2024-03-16T11:27:31.655 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-16T11:27:31.655 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --archive /home/teuthworker/archive/pdonnell-2024-03-15_13:00:41-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7602417 --name pdonnell-2024-03-15_13:00:41-fs-wip-batrick-testing-20240314.183243-distro-default-smithi --description fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/suites/dbench}} --owner scheduled_pdonnell@teuthology -- /home/teuthworker/archive/pdonnell-2024-03-15_13:00:41-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7602417/orig.config.yaml 2024-03-16T11:27:31.710 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-16T11:27:31.797 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/pdonnell-2024-03-15_13:00:41-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7602417 branch: wip-batrick-testing-20240314.183243 description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/suites/dbench}} email: pdonnell@redhat.com first_in_suite: false job_id: '7602417' kernel: client: sha1: distro kdb: true sha1: distro ktype: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: pdonnell-2024-03-15_13:00:41-fs-wip-batrick-testing-20240314.183243-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-batrick-testing-20240314.183243 ceph: cephfs: max_mds: 3 session_timeout: 300 standby_replay: true conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true mds_bal_replicate_threshold: 1 mds_export_ephemeral_random_max: 0.1 osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: cephsqlite lock renewal timeout: 900000 debug cephsqlite: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bluestore block size: 96636764160 bluestore compression mode: aggressive bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 osd shutdown pgref assert: true osd_max_omap_entries_per_request: 10 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - POOL_APP_NOT_ENABLED - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - MON_DOWN sha1: 3d78f5ba07bf18880172c07d4d7b4ac00c9d30af subvols: create: 2 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} ceph-fuse: client.0: mount_subvol_num: 0 check-counter: counters: mds: - mds.exported - mds.imported - min: 3 name: mds_cache.dir_update - min: 3 name: mds_cache.dir_update_receipt - mds.root_rsnaps - mds_server.req_mksnap_latency.avgcount - mds_server.req_rmsnap_latency.avgcount install: ceph: flavor: default sha1: 3d78f5ba07bf18880172c07d4d7b4ac00c9d30af kclient: client.0: mount_subvol_num: 1 mntopts: - ms_mode=legacy - nowsync syntax: v1 selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 - scontext=system_u:system_r:getty_t:s0 - scontext=system_u:system_r:getty_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-batrick-testing-20240314.183243 sha1: 3d78f5ba07bf18880172c07d4d7b4ac00c9d30af owner: scheduled_pdonnell@teuthology priority: 75 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - mds.a - mds.d - mds.g - mds.j - osd.0 - osd.3 - osd.6 - osd.9 - client.0 - - mon.b - mgr.y - mds.b - mds.e - mds.h - mds.k - osd.1 - osd.4 - osd.7 - osd.10 - - mon.c - mgr.z - mds.c - mds.f - mds.i - mds.l - osd.2 - osd.5 - osd.8 - osd.11 seed: 4770 sha1: 3d78f5ba07bf18880172c07d4d7b4ac00c9d30af sleep_before_teardown: 0 subset: 498/512 suite: fs suite_branch: wip-batrick-testing-20240314.183243 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 3d78f5ba07bf18880172c07d4d7b4ac00c9d30af syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi029.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDgj7nLjz/eATmFxlBE8SkOj64J+ZHgb7KgpfapFpAiVDePemMgh+wEjzllGImxykgPeb9oHGgbFSgKSubGOT4I= smithi152.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBODmpUUIT8s4S2XV01SxUGYFTYBPzWyZZP6Gtz6xLgKr1GoKR44RiyUzDqs9CeqaDwCNBMB8+wW3db/uSS77mp4= smithi154.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKELybuyIauRgYEU3ET5nVWAUfkL8SW3exbDQM8gO4XU8gg28OxV04H1Uz80XrozUoFzeXwIb3YOjjp4czJiImQ= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - cephadm: roleless: false - cephadm.shell: mon.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: mon.a: - ceph fs dump - ceph osd dump - fs.ready: timeout: 300 - kclient: null - exec: mon.a: - ceph fs set cephfs balance_automate false - ceph fs subvolumegroup pin cephfs qa random 0.10 - check-counter: {} - exec: mon.a: - ceph mgr module enable snap_schedule - ceph config set mgr mgr/snap_schedule/allow_m_granularity true - ceph config set mgr mgr/snap_schedule/dump_on_update true - ceph fs snap-schedule add --fs=cephfs --path=/ --snap_schedule=1m - ceph fs snap-schedule retention add --fs=cephfs --path=/ --retention-spec-or-period=6m3h - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - date +%s > START_TIME - full_sequential_finally: - exec: mon.a: - date +%s > END_TIME - START_TIME=$(cat START_TIME); END_TIME=$(cat END_TIME); DIFF_TIME=$((600-(END_TIME-START_TIME))); if [ "$DIFF_TIME" -gt 0 ]; then sleep "$DIFF_TIME"; fi - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - background_exec: mon.a: - while sleep 13; do ceph tell mds.cephfs:0 flush journal; done - workunit: clients: all: - suites/dbench.sh teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-15_13:00:41 tube: smithi user: pdonnell verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-16T11:27:31.798 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/qa; will attempt to use it 2024-03-16T11:27:31.798 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/qa/tasks 2024-03-16T11:27:31.799 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-16T11:27:31.800 INFO:teuthology.task.internal:Checking packages... 2024-03-16T11:27:31.824 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '3d78f5ba07bf18880172c07d4d7b4ac00c9d30af' 2024-03-16T11:27:31.824 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-16T11:27:31.824 INFO:teuthology.packaging:ref: None 2024-03-16T11:27:31.824 INFO:teuthology.packaging:tag: None 2024-03-16T11:27:31.824 INFO:teuthology.packaging:branch: wip-batrick-testing-20240314.183243 2024-03-16T11:27:31.824 INFO:teuthology.packaging:sha1: 3d78f5ba07bf18880172c07d4d7b4ac00c9d30af 2024-03-16T11:27:31.824 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-batrick-testing-20240314.183243 2024-03-16T11:27:31.966 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2162.g3d78f5ba 2024-03-16T11:27:31.968 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-16T11:27:31.977 INFO:teuthology.task.internal:no buildpackages task found 2024-03-16T11:27:31.977 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-16T11:27:31.990 INFO:teuthology.task.internal:Saving configuration 2024-03-16T11:27:32.010 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-16T11:27:32.021 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-16T11:27:32.047 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi029.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-03-15_13:00:41-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7602417', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-16 11:22:07.987657', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDgj7nLjz/eATmFxlBE8SkOj64J+ZHgb7KgpfapFpAiVDePemMgh+wEjzllGImxykgPeb9oHGgbFSgKSubGOT4I='} 2024-03-16T11:27:32.071 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi152.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-03-15_13:00:41-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7602417', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-16 11:22:07.990467', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBODmpUUIT8s4S2XV01SxUGYFTYBPzWyZZP6Gtz6xLgKr1GoKR44RiyUzDqs9CeqaDwCNBMB8+wW3db/uSS77mp4='} 2024-03-16T11:27:32.092 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi154.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-03-15_13:00:41-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7602417', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-16 11:22:07.989189', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKELybuyIauRgYEU3ET5nVWAUfkL8SW3exbDQM8gO4XU8gg28OxV04H1Uz80XrozUoFzeXwIb3YOjjp4czJiImQ='} 2024-03-16T11:27:32.092 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-16T11:27:32.103 INFO:teuthology.task.internal:roles: ubuntu@smithi029.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0'] 2024-03-16T11:27:32.103 INFO:teuthology.task.internal:roles: ubuntu@smithi152.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10'] 2024-03-16T11:27:32.103 INFO:teuthology.task.internal:roles: ubuntu@smithi154.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11'] 2024-03-16T11:27:32.103 INFO:teuthology.run_tasks:Running task console_log... 2024-03-16T11:27:32.209 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fba77bd6160>, signals=[15]) 2024-03-16T11:27:32.210 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-16T11:27:32.218 INFO:teuthology.task.internal:Opening connections... 2024-03-16T11:27:32.218 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi029.front.sepia.ceph.com 2024-03-16T11:27:32.220 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi029.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-16T11:27:32.327 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi152.front.sepia.ceph.com 2024-03-16T11:27:32.328 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi152.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-16T11:27:32.414 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi154.front.sepia.ceph.com 2024-03-16T11:27:32.415 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi154.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-16T11:27:32.502 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-16T11:27:32.510 DEBUG:teuthology.orchestra.run.smithi029:> uname -m 2024-03-16T11:27:32.528 INFO:teuthology.orchestra.run.smithi029.stdout:x86_64 2024-03-16T11:27:32.528 DEBUG:teuthology.orchestra.run.smithi029:> cat /etc/os-release 2024-03-16T11:27:32.587 INFO:teuthology.orchestra.run.smithi029.stdout:NAME="CentOS Stream" 2024-03-16T11:27:32.587 INFO:teuthology.orchestra.run.smithi029.stdout:VERSION="9" 2024-03-16T11:27:32.587 INFO:teuthology.orchestra.run.smithi029.stdout:ID="centos" 2024-03-16T11:27:32.587 INFO:teuthology.orchestra.run.smithi029.stdout:ID_LIKE="rhel fedora" 2024-03-16T11:27:32.587 INFO:teuthology.orchestra.run.smithi029.stdout:VERSION_ID="9" 2024-03-16T11:27:32.587 INFO:teuthology.orchestra.run.smithi029.stdout:PLATFORM_ID="platform:el9" 2024-03-16T11:27:32.587 INFO:teuthology.orchestra.run.smithi029.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-16T11:27:32.587 INFO:teuthology.orchestra.run.smithi029.stdout:ANSI_COLOR="0;31" 2024-03-16T11:27:32.587 INFO:teuthology.orchestra.run.smithi029.stdout:LOGO="fedora-logo-icon" 2024-03-16T11:27:32.587 INFO:teuthology.orchestra.run.smithi029.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-16T11:27:32.587 INFO:teuthology.orchestra.run.smithi029.stdout:HOME_URL="https://centos.org/" 2024-03-16T11:27:32.588 INFO:teuthology.orchestra.run.smithi029.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-16T11:27:32.588 INFO:teuthology.orchestra.run.smithi029.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-16T11:27:32.588 INFO:teuthology.orchestra.run.smithi029.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-16T11:27:32.588 INFO:teuthology.lock.ops:Updating smithi029.front.sepia.ceph.com on lock server 2024-03-16T11:27:32.608 DEBUG:teuthology.orchestra.run.smithi152:> uname -m 2024-03-16T11:27:32.626 INFO:teuthology.orchestra.run.smithi152.stdout:x86_64 2024-03-16T11:27:32.626 DEBUG:teuthology.orchestra.run.smithi152:> cat /etc/os-release 2024-03-16T11:27:32.682 INFO:teuthology.orchestra.run.smithi152.stdout:NAME="CentOS Stream" 2024-03-16T11:27:32.683 INFO:teuthology.orchestra.run.smithi152.stdout:VERSION="9" 2024-03-16T11:27:32.683 INFO:teuthology.orchestra.run.smithi152.stdout:ID="centos" 2024-03-16T11:27:32.683 INFO:teuthology.orchestra.run.smithi152.stdout:ID_LIKE="rhel fedora" 2024-03-16T11:27:32.683 INFO:teuthology.orchestra.run.smithi152.stdout:VERSION_ID="9" 2024-03-16T11:27:32.683 INFO:teuthology.orchestra.run.smithi152.stdout:PLATFORM_ID="platform:el9" 2024-03-16T11:27:32.683 INFO:teuthology.orchestra.run.smithi152.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-16T11:27:32.683 INFO:teuthology.orchestra.run.smithi152.stdout:ANSI_COLOR="0;31" 2024-03-16T11:27:32.683 INFO:teuthology.orchestra.run.smithi152.stdout:LOGO="fedora-logo-icon" 2024-03-16T11:27:32.683 INFO:teuthology.orchestra.run.smithi152.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-16T11:27:32.683 INFO:teuthology.orchestra.run.smithi152.stdout:HOME_URL="https://centos.org/" 2024-03-16T11:27:32.683 INFO:teuthology.orchestra.run.smithi152.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-16T11:27:32.684 INFO:teuthology.orchestra.run.smithi152.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-16T11:27:32.684 INFO:teuthology.orchestra.run.smithi152.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-16T11:27:32.684 INFO:teuthology.lock.ops:Updating smithi152.front.sepia.ceph.com on lock server 2024-03-16T11:27:32.699 DEBUG:teuthology.orchestra.run.smithi154:> uname -m 2024-03-16T11:27:32.717 INFO:teuthology.orchestra.run.smithi154.stdout:x86_64 2024-03-16T11:27:32.717 DEBUG:teuthology.orchestra.run.smithi154:> cat /etc/os-release 2024-03-16T11:27:32.774 INFO:teuthology.orchestra.run.smithi154.stdout:NAME="CentOS Stream" 2024-03-16T11:27:32.774 INFO:teuthology.orchestra.run.smithi154.stdout:VERSION="9" 2024-03-16T11:27:32.774 INFO:teuthology.orchestra.run.smithi154.stdout:ID="centos" 2024-03-16T11:27:32.775 INFO:teuthology.orchestra.run.smithi154.stdout:ID_LIKE="rhel fedora" 2024-03-16T11:27:32.775 INFO:teuthology.orchestra.run.smithi154.stdout:VERSION_ID="9" 2024-03-16T11:27:32.775 INFO:teuthology.orchestra.run.smithi154.stdout:PLATFORM_ID="platform:el9" 2024-03-16T11:27:32.775 INFO:teuthology.orchestra.run.smithi154.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-16T11:27:32.775 INFO:teuthology.orchestra.run.smithi154.stdout:ANSI_COLOR="0;31" 2024-03-16T11:27:32.775 INFO:teuthology.orchestra.run.smithi154.stdout:LOGO="fedora-logo-icon" 2024-03-16T11:27:32.775 INFO:teuthology.orchestra.run.smithi154.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-16T11:27:32.775 INFO:teuthology.orchestra.run.smithi154.stdout:HOME_URL="https://centos.org/" 2024-03-16T11:27:32.775 INFO:teuthology.orchestra.run.smithi154.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-16T11:27:32.775 INFO:teuthology.orchestra.run.smithi154.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-16T11:27:32.775 INFO:teuthology.orchestra.run.smithi154.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-16T11:27:32.775 INFO:teuthology.lock.ops:Updating smithi154.front.sepia.ceph.com on lock server 2024-03-16T11:27:32.800 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-16T11:27:32.816 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-16T11:27:32.824 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-16T11:27:32.825 DEBUG:teuthology.orchestra.run.smithi029:> test '!' -e /home/ubuntu/cephtest 2024-03-16T11:27:32.828 DEBUG:teuthology.orchestra.run.smithi152:> test '!' -e /home/ubuntu/cephtest 2024-03-16T11:27:32.831 DEBUG:teuthology.orchestra.run.smithi154:> test '!' -e /home/ubuntu/cephtest 2024-03-16T11:27:32.847 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-16T11:27:32.856 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-16T11:27:32.857 DEBUG:teuthology.orchestra.run.smithi029:> test -z $(ls -A /var/lib/ceph) 2024-03-16T11:27:32.885 DEBUG:teuthology.orchestra.run.smithi152:> test -z $(ls -A /var/lib/ceph) 2024-03-16T11:27:32.888 DEBUG:teuthology.orchestra.run.smithi154:> test -z $(ls -A /var/lib/ceph) 2024-03-16T11:27:32.986 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-16T11:27:33.064 INFO:teuthology.run_tasks:Running task kernel... 2024-03-16T11:27:33.081 INFO:teuthology.task.kernel:normalize config orig: {'client': {'sha1': 'distro'}, 'kdb': True, 'sha1': 'distro'} 2024-03-16T11:27:33.081 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'sha1': 'distro'}} 2024-03-16T11:27:33.082 INFO:teuthology.task.kernel:config {'client.0': {'sha1': 'distro'}}, timeout 300 2024-03-16T11:27:33.082 DEBUG:teuthology.orchestra.run.smithi029:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-16T11:27:33.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-16T11:27:33.099 DEBUG:teuthology.orchestra.run.smithi029:> uname -r 2024-03-16T11:27:33.158 INFO:teuthology.orchestra.run.smithi029.stdout:5.14.0-361.el9.x86_64 2024-03-16T11:27:33.158 INFO:teuthology.task.kernel:Running kernel on smithi029: 5.14.0-361.el9.x86_64 2024-03-16T11:27:33.158 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum install -y kernel 2024-03-16T11:27:36.264 INFO:teuthology.orchestra.run.smithi029.stdout:CentOS-9 - AppStream 13 MB/s | 6.7 MB 00:00 2024-03-16T11:27:37.717 INFO:teuthology.orchestra.run.smithi029.stdout:CentOS-9 - BaseOS 6.7 MB/s | 2.0 MB 00:00 2024-03-16T11:27:39.309 INFO:teuthology.orchestra.run.smithi029.stdout:CentOS Stream 9 - BaseOS 6.4 MB/s | 8.0 MB 00:01 2024-03-16T11:27:41.678 INFO:teuthology.orchestra.run.smithi029.stdout:CentOS Stream 9 - AppStream 15 MB/s | 19 MB 00:01 2024-03-16T11:27:45.992 INFO:teuthology.orchestra.run.smithi029.stdout:CentOS Stream 9 - CRB 8.9 MB/s | 6.1 MB 00:00 2024-03-16T11:27:47.691 INFO:teuthology.orchestra.run.smithi029.stdout:CentOS Stream 9 - Extras packages 29 kB/s | 15 kB 00:00 2024-03-16T11:28:00.018 INFO:teuthology.orchestra.run.smithi029.stdout:Extra Packages for Enterprise Linux 1.7 MB/s | 21 MB 00:12 2024-03-16T11:28:05.152 INFO:teuthology.orchestra.run.smithi029.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2024-03-16T11:28:07.486 INFO:teuthology.orchestra.run.smithi029.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-16T11:28:07.487 INFO:teuthology.orchestra.run.smithi029.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-16T11:28:07.596 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-03-16T11:28:07.611 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-03-16T11:28:07.611 INFO:teuthology.orchestra.run.smithi029.stdout: Package Architecture Version Repository Size 2024-03-16T11:28:07.612 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-03-16T11:28:07.612 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2024-03-16T11:28:07.612 INFO:teuthology.orchestra.run.smithi029.stdout: kernel x86_64 5.14.0-427.el9 baseos 6.0 M 2024-03-16T11:28:07.612 INFO:teuthology.orchestra.run.smithi029.stdout:Installing dependencies: 2024-03-16T11:28:07.612 INFO:teuthology.orchestra.run.smithi029.stdout: kernel-core x86_64 5.14.0-427.el9 baseos 21 M 2024-03-16T11:28:07.612 INFO:teuthology.orchestra.run.smithi029.stdout: kernel-modules x86_64 5.14.0-427.el9 baseos 39 M 2024-03-16T11:28:07.612 INFO:teuthology.orchestra.run.smithi029.stdout: kernel-modules-core x86_64 5.14.0-427.el9 baseos 33 M 2024-03-16T11:28:07.612 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-16T11:28:07.612 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2024-03-16T11:28:07.612 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-03-16T11:28:07.612 INFO:teuthology.orchestra.run.smithi029.stdout:Install 4 Packages 2024-03-16T11:28:07.612 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-16T11:28:07.613 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 100 M 2024-03-16T11:28:07.613 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 123 M 2024-03-16T11:28:07.613 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2024-03-16T11:28:08.381 INFO:teuthology.orchestra.run.smithi029.stdout:(1/4): kernel-5.14.0-427.el9.x86_64.rpm 12 MB/s | 6.0 MB 00:00 2024-03-16T11:28:08.999 INFO:teuthology.orchestra.run.smithi029.stdout:(2/4): kernel-core-5.14.0-427.el9.x86_64.rpm 19 MB/s | 21 MB 00:01 2024-03-16T11:28:09.883 INFO:teuthology.orchestra.run.smithi029.stdout:(3/4): kernel-modules-5.14.0-427.el9.x86_64.rpm 20 MB/s | 39 MB 00:01 2024-03-16T11:28:10.941 INFO:teuthology.orchestra.run.smithi029.stdout:(4/4): kernel-modules-core-5.14.0-427.el9.x86_6 13 MB/s | 33 MB 00:02 2024-03-16T11:28:10.942 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2024-03-16T11:28:10.942 INFO:teuthology.orchestra.run.smithi029.stdout:Total 30 MB/s | 100 MB 00:03 2024-03-16T11:28:11.863 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2024-03-16T11:28:12.332 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2024-03-16T11:28:12.333 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2024-03-16T11:28:13.722 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2024-03-16T11:28:13.722 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2024-03-16T11:28:16.594 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2024-03-16T11:28:17.379 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : kernel-modules-core-5.14.0-427.el9.x86_64 1/4 2024-03-16T11:28:17.573 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-16T11:28:20.647 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-16T11:28:20.958 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-16T11:28:26.059 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-16T11:28:26.208 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : kernel-5.14.0-427.el9.x86_64 4/4 2024-03-16T11:28:31.343 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-16T11:29:04.533 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 4/4 2024-03-16T11:29:04.533 INFO:teuthology.orchestra.run.smithi029.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-16T11:29:04.533 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-16T11:29:04.620 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 4/4 2024-03-16T11:29:05.143 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: kernel-5.14.0-427.el9.x86_64 4/4 2024-03-16T11:29:05.144 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : kernel-5.14.0-427.el9.x86_64 1/4 2024-03-16T11:29:05.144 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-16T11:29:05.144 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-16T11:29:05.604 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-16T11:29:05.605 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-16T11:29:05.605 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2024-03-16T11:29:05.605 INFO:teuthology.orchestra.run.smithi029.stdout: kernel-5.14.0-427.el9.x86_64 2024-03-16T11:29:05.605 INFO:teuthology.orchestra.run.smithi029.stdout: kernel-core-5.14.0-427.el9.x86_64 2024-03-16T11:29:05.605 INFO:teuthology.orchestra.run.smithi029.stdout: kernel-modules-5.14.0-427.el9.x86_64 2024-03-16T11:29:05.605 INFO:teuthology.orchestra.run.smithi029.stdout: kernel-modules-core-5.14.0-427.el9.x86_64 2024-03-16T11:29:05.605 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-16T11:29:05.605 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-03-16T11:29:06.344 DEBUG:teuthology.orchestra.run.smithi029:> rpm -q kernel | sort -rV | head -n 1 2024-03-16T11:29:06.528 INFO:teuthology.orchestra.run.smithi029.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-16T11:29:06.528 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-16T11:29:06.528 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-427.el9.x86_64 2024-03-16T11:29:06.528 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-16T11:29:06.528 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-16T11:29:06.529 INFO:teuthology.task.kernel:Installing distro kernel on client.0... 2024-03-16T11:29:06.529 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi029.front.sepia.ceph.com, path=None, version=distro) 2024-03-16T11:29:06.529 DEBUG:teuthology.orchestra.run.smithi029:> rpm -q kernel | sort -rV | head -n 1 2024-03-16T11:29:06.654 INFO:teuthology.orchestra.run.smithi029.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-16T11:29:06.655 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-16T11:29:06.655 DEBUG:teuthology.orchestra.run.smithi029:> sudo rpm -qi grub2-tools 2024-03-16T11:29:06.698 INFO:teuthology.orchestra.run.smithi029.stdout:Name : grub2-tools 2024-03-16T11:29:06.699 INFO:teuthology.orchestra.run.smithi029.stdout:Epoch : 1 2024-03-16T11:29:06.699 INFO:teuthology.orchestra.run.smithi029.stdout:Version : 2.06 2024-03-16T11:29:06.699 INFO:teuthology.orchestra.run.smithi029.stdout:Release : 68.el9 2024-03-16T11:29:06.699 INFO:teuthology.orchestra.run.smithi029.stdout:Architecture: x86_64 2024-03-16T11:29:06.699 INFO:teuthology.orchestra.run.smithi029.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-16T11:29:06.699 INFO:teuthology.orchestra.run.smithi029.stdout:Group : Unspecified 2024-03-16T11:29:06.699 INFO:teuthology.orchestra.run.smithi029.stdout:Size : 8264254 2024-03-16T11:29:06.699 INFO:teuthology.orchestra.run.smithi029.stdout:License : GPLv3+ 2024-03-16T11:29:06.699 INFO:teuthology.orchestra.run.smithi029.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-16T11:29:06.699 INFO:teuthology.orchestra.run.smithi029.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-16T11:29:06.699 INFO:teuthology.orchestra.run.smithi029.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-16T11:29:06.699 INFO:teuthology.orchestra.run.smithi029.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-16T11:29:06.699 INFO:teuthology.orchestra.run.smithi029.stdout:Packager : builder@centos.org 2024-03-16T11:29:06.699 INFO:teuthology.orchestra.run.smithi029.stdout:Vendor : CentOS 2024-03-16T11:29:06.700 INFO:teuthology.orchestra.run.smithi029.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-16T11:29:06.700 INFO:teuthology.orchestra.run.smithi029.stdout:Summary : Support tools for GRUB. 2024-03-16T11:29:06.700 INFO:teuthology.orchestra.run.smithi029.stdout:Description : 2024-03-16T11:29:06.700 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-16T11:29:06.700 INFO:teuthology.orchestra.run.smithi029.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-16T11:29:06.700 INFO:teuthology.orchestra.run.smithi029.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-16T11:29:06.700 INFO:teuthology.orchestra.run.smithi029.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-16T11:29:06.700 INFO:teuthology.orchestra.run.smithi029.stdout:hardware devices. 2024-03-16T11:29:06.700 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-16T11:29:06.700 INFO:teuthology.orchestra.run.smithi029.stdout:This subpackage provides tools for support of all platforms. 2024-03-16T11:29:06.701 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-16T11:29:06.701 INFO:teuthology.task.kernel:Updating grub on smithi029 to boot 5.14.0-427.el9.x86_64 2024-03-16T11:29:06.701 DEBUG:teuthology.orchestra.run.smithi029:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-16T11:29:07.452 INFO:teuthology.orchestra.run.smithi029.stderr:Generating grub configuration file ... 2024-03-16T11:29:09.129 INFO:teuthology.orchestra.run.smithi029.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-16T11:29:09.154 INFO:teuthology.orchestra.run.smithi029.stderr:done 2024-03-16T11:29:09.157 DEBUG:teuthology.orchestra.run.smithi029:> mktemp 2024-03-16T11:29:09.171 INFO:teuthology.orchestra.run.smithi029.stdout:/tmp/tmp.4eqEuZbhz4 2024-03-16T11:29:09.172 DEBUG:teuthology.orchestra.run.smithi029:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.4eqEuZbhz4 2024-03-16T11:29:09.243 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod 0666 /tmp/tmp.4eqEuZbhz4 2024-03-16T11:29:09.385 DEBUG:teuthology.orchestra.remote:smithi029:/tmp/tmp.4eqEuZbhz4 is 6KB 2024-03-16T11:29:09.434 DEBUG:teuthology.orchestra.run.smithi029:> rm -fr /tmp/tmp.4eqEuZbhz4 2024-03-16T11:29:09.449 DEBUG:teuthology.orchestra.run.smithi029:> sudo /bin/ls /boot/loader/entries || true 2024-03-16T11:29:09.519 INFO:teuthology.orchestra.run.smithi029.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-16T11:29:09.520 INFO:teuthology.orchestra.run.smithi029.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-16T11:29:09.520 INFO:teuthology.orchestra.run.smithi029.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-16T11:29:09.520 INFO:teuthology.orchestra.run.smithi029.stdout:15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64.conf 2024-03-16T11:29:09.521 DEBUG:teuthology.orchestra.run.smithi029:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64 2024-03-16T11:29:09.688 DEBUG:teuthology.orchestra.run.smithi029:> sudo shutdown -r now 2024-03-16T11:29:39.691 INFO:teuthology.task.kernel:Checking client client.0 for new kernel version... 2024-03-16T11:29:39.691 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi029.front.sepia.ceph.com' 2024-03-16T11:29:39.692 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi029.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-16T11:29:58.050 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.29 2024-03-16T11:30:07.051 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi029.front.sepia.ceph.com' 2024-03-16T11:30:07.052 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi029.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-16T11:31:07.054 DEBUG:teuthology.orchestra.remote:timed out 2024-03-16T11:31:19.055 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi029.front.sepia.ceph.com' 2024-03-16T11:31:19.056 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi029.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-16T11:31:19.493 DEBUG:teuthology.orchestra.run.smithi029:> true 2024-03-16T11:31:19.930 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi029.front.sepia.ceph.com' 2024-03-16T11:31:19.930 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "5.14.0-427.el9.x86_64"... 2024-03-16T11:31:19.930 DEBUG:teuthology.orchestra.run.smithi029:> uname -r 2024-03-16T11:31:19.947 INFO:teuthology.orchestra.run.smithi029.stdout:5.14.0-427.el9.x86_64 2024-03-16T11:31:19.947 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-427.el9.x86_64 vs 5.14.0-427.el9.x86_64 2024-03-16T11:31:19.947 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-16T11:31:19.947 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-16T11:31:20.948 DEBUG:teuthology.parallel:result is None 2024-03-16T11:31:20.948 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-16T11:31:20.956 INFO:teuthology.task.internal:Creating test directory... 2024-03-16T11:31:20.956 DEBUG:teuthology.orchestra.run.smithi029:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-16T11:31:20.959 DEBUG:teuthology.orchestra.run.smithi152:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-16T11:31:20.962 DEBUG:teuthology.orchestra.run.smithi154:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-16T11:31:20.980 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-16T11:31:20.986 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-16T11:31:20.997 INFO:teuthology.task.internal:Creating archive directory... 2024-03-16T11:31:20.997 DEBUG:teuthology.orchestra.run.smithi029:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-16T11:31:21.017 DEBUG:teuthology.orchestra.run.smithi152:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-16T11:31:21.020 DEBUG:teuthology.orchestra.run.smithi154:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-16T11:31:21.136 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-16T11:31:21.144 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-16T11:31:21.144 DEBUG:teuthology.orchestra.run.smithi152:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-16T11:31:21.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-16T11:31:21.160 DEBUG:teuthology.orchestra.run.smithi154:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-16T11:31:21.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-16T11:31:21.177 DEBUG:teuthology.orchestra.run.smithi029:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-16T11:31:21.179 DEBUG:teuthology.orchestra.run.smithi152:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-16T11:31:21.202 DEBUG:teuthology.orchestra.run.smithi154:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-16T11:31:21.254 INFO:teuthology.orchestra.run.smithi152.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-16T11:31:21.259 INFO:teuthology.orchestra.run.smithi154.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-16T11:31:21.269 INFO:teuthology.orchestra.run.smithi029.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-16T11:31:21.275 INFO:teuthology.orchestra.run.smithi152.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-16T11:31:21.278 INFO:teuthology.orchestra.run.smithi154.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-16T11:31:21.314 INFO:teuthology.orchestra.run.smithi029.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-16T11:31:21.315 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-16T11:31:21.323 INFO:teuthology.task.internal:Configuring sudo... 2024-03-16T11:31:21.323 DEBUG:teuthology.orchestra.run.smithi029:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-16T11:31:21.326 DEBUG:teuthology.orchestra.run.smithi152:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-16T11:31:21.328 DEBUG:teuthology.orchestra.run.smithi154:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-16T11:31:21.357 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-16T11:31:21.367 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-16T11:31:21.367 DEBUG:teuthology.orchestra.run.smithi029:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-16T11:31:21.396 DEBUG:teuthology.orchestra.run.smithi152:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-16T11:31:21.399 DEBUG:teuthology.orchestra.run.smithi154:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-16T11:31:21.414 DEBUG:teuthology.orchestra.run.smithi029:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-16T11:31:21.494 DEBUG:teuthology.orchestra.run.smithi029:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-16T11:31:21.554 DEBUG:teuthology.orchestra.run.smithi029:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-16T11:31:21.596 DEBUG:teuthology.orchestra.run.smithi029:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-16T11:31:21.639 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-03-16T11:31:21.639 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-16T11:31:21.707 DEBUG:teuthology.orchestra.run.smithi152:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-16T11:31:21.752 DEBUG:teuthology.orchestra.run.smithi152:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-16T11:31:21.802 DEBUG:teuthology.orchestra.run.smithi152:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-16T11:31:21.886 DEBUG:teuthology.orchestra.run.smithi152:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-16T11:31:21.931 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-03-16T11:31:21.931 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-16T11:31:21.999 DEBUG:teuthology.orchestra.run.smithi154:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-16T11:31:22.043 DEBUG:teuthology.orchestra.run.smithi154:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-16T11:31:22.107 DEBUG:teuthology.orchestra.run.smithi154:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-16T11:31:22.149 DEBUG:teuthology.orchestra.run.smithi154:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-16T11:31:22.194 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-03-16T11:31:22.194 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-16T11:31:22.264 DEBUG:teuthology.orchestra.run.smithi029:> sudo service rsyslog restart 2024-03-16T11:31:22.267 DEBUG:teuthology.orchestra.run.smithi152:> sudo service rsyslog restart 2024-03-16T11:31:22.269 DEBUG:teuthology.orchestra.run.smithi154:> sudo service rsyslog restart 2024-03-16T11:31:22.307 INFO:teuthology.orchestra.run.smithi152.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-16T11:31:22.307 INFO:teuthology.orchestra.run.smithi029.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-16T11:31:22.345 INFO:teuthology.orchestra.run.smithi154.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-16T11:31:22.658 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-16T11:31:22.665 INFO:teuthology.task.internal:Starting timer... 2024-03-16T11:31:22.665 INFO:teuthology.run_tasks:Running task pcp... 2024-03-16T11:31:22.675 INFO:teuthology.run_tasks:Running task selinux... 2024-03-16T11:31:22.683 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:getty_t:s0', 'scontext=system_u:system_r:getty_t:s0']} 2024-03-16T11:31:22.684 DEBUG:teuthology.orchestra.run.smithi029:> sudo service auditd rotate 2024-03-16T11:31:22.742 INFO:teuthology.orchestra.run.smithi029.stdout:Rotating logs: 2024-03-16T11:31:22.744 DEBUG:teuthology.orchestra.run.smithi152:> sudo service auditd rotate 2024-03-16T11:31:22.817 INFO:teuthology.orchestra.run.smithi152.stdout:Rotating logs: 2024-03-16T11:31:22.819 DEBUG:teuthology.orchestra.run.smithi154:> sudo service auditd rotate 2024-03-16T11:31:22.868 INFO:teuthology.orchestra.run.smithi154.stdout:Rotating logs: 2024-03-16T11:31:22.870 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-16T11:31:22.870 DEBUG:teuthology.orchestra.run.smithi029:> /usr/sbin/getenforce 2024-03-16T11:31:22.895 INFO:teuthology.orchestra.run.smithi029.stdout:Permissive 2024-03-16T11:31:22.895 DEBUG:teuthology.orchestra.run.smithi152:> /usr/sbin/getenforce 2024-03-16T11:31:22.914 INFO:teuthology.orchestra.run.smithi152.stdout:Permissive 2024-03-16T11:31:22.915 DEBUG:teuthology.orchestra.run.smithi154:> /usr/sbin/getenforce 2024-03-16T11:31:22.935 INFO:teuthology.orchestra.run.smithi154.stdout:Permissive 2024-03-16T11:31:22.935 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi029.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi152.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi154.front.sepia.ceph.com': 'permissive'} 2024-03-16T11:31:22.935 DEBUG:teuthology.orchestra.run.smithi029:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-16T11:31:22.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-16T11:31:22.965 DEBUG:teuthology.orchestra.run.smithi152:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-16T11:31:22.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-16T11:31:22.994 DEBUG:teuthology.orchestra.run.smithi154:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-16T11:31:23.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-16T11:31:23.021 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-16T11:31:23.021 DEBUG:teuthology.orchestra.run.smithi029:> sudo /usr/sbin/setenforce permissive 2024-03-16T11:31:23.055 DEBUG:teuthology.orchestra.run.smithi152:> sudo /usr/sbin/setenforce permissive 2024-03-16T11:31:23.092 DEBUG:teuthology.orchestra.run.smithi154:> sudo /usr/sbin/setenforce permissive 2024-03-16T11:31:23.129 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-16T11:31:23.138 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-16T11:31:23.155 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-16T11:31:23.157 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi029.front.sepia.ceph.com,smithi152.front.sepia.ceph.com,smithi154.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-16T11:35:50.923 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi029.front.sepia.ceph.com'), Remote(name='ubuntu@smithi152.front.sepia.ceph.com'), Remote(name='ubuntu@smithi154.front.sepia.ceph.com')] 2024-03-16T11:35:50.924 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi029.front.sepia.ceph.com' 2024-03-16T11:35:50.925 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi029.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-16T11:35:51.015 DEBUG:teuthology.orchestra.run.smithi029:> true 2024-03-16T11:35:51.092 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi029.front.sepia.ceph.com' 2024-03-16T11:35:51.092 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi152.front.sepia.ceph.com' 2024-03-16T11:35:51.093 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi152.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-16T11:35:51.177 DEBUG:teuthology.orchestra.run.smithi152:> true 2024-03-16T11:35:51.244 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi152.front.sepia.ceph.com' 2024-03-16T11:35:51.245 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi154.front.sepia.ceph.com' 2024-03-16T11:35:51.245 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi154.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-16T11:35:51.333 DEBUG:teuthology.orchestra.run.smithi154:> true 2024-03-16T11:35:51.401 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi154.front.sepia.ceph.com' 2024-03-16T11:35:51.402 INFO:teuthology.run_tasks:Running task clock... 2024-03-16T11:35:51.415 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-16T11:35:51.416 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-16T11:35:51.416 DEBUG:teuthology.orchestra.run.smithi029:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-16T11:35:51.418 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-16T11:35:51.418 DEBUG:teuthology.orchestra.run.smithi152:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-16T11:35:51.421 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-16T11:35:51.421 DEBUG:teuthology.orchestra.run.smithi154:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-16T11:35:51.447 INFO:teuthology.orchestra.run.smithi029.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-16T11:35:51.449 INFO:teuthology.orchestra.run.smithi152.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-16T11:35:51.464 INFO:teuthology.orchestra.run.smithi029.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-16T11:35:51.466 INFO:teuthology.orchestra.run.smithi152.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-16T11:35:51.473 INFO:teuthology.orchestra.run.smithi154.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-16T11:35:51.490 INFO:teuthology.orchestra.run.smithi154.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-16T11:35:51.493 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: ntpd: command not found 2024-03-16T11:35:51.495 INFO:teuthology.orchestra.run.smithi152.stderr:sudo: ntpd: command not found 2024-03-16T11:35:51.507 INFO:teuthology.orchestra.run.smithi029.stdout:506 Cannot talk to daemon 2024-03-16T11:35:51.508 INFO:teuthology.orchestra.run.smithi152.stdout:506 Cannot talk to daemon 2024-03-16T11:35:51.520 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: ntpd: command not found 2024-03-16T11:35:51.523 INFO:teuthology.orchestra.run.smithi029.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-16T11:35:51.525 INFO:teuthology.orchestra.run.smithi152.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-16T11:35:51.533 INFO:teuthology.orchestra.run.smithi154.stdout:506 Cannot talk to daemon 2024-03-16T11:35:51.540 INFO:teuthology.orchestra.run.smithi029.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-16T11:35:51.541 INFO:teuthology.orchestra.run.smithi152.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-16T11:35:51.549 INFO:teuthology.orchestra.run.smithi154.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-16T11:35:51.566 INFO:teuthology.orchestra.run.smithi154.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-16T11:35:51.585 INFO:teuthology.orchestra.run.smithi152.stderr:bash: line 1: ntpq: command not found 2024-03-16T11:35:51.589 INFO:teuthology.orchestra.run.smithi152.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-16T11:35:51.590 INFO:teuthology.orchestra.run.smithi152.stdout:=============================================================================== 2024-03-16T11:35:51.590 INFO:teuthology.orchestra.run.smithi152.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-16T11:35:51.590 INFO:teuthology.orchestra.run.smithi152.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-16T11:35:51.590 INFO:teuthology.orchestra.run.smithi152.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-16T11:35:51.590 INFO:teuthology.orchestra.run.smithi152.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-16T11:35:51.591 INFO:teuthology.orchestra.run.smithi029.stderr:bash: line 1: ntpq: command not found 2024-03-16T11:35:51.594 INFO:teuthology.orchestra.run.smithi029.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-16T11:35:51.594 INFO:teuthology.orchestra.run.smithi029.stdout:=============================================================================== 2024-03-16T11:35:51.594 INFO:teuthology.orchestra.run.smithi029.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-16T11:35:51.594 INFO:teuthology.orchestra.run.smithi029.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-16T11:35:51.594 INFO:teuthology.orchestra.run.smithi029.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-16T11:35:51.594 INFO:teuthology.orchestra.run.smithi029.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-16T11:35:51.609 INFO:teuthology.orchestra.run.smithi154.stderr:bash: line 1: ntpq: command not found 2024-03-16T11:35:51.613 INFO:teuthology.orchestra.run.smithi154.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-16T11:35:51.613 INFO:teuthology.orchestra.run.smithi154.stdout:=============================================================================== 2024-03-16T11:35:51.613 INFO:teuthology.orchestra.run.smithi154.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-16T11:35:51.613 INFO:teuthology.orchestra.run.smithi154.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-16T11:35:51.613 INFO:teuthology.orchestra.run.smithi154.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-16T11:35:51.613 INFO:teuthology.orchestra.run.smithi154.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-16T11:35:51.614 INFO:teuthology.run_tasks:Running task install... 2024-03-16T11:35:51.622 DEBUG:teuthology.task.install:project ceph 2024-03-16T11:35:51.622 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '3d78f5ba07bf18880172c07d4d7b4ac00c9d30af'}} 2024-03-16T11:35:51.623 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '3d78f5ba07bf18880172c07d4d7b4ac00c9d30af'} 2024-03-16T11:35:51.623 INFO:teuthology.task.install:Using flavor: default 2024-03-16T11:35:51.631 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-03-16T11:35:51.631 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-03-16T11:35:51.631 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '3d78f5ba07bf18880172c07d4d7b4ac00c9d30af', 'tag': None, 'wait_for_package': False} 2024-03-16T11:35:51.631 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3d78f5ba07bf18880172c07d4d7b4ac00c9d30af 2024-03-16T11:35:51.633 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '3d78f5ba07bf18880172c07d4d7b4ac00c9d30af', 'tag': None, 'wait_for_package': False} 2024-03-16T11:35:51.633 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3d78f5ba07bf18880172c07d4d7b4ac00c9d30af 2024-03-16T11:35:51.634 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '3d78f5ba07bf18880172c07d4d7b4ac00c9d30af', 'tag': None, 'wait_for_package': False} 2024-03-16T11:35:51.634 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3d78f5ba07bf18880172c07d4d7b4ac00c9d30af 2024-03-16T11:35:51.863 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/flavors/default/ 2024-03-16T11:35:51.863 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2162.g3d78f5ba 2024-03-16T11:35:51.866 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/flavors/default/ 2024-03-16T11:35:51.867 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2162.g3d78f5ba 2024-03-16T11:35:51.867 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/flavors/default/ 2024-03-16T11:35:51.868 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2162.g3d78f5ba 2024-03-16T11:35:51.974 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-16T11:35:51.975 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-03-16T11:35:51.975 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-16T11:35:51.983 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-16T11:35:51.984 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-03-16T11:35:51.984 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-16T11:35:52.002 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-16T11:35:52.003 DEBUG:teuthology.orchestra.run.smithi029:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-16T11:35:52.011 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-16T11:35:52.012 DEBUG:teuthology.orchestra.run.smithi154:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-16T11:35:52.036 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-16T11:35:52.037 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-03-16T11:35:52.037 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-16T11:35:52.064 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-16T11:35:52.065 DEBUG:teuthology.orchestra.run.smithi152:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-16T11:35:52.071 DEBUG:teuthology.orchestra.run.smithi029:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-16T11:35:52.079 DEBUG:teuthology.orchestra.run.smithi154:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-16T11:35:52.132 DEBUG:teuthology.orchestra.run.smithi152:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-16T11:35:52.151 DEBUG:teuthology.orchestra.run.smithi029:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-16T11:35:52.159 DEBUG:teuthology.orchestra.run.smithi154:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-16T11:35:52.212 DEBUG:teuthology.orchestra.run.smithi152:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-16T11:35:52.217 INFO:teuthology.orchestra.run.smithi029.stdout:check_obsoletes = 1 2024-03-16T11:35:52.219 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum clean all 2024-03-16T11:35:52.225 INFO:teuthology.orchestra.run.smithi154.stdout:check_obsoletes = 1 2024-03-16T11:35:52.227 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum clean all 2024-03-16T11:35:52.278 INFO:teuthology.orchestra.run.smithi152.stdout:check_obsoletes = 1 2024-03-16T11:35:52.280 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum clean all 2024-03-16T11:35:52.462 INFO:teuthology.orchestra.run.smithi154.stdout:74 files removed 2024-03-16T11:35:52.467 INFO:teuthology.orchestra.run.smithi029.stdout:74 files removed 2024-03-16T11:35:52.490 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-16T11:35:52.494 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-16T11:35:52.513 INFO:teuthology.orchestra.run.smithi152.stdout:74 files removed 2024-03-16T11:35:52.541 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-16T11:35:53.140 INFO:teuthology.orchestra.run.smithi152.stdout:ceph packages for x86_64 269 kB/s | 83 kB 00:00 2024-03-16T11:35:53.210 INFO:teuthology.orchestra.run.smithi154.stdout:ceph packages for x86_64 194 kB/s | 83 kB 00:00 2024-03-16T11:35:53.234 INFO:teuthology.orchestra.run.smithi029.stdout:ceph packages for x86_64 199 kB/s | 83 kB 00:00 2024-03-16T11:35:53.527 INFO:teuthology.orchestra.run.smithi154.stdout:ceph noarch packages 36 kB/s | 10 kB 00:00 2024-03-16T11:35:53.565 INFO:teuthology.orchestra.run.smithi152.stdout:ceph noarch packages 26 kB/s | 10 kB 00:00 2024-03-16T11:35:53.610 INFO:teuthology.orchestra.run.smithi029.stdout:ceph noarch packages 31 kB/s | 10 kB 00:00 2024-03-16T11:35:53.928 INFO:teuthology.orchestra.run.smithi154.stdout:ceph source packages 5.1 kB/s | 1.9 kB 00:00 2024-03-16T11:35:53.970 INFO:teuthology.orchestra.run.smithi029.stdout:ceph source packages 6.0 kB/s | 1.9 kB 00:00 2024-03-16T11:35:53.991 INFO:teuthology.orchestra.run.smithi152.stdout:ceph source packages 4.8 kB/s | 1.9 kB 00:00 2024-03-16T11:35:54.314 INFO:teuthology.orchestra.run.smithi154.stdout:CentOS-9 - AppStream 19 MB/s | 6.7 MB 00:00 2024-03-16T11:35:54.450 INFO:teuthology.orchestra.run.smithi029.stdout:CentOS-9 - AppStream 15 MB/s | 6.7 MB 00:00 2024-03-16T11:35:54.460 INFO:teuthology.orchestra.run.smithi152.stdout:CentOS-9 - AppStream 15 MB/s | 6.7 MB 00:00 2024-03-16T11:35:55.781 INFO:teuthology.orchestra.run.smithi154.stdout:CentOS-9 - BaseOS 5.8 MB/s | 2.0 MB 00:00 2024-03-16T11:35:55.844 INFO:teuthology.orchestra.run.smithi152.stdout:CentOS-9 - BaseOS 7.2 MB/s | 2.0 MB 00:00 2024-03-16T11:35:55.892 INFO:teuthology.orchestra.run.smithi029.stdout:CentOS-9 - BaseOS 7.0 MB/s | 2.0 MB 00:00 2024-03-16T11:35:56.732 INFO:teuthology.orchestra.run.smithi154.stdout:CentOS Stream 9 - BaseOS 13 MB/s | 8.0 MB 00:00 2024-03-16T11:35:56.995 INFO:teuthology.orchestra.run.smithi029.stdout:CentOS Stream 9 - BaseOS 11 MB/s | 8.0 MB 00:00 2024-03-16T11:35:58.947 INFO:teuthology.orchestra.run.smithi029.stdout:CentOS Stream 9 - AppStream 23 MB/s | 19 MB 00:00 2024-03-16T11:35:59.189 INFO:teuthology.orchestra.run.smithi154.stdout:CentOS Stream 9 - AppStream 14 MB/s | 19 MB 00:01 2024-03-16T11:36:03.148 INFO:teuthology.orchestra.run.smithi154.stdout:CentOS Stream 9 - CRB 15 MB/s | 6.1 MB 00:00 2024-03-16T11:36:03.225 INFO:teuthology.orchestra.run.smithi029.stdout:CentOS Stream 9 - CRB 9.4 MB/s | 6.1 MB 00:00 2024-03-16T11:36:04.627 INFO:teuthology.orchestra.run.smithi029.stdout:CentOS Stream 9 - Extras packages 72 kB/s | 15 kB 00:00 2024-03-16T11:36:04.732 INFO:teuthology.orchestra.run.smithi154.stdout:CentOS Stream 9 - Extras packages 36 kB/s | 15 kB 00:00 2024-03-16T11:36:05.451 INFO:teuthology.orchestra.run.smithi152.stdout:CentOS Stream 9 - BaseOS 887 kB/s | 8.0 MB 00:09 2024-03-16T11:36:05.590 INFO:teuthology.orchestra.run.smithi029.stdout:Extra Packages for Enterprise Linux 25 MB/s | 21 MB 00:00 2024-03-16T11:36:07.410 INFO:teuthology.orchestra.run.smithi152.stdout:CentOS Stream 9 - AppStream 22 MB/s | 19 MB 00:00 2024-03-16T11:36:07.900 INFO:teuthology.orchestra.run.smithi154.stdout:Extra Packages for Enterprise Linux 6.8 MB/s | 21 MB 00:03 2024-03-16T11:36:10.708 INFO:teuthology.orchestra.run.smithi029.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2024-03-16T11:36:12.850 INFO:teuthology.orchestra.run.smithi154.stdout:lab-extras 15 kB/s | 1.7 kB 00:00 2024-03-16T11:36:13.356 INFO:teuthology.orchestra.run.smithi029.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-16T11:36:13.357 INFO:teuthology.orchestra.run.smithi029.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-16T11:36:13.371 INFO:teuthology.orchestra.run.smithi029.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-03-16T11:36:13.372 INFO:teuthology.orchestra.run.smithi029.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-03-16T11:36:13.374 INFO:teuthology.orchestra.run.smithi029.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-03-16T11:36:13.375 INFO:teuthology.orchestra.run.smithi029.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-03-16T11:36:13.377 INFO:teuthology.orchestra.run.smithi029.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-03-16T11:36:13.380 INFO:teuthology.orchestra.run.smithi029.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-03-16T11:36:13.381 INFO:teuthology.orchestra.run.smithi029.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-03-16T11:36:13.467 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-03-16T11:36:13.499 INFO:teuthology.orchestra.run.smithi029.stdout:============================================================================================ 2024-03-16T11:36:13.499 INFO:teuthology.orchestra.run.smithi029.stdout: Package Arch Version Repository Size 2024-03-16T11:36:13.499 INFO:teuthology.orchestra.run.smithi029.stdout:============================================================================================ 2024-03-16T11:36:13.499 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2024-03-16T11:36:13.499 INFO:teuthology.orchestra.run.smithi029.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-03-16T11:36:13.499 INFO:teuthology.orchestra.run.smithi029.stdout: ceph x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 6.6 k 2024-03-16T11:36:13.499 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-base x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 5.4 M 2024-03-16T11:36:13.499 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-fuse x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 890 k 2024-03-16T11:36:13.499 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 145 k 2024-03-16T11:36:13.499 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 1.5 M 2024-03-16T11:36:13.499 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 139 k 2024-03-16T11:36:13.499 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 1.8 M 2024-03-16T11:36:13.499 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 7.4 M 2024-03-16T11:36:13.500 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-rook noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 49 k 2024-03-16T11:36:13.500 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-radosgw x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 9.8 M 2024-03-16T11:36:13.500 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-test x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 48 M 2024-03-16T11:36:13.500 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-volume noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 264 k 2024-03-16T11:36:13.500 INFO:teuthology.orchestra.run.smithi029.stdout: cephadm noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 748 k 2024-03-16T11:36:13.500 INFO:teuthology.orchestra.run.smithi029.stdout: cephfs-mirror x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 221 k 2024-03-16T11:36:13.500 INFO:teuthology.orchestra.run.smithi029.stdout: cephfs-shell x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 27 k 2024-03-16T11:36:13.500 INFO:teuthology.orchestra.run.smithi029.stdout: cephfs-top noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 18 k 2024-03-16T11:36:13.500 INFO:teuthology.orchestra.run.smithi029.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-03-16T11:36:13.500 INFO:teuthology.orchestra.run.smithi029.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-03-16T11:36:13.500 INFO:teuthology.orchestra.run.smithi029.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-03-16T11:36:13.500 INFO:teuthology.orchestra.run.smithi029.stdout: libcephfs-devel x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 32 k 2024-03-16T11:36:13.500 INFO:teuthology.orchestra.run.smithi029.stdout: libcephfs2 x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 747 k 2024-03-16T11:36:13.500 INFO:teuthology.orchestra.run.smithi029.stdout: librados-devel x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 127 k 2024-03-16T11:36:13.500 INFO:teuthology.orchestra.run.smithi029.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-03-16T11:36:13.501 INFO:teuthology.orchestra.run.smithi029.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-03-16T11:36:13.501 INFO:teuthology.orchestra.run.smithi029.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-03-16T11:36:13.501 INFO:teuthology.orchestra.run.smithi029.stdout: postgresql x86_64 13.11-1.el9 CentOS-AppStream 1.5 M 2024-03-16T11:36:13.501 INFO:teuthology.orchestra.run.smithi029.stdout: postgresql-contrib x86_64 13.11-1.el9 CentOS-AppStream 811 k 2024-03-16T11:36:13.501 INFO:teuthology.orchestra.run.smithi029.stdout: postgresql-server x86_64 13.11-1.el9 CentOS-AppStream 5.7 M 2024-03-16T11:36:13.501 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cephfs x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 162 k 2024-03-16T11:36:13.501 INFO:teuthology.orchestra.run.smithi029.stdout: python3-rados x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 320 k 2024-03-16T11:36:13.501 INFO:teuthology.orchestra.run.smithi029.stdout: python3-rbd x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 298 k 2024-03-16T11:36:13.501 INFO:teuthology.orchestra.run.smithi029.stdout: python3-rgw x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 99 k 2024-03-16T11:36:13.501 INFO:teuthology.orchestra.run.smithi029.stdout: rbd-fuse x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 85 k 2024-03-16T11:36:13.501 INFO:teuthology.orchestra.run.smithi029.stdout: rbd-mirror x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 3.1 M 2024-03-16T11:36:13.501 INFO:teuthology.orchestra.run.smithi029.stdout: rbd-nbd x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 170 k 2024-03-16T11:36:13.501 INFO:teuthology.orchestra.run.smithi029.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-03-16T11:36:13.501 INFO:teuthology.orchestra.run.smithi029.stdout:Upgrading: 2024-03-16T11:36:13.501 INFO:teuthology.orchestra.run.smithi029.stdout: NetworkManager x86_64 1:1.46.0-1.el9 baseos 2.3 M 2024-03-16T11:36:13.501 INFO:teuthology.orchestra.run.smithi029.stdout: NetworkManager-libnm x86_64 1:1.46.0-1.el9 baseos 1.8 M 2024-03-16T11:36:13.502 INFO:teuthology.orchestra.run.smithi029.stdout: NetworkManager-team x86_64 1:1.46.0-1.el9 baseos 42 k 2024-03-16T11:36:13.502 INFO:teuthology.orchestra.run.smithi029.stdout: NetworkManager-tui x86_64 1:1.46.0-1.el9 baseos 248 k 2024-03-16T11:36:13.502 INFO:teuthology.orchestra.run.smithi029.stdout: acl x86_64 2.3.1-4.el9 baseos 71 k 2024-03-16T11:36:13.502 INFO:teuthology.orchestra.run.smithi029.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 baseos 36 k 2024-03-16T11:36:13.502 INFO:teuthology.orchestra.run.smithi029.stdout: elfutils-libelf x86_64 0.190-2.el9 baseos 192 k 2024-03-16T11:36:13.502 INFO:teuthology.orchestra.run.smithi029.stdout: elfutils-libs x86_64 0.190-2.el9 baseos 256 k 2024-03-16T11:36:13.502 INFO:teuthology.orchestra.run.smithi029.stdout: libacl x86_64 2.3.1-4.el9 baseos 23 k 2024-03-16T11:36:13.502 INFO:teuthology.orchestra.run.smithi029.stdout: libacl-devel x86_64 2.3.1-4.el9 appstream 74 k 2024-03-16T11:36:13.502 INFO:teuthology.orchestra.run.smithi029.stdout: libblkid x86_64 2.37.4-18.el9 baseos 108 k 2024-03-16T11:36:13.502 INFO:teuthology.orchestra.run.smithi029.stdout: libfdisk x86_64 2.37.4-18.el9 baseos 155 k 2024-03-16T11:36:13.502 INFO:teuthology.orchestra.run.smithi029.stdout: libmount x86_64 2.37.4-18.el9 baseos 136 k 2024-03-16T11:36:13.502 INFO:teuthology.orchestra.run.smithi029.stdout: librados2 x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 3.4 M 2024-03-16T11:36:13.503 INFO:teuthology.orchestra.run.smithi029.stdout: librbd1 x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 3.1 M 2024-03-16T11:36:13.503 INFO:teuthology.orchestra.run.smithi029.stdout: libsmartcols x86_64 2.37.4-18.el9 baseos 64 k 2024-03-16T11:36:13.503 INFO:teuthology.orchestra.run.smithi029.stdout: libuuid x86_64 2.37.4-18.el9 baseos 29 k 2024-03-16T11:36:13.503 INFO:teuthology.orchestra.run.smithi029.stdout: openssl x86_64 1:3.0.7-27.el9 baseos 1.2 M 2024-03-16T11:36:13.503 INFO:teuthology.orchestra.run.smithi029.stdout: openssl-libs x86_64 1:3.0.7-27.el9 baseos 2.2 M 2024-03-16T11:36:13.503 INFO:teuthology.orchestra.run.smithi029.stdout: util-linux x86_64 2.37.4-18.el9 baseos 2.3 M 2024-03-16T11:36:13.503 INFO:teuthology.orchestra.run.smithi029.stdout: util-linux-core x86_64 2.37.4-18.el9 baseos 465 k 2024-03-16T11:36:13.503 INFO:teuthology.orchestra.run.smithi029.stdout: util-linux-user x86_64 2.37.4-18.el9 baseos 31 k 2024-03-16T11:36:13.503 INFO:teuthology.orchestra.run.smithi029.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-03-16T11:36:13.503 INFO:teuthology.orchestra.run.smithi029.stdout: xfsprogs x86_64 6.3.0-1.el9 baseos 1.1 M 2024-03-16T11:36:13.503 INFO:teuthology.orchestra.run.smithi029.stdout:Installing dependencies: 2024-03-16T11:36:13.503 INFO:teuthology.orchestra.run.smithi029.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-16T11:36:13.503 INFO:teuthology.orchestra.run.smithi029.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-03-16T11:36:13.503 INFO:teuthology.orchestra.run.smithi029.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-16T11:36:13.503 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-common x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 21 M 2024-03-16T11:36:13.504 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 34 k 2024-03-16T11:36:13.504 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mds x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 2.4 M 2024-03-16T11:36:13.504 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 248 k 2024-03-16T11:36:13.504 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mon x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 4.7 M 2024-03-16T11:36:13.504 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-osd x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 17 M 2024-03-16T11:36:13.504 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 16 k 2024-03-16T11:36:13.504 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-selinux x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 25 k 2024-03-16T11:36:13.504 INFO:teuthology.orchestra.run.smithi029.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-16T11:36:13.504 INFO:teuthology.orchestra.run.smithi029.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-16T11:36:13.504 INFO:teuthology.orchestra.run.smithi029.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-16T11:36:13.504 INFO:teuthology.orchestra.run.smithi029.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-16T11:36:13.504 INFO:teuthology.orchestra.run.smithi029.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-16T11:36:13.504 INFO:teuthology.orchestra.run.smithi029.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-16T11:36:13.504 INFO:teuthology.orchestra.run.smithi029.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-16T11:36:13.505 INFO:teuthology.orchestra.run.smithi029.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-16T11:36:13.505 INFO:teuthology.orchestra.run.smithi029.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-16T11:36:13.505 INFO:teuthology.orchestra.run.smithi029.stdout: libcephsqlite x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 163 k 2024-03-16T11:36:13.505 INFO:teuthology.orchestra.run.smithi029.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-16T11:36:13.505 INFO:teuthology.orchestra.run.smithi029.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-16T11:36:13.505 INFO:teuthology.orchestra.run.smithi029.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-16T11:36:13.505 INFO:teuthology.orchestra.run.smithi029.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-16T11:36:13.505 INFO:teuthology.orchestra.run.smithi029.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-16T11:36:13.505 INFO:teuthology.orchestra.run.smithi029.stdout: libradosstriper1 x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 501 k 2024-03-16T11:36:13.505 INFO:teuthology.orchestra.run.smithi029.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-16T11:36:13.505 INFO:teuthology.orchestra.run.smithi029.stdout: librgw2 x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 5.1 M 2024-03-16T11:36:13.505 INFO:teuthology.orchestra.run.smithi029.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-16T11:36:13.505 INFO:teuthology.orchestra.run.smithi029.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-03-16T11:36:13.505 INFO:teuthology.orchestra.run.smithi029.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-16T11:36:13.506 INFO:teuthology.orchestra.run.smithi029.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-16T11:36:13.506 INFO:teuthology.orchestra.run.smithi029.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-16T11:36:13.506 INFO:teuthology.orchestra.run.smithi029.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-16T11:36:13.506 INFO:teuthology.orchestra.run.smithi029.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-16T11:36:13.506 INFO:teuthology.orchestra.run.smithi029.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-16T11:36:13.506 INFO:teuthology.orchestra.run.smithi029.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-16T11:36:13.506 INFO:teuthology.orchestra.run.smithi029.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-16T11:36:13.506 INFO:teuthology.orchestra.run.smithi029.stdout: postgresql-private-libs x86_64 13.11-1.el9 CentOS-AppStream 134 k 2024-03-16T11:36:13.506 INFO:teuthology.orchestra.run.smithi029.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-16T11:36:13.506 INFO:teuthology.orchestra.run.smithi029.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-16T11:36:13.506 INFO:teuthology.orchestra.run.smithi029.stdout: python3-asyncssh noarch 2.13.2-4.el9 epel 548 k 2024-03-16T11:36:13.506 INFO:teuthology.orchestra.run.smithi029.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-03-16T11:36:13.506 INFO:teuthology.orchestra.run.smithi029.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-16T11:36:13.507 INFO:teuthology.orchestra.run.smithi029.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-16T11:36:13.507 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-16T11:36:13.507 INFO:teuthology.orchestra.run.smithi029.stdout: python3-ceph-argparse x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 45 k 2024-03-16T11:36:13.507 INFO:teuthology.orchestra.run.smithi029.stdout: python3-ceph-common x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 131 k 2024-03-16T11:36:13.507 INFO:teuthology.orchestra.run.smithi029.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-16T11:36:13.507 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-16T11:36:13.507 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-16T11:36:13.507 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-16T11:36:13.507 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-03-16T11:36:13.507 INFO:teuthology.orchestra.run.smithi029.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-03-16T11:36:13.507 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-16T11:36:13.507 INFO:teuthology.orchestra.run.smithi029.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-16T11:36:13.507 INFO:teuthology.orchestra.run.smithi029.stdout: python3-google-auth noarch 1:2.28.0-1.el9 epel 203 k 2024-03-16T11:36:13.507 INFO:teuthology.orchestra.run.smithi029.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-16T11:36:13.507 INFO:teuthology.orchestra.run.smithi029.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-16T11:36:13.508 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-16T11:36:13.508 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-16T11:36:13.508 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-16T11:36:13.508 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-16T11:36:13.508 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-16T11:36:13.508 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-16T11:36:13.508 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-16T11:36:13.508 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-16T11:36:13.508 INFO:teuthology.orchestra.run.smithi029.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-16T11:36:13.508 INFO:teuthology.orchestra.run.smithi029.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-16T11:36:13.508 INFO:teuthology.orchestra.run.smithi029.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-16T11:36:13.508 INFO:teuthology.orchestra.run.smithi029.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-16T11:36:13.508 INFO:teuthology.orchestra.run.smithi029.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-16T11:36:13.508 INFO:teuthology.orchestra.run.smithi029.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-16T11:36:13.508 INFO:teuthology.orchestra.run.smithi029.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-16T11:36:13.508 INFO:teuthology.orchestra.run.smithi029.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-16T11:36:13.509 INFO:teuthology.orchestra.run.smithi029.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-16T11:36:13.509 INFO:teuthology.orchestra.run.smithi029.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-16T11:36:13.509 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-16T11:36:13.509 INFO:teuthology.orchestra.run.smithi029.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-16T11:36:13.509 INFO:teuthology.orchestra.run.smithi029.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-16T11:36:13.509 INFO:teuthology.orchestra.run.smithi029.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-16T11:36:13.509 INFO:teuthology.orchestra.run.smithi029.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-16T11:36:13.509 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-16T11:36:13.509 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-16T11:36:13.509 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-16T11:36:13.509 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-16T11:36:13.509 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-03-16T11:36:13.509 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-16T11:36:13.509 INFO:teuthology.orchestra.run.smithi029.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-16T11:36:13.509 INFO:teuthology.orchestra.run.smithi029.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-16T11:36:13.509 INFO:teuthology.orchestra.run.smithi029.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-16T11:36:13.510 INFO:teuthology.orchestra.run.smithi029.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-16T11:36:13.510 INFO:teuthology.orchestra.run.smithi029.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-16T11:36:13.510 INFO:teuthology.orchestra.run.smithi029.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-16T11:36:13.510 INFO:teuthology.orchestra.run.smithi029.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-16T11:36:13.510 INFO:teuthology.orchestra.run.smithi029.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-16T11:36:13.510 INFO:teuthology.orchestra.run.smithi029.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-16T11:36:13.510 INFO:teuthology.orchestra.run.smithi029.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-16T11:36:13.510 INFO:teuthology.orchestra.run.smithi029.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-03-16T11:36:13.510 INFO:teuthology.orchestra.run.smithi029.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-16T11:36:13.510 INFO:teuthology.orchestra.run.smithi029.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-16T11:36:13.510 INFO:teuthology.orchestra.run.smithi029.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-16T11:36:13.510 INFO:teuthology.orchestra.run.smithi029.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-16T11:36:13.510 INFO:teuthology.orchestra.run.smithi029.stdout: qatlib x86_64 23.11.0-1.el9 appstream 216 k 2024-03-16T11:36:13.510 INFO:teuthology.orchestra.run.smithi029.stdout: qatzip-libs x86_64 1.1.2-2.el9 appstream 44 k 2024-03-16T11:36:13.510 INFO:teuthology.orchestra.run.smithi029.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-16T11:36:13.510 INFO:teuthology.orchestra.run.smithi029.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-16T11:36:13.511 INFO:teuthology.orchestra.run.smithi029.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-16T11:36:13.511 INFO:teuthology.orchestra.run.smithi029.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-03-16T11:36:13.511 INFO:teuthology.orchestra.run.smithi029.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-16T11:36:13.511 INFO:teuthology.orchestra.run.smithi029.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-03-16T11:36:13.511 INFO:teuthology.orchestra.run.smithi029.stdout:Installing weak dependencies: 2024-03-16T11:36:13.511 INFO:teuthology.orchestra.run.smithi029.stdout: qatlib-service x86_64 23.11.0-1.el9 appstream 35 k 2024-03-16T11:36:13.511 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-16T11:36:13.511 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2024-03-16T11:36:13.511 INFO:teuthology.orchestra.run.smithi029.stdout:============================================================================================ 2024-03-16T11:36:13.511 INFO:teuthology.orchestra.run.smithi029.stdout:Install 150 Packages 2024-03-16T11:36:13.511 INFO:teuthology.orchestra.run.smithi029.stdout:Upgrade 24 Packages 2024-03-16T11:36:13.511 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-16T11:36:13.512 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 224 M 2024-03-16T11:36:13.512 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2024-03-16T11:36:14.594 INFO:teuthology.orchestra.run.smithi029.stdout:(1/174): ceph-19.0.0-2162.g3d78f5ba.el9.x86_64. 37 kB/s | 6.6 kB 00:00 2024-03-16T11:36:14.994 INFO:teuthology.orchestra.run.smithi029.stdout:(2/174): ceph-fuse-19.0.0-2162.g3d78f5ba.el9.x8 2.2 MB/s | 890 kB 00:00 2024-03-16T11:36:15.161 INFO:teuthology.orchestra.run.smithi029.stdout:(3/174): ceph-base-19.0.0-2162.g3d78f5ba.el9.x8 7.2 MB/s | 5.4 MB 00:00 2024-03-16T11:36:15.187 INFO:teuthology.orchestra.run.smithi029.stdout:(4/174): ceph-immutable-object-cache-19.0.0-216 752 kB/s | 145 kB 00:00 2024-03-16T11:36:15.305 INFO:teuthology.orchestra.run.smithi154.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-16T11:36:15.306 INFO:teuthology.orchestra.run.smithi154.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-16T11:36:15.320 INFO:teuthology.orchestra.run.smithi154.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-03-16T11:36:15.321 INFO:teuthology.orchestra.run.smithi154.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-03-16T11:36:15.322 INFO:teuthology.orchestra.run.smithi154.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-03-16T11:36:15.323 INFO:teuthology.orchestra.run.smithi154.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-03-16T11:36:15.325 INFO:teuthology.orchestra.run.smithi154.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-03-16T11:36:15.328 INFO:teuthology.orchestra.run.smithi154.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-03-16T11:36:15.329 INFO:teuthology.orchestra.run.smithi154.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-03-16T11:36:15.409 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2024-03-16T11:36:15.421 INFO:teuthology.orchestra.run.smithi154.stdout:============================================================================================ 2024-03-16T11:36:15.421 INFO:teuthology.orchestra.run.smithi154.stdout: Package Arch Version Repository Size 2024-03-16T11:36:15.421 INFO:teuthology.orchestra.run.smithi154.stdout:============================================================================================ 2024-03-16T11:36:15.421 INFO:teuthology.orchestra.run.smithi154.stdout:Installing: 2024-03-16T11:36:15.421 INFO:teuthology.orchestra.run.smithi154.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-03-16T11:36:15.421 INFO:teuthology.orchestra.run.smithi154.stdout: ceph x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 6.6 k 2024-03-16T11:36:15.421 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-base x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 5.4 M 2024-03-16T11:36:15.421 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-fuse x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 890 k 2024-03-16T11:36:15.421 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 145 k 2024-03-16T11:36:15.421 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mgr x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 1.5 M 2024-03-16T11:36:15.422 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 139 k 2024-03-16T11:36:15.422 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 1.8 M 2024-03-16T11:36:15.422 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 7.4 M 2024-03-16T11:36:15.422 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mgr-rook noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 49 k 2024-03-16T11:36:15.422 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-radosgw x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 9.8 M 2024-03-16T11:36:15.422 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-test x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 48 M 2024-03-16T11:36:15.422 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-volume noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 264 k 2024-03-16T11:36:15.422 INFO:teuthology.orchestra.run.smithi154.stdout: cephadm noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 748 k 2024-03-16T11:36:15.422 INFO:teuthology.orchestra.run.smithi154.stdout: cephfs-mirror x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 221 k 2024-03-16T11:36:15.422 INFO:teuthology.orchestra.run.smithi154.stdout: cephfs-shell x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 27 k 2024-03-16T11:36:15.422 INFO:teuthology.orchestra.run.smithi154.stdout: cephfs-top noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 18 k 2024-03-16T11:36:15.422 INFO:teuthology.orchestra.run.smithi154.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-03-16T11:36:15.422 INFO:teuthology.orchestra.run.smithi154.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-03-16T11:36:15.422 INFO:teuthology.orchestra.run.smithi154.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-03-16T11:36:15.423 INFO:teuthology.orchestra.run.smithi154.stdout: libcephfs-devel x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 32 k 2024-03-16T11:36:15.423 INFO:teuthology.orchestra.run.smithi154.stdout: libcephfs2 x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 747 k 2024-03-16T11:36:15.423 INFO:teuthology.orchestra.run.smithi154.stdout: librados-devel x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 127 k 2024-03-16T11:36:15.423 INFO:teuthology.orchestra.run.smithi154.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-03-16T11:36:15.423 INFO:teuthology.orchestra.run.smithi154.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-03-16T11:36:15.423 INFO:teuthology.orchestra.run.smithi154.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-03-16T11:36:15.423 INFO:teuthology.orchestra.run.smithi154.stdout: postgresql x86_64 13.11-1.el9 CentOS-AppStream 1.5 M 2024-03-16T11:36:15.423 INFO:teuthology.orchestra.run.smithi154.stdout: postgresql-contrib x86_64 13.11-1.el9 CentOS-AppStream 811 k 2024-03-16T11:36:15.423 INFO:teuthology.orchestra.run.smithi154.stdout: postgresql-server x86_64 13.11-1.el9 CentOS-AppStream 5.7 M 2024-03-16T11:36:15.423 INFO:teuthology.orchestra.run.smithi154.stdout: python3-cephfs x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 162 k 2024-03-16T11:36:15.423 INFO:teuthology.orchestra.run.smithi154.stdout: python3-rados x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 320 k 2024-03-16T11:36:15.423 INFO:teuthology.orchestra.run.smithi154.stdout: python3-rbd x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 298 k 2024-03-16T11:36:15.423 INFO:teuthology.orchestra.run.smithi154.stdout: python3-rgw x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 99 k 2024-03-16T11:36:15.423 INFO:teuthology.orchestra.run.smithi154.stdout: rbd-fuse x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 85 k 2024-03-16T11:36:15.424 INFO:teuthology.orchestra.run.smithi154.stdout: rbd-mirror x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 3.1 M 2024-03-16T11:36:15.424 INFO:teuthology.orchestra.run.smithi154.stdout: rbd-nbd x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 170 k 2024-03-16T11:36:15.424 INFO:teuthology.orchestra.run.smithi154.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-03-16T11:36:15.424 INFO:teuthology.orchestra.run.smithi154.stdout:Upgrading: 2024-03-16T11:36:15.424 INFO:teuthology.orchestra.run.smithi154.stdout: NetworkManager x86_64 1:1.46.0-1.el9 baseos 2.3 M 2024-03-16T11:36:15.424 INFO:teuthology.orchestra.run.smithi154.stdout: NetworkManager-libnm x86_64 1:1.46.0-1.el9 baseos 1.8 M 2024-03-16T11:36:15.424 INFO:teuthology.orchestra.run.smithi154.stdout: NetworkManager-team x86_64 1:1.46.0-1.el9 baseos 42 k 2024-03-16T11:36:15.424 INFO:teuthology.orchestra.run.smithi154.stdout: NetworkManager-tui x86_64 1:1.46.0-1.el9 baseos 248 k 2024-03-16T11:36:15.424 INFO:teuthology.orchestra.run.smithi154.stdout: acl x86_64 2.3.1-4.el9 baseos 71 k 2024-03-16T11:36:15.424 INFO:teuthology.orchestra.run.smithi154.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 baseos 36 k 2024-03-16T11:36:15.424 INFO:teuthology.orchestra.run.smithi154.stdout: elfutils-libelf x86_64 0.190-2.el9 baseos 192 k 2024-03-16T11:36:15.424 INFO:teuthology.orchestra.run.smithi154.stdout: elfutils-libs x86_64 0.190-2.el9 baseos 256 k 2024-03-16T11:36:15.424 INFO:teuthology.orchestra.run.smithi154.stdout: libacl x86_64 2.3.1-4.el9 baseos 23 k 2024-03-16T11:36:15.424 INFO:teuthology.orchestra.run.smithi154.stdout: libacl-devel x86_64 2.3.1-4.el9 appstream 74 k 2024-03-16T11:36:15.424 INFO:teuthology.orchestra.run.smithi154.stdout: libblkid x86_64 2.37.4-18.el9 baseos 108 k 2024-03-16T11:36:15.424 INFO:teuthology.orchestra.run.smithi154.stdout: libfdisk x86_64 2.37.4-18.el9 baseos 155 k 2024-03-16T11:36:15.425 INFO:teuthology.orchestra.run.smithi154.stdout: libmount x86_64 2.37.4-18.el9 baseos 136 k 2024-03-16T11:36:15.425 INFO:teuthology.orchestra.run.smithi154.stdout: librados2 x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 3.4 M 2024-03-16T11:36:15.425 INFO:teuthology.orchestra.run.smithi154.stdout: librbd1 x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 3.1 M 2024-03-16T11:36:15.425 INFO:teuthology.orchestra.run.smithi154.stdout: libsmartcols x86_64 2.37.4-18.el9 baseos 64 k 2024-03-16T11:36:15.425 INFO:teuthology.orchestra.run.smithi154.stdout: libuuid x86_64 2.37.4-18.el9 baseos 29 k 2024-03-16T11:36:15.425 INFO:teuthology.orchestra.run.smithi154.stdout: openssl x86_64 1:3.0.7-27.el9 baseos 1.2 M 2024-03-16T11:36:15.425 INFO:teuthology.orchestra.run.smithi154.stdout: openssl-libs x86_64 1:3.0.7-27.el9 baseos 2.2 M 2024-03-16T11:36:15.425 INFO:teuthology.orchestra.run.smithi154.stdout: util-linux x86_64 2.37.4-18.el9 baseos 2.3 M 2024-03-16T11:36:15.425 INFO:teuthology.orchestra.run.smithi154.stdout: util-linux-core x86_64 2.37.4-18.el9 baseos 465 k 2024-03-16T11:36:15.425 INFO:teuthology.orchestra.run.smithi154.stdout: util-linux-user x86_64 2.37.4-18.el9 baseos 31 k 2024-03-16T11:36:15.425 INFO:teuthology.orchestra.run.smithi154.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-03-16T11:36:15.425 INFO:teuthology.orchestra.run.smithi154.stdout: xfsprogs x86_64 6.3.0-1.el9 baseos 1.1 M 2024-03-16T11:36:15.425 INFO:teuthology.orchestra.run.smithi154.stdout:Installing dependencies: 2024-03-16T11:36:15.425 INFO:teuthology.orchestra.run.smithi154.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-16T11:36:15.425 INFO:teuthology.orchestra.run.smithi154.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-03-16T11:36:15.426 INFO:teuthology.orchestra.run.smithi154.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-16T11:36:15.426 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-common x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 21 M 2024-03-16T11:36:15.426 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 34 k 2024-03-16T11:36:15.426 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mds x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 2.4 M 2024-03-16T11:36:15.426 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 248 k 2024-03-16T11:36:15.426 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mon x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 4.7 M 2024-03-16T11:36:15.426 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-osd x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 17 M 2024-03-16T11:36:15.426 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 16 k 2024-03-16T11:36:15.426 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-selinux x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 25 k 2024-03-16T11:36:15.426 INFO:teuthology.orchestra.run.smithi154.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-16T11:36:15.426 INFO:teuthology.orchestra.run.smithi154.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-16T11:36:15.426 INFO:teuthology.orchestra.run.smithi154.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-16T11:36:15.426 INFO:teuthology.orchestra.run.smithi154.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-16T11:36:15.426 INFO:teuthology.orchestra.run.smithi154.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-16T11:36:15.427 INFO:teuthology.orchestra.run.smithi154.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-16T11:36:15.427 INFO:teuthology.orchestra.run.smithi154.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-16T11:36:15.427 INFO:teuthology.orchestra.run.smithi154.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-16T11:36:15.427 INFO:teuthology.orchestra.run.smithi154.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-16T11:36:15.427 INFO:teuthology.orchestra.run.smithi154.stdout: libcephsqlite x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 163 k 2024-03-16T11:36:15.427 INFO:teuthology.orchestra.run.smithi154.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-16T11:36:15.427 INFO:teuthology.orchestra.run.smithi154.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-16T11:36:15.427 INFO:teuthology.orchestra.run.smithi154.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-16T11:36:15.427 INFO:teuthology.orchestra.run.smithi154.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-16T11:36:15.427 INFO:teuthology.orchestra.run.smithi154.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-16T11:36:15.427 INFO:teuthology.orchestra.run.smithi154.stdout: libradosstriper1 x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 501 k 2024-03-16T11:36:15.427 INFO:teuthology.orchestra.run.smithi154.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-16T11:36:15.427 INFO:teuthology.orchestra.run.smithi154.stdout: librgw2 x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 5.1 M 2024-03-16T11:36:15.427 INFO:teuthology.orchestra.run.smithi154.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-16T11:36:15.427 INFO:teuthology.orchestra.run.smithi154.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-03-16T11:36:15.428 INFO:teuthology.orchestra.run.smithi154.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-16T11:36:15.428 INFO:teuthology.orchestra.run.smithi154.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-16T11:36:15.428 INFO:teuthology.orchestra.run.smithi154.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-16T11:36:15.428 INFO:teuthology.orchestra.run.smithi154.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-16T11:36:15.428 INFO:teuthology.orchestra.run.smithi154.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-16T11:36:15.428 INFO:teuthology.orchestra.run.smithi154.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-16T11:36:15.428 INFO:teuthology.orchestra.run.smithi154.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-16T11:36:15.428 INFO:teuthology.orchestra.run.smithi154.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-16T11:36:15.428 INFO:teuthology.orchestra.run.smithi154.stdout: postgresql-private-libs x86_64 13.11-1.el9 CentOS-AppStream 134 k 2024-03-16T11:36:15.428 INFO:teuthology.orchestra.run.smithi154.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-16T11:36:15.428 INFO:teuthology.orchestra.run.smithi154.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-16T11:36:15.428 INFO:teuthology.orchestra.run.smithi154.stdout: python3-asyncssh noarch 2.13.2-4.el9 epel 548 k 2024-03-16T11:36:15.428 INFO:teuthology.orchestra.run.smithi154.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-03-16T11:36:15.428 INFO:teuthology.orchestra.run.smithi154.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-16T11:36:15.428 INFO:teuthology.orchestra.run.smithi154.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-16T11:36:15.429 INFO:teuthology.orchestra.run.smithi154.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-16T11:36:15.429 INFO:teuthology.orchestra.run.smithi154.stdout: python3-ceph-argparse x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 45 k 2024-03-16T11:36:15.429 INFO:teuthology.orchestra.run.smithi154.stdout: python3-ceph-common x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 131 k 2024-03-16T11:36:15.429 INFO:teuthology.orchestra.run.smithi154.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-16T11:36:15.429 INFO:teuthology.orchestra.run.smithi154.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-16T11:36:15.429 INFO:teuthology.orchestra.run.smithi154.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-16T11:36:15.429 INFO:teuthology.orchestra.run.smithi154.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-16T11:36:15.429 INFO:teuthology.orchestra.run.smithi154.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-03-16T11:36:15.429 INFO:teuthology.orchestra.run.smithi154.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-03-16T11:36:15.429 INFO:teuthology.orchestra.run.smithi154.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-16T11:36:15.429 INFO:teuthology.orchestra.run.smithi154.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-16T11:36:15.429 INFO:teuthology.orchestra.run.smithi154.stdout: python3-google-auth noarch 1:2.28.0-1.el9 epel 203 k 2024-03-16T11:36:15.429 INFO:teuthology.orchestra.run.smithi154.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-16T11:36:15.429 INFO:teuthology.orchestra.run.smithi154.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-16T11:36:15.429 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-16T11:36:15.429 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-16T11:36:15.429 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-16T11:36:15.430 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-16T11:36:15.430 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-16T11:36:15.430 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-16T11:36:15.430 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-16T11:36:15.430 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-16T11:36:15.430 INFO:teuthology.orchestra.run.smithi154.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-16T11:36:15.430 INFO:teuthology.orchestra.run.smithi154.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-16T11:36:15.430 INFO:teuthology.orchestra.run.smithi154.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-16T11:36:15.430 INFO:teuthology.orchestra.run.smithi154.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-16T11:36:15.430 INFO:teuthology.orchestra.run.smithi154.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-16T11:36:15.430 INFO:teuthology.orchestra.run.smithi154.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-16T11:36:15.430 INFO:teuthology.orchestra.run.smithi154.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-16T11:36:15.430 INFO:teuthology.orchestra.run.smithi154.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-16T11:36:15.430 INFO:teuthology.orchestra.run.smithi154.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-16T11:36:15.430 INFO:teuthology.orchestra.run.smithi154.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-16T11:36:15.430 INFO:teuthology.orchestra.run.smithi154.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-16T11:36:15.431 INFO:teuthology.orchestra.run.smithi154.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-16T11:36:15.431 INFO:teuthology.orchestra.run.smithi154.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-16T11:36:15.431 INFO:teuthology.orchestra.run.smithi154.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-16T11:36:15.431 INFO:teuthology.orchestra.run.smithi154.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-16T11:36:15.431 INFO:teuthology.orchestra.run.smithi154.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-16T11:36:15.431 INFO:teuthology.orchestra.run.smithi154.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-16T11:36:15.431 INFO:teuthology.orchestra.run.smithi154.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-16T11:36:15.431 INFO:teuthology.orchestra.run.smithi154.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-16T11:36:15.431 INFO:teuthology.orchestra.run.smithi154.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-03-16T11:36:15.431 INFO:teuthology.orchestra.run.smithi154.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-16T11:36:15.431 INFO:teuthology.orchestra.run.smithi154.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-16T11:36:15.431 INFO:teuthology.orchestra.run.smithi154.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-16T11:36:15.431 INFO:teuthology.orchestra.run.smithi154.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-16T11:36:15.431 INFO:teuthology.orchestra.run.smithi154.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-16T11:36:15.431 INFO:teuthology.orchestra.run.smithi154.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-16T11:36:15.431 INFO:teuthology.orchestra.run.smithi154.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-16T11:36:15.432 INFO:teuthology.orchestra.run.smithi154.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-16T11:36:15.432 INFO:teuthology.orchestra.run.smithi154.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-16T11:36:15.432 INFO:teuthology.orchestra.run.smithi154.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-16T11:36:15.432 INFO:teuthology.orchestra.run.smithi154.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-16T11:36:15.432 INFO:teuthology.orchestra.run.smithi154.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-03-16T11:36:15.432 INFO:teuthology.orchestra.run.smithi154.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-16T11:36:15.432 INFO:teuthology.orchestra.run.smithi154.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-16T11:36:15.432 INFO:teuthology.orchestra.run.smithi154.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-16T11:36:15.432 INFO:teuthology.orchestra.run.smithi154.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-16T11:36:15.432 INFO:teuthology.orchestra.run.smithi154.stdout: qatlib x86_64 23.11.0-1.el9 appstream 216 k 2024-03-16T11:36:15.432 INFO:teuthology.orchestra.run.smithi154.stdout: qatzip-libs x86_64 1.1.2-2.el9 appstream 44 k 2024-03-16T11:36:15.432 INFO:teuthology.orchestra.run.smithi154.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-16T11:36:15.432 INFO:teuthology.orchestra.run.smithi154.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-16T11:36:15.432 INFO:teuthology.orchestra.run.smithi154.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-16T11:36:15.432 INFO:teuthology.orchestra.run.smithi154.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-03-16T11:36:15.432 INFO:teuthology.orchestra.run.smithi154.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-16T11:36:15.432 INFO:teuthology.orchestra.run.smithi154.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-03-16T11:36:15.433 INFO:teuthology.orchestra.run.smithi154.stdout:Installing weak dependencies: 2024-03-16T11:36:15.433 INFO:teuthology.orchestra.run.smithi154.stdout: qatlib-service x86_64 23.11.0-1.el9 appstream 35 k 2024-03-16T11:36:15.433 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-03-16T11:36:15.433 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction Summary 2024-03-16T11:36:15.433 INFO:teuthology.orchestra.run.smithi154.stdout:============================================================================================ 2024-03-16T11:36:15.433 INFO:teuthology.orchestra.run.smithi154.stdout:Install 150 Packages 2024-03-16T11:36:15.433 INFO:teuthology.orchestra.run.smithi154.stdout:Upgrade 24 Packages 2024-03-16T11:36:15.433 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-03-16T11:36:15.433 INFO:teuthology.orchestra.run.smithi154.stdout:Total download size: 224 M 2024-03-16T11:36:15.433 INFO:teuthology.orchestra.run.smithi154.stdout:Downloading Packages: 2024-03-16T11:36:15.480 INFO:teuthology.orchestra.run.smithi029.stdout:(5/174): ceph-mds-19.0.0-2162.g3d78f5ba.el9.x86 7.4 MB/s | 2.4 MB 00:00 2024-03-16T11:36:15.714 INFO:teuthology.orchestra.run.smithi029.stdout:(6/174): ceph-mgr-19.0.0-2162.g3d78f5ba.el9.x86 2.8 MB/s | 1.5 MB 00:00 2024-03-16T11:36:16.015 INFO:teuthology.orchestra.run.smithi029.stdout:(7/174): ceph-mon-19.0.0-2162.g3d78f5ba.el9.x86 8.8 MB/s | 4.7 MB 00:00 2024-03-16T11:36:16.035 INFO:teuthology.orchestra.run.smithi154.stdout:(1/174): ceph-19.0.0-2162.g3d78f5ba.el9.x86_64. 31 kB/s | 6.6 kB 00:00 2024-03-16T11:36:16.349 INFO:teuthology.orchestra.run.smithi029.stdout:(8/174): ceph-common-19.0.0-2162.g3d78f5ba.el9. 11 MB/s | 21 MB 00:01 2024-03-16T11:36:16.376 INFO:teuthology.orchestra.run.smithi154.stdout:(2/174): ceph-fuse-19.0.0-2162.g3d78f5ba.el9.x8 2.5 MB/s | 890 kB 00:00 2024-03-16T11:36:16.424 INFO:teuthology.orchestra.run.smithi029.stdout:(9/174): ceph-selinux-19.0.0-2162.g3d78f5ba.el9 335 kB/s | 25 kB 00:00 2024-03-16T11:36:16.452 INFO:teuthology.orchestra.run.smithi154.stdout:(3/174): ceph-immutable-object-cache-19.0.0-216 1.9 MB/s | 145 kB 00:00 2024-03-16T11:36:17.327 INFO:teuthology.orchestra.run.smithi154.stdout:(4/174): ceph-mds-19.0.0-2162.g3d78f5ba.el9.x86 2.7 MB/s | 2.4 MB 00:00 2024-03-16T11:36:17.333 INFO:teuthology.orchestra.run.smithi029.stdout:(10/174): ceph-radosgw-19.0.0-2162.g3d78f5ba.el 7.5 MB/s | 9.8 MB 00:01 2024-03-16T11:36:17.442 INFO:teuthology.orchestra.run.smithi029.stdout:(11/174): cephfs-mirror-19.0.0-2162.g3d78f5ba.e 2.0 MB/s | 221 kB 00:00 2024-03-16T11:36:17.526 INFO:teuthology.orchestra.run.smithi029.stdout:(12/174): cephfs-shell-19.0.0-2162.g3d78f5ba.el 325 kB/s | 27 kB 00:00 2024-03-16T11:36:17.609 INFO:teuthology.orchestra.run.smithi029.stdout:(13/174): libcephfs-devel-19.0.0-2162.g3d78f5ba 380 kB/s | 32 kB 00:00 2024-03-16T11:36:17.661 INFO:teuthology.orchestra.run.smithi154.stdout:(5/174): ceph-base-19.0.0-2162.g3d78f5ba.el9.x8 2.9 MB/s | 5.4 MB 00:01 2024-03-16T11:36:17.761 INFO:teuthology.orchestra.run.smithi154.stdout:(6/174): ceph-mgr-19.0.0-2162.g3d78f5ba.el9.x86 3.4 MB/s | 1.5 MB 00:00 2024-03-16T11:36:17.868 INFO:teuthology.orchestra.run.smithi029.stdout:(14/174): libcephfs2-19.0.0-2162.g3d78f5ba.el9. 2.8 MB/s | 747 kB 00:00 2024-03-16T11:36:17.977 INFO:teuthology.orchestra.run.smithi029.stdout:(15/174): libcephsqlite-19.0.0-2162.g3d78f5ba.e 1.5 MB/s | 163 kB 00:00 2024-03-16T11:36:18.078 INFO:teuthology.orchestra.run.smithi029.stdout:(16/174): librados-devel-19.0.0-2162.g3d78f5ba. 1.2 MB/s | 127 kB 00:00 2024-03-16T11:36:18.320 INFO:teuthology.orchestra.run.smithi029.stdout:(17/174): libradosstriper1-19.0.0-2162.g3d78f5b 2.0 MB/s | 501 kB 00:00 2024-03-16T11:36:18.854 INFO:teuthology.orchestra.run.smithi029.stdout:(18/174): ceph-osd-19.0.0-2162.g3d78f5ba.el9.x8 5.4 MB/s | 17 MB 00:03 2024-03-16T11:36:18.886 INFO:teuthology.orchestra.run.smithi154.stdout:(7/174): ceph-mon-19.0.0-2162.g3d78f5ba.el9.x86 3.8 MB/s | 4.7 MB 00:01 2024-03-16T11:36:18.893 INFO:teuthology.orchestra.run.smithi152.stdout:CentOS Stream 9 - CRB 781 kB/s | 6.1 MB 00:08 2024-03-16T11:36:18.905 INFO:teuthology.orchestra.run.smithi029.stdout:(19/174): python3-ceph-argparse-19.0.0-2162.g3d 902 kB/s | 45 kB 00:00 2024-03-16T11:36:18.990 INFO:teuthology.orchestra.run.smithi029.stdout:(20/174): python3-ceph-common-19.0.0-2162.g3d78 1.5 MB/s | 131 kB 00:00 2024-03-16T11:36:19.081 INFO:teuthology.orchestra.run.smithi029.stdout:(21/174): python3-cephfs-19.0.0-2162.g3d78f5ba. 1.8 MB/s | 162 kB 00:00 2024-03-16T11:36:19.174 INFO:teuthology.orchestra.run.smithi029.stdout:(22/174): python3-rados-19.0.0-2162.g3d78f5ba.e 3.3 MB/s | 320 kB 00:00 2024-03-16T11:36:19.275 INFO:teuthology.orchestra.run.smithi029.stdout:(23/174): python3-rbd-19.0.0-2162.g3d78f5ba.el9 2.9 MB/s | 298 kB 00:00 2024-03-16T11:36:19.359 INFO:teuthology.orchestra.run.smithi029.stdout:(24/174): python3-rgw-19.0.0-2162.g3d78f5ba.el9 1.2 MB/s | 99 kB 00:00 2024-03-16T11:36:19.442 INFO:teuthology.orchestra.run.smithi029.stdout:(25/174): rbd-fuse-19.0.0-2162.g3d78f5ba.el9.x8 1.0 MB/s | 85 kB 00:00 2024-03-16T11:36:19.618 INFO:teuthology.orchestra.run.smithi029.stdout:(26/174): librgw2-19.0.0-2162.g3d78f5ba.el9.x86 4.0 MB/s | 5.1 MB 00:01 2024-03-16T11:36:19.710 INFO:teuthology.orchestra.run.smithi029.stdout:(27/174): rbd-nbd-19.0.0-2162.g3d78f5ba.el9.x86 1.8 MB/s | 170 kB 00:00 2024-03-16T11:36:19.794 INFO:teuthology.orchestra.run.smithi029.stdout:(28/174): ceph-grafana-dashboards-19.0.0-2162.g 407 kB/s | 34 kB 00:00 2024-03-16T11:36:19.886 INFO:teuthology.orchestra.run.smithi029.stdout:(29/174): ceph-mgr-cephadm-19.0.0-2162.g3d78f5b 1.5 MB/s | 139 kB 00:00 2024-03-16T11:36:20.204 INFO:teuthology.orchestra.run.smithi029.stdout:(30/174): rbd-mirror-19.0.0-2162.g3d78f5ba.el9. 4.1 MB/s | 3.1 MB 00:00 2024-03-16T11:36:20.413 INFO:teuthology.orchestra.run.smithi029.stdout:(31/174): ceph-mgr-dashboard-19.0.0-2162.g3d78f 3.4 MB/s | 1.8 MB 00:00 2024-03-16T11:36:20.436 INFO:teuthology.orchestra.run.smithi152.stdout:CentOS Stream 9 - Extras packages 38 kB/s | 15 kB 00:00 2024-03-16T11:36:20.488 INFO:teuthology.orchestra.run.smithi029.stdout:(32/174): ceph-mgr-modules-core-19.0.0-2162.g3d 3.2 MB/s | 248 kB 00:00 2024-03-16T11:36:20.536 INFO:teuthology.orchestra.run.smithi154.stdout:(8/174): ceph-osd-19.0.0-2162.g3d78f5ba.el9.x86 6.1 MB/s | 17 MB 00:02 2024-03-16T11:36:20.555 INFO:teuthology.orchestra.run.smithi029.stdout:(33/174): ceph-mgr-rook-19.0.0-2162.g3d78f5ba.e 738 kB/s | 49 kB 00:00 2024-03-16T11:36:20.603 INFO:teuthology.orchestra.run.smithi154.stdout:(9/174): ceph-selinux-19.0.0-2162.g3d78f5ba.el9 378 kB/s | 25 kB 00:00 2024-03-16T11:36:20.631 INFO:teuthology.orchestra.run.smithi029.stdout:(34/174): ceph-prometheus-alerts-19.0.0-2162.g3 218 kB/s | 16 kB 00:00 2024-03-16T11:36:20.740 INFO:teuthology.orchestra.run.smithi029.stdout:(35/174): ceph-volume-19.0.0-2162.g3d78f5ba.el9 2.4 MB/s | 264 kB 00:00 2024-03-16T11:36:20.975 INFO:teuthology.orchestra.run.smithi029.stdout:(36/174): cephadm-19.0.0-2162.g3d78f5ba.el9.noa 3.1 MB/s | 748 kB 00:00 2024-03-16T11:36:21.059 INFO:teuthology.orchestra.run.smithi029.stdout:(37/174): cephfs-top-19.0.0-2162.g3d78f5ba.el9. 218 kB/s | 18 kB 00:00 2024-03-16T11:36:21.226 INFO:teuthology.orchestra.run.smithi029.stdout:(38/174): automake-1.16.2-8.el9.noarch.rpm 3.9 MB/s | 662 kB 00:00 2024-03-16T11:36:21.302 INFO:teuthology.orchestra.run.smithi029.stdout:(39/174): bison-3.7.4-5.el9.x86_64.rpm 12 MB/s | 920 kB 00:00 2024-03-16T11:36:21.344 INFO:teuthology.orchestra.run.smithi029.stdout:(40/174): boost-program-options-1.75.0-8.el9.x8 2.5 MB/s | 106 kB 00:00 2024-03-16T11:36:21.386 INFO:teuthology.orchestra.run.smithi029.stdout:(41/174): flex-2.6.4-9.el9.x86_64.rpm 7.2 MB/s | 307 kB 00:00 2024-03-16T11:36:21.420 INFO:teuthology.orchestra.run.smithi029.stdout:(42/174): flexiblas-3.0.4-8.el9.x86_64.rpm 942 kB/s | 32 kB 00:00 2024-03-16T11:36:21.438 INFO:teuthology.orchestra.run.smithi152.stdout:Extra Packages for Enterprise Linux 23 MB/s | 21 MB 00:00 2024-03-16T11:36:21.529 INFO:teuthology.orchestra.run.smithi029.stdout:(43/174): flexiblas-netlib-3.0.4-8.el9.x86_64.r 28 MB/s | 3.0 MB 00:00 2024-03-16T11:36:21.663 INFO:teuthology.orchestra.run.smithi029.stdout:(44/174): ceph-mgr-diskprediction-local-19.0.0- 5.1 MB/s | 7.4 MB 00:01 2024-03-16T11:36:21.670 INFO:teuthology.orchestra.run.smithi154.stdout:(10/174): ceph-common-19.0.0-2162.g3d78f5ba.el9 3.5 MB/s | 21 MB 00:05 2024-03-16T11:36:21.688 INFO:teuthology.orchestra.run.smithi029.stdout:(45/174): flexiblas-openblas-openmp-3.0.4-8.el9 108 kB/s | 17 kB 00:00 2024-03-16T11:36:21.714 INFO:teuthology.orchestra.run.smithi029.stdout:(46/174): libaio-devel-0.3.111-13.el9.x86_64.rp 199 kB/s | 10 kB 00:00 2024-03-16T11:36:21.740 INFO:teuthology.orchestra.run.smithi029.stdout:(47/174): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.1 MB/s | 159 kB 00:00 2024-03-16T11:36:21.765 INFO:teuthology.orchestra.run.smithi029.stdout:(48/174): librabbitmq-0.11.0-7.el9.x86_64.rpm 875 kB/s | 44 kB 00:00 2024-03-16T11:36:21.778 INFO:teuthology.orchestra.run.smithi154.stdout:(11/174): ceph-radosgw-19.0.0-2162.g3d78f5ba.el 3.4 MB/s | 9.8 MB 00:02 2024-03-16T11:36:21.799 INFO:teuthology.orchestra.run.smithi029.stdout:(49/174): librdkafka-1.6.1-102.el9.x86_64.rpm 11 MB/s | 661 kB 00:00 2024-03-16T11:36:21.803 INFO:teuthology.orchestra.run.smithi154.stdout:(12/174): cephfs-mirror-19.0.0-2162.g3d78f5ba.e 1.6 MB/s | 221 kB 00:00 2024-03-16T11:36:21.833 INFO:teuthology.orchestra.run.smithi029.stdout:(50/174): libtool-2.4.6-45.el9.x86_64.rpm 8.5 MB/s | 578 kB 00:00 2024-03-16T11:36:21.837 INFO:teuthology.orchestra.run.smithi154.stdout:(13/174): cephfs-shell-19.0.0-2162.g3d78f5ba.el 464 kB/s | 27 kB 00:00 2024-03-16T11:36:21.860 INFO:teuthology.orchestra.run.smithi029.stdout:(51/174): libzstd-devel-1.5.1-2.el9.x86_64.rpm 777 kB/s | 46 kB 00:00 2024-03-16T11:36:21.871 INFO:teuthology.orchestra.run.smithi154.stdout:(14/174): libcephfs-devel-19.0.0-2162.g3d78f5ba 473 kB/s | 32 kB 00:00 2024-03-16T11:36:21.894 INFO:teuthology.orchestra.run.smithi029.stdout:(52/174): lttng-ust-2.12.0-6.el9.x86_64.rpm 4.6 MB/s | 282 kB 00:00 2024-03-16T11:36:21.928 INFO:teuthology.orchestra.run.smithi029.stdout:(53/174): lua-5.4.4-4.el9.x86_64.rpm 2.7 MB/s | 187 kB 00:00 2024-03-16T11:36:21.938 INFO:teuthology.orchestra.run.smithi154.stdout:(15/174): libcephsqlite-19.0.0-2162.g3d78f5ba.e 2.4 MB/s | 163 kB 00:00 2024-03-16T11:36:21.953 INFO:teuthology.orchestra.run.smithi029.stdout:(54/174): oniguruma-6.9.6-1.el9.5.x86_64.rpm 3.6 MB/s | 217 kB 00:00 2024-03-16T11:36:21.970 INFO:teuthology.orchestra.run.smithi029.stdout:(55/174): openblas-0.3.21-2.el9.x86_64.rpm 816 kB/s | 34 kB 00:00 2024-03-16T11:36:22.038 INFO:teuthology.orchestra.run.smithi154.stdout:(16/174): librados-devel-19.0.0-2162.g3d78f5ba. 1.2 MB/s | 127 kB 00:00 2024-03-16T11:36:22.080 INFO:teuthology.orchestra.run.smithi154.stdout:(17/174): libcephfs2-19.0.0-2162.g3d78f5ba.el9. 3.0 MB/s | 747 kB 00:00 2024-03-16T11:36:22.105 INFO:teuthology.orchestra.run.smithi029.stdout:(56/174): openblas-openmp-0.3.21-2.el9.x86_64.r 31 MB/s | 4.7 MB 00:00 2024-03-16T11:36:22.155 INFO:teuthology.orchestra.run.smithi029.stdout:(57/174): postgresql-13.11-1.el9.x86_64.rpm 8.3 MB/s | 1.5 MB 00:00 2024-03-16T11:36:22.156 INFO:teuthology.orchestra.run.smithi154.stdout:(18/174): libradosstriper1-19.0.0-2162.g3d78f5b 4.2 MB/s | 501 kB 00:00 2024-03-16T11:36:22.197 INFO:teuthology.orchestra.run.smithi029.stdout:(58/174): postgresql-contrib-13.11-1.el9.x86_64 8.9 MB/s | 811 kB 00:00 2024-03-16T11:36:22.223 INFO:teuthology.orchestra.run.smithi029.stdout:(59/174): postgresql-private-libs-13.11-1.el9.x 2.0 MB/s | 134 kB 00:00 2024-03-16T11:36:22.230 INFO:teuthology.orchestra.run.smithi154.stdout:(19/174): python3-ceph-argparse-19.0.0-2162.g3d 601 kB/s | 45 kB 00:00 2024-03-16T11:36:22.314 INFO:teuthology.orchestra.run.smithi154.stdout:(20/174): python3-ceph-common-19.0.0-2162.g3d78 1.5 MB/s | 131 kB 00:00 2024-03-16T11:36:22.389 INFO:teuthology.orchestra.run.smithi154.stdout:(21/174): python3-cephfs-19.0.0-2162.g3d78f5ba. 2.1 MB/s | 162 kB 00:00 2024-03-16T11:36:22.498 INFO:teuthology.orchestra.run.smithi154.stdout:(22/174): python3-rados-19.0.0-2162.g3d78f5ba.e 2.9 MB/s | 320 kB 00:00 2024-03-16T11:36:22.590 INFO:teuthology.orchestra.run.smithi154.stdout:(23/174): python3-rbd-19.0.0-2162.g3d78f5ba.el9 3.2 MB/s | 298 kB 00:00 2024-03-16T11:36:22.666 INFO:teuthology.orchestra.run.smithi154.stdout:(24/174): python3-rgw-19.0.0-2162.g3d78f5ba.el9 1.3 MB/s | 99 kB 00:00 2024-03-16T11:36:22.733 INFO:teuthology.orchestra.run.smithi154.stdout:(25/174): rbd-fuse-19.0.0-2162.g3d78f5ba.el9.x8 1.3 MB/s | 85 kB 00:00 2024-03-16T11:36:22.966 INFO:teuthology.orchestra.run.smithi154.stdout:(26/174): librgw2-19.0.0-2162.g3d78f5ba.el9.x86 5.8 MB/s | 5.1 MB 00:00 2024-03-16T11:36:22.998 INFO:teuthology.orchestra.run.smithi029.stdout:(60/174): ceph-test-19.0.0-2162.g3d78f5ba.el9.x 7.2 MB/s | 48 MB 00:06 2024-03-16T11:36:23.033 INFO:teuthology.orchestra.run.smithi154.stdout:(27/174): rbd-nbd-19.0.0-2162.g3d78f5ba.el9.x86 2.5 MB/s | 170 kB 00:00 2024-03-16T11:36:23.109 INFO:teuthology.orchestra.run.smithi154.stdout:(28/174): ceph-grafana-dashboards-19.0.0-2162.g 453 kB/s | 34 kB 00:00 2024-03-16T11:36:23.124 INFO:teuthology.orchestra.run.smithi029.stdout:(61/174): postgresql-server-13.11-1.el9.x86_64. 6.1 MB/s | 5.7 MB 00:00 2024-03-16T11:36:23.166 INFO:teuthology.orchestra.run.smithi029.stdout:(62/174): protobuf-3.14.0-13.el9.x86_64.rpm 1.1 MB/s | 1.0 MB 00:00 2024-03-16T11:36:23.184 INFO:teuthology.orchestra.run.smithi154.stdout:(29/174): ceph-mgr-cephadm-19.0.0-2162.g3d78f5b 1.8 MB/s | 139 kB 00:00 2024-03-16T11:36:23.192 INFO:teuthology.orchestra.run.smithi029.stdout:(63/174): python3-attrs-20.3.0-7.el9.noarch.rpm 409 kB/s | 78 kB 00:00 2024-03-16T11:36:23.317 INFO:teuthology.orchestra.run.smithi154.stdout:(30/174): rbd-mirror-19.0.0-2162.g3d78f5ba.el9. 5.3 MB/s | 3.1 MB 00:00 2024-03-16T11:36:23.351 INFO:teuthology.orchestra.run.smithi029.stdout:(64/174): python3-babel-2.9.1-2.el9.noarch.rpm 26 MB/s | 5.8 MB 00:00 2024-03-16T11:36:23.376 INFO:teuthology.orchestra.run.smithi029.stdout:(65/174): python3-jsonpatch-1.21-16.el9.noarch. 135 kB/s | 24 kB 00:00 2024-03-16T11:36:23.393 INFO:teuthology.orchestra.run.smithi029.stdout:(66/174): python3-jsonpointer-2.0-4.el9.noarch. 91 kB/s | 18 kB 00:00 2024-03-16T11:36:23.428 INFO:teuthology.orchestra.run.smithi029.stdout:(67/174): python3-mako-1.1.4-6.el9.noarch.rpm 1.8 MB/s | 144 kB 00:00 2024-03-16T11:36:23.454 INFO:teuthology.orchestra.run.smithi029.stdout:(68/174): python3-markupsafe-1.1.1-12.el9.x86_6 527 kB/s | 32 kB 00:00 2024-03-16T11:36:23.543 INFO:teuthology.orchestra.run.smithi154.stdout:(31/174): ceph-mgr-dashboard-19.0.0-2162.g3d78f 5.0 MB/s | 1.8 MB 00:00 2024-03-16T11:36:23.571 INFO:teuthology.orchestra.run.smithi029.stdout:(69/174): python3-numpy-1.20.1-5.el9.x86_64.rpm 27 MB/s | 4.8 MB 00:00 2024-03-16T11:36:23.605 INFO:teuthology.orchestra.run.smithi029.stdout:(70/174): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.2 MB/s | 217 kB 00:00 2024-03-16T11:36:23.626 INFO:teuthology.orchestra.run.smithi154.stdout:(32/174): ceph-mgr-modules-core-19.0.0-2162.g3d 2.9 MB/s | 248 kB 00:00 2024-03-16T11:36:23.630 INFO:teuthology.orchestra.run.smithi029.stdout:(71/174): python3-oauthlib-3.1.1-5.el9.noarch.r 1.0 MB/s | 174 kB 00:00 2024-03-16T11:36:23.656 INFO:teuthology.orchestra.run.smithi029.stdout:(72/174): python3-packaging-20.9-5.el9.noarch.r 1.3 MB/s | 69 kB 00:00 2024-03-16T11:36:23.681 INFO:teuthology.orchestra.run.smithi029.stdout:(73/174): python3-prettytable-0.7.2-27.el9.noar 546 kB/s | 41 kB 00:00 2024-03-16T11:36:23.701 INFO:teuthology.orchestra.run.smithi154.stdout:(33/174): ceph-mgr-rook-19.0.0-2162.g3d78f5ba.e 660 kB/s | 49 kB 00:00 2024-03-16T11:36:23.715 INFO:teuthology.orchestra.run.smithi029.stdout:(74/174): python3-protobuf-3.14.0-13.el9.noarch 2.8 MB/s | 238 kB 00:00 2024-03-16T11:36:23.749 INFO:teuthology.orchestra.run.smithi029.stdout:(75/174): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.9 MB/s | 132 kB 00:00 2024-03-16T11:36:23.768 INFO:teuthology.orchestra.run.smithi154.stdout:(34/174): ceph-prometheus-alerts-19.0.0-2162.g3 246 kB/s | 16 kB 00:00 2024-03-16T11:36:23.774 INFO:teuthology.orchestra.run.smithi029.stdout:(76/174): python3-pyasn1-modules-0.4.8-6.el9.no 2.2 MB/s | 211 kB 00:00 2024-03-16T11:36:23.799 INFO:teuthology.orchestra.run.smithi029.stdout:(77/174): python3-pytz-2021.1-5.el9.noarch.rpm 562 kB/s | 47 kB 00:00 2024-03-16T11:36:23.825 INFO:teuthology.orchestra.run.smithi029.stdout:(78/174): python3-requests-oauthlib-1.3.0-12.el 850 kB/s | 43 kB 00:00 2024-03-16T11:36:23.861 INFO:teuthology.orchestra.run.smithi154.stdout:(35/174): ceph-volume-19.0.0-2162.g3d78f5ba.el9 2.8 MB/s | 264 kB 00:00 2024-03-16T11:36:23.892 INFO:teuthology.orchestra.run.smithi029.stdout:(79/174): python3-toml-0.10.2-6.el9.noarch.rpm 419 kB/s | 39 kB 00:00 2024-03-16T11:36:24.036 INFO:teuthology.orchestra.run.smithi154.stdout:(36/174): cephadm-19.0.0-2162.g3d78f5ba.el9.noa 4.2 MB/s | 748 kB 00:00 2024-03-16T11:36:24.128 INFO:teuthology.orchestra.run.smithi154.stdout:(37/174): cephfs-top-19.0.0-2162.g3d78f5ba.el9. 199 kB/s | 18 kB 00:00 2024-03-16T11:36:24.218 INFO:teuthology.orchestra.run.smithi029.stdout:(80/174): python3-scipy-1.6.2-8.el9.x86_64.rpm 35 MB/s | 16 MB 00:00 2024-03-16T11:36:24.243 INFO:teuthology.orchestra.run.smithi029.stdout:(81/174): python3-wcwidth-0.2.5-8.el9.noarch.rp 99 kB/s | 41 kB 00:00 2024-03-16T11:36:24.253 INFO:teuthology.orchestra.run.smithi154.stdout:(38/174): automake-1.16.2-8.el9.noarch.rpm 5.2 MB/s | 662 kB 00:00 2024-03-16T11:36:24.268 INFO:teuthology.orchestra.run.smithi029.stdout:(82/174): socat-1.7.4.1-5.el9.x86_64.rpm 803 kB/s | 300 kB 00:00 2024-03-16T11:36:24.294 INFO:teuthology.orchestra.run.smithi029.stdout:(83/174): uuid-1.6.2-55.el9.x86_64.rpm 752 kB/s | 56 kB 00:00 2024-03-16T11:36:24.303 INFO:teuthology.orchestra.run.smithi154.stdout:(39/174): bison-3.7.4-5.el9.x86_64.rpm 18 MB/s | 920 kB 00:00 2024-03-16T11:36:24.321 INFO:teuthology.orchestra.run.smithi029.stdout:(84/174): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 63 kB 00:00 2024-03-16T11:36:24.337 INFO:teuthology.orchestra.run.smithi154.stdout:(40/174): boost-program-options-1.75.0-8.el9.x8 3.1 MB/s | 106 kB 00:00 2024-03-16T11:36:24.346 INFO:teuthology.orchestra.run.smithi029.stdout:(85/174): zlib-devel-1.2.11-40.el9.x86_64.rpm 570 kB/s | 44 kB 00:00 2024-03-16T11:36:24.373 INFO:teuthology.orchestra.run.smithi029.stdout:(86/174): fuse-2.9.9-15.el9.x86_64.rpm 1.0 MB/s | 79 kB 00:00 2024-03-16T11:36:24.378 INFO:teuthology.orchestra.run.smithi154.stdout:(41/174): flex-2.6.4-9.el9.x86_64.rpm 7.2 MB/s | 307 kB 00:00 2024-03-16T11:36:24.406 INFO:teuthology.orchestra.run.smithi029.stdout:(87/174): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.0 MB/s | 241 kB 00:00 2024-03-16T11:36:24.412 INFO:teuthology.orchestra.run.smithi154.stdout:(42/174): flexiblas-3.0.4-8.el9.x86_64.rpm 945 kB/s | 32 kB 00:00 2024-03-16T11:36:24.448 INFO:teuthology.orchestra.run.smithi029.stdout:(88/174): python3-cryptography-36.0.1-4.el9.x86 11 MB/s | 1.1 MB 00:00 2024-03-16T11:36:24.482 INFO:teuthology.orchestra.run.smithi029.stdout:(89/174): python3-ply-3.11-14.el9.noarch.rpm 942 kB/s | 103 kB 00:00 2024-03-16T11:36:24.515 INFO:teuthology.orchestra.run.smithi029.stdout:(90/174): python3-urllib3-1.26.5-5.el9.noarch.r 2.7 MB/s | 187 kB 00:00 2024-03-16T11:36:24.541 INFO:teuthology.orchestra.run.smithi029.stdout:(91/174): python3-pycparser-2.20-6.el9.noarch.r 1.3 MB/s | 124 kB 00:00 2024-03-16T11:36:24.708 INFO:teuthology.orchestra.run.smithi029.stdout:(92/174): jq-1.6-16.el9.x86_64.rpm 828 kB/s | 187 kB 00:00 2024-03-16T11:36:24.776 INFO:teuthology.orchestra.run.smithi029.stdout:(93/174): python3-requests-2.25.1-8.el9.noarch. 1.8 MB/s | 125 kB 00:00 2024-03-16T11:36:24.809 INFO:teuthology.orchestra.run.smithi029.stdout:(94/174): libquadmath-11.4.1-3.el9.x86_64.rpm 728 kB/s | 195 kB 00:00 2024-03-16T11:36:24.838 INFO:teuthology.orchestra.run.smithi154.stdout:(43/174): ceph-test-19.0.0-2162.g3d78f5ba.el9.x 11 MB/s | 48 MB 00:04 2024-03-16T11:36:24.851 INFO:teuthology.orchestra.run.smithi029.stdout:(95/174): libgfortran-11.4.1-3.el9.x86_64.rpm 2.3 MB/s | 803 kB 00:00 2024-03-16T11:36:24.896 INFO:teuthology.orchestra.run.smithi154.stdout:(44/174): flexiblas-netlib-3.0.4-8.el9.x86_64.r 6.2 MB/s | 3.0 MB 00:00 2024-03-16T11:36:24.921 INFO:teuthology.orchestra.run.smithi154.stdout:(45/174): flexiblas-openblas-openmp-3.0.4-8.el9 213 kB/s | 17 kB 00:00 2024-03-16T11:36:24.947 INFO:teuthology.orchestra.run.smithi154.stdout:(46/174): libaio-devel-0.3.111-13.el9.x86_64.rp 201 kB/s | 10 kB 00:00 2024-03-16T11:36:24.972 INFO:teuthology.orchestra.run.smithi154.stdout:(47/174): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.1 MB/s | 159 kB 00:00 2024-03-16T11:36:24.997 INFO:teuthology.orchestra.run.smithi154.stdout:(48/174): librabbitmq-0.11.0-7.el9.x86_64.rpm 884 kB/s | 44 kB 00:00 2024-03-16T11:36:25.023 INFO:teuthology.orchestra.run.smithi154.stdout:(49/174): librdkafka-1.6.1-102.el9.x86_64.rpm 13 MB/s | 661 kB 00:00 2024-03-16T11:36:25.048 INFO:teuthology.orchestra.run.smithi154.stdout:(50/174): libtool-2.4.6-45.el9.x86_64.rpm 12 MB/s | 578 kB 00:00 2024-03-16T11:36:25.073 INFO:teuthology.orchestra.run.smithi154.stdout:(51/174): libzstd-devel-1.5.1-2.el9.x86_64.rpm 943 kB/s | 46 kB 00:00 2024-03-16T11:36:25.099 INFO:teuthology.orchestra.run.smithi154.stdout:(52/174): lttng-ust-2.12.0-6.el9.x86_64.rpm 5.5 MB/s | 282 kB 00:00 2024-03-16T11:36:25.124 INFO:teuthology.orchestra.run.smithi154.stdout:(53/174): lua-5.4.4-4.el9.x86_64.rpm 3.6 MB/s | 187 kB 00:00 2024-03-16T11:36:25.149 INFO:teuthology.orchestra.run.smithi154.stdout:(54/174): oniguruma-6.9.6-1.el9.5.x86_64.rpm 4.3 MB/s | 217 kB 00:00 2024-03-16T11:36:25.175 INFO:teuthology.orchestra.run.smithi154.stdout:(55/174): openblas-0.3.21-2.el9.x86_64.rpm 688 kB/s | 34 kB 00:00 2024-03-16T11:36:25.275 INFO:teuthology.orchestra.run.smithi154.stdout:(56/174): ceph-mgr-diskprediction-local-19.0.0- 3.8 MB/s | 7.4 MB 00:01 2024-03-16T11:36:25.350 INFO:teuthology.orchestra.run.smithi154.stdout:(57/174): openblas-openmp-0.3.21-2.el9.x86_64.r 24 MB/s | 4.7 MB 00:00 2024-03-16T11:36:25.394 INFO:teuthology.orchestra.run.smithi154.stdout:(58/174): postgresql-13.11-1.el9.x86_64.rpm 7.1 MB/s | 1.5 MB 00:00 2024-03-16T11:36:25.417 INFO:teuthology.orchestra.run.smithi154.stdout:(59/174): postgresql-contrib-13.11-1.el9.x86_64 5.7 MB/s | 811 kB 00:00 2024-03-16T11:36:25.442 INFO:teuthology.orchestra.run.smithi154.stdout:(60/174): postgresql-private-libs-13.11-1.el9.x 1.4 MB/s | 134 kB 00:00 2024-03-16T11:36:25.493 INFO:teuthology.orchestra.run.smithi154.stdout:(61/174): protobuf-3.14.0-13.el9.x86_64.rpm 13 MB/s | 1.0 MB 00:00 2024-03-16T11:36:25.610 INFO:teuthology.orchestra.run.smithi154.stdout:(62/174): postgresql-server-13.11-1.el9.x86_64. 30 MB/s | 5.7 MB 00:00 2024-03-16T11:36:25.635 INFO:teuthology.orchestra.run.smithi154.stdout:(63/174): python3-attrs-20.3.0-7.el9.noarch.rpm 404 kB/s | 78 kB 00:00 2024-03-16T11:36:25.728 INFO:teuthology.orchestra.run.smithi154.stdout:(64/174): python3-babel-2.9.1-2.el9.noarch.rpm 25 MB/s | 5.8 MB 00:00 2024-03-16T11:36:25.744 INFO:teuthology.orchestra.run.smithi154.stdout:(65/174): python3-jsonpatch-1.21-16.el9.noarch. 186 kB/s | 24 kB 00:00 2024-03-16T11:36:25.761 INFO:teuthology.orchestra.run.smithi154.stdout:(66/174): python3-jsonpointer-2.0-4.el9.noarch. 147 kB/s | 18 kB 00:00 2024-03-16T11:36:25.786 INFO:teuthology.orchestra.run.smithi154.stdout:(67/174): python3-mako-1.1.4-6.el9.noarch.rpm 2.4 MB/s | 144 kB 00:00 2024-03-16T11:36:25.811 INFO:teuthology.orchestra.run.smithi154.stdout:(68/174): python3-markupsafe-1.1.1-12.el9.x86_6 634 kB/s | 32 kB 00:00 2024-03-16T11:36:25.837 INFO:teuthology.orchestra.run.smithi154.stdout:(69/174): python3-numpy-f2py-1.20.1-5.el9.x86_6 4.2 MB/s | 217 kB 00:00 2024-03-16T11:36:25.862 INFO:teuthology.orchestra.run.smithi154.stdout:(70/174): python3-oauthlib-3.1.1-5.el9.noarch.r 3.4 MB/s | 174 kB 00:00 2024-03-16T11:36:25.879 INFO:teuthology.orchestra.run.smithi154.stdout:(71/174): python3-packaging-20.9-5.el9.noarch.r 1.7 MB/s | 69 kB 00:00 2024-03-16T11:36:25.904 INFO:teuthology.orchestra.run.smithi154.stdout:(72/174): python3-prettytable-0.7.2-27.el9.noar 987 kB/s | 41 kB 00:00 2024-03-16T11:36:25.929 INFO:teuthology.orchestra.run.smithi154.stdout:(73/174): python3-protobuf-3.14.0-13.el9.noarch 4.9 MB/s | 238 kB 00:00 2024-03-16T11:36:25.955 INFO:teuthology.orchestra.run.smithi154.stdout:(74/174): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.6 MB/s | 132 kB 00:00 2024-03-16T11:36:26.038 INFO:teuthology.orchestra.run.smithi154.stdout:(75/174): python3-numpy-1.20.1-5.el9.x86_64.rpm 17 MB/s | 4.8 MB 00:00 2024-03-16T11:36:26.064 INFO:teuthology.orchestra.run.smithi154.stdout:(76/174): python3-pyasn1-modules-0.4.8-6.el9.no 1.5 MB/s | 211 kB 00:00 2024-03-16T11:36:26.089 INFO:teuthology.orchestra.run.smithi154.stdout:(77/174): python3-pytz-2021.1-5.el9.noarch.rpm 361 kB/s | 47 kB 00:00 2024-03-16T11:36:26.114 INFO:teuthology.orchestra.run.smithi154.stdout:(78/174): python3-requests-oauthlib-1.3.0-12.el 856 kB/s | 43 kB 00:00 2024-03-16T11:36:26.264 INFO:teuthology.orchestra.run.smithi154.stdout:(79/174): python3-toml-0.10.2-6.el9.noarch.rpm 224 kB/s | 39 kB 00:00 2024-03-16T11:36:26.292 INFO:teuthology.orchestra.run.smithi154.stdout:(80/174): python3-wcwidth-0.2.5-8.el9.noarch.rp 235 kB/s | 41 kB 00:00 2024-03-16T11:36:26.388 INFO:teuthology.orchestra.run.smithi152.stdout:lab-extras 41 kB/s | 1.7 kB 00:00 2024-03-16T11:36:26.442 INFO:teuthology.orchestra.run.smithi154.stdout:(81/174): python3-scipy-1.6.2-8.el9.x86_64.rpm 41 MB/s | 16 MB 00:00 2024-03-16T11:36:26.467 INFO:teuthology.orchestra.run.smithi154.stdout:(82/174): socat-1.7.4.1-5.el9.x86_64.rpm 1.5 MB/s | 300 kB 00:00 2024-03-16T11:36:26.492 INFO:teuthology.orchestra.run.smithi154.stdout:(83/174): uuid-1.6.2-55.el9.x86_64.rpm 282 kB/s | 56 kB 00:00 2024-03-16T11:36:26.518 INFO:teuthology.orchestra.run.smithi154.stdout:(84/174): zlib-devel-1.2.11-40.el9.x86_64.rpm 877 kB/s | 44 kB 00:00 2024-03-16T11:36:26.544 INFO:teuthology.orchestra.run.smithi154.stdout:(85/174): xmlstarlet-1.6.1-20.el9.x86_64.rpm 826 kB/s | 63 kB 00:00 2024-03-16T11:36:26.568 INFO:teuthology.orchestra.run.smithi154.stdout:(86/174): fuse-2.9.9-15.el9.x86_64.rpm 1.0 MB/s | 79 kB 00:00 2024-03-16T11:36:26.594 INFO:teuthology.orchestra.run.smithi154.stdout:(87/174): python3-cffi-1.14.5-5.el9.x86_64.rpm 3.1 MB/s | 241 kB 00:00 2024-03-16T11:36:26.619 INFO:teuthology.orchestra.run.smithi154.stdout:(88/174): python3-ply-3.11-14.el9.noarch.rpm 2.0 MB/s | 103 kB 00:00 2024-03-16T11:36:26.653 INFO:teuthology.orchestra.run.smithi154.stdout:(89/174): python3-cryptography-36.0.1-4.el9.x86 14 MB/s | 1.1 MB 00:00 2024-03-16T11:36:26.678 INFO:teuthology.orchestra.run.smithi154.stdout:(90/174): python3-pycparser-2.20-6.el9.noarch.r 1.4 MB/s | 124 kB 00:00 2024-03-16T11:36:26.703 INFO:teuthology.orchestra.run.smithi154.stdout:(91/174): python3-urllib3-1.26.5-5.el9.noarch.r 2.2 MB/s | 187 kB 00:00 2024-03-16T11:36:26.812 INFO:teuthology.orchestra.run.smithi154.stdout:(92/174): jq-1.6-16.el9.x86_64.rpm 1.4 MB/s | 187 kB 00:00 2024-03-16T11:36:26.837 INFO:teuthology.orchestra.run.smithi154.stdout:(93/174): libquadmath-11.4.1-3.el9.x86_64.rpm 1.4 MB/s | 195 kB 00:00 2024-03-16T11:36:26.862 INFO:teuthology.orchestra.run.smithi154.stdout:(94/174): python3-requests-2.25.1-8.el9.noarch. 2.4 MB/s | 125 kB 00:00 2024-03-16T11:36:26.896 INFO:teuthology.orchestra.run.smithi154.stdout:(95/174): libgfortran-11.4.1-3.el9.x86_64.rpm 3.6 MB/s | 803 kB 00:00 2024-03-16T11:36:26.988 INFO:teuthology.orchestra.run.smithi154.stdout:(96/174): elfutils-libelf-devel-0.190-2.el9.x86 143 kB/s | 21 kB 00:00 2024-03-16T11:36:27.013 INFO:teuthology.orchestra.run.smithi154.stdout:(97/174): libuuid-devel-2.37.4-18.el9.x86_64.rp 169 kB/s | 25 kB 00:00 2024-03-16T11:36:27.097 INFO:teuthology.orchestra.run.smithi154.stdout:(98/174): python3-devel-3.9.17-2.el9.x86_64.rpm 2.2 MB/s | 245 kB 00:00 2024-03-16T11:36:27.156 INFO:teuthology.orchestra.run.smithi154.stdout:(99/174): qatlib-23.11.0-1.el9.x86_64.rpm 3.6 MB/s | 216 kB 00:00 2024-03-16T11:36:27.181 INFO:teuthology.orchestra.run.smithi154.stdout:(100/174): python3-jinja2-2.11.3-5.el9.noarch.r 1.4 MB/s | 248 kB 00:00 2024-03-16T11:36:27.206 INFO:teuthology.orchestra.run.smithi154.stdout:(101/174): qatlib-service-23.11.0-1.el9.x86_64. 693 kB/s | 35 kB 00:00 2024-03-16T11:36:27.265 INFO:teuthology.orchestra.run.smithi154.stdout:(102/174): openssl-devel-3.0.7-27.el9.x86_64.rp 11 MB/s | 4.1 MB 00:00 2024-03-16T11:36:27.290 INFO:teuthology.orchestra.run.smithi154.stdout:(103/174): qatzip-libs-1.1.2-2.el9.x86_64.rpm 409 kB/s | 44 kB 00:00 2024-03-16T11:36:27.315 INFO:teuthology.orchestra.run.smithi154.stdout:(104/174): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 862 kB/s | 94 kB 00:00 2024-03-16T11:36:27.482 INFO:teuthology.orchestra.run.smithi154.stdout:(105/174): lua-devel-5.4.4-4.el9.x86_64.rpm 103 kB/s | 22 kB 00:00 2024-03-16T11:36:27.566 INFO:teuthology.orchestra.run.smithi154.stdout:(106/174): protobuf-compiler-3.14.0-13.el9.x86_ 3.1 MB/s | 863 kB 00:00 2024-03-16T11:36:27.658 INFO:teuthology.orchestra.run.smithi154.stdout:(107/174): abseil-cpp-20211102.0-3.el9.x86_64.r 1.6 MB/s | 548 kB 00:00 2024-03-16T11:36:27.733 INFO:teuthology.orchestra.run.smithi154.stdout:(108/174): grpc-data-1.46.7-10.el9.noarch.rpm 116 kB/s | 19 kB 00:00 2024-03-16T11:36:27.758 INFO:teuthology.orchestra.run.smithi154.stdout:(109/174): gperftools-libs-2.9.1-2.el9.x86_64.r 1.1 MB/s | 309 kB 00:00 2024-03-16T11:36:27.923 INFO:teuthology.orchestra.run.smithi154.stdout:(110/174): libarrow-doc-9.0.0-10.el9.noarch.rpm 137 kB/s | 26 kB 00:00 2024-03-16T11:36:28.017 INFO:teuthology.orchestra.run.smithi154.stdout:(111/174): liboath-2.6.7-2.el9.x86_64.rpm 188 kB/s | 49 kB 00:00 2024-03-16T11:36:28.059 INFO:teuthology.orchestra.run.smithi154.stdout:(112/174): libunwind-1.6.2-1.el9.x86_64.rpm 496 kB/s | 67 kB 00:00 2024-03-16T11:36:28.126 INFO:teuthology.orchestra.run.smithi154.stdout:(113/174): luarocks-3.9.2-1.el9.noarch.rpm 1.4 MB/s | 151 kB 00:00 2024-03-16T11:36:28.193 INFO:teuthology.orchestra.run.smithi154.stdout:(114/174): libarrow-9.0.0-10.el9.x86_64.rpm 8.3 MB/s | 4.4 MB 00:00 2024-03-16T11:36:28.252 INFO:teuthology.orchestra.run.smithi154.stdout:(115/174): python3-asyncssh-2.13.2-4.el9.noarch 4.3 MB/s | 548 kB 00:00 2024-03-16T11:36:28.286 INFO:teuthology.orchestra.run.smithi154.stdout:(116/174): python3-bcrypt-3.2.2-1.el9.x86_64.rp 472 kB/s | 43 kB 00:00 2024-03-16T11:36:28.328 INFO:teuthology.orchestra.run.smithi154.stdout:(117/174): parquet-libs-9.0.0-10.el9.x86_64.rpm 3.1 MB/s | 838 kB 00:00 2024-03-16T11:36:28.370 INFO:teuthology.orchestra.run.smithi154.stdout:(118/174): python3-cachetools-4.2.4-1.el9.noarc 274 kB/s | 32 kB 00:00 2024-03-16T11:36:28.486 INFO:teuthology.orchestra.run.smithi154.stdout:(119/174): python3-cherrypy-18.6.1-2.el9.noarch 3.0 MB/s | 358 kB 00:00 2024-03-16T11:36:28.512 INFO:teuthology.orchestra.run.smithi154.stdout:(120/174): python3-certifi-2021.10.8-2.el9.noar 66 kB/s | 15 kB 00:00 2024-03-16T11:36:28.545 INFO:teuthology.orchestra.run.smithi154.stdout:(121/174): python3-cheroot-8.6.0-4.el9.noarch.r 790 kB/s | 172 kB 00:00 2024-03-16T11:36:28.604 INFO:teuthology.orchestra.run.smithi154.stdout:(122/174): python3-cmd2-2.4.2-2.el9.noarch.rpm 2.5 MB/s | 303 kB 00:00 2024-03-16T11:36:28.629 INFO:teuthology.orchestra.run.smithi154.stdout:(123/174): python3-colorama-0.4.4-12.el9.noarch 301 kB/s | 35 kB 00:00 2024-03-16T11:36:28.663 INFO:teuthology.orchestra.run.smithi154.stdout:(124/174): python3-google-auth-2.28.0-1.el9.noa 1.7 MB/s | 203 kB 00:00 2024-03-16T11:36:28.730 INFO:teuthology.orchestra.run.smithi154.stdout:(125/174): python3-grpcio-tools-1.46.7-10.el9.x 1.4 MB/s | 144 kB 00:00 2024-03-16T11:36:28.755 INFO:teuthology.orchestra.run.smithi154.stdout:(126/174): python3-jaraco-8.2.1-3.el9.noarch.rp 116 kB/s | 11 kB 00:00 2024-03-16T11:36:28.814 INFO:teuthology.orchestra.run.smithi154.stdout:(127/174): python3-jaraco-classes-3.2.1-5.el9.n 211 kB/s | 18 kB 00:00 2024-03-16T11:36:28.864 INFO:teuthology.orchestra.run.smithi154.stdout:(128/174): python3-grpcio-1.46.7-10.el9.x86_64. 7.8 MB/s | 2.0 MB 00:00 2024-03-16T11:36:28.877 INFO:teuthology.orchestra.run.smithi152.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-16T11:36:28.877 INFO:teuthology.orchestra.run.smithi152.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-16T11:36:28.881 INFO:teuthology.orchestra.run.smithi154.stdout:(129/174): python3-jaraco-collections-3.0.0-8.e 184 kB/s | 23 kB 00:00 2024-03-16T11:36:28.891 INFO:teuthology.orchestra.run.smithi152.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-03-16T11:36:28.892 INFO:teuthology.orchestra.run.smithi152.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-03-16T11:36:28.893 INFO:teuthology.orchestra.run.smithi152.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-03-16T11:36:28.894 INFO:teuthology.orchestra.run.smithi152.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-03-16T11:36:28.896 INFO:teuthology.orchestra.run.smithi152.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-03-16T11:36:28.898 INFO:teuthology.orchestra.run.smithi154.stdout:(130/174): python3-jaraco-functools-3.5.0-2.el9 232 kB/s | 19 kB 00:00 2024-03-16T11:36:28.899 INFO:teuthology.orchestra.run.smithi152.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-03-16T11:36:28.900 INFO:teuthology.orchestra.run.smithi152.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-03-16T11:36:28.948 INFO:teuthology.orchestra.run.smithi154.stdout:(131/174): python3-jaraco-text-3.2.0-6.el9.noar 235 kB/s | 20 kB 00:00 2024-03-16T11:36:28.980 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-03-16T11:36:29.001 INFO:teuthology.orchestra.run.smithi152.stdout:============================================================================================ 2024-03-16T11:36:29.001 INFO:teuthology.orchestra.run.smithi152.stdout: Package Arch Version Repository Size 2024-03-16T11:36:29.001 INFO:teuthology.orchestra.run.smithi152.stdout:============================================================================================ 2024-03-16T11:36:29.001 INFO:teuthology.orchestra.run.smithi152.stdout:Installing: 2024-03-16T11:36:29.001 INFO:teuthology.orchestra.run.smithi152.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-03-16T11:36:29.001 INFO:teuthology.orchestra.run.smithi152.stdout: ceph x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 6.6 k 2024-03-16T11:36:29.001 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-base x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 5.4 M 2024-03-16T11:36:29.001 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-fuse x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 890 k 2024-03-16T11:36:29.001 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 145 k 2024-03-16T11:36:29.001 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mgr x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 1.5 M 2024-03-16T11:36:29.001 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 139 k 2024-03-16T11:36:29.001 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 1.8 M 2024-03-16T11:36:29.001 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 7.4 M 2024-03-16T11:36:29.002 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mgr-rook noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 49 k 2024-03-16T11:36:29.002 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-radosgw x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 9.8 M 2024-03-16T11:36:29.002 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-test x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 48 M 2024-03-16T11:36:29.002 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-volume noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 264 k 2024-03-16T11:36:29.002 INFO:teuthology.orchestra.run.smithi152.stdout: cephadm noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 748 k 2024-03-16T11:36:29.002 INFO:teuthology.orchestra.run.smithi152.stdout: cephfs-mirror x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 221 k 2024-03-16T11:36:29.002 INFO:teuthology.orchestra.run.smithi152.stdout: cephfs-shell x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 27 k 2024-03-16T11:36:29.002 INFO:teuthology.orchestra.run.smithi152.stdout: cephfs-top noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 18 k 2024-03-16T11:36:29.002 INFO:teuthology.orchestra.run.smithi152.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-03-16T11:36:29.002 INFO:teuthology.orchestra.run.smithi152.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-03-16T11:36:29.002 INFO:teuthology.orchestra.run.smithi152.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-03-16T11:36:29.002 INFO:teuthology.orchestra.run.smithi152.stdout: libcephfs-devel x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 32 k 2024-03-16T11:36:29.002 INFO:teuthology.orchestra.run.smithi152.stdout: libcephfs2 x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 747 k 2024-03-16T11:36:29.002 INFO:teuthology.orchestra.run.smithi152.stdout: librados-devel x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 127 k 2024-03-16T11:36:29.003 INFO:teuthology.orchestra.run.smithi152.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-03-16T11:36:29.003 INFO:teuthology.orchestra.run.smithi152.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-03-16T11:36:29.003 INFO:teuthology.orchestra.run.smithi152.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-03-16T11:36:29.003 INFO:teuthology.orchestra.run.smithi152.stdout: postgresql x86_64 13.11-1.el9 CentOS-AppStream 1.5 M 2024-03-16T11:36:29.003 INFO:teuthology.orchestra.run.smithi152.stdout: postgresql-contrib x86_64 13.11-1.el9 CentOS-AppStream 811 k 2024-03-16T11:36:29.003 INFO:teuthology.orchestra.run.smithi152.stdout: postgresql-server x86_64 13.11-1.el9 CentOS-AppStream 5.7 M 2024-03-16T11:36:29.003 INFO:teuthology.orchestra.run.smithi152.stdout: python3-cephfs x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 162 k 2024-03-16T11:36:29.003 INFO:teuthology.orchestra.run.smithi152.stdout: python3-rados x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 320 k 2024-03-16T11:36:29.003 INFO:teuthology.orchestra.run.smithi152.stdout: python3-rbd x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 298 k 2024-03-16T11:36:29.003 INFO:teuthology.orchestra.run.smithi152.stdout: python3-rgw x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 99 k 2024-03-16T11:36:29.003 INFO:teuthology.orchestra.run.smithi152.stdout: rbd-fuse x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 85 k 2024-03-16T11:36:29.003 INFO:teuthology.orchestra.run.smithi152.stdout: rbd-mirror x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 3.1 M 2024-03-16T11:36:29.003 INFO:teuthology.orchestra.run.smithi152.stdout: rbd-nbd x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 170 k 2024-03-16T11:36:29.003 INFO:teuthology.orchestra.run.smithi152.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-03-16T11:36:29.003 INFO:teuthology.orchestra.run.smithi152.stdout:Upgrading: 2024-03-16T11:36:29.004 INFO:teuthology.orchestra.run.smithi152.stdout: NetworkManager x86_64 1:1.46.0-1.el9 baseos 2.3 M 2024-03-16T11:36:29.004 INFO:teuthology.orchestra.run.smithi152.stdout: NetworkManager-libnm x86_64 1:1.46.0-1.el9 baseos 1.8 M 2024-03-16T11:36:29.004 INFO:teuthology.orchestra.run.smithi152.stdout: NetworkManager-team x86_64 1:1.46.0-1.el9 baseos 42 k 2024-03-16T11:36:29.004 INFO:teuthology.orchestra.run.smithi152.stdout: NetworkManager-tui x86_64 1:1.46.0-1.el9 baseos 248 k 2024-03-16T11:36:29.004 INFO:teuthology.orchestra.run.smithi152.stdout: acl x86_64 2.3.1-4.el9 baseos 71 k 2024-03-16T11:36:29.004 INFO:teuthology.orchestra.run.smithi152.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 baseos 36 k 2024-03-16T11:36:29.004 INFO:teuthology.orchestra.run.smithi152.stdout: elfutils-libelf x86_64 0.190-2.el9 baseos 192 k 2024-03-16T11:36:29.004 INFO:teuthology.orchestra.run.smithi152.stdout: elfutils-libs x86_64 0.190-2.el9 baseos 256 k 2024-03-16T11:36:29.004 INFO:teuthology.orchestra.run.smithi152.stdout: libacl x86_64 2.3.1-4.el9 baseos 23 k 2024-03-16T11:36:29.004 INFO:teuthology.orchestra.run.smithi152.stdout: libacl-devel x86_64 2.3.1-4.el9 appstream 74 k 2024-03-16T11:36:29.004 INFO:teuthology.orchestra.run.smithi152.stdout: libblkid x86_64 2.37.4-18.el9 baseos 108 k 2024-03-16T11:36:29.004 INFO:teuthology.orchestra.run.smithi152.stdout: libfdisk x86_64 2.37.4-18.el9 baseos 155 k 2024-03-16T11:36:29.004 INFO:teuthology.orchestra.run.smithi152.stdout: libmount x86_64 2.37.4-18.el9 baseos 136 k 2024-03-16T11:36:29.004 INFO:teuthology.orchestra.run.smithi152.stdout: librados2 x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 3.4 M 2024-03-16T11:36:29.004 INFO:teuthology.orchestra.run.smithi152.stdout: librbd1 x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 3.1 M 2024-03-16T11:36:29.004 INFO:teuthology.orchestra.run.smithi152.stdout: libsmartcols x86_64 2.37.4-18.el9 baseos 64 k 2024-03-16T11:36:29.005 INFO:teuthology.orchestra.run.smithi152.stdout: libuuid x86_64 2.37.4-18.el9 baseos 29 k 2024-03-16T11:36:29.005 INFO:teuthology.orchestra.run.smithi152.stdout: openssl x86_64 1:3.0.7-27.el9 baseos 1.2 M 2024-03-16T11:36:29.005 INFO:teuthology.orchestra.run.smithi152.stdout: openssl-libs x86_64 1:3.0.7-27.el9 baseos 2.2 M 2024-03-16T11:36:29.005 INFO:teuthology.orchestra.run.smithi152.stdout: util-linux x86_64 2.37.4-18.el9 baseos 2.3 M 2024-03-16T11:36:29.005 INFO:teuthology.orchestra.run.smithi152.stdout: util-linux-core x86_64 2.37.4-18.el9 baseos 465 k 2024-03-16T11:36:29.005 INFO:teuthology.orchestra.run.smithi152.stdout: util-linux-user x86_64 2.37.4-18.el9 baseos 31 k 2024-03-16T11:36:29.005 INFO:teuthology.orchestra.run.smithi152.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-03-16T11:36:29.005 INFO:teuthology.orchestra.run.smithi152.stdout: xfsprogs x86_64 6.3.0-1.el9 baseos 1.1 M 2024-03-16T11:36:29.005 INFO:teuthology.orchestra.run.smithi152.stdout:Installing dependencies: 2024-03-16T11:36:29.005 INFO:teuthology.orchestra.run.smithi152.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-16T11:36:29.005 INFO:teuthology.orchestra.run.smithi152.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-03-16T11:36:29.005 INFO:teuthology.orchestra.run.smithi152.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-16T11:36:29.005 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-common x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 21 M 2024-03-16T11:36:29.005 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 34 k 2024-03-16T11:36:29.005 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mds x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 2.4 M 2024-03-16T11:36:29.005 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 248 k 2024-03-16T11:36:29.006 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mon x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 4.7 M 2024-03-16T11:36:29.006 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-osd x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 17 M 2024-03-16T11:36:29.006 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 16 k 2024-03-16T11:36:29.006 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-selinux x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 25 k 2024-03-16T11:36:29.006 INFO:teuthology.orchestra.run.smithi152.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-16T11:36:29.006 INFO:teuthology.orchestra.run.smithi152.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-16T11:36:29.006 INFO:teuthology.orchestra.run.smithi152.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-16T11:36:29.006 INFO:teuthology.orchestra.run.smithi152.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-16T11:36:29.006 INFO:teuthology.orchestra.run.smithi152.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-16T11:36:29.006 INFO:teuthology.orchestra.run.smithi152.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-16T11:36:29.006 INFO:teuthology.orchestra.run.smithi152.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-16T11:36:29.006 INFO:teuthology.orchestra.run.smithi152.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-16T11:36:29.006 INFO:teuthology.orchestra.run.smithi152.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-16T11:36:29.006 INFO:teuthology.orchestra.run.smithi152.stdout: libcephsqlite x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 163 k 2024-03-16T11:36:29.006 INFO:teuthology.orchestra.run.smithi152.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-16T11:36:29.007 INFO:teuthology.orchestra.run.smithi152.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-16T11:36:29.007 INFO:teuthology.orchestra.run.smithi152.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-16T11:36:29.007 INFO:teuthology.orchestra.run.smithi152.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-16T11:36:29.007 INFO:teuthology.orchestra.run.smithi152.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-16T11:36:29.007 INFO:teuthology.orchestra.run.smithi152.stdout: libradosstriper1 x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 501 k 2024-03-16T11:36:29.007 INFO:teuthology.orchestra.run.smithi152.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-16T11:36:29.007 INFO:teuthology.orchestra.run.smithi152.stdout: librgw2 x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 5.1 M 2024-03-16T11:36:29.007 INFO:teuthology.orchestra.run.smithi152.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-16T11:36:29.007 INFO:teuthology.orchestra.run.smithi152.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-03-16T11:36:29.007 INFO:teuthology.orchestra.run.smithi152.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-16T11:36:29.007 INFO:teuthology.orchestra.run.smithi152.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-16T11:36:29.007 INFO:teuthology.orchestra.run.smithi152.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-16T11:36:29.007 INFO:teuthology.orchestra.run.smithi152.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-16T11:36:29.007 INFO:teuthology.orchestra.run.smithi152.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-16T11:36:29.008 INFO:teuthology.orchestra.run.smithi152.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-16T11:36:29.008 INFO:teuthology.orchestra.run.smithi152.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-16T11:36:29.008 INFO:teuthology.orchestra.run.smithi152.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-16T11:36:29.008 INFO:teuthology.orchestra.run.smithi152.stdout: postgresql-private-libs x86_64 13.11-1.el9 CentOS-AppStream 134 k 2024-03-16T11:36:29.008 INFO:teuthology.orchestra.run.smithi152.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-16T11:36:29.008 INFO:teuthology.orchestra.run.smithi152.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-16T11:36:29.008 INFO:teuthology.orchestra.run.smithi152.stdout: python3-asyncssh noarch 2.13.2-4.el9 epel 548 k 2024-03-16T11:36:29.008 INFO:teuthology.orchestra.run.smithi152.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-03-16T11:36:29.008 INFO:teuthology.orchestra.run.smithi152.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-16T11:36:29.008 INFO:teuthology.orchestra.run.smithi152.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-16T11:36:29.008 INFO:teuthology.orchestra.run.smithi152.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-16T11:36:29.008 INFO:teuthology.orchestra.run.smithi152.stdout: python3-ceph-argparse x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 45 k 2024-03-16T11:36:29.008 INFO:teuthology.orchestra.run.smithi152.stdout: python3-ceph-common x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 131 k 2024-03-16T11:36:29.008 INFO:teuthology.orchestra.run.smithi152.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-16T11:36:29.008 INFO:teuthology.orchestra.run.smithi152.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-16T11:36:29.008 INFO:teuthology.orchestra.run.smithi152.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-16T11:36:29.009 INFO:teuthology.orchestra.run.smithi152.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-16T11:36:29.009 INFO:teuthology.orchestra.run.smithi152.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-03-16T11:36:29.009 INFO:teuthology.orchestra.run.smithi152.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-03-16T11:36:29.009 INFO:teuthology.orchestra.run.smithi152.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-16T11:36:29.009 INFO:teuthology.orchestra.run.smithi152.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-16T11:36:29.009 INFO:teuthology.orchestra.run.smithi152.stdout: python3-google-auth noarch 1:2.28.0-1.el9 epel 203 k 2024-03-16T11:36:29.009 INFO:teuthology.orchestra.run.smithi152.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-16T11:36:29.009 INFO:teuthology.orchestra.run.smithi152.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-16T11:36:29.009 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-16T11:36:29.009 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-16T11:36:29.009 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-16T11:36:29.009 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-16T11:36:29.009 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-16T11:36:29.009 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-16T11:36:29.009 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-16T11:36:29.009 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-16T11:36:29.009 INFO:teuthology.orchestra.run.smithi152.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-16T11:36:29.010 INFO:teuthology.orchestra.run.smithi152.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-16T11:36:29.010 INFO:teuthology.orchestra.run.smithi152.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-16T11:36:29.010 INFO:teuthology.orchestra.run.smithi152.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-16T11:36:29.010 INFO:teuthology.orchestra.run.smithi152.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-16T11:36:29.010 INFO:teuthology.orchestra.run.smithi152.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-16T11:36:29.010 INFO:teuthology.orchestra.run.smithi152.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-16T11:36:29.013 INFO:teuthology.orchestra.run.smithi152.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-16T11:36:29.013 INFO:teuthology.orchestra.run.smithi152.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-16T11:36:29.013 INFO:teuthology.orchestra.run.smithi152.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-16T11:36:29.013 INFO:teuthology.orchestra.run.smithi152.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-16T11:36:29.013 INFO:teuthology.orchestra.run.smithi152.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-16T11:36:29.013 INFO:teuthology.orchestra.run.smithi152.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-16T11:36:29.013 INFO:teuthology.orchestra.run.smithi152.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-16T11:36:29.013 INFO:teuthology.orchestra.run.smithi152.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-16T11:36:29.013 INFO:teuthology.orchestra.run.smithi152.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-16T11:36:29.013 INFO:teuthology.orchestra.run.smithi152.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-16T11:36:29.013 INFO:teuthology.orchestra.run.smithi152.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-16T11:36:29.013 INFO:teuthology.orchestra.run.smithi152.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-16T11:36:29.013 INFO:teuthology.orchestra.run.smithi152.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-03-16T11:36:29.013 INFO:teuthology.orchestra.run.smithi152.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-16T11:36:29.014 INFO:teuthology.orchestra.run.smithi152.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-16T11:36:29.014 INFO:teuthology.orchestra.run.smithi152.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-16T11:36:29.014 INFO:teuthology.orchestra.run.smithi152.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-16T11:36:29.014 INFO:teuthology.orchestra.run.smithi152.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-16T11:36:29.014 INFO:teuthology.orchestra.run.smithi152.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-16T11:36:29.014 INFO:teuthology.orchestra.run.smithi152.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-16T11:36:29.014 INFO:teuthology.orchestra.run.smithi152.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-16T11:36:29.014 INFO:teuthology.orchestra.run.smithi152.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-16T11:36:29.014 INFO:teuthology.orchestra.run.smithi152.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-16T11:36:29.014 INFO:teuthology.orchestra.run.smithi152.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-16T11:36:29.014 INFO:teuthology.orchestra.run.smithi152.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-03-16T11:36:29.014 INFO:teuthology.orchestra.run.smithi152.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-16T11:36:29.014 INFO:teuthology.orchestra.run.smithi152.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-16T11:36:29.014 INFO:teuthology.orchestra.run.smithi152.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-16T11:36:29.015 INFO:teuthology.orchestra.run.smithi152.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-16T11:36:29.015 INFO:teuthology.orchestra.run.smithi152.stdout: qatlib x86_64 23.11.0-1.el9 appstream 216 k 2024-03-16T11:36:29.015 INFO:teuthology.orchestra.run.smithi152.stdout: qatzip-libs x86_64 1.1.2-2.el9 appstream 44 k 2024-03-16T11:36:29.015 INFO:teuthology.orchestra.run.smithi152.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-16T11:36:29.015 INFO:teuthology.orchestra.run.smithi152.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-16T11:36:29.015 INFO:teuthology.orchestra.run.smithi152.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-16T11:36:29.015 INFO:teuthology.orchestra.run.smithi152.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-03-16T11:36:29.015 INFO:teuthology.orchestra.run.smithi152.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-16T11:36:29.015 INFO:teuthology.orchestra.run.smithi152.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-03-16T11:36:29.015 INFO:teuthology.orchestra.run.smithi152.stdout:Installing weak dependencies: 2024-03-16T11:36:29.015 INFO:teuthology.orchestra.run.smithi152.stdout: qatlib-service x86_64 23.11.0-1.el9 appstream 35 k 2024-03-16T11:36:29.015 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-16T11:36:29.015 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction Summary 2024-03-16T11:36:29.015 INFO:teuthology.orchestra.run.smithi152.stdout:============================================================================================ 2024-03-16T11:36:29.015 INFO:teuthology.orchestra.run.smithi152.stdout:Install 150 Packages 2024-03-16T11:36:29.016 INFO:teuthology.orchestra.run.smithi152.stdout:Upgrade 24 Packages 2024-03-16T11:36:29.016 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-16T11:36:29.016 INFO:teuthology.orchestra.run.smithi152.stdout:Total download size: 224 M 2024-03-16T11:36:29.016 INFO:teuthology.orchestra.run.smithi152.stdout:Downloading Packages: 2024-03-16T11:36:29.065 INFO:teuthology.orchestra.run.smithi154.stdout:(132/174): python3-logutils-0.3.5-21.el9.noarch 278 kB/s | 46 kB 00:00 2024-03-16T11:36:29.082 INFO:teuthology.orchestra.run.smithi154.stdout:(133/174): python3-more-itertools-8.12.0-2.el9. 591 kB/s | 79 kB 00:00 2024-03-16T11:36:29.115 INFO:teuthology.orchestra.run.smithi154.stdout:(134/174): python3-kubernetes-26.1.0-2.el9.noar 4.7 MB/s | 1.0 MB 00:00 2024-03-16T11:36:29.161 INFO:teuthology.orchestra.run.smithi154.stdout:(135/174): python3-natsort-7.1.1-5.el9.noarch.r 606 kB/s | 58 kB 00:00 2024-03-16T11:36:29.203 INFO:teuthology.orchestra.run.smithi154.stdout:(136/174): python3-pecan-1.4.2-3.el9.noarch.rpm 2.2 MB/s | 272 kB 00:00 2024-03-16T11:36:29.228 INFO:teuthology.orchestra.run.smithi154.stdout:(137/174): python3-portend-3.1.0-2.el9.noarch.r 146 kB/s | 16 kB 00:00 2024-03-16T11:36:29.253 INFO:teuthology.orchestra.run.smithi154.stdout:(138/174): python3-pyOpenSSL-21.0.0-1.el9.noarc 974 kB/s | 90 kB 00:00 2024-03-16T11:36:29.287 INFO:teuthology.orchestra.run.smithi154.stdout:(139/174): python3-pyperclip-1.8.2-2.el9.noarch 336 kB/s | 28 kB 00:00 2024-03-16T11:36:29.312 INFO:teuthology.orchestra.run.smithi154.stdout:(140/174): python3-repoze-lru-0.7-16.el9.noarch 366 kB/s | 31 kB 00:00 2024-03-16T11:36:29.338 INFO:teuthology.orchestra.run.smithi154.stdout:(141/174): python3-routes-2.5.1-5.el9.noarch.rp 2.2 MB/s | 188 kB 00:00 2024-03-16T11:36:29.388 INFO:teuthology.orchestra.run.smithi154.stdout:(142/174): python3-rsa-4.9-2.el9.noarch.rpm 588 kB/s | 59 kB 00:00 2024-03-16T11:36:29.413 INFO:teuthology.orchestra.run.smithi154.stdout:(143/174): python3-tempora-5.0.0-2.el9.noarch.r 356 kB/s | 36 kB 00:00 2024-03-16T11:36:29.438 INFO:teuthology.orchestra.run.smithi154.stdout:(144/174): python3-typing-extensions-4.4.0-2.el 508 kB/s | 51 kB 00:00 2024-03-16T11:36:29.497 INFO:teuthology.orchestra.run.smithi154.stdout:(145/174): python3-webob-1.8.7-6.el9.noarch.rpm 2.1 MB/s | 230 kB 00:00 2024-03-16T11:36:29.522 INFO:teuthology.orchestra.run.smithi154.stdout:(146/174): python3-websocket-client-1.2.3-2.el9 824 kB/s | 90 kB 00:00 2024-03-16T11:36:29.556 INFO:teuthology.orchestra.run.smithi154.stdout:(147/174): python3-werkzeug-2.0.3-3.el9.1.noarc 3.6 MB/s | 427 kB 00:00 2024-03-16T11:36:29.581 INFO:teuthology.orchestra.run.smithi154.stdout:(148/174): python3-zc-lockfile-2.0-10.el9.noarc 238 kB/s | 20 kB 00:00 2024-03-16T11:36:29.657 INFO:teuthology.orchestra.run.smithi154.stdout:(149/174): re2-20211101-3.el9.x86_64.rpm 1.4 MB/s | 192 kB 00:00 2024-03-16T11:36:29.747 INFO:teuthology.orchestra.run.smithi152.stdout:(1/174): ceph-19.0.0-2162.g3d78f5ba.el9.x86_64. 29 kB/s | 6.6 kB 00:00 2024-03-16T11:36:29.777 INFO:teuthology.orchestra.run.smithi154.stdout:(150/174): thrift-0.15.0-2.el9.x86_64.rpm 7.2 MB/s | 1.6 MB 00:00 2024-03-16T11:36:29.890 INFO:teuthology.orchestra.run.smithi154.stdout:(151/174): xfsdump-3.1.12-4.el9.x86_64.rpm 2.8 MB/s | 321 kB 00:00 2024-03-16T11:36:30.099 INFO:teuthology.orchestra.run.smithi154.stdout:(152/174): NetworkManager-1.46.0-1.el9.x86_64.r 11 MB/s | 2.3 MB 00:00 2024-03-16T11:36:30.123 INFO:teuthology.orchestra.run.smithi152.stdout:(2/174): ceph-fuse-19.0.0-2162.g3d78f5ba.el9.x8 2.3 MB/s | 890 kB 00:00 2024-03-16T11:36:30.174 INFO:teuthology.orchestra.run.smithi154.stdout:(153/174): NetworkManager-libnm-1.46.0-1.el9.x8 24 MB/s | 1.8 MB 00:00 2024-03-16T11:36:30.215 INFO:teuthology.orchestra.run.smithi152.stdout:(3/174): ceph-immutable-object-cache-19.0.0-216 1.5 MB/s | 145 kB 00:00 2024-03-16T11:36:30.216 INFO:teuthology.orchestra.run.smithi154.stdout:(154/174): NetworkManager-team-1.46.0-1.el9.x86 1.0 MB/s | 42 kB 00:00 2024-03-16T11:36:30.258 INFO:teuthology.orchestra.run.smithi154.stdout:(155/174): NetworkManager-tui-1.46.0-1.el9.x86_ 5.8 MB/s | 248 kB 00:00 2024-03-16T11:36:30.300 INFO:teuthology.orchestra.run.smithi154.stdout:(156/174): acl-2.3.1-4.el9.x86_64.rpm 1.7 MB/s | 71 kB 00:00 2024-03-16T11:36:30.306 INFO:teuthology.orchestra.run.smithi152.stdout:(4/174): ceph-base-19.0.0-2162.g3d78f5ba.el9.x8 6.8 MB/s | 5.4 MB 00:00 2024-03-16T11:36:30.350 INFO:teuthology.orchestra.run.smithi154.stdout:(157/174): librados2-19.0.0-2162.g3d78f5ba.el9. 4.4 MB/s | 3.4 MB 00:00 2024-03-16T11:36:30.376 INFO:teuthology.orchestra.run.smithi154.stdout:(158/174): elfutils-debuginfod-client-0.190-2.e 485 kB/s | 36 kB 00:00 2024-03-16T11:36:30.417 INFO:teuthology.orchestra.run.smithi154.stdout:(159/174): elfutils-libs-0.190-2.el9.x86_64.rpm 6.0 MB/s | 256 kB 00:00 2024-03-16T11:36:30.459 INFO:teuthology.orchestra.run.smithi154.stdout:(160/174): libacl-2.3.1-4.el9.x86_64.rpm 554 kB/s | 23 kB 00:00 2024-03-16T11:36:30.485 INFO:teuthology.orchestra.run.smithi154.stdout:(161/174): elfutils-libelf-0.190-2.el9.x86_64.r 1.4 MB/s | 192 kB 00:00 2024-03-16T11:36:30.490 INFO:teuthology.orchestra.run.smithi152.stdout:(5/174): ceph-mgr-19.0.0-2162.g3d78f5ba.el9.x86 8.1 MB/s | 1.5 MB 00:00 2024-03-16T11:36:30.510 INFO:teuthology.orchestra.run.smithi154.stdout:(162/174): libblkid-2.37.4-18.el9.x86_64.rpm 2.1 MB/s | 108 kB 00:00 2024-03-16T11:36:30.535 INFO:teuthology.orchestra.run.smithi154.stdout:(163/174): libfdisk-2.37.4-18.el9.x86_64.rpm 3.0 MB/s | 155 kB 00:00 2024-03-16T11:36:30.561 INFO:teuthology.orchestra.run.smithi154.stdout:(164/174): libmount-2.37.4-18.el9.x86_64.rpm 2.7 MB/s | 136 kB 00:00 2024-03-16T11:36:30.577 INFO:teuthology.orchestra.run.smithi154.stdout:(165/174): libsmartcols-2.37.4-18.el9.x86_64.rp 1.5 MB/s | 64 kB 00:00 2024-03-16T11:36:30.594 INFO:teuthology.orchestra.run.smithi154.stdout:(166/174): libuuid-2.37.4-18.el9.x86_64.rpm 852 kB/s | 29 kB 00:00 2024-03-16T11:36:30.662 INFO:teuthology.orchestra.run.smithi154.stdout:(167/174): openssl-3.0.7-27.el9.x86_64.rpm 14 MB/s | 1.2 MB 00:00 2024-03-16T11:36:30.722 INFO:teuthology.orchestra.run.smithi154.stdout:(168/174): openssl-libs-3.0.7-27.el9.x86_64.rpm 17 MB/s | 2.2 MB 00:00 2024-03-16T11:36:30.781 INFO:teuthology.orchestra.run.smithi154.stdout:(169/174): util-linux-2.37.4-18.el9.x86_64.rpm 19 MB/s | 2.3 MB 00:00 2024-03-16T11:36:30.806 INFO:teuthology.orchestra.run.smithi154.stdout:(170/174): util-linux-core-2.37.4-18.el9.x86_64 5.4 MB/s | 465 kB 00:00 2024-03-16T11:36:30.831 INFO:teuthology.orchestra.run.smithi154.stdout:(171/174): util-linux-user-2.37.4-18.el9.x86_64 622 kB/s | 31 kB 00:00 2024-03-16T11:36:30.865 INFO:teuthology.orchestra.run.smithi154.stdout:(172/174): xfsprogs-6.3.0-1.el9.x86_64.rpm 18 MB/s | 1.1 MB 00:00 2024-03-16T11:36:30.865 INFO:teuthology.orchestra.run.smithi152.stdout:(6/174): ceph-mds-19.0.0-2162.g3d78f5ba.el9.x86 3.6 MB/s | 2.4 MB 00:00 2024-03-16T11:36:30.991 INFO:teuthology.orchestra.run.smithi152.stdout:(7/174): ceph-mon-19.0.0-2162.g3d78f5ba.el9.x86 9.4 MB/s | 4.7 MB 00:00 2024-03-16T11:36:31.090 INFO:teuthology.orchestra.run.smithi154.stdout:(173/174): libacl-devel-2.3.1-4.el9.x86_64.rpm 287 kB/s | 74 kB 00:00 2024-03-16T11:36:31.516 INFO:teuthology.orchestra.run.smithi152.stdout:(8/174): ceph-common-19.0.0-2162.g3d78f5ba.el9. 10 MB/s | 21 MB 00:01 2024-03-16T11:36:31.575 INFO:teuthology.orchestra.run.smithi152.stdout:(9/174): ceph-selinux-19.0.0-2162.g3d78f5ba.el9 434 kB/s | 25 kB 00:00 2024-03-16T11:36:32.100 INFO:teuthology.orchestra.run.smithi152.stdout:(10/174): ceph-radosgw-19.0.0-2162.g3d78f5ba.el 8.9 MB/s | 9.8 MB 00:01 2024-03-16T11:36:32.184 INFO:teuthology.orchestra.run.smithi152.stdout:(11/174): cephfs-mirror-19.0.0-2162.g3d78f5ba.e 2.6 MB/s | 221 kB 00:00 2024-03-16T11:36:32.240 INFO:teuthology.orchestra.run.smithi154.stdout:(174/174): librbd1-19.0.0-2162.g3d78f5ba.el9.x8 1.2 MB/s | 3.1 MB 00:02 2024-03-16T11:36:32.246 INFO:teuthology.orchestra.run.smithi154.stdout:-------------------------------------------------------------------------------- 2024-03-16T11:36:32.246 INFO:teuthology.orchestra.run.smithi154.stdout:Total 13 MB/s | 224 MB 00:16 2024-03-16T11:36:32.267 INFO:teuthology.orchestra.run.smithi152.stdout:(12/174): cephfs-shell-19.0.0-2162.g3d78f5ba.el 324 kB/s | 27 kB 00:00 2024-03-16T11:36:32.342 INFO:teuthology.orchestra.run.smithi152.stdout:(13/174): libcephfs-devel-19.0.0-2162.g3d78f5ba 424 kB/s | 32 kB 00:00 2024-03-16T11:36:32.713 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction check 2024-03-16T11:36:32.734 INFO:teuthology.orchestra.run.smithi152.stdout:(14/174): libcephfs2-19.0.0-2162.g3d78f5ba.el9. 1.9 MB/s | 747 kB 00:00 2024-03-16T11:36:32.818 INFO:teuthology.orchestra.run.smithi152.stdout:(15/174): libcephsqlite-19.0.0-2162.g3d78f5ba.e 1.9 MB/s | 163 kB 00:00 2024-03-16T11:36:32.902 INFO:teuthology.orchestra.run.smithi152.stdout:(16/174): librados-devel-19.0.0-2162.g3d78f5ba. 1.5 MB/s | 127 kB 00:00 2024-03-16T11:36:32.984 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction check succeeded. 2024-03-16T11:36:32.984 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction test 2024-03-16T11:36:33.060 INFO:teuthology.orchestra.run.smithi152.stdout:(17/174): ceph-osd-19.0.0-2162.g3d78f5ba.el9.x8 7.7 MB/s | 17 MB 00:02 2024-03-16T11:36:33.094 INFO:teuthology.orchestra.run.smithi152.stdout:(18/174): libradosstriper1-19.0.0-2162.g3d78f5b 2.6 MB/s | 501 kB 00:00 2024-03-16T11:36:33.144 INFO:teuthology.orchestra.run.smithi152.stdout:(19/174): python3-ceph-argparse-19.0.0-2162.g3d 896 kB/s | 45 kB 00:00 2024-03-16T11:36:33.228 INFO:teuthology.orchestra.run.smithi152.stdout:(20/174): python3-ceph-common-19.0.0-2162.g3d78 1.5 MB/s | 131 kB 00:00 2024-03-16T11:36:33.312 INFO:teuthology.orchestra.run.smithi152.stdout:(21/174): python3-cephfs-19.0.0-2162.g3d78f5ba. 1.9 MB/s | 162 kB 00:00 2024-03-16T11:36:33.404 INFO:teuthology.orchestra.run.smithi152.stdout:(22/174): python3-rados-19.0.0-2162.g3d78f5ba.e 3.4 MB/s | 320 kB 00:00 2024-03-16T11:36:33.496 INFO:teuthology.orchestra.run.smithi152.stdout:(23/174): python3-rbd-19.0.0-2162.g3d78f5ba.el9 3.2 MB/s | 298 kB 00:00 2024-03-16T11:36:33.588 INFO:teuthology.orchestra.run.smithi152.stdout:(24/174): python3-rgw-19.0.0-2162.g3d78f5ba.el9 1.1 MB/s | 99 kB 00:00 2024-03-16T11:36:33.663 INFO:teuthology.orchestra.run.smithi152.stdout:(25/174): librgw2-19.0.0-2162.g3d78f5ba.el9.x86 8.5 MB/s | 5.1 MB 00:00 2024-03-16T11:36:33.688 INFO:teuthology.orchestra.run.smithi152.stdout:(26/174): rbd-fuse-19.0.0-2162.g3d78f5ba.el9.x8 852 kB/s | 85 kB 00:00 2024-03-16T11:36:33.772 INFO:teuthology.orchestra.run.smithi152.stdout:(27/174): rbd-nbd-19.0.0-2162.g3d78f5ba.el9.x86 2.0 MB/s | 170 kB 00:00 2024-03-16T11:36:33.839 INFO:teuthology.orchestra.run.smithi152.stdout:(28/174): ceph-grafana-dashboards-19.0.0-2162.g 511 kB/s | 34 kB 00:00 2024-03-16T11:36:33.923 INFO:teuthology.orchestra.run.smithi152.stdout:(29/174): ceph-mgr-cephadm-19.0.0-2162.g3d78f5b 1.6 MB/s | 139 kB 00:00 2024-03-16T11:36:34.048 INFO:teuthology.orchestra.run.smithi152.stdout:(30/174): rbd-mirror-19.0.0-2162.g3d78f5ba.el9. 8.0 MB/s | 3.1 MB 00:00 2024-03-16T11:36:34.274 INFO:teuthology.orchestra.run.smithi152.stdout:(31/174): ceph-mgr-dashboard-19.0.0-2162.g3d78f 5.1 MB/s | 1.8 MB 00:00 2024-03-16T11:36:34.365 INFO:teuthology.orchestra.run.smithi152.stdout:(32/174): ceph-mgr-modules-core-19.0.0-2162.g3d 2.7 MB/s | 248 kB 00:00 2024-03-16T11:36:34.441 INFO:teuthology.orchestra.run.smithi152.stdout:(33/174): ceph-mgr-rook-19.0.0-2162.g3d78f5ba.e 658 kB/s | 49 kB 00:00 2024-03-16T11:36:34.691 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction test succeeded. 2024-03-16T11:36:34.691 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction 2024-03-16T11:36:34.875 INFO:teuthology.orchestra.run.smithi152.stdout:(34/174): ceph-test-19.0.0-2162.g3d78f5ba.el9.x 14 MB/s | 48 MB 00:03 2024-03-16T11:36:34.899 INFO:teuthology.orchestra.run.smithi152.stdout:(35/174): ceph-prometheus-alerts-19.0.0-2162.g3 36 kB/s | 16 kB 00:00 2024-03-16T11:36:34.925 INFO:teuthology.orchestra.run.smithi152.stdout:(36/174): ceph-volume-19.0.0-2162.g3d78f5ba.el9 5.2 MB/s | 264 kB 00:00 2024-03-16T11:36:35.025 INFO:teuthology.orchestra.run.smithi152.stdout:(37/174): ceph-mgr-diskprediction-local-19.0.0- 7.6 MB/s | 7.4 MB 00:00 2024-03-16T11:36:35.050 INFO:teuthology.orchestra.run.smithi152.stdout:(38/174): cephfs-top-19.0.0-2162.g3d78f5ba.el9. 146 kB/s | 18 kB 00:00 2024-03-16T11:36:35.093 INFO:teuthology.orchestra.run.smithi152.stdout:(39/174): cephadm-19.0.0-2162.g3d78f5ba.el9.noa 3.8 MB/s | 748 kB 00:00 2024-03-16T11:36:35.243 INFO:teuthology.orchestra.run.smithi152.stdout:(40/174): boost-program-options-1.75.0-8.el9.x8 1.1 MB/s | 106 kB 00:00 2024-03-16T11:36:35.268 INFO:teuthology.orchestra.run.smithi152.stdout:(41/174): automake-1.16.2-8.el9.noarch.rpm 2.7 MB/s | 662 kB 00:00 2024-03-16T11:36:35.302 INFO:teuthology.orchestra.run.smithi152.stdout:(42/174): bison-3.7.4-5.el9.x86_64.rpm 3.6 MB/s | 920 kB 00:00 2024-03-16T11:36:35.327 INFO:teuthology.orchestra.run.smithi152.stdout:(43/174): flex-2.6.4-9.el9.x86_64.rpm 3.6 MB/s | 307 kB 00:00 2024-03-16T11:36:35.352 INFO:teuthology.orchestra.run.smithi152.stdout:(44/174): flexiblas-3.0.4-8.el9.x86_64.rpm 638 kB/s | 32 kB 00:00 2024-03-16T11:36:35.419 INFO:teuthology.orchestra.run.smithi152.stdout:(45/174): flexiblas-netlib-3.0.4-8.el9.x86_64.r 26 MB/s | 3.0 MB 00:00 2024-03-16T11:36:35.453 INFO:teuthology.orchestra.run.smithi152.stdout:(46/174): flexiblas-openblas-openmp-3.0.4-8.el9 138 kB/s | 17 kB 00:00 2024-03-16T11:36:35.470 INFO:teuthology.orchestra.run.smithi152.stdout:(47/174): libaio-devel-0.3.111-13.el9.x86_64.rp 87 kB/s | 10 kB 00:00 2024-03-16T11:36:35.495 INFO:teuthology.orchestra.run.smithi152.stdout:(48/174): libpmemobj-1.12.1-1.el9.x86_64.rpm 2.1 MB/s | 159 kB 00:00 2024-03-16T11:36:35.512 INFO:teuthology.orchestra.run.smithi152.stdout:(49/174): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.0 MB/s | 44 kB 00:00 2024-03-16T11:36:35.537 INFO:teuthology.orchestra.run.smithi152.stdout:(50/174): librdkafka-1.6.1-102.el9.x86_64.rpm 9.7 MB/s | 661 kB 00:00 2024-03-16T11:36:35.571 INFO:teuthology.orchestra.run.smithi152.stdout:(51/174): libtool-2.4.6-45.el9.x86_64.rpm 7.6 MB/s | 578 kB 00:00 2024-03-16T11:36:35.596 INFO:teuthology.orchestra.run.smithi152.stdout:(52/174): libzstd-devel-1.5.1-2.el9.x86_64.rpm 806 kB/s | 46 kB 00:00 2024-03-16T11:36:35.629 INFO:teuthology.orchestra.run.smithi152.stdout:(53/174): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.0 MB/s | 282 kB 00:00 2024-03-16T11:36:35.655 INFO:teuthology.orchestra.run.smithi152.stdout:(54/174): lua-5.4.4-4.el9.x86_64.rpm 2.2 MB/s | 187 kB 00:00 2024-03-16T11:36:35.680 INFO:teuthology.orchestra.run.smithi152.stdout:(55/174): openblas-0.3.21-2.el9.x86_64.rpm 686 kB/s | 34 kB 00:00 2024-03-16T11:36:35.711 INFO:teuthology.orchestra.run.smithi152.stdout:(56/174): oniguruma-6.9.6-1.el9.5.x86_64.rpm 2.6 MB/s | 217 kB 00:00 2024-03-16T11:36:35.787 INFO:teuthology.orchestra.run.smithi152.stdout:(57/174): openblas-openmp-0.3.21-2.el9.x86_64.r 36 MB/s | 4.7 MB 00:00 2024-03-16T11:36:35.829 INFO:teuthology.orchestra.run.smithi152.stdout:(58/174): postgresql-13.11-1.el9.x86_64.rpm 10 MB/s | 1.5 MB 00:00 2024-03-16T11:36:35.862 INFO:teuthology.orchestra.run.smithi152.stdout:(59/174): postgresql-contrib-13.11-1.el9.x86_64 5.4 MB/s | 811 kB 00:00 2024-03-16T11:36:35.888 INFO:teuthology.orchestra.run.smithi152.stdout:(60/174): postgresql-private-libs-13.11-1.el9.x 1.3 MB/s | 134 kB 00:00 2024-03-16T11:36:35.930 INFO:teuthology.orchestra.run.smithi152.stdout:(61/174): protobuf-3.14.0-13.el9.x86_64.rpm 15 MB/s | 1.0 MB 00:00 2024-03-16T11:36:36.031 INFO:teuthology.orchestra.run.smithi152.stdout:(62/174): postgresql-server-13.11-1.el9.x86_64. 34 MB/s | 5.7 MB 00:00 2024-03-16T11:36:36.056 INFO:teuthology.orchestra.run.smithi152.stdout:(63/174): python3-attrs-20.3.0-7.el9.noarch.rpm 468 kB/s | 78 kB 00:00 2024-03-16T11:36:36.148 INFO:teuthology.orchestra.run.smithi152.stdout:(64/174): python3-babel-2.9.1-2.el9.noarch.rpm 27 MB/s | 5.8 MB 00:00 2024-03-16T11:36:36.173 INFO:teuthology.orchestra.run.smithi152.stdout:(65/174): python3-jsonpatch-1.21-16.el9.noarch. 175 kB/s | 24 kB 00:00 2024-03-16T11:36:36.207 INFO:teuthology.orchestra.run.smithi152.stdout:(66/174): python3-jsonpointer-2.0-4.el9.noarch. 122 kB/s | 18 kB 00:00 2024-03-16T11:36:36.232 INFO:teuthology.orchestra.run.smithi152.stdout:(67/174): python3-mako-1.1.4-6.el9.noarch.rpm 1.7 MB/s | 144 kB 00:00 2024-03-16T11:36:36.257 INFO:teuthology.orchestra.run.smithi152.stdout:(68/174): python3-markupsafe-1.1.1-12.el9.x86_6 631 kB/s | 32 kB 00:00 2024-03-16T11:36:36.333 INFO:teuthology.orchestra.run.smithi152.stdout:(69/174): python3-numpy-1.20.1-5.el9.x86_64.rpm 38 MB/s | 4.8 MB 00:00 2024-03-16T11:36:36.358 INFO:teuthology.orchestra.run.smithi152.stdout:(70/174): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.7 MB/s | 217 kB 00:00 2024-03-16T11:36:36.383 INFO:teuthology.orchestra.run.smithi152.stdout:(71/174): python3-oauthlib-3.1.1-5.el9.noarch.r 1.4 MB/s | 174 kB 00:00 2024-03-16T11:36:36.408 INFO:teuthology.orchestra.run.smithi152.stdout:(72/174): python3-packaging-20.9-5.el9.noarch.r 921 kB/s | 69 kB 00:00 2024-03-16T11:36:36.433 INFO:teuthology.orchestra.run.smithi152.stdout:(73/174): python3-prettytable-0.7.2-27.el9.noar 823 kB/s | 41 kB 00:00 2024-03-16T11:36:36.459 INFO:teuthology.orchestra.run.smithi152.stdout:(74/174): python3-protobuf-3.14.0-13.el9.noarch 3.1 MB/s | 238 kB 00:00 2024-03-16T11:36:36.484 INFO:teuthology.orchestra.run.smithi152.stdout:(75/174): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 132 kB 00:00 2024-03-16T11:36:36.509 INFO:teuthology.orchestra.run.smithi152.stdout:(76/174): python3-pytz-2021.1-5.el9.noarch.rpm 952 kB/s | 47 kB 00:00 2024-03-16T11:36:36.535 INFO:teuthology.orchestra.run.smithi152.stdout:(77/174): python3-pyasn1-modules-0.4.8-6.el9.no 2.7 MB/s | 211 kB 00:00 2024-03-16T11:36:36.559 INFO:teuthology.orchestra.run.smithi152.stdout:(78/174): python3-requests-oauthlib-1.3.0-12.el 568 kB/s | 43 kB 00:00 2024-03-16T11:36:36.719 INFO:teuthology.orchestra.run.smithi152.stdout:(79/174): python3-toml-0.10.2-6.el9.noarch.rpm 247 kB/s | 39 kB 00:00 2024-03-16T11:36:36.744 INFO:teuthology.orchestra.run.smithi152.stdout:(80/174): python3-wcwidth-0.2.5-8.el9.noarch.rp 226 kB/s | 41 kB 00:00 2024-03-16T11:36:36.902 INFO:teuthology.orchestra.run.smithi152.stdout:(81/174): python3-scipy-1.6.2-8.el9.x86_64.rpm 40 MB/s | 16 MB 00:00 2024-03-16T11:36:36.936 INFO:teuthology.orchestra.run.smithi152.stdout:(82/174): socat-1.7.4.1-5.el9.x86_64.rpm 1.3 MB/s | 300 kB 00:00 2024-03-16T11:36:36.961 INFO:teuthology.orchestra.run.smithi152.stdout:(83/174): uuid-1.6.2-55.el9.x86_64.rpm 260 kB/s | 56 kB 00:00 2024-03-16T11:36:36.986 INFO:teuthology.orchestra.run.smithi152.stdout:(84/174): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 63 kB 00:00 2024-03-16T11:36:37.011 INFO:teuthology.orchestra.run.smithi152.stdout:(85/174): zlib-devel-1.2.11-40.el9.x86_64.rpm 589 kB/s | 44 kB 00:00 2024-03-16T11:36:37.037 INFO:teuthology.orchestra.run.smithi152.stdout:(86/174): fuse-2.9.9-15.el9.x86_64.rpm 1.0 MB/s | 79 kB 00:00 2024-03-16T11:36:37.062 INFO:teuthology.orchestra.run.smithi152.stdout:(87/174): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.7 MB/s | 241 kB 00:00 2024-03-16T11:36:37.096 INFO:teuthology.orchestra.run.smithi152.stdout:(88/174): python3-cryptography-36.0.1-4.el9.x86 14 MB/s | 1.1 MB 00:00 2024-03-16T11:36:37.121 INFO:teuthology.orchestra.run.smithi152.stdout:(89/174): python3-ply-3.11-14.el9.noarch.rpm 1.2 MB/s | 103 kB 00:00 2024-03-16T11:36:37.146 INFO:teuthology.orchestra.run.smithi152.stdout:(90/174): python3-pycparser-2.20-6.el9.noarch.r 2.4 MB/s | 124 kB 00:00 2024-03-16T11:36:37.172 INFO:teuthology.orchestra.run.smithi152.stdout:(91/174): python3-urllib3-1.26.5-5.el9.noarch.r 2.4 MB/s | 187 kB 00:00 2024-03-16T11:36:37.280 INFO:teuthology.orchestra.run.smithi152.stdout:(92/174): jq-1.6-16.el9.x86_64.rpm 1.2 MB/s | 187 kB 00:00 2024-03-16T11:36:37.314 INFO:teuthology.orchestra.run.smithi152.stdout:(93/174): libquadmath-11.4.1-3.el9.x86_64.rpm 1.3 MB/s | 195 kB 00:00 2024-03-16T11:36:37.339 INFO:teuthology.orchestra.run.smithi152.stdout:(94/174): python3-requests-2.25.1-8.el9.noarch. 2.1 MB/s | 125 kB 00:00 2024-03-16T11:36:37.364 INFO:teuthology.orchestra.run.smithi152.stdout:(95/174): elfutils-libelf-devel-0.190-2.el9.x86 427 kB/s | 21 kB 00:00 2024-03-16T11:36:37.390 INFO:teuthology.orchestra.run.smithi152.stdout:(96/174): libuuid-devel-2.37.4-18.el9.x86_64.rp 503 kB/s | 25 kB 00:00 2024-03-16T11:36:37.432 INFO:teuthology.orchestra.run.smithi152.stdout:(97/174): libgfortran-11.4.1-3.el9.x86_64.rpm 3.0 MB/s | 803 kB 00:00 2024-03-16T11:36:37.465 INFO:teuthology.orchestra.run.smithi152.stdout:(98/174): python3-devel-3.9.17-2.el9.x86_64.rpm 3.2 MB/s | 245 kB 00:00 2024-03-16T11:36:37.491 INFO:teuthology.orchestra.run.smithi152.stdout:(99/174): python3-jinja2-2.11.3-5.el9.noarch.rp 4.1 MB/s | 248 kB 00:00 2024-03-16T11:36:37.516 INFO:teuthology.orchestra.run.smithi152.stdout:(100/174): qatlib-23.11.0-1.el9.x86_64.rpm 4.2 MB/s | 216 kB 00:00 2024-03-16T11:36:37.541 INFO:teuthology.orchestra.run.smithi152.stdout:(101/174): qatlib-service-23.11.0-1.el9.x86_64. 691 kB/s | 35 kB 00:00 2024-03-16T11:36:37.567 INFO:teuthology.orchestra.run.smithi152.stdout:(102/174): qatzip-libs-1.1.2-2.el9.x86_64.rpm 889 kB/s | 44 kB 00:00 2024-03-16T11:36:37.592 INFO:teuthology.orchestra.run.smithi152.stdout:(103/174): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 1.8 MB/s | 94 kB 00:00 2024-03-16T11:36:37.676 INFO:teuthology.orchestra.run.smithi152.stdout:(104/174): openssl-devel-3.0.7-27.el9.x86_64.rp 13 MB/s | 4.1 MB 00:00 2024-03-16T11:36:37.701 INFO:teuthology.orchestra.run.smithi152.stdout:(105/174): lua-devel-5.4.4-4.el9.x86_64.rpm 166 kB/s | 22 kB 00:00 2024-03-16T11:36:37.801 INFO:teuthology.orchestra.run.smithi152.stdout:(106/174): protobuf-compiler-3.14.0-13.el9.x86_ 4.0 MB/s | 863 kB 00:00 2024-03-16T11:36:37.918 INFO:teuthology.orchestra.run.smithi152.stdout:(107/174): grpc-data-1.46.7-10.el9.noarch.rpm 167 kB/s | 19 kB 00:00 2024-03-16T11:36:37.944 INFO:teuthology.orchestra.run.smithi152.stdout:(108/174): gperftools-libs-2.9.1-2.el9.x86_64.r 1.2 MB/s | 309 kB 00:00 2024-03-16T11:36:37.985 INFO:teuthology.orchestra.run.smithi152.stdout:(109/174): abseil-cpp-20211102.0-3.el9.x86_64.r 1.7 MB/s | 548 kB 00:00 2024-03-16T11:36:38.011 INFO:teuthology.orchestra.run.smithi152.stdout:(110/174): libarrow-doc-9.0.0-10.el9.noarch.rpm 388 kB/s | 26 kB 00:00 2024-03-16T11:36:38.039 INFO:teuthology.orchestra.run.smithi152.stdout:(111/174): liboath-2.6.7-2.el9.x86_64.rpm 906 kB/s | 49 kB 00:00 2024-03-16T11:36:38.073 INFO:teuthology.orchestra.run.smithi152.stdout:(112/174): libunwind-1.6.2-1.el9.x86_64.rpm 1.1 MB/s | 67 kB 00:00 2024-03-16T11:36:38.098 INFO:teuthology.orchestra.run.smithi152.stdout:(113/174): luarocks-3.9.2-1.el9.noarch.rpm 2.5 MB/s | 151 kB 00:00 2024-03-16T11:36:38.182 INFO:teuthology.orchestra.run.smithi152.stdout:(114/174): python3-asyncssh-2.13.2-4.el9.noarch 6.4 MB/s | 548 kB 00:00 2024-03-16T11:36:38.224 INFO:teuthology.orchestra.run.smithi152.stdout:(115/174): parquet-libs-9.0.0-10.el9.x86_64.rpm 5.4 MB/s | 838 kB 00:00 2024-03-16T11:36:38.249 INFO:teuthology.orchestra.run.smithi152.stdout:(116/174): python3-bcrypt-3.2.2-1.el9.x86_64.rp 650 kB/s | 43 kB 00:00 2024-03-16T11:36:38.274 INFO:teuthology.orchestra.run.smithi152.stdout:(117/174): python3-cachetools-4.2.4-1.el9.noarc 643 kB/s | 32 kB 00:00 2024-03-16T11:36:38.300 INFO:teuthology.orchestra.run.smithi152.stdout:(118/174): python3-certifi-2021.10.8-2.el9.noar 298 kB/s | 15 kB 00:00 2024-03-16T11:36:38.367 INFO:teuthology.orchestra.run.smithi152.stdout:(119/174): libarrow-9.0.0-10.el9.x86_64.rpm 9.9 MB/s | 4.4 MB 00:00 2024-03-16T11:36:38.484 INFO:teuthology.orchestra.run.smithi152.stdout:(120/174): python3-cheroot-8.6.0-4.el9.noarch.r 822 kB/s | 172 kB 00:00 2024-03-16T11:36:38.509 INFO:teuthology.orchestra.run.smithi152.stdout:(121/174): python3-cherrypy-18.6.1-2.el9.noarch 1.7 MB/s | 358 kB 00:00 2024-03-16T11:36:38.542 INFO:teuthology.orchestra.run.smithi152.stdout:(122/174): python3-cmd2-2.4.2-2.el9.noarch.rpm 1.7 MB/s | 303 kB 00:00 2024-03-16T11:36:38.568 INFO:teuthology.orchestra.run.smithi152.stdout:(123/174): python3-colorama-0.4.4-12.el9.noarch 420 kB/s | 35 kB 00:00 2024-03-16T11:36:38.601 INFO:teuthology.orchestra.run.smithi152.stdout:(124/174): python3-google-auth-2.28.0-1.el9.noa 2.1 MB/s | 203 kB 00:00 2024-03-16T11:36:38.627 INFO:teuthology.orchestra.run.smithi152.stdout:(125/174): python3-grpcio-tools-1.46.7-10.el9.x 2.4 MB/s | 144 kB 00:00 2024-03-16T11:36:38.652 INFO:teuthology.orchestra.run.smithi152.stdout:(126/174): python3-jaraco-8.2.1-3.el9.noarch.rp 213 kB/s | 11 kB 00:00 2024-03-16T11:36:38.703 INFO:teuthology.orchestra.run.smithi152.stdout:(127/174): python3-grpcio-1.46.7-10.el9.x86_64. 13 MB/s | 2.0 MB 00:00 2024-03-16T11:36:38.728 INFO:teuthology.orchestra.run.smithi152.stdout:(128/174): python3-jaraco-classes-3.2.1-5.el9.n 176 kB/s | 18 kB 00:00 2024-03-16T11:36:38.753 INFO:teuthology.orchestra.run.smithi152.stdout:(129/174): python3-jaraco-collections-3.0.0-8.e 231 kB/s | 23 kB 00:00 2024-03-16T11:36:38.778 INFO:teuthology.orchestra.run.smithi152.stdout:(130/174): python3-jaraco-functools-3.5.0-2.el9 258 kB/s | 19 kB 00:00 2024-03-16T11:36:38.803 INFO:teuthology.orchestra.run.smithi152.stdout:(131/174): python3-jaraco-text-3.2.0-6.el9.noar 395 kB/s | 20 kB 00:00 2024-03-16T11:36:38.845 INFO:teuthology.orchestra.run.smithi152.stdout:(132/174): python3-kubernetes-26.1.0-2.el9.noar 11 MB/s | 1.0 MB 00:00 2024-03-16T11:36:38.870 INFO:teuthology.orchestra.run.smithi152.stdout:(133/174): python3-logutils-0.3.5-21.el9.noarch 502 kB/s | 46 kB 00:00 2024-03-16T11:36:38.896 INFO:teuthology.orchestra.run.smithi152.stdout:(134/174): python3-more-itertools-8.12.0-2.el9. 855 kB/s | 79 kB 00:00 2024-03-16T11:36:38.921 INFO:teuthology.orchestra.run.smithi152.stdout:(135/174): python3-natsort-7.1.1-5.el9.noarch.r 766 kB/s | 58 kB 00:00 2024-03-16T11:36:38.946 INFO:teuthology.orchestra.run.smithi152.stdout:(136/174): python3-pecan-1.4.2-3.el9.noarch.rpm 3.5 MB/s | 272 kB 00:00 2024-03-16T11:36:38.972 INFO:teuthology.orchestra.run.smithi152.stdout:(137/174): python3-portend-3.1.0-2.el9.noarch.r 217 kB/s | 16 kB 00:00 2024-03-16T11:36:38.997 INFO:teuthology.orchestra.run.smithi152.stdout:(138/174): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.2 MB/s | 90 kB 00:00 2024-03-16T11:36:39.022 INFO:teuthology.orchestra.run.smithi152.stdout:(139/174): python3-pyperclip-1.8.2-2.el9.noarch 373 kB/s | 28 kB 00:00 2024-03-16T11:36:39.047 INFO:teuthology.orchestra.run.smithi152.stdout:(140/174): python3-repoze-lru-0.7-16.el9.noarch 408 kB/s | 31 kB 00:00 2024-03-16T11:36:39.073 INFO:teuthology.orchestra.run.smithi152.stdout:(141/174): python3-routes-2.5.1-5.el9.noarch.rp 2.4 MB/s | 188 kB 00:00 2024-03-16T11:36:39.098 INFO:teuthology.orchestra.run.smithi152.stdout:(142/174): python3-rsa-4.9-2.el9.noarch.rpm 783 kB/s | 59 kB 00:00 2024-03-16T11:36:39.123 INFO:teuthology.orchestra.run.smithi152.stdout:(143/174): python3-tempora-5.0.0-2.el9.noarch.r 474 kB/s | 36 kB 00:00 2024-03-16T11:36:39.148 INFO:teuthology.orchestra.run.smithi152.stdout:(144/174): python3-typing-extensions-4.4.0-2.el 677 kB/s | 51 kB 00:00 2024-03-16T11:36:39.173 INFO:teuthology.orchestra.run.smithi152.stdout:(145/174): python3-webob-1.8.7-6.el9.noarch.rpm 3.0 MB/s | 230 kB 00:00 2024-03-16T11:36:39.199 INFO:teuthology.orchestra.run.smithi152.stdout:(146/174): python3-websocket-client-1.2.3-2.el9 1.2 MB/s | 90 kB 00:00 2024-03-16T11:36:39.224 INFO:teuthology.orchestra.run.smithi152.stdout:(147/174): python3-werkzeug-2.0.3-3.el9.1.noarc 5.5 MB/s | 427 kB 00:00 2024-03-16T11:36:39.249 INFO:teuthology.orchestra.run.smithi152.stdout:(148/174): python3-zc-lockfile-2.0-10.el9.noarc 264 kB/s | 20 kB 00:00 2024-03-16T11:36:39.275 INFO:teuthology.orchestra.run.smithi152.stdout:(149/174): re2-20211101-3.el9.x86_64.rpm 2.5 MB/s | 192 kB 00:00 2024-03-16T11:36:39.325 INFO:teuthology.orchestra.run.smithi152.stdout:(150/174): thrift-0.15.0-2.el9.x86_64.rpm 16 MB/s | 1.6 MB 00:00 2024-03-16T11:36:39.392 INFO:teuthology.orchestra.run.smithi152.stdout:(151/174): xfsdump-3.1.12-4.el9.x86_64.rpm 7.8 MB/s | 321 kB 00:00 2024-03-16T11:36:39.559 INFO:teuthology.orchestra.run.smithi152.stdout:(152/174): NetworkManager-1.46.0-1.el9.x86_64.r 14 MB/s | 2.3 MB 00:00 2024-03-16T11:36:39.634 INFO:teuthology.orchestra.run.smithi152.stdout:(153/174): librados2-19.0.0-2162.g3d78f5ba.el9. 8.8 MB/s | 3.4 MB 00:00 2024-03-16T11:36:39.676 INFO:teuthology.orchestra.run.smithi152.stdout:(154/174): NetworkManager-libnm-1.46.0-1.el9.x8 16 MB/s | 1.8 MB 00:00 2024-03-16T11:36:39.693 INFO:teuthology.orchestra.run.smithi152.stdout:(155/174): NetworkManager-team-1.46.0-1.el9.x86 724 kB/s | 42 kB 00:00 2024-03-16T11:36:39.727 INFO:teuthology.orchestra.run.smithi152.stdout:(156/174): NetworkManager-tui-1.46.0-1.el9.x86_ 4.8 MB/s | 248 kB 00:00 2024-03-16T11:36:39.752 INFO:teuthology.orchestra.run.smithi152.stdout:(157/174): acl-2.3.1-4.el9.x86_64.rpm 1.2 MB/s | 71 kB 00:00 2024-03-16T11:36:39.777 INFO:teuthology.orchestra.run.smithi152.stdout:(158/174): elfutils-debuginfod-client-0.190-2.e 727 kB/s | 36 kB 00:00 2024-03-16T11:36:39.811 INFO:teuthology.orchestra.run.smithi152.stdout:(159/174): elfutils-libelf-0.190-2.el9.x86_64.r 3.2 MB/s | 192 kB 00:00 2024-03-16T11:36:39.836 INFO:teuthology.orchestra.run.smithi152.stdout:(160/174): elfutils-libs-0.190-2.el9.x86_64.rpm 4.3 MB/s | 256 kB 00:00 2024-03-16T11:36:39.861 INFO:teuthology.orchestra.run.smithi152.stdout:(161/174): libacl-2.3.1-4.el9.x86_64.rpm 459 kB/s | 23 kB 00:00 2024-03-16T11:36:39.887 INFO:teuthology.orchestra.run.smithi152.stdout:(162/174): libblkid-2.37.4-18.el9.x86_64.rpm 2.1 MB/s | 108 kB 00:00 2024-03-16T11:36:39.912 INFO:teuthology.orchestra.run.smithi152.stdout:(163/174): libfdisk-2.37.4-18.el9.x86_64.rpm 3.0 MB/s | 155 kB 00:00 2024-03-16T11:36:39.945 INFO:teuthology.orchestra.run.smithi152.stdout:(164/174): libmount-2.37.4-18.el9.x86_64.rpm 2.3 MB/s | 136 kB 00:00 2024-03-16T11:36:39.979 INFO:teuthology.orchestra.run.smithi152.stdout:(165/174): libsmartcols-2.37.4-18.el9.x86_64.rp 958 kB/s | 64 kB 00:00 2024-03-16T11:36:40.029 INFO:teuthology.orchestra.run.smithi152.stdout:(166/174): librbd1-19.0.0-2162.g3d78f5ba.el9.x8 4.2 MB/s | 3.1 MB 00:00 2024-03-16T11:36:40.055 INFO:teuthology.orchestra.run.smithi152.stdout:(167/174): libuuid-2.37.4-18.el9.x86_64.rpm 264 kB/s | 29 kB 00:00 2024-03-16T11:36:40.138 INFO:teuthology.orchestra.run.smithi152.stdout:(168/174): openssl-libs-3.0.7-27.el9.x86_64.rpm 20 MB/s | 2.2 MB 00:00 2024-03-16T11:36:40.197 INFO:teuthology.orchestra.run.smithi152.stdout:(169/174): util-linux-core-2.37.4-18.el9.x86_64 7.8 MB/s | 465 kB 00:00 2024-03-16T11:36:40.247 INFO:teuthology.orchestra.run.smithi152.stdout:(170/174): util-linux-2.37.4-18.el9.x86_64.rpm 12 MB/s | 2.3 MB 00:00 2024-03-16T11:36:40.272 INFO:teuthology.orchestra.run.smithi152.stdout:(171/174): util-linux-user-2.37.4-18.el9.x86_64 414 kB/s | 31 kB 00:00 2024-03-16T11:36:40.314 INFO:teuthology.orchestra.run.smithi152.stdout:(172/174): openssl-3.0.7-27.el9.x86_64.rpm 3.6 MB/s | 1.2 MB 00:00 2024-03-16T11:36:40.339 INFO:teuthology.orchestra.run.smithi152.stdout:(173/174): libacl-devel-2.3.1-4.el9.x86_64.rpm 1.1 MB/s | 74 kB 00:00 2024-03-16T11:36:40.373 INFO:teuthology.orchestra.run.smithi152.stdout:(174/174): xfsprogs-6.3.0-1.el9.x86_64.rpm 8.6 MB/s | 1.1 MB 00:00 2024-03-16T11:36:40.377 INFO:teuthology.orchestra.run.smithi152.stdout:-------------------------------------------------------------------------------- 2024-03-16T11:36:40.377 INFO:teuthology.orchestra.run.smithi152.stdout:Total 20 MB/s | 224 MB 00:11 2024-03-16T11:36:40.862 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction check 2024-03-16T11:36:41.136 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction check succeeded. 2024-03-16T11:36:41.137 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction test 2024-03-16T11:36:41.505 INFO:teuthology.orchestra.run.smithi154.stdout: Preparing : 1/1 2024-03-16T11:36:41.619 INFO:teuthology.orchestra.run.smithi154.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/198 2024-03-16T11:36:41.644 INFO:teuthology.orchestra.run.smithi154.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/198 2024-03-16T11:36:41.675 INFO:teuthology.orchestra.run.smithi154.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-16T11:36:41.684 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-16T11:36:41.724 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 4/198 2024-03-16T11:36:41.777 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/198 2024-03-16T11:36:41.812 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/198 2024-03-16T11:36:41.837 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 7/198 2024-03-16T11:36:41.962 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 8/198 2024-03-16T11:36:42.391 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 9/198 2024-03-16T11:36:42.455 INFO:teuthology.orchestra.run.smithi154.stdout: Upgrading : librados2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 10/198 2024-03-16T11:36:42.560 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: librados2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 10/198 2024-03-16T11:36:42.589 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : libcephfs2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 11/198 2024-03-16T11:36:42.681 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: libcephfs2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 11/198 2024-03-16T11:36:42.710 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-rados-2:19.0.0-2162.g3d78f5ba.el9.x86_64 12/198 2024-03-16T11:36:42.745 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : postgresql-private-libs-13.11-1.el9.x86_64 13/198 2024-03-16T11:36:42.863 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction test succeeded. 2024-03-16T11:36:42.864 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction 2024-03-16T11:36:42.925 INFO:teuthology.orchestra.run.smithi154.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 14/198 2024-03-16T11:36:42.974 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : postgresql-13.11-1.el9.x86_64 15/198 2024-03-16T11:36:43.002 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : libcephsqlite-2:19.0.0-2162.g3d78f5ba.el9.x86_64 16/198 2024-03-16T11:36:43.122 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2162.g3d78f5ba.el9.x86_64 16/198 2024-03-16T11:36:43.145 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : libradosstriper1-2:19.0.0-2162.g3d78f5ba.el9.x86 17/198 2024-03-16T11:36:43.210 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2162.g3d78f5ba.el9.x86 17/198 2024-03-16T11:36:43.249 INFO:teuthology.orchestra.run.smithi154.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/198 2024-03-16T11:36:43.272 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-16T11:36:43.287 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-16T11:36:43.480 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-16T11:36:43.511 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-16T11:36:43.542 INFO:teuthology.orchestra.run.smithi154.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/198 2024-03-16T11:36:43.639 INFO:teuthology.orchestra.run.smithi154.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/198 2024-03-16T11:36:43.736 INFO:teuthology.orchestra.run.smithi154.stdout: Upgrading : NetworkManager-libnm-1:1.46.0-1.el9.x86_64 23/198 2024-03-16T11:36:43.797 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: NetworkManager-libnm-1:1.46.0-1.el9.x86_64 23/198 2024-03-16T11:36:43.869 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: NetworkManager-1:1.46.0-1.el9.x86_64 24/198 2024-03-16T11:36:43.940 INFO:teuthology.orchestra.run.smithi154.stdout: Upgrading : NetworkManager-1:1.46.0-1.el9.x86_64 24/198 2024-03-16T11:36:44.155 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: NetworkManager-1:1.46.0-1.el9.x86_64 24/198 2024-03-16T11:36:44.270 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/198 2024-03-16T11:36:44.366 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/198 2024-03-16T11:36:44.395 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/198 2024-03-16T11:36:44.421 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/198 2024-03-16T11:36:44.465 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/198 2024-03-16T11:36:44.539 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/198 2024-03-16T11:36:44.601 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 31/198 2024-03-16T11:36:44.640 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/198 2024-03-16T11:36:44.704 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/198 2024-03-16T11:36:44.739 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/198 2024-03-16T11:36:44.782 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/198 2024-03-16T11:36:44.815 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/198 2024-03-16T11:36:44.845 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/198 2024-03-16T11:36:44.950 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/198 2024-03-16T11:36:44.981 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-ceph-common-2:19.0.0-2162.g3d78f5ba.el9. 39/198 2024-03-16T11:36:45.024 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-ceph-argparse-2:19.0.0-2162.g3d78f5ba.el 40/198 2024-03-16T11:36:45.053 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-cephfs-2:19.0.0-2162.g3d78f5ba.el9.x86_6 41/198 2024-03-16T11:36:45.121 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/198 2024-03-16T11:36:45.167 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/198 2024-03-16T11:36:45.240 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/198 2024-03-16T11:36:45.375 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/198 2024-03-16T11:36:45.428 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/198 2024-03-16T11:36:45.964 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/198 2024-03-16T11:36:46.048 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/198 2024-03-16T11:36:46.123 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/198 2024-03-16T11:36:46.159 INFO:teuthology.orchestra.run.smithi154.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-16T11:36:46.167 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-16T11:36:46.199 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 51/198 2024-03-16T11:36:46.248 INFO:teuthology.orchestra.run.smithi154.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 52/198 2024-03-16T11:36:46.286 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : librados-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_6 53/198 2024-03-16T11:36:46.316 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 54/198 2024-03-16T11:36:46.552 INFO:teuthology.orchestra.run.smithi154.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 55/198 2024-03-16T11:36:46.697 INFO:teuthology.orchestra.run.smithi154.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 56/198 2024-03-16T11:36:46.760 INFO:teuthology.orchestra.run.smithi154.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 57/198 2024-03-16T11:36:46.803 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 58/198 2024-03-16T11:36:46.934 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 59/198 2024-03-16T11:36:46.982 INFO:teuthology.orchestra.run.smithi154.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 60/198 2024-03-16T11:36:47.037 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 61/198 2024-03-16T11:36:47.110 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 62/198 2024-03-16T11:36:47.149 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 63/198 2024-03-16T11:36:47.185 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 64/198 2024-03-16T11:36:47.232 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 65/198 2024-03-16T11:36:47.268 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 66/198 2024-03-16T11:36:47.411 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 67/198 2024-03-16T11:36:47.453 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 68/198 2024-03-16T11:36:47.587 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 69/198 2024-03-16T11:36:47.633 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 70/198 2024-03-16T11:36:47.664 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 71/198 2024-03-16T11:36:47.706 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 72/198 2024-03-16T11:36:47.799 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 73/198 2024-03-16T11:36:47.839 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-google-auth-1:2.28.0-1.el9.noarch 74/198 2024-03-16T11:36:47.879 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 75/198 2024-03-16T11:36:47.904 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 76/198 2024-03-16T11:36:47.937 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 77/198 2024-03-16T11:36:48.017 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 78/198 2024-03-16T11:36:48.140 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 79/198 2024-03-16T11:36:48.237 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 80/198 2024-03-16T11:36:48.279 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 81/198 2024-03-16T11:36:48.327 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-ply-3.11-14.el9.noarch 82/198 2024-03-16T11:36:48.382 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 83/198 2024-03-16T11:36:48.569 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 84/198 2024-03-16T11:36:48.624 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 85/198 2024-03-16T11:36:48.771 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/198 2024-03-16T11:36:48.814 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-asyncssh-2.13.2-4.el9.noarch 87/198 2024-03-16T11:36:48.854 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 88/198 2024-03-16T11:36:48.886 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : fuse-2.9.9-15.el9.x86_64 89/198 2024-03-16T11:36:48.914 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 90/198 2024-03-16T11:36:48.948 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 91/198 2024-03-16T11:36:48.987 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : uuid-1.6.2-55.el9.x86_64 92/198 2024-03-16T11:36:49.027 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 93/198 2024-03-16T11:36:49.057 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 94/198 2024-03-16T11:36:49.134 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 95/198 2024-03-16T11:36:49.168 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 96/198 2024-03-16T11:36:49.200 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 97/198 2024-03-16T11:36:49.242 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 98/198 2024-03-16T11:36:49.277 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 99/198 2024-03-16T11:36:49.365 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 100/198 2024-03-16T11:36:49.640 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 101/198 2024-03-16T11:36:49.800 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 102/198 2024-03-16T11:36:49.805 INFO:teuthology.orchestra.run.smithi152.stdout: Preparing : 1/1 2024-03-16T11:36:49.831 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 103/198 2024-03-16T11:36:49.878 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 104/198 2024-03-16T11:36:49.918 INFO:teuthology.orchestra.run.smithi152.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/198 2024-03-16T11:36:49.952 INFO:teuthology.orchestra.run.smithi152.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/198 2024-03-16T11:36:49.975 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-packaging-20.9-5.el9.noarch 105/198 2024-03-16T11:36:49.983 INFO:teuthology.orchestra.run.smithi152.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-16T11:36:49.992 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-16T11:36:50.032 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 106/198 2024-03-16T11:36:50.040 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 4/198 2024-03-16T11:36:50.093 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/198 2024-03-16T11:36:50.128 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/198 2024-03-16T11:36:50.162 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 7/198 2024-03-16T11:36:50.204 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 8/198 2024-03-16T11:36:50.525 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 9/198 2024-03-16T11:36:50.588 INFO:teuthology.orchestra.run.smithi152.stdout: Upgrading : librados2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 10/198 2024-03-16T11:36:50.694 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: librados2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 10/198 2024-03-16T11:36:50.731 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : libcephfs2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 11/198 2024-03-16T11:36:50.772 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 107/198 2024-03-16T11:36:50.821 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 108/198 2024-03-16T11:36:50.822 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: libcephfs2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 11/198 2024-03-16T11:36:50.855 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 109/198 2024-03-16T11:36:50.859 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-rados-2:19.0.0-2162.g3d78f5ba.el9.x86_64 12/198 2024-03-16T11:36:50.897 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : postgresql-private-libs-13.11-1.el9.x86_64 13/198 2024-03-16T11:36:50.904 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 110/198 2024-03-16T11:36:50.992 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 111/198 2024-03-16T11:36:51.031 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 112/198 2024-03-16T11:36:51.059 INFO:teuthology.orchestra.run.smithi152.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 14/198 2024-03-16T11:36:51.107 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : postgresql-13.11-1.el9.x86_64 15/198 2024-03-16T11:36:51.136 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : libcephsqlite-2:19.0.0-2162.g3d78f5ba.el9.x86_64 16/198 2024-03-16T11:36:51.209 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : openblas-0.3.21-2.el9.x86_64 113/198 2024-03-16T11:36:51.222 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2162.g3d78f5ba.el9.x86_64 16/198 2024-03-16T11:36:51.237 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 114/198 2024-03-16T11:36:51.245 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : libradosstriper1-2:19.0.0-2162.g3d78f5ba.el9.x86 17/198 2024-03-16T11:36:51.309 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2162.g3d78f5ba.el9.x86 17/198 2024-03-16T11:36:51.314 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 115/198 2024-03-16T11:36:51.348 INFO:teuthology.orchestra.run.smithi152.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/198 2024-03-16T11:36:51.371 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-16T11:36:51.389 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-16T11:36:51.580 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-16T11:36:51.610 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-16T11:36:51.634 INFO:teuthology.orchestra.run.smithi152.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/198 2024-03-16T11:36:51.734 INFO:teuthology.orchestra.run.smithi152.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/198 2024-03-16T11:36:51.819 INFO:teuthology.orchestra.run.smithi152.stdout: Upgrading : NetworkManager-libnm-1:1.46.0-1.el9.x86_64 23/198 2024-03-16T11:36:51.880 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: NetworkManager-libnm-1:1.46.0-1.el9.x86_64 23/198 2024-03-16T11:36:51.944 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: NetworkManager-1:1.46.0-1.el9.x86_64 24/198 2024-03-16T11:36:51.970 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 116/198 2024-03-16T11:36:52.023 INFO:teuthology.orchestra.run.smithi152.stdout: Upgrading : NetworkManager-1:1.46.0-1.el9.x86_64 24/198 2024-03-16T11:36:52.076 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 117/198 2024-03-16T11:36:52.230 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: NetworkManager-1:1.46.0-1.el9.x86_64 24/198 2024-03-16T11:36:52.347 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/198 2024-03-16T11:36:52.448 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/198 2024-03-16T11:36:52.494 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/198 2024-03-16T11:36:52.520 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/198 2024-03-16T11:36:52.557 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/198 2024-03-16T11:36:52.631 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/198 2024-03-16T11:36:52.693 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 31/198 2024-03-16T11:36:52.731 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/198 2024-03-16T11:36:52.796 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/198 2024-03-16T11:36:52.830 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/198 2024-03-16T11:36:52.873 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/198 2024-03-16T11:36:52.905 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/198 2024-03-16T11:36:52.933 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/198 2024-03-16T11:36:53.002 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/198 2024-03-16T11:36:53.047 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-ceph-common-2:19.0.0-2162.g3d78f5ba.el9. 39/198 2024-03-16T11:36:53.099 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-ceph-argparse-2:19.0.0-2162.g3d78f5ba.el 40/198 2024-03-16T11:36:53.127 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-cephfs-2:19.0.0-2162.g3d78f5ba.el9.x86_6 41/198 2024-03-16T11:36:53.196 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/198 2024-03-16T11:36:53.242 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/198 2024-03-16T11:36:53.287 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 118/198 2024-03-16T11:36:53.315 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/198 2024-03-16T11:36:53.379 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 119/198 2024-03-16T11:36:53.410 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 120/198 2024-03-16T11:36:53.441 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : jq-1.6-16.el9.x86_64 121/198 2024-03-16T11:36:53.448 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/198 2024-03-16T11:36:53.468 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 122/198 2024-03-16T11:36:53.502 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/198 2024-03-16T11:36:53.782 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 123/198 2024-03-16T11:36:53.815 INFO:teuthology.orchestra.run.smithi154.stdout: Upgrading : librbd1-2:19.0.0-2162.g3d78f5ba.el9.x86_64 124/198 2024-03-16T11:36:53.905 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: librbd1-2:19.0.0-2162.g3d78f5ba.el9.x86_64 124/198 2024-03-16T11:36:53.937 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-rbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 125/198 2024-03-16T11:36:53.973 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 126/198 2024-03-16T11:36:54.033 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/198 2024-03-16T11:36:54.122 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/198 2024-03-16T11:36:54.208 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/198 2024-03-16T11:36:54.250 INFO:teuthology.orchestra.run.smithi152.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-16T11:36:54.258 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-16T11:36:54.299 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 51/198 2024-03-16T11:36:54.340 INFO:teuthology.orchestra.run.smithi152.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 52/198 2024-03-16T11:36:54.378 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : librados-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_6 53/198 2024-03-16T11:36:54.407 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 54/198 2024-03-16T11:36:54.444 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 127/198 2024-03-16T11:36:54.476 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : librgw2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 128/198 2024-03-16T11:36:54.548 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: librgw2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 128/198 2024-03-16T11:36:54.570 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-rgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 129/198 2024-03-16T11:36:54.668 INFO:teuthology.orchestra.run.smithi152.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 55/198 2024-03-16T11:36:54.841 INFO:teuthology.orchestra.run.smithi152.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 56/198 2024-03-16T11:36:54.926 INFO:teuthology.orchestra.run.smithi152.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 57/198 2024-03-16T11:36:54.978 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 58/198 2024-03-16T11:36:55.123 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 59/198 2024-03-16T11:36:55.173 INFO:teuthology.orchestra.run.smithi152.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 60/198 2024-03-16T11:36:55.227 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 61/198 2024-03-16T11:36:55.299 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 62/198 2024-03-16T11:36:55.532 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 63/198 2024-03-16T11:36:55.574 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 64/198 2024-03-16T11:36:55.621 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 65/198 2024-03-16T11:36:55.656 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 66/198 2024-03-16T11:36:55.699 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 67/198 2024-03-16T11:36:55.741 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 68/198 2024-03-16T11:36:55.875 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 69/198 2024-03-16T11:36:55.930 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 70/198 2024-03-16T11:36:55.960 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 71/198 2024-03-16T11:36:56.001 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 72/198 2024-03-16T11:36:56.096 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 73/198 2024-03-16T11:36:56.160 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-google-auth-1:2.28.0-1.el9.noarch 74/198 2024-03-16T11:36:56.200 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 75/198 2024-03-16T11:36:56.226 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 76/198 2024-03-16T11:36:56.250 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 77/198 2024-03-16T11:36:56.330 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 78/198 2024-03-16T11:36:56.459 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 79/198 2024-03-16T11:36:56.574 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 80/198 2024-03-16T11:36:56.616 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 81/198 2024-03-16T11:36:56.665 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-ply-3.11-14.el9.noarch 82/198 2024-03-16T11:36:56.720 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 83/198 2024-03-16T11:36:56.781 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 130/198 2024-03-16T11:36:56.817 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 130/198 2024-03-16T11:36:56.912 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 84/198 2024-03-16T11:36:56.962 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 85/198 2024-03-16T11:36:57.090 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/198 2024-03-16T11:36:57.126 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-asyncssh-2.13.2-4.el9.noarch 87/198 2024-03-16T11:36:57.158 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 88/198 2024-03-16T11:36:57.189 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : fuse-2.9.9-15.el9.x86_64 89/198 2024-03-16T11:36:57.217 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 90/198 2024-03-16T11:36:57.252 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 91/198 2024-03-16T11:36:57.291 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : uuid-1.6.2-55.el9.x86_64 92/198 2024-03-16T11:36:57.323 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 93/198 2024-03-16T11:36:57.353 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 130/198 2024-03-16T11:36:57.386 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-base-2:19.0.0-2162.g3d78f5ba.el9.x86_64 131/198 2024-03-16T11:36:57.451 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-base-2:19.0.0-2162.g3d78f5ba.el9.x86_64 131/198 2024-03-16T11:36:57.451 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-16T11:36:57.451 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-16T11:36:57.451 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-03-16T11:36:57.464 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 94/198 2024-03-16T11:36:57.479 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-selinux-2:19.0.0-2162.g3d78f5ba.el9.x86_64 132/198 2024-03-16T11:36:57.666 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 95/198 2024-03-16T11:36:57.888 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 96/198 2024-03-16T11:36:57.922 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 97/198 2024-03-16T11:36:57.963 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 98/198 2024-03-16T11:36:57.997 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 99/198 2024-03-16T11:36:58.088 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 100/198 2024-03-16T11:36:58.176 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 101/198 2024-03-16T11:36:58.219 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 102/198 2024-03-16T11:36:58.259 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 103/198 2024-03-16T11:36:58.306 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 104/198 2024-03-16T11:36:58.404 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-packaging-20.9-5.el9.noarch 105/198 2024-03-16T11:36:58.452 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 106/198 2024-03-16T11:36:59.192 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 107/198 2024-03-16T11:36:59.249 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 108/198 2024-03-16T11:36:59.291 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 109/198 2024-03-16T11:36:59.348 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 110/198 2024-03-16T11:36:59.436 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 111/198 2024-03-16T11:36:59.476 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 112/198 2024-03-16T11:36:59.655 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : openblas-0.3.21-2.el9.x86_64 113/198 2024-03-16T11:36:59.682 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 114/198 2024-03-16T11:36:59.759 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 115/198 2024-03-16T11:37:00.402 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 116/198 2024-03-16T11:37:00.504 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 117/198 2024-03-16T11:37:01.738 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 118/198 2024-03-16T11:37:01.831 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 119/198 2024-03-16T11:37:01.862 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 120/198 2024-03-16T11:37:01.893 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : jq-1.6-16.el9.x86_64 121/198 2024-03-16T11:37:01.920 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 122/198 2024-03-16T11:37:02.239 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 123/198 2024-03-16T11:37:02.267 INFO:teuthology.orchestra.run.smithi152.stdout: Upgrading : librbd1-2:19.0.0-2162.g3d78f5ba.el9.x86_64 124/198 2024-03-16T11:37:02.349 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: librbd1-2:19.0.0-2162.g3d78f5ba.el9.x86_64 124/198 2024-03-16T11:37:02.380 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-rbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 125/198 2024-03-16T11:37:02.416 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 126/198 2024-03-16T11:37:02.885 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 127/198 2024-03-16T11:37:02.920 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : librgw2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 128/198 2024-03-16T11:37:02.991 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: librgw2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 128/198 2024-03-16T11:37:03.022 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-rgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 129/198 2024-03-16T11:37:05.272 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 130/198 2024-03-16T11:37:05.326 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 130/198 2024-03-16T11:37:05.863 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 130/198 2024-03-16T11:37:05.904 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-base-2:19.0.0-2162.g3d78f5ba.el9.x86_64 131/198 2024-03-16T11:37:05.985 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-base-2:19.0.0-2162.g3d78f5ba.el9.x86_64 131/198 2024-03-16T11:37:05.985 INFO:teuthology.orchestra.run.smithi152.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-16T11:37:05.985 INFO:teuthology.orchestra.run.smithi152.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-16T11:37:05.985 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-16T11:37:06.013 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-selinux-2:19.0.0-2162.g3d78f5ba.el9.x86_64 132/198 2024-03-16T11:37:09.393 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2162.g3d78f5ba.el9.x86_64 132/198 2024-03-16T11:37:09.393 INFO:teuthology.orchestra.run.smithi154.stdout:skipping the directory /sys 2024-03-16T11:37:09.393 INFO:teuthology.orchestra.run.smithi154.stdout:skipping the directory /proc 2024-03-16T11:37:09.393 INFO:teuthology.orchestra.run.smithi154.stdout:skipping the directory /mnt 2024-03-16T11:37:09.394 INFO:teuthology.orchestra.run.smithi154.stdout:skipping the directory /var/tmp 2024-03-16T11:37:09.394 INFO:teuthology.orchestra.run.smithi154.stdout:skipping the directory /home 2024-03-16T11:37:09.394 INFO:teuthology.orchestra.run.smithi154.stdout:skipping the directory /root 2024-03-16T11:37:09.394 INFO:teuthology.orchestra.run.smithi154.stdout:skipping the directory /tmp 2024-03-16T11:37:09.394 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-03-16T11:37:09.558 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-volume-2:19.0.0-2162.g3d78f5ba.el9.noarch 133/198 2024-03-16T11:37:09.574 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-volume-2:19.0.0-2162.g3d78f5ba.el9.noarch 133/198 2024-03-16T11:37:09.574 INFO:teuthology.orchestra.run.smithi154.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-16T11:37:09.574 INFO:teuthology.orchestra.run.smithi154.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-16T11:37:09.574 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-03-16T11:37:11.044 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-osd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 134/198 2024-03-16T11:37:11.070 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-osd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 134/198 2024-03-16T11:37:11.070 INFO:teuthology.orchestra.run.smithi154.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-16T11:37:11.070 INFO:teuthology.orchestra.run.smithi154.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-16T11:37:11.070 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-16T11:37:11.070 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-16T11:37:11.070 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-03-16T11:37:11.295 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-mds-2:19.0.0-2162.g3d78f5ba.el9.x86_64 135/198 2024-03-16T11:37:11.317 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-mds-2:19.0.0-2162.g3d78f5ba.el9.x86_64 135/198 2024-03-16T11:37:11.317 INFO:teuthology.orchestra.run.smithi154.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-16T11:37:11.317 INFO:teuthology.orchestra.run.smithi154.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-16T11:37:11.317 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-16T11:37:11.317 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-16T11:37:11.318 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-03-16T11:37:11.762 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-mon-2:19.0.0-2162.g3d78f5ba.el9.x86_64 136/198 2024-03-16T11:37:11.784 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-mon-2:19.0.0-2162.g3d78f5ba.el9.x86_64 136/198 2024-03-16T11:37:11.785 INFO:teuthology.orchestra.run.smithi154.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-16T11:37:11.785 INFO:teuthology.orchestra.run.smithi154.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-16T11:37:11.785 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-16T11:37:11.785 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-16T11:37:11.785 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-03-16T11:37:11.889 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : automake-1.16.2-8.el9.noarch 137/198 2024-03-16T11:37:12.314 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: cephadm-2:19.0.0-2162.g3d78f5ba.el9.noarch 138/198 2024-03-16T11:37:12.346 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : cephadm-2:19.0.0-2162.g3d78f5ba.el9.noarch 138/198 2024-03-16T11:37:12.378 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2162.g3d78f5ba.e 139/198 2024-03-16T11:37:12.438 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2162.g3d78f5ba. 140/198 2024-03-16T11:37:12.460 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2162.g3d78f5ba.el9.noa 141/198 2024-03-16T11:37:12.694 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2162.g3d78f5ba.el9.noa 141/198 2024-03-16T11:37:12.720 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2162.g3d78f5ba.el9.n 142/198 2024-03-16T11:37:13.684 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2162.g3d78f5ba.el9.n 142/198 2024-03-16T11:37:13.712 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2162.g3d7 143/198 2024-03-16T11:37:13.810 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2162.g3d7 143/198 2024-03-16T11:37:13.975 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2162.g3d78f5ba.el 144/198 2024-03-16T11:37:14.005 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-mgr-2:19.0.0-2162.g3d78f5ba.el9.x86_64 145/198 2024-03-16T11:37:14.028 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2162.g3d78f5ba.el9.x86_64 145/198 2024-03-16T11:37:14.028 INFO:teuthology.orchestra.run.smithi154.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-16T11:37:14.029 INFO:teuthology.orchestra.run.smithi154.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-16T11:37:14.029 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-16T11:37:14.029 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-16T11:37:14.029 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-03-16T11:37:14.073 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-mgr-rook-2:19.0.0-2162.g3d78f5ba.el9.noarch 146/198 2024-03-16T11:37:14.084 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2162.g3d78f5ba.el9.noarch 146/198 2024-03-16T11:37:14.128 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-2:19.0.0-2162.g3d78f5ba.el9.x86_64 147/198 2024-03-16T11:37:15.003 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : libtool-2.4.6-45.el9.x86_64 148/198 2024-03-16T11:37:15.033 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-radosgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 149/198 2024-03-16T11:37:15.056 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 149/198 2024-03-16T11:37:15.056 INFO:teuthology.orchestra.run.smithi154.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-16T11:37:15.056 INFO:teuthology.orchestra.run.smithi154.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-16T11:37:15.056 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-16T11:37:15.056 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-16T11:37:15.056 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-03-16T11:37:15.101 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2162.g3d78f 150/198 2024-03-16T11:37:15.122 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2162.g3d78f 150/198 2024-03-16T11:37:15.122 INFO:teuthology.orchestra.run.smithi154.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-16T11:37:15.122 INFO:teuthology.orchestra.run.smithi154.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-16T11:37:15.122 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-03-16T11:37:15.168 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : cephfs-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 151/198 2024-03-16T11:37:15.190 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 151/198 2024-03-16T11:37:15.190 INFO:teuthology.orchestra.run.smithi154.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-16T11:37:15.190 INFO:teuthology.orchestra.run.smithi154.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-03-16T11:37:15.190 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-16T11:37:15.190 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-16T11:37:15.191 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-03-16T11:37:15.510 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : rbd-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 152/198 2024-03-16T11:37:15.532 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 152/198 2024-03-16T11:37:15.532 INFO:teuthology.orchestra.run.smithi154.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-16T11:37:15.532 INFO:teuthology.orchestra.run.smithi154.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-16T11:37:15.533 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-16T11:37:15.533 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-16T11:37:15.533 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-03-16T11:37:17.999 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2162.g3d78f5ba.el9.x86_64 132/198 2024-03-16T11:37:17.999 INFO:teuthology.orchestra.run.smithi152.stdout:skipping the directory /sys 2024-03-16T11:37:17.999 INFO:teuthology.orchestra.run.smithi152.stdout:skipping the directory /proc 2024-03-16T11:37:17.999 INFO:teuthology.orchestra.run.smithi152.stdout:skipping the directory /mnt 2024-03-16T11:37:18.000 INFO:teuthology.orchestra.run.smithi152.stdout:skipping the directory /var/tmp 2024-03-16T11:37:18.000 INFO:teuthology.orchestra.run.smithi152.stdout:skipping the directory /home 2024-03-16T11:37:18.000 INFO:teuthology.orchestra.run.smithi152.stdout:skipping the directory /root 2024-03-16T11:37:18.000 INFO:teuthology.orchestra.run.smithi152.stdout:skipping the directory /tmp 2024-03-16T11:37:18.000 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-16T11:37:18.168 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-volume-2:19.0.0-2162.g3d78f5ba.el9.noarch 133/198 2024-03-16T11:37:18.183 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-volume-2:19.0.0-2162.g3d78f5ba.el9.noarch 133/198 2024-03-16T11:37:18.183 INFO:teuthology.orchestra.run.smithi152.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-16T11:37:18.183 INFO:teuthology.orchestra.run.smithi152.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-16T11:37:18.183 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-16T11:37:19.712 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-osd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 134/198 2024-03-16T11:37:19.738 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-osd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 134/198 2024-03-16T11:37:19.738 INFO:teuthology.orchestra.run.smithi152.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-16T11:37:19.738 INFO:teuthology.orchestra.run.smithi152.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-16T11:37:19.738 INFO:teuthology.orchestra.run.smithi152.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-16T11:37:19.738 INFO:teuthology.orchestra.run.smithi152.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-16T11:37:19.738 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-16T11:37:19.963 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-mds-2:19.0.0-2162.g3d78f5ba.el9.x86_64 135/198 2024-03-16T11:37:19.966 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-test-2:19.0.0-2162.g3d78f5ba.el9.x86_64 153/198 2024-03-16T11:37:19.985 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-mds-2:19.0.0-2162.g3d78f5ba.el9.x86_64 135/198 2024-03-16T11:37:19.985 INFO:teuthology.orchestra.run.smithi152.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-16T11:37:19.985 INFO:teuthology.orchestra.run.smithi152.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-16T11:37:19.985 INFO:teuthology.orchestra.run.smithi152.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-16T11:37:19.985 INFO:teuthology.orchestra.run.smithi152.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-16T11:37:19.986 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-16T11:37:20.007 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : rbd-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 154/198 2024-03-16T11:37:20.037 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : rbd-nbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 155/198 2024-03-16T11:37:20.123 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 156/198 2024-03-16T11:37:20.303 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : cephfs-shell-2:19.0.0-2162.g3d78f5ba.el9.x86_64 157/198 2024-03-16T11:37:20.422 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-mon-2:19.0.0-2162.g3d78f5ba.el9.x86_64 136/198 2024-03-16T11:37:20.423 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : postgresql-contrib-13.11-1.el9.x86_64 158/198 2024-03-16T11:37:20.444 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-mon-2:19.0.0-2162.g3d78f5ba.el9.x86_64 136/198 2024-03-16T11:37:20.445 INFO:teuthology.orchestra.run.smithi152.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-16T11:37:20.445 INFO:teuthology.orchestra.run.smithi152.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-16T11:37:20.445 INFO:teuthology.orchestra.run.smithi152.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-16T11:37:20.445 INFO:teuthology.orchestra.run.smithi152.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-16T11:37:20.445 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-16T11:37:20.479 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 159/198 2024-03-16T11:37:20.518 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 160/198 2024-03-16T11:37:20.548 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : automake-1.16.2-8.el9.noarch 137/198 2024-03-16T11:37:20.571 INFO:teuthology.orchestra.run.smithi154.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 161/198 2024-03-16T11:37:20.614 INFO:teuthology.orchestra.run.smithi154.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 162/198 2024-03-16T11:37:20.647 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : libcephfs-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_ 163/198 2024-03-16T11:37:20.679 INFO:teuthology.orchestra.run.smithi154.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 164/198 2024-03-16T11:37:20.708 INFO:teuthology.orchestra.run.smithi154.stdout: Upgrading : NetworkManager-team-1:1.46.0-1.el9.x86_64 165/198 2024-03-16T11:37:20.762 INFO:teuthology.orchestra.run.smithi154.stdout: Upgrading : NetworkManager-tui-1:1.46.0-1.el9.x86_64 166/198 2024-03-16T11:37:20.823 INFO:teuthology.orchestra.run.smithi154.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 167/198 2024-03-16T11:37:20.865 INFO:teuthology.orchestra.run.smithi154.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 168/198 2024-03-16T11:37:20.865 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: cephadm-2:19.0.0-2162.g3d78f5ba.el9.noarch 138/198 2024-03-16T11:37:20.896 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : cephadm-2:19.0.0-2162.g3d78f5ba.el9.noarch 138/198 2024-03-16T11:37:20.928 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2162.g3d78f5ba.e 139/198 2024-03-16T11:37:20.988 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2162.g3d78f5ba. 140/198 2024-03-16T11:37:21.011 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2162.g3d78f5ba.el9.noa 141/198 2024-03-16T11:37:21.244 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2162.g3d78f5ba.el9.noa 141/198 2024-03-16T11:37:21.279 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2162.g3d78f5ba.el9.n 142/198 2024-03-16T11:37:21.440 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-16T11:37:21.480 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-16T11:37:21.499 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-16T11:37:22.239 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2162.g3d78f5ba.el9.n 142/198 2024-03-16T11:37:22.271 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2162.g3d7 143/198 2024-03-16T11:37:22.325 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : cephfs-top-2:19.0.0-2162.g3d78f5ba.el9.noarch 170/198 2024-03-16T11:37:22.371 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2162.g3d7 143/198 2024-03-16T11:37:22.400 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 171/198 2024-03-16T11:37:22.440 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 172/198 2024-03-16T11:37:22.520 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : flex-2.6.4-9.el9.x86_64 173/198 2024-03-16T11:37:22.533 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2162.g3d78f5ba.el 144/198 2024-03-16T11:37:22.562 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : bison-3.7.4-5.el9.x86_64 174/198 2024-03-16T11:37:22.564 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-mgr-2:19.0.0-2162.g3d78f5ba.el9.x86_64 145/198 2024-03-16T11:37:22.587 INFO:teuthology.orchestra.run.smithi154.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 175/198 2024-03-16T11:37:22.587 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2162.g3d78f5ba.el9.x86_64 145/198 2024-03-16T11:37:22.587 INFO:teuthology.orchestra.run.smithi152.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-16T11:37:22.587 INFO:teuthology.orchestra.run.smithi152.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-16T11:37:22.587 INFO:teuthology.orchestra.run.smithi152.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-16T11:37:22.587 INFO:teuthology.orchestra.run.smithi152.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-16T11:37:22.588 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-16T11:37:22.612 INFO:teuthology.orchestra.run.smithi154.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 176/198 2024-03-16T11:37:22.632 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-mgr-rook-2:19.0.0-2162.g3d78f5ba.el9.noarch 146/198 2024-03-16T11:37:22.638 INFO:teuthology.orchestra.run.smithi154.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 177/198 2024-03-16T11:37:22.639 INFO:teuthology.orchestra.run.smithi154.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-16T11:37:22.643 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2162.g3d78f5ba.el9.noarch 146/198 2024-03-16T11:37:22.688 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-2:19.0.0-2162.g3d78f5ba.el9.x86_64 147/198 2024-03-16T11:37:22.718 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-16T11:37:22.767 INFO:teuthology.orchestra.run.smithi154.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 179/198 2024-03-16T11:37:22.767 INFO:teuthology.orchestra.run.smithi154.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-16T11:37:22.849 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-16T11:37:22.875 INFO:teuthology.orchestra.run.smithi154.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 181/198 2024-03-16T11:37:22.917 INFO:teuthology.orchestra.run.smithi154.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 182/198 2024-03-16T11:37:22.942 INFO:teuthology.orchestra.run.smithi154.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 183/198 2024-03-16T11:37:22.967 INFO:teuthology.orchestra.run.smithi154.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 184/198 2024-03-16T11:37:23.000 INFO:teuthology.orchestra.run.smithi154.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 185/198 2024-03-16T11:37:23.040 INFO:teuthology.orchestra.run.smithi154.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 186/198 2024-03-16T11:37:23.072 INFO:teuthology.orchestra.run.smithi154.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 187/198 2024-03-16T11:37:23.097 INFO:teuthology.orchestra.run.smithi154.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 188/198 2024-03-16T11:37:23.122 INFO:teuthology.orchestra.run.smithi154.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 189/198 2024-03-16T11:37:23.148 INFO:teuthology.orchestra.run.smithi154.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 190/198 2024-03-16T11:37:23.173 INFO:teuthology.orchestra.run.smithi154.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 191/198 2024-03-16T11:37:23.179 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-16T11:37:23.179 INFO:teuthology.orchestra.run.smithi154.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-16T11:37:23.359 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-16T11:37:23.359 INFO:teuthology.orchestra.run.smithi154.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-16T11:37:23.442 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-16T11:37:23.466 INFO:teuthology.orchestra.run.smithi154.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 194/198 2024-03-16T11:37:23.492 INFO:teuthology.orchestra.run.smithi154.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 195/198 2024-03-16T11:37:23.533 INFO:teuthology.orchestra.run.smithi154.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 196/198 2024-03-16T11:37:23.560 INFO:teuthology.orchestra.run.smithi154.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 197/198 2024-03-16T11:37:23.562 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : libtool-2.4.6-45.el9.x86_64 148/198 2024-03-16T11:37:23.600 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-radosgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 149/198 2024-03-16T11:37:23.623 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 149/198 2024-03-16T11:37:23.623 INFO:teuthology.orchestra.run.smithi152.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-16T11:37:23.623 INFO:teuthology.orchestra.run.smithi152.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-16T11:37:23.623 INFO:teuthology.orchestra.run.smithi152.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-16T11:37:23.624 INFO:teuthology.orchestra.run.smithi152.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-16T11:37:23.624 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-16T11:37:23.625 INFO:teuthology.orchestra.run.smithi154.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-16T11:37:23.668 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2162.g3d78f 150/198 2024-03-16T11:37:23.690 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2162.g3d78f 150/198 2024-03-16T11:37:23.690 INFO:teuthology.orchestra.run.smithi152.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-16T11:37:23.690 INFO:teuthology.orchestra.run.smithi152.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-16T11:37:23.690 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-16T11:37:23.752 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : cephfs-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 151/198 2024-03-16T11:37:23.774 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 151/198 2024-03-16T11:37:23.831 INFO:teuthology.orchestra.run.smithi152.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-16T11:37:23.831 INFO:teuthology.orchestra.run.smithi152.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-03-16T11:37:23.831 INFO:teuthology.orchestra.run.smithi152.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-16T11:37:23.831 INFO:teuthology.orchestra.run.smithi152.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-16T11:37:23.831 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-16T11:37:24.086 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : rbd-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 152/198 2024-03-16T11:37:24.109 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 152/198 2024-03-16T11:37:24.109 INFO:teuthology.orchestra.run.smithi152.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-16T11:37:24.109 INFO:teuthology.orchestra.run.smithi152.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-16T11:37:24.109 INFO:teuthology.orchestra.run.smithi152.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-16T11:37:24.109 INFO:teuthology.orchestra.run.smithi152.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-16T11:37:24.109 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-16T11:37:28.557 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-test-2:19.0.0-2162.g3d78f5ba.el9.x86_64 153/198 2024-03-16T11:37:28.598 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : rbd-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 154/198 2024-03-16T11:37:28.624 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-16T11:37:28.624 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-2:19.0.0-2162.g3d78f5ba.el9.x86_64 1/198 2024-03-16T11:37:28.624 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-base-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2/198 2024-03-16T11:37:28.624 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 3/198 2024-03-16T11:37:28.624 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 4/198 2024-03-16T11:37:28.624 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2162.g3d78f 5/198 2024-03-16T11:37:28.624 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-mds-2:19.0.0-2162.g3d78f5ba.el9.x86_64 6/198 2024-03-16T11:37:28.625 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-mgr-2:19.0.0-2162.g3d78f5ba.el9.x86_64 7/198 2024-03-16T11:37:28.625 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-mon-2:19.0.0-2162.g3d78f5ba.el9.x86_64 8/198 2024-03-16T11:37:28.625 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-osd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 9/198 2024-03-16T11:37:28.625 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-radosgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 10/198 2024-03-16T11:37:28.625 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-selinux-2:19.0.0-2162.g3d78f5ba.el9.x86_64 11/198 2024-03-16T11:37:28.625 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-test-2:19.0.0-2162.g3d78f5ba.el9.x86_64 12/198 2024-03-16T11:37:28.625 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : cephfs-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 13/198 2024-03-16T11:37:28.625 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : cephfs-shell-2:19.0.0-2162.g3d78f5ba.el9.x86_64 14/198 2024-03-16T11:37:28.625 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : libcephfs-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_ 15/198 2024-03-16T11:37:28.625 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : libcephfs2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 16/198 2024-03-16T11:37:28.625 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : libcephsqlite-2:19.0.0-2162.g3d78f5ba.el9.x86_64 17/198 2024-03-16T11:37:28.625 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : librados-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_6 18/198 2024-03-16T11:37:28.625 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : libradosstriper1-2:19.0.0-2162.g3d78f5ba.el9.x86 19/198 2024-03-16T11:37:28.625 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : librgw2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 20/198 2024-03-16T11:37:28.626 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2162.g3d78f5ba.el 21/198 2024-03-16T11:37:28.626 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-ceph-common-2:19.0.0-2162.g3d78f5ba.el9. 22/198 2024-03-16T11:37:28.626 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-cephfs-2:19.0.0-2162.g3d78f5ba.el9.x86_6 23/198 2024-03-16T11:37:28.626 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-rados-2:19.0.0-2162.g3d78f5ba.el9.x86_64 24/198 2024-03-16T11:37:28.626 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-rbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 25/198 2024-03-16T11:37:28.626 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-rgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 26/198 2024-03-16T11:37:28.626 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : rbd-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 27/198 2024-03-16T11:37:28.626 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : rbd-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 28/198 2024-03-16T11:37:28.626 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : rbd-nbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 29/198 2024-03-16T11:37:28.632 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : rbd-nbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 155/198 2024-03-16T11:37:28.643 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2162.g3d78f5ba. 30/198 2024-03-16T11:37:28.643 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2162.g3d78f5ba.el9.noa 31/198 2024-03-16T11:37:28.643 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2162.g3d78f5ba.el9.n 32/198 2024-03-16T11:37:28.643 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2162.g3d7 33/198 2024-03-16T11:37:28.643 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2162.g3d78f5ba.el 34/198 2024-03-16T11:37:28.644 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2162.g3d78f5ba.el9.noarch 35/198 2024-03-16T11:37:28.644 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2162.g3d78f5ba.e 36/198 2024-03-16T11:37:28.644 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-volume-2:19.0.0-2162.g3d78f5ba.el9.noarch 37/198 2024-03-16T11:37:28.644 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : cephadm-2:19.0.0-2162.g3d78f5ba.el9.noarch 38/198 2024-03-16T11:37:28.644 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : cephfs-top-2:19.0.0-2162.g3d78f5ba.el9.noarch 39/198 2024-03-16T11:37:28.644 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/198 2024-03-16T11:37:28.644 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/198 2024-03-16T11:37:28.644 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/198 2024-03-16T11:37:28.644 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : flex-2.6.4-9.el9.x86_64 43/198 2024-03-16T11:37:28.644 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 44/198 2024-03-16T11:37:28.644 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 45/198 2024-03-16T11:37:28.644 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 46/198 2024-03-16T11:37:28.644 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 47/198 2024-03-16T11:37:28.644 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 48/198 2024-03-16T11:37:28.644 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 49/198 2024-03-16T11:37:28.645 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 50/198 2024-03-16T11:37:28.645 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 51/198 2024-03-16T11:37:28.645 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 52/198 2024-03-16T11:37:28.645 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 53/198 2024-03-16T11:37:28.645 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : lua-5.4.4-4.el9.x86_64 54/198 2024-03-16T11:37:28.645 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 55/198 2024-03-16T11:37:28.645 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 56/198 2024-03-16T11:37:28.645 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 57/198 2024-03-16T11:37:28.645 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : postgresql-13.11-1.el9.x86_64 58/198 2024-03-16T11:37:28.645 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : postgresql-contrib-13.11-1.el9.x86_64 59/198 2024-03-16T11:37:28.645 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : postgresql-private-libs-13.11-1.el9.x86_64 60/198 2024-03-16T11:37:28.645 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : postgresql-server-13.11-1.el9.x86_64 61/198 2024-03-16T11:37:28.645 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 62/198 2024-03-16T11:37:28.645 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 63/198 2024-03-16T11:37:28.645 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 64/198 2024-03-16T11:37:28.645 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 65/198 2024-03-16T11:37:28.645 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 66/198 2024-03-16T11:37:28.646 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 67/198 2024-03-16T11:37:28.646 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 68/198 2024-03-16T11:37:28.646 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 69/198 2024-03-16T11:37:28.646 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 70/198 2024-03-16T11:37:28.646 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 71/198 2024-03-16T11:37:28.646 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 72/198 2024-03-16T11:37:28.646 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 73/198 2024-03-16T11:37:28.646 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 74/198 2024-03-16T11:37:28.646 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 75/198 2024-03-16T11:37:28.646 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 76/198 2024-03-16T11:37:28.646 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 77/198 2024-03-16T11:37:28.646 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 78/198 2024-03-16T11:37:28.646 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 79/198 2024-03-16T11:37:28.647 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 80/198 2024-03-16T11:37:28.647 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 81/198 2024-03-16T11:37:28.647 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 82/198 2024-03-16T11:37:28.647 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 83/198 2024-03-16T11:37:28.647 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 84/198 2024-03-16T11:37:28.647 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 85/198 2024-03-16T11:37:28.647 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 86/198 2024-03-16T11:37:28.647 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 87/198 2024-03-16T11:37:28.647 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 88/198 2024-03-16T11:37:28.647 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-ply-3.11-14.el9.noarch 89/198 2024-03-16T11:37:28.647 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 90/198 2024-03-16T11:37:28.647 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 91/198 2024-03-16T11:37:28.647 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : jq-1.6-16.el9.x86_64 92/198 2024-03-16T11:37:28.647 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 93/198 2024-03-16T11:37:28.647 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 94/198 2024-03-16T11:37:28.648 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 95/198 2024-03-16T11:37:28.648 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 96/198 2024-03-16T11:37:28.648 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 97/198 2024-03-16T11:37:28.648 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 98/198 2024-03-16T11:37:28.648 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 99/198 2024-03-16T11:37:28.648 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 100/198 2024-03-16T11:37:28.648 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 101/198 2024-03-16T11:37:28.648 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 102/198 2024-03-16T11:37:28.648 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 103/198 2024-03-16T11:37:28.648 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 104/198 2024-03-16T11:37:28.648 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/198 2024-03-16T11:37:28.648 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/198 2024-03-16T11:37:28.648 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/198 2024-03-16T11:37:28.648 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/198 2024-03-16T11:37:28.648 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/198 2024-03-16T11:37:28.648 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/198 2024-03-16T11:37:28.648 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/198 2024-03-16T11:37:28.649 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/198 2024-03-16T11:37:28.649 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/198 2024-03-16T11:37:28.649 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/198 2024-03-16T11:37:28.649 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/198 2024-03-16T11:37:28.649 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-asyncssh-2.13.2-4.el9.noarch 116/198 2024-03-16T11:37:28.649 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/198 2024-03-16T11:37:28.649 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/198 2024-03-16T11:37:28.649 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/198 2024-03-16T11:37:28.649 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/198 2024-03-16T11:37:28.649 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/198 2024-03-16T11:37:28.649 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/198 2024-03-16T11:37:28.649 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/198 2024-03-16T11:37:28.649 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-google-auth-1:2.28.0-1.el9.noarch 124/198 2024-03-16T11:37:28.649 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/198 2024-03-16T11:37:28.649 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/198 2024-03-16T11:37:28.649 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/198 2024-03-16T11:37:28.649 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/198 2024-03-16T11:37:28.650 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/198 2024-03-16T11:37:28.650 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/198 2024-03-16T11:37:28.650 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/198 2024-03-16T11:37:28.650 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/198 2024-03-16T11:37:28.650 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/198 2024-03-16T11:37:28.650 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/198 2024-03-16T11:37:28.650 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/198 2024-03-16T11:37:28.650 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/198 2024-03-16T11:37:28.650 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/198 2024-03-16T11:37:28.650 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/198 2024-03-16T11:37:28.650 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/198 2024-03-16T11:37:28.650 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/198 2024-03-16T11:37:28.650 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/198 2024-03-16T11:37:28.650 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/198 2024-03-16T11:37:28.651 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/198 2024-03-16T11:37:28.651 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/198 2024-03-16T11:37:28.651 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/198 2024-03-16T11:37:28.651 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/198 2024-03-16T11:37:28.651 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/198 2024-03-16T11:37:28.651 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/198 2024-03-16T11:37:28.651 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/198 2024-03-16T11:37:28.651 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/198 2024-03-16T11:37:28.651 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : librados2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 151/198 2024-03-16T11:37:28.651 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/198 2024-03-16T11:37:28.651 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : librbd1-2:19.0.0-2162.g3d78f5ba.el9.x86_64 153/198 2024-03-16T11:37:28.651 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/198 2024-03-16T11:37:28.651 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 155/198 2024-03-16T11:37:28.651 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 156/198 2024-03-16T11:37:28.651 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : NetworkManager-1:1.46.0-1.el9.x86_64 157/198 2024-03-16T11:37:28.651 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 158/198 2024-03-16T11:37:28.652 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : NetworkManager-libnm-1:1.46.0-1.el9.x86_64 159/198 2024-03-16T11:37:28.652 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 160/198 2024-03-16T11:37:28.652 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : NetworkManager-team-1:1.46.0-1.el9.x86_64 161/198 2024-03-16T11:37:28.652 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 162/198 2024-03-16T11:37:28.652 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : NetworkManager-tui-1:1.46.0-1.el9.x86_64 163/198 2024-03-16T11:37:28.652 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 164/198 2024-03-16T11:37:28.652 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/198 2024-03-16T11:37:28.652 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/198 2024-03-16T11:37:28.652 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/198 2024-03-16T11:37:28.652 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/198 2024-03-16T11:37:28.652 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/198 2024-03-16T11:37:28.652 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/198 2024-03-16T11:37:28.652 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/198 2024-03-16T11:37:28.652 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/198 2024-03-16T11:37:28.652 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 173/198 2024-03-16T11:37:28.652 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 174/198 2024-03-16T11:37:28.652 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 175/198 2024-03-16T11:37:28.653 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 176/198 2024-03-16T11:37:28.653 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 177/198 2024-03-16T11:37:28.653 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 178/198 2024-03-16T11:37:28.653 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 179/198 2024-03-16T11:37:28.653 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 180/198 2024-03-16T11:37:28.653 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 181/198 2024-03-16T11:37:28.653 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 182/198 2024-03-16T11:37:28.653 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 183/198 2024-03-16T11:37:28.653 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 184/198 2024-03-16T11:37:28.653 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 185/198 2024-03-16T11:37:28.653 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 186/198 2024-03-16T11:37:28.653 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 187/198 2024-03-16T11:37:28.653 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 188/198 2024-03-16T11:37:28.653 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 189/198 2024-03-16T11:37:28.653 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 190/198 2024-03-16T11:37:28.653 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 191/198 2024-03-16T11:37:28.654 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 192/198 2024-03-16T11:37:28.654 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 193/198 2024-03-16T11:37:28.654 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 194/198 2024-03-16T11:37:28.654 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 195/198 2024-03-16T11:37:28.654 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 196/198 2024-03-16T11:37:28.654 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 197/198 2024-03-16T11:37:28.655 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 156/198 2024-03-16T11:37:28.834 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : cephfs-shell-2:19.0.0-2162.g3d78f5ba.el9.x86_64 157/198 2024-03-16T11:37:28.946 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : postgresql-contrib-13.11-1.el9.x86_64 158/198 2024-03-16T11:37:28.995 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 159/198 2024-03-16T11:37:29.058 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 160/198 2024-03-16T11:37:29.136 INFO:teuthology.orchestra.run.smithi152.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 161/198 2024-03-16T11:37:29.190 INFO:teuthology.orchestra.run.smithi152.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 162/198 2024-03-16T11:37:29.221 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : libcephfs-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_ 163/198 2024-03-16T11:37:29.261 INFO:teuthology.orchestra.run.smithi152.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 164/198 2024-03-16T11:37:29.290 INFO:teuthology.orchestra.run.smithi152.stdout: Upgrading : NetworkManager-team-1:1.46.0-1.el9.x86_64 165/198 2024-03-16T11:37:29.366 INFO:teuthology.orchestra.run.smithi152.stdout: Upgrading : NetworkManager-tui-1:1.46.0-1.el9.x86_64 166/198 2024-03-16T11:37:29.422 INFO:teuthology.orchestra.run.smithi152.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 167/198 2024-03-16T11:37:29.456 INFO:teuthology.orchestra.run.smithi152.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 168/198 2024-03-16T11:37:30.009 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-16T11:37:30.053 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-16T11:37:30.073 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-16T11:37:30.896 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : cephfs-top-2:19.0.0-2162.g3d78f5ba.el9.noarch 170/198 2024-03-16T11:37:30.973 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 171/198 2024-03-16T11:37:31.013 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 172/198 2024-03-16T11:37:31.093 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : flex-2.6.4-9.el9.x86_64 173/198 2024-03-16T11:37:31.126 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : bison-3.7.4-5.el9.x86_64 174/198 2024-03-16T11:37:31.151 INFO:teuthology.orchestra.run.smithi152.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 175/198 2024-03-16T11:37:31.176 INFO:teuthology.orchestra.run.smithi152.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 176/198 2024-03-16T11:37:31.203 INFO:teuthology.orchestra.run.smithi152.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 177/198 2024-03-16T11:37:31.203 INFO:teuthology.orchestra.run.smithi152.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-16T11:37:31.290 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-16T11:37:31.320 INFO:teuthology.orchestra.run.smithi152.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 179/198 2024-03-16T11:37:31.320 INFO:teuthology.orchestra.run.smithi152.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-16T11:37:31.388 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-16T11:37:31.414 INFO:teuthology.orchestra.run.smithi152.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 181/198 2024-03-16T11:37:31.448 INFO:teuthology.orchestra.run.smithi152.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 182/198 2024-03-16T11:37:31.481 INFO:teuthology.orchestra.run.smithi152.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 183/198 2024-03-16T11:37:31.506 INFO:teuthology.orchestra.run.smithi152.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 184/198 2024-03-16T11:37:31.539 INFO:teuthology.orchestra.run.smithi152.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 185/198 2024-03-16T11:37:31.587 INFO:teuthology.orchestra.run.smithi152.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 186/198 2024-03-16T11:37:31.627 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 198/198 2024-03-16T11:37:31.627 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-03-16T11:37:31.627 INFO:teuthology.orchestra.run.smithi154.stdout:Upgraded: 2024-03-16T11:37:31.627 INFO:teuthology.orchestra.run.smithi154.stdout: NetworkManager-1:1.46.0-1.el9.x86_64 2024-03-16T11:37:31.627 INFO:teuthology.orchestra.run.smithi154.stdout: NetworkManager-libnm-1:1.46.0-1.el9.x86_64 2024-03-16T11:37:31.627 INFO:teuthology.orchestra.run.smithi154.stdout: NetworkManager-team-1:1.46.0-1.el9.x86_64 2024-03-16T11:37:31.627 INFO:teuthology.orchestra.run.smithi154.stdout: NetworkManager-tui-1:1.46.0-1.el9.x86_64 2024-03-16T11:37:31.627 INFO:teuthology.orchestra.run.smithi154.stdout: acl-2.3.1-4.el9.x86_64 2024-03-16T11:37:31.628 INFO:teuthology.orchestra.run.smithi154.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-03-16T11:37:31.628 INFO:teuthology.orchestra.run.smithi154.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-03-16T11:37:31.628 INFO:teuthology.orchestra.run.smithi154.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-03-16T11:37:31.628 INFO:teuthology.orchestra.run.smithi154.stdout: libacl-2.3.1-4.el9.x86_64 2024-03-16T11:37:31.628 INFO:teuthology.orchestra.run.smithi154.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-03-16T11:37:31.628 INFO:teuthology.orchestra.run.smithi154.stdout: libblkid-2.37.4-18.el9.x86_64 2024-03-16T11:37:31.628 INFO:teuthology.orchestra.run.smithi154.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-03-16T11:37:31.628 INFO:teuthology.orchestra.run.smithi154.stdout: libmount-2.37.4-18.el9.x86_64 2024-03-16T11:37:31.628 INFO:teuthology.orchestra.run.smithi154.stdout: librados2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T11:37:31.628 INFO:teuthology.orchestra.run.smithi154.stdout: librbd1-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T11:37:31.628 INFO:teuthology.orchestra.run.smithi154.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-03-16T11:37:31.628 INFO:teuthology.orchestra.run.smithi154.stdout: libuuid-2.37.4-18.el9.x86_64 2024-03-16T11:37:31.628 INFO:teuthology.orchestra.run.smithi154.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-03-16T11:37:31.628 INFO:teuthology.orchestra.run.smithi154.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-03-16T11:37:31.629 INFO:teuthology.orchestra.run.smithi154.stdout: util-linux-2.37.4-18.el9.x86_64 2024-03-16T11:37:31.629 INFO:teuthology.orchestra.run.smithi154.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-03-16T11:37:31.629 INFO:teuthology.orchestra.run.smithi154.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-03-16T11:37:31.629 INFO:teuthology.orchestra.run.smithi154.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-03-16T11:37:31.629 INFO:teuthology.orchestra.run.smithi154.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-03-16T11:37:31.629 INFO:teuthology.orchestra.run.smithi154.stdout:Installed: 2024-03-16T11:37:31.629 INFO:teuthology.orchestra.run.smithi154.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-16T11:37:31.629 INFO:teuthology.orchestra.run.smithi154.stdout: automake-1.16.2-8.el9.noarch 2024-03-16T11:37:31.629 INFO:teuthology.orchestra.run.smithi154.stdout: bison-3.7.4-5.el9.x86_64 2024-03-16T11:37:31.629 INFO:teuthology.orchestra.run.smithi154.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-16T11:37:31.629 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T11:37:31.629 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-base-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T11:37:31.629 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T11:37:31.629 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T11:37:31.630 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-grafana-dashboards-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-16T11:37:31.630 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-immutable-object-cache-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T11:37:31.630 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mds-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T11:37:31.630 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mgr-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T11:37:31.630 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mgr-cephadm-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-16T11:37:31.630 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mgr-dashboard-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-16T11:37:31.630 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-16T11:37:31.630 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mgr-modules-core-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-16T11:37:31.630 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mgr-rook-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-16T11:37:31.630 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mon-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T11:37:31.630 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-osd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T11:37:31.630 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-prometheus-alerts-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-16T11:37:31.630 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-radosgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T11:37:31.630 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-selinux-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T11:37:31.631 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-test-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T11:37:31.631 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-volume-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-16T11:37:31.631 INFO:teuthology.orchestra.run.smithi154.stdout: cephadm-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-16T11:37:31.631 INFO:teuthology.orchestra.run.smithi154.stdout: cephfs-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T11:37:31.631 INFO:teuthology.orchestra.run.smithi154.stdout: cephfs-shell-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T11:37:31.631 INFO:teuthology.orchestra.run.smithi154.stdout: cephfs-top-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-16T11:37:31.631 INFO:teuthology.orchestra.run.smithi154.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-03-16T11:37:31.631 INFO:teuthology.orchestra.run.smithi154.stdout: flex-2.6.4-9.el9.x86_64 2024-03-16T11:37:31.631 INFO:teuthology.orchestra.run.smithi154.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-16T11:37:31.631 INFO:teuthology.orchestra.run.smithi154.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-16T11:37:31.631 INFO:teuthology.orchestra.run.smithi154.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-16T11:37:31.631 INFO:teuthology.orchestra.run.smithi154.stdout: fuse-2.9.9-15.el9.x86_64 2024-03-16T11:37:31.631 INFO:teuthology.orchestra.run.smithi154.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-16T11:37:31.632 INFO:teuthology.orchestra.run.smithi154.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-16T11:37:31.632 INFO:teuthology.orchestra.run.smithi154.stdout: jq-1.6-16.el9.x86_64 2024-03-16T11:37:31.632 INFO:teuthology.orchestra.run.smithi154.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-03-16T11:37:31.632 INFO:teuthology.orchestra.run.smithi154.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-16T11:37:31.632 INFO:teuthology.orchestra.run.smithi154.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-16T11:37:31.632 INFO:teuthology.orchestra.run.smithi154.stdout: libcephfs-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T11:37:31.632 INFO:teuthology.orchestra.run.smithi154.stdout: libcephfs2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T11:37:31.632 INFO:teuthology.orchestra.run.smithi154.stdout: libcephsqlite-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T11:37:31.632 INFO:teuthology.orchestra.run.smithi154.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-16T11:37:31.632 INFO:teuthology.orchestra.run.smithi154.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-16T11:37:31.632 INFO:teuthology.orchestra.run.smithi154.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-16T11:37:31.632 INFO:teuthology.orchestra.run.smithi154.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-16T11:37:31.632 INFO:teuthology.orchestra.run.smithi154.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-16T11:37:31.632 INFO:teuthology.orchestra.run.smithi154.stdout: librados-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T11:37:31.632 INFO:teuthology.orchestra.run.smithi154.stdout: libradosstriper1-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T11:37:31.633 INFO:teuthology.orchestra.run.smithi154.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-16T11:37:31.633 INFO:teuthology.orchestra.run.smithi154.stdout: librgw2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T11:37:31.633 INFO:teuthology.orchestra.run.smithi154.stdout: libtool-2.4.6-45.el9.x86_64 2024-03-16T11:37:31.633 INFO:teuthology.orchestra.run.smithi154.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-16T11:37:31.633 INFO:teuthology.orchestra.run.smithi154.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-03-16T11:37:31.633 INFO:teuthology.orchestra.run.smithi154.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-03-16T11:37:31.633 INFO:teuthology.orchestra.run.smithi154.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-16T11:37:31.633 INFO:teuthology.orchestra.run.smithi154.stdout: lua-5.4.4-4.el9.x86_64 2024-03-16T11:37:31.633 INFO:teuthology.orchestra.run.smithi154.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-16T11:37:31.633 INFO:teuthology.orchestra.run.smithi154.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-16T11:37:31.633 INFO:teuthology.orchestra.run.smithi154.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-16T11:37:31.633 INFO:teuthology.orchestra.run.smithi154.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-16T11:37:31.633 INFO:teuthology.orchestra.run.smithi154.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-16T11:37:31.633 INFO:teuthology.orchestra.run.smithi154.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-03-16T11:37:31.633 INFO:teuthology.orchestra.run.smithi154.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-16T11:37:31.634 INFO:teuthology.orchestra.run.smithi154.stdout: postgresql-13.11-1.el9.x86_64 2024-03-16T11:37:31.634 INFO:teuthology.orchestra.run.smithi154.stdout: postgresql-contrib-13.11-1.el9.x86_64 2024-03-16T11:37:31.634 INFO:teuthology.orchestra.run.smithi154.stdout: postgresql-private-libs-13.11-1.el9.x86_64 2024-03-16T11:37:31.634 INFO:teuthology.orchestra.run.smithi154.stdout: postgresql-server-13.11-1.el9.x86_64 2024-03-16T11:37:31.634 INFO:teuthology.orchestra.run.smithi154.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-16T11:37:31.634 INFO:teuthology.orchestra.run.smithi154.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-16T11:37:31.634 INFO:teuthology.orchestra.run.smithi154.stdout: python3-asyncssh-2.13.2-4.el9.noarch 2024-03-16T11:37:31.634 INFO:teuthology.orchestra.run.smithi154.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-03-16T11:37:31.634 INFO:teuthology.orchestra.run.smithi154.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-16T11:37:31.634 INFO:teuthology.orchestra.run.smithi154.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-16T11:37:31.634 INFO:teuthology.orchestra.run.smithi154.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-16T11:37:31.634 INFO:teuthology.orchestra.run.smithi154.stdout: python3-ceph-argparse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T11:37:31.634 INFO:teuthology.orchestra.run.smithi154.stdout: python3-ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T11:37:31.634 INFO:teuthology.orchestra.run.smithi154.stdout: python3-cephfs-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T11:37:31.634 INFO:teuthology.orchestra.run.smithi154.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-16T11:37:31.635 INFO:teuthology.orchestra.run.smithi154.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-16T11:37:31.635 INFO:teuthology.orchestra.run.smithi154.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-16T11:37:31.635 INFO:teuthology.orchestra.run.smithi154.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-16T11:37:31.635 INFO:teuthology.orchestra.run.smithi154.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-03-16T11:37:31.635 INFO:teuthology.orchestra.run.smithi154.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-03-16T11:37:31.635 INFO:teuthology.orchestra.run.smithi154.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-16T11:37:31.635 INFO:teuthology.orchestra.run.smithi154.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-16T11:37:31.635 INFO:teuthology.orchestra.run.smithi154.stdout: python3-google-auth-1:2.28.0-1.el9.noarch 2024-03-16T11:37:31.635 INFO:teuthology.orchestra.run.smithi154.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-16T11:37:31.635 INFO:teuthology.orchestra.run.smithi154.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-16T11:37:31.635 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-16T11:37:31.635 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-16T11:37:31.635 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-16T11:37:31.635 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-16T11:37:31.636 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-16T11:37:31.636 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-16T11:37:31.636 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-16T11:37:31.636 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-16T11:37:31.636 INFO:teuthology.orchestra.run.smithi154.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-16T11:37:31.636 INFO:teuthology.orchestra.run.smithi154.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-16T11:37:31.636 INFO:teuthology.orchestra.run.smithi154.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-16T11:37:31.636 INFO:teuthology.orchestra.run.smithi154.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-16T11:37:31.636 INFO:teuthology.orchestra.run.smithi154.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-16T11:37:31.636 INFO:teuthology.orchestra.run.smithi154.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-16T11:37:31.636 INFO:teuthology.orchestra.run.smithi154.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-16T11:37:31.636 INFO:teuthology.orchestra.run.smithi154.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-16T11:37:31.636 INFO:teuthology.orchestra.run.smithi154.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-16T11:37:31.636 INFO:teuthology.orchestra.run.smithi154.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-16T11:37:31.636 INFO:teuthology.orchestra.run.smithi154.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-16T11:37:31.636 INFO:teuthology.orchestra.run.smithi154.stdout: python3-ply-3.11-14.el9.noarch 2024-03-16T11:37:31.636 INFO:teuthology.orchestra.run.smithi154.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-16T11:37:31.637 INFO:teuthology.orchestra.run.smithi154.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-16T11:37:31.637 INFO:teuthology.orchestra.run.smithi154.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-16T11:37:31.637 INFO:teuthology.orchestra.run.smithi154.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-16T11:37:31.637 INFO:teuthology.orchestra.run.smithi154.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-16T11:37:31.637 INFO:teuthology.orchestra.run.smithi154.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-16T11:37:31.637 INFO:teuthology.orchestra.run.smithi154.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-16T11:37:31.637 INFO:teuthology.orchestra.run.smithi154.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-03-16T11:37:31.637 INFO:teuthology.orchestra.run.smithi154.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-16T11:37:31.637 INFO:teuthology.orchestra.run.smithi154.stdout: python3-rados-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T11:37:31.637 INFO:teuthology.orchestra.run.smithi154.stdout: python3-rbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T11:37:31.637 INFO:teuthology.orchestra.run.smithi154.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-16T11:37:31.637 INFO:teuthology.orchestra.run.smithi154.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-16T11:37:31.637 INFO:teuthology.orchestra.run.smithi154.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-16T11:37:31.637 INFO:teuthology.orchestra.run.smithi154.stdout: python3-rgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T11:37:31.637 INFO:teuthology.orchestra.run.smithi154.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-16T11:37:31.638 INFO:teuthology.orchestra.run.smithi154.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-16T11:37:31.638 INFO:teuthology.orchestra.run.smithi154.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-16T11:37:31.638 INFO:teuthology.orchestra.run.smithi154.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-16T11:37:31.638 INFO:teuthology.orchestra.run.smithi154.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-16T11:37:31.638 INFO:teuthology.orchestra.run.smithi154.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-16T11:37:31.638 INFO:teuthology.orchestra.run.smithi154.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-16T11:37:31.638 INFO:teuthology.orchestra.run.smithi154.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-03-16T11:37:31.638 INFO:teuthology.orchestra.run.smithi154.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-16T11:37:31.638 INFO:teuthology.orchestra.run.smithi154.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-16T11:37:31.638 INFO:teuthology.orchestra.run.smithi154.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-16T11:37:31.638 INFO:teuthology.orchestra.run.smithi154.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-16T11:37:31.638 INFO:teuthology.orchestra.run.smithi154.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-16T11:37:31.638 INFO:teuthology.orchestra.run.smithi154.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-16T11:37:31.639 INFO:teuthology.orchestra.run.smithi154.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-16T11:37:31.639 INFO:teuthology.orchestra.run.smithi154.stdout: rbd-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T11:37:31.639 INFO:teuthology.orchestra.run.smithi154.stdout: rbd-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T11:37:31.639 INFO:teuthology.orchestra.run.smithi154.stdout: rbd-nbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T11:37:31.639 INFO:teuthology.orchestra.run.smithi154.stdout: re2-1:20211101-3.el9.x86_64 2024-03-16T11:37:31.639 INFO:teuthology.orchestra.run.smithi154.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-16T11:37:31.639 INFO:teuthology.orchestra.run.smithi154.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-16T11:37:31.639 INFO:teuthology.orchestra.run.smithi154.stdout: uuid-1.6.2-55.el9.x86_64 2024-03-16T11:37:31.639 INFO:teuthology.orchestra.run.smithi154.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-03-16T11:37:31.639 INFO:teuthology.orchestra.run.smithi154.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-16T11:37:31.640 INFO:teuthology.orchestra.run.smithi152.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 187/198 2024-03-16T11:37:31.640 INFO:teuthology.orchestra.run.smithi154.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-03-16T11:37:31.640 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-03-16T11:37:31.640 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2024-03-16T11:37:31.652 INFO:teuthology.orchestra.run.smithi152.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 188/198 2024-03-16T11:37:31.678 INFO:teuthology.orchestra.run.smithi152.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 189/198 2024-03-16T11:37:31.703 INFO:teuthology.orchestra.run.smithi152.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 190/198 2024-03-16T11:37:31.728 INFO:teuthology.orchestra.run.smithi152.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 191/198 2024-03-16T11:37:31.735 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-16T11:37:31.735 INFO:teuthology.orchestra.run.smithi152.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-16T11:37:31.915 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-16T11:37:31.915 INFO:teuthology.orchestra.run.smithi152.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-16T11:37:32.014 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-16T11:37:32.039 INFO:teuthology.orchestra.run.smithi152.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 194/198 2024-03-16T11:37:32.065 INFO:teuthology.orchestra.run.smithi152.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 195/198 2024-03-16T11:37:32.098 INFO:teuthology.orchestra.run.smithi152.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 196/198 2024-03-16T11:37:32.124 INFO:teuthology.orchestra.run.smithi152.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 197/198 2024-03-16T11:37:32.132 DEBUG:teuthology.parallel:result is None 2024-03-16T11:37:32.187 INFO:teuthology.orchestra.run.smithi152.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-16T11:37:37.365 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-16T11:37:37.365 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-2:19.0.0-2162.g3d78f5ba.el9.x86_64 1/198 2024-03-16T11:37:37.365 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-base-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2/198 2024-03-16T11:37:37.365 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 3/198 2024-03-16T11:37:37.365 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 4/198 2024-03-16T11:37:37.365 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2162.g3d78f 5/198 2024-03-16T11:37:37.365 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-mds-2:19.0.0-2162.g3d78f5ba.el9.x86_64 6/198 2024-03-16T11:37:37.365 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-mgr-2:19.0.0-2162.g3d78f5ba.el9.x86_64 7/198 2024-03-16T11:37:37.365 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-mon-2:19.0.0-2162.g3d78f5ba.el9.x86_64 8/198 2024-03-16T11:37:37.365 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-osd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 9/198 2024-03-16T11:37:37.366 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-radosgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 10/198 2024-03-16T11:37:37.366 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-selinux-2:19.0.0-2162.g3d78f5ba.el9.x86_64 11/198 2024-03-16T11:37:37.366 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-test-2:19.0.0-2162.g3d78f5ba.el9.x86_64 12/198 2024-03-16T11:37:37.366 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : cephfs-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 13/198 2024-03-16T11:37:37.366 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : cephfs-shell-2:19.0.0-2162.g3d78f5ba.el9.x86_64 14/198 2024-03-16T11:37:37.366 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : libcephfs-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_ 15/198 2024-03-16T11:37:37.366 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : libcephfs2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 16/198 2024-03-16T11:37:37.366 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : libcephsqlite-2:19.0.0-2162.g3d78f5ba.el9.x86_64 17/198 2024-03-16T11:37:37.366 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : librados-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_6 18/198 2024-03-16T11:37:37.366 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : libradosstriper1-2:19.0.0-2162.g3d78f5ba.el9.x86 19/198 2024-03-16T11:37:37.366 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : librgw2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 20/198 2024-03-16T11:37:37.366 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2162.g3d78f5ba.el 21/198 2024-03-16T11:37:37.366 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-ceph-common-2:19.0.0-2162.g3d78f5ba.el9. 22/198 2024-03-16T11:37:37.366 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-cephfs-2:19.0.0-2162.g3d78f5ba.el9.x86_6 23/198 2024-03-16T11:37:37.366 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-rados-2:19.0.0-2162.g3d78f5ba.el9.x86_64 24/198 2024-03-16T11:37:37.367 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-rbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 25/198 2024-03-16T11:37:37.367 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-rgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 26/198 2024-03-16T11:37:37.367 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : rbd-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 27/198 2024-03-16T11:37:37.367 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : rbd-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 28/198 2024-03-16T11:37:37.367 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : rbd-nbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 29/198 2024-03-16T11:37:37.367 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2162.g3d78f5ba. 30/198 2024-03-16T11:37:37.367 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2162.g3d78f5ba.el9.noa 31/198 2024-03-16T11:37:37.367 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2162.g3d78f5ba.el9.n 32/198 2024-03-16T11:37:37.367 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2162.g3d7 33/198 2024-03-16T11:37:37.367 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2162.g3d78f5ba.el 34/198 2024-03-16T11:37:37.367 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2162.g3d78f5ba.el9.noarch 35/198 2024-03-16T11:37:37.367 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2162.g3d78f5ba.e 36/198 2024-03-16T11:37:37.367 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-volume-2:19.0.0-2162.g3d78f5ba.el9.noarch 37/198 2024-03-16T11:37:37.367 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : cephadm-2:19.0.0-2162.g3d78f5ba.el9.noarch 38/198 2024-03-16T11:37:37.367 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : cephfs-top-2:19.0.0-2162.g3d78f5ba.el9.noarch 39/198 2024-03-16T11:37:37.368 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/198 2024-03-16T11:37:37.368 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/198 2024-03-16T11:37:37.368 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/198 2024-03-16T11:37:37.368 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : flex-2.6.4-9.el9.x86_64 43/198 2024-03-16T11:37:37.368 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 44/198 2024-03-16T11:37:37.368 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 45/198 2024-03-16T11:37:37.368 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 46/198 2024-03-16T11:37:37.368 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 47/198 2024-03-16T11:37:37.368 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 48/198 2024-03-16T11:37:37.368 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 49/198 2024-03-16T11:37:37.368 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 50/198 2024-03-16T11:37:37.368 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 51/198 2024-03-16T11:37:37.368 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 52/198 2024-03-16T11:37:37.368 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 53/198 2024-03-16T11:37:37.368 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : lua-5.4.4-4.el9.x86_64 54/198 2024-03-16T11:37:37.368 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 55/198 2024-03-16T11:37:37.369 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 56/198 2024-03-16T11:37:37.369 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 57/198 2024-03-16T11:37:37.369 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : postgresql-13.11-1.el9.x86_64 58/198 2024-03-16T11:37:37.369 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : postgresql-contrib-13.11-1.el9.x86_64 59/198 2024-03-16T11:37:37.369 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : postgresql-private-libs-13.11-1.el9.x86_64 60/198 2024-03-16T11:37:37.369 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : postgresql-server-13.11-1.el9.x86_64 61/198 2024-03-16T11:37:37.369 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 62/198 2024-03-16T11:37:37.369 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 63/198 2024-03-16T11:37:37.369 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 64/198 2024-03-16T11:37:37.369 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 65/198 2024-03-16T11:37:37.369 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 66/198 2024-03-16T11:37:37.369 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 67/198 2024-03-16T11:37:37.369 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 68/198 2024-03-16T11:37:37.369 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 69/198 2024-03-16T11:37:37.369 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 70/198 2024-03-16T11:37:37.370 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 71/198 2024-03-16T11:37:37.370 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 72/198 2024-03-16T11:37:37.370 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 73/198 2024-03-16T11:37:37.370 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 74/198 2024-03-16T11:37:37.370 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 75/198 2024-03-16T11:37:37.370 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 76/198 2024-03-16T11:37:37.370 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 77/198 2024-03-16T11:37:37.370 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 78/198 2024-03-16T11:37:37.370 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 79/198 2024-03-16T11:37:37.370 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 80/198 2024-03-16T11:37:37.370 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 81/198 2024-03-16T11:37:37.370 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 82/198 2024-03-16T11:37:37.370 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 83/198 2024-03-16T11:37:37.370 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 84/198 2024-03-16T11:37:37.370 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 85/198 2024-03-16T11:37:37.371 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 86/198 2024-03-16T11:37:37.371 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 87/198 2024-03-16T11:37:37.371 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 88/198 2024-03-16T11:37:37.371 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-ply-3.11-14.el9.noarch 89/198 2024-03-16T11:37:37.371 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 90/198 2024-03-16T11:37:37.371 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 91/198 2024-03-16T11:37:37.371 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : jq-1.6-16.el9.x86_64 92/198 2024-03-16T11:37:37.371 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 93/198 2024-03-16T11:37:37.371 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 94/198 2024-03-16T11:37:37.371 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 95/198 2024-03-16T11:37:37.371 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 96/198 2024-03-16T11:37:37.371 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 97/198 2024-03-16T11:37:37.371 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 98/198 2024-03-16T11:37:37.371 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 99/198 2024-03-16T11:37:37.372 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 100/198 2024-03-16T11:37:37.372 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 101/198 2024-03-16T11:37:37.372 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 102/198 2024-03-16T11:37:37.372 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 103/198 2024-03-16T11:37:37.372 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 104/198 2024-03-16T11:37:37.372 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/198 2024-03-16T11:37:37.372 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/198 2024-03-16T11:37:37.372 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/198 2024-03-16T11:37:37.372 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/198 2024-03-16T11:37:37.372 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/198 2024-03-16T11:37:37.372 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/198 2024-03-16T11:37:37.372 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/198 2024-03-16T11:37:37.372 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/198 2024-03-16T11:37:37.372 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/198 2024-03-16T11:37:37.372 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/198 2024-03-16T11:37:37.372 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/198 2024-03-16T11:37:37.373 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-asyncssh-2.13.2-4.el9.noarch 116/198 2024-03-16T11:37:37.373 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/198 2024-03-16T11:37:37.373 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/198 2024-03-16T11:37:37.373 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/198 2024-03-16T11:37:37.373 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/198 2024-03-16T11:37:37.373 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/198 2024-03-16T11:37:37.373 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/198 2024-03-16T11:37:37.373 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/198 2024-03-16T11:37:37.373 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-google-auth-1:2.28.0-1.el9.noarch 124/198 2024-03-16T11:37:37.373 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/198 2024-03-16T11:37:37.373 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/198 2024-03-16T11:37:37.373 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/198 2024-03-16T11:37:37.373 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/198 2024-03-16T11:37:37.373 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/198 2024-03-16T11:37:37.373 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/198 2024-03-16T11:37:37.374 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/198 2024-03-16T11:37:37.374 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/198 2024-03-16T11:37:37.374 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/198 2024-03-16T11:37:37.374 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/198 2024-03-16T11:37:37.374 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/198 2024-03-16T11:37:37.374 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/198 2024-03-16T11:37:37.374 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/198 2024-03-16T11:37:37.374 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/198 2024-03-16T11:37:37.374 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/198 2024-03-16T11:37:37.374 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/198 2024-03-16T11:37:37.374 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/198 2024-03-16T11:37:37.374 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/198 2024-03-16T11:37:37.374 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/198 2024-03-16T11:37:37.374 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/198 2024-03-16T11:37:37.374 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/198 2024-03-16T11:37:37.375 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/198 2024-03-16T11:37:37.375 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/198 2024-03-16T11:37:37.375 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/198 2024-03-16T11:37:37.375 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/198 2024-03-16T11:37:37.375 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/198 2024-03-16T11:37:37.375 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : librados2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 151/198 2024-03-16T11:37:37.375 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/198 2024-03-16T11:37:37.375 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : librbd1-2:19.0.0-2162.g3d78f5ba.el9.x86_64 153/198 2024-03-16T11:37:37.375 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/198 2024-03-16T11:37:37.375 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 155/198 2024-03-16T11:37:37.375 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 156/198 2024-03-16T11:37:37.375 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : NetworkManager-1:1.46.0-1.el9.x86_64 157/198 2024-03-16T11:37:37.375 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 158/198 2024-03-16T11:37:37.375 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : NetworkManager-libnm-1:1.46.0-1.el9.x86_64 159/198 2024-03-16T11:37:37.375 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 160/198 2024-03-16T11:37:37.375 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : NetworkManager-team-1:1.46.0-1.el9.x86_64 161/198 2024-03-16T11:37:37.376 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 162/198 2024-03-16T11:37:37.376 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : NetworkManager-tui-1:1.46.0-1.el9.x86_64 163/198 2024-03-16T11:37:37.376 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 164/198 2024-03-16T11:37:37.376 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/198 2024-03-16T11:37:37.376 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/198 2024-03-16T11:37:37.376 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/198 2024-03-16T11:37:37.376 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/198 2024-03-16T11:37:37.376 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/198 2024-03-16T11:37:37.376 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/198 2024-03-16T11:37:37.376 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/198 2024-03-16T11:37:37.376 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/198 2024-03-16T11:37:37.376 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 173/198 2024-03-16T11:37:37.376 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 174/198 2024-03-16T11:37:37.376 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 175/198 2024-03-16T11:37:37.376 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 176/198 2024-03-16T11:37:37.376 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 177/198 2024-03-16T11:37:37.376 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 178/198 2024-03-16T11:37:37.377 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 179/198 2024-03-16T11:37:37.377 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 180/198 2024-03-16T11:37:37.377 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 181/198 2024-03-16T11:37:37.377 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 182/198 2024-03-16T11:37:37.377 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 183/198 2024-03-16T11:37:37.377 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 184/198 2024-03-16T11:37:37.377 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 185/198 2024-03-16T11:37:37.377 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 186/198 2024-03-16T11:37:37.377 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 187/198 2024-03-16T11:37:37.377 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 188/198 2024-03-16T11:37:37.377 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 189/198 2024-03-16T11:37:37.377 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 190/198 2024-03-16T11:37:37.377 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 191/198 2024-03-16T11:37:37.377 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 192/198 2024-03-16T11:37:37.377 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 193/198 2024-03-16T11:37:37.377 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 194/198 2024-03-16T11:37:37.377 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 195/198 2024-03-16T11:37:37.378 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 196/198 2024-03-16T11:37:37.378 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 197/198 2024-03-16T11:37:40.091 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 198/198 2024-03-16T11:37:40.092 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-16T11:37:40.092 INFO:teuthology.orchestra.run.smithi152.stdout:Upgraded: 2024-03-16T11:37:40.092 INFO:teuthology.orchestra.run.smithi152.stdout: NetworkManager-1:1.46.0-1.el9.x86_64 2024-03-16T11:37:40.092 INFO:teuthology.orchestra.run.smithi152.stdout: NetworkManager-libnm-1:1.46.0-1.el9.x86_64 2024-03-16T11:37:40.092 INFO:teuthology.orchestra.run.smithi152.stdout: NetworkManager-team-1:1.46.0-1.el9.x86_64 2024-03-16T11:37:40.092 INFO:teuthology.orchestra.run.smithi152.stdout: NetworkManager-tui-1:1.46.0-1.el9.x86_64 2024-03-16T11:37:40.092 INFO:teuthology.orchestra.run.smithi152.stdout: acl-2.3.1-4.el9.x86_64 2024-03-16T11:37:40.092 INFO:teuthology.orchestra.run.smithi152.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-03-16T11:37:40.092 INFO:teuthology.orchestra.run.smithi152.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-03-16T11:37:40.092 INFO:teuthology.orchestra.run.smithi152.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-03-16T11:37:40.092 INFO:teuthology.orchestra.run.smithi152.stdout: libacl-2.3.1-4.el9.x86_64 2024-03-16T11:37:40.092 INFO:teuthology.orchestra.run.smithi152.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-03-16T11:37:40.092 INFO:teuthology.orchestra.run.smithi152.stdout: libblkid-2.37.4-18.el9.x86_64 2024-03-16T11:37:40.092 INFO:teuthology.orchestra.run.smithi152.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-03-16T11:37:40.093 INFO:teuthology.orchestra.run.smithi152.stdout: libmount-2.37.4-18.el9.x86_64 2024-03-16T11:37:40.093 INFO:teuthology.orchestra.run.smithi152.stdout: librados2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T11:37:40.093 INFO:teuthology.orchestra.run.smithi152.stdout: librbd1-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T11:37:40.093 INFO:teuthology.orchestra.run.smithi152.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-03-16T11:37:40.093 INFO:teuthology.orchestra.run.smithi152.stdout: libuuid-2.37.4-18.el9.x86_64 2024-03-16T11:37:40.093 INFO:teuthology.orchestra.run.smithi152.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-03-16T11:37:40.093 INFO:teuthology.orchestra.run.smithi152.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-03-16T11:37:40.093 INFO:teuthology.orchestra.run.smithi152.stdout: util-linux-2.37.4-18.el9.x86_64 2024-03-16T11:37:40.093 INFO:teuthology.orchestra.run.smithi152.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-03-16T11:37:40.093 INFO:teuthology.orchestra.run.smithi152.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-03-16T11:37:40.093 INFO:teuthology.orchestra.run.smithi152.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-03-16T11:37:40.093 INFO:teuthology.orchestra.run.smithi152.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-03-16T11:37:40.093 INFO:teuthology.orchestra.run.smithi152.stdout:Installed: 2024-03-16T11:37:40.093 INFO:teuthology.orchestra.run.smithi152.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-16T11:37:40.094 INFO:teuthology.orchestra.run.smithi152.stdout: automake-1.16.2-8.el9.noarch 2024-03-16T11:37:40.094 INFO:teuthology.orchestra.run.smithi152.stdout: bison-3.7.4-5.el9.x86_64 2024-03-16T11:37:40.094 INFO:teuthology.orchestra.run.smithi152.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-16T11:37:40.094 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T11:37:40.094 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-base-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T11:37:40.094 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T11:37:40.094 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T11:37:40.094 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-grafana-dashboards-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-16T11:37:40.094 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-immutable-object-cache-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T11:37:40.094 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mds-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T11:37:40.094 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mgr-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T11:37:40.094 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mgr-cephadm-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-16T11:37:40.094 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mgr-dashboard-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-16T11:37:40.095 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-16T11:37:40.095 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mgr-modules-core-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-16T11:37:40.095 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mgr-rook-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-16T11:37:40.095 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mon-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T11:37:40.095 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-osd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T11:37:40.095 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-prometheus-alerts-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-16T11:37:40.095 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-radosgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T11:37:40.095 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-selinux-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T11:37:40.095 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-test-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T11:37:40.095 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-volume-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-16T11:37:40.095 INFO:teuthology.orchestra.run.smithi152.stdout: cephadm-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-16T11:37:40.095 INFO:teuthology.orchestra.run.smithi152.stdout: cephfs-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T11:37:40.095 INFO:teuthology.orchestra.run.smithi152.stdout: cephfs-shell-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T11:37:40.095 INFO:teuthology.orchestra.run.smithi152.stdout: cephfs-top-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-16T11:37:40.095 INFO:teuthology.orchestra.run.smithi152.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-03-16T11:37:40.095 INFO:teuthology.orchestra.run.smithi152.stdout: flex-2.6.4-9.el9.x86_64 2024-03-16T11:37:40.096 INFO:teuthology.orchestra.run.smithi152.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-16T11:37:40.096 INFO:teuthology.orchestra.run.smithi152.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-16T11:37:40.096 INFO:teuthology.orchestra.run.smithi152.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-16T11:37:40.096 INFO:teuthology.orchestra.run.smithi152.stdout: fuse-2.9.9-15.el9.x86_64 2024-03-16T11:37:40.096 INFO:teuthology.orchestra.run.smithi152.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-16T11:37:40.096 INFO:teuthology.orchestra.run.smithi152.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-16T11:37:40.096 INFO:teuthology.orchestra.run.smithi152.stdout: jq-1.6-16.el9.x86_64 2024-03-16T11:37:40.096 INFO:teuthology.orchestra.run.smithi152.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-03-16T11:37:40.096 INFO:teuthology.orchestra.run.smithi152.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-16T11:37:40.096 INFO:teuthology.orchestra.run.smithi152.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-16T11:37:40.096 INFO:teuthology.orchestra.run.smithi152.stdout: libcephfs-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T11:37:40.096 INFO:teuthology.orchestra.run.smithi152.stdout: libcephfs2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T11:37:40.096 INFO:teuthology.orchestra.run.smithi152.stdout: libcephsqlite-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T11:37:40.096 INFO:teuthology.orchestra.run.smithi152.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-16T11:37:40.096 INFO:teuthology.orchestra.run.smithi152.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-16T11:37:40.097 INFO:teuthology.orchestra.run.smithi152.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-16T11:37:40.097 INFO:teuthology.orchestra.run.smithi152.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-16T11:37:40.097 INFO:teuthology.orchestra.run.smithi152.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-16T11:37:40.097 INFO:teuthology.orchestra.run.smithi152.stdout: librados-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T11:37:40.097 INFO:teuthology.orchestra.run.smithi152.stdout: libradosstriper1-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T11:37:40.097 INFO:teuthology.orchestra.run.smithi152.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-16T11:37:40.097 INFO:teuthology.orchestra.run.smithi152.stdout: librgw2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T11:37:40.097 INFO:teuthology.orchestra.run.smithi152.stdout: libtool-2.4.6-45.el9.x86_64 2024-03-16T11:37:40.097 INFO:teuthology.orchestra.run.smithi152.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-16T11:37:40.097 INFO:teuthology.orchestra.run.smithi152.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-03-16T11:37:40.097 INFO:teuthology.orchestra.run.smithi152.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-03-16T11:37:40.097 INFO:teuthology.orchestra.run.smithi152.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-16T11:37:40.097 INFO:teuthology.orchestra.run.smithi152.stdout: lua-5.4.4-4.el9.x86_64 2024-03-16T11:37:40.097 INFO:teuthology.orchestra.run.smithi152.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-16T11:37:40.097 INFO:teuthology.orchestra.run.smithi152.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-16T11:37:40.098 INFO:teuthology.orchestra.run.smithi152.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-16T11:37:40.098 INFO:teuthology.orchestra.run.smithi152.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-16T11:37:40.098 INFO:teuthology.orchestra.run.smithi152.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-16T11:37:40.098 INFO:teuthology.orchestra.run.smithi152.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-03-16T11:37:40.098 INFO:teuthology.orchestra.run.smithi152.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-16T11:37:40.098 INFO:teuthology.orchestra.run.smithi152.stdout: postgresql-13.11-1.el9.x86_64 2024-03-16T11:37:40.098 INFO:teuthology.orchestra.run.smithi152.stdout: postgresql-contrib-13.11-1.el9.x86_64 2024-03-16T11:37:40.098 INFO:teuthology.orchestra.run.smithi152.stdout: postgresql-private-libs-13.11-1.el9.x86_64 2024-03-16T11:37:40.098 INFO:teuthology.orchestra.run.smithi152.stdout: postgresql-server-13.11-1.el9.x86_64 2024-03-16T11:37:40.098 INFO:teuthology.orchestra.run.smithi152.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-16T11:37:40.098 INFO:teuthology.orchestra.run.smithi152.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-16T11:37:40.098 INFO:teuthology.orchestra.run.smithi152.stdout: python3-asyncssh-2.13.2-4.el9.noarch 2024-03-16T11:37:40.098 INFO:teuthology.orchestra.run.smithi152.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-03-16T11:37:40.098 INFO:teuthology.orchestra.run.smithi152.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-16T11:37:40.099 INFO:teuthology.orchestra.run.smithi152.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-16T11:37:40.099 INFO:teuthology.orchestra.run.smithi152.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-16T11:37:40.099 INFO:teuthology.orchestra.run.smithi152.stdout: python3-ceph-argparse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T11:37:40.099 INFO:teuthology.orchestra.run.smithi152.stdout: python3-ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T11:37:40.099 INFO:teuthology.orchestra.run.smithi152.stdout: python3-cephfs-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T11:37:40.099 INFO:teuthology.orchestra.run.smithi152.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-16T11:37:40.099 INFO:teuthology.orchestra.run.smithi152.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-16T11:37:40.099 INFO:teuthology.orchestra.run.smithi152.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-16T11:37:40.099 INFO:teuthology.orchestra.run.smithi152.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-16T11:37:40.099 INFO:teuthology.orchestra.run.smithi152.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-03-16T11:37:40.099 INFO:teuthology.orchestra.run.smithi152.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-03-16T11:37:40.099 INFO:teuthology.orchestra.run.smithi152.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-16T11:37:40.099 INFO:teuthology.orchestra.run.smithi152.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-16T11:37:40.099 INFO:teuthology.orchestra.run.smithi152.stdout: python3-google-auth-1:2.28.0-1.el9.noarch 2024-03-16T11:37:40.099 INFO:teuthology.orchestra.run.smithi152.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-16T11:37:40.100 INFO:teuthology.orchestra.run.smithi152.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-16T11:37:40.100 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-16T11:37:40.100 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-16T11:37:40.100 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-16T11:37:40.100 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-16T11:37:40.100 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-16T11:37:40.100 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-16T11:37:40.100 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-16T11:37:40.100 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-16T11:37:40.100 INFO:teuthology.orchestra.run.smithi152.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-16T11:37:40.100 INFO:teuthology.orchestra.run.smithi152.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-16T11:37:40.100 INFO:teuthology.orchestra.run.smithi152.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-16T11:37:40.100 INFO:teuthology.orchestra.run.smithi152.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-16T11:37:40.100 INFO:teuthology.orchestra.run.smithi152.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-16T11:37:40.100 INFO:teuthology.orchestra.run.smithi152.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-16T11:37:40.101 INFO:teuthology.orchestra.run.smithi152.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-16T11:37:40.101 INFO:teuthology.orchestra.run.smithi152.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-16T11:37:40.101 INFO:teuthology.orchestra.run.smithi152.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-16T11:37:40.101 INFO:teuthology.orchestra.run.smithi152.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-16T11:37:40.101 INFO:teuthology.orchestra.run.smithi152.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-16T11:37:40.101 INFO:teuthology.orchestra.run.smithi152.stdout: python3-ply-3.11-14.el9.noarch 2024-03-16T11:37:40.101 INFO:teuthology.orchestra.run.smithi152.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-16T11:37:40.101 INFO:teuthology.orchestra.run.smithi152.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-16T11:37:40.101 INFO:teuthology.orchestra.run.smithi152.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-16T11:37:40.101 INFO:teuthology.orchestra.run.smithi152.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-16T11:37:40.101 INFO:teuthology.orchestra.run.smithi152.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-16T11:37:40.101 INFO:teuthology.orchestra.run.smithi152.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-16T11:37:40.101 INFO:teuthology.orchestra.run.smithi152.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-16T11:37:40.101 INFO:teuthology.orchestra.run.smithi152.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-03-16T11:37:40.102 INFO:teuthology.orchestra.run.smithi152.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-16T11:37:40.102 INFO:teuthology.orchestra.run.smithi152.stdout: python3-rados-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T11:37:40.102 INFO:teuthology.orchestra.run.smithi152.stdout: python3-rbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T11:37:40.102 INFO:teuthology.orchestra.run.smithi152.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-16T11:37:40.102 INFO:teuthology.orchestra.run.smithi152.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-16T11:37:40.102 INFO:teuthology.orchestra.run.smithi152.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-16T11:37:40.102 INFO:teuthology.orchestra.run.smithi152.stdout: python3-rgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T11:37:40.102 INFO:teuthology.orchestra.run.smithi152.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-16T11:37:40.102 INFO:teuthology.orchestra.run.smithi152.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-16T11:37:40.102 INFO:teuthology.orchestra.run.smithi152.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-16T11:37:40.102 INFO:teuthology.orchestra.run.smithi152.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-16T11:37:40.102 INFO:teuthology.orchestra.run.smithi152.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-16T11:37:40.102 INFO:teuthology.orchestra.run.smithi152.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-16T11:37:40.103 INFO:teuthology.orchestra.run.smithi152.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-16T11:37:40.103 INFO:teuthology.orchestra.run.smithi152.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-03-16T11:37:40.103 INFO:teuthology.orchestra.run.smithi152.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-16T11:37:40.103 INFO:teuthology.orchestra.run.smithi152.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-16T11:37:40.103 INFO:teuthology.orchestra.run.smithi152.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-16T11:37:40.103 INFO:teuthology.orchestra.run.smithi152.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-16T11:37:40.103 INFO:teuthology.orchestra.run.smithi152.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-16T11:37:40.103 INFO:teuthology.orchestra.run.smithi152.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-16T11:37:40.103 INFO:teuthology.orchestra.run.smithi152.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-16T11:37:40.103 INFO:teuthology.orchestra.run.smithi152.stdout: rbd-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T11:37:40.103 INFO:teuthology.orchestra.run.smithi152.stdout: rbd-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T11:37:40.103 INFO:teuthology.orchestra.run.smithi152.stdout: rbd-nbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T11:37:40.103 INFO:teuthology.orchestra.run.smithi152.stdout: re2-1:20211101-3.el9.x86_64 2024-03-16T11:37:40.103 INFO:teuthology.orchestra.run.smithi152.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-16T11:37:40.103 INFO:teuthology.orchestra.run.smithi152.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-16T11:37:40.104 INFO:teuthology.orchestra.run.smithi152.stdout: uuid-1.6.2-55.el9.x86_64 2024-03-16T11:37:40.104 INFO:teuthology.orchestra.run.smithi152.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-03-16T11:37:40.104 INFO:teuthology.orchestra.run.smithi152.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-16T11:37:40.104 INFO:teuthology.orchestra.run.smithi152.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-03-16T11:37:40.104 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-16T11:37:40.104 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-03-16T11:37:40.561 DEBUG:teuthology.parallel:result is None 2024-03-16T11:37:58.121 INFO:teuthology.orchestra.run.smithi029.stdout:(96/174): elfutils-libelf-devel-0.190-2.el9.x86 235 B/s | 21 kB 01:33 2024-03-16T11:37:58.138 INFO:teuthology.orchestra.run.smithi029.stdout:(97/174): libuuid-devel-2.37.4-18.el9.x86_64.rp 279 B/s | 25 kB 01:33 2024-03-16T11:37:58.364 INFO:teuthology.orchestra.run.smithi029.stdout:(98/174): python3-jinja2-2.11.3-5.el9.noarch.rp 1.1 MB/s | 248 kB 00:00 2024-03-16T11:37:58.397 INFO:teuthology.orchestra.run.smithi029.stdout:(99/174): python3-devel-3.9.17-2.el9.x86_64.rpm 959 kB/s | 245 kB 00:00 2024-03-16T11:37:58.648 INFO:teuthology.orchestra.run.smithi029.stdout:(100/174): qatlib-23.11.0-1.el9.x86_64.rpm 762 kB/s | 216 kB 00:00 2024-03-16T11:37:58.765 INFO:teuthology.orchestra.run.smithi029.stdout:(101/174): qatzip-libs-1.1.2-2.el9.x86_64.rpm 381 kB/s | 44 kB 00:00 2024-03-16T11:37:58.890 INFO:teuthology.orchestra.run.smithi029.stdout:(102/174): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 748 kB/s | 94 kB 00:00 2024-03-16T11:37:58.924 INFO:teuthology.orchestra.run.smithi029.stdout:(103/174): qatlib-service-23.11.0-1.el9.x86_64. 66 kB/s | 35 kB 00:00 2024-03-16T11:37:58.983 INFO:teuthology.orchestra.run.smithi029.stdout:(104/174): lua-devel-5.4.4-4.el9.x86_64.rpm 241 kB/s | 22 kB 00:00 2024-03-16T11:37:59.125 INFO:teuthology.orchestra.run.smithi029.stdout:(105/174): openssl-devel-3.0.7-27.el9.x86_64.rp 45 kB/s | 4.1 MB 01:34 2024-03-16T11:37:59.293 INFO:teuthology.orchestra.run.smithi029.stdout:(106/174): abseil-cpp-20211102.0-3.el9.x86_64.r 1.7 MB/s | 548 kB 00:00 2024-03-16T11:37:59.326 INFO:teuthology.orchestra.run.smithi029.stdout:(107/174): gperftools-libs-2.9.1-2.el9.x86_64.r 1.5 MB/s | 309 kB 00:00 2024-03-16T11:37:59.352 INFO:teuthology.orchestra.run.smithi029.stdout:(108/174): grpc-data-1.46.7-10.el9.noarch.rpm 332 kB/s | 19 kB 00:00 2024-03-16T11:37:59.394 INFO:teuthology.orchestra.run.smithi029.stdout:(109/174): protobuf-compiler-3.14.0-13.el9.x86_ 1.8 MB/s | 863 kB 00:00 2024-03-16T11:37:59.545 INFO:teuthology.orchestra.run.smithi029.stdout:(110/174): libarrow-9.0.0-10.el9.x86_64.rpm 20 MB/s | 4.4 MB 00:00 2024-03-16T11:37:59.572 INFO:teuthology.orchestra.run.smithi029.stdout:(111/174): libarrow-doc-9.0.0-10.el9.noarch.rpm 119 kB/s | 26 kB 00:00 2024-03-16T11:37:59.597 INFO:teuthology.orchestra.run.smithi029.stdout:(112/174): liboath-2.6.7-2.el9.x86_64.rpm 242 kB/s | 49 kB 00:00 2024-03-16T11:37:59.622 INFO:teuthology.orchestra.run.smithi029.stdout:(113/174): libunwind-1.6.2-1.el9.x86_64.rpm 873 kB/s | 67 kB 00:00 2024-03-16T11:37:59.656 INFO:teuthology.orchestra.run.smithi029.stdout:(114/174): luarocks-3.9.2-1.el9.noarch.rpm 2.5 MB/s | 151 kB 00:00 2024-03-16T11:37:59.699 INFO:teuthology.orchestra.run.smithi029.stdout:(115/174): parquet-libs-9.0.0-10.el9.x86_64.rpm 8.1 MB/s | 838 kB 00:00 2024-03-16T11:37:59.732 INFO:teuthology.orchestra.run.smithi029.stdout:(116/174): python3-asyncssh-2.13.2-4.el9.noarch 4.9 MB/s | 548 kB 00:00 2024-03-16T11:37:59.758 INFO:teuthology.orchestra.run.smithi029.stdout:(117/174): python3-bcrypt-3.2.2-1.el9.x86_64.rp 439 kB/s | 43 kB 00:00 2024-03-16T11:37:59.783 INFO:teuthology.orchestra.run.smithi029.stdout:(118/174): python3-cachetools-4.2.4-1.el9.noarc 638 kB/s | 32 kB 00:00 2024-03-16T11:37:59.808 INFO:teuthology.orchestra.run.smithi029.stdout:(119/174): python3-certifi-2021.10.8-2.el9.noar 198 kB/s | 15 kB 00:00 2024-03-16T11:37:59.834 INFO:teuthology.orchestra.run.smithi029.stdout:(120/174): python3-cheroot-8.6.0-4.el9.noarch.r 2.2 MB/s | 172 kB 00:00 2024-03-16T11:37:59.868 INFO:teuthology.orchestra.run.smithi029.stdout:(121/174): python3-cherrypy-18.6.1-2.el9.noarch 6.0 MB/s | 358 kB 00:00 2024-03-16T11:37:59.902 INFO:teuthology.orchestra.run.smithi029.stdout:(122/174): python3-cmd2-2.4.2-2.el9.noarch.rpm 3.2 MB/s | 303 kB 00:00 2024-03-16T11:37:59.927 INFO:teuthology.orchestra.run.smithi029.stdout:(123/174): python3-colorama-0.4.4-12.el9.noarch 384 kB/s | 35 kB 00:00 2024-03-16T11:37:59.961 INFO:teuthology.orchestra.run.smithi029.stdout:(124/174): python3-google-auth-2.28.0-1.el9.noa 2.2 MB/s | 203 kB 00:00 2024-03-16T11:37:59.995 INFO:teuthology.orchestra.run.smithi029.stdout:(125/174): python3-grpcio-tools-1.46.7-10.el9.x 2.1 MB/s | 144 kB 00:00 2024-03-16T11:38:00.054 INFO:teuthology.orchestra.run.smithi029.stdout:(126/174): python3-grpcio-1.46.7-10.el9.x86_64. 16 MB/s | 2.0 MB 00:00 2024-03-16T11:38:00.079 INFO:teuthology.orchestra.run.smithi029.stdout:(127/174): python3-jaraco-8.2.1-3.el9.noarch.rp 91 kB/s | 11 kB 00:00 2024-03-16T11:38:00.106 INFO:teuthology.orchestra.run.smithi029.stdout:(128/174): python3-jaraco-classes-3.2.1-5.el9.n 164 kB/s | 18 kB 00:00 2024-03-16T11:38:00.131 INFO:teuthology.orchestra.run.smithi029.stdout:(129/174): python3-jaraco-collections-3.0.0-8.e 444 kB/s | 23 kB 00:00 2024-03-16T11:38:00.157 INFO:teuthology.orchestra.run.smithi029.stdout:(130/174): python3-jaraco-functools-3.5.0-2.el9 251 kB/s | 19 kB 00:00 2024-03-16T11:38:00.182 INFO:teuthology.orchestra.run.smithi029.stdout:(131/174): python3-jaraco-text-3.2.0-6.el9.noar 260 kB/s | 20 kB 00:00 2024-03-16T11:38:00.225 INFO:teuthology.orchestra.run.smithi029.stdout:(132/174): python3-kubernetes-26.1.0-2.el9.noar 15 MB/s | 1.0 MB 00:00 2024-03-16T11:38:00.250 INFO:teuthology.orchestra.run.smithi029.stdout:(133/174): python3-logutils-0.3.5-21.el9.noarch 500 kB/s | 46 kB 00:00 2024-03-16T11:38:00.275 INFO:teuthology.orchestra.run.smithi029.stdout:(134/174): python3-more-itertools-8.12.0-2.el9. 858 kB/s | 79 kB 00:00 2024-03-16T11:38:00.301 INFO:teuthology.orchestra.run.smithi029.stdout:(135/174): python3-natsort-7.1.1-5.el9.noarch.r 762 kB/s | 58 kB 00:00 2024-03-16T11:38:00.326 INFO:teuthology.orchestra.run.smithi029.stdout:(136/174): python3-pecan-1.4.2-3.el9.noarch.rpm 5.3 MB/s | 272 kB 00:00 2024-03-16T11:38:00.352 INFO:teuthology.orchestra.run.smithi029.stdout:(137/174): python3-portend-3.1.0-2.el9.noarch.r 216 kB/s | 16 kB 00:00 2024-03-16T11:38:00.377 INFO:teuthology.orchestra.run.smithi029.stdout:(138/174): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.2 MB/s | 90 kB 00:00 2024-03-16T11:38:00.403 INFO:teuthology.orchestra.run.smithi029.stdout:(139/174): python3-pyperclip-1.8.2-2.el9.noarch 561 kB/s | 28 kB 00:00 2024-03-16T11:38:00.428 INFO:teuthology.orchestra.run.smithi029.stdout:(140/174): python3-repoze-lru-0.7-16.el9.noarch 407 kB/s | 31 kB 00:00 2024-03-16T11:38:00.462 INFO:teuthology.orchestra.run.smithi029.stdout:(141/174): python3-routes-2.5.1-5.el9.noarch.rp 2.2 MB/s | 188 kB 00:00 2024-03-16T11:38:00.487 INFO:teuthology.orchestra.run.smithi029.stdout:(142/174): python3-rsa-4.9-2.el9.noarch.rpm 1.0 MB/s | 59 kB 00:00 2024-03-16T11:38:00.513 INFO:teuthology.orchestra.run.smithi029.stdout:(143/174): python3-tempora-5.0.0-2.el9.noarch.r 427 kB/s | 36 kB 00:00 2024-03-16T11:38:00.538 INFO:teuthology.orchestra.run.smithi029.stdout:(144/174): python3-typing-extensions-4.4.0-2.el 674 kB/s | 51 kB 00:00 2024-03-16T11:38:00.564 INFO:teuthology.orchestra.run.smithi029.stdout:(145/174): python3-webob-1.8.7-6.el9.noarch.rpm 4.5 MB/s | 230 kB 00:00 2024-03-16T11:38:00.589 INFO:teuthology.orchestra.run.smithi029.stdout:(146/174): python3-websocket-client-1.2.3-2.el9 1.2 MB/s | 90 kB 00:00 2024-03-16T11:38:00.633 INFO:teuthology.orchestra.run.smithi029.stdout:(147/174): python3-werkzeug-2.0.3-3.el9.1.noarc 4.5 MB/s | 427 kB 00:00 2024-03-16T11:38:00.666 INFO:teuthology.orchestra.run.smithi029.stdout:(148/174): python3-zc-lockfile-2.0-10.el9.noarc 262 kB/s | 20 kB 00:00 2024-03-16T11:38:00.700 INFO:teuthology.orchestra.run.smithi029.stdout:(149/174): re2-20211101-3.el9.x86_64.rpm 1.7 MB/s | 192 kB 00:00 2024-03-16T11:38:00.751 INFO:teuthology.orchestra.run.smithi029.stdout:(150/174): thrift-0.15.0-2.el9.x86_64.rpm 13 MB/s | 1.6 MB 00:00 2024-03-16T11:38:00.793 INFO:teuthology.orchestra.run.smithi029.stdout:(151/174): xfsdump-3.1.12-4.el9.x86_64.rpm 7.5 MB/s | 321 kB 00:00 2024-03-16T11:38:01.202 INFO:teuthology.orchestra.run.smithi029.stdout:(152/174): librbd1-19.0.0-2162.g3d78f5ba.el9.x8 6.3 MB/s | 3.1 MB 00:00 2024-03-16T11:38:01.294 INFO:teuthology.orchestra.run.smithi029.stdout:(153/174): NetworkManager-1.46.0-1.el9.x86_64.r 5.1 MB/s | 2.3 MB 00:00 2024-03-16T11:38:01.353 INFO:teuthology.orchestra.run.smithi029.stdout:(154/174): NetworkManager-team-1.46.0-1.el9.x86 726 kB/s | 42 kB 00:00 2024-03-16T11:38:01.420 INFO:teuthology.orchestra.run.smithi029.stdout:(155/174): NetworkManager-tui-1.46.0-1.el9.x86_ 3.6 MB/s | 248 kB 00:00 2024-03-16T11:38:01.479 INFO:teuthology.orchestra.run.smithi029.stdout:(156/174): acl-2.3.1-4.el9.x86_64.rpm 1.2 MB/s | 71 kB 00:00 2024-03-16T11:38:01.538 INFO:teuthology.orchestra.run.smithi029.stdout:(157/174): elfutils-debuginfod-client-0.190-2.e 622 kB/s | 36 kB 00:00 2024-03-16T11:38:01.630 INFO:teuthology.orchestra.run.smithi029.stdout:(158/174): NetworkManager-libnm-1.46.0-1.el9.x8 4.3 MB/s | 1.8 MB 00:00 2024-03-16T11:38:01.655 INFO:teuthology.orchestra.run.smithi029.stdout:(159/174): elfutils-libelf-0.190-2.el9.x86_64.r 1.6 MB/s | 192 kB 00:00 2024-03-16T11:38:01.689 INFO:teuthology.orchestra.run.smithi029.stdout:(160/174): elfutils-libs-0.190-2.el9.x86_64.rpm 4.2 MB/s | 256 kB 00:00 2024-03-16T11:38:01.715 INFO:teuthology.orchestra.run.smithi029.stdout:(161/174): libacl-2.3.1-4.el9.x86_64.rpm 390 kB/s | 23 kB 00:00 2024-03-16T11:38:01.790 INFO:teuthology.orchestra.run.smithi029.stdout:(162/174): librados2-19.0.0-2162.g3d78f5ba.el9. 3.1 MB/s | 3.4 MB 00:01 2024-03-16T11:38:01.826 INFO:teuthology.orchestra.run.smithi029.stdout:(163/174): libblkid-2.37.4-18.el9.x86_64.rpm 796 kB/s | 108 kB 00:00 2024-03-16T11:38:01.851 INFO:teuthology.orchestra.run.smithi029.stdout:(164/174): libfdisk-2.37.4-18.el9.x86_64.rpm 1.1 MB/s | 155 kB 00:00 2024-03-16T11:38:01.877 INFO:teuthology.orchestra.run.smithi029.stdout:(165/174): libsmartcols-2.37.4-18.el9.x86_64.rp 1.2 MB/s | 64 kB 00:00 2024-03-16T11:38:01.903 INFO:teuthology.orchestra.run.smithi029.stdout:(166/174): libuuid-2.37.4-18.el9.x86_64.rpm 565 kB/s | 29 kB 00:00 2024-03-16T11:38:01.995 INFO:teuthology.orchestra.run.smithi029.stdout:(167/174): openssl-3.0.7-27.el9.x86_64.rpm 10 MB/s | 1.2 MB 00:00 2024-03-16T11:38:02.020 INFO:teuthology.orchestra.run.smithi029.stdout:(168/174): libmount-2.37.4-18.el9.x86_64.rpm 591 kB/s | 136 kB 00:00 2024-03-16T11:38:02.112 INFO:teuthology.orchestra.run.smithi029.stdout:(169/174): openssl-libs-3.0.7-27.el9.x86_64.rpm 10 MB/s | 2.2 MB 00:00 2024-03-16T11:38:02.171 INFO:teuthology.orchestra.run.smithi029.stdout:(170/174): util-linux-user-2.37.4-18.el9.x86_64 531 kB/s | 31 kB 00:00 2024-03-16T11:38:02.205 INFO:teuthology.orchestra.run.smithi029.stdout:(171/174): util-linux-core-2.37.4-18.el9.x86_64 2.5 MB/s | 465 kB 00:00 2024-03-16T11:38:02.272 INFO:teuthology.orchestra.run.smithi029.stdout:(172/174): util-linux-2.37.4-18.el9.x86_64.rpm 8.2 MB/s | 2.3 MB 00:00 2024-03-16T11:38:02.322 INFO:teuthology.orchestra.run.smithi029.stdout:(173/174): xfsprogs-6.3.0-1.el9.x86_64.rpm 7.1 MB/s | 1.1 MB 00:00 2024-03-16T11:38:02.348 INFO:teuthology.orchestra.run.smithi029.stdout:(174/174): libacl-devel-2.3.1-4.el9.x86_64.rpm 522 kB/s | 74 kB 00:00 2024-03-16T11:38:02.353 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2024-03-16T11:38:02.353 INFO:teuthology.orchestra.run.smithi029.stdout:Total 2.1 MB/s | 224 MB 01:48 2024-03-16T11:38:02.811 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2024-03-16T11:38:03.145 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2024-03-16T11:38:03.145 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2024-03-16T11:38:05.288 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2024-03-16T11:38:05.288 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2024-03-16T11:38:12.309 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2024-03-16T11:38:12.448 INFO:teuthology.orchestra.run.smithi029.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/198 2024-03-16T11:38:12.484 INFO:teuthology.orchestra.run.smithi029.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/198 2024-03-16T11:38:12.522 INFO:teuthology.orchestra.run.smithi029.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-16T11:38:12.530 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-16T11:38:12.580 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 4/198 2024-03-16T11:38:12.635 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/198 2024-03-16T11:38:12.673 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/198 2024-03-16T11:38:12.716 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 7/198 2024-03-16T11:38:12.769 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 8/198 2024-03-16T11:38:13.124 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 9/198 2024-03-16T11:38:13.271 INFO:teuthology.orchestra.run.smithi029.stdout: Upgrading : librados2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 10/198 2024-03-16T11:38:13.395 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: librados2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 10/198 2024-03-16T11:38:13.432 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libcephfs2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 11/198 2024-03-16T11:38:13.524 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: libcephfs2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 11/198 2024-03-16T11:38:13.568 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-rados-2:19.0.0-2162.g3d78f5ba.el9.x86_64 12/198 2024-03-16T11:38:13.625 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : postgresql-private-libs-13.11-1.el9.x86_64 13/198 2024-03-16T11:38:13.813 INFO:teuthology.orchestra.run.smithi029.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 14/198 2024-03-16T11:38:13.873 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : postgresql-13.11-1.el9.x86_64 15/198 2024-03-16T11:38:13.901 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libcephsqlite-2:19.0.0-2162.g3d78f5ba.el9.x86_64 16/198 2024-03-16T11:38:14.007 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2162.g3d78f5ba.el9.x86_64 16/198 2024-03-16T11:38:14.038 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libradosstriper1-2:19.0.0-2162.g3d78f5ba.el9.x86 17/198 2024-03-16T11:38:14.120 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2162.g3d78f5ba.el9.x86 17/198 2024-03-16T11:38:14.168 INFO:teuthology.orchestra.run.smithi029.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/198 2024-03-16T11:38:14.199 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-16T11:38:14.214 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-16T11:38:14.416 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-16T11:38:14.450 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-16T11:38:14.491 INFO:teuthology.orchestra.run.smithi029.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/198 2024-03-16T11:38:14.806 INFO:teuthology.orchestra.run.smithi029.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/198 2024-03-16T11:38:14.910 INFO:teuthology.orchestra.run.smithi029.stdout: Upgrading : NetworkManager-libnm-1:1.46.0-1.el9.x86_64 23/198 2024-03-16T11:38:14.971 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: NetworkManager-libnm-1:1.46.0-1.el9.x86_64 23/198 2024-03-16T11:38:15.143 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: NetworkManager-1:1.46.0-1.el9.x86_64 24/198 2024-03-16T11:38:15.209 INFO:teuthology.orchestra.run.smithi029.stdout: Upgrading : NetworkManager-1:1.46.0-1.el9.x86_64 24/198 2024-03-16T11:38:15.441 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: NetworkManager-1:1.46.0-1.el9.x86_64 24/198 2024-03-16T11:38:15.551 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/198 2024-03-16T11:38:15.655 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/198 2024-03-16T11:38:15.691 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/198 2024-03-16T11:38:15.729 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/198 2024-03-16T11:38:15.773 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/198 2024-03-16T11:38:15.859 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/198 2024-03-16T11:38:15.921 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 31/198 2024-03-16T11:38:15.958 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/198 2024-03-16T11:38:16.033 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/198 2024-03-16T11:38:16.077 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/198 2024-03-16T11:38:16.121 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/198 2024-03-16T11:38:16.163 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/198 2024-03-16T11:38:16.199 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/198 2024-03-16T11:38:16.280 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/198 2024-03-16T11:38:16.322 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-ceph-common-2:19.0.0-2162.g3d78f5ba.el9. 39/198 2024-03-16T11:38:16.370 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-ceph-argparse-2:19.0.0-2162.g3d78f5ba.el 40/198 2024-03-16T11:38:16.403 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-cephfs-2:19.0.0-2162.g3d78f5ba.el9.x86_6 41/198 2024-03-16T11:38:16.476 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/198 2024-03-16T11:38:16.529 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/198 2024-03-16T11:38:16.615 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/198 2024-03-16T11:38:16.769 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/198 2024-03-16T11:38:16.827 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/198 2024-03-16T11:38:17.389 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/198 2024-03-16T11:38:17.478 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/198 2024-03-16T11:38:17.576 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/198 2024-03-16T11:38:17.633 INFO:teuthology.orchestra.run.smithi029.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-16T11:38:17.646 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-16T11:38:17.682 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 51/198 2024-03-16T11:38:17.731 INFO:teuthology.orchestra.run.smithi029.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 52/198 2024-03-16T11:38:17.770 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : librados-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_6 53/198 2024-03-16T11:38:17.810 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 54/198 2024-03-16T11:38:18.086 INFO:teuthology.orchestra.run.smithi029.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 55/198 2024-03-16T11:38:18.301 INFO:teuthology.orchestra.run.smithi029.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 56/198 2024-03-16T11:38:18.388 INFO:teuthology.orchestra.run.smithi029.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 57/198 2024-03-16T11:38:18.435 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 58/198 2024-03-16T11:38:18.579 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 59/198 2024-03-16T11:38:18.632 INFO:teuthology.orchestra.run.smithi029.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 60/198 2024-03-16T11:38:18.698 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 61/198 2024-03-16T11:38:18.772 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 62/198 2024-03-16T11:38:18.811 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 63/198 2024-03-16T11:38:18.848 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 64/198 2024-03-16T11:38:18.906 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 65/198 2024-03-16T11:38:18.949 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 66/198 2024-03-16T11:38:18.993 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 67/198 2024-03-16T11:38:19.036 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 68/198 2024-03-16T11:38:19.187 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 69/198 2024-03-16T11:38:19.237 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 70/198 2024-03-16T11:38:19.275 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 71/198 2024-03-16T11:38:19.318 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 72/198 2024-03-16T11:38:19.424 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 73/198 2024-03-16T11:38:19.462 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-google-auth-1:2.28.0-1.el9.noarch 74/198 2024-03-16T11:38:19.503 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 75/198 2024-03-16T11:38:19.537 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 76/198 2024-03-16T11:38:19.562 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 77/198 2024-03-16T11:38:19.659 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 78/198 2024-03-16T11:38:19.798 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 79/198 2024-03-16T11:38:19.934 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 80/198 2024-03-16T11:38:19.980 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 81/198 2024-03-16T11:38:20.039 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-ply-3.11-14.el9.noarch 82/198 2024-03-16T11:38:20.103 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 83/198 2024-03-16T11:38:20.305 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 84/198 2024-03-16T11:38:20.363 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 85/198 2024-03-16T11:38:20.507 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/198 2024-03-16T11:38:20.547 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-asyncssh-2.13.2-4.el9.noarch 87/198 2024-03-16T11:38:20.588 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 88/198 2024-03-16T11:38:20.619 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : fuse-2.9.9-15.el9.x86_64 89/198 2024-03-16T11:38:20.656 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 90/198 2024-03-16T11:38:20.691 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 91/198 2024-03-16T11:38:20.731 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : uuid-1.6.2-55.el9.x86_64 92/198 2024-03-16T11:38:20.773 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 93/198 2024-03-16T11:38:20.811 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 94/198 2024-03-16T11:38:20.890 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 95/198 2024-03-16T11:38:20.923 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 96/198 2024-03-16T11:38:20.957 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 97/198 2024-03-16T11:38:20.999 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 98/198 2024-03-16T11:38:21.027 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 99/198 2024-03-16T11:38:21.121 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 100/198 2024-03-16T11:38:21.211 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 101/198 2024-03-16T11:38:21.261 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 102/198 2024-03-16T11:38:21.301 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 103/198 2024-03-16T11:38:21.349 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 104/198 2024-03-16T11:38:21.459 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-packaging-20.9-5.el9.noarch 105/198 2024-03-16T11:38:21.516 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 106/198 2024-03-16T11:38:22.276 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 107/198 2024-03-16T11:38:22.342 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 108/198 2024-03-16T11:38:22.376 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 109/198 2024-03-16T11:38:22.503 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 110/198 2024-03-16T11:38:22.806 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 111/198 2024-03-16T11:38:23.031 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 112/198 2024-03-16T11:38:23.215 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : openblas-0.3.21-2.el9.x86_64 113/198 2024-03-16T11:38:23.245 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 114/198 2024-03-16T11:38:23.332 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 115/198 2024-03-16T11:38:24.020 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 116/198 2024-03-16T11:38:24.155 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 117/198 2024-03-16T11:38:25.450 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 118/198 2024-03-16T11:38:25.553 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 119/198 2024-03-16T11:38:25.586 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 120/198 2024-03-16T11:38:25.617 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : jq-1.6-16.el9.x86_64 121/198 2024-03-16T11:38:25.654 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 122/198 2024-03-16T11:38:25.983 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 123/198 2024-03-16T11:38:26.010 INFO:teuthology.orchestra.run.smithi029.stdout: Upgrading : librbd1-2:19.0.0-2162.g3d78f5ba.el9.x86_64 124/198 2024-03-16T11:38:26.102 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: librbd1-2:19.0.0-2162.g3d78f5ba.el9.x86_64 124/198 2024-03-16T11:38:26.126 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-rbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 125/198 2024-03-16T11:38:26.171 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 126/198 2024-03-16T11:38:26.657 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 127/198 2024-03-16T11:38:26.692 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : librgw2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 128/198 2024-03-16T11:38:26.764 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: librgw2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 128/198 2024-03-16T11:38:26.795 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-rgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 129/198 2024-03-16T11:38:29.164 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 130/198 2024-03-16T11:38:29.206 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 130/198 2024-03-16T11:38:29.773 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 130/198 2024-03-16T11:38:29.822 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-base-2:19.0.0-2162.g3d78f5ba.el9.x86_64 131/198 2024-03-16T11:38:29.914 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-base-2:19.0.0-2162.g3d78f5ba.el9.x86_64 131/198 2024-03-16T11:38:29.914 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-16T11:38:29.914 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-16T11:38:29.914 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-16T11:38:29.958 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-selinux-2:19.0.0-2162.g3d78f5ba.el9.x86_64 132/198 2024-03-16T11:38:46.691 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2162.g3d78f5ba.el9.x86_64 132/198 2024-03-16T11:38:46.691 INFO:teuthology.orchestra.run.smithi029.stdout:skipping the directory /sys 2024-03-16T11:38:46.691 INFO:teuthology.orchestra.run.smithi029.stdout:skipping the directory /proc 2024-03-16T11:38:46.691 INFO:teuthology.orchestra.run.smithi029.stdout:skipping the directory /mnt 2024-03-16T11:38:46.691 INFO:teuthology.orchestra.run.smithi029.stdout:skipping the directory /var/tmp 2024-03-16T11:38:46.691 INFO:teuthology.orchestra.run.smithi029.stdout:skipping the directory /home 2024-03-16T11:38:46.691 INFO:teuthology.orchestra.run.smithi029.stdout:skipping the directory /root 2024-03-16T11:38:46.691 INFO:teuthology.orchestra.run.smithi029.stdout:skipping the directory /tmp 2024-03-16T11:38:46.691 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-16T11:38:46.883 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-volume-2:19.0.0-2162.g3d78f5ba.el9.noarch 133/198 2024-03-16T11:38:46.897 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-volume-2:19.0.0-2162.g3d78f5ba.el9.noarch 133/198 2024-03-16T11:38:46.898 INFO:teuthology.orchestra.run.smithi029.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-16T11:38:46.898 INFO:teuthology.orchestra.run.smithi029.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-16T11:38:46.898 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-16T11:38:48.430 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-osd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 134/198 2024-03-16T11:38:48.456 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-osd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 134/198 2024-03-16T11:38:48.456 INFO:teuthology.orchestra.run.smithi029.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-16T11:38:48.456 INFO:teuthology.orchestra.run.smithi029.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-16T11:38:48.456 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-16T11:38:48.456 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-16T11:38:48.456 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-16T11:38:48.691 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-mds-2:19.0.0-2162.g3d78f5ba.el9.x86_64 135/198 2024-03-16T11:38:48.713 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-mds-2:19.0.0-2162.g3d78f5ba.el9.x86_64 135/198 2024-03-16T11:38:48.714 INFO:teuthology.orchestra.run.smithi029.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-16T11:38:48.714 INFO:teuthology.orchestra.run.smithi029.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-16T11:38:48.714 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-16T11:38:48.714 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-16T11:38:48.714 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-16T11:38:49.184 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-mon-2:19.0.0-2162.g3d78f5ba.el9.x86_64 136/198 2024-03-16T11:38:49.206 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-mon-2:19.0.0-2162.g3d78f5ba.el9.x86_64 136/198 2024-03-16T11:38:49.206 INFO:teuthology.orchestra.run.smithi029.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-16T11:38:49.206 INFO:teuthology.orchestra.run.smithi029.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-16T11:38:49.207 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-16T11:38:49.207 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-16T11:38:49.207 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-16T11:38:49.304 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : automake-1.16.2-8.el9.noarch 137/198 2024-03-16T11:38:49.778 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: cephadm-2:19.0.0-2162.g3d78f5ba.el9.noarch 138/198 2024-03-16T11:38:49.816 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : cephadm-2:19.0.0-2162.g3d78f5ba.el9.noarch 138/198 2024-03-16T11:38:49.857 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2162.g3d78f5ba.e 139/198 2024-03-16T11:38:49.928 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2162.g3d78f5ba. 140/198 2024-03-16T11:38:49.957 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2162.g3d78f5ba.el9.noa 141/198 2024-03-16T11:38:50.204 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2162.g3d78f5ba.el9.noa 141/198 2024-03-16T11:38:50.242 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2162.g3d78f5ba.el9.n 142/198 2024-03-16T11:38:51.256 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2162.g3d78f5ba.el9.n 142/198 2024-03-16T11:38:51.286 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2162.g3d7 143/198 2024-03-16T11:38:51.393 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2162.g3d7 143/198 2024-03-16T11:38:51.566 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2162.g3d78f5ba.el 144/198 2024-03-16T11:38:51.615 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-mgr-2:19.0.0-2162.g3d78f5ba.el9.x86_64 145/198 2024-03-16T11:38:51.637 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2162.g3d78f5ba.el9.x86_64 145/198 2024-03-16T11:38:51.637 INFO:teuthology.orchestra.run.smithi029.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-16T11:38:51.638 INFO:teuthology.orchestra.run.smithi029.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-16T11:38:51.638 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-16T11:38:51.638 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-16T11:38:51.638 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-16T11:38:51.691 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-mgr-rook-2:19.0.0-2162.g3d78f5ba.el9.noarch 146/198 2024-03-16T11:38:51.702 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2162.g3d78f5ba.el9.noarch 146/198 2024-03-16T11:38:51.763 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-2:19.0.0-2162.g3d78f5ba.el9.x86_64 147/198 2024-03-16T11:38:52.685 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libtool-2.4.6-45.el9.x86_64 148/198 2024-03-16T11:38:52.714 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-radosgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 149/198 2024-03-16T11:38:52.736 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 149/198 2024-03-16T11:38:52.736 INFO:teuthology.orchestra.run.smithi029.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-16T11:38:52.736 INFO:teuthology.orchestra.run.smithi029.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-16T11:38:52.736 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-16T11:38:52.736 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-16T11:38:52.736 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-16T11:38:52.781 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2162.g3d78f 150/198 2024-03-16T11:38:52.803 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2162.g3d78f 150/198 2024-03-16T11:38:52.803 INFO:teuthology.orchestra.run.smithi029.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-16T11:38:52.803 INFO:teuthology.orchestra.run.smithi029.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-16T11:38:52.803 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-16T11:38:52.866 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : cephfs-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 151/198 2024-03-16T11:38:52.888 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 151/198 2024-03-16T11:38:52.888 INFO:teuthology.orchestra.run.smithi029.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-16T11:38:52.888 INFO:teuthology.orchestra.run.smithi029.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-03-16T11:38:52.888 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-16T11:38:52.888 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-16T11:38:52.888 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-16T11:38:53.209 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : rbd-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 152/198 2024-03-16T11:38:53.231 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 152/198 2024-03-16T11:38:53.231 INFO:teuthology.orchestra.run.smithi029.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-16T11:38:53.231 INFO:teuthology.orchestra.run.smithi029.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-16T11:38:53.231 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-16T11:38:53.231 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-16T11:38:53.231 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-16T11:38:57.945 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-test-2:19.0.0-2162.g3d78f5ba.el9.x86_64 153/198 2024-03-16T11:38:57.998 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : rbd-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 154/198 2024-03-16T11:38:58.028 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : rbd-nbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 155/198 2024-03-16T11:38:58.064 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 156/198 2024-03-16T11:38:58.254 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : cephfs-shell-2:19.0.0-2162.g3d78f5ba.el9.x86_64 157/198 2024-03-16T11:38:58.381 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : postgresql-contrib-13.11-1.el9.x86_64 158/198 2024-03-16T11:38:58.428 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 159/198 2024-03-16T11:38:58.504 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 160/198 2024-03-16T11:38:58.563 INFO:teuthology.orchestra.run.smithi029.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 161/198 2024-03-16T11:38:58.616 INFO:teuthology.orchestra.run.smithi029.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 162/198 2024-03-16T11:38:58.649 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libcephfs-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_ 163/198 2024-03-16T11:38:58.682 INFO:teuthology.orchestra.run.smithi029.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 164/198 2024-03-16T11:38:58.712 INFO:teuthology.orchestra.run.smithi029.stdout: Upgrading : NetworkManager-team-1:1.46.0-1.el9.x86_64 165/198 2024-03-16T11:38:58.793 INFO:teuthology.orchestra.run.smithi029.stdout: Upgrading : NetworkManager-tui-1:1.46.0-1.el9.x86_64 166/198 2024-03-16T11:38:58.869 INFO:teuthology.orchestra.run.smithi029.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 167/198 2024-03-16T11:38:58.906 INFO:teuthology.orchestra.run.smithi029.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 168/198 2024-03-16T11:38:59.520 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-16T11:38:59.572 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-16T11:38:59.591 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-16T11:39:00.471 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : cephfs-top-2:19.0.0-2162.g3d78f5ba.el9.noarch 170/198 2024-03-16T11:39:00.572 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 171/198 2024-03-16T11:39:00.614 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 172/198 2024-03-16T11:39:00.708 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : flex-2.6.4-9.el9.x86_64 173/198 2024-03-16T11:39:00.747 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : bison-3.7.4-5.el9.x86_64 174/198 2024-03-16T11:39:00.780 INFO:teuthology.orchestra.run.smithi029.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 175/198 2024-03-16T11:39:00.814 INFO:teuthology.orchestra.run.smithi029.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 176/198 2024-03-16T11:39:00.842 INFO:teuthology.orchestra.run.smithi029.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 177/198 2024-03-16T11:39:00.842 INFO:teuthology.orchestra.run.smithi029.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-16T11:39:00.939 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-16T11:39:00.979 INFO:teuthology.orchestra.run.smithi029.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 179/198 2024-03-16T11:39:00.979 INFO:teuthology.orchestra.run.smithi029.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-16T11:39:01.081 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-16T11:39:01.116 INFO:teuthology.orchestra.run.smithi029.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 181/198 2024-03-16T11:39:01.150 INFO:teuthology.orchestra.run.smithi029.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 182/198 2024-03-16T11:39:01.184 INFO:teuthology.orchestra.run.smithi029.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 183/198 2024-03-16T11:39:01.209 INFO:teuthology.orchestra.run.smithi029.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 184/198 2024-03-16T11:39:01.244 INFO:teuthology.orchestra.run.smithi029.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 185/198 2024-03-16T11:39:01.303 INFO:teuthology.orchestra.run.smithi029.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 186/198 2024-03-16T11:39:01.345 INFO:teuthology.orchestra.run.smithi029.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 187/198 2024-03-16T11:39:01.379 INFO:teuthology.orchestra.run.smithi029.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 188/198 2024-03-16T11:39:01.404 INFO:teuthology.orchestra.run.smithi029.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 189/198 2024-03-16T11:39:01.430 INFO:teuthology.orchestra.run.smithi029.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 190/198 2024-03-16T11:39:01.465 INFO:teuthology.orchestra.run.smithi029.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 191/198 2024-03-16T11:39:01.472 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-16T11:39:01.472 INFO:teuthology.orchestra.run.smithi029.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-16T11:39:01.660 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-16T11:39:01.661 INFO:teuthology.orchestra.run.smithi029.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-16T11:39:01.753 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-16T11:39:01.786 INFO:teuthology.orchestra.run.smithi029.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 194/198 2024-03-16T11:39:01.811 INFO:teuthology.orchestra.run.smithi029.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 195/198 2024-03-16T11:39:01.845 INFO:teuthology.orchestra.run.smithi029.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 196/198 2024-03-16T11:39:01.872 INFO:teuthology.orchestra.run.smithi029.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 197/198 2024-03-16T11:39:01.950 INFO:teuthology.orchestra.run.smithi029.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-16T11:39:10.591 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-16T11:39:10.591 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-2:19.0.0-2162.g3d78f5ba.el9.x86_64 1/198 2024-03-16T11:39:10.591 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-base-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2/198 2024-03-16T11:39:10.591 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 3/198 2024-03-16T11:39:10.592 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 4/198 2024-03-16T11:39:10.592 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2162.g3d78f 5/198 2024-03-16T11:39:10.592 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-mds-2:19.0.0-2162.g3d78f5ba.el9.x86_64 6/198 2024-03-16T11:39:10.592 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-mgr-2:19.0.0-2162.g3d78f5ba.el9.x86_64 7/198 2024-03-16T11:39:10.592 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-mon-2:19.0.0-2162.g3d78f5ba.el9.x86_64 8/198 2024-03-16T11:39:10.592 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-osd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 9/198 2024-03-16T11:39:10.592 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-radosgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 10/198 2024-03-16T11:39:10.592 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-selinux-2:19.0.0-2162.g3d78f5ba.el9.x86_64 11/198 2024-03-16T11:39:10.592 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-test-2:19.0.0-2162.g3d78f5ba.el9.x86_64 12/198 2024-03-16T11:39:10.592 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : cephfs-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 13/198 2024-03-16T11:39:10.592 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : cephfs-shell-2:19.0.0-2162.g3d78f5ba.el9.x86_64 14/198 2024-03-16T11:39:10.592 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libcephfs-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_ 15/198 2024-03-16T11:39:10.592 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libcephfs2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 16/198 2024-03-16T11:39:10.593 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libcephsqlite-2:19.0.0-2162.g3d78f5ba.el9.x86_64 17/198 2024-03-16T11:39:10.593 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : librados-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_6 18/198 2024-03-16T11:39:10.593 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libradosstriper1-2:19.0.0-2162.g3d78f5ba.el9.x86 19/198 2024-03-16T11:39:10.593 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : librgw2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 20/198 2024-03-16T11:39:10.593 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2162.g3d78f5ba.el 21/198 2024-03-16T11:39:10.593 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-ceph-common-2:19.0.0-2162.g3d78f5ba.el9. 22/198 2024-03-16T11:39:10.593 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-cephfs-2:19.0.0-2162.g3d78f5ba.el9.x86_6 23/198 2024-03-16T11:39:10.593 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-rados-2:19.0.0-2162.g3d78f5ba.el9.x86_64 24/198 2024-03-16T11:39:10.593 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-rbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 25/198 2024-03-16T11:39:10.593 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-rgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 26/198 2024-03-16T11:39:10.593 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : rbd-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 27/198 2024-03-16T11:39:10.593 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : rbd-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 28/198 2024-03-16T11:39:10.593 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : rbd-nbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 29/198 2024-03-16T11:39:10.593 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2162.g3d78f5ba. 30/198 2024-03-16T11:39:10.608 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2162.g3d78f5ba.el9.noa 31/198 2024-03-16T11:39:10.609 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2162.g3d78f5ba.el9.n 32/198 2024-03-16T11:39:10.609 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2162.g3d7 33/198 2024-03-16T11:39:10.609 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2162.g3d78f5ba.el 34/198 2024-03-16T11:39:10.609 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2162.g3d78f5ba.el9.noarch 35/198 2024-03-16T11:39:10.609 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2162.g3d78f5ba.e 36/198 2024-03-16T11:39:10.609 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-volume-2:19.0.0-2162.g3d78f5ba.el9.noarch 37/198 2024-03-16T11:39:10.609 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : cephadm-2:19.0.0-2162.g3d78f5ba.el9.noarch 38/198 2024-03-16T11:39:10.609 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : cephfs-top-2:19.0.0-2162.g3d78f5ba.el9.noarch 39/198 2024-03-16T11:39:10.609 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/198 2024-03-16T11:39:10.609 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/198 2024-03-16T11:39:10.609 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/198 2024-03-16T11:39:10.609 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : flex-2.6.4-9.el9.x86_64 43/198 2024-03-16T11:39:10.609 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 44/198 2024-03-16T11:39:10.609 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 45/198 2024-03-16T11:39:10.610 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 46/198 2024-03-16T11:39:10.610 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 47/198 2024-03-16T11:39:10.610 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 48/198 2024-03-16T11:39:10.610 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 49/198 2024-03-16T11:39:10.610 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 50/198 2024-03-16T11:39:10.610 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 51/198 2024-03-16T11:39:10.610 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 52/198 2024-03-16T11:39:10.610 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 53/198 2024-03-16T11:39:10.610 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : lua-5.4.4-4.el9.x86_64 54/198 2024-03-16T11:39:10.610 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 55/198 2024-03-16T11:39:10.610 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 56/198 2024-03-16T11:39:10.610 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 57/198 2024-03-16T11:39:10.610 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : postgresql-13.11-1.el9.x86_64 58/198 2024-03-16T11:39:10.610 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : postgresql-contrib-13.11-1.el9.x86_64 59/198 2024-03-16T11:39:10.611 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : postgresql-private-libs-13.11-1.el9.x86_64 60/198 2024-03-16T11:39:10.611 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : postgresql-server-13.11-1.el9.x86_64 61/198 2024-03-16T11:39:10.611 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 62/198 2024-03-16T11:39:10.611 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 63/198 2024-03-16T11:39:10.611 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 64/198 2024-03-16T11:39:10.611 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 65/198 2024-03-16T11:39:10.611 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 66/198 2024-03-16T11:39:10.611 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 67/198 2024-03-16T11:39:10.611 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 68/198 2024-03-16T11:39:10.611 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 69/198 2024-03-16T11:39:10.611 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 70/198 2024-03-16T11:39:10.611 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 71/198 2024-03-16T11:39:10.611 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 72/198 2024-03-16T11:39:10.611 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 73/198 2024-03-16T11:39:10.611 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 74/198 2024-03-16T11:39:10.612 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 75/198 2024-03-16T11:39:10.612 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 76/198 2024-03-16T11:39:10.612 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 77/198 2024-03-16T11:39:10.612 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 78/198 2024-03-16T11:39:10.612 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 79/198 2024-03-16T11:39:10.612 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 80/198 2024-03-16T11:39:10.612 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 81/198 2024-03-16T11:39:10.612 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 82/198 2024-03-16T11:39:10.612 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 83/198 2024-03-16T11:39:10.612 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 84/198 2024-03-16T11:39:10.612 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 85/198 2024-03-16T11:39:10.612 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 86/198 2024-03-16T11:39:10.612 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 87/198 2024-03-16T11:39:10.612 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 88/198 2024-03-16T11:39:10.612 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-ply-3.11-14.el9.noarch 89/198 2024-03-16T11:39:10.613 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 90/198 2024-03-16T11:39:10.613 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 91/198 2024-03-16T11:39:10.613 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : jq-1.6-16.el9.x86_64 92/198 2024-03-16T11:39:10.613 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 93/198 2024-03-16T11:39:10.613 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 94/198 2024-03-16T11:39:10.613 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 95/198 2024-03-16T11:39:10.613 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 96/198 2024-03-16T11:39:10.613 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 97/198 2024-03-16T11:39:10.613 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 98/198 2024-03-16T11:39:10.613 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 99/198 2024-03-16T11:39:10.613 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 100/198 2024-03-16T11:39:10.613 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 101/198 2024-03-16T11:39:10.613 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 102/198 2024-03-16T11:39:10.613 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 103/198 2024-03-16T11:39:10.613 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 104/198 2024-03-16T11:39:10.614 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/198 2024-03-16T11:39:10.614 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/198 2024-03-16T11:39:10.614 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/198 2024-03-16T11:39:10.614 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/198 2024-03-16T11:39:10.614 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/198 2024-03-16T11:39:10.614 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/198 2024-03-16T11:39:10.614 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/198 2024-03-16T11:39:10.614 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/198 2024-03-16T11:39:10.614 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/198 2024-03-16T11:39:10.614 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/198 2024-03-16T11:39:10.614 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/198 2024-03-16T11:39:10.614 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-asyncssh-2.13.2-4.el9.noarch 116/198 2024-03-16T11:39:10.614 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/198 2024-03-16T11:39:10.614 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/198 2024-03-16T11:39:10.614 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/198 2024-03-16T11:39:10.615 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/198 2024-03-16T11:39:10.615 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/198 2024-03-16T11:39:10.615 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/198 2024-03-16T11:39:10.615 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/198 2024-03-16T11:39:10.615 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-google-auth-1:2.28.0-1.el9.noarch 124/198 2024-03-16T11:39:10.615 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/198 2024-03-16T11:39:10.615 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/198 2024-03-16T11:39:10.615 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/198 2024-03-16T11:39:10.615 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/198 2024-03-16T11:39:10.615 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/198 2024-03-16T11:39:10.615 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/198 2024-03-16T11:39:10.615 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/198 2024-03-16T11:39:10.615 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/198 2024-03-16T11:39:10.615 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/198 2024-03-16T11:39:10.615 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/198 2024-03-16T11:39:10.616 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/198 2024-03-16T11:39:10.616 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/198 2024-03-16T11:39:10.616 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/198 2024-03-16T11:39:10.616 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/198 2024-03-16T11:39:10.616 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/198 2024-03-16T11:39:10.616 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/198 2024-03-16T11:39:10.616 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/198 2024-03-16T11:39:10.616 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/198 2024-03-16T11:39:10.616 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/198 2024-03-16T11:39:10.616 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/198 2024-03-16T11:39:10.616 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/198 2024-03-16T11:39:10.616 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/198 2024-03-16T11:39:10.616 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/198 2024-03-16T11:39:10.616 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/198 2024-03-16T11:39:10.616 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/198 2024-03-16T11:39:10.616 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/198 2024-03-16T11:39:10.617 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : librados2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 151/198 2024-03-16T11:39:10.617 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/198 2024-03-16T11:39:10.617 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : librbd1-2:19.0.0-2162.g3d78f5ba.el9.x86_64 153/198 2024-03-16T11:39:10.617 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/198 2024-03-16T11:39:10.617 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 155/198 2024-03-16T11:39:10.617 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 156/198 2024-03-16T11:39:10.617 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : NetworkManager-1:1.46.0-1.el9.x86_64 157/198 2024-03-16T11:39:10.617 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 158/198 2024-03-16T11:39:10.617 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : NetworkManager-libnm-1:1.46.0-1.el9.x86_64 159/198 2024-03-16T11:39:10.617 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 160/198 2024-03-16T11:39:10.617 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : NetworkManager-team-1:1.46.0-1.el9.x86_64 161/198 2024-03-16T11:39:10.617 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 162/198 2024-03-16T11:39:10.617 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : NetworkManager-tui-1:1.46.0-1.el9.x86_64 163/198 2024-03-16T11:39:10.617 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 164/198 2024-03-16T11:39:10.617 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/198 2024-03-16T11:39:10.617 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/198 2024-03-16T11:39:10.618 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/198 2024-03-16T11:39:10.618 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/198 2024-03-16T11:39:10.618 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/198 2024-03-16T11:39:10.618 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/198 2024-03-16T11:39:10.618 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/198 2024-03-16T11:39:10.618 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/198 2024-03-16T11:39:10.618 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 173/198 2024-03-16T11:39:10.618 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 174/198 2024-03-16T11:39:10.618 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 175/198 2024-03-16T11:39:10.618 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 176/198 2024-03-16T11:39:10.618 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 177/198 2024-03-16T11:39:10.618 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 178/198 2024-03-16T11:39:10.618 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 179/198 2024-03-16T11:39:10.618 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 180/198 2024-03-16T11:39:10.618 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 181/198 2024-03-16T11:39:10.618 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 182/198 2024-03-16T11:39:10.619 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 183/198 2024-03-16T11:39:10.619 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 184/198 2024-03-16T11:39:10.619 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 185/198 2024-03-16T11:39:10.619 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 186/198 2024-03-16T11:39:10.619 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 187/198 2024-03-16T11:39:10.619 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 188/198 2024-03-16T11:39:10.619 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 189/198 2024-03-16T11:39:10.619 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 190/198 2024-03-16T11:39:10.619 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 191/198 2024-03-16T11:39:10.619 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 192/198 2024-03-16T11:39:10.619 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 193/198 2024-03-16T11:39:10.619 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 194/198 2024-03-16T11:39:10.619 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 195/198 2024-03-16T11:39:10.619 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 196/198 2024-03-16T11:39:10.619 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 197/198 2024-03-16T11:39:13.710 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 198/198 2024-03-16T11:39:13.711 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-16T11:39:13.711 INFO:teuthology.orchestra.run.smithi029.stdout:Upgraded: 2024-03-16T11:39:13.711 INFO:teuthology.orchestra.run.smithi029.stdout: NetworkManager-1:1.46.0-1.el9.x86_64 2024-03-16T11:39:13.711 INFO:teuthology.orchestra.run.smithi029.stdout: NetworkManager-libnm-1:1.46.0-1.el9.x86_64 2024-03-16T11:39:13.711 INFO:teuthology.orchestra.run.smithi029.stdout: NetworkManager-team-1:1.46.0-1.el9.x86_64 2024-03-16T11:39:13.711 INFO:teuthology.orchestra.run.smithi029.stdout: NetworkManager-tui-1:1.46.0-1.el9.x86_64 2024-03-16T11:39:13.711 INFO:teuthology.orchestra.run.smithi029.stdout: acl-2.3.1-4.el9.x86_64 2024-03-16T11:39:13.711 INFO:teuthology.orchestra.run.smithi029.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-03-16T11:39:13.711 INFO:teuthology.orchestra.run.smithi029.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-03-16T11:39:13.711 INFO:teuthology.orchestra.run.smithi029.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-03-16T11:39:13.711 INFO:teuthology.orchestra.run.smithi029.stdout: libacl-2.3.1-4.el9.x86_64 2024-03-16T11:39:13.711 INFO:teuthology.orchestra.run.smithi029.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-03-16T11:39:13.712 INFO:teuthology.orchestra.run.smithi029.stdout: libblkid-2.37.4-18.el9.x86_64 2024-03-16T11:39:13.712 INFO:teuthology.orchestra.run.smithi029.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-03-16T11:39:13.712 INFO:teuthology.orchestra.run.smithi029.stdout: libmount-2.37.4-18.el9.x86_64 2024-03-16T11:39:13.712 INFO:teuthology.orchestra.run.smithi029.stdout: librados2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T11:39:13.712 INFO:teuthology.orchestra.run.smithi029.stdout: librbd1-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T11:39:13.712 INFO:teuthology.orchestra.run.smithi029.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-03-16T11:39:13.712 INFO:teuthology.orchestra.run.smithi029.stdout: libuuid-2.37.4-18.el9.x86_64 2024-03-16T11:39:13.712 INFO:teuthology.orchestra.run.smithi029.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-03-16T11:39:13.712 INFO:teuthology.orchestra.run.smithi029.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-03-16T11:39:13.712 INFO:teuthology.orchestra.run.smithi029.stdout: util-linux-2.37.4-18.el9.x86_64 2024-03-16T11:39:13.712 INFO:teuthology.orchestra.run.smithi029.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-03-16T11:39:13.712 INFO:teuthology.orchestra.run.smithi029.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-03-16T11:39:13.712 INFO:teuthology.orchestra.run.smithi029.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-03-16T11:39:13.712 INFO:teuthology.orchestra.run.smithi029.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-03-16T11:39:13.713 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2024-03-16T11:39:13.713 INFO:teuthology.orchestra.run.smithi029.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-16T11:39:13.713 INFO:teuthology.orchestra.run.smithi029.stdout: automake-1.16.2-8.el9.noarch 2024-03-16T11:39:13.713 INFO:teuthology.orchestra.run.smithi029.stdout: bison-3.7.4-5.el9.x86_64 2024-03-16T11:39:13.713 INFO:teuthology.orchestra.run.smithi029.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-16T11:39:13.713 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T11:39:13.713 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-base-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T11:39:13.713 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T11:39:13.713 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T11:39:13.713 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-grafana-dashboards-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-16T11:39:13.713 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-immutable-object-cache-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T11:39:13.713 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mds-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T11:39:13.713 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T11:39:13.713 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-cephadm-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-16T11:39:13.714 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-dashboard-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-16T11:39:13.714 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-16T11:39:13.714 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-modules-core-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-16T11:39:13.714 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-rook-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-16T11:39:13.714 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mon-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T11:39:13.714 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-osd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T11:39:13.714 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-prometheus-alerts-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-16T11:39:13.714 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-radosgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T11:39:13.714 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-selinux-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T11:39:13.714 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-test-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T11:39:13.714 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-volume-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-16T11:39:13.714 INFO:teuthology.orchestra.run.smithi029.stdout: cephadm-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-16T11:39:13.714 INFO:teuthology.orchestra.run.smithi029.stdout: cephfs-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T11:39:13.714 INFO:teuthology.orchestra.run.smithi029.stdout: cephfs-shell-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T11:39:13.715 INFO:teuthology.orchestra.run.smithi029.stdout: cephfs-top-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-16T11:39:13.715 INFO:teuthology.orchestra.run.smithi029.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-03-16T11:39:13.715 INFO:teuthology.orchestra.run.smithi029.stdout: flex-2.6.4-9.el9.x86_64 2024-03-16T11:39:13.715 INFO:teuthology.orchestra.run.smithi029.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-16T11:39:13.715 INFO:teuthology.orchestra.run.smithi029.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-16T11:39:13.715 INFO:teuthology.orchestra.run.smithi029.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-16T11:39:13.715 INFO:teuthology.orchestra.run.smithi029.stdout: fuse-2.9.9-15.el9.x86_64 2024-03-16T11:39:13.715 INFO:teuthology.orchestra.run.smithi029.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-16T11:39:13.715 INFO:teuthology.orchestra.run.smithi029.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-16T11:39:13.715 INFO:teuthology.orchestra.run.smithi029.stdout: jq-1.6-16.el9.x86_64 2024-03-16T11:39:13.715 INFO:teuthology.orchestra.run.smithi029.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-03-16T11:39:13.715 INFO:teuthology.orchestra.run.smithi029.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-16T11:39:13.715 INFO:teuthology.orchestra.run.smithi029.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-16T11:39:13.715 INFO:teuthology.orchestra.run.smithi029.stdout: libcephfs-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T11:39:13.715 INFO:teuthology.orchestra.run.smithi029.stdout: libcephfs2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T11:39:13.716 INFO:teuthology.orchestra.run.smithi029.stdout: libcephsqlite-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T11:39:13.716 INFO:teuthology.orchestra.run.smithi029.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-16T11:39:13.716 INFO:teuthology.orchestra.run.smithi029.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-16T11:39:13.716 INFO:teuthology.orchestra.run.smithi029.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-16T11:39:13.716 INFO:teuthology.orchestra.run.smithi029.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-16T11:39:13.716 INFO:teuthology.orchestra.run.smithi029.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-16T11:39:13.716 INFO:teuthology.orchestra.run.smithi029.stdout: librados-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T11:39:13.716 INFO:teuthology.orchestra.run.smithi029.stdout: libradosstriper1-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T11:39:13.716 INFO:teuthology.orchestra.run.smithi029.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-16T11:39:13.716 INFO:teuthology.orchestra.run.smithi029.stdout: librgw2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T11:39:13.716 INFO:teuthology.orchestra.run.smithi029.stdout: libtool-2.4.6-45.el9.x86_64 2024-03-16T11:39:13.716 INFO:teuthology.orchestra.run.smithi029.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-16T11:39:13.716 INFO:teuthology.orchestra.run.smithi029.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-03-16T11:39:13.716 INFO:teuthology.orchestra.run.smithi029.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-03-16T11:39:13.716 INFO:teuthology.orchestra.run.smithi029.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-16T11:39:13.717 INFO:teuthology.orchestra.run.smithi029.stdout: lua-5.4.4-4.el9.x86_64 2024-03-16T11:39:13.717 INFO:teuthology.orchestra.run.smithi029.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-16T11:39:13.717 INFO:teuthology.orchestra.run.smithi029.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-16T11:39:13.717 INFO:teuthology.orchestra.run.smithi029.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-16T11:39:13.717 INFO:teuthology.orchestra.run.smithi029.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-16T11:39:13.717 INFO:teuthology.orchestra.run.smithi029.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-16T11:39:13.717 INFO:teuthology.orchestra.run.smithi029.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-03-16T11:39:13.717 INFO:teuthology.orchestra.run.smithi029.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-16T11:39:13.717 INFO:teuthology.orchestra.run.smithi029.stdout: postgresql-13.11-1.el9.x86_64 2024-03-16T11:39:13.717 INFO:teuthology.orchestra.run.smithi029.stdout: postgresql-contrib-13.11-1.el9.x86_64 2024-03-16T11:39:13.717 INFO:teuthology.orchestra.run.smithi029.stdout: postgresql-private-libs-13.11-1.el9.x86_64 2024-03-16T11:39:13.717 INFO:teuthology.orchestra.run.smithi029.stdout: postgresql-server-13.11-1.el9.x86_64 2024-03-16T11:39:13.717 INFO:teuthology.orchestra.run.smithi029.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-16T11:39:13.717 INFO:teuthology.orchestra.run.smithi029.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-16T11:39:13.718 INFO:teuthology.orchestra.run.smithi029.stdout: python3-asyncssh-2.13.2-4.el9.noarch 2024-03-16T11:39:13.718 INFO:teuthology.orchestra.run.smithi029.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-03-16T11:39:13.718 INFO:teuthology.orchestra.run.smithi029.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-16T11:39:13.718 INFO:teuthology.orchestra.run.smithi029.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-16T11:39:13.718 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-16T11:39:13.718 INFO:teuthology.orchestra.run.smithi029.stdout: python3-ceph-argparse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T11:39:13.718 INFO:teuthology.orchestra.run.smithi029.stdout: python3-ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T11:39:13.718 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cephfs-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T11:39:13.718 INFO:teuthology.orchestra.run.smithi029.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-16T11:39:13.718 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-16T11:39:13.718 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-16T11:39:13.718 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-16T11:39:13.718 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-03-16T11:39:13.718 INFO:teuthology.orchestra.run.smithi029.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-03-16T11:39:13.719 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-16T11:39:13.719 INFO:teuthology.orchestra.run.smithi029.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-16T11:39:13.719 INFO:teuthology.orchestra.run.smithi029.stdout: python3-google-auth-1:2.28.0-1.el9.noarch 2024-03-16T11:39:13.719 INFO:teuthology.orchestra.run.smithi029.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-16T11:39:13.719 INFO:teuthology.orchestra.run.smithi029.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-16T11:39:13.719 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-16T11:39:13.719 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-16T11:39:13.719 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-16T11:39:13.719 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-16T11:39:13.719 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-16T11:39:13.719 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-16T11:39:13.719 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-16T11:39:13.719 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-16T11:39:13.719 INFO:teuthology.orchestra.run.smithi029.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-16T11:39:13.720 INFO:teuthology.orchestra.run.smithi029.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-16T11:39:13.720 INFO:teuthology.orchestra.run.smithi029.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-16T11:39:13.720 INFO:teuthology.orchestra.run.smithi029.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-16T11:39:13.720 INFO:teuthology.orchestra.run.smithi029.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-16T11:39:13.720 INFO:teuthology.orchestra.run.smithi029.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-16T11:39:13.720 INFO:teuthology.orchestra.run.smithi029.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-16T11:39:13.720 INFO:teuthology.orchestra.run.smithi029.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-16T11:39:13.720 INFO:teuthology.orchestra.run.smithi029.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-16T11:39:13.720 INFO:teuthology.orchestra.run.smithi029.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-16T11:39:13.720 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-16T11:39:13.720 INFO:teuthology.orchestra.run.smithi029.stdout: python3-ply-3.11-14.el9.noarch 2024-03-16T11:39:13.720 INFO:teuthology.orchestra.run.smithi029.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-16T11:39:13.720 INFO:teuthology.orchestra.run.smithi029.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-16T11:39:13.720 INFO:teuthology.orchestra.run.smithi029.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-16T11:39:13.720 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-16T11:39:13.721 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-16T11:39:13.721 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-16T11:39:13.721 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-16T11:39:13.721 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-03-16T11:39:13.721 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-16T11:39:13.721 INFO:teuthology.orchestra.run.smithi029.stdout: python3-rados-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T11:39:13.721 INFO:teuthology.orchestra.run.smithi029.stdout: python3-rbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T11:39:13.721 INFO:teuthology.orchestra.run.smithi029.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-16T11:39:13.721 INFO:teuthology.orchestra.run.smithi029.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-16T11:39:13.721 INFO:teuthology.orchestra.run.smithi029.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-16T11:39:13.721 INFO:teuthology.orchestra.run.smithi029.stdout: python3-rgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T11:39:13.721 INFO:teuthology.orchestra.run.smithi029.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-16T11:39:13.721 INFO:teuthology.orchestra.run.smithi029.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-16T11:39:13.721 INFO:teuthology.orchestra.run.smithi029.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-16T11:39:13.721 INFO:teuthology.orchestra.run.smithi029.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-16T11:39:13.722 INFO:teuthology.orchestra.run.smithi029.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-16T11:39:13.722 INFO:teuthology.orchestra.run.smithi029.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-16T11:39:13.722 INFO:teuthology.orchestra.run.smithi029.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-16T11:39:13.722 INFO:teuthology.orchestra.run.smithi029.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-03-16T11:39:13.722 INFO:teuthology.orchestra.run.smithi029.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-16T11:39:13.722 INFO:teuthology.orchestra.run.smithi029.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-16T11:39:13.722 INFO:teuthology.orchestra.run.smithi029.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-16T11:39:13.722 INFO:teuthology.orchestra.run.smithi029.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-16T11:39:13.722 INFO:teuthology.orchestra.run.smithi029.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-16T11:39:13.722 INFO:teuthology.orchestra.run.smithi029.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-16T11:39:13.722 INFO:teuthology.orchestra.run.smithi029.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-16T11:39:13.722 INFO:teuthology.orchestra.run.smithi029.stdout: rbd-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T11:39:13.722 INFO:teuthology.orchestra.run.smithi029.stdout: rbd-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T11:39:13.722 INFO:teuthology.orchestra.run.smithi029.stdout: rbd-nbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-16T11:39:13.723 INFO:teuthology.orchestra.run.smithi029.stdout: re2-1:20211101-3.el9.x86_64 2024-03-16T11:39:13.723 INFO:teuthology.orchestra.run.smithi029.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-16T11:39:13.723 INFO:teuthology.orchestra.run.smithi029.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-16T11:39:13.723 INFO:teuthology.orchestra.run.smithi029.stdout: uuid-1.6.2-55.el9.x86_64 2024-03-16T11:39:13.723 INFO:teuthology.orchestra.run.smithi029.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-03-16T11:39:13.723 INFO:teuthology.orchestra.run.smithi029.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-16T11:39:13.723 INFO:teuthology.orchestra.run.smithi029.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-03-16T11:39:13.723 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-16T11:39:13.723 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-03-16T11:39:14.375 DEBUG:teuthology.parallel:result is None 2024-03-16T11:39:14.376 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3d78f5ba07bf18880172c07d4d7b4ac00c9d30af 2024-03-16T11:39:14.522 DEBUG:teuthology.orchestra.run.smithi029:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-16T11:39:14.546 INFO:teuthology.orchestra.run.smithi029.stdout:19.0.0-2162.g3d78f5ba.el9 2024-03-16T11:39:14.547 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2162.g3d78f5ba.el9 2024-03-16T11:39:14.547 INFO:teuthology.task.install:The correct ceph version 19.0.0-2162.g3d78f5ba is installed. 2024-03-16T11:39:14.549 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3d78f5ba07bf18880172c07d4d7b4ac00c9d30af 2024-03-16T11:39:14.761 DEBUG:teuthology.orchestra.run.smithi152:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-16T11:39:14.787 INFO:teuthology.orchestra.run.smithi152.stdout:19.0.0-2162.g3d78f5ba.el9 2024-03-16T11:39:14.787 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2162.g3d78f5ba.el9 2024-03-16T11:39:14.787 INFO:teuthology.task.install:The correct ceph version 19.0.0-2162.g3d78f5ba is installed. 2024-03-16T11:39:14.789 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3d78f5ba07bf18880172c07d4d7b4ac00c9d30af 2024-03-16T11:39:14.931 DEBUG:teuthology.orchestra.run.smithi154:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-16T11:39:14.956 INFO:teuthology.orchestra.run.smithi154.stdout:19.0.0-2162.g3d78f5ba.el9 2024-03-16T11:39:14.956 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2162.g3d78f5ba.el9 2024-03-16T11:39:14.957 INFO:teuthology.task.install:The correct ceph version 19.0.0-2162.g3d78f5ba is installed. 2024-03-16T11:39:14.958 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-16T11:39:14.959 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-03-16T11:39:14.959 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-16T11:39:14.993 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-03-16T11:39:14.993 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-16T11:39:15.028 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-03-16T11:39:15.028 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-16T11:39:15.062 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-16T11:39:15.063 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-03-16T11:39:15.063 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd of=/usr/bin/daemon-helper 2024-03-16T11:39:15.093 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-16T11:39:15.162 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-03-16T11:39:15.163 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd of=/usr/bin/daemon-helper 2024-03-16T11:39:15.194 DEBUG:teuthology.orchestra.run.smithi152:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-16T11:39:15.265 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-03-16T11:39:15.265 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd of=/usr/bin/daemon-helper 2024-03-16T11:39:15.295 DEBUG:teuthology.orchestra.run.smithi154:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-16T11:39:15.363 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-16T11:39:15.363 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-03-16T11:39:15.363 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-16T11:39:15.393 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-16T11:39:15.462 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-03-16T11:39:15.462 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-16T11:39:15.492 DEBUG:teuthology.orchestra.run.smithi152:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-16T11:39:15.561 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-03-16T11:39:15.561 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-16T11:39:15.589 DEBUG:teuthology.orchestra.run.smithi154:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-16T11:39:15.657 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-16T11:39:15.658 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-03-16T11:39:15.658 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd of=/usr/bin/stdin-killer 2024-03-16T11:39:15.688 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-16T11:39:15.757 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-03-16T11:39:15.757 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd of=/usr/bin/stdin-killer 2024-03-16T11:39:15.785 DEBUG:teuthology.orchestra.run.smithi152:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-16T11:39:15.852 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-03-16T11:39:15.853 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd of=/usr/bin/stdin-killer 2024-03-16T11:39:15.880 DEBUG:teuthology.orchestra.run.smithi154:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-16T11:39:15.948 INFO:teuthology.run_tasks:Running task cephadm... 2024-03-16T11:39:16.046 INFO:tasks.cephadm:Config: {'roleless': False, 'cephfs': {'max_mds': 3, 'session_timeout': 300, 'standby_replay': True}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_bal_replicate_threshold': 1, 'mds_export_ephemeral_random_max': 0.1, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'cephsqlite lock renewal timeout': 900000, 'debug cephsqlite': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180, 'osd shutdown pgref assert': True, 'osd_max_omap_entries_per_request': 10}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'MON_DOWN'], 'sha1': '3d78f5ba07bf18880172c07d4d7b4ac00c9d30af', 'subvols': {'create': 2}} 2024-03-16T11:39:16.046 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af 2024-03-16T11:39:16.047 INFO:tasks.cephadm:Cluster fsid is d1887f28-e389-11ee-95c9-87774f69a715 2024-03-16T11:39:16.047 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-03-16T11:39:16.047 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.29', 'mon.b': '172.21.15.152', 'mon.c': '172.21.15.154'} 2024-03-16T11:39:16.047 INFO:tasks.cephadm:First mon is mon.a on smithi029 2024-03-16T11:39:16.047 INFO:tasks.cephadm:First mgr is x 2024-03-16T11:39:16.047 INFO:tasks.cephadm:Normalizing hostnames... 2024-03-16T11:39:16.048 DEBUG:teuthology.orchestra.run.smithi029:> sudo hostname $(hostname -s) 2024-03-16T11:39:16.080 DEBUG:teuthology.orchestra.run.smithi152:> sudo hostname $(hostname -s) 2024-03-16T11:39:16.108 DEBUG:teuthology.orchestra.run.smithi154:> sudo hostname $(hostname -s) 2024-03-16T11:39:16.134 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-03-16T11:39:16.135 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3d78f5ba07bf18880172c07d4d7b4ac00c9d30af 2024-03-16T11:39:16.351 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': '3d78f5ba07bf18880172c07d4d7b4ac00c9d30af', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/77658/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-2162-g3d78f5ba', 'node_name': '172.21.5.34+adami04', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-2162.g3d78f5ba'}, 'url': 'https://3.chacra.ceph.com/r/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-03-15 01:33:04.210510', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-batrick-testing-20240314.183243', 'chacra_url': 'https://3.chacra.ceph.com/repos/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/flavors/default/', 'archs': ['x86_64', 'source'], 'distro': 'centos'}] 2024-03-16T11:39:16.491 INFO:tasks.util.chacra:got chacra host 3.chacra.ceph.com, ref wip-batrick-testing-20240314.183243, sha1 3d78f5ba07bf18880172c07d4d7b4ac00c9d30af from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=3d78f5ba07bf18880172c07d4d7b4ac00c9d30af 2024-03-16T11:39:16.492 INFO:tasks.cephadm:Discovered cachra url: https://3.chacra.ceph.com/binaries/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/x86_64/flavors/default/cephadm 2024-03-16T11:39:16.493 DEBUG:teuthology.orchestra.run.smithi029:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-16T11:39:17.071 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r--. 1 ubuntu ubuntu 767536 Mar 16 11:39 /home/ubuntu/cephtest/cephadm 2024-03-16T11:39:17.072 DEBUG:teuthology.orchestra.run.smithi152:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-16T11:39:17.505 INFO:teuthology.orchestra.run.smithi152.stdout:-rw-r--r--. 1 ubuntu ubuntu 767536 Mar 16 11:39 /home/ubuntu/cephtest/cephadm 2024-03-16T11:39:17.506 DEBUG:teuthology.orchestra.run.smithi154:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-16T11:39:18.107 INFO:teuthology.orchestra.run.smithi154.stdout:-rw-r--r--. 1 ubuntu ubuntu 767536 Mar 16 11:39 /home/ubuntu/cephtest/cephadm 2024-03-16T11:39:18.107 DEBUG:teuthology.orchestra.run.smithi029:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-16T11:39:18.125 DEBUG:teuthology.orchestra.run.smithi152:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-16T11:39:18.143 DEBUG:teuthology.orchestra.run.smithi154:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-16T11:39:18.177 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af on all hosts... 2024-03-16T11:39:18.177 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af pull 2024-03-16T11:39:18.179 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af pull 2024-03-16T11:39:18.185 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af pull 2024-03-16T11:39:19.000 INFO:teuthology.orchestra.run.smithi154.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af... 2024-03-16T11:39:19.032 INFO:teuthology.orchestra.run.smithi152.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af... 2024-03-16T11:39:19.233 INFO:teuthology.orchestra.run.smithi029.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af... 2024-03-16T11:39:55.196 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-03-16T11:39:55.197 INFO:teuthology.orchestra.run.smithi029.stdout: "ceph_version": "ceph version 19.0.0-2162-g3d78f5ba (3d78f5ba07bf18880172c07d4d7b4ac00c9d30af) squid (dev)", 2024-03-16T11:39:55.197 INFO:teuthology.orchestra.run.smithi029.stdout: "image_id": "be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae", 2024-03-16T11:39:55.197 INFO:teuthology.orchestra.run.smithi029.stdout: "repo_digests": [ 2024-03-16T11:39:55.197 INFO:teuthology.orchestra.run.smithi029.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513" 2024-03-16T11:39:55.197 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-16T11:39:55.197 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-03-16T11:40:06.591 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-03-16T11:40:06.591 INFO:teuthology.orchestra.run.smithi154.stdout: "ceph_version": "ceph version 19.0.0-2162-g3d78f5ba (3d78f5ba07bf18880172c07d4d7b4ac00c9d30af) squid (dev)", 2024-03-16T11:40:06.591 INFO:teuthology.orchestra.run.smithi154.stdout: "image_id": "be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae", 2024-03-16T11:40:06.591 INFO:teuthology.orchestra.run.smithi154.stdout: "repo_digests": [ 2024-03-16T11:40:06.591 INFO:teuthology.orchestra.run.smithi154.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513" 2024-03-16T11:40:06.591 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-03-16T11:40:06.591 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-03-16T11:40:07.301 INFO:teuthology.orchestra.run.smithi152.stdout:{ 2024-03-16T11:40:07.301 INFO:teuthology.orchestra.run.smithi152.stdout: "ceph_version": "ceph version 19.0.0-2162-g3d78f5ba (3d78f5ba07bf18880172c07d4d7b4ac00c9d30af) squid (dev)", 2024-03-16T11:40:07.301 INFO:teuthology.orchestra.run.smithi152.stdout: "image_id": "be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae", 2024-03-16T11:40:07.301 INFO:teuthology.orchestra.run.smithi152.stdout: "repo_digests": [ 2024-03-16T11:40:07.301 INFO:teuthology.orchestra.run.smithi152.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513" 2024-03-16T11:40:07.301 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-03-16T11:40:07.302 INFO:teuthology.orchestra.run.smithi152.stdout:} 2024-03-16T11:40:07.331 DEBUG:teuthology.orchestra.run.smithi029:> sudo mkdir -p /etc/ceph 2024-03-16T11:40:07.368 DEBUG:teuthology.orchestra.run.smithi152:> sudo mkdir -p /etc/ceph 2024-03-16T11:40:07.397 DEBUG:teuthology.orchestra.run.smithi154:> sudo mkdir -p /etc/ceph 2024-03-16T11:40:07.432 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod 777 /etc/ceph 2024-03-16T11:40:07.462 DEBUG:teuthology.orchestra.run.smithi152:> sudo chmod 777 /etc/ceph 2024-03-16T11:40:07.489 DEBUG:teuthology.orchestra.run.smithi154:> sudo chmod 777 /etc/ceph 2024-03-16T11:40:07.519 INFO:tasks.cephadm:Writing seed config... 2024-03-16T11:40:07.520 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-03-16T11:40:07.520 INFO:tasks.cephadm: override: [client] debug client = 20 2024-03-16T11:40:07.520 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-03-16T11:40:07.520 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-03-16T11:40:07.520 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-03-16T11:40:07.520 INFO:tasks.cephadm: override: [global] ms die on skipped message = False 2024-03-16T11:40:07.520 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-03-16T11:40:07.521 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-03-16T11:40:07.521 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-03-16T11:40:07.521 INFO:tasks.cephadm: override: [mds] mds bal fragment size max = 10000 2024-03-16T11:40:07.521 INFO:tasks.cephadm: override: [mds] mds bal merge size = 5 2024-03-16T11:40:07.521 INFO:tasks.cephadm: override: [mds] mds bal split bits = 3 2024-03-16T11:40:07.521 INFO:tasks.cephadm: override: [mds] mds bal split size = 100 2024-03-16T11:40:07.521 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-03-16T11:40:07.521 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-03-16T11:40:07.521 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-03-16T11:40:07.521 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-03-16T11:40:07.521 INFO:tasks.cephadm: override: [mds] mds_bal_replicate_threshold = 1 2024-03-16T11:40:07.521 INFO:tasks.cephadm: override: [mds] mds_export_ephemeral_random_max = 0.1 2024-03-16T11:40:07.521 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-03-16T11:40:07.521 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-03-16T11:40:07.521 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-03-16T11:40:07.521 INFO:tasks.cephadm: override: [mgr] cephsqlite lock renewal timeout = 900000 2024-03-16T11:40:07.521 INFO:tasks.cephadm: override: [mgr] debug cephsqlite = 20 2024-03-16T11:40:07.522 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-03-16T11:40:07.522 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-03-16T11:40:07.522 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-03-16T11:40:07.522 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-03-16T11:40:07.522 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-03-16T11:40:07.522 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-03-16T11:40:07.522 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-03-16T11:40:07.522 INFO:tasks.cephadm: override: [osd] bluestore compression mode = aggressive 2024-03-16T11:40:07.522 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-03-16T11:40:07.522 INFO:tasks.cephadm: override: [osd] debug bluefs = 20 2024-03-16T11:40:07.522 INFO:tasks.cephadm: override: [osd] debug bluestore = 20 2024-03-16T11:40:07.522 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-03-16T11:40:07.522 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-03-16T11:40:07.522 INFO:tasks.cephadm: override: [osd] debug rocksdb = 10 2024-03-16T11:40:07.522 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-03-16T11:40:07.522 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-03-16T11:40:07.523 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-03-16T11:40:07.523 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-03-16T11:40:07.523 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-03-16T11:40:07.523 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-03-16T11:40:07.523 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-03-16T11:40:07.523 INFO:tasks.cephadm: override: [osd] osd_max_omap_entries_per_request = 10 2024-03-16T11:40:07.523 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-03-16T11:40:07.524 DEBUG:teuthology.orchestra.run.smithi029:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-03-16T11:40:07.538 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = d1887f28-e389-11ee-95c9-87774f69a715 ms die on skipped message = False [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true bluestore block size = 96636764160 bluestore compression mode = aggressive bluestore fsck on mount = True debug bluefs = 20 debug bluestore = 20 debug ms = 1 debug osd = 20 debug rocksdb = 10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 osd_max_omap_entries_per_request = 10 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false cephsqlite lock renewal timeout = 900000 debug cephsqlite = 20 debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds bal fragment size max = 10000 mds bal merge size = 5 mds bal split bits = 3 mds bal split size = 100 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True mds_bal_replicate_threshold = 1 mds_export_ephemeral_random_max = 0.1 osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-03-16T11:40:07.538 DEBUG:teuthology.orchestra.run.smithi029:mon.a> sudo journalctl -f -n 0 -u ceph-d1887f28-e389-11ee-95c9-87774f69a715@mon.a.service 2024-03-16T11:40:07.580 DEBUG:teuthology.orchestra.run.smithi029:mgr.x> sudo journalctl -f -n 0 -u ceph-d1887f28-e389-11ee-95c9-87774f69a715@mgr.x.service 2024-03-16T11:40:07.622 INFO:tasks.cephadm:Bootstrapping... 2024-03-16T11:40:07.623 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af -v bootstrap --fsid d1887f28-e389-11ee-95c9-87774f69a715 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.29 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-03-16T11:40:07.820 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2024-03-16T11:40:07.820 INFO:teuthology.orchestra.run.smithi029.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af', '-v', 'bootstrap', '--fsid', 'd1887f28-e389-11ee-95c9-87774f69a715', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'x', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.29', '--skip-admin-label'] 2024-03-16T11:40:07.845 INFO:teuthology.orchestra.run.smithi029.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-16T11:40:07.845 INFO:teuthology.orchestra.run.smithi029.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-03-16T11:40:07.846 INFO:teuthology.orchestra.run.smithi029.stdout:Verifying podman|docker is present... 2024-03-16T11:40:07.868 INFO:teuthology.orchestra.run.smithi029.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-16T11:40:07.868 INFO:teuthology.orchestra.run.smithi029.stdout:Verifying lvm2 is present... 2024-03-16T11:40:07.868 INFO:teuthology.orchestra.run.smithi029.stdout:Verifying time synchronization is in place... 2024-03-16T11:40:07.876 INFO:teuthology.orchestra.run.smithi029.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-16T11:40:07.876 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-16T11:40:07.882 INFO:teuthology.orchestra.run.smithi029.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-16T11:40:07.882 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl: stdout inactive 2024-03-16T11:40:07.888 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl: stdout enabled 2024-03-16T11:40:07.894 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl: stdout active 2024-03-16T11:40:07.894 INFO:teuthology.orchestra.run.smithi029.stdout:Unit chronyd.service is enabled and running 2024-03-16T11:40:07.894 INFO:teuthology.orchestra.run.smithi029.stdout:Repeating the final host check... 2024-03-16T11:40:07.916 INFO:teuthology.orchestra.run.smithi029.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-16T11:40:07.916 INFO:teuthology.orchestra.run.smithi029.stdout:podman (/bin/podman) version 4.9.3 is present 2024-03-16T11:40:07.916 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl is present 2024-03-16T11:40:07.916 INFO:teuthology.orchestra.run.smithi029.stdout:lvcreate is present 2024-03-16T11:40:07.922 INFO:teuthology.orchestra.run.smithi029.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-16T11:40:07.922 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-16T11:40:07.928 INFO:teuthology.orchestra.run.smithi029.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-16T11:40:07.928 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl: stdout inactive 2024-03-16T11:40:07.934 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl: stdout enabled 2024-03-16T11:40:07.940 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl: stdout active 2024-03-16T11:40:07.940 INFO:teuthology.orchestra.run.smithi029.stdout:Unit chronyd.service is enabled and running 2024-03-16T11:40:07.940 INFO:teuthology.orchestra.run.smithi029.stdout:Host looks OK 2024-03-16T11:40:07.940 INFO:teuthology.orchestra.run.smithi029.stdout:Cluster fsid: d1887f28-e389-11ee-95c9-87774f69a715 2024-03-16T11:40:07.940 INFO:teuthology.orchestra.run.smithi029.stdout:Acquiring lock 140697053740480 on /run/cephadm/d1887f28-e389-11ee-95c9-87774f69a715.lock 2024-03-16T11:40:07.940 INFO:teuthology.orchestra.run.smithi029.stdout:Lock 140697053740480 acquired on /run/cephadm/d1887f28-e389-11ee-95c9-87774f69a715.lock 2024-03-16T11:40:07.941 INFO:teuthology.orchestra.run.smithi029.stdout:Verifying IP 172.21.15.29 port 3300 ... 2024-03-16T11:40:07.941 INFO:teuthology.orchestra.run.smithi029.stdout:Verifying IP 172.21.15.29 port 6789 ... 2024-03-16T11:40:07.941 INFO:teuthology.orchestra.run.smithi029.stdout:Base mon IP(s) is [172.21.15.29:3300, 172.21.15.29:6789], mon addrv is [v2:172.21.15.29:3300,v1:172.21.15.29:6789] 2024-03-16T11:40:07.944 INFO:teuthology.orchestra.run.smithi029.stdout:/sbin/ip: stdout default via 172.21.15.254 dev ens1f0 proto dhcp src 172.21.15.29 metric 100 2024-03-16T11:40:07.944 INFO:teuthology.orchestra.run.smithi029.stdout:/sbin/ip: stdout 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.29 metric 100 2024-03-16T11:40:07.947 INFO:teuthology.orchestra.run.smithi029.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-03-16T11:40:07.947 INFO:teuthology.orchestra.run.smithi029.stdout:/sbin/ip: stdout fe80::/64 dev ens1f0 proto kernel metric 1024 pref medium 2024-03-16T11:40:07.947 INFO:teuthology.orchestra.run.smithi029.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 100 pref medium 2024-03-16T11:40:07.949 INFO:teuthology.orchestra.run.smithi029.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-03-16T11:40:07.949 INFO:teuthology.orchestra.run.smithi029.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-03-16T11:40:07.949 INFO:teuthology.orchestra.run.smithi029.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-16T11:40:07.949 INFO:teuthology.orchestra.run.smithi029.stdout:/sbin/ip: stdout 4: ens1f0: mtu 1500 state UP qlen 1000 2024-03-16T11:40:07.949 INFO:teuthology.orchestra.run.smithi029.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:febd:125a/64 scope link noprefixroute 2024-03-16T11:40:07.949 INFO:teuthology.orchestra.run.smithi029.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-16T11:40:07.950 INFO:teuthology.orchestra.run.smithi029.stdout:Mon IP `172.21.15.29` is in CIDR network `172.21.0.0/20` 2024-03-16T11:40:07.950 INFO:teuthology.orchestra.run.smithi029.stdout:Mon IP `172.21.15.29` is in CIDR network `172.21.0.0/20` 2024-03-16T11:40:07.950 INFO:teuthology.orchestra.run.smithi029.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-03-16T11:40:07.950 INFO:teuthology.orchestra.run.smithi029.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-03-16T11:40:07.951 INFO:teuthology.orchestra.run.smithi029.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af... 2024-03-16T11:40:08.443 INFO:teuthology.orchestra.run.smithi029.stdout:/bin/podman: stdout be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae 2024-03-16T11:40:08.443 INFO:teuthology.orchestra.run.smithi029.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af... 2024-03-16T11:40:08.443 INFO:teuthology.orchestra.run.smithi029.stdout:/bin/podman: stderr Getting image source signatures 2024-03-16T11:40:08.443 INFO:teuthology.orchestra.run.smithi029.stdout:/bin/podman: stderr Copying blob sha256:66487374cb611c95435850c17b53ebb64c22ed9555f72cd43d678b60262a12b1 2024-03-16T11:40:08.443 INFO:teuthology.orchestra.run.smithi029.stdout:/bin/podman: stderr Copying blob sha256:f58b4670ef4403d11d1c912aa26a4dbcf73447df5302e9dc66f6a7fc0d9d1e47 2024-03-16T11:40:08.443 INFO:teuthology.orchestra.run.smithi029.stdout:/bin/podman: stderr Copying config sha256:be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae 2024-03-16T11:40:08.444 INFO:teuthology.orchestra.run.smithi029.stdout:/bin/podman: stderr Writing manifest to image destination 2024-03-16T11:40:09.379 INFO:teuthology.orchestra.run.smithi029.stdout:ceph: stdout ceph version 19.0.0-2162-g3d78f5ba (3d78f5ba07bf18880172c07d4d7b4ac00c9d30af) squid (dev) 2024-03-16T11:40:09.380 INFO:teuthology.orchestra.run.smithi029.stdout:Ceph version: ceph version 19.0.0-2162-g3d78f5ba (3d78f5ba07bf18880172c07d4d7b4ac00c9d30af) squid (dev) 2024-03-16T11:40:09.380 INFO:teuthology.orchestra.run.smithi029.stdout:Extracting ceph user uid/gid from container image... 2024-03-16T11:40:10.267 INFO:teuthology.orchestra.run.smithi029.stdout:stat: stdout 167 167 2024-03-16T11:40:10.267 INFO:teuthology.orchestra.run.smithi029.stdout:Creating initial keys... 2024-03-16T11:40:11.221 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph-authtool: stdout AQAahfVlu2ADJhAAOC2a4yhBC8zuf6ArvfuCiA== 2024-03-16T11:40:12.225 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph-authtool: stdout AQAbhfVltA/SIBAAqFfCwT3wWISLJXXRnp3opQ== 2024-03-16T11:40:13.121 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph-authtool: stdout AQAchfVl3qoPIRAA8ip5ASwpQwHAkSNrLT6aqg== 2024-03-16T11:40:13.122 INFO:teuthology.orchestra.run.smithi029.stdout:Creating initial monmap... 2024-03-16T11:40:14.059 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-16T11:40:14.059 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-03-16T11:40:14.059 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to d1887f28-e389-11ee-95c9-87774f69a715 2024-03-16T11:40:14.059 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-16T11:40:14.059 INFO:teuthology.orchestra.run.smithi029.stdout:monmaptool for a [v2:172.21.15.29:3300,v1:172.21.15.29:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-16T11:40:14.059 INFO:teuthology.orchestra.run.smithi029.stdout:setting min_mon_release = quincy 2024-03-16T11:40:14.059 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/monmaptool: set fsid to d1887f28-e389-11ee-95c9-87774f69a715 2024-03-16T11:40:14.059 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-16T11:40:14.059 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-16T11:40:14.059 INFO:teuthology.orchestra.run.smithi029.stdout:Creating mon... 2024-03-16T11:40:14.954 INFO:teuthology.orchestra.run.smithi029.stdout:create mon.a on 2024-03-16T11:40:15.173 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-03-16T11:40:15.335 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-03-16T11:40:15.528 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-d1887f28-e389-11ee-95c9-87774f69a715.target → /etc/systemd/system/ceph-d1887f28-e389-11ee-95c9-87774f69a715.target. 2024-03-16T11:40:15.528 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-d1887f28-e389-11ee-95c9-87774f69a715.target → /etc/systemd/system/ceph-d1887f28-e389-11ee-95c9-87774f69a715.target. 2024-03-16T11:40:15.794 INFO:teuthology.orchestra.run.smithi029.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-d1887f28-e389-11ee-95c9-87774f69a715@mon.a 2024-03-16T11:40:15.795 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl: stderr Failed to reset failed state of unit ceph-d1887f28-e389-11ee-95c9-87774f69a715@mon.a.service: Unit ceph-d1887f28-e389-11ee-95c9-87774f69a715@mon.a.service not loaded. 2024-03-16T11:40:15.966 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-d1887f28-e389-11ee-95c9-87774f69a715.target.wants/ceph-d1887f28-e389-11ee-95c9-87774f69a715@mon.a.service → /etc/systemd/system/ceph-d1887f28-e389-11ee-95c9-87774f69a715@.service. 2024-03-16T11:40:16.502 INFO:teuthology.orchestra.run.smithi029.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-16T11:40:16.502 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl: stdout disabled 2024-03-16T11:40:16.508 INFO:teuthology.orchestra.run.smithi029.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-16T11:40:16.508 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl: stdout inactive 2024-03-16T11:40:16.508 INFO:teuthology.orchestra.run.smithi029.stdout:firewalld.service is not enabled 2024-03-16T11:40:16.508 INFO:teuthology.orchestra.run.smithi029.stdout:Not possible to enable service . firewalld.service is not available 2024-03-16T11:40:16.509 INFO:teuthology.orchestra.run.smithi029.stdout:Waiting for mon to start... 2024-03-16T11:40:16.509 INFO:teuthology.orchestra.run.smithi029.stdout:Waiting for mon... 2024-03-16T11:40:17.560 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:17 smithi029 ceph-mon[31592]: from='client.? 172.21.15.29:0/2503057652' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-03-16T11:40:17.569 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout cluster: 2024-03-16T11:40:17.569 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout id: d1887f28-e389-11ee-95c9-87774f69a715 2024-03-16T11:40:17.569 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-03-16T11:40:17.569 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout 2024-03-16T11:40:17.569 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout services: 2024-03-16T11:40:17.569 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.517953s) 2024-03-16T11:40:17.569 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-03-16T11:40:17.569 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-03-16T11:40:17.569 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout 2024-03-16T11:40:17.569 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout data: 2024-03-16T11:40:17.569 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-03-16T11:40:17.569 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-03-16T11:40:17.569 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-03-16T11:40:17.570 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout pgs: 2024-03-16T11:40:17.570 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout 2024-03-16T11:40:17.570 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:16.910+0000 7f7c2866a700 1 Processor -- start 2024-03-16T11:40:17.570 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:16.910+0000 7f7c2866a700 1 -- start start 2024-03-16T11:40:17.570 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:16.911+0000 7f7c2866a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7c20108790 0x7f7c2010ab80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:40:17.570 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:16.911+0000 7f7c2866a700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f7c2007be50 con 0x7f7c20108790 2024-03-16T11:40:17.570 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:16.911+0000 7f7c26406700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7c20108790 0x7f7c2010ab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:40:17.570 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:16.911+0000 7f7c26406700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7c20108790 0x7f7c2010ab80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45910/0 (socket says 172.21.15.29:45910) 2024-03-16T11:40:17.570 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:16.911+0000 7f7c26406700 1 -- 172.21.15.29:0/2559108550 learned_addr learned my addr 172.21.15.29:0/2559108550 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:40:17.570 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:16.911+0000 7f7c26406700 1 -- 172.21.15.29:0/2559108550 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c2010b0c0 con 0x7f7c20108790 2024-03-16T11:40:17.570 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:16.911+0000 7f7c26406700 1 --2- 172.21.15.29:0/2559108550 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7c20108790 0x7f7c2010ab80 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f7c1c0098a0 tx=0x7f7c1c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ce0afe812f79474 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:40:17.570 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:16.912+0000 7f7c25404700 1 -- 172.21.15.29:0/2559108550 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7c1c004160 con 0x7f7c20108790 2024-03-16T11:40:17.570 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:16.912+0000 7f7c25404700 1 -- 172.21.15.29:0/2559108550 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7c1c004300 con 0x7f7c20108790 2024-03-16T11:40:17.570 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:16.912+0000 7f7c25404700 1 -- 172.21.15.29:0/2559108550 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7c1c0036a0 con 0x7f7c20108790 2024-03-16T11:40:17.571 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:16.912+0000 7f7c2866a700 1 -- 172.21.15.29:0/2559108550 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7c20108790 msgr2=0x7f7c2010ab80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:40:17.571 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:16.912+0000 7f7c2866a700 1 --2- 172.21.15.29:0/2559108550 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7c20108790 0x7f7c2010ab80 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f7c1c0098a0 tx=0x7f7c1c009bb0 comp rx=0 tx=0).stop 2024-03-16T11:40:17.571 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:16.912+0000 7f7c2866a700 1 -- 172.21.15.29:0/2559108550 shutdown_connections 2024-03-16T11:40:17.571 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:16.912+0000 7f7c2866a700 1 --2- 172.21.15.29:0/2559108550 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7c20108790 0x7f7c2010ab80 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:40:17.571 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:16.912+0000 7f7c2866a700 1 -- 172.21.15.29:0/2559108550 >> 172.21.15.29:0/2559108550 conn(0x7f7c201021a0 msgr2=0x7f7c201045e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:40:17.571 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:16.912+0000 7f7c2866a700 1 -- 172.21.15.29:0/2559108550 shutdown_connections 2024-03-16T11:40:17.571 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:16.912+0000 7f7c2866a700 1 -- 172.21.15.29:0/2559108550 wait complete. 2024-03-16T11:40:17.571 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:16.913+0000 7f7c2866a700 1 Processor -- start 2024-03-16T11:40:17.571 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:16.913+0000 7f7c2866a700 1 -- start start 2024-03-16T11:40:17.571 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:16.913+0000 7f7c2866a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7c20108790 0x7f7c20108010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:40:17.571 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:16.913+0000 7f7c2866a700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f7c2010b530 con 0x7f7c20108790 2024-03-16T11:40:17.571 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:16.913+0000 7f7c26406700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7c20108790 0x7f7c20108010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:40:17.571 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:16.913+0000 7f7c26406700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7c20108790 0x7f7c20108010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45924/0 (socket says 172.21.15.29:45924) 2024-03-16T11:40:17.571 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:16.913+0000 7f7c26406700 1 -- 172.21.15.29:0/2503057652 learned_addr learned my addr 172.21.15.29:0/2503057652 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:40:17.571 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:16.913+0000 7f7c26406700 1 -- 172.21.15.29:0/2503057652 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c20108550 con 0x7f7c20108790 2024-03-16T11:40:17.572 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:16.914+0000 7f7c26406700 1 --2- 172.21.15.29:0/2503057652 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7c20108790 0x7f7c20108010 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f7c1c000c00 tx=0x7f7c1c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:40:17.572 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:16.914+0000 7f7c137fe700 1 -- 172.21.15.29:0/2503057652 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7c1c003db0 con 0x7f7c20108790 2024-03-16T11:40:17.572 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:16.914+0000 7f7c137fe700 1 -- 172.21.15.29:0/2503057652 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7c1c011630 con 0x7f7c20108790 2024-03-16T11:40:17.572 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:16.914+0000 7f7c137fe700 1 -- 172.21.15.29:0/2503057652 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7c1c011950 con 0x7f7c20108790 2024-03-16T11:40:17.572 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:16.914+0000 7f7c2866a700 1 -- 172.21.15.29:0/2503057652 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7c20106740 con 0x7f7c20108790 2024-03-16T11:40:17.572 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:16.914+0000 7f7c2866a700 1 -- 172.21.15.29:0/2503057652 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7c20106c80 con 0x7f7c20108790 2024-03-16T11:40:17.572 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:16.914+0000 7f7c137fe700 1 -- 172.21.15.29:0/2503057652 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f7c1c01a460 con 0x7f7c20108790 2024-03-16T11:40:17.572 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:16.914+0000 7f7c137fe700 1 -- 172.21.15.29:0/2503057652 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f7c1c01b780 con 0x7f7c20108790 2024-03-16T11:40:17.572 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:16.915+0000 7f7c2866a700 1 -- 172.21.15.29:0/2503057652 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7c20063900 con 0x7f7c20108790 2024-03-16T11:40:17.572 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:16.917+0000 7f7c137fe700 1 -- 172.21.15.29:0/2503057652 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f7c1c01ba20 con 0x7f7c20108790 2024-03-16T11:40:17.572 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:16.990+0000 7f7c2866a700 1 -- 172.21.15.29:0/2503057652 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "status"} v 0) -- 0x7f7c20194fe0 con 0x7f7c20108790 2024-03-16T11:40:17.572 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:16.990+0000 7f7c137fe700 1 -- 172.21.15.29:0/2503057652 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) ==== 54+0+317 (secure 0 0 0) 0x7f7c1c01bc00 con 0x7f7c20108790 2024-03-16T11:40:17.572 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:16.991+0000 7f7c2866a700 1 -- 172.21.15.29:0/2503057652 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7c20108790 msgr2=0x7f7c20108010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:40:17.572 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:16.991+0000 7f7c2866a700 1 --2- 172.21.15.29:0/2503057652 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7c20108790 0x7f7c20108010 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f7c1c000c00 tx=0x7f7c1c004750 comp rx=0 tx=0).stop 2024-03-16T11:40:17.572 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:16.991+0000 7f7c2866a700 1 -- 172.21.15.29:0/2503057652 shutdown_connections 2024-03-16T11:40:17.573 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:16.991+0000 7f7c2866a700 1 --2- 172.21.15.29:0/2503057652 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7c20108790 0x7f7c20108010 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:40:17.573 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:16.991+0000 7f7c2866a700 1 -- 172.21.15.29:0/2503057652 >> 172.21.15.29:0/2503057652 conn(0x7f7c201021a0 msgr2=0x7f7c20103070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:40:17.573 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:16.991+0000 7f7c2866a700 1 -- 172.21.15.29:0/2503057652 shutdown_connections 2024-03-16T11:40:17.573 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:16.991+0000 7f7c2866a700 1 -- 172.21.15.29:0/2503057652 wait complete. 2024-03-16T11:40:17.573 INFO:teuthology.orchestra.run.smithi029.stdout:mon is available 2024-03-16T11:40:17.573 INFO:teuthology.orchestra.run.smithi029.stdout:Assimilating anything we can from ceph.conf... 2024-03-16T11:40:18.642 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:18 smithi029 ceph-mon[31592]: from='client.? 172.21.15.29:0/1071180264' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-16T11:40:18.642 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:18 smithi029 ceph-mon[31592]: from='client.? 172.21.15.29:0/1071180264' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-03-16T11:40:18.651 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout 2024-03-16T11:40:18.651 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout [global] 2024-03-16T11:40:18.651 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout fsid = d1887f28-e389-11ee-95c9-87774f69a715 2024-03-16T11:40:18.651 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.29:3300,v1:172.21.15.29:6789] 2024-03-16T11:40:18.651 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-16T11:40:18.651 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-16T11:40:18.651 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-16T11:40:18.652 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-16T11:40:18.652 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout 2024-03-16T11:40:18.652 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-16T11:40:18.652 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-16T11:40:18.652 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout 2024-03-16T11:40:18.652 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout [osd] 2024-03-16T11:40:18.652 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-03-16T11:40:18.652 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-03-16T11:40:18.652 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-03-16T11:40:18.652 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-16T11:40:18.652 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-16T11:40:18.652 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:17.987+0000 7f8db056e700 1 Processor -- start 2024-03-16T11:40:18.652 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:17.987+0000 7f8db056e700 1 -- start start 2024-03-16T11:40:18.652 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:17.987+0000 7f8db056e700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8da8107d20 0x7f8da8108140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:40:18.653 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:17.987+0000 7f8db056e700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f8da8108710 con 0x7f8da8107d20 2024-03-16T11:40:18.653 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:17.988+0000 7f8dae30a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8da8107d20 0x7f8da8108140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:40:18.653 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:17.988+0000 7f8dae30a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8da8107d20 0x7f8da8108140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45926/0 (socket says 172.21.15.29:45926) 2024-03-16T11:40:18.653 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:17.988+0000 7f8dae30a700 1 -- 172.21.15.29:0/4186398564 learned_addr learned my addr 172.21.15.29:0/4186398564 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:40:18.653 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:17.988+0000 7f8dae30a700 1 -- 172.21.15.29:0/4186398564 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8da8108ed0 con 0x7f8da8107d20 2024-03-16T11:40:18.653 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:17.988+0000 7f8dae30a700 1 --2- 172.21.15.29:0/4186398564 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8da8107d20 0x7f8da8108140 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f8d980098a0 tx=0x7f8d98009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f062b7d3809e2659 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:40:18.653 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:17.988+0000 7f8dad308700 1 -- 172.21.15.29:0/4186398564 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8d98004160 con 0x7f8da8107d20 2024-03-16T11:40:18.653 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:17.988+0000 7f8dad308700 1 -- 172.21.15.29:0/4186398564 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8d98004300 con 0x7f8da8107d20 2024-03-16T11:40:18.653 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:17.988+0000 7f8dad308700 1 -- 172.21.15.29:0/4186398564 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8d980036a0 con 0x7f8da8107d20 2024-03-16T11:40:18.653 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:17.989+0000 7f8db056e700 1 -- 172.21.15.29:0/4186398564 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8da8107d20 msgr2=0x7f8da8108140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:40:18.653 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:17.989+0000 7f8db056e700 1 --2- 172.21.15.29:0/4186398564 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8da8107d20 0x7f8da8108140 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f8d980098a0 tx=0x7f8d98009bb0 comp rx=0 tx=0).stop 2024-03-16T11:40:18.653 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:17.989+0000 7f8db056e700 1 -- 172.21.15.29:0/4186398564 shutdown_connections 2024-03-16T11:40:18.653 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:17.989+0000 7f8db056e700 1 --2- 172.21.15.29:0/4186398564 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8da8107d20 0x7f8da8108140 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:40:18.653 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:17.989+0000 7f8db056e700 1 -- 172.21.15.29:0/4186398564 >> 172.21.15.29:0/4186398564 conn(0x7f8da8103300 msgr2=0x7f8da8105720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:40:18.653 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:17.989+0000 7f8db056e700 1 -- 172.21.15.29:0/4186398564 shutdown_connections 2024-03-16T11:40:18.654 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:17.989+0000 7f8db056e700 1 -- 172.21.15.29:0/4186398564 wait complete. 2024-03-16T11:40:18.654 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:17.989+0000 7f8db056e700 1 Processor -- start 2024-03-16T11:40:18.654 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:17.989+0000 7f8db056e700 1 -- start start 2024-03-16T11:40:18.654 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:17.990+0000 7f8db056e700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8da8107d20 0x7f8da819b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:40:18.654 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:17.990+0000 7f8db056e700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f8da8109400 con 0x7f8da8107d20 2024-03-16T11:40:18.654 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:17.990+0000 7f8dae30a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8da8107d20 0x7f8da819b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:40:18.654 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:17.990+0000 7f8dae30a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8da8107d20 0x7f8da819b4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45928/0 (socket says 172.21.15.29:45928) 2024-03-16T11:40:18.654 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:17.990+0000 7f8dae30a700 1 -- 172.21.15.29:0/1071180264 learned_addr learned my addr 172.21.15.29:0/1071180264 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:40:18.654 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:17.990+0000 7f8dae30a700 1 -- 172.21.15.29:0/1071180264 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8da819ba00 con 0x7f8da8107d20 2024-03-16T11:40:18.654 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:17.990+0000 7f8dae30a700 1 --2- 172.21.15.29:0/1071180264 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8da8107d20 0x7f8da819b4c0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f8d98000c00 tx=0x7f8d98004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:40:18.654 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:17.990+0000 7f8d9f7fe700 1 -- 172.21.15.29:0/1071180264 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8d98003db0 con 0x7f8da8107d20 2024-03-16T11:40:18.654 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:17.990+0000 7f8d9f7fe700 1 -- 172.21.15.29:0/1071180264 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8d98011630 con 0x7f8da8107d20 2024-03-16T11:40:18.654 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:17.990+0000 7f8db056e700 1 -- 172.21.15.29:0/1071180264 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8da819bc90 con 0x7f8da8107d20 2024-03-16T11:40:18.654 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:17.990+0000 7f8d9f7fe700 1 -- 172.21.15.29:0/1071180264 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8d98011950 con 0x7f8da8107d20 2024-03-16T11:40:18.654 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:17.990+0000 7f8db056e700 1 -- 172.21.15.29:0/1071180264 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8da819e930 con 0x7f8da8107d20 2024-03-16T11:40:18.655 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:17.991+0000 7f8d9f7fe700 1 -- 172.21.15.29:0/1071180264 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f8d9801a460 con 0x7f8da8107d20 2024-03-16T11:40:18.655 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:17.991+0000 7f8d9f7fe700 1 -- 172.21.15.29:0/1071180264 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f8d9801b780 con 0x7f8da8107d20 2024-03-16T11:40:18.655 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:17.991+0000 7f8db056e700 1 -- 172.21.15.29:0/1071180264 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8da8063900 con 0x7f8da8107d20 2024-03-16T11:40:18.655 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:17.994+0000 7f8d9f7fe700 1 -- 172.21.15.29:0/1071180264 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f8d98031020 con 0x7f8da8107d20 2024-03-16T11:40:18.655 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:18.064+0000 7f8db056e700 1 -- 172.21.15.29:0/1071180264 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7f8da819c170 con 0x7f8da8107d20 2024-03-16T11:40:18.655 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:18.067+0000 7f8d9f7fe700 1 -- 172.21.15.29:0/1071180264 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) ==== 70+0+429 (secure 0 0 0) 0x7f8da819c170 con 0x7f8da8107d20 2024-03-16T11:40:18.655 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:18.068+0000 7f8d9f7fe700 1 -- 172.21.15.29:0/1071180264 <== mon.0 v2:172.21.15.29:3300/0 8 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f8d9801a8e0 con 0x7f8da8107d20 2024-03-16T11:40:18.655 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:18.068+0000 7f8db056e700 1 -- 172.21.15.29:0/1071180264 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8da8107d20 msgr2=0x7f8da819b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:40:18.655 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:18.068+0000 7f8db056e700 1 --2- 172.21.15.29:0/1071180264 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8da8107d20 0x7f8da819b4c0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f8d98000c00 tx=0x7f8d98004750 comp rx=0 tx=0).stop 2024-03-16T11:40:18.655 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:18.069+0000 7f8db056e700 1 -- 172.21.15.29:0/1071180264 shutdown_connections 2024-03-16T11:40:18.655 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:18.069+0000 7f8db056e700 1 --2- 172.21.15.29:0/1071180264 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8da8107d20 0x7f8da819b4c0 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:40:18.655 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:18.069+0000 7f8db056e700 1 -- 172.21.15.29:0/1071180264 >> 172.21.15.29:0/1071180264 conn(0x7f8da8103300 msgr2=0x7f8da8192020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:40:18.655 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:18.069+0000 7f8db056e700 1 -- 172.21.15.29:0/1071180264 shutdown_connections 2024-03-16T11:40:18.655 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:18.069+0000 7f8db056e700 1 -- 172.21.15.29:0/1071180264 wait complete. 2024-03-16T11:40:18.655 INFO:teuthology.orchestra.run.smithi029.stdout:Generating new minimal ceph.conf... 2024-03-16T11:40:19.613 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:19.026+0000 7fb6e3c6c700 1 Processor -- start 2024-03-16T11:40:19.613 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:19.026+0000 7fb6e3c6c700 1 -- start start 2024-03-16T11:40:19.613 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:19.026+0000 7fb6e3c6c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb6dc108780 0x7fb6dc10ab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:40:19.613 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:19.026+0000 7fb6e3c6c700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fb6dc07be70 con 0x7fb6dc108780 2024-03-16T11:40:19.613 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:19.026+0000 7fb6e1a08700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb6dc108780 0x7fb6dc10ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:40:19.613 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:19.026+0000 7fb6e1a08700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb6dc108780 0x7fb6dc10ab70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45932/0 (socket says 172.21.15.29:45932) 2024-03-16T11:40:19.613 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:19.026+0000 7fb6e1a08700 1 -- 172.21.15.29:0/30932923 learned_addr learned my addr 172.21.15.29:0/30932923 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:40:19.613 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:19.027+0000 7fb6e1a08700 1 -- 172.21.15.29:0/30932923 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6dc10b0b0 con 0x7fb6dc108780 2024-03-16T11:40:19.613 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:19.027+0000 7fb6e1a08700 1 --2- 172.21.15.29:0/30932923 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb6dc108780 0x7fb6dc10ab70 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fb6cc0098a0 tx=0x7fb6cc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63eb8970e96bfac1 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:40:19.614 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:19.027+0000 7fb6e0a06700 1 -- 172.21.15.29:0/30932923 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb6cc004160 con 0x7fb6dc108780 2024-03-16T11:40:19.614 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:19.027+0000 7fb6e0a06700 1 -- 172.21.15.29:0/30932923 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fb6cc004300 con 0x7fb6dc108780 2024-03-16T11:40:19.614 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:19.027+0000 7fb6e0a06700 1 -- 172.21.15.29:0/30932923 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb6cc011b50 con 0x7fb6dc108780 2024-03-16T11:40:19.614 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:19.028+0000 7fb6e3c6c700 1 -- 172.21.15.29:0/30932923 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb6dc108780 msgr2=0x7fb6dc10ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:40:19.614 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:19.028+0000 7fb6e3c6c700 1 --2- 172.21.15.29:0/30932923 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb6dc108780 0x7fb6dc10ab70 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fb6cc0098a0 tx=0x7fb6cc009bb0 comp rx=0 tx=0).stop 2024-03-16T11:40:19.614 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:19.028+0000 7fb6e3c6c700 1 -- 172.21.15.29:0/30932923 shutdown_connections 2024-03-16T11:40:19.614 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:19.028+0000 7fb6e3c6c700 1 --2- 172.21.15.29:0/30932923 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb6dc108780 0x7fb6dc10ab70 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:40:19.614 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:19.028+0000 7fb6e3c6c700 1 -- 172.21.15.29:0/30932923 >> 172.21.15.29:0/30932923 conn(0x7fb6dc1021b0 msgr2=0x7fb6dc1045f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:40:19.615 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:19.028+0000 7fb6e3c6c700 1 -- 172.21.15.29:0/30932923 shutdown_connections 2024-03-16T11:40:19.615 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:19.028+0000 7fb6e3c6c700 1 -- 172.21.15.29:0/30932923 wait complete. 2024-03-16T11:40:19.615 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:19.028+0000 7fb6e3c6c700 1 Processor -- start 2024-03-16T11:40:19.615 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:19.028+0000 7fb6e3c6c700 1 -- start start 2024-03-16T11:40:19.615 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:19.028+0000 7fb6e3c6c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb6dc108780 0x7fb6dc19bd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:40:19.615 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:19.028+0000 7fb6e3c6c700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fb6dc10c0e0 con 0x7fb6dc108780 2024-03-16T11:40:19.615 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:19.029+0000 7fb6e1a08700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb6dc108780 0x7fb6dc19bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:40:19.615 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:19.029+0000 7fb6e1a08700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb6dc108780 0x7fb6dc19bd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45934/0 (socket says 172.21.15.29:45934) 2024-03-16T11:40:19.615 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:19.029+0000 7fb6e1a08700 1 -- 172.21.15.29:0/965081858 learned_addr learned my addr 172.21.15.29:0/965081858 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:40:19.615 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:19.029+0000 7fb6e1a08700 1 -- 172.21.15.29:0/965081858 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6dc19c250 con 0x7fb6dc108780 2024-03-16T11:40:19.615 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:19.029+0000 7fb6e1a08700 1 --2- 172.21.15.29:0/965081858 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb6dc108780 0x7fb6dc19bd10 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fb6cc009fb0 tx=0x7fb6cc011bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:40:19.615 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:19.029+0000 7fb6d2ffd700 1 -- 172.21.15.29:0/965081858 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb6cc0038b0 con 0x7fb6dc108780 2024-03-16T11:40:19.616 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:19.029+0000 7fb6d2ffd700 1 -- 172.21.15.29:0/965081858 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fb6cc011e50 con 0x7fb6dc108780 2024-03-16T11:40:19.616 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:19.029+0000 7fb6e3c6c700 1 -- 172.21.15.29:0/965081858 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb6dc19c4e0 con 0x7fb6dc108780 2024-03-16T11:40:19.616 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:19.029+0000 7fb6d2ffd700 1 -- 172.21.15.29:0/965081858 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb6cc019420 con 0x7fb6dc108780 2024-03-16T11:40:19.616 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:19.029+0000 7fb6e3c6c700 1 -- 172.21.15.29:0/965081858 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb6dc19f1c0 con 0x7fb6dc108780 2024-03-16T11:40:19.616 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:19.029+0000 7fb6d2ffd700 1 -- 172.21.15.29:0/965081858 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fb6cc019ad0 con 0x7fb6dc108780 2024-03-16T11:40:19.616 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:19.029+0000 7fb6d2ffd700 1 -- 172.21.15.29:0/965081858 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fb6cc024730 con 0x7fb6dc108780 2024-03-16T11:40:19.617 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:19.030+0000 7fb6e3c6c700 1 -- 172.21.15.29:0/965081858 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb6dc063900 con 0x7fb6dc108780 2024-03-16T11:40:19.617 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:19.032+0000 7fb6d2ffd700 1 -- 172.21.15.29:0/965081858 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7fb6cc043dd0 con 0x7fb6dc108780 2024-03-16T11:40:19.617 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:19.103+0000 7fb6e3c6c700 1 -- 172.21.15.29:0/965081858 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7fb6dc19f390 con 0x7fb6dc108780 2024-03-16T11:40:19.617 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:19.103+0000 7fb6d2ffd700 1 -- 172.21.15.29:0/965081858 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) ==== 76+0+175 (secure 0 0 0) 0x7fb6cc032030 con 0x7fb6dc108780 2024-03-16T11:40:19.617 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:19.104+0000 7fb6e3c6c700 1 -- 172.21.15.29:0/965081858 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb6dc108780 msgr2=0x7fb6dc19bd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:40:19.617 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:19.104+0000 7fb6e3c6c700 1 --2- 172.21.15.29:0/965081858 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb6dc108780 0x7fb6dc19bd10 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fb6cc009fb0 tx=0x7fb6cc011bf0 comp rx=0 tx=0).stop 2024-03-16T11:40:19.617 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:19.104+0000 7fb6e3c6c700 1 -- 172.21.15.29:0/965081858 shutdown_connections 2024-03-16T11:40:19.617 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:19.104+0000 7fb6e3c6c700 1 --2- 172.21.15.29:0/965081858 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb6dc108780 0x7fb6dc19bd10 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:40:19.617 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:19.104+0000 7fb6e3c6c700 1 -- 172.21.15.29:0/965081858 >> 172.21.15.29:0/965081858 conn(0x7fb6dc1021b0 msgr2=0x7fb6dc102cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:40:19.617 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:19.104+0000 7fb6e3c6c700 1 -- 172.21.15.29:0/965081858 shutdown_connections 2024-03-16T11:40:19.618 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:19.104+0000 7fb6e3c6c700 1 -- 172.21.15.29:0/965081858 wait complete. 2024-03-16T11:40:19.618 INFO:teuthology.orchestra.run.smithi029.stdout:Restarting the monitor... 2024-03-16T11:40:19.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:19 smithi029 ceph-mon[31592]: from='client.? 172.21.15.29:0/965081858' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:40:19.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:19 smithi029 systemd[1]: Stopping Ceph mon.a for d1887f28-e389-11ee-95c9-87774f69a715... 2024-03-16T11:40:20.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:19 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mon-a[31588]: 2024-03-16T11:40:19.876+0000 7f8cfcd0e700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-16T11:40:20.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:19 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mon-a[31588]: 2024-03-16T11:40:19.876+0000 7f8cfcd0e700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-03-16T11:40:20.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:19 smithi029 podman[31837]: 2024-03-16 11:40:19.969569747 +0000 UTC m=+0.195174031 container died b7037c935f23bf42ef895702e9ccb93ae9b9c7690e4b3672c651255d499a5cf3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af, name=ceph-d1887f28-e389-11ee-95c9-87774f69a715-mon-a, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307) 2024-03-16T11:40:23.840 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:23 smithi029 podman[31837]: 2024-03-16 11:40:23.383955456 +0000 UTC m=+3.609559739 container cleanup b7037c935f23bf42ef895702e9ccb93ae9b9c7690e4b3672c651255d499a5cf3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af, name=ceph-d1887f28-e389-11ee-95c9-87774f69a715-mon-a, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307) 2024-03-16T11:40:23.840 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:23 smithi029 bash[31837]: ceph-d1887f28-e389-11ee-95c9-87774f69a715-mon-a 2024-03-16T11:40:27.164 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:27 smithi029 podman[31851]: 2024-03-16 11:40:27.03609039 +0000 UTC m=+7.063281370 container remove b7037c935f23bf42ef895702e9ccb93ae9b9c7690e4b3672c651255d499a5cf3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af, name=ceph-d1887f28-e389-11ee-95c9-87774f69a715-mon-a, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD) 2024-03-16T11:40:27.537 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:27 smithi029 systemd[1]: ceph-d1887f28-e389-11ee-95c9-87774f69a715@mon.a.service: Deactivated successfully. 2024-03-16T11:40:27.537 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:27 smithi029 systemd[1]: Stopped Ceph mon.a for d1887f28-e389-11ee-95c9-87774f69a715. 2024-03-16T11:40:27.537 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:27 smithi029 systemd[1]: Starting Ceph mon.a for d1887f28-e389-11ee-95c9-87774f69a715... 2024-03-16T11:40:27.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:27 smithi029 podman[31955]: 2024-03-16 11:40:27.441896902 +0000 UTC m=+0.024039091 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af 2024-03-16T11:40:29.588 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:29 smithi029 podman[31955]: 2024-03-16 11:40:29.286991512 +0000 UTC m=+1.869133697 container create 4277a6990573405f3d85e1862504a887d6b73d20f334d2d4896eac22a55e389d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af, name=ceph-d1887f28-e389-11ee-95c9-87774f69a715-mon-a, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240307, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2) 2024-03-16T11:40:31.839 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 podman[31955]: 2024-03-16 11:40:31.584199944 +0000 UTC m=+4.166342135 container init 4277a6990573405f3d85e1862504a887d6b73d20f334d2d4896eac22a55e389d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af, name=ceph-d1887f28-e389-11ee-95c9-87774f69a715-mon-a, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.build-date=20240307) 2024-03-16T11:40:31.839 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 podman[31955]: 2024-03-16 11:40:31.587821226 +0000 UTC m=+4.169963411 container start 4277a6990573405f3d85e1862504a887d6b73d20f334d2d4896eac22a55e389d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af, name=ceph-d1887f28-e389-11ee-95c9-87774f69a715-mon-a, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-03-16T11:40:31.839 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: set uid:gid to 167:167 (ceph:ceph) 2024-03-16T11:40:31.839 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: ceph version 19.0.0-2162-g3d78f5ba (3d78f5ba07bf18880172c07d4d7b4ac00c9d30af) squid (dev), process ceph-mon, pid 2 2024-03-16T11:40:31.839 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: pidfile_write: ignore empty --pid-file 2024-03-16T11:40:31.839 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: load: jerasure load: lrc 2024-03-16T11:40:31.839 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: rocksdb: RocksDB version: 7.9.2 2024-03-16T11:40:31.839 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: rocksdb: Git sha 0 2024-03-16T11:40:31.839 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: rocksdb: Compile date 2024-03-15 00:33:34 2024-03-16T11:40:31.840 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: rocksdb: DB SUMMARY 2024-03-16T11:40:31.840 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: rocksdb: DB Session ID: QGQ28C6UP1LGQS0COUWZ 2024-03-16T11:40:31.840 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: rocksdb: CURRENT file: CURRENT 2024-03-16T11:40:31.840 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: rocksdb: IDENTITY file: IDENTITY 2024-03-16T11:40:31.840 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-03-16T11:40:31.840 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-03-16T11:40:31.840 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 105338 ; 2024-03-16T11:40:31.840 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: rocksdb: Options.error_if_exists: 0 2024-03-16T11:40:31.840 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: rocksdb: Options.create_if_missing: 0 2024-03-16T11:40:31.840 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: rocksdb: Options.paranoid_checks: 1 2024-03-16T11:40:31.840 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-16T11:40:31.840 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-16T11:40:31.840 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-16T11:40:31.841 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: rocksdb: Options.env: 0x555f94caec60 2024-03-16T11:40:31.841 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: rocksdb: Options.fs: PosixFileSystem 2024-03-16T11:40:31.841 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: rocksdb: Options.info_log: 0x555f96e8dac0 2024-03-16T11:40:31.841 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: rocksdb: Options.max_file_opening_threads: 16 2024-03-16T11:40:31.841 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: rocksdb: Options.statistics: (nil) 2024-03-16T11:40:31.841 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: rocksdb: Options.use_fsync: 0 2024-03-16T11:40:31.841 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: rocksdb: Options.max_log_file_size: 0 2024-03-16T11:40:31.841 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-16T11:40:31.841 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-16T11:40:31.841 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: rocksdb: Options.keep_log_file_num: 1000 2024-03-16T11:40:31.841 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: rocksdb: Options.recycle_log_file_num: 0 2024-03-16T11:40:31.841 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: rocksdb: Options.allow_fallocate: 1 2024-03-16T11:40:31.841 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: rocksdb: Options.allow_mmap_reads: 0 2024-03-16T11:40:31.841 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: rocksdb: Options.allow_mmap_writes: 0 2024-03-16T11:40:31.841 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: rocksdb: Options.use_direct_reads: 0 2024-03-16T11:40:31.842 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-16T11:40:31.842 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: rocksdb: Options.create_missing_column_families: 0 2024-03-16T11:40:31.842 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: rocksdb: Options.db_log_dir: 2024-03-16T11:40:31.842 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: rocksdb: Options.wal_dir: 2024-03-16T11:40:31.842 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-16T11:40:31.842 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-16T11:40:31.842 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-16T11:40:31.842 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-16T11:40:31.842 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-16T11:40:31.842 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-16T11:40:31.843 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: rocksdb: Options.advise_random_on_open: 1 2024-03-16T11:40:31.843 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: rocksdb: Options.db_write_buffer_size: 0 2024-03-16T11:40:31.843 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: rocksdb: Options.write_buffer_manager: 0x555f96e601e0 2024-03-16T11:40:31.843 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-16T11:40:31.843 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-16T11:40:31.843 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-16T11:40:31.843 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: rocksdb: Options.rate_limiter: (nil) 2024-03-16T11:40:31.843 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-16T11:40:31.843 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: rocksdb: Options.wal_recovery_mode: 2 2024-03-16T11:40:31.843 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: rocksdb: Options.enable_thread_tracking: 0 2024-03-16T11:40:31.844 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: rocksdb: Options.enable_pipelined_write: 0 2024-03-16T11:40:31.844 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: rocksdb: Options.unordered_write: 0 2024-03-16T11:40:31.844 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-16T11:40:31.844 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-16T11:40:31.844 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-16T11:40:31.844 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-16T11:40:31.844 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: rocksdb: Options.row_cache: None 2024-03-16T11:40:31.844 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: rocksdb: Options.wal_filter: None 2024-03-16T11:40:31.845 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-16T11:40:31.845 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: rocksdb: Options.allow_ingest_behind: 0 2024-03-16T11:40:31.845 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: rocksdb: Options.two_write_queues: 0 2024-03-16T11:40:31.845 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: rocksdb: Options.manual_wal_flush: 0 2024-03-16T11:40:31.845 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: rocksdb: Options.wal_compression: 0 2024-03-16T11:40:31.846 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: rocksdb: Options.atomic_flush: 0 2024-03-16T11:40:31.846 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-16T11:40:31.846 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-16T11:40:31.846 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-16T11:40:31.846 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: rocksdb: Options.log_readahead_size: 0 2024-03-16T11:40:31.846 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-16T11:40:31.846 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: rocksdb: Options.best_efforts_recovery: 0 2024-03-16T11:40:31.846 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-16T11:40:31.846 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-16T11:40:31.846 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: rocksdb: Options.allow_data_in_errors: 0 2024-03-16T11:40:31.846 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: rocksdb: Options.db_host_id: __hostname__ 2024-03-16T11:40:31.846 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-16T11:40:31.846 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: rocksdb: Options.max_background_jobs: 2 2024-03-16T11:40:31.846 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: rocksdb: Options.max_background_compactions: -1 2024-03-16T11:40:31.846 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: rocksdb: Options.max_subcompactions: 1 2024-03-16T11:40:31.847 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-16T11:40:31.847 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-16T11:40:31.847 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-16T11:40:31.847 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: rocksdb: Options.max_total_wal_size: 0 2024-03-16T11:40:31.847 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-16T11:40:31.847 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-16T11:40:31.847 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-16T11:40:31.847 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-16T11:40:31.847 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: rocksdb: Options.max_open_files: -1 2024-03-16T11:40:31.847 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: rocksdb: Options.bytes_per_sync: 0 2024-03-16T11:40:31.847 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-16T11:40:31.847 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-16T11:40:31.847 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: rocksdb: Options.compaction_readahead_size: 0 2024-03-16T11:40:31.847 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: rocksdb: Options.max_background_flushes: -1 2024-03-16T11:40:31.848 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: rocksdb: Compression algorithms supported: 2024-03-16T11:40:31.848 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-16T11:40:31.848 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: rocksdb: kZSTD supported: 0 2024-03-16T11:40:31.848 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: rocksdb: kXpressCompression supported: 0 2024-03-16T11:40:31.848 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: rocksdb: kLZ4HCCompression supported: 1 2024-03-16T11:40:31.848 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: rocksdb: kZlibCompression supported: 1 2024-03-16T11:40:31.848 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: rocksdb: kSnappyCompression supported: 1 2024-03-16T11:40:31.848 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: rocksdb: kLZ4Compression supported: 1 2024-03-16T11:40:31.848 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: rocksdb: kBZip2Compression supported: 0 2024-03-16T11:40:31.848 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-16T11:40:31.848 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-16T11:40:31.848 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-03-16T11:40:31.848 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-16T11:40:31.849 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-16T11:40:31.849 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: rocksdb: Options.merge_operator: 2024-03-16T11:40:31.849 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: rocksdb: Options.compaction_filter: None 2024-03-16T11:40:31.849 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: rocksdb: Options.compaction_filter_factory: None 2024-03-16T11:40:31.849 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: rocksdb: Options.sst_partitioner_factory: None 2024-03-16T11:40:31.849 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-16T11:40:31.849 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-16T11:40:31.849 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x555f96e8dbc0) 2024-03-16T11:40:31.849 INFO:journalctl@ceph.mon.a.smithi029.stdout: cache_index_and_filter_blocks: 1 2024-03-16T11:40:31.849 INFO:journalctl@ceph.mon.a.smithi029.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-16T11:40:31.849 INFO:journalctl@ceph.mon.a.smithi029.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-16T11:40:31.850 INFO:journalctl@ceph.mon.a.smithi029.stdout: pin_top_level_index_and_filter: 1 2024-03-16T11:40:31.850 INFO:journalctl@ceph.mon.a.smithi029.stdout: index_type: 0 2024-03-16T11:40:31.850 INFO:journalctl@ceph.mon.a.smithi029.stdout: data_block_index_type: 0 2024-03-16T11:40:31.850 INFO:journalctl@ceph.mon.a.smithi029.stdout: index_shortening: 1 2024-03-16T11:40:31.850 INFO:journalctl@ceph.mon.a.smithi029.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-16T11:40:31.850 INFO:journalctl@ceph.mon.a.smithi029.stdout: checksum: 4 2024-03-16T11:40:31.850 INFO:journalctl@ceph.mon.a.smithi029.stdout: no_block_cache: 0 2024-03-16T11:40:31.850 INFO:journalctl@ceph.mon.a.smithi029.stdout: block_cache: 0x555f96e93090 2024-03-16T11:40:31.850 INFO:journalctl@ceph.mon.a.smithi029.stdout: block_cache_name: BinnedLRUCache 2024-03-16T11:40:31.850 INFO:journalctl@ceph.mon.a.smithi029.stdout: block_cache_options: 2024-03-16T11:40:31.850 INFO:journalctl@ceph.mon.a.smithi029.stdout: capacity : 536870912 2024-03-16T11:40:31.850 INFO:journalctl@ceph.mon.a.smithi029.stdout: num_shard_bits : 4 2024-03-16T11:40:31.850 INFO:journalctl@ceph.mon.a.smithi029.stdout: strict_capacity_limit : 0 2024-03-16T11:40:31.851 INFO:journalctl@ceph.mon.a.smithi029.stdout: high_pri_pool_ratio: 0.000 2024-03-16T11:40:31.851 INFO:journalctl@ceph.mon.a.smithi029.stdout: block_cache_compressed: (nil) 2024-03-16T11:40:31.851 INFO:journalctl@ceph.mon.a.smithi029.stdout: persistent_cache: (nil) 2024-03-16T11:40:31.851 INFO:journalctl@ceph.mon.a.smithi029.stdout: block_size: 4096 2024-03-16T11:40:31.851 INFO:journalctl@ceph.mon.a.smithi029.stdout: block_size_deviation: 10 2024-03-16T11:40:31.851 INFO:journalctl@ceph.mon.a.smithi029.stdout: block_restart_interval: 16 2024-03-16T11:40:31.851 INFO:journalctl@ceph.mon.a.smithi029.stdout: index_block_restart_interval: 1 2024-03-16T11:40:31.851 INFO:journalctl@ceph.mon.a.smithi029.stdout: metadata_block_size: 4096 2024-03-16T11:40:31.851 INFO:journalctl@ceph.mon.a.smithi029.stdout: partition_filters: 0 2024-03-16T11:40:31.851 INFO:journalctl@ceph.mon.a.smithi029.stdout: use_delta_encoding: 1 2024-03-16T11:40:31.851 INFO:journalctl@ceph.mon.a.smithi029.stdout: filter_policy: bloomfilter 2024-03-16T11:40:31.851 INFO:journalctl@ceph.mon.a.smithi029.stdout: whole_key_filtering: 1 2024-03-16T11:40:31.851 INFO:journalctl@ceph.mon.a.smithi029.stdout: verify_compression: 0 2024-03-16T11:40:31.851 INFO:journalctl@ceph.mon.a.smithi029.stdout: read_amp_bytes_per_bit: 0 2024-03-16T11:40:31.851 INFO:journalctl@ceph.mon.a.smithi029.stdout: format_version: 5 2024-03-16T11:40:31.851 INFO:journalctl@ceph.mon.a.smithi029.stdout: enable_index_compression: 1 2024-03-16T11:40:31.852 INFO:journalctl@ceph.mon.a.smithi029.stdout: block_align: 0 2024-03-16T11:40:31.852 INFO:journalctl@ceph.mon.a.smithi029.stdout: max_auto_readahead_size: 262144 2024-03-16T11:40:31.852 INFO:journalctl@ceph.mon.a.smithi029.stdout: prepopulate_block_cache: 0 2024-03-16T11:40:31.852 INFO:journalctl@ceph.mon.a.smithi029.stdout: initial_auto_readahead_size: 8192 2024-03-16T11:40:31.852 INFO:journalctl@ceph.mon.a.smithi029.stdout: num_file_reads_for_auto_readahead: 2 2024-03-16T11:40:31.852 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: rocksdb: Options.write_buffer_size: 33554432 2024-03-16T11:40:31.852 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: rocksdb: Options.max_write_buffer_number: 2 2024-03-16T11:40:31.852 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: rocksdb: Options.compression: NoCompression 2024-03-16T11:40:31.852 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: rocksdb: Options.bottommost_compression: Disabled 2024-03-16T11:40:31.852 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: rocksdb: Options.prefix_extractor: nullptr 2024-03-16T11:40:31.852 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-16T11:40:31.852 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: rocksdb: Options.num_levels: 7 2024-03-16T11:40:31.852 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-16T11:40:31.852 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-16T11:40:31.853 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-16T11:40:31.853 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-16T11:40:31.853 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-16T11:40:31.853 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-16T11:40:31.853 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-16T11:40:31.853 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-16T11:40:31.853 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-16T11:40:31.853 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-16T11:40:31.853 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-16T11:40:31.853 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-16T11:40:31.853 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-16T11:40:31.854 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: rocksdb: Options.compression_opts.level: 32767 2024-03-16T11:40:31.854 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: rocksdb: Options.compression_opts.strategy: 0 2024-03-16T11:40:31.854 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-16T11:40:31.854 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-16T11:40:31.854 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-16T11:40:31.854 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-16T11:40:31.854 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: rocksdb: Options.compression_opts.enabled: false 2024-03-16T11:40:31.854 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-16T11:40:31.854 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-16T11:40:31.854 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-16T11:40:31.854 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-16T11:40:31.854 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: rocksdb: Options.target_file_size_base: 67108864 2024-03-16T11:40:31.854 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-16T11:40:31.854 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-16T11:40:31.854 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-16T11:40:31.855 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-16T11:40:31.855 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-16T11:40:31.855 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-16T11:40:31.855 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-16T11:40:31.855 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-16T11:40:31.855 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-16T11:40:31.855 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-16T11:40:31.855 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-16T11:40:31.855 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-16T11:40:31.855 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-16T11:40:31.855 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-16T11:40:31.855 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: rocksdb: Options.arena_block_size: 1048576 2024-03-16T11:40:31.855 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-16T11:40:31.855 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-16T11:40:31.855 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: rocksdb: Options.disable_auto_compactions: 0 2024-03-16T11:40:31.855 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-16T11:40:31.856 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-16T11:40:31.856 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-16T11:40:31.856 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-16T11:40:31.856 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-16T11:40:31.856 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-16T11:40:31.856 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-16T11:40:31.856 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-16T11:40:31.856 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-16T11:40:31.856 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-16T11:40:31.856 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: rocksdb: Options.table_properties_collectors: 2024-03-16T11:40:31.856 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: rocksdb: Options.inplace_update_support: 0 2024-03-16T11:40:31.856 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-16T11:40:31.856 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-16T11:40:31.856 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-16T11:40:31.856 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-16T11:40:31.856 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: rocksdb: Options.bloom_locality: 0 2024-03-16T11:40:31.857 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: rocksdb: Options.max_successive_merges: 0 2024-03-16T11:40:31.857 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-16T11:40:31.857 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: rocksdb: Options.paranoid_file_checks: 0 2024-03-16T11:40:31.857 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: rocksdb: Options.force_consistency_checks: 1 2024-03-16T11:40:31.857 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: rocksdb: Options.report_bg_io_stats: 0 2024-03-16T11:40:31.857 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: rocksdb: Options.ttl: 2592000 2024-03-16T11:40:31.857 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-16T11:40:31.857 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-16T11:40:31.857 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-16T11:40:31.857 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: rocksdb: Options.enable_blob_files: false 2024-03-16T11:40:31.857 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: rocksdb: Options.min_blob_size: 0 2024-03-16T11:40:31.857 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: rocksdb: Options.blob_file_size: 268435456 2024-03-16T11:40:31.857 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-16T11:40:31.858 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-16T11:40:31.858 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-16T11:40:31.858 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-16T11:40:31.858 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-16T11:40:31.858 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: rocksdb: Options.blob_file_starting_level: 0 2024-03-16T11:40:31.858 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-16T11:40:31.858 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-03-16T11:40:31.858 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-03-16T11:40:31.858 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 848022e0-9701-4723-9c3f-317bd7770510 2024-03-16T11:40:31.858 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710589231622279, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-03-16T11:40:31.858 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-03-16T11:40:31.858 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710589231623301, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 100449, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 326, "table_properties": {"data_size": 98360, "index_size": 307, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 773, "raw_key_size": 14219, "raw_average_key_size": 49, "raw_value_size": 90624, "raw_average_value_size": 317, "num_data_blocks": 12, "num_entries": 285, "num_filter_entries": 285, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1710589231, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "848022e0-9701-4723-9c3f-317bd7770510", "db_session_id": "QGQ28C6UP1LGQS0COUWZ", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-03-16T11:40:31.858 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710589231623405, "job": 1, "event": "recovery_finished"} 2024-03-16T11:40:31.859 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-03-16T11:40:31.859 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-16T11:40:31.859 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x555f96f86000 2024-03-16T11:40:31.859 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: rocksdb: DB pointer 0x555f96f70000 2024-03-16T11:40:31.859 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-16T11:40:31.859 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-16T11:40:31.859 INFO:journalctl@ceph.mon.a.smithi029.stdout: ** DB Stats ** 2024-03-16T11:40:31.859 INFO:journalctl@ceph.mon.a.smithi029.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-16T11:40:31.859 INFO:journalctl@ceph.mon.a.smithi029.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-16T11:40:31.859 INFO:journalctl@ceph.mon.a.smithi029.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-16T11:40:31.859 INFO:journalctl@ceph.mon.a.smithi029.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-16T11:40:31.859 INFO:journalctl@ceph.mon.a.smithi029.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-16T11:40:31.859 INFO:journalctl@ceph.mon.a.smithi029.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-16T11:40:31.859 INFO:journalctl@ceph.mon.a.smithi029.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-16T11:40:31.860 INFO:journalctl@ceph.mon.a.smithi029.stdout: 2024-03-16T11:40:31.860 INFO:journalctl@ceph.mon.a.smithi029.stdout: ** Compaction Stats [default] ** 2024-03-16T11:40:31.860 INFO:journalctl@ceph.mon.a.smithi029.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-16T11:40:31.860 INFO:journalctl@ceph.mon.a.smithi029.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-16T11:40:31.860 INFO:journalctl@ceph.mon.a.smithi029.stdout: L0 2/0 99.93 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 143.8 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-16T11:40:31.860 INFO:journalctl@ceph.mon.a.smithi029.stdout: Sum 2/0 99.93 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 143.8 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-16T11:40:31.860 INFO:journalctl@ceph.mon.a.smithi029.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 143.8 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-16T11:40:31.860 INFO:journalctl@ceph.mon.a.smithi029.stdout: 2024-03-16T11:40:31.860 INFO:journalctl@ceph.mon.a.smithi029.stdout: ** Compaction Stats [default] ** 2024-03-16T11:40:31.860 INFO:journalctl@ceph.mon.a.smithi029.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-16T11:40:31.860 INFO:journalctl@ceph.mon.a.smithi029.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-16T11:40:31.860 INFO:journalctl@ceph.mon.a.smithi029.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 143.8 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-16T11:40:31.861 INFO:journalctl@ceph.mon.a.smithi029.stdout: 2024-03-16T11:40:31.861 INFO:journalctl@ceph.mon.a.smithi029.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-16T11:40:31.861 INFO:journalctl@ceph.mon.a.smithi029.stdout: 2024-03-16T11:40:31.861 INFO:journalctl@ceph.mon.a.smithi029.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-16T11:40:31.861 INFO:journalctl@ceph.mon.a.smithi029.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-16T11:40:31.861 INFO:journalctl@ceph.mon.a.smithi029.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-16T11:40:31.861 INFO:journalctl@ceph.mon.a.smithi029.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-16T11:40:31.861 INFO:journalctl@ceph.mon.a.smithi029.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-16T11:40:31.861 INFO:journalctl@ceph.mon.a.smithi029.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-16T11:40:31.861 INFO:journalctl@ceph.mon.a.smithi029.stdout: Cumulative compaction: 0.00 GB write, 25.79 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-16T11:40:31.861 INFO:journalctl@ceph.mon.a.smithi029.stdout: Interval compaction: 0.00 GB write, 25.79 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-16T11:40:31.861 INFO:journalctl@ceph.mon.a.smithi029.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-16T11:40:31.861 INFO:journalctl@ceph.mon.a.smithi029.stdout: Block cache BinnedLRUCache@0x555f96e93090#2 capacity: 512.00 MB usage: 1.52 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.2e-05 secs_since: 0 2024-03-16T11:40:31.861 INFO:journalctl@ceph.mon.a.smithi029.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,1.02 KB,0.000193715%) IndexBlock(2,0.50 KB,9.53674e-05%) Misc(1,0.00 KB,0%) 2024-03-16T11:40:31.861 INFO:journalctl@ceph.mon.a.smithi029.stdout: 2024-03-16T11:40:31.861 INFO:journalctl@ceph.mon.a.smithi029.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-16T11:40:31.862 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: starting mon.a rank 0 at public addrs [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] at bind addrs [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid d1887f28-e389-11ee-95c9-87774f69a715 2024-03-16T11:40:31.862 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: mon.a@-1(???) e1 preinit fsid d1887f28-e389-11ee-95c9-87774f69a715 2024-03-16T11:40:31.862 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: mon.a@-1(???).mds e1 new map 2024-03-16T11:40:31.862 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: mon.a@-1(???).mds e1 print_map 2024-03-16T11:40:31.862 INFO:journalctl@ceph.mon.a.smithi029.stdout: e1 2024-03-16T11:40:31.862 INFO:journalctl@ceph.mon.a.smithi029.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-16T11:40:31.862 INFO:journalctl@ceph.mon.a.smithi029.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-03-16T11:40:31.862 INFO:journalctl@ceph.mon.a.smithi029.stdout: legacy client fscid: -1 2024-03-16T11:40:31.862 INFO:journalctl@ceph.mon.a.smithi029.stdout: 2024-03-16T11:40:31.862 INFO:journalctl@ceph.mon.a.smithi029.stdout: No filesystems configured 2024-03-16T11:40:31.862 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-16T11:40:31.862 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-16T11:40:31.862 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-16T11:40:31.862 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-16T11:40:31.862 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-03-16T11:40:31.862 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-16T11:40:31.863 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: monmap epoch 1 2024-03-16T11:40:31.863 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: fsid d1887f28-e389-11ee-95c9-87774f69a715 2024-03-16T11:40:31.863 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: last_changed 2024-03-16T11:40:13.432436+0000 2024-03-16T11:40:31.863 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: created 2024-03-16T11:40:13.432436+0000 2024-03-16T11:40:31.863 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: min_mon_release 19 (squid) 2024-03-16T11:40:31.863 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: election_strategy: 1 2024-03-16T11:40:31.863 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: 0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2024-03-16T11:40:31.863 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: fsmap 2024-03-16T11:40:31.863 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: osdmap e1: 0 total, 0 up, 0 in 2024-03-16T11:40:31.863 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:31 smithi029 ceph-mon[31975]: mgrmap e1: no daemons active 2024-03-16T11:40:33.366 INFO:teuthology.orchestra.run.smithi029.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-03-16T11:40:33.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:33 smithi029 bash[31955]: 4277a6990573405f3d85e1862504a887d6b73d20f334d2d4896eac22a55e389d 2024-03-16T11:40:33.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:33 smithi029 systemd[1]: Started Ceph mon.a for d1887f28-e389-11ee-95c9-87774f69a715. 2024-03-16T11:40:39.600 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:38.341+0000 7f750740c700 1 Processor -- start 2024-03-16T11:40:39.606 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:38.341+0000 7f750740c700 1 -- start start 2024-03-16T11:40:39.606 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:38.341+0000 7f750740c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f75001087f0 0x7f750010abe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:40:39.606 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:38.341+0000 7f750740c700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f750007be10 con 0x7f75001087f0 2024-03-16T11:40:39.606 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:38.342+0000 7f75051a8700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f75001087f0 0x7f750010abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:40:39.606 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:38.342+0000 7f75051a8700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f75001087f0 0x7f750010abe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:35694/0 (socket says 172.21.15.29:35694) 2024-03-16T11:40:39.606 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:38.342+0000 7f75051a8700 1 -- 172.21.15.29:0/839174987 learned_addr learned my addr 172.21.15.29:0/839174987 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:40:39.606 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:38.342+0000 7f75051a8700 1 -- 172.21.15.29:0/839174987 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f750010b120 con 0x7f75001087f0 2024-03-16T11:40:39.606 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:38.342+0000 7f75051a8700 1 --2- 172.21.15.29:0/839174987 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f75001087f0 0x7f750010abe0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f74f0009e80 tx=0x7f74f000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5d1868366694644 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:40:39.606 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:38.342+0000 7f74fffff700 1 -- 172.21.15.29:0/839174987 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f74f0017070 con 0x7f75001087f0 2024-03-16T11:40:39.607 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:38.342+0000 7f74fffff700 1 -- 172.21.15.29:0/839174987 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f74f000b8c0 con 0x7f75001087f0 2024-03-16T11:40:39.607 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:38.342+0000 7f74fffff700 1 -- 172.21.15.29:0/839174987 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f74f00045c0 con 0x7f75001087f0 2024-03-16T11:40:39.607 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:38.343+0000 7f750740c700 1 -- 172.21.15.29:0/839174987 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f75001087f0 msgr2=0x7f750010abe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:40:39.607 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:38.343+0000 7f750740c700 1 --2- 172.21.15.29:0/839174987 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f75001087f0 0x7f750010abe0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f74f0009e80 tx=0x7f74f000b330 comp rx=0 tx=0).stop 2024-03-16T11:40:39.607 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:38.343+0000 7f750740c700 1 -- 172.21.15.29:0/839174987 shutdown_connections 2024-03-16T11:40:39.607 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:38.343+0000 7f750740c700 1 --2- 172.21.15.29:0/839174987 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f75001087f0 0x7f750010abe0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:40:39.607 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:38.343+0000 7f750740c700 1 -- 172.21.15.29:0/839174987 >> 172.21.15.29:0/839174987 conn(0x7f75001021c0 msgr2=0x7f7500104600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:40:39.607 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:38.343+0000 7f750740c700 1 -- 172.21.15.29:0/839174987 shutdown_connections 2024-03-16T11:40:39.607 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:38.343+0000 7f750740c700 1 -- 172.21.15.29:0/839174987 wait complete. 2024-03-16T11:40:39.607 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:38.344+0000 7f750740c700 1 Processor -- start 2024-03-16T11:40:39.607 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:38.344+0000 7f750740c700 1 -- start start 2024-03-16T11:40:39.607 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:38.344+0000 7f750740c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f75001087f0 0x7f750019bcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:40:39.608 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:38.344+0000 7f750740c700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f750010c0f0 con 0x7f75001087f0 2024-03-16T11:40:39.608 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:38.344+0000 7f75051a8700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f75001087f0 0x7f750019bcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:40:39.608 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:38.344+0000 7f75051a8700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f75001087f0 0x7f750019bcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:35706/0 (socket says 172.21.15.29:35706) 2024-03-16T11:40:39.608 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:38.344+0000 7f75051a8700 1 -- 172.21.15.29:0/1295810821 learned_addr learned my addr 172.21.15.29:0/1295810821 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:40:39.608 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:38.344+0000 7f75051a8700 1 -- 172.21.15.29:0/1295810821 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f750019c1f0 con 0x7f75001087f0 2024-03-16T11:40:39.608 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:38.344+0000 7f75051a8700 1 --2- 172.21.15.29:0/1295810821 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f75001087f0 0x7f750019bcb0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f74f0009e50 tx=0x7f74f000bd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:40:39.608 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:38.344+0000 7f74fe7fc700 1 -- 172.21.15.29:0/1295810821 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f74f0020020 con 0x7f75001087f0 2024-03-16T11:40:39.608 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:38.345+0000 7f74fe7fc700 1 -- 172.21.15.29:0/1295810821 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f74f0003b50 con 0x7f75001087f0 2024-03-16T11:40:39.608 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:38.345+0000 7f750740c700 1 -- 172.21.15.29:0/1295810821 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f750019c480 con 0x7f75001087f0 2024-03-16T11:40:39.608 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:38.345+0000 7f74fe7fc700 1 -- 172.21.15.29:0/1295810821 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f74f0017040 con 0x7f75001087f0 2024-03-16T11:40:39.608 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:38.345+0000 7f750740c700 1 -- 172.21.15.29:0/1295810821 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f750019f170 con 0x7f75001087f0 2024-03-16T11:40:39.608 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:38.345+0000 7f74fe7fc700 1 -- 172.21.15.29:0/1295810821 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f74f0023020 con 0x7f75001087f0 2024-03-16T11:40:39.608 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:38.345+0000 7f74fe7fc700 1 -- 172.21.15.29:0/1295810821 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f74f000f050 con 0x7f75001087f0 2024-03-16T11:40:39.609 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:38.345+0000 7f750740c700 1 -- 172.21.15.29:0/1295810821 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7500063900 con 0x7f75001087f0 2024-03-16T11:40:39.609 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:38.348+0000 7f74fe7fc700 1 -- 172.21.15.29:0/1295810821 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f74f002a020 con 0x7f75001087f0 2024-03-16T11:40:39.609 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:38.418+0000 7f750740c700 1 -- 172.21.15.29:0/1295810821 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) -- 0x7f750019f7a0 con 0x7f75001087f0 2024-03-16T11:40:39.609 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:38.420+0000 7f74fe7fc700 1 -- 172.21.15.29:0/1295810821 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) ==== 124+0+0 (secure 0 0 0) 0x7f74f0003d00 con 0x7f75001087f0 2024-03-16T11:40:39.609 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:38.421+0000 7f750740c700 1 -- 172.21.15.29:0/1295810821 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f75001087f0 msgr2=0x7f750019bcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:40:39.609 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:38.421+0000 7f750740c700 1 --2- 172.21.15.29:0/1295810821 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f75001087f0 0x7f750019bcb0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f74f0009e50 tx=0x7f74f000bd30 comp rx=0 tx=0).stop 2024-03-16T11:40:39.609 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:38.421+0000 7f750740c700 1 -- 172.21.15.29:0/1295810821 shutdown_connections 2024-03-16T11:40:39.609 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:38.421+0000 7f750740c700 1 --2- 172.21.15.29:0/1295810821 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f75001087f0 0x7f750019bcb0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:40:39.609 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:38.421+0000 7f750740c700 1 -- 172.21.15.29:0/1295810821 >> 172.21.15.29:0/1295810821 conn(0x7f75001021c0 msgr2=0x7f7500102cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:40:39.609 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:38.421+0000 7f750740c700 1 -- 172.21.15.29:0/1295810821 shutdown_connections 2024-03-16T11:40:39.609 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:38.421+0000 7f750740c700 1 -- 172.21.15.29:0/1295810821 wait complete. 2024-03-16T11:40:39.609 INFO:teuthology.orchestra.run.smithi029.stdout:Wrote config to /etc/ceph/ceph.conf 2024-03-16T11:40:39.631 INFO:teuthology.orchestra.run.smithi029.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-03-16T11:40:39.631 INFO:teuthology.orchestra.run.smithi029.stdout:Creating mgr... 2024-03-16T11:40:39.631 INFO:teuthology.orchestra.run.smithi029.stdout:Verifying port 0.0.0.0:9283 ... 2024-03-16T11:40:39.632 INFO:teuthology.orchestra.run.smithi029.stdout:Verifying port 0.0.0.0:8765 ... 2024-03-16T11:40:39.716 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:39 smithi029 ceph-mon[31975]: from='client.? 172.21.15.29:0/1295810821' entity='client.admin' 2024-03-16T11:40:39.889 INFO:teuthology.orchestra.run.smithi029.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-d1887f28-e389-11ee-95c9-87774f69a715@mgr.x 2024-03-16T11:40:39.889 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl: stderr Failed to reset failed state of unit ceph-d1887f28-e389-11ee-95c9-87774f69a715@mgr.x.service: Unit ceph-d1887f28-e389-11ee-95c9-87774f69a715@mgr.x.service not loaded. 2024-03-16T11:40:40.028 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-d1887f28-e389-11ee-95c9-87774f69a715.target.wants/ceph-d1887f28-e389-11ee-95c9-87774f69a715@mgr.x.service → /etc/systemd/system/ceph-d1887f28-e389-11ee-95c9-87774f69a715@.service. 2024-03-16T11:40:40.338 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 16 11:40:40 smithi029 podman[32196]: 2024-03-16 11:40:40.16557726 +0000 UTC m=+0.017095058 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af 2024-03-16T11:40:40.554 INFO:teuthology.orchestra.run.smithi029.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-16T11:40:40.554 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl: stdout disabled 2024-03-16T11:40:40.561 INFO:teuthology.orchestra.run.smithi029.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-16T11:40:40.561 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl: stdout inactive 2024-03-16T11:40:40.561 INFO:teuthology.orchestra.run.smithi029.stdout:firewalld.service is not enabled 2024-03-16T11:40:40.561 INFO:teuthology.orchestra.run.smithi029.stdout:Not possible to enable service . firewalld.service is not available 2024-03-16T11:40:40.569 INFO:teuthology.orchestra.run.smithi029.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-16T11:40:40.569 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl: stdout disabled 2024-03-16T11:40:40.576 INFO:teuthology.orchestra.run.smithi029.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-16T11:40:40.576 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl: stdout inactive 2024-03-16T11:40:40.576 INFO:teuthology.orchestra.run.smithi029.stdout:firewalld.service is not enabled 2024-03-16T11:40:40.576 INFO:teuthology.orchestra.run.smithi029.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-03-16T11:40:40.576 INFO:teuthology.orchestra.run.smithi029.stdout:Waiting for mgr to start... 2024-03-16T11:40:40.576 INFO:teuthology.orchestra.run.smithi029.stdout:Waiting for mgr... 2024-03-16T11:40:40.701 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 16 11:40:40 smithi029 podman[32196]: 2024-03-16 11:40:40.441762034 +0000 UTC m=+0.293279820 container init 4ae154094330b5ca37cf71565bb7076bf30261ef14b9197acf83d63f1bf85c9f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af, name=ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-x, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-03-16T11:40:40.701 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 16 11:40:40 smithi029 podman[32196]: 2024-03-16 11:40:40.444236665 +0000 UTC m=+0.295754461 container start 4ae154094330b5ca37cf71565bb7076bf30261ef14b9197acf83d63f1bf85c9f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af, name=ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-x, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b) 2024-03-16T11:40:40.701 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 16 11:40:40 smithi029 bash[32196]: 4ae154094330b5ca37cf71565bb7076bf30261ef14b9197acf83d63f1bf85c9f 2024-03-16T11:40:40.701 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 16 11:40:40 smithi029 systemd[1]: Started Ceph mgr.x for d1887f28-e389-11ee-95c9-87774f69a715. 2024-03-16T11:40:41.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:41 smithi029 ceph-mon[31975]: from='client.? 172.21.15.29:0/1927148736' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-16T11:40:41.338 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 16 11:40:41 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-x[32210]: 2024-03-16T11:40:41.007+0000 7fb9410d4200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-16T11:40:41.338 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 16 11:40:41 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-x[32210]: 2024-03-16T11:40:41.106+0000 7fb9410d4200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-16T11:40:41.729 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout 2024-03-16T11:40:41.729 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout { 2024-03-16T11:40:41.729 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "fsid": "d1887f28-e389-11ee-95c9-87774f69a715", 2024-03-16T11:40:41.729 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "health": { 2024-03-16T11:40:41.729 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-16T11:40:41.729 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-16T11:40:41.730 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-16T11:40:41.730 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout }, 2024-03-16T11:40:41.730 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-16T11:40:41.730 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-16T11:40:41.730 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout 0 2024-03-16T11:40:41.730 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout ], 2024-03-16T11:40:41.730 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-16T11:40:41.730 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "a" 2024-03-16T11:40:41.730 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout ], 2024-03-16T11:40:41.730 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "quorum_age": 9, 2024-03-16T11:40:41.730 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-16T11:40:41.730 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-16T11:40:41.730 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-16T11:40:41.730 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-16T11:40:41.730 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout }, 2024-03-16T11:40:41.731 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-16T11:40:41.731 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-16T11:40:41.731 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-16T11:40:41.731 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-16T11:40:41.731 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-16T11:40:41.739 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-16T11:40:41.739 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-16T11:40:41.739 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-16T11:40:41.739 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout }, 2024-03-16T11:40:41.739 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-16T11:40:41.739 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-16T11:40:41.739 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-16T11:40:41.739 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-16T11:40:41.739 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-16T11:40:41.739 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-16T11:40:41.740 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-16T11:40:41.740 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-16T11:40:41.740 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-16T11:40:41.740 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout }, 2024-03-16T11:40:41.740 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-16T11:40:41.740 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-16T11:40:41.740 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-16T11:40:41.740 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-16T11:40:41.740 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout }, 2024-03-16T11:40:41.740 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-16T11:40:41.740 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-16T11:40:41.740 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-16T11:40:41.740 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-16T11:40:41.740 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-16T11:40:41.740 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-16T11:40:41.741 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "restful" 2024-03-16T11:40:41.741 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout ], 2024-03-16T11:40:41.741 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-16T11:40:41.741 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout }, 2024-03-16T11:40:41.741 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-16T11:40:41.741 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-16T11:40:41.741 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "modified": "2024-03-16T11:40:16.475564+0000", 2024-03-16T11:40:41.741 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-16T11:40:41.741 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout }, 2024-03-16T11:40:41.741 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-16T11:40:41.741 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout } 2024-03-16T11:40:41.741 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:41.025+0000 7f0a9f535700 1 Processor -- start 2024-03-16T11:40:41.741 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:41.025+0000 7f0a9f535700 1 -- start start 2024-03-16T11:40:41.741 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:41.025+0000 7f0a9f535700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0a981087f0 0x7f0a9810abe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:40:41.741 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:41.025+0000 7f0a9f535700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f0a9807be10 con 0x7f0a981087f0 2024-03-16T11:40:41.742 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:41.026+0000 7f0a9d2d1700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0a981087f0 0x7f0a9810abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:40:41.742 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:41.026+0000 7f0a9d2d1700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0a981087f0 0x7f0a9810abe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56068/0 (socket says 172.21.15.29:56068) 2024-03-16T11:40:41.742 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:41.026+0000 7f0a9d2d1700 1 -- 172.21.15.29:0/956429342 learned_addr learned my addr 172.21.15.29:0/956429342 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:40:41.742 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:41.026+0000 7f0a9d2d1700 1 -- 172.21.15.29:0/956429342 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a9810b120 con 0x7f0a981087f0 2024-03-16T11:40:41.742 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:41.026+0000 7f0a9d2d1700 1 --2- 172.21.15.29:0/956429342 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0a981087f0 0x7f0a9810abe0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f0a940098a0 tx=0x7f0a94009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6e882dd1a3057d0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:40:41.742 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:41.026+0000 7f0a8bfff700 1 -- 172.21.15.29:0/956429342 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0a94004160 con 0x7f0a981087f0 2024-03-16T11:40:41.742 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:41.026+0000 7f0a8bfff700 1 -- 172.21.15.29:0/956429342 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f0a94004300 con 0x7f0a981087f0 2024-03-16T11:40:41.742 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:41.026+0000 7f0a8bfff700 1 -- 172.21.15.29:0/956429342 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0a94011b50 con 0x7f0a981087f0 2024-03-16T11:40:41.742 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:41.027+0000 7f0a9f535700 1 -- 172.21.15.29:0/956429342 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0a981087f0 msgr2=0x7f0a9810abe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:40:41.742 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:41.027+0000 7f0a9f535700 1 --2- 172.21.15.29:0/956429342 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0a981087f0 0x7f0a9810abe0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f0a940098a0 tx=0x7f0a94009bb0 comp rx=0 tx=0).stop 2024-03-16T11:40:41.742 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:41.027+0000 7f0a9f535700 1 -- 172.21.15.29:0/956429342 shutdown_connections 2024-03-16T11:40:41.742 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:41.027+0000 7f0a9f535700 1 --2- 172.21.15.29:0/956429342 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0a981087f0 0x7f0a9810abe0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:40:41.742 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:41.027+0000 7f0a9f535700 1 -- 172.21.15.29:0/956429342 >> 172.21.15.29:0/956429342 conn(0x7f0a981021c0 msgr2=0x7f0a98104600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:40:41.743 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:41.027+0000 7f0a9f535700 1 -- 172.21.15.29:0/956429342 shutdown_connections 2024-03-16T11:40:41.743 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:41.027+0000 7f0a9f535700 1 -- 172.21.15.29:0/956429342 wait complete. 2024-03-16T11:40:41.743 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:41.028+0000 7f0a9f535700 1 Processor -- start 2024-03-16T11:40:41.743 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:41.028+0000 7f0a9f535700 1 -- start start 2024-03-16T11:40:41.743 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:41.028+0000 7f0a9f535700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0a981087f0 0x7f0a9819df50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:40:41.743 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:41.028+0000 7f0a9f535700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f0a9810c0f0 con 0x7f0a981087f0 2024-03-16T11:40:41.743 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:41.028+0000 7f0a9d2d1700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0a981087f0 0x7f0a9819df50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:40:41.743 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:41.028+0000 7f0a9d2d1700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0a981087f0 0x7f0a9819df50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56072/0 (socket says 172.21.15.29:56072) 2024-03-16T11:40:41.743 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:41.028+0000 7f0a9d2d1700 1 -- 172.21.15.29:0/1927148736 learned_addr learned my addr 172.21.15.29:0/1927148736 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:40:41.743 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:41.028+0000 7f0a9d2d1700 1 -- 172.21.15.29:0/1927148736 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a9819e490 con 0x7f0a981087f0 2024-03-16T11:40:41.743 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:41.029+0000 7f0a9d2d1700 1 --2- 172.21.15.29:0/1927148736 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0a981087f0 0x7f0a9819df50 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f0a94009fb0 tx=0x7f0a94011bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:40:41.743 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:41.029+0000 7f0a8a7fc700 1 -- 172.21.15.29:0/1927148736 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0a940038b0 con 0x7f0a981087f0 2024-03-16T11:40:41.743 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:41.029+0000 7f0a9f535700 1 -- 172.21.15.29:0/1927148736 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0a9819e720 con 0x7f0a981087f0 2024-03-16T11:40:41.743 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:41.029+0000 7f0a8a7fc700 1 -- 172.21.15.29:0/1927148736 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f0a94011e50 con 0x7f0a981087f0 2024-03-16T11:40:41.743 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:41.029+0000 7f0a8a7fc700 1 -- 172.21.15.29:0/1927148736 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0a94019420 con 0x7f0a981087f0 2024-03-16T11:40:41.744 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:41.029+0000 7f0a9f535700 1 -- 172.21.15.29:0/1927148736 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0a981a1410 con 0x7f0a981087f0 2024-03-16T11:40:41.744 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:41.029+0000 7f0a8a7fc700 1 -- 172.21.15.29:0/1927148736 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f0a94019b30 con 0x7f0a981087f0 2024-03-16T11:40:41.744 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:41.029+0000 7f0a8a7fc700 1 -- 172.21.15.29:0/1927148736 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f0a9402d730 con 0x7f0a981087f0 2024-03-16T11:40:41.744 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:41.030+0000 7f0a9f535700 1 -- 172.21.15.29:0/1927148736 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0a98063900 con 0x7f0a981087f0 2024-03-16T11:40:41.744 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:41.032+0000 7f0a8a7fc700 1 -- 172.21.15.29:0/1927148736 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f0a94032070 con 0x7f0a981087f0 2024-03-16T11:40:41.744 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:41.105+0000 7f0a9f535700 1 -- 172.21.15.29:0/1927148736 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f0a981a1ab0 con 0x7f0a981087f0 2024-03-16T11:40:41.744 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:41.106+0000 7f0a8a7fc700 1 -- 172.21.15.29:0/1927148736 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7f0a94003a50 con 0x7f0a981087f0 2024-03-16T11:40:41.744 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:41.107+0000 7f0a9f535700 1 -- 172.21.15.29:0/1927148736 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0a981087f0 msgr2=0x7f0a9819df50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:40:41.744 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:41.107+0000 7f0a9f535700 1 --2- 172.21.15.29:0/1927148736 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0a981087f0 0x7f0a9819df50 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f0a94009fb0 tx=0x7f0a94011bf0 comp rx=0 tx=0).stop 2024-03-16T11:40:41.744 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:41.107+0000 7f0a9f535700 1 -- 172.21.15.29:0/1927148736 shutdown_connections 2024-03-16T11:40:41.744 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:41.107+0000 7f0a9f535700 1 --2- 172.21.15.29:0/1927148736 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0a981087f0 0x7f0a9819df50 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:40:41.744 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:41.107+0000 7f0a9f535700 1 -- 172.21.15.29:0/1927148736 >> 172.21.15.29:0/1927148736 conn(0x7f0a981021c0 msgr2=0x7f0a98102ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:40:41.744 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:41.107+0000 7f0a9f535700 1 -- 172.21.15.29:0/1927148736 shutdown_connections 2024-03-16T11:40:41.744 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:41.107+0000 7f0a9f535700 1 -- 172.21.15.29:0/1927148736 wait complete. 2024-03-16T11:40:41.745 INFO:teuthology.orchestra.run.smithi029.stdout:mgr not available, waiting (1/15)... 2024-03-16T11:40:42.016 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 16 11:40:41 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-x[32210]: 2024-03-16T11:40:41.778+0000 7fb9410d4200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-16T11:40:42.016 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 16 11:40:41 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-x[32210]: 2024-03-16T11:40:41.873+0000 7fb9410d4200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-16T11:40:42.338 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 16 11:40:42 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-x[32210]: 2024-03-16T11:40:42.016+0000 7fb9410d4200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-16T11:40:42.338 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 16 11:40:42 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-x[32210]: 2024-03-16T11:40:42.113+0000 7fb9410d4200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-16T11:40:42.338 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 16 11:40:42 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-x[32210]: 2024-03-16T11:40:42.198+0000 7fb9410d4200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-16T11:40:42.761 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 16 11:40:42 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-x[32210]: 2024-03-16T11:40:42.465+0000 7fb9410d4200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-16T11:40:42.761 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 16 11:40:42 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-x[32210]: 2024-03-16T11:40:42.577+0000 7fb9410d4200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-16T11:40:42.761 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 16 11:40:42 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-x[32210]: 2024-03-16T11:40:42.664+0000 7fb9410d4200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-16T11:40:42.761 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 16 11:40:42 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-x[32210]: 2024-03-16T11:40:42.761+0000 7fb9410d4200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-16T11:40:43.088 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 16 11:40:42 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-x[32210]: 2024-03-16T11:40:42.938+0000 7fb9410d4200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-16T11:40:44.182 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 16 11:40:43 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-x[32210]: 2024-03-16T11:40:43.937+0000 7fb9410d4200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-16T11:40:44.182 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 16 11:40:44 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-x[32210]: 2024-03-16T11:40:44.072+0000 7fb9410d4200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-16T11:40:44.452 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 16 11:40:44 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-x[32210]: 2024-03-16T11:40:44.415+0000 7fb9410d4200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-16T11:40:44.710 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:44 smithi029 ceph-mon[31975]: from='client.? 172.21.15.29:0/3658421645' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-16T11:40:44.711 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 16 11:40:44 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-x[32210]: 2024-03-16T11:40:44.510+0000 7fb9410d4200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-16T11:40:44.711 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 16 11:40:44 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-x[32210]: 2024-03-16T11:40:44.711+0000 7fb9410d4200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-16T11:40:45.061 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 16 11:40:44 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-x[32210]: 2024-03-16T11:40:44.900+0000 7fb9410d4200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-16T11:40:45.084 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout 2024-03-16T11:40:45.084 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout { 2024-03-16T11:40:45.084 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "fsid": "d1887f28-e389-11ee-95c9-87774f69a715", 2024-03-16T11:40:45.084 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "health": { 2024-03-16T11:40:45.084 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-16T11:40:45.084 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-16T11:40:45.084 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-16T11:40:45.084 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout }, 2024-03-16T11:40:45.084 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-16T11:40:45.084 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-16T11:40:45.084 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout 0 2024-03-16T11:40:45.085 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout ], 2024-03-16T11:40:45.085 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-16T11:40:45.085 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "a" 2024-03-16T11:40:45.085 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout ], 2024-03-16T11:40:45.085 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "quorum_age": 12, 2024-03-16T11:40:45.085 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-16T11:40:45.085 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-16T11:40:45.085 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-16T11:40:45.085 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-16T11:40:45.085 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout }, 2024-03-16T11:40:45.085 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-16T11:40:45.085 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-16T11:40:45.085 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-16T11:40:45.086 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-16T11:40:45.086 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-16T11:40:45.086 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-16T11:40:45.086 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-16T11:40:45.086 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-16T11:40:45.086 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout }, 2024-03-16T11:40:45.086 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-16T11:40:45.086 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-16T11:40:45.086 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-16T11:40:45.086 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-16T11:40:45.086 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-16T11:40:45.086 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-16T11:40:45.086 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-16T11:40:45.087 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-16T11:40:45.087 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-16T11:40:45.087 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout }, 2024-03-16T11:40:45.087 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-16T11:40:45.087 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-16T11:40:45.087 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-16T11:40:45.087 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-16T11:40:45.087 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout }, 2024-03-16T11:40:45.087 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-16T11:40:45.087 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-16T11:40:45.087 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-16T11:40:45.087 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-16T11:40:45.088 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-16T11:40:45.088 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-16T11:40:45.088 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "restful" 2024-03-16T11:40:45.088 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout ], 2024-03-16T11:40:45.088 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-16T11:40:45.088 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout }, 2024-03-16T11:40:45.088 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-16T11:40:45.088 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-16T11:40:45.088 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "modified": "2024-03-16T11:40:16.475564+0000", 2024-03-16T11:40:45.088 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-16T11:40:45.088 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout }, 2024-03-16T11:40:45.088 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-16T11:40:45.088 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout } 2024-03-16T11:40:45.089 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:44.372+0000 7f62e901c700 1 Processor -- start 2024-03-16T11:40:45.089 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:44.373+0000 7f62e901c700 1 -- start start 2024-03-16T11:40:45.089 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:44.373+0000 7f62e901c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f62e4106c00 0x7f62e4107020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:40:45.089 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:44.373+0000 7f62e901c700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f62e41075f0 con 0x7f62e4106c00 2024-03-16T11:40:45.089 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:44.373+0000 7f62e2d9d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f62e4106c00 0x7f62e4107020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:40:45.089 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:44.373+0000 7f62e2d9d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f62e4106c00 0x7f62e4107020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56088/0 (socket says 172.21.15.29:56088) 2024-03-16T11:40:45.089 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:44.373+0000 7f62e2d9d700 1 -- 172.21.15.29:0/1112740364 learned_addr learned my addr 172.21.15.29:0/1112740364 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:40:45.089 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:44.373+0000 7f62e2d9d700 1 -- 172.21.15.29:0/1112740364 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62e4107e00 con 0x7f62e4106c00 2024-03-16T11:40:45.089 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:44.373+0000 7f62e2d9d700 1 --2- 172.21.15.29:0/1112740364 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f62e4106c00 0x7f62e4107020 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f62d40098a0 tx=0x7f62d4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d2a6f82e5560d80 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:40:45.089 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:44.374+0000 7f62e1d9b700 1 -- 172.21.15.29:0/1112740364 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f62d4004160 con 0x7f62e4106c00 2024-03-16T11:40:45.090 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:44.374+0000 7f62e1d9b700 1 -- 172.21.15.29:0/1112740364 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f62d4004300 con 0x7f62e4106c00 2024-03-16T11:40:45.090 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:44.374+0000 7f62e1d9b700 1 -- 172.21.15.29:0/1112740364 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f62d4011b50 con 0x7f62e4106c00 2024-03-16T11:40:45.090 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:44.374+0000 7f62e901c700 1 -- 172.21.15.29:0/1112740364 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f62e4106c00 msgr2=0x7f62e4107020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:40:45.090 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:44.374+0000 7f62e901c700 1 --2- 172.21.15.29:0/1112740364 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f62e4106c00 0x7f62e4107020 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f62d40098a0 tx=0x7f62d4009bb0 comp rx=0 tx=0).stop 2024-03-16T11:40:45.090 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:44.374+0000 7f62e901c700 1 -- 172.21.15.29:0/1112740364 shutdown_connections 2024-03-16T11:40:45.090 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:44.374+0000 7f62e901c700 1 --2- 172.21.15.29:0/1112740364 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f62e4106c00 0x7f62e4107020 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:40:45.090 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:44.374+0000 7f62e901c700 1 -- 172.21.15.29:0/1112740364 >> 172.21.15.29:0/1112740364 conn(0x7f62e41021a0 msgr2=0x7f62e41045e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:40:45.090 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:44.374+0000 7f62e901c700 1 -- 172.21.15.29:0/1112740364 shutdown_connections 2024-03-16T11:40:45.090 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:44.374+0000 7f62e901c700 1 -- 172.21.15.29:0/1112740364 wait complete. 2024-03-16T11:40:45.090 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:44.375+0000 7f62e901c700 1 Processor -- start 2024-03-16T11:40:45.090 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:44.375+0000 7f62e901c700 1 -- start start 2024-03-16T11:40:45.090 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:44.375+0000 7f62e901c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f62e4106c00 0x7f62e419df40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:40:45.091 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:44.375+0000 7f62e901c700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f62e407af90 con 0x7f62e4106c00 2024-03-16T11:40:45.091 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:44.375+0000 7f62e2d9d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f62e4106c00 0x7f62e419df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:40:45.091 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:44.375+0000 7f62e2d9d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f62e4106c00 0x7f62e419df40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56104/0 (socket says 172.21.15.29:56104) 2024-03-16T11:40:45.091 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:44.376+0000 7f62e2d9d700 1 -- 172.21.15.29:0/3658421645 learned_addr learned my addr 172.21.15.29:0/3658421645 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:40:45.091 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:44.376+0000 7f62e2d9d700 1 -- 172.21.15.29:0/3658421645 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62e419e480 con 0x7f62e4106c00 2024-03-16T11:40:45.091 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:44.376+0000 7f62e2d9d700 1 --2- 172.21.15.29:0/3658421645 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f62e4106c00 0x7f62e419df40 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f62d4011850 tx=0x7f62d4011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:40:45.091 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:44.376+0000 7f62cbfff700 1 -- 172.21.15.29:0/3658421645 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f62d4003af0 con 0x7f62e4106c00 2024-03-16T11:40:45.091 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:44.376+0000 7f62cbfff700 1 -- 172.21.15.29:0/3658421645 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f62d4020070 con 0x7f62e4106c00 2024-03-16T11:40:45.091 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:44.376+0000 7f62e901c700 1 -- 172.21.15.29:0/3658421645 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f62e419e710 con 0x7f62e4106c00 2024-03-16T11:40:45.091 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:44.376+0000 7f62cbfff700 1 -- 172.21.15.29:0/3658421645 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f62d40195a0 con 0x7f62e4106c00 2024-03-16T11:40:45.091 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:44.376+0000 7f62e901c700 1 -- 172.21.15.29:0/3658421645 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f62e41a13b0 con 0x7f62e4106c00 2024-03-16T11:40:45.091 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:44.376+0000 7f62cbfff700 1 -- 172.21.15.29:0/3658421645 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f62d4019c30 con 0x7f62e4106c00 2024-03-16T11:40:45.092 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:44.376+0000 7f62cbfff700 1 -- 172.21.15.29:0/3658421645 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f62d4011e00 con 0x7f62e4106c00 2024-03-16T11:40:45.092 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:44.377+0000 7f62e901c700 1 -- 172.21.15.29:0/3658421645 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f62e4063900 con 0x7f62e4106c00 2024-03-16T11:40:45.092 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:44.379+0000 7f62cbfff700 1 -- 172.21.15.29:0/3658421645 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f62d4003c90 con 0x7f62e4106c00 2024-03-16T11:40:45.092 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:44.452+0000 7f62e901c700 1 -- 172.21.15.29:0/3658421645 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f62e40008d0 con 0x7f62e4106c00 2024-03-16T11:40:45.092 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:44.453+0000 7f62cbfff700 1 -- 172.21.15.29:0/3658421645 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1240 (secure 0 0 0) 0x7f62d401a430 con 0x7f62e4106c00 2024-03-16T11:40:45.092 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:44.454+0000 7f62e901c700 1 -- 172.21.15.29:0/3658421645 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f62e4106c00 msgr2=0x7f62e419df40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:40:45.092 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:44.454+0000 7f62e901c700 1 --2- 172.21.15.29:0/3658421645 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f62e4106c00 0x7f62e419df40 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f62d4011850 tx=0x7f62d4011c10 comp rx=0 tx=0).stop 2024-03-16T11:40:45.092 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:44.454+0000 7f62e901c700 1 -- 172.21.15.29:0/3658421645 shutdown_connections 2024-03-16T11:40:45.092 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:44.454+0000 7f62e901c700 1 --2- 172.21.15.29:0/3658421645 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f62e4106c00 0x7f62e419df40 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:40:45.092 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:44.454+0000 7f62e901c700 1 -- 172.21.15.29:0/3658421645 >> 172.21.15.29:0/3658421645 conn(0x7f62e41021a0 msgr2=0x7f62e41905d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:40:45.092 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:44.454+0000 7f62e901c700 1 -- 172.21.15.29:0/3658421645 shutdown_connections 2024-03-16T11:40:45.092 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:44.454+0000 7f62e901c700 1 -- 172.21.15.29:0/3658421645 wait complete. 2024-03-16T11:40:45.092 INFO:teuthology.orchestra.run.smithi029.stdout:mgr not available, waiting (2/15)... 2024-03-16T11:40:45.338 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 16 11:40:45 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-x[32210]: 2024-03-16T11:40:45.190+0000 7fb9410d4200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-16T11:40:45.838 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 16 11:40:45 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-x[32210]: 2024-03-16T11:40:45.450+0000 7fb9410d4200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-16T11:40:45.838 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 16 11:40:45 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-x[32210]: 2024-03-16T11:40:45.691+0000 7fb9410d4200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-16T11:40:46.838 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 16 11:40:46 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-x[32210]: 2024-03-16T11:40:46.512+0000 7fb9410d4200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-16T11:40:47.419 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 16 11:40:47 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-x[32210]: 2024-03-16T11:40:47.095+0000 7fb9410d4200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-16T11:40:47.419 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 16 11:40:47 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-x[32210]: 2024-03-16T11:40:47.201+0000 7fb9410d4200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-16T11:40:47.838 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 16 11:40:47 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-x[32210]: 2024-03-16T11:40:47.426+0000 7fb9410d4200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-16T11:40:47.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:47 smithi029 ceph-mon[31975]: Activating manager daemon x 2024-03-16T11:40:47.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:47 smithi029 ceph-mon[31975]: mgrmap e2: x(active, starting, since 0.00314416s) 2024-03-16T11:40:47.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:47 smithi029 ceph-mon[31975]: from='mgr.14100 172.21.15.29:0/880899150' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-16T11:40:47.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:47 smithi029 ceph-mon[31975]: from='mgr.14100 172.21.15.29:0/880899150' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-16T11:40:47.839 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:47 smithi029 ceph-mon[31975]: from='mgr.14100 172.21.15.29:0/880899150' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-16T11:40:47.839 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:47 smithi029 ceph-mon[31975]: from='mgr.14100 172.21.15.29:0/880899150' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-16T11:40:47.839 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:47 smithi029 ceph-mon[31975]: from='mgr.14100 172.21.15.29:0/880899150' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-16T11:40:47.839 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:47 smithi029 ceph-mon[31975]: Manager daemon x is now available 2024-03-16T11:40:47.839 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:47 smithi029 ceph-mon[31975]: from='mgr.14100 172.21.15.29:0/880899150' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-16T11:40:47.839 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:47 smithi029 ceph-mon[31975]: from='mgr.14100 172.21.15.29:0/880899150' entity='mgr.x' 2024-03-16T11:40:47.839 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:47 smithi029 ceph-mon[31975]: from='mgr.14100 172.21.15.29:0/880899150' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-16T11:40:47.839 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:47 smithi029 ceph-mon[31975]: from='mgr.14100 172.21.15.29:0/880899150' entity='mgr.x' 2024-03-16T11:40:47.839 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:47 smithi029 ceph-mon[31975]: from='mgr.14100 172.21.15.29:0/880899150' entity='mgr.x' 2024-03-16T11:40:48.199 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout 2024-03-16T11:40:48.199 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout { 2024-03-16T11:40:48.199 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "fsid": "d1887f28-e389-11ee-95c9-87774f69a715", 2024-03-16T11:40:48.199 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "health": { 2024-03-16T11:40:48.199 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-16T11:40:48.199 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-16T11:40:48.199 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-16T11:40:48.200 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout }, 2024-03-16T11:40:48.200 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-16T11:40:48.200 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-16T11:40:48.200 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout 0 2024-03-16T11:40:48.200 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout ], 2024-03-16T11:40:48.200 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-16T11:40:48.200 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "a" 2024-03-16T11:40:48.200 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout ], 2024-03-16T11:40:48.200 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "quorum_age": 15, 2024-03-16T11:40:48.200 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-16T11:40:48.200 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-16T11:40:48.200 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-16T11:40:48.200 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-16T11:40:48.200 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout }, 2024-03-16T11:40:48.200 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-16T11:40:48.208 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-16T11:40:48.208 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-16T11:40:48.208 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-16T11:40:48.209 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-16T11:40:48.209 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-16T11:40:48.209 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-16T11:40:48.209 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-16T11:40:48.209 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout }, 2024-03-16T11:40:48.209 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-16T11:40:48.209 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-16T11:40:48.209 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-16T11:40:48.209 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-16T11:40:48.209 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-16T11:40:48.209 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-16T11:40:48.209 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-16T11:40:48.209 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-16T11:40:48.209 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-16T11:40:48.209 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout }, 2024-03-16T11:40:48.210 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-16T11:40:48.210 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-16T11:40:48.210 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-16T11:40:48.210 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-16T11:40:48.210 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout }, 2024-03-16T11:40:48.210 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-16T11:40:48.210 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-16T11:40:48.210 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-16T11:40:48.210 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-16T11:40:48.210 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-16T11:40:48.210 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-16T11:40:48.210 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "restful" 2024-03-16T11:40:48.210 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout ], 2024-03-16T11:40:48.210 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-16T11:40:48.210 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout }, 2024-03-16T11:40:48.210 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-16T11:40:48.211 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-16T11:40:48.211 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "modified": "2024-03-16T11:40:16.475564+0000", 2024-03-16T11:40:48.211 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-16T11:40:48.211 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout }, 2024-03-16T11:40:48.211 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-16T11:40:48.211 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout } 2024-03-16T11:40:48.211 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:47.523+0000 7f88ac86a700 1 Processor -- start 2024-03-16T11:40:48.211 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:47.524+0000 7f88ac86a700 1 -- start start 2024-03-16T11:40:48.211 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:47.524+0000 7f88ac86a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f88a8108790 0x7f88a810ab80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:40:48.211 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:47.524+0000 7f88ac86a700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f88a807be30 con 0x7f88a8108790 2024-03-16T11:40:48.211 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:47.524+0000 7f88a659c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f88a8108790 0x7f88a810ab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:40:48.211 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:47.524+0000 7f88a659c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f88a8108790 0x7f88a810ab80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56166/0 (socket says 172.21.15.29:56166) 2024-03-16T11:40:48.211 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:47.524+0000 7f88a659c700 1 -- 172.21.15.29:0/2670743244 learned_addr learned my addr 172.21.15.29:0/2670743244 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:40:48.211 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:47.524+0000 7f88a659c700 1 -- 172.21.15.29:0/2670743244 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88a810b0c0 con 0x7f88a8108790 2024-03-16T11:40:48.211 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:47.524+0000 7f88a659c700 1 --2- 172.21.15.29:0/2670743244 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f88a8108790 0x7f88a810ab80 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f88900098a0 tx=0x7f8890009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81480551b98c8e36 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:40:48.211 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:47.525+0000 7f88a559a700 1 -- 172.21.15.29:0/2670743244 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8890004160 con 0x7f88a8108790 2024-03-16T11:40:48.212 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:47.525+0000 7f88a559a700 1 -- 172.21.15.29:0/2670743244 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f8890004300 con 0x7f88a8108790 2024-03-16T11:40:48.212 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:47.525+0000 7f88a559a700 1 -- 172.21.15.29:0/2670743244 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8890011b50 con 0x7f88a8108790 2024-03-16T11:40:48.212 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:47.525+0000 7f88ac86a700 1 -- 172.21.15.29:0/2670743244 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f88a8108790 msgr2=0x7f88a810ab80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:40:48.212 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:47.525+0000 7f88ac86a700 1 --2- 172.21.15.29:0/2670743244 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f88a8108790 0x7f88a810ab80 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f88900098a0 tx=0x7f8890009bb0 comp rx=0 tx=0).stop 2024-03-16T11:40:48.212 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:47.525+0000 7f88ac86a700 1 -- 172.21.15.29:0/2670743244 shutdown_connections 2024-03-16T11:40:48.212 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:47.525+0000 7f88ac86a700 1 --2- 172.21.15.29:0/2670743244 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f88a8108790 0x7f88a810ab80 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:40:48.212 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:47.525+0000 7f88ac86a700 1 -- 172.21.15.29:0/2670743244 >> 172.21.15.29:0/2670743244 conn(0x7f88a8102180 msgr2=0x7f88a81045c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:40:48.212 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:47.525+0000 7f88ac86a700 1 -- 172.21.15.29:0/2670743244 shutdown_connections 2024-03-16T11:40:48.212 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:47.525+0000 7f88ac86a700 1 -- 172.21.15.29:0/2670743244 wait complete. 2024-03-16T11:40:48.212 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:47.526+0000 7f88ac86a700 1 Processor -- start 2024-03-16T11:40:48.212 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:47.526+0000 7f88ac86a700 1 -- start start 2024-03-16T11:40:48.212 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:47.526+0000 7f88ac86a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f88a8108790 0x7f88a819bcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:40:48.212 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:47.526+0000 7f88ac86a700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f88a810c0e0 con 0x7f88a8108790 2024-03-16T11:40:48.212 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:47.526+0000 7f88a659c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f88a8108790 0x7f88a819bcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:40:48.212 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:47.526+0000 7f88a659c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f88a8108790 0x7f88a819bcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56170/0 (socket says 172.21.15.29:56170) 2024-03-16T11:40:48.213 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:47.526+0000 7f88a659c700 1 -- 172.21.15.29:0/2777003646 learned_addr learned my addr 172.21.15.29:0/2777003646 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:40:48.213 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:47.527+0000 7f88a659c700 1 -- 172.21.15.29:0/2777003646 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88a819c1f0 con 0x7f88a8108790 2024-03-16T11:40:48.213 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:47.527+0000 7f88a659c700 1 --2- 172.21.15.29:0/2777003646 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f88a8108790 0x7f88a819bcb0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f8890009fb0 tx=0x7f8890011bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:40:48.213 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:47.527+0000 7f889f7fe700 1 -- 172.21.15.29:0/2777003646 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f88900038b0 con 0x7f88a8108790 2024-03-16T11:40:48.213 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:47.527+0000 7f88ac86a700 1 -- 172.21.15.29:0/2777003646 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f88a819c480 con 0x7f88a8108790 2024-03-16T11:40:48.213 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:47.527+0000 7f889f7fe700 1 -- 172.21.15.29:0/2777003646 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f8890011e50 con 0x7f88a8108790 2024-03-16T11:40:48.213 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:47.527+0000 7f889f7fe700 1 -- 172.21.15.29:0/2777003646 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8890019420 con 0x7f88a8108790 2024-03-16T11:40:48.213 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:47.527+0000 7f88ac86a700 1 -- 172.21.15.29:0/2777003646 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f88a819f170 con 0x7f88a8108790 2024-03-16T11:40:48.213 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:47.528+0000 7f889f7fe700 1 -- 172.21.15.29:0/2777003646 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 2) ==== 45862+0+0 (secure 0 0 0) 0x7f8890003a50 con 0x7f88a8108790 2024-03-16T11:40:48.213 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:47.528+0000 7f88ac86a700 1 -- 172.21.15.29:0/2777003646 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f88a8063900 con 0x7f88a8108790 2024-03-16T11:40:48.213 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:47.528+0000 7f889f7fe700 1 -- 172.21.15.29:0/2777003646 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f889004cd00 con 0x7f88a8108790 2024-03-16T11:40:48.213 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:47.530+0000 7f889f7fe700 1 -- 172.21.15.29:0/2777003646 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f889001a4e0 con 0x7f88a8108790 2024-03-16T11:40:48.213 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:47.604+0000 7f88ac86a700 1 -- 172.21.15.29:0/2777003646 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f88a819f340 con 0x7f88a8108790 2024-03-16T11:40:48.213 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:47.604+0000 7f889f7fe700 1 -- 172.21.15.29:0/2777003646 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1240 (secure 0 0 0) 0x7f88a819f340 con 0x7f88a8108790 2024-03-16T11:40:48.214 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:47.605+0000 7f88ac86a700 1 -- 172.21.15.29:0/2777003646 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f88a8108790 msgr2=0x7f88a819bcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:40:48.214 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:47.605+0000 7f88ac86a700 1 --2- 172.21.15.29:0/2777003646 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f88a8108790 0x7f88a819bcb0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f8890009fb0 tx=0x7f8890011bf0 comp rx=0 tx=0).stop 2024-03-16T11:40:48.214 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:47.605+0000 7f88ac86a700 1 -- 172.21.15.29:0/2777003646 shutdown_connections 2024-03-16T11:40:48.214 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:47.605+0000 7f88ac86a700 1 --2- 172.21.15.29:0/2777003646 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f88a8108790 0x7f88a819bcb0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:40:48.214 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:47.605+0000 7f88ac86a700 1 -- 172.21.15.29:0/2777003646 >> 172.21.15.29:0/2777003646 conn(0x7f88a8102180 msgr2=0x7f88a8102ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:40:48.214 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:47.605+0000 7f88ac86a700 1 -- 172.21.15.29:0/2777003646 shutdown_connections 2024-03-16T11:40:48.214 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:47.605+0000 7f88ac86a700 1 -- 172.21.15.29:0/2777003646 wait complete. 2024-03-16T11:40:48.214 INFO:teuthology.orchestra.run.smithi029.stdout:mgr not available, waiting (3/15)... 2024-03-16T11:40:48.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:48 smithi029 ceph-mon[31975]: from='client.? 172.21.15.29:0/2777003646' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-16T11:40:48.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:48 smithi029 ceph-mon[31975]: mgrmap e3: x(active, since 1.00709s) 2024-03-16T11:40:50.588 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:50 smithi029 ceph-mon[31975]: mgrmap e4: x(active, since 2s) 2024-03-16T11:40:51.453 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout 2024-03-16T11:40:51.453 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout { 2024-03-16T11:40:51.453 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "fsid": "d1887f28-e389-11ee-95c9-87774f69a715", 2024-03-16T11:40:51.453 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "health": { 2024-03-16T11:40:51.453 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-16T11:40:51.453 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-16T11:40:51.453 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-16T11:40:51.453 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout }, 2024-03-16T11:40:51.453 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-16T11:40:51.453 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-16T11:40:51.453 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout 0 2024-03-16T11:40:51.453 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout ], 2024-03-16T11:40:51.454 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-16T11:40:51.454 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "a" 2024-03-16T11:40:51.454 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout ], 2024-03-16T11:40:51.454 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "quorum_age": 19, 2024-03-16T11:40:51.454 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-16T11:40:51.454 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-16T11:40:51.454 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-16T11:40:51.454 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-16T11:40:51.462 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout }, 2024-03-16T11:40:51.462 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-16T11:40:51.462 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-16T11:40:51.462 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-16T11:40:51.462 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-16T11:40:51.462 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-16T11:40:51.463 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-16T11:40:51.463 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-16T11:40:51.463 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-16T11:40:51.463 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout }, 2024-03-16T11:40:51.463 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-16T11:40:51.463 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-16T11:40:51.463 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-16T11:40:51.463 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-16T11:40:51.463 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-16T11:40:51.463 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-16T11:40:51.463 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-16T11:40:51.463 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-16T11:40:51.463 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-16T11:40:51.463 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout }, 2024-03-16T11:40:51.463 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-16T11:40:51.463 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-16T11:40:51.464 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-16T11:40:51.464 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-16T11:40:51.464 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout }, 2024-03-16T11:40:51.464 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-16T11:40:51.464 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-16T11:40:51.464 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-16T11:40:51.464 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-16T11:40:51.464 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-16T11:40:51.464 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-16T11:40:51.464 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "restful" 2024-03-16T11:40:51.464 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout ], 2024-03-16T11:40:51.464 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-16T11:40:51.464 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout }, 2024-03-16T11:40:51.464 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-16T11:40:51.464 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-16T11:40:51.464 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "modified": "2024-03-16T11:40:16.475564+0000", 2024-03-16T11:40:51.465 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-16T11:40:51.465 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout }, 2024-03-16T11:40:51.465 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-16T11:40:51.465 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout } 2024-03-16T11:40:51.465 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:50.620+0000 7fadfd4d8700 1 Processor -- start 2024-03-16T11:40:51.465 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:50.620+0000 7fadfd4d8700 1 -- start start 2024-03-16T11:40:51.465 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:50.620+0000 7fadfd4d8700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fadf8106c00 0x7fadf8107020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:40:51.465 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:50.620+0000 7fadfd4d8700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fadf81075f0 con 0x7fadf8106c00 2024-03-16T11:40:51.465 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:50.620+0000 7fadf6ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fadf8106c00 0x7fadf8107020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:40:51.465 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:50.620+0000 7fadf6ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fadf8106c00 0x7fadf8107020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60820/0 (socket says 172.21.15.29:60820) 2024-03-16T11:40:51.465 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:50.620+0000 7fadf6ffd700 1 -- 172.21.15.29:0/4219396676 learned_addr learned my addr 172.21.15.29:0/4219396676 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:40:51.465 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:50.620+0000 7fadf6ffd700 1 -- 172.21.15.29:0/4219396676 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fadf8107e00 con 0x7fadf8106c00 2024-03-16T11:40:51.465 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:50.621+0000 7fadf6ffd700 1 --2- 172.21.15.29:0/4219396676 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fadf8106c00 0x7fadf8107020 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fade80098a0 tx=0x7fade8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b3160a540bb0e4c server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:40:51.465 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:50.621+0000 7fadf5ffb700 1 -- 172.21.15.29:0/4219396676 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fade8004160 con 0x7fadf8106c00 2024-03-16T11:40:51.465 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:50.621+0000 7fadf5ffb700 1 -- 172.21.15.29:0/4219396676 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fade8004300 con 0x7fadf8106c00 2024-03-16T11:40:51.466 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:50.621+0000 7fadf5ffb700 1 -- 172.21.15.29:0/4219396676 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fade8011b50 con 0x7fadf8106c00 2024-03-16T11:40:51.466 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:50.621+0000 7fadfd4d8700 1 -- 172.21.15.29:0/4219396676 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fadf8106c00 msgr2=0x7fadf8107020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:40:51.466 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:50.621+0000 7fadfd4d8700 1 --2- 172.21.15.29:0/4219396676 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fadf8106c00 0x7fadf8107020 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fade80098a0 tx=0x7fade8009bb0 comp rx=0 tx=0).stop 2024-03-16T11:40:51.466 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:50.621+0000 7fadfd4d8700 1 -- 172.21.15.29:0/4219396676 shutdown_connections 2024-03-16T11:40:51.466 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:50.621+0000 7fadfd4d8700 1 --2- 172.21.15.29:0/4219396676 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fadf8106c00 0x7fadf8107020 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:40:51.466 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:50.621+0000 7fadfd4d8700 1 -- 172.21.15.29:0/4219396676 >> 172.21.15.29:0/4219396676 conn(0x7fadf81021a0 msgr2=0x7fadf81045e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:40:51.466 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:50.621+0000 7fadfd4d8700 1 -- 172.21.15.29:0/4219396676 shutdown_connections 2024-03-16T11:40:51.466 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:50.621+0000 7fadfd4d8700 1 -- 172.21.15.29:0/4219396676 wait complete. 2024-03-16T11:40:51.466 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:50.622+0000 7fadfd4d8700 1 Processor -- start 2024-03-16T11:40:51.466 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:50.622+0000 7fadfd4d8700 1 -- start start 2024-03-16T11:40:51.466 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:50.622+0000 7fadfd4d8700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fadf8106c00 0x7fadf819bd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:40:51.466 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:50.622+0000 7fadfd4d8700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fadf810b1b0 con 0x7fadf8106c00 2024-03-16T11:40:51.466 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:50.622+0000 7fadf6ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fadf8106c00 0x7fadf819bd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:40:51.466 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:50.623+0000 7fadf6ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fadf8106c00 0x7fadf819bd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60826/0 (socket says 172.21.15.29:60826) 2024-03-16T11:40:51.466 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:50.623+0000 7fadf6ffd700 1 -- 172.21.15.29:0/1606204102 learned_addr learned my addr 172.21.15.29:0/1606204102 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:40:51.467 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:50.623+0000 7fadf6ffd700 1 -- 172.21.15.29:0/1606204102 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fadf819c240 con 0x7fadf8106c00 2024-03-16T11:40:51.467 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:50.623+0000 7fadf6ffd700 1 --2- 172.21.15.29:0/1606204102 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fadf8106c00 0x7fadf819bd00 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fade8011850 tx=0x7fade8011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:40:51.467 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:50.623+0000 7faddffff700 1 -- 172.21.15.29:0/1606204102 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fade8003af0 con 0x7fadf8106c00 2024-03-16T11:40:51.467 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:50.623+0000 7faddffff700 1 -- 172.21.15.29:0/1606204102 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fade8020070 con 0x7fadf8106c00 2024-03-16T11:40:51.467 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:50.623+0000 7fadfd4d8700 1 -- 172.21.15.29:0/1606204102 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fadf819c4d0 con 0x7fadf8106c00 2024-03-16T11:40:51.467 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:50.623+0000 7faddffff700 1 -- 172.21.15.29:0/1606204102 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fade80195a0 con 0x7fadf8106c00 2024-03-16T11:40:51.467 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:50.623+0000 7fadfd4d8700 1 -- 172.21.15.29:0/1606204102 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fadf819f1b0 con 0x7fadf8106c00 2024-03-16T11:40:51.467 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:50.624+0000 7faddffff700 1 -- 172.21.15.29:0/1606204102 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 4) ==== 46095+0+0 (secure 0 0 0) 0x7fade80198a0 con 0x7fadf8106c00 2024-03-16T11:40:51.467 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:50.624+0000 7faddffff700 1 --2- 172.21.15.29:0/1606204102 >> [v2:172.21.15.29:6800/2275315654,v1:172.21.15.29:6801/2275315654] conn(0x7fade0038ed0 0x7fade003b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:40:51.467 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:50.624+0000 7faddffff700 1 -- 172.21.15.29:0/1606204102 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fade8011e00 con 0x7fadf8106c00 2024-03-16T11:40:51.467 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:50.624+0000 7fadfd4d8700 1 -- 172.21.15.29:0/1606204102 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fadf8063900 con 0x7fadf8106c00 2024-03-16T11:40:51.467 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:50.624+0000 7fadf67fc700 1 --2- 172.21.15.29:0/1606204102 >> [v2:172.21.15.29:6800/2275315654,v1:172.21.15.29:6801/2275315654] conn(0x7fade0038ed0 0x7fade003b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:40:51.467 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:50.625+0000 7fadf67fc700 1 --2- 172.21.15.29:0/1606204102 >> [v2:172.21.15.29:6800/2275315654,v1:172.21.15.29:6801/2275315654] conn(0x7fade0038ed0 0x7fade003b390 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fadec006fd0 tx=0x7fadec006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:40:51.467 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:50.630+0000 7faddffff700 1 -- 172.21.15.29:0/1606204102 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fade8003c90 con 0x7fadf8106c00 2024-03-16T11:40:51.468 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:50.919+0000 7fadfd4d8700 1 -- 172.21.15.29:0/1606204102 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7fadf802dd70 con 0x7fadf8106c00 2024-03-16T11:40:51.468 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:50.919+0000 7faddffff700 1 -- 172.21.15.29:0/1606204102 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7fade8023530 con 0x7fadf8106c00 2024-03-16T11:40:51.468 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:50.922+0000 7fadfd4d8700 1 -- 172.21.15.29:0/1606204102 >> [v2:172.21.15.29:6800/2275315654,v1:172.21.15.29:6801/2275315654] conn(0x7fade0038ed0 msgr2=0x7fade003b390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:40:51.468 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:50.922+0000 7fadfd4d8700 1 --2- 172.21.15.29:0/1606204102 >> [v2:172.21.15.29:6800/2275315654,v1:172.21.15.29:6801/2275315654] conn(0x7fade0038ed0 0x7fade003b390 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fadec006fd0 tx=0x7fadec006e40 comp rx=0 tx=0).stop 2024-03-16T11:40:51.468 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:50.922+0000 7fadfd4d8700 1 -- 172.21.15.29:0/1606204102 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fadf8106c00 msgr2=0x7fadf819bd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:40:51.468 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:50.922+0000 7fadfd4d8700 1 --2- 172.21.15.29:0/1606204102 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fadf8106c00 0x7fadf819bd00 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fade8011850 tx=0x7fade8011c10 comp rx=0 tx=0).stop 2024-03-16T11:40:51.468 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:50.922+0000 7fadfd4d8700 1 -- 172.21.15.29:0/1606204102 shutdown_connections 2024-03-16T11:40:51.468 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:50.922+0000 7fadfd4d8700 1 --2- 172.21.15.29:0/1606204102 >> [v2:172.21.15.29:6800/2275315654,v1:172.21.15.29:6801/2275315654] conn(0x7fade0038ed0 0x7fade003b390 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:40:51.468 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:50.922+0000 7fadfd4d8700 1 --2- 172.21.15.29:0/1606204102 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fadf8106c00 0x7fadf819bd00 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:40:51.468 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:50.922+0000 7fadfd4d8700 1 -- 172.21.15.29:0/1606204102 >> 172.21.15.29:0/1606204102 conn(0x7fadf81021a0 msgr2=0x7fadf8103090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:40:51.468 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:50.922+0000 7fadfd4d8700 1 -- 172.21.15.29:0/1606204102 shutdown_connections 2024-03-16T11:40:51.468 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:50.922+0000 7fadfd4d8700 1 -- 172.21.15.29:0/1606204102 wait complete. 2024-03-16T11:40:51.468 INFO:teuthology.orchestra.run.smithi029.stdout:mgr is available 2024-03-16T11:40:51.588 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:51 smithi029 ceph-mon[31975]: from='client.? 172.21.15.29:0/1606204102' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-16T11:40:52.724 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:52 smithi029 ceph-mon[31975]: from='client.? 172.21.15.29:0/1868334343' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-16T11:40:52.728 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout 2024-03-16T11:40:52.728 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout [global] 2024-03-16T11:40:52.728 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout fsid = d1887f28-e389-11ee-95c9-87774f69a715 2024-03-16T11:40:52.728 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-16T11:40:52.729 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-16T11:40:52.729 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-16T11:40:52.729 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-16T11:40:52.729 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout 2024-03-16T11:40:52.729 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-16T11:40:52.729 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-16T11:40:52.729 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout 2024-03-16T11:40:52.729 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout [osd] 2024-03-16T11:40:52.729 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-03-16T11:40:52.729 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-03-16T11:40:52.729 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-03-16T11:40:52.729 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-16T11:40:52.729 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-16T11:40:52.729 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:51.882+0000 7f800b71b700 1 Processor -- start 2024-03-16T11:40:52.729 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:51.882+0000 7f800b71b700 1 -- start start 2024-03-16T11:40:52.730 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:51.882+0000 7f800b71b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8004106390 0x7f80041067b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:40:52.734 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:51.882+0000 7f800b71b700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f8004106d80 con 0x7f8004106390 2024-03-16T11:40:52.735 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:51.882+0000 7f80094b7700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8004106390 0x7f80041067b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:40:52.735 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:51.882+0000 7f80094b7700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8004106390 0x7f80041067b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60828/0 (socket says 172.21.15.29:60828) 2024-03-16T11:40:52.735 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:51.882+0000 7f80094b7700 1 -- 172.21.15.29:0/1490087045 learned_addr learned my addr 172.21.15.29:0/1490087045 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:40:52.735 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:51.883+0000 7f80094b7700 1 -- 172.21.15.29:0/1490087045 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8004107590 con 0x7f8004106390 2024-03-16T11:40:52.735 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:51.883+0000 7f80094b7700 1 --2- 172.21.15.29:0/1490087045 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8004106390 0x7f80041067b0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f800000aa40 tx=0x7f800000ad50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a978d68e52bb6a12 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:40:52.735 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:51.883+0000 7f7ffbfff700 1 -- 172.21.15.29:0/1490087045 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f800000b040 con 0x7f8004106390 2024-03-16T11:40:52.735 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:51.883+0000 7f7ffbfff700 1 -- 172.21.15.29:0/1490087045 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f800000c8f0 con 0x7f8004106390 2024-03-16T11:40:52.735 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:51.883+0000 7f7ffbfff700 1 -- 172.21.15.29:0/1490087045 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8000003c10 con 0x7f8004106390 2024-03-16T11:40:52.735 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:51.883+0000 7f800b71b700 1 -- 172.21.15.29:0/1490087045 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8004106390 msgr2=0x7f80041067b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:40:52.735 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:51.883+0000 7f800b71b700 1 --2- 172.21.15.29:0/1490087045 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8004106390 0x7f80041067b0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f800000aa40 tx=0x7f800000ad50 comp rx=0 tx=0).stop 2024-03-16T11:40:52.735 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:51.883+0000 7f800b71b700 1 -- 172.21.15.29:0/1490087045 shutdown_connections 2024-03-16T11:40:52.735 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:51.883+0000 7f800b71b700 1 --2- 172.21.15.29:0/1490087045 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8004106390 0x7f80041067b0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:40:52.735 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:51.884+0000 7f800b71b700 1 -- 172.21.15.29:0/1490087045 >> 172.21.15.29:0/1490087045 conn(0x7f8004101930 msgr2=0x7f8004103d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:40:52.735 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:51.884+0000 7f800b71b700 1 -- 172.21.15.29:0/1490087045 shutdown_connections 2024-03-16T11:40:52.735 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:51.884+0000 7f800b71b700 1 -- 172.21.15.29:0/1490087045 wait complete. 2024-03-16T11:40:52.736 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:51.884+0000 7f800b71b700 1 Processor -- start 2024-03-16T11:40:52.736 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:51.884+0000 7f800b71b700 1 -- start start 2024-03-16T11:40:52.736 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:51.884+0000 7f800b71b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8004106390 0x7f800418ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:40:52.736 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:51.884+0000 7f800b71b700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f8004107d10 con 0x7f8004106390 2024-03-16T11:40:52.736 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:51.884+0000 7f80094b7700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8004106390 0x7f800418ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:40:52.736 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:51.884+0000 7f80094b7700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8004106390 0x7f800418ef50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60840/0 (socket says 172.21.15.29:60840) 2024-03-16T11:40:52.736 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:51.884+0000 7f80094b7700 1 -- 172.21.15.29:0/1868334343 learned_addr learned my addr 172.21.15.29:0/1868334343 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:40:52.736 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:51.885+0000 7f80094b7700 1 -- 172.21.15.29:0/1868334343 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f800418f490 con 0x7f8004106390 2024-03-16T11:40:52.736 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:51.885+0000 7f80094b7700 1 --2- 172.21.15.29:0/1868334343 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8004106390 0x7f800418ef50 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f800000afd0 tx=0x7f8000003e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:40:52.736 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:51.885+0000 7f7ffa7fc700 1 -- 172.21.15.29:0/1868334343 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8000025070 con 0x7f8004106390 2024-03-16T11:40:52.736 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:51.885+0000 7f800b71b700 1 -- 172.21.15.29:0/1868334343 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f800418d680 con 0x7f8004106390 2024-03-16T11:40:52.736 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:51.885+0000 7f7ffa7fc700 1 -- 172.21.15.29:0/1868334343 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f8000004410 con 0x7f8004106390 2024-03-16T11:40:52.736 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:51.885+0000 7f7ffa7fc700 1 -- 172.21.15.29:0/1868334343 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f80000107b0 con 0x7f8004106390 2024-03-16T11:40:52.736 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:51.885+0000 7f800b71b700 1 -- 172.21.15.29:0/1868334343 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f800418dbc0 con 0x7f8004106390 2024-03-16T11:40:52.736 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:51.886+0000 7f7ffa7fc700 1 -- 172.21.15.29:0/1868334343 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 4) ==== 46095+0+0 (secure 0 0 0) 0x7f8000028070 con 0x7f8004106390 2024-03-16T11:40:52.736 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:51.886+0000 7f800b71b700 1 -- 172.21.15.29:0/1868334343 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8004063900 con 0x7f8004106390 2024-03-16T11:40:52.737 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:51.886+0000 7f7ffa7fc700 1 --2- 172.21.15.29:0/1868334343 >> [v2:172.21.15.29:6800/2275315654,v1:172.21.15.29:6801/2275315654] conn(0x7f7ff0038e80 0x7f7ff003b340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:40:52.737 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:51.886+0000 7f7ffa7fc700 1 -- 172.21.15.29:0/1868334343 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f800004ce00 con 0x7f8004106390 2024-03-16T11:40:52.737 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:51.886+0000 7f8008cb6700 1 --2- 172.21.15.29:0/1868334343 >> [v2:172.21.15.29:6800/2275315654,v1:172.21.15.29:6801/2275315654] conn(0x7f7ff0038e80 0x7f7ff003b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:40:52.737 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:51.887+0000 7f8008cb6700 1 --2- 172.21.15.29:0/1868334343 >> [v2:172.21.15.29:6800/2275315654,v1:172.21.15.29:6801/2275315654] conn(0x7f7ff0038e80 0x7f7ff003b340 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f7ff4006fd0 tx=0x7f7ff4006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:40:52.737 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:51.891+0000 7f7ffa7fc700 1 -- 172.21.15.29:0/1868334343 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f8000014070 con 0x7f8004106390 2024-03-16T11:40:52.737 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:52.094+0000 7f800b71b700 1 -- 172.21.15.29:0/1868334343 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7f80041955b0 con 0x7f8004106390 2024-03-16T11:40:52.737 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:52.095+0000 7f7ffa7fc700 1 -- 172.21.15.29:0/1868334343 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) ==== 70+0+373 (secure 0 0 0) 0x7f800004c820 con 0x7f8004106390 2024-03-16T11:40:52.737 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:52.097+0000 7f800b71b700 1 -- 172.21.15.29:0/1868334343 >> [v2:172.21.15.29:6800/2275315654,v1:172.21.15.29:6801/2275315654] conn(0x7f7ff0038e80 msgr2=0x7f7ff003b340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:40:52.737 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:52.097+0000 7f800b71b700 1 --2- 172.21.15.29:0/1868334343 >> [v2:172.21.15.29:6800/2275315654,v1:172.21.15.29:6801/2275315654] conn(0x7f7ff0038e80 0x7f7ff003b340 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f7ff4006fd0 tx=0x7f7ff4006e40 comp rx=0 tx=0).stop 2024-03-16T11:40:52.737 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:52.097+0000 7f800b71b700 1 -- 172.21.15.29:0/1868334343 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8004106390 msgr2=0x7f800418ef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:40:52.737 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:52.097+0000 7f800b71b700 1 --2- 172.21.15.29:0/1868334343 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8004106390 0x7f800418ef50 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f800000afd0 tx=0x7f8000003e40 comp rx=0 tx=0).stop 2024-03-16T11:40:52.737 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:52.097+0000 7f800b71b700 1 -- 172.21.15.29:0/1868334343 shutdown_connections 2024-03-16T11:40:52.737 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:52.097+0000 7f800b71b700 1 --2- 172.21.15.29:0/1868334343 >> [v2:172.21.15.29:6800/2275315654,v1:172.21.15.29:6801/2275315654] conn(0x7f7ff0038e80 0x7f7ff003b340 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:40:52.737 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:52.097+0000 7f800b71b700 1 --2- 172.21.15.29:0/1868334343 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8004106390 0x7f800418ef50 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:40:52.737 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:52.097+0000 7f800b71b700 1 -- 172.21.15.29:0/1868334343 >> 172.21.15.29:0/1868334343 conn(0x7f8004101930 msgr2=0x7f8004192880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:40:52.738 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:52.097+0000 7f800b71b700 1 -- 172.21.15.29:0/1868334343 shutdown_connections 2024-03-16T11:40:52.738 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:52.097+0000 7f800b71b700 1 -- 172.21.15.29:0/1868334343 wait complete. 2024-03-16T11:40:52.738 INFO:teuthology.orchestra.run.smithi029.stdout:Enabling cephadm module... 2024-03-16T11:40:53.802 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:53 smithi029 ceph-mon[31975]: from='client.? 172.21.15.29:0/2018990881' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-03-16T11:40:53.803 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 16 11:40:53 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-x[32210]: ignoring --setuser ceph since I am not root 2024-03-16T11:40:53.803 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 16 11:40:53 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-x[32210]: ignoring --setgroup ceph since I am not root 2024-03-16T11:40:54.091 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:53.217+0000 7fd61d50e700 1 Processor -- start 2024-03-16T11:40:54.091 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:53.217+0000 7fd61d50e700 1 -- start start 2024-03-16T11:40:54.091 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:53.217+0000 7fd61d50e700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd618106c10 0x7fd618107030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:40:54.091 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:53.217+0000 7fd61d50e700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fd618107600 con 0x7fd618106c10 2024-03-16T11:40:54.091 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:53.218+0000 7fd616ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd618106c10 0x7fd618107030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:40:54.091 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:53.218+0000 7fd616ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd618106c10 0x7fd618107030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60854/0 (socket says 172.21.15.29:60854) 2024-03-16T11:40:54.091 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:53.218+0000 7fd616ffd700 1 -- 172.21.15.29:0/3731897244 learned_addr learned my addr 172.21.15.29:0/3731897244 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:40:54.091 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:53.218+0000 7fd616ffd700 1 -- 172.21.15.29:0/3731897244 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd618107e10 con 0x7fd618106c10 2024-03-16T11:40:54.091 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:53.218+0000 7fd616ffd700 1 --2- 172.21.15.29:0/3731897244 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd618106c10 0x7fd618107030 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fd61000d180 tx=0x7fd61000d490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d861c59b1464de3f server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:40:54.091 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:53.218+0000 7fd615ffb700 1 -- 172.21.15.29:0/3731897244 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd610019070 con 0x7fd618106c10 2024-03-16T11:40:54.091 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:53.218+0000 7fd615ffb700 1 -- 172.21.15.29:0/3731897244 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fd61000de20 con 0x7fd618106c10 2024-03-16T11:40:54.091 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:53.218+0000 7fd615ffb700 1 -- 172.21.15.29:0/3731897244 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd610009b90 con 0x7fd618106c10 2024-03-16T11:40:54.091 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:53.219+0000 7fd61d50e700 1 -- 172.21.15.29:0/3731897244 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd618106c10 msgr2=0x7fd618107030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:40:54.091 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:53.219+0000 7fd61d50e700 1 --2- 172.21.15.29:0/3731897244 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd618106c10 0x7fd618107030 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fd61000d180 tx=0x7fd61000d490 comp rx=0 tx=0).stop 2024-03-16T11:40:54.092 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:53.219+0000 7fd61d50e700 1 -- 172.21.15.29:0/3731897244 shutdown_connections 2024-03-16T11:40:54.092 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:53.219+0000 7fd61d50e700 1 --2- 172.21.15.29:0/3731897244 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd618106c10 0x7fd618107030 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:40:54.092 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:53.219+0000 7fd61d50e700 1 -- 172.21.15.29:0/3731897244 >> 172.21.15.29:0/3731897244 conn(0x7fd6181021b0 msgr2=0x7fd6181045f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:40:54.092 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:53.219+0000 7fd61d50e700 1 -- 172.21.15.29:0/3731897244 shutdown_connections 2024-03-16T11:40:54.092 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:53.219+0000 7fd61d50e700 1 -- 172.21.15.29:0/3731897244 wait complete. 2024-03-16T11:40:54.092 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:53.220+0000 7fd61d50e700 1 Processor -- start 2024-03-16T11:40:54.092 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:53.220+0000 7fd61d50e700 1 -- start start 2024-03-16T11:40:54.092 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:53.220+0000 7fd61d50e700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd618106c10 0x7fd618199b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:40:54.092 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:53.220+0000 7fd61d50e700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fd61807af90 con 0x7fd618106c10 2024-03-16T11:40:54.092 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:53.220+0000 7fd616ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd618106c10 0x7fd618199b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:40:54.092 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:53.220+0000 7fd616ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd618106c10 0x7fd618199b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60868/0 (socket says 172.21.15.29:60868) 2024-03-16T11:40:54.092 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:53.220+0000 7fd616ffd700 1 -- 172.21.15.29:0/2018990881 learned_addr learned my addr 172.21.15.29:0/2018990881 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:40:54.092 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:53.220+0000 7fd616ffd700 1 -- 172.21.15.29:0/2018990881 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd61819a080 con 0x7fd618106c10 2024-03-16T11:40:54.092 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:53.221+0000 7fd616ffd700 1 --2- 172.21.15.29:0/2018990881 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd618106c10 0x7fd618199b40 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fd61000d150 tx=0x7fd61000b820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:40:54.092 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:53.221+0000 7fd60ffff700 1 -- 172.21.15.29:0/2018990881 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd610019050 con 0x7fd618106c10 2024-03-16T11:40:54.092 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:53.221+0000 7fd60ffff700 1 -- 172.21.15.29:0/2018990881 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fd610025070 con 0x7fd618106c10 2024-03-16T11:40:54.093 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:53.221+0000 7fd61d50e700 1 -- 172.21.15.29:0/2018990881 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd61819a310 con 0x7fd618106c10 2024-03-16T11:40:54.093 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:53.221+0000 7fd60ffff700 1 -- 172.21.15.29:0/2018990881 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd6100045a0 con 0x7fd618106c10 2024-03-16T11:40:54.093 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:53.221+0000 7fd61d50e700 1 -- 172.21.15.29:0/2018990881 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd61819cfb0 con 0x7fd618106c10 2024-03-16T11:40:54.093 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:53.222+0000 7fd60ffff700 1 -- 172.21.15.29:0/2018990881 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 4) ==== 46095+0+0 (secure 0 0 0) 0x7fd610004820 con 0x7fd618106c10 2024-03-16T11:40:54.093 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:53.222+0000 7fd60ffff700 1 --2- 172.21.15.29:0/2018990881 >> [v2:172.21.15.29:6800/2275315654,v1:172.21.15.29:6801/2275315654] conn(0x7fd604038e80 0x7fd60403b340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:40:54.093 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:53.222+0000 7fd61d50e700 1 -- 172.21.15.29:0/2018990881 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd618063900 con 0x7fd618106c10 2024-03-16T11:40:54.093 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:53.222+0000 7fd60ffff700 1 -- 172.21.15.29:0/2018990881 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fd610011050 con 0x7fd618106c10 2024-03-16T11:40:54.093 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:53.222+0000 7fd6167fc700 1 --2- 172.21.15.29:0/2018990881 >> [v2:172.21.15.29:6800/2275315654,v1:172.21.15.29:6801/2275315654] conn(0x7fd604038e80 0x7fd60403b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:40:54.093 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:53.222+0000 7fd6167fc700 1 --2- 172.21.15.29:0/2018990881 >> [v2:172.21.15.29:6800/2275315654,v1:172.21.15.29:6801/2275315654] conn(0x7fd604038e80 0x7fd60403b340 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fd600006fd0 tx=0x7fd600006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:40:54.093 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:53.227+0000 7fd60ffff700 1 -- 172.21.15.29:0/2018990881 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fd61002ad00 con 0x7fd618106c10 2024-03-16T11:40:54.093 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:53.492+0000 7fd61d50e700 1 -- 172.21.15.29:0/2018990881 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) -- 0x7fd6180008d0 con 0x7fd618106c10 2024-03-16T11:40:54.093 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:53.496+0000 7fd60ffff700 1 -- 172.21.15.29:0/2018990881 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) ==== 86+0+0 (secure 0 0 0) 0x7fd61004f270 con 0x7fd618106c10 2024-03-16T11:40:54.093 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:53.498+0000 7fd60ffff700 1 -- 172.21.15.29:0/2018990881 <== mon.0 v2:172.21.15.29:3300/0 8 ==== mgrmap(e 5) ==== 46106+0+0 (secure 0 0 0) 0x7fd610026cb0 con 0x7fd618106c10 2024-03-16T11:40:54.093 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:53.499+0000 7fd61d50e700 1 -- 172.21.15.29:0/2018990881 >> [v2:172.21.15.29:6800/2275315654,v1:172.21.15.29:6801/2275315654] conn(0x7fd604038e80 msgr2=0x7fd60403b340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:40:54.093 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:53.499+0000 7fd61d50e700 1 --2- 172.21.15.29:0/2018990881 >> [v2:172.21.15.29:6800/2275315654,v1:172.21.15.29:6801/2275315654] conn(0x7fd604038e80 0x7fd60403b340 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fd600006fd0 tx=0x7fd600006e40 comp rx=0 tx=0).stop 2024-03-16T11:40:54.094 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:53.499+0000 7fd61d50e700 1 -- 172.21.15.29:0/2018990881 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd618106c10 msgr2=0x7fd618199b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:40:54.094 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:53.499+0000 7fd61d50e700 1 --2- 172.21.15.29:0/2018990881 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd618106c10 0x7fd618199b40 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fd61000d150 tx=0x7fd61000b820 comp rx=0 tx=0).stop 2024-03-16T11:40:54.094 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:53.499+0000 7fd61d50e700 1 -- 172.21.15.29:0/2018990881 shutdown_connections 2024-03-16T11:40:54.094 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:53.499+0000 7fd61d50e700 1 --2- 172.21.15.29:0/2018990881 >> [v2:172.21.15.29:6800/2275315654,v1:172.21.15.29:6801/2275315654] conn(0x7fd604038e80 0x7fd60403b340 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:40:54.094 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:53.499+0000 7fd61d50e700 1 --2- 172.21.15.29:0/2018990881 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd618106c10 0x7fd618199b40 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:40:54.094 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:53.499+0000 7fd61d50e700 1 -- 172.21.15.29:0/2018990881 >> 172.21.15.29:0/2018990881 conn(0x7fd6181021b0 msgr2=0x7fd6181905e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:40:54.094 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:53.499+0000 7fd61d50e700 1 -- 172.21.15.29:0/2018990881 shutdown_connections 2024-03-16T11:40:54.094 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:53.499+0000 7fd61d50e700 1 -- 172.21.15.29:0/2018990881 wait complete. 2024-03-16T11:40:54.337 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 16 11:40:54 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-x[32210]: 2024-03-16T11:40:54.069+0000 7ff0c0562200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-16T11:40:54.338 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 16 11:40:54 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-x[32210]: 2024-03-16T11:40:54.169+0000 7ff0c0562200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-16T11:40:54.676 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:54 smithi029 ceph-mon[31975]: from='client.? 172.21.15.29:0/2018990881' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-03-16T11:40:54.676 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:54 smithi029 ceph-mon[31975]: mgrmap e5: x(active, since 6s) 2024-03-16T11:40:54.932 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 16 11:40:54 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-x[32210]: 2024-03-16T11:40:54.837+0000 7ff0c0562200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-16T11:40:54.932 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 16 11:40:54 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-x[32210]: 2024-03-16T11:40:54.932+0000 7ff0c0562200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-16T11:40:55.199 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 16 11:40:55 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-x[32210]: 2024-03-16T11:40:55.079+0000 7ff0c0562200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-16T11:40:55.199 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 16 11:40:55 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-x[32210]: 2024-03-16T11:40:55.178+0000 7ff0c0562200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-16T11:40:55.461 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 16 11:40:55 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-x[32210]: 2024-03-16T11:40:55.263+0000 7ff0c0562200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-16T11:40:55.466 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout { 2024-03-16T11:40:55.466 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-03-16T11:40:55.466 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-16T11:40:55.466 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-03-16T11:40:55.466 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-16T11:40:55.466 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout } 2024-03-16T11:40:55.466 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:54.677+0000 7fa7f894e700 1 Processor -- start 2024-03-16T11:40:55.466 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:54.677+0000 7fa7f894e700 1 -- start start 2024-03-16T11:40:55.466 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:54.677+0000 7fa7f894e700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa7f407de60 0x7fa7f407c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:40:55.466 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:54.677+0000 7fa7f894e700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fa7f407c890 con 0x7fa7f407de60 2024-03-16T11:40:55.466 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:54.677+0000 7fa7f259c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa7f407de60 0x7fa7f407c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:40:55.466 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:54.677+0000 7fa7f259c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa7f407de60 0x7fa7f407c2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60908/0 (socket says 172.21.15.29:60908) 2024-03-16T11:40:55.466 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:54.677+0000 7fa7f259c700 1 -- 172.21.15.29:0/3621747077 learned_addr learned my addr 172.21.15.29:0/3621747077 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:40:55.467 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:54.677+0000 7fa7f259c700 1 -- 172.21.15.29:0/3621747077 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7f407ca10 con 0x7fa7f407de60 2024-03-16T11:40:55.467 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:54.678+0000 7fa7f259c700 1 --2- 172.21.15.29:0/3621747077 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa7f407de60 0x7fa7f407c2c0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fa7e4009e80 tx=0x7fa7e400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7cfef9a49b756128 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:40:55.467 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:54.678+0000 7fa7f1d9b700 1 -- 172.21.15.29:0/3621747077 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa7e4017070 con 0x7fa7f407de60 2024-03-16T11:40:55.467 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:54.678+0000 7fa7f1d9b700 1 -- 172.21.15.29:0/3621747077 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fa7e400b8c0 con 0x7fa7f407de60 2024-03-16T11:40:55.467 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:54.678+0000 7fa7f1d9b700 1 -- 172.21.15.29:0/3621747077 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa7e40045c0 con 0x7fa7f407de60 2024-03-16T11:40:55.467 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:54.678+0000 7fa7f894e700 1 -- 172.21.15.29:0/3621747077 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa7f407de60 msgr2=0x7fa7f407c2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:40:55.467 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:54.678+0000 7fa7f894e700 1 --2- 172.21.15.29:0/3621747077 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa7f407de60 0x7fa7f407c2c0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fa7e4009e80 tx=0x7fa7e400b330 comp rx=0 tx=0).stop 2024-03-16T11:40:55.467 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:54.678+0000 7fa7f894e700 1 -- 172.21.15.29:0/3621747077 shutdown_connections 2024-03-16T11:40:55.467 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:54.679+0000 7fa7f894e700 1 --2- 172.21.15.29:0/3621747077 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa7f407de60 0x7fa7f407c2c0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:40:55.467 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:54.679+0000 7fa7f894e700 1 -- 172.21.15.29:0/3621747077 >> 172.21.15.29:0/3621747077 conn(0x7fa7f41032f0 msgr2=0x7fa7f4105710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:40:55.467 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:54.679+0000 7fa7f894e700 1 -- 172.21.15.29:0/3621747077 shutdown_connections 2024-03-16T11:40:55.467 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:54.679+0000 7fa7f894e700 1 -- 172.21.15.29:0/3621747077 wait complete. 2024-03-16T11:40:55.467 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:54.679+0000 7fa7f894e700 1 Processor -- start 2024-03-16T11:40:55.467 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:54.679+0000 7fa7f894e700 1 -- start start 2024-03-16T11:40:55.471 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:54.680+0000 7fa7f894e700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa7f407de60 0x7fa7f419dee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:40:55.471 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:54.680+0000 7fa7f894e700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fa7f410e340 con 0x7fa7f407de60 2024-03-16T11:40:55.471 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:54.680+0000 7fa7f259c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa7f407de60 0x7fa7f419dee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:40:55.471 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:54.680+0000 7fa7f259c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa7f407de60 0x7fa7f419dee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60914/0 (socket says 172.21.15.29:60914) 2024-03-16T11:40:55.471 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:54.680+0000 7fa7f259c700 1 -- 172.21.15.29:0/3776755557 learned_addr learned my addr 172.21.15.29:0/3776755557 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:40:55.471 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:54.680+0000 7fa7f259c700 1 -- 172.21.15.29:0/3776755557 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7f419e420 con 0x7fa7f407de60 2024-03-16T11:40:55.471 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:54.680+0000 7fa7f259c700 1 --2- 172.21.15.29:0/3776755557 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa7f407de60 0x7fa7f419dee0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fa7e4009e50 tx=0x7fa7e400bd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:40:55.471 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:54.680+0000 7fa7ebfff700 1 -- 172.21.15.29:0/3776755557 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa7e4020020 con 0x7fa7f407de60 2024-03-16T11:40:55.471 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:54.680+0000 7fa7f894e700 1 -- 172.21.15.29:0/3776755557 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa7f419e6b0 con 0x7fa7f407de60 2024-03-16T11:40:55.471 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:54.680+0000 7fa7ebfff700 1 -- 172.21.15.29:0/3776755557 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fa7e4003b50 con 0x7fa7f407de60 2024-03-16T11:40:55.471 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:54.680+0000 7fa7ebfff700 1 -- 172.21.15.29:0/3776755557 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa7e4017040 con 0x7fa7f407de60 2024-03-16T11:40:55.471 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:54.680+0000 7fa7f894e700 1 -- 172.21.15.29:0/3776755557 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa7f41a13a0 con 0x7fa7f407de60 2024-03-16T11:40:55.472 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:54.682+0000 7fa7f894e700 1 -- 172.21.15.29:0/3776755557 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa7f4063900 con 0x7fa7f407de60 2024-03-16T11:40:55.472 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:54.682+0000 7fa7ebfff700 1 -- 172.21.15.29:0/3776755557 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 46106+0+0 (secure 0 0 0) 0x7fa7e4023020 con 0x7fa7f407de60 2024-03-16T11:40:55.472 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:54.682+0000 7fa7ebfff700 1 --2- 172.21.15.29:0/3776755557 >> [v2:172.21.15.29:6800/2275315654,v1:172.21.15.29:6801/2275315654] conn(0x7fa7e0038ed0 0x7fa7e003b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:40:55.472 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:54.682+0000 7fa7ebfff700 1 -- 172.21.15.29:0/3776755557 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fa7e400f050 con 0x7fa7f407de60 2024-03-16T11:40:55.472 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:54.682+0000 7fa7e9d9b700 1 -- 172.21.15.29:0/3776755557 >> [v2:172.21.15.29:6800/2275315654,v1:172.21.15.29:6801/2275315654] conn(0x7fa7e0038ed0 msgr2=0x7fa7e003b390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6800/2275315654 2024-03-16T11:40:55.472 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:54.682+0000 7fa7e9d9b700 1 --2- 172.21.15.29:0/3776755557 >> [v2:172.21.15.29:6800/2275315654,v1:172.21.15.29:6801/2275315654] conn(0x7fa7e0038ed0 0x7fa7e003b390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-16T11:40:55.472 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:54.691+0000 7fa7ebfff700 1 -- 172.21.15.29:0/3776755557 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fa7e402a020 con 0x7fa7f407de60 2024-03-16T11:40:55.472 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:54.883+0000 7fa7e9d9b700 1 -- 172.21.15.29:0/3776755557 >> [v2:172.21.15.29:6800/2275315654,v1:172.21.15.29:6801/2275315654] conn(0x7fa7e0038ed0 msgr2=0x7fa7e003b390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6800/2275315654 2024-03-16T11:40:55.472 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:54.883+0000 7fa7e9d9b700 1 --2- 172.21.15.29:0/3776755557 >> [v2:172.21.15.29:6800/2275315654,v1:172.21.15.29:6801/2275315654] conn(0x7fa7e0038ed0 0x7fa7e003b390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-16T11:40:55.472 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:54.956+0000 7fa7f894e700 1 -- 172.21.15.29:0/3776755557 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7fa7f41a1570 con 0x7fa7f407de60 2024-03-16T11:40:55.472 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:54.956+0000 7fa7ebfff700 1 -- 172.21.15.29:0/3776755557 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) ==== 56+0+88 (secure 0 0 0) 0x7fa7f41a1570 con 0x7fa7f407de60 2024-03-16T11:40:55.472 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:54.959+0000 7fa7f894e700 1 -- 172.21.15.29:0/3776755557 >> [v2:172.21.15.29:6800/2275315654,v1:172.21.15.29:6801/2275315654] conn(0x7fa7e0038ed0 msgr2=0x7fa7e003b390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-16T11:40:55.472 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:54.959+0000 7fa7f894e700 1 --2- 172.21.15.29:0/3776755557 >> [v2:172.21.15.29:6800/2275315654,v1:172.21.15.29:6801/2275315654] conn(0x7fa7e0038ed0 0x7fa7e003b390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:40:55.472 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:54.959+0000 7fa7f894e700 1 -- 172.21.15.29:0/3776755557 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa7f407de60 msgr2=0x7fa7f419dee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:40:55.473 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:54.959+0000 7fa7f894e700 1 --2- 172.21.15.29:0/3776755557 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa7f407de60 0x7fa7f419dee0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fa7e4009e50 tx=0x7fa7e400bd30 comp rx=0 tx=0).stop 2024-03-16T11:40:55.473 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:54.959+0000 7fa7f894e700 1 -- 172.21.15.29:0/3776755557 shutdown_connections 2024-03-16T11:40:55.473 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:54.959+0000 7fa7f894e700 1 --2- 172.21.15.29:0/3776755557 >> [v2:172.21.15.29:6800/2275315654,v1:172.21.15.29:6801/2275315654] conn(0x7fa7e0038ed0 0x7fa7e003b390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:40:55.473 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:54.959+0000 7fa7f894e700 1 --2- 172.21.15.29:0/3776755557 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa7f407de60 0x7fa7f419dee0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:40:55.473 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:54.959+0000 7fa7f894e700 1 -- 172.21.15.29:0/3776755557 >> 172.21.15.29:0/3776755557 conn(0x7fa7f41032f0 msgr2=0x7fa7f4103e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:40:55.473 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:54.959+0000 7fa7f894e700 1 -- 172.21.15.29:0/3776755557 shutdown_connections 2024-03-16T11:40:55.473 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:54.959+0000 7fa7f894e700 1 -- 172.21.15.29:0/3776755557 wait complete. 2024-03-16T11:40:55.473 INFO:teuthology.orchestra.run.smithi029.stdout:Waiting for the mgr to restart... 2024-03-16T11:40:55.473 INFO:teuthology.orchestra.run.smithi029.stdout:Waiting for mgr epoch 5... 2024-03-16T11:40:55.744 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:40:55 smithi029 ceph-mon[31975]: from='client.? 172.21.15.29:0/3776755557' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-16T11:40:55.745 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 16 11:40:55 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-x[32210]: 2024-03-16T11:40:55.542+0000 7ff0c0562200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-16T11:40:55.745 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 16 11:40:55 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-x[32210]: 2024-03-16T11:40:55.654+0000 7ff0c0562200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-16T11:40:55.745 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 16 11:40:55 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-x[32210]: 2024-03-16T11:40:55.745+0000 7ff0c0562200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-16T11:40:56.024 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 16 11:40:55 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-x[32210]: 2024-03-16T11:40:55.845+0000 7ff0c0562200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-16T11:40:56.025 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 16 11:40:56 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-x[32210]: 2024-03-16T11:40:56.025+0000 7ff0c0562200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-16T11:40:57.274 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 16 11:40:57 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-x[32210]: 2024-03-16T11:40:57.006+0000 7ff0c0562200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-16T11:40:57.274 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 16 11:40:57 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-x[32210]: 2024-03-16T11:40:57.144+0000 7ff0c0562200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-16T11:40:57.582 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 16 11:40:57 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-x[32210]: 2024-03-16T11:40:57.488+0000 7ff0c0562200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-16T11:40:57.582 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 16 11:40:57 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-x[32210]: 2024-03-16T11:40:57.583+0000 7ff0c0562200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-16T11:40:57.838 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 16 11:40:57 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-x[32210]: 2024-03-16T11:40:57.787+0000 7ff0c0562200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-16T11:40:58.268 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 16 11:40:57 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-x[32210]: 2024-03-16T11:40:57.978+0000 7ff0c0562200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-16T11:40:58.531 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 16 11:40:58 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-x[32210]: 2024-03-16T11:40:58.272+0000 7ff0c0562200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-16T11:40:58.532 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 16 11:40:58 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-x[32210]: 2024-03-16T11:40:58.532+0000 7ff0c0562200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-16T11:40:58.838 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 16 11:40:58 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-x[32210]: 2024-03-16T11:40:58.773+0000 7ff0c0562200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-16T11:41:00.088 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 16 11:40:59 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-x[32210]: 2024-03-16T11:40:59.594+0000 7ff0c0562200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-16T11:41:00.491 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 16 11:41:00 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-x[32210]: 2024-03-16T11:41:00.172+0000 7ff0c0562200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-16T11:41:00.491 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 16 11:41:00 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-x[32210]: 2024-03-16T11:41:00.272+0000 7ff0c0562200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-16T11:41:00.491 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 16 11:41:00 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-x[32210]: 2024-03-16T11:41:00.491+0000 7ff0c0562200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-16T11:41:00.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:41:00 smithi029 ceph-mon[31975]: Active manager daemon x restarted 2024-03-16T11:41:00.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:41:00 smithi029 ceph-mon[31975]: Activating manager daemon x 2024-03-16T11:41:00.839 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:41:00 smithi029 ceph-mon[31975]: osdmap e2: 0 total, 0 up, 0 in 2024-03-16T11:41:00.839 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:41:00 smithi029 ceph-mon[31975]: mgrmap e6: x(active, starting, since 0.00335073s) 2024-03-16T11:41:00.839 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:41:00 smithi029 ceph-mon[31975]: from='mgr.14120 172.21.15.29:0/4098640102' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-16T11:41:00.839 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:41:00 smithi029 ceph-mon[31975]: from='mgr.14120 172.21.15.29:0/4098640102' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-16T11:41:00.839 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:41:00 smithi029 ceph-mon[31975]: from='mgr.14120 172.21.15.29:0/4098640102' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-16T11:41:00.839 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:41:00 smithi029 ceph-mon[31975]: from='mgr.14120 172.21.15.29:0/4098640102' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-16T11:41:00.839 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:41:00 smithi029 ceph-mon[31975]: from='mgr.14120 172.21.15.29:0/4098640102' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-16T11:41:00.839 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:41:00 smithi029 ceph-mon[31975]: Manager daemon x is now available 2024-03-16T11:41:00.839 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:41:00 smithi029 ceph-mon[31975]: Found migration_current of "None". Setting to last migration. 2024-03-16T11:41:00.839 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:41:00 smithi029 ceph-mon[31975]: from='mgr.14120 172.21.15.29:0/4098640102' entity='mgr.x' 2024-03-16T11:41:00.839 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:41:00 smithi029 ceph-mon[31975]: from='mgr.14120 172.21.15.29:0/4098640102' entity='mgr.x' 2024-03-16T11:41:00.839 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:41:00 smithi029 ceph-mon[31975]: from='mgr.14120 172.21.15.29:0/4098640102' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T11:41:00.839 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:41:00 smithi029 ceph-mon[31975]: from='mgr.14120 172.21.15.29:0/4098640102' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T11:41:01.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:41:01 smithi029 ceph-mon[31975]: from='mgr.14120 172.21.15.29:0/4098640102' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T11:41:01.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:41:01 smithi029 ceph-mon[31975]: from='mgr.14120 172.21.15.29:0/4098640102' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-16T11:41:01.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:41:01 smithi029 ceph-mon[31975]: from='mgr.14120 172.21.15.29:0/4098640102' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-16T11:41:01.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:41:01 smithi029 ceph-mon[31975]: from='mgr.14120 172.21.15.29:0/4098640102' entity='mgr.x' 2024-03-16T11:41:01.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:41:01 smithi029 ceph-mon[31975]: from='mgr.14120 172.21.15.29:0/4098640102' entity='mgr.x' 2024-03-16T11:41:01.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:41:01 smithi029 ceph-mon[31975]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-16T11:41:01.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:41:01 smithi029 ceph-mon[31975]: mgrmap e7: x(active, since 1.00522s) 2024-03-16T11:41:01.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:41:01 smithi029 ceph-mon[31975]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-16T11:41:02.124 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout { 2024-03-16T11:41:02.124 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-03-16T11:41:02.124 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-16T11:41:02.124 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout } 2024-03-16T11:41:02.124 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:55.867+0000 7f52d3478700 1 Processor -- start 2024-03-16T11:41:02.125 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:55.867+0000 7f52d3478700 1 -- start start 2024-03-16T11:41:02.125 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:55.868+0000 7f52d3478700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f52cc1087b0 0x7f52cc10aba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:41:02.125 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:55.868+0000 7f52d3478700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f52cc07be30 con 0x7f52cc1087b0 2024-03-16T11:41:02.125 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:55.868+0000 7f52d1214700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f52cc1087b0 0x7f52cc10aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:41:02.125 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:55.868+0000 7f52d1214700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f52cc1087b0 0x7f52cc10aba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60930/0 (socket says 172.21.15.29:60930) 2024-03-16T11:41:02.125 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:55.868+0000 7f52d1214700 1 -- 172.21.15.29:0/372693350 learned_addr learned my addr 172.21.15.29:0/372693350 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:41:02.125 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:55.868+0000 7f52d1214700 1 -- 172.21.15.29:0/372693350 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52cc10b0e0 con 0x7f52cc1087b0 2024-03-16T11:41:02.125 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:55.868+0000 7f52d1214700 1 --2- 172.21.15.29:0/372693350 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f52cc1087b0 0x7f52cc10aba0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f52c80098a0 tx=0x7f52c8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44e2c3717390e091 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:41:02.125 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:55.868+0000 7f52c3fff700 1 -- 172.21.15.29:0/372693350 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f52c8004160 con 0x7f52cc1087b0 2024-03-16T11:41:02.126 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:55.868+0000 7f52c3fff700 1 -- 172.21.15.29:0/372693350 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f52c8004300 con 0x7f52cc1087b0 2024-03-16T11:41:02.126 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:55.868+0000 7f52c3fff700 1 -- 172.21.15.29:0/372693350 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f52c8011b50 con 0x7f52cc1087b0 2024-03-16T11:41:02.126 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:55.869+0000 7f52d3478700 1 -- 172.21.15.29:0/372693350 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f52cc1087b0 msgr2=0x7f52cc10aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:41:02.126 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:55.869+0000 7f52d3478700 1 --2- 172.21.15.29:0/372693350 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f52cc1087b0 0x7f52cc10aba0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f52c80098a0 tx=0x7f52c8009bb0 comp rx=0 tx=0).stop 2024-03-16T11:41:02.126 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:55.869+0000 7f52d3478700 1 -- 172.21.15.29:0/372693350 shutdown_connections 2024-03-16T11:41:02.126 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:55.869+0000 7f52d3478700 1 --2- 172.21.15.29:0/372693350 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f52cc1087b0 0x7f52cc10aba0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:41:02.126 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:55.869+0000 7f52d3478700 1 -- 172.21.15.29:0/372693350 >> 172.21.15.29:0/372693350 conn(0x7f52cc1021a0 msgr2=0x7f52cc1045e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:41:02.126 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:55.869+0000 7f52d3478700 1 -- 172.21.15.29:0/372693350 shutdown_connections 2024-03-16T11:41:02.126 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:55.869+0000 7f52d3478700 1 -- 172.21.15.29:0/372693350 wait complete. 2024-03-16T11:41:02.127 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:55.870+0000 7f52d3478700 1 Processor -- start 2024-03-16T11:41:02.127 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:55.870+0000 7f52d3478700 1 -- start start 2024-03-16T11:41:02.127 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:55.870+0000 7f52d3478700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f52cc1087b0 0x7f52cc19bcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:41:02.127 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:55.870+0000 7f52d3478700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f52cc10c100 con 0x7f52cc1087b0 2024-03-16T11:41:02.127 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:55.870+0000 7f52d1214700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f52cc1087b0 0x7f52cc19bcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:41:02.127 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:55.870+0000 7f52d1214700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f52cc1087b0 0x7f52cc19bcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60938/0 (socket says 172.21.15.29:60938) 2024-03-16T11:41:02.127 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:55.870+0000 7f52d1214700 1 -- 172.21.15.29:0/2782034243 learned_addr learned my addr 172.21.15.29:0/2782034243 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:41:02.127 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:55.870+0000 7f52d1214700 1 -- 172.21.15.29:0/2782034243 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52cc19c1f0 con 0x7f52cc1087b0 2024-03-16T11:41:02.127 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:55.871+0000 7f52d1214700 1 --2- 172.21.15.29:0/2782034243 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f52cc1087b0 0x7f52cc19bcb0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f52c8011b60 tx=0x7f52c8011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:41:02.128 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:55.871+0000 7f52c27fc700 1 -- 172.21.15.29:0/2782034243 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f52c8003970 con 0x7f52cc1087b0 2024-03-16T11:41:02.128 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:55.871+0000 7f52c27fc700 1 -- 172.21.15.29:0/2782034243 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f52c8003b10 con 0x7f52cc1087b0 2024-03-16T11:41:02.128 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:55.871+0000 7f52c27fc700 1 -- 172.21.15.29:0/2782034243 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f52c8024610 con 0x7f52cc1087b0 2024-03-16T11:41:02.128 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:55.871+0000 7f52d3478700 1 -- 172.21.15.29:0/2782034243 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f52cc19c480 con 0x7f52cc1087b0 2024-03-16T11:41:02.128 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:55.871+0000 7f52d3478700 1 -- 172.21.15.29:0/2782034243 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f52cc19f170 con 0x7f52cc1087b0 2024-03-16T11:41:02.128 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:55.872+0000 7f52c27fc700 1 -- 172.21.15.29:0/2782034243 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 46106+0+0 (secure 0 0 0) 0x7f52c8024910 con 0x7f52cc1087b0 2024-03-16T11:41:02.128 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:55.872+0000 7f52c27fc700 1 --2- 172.21.15.29:0/2782034243 >> [v2:172.21.15.29:6800/2275315654,v1:172.21.15.29:6801/2275315654] conn(0x7f52b8038f20 0x7f52b803b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:41:02.128 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:55.872+0000 7f52c27fc700 1 -- 172.21.15.29:0/2782034243 --> [v2:172.21.15.29:6800/2275315654,v1:172.21.15.29:6801/2275315654] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f52b803baf0 con 0x7f52b8038f20 2024-03-16T11:41:02.128 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:55.872+0000 7f52c27fc700 1 -- 172.21.15.29:0/2782034243 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f52c8011e00 con 0x7f52cc1087b0 2024-03-16T11:41:02.129 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:55.872+0000 7f52d0a13700 1 -- 172.21.15.29:0/2782034243 >> [v2:172.21.15.29:6800/2275315654,v1:172.21.15.29:6801/2275315654] conn(0x7f52b8038f20 msgr2=0x7f52b803b3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6800/2275315654 2024-03-16T11:41:02.129 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:55.872+0000 7f52d0a13700 1 --2- 172.21.15.29:0/2782034243 >> [v2:172.21.15.29:6800/2275315654,v1:172.21.15.29:6801/2275315654] conn(0x7f52b8038f20 0x7f52b803b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-16T11:41:02.129 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:56.073+0000 7f52d0a13700 1 -- 172.21.15.29:0/2782034243 >> [v2:172.21.15.29:6800/2275315654,v1:172.21.15.29:6801/2275315654] conn(0x7f52b8038f20 msgr2=0x7f52b803b3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6800/2275315654 2024-03-16T11:41:02.129 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:56.073+0000 7f52d0a13700 1 --2- 172.21.15.29:0/2782034243 >> [v2:172.21.15.29:6800/2275315654,v1:172.21.15.29:6801/2275315654] conn(0x7f52b8038f20 0x7f52b803b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-16T11:41:02.129 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:56.474+0000 7f52d0a13700 1 -- 172.21.15.29:0/2782034243 >> [v2:172.21.15.29:6800/2275315654,v1:172.21.15.29:6801/2275315654] conn(0x7f52b8038f20 msgr2=0x7f52b803b3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6800/2275315654 2024-03-16T11:41:02.129 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:56.474+0000 7f52d0a13700 1 --2- 172.21.15.29:0/2782034243 >> [v2:172.21.15.29:6800/2275315654,v1:172.21.15.29:6801/2275315654] conn(0x7f52b8038f20 0x7f52b803b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-16T11:41:02.129 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:57.275+0000 7f52d0a13700 1 -- 172.21.15.29:0/2782034243 >> [v2:172.21.15.29:6800/2275315654,v1:172.21.15.29:6801/2275315654] conn(0x7f52b8038f20 msgr2=0x7f52b803b3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6800/2275315654 2024-03-16T11:41:02.129 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:57.275+0000 7f52d0a13700 1 --2- 172.21.15.29:0/2782034243 >> [v2:172.21.15.29:6800/2275315654,v1:172.21.15.29:6801/2275315654] conn(0x7f52b8038f20 0x7f52b803b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-16T11:41:02.129 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:58.876+0000 7f52d0a13700 1 -- 172.21.15.29:0/2782034243 >> [v2:172.21.15.29:6800/2275315654,v1:172.21.15.29:6801/2275315654] conn(0x7f52b8038f20 msgr2=0x7f52b803b3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6800/2275315654 2024-03-16T11:41:02.129 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:40:58.876+0000 7f52d0a13700 1 --2- 172.21.15.29:0/2782034243 >> [v2:172.21.15.29:6800/2275315654,v1:172.21.15.29:6801/2275315654] conn(0x7f52b8038f20 0x7f52b803b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-16T11:41:02.130 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:00.507+0000 7f52c27fc700 1 -- 172.21.15.29:0/2782034243 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mgrmap(e 6) ==== 45873+0+0 (secure 0 0 0) 0x7f52c8024c00 con 0x7f52cc1087b0 2024-03-16T11:41:02.130 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:00.507+0000 7f52c27fc700 1 -- 172.21.15.29:0/2782034243 >> [v2:172.21.15.29:6800/2275315654,v1:172.21.15.29:6801/2275315654] conn(0x7f52b8038f20 msgr2=0x7f52b803b3e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-16T11:41:02.130 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:00.507+0000 7f52c27fc700 1 --2- 172.21.15.29:0/2782034243 >> [v2:172.21.15.29:6800/2275315654,v1:172.21.15.29:6801/2275315654] conn(0x7f52b8038f20 0x7f52b803b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:41:02.130 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:01.508+0000 7f52c27fc700 1 -- 172.21.15.29:0/2782034243 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mgrmap(e 7) ==== 46000+0+0 (secure 0 0 0) 0x7f52c804efa0 con 0x7f52cc1087b0 2024-03-16T11:41:02.130 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:01.508+0000 7f52c27fc700 1 --2- 172.21.15.29:0/2782034243 >> [v2:172.21.15.29:6800/2798831128,v1:172.21.15.29:6801/2798831128] conn(0x7f52b803c810 0x7f52b803ec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:41:02.130 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:01.508+0000 7f52c27fc700 1 -- 172.21.15.29:0/2782034243 --> [v2:172.21.15.29:6800/2798831128,v1:172.21.15.29:6801/2798831128] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f52b803baf0 con 0x7f52b803c810 2024-03-16T11:41:02.130 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:01.508+0000 7f52d0a13700 1 --2- 172.21.15.29:0/2782034243 >> [v2:172.21.15.29:6800/2798831128,v1:172.21.15.29:6801/2798831128] conn(0x7f52b803c810 0x7f52b803ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:41:02.130 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:01.509+0000 7f52d0a13700 1 --2- 172.21.15.29:0/2782034243 >> [v2:172.21.15.29:6800/2798831128,v1:172.21.15.29:6801/2798831128] conn(0x7f52b803c810 0x7f52b803ec00 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f52bc003a10 tx=0x7f52bc009290 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:41:02.131 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:01.510+0000 7f52c27fc700 1 -- 172.21.15.29:0/2782034243 <== mgr.14120 v2:172.21.15.29:6800/2798831128 1 ==== command_reply(tid 0: 0 ) ==== 8+0+7774 (secure 0 0 0) 0x7f52b803baf0 con 0x7f52b803c810 2024-03-16T11:41:02.131 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:01.516+0000 7f52d3478700 1 -- 172.21.15.29:0/2782034243 --> [v2:172.21.15.29:6800/2798831128,v1:172.21.15.29:6801/2798831128] -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7f52cc063900 con 0x7f52b803c810 2024-03-16T11:41:02.131 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:01.516+0000 7f52c27fc700 1 -- 172.21.15.29:0/2782034243 <== mgr.14120 v2:172.21.15.29:6800/2798831128 2 ==== command_reply(tid 1: 0 ) ==== 8+0+51 (secure 0 0 0) 0x7f52cc063900 con 0x7f52b803c810 2024-03-16T11:41:02.131 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:01.516+0000 7f52d3478700 1 -- 172.21.15.29:0/2782034243 >> [v2:172.21.15.29:6800/2798831128,v1:172.21.15.29:6801/2798831128] conn(0x7f52b803c810 msgr2=0x7f52b803ec00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:41:02.131 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:01.516+0000 7f52d3478700 1 --2- 172.21.15.29:0/2782034243 >> [v2:172.21.15.29:6800/2798831128,v1:172.21.15.29:6801/2798831128] conn(0x7f52b803c810 0x7f52b803ec00 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f52bc003a10 tx=0x7f52bc009290 comp rx=0 tx=0).stop 2024-03-16T11:41:02.131 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:01.516+0000 7f52d3478700 1 -- 172.21.15.29:0/2782034243 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f52cc1087b0 msgr2=0x7f52cc19bcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:41:02.131 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:01.516+0000 7f52d3478700 1 --2- 172.21.15.29:0/2782034243 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f52cc1087b0 0x7f52cc19bcb0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f52c8011b60 tx=0x7f52c8011c40 comp rx=0 tx=0).stop 2024-03-16T11:41:02.131 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:01.517+0000 7f52d3478700 1 -- 172.21.15.29:0/2782034243 shutdown_connections 2024-03-16T11:41:02.131 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:01.517+0000 7f52d3478700 1 --2- 172.21.15.29:0/2782034243 >> [v2:172.21.15.29:6800/2798831128,v1:172.21.15.29:6801/2798831128] conn(0x7f52b803c810 0x7f52b803ec00 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:41:02.132 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:01.517+0000 7f52d3478700 1 --2- 172.21.15.29:0/2782034243 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f52cc1087b0 0x7f52cc19bcb0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:41:02.132 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:01.517+0000 7f52d3478700 1 --2- 172.21.15.29:0/2782034243 >> [v2:172.21.15.29:6800/2275315654,v1:172.21.15.29:6801/2275315654] conn(0x7f52b8038f20 0x7f52b803b3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:41:02.132 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:01.517+0000 7f52d3478700 1 -- 172.21.15.29:0/2782034243 >> 172.21.15.29:0/2782034243 conn(0x7f52cc1021a0 msgr2=0x7f52cc102c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:41:02.132 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:01.517+0000 7f52d3478700 1 -- 172.21.15.29:0/2782034243 shutdown_connections 2024-03-16T11:41:02.132 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:01.517+0000 7f52d3478700 1 -- 172.21.15.29:0/2782034243 wait complete. 2024-03-16T11:41:02.132 INFO:teuthology.orchestra.run.smithi029.stdout:mgr epoch 5 is available 2024-03-16T11:41:02.132 INFO:teuthology.orchestra.run.smithi029.stdout:Setting orchestrator backend to cephadm... 2024-03-16T11:41:02.748 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:41:02 smithi029 ceph-mon[31975]: [16/Mar/2024:11:41:01] ENGINE Bus STARTING 2024-03-16T11:41:02.748 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:41:02 smithi029 ceph-mon[31975]: [16/Mar/2024:11:41:01] ENGINE Serving on http://172.21.15.29:8765 2024-03-16T11:41:02.748 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:41:02 smithi029 ceph-mon[31975]: [16/Mar/2024:11:41:01] ENGINE Serving on https://172.21.15.29:7150 2024-03-16T11:41:02.748 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:41:02 smithi029 ceph-mon[31975]: [16/Mar/2024:11:41:01] ENGINE Bus STARTED 2024-03-16T11:41:02.748 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:41:02 smithi029 ceph-mon[31975]: from='mgr.14120 172.21.15.29:0/4098640102' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T11:41:03.378 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:02.514+0000 7f65cef41700 1 Processor -- start 2024-03-16T11:41:03.378 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:02.514+0000 7f65cef41700 1 -- start start 2024-03-16T11:41:03.378 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:02.514+0000 7f65cef41700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f65c8106610 0x7f65c8106a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:41:03.379 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:02.514+0000 7f65cef41700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f65c807be10 con 0x7f65c8106610 2024-03-16T11:41:03.379 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:02.514+0000 7f65cccdd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f65c8106610 0x7f65c8106a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:41:03.379 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:02.514+0000 7f65cccdd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f65c8106610 0x7f65c8106a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45686/0 (socket says 172.21.15.29:45686) 2024-03-16T11:41:03.379 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:02.514+0000 7f65cccdd700 1 -- 172.21.15.29:0/3174528561 learned_addr learned my addr 172.21.15.29:0/3174528561 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:41:03.379 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:02.515+0000 7f65cccdd700 1 -- 172.21.15.29:0/3174528561 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65c8106f70 con 0x7f65c8106610 2024-03-16T11:41:03.379 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:02.515+0000 7f65cccdd700 1 --2- 172.21.15.29:0/3174528561 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f65c8106610 0x7f65c8106a30 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f65b80098a0 tx=0x7f65b8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd81939295fbb112 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:41:03.379 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:02.515+0000 7f65c77fe700 1 -- 172.21.15.29:0/3174528561 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f65b8004160 con 0x7f65c8106610 2024-03-16T11:41:03.379 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:02.515+0000 7f65c77fe700 1 -- 172.21.15.29:0/3174528561 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f65b8004300 con 0x7f65c8106610 2024-03-16T11:41:03.379 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:02.515+0000 7f65c77fe700 1 -- 172.21.15.29:0/3174528561 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f65b8011b50 con 0x7f65c8106610 2024-03-16T11:41:03.379 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:02.516+0000 7f65cef41700 1 -- 172.21.15.29:0/3174528561 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f65c8106610 msgr2=0x7f65c8106a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:41:03.379 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:02.516+0000 7f65cef41700 1 --2- 172.21.15.29:0/3174528561 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f65c8106610 0x7f65c8106a30 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f65b80098a0 tx=0x7f65b8009bb0 comp rx=0 tx=0).stop 2024-03-16T11:41:03.379 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:02.516+0000 7f65cef41700 1 -- 172.21.15.29:0/3174528561 shutdown_connections 2024-03-16T11:41:03.379 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:02.516+0000 7f65cef41700 1 --2- 172.21.15.29:0/3174528561 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f65c8106610 0x7f65c8106a30 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:41:03.379 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:02.516+0000 7f65cef41700 1 -- 172.21.15.29:0/3174528561 >> 172.21.15.29:0/3174528561 conn(0x7f65c8102210 msgr2=0x7f65c8104630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:41:03.379 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:02.516+0000 7f65cef41700 1 -- 172.21.15.29:0/3174528561 shutdown_connections 2024-03-16T11:41:03.380 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:02.516+0000 7f65cef41700 1 -- 172.21.15.29:0/3174528561 wait complete. 2024-03-16T11:41:03.380 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:02.516+0000 7f65cef41700 1 Processor -- start 2024-03-16T11:41:03.380 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:02.516+0000 7f65cef41700 1 -- start start 2024-03-16T11:41:03.383 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:02.516+0000 7f65cef41700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f65c8106610 0x7f65c819df20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:41:03.383 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:02.516+0000 7f65cef41700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f65c8108000 con 0x7f65c8106610 2024-03-16T11:41:03.383 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:02.517+0000 7f65cccdd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f65c8106610 0x7f65c819df20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:41:03.383 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:02.517+0000 7f65cccdd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f65c8106610 0x7f65c819df20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45688/0 (socket says 172.21.15.29:45688) 2024-03-16T11:41:03.383 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:02.517+0000 7f65cccdd700 1 -- 172.21.15.29:0/2280745839 learned_addr learned my addr 172.21.15.29:0/2280745839 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:41:03.383 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:02.517+0000 7f65cccdd700 1 -- 172.21.15.29:0/2280745839 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65c819e460 con 0x7f65c8106610 2024-03-16T11:41:03.384 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:02.517+0000 7f65cccdd700 1 --2- 172.21.15.29:0/2280745839 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f65c8106610 0x7f65c819df20 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f65b8011b60 tx=0x7f65b8011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:41:03.384 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:02.517+0000 7f65c5ffb700 1 -- 172.21.15.29:0/2280745839 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f65b8003970 con 0x7f65c8106610 2024-03-16T11:41:03.384 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:02.517+0000 7f65c5ffb700 1 -- 172.21.15.29:0/2280745839 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f65b8003b10 con 0x7f65c8106610 2024-03-16T11:41:03.384 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:02.517+0000 7f65cef41700 1 -- 172.21.15.29:0/2280745839 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f65c819e6f0 con 0x7f65c8106610 2024-03-16T11:41:03.384 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:02.517+0000 7f65c5ffb700 1 -- 172.21.15.29:0/2280745839 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f65b8024610 con 0x7f65c8106610 2024-03-16T11:41:03.384 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:02.517+0000 7f65cef41700 1 -- 172.21.15.29:0/2280745839 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f65c81a13e0 con 0x7f65c8106610 2024-03-16T11:41:03.384 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:02.518+0000 7f65c5ffb700 1 -- 172.21.15.29:0/2280745839 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 7) ==== 46000+0+0 (secure 0 0 0) 0x7f65b8024910 con 0x7f65c8106610 2024-03-16T11:41:03.384 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:02.518+0000 7f65cef41700 1 -- 172.21.15.29:0/2280745839 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f65c8063900 con 0x7f65c8106610 2024-03-16T11:41:03.384 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:02.518+0000 7f65c5ffb700 1 --2- 172.21.15.29:0/2280745839 >> [v2:172.21.15.29:6800/2798831128,v1:172.21.15.29:6801/2798831128] conn(0x7f65b0038e00 0x7f65b003b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:41:03.384 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:02.518+0000 7f65c5ffb700 1 -- 172.21.15.29:0/2280745839 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f65b8011e00 con 0x7f65c8106610 2024-03-16T11:41:03.384 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:02.519+0000 7f65c7fff700 1 --2- 172.21.15.29:0/2280745839 >> [v2:172.21.15.29:6800/2798831128,v1:172.21.15.29:6801/2798831128] conn(0x7f65b0038e00 0x7f65b003b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:41:03.384 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:02.519+0000 7f65c7fff700 1 --2- 172.21.15.29:0/2280745839 >> [v2:172.21.15.29:6800/2798831128,v1:172.21.15.29:6801/2798831128] conn(0x7f65b0038e00 0x7f65b003b2c0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f65bc006fd0 tx=0x7f65bc006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:41:03.384 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:02.524+0000 7f65c5ffb700 1 -- 172.21.15.29:0/2280745839 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f65b8052020 con 0x7f65c8106610 2024-03-16T11:41:03.384 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:02.559+0000 7f65c5ffb700 1 -- 172.21.15.29:0/2280745839 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mgrmap(e 8) ==== 46106+0+0 (secure 0 0 0) 0x7f65b802d3f0 con 0x7f65c8106610 2024-03-16T11:41:03.384 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:02.748+0000 7f65cef41700 1 -- 172.21.15.29:0/2280745839 --> [v2:172.21.15.29:6800/2798831128,v1:172.21.15.29:6801/2798831128] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) -- 0x7f65c80008d0 con 0x7f65b0038e00 2024-03-16T11:41:03.384 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:02.754+0000 7f65c5ffb700 1 -- 172.21.15.29:0/2280745839 <== mgr.14120 v2:172.21.15.29:6800/2798831128 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f65c80008d0 con 0x7f65b0038e00 2024-03-16T11:41:03.385 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:02.756+0000 7f65cef41700 1 -- 172.21.15.29:0/2280745839 >> [v2:172.21.15.29:6800/2798831128,v1:172.21.15.29:6801/2798831128] conn(0x7f65b0038e00 msgr2=0x7f65b003b2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:41:03.385 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:02.756+0000 7f65cef41700 1 --2- 172.21.15.29:0/2280745839 >> [v2:172.21.15.29:6800/2798831128,v1:172.21.15.29:6801/2798831128] conn(0x7f65b0038e00 0x7f65b003b2c0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f65bc006fd0 tx=0x7f65bc006e40 comp rx=0 tx=0).stop 2024-03-16T11:41:03.385 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:02.756+0000 7f65cef41700 1 -- 172.21.15.29:0/2280745839 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f65c8106610 msgr2=0x7f65c819df20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:41:03.385 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:02.756+0000 7f65cef41700 1 --2- 172.21.15.29:0/2280745839 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f65c8106610 0x7f65c819df20 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f65b8011b60 tx=0x7f65b8011c40 comp rx=0 tx=0).stop 2024-03-16T11:41:03.385 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:02.756+0000 7f65cef41700 1 -- 172.21.15.29:0/2280745839 shutdown_connections 2024-03-16T11:41:03.385 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:02.756+0000 7f65cef41700 1 --2- 172.21.15.29:0/2280745839 >> [v2:172.21.15.29:6800/2798831128,v1:172.21.15.29:6801/2798831128] conn(0x7f65b0038e00 0x7f65b003b2c0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:41:03.385 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:02.756+0000 7f65cef41700 1 --2- 172.21.15.29:0/2280745839 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f65c8106610 0x7f65c819df20 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:41:03.385 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:02.756+0000 7f65cef41700 1 -- 172.21.15.29:0/2280745839 >> 172.21.15.29:0/2280745839 conn(0x7f65c8102210 msgr2=0x7f65c8194bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:41:03.385 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:02.757+0000 7f65cef41700 1 -- 172.21.15.29:0/2280745839 shutdown_connections 2024-03-16T11:41:03.385 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:02.757+0000 7f65cef41700 1 -- 172.21.15.29:0/2280745839 wait complete. 2024-03-16T11:41:03.703 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:41:03 smithi029 ceph-mon[31975]: mgrmap e8: x(active, since 2s) 2024-03-16T11:41:03.703 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:41:03 smithi029 ceph-mon[31975]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T11:41:03.703 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:41:03 smithi029 ceph-mon[31975]: from='mgr.14120 172.21.15.29:0/4098640102' entity='mgr.x' 2024-03-16T11:41:03.703 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:41:03 smithi029 ceph-mon[31975]: from='mgr.14120 172.21.15.29:0/4098640102' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T11:41:04.588 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:41:04 smithi029 ceph-mon[31975]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T11:41:04.619 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout value unchanged 2024-03-16T11:41:04.619 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:03.808+0000 7fef8e2bb700 1 Processor -- start 2024-03-16T11:41:04.619 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:03.808+0000 7fef8e2bb700 1 -- start start 2024-03-16T11:41:04.619 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:03.808+0000 7fef8e2bb700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fef88107d30 0x7fef88108150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:41:04.620 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:03.808+0000 7fef8e2bb700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fef88108720 con 0x7fef88107d30 2024-03-16T11:41:04.620 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:03.808+0000 7fef87fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fef88107d30 0x7fef88108150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:41:04.620 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:03.808+0000 7fef87fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fef88107d30 0x7fef88108150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45704/0 (socket says 172.21.15.29:45704) 2024-03-16T11:41:04.620 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:03.808+0000 7fef87fff700 1 -- 172.21.15.29:0/103445702 learned_addr learned my addr 172.21.15.29:0/103445702 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:41:04.620 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:03.809+0000 7fef87fff700 1 -- 172.21.15.29:0/103445702 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef88108f30 con 0x7fef88107d30 2024-03-16T11:41:04.620 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:03.809+0000 7fef87fff700 1 --2- 172.21.15.29:0/103445702 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fef88107d30 0x7fef88108150 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fef780098a0 tx=0x7fef78009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a6c7bec46c64508 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:41:04.620 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:03.809+0000 7fef86ffd700 1 -- 172.21.15.29:0/103445702 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fef78004160 con 0x7fef88107d30 2024-03-16T11:41:04.620 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:03.809+0000 7fef86ffd700 1 -- 172.21.15.29:0/103445702 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fef78004300 con 0x7fef88107d30 2024-03-16T11:41:04.620 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:03.809+0000 7fef86ffd700 1 -- 172.21.15.29:0/103445702 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fef78011b50 con 0x7fef88107d30 2024-03-16T11:41:04.620 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:03.810+0000 7fef8e2bb700 1 -- 172.21.15.29:0/103445702 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fef88107d30 msgr2=0x7fef88108150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:41:04.620 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:03.810+0000 7fef8e2bb700 1 --2- 172.21.15.29:0/103445702 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fef88107d30 0x7fef88108150 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fef780098a0 tx=0x7fef78009bb0 comp rx=0 tx=0).stop 2024-03-16T11:41:04.620 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:03.810+0000 7fef8e2bb700 1 -- 172.21.15.29:0/103445702 shutdown_connections 2024-03-16T11:41:04.620 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:03.810+0000 7fef8e2bb700 1 --2- 172.21.15.29:0/103445702 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fef88107d30 0x7fef88108150 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:41:04.620 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:03.810+0000 7fef8e2bb700 1 -- 172.21.15.29:0/103445702 >> 172.21.15.29:0/103445702 conn(0x7fef881032f0 msgr2=0x7fef88105710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:41:04.620 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:03.810+0000 7fef8e2bb700 1 -- 172.21.15.29:0/103445702 shutdown_connections 2024-03-16T11:41:04.621 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:03.810+0000 7fef8e2bb700 1 -- 172.21.15.29:0/103445702 wait complete. 2024-03-16T11:41:04.621 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:03.810+0000 7fef8e2bb700 1 Processor -- start 2024-03-16T11:41:04.621 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:03.810+0000 7fef8e2bb700 1 -- start start 2024-03-16T11:41:04.621 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:03.811+0000 7fef8e2bb700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fef88107d30 0x7fef8819bd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:41:04.621 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:03.811+0000 7fef8e2bb700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fef8810c2e0 con 0x7fef88107d30 2024-03-16T11:41:04.621 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:03.811+0000 7fef87fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fef88107d30 0x7fef8819bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:41:04.621 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:03.811+0000 7fef87fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fef88107d30 0x7fef8819bd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45710/0 (socket says 172.21.15.29:45710) 2024-03-16T11:41:04.621 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:03.811+0000 7fef87fff700 1 -- 172.21.15.29:0/2505108835 learned_addr learned my addr 172.21.15.29:0/2505108835 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:41:04.621 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:03.811+0000 7fef87fff700 1 -- 172.21.15.29:0/2505108835 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef8819c290 con 0x7fef88107d30 2024-03-16T11:41:04.621 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:03.811+0000 7fef87fff700 1 --2- 172.21.15.29:0/2505108835 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fef88107d30 0x7fef8819bd50 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fef78011850 tx=0x7fef78011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:41:04.621 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:03.811+0000 7fef857fa700 1 -- 172.21.15.29:0/2505108835 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fef78003af0 con 0x7fef88107d30 2024-03-16T11:41:04.621 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:03.811+0000 7fef857fa700 1 -- 172.21.15.29:0/2505108835 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fef78020070 con 0x7fef88107d30 2024-03-16T11:41:04.621 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:03.811+0000 7fef8e2bb700 1 -- 172.21.15.29:0/2505108835 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fef8819c520 con 0x7fef88107d30 2024-03-16T11:41:04.621 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:03.811+0000 7fef857fa700 1 -- 172.21.15.29:0/2505108835 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fef780195a0 con 0x7fef88107d30 2024-03-16T11:41:04.621 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:03.811+0000 7fef8e2bb700 1 -- 172.21.15.29:0/2505108835 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fef8819f210 con 0x7fef88107d30 2024-03-16T11:41:04.621 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:03.812+0000 7fef857fa700 1 -- 172.21.15.29:0/2505108835 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 8) ==== 46106+0+0 (secure 0 0 0) 0x7fef78019850 con 0x7fef88107d30 2024-03-16T11:41:04.622 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:03.812+0000 7fef8e2bb700 1 -- 172.21.15.29:0/2505108835 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fef88063900 con 0x7fef88107d30 2024-03-16T11:41:04.622 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:03.812+0000 7fef857fa700 1 --2- 172.21.15.29:0/2505108835 >> [v2:172.21.15.29:6800/2798831128,v1:172.21.15.29:6801/2798831128] conn(0x7fef70038f70 0x7fef7003b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:41:04.622 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:03.813+0000 7fef857fa700 1 -- 172.21.15.29:0/2505108835 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fef78011e00 con 0x7fef88107d30 2024-03-16T11:41:04.622 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:03.813+0000 7fef877fe700 1 --2- 172.21.15.29:0/2505108835 >> [v2:172.21.15.29:6800/2798831128,v1:172.21.15.29:6801/2798831128] conn(0x7fef70038f70 0x7fef7003b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:41:04.622 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:03.813+0000 7fef877fe700 1 --2- 172.21.15.29:0/2505108835 >> [v2:172.21.15.29:6800/2798831128,v1:172.21.15.29:6801/2798831128] conn(0x7fef70038f70 0x7fef7003b430 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fef7c006fd0 tx=0x7fef7c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:41:04.622 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:03.818+0000 7fef857fa700 1 -- 172.21.15.29:0/2505108835 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fef78003c90 con 0x7fef88107d30 2024-03-16T11:41:04.622 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:04.023+0000 7fef8e2bb700 1 -- 172.21.15.29:0/2505108835 --> [v2:172.21.15.29:6800/2798831128,v1:172.21.15.29:6801/2798831128] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) -- 0x7fef881045a0 con 0x7fef70038f70 2024-03-16T11:41:04.622 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:04.024+0000 7fef857fa700 1 -- 172.21.15.29:0/2505108835 <== mgr.14120 v2:172.21.15.29:6800/2798831128 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+16 (secure 0 0 0) 0x7fef881045a0 con 0x7fef70038f70 2024-03-16T11:41:04.622 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:04.026+0000 7fef8e2bb700 1 -- 172.21.15.29:0/2505108835 >> [v2:172.21.15.29:6800/2798831128,v1:172.21.15.29:6801/2798831128] conn(0x7fef70038f70 msgr2=0x7fef7003b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:41:04.622 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:04.026+0000 7fef8e2bb700 1 --2- 172.21.15.29:0/2505108835 >> [v2:172.21.15.29:6800/2798831128,v1:172.21.15.29:6801/2798831128] conn(0x7fef70038f70 0x7fef7003b430 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fef7c006fd0 tx=0x7fef7c006e40 comp rx=0 tx=0).stop 2024-03-16T11:41:04.622 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:04.026+0000 7fef8e2bb700 1 -- 172.21.15.29:0/2505108835 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fef88107d30 msgr2=0x7fef8819bd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:41:04.622 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:04.026+0000 7fef8e2bb700 1 --2- 172.21.15.29:0/2505108835 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fef88107d30 0x7fef8819bd50 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fef78011850 tx=0x7fef78011c10 comp rx=0 tx=0).stop 2024-03-16T11:41:04.622 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:04.026+0000 7fef8e2bb700 1 -- 172.21.15.29:0/2505108835 shutdown_connections 2024-03-16T11:41:04.622 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:04.026+0000 7fef8e2bb700 1 --2- 172.21.15.29:0/2505108835 >> [v2:172.21.15.29:6800/2798831128,v1:172.21.15.29:6801/2798831128] conn(0x7fef70038f70 0x7fef7003b430 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:41:04.623 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:04.026+0000 7fef8e2bb700 1 --2- 172.21.15.29:0/2505108835 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fef88107d30 0x7fef8819bd50 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:41:04.623 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:04.026+0000 7fef8e2bb700 1 -- 172.21.15.29:0/2505108835 >> 172.21.15.29:0/2505108835 conn(0x7fef881032f0 msgr2=0x7fef88103e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:41:04.623 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:04.026+0000 7fef8e2bb700 1 -- 172.21.15.29:0/2505108835 shutdown_connections 2024-03-16T11:41:04.623 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:04.026+0000 7fef8e2bb700 1 -- 172.21.15.29:0/2505108835 wait complete. 2024-03-16T11:41:04.623 INFO:teuthology.orchestra.run.smithi029.stdout:Generating ssh key... 2024-03-16T11:41:05.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:41:05 smithi029 ceph-mon[31975]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T11:41:05.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:41:05 smithi029 ceph-mon[31975]: Generating ssh key... 2024-03-16T11:41:05.839 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 16 11:41:05 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-x[32210]: Generating public/private rsa key pair. 2024-03-16T11:41:05.839 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 16 11:41:05 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-x[32210]: Your identification has been saved in /tmp/tmpxbzv74f6/key. 2024-03-16T11:41:05.839 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 16 11:41:05 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-x[32210]: Your public key has been saved in /tmp/tmpxbzv74f6/key.pub. 2024-03-16T11:41:05.839 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 16 11:41:05 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-x[32210]: The key fingerprint is: 2024-03-16T11:41:05.839 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 16 11:41:05 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-x[32210]: SHA256:07m6GmTcMu39DQvi+23iFZHyXXuZzrA0oveTmiy1Ju0 ceph-d1887f28-e389-11ee-95c9-87774f69a715 2024-03-16T11:41:05.839 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 16 11:41:05 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-x[32210]: The key's randomart image is: 2024-03-16T11:41:05.839 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 16 11:41:05 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-x[32210]: +---[RSA 3072]----+ 2024-03-16T11:41:05.839 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 16 11:41:05 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-x[32210]: | | 2024-03-16T11:41:05.839 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 16 11:41:05 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-x[32210]: | . | 2024-03-16T11:41:05.839 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 16 11:41:05 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-x[32210]: | . o .| 2024-03-16T11:41:05.839 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 16 11:41:05 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-x[32210]: | . o . + o .+| 2024-03-16T11:41:05.839 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 16 11:41:05 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-x[32210]: | * S o.o+.+.| 2024-03-16T11:41:05.839 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 16 11:41:05 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-x[32210]: | o + o.oo.* .| 2024-03-16T11:41:05.839 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 16 11:41:05 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-x[32210]: | . o.*.+..o | 2024-03-16T11:41:05.839 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 16 11:41:05 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-x[32210]: | o =+O+* | 2024-03-16T11:41:05.839 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 16 11:41:05 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-x[32210]: | ..==BE=.o | 2024-03-16T11:41:05.840 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 16 11:41:05 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-x[32210]: +----[SHA256]-----+ 2024-03-16T11:41:06.169 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:05.050+0000 7fadacbf7700 1 Processor -- start 2024-03-16T11:41:06.169 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:05.050+0000 7fadacbf7700 1 -- start start 2024-03-16T11:41:06.169 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:05.050+0000 7fadacbf7700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fada81059f0 0x7fada8105e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:41:06.169 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:05.050+0000 7fadacbf7700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fada8106350 con 0x7fada81059f0 2024-03-16T11:41:06.169 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:05.050+0000 7fada659c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fada81059f0 0x7fada8105e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:41:06.169 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:05.050+0000 7fada659c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fada81059f0 0x7fada8105e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45720/0 (socket says 172.21.15.29:45720) 2024-03-16T11:41:06.169 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:05.050+0000 7fada659c700 1 -- 172.21.15.29:0/3765061896 learned_addr learned my addr 172.21.15.29:0/3765061896 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:41:06.169 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:05.051+0000 7fada659c700 1 -- 172.21.15.29:0/3765061896 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fada81064d0 con 0x7fada81059f0 2024-03-16T11:41:06.169 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:05.051+0000 7fada659c700 1 --2- 172.21.15.29:0/3765061896 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fada81059f0 0x7fada8105e10 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fad980098a0 tx=0x7fad98009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58bec674c6b49e73 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:41:06.169 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:05.051+0000 7fada559a700 1 -- 172.21.15.29:0/3765061896 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fad98004160 con 0x7fada81059f0 2024-03-16T11:41:06.169 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:05.051+0000 7fada559a700 1 -- 172.21.15.29:0/3765061896 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fad98004300 con 0x7fada81059f0 2024-03-16T11:41:06.169 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:05.051+0000 7fada559a700 1 -- 172.21.15.29:0/3765061896 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fad98011b50 con 0x7fada81059f0 2024-03-16T11:41:06.170 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:05.051+0000 7fadacbf7700 1 -- 172.21.15.29:0/3765061896 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fada81059f0 msgr2=0x7fada8105e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:41:06.170 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:05.052+0000 7fadacbf7700 1 --2- 172.21.15.29:0/3765061896 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fada81059f0 0x7fada8105e10 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fad980098a0 tx=0x7fad98009bb0 comp rx=0 tx=0).stop 2024-03-16T11:41:06.170 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:05.052+0000 7fadacbf7700 1 -- 172.21.15.29:0/3765061896 shutdown_connections 2024-03-16T11:41:06.170 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:05.052+0000 7fadacbf7700 1 --2- 172.21.15.29:0/3765061896 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fada81059f0 0x7fada8105e10 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:41:06.170 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:05.052+0000 7fadacbf7700 1 -- 172.21.15.29:0/3765061896 >> 172.21.15.29:0/3765061896 conn(0x7fada8101470 msgr2=0x7fada81038d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:41:06.170 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:05.052+0000 7fadacbf7700 1 -- 172.21.15.29:0/3765061896 shutdown_connections 2024-03-16T11:41:06.170 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:05.052+0000 7fadacbf7700 1 -- 172.21.15.29:0/3765061896 wait complete. 2024-03-16T11:41:06.170 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:05.052+0000 7fadacbf7700 1 Processor -- start 2024-03-16T11:41:06.170 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:05.053+0000 7fadacbf7700 1 -- start start 2024-03-16T11:41:06.170 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:05.053+0000 7fadacbf7700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fada81059f0 0x7fada819bcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:41:06.170 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:05.053+0000 7fadacbf7700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fada810c080 con 0x7fada81059f0 2024-03-16T11:41:06.170 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:05.053+0000 7fada659c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fada81059f0 0x7fada819bcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:41:06.170 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:05.053+0000 7fada659c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fada81059f0 0x7fada819bcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45728/0 (socket says 172.21.15.29:45728) 2024-03-16T11:41:06.170 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:05.053+0000 7fada659c700 1 -- 172.21.15.29:0/3746299655 learned_addr learned my addr 172.21.15.29:0/3746299655 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:41:06.171 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:05.053+0000 7fada659c700 1 -- 172.21.15.29:0/3746299655 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fada819c1f0 con 0x7fada81059f0 2024-03-16T11:41:06.171 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:05.053+0000 7fada659c700 1 --2- 172.21.15.29:0/3746299655 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fada81059f0 0x7fada819bcb0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fad98009fb0 tx=0x7fad98011bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:41:06.171 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:05.053+0000 7fad977fe700 1 -- 172.21.15.29:0/3746299655 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fad980038b0 con 0x7fada81059f0 2024-03-16T11:41:06.171 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:05.053+0000 7fad977fe700 1 -- 172.21.15.29:0/3746299655 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fad98011e50 con 0x7fada81059f0 2024-03-16T11:41:06.171 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:05.053+0000 7fadacbf7700 1 -- 172.21.15.29:0/3746299655 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fada819c480 con 0x7fada81059f0 2024-03-16T11:41:06.171 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:05.054+0000 7fad977fe700 1 -- 172.21.15.29:0/3746299655 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fad98019420 con 0x7fada81059f0 2024-03-16T11:41:06.171 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:05.054+0000 7fadacbf7700 1 -- 172.21.15.29:0/3746299655 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fada819f170 con 0x7fada81059f0 2024-03-16T11:41:06.171 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:05.054+0000 7fad977fe700 1 -- 172.21.15.29:0/3746299655 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 8) ==== 46106+0+0 (secure 0 0 0) 0x7fad98019720 con 0x7fada81059f0 2024-03-16T11:41:06.171 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:05.055+0000 7fad977fe700 1 --2- 172.21.15.29:0/3746299655 >> [v2:172.21.15.29:6800/2798831128,v1:172.21.15.29:6801/2798831128] conn(0x7fad90038ed0 0x7fad9003b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:41:06.171 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:05.055+0000 7fadacbf7700 1 -- 172.21.15.29:0/3746299655 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fada8063900 con 0x7fada81059f0 2024-03-16T11:41:06.171 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:05.055+0000 7fad977fe700 1 -- 172.21.15.29:0/3746299655 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fad9804d2b0 con 0x7fada81059f0 2024-03-16T11:41:06.171 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:05.055+0000 7fada5d9b700 1 --2- 172.21.15.29:0/3746299655 >> [v2:172.21.15.29:6800/2798831128,v1:172.21.15.29:6801/2798831128] conn(0x7fad90038ed0 0x7fad9003b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:41:06.171 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:05.055+0000 7fada5d9b700 1 --2- 172.21.15.29:0/3746299655 >> [v2:172.21.15.29:6800/2798831128,v1:172.21.15.29:6801/2798831128] conn(0x7fad90038ed0 0x7fad9003b390 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fad9c006fd0 tx=0x7fad9c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:41:06.171 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:05.060+0000 7fad977fe700 1 -- 172.21.15.29:0/3746299655 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fad9800bc70 con 0x7fada81059f0 2024-03-16T11:41:06.171 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:05.265+0000 7fadacbf7700 1 -- 172.21.15.29:0/3746299655 --> [v2:172.21.15.29:6800/2798831128,v1:172.21.15.29:6801/2798831128] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) -- 0x7fada8102730 con 0x7fad90038ed0 2024-03-16T11:41:06.172 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:05.634+0000 7fad977fe700 1 -- 172.21.15.29:0/3746299655 <== mgr.14120 v2:172.21.15.29:6800/2798831128 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fada8102730 con 0x7fad90038ed0 2024-03-16T11:41:06.172 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:05.637+0000 7fadacbf7700 1 -- 172.21.15.29:0/3746299655 >> [v2:172.21.15.29:6800/2798831128,v1:172.21.15.29:6801/2798831128] conn(0x7fad90038ed0 msgr2=0x7fad9003b390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:41:06.172 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:05.637+0000 7fadacbf7700 1 --2- 172.21.15.29:0/3746299655 >> [v2:172.21.15.29:6800/2798831128,v1:172.21.15.29:6801/2798831128] conn(0x7fad90038ed0 0x7fad9003b390 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fad9c006fd0 tx=0x7fad9c006e40 comp rx=0 tx=0).stop 2024-03-16T11:41:06.172 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:05.637+0000 7fadacbf7700 1 -- 172.21.15.29:0/3746299655 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fada81059f0 msgr2=0x7fada819bcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:41:06.172 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:05.637+0000 7fadacbf7700 1 --2- 172.21.15.29:0/3746299655 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fada81059f0 0x7fada819bcb0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fad98009fb0 tx=0x7fad98011bf0 comp rx=0 tx=0).stop 2024-03-16T11:41:06.172 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:05.637+0000 7fadacbf7700 1 -- 172.21.15.29:0/3746299655 shutdown_connections 2024-03-16T11:41:06.172 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:05.637+0000 7fadacbf7700 1 --2- 172.21.15.29:0/3746299655 >> [v2:172.21.15.29:6800/2798831128,v1:172.21.15.29:6801/2798831128] conn(0x7fad90038ed0 0x7fad9003b390 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:41:06.172 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:05.637+0000 7fadacbf7700 1 --2- 172.21.15.29:0/3746299655 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fada81059f0 0x7fada819bcb0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:41:06.172 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:05.637+0000 7fadacbf7700 1 -- 172.21.15.29:0/3746299655 >> 172.21.15.29:0/3746299655 conn(0x7fada8101470 msgr2=0x7fada8102020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:41:06.172 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:05.637+0000 7fadacbf7700 1 -- 172.21.15.29:0/3746299655 shutdown_connections 2024-03-16T11:41:06.172 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:05.637+0000 7fadacbf7700 1 -- 172.21.15.29:0/3746299655 wait complete. 2024-03-16T11:41:06.811 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:41:06 smithi029 ceph-mon[31975]: from='mgr.14120 172.21.15.29:0/4098640102' entity='mgr.x' 2024-03-16T11:41:06.811 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:41:06 smithi029 ceph-mon[31975]: from='mgr.14120 172.21.15.29:0/4098640102' entity='mgr.x' 2024-03-16T11:41:07.383 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC8KkuD7XYsg7qg5vvm6f9C9F+bVy31FMlXBVVQr5jgiqFnoJJ2WHe/FyeDly3kAIcxbqx/iXf/RyV7hp8dJKclLomINMbzqk9ZVgPp5Y026gBfLWwF9v5PR8XEp69/TpfLMXEHTfnPIhrkq8TLxGdCSf9KZnNkxbAAydQq1I3bdKkNHmLBSduPJJXigbrrxZKltcOiUv1HHWJ+izQLpohLvOnkkoVcIKjZhJg/W6MvM36jmQ8QSumUvV48B6/RpXrRbRmJZk/pJXuKTI6iUtSgjRVZU3e2lv3z62fkqRfnax/PlA38ddonynlcCkwcFmFZpo9kgH+0d7HqJmt1bQ+dfih97Hj4Cepr7q6of9GgCR5m+6bqM6E5XYAiskWUVcQiMuUdtWIGWLwZLSTKMPn/ShpirlzVKZj0+UErz4t+pH3x0SrEv5yFbvV06xs067rn0AORK42XwURRlffKbMhQvcLKYfNWpyRC2Uhx6AH5JC7UAFo4S8XLr4jNfxmjnbk= ceph-d1887f28-e389-11ee-95c9-87774f69a715 2024-03-16T11:41:07.383 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:06.582+0000 7f9f19f35700 1 Processor -- start 2024-03-16T11:41:07.383 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:06.582+0000 7f9f19f35700 1 -- start start 2024-03-16T11:41:07.383 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:06.583+0000 7f9f19f35700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9f14106620 0x7f9f14106a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:41:07.383 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:06.583+0000 7f9f19f35700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f9f1407bdd0 con 0x7f9f14106620 2024-03-16T11:41:07.383 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:06.583+0000 7f9f137fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9f14106620 0x7f9f14106a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:41:07.383 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:06.583+0000 7f9f137fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9f14106620 0x7f9f14106a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45744/0 (socket says 172.21.15.29:45744) 2024-03-16T11:41:07.383 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:06.583+0000 7f9f137fe700 1 -- 172.21.15.29:0/3008460905 learned_addr learned my addr 172.21.15.29:0/3008460905 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:41:07.383 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:06.583+0000 7f9f137fe700 1 -- 172.21.15.29:0/3008460905 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f14106f80 con 0x7f9f14106620 2024-03-16T11:41:07.383 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:06.583+0000 7f9f137fe700 1 --2- 172.21.15.29:0/3008460905 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9f14106620 0x7f9f14106a40 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f9f040098a0 tx=0x7f9f04009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26edc9b8eef6a5b5 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:41:07.383 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:06.583+0000 7f9f127fc700 1 -- 172.21.15.29:0/3008460905 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9f04004160 con 0x7f9f14106620 2024-03-16T11:41:07.383 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:06.583+0000 7f9f127fc700 1 -- 172.21.15.29:0/3008460905 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f9f04004300 con 0x7f9f14106620 2024-03-16T11:41:07.384 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:06.583+0000 7f9f127fc700 1 -- 172.21.15.29:0/3008460905 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9f04011b50 con 0x7f9f14106620 2024-03-16T11:41:07.384 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:06.584+0000 7f9f19f35700 1 -- 172.21.15.29:0/3008460905 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9f14106620 msgr2=0x7f9f14106a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:41:07.384 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:06.584+0000 7f9f19f35700 1 --2- 172.21.15.29:0/3008460905 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9f14106620 0x7f9f14106a40 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f9f040098a0 tx=0x7f9f04009bb0 comp rx=0 tx=0).stop 2024-03-16T11:41:07.384 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:06.584+0000 7f9f19f35700 1 -- 172.21.15.29:0/3008460905 shutdown_connections 2024-03-16T11:41:07.384 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:06.584+0000 7f9f19f35700 1 --2- 172.21.15.29:0/3008460905 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9f14106620 0x7f9f14106a40 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:41:07.384 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:06.584+0000 7f9f19f35700 1 -- 172.21.15.29:0/3008460905 >> 172.21.15.29:0/3008460905 conn(0x7f9f141021c0 msgr2=0x7f9f14104600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:41:07.384 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:06.584+0000 7f9f19f35700 1 -- 172.21.15.29:0/3008460905 shutdown_connections 2024-03-16T11:41:07.384 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:06.584+0000 7f9f19f35700 1 -- 172.21.15.29:0/3008460905 wait complete. 2024-03-16T11:41:07.384 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:06.585+0000 7f9f19f35700 1 Processor -- start 2024-03-16T11:41:07.384 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:06.585+0000 7f9f19f35700 1 -- start start 2024-03-16T11:41:07.384 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:06.585+0000 7f9f19f35700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9f14106620 0x7f9f1419df00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:41:07.384 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:06.585+0000 7f9f19f35700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f9f14108010 con 0x7f9f14106620 2024-03-16T11:41:07.384 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:06.585+0000 7f9f137fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9f14106620 0x7f9f1419df00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:41:07.384 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:06.585+0000 7f9f137fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9f14106620 0x7f9f1419df00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45746/0 (socket says 172.21.15.29:45746) 2024-03-16T11:41:07.384 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:06.585+0000 7f9f137fe700 1 -- 172.21.15.29:0/786886080 learned_addr learned my addr 172.21.15.29:0/786886080 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:41:07.385 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:06.585+0000 7f9f137fe700 1 -- 172.21.15.29:0/786886080 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f1419e440 con 0x7f9f14106620 2024-03-16T11:41:07.385 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:06.586+0000 7f9f137fe700 1 --2- 172.21.15.29:0/786886080 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9f14106620 0x7f9f1419df00 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f9f04009fb0 tx=0x7f9f04011bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:41:07.385 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:06.586+0000 7f9f10ff9700 1 -- 172.21.15.29:0/786886080 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9f040038b0 con 0x7f9f14106620 2024-03-16T11:41:07.385 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:06.586+0000 7f9f10ff9700 1 -- 172.21.15.29:0/786886080 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f9f04011e50 con 0x7f9f14106620 2024-03-16T11:41:07.385 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:06.586+0000 7f9f19f35700 1 -- 172.21.15.29:0/786886080 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9f1419e6d0 con 0x7f9f14106620 2024-03-16T11:41:07.385 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:06.586+0000 7f9f10ff9700 1 -- 172.21.15.29:0/786886080 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9f04019420 con 0x7f9f14106620 2024-03-16T11:41:07.385 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:06.586+0000 7f9f19f35700 1 -- 172.21.15.29:0/786886080 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9f141a13c0 con 0x7f9f14106620 2024-03-16T11:41:07.385 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:06.587+0000 7f9f10ff9700 1 -- 172.21.15.29:0/786886080 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 8) ==== 46106+0+0 (secure 0 0 0) 0x7f9f040196f0 con 0x7f9f14106620 2024-03-16T11:41:07.385 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:06.587+0000 7f9f19f35700 1 -- 172.21.15.29:0/786886080 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9f14063900 con 0x7f9f14106620 2024-03-16T11:41:07.385 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:06.587+0000 7f9f10ff9700 1 --2- 172.21.15.29:0/786886080 >> [v2:172.21.15.29:6800/2798831128,v1:172.21.15.29:6801/2798831128] conn(0x7f9efc038f20 0x7f9efc03b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:41:07.385 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:06.587+0000 7f9f10ff9700 1 -- 172.21.15.29:0/786886080 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f9f0404d250 con 0x7f9f14106620 2024-03-16T11:41:07.385 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:06.587+0000 7f9f12ffd700 1 --2- 172.21.15.29:0/786886080 >> [v2:172.21.15.29:6800/2798831128,v1:172.21.15.29:6801/2798831128] conn(0x7f9efc038f20 0x7f9efc03b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:41:07.385 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:06.587+0000 7f9f12ffd700 1 --2- 172.21.15.29:0/786886080 >> [v2:172.21.15.29:6800/2798831128,v1:172.21.15.29:6801/2798831128] conn(0x7f9efc038f20 0x7f9efc03b3e0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f9f08006fd0 tx=0x7f9f08006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:41:07.385 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:06.592+0000 7f9f10ff9700 1 -- 172.21.15.29:0/786886080 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f9f0400ba30 con 0x7f9f14106620 2024-03-16T11:41:07.385 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:06.798+0000 7f9f19f35700 1 -- 172.21.15.29:0/786886080 --> [v2:172.21.15.29:6800/2798831128,v1:172.21.15.29:6801/2798831128] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) -- 0x7f9f140008d0 con 0x7f9efc038f20 2024-03-16T11:41:07.385 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:06.799+0000 7f9f10ff9700 1 -- 172.21.15.29:0/786886080 <== mgr.14120 v2:172.21.15.29:6800/2798831128 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+595 (secure 0 0 0) 0x7f9f140008d0 con 0x7f9efc038f20 2024-03-16T11:41:07.386 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:06.801+0000 7f9f19f35700 1 -- 172.21.15.29:0/786886080 >> [v2:172.21.15.29:6800/2798831128,v1:172.21.15.29:6801/2798831128] conn(0x7f9efc038f20 msgr2=0x7f9efc03b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:41:07.386 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:06.801+0000 7f9f19f35700 1 --2- 172.21.15.29:0/786886080 >> [v2:172.21.15.29:6800/2798831128,v1:172.21.15.29:6801/2798831128] conn(0x7f9efc038f20 0x7f9efc03b3e0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f9f08006fd0 tx=0x7f9f08006e40 comp rx=0 tx=0).stop 2024-03-16T11:41:07.386 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:06.801+0000 7f9f19f35700 1 -- 172.21.15.29:0/786886080 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9f14106620 msgr2=0x7f9f1419df00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:41:07.386 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:06.801+0000 7f9f19f35700 1 --2- 172.21.15.29:0/786886080 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9f14106620 0x7f9f1419df00 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f9f04009fb0 tx=0x7f9f04011bf0 comp rx=0 tx=0).stop 2024-03-16T11:41:07.386 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:06.801+0000 7f9f19f35700 1 -- 172.21.15.29:0/786886080 shutdown_connections 2024-03-16T11:41:07.386 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:06.801+0000 7f9f19f35700 1 --2- 172.21.15.29:0/786886080 >> [v2:172.21.15.29:6800/2798831128,v1:172.21.15.29:6801/2798831128] conn(0x7f9efc038f20 0x7f9efc03b3e0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:41:07.386 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:06.801+0000 7f9f19f35700 1 --2- 172.21.15.29:0/786886080 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9f14106620 0x7f9f1419df00 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:41:07.386 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:06.801+0000 7f9f19f35700 1 -- 172.21.15.29:0/786886080 >> 172.21.15.29:0/786886080 conn(0x7f9f141021c0 msgr2=0x7f9f14194b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:41:07.386 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:06.801+0000 7f9f19f35700 1 -- 172.21.15.29:0/786886080 shutdown_connections 2024-03-16T11:41:07.386 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:06.801+0000 7f9f19f35700 1 -- 172.21.15.29:0/786886080 wait complete. 2024-03-16T11:41:07.386 INFO:teuthology.orchestra.run.smithi029.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-03-16T11:41:07.386 INFO:teuthology.orchestra.run.smithi029.stdout:Adding key to root@localhost authorized_keys... 2024-03-16T11:41:07.386 INFO:teuthology.orchestra.run.smithi029.stdout:Adding host smithi029... 2024-03-16T11:41:07.753 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:41:07 smithi029 ceph-mon[31975]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T11:41:08.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:41:08 smithi029 ceph-mon[31975]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi029", "addr": "172.21.15.29", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T11:41:10.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:41:09 smithi029 ceph-mon[31975]: Deploying cephadm binary to smithi029 2024-03-16T11:41:13.019 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout Added host 'smithi029' with addr '172.21.15.29' 2024-03-16T11:41:13.019 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:07.828+0000 7ff91806a700 1 Processor -- start 2024-03-16T11:41:13.019 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:07.828+0000 7ff91806a700 1 -- start start 2024-03-16T11:41:13.019 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:07.828+0000 7ff91806a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff910107d00 0x7ff910108120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:41:13.019 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:07.828+0000 7ff91806a700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7ff9101086f0 con 0x7ff910107d00 2024-03-16T11:41:13.019 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:07.829+0000 7ff915e06700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff910107d00 0x7ff910108120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:41:13.019 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:07.829+0000 7ff915e06700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff910107d00 0x7ff910108120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45750/0 (socket says 172.21.15.29:45750) 2024-03-16T11:41:13.019 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:07.829+0000 7ff915e06700 1 -- 172.21.15.29:0/3714409919 learned_addr learned my addr 172.21.15.29:0/3714409919 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:41:13.019 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:07.829+0000 7ff915e06700 1 -- 172.21.15.29:0/3714409919 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff910108f00 con 0x7ff910107d00 2024-03-16T11:41:13.020 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:07.829+0000 7ff915e06700 1 --2- 172.21.15.29:0/3714409919 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff910107d00 0x7ff910108120 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7ff900009e80 tx=0x7ff90000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a381b762a4591ee1 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:41:13.020 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:07.829+0000 7ff914e04700 1 -- 172.21.15.29:0/3714409919 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff900017070 con 0x7ff910107d00 2024-03-16T11:41:13.020 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:07.829+0000 7ff914e04700 1 -- 172.21.15.29:0/3714409919 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7ff90000b8c0 con 0x7ff910107d00 2024-03-16T11:41:13.020 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:07.829+0000 7ff914e04700 1 -- 172.21.15.29:0/3714409919 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff9000045c0 con 0x7ff910107d00 2024-03-16T11:41:13.020 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:07.830+0000 7ff91806a700 1 -- 172.21.15.29:0/3714409919 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff910107d00 msgr2=0x7ff910108120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:41:13.020 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:07.830+0000 7ff91806a700 1 --2- 172.21.15.29:0/3714409919 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff910107d00 0x7ff910108120 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7ff900009e80 tx=0x7ff90000b330 comp rx=0 tx=0).stop 2024-03-16T11:41:13.020 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:07.830+0000 7ff91806a700 1 -- 172.21.15.29:0/3714409919 shutdown_connections 2024-03-16T11:41:13.020 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:07.830+0000 7ff91806a700 1 --2- 172.21.15.29:0/3714409919 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff910107d00 0x7ff910108120 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:41:13.020 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:07.830+0000 7ff91806a700 1 -- 172.21.15.29:0/3714409919 >> 172.21.15.29:0/3714409919 conn(0x7ff9101032a0 msgr2=0x7ff9101056e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:41:13.020 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:07.830+0000 7ff91806a700 1 -- 172.21.15.29:0/3714409919 shutdown_connections 2024-03-16T11:41:13.020 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:07.830+0000 7ff91806a700 1 -- 172.21.15.29:0/3714409919 wait complete. 2024-03-16T11:41:13.020 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:07.831+0000 7ff91806a700 1 Processor -- start 2024-03-16T11:41:13.020 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:07.831+0000 7ff91806a700 1 -- start start 2024-03-16T11:41:13.021 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:07.831+0000 7ff91806a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff910107d00 0x7ff91019bcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:41:13.021 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:07.831+0000 7ff91806a700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7ff91010c2b0 con 0x7ff910107d00 2024-03-16T11:41:13.021 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:07.831+0000 7ff915e06700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff910107d00 0x7ff91019bcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:41:13.021 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:07.831+0000 7ff915e06700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff910107d00 0x7ff91019bcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45764/0 (socket says 172.21.15.29:45764) 2024-03-16T11:41:13.021 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:07.831+0000 7ff915e06700 1 -- 172.21.15.29:0/641590233 learned_addr learned my addr 172.21.15.29:0/641590233 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:41:13.021 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:07.831+0000 7ff915e06700 1 -- 172.21.15.29:0/641590233 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff91019c210 con 0x7ff910107d00 2024-03-16T11:41:13.021 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:07.831+0000 7ff915e06700 1 --2- 172.21.15.29:0/641590233 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff910107d00 0x7ff91019bcd0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7ff900004210 tx=0x7ff90000bd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:41:13.021 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:07.832+0000 7ff90effd700 1 -- 172.21.15.29:0/641590233 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff900020020 con 0x7ff910107d00 2024-03-16T11:41:13.021 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:07.832+0000 7ff90effd700 1 -- 172.21.15.29:0/641590233 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7ff900003b50 con 0x7ff910107d00 2024-03-16T11:41:13.021 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:07.832+0000 7ff91806a700 1 -- 172.21.15.29:0/641590233 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff91019c4a0 con 0x7ff910107d00 2024-03-16T11:41:13.021 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:07.832+0000 7ff90effd700 1 -- 172.21.15.29:0/641590233 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff900017040 con 0x7ff910107d00 2024-03-16T11:41:13.021 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:07.832+0000 7ff91806a700 1 -- 172.21.15.29:0/641590233 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff91019f140 con 0x7ff910107d00 2024-03-16T11:41:13.021 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:07.833+0000 7ff90effd700 1 -- 172.21.15.29:0/641590233 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 8) ==== 46106+0+0 (secure 0 0 0) 0x7ff900023020 con 0x7ff910107d00 2024-03-16T11:41:13.021 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:07.833+0000 7ff91806a700 1 -- 172.21.15.29:0/641590233 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff910063900 con 0x7ff910107d00 2024-03-16T11:41:13.022 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:07.833+0000 7ff90effd700 1 --2- 172.21.15.29:0/641590233 >> [v2:172.21.15.29:6800/2798831128,v1:172.21.15.29:6801/2798831128] conn(0x7ff8f8038f70 0x7ff8f803b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:41:13.022 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:07.833+0000 7ff90effd700 1 -- 172.21.15.29:0/641590233 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7ff90000f050 con 0x7ff910107d00 2024-03-16T11:41:13.022 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:07.833+0000 7ff915605700 1 --2- 172.21.15.29:0/641590233 >> [v2:172.21.15.29:6800/2798831128,v1:172.21.15.29:6801/2798831128] conn(0x7ff8f8038f70 0x7ff8f803b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:41:13.022 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:07.833+0000 7ff915605700 1 --2- 172.21.15.29:0/641590233 >> [v2:172.21.15.29:6800/2798831128,v1:172.21.15.29:6801/2798831128] conn(0x7ff8f8038f70 0x7ff8f803b430 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7ff904006fd0 tx=0x7ff904006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:41:13.022 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:07.838+0000 7ff90effd700 1 -- 172.21.15.29:0/641590233 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7ff90002a020 con 0x7ff910107d00 2024-03-16T11:41:13.022 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:08.046+0000 7ff91806a700 1 -- 172.21.15.29:0/641590233 --> [v2:172.21.15.29:6800/2798831128,v1:172.21.15.29:6801/2798831128] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi029", "addr": "172.21.15.29", "target": ["mon-mgr", ""]}) -- 0x7ff9101044c0 con 0x7ff8f8038f70 2024-03-16T11:41:13.022 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:12.460+0000 7ff90effd700 1 -- 172.21.15.29:0/641590233 <== mgr.14120 v2:172.21.15.29:6800/2798831128 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+48 (secure 0 0 0) 0x7ff9101044c0 con 0x7ff8f8038f70 2024-03-16T11:41:13.022 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:12.463+0000 7ff91806a700 1 -- 172.21.15.29:0/641590233 >> [v2:172.21.15.29:6800/2798831128,v1:172.21.15.29:6801/2798831128] conn(0x7ff8f8038f70 msgr2=0x7ff8f803b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:41:13.022 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:12.463+0000 7ff91806a700 1 --2- 172.21.15.29:0/641590233 >> [v2:172.21.15.29:6800/2798831128,v1:172.21.15.29:6801/2798831128] conn(0x7ff8f8038f70 0x7ff8f803b430 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7ff904006fd0 tx=0x7ff904006e40 comp rx=0 tx=0).stop 2024-03-16T11:41:13.022 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:12.463+0000 7ff91806a700 1 -- 172.21.15.29:0/641590233 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff910107d00 msgr2=0x7ff91019bcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:41:13.022 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:12.463+0000 7ff91806a700 1 --2- 172.21.15.29:0/641590233 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff910107d00 0x7ff91019bcd0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7ff900004210 tx=0x7ff90000bd50 comp rx=0 tx=0).stop 2024-03-16T11:41:13.022 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:12.463+0000 7ff91806a700 1 -- 172.21.15.29:0/641590233 shutdown_connections 2024-03-16T11:41:13.022 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:12.463+0000 7ff91806a700 1 --2- 172.21.15.29:0/641590233 >> [v2:172.21.15.29:6800/2798831128,v1:172.21.15.29:6801/2798831128] conn(0x7ff8f8038f70 0x7ff8f803b430 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:41:13.022 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:12.463+0000 7ff91806a700 1 --2- 172.21.15.29:0/641590233 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff910107d00 0x7ff91019bcd0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:41:13.023 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:12.463+0000 7ff91806a700 1 -- 172.21.15.29:0/641590233 >> 172.21.15.29:0/641590233 conn(0x7ff9101032a0 msgr2=0x7ff910103db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:41:13.023 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:12.463+0000 7ff91806a700 1 -- 172.21.15.29:0/641590233 shutdown_connections 2024-03-16T11:41:13.023 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:12.463+0000 7ff91806a700 1 -- 172.21.15.29:0/641590233 wait complete. 2024-03-16T11:41:13.023 INFO:teuthology.orchestra.run.smithi029.stdout:Deploying unmanaged mon service... 2024-03-16T11:41:13.761 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:41:13 smithi029 ceph-mon[31975]: from='mgr.14120 172.21.15.29:0/4098640102' entity='mgr.x' 2024-03-16T11:41:13.761 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:41:13 smithi029 ceph-mon[31975]: Added host smithi029 2024-03-16T11:41:13.761 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:41:13 smithi029 ceph-mon[31975]: from='mgr.14120 172.21.15.29:0/4098640102' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T11:41:14.477 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-03-16T11:41:14.477 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:13.524+0000 7fadabb3d700 1 Processor -- start 2024-03-16T11:41:14.477 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:13.524+0000 7fadabb3d700 1 -- start start 2024-03-16T11:41:14.477 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:13.525+0000 7fadabb3d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fada41087b0 0x7fada410aba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:41:14.477 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:13.525+0000 7fadabb3d700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fada407be30 con 0x7fada41087b0 2024-03-16T11:41:14.477 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:13.525+0000 7fada98d9700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fada41087b0 0x7fada410aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:41:14.478 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:13.525+0000 7fada98d9700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fada41087b0 0x7fada410aba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49032/0 (socket says 172.21.15.29:49032) 2024-03-16T11:41:14.478 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:13.525+0000 7fada98d9700 1 -- 172.21.15.29:0/317832508 learned_addr learned my addr 172.21.15.29:0/317832508 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:41:14.478 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:13.525+0000 7fada98d9700 1 -- 172.21.15.29:0/317832508 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fada410b0e0 con 0x7fada41087b0 2024-03-16T11:41:14.478 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:13.525+0000 7fada98d9700 1 --2- 172.21.15.29:0/317832508 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fada41087b0 0x7fada410aba0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fad940098a0 tx=0x7fad94009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e78e3673b4fc978a server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:41:14.478 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:13.525+0000 7fada88d7700 1 -- 172.21.15.29:0/317832508 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fad94004160 con 0x7fada41087b0 2024-03-16T11:41:14.478 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:13.525+0000 7fada88d7700 1 -- 172.21.15.29:0/317832508 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fad94004300 con 0x7fada41087b0 2024-03-16T11:41:14.478 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:13.525+0000 7fada88d7700 1 -- 172.21.15.29:0/317832508 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fad94011b50 con 0x7fada41087b0 2024-03-16T11:41:14.478 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:13.526+0000 7fadabb3d700 1 -- 172.21.15.29:0/317832508 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fada41087b0 msgr2=0x7fada410aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:41:14.478 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:13.526+0000 7fadabb3d700 1 --2- 172.21.15.29:0/317832508 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fada41087b0 0x7fada410aba0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fad940098a0 tx=0x7fad94009bb0 comp rx=0 tx=0).stop 2024-03-16T11:41:14.478 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:13.526+0000 7fadabb3d700 1 -- 172.21.15.29:0/317832508 shutdown_connections 2024-03-16T11:41:14.478 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:13.526+0000 7fadabb3d700 1 --2- 172.21.15.29:0/317832508 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fada41087b0 0x7fada410aba0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:41:14.478 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:13.526+0000 7fadabb3d700 1 -- 172.21.15.29:0/317832508 >> 172.21.15.29:0/317832508 conn(0x7fada41021a0 msgr2=0x7fada41045e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:41:14.478 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:13.526+0000 7fadabb3d700 1 -- 172.21.15.29:0/317832508 shutdown_connections 2024-03-16T11:41:14.478 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:13.526+0000 7fadabb3d700 1 -- 172.21.15.29:0/317832508 wait complete. 2024-03-16T11:41:14.479 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:13.527+0000 7fadabb3d700 1 Processor -- start 2024-03-16T11:41:14.479 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:13.527+0000 7fadabb3d700 1 -- start start 2024-03-16T11:41:14.479 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:13.527+0000 7fadabb3d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fada41087b0 0x7fada419bcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:41:14.479 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:13.527+0000 7fadabb3d700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fada410c100 con 0x7fada41087b0 2024-03-16T11:41:14.479 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:13.527+0000 7fada98d9700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fada41087b0 0x7fada419bcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:41:14.479 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:13.527+0000 7fada98d9700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fada41087b0 0x7fada419bcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49048/0 (socket says 172.21.15.29:49048) 2024-03-16T11:41:14.479 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:13.527+0000 7fada98d9700 1 -- 172.21.15.29:0/3840830958 learned_addr learned my addr 172.21.15.29:0/3840830958 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:41:14.479 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:13.527+0000 7fada98d9700 1 -- 172.21.15.29:0/3840830958 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fada419c1f0 con 0x7fada41087b0 2024-03-16T11:41:14.479 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:13.527+0000 7fada98d9700 1 --2- 172.21.15.29:0/3840830958 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fada41087b0 0x7fada419bcb0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fad94009fb0 tx=0x7fad94011bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:41:14.479 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:13.527+0000 7fad9affd700 1 -- 172.21.15.29:0/3840830958 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fad940038b0 con 0x7fada41087b0 2024-03-16T11:41:14.479 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:13.527+0000 7fad9affd700 1 -- 172.21.15.29:0/3840830958 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fad94011e50 con 0x7fada41087b0 2024-03-16T11:41:14.479 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:13.528+0000 7fadabb3d700 1 -- 172.21.15.29:0/3840830958 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fada419c480 con 0x7fada41087b0 2024-03-16T11:41:14.479 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:13.528+0000 7fad9affd700 1 -- 172.21.15.29:0/3840830958 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fad94019420 con 0x7fada41087b0 2024-03-16T11:41:14.479 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:13.528+0000 7fadabb3d700 1 -- 172.21.15.29:0/3840830958 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fada419f170 con 0x7fada41087b0 2024-03-16T11:41:14.480 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:13.528+0000 7fad9affd700 1 -- 172.21.15.29:0/3840830958 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 8) ==== 46106+0+0 (secure 0 0 0) 0x7fad940196d0 con 0x7fada41087b0 2024-03-16T11:41:14.480 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:13.528+0000 7fadabb3d700 1 -- 172.21.15.29:0/3840830958 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fada4063900 con 0x7fada41087b0 2024-03-16T11:41:14.480 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:13.528+0000 7fad9affd700 1 --2- 172.21.15.29:0/3840830958 >> [v2:172.21.15.29:6800/2798831128,v1:172.21.15.29:6801/2798831128] conn(0x7fad90038f20 0x7fad9003b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:41:14.480 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:13.529+0000 7fad9affd700 1 -- 172.21.15.29:0/3840830958 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fad9404d270 con 0x7fada41087b0 2024-03-16T11:41:14.480 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:13.529+0000 7fada90d8700 1 --2- 172.21.15.29:0/3840830958 >> [v2:172.21.15.29:6800/2798831128,v1:172.21.15.29:6801/2798831128] conn(0x7fad90038f20 0x7fad9003b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:41:14.480 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:13.529+0000 7fada90d8700 1 --2- 172.21.15.29:0/3840830958 >> [v2:172.21.15.29:6800/2798831128,v1:172.21.15.29:6801/2798831128] conn(0x7fad90038f20 0x7fad9003b3e0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fada0006fd0 tx=0x7fada0006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:41:14.480 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:13.534+0000 7fad9affd700 1 -- 172.21.15.29:0/3840830958 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fad94003ac0 con 0x7fada41087b0 2024-03-16T11:41:14.480 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:13.761+0000 7fadabb3d700 1 -- 172.21.15.29:0/3840830958 --> [v2:172.21.15.29:6800/2798831128,v1:172.21.15.29:6801/2798831128] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7fada41033f0 con 0x7fad90038f20 2024-03-16T11:41:14.480 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:13.765+0000 7fad9affd700 1 -- 172.21.15.29:0/3840830958 <== mgr.14120 v2:172.21.15.29:6800/2798831128 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7fada41033f0 con 0x7fad90038f20 2024-03-16T11:41:14.480 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:13.767+0000 7fadabb3d700 1 -- 172.21.15.29:0/3840830958 >> [v2:172.21.15.29:6800/2798831128,v1:172.21.15.29:6801/2798831128] conn(0x7fad90038f20 msgr2=0x7fad9003b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:41:14.480 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:13.767+0000 7fadabb3d700 1 --2- 172.21.15.29:0/3840830958 >> [v2:172.21.15.29:6800/2798831128,v1:172.21.15.29:6801/2798831128] conn(0x7fad90038f20 0x7fad9003b3e0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fada0006fd0 tx=0x7fada0006e40 comp rx=0 tx=0).stop 2024-03-16T11:41:14.480 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:13.767+0000 7fadabb3d700 1 -- 172.21.15.29:0/3840830958 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fada41087b0 msgr2=0x7fada419bcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:41:14.480 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:13.767+0000 7fadabb3d700 1 --2- 172.21.15.29:0/3840830958 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fada41087b0 0x7fada419bcb0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fad94009fb0 tx=0x7fad94011bf0 comp rx=0 tx=0).stop 2024-03-16T11:41:14.481 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:13.768+0000 7fadabb3d700 1 -- 172.21.15.29:0/3840830958 shutdown_connections 2024-03-16T11:41:14.481 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:13.768+0000 7fadabb3d700 1 --2- 172.21.15.29:0/3840830958 >> [v2:172.21.15.29:6800/2798831128,v1:172.21.15.29:6801/2798831128] conn(0x7fad90038f20 0x7fad9003b3e0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:41:14.481 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:13.768+0000 7fadabb3d700 1 --2- 172.21.15.29:0/3840830958 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fada41087b0 0x7fada419bcb0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:41:14.481 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:13.768+0000 7fadabb3d700 1 -- 172.21.15.29:0/3840830958 >> 172.21.15.29:0/3840830958 conn(0x7fada41021a0 msgr2=0x7fada4102ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:41:14.481 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:13.768+0000 7fadabb3d700 1 -- 172.21.15.29:0/3840830958 shutdown_connections 2024-03-16T11:41:14.481 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:13.768+0000 7fadabb3d700 1 -- 172.21.15.29:0/3840830958 wait complete. 2024-03-16T11:41:14.481 INFO:teuthology.orchestra.run.smithi029.stdout:Deploying unmanaged mgr service... 2024-03-16T11:41:15.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:41:14 smithi029 ceph-mon[31975]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T11:41:15.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:41:14 smithi029 ceph-mon[31975]: Saving service mon spec with placement count:5 2024-03-16T11:41:15.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:41:14 smithi029 ceph-mon[31975]: from='mgr.14120 172.21.15.29:0/4098640102' entity='mgr.x' 2024-03-16T11:41:16.344 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-03-16T11:41:16.344 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:15.274+0000 7f4bca97d700 1 Processor -- start 2024-03-16T11:41:16.344 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:15.274+0000 7f4bca97d700 1 -- start start 2024-03-16T11:41:16.344 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:15.274+0000 7f4bca97d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4bc4107da0 0x7f4bc41081c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:41:16.344 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:15.274+0000 7f4bca97d700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f4bc4108790 con 0x7f4bc4107da0 2024-03-16T11:41:16.344 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:15.274+0000 7f4bc3fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4bc4107da0 0x7f4bc41081c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:41:16.344 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:15.274+0000 7f4bc3fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4bc4107da0 0x7f4bc41081c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49056/0 (socket says 172.21.15.29:49056) 2024-03-16T11:41:16.344 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:15.274+0000 7f4bc3fff700 1 -- 172.21.15.29:0/4061157701 learned_addr learned my addr 172.21.15.29:0/4061157701 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:41:16.345 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:15.275+0000 7f4bc3fff700 1 -- 172.21.15.29:0/4061157701 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4bc4108f00 con 0x7f4bc4107da0 2024-03-16T11:41:16.345 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:15.275+0000 7f4bc3fff700 1 --2- 172.21.15.29:0/4061157701 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4bc4107da0 0x7f4bc41081c0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f4bbc009e80 tx=0x7f4bbc00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35581d420271f518 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:41:16.345 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:15.275+0000 7f4bc2ffd700 1 -- 172.21.15.29:0/4061157701 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4bbc017070 con 0x7f4bc4107da0 2024-03-16T11:41:16.345 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:15.275+0000 7f4bc2ffd700 1 -- 172.21.15.29:0/4061157701 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f4bbc00b8c0 con 0x7f4bc4107da0 2024-03-16T11:41:16.345 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:15.275+0000 7f4bc2ffd700 1 -- 172.21.15.29:0/4061157701 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4bbc0045c0 con 0x7f4bc4107da0 2024-03-16T11:41:16.345 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:15.276+0000 7f4bca97d700 1 -- 172.21.15.29:0/4061157701 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4bc4107da0 msgr2=0x7f4bc41081c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:41:16.345 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:15.276+0000 7f4bca97d700 1 --2- 172.21.15.29:0/4061157701 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4bc4107da0 0x7f4bc41081c0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f4bbc009e80 tx=0x7f4bbc00b330 comp rx=0 tx=0).stop 2024-03-16T11:41:16.345 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:15.276+0000 7f4bca97d700 1 -- 172.21.15.29:0/4061157701 shutdown_connections 2024-03-16T11:41:16.345 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:15.276+0000 7f4bca97d700 1 --2- 172.21.15.29:0/4061157701 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4bc4107da0 0x7f4bc41081c0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:41:16.345 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:15.276+0000 7f4bca97d700 1 -- 172.21.15.29:0/4061157701 >> 172.21.15.29:0/4061157701 conn(0x7f4bc4103360 msgr2=0x7f4bc4105780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:41:16.345 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:15.276+0000 7f4bca97d700 1 -- 172.21.15.29:0/4061157701 shutdown_connections 2024-03-16T11:41:16.345 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:15.276+0000 7f4bca97d700 1 -- 172.21.15.29:0/4061157701 wait complete. 2024-03-16T11:41:16.345 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:15.276+0000 7f4bca97d700 1 Processor -- start 2024-03-16T11:41:16.345 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:15.276+0000 7f4bca97d700 1 -- start start 2024-03-16T11:41:16.346 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:15.277+0000 7f4bca97d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4bc4107da0 0x7f4bc419bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:41:16.346 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:15.277+0000 7f4bca97d700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f4bc410c2b0 con 0x7f4bc4107da0 2024-03-16T11:41:16.346 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:15.277+0000 7f4bc3fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4bc4107da0 0x7f4bc419bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:41:16.346 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:15.277+0000 7f4bc3fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4bc4107da0 0x7f4bc419bcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49072/0 (socket says 172.21.15.29:49072) 2024-03-16T11:41:16.346 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:15.277+0000 7f4bc3fff700 1 -- 172.21.15.29:0/2274744087 learned_addr learned my addr 172.21.15.29:0/2274744087 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:41:16.346 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:15.277+0000 7f4bc3fff700 1 -- 172.21.15.29:0/2274744087 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4bc419c230 con 0x7f4bc4107da0 2024-03-16T11:41:16.346 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:15.277+0000 7f4bc3fff700 1 --2- 172.21.15.29:0/2274744087 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4bc4107da0 0x7f4bc419bcf0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f4bbc004210 tx=0x7f4bbc00bd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:41:16.346 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:15.277+0000 7f4bc17fa700 1 -- 172.21.15.29:0/2274744087 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4bbc020020 con 0x7f4bc4107da0 2024-03-16T11:41:16.346 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:15.277+0000 7f4bc17fa700 1 -- 172.21.15.29:0/2274744087 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f4bbc003b50 con 0x7f4bc4107da0 2024-03-16T11:41:16.346 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:15.277+0000 7f4bca97d700 1 -- 172.21.15.29:0/2274744087 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4bc419c4c0 con 0x7f4bc4107da0 2024-03-16T11:41:16.346 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:15.277+0000 7f4bc17fa700 1 -- 172.21.15.29:0/2274744087 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4bbc017040 con 0x7f4bc4107da0 2024-03-16T11:41:16.346 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:15.277+0000 7f4bca97d700 1 -- 172.21.15.29:0/2274744087 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4bc419f1a0 con 0x7f4bc4107da0 2024-03-16T11:41:16.346 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:15.278+0000 7f4bc17fa700 1 -- 172.21.15.29:0/2274744087 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 8) ==== 46106+0+0 (secure 0 0 0) 0x7f4bbc023020 con 0x7f4bc4107da0 2024-03-16T11:41:16.346 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:15.278+0000 7f4bc17fa700 1 --2- 172.21.15.29:0/2274744087 >> [v2:172.21.15.29:6800/2798831128,v1:172.21.15.29:6801/2798831128] conn(0x7f4bac038f20 0x7f4bac03b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:41:16.347 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:15.278+0000 7f4bc17fa700 1 -- 172.21.15.29:0/2274744087 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f4bbc00f050 con 0x7f4bc4107da0 2024-03-16T11:41:16.347 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:15.279+0000 7f4bc37fe700 1 --2- 172.21.15.29:0/2274744087 >> [v2:172.21.15.29:6800/2798831128,v1:172.21.15.29:6801/2798831128] conn(0x7f4bac038f20 0x7f4bac03b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:41:16.347 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:15.279+0000 7f4bca97d700 1 -- 172.21.15.29:0/2274744087 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4bc4063900 con 0x7f4bc4107da0 2024-03-16T11:41:16.347 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:15.279+0000 7f4bc37fe700 1 --2- 172.21.15.29:0/2274744087 >> [v2:172.21.15.29:6800/2798831128,v1:172.21.15.29:6801/2798831128] conn(0x7f4bac038f20 0x7f4bac03b3e0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f4bb8006fd0 tx=0x7f4bb8006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:41:16.347 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:15.284+0000 7f4bc17fa700 1 -- 172.21.15.29:0/2274744087 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f4bbc048080 con 0x7f4bc4107da0 2024-03-16T11:41:16.347 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:15.503+0000 7f4bca97d700 1 -- 172.21.15.29:0/2274744087 --> [v2:172.21.15.29:6800/2798831128,v1:172.21.15.29:6801/2798831128] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7f4bc41045d0 con 0x7f4bac038f20 2024-03-16T11:41:16.347 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:15.508+0000 7f4bc17fa700 1 -- 172.21.15.29:0/2274744087 <== mgr.14120 v2:172.21.15.29:6800/2798831128 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f4bc41045d0 con 0x7f4bac038f20 2024-03-16T11:41:16.347 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:15.510+0000 7f4bca97d700 1 -- 172.21.15.29:0/2274744087 >> [v2:172.21.15.29:6800/2798831128,v1:172.21.15.29:6801/2798831128] conn(0x7f4bac038f20 msgr2=0x7f4bac03b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:41:16.347 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:15.510+0000 7f4bca97d700 1 --2- 172.21.15.29:0/2274744087 >> [v2:172.21.15.29:6800/2798831128,v1:172.21.15.29:6801/2798831128] conn(0x7f4bac038f20 0x7f4bac03b3e0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f4bb8006fd0 tx=0x7f4bb8006e40 comp rx=0 tx=0).stop 2024-03-16T11:41:16.347 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:15.510+0000 7f4bca97d700 1 -- 172.21.15.29:0/2274744087 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4bc4107da0 msgr2=0x7f4bc419bcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:41:16.347 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:15.510+0000 7f4bca97d700 1 --2- 172.21.15.29:0/2274744087 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4bc4107da0 0x7f4bc419bcf0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f4bbc004210 tx=0x7f4bbc00bd50 comp rx=0 tx=0).stop 2024-03-16T11:41:16.347 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:15.510+0000 7f4bca97d700 1 -- 172.21.15.29:0/2274744087 shutdown_connections 2024-03-16T11:41:16.347 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:15.510+0000 7f4bca97d700 1 --2- 172.21.15.29:0/2274744087 >> [v2:172.21.15.29:6800/2798831128,v1:172.21.15.29:6801/2798831128] conn(0x7f4bac038f20 0x7f4bac03b3e0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:41:16.347 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:15.510+0000 7f4bca97d700 1 --2- 172.21.15.29:0/2274744087 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4bc4107da0 0x7f4bc419bcf0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:41:16.348 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:15.510+0000 7f4bca97d700 1 -- 172.21.15.29:0/2274744087 >> 172.21.15.29:0/2274744087 conn(0x7f4bc4103360 msgr2=0x7f4bc4103ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:41:16.348 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:15.510+0000 7f4bca97d700 1 -- 172.21.15.29:0/2274744087 shutdown_connections 2024-03-16T11:41:16.348 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:15.510+0000 7f4bca97d700 1 -- 172.21.15.29:0/2274744087 wait complete. 2024-03-16T11:41:16.588 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:41:16 smithi029 ceph-mon[31975]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T11:41:16.588 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:41:16 smithi029 ceph-mon[31975]: Saving service mgr spec with placement count:2 2024-03-16T11:41:16.588 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:41:16 smithi029 ceph-mon[31975]: from='mgr.14120 172.21.15.29:0/4098640102' entity='mgr.x' 2024-03-16T11:41:16.588 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:41:16 smithi029 ceph-mon[31975]: from='mgr.14120 172.21.15.29:0/4098640102' entity='mgr.x' 2024-03-16T11:41:16.588 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:41:16 smithi029 ceph-mon[31975]: from='mgr.14120 172.21.15.29:0/4098640102' entity='mgr.x' 2024-03-16T11:41:18.122 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:16.837+0000 7f69103b0700 1 Processor -- start 2024-03-16T11:41:18.122 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:16.837+0000 7f69103b0700 1 -- start start 2024-03-16T11:41:18.122 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:16.838+0000 7f69103b0700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6908105fe0 0x7f6908106400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:41:18.122 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:16.838+0000 7f69103b0700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f69081069d0 con 0x7f6908105fe0 2024-03-16T11:41:18.122 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:16.838+0000 7f690e14c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6908105fe0 0x7f6908106400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:41:18.122 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:16.838+0000 7f690e14c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6908105fe0 0x7f6908106400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49078/0 (socket says 172.21.15.29:49078) 2024-03-16T11:41:18.123 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:16.838+0000 7f690e14c700 1 -- 172.21.15.29:0/3431681558 learned_addr learned my addr 172.21.15.29:0/3431681558 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:41:18.123 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:16.838+0000 7f690e14c700 1 -- 172.21.15.29:0/3431681558 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6908108f70 con 0x7f6908105fe0 2024-03-16T11:41:18.123 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:16.838+0000 7f690e14c700 1 --2- 172.21.15.29:0/3431681558 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6908105fe0 0x7f6908106400 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f69040098a0 tx=0x7f6904009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c3b3568f57289f6 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:41:18.123 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:16.839+0000 7f690d14a700 1 -- 172.21.15.29:0/3431681558 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6904004160 con 0x7f6908105fe0 2024-03-16T11:41:18.123 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:16.839+0000 7f690d14a700 1 -- 172.21.15.29:0/3431681558 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6904004300 con 0x7f6908105fe0 2024-03-16T11:41:18.123 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:16.839+0000 7f690d14a700 1 -- 172.21.15.29:0/3431681558 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6904011b20 con 0x7f6908105fe0 2024-03-16T11:41:18.123 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:16.839+0000 7f69103b0700 1 -- 172.21.15.29:0/3431681558 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6908105fe0 msgr2=0x7f6908106400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:41:18.123 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:16.839+0000 7f69103b0700 1 --2- 172.21.15.29:0/3431681558 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6908105fe0 0x7f6908106400 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f69040098a0 tx=0x7f6904009bb0 comp rx=0 tx=0).stop 2024-03-16T11:41:18.123 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:16.840+0000 7f69103b0700 1 -- 172.21.15.29:0/3431681558 shutdown_connections 2024-03-16T11:41:18.123 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:16.840+0000 7f69103b0700 1 --2- 172.21.15.29:0/3431681558 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6908105fe0 0x7f6908106400 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:41:18.123 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:16.840+0000 7f69103b0700 1 -- 172.21.15.29:0/3431681558 >> 172.21.15.29:0/3431681558 conn(0x7f6908101930 msgr2=0x7f6908103d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:41:18.123 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:16.840+0000 7f69103b0700 1 -- 172.21.15.29:0/3431681558 shutdown_connections 2024-03-16T11:41:18.123 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:16.840+0000 7f69103b0700 1 -- 172.21.15.29:0/3431681558 wait complete. 2024-03-16T11:41:18.123 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:16.840+0000 7f69103b0700 1 Processor -- start 2024-03-16T11:41:18.123 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:16.840+0000 7f69103b0700 1 -- start start 2024-03-16T11:41:18.124 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:16.840+0000 7f69103b0700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6908105fe0 0x7f690819bc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:41:18.124 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:16.840+0000 7f69103b0700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f690810c320 con 0x7f6908105fe0 2024-03-16T11:41:18.124 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:16.841+0000 7f690e14c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6908105fe0 0x7f690819bc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:41:18.124 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:16.841+0000 7f690e14c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6908105fe0 0x7f690819bc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49094/0 (socket says 172.21.15.29:49094) 2024-03-16T11:41:18.124 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:16.841+0000 7f690e14c700 1 -- 172.21.15.29:0/3800426879 learned_addr learned my addr 172.21.15.29:0/3800426879 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:41:18.124 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:16.841+0000 7f690e14c700 1 -- 172.21.15.29:0/3800426879 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f690819c180 con 0x7f6908105fe0 2024-03-16T11:41:18.124 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:16.841+0000 7f690e14c700 1 --2- 172.21.15.29:0/3800426879 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6908105fe0 0x7f690819bc40 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f6904000c00 tx=0x7f6904011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:41:18.124 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:16.841+0000 7f68fb7fe700 1 -- 172.21.15.29:0/3800426879 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6904003b60 con 0x7f6908105fe0 2024-03-16T11:41:18.124 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:16.841+0000 7f68fb7fe700 1 -- 172.21.15.29:0/3800426879 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f690401a910 con 0x7f6908105fe0 2024-03-16T11:41:18.124 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:16.841+0000 7f69103b0700 1 -- 172.21.15.29:0/3800426879 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f690819c410 con 0x7f6908105fe0 2024-03-16T11:41:18.124 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:16.841+0000 7f68fb7fe700 1 -- 172.21.15.29:0/3800426879 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6904024930 con 0x7f6908105fe0 2024-03-16T11:41:18.124 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:16.841+0000 7f69103b0700 1 -- 172.21.15.29:0/3800426879 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f690819c870 con 0x7f6908105fe0 2024-03-16T11:41:18.124 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:16.842+0000 7f68fb7fe700 1 -- 172.21.15.29:0/3800426879 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 8) ==== 46106+0+0 (secure 0 0 0) 0x7f6904022020 con 0x7f6908105fe0 2024-03-16T11:41:18.124 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:16.842+0000 7f69103b0700 1 -- 172.21.15.29:0/3800426879 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6908063900 con 0x7f6908105fe0 2024-03-16T11:41:18.124 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:16.842+0000 7f68fb7fe700 1 --2- 172.21.15.29:0/3800426879 >> [v2:172.21.15.29:6800/2798831128,v1:172.21.15.29:6801/2798831128] conn(0x7f68f4038f20 0x7f68f403b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:41:18.125 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:16.842+0000 7f68fb7fe700 1 -- 172.21.15.29:0/3800426879 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f690404d010 con 0x7f6908105fe0 2024-03-16T11:41:18.125 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:16.842+0000 7f690d94b700 1 --2- 172.21.15.29:0/3800426879 >> [v2:172.21.15.29:6800/2798831128,v1:172.21.15.29:6801/2798831128] conn(0x7f68f4038f20 0x7f68f403b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:41:18.125 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:16.843+0000 7f690d94b700 1 --2- 172.21.15.29:0/3800426879 >> [v2:172.21.15.29:6800/2798831128,v1:172.21.15.29:6801/2798831128] conn(0x7f68f4038f20 0x7f68f403b3e0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f68fc006fd0 tx=0x7f68fc006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:41:18.125 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:16.852+0000 7f68fb7fe700 1 -- 172.21.15.29:0/3800426879 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f690401aac0 con 0x7f6908105fe0 2024-03-16T11:41:18.125 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:17.073+0000 7f69103b0700 1 -- 172.21.15.29:0/3800426879 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) -- 0x7f690819f100 con 0x7f6908105fe0 2024-03-16T11:41:18.125 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:17.076+0000 7f68fb7fe700 1 -- 172.21.15.29:0/3800426879 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) ==== 142+0+0 (secure 0 0 0) 0x7f690400b3e0 con 0x7f6908105fe0 2024-03-16T11:41:18.125 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:17.078+0000 7f69103b0700 1 -- 172.21.15.29:0/3800426879 >> [v2:172.21.15.29:6800/2798831128,v1:172.21.15.29:6801/2798831128] conn(0x7f68f4038f20 msgr2=0x7f68f403b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:41:18.125 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:17.078+0000 7f69103b0700 1 --2- 172.21.15.29:0/3800426879 >> [v2:172.21.15.29:6800/2798831128,v1:172.21.15.29:6801/2798831128] conn(0x7f68f4038f20 0x7f68f403b3e0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f68fc006fd0 tx=0x7f68fc006e40 comp rx=0 tx=0).stop 2024-03-16T11:41:18.125 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:17.078+0000 7f69103b0700 1 -- 172.21.15.29:0/3800426879 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6908105fe0 msgr2=0x7f690819bc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:41:18.125 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:17.078+0000 7f69103b0700 1 --2- 172.21.15.29:0/3800426879 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6908105fe0 0x7f690819bc40 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f6904000c00 tx=0x7f6904011c10 comp rx=0 tx=0).stop 2024-03-16T11:41:18.125 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:17.078+0000 7f69103b0700 1 -- 172.21.15.29:0/3800426879 shutdown_connections 2024-03-16T11:41:18.125 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:17.078+0000 7f69103b0700 1 --2- 172.21.15.29:0/3800426879 >> [v2:172.21.15.29:6800/2798831128,v1:172.21.15.29:6801/2798831128] conn(0x7f68f4038f20 0x7f68f403b3e0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:41:18.125 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:17.078+0000 7f69103b0700 1 --2- 172.21.15.29:0/3800426879 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6908105fe0 0x7f690819bc40 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:41:18.125 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:17.078+0000 7f69103b0700 1 -- 172.21.15.29:0/3800426879 >> 172.21.15.29:0/3800426879 conn(0x7f6908101930 msgr2=0x7f6908102390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:41:18.125 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:17.078+0000 7f69103b0700 1 -- 172.21.15.29:0/3800426879 shutdown_connections 2024-03-16T11:41:18.126 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:17.078+0000 7f69103b0700 1 -- 172.21.15.29:0/3800426879 wait complete. 2024-03-16T11:41:18.244 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:41:18 smithi029 ceph-mon[31975]: from='client.? 172.21.15.29:0/3800426879' entity='client.admin' 2024-03-16T11:41:19.405 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:41:19 smithi029 ceph-mon[31975]: from='mgr.14120 172.21.15.29:0/4098640102' entity='mgr.x' 2024-03-16T11:41:19.405 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:41:19 smithi029 ceph-mon[31975]: from='client.? 172.21.15.29:0/3814231294' entity='client.admin' 2024-03-16T11:41:19.413 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:18.568+0000 7f12cc2c2700 1 Processor -- start 2024-03-16T11:41:19.413 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:18.568+0000 7f12cc2c2700 1 -- start start 2024-03-16T11:41:19.413 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:18.568+0000 7f12cc2c2700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f12c407de60 0x7f12c407c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:41:19.413 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:18.568+0000 7f12cc2c2700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f12c407c890 con 0x7f12c407de60 2024-03-16T11:41:19.413 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:18.568+0000 7f12ca05e700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f12c407de60 0x7f12c407c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:41:19.413 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:18.568+0000 7f12ca05e700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f12c407de60 0x7f12c407c2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49102/0 (socket says 172.21.15.29:49102) 2024-03-16T11:41:19.414 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:18.568+0000 7f12ca05e700 1 -- 172.21.15.29:0/25658396 learned_addr learned my addr 172.21.15.29:0/25658396 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:41:19.414 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:18.568+0000 7f12ca05e700 1 -- 172.21.15.29:0/25658396 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12c407ca10 con 0x7f12c407de60 2024-03-16T11:41:19.414 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:18.569+0000 7f12ca05e700 1 --2- 172.21.15.29:0/25658396 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f12c407de60 0x7f12c407c2c0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f12b40098a0 tx=0x7f12b4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c614c2a3e1a8505 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:41:19.414 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:18.569+0000 7f12c905c700 1 -- 172.21.15.29:0/25658396 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f12b4004160 con 0x7f12c407de60 2024-03-16T11:41:19.414 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:18.569+0000 7f12c905c700 1 -- 172.21.15.29:0/25658396 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f12b4004300 con 0x7f12c407de60 2024-03-16T11:41:19.414 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:18.569+0000 7f12c905c700 1 -- 172.21.15.29:0/25658396 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f12b4011b20 con 0x7f12c407de60 2024-03-16T11:41:19.414 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:18.569+0000 7f12cc2c2700 1 -- 172.21.15.29:0/25658396 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f12c407de60 msgr2=0x7f12c407c2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:41:19.414 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:18.569+0000 7f12cc2c2700 1 --2- 172.21.15.29:0/25658396 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f12c407de60 0x7f12c407c2c0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f12b40098a0 tx=0x7f12b4009bb0 comp rx=0 tx=0).stop 2024-03-16T11:41:19.414 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:18.570+0000 7f12cc2c2700 1 -- 172.21.15.29:0/25658396 shutdown_connections 2024-03-16T11:41:19.414 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:18.570+0000 7f12cc2c2700 1 --2- 172.21.15.29:0/25658396 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f12c407de60 0x7f12c407c2c0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:41:19.414 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:18.570+0000 7f12cc2c2700 1 -- 172.21.15.29:0/25658396 >> 172.21.15.29:0/25658396 conn(0x7f12c41032f0 msgr2=0x7f12c4105710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:41:19.414 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:18.570+0000 7f12cc2c2700 1 -- 172.21.15.29:0/25658396 shutdown_connections 2024-03-16T11:41:19.414 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:18.570+0000 7f12cc2c2700 1 -- 172.21.15.29:0/25658396 wait complete. 2024-03-16T11:41:19.415 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:18.570+0000 7f12cc2c2700 1 Processor -- start 2024-03-16T11:41:19.415 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:18.570+0000 7f12cc2c2700 1 -- start start 2024-03-16T11:41:19.415 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:18.570+0000 7f12cc2c2700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f12c407de60 0x7f12c419def0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:41:19.415 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:18.570+0000 7f12cc2c2700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f12c410e390 con 0x7f12c407de60 2024-03-16T11:41:19.415 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:18.571+0000 7f12ca05e700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f12c407de60 0x7f12c419def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:41:19.415 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:18.571+0000 7f12ca05e700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f12c407de60 0x7f12c419def0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49104/0 (socket says 172.21.15.29:49104) 2024-03-16T11:41:19.415 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:18.571+0000 7f12ca05e700 1 -- 172.21.15.29:0/3814231294 learned_addr learned my addr 172.21.15.29:0/3814231294 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:41:19.415 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:18.571+0000 7f12ca05e700 1 -- 172.21.15.29:0/3814231294 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12c419e430 con 0x7f12c407de60 2024-03-16T11:41:19.415 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:18.571+0000 7f12ca05e700 1 --2- 172.21.15.29:0/3814231294 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f12c407de60 0x7f12c419def0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f12b40116a0 tx=0x7f12b4011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:41:19.415 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:18.571+0000 7f12bb7fe700 1 -- 172.21.15.29:0/3814231294 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f12b4003a40 con 0x7f12c407de60 2024-03-16T11:41:19.415 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:18.571+0000 7f12bb7fe700 1 -- 172.21.15.29:0/3814231294 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f12b4003be0 con 0x7f12c407de60 2024-03-16T11:41:19.415 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:18.571+0000 7f12cc2c2700 1 -- 172.21.15.29:0/3814231294 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f12c419e6c0 con 0x7f12c407de60 2024-03-16T11:41:19.415 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:18.571+0000 7f12cc2c2700 1 -- 172.21.15.29:0/3814231294 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f12c419eb20 con 0x7f12c407de60 2024-03-16T11:41:19.415 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:18.571+0000 7f12bb7fe700 1 -- 172.21.15.29:0/3814231294 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f12b401a720 con 0x7f12c407de60 2024-03-16T11:41:19.415 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:18.572+0000 7f12cc2c2700 1 -- 172.21.15.29:0/3814231294 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f12a8005360 con 0x7f12c407de60 2024-03-16T11:41:19.416 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:18.572+0000 7f12bb7fe700 1 -- 172.21.15.29:0/3814231294 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 8) ==== 46106+0+0 (secure 0 0 0) 0x7f12b401aa90 con 0x7f12c407de60 2024-03-16T11:41:19.416 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:18.573+0000 7f12bb7fe700 1 --2- 172.21.15.29:0/3814231294 >> [v2:172.21.15.29:6800/2798831128,v1:172.21.15.29:6801/2798831128] conn(0x7f12b0038f20 0x7f12b003b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:41:19.416 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:18.573+0000 7f12bb7fe700 1 -- 172.21.15.29:0/3814231294 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f12b404d4b0 con 0x7f12c407de60 2024-03-16T11:41:19.416 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:18.573+0000 7f12c985d700 1 --2- 172.21.15.29:0/3814231294 >> [v2:172.21.15.29:6800/2798831128,v1:172.21.15.29:6801/2798831128] conn(0x7f12b0038f20 0x7f12b003b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:41:19.416 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:18.573+0000 7f12c985d700 1 --2- 172.21.15.29:0/3814231294 >> [v2:172.21.15.29:6800/2798831128,v1:172.21.15.29:6801/2798831128] conn(0x7f12b0038f20 0x7f12b003b3e0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f12c0006fd0 tx=0x7f12c0006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:41:19.416 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:18.578+0000 7f12bb7fe700 1 -- 172.21.15.29:0/3814231294 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f12b4011e20 con 0x7f12c407de60 2024-03-16T11:41:19.416 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:18.785+0000 7f12cc2c2700 1 -- 172.21.15.29:0/3814231294 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) -- 0x7f12a8005650 con 0x7f12c407de60 2024-03-16T11:41:19.416 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:18.787+0000 7f12bb7fe700 1 -- 172.21.15.29:0/3814231294 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) ==== 130+0+0 (secure 0 0 0) 0x7f12b400bc30 con 0x7f12c407de60 2024-03-16T11:41:19.416 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:18.789+0000 7f12cc2c2700 1 -- 172.21.15.29:0/3814231294 >> [v2:172.21.15.29:6800/2798831128,v1:172.21.15.29:6801/2798831128] conn(0x7f12b0038f20 msgr2=0x7f12b003b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:41:19.416 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:18.790+0000 7f12cc2c2700 1 --2- 172.21.15.29:0/3814231294 >> [v2:172.21.15.29:6800/2798831128,v1:172.21.15.29:6801/2798831128] conn(0x7f12b0038f20 0x7f12b003b3e0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f12c0006fd0 tx=0x7f12c0006e40 comp rx=0 tx=0).stop 2024-03-16T11:41:19.416 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:18.790+0000 7f12cc2c2700 1 -- 172.21.15.29:0/3814231294 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f12c407de60 msgr2=0x7f12c419def0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:41:19.416 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:18.790+0000 7f12cc2c2700 1 --2- 172.21.15.29:0/3814231294 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f12c407de60 0x7f12c419def0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f12b40116a0 tx=0x7f12b4011ba0 comp rx=0 tx=0).stop 2024-03-16T11:41:19.416 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:18.790+0000 7f12cc2c2700 1 -- 172.21.15.29:0/3814231294 shutdown_connections 2024-03-16T11:41:19.416 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:18.790+0000 7f12cc2c2700 1 --2- 172.21.15.29:0/3814231294 >> [v2:172.21.15.29:6800/2798831128,v1:172.21.15.29:6801/2798831128] conn(0x7f12b0038f20 0x7f12b003b3e0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:41:19.416 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:18.790+0000 7f12cc2c2700 1 --2- 172.21.15.29:0/3814231294 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f12c407de60 0x7f12c419def0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:41:19.417 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:18.790+0000 7f12cc2c2700 1 -- 172.21.15.29:0/3814231294 >> 172.21.15.29:0/3814231294 conn(0x7f12c41032f0 msgr2=0x7f12c4103e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:41:19.417 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:18.790+0000 7f12cc2c2700 1 -- 172.21.15.29:0/3814231294 shutdown_connections 2024-03-16T11:41:19.417 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:18.790+0000 7f12cc2c2700 1 -- 172.21.15.29:0/3814231294 wait complete. 2024-03-16T11:41:19.417 INFO:teuthology.orchestra.run.smithi029.stdout:Enabling the dashboard module... 2024-03-16T11:41:20.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:41:20 smithi029 ceph-mon[31975]: from='mgr.14120 172.21.15.29:0/4098640102' entity='mgr.x' 2024-03-16T11:41:20.839 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:41:20 smithi029 ceph-mon[31975]: from='client.? 172.21.15.29:0/277985651' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-03-16T11:41:20.839 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 16 11:41:20 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-x[32210]: ignoring --setuser ceph since I am not root 2024-03-16T11:41:20.839 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 16 11:41:20 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-x[32210]: ignoring --setgroup ceph since I am not root 2024-03-16T11:41:21.262 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 16 11:41:21 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-x[32210]: 2024-03-16T11:41:21.086+0000 7f7ea1243200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-16T11:41:21.262 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 16 11:41:21 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-x[32210]: 2024-03-16T11:41:21.181+0000 7f7ea1243200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-16T11:41:21.667 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:19.884+0000 7f3c22422700 1 Processor -- start 2024-03-16T11:41:21.667 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:19.884+0000 7f3c22422700 1 -- start start 2024-03-16T11:41:21.667 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:19.884+0000 7f3c22422700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c1c108800 0x7f3c1c10abf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:41:21.668 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:19.884+0000 7f3c22422700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f3c1c07bdf0 con 0x7f3c1c108800 2024-03-16T11:41:21.668 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:19.884+0000 7f3c1bfff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c1c108800 0x7f3c1c10abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:41:21.668 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:19.884+0000 7f3c1bfff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c1c108800 0x7f3c1c10abf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47638/0 (socket says 172.21.15.29:47638) 2024-03-16T11:41:21.668 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:19.884+0000 7f3c1bfff700 1 -- 172.21.15.29:0/1829825439 learned_addr learned my addr 172.21.15.29:0/1829825439 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:41:21.668 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:19.885+0000 7f3c1bfff700 1 -- 172.21.15.29:0/1829825439 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c1c10b130 con 0x7f3c1c108800 2024-03-16T11:41:21.668 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:19.885+0000 7f3c1bfff700 1 --2- 172.21.15.29:0/1829825439 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c1c108800 0x7f3c1c10abf0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f3c0c0098a0 tx=0x7f3c0c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1ba664d023e60e0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:41:21.668 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:19.885+0000 7f3c1affd700 1 -- 172.21.15.29:0/1829825439 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3c0c004160 con 0x7f3c1c108800 2024-03-16T11:41:21.668 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:19.885+0000 7f3c1affd700 1 -- 172.21.15.29:0/1829825439 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3c0c004300 con 0x7f3c1c108800 2024-03-16T11:41:21.668 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:19.885+0000 7f3c1affd700 1 -- 172.21.15.29:0/1829825439 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3c0c011b20 con 0x7f3c1c108800 2024-03-16T11:41:21.668 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:19.886+0000 7f3c22422700 1 -- 172.21.15.29:0/1829825439 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c1c108800 msgr2=0x7f3c1c10abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:41:21.668 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:19.886+0000 7f3c22422700 1 --2- 172.21.15.29:0/1829825439 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c1c108800 0x7f3c1c10abf0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f3c0c0098a0 tx=0x7f3c0c009bb0 comp rx=0 tx=0).stop 2024-03-16T11:41:21.668 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:19.886+0000 7f3c22422700 1 -- 172.21.15.29:0/1829825439 shutdown_connections 2024-03-16T11:41:21.668 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:19.886+0000 7f3c22422700 1 --2- 172.21.15.29:0/1829825439 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c1c108800 0x7f3c1c10abf0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:41:21.668 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:19.886+0000 7f3c22422700 1 -- 172.21.15.29:0/1829825439 >> 172.21.15.29:0/1829825439 conn(0x7f3c1c1021b0 msgr2=0x7f3c1c1045f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:41:21.668 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:19.886+0000 7f3c22422700 1 -- 172.21.15.29:0/1829825439 shutdown_connections 2024-03-16T11:41:21.669 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:19.886+0000 7f3c22422700 1 -- 172.21.15.29:0/1829825439 wait complete. 2024-03-16T11:41:21.669 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:19.886+0000 7f3c22422700 1 Processor -- start 2024-03-16T11:41:21.669 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:19.886+0000 7f3c22422700 1 -- start start 2024-03-16T11:41:21.669 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:19.887+0000 7f3c22422700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c1c108800 0x7f3c1c108080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:41:21.669 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:19.887+0000 7f3c22422700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f3c1c10c130 con 0x7f3c1c108800 2024-03-16T11:41:21.669 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:19.887+0000 7f3c1bfff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c1c108800 0x7f3c1c108080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:41:21.669 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:19.887+0000 7f3c1bfff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c1c108800 0x7f3c1c108080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47650/0 (socket says 172.21.15.29:47650) 2024-03-16T11:41:21.669 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:19.887+0000 7f3c1bfff700 1 -- 172.21.15.29:0/277985651 learned_addr learned my addr 172.21.15.29:0/277985651 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:41:21.669 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:19.887+0000 7f3c1bfff700 1 -- 172.21.15.29:0/277985651 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c1c1085c0 con 0x7f3c1c108800 2024-03-16T11:41:21.669 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:19.887+0000 7f3c1bfff700 1 --2- 172.21.15.29:0/277985651 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c1c108800 0x7f3c1c108080 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f3c0c0116a0 tx=0x7f3c0c011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:41:21.669 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:19.887+0000 7f3c197fa700 1 -- 172.21.15.29:0/277985651 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3c0c003a40 con 0x7f3c1c108800 2024-03-16T11:41:21.669 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:19.887+0000 7f3c197fa700 1 -- 172.21.15.29:0/277985651 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3c0c003be0 con 0x7f3c1c108800 2024-03-16T11:41:21.669 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:19.887+0000 7f3c22422700 1 -- 172.21.15.29:0/277985651 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3c1c1067b0 con 0x7f3c1c108800 2024-03-16T11:41:21.669 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:19.887+0000 7f3c197fa700 1 -- 172.21.15.29:0/277985651 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3c0c01a750 con 0x7f3c1c108800 2024-03-16T11:41:21.669 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:19.887+0000 7f3c22422700 1 -- 172.21.15.29:0/277985651 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3c1c106c70 con 0x7f3c1c108800 2024-03-16T11:41:21.669 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:19.888+0000 7f3c197fa700 1 -- 172.21.15.29:0/277985651 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 8) ==== 46106+0+0 (secure 0 0 0) 0x7f3c0c01aa50 con 0x7f3c1c108800 2024-03-16T11:41:21.670 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:19.888+0000 7f3c197fa700 1 --2- 172.21.15.29:0/277985651 >> [v2:172.21.15.29:6800/2798831128,v1:172.21.15.29:6801/2798831128] conn(0x7f3c04038ed0 0x7f3c0403b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:41:21.670 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:19.888+0000 7f3c22422700 1 -- 172.21.15.29:0/277985651 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3c1c063900 con 0x7f3c1c108800 2024-03-16T11:41:21.670 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:19.888+0000 7f3c197fa700 1 -- 172.21.15.29:0/277985651 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f3c0c04d500 con 0x7f3c1c108800 2024-03-16T11:41:21.670 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:19.889+0000 7f3c1b7fe700 1 --2- 172.21.15.29:0/277985651 >> [v2:172.21.15.29:6800/2798831128,v1:172.21.15.29:6801/2798831128] conn(0x7f3c04038ed0 0x7f3c0403b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:41:21.670 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:19.889+0000 7f3c1b7fe700 1 --2- 172.21.15.29:0/277985651 >> [v2:172.21.15.29:6800/2798831128,v1:172.21.15.29:6801/2798831128] conn(0x7f3c04038ed0 0x7f3c0403b390 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f3c10006fd0 tx=0x7f3c10006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:41:21.670 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:19.894+0000 7f3c197fa700 1 -- 172.21.15.29:0/277985651 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f3c0c011e20 con 0x7f3c1c108800 2024-03-16T11:41:21.670 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:20.173+0000 7f3c22422700 1 -- 172.21.15.29:0/277985651 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) -- 0x7f3c1c106ef0 con 0x7f3c1c108800 2024-03-16T11:41:21.670 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:20.510+0000 7f3c197fa700 1 -- 172.21.15.29:0/277985651 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) ==== 88+0+0 (secure 0 0 0) 0x7f3c0c00ba30 con 0x7f3c1c108800 2024-03-16T11:41:21.670 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:20.512+0000 7f3c197fa700 1 -- 172.21.15.29:0/277985651 <== mon.0 v2:172.21.15.29:3300/0 8 ==== mgrmap(e 9) ==== 46119+0+0 (secure 0 0 0) 0x7f3c0c02d910 con 0x7f3c1c108800 2024-03-16T11:41:21.670 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:20.514+0000 7f3c22422700 1 -- 172.21.15.29:0/277985651 >> [v2:172.21.15.29:6800/2798831128,v1:172.21.15.29:6801/2798831128] conn(0x7f3c04038ed0 msgr2=0x7f3c0403b390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:41:21.670 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:20.514+0000 7f3c22422700 1 --2- 172.21.15.29:0/277985651 >> [v2:172.21.15.29:6800/2798831128,v1:172.21.15.29:6801/2798831128] conn(0x7f3c04038ed0 0x7f3c0403b390 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f3c10006fd0 tx=0x7f3c10006e40 comp rx=0 tx=0).stop 2024-03-16T11:41:21.670 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:20.514+0000 7f3c22422700 1 -- 172.21.15.29:0/277985651 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c1c108800 msgr2=0x7f3c1c108080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:41:21.670 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:20.514+0000 7f3c22422700 1 --2- 172.21.15.29:0/277985651 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c1c108800 0x7f3c1c108080 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f3c0c0116a0 tx=0x7f3c0c011ba0 comp rx=0 tx=0).stop 2024-03-16T11:41:21.670 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:20.514+0000 7f3c22422700 1 -- 172.21.15.29:0/277985651 shutdown_connections 2024-03-16T11:41:21.671 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:20.514+0000 7f3c22422700 1 --2- 172.21.15.29:0/277985651 >> [v2:172.21.15.29:6800/2798831128,v1:172.21.15.29:6801/2798831128] conn(0x7f3c04038ed0 0x7f3c0403b390 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:41:21.671 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:20.514+0000 7f3c22422700 1 --2- 172.21.15.29:0/277985651 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c1c108800 0x7f3c1c108080 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:41:21.671 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:20.514+0000 7f3c22422700 1 -- 172.21.15.29:0/277985651 >> 172.21.15.29:0/277985651 conn(0x7f3c1c1021b0 msgr2=0x7f3c1c102cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:41:21.671 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:20.514+0000 7f3c22422700 1 -- 172.21.15.29:0/277985651 shutdown_connections 2024-03-16T11:41:21.671 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:20.514+0000 7f3c22422700 1 -- 172.21.15.29:0/277985651 wait complete. 2024-03-16T11:41:21.785 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:41:21 smithi029 ceph-mon[31975]: from='client.? 172.21.15.29:0/277985651' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-03-16T11:41:21.786 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:41:21 smithi029 ceph-mon[31975]: mgrmap e9: x(active, since 20s) 2024-03-16T11:41:22.066 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 16 11:41:21 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-x[32210]: 2024-03-16T11:41:21.847+0000 7f7ea1243200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-16T11:41:22.066 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 16 11:41:21 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-x[32210]: 2024-03-16T11:41:21.944+0000 7f7ea1243200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-16T11:41:22.338 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 16 11:41:22 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-x[32210]: 2024-03-16T11:41:22.092+0000 7f7ea1243200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-16T11:41:22.338 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 16 11:41:22 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-x[32210]: 2024-03-16T11:41:22.191+0000 7f7ea1243200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-16T11:41:22.338 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 16 11:41:22 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-x[32210]: 2024-03-16T11:41:22.276+0000 7f7ea1243200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-16T11:41:22.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:41:22 smithi029 ceph-mon[31975]: from='client.? 172.21.15.29:0/4154337527' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-16T11:41:22.667 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 16 11:41:22 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-x[32210]: 2024-03-16T11:41:22.555+0000 7f7ea1243200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-16T11:41:22.667 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 16 11:41:22 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-x[32210]: 2024-03-16T11:41:22.667+0000 7f7ea1243200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-16T11:41:22.956 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 16 11:41:22 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-x[32210]: 2024-03-16T11:41:22.756+0000 7f7ea1243200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-16T11:41:22.956 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 16 11:41:22 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-x[32210]: 2024-03-16T11:41:22.853+0000 7f7ea1243200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-16T11:41:22.965 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout { 2024-03-16T11:41:22.965 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-03-16T11:41:22.965 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-16T11:41:22.965 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-03-16T11:41:22.965 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-16T11:41:22.965 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout } 2024-03-16T11:41:22.965 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:22.088+0000 7fd17bd0f700 1 Processor -- start 2024-03-16T11:41:22.965 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:22.088+0000 7fd17bd0f700 1 -- start start 2024-03-16T11:41:22.965 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:22.088+0000 7fd17bd0f700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd174108790 0x7fd17410ab80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:41:22.966 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:22.088+0000 7fd17bd0f700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fd17407be70 con 0x7fd174108790 2024-03-16T11:41:22.966 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:22.088+0000 7fd179aab700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd174108790 0x7fd17410ab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:41:22.966 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:22.088+0000 7fd179aab700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd174108790 0x7fd17410ab80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47662/0 (socket says 172.21.15.29:47662) 2024-03-16T11:41:22.966 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:22.088+0000 7fd179aab700 1 -- 172.21.15.29:0/411626957 learned_addr learned my addr 172.21.15.29:0/411626957 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:41:22.966 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:22.089+0000 7fd179aab700 1 -- 172.21.15.29:0/411626957 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd17410b0c0 con 0x7fd174108790 2024-03-16T11:41:22.966 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:22.089+0000 7fd179aab700 1 --2- 172.21.15.29:0/411626957 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd174108790 0x7fd17410ab80 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fd1700098a0 tx=0x7fd170009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ec946b5c55ff09e server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:41:22.966 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:22.089+0000 7fd178aa9700 1 -- 172.21.15.29:0/411626957 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd170004160 con 0x7fd174108790 2024-03-16T11:41:22.966 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:22.089+0000 7fd178aa9700 1 -- 172.21.15.29:0/411626957 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd170004300 con 0x7fd174108790 2024-03-16T11:41:22.966 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:22.089+0000 7fd178aa9700 1 -- 172.21.15.29:0/411626957 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd170011b20 con 0x7fd174108790 2024-03-16T11:41:22.966 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:22.090+0000 7fd17bd0f700 1 -- 172.21.15.29:0/411626957 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd174108790 msgr2=0x7fd17410ab80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:41:22.966 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:22.090+0000 7fd17bd0f700 1 --2- 172.21.15.29:0/411626957 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd174108790 0x7fd17410ab80 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fd1700098a0 tx=0x7fd170009bb0 comp rx=0 tx=0).stop 2024-03-16T11:41:22.966 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:22.090+0000 7fd17bd0f700 1 -- 172.21.15.29:0/411626957 shutdown_connections 2024-03-16T11:41:22.966 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:22.090+0000 7fd17bd0f700 1 --2- 172.21.15.29:0/411626957 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd174108790 0x7fd17410ab80 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:41:22.966 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:22.090+0000 7fd17bd0f700 1 -- 172.21.15.29:0/411626957 >> 172.21.15.29:0/411626957 conn(0x7fd1741021c0 msgr2=0x7fd174104600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:41:22.967 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:22.090+0000 7fd17bd0f700 1 -- 172.21.15.29:0/411626957 shutdown_connections 2024-03-16T11:41:22.967 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:22.090+0000 7fd17bd0f700 1 -- 172.21.15.29:0/411626957 wait complete. 2024-03-16T11:41:22.967 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:22.090+0000 7fd17bd0f700 1 Processor -- start 2024-03-16T11:41:22.967 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:22.091+0000 7fd17bd0f700 1 -- start start 2024-03-16T11:41:22.967 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:22.091+0000 7fd17bd0f700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd174108790 0x7fd174108010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:41:22.967 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:22.091+0000 7fd17bd0f700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fd17410c130 con 0x7fd174108790 2024-03-16T11:41:22.967 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:22.091+0000 7fd179aab700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd174108790 0x7fd174108010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:41:22.967 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:22.091+0000 7fd179aab700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd174108790 0x7fd174108010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47670/0 (socket says 172.21.15.29:47670) 2024-03-16T11:41:22.967 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:22.091+0000 7fd179aab700 1 -- 172.21.15.29:0/4154337527 learned_addr learned my addr 172.21.15.29:0/4154337527 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:41:22.967 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:22.091+0000 7fd179aab700 1 -- 172.21.15.29:0/4154337527 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd174108550 con 0x7fd174108790 2024-03-16T11:41:22.967 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:22.091+0000 7fd179aab700 1 --2- 172.21.15.29:0/4154337527 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd174108790 0x7fd174108010 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fd1700116a0 tx=0x7fd170011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:41:22.967 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:22.091+0000 7fd16affd700 1 -- 172.21.15.29:0/4154337527 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd170003a40 con 0x7fd174108790 2024-03-16T11:41:22.967 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:22.091+0000 7fd16affd700 1 -- 172.21.15.29:0/4154337527 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd170003be0 con 0x7fd174108790 2024-03-16T11:41:22.967 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:22.092+0000 7fd17bd0f700 1 -- 172.21.15.29:0/4154337527 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd174106740 con 0x7fd174108790 2024-03-16T11:41:22.968 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:22.092+0000 7fd16affd700 1 -- 172.21.15.29:0/4154337527 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd17001a750 con 0x7fd174108790 2024-03-16T11:41:22.968 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:22.092+0000 7fd17bd0f700 1 -- 172.21.15.29:0/4154337527 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd174106c00 con 0x7fd174108790 2024-03-16T11:41:22.968 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:22.092+0000 7fd16affd700 1 -- 172.21.15.29:0/4154337527 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 9) ==== 46119+0+0 (secure 0 0 0) 0x7fd17001aa50 con 0x7fd174108790 2024-03-16T11:41:22.968 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:22.093+0000 7fd17bd0f700 1 -- 172.21.15.29:0/4154337527 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd174063900 con 0x7fd174108790 2024-03-16T11:41:22.968 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:22.093+0000 7fd16affd700 1 --2- 172.21.15.29:0/4154337527 >> [v2:172.21.15.29:6800/2798831128,v1:172.21.15.29:6801/2798831128] conn(0x7fd160038f20 0x7fd16003b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:41:22.968 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:22.093+0000 7fd16affd700 1 -- 172.21.15.29:0/4154337527 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fd17004d7b0 con 0x7fd174108790 2024-03-16T11:41:22.968 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:22.093+0000 7fd1792aa700 1 -- 172.21.15.29:0/4154337527 >> [v2:172.21.15.29:6800/2798831128,v1:172.21.15.29:6801/2798831128] conn(0x7fd160038f20 msgr2=0x7fd16003b3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6800/2798831128 2024-03-16T11:41:22.968 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:22.093+0000 7fd1792aa700 1 --2- 172.21.15.29:0/4154337527 >> [v2:172.21.15.29:6800/2798831128,v1:172.21.15.29:6801/2798831128] conn(0x7fd160038f20 0x7fd16003b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-16T11:41:22.968 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:22.098+0000 7fd16affd700 1 -- 172.21.15.29:0/4154337527 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fd170011e20 con 0x7fd174108790 2024-03-16T11:41:22.968 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:22.294+0000 7fd1792aa700 1 -- 172.21.15.29:0/4154337527 >> [v2:172.21.15.29:6800/2798831128,v1:172.21.15.29:6801/2798831128] conn(0x7fd160038f20 msgr2=0x7fd16003b3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6800/2798831128 2024-03-16T11:41:22.968 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:22.294+0000 7fd1792aa700 1 --2- 172.21.15.29:0/4154337527 >> [v2:172.21.15.29:6800/2798831128,v1:172.21.15.29:6801/2798831128] conn(0x7fd160038f20 0x7fd16003b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-16T11:41:22.968 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:22.364+0000 7fd17bd0f700 1 -- 172.21.15.29:0/4154337527 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7fd1741074a0 con 0x7fd174108790 2024-03-16T11:41:22.968 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:22.364+0000 7fd16affd700 1 -- 172.21.15.29:0/4154337527 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) ==== 56+0+88 (secure 0 0 0) 0x7fd17000ba30 con 0x7fd174108790 2024-03-16T11:41:22.968 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:22.366+0000 7fd17bd0f700 1 -- 172.21.15.29:0/4154337527 >> [v2:172.21.15.29:6800/2798831128,v1:172.21.15.29:6801/2798831128] conn(0x7fd160038f20 msgr2=0x7fd16003b3e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-16T11:41:22.968 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:22.366+0000 7fd17bd0f700 1 --2- 172.21.15.29:0/4154337527 >> [v2:172.21.15.29:6800/2798831128,v1:172.21.15.29:6801/2798831128] conn(0x7fd160038f20 0x7fd16003b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:41:22.969 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:22.366+0000 7fd17bd0f700 1 -- 172.21.15.29:0/4154337527 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd174108790 msgr2=0x7fd174108010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:41:22.969 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:22.366+0000 7fd17bd0f700 1 --2- 172.21.15.29:0/4154337527 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd174108790 0x7fd174108010 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fd1700116a0 tx=0x7fd170011ba0 comp rx=0 tx=0).stop 2024-03-16T11:41:22.969 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:22.366+0000 7fd17bd0f700 1 -- 172.21.15.29:0/4154337527 shutdown_connections 2024-03-16T11:41:22.969 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:22.366+0000 7fd17bd0f700 1 --2- 172.21.15.29:0/4154337527 >> [v2:172.21.15.29:6800/2798831128,v1:172.21.15.29:6801/2798831128] conn(0x7fd160038f20 0x7fd16003b3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:41:22.969 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:22.366+0000 7fd17bd0f700 1 --2- 172.21.15.29:0/4154337527 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd174108790 0x7fd174108010 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:41:22.969 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:22.366+0000 7fd17bd0f700 1 -- 172.21.15.29:0/4154337527 >> 172.21.15.29:0/4154337527 conn(0x7fd1741021c0 msgr2=0x7fd174102cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:41:22.969 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:22.367+0000 7fd17bd0f700 1 -- 172.21.15.29:0/4154337527 shutdown_connections 2024-03-16T11:41:22.969 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:22.367+0000 7fd17bd0f700 1 -- 172.21.15.29:0/4154337527 wait complete. 2024-03-16T11:41:22.969 INFO:teuthology.orchestra.run.smithi029.stdout:Waiting for the mgr to restart... 2024-03-16T11:41:22.969 INFO:teuthology.orchestra.run.smithi029.stdout:Waiting for mgr epoch 9... 2024-03-16T11:41:23.206 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 16 11:41:23 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-x[32210]: 2024-03-16T11:41:23.037+0000 7f7ea1243200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-16T11:41:24.338 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 16 11:41:24 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-x[32210]: 2024-03-16T11:41:24.028+0000 7f7ea1243200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-16T11:41:24.339 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 16 11:41:24 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-x[32210]: 2024-03-16T11:41:24.162+0000 7f7ea1243200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-16T11:41:24.791 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 16 11:41:24 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-x[32210]: 2024-03-16T11:41:24.499+0000 7f7ea1243200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-16T11:41:24.791 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 16 11:41:24 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-x[32210]: 2024-03-16T11:41:24.591+0000 7f7ea1243200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-16T11:41:24.791 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 16 11:41:24 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-x[32210]: 2024-03-16T11:41:24.792+0000 7f7ea1243200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-16T11:41:25.088 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 16 11:41:24 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-x[32210]: 2024-03-16T11:41:24.982+0000 7f7ea1243200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-16T11:41:25.533 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 16 11:41:25 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-x[32210]: 2024-03-16T11:41:25.273+0000 7f7ea1243200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-16T11:41:25.533 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 16 11:41:25 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-x[32210]: 2024-03-16T11:41:25.533+0000 7f7ea1243200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-16T11:41:25.838 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 16 11:41:25 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-x[32210]: 2024-03-16T11:41:25.775+0000 7f7ea1243200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-16T11:41:26.739 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 16 11:41:26 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-x[32210]: 2024-03-16T11:41:26.600+0000 7f7ea1243200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-16T11:41:27.294 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 16 11:41:27 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-x[32210]: 2024-03-16T11:41:27.192+0000 7f7ea1243200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-16T11:41:27.294 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 16 11:41:27 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-x[32210]: 2024-03-16T11:41:27.294+0000 7f7ea1243200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-16T11:41:27.581 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 16 11:41:27 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-x[32210]: 2024-03-16T11:41:27.517+0000 7f7ea1243200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-16T11:41:27.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:41:27 smithi029 ceph-mon[31975]: Active manager daemon x restarted 2024-03-16T11:41:27.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:41:27 smithi029 ceph-mon[31975]: Activating manager daemon x 2024-03-16T11:41:27.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:41:27 smithi029 ceph-mon[31975]: osdmap e3: 0 total, 0 up, 0 in 2024-03-16T11:41:27.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:41:27 smithi029 ceph-mon[31975]: mgrmap e10: x(active, starting, since 0.00356905s) 2024-03-16T11:41:27.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:41:27 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-16T11:41:27.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:41:27 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-16T11:41:27.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:41:27 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-16T11:41:27.839 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:41:27 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-16T11:41:27.839 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:41:27 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-16T11:41:27.839 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:41:27 smithi029 ceph-mon[31975]: Manager daemon x is now available 2024-03-16T11:41:28.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:41:28 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T11:41:28.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:41:28 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T11:41:28.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:41:28 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-16T11:41:28.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:41:28 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-16T11:41:28.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:41:28 smithi029 ceph-mon[31975]: [16/Mar/2024:11:41:28] ENGINE Bus STARTING 2024-03-16T11:41:28.839 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:41:28 smithi029 ceph-mon[31975]: [16/Mar/2024:11:41:28] ENGINE Serving on http://172.21.15.29:8765 2024-03-16T11:41:28.839 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:41:28 smithi029 ceph-mon[31975]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-16T11:41:28.839 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:41:28 smithi029 ceph-mon[31975]: mgrmap e11: x(active, since 1.00567s) 2024-03-16T11:41:28.839 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:41:28 smithi029 ceph-mon[31975]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-16T11:41:29.464 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout { 2024-03-16T11:41:29.464 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-03-16T11:41:29.464 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-16T11:41:29.464 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout } 2024-03-16T11:41:29.464 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:23.402+0000 7f9ce091e700 1 Processor -- start 2024-03-16T11:41:29.464 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:23.406+0000 7f9ce091e700 1 -- start start 2024-03-16T11:41:29.464 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:23.406+0000 7f9ce091e700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9cdc106c00 0x7f9cdc107020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:41:29.464 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:23.406+0000 7f9ce091e700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f9cdc1075f0 con 0x7f9cdc106c00 2024-03-16T11:41:29.464 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:23.406+0000 7f9cda59c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9cdc106c00 0x7f9cdc107020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:41:29.465 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:23.406+0000 7f9cda59c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9cdc106c00 0x7f9cdc107020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47682/0 (socket says 172.21.15.29:47682) 2024-03-16T11:41:29.465 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:23.406+0000 7f9cda59c700 1 -- 172.21.15.29:0/1632480736 learned_addr learned my addr 172.21.15.29:0/1632480736 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:41:29.465 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:23.412+0000 7f9cda59c700 1 -- 172.21.15.29:0/1632480736 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9cdc107e00 con 0x7f9cdc106c00 2024-03-16T11:41:29.465 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:23.417+0000 7f9cda59c700 1 --2- 172.21.15.29:0/1632480736 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9cdc106c00 0x7f9cdc107020 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f9ccc0098a0 tx=0x7f9ccc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d945e5bf12c89591 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:41:29.465 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:23.419+0000 7f9cd959a700 1 -- 172.21.15.29:0/1632480736 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9ccc004160 con 0x7f9cdc106c00 2024-03-16T11:41:29.465 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:23.420+0000 7f9cd959a700 1 -- 172.21.15.29:0/1632480736 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9ccc004300 con 0x7f9cdc106c00 2024-03-16T11:41:29.465 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:23.420+0000 7f9cd959a700 1 -- 172.21.15.29:0/1632480736 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9ccc011ad0 con 0x7f9cdc106c00 2024-03-16T11:41:29.465 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:23.421+0000 7f9ce091e700 1 -- 172.21.15.29:0/1632480736 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9cdc106c00 msgr2=0x7f9cdc107020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:41:29.465 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:23.421+0000 7f9ce091e700 1 --2- 172.21.15.29:0/1632480736 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9cdc106c00 0x7f9cdc107020 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f9ccc0098a0 tx=0x7f9ccc009bb0 comp rx=0 tx=0).stop 2024-03-16T11:41:29.465 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:23.423+0000 7f9ce091e700 1 -- 172.21.15.29:0/1632480736 shutdown_connections 2024-03-16T11:41:29.465 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:23.423+0000 7f9ce091e700 1 --2- 172.21.15.29:0/1632480736 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9cdc106c00 0x7f9cdc107020 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:41:29.465 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:23.423+0000 7f9ce091e700 1 -- 172.21.15.29:0/1632480736 >> 172.21.15.29:0/1632480736 conn(0x7f9cdc1021a0 msgr2=0x7f9cdc1045e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:41:29.465 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:23.426+0000 7f9ce091e700 1 -- 172.21.15.29:0/1632480736 shutdown_connections 2024-03-16T11:41:29.465 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:23.430+0000 7f9ce091e700 1 -- 172.21.15.29:0/1632480736 wait complete. 2024-03-16T11:41:29.465 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:23.430+0000 7f9ce091e700 1 Processor -- start 2024-03-16T11:41:29.466 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:23.431+0000 7f9ce091e700 1 -- start start 2024-03-16T11:41:29.466 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:23.431+0000 7f9ce091e700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9cdc106c00 0x7f9cdc199a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:41:29.466 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:23.431+0000 7f9ce091e700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f9cdc10b1b0 con 0x7f9cdc106c00 2024-03-16T11:41:29.466 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:23.437+0000 7f9cda59c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9cdc106c00 0x7f9cdc199a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:41:29.466 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:23.437+0000 7f9cda59c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9cdc106c00 0x7f9cdc199a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47686/0 (socket says 172.21.15.29:47686) 2024-03-16T11:41:29.466 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:23.437+0000 7f9cda59c700 1 -- 172.21.15.29:0/2080553123 learned_addr learned my addr 172.21.15.29:0/2080553123 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:41:29.466 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:23.437+0000 7f9cda59c700 1 -- 172.21.15.29:0/2080553123 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9cdc199fa0 con 0x7f9cdc106c00 2024-03-16T11:41:29.466 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:23.437+0000 7f9cda59c700 1 --2- 172.21.15.29:0/2080553123 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9cdc106c00 0x7f9cdc199a60 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f9ccc000c00 tx=0x7f9ccc011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:41:29.466 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:23.439+0000 7f9ccb7fe700 1 -- 172.21.15.29:0/2080553123 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9ccc003a70 con 0x7f9cdc106c00 2024-03-16T11:41:29.466 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:23.439+0000 7f9ccb7fe700 1 -- 172.21.15.29:0/2080553123 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9ccc003c10 con 0x7f9cdc106c00 2024-03-16T11:41:29.466 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:23.439+0000 7f9ccb7fe700 1 -- 172.21.15.29:0/2080553123 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9ccc01a820 con 0x7f9cdc106c00 2024-03-16T11:41:29.466 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:23.439+0000 7f9ce091e700 1 -- 172.21.15.29:0/2080553123 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9cdc19a230 con 0x7f9cdc106c00 2024-03-16T11:41:29.466 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:23.439+0000 7f9ce091e700 1 -- 172.21.15.29:0/2080553123 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9cdc19a690 con 0x7f9cdc106c00 2024-03-16T11:41:29.466 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:23.451+0000 7f9ccb7fe700 1 -- 172.21.15.29:0/2080553123 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 9) ==== 46119+0+0 (secure 0 0 0) 0x7f9ccc024b90 con 0x7f9cdc106c00 2024-03-16T11:41:29.466 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:23.451+0000 7f9ccb7fe700 1 --2- 172.21.15.29:0/2080553123 >> [v2:172.21.15.29:6800/2798831128,v1:172.21.15.29:6801/2798831128] conn(0x7f9cc4039070 0x7f9cc403b530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:41:29.467 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:23.451+0000 7f9ccb7fe700 1 -- 172.21.15.29:0/2080553123 --> [v2:172.21.15.29:6800/2798831128,v1:172.21.15.29:6801/2798831128] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f9cc403bc40 con 0x7f9cc4039070 2024-03-16T11:41:29.467 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:23.451+0000 7f9ccb7fe700 1 -- 172.21.15.29:0/2080553123 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f9ccc04d380 con 0x7f9cdc106c00 2024-03-16T11:41:29.467 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:23.451+0000 7f9cd9d9b700 1 -- 172.21.15.29:0/2080553123 >> [v2:172.21.15.29:6800/2798831128,v1:172.21.15.29:6801/2798831128] conn(0x7f9cc4039070 msgr2=0x7f9cc403b530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6800/2798831128 2024-03-16T11:41:29.467 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:23.451+0000 7f9cd9d9b700 1 --2- 172.21.15.29:0/2080553123 >> [v2:172.21.15.29:6800/2798831128,v1:172.21.15.29:6801/2798831128] conn(0x7f9cc4039070 0x7f9cc403b530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-16T11:41:29.467 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:23.652+0000 7f9cd9d9b700 1 -- 172.21.15.29:0/2080553123 >> [v2:172.21.15.29:6800/2798831128,v1:172.21.15.29:6801/2798831128] conn(0x7f9cc4039070 msgr2=0x7f9cc403b530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6800/2798831128 2024-03-16T11:41:29.467 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:23.652+0000 7f9cd9d9b700 1 --2- 172.21.15.29:0/2080553123 >> [v2:172.21.15.29:6800/2798831128,v1:172.21.15.29:6801/2798831128] conn(0x7f9cc4039070 0x7f9cc403b530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-16T11:41:29.467 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:24.053+0000 7f9cd9d9b700 1 -- 172.21.15.29:0/2080553123 >> [v2:172.21.15.29:6800/2798831128,v1:172.21.15.29:6801/2798831128] conn(0x7f9cc4039070 msgr2=0x7f9cc403b530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6800/2798831128 2024-03-16T11:41:29.467 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:24.053+0000 7f9cd9d9b700 1 --2- 172.21.15.29:0/2080553123 >> [v2:172.21.15.29:6800/2798831128,v1:172.21.15.29:6801/2798831128] conn(0x7f9cc4039070 0x7f9cc403b530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-16T11:41:29.467 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:24.854+0000 7f9cd9d9b700 1 -- 172.21.15.29:0/2080553123 >> [v2:172.21.15.29:6800/2798831128,v1:172.21.15.29:6801/2798831128] conn(0x7f9cc4039070 msgr2=0x7f9cc403b530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6800/2798831128 2024-03-16T11:41:29.467 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:24.854+0000 7f9cd9d9b700 1 --2- 172.21.15.29:0/2080553123 >> [v2:172.21.15.29:6800/2798831128,v1:172.21.15.29:6801/2798831128] conn(0x7f9cc4039070 0x7f9cc403b530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-16T11:41:29.467 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:26.455+0000 7f9cd9d9b700 1 -- 172.21.15.29:0/2080553123 >> [v2:172.21.15.29:6800/2798831128,v1:172.21.15.29:6801/2798831128] conn(0x7f9cc4039070 msgr2=0x7f9cc403b530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6800/2798831128 2024-03-16T11:41:29.467 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:26.455+0000 7f9cd9d9b700 1 --2- 172.21.15.29:0/2080553123 >> [v2:172.21.15.29:6800/2798831128,v1:172.21.15.29:6801/2798831128] conn(0x7f9cc4039070 0x7f9cc403b530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-16T11:41:29.467 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:27.533+0000 7f9ccb7fe700 1 -- 172.21.15.29:0/2080553123 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mgrmap(e 10) ==== 45886+0+0 (secure 0 0 0) 0x7f9ccc024440 con 0x7f9cdc106c00 2024-03-16T11:41:29.467 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:27.533+0000 7f9ccb7fe700 1 -- 172.21.15.29:0/2080553123 >> [v2:172.21.15.29:6800/2798831128,v1:172.21.15.29:6801/2798831128] conn(0x7f9cc4039070 msgr2=0x7f9cc403b530 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-16T11:41:29.468 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:27.533+0000 7f9ccb7fe700 1 --2- 172.21.15.29:0/2080553123 >> [v2:172.21.15.29:6800/2798831128,v1:172.21.15.29:6801/2798831128] conn(0x7f9cc4039070 0x7f9cc403b530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:41:29.468 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:28.534+0000 7f9ccb7fe700 1 -- 172.21.15.29:0/2080553123 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mgrmap(e 11) ==== 46013+0+0 (secure 0 0 0) 0x7f9ccc04ece0 con 0x7f9cdc106c00 2024-03-16T11:41:29.468 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:28.534+0000 7f9ccb7fe700 1 --2- 172.21.15.29:0/2080553123 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f9cc403cb40 0x7f9cc403ef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:41:29.468 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:28.535+0000 7f9ccb7fe700 1 -- 172.21.15.29:0/2080553123 --> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f9ccc0463e0 con 0x7f9cc403cb40 2024-03-16T11:41:29.468 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:28.535+0000 7f9cd9d9b700 1 --2- 172.21.15.29:0/2080553123 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f9cc403cb40 0x7f9cc403ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:41:29.468 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:28.535+0000 7f9cd9d9b700 1 --2- 172.21.15.29:0/2080553123 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f9cc403cb40 0x7f9cc403ef30 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f9cd0003a10 tx=0x7f9cd00092b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:41:29.468 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:28.536+0000 7f9ccb7fe700 1 -- 172.21.15.29:0/2080553123 <== mgr.14152 v2:172.21.15.29:6800/4192064640 1 ==== command_reply(tid 0: 0 ) ==== 8+0+7774 (secure 0 0 0) 0x7f9ccc0463e0 con 0x7f9cc403cb40 2024-03-16T11:41:29.468 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:28.542+0000 7f9cc97fa700 1 -- 172.21.15.29:0/2080553123 --> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7f9cbc002600 con 0x7f9cc403cb40 2024-03-16T11:41:29.468 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:28.543+0000 7f9ccb7fe700 1 -- 172.21.15.29:0/2080553123 <== mgr.14152 v2:172.21.15.29:6800/4192064640 2 ==== command_reply(tid 1: 0 ) ==== 8+0+52 (secure 0 0 0) 0x7f9cbc002600 con 0x7f9cc403cb40 2024-03-16T11:41:29.468 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:28.543+0000 7f9cc97fa700 1 -- 172.21.15.29:0/2080553123 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f9cc403cb40 msgr2=0x7f9cc403ef30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:41:29.468 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:28.543+0000 7f9cc97fa700 1 --2- 172.21.15.29:0/2080553123 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f9cc403cb40 0x7f9cc403ef30 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f9cd0003a10 tx=0x7f9cd00092b0 comp rx=0 tx=0).stop 2024-03-16T11:41:29.468 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:28.543+0000 7f9cc97fa700 1 -- 172.21.15.29:0/2080553123 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9cdc106c00 msgr2=0x7f9cdc199a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:41:29.468 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:28.543+0000 7f9cc97fa700 1 --2- 172.21.15.29:0/2080553123 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9cdc106c00 0x7f9cdc199a60 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f9ccc000c00 tx=0x7f9ccc011c10 comp rx=0 tx=0).stop 2024-03-16T11:41:29.468 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:28.543+0000 7f9cc97fa700 1 -- 172.21.15.29:0/2080553123 shutdown_connections 2024-03-16T11:41:29.468 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:28.543+0000 7f9cc97fa700 1 --2- 172.21.15.29:0/2080553123 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9cdc106c00 0x7f9cdc199a60 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:41:29.469 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:28.543+0000 7f9cc97fa700 1 --2- 172.21.15.29:0/2080553123 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f9cc403cb40 0x7f9cc403ef30 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:41:29.469 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:28.543+0000 7f9cc97fa700 1 --2- 172.21.15.29:0/2080553123 >> [v2:172.21.15.29:6800/2798831128,v1:172.21.15.29:6801/2798831128] conn(0x7f9cc4039070 0x7f9cc403b530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:41:29.469 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:28.543+0000 7f9cc97fa700 1 -- 172.21.15.29:0/2080553123 >> 172.21.15.29:0/2080553123 conn(0x7f9cdc1021a0 msgr2=0x7f9cdc103000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:41:29.469 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:28.543+0000 7f9cc97fa700 1 -- 172.21.15.29:0/2080553123 shutdown_connections 2024-03-16T11:41:29.469 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:28.543+0000 7f9cc97fa700 1 -- 172.21.15.29:0/2080553123 wait complete. 2024-03-16T11:41:29.469 INFO:teuthology.orchestra.run.smithi029.stdout:mgr epoch 9 is available 2024-03-16T11:41:29.469 INFO:teuthology.orchestra.run.smithi029.stdout:Generating a dashboard self-signed certificate... 2024-03-16T11:41:30.289 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:41:30 smithi029 ceph-mon[31975]: [16/Mar/2024:11:41:28] ENGINE Serving on https://172.21.15.29:7150 2024-03-16T11:41:30.290 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:41:30 smithi029 ceph-mon[31975]: [16/Mar/2024:11:41:28] ENGINE Bus STARTED 2024-03-16T11:41:30.290 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:41:30 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:41:30.859 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-03-16T11:41:30.859 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:29.910+0000 7f53d3246700 1 Processor -- start 2024-03-16T11:41:30.859 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:29.910+0000 7f53d3246700 1 -- start start 2024-03-16T11:41:30.859 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:29.910+0000 7f53d3246700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f53cc108770 0x7f53cc10ab60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:41:30.860 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:29.910+0000 7f53d3246700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f53cc07be70 con 0x7f53cc108770 2024-03-16T11:41:30.860 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:29.911+0000 7f53d0fe2700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f53cc108770 0x7f53cc10ab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:41:30.860 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:29.911+0000 7f53d0fe2700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f53cc108770 0x7f53cc10ab60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59958/0 (socket says 172.21.15.29:59958) 2024-03-16T11:41:30.860 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:29.911+0000 7f53d0fe2700 1 -- 172.21.15.29:0/3852552777 learned_addr learned my addr 172.21.15.29:0/3852552777 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:41:30.860 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:29.911+0000 7f53d0fe2700 1 -- 172.21.15.29:0/3852552777 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53cc10b0a0 con 0x7f53cc108770 2024-03-16T11:41:30.860 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:29.911+0000 7f53d0fe2700 1 --2- 172.21.15.29:0/3852552777 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f53cc108770 0x7f53cc10ab60 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f53c4009e80 tx=0x7f53c400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb1c52f674a7e20e server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:41:30.860 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:29.911+0000 7f53cb7fe700 1 -- 172.21.15.29:0/3852552777 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f53c4017070 con 0x7f53cc108770 2024-03-16T11:41:30.860 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:29.911+0000 7f53cb7fe700 1 -- 172.21.15.29:0/3852552777 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f53c400b8c0 con 0x7f53cc108770 2024-03-16T11:41:30.860 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:29.911+0000 7f53cb7fe700 1 -- 172.21.15.29:0/3852552777 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f53c40045c0 con 0x7f53cc108770 2024-03-16T11:41:30.860 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:29.912+0000 7f53d3246700 1 -- 172.21.15.29:0/3852552777 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f53cc108770 msgr2=0x7f53cc10ab60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:41:30.860 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:29.912+0000 7f53d3246700 1 --2- 172.21.15.29:0/3852552777 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f53cc108770 0x7f53cc10ab60 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f53c4009e80 tx=0x7f53c400b330 comp rx=0 tx=0).stop 2024-03-16T11:41:30.860 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:29.912+0000 7f53d3246700 1 -- 172.21.15.29:0/3852552777 shutdown_connections 2024-03-16T11:41:30.860 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:29.912+0000 7f53d3246700 1 --2- 172.21.15.29:0/3852552777 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f53cc108770 0x7f53cc10ab60 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:41:30.861 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:29.912+0000 7f53d3246700 1 -- 172.21.15.29:0/3852552777 >> 172.21.15.29:0/3852552777 conn(0x7f53cc1021a0 msgr2=0x7f53cc1045e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:41:30.861 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:29.912+0000 7f53d3246700 1 -- 172.21.15.29:0/3852552777 shutdown_connections 2024-03-16T11:41:30.861 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:29.912+0000 7f53d3246700 1 -- 172.21.15.29:0/3852552777 wait complete. 2024-03-16T11:41:30.861 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:29.912+0000 7f53d3246700 1 Processor -- start 2024-03-16T11:41:30.861 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:29.912+0000 7f53d3246700 1 -- start start 2024-03-16T11:41:30.861 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:29.913+0000 7f53d3246700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f53cc108770 0x7f53cc19bc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:41:30.861 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:29.913+0000 7f53d3246700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f53cc10c110 con 0x7f53cc108770 2024-03-16T11:41:30.861 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:29.913+0000 7f53d0fe2700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f53cc108770 0x7f53cc19bc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:41:30.861 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:29.913+0000 7f53d0fe2700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f53cc108770 0x7f53cc19bc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59964/0 (socket says 172.21.15.29:59964) 2024-03-16T11:41:30.861 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:29.913+0000 7f53d0fe2700 1 -- 172.21.15.29:0/2205028590 learned_addr learned my addr 172.21.15.29:0/2205028590 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:41:30.861 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:29.913+0000 7f53d0fe2700 1 -- 172.21.15.29:0/2205028590 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53cc19c1d0 con 0x7f53cc108770 2024-03-16T11:41:30.861 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:29.913+0000 7f53d0fe2700 1 --2- 172.21.15.29:0/2205028590 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f53cc108770 0x7f53cc19bc90 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f53c4009530 tx=0x7f53c4003d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:41:30.861 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:29.913+0000 7f53c9ffb700 1 -- 172.21.15.29:0/2205028590 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f53c4020020 con 0x7f53cc108770 2024-03-16T11:41:30.861 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:29.913+0000 7f53c9ffb700 1 -- 172.21.15.29:0/2205028590 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f53c400bba0 con 0x7f53cc108770 2024-03-16T11:41:30.861 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:29.913+0000 7f53c9ffb700 1 -- 172.21.15.29:0/2205028590 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f53c4017040 con 0x7f53cc108770 2024-03-16T11:41:30.861 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:29.913+0000 7f53d3246700 1 -- 172.21.15.29:0/2205028590 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f53cc19c460 con 0x7f53cc108770 2024-03-16T11:41:30.862 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:29.914+0000 7f53d3246700 1 -- 172.21.15.29:0/2205028590 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f53cc19f150 con 0x7f53cc108770 2024-03-16T11:41:30.862 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:29.914+0000 7f53c9ffb700 1 -- 172.21.15.29:0/2205028590 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 11) ==== 46013+0+0 (secure 0 0 0) 0x7f53c4023020 con 0x7f53cc108770 2024-03-16T11:41:30.862 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:29.914+0000 7f53d3246700 1 -- 172.21.15.29:0/2205028590 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f53cc063900 con 0x7f53cc108770 2024-03-16T11:41:30.862 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:29.914+0000 7f53c9ffb700 1 --2- 172.21.15.29:0/2205028590 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f53b4038ec0 0x7f53b403b380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:41:30.862 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:29.915+0000 7f53c9ffb700 1 -- 172.21.15.29:0/2205028590 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f53c400f050 con 0x7f53cc108770 2024-03-16T11:41:30.862 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:29.915+0000 7f53cbfff700 1 --2- 172.21.15.29:0/2205028590 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f53b4038ec0 0x7f53b403b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:41:30.862 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:29.915+0000 7f53cbfff700 1 --2- 172.21.15.29:0/2205028590 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f53b4038ec0 0x7f53b403b380 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f53c0006fd0 tx=0x7f53c0006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:41:30.862 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:29.920+0000 7f53c9ffb700 1 -- 172.21.15.29:0/2205028590 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f53c402a020 con 0x7f53cc108770 2024-03-16T11:41:30.862 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:30.116+0000 7f53c9ffb700 1 -- 172.21.15.29:0/2205028590 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mgrmap(e 12) ==== 46119+0+0 (secure 0 0 0) 0x7f53c4012430 con 0x7f53cc108770 2024-03-16T11:41:30.862 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:30.133+0000 7f53d3246700 1 -- 172.21.15.29:0/2205028590 --> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) -- 0x7f53cc1033f0 con 0x7f53b4038ec0 2024-03-16T11:41:30.862 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:30.290+0000 7f53c9ffb700 1 -- 172.21.15.29:0/2205028590 <== mgr.14152 v2:172.21.15.29:6800/4192064640 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+32 (secure 0 0 0) 0x7f53cc1033f0 con 0x7f53b4038ec0 2024-03-16T11:41:30.862 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:30.292+0000 7f53d3246700 1 -- 172.21.15.29:0/2205028590 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f53b4038ec0 msgr2=0x7f53b403b380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:41:30.862 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:30.292+0000 7f53d3246700 1 --2- 172.21.15.29:0/2205028590 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f53b4038ec0 0x7f53b403b380 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f53c0006fd0 tx=0x7f53c0006e40 comp rx=0 tx=0).stop 2024-03-16T11:41:30.862 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:30.292+0000 7f53d3246700 1 -- 172.21.15.29:0/2205028590 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f53cc108770 msgr2=0x7f53cc19bc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:41:30.863 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:30.292+0000 7f53d3246700 1 --2- 172.21.15.29:0/2205028590 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f53cc108770 0x7f53cc19bc90 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f53c4009530 tx=0x7f53c4003d00 comp rx=0 tx=0).stop 2024-03-16T11:41:30.863 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:30.293+0000 7f53d3246700 1 -- 172.21.15.29:0/2205028590 shutdown_connections 2024-03-16T11:41:30.863 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:30.293+0000 7f53d3246700 1 --2- 172.21.15.29:0/2205028590 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f53b4038ec0 0x7f53b403b380 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:41:30.863 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:30.293+0000 7f53d3246700 1 --2- 172.21.15.29:0/2205028590 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f53cc108770 0x7f53cc19bc90 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:41:30.863 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:30.293+0000 7f53d3246700 1 -- 172.21.15.29:0/2205028590 >> 172.21.15.29:0/2205028590 conn(0x7f53cc1021a0 msgr2=0x7f53cc102ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:41:30.863 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:30.293+0000 7f53d3246700 1 -- 172.21.15.29:0/2205028590 shutdown_connections 2024-03-16T11:41:30.863 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:30.293+0000 7f53d3246700 1 -- 172.21.15.29:0/2205028590 wait complete. 2024-03-16T11:41:30.863 INFO:teuthology.orchestra.run.smithi029.stdout:Creating initial admin user... 2024-03-16T11:41:31.309 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:41:31 smithi029 ceph-mon[31975]: mgrmap e12: x(active, since 2s) 2024-03-16T11:41:31.309 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:41:31 smithi029 ceph-mon[31975]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T11:41:31.309 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:41:31 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:41:31.309 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:41:31 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:41:32.389 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$w1eiijZEL3kX7/uQUhbxhOaVMtmLDfpytVmp09H3CuqjKt7IsKAe.", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1710589291, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-03-16T11:41:32.389 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:31.333+0000 7f927cc3d700 1 Processor -- start 2024-03-16T11:41:32.389 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:31.333+0000 7f927cc3d700 1 -- start start 2024-03-16T11:41:32.389 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:31.333+0000 7f927cc3d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9278106c00 0x7f9278107020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:41:32.389 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:31.333+0000 7f927cc3d700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f92781075f0 con 0x7f9278106c00 2024-03-16T11:41:32.389 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:31.334+0000 7f927659c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9278106c00 0x7f9278107020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:41:32.389 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:31.334+0000 7f927659c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9278106c00 0x7f9278107020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59976/0 (socket says 172.21.15.29:59976) 2024-03-16T11:41:32.389 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:31.334+0000 7f927659c700 1 -- 172.21.15.29:0/1303285822 learned_addr learned my addr 172.21.15.29:0/1303285822 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:41:32.389 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:31.334+0000 7f927659c700 1 -- 172.21.15.29:0/1303285822 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9278107e00 con 0x7f9278106c00 2024-03-16T11:41:32.390 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:31.334+0000 7f927659c700 1 --2- 172.21.15.29:0/1303285822 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9278106c00 0x7f9278107020 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f92680098a0 tx=0x7f9268009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec69375cadc21adf server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:41:32.390 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:31.334+0000 7f927559a700 1 -- 172.21.15.29:0/1303285822 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9268004160 con 0x7f9278106c00 2024-03-16T11:41:32.390 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:31.334+0000 7f927559a700 1 -- 172.21.15.29:0/1303285822 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9268004300 con 0x7f9278106c00 2024-03-16T11:41:32.390 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:31.334+0000 7f927559a700 1 -- 172.21.15.29:0/1303285822 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9268011b20 con 0x7f9278106c00 2024-03-16T11:41:32.390 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:31.335+0000 7f927cc3d700 1 -- 172.21.15.29:0/1303285822 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9278106c00 msgr2=0x7f9278107020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:41:32.390 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:31.335+0000 7f927cc3d700 1 --2- 172.21.15.29:0/1303285822 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9278106c00 0x7f9278107020 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f92680098a0 tx=0x7f9268009bb0 comp rx=0 tx=0).stop 2024-03-16T11:41:32.390 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:31.335+0000 7f927cc3d700 1 -- 172.21.15.29:0/1303285822 shutdown_connections 2024-03-16T11:41:32.390 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:31.335+0000 7f927cc3d700 1 --2- 172.21.15.29:0/1303285822 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9278106c00 0x7f9278107020 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:41:32.390 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:31.335+0000 7f927cc3d700 1 -- 172.21.15.29:0/1303285822 >> 172.21.15.29:0/1303285822 conn(0x7f92781021a0 msgr2=0x7f92781045e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:41:32.390 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:31.335+0000 7f927cc3d700 1 -- 172.21.15.29:0/1303285822 shutdown_connections 2024-03-16T11:41:32.390 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:31.335+0000 7f927cc3d700 1 -- 172.21.15.29:0/1303285822 wait complete. 2024-03-16T11:41:32.390 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:31.336+0000 7f927cc3d700 1 Processor -- start 2024-03-16T11:41:32.394 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:31.336+0000 7f927cc3d700 1 -- start start 2024-03-16T11:41:32.394 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:31.336+0000 7f927cc3d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9278106c00 0x7f927819bcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:41:32.394 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:31.336+0000 7f927cc3d700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f927810b1b0 con 0x7f9278106c00 2024-03-16T11:41:32.394 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:31.336+0000 7f927659c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9278106c00 0x7f927819bcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:41:32.394 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:31.336+0000 7f927659c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9278106c00 0x7f927819bcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59980/0 (socket says 172.21.15.29:59980) 2024-03-16T11:41:32.394 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:31.336+0000 7f927659c700 1 -- 172.21.15.29:0/3958510122 learned_addr learned my addr 172.21.15.29:0/3958510122 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:41:32.394 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:31.336+0000 7f927659c700 1 -- 172.21.15.29:0/3958510122 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f927819c1f0 con 0x7f9278106c00 2024-03-16T11:41:32.394 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:31.337+0000 7f927659c700 1 --2- 172.21.15.29:0/3958510122 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9278106c00 0x7f927819bcb0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f9268000c00 tx=0x7f9268011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:41:32.394 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:31.337+0000 7f92677fe700 1 -- 172.21.15.29:0/3958510122 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9268003b60 con 0x7f9278106c00 2024-03-16T11:41:32.394 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:31.337+0000 7f92677fe700 1 -- 172.21.15.29:0/3958510122 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f926801a910 con 0x7f9278106c00 2024-03-16T11:41:32.394 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:31.337+0000 7f927cc3d700 1 -- 172.21.15.29:0/3958510122 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f927819c480 con 0x7f9278106c00 2024-03-16T11:41:32.394 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:31.337+0000 7f92677fe700 1 -- 172.21.15.29:0/3958510122 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9268024930 con 0x7f9278106c00 2024-03-16T11:41:32.394 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:31.337+0000 7f927cc3d700 1 -- 172.21.15.29:0/3958510122 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f927819c8e0 con 0x7f9278106c00 2024-03-16T11:41:32.395 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:31.338+0000 7f92677fe700 1 -- 172.21.15.29:0/3958510122 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 12) ==== 46119+0+0 (secure 0 0 0) 0x7f9268022020 con 0x7f9278106c00 2024-03-16T11:41:32.395 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:31.338+0000 7f927cc3d700 1 -- 172.21.15.29:0/3958510122 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9278063900 con 0x7f9278106c00 2024-03-16T11:41:32.395 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:31.338+0000 7f92677fe700 1 --2- 172.21.15.29:0/3958510122 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f9260038f70 0x7f926003b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:41:32.395 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:31.338+0000 7f92677fe700 1 -- 172.21.15.29:0/3958510122 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f926804d340 con 0x7f9278106c00 2024-03-16T11:41:32.395 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:31.338+0000 7f9275d9b700 1 --2- 172.21.15.29:0/3958510122 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f9260038f70 0x7f926003b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:41:32.395 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:31.338+0000 7f9275d9b700 1 --2- 172.21.15.29:0/3958510122 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f9260038f70 0x7f926003b430 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f926c006fd0 tx=0x7f926c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:41:32.395 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:31.343+0000 7f92677fe700 1 -- 172.21.15.29:0/3958510122 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f926801ab30 con 0x7f9278106c00 2024-03-16T11:41:32.395 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:31.572+0000 7f927cc3d700 1 -- 172.21.15.29:0/3958510122 --> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) -- 0x7f9278195ce0 con 0x7f9260038f70 2024-03-16T11:41:32.395 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:31.819+0000 7f92677fe700 1 -- 172.21.15.29:0/3958510122 <== mgr.14152 v2:172.21.15.29:6800/4192064640 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+252 (secure 0 0 0) 0x7f9278195ce0 con 0x7f9260038f70 2024-03-16T11:41:32.395 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:31.821+0000 7f927cc3d700 1 -- 172.21.15.29:0/3958510122 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f9260038f70 msgr2=0x7f926003b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:41:32.395 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:31.821+0000 7f927cc3d700 1 --2- 172.21.15.29:0/3958510122 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f9260038f70 0x7f926003b430 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f926c006fd0 tx=0x7f926c006e40 comp rx=0 tx=0).stop 2024-03-16T11:41:32.395 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:31.821+0000 7f927cc3d700 1 -- 172.21.15.29:0/3958510122 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9278106c00 msgr2=0x7f927819bcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:41:32.395 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:31.821+0000 7f927cc3d700 1 --2- 172.21.15.29:0/3958510122 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9278106c00 0x7f927819bcb0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f9268000c00 tx=0x7f9268011c10 comp rx=0 tx=0).stop 2024-03-16T11:41:32.395 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:31.821+0000 7f927cc3d700 1 -- 172.21.15.29:0/3958510122 shutdown_connections 2024-03-16T11:41:32.396 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:31.821+0000 7f927cc3d700 1 --2- 172.21.15.29:0/3958510122 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f9260038f70 0x7f926003b430 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:41:32.396 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:31.821+0000 7f927cc3d700 1 --2- 172.21.15.29:0/3958510122 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9278106c00 0x7f927819bcb0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:41:32.396 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:31.821+0000 7f927cc3d700 1 -- 172.21.15.29:0/3958510122 >> 172.21.15.29:0/3958510122 conn(0x7f92781021a0 msgr2=0x7f9278103020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:41:32.396 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:31.821+0000 7f927cc3d700 1 -- 172.21.15.29:0/3958510122 shutdown_connections 2024-03-16T11:41:32.396 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:31.821+0000 7f927cc3d700 1 -- 172.21.15.29:0/3958510122 wait complete. 2024-03-16T11:41:32.396 INFO:teuthology.orchestra.run.smithi029.stdout:Fetching dashboard port number... 2024-03-16T11:41:32.706 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:41:32 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:41:32.706 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:41:32 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:41:33.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:41:33 smithi029 ceph-mon[31975]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T11:41:33.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:41:33 smithi029 ceph-mon[31975]: from='client.? 172.21.15.29:0/1214597056' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-03-16T11:41:34.571 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout 8443 2024-03-16T11:41:34.571 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:32.808+0000 7f6caf3e2700 1 Processor -- start 2024-03-16T11:41:34.571 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:32.808+0000 7f6caf3e2700 1 -- start start 2024-03-16T11:41:34.572 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:32.809+0000 7f6caf3e2700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ca8107ce0 0x7f6ca8108100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:41:34.572 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:32.809+0000 7f6caf3e2700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f6ca81086d0 con 0x7f6ca8107ce0 2024-03-16T11:41:34.572 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:32.809+0000 7f6cad17e700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ca8107ce0 0x7f6ca8108100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:41:34.572 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:32.809+0000 7f6cad17e700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ca8107ce0 0x7f6ca8108100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59994/0 (socket says 172.21.15.29:59994) 2024-03-16T11:41:34.572 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:32.809+0000 7f6cad17e700 1 -- 172.21.15.29:0/4087171010 learned_addr learned my addr 172.21.15.29:0/4087171010 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:41:34.572 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:32.809+0000 7f6cad17e700 1 -- 172.21.15.29:0/4087171010 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ca8108ee0 con 0x7f6ca8107ce0 2024-03-16T11:41:34.572 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:32.809+0000 7f6cad17e700 1 --2- 172.21.15.29:0/4087171010 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ca8107ce0 0x7f6ca8108100 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f6c980098a0 tx=0x7f6c98009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b0ccac3dd77533e server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:41:34.572 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:32.809+0000 7f6c9ffff700 1 -- 172.21.15.29:0/4087171010 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6c98004160 con 0x7f6ca8107ce0 2024-03-16T11:41:34.572 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:32.809+0000 7f6c9ffff700 1 -- 172.21.15.29:0/4087171010 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6c98004300 con 0x7f6ca8107ce0 2024-03-16T11:41:34.572 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:32.809+0000 7f6c9ffff700 1 -- 172.21.15.29:0/4087171010 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6c98011b20 con 0x7f6ca8107ce0 2024-03-16T11:41:34.572 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:32.810+0000 7f6caf3e2700 1 -- 172.21.15.29:0/4087171010 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ca8107ce0 msgr2=0x7f6ca8108100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:41:34.572 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:32.810+0000 7f6caf3e2700 1 --2- 172.21.15.29:0/4087171010 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ca8107ce0 0x7f6ca8108100 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f6c980098a0 tx=0x7f6c98009bb0 comp rx=0 tx=0).stop 2024-03-16T11:41:34.572 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:32.810+0000 7f6caf3e2700 1 -- 172.21.15.29:0/4087171010 shutdown_connections 2024-03-16T11:41:34.572 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:32.810+0000 7f6caf3e2700 1 --2- 172.21.15.29:0/4087171010 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ca8107ce0 0x7f6ca8108100 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:41:34.573 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:32.810+0000 7f6caf3e2700 1 -- 172.21.15.29:0/4087171010 >> 172.21.15.29:0/4087171010 conn(0x7f6ca81032a0 msgr2=0x7f6ca81056c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:41:34.573 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:32.810+0000 7f6caf3e2700 1 -- 172.21.15.29:0/4087171010 shutdown_connections 2024-03-16T11:41:34.573 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:32.810+0000 7f6caf3e2700 1 -- 172.21.15.29:0/4087171010 wait complete. 2024-03-16T11:41:34.573 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:32.811+0000 7f6caf3e2700 1 Processor -- start 2024-03-16T11:41:34.573 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:32.811+0000 7f6caf3e2700 1 -- start start 2024-03-16T11:41:34.574 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:32.811+0000 7f6caf3e2700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ca8107ce0 0x7f6ca819bc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:41:34.574 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:32.811+0000 7f6caf3e2700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f6ca810c290 con 0x7f6ca8107ce0 2024-03-16T11:41:34.574 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:32.811+0000 7f6cad17e700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ca8107ce0 0x7f6ca819bc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:41:34.574 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:32.811+0000 7f6cad17e700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ca8107ce0 0x7f6ca819bc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59998/0 (socket says 172.21.15.29:59998) 2024-03-16T11:41:34.574 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:32.811+0000 7f6cad17e700 1 -- 172.21.15.29:0/1214597056 learned_addr learned my addr 172.21.15.29:0/1214597056 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:41:34.574 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:32.811+0000 7f6cad17e700 1 -- 172.21.15.29:0/1214597056 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ca819c1b0 con 0x7f6ca8107ce0 2024-03-16T11:41:34.574 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:32.812+0000 7f6cad17e700 1 --2- 172.21.15.29:0/1214597056 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ca8107ce0 0x7f6ca819bc70 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f6c980116a0 tx=0x7f6c98011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:41:34.574 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:32.812+0000 7f6c9e7fc700 1 -- 172.21.15.29:0/1214597056 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6c98003a40 con 0x7f6ca8107ce0 2024-03-16T11:41:34.574 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:32.812+0000 7f6c9e7fc700 1 -- 172.21.15.29:0/1214597056 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6c98003be0 con 0x7f6ca8107ce0 2024-03-16T11:41:34.574 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:32.812+0000 7f6caf3e2700 1 -- 172.21.15.29:0/1214597056 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6ca819c440 con 0x7f6ca8107ce0 2024-03-16T11:41:34.575 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:32.812+0000 7f6c9e7fc700 1 -- 172.21.15.29:0/1214597056 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6c9801a720 con 0x7f6ca8107ce0 2024-03-16T11:41:34.575 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:32.812+0000 7f6caf3e2700 1 -- 172.21.15.29:0/1214597056 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6ca819c8a0 con 0x7f6ca8107ce0 2024-03-16T11:41:34.575 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:32.813+0000 7f6c9e7fc700 1 -- 172.21.15.29:0/1214597056 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 12) ==== 46119+0+0 (secure 0 0 0) 0x7f6c9801aaa0 con 0x7f6ca8107ce0 2024-03-16T11:41:34.575 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:32.813+0000 7f6caf3e2700 1 -- 172.21.15.29:0/1214597056 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6ca8063900 con 0x7f6ca8107ce0 2024-03-16T11:41:34.575 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:32.813+0000 7f6c9e7fc700 1 --2- 172.21.15.29:0/1214597056 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f6c94038f20 0x7f6c9403b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:41:34.575 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:32.813+0000 7f6c9e7fc700 1 -- 172.21.15.29:0/1214597056 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f6c9804d5d0 con 0x7f6ca8107ce0 2024-03-16T11:41:34.575 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:32.813+0000 7f6cac97d700 1 --2- 172.21.15.29:0/1214597056 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f6c94038f20 0x7f6c9403b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:41:34.575 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:32.814+0000 7f6cac97d700 1 --2- 172.21.15.29:0/1214597056 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f6c94038f20 0x7f6c9403b3e0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f6ca4006fd0 tx=0x7f6ca4006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:41:34.575 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:32.818+0000 7f6c9e7fc700 1 -- 172.21.15.29:0/1214597056 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f6c98011e20 con 0x7f6ca8107ce0 2024-03-16T11:41:34.575 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:33.051+0000 7f6caf3e2700 1 -- 172.21.15.29:0/1214597056 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) -- 0x7f6ca819f310 con 0x7f6ca8107ce0 2024-03-16T11:41:34.575 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:33.051+0000 7f6c9e7fc700 1 -- 172.21.15.29:0/1214597056 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) ==== 112+0+5 (secure 0 0 0) 0x7f6c9800bc30 con 0x7f6ca8107ce0 2024-03-16T11:41:34.575 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:33.053+0000 7f6caf3e2700 1 -- 172.21.15.29:0/1214597056 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f6c94038f20 msgr2=0x7f6c9403b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:41:34.575 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:33.053+0000 7f6caf3e2700 1 --2- 172.21.15.29:0/1214597056 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f6c94038f20 0x7f6c9403b3e0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f6ca4006fd0 tx=0x7f6ca4006e40 comp rx=0 tx=0).stop 2024-03-16T11:41:34.576 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:33.053+0000 7f6caf3e2700 1 -- 172.21.15.29:0/1214597056 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ca8107ce0 msgr2=0x7f6ca819bc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:41:34.576 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:33.053+0000 7f6caf3e2700 1 --2- 172.21.15.29:0/1214597056 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ca8107ce0 0x7f6ca819bc70 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f6c980116a0 tx=0x7f6c98011ba0 comp rx=0 tx=0).stop 2024-03-16T11:41:34.576 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:33.053+0000 7f6caf3e2700 1 -- 172.21.15.29:0/1214597056 shutdown_connections 2024-03-16T11:41:34.576 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:33.053+0000 7f6caf3e2700 1 --2- 172.21.15.29:0/1214597056 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f6c94038f20 0x7f6c9403b3e0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:41:34.576 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:33.053+0000 7f6caf3e2700 1 --2- 172.21.15.29:0/1214597056 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ca8107ce0 0x7f6ca819bc70 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:41:34.576 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:33.053+0000 7f6caf3e2700 1 -- 172.21.15.29:0/1214597056 >> 172.21.15.29:0/1214597056 conn(0x7f6ca81032a0 msgr2=0x7f6ca8103d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:41:34.576 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:33.053+0000 7f6caf3e2700 1 -- 172.21.15.29:0/1214597056 shutdown_connections 2024-03-16T11:41:34.576 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:33.054+0000 7f6caf3e2700 1 -- 172.21.15.29:0/1214597056 wait complete. 2024-03-16T11:41:34.579 INFO:teuthology.orchestra.run.smithi029.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-16T11:41:34.579 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl: stdout disabled 2024-03-16T11:41:34.587 INFO:teuthology.orchestra.run.smithi029.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-16T11:41:34.587 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl: stdout inactive 2024-03-16T11:41:34.587 INFO:teuthology.orchestra.run.smithi029.stdout:firewalld.service is not enabled 2024-03-16T11:41:34.587 INFO:teuthology.orchestra.run.smithi029.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-03-16T11:41:34.589 INFO:teuthology.orchestra.run.smithi029.stdout:Ceph Dashboard is now available at: 2024-03-16T11:41:34.589 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-16T11:41:34.589 INFO:teuthology.orchestra.run.smithi029.stdout: URL: https://smithi029.front.sepia.ceph.com:8443/ 2024-03-16T11:41:34.589 INFO:teuthology.orchestra.run.smithi029.stdout: User: admin 2024-03-16T11:41:34.589 INFO:teuthology.orchestra.run.smithi029.stdout: Password: gqyk6g857u 2024-03-16T11:41:34.590 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-16T11:41:34.590 INFO:teuthology.orchestra.run.smithi029.stdout:Saving cluster configuration to /var/lib/ceph/d1887f28-e389-11ee-95c9-87774f69a715/config directory 2024-03-16T11:41:34.590 INFO:teuthology.orchestra.run.smithi029.stdout:Enabling autotune for osd_memory_target 2024-03-16T11:41:35.541 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:41:35 smithi029 ceph-mon[31975]: mgrmap e13: x(active, since 6s) 2024-03-16T11:41:35.815 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:35.009+0000 7f48c3630700 1 Processor -- start 2024-03-16T11:41:35.815 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:35.009+0000 7f48c3630700 1 -- start start 2024-03-16T11:41:35.815 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:35.009+0000 7f48c3630700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f48bc108780 0x7f48bc10ab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:41:35.815 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:35.009+0000 7f48c3630700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f48bc07be70 con 0x7f48bc108780 2024-03-16T11:41:35.815 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:35.009+0000 7f48c13cc700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f48bc108780 0x7f48bc10ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:41:35.815 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:35.009+0000 7f48c13cc700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f48bc108780 0x7f48bc10ab70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60004/0 (socket says 172.21.15.29:60004) 2024-03-16T11:41:35.816 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:35.009+0000 7f48c13cc700 1 -- 172.21.15.29:0/1715203735 learned_addr learned my addr 172.21.15.29:0/1715203735 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:41:35.816 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:35.010+0000 7f48c13cc700 1 -- 172.21.15.29:0/1715203735 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48bc10b0b0 con 0x7f48bc108780 2024-03-16T11:41:35.816 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:35.010+0000 7f48c13cc700 1 --2- 172.21.15.29:0/1715203735 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f48bc108780 0x7f48bc10ab70 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f48b80098a0 tx=0x7f48b8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a15462a0d0f7b33 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:41:35.816 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:35.010+0000 7f48b3fff700 1 -- 172.21.15.29:0/1715203735 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f48b8004160 con 0x7f48bc108780 2024-03-16T11:41:35.816 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:35.010+0000 7f48b3fff700 1 -- 172.21.15.29:0/1715203735 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f48b8004300 con 0x7f48bc108780 2024-03-16T11:41:35.816 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:35.010+0000 7f48b3fff700 1 -- 172.21.15.29:0/1715203735 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f48b8011b20 con 0x7f48bc108780 2024-03-16T11:41:35.816 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:35.010+0000 7f48c3630700 1 -- 172.21.15.29:0/1715203735 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f48bc108780 msgr2=0x7f48bc10ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:41:35.816 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:35.010+0000 7f48c3630700 1 --2- 172.21.15.29:0/1715203735 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f48bc108780 0x7f48bc10ab70 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f48b80098a0 tx=0x7f48b8009bb0 comp rx=0 tx=0).stop 2024-03-16T11:41:35.816 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:35.011+0000 7f48c3630700 1 -- 172.21.15.29:0/1715203735 shutdown_connections 2024-03-16T11:41:35.816 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:35.011+0000 7f48c3630700 1 --2- 172.21.15.29:0/1715203735 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f48bc108780 0x7f48bc10ab70 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:41:35.816 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:35.011+0000 7f48c3630700 1 -- 172.21.15.29:0/1715203735 >> 172.21.15.29:0/1715203735 conn(0x7f48bc1021b0 msgr2=0x7f48bc1045f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:41:35.816 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:35.011+0000 7f48c3630700 1 -- 172.21.15.29:0/1715203735 shutdown_connections 2024-03-16T11:41:35.816 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:35.011+0000 7f48c3630700 1 -- 172.21.15.29:0/1715203735 wait complete. 2024-03-16T11:41:35.816 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:35.011+0000 7f48c3630700 1 Processor -- start 2024-03-16T11:41:35.816 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:35.011+0000 7f48c3630700 1 -- start start 2024-03-16T11:41:35.820 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:35.012+0000 7f48c3630700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f48bc108780 0x7f48bc19bca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:41:35.820 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:35.012+0000 7f48c3630700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f48bc10c120 con 0x7f48bc108780 2024-03-16T11:41:35.820 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:35.012+0000 7f48c13cc700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f48bc108780 0x7f48bc19bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:41:35.820 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:35.012+0000 7f48c13cc700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f48bc108780 0x7f48bc19bca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60006/0 (socket says 172.21.15.29:60006) 2024-03-16T11:41:35.820 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:35.012+0000 7f48c13cc700 1 -- 172.21.15.29:0/1683849538 learned_addr learned my addr 172.21.15.29:0/1683849538 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:41:35.820 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:35.012+0000 7f48c13cc700 1 -- 172.21.15.29:0/1683849538 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48bc19c1e0 con 0x7f48bc108780 2024-03-16T11:41:35.820 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:35.012+0000 7f48c13cc700 1 --2- 172.21.15.29:0/1683849538 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f48bc108780 0x7f48bc19bca0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f48b80116a0 tx=0x7f48b8011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:41:35.820 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:35.012+0000 7f48b27fc700 1 -- 172.21.15.29:0/1683849538 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f48b8003a40 con 0x7f48bc108780 2024-03-16T11:41:35.820 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:35.012+0000 7f48b27fc700 1 -- 172.21.15.29:0/1683849538 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f48b8003be0 con 0x7f48bc108780 2024-03-16T11:41:35.820 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:35.012+0000 7f48b27fc700 1 -- 172.21.15.29:0/1683849538 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f48b801a750 con 0x7f48bc108780 2024-03-16T11:41:35.820 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:35.012+0000 7f48c3630700 1 -- 172.21.15.29:0/1683849538 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f48bc19c470 con 0x7f48bc108780 2024-03-16T11:41:35.820 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:35.013+0000 7f48c3630700 1 -- 172.21.15.29:0/1683849538 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f48bc19c8d0 con 0x7f48bc108780 2024-03-16T11:41:35.821 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:35.013+0000 7f48b27fc700 1 -- 172.21.15.29:0/1683849538 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 13) ==== 46162+0+0 (secure 0 0 0) 0x7f48b8024aa0 con 0x7f48bc108780 2024-03-16T11:41:35.821 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:35.013+0000 7f48c3630700 1 -- 172.21.15.29:0/1683849538 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f48bc063900 con 0x7f48bc108780 2024-03-16T11:41:35.821 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:35.014+0000 7f48b27fc700 1 --2- 172.21.15.29:0/1683849538 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f48a8039020 0x7f48a803b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:41:35.821 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:35.014+0000 7f48b27fc700 1 -- 172.21.15.29:0/1683849538 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f48b804d610 con 0x7f48bc108780 2024-03-16T11:41:35.821 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:35.014+0000 7f48c0bcb700 1 --2- 172.21.15.29:0/1683849538 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f48a8039020 0x7f48a803b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:41:35.821 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:35.014+0000 7f48c0bcb700 1 --2- 172.21.15.29:0/1683849538 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f48a8039020 0x7f48a803b4e0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f48ac006fd0 tx=0x7f48ac006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:41:35.821 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:35.019+0000 7f48b27fc700 1 -- 172.21.15.29:0/1683849538 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f48b8011e20 con 0x7f48bc108780 2024-03-16T11:41:35.821 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:35.226+0000 7f48c3630700 1 -- 172.21.15.29:0/1683849538 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) -- 0x7f48bc19f340 con 0x7f48bc108780 2024-03-16T11:41:35.821 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:35.226+0000 7f48b27fc700 1 -- 172.21.15.29:0/1683849538 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) ==== 127+0+0 (secure 0 0 0) 0x7f48b802d770 con 0x7f48bc108780 2024-03-16T11:41:35.821 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:35.228+0000 7f48c3630700 1 -- 172.21.15.29:0/1683849538 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f48a8039020 msgr2=0x7f48a803b4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:41:35.821 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:35.228+0000 7f48c3630700 1 --2- 172.21.15.29:0/1683849538 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f48a8039020 0x7f48a803b4e0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f48ac006fd0 tx=0x7f48ac006e40 comp rx=0 tx=0).stop 2024-03-16T11:41:35.821 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:35.228+0000 7f48c3630700 1 -- 172.21.15.29:0/1683849538 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f48bc108780 msgr2=0x7f48bc19bca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:41:35.821 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:35.228+0000 7f48c3630700 1 --2- 172.21.15.29:0/1683849538 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f48bc108780 0x7f48bc19bca0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f48b80116a0 tx=0x7f48b8011ba0 comp rx=0 tx=0).stop 2024-03-16T11:41:35.821 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:35.228+0000 7f48c3630700 1 -- 172.21.15.29:0/1683849538 shutdown_connections 2024-03-16T11:41:35.821 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:35.228+0000 7f48c3630700 1 --2- 172.21.15.29:0/1683849538 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f48a8039020 0x7f48a803b4e0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:41:35.821 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:35.228+0000 7f48c3630700 1 --2- 172.21.15.29:0/1683849538 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f48bc108780 0x7f48bc19bca0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:41:35.822 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:35.228+0000 7f48c3630700 1 -- 172.21.15.29:0/1683849538 >> 172.21.15.29:0/1683849538 conn(0x7f48bc1021b0 msgr2=0x7f48bc102ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:41:35.822 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:35.228+0000 7f48c3630700 1 -- 172.21.15.29:0/1683849538 shutdown_connections 2024-03-16T11:41:35.822 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:35.228+0000 7f48c3630700 1 -- 172.21.15.29:0/1683849538 wait complete. 2024-03-16T11:41:37.131 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:36.246+0000 7efe5df67700 1 Processor -- start 2024-03-16T11:41:37.131 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:36.246+0000 7efe5df67700 1 -- start start 2024-03-16T11:41:37.131 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:36.246+0000 7efe5df67700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efe5807ddc0 0x7efe5807c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:41:37.131 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:36.246+0000 7efe5df67700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7efe5807c800 con 0x7efe5807ddc0 2024-03-16T11:41:37.131 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:36.246+0000 7efe577fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efe5807ddc0 0x7efe5807c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:41:37.131 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:36.246+0000 7efe577fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efe5807ddc0 0x7efe5807c2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60014/0 (socket says 172.21.15.29:60014) 2024-03-16T11:41:37.131 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:36.246+0000 7efe577fe700 1 -- 172.21.15.29:0/106323337 learned_addr learned my addr 172.21.15.29:0/106323337 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:41:37.131 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:36.247+0000 7efe577fe700 1 -- 172.21.15.29:0/106323337 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe5807c980 con 0x7efe5807ddc0 2024-03-16T11:41:37.132 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:36.247+0000 7efe577fe700 1 --2- 172.21.15.29:0/106323337 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efe5807ddc0 0x7efe5807c2c0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7efe48009e80 tx=0x7efe4800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c815b88c73ba9bd7 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:41:37.132 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:36.247+0000 7efe567fc700 1 -- 172.21.15.29:0/106323337 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7efe48017070 con 0x7efe5807ddc0 2024-03-16T11:41:37.132 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:36.247+0000 7efe567fc700 1 -- 172.21.15.29:0/106323337 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efe4800b8c0 con 0x7efe5807ddc0 2024-03-16T11:41:37.132 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:36.247+0000 7efe567fc700 1 -- 172.21.15.29:0/106323337 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7efe480045c0 con 0x7efe5807ddc0 2024-03-16T11:41:37.132 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:36.248+0000 7efe5df67700 1 -- 172.21.15.29:0/106323337 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efe5807ddc0 msgr2=0x7efe5807c2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:41:37.132 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:36.248+0000 7efe5df67700 1 --2- 172.21.15.29:0/106323337 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efe5807ddc0 0x7efe5807c2c0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7efe48009e80 tx=0x7efe4800b330 comp rx=0 tx=0).stop 2024-03-16T11:41:37.132 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:36.248+0000 7efe5df67700 1 -- 172.21.15.29:0/106323337 shutdown_connections 2024-03-16T11:41:37.132 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:36.248+0000 7efe5df67700 1 --2- 172.21.15.29:0/106323337 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efe5807ddc0 0x7efe5807c2c0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:41:37.132 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:36.248+0000 7efe5df67700 1 -- 172.21.15.29:0/106323337 >> 172.21.15.29:0/106323337 conn(0x7efe58103360 msgr2=0x7efe58105780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:41:37.132 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:36.248+0000 7efe5df67700 1 -- 172.21.15.29:0/106323337 shutdown_connections 2024-03-16T11:41:37.132 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:36.248+0000 7efe5df67700 1 -- 172.21.15.29:0/106323337 wait complete. 2024-03-16T11:41:37.132 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:36.248+0000 7efe5df67700 1 Processor -- start 2024-03-16T11:41:37.132 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:36.249+0000 7efe5df67700 1 -- start start 2024-03-16T11:41:37.132 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:36.249+0000 7efe5df67700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efe5807ddc0 0x7efe5819dec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:41:37.133 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:36.249+0000 7efe5df67700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7efe5810e340 con 0x7efe5807ddc0 2024-03-16T11:41:37.136 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:36.249+0000 7efe577fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efe5807ddc0 0x7efe5819dec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:41:37.136 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:36.249+0000 7efe577fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efe5807ddc0 0x7efe5819dec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60016/0 (socket says 172.21.15.29:60016) 2024-03-16T11:41:37.136 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:36.249+0000 7efe577fe700 1 -- 172.21.15.29:0/707230044 learned_addr learned my addr 172.21.15.29:0/707230044 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:41:37.136 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:36.249+0000 7efe577fe700 1 -- 172.21.15.29:0/707230044 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe5819e400 con 0x7efe5807ddc0 2024-03-16T11:41:37.136 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:36.249+0000 7efe577fe700 1 --2- 172.21.15.29:0/707230044 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efe5807ddc0 0x7efe5819dec0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7efe48004350 tx=0x7efe4800bd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:41:37.136 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:36.249+0000 7efe54ff9700 1 -- 172.21.15.29:0/707230044 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7efe48020020 con 0x7efe5807ddc0 2024-03-16T11:41:37.137 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:36.250+0000 7efe54ff9700 1 -- 172.21.15.29:0/707230044 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efe48003d90 con 0x7efe5807ddc0 2024-03-16T11:41:37.137 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:36.250+0000 7efe5df67700 1 -- 172.21.15.29:0/707230044 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efe5819e690 con 0x7efe5807ddc0 2024-03-16T11:41:37.137 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:36.250+0000 7efe54ff9700 1 -- 172.21.15.29:0/707230044 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7efe48017040 con 0x7efe5807ddc0 2024-03-16T11:41:37.137 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:36.250+0000 7efe5df67700 1 -- 172.21.15.29:0/707230044 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efe581a1380 con 0x7efe5807ddc0 2024-03-16T11:41:37.137 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:36.250+0000 7efe54ff9700 1 -- 172.21.15.29:0/707230044 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 13) ==== 46162+0+0 (secure 0 0 0) 0x7efe48023020 con 0x7efe5807ddc0 2024-03-16T11:41:37.137 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:36.251+0000 7efe5df67700 1 -- 172.21.15.29:0/707230044 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efe58063900 con 0x7efe5807ddc0 2024-03-16T11:41:37.137 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:36.251+0000 7efe54ff9700 1 --2- 172.21.15.29:0/707230044 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7efe44038fc0 0x7efe4403b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:41:37.137 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:36.251+0000 7efe54ff9700 1 -- 172.21.15.29:0/707230044 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7efe4800f050 con 0x7efe5807ddc0 2024-03-16T11:41:37.137 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:36.251+0000 7efe56ffd700 1 --2- 172.21.15.29:0/707230044 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7efe44038fc0 0x7efe4403b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:41:37.137 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:36.251+0000 7efe56ffd700 1 --2- 172.21.15.29:0/707230044 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7efe44038fc0 0x7efe4403b480 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7efe40006fd0 tx=0x7efe40006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:41:37.137 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:36.256+0000 7efe54ff9700 1 -- 172.21.15.29:0/707230044 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7efe4802a020 con 0x7efe5807ddc0 2024-03-16T11:41:37.137 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:36.575+0000 7efe5df67700 1 -- 172.21.15.29:0/707230044 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) -- 0x7efe581a1950 con 0x7efe5807ddc0 2024-03-16T11:41:37.137 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:36.578+0000 7efe54ff9700 1 -- 172.21.15.29:0/707230044 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v28)=0 set mgr/dashboard/cluster/status v28) ==== 153+0+0 (secure 0 0 0) 0x7efe48023dd0 con 0x7efe5807ddc0 2024-03-16T11:41:37.137 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:36.580+0000 7efe5df67700 1 -- 172.21.15.29:0/707230044 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7efe44038fc0 msgr2=0x7efe4403b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:41:37.138 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:36.580+0000 7efe5df67700 1 --2- 172.21.15.29:0/707230044 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7efe44038fc0 0x7efe4403b480 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7efe40006fd0 tx=0x7efe40006e40 comp rx=0 tx=0).stop 2024-03-16T11:41:37.138 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:36.580+0000 7efe5df67700 1 -- 172.21.15.29:0/707230044 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efe5807ddc0 msgr2=0x7efe5819dec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:41:37.138 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:36.580+0000 7efe5df67700 1 --2- 172.21.15.29:0/707230044 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efe5807ddc0 0x7efe5819dec0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7efe48004350 tx=0x7efe4800bd50 comp rx=0 tx=0).stop 2024-03-16T11:41:37.138 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:36.580+0000 7efe5df67700 1 -- 172.21.15.29:0/707230044 shutdown_connections 2024-03-16T11:41:37.138 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:36.580+0000 7efe5df67700 1 --2- 172.21.15.29:0/707230044 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7efe44038fc0 0x7efe4403b480 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:41:37.138 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:36.580+0000 7efe5df67700 1 --2- 172.21.15.29:0/707230044 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efe5807ddc0 0x7efe5819dec0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:41:37.138 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:36.580+0000 7efe5df67700 1 -- 172.21.15.29:0/707230044 >> 172.21.15.29:0/707230044 conn(0x7efe58103360 msgr2=0x7efe58103ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:41:37.138 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:36.580+0000 7efe5df67700 1 -- 172.21.15.29:0/707230044 shutdown_connections 2024-03-16T11:41:37.138 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr 2024-03-16T11:41:36.580+0000 7efe5df67700 1 -- 172.21.15.29:0/707230044 wait complete. 2024-03-16T11:41:37.138 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-03-16T11:41:37.138 INFO:teuthology.orchestra.run.smithi029.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-03-16T11:41:37.138 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-16T11:41:37.138 INFO:teuthology.orchestra.run.smithi029.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid d1887f28-e389-11ee-95c9-87774f69a715 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-03-16T11:41:37.138 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-16T11:41:37.139 INFO:teuthology.orchestra.run.smithi029.stdout:Or, if you are only running a single cluster on this host: 2024-03-16T11:41:37.139 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-16T11:41:37.139 INFO:teuthology.orchestra.run.smithi029.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-03-16T11:41:37.139 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-16T11:41:37.139 INFO:teuthology.orchestra.run.smithi029.stdout:Please consider enabling telemetry to help improve Ceph: 2024-03-16T11:41:37.139 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-16T11:41:37.139 INFO:teuthology.orchestra.run.smithi029.stdout: ceph telemetry on 2024-03-16T11:41:37.139 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-16T11:41:37.139 INFO:teuthology.orchestra.run.smithi029.stdout:For more information see: 2024-03-16T11:41:37.139 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-16T11:41:37.139 INFO:teuthology.orchestra.run.smithi029.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-03-16T11:41:37.139 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-16T11:41:37.139 INFO:teuthology.orchestra.run.smithi029.stdout:Bootstrap complete. 2024-03-16T11:41:37.178 INFO:tasks.cephadm:Fetching config... 2024-03-16T11:41:37.179 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-03-16T11:41:37.179 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-03-16T11:41:37.195 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-03-16T11:41:37.195 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-03-16T11:41:37.195 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-03-16T11:41:37.250 INFO:tasks.cephadm:Fetching mon keyring... 2024-03-16T11:41:37.250 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-03-16T11:41:37.250 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/var/lib/ceph/d1887f28-e389-11ee-95c9-87774f69a715/mon.a/keyring of=/dev/stdout 2024-03-16T11:41:37.318 INFO:tasks.cephadm:Fetching pub ssh key... 2024-03-16T11:41:37.318 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-03-16T11:41:37.318 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-03-16T11:41:37.373 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-03-16T11:41:37.373 DEBUG:teuthology.orchestra.run.smithi029:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC8KkuD7XYsg7qg5vvm6f9C9F+bVy31FMlXBVVQr5jgiqFnoJJ2WHe/FyeDly3kAIcxbqx/iXf/RyV7hp8dJKclLomINMbzqk9ZVgPp5Y026gBfLWwF9v5PR8XEp69/TpfLMXEHTfnPIhrkq8TLxGdCSf9KZnNkxbAAydQq1I3bdKkNHmLBSduPJJXigbrrxZKltcOiUv1HHWJ+izQLpohLvOnkkoVcIKjZhJg/W6MvM36jmQ8QSumUvV48B6/RpXrRbRmJZk/pJXuKTI6iUtSgjRVZU3e2lv3z62fkqRfnax/PlA38ddonynlcCkwcFmFZpo9kgH+0d7HqJmt1bQ+dfih97Hj4Cepr7q6of9GgCR5m+6bqM6E5XYAiskWUVcQiMuUdtWIGWLwZLSTKMPn/ShpirlzVKZj0+UErz4t+pH3x0SrEv5yFbvV06xs067rn0AORK42XwURRlffKbMhQvcLKYfNWpyRC2Uhx6AH5JC7UAFo4S8XLr4jNfxmjnbk= ceph-d1887f28-e389-11ee-95c9-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-16T11:41:37.461 INFO:teuthology.orchestra.run.smithi029.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC8KkuD7XYsg7qg5vvm6f9C9F+bVy31FMlXBVVQr5jgiqFnoJJ2WHe/FyeDly3kAIcxbqx/iXf/RyV7hp8dJKclLomINMbzqk9ZVgPp5Y026gBfLWwF9v5PR8XEp69/TpfLMXEHTfnPIhrkq8TLxGdCSf9KZnNkxbAAydQq1I3bdKkNHmLBSduPJJXigbrrxZKltcOiUv1HHWJ+izQLpohLvOnkkoVcIKjZhJg/W6MvM36jmQ8QSumUvV48B6/RpXrRbRmJZk/pJXuKTI6iUtSgjRVZU3e2lv3z62fkqRfnax/PlA38ddonynlcCkwcFmFZpo9kgH+0d7HqJmt1bQ+dfih97Hj4Cepr7q6of9GgCR5m+6bqM6E5XYAiskWUVcQiMuUdtWIGWLwZLSTKMPn/ShpirlzVKZj0+UErz4t+pH3x0SrEv5yFbvV06xs067rn0AORK42XwURRlffKbMhQvcLKYfNWpyRC2Uhx6AH5JC7UAFo4S8XLr4jNfxmjnbk= ceph-d1887f28-e389-11ee-95c9-87774f69a715 2024-03-16T11:41:37.475 DEBUG:teuthology.orchestra.run.smithi152:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC8KkuD7XYsg7qg5vvm6f9C9F+bVy31FMlXBVVQr5jgiqFnoJJ2WHe/FyeDly3kAIcxbqx/iXf/RyV7hp8dJKclLomINMbzqk9ZVgPp5Y026gBfLWwF9v5PR8XEp69/TpfLMXEHTfnPIhrkq8TLxGdCSf9KZnNkxbAAydQq1I3bdKkNHmLBSduPJJXigbrrxZKltcOiUv1HHWJ+izQLpohLvOnkkoVcIKjZhJg/W6MvM36jmQ8QSumUvV48B6/RpXrRbRmJZk/pJXuKTI6iUtSgjRVZU3e2lv3z62fkqRfnax/PlA38ddonynlcCkwcFmFZpo9kgH+0d7HqJmt1bQ+dfih97Hj4Cepr7q6of9GgCR5m+6bqM6E5XYAiskWUVcQiMuUdtWIGWLwZLSTKMPn/ShpirlzVKZj0+UErz4t+pH3x0SrEv5yFbvV06xs067rn0AORK42XwURRlffKbMhQvcLKYfNWpyRC2Uhx6AH5JC7UAFo4S8XLr4jNfxmjnbk= ceph-d1887f28-e389-11ee-95c9-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-16T11:41:37.523 INFO:teuthology.orchestra.run.smithi152.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC8KkuD7XYsg7qg5vvm6f9C9F+bVy31FMlXBVVQr5jgiqFnoJJ2WHe/FyeDly3kAIcxbqx/iXf/RyV7hp8dJKclLomINMbzqk9ZVgPp5Y026gBfLWwF9v5PR8XEp69/TpfLMXEHTfnPIhrkq8TLxGdCSf9KZnNkxbAAydQq1I3bdKkNHmLBSduPJJXigbrrxZKltcOiUv1HHWJ+izQLpohLvOnkkoVcIKjZhJg/W6MvM36jmQ8QSumUvV48B6/RpXrRbRmJZk/pJXuKTI6iUtSgjRVZU3e2lv3z62fkqRfnax/PlA38ddonynlcCkwcFmFZpo9kgH+0d7HqJmt1bQ+dfih97Hj4Cepr7q6of9GgCR5m+6bqM6E5XYAiskWUVcQiMuUdtWIGWLwZLSTKMPn/ShpirlzVKZj0+UErz4t+pH3x0SrEv5yFbvV06xs067rn0AORK42XwURRlffKbMhQvcLKYfNWpyRC2Uhx6AH5JC7UAFo4S8XLr4jNfxmjnbk= ceph-d1887f28-e389-11ee-95c9-87774f69a715 2024-03-16T11:41:37.540 DEBUG:teuthology.orchestra.run.smithi154:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC8KkuD7XYsg7qg5vvm6f9C9F+bVy31FMlXBVVQr5jgiqFnoJJ2WHe/FyeDly3kAIcxbqx/iXf/RyV7hp8dJKclLomINMbzqk9ZVgPp5Y026gBfLWwF9v5PR8XEp69/TpfLMXEHTfnPIhrkq8TLxGdCSf9KZnNkxbAAydQq1I3bdKkNHmLBSduPJJXigbrrxZKltcOiUv1HHWJ+izQLpohLvOnkkoVcIKjZhJg/W6MvM36jmQ8QSumUvV48B6/RpXrRbRmJZk/pJXuKTI6iUtSgjRVZU3e2lv3z62fkqRfnax/PlA38ddonynlcCkwcFmFZpo9kgH+0d7HqJmt1bQ+dfih97Hj4Cepr7q6of9GgCR5m+6bqM6E5XYAiskWUVcQiMuUdtWIGWLwZLSTKMPn/ShpirlzVKZj0+UErz4t+pH3x0SrEv5yFbvV06xs067rn0AORK42XwURRlffKbMhQvcLKYfNWpyRC2Uhx6AH5JC7UAFo4S8XLr4jNfxmjnbk= ceph-d1887f28-e389-11ee-95c9-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-16T11:41:37.590 INFO:teuthology.orchestra.run.smithi154.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC8KkuD7XYsg7qg5vvm6f9C9F+bVy31FMlXBVVQr5jgiqFnoJJ2WHe/FyeDly3kAIcxbqx/iXf/RyV7hp8dJKclLomINMbzqk9ZVgPp5Y026gBfLWwF9v5PR8XEp69/TpfLMXEHTfnPIhrkq8TLxGdCSf9KZnNkxbAAydQq1I3bdKkNHmLBSduPJJXigbrrxZKltcOiUv1HHWJ+izQLpohLvOnkkoVcIKjZhJg/W6MvM36jmQ8QSumUvV48B6/RpXrRbRmJZk/pJXuKTI6iUtSgjRVZU3e2lv3z62fkqRfnax/PlA38ddonynlcCkwcFmFZpo9kgH+0d7HqJmt1bQ+dfih97Hj4Cepr7q6of9GgCR5m+6bqM6E5XYAiskWUVcQiMuUdtWIGWLwZLSTKMPn/ShpirlzVKZj0+UErz4t+pH3x0SrEv5yFbvV06xs067rn0AORK42XwURRlffKbMhQvcLKYfNWpyRC2Uhx6AH5JC7UAFo4S8XLr4jNfxmjnbk= ceph-d1887f28-e389-11ee-95c9-87774f69a715 2024-03-16T11:41:37.606 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d1887f28-e389-11ee-95c9-87774f69a715 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-03-16T11:41:37.682 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:41:37 smithi029 ceph-mon[31975]: from='client.? 172.21.15.29:0/707230044' entity='client.admin' 2024-03-16T11:41:37.846 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/d1887f28-e389-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T11:41:39.588 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:41:39.587+0000 7fdfddf0f700 1 -- 172.21.15.29:0/2764439424 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdfd81027b0 msgr2=0x7fdfd8104ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:41:39.588 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:41:39.587+0000 7fdfddf0f700 1 --2- 172.21.15.29:0/2764439424 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdfd81027b0 0x7fdfd8104ba0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fdfc8009910 tx=0x7fdfc8009c20 comp rx=0 tx=0).stop 2024-03-16T11:41:39.588 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:41:39.587+0000 7fdfddf0f700 1 -- 172.21.15.29:0/2764439424 shutdown_connections 2024-03-16T11:41:39.588 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:41:39.587+0000 7fdfddf0f700 1 --2- 172.21.15.29:0/2764439424 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdfd81027b0 0x7fdfd8104ba0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:41:39.589 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:41:39.587+0000 7fdfddf0f700 1 -- 172.21.15.29:0/2764439424 >> 172.21.15.29:0/2764439424 conn(0x7fdfd80fc1e0 msgr2=0x7fdfd80fe600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:41:39.589 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:41:39.587+0000 7fdfddf0f700 1 -- 172.21.15.29:0/2764439424 shutdown_connections 2024-03-16T11:41:39.589 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:41:39.587+0000 7fdfddf0f700 1 -- 172.21.15.29:0/2764439424 wait complete. 2024-03-16T11:41:39.589 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:41:39.587+0000 7fdfddf0f700 1 Processor -- start 2024-03-16T11:41:39.589 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:41:39.588+0000 7fdfddf0f700 1 -- start start 2024-03-16T11:41:39.589 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:41:39.588+0000 7fdfddf0f700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdfd81027b0 0x7fdfd81973a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:41:39.589 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:41:39.588+0000 7fdfddf0f700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fdfd8107d50 con 0x7fdfd81027b0 2024-03-16T11:41:39.589 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:41:39.588+0000 7fdfd77fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdfd81027b0 0x7fdfd81973a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:41:39.589 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:41:39.588+0000 7fdfd77fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdfd81027b0 0x7fdfd81973a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:38518/0 (socket says 172.21.15.29:38518) 2024-03-16T11:41:39.589 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:41:39.588+0000 7fdfd77fe700 1 -- 172.21.15.29:0/2895910756 learned_addr learned my addr 172.21.15.29:0/2895910756 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:41:39.589 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:41:39.588+0000 7fdfd77fe700 1 -- 172.21.15.29:0/2895910756 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdfd81978e0 con 0x7fdfd81027b0 2024-03-16T11:41:39.589 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:41:39.588+0000 7fdfd77fe700 1 --2- 172.21.15.29:0/2895910756 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdfd81027b0 0x7fdfd81973a0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fdfc8004fa0 tx=0x7fdfc8005080 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:41:39.589 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:41:39.588+0000 7fdfd4ff9700 1 -- 172.21.15.29:0/2895910756 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdfc8027070 con 0x7fdfd81027b0 2024-03-16T11:41:39.589 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:41:39.589+0000 7fdfd4ff9700 1 -- 172.21.15.29:0/2895910756 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdfc800be40 con 0x7fdfd81027b0 2024-03-16T11:41:39.589 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:41:39.589+0000 7fdfd4ff9700 1 -- 172.21.15.29:0/2895910756 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdfc800f460 con 0x7fdfd81027b0 2024-03-16T11:41:39.590 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:41:39.589+0000 7fdfddf0f700 1 -- 172.21.15.29:0/2895910756 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdfd8197bd0 con 0x7fdfd81027b0 2024-03-16T11:41:39.590 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:41:39.589+0000 7fdfddf0f700 1 -- 172.21.15.29:0/2895910756 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdfd819a810 con 0x7fdfd81027b0 2024-03-16T11:41:39.590 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:41:39.589+0000 7fdfd4ff9700 1 -- 172.21.15.29:0/2895910756 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 13) ==== 46162+0+0 (secure 0 0 0) 0x7fdfc802a070 con 0x7fdfd81027b0 2024-03-16T11:41:39.590 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:41:39.590+0000 7fdfd4ff9700 1 --2- 172.21.15.29:0/2895910756 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fdfc0038fc0 0x7fdfc003b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:41:39.591 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:41:39.590+0000 7fdfd4ff9700 1 -- 172.21.15.29:0/2895910756 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fdfc801e030 con 0x7fdfd81027b0 2024-03-16T11:41:39.591 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:41:39.590+0000 7fdfddf0f700 1 -- 172.21.15.29:0/2895910756 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdfd81084d0 con 0x7fdfd81027b0 2024-03-16T11:41:39.591 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:41:39.590+0000 7fdfd6ffd700 1 --2- 172.21.15.29:0/2895910756 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fdfc0038fc0 0x7fdfc003b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:41:39.591 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:41:39.590+0000 7fdfd6ffd700 1 --2- 172.21.15.29:0/2895910756 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fdfc0038fc0 0x7fdfc003b480 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fdfcc006fd0 tx=0x7fdfcc006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:41:39.594 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:41:39.595+0000 7fdfd4ff9700 1 -- 172.21.15.29:0/2895910756 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fdfc8013070 con 0x7fdfd81027b0 2024-03-16T11:41:39.802 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:41:39.802+0000 7fdfddf0f700 1 -- 172.21.15.29:0/2895910756 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) -- 0x7fdfd802dc80 con 0x7fdfd81027b0 2024-03-16T11:41:39.804 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:41:39.805+0000 7fdfd4ff9700 1 -- 172.21.15.29:0/2895910756 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) ==== 125+0+0 (secure 0 0 0) 0x7fdfc8020c90 con 0x7fdfd81027b0 2024-03-16T11:41:39.807 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:41:39.807+0000 7fdfddf0f700 1 -- 172.21.15.29:0/2895910756 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fdfc0038fc0 msgr2=0x7fdfc003b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:41:39.807 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:41:39.807+0000 7fdfddf0f700 1 --2- 172.21.15.29:0/2895910756 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fdfc0038fc0 0x7fdfc003b480 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fdfcc006fd0 tx=0x7fdfcc006e40 comp rx=0 tx=0).stop 2024-03-16T11:41:39.807 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:41:39.807+0000 7fdfddf0f700 1 -- 172.21.15.29:0/2895910756 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdfd81027b0 msgr2=0x7fdfd81973a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:41:39.807 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:41:39.807+0000 7fdfddf0f700 1 --2- 172.21.15.29:0/2895910756 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdfd81027b0 0x7fdfd81973a0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fdfc8004fa0 tx=0x7fdfc8005080 comp rx=0 tx=0).stop 2024-03-16T11:41:39.807 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:41:39.807+0000 7fdfddf0f700 1 -- 172.21.15.29:0/2895910756 shutdown_connections 2024-03-16T11:41:39.807 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:41:39.807+0000 7fdfddf0f700 1 --2- 172.21.15.29:0/2895910756 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fdfc0038fc0 0x7fdfc003b480 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:41:39.807 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:41:39.807+0000 7fdfddf0f700 1 --2- 172.21.15.29:0/2895910756 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdfd81027b0 0x7fdfd81973a0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:41:39.807 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:41:39.807+0000 7fdfddf0f700 1 -- 172.21.15.29:0/2895910756 >> 172.21.15.29:0/2895910756 conn(0x7fdfd80fc1e0 msgr2=0x7fdfd80fcbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:41:39.807 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:41:39.807+0000 7fdfddf0f700 1 -- 172.21.15.29:0/2895910756 shutdown_connections 2024-03-16T11:41:39.807 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:41:39.807+0000 7fdfddf0f700 1 -- 172.21.15.29:0/2895910756 wait complete. 2024-03-16T11:41:40.597 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-03-16T11:41:40.597 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d1887f28-e389-11ee-95c9-87774f69a715 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-03-16T11:41:41.061 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:41:40 smithi029 ceph-mon[31975]: from='client.? 172.21.15.29:0/2895910756' entity='client.admin' 2024-03-16T11:41:41.098 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/d1887f28-e389-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T11:41:43.131 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:41:43.131+0000 7fe40600b700 1 -- 172.21.15.29:0/2011284265 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe400101d20 msgr2=0x7fe400102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:41:43.131 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:41:43.131+0000 7fe40600b700 1 --2- 172.21.15.29:0/2011284265 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe400101d20 0x7fe400102140 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fe3f0009910 tx=0x7fe3f0009c20 comp rx=0 tx=0).stop 2024-03-16T11:41:43.132 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:41:43.131+0000 7fe40600b700 1 -- 172.21.15.29:0/2011284265 shutdown_connections 2024-03-16T11:41:43.132 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:41:43.131+0000 7fe40600b700 1 --2- 172.21.15.29:0/2011284265 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe400101d20 0x7fe400102140 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:41:43.132 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:41:43.131+0000 7fe40600b700 1 -- 172.21.15.29:0/2011284265 >> 172.21.15.29:0/2011284265 conn(0x7fe4000fd2c0 msgr2=0x7fe4000ff700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:41:43.132 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:41:43.132+0000 7fe40600b700 1 -- 172.21.15.29:0/2011284265 shutdown_connections 2024-03-16T11:41:43.132 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:41:43.132+0000 7fe40600b700 1 -- 172.21.15.29:0/2011284265 wait complete. 2024-03-16T11:41:43.134 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:41:43.132+0000 7fe40600b700 1 Processor -- start 2024-03-16T11:41:43.134 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:41:43.132+0000 7fe40600b700 1 -- start start 2024-03-16T11:41:43.134 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:41:43.133+0000 7fe40600b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe400101d20 0x7fe4001974b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:41:43.134 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:41:43.133+0000 7fe40600b700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fe400107db0 con 0x7fe400101d20 2024-03-16T11:41:43.134 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:41:43.133+0000 7fe3ff7fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe400101d20 0x7fe4001974b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:41:43.134 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:41:43.133+0000 7fe3ff7fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe400101d20 0x7fe4001974b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:38542/0 (socket says 172.21.15.29:38542) 2024-03-16T11:41:43.134 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:41:43.133+0000 7fe3ff7fe700 1 -- 172.21.15.29:0/3226918636 learned_addr learned my addr 172.21.15.29:0/3226918636 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:41:43.134 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:41:43.133+0000 7fe3ff7fe700 1 -- 172.21.15.29:0/3226918636 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4001979f0 con 0x7fe400101d20 2024-03-16T11:41:43.135 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:41:43.133+0000 7fe3ff7fe700 1 --2- 172.21.15.29:0/3226918636 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe400101d20 0x7fe4001974b0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fe3f0006010 tx=0x7fe3f0005050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:41:43.135 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:41:43.133+0000 7fe3fcff9700 1 -- 172.21.15.29:0/3226918636 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe3f0027070 con 0x7fe400101d20 2024-03-16T11:41:43.135 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:41:43.133+0000 7fe3fcff9700 1 -- 172.21.15.29:0/3226918636 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe3f000be40 con 0x7fe400101d20 2024-03-16T11:41:43.135 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:41:43.133+0000 7fe40600b700 1 -- 172.21.15.29:0/3226918636 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe400197c80 con 0x7fe400101d20 2024-03-16T11:41:43.135 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:41:43.133+0000 7fe3fcff9700 1 -- 172.21.15.29:0/3226918636 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe3f000f460 con 0x7fe400101d20 2024-03-16T11:41:43.135 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:41:43.133+0000 7fe40600b700 1 -- 172.21.15.29:0/3226918636 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe40019a920 con 0x7fe400101d20 2024-03-16T11:41:43.135 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:41:43.134+0000 7fe3fcff9700 1 -- 172.21.15.29:0/3226918636 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 13) ==== 46162+0+0 (secure 0 0 0) 0x7fe3f002a070 con 0x7fe400101d20 2024-03-16T11:41:43.135 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:41:43.134+0000 7fe40600b700 1 -- 172.21.15.29:0/3226918636 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe400108530 con 0x7fe400101d20 2024-03-16T11:41:43.135 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:41:43.134+0000 7fe3fcff9700 1 --2- 172.21.15.29:0/3226918636 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fe3e8038fc0 0x7fe3e803b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:41:43.135 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:41:43.134+0000 7fe3fcff9700 1 -- 172.21.15.29:0/3226918636 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fe3f001e030 con 0x7fe400101d20 2024-03-16T11:41:43.135 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:41:43.135+0000 7fe3feffd700 1 --2- 172.21.15.29:0/3226918636 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fe3e8038fc0 0x7fe3e803b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:41:43.135 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:41:43.135+0000 7fe3feffd700 1 --2- 172.21.15.29:0/3226918636 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fe3e8038fc0 0x7fe3e803b480 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fe3f4006fd0 tx=0x7fe3f4006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:41:43.139 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:41:43.140+0000 7fe3fcff9700 1 -- 172.21.15.29:0/3226918636 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fe3f0013070 con 0x7fe400101d20 2024-03-16T11:41:43.347 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:41:43.347+0000 7fe40600b700 1 -- 172.21.15.29:0/3226918636 --> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) -- 0x7fe40002dc80 con 0x7fe3e8038fc0 2024-03-16T11:41:43.350 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:41:43.351+0000 7fe3fcff9700 1 -- 172.21.15.29:0/3226918636 <== mgr.14152 v2:172.21.15.29:6800/4192064640 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fe40002dc80 con 0x7fe3e8038fc0 2024-03-16T11:41:43.352 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:41:43.353+0000 7fe40600b700 1 -- 172.21.15.29:0/3226918636 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fe3e8038fc0 msgr2=0x7fe3e803b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:41:43.352 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:41:43.353+0000 7fe40600b700 1 --2- 172.21.15.29:0/3226918636 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fe3e8038fc0 0x7fe3e803b480 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fe3f4006fd0 tx=0x7fe3f4006e40 comp rx=0 tx=0).stop 2024-03-16T11:41:43.352 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:41:43.353+0000 7fe40600b700 1 -- 172.21.15.29:0/3226918636 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe400101d20 msgr2=0x7fe4001974b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:41:43.353 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:41:43.353+0000 7fe40600b700 1 --2- 172.21.15.29:0/3226918636 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe400101d20 0x7fe4001974b0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fe3f0006010 tx=0x7fe3f0005050 comp rx=0 tx=0).stop 2024-03-16T11:41:43.353 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:41:43.353+0000 7fe40600b700 1 -- 172.21.15.29:0/3226918636 shutdown_connections 2024-03-16T11:41:43.353 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:41:43.353+0000 7fe40600b700 1 --2- 172.21.15.29:0/3226918636 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fe3e8038fc0 0x7fe3e803b480 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:41:43.353 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:41:43.353+0000 7fe40600b700 1 --2- 172.21.15.29:0/3226918636 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe400101d20 0x7fe4001974b0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:41:43.353 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:41:43.353+0000 7fe40600b700 1 -- 172.21.15.29:0/3226918636 >> 172.21.15.29:0/3226918636 conn(0x7fe4000fd2c0 msgr2=0x7fe4000fddf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:41:43.353 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:41:43.353+0000 7fe40600b700 1 -- 172.21.15.29:0/3226918636 shutdown_connections 2024-03-16T11:41:43.353 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:41:43.353+0000 7fe40600b700 1 -- 172.21.15.29:0/3226918636 wait complete. 2024-03-16T11:41:44.054 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:41:43 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:41:44.054 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:41:43 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:41:44.055 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:41:43 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi029", "name": "osd_memory_target"}]: dispatch 2024-03-16T11:41:44.055 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:41:43 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:41:44.055 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:41:43 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T11:41:44.055 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:41:43 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:41:44.055 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:41:43 smithi029 ceph-mon[31975]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T11:41:44.055 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:41:43 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:41:44.055 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:41:43 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T11:41:44.055 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:41:43 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:41:44.055 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:41:43 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T11:41:44.055 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:41:43 smithi029 ceph-mon[31975]: Updating smithi029:/etc/ceph/ceph.conf 2024-03-16T11:41:44.083 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi152 2024-03-16T11:41:44.083 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-03-16T11:41:44.083 DEBUG:teuthology.orchestra.run.smithi152:> dd of=/etc/ceph/ceph.conf 2024-03-16T11:41:44.102 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-03-16T11:41:44.102 DEBUG:teuthology.orchestra.run.smithi152:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-16T11:41:44.159 INFO:tasks.cephadm:Adding host smithi152 to orchestrator... 2024-03-16T11:41:44.159 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d1887f28-e389-11ee-95c9-87774f69a715 -- ceph orch host add smithi152 2024-03-16T11:41:44.410 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/d1887f28-e389-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T11:41:45.453 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:41:45 smithi029 ceph-mon[31975]: Updating smithi029:/var/lib/ceph/d1887f28-e389-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-16T11:41:45.453 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:41:45 smithi029 ceph-mon[31975]: Updating smithi029:/etc/ceph/ceph.client.admin.keyring 2024-03-16T11:41:45.453 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:41:45 smithi029 ceph-mon[31975]: Updating smithi029:/var/lib/ceph/d1887f28-e389-11ee-95c9-87774f69a715/config/ceph.client.admin.keyring 2024-03-16T11:41:45.453 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:41:45 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:41:45.454 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:41:45 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:41:45.454 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:41:45 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:41:45.969 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:41:45.969+0000 7ff9598db700 1 -- 172.21.15.29:0/2146565207 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff954101f10 msgr2=0x7ff9541022f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:41:46.035 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:41:45.969+0000 7ff9598db700 1 --2- 172.21.15.29:0/2146565207 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff954101f10 0x7ff9541022f0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7ff944009910 tx=0x7ff944009c20 comp rx=0 tx=0).stop 2024-03-16T11:41:46.035 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:41:45.969+0000 7ff9598db700 1 -- 172.21.15.29:0/2146565207 shutdown_connections 2024-03-16T11:41:46.035 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:41:45.969+0000 7ff9598db700 1 --2- 172.21.15.29:0/2146565207 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff954101f10 0x7ff9541022f0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:41:46.035 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:41:45.969+0000 7ff9598db700 1 -- 172.21.15.29:0/2146565207 >> 172.21.15.29:0/2146565207 conn(0x7ff9540fd7e0 msgr2=0x7ff9540ffc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:41:46.035 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:41:45.969+0000 7ff9598db700 1 -- 172.21.15.29:0/2146565207 shutdown_connections 2024-03-16T11:41:46.035 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:41:45.969+0000 7ff9598db700 1 -- 172.21.15.29:0/2146565207 wait complete. 2024-03-16T11:41:46.037 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:41:45.970+0000 7ff9598db700 1 Processor -- start 2024-03-16T11:41:46.037 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:41:45.970+0000 7ff9598db700 1 -- start start 2024-03-16T11:41:46.038 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:41:45.970+0000 7ff9598db700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff954101f10 0x7ff9540745d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:41:46.038 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:41:45.970+0000 7ff9598db700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7ff95410c2e0 con 0x7ff954101f10 2024-03-16T11:41:46.038 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:41:45.971+0000 7ff952ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff954101f10 0x7ff9540745d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:41:46.038 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:41:45.971+0000 7ff952ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff954101f10 0x7ff9540745d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:38560/0 (socket says 172.21.15.29:38560) 2024-03-16T11:41:46.038 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:41:45.971+0000 7ff952ffd700 1 -- 172.21.15.29:0/578550746 learned_addr learned my addr 172.21.15.29:0/578550746 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:41:46.038 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:41:45.971+0000 7ff952ffd700 1 -- 172.21.15.29:0/578550746 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9540764c0 con 0x7ff954101f10 2024-03-16T11:41:46.038 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:41:45.971+0000 7ff952ffd700 1 --2- 172.21.15.29:0/578550746 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff954101f10 0x7ff9540745d0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7ff944005850 tx=0x7ff944005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:41:46.038 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:41:45.971+0000 7ff9588d9700 1 -- 172.21.15.29:0/578550746 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff944026070 con 0x7ff954101f10 2024-03-16T11:41:46.038 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:41:45.971+0000 7ff9588d9700 1 -- 172.21.15.29:0/578550746 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff94400bc30 con 0x7ff954101f10 2024-03-16T11:41:46.038 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:41:45.971+0000 7ff9598db700 1 -- 172.21.15.29:0/578550746 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff954074b70 con 0x7ff954101f10 2024-03-16T11:41:46.038 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:41:45.971+0000 7ff9588d9700 1 -- 172.21.15.29:0/578550746 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff944003a20 con 0x7ff954101f10 2024-03-16T11:41:46.038 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:41:45.971+0000 7ff9598db700 1 -- 172.21.15.29:0/578550746 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff954075030 con 0x7ff954101f10 2024-03-16T11:41:46.038 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:41:45.972+0000 7ff9588d9700 1 -- 172.21.15.29:0/578550746 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 13) ==== 46162+0+0 (secure 0 0 0) 0x7ff944029070 con 0x7ff954101f10 2024-03-16T11:41:46.039 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:41:45.972+0000 7ff9598db700 1 -- 172.21.15.29:0/578550746 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff9541033f0 con 0x7ff954101f10 2024-03-16T11:41:46.039 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:41:45.973+0000 7ff9588d9700 1 --2- 172.21.15.29:0/578550746 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7ff93c039020 0x7ff93c03b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:41:46.039 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:41:45.973+0000 7ff9588d9700 1 -- 172.21.15.29:0/578550746 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7ff94401e070 con 0x7ff954101f10 2024-03-16T11:41:46.039 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:41:45.973+0000 7ff9527fc700 1 --2- 172.21.15.29:0/578550746 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7ff93c039020 0x7ff93c03b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:41:46.039 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:41:45.973+0000 7ff9527fc700 1 --2- 172.21.15.29:0/578550746 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7ff93c039020 0x7ff93c03b4e0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7ff948006fd0 tx=0x7ff948006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:41:46.039 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:41:45.978+0000 7ff9588d9700 1 -- 172.21.15.29:0/578550746 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7ff944013070 con 0x7ff954101f10 2024-03-16T11:41:46.187 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:41:46.188+0000 7ff9598db700 1 -- 172.21.15.29:0/578550746 --> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi152", "target": ["mon-mgr", ""]}) -- 0x7ff954075b00 con 0x7ff93c039020 2024-03-16T11:41:47.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:41:46 smithi029 ceph-mon[31975]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi152", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T11:41:48.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:41:47 smithi029 ceph-mon[31975]: Deploying cephadm binary to smithi152 2024-03-16T11:41:48.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:41:47 smithi029 ceph-mon[31975]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T11:41:49.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:41:49 smithi029 ceph-mon[31975]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T11:41:50.539 INFO:teuthology.orchestra.run.smithi029.stdout:Added host 'smithi152' with addr '172.21.15.152' 2024-03-16T11:41:50.539 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:41:50.538+0000 7ff9588d9700 1 -- 172.21.15.29:0/578550746 <== mgr.14152 v2:172.21.15.29:6800/4192064640 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+49 (secure 0 0 0) 0x7ff954075b00 con 0x7ff93c039020 2024-03-16T11:41:50.541 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:41:50.541+0000 7ff9598db700 1 -- 172.21.15.29:0/578550746 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7ff93c039020 msgr2=0x7ff93c03b4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:41:50.541 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:41:50.541+0000 7ff9598db700 1 --2- 172.21.15.29:0/578550746 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7ff93c039020 0x7ff93c03b4e0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7ff948006fd0 tx=0x7ff948006e40 comp rx=0 tx=0).stop 2024-03-16T11:41:50.541 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:41:50.541+0000 7ff9598db700 1 -- 172.21.15.29:0/578550746 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff954101f10 msgr2=0x7ff9540745d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:41:50.541 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:41:50.541+0000 7ff9598db700 1 --2- 172.21.15.29:0/578550746 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff954101f10 0x7ff9540745d0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7ff944005850 tx=0x7ff944005030 comp rx=0 tx=0).stop 2024-03-16T11:41:50.541 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:41:50.541+0000 7ff9598db700 1 -- 172.21.15.29:0/578550746 shutdown_connections 2024-03-16T11:41:50.541 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:41:50.541+0000 7ff9598db700 1 --2- 172.21.15.29:0/578550746 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7ff93c039020 0x7ff93c03b4e0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:41:50.541 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:41:50.541+0000 7ff9598db700 1 --2- 172.21.15.29:0/578550746 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff954101f10 0x7ff9540745d0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:41:50.541 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:41:50.541+0000 7ff9598db700 1 -- 172.21.15.29:0/578550746 >> 172.21.15.29:0/578550746 conn(0x7ff9540fd7e0 msgr2=0x7ff954104d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:41:50.541 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:41:50.541+0000 7ff9598db700 1 -- 172.21.15.29:0/578550746 shutdown_connections 2024-03-16T11:41:50.541 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:41:50.541+0000 7ff9598db700 1 -- 172.21.15.29:0/578550746 wait complete. 2024-03-16T11:41:51.202 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d1887f28-e389-11ee-95c9-87774f69a715 -- ceph orch host ls --format=json 2024-03-16T11:41:51.451 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/d1887f28-e389-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T11:41:51.787 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:41:51 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:41:51.787 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:41:51 smithi029 ceph-mon[31975]: Added host smithi152 2024-03-16T11:41:51.787 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:41:51 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T11:41:51.787 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:41:51 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:41:52.795 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:41:52 smithi029 ceph-mon[31975]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T11:41:52.795 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:41:52 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:41:53.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:41:53.052+0000 7f9713636700 1 -- 172.21.15.29:0/1168387082 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f970c100420 msgr2=0x7f970c100800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:41:53.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:41:53.052+0000 7f9713636700 1 --2- 172.21.15.29:0/1168387082 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f970c100420 0x7f970c100800 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f96fc009910 tx=0x7f96fc009c20 comp rx=0 tx=0).stop 2024-03-16T11:41:53.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:41:53.052+0000 7f9713636700 1 -- 172.21.15.29:0/1168387082 shutdown_connections 2024-03-16T11:41:53.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:41:53.052+0000 7f9713636700 1 --2- 172.21.15.29:0/1168387082 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f970c100420 0x7f970c100800 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:41:53.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:41:53.052+0000 7f9713636700 1 -- 172.21.15.29:0/1168387082 >> 172.21.15.29:0/1168387082 conn(0x7f970c0fbe20 msgr2=0x7f970c0fe240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:41:53.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:41:53.052+0000 7f9713636700 1 -- 172.21.15.29:0/1168387082 shutdown_connections 2024-03-16T11:41:53.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:41:53.052+0000 7f9713636700 1 -- 172.21.15.29:0/1168387082 wait complete. 2024-03-16T11:41:53.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:41:53.053+0000 7f9713636700 1 Processor -- start 2024-03-16T11:41:53.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:41:53.053+0000 7f9713636700 1 -- start start 2024-03-16T11:41:53.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:41:53.053+0000 7f9713636700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f970c100420 0x7f970c1013a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:41:53.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:41:53.053+0000 7f9713636700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f970c10c2e0 con 0x7f970c100420 2024-03-16T11:41:53.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:41:53.053+0000 7f97113d2700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f970c100420 0x7f970c1013a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:41:53.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:41:53.053+0000 7f97113d2700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f970c100420 0x7f970c1013a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53632/0 (socket says 172.21.15.29:53632) 2024-03-16T11:41:53.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:41:53.053+0000 7f97113d2700 1 -- 172.21.15.29:0/545219156 learned_addr learned my addr 172.21.15.29:0/545219156 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:41:53.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:41:53.054+0000 7f97113d2700 1 -- 172.21.15.29:0/545219156 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f970c103290 con 0x7f970c100420 2024-03-16T11:41:53.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:41:53.054+0000 7f97113d2700 1 --2- 172.21.15.29:0/545219156 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f970c100420 0x7f970c1013a0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f96fc005850 tx=0x7f96fc005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:41:53.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:41:53.054+0000 7f97027fc700 1 -- 172.21.15.29:0/545219156 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f96fc026070 con 0x7f970c100420 2024-03-16T11:41:53.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:41:53.054+0000 7f9713636700 1 -- 172.21.15.29:0/545219156 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f970c101940 con 0x7f970c100420 2024-03-16T11:41:53.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:41:53.054+0000 7f97027fc700 1 -- 172.21.15.29:0/545219156 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f96fc00bc30 con 0x7f970c100420 2024-03-16T11:41:53.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:41:53.054+0000 7f97027fc700 1 -- 172.21.15.29:0/545219156 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f96fc003a20 con 0x7f970c100420 2024-03-16T11:41:53.058 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:41:53.054+0000 7f9713636700 1 -- 172.21.15.29:0/545219156 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f970c101e00 con 0x7f970c100420 2024-03-16T11:41:53.058 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:41:53.055+0000 7f97027fc700 1 -- 172.21.15.29:0/545219156 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 13) ==== 46162+0+0 (secure 0 0 0) 0x7f96fc029070 con 0x7f970c100420 2024-03-16T11:41:53.058 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:41:53.055+0000 7f9713636700 1 -- 172.21.15.29:0/545219156 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f970c063900 con 0x7f970c100420 2024-03-16T11:41:53.058 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:41:53.055+0000 7f97027fc700 1 --2- 172.21.15.29:0/545219156 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f96f8038fc0 0x7f96f803b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:41:53.058 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:41:53.055+0000 7f97027fc700 1 -- 172.21.15.29:0/545219156 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f96fc01e070 con 0x7f970c100420 2024-03-16T11:41:53.058 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:41:53.055+0000 7f9710bd1700 1 --2- 172.21.15.29:0/545219156 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f96f8038fc0 0x7f96f803b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:41:53.058 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:41:53.056+0000 7f9710bd1700 1 --2- 172.21.15.29:0/545219156 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f96f8038fc0 0x7f96f803b480 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f9708006fd0 tx=0x7f9708006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:41:53.059 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:41:53.060+0000 7f97027fc700 1 -- 172.21.15.29:0/545219156 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f96fc013070 con 0x7f970c100420 2024-03-16T11:41:53.268 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:41:53.269+0000 7f9713636700 1 -- 172.21.15.29:0/545219156 --> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f970c1034d0 con 0x7f96f8038fc0 2024-03-16T11:41:53.269 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:41:53.270+0000 7f97027fc700 1 -- 172.21.15.29:0/545219156 <== mgr.14152 v2:172.21.15.29:6800/4192064640 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+160 (secure 0 0 0) 0x7f970c1034d0 con 0x7f96f8038fc0 2024-03-16T11:41:53.269 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-16T11:41:53.270 INFO:teuthology.orchestra.run.smithi029.stdout:[{"addr": "172.21.15.29", "hostname": "smithi029", "labels": [], "status": ""}, {"addr": "172.21.15.152", "hostname": "smithi152", "labels": [], "status": ""}] 2024-03-16T11:41:53.272 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:41:53.272+0000 7f9713636700 1 -- 172.21.15.29:0/545219156 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f96f8038fc0 msgr2=0x7f96f803b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:41:53.272 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:41:53.272+0000 7f9713636700 1 --2- 172.21.15.29:0/545219156 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f96f8038fc0 0x7f96f803b480 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f9708006fd0 tx=0x7f9708006e40 comp rx=0 tx=0).stop 2024-03-16T11:41:53.272 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:41:53.272+0000 7f9713636700 1 -- 172.21.15.29:0/545219156 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f970c100420 msgr2=0x7f970c1013a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:41:53.272 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:41:53.272+0000 7f9713636700 1 --2- 172.21.15.29:0/545219156 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f970c100420 0x7f970c1013a0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f96fc005850 tx=0x7f96fc005030 comp rx=0 tx=0).stop 2024-03-16T11:41:53.272 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:41:53.272+0000 7f9713636700 1 -- 172.21.15.29:0/545219156 shutdown_connections 2024-03-16T11:41:53.272 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:41:53.272+0000 7f9713636700 1 --2- 172.21.15.29:0/545219156 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f96f8038fc0 0x7f96f803b480 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:41:53.272 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:41:53.272+0000 7f9713636700 1 --2- 172.21.15.29:0/545219156 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f970c100420 0x7f970c1013a0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:41:53.272 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:41:53.272+0000 7f9713636700 1 -- 172.21.15.29:0/545219156 >> 172.21.15.29:0/545219156 conn(0x7f970c0fbe20 msgr2=0x7f970c1038e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:41:53.272 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:41:53.272+0000 7f9713636700 1 -- 172.21.15.29:0/545219156 shutdown_connections 2024-03-16T11:41:53.272 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:41:53.272+0000 7f9713636700 1 -- 172.21.15.29:0/545219156 wait complete. 2024-03-16T11:41:53.871 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:41:53 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:41:53.871 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:41:53 smithi029 ceph-mon[31975]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-16T11:41:53.871 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:41:53 smithi029 ceph-mon[31975]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T11:41:53.898 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi154 2024-03-16T11:41:53.898 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-03-16T11:41:53.899 DEBUG:teuthology.orchestra.run.smithi154:> dd of=/etc/ceph/ceph.conf 2024-03-16T11:41:53.917 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-03-16T11:41:53.917 DEBUG:teuthology.orchestra.run.smithi154:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-16T11:41:53.974 INFO:tasks.cephadm:Adding host smithi154 to orchestrator... 2024-03-16T11:41:53.974 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d1887f28-e389-11ee-95c9-87774f69a715 -- ceph orch host add smithi154 2024-03-16T11:41:54.214 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/d1887f28-e389-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T11:41:55.807 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:41:55.807+0000 7f057f3ca700 1 -- 172.21.15.29:0/4226221602 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0578101f00 msgr2=0x7f05781022e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:41:55.807 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:41:55.807+0000 7f057f3ca700 1 --2- 172.21.15.29:0/4226221602 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0578101f00 0x7f05781022e0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f0574009910 tx=0x7f0574009c20 comp rx=0 tx=0).stop 2024-03-16T11:41:55.807 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:41:55.807+0000 7f057f3ca700 1 -- 172.21.15.29:0/4226221602 shutdown_connections 2024-03-16T11:41:55.807 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:41:55.807+0000 7f057f3ca700 1 --2- 172.21.15.29:0/4226221602 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0578101f00 0x7f05781022e0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:41:55.807 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:41:55.807+0000 7f057f3ca700 1 -- 172.21.15.29:0/4226221602 >> 172.21.15.29:0/4226221602 conn(0x7f05780fd7b0 msgr2=0x7f05780ffbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:41:55.807 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:41:55.807+0000 7f057f3ca700 1 -- 172.21.15.29:0/4226221602 shutdown_connections 2024-03-16T11:41:55.807 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:41:55.807+0000 7f057f3ca700 1 -- 172.21.15.29:0/4226221602 wait complete. 2024-03-16T11:41:55.809 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:41:55.808+0000 7f057f3ca700 1 Processor -- start 2024-03-16T11:41:55.809 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:41:55.808+0000 7f057f3ca700 1 -- start start 2024-03-16T11:41:55.809 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:41:55.808+0000 7f057f3ca700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0578101f00 0x7f057819d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:41:55.810 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:41:55.808+0000 7f057f3ca700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f057810c2d0 con 0x7f0578101f00 2024-03-16T11:41:55.810 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:41:55.808+0000 7f057d166700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0578101f00 0x7f057819d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:41:55.810 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:41:55.808+0000 7f057d166700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0578101f00 0x7f057819d450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53642/0 (socket says 172.21.15.29:53642) 2024-03-16T11:41:55.810 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:41:55.808+0000 7f057d166700 1 -- 172.21.15.29:0/3734313727 learned_addr learned my addr 172.21.15.29:0/3734313727 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:41:55.810 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:41:55.808+0000 7f057d166700 1 -- 172.21.15.29:0/3734313727 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f057819d990 con 0x7f0578101f00 2024-03-16T11:41:55.810 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:41:55.809+0000 7f057d166700 1 --2- 172.21.15.29:0/3734313727 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0578101f00 0x7f057819d450 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f0574005850 tx=0x7f0574005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:41:55.810 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:41:55.809+0000 7f056e7fc700 1 -- 172.21.15.29:0/3734313727 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0574026070 con 0x7f0578101f00 2024-03-16T11:41:55.810 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:41:55.809+0000 7f056e7fc700 1 -- 172.21.15.29:0/3734313727 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f057400bbd0 con 0x7f0578101f00 2024-03-16T11:41:55.810 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:41:55.809+0000 7f056e7fc700 1 -- 172.21.15.29:0/3734313727 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0574003970 con 0x7f0578101f00 2024-03-16T11:41:55.810 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:41:55.809+0000 7f057f3ca700 1 -- 172.21.15.29:0/3734313727 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f057819dc80 con 0x7f0578101f00 2024-03-16T11:41:55.810 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:41:55.809+0000 7f057f3ca700 1 -- 172.21.15.29:0/3734313727 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0578197610 con 0x7f0578101f00 2024-03-16T11:41:55.810 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:41:55.810+0000 7f056e7fc700 1 -- 172.21.15.29:0/3734313727 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 13) ==== 46162+0+0 (secure 0 0 0) 0x7f0574029070 con 0x7f0578101f00 2024-03-16T11:41:55.810 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:41:55.810+0000 7f057f3ca700 1 -- 172.21.15.29:0/3734313727 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f05781033e0 con 0x7f0578101f00 2024-03-16T11:41:55.811 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:41:55.810+0000 7f056e7fc700 1 --2- 172.21.15.29:0/3734313727 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f0564038fc0 0x7f056403b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:41:55.811 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:41:55.810+0000 7f056e7fc700 1 -- 172.21.15.29:0/3734313727 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f057401e070 con 0x7f0578101f00 2024-03-16T11:41:55.811 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:41:55.810+0000 7f057c965700 1 --2- 172.21.15.29:0/3734313727 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f0564038fc0 0x7f056403b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:41:55.811 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:41:55.810+0000 7f057c965700 1 --2- 172.21.15.29:0/3734313727 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f0564038fc0 0x7f056403b480 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f0568006fd0 tx=0x7f0568006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:41:55.814 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:41:55.815+0000 7f056e7fc700 1 -- 172.21.15.29:0/3734313727 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f0574013070 con 0x7f0578101f00 2024-03-16T11:41:55.837 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:41:55 smithi029 ceph-mon[31975]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T11:41:56.026 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:41:56.027+0000 7f057f3ca700 1 -- 172.21.15.29:0/3734313727 --> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi154", "target": ["mon-mgr", ""]}) -- 0x7f0578198b40 con 0x7f0564038fc0 2024-03-16T11:41:56.897 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:41:56 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:41:56.897 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:41:56 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:41:56.898 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:41:56 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:41:56.898 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:41:56 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:41:56.898 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:41:56 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi152", "name": "osd_memory_target"}]: dispatch 2024-03-16T11:41:56.898 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:41:56 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:41:56.898 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:41:56 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T11:41:56.898 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:41:56 smithi029 ceph-mon[31975]: Updating smithi152:/etc/ceph/ceph.conf 2024-03-16T11:41:56.898 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:41:56 smithi029 ceph-mon[31975]: Updating smithi152:/var/lib/ceph/d1887f28-e389-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-16T11:41:56.898 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:41:56 smithi029 ceph-mon[31975]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi154", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T11:41:56.898 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:41:56 smithi029 ceph-mon[31975]: Updating smithi152:/etc/ceph/ceph.client.admin.keyring 2024-03-16T11:41:56.898 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:41:56 smithi029 ceph-mon[31975]: Updating smithi152:/var/lib/ceph/d1887f28-e389-11ee-95c9-87774f69a715/config/ceph.client.admin.keyring 2024-03-16T11:41:56.898 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:41:56 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:41:56.898 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:41:56 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:41:56.898 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:41:56 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:41:58.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:41:57 smithi029 ceph-mon[31975]: Deploying cephadm binary to smithi154 2024-03-16T11:41:58.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:41:57 smithi029 ceph-mon[31975]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T11:42:00.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:41:59 smithi029 ceph-mon[31975]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T11:42:00.296 INFO:teuthology.orchestra.run.smithi029.stdout:Added host 'smithi154' with addr '172.21.15.154' 2024-03-16T11:42:00.297 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:42:00.296+0000 7f056e7fc700 1 -- 172.21.15.29:0/3734313727 <== mgr.14152 v2:172.21.15.29:6800/4192064640 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+49 (secure 0 0 0) 0x7f0578198b40 con 0x7f0564038fc0 2024-03-16T11:42:00.298 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:42:00.299+0000 7f057f3ca700 1 -- 172.21.15.29:0/3734313727 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f0564038fc0 msgr2=0x7f056403b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:42:00.298 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:42:00.299+0000 7f057f3ca700 1 --2- 172.21.15.29:0/3734313727 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f0564038fc0 0x7f056403b480 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f0568006fd0 tx=0x7f0568006e40 comp rx=0 tx=0).stop 2024-03-16T11:42:00.298 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:42:00.299+0000 7f057f3ca700 1 -- 172.21.15.29:0/3734313727 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0578101f00 msgr2=0x7f057819d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:42:00.298 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:42:00.299+0000 7f057f3ca700 1 --2- 172.21.15.29:0/3734313727 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0578101f00 0x7f057819d450 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f0574005850 tx=0x7f0574005030 comp rx=0 tx=0).stop 2024-03-16T11:42:00.299 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:42:00.299+0000 7f057f3ca700 1 -- 172.21.15.29:0/3734313727 shutdown_connections 2024-03-16T11:42:00.299 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:42:00.299+0000 7f057f3ca700 1 --2- 172.21.15.29:0/3734313727 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f0564038fc0 0x7f056403b480 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:42:00.299 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:42:00.299+0000 7f057f3ca700 1 --2- 172.21.15.29:0/3734313727 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0578101f00 0x7f057819d450 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:42:00.299 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:42:00.299+0000 7f057f3ca700 1 -- 172.21.15.29:0/3734313727 >> 172.21.15.29:0/3734313727 conn(0x7f05780fd7b0 msgr2=0x7f05781038d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:42:00.299 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:42:00.299+0000 7f057f3ca700 1 -- 172.21.15.29:0/3734313727 shutdown_connections 2024-03-16T11:42:00.299 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:42:00.299+0000 7f057f3ca700 1 -- 172.21.15.29:0/3734313727 wait complete. 2024-03-16T11:42:00.901 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d1887f28-e389-11ee-95c9-87774f69a715 -- ceph orch host ls --format=json 2024-03-16T11:42:01.147 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/d1887f28-e389-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T11:42:01.588 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:01 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:42:01.588 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:01 smithi029 ceph-mon[31975]: Added host smithi154 2024-03-16T11:42:01.588 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:01 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T11:42:01.588 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:01 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:42:02.507 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:02 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:42:02.508 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:02 smithi029 ceph-mon[31975]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T11:42:02.735 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:42:02.735+0000 7f534c93b700 1 -- 172.21.15.29:0/1007763002 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5348102eb0 msgr2=0x7f5348103290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:42:02.735 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:42:02.736+0000 7f534c93b700 1 --2- 172.21.15.29:0/1007763002 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5348102eb0 0x7f5348103290 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f534000b0a0 tx=0x7f534000b3b0 comp rx=0 tx=0).stop 2024-03-16T11:42:02.735 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:42:02.736+0000 7f534c93b700 1 -- 172.21.15.29:0/1007763002 shutdown_connections 2024-03-16T11:42:02.735 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:42:02.736+0000 7f534c93b700 1 --2- 172.21.15.29:0/1007763002 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5348102eb0 0x7f5348103290 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:42:02.735 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:42:02.736+0000 7f534c93b700 1 -- 172.21.15.29:0/1007763002 >> 172.21.15.29:0/1007763002 conn(0x7f53480fc660 msgr2=0x7f53480fea80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:42:02.735 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:42:02.736+0000 7f534c93b700 1 -- 172.21.15.29:0/1007763002 shutdown_connections 2024-03-16T11:42:02.736 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:42:02.736+0000 7f534c93b700 1 -- 172.21.15.29:0/1007763002 wait complete. 2024-03-16T11:42:02.737 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:42:02.736+0000 7f534c93b700 1 Processor -- start 2024-03-16T11:42:02.737 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:42:02.736+0000 7f534c93b700 1 -- start start 2024-03-16T11:42:02.737 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:42:02.737+0000 7f534c93b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5348102eb0 0x7f534819d460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:42:02.737 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:42:02.737+0000 7f534c93b700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f534810c330 con 0x7f5348102eb0 2024-03-16T11:42:02.738 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:42:02.737+0000 7f534659c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5348102eb0 0x7f534819d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:42:02.738 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:42:02.737+0000 7f534659c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5348102eb0 0x7f534819d460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41682/0 (socket says 172.21.15.29:41682) 2024-03-16T11:42:02.738 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:42:02.737+0000 7f534659c700 1 -- 172.21.15.29:0/1193201933 learned_addr learned my addr 172.21.15.29:0/1193201933 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:42:02.738 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:42:02.737+0000 7f534659c700 1 -- 172.21.15.29:0/1193201933 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f534819d9a0 con 0x7f5348102eb0 2024-03-16T11:42:02.738 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:42:02.737+0000 7f534659c700 1 --2- 172.21.15.29:0/1193201933 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5348102eb0 0x7f534819d460 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f5340000f80 tx=0x7f534000bd20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:42:02.738 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:42:02.737+0000 7f533f7fe700 1 -- 172.21.15.29:0/1193201933 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5340020020 con 0x7f5348102eb0 2024-03-16T11:42:02.738 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:42:02.737+0000 7f533f7fe700 1 -- 172.21.15.29:0/1193201933 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5340018040 con 0x7f5348102eb0 2024-03-16T11:42:02.738 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:42:02.737+0000 7f533f7fe700 1 -- 172.21.15.29:0/1193201933 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5340017070 con 0x7f5348102eb0 2024-03-16T11:42:02.738 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:42:02.737+0000 7f534c93b700 1 -- 172.21.15.29:0/1193201933 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f534819dc30 con 0x7f5348102eb0 2024-03-16T11:42:02.738 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:42:02.737+0000 7f534c93b700 1 -- 172.21.15.29:0/1193201933 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5348197620 con 0x7f5348102eb0 2024-03-16T11:42:02.739 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:42:02.738+0000 7f533f7fe700 1 -- 172.21.15.29:0/1193201933 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 13) ==== 46162+0+0 (secure 0 0 0) 0x7f5340023020 con 0x7f5348102eb0 2024-03-16T11:42:02.739 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:42:02.738+0000 7f534c93b700 1 -- 172.21.15.29:0/1193201933 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f534806a9d0 con 0x7f5348102eb0 2024-03-16T11:42:02.739 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:42:02.738+0000 7f533f7fe700 1 --2- 172.21.15.29:0/1193201933 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f53340390a0 0x7f533403b560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:42:02.739 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:42:02.739+0000 7f533f7fe700 1 -- 172.21.15.29:0/1193201933 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f534000f050 con 0x7f5348102eb0 2024-03-16T11:42:02.739 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:42:02.739+0000 7f5345d9b700 1 --2- 172.21.15.29:0/1193201933 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f53340390a0 0x7f533403b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:42:02.739 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:42:02.739+0000 7f5345d9b700 1 --2- 172.21.15.29:0/1193201933 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f53340390a0 0x7f533403b560 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f5330006fd0 tx=0x7f5330006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:42:02.743 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:42:02.744+0000 7f533f7fe700 1 -- 172.21.15.29:0/1193201933 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f534002a020 con 0x7f5348102eb0 2024-03-16T11:42:02.957 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:42:02.957+0000 7f534c93b700 1 -- 172.21.15.29:0/1193201933 --> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f5348198b50 con 0x7f53340390a0 2024-03-16T11:42:02.958 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:42:02.958+0000 7f533f7fe700 1 -- 172.21.15.29:0/1193201933 <== mgr.14152 v2:172.21.15.29:6800/4192064640 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+240 (secure 0 0 0) 0x7f5348198b50 con 0x7f53340390a0 2024-03-16T11:42:02.958 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-16T11:42:02.958 INFO:teuthology.orchestra.run.smithi029.stdout:[{"addr": "172.21.15.29", "hostname": "smithi029", "labels": [], "status": ""}, {"addr": "172.21.15.152", "hostname": "smithi152", "labels": [], "status": ""}, {"addr": "172.21.15.154", "hostname": "smithi154", "labels": [], "status": ""}] 2024-03-16T11:42:02.960 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:42:02.960+0000 7f534c93b700 1 -- 172.21.15.29:0/1193201933 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f53340390a0 msgr2=0x7f533403b560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:42:02.960 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:42:02.960+0000 7f534c93b700 1 --2- 172.21.15.29:0/1193201933 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f53340390a0 0x7f533403b560 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f5330006fd0 tx=0x7f5330006e40 comp rx=0 tx=0).stop 2024-03-16T11:42:02.960 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:42:02.960+0000 7f534c93b700 1 -- 172.21.15.29:0/1193201933 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5348102eb0 msgr2=0x7f534819d460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:42:02.960 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:42:02.960+0000 7f534c93b700 1 --2- 172.21.15.29:0/1193201933 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5348102eb0 0x7f534819d460 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f5340000f80 tx=0x7f534000bd20 comp rx=0 tx=0).stop 2024-03-16T11:42:02.960 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:42:02.961+0000 7f534c93b700 1 -- 172.21.15.29:0/1193201933 shutdown_connections 2024-03-16T11:42:02.960 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:42:02.961+0000 7f534c93b700 1 --2- 172.21.15.29:0/1193201933 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f53340390a0 0x7f533403b560 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:42:02.960 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:42:02.961+0000 7f534c93b700 1 --2- 172.21.15.29:0/1193201933 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5348102eb0 0x7f534819d460 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:42:02.961 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:42:02.961+0000 7f534c93b700 1 -- 172.21.15.29:0/1193201933 >> 172.21.15.29:0/1193201933 conn(0x7f53480fc660 msgr2=0x7f5348103c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:42:02.961 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:42:02.961+0000 7f534c93b700 1 -- 172.21.15.29:0/1193201933 shutdown_connections 2024-03-16T11:42:02.961 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:42:02.961+0000 7f534c93b700 1 -- 172.21.15.29:0/1193201933 wait complete. 2024-03-16T11:42:03.567 INFO:tasks.cephadm:Setting crush tunables to default 2024-03-16T11:42:03.567 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d1887f28-e389-11ee-95c9-87774f69a715 -- ceph osd crush tunables default 2024-03-16T11:42:03.808 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/d1887f28-e389-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T11:42:04.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:03 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:42:04.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:03 smithi029 ceph-mon[31975]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-16T11:42:04.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:03 smithi029 ceph-mon[31975]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T11:42:05.348 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:42:05.347+0000 7f244a236700 1 -- 172.21.15.29:0/3616224862 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2444101f10 msgr2=0x7f24441022f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:42:05.349 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:42:05.347+0000 7f244a236700 1 --2- 172.21.15.29:0/3616224862 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2444101f10 0x7f24441022f0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f2434009910 tx=0x7f2434009c20 comp rx=0 tx=0).stop 2024-03-16T11:42:05.349 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:42:05.347+0000 7f244a236700 1 -- 172.21.15.29:0/3616224862 shutdown_connections 2024-03-16T11:42:05.349 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:42:05.347+0000 7f244a236700 1 --2- 172.21.15.29:0/3616224862 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2444101f10 0x7f24441022f0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:42:05.349 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:42:05.347+0000 7f244a236700 1 -- 172.21.15.29:0/3616224862 >> 172.21.15.29:0/3616224862 conn(0x7f24440fd7e0 msgr2=0x7f24440ffc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:42:05.349 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:42:05.347+0000 7f244a236700 1 -- 172.21.15.29:0/3616224862 shutdown_connections 2024-03-16T11:42:05.349 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:42:05.347+0000 7f244a236700 1 -- 172.21.15.29:0/3616224862 wait complete. 2024-03-16T11:42:05.349 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:42:05.347+0000 7f244a236700 1 Processor -- start 2024-03-16T11:42:05.349 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:42:05.347+0000 7f244a236700 1 -- start start 2024-03-16T11:42:05.349 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:42:05.347+0000 7f244a236700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2444101f10 0x7f24440745d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:42:05.349 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:42:05.347+0000 7f244a236700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f244410c2e0 con 0x7f2444101f10 2024-03-16T11:42:05.349 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:42:05.348+0000 7f24437fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2444101f10 0x7f24440745d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:42:05.350 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:42:05.348+0000 7f24437fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2444101f10 0x7f24440745d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41702/0 (socket says 172.21.15.29:41702) 2024-03-16T11:42:05.350 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:42:05.348+0000 7f24437fe700 1 -- 172.21.15.29:0/1485977759 learned_addr learned my addr 172.21.15.29:0/1485977759 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:42:05.350 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:42:05.348+0000 7f24437fe700 1 -- 172.21.15.29:0/1485977759 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24440764c0 con 0x7f2444101f10 2024-03-16T11:42:05.350 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:42:05.348+0000 7f24437fe700 1 --2- 172.21.15.29:0/1485977759 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2444101f10 0x7f24440745d0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f2434005850 tx=0x7f2434005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:42:05.350 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:42:05.348+0000 7f24417fa700 1 -- 172.21.15.29:0/1485977759 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2434026070 con 0x7f2444101f10 2024-03-16T11:42:05.350 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:42:05.348+0000 7f24417fa700 1 -- 172.21.15.29:0/1485977759 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f243400bc30 con 0x7f2444101f10 2024-03-16T11:42:05.350 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:42:05.348+0000 7f244a236700 1 -- 172.21.15.29:0/1485977759 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2444074b70 con 0x7f2444101f10 2024-03-16T11:42:05.350 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:42:05.348+0000 7f24417fa700 1 -- 172.21.15.29:0/1485977759 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2434003a20 con 0x7f2444101f10 2024-03-16T11:42:05.350 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:42:05.348+0000 7f244a236700 1 -- 172.21.15.29:0/1485977759 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2444075030 con 0x7f2444101f10 2024-03-16T11:42:05.351 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:42:05.349+0000 7f24417fa700 1 -- 172.21.15.29:0/1485977759 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 13) ==== 46162+0+0 (secure 0 0 0) 0x7f2434029070 con 0x7f2444101f10 2024-03-16T11:42:05.351 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:42:05.349+0000 7f244a236700 1 -- 172.21.15.29:0/1485977759 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f24441033f0 con 0x7f2444101f10 2024-03-16T11:42:05.351 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:42:05.349+0000 7f24417fa700 1 --2- 172.21.15.29:0/1485977759 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f2430038fc0 0x7f243003b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:42:05.351 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:42:05.349+0000 7f24417fa700 1 -- 172.21.15.29:0/1485977759 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f243401e070 con 0x7f2444101f10 2024-03-16T11:42:05.351 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:42:05.349+0000 7f243bfff700 1 --2- 172.21.15.29:0/1485977759 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f2430038fc0 0x7f243003b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:42:05.351 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:42:05.350+0000 7f243bfff700 1 --2- 172.21.15.29:0/1485977759 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f2430038fc0 0x7f243003b480 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f242c006fd0 tx=0x7f242c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:42:05.354 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:42:05.354+0000 7f24417fa700 1 -- 172.21.15.29:0/1485977759 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f2434013070 con 0x7f2444101f10 2024-03-16T11:42:05.565 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:42:05.564+0000 7f244a236700 1 -- 172.21.15.29:0/1485977759 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) -- 0x7f2444075b00 con 0x7f2444101f10 2024-03-16T11:42:05.615 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:42:05.615+0000 7f24417fa700 1 -- 172.21.15.29:0/1485977759 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) ==== 124+0+0 (secure 0 0 0) 0x7f2434030020 con 0x7f2444101f10 2024-03-16T11:42:05.618 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:42:05.618+0000 7f244a236700 1 -- 172.21.15.29:0/1485977759 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f2430038fc0 msgr2=0x7f243003b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:42:05.618 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:42:05.618+0000 7f244a236700 1 --2- 172.21.15.29:0/1485977759 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f2430038fc0 0x7f243003b480 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f242c006fd0 tx=0x7f242c006e40 comp rx=0 tx=0).stop 2024-03-16T11:42:05.618 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:42:05.618+0000 7f244a236700 1 -- 172.21.15.29:0/1485977759 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2444101f10 msgr2=0x7f24440745d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:42:05.618 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:42:05.618+0000 7f244a236700 1 --2- 172.21.15.29:0/1485977759 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2444101f10 0x7f24440745d0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f2434005850 tx=0x7f2434005030 comp rx=0 tx=0).stop 2024-03-16T11:42:05.618 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:42:05.618+0000 7f244a236700 1 -- 172.21.15.29:0/1485977759 shutdown_connections 2024-03-16T11:42:05.618 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:42:05.618+0000 7f244a236700 1 --2- 172.21.15.29:0/1485977759 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f2430038fc0 0x7f243003b480 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:42:05.618 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:42:05.618+0000 7f244a236700 1 --2- 172.21.15.29:0/1485977759 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2444101f10 0x7f24440745d0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:42:05.619 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:42:05.618+0000 7f244a236700 1 -- 172.21.15.29:0/1485977759 >> 172.21.15.29:0/1485977759 conn(0x7f24440fd7e0 msgr2=0x7f2444104d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:42:05.619 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:42:05.618+0000 7f244a236700 1 -- 172.21.15.29:0/1485977759 shutdown_connections 2024-03-16T11:42:05.619 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:42:05.618+0000 7f244a236700 1 -- 172.21.15.29:0/1485977759 wait complete. 2024-03-16T11:42:05.619 INFO:teuthology.orchestra.run.smithi029.stderr:adjusted tunables profile to default 2024-03-16T11:42:05.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:05 smithi029 ceph-mon[31975]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T11:42:05.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:05 smithi029 ceph-mon[31975]: from='client.? 172.21.15.29:0/1485977759' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-16T11:42:06.207 INFO:tasks.cephadm:Adding mon.a on smithi029 2024-03-16T11:42:06.208 INFO:tasks.cephadm:Adding mon.b on smithi152 2024-03-16T11:42:06.208 INFO:tasks.cephadm:Adding mon.c on smithi154 2024-03-16T11:42:06.208 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d1887f28-e389-11ee-95c9-87774f69a715 -- ceph orch apply mon '3;smithi029:172.21.15.29=a;smithi152:172.21.15.152=b;smithi154:172.21.15.154=c' 2024-03-16T11:42:06.441 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-16T11:42:06.489 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-16T11:42:07.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:06 smithi029 ceph-mon[31975]: from='client.? 172.21.15.29:0/1485977759' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-16T11:42:07.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:06 smithi029 ceph-mon[31975]: osdmap e4: 0 total, 0 up, 0 in 2024-03-16T11:42:07.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:06 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:42:07.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:06 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:42:07.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:06 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:42:07.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:06 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:42:07.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:06 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi154", "name": "osd_memory_target"}]: dispatch 2024-03-16T11:42:07.089 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:06 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:42:07.089 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:06 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T11:42:07.089 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:06 smithi029 ceph-mon[31975]: Updating smithi154:/etc/ceph/ceph.conf 2024-03-16T11:42:07.089 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:06 smithi029 ceph-mon[31975]: Updating smithi154:/var/lib/ceph/d1887f28-e389-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-16T11:42:07.089 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:06 smithi029 ceph-mon[31975]: Updating smithi154:/etc/ceph/ceph.client.admin.keyring 2024-03-16T11:42:07.935 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:07.933+0000 7fb936cf7700 1 -- 172.21.15.154:0/2319319626 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb930100e10 msgr2=0x7fb9301011f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:42:07.935 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:07.933+0000 7fb936cf7700 1 --2- 172.21.15.154:0/2319319626 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb930100e10 0x7fb9301011f0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fb924009910 tx=0x7fb924009c20 comp rx=0 tx=0).stop 2024-03-16T11:42:07.935 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:07.933+0000 7fb936cf7700 1 -- 172.21.15.154:0/2319319626 shutdown_connections 2024-03-16T11:42:07.935 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:07.933+0000 7fb936cf7700 1 --2- 172.21.15.154:0/2319319626 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb930100e10 0x7fb9301011f0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:42:07.935 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:07.933+0000 7fb936cf7700 1 -- 172.21.15.154:0/2319319626 >> 172.21.15.154:0/2319319626 conn(0x7fb9300fc6c0 msgr2=0x7fb9300feae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:42:07.936 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:07.933+0000 7fb936cf7700 1 -- 172.21.15.154:0/2319319626 shutdown_connections 2024-03-16T11:42:07.936 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:07.933+0000 7fb936cf7700 1 -- 172.21.15.154:0/2319319626 wait complete. 2024-03-16T11:42:07.936 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:07.934+0000 7fb936cf7700 1 Processor -- start 2024-03-16T11:42:07.936 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:07.934+0000 7fb936cf7700 1 -- start start 2024-03-16T11:42:07.936 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:07.934+0000 7fb936cf7700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb930100e10 0x7fb93019f690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:42:07.936 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:07.934+0000 7fb936cf7700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fb93010b1e0 con 0x7fb930100e10 2024-03-16T11:42:07.936 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:07.934+0000 7fb934a93700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb930100e10 0x7fb93019f690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:42:07.936 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:07.934+0000 7fb934a93700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb930100e10 0x7fb93019f690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.154:50532/0 (socket says 172.21.15.154:50532) 2024-03-16T11:42:07.936 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:07.934+0000 7fb934a93700 1 -- 172.21.15.154:0/4220525677 learned_addr learned my addr 172.21.15.154:0/4220525677 (peer_addr_for_me v2:172.21.15.154:0/0) 2024-03-16T11:42:07.936 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:07.935+0000 7fb934a93700 1 -- 172.21.15.154:0/4220525677 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb93019fbd0 con 0x7fb930100e10 2024-03-16T11:42:07.936 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:07.935+0000 7fb934a93700 1 --2- 172.21.15.154:0/4220525677 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb930100e10 0x7fb93019f690 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fb924006010 tx=0x7fb924005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:42:07.937 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:07.935+0000 7fb92dffb700 1 -- 172.21.15.154:0/4220525677 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb924026070 con 0x7fb930100e10 2024-03-16T11:42:07.937 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:07.935+0000 7fb92dffb700 1 -- 172.21.15.154:0/4220525677 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb92400bbd0 con 0x7fb930100e10 2024-03-16T11:42:07.937 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:07.935+0000 7fb936cf7700 1 -- 172.21.15.154:0/4220525677 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb93019fec0 con 0x7fb930100e10 2024-03-16T11:42:07.937 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:07.935+0000 7fb92dffb700 1 -- 172.21.15.154:0/4220525677 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb924003910 con 0x7fb930100e10 2024-03-16T11:42:07.937 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:07.935+0000 7fb936cf7700 1 -- 172.21.15.154:0/4220525677 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb930199860 con 0x7fb930100e10 2024-03-16T11:42:07.938 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:07.936+0000 7fb936cf7700 1 -- 172.21.15.154:0/4220525677 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb9301022f0 con 0x7fb930100e10 2024-03-16T11:42:07.938 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:07.936+0000 7fb92dffb700 1 -- 172.21.15.154:0/4220525677 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 13) ==== 46162+0+0 (secure 0 0 0) 0x7fb924029070 con 0x7fb930100e10 2024-03-16T11:42:07.938 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:07.936+0000 7fb92dffb700 1 --2- 172.21.15.154:0/4220525677 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fb920038fc0 0x7fb92003b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:42:07.939 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:07.936+0000 7fb92dffb700 1 -- 172.21.15.154:0/4220525677 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fb92401e070 con 0x7fb930100e10 2024-03-16T11:42:07.939 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:07.936+0000 7fb92ffff700 1 --2- 172.21.15.154:0/4220525677 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fb920038fc0 0x7fb92003b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:42:07.939 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:07.937+0000 7fb92ffff700 1 --2- 172.21.15.154:0/4220525677 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fb920038fc0 0x7fb92003b480 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fb91c006fd0 tx=0x7fb91c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:42:07.941 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:07.942+0000 7fb92dffb700 1 -- 172.21.15.154:0/4220525677 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fb924013070 con 0x7fb930100e10 2024-03-16T11:42:08.148 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:08.148+0000 7fb936cf7700 1 -- 172.21.15.154:0/4220525677 --> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi029:172.21.15.29=a;smithi152:172.21.15.152=b;smithi154:172.21.15.154=c", "target": ["mon-mgr", ""]}) -- 0x7fb93019ae00 con 0x7fb920038fc0 2024-03-16T11:42:08.153 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:08.154+0000 7fb92dffb700 1 -- 172.21.15.154:0/4220525677 <== mgr.14152 v2:172.21.15.29:6800/4192064640 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7fb93019ae00 con 0x7fb920038fc0 2024-03-16T11:42:08.154 INFO:teuthology.orchestra.run.smithi154.stdout:Scheduled mon update... 2024-03-16T11:42:08.156 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:08.156+0000 7fb936cf7700 1 -- 172.21.15.154:0/4220525677 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fb920038fc0 msgr2=0x7fb92003b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:42:08.156 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:08.156+0000 7fb936cf7700 1 --2- 172.21.15.154:0/4220525677 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fb920038fc0 0x7fb92003b480 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fb91c006fd0 tx=0x7fb91c006e40 comp rx=0 tx=0).stop 2024-03-16T11:42:08.156 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:08.156+0000 7fb936cf7700 1 -- 172.21.15.154:0/4220525677 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb930100e10 msgr2=0x7fb93019f690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:42:08.156 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:08.156+0000 7fb936cf7700 1 --2- 172.21.15.154:0/4220525677 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb930100e10 0x7fb93019f690 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fb924006010 tx=0x7fb924005030 comp rx=0 tx=0).stop 2024-03-16T11:42:08.157 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:08.156+0000 7fb936cf7700 1 -- 172.21.15.154:0/4220525677 shutdown_connections 2024-03-16T11:42:08.157 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:08.156+0000 7fb936cf7700 1 --2- 172.21.15.154:0/4220525677 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fb920038fc0 0x7fb92003b480 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:42:08.157 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:08.156+0000 7fb936cf7700 1 --2- 172.21.15.154:0/4220525677 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb930100e10 0x7fb93019f690 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:42:08.157 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:08.156+0000 7fb936cf7700 1 -- 172.21.15.154:0/4220525677 >> 172.21.15.154:0/4220525677 conn(0x7fb9300fc6c0 msgr2=0x7fb93006a150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:42:08.157 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:08.156+0000 7fb936cf7700 1 -- 172.21.15.154:0/4220525677 shutdown_connections 2024-03-16T11:42:08.157 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:08.156+0000 7fb936cf7700 1 -- 172.21.15.154:0/4220525677 wait complete. 2024-03-16T11:42:08.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:07 smithi029 ceph-mon[31975]: Updating smithi154:/var/lib/ceph/d1887f28-e389-11ee-95c9-87774f69a715/config/ceph.client.admin.keyring 2024-03-16T11:42:08.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:07 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:42:08.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:07 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:42:08.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:07 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:42:08.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:07 smithi029 ceph-mon[31975]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T11:42:08.741 DEBUG:teuthology.orchestra.run.smithi152:mon.b> sudo journalctl -f -n 0 -u ceph-d1887f28-e389-11ee-95c9-87774f69a715@mon.b.service 2024-03-16T11:42:08.744 DEBUG:teuthology.orchestra.run.smithi154:mon.c> sudo journalctl -f -n 0 -u ceph-d1887f28-e389-11ee-95c9-87774f69a715@mon.c.service 2024-03-16T11:42:08.747 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-16T11:42:08.747 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d1887f28-e389-11ee-95c9-87774f69a715 -- ceph mon dump -f json 2024-03-16T11:42:09.421 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/d1887f28-e389-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-16T11:42:09.588 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:09 smithi029 ceph-mon[31975]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi029:172.21.15.29=a;smithi152:172.21.15.152=b;smithi154:172.21.15.154=c", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T11:42:09.588 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:09 smithi029 ceph-mon[31975]: Saving service mon spec with placement smithi029:172.21.15.29=a;smithi152:172.21.15.152=b;smithi154:172.21.15.154=c;count:3 2024-03-16T11:42:09.588 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:09 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:42:09.589 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:09 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T11:42:09.589 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:09 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:42:09.589 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:09 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T11:42:09.589 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:09 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:42:09.589 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:09 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-16T11:42:09.589 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:09 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:42:09.589 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:09 smithi029 ceph-mon[31975]: Deploying daemon mon.c on smithi154 2024-03-16T11:42:10.588 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:10 smithi029 ceph-mon[31975]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T11:42:12.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:11 smithi029 ceph-mon[31975]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T11:42:12.539 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:12.539+0000 7fdbab08f700 1 -- 172.21.15.154:0/128490710 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdba4101f10 msgr2=0x7fdba41022f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:42:12.539 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:12.539+0000 7fdbab08f700 1 --2- 172.21.15.154:0/128490710 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdba4101f10 0x7fdba41022f0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fdb9c00b0a0 tx=0x7fdb9c00b3b0 comp rx=0 tx=0).stop 2024-03-16T11:42:12.539 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:12.539+0000 7fdbab08f700 1 -- 172.21.15.154:0/128490710 shutdown_connections 2024-03-16T11:42:12.539 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:12.539+0000 7fdbab08f700 1 --2- 172.21.15.154:0/128490710 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdba4101f10 0x7fdba41022f0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:42:12.539 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:12.539+0000 7fdbab08f700 1 -- 172.21.15.154:0/128490710 >> 172.21.15.154:0/128490710 conn(0x7fdba40fd7c0 msgr2=0x7fdba40ffbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:42:12.539 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:12.539+0000 7fdbab08f700 1 -- 172.21.15.154:0/128490710 shutdown_connections 2024-03-16T11:42:12.539 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:12.539+0000 7fdbab08f700 1 -- 172.21.15.154:0/128490710 wait complete. 2024-03-16T11:42:12.540 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:12.540+0000 7fdbab08f700 1 Processor -- start 2024-03-16T11:42:12.540 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:12.540+0000 7fdbab08f700 1 -- start start 2024-03-16T11:42:12.540 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:12.540+0000 7fdbab08f700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdba4101f10 0x7fdba4197740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:42:12.540 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:12.540+0000 7fdbab08f700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fdba410c2e0 con 0x7fdba4101f10 2024-03-16T11:42:12.540 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:12.540+0000 7fdba8e2b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdba4101f10 0x7fdba4197740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:42:12.540 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:12.540+0000 7fdba8e2b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdba4101f10 0x7fdba4197740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.154:43660/0 (socket says 172.21.15.154:43660) 2024-03-16T11:42:12.540 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:12.540+0000 7fdba8e2b700 1 -- 172.21.15.154:0/3233572200 learned_addr learned my addr 172.21.15.154:0/3233572200 (peer_addr_for_me v2:172.21.15.154:0/0) 2024-03-16T11:42:12.540 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:12.540+0000 7fdba8e2b700 1 -- 172.21.15.154:0/3233572200 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdba4197c80 con 0x7fdba4101f10 2024-03-16T11:42:12.540 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:12.541+0000 7fdba8e2b700 1 --2- 172.21.15.154:0/3233572200 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdba4101f10 0x7fdba4197740 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fdb9c015040 tx=0x7fdb9c003d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:42:12.541 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:12.541+0000 7fdba1ffb700 1 -- 172.21.15.154:0/3233572200 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdb9c017040 con 0x7fdba4101f10 2024-03-16T11:42:12.541 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:12.541+0000 7fdba1ffb700 1 -- 172.21.15.154:0/3233572200 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdb9c022070 con 0x7fdba4101f10 2024-03-16T11:42:12.541 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:12.541+0000 7fdba1ffb700 1 -- 172.21.15.154:0/3233572200 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdb9c01d5d0 con 0x7fdba4101f10 2024-03-16T11:42:12.541 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:12.541+0000 7fdbab08f700 1 -- 172.21.15.154:0/3233572200 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdba4197f10 con 0x7fdba4101f10 2024-03-16T11:42:12.541 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:12.541+0000 7fdbab08f700 1 -- 172.21.15.154:0/3233572200 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdba419c010 con 0x7fdba4101f10 2024-03-16T11:42:12.542 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:12.542+0000 7fdba1ffb700 1 -- 172.21.15.154:0/3233572200 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 13) ==== 46162+0+0 (secure 0 0 0) 0x7fdb9c004000 con 0x7fdba4101f10 2024-03-16T11:42:12.542 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:12.542+0000 7fdba1ffb700 1 --2- 172.21.15.154:0/3233572200 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fdb90038fc0 0x7fdb9003b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:42:12.542 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:12.543+0000 7fdba1ffb700 1 -- 172.21.15.154:0/3233572200 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fdb9c00f050 con 0x7fdba4101f10 2024-03-16T11:42:12.542 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:12.543+0000 7fdbab08f700 1 -- 172.21.15.154:0/3233572200 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdba41033f0 con 0x7fdba4101f10 2024-03-16T11:42:12.542 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:12.543+0000 7fdba3fff700 1 --2- 172.21.15.154:0/3233572200 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fdb90038fc0 0x7fdb9003b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:42:12.542 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:12.543+0000 7fdba3fff700 1 --2- 172.21.15.154:0/3233572200 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fdb90038fc0 0x7fdb9003b480 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fdb94006fd0 tx=0x7fdb94006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:42:12.547 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:12.548+0000 7fdba1ffb700 1 -- 172.21.15.154:0/3233572200 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fdb9c007e20 con 0x7fdba4101f10 2024-03-16T11:42:12.751 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:12 smithi154 systemd[1]: Starting Ceph mon.c for d1887f28-e389-11ee-95c9-87774f69a715... 2024-03-16T11:42:12.838 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:12.839+0000 7fdbab08f700 1 -- 172.21.15.154:0/3233572200 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7fdba4063900 con 0x7fdba4101f10 2024-03-16T11:42:12.838 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:12.840+0000 7fdba1ffb700 1 -- 172.21.15.154:0/3233572200 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) ==== 95+0+745 (secure 0 0 0) 0x7fdb9c00bc10 con 0x7fdba4101f10 2024-03-16T11:42:12.838 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-03-16T11:42:12.839 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":1,"fsid":"d1887f28-e389-11ee-95c9-87774f69a715","modified":"2024-03-16T11:40:13.432436Z","created":"2024-03-16T11:40:13.432436Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-16T11:42:12.841 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:12.842+0000 7fdbab08f700 1 -- 172.21.15.154:0/3233572200 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fdb90038fc0 msgr2=0x7fdb9003b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:42:12.841 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:12.842+0000 7fdbab08f700 1 --2- 172.21.15.154:0/3233572200 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fdb90038fc0 0x7fdb9003b480 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fdb94006fd0 tx=0x7fdb94006e40 comp rx=0 tx=0).stop 2024-03-16T11:42:12.841 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:12.842+0000 7fdbab08f700 1 -- 172.21.15.154:0/3233572200 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdba4101f10 msgr2=0x7fdba4197740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:42:12.841 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:12.842+0000 7fdbab08f700 1 --2- 172.21.15.154:0/3233572200 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdba4101f10 0x7fdba4197740 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fdb9c015040 tx=0x7fdb9c003d50 comp rx=0 tx=0).stop 2024-03-16T11:42:12.841 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:12.842+0000 7fdbab08f700 1 -- 172.21.15.154:0/3233572200 shutdown_connections 2024-03-16T11:42:12.841 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:12.842+0000 7fdbab08f700 1 --2- 172.21.15.154:0/3233572200 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fdb90038fc0 0x7fdb9003b480 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:42:12.841 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:12.842+0000 7fdbab08f700 1 --2- 172.21.15.154:0/3233572200 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdba4101f10 0x7fdba4197740 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:42:12.841 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:12.842+0000 7fdbab08f700 1 -- 172.21.15.154:0/3233572200 >> 172.21.15.154:0/3233572200 conn(0x7fdba40fd7c0 msgr2=0x7fdba4104c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:42:12.842 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:12.842+0000 7fdbab08f700 1 -- 172.21.15.154:0/3233572200 shutdown_connections 2024-03-16T11:42:12.842 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:12.842+0000 7fdbab08f700 1 -- 172.21.15.154:0/3233572200 wait complete. 2024-03-16T11:42:12.842 INFO:teuthology.orchestra.run.smithi154.stderr:dumped monmap epoch 1 2024-03-16T11:42:13.023 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:12 smithi154 podman[36288]: 2024-03-16 11:42:12.752651064 +0000 UTC m=+0.101446746 container create 79732bdfb11e0070e64dcea232db0dced05450c207f15255f8c8ea2eb58f0fb1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d1887f28-e389-11ee-95c9-87774f69a715-mon-c, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-03-16T11:42:13.023 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:12 smithi154 podman[36288]: 2024-03-16 11:42:12.669486046 +0000 UTC m=+0.018281748 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-16T11:42:13.023 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:12 smithi154 podman[36288]: 2024-03-16 11:42:12.997490631 +0000 UTC m=+0.346286312 container init 79732bdfb11e0070e64dcea232db0dced05450c207f15255f8c8ea2eb58f0fb1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d1887f28-e389-11ee-95c9-87774f69a715-mon-c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True) 2024-03-16T11:42:13.023 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 podman[36288]: 2024-03-16 11:42:13.000798371 +0000 UTC m=+0.349594062 container start 79732bdfb11e0070e64dcea232db0dced05450c207f15255f8c8ea2eb58f0fb1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d1887f28-e389-11ee-95c9-87774f69a715-mon-c, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240307, ceph=True, org.label-schema.license=GPLv2) 2024-03-16T11:42:13.296 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: set uid:gid to 167:167 (ceph:ceph) 2024-03-16T11:42:13.296 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: ceph version 19.0.0-2162-g3d78f5ba (3d78f5ba07bf18880172c07d4d7b4ac00c9d30af) squid (dev), process ceph-mon, pid 2 2024-03-16T11:42:13.296 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: pidfile_write: ignore empty --pid-file 2024-03-16T11:42:13.296 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: load: jerasure load: lrc 2024-03-16T11:42:13.296 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: rocksdb: RocksDB version: 7.9.2 2024-03-16T11:42:13.296 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: rocksdb: Git sha 0 2024-03-16T11:42:13.296 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: rocksdb: Compile date 2024-03-15 00:33:34 2024-03-16T11:42:13.296 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: rocksdb: DB SUMMARY 2024-03-16T11:42:13.296 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: rocksdb: DB Session ID: KRA2N8S36Z0RRJFHVNV0 2024-03-16T11:42:13.296 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: rocksdb: CURRENT file: CURRENT 2024-03-16T11:42:13.297 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: rocksdb: IDENTITY file: IDENTITY 2024-03-16T11:42:13.297 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-03-16T11:42:13.297 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2024-03-16T11:42:13.297 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 511 ; 2024-03-16T11:42:13.297 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: rocksdb: Options.error_if_exists: 0 2024-03-16T11:42:13.297 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: rocksdb: Options.create_if_missing: 0 2024-03-16T11:42:13.297 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: rocksdb: Options.paranoid_checks: 1 2024-03-16T11:42:13.297 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-16T11:42:13.297 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-16T11:42:13.297 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-16T11:42:13.297 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: rocksdb: Options.env: 0x55eb2b2c8c60 2024-03-16T11:42:13.297 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: rocksdb: Options.fs: PosixFileSystem 2024-03-16T11:42:13.298 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: rocksdb: Options.info_log: 0x55eb2d195ac0 2024-03-16T11:42:13.298 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: rocksdb: Options.max_file_opening_threads: 16 2024-03-16T11:42:13.298 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: rocksdb: Options.statistics: (nil) 2024-03-16T11:42:13.298 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: rocksdb: Options.use_fsync: 0 2024-03-16T11:42:13.298 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: rocksdb: Options.max_log_file_size: 0 2024-03-16T11:42:13.298 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-16T11:42:13.298 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-16T11:42:13.298 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: rocksdb: Options.keep_log_file_num: 1000 2024-03-16T11:42:13.298 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: rocksdb: Options.recycle_log_file_num: 0 2024-03-16T11:42:13.298 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: rocksdb: Options.allow_fallocate: 1 2024-03-16T11:42:13.298 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: rocksdb: Options.allow_mmap_reads: 0 2024-03-16T11:42:13.299 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: rocksdb: Options.allow_mmap_writes: 0 2024-03-16T11:42:13.299 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: rocksdb: Options.use_direct_reads: 0 2024-03-16T11:42:13.299 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-16T11:42:13.299 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: rocksdb: Options.create_missing_column_families: 0 2024-03-16T11:42:13.299 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: rocksdb: Options.db_log_dir: 2024-03-16T11:42:13.299 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: rocksdb: Options.wal_dir: 2024-03-16T11:42:13.299 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-16T11:42:13.299 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-16T11:42:13.299 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-16T11:42:13.299 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-16T11:42:13.299 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-16T11:42:13.299 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-16T11:42:13.299 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: rocksdb: Options.advise_random_on_open: 1 2024-03-16T11:42:13.300 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: rocksdb: Options.db_write_buffer_size: 0 2024-03-16T11:42:13.300 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: rocksdb: Options.write_buffer_manager: 0x55eb2d1681e0 2024-03-16T11:42:13.300 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-16T11:42:13.300 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-16T11:42:13.300 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-16T11:42:13.300 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: rocksdb: Options.rate_limiter: (nil) 2024-03-16T11:42:13.300 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-16T11:42:13.300 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: rocksdb: Options.wal_recovery_mode: 2 2024-03-16T11:42:13.300 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: rocksdb: Options.enable_thread_tracking: 0 2024-03-16T11:42:13.300 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: rocksdb: Options.enable_pipelined_write: 0 2024-03-16T11:42:13.300 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: rocksdb: Options.unordered_write: 0 2024-03-16T11:42:13.300 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-16T11:42:13.300 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-16T11:42:13.301 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-16T11:42:13.301 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-16T11:42:13.301 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: rocksdb: Options.row_cache: None 2024-03-16T11:42:13.301 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: rocksdb: Options.wal_filter: None 2024-03-16T11:42:13.301 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-16T11:42:13.301 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: rocksdb: Options.allow_ingest_behind: 0 2024-03-16T11:42:13.301 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: rocksdb: Options.two_write_queues: 0 2024-03-16T11:42:13.301 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: rocksdb: Options.manual_wal_flush: 0 2024-03-16T11:42:13.301 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: rocksdb: Options.wal_compression: 0 2024-03-16T11:42:13.301 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: rocksdb: Options.atomic_flush: 0 2024-03-16T11:42:13.301 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-16T11:42:13.301 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-16T11:42:13.301 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-16T11:42:13.301 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: rocksdb: Options.log_readahead_size: 0 2024-03-16T11:42:13.302 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-16T11:42:13.302 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: rocksdb: Options.best_efforts_recovery: 0 2024-03-16T11:42:13.302 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-16T11:42:13.302 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-16T11:42:13.302 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: rocksdb: Options.allow_data_in_errors: 0 2024-03-16T11:42:13.302 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: rocksdb: Options.db_host_id: __hostname__ 2024-03-16T11:42:13.302 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-16T11:42:13.304 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: rocksdb: Options.max_background_jobs: 2 2024-03-16T11:42:13.304 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: rocksdb: Options.max_background_compactions: -1 2024-03-16T11:42:13.304 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: rocksdb: Options.max_subcompactions: 1 2024-03-16T11:42:13.304 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-16T11:42:13.304 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-16T11:42:13.304 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-16T11:42:13.304 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: rocksdb: Options.max_total_wal_size: 0 2024-03-16T11:42:13.304 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-16T11:42:13.304 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-16T11:42:13.304 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-16T11:42:13.305 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-16T11:42:13.305 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: rocksdb: Options.max_open_files: -1 2024-03-16T11:42:13.305 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: rocksdb: Options.bytes_per_sync: 0 2024-03-16T11:42:13.305 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-16T11:42:13.305 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-16T11:42:13.305 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: rocksdb: Options.compaction_readahead_size: 0 2024-03-16T11:42:13.305 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: rocksdb: Options.max_background_flushes: -1 2024-03-16T11:42:13.305 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: rocksdb: Compression algorithms supported: 2024-03-16T11:42:13.305 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-16T11:42:13.305 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: rocksdb: kZSTD supported: 0 2024-03-16T11:42:13.305 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: rocksdb: kXpressCompression supported: 0 2024-03-16T11:42:13.305 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: rocksdb: kLZ4HCCompression supported: 1 2024-03-16T11:42:13.305 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: rocksdb: kZlibCompression supported: 1 2024-03-16T11:42:13.305 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: rocksdb: kSnappyCompression supported: 1 2024-03-16T11:42:13.306 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: rocksdb: kLZ4Compression supported: 1 2024-03-16T11:42:13.306 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: rocksdb: kBZip2Compression supported: 0 2024-03-16T11:42:13.306 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-16T11:42:13.306 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-16T11:42:13.306 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 2024-03-16T11:42:13.306 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-16T11:42:13.306 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-16T11:42:13.306 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: rocksdb: Options.merge_operator: 2024-03-16T11:42:13.306 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: rocksdb: Options.compaction_filter: None 2024-03-16T11:42:13.306 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: rocksdb: Options.compaction_filter_factory: None 2024-03-16T11:42:13.306 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: rocksdb: Options.sst_partitioner_factory: None 2024-03-16T11:42:13.306 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-16T11:42:13.307 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-16T11:42:13.307 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55eb2d195bc0) 2024-03-16T11:42:13.307 INFO:journalctl@ceph.mon.c.smithi154.stdout: cache_index_and_filter_blocks: 1 2024-03-16T11:42:13.307 INFO:journalctl@ceph.mon.c.smithi154.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-16T11:42:13.307 INFO:journalctl@ceph.mon.c.smithi154.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-16T11:42:13.307 INFO:journalctl@ceph.mon.c.smithi154.stdout: pin_top_level_index_and_filter: 1 2024-03-16T11:42:13.307 INFO:journalctl@ceph.mon.c.smithi154.stdout: index_type: 0 2024-03-16T11:42:13.307 INFO:journalctl@ceph.mon.c.smithi154.stdout: data_block_index_type: 0 2024-03-16T11:42:13.307 INFO:journalctl@ceph.mon.c.smithi154.stdout: index_shortening: 1 2024-03-16T11:42:13.307 INFO:journalctl@ceph.mon.c.smithi154.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-16T11:42:13.307 INFO:journalctl@ceph.mon.c.smithi154.stdout: checksum: 4 2024-03-16T11:42:13.307 INFO:journalctl@ceph.mon.c.smithi154.stdout: no_block_cache: 0 2024-03-16T11:42:13.307 INFO:journalctl@ceph.mon.c.smithi154.stdout: block_cache: 0x55eb2d19b090 2024-03-16T11:42:13.307 INFO:journalctl@ceph.mon.c.smithi154.stdout: block_cache_name: BinnedLRUCache 2024-03-16T11:42:13.308 INFO:journalctl@ceph.mon.c.smithi154.stdout: block_cache_options: 2024-03-16T11:42:13.308 INFO:journalctl@ceph.mon.c.smithi154.stdout: capacity : 536870912 2024-03-16T11:42:13.308 INFO:journalctl@ceph.mon.c.smithi154.stdout: num_shard_bits : 4 2024-03-16T11:42:13.308 INFO:journalctl@ceph.mon.c.smithi154.stdout: strict_capacity_limit : 0 2024-03-16T11:42:13.308 INFO:journalctl@ceph.mon.c.smithi154.stdout: high_pri_pool_ratio: 0.000 2024-03-16T11:42:13.308 INFO:journalctl@ceph.mon.c.smithi154.stdout: block_cache_compressed: (nil) 2024-03-16T11:42:13.308 INFO:journalctl@ceph.mon.c.smithi154.stdout: persistent_cache: (nil) 2024-03-16T11:42:13.308 INFO:journalctl@ceph.mon.c.smithi154.stdout: block_size: 4096 2024-03-16T11:42:13.308 INFO:journalctl@ceph.mon.c.smithi154.stdout: block_size_deviation: 10 2024-03-16T11:42:13.308 INFO:journalctl@ceph.mon.c.smithi154.stdout: block_restart_interval: 16 2024-03-16T11:42:13.308 INFO:journalctl@ceph.mon.c.smithi154.stdout: index_block_restart_interval: 1 2024-03-16T11:42:13.308 INFO:journalctl@ceph.mon.c.smithi154.stdout: metadata_block_size: 4096 2024-03-16T11:42:13.308 INFO:journalctl@ceph.mon.c.smithi154.stdout: partition_filters: 0 2024-03-16T11:42:13.309 INFO:journalctl@ceph.mon.c.smithi154.stdout: use_delta_encoding: 1 2024-03-16T11:42:13.309 INFO:journalctl@ceph.mon.c.smithi154.stdout: filter_policy: bloomfilter 2024-03-16T11:42:13.309 INFO:journalctl@ceph.mon.c.smithi154.stdout: whole_key_filtering: 1 2024-03-16T11:42:13.309 INFO:journalctl@ceph.mon.c.smithi154.stdout: verify_compression: 0 2024-03-16T11:42:13.309 INFO:journalctl@ceph.mon.c.smithi154.stdout: read_amp_bytes_per_bit: 0 2024-03-16T11:42:13.309 INFO:journalctl@ceph.mon.c.smithi154.stdout: format_version: 5 2024-03-16T11:42:13.309 INFO:journalctl@ceph.mon.c.smithi154.stdout: enable_index_compression: 1 2024-03-16T11:42:13.309 INFO:journalctl@ceph.mon.c.smithi154.stdout: block_align: 0 2024-03-16T11:42:13.309 INFO:journalctl@ceph.mon.c.smithi154.stdout: max_auto_readahead_size: 262144 2024-03-16T11:42:13.309 INFO:journalctl@ceph.mon.c.smithi154.stdout: prepopulate_block_cache: 0 2024-03-16T11:42:13.309 INFO:journalctl@ceph.mon.c.smithi154.stdout: initial_auto_readahead_size: 8192 2024-03-16T11:42:13.309 INFO:journalctl@ceph.mon.c.smithi154.stdout: num_file_reads_for_auto_readahead: 2 2024-03-16T11:42:13.309 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: rocksdb: Options.write_buffer_size: 33554432 2024-03-16T11:42:13.310 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: rocksdb: Options.max_write_buffer_number: 2 2024-03-16T11:42:13.310 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: rocksdb: Options.compression: NoCompression 2024-03-16T11:42:13.310 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: rocksdb: Options.bottommost_compression: Disabled 2024-03-16T11:42:13.310 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: rocksdb: Options.prefix_extractor: nullptr 2024-03-16T11:42:13.310 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-16T11:42:13.310 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: rocksdb: Options.num_levels: 7 2024-03-16T11:42:13.310 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-16T11:42:13.310 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-16T11:42:13.310 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-16T11:42:13.310 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-16T11:42:13.310 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-16T11:42:13.310 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-16T11:42:13.310 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-16T11:42:13.310 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-16T11:42:13.311 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-16T11:42:13.311 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-16T11:42:13.311 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-16T11:42:13.311 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-16T11:42:13.311 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-16T11:42:13.311 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: rocksdb: Options.compression_opts.level: 32767 2024-03-16T11:42:13.311 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: rocksdb: Options.compression_opts.strategy: 0 2024-03-16T11:42:13.311 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-16T11:42:13.311 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-16T11:42:13.311 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-16T11:42:13.311 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-16T11:42:13.311 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: rocksdb: Options.compression_opts.enabled: false 2024-03-16T11:42:13.312 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-16T11:42:13.312 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-16T11:42:13.312 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-16T11:42:13.312 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-16T11:42:13.312 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: rocksdb: Options.target_file_size_base: 67108864 2024-03-16T11:42:13.312 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-16T11:42:13.312 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-16T11:42:13.312 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-16T11:42:13.312 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-16T11:42:13.312 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-16T11:42:13.312 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-16T11:42:13.312 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-16T11:42:13.312 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-16T11:42:13.313 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-16T11:42:13.313 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-16T11:42:13.313 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-16T11:42:13.313 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-16T11:42:13.313 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-16T11:42:13.313 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-16T11:42:13.313 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: rocksdb: Options.arena_block_size: 1048576 2024-03-16T11:42:13.313 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-16T11:42:13.313 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-16T11:42:13.313 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: rocksdb: Options.disable_auto_compactions: 0 2024-03-16T11:42:13.313 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-16T11:42:13.313 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-16T11:42:13.313 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-16T11:42:13.314 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-16T11:42:13.314 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-16T11:42:13.314 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-16T11:42:13.314 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-16T11:42:13.314 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-16T11:42:13.314 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-16T11:42:13.314 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-16T11:42:13.314 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: rocksdb: Options.table_properties_collectors: 2024-03-16T11:42:13.314 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: rocksdb: Options.inplace_update_support: 0 2024-03-16T11:42:13.314 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-16T11:42:13.314 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-16T11:42:13.314 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-16T11:42:13.315 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-16T11:42:13.315 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: rocksdb: Options.bloom_locality: 0 2024-03-16T11:42:13.315 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: rocksdb: Options.max_successive_merges: 0 2024-03-16T11:42:13.315 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-16T11:42:13.315 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: rocksdb: Options.paranoid_file_checks: 0 2024-03-16T11:42:13.315 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: rocksdb: Options.force_consistency_checks: 1 2024-03-16T11:42:13.315 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: rocksdb: Options.report_bg_io_stats: 0 2024-03-16T11:42:13.315 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: rocksdb: Options.ttl: 2592000 2024-03-16T11:42:13.315 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-16T11:42:13.315 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-16T11:42:13.315 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-16T11:42:13.315 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: rocksdb: Options.enable_blob_files: false 2024-03-16T11:42:13.315 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: rocksdb: Options.min_blob_size: 0 2024-03-16T11:42:13.315 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: rocksdb: Options.blob_file_size: 268435456 2024-03-16T11:42:13.316 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-16T11:42:13.316 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-16T11:42:13.316 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-16T11:42:13.316 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-16T11:42:13.316 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-16T11:42:13.316 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: rocksdb: Options.blob_file_starting_level: 0 2024-03-16T11:42:13.316 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-16T11:42:13.316 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-03-16T11:42:13.316 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-03-16T11:42:13.316 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 3971bf74-1f21-48b1-b58b-899b0740380a 2024-03-16T11:42:13.316 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710589333030835, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-03-16T11:42:13.316 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-03-16T11:42:13.316 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710589333031272, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1622, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1710589333, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "3971bf74-1f21-48b1-b58b-899b0740380a", "db_session_id": "KRA2N8S36Z0RRJFHVNV0", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-03-16T11:42:13.317 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710589333031352, "job": 1, "event": "recovery_finished"} 2024-03-16T11:42:13.317 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-03-16T11:42:13.317 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-16T11:42:13.317 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55eb2d28e000 2024-03-16T11:42:13.317 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: rocksdb: DB pointer 0x55eb2d278000 2024-03-16T11:42:13.317 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-16T11:42:13.317 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-16T11:42:13.317 INFO:journalctl@ceph.mon.c.smithi154.stdout: ** DB Stats ** 2024-03-16T11:42:13.317 INFO:journalctl@ceph.mon.c.smithi154.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-16T11:42:13.317 INFO:journalctl@ceph.mon.c.smithi154.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-16T11:42:13.317 INFO:journalctl@ceph.mon.c.smithi154.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-16T11:42:13.317 INFO:journalctl@ceph.mon.c.smithi154.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-16T11:42:13.317 INFO:journalctl@ceph.mon.c.smithi154.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-16T11:42:13.318 INFO:journalctl@ceph.mon.c.smithi154.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-16T11:42:13.318 INFO:journalctl@ceph.mon.c.smithi154.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-16T11:42:13.318 INFO:journalctl@ceph.mon.c.smithi154.stdout: 2024-03-16T11:42:13.318 INFO:journalctl@ceph.mon.c.smithi154.stdout: ** Compaction Stats [default] ** 2024-03-16T11:42:13.318 INFO:journalctl@ceph.mon.c.smithi154.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-16T11:42:13.318 INFO:journalctl@ceph.mon.c.smithi154.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-16T11:42:13.318 INFO:journalctl@ceph.mon.c.smithi154.stdout: L0 1/0 1.58 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.7 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-16T11:42:13.318 INFO:journalctl@ceph.mon.c.smithi154.stdout: Sum 1/0 1.58 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.7 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-16T11:42:13.318 INFO:journalctl@ceph.mon.c.smithi154.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.7 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-16T11:42:13.318 INFO:journalctl@ceph.mon.c.smithi154.stdout: 2024-03-16T11:42:13.318 INFO:journalctl@ceph.mon.c.smithi154.stdout: ** Compaction Stats [default] ** 2024-03-16T11:42:13.318 INFO:journalctl@ceph.mon.c.smithi154.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-16T11:42:13.318 INFO:journalctl@ceph.mon.c.smithi154.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-16T11:42:13.319 INFO:journalctl@ceph.mon.c.smithi154.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 3.7 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-16T11:42:13.319 INFO:journalctl@ceph.mon.c.smithi154.stdout: 2024-03-16T11:42:13.319 INFO:journalctl@ceph.mon.c.smithi154.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-16T11:42:13.319 INFO:journalctl@ceph.mon.c.smithi154.stdout: 2024-03-16T11:42:13.319 INFO:journalctl@ceph.mon.c.smithi154.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-16T11:42:13.319 INFO:journalctl@ceph.mon.c.smithi154.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-16T11:42:13.319 INFO:journalctl@ceph.mon.c.smithi154.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-16T11:42:13.319 INFO:journalctl@ceph.mon.c.smithi154.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-16T11:42:13.319 INFO:journalctl@ceph.mon.c.smithi154.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-16T11:42:13.319 INFO:journalctl@ceph.mon.c.smithi154.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-16T11:42:13.319 INFO:journalctl@ceph.mon.c.smithi154.stdout: Cumulative compaction: 0.00 GB write, 0.63 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-16T11:42:13.319 INFO:journalctl@ceph.mon.c.smithi154.stdout: Interval compaction: 0.00 GB write, 0.63 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-16T11:42:13.319 INFO:journalctl@ceph.mon.c.smithi154.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-16T11:42:13.319 INFO:journalctl@ceph.mon.c.smithi154.stdout: Block cache BinnedLRUCache@0x55eb2d19b090#2 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 8e-06 secs_since: 0 2024-03-16T11:42:13.320 INFO:journalctl@ceph.mon.c.smithi154.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-03-16T11:42:13.320 INFO:journalctl@ceph.mon.c.smithi154.stdout: 2024-03-16T11:42:13.320 INFO:journalctl@ceph.mon.c.smithi154.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-16T11:42:13.320 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: mon.c does not exist in monmap, will attempt to join an existing cluster 2024-03-16T11:42:13.320 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: using public_addr v2:172.21.15.154:0/0 -> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-03-16T11:42:13.320 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: starting mon.c rank -1 at public addrs [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] at bind addrs [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon_data /var/lib/ceph/mon/ceph-c fsid d1887f28-e389-11ee-95c9-87774f69a715 2024-03-16T11:42:13.320 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: mon.c@-1(???) e0 preinit fsid d1887f28-e389-11ee-95c9-87774f69a715 2024-03-16T11:42:13.320 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: mon.c@-1(synchronizing).mds e1 new map 2024-03-16T11:42:13.320 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: mon.c@-1(synchronizing).mds e1 print_map 2024-03-16T11:42:13.320 INFO:journalctl@ceph.mon.c.smithi154.stdout: e1 2024-03-16T11:42:13.320 INFO:journalctl@ceph.mon.c.smithi154.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-16T11:42:13.320 INFO:journalctl@ceph.mon.c.smithi154.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-03-16T11:42:13.321 INFO:journalctl@ceph.mon.c.smithi154.stdout: legacy client fscid: -1 2024-03-16T11:42:13.321 INFO:journalctl@ceph.mon.c.smithi154.stdout: 2024-03-16T11:42:13.321 INFO:journalctl@ceph.mon.c.smithi154.stdout: No filesystems configured 2024-03-16T11:42:13.321 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-03-16T11:42:13.321 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-03-16T11:42:13.321 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-03-16T11:42:13.321 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-03-16T11:42:13.321 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-03-16T11:42:13.321 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-03-16T11:42:13.321 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-16T11:42:13.321 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-16T11:42:13.321 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-16T11:42:13.321 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-16T11:42:13.321 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:42:13.322 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-16T11:42:13.322 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T11:42:13.322 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T11:42:13.322 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:42:13.322 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:42:13.322 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:42:13.322 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:42:13.322 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi152", "name": "osd_memory_target"}]: dispatch 2024-03-16T11:42:13.322 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:42:13.322 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T11:42:13.322 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: Updating smithi152:/etc/ceph/ceph.conf 2024-03-16T11:42:13.322 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: Updating smithi152:/var/lib/ceph/d1887f28-e389-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-16T11:42:13.322 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi154", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T11:42:13.323 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: Updating smithi152:/etc/ceph/ceph.client.admin.keyring 2024-03-16T11:42:13.323 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: Updating smithi152:/var/lib/ceph/d1887f28-e389-11ee-95c9-87774f69a715/config/ceph.client.admin.keyring 2024-03-16T11:42:13.323 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:42:13.323 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:42:13.323 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:42:13.323 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: Deploying cephadm binary to smithi154 2024-03-16T11:42:13.323 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T11:42:13.323 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T11:42:13.323 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:42:13.323 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: Added host smithi154 2024-03-16T11:42:13.323 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T11:42:13.323 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:42:13.323 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:42:13.324 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T11:42:13.324 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:42:13.324 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-16T11:42:13.324 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T11:42:13.324 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T11:42:13.324 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: from='client.? 172.21.15.29:0/1485977759' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-16T11:42:13.324 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: from='client.? 172.21.15.29:0/1485977759' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-16T11:42:13.324 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: osdmap e4: 0 total, 0 up, 0 in 2024-03-16T11:42:13.324 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:42:13.324 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:42:13.324 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:42:13.324 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:42:13.324 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi154", "name": "osd_memory_target"}]: dispatch 2024-03-16T11:42:13.324 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:42:13.325 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T11:42:13.325 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: Updating smithi154:/etc/ceph/ceph.conf 2024-03-16T11:42:13.325 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: Updating smithi154:/var/lib/ceph/d1887f28-e389-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-16T11:42:13.325 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: Updating smithi154:/etc/ceph/ceph.client.admin.keyring 2024-03-16T11:42:13.325 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: Updating smithi154:/var/lib/ceph/d1887f28-e389-11ee-95c9-87774f69a715/config/ceph.client.admin.keyring 2024-03-16T11:42:13.325 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:42:13.325 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:42:13.325 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:42:13.325 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T11:42:13.325 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi029:172.21.15.29=a;smithi152:172.21.15.152=b;smithi154:172.21.15.154=c", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T11:42:13.325 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: Saving service mon spec with placement smithi029:172.21.15.29=a;smithi152:172.21.15.152=b;smithi154:172.21.15.154=c;count:3 2024-03-16T11:42:13.325 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:42:13.325 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T11:42:13.326 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:42:13.326 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T11:42:13.326 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:42:13.326 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-16T11:42:13.326 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:42:13.326 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: Deploying daemon mon.c on smithi154 2024-03-16T11:42:13.326 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T11:42:13.326 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T11:42:13.326 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: from='client.? 172.21.15.154:0/3233572200' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-16T11:42:13.326 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 ceph-mon[36320]: mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-03-16T11:42:13.326 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 bash[36288]: 79732bdfb11e0070e64dcea232db0dced05450c207f15255f8c8ea2eb58f0fb1 2024-03-16T11:42:13.326 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:13 smithi154 systemd[1]: Started Ceph mon.c for d1887f28-e389-11ee-95c9-87774f69a715. 2024-03-16T11:42:13.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:12 smithi029 ceph-mon[31975]: from='client.? 172.21.15.154:0/3233572200' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-16T11:42:14.626 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-16T11:42:14.626 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d1887f28-e389-11ee-95c9-87774f69a715 -- ceph mon dump -f json 2024-03-16T11:42:14.873 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/d1887f28-e389-11ee-95c9-87774f69a715/mon.c/config 2024-03-16T11:42:18.546 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:18 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-16T11:42:18.546 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:18 smithi154 ceph-mon[36320]: mon.a calling monitor election 2024-03-16T11:42:18.546 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:18 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-16T11:42:18.546 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:18 smithi154 ceph-mon[36320]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T11:42:18.546 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:18 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-16T11:42:18.546 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:18 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-16T11:42:18.546 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:18 smithi154 ceph-mon[36320]: mon.c calling monitor election 2024-03-16T11:42:18.546 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:18 smithi154 ceph-mon[36320]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T11:42:18.546 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:18 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-16T11:42:18.546 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:18 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-16T11:42:18.547 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:18 smithi154 ceph-mon[36320]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T11:42:18.547 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:18 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-16T11:42:18.547 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:18 smithi154 ceph-mon[36320]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-16T11:42:18.547 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:18 smithi154 ceph-mon[36320]: monmap epoch 2 2024-03-16T11:42:18.547 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:18 smithi154 ceph-mon[36320]: fsid d1887f28-e389-11ee-95c9-87774f69a715 2024-03-16T11:42:18.547 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:18 smithi154 ceph-mon[36320]: last_changed 2024-03-16T11:42:13.055033+0000 2024-03-16T11:42:18.547 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:18 smithi154 ceph-mon[36320]: created 2024-03-16T11:40:13.432436+0000 2024-03-16T11:42:18.547 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:18 smithi154 ceph-mon[36320]: min_mon_release 19 (squid) 2024-03-16T11:42:18.547 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:18 smithi154 ceph-mon[36320]: election_strategy: 1 2024-03-16T11:42:18.547 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:18 smithi154 ceph-mon[36320]: 0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2024-03-16T11:42:18.547 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:18 smithi154 ceph-mon[36320]: 1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.c 2024-03-16T11:42:18.547 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:18 smithi154 ceph-mon[36320]: fsmap 2024-03-16T11:42:18.547 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:18 smithi154 ceph-mon[36320]: osdmap e4: 0 total, 0 up, 0 in 2024-03-16T11:42:18.547 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:18 smithi154 ceph-mon[36320]: mgrmap e13: x(active, since 50s) 2024-03-16T11:42:18.547 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:18 smithi154 ceph-mon[36320]: overall HEALTH_OK 2024-03-16T11:42:18.548 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:18 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:42:18.548 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:18 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:42:18.548 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:18 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:42:18.548 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:18 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-16T11:42:18.548 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:18 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:42:18.588 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:18 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-16T11:42:18.588 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:18 smithi029 ceph-mon[31975]: mon.a calling monitor election 2024-03-16T11:42:18.589 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:18 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-16T11:42:18.589 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:18 smithi029 ceph-mon[31975]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T11:42:18.589 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:18 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-16T11:42:18.589 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:18 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-16T11:42:18.589 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:18 smithi029 ceph-mon[31975]: mon.c calling monitor election 2024-03-16T11:42:18.589 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:18 smithi029 ceph-mon[31975]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T11:42:18.589 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:18 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-16T11:42:18.589 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:18 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-16T11:42:18.589 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:18 smithi029 ceph-mon[31975]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T11:42:18.589 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:18 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-16T11:42:18.589 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:18 smithi029 ceph-mon[31975]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-16T11:42:18.589 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:18 smithi029 ceph-mon[31975]: monmap epoch 2 2024-03-16T11:42:18.589 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:18 smithi029 ceph-mon[31975]: fsid d1887f28-e389-11ee-95c9-87774f69a715 2024-03-16T11:42:18.589 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:18 smithi029 ceph-mon[31975]: last_changed 2024-03-16T11:42:13.055033+0000 2024-03-16T11:42:18.589 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:18 smithi029 ceph-mon[31975]: created 2024-03-16T11:40:13.432436+0000 2024-03-16T11:42:18.589 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:18 smithi029 ceph-mon[31975]: min_mon_release 19 (squid) 2024-03-16T11:42:18.589 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:18 smithi029 ceph-mon[31975]: election_strategy: 1 2024-03-16T11:42:18.590 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:18 smithi029 ceph-mon[31975]: 0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2024-03-16T11:42:18.590 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:18 smithi029 ceph-mon[31975]: 1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.c 2024-03-16T11:42:18.590 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:18 smithi029 ceph-mon[31975]: fsmap 2024-03-16T11:42:18.590 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:18 smithi029 ceph-mon[31975]: osdmap e4: 0 total, 0 up, 0 in 2024-03-16T11:42:18.590 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:18 smithi029 ceph-mon[31975]: mgrmap e13: x(active, since 50s) 2024-03-16T11:42:18.590 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:18 smithi029 ceph-mon[31975]: overall HEALTH_OK 2024-03-16T11:42:18.590 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:18 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:42:18.590 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:18 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:42:18.590 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:18 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:42:18.590 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:18 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-16T11:42:18.590 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:18 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:42:19.287 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:19.287+0000 7fe7da932700 1 -- 172.21.15.154:0/3739899748 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe7d4102010 msgr2=0x7fe7bc005610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:42:19.287 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:19.287+0000 7fe7da932700 1 --2- 172.21.15.154:0/3739899748 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe7d4102010 0x7fe7bc005610 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fe7c4005fa0 tx=0x7fe7c400ffc0 comp rx=0 tx=0).stop 2024-03-16T11:42:19.287 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:19.287+0000 7fe7da932700 1 -- 172.21.15.154:0/3739899748 shutdown_connections 2024-03-16T11:42:19.287 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:19.287+0000 7fe7da932700 1 --2- 172.21.15.154:0/3739899748 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe7d4102010 0x7fe7bc005610 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:42:19.287 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:19.287+0000 7fe7da932700 1 -- 172.21.15.154:0/3739899748 >> 172.21.15.154:0/3739899748 conn(0x7fe7d40fd8a0 msgr2=0x7fe7d40ffcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:42:19.288 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:19.287+0000 7fe7da932700 1 -- 172.21.15.154:0/3739899748 shutdown_connections 2024-03-16T11:42:19.288 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:19.287+0000 7fe7da932700 1 -- 172.21.15.154:0/3739899748 wait complete. 2024-03-16T11:42:19.288 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:19 smithi154 ceph-mon[36320]: Deploying daemon mon.b on smithi152 2024-03-16T11:42:19.288 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:19 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-16T11:42:19.289 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:19.288+0000 7fe7da932700 1 Processor -- start 2024-03-16T11:42:19.289 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:19.288+0000 7fe7da932700 1 -- start start 2024-03-16T11:42:19.289 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:19.289+0000 7fe7da932700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe7d4102010 0x7fe7d419f710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:42:19.289 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:19.289+0000 7fe7da932700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe7d419fc50 0x7fe7d41998d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:42:19.289 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:19.289+0000 7fe7da932700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fe7d4109e40 con 0x7fe7d419fc50 2024-03-16T11:42:19.289 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:19.289+0000 7fe7da932700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fe7d4109cc0 con 0x7fe7d4102010 2024-03-16T11:42:19.289 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:19.289+0000 7fe7d3fff700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe7d4102010 0x7fe7d419f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:42:19.289 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:19.289+0000 7fe7d3fff700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe7d4102010 0x7fe7d419f710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.154:54110/0 (socket says 172.21.15.154:54110) 2024-03-16T11:42:19.289 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:19.289+0000 7fe7d3fff700 1 -- 172.21.15.154:0/1617066001 learned_addr learned my addr 172.21.15.154:0/1617066001 (peer_addr_for_me v2:172.21.15.154:0/0) 2024-03-16T11:42:19.289 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:19.289+0000 7fe7d3fff700 1 -- 172.21.15.154:0/1617066001 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe7d4102010 msgr2=0x7fe7d419f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-03-16T11:42:19.290 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:19.289+0000 7fe7d3fff700 1 -- 172.21.15.154:0/1617066001 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe7d4102010 msgr2=0x7fe7d419f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-16T11:42:19.290 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:19.289+0000 7fe7d37fe700 1 --2- 172.21.15.154:0/1617066001 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe7d419fc50 0x7fe7d41998d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:42:19.290 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:19.289+0000 7fe7d3fff700 1 --2- 172.21.15.154:0/1617066001 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe7d4102010 0x7fe7d419f710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-16T11:42:19.290 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:19.289+0000 7fe7d3fff700 1 --2- 172.21.15.154:0/1617066001 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe7d4102010 0x7fe7d419f710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-16T11:42:19.291 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:19.290+0000 7fe7d37fe700 1 -- 172.21.15.154:0/1617066001 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe7d4102010 msgr2=0x7fe7d419f710 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-16T11:42:19.291 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:19.290+0000 7fe7d37fe700 1 --2- 172.21.15.154:0/1617066001 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe7d4102010 0x7fe7d419f710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:42:19.291 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:19.290+0000 7fe7d37fe700 1 -- 172.21.15.154:0/1617066001 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7d41a0290 con 0x7fe7d419fc50 2024-03-16T11:42:19.291 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:19.290+0000 7fe7d37fe700 1 --2- 172.21.15.154:0/1617066001 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe7d419fc50 0x7fe7d41998d0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fe7c800b4d0 tx=0x7fe7c800b890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:42:19.292 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:19.290+0000 7fe7d17fa700 1 -- 172.21.15.154:0/1617066001 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7fe7c8011630 con 0x7fe7d419fc50 2024-03-16T11:42:19.292 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:19.290+0000 7fe7da932700 1 -- 172.21.15.154:0/1617066001 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe7d4199e10 con 0x7fe7d419fc50 2024-03-16T11:42:19.292 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:19.290+0000 7fe7d17fa700 1 -- 172.21.15.154:0/1617066001 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe7c8011cb0 con 0x7fe7d419fc50 2024-03-16T11:42:19.292 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:19.290+0000 7fe7d17fa700 1 -- 172.21.15.154:0/1617066001 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7fe7c8010e60 con 0x7fe7d419fc50 2024-03-16T11:42:19.292 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:19.290+0000 7fe7da932700 1 -- 172.21.15.154:0/1617066001 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe7d419a3a0 con 0x7fe7d419fc50 2024-03-16T11:42:19.292 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:19.291+0000 7fe7d17fa700 1 -- 172.21.15.154:0/1617066001 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 13) ==== 46162+0+0 (secure 0 0 0) 0x7fe7c8018020 con 0x7fe7d419fc50 2024-03-16T11:42:19.292 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:19.292+0000 7fe7d17fa700 1 --2- 172.21.15.154:0/1617066001 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fe7c0038d00 0x7fe7c003b1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:42:19.292 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:19.292+0000 7fe7da932700 1 -- 172.21.15.154:0/1617066001 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe7d41034f0 con 0x7fe7d419fc50 2024-03-16T11:42:19.292 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:19.292+0000 7fe7d17fa700 1 -- 172.21.15.154:0/1617066001 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fe7c804ba60 con 0x7fe7d419fc50 2024-03-16T11:42:19.292 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:19.292+0000 7fe7d3fff700 1 --2- 172.21.15.154:0/1617066001 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fe7c0038d00 0x7fe7c003b1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:42:19.293 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:19.292+0000 7fe7d3fff700 1 --2- 172.21.15.154:0/1617066001 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fe7c0038d00 0x7fe7c003b1c0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fe7d419aa10 tx=0x7fe7c401d040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:42:19.297 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:19.298+0000 7fe7d17fa700 1 -- 172.21.15.154:0/1617066001 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fe7c8040020 con 0x7fe7d419fc50 2024-03-16T11:42:19.580 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:19.581+0000 7fe7da932700 1 -- 172.21.15.154:0/1617066001 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7fe7d419aea0 con 0x7fe7d419fc50 2024-03-16T11:42:19.581 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:19.581+0000 7fe7d17fa700 1 -- 172.21.15.154:0/1617066001 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) ==== 95+0+1015 (secure 0 0 0) 0x7fe7c801f020 con 0x7fe7d419fc50 2024-03-16T11:42:19.581 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-03-16T11:42:19.581 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":2,"fsid":"d1887f28-e389-11ee-95c9-87774f69a715","modified":"2024-03-16T11:42:13.055033Z","created":"2024-03-16T11:40:13.432436Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-03-16T11:42:19.583 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:19.583+0000 7fe7da932700 1 -- 172.21.15.154:0/1617066001 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fe7c0038d00 msgr2=0x7fe7c003b1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:42:19.584 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:19.583+0000 7fe7da932700 1 --2- 172.21.15.154:0/1617066001 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fe7c0038d00 0x7fe7c003b1c0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fe7d419aa10 tx=0x7fe7c401d040 comp rx=0 tx=0).stop 2024-03-16T11:42:19.584 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:19.583+0000 7fe7da932700 1 -- 172.21.15.154:0/1617066001 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe7d419fc50 msgr2=0x7fe7d41998d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:42:19.584 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:19.584+0000 7fe7da932700 1 --2- 172.21.15.154:0/1617066001 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe7d419fc50 0x7fe7d41998d0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fe7c800b4d0 tx=0x7fe7c800b890 comp rx=0 tx=0).stop 2024-03-16T11:42:19.584 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:19.584+0000 7fe7da932700 1 -- 172.21.15.154:0/1617066001 shutdown_connections 2024-03-16T11:42:19.584 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:19.584+0000 7fe7da932700 1 --2- 172.21.15.154:0/1617066001 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fe7c0038d00 0x7fe7c003b1c0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:42:19.584 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:19.584+0000 7fe7da932700 1 --2- 172.21.15.154:0/1617066001 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe7d4102010 0x7fe7d419f710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:42:19.584 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:19.584+0000 7fe7da932700 1 --2- 172.21.15.154:0/1617066001 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe7d419fc50 0x7fe7d41998d0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:42:19.584 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:19.584+0000 7fe7da932700 1 -- 172.21.15.154:0/1617066001 >> 172.21.15.154:0/1617066001 conn(0x7fe7d40fd8a0 msgr2=0x7fe7d41039e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:42:19.584 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:19.584+0000 7fe7da932700 1 -- 172.21.15.154:0/1617066001 shutdown_connections 2024-03-16T11:42:19.584 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:19.584+0000 7fe7da932700 1 -- 172.21.15.154:0/1617066001 wait complete. 2024-03-16T11:42:19.584 INFO:teuthology.orchestra.run.smithi154.stderr:dumped monmap epoch 2 2024-03-16T11:42:19.588 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:19 smithi029 ceph-mon[31975]: Deploying daemon mon.b on smithi152 2024-03-16T11:42:19.588 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:19 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-16T11:42:20.545 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:20 smithi154 ceph-mon[36320]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T11:42:20.545 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:20 smithi154 ceph-mon[36320]: from='client.? 172.21.15.154:0/1617066001' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-16T11:42:20.588 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:20 smithi029 ceph-mon[31975]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T11:42:20.588 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:20 smithi029 ceph-mon[31975]: from='client.? 172.21.15.154:0/1617066001' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-16T11:42:21.203 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-16T11:42:21.204 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d1887f28-e389-11ee-95c9-87774f69a715 -- ceph mon dump -f json 2024-03-16T11:42:21.442 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/d1887f28-e389-11ee-95c9-87774f69a715/mon.c/config 2024-03-16T11:42:21.805 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:21 smithi154 ceph-mon[36320]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T11:42:22.054 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:22 smithi152 ceph-mon[36291]: mon.b@-1(synchronizing).mds e1 new map 2024-03-16T11:42:22.054 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:22 smithi152 ceph-mon[36291]: mon.b@-1(synchronizing).mds e1 print_map 2024-03-16T11:42:22.054 INFO:journalctl@ceph.mon.b.smithi152.stdout: e1 2024-03-16T11:42:22.054 INFO:journalctl@ceph.mon.b.smithi152.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-16T11:42:22.054 INFO:journalctl@ceph.mon.b.smithi152.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-03-16T11:42:22.054 INFO:journalctl@ceph.mon.b.smithi152.stdout: legacy client fscid: -1 2024-03-16T11:42:22.054 INFO:journalctl@ceph.mon.b.smithi152.stdout: 2024-03-16T11:42:22.054 INFO:journalctl@ceph.mon.b.smithi152.stdout: No filesystems configured 2024-03-16T11:42:22.054 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:22 smithi152 ceph-mon[36291]: mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-03-16T11:42:22.054 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:22 smithi152 ceph-mon[36291]: mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-03-16T11:42:22.054 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:22 smithi152 ceph-mon[36291]: mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-03-16T11:42:22.054 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:22 smithi152 ceph-mon[36291]: mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-03-16T11:42:22.054 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:22 smithi152 ceph-mon[36291]: mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-03-16T11:42:22.055 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:22 smithi152 ceph-mon[36291]: mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-03-16T11:42:22.055 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:22 smithi152 ceph-mon[36291]: mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-16T11:42:22.055 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:22 smithi152 ceph-mon[36291]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-16T11:42:22.055 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:22 smithi152 ceph-mon[36291]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-16T11:42:22.055 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:22 smithi152 ceph-mon[36291]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-16T11:42:22.055 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:22 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:42:22.055 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:22 smithi152 ceph-mon[36291]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-16T11:42:22.055 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:22 smithi152 ceph-mon[36291]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T11:42:22.055 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:22 smithi152 ceph-mon[36291]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T11:42:22.055 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:22 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:42:22.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:21 smithi029 ceph-mon[31975]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T11:42:22.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:22 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:42:22.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:22 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:42:22.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:22 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:42:22.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:22 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi152", "name": "osd_memory_target"}]: dispatch 2024-03-16T11:42:22.378 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:22 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:42:22.378 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:22 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T11:42:22.378 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:22 smithi152 ceph-mon[36291]: Updating smithi152:/etc/ceph/ceph.conf 2024-03-16T11:42:22.378 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:22 smithi152 ceph-mon[36291]: Updating smithi152:/var/lib/ceph/d1887f28-e389-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-16T11:42:22.378 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:22 smithi152 ceph-mon[36291]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi154", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T11:42:22.378 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:22 smithi152 ceph-mon[36291]: Updating smithi152:/etc/ceph/ceph.client.admin.keyring 2024-03-16T11:42:22.378 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:22 smithi152 ceph-mon[36291]: Updating smithi152:/var/lib/ceph/d1887f28-e389-11ee-95c9-87774f69a715/config/ceph.client.admin.keyring 2024-03-16T11:42:22.378 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:22 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:42:22.378 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:22 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:42:22.378 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:22 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:42:22.378 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:22 smithi152 ceph-mon[36291]: Deploying cephadm binary to smithi154 2024-03-16T11:42:22.378 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:22 smithi152 ceph-mon[36291]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T11:42:22.378 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:22 smithi152 ceph-mon[36291]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T11:42:22.378 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:22 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:42:22.379 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:22 smithi152 ceph-mon[36291]: Added host smithi154 2024-03-16T11:42:22.379 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:22 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T11:42:22.379 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:22 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:42:22.379 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:22 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:42:22.379 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:22 smithi152 ceph-mon[36291]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T11:42:22.379 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:22 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:42:22.379 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:22 smithi152 ceph-mon[36291]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-16T11:42:22.379 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:22 smithi152 ceph-mon[36291]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T11:42:22.379 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:22 smithi152 ceph-mon[36291]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T11:42:22.379 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:22 smithi152 ceph-mon[36291]: from='client.? 172.21.15.29:0/1485977759' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-16T11:42:22.379 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:22 smithi152 ceph-mon[36291]: from='client.? 172.21.15.29:0/1485977759' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-16T11:42:22.379 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:22 smithi152 ceph-mon[36291]: osdmap e4: 0 total, 0 up, 0 in 2024-03-16T11:42:22.379 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:22 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:42:22.379 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:22 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:42:22.380 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:22 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:42:22.380 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:22 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:42:22.380 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:22 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi154", "name": "osd_memory_target"}]: dispatch 2024-03-16T11:42:22.380 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:22 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:42:22.380 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:22 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T11:42:22.380 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:22 smithi152 ceph-mon[36291]: Updating smithi154:/etc/ceph/ceph.conf 2024-03-16T11:42:22.380 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:22 smithi152 ceph-mon[36291]: Updating smithi154:/var/lib/ceph/d1887f28-e389-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-16T11:42:22.380 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:22 smithi152 ceph-mon[36291]: Updating smithi154:/etc/ceph/ceph.client.admin.keyring 2024-03-16T11:42:22.380 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:22 smithi152 ceph-mon[36291]: Updating smithi154:/var/lib/ceph/d1887f28-e389-11ee-95c9-87774f69a715/config/ceph.client.admin.keyring 2024-03-16T11:42:22.380 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:22 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:42:22.380 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:22 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:42:22.380 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:22 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:42:22.380 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:22 smithi152 ceph-mon[36291]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T11:42:22.380 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:22 smithi152 ceph-mon[36291]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi029:172.21.15.29=a;smithi152:172.21.15.152=b;smithi154:172.21.15.154=c", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T11:42:22.380 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:22 smithi152 ceph-mon[36291]: Saving service mon spec with placement smithi029:172.21.15.29=a;smithi152:172.21.15.152=b;smithi154:172.21.15.154=c;count:3 2024-03-16T11:42:22.381 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:22 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:42:22.381 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:22 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T11:42:22.381 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:22 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:42:22.381 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:22 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T11:42:22.381 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:22 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:42:22.381 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:22 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-16T11:42:22.381 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:22 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:42:22.381 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:22 smithi152 ceph-mon[36291]: Deploying daemon mon.c on smithi154 2024-03-16T11:42:22.381 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:22 smithi152 ceph-mon[36291]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T11:42:22.381 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:22 smithi152 ceph-mon[36291]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T11:42:22.381 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:22 smithi152 ceph-mon[36291]: from='client.? 172.21.15.154:0/3233572200' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-16T11:42:22.381 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:22 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-16T11:42:22.381 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:22 smithi152 ceph-mon[36291]: mon.a calling monitor election 2024-03-16T11:42:22.381 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:22 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-16T11:42:22.381 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:22 smithi152 ceph-mon[36291]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T11:42:22.381 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:22 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-16T11:42:22.382 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:22 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-16T11:42:22.382 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:22 smithi152 ceph-mon[36291]: mon.c calling monitor election 2024-03-16T11:42:22.382 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:22 smithi152 ceph-mon[36291]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T11:42:22.382 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:22 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-16T11:42:22.382 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:22 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-16T11:42:22.382 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:22 smithi152 ceph-mon[36291]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T11:42:22.382 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:22 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-16T11:42:22.382 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:22 smithi152 ceph-mon[36291]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-16T11:42:22.382 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:22 smithi152 ceph-mon[36291]: monmap epoch 2 2024-03-16T11:42:22.382 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:22 smithi152 ceph-mon[36291]: fsid d1887f28-e389-11ee-95c9-87774f69a715 2024-03-16T11:42:22.383 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:22 smithi152 ceph-mon[36291]: last_changed 2024-03-16T11:42:13.055033+0000 2024-03-16T11:42:22.383 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:22 smithi152 ceph-mon[36291]: created 2024-03-16T11:40:13.432436+0000 2024-03-16T11:42:22.383 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:22 smithi152 ceph-mon[36291]: min_mon_release 19 (squid) 2024-03-16T11:42:22.383 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:22 smithi152 ceph-mon[36291]: election_strategy: 1 2024-03-16T11:42:22.383 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:22 smithi152 ceph-mon[36291]: 0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2024-03-16T11:42:22.383 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:22 smithi152 ceph-mon[36291]: 1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.c 2024-03-16T11:42:22.383 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:22 smithi152 ceph-mon[36291]: fsmap 2024-03-16T11:42:22.383 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:22 smithi152 ceph-mon[36291]: osdmap e4: 0 total, 0 up, 0 in 2024-03-16T11:42:22.383 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:22 smithi152 ceph-mon[36291]: mgrmap e13: x(active, since 50s) 2024-03-16T11:42:22.383 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:22 smithi152 ceph-mon[36291]: overall HEALTH_OK 2024-03-16T11:42:22.383 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:22 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:42:22.383 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:22 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:42:22.383 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:22 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:42:22.383 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:22 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-16T11:42:22.383 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:22 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:42:22.384 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:22 smithi152 ceph-mon[36291]: Deploying daemon mon.b on smithi152 2024-03-16T11:42:22.384 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:22 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-16T11:42:22.384 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:22 smithi152 ceph-mon[36291]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T11:42:22.384 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:22 smithi152 ceph-mon[36291]: from='client.? 172.21.15.154:0/1617066001' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-16T11:42:22.384 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:22 smithi152 ceph-mon[36291]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T11:42:22.384 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:22 smithi152 ceph-mon[36291]: mon.b@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-03-16T11:42:22.384 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:22 smithi152 bash[36269]: 315b459181464679894a446a6cdc2dc8623c966f3b2fb42bda423483e205f7bd 2024-03-16T11:42:22.384 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:22 smithi152 systemd[1]: Started Ceph mon.b for d1887f28-e389-11ee-95c9-87774f69a715. 2024-03-16T11:42:27.272 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:27.272+0000 7f32bc11f700 1 -- 172.21.15.154:0/2767828986 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f329c0057d0 msgr2=0x7f329c007c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:42:27.272 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:27.272+0000 7f32bc11f700 1 --2- 172.21.15.154:0/2767828986 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f329c0057d0 0x7f329c007c60 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f32b4100c00 tx=0x7f32ac009e80 comp rx=0 tx=0).stop 2024-03-16T11:42:27.273 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:27.272+0000 7f32bc11f700 1 -- 172.21.15.154:0/2767828986 shutdown_connections 2024-03-16T11:42:27.273 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:27.272+0000 7f32bc11f700 1 --2- 172.21.15.154:0/2767828986 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f329c0057d0 0x7f329c007c60 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:42:27.273 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:27.272+0000 7f32bc11f700 1 -- 172.21.15.154:0/2767828986 >> 172.21.15.154:0/2767828986 conn(0x7f32b40fc7e0 msgr2=0x7f32b40fec00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:42:27.273 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:27.273+0000 7f32bc11f700 1 -- 172.21.15.154:0/2767828986 shutdown_connections 2024-03-16T11:42:27.273 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:27.273+0000 7f32bc11f700 1 -- 172.21.15.154:0/2767828986 wait complete. 2024-03-16T11:42:27.274 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:27.274+0000 7f32bc11f700 1 Processor -- start 2024-03-16T11:42:27.274 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:27.274+0000 7f32bc11f700 1 -- start start 2024-03-16T11:42:27.274 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:27.274+0000 7f32bc11f700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f329c0057d0 0x7f32b419f870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:42:27.274 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:27.274+0000 7f32bc11f700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f32b419fdb0 0x7f32b4199a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:42:27.274 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:27.274+0000 7f32bc11f700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f32b4199f70 0x7f32b419a420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:42:27.274 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:27.274+0000 7f32bc11f700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f32b4107d30 con 0x7f329c0057d0 2024-03-16T11:42:27.274 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:27.274+0000 7f32bc11f700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f32b4107bb0 con 0x7f32b4199f70 2024-03-16T11:42:27.275 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:27.274+0000 7f32bc11f700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f32b4107eb0 con 0x7f32b419fdb0 2024-03-16T11:42:27.275 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:27.274+0000 7f32b9ebb700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f329c0057d0 0x7f32b419f870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:42:27.275 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:27.274+0000 7f32b9ebb700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f329c0057d0 0x7f32b419f870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.154:53864/0 (socket says 172.21.15.154:53864) 2024-03-16T11:42:27.275 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:27.274+0000 7f32b9ebb700 1 -- 172.21.15.154:0/887345139 learned_addr learned my addr 172.21.15.154:0/887345139 (peer_addr_for_me v2:172.21.15.154:0/0) 2024-03-16T11:42:27.275 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:27.274+0000 7f32b96ba700 1 --2- 172.21.15.154:0/887345139 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f32b419fdb0 0x7f32b4199a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:42:27.275 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:27.274+0000 7f32ba6bc700 1 --2- 172.21.15.154:0/887345139 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f32b4199f70 0x7f32b419a420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:42:27.275 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:27.274+0000 7f32b96ba700 1 -- 172.21.15.154:0/887345139 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f32b419fdb0 msgr2=0x7f32b4199a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-03-16T11:42:27.275 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:27.275+0000 7f32b96ba700 1 -- 172.21.15.154:0/887345139 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f32b419fdb0 msgr2=0x7f32b4199a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-16T11:42:27.275 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:27.275+0000 7f32b96ba700 1 --2- 172.21.15.154:0/887345139 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f32b419fdb0 0x7f32b4199a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-16T11:42:27.275 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:27.275+0000 7f32ba6bc700 1 -- 172.21.15.154:0/887345139 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f32b4199f70 msgr2=0x7f32b419a420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2024-03-16T11:42:27.275 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:27.275+0000 7f32ba6bc700 1 -- 172.21.15.154:0/887345139 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f32b4199f70 msgr2=0x7f32b419a420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-16T11:42:27.275 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:27.275+0000 7f32ba6bc700 1 --2- 172.21.15.154:0/887345139 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f32b4199f70 0x7f32b419a420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-16T11:42:27.275 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:27.275+0000 7f32b96ba700 1 --2- 172.21.15.154:0/887345139 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f32b419fdb0 0x7f32b4199a30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-16T11:42:27.275 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:27.275+0000 7f32ba6bc700 1 --2- 172.21.15.154:0/887345139 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f32b4199f70 0x7f32b419a420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-16T11:42:27.275 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:27.275+0000 7f32b9ebb700 1 -- 172.21.15.154:0/887345139 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f32b419fdb0 msgr2=0x7f32b4199a30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-16T11:42:27.276 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:27.275+0000 7f32b9ebb700 1 --2- 172.21.15.154:0/887345139 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f32b419fdb0 0x7f32b4199a30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:42:27.276 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:27.275+0000 7f32b9ebb700 1 -- 172.21.15.154:0/887345139 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f32b4199f70 msgr2=0x7f32b419a420 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-16T11:42:27.276 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:27.275+0000 7f32b9ebb700 1 --2- 172.21.15.154:0/887345139 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f32b4199f70 0x7f32b419a420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:42:27.276 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:27.275+0000 7f32b9ebb700 1 -- 172.21.15.154:0/887345139 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32b41a6740 con 0x7f329c0057d0 2024-03-16T11:42:27.277 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:27.275+0000 7f32b9ebb700 1 --2- 172.21.15.154:0/887345139 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f329c0057d0 0x7f32b419f870 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f32ac002230 tx=0x7f32ac0048c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:42:27.277 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:27.275+0000 7f32b2ffd700 1 -- 172.21.15.154:0/887345139 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32ac01a7b0 con 0x7f329c0057d0 2024-03-16T11:42:27.277 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:27.275+0000 7f32b2ffd700 1 -- 172.21.15.154:0/887345139 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f32ac026030 con 0x7f329c0057d0 2024-03-16T11:42:27.277 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:27.275+0000 7f32bc11f700 1 -- 172.21.15.154:0/887345139 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f32b41a6910 con 0x7f329c0057d0 2024-03-16T11:42:27.277 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:27.276+0000 7f32b2ffd700 1 -- 172.21.15.154:0/887345139 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32ac003a00 con 0x7f329c0057d0 2024-03-16T11:42:27.277 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:27.276+0000 7f32bc11f700 1 -- 172.21.15.154:0/887345139 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f32b41a6ea0 con 0x7f329c0057d0 2024-03-16T11:42:27.277 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:27.276+0000 7f32b2ffd700 1 -- 172.21.15.154:0/887345139 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 13) ==== 46162+0+0 (secure 0 0 0) 0x7f32ac015040 con 0x7f329c0057d0 2024-03-16T11:42:27.277 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:27.276+0000 7f32b2ffd700 1 --2- 172.21.15.154:0/887345139 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f32a0039450 0x7f32a003b910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:42:27.277 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:27.276+0000 7f32b2ffd700 1 -- 172.21.15.154:0/887345139 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f32ac04f090 con 0x7f329c0057d0 2024-03-16T11:42:27.278 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:27.277+0000 7f32b96ba700 1 --2- 172.21.15.154:0/887345139 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f32a0039450 0x7f32a003b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:42:27.278 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:27.277+0000 7f32bc11f700 1 -- 172.21.15.154:0/887345139 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f32b4063900 con 0x7f329c0057d0 2024-03-16T11:42:27.278 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:27.277+0000 7f32b96ba700 1 --2- 172.21.15.154:0/887345139 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f32a0039450 0x7f32a003b910 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f32a8009850 tx=0x7f32a8008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:42:27.282 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:27.282+0000 7f32b2ffd700 1 -- 172.21.15.154:0/887345139 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f32ac02a7d0 con 0x7f329c0057d0 2024-03-16T11:42:27.495 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:27 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-16T11:42:27.495 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:27 smithi029 ceph-mon[31975]: mon.c calling monitor election 2024-03-16T11:42:27.495 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:27 smithi029 ceph-mon[31975]: mon.a calling monitor election 2024-03-16T11:42:27.495 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:27 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T11:42:27.495 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:27 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-16T11:42:27.495 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:27 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T11:42:27.495 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:27 smithi029 ceph-mon[31975]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T11:42:27.495 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:27 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T11:42:27.496 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:27 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T11:42:27.496 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:27 smithi029 ceph-mon[31975]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T11:42:27.496 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:27 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T11:42:27.496 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:27 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T11:42:27.496 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:27 smithi029 ceph-mon[31975]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-16T11:42:27.496 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:27 smithi029 ceph-mon[31975]: monmap epoch 3 2024-03-16T11:42:27.496 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:27 smithi029 ceph-mon[31975]: fsid d1887f28-e389-11ee-95c9-87774f69a715 2024-03-16T11:42:27.496 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:27 smithi029 ceph-mon[31975]: last_changed 2024-03-16T11:42:22.059913+0000 2024-03-16T11:42:27.496 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:27 smithi029 ceph-mon[31975]: created 2024-03-16T11:40:13.432436+0000 2024-03-16T11:42:27.496 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:27 smithi029 ceph-mon[31975]: min_mon_release 19 (squid) 2024-03-16T11:42:27.496 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:27 smithi029 ceph-mon[31975]: election_strategy: 1 2024-03-16T11:42:27.496 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:27 smithi029 ceph-mon[31975]: 0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2024-03-16T11:42:27.496 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:27 smithi029 ceph-mon[31975]: 1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.c 2024-03-16T11:42:27.496 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:27 smithi029 ceph-mon[31975]: 2: [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] mon.b 2024-03-16T11:42:27.496 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:27 smithi029 ceph-mon[31975]: fsmap 2024-03-16T11:42:27.496 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:27 smithi029 ceph-mon[31975]: osdmap e4: 0 total, 0 up, 0 in 2024-03-16T11:42:27.497 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:27 smithi029 ceph-mon[31975]: mgrmap e13: x(active, since 59s) 2024-03-16T11:42:27.497 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:27 smithi029 ceph-mon[31975]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-03-16T11:42:27.497 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:27 smithi029 ceph-mon[31975]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-03-16T11:42:27.497 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:27 smithi029 ceph-mon[31975]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-16T11:42:27.497 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:27 smithi029 ceph-mon[31975]: mon.b (rank 2) addr [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] is down (out of quorum) 2024-03-16T11:42:27.497 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:27 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:42:27.497 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:27 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:42:27.497 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:27 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:42:27.497 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:27 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:42:27.546 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:27 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-16T11:42:27.546 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:27 smithi154 ceph-mon[36320]: mon.c calling monitor election 2024-03-16T11:42:27.546 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:27 smithi154 ceph-mon[36320]: mon.a calling monitor election 2024-03-16T11:42:27.546 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:27 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T11:42:27.546 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:27 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-16T11:42:27.546 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:27 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T11:42:27.546 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:27 smithi154 ceph-mon[36320]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T11:42:27.547 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:27 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T11:42:27.547 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:27 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T11:42:27.547 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:27 smithi154 ceph-mon[36320]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T11:42:27.547 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:27 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T11:42:27.547 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:27 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T11:42:27.547 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:27 smithi154 ceph-mon[36320]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-16T11:42:27.547 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:27 smithi154 ceph-mon[36320]: monmap epoch 3 2024-03-16T11:42:27.547 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:27 smithi154 ceph-mon[36320]: fsid d1887f28-e389-11ee-95c9-87774f69a715 2024-03-16T11:42:27.547 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:27 smithi154 ceph-mon[36320]: last_changed 2024-03-16T11:42:22.059913+0000 2024-03-16T11:42:27.547 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:27 smithi154 ceph-mon[36320]: created 2024-03-16T11:40:13.432436+0000 2024-03-16T11:42:27.547 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:27 smithi154 ceph-mon[36320]: min_mon_release 19 (squid) 2024-03-16T11:42:27.547 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:27 smithi154 ceph-mon[36320]: election_strategy: 1 2024-03-16T11:42:27.547 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:27 smithi154 ceph-mon[36320]: 0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2024-03-16T11:42:27.547 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:27 smithi154 ceph-mon[36320]: 1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.c 2024-03-16T11:42:27.547 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:27 smithi154 ceph-mon[36320]: 2: [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] mon.b 2024-03-16T11:42:27.548 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:27 smithi154 ceph-mon[36320]: fsmap 2024-03-16T11:42:27.548 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:27 smithi154 ceph-mon[36320]: osdmap e4: 0 total, 0 up, 0 in 2024-03-16T11:42:27.548 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:27 smithi154 ceph-mon[36320]: mgrmap e13: x(active, since 59s) 2024-03-16T11:42:27.548 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:27 smithi154 ceph-mon[36320]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-03-16T11:42:27.548 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:27 smithi154 ceph-mon[36320]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-03-16T11:42:27.548 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:27 smithi154 ceph-mon[36320]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-16T11:42:27.548 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:27 smithi154 ceph-mon[36320]: mon.b (rank 2) addr [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] is down (out of quorum) 2024-03-16T11:42:27.548 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:27 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:42:27.548 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:27 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:42:27.548 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:27 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:42:27.548 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:27 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:42:27.571 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:27.571+0000 7f32bc11f700 1 -- 172.21.15.154:0/887345139 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f32b402db70 con 0x7f329c0057d0 2024-03-16T11:42:27.571 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:27.572+0000 7f32b2ffd700 1 -- 172.21.15.154:0/887345139 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 3 v3) ==== 95+0+1283 (secure 0 0 0) 0x7f32ac022420 con 0x7f329c0057d0 2024-03-16T11:42:27.572 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-03-16T11:42:27.572 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":3,"fsid":"d1887f28-e389-11ee-95c9-87774f69a715","modified":"2024-03-16T11:42:22.059913Z","created":"2024-03-16T11:40:13.432436Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:3300","nonce":0},{"type":"v1","addr":"172.21.15.152:6789","nonce":0}]},"addr":"172.21.15.152:6789/0","public_addr":"172.21.15.152:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-03-16T11:42:27.574 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:27.574+0000 7f32bc11f700 1 -- 172.21.15.154:0/887345139 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f32a0039450 msgr2=0x7f32a003b910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:42:27.574 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:27.574+0000 7f32bc11f700 1 --2- 172.21.15.154:0/887345139 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f32a0039450 0x7f32a003b910 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f32a8009850 tx=0x7f32a8008040 comp rx=0 tx=0).stop 2024-03-16T11:42:27.574 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:27.574+0000 7f32bc11f700 1 -- 172.21.15.154:0/887345139 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f329c0057d0 msgr2=0x7f32b419f870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:42:27.574 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:27.574+0000 7f32bc11f700 1 --2- 172.21.15.154:0/887345139 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f329c0057d0 0x7f32b419f870 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f32ac002230 tx=0x7f32ac0048c0 comp rx=0 tx=0).stop 2024-03-16T11:42:27.574 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:27.574+0000 7f32bc11f700 1 -- 172.21.15.154:0/887345139 shutdown_connections 2024-03-16T11:42:27.574 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:27.574+0000 7f32bc11f700 1 --2- 172.21.15.154:0/887345139 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f32b4199f70 0x7f32b419a420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:42:27.575 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:27.574+0000 7f32bc11f700 1 --2- 172.21.15.154:0/887345139 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f329c0057d0 0x7f32b419f870 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:42:27.575 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:27.574+0000 7f32bc11f700 1 --2- 172.21.15.154:0/887345139 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f32a0039450 0x7f32a003b910 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:42:27.575 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:27.574+0000 7f32bc11f700 1 --2- 172.21.15.154:0/887345139 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f32b419fdb0 0x7f32b4199a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:42:27.575 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:27.574+0000 7f32bc11f700 1 -- 172.21.15.154:0/887345139 >> 172.21.15.154:0/887345139 conn(0x7f32b40fc7e0 msgr2=0x7f32b40fd940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:42:27.575 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:27.574+0000 7f32bc11f700 1 -- 172.21.15.154:0/887345139 shutdown_connections 2024-03-16T11:42:27.575 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:27.574+0000 7f32bc11f700 1 -- 172.21.15.154:0/887345139 wait complete. 2024-03-16T11:42:27.575 INFO:teuthology.orchestra.run.smithi154.stderr:dumped monmap epoch 3 2024-03-16T11:42:28.209 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-03-16T11:42:28.209 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d1887f28-e389-11ee-95c9-87774f69a715 -- ceph config generate-minimal-conf 2024-03-16T11:42:28.458 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/d1887f28-e389-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T11:42:28.545 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:28 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T11:42:28.546 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:28 smithi154 ceph-mon[36320]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T11:42:28.546 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:28 smithi154 ceph-mon[36320]: from='client.? 172.21.15.154:0/887345139' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-16T11:42:28.546 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:28 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:42:28.546 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:28 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T11:42:28.580 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:28 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T11:42:28.580 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:28 smithi029 ceph-mon[31975]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T11:42:28.580 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:28 smithi029 ceph-mon[31975]: from='client.? 172.21.15.154:0/887345139' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-16T11:42:28.580 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:28 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:42:28.580 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:28 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T11:42:29.378 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:29 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-16T11:42:29.378 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:29 smithi152 ceph-mon[36291]: mon.c calling monitor election 2024-03-16T11:42:29.378 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:29 smithi152 ceph-mon[36291]: mon.a calling monitor election 2024-03-16T11:42:29.378 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:29 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T11:42:29.378 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:29 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-16T11:42:29.378 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:29 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T11:42:29.378 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:29 smithi152 ceph-mon[36291]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T11:42:29.378 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:29 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T11:42:29.379 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:29 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T11:42:29.379 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:29 smithi152 ceph-mon[36291]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T11:42:29.379 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:29 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T11:42:29.379 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:29 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T11:42:29.379 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:29 smithi152 ceph-mon[36291]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-16T11:42:29.379 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:29 smithi152 ceph-mon[36291]: monmap epoch 3 2024-03-16T11:42:29.379 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:29 smithi152 ceph-mon[36291]: fsid d1887f28-e389-11ee-95c9-87774f69a715 2024-03-16T11:42:29.379 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:29 smithi152 ceph-mon[36291]: last_changed 2024-03-16T11:42:22.059913+0000 2024-03-16T11:42:29.379 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:29 smithi152 ceph-mon[36291]: created 2024-03-16T11:40:13.432436+0000 2024-03-16T11:42:29.379 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:29 smithi152 ceph-mon[36291]: min_mon_release 19 (squid) 2024-03-16T11:42:29.379 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:29 smithi152 ceph-mon[36291]: election_strategy: 1 2024-03-16T11:42:29.379 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:29 smithi152 ceph-mon[36291]: 0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2024-03-16T11:42:29.379 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:29 smithi152 ceph-mon[36291]: 1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.c 2024-03-16T11:42:29.379 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:29 smithi152 ceph-mon[36291]: 2: [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] mon.b 2024-03-16T11:42:29.379 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:29 smithi152 ceph-mon[36291]: fsmap 2024-03-16T11:42:29.379 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:29 smithi152 ceph-mon[36291]: osdmap e4: 0 total, 0 up, 0 in 2024-03-16T11:42:29.380 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:29 smithi152 ceph-mon[36291]: mgrmap e13: x(active, since 59s) 2024-03-16T11:42:29.380 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:29 smithi152 ceph-mon[36291]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-03-16T11:42:29.380 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:29 smithi152 ceph-mon[36291]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-03-16T11:42:29.380 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:29 smithi152 ceph-mon[36291]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-16T11:42:29.380 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:29 smithi152 ceph-mon[36291]: mon.b (rank 2) addr [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] is down (out of quorum) 2024-03-16T11:42:29.380 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:29 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:42:29.380 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:29 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:42:29.380 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:29 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:42:29.380 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:29 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:42:29.380 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:29 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T11:42:29.380 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:29 smithi152 ceph-mon[36291]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T11:42:29.380 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:29 smithi152 ceph-mon[36291]: from='client.? 172.21.15.154:0/887345139' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-16T11:42:29.380 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:29 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:42:29.380 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:29 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T11:42:30.099 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:42:30.097+0000 7f84b97de700 1 -- 172.21.15.29:0/444048864 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f84b4102e60 msgr2=0x7f84b4103240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:42:30.099 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:42:30.097+0000 7f84b97de700 1 --2- 172.21.15.29:0/444048864 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f84b4102e60 0x7f84b4103240 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f849c009910 tx=0x7f849c009c20 comp rx=0 tx=0).stop 2024-03-16T11:42:30.099 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:42:30.097+0000 7f84b97de700 1 -- 172.21.15.29:0/444048864 shutdown_connections 2024-03-16T11:42:30.099 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:42:30.097+0000 7f84b97de700 1 --2- 172.21.15.29:0/444048864 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f84b4102e60 0x7f84b4103240 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:42:30.099 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:42:30.098+0000 7f84b97de700 1 -- 172.21.15.29:0/444048864 >> 172.21.15.29:0/444048864 conn(0x7f84b4077810 msgr2=0x7f84b40feae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:42:30.099 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:42:30.098+0000 7f84b97de700 1 -- 172.21.15.29:0/444048864 shutdown_connections 2024-03-16T11:42:30.099 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:42:30.098+0000 7f84b97de700 1 -- 172.21.15.29:0/444048864 wait complete. 2024-03-16T11:42:30.099 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:42:30.098+0000 7f84b97de700 1 Processor -- start 2024-03-16T11:42:30.100 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:42:30.098+0000 7f84b97de700 1 -- start start 2024-03-16T11:42:30.100 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:42:30.098+0000 7f84b97de700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f84b4102e60 0x7f84b4100ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:42:30.100 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:42:30.098+0000 7f84b97de700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f84b4101000 0x7f84b4101480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:42:30.100 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:42:30.099+0000 7f84b97de700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f84b41a3c90 0x7f84b41a6080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:42:30.100 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:42:30.099+0000 7f84b97de700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f84b410c350 con 0x7f84b4101000 2024-03-16T11:42:30.100 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:42:30.099+0000 7f84b97de700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f84b410c1d0 con 0x7f84b4102e60 2024-03-16T11:42:30.100 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:42:30.099+0000 7f84b97de700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f84b410c4d0 con 0x7f84b41a3c90 2024-03-16T11:42:30.100 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:42:30.099+0000 7f84b27fc700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f84b4101000 0x7f84b4101480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:42:30.100 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:42:30.099+0000 7f84b27fc700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f84b4101000 0x7f84b4101480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36096/0 (socket says 172.21.15.29:36096) 2024-03-16T11:42:30.100 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:42:30.099+0000 7f84b27fc700 1 -- 172.21.15.29:0/279579399 learned_addr learned my addr 172.21.15.29:0/279579399 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:42:30.100 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:42:30.099+0000 7f84b2ffd700 1 --2- 172.21.15.29:0/279579399 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f84b4102e60 0x7f84b4100ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:42:30.100 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:42:30.099+0000 7f84b37fe700 1 --2- 172.21.15.29:0/279579399 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f84b41a3c90 0x7f84b41a6080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:42:30.100 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:42:30.099+0000 7f84b27fc700 1 -- 172.21.15.29:0/279579399 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f84b41a3c90 msgr2=0x7f84b41a6080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:42:30.100 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:42:30.099+0000 7f84b27fc700 1 --2- 172.21.15.29:0/279579399 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f84b41a3c90 0x7f84b41a6080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:42:30.101 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:42:30.099+0000 7f84b27fc700 1 -- 172.21.15.29:0/279579399 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f84b4102e60 msgr2=0x7f84b4100ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:42:30.101 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:42:30.099+0000 7f84b27fc700 1 --2- 172.21.15.29:0/279579399 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f84b4102e60 0x7f84b4100ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:42:30.101 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:42:30.099+0000 7f84b27fc700 1 -- 172.21.15.29:0/279579399 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84b41a67b0 con 0x7f84b4101000 2024-03-16T11:42:30.101 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:42:30.099+0000 7f84b27fc700 1 --2- 172.21.15.29:0/279579399 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f84b4101000 0x7f84b4101480 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f84a400b780 tx=0x7f84a400ba90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:42:30.101 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:42:30.099+0000 7f84abfff700 1 -- 172.21.15.29:0/279579399 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84a4004230 con 0x7f84b4101000 2024-03-16T11:42:30.101 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:42:30.099+0000 7f84b97de700 1 -- 172.21.15.29:0/279579399 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f84b41a6aa0 con 0x7f84b4101000 2024-03-16T11:42:30.101 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:42:30.099+0000 7f84abfff700 1 -- 172.21.15.29:0/279579399 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f84a40043d0 con 0x7f84b4101000 2024-03-16T11:42:30.101 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:42:30.099+0000 7f84abfff700 1 -- 172.21.15.29:0/279579399 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84a4011c90 con 0x7f84b4101000 2024-03-16T11:42:30.101 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:42:30.099+0000 7f84b97de700 1 -- 172.21.15.29:0/279579399 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f84b41a7030 con 0x7f84b4101000 2024-03-16T11:42:30.102 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:42:30.100+0000 7f84abfff700 1 -- 172.21.15.29:0/279579399 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 13) ==== 46162+0+0 (secure 0 0 0) 0x7f84a401a400 con 0x7f84b4101000 2024-03-16T11:42:30.102 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:42:30.100+0000 7f84b97de700 1 -- 172.21.15.29:0/279579399 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f84b4069fa0 con 0x7f84b4101000 2024-03-16T11:42:30.102 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:42:30.100+0000 7f84abfff700 1 --2- 172.21.15.29:0/279579399 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f84a0039450 0x7f84a003b910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:42:30.102 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:42:30.100+0000 7f84abfff700 1 -- 172.21.15.29:0/279579399 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f84a404bb20 con 0x7f84b4101000 2024-03-16T11:42:30.102 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:42:30.101+0000 7f84b2ffd700 1 --2- 172.21.15.29:0/279579399 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f84a0039450 0x7f84a003b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:42:30.102 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:42:30.101+0000 7f84b2ffd700 1 --2- 172.21.15.29:0/279579399 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f84a0039450 0x7f84a003b910 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f849c006010 tx=0x7f849c0058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:42:30.105 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:42:30.105+0000 7f84abfff700 1 -- 172.21.15.29:0/279579399 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f84a4020500 con 0x7f84b4101000 2024-03-16T11:42:30.257 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:29 smithi029 ceph-mon[31975]: mon.b calling monitor election 2024-03-16T11:42:30.257 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:29 smithi029 ceph-mon[31975]: Updating smithi029:/etc/ceph/ceph.conf 2024-03-16T11:42:30.258 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:29 smithi029 ceph-mon[31975]: Updating smithi152:/etc/ceph/ceph.conf 2024-03-16T11:42:30.258 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:29 smithi029 ceph-mon[31975]: Updating smithi154:/etc/ceph/ceph.conf 2024-03-16T11:42:30.258 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:29 smithi029 ceph-mon[31975]: mon.b calling monitor election 2024-03-16T11:42:30.258 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:29 smithi029 ceph-mon[31975]: mon.c calling monitor election 2024-03-16T11:42:30.258 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:29 smithi029 ceph-mon[31975]: mon.a calling monitor election 2024-03-16T11:42:30.258 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:29 smithi029 ceph-mon[31975]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-16T11:42:30.258 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:29 smithi029 ceph-mon[31975]: monmap epoch 3 2024-03-16T11:42:30.258 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:29 smithi029 ceph-mon[31975]: fsid d1887f28-e389-11ee-95c9-87774f69a715 2024-03-16T11:42:30.258 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:29 smithi029 ceph-mon[31975]: last_changed 2024-03-16T11:42:22.059913+0000 2024-03-16T11:42:30.258 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:29 smithi029 ceph-mon[31975]: created 2024-03-16T11:40:13.432436+0000 2024-03-16T11:42:30.258 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:29 smithi029 ceph-mon[31975]: min_mon_release 19 (squid) 2024-03-16T11:42:30.258 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:29 smithi029 ceph-mon[31975]: election_strategy: 1 2024-03-16T11:42:30.258 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:29 smithi029 ceph-mon[31975]: 0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2024-03-16T11:42:30.258 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:29 smithi029 ceph-mon[31975]: 1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.c 2024-03-16T11:42:30.259 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:29 smithi029 ceph-mon[31975]: 2: [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] mon.b 2024-03-16T11:42:30.259 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:29 smithi029 ceph-mon[31975]: fsmap 2024-03-16T11:42:30.259 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:29 smithi029 ceph-mon[31975]: osdmap e4: 0 total, 0 up, 0 in 2024-03-16T11:42:30.259 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:29 smithi029 ceph-mon[31975]: mgrmap e13: x(active, since 61s) 2024-03-16T11:42:30.259 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:29 smithi029 ceph-mon[31975]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-03-16T11:42:30.259 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:29 smithi029 ceph-mon[31975]: Cluster is now healthy 2024-03-16T11:42:30.259 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:29 smithi029 ceph-mon[31975]: overall HEALTH_OK 2024-03-16T11:42:30.259 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:29 smithi029 ceph-mon[31975]: Updating smithi029:/var/lib/ceph/d1887f28-e389-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-16T11:42:30.259 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:29 smithi029 ceph-mon[31975]: Updating smithi154:/var/lib/ceph/d1887f28-e389-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-16T11:42:30.259 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:29 smithi029 ceph-mon[31975]: Updating smithi152:/var/lib/ceph/d1887f28-e389-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-16T11:42:30.259 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:29 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:42:30.259 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:29 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:42:30.259 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:29 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:42:30.259 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:29 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:42:30.259 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:29 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:42:30.259 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:29 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:42:30.260 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:29 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:42:30.260 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:29 smithi029 ceph-mon[31975]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T11:42:30.260 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:29 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:42:30.260 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:29 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:42:30.260 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:29 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:42:30.260 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:29 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:42:30.260 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:29 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-16T11:42:30.260 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:29 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-16T11:42:30.260 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:29 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:42:30.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:29 smithi154 ceph-mon[36320]: mon.b calling monitor election 2024-03-16T11:42:30.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:29 smithi154 ceph-mon[36320]: Updating smithi029:/etc/ceph/ceph.conf 2024-03-16T11:42:30.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:29 smithi154 ceph-mon[36320]: Updating smithi152:/etc/ceph/ceph.conf 2024-03-16T11:42:30.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:29 smithi154 ceph-mon[36320]: Updating smithi154:/etc/ceph/ceph.conf 2024-03-16T11:42:30.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:29 smithi154 ceph-mon[36320]: mon.b calling monitor election 2024-03-16T11:42:30.296 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:29 smithi154 ceph-mon[36320]: mon.c calling monitor election 2024-03-16T11:42:30.296 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:29 smithi154 ceph-mon[36320]: mon.a calling monitor election 2024-03-16T11:42:30.296 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:29 smithi154 ceph-mon[36320]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-16T11:42:30.296 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:29 smithi154 ceph-mon[36320]: monmap epoch 3 2024-03-16T11:42:30.296 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:29 smithi154 ceph-mon[36320]: fsid d1887f28-e389-11ee-95c9-87774f69a715 2024-03-16T11:42:30.296 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:29 smithi154 ceph-mon[36320]: last_changed 2024-03-16T11:42:22.059913+0000 2024-03-16T11:42:30.296 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:29 smithi154 ceph-mon[36320]: created 2024-03-16T11:40:13.432436+0000 2024-03-16T11:42:30.296 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:29 smithi154 ceph-mon[36320]: min_mon_release 19 (squid) 2024-03-16T11:42:30.296 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:29 smithi154 ceph-mon[36320]: election_strategy: 1 2024-03-16T11:42:30.296 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:29 smithi154 ceph-mon[36320]: 0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2024-03-16T11:42:30.296 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:29 smithi154 ceph-mon[36320]: 1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.c 2024-03-16T11:42:30.296 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:29 smithi154 ceph-mon[36320]: 2: [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] mon.b 2024-03-16T11:42:30.296 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:29 smithi154 ceph-mon[36320]: fsmap 2024-03-16T11:42:30.296 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:29 smithi154 ceph-mon[36320]: osdmap e4: 0 total, 0 up, 0 in 2024-03-16T11:42:30.296 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:29 smithi154 ceph-mon[36320]: mgrmap e13: x(active, since 61s) 2024-03-16T11:42:30.296 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:29 smithi154 ceph-mon[36320]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-03-16T11:42:30.297 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:29 smithi154 ceph-mon[36320]: Cluster is now healthy 2024-03-16T11:42:30.297 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:29 smithi154 ceph-mon[36320]: overall HEALTH_OK 2024-03-16T11:42:30.297 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:29 smithi154 ceph-mon[36320]: Updating smithi029:/var/lib/ceph/d1887f28-e389-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-16T11:42:30.297 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:29 smithi154 ceph-mon[36320]: Updating smithi154:/var/lib/ceph/d1887f28-e389-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-16T11:42:30.297 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:29 smithi154 ceph-mon[36320]: Updating smithi152:/var/lib/ceph/d1887f28-e389-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-16T11:42:30.297 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:29 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:42:30.297 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:29 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:42:30.297 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:29 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:42:30.297 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:29 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:42:30.297 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:29 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:42:30.297 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:29 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:42:30.297 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:29 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:42:30.297 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:29 smithi154 ceph-mon[36320]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T11:42:30.297 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:29 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:42:30.297 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:29 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:42:30.298 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:29 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:42:30.298 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:29 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:42:30.298 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:29 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-16T11:42:30.298 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:29 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-16T11:42:30.298 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:29 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:42:30.323 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:42:30.324+0000 7f84b97de700 1 -- 172.21.15.29:0/279579399 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7f84b41a7300 con 0x7f84b4101000 2024-03-16T11:42:30.324 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:42:30.324+0000 7f84abfff700 1 -- 172.21.15.29:0/279579399 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v9) ==== 76+0+275 (secure 0 0 0) 0x7f84a40202a0 con 0x7f84b4101000 2024-03-16T11:42:30.324 INFO:teuthology.orchestra.run.smithi029.stdout:# minimal ceph.conf for d1887f28-e389-11ee-95c9-87774f69a715 2024-03-16T11:42:30.324 INFO:teuthology.orchestra.run.smithi029.stdout:[global] 2024-03-16T11:42:30.324 INFO:teuthology.orchestra.run.smithi029.stdout: fsid = d1887f28-e389-11ee-95c9-87774f69a715 2024-03-16T11:42:30.324 INFO:teuthology.orchestra.run.smithi029.stdout: mon_host = [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-03-16T11:42:30.326 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:42:30.326+0000 7f84b97de700 1 -- 172.21.15.29:0/279579399 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f84a0039450 msgr2=0x7f84a003b910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:42:30.326 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:42:30.326+0000 7f84b97de700 1 --2- 172.21.15.29:0/279579399 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f84a0039450 0x7f84a003b910 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f849c006010 tx=0x7f849c0058e0 comp rx=0 tx=0).stop 2024-03-16T11:42:30.326 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:42:30.326+0000 7f84b97de700 1 -- 172.21.15.29:0/279579399 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f84b4101000 msgr2=0x7f84b4101480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:42:30.326 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:42:30.326+0000 7f84b97de700 1 --2- 172.21.15.29:0/279579399 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f84b4101000 0x7f84b4101480 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f84a400b780 tx=0x7f84a400ba90 comp rx=0 tx=0).stop 2024-03-16T11:42:30.326 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:42:30.326+0000 7f84b97de700 1 -- 172.21.15.29:0/279579399 shutdown_connections 2024-03-16T11:42:30.326 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:42:30.326+0000 7f84b97de700 1 --2- 172.21.15.29:0/279579399 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f84a0039450 0x7f84a003b910 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:42:30.326 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:42:30.326+0000 7f84b97de700 1 --2- 172.21.15.29:0/279579399 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f84b41a3c90 0x7f84b41a6080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:42:30.326 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:42:30.326+0000 7f84b97de700 1 --2- 172.21.15.29:0/279579399 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f84b4102e60 0x7f84b4100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:42:30.326 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:42:30.326+0000 7f84b97de700 1 --2- 172.21.15.29:0/279579399 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f84b4101000 0x7f84b4101480 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:42:30.327 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:42:30.326+0000 7f84b97de700 1 -- 172.21.15.29:0/279579399 >> 172.21.15.29:0/279579399 conn(0x7f84b4077810 msgr2=0x7f84b4103d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:42:30.327 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:42:30.327+0000 7f84b97de700 1 -- 172.21.15.29:0/279579399 shutdown_connections 2024-03-16T11:42:30.327 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:42:30.327+0000 7f84b97de700 1 -- 172.21.15.29:0/279579399 wait complete. 2024-03-16T11:42:30.378 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:29 smithi152 ceph-mon[36291]: mon.b calling monitor election 2024-03-16T11:42:30.378 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:29 smithi152 ceph-mon[36291]: Updating smithi029:/etc/ceph/ceph.conf 2024-03-16T11:42:30.378 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:29 smithi152 ceph-mon[36291]: Updating smithi152:/etc/ceph/ceph.conf 2024-03-16T11:42:30.378 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:29 smithi152 ceph-mon[36291]: Updating smithi154:/etc/ceph/ceph.conf 2024-03-16T11:42:30.378 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:29 smithi152 ceph-mon[36291]: mon.b calling monitor election 2024-03-16T11:42:30.378 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:29 smithi152 ceph-mon[36291]: mon.c calling monitor election 2024-03-16T11:42:30.378 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:29 smithi152 ceph-mon[36291]: mon.a calling monitor election 2024-03-16T11:42:30.378 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:29 smithi152 ceph-mon[36291]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-16T11:42:30.378 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:29 smithi152 ceph-mon[36291]: monmap epoch 3 2024-03-16T11:42:30.378 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:29 smithi152 ceph-mon[36291]: fsid d1887f28-e389-11ee-95c9-87774f69a715 2024-03-16T11:42:30.378 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:29 smithi152 ceph-mon[36291]: last_changed 2024-03-16T11:42:22.059913+0000 2024-03-16T11:42:30.378 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:29 smithi152 ceph-mon[36291]: created 2024-03-16T11:40:13.432436+0000 2024-03-16T11:42:30.379 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:29 smithi152 ceph-mon[36291]: min_mon_release 19 (squid) 2024-03-16T11:42:30.379 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:29 smithi152 ceph-mon[36291]: election_strategy: 1 2024-03-16T11:42:30.379 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:29 smithi152 ceph-mon[36291]: 0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2024-03-16T11:42:30.379 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:29 smithi152 ceph-mon[36291]: 1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.c 2024-03-16T11:42:30.379 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:29 smithi152 ceph-mon[36291]: 2: [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] mon.b 2024-03-16T11:42:30.379 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:29 smithi152 ceph-mon[36291]: fsmap 2024-03-16T11:42:30.379 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:29 smithi152 ceph-mon[36291]: osdmap e4: 0 total, 0 up, 0 in 2024-03-16T11:42:30.379 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:29 smithi152 ceph-mon[36291]: mgrmap e13: x(active, since 61s) 2024-03-16T11:42:30.379 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:29 smithi152 ceph-mon[36291]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-03-16T11:42:30.379 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:29 smithi152 ceph-mon[36291]: Cluster is now healthy 2024-03-16T11:42:30.379 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:29 smithi152 ceph-mon[36291]: overall HEALTH_OK 2024-03-16T11:42:30.379 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:29 smithi152 ceph-mon[36291]: Updating smithi029:/var/lib/ceph/d1887f28-e389-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-16T11:42:30.379 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:29 smithi152 ceph-mon[36291]: Updating smithi154:/var/lib/ceph/d1887f28-e389-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-16T11:42:30.379 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:29 smithi152 ceph-mon[36291]: Updating smithi152:/var/lib/ceph/d1887f28-e389-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-16T11:42:30.379 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:29 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:42:30.379 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:29 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:42:30.379 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:29 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:42:30.379 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:29 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:42:30.380 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:29 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:42:30.380 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:29 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:42:30.380 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:29 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:42:30.380 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:29 smithi152 ceph-mon[36291]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T11:42:30.380 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:29 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:42:30.380 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:29 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:42:30.380 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:29 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:42:30.380 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:29 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:42:30.380 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:29 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-16T11:42:30.380 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:29 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-16T11:42:30.380 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:29 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:42:31.041 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:30 smithi029 ceph-mon[31975]: Reconfiguring mon.a (unknown last config time)... 2024-03-16T11:42:31.041 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:30 smithi029 ceph-mon[31975]: Reconfiguring daemon mon.a on smithi029 2024-03-16T11:42:31.041 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:30 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T11:42:31.041 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:30 smithi029 ceph-mon[31975]: from='client.? 172.21.15.29:0/279579399' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:42:31.074 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-03-16T11:42:31.074 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-03-16T11:42:31.074 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd of=/etc/ceph/ceph.conf 2024-03-16T11:42:31.107 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-03-16T11:42:31.107 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-16T11:42:31.176 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-03-16T11:42:31.176 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd of=/etc/ceph/ceph.conf 2024-03-16T11:42:31.206 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:30 smithi152 ceph-mon[36291]: Reconfiguring mon.a (unknown last config time)... 2024-03-16T11:42:31.206 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:30 smithi152 ceph-mon[36291]: Reconfiguring daemon mon.a on smithi029 2024-03-16T11:42:31.206 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:30 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T11:42:31.206 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:30 smithi152 ceph-mon[36291]: from='client.? 172.21.15.29:0/279579399' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:42:31.211 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-03-16T11:42:31.211 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-16T11:42:31.281 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-03-16T11:42:31.281 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd of=/etc/ceph/ceph.conf 2024-03-16T11:42:31.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:30 smithi154 ceph-mon[36320]: Reconfiguring mon.a (unknown last config time)... 2024-03-16T11:42:31.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:30 smithi154 ceph-mon[36320]: Reconfiguring daemon mon.a on smithi029 2024-03-16T11:42:31.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:30 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T11:42:31.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:30 smithi154 ceph-mon[36320]: from='client.? 172.21.15.29:0/279579399' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:42:31.316 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-03-16T11:42:31.316 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-16T11:42:31.386 INFO:tasks.cephadm:Adding mgr.x on smithi029 2024-03-16T11:42:31.387 INFO:tasks.cephadm:Adding mgr.y on smithi152 2024-03-16T11:42:31.387 INFO:tasks.cephadm:Adding mgr.z on smithi154 2024-03-16T11:42:31.387 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d1887f28-e389-11ee-95c9-87774f69a715 -- ceph orch apply mgr '3;smithi029=x;smithi152=y;smithi154=z' 2024-03-16T11:42:31.659 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/d1887f28-e389-11ee-95c9-87774f69a715/mon.c/config 2024-03-16T11:42:32.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:31 smithi154 ceph-mon[36320]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T11:42:32.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:31 smithi152 ceph-mon[36291]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T11:42:32.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:31 smithi029 ceph-mon[31975]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T11:42:33.123 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:33.121+0000 7f34d238a700 1 -- 172.21.15.154:0/3925659447 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f34cc100cf0 msgr2=0x7f34cc101110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:42:33.123 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:33.121+0000 7f34d238a700 1 --2- 172.21.15.154:0/3925659447 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f34cc100cf0 0x7f34cc101110 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f34bc009910 tx=0x7f34bc009c20 comp rx=0 tx=0).stop 2024-03-16T11:42:33.123 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:33.121+0000 7f34d238a700 1 -- 172.21.15.154:0/3925659447 shutdown_connections 2024-03-16T11:42:33.124 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:33.121+0000 7f34d238a700 1 --2- 172.21.15.154:0/3925659447 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f34cc100cf0 0x7f34cc101110 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:42:33.124 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:33.121+0000 7f34d238a700 1 -- 172.21.15.154:0/3925659447 >> 172.21.15.154:0/3925659447 conn(0x7f34cc0fc2b0 msgr2=0x7f34cc0fe6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:42:33.124 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:33.121+0000 7f34d238a700 1 -- 172.21.15.154:0/3925659447 shutdown_connections 2024-03-16T11:42:33.124 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:33.121+0000 7f34d238a700 1 -- 172.21.15.154:0/3925659447 wait complete. 2024-03-16T11:42:33.124 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:33.122+0000 7f34d238a700 1 Processor -- start 2024-03-16T11:42:33.124 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:33.122+0000 7f34d238a700 1 -- start start 2024-03-16T11:42:33.124 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:33.122+0000 7f34d238a700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f34cc100cf0 0x7f34cc06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:42:33.124 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:33.122+0000 7f34d238a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f34cc19a380 0x7f34cc19c770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:42:33.124 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:33.122+0000 7f34d238a700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f34cc19cd00 0x7f34cc19f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:42:33.124 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:33.122+0000 7f34d238a700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f34cc106d30 con 0x7f34cc19a380 2024-03-16T11:42:33.124 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:33.122+0000 7f34d238a700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f34cc106bb0 con 0x7f34cc100cf0 2024-03-16T11:42:33.124 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:33.122+0000 7f34d238a700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f34cc106eb0 con 0x7f34cc19cd00 2024-03-16T11:42:33.124 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:33.123+0000 7f34d0927700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f34cc19cd00 0x7f34cc19f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:42:33.124 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:33.123+0000 7f34d0927700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f34cc19cd00 0x7f34cc19f1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.154:48494/0 (socket says 172.21.15.154:48494) 2024-03-16T11:42:33.124 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:33.123+0000 7f34d0927700 1 -- 172.21.15.154:0/2589794609 learned_addr learned my addr 172.21.15.154:0/2589794609 (peer_addr_for_me v2:172.21.15.154:0/0) 2024-03-16T11:42:33.125 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:33.123+0000 7f34cb7fe700 1 --2- 172.21.15.154:0/2589794609 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f34cc19a380 0x7f34cc19c770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:42:33.125 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:33.123+0000 7f34cbfff700 1 --2- 172.21.15.154:0/2589794609 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f34cc100cf0 0x7f34cc06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:42:33.125 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:33.123+0000 7f34d0927700 1 -- 172.21.15.154:0/2589794609 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f34cc100cf0 msgr2=0x7f34cc06a7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:42:33.125 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:33.123+0000 7f34d0927700 1 --2- 172.21.15.154:0/2589794609 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f34cc100cf0 0x7f34cc06a7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:42:33.125 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:33.123+0000 7f34d0927700 1 -- 172.21.15.154:0/2589794609 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f34cc19a380 msgr2=0x7f34cc19c770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:42:33.125 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:33.123+0000 7f34d0927700 1 --2- 172.21.15.154:0/2589794609 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f34cc19a380 0x7f34cc19c770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:42:33.125 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:33.123+0000 7f34d0927700 1 -- 172.21.15.154:0/2589794609 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34cc19fa70 con 0x7f34cc19cd00 2024-03-16T11:42:33.125 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:33.123+0000 7f34cbfff700 1 --2- 172.21.15.154:0/2589794609 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f34cc100cf0 0x7f34cc06a7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T11:42:33.125 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:33.123+0000 7f34cb7fe700 1 --2- 172.21.15.154:0/2589794609 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f34cc19a380 0x7f34cc19c770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:42:33.125 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:33.123+0000 7f34d0927700 1 --2- 172.21.15.154:0/2589794609 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f34cc19cd00 0x7f34cc19f1a0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f34c400ba50 tx=0x7f34c400bd60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:42:33.125 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:33.123+0000 7f34c97fa700 1 -- 172.21.15.154:0/2589794609 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34c4019070 con 0x7f34cc19cd00 2024-03-16T11:42:33.125 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:33.123+0000 7f34c97fa700 1 -- 172.21.15.154:0/2589794609 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f34c4014440 con 0x7f34cc19cd00 2024-03-16T11:42:33.125 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:33.123+0000 7f34c97fa700 1 -- 172.21.15.154:0/2589794609 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34c40075b0 con 0x7f34cc19cd00 2024-03-16T11:42:33.125 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:33.123+0000 7f34d238a700 1 -- 172.21.15.154:0/2589794609 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f34cc1961c0 con 0x7f34cc19cd00 2024-03-16T11:42:33.125 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:33.124+0000 7f34d238a700 1 -- 172.21.15.154:0/2589794609 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f34cc196730 con 0x7f34cc19cd00 2024-03-16T11:42:33.126 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:33.124+0000 7f34c97fa700 1 -- 172.21.15.154:0/2589794609 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 13) ==== 46162+0+0 (secure 0 0 0) 0x7f34c401c070 con 0x7f34cc19cd00 2024-03-16T11:42:33.126 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:33.124+0000 7f34c97fa700 1 --2- 172.21.15.154:0/2589794609 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f34b4039450 0x7f34b403b910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:42:33.127 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:33.124+0000 7f34c97fa700 1 -- 172.21.15.154:0/2589794609 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f34c404b860 con 0x7f34cc19cd00 2024-03-16T11:42:33.127 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:33.124+0000 7f34d238a700 1 -- 172.21.15.154:0/2589794609 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f34cc063900 con 0x7f34cc19cd00 2024-03-16T11:42:33.127 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:33.125+0000 7f34cbfff700 1 --2- 172.21.15.154:0/2589794609 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f34b4039450 0x7f34b403b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:42:33.127 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:33.125+0000 7f34cbfff700 1 --2- 172.21.15.154:0/2589794609 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f34b4039450 0x7f34b403b910 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f34bc00b5c0 tx=0x7f34bc01a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:42:33.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:32 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:42:33.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:32 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:42:33.128 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:32 smithi152 ceph-mon[36291]: Reconfiguring mon.b (monmap changed)... 2024-03-16T11:42:33.128 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:32 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-16T11:42:33.128 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:32 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-16T11:42:33.128 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:32 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:42:33.128 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:32 smithi152 ceph-mon[36291]: Reconfiguring daemon mon.b on smithi152 2024-03-16T11:42:33.129 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:33.130+0000 7f34c97fa700 1 -- 172.21.15.154:0/2589794609 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f34c4012740 con 0x7f34cc19cd00 2024-03-16T11:42:33.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:32 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:42:33.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:32 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:42:33.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:32 smithi154 ceph-mon[36320]: Reconfiguring mon.b (monmap changed)... 2024-03-16T11:42:33.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:32 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-16T11:42:33.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:32 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-16T11:42:33.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:32 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:42:33.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:32 smithi154 ceph-mon[36320]: Reconfiguring daemon mon.b on smithi152 2024-03-16T11:42:33.334 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:33.334+0000 7f34d238a700 1 -- 172.21.15.154:0/2589794609 --> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi029=x;smithi152=y;smithi154=z", "target": ["mon-mgr", ""]}) -- 0x7f34cc197be0 con 0x7f34b4039450 2024-03-16T11:42:33.337 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:32 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:42:33.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:32 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:42:33.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:32 smithi029 ceph-mon[31975]: Reconfiguring mon.b (monmap changed)... 2024-03-16T11:42:33.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:32 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-16T11:42:33.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:32 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-16T11:42:33.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:32 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:42:33.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:32 smithi029 ceph-mon[31975]: Reconfiguring daemon mon.b on smithi152 2024-03-16T11:42:33.340 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:33.341+0000 7f34c97fa700 1 -- 172.21.15.154:0/2589794609 <== mgr.14152 v2:172.21.15.29:6800/4192064640 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f34cc197be0 con 0x7f34b4039450 2024-03-16T11:42:33.340 INFO:teuthology.orchestra.run.smithi154.stdout:Scheduled mgr update... 2024-03-16T11:42:33.343 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:33.344+0000 7f34d238a700 1 -- 172.21.15.154:0/2589794609 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f34b4039450 msgr2=0x7f34b403b910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:42:33.344 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:33.344+0000 7f34d238a700 1 --2- 172.21.15.154:0/2589794609 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f34b4039450 0x7f34b403b910 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f34bc00b5c0 tx=0x7f34bc01a040 comp rx=0 tx=0).stop 2024-03-16T11:42:33.344 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:33.344+0000 7f34d238a700 1 -- 172.21.15.154:0/2589794609 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f34cc19cd00 msgr2=0x7f34cc19f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:42:33.344 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:33.344+0000 7f34d238a700 1 --2- 172.21.15.154:0/2589794609 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f34cc19cd00 0x7f34cc19f1a0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f34c400ba50 tx=0x7f34c400bd60 comp rx=0 tx=0).stop 2024-03-16T11:42:33.344 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:33.344+0000 7f34d238a700 1 -- 172.21.15.154:0/2589794609 shutdown_connections 2024-03-16T11:42:33.344 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:33.344+0000 7f34d238a700 1 --2- 172.21.15.154:0/2589794609 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f34b4039450 0x7f34b403b910 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:42:33.344 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:33.344+0000 7f34d238a700 1 --2- 172.21.15.154:0/2589794609 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f34cc19cd00 0x7f34cc19f1a0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:42:33.344 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:33.344+0000 7f34d238a700 1 --2- 172.21.15.154:0/2589794609 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f34cc100cf0 0x7f34cc06a7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:42:33.344 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:33.344+0000 7f34d238a700 1 --2- 172.21.15.154:0/2589794609 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f34cc19a380 0x7f34cc19c770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:42:33.344 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:33.344+0000 7f34d238a700 1 -- 172.21.15.154:0/2589794609 >> 172.21.15.154:0/2589794609 conn(0x7f34cc0fc2b0 msgr2=0x7f34cc0fcf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:42:33.344 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:33.344+0000 7f34d238a700 1 -- 172.21.15.154:0/2589794609 shutdown_connections 2024-03-16T11:42:33.344 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:42:33.344+0000 7f34d238a700 1 -- 172.21.15.154:0/2589794609 wait complete. 2024-03-16T11:42:33.969 DEBUG:teuthology.orchestra.run.smithi152:mgr.y> sudo journalctl -f -n 0 -u ceph-d1887f28-e389-11ee-95c9-87774f69a715@mgr.y.service 2024-03-16T11:42:33.972 DEBUG:teuthology.orchestra.run.smithi154:mgr.z> sudo journalctl -f -n 0 -u ceph-d1887f28-e389-11ee-95c9-87774f69a715@mgr.z.service 2024-03-16T11:42:33.974 INFO:tasks.cephadm:Deploying OSDs... 2024-03-16T11:42:33.975 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-03-16T11:42:33.975 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/scratch_devs of=/dev/stdout 2024-03-16T11:42:33.993 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-16T11:42:33.994 DEBUG:teuthology.orchestra.run.smithi029:> stat /dev/vg_nvme/lv_1 2024-03-16T11:42:34.052 INFO:teuthology.orchestra.run.smithi029.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-16T11:42:34.052 INFO:teuthology.orchestra.run.smithi029.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-16T11:42:34.052 INFO:teuthology.orchestra.run.smithi029.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-03-16T11:42:34.052 INFO:teuthology.orchestra.run.smithi029.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-16T11:42:34.052 INFO:teuthology.orchestra.run.smithi029.stdout:Context: system_u:object_r:device_t:s0 2024-03-16T11:42:34.052 INFO:teuthology.orchestra.run.smithi029.stdout:Access: 2024-03-16 11:41:41.465385266 +0000 2024-03-16T11:42:34.052 INFO:teuthology.orchestra.run.smithi029.stdout:Modify: 2024-03-16 11:35:22.779633183 +0000 2024-03-16T11:42:34.052 INFO:teuthology.orchestra.run.smithi029.stdout:Change: 2024-03-16 11:35:22.779633183 +0000 2024-03-16T11:42:34.052 INFO:teuthology.orchestra.run.smithi029.stdout: Birth: 2024-03-16 11:35:22.779633183 +0000 2024-03-16T11:42:34.052 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-16T11:42:34.122 INFO:teuthology.orchestra.run.smithi029.stderr:1+0 records in 2024-03-16T11:42:34.122 INFO:teuthology.orchestra.run.smithi029.stderr:1+0 records out 2024-03-16T11:42:34.122 INFO:teuthology.orchestra.run.smithi029.stderr:512 bytes copied, 0.000114154 s, 4.5 MB/s 2024-03-16T11:42:34.123 DEBUG:teuthology.orchestra.run.smithi029:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-16T11:42:34.181 DEBUG:teuthology.orchestra.run.smithi029:> stat /dev/vg_nvme/lv_2 2024-03-16T11:42:34.237 INFO:teuthology.orchestra.run.smithi029.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-16T11:42:34.237 INFO:teuthology.orchestra.run.smithi029.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-16T11:42:34.238 INFO:teuthology.orchestra.run.smithi029.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-03-16T11:42:34.238 INFO:teuthology.orchestra.run.smithi029.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-16T11:42:34.238 INFO:teuthology.orchestra.run.smithi029.stdout:Context: system_u:object_r:device_t:s0 2024-03-16T11:42:34.238 INFO:teuthology.orchestra.run.smithi029.stdout:Access: 2024-03-16 11:41:41.493384781 +0000 2024-03-16T11:42:34.238 INFO:teuthology.orchestra.run.smithi029.stdout:Modify: 2024-03-16 11:35:23.115627450 +0000 2024-03-16T11:42:34.238 INFO:teuthology.orchestra.run.smithi029.stdout:Change: 2024-03-16 11:35:23.115627450 +0000 2024-03-16T11:42:34.238 INFO:teuthology.orchestra.run.smithi029.stdout: Birth: 2024-03-16 11:35:23.115627450 +0000 2024-03-16T11:42:34.238 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-16T11:42:34.306 INFO:teuthology.orchestra.run.smithi029.stderr:1+0 records in 2024-03-16T11:42:34.306 INFO:teuthology.orchestra.run.smithi029.stderr:1+0 records out 2024-03-16T11:42:34.306 INFO:teuthology.orchestra.run.smithi029.stderr:512 bytes copied, 0.000101264 s, 5.1 MB/s 2024-03-16T11:42:34.307 DEBUG:teuthology.orchestra.run.smithi029:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-16T11:42:34.363 DEBUG:teuthology.orchestra.run.smithi029:> stat /dev/vg_nvme/lv_3 2024-03-16T11:42:34.419 INFO:teuthology.orchestra.run.smithi029.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-16T11:42:34.420 INFO:teuthology.orchestra.run.smithi029.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-16T11:42:34.420 INFO:teuthology.orchestra.run.smithi029.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-03-16T11:42:34.420 INFO:teuthology.orchestra.run.smithi029.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-16T11:42:34.420 INFO:teuthology.orchestra.run.smithi029.stdout:Context: system_u:object_r:device_t:s0 2024-03-16T11:42:34.420 INFO:teuthology.orchestra.run.smithi029.stdout:Access: 2024-03-16 11:41:41.520384314 +0000 2024-03-16T11:42:34.420 INFO:teuthology.orchestra.run.smithi029.stdout:Modify: 2024-03-16 11:35:23.452621701 +0000 2024-03-16T11:42:34.420 INFO:teuthology.orchestra.run.smithi029.stdout:Change: 2024-03-16 11:35:23.452621701 +0000 2024-03-16T11:42:34.420 INFO:teuthology.orchestra.run.smithi029.stdout: Birth: 2024-03-16 11:35:23.452621701 +0000 2024-03-16T11:42:34.420 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-16T11:42:34.485 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:34 smithi029 ceph-mon[31975]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi029=x;smithi152=y;smithi154=z", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T11:42:34.485 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:34 smithi029 ceph-mon[31975]: Saving service mgr spec with placement smithi029=x;smithi152=y;smithi154=z;count:3 2024-03-16T11:42:34.485 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:34 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:42:34.485 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:34 smithi029 ceph-mon[31975]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T11:42:34.485 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:34 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:42:34.485 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:34 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:42:34.485 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:34 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-16T11:42:34.485 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:34 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-16T11:42:34.485 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:34 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:42:34.486 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:34 smithi154 ceph-mon[36320]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi029=x;smithi152=y;smithi154=z", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T11:42:34.486 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:34 smithi154 ceph-mon[36320]: Saving service mgr spec with placement smithi029=x;smithi152=y;smithi154=z;count:3 2024-03-16T11:42:34.486 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:34 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:42:34.486 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:34 smithi154 ceph-mon[36320]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T11:42:34.486 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:34 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:42:34.486 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:34 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:42:34.486 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:34 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-16T11:42:34.486 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:34 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-16T11:42:34.486 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:34 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:42:34.487 INFO:teuthology.orchestra.run.smithi029.stderr:1+0 records in 2024-03-16T11:42:34.487 INFO:teuthology.orchestra.run.smithi029.stderr:1+0 records out 2024-03-16T11:42:34.487 INFO:teuthology.orchestra.run.smithi029.stderr:512 bytes copied, 0.000106651 s, 4.8 MB/s 2024-03-16T11:42:34.488 DEBUG:teuthology.orchestra.run.smithi029:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-16T11:42:34.545 DEBUG:teuthology.orchestra.run.smithi029:> stat /dev/vg_nvme/lv_4 2024-03-16T11:42:34.600 INFO:teuthology.orchestra.run.smithi029.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-16T11:42:34.600 INFO:teuthology.orchestra.run.smithi029.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-16T11:42:34.600 INFO:teuthology.orchestra.run.smithi029.stdout:Device: 5h/5d Inode: 725 Links: 1 2024-03-16T11:42:34.600 INFO:teuthology.orchestra.run.smithi029.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-16T11:42:34.600 INFO:teuthology.orchestra.run.smithi029.stdout:Context: system_u:object_r:device_t:s0 2024-03-16T11:42:34.600 INFO:teuthology.orchestra.run.smithi029.stdout:Access: 2024-03-16 11:41:41.546383864 +0000 2024-03-16T11:42:34.601 INFO:teuthology.orchestra.run.smithi029.stdout:Modify: 2024-03-16 11:35:23.783616054 +0000 2024-03-16T11:42:34.601 INFO:teuthology.orchestra.run.smithi029.stdout:Change: 2024-03-16 11:35:23.783616054 +0000 2024-03-16T11:42:34.601 INFO:teuthology.orchestra.run.smithi029.stdout: Birth: 2024-03-16 11:35:23.783616054 +0000 2024-03-16T11:42:34.601 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-16T11:42:34.627 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:34 smithi152 ceph-mon[36291]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi029=x;smithi152=y;smithi154=z", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T11:42:34.627 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:34 smithi152 ceph-mon[36291]: Saving service mgr spec with placement smithi029=x;smithi152=y;smithi154=z;count:3 2024-03-16T11:42:34.628 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:34 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:42:34.628 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:34 smithi152 ceph-mon[36291]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T11:42:34.628 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:34 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:42:34.628 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:34 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:42:34.628 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:34 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-16T11:42:34.628 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:34 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-16T11:42:34.628 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:34 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:42:34.666 INFO:teuthology.orchestra.run.smithi029.stderr:1+0 records in 2024-03-16T11:42:34.667 INFO:teuthology.orchestra.run.smithi029.stderr:1+0 records out 2024-03-16T11:42:34.667 INFO:teuthology.orchestra.run.smithi029.stderr:512 bytes copied, 0.000109359 s, 4.7 MB/s 2024-03-16T11:42:34.668 DEBUG:teuthology.orchestra.run.smithi029:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-16T11:42:34.724 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-03-16T11:42:34.724 DEBUG:teuthology.orchestra.run.smithi152:> dd if=/scratch_devs of=/dev/stdout 2024-03-16T11:42:34.743 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-16T11:42:34.743 DEBUG:teuthology.orchestra.run.smithi152:> stat /dev/vg_nvme/lv_1 2024-03-16T11:42:34.801 INFO:teuthology.orchestra.run.smithi152.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-16T11:42:34.801 INFO:teuthology.orchestra.run.smithi152.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-16T11:42:34.801 INFO:teuthology.orchestra.run.smithi152.stdout:Device: 5h/5d Inode: 664 Links: 1 2024-03-16T11:42:34.801 INFO:teuthology.orchestra.run.smithi152.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-16T11:42:34.801 INFO:teuthology.orchestra.run.smithi152.stdout:Context: system_u:object_r:device_t:s0 2024-03-16T11:42:34.801 INFO:teuthology.orchestra.run.smithi152.stdout:Access: 2024-03-16 11:42:21.481107755 +0000 2024-03-16T11:42:34.801 INFO:teuthology.orchestra.run.smithi152.stdout:Modify: 2024-03-16 11:35:29.024861405 +0000 2024-03-16T11:42:34.801 INFO:teuthology.orchestra.run.smithi152.stdout:Change: 2024-03-16 11:35:29.024861405 +0000 2024-03-16T11:42:34.801 INFO:teuthology.orchestra.run.smithi152.stdout: Birth: 2024-03-16 11:35:29.024861405 +0000 2024-03-16T11:42:34.802 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-16T11:42:34.871 INFO:teuthology.orchestra.run.smithi152.stderr:1+0 records in 2024-03-16T11:42:34.871 INFO:teuthology.orchestra.run.smithi152.stderr:1+0 records out 2024-03-16T11:42:34.872 INFO:teuthology.orchestra.run.smithi152.stderr:512 bytes copied, 0.000237284 s, 2.2 MB/s 2024-03-16T11:42:34.873 DEBUG:teuthology.orchestra.run.smithi152:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-16T11:42:34.931 DEBUG:teuthology.orchestra.run.smithi152:> stat /dev/vg_nvme/lv_2 2024-03-16T11:42:34.987 INFO:teuthology.orchestra.run.smithi152.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-16T11:42:34.987 INFO:teuthology.orchestra.run.smithi152.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-16T11:42:34.987 INFO:teuthology.orchestra.run.smithi152.stdout:Device: 5h/5d Inode: 682 Links: 1 2024-03-16T11:42:34.987 INFO:teuthology.orchestra.run.smithi152.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-16T11:42:34.988 INFO:teuthology.orchestra.run.smithi152.stdout:Context: system_u:object_r:device_t:s0 2024-03-16T11:42:34.988 INFO:teuthology.orchestra.run.smithi152.stdout:Access: 2024-03-16 11:42:21.481107755 +0000 2024-03-16T11:42:34.988 INFO:teuthology.orchestra.run.smithi152.stdout:Modify: 2024-03-16 11:35:29.365856884 +0000 2024-03-16T11:42:34.988 INFO:teuthology.orchestra.run.smithi152.stdout:Change: 2024-03-16 11:35:29.365856884 +0000 2024-03-16T11:42:34.988 INFO:teuthology.orchestra.run.smithi152.stdout: Birth: 2024-03-16 11:35:29.365856884 +0000 2024-03-16T11:42:34.988 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-16T11:42:35.055 INFO:teuthology.orchestra.run.smithi152.stderr:1+0 records in 2024-03-16T11:42:35.055 INFO:teuthology.orchestra.run.smithi152.stderr:1+0 records out 2024-03-16T11:42:35.055 INFO:teuthology.orchestra.run.smithi152.stderr:512 bytes copied, 0.000239877 s, 2.1 MB/s 2024-03-16T11:42:35.056 DEBUG:teuthology.orchestra.run.smithi152:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-16T11:42:35.114 DEBUG:teuthology.orchestra.run.smithi152:> stat /dev/vg_nvme/lv_3 2024-03-16T11:42:35.170 INFO:teuthology.orchestra.run.smithi152.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-16T11:42:35.170 INFO:teuthology.orchestra.run.smithi152.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-16T11:42:35.170 INFO:teuthology.orchestra.run.smithi152.stdout:Device: 5h/5d Inode: 702 Links: 1 2024-03-16T11:42:35.170 INFO:teuthology.orchestra.run.smithi152.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-16T11:42:35.170 INFO:teuthology.orchestra.run.smithi152.stdout:Context: system_u:object_r:device_t:s0 2024-03-16T11:42:35.170 INFO:teuthology.orchestra.run.smithi152.stdout:Access: 2024-03-16 11:42:21.481107755 +0000 2024-03-16T11:42:35.170 INFO:teuthology.orchestra.run.smithi152.stdout:Modify: 2024-03-16 11:35:29.700852443 +0000 2024-03-16T11:42:35.170 INFO:teuthology.orchestra.run.smithi152.stdout:Change: 2024-03-16 11:35:29.700852443 +0000 2024-03-16T11:42:35.170 INFO:teuthology.orchestra.run.smithi152.stdout: Birth: 2024-03-16 11:35:29.700852443 +0000 2024-03-16T11:42:35.171 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-16T11:42:35.236 INFO:teuthology.orchestra.run.smithi152.stderr:1+0 records in 2024-03-16T11:42:35.236 INFO:teuthology.orchestra.run.smithi152.stderr:1+0 records out 2024-03-16T11:42:35.236 INFO:teuthology.orchestra.run.smithi152.stderr:512 bytes copied, 0.000245502 s, 2.1 MB/s 2024-03-16T11:42:35.238 DEBUG:teuthology.orchestra.run.smithi152:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-16T11:42:35.294 DEBUG:teuthology.orchestra.run.smithi152:> stat /dev/vg_nvme/lv_4 2024-03-16T11:42:35.349 INFO:teuthology.orchestra.run.smithi152.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-16T11:42:35.349 INFO:teuthology.orchestra.run.smithi152.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-16T11:42:35.349 INFO:teuthology.orchestra.run.smithi152.stdout:Device: 5h/5d Inode: 721 Links: 1 2024-03-16T11:42:35.349 INFO:teuthology.orchestra.run.smithi152.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-16T11:42:35.349 INFO:teuthology.orchestra.run.smithi152.stdout:Context: system_u:object_r:device_t:s0 2024-03-16T11:42:35.349 INFO:teuthology.orchestra.run.smithi152.stdout:Access: 2024-03-16 11:42:21.482107742 +0000 2024-03-16T11:42:35.349 INFO:teuthology.orchestra.run.smithi152.stdout:Modify: 2024-03-16 11:35:30.036847989 +0000 2024-03-16T11:42:35.349 INFO:teuthology.orchestra.run.smithi152.stdout:Change: 2024-03-16 11:35:30.036847989 +0000 2024-03-16T11:42:35.349 INFO:teuthology.orchestra.run.smithi152.stdout: Birth: 2024-03-16 11:35:30.036847989 +0000 2024-03-16T11:42:35.350 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-16T11:42:35.415 INFO:teuthology.orchestra.run.smithi152.stderr:1+0 records in 2024-03-16T11:42:35.415 INFO:teuthology.orchestra.run.smithi152.stderr:1+0 records out 2024-03-16T11:42:35.415 INFO:teuthology.orchestra.run.smithi152.stderr:512 bytes copied, 0.000177862 s, 2.9 MB/s 2024-03-16T11:42:35.416 DEBUG:teuthology.orchestra.run.smithi152:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-16T11:42:35.473 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-03-16T11:42:35.473 DEBUG:teuthology.orchestra.run.smithi154:> dd if=/scratch_devs of=/dev/stdout 2024-03-16T11:42:35.488 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-16T11:42:35.488 DEBUG:teuthology.orchestra.run.smithi154:> stat /dev/vg_nvme/lv_1 2024-03-16T11:42:35.543 INFO:teuthology.orchestra.run.smithi154.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-16T11:42:35.543 INFO:teuthology.orchestra.run.smithi154.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-16T11:42:35.543 INFO:teuthology.orchestra.run.smithi154.stdout:Device: 5h/5d Inode: 660 Links: 1 2024-03-16T11:42:35.543 INFO:teuthology.orchestra.run.smithi154.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-16T11:42:35.543 INFO:teuthology.orchestra.run.smithi154.stdout:Context: system_u:object_r:device_t:s0 2024-03-16T11:42:35.543 INFO:teuthology.orchestra.run.smithi154.stdout:Access: 2024-03-16 11:42:12.474028741 +0000 2024-03-16T11:42:35.543 INFO:teuthology.orchestra.run.smithi154.stdout:Modify: 2024-03-16 11:35:28.725683702 +0000 2024-03-16T11:42:35.543 INFO:teuthology.orchestra.run.smithi154.stdout:Change: 2024-03-16 11:35:28.725683702 +0000 2024-03-16T11:42:35.543 INFO:teuthology.orchestra.run.smithi154.stdout: Birth: 2024-03-16 11:35:28.725683702 +0000 2024-03-16T11:42:35.543 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-16T11:42:35.607 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:35 smithi154 ceph-mon[36320]: Reconfiguring mon.c (monmap changed)... 2024-03-16T11:42:35.607 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:35 smithi154 ceph-mon[36320]: Reconfiguring daemon mon.c on smithi154 2024-03-16T11:42:35.607 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:35 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:42:35.607 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:35 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:42:35.607 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:35 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T11:42:35.610 INFO:teuthology.orchestra.run.smithi154.stderr:1+0 records in 2024-03-16T11:42:35.610 INFO:teuthology.orchestra.run.smithi154.stderr:1+0 records out 2024-03-16T11:42:35.610 INFO:teuthology.orchestra.run.smithi154.stderr:512 bytes copied, 0.000270781 s, 1.9 MB/s 2024-03-16T11:42:35.611 DEBUG:teuthology.orchestra.run.smithi154:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-16T11:42:35.627 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:35 smithi152 ceph-mon[36291]: Reconfiguring mon.c (monmap changed)... 2024-03-16T11:42:35.627 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:35 smithi152 ceph-mon[36291]: Reconfiguring daemon mon.c on smithi154 2024-03-16T11:42:35.627 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:35 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:42:35.627 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:35 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:42:35.627 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:35 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T11:42:35.666 DEBUG:teuthology.orchestra.run.smithi154:> stat /dev/vg_nvme/lv_2 2024-03-16T11:42:35.721 INFO:teuthology.orchestra.run.smithi154.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-16T11:42:35.721 INFO:teuthology.orchestra.run.smithi154.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-16T11:42:35.721 INFO:teuthology.orchestra.run.smithi154.stdout:Device: 5h/5d Inode: 679 Links: 1 2024-03-16T11:42:35.721 INFO:teuthology.orchestra.run.smithi154.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-16T11:42:35.721 INFO:teuthology.orchestra.run.smithi154.stdout:Context: system_u:object_r:device_t:s0 2024-03-16T11:42:35.721 INFO:teuthology.orchestra.run.smithi154.stdout:Access: 2024-03-16 11:42:12.474028741 +0000 2024-03-16T11:42:35.721 INFO:teuthology.orchestra.run.smithi154.stdout:Modify: 2024-03-16 11:35:29.058675074 +0000 2024-03-16T11:42:35.721 INFO:teuthology.orchestra.run.smithi154.stdout:Change: 2024-03-16 11:35:29.058675074 +0000 2024-03-16T11:42:35.721 INFO:teuthology.orchestra.run.smithi154.stdout: Birth: 2024-03-16 11:35:29.058675074 +0000 2024-03-16T11:42:35.721 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-16T11:42:35.786 INFO:teuthology.orchestra.run.smithi154.stderr:1+0 records in 2024-03-16T11:42:35.786 INFO:teuthology.orchestra.run.smithi154.stderr:1+0 records out 2024-03-16T11:42:35.786 INFO:teuthology.orchestra.run.smithi154.stderr:512 bytes copied, 0.000199639 s, 2.6 MB/s 2024-03-16T11:42:35.788 DEBUG:teuthology.orchestra.run.smithi154:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-16T11:42:35.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:35 smithi029 ceph-mon[31975]: Reconfiguring mon.c (monmap changed)... 2024-03-16T11:42:35.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:35 smithi029 ceph-mon[31975]: Reconfiguring daemon mon.c on smithi154 2024-03-16T11:42:35.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:35 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:42:35.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:35 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:42:35.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:35 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T11:42:35.845 DEBUG:teuthology.orchestra.run.smithi154:> stat /dev/vg_nvme/lv_3 2024-03-16T11:42:35.900 INFO:teuthology.orchestra.run.smithi154.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-16T11:42:35.900 INFO:teuthology.orchestra.run.smithi154.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-16T11:42:35.900 INFO:teuthology.orchestra.run.smithi154.stdout:Device: 5h/5d Inode: 700 Links: 1 2024-03-16T11:42:35.900 INFO:teuthology.orchestra.run.smithi154.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-16T11:42:35.900 INFO:teuthology.orchestra.run.smithi154.stdout:Context: system_u:object_r:device_t:s0 2024-03-16T11:42:35.900 INFO:teuthology.orchestra.run.smithi154.stdout:Access: 2024-03-16 11:42:12.474028741 +0000 2024-03-16T11:42:35.900 INFO:teuthology.orchestra.run.smithi154.stdout:Modify: 2024-03-16 11:35:29.401666188 +0000 2024-03-16T11:42:35.900 INFO:teuthology.orchestra.run.smithi154.stdout:Change: 2024-03-16 11:35:29.401666188 +0000 2024-03-16T11:42:35.901 INFO:teuthology.orchestra.run.smithi154.stdout: Birth: 2024-03-16 11:35:29.401666188 +0000 2024-03-16T11:42:35.901 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-16T11:42:35.967 INFO:teuthology.orchestra.run.smithi154.stderr:1+0 records in 2024-03-16T11:42:35.967 INFO:teuthology.orchestra.run.smithi154.stderr:1+0 records out 2024-03-16T11:42:35.967 INFO:teuthology.orchestra.run.smithi154.stderr:512 bytes copied, 0.000199133 s, 2.6 MB/s 2024-03-16T11:42:35.968 DEBUG:teuthology.orchestra.run.smithi154:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-16T11:42:36.024 DEBUG:teuthology.orchestra.run.smithi154:> stat /dev/vg_nvme/lv_4 2024-03-16T11:42:36.079 INFO:teuthology.orchestra.run.smithi154.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-16T11:42:36.079 INFO:teuthology.orchestra.run.smithi154.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-16T11:42:36.079 INFO:teuthology.orchestra.run.smithi154.stdout:Device: 5h/5d Inode: 718 Links: 1 2024-03-16T11:42:36.079 INFO:teuthology.orchestra.run.smithi154.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-16T11:42:36.079 INFO:teuthology.orchestra.run.smithi154.stdout:Context: system_u:object_r:device_t:s0 2024-03-16T11:42:36.079 INFO:teuthology.orchestra.run.smithi154.stdout:Access: 2024-03-16 11:42:12.475028716 +0000 2024-03-16T11:42:36.079 INFO:teuthology.orchestra.run.smithi154.stdout:Modify: 2024-03-16 11:35:29.735657534 +0000 2024-03-16T11:42:36.079 INFO:teuthology.orchestra.run.smithi154.stdout:Change: 2024-03-16 11:35:29.735657534 +0000 2024-03-16T11:42:36.079 INFO:teuthology.orchestra.run.smithi154.stdout: Birth: 2024-03-16 11:35:29.735657534 +0000 2024-03-16T11:42:36.079 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-16T11:42:36.144 INFO:teuthology.orchestra.run.smithi154.stderr:1+0 records in 2024-03-16T11:42:36.145 INFO:teuthology.orchestra.run.smithi154.stderr:1+0 records out 2024-03-16T11:42:36.145 INFO:teuthology.orchestra.run.smithi154.stderr:512 bytes copied, 0.000176088 s, 2.9 MB/s 2024-03-16T11:42:36.146 DEBUG:teuthology.orchestra.run.smithi154:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-16T11:42:36.201 INFO:tasks.cephadm:Deploying osd.0 on smithi029 with /dev/vg_nvme/lv_4... 2024-03-16T11:42:36.201 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d1887f28-e389-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-16T11:42:36.442 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/d1887f28-e389-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T11:42:36.564 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:36 smithi029 ceph-mon[31975]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T11:42:36.564 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:36 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:42:36.564 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:36 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T11:42:36.564 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:36 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:42:36.564 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:36 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-16T11:42:36.564 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:36 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-16T11:42:36.564 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:36 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-16T11:42:36.564 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:36 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:42:36.627 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:36 smithi152 ceph-mon[36291]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T11:42:36.627 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:36 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:42:36.628 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:36 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T11:42:36.628 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:36 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:42:36.628 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:36 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-16T11:42:36.628 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:36 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-16T11:42:36.628 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:36 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-16T11:42:36.628 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:36 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:42:36.725 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:36 smithi154 ceph-mon[36320]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T11:42:36.725 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:36 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:42:36.725 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:36 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T11:42:36.725 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:36 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:42:36.725 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:36 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-16T11:42:36.725 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:36 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-16T11:42:36.725 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:36 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-16T11:42:36.725 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:36 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:42:37.523 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:37 smithi154 ceph-mon[36320]: Deploying daemon mgr.z on smithi154 2024-03-16T11:42:37.627 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:37 smithi152 ceph-mon[36291]: Deploying daemon mgr.z on smithi154 2024-03-16T11:42:37.631 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:37 smithi029 ceph-mon[31975]: Deploying daemon mgr.z on smithi154 2024-03-16T11:42:37.795 INFO:journalctl@ceph.mgr.z.smithi154.stdout:Mar 16 11:42:37 smithi154 systemd[1]: Starting Ceph mgr.z for d1887f28-e389-11ee-95c9-87774f69a715... 2024-03-16T11:42:38.218 INFO:journalctl@ceph.mgr.z.smithi154.stdout:Mar 16 11:42:37 smithi154 podman[38167]: 2024-03-16 11:42:37.821797155 +0000 UTC m=+0.017501687 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-16T11:42:38.218 INFO:journalctl@ceph.mgr.z.smithi154.stdout:Mar 16 11:42:37 smithi154 podman[38167]: 2024-03-16 11:42:37.931180553 +0000 UTC m=+0.126885071 container create d145605c62c862ce9ca2f08b5e8acb8ddeb48d489ef448d22e9da39d63f5455a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-z, ceph=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-03-16T11:42:38.218 INFO:journalctl@ceph.mgr.z.smithi154.stdout:Mar 16 11:42:38 smithi154 podman[38167]: 2024-03-16 11:42:38.116813222 +0000 UTC m=+0.312517749 container init d145605c62c862ce9ca2f08b5e8acb8ddeb48d489ef448d22e9da39d63f5455a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-z, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-03-16T11:42:38.218 INFO:journalctl@ceph.mgr.z.smithi154.stdout:Mar 16 11:42:38 smithi154 podman[38167]: 2024-03-16 11:42:38.119429495 +0000 UTC m=+0.315134014 container start d145605c62c862ce9ca2f08b5e8acb8ddeb48d489ef448d22e9da39d63f5455a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-z, ceph=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-03-16T11:42:38.218 INFO:journalctl@ceph.mgr.z.smithi154.stdout:Mar 16 11:42:38 smithi154 ceph-mgr[38186]: -- 172.21.15.154:0/4155469388 <== mon.1 v2:172.21.15.154:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55a08cfd05a0 con 0x55a08cfa5000 2024-03-16T11:42:38.218 INFO:journalctl@ceph.mgr.z.smithi154.stdout:Mar 16 11:42:38 smithi154 bash[38167]: d145605c62c862ce9ca2f08b5e8acb8ddeb48d489ef448d22e9da39d63f5455a 2024-03-16T11:42:38.297 INFO:teuthology.orchestra.run.smithi029.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-16T11:42:38.297 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-16T11:42:38.297 INFO:teuthology.orchestra.run.smithi029.stderr: stderr: 10+0 records in 2024-03-16T11:42:38.297 INFO:teuthology.orchestra.run.smithi029.stderr:10+0 records out 2024-03-16T11:42:38.297 INFO:teuthology.orchestra.run.smithi029.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00930487 s, 1.1 GB/s 2024-03-16T11:42:38.298 INFO:teuthology.orchestra.run.smithi029.stderr:--> Zapping successful for: 2024-03-16T11:42:38.545 INFO:journalctl@ceph.mgr.z.smithi154.stdout:Mar 16 11:42:38 smithi154 systemd[1]: Started Ceph mgr.z for d1887f28-e389-11ee-95c9-87774f69a715. 2024-03-16T11:42:38.545 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:38 smithi154 ceph-mon[36320]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T11:42:38.545 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:38 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:42:38.545 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:38 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:42:38.545 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:38 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:42:38.545 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:38 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-16T11:42:38.546 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:38 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-16T11:42:38.588 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:38 smithi029 ceph-mon[31975]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T11:42:38.588 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:38 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:42:38.588 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:38 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:42:38.588 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:38 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:42:38.588 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:38 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-16T11:42:38.588 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:38 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-16T11:42:38.627 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:38 smithi152 ceph-mon[36291]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T11:42:38.627 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:38 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:42:38.627 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:38 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:42:38.627 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:38 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:42:38.627 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:38 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-16T11:42:38.628 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:38 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-16T11:42:38.901 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d1887f28-e389-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi029:vg_nvme/lv_4 2024-03-16T11:42:39.045 INFO:journalctl@ceph.mgr.z.smithi154.stdout:Mar 16 11:42:38 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-z[38182]: 2024-03-16T11:42:38.668+0000 7fb6b8963200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-16T11:42:39.045 INFO:journalctl@ceph.mgr.z.smithi154.stdout:Mar 16 11:42:38 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-z[38182]: 2024-03-16T11:42:38.761+0000 7fb6b8963200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-16T11:42:39.141 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/d1887f28-e389-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T11:42:39.560 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:39 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-16T11:42:39.560 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:39 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:42:39.560 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:39 smithi154 ceph-mon[36320]: Deploying daemon mgr.y on smithi152 2024-03-16T11:42:39.560 INFO:journalctl@ceph.mgr.z.smithi154.stdout:Mar 16 11:42:39 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-z[38182]: 2024-03-16T11:42:39.406+0000 7fb6b8963200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-16T11:42:39.560 INFO:journalctl@ceph.mgr.z.smithi154.stdout:Mar 16 11:42:39 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-z[38182]: 2024-03-16T11:42:39.500+0000 7fb6b8963200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-16T11:42:39.588 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:39 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-16T11:42:39.588 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:39 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:42:39.588 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:39 smithi029 ceph-mon[31975]: Deploying daemon mgr.y on smithi152 2024-03-16T11:42:39.627 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:39 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-16T11:42:39.627 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:39 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:42:39.627 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:39 smithi152 ceph-mon[36291]: Deploying daemon mgr.y on smithi152 2024-03-16T11:42:39.837 INFO:journalctl@ceph.mgr.z.smithi154.stdout:Mar 16 11:42:39 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-z[38182]: 2024-03-16T11:42:39.650+0000 7fb6b8963200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-16T11:42:39.837 INFO:journalctl@ceph.mgr.z.smithi154.stdout:Mar 16 11:42:39 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-z[38182]: 2024-03-16T11:42:39.751+0000 7fb6b8963200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-16T11:42:39.837 INFO:journalctl@ceph.mgr.z.smithi154.stdout:Mar 16 11:42:39 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-z[38182]: 2024-03-16T11:42:39.837+0000 7fb6b8963200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-16T11:42:40.367 INFO:journalctl@ceph.mgr.z.smithi154.stdout:Mar 16 11:42:40 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-z[38182]: 2024-03-16T11:42:40.117+0000 7fb6b8963200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-16T11:42:40.367 INFO:journalctl@ceph.mgr.z.smithi154.stdout:Mar 16 11:42:40 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-z[38182]: 2024-03-16T11:42:40.228+0000 7fb6b8963200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-16T11:42:40.367 INFO:journalctl@ceph.mgr.z.smithi154.stdout:Mar 16 11:42:40 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-z[38182]: 2024-03-16T11:42:40.315+0000 7fb6b8963200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-16T11:42:40.368 INFO:journalctl@ceph.mgr.y.smithi152.stdout:Mar 16 11:42:40 smithi152 systemd[1]: Starting Ceph mgr.y for d1887f28-e389-11ee-95c9-87774f69a715... 2024-03-16T11:42:40.499 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:40 smithi029 ceph-mon[31975]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T11:42:40.627 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:40 smithi152 ceph-mon[36291]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T11:42:40.627 INFO:journalctl@ceph.mgr.y.smithi152.stdout:Mar 16 11:42:40 smithi152 podman[37653]: 2024-03-16 11:42:40.488511777 +0000 UTC m=+0.114340600 container create eec9dd63927ae55b173ae886cdfe60f2852fe2cef0c8382946c155134dd686ec (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-y, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-16T11:42:40.627 INFO:journalctl@ceph.mgr.y.smithi152.stdout:Mar 16 11:42:40 smithi152 podman[37653]: 2024-03-16 11:42:40.391861423 +0000 UTC m=+0.017690262 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-16T11:42:40.731 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:42:40.732+0000 7fc16b383700 1 -- 172.21.15.29:0/3619085610 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc164102470 msgr2=0x7fc1641028d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:42:40.732 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:42:40.732+0000 7fc16b383700 1 --2- 172.21.15.29:0/3619085610 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc164102470 0x7fc1641028d0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fc1540099c0 tx=0x7fc154009cd0 comp rx=0 tx=0).stop 2024-03-16T11:42:40.732 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:42:40.732+0000 7fc16b383700 1 -- 172.21.15.29:0/3619085610 shutdown_connections 2024-03-16T11:42:40.732 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:42:40.732+0000 7fc16b383700 1 --2- 172.21.15.29:0/3619085610 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc164102e10 0x7fc1641096c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:42:40.732 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:42:40.732+0000 7fc16b383700 1 --2- 172.21.15.29:0/3619085610 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc164101270 0x7fc164101690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:42:40.732 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:42:40.732+0000 7fc16b383700 1 --2- 172.21.15.29:0/3619085610 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc164102470 0x7fc1641028d0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:42:40.732 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:42:40.732+0000 7fc16b383700 1 -- 172.21.15.29:0/3619085610 >> 172.21.15.29:0/3619085610 conn(0x7fc1640fc830 msgr2=0x7fc1640fec50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:42:40.732 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:42:40.732+0000 7fc16b383700 1 -- 172.21.15.29:0/3619085610 shutdown_connections 2024-03-16T11:42:40.732 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:42:40.732+0000 7fc16b383700 1 -- 172.21.15.29:0/3619085610 wait complete. 2024-03-16T11:42:40.734 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:42:40.733+0000 7fc16b383700 1 Processor -- start 2024-03-16T11:42:40.734 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:42:40.733+0000 7fc16b383700 1 -- start start 2024-03-16T11:42:40.735 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:42:40.733+0000 7fc16b383700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc164101270 0x7fc1641984f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:42:40.735 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:42:40.733+0000 7fc16b383700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc164102470 0x7fc164198a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:42:40.735 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:42:40.733+0000 7fc16b383700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc164102e10 0x7fc16419fab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:42:40.735 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:42:40.733+0000 7fc16b383700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fc16410cdc0 con 0x7fc164101270 2024-03-16T11:42:40.735 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:42:40.733+0000 7fc16b383700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fc16410cc40 con 0x7fc164102e10 2024-03-16T11:42:40.735 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:42:40.733+0000 7fc16b383700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fc16410cf40 con 0x7fc164102470 2024-03-16T11:42:40.735 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:42:40.733+0000 7fc16911f700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc164101270 0x7fc1641984f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:42:40.735 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:42:40.733+0000 7fc16911f700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc164101270 0x7fc1641984f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47164/0 (socket says 172.21.15.29:47164) 2024-03-16T11:42:40.735 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:42:40.733+0000 7fc16911f700 1 -- 172.21.15.29:0/2327442400 learned_addr learned my addr 172.21.15.29:0/2327442400 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:42:40.735 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:42:40.733+0000 7fc169920700 1 --2- 172.21.15.29:0/2327442400 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc164102e10 0x7fc16419fab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:42:40.735 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:42:40.733+0000 7fc16891e700 1 --2- 172.21.15.29:0/2327442400 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc164102470 0x7fc164198a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:42:40.735 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:42:40.733+0000 7fc16911f700 1 -- 172.21.15.29:0/2327442400 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc164102470 msgr2=0x7fc164198a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:42:40.735 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:42:40.733+0000 7fc16911f700 1 --2- 172.21.15.29:0/2327442400 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc164102470 0x7fc164198a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:42:40.735 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:42:40.733+0000 7fc16911f700 1 -- 172.21.15.29:0/2327442400 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc164102e10 msgr2=0x7fc16419fab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:42:40.735 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:42:40.733+0000 7fc16911f700 1 --2- 172.21.15.29:0/2327442400 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc164102e10 0x7fc16419fab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:42:40.736 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:42:40.733+0000 7fc16911f700 1 -- 172.21.15.29:0/2327442400 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1641a0150 con 0x7fc164101270 2024-03-16T11:42:40.736 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:42:40.734+0000 7fc16911f700 1 --2- 172.21.15.29:0/2327442400 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc164101270 0x7fc1641984f0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fc16000d9b0 tx=0x7fc16000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:42:40.736 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:42:40.734+0000 7fc15a7fc700 1 -- 172.21.15.29:0/2327442400 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc160017030 con 0x7fc164101270 2024-03-16T11:42:40.736 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:42:40.734+0000 7fc15a7fc700 1 -- 172.21.15.29:0/2327442400 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc160004500 con 0x7fc164101270 2024-03-16T11:42:40.736 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:42:40.734+0000 7fc16b383700 1 -- 172.21.15.29:0/2327442400 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc1641a0440 con 0x7fc164101270 2024-03-16T11:42:40.736 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:42:40.734+0000 7fc15a7fc700 1 -- 172.21.15.29:0/2327442400 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc160003970 con 0x7fc164101270 2024-03-16T11:42:40.736 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:42:40.734+0000 7fc16b383700 1 -- 172.21.15.29:0/2327442400 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc1641a0980 con 0x7fc164101270 2024-03-16T11:42:40.736 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:42:40.735+0000 7fc15a7fc700 1 -- 172.21.15.29:0/2327442400 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 13) ==== 46162+0+0 (secure 0 0 0) 0x7fc16001a070 con 0x7fc164101270 2024-03-16T11:42:40.736 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:42:40.735+0000 7fc16b383700 1 -- 172.21.15.29:0/2327442400 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc164068f90 con 0x7fc164101270 2024-03-16T11:42:40.736 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:42:40.735+0000 7fc15a7fc700 1 --2- 172.21.15.29:0/2327442400 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fc150039200 0x7fc15003b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:42:40.736 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:42:40.735+0000 7fc15a7fc700 1 -- 172.21.15.29:0/2327442400 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fc16004b9d0 con 0x7fc164101270 2024-03-16T11:42:40.736 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:42:40.735+0000 7fc16891e700 1 --2- 172.21.15.29:0/2327442400 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fc150039200 0x7fc15003b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:42:40.736 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:42:40.735+0000 7fc16891e700 1 --2- 172.21.15.29:0/2327442400 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fc150039200 0x7fc15003b6c0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fc154000c00 tx=0x7fc154005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:42:40.739 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:42:40.740+0000 7fc15a7fc700 1 -- 172.21.15.29:0/2327442400 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fc16001ce00 con 0x7fc164101270 2024-03-16T11:42:40.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:40 smithi154 ceph-mon[36320]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T11:42:40.795 INFO:journalctl@ceph.mgr.z.smithi154.stdout:Mar 16 11:42:40 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-z[38182]: 2024-03-16T11:42:40.413+0000 7fb6b8963200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-16T11:42:40.795 INFO:journalctl@ceph.mgr.z.smithi154.stdout:Mar 16 11:42:40 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-z[38182]: 2024-03-16T11:42:40.594+0000 7fb6b8963200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-16T11:42:40.969 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:42:40.970+0000 7fc16b383700 1 -- 172.21.15.29:0/2327442400 --> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi029:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7fc1640fea00 con 0x7fc150039200 2024-03-16T11:42:41.086 INFO:journalctl@ceph.mgr.y.smithi152.stdout:Mar 16 11:42:40 smithi152 podman[37653]: 2024-03-16 11:42:40.653955034 +0000 UTC m=+0.279783861 container init eec9dd63927ae55b173ae886cdfe60f2852fe2cef0c8382946c155134dd686ec (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-y, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-03-16T11:42:41.087 INFO:journalctl@ceph.mgr.y.smithi152.stdout:Mar 16 11:42:40 smithi152 podman[37653]: 2024-03-16 11:42:40.65676285 +0000 UTC m=+0.282591668 container start eec9dd63927ae55b173ae886cdfe60f2852fe2cef0c8382946c155134dd686ec (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-y, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307) 2024-03-16T11:42:41.087 INFO:journalctl@ceph.mgr.y.smithi152.stdout:Mar 16 11:42:40 smithi152 ceph-mgr[37672]: -- 172.21.15.152:0/370400462 <== mon.2 v2:172.21.15.152:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55804ec925a0 con 0x55804ec66c00 2024-03-16T11:42:41.087 INFO:journalctl@ceph.mgr.y.smithi152.stdout:Mar 16 11:42:40 smithi152 bash[37653]: eec9dd63927ae55b173ae886cdfe60f2852fe2cef0c8382946c155134dd686ec 2024-03-16T11:42:41.087 INFO:journalctl@ceph.mgr.y.smithi152.stdout:Mar 16 11:42:40 smithi152 systemd[1]: Started Ceph mgr.y for d1887f28-e389-11ee-95c9-87774f69a715. 2024-03-16T11:42:41.352 INFO:journalctl@ceph.mgr.y.smithi152.stdout:Mar 16 11:42:41 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-y[37668]: 2024-03-16T11:42:41.216+0000 7f99d79de200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-16T11:42:41.352 INFO:journalctl@ceph.mgr.y.smithi152.stdout:Mar 16 11:42:41 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-y[37668]: 2024-03-16T11:42:41.314+0000 7f99d79de200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-16T11:42:41.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:41 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:42:41.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:41 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:42:41.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:41 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:42:41.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:41 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:42:41.839 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:41 smithi029 ceph-mon[31975]: from='client.14229 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi029:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T11:42:41.839 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:41 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T11:42:41.839 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:41 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T11:42:41.839 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:41 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:42:41.839 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:41 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T11:42:41.839 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:41 smithi029 ceph-mon[31975]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T11:42:41.869 INFO:journalctl@ceph.mgr.z.smithi154.stdout:Mar 16 11:42:41 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-z[38182]: 2024-03-16T11:42:41.607+0000 7fb6b8963200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-16T11:42:41.869 INFO:journalctl@ceph.mgr.z.smithi154.stdout:Mar 16 11:42:41 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-z[38182]: 2024-03-16T11:42:41.748+0000 7fb6b8963200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-16T11:42:41.869 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:41 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:42:41.870 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:41 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:42:41.870 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:41 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:42:41.870 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:41 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:42:41.870 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:41 smithi154 ceph-mon[36320]: from='client.14229 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi029:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T11:42:41.870 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:41 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T11:42:41.870 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:41 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T11:42:41.870 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:41 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:42:41.870 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:41 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T11:42:41.870 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:41 smithi154 ceph-mon[36320]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T11:42:42.124 INFO:journalctl@ceph.mgr.y.smithi152.stdout:Mar 16 11:42:42 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-y[37668]: 2024-03-16T11:42:42.028+0000 7f99d79de200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-16T11:42:42.124 INFO:journalctl@ceph.mgr.y.smithi152.stdout:Mar 16 11:42:42 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-y[37668]: 2024-03-16T11:42:42.124+0000 7f99d79de200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-16T11:42:42.125 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:41 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:42:42.125 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:41 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:42:42.125 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:41 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:42:42.125 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:41 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:42:42.125 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:41 smithi152 ceph-mon[36291]: from='client.14229 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi029:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T11:42:42.125 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:41 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T11:42:42.125 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:41 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T11:42:42.125 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:41 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:42:42.125 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:41 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T11:42:42.125 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:41 smithi152 ceph-mon[36291]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T11:42:42.186 INFO:journalctl@ceph.mgr.z.smithi154.stdout:Mar 16 11:42:42 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-z[38182]: 2024-03-16T11:42:42.095+0000 7fb6b8963200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-16T11:42:42.186 INFO:journalctl@ceph.mgr.z.smithi154.stdout:Mar 16 11:42:42 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-z[38182]: 2024-03-16T11:42:42.186+0000 7fb6b8963200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-16T11:42:42.377 INFO:journalctl@ceph.mgr.y.smithi152.stdout:Mar 16 11:42:42 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-y[37668]: 2024-03-16T11:42:42.272+0000 7f99d79de200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-16T11:42:42.377 INFO:journalctl@ceph.mgr.y.smithi152.stdout:Mar 16 11:42:42 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-y[37668]: 2024-03-16T11:42:42.373+0000 7f99d79de200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-16T11:42:42.545 INFO:journalctl@ceph.mgr.z.smithi154.stdout:Mar 16 11:42:42 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-z[38182]: 2024-03-16T11:42:42.390+0000 7fb6b8963200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-16T11:42:42.731 INFO:journalctl@ceph.mgr.y.smithi152.stdout:Mar 16 11:42:42 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-y[37668]: 2024-03-16T11:42:42.463+0000 7f99d79de200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-16T11:42:42.731 INFO:journalctl@ceph.mgr.y.smithi152.stdout:Mar 16 11:42:42 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-y[37668]: 2024-03-16T11:42:42.732+0000 7f99d79de200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-16T11:42:42.856 INFO:journalctl@ceph.mgr.z.smithi154.stdout:Mar 16 11:42:42 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-z[38182]: 2024-03-16T11:42:42.575+0000 7fb6b8963200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-16T11:42:43.020 INFO:journalctl@ceph.mgr.y.smithi152.stdout:Mar 16 11:42:42 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-y[37668]: 2024-03-16T11:42:42.840+0000 7f99d79de200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-16T11:42:43.020 INFO:journalctl@ceph.mgr.y.smithi152.stdout:Mar 16 11:42:42 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-y[37668]: 2024-03-16T11:42:42.926+0000 7f99d79de200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-16T11:42:43.020 INFO:journalctl@ceph.mgr.y.smithi152.stdout:Mar 16 11:42:43 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-y[37668]: 2024-03-16T11:42:43.020+0000 7f99d79de200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-16T11:42:43.118 INFO:journalctl@ceph.mgr.z.smithi154.stdout:Mar 16 11:42:42 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-z[38182]: 2024-03-16T11:42:42.857+0000 7fb6b8963200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-16T11:42:43.118 INFO:journalctl@ceph.mgr.z.smithi154.stdout:Mar 16 11:42:43 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-z[38182]: 2024-03-16T11:42:43.118+0000 7fb6b8963200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-16T11:42:43.377 INFO:journalctl@ceph.mgr.y.smithi152.stdout:Mar 16 11:42:43 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-y[37668]: 2024-03-16T11:42:43.196+0000 7f99d79de200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-16T11:42:43.392 INFO:journalctl@ceph.mgr.z.smithi154.stdout:Mar 16 11:42:43 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-z[38182]: 2024-03-16T11:42:43.352+0000 7fb6b8963200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-16T11:42:43.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:43 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:42:43.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:43 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:42:43.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:43 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:42:43.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:43 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:42:43.796 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:43 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:42:43.796 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:43 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T11:42:43.796 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:43 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:42:43.796 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:43 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-16T11:42:43.796 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:43 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-16T11:42:43.796 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:43 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:42:43.796 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:43 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:42:43.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:43 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:42:43.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:43 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:42:43.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:43 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:42:43.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:43 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:42:43.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:43 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:42:43.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:43 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T11:42:43.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:43 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:42:43.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:43 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-16T11:42:43.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:43 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-16T11:42:43.839 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:43 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:42:43.839 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:43 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:42:43.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:43 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:42:43.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:43 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:42:43.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:43 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:42:43.878 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:43 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:42:43.878 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:43 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:42:43.878 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:43 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T11:42:43.878 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:43 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:42:43.878 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:43 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-16T11:42:43.878 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:43 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-16T11:42:43.878 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:43 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:42:43.878 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:43 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:42:44.545 INFO:journalctl@ceph.mgr.z.smithi154.stdout:Mar 16 11:42:44 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-z[38182]: 2024-03-16T11:42:44.147+0000 7fb6b8963200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-16T11:42:44.545 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:44 smithi154 ceph-mon[36320]: Reconfiguring mgr.x (unknown last config time)... 2024-03-16T11:42:44.545 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:44 smithi154 ceph-mon[36320]: Reconfiguring daemon mgr.x on smithi029 2024-03-16T11:42:44.546 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:44 smithi154 ceph-mon[36320]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T11:42:44.546 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:44 smithi154 ceph-mon[36320]: from='client.? 172.21.15.29:0/3697918714' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "60f5608f-a408-45ba-bdf6-5b53758f88fc"}]: dispatch 2024-03-16T11:42:44.546 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:44 smithi154 ceph-mon[36320]: from='client.? 172.21.15.29:0/3697918714' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "60f5608f-a408-45ba-bdf6-5b53758f88fc"}]': finished 2024-03-16T11:42:44.546 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:44 smithi154 ceph-mon[36320]: osdmap e5: 1 total, 0 up, 1 in 2024-03-16T11:42:44.546 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:44 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-16T11:42:44.546 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:44 smithi154 ceph-mon[36320]: from='client.? 172.21.15.29:0/1585092319' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T11:42:44.627 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:44 smithi152 ceph-mon[36291]: Reconfiguring mgr.x (unknown last config time)... 2024-03-16T11:42:44.627 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:44 smithi152 ceph-mon[36291]: Reconfiguring daemon mgr.x on smithi029 2024-03-16T11:42:44.627 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:44 smithi152 ceph-mon[36291]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T11:42:44.627 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:44 smithi152 ceph-mon[36291]: from='client.? 172.21.15.29:0/3697918714' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "60f5608f-a408-45ba-bdf6-5b53758f88fc"}]: dispatch 2024-03-16T11:42:44.627 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:44 smithi152 ceph-mon[36291]: from='client.? 172.21.15.29:0/3697918714' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "60f5608f-a408-45ba-bdf6-5b53758f88fc"}]': finished 2024-03-16T11:42:44.628 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:44 smithi152 ceph-mon[36291]: osdmap e5: 1 total, 0 up, 1 in 2024-03-16T11:42:44.628 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:44 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-16T11:42:44.628 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:44 smithi152 ceph-mon[36291]: from='client.? 172.21.15.29:0/1585092319' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T11:42:44.628 INFO:journalctl@ceph.mgr.y.smithi152.stdout:Mar 16 11:42:44 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-y[37668]: 2024-03-16T11:42:44.166+0000 7f99d79de200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-16T11:42:44.628 INFO:journalctl@ceph.mgr.y.smithi152.stdout:Mar 16 11:42:44 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-y[37668]: 2024-03-16T11:42:44.295+0000 7f99d79de200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-16T11:42:44.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:44 smithi029 ceph-mon[31975]: Reconfiguring mgr.x (unknown last config time)... 2024-03-16T11:42:44.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:44 smithi029 ceph-mon[31975]: Reconfiguring daemon mgr.x on smithi029 2024-03-16T11:42:44.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:44 smithi029 ceph-mon[31975]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T11:42:44.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:44 smithi029 ceph-mon[31975]: from='client.? 172.21.15.29:0/3697918714' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "60f5608f-a408-45ba-bdf6-5b53758f88fc"}]: dispatch 2024-03-16T11:42:44.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:44 smithi029 ceph-mon[31975]: from='client.? 172.21.15.29:0/3697918714' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "60f5608f-a408-45ba-bdf6-5b53758f88fc"}]': finished 2024-03-16T11:42:44.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:44 smithi029 ceph-mon[31975]: osdmap e5: 1 total, 0 up, 1 in 2024-03-16T11:42:44.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:44 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-16T11:42:44.839 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:44 smithi029 ceph-mon[31975]: from='client.? 172.21.15.29:0/1585092319' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T11:42:44.914 INFO:journalctl@ceph.mgr.y.smithi152.stdout:Mar 16 11:42:44 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-y[37668]: 2024-03-16T11:42:44.631+0000 7f99d79de200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-16T11:42:44.914 INFO:journalctl@ceph.mgr.y.smithi152.stdout:Mar 16 11:42:44 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-y[37668]: 2024-03-16T11:42:44.721+0000 7f99d79de200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-16T11:42:44.914 INFO:journalctl@ceph.mgr.y.smithi152.stdout:Mar 16 11:42:44 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-y[37668]: 2024-03-16T11:42:44.914+0000 7f99d79de200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-16T11:42:45.028 INFO:journalctl@ceph.mgr.z.smithi154.stdout:Mar 16 11:42:44 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-z[38182]: 2024-03-16T11:42:44.718+0000 7fb6b8963200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-16T11:42:45.029 INFO:journalctl@ceph.mgr.z.smithi154.stdout:Mar 16 11:42:44 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-z[38182]: 2024-03-16T11:42:44.814+0000 7fb6b8963200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-16T11:42:45.029 INFO:journalctl@ceph.mgr.z.smithi154.stdout:Mar 16 11:42:45 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-z[38182]: 2024-03-16T11:42:45.029+0000 7fb6b8963200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-16T11:42:45.377 INFO:journalctl@ceph.mgr.y.smithi152.stdout:Mar 16 11:42:45 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-y[37668]: 2024-03-16T11:42:45.097+0000 7f99d79de200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-16T11:42:45.404 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:42:45.404+0000 7fc15a7fc700 1 -- 172.21.15.29:0/2327442400 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mgrmap(e 14) ==== 91698+0+0 (secure 0 0 0) 0x7fc160004020 con 0x7fc164101270 2024-03-16T11:42:45.642 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:45 smithi152 ceph-mon[36291]: Standby manager daemon z started 2024-03-16T11:42:45.642 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:45 smithi152 ceph-mon[36291]: from='mgr.? 172.21.15.154:0/1651235088' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-16T11:42:45.643 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:45 smithi152 ceph-mon[36291]: from='mgr.? 172.21.15.154:0/1651235088' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-16T11:42:45.643 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:45 smithi152 ceph-mon[36291]: from='mgr.? 172.21.15.154:0/1651235088' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-16T11:42:45.643 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:45 smithi152 ceph-mon[36291]: from='mgr.? 172.21.15.154:0/1651235088' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-16T11:42:45.643 INFO:journalctl@ceph.mgr.y.smithi152.stdout:Mar 16 11:42:45 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-y[37668]: 2024-03-16T11:42:45.380+0000 7f99d79de200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-16T11:42:45.643 INFO:journalctl@ceph.mgr.y.smithi152.stdout:Mar 16 11:42:45 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-y[37668]: 2024-03-16T11:42:45.643+0000 7f99d79de200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-16T11:42:45.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:45 smithi154 ceph-mon[36320]: Standby manager daemon z started 2024-03-16T11:42:45.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:45 smithi154 ceph-mon[36320]: from='mgr.? 172.21.15.154:0/1651235088' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-16T11:42:45.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:45 smithi154 ceph-mon[36320]: from='mgr.? 172.21.15.154:0/1651235088' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-16T11:42:45.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:45 smithi154 ceph-mon[36320]: from='mgr.? 172.21.15.154:0/1651235088' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-16T11:42:45.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:45 smithi154 ceph-mon[36320]: from='mgr.? 172.21.15.154:0/1651235088' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-16T11:42:45.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:45 smithi029 ceph-mon[31975]: Standby manager daemon z started 2024-03-16T11:42:45.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:45 smithi029 ceph-mon[31975]: from='mgr.? 172.21.15.154:0/1651235088' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-16T11:42:45.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:45 smithi029 ceph-mon[31975]: from='mgr.? 172.21.15.154:0/1651235088' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-16T11:42:45.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:45 smithi029 ceph-mon[31975]: from='mgr.? 172.21.15.154:0/1651235088' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-16T11:42:45.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:45 smithi029 ceph-mon[31975]: from='mgr.? 172.21.15.154:0/1651235088' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-16T11:42:46.127 INFO:journalctl@ceph.mgr.y.smithi152.stdout:Mar 16 11:42:45 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-y[37668]: 2024-03-16T11:42:45.877+0000 7f99d79de200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-16T11:42:46.673 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:46 smithi152 ceph-mon[36291]: mgrmap e14: x(active, since 77s), standbys: z 2024-03-16T11:42:46.674 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:46 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-16T11:42:46.674 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:46 smithi152 ceph-mon[36291]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T11:42:46.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:46 smithi154 ceph-mon[36320]: mgrmap e14: x(active, since 77s), standbys: z 2024-03-16T11:42:46.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:46 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-16T11:42:46.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:46 smithi154 ceph-mon[36320]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T11:42:46.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:46 smithi029 ceph-mon[31975]: mgrmap e14: x(active, since 77s), standbys: z 2024-03-16T11:42:46.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:46 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-16T11:42:46.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:46 smithi029 ceph-mon[31975]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T11:42:47.127 INFO:journalctl@ceph.mgr.y.smithi152.stdout:Mar 16 11:42:46 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-y[37668]: 2024-03-16T11:42:46.674+0000 7f99d79de200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-16T11:42:47.554 INFO:journalctl@ceph.mgr.y.smithi152.stdout:Mar 16 11:42:47 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-y[37668]: 2024-03-16T11:42:47.242+0000 7f99d79de200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-16T11:42:47.555 INFO:journalctl@ceph.mgr.y.smithi152.stdout:Mar 16 11:42:47 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-y[37668]: 2024-03-16T11:42:47.339+0000 7f99d79de200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-16T11:42:47.555 INFO:journalctl@ceph.mgr.y.smithi152.stdout:Mar 16 11:42:47 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-y[37668]: 2024-03-16T11:42:47.555+0000 7f99d79de200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-16T11:42:47.626 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:42:47.626+0000 7fc15a7fc700 1 -- 172.21.15.29:0/2327442400 <== mon.0 v2:172.21.15.29:3300/0 8 ==== mgrmap(e 15) ==== 137234+0+0 (secure 0 0 0) 0x7fc160031ac0 con 0x7fc164101270 2024-03-16T11:42:47.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:47 smithi152 ceph-mon[36291]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T11:42:47.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:47 smithi152 ceph-mon[36291]: Standby manager daemon y started 2024-03-16T11:42:47.878 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:47 smithi152 ceph-mon[36291]: from='mgr.? 172.21.15.152:0/3452788581' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-16T11:42:47.878 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:47 smithi152 ceph-mon[36291]: from='mgr.? 172.21.15.152:0/3452788581' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-16T11:42:47.878 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:47 smithi152 ceph-mon[36291]: from='mgr.? 172.21.15.152:0/3452788581' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-16T11:42:47.878 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:47 smithi152 ceph-mon[36291]: from='mgr.? 172.21.15.152:0/3452788581' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-16T11:42:47.978 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:47 smithi029 ceph-mon[31975]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T11:42:47.978 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:47 smithi029 ceph-mon[31975]: Standby manager daemon y started 2024-03-16T11:42:47.978 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:47 smithi029 ceph-mon[31975]: from='mgr.? 172.21.15.152:0/3452788581' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-16T11:42:47.978 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:47 smithi029 ceph-mon[31975]: from='mgr.? 172.21.15.152:0/3452788581' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-16T11:42:47.978 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:47 smithi029 ceph-mon[31975]: from='mgr.? 172.21.15.152:0/3452788581' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-16T11:42:47.978 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:47 smithi029 ceph-mon[31975]: from='mgr.? 172.21.15.152:0/3452788581' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-16T11:42:48.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:47 smithi154 ceph-mon[36320]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T11:42:48.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:47 smithi154 ceph-mon[36320]: Standby manager daemon y started 2024-03-16T11:42:48.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:47 smithi154 ceph-mon[36320]: from='mgr.? 172.21.15.152:0/3452788581' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-16T11:42:48.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:47 smithi154 ceph-mon[36320]: from='mgr.? 172.21.15.152:0/3452788581' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-16T11:42:48.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:47 smithi154 ceph-mon[36320]: from='mgr.? 172.21.15.152:0/3452788581' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-16T11:42:48.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:47 smithi154 ceph-mon[36320]: from='mgr.? 172.21.15.152:0/3452788581' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-16T11:42:48.708 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:48 smithi029 ceph-mon[31975]: mgrmap e15: x(active, since 80s), standbys: z, y 2024-03-16T11:42:48.709 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:48 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-16T11:42:49.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:48 smithi154 ceph-mon[36320]: mgrmap e15: x(active, since 80s), standbys: z, y 2024-03-16T11:42:49.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:48 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-16T11:42:49.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:48 smithi152 ceph-mon[36291]: mgrmap e15: x(active, since 80s), standbys: z, y 2024-03-16T11:42:49.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:48 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-16T11:42:50.529 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:50 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:42:50.529 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:50 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:42:50.530 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:50 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T11:42:50.530 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:50 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:42:50.530 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:50 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T11:42:50.530 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:50 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:42:50.530 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:50 smithi029 ceph-mon[31975]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T11:42:50.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:50 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:42:50.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:50 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:42:50.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:50 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T11:42:50.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:50 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:42:50.796 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:50 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T11:42:50.796 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:50 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:42:50.796 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:50 smithi154 ceph-mon[36320]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T11:42:50.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:50 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:42:50.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:50 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:42:50.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:50 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T11:42:50.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:50 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:42:50.878 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:50 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T11:42:50.878 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:50 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:42:50.878 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:50 smithi152 ceph-mon[36291]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T11:42:52.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:51 smithi154 ceph-mon[36320]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T11:42:52.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:51 smithi029 ceph-mon[31975]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T11:42:52.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:51 smithi152 ceph-mon[36291]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T11:42:52.749 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:52 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-16T11:42:52.749 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:52 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:42:52.749 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:52 smithi029 ceph-mon[31975]: Deploying daemon osd.0 on smithi029 2024-03-16T11:42:52.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:52 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-16T11:42:52.878 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:52 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:42:52.878 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:52 smithi152 ceph-mon[36291]: Deploying daemon osd.0 on smithi029 2024-03-16T11:42:53.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:52 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-16T11:42:53.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:52 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:42:53.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:52 smithi154 ceph-mon[36320]: Deploying daemon osd.0 on smithi029 2024-03-16T11:42:53.836 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:53 smithi029 ceph-mon[31975]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T11:42:54.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:53 smithi154 ceph-mon[36320]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T11:42:54.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:53 smithi152 ceph-mon[36291]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T11:42:55.815 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:55 smithi029 ceph-mon[31975]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T11:42:56.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:55 smithi154 ceph-mon[36320]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T11:42:56.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:55 smithi152 ceph-mon[36291]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T11:42:58.414 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:58 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:42:58.414 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:58 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:42:58.414 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:58 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T11:42:58.414 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:58 smithi029 ceph-mon[31975]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T11:42:58.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:58 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:42:58.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:58 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:42:58.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:58 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T11:42:58.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:58 smithi154 ceph-mon[36320]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T11:42:58.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:58 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:42:58.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:58 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:42:58.878 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:58 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T11:42:58.878 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:58 smithi152 ceph-mon[36291]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T11:42:59.902 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:42:59 smithi029 ceph-mon[31975]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T11:43:00.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:42:59 smithi154 ceph-mon[36320]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T11:43:00.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:42:59 smithi152 ceph-mon[36291]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T11:43:00.593 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:43:00.594+0000 7fc15a7fc700 1 -- 172.21.15.29:0/2327442400 <== mgr.14152 v2:172.21.15.29:6800/4192064640 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fc1640fea00 con 0x7fc150039200 2024-03-16T11:43:00.594 INFO:teuthology.orchestra.run.smithi029.stdout:Created osd(s) 0 on host 'smithi029' 2024-03-16T11:43:00.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:43:00.596+0000 7fc16b383700 1 -- 172.21.15.29:0/2327442400 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fc150039200 msgr2=0x7fc15003b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:43:00.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:43:00.596+0000 7fc16b383700 1 --2- 172.21.15.29:0/2327442400 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fc150039200 0x7fc15003b6c0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fc154000c00 tx=0x7fc154005fb0 comp rx=0 tx=0).stop 2024-03-16T11:43:00.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:43:00.597+0000 7fc16b383700 1 -- 172.21.15.29:0/2327442400 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc164101270 msgr2=0x7fc1641984f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:43:00.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:43:00.597+0000 7fc16b383700 1 --2- 172.21.15.29:0/2327442400 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc164101270 0x7fc1641984f0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fc16000d9b0 tx=0x7fc16000dcc0 comp rx=0 tx=0).stop 2024-03-16T11:43:00.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:43:00.597+0000 7fc16b383700 1 -- 172.21.15.29:0/2327442400 shutdown_connections 2024-03-16T11:43:00.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:43:00.597+0000 7fc16b383700 1 --2- 172.21.15.29:0/2327442400 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc164102e10 0x7fc16419fab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:43:00.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:43:00.597+0000 7fc16b383700 1 --2- 172.21.15.29:0/2327442400 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc164101270 0x7fc1641984f0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:43:00.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:43:00.597+0000 7fc16b383700 1 --2- 172.21.15.29:0/2327442400 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fc150039200 0x7fc15003b6c0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:43:00.597 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:43:00.597+0000 7fc16b383700 1 --2- 172.21.15.29:0/2327442400 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc164102470 0x7fc164198a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:43:00.597 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:43:00.597+0000 7fc16b383700 1 -- 172.21.15.29:0/2327442400 >> 172.21.15.29:0/2327442400 conn(0x7fc1640fc830 msgr2=0x7fc1640fe2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:43:00.597 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:43:00.597+0000 7fc16b383700 1 -- 172.21.15.29:0/2327442400 shutdown_connections 2024-03-16T11:43:00.597 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:43:00.597+0000 7fc16b383700 1 -- 172.21.15.29:0/2327442400 wait complete. 2024-03-16T11:43:01.263 DEBUG:teuthology.orchestra.run.smithi029:osd.0> sudo journalctl -f -n 0 -u ceph-d1887f28-e389-11ee-95c9-87774f69a715@osd.0.service 2024-03-16T11:43:01.265 INFO:tasks.cephadm:Deploying osd.1 on smithi152 with /dev/vg_nvme/lv_4... 2024-03-16T11:43:01.266 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d1887f28-e389-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-16T11:43:01.511 INFO:teuthology.orchestra.run.smithi152.stderr:Inferring config /var/lib/ceph/d1887f28-e389-11ee-95c9-87774f69a715/mon.b/config 2024-03-16T11:43:01.836 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:43:01 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:43:01.837 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:43:01 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:43:01.837 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:43:01 smithi152 ceph-mon[36291]: from='osd.0 [v2:172.21.15.29:6802/2748947913,v1:172.21.15.29:6803/2748947913]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-16T11:43:01.837 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:43:01 smithi152 ceph-mon[36291]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T11:43:01.837 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:43:01 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:43:01.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:43:01 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:43:01.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:43:01 smithi029 ceph-mon[31975]: from='osd.0 [v2:172.21.15.29:6802/2748947913,v1:172.21.15.29:6803/2748947913]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-16T11:43:01.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:43:01 smithi029 ceph-mon[31975]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T11:43:02.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:43:01 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:43:02.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:43:01 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:43:02.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:43:01 smithi154 ceph-mon[36320]: from='osd.0 [v2:172.21.15.29:6802/2748947913,v1:172.21.15.29:6803/2748947913]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-16T11:43:02.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:43:01 smithi154 ceph-mon[36320]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T11:43:02.700 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:43:02 smithi152 ceph-mon[36291]: from='osd.0 [v2:172.21.15.29:6802/2748947913,v1:172.21.15.29:6803/2748947913]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-16T11:43:02.700 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:43:02 smithi152 ceph-mon[36291]: osdmap e6: 1 total, 0 up, 1 in 2024-03-16T11:43:02.700 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:43:02 smithi152 ceph-mon[36291]: from='osd.0 [v2:172.21.15.29:6802/2748947913,v1:172.21.15.29:6803/2748947913]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]: dispatch 2024-03-16T11:43:02.700 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:43:02 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-16T11:43:02.700 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:43:02 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:43:02.700 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:43:02 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:43:02.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:43:02 smithi029 ceph-mon[31975]: from='osd.0 [v2:172.21.15.29:6802/2748947913,v1:172.21.15.29:6803/2748947913]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-16T11:43:02.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:43:02 smithi029 ceph-mon[31975]: osdmap e6: 1 total, 0 up, 1 in 2024-03-16T11:43:02.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:43:02 smithi029 ceph-mon[31975]: from='osd.0 [v2:172.21.15.29:6802/2748947913,v1:172.21.15.29:6803/2748947913]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]: dispatch 2024-03-16T11:43:02.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:43:02 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-16T11:43:02.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:43:02 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:43:02.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:43:02 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:43:02.839 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 16 11:43:02 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-0[46776]: 2024-03-16T11:43:02.603+0000 7f49be694700 -1 osd.0 0 waiting for initial osdmap 2024-03-16T11:43:02.839 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 16 11:43:02 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-0[46776]: 2024-03-16T11:43:02.606+0000 7f49ba487700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-16T11:43:03.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:43:02 smithi154 ceph-mon[36320]: from='osd.0 [v2:172.21.15.29:6802/2748947913,v1:172.21.15.29:6803/2748947913]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-16T11:43:03.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:43:02 smithi154 ceph-mon[36320]: osdmap e6: 1 total, 0 up, 1 in 2024-03-16T11:43:03.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:43:02 smithi154 ceph-mon[36320]: from='osd.0 [v2:172.21.15.29:6802/2748947913,v1:172.21.15.29:6803/2748947913]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]: dispatch 2024-03-16T11:43:03.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:43:02 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-16T11:43:03.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:43:02 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:43:03.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:43:02 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:43:03.224 INFO:teuthology.orchestra.run.smithi152.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-16T11:43:03.224 INFO:teuthology.orchestra.run.smithi152.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-16T11:43:03.224 INFO:teuthology.orchestra.run.smithi152.stderr: stderr: 10+0 records in 2024-03-16T11:43:03.224 INFO:teuthology.orchestra.run.smithi152.stderr:10+0 records out 2024-03-16T11:43:03.224 INFO:teuthology.orchestra.run.smithi152.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0131412 s, 798 MB/s 2024-03-16T11:43:03.224 INFO:teuthology.orchestra.run.smithi152.stderr: stderr: 2024-03-16T11:43:03.224 INFO:teuthology.orchestra.run.smithi152.stderr:--> Zapping successful for: 2024-03-16T11:43:03.706 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:43:03 smithi029 ceph-mon[31975]: from='osd.0 [v2:172.21.15.29:6802/2748947913,v1:172.21.15.29:6803/2748947913]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]': finished 2024-03-16T11:43:03.707 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:43:03 smithi029 ceph-mon[31975]: osdmap e7: 1 total, 0 up, 1 in 2024-03-16T11:43:03.707 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:43:03 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-16T11:43:03.707 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:43:03 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-16T11:43:03.707 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:43:03 smithi029 ceph-mon[31975]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T11:43:03.789 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:43:03 smithi152 ceph-mon[36291]: from='osd.0 [v2:172.21.15.29:6802/2748947913,v1:172.21.15.29:6803/2748947913]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]': finished 2024-03-16T11:43:03.789 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:43:03 smithi152 ceph-mon[36291]: osdmap e7: 1 total, 0 up, 1 in 2024-03-16T11:43:03.789 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:43:03 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-16T11:43:03.789 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:43:03 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-16T11:43:03.789 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:43:03 smithi152 ceph-mon[36291]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T11:43:03.815 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d1887f28-e389-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi152:vg_nvme/lv_4 2024-03-16T11:43:04.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:43:03 smithi154 ceph-mon[36320]: from='osd.0 [v2:172.21.15.29:6802/2748947913,v1:172.21.15.29:6803/2748947913]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]': finished 2024-03-16T11:43:04.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:43:03 smithi154 ceph-mon[36320]: osdmap e7: 1 total, 0 up, 1 in 2024-03-16T11:43:04.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:43:03 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-16T11:43:04.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:43:03 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-16T11:43:04.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:43:03 smithi154 ceph-mon[36320]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T11:43:04.057 INFO:teuthology.orchestra.run.smithi152.stderr:Inferring config /var/lib/ceph/d1887f28-e389-11ee-95c9-87774f69a715/mon.b/config 2024-03-16T11:43:04.753 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:43:04 smithi152 ceph-mon[36291]: purged_snaps scrub starts 2024-03-16T11:43:04.753 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:43:04 smithi152 ceph-mon[36291]: purged_snaps scrub ok 2024-03-16T11:43:04.753 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:43:04 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-16T11:43:04.753 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:43:04 smithi152 ceph-mon[36291]: osd.0 [v2:172.21.15.29:6802/2748947913,v1:172.21.15.29:6803/2748947913] boot 2024-03-16T11:43:04.753 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:43:04 smithi152 ceph-mon[36291]: osdmap e8: 1 total, 1 up, 1 in 2024-03-16T11:43:04.753 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:43:04 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-16T11:43:04.830 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:43:04 smithi029 ceph-mon[31975]: purged_snaps scrub starts 2024-03-16T11:43:04.830 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:43:04 smithi029 ceph-mon[31975]: purged_snaps scrub ok 2024-03-16T11:43:04.830 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:43:04 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-16T11:43:04.830 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:43:04 smithi029 ceph-mon[31975]: osd.0 [v2:172.21.15.29:6802/2748947913,v1:172.21.15.29:6803/2748947913] boot 2024-03-16T11:43:04.830 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:43:04 smithi029 ceph-mon[31975]: osdmap e8: 1 total, 1 up, 1 in 2024-03-16T11:43:04.830 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:43:04 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-16T11:43:05.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:43:04 smithi154 ceph-mon[36320]: purged_snaps scrub starts 2024-03-16T11:43:05.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:43:04 smithi154 ceph-mon[36320]: purged_snaps scrub ok 2024-03-16T11:43:05.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:43:04 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-16T11:43:05.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:43:04 smithi154 ceph-mon[36320]: osd.0 [v2:172.21.15.29:6802/2748947913,v1:172.21.15.29:6803/2748947913] boot 2024-03-16T11:43:05.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:43:04 smithi154 ceph-mon[36320]: osdmap e8: 1 total, 1 up, 1 in 2024-03-16T11:43:05.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:43:04 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-16T11:43:05.536 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-16T11:43:05.534+0000 7f8576c82700 1 -- 172.21.15.152:0/3154685598 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8570102450 msgr2=0x7f85701028b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:43:05.536 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-16T11:43:05.534+0000 7f8576c82700 1 --2- 172.21.15.152:0/3154685598 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8570102450 0x7f85701028b0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f855c0099c0 tx=0x7f855c009cd0 comp rx=0 tx=0).stop 2024-03-16T11:43:05.536 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-16T11:43:05.534+0000 7f8576c82700 1 -- 172.21.15.152:0/3154685598 shutdown_connections 2024-03-16T11:43:05.536 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-16T11:43:05.534+0000 7f8576c82700 1 --2- 172.21.15.152:0/3154685598 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8570102df0 0x7f85701096a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:43:05.536 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-16T11:43:05.534+0000 7f8576c82700 1 --2- 172.21.15.152:0/3154685598 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8570101250 0x7f8570101670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:43:05.536 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-16T11:43:05.534+0000 7f8576c82700 1 --2- 172.21.15.152:0/3154685598 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8570102450 0x7f85701028b0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:43:05.536 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-16T11:43:05.534+0000 7f8576c82700 1 -- 172.21.15.152:0/3154685598 >> 172.21.15.152:0/3154685598 conn(0x7f85700fc7f0 msgr2=0x7f85700fec30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:43:05.536 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-16T11:43:05.534+0000 7f8576c82700 1 -- 172.21.15.152:0/3154685598 shutdown_connections 2024-03-16T11:43:05.536 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-16T11:43:05.534+0000 7f8576c82700 1 -- 172.21.15.152:0/3154685598 wait complete. 2024-03-16T11:43:05.537 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-16T11:43:05.535+0000 7f8576c82700 1 Processor -- start 2024-03-16T11:43:05.537 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-16T11:43:05.535+0000 7f8576c82700 1 -- start start 2024-03-16T11:43:05.537 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-16T11:43:05.535+0000 7f8576c82700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8570101250 0x7f8570197430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:43:05.537 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-16T11:43:05.535+0000 7f8576c82700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8570102450 0x7f8570197970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:43:05.537 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-16T11:43:05.535+0000 7f8576c82700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8570102df0 0x7f857019e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:43:05.537 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-16T11:43:05.535+0000 7f8576c82700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f857010ce20 con 0x7f8570102450 2024-03-16T11:43:05.537 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-16T11:43:05.535+0000 7f8576c82700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f857010cca0 con 0x7f8570101250 2024-03-16T11:43:05.537 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-16T11:43:05.535+0000 7f8576c82700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f857010cfa0 con 0x7f8570102df0 2024-03-16T11:43:05.537 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-16T11:43:05.535+0000 7f8574a1e700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8570101250 0x7f8570197430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:43:05.537 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-16T11:43:05.535+0000 7f8574a1e700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8570101250 0x7f8570197430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.152:34876/0 (socket says 172.21.15.152:34876) 2024-03-16T11:43:05.537 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-16T11:43:05.535+0000 7f8574a1e700 1 -- 172.21.15.152:0/262737243 learned_addr learned my addr 172.21.15.152:0/262737243 (peer_addr_for_me v2:172.21.15.152:0/0) 2024-03-16T11:43:05.537 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-16T11:43:05.535+0000 7f857521f700 1 --2- 172.21.15.152:0/262737243 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8570102df0 0x7f857019e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:43:05.537 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-16T11:43:05.535+0000 7f856ffff700 1 --2- 172.21.15.152:0/262737243 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8570102450 0x7f8570197970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:43:05.537 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-16T11:43:05.536+0000 7f8574a1e700 1 -- 172.21.15.152:0/262737243 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8570102df0 msgr2=0x7f857019e9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:43:05.538 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-16T11:43:05.536+0000 7f8574a1e700 1 --2- 172.21.15.152:0/262737243 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8570102df0 0x7f857019e9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:43:05.538 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-16T11:43:05.536+0000 7f8574a1e700 1 -- 172.21.15.152:0/262737243 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8570102450 msgr2=0x7f8570197970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:43:05.538 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-16T11:43:05.536+0000 7f8574a1e700 1 --2- 172.21.15.152:0/262737243 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8570102450 0x7f8570197970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:43:05.538 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-16T11:43:05.536+0000 7f8574a1e700 1 -- 172.21.15.152:0/262737243 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f857019f090 con 0x7f8570101250 2024-03-16T11:43:05.538 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-16T11:43:05.536+0000 7f856ffff700 1 --2- 172.21.15.152:0/262737243 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8570102450 0x7f8570197970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T11:43:05.538 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-16T11:43:05.536+0000 7f8574a1e700 1 --2- 172.21.15.152:0/262737243 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8570101250 0x7f8570197430 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f856400d980 tx=0x7f856400dc90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:43:05.538 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-16T11:43:05.536+0000 7f856dffb700 1 -- 172.21.15.152:0/262737243 <== mon.2 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8564016070 con 0x7f8570101250 2024-03-16T11:43:05.538 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-16T11:43:05.536+0000 7f856dffb700 1 -- 172.21.15.152:0/262737243 <== mon.2 v2:172.21.15.152:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f856400be30 con 0x7f8570101250 2024-03-16T11:43:05.538 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-16T11:43:05.536+0000 7f8576c82700 1 -- 172.21.15.152:0/262737243 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f857019f380 con 0x7f8570101250 2024-03-16T11:43:05.538 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-16T11:43:05.536+0000 7f856dffb700 1 -- 172.21.15.152:0/262737243 <== mon.2 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f856400f3c0 con 0x7f8570101250 2024-03-16T11:43:05.538 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-16T11:43:05.536+0000 7f8576c82700 1 -- 172.21.15.152:0/262737243 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f857019f910 con 0x7f8570101250 2024-03-16T11:43:05.539 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-16T11:43:05.537+0000 7f8576c82700 1 -- 172.21.15.152:0/262737243 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8570068f90 con 0x7f8570101250 2024-03-16T11:43:05.539 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-16T11:43:05.538+0000 7f856dffb700 1 -- 172.21.15.152:0/262737243 <== mon.2 v2:172.21.15.152:3300/0 4 ==== mgrmap(e 15) ==== 137234+0+0 (secure 0 0 0) 0x7f8564019070 con 0x7f8570101250 2024-03-16T11:43:05.539 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-16T11:43:05.538+0000 7f856dffb700 1 --2- 172.21.15.152:0/262737243 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f85600a2880 0x7f85600a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:43:05.540 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-16T11:43:05.539+0000 7f856dffb700 1 -- 172.21.15.152:0/262737243 <== mon.2 v2:172.21.15.152:3300/0 5 ==== osd_map(9..9 src has 1..9) ==== 1767+0+0 (secure 0 0 0) 0x7f85640ccae0 con 0x7f8570101250 2024-03-16T11:43:05.540 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-16T11:43:05.539+0000 7f856ffff700 1 --2- 172.21.15.152:0/262737243 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f85600a2880 0x7f85600a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:43:05.540 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-16T11:43:05.539+0000 7f856ffff700 1 --2- 172.21.15.152:0/262737243 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f85600a2880 0x7f85600a4d40 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f855c000c00 tx=0x7f855c003960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:43:05.542 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-16T11:43:05.543+0000 7f856dffb700 1 -- 172.21.15.152:0/262737243 <== mon.2 v2:172.21.15.152:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f8564020030 con 0x7f8570101250 2024-03-16T11:43:05.751 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-16T11:43:05.751+0000 7f8576c82700 1 -- 172.21.15.152:0/262737243 --> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi152:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f8570062710 con 0x7f85600a2880 2024-03-16T11:43:05.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:43:05 smithi152 ceph-mon[36291]: osdmap e9: 1 total, 1 up, 1 in 2024-03-16T11:43:05.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:43:05 smithi152 ceph-mon[36291]: Detected new or changed devices on smithi029 2024-03-16T11:43:05.878 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:43:05 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:43:05.878 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:43:05 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:43:05.878 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:43:05 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-16T11:43:05.878 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:43:05 smithi152 ceph-mon[36291]: Adjusting osd_memory_target on smithi029 to 17093M 2024-03-16T11:43:05.878 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:43:05 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:43:05.878 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:43:05 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:43:05.878 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:43:05 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T11:43:05.878 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:43:05 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:43:05.878 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:43:05 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T11:43:05.878 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:43:05 smithi152 ceph-mon[36291]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-16T11:43:06.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:43:05 smithi154 ceph-mon[36320]: osdmap e9: 1 total, 1 up, 1 in 2024-03-16T11:43:06.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:43:05 smithi154 ceph-mon[36320]: Detected new or changed devices on smithi029 2024-03-16T11:43:06.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:43:05 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:43:06.046 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:43:05 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:43:06.046 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:43:05 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-16T11:43:06.046 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:43:05 smithi154 ceph-mon[36320]: Adjusting osd_memory_target on smithi029 to 17093M 2024-03-16T11:43:06.046 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:43:05 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:43:06.046 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:43:05 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:43:06.046 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:43:05 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T11:43:06.046 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:43:05 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:43:06.046 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:43:05 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T11:43:06.046 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:43:05 smithi154 ceph-mon[36320]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-16T11:43:06.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:43:05 smithi029 ceph-mon[31975]: osdmap e9: 1 total, 1 up, 1 in 2024-03-16T11:43:06.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:43:05 smithi029 ceph-mon[31975]: Detected new or changed devices on smithi029 2024-03-16T11:43:06.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:43:05 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:43:06.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:43:05 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:43:06.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:43:05 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-16T11:43:06.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:43:05 smithi029 ceph-mon[31975]: Adjusting osd_memory_target on smithi029 to 17093M 2024-03-16T11:43:06.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:43:05 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:43:06.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:43:05 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:43:06.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:43:05 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T11:43:06.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:43:05 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:43:06.089 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:43:05 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T11:43:06.089 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:43:05 smithi029 ceph-mon[31975]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-16T11:43:06.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:43:06 smithi152 ceph-mon[36291]: from='client.24143 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi152:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T11:43:06.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:43:06 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T11:43:06.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:43:06 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T11:43:06.878 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:43:06 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:43:06.878 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:43:06 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:43:06.878 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:43:06 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T11:43:06.878 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:43:06 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:43:07.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:43:06 smithi154 ceph-mon[36320]: from='client.24143 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi152:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T11:43:07.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:43:06 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T11:43:07.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:43:06 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T11:43:07.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:43:06 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:43:07.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:43:06 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:43:07.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:43:06 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T11:43:07.046 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:43:06 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:43:07.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:43:06 smithi029 ceph-mon[31975]: from='client.24143 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi152:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T11:43:07.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:43:06 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T11:43:07.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:43:06 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T11:43:07.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:43:06 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:43:07.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:43:06 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:43:07.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:43:06 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T11:43:07.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:43:06 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:43:07.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:43:07 smithi152 ceph-mon[36291]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-16T11:43:07.946 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:43:07 smithi154 ceph-mon[36320]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-16T11:43:08.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:43:07 smithi029 ceph-mon[31975]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-16T11:43:08.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:43:08 smithi152 ceph-mon[36291]: from='client.? 172.21.15.152:0/3783828345' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e70b53d6-ee78-4e56-9a2c-263ae0501b43"}]: dispatch 2024-03-16T11:43:08.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:43:08 smithi152 ceph-mon[36291]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e70b53d6-ee78-4e56-9a2c-263ae0501b43"}]: dispatch 2024-03-16T11:43:08.878 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:43:08 smithi152 ceph-mon[36291]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e70b53d6-ee78-4e56-9a2c-263ae0501b43"}]': finished 2024-03-16T11:43:08.878 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:43:08 smithi152 ceph-mon[36291]: osdmap e10: 2 total, 1 up, 2 in 2024-03-16T11:43:08.878 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:43:08 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-16T11:43:09.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:43:08 smithi154 ceph-mon[36320]: from='client.? 172.21.15.152:0/3783828345' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e70b53d6-ee78-4e56-9a2c-263ae0501b43"}]: dispatch 2024-03-16T11:43:09.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:43:08 smithi154 ceph-mon[36320]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e70b53d6-ee78-4e56-9a2c-263ae0501b43"}]: dispatch 2024-03-16T11:43:09.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:43:08 smithi154 ceph-mon[36320]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e70b53d6-ee78-4e56-9a2c-263ae0501b43"}]': finished 2024-03-16T11:43:09.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:43:08 smithi154 ceph-mon[36320]: osdmap e10: 2 total, 1 up, 2 in 2024-03-16T11:43:09.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:43:08 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-16T11:43:09.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:43:08 smithi029 ceph-mon[31975]: from='client.? 172.21.15.152:0/3783828345' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e70b53d6-ee78-4e56-9a2c-263ae0501b43"}]: dispatch 2024-03-16T11:43:09.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:43:08 smithi029 ceph-mon[31975]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e70b53d6-ee78-4e56-9a2c-263ae0501b43"}]: dispatch 2024-03-16T11:43:09.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:43:08 smithi029 ceph-mon[31975]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e70b53d6-ee78-4e56-9a2c-263ae0501b43"}]': finished 2024-03-16T11:43:09.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:43:08 smithi029 ceph-mon[31975]: osdmap e10: 2 total, 1 up, 2 in 2024-03-16T11:43:09.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:43:08 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-16T11:43:09.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:43:09 smithi152 ceph-mon[36291]: from='client.? 172.21.15.152:0/1840884617' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T11:43:09.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:43:09 smithi152 ceph-mon[36291]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-16T11:43:10.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:43:09 smithi154 ceph-mon[36320]: from='client.? 172.21.15.152:0/1840884617' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T11:43:10.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:43:09 smithi154 ceph-mon[36320]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-16T11:43:10.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:43:09 smithi029 ceph-mon[31975]: from='client.? 172.21.15.152:0/1840884617' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T11:43:10.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:43:09 smithi029 ceph-mon[31975]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-16T11:43:11.942 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:43:11 smithi152 ceph-mon[36291]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-16T11:43:12.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:43:11 smithi154 ceph-mon[36320]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-16T11:43:12.087 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:43:11 smithi029 ceph-mon[31975]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-16T11:43:13.878 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:43:13 smithi152 ceph-mon[36291]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-16T11:43:14.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:43:13 smithi154 ceph-mon[36320]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-16T11:43:14.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:43:13 smithi029 ceph-mon[31975]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-16T11:43:15.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:43:15 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-16T11:43:15.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:43:15 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:43:15.545 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:43:15 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-16T11:43:15.545 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:43:15 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:43:15.588 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:43:15 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-16T11:43:15.588 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:43:15 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:43:16.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:43:16 smithi152 ceph-mon[36291]: Deploying daemon osd.1 on smithi152 2024-03-16T11:43:16.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:43:16 smithi152 ceph-mon[36291]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-16T11:43:16.545 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:43:16 smithi154 ceph-mon[36320]: Deploying daemon osd.1 on smithi152 2024-03-16T11:43:16.545 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:43:16 smithi154 ceph-mon[36320]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-16T11:43:16.588 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:43:16 smithi029 ceph-mon[31975]: Deploying daemon osd.1 on smithi152 2024-03-16T11:43:16.588 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:43:16 smithi029 ceph-mon[31975]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-16T11:43:17.939 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:43:17 smithi152 ceph-mon[36291]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-16T11:43:18.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:43:17 smithi154 ceph-mon[36320]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-16T11:43:18.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:43:17 smithi029 ceph-mon[31975]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-16T11:43:19.782 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:43:19 smithi152 ceph-mon[36291]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-16T11:43:20.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:43:19 smithi154 ceph-mon[36320]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-16T11:43:20.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:43:19 smithi029 ceph-mon[31975]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-16T11:43:21.627 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:43:21 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:43:21.628 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:43:21 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:43:21.628 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:43:21 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T11:43:21.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:43:21 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:43:21.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:43:21 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:43:21.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:43:21 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T11:43:21.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:43:21 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:43:21.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:43:21 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:43:21.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:43:21 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T11:43:22.627 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:43:22 smithi152 ceph-mon[36291]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-16T11:43:22.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:43:22 smithi154 ceph-mon[36320]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-16T11:43:22.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:43:22 smithi029 ceph-mon[31975]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-16T11:43:23.711 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:43:23 smithi152 ceph-mon[36291]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-16T11:43:23.765 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-16T11:43:23.766+0000 7f856dffb700 1 -- 172.21.15.152:0/262737243 <== mgr.14152 v2:172.21.15.29:6800/4192064640 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f8570062710 con 0x7f85600a2880 2024-03-16T11:43:23.766 INFO:teuthology.orchestra.run.smithi152.stdout:Created osd(s) 1 on host 'smithi152' 2024-03-16T11:43:23.768 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-16T11:43:23.769+0000 7f8576c82700 1 -- 172.21.15.152:0/262737243 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f85600a2880 msgr2=0x7f85600a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:43:23.768 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-16T11:43:23.769+0000 7f8576c82700 1 --2- 172.21.15.152:0/262737243 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f85600a2880 0x7f85600a4d40 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f855c000c00 tx=0x7f855c003960 comp rx=0 tx=0).stop 2024-03-16T11:43:23.768 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-16T11:43:23.769+0000 7f8576c82700 1 -- 172.21.15.152:0/262737243 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8570101250 msgr2=0x7f8570197430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:43:23.769 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-16T11:43:23.769+0000 7f8576c82700 1 --2- 172.21.15.152:0/262737243 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8570101250 0x7f8570197430 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f856400d980 tx=0x7f856400dc90 comp rx=0 tx=0).stop 2024-03-16T11:43:23.769 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-16T11:43:23.769+0000 7f8576c82700 1 -- 172.21.15.152:0/262737243 shutdown_connections 2024-03-16T11:43:23.769 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-16T11:43:23.769+0000 7f8576c82700 1 --2- 172.21.15.152:0/262737243 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f85600a2880 0x7f85600a4d40 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:43:23.769 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-16T11:43:23.769+0000 7f8576c82700 1 --2- 172.21.15.152:0/262737243 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8570102df0 0x7f857019e9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:43:23.769 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-16T11:43:23.769+0000 7f8576c82700 1 --2- 172.21.15.152:0/262737243 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8570101250 0x7f8570197430 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:43:23.769 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-16T11:43:23.769+0000 7f8576c82700 1 --2- 172.21.15.152:0/262737243 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8570102450 0x7f8570197970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:43:23.769 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-16T11:43:23.769+0000 7f8576c82700 1 -- 172.21.15.152:0/262737243 >> 172.21.15.152:0/262737243 conn(0x7f85700fc7f0 msgr2=0x7f8570105750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:43:23.769 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-16T11:43:23.769+0000 7f8576c82700 1 -- 172.21.15.152:0/262737243 shutdown_connections 2024-03-16T11:43:23.769 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-16T11:43:23.769+0000 7f8576c82700 1 -- 172.21.15.152:0/262737243 wait complete. 2024-03-16T11:43:24.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:43:23 smithi154 ceph-mon[36320]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-16T11:43:24.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:43:23 smithi029 ceph-mon[31975]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-16T11:43:24.408 DEBUG:teuthology.orchestra.run.smithi152:osd.1> sudo journalctl -f -n 0 -u ceph-d1887f28-e389-11ee-95c9-87774f69a715@osd.1.service 2024-03-16T11:43:24.410 INFO:tasks.cephadm:Deploying osd.2 on smithi154 with /dev/vg_nvme/lv_4... 2024-03-16T11:43:24.410 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d1887f28-e389-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-16T11:43:24.656 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/d1887f28-e389-11ee-95c9-87774f69a715/mon.c/config 2024-03-16T11:43:24.996 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:43:24 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:43:24.996 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:43:24 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:43:24.996 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:43:24 smithi154 ceph-mon[36320]: from='osd.1 [v2:172.21.15.152:6800/2339408422,v1:172.21.15.152:6801/2339408422]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-16T11:43:24.996 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:43:24 smithi154 ceph-mon[36320]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-16T11:43:24.996 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:43:24 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:43:24.996 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:43:24 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:43:25.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:43:24 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:43:25.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:43:24 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:43:25.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:43:24 smithi029 ceph-mon[31975]: from='osd.1 [v2:172.21.15.152:6800/2339408422,v1:172.21.15.152:6801/2339408422]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-16T11:43:25.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:43:24 smithi029 ceph-mon[31975]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-16T11:43:25.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:43:24 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:43:25.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:43:24 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:43:25.110 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:43:24 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:43:25.110 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:43:24 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:43:25.111 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:43:24 smithi152 ceph-mon[36291]: from='osd.1 [v2:172.21.15.152:6800/2339408422,v1:172.21.15.152:6801/2339408422]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-16T11:43:25.111 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:43:24 smithi152 ceph-mon[36291]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-16T11:43:25.111 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:43:24 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:43:25.111 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:43:24 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:43:25.898 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:43:25 smithi154 ceph-mon[36320]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-16T11:43:25.898 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:43:25 smithi154 ceph-mon[36320]: from='osd.1 [v2:172.21.15.152:6800/2339408422,v1:172.21.15.152:6801/2339408422]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]: dispatch 2024-03-16T11:43:25.898 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:43:25 smithi154 ceph-mon[36320]: osdmap e11: 2 total, 1 up, 2 in 2024-03-16T11:43:25.898 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:43:25 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-16T11:43:25.898 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:43:25 smithi154 ceph-mon[36320]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]: dispatch 2024-03-16T11:43:25.898 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:43:25 smithi154 ceph-mon[36320]: pgmap v60: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-16T11:43:26.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:43:25 smithi029 ceph-mon[31975]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-16T11:43:26.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:43:25 smithi029 ceph-mon[31975]: from='osd.1 [v2:172.21.15.152:6800/2339408422,v1:172.21.15.152:6801/2339408422]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]: dispatch 2024-03-16T11:43:26.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:43:25 smithi029 ceph-mon[31975]: osdmap e11: 2 total, 1 up, 2 in 2024-03-16T11:43:26.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:43:25 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-16T11:43:26.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:43:25 smithi029 ceph-mon[31975]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]: dispatch 2024-03-16T11:43:26.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:43:25 smithi029 ceph-mon[31975]: pgmap v60: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-16T11:43:26.106 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:43:25 smithi152 ceph-mon[36291]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-16T11:43:26.106 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:43:25 smithi152 ceph-mon[36291]: from='osd.1 [v2:172.21.15.152:6800/2339408422,v1:172.21.15.152:6801/2339408422]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]: dispatch 2024-03-16T11:43:26.106 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:43:25 smithi152 ceph-mon[36291]: osdmap e11: 2 total, 1 up, 2 in 2024-03-16T11:43:26.106 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:43:25 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-16T11:43:26.106 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:43:25 smithi152 ceph-mon[36291]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]: dispatch 2024-03-16T11:43:26.106 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:43:25 smithi152 ceph-mon[36291]: pgmap v60: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-16T11:43:26.377 INFO:journalctl@ceph.osd.1.smithi152.stdout:Mar 16 11:43:26 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-1[40893]: 2024-03-16T11:43:26.213+0000 7f4fe9c38700 -1 osd.1 0 waiting for initial osdmap 2024-03-16T11:43:26.377 INFO:journalctl@ceph.osd.1.smithi152.stdout:Mar 16 11:43:26 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-1[40893]: 2024-03-16T11:43:26.218+0000 7f4fe1a23700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-16T11:43:26.474 INFO:teuthology.orchestra.run.smithi154.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-16T11:43:26.474 INFO:teuthology.orchestra.run.smithi154.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-16T11:43:26.474 INFO:teuthology.orchestra.run.smithi154.stderr: stderr: 10+0 records in 2024-03-16T11:43:26.474 INFO:teuthology.orchestra.run.smithi154.stderr:10+0 records out 2024-03-16T11:43:26.475 INFO:teuthology.orchestra.run.smithi154.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.011265 s, 931 MB/s 2024-03-16T11:43:26.475 INFO:teuthology.orchestra.run.smithi154.stderr:--> Zapping successful for: 2024-03-16T11:43:27.011 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:43:26 smithi154 ceph-mon[36320]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]': finished 2024-03-16T11:43:27.012 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:43:26 smithi154 ceph-mon[36320]: osdmap e12: 2 total, 1 up, 2 in 2024-03-16T11:43:27.012 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:43:26 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-16T11:43:27.012 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:43:26 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-16T11:43:27.012 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:43:26 smithi154 ceph-mon[36320]: from='osd.1 ' entity='osd.1' 2024-03-16T11:43:27.039 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d1887f28-e389-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi154:vg_nvme/lv_4 2024-03-16T11:43:27.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:43:26 smithi029 ceph-mon[31975]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]': finished 2024-03-16T11:43:27.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:43:26 smithi029 ceph-mon[31975]: osdmap e12: 2 total, 1 up, 2 in 2024-03-16T11:43:27.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:43:26 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-16T11:43:27.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:43:26 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-16T11:43:27.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:43:26 smithi029 ceph-mon[31975]: from='osd.1 ' entity='osd.1' 2024-03-16T11:43:27.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:43:26 smithi152 ceph-mon[36291]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]': finished 2024-03-16T11:43:27.128 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:43:26 smithi152 ceph-mon[36291]: osdmap e12: 2 total, 1 up, 2 in 2024-03-16T11:43:27.128 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:43:26 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-16T11:43:27.128 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:43:26 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-16T11:43:27.128 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:43:26 smithi152 ceph-mon[36291]: from='osd.1 ' entity='osd.1' 2024-03-16T11:43:27.292 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/d1887f28-e389-11ee-95c9-87774f69a715/mon.c/config 2024-03-16T11:43:27.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:43:27 smithi152 ceph-mon[36291]: purged_snaps scrub starts 2024-03-16T11:43:27.878 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:43:27 smithi152 ceph-mon[36291]: purged_snaps scrub ok 2024-03-16T11:43:27.878 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:43:27 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-16T11:43:27.878 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:43:27 smithi152 ceph-mon[36291]: osd.1 [v2:172.21.15.152:6800/2339408422,v1:172.21.15.152:6801/2339408422] boot 2024-03-16T11:43:27.878 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:43:27 smithi152 ceph-mon[36291]: osdmap e13: 2 total, 2 up, 2 in 2024-03-16T11:43:27.878 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:43:27 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-16T11:43:27.878 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:43:27 smithi152 ceph-mon[36291]: pgmap v63: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-16T11:43:28.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:43:27 smithi154 ceph-mon[36320]: purged_snaps scrub starts 2024-03-16T11:43:28.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:43:27 smithi154 ceph-mon[36320]: purged_snaps scrub ok 2024-03-16T11:43:28.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:43:27 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-16T11:43:28.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:43:27 smithi154 ceph-mon[36320]: osd.1 [v2:172.21.15.152:6800/2339408422,v1:172.21.15.152:6801/2339408422] boot 2024-03-16T11:43:28.046 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:43:27 smithi154 ceph-mon[36320]: osdmap e13: 2 total, 2 up, 2 in 2024-03-16T11:43:28.046 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:43:27 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-16T11:43:28.046 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:43:27 smithi154 ceph-mon[36320]: pgmap v63: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-16T11:43:28.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:43:27 smithi029 ceph-mon[31975]: purged_snaps scrub starts 2024-03-16T11:43:28.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:43:27 smithi029 ceph-mon[31975]: purged_snaps scrub ok 2024-03-16T11:43:28.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:43:27 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-16T11:43:28.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:43:27 smithi029 ceph-mon[31975]: osd.1 [v2:172.21.15.152:6800/2339408422,v1:172.21.15.152:6801/2339408422] boot 2024-03-16T11:43:28.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:43:27 smithi029 ceph-mon[31975]: osdmap e13: 2 total, 2 up, 2 in 2024-03-16T11:43:28.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:43:27 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-16T11:43:28.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:43:27 smithi029 ceph-mon[31975]: pgmap v63: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-16T11:43:28.843 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:43:28.841+0000 7f9eac7a9700 1 -- 172.21.15.154:0/4092420484 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9ea4102300 msgr2=0x7f9ea4102720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:43:28.843 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:43:28.841+0000 7f9eac7a9700 1 --2- 172.21.15.154:0/4092420484 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9ea4102300 0x7f9ea4102720 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f9e9400b0a0 tx=0x7f9e9400b3b0 comp rx=0 tx=0).stop 2024-03-16T11:43:28.843 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:43:28.841+0000 7f9eac7a9700 1 -- 172.21.15.154:0/4092420484 shutdown_connections 2024-03-16T11:43:28.843 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:43:28.841+0000 7f9eac7a9700 1 --2- 172.21.15.154:0/4092420484 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9ea4103ea0 0x7f9ea410a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:43:28.843 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:43:28.841+0000 7f9eac7a9700 1 --2- 172.21.15.154:0/4092420484 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9ea4102300 0x7f9ea4102720 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:43:28.843 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:43:28.841+0000 7f9eac7a9700 1 --2- 172.21.15.154:0/4092420484 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9ea4103500 0x7f9ea4103960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:43:28.843 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:43:28.841+0000 7f9eac7a9700 1 -- 172.21.15.154:0/4092420484 >> 172.21.15.154:0/4092420484 conn(0x7f9ea40fd8e0 msgr2=0x7f9ea40ffd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:43:28.843 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:43:28.841+0000 7f9eac7a9700 1 -- 172.21.15.154:0/4092420484 shutdown_connections 2024-03-16T11:43:28.843 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:43:28.841+0000 7f9eac7a9700 1 -- 172.21.15.154:0/4092420484 wait complete. 2024-03-16T11:43:28.843 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:43:28.842+0000 7f9eac7a9700 1 Processor -- start 2024-03-16T11:43:28.843 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:43:28.842+0000 7f9eac7a9700 1 -- start start 2024-03-16T11:43:28.843 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:43:28.842+0000 7f9eac7a9700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9ea4102300 0x7f9ea4198610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:43:28.844 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:43:28.842+0000 7f9eac7a9700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9ea4103500 0x7f9ea4198b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:43:28.844 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:43:28.842+0000 7f9eac7a9700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9ea4103ea0 0x7f9ea419fbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:43:28.844 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:43:28.842+0000 7f9eac7a9700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f9ea410df90 con 0x7f9ea4103500 2024-03-16T11:43:28.844 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:43:28.842+0000 7f9eac7a9700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f9ea410de10 con 0x7f9ea4102300 2024-03-16T11:43:28.844 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:43:28.842+0000 7f9eac7a9700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f9ea410e110 con 0x7f9ea4103ea0 2024-03-16T11:43:28.844 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:43:28.842+0000 7f9eaad46700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9ea4103ea0 0x7f9ea419fbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:43:28.844 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:43:28.842+0000 7f9eaad46700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9ea4103ea0 0x7f9ea419fbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.154:41766/0 (socket says 172.21.15.154:41766) 2024-03-16T11:43:28.844 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:43:28.842+0000 7f9eaad46700 1 -- 172.21.15.154:0/2225424398 learned_addr learned my addr 172.21.15.154:0/2225424398 (peer_addr_for_me v2:172.21.15.154:0/0) 2024-03-16T11:43:28.844 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:43:28.842+0000 7f9ea9d44700 1 --2- 172.21.15.154:0/2225424398 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9ea4103500 0x7f9ea4198b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:43:28.844 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:43:28.842+0000 7f9eaa545700 1 --2- 172.21.15.154:0/2225424398 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9ea4102300 0x7f9ea4198610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:43:28.844 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:43:28.842+0000 7f9eaad46700 1 -- 172.21.15.154:0/2225424398 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9ea4102300 msgr2=0x7f9ea4198610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:43:28.844 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:43:28.842+0000 7f9eaad46700 1 --2- 172.21.15.154:0/2225424398 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9ea4102300 0x7f9ea4198610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:43:28.844 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:43:28.842+0000 7f9eaad46700 1 -- 172.21.15.154:0/2225424398 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9ea4103500 msgr2=0x7f9ea4198b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:43:28.844 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:43:28.842+0000 7f9eaad46700 1 --2- 172.21.15.154:0/2225424398 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9ea4103500 0x7f9ea4198b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:43:28.845 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:43:28.842+0000 7f9eaad46700 1 -- 172.21.15.154:0/2225424398 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ea41a0270 con 0x7f9ea4103ea0 2024-03-16T11:43:28.845 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:43:28.842+0000 7f9ea9d44700 1 --2- 172.21.15.154:0/2225424398 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9ea4103500 0x7f9ea4198b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-16T11:43:28.845 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:43:28.843+0000 7f9eaad46700 1 --2- 172.21.15.154:0/2225424398 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9ea4103ea0 0x7f9ea419fbd0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f9e9c00d9b0 tx=0x7f9e9c00dd70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:43:28.845 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:43:28.843+0000 7f9ea37fe700 1 -- 172.21.15.154:0/2225424398 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e9c017070 con 0x7f9ea4103ea0 2024-03-16T11:43:28.845 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:43:28.843+0000 7f9ea37fe700 1 -- 172.21.15.154:0/2225424398 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9e9c009e20 con 0x7f9ea4103ea0 2024-03-16T11:43:28.845 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:43:28.843+0000 7f9eac7a9700 1 -- 172.21.15.154:0/2225424398 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9ea41a0560 con 0x7f9ea4103ea0 2024-03-16T11:43:28.845 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:43:28.843+0000 7f9ea37fe700 1 -- 172.21.15.154:0/2225424398 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e9c005410 con 0x7f9ea4103ea0 2024-03-16T11:43:28.845 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:43:28.843+0000 7f9eac7a9700 1 -- 172.21.15.154:0/2225424398 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9ea41a0b20 con 0x7f9ea4103ea0 2024-03-16T11:43:28.846 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:43:28.844+0000 7f9eac7a9700 1 -- 172.21.15.154:0/2225424398 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9ea4068f90 con 0x7f9ea4103ea0 2024-03-16T11:43:28.846 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:43:28.845+0000 7f9ea37fe700 1 -- 172.21.15.154:0/2225424398 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 15) ==== 137234+0+0 (secure 0 0 0) 0x7f9e9c0055b0 con 0x7f9ea4103ea0 2024-03-16T11:43:28.846 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:43:28.845+0000 7f9ea37fe700 1 --2- 172.21.15.154:0/2225424398 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f9e8c0a2800 0x7f9e8c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:43:28.846 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:43:28.845+0000 7f9ea37fe700 1 -- 172.21.15.154:0/2225424398 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(14..14 src has 1..14) ==== 2309+0+0 (secure 0 0 0) 0x7f9e9c0cd050 con 0x7f9ea4103ea0 2024-03-16T11:43:28.846 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:43:28.846+0000 7f9eaa545700 1 --2- 172.21.15.154:0/2225424398 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f9e8c0a2800 0x7f9e8c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:43:28.846 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:43:28.846+0000 7f9eaa545700 1 --2- 172.21.15.154:0/2225424398 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f9e8c0a2800 0x7f9e8c0a4cc0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f9e9400b830 tx=0x7f9e9400bdd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:43:28.849 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:43:28.851+0000 7f9ea37fe700 1 -- 172.21.15.154:0/2225424398 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f9e9c020020 con 0x7f9ea4103ea0 2024-03-16T11:43:29.057 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:43:29.057+0000 7f9eac7a9700 1 -- 172.21.15.154:0/2225424398 --> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi154:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f9ea4062710 con 0x7f9e8c0a2800 2024-03-16T11:43:29.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:43:28 smithi154 ceph-mon[36320]: Detected new or changed devices on smithi152 2024-03-16T11:43:29.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:43:28 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:43:29.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:43:28 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:43:29.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:43:28 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-16T11:43:29.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:43:28 smithi154 ceph-mon[36320]: Adjusting osd_memory_target on smithi152 to 17080M 2024-03-16T11:43:29.296 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:43:28 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:43:29.296 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:43:28 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:43:29.296 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:43:28 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T11:43:29.296 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:43:28 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:43:29.296 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:43:28 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T11:43:29.296 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:43:28 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:43:29.296 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:43:28 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T11:43:29.296 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:43:28 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:43:29.296 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:43:28 smithi154 ceph-mon[36320]: osdmap e14: 2 total, 2 up, 2 in 2024-03-16T11:43:29.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:43:28 smithi029 ceph-mon[31975]: Detected new or changed devices on smithi152 2024-03-16T11:43:29.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:43:28 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:43:29.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:43:28 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:43:29.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:43:28 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-16T11:43:29.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:43:28 smithi029 ceph-mon[31975]: Adjusting osd_memory_target on smithi152 to 17080M 2024-03-16T11:43:29.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:43:28 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:43:29.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:43:28 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:43:29.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:43:28 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T11:43:29.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:43:28 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:43:29.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:43:28 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T11:43:29.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:43:28 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:43:29.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:43:28 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T11:43:29.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:43:28 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:43:29.339 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:43:28 smithi029 ceph-mon[31975]: osdmap e14: 2 total, 2 up, 2 in 2024-03-16T11:43:29.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:43:28 smithi152 ceph-mon[36291]: Detected new or changed devices on smithi152 2024-03-16T11:43:29.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:43:28 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:43:29.378 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:43:28 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:43:29.378 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:43:28 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-16T11:43:29.378 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:43:28 smithi152 ceph-mon[36291]: Adjusting osd_memory_target on smithi152 to 17080M 2024-03-16T11:43:29.378 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:43:28 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:43:29.378 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:43:28 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:43:29.378 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:43:28 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T11:43:29.378 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:43:28 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:43:29.378 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:43:28 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T11:43:29.378 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:43:28 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:43:29.379 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:43:28 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T11:43:29.379 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:43:28 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:43:29.379 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:43:28 smithi152 ceph-mon[36291]: osdmap e14: 2 total, 2 up, 2 in 2024-03-16T11:43:30.285 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:43:29 smithi154 ceph-mon[36320]: from='client.24172 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi154:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T11:43:30.285 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:43:29 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T11:43:30.285 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:43:29 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T11:43:30.285 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:43:29 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:43:30.285 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:43:29 smithi154 ceph-mon[36320]: pgmap v65: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-16T11:43:30.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:43:29 smithi029 ceph-mon[31975]: from='client.24172 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi154:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T11:43:30.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:43:29 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T11:43:30.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:43:29 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T11:43:30.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:43:29 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:43:30.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:43:29 smithi029 ceph-mon[31975]: pgmap v65: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-16T11:43:30.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:43:29 smithi152 ceph-mon[36291]: from='client.24172 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi154:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T11:43:30.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:43:29 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T11:43:30.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:43:29 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T11:43:30.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:43:29 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:43:30.378 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:43:29 smithi152 ceph-mon[36291]: pgmap v65: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-16T11:43:31.962 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:43:31 smithi154 ceph-mon[36320]: pgmap v66: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-16T11:43:31.962 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:43:31 smithi154 ceph-mon[36320]: from='client.? 172.21.15.154:0/4164976063' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ff39389a-dfac-44b5-80ff-36abca316e78"}]: dispatch 2024-03-16T11:43:31.962 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:43:31 smithi154 ceph-mon[36320]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ff39389a-dfac-44b5-80ff-36abca316e78"}]: dispatch 2024-03-16T11:43:31.962 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:43:31 smithi154 ceph-mon[36320]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ff39389a-dfac-44b5-80ff-36abca316e78"}]': finished 2024-03-16T11:43:31.962 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:43:31 smithi154 ceph-mon[36320]: osdmap e15: 3 total, 2 up, 3 in 2024-03-16T11:43:31.962 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:43:31 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-16T11:43:32.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:43:31 smithi029 ceph-mon[31975]: pgmap v66: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-16T11:43:32.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:43:31 smithi029 ceph-mon[31975]: from='client.? 172.21.15.154:0/4164976063' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ff39389a-dfac-44b5-80ff-36abca316e78"}]: dispatch 2024-03-16T11:43:32.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:43:31 smithi029 ceph-mon[31975]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ff39389a-dfac-44b5-80ff-36abca316e78"}]: dispatch 2024-03-16T11:43:32.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:43:31 smithi029 ceph-mon[31975]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ff39389a-dfac-44b5-80ff-36abca316e78"}]': finished 2024-03-16T11:43:32.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:43:31 smithi029 ceph-mon[31975]: osdmap e15: 3 total, 2 up, 3 in 2024-03-16T11:43:32.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:43:31 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-16T11:43:32.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:43:31 smithi152 ceph-mon[36291]: pgmap v66: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-16T11:43:32.128 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:43:31 smithi152 ceph-mon[36291]: from='client.? 172.21.15.154:0/4164976063' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ff39389a-dfac-44b5-80ff-36abca316e78"}]: dispatch 2024-03-16T11:43:32.128 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:43:31 smithi152 ceph-mon[36291]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ff39389a-dfac-44b5-80ff-36abca316e78"}]: dispatch 2024-03-16T11:43:32.128 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:43:31 smithi152 ceph-mon[36291]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ff39389a-dfac-44b5-80ff-36abca316e78"}]': finished 2024-03-16T11:43:32.128 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:43:31 smithi152 ceph-mon[36291]: osdmap e15: 3 total, 2 up, 3 in 2024-03-16T11:43:32.128 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:43:31 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-16T11:43:33.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:43:32 smithi154 ceph-mon[36320]: from='client.? 172.21.15.154:0/40858994' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T11:43:33.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:43:32 smithi029 ceph-mon[31975]: from='client.? 172.21.15.154:0/40858994' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T11:43:33.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:43:32 smithi152 ceph-mon[36291]: from='client.? 172.21.15.154:0/40858994' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T11:43:34.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:43:33 smithi154 ceph-mon[36320]: pgmap v68: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-16T11:43:34.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:43:33 smithi029 ceph-mon[31975]: pgmap v68: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-16T11:43:34.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:43:33 smithi152 ceph-mon[36291]: pgmap v68: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-16T11:43:35.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:43:35 smithi154 ceph-mon[36320]: pgmap v69: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-16T11:43:36.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:43:35 smithi029 ceph-mon[31975]: pgmap v69: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-16T11:43:36.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:43:35 smithi152 ceph-mon[36291]: pgmap v69: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-16T11:43:38.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:43:37 smithi154 ceph-mon[36320]: pgmap v70: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-16T11:43:38.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:43:37 smithi029 ceph-mon[31975]: pgmap v70: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-16T11:43:38.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:43:37 smithi152 ceph-mon[36291]: pgmap v70: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-16T11:43:38.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:43:38 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-16T11:43:38.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:43:38 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:43:39.028 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:43:38 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-16T11:43:39.029 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:43:38 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:43:39.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:43:38 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-16T11:43:39.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:43:38 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:43:39.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:43:39 smithi154 ceph-mon[36320]: Deploying daemon osd.2 on smithi154 2024-03-16T11:43:39.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:43:39 smithi154 ceph-mon[36320]: pgmap v71: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-16T11:43:40.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:43:39 smithi029 ceph-mon[31975]: Deploying daemon osd.2 on smithi154 2024-03-16T11:43:40.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:43:39 smithi029 ceph-mon[31975]: pgmap v71: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-16T11:43:40.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:43:39 smithi152 ceph-mon[36291]: Deploying daemon osd.2 on smithi154 2024-03-16T11:43:40.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:43:39 smithi152 ceph-mon[36291]: pgmap v71: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-16T11:43:41.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:43:41 smithi154 ceph-mon[36320]: pgmap v72: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-16T11:43:42.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:43:41 smithi029 ceph-mon[31975]: pgmap v72: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-16T11:43:42.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:43:41 smithi152 ceph-mon[36291]: pgmap v72: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-16T11:43:43.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:43:43 smithi154 ceph-mon[36320]: pgmap v73: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-16T11:43:44.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:43:43 smithi029 ceph-mon[31975]: pgmap v73: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-16T11:43:44.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:43:43 smithi152 ceph-mon[36291]: pgmap v73: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-16T11:43:45.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:43:45 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:43:45.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:43:45 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:43:45.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:43:45 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T11:43:45.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:43:45 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:43:45.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:43:45 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:43:45.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:43:45 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T11:43:45.545 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:43:45 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:43:45.545 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:43:45 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:43:45.545 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:43:45 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T11:43:46.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:43:46 smithi154 ceph-mon[36320]: pgmap v74: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-16T11:43:46.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:43:46 smithi029 ceph-mon[31975]: pgmap v74: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-16T11:43:46.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:43:46 smithi152 ceph-mon[36291]: pgmap v74: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-16T11:43:47.424 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:43:47.424+0000 7f9ea37fe700 1 -- 172.21.15.154:0/2225424398 <== mgr.14152 v2:172.21.15.29:6800/4192064640 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f9ea4062710 con 0x7f9e8c0a2800 2024-03-16T11:43:47.424 INFO:teuthology.orchestra.run.smithi154.stdout:Created osd(s) 2 on host 'smithi154' 2024-03-16T11:43:47.426 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:43:47.427+0000 7f9eac7a9700 1 -- 172.21.15.154:0/2225424398 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f9e8c0a2800 msgr2=0x7f9e8c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:43:47.426 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:43:47.427+0000 7f9eac7a9700 1 --2- 172.21.15.154:0/2225424398 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f9e8c0a2800 0x7f9e8c0a4cc0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f9e9400b830 tx=0x7f9e9400bdd0 comp rx=0 tx=0).stop 2024-03-16T11:43:47.427 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:43:47.427+0000 7f9eac7a9700 1 -- 172.21.15.154:0/2225424398 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9ea4103ea0 msgr2=0x7f9ea419fbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:43:47.427 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:43:47.427+0000 7f9eac7a9700 1 --2- 172.21.15.154:0/2225424398 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9ea4103ea0 0x7f9ea419fbd0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f9e9c00d9b0 tx=0x7f9e9c00dd70 comp rx=0 tx=0).stop 2024-03-16T11:43:47.427 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:43:47.427+0000 7f9eac7a9700 1 -- 172.21.15.154:0/2225424398 shutdown_connections 2024-03-16T11:43:47.427 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:43:47.427+0000 7f9eac7a9700 1 --2- 172.21.15.154:0/2225424398 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f9e8c0a2800 0x7f9e8c0a4cc0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:43:47.427 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:43:47.427+0000 7f9eac7a9700 1 --2- 172.21.15.154:0/2225424398 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9ea4103ea0 0x7f9ea419fbd0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:43:47.427 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:43:47.427+0000 7f9eac7a9700 1 --2- 172.21.15.154:0/2225424398 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9ea4102300 0x7f9ea4198610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:43:47.427 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:43:47.427+0000 7f9eac7a9700 1 --2- 172.21.15.154:0/2225424398 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9ea4103500 0x7f9ea4198b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:43:47.427 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:43:47.427+0000 7f9eac7a9700 1 -- 172.21.15.154:0/2225424398 >> 172.21.15.154:0/2225424398 conn(0x7f9ea40fd8e0 msgr2=0x7f9ea4106870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:43:47.427 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:43:47.427+0000 7f9eac7a9700 1 -- 172.21.15.154:0/2225424398 shutdown_connections 2024-03-16T11:43:47.427 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:43:47.427+0000 7f9eac7a9700 1 -- 172.21.15.154:0/2225424398 wait complete. 2024-03-16T11:43:47.988 DEBUG:teuthology.orchestra.run.smithi154:osd.2> sudo journalctl -f -n 0 -u ceph-d1887f28-e389-11ee-95c9-87774f69a715@osd.2.service 2024-03-16T11:43:47.991 INFO:tasks.cephadm:Deploying osd.3 on smithi029 with /dev/vg_nvme/lv_3... 2024-03-16T11:43:47.991 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d1887f28-e389-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-16T11:43:48.245 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/d1887f28-e389-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T11:43:48.614 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:43:48 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:43:48.614 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:43:48 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:43:48.614 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:43:48 smithi029 ceph-mon[31975]: pgmap v75: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-16T11:43:48.614 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:43:48 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:43:48.614 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:43:48 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:43:48.614 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:43:48 smithi029 ceph-mon[31975]: from='osd.2 [v2:172.21.15.154:6800/4288325280,v1:172.21.15.154:6801/4288325280]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-16T11:43:48.614 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:43:48 smithi029 ceph-mon[31975]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-16T11:43:48.666 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:43:48 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:43:48.667 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:43:48 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:43:48.667 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:43:48 smithi154 ceph-mon[36320]: pgmap v75: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-16T11:43:48.667 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:43:48 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:43:48.667 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:43:48 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:43:48.667 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:43:48 smithi154 ceph-mon[36320]: from='osd.2 [v2:172.21.15.154:6800/4288325280,v1:172.21.15.154:6801/4288325280]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-16T11:43:48.667 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:43:48 smithi154 ceph-mon[36320]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-16T11:43:48.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:43:48 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:43:48.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:43:48 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:43:48.878 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:43:48 smithi152 ceph-mon[36291]: pgmap v75: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-16T11:43:48.878 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:43:48 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:43:48.878 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:43:48 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:43:48.878 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:43:48 smithi152 ceph-mon[36291]: from='osd.2 [v2:172.21.15.154:6800/4288325280,v1:172.21.15.154:6801/4288325280]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-16T11:43:48.878 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:43:48 smithi152 ceph-mon[36291]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-16T11:43:50.046 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:43:49 smithi154 ceph-mon[36320]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-16T11:43:50.046 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:43:49 smithi154 ceph-mon[36320]: osdmap e16: 3 total, 2 up, 3 in 2024-03-16T11:43:50.046 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:43:49 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-16T11:43:50.046 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:43:49 smithi154 ceph-mon[36320]: from='osd.2 [v2:172.21.15.154:6800/4288325280,v1:172.21.15.154:6801/4288325280]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]: dispatch 2024-03-16T11:43:50.046 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:43:49 smithi154 ceph-mon[36320]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]: dispatch 2024-03-16T11:43:50.046 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:43:49 smithi154 ceph-mon[36320]: pgmap v77: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-16T11:43:50.087 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:43:49 smithi029 ceph-mon[31975]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-16T11:43:50.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:43:49 smithi029 ceph-mon[31975]: osdmap e16: 3 total, 2 up, 3 in 2024-03-16T11:43:50.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:43:49 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-16T11:43:50.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:43:49 smithi029 ceph-mon[31975]: from='osd.2 [v2:172.21.15.154:6800/4288325280,v1:172.21.15.154:6801/4288325280]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]: dispatch 2024-03-16T11:43:50.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:43:49 smithi029 ceph-mon[31975]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]: dispatch 2024-03-16T11:43:50.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:43:49 smithi029 ceph-mon[31975]: pgmap v77: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-16T11:43:50.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:43:49 smithi152 ceph-mon[36291]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-16T11:43:50.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:43:49 smithi152 ceph-mon[36291]: osdmap e16: 3 total, 2 up, 3 in 2024-03-16T11:43:50.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:43:49 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-16T11:43:50.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:43:49 smithi152 ceph-mon[36291]: from='osd.2 [v2:172.21.15.154:6800/4288325280,v1:172.21.15.154:6801/4288325280]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]: dispatch 2024-03-16T11:43:50.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:43:49 smithi152 ceph-mon[36291]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]: dispatch 2024-03-16T11:43:50.128 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:43:49 smithi152 ceph-mon[36291]: pgmap v77: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-16T11:43:50.319 INFO:teuthology.orchestra.run.smithi029.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-16T11:43:50.319 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-16T11:43:50.319 INFO:teuthology.orchestra.run.smithi029.stderr: stderr: 10+0 records in 2024-03-16T11:43:50.320 INFO:teuthology.orchestra.run.smithi029.stderr:10+0 records out 2024-03-16T11:43:50.320 INFO:teuthology.orchestra.run.smithi029.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00862827 s, 1.2 GB/s 2024-03-16T11:43:50.320 INFO:teuthology.orchestra.run.smithi029.stderr:--> Zapping successful for: 2024-03-16T11:43:50.477 INFO:journalctl@ceph.osd.2.smithi154.stdout:Mar 16 11:43:50 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-2[41481]: 2024-03-16T11:43:50.177+0000 7fee68a50700 -1 osd.2 0 waiting for initial osdmap 2024-03-16T11:43:50.477 INFO:journalctl@ceph.osd.2.smithi154.stdout:Mar 16 11:43:50 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-2[41481]: 2024-03-16T11:43:50.182+0000 7fee6003a700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-16T11:43:51.016 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:43:50 smithi029 ceph-mon[31975]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]': finished 2024-03-16T11:43:51.016 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:43:50 smithi029 ceph-mon[31975]: osdmap e17: 3 total, 2 up, 3 in 2024-03-16T11:43:51.016 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:43:50 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-16T11:43:51.016 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:43:50 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-16T11:43:51.016 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:43:50 smithi029 ceph-mon[31975]: from='osd.2 ' entity='osd.2' 2024-03-16T11:43:51.042 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d1887f28-e389-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi029:vg_nvme/lv_3 2024-03-16T11:43:51.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:43:50 smithi154 ceph-mon[36320]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]': finished 2024-03-16T11:43:51.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:43:50 smithi154 ceph-mon[36320]: osdmap e17: 3 total, 2 up, 3 in 2024-03-16T11:43:51.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:43:50 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-16T11:43:51.046 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:43:50 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-16T11:43:51.046 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:43:50 smithi154 ceph-mon[36320]: from='osd.2 ' entity='osd.2' 2024-03-16T11:43:51.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:43:50 smithi152 ceph-mon[36291]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]': finished 2024-03-16T11:43:51.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:43:50 smithi152 ceph-mon[36291]: osdmap e17: 3 total, 2 up, 3 in 2024-03-16T11:43:51.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:43:50 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-16T11:43:51.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:43:50 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-16T11:43:51.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:43:50 smithi152 ceph-mon[36291]: from='osd.2 ' entity='osd.2' 2024-03-16T11:43:51.298 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/d1887f28-e389-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T11:43:52.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:43:51 smithi029 ceph-mon[31975]: purged_snaps scrub starts 2024-03-16T11:43:52.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:43:51 smithi029 ceph-mon[31975]: purged_snaps scrub ok 2024-03-16T11:43:52.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:43:51 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-16T11:43:52.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:43:51 smithi029 ceph-mon[31975]: Detected new or changed devices on smithi154 2024-03-16T11:43:52.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:43:51 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:43:52.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:43:51 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:43:52.089 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:43:51 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-16T11:43:52.089 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:43:51 smithi029 ceph-mon[31975]: Adjusting osd_memory_target on smithi154 to 17080M 2024-03-16T11:43:52.089 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:43:51 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:43:52.089 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:43:51 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:43:52.089 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:43:51 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T11:43:52.089 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:43:51 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:43:52.089 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:43:51 smithi029 ceph-mon[31975]: osd.2 [v2:172.21.15.154:6800/4288325280,v1:172.21.15.154:6801/4288325280] boot 2024-03-16T11:43:52.089 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:43:51 smithi029 ceph-mon[31975]: osdmap e18: 3 total, 3 up, 3 in 2024-03-16T11:43:52.089 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:43:51 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-16T11:43:52.089 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:43:51 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T11:43:52.089 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:43:51 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:43:52.089 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:43:51 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T11:43:52.089 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:43:51 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:43:52.089 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:43:51 smithi029 ceph-mon[31975]: pgmap v80: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-16T11:43:52.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:43:51 smithi154 ceph-mon[36320]: purged_snaps scrub starts 2024-03-16T11:43:52.296 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:43:51 smithi154 ceph-mon[36320]: purged_snaps scrub ok 2024-03-16T11:43:52.296 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:43:51 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-16T11:43:52.296 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:43:51 smithi154 ceph-mon[36320]: Detected new or changed devices on smithi154 2024-03-16T11:43:52.296 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:43:51 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:43:52.296 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:43:51 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:43:52.296 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:43:51 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-16T11:43:52.296 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:43:51 smithi154 ceph-mon[36320]: Adjusting osd_memory_target on smithi154 to 17080M 2024-03-16T11:43:52.296 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:43:51 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:43:52.296 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:43:51 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:43:52.296 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:43:51 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T11:43:52.296 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:43:51 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:43:52.296 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:43:51 smithi154 ceph-mon[36320]: osd.2 [v2:172.21.15.154:6800/4288325280,v1:172.21.15.154:6801/4288325280] boot 2024-03-16T11:43:52.296 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:43:51 smithi154 ceph-mon[36320]: osdmap e18: 3 total, 3 up, 3 in 2024-03-16T11:43:52.296 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:43:51 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-16T11:43:52.297 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:43:51 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T11:43:52.297 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:43:51 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:43:52.297 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:43:51 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T11:43:52.297 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:43:51 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:43:52.297 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:43:51 smithi154 ceph-mon[36320]: pgmap v80: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-16T11:43:52.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:43:51 smithi152 ceph-mon[36291]: purged_snaps scrub starts 2024-03-16T11:43:52.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:43:51 smithi152 ceph-mon[36291]: purged_snaps scrub ok 2024-03-16T11:43:52.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:43:51 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-16T11:43:52.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:43:51 smithi152 ceph-mon[36291]: Detected new or changed devices on smithi154 2024-03-16T11:43:52.378 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:43:51 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:43:52.378 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:43:51 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:43:52.378 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:43:51 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-16T11:43:52.378 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:43:51 smithi152 ceph-mon[36291]: Adjusting osd_memory_target on smithi154 to 17080M 2024-03-16T11:43:52.378 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:43:51 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:43:52.378 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:43:51 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:43:52.378 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:43:51 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T11:43:52.378 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:43:51 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:43:52.378 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:43:51 smithi152 ceph-mon[36291]: osd.2 [v2:172.21.15.154:6800/4288325280,v1:172.21.15.154:6801/4288325280] boot 2024-03-16T11:43:52.378 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:43:51 smithi152 ceph-mon[36291]: osdmap e18: 3 total, 3 up, 3 in 2024-03-16T11:43:52.378 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:43:51 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-16T11:43:52.378 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:43:51 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T11:43:52.378 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:43:51 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:43:52.379 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:43:51 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T11:43:52.379 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:43:51 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:43:52.379 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:43:51 smithi152 ceph-mon[36291]: pgmap v80: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-16T11:43:53.032 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:43:53.021+0000 7f2c6a171700 1 -- 172.21.15.29:0/3499261759 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2c64102300 msgr2=0x7f2c64102720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:43:53.032 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:43:53.021+0000 7f2c6a171700 1 --2- 172.21.15.29:0/3499261759 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2c64102300 0x7f2c64102720 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f2c5400b0d0 tx=0x7f2c5400b3e0 comp rx=0 tx=0).stop 2024-03-16T11:43:53.032 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:43:53.021+0000 7f2c6a171700 1 -- 172.21.15.29:0/3499261759 shutdown_connections 2024-03-16T11:43:53.032 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:43:53.021+0000 7f2c6a171700 1 --2- 172.21.15.29:0/3499261759 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2c64103ea0 0x7f2c6410a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:43:53.032 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:43:53.021+0000 7f2c6a171700 1 --2- 172.21.15.29:0/3499261759 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2c64102300 0x7f2c64102720 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:43:53.032 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:43:53.021+0000 7f2c6a171700 1 --2- 172.21.15.29:0/3499261759 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2c64103500 0x7f2c64103960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:43:53.033 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:43:53.021+0000 7f2c6a171700 1 -- 172.21.15.29:0/3499261759 >> 172.21.15.29:0/3499261759 conn(0x7f2c640fd8e0 msgr2=0x7f2c640ffd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:43:53.033 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:43:53.021+0000 7f2c6a171700 1 -- 172.21.15.29:0/3499261759 shutdown_connections 2024-03-16T11:43:53.033 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:43:53.021+0000 7f2c6a171700 1 -- 172.21.15.29:0/3499261759 wait complete. 2024-03-16T11:43:53.033 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:43:53.021+0000 7f2c6a171700 1 Processor -- start 2024-03-16T11:43:53.033 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:43:53.022+0000 7f2c6a171700 1 -- start start 2024-03-16T11:43:53.033 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:43:53.022+0000 7f2c6a171700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2c64102300 0x7f2c641984d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:43:53.033 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:43:53.022+0000 7f2c6a171700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2c64103500 0x7f2c64198a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:43:53.033 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:43:53.022+0000 7f2c6a171700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2c64103ea0 0x7f2c6419fa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:43:53.033 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:43:53.022+0000 7f2c6a171700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f2c6410dec0 con 0x7f2c64102300 2024-03-16T11:43:53.033 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:43:53.022+0000 7f2c6a171700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f2c6410dd40 con 0x7f2c64103ea0 2024-03-16T11:43:53.033 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:43:53.022+0000 7f2c6a171700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f2c6410e040 con 0x7f2c64103500 2024-03-16T11:43:53.033 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:43:53.022+0000 7f2c637fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2c64102300 0x7f2c641984d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:43:53.033 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:43:53.022+0000 7f2c637fe700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2c64102300 0x7f2c641984d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55468/0 (socket says 172.21.15.29:55468) 2024-03-16T11:43:53.033 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:43:53.022+0000 7f2c637fe700 1 -- 172.21.15.29:0/2883518783 learned_addr learned my addr 172.21.15.29:0/2883518783 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:43:53.034 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:43:53.022+0000 7f2c62ffd700 1 --2- 172.21.15.29:0/2883518783 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2c64103500 0x7f2c64198a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:43:53.034 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:43:53.022+0000 7f2c63fff700 1 --2- 172.21.15.29:0/2883518783 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2c64103ea0 0x7f2c6419fa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:43:53.034 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:43:53.022+0000 7f2c637fe700 1 -- 172.21.15.29:0/2883518783 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2c64103500 msgr2=0x7f2c64198a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:43:53.034 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:43:53.022+0000 7f2c637fe700 1 --2- 172.21.15.29:0/2883518783 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2c64103500 0x7f2c64198a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:43:53.034 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:43:53.022+0000 7f2c637fe700 1 -- 172.21.15.29:0/2883518783 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2c64103ea0 msgr2=0x7f2c6419fa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:43:53.034 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:43:53.022+0000 7f2c637fe700 1 --2- 172.21.15.29:0/2883518783 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2c64103ea0 0x7f2c6419fa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:43:53.034 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:43:53.022+0000 7f2c637fe700 1 -- 172.21.15.29:0/2883518783 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c641a0130 con 0x7f2c64102300 2024-03-16T11:43:53.034 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:43:53.022+0000 7f2c637fe700 1 --2- 172.21.15.29:0/2883518783 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2c64102300 0x7f2c641984d0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f2c54007d10 tx=0x7f2c54007d40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:43:53.034 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:43:53.022+0000 7f2c60ff9700 1 -- 172.21.15.29:0/2883518783 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c54021020 con 0x7f2c64102300 2024-03-16T11:43:53.034 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:43:53.022+0000 7f2c6a171700 1 -- 172.21.15.29:0/2883518783 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2c641a03c0 con 0x7f2c64102300 2024-03-16T11:43:53.034 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:43:53.023+0000 7f2c60ff9700 1 -- 172.21.15.29:0/2883518783 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2c5401d950 con 0x7f2c64102300 2024-03-16T11:43:53.034 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:43:53.023+0000 7f2c6a171700 1 -- 172.21.15.29:0/2883518783 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2c641a0950 con 0x7f2c64102300 2024-03-16T11:43:53.034 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:43:53.023+0000 7f2c60ff9700 1 -- 172.21.15.29:0/2883518783 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c54005430 con 0x7f2c64102300 2024-03-16T11:43:53.034 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:43:53.023+0000 7f2c6a171700 1 -- 172.21.15.29:0/2883518783 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2c64068f90 con 0x7f2c64102300 2024-03-16T11:43:53.034 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:43:53.025+0000 7f2c60ff9700 1 -- 172.21.15.29:0/2883518783 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 15) ==== 137234+0+0 (secure 0 0 0) 0x7f2c54024020 con 0x7f2c64102300 2024-03-16T11:43:53.035 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:43:53.025+0000 7f2c60ff9700 1 --2- 172.21.15.29:0/2883518783 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f2c500a2800 0x7f2c500a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:43:53.035 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:43:53.025+0000 7f2c60ff9700 1 -- 172.21.15.29:0/2883518783 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(19..19 src has 1..19) ==== 3226+0+0 (secure 0 0 0) 0x7f2c5400f020 con 0x7f2c64102300 2024-03-16T11:43:53.035 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:43:53.025+0000 7f2c62ffd700 1 --2- 172.21.15.29:0/2883518783 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f2c500a2800 0x7f2c500a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:43:53.035 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:43:53.026+0000 7f2c62ffd700 1 --2- 172.21.15.29:0/2883518783 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f2c500a2800 0x7f2c500a4cc0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f2c4c005950 tx=0x7f2c4c00b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:43:53.035 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:43:53.029+0000 7f2c60ff9700 1 -- 172.21.15.29:0/2883518783 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f2c5409d4d0 con 0x7f2c64102300 2024-03-16T11:43:53.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:43:52 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-16T11:43:53.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:43:52 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-16T11:43:53.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:43:52 smithi029 ceph-mon[31975]: osdmap e19: 3 total, 3 up, 3 in 2024-03-16T11:43:53.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:43:52 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-16T11:43:53.241 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:43:53.242+0000 7f2c6a171700 1 -- 172.21.15.29:0/2883518783 --> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi029:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f2c6402dfa0 con 0x7f2c500a2800 2024-03-16T11:43:53.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:43:52 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-16T11:43:53.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:43:52 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-16T11:43:53.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:43:52 smithi154 ceph-mon[36320]: osdmap e19: 3 total, 3 up, 3 in 2024-03-16T11:43:53.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:43:52 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-16T11:43:53.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:43:52 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-16T11:43:53.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:43:52 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-16T11:43:53.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:43:52 smithi152 ceph-mon[36291]: osdmap e19: 3 total, 3 up, 3 in 2024-03-16T11:43:53.378 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:43:52 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-16T11:43:54.019 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 16 11:43:53 smithi029 sudo[51122]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-16T11:43:54.019 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 16 11:43:53 smithi029 sudo[51122]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-16T11:43:54.019 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 16 11:43:53 smithi029 sudo[51122]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-16T11:43:54.019 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 16 11:43:53 smithi029 sudo[51122]: pam_unix(sudo:session): session closed for user root 2024-03-16T11:43:54.020 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:43:53 smithi029 ceph-mon[31975]: from='client.14289 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi029:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T11:43:54.020 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:43:53 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T11:43:54.020 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:43:53 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T11:43:54.020 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:43:53 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:43:54.020 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:43:53 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-16T11:43:54.020 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:43:53 smithi029 ceph-mon[31975]: osdmap e20: 3 total, 3 up, 3 in 2024-03-16T11:43:54.020 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:43:53 smithi029 ceph-mon[31975]: pgmap v83: 1 pgs: 1 creating+peering; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-16T11:43:54.290 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:43:53 smithi152 ceph-mon[36291]: from='client.14289 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi029:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T11:43:54.290 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:43:53 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T11:43:54.290 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:43:53 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T11:43:54.290 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:43:53 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:43:54.290 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:43:53 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-16T11:43:54.290 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:43:53 smithi152 ceph-mon[36291]: osdmap e20: 3 total, 3 up, 3 in 2024-03-16T11:43:54.290 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:43:53 smithi152 ceph-mon[36291]: pgmap v83: 1 pgs: 1 creating+peering; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-16T11:43:54.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:43:53 smithi154 ceph-mon[36320]: from='client.14289 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi029:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T11:43:54.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:43:53 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T11:43:54.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:43:53 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T11:43:54.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:43:53 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:43:54.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:43:53 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-16T11:43:54.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:43:53 smithi154 ceph-mon[36320]: osdmap e20: 3 total, 3 up, 3 in 2024-03-16T11:43:54.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:43:53 smithi154 ceph-mon[36320]: pgmap v83: 1 pgs: 1 creating+peering; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-16T11:43:54.338 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 16 11:43:54 smithi029 sudo[51146]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-16T11:43:54.338 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 16 11:43:54 smithi029 sudo[51146]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-16T11:43:54.338 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 16 11:43:54 smithi029 sudo[51146]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-16T11:43:54.338 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 16 11:43:54 smithi029 sudo[51146]: pam_unix(sudo:session): session closed for user root 2024-03-16T11:43:54.545 INFO:journalctl@ceph.osd.1.smithi152.stdout:Mar 16 11:43:54 smithi152 sudo[44924]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-16T11:43:54.546 INFO:journalctl@ceph.osd.1.smithi152.stdout:Mar 16 11:43:54 smithi152 sudo[44924]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-16T11:43:54.546 INFO:journalctl@ceph.osd.1.smithi152.stdout:Mar 16 11:43:54 smithi152 sudo[44924]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-16T11:43:54.546 INFO:journalctl@ceph.osd.1.smithi152.stdout:Mar 16 11:43:54 smithi152 sudo[44924]: pam_unix(sudo:session): session closed for user root 2024-03-16T11:43:54.877 INFO:journalctl@ceph.osd.1.smithi152.stdout:Mar 16 11:43:54 smithi152 sudo[44927]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-16T11:43:54.877 INFO:journalctl@ceph.osd.1.smithi152.stdout:Mar 16 11:43:54 smithi152 sudo[44927]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-16T11:43:54.877 INFO:journalctl@ceph.osd.1.smithi152.stdout:Mar 16 11:43:54 smithi152 sudo[44927]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-16T11:43:54.878 INFO:journalctl@ceph.osd.1.smithi152.stdout:Mar 16 11:43:54 smithi152 sudo[44927]: pam_unix(sudo:session): session closed for user root 2024-03-16T11:43:55.070 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:43:55 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:43:55.070 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:43:55 smithi154 ceph-mon[36320]: osdmap e21: 3 total, 3 up, 3 in 2024-03-16T11:43:55.070 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:43:55 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:43:55.070 INFO:journalctl@ceph.osd.2.smithi154.stdout:Mar 16 11:43:54 smithi154 sudo[45565]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-16T11:43:55.070 INFO:journalctl@ceph.osd.2.smithi154.stdout:Mar 16 11:43:54 smithi154 sudo[45565]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-16T11:43:55.070 INFO:journalctl@ceph.osd.2.smithi154.stdout:Mar 16 11:43:54 smithi154 sudo[45565]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-16T11:43:55.071 INFO:journalctl@ceph.osd.2.smithi154.stdout:Mar 16 11:43:54 smithi154 sudo[45565]: pam_unix(sudo:session): session closed for user root 2024-03-16T11:43:55.256 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:43:55 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:43:55.256 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:43:55 smithi029 ceph-mon[31975]: osdmap e21: 3 total, 3 up, 3 in 2024-03-16T11:43:55.256 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:43:55 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:43:55.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:43:55 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:43:55.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:43:55 smithi152 ceph-mon[36291]: osdmap e21: 3 total, 3 up, 3 in 2024-03-16T11:43:55.378 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:43:55 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:43:55.545 INFO:journalctl@ceph.osd.2.smithi154.stdout:Mar 16 11:43:55 smithi154 sudo[45568]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-16T11:43:55.545 INFO:journalctl@ceph.osd.2.smithi154.stdout:Mar 16 11:43:55 smithi154 sudo[45568]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-16T11:43:55.545 INFO:journalctl@ceph.osd.2.smithi154.stdout:Mar 16 11:43:55 smithi154 sudo[45568]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-16T11:43:55.545 INFO:journalctl@ceph.osd.2.smithi154.stdout:Mar 16 11:43:55 smithi154 sudo[45568]: pam_unix(sudo:session): session closed for user root 2024-03-16T11:43:55.588 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:43:55 smithi029 sudo[51191]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-16T11:43:55.588 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:43:55 smithi029 sudo[51191]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-16T11:43:55.588 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:43:55 smithi029 sudo[51191]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-16T11:43:55.588 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:43:55 smithi029 sudo[51191]: pam_unix(sudo:session): session closed for user root 2024-03-16T11:43:55.632 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:43:55.632+0000 7f2c60ff9700 1 -- 172.21.15.29:0/2883518783 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f2c54099440 con 0x7f2c64102300 2024-03-16T11:43:56.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:43:55 smithi029 sudo[51202]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-16T11:43:56.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:43:55 smithi029 sudo[51202]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-16T11:43:56.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:43:55 smithi029 sudo[51202]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-16T11:43:56.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:43:55 smithi029 sudo[51202]: pam_unix(sudo:session): session closed for user root 2024-03-16T11:43:56.115 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:43:55 smithi154 sudo[45571]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-16T11:43:56.115 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:43:55 smithi154 sudo[45571]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-16T11:43:56.116 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:43:55 smithi154 sudo[45571]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-16T11:43:56.116 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:43:55 smithi154 sudo[45571]: pam_unix(sudo:session): session closed for user root 2024-03-16T11:43:56.116 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:43:56 smithi154 sudo[45574]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-16T11:43:56.116 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:43:56 smithi154 sudo[45574]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-16T11:43:56.116 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:43:56 smithi154 sudo[45574]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-16T11:43:56.497 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:43:56 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:43:56.497 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:43:56 smithi029 ceph-mon[31975]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-16T11:43:56.497 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:43:56 smithi029 ceph-mon[31975]: pgmap v85: 1 pgs: 1 creating+peering; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-16T11:43:56.497 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:43:56 smithi029 ceph-mon[31975]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-16T11:43:56.497 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:43:56 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-16T11:43:56.497 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:43:56 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T11:43:56.497 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:43:56 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-16T11:43:56.497 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:43:56 smithi029 ceph-mon[31975]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-03-16T11:43:56.497 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:43:56 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-16T11:43:56.497 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:43:56 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T11:43:56.497 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:43:56 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-16T11:43:56.545 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:43:56 smithi154 sudo[45574]: pam_unix(sudo:session): session closed for user root 2024-03-16T11:43:56.545 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:43:56 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:43:56.545 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:43:56 smithi154 ceph-mon[36320]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-16T11:43:56.545 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:43:56 smithi154 ceph-mon[36320]: pgmap v85: 1 pgs: 1 creating+peering; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-16T11:43:56.545 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:43:56 smithi154 ceph-mon[36320]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-16T11:43:56.545 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:43:56 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-16T11:43:56.545 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:43:56 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T11:43:56.545 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:43:56 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-16T11:43:56.546 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:43:56 smithi154 ceph-mon[36320]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-03-16T11:43:56.546 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:43:56 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-16T11:43:56.546 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:43:56 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T11:43:56.546 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:43:56 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-16T11:43:56.610 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:43:56 smithi152 sudo[44930]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-16T11:43:56.611 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:43:56 smithi152 sudo[44930]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-16T11:43:56.611 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:43:56 smithi152 sudo[44930]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-16T11:43:56.611 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:43:56 smithi152 sudo[44930]: pam_unix(sudo:session): session closed for user root 2024-03-16T11:43:56.878 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:43:56 smithi152 sudo[44933]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-16T11:43:56.878 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:43:56 smithi152 sudo[44933]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-16T11:43:56.878 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:43:56 smithi152 sudo[44933]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-16T11:43:56.878 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:43:56 smithi152 sudo[44933]: pam_unix(sudo:session): session closed for user root 2024-03-16T11:43:56.878 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:43:56 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:43:56.878 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:43:56 smithi152 ceph-mon[36291]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-16T11:43:56.878 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:43:56 smithi152 ceph-mon[36291]: pgmap v85: 1 pgs: 1 creating+peering; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-16T11:43:56.879 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:43:56 smithi152 ceph-mon[36291]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-16T11:43:56.879 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:43:56 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-16T11:43:56.879 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:43:56 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T11:43:56.879 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:43:56 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-16T11:43:56.879 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:43:56 smithi152 ceph-mon[36291]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-03-16T11:43:56.879 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:43:56 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-16T11:43:56.879 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:43:56 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T11:43:56.879 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:43:56 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-16T11:43:57.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:43:57 smithi152 ceph-mon[36291]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-16T11:43:57.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:43:57 smithi152 ceph-mon[36291]: from='client.? 172.21.15.29:0/1239465055' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "069e1795-55bc-4324-b94c-eb24ef2334c0"}]: dispatch 2024-03-16T11:43:57.378 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:43:57 smithi152 ceph-mon[36291]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-16T11:43:57.378 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:43:57 smithi152 ceph-mon[36291]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-16T11:43:57.378 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:43:57 smithi152 ceph-mon[36291]: from='client.? 172.21.15.29:0/1239465055' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "069e1795-55bc-4324-b94c-eb24ef2334c0"}]': finished 2024-03-16T11:43:57.378 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:43:57 smithi152 ceph-mon[36291]: osdmap e22: 4 total, 3 up, 4 in 2024-03-16T11:43:57.378 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:43:57 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-16T11:43:57.378 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:43:57 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T11:43:57.378 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:43:57 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-16T11:43:57.378 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:43:57 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-16T11:43:57.378 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:43:57 smithi152 ceph-mon[36291]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-16T11:43:57.378 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:43:57 smithi152 ceph-mon[36291]: from='client.? 172.21.15.29:0/1213783593' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T11:43:57.545 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:43:57 smithi154 ceph-mon[36320]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-16T11:43:57.545 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:43:57 smithi154 ceph-mon[36320]: from='client.? 172.21.15.29:0/1239465055' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "069e1795-55bc-4324-b94c-eb24ef2334c0"}]: dispatch 2024-03-16T11:43:57.546 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:43:57 smithi154 ceph-mon[36320]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-16T11:43:57.546 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:43:57 smithi154 ceph-mon[36320]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-16T11:43:57.546 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:43:57 smithi154 ceph-mon[36320]: from='client.? 172.21.15.29:0/1239465055' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "069e1795-55bc-4324-b94c-eb24ef2334c0"}]': finished 2024-03-16T11:43:57.546 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:43:57 smithi154 ceph-mon[36320]: osdmap e22: 4 total, 3 up, 4 in 2024-03-16T11:43:57.546 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:43:57 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-16T11:43:57.546 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:43:57 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T11:43:57.546 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:43:57 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-16T11:43:57.546 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:43:57 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-16T11:43:57.546 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:43:57 smithi154 ceph-mon[36320]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-16T11:43:57.546 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:43:57 smithi154 ceph-mon[36320]: from='client.? 172.21.15.29:0/1213783593' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T11:43:57.588 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:43:57 smithi029 ceph-mon[31975]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-16T11:43:57.588 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:43:57 smithi029 ceph-mon[31975]: from='client.? 172.21.15.29:0/1239465055' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "069e1795-55bc-4324-b94c-eb24ef2334c0"}]: dispatch 2024-03-16T11:43:57.588 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:43:57 smithi029 ceph-mon[31975]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-16T11:43:57.588 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:43:57 smithi029 ceph-mon[31975]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-16T11:43:57.588 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:43:57 smithi029 ceph-mon[31975]: from='client.? 172.21.15.29:0/1239465055' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "069e1795-55bc-4324-b94c-eb24ef2334c0"}]': finished 2024-03-16T11:43:57.588 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:43:57 smithi029 ceph-mon[31975]: osdmap e22: 4 total, 3 up, 4 in 2024-03-16T11:43:57.588 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:43:57 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-16T11:43:57.588 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:43:57 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T11:43:57.588 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:43:57 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-16T11:43:57.588 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:43:57 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-16T11:43:57.588 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:43:57 smithi029 ceph-mon[31975]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-16T11:43:57.588 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:43:57 smithi029 ceph-mon[31975]: from='client.? 172.21.15.29:0/1213783593' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T11:43:58.545 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:43:58 smithi154 ceph-mon[36320]: pgmap v87: 1 pgs: 1 creating+peering; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-16T11:43:58.588 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:43:58 smithi029 ceph-mon[31975]: pgmap v87: 1 pgs: 1 creating+peering; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-16T11:43:58.627 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:43:58 smithi152 ceph-mon[36291]: pgmap v87: 1 pgs: 1 creating+peering; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-16T11:44:00.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:43:59 smithi154 ceph-mon[36320]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-16T11:44:00.087 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:43:59 smithi029 ceph-mon[31975]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-16T11:44:00.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:43:59 smithi152 ceph-mon[36291]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-16T11:44:01.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:44:01 smithi029 ceph-mon[31975]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-16T11:44:02.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:44:01 smithi154 ceph-mon[36320]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-16T11:44:02.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:44:01 smithi152 ceph-mon[36291]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-16T11:44:03.793 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:44:03 smithi029 ceph-mon[31975]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-16T11:44:04.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:44:03 smithi154 ceph-mon[36320]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-16T11:44:04.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:44:03 smithi152 ceph-mon[36291]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-16T11:44:04.715 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:44:04 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-16T11:44:04.715 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:44:04 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:44:04.715 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:44:04 smithi029 ceph-mon[31975]: Deploying daemon osd.3 on smithi029 2024-03-16T11:44:05.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:44:04 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-16T11:44:05.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:44:04 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:44:05.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:44:04 smithi154 ceph-mon[36320]: Deploying daemon osd.3 on smithi029 2024-03-16T11:44:05.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:44:04 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-16T11:44:05.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:44:04 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:44:05.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:44:04 smithi152 ceph-mon[36291]: Deploying daemon osd.3 on smithi029 2024-03-16T11:44:05.821 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:44:05 smithi029 ceph-mon[31975]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-16T11:44:06.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:44:05 smithi154 ceph-mon[36320]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-16T11:44:06.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:44:05 smithi152 ceph-mon[36291]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-16T11:44:07.909 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:44:07 smithi029 ceph-mon[31975]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-16T11:44:08.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:44:07 smithi154 ceph-mon[36320]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-16T11:44:08.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:44:07 smithi152 ceph-mon[36291]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-16T11:44:09.929 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:44:09 smithi152 ceph-mon[36291]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-16T11:44:09.930 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:44:09 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:44:09.930 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:44:09 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:44:10.032 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:44:09 smithi029 ceph-mon[31975]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-16T11:44:10.032 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:44:09 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:44:10.032 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:44:09 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:44:10.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:44:09 smithi154 ceph-mon[36320]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-16T11:44:10.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:44:09 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:44:10.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:44:09 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:44:11.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:44:10 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T11:44:11.087 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:44:10 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T11:44:11.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:44:10 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T11:44:11.763 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:44:11 smithi029 ceph-mon[31975]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-16T11:44:12.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:44:11 smithi154 ceph-mon[36320]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-16T11:44:12.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:44:11 smithi152 ceph-mon[36291]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-16T11:44:13.524 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:44:13.524+0000 7f2c60ff9700 1 -- 172.21.15.29:0/2883518783 <== mgr.14152 v2:172.21.15.29:6800/4192064640 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f2c6402dfa0 con 0x7f2c500a2800 2024-03-16T11:44:13.524 INFO:teuthology.orchestra.run.smithi029.stdout:Created osd(s) 3 on host 'smithi029' 2024-03-16T11:44:13.526 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:44:13.527+0000 7f2c6a171700 1 -- 172.21.15.29:0/2883518783 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f2c500a2800 msgr2=0x7f2c500a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:44:13.527 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:44:13.527+0000 7f2c6a171700 1 --2- 172.21.15.29:0/2883518783 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f2c500a2800 0x7f2c500a4cc0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f2c4c005950 tx=0x7f2c4c00b410 comp rx=0 tx=0).stop 2024-03-16T11:44:13.527 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:44:13.527+0000 7f2c6a171700 1 -- 172.21.15.29:0/2883518783 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2c64102300 msgr2=0x7f2c641984d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:44:13.527 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:44:13.527+0000 7f2c6a171700 1 --2- 172.21.15.29:0/2883518783 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2c64102300 0x7f2c641984d0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f2c54007d10 tx=0x7f2c54007d40 comp rx=0 tx=0).stop 2024-03-16T11:44:13.527 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:44:13.527+0000 7f2c6a171700 1 -- 172.21.15.29:0/2883518783 shutdown_connections 2024-03-16T11:44:13.527 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:44:13.527+0000 7f2c6a171700 1 --2- 172.21.15.29:0/2883518783 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2c64103ea0 0x7f2c6419fa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:44:13.527 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:44:13.527+0000 7f2c6a171700 1 --2- 172.21.15.29:0/2883518783 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2c64102300 0x7f2c641984d0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:44:13.527 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:44:13.527+0000 7f2c6a171700 1 --2- 172.21.15.29:0/2883518783 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f2c500a2800 0x7f2c500a4cc0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:44:13.527 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:44:13.527+0000 7f2c6a171700 1 --2- 172.21.15.29:0/2883518783 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2c64103500 0x7f2c64198a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:44:13.527 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:44:13.527+0000 7f2c6a171700 1 -- 172.21.15.29:0/2883518783 >> 172.21.15.29:0/2883518783 conn(0x7f2c640fd8e0 msgr2=0x7f2c64106800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:44:13.527 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:44:13.528+0000 7f2c6a171700 1 -- 172.21.15.29:0/2883518783 shutdown_connections 2024-03-16T11:44:13.527 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:44:13.528+0000 7f2c6a171700 1 -- 172.21.15.29:0/2883518783 wait complete. 2024-03-16T11:44:13.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:44:13 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:44:13.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:44:13 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:44:13.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:44:13 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:44:13.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:44:13 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T11:44:13.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:44:13 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:44:13.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:44:13 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T11:44:13.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:44:13 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:44:13.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:44:13 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:44:13.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:44:13 smithi029 ceph-mon[31975]: from='osd.3 [v2:172.21.15.29:6810/1538750397,v1:172.21.15.29:6811/1538750397]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-16T11:44:13.839 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:44:13 smithi029 ceph-mon[31975]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-16T11:44:13.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:44:13 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:44:13.878 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:44:13 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:44:13.878 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:44:13 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:44:13.878 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:44:13 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T11:44:13.878 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:44:13 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:44:13.878 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:44:13 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T11:44:13.878 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:44:13 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:44:13.878 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:44:13 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:44:13.878 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:44:13 smithi152 ceph-mon[36291]: from='osd.3 [v2:172.21.15.29:6810/1538750397,v1:172.21.15.29:6811/1538750397]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-16T11:44:13.878 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:44:13 smithi152 ceph-mon[36291]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-16T11:44:14.046 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:44:13 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:44:14.046 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:44:13 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:44:14.046 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:44:13 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:44:14.046 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:44:13 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T11:44:14.046 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:44:13 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:44:14.046 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:44:13 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T11:44:14.046 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:44:13 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:44:14.046 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:44:13 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:44:14.046 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:44:13 smithi154 ceph-mon[36320]: from='osd.3 [v2:172.21.15.29:6810/1538750397,v1:172.21.15.29:6811/1538750397]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-16T11:44:14.046 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:44:13 smithi154 ceph-mon[36320]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-16T11:44:14.279 DEBUG:teuthology.orchestra.run.smithi029:osd.3> sudo journalctl -f -n 0 -u ceph-d1887f28-e389-11ee-95c9-87774f69a715@osd.3.service 2024-03-16T11:44:14.281 INFO:tasks.cephadm:Deploying osd.4 on smithi152 with /dev/vg_nvme/lv_3... 2024-03-16T11:44:14.281 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d1887f28-e389-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-16T11:44:14.528 INFO:teuthology.orchestra.run.smithi152.stderr:Inferring config /var/lib/ceph/d1887f28-e389-11ee-95c9-87774f69a715/mon.b/config 2024-03-16T11:44:15.780 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:44:15 smithi029 ceph-mon[31975]: from='osd.3 [v2:172.21.15.29:6810/1538750397,v1:172.21.15.29:6811/1538750397]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-16T11:44:15.781 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:44:15 smithi029 ceph-mon[31975]: osdmap e23: 4 total, 3 up, 4 in 2024-03-16T11:44:15.781 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:44:15 smithi029 ceph-mon[31975]: from='osd.3 [v2:172.21.15.29:6810/1538750397,v1:172.21.15.29:6811/1538750397]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]: dispatch 2024-03-16T11:44:15.781 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:44:15 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-16T11:44:15.787 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:44:15 smithi152 ceph-mon[36291]: from='osd.3 [v2:172.21.15.29:6810/1538750397,v1:172.21.15.29:6811/1538750397]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-16T11:44:15.787 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:44:15 smithi152 ceph-mon[36291]: osdmap e23: 4 total, 3 up, 4 in 2024-03-16T11:44:15.787 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:44:15 smithi152 ceph-mon[36291]: from='osd.3 [v2:172.21.15.29:6810/1538750397,v1:172.21.15.29:6811/1538750397]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]: dispatch 2024-03-16T11:44:15.787 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:44:15 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-16T11:44:15.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:44:15 smithi154 ceph-mon[36320]: from='osd.3 [v2:172.21.15.29:6810/1538750397,v1:172.21.15.29:6811/1538750397]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-16T11:44:15.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:44:15 smithi154 ceph-mon[36320]: osdmap e23: 4 total, 3 up, 4 in 2024-03-16T11:44:15.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:44:15 smithi154 ceph-mon[36320]: from='osd.3 [v2:172.21.15.29:6810/1538750397,v1:172.21.15.29:6811/1538750397]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]: dispatch 2024-03-16T11:44:15.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:44:15 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-16T11:44:16.088 INFO:journalctl@ceph.osd.3.smithi029.stdout:Mar 16 11:44:15 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-3[53281]: 2024-03-16T11:44:15.947+0000 7ffa10a4f700 -1 osd.3 0 waiting for initial osdmap 2024-03-16T11:44:16.088 INFO:journalctl@ceph.osd.3.smithi029.stdout:Mar 16 11:44:15 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-3[53281]: 2024-03-16T11:44:15.953+0000 7ffa0883a700 -1 osd.3 24 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-16T11:44:16.501 INFO:teuthology.orchestra.run.smithi152.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-16T11:44:16.501 INFO:teuthology.orchestra.run.smithi152.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-16T11:44:16.501 INFO:teuthology.orchestra.run.smithi152.stderr: stderr: 10+0 records in 2024-03-16T11:44:16.502 INFO:teuthology.orchestra.run.smithi152.stderr:10+0 records out 2024-03-16T11:44:16.502 INFO:teuthology.orchestra.run.smithi152.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0120294 s, 872 MB/s 2024-03-16T11:44:16.502 INFO:teuthology.orchestra.run.smithi152.stderr:--> Zapping successful for: 2024-03-16T11:44:16.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:44:16 smithi154 ceph-mon[36320]: from='osd.3 [v2:172.21.15.29:6810/1538750397,v1:172.21.15.29:6811/1538750397]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]': finished 2024-03-16T11:44:16.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:44:16 smithi154 ceph-mon[36320]: osdmap e24: 4 total, 3 up, 4 in 2024-03-16T11:44:16.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:44:16 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-16T11:44:16.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:44:16 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-16T11:44:16.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:44:16 smithi154 ceph-mon[36320]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-16T11:44:16.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:44:16 smithi154 ceph-mon[36320]: from='osd.3 [v2:172.21.15.29:6810/1538750397,v1:172.21.15.29:6811/1538750397]' entity='osd.3' 2024-03-16T11:44:16.796 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:44:16 smithi154 ceph-mon[36320]: osd.3 [v2:172.21.15.29:6810/1538750397,v1:172.21.15.29:6811/1538750397] boot 2024-03-16T11:44:16.796 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:44:16 smithi154 ceph-mon[36320]: osdmap e25: 4 total, 4 up, 4 in 2024-03-16T11:44:16.796 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:44:16 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-16T11:44:16.826 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:44:16 smithi152 ceph-mon[36291]: from='osd.3 [v2:172.21.15.29:6810/1538750397,v1:172.21.15.29:6811/1538750397]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]': finished 2024-03-16T11:44:16.826 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:44:16 smithi152 ceph-mon[36291]: osdmap e24: 4 total, 3 up, 4 in 2024-03-16T11:44:16.826 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:44:16 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-16T11:44:16.826 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:44:16 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-16T11:44:16.826 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:44:16 smithi152 ceph-mon[36291]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-16T11:44:16.826 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:44:16 smithi152 ceph-mon[36291]: from='osd.3 [v2:172.21.15.29:6810/1538750397,v1:172.21.15.29:6811/1538750397]' entity='osd.3' 2024-03-16T11:44:16.826 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:44:16 smithi152 ceph-mon[36291]: osd.3 [v2:172.21.15.29:6810/1538750397,v1:172.21.15.29:6811/1538750397] boot 2024-03-16T11:44:16.827 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:44:16 smithi152 ceph-mon[36291]: osdmap e25: 4 total, 4 up, 4 in 2024-03-16T11:44:16.827 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:44:16 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-16T11:44:16.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:44:16 smithi029 ceph-mon[31975]: from='osd.3 [v2:172.21.15.29:6810/1538750397,v1:172.21.15.29:6811/1538750397]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]': finished 2024-03-16T11:44:16.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:44:16 smithi029 ceph-mon[31975]: osdmap e24: 4 total, 3 up, 4 in 2024-03-16T11:44:16.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:44:16 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-16T11:44:16.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:44:16 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-16T11:44:16.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:44:16 smithi029 ceph-mon[31975]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-16T11:44:16.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:44:16 smithi029 ceph-mon[31975]: from='osd.3 [v2:172.21.15.29:6810/1538750397,v1:172.21.15.29:6811/1538750397]' entity='osd.3' 2024-03-16T11:44:16.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:44:16 smithi029 ceph-mon[31975]: osd.3 [v2:172.21.15.29:6810/1538750397,v1:172.21.15.29:6811/1538750397] boot 2024-03-16T11:44:16.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:44:16 smithi029 ceph-mon[31975]: osdmap e25: 4 total, 4 up, 4 in 2024-03-16T11:44:16.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:44:16 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-16T11:44:17.137 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d1887f28-e389-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi152:vg_nvme/lv_3 2024-03-16T11:44:17.371 INFO:teuthology.orchestra.run.smithi152.stderr:Inferring config /var/lib/ceph/d1887f28-e389-11ee-95c9-87774f69a715/mon.b/config 2024-03-16T11:44:17.790 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:44:17 smithi152 ceph-mon[36291]: purged_snaps scrub starts 2024-03-16T11:44:17.791 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:44:17 smithi152 ceph-mon[36291]: purged_snaps scrub ok 2024-03-16T11:44:17.791 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:44:17 smithi152 ceph-mon[36291]: osdmap e26: 4 total, 4 up, 4 in 2024-03-16T11:44:17.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:44:17 smithi154 ceph-mon[36320]: purged_snaps scrub starts 2024-03-16T11:44:17.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:44:17 smithi154 ceph-mon[36320]: purged_snaps scrub ok 2024-03-16T11:44:17.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:44:17 smithi154 ceph-mon[36320]: osdmap e26: 4 total, 4 up, 4 in 2024-03-16T11:44:17.837 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:44:17 smithi029 ceph-mon[31975]: purged_snaps scrub starts 2024-03-16T11:44:17.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:44:17 smithi029 ceph-mon[31975]: purged_snaps scrub ok 2024-03-16T11:44:17.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:44:17 smithi029 ceph-mon[31975]: osdmap e26: 4 total, 4 up, 4 in 2024-03-16T11:44:18.746 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:44:18 smithi152 ceph-mon[36291]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 129 MiB used, 357 GiB / 358 GiB avail 2024-03-16T11:44:18.747 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:44:18 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:44:18.747 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:44:18 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:44:18.747 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:44:18 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-16T11:44:18.747 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:44:18 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-16T11:44:18.747 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:44:18 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:44:18.747 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:44:18 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:44:18.747 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:44:18 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T11:44:18.747 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:44:18 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:44:18.747 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:44:18 smithi152 ceph-mon[36291]: osdmap e27: 4 total, 4 up, 4 in 2024-03-16T11:44:18.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:44:18 smithi154 ceph-mon[36320]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 129 MiB used, 357 GiB / 358 GiB avail 2024-03-16T11:44:18.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:44:18 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:44:18.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:44:18 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:44:18.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:44:18 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-16T11:44:18.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:44:18 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-16T11:44:18.796 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:44:18 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:44:18.796 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:44:18 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:44:18.796 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:44:18 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T11:44:18.796 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:44:18 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:44:18.796 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:44:18 smithi154 ceph-mon[36320]: osdmap e27: 4 total, 4 up, 4 in 2024-03-16T11:44:18.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:44:18 smithi029 ceph-mon[31975]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 129 MiB used, 357 GiB / 358 GiB avail 2024-03-16T11:44:18.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:44:18 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:44:18.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:44:18 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:44:18.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:44:18 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-16T11:44:18.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:44:18 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-16T11:44:18.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:44:18 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:44:18.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:44:18 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:44:18.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:44:18 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T11:44:18.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:44:18 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:44:18.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:44:18 smithi029 ceph-mon[31975]: osdmap e27: 4 total, 4 up, 4 in 2024-03-16T11:44:18.973 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-16T11:44:18.970+0000 7efd391a4700 1 -- 172.21.15.152:0/1491482373 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7efd341035f0 msgr2=0x7efd34103a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:44:18.974 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-16T11:44:18.970+0000 7efd391a4700 1 --2- 172.21.15.152:0/1491482373 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7efd341035f0 0x7efd34103a50 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7efd28009990 tx=0x7efd28009ca0 comp rx=0 tx=0).stop 2024-03-16T11:44:18.974 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-16T11:44:18.970+0000 7efd391a4700 1 -- 172.21.15.152:0/1491482373 shutdown_connections 2024-03-16T11:44:18.974 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-16T11:44:18.970+0000 7efd391a4700 1 --2- 172.21.15.152:0/1491482373 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efd34103f90 0x7efd3410a840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:44:18.974 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-16T11:44:18.970+0000 7efd391a4700 1 --2- 172.21.15.152:0/1491482373 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efd34100610 0x7efd34100a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:44:18.974 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-16T11:44:18.970+0000 7efd391a4700 1 --2- 172.21.15.152:0/1491482373 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7efd341035f0 0x7efd34103a50 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:44:18.975 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-16T11:44:18.970+0000 7efd391a4700 1 -- 172.21.15.152:0/1491482373 >> 172.21.15.152:0/1491482373 conn(0x7efd340fbf70 msgr2=0x7efd340fe3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:44:18.975 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-16T11:44:18.970+0000 7efd391a4700 1 -- 172.21.15.152:0/1491482373 shutdown_connections 2024-03-16T11:44:18.975 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-16T11:44:18.970+0000 7efd391a4700 1 -- 172.21.15.152:0/1491482373 wait complete. 2024-03-16T11:44:18.975 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-16T11:44:18.971+0000 7efd391a4700 1 Processor -- start 2024-03-16T11:44:18.975 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-16T11:44:18.971+0000 7efd391a4700 1 -- start start 2024-03-16T11:44:18.975 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-16T11:44:18.972+0000 7efd391a4700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efd34100610 0x7efd34198460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:44:18.976 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-16T11:44:18.972+0000 7efd391a4700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7efd341035f0 0x7efd341989a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:44:18.976 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-16T11:44:18.972+0000 7efd391a4700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efd34103f90 0x7efd3419fa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:44:18.976 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-16T11:44:18.972+0000 7efd391a4700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7efd3410df10 con 0x7efd34103f90 2024-03-16T11:44:18.976 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-16T11:44:18.972+0000 7efd391a4700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7efd3410dd90 con 0x7efd341035f0 2024-03-16T11:44:18.976 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-16T11:44:18.972+0000 7efd391a4700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7efd3410e090 con 0x7efd34100610 2024-03-16T11:44:18.976 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-16T11:44:18.972+0000 7efd3259c700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7efd341035f0 0x7efd341989a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:44:18.976 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-16T11:44:18.972+0000 7efd3259c700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7efd341035f0 0x7efd341989a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.152:46548/0 (socket says 172.21.15.152:46548) 2024-03-16T11:44:18.977 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-16T11:44:18.972+0000 7efd3259c700 1 -- 172.21.15.152:0/2573608465 learned_addr learned my addr 172.21.15.152:0/2573608465 (peer_addr_for_me v2:172.21.15.152:0/0) 2024-03-16T11:44:18.977 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-16T11:44:18.972+0000 7efd3359e700 1 --2- 172.21.15.152:0/2573608465 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efd34103f90 0x7efd3419fa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:44:18.977 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-16T11:44:18.972+0000 7efd32d9d700 1 --2- 172.21.15.152:0/2573608465 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efd34100610 0x7efd34198460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:44:18.977 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-16T11:44:18.972+0000 7efd3259c700 1 -- 172.21.15.152:0/2573608465 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efd34100610 msgr2=0x7efd34198460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:44:18.977 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-16T11:44:18.972+0000 7efd3259c700 1 --2- 172.21.15.152:0/2573608465 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efd34100610 0x7efd34198460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:44:18.978 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-16T11:44:18.972+0000 7efd3259c700 1 -- 172.21.15.152:0/2573608465 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efd34103f90 msgr2=0x7efd3419fa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:44:18.978 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-16T11:44:18.972+0000 7efd3259c700 1 --2- 172.21.15.152:0/2573608465 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efd34103f90 0x7efd3419fa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:44:18.978 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-16T11:44:18.972+0000 7efd3259c700 1 -- 172.21.15.152:0/2573608465 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd341a00c0 con 0x7efd341035f0 2024-03-16T11:44:18.978 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-16T11:44:18.972+0000 7efd3259c700 1 --2- 172.21.15.152:0/2573608465 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7efd341035f0 0x7efd341989a0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7efd280098e0 tx=0x7efd280057c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:44:18.978 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-16T11:44:18.972+0000 7efd1bfff700 1 -- 172.21.15.152:0/2573608465 <== mon.2 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd28027070 con 0x7efd341035f0 2024-03-16T11:44:18.978 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-16T11:44:18.972+0000 7efd391a4700 1 -- 172.21.15.152:0/2573608465 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efd341a0350 con 0x7efd341035f0 2024-03-16T11:44:18.979 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-16T11:44:18.973+0000 7efd391a4700 1 -- 172.21.15.152:0/2573608465 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efd341a0800 con 0x7efd341035f0 2024-03-16T11:44:18.979 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-16T11:44:18.973+0000 7efd1bfff700 1 -- 172.21.15.152:0/2573608465 <== mon.2 v2:172.21.15.152:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efd28004150 con 0x7efd341035f0 2024-03-16T11:44:18.979 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-16T11:44:18.973+0000 7efd1bfff700 1 -- 172.21.15.152:0/2573608465 <== mon.2 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd28022c30 con 0x7efd341035f0 2024-03-16T11:44:18.979 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-16T11:44:18.973+0000 7efd391a4700 1 -- 172.21.15.152:0/2573608465 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efd20005360 con 0x7efd341035f0 2024-03-16T11:44:18.981 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-16T11:44:18.975+0000 7efd1bfff700 1 -- 172.21.15.152:0/2573608465 <== mon.2 v2:172.21.15.152:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7efd28022470 con 0x7efd341035f0 2024-03-16T11:44:18.981 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-16T11:44:18.975+0000 7efd1bfff700 1 --2- 172.21.15.152:0/2573608465 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7efd1c0a2890 0x7efd1c0a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:44:18.981 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-16T11:44:18.976+0000 7efd1bfff700 1 -- 172.21.15.152:0/2573608465 <== mon.2 v2:172.21.15.152:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 3669+0+0 (secure 0 0 0) 0x7efd280cf1c0 con 0x7efd341035f0 2024-03-16T11:44:18.981 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-16T11:44:18.976+0000 7efd32d9d700 1 --2- 172.21.15.152:0/2573608465 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7efd1c0a2890 0x7efd1c0a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:44:18.981 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-16T11:44:18.976+0000 7efd32d9d700 1 --2- 172.21.15.152:0/2573608465 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7efd1c0a2890 0x7efd1c0a4d50 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7efd3406a7f0 tx=0x7efd2400a3b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:44:18.981 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-16T11:44:18.979+0000 7efd1bfff700 1 -- 172.21.15.152:0/2573608465 <== mon.2 v2:172.21.15.152:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7efd28030070 con 0x7efd341035f0 2024-03-16T11:44:19.184 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-16T11:44:19.184+0000 7efd391a4700 1 -- 172.21.15.152:0/2573608465 --> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi152:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7efd20000bf0 con 0x7efd1c0a2890 2024-03-16T11:44:19.783 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:44:19 smithi152 ceph-mon[36291]: Detected new or changed devices on smithi029 2024-03-16T11:44:19.783 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:44:19 smithi152 ceph-mon[36291]: Adjusting osd_memory_target on smithi029 to 8546M 2024-03-16T11:44:19.783 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:44:19 smithi152 ceph-mon[36291]: osdmap e28: 4 total, 4 up, 4 in 2024-03-16T11:44:19.783 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:44:19 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T11:44:19.783 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:44:19 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T11:44:19.783 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:44:19 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:44:19.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:44:19 smithi154 ceph-mon[36320]: Detected new or changed devices on smithi029 2024-03-16T11:44:19.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:44:19 smithi154 ceph-mon[36320]: Adjusting osd_memory_target on smithi029 to 8546M 2024-03-16T11:44:19.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:44:19 smithi154 ceph-mon[36320]: osdmap e28: 4 total, 4 up, 4 in 2024-03-16T11:44:19.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:44:19 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T11:44:19.796 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:44:19 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T11:44:19.796 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:44:19 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:44:19.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:44:19 smithi029 ceph-mon[31975]: Detected new or changed devices on smithi029 2024-03-16T11:44:19.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:44:19 smithi029 ceph-mon[31975]: Adjusting osd_memory_target on smithi029 to 8546M 2024-03-16T11:44:19.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:44:19 smithi029 ceph-mon[31975]: osdmap e28: 4 total, 4 up, 4 in 2024-03-16T11:44:19.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:44:19 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T11:44:19.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:44:19 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T11:44:19.839 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:44:19 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:44:20.786 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:44:20 smithi152 ceph-mon[36291]: from='client.24203 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi152:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T11:44:20.786 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:44:20 smithi152 ceph-mon[36291]: pgmap v104: 1 pgs: 1 remapped+peering; 577 KiB data, 128 MiB used, 357 GiB / 358 GiB avail 2024-03-16T11:44:20.786 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:44:20 smithi152 ceph-mon[36291]: osdmap e29: 4 total, 4 up, 4 in 2024-03-16T11:44:20.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:44:20 smithi154 ceph-mon[36320]: from='client.24203 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi152:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T11:44:20.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:44:20 smithi154 ceph-mon[36320]: pgmap v104: 1 pgs: 1 remapped+peering; 577 KiB data, 128 MiB used, 357 GiB / 358 GiB avail 2024-03-16T11:44:20.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:44:20 smithi154 ceph-mon[36320]: osdmap e29: 4 total, 4 up, 4 in 2024-03-16T11:44:20.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:44:20 smithi029 ceph-mon[31975]: from='client.24203 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi152:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T11:44:20.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:44:20 smithi029 ceph-mon[31975]: pgmap v104: 1 pgs: 1 remapped+peering; 577 KiB data, 128 MiB used, 357 GiB / 358 GiB avail 2024-03-16T11:44:20.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:44:20 smithi029 ceph-mon[31975]: osdmap e29: 4 total, 4 up, 4 in 2024-03-16T11:44:22.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:44:21 smithi154 ceph-mon[36320]: pgmap v106: 1 pgs: 1 remapped+peering; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-03-16T11:44:22.087 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:44:21 smithi029 ceph-mon[31975]: pgmap v106: 1 pgs: 1 remapped+peering; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-03-16T11:44:22.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:44:21 smithi152 ceph-mon[36291]: pgmap v106: 1 pgs: 1 remapped+peering; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-03-16T11:44:23.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:44:22 smithi154 ceph-mon[36320]: from='client.? 172.21.15.152:0/1193813635' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5a9d43a1-e8f5-4ab5-a441-afbc6ed4066e"}]: dispatch 2024-03-16T11:44:23.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:44:22 smithi154 ceph-mon[36320]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5a9d43a1-e8f5-4ab5-a441-afbc6ed4066e"}]: dispatch 2024-03-16T11:44:23.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:44:22 smithi154 ceph-mon[36320]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5a9d43a1-e8f5-4ab5-a441-afbc6ed4066e"}]': finished 2024-03-16T11:44:23.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:44:22 smithi154 ceph-mon[36320]: osdmap e30: 5 total, 4 up, 5 in 2024-03-16T11:44:23.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:44:22 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-16T11:44:23.046 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:44:22 smithi154 ceph-mon[36320]: from='client.? 172.21.15.152:0/3960210984' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T11:44:23.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:44:22 smithi029 ceph-mon[31975]: from='client.? 172.21.15.152:0/1193813635' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5a9d43a1-e8f5-4ab5-a441-afbc6ed4066e"}]: dispatch 2024-03-16T11:44:23.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:44:22 smithi029 ceph-mon[31975]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5a9d43a1-e8f5-4ab5-a441-afbc6ed4066e"}]: dispatch 2024-03-16T11:44:23.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:44:22 smithi029 ceph-mon[31975]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5a9d43a1-e8f5-4ab5-a441-afbc6ed4066e"}]': finished 2024-03-16T11:44:23.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:44:22 smithi029 ceph-mon[31975]: osdmap e30: 5 total, 4 up, 5 in 2024-03-16T11:44:23.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:44:22 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-16T11:44:23.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:44:22 smithi029 ceph-mon[31975]: from='client.? 172.21.15.152:0/3960210984' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T11:44:23.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:44:22 smithi152 ceph-mon[36291]: from='client.? 172.21.15.152:0/1193813635' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5a9d43a1-e8f5-4ab5-a441-afbc6ed4066e"}]: dispatch 2024-03-16T11:44:23.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:44:22 smithi152 ceph-mon[36291]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5a9d43a1-e8f5-4ab5-a441-afbc6ed4066e"}]: dispatch 2024-03-16T11:44:23.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:44:22 smithi152 ceph-mon[36291]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5a9d43a1-e8f5-4ab5-a441-afbc6ed4066e"}]': finished 2024-03-16T11:44:23.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:44:22 smithi152 ceph-mon[36291]: osdmap e30: 5 total, 4 up, 5 in 2024-03-16T11:44:23.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:44:22 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-16T11:44:23.128 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:44:22 smithi152 ceph-mon[36291]: from='client.? 172.21.15.152:0/3960210984' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T11:44:24.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:44:23 smithi154 ceph-mon[36320]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail; 102 KiB/s, 0 objects/s recovering 2024-03-16T11:44:24.087 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:44:23 smithi029 ceph-mon[31975]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail; 102 KiB/s, 0 objects/s recovering 2024-03-16T11:44:24.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:44:23 smithi152 ceph-mon[36291]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail; 102 KiB/s, 0 objects/s recovering 2024-03-16T11:44:25.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:44:25 smithi152 ceph-mon[36291]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail; 87 KiB/s, 0 objects/s recovering 2024-03-16T11:44:26.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:44:25 smithi154 ceph-mon[36320]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail; 87 KiB/s, 0 objects/s recovering 2024-03-16T11:44:26.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:44:25 smithi029 ceph-mon[31975]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail; 87 KiB/s, 0 objects/s recovering 2024-03-16T11:44:27.904 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:44:27 smithi029 ceph-mon[31975]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-16T11:44:28.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:44:27 smithi154 ceph-mon[36320]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-16T11:44:28.068 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:44:27 smithi152 ceph-mon[36291]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-16T11:44:29.588 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:44:29 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-16T11:44:29.588 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:44:29 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:44:29.627 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:44:29 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-16T11:44:29.627 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:44:29 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:44:29.797 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:44:29 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-16T11:44:29.797 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:44:29 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:44:30.627 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:44:30 smithi152 ceph-mon[36291]: Deploying daemon osd.4 on smithi152 2024-03-16T11:44:30.627 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:44:30 smithi152 ceph-mon[36291]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail; 60 KiB/s, 0 objects/s recovering 2024-03-16T11:44:30.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:44:30 smithi154 ceph-mon[36320]: Deploying daemon osd.4 on smithi152 2024-03-16T11:44:30.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:44:30 smithi154 ceph-mon[36320]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail; 60 KiB/s, 0 objects/s recovering 2024-03-16T11:44:30.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:44:30 smithi029 ceph-mon[31975]: Deploying daemon osd.4 on smithi152 2024-03-16T11:44:30.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:44:30 smithi029 ceph-mon[31975]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail; 60 KiB/s, 0 objects/s recovering 2024-03-16T11:44:31.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:44:31 smithi152 ceph-mon[36291]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-16T11:44:32.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:44:31 smithi154 ceph-mon[36320]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-16T11:44:32.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:44:31 smithi029 ceph-mon[31975]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-16T11:44:33.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:44:33 smithi152 ceph-mon[36291]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail; 49 KiB/s, 0 objects/s recovering 2024-03-16T11:44:34.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:44:33 smithi154 ceph-mon[36320]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail; 49 KiB/s, 0 objects/s recovering 2024-03-16T11:44:34.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:44:33 smithi029 ceph-mon[31975]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail; 49 KiB/s, 0 objects/s recovering 2024-03-16T11:44:36.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:44:35 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:44:36.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:44:35 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:44:36.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:44:35 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T11:44:36.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:44:35 smithi154 ceph-mon[36320]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-03-16T11:44:36.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:44:35 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:44:36.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:44:35 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:44:36.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:44:35 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T11:44:36.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:44:35 smithi029 ceph-mon[31975]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-03-16T11:44:36.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:44:35 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:44:36.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:44:35 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:44:36.128 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:44:35 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T11:44:36.128 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:44:35 smithi152 ceph-mon[36291]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-03-16T11:44:37.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:44:37 smithi152 ceph-mon[36291]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-03-16T11:44:38.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:44:37 smithi154 ceph-mon[36320]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-03-16T11:44:38.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:44:37 smithi029 ceph-mon[31975]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-03-16T11:44:38.301 INFO:teuthology.orchestra.run.smithi152.stdout:Created osd(s) 4 on host 'smithi152' 2024-03-16T11:44:38.301 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-16T11:44:38.301+0000 7efd1bfff700 1 -- 172.21.15.152:0/2573608465 <== mgr.14152 v2:172.21.15.29:6800/4192064640 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7efd20000bf0 con 0x7efd1c0a2890 2024-03-16T11:44:38.303 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-16T11:44:38.303+0000 7efd391a4700 1 -- 172.21.15.152:0/2573608465 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7efd1c0a2890 msgr2=0x7efd1c0a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:44:38.303 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-16T11:44:38.303+0000 7efd391a4700 1 --2- 172.21.15.152:0/2573608465 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7efd1c0a2890 0x7efd1c0a4d50 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7efd3406a7f0 tx=0x7efd2400a3b0 comp rx=0 tx=0).stop 2024-03-16T11:44:38.303 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-16T11:44:38.303+0000 7efd391a4700 1 -- 172.21.15.152:0/2573608465 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7efd341035f0 msgr2=0x7efd341989a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:44:38.303 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-16T11:44:38.303+0000 7efd391a4700 1 --2- 172.21.15.152:0/2573608465 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7efd341035f0 0x7efd341989a0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7efd280098e0 tx=0x7efd280057c0 comp rx=0 tx=0).stop 2024-03-16T11:44:38.303 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-16T11:44:38.303+0000 7efd391a4700 1 -- 172.21.15.152:0/2573608465 shutdown_connections 2024-03-16T11:44:38.303 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-16T11:44:38.303+0000 7efd391a4700 1 --2- 172.21.15.152:0/2573608465 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efd34103f90 0x7efd3419fa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:44:38.303 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-16T11:44:38.303+0000 7efd391a4700 1 --2- 172.21.15.152:0/2573608465 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7efd1c0a2890 0x7efd1c0a4d50 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:44:38.304 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-16T11:44:38.303+0000 7efd391a4700 1 --2- 172.21.15.152:0/2573608465 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efd34100610 0x7efd34198460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:44:38.304 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-16T11:44:38.303+0000 7efd391a4700 1 --2- 172.21.15.152:0/2573608465 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7efd341035f0 0x7efd341989a0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:44:38.304 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-16T11:44:38.303+0000 7efd391a4700 1 -- 172.21.15.152:0/2573608465 >> 172.21.15.152:0/2573608465 conn(0x7efd340fbf70 msgr2=0x7efd34106890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:44:38.304 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-16T11:44:38.304+0000 7efd391a4700 1 -- 172.21.15.152:0/2573608465 shutdown_connections 2024-03-16T11:44:38.304 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-16T11:44:38.304+0000 7efd391a4700 1 -- 172.21.15.152:0/2573608465 wait complete. 2024-03-16T11:44:38.884 DEBUG:teuthology.orchestra.run.smithi152:osd.4> sudo journalctl -f -n 0 -u ceph-d1887f28-e389-11ee-95c9-87774f69a715@osd.4.service 2024-03-16T11:44:38.886 INFO:tasks.cephadm:Deploying osd.5 on smithi154 with /dev/vg_nvme/lv_3... 2024-03-16T11:44:38.886 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d1887f28-e389-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-16T11:44:39.127 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/d1887f28-e389-11ee-95c9-87774f69a715/mon.c/config 2024-03-16T11:44:39.515 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:44:39 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:44:39.515 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:44:39 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:44:39.515 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:44:39 smithi154 ceph-mon[36320]: from='osd.4 [v2:172.21.15.152:6808/2795081750,v1:172.21.15.152:6809/2795081750]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-16T11:44:39.515 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:44:39 smithi154 ceph-mon[36320]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-16T11:44:39.515 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:44:39 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:44:39.515 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:44:39 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:44:39.588 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:44:39 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:44:39.588 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:44:39 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:44:39.588 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:44:39 smithi029 ceph-mon[31975]: from='osd.4 [v2:172.21.15.152:6808/2795081750,v1:172.21.15.152:6809/2795081750]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-16T11:44:39.588 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:44:39 smithi029 ceph-mon[31975]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-16T11:44:39.588 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:44:39 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:44:39.588 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:44:39 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:44:39.603 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:44:39 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:44:39.603 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:44:39 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:44:39.603 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:44:39 smithi152 ceph-mon[36291]: from='osd.4 [v2:172.21.15.152:6808/2795081750,v1:172.21.15.152:6809/2795081750]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-16T11:44:39.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:44:39 smithi152 ceph-mon[36291]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-16T11:44:39.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:44:39 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:44:39.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:44:39 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:44:40.545 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:44:40 smithi154 ceph-mon[36320]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-16T11:44:40.545 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:44:40 smithi154 ceph-mon[36320]: osdmap e31: 5 total, 4 up, 5 in 2024-03-16T11:44:40.545 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:44:40 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-16T11:44:40.545 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:44:40 smithi154 ceph-mon[36320]: from='osd.4 [v2:172.21.15.152:6808/2795081750,v1:172.21.15.152:6809/2795081750]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]: dispatch 2024-03-16T11:44:40.545 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:44:40 smithi154 ceph-mon[36320]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]: dispatch 2024-03-16T11:44:40.546 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:44:40 smithi154 ceph-mon[36320]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-03-16T11:44:40.588 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:44:40 smithi029 ceph-mon[31975]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-16T11:44:40.588 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:44:40 smithi029 ceph-mon[31975]: osdmap e31: 5 total, 4 up, 5 in 2024-03-16T11:44:40.588 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:44:40 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-16T11:44:40.588 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:44:40 smithi029 ceph-mon[31975]: from='osd.4 [v2:172.21.15.152:6808/2795081750,v1:172.21.15.152:6809/2795081750]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]: dispatch 2024-03-16T11:44:40.588 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:44:40 smithi029 ceph-mon[31975]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]: dispatch 2024-03-16T11:44:40.588 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:44:40 smithi029 ceph-mon[31975]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-03-16T11:44:40.600 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:44:40 smithi152 ceph-mon[36291]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-16T11:44:40.600 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:44:40 smithi152 ceph-mon[36291]: osdmap e31: 5 total, 4 up, 5 in 2024-03-16T11:44:40.600 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:44:40 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-16T11:44:40.600 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:44:40 smithi152 ceph-mon[36291]: from='osd.4 [v2:172.21.15.152:6808/2795081750,v1:172.21.15.152:6809/2795081750]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]: dispatch 2024-03-16T11:44:40.600 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:44:40 smithi152 ceph-mon[36291]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]: dispatch 2024-03-16T11:44:40.600 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:44:40 smithi152 ceph-mon[36291]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-03-16T11:44:40.601 INFO:journalctl@ceph.osd.4.smithi152.stdout:Mar 16 11:44:40 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-4[47757]: 2024-03-16T11:44:40.275+0000 7fd438bb4700 -1 osd.4 0 waiting for initial osdmap 2024-03-16T11:44:40.601 INFO:journalctl@ceph.osd.4.smithi152.stdout:Mar 16 11:44:40 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-4[47757]: 2024-03-16T11:44:40.280+0000 7fd4349a7700 -1 osd.4 32 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-16T11:44:41.132 INFO:teuthology.orchestra.run.smithi154.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-16T11:44:41.133 INFO:teuthology.orchestra.run.smithi154.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-16T11:44:41.133 INFO:teuthology.orchestra.run.smithi154.stderr: stderr: 10+0 records in 2024-03-16T11:44:41.133 INFO:teuthology.orchestra.run.smithi154.stderr:10+0 records out 2024-03-16T11:44:41.133 INFO:teuthology.orchestra.run.smithi154.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0146788 s, 714 MB/s 2024-03-16T11:44:41.133 INFO:teuthology.orchestra.run.smithi154.stderr:--> Zapping successful for: 2024-03-16T11:44:41.427 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:44:41 smithi154 ceph-mon[36320]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]': finished 2024-03-16T11:44:41.427 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:44:41 smithi154 ceph-mon[36320]: osdmap e32: 5 total, 4 up, 5 in 2024-03-16T11:44:41.427 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:44:41 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-16T11:44:41.427 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:44:41 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-16T11:44:41.588 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:44:41 smithi029 ceph-mon[31975]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]': finished 2024-03-16T11:44:41.588 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:44:41 smithi029 ceph-mon[31975]: osdmap e32: 5 total, 4 up, 5 in 2024-03-16T11:44:41.588 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:44:41 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-16T11:44:41.588 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:44:41 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-16T11:44:41.628 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:44:41 smithi152 ceph-mon[36291]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]': finished 2024-03-16T11:44:41.628 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:44:41 smithi152 ceph-mon[36291]: osdmap e32: 5 total, 4 up, 5 in 2024-03-16T11:44:41.628 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:44:41 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-16T11:44:41.628 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:44:41 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-16T11:44:41.751 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d1887f28-e389-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi154:vg_nvme/lv_3 2024-03-16T11:44:41.987 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/d1887f28-e389-11ee-95c9-87774f69a715/mon.c/config 2024-03-16T11:44:42.373 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:44:42 smithi154 ceph-mon[36320]: purged_snaps scrub starts 2024-03-16T11:44:42.373 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:44:42 smithi154 ceph-mon[36320]: purged_snaps scrub ok 2024-03-16T11:44:42.373 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:44:42 smithi154 ceph-mon[36320]: osd.4 [v2:172.21.15.152:6808/2795081750,v1:172.21.15.152:6809/2795081750] boot 2024-03-16T11:44:42.373 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:44:42 smithi154 ceph-mon[36320]: osdmap e33: 5 total, 5 up, 5 in 2024-03-16T11:44:42.373 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:44:42 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-16T11:44:42.373 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:44:42 smithi154 ceph-mon[36320]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-03-16T11:44:42.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:44:42 smithi152 ceph-mon[36291]: purged_snaps scrub starts 2024-03-16T11:44:42.378 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:44:42 smithi152 ceph-mon[36291]: purged_snaps scrub ok 2024-03-16T11:44:42.378 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:44:42 smithi152 ceph-mon[36291]: osd.4 [v2:172.21.15.152:6808/2795081750,v1:172.21.15.152:6809/2795081750] boot 2024-03-16T11:44:42.378 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:44:42 smithi152 ceph-mon[36291]: osdmap e33: 5 total, 5 up, 5 in 2024-03-16T11:44:42.378 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:44:42 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-16T11:44:42.378 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:44:42 smithi152 ceph-mon[36291]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-03-16T11:44:42.588 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:44:42 smithi029 ceph-mon[31975]: purged_snaps scrub starts 2024-03-16T11:44:42.588 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:44:42 smithi029 ceph-mon[31975]: purged_snaps scrub ok 2024-03-16T11:44:42.588 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:44:42 smithi029 ceph-mon[31975]: osd.4 [v2:172.21.15.152:6808/2795081750,v1:172.21.15.152:6809/2795081750] boot 2024-03-16T11:44:42.588 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:44:42 smithi029 ceph-mon[31975]: osdmap e33: 5 total, 5 up, 5 in 2024-03-16T11:44:42.588 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:44:42 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-16T11:44:42.588 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:44:42 smithi029 ceph-mon[31975]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-03-16T11:44:43.545 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:44:43 smithi154 ceph-mon[36320]: osdmap e34: 5 total, 5 up, 5 in 2024-03-16T11:44:43.546 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:44:43 smithi154 ceph-mon[36320]: Detected new or changed devices on smithi152 2024-03-16T11:44:43.546 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:44:43 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:44:43.546 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:44:43 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:44:43.546 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:44:43 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-16T11:44:43.546 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:44:43 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-16T11:44:43.546 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:44:43 smithi154 ceph-mon[36320]: Adjusting osd_memory_target on smithi152 to 8540M 2024-03-16T11:44:43.546 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:44:43 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:44:43.546 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:44:43 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:44:43.546 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:44:43 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T11:44:43.546 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:44:43 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:44:43.546 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:44:43 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T11:44:43.547 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:44:43 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:44:43.547 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:44:43 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T11:44:43.547 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:44:43 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:44:43.588 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:44:43 smithi029 ceph-mon[31975]: osdmap e34: 5 total, 5 up, 5 in 2024-03-16T11:44:43.588 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:44:43 smithi029 ceph-mon[31975]: Detected new or changed devices on smithi152 2024-03-16T11:44:43.588 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:44:43 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:44:43.588 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:44:43 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:44:43.588 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:44:43 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-16T11:44:43.588 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:44:43 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-16T11:44:43.588 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:44:43 smithi029 ceph-mon[31975]: Adjusting osd_memory_target on smithi152 to 8540M 2024-03-16T11:44:43.588 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:44:43 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:44:43.588 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:44:43 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:44:43.588 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:44:43 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T11:44:43.588 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:44:43 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:44:43.589 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:44:43 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T11:44:43.589 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:44:43 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:44:43.589 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:44:43 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T11:44:43.589 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:44:43 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:44:43.597 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:44:43.592+0000 7efe5de4f700 1 -- 172.21.15.154:0/3852042986 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efe581022f0 msgr2=0x7efe58102710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:44:43.597 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:44:43.592+0000 7efe5de4f700 1 --2- 172.21.15.154:0/3852042986 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efe581022f0 0x7efe58102710 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7efe48009990 tx=0x7efe48009ca0 comp rx=0 tx=0).stop 2024-03-16T11:44:43.597 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:44:43.592+0000 7efe5de4f700 1 -- 172.21.15.154:0/3852042986 shutdown_connections 2024-03-16T11:44:43.597 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:44:43.592+0000 7efe5de4f700 1 --2- 172.21.15.154:0/3852042986 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efe58103e90 0x7efe5810a740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:44:43.597 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:44:43.592+0000 7efe5de4f700 1 --2- 172.21.15.154:0/3852042986 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efe581022f0 0x7efe58102710 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:44:43.597 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:44:43.592+0000 7efe5de4f700 1 --2- 172.21.15.154:0/3852042986 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7efe581034f0 0x7efe58103950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:44:43.597 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:44:43.592+0000 7efe5de4f700 1 -- 172.21.15.154:0/3852042986 >> 172.21.15.154:0/3852042986 conn(0x7efe580fd8b0 msgr2=0x7efe580ffcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:44:43.597 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:44:43.592+0000 7efe5de4f700 1 -- 172.21.15.154:0/3852042986 shutdown_connections 2024-03-16T11:44:43.597 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:44:43.592+0000 7efe5de4f700 1 -- 172.21.15.154:0/3852042986 wait complete. 2024-03-16T11:44:43.597 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:44:43.593+0000 7efe5de4f700 1 Processor -- start 2024-03-16T11:44:43.598 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:44:43.593+0000 7efe5de4f700 1 -- start start 2024-03-16T11:44:43.598 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:44:43.593+0000 7efe5de4f700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7efe581022f0 0x7efe58076070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:44:43.598 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:44:43.593+0000 7efe5de4f700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efe581034f0 0x7efe580746c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:44:43.598 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:44:43.593+0000 7efe5de4f700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efe58103e90 0x7efe58074c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:44:43.598 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:44:43.593+0000 7efe5de4f700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7efe5810ded0 con 0x7efe581034f0 2024-03-16T11:44:43.598 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:44:43.593+0000 7efe5de4f700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7efe5810dd50 con 0x7efe581022f0 2024-03-16T11:44:43.598 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:44:43.593+0000 7efe5de4f700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7efe5810e050 con 0x7efe58103e90 2024-03-16T11:44:43.598 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:44:43.593+0000 7efe57fff700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efe58103e90 0x7efe58074c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:44:43.598 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:44:43.593+0000 7efe57fff700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efe58103e90 0x7efe58074c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.154:33294/0 (socket says 172.21.15.154:33294) 2024-03-16T11:44:43.598 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:44:43.593+0000 7efe57fff700 1 -- 172.21.15.154:0/2746270604 learned_addr learned my addr 172.21.15.154:0/2746270604 (peer_addr_for_me v2:172.21.15.154:0/0) 2024-03-16T11:44:43.598 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:44:43.593+0000 7efe577fe700 1 --2- 172.21.15.154:0/2746270604 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7efe581022f0 0x7efe58076070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:44:43.598 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:44:43.593+0000 7efe57fff700 1 -- 172.21.15.154:0/2746270604 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7efe581022f0 msgr2=0x7efe58076070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:44:43.598 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:44:43.593+0000 7efe57fff700 1 --2- 172.21.15.154:0/2746270604 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7efe581022f0 0x7efe58076070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:44:43.599 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:44:43.593+0000 7efe56ffd700 1 --2- 172.21.15.154:0/2746270604 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efe581034f0 0x7efe580746c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:44:43.599 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:44:43.593+0000 7efe57fff700 1 -- 172.21.15.154:0/2746270604 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efe581034f0 msgr2=0x7efe580746c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:44:43.599 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:44:43.593+0000 7efe57fff700 1 --2- 172.21.15.154:0/2746270604 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efe581034f0 0x7efe580746c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:44:43.599 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:44:43.593+0000 7efe57fff700 1 -- 172.21.15.154:0/2746270604 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe580754b0 con 0x7efe58103e90 2024-03-16T11:44:43.599 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:44:43.593+0000 7efe56ffd700 1 --2- 172.21.15.154:0/2746270604 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efe581034f0 0x7efe580746c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-16T11:44:43.599 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:44:43.594+0000 7efe57fff700 1 --2- 172.21.15.154:0/2746270604 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efe58103e90 0x7efe58074c00 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7efe50007ee0 tx=0x7efe5000d3b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:44:43.599 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:44:43.594+0000 7efe54ff9700 1 -- 172.21.15.154:0/2746270604 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe5000dbd0 con 0x7efe58103e90 2024-03-16T11:44:43.599 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:44:43.594+0000 7efe5de4f700 1 -- 172.21.15.154:0/2746270604 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efe581a26d0 con 0x7efe58103e90 2024-03-16T11:44:43.599 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:44:43.594+0000 7efe54ff9700 1 -- 172.21.15.154:0/2746270604 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efe5000dd70 con 0x7efe58103e90 2024-03-16T11:44:43.599 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:44:43.594+0000 7efe54ff9700 1 -- 172.21.15.154:0/2746270604 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe500133f0 con 0x7efe58103e90 2024-03-16T11:44:43.599 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:44:43.594+0000 7efe5de4f700 1 -- 172.21.15.154:0/2746270604 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efe581a2be0 con 0x7efe58103e90 2024-03-16T11:44:43.599 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:44:43.595+0000 7efe5de4f700 1 -- 172.21.15.154:0/2746270604 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efe58068f90 con 0x7efe58103e90 2024-03-16T11:44:43.599 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:44:43.596+0000 7efe54ff9700 1 -- 172.21.15.154:0/2746270604 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7efe5001a020 con 0x7efe58103e90 2024-03-16T11:44:43.599 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:44:43.596+0000 7efe54ff9700 1 --2- 172.21.15.154:0/2746270604 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7efe400a2890 0x7efe400a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:44:43.599 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:44:43.597+0000 7efe54ff9700 1 -- 172.21.15.154:0/2746270604 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 4101+0+0 (secure 0 0 0) 0x7efe500cd8a0 con 0x7efe58103e90 2024-03-16T11:44:43.600 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:44:43.597+0000 7efe577fe700 1 --2- 172.21.15.154:0/2746270604 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7efe400a2890 0x7efe400a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:44:43.600 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:44:43.597+0000 7efe577fe700 1 --2- 172.21.15.154:0/2746270604 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7efe400a2890 0x7efe400a4d50 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7efe48006010 tx=0x7efe48005e80 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:44:43.600 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:44:43.600+0000 7efe54ff9700 1 -- 172.21.15.154:0/2746270604 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7efe50010070 con 0x7efe58103e90 2024-03-16T11:44:43.627 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:44:43 smithi152 ceph-mon[36291]: osdmap e34: 5 total, 5 up, 5 in 2024-03-16T11:44:43.627 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:44:43 smithi152 ceph-mon[36291]: Detected new or changed devices on smithi152 2024-03-16T11:44:43.627 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:44:43 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:44:43.627 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:44:43 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:44:43.628 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:44:43 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-16T11:44:43.628 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:44:43 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-16T11:44:43.628 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:44:43 smithi152 ceph-mon[36291]: Adjusting osd_memory_target on smithi152 to 8540M 2024-03-16T11:44:43.628 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:44:43 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:44:43.628 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:44:43 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:44:43.628 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:44:43 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T11:44:43.628 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:44:43 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:44:43.628 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:44:43 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T11:44:43.628 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:44:43 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:44:43.628 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:44:43 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T11:44:43.628 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:44:43 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:44:43.805 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:44:43.805+0000 7efe5de4f700 1 -- 172.21.15.154:0/2746270604 --> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi154:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7efe58062710 con 0x7efe400a2890 2024-03-16T11:44:44.545 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:44:44 smithi154 ceph-mon[36320]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-03-16T11:44:44.545 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:44:44 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T11:44:44.546 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:44:44 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T11:44:44.546 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:44:44 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:44:44.588 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:44:44 smithi029 ceph-mon[31975]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-03-16T11:44:44.588 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:44:44 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T11:44:44.588 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:44:44 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T11:44:44.588 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:44:44 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:44:44.627 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:44:44 smithi152 ceph-mon[36291]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-03-16T11:44:44.627 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:44:44 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T11:44:44.627 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:44:44 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T11:44:44.628 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:44:44 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:44:45.379 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:44:45 smithi154 ceph-mon[36320]: from='client.24247 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi154:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T11:44:45.588 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:44:45 smithi029 ceph-mon[31975]: from='client.24247 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi154:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T11:44:45.627 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:44:45 smithi152 ceph-mon[36291]: from='client.24247 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi154:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T11:44:46.545 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:44:46 smithi154 ceph-mon[36320]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-03-16T11:44:46.588 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:44:46 smithi029 ceph-mon[31975]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-03-16T11:44:46.627 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:44:46 smithi152 ceph-mon[36291]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-03-16T11:44:47.588 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:44:47 smithi029 ceph-mon[31975]: from='client.? 172.21.15.154:0/4101608304' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "234de428-4ac7-44be-9a54-7b58331fdf8c"}]: dispatch 2024-03-16T11:44:47.588 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:44:47 smithi029 ceph-mon[31975]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "234de428-4ac7-44be-9a54-7b58331fdf8c"}]: dispatch 2024-03-16T11:44:47.588 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:44:47 smithi029 ceph-mon[31975]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "234de428-4ac7-44be-9a54-7b58331fdf8c"}]': finished 2024-03-16T11:44:47.588 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:44:47 smithi029 ceph-mon[31975]: osdmap e35: 6 total, 5 up, 6 in 2024-03-16T11:44:47.588 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:44:47 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-16T11:44:47.588 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:44:47 smithi029 ceph-mon[31975]: from='client.? 172.21.15.154:0/103165092' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T11:44:47.627 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:44:47 smithi152 ceph-mon[36291]: from='client.? 172.21.15.154:0/4101608304' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "234de428-4ac7-44be-9a54-7b58331fdf8c"}]: dispatch 2024-03-16T11:44:47.627 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:44:47 smithi152 ceph-mon[36291]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "234de428-4ac7-44be-9a54-7b58331fdf8c"}]: dispatch 2024-03-16T11:44:47.627 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:44:47 smithi152 ceph-mon[36291]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "234de428-4ac7-44be-9a54-7b58331fdf8c"}]': finished 2024-03-16T11:44:47.628 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:44:47 smithi152 ceph-mon[36291]: osdmap e35: 6 total, 5 up, 6 in 2024-03-16T11:44:47.628 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:44:47 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-16T11:44:47.628 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:44:47 smithi152 ceph-mon[36291]: from='client.? 172.21.15.154:0/103165092' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T11:44:47.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:44:47 smithi154 ceph-mon[36320]: from='client.? 172.21.15.154:0/4101608304' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "234de428-4ac7-44be-9a54-7b58331fdf8c"}]: dispatch 2024-03-16T11:44:47.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:44:47 smithi154 ceph-mon[36320]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "234de428-4ac7-44be-9a54-7b58331fdf8c"}]: dispatch 2024-03-16T11:44:47.796 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:44:47 smithi154 ceph-mon[36320]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "234de428-4ac7-44be-9a54-7b58331fdf8c"}]': finished 2024-03-16T11:44:47.796 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:44:47 smithi154 ceph-mon[36320]: osdmap e35: 6 total, 5 up, 6 in 2024-03-16T11:44:47.796 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:44:47 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-16T11:44:47.796 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:44:47 smithi154 ceph-mon[36320]: from='client.? 172.21.15.154:0/103165092' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T11:44:48.588 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:44:48 smithi029 ceph-mon[31975]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-03-16T11:44:48.627 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:44:48 smithi152 ceph-mon[36291]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-03-16T11:44:48.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:44:48 smithi154 ceph-mon[36320]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-03-16T11:44:50.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:44:49 smithi154 ceph-mon[36320]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-03-16T11:44:50.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:44:49 smithi029 ceph-mon[31975]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-03-16T11:44:50.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:44:49 smithi152 ceph-mon[36291]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-03-16T11:44:51.758 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:44:51 smithi154 ceph-mon[36320]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-03-16T11:44:51.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:44:51 smithi152 ceph-mon[36291]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-03-16T11:44:52.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:44:51 smithi029 ceph-mon[31975]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-03-16T11:44:53.744 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:44:53 smithi154 ceph-mon[36320]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-03-16T11:44:54.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:44:53 smithi029 ceph-mon[31975]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-03-16T11:44:54.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:44:53 smithi152 ceph-mon[36291]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-03-16T11:44:55.041 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:44:54 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-16T11:44:55.041 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:44:54 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:44:55.041 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:44:54 smithi154 ceph-mon[36320]: Deploying daemon osd.5 on smithi154 2024-03-16T11:44:55.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:44:54 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-16T11:44:55.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:44:54 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:44:55.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:44:54 smithi029 ceph-mon[31975]: Deploying daemon osd.5 on smithi154 2024-03-16T11:44:55.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:44:54 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-16T11:44:55.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:44:54 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:44:55.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:44:54 smithi152 ceph-mon[36291]: Deploying daemon osd.5 on smithi154 2024-03-16T11:44:55.988 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:44:55 smithi154 ceph-mon[36320]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-03-16T11:44:56.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:44:55 smithi029 ceph-mon[31975]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-03-16T11:44:56.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:44:55 smithi152 ceph-mon[36291]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-03-16T11:44:57.904 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:44:57 smithi029 ceph-mon[31975]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-03-16T11:44:58.015 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:44:57 smithi154 ceph-mon[36320]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-03-16T11:44:58.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:44:57 smithi152 ceph-mon[36291]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-03-16T11:45:00.545 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:45:00 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:45:00.545 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:45:00 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:45:00.545 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:45:00 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T11:45:00.545 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:45:00 smithi154 ceph-mon[36320]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-03-16T11:45:00.588 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:45:00 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:45:00.588 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:45:00 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:45:00.588 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:45:00 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T11:45:00.588 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:45:00 smithi029 ceph-mon[31975]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-03-16T11:45:00.628 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:45:00 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:45:00.628 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:45:00 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:45:00.628 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:45:00 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T11:45:00.628 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:45:00 smithi152 ceph-mon[36291]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-03-16T11:45:01.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:45:01 smithi154 ceph-mon[36320]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-03-16T11:45:02.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:45:01 smithi029 ceph-mon[31975]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-03-16T11:45:02.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:45:01 smithi152 ceph-mon[36291]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-03-16T11:45:03.070 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:45:03.070+0000 7efe54ff9700 1 -- 172.21.15.154:0/2746270604 <== mgr.14152 v2:172.21.15.29:6800/4192064640 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7efe58062710 con 0x7efe400a2890 2024-03-16T11:45:03.070 INFO:teuthology.orchestra.run.smithi154.stdout:Created osd(s) 5 on host 'smithi154' 2024-03-16T11:45:03.072 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:45:03.072+0000 7efe5de4f700 1 -- 172.21.15.154:0/2746270604 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7efe400a2890 msgr2=0x7efe400a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:45:03.072 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:45:03.072+0000 7efe5de4f700 1 --2- 172.21.15.154:0/2746270604 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7efe400a2890 0x7efe400a4d50 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7efe48006010 tx=0x7efe48005e80 comp rx=0 tx=0).stop 2024-03-16T11:45:03.072 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:45:03.072+0000 7efe5de4f700 1 -- 172.21.15.154:0/2746270604 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efe58103e90 msgr2=0x7efe58074c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:45:03.072 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:45:03.072+0000 7efe5de4f700 1 --2- 172.21.15.154:0/2746270604 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efe58103e90 0x7efe58074c00 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7efe50007ee0 tx=0x7efe5000d3b0 comp rx=0 tx=0).stop 2024-03-16T11:45:03.072 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:45:03.072+0000 7efe5de4f700 1 -- 172.21.15.154:0/2746270604 shutdown_connections 2024-03-16T11:45:03.072 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:45:03.072+0000 7efe5de4f700 1 --2- 172.21.15.154:0/2746270604 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7efe400a2890 0x7efe400a4d50 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:45:03.072 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:45:03.072+0000 7efe5de4f700 1 --2- 172.21.15.154:0/2746270604 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efe58103e90 0x7efe58074c00 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:45:03.073 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:45:03.072+0000 7efe5de4f700 1 --2- 172.21.15.154:0/2746270604 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7efe581022f0 0x7efe58076070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:45:03.073 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:45:03.072+0000 7efe5de4f700 1 --2- 172.21.15.154:0/2746270604 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efe581034f0 0x7efe580746c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:45:03.073 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:45:03.072+0000 7efe5de4f700 1 -- 172.21.15.154:0/2746270604 >> 172.21.15.154:0/2746270604 conn(0x7efe580fd8b0 msgr2=0x7efe58106870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:45:03.073 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:45:03.073+0000 7efe5de4f700 1 -- 172.21.15.154:0/2746270604 shutdown_connections 2024-03-16T11:45:03.073 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:45:03.073+0000 7efe5de4f700 1 -- 172.21.15.154:0/2746270604 wait complete. 2024-03-16T11:45:03.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:45:03 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:45:03.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:45:03 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:45:03.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:45:03 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:45:03.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:45:03 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T11:45:03.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:45:03 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:45:03.296 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:45:03 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:45:03.296 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:45:03 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:45:03.296 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:45:03 smithi154 ceph-mon[36320]: from='osd.5 [v2:172.21.15.154:6808/1472686560,v1:172.21.15.154:6809/1472686560]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-16T11:45:03.296 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:45:03 smithi154 ceph-mon[36320]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-16T11:45:03.296 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:45:03 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T11:45:03.588 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:45:03 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:45:03.588 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:45:03 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:45:03.588 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:45:03 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:45:03.588 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:45:03 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T11:45:03.588 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:45:03 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:45:03.589 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:45:03 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:45:03.589 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:45:03 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:45:03.589 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:45:03 smithi029 ceph-mon[31975]: from='osd.5 [v2:172.21.15.154:6808/1472686560,v1:172.21.15.154:6809/1472686560]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-16T11:45:03.589 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:45:03 smithi029 ceph-mon[31975]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-16T11:45:03.589 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:45:03 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T11:45:03.627 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:45:03 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:45:03.627 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:45:03 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:45:03.627 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:45:03 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:45:03.627 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:45:03 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T11:45:03.628 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:45:03 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:45:03.628 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:45:03 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:45:03.628 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:45:03 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:45:03.628 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:45:03 smithi152 ceph-mon[36291]: from='osd.5 [v2:172.21.15.154:6808/1472686560,v1:172.21.15.154:6809/1472686560]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-16T11:45:03.628 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:45:03 smithi152 ceph-mon[36291]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-16T11:45:03.628 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:45:03 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T11:45:03.683 DEBUG:teuthology.orchestra.run.smithi154:osd.5> sudo journalctl -f -n 0 -u ceph-d1887f28-e389-11ee-95c9-87774f69a715@osd.5.service 2024-03-16T11:45:03.685 INFO:tasks.cephadm:Deploying osd.6 on smithi029 with /dev/vg_nvme/lv_2... 2024-03-16T11:45:03.685 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d1887f28-e389-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-16T11:45:03.936 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/d1887f28-e389-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T11:45:04.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:45:04 smithi154 ceph-mon[36320]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-03-16T11:45:04.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:45:04 smithi154 ceph-mon[36320]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-16T11:45:04.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:45:04 smithi154 ceph-mon[36320]: osdmap e36: 6 total, 5 up, 6 in 2024-03-16T11:45:04.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:45:04 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-16T11:45:04.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:45:04 smithi154 ceph-mon[36320]: from='osd.5 [v2:172.21.15.154:6808/1472686560,v1:172.21.15.154:6809/1472686560]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]: dispatch 2024-03-16T11:45:04.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:45:04 smithi154 ceph-mon[36320]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]: dispatch 2024-03-16T11:45:04.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:45:04 smithi029 ceph-mon[31975]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-03-16T11:45:04.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:45:04 smithi029 ceph-mon[31975]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-16T11:45:04.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:45:04 smithi029 ceph-mon[31975]: osdmap e36: 6 total, 5 up, 6 in 2024-03-16T11:45:04.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:45:04 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-16T11:45:04.339 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:45:04 smithi029 ceph-mon[31975]: from='osd.5 [v2:172.21.15.154:6808/1472686560,v1:172.21.15.154:6809/1472686560]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]: dispatch 2024-03-16T11:45:04.339 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:45:04 smithi029 ceph-mon[31975]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]: dispatch 2024-03-16T11:45:04.627 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:45:04 smithi152 ceph-mon[36291]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-03-16T11:45:04.627 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:45:04 smithi152 ceph-mon[36291]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-16T11:45:04.628 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:45:04 smithi152 ceph-mon[36291]: osdmap e36: 6 total, 5 up, 6 in 2024-03-16T11:45:04.628 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:45:04 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-16T11:45:04.628 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:45:04 smithi152 ceph-mon[36291]: from='osd.5 [v2:172.21.15.154:6808/1472686560,v1:172.21.15.154:6809/1472686560]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]: dispatch 2024-03-16T11:45:04.628 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:45:04 smithi152 ceph-mon[36291]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]: dispatch 2024-03-16T11:45:05.796 INFO:journalctl@ceph.osd.5.smithi154.stdout:Mar 16 11:45:05 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-5[48458]: 2024-03-16T11:45:05.517+0000 7fc65cb14700 -1 osd.5 0 waiting for initial osdmap 2024-03-16T11:45:05.796 INFO:journalctl@ceph.osd.5.smithi154.stdout:Mar 16 11:45:05 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-5[48458]: 2024-03-16T11:45:05.526+0000 7fc6548ff700 -1 osd.5 37 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-16T11:45:06.221 INFO:teuthology.orchestra.run.smithi029.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-16T11:45:06.221 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-16T11:45:06.221 INFO:teuthology.orchestra.run.smithi029.stderr: stderr: 10+0 records in 2024-03-16T11:45:06.221 INFO:teuthology.orchestra.run.smithi029.stderr:10+0 records out 2024-03-16T11:45:06.221 INFO:teuthology.orchestra.run.smithi029.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00858198 s, 1.2 GB/s 2024-03-16T11:45:06.221 INFO:teuthology.orchestra.run.smithi029.stderr:--> Zapping successful for: 2024-03-16T11:45:06.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:45:06 smithi154 ceph-mon[36320]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]': finished 2024-03-16T11:45:06.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:45:06 smithi154 ceph-mon[36320]: osdmap e37: 6 total, 5 up, 6 in 2024-03-16T11:45:06.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:45:06 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-16T11:45:06.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:45:06 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-16T11:45:06.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:45:06 smithi154 ceph-mon[36320]: from='osd.5 ' entity='osd.5' 2024-03-16T11:45:06.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:45:06 smithi154 ceph-mon[36320]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-03-16T11:45:06.337 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:45:06 smithi029 ceph-mon[31975]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]': finished 2024-03-16T11:45:06.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:45:06 smithi029 ceph-mon[31975]: osdmap e37: 6 total, 5 up, 6 in 2024-03-16T11:45:06.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:45:06 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-16T11:45:06.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:45:06 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-16T11:45:06.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:45:06 smithi029 ceph-mon[31975]: from='osd.5 ' entity='osd.5' 2024-03-16T11:45:06.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:45:06 smithi029 ceph-mon[31975]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-03-16T11:45:06.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:45:06 smithi152 ceph-mon[36291]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]': finished 2024-03-16T11:45:06.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:45:06 smithi152 ceph-mon[36291]: osdmap e37: 6 total, 5 up, 6 in 2024-03-16T11:45:06.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:45:06 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-16T11:45:06.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:45:06 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-16T11:45:06.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:45:06 smithi152 ceph-mon[36291]: from='osd.5 ' entity='osd.5' 2024-03-16T11:45:06.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:45:06 smithi152 ceph-mon[36291]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-03-16T11:45:06.926 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d1887f28-e389-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi029:vg_nvme/lv_2 2024-03-16T11:45:07.179 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/d1887f28-e389-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T11:45:07.301 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:45:07 smithi029 ceph-mon[31975]: purged_snaps scrub starts 2024-03-16T11:45:07.301 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:45:07 smithi029 ceph-mon[31975]: purged_snaps scrub ok 2024-03-16T11:45:07.301 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:45:07 smithi029 ceph-mon[31975]: osd.5 [v2:172.21.15.154:6808/1472686560,v1:172.21.15.154:6809/1472686560] boot 2024-03-16T11:45:07.301 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:45:07 smithi029 ceph-mon[31975]: osdmap e38: 6 total, 6 up, 6 in 2024-03-16T11:45:07.301 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:45:07 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-16T11:45:07.301 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:45:07 smithi029 ceph-mon[31975]: osdmap e39: 6 total, 6 up, 6 in 2024-03-16T11:45:07.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:45:07 smithi152 ceph-mon[36291]: purged_snaps scrub starts 2024-03-16T11:45:07.378 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:45:07 smithi152 ceph-mon[36291]: purged_snaps scrub ok 2024-03-16T11:45:07.378 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:45:07 smithi152 ceph-mon[36291]: osd.5 [v2:172.21.15.154:6808/1472686560,v1:172.21.15.154:6809/1472686560] boot 2024-03-16T11:45:07.378 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:45:07 smithi152 ceph-mon[36291]: osdmap e38: 6 total, 6 up, 6 in 2024-03-16T11:45:07.378 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:45:07 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-16T11:45:07.378 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:45:07 smithi152 ceph-mon[36291]: osdmap e39: 6 total, 6 up, 6 in 2024-03-16T11:45:07.545 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:45:07 smithi154 ceph-mon[36320]: purged_snaps scrub starts 2024-03-16T11:45:07.545 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:45:07 smithi154 ceph-mon[36320]: purged_snaps scrub ok 2024-03-16T11:45:07.545 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:45:07 smithi154 ceph-mon[36320]: osd.5 [v2:172.21.15.154:6808/1472686560,v1:172.21.15.154:6809/1472686560] boot 2024-03-16T11:45:07.545 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:45:07 smithi154 ceph-mon[36320]: osdmap e38: 6 total, 6 up, 6 in 2024-03-16T11:45:07.545 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:45:07 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-16T11:45:07.545 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:45:07 smithi154 ceph-mon[36320]: osdmap e39: 6 total, 6 up, 6 in 2024-03-16T11:45:08.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:45:08 smithi029 ceph-mon[31975]: Detected new or changed devices on smithi154 2024-03-16T11:45:08.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:45:08 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:45:08.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:45:08 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:45:08.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:45:08 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-16T11:45:08.339 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:45:08 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-16T11:45:08.339 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:45:08 smithi029 ceph-mon[31975]: Adjusting osd_memory_target on smithi154 to 8540M 2024-03-16T11:45:08.339 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:45:08 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:45:08.339 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:45:08 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:45:08.339 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:45:08 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T11:45:08.339 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:45:08 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:45:08.339 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:45:08 smithi029 ceph-mon[31975]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-03-16T11:45:08.339 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:45:08 smithi029 ceph-mon[31975]: osdmap e40: 6 total, 6 up, 6 in 2024-03-16T11:45:08.545 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:45:08 smithi154 ceph-mon[36320]: Detected new or changed devices on smithi154 2024-03-16T11:45:08.545 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:45:08 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:45:08.545 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:45:08 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:45:08.545 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:45:08 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-16T11:45:08.545 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:45:08 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-16T11:45:08.545 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:45:08 smithi154 ceph-mon[36320]: Adjusting osd_memory_target on smithi154 to 8540M 2024-03-16T11:45:08.546 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:45:08 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:45:08.546 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:45:08 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:45:08.546 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:45:08 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T11:45:08.546 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:45:08 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:45:08.546 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:45:08 smithi154 ceph-mon[36320]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-03-16T11:45:08.546 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:45:08 smithi154 ceph-mon[36320]: osdmap e40: 6 total, 6 up, 6 in 2024-03-16T11:45:08.627 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:45:08 smithi152 ceph-mon[36291]: Detected new or changed devices on smithi154 2024-03-16T11:45:08.627 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:45:08 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:45:08.628 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:45:08 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:45:08.628 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:45:08 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-16T11:45:08.628 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:45:08 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-16T11:45:08.628 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:45:08 smithi152 ceph-mon[36291]: Adjusting osd_memory_target on smithi154 to 8540M 2024-03-16T11:45:08.628 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:45:08 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:45:08.628 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:45:08 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:45:08.628 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:45:08 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T11:45:08.628 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:45:08 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:45:08.628 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:45:08 smithi152 ceph-mon[36291]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-03-16T11:45:08.628 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:45:08 smithi152 ceph-mon[36291]: osdmap e40: 6 total, 6 up, 6 in 2024-03-16T11:45:09.123 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:45:09.124+0000 7f07f1bfd700 1 -- 172.21.15.29:0/1042270790 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f07ec103520 msgr2=0x7f07ec103980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:45:09.123 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:45:09.124+0000 7f07f1bfd700 1 --2- 172.21.15.29:0/1042270790 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f07ec103520 0x7f07ec103980 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f07e00099c0 tx=0x7f07e0009cd0 comp rx=0 tx=0).stop 2024-03-16T11:45:09.123 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:45:09.124+0000 7f07f1bfd700 1 -- 172.21.15.29:0/1042270790 shutdown_connections 2024-03-16T11:45:09.123 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:45:09.124+0000 7f07f1bfd700 1 --2- 172.21.15.29:0/1042270790 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f07ec103ec0 0x7f07ec10a770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:45:09.124 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:45:09.124+0000 7f07f1bfd700 1 --2- 172.21.15.29:0/1042270790 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f07ec102320 0x7f07ec102740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:45:09.124 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:45:09.124+0000 7f07f1bfd700 1 --2- 172.21.15.29:0/1042270790 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f07ec103520 0x7f07ec103980 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:45:09.124 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:45:09.124+0000 7f07f1bfd700 1 -- 172.21.15.29:0/1042270790 >> 172.21.15.29:0/1042270790 conn(0x7f07ec0fd8e0 msgr2=0x7f07ec0ffd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:45:09.124 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:45:09.124+0000 7f07f1bfd700 1 -- 172.21.15.29:0/1042270790 shutdown_connections 2024-03-16T11:45:09.124 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:45:09.124+0000 7f07f1bfd700 1 -- 172.21.15.29:0/1042270790 wait complete. 2024-03-16T11:45:09.125 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:45:09.124+0000 7f07f1bfd700 1 Processor -- start 2024-03-16T11:45:09.125 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:45:09.125+0000 7f07f1bfd700 1 -- start start 2024-03-16T11:45:09.126 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:45:09.125+0000 7f07f1bfd700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f07ec102320 0x7f07ec198530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:45:09.126 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:45:09.125+0000 7f07f1bfd700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f07ec103520 0x7f07ec198a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:45:09.126 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:45:09.125+0000 7f07f1bfd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f07ec103ec0 0x7f07ec19faf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:45:09.126 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:45:09.125+0000 7f07f1bfd700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f07ec10df10 con 0x7f07ec103ec0 2024-03-16T11:45:09.126 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:45:09.125+0000 7f07f1bfd700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f07ec10dd90 con 0x7f07ec102320 2024-03-16T11:45:09.126 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:45:09.125+0000 7f07f1bfd700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f07ec10e090 con 0x7f07ec103520 2024-03-16T11:45:09.126 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:45:09.125+0000 7f07ebfff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f07ec103ec0 0x7f07ec19faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:45:09.126 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:45:09.125+0000 7f07ebfff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f07ec103ec0 0x7f07ec19faf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46146/0 (socket says 172.21.15.29:46146) 2024-03-16T11:45:09.126 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:45:09.125+0000 7f07ebfff700 1 -- 172.21.15.29:0/2774492872 learned_addr learned my addr 172.21.15.29:0/2774492872 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:45:09.126 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:45:09.125+0000 7f07eaffd700 1 --2- 172.21.15.29:0/2774492872 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f07ec103520 0x7f07ec198a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:45:09.126 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:45:09.125+0000 7f07eb7fe700 1 --2- 172.21.15.29:0/2774492872 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f07ec102320 0x7f07ec198530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:45:09.126 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:45:09.125+0000 7f07ebfff700 1 -- 172.21.15.29:0/2774492872 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f07ec103520 msgr2=0x7f07ec198a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:45:09.127 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:45:09.125+0000 7f07ebfff700 1 --2- 172.21.15.29:0/2774492872 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f07ec103520 0x7f07ec198a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:45:09.127 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:45:09.125+0000 7f07ebfff700 1 -- 172.21.15.29:0/2774492872 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f07ec102320 msgr2=0x7f07ec198530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:45:09.127 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:45:09.125+0000 7f07ebfff700 1 --2- 172.21.15.29:0/2774492872 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f07ec102320 0x7f07ec198530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:45:09.127 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:45:09.125+0000 7f07ebfff700 1 -- 172.21.15.29:0/2774492872 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07ec1a0190 con 0x7f07ec103ec0 2024-03-16T11:45:09.127 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:45:09.125+0000 7f07eb7fe700 1 --2- 172.21.15.29:0/2774492872 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f07ec102320 0x7f07ec198530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T11:45:09.127 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:45:09.125+0000 7f07eaffd700 1 --2- 172.21.15.29:0/2774492872 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f07ec103520 0x7f07ec198a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:45:09.127 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:45:09.126+0000 7f07ebfff700 1 --2- 172.21.15.29:0/2774492872 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f07ec103ec0 0x7f07ec19faf0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f07e400add0 tx=0x7f07e400c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:45:09.127 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:45:09.126+0000 7f07e8ff9700 1 -- 172.21.15.29:0/2774492872 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07e4017020 con 0x7f07ec103ec0 2024-03-16T11:45:09.127 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:45:09.126+0000 7f07f1bfd700 1 -- 172.21.15.29:0/2774492872 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f07ec1a0480 con 0x7f07ec103ec0 2024-03-16T11:45:09.127 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:45:09.126+0000 7f07e8ff9700 1 -- 172.21.15.29:0/2774492872 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f07e400ce20 con 0x7f07ec103ec0 2024-03-16T11:45:09.127 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:45:09.126+0000 7f07e8ff9700 1 -- 172.21.15.29:0/2774492872 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07e40053e0 con 0x7f07ec103ec0 2024-03-16T11:45:09.127 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:45:09.126+0000 7f07f1bfd700 1 -- 172.21.15.29:0/2774492872 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f07ec1a0970 con 0x7f07ec103ec0 2024-03-16T11:45:09.128 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:45:09.127+0000 7f07f1bfd700 1 -- 172.21.15.29:0/2774492872 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f07ec068f90 con 0x7f07ec103ec0 2024-03-16T11:45:09.128 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:45:09.128+0000 7f07e8ff9700 1 -- 172.21.15.29:0/2774492872 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f07e400c940 con 0x7f07ec103ec0 2024-03-16T11:45:09.128 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:45:09.128+0000 7f07e8ff9700 1 --2- 172.21.15.29:0/2774492872 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f07d40a2890 0x7f07d40a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:45:09.128 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:45:09.129+0000 7f07eb7fe700 1 --2- 172.21.15.29:0/2774492872 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f07d40a2890 0x7f07d40a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:45:09.128 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:45:09.129+0000 7f07e8ff9700 1 -- 172.21.15.29:0/2774492872 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 4562+0+0 (secure 0 0 0) 0x7f07e40cd430 con 0x7f07ec103ec0 2024-03-16T11:45:09.129 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:45:09.129+0000 7f07eb7fe700 1 --2- 172.21.15.29:0/2774492872 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f07d40a2890 0x7f07d40a4d50 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f07ec103380 tx=0x7f07dc009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:45:09.132 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:45:09.133+0000 7f07e8ff9700 1 -- 172.21.15.29:0/2774492872 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f07e4021020 con 0x7f07ec103ec0 2024-03-16T11:45:09.350 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:45:09.351+0000 7f07f1bfd700 1 -- 172.21.15.29:0/2774492872 --> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi029:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f07ec062710 con 0x7f07d40a2890 2024-03-16T11:45:10.239 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:45:10 smithi029 ceph-mon[31975]: osdmap e41: 6 total, 6 up, 6 in 2024-03-16T11:45:10.239 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:45:10 smithi029 ceph-mon[31975]: from='client.14370 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi029:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T11:45:10.239 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:45:10 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T11:45:10.239 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:45:10 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T11:45:10.239 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:45:10 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:45:10.240 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:45:10 smithi029 ceph-mon[31975]: pgmap v142: 1 pgs: 1 remapped+peering; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-03-16T11:45:10.446 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:45:10 smithi152 ceph-mon[36291]: osdmap e41: 6 total, 6 up, 6 in 2024-03-16T11:45:10.446 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:45:10 smithi152 ceph-mon[36291]: from='client.14370 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi029:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T11:45:10.446 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:45:10 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T11:45:10.446 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:45:10 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T11:45:10.446 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:45:10 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:45:10.446 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:45:10 smithi152 ceph-mon[36291]: pgmap v142: 1 pgs: 1 remapped+peering; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-03-16T11:45:10.545 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:45:10 smithi154 ceph-mon[36320]: osdmap e41: 6 total, 6 up, 6 in 2024-03-16T11:45:10.545 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:45:10 smithi154 ceph-mon[36320]: from='client.14370 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi029:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T11:45:10.545 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:45:10 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T11:45:10.545 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:45:10 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T11:45:10.545 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:45:10 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:45:10.545 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:45:10 smithi154 ceph-mon[36320]: pgmap v142: 1 pgs: 1 remapped+peering; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-03-16T11:45:11.295 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:45:11 smithi029 ceph-mon[31975]: osdmap e42: 6 total, 6 up, 6 in 2024-03-16T11:45:11.545 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:45:11 smithi154 ceph-mon[36320]: osdmap e42: 6 total, 6 up, 6 in 2024-03-16T11:45:11.621 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:45:11 smithi152 ceph-mon[36291]: osdmap e42: 6 total, 6 up, 6 in 2024-03-16T11:45:12.545 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:45:12 smithi154 ceph-mon[36320]: pgmap v144: 1 pgs: 1 remapped+peering; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail 2024-03-16T11:45:12.588 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:45:12 smithi029 ceph-mon[31975]: pgmap v144: 1 pgs: 1 remapped+peering; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail 2024-03-16T11:45:12.627 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:45:12 smithi152 ceph-mon[36291]: pgmap v144: 1 pgs: 1 remapped+peering; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail 2024-03-16T11:45:13.545 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:45:13 smithi154 ceph-mon[36320]: from='client.? 172.21.15.29:0/312264848' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ae45b8d9-7883-40c9-a9cd-4c92b601d4f2"}]: dispatch 2024-03-16T11:45:13.545 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:45:13 smithi154 ceph-mon[36320]: from='client.? 172.21.15.29:0/312264848' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ae45b8d9-7883-40c9-a9cd-4c92b601d4f2"}]': finished 2024-03-16T11:45:13.545 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:45:13 smithi154 ceph-mon[36320]: osdmap e43: 7 total, 6 up, 7 in 2024-03-16T11:45:13.545 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:45:13 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-16T11:45:13.545 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:45:13 smithi154 ceph-mon[36320]: from='client.? 172.21.15.29:0/2815146849' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T11:45:13.588 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:45:13 smithi029 ceph-mon[31975]: from='client.? 172.21.15.29:0/312264848' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ae45b8d9-7883-40c9-a9cd-4c92b601d4f2"}]: dispatch 2024-03-16T11:45:13.588 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:45:13 smithi029 ceph-mon[31975]: from='client.? 172.21.15.29:0/312264848' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ae45b8d9-7883-40c9-a9cd-4c92b601d4f2"}]': finished 2024-03-16T11:45:13.588 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:45:13 smithi029 ceph-mon[31975]: osdmap e43: 7 total, 6 up, 7 in 2024-03-16T11:45:13.588 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:45:13 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-16T11:45:13.588 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:45:13 smithi029 ceph-mon[31975]: from='client.? 172.21.15.29:0/2815146849' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T11:45:13.627 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:45:13 smithi152 ceph-mon[36291]: from='client.? 172.21.15.29:0/312264848' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ae45b8d9-7883-40c9-a9cd-4c92b601d4f2"}]: dispatch 2024-03-16T11:45:13.627 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:45:13 smithi152 ceph-mon[36291]: from='client.? 172.21.15.29:0/312264848' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ae45b8d9-7883-40c9-a9cd-4c92b601d4f2"}]': finished 2024-03-16T11:45:13.628 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:45:13 smithi152 ceph-mon[36291]: osdmap e43: 7 total, 6 up, 7 in 2024-03-16T11:45:13.628 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:45:13 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-16T11:45:13.628 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:45:13 smithi152 ceph-mon[36291]: from='client.? 172.21.15.29:0/2815146849' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T11:45:14.545 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:45:14 smithi154 ceph-mon[36320]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail; 102 KiB/s, 0 objects/s recovering 2024-03-16T11:45:14.587 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:45:14 smithi029 ceph-mon[31975]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail; 102 KiB/s, 0 objects/s recovering 2024-03-16T11:45:14.627 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:45:14 smithi152 ceph-mon[36291]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail; 102 KiB/s, 0 objects/s recovering 2024-03-16T11:45:16.030 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:45:15 smithi029 ceph-mon[31975]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail; 90 KiB/s, 0 objects/s recovering 2024-03-16T11:45:16.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:45:15 smithi154 ceph-mon[36320]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail; 90 KiB/s, 0 objects/s recovering 2024-03-16T11:45:16.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:45:15 smithi152 ceph-mon[36291]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail; 90 KiB/s, 0 objects/s recovering 2024-03-16T11:45:17.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:45:17 smithi029 ceph-mon[31975]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-16T11:45:18.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:45:17 smithi154 ceph-mon[36320]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-16T11:45:18.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:45:17 smithi152 ceph-mon[36291]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-16T11:45:19.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:45:19 smithi029 ceph-mon[31975]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail; 61 KiB/s, 0 objects/s recovering 2024-03-16T11:45:20.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:45:19 smithi154 ceph-mon[36320]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail; 61 KiB/s, 0 objects/s recovering 2024-03-16T11:45:20.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:45:19 smithi152 ceph-mon[36291]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail; 61 KiB/s, 0 objects/s recovering 2024-03-16T11:45:21.041 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:45:20 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-16T11:45:21.041 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:45:20 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:45:21.041 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:45:20 smithi029 ceph-mon[31975]: Deploying daemon osd.6 on smithi029 2024-03-16T11:45:21.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:45:20 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-16T11:45:21.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:45:20 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:45:21.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:45:20 smithi154 ceph-mon[36320]: Deploying daemon osd.6 on smithi029 2024-03-16T11:45:21.128 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:45:20 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-16T11:45:21.129 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:45:20 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:45:21.129 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:45:20 smithi152 ceph-mon[36291]: Deploying daemon osd.6 on smithi029 2024-03-16T11:45:22.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:45:21 smithi154 ceph-mon[36320]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-16T11:45:22.068 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:45:21 smithi029 ceph-mon[31975]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-16T11:45:22.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:45:21 smithi152 ceph-mon[36291]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-16T11:45:23.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:45:23 smithi029 ceph-mon[31975]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail; 51 KiB/s, 0 objects/s recovering 2024-03-16T11:45:24.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:45:23 smithi154 ceph-mon[36320]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail; 51 KiB/s, 0 objects/s recovering 2024-03-16T11:45:24.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:45:23 smithi152 ceph-mon[36291]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail; 51 KiB/s, 0 objects/s recovering 2024-03-16T11:45:25.731 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:45:25 smithi029 ceph-mon[31975]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail 2024-03-16T11:45:26.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:45:25 smithi154 ceph-mon[36320]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail 2024-03-16T11:45:26.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:45:25 smithi152 ceph-mon[36291]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail 2024-03-16T11:45:27.588 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:45:27 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:45:27.588 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:45:27 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:45:27.588 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:45:27 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T11:45:27.627 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:45:27 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:45:27.627 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:45:27 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:45:27.627 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:45:27 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T11:45:27.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:45:27 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:45:27.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:45:27 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:45:27.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:45:27 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T11:45:28.456 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:45:28 smithi029 ceph-mon[31975]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail 2024-03-16T11:45:28.627 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:45:28 smithi152 ceph-mon[36291]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail 2024-03-16T11:45:28.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:45:28 smithi154 ceph-mon[36320]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail 2024-03-16T11:45:30.377 INFO:teuthology.orchestra.run.smithi029.stdout:Created osd(s) 6 on host 'smithi029' 2024-03-16T11:45:30.377 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:45:30.377+0000 7f07e8ff9700 1 -- 172.21.15.29:0/2774492872 <== mgr.14152 v2:172.21.15.29:6800/4192064640 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f07ec062710 con 0x7f07d40a2890 2024-03-16T11:45:30.379 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:45:30.379+0000 7f07f1bfd700 1 -- 172.21.15.29:0/2774492872 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f07d40a2890 msgr2=0x7f07d40a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:45:30.379 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:45:30.379+0000 7f07f1bfd700 1 --2- 172.21.15.29:0/2774492872 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f07d40a2890 0x7f07d40a4d50 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f07ec103380 tx=0x7f07dc009260 comp rx=0 tx=0).stop 2024-03-16T11:45:30.379 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:45:30.379+0000 7f07f1bfd700 1 -- 172.21.15.29:0/2774492872 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f07ec103ec0 msgr2=0x7f07ec19faf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:45:30.379 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:45:30.379+0000 7f07f1bfd700 1 --2- 172.21.15.29:0/2774492872 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f07ec103ec0 0x7f07ec19faf0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f07e400add0 tx=0x7f07e400c330 comp rx=0 tx=0).stop 2024-03-16T11:45:30.379 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:45:30.379+0000 7f07f1bfd700 1 -- 172.21.15.29:0/2774492872 shutdown_connections 2024-03-16T11:45:30.379 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:45:30.379+0000 7f07f1bfd700 1 --2- 172.21.15.29:0/2774492872 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f07ec103ec0 0x7f07ec19faf0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:45:30.379 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:45:30.379+0000 7f07f1bfd700 1 --2- 172.21.15.29:0/2774492872 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f07ec102320 0x7f07ec198530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:45:30.379 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:45:30.379+0000 7f07f1bfd700 1 --2- 172.21.15.29:0/2774492872 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f07d40a2890 0x7f07d40a4d50 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:45:30.379 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:45:30.379+0000 7f07f1bfd700 1 --2- 172.21.15.29:0/2774492872 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f07ec103520 0x7f07ec198a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:45:30.379 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:45:30.379+0000 7f07f1bfd700 1 -- 172.21.15.29:0/2774492872 >> 172.21.15.29:0/2774492872 conn(0x7f07ec0fd8e0 msgr2=0x7f07ec106820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:45:30.379 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:45:30.380+0000 7f07f1bfd700 1 -- 172.21.15.29:0/2774492872 shutdown_connections 2024-03-16T11:45:30.379 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:45:30.380+0000 7f07f1bfd700 1 -- 172.21.15.29:0/2774492872 wait complete. 2024-03-16T11:45:30.588 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:45:30 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:45:30.588 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:45:30 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:45:30.588 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:45:30 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:45:30.588 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:45:30 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T11:45:30.588 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:45:30 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:45:30.588 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:45:30 smithi029 ceph-mon[31975]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail 2024-03-16T11:45:30.588 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:45:30 smithi029 ceph-mon[31975]: from='osd.6 [v2:172.21.15.29:6818/816506244,v1:172.21.15.29:6819/816506244]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-16T11:45:30.588 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:45:30 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:45:30.588 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:45:30 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:45:30.588 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:45:30 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T11:45:30.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:45:30 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:45:30.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:45:30 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:45:30.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:45:30 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:45:30.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:45:30 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T11:45:30.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:45:30 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:45:30.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:45:30 smithi154 ceph-mon[36320]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail 2024-03-16T11:45:30.796 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:45:30 smithi154 ceph-mon[36320]: from='osd.6 [v2:172.21.15.29:6818/816506244,v1:172.21.15.29:6819/816506244]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-16T11:45:30.796 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:45:30 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:45:30.796 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:45:30 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:45:30.796 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:45:30 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T11:45:30.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:45:30 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:45:30.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:45:30 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:45:30.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:45:30 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:45:30.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:45:30 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T11:45:30.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:45:30 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:45:30.878 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:45:30 smithi152 ceph-mon[36291]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail 2024-03-16T11:45:30.878 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:45:30 smithi152 ceph-mon[36291]: from='osd.6 [v2:172.21.15.29:6818/816506244,v1:172.21.15.29:6819/816506244]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-16T11:45:30.878 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:45:30 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:45:30.878 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:45:30 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:45:30.878 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:45:30 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T11:45:31.123 DEBUG:teuthology.orchestra.run.smithi029:osd.6> sudo journalctl -f -n 0 -u ceph-d1887f28-e389-11ee-95c9-87774f69a715@osd.6.service 2024-03-16T11:45:31.125 INFO:tasks.cephadm:Deploying osd.7 on smithi152 with /dev/vg_nvme/lv_2... 2024-03-16T11:45:31.125 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d1887f28-e389-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-16T11:45:31.372 INFO:teuthology.orchestra.run.smithi152.stderr:Inferring config /var/lib/ceph/d1887f28-e389-11ee-95c9-87774f69a715/mon.b/config 2024-03-16T11:45:31.785 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:45:31 smithi152 ceph-mon[36291]: from='osd.6 [v2:172.21.15.29:6818/816506244,v1:172.21.15.29:6819/816506244]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-16T11:45:31.785 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:45:31 smithi152 ceph-mon[36291]: osdmap e44: 7 total, 6 up, 7 in 2024-03-16T11:45:31.785 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:45:31 smithi152 ceph-mon[36291]: from='osd.6 [v2:172.21.15.29:6818/816506244,v1:172.21.15.29:6819/816506244]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]: dispatch 2024-03-16T11:45:31.785 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:45:31 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-16T11:45:31.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:45:31 smithi154 ceph-mon[36320]: from='osd.6 [v2:172.21.15.29:6818/816506244,v1:172.21.15.29:6819/816506244]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-16T11:45:31.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:45:31 smithi154 ceph-mon[36320]: osdmap e44: 7 total, 6 up, 7 in 2024-03-16T11:45:31.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:45:31 smithi154 ceph-mon[36320]: from='osd.6 [v2:172.21.15.29:6818/816506244,v1:172.21.15.29:6819/816506244]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]: dispatch 2024-03-16T11:45:31.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:45:31 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-16T11:45:31.837 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:45:31 smithi029 ceph-mon[31975]: from='osd.6 [v2:172.21.15.29:6818/816506244,v1:172.21.15.29:6819/816506244]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-16T11:45:31.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:45:31 smithi029 ceph-mon[31975]: osdmap e44: 7 total, 6 up, 7 in 2024-03-16T11:45:31.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:45:31 smithi029 ceph-mon[31975]: from='osd.6 [v2:172.21.15.29:6818/816506244,v1:172.21.15.29:6819/816506244]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]: dispatch 2024-03-16T11:45:31.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:45:31 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-16T11:45:32.325 INFO:journalctl@ceph.osd.6.smithi029.stdout:Mar 16 11:45:31 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-6[60123]: 2024-03-16T11:45:31.947+0000 7fb560e01700 -1 osd.6 0 waiting for initial osdmap 2024-03-16T11:45:32.325 INFO:journalctl@ceph.osd.6.smithi029.stdout:Mar 16 11:45:31 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-6[60123]: 2024-03-16T11:45:31.955+0000 7fb558bec700 -1 osd.6 45 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-16T11:45:32.588 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:45:32 smithi029 ceph-mon[31975]: from='osd.6 [v2:172.21.15.29:6818/816506244,v1:172.21.15.29:6819/816506244]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]': finished 2024-03-16T11:45:32.732 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:45:32 smithi152 ceph-mon[36291]: from='osd.6 [v2:172.21.15.29:6818/816506244,v1:172.21.15.29:6819/816506244]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]': finished 2024-03-16T11:45:32.732 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:45:32 smithi152 ceph-mon[36291]: osdmap e45: 7 total, 6 up, 7 in 2024-03-16T11:45:32.732 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:45:32 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-16T11:45:32.732 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:45:32 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-16T11:45:32.732 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:45:32 smithi152 ceph-mon[36291]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail 2024-03-16T11:45:32.732 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:45:32 smithi152 ceph-mon[36291]: from='osd.6 [v2:172.21.15.29:6818/816506244,v1:172.21.15.29:6819/816506244]' entity='osd.6' 2024-03-16T11:45:32.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:45:32 smithi154 ceph-mon[36320]: from='osd.6 [v2:172.21.15.29:6818/816506244,v1:172.21.15.29:6819/816506244]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]': finished 2024-03-16T11:45:32.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:45:32 smithi154 ceph-mon[36320]: osdmap e45: 7 total, 6 up, 7 in 2024-03-16T11:45:32.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:45:32 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-16T11:45:32.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:45:32 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-16T11:45:32.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:45:32 smithi154 ceph-mon[36320]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail 2024-03-16T11:45:32.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:45:32 smithi154 ceph-mon[36320]: from='osd.6 [v2:172.21.15.29:6818/816506244,v1:172.21.15.29:6819/816506244]' entity='osd.6' 2024-03-16T11:45:33.057 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:45:32 smithi029 ceph-mon[31975]: osdmap e45: 7 total, 6 up, 7 in 2024-03-16T11:45:33.057 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:45:32 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-16T11:45:33.057 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:45:32 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-16T11:45:33.057 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:45:32 smithi029 ceph-mon[31975]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail 2024-03-16T11:45:33.057 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:45:32 smithi029 ceph-mon[31975]: from='osd.6 [v2:172.21.15.29:6818/816506244,v1:172.21.15.29:6819/816506244]' entity='osd.6' 2024-03-16T11:45:33.507 INFO:teuthology.orchestra.run.smithi152.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-16T11:45:33.507 INFO:teuthology.orchestra.run.smithi152.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-16T11:45:33.507 INFO:teuthology.orchestra.run.smithi152.stderr: stderr: 10+0 records in 2024-03-16T11:45:33.507 INFO:teuthology.orchestra.run.smithi152.stderr:10+0 records out 2024-03-16T11:45:33.507 INFO:teuthology.orchestra.run.smithi152.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0185511 s, 565 MB/s 2024-03-16T11:45:33.508 INFO:teuthology.orchestra.run.smithi152.stderr:--> Zapping successful for: 2024-03-16T11:45:33.588 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:45:33 smithi029 ceph-mon[31975]: purged_snaps scrub starts 2024-03-16T11:45:33.588 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:45:33 smithi029 ceph-mon[31975]: purged_snaps scrub ok 2024-03-16T11:45:33.588 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:45:33 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-16T11:45:33.588 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:45:33 smithi029 ceph-mon[31975]: osd.6 [v2:172.21.15.29:6818/816506244,v1:172.21.15.29:6819/816506244] boot 2024-03-16T11:45:33.588 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:45:33 smithi029 ceph-mon[31975]: osdmap e46: 7 total, 7 up, 7 in 2024-03-16T11:45:33.588 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:45:33 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-16T11:45:33.627 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:45:33 smithi152 ceph-mon[36291]: purged_snaps scrub starts 2024-03-16T11:45:33.627 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:45:33 smithi152 ceph-mon[36291]: purged_snaps scrub ok 2024-03-16T11:45:33.627 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:45:33 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-16T11:45:33.627 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:45:33 smithi152 ceph-mon[36291]: osd.6 [v2:172.21.15.29:6818/816506244,v1:172.21.15.29:6819/816506244] boot 2024-03-16T11:45:33.627 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:45:33 smithi152 ceph-mon[36291]: osdmap e46: 7 total, 7 up, 7 in 2024-03-16T11:45:33.627 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:45:33 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-16T11:45:33.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:45:33 smithi154 ceph-mon[36320]: purged_snaps scrub starts 2024-03-16T11:45:33.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:45:33 smithi154 ceph-mon[36320]: purged_snaps scrub ok 2024-03-16T11:45:33.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:45:33 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-16T11:45:33.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:45:33 smithi154 ceph-mon[36320]: osd.6 [v2:172.21.15.29:6818/816506244,v1:172.21.15.29:6819/816506244] boot 2024-03-16T11:45:33.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:45:33 smithi154 ceph-mon[36320]: osdmap e46: 7 total, 7 up, 7 in 2024-03-16T11:45:33.796 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:45:33 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-16T11:45:34.216 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d1887f28-e389-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi152:vg_nvme/lv_2 2024-03-16T11:45:34.454 INFO:teuthology.orchestra.run.smithi152.stderr:Inferring config /var/lib/ceph/d1887f28-e389-11ee-95c9-87774f69a715/mon.b/config 2024-03-16T11:45:34.780 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:45:34 smithi029 ceph-mon[31975]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-03-16T11:45:34.781 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:45:34 smithi029 ceph-mon[31975]: osdmap e47: 7 total, 7 up, 7 in 2024-03-16T11:45:34.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:45:34 smithi154 ceph-mon[36320]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-03-16T11:45:34.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:45:34 smithi154 ceph-mon[36320]: osdmap e47: 7 total, 7 up, 7 in 2024-03-16T11:45:34.835 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:45:34 smithi152 ceph-mon[36291]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-03-16T11:45:34.835 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:45:34 smithi152 ceph-mon[36291]: osdmap e47: 7 total, 7 up, 7 in 2024-03-16T11:45:36.038 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-16T11:45:36.038+0000 7f59b315a700 1 -- 172.21.15.152:0/1305426241 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f59ac103f90 msgr2=0x7f59ac10a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:45:36.038 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-16T11:45:36.038+0000 7f59b315a700 1 --2- 172.21.15.152:0/1305426241 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f59ac103f90 0x7f59ac10a840 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f59a400b0a0 tx=0x7f59a400b3b0 comp rx=0 tx=0).stop 2024-03-16T11:45:36.038 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-16T11:45:36.038+0000 7f59b315a700 1 -- 172.21.15.152:0/1305426241 shutdown_connections 2024-03-16T11:45:36.038 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-16T11:45:36.038+0000 7f59b315a700 1 --2- 172.21.15.152:0/1305426241 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f59ac103f90 0x7f59ac10a840 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:45:36.039 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-16T11:45:36.038+0000 7f59b315a700 1 --2- 172.21.15.152:0/1305426241 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f59ac100610 0x7f59ac100a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:45:36.039 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-16T11:45:36.038+0000 7f59b315a700 1 --2- 172.21.15.152:0/1305426241 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f59ac1035f0 0x7f59ac103a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:45:36.039 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-16T11:45:36.038+0000 7f59b315a700 1 -- 172.21.15.152:0/1305426241 >> 172.21.15.152:0/1305426241 conn(0x7f59ac0fbf70 msgr2=0x7f59ac0fe3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:45:36.039 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-16T11:45:36.039+0000 7f59b315a700 1 -- 172.21.15.152:0/1305426241 shutdown_connections 2024-03-16T11:45:36.039 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-16T11:45:36.039+0000 7f59b315a700 1 -- 172.21.15.152:0/1305426241 wait complete. 2024-03-16T11:45:36.041 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-16T11:45:36.039+0000 7f59b315a700 1 Processor -- start 2024-03-16T11:45:36.041 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-16T11:45:36.039+0000 7f59b315a700 1 -- start start 2024-03-16T11:45:36.041 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-16T11:45:36.039+0000 7f59b315a700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f59ac100610 0x7f59ac198480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:45:36.041 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-16T11:45:36.039+0000 7f59b315a700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f59ac1035f0 0x7f59ac1989c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:45:36.041 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-16T11:45:36.040+0000 7f59b315a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f59ac103f90 0x7f59ac19fa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:45:36.041 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-16T11:45:36.040+0000 7f59b315a700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f59ac10df30 con 0x7f59ac103f90 2024-03-16T11:45:36.041 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-16T11:45:36.040+0000 7f59b315a700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f59ac10ddb0 con 0x7f59ac1035f0 2024-03-16T11:45:36.041 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-16T11:45:36.040+0000 7f59b315a700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f59ac10e0b0 con 0x7f59ac100610 2024-03-16T11:45:36.041 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-16T11:45:36.040+0000 7f59abfff700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f59ac1035f0 0x7f59ac1989c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:45:36.041 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-16T11:45:36.040+0000 7f59abfff700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f59ac1035f0 0x7f59ac1989c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.152:45152/0 (socket says 172.21.15.152:45152) 2024-03-16T11:45:36.042 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-16T11:45:36.040+0000 7f59abfff700 1 -- 172.21.15.152:0/4029973996 learned_addr learned my addr 172.21.15.152:0/4029973996 (peer_addr_for_me v2:172.21.15.152:0/0) 2024-03-16T11:45:36.042 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-16T11:45:36.040+0000 7f59b0ef6700 1 --2- 172.21.15.152:0/4029973996 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f59ac100610 0x7f59ac198480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:45:36.042 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-16T11:45:36.040+0000 7f59b16f7700 1 --2- 172.21.15.152:0/4029973996 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f59ac103f90 0x7f59ac19fa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:45:36.042 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-16T11:45:36.040+0000 7f59abfff700 1 -- 172.21.15.152:0/4029973996 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f59ac100610 msgr2=0x7f59ac198480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:45:36.042 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-16T11:45:36.040+0000 7f59abfff700 1 --2- 172.21.15.152:0/4029973996 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f59ac100610 0x7f59ac198480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:45:36.042 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-16T11:45:36.040+0000 7f59abfff700 1 -- 172.21.15.152:0/4029973996 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f59ac103f90 msgr2=0x7f59ac19fa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:45:36.042 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-16T11:45:36.040+0000 7f59abfff700 1 --2- 172.21.15.152:0/4029973996 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f59ac103f90 0x7f59ac19fa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:45:36.042 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-16T11:45:36.040+0000 7f59abfff700 1 -- 172.21.15.152:0/4029973996 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59ac1a00e0 con 0x7f59ac1035f0 2024-03-16T11:45:36.042 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-16T11:45:36.040+0000 7f59abfff700 1 --2- 172.21.15.152:0/4029973996 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f59ac1035f0 0x7f59ac1989c0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f599800d980 tx=0x7f599800dc90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:45:36.042 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-16T11:45:36.040+0000 7f59a9ffb700 1 -- 172.21.15.152:0/4029973996 <== mon.2 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5998017070 con 0x7f59ac1035f0 2024-03-16T11:45:36.042 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-16T11:45:36.040+0000 7f59b315a700 1 -- 172.21.15.152:0/4029973996 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f59ac1a03d0 con 0x7f59ac1035f0 2024-03-16T11:45:36.042 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-16T11:45:36.040+0000 7f59a9ffb700 1 -- 172.21.15.152:0/4029973996 <== mon.2 v2:172.21.15.152:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5998005020 con 0x7f59ac1035f0 2024-03-16T11:45:36.042 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-16T11:45:36.040+0000 7f59a9ffb700 1 -- 172.21.15.152:0/4029973996 <== mon.2 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f599800fbf0 con 0x7f59ac1035f0 2024-03-16T11:45:36.042 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-16T11:45:36.041+0000 7f59b315a700 1 -- 172.21.15.152:0/4029973996 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f59ac1a0910 con 0x7f59ac1035f0 2024-03-16T11:45:36.042 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-16T11:45:36.042+0000 7f59b315a700 1 -- 172.21.15.152:0/4029973996 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f59ac068f90 con 0x7f59ac1035f0 2024-03-16T11:45:36.043 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-16T11:45:36.043+0000 7f59a9ffb700 1 -- 172.21.15.152:0/4029973996 <== mon.2 v2:172.21.15.152:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f59980051d0 con 0x7f59ac1035f0 2024-03-16T11:45:36.043 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-16T11:45:36.043+0000 7f59a9ffb700 1 --2- 172.21.15.152:0/4029973996 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f599c0a2890 0x7f599c0a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:45:36.043 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-16T11:45:36.043+0000 7f59a9ffb700 1 -- 172.21.15.152:0/4029973996 <== mon.2 v2:172.21.15.152:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 4965+0+0 (secure 0 0 0) 0x7f59980cdf90 con 0x7f59ac1035f0 2024-03-16T11:45:36.044 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-16T11:45:36.043+0000 7f59b0ef6700 1 --2- 172.21.15.152:0/4029973996 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f599c0a2890 0x7f599c0a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:45:36.044 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-16T11:45:36.044+0000 7f59b0ef6700 1 --2- 172.21.15.152:0/4029973996 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f599c0a2890 0x7f599c0a4d50 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f59ac06a7f0 tx=0x7f59a000b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:45:36.046 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-16T11:45:36.047+0000 7f59a9ffb700 1 -- 172.21.15.152:0/4029973996 <== mon.2 v2:172.21.15.152:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f5998020020 con 0x7f59ac1035f0 2024-03-16T11:45:36.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:45:35 smithi152 ceph-mon[36291]: Detected new or changed devices on smithi029 2024-03-16T11:45:36.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:45:35 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:45:36.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:45:35 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:45:36.128 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:45:35 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-16T11:45:36.128 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:45:35 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-16T11:45:36.128 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:45:35 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-16T11:45:36.128 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:45:35 smithi152 ceph-mon[36291]: Adjusting osd_memory_target on smithi029 to 5697M 2024-03-16T11:45:36.128 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:45:35 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:45:36.128 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:45:35 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:45:36.128 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:45:35 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T11:45:36.128 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:45:35 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:45:36.128 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:45:35 smithi152 ceph-mon[36291]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-03-16T11:45:36.250 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-16T11:45:36.250+0000 7f59b315a700 1 -- 172.21.15.152:0/4029973996 --> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi152:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f59ac0627d0 con 0x7f599c0a2890 2024-03-16T11:45:36.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:45:35 smithi154 ceph-mon[36320]: Detected new or changed devices on smithi029 2024-03-16T11:45:36.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:45:35 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:45:36.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:45:35 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:45:36.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:45:35 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-16T11:45:36.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:45:35 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-16T11:45:36.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:45:35 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-16T11:45:36.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:45:35 smithi154 ceph-mon[36320]: Adjusting osd_memory_target on smithi029 to 5697M 2024-03-16T11:45:36.296 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:45:35 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:45:36.296 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:45:35 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:45:36.296 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:45:35 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T11:45:36.296 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:45:35 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:45:36.296 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:45:35 smithi154 ceph-mon[36320]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-03-16T11:45:36.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:45:35 smithi029 ceph-mon[31975]: Detected new or changed devices on smithi029 2024-03-16T11:45:36.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:45:35 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:45:36.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:45:35 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:45:36.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:45:35 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-16T11:45:36.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:45:35 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-16T11:45:36.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:45:35 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-16T11:45:36.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:45:35 smithi029 ceph-mon[31975]: Adjusting osd_memory_target on smithi029 to 5697M 2024-03-16T11:45:36.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:45:35 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:45:36.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:45:35 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:45:36.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:45:35 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T11:45:36.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:45:35 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:45:36.339 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:45:35 smithi029 ceph-mon[31975]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-03-16T11:45:37.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:45:36 smithi152 ceph-mon[36291]: from='client.24266 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi152:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T11:45:37.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:45:36 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T11:45:37.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:45:36 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T11:45:37.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:45:36 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:45:37.197 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:45:36 smithi154 ceph-mon[36320]: from='client.24266 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi152:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T11:45:37.197 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:45:36 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T11:45:37.197 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:45:36 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T11:45:37.197 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:45:36 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:45:37.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:45:36 smithi029 ceph-mon[31975]: from='client.24266 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi152:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T11:45:37.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:45:36 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T11:45:37.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:45:36 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T11:45:37.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:45:36 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:45:38.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:45:37 smithi152 ceph-mon[36291]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-03-16T11:45:38.211 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:45:37 smithi154 ceph-mon[36320]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-03-16T11:45:38.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:45:37 smithi029 ceph-mon[31975]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-03-16T11:45:39.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:45:39 smithi029 ceph-mon[31975]: from='client.? 172.21.15.152:0/2016305260' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6f687b28-4fce-4d4a-8b4e-294173d9d6c5"}]: dispatch 2024-03-16T11:45:39.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:45:39 smithi029 ceph-mon[31975]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6f687b28-4fce-4d4a-8b4e-294173d9d6c5"}]: dispatch 2024-03-16T11:45:39.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:45:39 smithi029 ceph-mon[31975]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6f687b28-4fce-4d4a-8b4e-294173d9d6c5"}]': finished 2024-03-16T11:45:39.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:45:39 smithi029 ceph-mon[31975]: osdmap e48: 8 total, 7 up, 8 in 2024-03-16T11:45:39.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:45:39 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-16T11:45:39.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:45:39 smithi152 ceph-mon[36291]: from='client.? 172.21.15.152:0/2016305260' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6f687b28-4fce-4d4a-8b4e-294173d9d6c5"}]: dispatch 2024-03-16T11:45:39.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:45:39 smithi152 ceph-mon[36291]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6f687b28-4fce-4d4a-8b4e-294173d9d6c5"}]: dispatch 2024-03-16T11:45:39.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:45:39 smithi152 ceph-mon[36291]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6f687b28-4fce-4d4a-8b4e-294173d9d6c5"}]': finished 2024-03-16T11:45:39.378 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:45:39 smithi152 ceph-mon[36291]: osdmap e48: 8 total, 7 up, 8 in 2024-03-16T11:45:39.378 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:45:39 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-16T11:45:39.391 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:45:39 smithi154 ceph-mon[36320]: from='client.? 172.21.15.152:0/2016305260' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6f687b28-4fce-4d4a-8b4e-294173d9d6c5"}]: dispatch 2024-03-16T11:45:39.391 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:45:39 smithi154 ceph-mon[36320]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6f687b28-4fce-4d4a-8b4e-294173d9d6c5"}]: dispatch 2024-03-16T11:45:39.391 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:45:39 smithi154 ceph-mon[36320]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6f687b28-4fce-4d4a-8b4e-294173d9d6c5"}]': finished 2024-03-16T11:45:39.391 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:45:39 smithi154 ceph-mon[36320]: osdmap e48: 8 total, 7 up, 8 in 2024-03-16T11:45:39.391 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:45:39 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-16T11:45:40.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:45:40 smithi029 ceph-mon[31975]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-03-16T11:45:40.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:45:40 smithi029 ceph-mon[31975]: from='client.? 172.21.15.152:0/3029908853' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T11:45:40.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:45:40 smithi152 ceph-mon[36291]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-03-16T11:45:40.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:45:40 smithi152 ceph-mon[36291]: from='client.? 172.21.15.152:0/3029908853' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T11:45:40.545 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:45:40 smithi154 ceph-mon[36320]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-03-16T11:45:40.545 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:45:40 smithi154 ceph-mon[36320]: from='client.? 172.21.15.152:0/3029908853' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T11:45:42.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:45:41 smithi154 ceph-mon[36320]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-03-16T11:45:42.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:45:41 smithi029 ceph-mon[31975]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-03-16T11:45:42.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:45:41 smithi152 ceph-mon[36291]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-03-16T11:45:43.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:45:43 smithi152 ceph-mon[36291]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-03-16T11:45:44.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:45:43 smithi154 ceph-mon[36320]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-03-16T11:45:44.087 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:45:43 smithi029 ceph-mon[31975]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-03-16T11:45:46.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:45:45 smithi154 ceph-mon[36320]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-03-16T11:45:46.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:45:45 smithi029 ceph-mon[31975]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-03-16T11:45:46.110 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:45:45 smithi152 ceph-mon[36291]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-03-16T11:45:46.878 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:45:46 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-16T11:45:46.878 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:45:46 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:45:46.878 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:45:46 smithi152 ceph-mon[36291]: Deploying daemon osd.7 on smithi152 2024-03-16T11:45:47.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:45:46 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-16T11:45:47.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:45:46 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:45:47.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:45:46 smithi154 ceph-mon[36320]: Deploying daemon osd.7 on smithi152 2024-03-16T11:45:47.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:45:46 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-16T11:45:47.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:45:46 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:45:47.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:45:46 smithi029 ceph-mon[31975]: Deploying daemon osd.7 on smithi152 2024-03-16T11:45:47.987 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:45:47 smithi152 ceph-mon[36291]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-03-16T11:45:48.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:45:47 smithi154 ceph-mon[36320]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-03-16T11:45:48.087 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:45:47 smithi029 ceph-mon[31975]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-03-16T11:45:49.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:45:49 smithi154 ceph-mon[36320]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-03-16T11:45:49.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:45:49 smithi152 ceph-mon[36291]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-03-16T11:45:50.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:45:49 smithi029 ceph-mon[31975]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-03-16T11:45:51.838 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:45:51 smithi152 ceph-mon[36291]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-03-16T11:45:52.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:45:51 smithi154 ceph-mon[36320]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-03-16T11:45:52.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:45:51 smithi029 ceph-mon[31975]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-03-16T11:45:53.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:45:53 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:45:53.378 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:45:53 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:45:53.378 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:45:53 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T11:45:53.545 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:45:53 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:45:53.545 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:45:53 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:45:53.545 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:45:53 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T11:45:53.587 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:45:53 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:45:53.588 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:45:53 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:45:53.588 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:45:53 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T11:45:54.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:45:54 smithi152 ceph-mon[36291]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-03-16T11:45:54.545 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:45:54 smithi154 ceph-mon[36320]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-03-16T11:45:54.588 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:45:54 smithi029 ceph-mon[31975]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-03-16T11:45:55.918 INFO:teuthology.orchestra.run.smithi152.stdout:Created osd(s) 7 on host 'smithi152' 2024-03-16T11:45:55.919 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-16T11:45:55.918+0000 7f59a9ffb700 1 -- 172.21.15.152:0/4029973996 <== mgr.14152 v2:172.21.15.29:6800/4192064640 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f59ac0627d0 con 0x7f599c0a2890 2024-03-16T11:45:55.921 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-16T11:45:55.921+0000 7f59b315a700 1 -- 172.21.15.152:0/4029973996 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f599c0a2890 msgr2=0x7f599c0a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:45:55.921 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-16T11:45:55.921+0000 7f59b315a700 1 --2- 172.21.15.152:0/4029973996 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f599c0a2890 0x7f599c0a4d50 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f59ac06a7f0 tx=0x7f59a000b410 comp rx=0 tx=0).stop 2024-03-16T11:45:55.921 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-16T11:45:55.921+0000 7f59b315a700 1 -- 172.21.15.152:0/4029973996 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f59ac1035f0 msgr2=0x7f59ac1989c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:45:55.921 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-16T11:45:55.921+0000 7f59b315a700 1 --2- 172.21.15.152:0/4029973996 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f59ac1035f0 0x7f59ac1989c0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f599800d980 tx=0x7f599800dc90 comp rx=0 tx=0).stop 2024-03-16T11:45:55.921 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-16T11:45:55.921+0000 7f59b315a700 1 -- 172.21.15.152:0/4029973996 shutdown_connections 2024-03-16T11:45:55.921 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-16T11:45:55.921+0000 7f59b315a700 1 --2- 172.21.15.152:0/4029973996 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f59ac103f90 0x7f59ac19fa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:45:55.921 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-16T11:45:55.921+0000 7f59b315a700 1 --2- 172.21.15.152:0/4029973996 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f599c0a2890 0x7f599c0a4d50 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:45:55.921 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-16T11:45:55.921+0000 7f59b315a700 1 --2- 172.21.15.152:0/4029973996 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f59ac100610 0x7f59ac198480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:45:55.921 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-16T11:45:55.921+0000 7f59b315a700 1 --2- 172.21.15.152:0/4029973996 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f59ac1035f0 0x7f59ac1989c0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:45:55.921 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-16T11:45:55.921+0000 7f59b315a700 1 -- 172.21.15.152:0/4029973996 >> 172.21.15.152:0/4029973996 conn(0x7f59ac0fbf70 msgr2=0x7f59ac106820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:45:55.921 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-16T11:45:55.921+0000 7f59b315a700 1 -- 172.21.15.152:0/4029973996 shutdown_connections 2024-03-16T11:45:55.921 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-16T11:45:55.921+0000 7f59b315a700 1 -- 172.21.15.152:0/4029973996 wait complete. 2024-03-16T11:45:56.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:45:56 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:45:56.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:45:56 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:45:56.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:45:56 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:45:56.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:45:56 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T11:45:56.128 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:45:56 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:45:56.128 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:45:56 smithi152 ceph-mon[36291]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-03-16T11:45:56.128 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:45:56 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:45:56.128 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:45:56 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:45:56.128 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:45:56 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T11:45:56.128 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:45:56 smithi152 ceph-mon[36291]: from='osd.7 [v2:172.21.15.152:6816/2924651787,v1:172.21.15.152:6817/2924651787]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-16T11:45:56.128 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:45:56 smithi152 ceph-mon[36291]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-16T11:45:56.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:45:56 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:45:56.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:45:56 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:45:56.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:45:56 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:45:56.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:45:56 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T11:45:56.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:45:56 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:45:56.296 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:45:56 smithi154 ceph-mon[36320]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-03-16T11:45:56.296 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:45:56 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:45:56.296 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:45:56 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:45:56.296 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:45:56 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T11:45:56.296 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:45:56 smithi154 ceph-mon[36320]: from='osd.7 [v2:172.21.15.152:6816/2924651787,v1:172.21.15.152:6817/2924651787]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-16T11:45:56.296 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:45:56 smithi154 ceph-mon[36320]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-16T11:45:56.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:45:56 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:45:56.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:45:56 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:45:56.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:45:56 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:45:56.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:45:56 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T11:45:56.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:45:56 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:45:56.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:45:56 smithi029 ceph-mon[31975]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-03-16T11:45:56.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:45:56 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:45:56.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:45:56 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:45:56.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:45:56 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T11:45:56.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:45:56 smithi029 ceph-mon[31975]: from='osd.7 [v2:172.21.15.152:6816/2924651787,v1:172.21.15.152:6817/2924651787]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-16T11:45:56.339 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:45:56 smithi029 ceph-mon[31975]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-16T11:45:56.629 DEBUG:teuthology.orchestra.run.smithi152:osd.7> sudo journalctl -f -n 0 -u ceph-d1887f28-e389-11ee-95c9-87774f69a715@osd.7.service 2024-03-16T11:45:56.631 INFO:tasks.cephadm:Deploying osd.8 on smithi154 with /dev/vg_nvme/lv_2... 2024-03-16T11:45:56.631 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d1887f28-e389-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-16T11:45:56.878 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/d1887f28-e389-11ee-95c9-87774f69a715/mon.c/config 2024-03-16T11:45:58.187 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:45:57 smithi154 ceph-mon[36320]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-16T11:45:58.187 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:45:57 smithi154 ceph-mon[36320]: osdmap e49: 8 total, 7 up, 8 in 2024-03-16T11:45:58.187 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:45:57 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-16T11:45:58.187 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:45:57 smithi154 ceph-mon[36320]: from='osd.7 [v2:172.21.15.152:6816/2924651787,v1:172.21.15.152:6817/2924651787]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]: dispatch 2024-03-16T11:45:58.187 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:45:57 smithi154 ceph-mon[36320]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]: dispatch 2024-03-16T11:45:58.187 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:45:57 smithi154 ceph-mon[36320]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-03-16T11:45:58.188 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:45:57 smithi152 ceph-mon[36291]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-16T11:45:58.188 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:45:57 smithi152 ceph-mon[36291]: osdmap e49: 8 total, 7 up, 8 in 2024-03-16T11:45:58.188 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:45:57 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-16T11:45:58.188 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:45:57 smithi152 ceph-mon[36291]: from='osd.7 [v2:172.21.15.152:6816/2924651787,v1:172.21.15.152:6817/2924651787]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]: dispatch 2024-03-16T11:45:58.188 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:45:57 smithi152 ceph-mon[36291]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]: dispatch 2024-03-16T11:45:58.188 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:45:57 smithi152 ceph-mon[36291]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-03-16T11:45:58.189 INFO:journalctl@ceph.osd.7.smithi152.stdout:Mar 16 11:45:57 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-7[54789]: 2024-03-16T11:45:57.922+0000 7f6c63651700 -1 osd.7 0 waiting for initial osdmap 2024-03-16T11:45:58.189 INFO:journalctl@ceph.osd.7.smithi152.stdout:Mar 16 11:45:57 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-7[54789]: 2024-03-16T11:45:57.931+0000 7f6c5f444700 -1 osd.7 50 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-16T11:45:58.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:45:57 smithi029 ceph-mon[31975]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-16T11:45:58.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:45:57 smithi029 ceph-mon[31975]: osdmap e49: 8 total, 7 up, 8 in 2024-03-16T11:45:58.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:45:57 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-16T11:45:58.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:45:57 smithi029 ceph-mon[31975]: from='osd.7 [v2:172.21.15.152:6816/2924651787,v1:172.21.15.152:6817/2924651787]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]: dispatch 2024-03-16T11:45:58.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:45:57 smithi029 ceph-mon[31975]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]: dispatch 2024-03-16T11:45:58.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:45:57 smithi029 ceph-mon[31975]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-03-16T11:45:58.915 INFO:teuthology.orchestra.run.smithi154.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-16T11:45:58.915 INFO:teuthology.orchestra.run.smithi154.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-16T11:45:58.916 INFO:teuthology.orchestra.run.smithi154.stderr: stderr: 10+0 records in 2024-03-16T11:45:58.916 INFO:teuthology.orchestra.run.smithi154.stderr:10+0 records out 2024-03-16T11:45:58.916 INFO:teuthology.orchestra.run.smithi154.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0132686 s, 790 MB/s 2024-03-16T11:45:58.916 INFO:teuthology.orchestra.run.smithi154.stderr:--> Zapping successful for: 2024-03-16T11:45:59.259 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:45:58 smithi154 ceph-mon[36320]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]': finished 2024-03-16T11:45:59.259 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:45:58 smithi154 ceph-mon[36320]: osdmap e50: 8 total, 7 up, 8 in 2024-03-16T11:45:59.259 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:45:58 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-16T11:45:59.259 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:45:58 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-16T11:45:59.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:45:58 smithi029 ceph-mon[31975]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]': finished 2024-03-16T11:45:59.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:45:58 smithi029 ceph-mon[31975]: osdmap e50: 8 total, 7 up, 8 in 2024-03-16T11:45:59.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:45:58 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-16T11:45:59.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:45:58 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-16T11:45:59.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:45:58 smithi152 ceph-mon[36291]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]': finished 2024-03-16T11:45:59.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:45:58 smithi152 ceph-mon[36291]: osdmap e50: 8 total, 7 up, 8 in 2024-03-16T11:45:59.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:45:58 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-16T11:45:59.378 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:45:58 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-16T11:45:59.599 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d1887f28-e389-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi154:vg_nvme/lv_2 2024-03-16T11:45:59.846 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/d1887f28-e389-11ee-95c9-87774f69a715/mon.c/config 2024-03-16T11:46:00.124 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:45:59 smithi154 ceph-mon[36320]: purged_snaps scrub starts 2024-03-16T11:46:00.124 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:45:59 smithi154 ceph-mon[36320]: purged_snaps scrub ok 2024-03-16T11:46:00.124 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:45:59 smithi154 ceph-mon[36320]: osd.7 [v2:172.21.15.152:6816/2924651787,v1:172.21.15.152:6817/2924651787] boot 2024-03-16T11:46:00.124 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:45:59 smithi154 ceph-mon[36320]: osdmap e51: 8 total, 8 up, 8 in 2024-03-16T11:46:00.124 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:45:59 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-16T11:46:00.124 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:45:59 smithi154 ceph-mon[36320]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 715 GiB / 715 GiB avail 2024-03-16T11:46:00.124 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:45:59 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:46:00.124 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:45:59 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:46:00.124 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:45:59 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-16T11:46:00.124 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:45:59 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-16T11:46:00.124 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:45:59 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-16T11:46:00.124 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:45:59 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:46:00.124 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:45:59 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:46:00.125 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:45:59 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T11:46:00.125 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:45:59 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:46:00.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:45:59 smithi152 ceph-mon[36291]: purged_snaps scrub starts 2024-03-16T11:46:00.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:45:59 smithi152 ceph-mon[36291]: purged_snaps scrub ok 2024-03-16T11:46:00.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:45:59 smithi152 ceph-mon[36291]: osd.7 [v2:172.21.15.152:6816/2924651787,v1:172.21.15.152:6817/2924651787] boot 2024-03-16T11:46:00.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:45:59 smithi152 ceph-mon[36291]: osdmap e51: 8 total, 8 up, 8 in 2024-03-16T11:46:00.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:45:59 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-16T11:46:00.128 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:45:59 smithi152 ceph-mon[36291]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 715 GiB / 715 GiB avail 2024-03-16T11:46:00.128 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:45:59 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:46:00.128 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:45:59 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:46:00.128 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:45:59 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-16T11:46:00.128 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:45:59 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-16T11:46:00.128 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:45:59 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-16T11:46:00.128 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:45:59 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:46:00.128 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:45:59 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:46:00.128 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:45:59 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T11:46:00.128 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:45:59 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:46:00.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:45:59 smithi029 ceph-mon[31975]: purged_snaps scrub starts 2024-03-16T11:46:00.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:45:59 smithi029 ceph-mon[31975]: purged_snaps scrub ok 2024-03-16T11:46:00.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:45:59 smithi029 ceph-mon[31975]: osd.7 [v2:172.21.15.152:6816/2924651787,v1:172.21.15.152:6817/2924651787] boot 2024-03-16T11:46:00.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:45:59 smithi029 ceph-mon[31975]: osdmap e51: 8 total, 8 up, 8 in 2024-03-16T11:46:00.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:45:59 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-16T11:46:00.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:45:59 smithi029 ceph-mon[31975]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 715 GiB / 715 GiB avail 2024-03-16T11:46:00.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:45:59 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:46:00.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:45:59 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:46:00.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:45:59 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-16T11:46:00.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:45:59 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-16T11:46:00.339 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:45:59 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-16T11:46:00.339 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:45:59 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:46:00.339 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:45:59 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:46:00.339 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:45:59 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T11:46:00.339 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:45:59 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:46:01.018 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:46:00 smithi154 ceph-mon[36320]: Detected new or changed devices on smithi152 2024-03-16T11:46:01.018 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:46:00 smithi154 ceph-mon[36320]: Adjusting osd_memory_target on smithi152 to 5693M 2024-03-16T11:46:01.018 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:46:00 smithi154 ceph-mon[36320]: osdmap e52: 8 total, 8 up, 8 in 2024-03-16T11:46:01.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:46:00 smithi029 ceph-mon[31975]: Detected new or changed devices on smithi152 2024-03-16T11:46:01.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:46:00 smithi029 ceph-mon[31975]: Adjusting osd_memory_target on smithi152 to 5693M 2024-03-16T11:46:01.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:46:00 smithi029 ceph-mon[31975]: osdmap e52: 8 total, 8 up, 8 in 2024-03-16T11:46:01.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:46:00 smithi152 ceph-mon[36291]: Detected new or changed devices on smithi152 2024-03-16T11:46:01.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:46:00 smithi152 ceph-mon[36291]: Adjusting osd_memory_target on smithi152 to 5693M 2024-03-16T11:46:01.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:46:00 smithi152 ceph-mon[36291]: osdmap e52: 8 total, 8 up, 8 in 2024-03-16T11:46:01.552 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:46:01.550+0000 7f023fd9b700 1 -- 172.21.15.154:0/1573462974 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0238100d50 msgr2=0x7f023810a920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:46:01.552 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:46:01.550+0000 7f023fd9b700 1 --2- 172.21.15.154:0/1573462974 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0238100d50 0x7f023810a920 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f023000b0a0 tx=0x7f023000b3b0 comp rx=0 tx=0).stop 2024-03-16T11:46:01.552 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:46:01.550+0000 7f023fd9b700 1 -- 172.21.15.154:0/1573462974 shutdown_connections 2024-03-16T11:46:01.552 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:46:01.550+0000 7f023fd9b700 1 --2- 172.21.15.154:0/1573462974 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0238100d50 0x7f023810a920 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:46:01.553 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:46:01.550+0000 7f023fd9b700 1 --2- 172.21.15.154:0/1573462974 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0238103930 0x7f0238103d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:46:01.553 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:46:01.550+0000 7f023fd9b700 1 --2- 172.21.15.154:0/1573462974 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f02381003b0 0x7f0238100810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:46:01.553 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:46:01.550+0000 7f023fd9b700 1 -- 172.21.15.154:0/1573462974 >> 172.21.15.154:0/1573462974 conn(0x7f02380fbf70 msgr2=0x7f02380fe3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:46:01.553 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:46:01.551+0000 7f023fd9b700 1 -- 172.21.15.154:0/1573462974 shutdown_connections 2024-03-16T11:46:01.553 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:46:01.551+0000 7f023fd9b700 1 -- 172.21.15.154:0/1573462974 wait complete. 2024-03-16T11:46:01.553 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:46:01.551+0000 7f023fd9b700 1 Processor -- start 2024-03-16T11:46:01.553 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:46:01.551+0000 7f023fd9b700 1 -- start start 2024-03-16T11:46:01.553 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:46:01.551+0000 7f023fd9b700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f02381003b0 0x7f02381984d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:46:01.553 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:46:01.551+0000 7f023fd9b700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0238100d50 0x7f0238198a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:46:01.553 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:46:01.551+0000 7f023fd9b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0238103930 0x7f023819fa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:46:01.553 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:46:01.551+0000 7f023fd9b700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f023810dee0 con 0x7f0238103930 2024-03-16T11:46:01.553 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:46:01.551+0000 7f023fd9b700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f023810dd60 con 0x7f0238100d50 2024-03-16T11:46:01.553 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:46:01.551+0000 7f023fd9b700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f023810e060 con 0x7f02381003b0 2024-03-16T11:46:01.553 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:46:01.551+0000 7f023db37700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f02381003b0 0x7f02381984d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:46:01.553 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:46:01.551+0000 7f023db37700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f02381003b0 0x7f02381984d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.154:36648/0 (socket says 172.21.15.154:36648) 2024-03-16T11:46:01.554 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:46:01.552+0000 7f023db37700 1 -- 172.21.15.154:0/256767430 learned_addr learned my addr 172.21.15.154:0/256767430 (peer_addr_for_me v2:172.21.15.154:0/0) 2024-03-16T11:46:01.554 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:46:01.552+0000 7f023e338700 1 --2- 172.21.15.154:0/256767430 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0238103930 0x7f023819fa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:46:01.554 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:46:01.552+0000 7f023d336700 1 --2- 172.21.15.154:0/256767430 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0238100d50 0x7f0238198a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:46:01.554 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:46:01.552+0000 7f023db37700 1 -- 172.21.15.154:0/256767430 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0238100d50 msgr2=0x7f0238198a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:46:01.554 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:46:01.552+0000 7f023db37700 1 --2- 172.21.15.154:0/256767430 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0238100d50 0x7f0238198a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:46:01.554 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:46:01.552+0000 7f023db37700 1 -- 172.21.15.154:0/256767430 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0238103930 msgr2=0x7f023819fa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:46:01.554 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:46:01.552+0000 7f023db37700 1 --2- 172.21.15.154:0/256767430 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0238103930 0x7f023819fa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:46:01.554 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:46:01.552+0000 7f023db37700 1 -- 172.21.15.154:0/256767430 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02381a0130 con 0x7f02381003b0 2024-03-16T11:46:01.554 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:46:01.552+0000 7f023d336700 1 --2- 172.21.15.154:0/256767430 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0238100d50 0x7f0238198a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-16T11:46:01.554 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:46:01.552+0000 7f023db37700 1 --2- 172.21.15.154:0/256767430 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f02381003b0 0x7f02381984d0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f022800d9b0 tx=0x7f022800dcc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:46:01.554 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:46:01.552+0000 7f022effd700 1 -- 172.21.15.154:0/256767430 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0228017030 con 0x7f02381003b0 2024-03-16T11:46:01.554 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:46:01.552+0000 7f023fd9b700 1 -- 172.21.15.154:0/256767430 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f02381a0420 con 0x7f02381003b0 2024-03-16T11:46:01.554 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:46:01.552+0000 7f022effd700 1 -- 172.21.15.154:0/256767430 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0228004500 con 0x7f02381003b0 2024-03-16T11:46:01.554 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:46:01.552+0000 7f022effd700 1 -- 172.21.15.154:0/256767430 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0228003aa0 con 0x7f02381003b0 2024-03-16T11:46:01.554 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:46:01.552+0000 7f023fd9b700 1 -- 172.21.15.154:0/256767430 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f02381a09b0 con 0x7f02381003b0 2024-03-16T11:46:01.555 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:46:01.553+0000 7f023fd9b700 1 -- 172.21.15.154:0/256767430 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0238068f90 con 0x7f02381003b0 2024-03-16T11:46:01.555 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:46:01.554+0000 7f022effd700 1 -- 172.21.15.154:0/256767430 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f022801a070 con 0x7f02381003b0 2024-03-16T11:46:01.556 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:46:01.555+0000 7f022effd700 1 --2- 172.21.15.154:0/256767430 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f02240a2890 0x7f02240a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:46:01.556 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:46:01.555+0000 7f022effd700 1 -- 172.21.15.154:0/256767430 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(53..53 src has 1..53) ==== 5426+0+0 (secure 0 0 0) 0x7f02280cd850 con 0x7f02381003b0 2024-03-16T11:46:01.556 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:46:01.555+0000 7f023d336700 1 --2- 172.21.15.154:0/256767430 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f02240a2890 0x7f02240a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:46:01.556 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:46:01.556+0000 7f023d336700 1 --2- 172.21.15.154:0/256767430 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f02240a2890 0x7f02240a4d50 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f0234007900 tx=0x7f0234008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:46:01.559 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:46:01.560+0000 7f022effd700 1 -- 172.21.15.154:0/256767430 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f02280232f0 con 0x7f02381003b0 2024-03-16T11:46:01.764 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:46:01.764+0000 7f023fd9b700 1 -- 172.21.15.154:0/256767430 --> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi154:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f0238062710 con 0x7f02240a2890 2024-03-16T11:46:02.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:46:01 smithi154 ceph-mon[36320]: osdmap e53: 8 total, 8 up, 8 in 2024-03-16T11:46:02.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:46:01 smithi154 ceph-mon[36320]: pgmap v180: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 243 MiB used, 715 GiB / 715 GiB avail 2024-03-16T11:46:02.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:46:01 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T11:46:02.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:46:01 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T11:46:02.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:46:01 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:46:02.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:46:01 smithi154 ceph-mon[36320]: osdmap e54: 8 total, 8 up, 8 in 2024-03-16T11:46:02.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:46:01 smithi029 ceph-mon[31975]: osdmap e53: 8 total, 8 up, 8 in 2024-03-16T11:46:02.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:46:01 smithi029 ceph-mon[31975]: pgmap v180: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 243 MiB used, 715 GiB / 715 GiB avail 2024-03-16T11:46:02.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:46:01 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T11:46:02.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:46:01 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T11:46:02.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:46:01 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:46:02.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:46:01 smithi029 ceph-mon[31975]: osdmap e54: 8 total, 8 up, 8 in 2024-03-16T11:46:02.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:46:01 smithi152 ceph-mon[36291]: osdmap e53: 8 total, 8 up, 8 in 2024-03-16T11:46:02.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:46:01 smithi152 ceph-mon[36291]: pgmap v180: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 243 MiB used, 715 GiB / 715 GiB avail 2024-03-16T11:46:02.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:46:01 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T11:46:02.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:46:01 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T11:46:02.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:46:01 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:46:02.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:46:01 smithi152 ceph-mon[36291]: osdmap e54: 8 total, 8 up, 8 in 2024-03-16T11:46:03.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:46:02 smithi154 ceph-mon[36320]: from='client.24319 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi154:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T11:46:03.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:46:02 smithi154 ceph-mon[36320]: osdmap e55: 8 total, 8 up, 8 in 2024-03-16T11:46:03.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:46:02 smithi029 ceph-mon[31975]: from='client.24319 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi154:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T11:46:03.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:46:02 smithi029 ceph-mon[31975]: osdmap e55: 8 total, 8 up, 8 in 2024-03-16T11:46:03.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:46:02 smithi152 ceph-mon[36291]: from='client.24319 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi154:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T11:46:03.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:46:02 smithi152 ceph-mon[36291]: osdmap e55: 8 total, 8 up, 8 in 2024-03-16T11:46:04.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:46:03 smithi154 ceph-mon[36320]: pgmap v183: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-03-16T11:46:04.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:46:03 smithi029 ceph-mon[31975]: pgmap v183: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-03-16T11:46:04.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:46:03 smithi152 ceph-mon[36291]: pgmap v183: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-03-16T11:46:05.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:46:04 smithi154 ceph-mon[36320]: from='client.? 172.21.15.154:0/2970638845' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a25fb741-caa7-429a-b540-e394d034b446"}]: dispatch 2024-03-16T11:46:05.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:46:04 smithi154 ceph-mon[36320]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a25fb741-caa7-429a-b540-e394d034b446"}]: dispatch 2024-03-16T11:46:05.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:46:04 smithi154 ceph-mon[36320]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a25fb741-caa7-429a-b540-e394d034b446"}]': finished 2024-03-16T11:46:05.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:46:04 smithi154 ceph-mon[36320]: osdmap e56: 9 total, 8 up, 9 in 2024-03-16T11:46:05.296 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:46:04 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-16T11:46:05.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:46:04 smithi029 ceph-mon[31975]: from='client.? 172.21.15.154:0/2970638845' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a25fb741-caa7-429a-b540-e394d034b446"}]: dispatch 2024-03-16T11:46:05.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:46:04 smithi029 ceph-mon[31975]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a25fb741-caa7-429a-b540-e394d034b446"}]: dispatch 2024-03-16T11:46:05.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:46:04 smithi029 ceph-mon[31975]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a25fb741-caa7-429a-b540-e394d034b446"}]': finished 2024-03-16T11:46:05.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:46:04 smithi029 ceph-mon[31975]: osdmap e56: 9 total, 8 up, 9 in 2024-03-16T11:46:05.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:46:04 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-16T11:46:05.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:46:04 smithi152 ceph-mon[36291]: from='client.? 172.21.15.154:0/2970638845' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a25fb741-caa7-429a-b540-e394d034b446"}]: dispatch 2024-03-16T11:46:05.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:46:04 smithi152 ceph-mon[36291]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a25fb741-caa7-429a-b540-e394d034b446"}]: dispatch 2024-03-16T11:46:05.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:46:04 smithi152 ceph-mon[36291]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a25fb741-caa7-429a-b540-e394d034b446"}]': finished 2024-03-16T11:46:05.378 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:46:04 smithi152 ceph-mon[36291]: osdmap e56: 9 total, 8 up, 9 in 2024-03-16T11:46:05.378 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:46:04 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-16T11:46:06.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:46:05 smithi154 ceph-mon[36320]: from='client.? 172.21.15.154:0/465330636' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T11:46:06.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:46:05 smithi154 ceph-mon[36320]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-03-16T11:46:06.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:46:05 smithi029 ceph-mon[31975]: from='client.? 172.21.15.154:0/465330636' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T11:46:06.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:46:05 smithi029 ceph-mon[31975]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-03-16T11:46:06.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:46:05 smithi152 ceph-mon[36291]: from='client.? 172.21.15.154:0/465330636' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T11:46:06.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:46:05 smithi152 ceph-mon[36291]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-03-16T11:46:08.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:46:07 smithi154 ceph-mon[36320]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-03-16T11:46:08.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:46:07 smithi029 ceph-mon[31975]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-03-16T11:46:08.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:46:07 smithi152 ceph-mon[36291]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-03-16T11:46:09.764 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:46:09 smithi154 ceph-mon[36320]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-03-16T11:46:10.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:46:09 smithi029 ceph-mon[31975]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-03-16T11:46:10.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:46:09 smithi152 ceph-mon[36291]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-03-16T11:46:11.895 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:46:11 smithi154 ceph-mon[36320]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-03-16T11:46:11.929 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:46:11 smithi152 ceph-mon[36291]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-03-16T11:46:12.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:46:11 smithi029 ceph-mon[31975]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-03-16T11:46:12.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:46:12 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-03-16T11:46:12.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:46:12 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:46:12.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:46:12 smithi154 ceph-mon[36320]: Deploying daemon osd.8 on smithi154 2024-03-16T11:46:13.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:46:12 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-03-16T11:46:13.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:46:12 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:46:13.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:46:12 smithi029 ceph-mon[31975]: Deploying daemon osd.8 on smithi154 2024-03-16T11:46:13.124 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:46:12 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-03-16T11:46:13.124 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:46:12 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:46:13.124 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:46:12 smithi152 ceph-mon[36291]: Deploying daemon osd.8 on smithi154 2024-03-16T11:46:13.915 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:46:13 smithi154 ceph-mon[36320]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-03-16T11:46:14.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:46:13 smithi029 ceph-mon[31975]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-03-16T11:46:14.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:46:13 smithi152 ceph-mon[36291]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-03-16T11:46:15.919 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:46:15 smithi154 ceph-mon[36320]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-03-16T11:46:16.087 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:46:15 smithi029 ceph-mon[31975]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-03-16T11:46:16.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:46:15 smithi152 ceph-mon[36291]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-03-16T11:46:18.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:46:17 smithi154 ceph-mon[36320]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-03-16T11:46:18.087 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:46:17 smithi029 ceph-mon[31975]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-03-16T11:46:18.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:46:17 smithi152 ceph-mon[36291]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-03-16T11:46:18.676 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:46:18 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T11:46:18.677 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:46:18 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:46:18.677 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:46:18 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:46:19.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:46:18 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T11:46:19.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:46:18 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:46:19.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:46:18 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:46:19.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:46:18 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T11:46:19.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:46:18 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:46:19.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:46:18 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:46:20.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:46:19 smithi154 ceph-mon[36320]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-03-16T11:46:20.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:46:19 smithi029 ceph-mon[31975]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-03-16T11:46:20.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:46:19 smithi152 ceph-mon[36291]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-03-16T11:46:21.582 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:46:21.583+0000 7f022effd700 1 -- 172.21.15.154:0/256767430 <== mgr.14152 v2:172.21.15.29:6800/4192064640 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f0238062710 con 0x7f02240a2890 2024-03-16T11:46:21.583 INFO:teuthology.orchestra.run.smithi154.stdout:Created osd(s) 8 on host 'smithi154' 2024-03-16T11:46:21.585 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:46:21.585+0000 7f023fd9b700 1 -- 172.21.15.154:0/256767430 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f02240a2890 msgr2=0x7f02240a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:46:21.585 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:46:21.585+0000 7f023fd9b700 1 --2- 172.21.15.154:0/256767430 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f02240a2890 0x7f02240a4d50 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f0234007900 tx=0x7f0234008040 comp rx=0 tx=0).stop 2024-03-16T11:46:21.585 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:46:21.585+0000 7f023fd9b700 1 -- 172.21.15.154:0/256767430 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f02381003b0 msgr2=0x7f02381984d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:46:21.585 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:46:21.585+0000 7f023fd9b700 1 --2- 172.21.15.154:0/256767430 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f02381003b0 0x7f02381984d0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f022800d9b0 tx=0x7f022800dcc0 comp rx=0 tx=0).stop 2024-03-16T11:46:21.585 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:46:21.585+0000 7f023fd9b700 1 -- 172.21.15.154:0/256767430 shutdown_connections 2024-03-16T11:46:21.585 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:46:21.585+0000 7f023fd9b700 1 --2- 172.21.15.154:0/256767430 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0238103930 0x7f023819fa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:46:21.585 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:46:21.585+0000 7f023fd9b700 1 --2- 172.21.15.154:0/256767430 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f02240a2890 0x7f02240a4d50 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:46:21.586 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:46:21.585+0000 7f023fd9b700 1 --2- 172.21.15.154:0/256767430 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f02381003b0 0x7f02381984d0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:46:21.586 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:46:21.585+0000 7f023fd9b700 1 --2- 172.21.15.154:0/256767430 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0238100d50 0x7f0238198a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:46:21.586 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:46:21.585+0000 7f023fd9b700 1 -- 172.21.15.154:0/256767430 >> 172.21.15.154:0/256767430 conn(0x7f02380fbf70 msgr2=0x7f0238106970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:46:21.586 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:46:21.585+0000 7f023fd9b700 1 -- 172.21.15.154:0/256767430 shutdown_connections 2024-03-16T11:46:21.586 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:46:21.585+0000 7f023fd9b700 1 -- 172.21.15.154:0/256767430 wait complete. 2024-03-16T11:46:21.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:46:21 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:46:21.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:46:21 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:46:21.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:46:21 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:46:21.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:46:21 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T11:46:21.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:46:21 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:46:21.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:46:21 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:46:21.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:46:21 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:46:21.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:46:21 smithi154 ceph-mon[36320]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-03-16T11:46:21.796 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:46:21 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T11:46:22.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:46:21 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:46:22.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:46:21 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:46:22.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:46:21 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:46:22.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:46:21 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T11:46:22.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:46:21 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:46:22.089 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:46:21 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:46:22.089 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:46:21 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:46:22.089 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:46:21 smithi029 ceph-mon[31975]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-03-16T11:46:22.089 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:46:21 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T11:46:22.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:46:21 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:46:22.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:46:21 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:46:22.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:46:21 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:46:22.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:46:21 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T11:46:22.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:46:21 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:46:22.128 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:46:21 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:46:22.128 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:46:21 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:46:22.128 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:46:21 smithi152 ceph-mon[36291]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-03-16T11:46:22.128 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:46:21 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T11:46:22.236 DEBUG:teuthology.orchestra.run.smithi154:osd.8> sudo journalctl -f -n 0 -u ceph-d1887f28-e389-11ee-95c9-87774f69a715@osd.8.service 2024-03-16T11:46:22.238 INFO:tasks.cephadm:Deploying osd.9 on smithi029 with /dev/vg_nvme/lv_1... 2024-03-16T11:46:22.238 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d1887f28-e389-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-16T11:46:22.492 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/d1887f28-e389-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T11:46:22.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:46:22 smithi154 ceph-mon[36320]: from='osd.8 [v2:172.21.15.154:6816/129974156,v1:172.21.15.154:6817/129974156]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-16T11:46:22.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:46:22 smithi154 ceph-mon[36320]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-16T11:46:22.907 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:46:22 smithi029 ceph-mon[31975]: from='osd.8 [v2:172.21.15.154:6816/129974156,v1:172.21.15.154:6817/129974156]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-16T11:46:22.908 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:46:22 smithi029 ceph-mon[31975]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-16T11:46:23.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:46:22 smithi152 ceph-mon[36291]: from='osd.8 [v2:172.21.15.154:6816/129974156,v1:172.21.15.154:6817/129974156]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-16T11:46:23.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:46:22 smithi152 ceph-mon[36291]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-16T11:46:23.968 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:46:23 smithi154 ceph-mon[36320]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-03-16T11:46:23.968 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:46:23 smithi154 ceph-mon[36320]: osdmap e57: 9 total, 8 up, 9 in 2024-03-16T11:46:23.968 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:46:23 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-16T11:46:23.968 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:46:23 smithi154 ceph-mon[36320]: from='osd.8 [v2:172.21.15.154:6816/129974156,v1:172.21.15.154:6817/129974156]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]: dispatch 2024-03-16T11:46:23.968 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:46:23 smithi154 ceph-mon[36320]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]: dispatch 2024-03-16T11:46:23.968 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:46:23 smithi154 ceph-mon[36320]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-03-16T11:46:24.032 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:46:23 smithi029 ceph-mon[31975]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-03-16T11:46:24.032 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:46:23 smithi029 ceph-mon[31975]: osdmap e57: 9 total, 8 up, 9 in 2024-03-16T11:46:24.032 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:46:23 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-16T11:46:24.032 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:46:23 smithi029 ceph-mon[31975]: from='osd.8 [v2:172.21.15.154:6816/129974156,v1:172.21.15.154:6817/129974156]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]: dispatch 2024-03-16T11:46:24.033 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:46:23 smithi029 ceph-mon[31975]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]: dispatch 2024-03-16T11:46:24.033 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:46:23 smithi029 ceph-mon[31975]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-03-16T11:46:24.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:46:23 smithi152 ceph-mon[36291]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-03-16T11:46:24.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:46:23 smithi152 ceph-mon[36291]: osdmap e57: 9 total, 8 up, 9 in 2024-03-16T11:46:24.128 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:46:23 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-16T11:46:24.128 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:46:23 smithi152 ceph-mon[36291]: from='osd.8 [v2:172.21.15.154:6816/129974156,v1:172.21.15.154:6817/129974156]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]: dispatch 2024-03-16T11:46:24.128 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:46:23 smithi152 ceph-mon[36291]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]: dispatch 2024-03-16T11:46:24.128 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:46:23 smithi152 ceph-mon[36291]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-03-16T11:46:24.295 INFO:journalctl@ceph.osd.8.smithi154.stdout:Mar 16 11:46:24 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-8[55054]: 2024-03-16T11:46:24.138+0000 7f3f0cfd2700 -1 osd.8 0 waiting for initial osdmap 2024-03-16T11:46:24.295 INFO:journalctl@ceph.osd.8.smithi154.stdout:Mar 16 11:46:24 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-8[55054]: 2024-03-16T11:46:24.149+0000 7f3f04dbd700 -1 osd.8 58 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-16T11:46:24.745 INFO:teuthology.orchestra.run.smithi029.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-16T11:46:24.746 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-16T11:46:24.746 INFO:teuthology.orchestra.run.smithi029.stderr: stderr: 10+0 records in 2024-03-16T11:46:24.746 INFO:teuthology.orchestra.run.smithi029.stderr:10+0 records out 2024-03-16T11:46:24.746 INFO:teuthology.orchestra.run.smithi029.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00836745 s, 1.3 GB/s 2024-03-16T11:46:24.746 INFO:teuthology.orchestra.run.smithi029.stderr:--> Zapping successful for: 2024-03-16T11:46:24.922 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:46:24 smithi154 ceph-mon[36320]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]': finished 2024-03-16T11:46:24.922 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:46:24 smithi154 ceph-mon[36320]: osdmap e58: 9 total, 8 up, 9 in 2024-03-16T11:46:24.922 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:46:24 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-16T11:46:24.922 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:46:24 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-16T11:46:24.922 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:46:24 smithi154 ceph-mon[36320]: from='osd.8 ' entity='osd.8' 2024-03-16T11:46:25.056 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:46:24 smithi029 ceph-mon[31975]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]': finished 2024-03-16T11:46:25.056 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:46:24 smithi029 ceph-mon[31975]: osdmap e58: 9 total, 8 up, 9 in 2024-03-16T11:46:25.056 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:46:24 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-16T11:46:25.056 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:46:24 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-16T11:46:25.056 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:46:24 smithi029 ceph-mon[31975]: from='osd.8 ' entity='osd.8' 2024-03-16T11:46:25.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:46:24 smithi152 ceph-mon[36291]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]': finished 2024-03-16T11:46:25.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:46:24 smithi152 ceph-mon[36291]: osdmap e58: 9 total, 8 up, 9 in 2024-03-16T11:46:25.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:46:24 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-16T11:46:25.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:46:24 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-16T11:46:25.128 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:46:24 smithi152 ceph-mon[36291]: from='osd.8 ' entity='osd.8' 2024-03-16T11:46:25.448 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d1887f28-e389-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi029:vg_nvme/lv_1 2024-03-16T11:46:25.698 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/d1887f28-e389-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T11:46:26.032 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:46:25 smithi029 ceph-mon[31975]: purged_snaps scrub starts 2024-03-16T11:46:26.032 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:46:25 smithi029 ceph-mon[31975]: purged_snaps scrub ok 2024-03-16T11:46:26.032 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:46:25 smithi029 ceph-mon[31975]: osd.8 [v2:172.21.15.154:6816/129974156,v1:172.21.15.154:6817/129974156] boot 2024-03-16T11:46:26.032 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:46:25 smithi029 ceph-mon[31975]: osdmap e59: 9 total, 9 up, 9 in 2024-03-16T11:46:26.032 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:46:25 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-16T11:46:26.032 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:46:25 smithi029 ceph-mon[31975]: Detected new or changed devices on smithi154 2024-03-16T11:46:26.032 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:46:25 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:46:26.033 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:46:25 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:46:26.033 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:46:25 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-16T11:46:26.033 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:46:25 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-16T11:46:26.033 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:46:25 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-16T11:46:26.033 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:46:25 smithi029 ceph-mon[31975]: Adjusting osd_memory_target on smithi154 to 5693M 2024-03-16T11:46:26.033 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:46:25 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:46:26.033 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:46:25 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:46:26.033 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:46:25 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T11:46:26.033 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:46:25 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:46:26.033 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:46:25 smithi029 ceph-mon[31975]: pgmap v198: 1 pgs: 1 peering; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-03-16T11:46:26.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:46:25 smithi154 ceph-mon[36320]: purged_snaps scrub starts 2024-03-16T11:46:26.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:46:25 smithi154 ceph-mon[36320]: purged_snaps scrub ok 2024-03-16T11:46:26.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:46:25 smithi154 ceph-mon[36320]: osd.8 [v2:172.21.15.154:6816/129974156,v1:172.21.15.154:6817/129974156] boot 2024-03-16T11:46:26.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:46:25 smithi154 ceph-mon[36320]: osdmap e59: 9 total, 9 up, 9 in 2024-03-16T11:46:26.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:46:25 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-16T11:46:26.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:46:25 smithi154 ceph-mon[36320]: Detected new or changed devices on smithi154 2024-03-16T11:46:26.046 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:46:25 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:46:26.046 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:46:25 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:46:26.046 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:46:25 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-16T11:46:26.046 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:46:25 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-16T11:46:26.046 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:46:25 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-16T11:46:26.046 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:46:25 smithi154 ceph-mon[36320]: Adjusting osd_memory_target on smithi154 to 5693M 2024-03-16T11:46:26.046 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:46:25 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:46:26.046 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:46:25 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:46:26.046 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:46:25 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T11:46:26.046 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:46:25 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:46:26.046 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:46:25 smithi154 ceph-mon[36320]: pgmap v198: 1 pgs: 1 peering; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-03-16T11:46:26.128 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:46:25 smithi152 ceph-mon[36291]: purged_snaps scrub starts 2024-03-16T11:46:26.128 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:46:25 smithi152 ceph-mon[36291]: purged_snaps scrub ok 2024-03-16T11:46:26.128 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:46:25 smithi152 ceph-mon[36291]: osd.8 [v2:172.21.15.154:6816/129974156,v1:172.21.15.154:6817/129974156] boot 2024-03-16T11:46:26.128 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:46:25 smithi152 ceph-mon[36291]: osdmap e59: 9 total, 9 up, 9 in 2024-03-16T11:46:26.128 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:46:25 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-16T11:46:26.128 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:46:25 smithi152 ceph-mon[36291]: Detected new or changed devices on smithi154 2024-03-16T11:46:26.128 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:46:25 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:46:26.128 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:46:25 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:46:26.128 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:46:25 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-16T11:46:26.128 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:46:25 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-16T11:46:26.128 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:46:25 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-16T11:46:26.128 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:46:25 smithi152 ceph-mon[36291]: Adjusting osd_memory_target on smithi154 to 5693M 2024-03-16T11:46:26.129 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:46:25 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:46:26.129 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:46:25 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:46:26.129 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:46:25 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T11:46:26.129 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:46:25 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:46:26.129 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:46:25 smithi152 ceph-mon[36291]: pgmap v198: 1 pgs: 1 peering; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-03-16T11:46:26.964 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:46:26 smithi029 ceph-mon[31975]: osdmap e60: 9 total, 9 up, 9 in 2024-03-16T11:46:27.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:46:26 smithi154 ceph-mon[36320]: osdmap e60: 9 total, 9 up, 9 in 2024-03-16T11:46:27.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:46:26 smithi152 ceph-mon[36291]: osdmap e60: 9 total, 9 up, 9 in 2024-03-16T11:46:27.522 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:46:27.514+0000 7fbb9ed65700 1 -- 172.21.15.29:0/1955749613 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbb98102e10 msgr2=0x7fbb981096c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:46:27.523 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:46:27.514+0000 7fbb9ed65700 1 --2- 172.21.15.29:0/1955749613 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbb98102e10 0x7fbb981096c0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fbb9000b0d0 tx=0x7fbb9000b3e0 comp rx=0 tx=0).stop 2024-03-16T11:46:27.523 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:46:27.514+0000 7fbb9ed65700 1 -- 172.21.15.29:0/1955749613 shutdown_connections 2024-03-16T11:46:27.523 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:46:27.514+0000 7fbb9ed65700 1 --2- 172.21.15.29:0/1955749613 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbb98102e10 0x7fbb981096c0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:46:27.523 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:46:27.514+0000 7fbb9ed65700 1 --2- 172.21.15.29:0/1955749613 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbb98101270 0x7fbb98101690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:46:27.523 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:46:27.514+0000 7fbb9ed65700 1 --2- 172.21.15.29:0/1955749613 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbb98102470 0x7fbb981028d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:46:27.523 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:46:27.514+0000 7fbb9ed65700 1 -- 172.21.15.29:0/1955749613 >> 172.21.15.29:0/1955749613 conn(0x7fbb980fc830 msgr2=0x7fbb980fec50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:46:27.523 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:46:27.514+0000 7fbb9ed65700 1 -- 172.21.15.29:0/1955749613 shutdown_connections 2024-03-16T11:46:27.523 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:46:27.515+0000 7fbb9ed65700 1 -- 172.21.15.29:0/1955749613 wait complete. 2024-03-16T11:46:27.523 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:46:27.515+0000 7fbb9ed65700 1 Processor -- start 2024-03-16T11:46:27.523 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:46:27.515+0000 7fbb9ed65700 1 -- start start 2024-03-16T11:46:27.523 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:46:27.515+0000 7fbb9ed65700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbb98101270 0x7fbb98197370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:46:27.524 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:46:27.515+0000 7fbb9ed65700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbb98102470 0x7fbb981978b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:46:27.524 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:46:27.515+0000 7fbb9ed65700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbb98102e10 0x7fbb9819e930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:46:27.524 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:46:27.515+0000 7fbb9ed65700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fbb9810ce50 con 0x7fbb98101270 2024-03-16T11:46:27.524 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:46:27.515+0000 7fbb9ed65700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fbb9810ccd0 con 0x7fbb98102e10 2024-03-16T11:46:27.524 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:46:27.515+0000 7fbb9ed65700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fbb9810cfd0 con 0x7fbb98102470 2024-03-16T11:46:27.524 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:46:27.515+0000 7fbb9cb01700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbb98101270 0x7fbb98197370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:46:27.524 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:46:27.515+0000 7fbb9cb01700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbb98101270 0x7fbb98197370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49286/0 (socket says 172.21.15.29:49286) 2024-03-16T11:46:27.524 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:46:27.515+0000 7fbb9cb01700 1 -- 172.21.15.29:0/4144739245 learned_addr learned my addr 172.21.15.29:0/4144739245 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:46:27.524 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:46:27.515+0000 7fbb9d302700 1 --2- 172.21.15.29:0/4144739245 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbb98102e10 0x7fbb9819e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:46:27.524 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:46:27.516+0000 7fbb9cb01700 1 -- 172.21.15.29:0/4144739245 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbb98102470 msgr2=0x7fbb981978b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:46:27.524 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:46:27.516+0000 7fbb9cb01700 1 --2- 172.21.15.29:0/4144739245 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbb98102470 0x7fbb981978b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:46:27.524 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:46:27.516+0000 7fbb9cb01700 1 -- 172.21.15.29:0/4144739245 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbb98102e10 msgr2=0x7fbb9819e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:46:27.524 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:46:27.516+0000 7fbb9cb01700 1 --2- 172.21.15.29:0/4144739245 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbb98102e10 0x7fbb9819e930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:46:27.524 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:46:27.516+0000 7fbb9cb01700 1 -- 172.21.15.29:0/4144739245 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb9819efd0 con 0x7fbb98101270 2024-03-16T11:46:27.525 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:46:27.516+0000 7fbb9cb01700 1 --2- 172.21.15.29:0/4144739245 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbb98101270 0x7fbb98197370 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fbb8c00c9c0 tx=0x7fbb8c00cd80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:46:27.525 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:46:27.516+0000 7fbb95ffb700 1 -- 172.21.15.29:0/4144739245 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb8c007ac0 con 0x7fbb98101270 2024-03-16T11:46:27.525 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:46:27.516+0000 7fbb9ed65700 1 -- 172.21.15.29:0/4144739245 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbb9806aa30 con 0x7fbb98101270 2024-03-16T11:46:27.525 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:46:27.516+0000 7fbb95ffb700 1 -- 172.21.15.29:0/4144739245 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbb8c007c60 con 0x7fbb98101270 2024-03-16T11:46:27.525 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:46:27.516+0000 7fbb95ffb700 1 -- 172.21.15.29:0/4144739245 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb8c0186f0 con 0x7fbb98101270 2024-03-16T11:46:27.525 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:46:27.516+0000 7fbb9ed65700 1 -- 172.21.15.29:0/4144739245 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbb9806b040 con 0x7fbb98101270 2024-03-16T11:46:27.525 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:46:27.517+0000 7fbb9ed65700 1 -- 172.21.15.29:0/4144739245 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbb98068f90 con 0x7fbb98101270 2024-03-16T11:46:27.525 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:46:27.519+0000 7fbb95ffb700 1 -- 172.21.15.29:0/4144739245 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fbb8c004030 con 0x7fbb98101270 2024-03-16T11:46:27.525 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:46:27.519+0000 7fbb95ffb700 1 --2- 172.21.15.29:0/4144739245 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fbb880a2890 0x7fbb880a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:46:27.525 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:46:27.520+0000 7fbb97fff700 1 --2- 172.21.15.29:0/4144739245 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fbb880a2890 0x7fbb880a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:46:27.525 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:46:27.520+0000 7fbb95ffb700 1 -- 172.21.15.29:0/4144739245 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 5829+0+0 (secure 0 0 0) 0x7fbb8c0ce260 con 0x7fbb98101270 2024-03-16T11:46:27.525 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:46:27.520+0000 7fbb97fff700 1 --2- 172.21.15.29:0/4144739245 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fbb880a2890 0x7fbb880a4d50 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fbb840097b0 tx=0x7fbb84006d20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:46:27.525 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:46:27.523+0000 7fbb95ffb700 1 -- 172.21.15.29:0/4144739245 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fbb8c014030 con 0x7fbb98101270 2024-03-16T11:46:27.740 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:46:27.741+0000 7fbb9ed65700 1 -- 172.21.15.29:0/4144739245 --> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi029:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7fbb9806b300 con 0x7fbb880a2890 2024-03-16T11:46:28.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:46:27 smithi154 ceph-mon[36320]: osdmap e61: 9 total, 9 up, 9 in 2024-03-16T11:46:28.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:46:27 smithi154 ceph-mon[36320]: osdmap e62: 9 total, 9 up, 9 in 2024-03-16T11:46:28.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:46:27 smithi154 ceph-mon[36320]: pgmap v202: 1 pgs: 1 peering; 577 KiB data, 275 MiB used, 804 GiB / 805 GiB avail 2024-03-16T11:46:28.087 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:46:27 smithi029 ceph-mon[31975]: osdmap e61: 9 total, 9 up, 9 in 2024-03-16T11:46:28.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:46:27 smithi029 ceph-mon[31975]: osdmap e62: 9 total, 9 up, 9 in 2024-03-16T11:46:28.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:46:27 smithi029 ceph-mon[31975]: pgmap v202: 1 pgs: 1 peering; 577 KiB data, 275 MiB used, 804 GiB / 805 GiB avail 2024-03-16T11:46:28.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:46:27 smithi152 ceph-mon[36291]: osdmap e61: 9 total, 9 up, 9 in 2024-03-16T11:46:28.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:46:27 smithi152 ceph-mon[36291]: osdmap e62: 9 total, 9 up, 9 in 2024-03-16T11:46:28.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:46:27 smithi152 ceph-mon[36291]: pgmap v202: 1 pgs: 1 peering; 577 KiB data, 275 MiB used, 804 GiB / 805 GiB avail 2024-03-16T11:46:29.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:46:28 smithi154 ceph-mon[36320]: from='client.14442 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi029:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T11:46:29.046 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:46:28 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T11:46:29.046 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:46:28 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T11:46:29.046 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:46:28 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:46:29.046 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:46:28 smithi154 ceph-mon[36320]: osdmap e63: 9 total, 9 up, 9 in 2024-03-16T11:46:29.087 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:46:28 smithi029 ceph-mon[31975]: from='client.14442 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi029:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T11:46:29.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:46:28 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T11:46:29.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:46:28 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T11:46:29.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:46:28 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:46:29.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:46:28 smithi029 ceph-mon[31975]: osdmap e63: 9 total, 9 up, 9 in 2024-03-16T11:46:29.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:46:28 smithi152 ceph-mon[36291]: from='client.14442 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi029:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T11:46:29.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:46:28 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T11:46:29.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:46:28 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T11:46:29.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:46:28 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:46:29.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:46:28 smithi152 ceph-mon[36291]: osdmap e63: 9 total, 9 up, 9 in 2024-03-16T11:46:29.994 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:46:29 smithi029 ceph-mon[31975]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 275 MiB used, 804 GiB / 805 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-16T11:46:30.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:46:29 smithi154 ceph-mon[36320]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 275 MiB used, 804 GiB / 805 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-16T11:46:30.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:46:29 smithi152 ceph-mon[36291]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 275 MiB used, 804 GiB / 805 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-16T11:46:31.258 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:46:30 smithi029 ceph-mon[31975]: from='client.? 172.21.15.29:0/2154116101' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1071cb0e-8c5f-4cd8-9805-0045e220d946"}]: dispatch 2024-03-16T11:46:31.258 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:46:30 smithi029 ceph-mon[31975]: from='client.? 172.21.15.29:0/2154116101' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1071cb0e-8c5f-4cd8-9805-0045e220d946"}]': finished 2024-03-16T11:46:31.258 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:46:30 smithi029 ceph-mon[31975]: osdmap e64: 10 total, 9 up, 10 in 2024-03-16T11:46:31.258 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:46:30 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-16T11:46:31.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:46:30 smithi154 ceph-mon[36320]: from='client.? 172.21.15.29:0/2154116101' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1071cb0e-8c5f-4cd8-9805-0045e220d946"}]: dispatch 2024-03-16T11:46:31.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:46:30 smithi154 ceph-mon[36320]: from='client.? 172.21.15.29:0/2154116101' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1071cb0e-8c5f-4cd8-9805-0045e220d946"}]': finished 2024-03-16T11:46:31.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:46:30 smithi154 ceph-mon[36320]: osdmap e64: 10 total, 9 up, 10 in 2024-03-16T11:46:31.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:46:30 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-16T11:46:31.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:46:30 smithi152 ceph-mon[36291]: from='client.? 172.21.15.29:0/2154116101' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1071cb0e-8c5f-4cd8-9805-0045e220d946"}]: dispatch 2024-03-16T11:46:31.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:46:30 smithi152 ceph-mon[36291]: from='client.? 172.21.15.29:0/2154116101' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1071cb0e-8c5f-4cd8-9805-0045e220d946"}]': finished 2024-03-16T11:46:31.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:46:30 smithi152 ceph-mon[36291]: osdmap e64: 10 total, 9 up, 10 in 2024-03-16T11:46:31.378 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:46:30 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-16T11:46:32.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:46:31 smithi154 ceph-mon[36320]: from='client.? 172.21.15.29:0/3926270046' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T11:46:32.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:46:31 smithi154 ceph-mon[36320]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 276 MiB used, 804 GiB / 805 GiB avail; 118 KiB/s, 0 objects/s recovering 2024-03-16T11:46:32.337 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:46:31 smithi029 ceph-mon[31975]: from='client.? 172.21.15.29:0/3926270046' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T11:46:32.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:46:31 smithi029 ceph-mon[31975]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 276 MiB used, 804 GiB / 805 GiB avail; 118 KiB/s, 0 objects/s recovering 2024-03-16T11:46:32.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:46:31 smithi152 ceph-mon[36291]: from='client.? 172.21.15.29:0/3926270046' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T11:46:32.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:46:31 smithi152 ceph-mon[36291]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 276 MiB used, 804 GiB / 805 GiB avail; 118 KiB/s, 0 objects/s recovering 2024-03-16T11:46:34.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:46:33 smithi154 ceph-mon[36320]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 276 MiB used, 804 GiB / 805 GiB avail; 87 KiB/s, 0 objects/s recovering 2024-03-16T11:46:34.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:46:33 smithi029 ceph-mon[31975]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 276 MiB used, 804 GiB / 805 GiB avail; 87 KiB/s, 0 objects/s recovering 2024-03-16T11:46:34.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:46:33 smithi152 ceph-mon[36291]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 276 MiB used, 804 GiB / 805 GiB avail; 87 KiB/s, 0 objects/s recovering 2024-03-16T11:46:36.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:46:35 smithi154 ceph-mon[36320]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 276 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-16T11:46:36.065 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:46:35 smithi029 ceph-mon[31975]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 276 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-16T11:46:36.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:46:35 smithi152 ceph-mon[36291]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 276 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-16T11:46:37.976 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:46:37 smithi029 ceph-mon[31975]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 276 MiB used, 804 GiB / 805 GiB avail; 60 KiB/s, 0 objects/s recovering 2024-03-16T11:46:38.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:46:37 smithi154 ceph-mon[36320]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 276 MiB used, 804 GiB / 805 GiB avail; 60 KiB/s, 0 objects/s recovering 2024-03-16T11:46:38.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:46:37 smithi152 ceph-mon[36291]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 276 MiB used, 804 GiB / 805 GiB avail; 60 KiB/s, 0 objects/s recovering 2024-03-16T11:46:39.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:46:38 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-03-16T11:46:39.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:46:38 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:46:39.337 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:46:38 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-03-16T11:46:39.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:46:38 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:46:39.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:46:38 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-03-16T11:46:39.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:46:38 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:46:40.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:46:39 smithi029 ceph-mon[31975]: Deploying daemon osd.9 on smithi029 2024-03-16T11:46:40.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:46:39 smithi029 ceph-mon[31975]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 276 MiB used, 804 GiB / 805 GiB avail 2024-03-16T11:46:40.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:46:39 smithi154 ceph-mon[36320]: Deploying daemon osd.9 on smithi029 2024-03-16T11:46:40.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:46:39 smithi154 ceph-mon[36320]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 276 MiB used, 804 GiB / 805 GiB avail 2024-03-16T11:46:40.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:46:39 smithi152 ceph-mon[36291]: Deploying daemon osd.9 on smithi029 2024-03-16T11:46:40.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:46:39 smithi152 ceph-mon[36291]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 276 MiB used, 804 GiB / 805 GiB avail 2024-03-16T11:46:41.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:46:41 smithi029 ceph-mon[31975]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 276 MiB used, 804 GiB / 805 GiB avail 2024-03-16T11:46:42.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:46:41 smithi154 ceph-mon[36320]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 276 MiB used, 804 GiB / 805 GiB avail 2024-03-16T11:46:42.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:46:41 smithi152 ceph-mon[36291]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 276 MiB used, 804 GiB / 805 GiB avail 2024-03-16T11:46:44.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:46:43 smithi154 ceph-mon[36320]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 276 MiB used, 804 GiB / 805 GiB avail 2024-03-16T11:46:44.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:46:43 smithi029 ceph-mon[31975]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 276 MiB used, 804 GiB / 805 GiB avail 2024-03-16T11:46:44.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:46:43 smithi152 ceph-mon[36291]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 276 MiB used, 804 GiB / 805 GiB avail 2024-03-16T11:46:46.545 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:46:46 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:46:46.545 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:46:46 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:46:46.545 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:46:46 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T11:46:46.545 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:46:46 smithi154 ceph-mon[36320]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 276 MiB used, 804 GiB / 805 GiB avail 2024-03-16T11:46:46.588 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:46:46 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:46:46.588 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:46:46 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:46:46.588 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:46:46 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T11:46:46.588 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:46:46 smithi029 ceph-mon[31975]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 276 MiB used, 804 GiB / 805 GiB avail 2024-03-16T11:46:46.627 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:46:46 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:46:46.627 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:46:46 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:46:46.627 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:46:46 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T11:46:46.627 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:46:46 smithi152 ceph-mon[36291]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 276 MiB used, 804 GiB / 805 GiB avail 2024-03-16T11:46:47.925 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:46:47 smithi029 ceph-mon[31975]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 276 MiB used, 804 GiB / 805 GiB avail 2024-03-16T11:46:48.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:46:47 smithi154 ceph-mon[36320]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 276 MiB used, 804 GiB / 805 GiB avail 2024-03-16T11:46:48.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:46:47 smithi152 ceph-mon[36291]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 276 MiB used, 804 GiB / 805 GiB avail 2024-03-16T11:46:48.673 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:46:48.674+0000 7fbb95ffb700 1 -- 172.21.15.29:0/4144739245 <== mgr.14152 v2:172.21.15.29:6800/4192064640 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fbb9806b300 con 0x7fbb880a2890 2024-03-16T11:46:48.674 INFO:teuthology.orchestra.run.smithi029.stdout:Created osd(s) 9 on host 'smithi029' 2024-03-16T11:46:48.676 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:46:48.676+0000 7fbb9ed65700 1 -- 172.21.15.29:0/4144739245 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fbb880a2890 msgr2=0x7fbb880a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:46:48.676 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:46:48.676+0000 7fbb9ed65700 1 --2- 172.21.15.29:0/4144739245 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fbb880a2890 0x7fbb880a4d50 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fbb840097b0 tx=0x7fbb84006d20 comp rx=0 tx=0).stop 2024-03-16T11:46:48.676 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:46:48.676+0000 7fbb9ed65700 1 -- 172.21.15.29:0/4144739245 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbb98101270 msgr2=0x7fbb98197370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:46:48.676 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:46:48.676+0000 7fbb9ed65700 1 --2- 172.21.15.29:0/4144739245 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbb98101270 0x7fbb98197370 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fbb8c00c9c0 tx=0x7fbb8c00cd80 comp rx=0 tx=0).stop 2024-03-16T11:46:48.676 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:46:48.676+0000 7fbb9ed65700 1 -- 172.21.15.29:0/4144739245 shutdown_connections 2024-03-16T11:46:48.676 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:46:48.676+0000 7fbb9ed65700 1 --2- 172.21.15.29:0/4144739245 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbb98102e10 0x7fbb9819e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:46:48.676 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:46:48.676+0000 7fbb9ed65700 1 --2- 172.21.15.29:0/4144739245 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbb98101270 0x7fbb98197370 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:46:48.676 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:46:48.676+0000 7fbb9ed65700 1 --2- 172.21.15.29:0/4144739245 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fbb880a2890 0x7fbb880a4d50 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:46:48.676 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:46:48.676+0000 7fbb9ed65700 1 --2- 172.21.15.29:0/4144739245 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbb98102470 0x7fbb981978b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:46:48.676 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:46:48.676+0000 7fbb9ed65700 1 -- 172.21.15.29:0/4144739245 >> 172.21.15.29:0/4144739245 conn(0x7fbb980fc830 msgr2=0x7fbb981056a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:46:48.676 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:46:48.676+0000 7fbb9ed65700 1 -- 172.21.15.29:0/4144739245 shutdown_connections 2024-03-16T11:46:48.677 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:46:48.676+0000 7fbb9ed65700 1 -- 172.21.15.29:0/4144739245 wait complete. 2024-03-16T11:46:49.687 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:46:49 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:46:49.688 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:46:49 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:46:49.688 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:46:49 smithi029 ceph-mon[31975]: from='osd.9 [v2:172.21.15.29:6826/2951414243,v1:172.21.15.29:6827/2951414243]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-03-16T11:46:49.688 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:46:49 smithi029 ceph-mon[31975]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 276 MiB used, 804 GiB / 805 GiB avail 2024-03-16T11:46:49.715 DEBUG:teuthology.orchestra.run.smithi029:osd.9> sudo journalctl -f -n 0 -u ceph-d1887f28-e389-11ee-95c9-87774f69a715@osd.9.service 2024-03-16T11:46:49.718 INFO:tasks.cephadm:Deploying osd.10 on smithi152 with /dev/vg_nvme/lv_1... 2024-03-16T11:46:49.718 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d1887f28-e389-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-16T11:46:49.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:46:49 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:46:49.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:46:49 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:46:49.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:46:49 smithi152 ceph-mon[36291]: from='osd.9 [v2:172.21.15.29:6826/2951414243,v1:172.21.15.29:6827/2951414243]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-03-16T11:46:49.878 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:46:49 smithi152 ceph-mon[36291]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 276 MiB used, 804 GiB / 805 GiB avail 2024-03-16T11:46:49.965 INFO:teuthology.orchestra.run.smithi152.stderr:Inferring config /var/lib/ceph/d1887f28-e389-11ee-95c9-87774f69a715/mon.b/config 2024-03-16T11:46:50.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:46:49 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:46:50.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:46:49 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:46:50.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:46:49 smithi154 ceph-mon[36320]: from='osd.9 [v2:172.21.15.29:6826/2951414243,v1:172.21.15.29:6827/2951414243]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-03-16T11:46:50.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:46:49 smithi154 ceph-mon[36320]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 276 MiB used, 804 GiB / 805 GiB avail 2024-03-16T11:46:50.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:46:50 smithi152 ceph-mon[36291]: from='osd.9 [v2:172.21.15.29:6826/2951414243,v1:172.21.15.29:6827/2951414243]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-03-16T11:46:50.878 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:46:50 smithi152 ceph-mon[36291]: osdmap e65: 10 total, 9 up, 10 in 2024-03-16T11:46:50.878 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:46:50 smithi152 ceph-mon[36291]: from='osd.9 [v2:172.21.15.29:6826/2951414243,v1:172.21.15.29:6827/2951414243]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]: dispatch 2024-03-16T11:46:50.878 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:46:50 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-16T11:46:50.878 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:46:50 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:46:50.878 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:46:50 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:46:50.945 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:46:50 smithi029 ceph-mon[31975]: from='osd.9 [v2:172.21.15.29:6826/2951414243,v1:172.21.15.29:6827/2951414243]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-03-16T11:46:50.945 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:46:50 smithi029 ceph-mon[31975]: osdmap e65: 10 total, 9 up, 10 in 2024-03-16T11:46:50.945 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:46:50 smithi029 ceph-mon[31975]: from='osd.9 [v2:172.21.15.29:6826/2951414243,v1:172.21.15.29:6827/2951414243]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]: dispatch 2024-03-16T11:46:50.945 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:46:50 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-16T11:46:50.945 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:46:50 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:46:50.945 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:46:50 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:46:50.945 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 16 11:46:50 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-9[67182]: 2024-03-16T11:46:50.639+0000 7fe1d1895700 -1 osd.9 0 waiting for initial osdmap 2024-03-16T11:46:50.945 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 16 11:46:50 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-9[67182]: 2024-03-16T11:46:50.657+0000 7fe1cd688700 -1 osd.9 66 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-16T11:46:51.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:46:50 smithi154 ceph-mon[36320]: from='osd.9 [v2:172.21.15.29:6826/2951414243,v1:172.21.15.29:6827/2951414243]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-03-16T11:46:51.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:46:50 smithi154 ceph-mon[36320]: osdmap e65: 10 total, 9 up, 10 in 2024-03-16T11:46:51.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:46:50 smithi154 ceph-mon[36320]: from='osd.9 [v2:172.21.15.29:6826/2951414243,v1:172.21.15.29:6827/2951414243]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]: dispatch 2024-03-16T11:46:51.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:46:50 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-16T11:46:51.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:46:50 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:46:51.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:46:50 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:46:51.670 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:46:51 smithi152 ceph-mon[36291]: from='osd.9 [v2:172.21.15.29:6826/2951414243,v1:172.21.15.29:6827/2951414243]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]': finished 2024-03-16T11:46:51.670 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:46:51 smithi152 ceph-mon[36291]: osdmap e66: 10 total, 9 up, 10 in 2024-03-16T11:46:51.670 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:46:51 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-16T11:46:51.670 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:46:51 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-16T11:46:51.671 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:46:51 smithi152 ceph-mon[36291]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 276 MiB used, 804 GiB / 805 GiB avail 2024-03-16T11:46:52.030 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:46:51 smithi029 ceph-mon[31975]: from='osd.9 [v2:172.21.15.29:6826/2951414243,v1:172.21.15.29:6827/2951414243]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]': finished 2024-03-16T11:46:52.030 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:46:51 smithi029 ceph-mon[31975]: osdmap e66: 10 total, 9 up, 10 in 2024-03-16T11:46:52.030 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:46:51 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-16T11:46:52.030 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:46:51 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-16T11:46:52.030 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:46:51 smithi029 ceph-mon[31975]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 276 MiB used, 804 GiB / 805 GiB avail 2024-03-16T11:46:52.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:46:51 smithi154 ceph-mon[36320]: from='osd.9 [v2:172.21.15.29:6826/2951414243,v1:172.21.15.29:6827/2951414243]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]': finished 2024-03-16T11:46:52.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:46:51 smithi154 ceph-mon[36320]: osdmap e66: 10 total, 9 up, 10 in 2024-03-16T11:46:52.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:46:51 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-16T11:46:52.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:46:51 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-16T11:46:52.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:46:51 smithi154 ceph-mon[36320]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 276 MiB used, 804 GiB / 805 GiB avail 2024-03-16T11:46:52.130 INFO:teuthology.orchestra.run.smithi152.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-16T11:46:52.131 INFO:teuthology.orchestra.run.smithi152.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-16T11:46:52.131 INFO:teuthology.orchestra.run.smithi152.stderr: stderr: 10+0 records in 2024-03-16T11:46:52.131 INFO:teuthology.orchestra.run.smithi152.stderr:10+0 records out 2024-03-16T11:46:52.131 INFO:teuthology.orchestra.run.smithi152.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0113892 s, 921 MB/s 2024-03-16T11:46:52.131 INFO:teuthology.orchestra.run.smithi152.stderr:--> Zapping successful for: 2024-03-16T11:46:52.834 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:46:52 smithi152 ceph-mon[36291]: purged_snaps scrub starts 2024-03-16T11:46:52.835 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:46:52 smithi152 ceph-mon[36291]: purged_snaps scrub ok 2024-03-16T11:46:52.835 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:46:52 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-16T11:46:52.835 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:46:52 smithi152 ceph-mon[36291]: osd.9 [v2:172.21.15.29:6826/2951414243,v1:172.21.15.29:6827/2951414243] boot 2024-03-16T11:46:52.835 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:46:52 smithi152 ceph-mon[36291]: osdmap e67: 10 total, 10 up, 10 in 2024-03-16T11:46:52.835 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:46:52 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-16T11:46:52.861 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d1887f28-e389-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi152:vg_nvme/lv_1 2024-03-16T11:46:53.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:46:52 smithi154 ceph-mon[36320]: purged_snaps scrub starts 2024-03-16T11:46:53.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:46:52 smithi154 ceph-mon[36320]: purged_snaps scrub ok 2024-03-16T11:46:53.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:46:52 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-16T11:46:53.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:46:52 smithi154 ceph-mon[36320]: osd.9 [v2:172.21.15.29:6826/2951414243,v1:172.21.15.29:6827/2951414243] boot 2024-03-16T11:46:53.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:46:52 smithi154 ceph-mon[36320]: osdmap e67: 10 total, 10 up, 10 in 2024-03-16T11:46:53.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:46:52 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-16T11:46:53.054 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:46:52 smithi029 ceph-mon[31975]: purged_snaps scrub starts 2024-03-16T11:46:53.054 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:46:52 smithi029 ceph-mon[31975]: purged_snaps scrub ok 2024-03-16T11:46:53.054 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:46:52 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-16T11:46:53.054 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:46:52 smithi029 ceph-mon[31975]: osd.9 [v2:172.21.15.29:6826/2951414243,v1:172.21.15.29:6827/2951414243] boot 2024-03-16T11:46:53.054 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:46:52 smithi029 ceph-mon[31975]: osdmap e67: 10 total, 10 up, 10 in 2024-03-16T11:46:53.055 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:46:52 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-16T11:46:53.097 INFO:teuthology.orchestra.run.smithi152.stderr:Inferring config /var/lib/ceph/d1887f28-e389-11ee-95c9-87774f69a715/mon.b/config 2024-03-16T11:46:53.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:46:53 smithi029 ceph-mon[31975]: osdmap e68: 10 total, 10 up, 10 in 2024-03-16T11:46:53.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:46:53 smithi029 ceph-mon[31975]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 303 MiB used, 894 GiB / 894 GiB avail 2024-03-16T11:46:53.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:46:53 smithi152 ceph-mon[36291]: osdmap e68: 10 total, 10 up, 10 in 2024-03-16T11:46:53.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:46:53 smithi152 ceph-mon[36291]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 303 MiB used, 894 GiB / 894 GiB avail 2024-03-16T11:46:54.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:46:53 smithi154 ceph-mon[36320]: osdmap e68: 10 total, 10 up, 10 in 2024-03-16T11:46:54.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:46:53 smithi154 ceph-mon[36320]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 303 MiB used, 894 GiB / 894 GiB avail 2024-03-16T11:46:54.839 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-16T11:46:54.837+0000 7f8bb17ba700 1 -- 172.21.15.152:0/3871491585 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8bac101230 msgr2=0x7f8bac101650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:46:54.839 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-16T11:46:54.837+0000 7f8bb17ba700 1 --2- 172.21.15.152:0/3871491585 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8bac101230 0x7f8bac101650 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f8b9c0099c0 tx=0x7f8b9c009cd0 comp rx=0 tx=0).stop 2024-03-16T11:46:54.839 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-16T11:46:54.837+0000 7f8bb17ba700 1 -- 172.21.15.152:0/3871491585 shutdown_connections 2024-03-16T11:46:54.839 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-16T11:46:54.837+0000 7f8bb17ba700 1 --2- 172.21.15.152:0/3871491585 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8bac102dd0 0x7f8bac109680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:46:54.839 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-16T11:46:54.837+0000 7f8bb17ba700 1 --2- 172.21.15.152:0/3871491585 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8bac101230 0x7f8bac101650 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:46:54.839 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-16T11:46:54.837+0000 7f8bb17ba700 1 --2- 172.21.15.152:0/3871491585 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8bac102430 0x7f8bac102890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:46:54.839 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-16T11:46:54.837+0000 7f8bb17ba700 1 -- 172.21.15.152:0/3871491585 >> 172.21.15.152:0/3871491585 conn(0x7f8bac0fc7d0 msgr2=0x7f8bac0fec10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:46:54.839 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-16T11:46:54.837+0000 7f8bb17ba700 1 -- 172.21.15.152:0/3871491585 shutdown_connections 2024-03-16T11:46:54.840 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-16T11:46:54.837+0000 7f8bb17ba700 1 -- 172.21.15.152:0/3871491585 wait complete. 2024-03-16T11:46:54.840 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-16T11:46:54.838+0000 7f8bb17ba700 1 Processor -- start 2024-03-16T11:46:54.840 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-16T11:46:54.838+0000 7f8bb17ba700 1 -- start start 2024-03-16T11:46:54.840 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-16T11:46:54.838+0000 7f8bb17ba700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8bac101230 0x7f8bac197480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:46:54.840 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-16T11:46:54.838+0000 7f8bb17ba700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8bac102430 0x7f8bac1979c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:46:54.840 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-16T11:46:54.838+0000 7f8bb17ba700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8bac102dd0 0x7f8bac19ea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:46:54.840 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-16T11:46:54.838+0000 7f8bb17ba700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f8bac10ce70 con 0x7f8bac102430 2024-03-16T11:46:54.840 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-16T11:46:54.838+0000 7f8bb17ba700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f8bac10ccf0 con 0x7f8bac101230 2024-03-16T11:46:54.840 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-16T11:46:54.838+0000 7f8bb17ba700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f8bac10cff0 con 0x7f8bac102dd0 2024-03-16T11:46:54.840 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-16T11:46:54.838+0000 7f8baaffd700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8bac101230 0x7f8bac197480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:46:54.840 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-16T11:46:54.838+0000 7f8baaffd700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8bac101230 0x7f8bac197480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.152:33706/0 (socket says 172.21.15.152:33706) 2024-03-16T11:46:54.840 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-16T11:46:54.838+0000 7f8baaffd700 1 -- 172.21.15.152:0/2735553514 learned_addr learned my addr 172.21.15.152:0/2735553514 (peer_addr_for_me v2:172.21.15.152:0/0) 2024-03-16T11:46:54.840 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-16T11:46:54.838+0000 7f8ba27fc700 1 --2- 172.21.15.152:0/2735553514 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8bac102430 0x7f8bac1979c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:46:54.841 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-16T11:46:54.839+0000 7f8bab7fe700 1 --2- 172.21.15.152:0/2735553514 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8bac102dd0 0x7f8bac19ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:46:54.841 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-16T11:46:54.839+0000 7f8baaffd700 1 -- 172.21.15.152:0/2735553514 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8bac102dd0 msgr2=0x7f8bac19ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:46:54.841 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-16T11:46:54.839+0000 7f8baaffd700 1 --2- 172.21.15.152:0/2735553514 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8bac102dd0 0x7f8bac19ea40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:46:54.841 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-16T11:46:54.839+0000 7f8baaffd700 1 -- 172.21.15.152:0/2735553514 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8bac102430 msgr2=0x7f8bac1979c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:46:54.841 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-16T11:46:54.839+0000 7f8baaffd700 1 --2- 172.21.15.152:0/2735553514 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8bac102430 0x7f8bac1979c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:46:54.841 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-16T11:46:54.839+0000 7f8baaffd700 1 -- 172.21.15.152:0/2735553514 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8bac19f0e0 con 0x7f8bac101230 2024-03-16T11:46:54.841 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-16T11:46:54.839+0000 7f8baaffd700 1 --2- 172.21.15.152:0/2735553514 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8bac101230 0x7f8bac197480 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f8b9c00fa80 tx=0x7f8b9c00fb60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:46:54.841 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-16T11:46:54.839+0000 7f8ba8ff9700 1 -- 172.21.15.152:0/2735553514 <== mon.2 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b9c027070 con 0x7f8bac101230 2024-03-16T11:46:54.841 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-16T11:46:54.839+0000 7f8ba8ff9700 1 -- 172.21.15.152:0/2735553514 <== mon.2 v2:172.21.15.152:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8b9c00fe30 con 0x7f8bac101230 2024-03-16T11:46:54.841 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-16T11:46:54.839+0000 7f8bb17ba700 1 -- 172.21.15.152:0/2735553514 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8bac19f370 con 0x7f8bac101230 2024-03-16T11:46:54.841 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-16T11:46:54.839+0000 7f8ba8ff9700 1 -- 172.21.15.152:0/2735553514 <== mon.2 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b9c022470 con 0x7f8bac101230 2024-03-16T11:46:54.841 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-16T11:46:54.839+0000 7f8bb17ba700 1 -- 172.21.15.152:0/2735553514 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8bac19f8d0 con 0x7f8bac101230 2024-03-16T11:46:54.842 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-16T11:46:54.840+0000 7f8bb17ba700 1 -- 172.21.15.152:0/2735553514 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8bac068f90 con 0x7f8bac101230 2024-03-16T11:46:54.842 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-16T11:46:54.841+0000 7f8ba8ff9700 1 -- 172.21.15.152:0/2735553514 <== mon.2 v2:172.21.15.152:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f8b9c02a070 con 0x7f8bac101230 2024-03-16T11:46:54.843 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-16T11:46:54.842+0000 7f8ba8ff9700 1 --2- 172.21.15.152:0/2735553514 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f8b980a2890 0x7f8b980a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:46:54.843 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-16T11:46:54.842+0000 7f8ba8ff9700 1 -- 172.21.15.152:0/2735553514 <== mon.2 v2:172.21.15.152:3300/0 5 ==== osd_map(68..68 src has 1..68) ==== 6261+0+0 (secure 0 0 0) 0x7f8b9c0cf0c0 con 0x7f8bac101230 2024-03-16T11:46:54.843 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-16T11:46:54.842+0000 7f8ba27fc700 1 --2- 172.21.15.152:0/2735553514 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f8b980a2890 0x7f8b980a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:46:54.843 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-16T11:46:54.843+0000 7f8ba27fc700 1 --2- 172.21.15.152:0/2735553514 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f8b980a2890 0x7f8b980a4d50 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f8b94006fd0 tx=0x7f8b94008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:46:54.845 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-16T11:46:54.845+0000 7f8ba8ff9700 1 -- 172.21.15.152:0/2735553514 <== mon.2 v2:172.21.15.152:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f8b9c05e740 con 0x7f8bac101230 2024-03-16T11:46:55.048 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-16T11:46:55.048+0000 7f8bb17ba700 1 -- 172.21.15.152:0/2735553514 --> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi152:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7f8bac062710 con 0x7f8b980a2890 2024-03-16T11:46:55.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:46:55 smithi154 ceph-mon[36320]: Detected new or changed devices on smithi029 2024-03-16T11:46:55.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:46:55 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:46:55.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:46:55 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:46:55.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:46:55 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-16T11:46:55.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:46:55 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-16T11:46:55.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:46:55 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-16T11:46:55.296 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:46:55 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-16T11:46:55.296 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:46:55 smithi154 ceph-mon[36320]: Adjusting osd_memory_target on smithi029 to 4273M 2024-03-16T11:46:55.296 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:46:55 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:46:55.296 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:46:55 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:46:55.296 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:46:55 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T11:46:55.296 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:46:55 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:46:55.296 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:46:55 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T11:46:55.296 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:46:55 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:46:55.296 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:46:55 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T11:46:55.296 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:46:55 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:46:55.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:46:55 smithi029 ceph-mon[31975]: Detected new or changed devices on smithi029 2024-03-16T11:46:55.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:46:55 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:46:55.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:46:55 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:46:55.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:46:55 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-16T11:46:55.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:46:55 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-16T11:46:55.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:46:55 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-16T11:46:55.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:46:55 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-16T11:46:55.339 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:46:55 smithi029 ceph-mon[31975]: Adjusting osd_memory_target on smithi029 to 4273M 2024-03-16T11:46:55.339 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:46:55 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:46:55.339 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:46:55 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:46:55.339 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:46:55 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T11:46:55.339 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:46:55 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:46:55.339 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:46:55 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T11:46:55.339 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:46:55 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:46:55.339 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:46:55 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T11:46:55.339 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:46:55 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:46:55.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:46:55 smithi152 ceph-mon[36291]: Detected new or changed devices on smithi029 2024-03-16T11:46:55.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:46:55 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:46:55.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:46:55 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:46:55.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:46:55 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-16T11:46:55.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:46:55 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-16T11:46:55.378 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:46:55 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-16T11:46:55.378 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:46:55 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-16T11:46:55.378 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:46:55 smithi152 ceph-mon[36291]: Adjusting osd_memory_target on smithi029 to 4273M 2024-03-16T11:46:55.378 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:46:55 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:46:55.378 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:46:55 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:46:55.378 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:46:55 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T11:46:55.378 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:46:55 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:46:55.378 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:46:55 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T11:46:55.378 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:46:55 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:46:55.378 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:46:55 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T11:46:55.378 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:46:55 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:46:56.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:46:56 smithi154 ceph-mon[36320]: from='client.24314 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi152:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T11:46:56.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:46:56 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T11:46:56.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:46:56 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T11:46:56.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:46:56 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:46:56.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:46:56 smithi154 ceph-mon[36320]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 303 MiB used, 894 GiB / 894 GiB avail 2024-03-16T11:46:56.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:46:56 smithi029 ceph-mon[31975]: from='client.24314 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi152:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T11:46:56.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:46:56 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T11:46:56.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:46:56 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T11:46:56.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:46:56 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:46:56.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:46:56 smithi029 ceph-mon[31975]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 303 MiB used, 894 GiB / 894 GiB avail 2024-03-16T11:46:56.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:46:56 smithi152 ceph-mon[36291]: from='client.24314 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi152:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T11:46:56.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:46:56 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T11:46:56.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:46:56 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T11:46:56.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:46:56 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:46:56.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:46:56 smithi152 ceph-mon[36291]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 303 MiB used, 894 GiB / 894 GiB avail 2024-03-16T11:46:58.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:46:57 smithi154 ceph-mon[36320]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 303 MiB used, 894 GiB / 894 GiB avail 2024-03-16T11:46:58.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:46:57 smithi029 ceph-mon[31975]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 303 MiB used, 894 GiB / 894 GiB avail 2024-03-16T11:46:58.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:46:57 smithi152 ceph-mon[36291]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 303 MiB used, 894 GiB / 894 GiB avail 2024-03-16T11:46:59.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:46:58 smithi154 ceph-mon[36320]: from='client.? 172.21.15.152:0/3391563840' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ee7c7d74-3f39-4cd4-be80-843fb1dd9ecc"}]: dispatch 2024-03-16T11:46:59.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:46:58 smithi154 ceph-mon[36320]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ee7c7d74-3f39-4cd4-be80-843fb1dd9ecc"}]: dispatch 2024-03-16T11:46:59.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:46:58 smithi154 ceph-mon[36320]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ee7c7d74-3f39-4cd4-be80-843fb1dd9ecc"}]': finished 2024-03-16T11:46:59.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:46:58 smithi154 ceph-mon[36320]: osdmap e69: 11 total, 10 up, 11 in 2024-03-16T11:46:59.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:46:58 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-16T11:46:59.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:46:58 smithi154 ceph-mon[36320]: from='client.? 172.21.15.152:0/1654233562' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T11:46:59.087 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:46:58 smithi029 ceph-mon[31975]: from='client.? 172.21.15.152:0/3391563840' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ee7c7d74-3f39-4cd4-be80-843fb1dd9ecc"}]: dispatch 2024-03-16T11:46:59.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:46:58 smithi029 ceph-mon[31975]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ee7c7d74-3f39-4cd4-be80-843fb1dd9ecc"}]: dispatch 2024-03-16T11:46:59.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:46:58 smithi029 ceph-mon[31975]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ee7c7d74-3f39-4cd4-be80-843fb1dd9ecc"}]': finished 2024-03-16T11:46:59.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:46:58 smithi029 ceph-mon[31975]: osdmap e69: 11 total, 10 up, 11 in 2024-03-16T11:46:59.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:46:58 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-16T11:46:59.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:46:58 smithi029 ceph-mon[31975]: from='client.? 172.21.15.152:0/1654233562' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T11:46:59.128 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:46:58 smithi152 ceph-mon[36291]: from='client.? 172.21.15.152:0/3391563840' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ee7c7d74-3f39-4cd4-be80-843fb1dd9ecc"}]: dispatch 2024-03-16T11:46:59.128 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:46:58 smithi152 ceph-mon[36291]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ee7c7d74-3f39-4cd4-be80-843fb1dd9ecc"}]: dispatch 2024-03-16T11:46:59.128 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:46:58 smithi152 ceph-mon[36291]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ee7c7d74-3f39-4cd4-be80-843fb1dd9ecc"}]': finished 2024-03-16T11:46:59.128 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:46:58 smithi152 ceph-mon[36291]: osdmap e69: 11 total, 10 up, 11 in 2024-03-16T11:46:59.128 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:46:58 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-16T11:46:59.128 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:46:58 smithi152 ceph-mon[36291]: from='client.? 172.21.15.152:0/1654233562' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T11:47:00.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:46:59 smithi154 ceph-mon[36320]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 303 MiB used, 894 GiB / 894 GiB avail 2024-03-16T11:47:00.087 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:46:59 smithi029 ceph-mon[31975]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 303 MiB used, 894 GiB / 894 GiB avail 2024-03-16T11:47:00.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:46:59 smithi152 ceph-mon[36291]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 303 MiB used, 894 GiB / 894 GiB avail 2024-03-16T11:47:02.002 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:47:01 smithi152 ceph-mon[36291]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 303 MiB used, 894 GiB / 894 GiB avail 2024-03-16T11:47:02.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:47:01 smithi154 ceph-mon[36320]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 303 MiB used, 894 GiB / 894 GiB avail 2024-03-16T11:47:02.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:47:01 smithi029 ceph-mon[31975]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 303 MiB used, 894 GiB / 894 GiB avail 2024-03-16T11:47:03.941 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:47:03 smithi152 ceph-mon[36291]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 303 MiB used, 894 GiB / 894 GiB avail 2024-03-16T11:47:04.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:47:03 smithi154 ceph-mon[36320]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 303 MiB used, 894 GiB / 894 GiB avail 2024-03-16T11:47:04.087 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:47:03 smithi029 ceph-mon[31975]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 303 MiB used, 894 GiB / 894 GiB avail 2024-03-16T11:47:05.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:47:05 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-03-16T11:47:05.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:47:05 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:47:05.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:47:05 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-03-16T11:47:05.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:47:05 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:47:06.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:47:05 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-03-16T11:47:06.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:47:05 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:47:06.830 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:47:06 smithi152 ceph-mon[36291]: Deploying daemon osd.10 on smithi152 2024-03-16T11:47:06.830 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:47:06 smithi152 ceph-mon[36291]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 303 MiB used, 894 GiB / 894 GiB avail 2024-03-16T11:47:06.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:47:06 smithi029 ceph-mon[31975]: Deploying daemon osd.10 on smithi152 2024-03-16T11:47:06.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:47:06 smithi029 ceph-mon[31975]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 303 MiB used, 894 GiB / 894 GiB avail 2024-03-16T11:47:07.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:47:06 smithi154 ceph-mon[36320]: Deploying daemon osd.10 on smithi152 2024-03-16T11:47:07.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:47:06 smithi154 ceph-mon[36320]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 303 MiB used, 894 GiB / 894 GiB avail 2024-03-16T11:47:07.867 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:47:07 smithi152 ceph-mon[36291]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 303 MiB used, 894 GiB / 894 GiB avail 2024-03-16T11:47:08.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:47:07 smithi154 ceph-mon[36320]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 303 MiB used, 894 GiB / 894 GiB avail 2024-03-16T11:47:08.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:47:07 smithi029 ceph-mon[31975]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 303 MiB used, 894 GiB / 894 GiB avail 2024-03-16T11:47:09.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:47:09 smithi152 ceph-mon[36291]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 303 MiB used, 894 GiB / 894 GiB avail 2024-03-16T11:47:10.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:47:09 smithi154 ceph-mon[36320]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 303 MiB used, 894 GiB / 894 GiB avail 2024-03-16T11:47:10.087 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:47:09 smithi029 ceph-mon[31975]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 303 MiB used, 894 GiB / 894 GiB avail 2024-03-16T11:47:12.627 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:47:12 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:47:12.627 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:47:12 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:47:12.627 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:47:12 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T11:47:12.627 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:47:12 smithi152 ceph-mon[36291]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 303 MiB used, 894 GiB / 894 GiB avail 2024-03-16T11:47:12.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:47:12 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:47:12.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:47:12 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:47:12.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:47:12 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T11:47:12.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:47:12 smithi154 ceph-mon[36320]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 303 MiB used, 894 GiB / 894 GiB avail 2024-03-16T11:47:12.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:47:12 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:47:12.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:47:12 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:47:12.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:47:12 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T11:47:12.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:47:12 smithi029 ceph-mon[31975]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 303 MiB used, 894 GiB / 894 GiB avail 2024-03-16T11:47:13.922 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:47:13 smithi152 ceph-mon[36291]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 303 MiB used, 894 GiB / 894 GiB avail 2024-03-16T11:47:14.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:47:13 smithi154 ceph-mon[36320]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 303 MiB used, 894 GiB / 894 GiB avail 2024-03-16T11:47:14.087 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:47:13 smithi029 ceph-mon[31975]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 303 MiB used, 894 GiB / 894 GiB avail 2024-03-16T11:47:14.964 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-16T11:47:14.965+0000 7f8ba8ff9700 1 -- 172.21.15.152:0/2735553514 <== mgr.14152 v2:172.21.15.29:6800/4192064640 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7f8bac062710 con 0x7f8b980a2890 2024-03-16T11:47:14.965 INFO:teuthology.orchestra.run.smithi152.stdout:Created osd(s) 10 on host 'smithi152' 2024-03-16T11:47:14.967 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-16T11:47:14.968+0000 7f8bb17ba700 1 -- 172.21.15.152:0/2735553514 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f8b980a2890 msgr2=0x7f8b980a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:47:14.968 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-16T11:47:14.968+0000 7f8bb17ba700 1 --2- 172.21.15.152:0/2735553514 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f8b980a2890 0x7f8b980a4d50 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f8b94006fd0 tx=0x7f8b94008040 comp rx=0 tx=0).stop 2024-03-16T11:47:14.968 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-16T11:47:14.968+0000 7f8bb17ba700 1 -- 172.21.15.152:0/2735553514 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8bac101230 msgr2=0x7f8bac197480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:47:14.968 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-16T11:47:14.968+0000 7f8bb17ba700 1 --2- 172.21.15.152:0/2735553514 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8bac101230 0x7f8bac197480 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f8b9c00fa80 tx=0x7f8b9c00fb60 comp rx=0 tx=0).stop 2024-03-16T11:47:14.968 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-16T11:47:14.968+0000 7f8bb17ba700 1 -- 172.21.15.152:0/2735553514 shutdown_connections 2024-03-16T11:47:14.968 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-16T11:47:14.968+0000 7f8bb17ba700 1 --2- 172.21.15.152:0/2735553514 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f8b980a2890 0x7f8b980a4d50 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:47:14.968 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-16T11:47:14.968+0000 7f8bb17ba700 1 --2- 172.21.15.152:0/2735553514 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8bac102dd0 0x7f8bac19ea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:47:14.968 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-16T11:47:14.968+0000 7f8bb17ba700 1 --2- 172.21.15.152:0/2735553514 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8bac101230 0x7f8bac197480 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:47:14.968 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-16T11:47:14.968+0000 7f8bb17ba700 1 --2- 172.21.15.152:0/2735553514 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8bac102430 0x7f8bac1979c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:47:14.968 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-16T11:47:14.968+0000 7f8bb17ba700 1 -- 172.21.15.152:0/2735553514 >> 172.21.15.152:0/2735553514 conn(0x7f8bac0fc7d0 msgr2=0x7f8bac105730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:47:14.968 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-16T11:47:14.968+0000 7f8bb17ba700 1 -- 172.21.15.152:0/2735553514 shutdown_connections 2024-03-16T11:47:14.968 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-16T11:47:14.968+0000 7f8bb17ba700 1 -- 172.21.15.152:0/2735553514 wait complete. 2024-03-16T11:47:15.623 DEBUG:teuthology.orchestra.run.smithi152:osd.10> sudo journalctl -f -n 0 -u ceph-d1887f28-e389-11ee-95c9-87774f69a715@osd.10.service 2024-03-16T11:47:15.625 INFO:tasks.cephadm:Deploying osd.11 on smithi154 with /dev/vg_nvme/lv_1... 2024-03-16T11:47:15.626 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d1887f28-e389-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-16T11:47:15.874 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/d1887f28-e389-11ee-95c9-87774f69a715/mon.c/config 2024-03-16T11:47:16.239 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:47:15 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:47:16.239 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:47:15 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:47:16.239 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:47:15 smithi154 ceph-mon[36320]: from='osd.10 [v2:172.21.15.152:6824/3412525480,v1:172.21.15.152:6825/3412525480]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-16T11:47:16.239 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:47:15 smithi154 ceph-mon[36320]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-16T11:47:16.239 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:47:15 smithi154 ceph-mon[36320]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 303 MiB used, 894 GiB / 894 GiB avail 2024-03-16T11:47:16.239 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:47:15 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:47:16.239 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:47:15 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:47:16.328 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:47:15 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:47:16.329 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:47:15 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:47:16.329 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:47:15 smithi152 ceph-mon[36291]: from='osd.10 [v2:172.21.15.152:6824/3412525480,v1:172.21.15.152:6825/3412525480]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-16T11:47:16.329 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:47:15 smithi152 ceph-mon[36291]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-16T11:47:16.329 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:47:15 smithi152 ceph-mon[36291]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 303 MiB used, 894 GiB / 894 GiB avail 2024-03-16T11:47:16.329 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:47:15 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:47:16.329 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:47:15 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:47:16.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:47:15 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:47:16.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:47:15 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:47:16.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:47:15 smithi029 ceph-mon[31975]: from='osd.10 [v2:172.21.15.152:6824/3412525480,v1:172.21.15.152:6825/3412525480]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-16T11:47:16.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:47:15 smithi029 ceph-mon[31975]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-16T11:47:16.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:47:15 smithi029 ceph-mon[31975]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 303 MiB used, 894 GiB / 894 GiB avail 2024-03-16T11:47:16.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:47:15 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:47:16.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:47:15 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:47:17.234 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:47:16 smithi154 ceph-mon[36320]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-03-16T11:47:17.234 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:47:16 smithi154 ceph-mon[36320]: osdmap e70: 11 total, 10 up, 11 in 2024-03-16T11:47:17.234 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:47:16 smithi154 ceph-mon[36320]: from='osd.10 [v2:172.21.15.152:6824/3412525480,v1:172.21.15.152:6825/3412525480]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]: dispatch 2024-03-16T11:47:17.234 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:47:16 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-16T11:47:17.234 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:47:16 smithi154 ceph-mon[36320]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]: dispatch 2024-03-16T11:47:17.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:47:16 smithi029 ceph-mon[31975]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-03-16T11:47:17.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:47:16 smithi029 ceph-mon[31975]: osdmap e70: 11 total, 10 up, 11 in 2024-03-16T11:47:17.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:47:16 smithi029 ceph-mon[31975]: from='osd.10 [v2:172.21.15.152:6824/3412525480,v1:172.21.15.152:6825/3412525480]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]: dispatch 2024-03-16T11:47:17.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:47:16 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-16T11:47:17.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:47:16 smithi029 ceph-mon[31975]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]: dispatch 2024-03-16T11:47:17.378 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:47:16 smithi152 ceph-mon[36291]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-03-16T11:47:17.378 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:47:16 smithi152 ceph-mon[36291]: osdmap e70: 11 total, 10 up, 11 in 2024-03-16T11:47:17.378 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:47:16 smithi152 ceph-mon[36291]: from='osd.10 [v2:172.21.15.152:6824/3412525480,v1:172.21.15.152:6825/3412525480]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]: dispatch 2024-03-16T11:47:17.378 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:47:16 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-16T11:47:17.378 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:47:16 smithi152 ceph-mon[36291]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]: dispatch 2024-03-16T11:47:17.378 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 11:47:16 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T11:47:16.969+0000 7fe127f57700 -1 osd.10 0 waiting for initial osdmap 2024-03-16T11:47:17.378 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 11:47:16 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T11:47:16.982+0000 7fe123d4a700 -1 osd.10 71 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-16T11:47:17.964 INFO:teuthology.orchestra.run.smithi154.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-16T11:47:17.965 INFO:teuthology.orchestra.run.smithi154.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-16T11:47:17.965 INFO:teuthology.orchestra.run.smithi154.stderr: stderr: 10+0 records in 2024-03-16T11:47:17.965 INFO:teuthology.orchestra.run.smithi154.stderr:10+0 records out 2024-03-16T11:47:17.965 INFO:teuthology.orchestra.run.smithi154.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0112469 s, 932 MB/s 2024-03-16T11:47:17.965 INFO:teuthology.orchestra.run.smithi154.stderr: stderr: 2024-03-16T11:47:17.965 INFO:teuthology.orchestra.run.smithi154.stderr:--> Zapping successful for: 2024-03-16T11:47:18.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:47:17 smithi152 ceph-mon[36291]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]': finished 2024-03-16T11:47:18.128 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:47:17 smithi152 ceph-mon[36291]: osdmap e71: 11 total, 10 up, 11 in 2024-03-16T11:47:18.128 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:47:17 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-16T11:47:18.128 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:47:17 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-16T11:47:18.128 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:47:17 smithi152 ceph-mon[36291]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 303 MiB used, 894 GiB / 894 GiB avail 2024-03-16T11:47:18.269 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:47:17 smithi154 ceph-mon[36320]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]': finished 2024-03-16T11:47:18.269 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:47:17 smithi154 ceph-mon[36320]: osdmap e71: 11 total, 10 up, 11 in 2024-03-16T11:47:18.269 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:47:17 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-16T11:47:18.269 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:47:17 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-16T11:47:18.269 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:47:17 smithi154 ceph-mon[36320]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 303 MiB used, 894 GiB / 894 GiB avail 2024-03-16T11:47:18.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:47:17 smithi029 ceph-mon[31975]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]': finished 2024-03-16T11:47:18.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:47:17 smithi029 ceph-mon[31975]: osdmap e71: 11 total, 10 up, 11 in 2024-03-16T11:47:18.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:47:17 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-16T11:47:18.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:47:17 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-16T11:47:18.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:47:17 smithi029 ceph-mon[31975]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 303 MiB used, 894 GiB / 894 GiB avail 2024-03-16T11:47:18.638 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d1887f28-e389-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi154:vg_nvme/lv_1 2024-03-16T11:47:18.894 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/d1887f28-e389-11ee-95c9-87774f69a715/mon.c/config 2024-03-16T11:47:19.231 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:47:18 smithi154 ceph-mon[36320]: purged_snaps scrub starts 2024-03-16T11:47:19.231 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:47:18 smithi154 ceph-mon[36320]: purged_snaps scrub ok 2024-03-16T11:47:19.231 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:47:18 smithi154 ceph-mon[36320]: osd.10 [v2:172.21.15.152:6824/3412525480,v1:172.21.15.152:6825/3412525480] boot 2024-03-16T11:47:19.231 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:47:18 smithi154 ceph-mon[36320]: osdmap e72: 11 total, 11 up, 11 in 2024-03-16T11:47:19.231 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:47:18 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-16T11:47:19.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:47:18 smithi029 ceph-mon[31975]: purged_snaps scrub starts 2024-03-16T11:47:19.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:47:18 smithi029 ceph-mon[31975]: purged_snaps scrub ok 2024-03-16T11:47:19.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:47:18 smithi029 ceph-mon[31975]: osd.10 [v2:172.21.15.152:6824/3412525480,v1:172.21.15.152:6825/3412525480] boot 2024-03-16T11:47:19.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:47:18 smithi029 ceph-mon[31975]: osdmap e72: 11 total, 11 up, 11 in 2024-03-16T11:47:19.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:47:18 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-16T11:47:19.376 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:47:18 smithi152 ceph-mon[36291]: purged_snaps scrub starts 2024-03-16T11:47:19.376 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:47:18 smithi152 ceph-mon[36291]: purged_snaps scrub ok 2024-03-16T11:47:19.376 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:47:18 smithi152 ceph-mon[36291]: osd.10 [v2:172.21.15.152:6824/3412525480,v1:172.21.15.152:6825/3412525480] boot 2024-03-16T11:47:19.376 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:47:18 smithi152 ceph-mon[36291]: osdmap e72: 11 total, 11 up, 11 in 2024-03-16T11:47:19.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:47:18 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-16T11:47:20.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:47:19 smithi154 ceph-mon[36320]: osdmap e73: 11 total, 11 up, 11 in 2024-03-16T11:47:20.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:47:19 smithi154 ceph-mon[36320]: Detected new or changed devices on smithi152 2024-03-16T11:47:20.296 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:47:19 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:47:20.296 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:47:19 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:47:20.296 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:47:19 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-16T11:47:20.296 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:47:19 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-16T11:47:20.296 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:47:19 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-16T11:47:20.296 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:47:19 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-16T11:47:20.296 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:47:19 smithi154 ceph-mon[36320]: Adjusting osd_memory_target on smithi152 to 4270M 2024-03-16T11:47:20.296 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:47:19 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:47:20.296 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:47:19 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:47:20.296 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:47:19 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T11:47:20.296 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:47:19 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:47:20.296 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:47:19 smithi154 ceph-mon[36320]: pgmap v239: 1 pgs: 1 remapped+peering; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-03-16T11:47:20.296 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:47:19 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T11:47:20.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:47:19 smithi029 ceph-mon[31975]: osdmap e73: 11 total, 11 up, 11 in 2024-03-16T11:47:20.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:47:19 smithi029 ceph-mon[31975]: Detected new or changed devices on smithi152 2024-03-16T11:47:20.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:47:19 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:47:20.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:47:19 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:47:20.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:47:19 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-16T11:47:20.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:47:19 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-16T11:47:20.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:47:19 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-16T11:47:20.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:47:19 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-16T11:47:20.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:47:19 smithi029 ceph-mon[31975]: Adjusting osd_memory_target on smithi152 to 4270M 2024-03-16T11:47:20.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:47:19 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:47:20.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:47:19 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:47:20.339 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:47:19 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T11:47:20.339 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:47:19 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:47:20.339 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:47:19 smithi029 ceph-mon[31975]: pgmap v239: 1 pgs: 1 remapped+peering; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-03-16T11:47:20.339 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:47:19 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T11:47:20.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:47:19 smithi152 ceph-mon[36291]: osdmap e73: 11 total, 11 up, 11 in 2024-03-16T11:47:20.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:47:19 smithi152 ceph-mon[36291]: Detected new or changed devices on smithi152 2024-03-16T11:47:20.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:47:19 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:47:20.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:47:19 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:47:20.378 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:47:19 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-16T11:47:20.378 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:47:19 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-16T11:47:20.378 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:47:19 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-16T11:47:20.378 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:47:19 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-16T11:47:20.378 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:47:19 smithi152 ceph-mon[36291]: Adjusting osd_memory_target on smithi152 to 4270M 2024-03-16T11:47:20.378 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:47:19 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:47:20.378 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:47:19 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:47:20.378 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:47:19 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T11:47:20.378 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:47:19 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:47:20.378 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:47:19 smithi152 ceph-mon[36291]: pgmap v239: 1 pgs: 1 remapped+peering; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-03-16T11:47:20.378 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:47:19 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T11:47:20.708 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:47:20.706+0000 7fa8b27f0700 1 -- 172.21.15.154:0/2090955306 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa8ac102e10 msgr2=0x7fa8ac1096c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:47:20.708 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:47:20.706+0000 7fa8b27f0700 1 --2- 172.21.15.154:0/2090955306 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa8ac102e10 0x7fa8ac1096c0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fa8a400b0a0 tx=0x7fa8a400b3b0 comp rx=0 tx=0).stop 2024-03-16T11:47:20.708 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:47:20.706+0000 7fa8b27f0700 1 -- 172.21.15.154:0/2090955306 shutdown_connections 2024-03-16T11:47:20.709 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:47:20.706+0000 7fa8b27f0700 1 --2- 172.21.15.154:0/2090955306 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa8ac102e10 0x7fa8ac1096c0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:47:20.709 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:47:20.706+0000 7fa8b27f0700 1 --2- 172.21.15.154:0/2090955306 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa8ac101270 0x7fa8ac101690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:47:20.709 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:47:20.706+0000 7fa8b27f0700 1 --2- 172.21.15.154:0/2090955306 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa8ac102470 0x7fa8ac1028d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:47:20.709 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:47:20.706+0000 7fa8b27f0700 1 -- 172.21.15.154:0/2090955306 >> 172.21.15.154:0/2090955306 conn(0x7fa8ac0fc830 msgr2=0x7fa8ac0fec50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:47:20.709 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:47:20.706+0000 7fa8b27f0700 1 -- 172.21.15.154:0/2090955306 shutdown_connections 2024-03-16T11:47:20.709 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:47:20.706+0000 7fa8b27f0700 1 -- 172.21.15.154:0/2090955306 wait complete. 2024-03-16T11:47:20.709 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:47:20.706+0000 7fa8b27f0700 1 Processor -- start 2024-03-16T11:47:20.709 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:47:20.706+0000 7fa8b27f0700 1 -- start start 2024-03-16T11:47:20.709 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:47:20.707+0000 7fa8b27f0700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa8ac101270 0x7fa8ac198460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:47:20.709 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:47:20.707+0000 7fa8b27f0700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa8ac102470 0x7fa8ac1989a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:47:20.709 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:47:20.707+0000 7fa8b27f0700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa8ac102e10 0x7fa8ac19fa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:47:20.709 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:47:20.707+0000 7fa8b27f0700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fa8ac10df60 con 0x7fa8ac101270 2024-03-16T11:47:20.709 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:47:20.707+0000 7fa8b27f0700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fa8ac10dde0 con 0x7fa8ac102e10 2024-03-16T11:47:20.709 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:47:20.707+0000 7fa8b27f0700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fa8ac10e0e0 con 0x7fa8ac102470 2024-03-16T11:47:20.710 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:47:20.707+0000 7fa8ab7fe700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa8ac102470 0x7fa8ac1989a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:47:20.710 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:47:20.707+0000 7fa8ab7fe700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa8ac102470 0x7fa8ac1989a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.154:42306/0 (socket says 172.21.15.154:42306) 2024-03-16T11:47:20.710 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:47:20.707+0000 7fa8ab7fe700 1 -- 172.21.15.154:0/278079963 learned_addr learned my addr 172.21.15.154:0/278079963 (peer_addr_for_me v2:172.21.15.154:0/0) 2024-03-16T11:47:20.710 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:47:20.707+0000 7fa8b0d8d700 1 --2- 172.21.15.154:0/278079963 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa8ac102e10 0x7fa8ac19fa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:47:20.710 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:47:20.707+0000 7fa8abfff700 1 --2- 172.21.15.154:0/278079963 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa8ac101270 0x7fa8ac198460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:47:20.710 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:47:20.707+0000 7fa8ab7fe700 1 -- 172.21.15.154:0/278079963 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa8ac102e10 msgr2=0x7fa8ac19fa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:47:20.710 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:47:20.707+0000 7fa8ab7fe700 1 --2- 172.21.15.154:0/278079963 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa8ac102e10 0x7fa8ac19fa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:47:20.710 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:47:20.707+0000 7fa8ab7fe700 1 -- 172.21.15.154:0/278079963 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa8ac101270 msgr2=0x7fa8ac198460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:47:20.710 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:47:20.707+0000 7fa8ab7fe700 1 --2- 172.21.15.154:0/278079963 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa8ac101270 0x7fa8ac198460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:47:20.710 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:47:20.707+0000 7fa8ab7fe700 1 -- 172.21.15.154:0/278079963 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8ac1a00c0 con 0x7fa8ac102470 2024-03-16T11:47:20.710 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:47:20.707+0000 7fa8ab7fe700 1 --2- 172.21.15.154:0/278079963 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa8ac102470 0x7fa8ac1989a0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fa8a000d9b0 tx=0x7fa8a000dcc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:47:20.710 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:47:20.707+0000 7fa8a97fa700 1 -- 172.21.15.154:0/278079963 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8a0016070 con 0x7fa8ac102470 2024-03-16T11:47:20.710 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:47:20.708+0000 7fa8b27f0700 1 -- 172.21.15.154:0/278079963 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa8ac1a03b0 con 0x7fa8ac102470 2024-03-16T11:47:20.711 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:47:20.708+0000 7fa8a97fa700 1 -- 172.21.15.154:0/278079963 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa8a0004d10 con 0x7fa8ac102470 2024-03-16T11:47:20.711 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:47:20.708+0000 7fa8a97fa700 1 -- 172.21.15.154:0/278079963 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8a0005630 con 0x7fa8ac102470 2024-03-16T11:47:20.711 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:47:20.708+0000 7fa8b27f0700 1 -- 172.21.15.154:0/278079963 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa8ac1a08f0 con 0x7fa8ac102470 2024-03-16T11:47:20.711 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:47:20.709+0000 7fa8b27f0700 1 -- 172.21.15.154:0/278079963 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa8ac068f90 con 0x7fa8ac102470 2024-03-16T11:47:20.711 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:47:20.710+0000 7fa8a97fa700 1 -- 172.21.15.154:0/278079963 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fa8a0004750 con 0x7fa8ac102470 2024-03-16T11:47:20.711 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:47:20.711+0000 7fa8a97fa700 1 --2- 172.21.15.154:0/278079963 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fa8940a2890 0x7fa8940a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:47:20.711 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:47:20.711+0000 7fa8abfff700 1 --2- 172.21.15.154:0/278079963 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fa8940a2890 0x7fa8940a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:47:20.712 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:47:20.711+0000 7fa8a97fa700 1 -- 172.21.15.154:0/278079963 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 6723+0+0 (secure 0 0 0) 0x7fa8a00ce250 con 0x7fa8ac102470 2024-03-16T11:47:20.712 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:47:20.711+0000 7fa8abfff700 1 --2- 172.21.15.154:0/278079963 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fa8940a2890 0x7fa8940a4d50 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fa89c005950 tx=0x7fa89c0058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:47:20.713 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:47:20.715+0000 7fa8a97fa700 1 -- 172.21.15.154:0/278079963 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fa8a001f020 con 0x7fa8ac102470 2024-03-16T11:47:20.918 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:47:20.919+0000 7fa8b27f0700 1 -- 172.21.15.154:0/278079963 --> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi154:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7fa8ac062710 con 0x7fa8940a2890 2024-03-16T11:47:21.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:47:20 smithi154 ceph-mon[36320]: osdmap e74: 11 total, 11 up, 11 in 2024-03-16T11:47:21.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:47:20 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:47:21.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:47:20 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T11:47:21.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:47:20 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:47:21.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:47:20 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T11:47:21.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:47:20 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T11:47:21.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:47:20 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:47:21.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:47:20 smithi029 ceph-mon[31975]: osdmap e74: 11 total, 11 up, 11 in 2024-03-16T11:47:21.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:47:20 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:47:21.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:47:20 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T11:47:21.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:47:20 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:47:21.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:47:20 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T11:47:21.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:47:20 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T11:47:21.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:47:20 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:47:21.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:47:20 smithi152 ceph-mon[36291]: osdmap e74: 11 total, 11 up, 11 in 2024-03-16T11:47:21.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:47:20 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:47:21.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:47:20 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T11:47:21.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:47:20 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:47:21.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:47:20 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T11:47:21.378 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:47:20 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T11:47:21.378 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:47:20 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:47:22.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:47:21 smithi154 ceph-mon[36320]: from='client.24361 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi154:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T11:47:22.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:47:21 smithi154 ceph-mon[36320]: osdmap e75: 11 total, 11 up, 11 in 2024-03-16T11:47:22.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:47:21 smithi154 ceph-mon[36320]: pgmap v242: 1 pgs: 1 remapped+peering; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-03-16T11:47:22.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:47:21 smithi029 ceph-mon[31975]: from='client.24361 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi154:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T11:47:22.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:47:21 smithi029 ceph-mon[31975]: osdmap e75: 11 total, 11 up, 11 in 2024-03-16T11:47:22.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:47:21 smithi029 ceph-mon[31975]: pgmap v242: 1 pgs: 1 remapped+peering; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-03-16T11:47:22.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:47:21 smithi152 ceph-mon[36291]: from='client.24361 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi154:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T11:47:22.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:47:21 smithi152 ceph-mon[36291]: osdmap e75: 11 total, 11 up, 11 in 2024-03-16T11:47:22.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:47:21 smithi152 ceph-mon[36291]: pgmap v242: 1 pgs: 1 remapped+peering; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-03-16T11:47:23.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:47:22 smithi152 ceph-mon[36291]: osdmap e76: 11 total, 11 up, 11 in 2024-03-16T11:47:23.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:47:22 smithi154 ceph-mon[36320]: osdmap e76: 11 total, 11 up, 11 in 2024-03-16T11:47:23.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:47:22 smithi029 ceph-mon[31975]: osdmap e76: 11 total, 11 up, 11 in 2024-03-16T11:47:24.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:47:24 smithi154 ceph-mon[36320]: pgmap v244: 1 pgs: 1 remapped+peering; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-03-16T11:47:24.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:47:24 smithi154 ceph-mon[36320]: from='client.? 172.21.15.154:0/1611285094' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4598c803-984a-4de6-ac22-066502c17889"}]: dispatch 2024-03-16T11:47:24.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:47:24 smithi154 ceph-mon[36320]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4598c803-984a-4de6-ac22-066502c17889"}]: dispatch 2024-03-16T11:47:24.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:47:24 smithi154 ceph-mon[36320]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4598c803-984a-4de6-ac22-066502c17889"}]': finished 2024-03-16T11:47:24.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:47:24 smithi154 ceph-mon[36320]: osdmap e77: 12 total, 11 up, 12 in 2024-03-16T11:47:24.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:47:24 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-16T11:47:24.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:47:24 smithi029 ceph-mon[31975]: pgmap v244: 1 pgs: 1 remapped+peering; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-03-16T11:47:24.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:47:24 smithi029 ceph-mon[31975]: from='client.? 172.21.15.154:0/1611285094' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4598c803-984a-4de6-ac22-066502c17889"}]: dispatch 2024-03-16T11:47:24.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:47:24 smithi029 ceph-mon[31975]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4598c803-984a-4de6-ac22-066502c17889"}]: dispatch 2024-03-16T11:47:24.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:47:24 smithi029 ceph-mon[31975]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4598c803-984a-4de6-ac22-066502c17889"}]': finished 2024-03-16T11:47:24.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:47:24 smithi029 ceph-mon[31975]: osdmap e77: 12 total, 11 up, 12 in 2024-03-16T11:47:24.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:47:24 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-16T11:47:24.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:47:24 smithi152 ceph-mon[36291]: pgmap v244: 1 pgs: 1 remapped+peering; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-03-16T11:47:24.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:47:24 smithi152 ceph-mon[36291]: from='client.? 172.21.15.154:0/1611285094' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4598c803-984a-4de6-ac22-066502c17889"}]: dispatch 2024-03-16T11:47:24.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:47:24 smithi152 ceph-mon[36291]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4598c803-984a-4de6-ac22-066502c17889"}]: dispatch 2024-03-16T11:47:24.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:47:24 smithi152 ceph-mon[36291]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4598c803-984a-4de6-ac22-066502c17889"}]': finished 2024-03-16T11:47:24.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:47:24 smithi152 ceph-mon[36291]: osdmap e77: 12 total, 11 up, 12 in 2024-03-16T11:47:24.378 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:47:24 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-16T11:47:25.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:47:25 smithi154 ceph-mon[36320]: from='client.? 172.21.15.154:0/3637729188' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T11:47:25.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:47:25 smithi029 ceph-mon[31975]: from='client.? 172.21.15.154:0/3637729188' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T11:47:25.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:47:25 smithi152 ceph-mon[36291]: from='client.? 172.21.15.154:0/3637729188' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T11:47:26.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:47:26 smithi154 ceph-mon[36320]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail; 103 KiB/s, 0 objects/s recovering 2024-03-16T11:47:26.337 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:47:26 smithi029 ceph-mon[31975]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail; 103 KiB/s, 0 objects/s recovering 2024-03-16T11:47:26.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:47:26 smithi152 ceph-mon[36291]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail; 103 KiB/s, 0 objects/s recovering 2024-03-16T11:47:28.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:47:27 smithi154 ceph-mon[36320]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail; 87 KiB/s, 0 objects/s recovering 2024-03-16T11:47:28.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:47:27 smithi029 ceph-mon[31975]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail; 87 KiB/s, 0 objects/s recovering 2024-03-16T11:47:28.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:47:27 smithi152 ceph-mon[36291]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail; 87 KiB/s, 0 objects/s recovering 2024-03-16T11:47:30.001 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:47:29 smithi154 ceph-mon[36320]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-16T11:47:30.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:47:29 smithi029 ceph-mon[31975]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-16T11:47:30.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:47:29 smithi152 ceph-mon[36291]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-16T11:47:31.545 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:47:31 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-03-16T11:47:31.545 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:47:31 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:47:31.587 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:47:31 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-03-16T11:47:31.588 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:47:31 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:47:31.627 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:47:31 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-03-16T11:47:31.627 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:47:31 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:47:32.545 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:47:32 smithi154 ceph-mon[36320]: Deploying daemon osd.11 on smithi154 2024-03-16T11:47:32.545 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:47:32 smithi154 ceph-mon[36320]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail; 60 KiB/s, 0 objects/s recovering 2024-03-16T11:47:32.588 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:47:32 smithi029 ceph-mon[31975]: Deploying daemon osd.11 on smithi154 2024-03-16T11:47:32.588 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:47:32 smithi029 ceph-mon[31975]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail; 60 KiB/s, 0 objects/s recovering 2024-03-16T11:47:32.627 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:47:32 smithi152 ceph-mon[36291]: Deploying daemon osd.11 on smithi154 2024-03-16T11:47:32.627 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:47:32 smithi152 ceph-mon[36291]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail; 60 KiB/s, 0 objects/s recovering 2024-03-16T11:47:33.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:47:33 smithi154 ceph-mon[36320]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-16T11:47:34.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:47:33 smithi029 ceph-mon[31975]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-16T11:47:34.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:47:33 smithi152 ceph-mon[36291]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-16T11:47:35.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:47:35 smithi154 ceph-mon[36320]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-03-16T11:47:36.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:47:35 smithi029 ceph-mon[31975]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-03-16T11:47:36.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:47:35 smithi152 ceph-mon[36291]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-03-16T11:47:38.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:47:38 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:47:38.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:47:38 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:47:38.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:47:38 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T11:47:38.796 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:47:38 smithi154 ceph-mon[36320]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-03-16T11:47:38.837 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:47:38 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:47:38.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:47:38 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:47:38.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:47:38 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T11:47:38.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:47:38 smithi029 ceph-mon[31975]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-03-16T11:47:38.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:47:38 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:47:38.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:47:38 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:47:38.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:47:38 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T11:47:38.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:47:38 smithi152 ceph-mon[36291]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-03-16T11:47:39.923 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:47:39 smithi154 ceph-mon[36320]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-03-16T11:47:40.087 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:47:39 smithi029 ceph-mon[31975]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-03-16T11:47:40.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:47:39 smithi152 ceph-mon[36291]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-03-16T11:47:40.983 INFO:teuthology.orchestra.run.smithi154.stdout:Created osd(s) 11 on host 'smithi154' 2024-03-16T11:47:40.984 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:47:40.983+0000 7fa8a97fa700 1 -- 172.21.15.154:0/278079963 <== mgr.14152 v2:172.21.15.29:6800/4192064640 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7fa8ac062710 con 0x7fa8940a2890 2024-03-16T11:47:40.985 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:47:40.986+0000 7fa8b27f0700 1 -- 172.21.15.154:0/278079963 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fa8940a2890 msgr2=0x7fa8940a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:47:40.985 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:47:40.986+0000 7fa8b27f0700 1 --2- 172.21.15.154:0/278079963 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fa8940a2890 0x7fa8940a4d50 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fa89c005950 tx=0x7fa89c0058e0 comp rx=0 tx=0).stop 2024-03-16T11:47:40.985 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:47:40.986+0000 7fa8b27f0700 1 -- 172.21.15.154:0/278079963 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa8ac102470 msgr2=0x7fa8ac1989a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:47:40.986 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:47:40.986+0000 7fa8b27f0700 1 --2- 172.21.15.154:0/278079963 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa8ac102470 0x7fa8ac1989a0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fa8a000d9b0 tx=0x7fa8a000dcc0 comp rx=0 tx=0).stop 2024-03-16T11:47:40.986 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:47:40.986+0000 7fa8b27f0700 1 -- 172.21.15.154:0/278079963 shutdown_connections 2024-03-16T11:47:40.986 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:47:40.986+0000 7fa8b27f0700 1 --2- 172.21.15.154:0/278079963 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa8ac102e10 0x7fa8ac19fa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:47:40.986 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:47:40.986+0000 7fa8b27f0700 1 --2- 172.21.15.154:0/278079963 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa8ac101270 0x7fa8ac198460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:47:40.986 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:47:40.986+0000 7fa8b27f0700 1 --2- 172.21.15.154:0/278079963 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fa8940a2890 0x7fa8940a4d50 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:47:40.986 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:47:40.986+0000 7fa8b27f0700 1 --2- 172.21.15.154:0/278079963 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa8ac102470 0x7fa8ac1989a0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:47:40.986 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:47:40.986+0000 7fa8b27f0700 1 -- 172.21.15.154:0/278079963 >> 172.21.15.154:0/278079963 conn(0x7fa8ac0fc830 msgr2=0x7fa8ac105710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:47:40.986 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:47:40.987+0000 7fa8b27f0700 1 -- 172.21.15.154:0/278079963 shutdown_connections 2024-03-16T11:47:40.986 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:47:40.987+0000 7fa8b27f0700 1 -- 172.21.15.154:0/278079963 wait complete. 2024-03-16T11:47:41.694 DEBUG:teuthology.orchestra.run.smithi154:osd.11> sudo journalctl -f -n 0 -u ceph-d1887f28-e389-11ee-95c9-87774f69a715@osd.11.service 2024-03-16T11:47:41.696 INFO:tasks.cephadm:Waiting for 12 OSDs to come up... 2024-03-16T11:47:41.697 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d1887f28-e389-11ee-95c9-87774f69a715 -- ceph osd stat -f json 2024-03-16T11:47:41.954 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/d1887f28-e389-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T11:47:42.076 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:47:41 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:47:42.076 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:47:41 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:47:42.076 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:47:41 smithi029 ceph-mon[31975]: from='osd.11 [v2:172.21.15.154:6824/1160927966,v1:172.21.15.154:6825/1160927966]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-16T11:47:42.076 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:47:41 smithi029 ceph-mon[31975]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-16T11:47:42.076 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:47:41 smithi029 ceph-mon[31975]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-03-16T11:47:42.076 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:47:41 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:47:42.076 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:47:41 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:47:42.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:47:41 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:47:42.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:47:41 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:47:42.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:47:41 smithi154 ceph-mon[36320]: from='osd.11 [v2:172.21.15.154:6824/1160927966,v1:172.21.15.154:6825/1160927966]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-16T11:47:42.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:47:41 smithi154 ceph-mon[36320]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-16T11:47:42.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:47:41 smithi154 ceph-mon[36320]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-03-16T11:47:42.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:47:41 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:47:42.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:47:41 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:47:42.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:47:41 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:47:42.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:47:41 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:47:42.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:47:41 smithi152 ceph-mon[36291]: from='osd.11 [v2:172.21.15.154:6824/1160927966,v1:172.21.15.154:6825/1160927966]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-16T11:47:42.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:47:41 smithi152 ceph-mon[36291]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-16T11:47:42.378 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:47:41 smithi152 ceph-mon[36291]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-03-16T11:47:42.378 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:47:41 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:47:42.378 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:47:41 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:47:43.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:47:42 smithi154 ceph-mon[36320]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-03-16T11:47:43.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:47:42 smithi154 ceph-mon[36320]: from='osd.11 [v2:172.21.15.154:6824/1160927966,v1:172.21.15.154:6825/1160927966]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]: dispatch 2024-03-16T11:47:43.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:47:42 smithi154 ceph-mon[36320]: osdmap e78: 12 total, 11 up, 12 in 2024-03-16T11:47:43.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:47:42 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-16T11:47:43.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:47:42 smithi154 ceph-mon[36320]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]: dispatch 2024-03-16T11:47:43.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:47:42 smithi029 ceph-mon[31975]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-03-16T11:47:43.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:47:42 smithi029 ceph-mon[31975]: from='osd.11 [v2:172.21.15.154:6824/1160927966,v1:172.21.15.154:6825/1160927966]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]: dispatch 2024-03-16T11:47:43.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:47:42 smithi029 ceph-mon[31975]: osdmap e78: 12 total, 11 up, 12 in 2024-03-16T11:47:43.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:47:42 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-16T11:47:43.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:47:42 smithi029 ceph-mon[31975]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]: dispatch 2024-03-16T11:47:43.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:47:42 smithi152 ceph-mon[36291]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-03-16T11:47:43.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:47:42 smithi152 ceph-mon[36291]: from='osd.11 [v2:172.21.15.154:6824/1160927966,v1:172.21.15.154:6825/1160927966]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]: dispatch 2024-03-16T11:47:43.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:47:42 smithi152 ceph-mon[36291]: osdmap e78: 12 total, 11 up, 12 in 2024-03-16T11:47:43.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:47:42 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-16T11:47:43.378 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:47:42 smithi152 ceph-mon[36291]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]: dispatch 2024-03-16T11:47:43.653 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 11:47:43 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T11:47:43.346+0000 7f8cfb053700 -1 osd.11 0 waiting for initial osdmap 2024-03-16T11:47:43.653 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 11:47:43 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T11:47:43.359+0000 7f8cf2e3e700 -1 osd.11 79 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-16T11:47:43.972 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:47:43 smithi029 ceph-mon[31975]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]': finished 2024-03-16T11:47:43.972 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:47:43 smithi029 ceph-mon[31975]: osdmap e79: 12 total, 11 up, 12 in 2024-03-16T11:47:43.972 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:47:43 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-16T11:47:43.973 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:47:43 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-16T11:47:43.973 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:47:43 smithi029 ceph-mon[31975]: from='osd.11 ' entity='osd.11' 2024-03-16T11:47:43.973 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:47:43 smithi029 ceph-mon[31975]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-03-16T11:47:43.974 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:43.972+0000 7f2bc4cd2700 1 -- 172.21.15.29:0/1834496813 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2bc01056f0 msgr2=0x7f2bc0107ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:47:43.975 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:43.972+0000 7f2bc4cd2700 1 --2- 172.21.15.29:0/1834496813 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2bc01056f0 0x7f2bc0107ae0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f2bb4009990 tx=0x7f2bb4009ca0 comp rx=0 tx=0).stop 2024-03-16T11:47:43.975 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:43.972+0000 7f2bc4cd2700 1 -- 172.21.15.29:0/1834496813 shutdown_connections 2024-03-16T11:47:43.975 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:43.972+0000 7f2bc4cd2700 1 --2- 172.21.15.29:0/1834496813 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2bc0108020 0x7f2bc010a490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:47:43.975 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:43.972+0000 7f2bc4cd2700 1 --2- 172.21.15.29:0/1834496813 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2bc0102dc0 0x7f2bc01051b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:47:43.975 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:43.972+0000 7f2bc4cd2700 1 --2- 172.21.15.29:0/1834496813 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2bc01056f0 0x7f2bc0107ae0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:47:43.975 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:43.972+0000 7f2bc4cd2700 1 -- 172.21.15.29:0/1834496813 >> 172.21.15.29:0/1834496813 conn(0x7f2bc00fc7f0 msgr2=0x7f2bc00fec10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:47:43.975 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:43.972+0000 7f2bc4cd2700 1 -- 172.21.15.29:0/1834496813 shutdown_connections 2024-03-16T11:47:43.975 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:43.972+0000 7f2bc4cd2700 1 -- 172.21.15.29:0/1834496813 wait complete. 2024-03-16T11:47:43.975 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:43.973+0000 7f2bc4cd2700 1 Processor -- start 2024-03-16T11:47:43.975 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:43.973+0000 7f2bc4cd2700 1 -- start start 2024-03-16T11:47:43.975 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:43.973+0000 7f2bc4cd2700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2bc0102dc0 0x7f2bc0198590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:47:43.975 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:43.973+0000 7f2bc4cd2700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2bc01056f0 0x7f2bc0198ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:47:43.975 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:43.973+0000 7f2bc4cd2700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2bc0108020 0x7f2bc019fb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:47:43.976 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:43.973+0000 7f2bc4cd2700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f2bc010df90 con 0x7f2bc0108020 2024-03-16T11:47:43.976 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:43.973+0000 7f2bc4cd2700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f2bc010de10 con 0x7f2bc0102dc0 2024-03-16T11:47:43.976 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:43.973+0000 7f2bc4cd2700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f2bc010e110 con 0x7f2bc01056f0 2024-03-16T11:47:43.976 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:43.973+0000 7f2bbed9d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2bc0108020 0x7f2bc019fb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:47:43.976 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:43.973+0000 7f2bbe59c700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2bc0102dc0 0x7f2bc0198590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:47:43.976 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:43.973+0000 7f2bbed9d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2bc0108020 0x7f2bc019fb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36680/0 (socket says 172.21.15.29:36680) 2024-03-16T11:47:43.976 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:43.973+0000 7f2bbed9d700 1 -- 172.21.15.29:0/2241622383 learned_addr learned my addr 172.21.15.29:0/2241622383 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:47:43.976 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:43.973+0000 7f2bbdd9b700 1 --2- 172.21.15.29:0/2241622383 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2bc01056f0 0x7f2bc0198ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:47:43.976 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:43.974+0000 7f2bbed9d700 1 -- 172.21.15.29:0/2241622383 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2bc01056f0 msgr2=0x7f2bc0198ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:47:43.976 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:43.974+0000 7f2bbed9d700 1 --2- 172.21.15.29:0/2241622383 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2bc01056f0 0x7f2bc0198ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:47:43.976 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:43.974+0000 7f2bbed9d700 1 -- 172.21.15.29:0/2241622383 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2bc0102dc0 msgr2=0x7f2bc0198590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:47:43.976 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:43.974+0000 7f2bbed9d700 1 --2- 172.21.15.29:0/2241622383 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2bc0102dc0 0x7f2bc0198590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:47:43.976 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:43.974+0000 7f2bbed9d700 1 -- 172.21.15.29:0/2241622383 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2bc01a01f0 con 0x7f2bc0108020 2024-03-16T11:47:43.977 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:43.974+0000 7f2bbed9d700 1 --2- 172.21.15.29:0/2241622383 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2bc0108020 0x7f2bc019fb50 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f2bb800efc0 tx=0x7f2bb800c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:47:43.977 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:43.974+0000 7f2baf7fe700 1 -- 172.21.15.29:0/2241622383 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bb80092e0 con 0x7f2bc0108020 2024-03-16T11:47:43.977 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:43.974+0000 7f2baf7fe700 1 -- 172.21.15.29:0/2241622383 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2bb8004bb0 con 0x7f2bc0108020 2024-03-16T11:47:43.977 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:43.974+0000 7f2bc4cd2700 1 -- 172.21.15.29:0/2241622383 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2bc01a0480 con 0x7f2bc0108020 2024-03-16T11:47:43.977 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:43.974+0000 7f2bc4cd2700 1 -- 172.21.15.29:0/2241622383 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2bc01a0970 con 0x7f2bc0108020 2024-03-16T11:47:43.977 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:43.975+0000 7f2baf7fe700 1 -- 172.21.15.29:0/2241622383 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bb8004280 con 0x7f2bc0108020 2024-03-16T11:47:43.977 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:43.975+0000 7f2bc4cd2700 1 -- 172.21.15.29:0/2241622383 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2bc0068f90 con 0x7f2bc0108020 2024-03-16T11:47:43.978 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:43.977+0000 7f2baf7fe700 1 -- 172.21.15.29:0/2241622383 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f2bb801b030 con 0x7f2bc0108020 2024-03-16T11:47:43.978 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:43.978+0000 7f2baf7fe700 1 --2- 172.21.15.29:0/2241622383 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f2ba80a2770 0x7f2ba80a4c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:47:43.978 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:43.978+0000 7f2bbe59c700 1 --2- 172.21.15.29:0/2241622383 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f2ba80a2770 0x7f2ba80a4c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:47:43.978 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:43.978+0000 7f2baf7fe700 1 -- 172.21.15.29:0/2241622383 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 6847+0+0 (secure 0 0 0) 0x7f2bb80ce900 con 0x7f2bc0108020 2024-03-16T11:47:43.978 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:43.978+0000 7f2bbe59c700 1 --2- 172.21.15.29:0/2241622383 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f2ba80a2770 0x7f2ba80a4c30 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f2bb0009c60 tx=0x7f2bb0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:47:43.981 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:43.982+0000 7f2baf7fe700 1 -- 172.21.15.29:0/2241622383 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f2bb8022020 con 0x7f2bc0108020 2024-03-16T11:47:44.188 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:44.189+0000 7f2bc4cd2700 1 -- 172.21.15.29:0/2241622383 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) -- 0x7f2bc002db70 con 0x7f2bc0108020 2024-03-16T11:47:44.188 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:44.189+0000 7f2baf7fe700 1 -- 172.21.15.29:0/2241622383 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v79) ==== 74+0+133 (secure 0 0 0) 0x7f2bb8010040 con 0x7f2bc0108020 2024-03-16T11:47:44.188 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-16T11:47:44.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:44.191+0000 7f2bc4cd2700 1 -- 172.21.15.29:0/2241622383 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f2ba80a2770 msgr2=0x7f2ba80a4c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:47:44.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:44.191+0000 7f2bc4cd2700 1 --2- 172.21.15.29:0/2241622383 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f2ba80a2770 0x7f2ba80a4c30 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f2bb0009c60 tx=0x7f2bb0009260 comp rx=0 tx=0).stop 2024-03-16T11:47:44.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:44.191+0000 7f2bc4cd2700 1 -- 172.21.15.29:0/2241622383 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2bc0108020 msgr2=0x7f2bc019fb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:47:44.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:44.191+0000 7f2bc4cd2700 1 --2- 172.21.15.29:0/2241622383 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2bc0108020 0x7f2bc019fb50 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f2bb800efc0 tx=0x7f2bb800c3a0 comp rx=0 tx=0).stop 2024-03-16T11:47:44.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:44.191+0000 7f2bc4cd2700 1 -- 172.21.15.29:0/2241622383 shutdown_connections 2024-03-16T11:47:44.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:44.191+0000 7f2bc4cd2700 1 --2- 172.21.15.29:0/2241622383 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2bc0108020 0x7f2bc019fb50 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:47:44.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:44.191+0000 7f2bc4cd2700 1 --2- 172.21.15.29:0/2241622383 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2bc0102dc0 0x7f2bc0198590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:47:44.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:44.191+0000 7f2bc4cd2700 1 --2- 172.21.15.29:0/2241622383 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f2ba80a2770 0x7f2ba80a4c30 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:47:44.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:44.191+0000 7f2bc4cd2700 1 --2- 172.21.15.29:0/2241622383 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2bc01056f0 0x7f2bc0198ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:47:44.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:44.191+0000 7f2bc4cd2700 1 -- 172.21.15.29:0/2241622383 >> 172.21.15.29:0/2241622383 conn(0x7f2bc00fc7f0 msgr2=0x7f2bc0103820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:47:44.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:44.191+0000 7f2bc4cd2700 1 -- 172.21.15.29:0/2241622383 shutdown_connections 2024-03-16T11:47:44.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:44.191+0000 7f2bc4cd2700 1 -- 172.21.15.29:0/2241622383 wait complete. 2024-03-16T11:47:44.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:47:43 smithi154 ceph-mon[36320]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]': finished 2024-03-16T11:47:44.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:47:43 smithi154 ceph-mon[36320]: osdmap e79: 12 total, 11 up, 12 in 2024-03-16T11:47:44.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:47:43 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-16T11:47:44.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:47:43 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-16T11:47:44.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:47:43 smithi154 ceph-mon[36320]: from='osd.11 ' entity='osd.11' 2024-03-16T11:47:44.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:47:43 smithi154 ceph-mon[36320]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-03-16T11:47:44.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:47:43 smithi152 ceph-mon[36291]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]': finished 2024-03-16T11:47:44.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:47:43 smithi152 ceph-mon[36291]: osdmap e79: 12 total, 11 up, 12 in 2024-03-16T11:47:44.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:47:43 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-16T11:47:44.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:47:43 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-16T11:47:44.378 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:47:43 smithi152 ceph-mon[36291]: from='osd.11 ' entity='osd.11' 2024-03-16T11:47:44.378 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:47:43 smithi152 ceph-mon[36291]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-03-16T11:47:44.909 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":79,"num_osds":12,"num_up_osds":11,"osd_up_since":1710589637,"num_in_osds":12,"osd_in_since":1710589643,"num_remapped_pgs":0} 2024-03-16T11:47:45.236 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:47:44 smithi154 ceph-mon[36320]: purged_snaps scrub starts 2024-03-16T11:47:45.236 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:47:44 smithi154 ceph-mon[36320]: purged_snaps scrub ok 2024-03-16T11:47:45.237 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:47:44 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-16T11:47:45.237 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:47:44 smithi154 ceph-mon[36320]: from='client.? 172.21.15.29:0/2241622383' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-16T11:47:45.237 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:47:44 smithi154 ceph-mon[36320]: osd.11 [v2:172.21.15.154:6824/1160927966,v1:172.21.15.154:6825/1160927966] boot 2024-03-16T11:47:45.237 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:47:44 smithi154 ceph-mon[36320]: osdmap e80: 12 total, 12 up, 12 in 2024-03-16T11:47:45.237 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:47:44 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-16T11:47:45.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:47:44 smithi029 ceph-mon[31975]: purged_snaps scrub starts 2024-03-16T11:47:45.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:47:44 smithi029 ceph-mon[31975]: purged_snaps scrub ok 2024-03-16T11:47:45.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:47:44 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-16T11:47:45.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:47:44 smithi029 ceph-mon[31975]: from='client.? 172.21.15.29:0/2241622383' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-16T11:47:45.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:47:44 smithi029 ceph-mon[31975]: osd.11 [v2:172.21.15.154:6824/1160927966,v1:172.21.15.154:6825/1160927966] boot 2024-03-16T11:47:45.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:47:44 smithi029 ceph-mon[31975]: osdmap e80: 12 total, 12 up, 12 in 2024-03-16T11:47:45.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:47:44 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-16T11:47:45.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:47:44 smithi152 ceph-mon[36291]: purged_snaps scrub starts 2024-03-16T11:47:45.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:47:44 smithi152 ceph-mon[36291]: purged_snaps scrub ok 2024-03-16T11:47:45.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:47:44 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-16T11:47:45.378 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:47:44 smithi152 ceph-mon[36291]: from='client.? 172.21.15.29:0/2241622383' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-16T11:47:45.378 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:47:44 smithi152 ceph-mon[36291]: osd.11 [v2:172.21.15.154:6824/1160927966,v1:172.21.15.154:6825/1160927966] boot 2024-03-16T11:47:45.378 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:47:44 smithi152 ceph-mon[36291]: osdmap e80: 12 total, 12 up, 12 in 2024-03-16T11:47:45.378 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:47:44 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-16T11:47:45.910 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d1887f28-e389-11ee-95c9-87774f69a715 -- ceph osd stat -f json 2024-03-16T11:47:46.162 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/d1887f28-e389-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T11:47:46.560 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:47:46 smithi029 ceph-mon[31975]: Detected new or changed devices on smithi154 2024-03-16T11:47:46.561 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:47:46 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:47:46.561 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:47:46 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:47:46.561 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:47:46 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-16T11:47:46.561 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:47:46 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-16T11:47:46.561 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:47:46 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-16T11:47:46.561 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:47:46 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-16T11:47:46.561 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:47:46 smithi029 ceph-mon[31975]: Adjusting osd_memory_target on smithi154 to 4270M 2024-03-16T11:47:46.561 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:47:46 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:47:46.561 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:47:46 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:47:46.561 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:47:46 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T11:47:46.561 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:47:46 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:47:46.561 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:47:46 smithi029 ceph-mon[31975]: osdmap e81: 12 total, 12 up, 12 in 2024-03-16T11:47:46.561 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:47:46 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T11:47:46.561 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:47:46 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:47:46.561 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:47:46 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T11:47:46.562 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:47:46 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:47:46.562 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:47:46 smithi029 ceph-mon[31975]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T11:47:46.627 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:47:46 smithi152 ceph-mon[36291]: Detected new or changed devices on smithi154 2024-03-16T11:47:46.628 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:47:46 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:47:46.628 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:47:46 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:47:46.628 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:47:46 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-16T11:47:46.628 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:47:46 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-16T11:47:46.628 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:47:46 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-16T11:47:46.628 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:47:46 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-16T11:47:46.628 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:47:46 smithi152 ceph-mon[36291]: Adjusting osd_memory_target on smithi154 to 4270M 2024-03-16T11:47:46.628 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:47:46 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:47:46.628 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:47:46 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:47:46.628 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:47:46 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T11:47:46.628 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:47:46 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:47:46.628 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:47:46 smithi152 ceph-mon[36291]: osdmap e81: 12 total, 12 up, 12 in 2024-03-16T11:47:46.628 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:47:46 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T11:47:46.628 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:47:46 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:47:46.629 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:47:46 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T11:47:46.629 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:47:46 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:47:46.629 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:47:46 smithi152 ceph-mon[36291]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T11:47:46.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:47:46 smithi154 ceph-mon[36320]: Detected new or changed devices on smithi154 2024-03-16T11:47:46.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:47:46 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:47:46.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:47:46 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:47:46.796 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:47:46 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-16T11:47:46.796 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:47:46 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-16T11:47:46.796 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:47:46 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-16T11:47:46.796 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:47:46 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-16T11:47:46.796 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:47:46 smithi154 ceph-mon[36320]: Adjusting osd_memory_target on smithi154 to 4270M 2024-03-16T11:47:46.796 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:47:46 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:47:46.796 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:47:46 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:47:46.796 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:47:46 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T11:47:46.796 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:47:46 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:47:46.796 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:47:46 smithi154 ceph-mon[36320]: osdmap e81: 12 total, 12 up, 12 in 2024-03-16T11:47:46.796 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:47:46 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T11:47:46.796 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:47:46 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:47:46.796 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:47:46 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T11:47:46.796 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:47:46 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:47:46.796 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:47:46 smithi154 ceph-mon[36320]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T11:47:47.911 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:47:47 smithi029 ceph-mon[31975]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T11:47:48.029 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:48.018+0000 7f07c3f34700 1 -- 172.21.15.29:0/2343295887 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f07bc108050 msgr2=0x7f07bc10a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:47:48.029 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:48.018+0000 7f07c3f34700 1 --2- 172.21.15.29:0/2343295887 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f07bc108050 0x7f07bc10a460 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f07b400b0a0 tx=0x7f07b400b3b0 comp rx=0 tx=0).stop 2024-03-16T11:47:48.029 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:48.018+0000 7f07c3f34700 1 -- 172.21.15.29:0/2343295887 shutdown_connections 2024-03-16T11:47:48.029 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:48.018+0000 7f07c3f34700 1 --2- 172.21.15.29:0/2343295887 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f07bc108050 0x7f07bc10a460 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:47:48.029 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:48.018+0000 7f07c3f34700 1 --2- 172.21.15.29:0/2343295887 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f07bc102df0 0x7f07bc1051e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:47:48.029 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:48.018+0000 7f07c3f34700 1 --2- 172.21.15.29:0/2343295887 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f07bc105720 0x7f07bc107b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:47:48.029 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:48.018+0000 7f07c3f34700 1 -- 172.21.15.29:0/2343295887 >> 172.21.15.29:0/2343295887 conn(0x7f07bc0fc7a0 msgr2=0x7f07bc0febe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:47:48.029 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:48.018+0000 7f07c3f34700 1 -- 172.21.15.29:0/2343295887 shutdown_connections 2024-03-16T11:47:48.029 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:48.018+0000 7f07c3f34700 1 -- 172.21.15.29:0/2343295887 wait complete. 2024-03-16T11:47:48.030 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:48.019+0000 7f07c3f34700 1 Processor -- start 2024-03-16T11:47:48.030 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:48.019+0000 7f07c3f34700 1 -- start start 2024-03-16T11:47:48.030 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:48.019+0000 7f07c3f34700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f07bc102df0 0x7f07bc198400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:47:48.030 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:48.019+0000 7f07c3f34700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f07bc105720 0x7f07bc198940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:47:48.030 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:48.019+0000 7f07c3f34700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f07bc108050 0x7f07bc19f9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:47:48.030 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:48.019+0000 7f07c3f34700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f07bc10df10 con 0x7f07bc105720 2024-03-16T11:47:48.030 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:48.019+0000 7f07c3f34700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f07bc10dd90 con 0x7f07bc108050 2024-03-16T11:47:48.030 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:48.019+0000 7f07c3f34700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f07bc10e090 con 0x7f07bc102df0 2024-03-16T11:47:48.030 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:48.019+0000 7f07c14cf700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f07bc105720 0x7f07bc198940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:47:48.030 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:48.019+0000 7f07c14cf700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f07bc105720 0x7f07bc198940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36698/0 (socket says 172.21.15.29:36698) 2024-03-16T11:47:48.030 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:48.019+0000 7f07c14cf700 1 -- 172.21.15.29:0/3407647713 learned_addr learned my addr 172.21.15.29:0/3407647713 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:47:48.030 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:48.019+0000 7f07c1cd0700 1 --2- 172.21.15.29:0/3407647713 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f07bc102df0 0x7f07bc198400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:47:48.030 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:48.020+0000 7f07c24d1700 1 --2- 172.21.15.29:0/3407647713 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f07bc108050 0x7f07bc19f9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:47:48.030 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:48.020+0000 7f07c14cf700 1 -- 172.21.15.29:0/3407647713 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f07bc102df0 msgr2=0x7f07bc198400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:47:48.031 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:48.020+0000 7f07c14cf700 1 --2- 172.21.15.29:0/3407647713 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f07bc102df0 0x7f07bc198400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:47:48.031 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:48.020+0000 7f07c14cf700 1 -- 172.21.15.29:0/3407647713 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f07bc108050 msgr2=0x7f07bc19f9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:47:48.031 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:48.020+0000 7f07c14cf700 1 --2- 172.21.15.29:0/3407647713 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f07bc108050 0x7f07bc19f9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:47:48.031 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:48.020+0000 7f07c14cf700 1 -- 172.21.15.29:0/3407647713 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07bc1a0060 con 0x7f07bc105720 2024-03-16T11:47:48.031 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:48.020+0000 7f07c14cf700 1 --2- 172.21.15.29:0/3407647713 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f07bc105720 0x7f07bc198940 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f07b800c9e0 tx=0x7f07b800cda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:47:48.031 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:48.020+0000 7f07b2ffd700 1 -- 172.21.15.29:0/3407647713 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07b8007ac0 con 0x7f07bc105720 2024-03-16T11:47:48.031 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:48.020+0000 7f07b2ffd700 1 -- 172.21.15.29:0/3407647713 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f07b8007c60 con 0x7f07bc105720 2024-03-16T11:47:48.031 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:48.020+0000 7f07c3f34700 1 -- 172.21.15.29:0/3407647713 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f07bc1a0350 con 0x7f07bc105720 2024-03-16T11:47:48.031 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:48.020+0000 7f07b2ffd700 1 -- 172.21.15.29:0/3407647713 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07b8018710 con 0x7f07bc105720 2024-03-16T11:47:48.031 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:48.020+0000 7f07c3f34700 1 -- 172.21.15.29:0/3407647713 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f07bc1a0890 con 0x7f07bc105720 2024-03-16T11:47:48.031 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:48.021+0000 7f07c3f34700 1 -- 172.21.15.29:0/3407647713 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f07bc068f90 con 0x7f07bc105720 2024-03-16T11:47:48.031 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:48.022+0000 7f07b2ffd700 1 -- 172.21.15.29:0/3407647713 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f07b80188d0 con 0x7f07bc105720 2024-03-16T11:47:48.031 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:48.023+0000 7f07b2ffd700 1 --2- 172.21.15.29:0/3407647713 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f07a80a2890 0x7f07a80a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:47:48.031 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:48.023+0000 7f07c1cd0700 1 --2- 172.21.15.29:0/3407647713 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f07a80a2890 0x7f07a80a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:47:48.031 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:48.023+0000 7f07b2ffd700 1 -- 172.21.15.29:0/3407647713 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f07b80ce7f0 con 0x7f07bc105720 2024-03-16T11:47:48.032 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:48.023+0000 7f07c1cd0700 1 --2- 172.21.15.29:0/3407647713 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f07a80a2890 0x7f07a80a4d50 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f07ac005fd0 tx=0x7f07ac00a560 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:47:48.032 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:48.027+0000 7f07b2ffd700 1 -- 172.21.15.29:0/3407647713 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f07b8014030 con 0x7f07bc105720 2024-03-16T11:47:48.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:47:47 smithi154 ceph-mon[36320]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T11:47:48.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:47:47 smithi152 ceph-mon[36291]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T11:47:48.230 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:48.231+0000 7f07c3f34700 1 -- 172.21.15.29:0/3407647713 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) -- 0x7f07bc02dba0 con 0x7f07bc105720 2024-03-16T11:47:48.230 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:48.231+0000 7f07b2ffd700 1 -- 172.21.15.29:0/3407647713 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v81) ==== 74+0+133 (secure 0 0 0) 0x7f07b809bd30 con 0x7f07bc105720 2024-03-16T11:47:48.231 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-16T11:47:48.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:48.234+0000 7f07c3f34700 1 -- 172.21.15.29:0/3407647713 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f07a80a2890 msgr2=0x7f07a80a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:47:48.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:48.234+0000 7f07c3f34700 1 --2- 172.21.15.29:0/3407647713 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f07a80a2890 0x7f07a80a4d50 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f07ac005fd0 tx=0x7f07ac00a560 comp rx=0 tx=0).stop 2024-03-16T11:47:48.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:48.234+0000 7f07c3f34700 1 -- 172.21.15.29:0/3407647713 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f07bc105720 msgr2=0x7f07bc198940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:47:48.234 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:48.234+0000 7f07c3f34700 1 --2- 172.21.15.29:0/3407647713 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f07bc105720 0x7f07bc198940 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f07b800c9e0 tx=0x7f07b800cda0 comp rx=0 tx=0).stop 2024-03-16T11:47:48.234 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:48.234+0000 7f07c3f34700 1 -- 172.21.15.29:0/3407647713 shutdown_connections 2024-03-16T11:47:48.234 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:48.234+0000 7f07c3f34700 1 --2- 172.21.15.29:0/3407647713 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f07bc108050 0x7f07bc19f9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:47:48.234 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:48.234+0000 7f07c3f34700 1 --2- 172.21.15.29:0/3407647713 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f07a80a2890 0x7f07a80a4d50 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:47:48.234 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:48.234+0000 7f07c3f34700 1 --2- 172.21.15.29:0/3407647713 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f07bc102df0 0x7f07bc198400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:47:48.234 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:48.234+0000 7f07c3f34700 1 --2- 172.21.15.29:0/3407647713 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f07bc105720 0x7f07bc198940 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:47:48.234 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:48.234+0000 7f07c3f34700 1 -- 172.21.15.29:0/3407647713 >> 172.21.15.29:0/3407647713 conn(0x7f07bc0fc7a0 msgr2=0x7f07bc1036f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:47:48.234 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:48.234+0000 7f07c3f34700 1 -- 172.21.15.29:0/3407647713 shutdown_connections 2024-03-16T11:47:48.234 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:48.234+0000 7f07c3f34700 1 -- 172.21.15.29:0/3407647713 wait complete. 2024-03-16T11:47:49.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:47:48 smithi154 ceph-mon[36320]: from='client.? 172.21.15.29:0/3407647713' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-16T11:47:49.084 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:47:48 smithi029 ceph-mon[31975]: from='client.? 172.21.15.29:0/3407647713' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-16T11:47:49.117 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":81,"num_osds":12,"num_up_osds":12,"osd_up_since":1710589664,"num_in_osds":12,"osd_in_since":1710589643,"num_remapped_pgs":0} 2024-03-16T11:47:49.118 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid d1887f28-e389-11ee-95c9-87774f69a715 -- ceph osd dump --format=json 2024-03-16T11:47:49.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:47:48 smithi152 ceph-mon[36291]: from='client.? 172.21.15.29:0/3407647713' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-16T11:47:49.365 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/d1887f28-e389-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T11:47:49.774 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:47:49 smithi029 ceph-mon[31975]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T11:47:50.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:47:49 smithi154 ceph-mon[36320]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T11:47:50.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:47:49 smithi152 ceph-mon[36291]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T11:47:51.229 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:51.224+0000 7f49b6ee2700 1 -- 172.21.15.29:0/4294453102 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f49b01064f0 msgr2=0x7f49b010e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:47:51.229 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:51.224+0000 7f49b6ee2700 1 --2- 172.21.15.29:0/4294453102 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f49b01064f0 0x7f49b010e0c0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f49a800b0a0 tx=0x7f49a800b3b0 comp rx=0 tx=0).stop 2024-03-16T11:47:51.229 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:51.225+0000 7f49b6ee2700 1 -- 172.21.15.29:0/4294453102 shutdown_connections 2024-03-16T11:47:51.229 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:51.225+0000 7f49b6ee2700 1 --2- 172.21.15.29:0/4294453102 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f49b01064f0 0x7f49b010e0c0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:47:51.229 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:51.225+0000 7f49b6ee2700 1 --2- 172.21.15.29:0/4294453102 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f49b0101460 0x7f49b0101840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:47:51.229 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:51.225+0000 7f49b6ee2700 1 --2- 172.21.15.29:0/4294453102 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f49b0101e10 0x7f49b0105e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:47:51.229 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:51.225+0000 7f49b6ee2700 1 -- 172.21.15.29:0/4294453102 >> 172.21.15.29:0/4294453102 conn(0x7f49b00771f0 msgr2=0x7f49b0077600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:47:51.229 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:51.225+0000 7f49b6ee2700 1 -- 172.21.15.29:0/4294453102 shutdown_connections 2024-03-16T11:47:51.229 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:51.225+0000 7f49b6ee2700 1 -- 172.21.15.29:0/4294453102 wait complete. 2024-03-16T11:47:51.229 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:51.225+0000 7f49b6ee2700 1 Processor -- start 2024-03-16T11:47:51.229 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:51.225+0000 7f49b6ee2700 1 -- start start 2024-03-16T11:47:51.230 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:51.226+0000 7f49b6ee2700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f49b0101460 0x7f49b0199890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:47:51.230 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:51.226+0000 7f49b6ee2700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f49b0101e10 0x7f49b0199dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:47:51.230 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:51.226+0000 7f49b6ee2700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f49b01064f0 0x7f49b01a4910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:47:51.230 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:51.226+0000 7f49b6ee2700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f49b0111530 con 0x7f49b01064f0 2024-03-16T11:47:51.230 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:51.226+0000 7f49b6ee2700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f49b01113b0 con 0x7f49b0101460 2024-03-16T11:47:51.230 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:51.226+0000 7f49b6ee2700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f49b01116b0 con 0x7f49b0101e10 2024-03-16T11:47:51.230 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:51.226+0000 7f49b547f700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f49b01064f0 0x7f49b01a4910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:47:51.230 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:51.226+0000 7f49b547f700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f49b01064f0 0x7f49b01a4910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57458/0 (socket says 172.21.15.29:57458) 2024-03-16T11:47:51.230 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:51.226+0000 7f49b547f700 1 -- 172.21.15.29:0/1119599039 learned_addr learned my addr 172.21.15.29:0/1119599039 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:47:51.230 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:51.226+0000 7f49affff700 1 --2- 172.21.15.29:0/1119599039 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f49b0101e10 0x7f49b0199dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:47:51.230 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:51.226+0000 7f49b4c7e700 1 --2- 172.21.15.29:0/1119599039 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f49b0101460 0x7f49b0199890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:47:51.230 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:51.226+0000 7f49b547f700 1 -- 172.21.15.29:0/1119599039 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f49b0101e10 msgr2=0x7f49b0199dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:47:51.230 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:51.226+0000 7f49b547f700 1 --2- 172.21.15.29:0/1119599039 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f49b0101e10 0x7f49b0199dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:47:51.230 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:51.226+0000 7f49b547f700 1 -- 172.21.15.29:0/1119599039 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f49b0101460 msgr2=0x7f49b0199890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:47:51.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:51.226+0000 7f49b547f700 1 --2- 172.21.15.29:0/1119599039 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f49b0101460 0x7f49b0199890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:47:51.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:51.226+0000 7f49b547f700 1 -- 172.21.15.29:0/1119599039 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49b01a4e50 con 0x7f49b01064f0 2024-03-16T11:47:51.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:51.226+0000 7f49b547f700 1 --2- 172.21.15.29:0/1119599039 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f49b01064f0 0x7f49b01a4910 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f49a80062a0 tx=0x7f49a800bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:47:51.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:51.226+0000 7f49adffb700 1 -- 172.21.15.29:0/1119599039 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49a8021070 con 0x7f49b01064f0 2024-03-16T11:47:51.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:51.226+0000 7f49b6ee2700 1 -- 172.21.15.29:0/1119599039 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f49b01a5020 con 0x7f49b01064f0 2024-03-16T11:47:51.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:51.226+0000 7f49adffb700 1 -- 172.21.15.29:0/1119599039 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f49a8007db0 con 0x7f49b01064f0 2024-03-16T11:47:51.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:51.226+0000 7f49adffb700 1 -- 172.21.15.29:0/1119599039 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49a8005460 con 0x7f49b01064f0 2024-03-16T11:47:51.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:51.227+0000 7f49b6ee2700 1 -- 172.21.15.29:0/1119599039 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f49b01a5520 con 0x7f49b01064f0 2024-03-16T11:47:51.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:51.227+0000 7f49b6ee2700 1 -- 172.21.15.29:0/1119599039 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f49b01028f0 con 0x7f49b01064f0 2024-03-16T11:47:51.232 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:51.229+0000 7f49adffb700 1 -- 172.21.15.29:0/1119599039 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f49a8024070 con 0x7f49b01064f0 2024-03-16T11:47:51.232 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:51.230+0000 7f49adffb700 1 --2- 172.21.15.29:0/1119599039 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f49a00a2890 0x7f49a00a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:47:51.232 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:51.230+0000 7f49b4c7e700 1 --2- 172.21.15.29:0/1119599039 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f49a00a2890 0x7f49a00a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:47:51.232 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:51.230+0000 7f49adffb700 1 -- 172.21.15.29:0/1119599039 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f49a800f020 con 0x7f49b01064f0 2024-03-16T11:47:51.232 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:51.230+0000 7f49b4c7e700 1 --2- 172.21.15.29:0/1119599039 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f49a00a2890 0x7f49a00a4d50 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f49a4005ea0 tx=0x7f49a4005e30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:47:51.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:51.234+0000 7f49adffb700 1 -- 172.21.15.29:0/1119599039 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f49a802b020 con 0x7f49b01064f0 2024-03-16T11:47:51.437 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:51.437+0000 7f49b6ee2700 1 -- 172.21.15.29:0/1119599039 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f49b002dc90 con 0x7f49b01064f0 2024-03-16T11:47:51.438 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:51.438+0000 7f49adffb700 1 -- 172.21.15.29:0/1119599039 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) ==== 74+0+18470 (secure 0 0 0) 0x7f49a809d6b0 con 0x7f49b01064f0 2024-03-16T11:47:51.439 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-16T11:47:51.439 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":81,"fsid":"d1887f28-e389-11ee-95c9-87774f69a715","created":"2024-03-16T11:40:16.475063+0000","modified":"2024-03-16T11:47:45.354317+0000","last_up_change":"2024-03-16T11:47:44.349348+0000","last_in_change":"2024-03-16T11:47:23.672432+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-16T11:43:51.892652+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"60f5608f-a408-45ba-bdf6-5b53758f88fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":2748947913},{"type":"v1","addr":"172.21.15.29:6803","nonce":2748947913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":2748947913},{"type":"v1","addr":"172.21.15.29:6805","nonce":2748947913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":2748947913},{"type":"v1","addr":"172.21.15.29:6809","nonce":2748947913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":2748947913},{"type":"v1","addr":"172.21.15.29:6807","nonce":2748947913}]},"public_addr":"172.21.15.29:6803/2748947913","cluster_addr":"172.21.15.29:6805/2748947913","heartbeat_back_addr":"172.21.15.29:6809/2748947913","heartbeat_front_addr":"172.21.15.29:6807/2748947913","state":["exists","up"]},{"osd":1,"uuid":"e70b53d6-ee78-4e56-9a2c-263ae0501b43","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6800","nonce":2339408422},{"type":"v1","addr":"172.21.15.152:6801","nonce":2339408422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6802","nonce":2339408422},{"type":"v1","addr":"172.21.15.152:6803","nonce":2339408422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6806","nonce":2339408422},{"type":"v1","addr":"172.21.15.152:6807","nonce":2339408422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6804","nonce":2339408422},{"type":"v1","addr":"172.21.15.152:6805","nonce":2339408422}]},"public_addr":"172.21.15.152:6801/2339408422","cluster_addr":"172.21.15.152:6803/2339408422","heartbeat_back_addr":"172.21.15.152:6807/2339408422","heartbeat_front_addr":"172.21.15.152:6805/2339408422","state":["exists","up"]},{"osd":2,"uuid":"ff39389a-dfac-44b5-80ff-36abca316e78","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6800","nonce":4288325280},{"type":"v1","addr":"172.21.15.154:6801","nonce":4288325280}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6802","nonce":4288325280},{"type":"v1","addr":"172.21.15.154:6803","nonce":4288325280}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6806","nonce":4288325280},{"type":"v1","addr":"172.21.15.154:6807","nonce":4288325280}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":4288325280},{"type":"v1","addr":"172.21.15.154:6805","nonce":4288325280}]},"public_addr":"172.21.15.154:6801/4288325280","cluster_addr":"172.21.15.154:6803/4288325280","heartbeat_back_addr":"172.21.15.154:6807/4288325280","heartbeat_front_addr":"172.21.15.154:6805/4288325280","state":["exists","up"]},{"osd":3,"uuid":"069e1795-55bc-4324-b94c-eb24ef2334c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":1538750397},{"type":"v1","addr":"172.21.15.29:6811","nonce":1538750397}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":1538750397},{"type":"v1","addr":"172.21.15.29:6813","nonce":1538750397}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":1538750397},{"type":"v1","addr":"172.21.15.29:6817","nonce":1538750397}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":1538750397},{"type":"v1","addr":"172.21.15.29:6815","nonce":1538750397}]},"public_addr":"172.21.15.29:6811/1538750397","cluster_addr":"172.21.15.29:6813/1538750397","heartbeat_back_addr":"172.21.15.29:6817/1538750397","heartbeat_front_addr":"172.21.15.29:6815/1538750397","state":["exists","up"]},{"osd":4,"uuid":"5a9d43a1-e8f5-4ab5-a441-afbc6ed4066e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6808","nonce":2795081750},{"type":"v1","addr":"172.21.15.152:6809","nonce":2795081750}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6810","nonce":2795081750},{"type":"v1","addr":"172.21.15.152:6811","nonce":2795081750}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6814","nonce":2795081750},{"type":"v1","addr":"172.21.15.152:6815","nonce":2795081750}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6812","nonce":2795081750},{"type":"v1","addr":"172.21.15.152:6813","nonce":2795081750}]},"public_addr":"172.21.15.152:6809/2795081750","cluster_addr":"172.21.15.152:6811/2795081750","heartbeat_back_addr":"172.21.15.152:6815/2795081750","heartbeat_front_addr":"172.21.15.152:6813/2795081750","state":["exists","up"]},{"osd":5,"uuid":"234de428-4ac7-44be-9a54-7b58331fdf8c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":1472686560},{"type":"v1","addr":"172.21.15.154:6809","nonce":1472686560}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6810","nonce":1472686560},{"type":"v1","addr":"172.21.15.154:6811","nonce":1472686560}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6814","nonce":1472686560},{"type":"v1","addr":"172.21.15.154:6815","nonce":1472686560}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6812","nonce":1472686560},{"type":"v1","addr":"172.21.15.154:6813","nonce":1472686560}]},"public_addr":"172.21.15.154:6809/1472686560","cluster_addr":"172.21.15.154:6811/1472686560","heartbeat_back_addr":"172.21.15.154:6815/1472686560","heartbeat_front_addr":"172.21.15.154:6813/1472686560","state":["exists","up"]},{"osd":6,"uuid":"ae45b8d9-7883-40c9-a9cd-4c92b601d4f2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":816506244},{"type":"v1","addr":"172.21.15.29:6819","nonce":816506244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":816506244},{"type":"v1","addr":"172.21.15.29:6821","nonce":816506244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":816506244},{"type":"v1","addr":"172.21.15.29:6825","nonce":816506244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":816506244},{"type":"v1","addr":"172.21.15.29:6823","nonce":816506244}]},"public_addr":"172.21.15.29:6819/816506244","cluster_addr":"172.21.15.29:6821/816506244","heartbeat_back_addr":"172.21.15.29:6825/816506244","heartbeat_front_addr":"172.21.15.29:6823/816506244","state":["exists","up"]},{"osd":7,"uuid":"6f687b28-4fce-4d4a-8b4e-294173d9d6c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6816","nonce":2924651787},{"type":"v1","addr":"172.21.15.152:6817","nonce":2924651787}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6818","nonce":2924651787},{"type":"v1","addr":"172.21.15.152:6819","nonce":2924651787}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6822","nonce":2924651787},{"type":"v1","addr":"172.21.15.152:6823","nonce":2924651787}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6820","nonce":2924651787},{"type":"v1","addr":"172.21.15.152:6821","nonce":2924651787}]},"public_addr":"172.21.15.152:6817/2924651787","cluster_addr":"172.21.15.152:6819/2924651787","heartbeat_back_addr":"172.21.15.152:6823/2924651787","heartbeat_front_addr":"172.21.15.152:6821/2924651787","state":["exists","up"]},{"osd":8,"uuid":"a25fb741-caa7-429a-b540-e394d034b446","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6816","nonce":129974156},{"type":"v1","addr":"172.21.15.154:6817","nonce":129974156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6818","nonce":129974156},{"type":"v1","addr":"172.21.15.154:6819","nonce":129974156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6822","nonce":129974156},{"type":"v1","addr":"172.21.15.154:6823","nonce":129974156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6820","nonce":129974156},{"type":"v1","addr":"172.21.15.154:6821","nonce":129974156}]},"public_addr":"172.21.15.154:6817/129974156","cluster_addr":"172.21.15.154:6819/129974156","heartbeat_back_addr":"172.21.15.154:6823/129974156","heartbeat_front_addr":"172.21.15.154:6821/129974156","state":["exists","up"]},{"osd":9,"uuid":"1071cb0e-8c5f-4cd8-9805-0045e220d946","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6826","nonce":2951414243},{"type":"v1","addr":"172.21.15.29:6827","nonce":2951414243}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":2951414243},{"type":"v1","addr":"172.21.15.29:6829","nonce":2951414243}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":2951414243},{"type":"v1","addr":"172.21.15.29:6833","nonce":2951414243}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":2951414243},{"type":"v1","addr":"172.21.15.29:6831","nonce":2951414243}]},"public_addr":"172.21.15.29:6827/2951414243","cluster_addr":"172.21.15.29:6829/2951414243","heartbeat_back_addr":"172.21.15.29:6833/2951414243","heartbeat_front_addr":"172.21.15.29:6831/2951414243","state":["exists","up"]},{"osd":10,"uuid":"ee7c7d74-3f39-4cd4-be80-843fb1dd9ecc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6824","nonce":3412525480},{"type":"v1","addr":"172.21.15.152:6825","nonce":3412525480}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6826","nonce":3412525480},{"type":"v1","addr":"172.21.15.152:6827","nonce":3412525480}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6830","nonce":3412525480},{"type":"v1","addr":"172.21.15.152:6831","nonce":3412525480}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6828","nonce":3412525480},{"type":"v1","addr":"172.21.15.152:6829","nonce":3412525480}]},"public_addr":"172.21.15.152:6825/3412525480","cluster_addr":"172.21.15.152:6827/3412525480","heartbeat_back_addr":"172.21.15.152:6831/3412525480","heartbeat_front_addr":"172.21.15.152:6829/3412525480","state":["exists","up"]},{"osd":11,"uuid":"4598c803-984a-4de6-ac22-066502c17889","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6824","nonce":1160927966},{"type":"v1","addr":"172.21.15.154:6825","nonce":1160927966}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6826","nonce":1160927966},{"type":"v1","addr":"172.21.15.154:6827","nonce":1160927966}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6830","nonce":1160927966},{"type":"v1","addr":"172.21.15.154:6831","nonce":1160927966}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6828","nonce":1160927966},{"type":"v1","addr":"172.21.15.154:6829","nonce":1160927966}]},"public_addr":"172.21.15.154:6825/1160927966","cluster_addr":"172.21.15.154:6827/1160927966","heartbeat_back_addr":"172.21.15.154:6831/1160927966","heartbeat_front_addr":"172.21.15.154:6829/1160927966","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:43:02.343393+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:43:25.209151+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:43:48.946164+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:44:14.588424+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:44:39.700025+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:45:04.061197+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:45:31.214508+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:45:56.937296+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:46:22.768915+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:46:50.101011+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:47:16.298041+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:47:42.402095+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.29:0/1853086886":"2024-03-17T11:41:27.531724+0000","172.21.15.29:0/1211659364":"2024-03-17T11:41:27.531724+0000","172.21.15.29:6800/2798831128":"2024-03-17T11:41:27.531724+0000","172.21.15.29:0/1281028285":"2024-03-17T11:41:00.505678+0000","172.21.15.29:6800/2275315654":"2024-03-17T11:41:00.505678+0000","172.21.15.29:6801/2798831128":"2024-03-17T11:41:27.531724+0000","172.21.15.29:6801/2275315654":"2024-03-17T11:41:00.505678+0000","172.21.15.29:0/1330100009":"2024-03-17T11:41:00.505678+0000","172.21.15.29:0/1784249039":"2024-03-17T11:41:00.505678+0000","172.21.15.29:0/1116486932":"2024-03-17T11:41:27.531724+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-16T11:47:51.441 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:51.440+0000 7f49b6ee2700 1 -- 172.21.15.29:0/1119599039 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f49a00a2890 msgr2=0x7f49a00a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:47:51.441 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:51.440+0000 7f49b6ee2700 1 --2- 172.21.15.29:0/1119599039 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f49a00a2890 0x7f49a00a4d50 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f49a4005ea0 tx=0x7f49a4005e30 comp rx=0 tx=0).stop 2024-03-16T11:47:51.441 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:51.440+0000 7f49b6ee2700 1 -- 172.21.15.29:0/1119599039 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f49b01064f0 msgr2=0x7f49b01a4910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:47:51.441 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:51.440+0000 7f49b6ee2700 1 --2- 172.21.15.29:0/1119599039 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f49b01064f0 0x7f49b01a4910 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f49a80062a0 tx=0x7f49a800bf10 comp rx=0 tx=0).stop 2024-03-16T11:47:51.441 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:51.440+0000 7f49b6ee2700 1 -- 172.21.15.29:0/1119599039 shutdown_connections 2024-03-16T11:47:51.441 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:51.441+0000 7f49b6ee2700 1 --2- 172.21.15.29:0/1119599039 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f49b01064f0 0x7f49b01a4910 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:47:51.441 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:51.441+0000 7f49b6ee2700 1 --2- 172.21.15.29:0/1119599039 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f49b0101460 0x7f49b0199890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:47:51.441 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:51.441+0000 7f49b6ee2700 1 --2- 172.21.15.29:0/1119599039 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f49a00a2890 0x7f49a00a4d50 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:47:51.442 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:51.441+0000 7f49b6ee2700 1 --2- 172.21.15.29:0/1119599039 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f49b0101e10 0x7f49b0199dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:47:51.442 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:51.441+0000 7f49b6ee2700 1 -- 172.21.15.29:0/1119599039 >> 172.21.15.29:0/1119599039 conn(0x7f49b00771f0 msgr2=0x7f49b00ff1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:47:51.442 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:51.441+0000 7f49b6ee2700 1 -- 172.21.15.29:0/1119599039 shutdown_connections 2024-03-16T11:47:51.442 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:51.441+0000 7f49b6ee2700 1 -- 172.21.15.29:0/1119599039 wait complete. 2024-03-16T11:47:51.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:47:51 smithi154 ceph-mon[36320]: from='client.? 172.21.15.29:0/1119599039' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T11:47:51.813 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:47:51 smithi029 ceph-mon[31975]: from='client.? 172.21.15.29:0/1119599039' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T11:47:51.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:47:51 smithi152 ceph-mon[36291]: from='client.? 172.21.15.29:0/1119599039' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T11:47:52.273 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-16T11:43:51.892652+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-16T11:47:52.274 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid d1887f28-e389-11ee-95c9-87774f69a715 -- ceph osd pool get .mgr pg_num 2024-03-16T11:47:52.521 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/d1887f28-e389-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T11:47:52.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:47:52 smithi154 ceph-mon[36320]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T11:47:52.797 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:47:52 smithi029 ceph-mon[31975]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T11:47:52.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:47:52 smithi152 ceph-mon[36291]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T11:47:53.970 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:47:53 smithi029 ceph-mon[31975]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T11:47:54.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:47:53 smithi154 ceph-mon[36320]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T11:47:54.086 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:47:53 smithi152 ceph-mon[36291]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T11:47:54.382 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:54.370+0000 7f1ec5672700 1 -- 172.21.15.29:0/880021504 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1ec0102320 msgr2=0x7f1ec01027a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:47:54.382 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:54.370+0000 7f1ec5672700 1 --2- 172.21.15.29:0/880021504 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1ec0102320 0x7f1ec01027a0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f1eb00099c0 tx=0x7f1eb0009cd0 comp rx=0 tx=0).stop 2024-03-16T11:47:54.382 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:54.370+0000 7f1ec5672700 1 -- 172.21.15.29:0/880021504 shutdown_connections 2024-03-16T11:47:54.382 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:54.370+0000 7f1ec5672700 1 --2- 172.21.15.29:0/880021504 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1ec0102ce0 0x7f1ec010f120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:47:54.382 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:54.370+0000 7f1ec5672700 1 --2- 172.21.15.29:0/880021504 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1ec0108300 0x7f1ec01086e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:47:54.382 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:54.370+0000 7f1ec5672700 1 --2- 172.21.15.29:0/880021504 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1ec0102320 0x7f1ec01027a0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:47:54.382 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:54.371+0000 7f1ec5672700 1 -- 172.21.15.29:0/880021504 >> 172.21.15.29:0/880021504 conn(0x7f1ec0079400 msgr2=0x7f1ec0079810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:47:54.382 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:54.371+0000 7f1ec5672700 1 -- 172.21.15.29:0/880021504 shutdown_connections 2024-03-16T11:47:54.382 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:54.371+0000 7f1ec5672700 1 -- 172.21.15.29:0/880021504 wait complete. 2024-03-16T11:47:54.382 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:54.371+0000 7f1ec5672700 1 Processor -- start 2024-03-16T11:47:54.382 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:54.371+0000 7f1ec5672700 1 -- start start 2024-03-16T11:47:54.382 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:54.372+0000 7f1ec5672700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1ec0102320 0x7f1ec00745d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:47:54.382 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:54.372+0000 7f1ec5672700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1ec0102ce0 0x7f1ec0074b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:47:54.383 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:54.372+0000 7f1ec5672700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1ec0108300 0x7f1ec0075050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:47:54.383 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:54.372+0000 7f1ec5672700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f1ec0112580 con 0x7f1ec0102ce0 2024-03-16T11:47:54.383 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:54.372+0000 7f1ec5672700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f1ec0112400 con 0x7f1ec0102320 2024-03-16T11:47:54.383 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:54.372+0000 7f1ec5672700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f1ec0112700 con 0x7f1ec0108300 2024-03-16T11:47:54.383 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:54.372+0000 7f1ebe7fc700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1ec0102ce0 0x7f1ec0074b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:47:54.383 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:54.372+0000 7f1ebe7fc700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1ec0102ce0 0x7f1ec0074b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57474/0 (socket says 172.21.15.29:57474) 2024-03-16T11:47:54.383 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:54.372+0000 7f1ebe7fc700 1 -- 172.21.15.29:0/3247906892 learned_addr learned my addr 172.21.15.29:0/3247906892 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:47:54.383 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:54.372+0000 7f1ebeffd700 1 --2- 172.21.15.29:0/3247906892 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1ec0102320 0x7f1ec00745d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:47:54.383 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:54.372+0000 7f1ebf7fe700 1 --2- 172.21.15.29:0/3247906892 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1ec0108300 0x7f1ec0075050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:47:54.383 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:54.372+0000 7f1ebe7fc700 1 -- 172.21.15.29:0/3247906892 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1ec0108300 msgr2=0x7f1ec0075050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:47:54.383 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:54.372+0000 7f1ebe7fc700 1 --2- 172.21.15.29:0/3247906892 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1ec0108300 0x7f1ec0075050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:47:54.383 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:54.372+0000 7f1ebe7fc700 1 -- 172.21.15.29:0/3247906892 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1ec0102320 msgr2=0x7f1ec00745d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:47:54.383 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:54.372+0000 7f1ebe7fc700 1 --2- 172.21.15.29:0/3247906892 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1ec0102320 0x7f1ec00745d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:47:54.383 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:54.372+0000 7f1ebe7fc700 1 -- 172.21.15.29:0/3247906892 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ec01a6f40 con 0x7f1ec0102ce0 2024-03-16T11:47:54.383 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:54.372+0000 7f1ebeffd700 1 --2- 172.21.15.29:0/3247906892 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1ec0102320 0x7f1ec00745d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-16T11:47:54.383 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:54.372+0000 7f1ebf7fe700 1 --2- 172.21.15.29:0/3247906892 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1ec0108300 0x7f1ec0075050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-16T11:47:54.384 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:54.372+0000 7f1ebe7fc700 1 --2- 172.21.15.29:0/3247906892 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1ec0102ce0 0x7f1ec0074b10 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f1eb0006010 tx=0x7f1eb0005640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:47:54.384 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:54.372+0000 7f1ea7fff700 1 -- 172.21.15.29:0/3247906892 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1eb0027070 con 0x7f1ec0102ce0 2024-03-16T11:47:54.384 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:54.372+0000 7f1ec5672700 1 -- 172.21.15.29:0/3247906892 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1ec01a71d0 con 0x7f1ec0102ce0 2024-03-16T11:47:54.384 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:54.372+0000 7f1ea7fff700 1 -- 172.21.15.29:0/3247906892 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1eb00040d0 con 0x7f1ec0102ce0 2024-03-16T11:47:54.384 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:54.372+0000 7f1ea7fff700 1 -- 172.21.15.29:0/3247906892 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1eb000f460 con 0x7f1ec0102ce0 2024-03-16T11:47:54.384 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:54.372+0000 7f1ec5672700 1 -- 172.21.15.29:0/3247906892 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1ec01a7620 con 0x7f1ec0102ce0 2024-03-16T11:47:54.384 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:54.373+0000 7f1ec5672700 1 -- 172.21.15.29:0/3247906892 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1ec0068f90 con 0x7f1ec0102ce0 2024-03-16T11:47:54.384 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:54.374+0000 7f1ea7fff700 1 -- 172.21.15.29:0/3247906892 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f1eb002a070 con 0x7f1ec0102ce0 2024-03-16T11:47:54.384 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:54.375+0000 7f1ea7fff700 1 --2- 172.21.15.29:0/3247906892 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f1ea80a2890 0x7f1ea80a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:47:54.384 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:54.375+0000 7f1ebeffd700 1 --2- 172.21.15.29:0/3247906892 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f1ea80a2890 0x7f1ea80a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:47:54.384 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:54.375+0000 7f1ea7fff700 1 -- 172.21.15.29:0/3247906892 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f1eb00cf430 con 0x7f1ec0102ce0 2024-03-16T11:47:54.384 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:54.375+0000 7f1ebeffd700 1 --2- 172.21.15.29:0/3247906892 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f1ea80a2890 0x7f1ea80a4d50 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f1eb400baa0 tx=0x7f1eb4005db0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:47:54.384 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:54.380+0000 7f1ea7fff700 1 -- 172.21.15.29:0/3247906892 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f1eb005e740 con 0x7f1ec0102ce0 2024-03-16T11:47:54.583 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:54.583+0000 7f1ec5672700 1 -- 172.21.15.29:0/3247906892 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f1ec002dd30 con 0x7f1ec0102ce0 2024-03-16T11:47:54.583 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:54.583+0000 7f1ea7fff700 1 -- 172.21.15.29:0/3247906892 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v81) ==== 93+0+10 (secure 0 0 0) 0x7f1eb0013070 con 0x7f1ec0102ce0 2024-03-16T11:47:54.583 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2024-03-16T11:47:54.586 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:54.586+0000 7f1ec5672700 1 -- 172.21.15.29:0/3247906892 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f1ea80a2890 msgr2=0x7f1ea80a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:47:54.587 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:54.586+0000 7f1ec5672700 1 --2- 172.21.15.29:0/3247906892 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f1ea80a2890 0x7f1ea80a4d50 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f1eb400baa0 tx=0x7f1eb4005db0 comp rx=0 tx=0).stop 2024-03-16T11:47:54.587 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:54.586+0000 7f1ec5672700 1 -- 172.21.15.29:0/3247906892 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1ec0102ce0 msgr2=0x7f1ec0074b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:47:54.587 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:54.586+0000 7f1ec5672700 1 --2- 172.21.15.29:0/3247906892 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1ec0102ce0 0x7f1ec0074b10 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f1eb0006010 tx=0x7f1eb0005640 comp rx=0 tx=0).stop 2024-03-16T11:47:54.587 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:54.586+0000 7f1ec5672700 1 -- 172.21.15.29:0/3247906892 shutdown_connections 2024-03-16T11:47:54.587 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:54.586+0000 7f1ec5672700 1 --2- 172.21.15.29:0/3247906892 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f1ea80a2890 0x7f1ea80a4d50 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:47:54.587 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:54.586+0000 7f1ec5672700 1 --2- 172.21.15.29:0/3247906892 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1ec0108300 0x7f1ec0075050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:47:54.587 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:54.586+0000 7f1ec5672700 1 --2- 172.21.15.29:0/3247906892 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1ec0102320 0x7f1ec00745d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:47:54.587 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:54.586+0000 7f1ec5672700 1 --2- 172.21.15.29:0/3247906892 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1ec0102ce0 0x7f1ec0074b10 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:47:54.587 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:54.586+0000 7f1ec5672700 1 -- 172.21.15.29:0/3247906892 >> 172.21.15.29:0/3247906892 conn(0x7f1ec0079400 msgr2=0x7f1ec0100f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:47:54.587 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:54.586+0000 7f1ec5672700 1 -- 172.21.15.29:0/3247906892 shutdown_connections 2024-03-16T11:47:54.587 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:54.586+0000 7f1ec5672700 1 -- 172.21.15.29:0/3247906892 wait complete. 2024-03-16T11:47:54.837 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:47:54 smithi029 ceph-mon[31975]: from='client.? 172.21.15.29:0/3247906892' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-16T11:47:55.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:47:54 smithi154 ceph-mon[36320]: from='client.? 172.21.15.29:0/3247906892' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-16T11:47:55.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:47:54 smithi152 ceph-mon[36291]: from='client.? 172.21.15.29:0/3247906892' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-16T11:47:55.450 INFO:tasks.cephadm:Adding mds.a on smithi029 2024-03-16T11:47:55.450 INFO:tasks.cephadm:Adding mds.d on smithi029 2024-03-16T11:47:55.450 INFO:tasks.cephadm:Adding mds.g on smithi029 2024-03-16T11:47:55.450 INFO:tasks.cephadm:Adding mds.j on smithi029 2024-03-16T11:47:55.451 INFO:tasks.cephadm:Adding mds.b on smithi152 2024-03-16T11:47:55.451 INFO:tasks.cephadm:Adding mds.e on smithi152 2024-03-16T11:47:55.451 INFO:tasks.cephadm:Adding mds.h on smithi152 2024-03-16T11:47:55.451 INFO:tasks.cephadm:Adding mds.k on smithi152 2024-03-16T11:47:55.451 INFO:tasks.cephadm:Adding mds.c on smithi154 2024-03-16T11:47:55.451 INFO:tasks.cephadm:Adding mds.f on smithi154 2024-03-16T11:47:55.451 INFO:tasks.cephadm:Adding mds.i on smithi154 2024-03-16T11:47:55.451 INFO:tasks.cephadm:Adding mds.l on smithi154 2024-03-16T11:47:55.451 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d1887f28-e389-11ee-95c9-87774f69a715 -- ceph orch apply mds all '12;smithi029=a;smithi029=d;smithi029=g;smithi029=j;smithi152=b;smithi152=e;smithi152=h;smithi152=k;smithi154=c;smithi154=f;smithi154=i;smithi154=l' 2024-03-16T11:47:55.703 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/d1887f28-e389-11ee-95c9-87774f69a715/mon.c/config 2024-03-16T11:47:55.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:47:55 smithi154 ceph-mon[36320]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T11:47:56.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:47:55 smithi029 ceph-mon[31975]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T11:47:56.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:47:55 smithi152 ceph-mon[36291]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T11:47:57.309 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:47:57.309+0000 7f96c7a26700 1 -- 172.21.15.154:0/2200850188 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f96c0103ea0 msgr2=0x7f96c010a750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:47:57.309 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:47:57.309+0000 7f96c7a26700 1 --2- 172.21.15.154:0/2200850188 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f96c0103ea0 0x7f96c010a750 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f96b800b0a0 tx=0x7f96b800b3b0 comp rx=0 tx=0).stop 2024-03-16T11:47:57.309 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:47:57.309+0000 7f96c7a26700 1 -- 172.21.15.154:0/2200850188 shutdown_connections 2024-03-16T11:47:57.309 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:47:57.309+0000 7f96c7a26700 1 --2- 172.21.15.154:0/2200850188 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f96c0103ea0 0x7f96c010a750 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:47:57.309 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:47:57.309+0000 7f96c7a26700 1 --2- 172.21.15.154:0/2200850188 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f96c0102300 0x7f96c0102720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:47:57.309 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:47:57.309+0000 7f96c7a26700 1 --2- 172.21.15.154:0/2200850188 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f96c0103500 0x7f96c0103960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:47:57.309 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:47:57.309+0000 7f96c7a26700 1 -- 172.21.15.154:0/2200850188 >> 172.21.15.154:0/2200850188 conn(0x7f96c00fd8e0 msgr2=0x7f96c00ffd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:47:57.309 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:47:57.310+0000 7f96c7a26700 1 -- 172.21.15.154:0/2200850188 shutdown_connections 2024-03-16T11:47:57.310 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:47:57.310+0000 7f96c7a26700 1 -- 172.21.15.154:0/2200850188 wait complete. 2024-03-16T11:47:57.311 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:47:57.310+0000 7f96c7a26700 1 Processor -- start 2024-03-16T11:47:57.311 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:47:57.310+0000 7f96c7a26700 1 -- start start 2024-03-16T11:47:57.311 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:47:57.310+0000 7f96c7a26700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f96c0102300 0x7f96c0198400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:47:57.312 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:47:57.310+0000 7f96c7a26700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f96c0103500 0x7f96c0198940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:47:57.312 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:47:57.310+0000 7f96c7a26700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f96c0103ea0 0x7f96c019f9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:47:57.312 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:47:57.310+0000 7f96c7a26700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f96c010dee0 con 0x7f96c0102300 2024-03-16T11:47:57.312 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:47:57.310+0000 7f96c7a26700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f96c010dd60 con 0x7f96c0103500 2024-03-16T11:47:57.312 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:47:57.310+0000 7f96c7a26700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f96c010e060 con 0x7f96c0103ea0 2024-03-16T11:47:57.312 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:47:57.310+0000 7f96c5fc3700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f96c0103ea0 0x7f96c019f9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:47:57.312 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:47:57.310+0000 7f96c5fc3700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f96c0103ea0 0x7f96c019f9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.154:39430/0 (socket says 172.21.15.154:39430) 2024-03-16T11:47:57.312 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:47:57.310+0000 7f96c5fc3700 1 -- 172.21.15.154:0/156035343 learned_addr learned my addr 172.21.15.154:0/156035343 (peer_addr_for_me v2:172.21.15.154:0/0) 2024-03-16T11:47:57.312 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:47:57.311+0000 7f96c57c2700 1 --2- 172.21.15.154:0/156035343 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f96c0102300 0x7f96c0198400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:47:57.312 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:47:57.311+0000 7f96c4fc1700 1 --2- 172.21.15.154:0/156035343 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f96c0103500 0x7f96c0198940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:47:57.312 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:47:57.311+0000 7f96c5fc3700 1 -- 172.21.15.154:0/156035343 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f96c0103500 msgr2=0x7f96c0198940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:47:57.312 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:47:57.311+0000 7f96c5fc3700 1 --2- 172.21.15.154:0/156035343 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f96c0103500 0x7f96c0198940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:47:57.312 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:47:57.311+0000 7f96c5fc3700 1 -- 172.21.15.154:0/156035343 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f96c0102300 msgr2=0x7f96c0198400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:47:57.312 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:47:57.311+0000 7f96c5fc3700 1 --2- 172.21.15.154:0/156035343 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f96c0102300 0x7f96c0198400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:47:57.312 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:47:57.311+0000 7f96c5fc3700 1 -- 172.21.15.154:0/156035343 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96c01a0060 con 0x7f96c0103ea0 2024-03-16T11:47:57.313 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:47:57.311+0000 7f96c57c2700 1 --2- 172.21.15.154:0/156035343 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f96c0102300 0x7f96c0198400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-16T11:47:57.313 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:47:57.311+0000 7f96c5fc3700 1 --2- 172.21.15.154:0/156035343 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f96c0103ea0 0x7f96c019f9c0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f96b800be60 tx=0x7f96b8007820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:47:57.313 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:47:57.311+0000 7f96b67fc700 1 -- 172.21.15.154:0/156035343 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96b8021020 con 0x7f96c0103ea0 2024-03-16T11:47:57.313 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:47:57.311+0000 7f96b67fc700 1 -- 172.21.15.154:0/156035343 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f96b8005520 con 0x7f96c0103ea0 2024-03-16T11:47:57.313 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:47:57.311+0000 7f96c7a26700 1 -- 172.21.15.154:0/156035343 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f96c01a02f0 con 0x7f96c0103ea0 2024-03-16T11:47:57.313 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:47:57.311+0000 7f96b67fc700 1 -- 172.21.15.154:0/156035343 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96b801ba00 con 0x7f96c0103ea0 2024-03-16T11:47:57.313 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:47:57.311+0000 7f96c7a26700 1 -- 172.21.15.154:0/156035343 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f96c01a07f0 con 0x7f96c0103ea0 2024-03-16T11:47:57.313 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:47:57.312+0000 7f96c7a26700 1 -- 172.21.15.154:0/156035343 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f96c0068f90 con 0x7f96c0103ea0 2024-03-16T11:47:57.314 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:47:57.314+0000 7f96b67fc700 1 -- 172.21.15.154:0/156035343 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f96b8024020 con 0x7f96c0103ea0 2024-03-16T11:47:57.314 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:47:57.314+0000 7f96b67fc700 1 --2- 172.21.15.154:0/156035343 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f96ac0a2910 0x7f96ac0a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:47:57.314 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:47:57.315+0000 7f96b67fc700 1 -- 172.21.15.154:0/156035343 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f96b800f020 con 0x7f96c0103ea0 2024-03-16T11:47:57.314 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:47:57.315+0000 7f96c57c2700 1 --2- 172.21.15.154:0/156035343 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f96ac0a2910 0x7f96ac0a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:47:57.314 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:47:57.315+0000 7f96c57c2700 1 --2- 172.21.15.154:0/156035343 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f96ac0a2910 0x7f96ac0a4dd0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f96bc005fd0 tx=0x7f96bc005e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:47:57.317 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:47:57.318+0000 7f96b67fc700 1 -- 172.21.15.154:0/156035343 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f96b809d250 con 0x7f96c0103ea0 2024-03-16T11:47:57.522 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:47:57.523+0000 7f96c7a26700 1 -- 172.21.15.154:0/156035343 --> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] -- mgr_command(tid 0: {"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi029=a;smithi029=d;smithi029=g;smithi029=j;smithi152=b;smithi152=e;smithi152=h;smithi152=k;smithi154=c;smithi154=f;smithi154=i;smithi154=l", "target": ["mon-mgr", ""]}) -- 0x7f96c0062910 con 0x7f96ac0a2910 2024-03-16T11:47:57.530 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:47:57.531+0000 7f96b67fc700 1 -- 172.21.15.154:0/156035343 <== mgr.14152 v2:172.21.15.29:6800/4192064640 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+28 (secure 0 0 0) 0x7f96c0062910 con 0x7f96ac0a2910 2024-03-16T11:47:57.531 INFO:teuthology.orchestra.run.smithi154.stdout:Scheduled mds.all update... 2024-03-16T11:47:57.533 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:47:57.534+0000 7f96c7a26700 1 -- 172.21.15.154:0/156035343 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f96ac0a2910 msgr2=0x7f96ac0a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:47:57.533 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:47:57.534+0000 7f96c7a26700 1 --2- 172.21.15.154:0/156035343 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f96ac0a2910 0x7f96ac0a4dd0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f96bc005fd0 tx=0x7f96bc005e40 comp rx=0 tx=0).stop 2024-03-16T11:47:57.533 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:47:57.534+0000 7f96c7a26700 1 -- 172.21.15.154:0/156035343 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f96c0103ea0 msgr2=0x7f96c019f9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:47:57.533 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:47:57.534+0000 7f96c7a26700 1 --2- 172.21.15.154:0/156035343 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f96c0103ea0 0x7f96c019f9c0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f96b800be60 tx=0x7f96b8007820 comp rx=0 tx=0).stop 2024-03-16T11:47:57.534 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:47:57.534+0000 7f96c7a26700 1 -- 172.21.15.154:0/156035343 shutdown_connections 2024-03-16T11:47:57.534 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:47:57.534+0000 7f96c7a26700 1 --2- 172.21.15.154:0/156035343 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f96ac0a2910 0x7f96ac0a4dd0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:47:57.534 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:47:57.534+0000 7f96c7a26700 1 --2- 172.21.15.154:0/156035343 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f96c0103ea0 0x7f96c019f9c0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:47:57.534 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:47:57.534+0000 7f96c7a26700 1 --2- 172.21.15.154:0/156035343 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f96c0102300 0x7f96c0198400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:47:57.534 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:47:57.534+0000 7f96c7a26700 1 --2- 172.21.15.154:0/156035343 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f96c0103500 0x7f96c0198940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:47:57.534 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:47:57.534+0000 7f96c7a26700 1 -- 172.21.15.154:0/156035343 >> 172.21.15.154:0/156035343 conn(0x7f96c00fd8e0 msgr2=0x7f96c01067a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:47:57.534 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:47:57.534+0000 7f96c7a26700 1 -- 172.21.15.154:0/156035343 shutdown_connections 2024-03-16T11:47:57.534 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-16T11:47:57.534+0000 7f96c7a26700 1 -- 172.21.15.154:0/156035343 wait complete. 2024-03-16T11:47:58.231 DEBUG:teuthology.orchestra.run.smithi029:mds.a> sudo journalctl -f -n 0 -u ceph-d1887f28-e389-11ee-95c9-87774f69a715@mds.a.service 2024-03-16T11:47:58.234 DEBUG:teuthology.orchestra.run.smithi029:mds.d> sudo journalctl -f -n 0 -u ceph-d1887f28-e389-11ee-95c9-87774f69a715@mds.d.service 2024-03-16T11:47:58.277 DEBUG:teuthology.orchestra.run.smithi029:mds.g> sudo journalctl -f -n 0 -u ceph-d1887f28-e389-11ee-95c9-87774f69a715@mds.g.service 2024-03-16T11:47:58.321 DEBUG:teuthology.orchestra.run.smithi029:mds.j> sudo journalctl -f -n 0 -u ceph-d1887f28-e389-11ee-95c9-87774f69a715@mds.j.service 2024-03-16T11:47:58.364 DEBUG:teuthology.orchestra.run.smithi152:mds.b> sudo journalctl -f -n 0 -u ceph-d1887f28-e389-11ee-95c9-87774f69a715@mds.b.service 2024-03-16T11:47:58.366 DEBUG:teuthology.orchestra.run.smithi152:mds.e> sudo journalctl -f -n 0 -u ceph-d1887f28-e389-11ee-95c9-87774f69a715@mds.e.service 2024-03-16T11:47:58.409 DEBUG:teuthology.orchestra.run.smithi152:mds.h> sudo journalctl -f -n 0 -u ceph-d1887f28-e389-11ee-95c9-87774f69a715@mds.h.service 2024-03-16T11:47:58.452 DEBUG:teuthology.orchestra.run.smithi152:mds.k> sudo journalctl -f -n 0 -u ceph-d1887f28-e389-11ee-95c9-87774f69a715@mds.k.service 2024-03-16T11:47:58.495 DEBUG:teuthology.orchestra.run.smithi154:mds.c> sudo journalctl -f -n 0 -u ceph-d1887f28-e389-11ee-95c9-87774f69a715@mds.c.service 2024-03-16T11:47:58.497 DEBUG:teuthology.orchestra.run.smithi154:mds.f> sudo journalctl -f -n 0 -u ceph-d1887f28-e389-11ee-95c9-87774f69a715@mds.f.service 2024-03-16T11:47:58.540 DEBUG:teuthology.orchestra.run.smithi154:mds.i> sudo journalctl -f -n 0 -u ceph-d1887f28-e389-11ee-95c9-87774f69a715@mds.i.service 2024-03-16T11:47:58.584 DEBUG:teuthology.orchestra.run.smithi154:mds.l> sudo journalctl -f -n 0 -u ceph-d1887f28-e389-11ee-95c9-87774f69a715@mds.l.service 2024-03-16T11:47:58.628 INFO:tasks.cephadm:Setting up CephFS filesystem(s)... 2024-03-16T11:47:58.628 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-16T11:47:58.749 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:58.746+0000 7fb7cdadc640 1 -- 172.21.15.29:0/1097819046 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb7c81022b0 msgr2=0x7fb7c8108b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:47:58.749 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:58.746+0000 7fb7cdadc640 1 --2- 172.21.15.29:0/1097819046 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb7c81022b0 0x7fb7c8108b40 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fb7c0009ca0 tx=0x7fb7c002c0f0 comp rx=0 tx=0).stop 2024-03-16T11:47:58.749 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:58.746+0000 7fb7cdadc640 1 -- 172.21.15.29:0/1097819046 shutdown_connections 2024-03-16T11:47:58.749 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:58.746+0000 7fb7cdadc640 1 --2- 172.21.15.29:0/1097819046 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb7c81022b0 0x7fb7c8108b40 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:47:58.749 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:58.746+0000 7fb7cdadc640 1 --2- 172.21.15.29:0/1097819046 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb7c81018f0 0x7fb7c8101d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:47:58.749 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:58.746+0000 7fb7cdadc640 1 --2- 172.21.15.29:0/1097819046 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb7c8100700 0x7fb7c8100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:47:58.749 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:58.746+0000 7fb7cdadc640 1 -- 172.21.15.29:0/1097819046 >> 172.21.15.29:0/1097819046 conn(0x7fb7c80fbc70 msgr2=0x7fb7c80fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:47:58.749 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:58.746+0000 7fb7cdadc640 1 -- 172.21.15.29:0/1097819046 shutdown_connections 2024-03-16T11:47:58.749 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:58.746+0000 7fb7cdadc640 1 -- 172.21.15.29:0/1097819046 wait complete. 2024-03-16T11:47:58.749 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:58.747+0000 7fb7cdadc640 1 Processor -- start 2024-03-16T11:47:58.749 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:58.747+0000 7fb7cdadc640 1 -- start start 2024-03-16T11:47:58.750 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:58.747+0000 7fb7cdadc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb7c8100700 0x7fb7c81969a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:47:58.750 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:58.747+0000 7fb7cdadc640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb7c81018f0 0x7fb7c8196ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:47:58.750 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:58.747+0000 7fb7cdadc640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb7c81022b0 0x7fb7c819df50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:47:58.750 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:58.747+0000 7fb7cdadc640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fb7c810c360 con 0x7fb7c8100700 2024-03-16T11:47:58.750 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:58.747+0000 7fb7cdadc640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fb7c810c1e0 con 0x7fb7c81018f0 2024-03-16T11:47:58.750 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:58.747+0000 7fb7cdadc640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fb7c810c4e0 con 0x7fb7c81022b0 2024-03-16T11:47:58.750 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:58.748+0000 7fb7c77fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb7c8100700 0x7fb7c81969a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:47:58.750 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:58.748+0000 7fb7c77fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb7c8100700 0x7fb7c81969a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57486/0 (socket says 172.21.15.29:57486) 2024-03-16T11:47:58.750 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:58.748+0000 7fb7c77fe640 1 -- 172.21.15.29:0/3024990105 learned_addr learned my addr 172.21.15.29:0/3024990105 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:47:58.750 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:58.748+0000 7fb7c7fff640 1 --2- 172.21.15.29:0/3024990105 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb7c81022b0 0x7fb7c819df50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:47:58.750 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:58.748+0000 7fb7c6ffd640 1 --2- 172.21.15.29:0/3024990105 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb7c81018f0 0x7fb7c8196ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:47:58.750 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:58.748+0000 7fb7c77fe640 1 -- 172.21.15.29:0/3024990105 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb7c81022b0 msgr2=0x7fb7c819df50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:47:58.750 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:58.748+0000 7fb7c77fe640 1 --2- 172.21.15.29:0/3024990105 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb7c81022b0 0x7fb7c819df50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:47:58.750 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:58.748+0000 7fb7c77fe640 1 -- 172.21.15.29:0/3024990105 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb7c81018f0 msgr2=0x7fb7c8196ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:47:58.751 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:58.748+0000 7fb7c77fe640 1 --2- 172.21.15.29:0/3024990105 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb7c81018f0 0x7fb7c8196ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:47:58.751 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:58.748+0000 7fb7c77fe640 1 -- 172.21.15.29:0/3024990105 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7c819e5c0 con 0x7fb7c8100700 2024-03-16T11:47:58.751 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:58.748+0000 7fb7c6ffd640 1 --2- 172.21.15.29:0/3024990105 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb7c81018f0 0x7fb7c8196ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T11:47:58.751 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:58.748+0000 7fb7c77fe640 1 --2- 172.21.15.29:0/3024990105 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb7c8100700 0x7fb7c81969a0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fb7b800af10 tx=0x7fb7b800cd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:47:58.751 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:58.748+0000 7fb7c4ff9640 1 -- 172.21.15.29:0/3024990105 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7b8004220 con 0x7fb7c8100700 2024-03-16T11:47:58.751 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:58.748+0000 7fb7c4ff9640 1 -- 172.21.15.29:0/3024990105 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb7b80043c0 con 0x7fb7c8100700 2024-03-16T11:47:58.751 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:58.748+0000 7fb7c4ff9640 1 -- 172.21.15.29:0/3024990105 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7b800f550 con 0x7fb7c8100700 2024-03-16T11:47:58.751 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:58.748+0000 7fb7cdadc640 1 -- 172.21.15.29:0/3024990105 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb7c80689a0 con 0x7fb7c8100700 2024-03-16T11:47:58.751 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:58.748+0000 7fb7cdadc640 1 -- 172.21.15.29:0/3024990105 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb7c8068f40 con 0x7fb7c8100700 2024-03-16T11:47:58.752 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:58.749+0000 7fb7cdadc640 1 -- 172.21.15.29:0/3024990105 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb7c80685d0 con 0x7fb7c8100700 2024-03-16T11:47:58.752 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:58.750+0000 7fb7c4ff9640 1 -- 172.21.15.29:0/3024990105 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fb7b80079b0 con 0x7fb7c8100700 2024-03-16T11:47:58.752 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:58.751+0000 7fb7c4ff9640 1 --2- 172.21.15.29:0/3024990105 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fb7b00a2880 0x7fb7b00a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:47:58.752 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:58.751+0000 7fb7c4ff9640 1 -- 172.21.15.29:0/3024990105 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fb7b80d0390 con 0x7fb7c8100700 2024-03-16T11:47:58.752 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:58.751+0000 7fb7c6ffd640 1 --2- 172.21.15.29:0/3024990105 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fb7b00a2880 0x7fb7b00a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:47:58.752 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:58.751+0000 7fb7c6ffd640 1 --2- 172.21.15.29:0/3024990105 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fb7b00a2880 0x7fb7b00a4d40 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fb7c8197eb0 tx=0x7fb7bc005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:47:58.755 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:58.755+0000 7fb7c4ff9640 1 -- 172.21.15.29:0/3024990105 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fb7b809d8d0 con 0x7fb7c8100700 2024-03-16T11:47:58.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:47:58 smithi154 ceph-mon[36320]: from='client.24400 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi029=a;smithi029=d;smithi029=g;smithi029=j;smithi152=b;smithi152=e;smithi152=h;smithi152=k;smithi154=c;smithi154=f;smithi154=i;smithi154=l", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T11:47:58.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:47:58 smithi154 ceph-mon[36320]: Saving service mds.all spec with placement smithi029=a;smithi029=d;smithi029=g;smithi029=j;smithi152=b;smithi152=e;smithi152=h;smithi152=k;smithi154=c;smithi154=f;smithi154=i;smithi154=l;count:12 2024-03-16T11:47:58.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:47:58 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:47:58.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:47:58 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T11:47:58.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:47:58 smithi154 ceph-mon[36320]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T11:47:58.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:47:58 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:47:58.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:47:58 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T11:47:58.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:47:58 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:47:58.796 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:47:58 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-16T11:47:58.796 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:47:58 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-16T11:47:58.796 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:47:58 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:47:58.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:47:58 smithi029 ceph-mon[31975]: from='client.24400 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi029=a;smithi029=d;smithi029=g;smithi029=j;smithi152=b;smithi152=e;smithi152=h;smithi152=k;smithi154=c;smithi154=f;smithi154=i;smithi154=l", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T11:47:58.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:47:58 smithi029 ceph-mon[31975]: Saving service mds.all spec with placement smithi029=a;smithi029=d;smithi029=g;smithi029=j;smithi152=b;smithi152=e;smithi152=h;smithi152=k;smithi154=c;smithi154=f;smithi154=i;smithi154=l;count:12 2024-03-16T11:47:58.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:47:58 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:47:58.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:47:58 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T11:47:58.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:47:58 smithi029 ceph-mon[31975]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T11:47:58.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:47:58 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:47:58.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:47:58 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T11:47:58.839 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:47:58 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:47:58.839 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:47:58 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-16T11:47:58.839 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:47:58 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-16T11:47:58.839 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:47:58 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:47:58.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:47:58 smithi152 ceph-mon[36291]: from='client.24400 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi029=a;smithi029=d;smithi029=g;smithi029=j;smithi152=b;smithi152=e;smithi152=h;smithi152=k;smithi154=c;smithi154=f;smithi154=i;smithi154=l", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T11:47:58.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:47:58 smithi152 ceph-mon[36291]: Saving service mds.all spec with placement smithi029=a;smithi029=d;smithi029=g;smithi029=j;smithi152=b;smithi152=e;smithi152=h;smithi152=k;smithi154=c;smithi154=f;smithi154=i;smithi154=l;count:12 2024-03-16T11:47:58.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:47:58 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:47:58.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:47:58 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T11:47:58.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:47:58 smithi152 ceph-mon[36291]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T11:47:58.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:47:58 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:47:58.878 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:47:58 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T11:47:58.878 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:47:58 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:47:58.878 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:47:58 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-16T11:47:58.878 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:47:58 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-16T11:47:58.878 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:47:58 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:47:58.940 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:58.940+0000 7fb7cdadc640 1 -- 172.21.15.29:0/3024990105 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb7c8107740 con 0x7fb7c8100700 2024-03-16T11:47:58.941 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:58.941+0000 7fb7c4ff9640 1 -- 172.21.15.29:0/3024990105 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) ==== 74+0+18470 (secure 0 0 0) 0x7fb7b809d1d0 con 0x7fb7c8100700 2024-03-16T11:47:58.941 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-16T11:47:58.941 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":81,"fsid":"d1887f28-e389-11ee-95c9-87774f69a715","created":"2024-03-16T11:40:16.475063+0000","modified":"2024-03-16T11:47:45.354317+0000","last_up_change":"2024-03-16T11:47:44.349348+0000","last_in_change":"2024-03-16T11:47:23.672432+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-16T11:43:51.892652+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"60f5608f-a408-45ba-bdf6-5b53758f88fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":2748947913},{"type":"v1","addr":"172.21.15.29:6803","nonce":2748947913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":2748947913},{"type":"v1","addr":"172.21.15.29:6805","nonce":2748947913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":2748947913},{"type":"v1","addr":"172.21.15.29:6809","nonce":2748947913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":2748947913},{"type":"v1","addr":"172.21.15.29:6807","nonce":2748947913}]},"public_addr":"172.21.15.29:6803/2748947913","cluster_addr":"172.21.15.29:6805/2748947913","heartbeat_back_addr":"172.21.15.29:6809/2748947913","heartbeat_front_addr":"172.21.15.29:6807/2748947913","state":["exists","up"]},{"osd":1,"uuid":"e70b53d6-ee78-4e56-9a2c-263ae0501b43","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6800","nonce":2339408422},{"type":"v1","addr":"172.21.15.152:6801","nonce":2339408422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6802","nonce":2339408422},{"type":"v1","addr":"172.21.15.152:6803","nonce":2339408422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6806","nonce":2339408422},{"type":"v1","addr":"172.21.15.152:6807","nonce":2339408422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6804","nonce":2339408422},{"type":"v1","addr":"172.21.15.152:6805","nonce":2339408422}]},"public_addr":"172.21.15.152:6801/2339408422","cluster_addr":"172.21.15.152:6803/2339408422","heartbeat_back_addr":"172.21.15.152:6807/2339408422","heartbeat_front_addr":"172.21.15.152:6805/2339408422","state":["exists","up"]},{"osd":2,"uuid":"ff39389a-dfac-44b5-80ff-36abca316e78","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6800","nonce":4288325280},{"type":"v1","addr":"172.21.15.154:6801","nonce":4288325280}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6802","nonce":4288325280},{"type":"v1","addr":"172.21.15.154:6803","nonce":4288325280}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6806","nonce":4288325280},{"type":"v1","addr":"172.21.15.154:6807","nonce":4288325280}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":4288325280},{"type":"v1","addr":"172.21.15.154:6805","nonce":4288325280}]},"public_addr":"172.21.15.154:6801/4288325280","cluster_addr":"172.21.15.154:6803/4288325280","heartbeat_back_addr":"172.21.15.154:6807/4288325280","heartbeat_front_addr":"172.21.15.154:6805/4288325280","state":["exists","up"]},{"osd":3,"uuid":"069e1795-55bc-4324-b94c-eb24ef2334c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":1538750397},{"type":"v1","addr":"172.21.15.29:6811","nonce":1538750397}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":1538750397},{"type":"v1","addr":"172.21.15.29:6813","nonce":1538750397}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":1538750397},{"type":"v1","addr":"172.21.15.29:6817","nonce":1538750397}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":1538750397},{"type":"v1","addr":"172.21.15.29:6815","nonce":1538750397}]},"public_addr":"172.21.15.29:6811/1538750397","cluster_addr":"172.21.15.29:6813/1538750397","heartbeat_back_addr":"172.21.15.29:6817/1538750397","heartbeat_front_addr":"172.21.15.29:6815/1538750397","state":["exists","up"]},{"osd":4,"uuid":"5a9d43a1-e8f5-4ab5-a441-afbc6ed4066e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6808","nonce":2795081750},{"type":"v1","addr":"172.21.15.152:6809","nonce":2795081750}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6810","nonce":2795081750},{"type":"v1","addr":"172.21.15.152:6811","nonce":2795081750}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6814","nonce":2795081750},{"type":"v1","addr":"172.21.15.152:6815","nonce":2795081750}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6812","nonce":2795081750},{"type":"v1","addr":"172.21.15.152:6813","nonce":2795081750}]},"public_addr":"172.21.15.152:6809/2795081750","cluster_addr":"172.21.15.152:6811/2795081750","heartbeat_back_addr":"172.21.15.152:6815/2795081750","heartbeat_front_addr":"172.21.15.152:6813/2795081750","state":["exists","up"]},{"osd":5,"uuid":"234de428-4ac7-44be-9a54-7b58331fdf8c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":1472686560},{"type":"v1","addr":"172.21.15.154:6809","nonce":1472686560}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6810","nonce":1472686560},{"type":"v1","addr":"172.21.15.154:6811","nonce":1472686560}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6814","nonce":1472686560},{"type":"v1","addr":"172.21.15.154:6815","nonce":1472686560}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6812","nonce":1472686560},{"type":"v1","addr":"172.21.15.154:6813","nonce":1472686560}]},"public_addr":"172.21.15.154:6809/1472686560","cluster_addr":"172.21.15.154:6811/1472686560","heartbeat_back_addr":"172.21.15.154:6815/1472686560","heartbeat_front_addr":"172.21.15.154:6813/1472686560","state":["exists","up"]},{"osd":6,"uuid":"ae45b8d9-7883-40c9-a9cd-4c92b601d4f2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":816506244},{"type":"v1","addr":"172.21.15.29:6819","nonce":816506244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":816506244},{"type":"v1","addr":"172.21.15.29:6821","nonce":816506244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":816506244},{"type":"v1","addr":"172.21.15.29:6825","nonce":816506244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":816506244},{"type":"v1","addr":"172.21.15.29:6823","nonce":816506244}]},"public_addr":"172.21.15.29:6819/816506244","cluster_addr":"172.21.15.29:6821/816506244","heartbeat_back_addr":"172.21.15.29:6825/816506244","heartbeat_front_addr":"172.21.15.29:6823/816506244","state":["exists","up"]},{"osd":7,"uuid":"6f687b28-4fce-4d4a-8b4e-294173d9d6c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6816","nonce":2924651787},{"type":"v1","addr":"172.21.15.152:6817","nonce":2924651787}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6818","nonce":2924651787},{"type":"v1","addr":"172.21.15.152:6819","nonce":2924651787}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6822","nonce":2924651787},{"type":"v1","addr":"172.21.15.152:6823","nonce":2924651787}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6820","nonce":2924651787},{"type":"v1","addr":"172.21.15.152:6821","nonce":2924651787}]},"public_addr":"172.21.15.152:6817/2924651787","cluster_addr":"172.21.15.152:6819/2924651787","heartbeat_back_addr":"172.21.15.152:6823/2924651787","heartbeat_front_addr":"172.21.15.152:6821/2924651787","state":["exists","up"]},{"osd":8,"uuid":"a25fb741-caa7-429a-b540-e394d034b446","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6816","nonce":129974156},{"type":"v1","addr":"172.21.15.154:6817","nonce":129974156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6818","nonce":129974156},{"type":"v1","addr":"172.21.15.154:6819","nonce":129974156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6822","nonce":129974156},{"type":"v1","addr":"172.21.15.154:6823","nonce":129974156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6820","nonce":129974156},{"type":"v1","addr":"172.21.15.154:6821","nonce":129974156}]},"public_addr":"172.21.15.154:6817/129974156","cluster_addr":"172.21.15.154:6819/129974156","heartbeat_back_addr":"172.21.15.154:6823/129974156","heartbeat_front_addr":"172.21.15.154:6821/129974156","state":["exists","up"]},{"osd":9,"uuid":"1071cb0e-8c5f-4cd8-9805-0045e220d946","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6826","nonce":2951414243},{"type":"v1","addr":"172.21.15.29:6827","nonce":2951414243}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":2951414243},{"type":"v1","addr":"172.21.15.29:6829","nonce":2951414243}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":2951414243},{"type":"v1","addr":"172.21.15.29:6833","nonce":2951414243}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":2951414243},{"type":"v1","addr":"172.21.15.29:6831","nonce":2951414243}]},"public_addr":"172.21.15.29:6827/2951414243","cluster_addr":"172.21.15.29:6829/2951414243","heartbeat_back_addr":"172.21.15.29:6833/2951414243","heartbeat_front_addr":"172.21.15.29:6831/2951414243","state":["exists","up"]},{"osd":10,"uuid":"ee7c7d74-3f39-4cd4-be80-843fb1dd9ecc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6824","nonce":3412525480},{"type":"v1","addr":"172.21.15.152:6825","nonce":3412525480}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6826","nonce":3412525480},{"type":"v1","addr":"172.21.15.152:6827","nonce":3412525480}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6830","nonce":3412525480},{"type":"v1","addr":"172.21.15.152:6831","nonce":3412525480}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6828","nonce":3412525480},{"type":"v1","addr":"172.21.15.152:6829","nonce":3412525480}]},"public_addr":"172.21.15.152:6825/3412525480","cluster_addr":"172.21.15.152:6827/3412525480","heartbeat_back_addr":"172.21.15.152:6831/3412525480","heartbeat_front_addr":"172.21.15.152:6829/3412525480","state":["exists","up"]},{"osd":11,"uuid":"4598c803-984a-4de6-ac22-066502c17889","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6824","nonce":1160927966},{"type":"v1","addr":"172.21.15.154:6825","nonce":1160927966}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6826","nonce":1160927966},{"type":"v1","addr":"172.21.15.154:6827","nonce":1160927966}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6830","nonce":1160927966},{"type":"v1","addr":"172.21.15.154:6831","nonce":1160927966}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6828","nonce":1160927966},{"type":"v1","addr":"172.21.15.154:6829","nonce":1160927966}]},"public_addr":"172.21.15.154:6825/1160927966","cluster_addr":"172.21.15.154:6827/1160927966","heartbeat_back_addr":"172.21.15.154:6831/1160927966","heartbeat_front_addr":"172.21.15.154:6829/1160927966","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:43:02.343393+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:43:25.209151+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:43:48.946164+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:44:14.588424+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:44:39.700025+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:45:04.061197+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:45:31.214508+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:45:56.937296+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:46:22.768915+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:46:50.101011+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:47:16.298041+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:47:42.402095+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.29:0/1853086886":"2024-03-17T11:41:27.531724+0000","172.21.15.29:0/1211659364":"2024-03-17T11:41:27.531724+0000","172.21.15.29:6800/2798831128":"2024-03-17T11:41:27.531724+0000","172.21.15.29:0/1281028285":"2024-03-17T11:41:00.505678+0000","172.21.15.29:6800/2275315654":"2024-03-17T11:41:00.505678+0000","172.21.15.29:6801/2798831128":"2024-03-17T11:41:27.531724+0000","172.21.15.29:6801/2275315654":"2024-03-17T11:41:00.505678+0000","172.21.15.29:0/1330100009":"2024-03-17T11:41:00.505678+0000","172.21.15.29:0/1784249039":"2024-03-17T11:41:00.505678+0000","172.21.15.29:0/1116486932":"2024-03-17T11:41:27.531724+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-16T11:47:58.943 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:58.943+0000 7fb7cdadc640 1 -- 172.21.15.29:0/3024990105 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fb7b00a2880 msgr2=0x7fb7b00a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:47:58.943 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:58.943+0000 7fb7cdadc640 1 --2- 172.21.15.29:0/3024990105 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fb7b00a2880 0x7fb7b00a4d40 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fb7c8197eb0 tx=0x7fb7bc005f90 comp rx=0 tx=0).stop 2024-03-16T11:47:58.943 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:58.943+0000 7fb7cdadc640 1 -- 172.21.15.29:0/3024990105 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb7c8100700 msgr2=0x7fb7c81969a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:47:58.943 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:58.943+0000 7fb7cdadc640 1 --2- 172.21.15.29:0/3024990105 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb7c8100700 0x7fb7c81969a0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fb7b800af10 tx=0x7fb7b800cd40 comp rx=0 tx=0).stop 2024-03-16T11:47:58.943 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:58.943+0000 7fb7cdadc640 1 -- 172.21.15.29:0/3024990105 shutdown_connections 2024-03-16T11:47:58.944 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:58.943+0000 7fb7cdadc640 1 --2- 172.21.15.29:0/3024990105 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fb7b00a2880 0x7fb7b00a4d40 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:47:58.944 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:58.943+0000 7fb7cdadc640 1 --2- 172.21.15.29:0/3024990105 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb7c81022b0 0x7fb7c819df50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:47:58.944 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:58.943+0000 7fb7cdadc640 1 --2- 172.21.15.29:0/3024990105 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb7c81018f0 0x7fb7c8196ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:47:58.944 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:58.943+0000 7fb7cdadc640 1 --2- 172.21.15.29:0/3024990105 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb7c8100700 0x7fb7c81969a0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:47:58.944 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:58.943+0000 7fb7cdadc640 1 -- 172.21.15.29:0/3024990105 >> 172.21.15.29:0/3024990105 conn(0x7fb7c80fbc70 msgr2=0x7fb7c8104bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:47:58.944 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:58.943+0000 7fb7cdadc640 1 -- 172.21.15.29:0/3024990105 shutdown_connections 2024-03-16T11:47:58.944 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:58.943+0000 7fb7cdadc640 1 -- 172.21.15.29:0/3024990105 wait complete. 2024-03-16T11:47:58.954 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-16T11:43:51.892652+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-16T11:47:58.955 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-16T11:47:59.064 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:59.063+0000 7fe9b8861640 1 -- 172.21.15.29:0/2083380257 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe9b41006e0 msgr2=0x7fe9b4100b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:47:59.064 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:59.063+0000 7fe9b8861640 1 --2- 172.21.15.29:0/2083380257 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe9b41006e0 0x7fe9b4100b00 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fe9ac009ca0 tx=0x7fe9ac02c0f0 comp rx=0 tx=0).stop 2024-03-16T11:47:59.064 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:59.064+0000 7fe9b8861640 1 -- 172.21.15.29:0/2083380257 shutdown_connections 2024-03-16T11:47:59.064 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:59.064+0000 7fe9b8861640 1 --2- 172.21.15.29:0/2083380257 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe9b4102290 0x7fe9b4108b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:47:59.064 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:59.064+0000 7fe9b8861640 1 --2- 172.21.15.29:0/2083380257 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe9b41018d0 0x7fe9b4101d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:47:59.064 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:59.064+0000 7fe9b8861640 1 --2- 172.21.15.29:0/2083380257 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe9b41006e0 0x7fe9b4100b00 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:47:59.064 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:59.064+0000 7fe9b8861640 1 -- 172.21.15.29:0/2083380257 >> 172.21.15.29:0/2083380257 conn(0x7fe9b40fbc70 msgr2=0x7fe9b40fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:47:59.064 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:59.064+0000 7fe9b8861640 1 -- 172.21.15.29:0/2083380257 shutdown_connections 2024-03-16T11:47:59.064 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:59.064+0000 7fe9b8861640 1 -- 172.21.15.29:0/2083380257 wait complete. 2024-03-16T11:47:59.067 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:59.064+0000 7fe9b8861640 1 Processor -- start 2024-03-16T11:47:59.067 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:59.064+0000 7fe9b8861640 1 -- start start 2024-03-16T11:47:59.068 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:59.064+0000 7fe9b8861640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe9b41006e0 0x7fe9b40694b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:47:59.068 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:59.064+0000 7fe9b8861640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe9b41018d0 0x7fe9b419c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:47:59.068 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:59.064+0000 7fe9b8861640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe9b4102290 0x7fe9b419eef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:47:59.068 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:59.064+0000 7fe9b8861640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fe9b410c330 con 0x7fe9b41006e0 2024-03-16T11:47:59.068 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:59.064+0000 7fe9b8861640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fe9b410c1b0 con 0x7fe9b41018d0 2024-03-16T11:47:59.068 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:59.065+0000 7fe9b8861640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fe9b410c4b0 con 0x7fe9b4102290 2024-03-16T11:47:59.068 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:59.065+0000 7fe9b259c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe9b41006e0 0x7fe9b40694b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:47:59.068 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:59.065+0000 7fe9b259c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe9b41006e0 0x7fe9b40694b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57512/0 (socket says 172.21.15.29:57512) 2024-03-16T11:47:59.068 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:59.065+0000 7fe9b259c640 1 -- 172.21.15.29:0/3593325878 learned_addr learned my addr 172.21.15.29:0/3593325878 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:47:59.068 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:59.065+0000 7fe9b2d9d640 1 --2- 172.21.15.29:0/3593325878 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe9b4102290 0x7fe9b419eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:47:59.068 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:59.065+0000 7fe9b1d9b640 1 --2- 172.21.15.29:0/3593325878 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe9b41018d0 0x7fe9b419c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:47:59.068 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:59.065+0000 7fe9b259c640 1 -- 172.21.15.29:0/3593325878 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe9b4102290 msgr2=0x7fe9b419eef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:47:59.068 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:59.065+0000 7fe9b259c640 1 --2- 172.21.15.29:0/3593325878 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe9b4102290 0x7fe9b419eef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:47:59.068 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:59.065+0000 7fe9b259c640 1 -- 172.21.15.29:0/3593325878 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe9b41018d0 msgr2=0x7fe9b419c8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:47:59.069 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:59.065+0000 7fe9b259c640 1 --2- 172.21.15.29:0/3593325878 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe9b41018d0 0x7fe9b419c8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:47:59.069 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:59.065+0000 7fe9b259c640 1 -- 172.21.15.29:0/3593325878 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9b419f800 con 0x7fe9b41006e0 2024-03-16T11:47:59.069 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:59.065+0000 7fe9b259c640 1 --2- 172.21.15.29:0/3593325878 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe9b41006e0 0x7fe9b40694b0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fe9ac0324c0 tx=0x7fe9ac032f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:47:59.069 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:59.065+0000 7fe9ab7fe640 1 -- 172.21.15.29:0/3593325878 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9ac039040 con 0x7fe9b41006e0 2024-03-16T11:47:59.069 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:59.065+0000 7fe9ab7fe640 1 -- 172.21.15.29:0/3593325878 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe9ac044070 con 0x7fe9b41006e0 2024-03-16T11:47:59.069 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:59.065+0000 7fe9b8861640 1 -- 172.21.15.29:0/3593325878 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe9b419fa30 con 0x7fe9b41006e0 2024-03-16T11:47:59.069 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:59.065+0000 7fe9ab7fe640 1 -- 172.21.15.29:0/3593325878 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9ac0042a0 con 0x7fe9b41006e0 2024-03-16T11:47:59.069 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:59.065+0000 7fe9b8861640 1 -- 172.21.15.29:0/3593325878 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe9b419ff20 con 0x7fe9b41006e0 2024-03-16T11:47:59.069 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:59.066+0000 7fe9b8861640 1 -- 172.21.15.29:0/3593325878 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe9b4107ac0 con 0x7fe9b41006e0 2024-03-16T11:47:59.069 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:59.067+0000 7fe9ab7fe640 1 -- 172.21.15.29:0/3593325878 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fe9ac003970 con 0x7fe9b41006e0 2024-03-16T11:47:59.070 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:59.068+0000 7fe9ab7fe640 1 --2- 172.21.15.29:0/3593325878 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fe9a00a2800 0x7fe9a00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:47:59.070 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:59.068+0000 7fe9ab7fe640 1 -- 172.21.15.29:0/3593325878 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fe9ac0f1590 con 0x7fe9b41006e0 2024-03-16T11:47:59.070 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:59.068+0000 7fe9b1d9b640 1 --2- 172.21.15.29:0/3593325878 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fe9a00a2800 0x7fe9a00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:47:59.070 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:59.069+0000 7fe9b1d9b640 1 --2- 172.21.15.29:0/3593325878 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fe9a00a2800 0x7fe9a00a4cc0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fe99c005d00 tx=0x7fe99c005c50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:47:59.072 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:59.072+0000 7fe9ab7fe640 1 -- 172.21.15.29:0/3593325878 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fe9ac0beac0 con 0x7fe9b41006e0 2024-03-16T11:47:59.255 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:59.255+0000 7fe9b8861640 1 -- 172.21.15.29:0/3593325878 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fe9b402d400 con 0x7fe9b41006e0 2024-03-16T11:47:59.256 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:59.256+0000 7fe9ab7fe640 1 -- 172.21.15.29:0/3593325878 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v81) ==== 93+0+10 (secure 0 0 0) 0x7fe9ac0be3c0 con 0x7fe9b41006e0 2024-03-16T11:47:59.256 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2024-03-16T11:47:59.258 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:59.258+0000 7fe9b8861640 1 -- 172.21.15.29:0/3593325878 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fe9a00a2800 msgr2=0x7fe9a00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:47:59.258 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:59.258+0000 7fe9b8861640 1 --2- 172.21.15.29:0/3593325878 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fe9a00a2800 0x7fe9a00a4cc0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fe99c005d00 tx=0x7fe99c005c50 comp rx=0 tx=0).stop 2024-03-16T11:47:59.258 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:59.258+0000 7fe9b8861640 1 -- 172.21.15.29:0/3593325878 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe9b41006e0 msgr2=0x7fe9b40694b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:47:59.258 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:59.258+0000 7fe9b8861640 1 --2- 172.21.15.29:0/3593325878 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe9b41006e0 0x7fe9b40694b0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fe9ac0324c0 tx=0x7fe9ac032f20 comp rx=0 tx=0).stop 2024-03-16T11:47:59.259 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:59.258+0000 7fe9b8861640 1 -- 172.21.15.29:0/3593325878 shutdown_connections 2024-03-16T11:47:59.259 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:59.258+0000 7fe9b8861640 1 --2- 172.21.15.29:0/3593325878 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fe9a00a2800 0x7fe9a00a4cc0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:47:59.259 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:59.258+0000 7fe9b8861640 1 --2- 172.21.15.29:0/3593325878 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe9b4102290 0x7fe9b419eef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:47:59.259 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:59.258+0000 7fe9b8861640 1 --2- 172.21.15.29:0/3593325878 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe9b41018d0 0x7fe9b419c8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:47:59.259 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:59.258+0000 7fe9b8861640 1 --2- 172.21.15.29:0/3593325878 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe9b41006e0 0x7fe9b40694b0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:47:59.259 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:59.258+0000 7fe9b8861640 1 -- 172.21.15.29:0/3593325878 >> 172.21.15.29:0/3593325878 conn(0x7fe9b40fbc70 msgr2=0x7fe9b4104c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:47:59.259 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:59.258+0000 7fe9b8861640 1 -- 172.21.15.29:0/3593325878 shutdown_connections 2024-03-16T11:47:59.259 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:59.258+0000 7fe9b8861640 1 -- 172.21.15.29:0/3593325878 wait complete. 2024-03-16T11:47:59.269 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T11:47:59.377 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:59.374+0000 7fc887fa9640 1 -- 172.21.15.29:0/4164641677 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc880100700 msgr2=0x7fc880100b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:47:59.377 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:59.374+0000 7fc887fa9640 1 --2- 172.21.15.29:0/4164641677 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc880100700 0x7fc880100b20 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fc87c009900 tx=0x7fc87c02bc30 comp rx=0 tx=0).stop 2024-03-16T11:47:59.377 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:59.374+0000 7fc887fa9640 1 -- 172.21.15.29:0/4164641677 shutdown_connections 2024-03-16T11:47:59.377 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:59.374+0000 7fc887fa9640 1 --2- 172.21.15.29:0/4164641677 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc8801022b0 0x7fc880108b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:47:59.377 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:59.374+0000 7fc887fa9640 1 --2- 172.21.15.29:0/4164641677 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc8801018f0 0x7fc880101d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:47:59.378 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:59.374+0000 7fc887fa9640 1 --2- 172.21.15.29:0/4164641677 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc880100700 0x7fc880100b20 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:47:59.378 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:59.374+0000 7fc887fa9640 1 -- 172.21.15.29:0/4164641677 >> 172.21.15.29:0/4164641677 conn(0x7fc8800fbc70 msgr2=0x7fc8800fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:47:59.378 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:59.374+0000 7fc887fa9640 1 -- 172.21.15.29:0/4164641677 shutdown_connections 2024-03-16T11:47:59.378 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:59.374+0000 7fc887fa9640 1 -- 172.21.15.29:0/4164641677 wait complete. 2024-03-16T11:47:59.378 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:59.375+0000 7fc887fa9640 1 Processor -- start 2024-03-16T11:47:59.378 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:59.375+0000 7fc887fa9640 1 -- start start 2024-03-16T11:47:59.378 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:59.375+0000 7fc887fa9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc880100700 0x7fc880199d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:47:59.378 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:59.375+0000 7fc887fa9640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc8801018f0 0x7fc88019a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:47:59.378 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:59.375+0000 7fc887fa9640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc8801022b0 0x7fc8801a1320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:47:59.378 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:59.375+0000 7fc887fa9640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fc88010c370 con 0x7fc880100700 2024-03-16T11:47:59.378 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:59.375+0000 7fc887fa9640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fc88010c1f0 con 0x7fc8801022b0 2024-03-16T11:47:59.378 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:59.375+0000 7fc887fa9640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fc88010c4f0 con 0x7fc8801018f0 2024-03-16T11:47:59.379 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:59.375+0000 7fc885d45640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc880100700 0x7fc880199d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:47:59.379 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:59.375+0000 7fc885d45640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc880100700 0x7fc880199d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57534/0 (socket says 172.21.15.29:57534) 2024-03-16T11:47:59.379 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:59.375+0000 7fc885d45640 1 -- 172.21.15.29:0/1953420993 learned_addr learned my addr 172.21.15.29:0/1953420993 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:47:59.379 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:59.375+0000 7fc885544640 1 --2- 172.21.15.29:0/1953420993 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc8801018f0 0x7fc88019a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:47:59.379 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:59.376+0000 7fc886546640 1 --2- 172.21.15.29:0/1953420993 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc8801022b0 0x7fc8801a1320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:47:59.379 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:59.376+0000 7fc885d45640 1 -- 172.21.15.29:0/1953420993 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc8801018f0 msgr2=0x7fc88019a2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:47:59.379 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:59.376+0000 7fc885d45640 1 --2- 172.21.15.29:0/1953420993 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc8801018f0 0x7fc88019a2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:47:59.379 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:59.376+0000 7fc885d45640 1 -- 172.21.15.29:0/1953420993 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc8801022b0 msgr2=0x7fc8801a1320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:47:59.379 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:59.376+0000 7fc885d45640 1 --2- 172.21.15.29:0/1953420993 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc8801022b0 0x7fc8801a1320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:47:59.379 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:59.376+0000 7fc885d45640 1 -- 172.21.15.29:0/1953420993 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8801a1990 con 0x7fc880100700 2024-03-16T11:47:59.379 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:59.376+0000 7fc885544640 1 --2- 172.21.15.29:0/1953420993 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc8801018f0 0x7fc88019a2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:47:59.379 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:59.376+0000 7fc886546640 1 --2- 172.21.15.29:0/1953420993 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc8801022b0 0x7fc8801a1320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T11:47:59.380 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:59.376+0000 7fc885d45640 1 --2- 172.21.15.29:0/1953420993 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc880100700 0x7fc880199d70 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fc87c000c00 tx=0x7fc87c02c640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:47:59.380 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:59.376+0000 7fc876ffd640 1 -- 172.21.15.29:0/1953420993 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc87c041070 con 0x7fc880100700 2024-03-16T11:47:59.380 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:59.376+0000 7fc876ffd640 1 -- 172.21.15.29:0/1953420993 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc87c004450 con 0x7fc880100700 2024-03-16T11:47:59.380 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:59.376+0000 7fc887fa9640 1 -- 172.21.15.29:0/1953420993 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc8801a1c20 con 0x7fc880100700 2024-03-16T11:47:59.380 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:59.376+0000 7fc876ffd640 1 -- 172.21.15.29:0/1953420993 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc87c034440 con 0x7fc880100700 2024-03-16T11:47:59.380 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:59.376+0000 7fc887fa9640 1 -- 172.21.15.29:0/1953420993 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc8801a2110 con 0x7fc880100700 2024-03-16T11:47:59.380 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:59.377+0000 7fc887fa9640 1 -- 172.21.15.29:0/1953420993 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc8801a23a0 con 0x7fc880100700 2024-03-16T11:47:59.381 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:59.379+0000 7fc876ffd640 1 -- 172.21.15.29:0/1953420993 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fc87c044070 con 0x7fc880100700 2024-03-16T11:47:59.381 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:59.379+0000 7fc876ffd640 1 --2- 172.21.15.29:0/1953420993 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fc86c0a2800 0x7fc86c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:47:59.381 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:59.380+0000 7fc885544640 1 --2- 172.21.15.29:0/1953420993 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fc86c0a2800 0x7fc86c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:47:59.381 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:59.380+0000 7fc876ffd640 1 -- 172.21.15.29:0/1953420993 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fc87c0f01c0 con 0x7fc880100700 2024-03-16T11:47:59.381 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:59.380+0000 7fc885544640 1 --2- 172.21.15.29:0/1953420993 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fc86c0a2800 0x7fc86c0a4cc0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fc88019b280 tx=0x7fc870005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:47:59.384 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:59.384+0000 7fc876ffd640 1 -- 172.21.15.29:0/1953420993 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fc87c0bd6f0 con 0x7fc880100700 2024-03-16T11:47:59.609 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:59.608+0000 7fc887fa9640 1 -- 172.21.15.29:0/1953420993 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc8801a2730 con 0x7fc880100700 2024-03-16T11:47:59.609 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:59.609+0000 7fc876ffd640 1 -- 172.21.15.29:0/1953420993 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 1 v1) ==== 93+0+569 (secure 0 0 0) 0x7fc87c0bcff0 con 0x7fc880100700 2024-03-16T11:47:59.609 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-16T11:47:59.609 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":1,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[]} 2024-03-16T11:47:59.610 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 1 2024-03-16T11:47:59.611 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:59.611+0000 7fc887fa9640 1 -- 172.21.15.29:0/1953420993 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fc86c0a2800 msgr2=0x7fc86c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:47:59.611 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:59.611+0000 7fc887fa9640 1 --2- 172.21.15.29:0/1953420993 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fc86c0a2800 0x7fc86c0a4cc0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fc88019b280 tx=0x7fc870005f90 comp rx=0 tx=0).stop 2024-03-16T11:47:59.611 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:59.611+0000 7fc887fa9640 1 -- 172.21.15.29:0/1953420993 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc880100700 msgr2=0x7fc880199d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:47:59.612 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:59.611+0000 7fc887fa9640 1 --2- 172.21.15.29:0/1953420993 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc880100700 0x7fc880199d70 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fc87c000c00 tx=0x7fc87c02c640 comp rx=0 tx=0).stop 2024-03-16T11:47:59.612 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:59.611+0000 7fc887fa9640 1 -- 172.21.15.29:0/1953420993 shutdown_connections 2024-03-16T11:47:59.612 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:59.611+0000 7fc887fa9640 1 --2- 172.21.15.29:0/1953420993 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fc86c0a2800 0x7fc86c0a4cc0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:47:59.612 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:59.611+0000 7fc887fa9640 1 --2- 172.21.15.29:0/1953420993 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc8801022b0 0x7fc8801a1320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:47:59.612 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:59.611+0000 7fc887fa9640 1 --2- 172.21.15.29:0/1953420993 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc8801018f0 0x7fc88019a2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:47:59.612 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:59.611+0000 7fc887fa9640 1 --2- 172.21.15.29:0/1953420993 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc880100700 0x7fc880199d70 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:47:59.612 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:59.611+0000 7fc887fa9640 1 -- 172.21.15.29:0/1953420993 >> 172.21.15.29:0/1953420993 conn(0x7fc8800fbc70 msgr2=0x7fc8800fd7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:47:59.612 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:59.611+0000 7fc887fa9640 1 -- 172.21.15.29:0/1953420993 shutdown_connections 2024-03-16T11:47:59.612 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:47:59.611+0000 7fc887fa9640 1 -- 172.21.15.29:0/1953420993 wait complete. 2024-03-16T11:47:59.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:47:59 smithi154 ceph-mon[36320]: Deploying daemon mds.l on smithi154 2024-03-16T11:47:59.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:47:59 smithi154 ceph-mon[36320]: from='client.? 172.21.15.29:0/3024990105' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T11:47:59.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:47:59 smithi154 ceph-mon[36320]: from='client.? 172.21.15.29:0/3593325878' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-16T11:47:59.837 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:47:59 smithi029 ceph-mon[31975]: Deploying daemon mds.l on smithi154 2024-03-16T11:47:59.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:47:59 smithi029 ceph-mon[31975]: from='client.? 172.21.15.29:0/3024990105' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T11:47:59.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:47:59 smithi029 ceph-mon[31975]: from='client.? 172.21.15.29:0/3593325878' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-16T11:47:59.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:47:59 smithi152 ceph-mon[36291]: Deploying daemon mds.l on smithi154 2024-03-16T11:47:59.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:47:59 smithi152 ceph-mon[36291]: from='client.? 172.21.15.29:0/3024990105' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T11:47:59.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:47:59 smithi152 ceph-mon[36291]: from='client.? 172.21.15.29:0/3593325878' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-16T11:48:00.766 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:00 smithi154 ceph-mon[36320]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T11:48:00.766 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:00 smithi154 ceph-mon[36320]: from='client.? 172.21.15.29:0/1953420993' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T11:48:00.767 INFO:journalctl@ceph.mds.l.smithi154.stdout:Mar 16 11:48:00 smithi154 systemd[1]: Starting Ceph mds.l for d1887f28-e389-11ee-95c9-87774f69a715... 2024-03-16T11:48:00.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:00 smithi029 ceph-mon[31975]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T11:48:00.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:00 smithi029 ceph-mon[31975]: from='client.? 172.21.15.29:0/1953420993' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T11:48:00.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:00 smithi152 ceph-mon[36291]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T11:48:00.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:00 smithi152 ceph-mon[36291]: from='client.? 172.21.15.29:0/1953420993' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T11:48:01.034 INFO:journalctl@ceph.mds.l.smithi154.stdout:Mar 16 11:48:00 smithi154 podman[66094]: 2024-03-16 11:48:00.670707258 +0000 UTC m=+0.017979461 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-16T11:48:01.034 INFO:journalctl@ceph.mds.l.smithi154.stdout:Mar 16 11:48:00 smithi154 podman[66094]: 2024-03-16 11:48:00.802214628 +0000 UTC m=+0.149486809 container create 5a6f0df597bee85538643390a4f3f7a6ce3e33e3a2112a5754295e1032b0d42d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d1887f28-e389-11ee-95c9-87774f69a715-mds-l, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307) 2024-03-16T11:48:01.034 INFO:journalctl@ceph.mds.l.smithi154.stdout:Mar 16 11:48:01 smithi154 podman[66094]: 2024-03-16 11:48:01.035969919 +0000 UTC m=+0.383242122 container init 5a6f0df597bee85538643390a4f3f7a6ce3e33e3a2112a5754295e1032b0d42d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d1887f28-e389-11ee-95c9-87774f69a715-mds-l, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5) 2024-03-16T11:48:01.295 INFO:journalctl@ceph.mds.l.smithi154.stdout:Mar 16 11:48:01 smithi154 podman[66094]: 2024-03-16 11:48:01.038749997 +0000 UTC m=+0.386022192 container start 5a6f0df597bee85538643390a4f3f7a6ce3e33e3a2112a5754295e1032b0d42d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d1887f28-e389-11ee-95c9-87774f69a715-mds-l, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-03-16T11:48:01.295 INFO:journalctl@ceph.mds.l.smithi154.stdout:Mar 16 11:48:01 smithi154 ceph-mds[66113]: -- 172.21.15.154:0/4270251112 <== mon.1 v2:172.21.15.154:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55a8c799da40 con 0x55a8c7971000 2024-03-16T11:48:01.295 INFO:journalctl@ceph.mds.l.smithi154.stdout:Mar 16 11:48:01 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mds-l[66109]: starting mds.l at 2024-03-16T11:48:01.295 INFO:journalctl@ceph.mds.l.smithi154.stdout:Mar 16 11:48:01 smithi154 bash[66094]: 5a6f0df597bee85538643390a4f3f7a6ce3e33e3a2112a5754295e1032b0d42d 2024-03-16T11:48:01.295 INFO:journalctl@ceph.mds.l.smithi154.stdout:Mar 16 11:48:01 smithi154 systemd[1]: Started Ceph mds.l for d1887f28-e389-11ee-95c9-87774f69a715. 2024-03-16T11:48:01.622 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T11:48:01.739 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:01.737+0000 7f99af768640 1 -- 172.21.15.29:0/3821050512 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f99a81033a0 msgr2=0x7f99a8109c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:48:01.739 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:01.737+0000 7f99af768640 1 --2- 172.21.15.29:0/3821050512 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f99a81033a0 0x7f99a8109c30 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f9998009ca0 tx=0x7f999802c0f0 comp rx=0 tx=0).stop 2024-03-16T11:48:01.739 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:01.737+0000 7f99af768640 1 -- 172.21.15.29:0/3821050512 shutdown_connections 2024-03-16T11:48:01.739 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:01.737+0000 7f99af768640 1 --2- 172.21.15.29:0/3821050512 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f99a81033a0 0x7f99a8109c30 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:01.739 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:01.737+0000 7f99af768640 1 --2- 172.21.15.29:0/3821050512 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f99a81029e0 0x7f99a8102e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:01.739 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:01.737+0000 7f99af768640 1 --2- 172.21.15.29:0/3821050512 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f99a81017f0 0x7f99a8101c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:01.740 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:01.737+0000 7f99af768640 1 -- 172.21.15.29:0/3821050512 >> 172.21.15.29:0/3821050512 conn(0x7f99a80fcdc0 msgr2=0x7f99a80ff1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:48:01.740 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:01.737+0000 7f99af768640 1 -- 172.21.15.29:0/3821050512 shutdown_connections 2024-03-16T11:48:01.740 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:01.737+0000 7f99af768640 1 -- 172.21.15.29:0/3821050512 wait complete. 2024-03-16T11:48:01.740 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:01.737+0000 7f99af768640 1 Processor -- start 2024-03-16T11:48:01.740 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:01.737+0000 7f99af768640 1 -- start start 2024-03-16T11:48:01.740 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:01.738+0000 7f99af768640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f99a81017f0 0x7f99a8197ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:48:01.740 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:01.738+0000 7f99af768640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f99a81029e0 0x7f99a8198000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:48:01.740 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:01.738+0000 7f99af768640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f99a81033a0 0x7f99a819f070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:48:01.740 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:01.738+0000 7f99af768640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f99a810d450 con 0x7f99a81017f0 2024-03-16T11:48:01.740 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:01.738+0000 7f99af768640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f99a810d2d0 con 0x7f99a81033a0 2024-03-16T11:48:01.740 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:01.738+0000 7f99af768640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f99a810d5d0 con 0x7f99a81029e0 2024-03-16T11:48:01.740 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:01.738+0000 7f99ad504640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f99a81017f0 0x7f99a8197ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:48:01.740 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:01.738+0000 7f99ad504640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f99a81017f0 0x7f99a8197ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:38320/0 (socket says 172.21.15.29:38320) 2024-03-16T11:48:01.740 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:01.738+0000 7f99ad504640 1 -- 172.21.15.29:0/2096890464 learned_addr learned my addr 172.21.15.29:0/2096890464 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:48:01.741 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:01.738+0000 7f99acd03640 1 --2- 172.21.15.29:0/2096890464 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f99a81029e0 0x7f99a8198000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:48:01.741 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:01.738+0000 7f99add05640 1 --2- 172.21.15.29:0/2096890464 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f99a81033a0 0x7f99a819f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:48:01.741 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:01.738+0000 7f99ad504640 1 -- 172.21.15.29:0/2096890464 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f99a81029e0 msgr2=0x7f99a8198000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:48:01.741 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:01.738+0000 7f99ad504640 1 --2- 172.21.15.29:0/2096890464 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f99a81029e0 0x7f99a8198000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:01.741 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:01.738+0000 7f99ad504640 1 -- 172.21.15.29:0/2096890464 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f99a81033a0 msgr2=0x7f99a819f070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:48:01.741 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:01.738+0000 7f99ad504640 1 --2- 172.21.15.29:0/2096890464 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f99a81033a0 0x7f99a819f070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:01.741 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:01.738+0000 7f99ad504640 1 -- 172.21.15.29:0/2096890464 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99a819f6e0 con 0x7f99a81017f0 2024-03-16T11:48:01.741 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:01.738+0000 7f99ad504640 1 --2- 172.21.15.29:0/2096890464 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f99a81017f0 0x7f99a8197ac0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f99a000c970 tx=0x7f99a000ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:48:01.741 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:01.738+0000 7f99a67fc640 1 -- 172.21.15.29:0/2096890464 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99a0004220 con 0x7f99a81017f0 2024-03-16T11:48:01.741 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:01.738+0000 7f99a67fc640 1 -- 172.21.15.29:0/2096890464 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f99a00043c0 con 0x7f99a81017f0 2024-03-16T11:48:01.741 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:01.738+0000 7f99af768640 1 -- 172.21.15.29:0/2096890464 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f99a819f9d0 con 0x7f99a81017f0 2024-03-16T11:48:01.741 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:01.738+0000 7f99a67fc640 1 -- 172.21.15.29:0/2096890464 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99a000f450 con 0x7f99a81017f0 2024-03-16T11:48:01.741 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:01.738+0000 7f99af768640 1 -- 172.21.15.29:0/2096890464 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f99a819ff70 con 0x7f99a81017f0 2024-03-16T11:48:01.742 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:01.739+0000 7f99af768640 1 -- 172.21.15.29:0/2096890464 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f99a8108b10 con 0x7f99a81017f0 2024-03-16T11:48:01.742 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:01.740+0000 7f99a67fc640 1 -- 172.21.15.29:0/2096890464 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f99a00079b0 con 0x7f99a81017f0 2024-03-16T11:48:01.743 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:01.741+0000 7f99a67fc640 1 --2- 172.21.15.29:0/2096890464 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f99900a2880 0x7f99900a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:48:01.743 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:01.741+0000 7f99acd03640 1 --2- 172.21.15.29:0/2096890464 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f99900a2880 0x7f99900a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:48:01.743 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:01.741+0000 7f99a67fc640 1 -- 172.21.15.29:0/2096890464 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f99a00ce1f0 con 0x7f99a81017f0 2024-03-16T11:48:01.743 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:01.742+0000 7f99acd03640 1 --2- 172.21.15.29:0/2096890464 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f99900a2880 0x7f99900a4d40 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f99a8198fd0 tx=0x7f999c00b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:48:01.746 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:01.746+0000 7f99a67fc640 1 -- 172.21.15.29:0/2096890464 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f99a0014030 con 0x7f99a81017f0 2024-03-16T11:48:01.968 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:01.968+0000 7f99af768640 1 -- 172.21.15.29:0/2096890464 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f99a802d1b0 con 0x7f99a81017f0 2024-03-16T11:48:01.968 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:01.968+0000 7f99a67fc640 1 -- 172.21.15.29:0/2096890464 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) ==== 93+0+1396 (secure 0 0 0) 0x7f99a802d1b0 con 0x7f99a81017f0 2024-03-16T11:48:01.968 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-16T11:48:01.969 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24421,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6833/2677426282","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":2677426282},{"type":"v1","addr":"172.21.15.154:6833","nonce":2677426282}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-03-16T11:48:01.969 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 2 2024-03-16T11:48:01.970 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:01.970+0000 7f99af768640 1 -- 172.21.15.29:0/2096890464 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f99900a2880 msgr2=0x7f99900a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:48:01.970 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:01.970+0000 7f99af768640 1 --2- 172.21.15.29:0/2096890464 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f99900a2880 0x7f99900a4d40 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f99a8198fd0 tx=0x7f999c00b390 comp rx=0 tx=0).stop 2024-03-16T11:48:01.970 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:01.970+0000 7f99af768640 1 -- 172.21.15.29:0/2096890464 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f99a81017f0 msgr2=0x7f99a8197ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:48:01.970 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:01.970+0000 7f99af768640 1 --2- 172.21.15.29:0/2096890464 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f99a81017f0 0x7f99a8197ac0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f99a000c970 tx=0x7f99a000ce30 comp rx=0 tx=0).stop 2024-03-16T11:48:01.971 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:01.970+0000 7f99af768640 1 -- 172.21.15.29:0/2096890464 shutdown_connections 2024-03-16T11:48:01.971 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:01.970+0000 7f99af768640 1 --2- 172.21.15.29:0/2096890464 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f99900a2880 0x7f99900a4d40 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:01.971 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:01.970+0000 7f99af768640 1 --2- 172.21.15.29:0/2096890464 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f99a81033a0 0x7f99a819f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:01.971 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:01.970+0000 7f99af768640 1 --2- 172.21.15.29:0/2096890464 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f99a81029e0 0x7f99a8198000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:01.971 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:01.970+0000 7f99af768640 1 --2- 172.21.15.29:0/2096890464 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f99a81017f0 0x7f99a8197ac0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:01.971 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:01.970+0000 7f99af768640 1 -- 172.21.15.29:0/2096890464 >> 172.21.15.29:0/2096890464 conn(0x7f99a80fcdc0 msgr2=0x7f99a8105ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:48:01.971 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:01.970+0000 7f99af768640 1 -- 172.21.15.29:0/2096890464 shutdown_connections 2024-03-16T11:48:01.971 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:01.970+0000 7f99af768640 1 -- 172.21.15.29:0/2096890464 wait complete. 2024-03-16T11:48:02.332 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:02 smithi152 ceph-mon[36291]: mds.? [v2:172.21.15.154:6832/2677426282,v1:172.21.15.154:6833/2677426282] up:boot 2024-03-16T11:48:02.332 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:02 smithi152 ceph-mon[36291]: fsmap 1 up:standby 2024-03-16T11:48:02.332 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:02 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-16T11:48:02.332 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:02 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:48:02.332 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:02 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:48:02.332 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:02 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:48:02.332 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:02 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-16T11:48:02.332 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:02 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-16T11:48:02.332 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:02 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:48:02.332 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:02 smithi152 ceph-mon[36291]: Deploying daemon mds.k on smithi152 2024-03-16T11:48:02.332 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:02 smithi152 ceph-mon[36291]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T11:48:02.332 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:02 smithi152 ceph-mon[36291]: from='client.? 172.21.15.29:0/2096890464' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T11:48:02.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:02 smithi029 ceph-mon[31975]: mds.? [v2:172.21.15.154:6832/2677426282,v1:172.21.15.154:6833/2677426282] up:boot 2024-03-16T11:48:02.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:02 smithi029 ceph-mon[31975]: fsmap 1 up:standby 2024-03-16T11:48:02.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:02 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-16T11:48:02.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:02 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:48:02.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:02 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:48:02.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:02 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:48:02.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:02 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-16T11:48:02.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:02 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-16T11:48:02.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:02 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:48:02.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:02 smithi029 ceph-mon[31975]: Deploying daemon mds.k on smithi152 2024-03-16T11:48:02.339 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:02 smithi029 ceph-mon[31975]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T11:48:02.339 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:02 smithi029 ceph-mon[31975]: from='client.? 172.21.15.29:0/2096890464' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T11:48:02.545 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:02 smithi154 ceph-mon[36320]: mds.? [v2:172.21.15.154:6832/2677426282,v1:172.21.15.154:6833/2677426282] up:boot 2024-03-16T11:48:02.545 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:02 smithi154 ceph-mon[36320]: fsmap 1 up:standby 2024-03-16T11:48:02.545 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:02 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-16T11:48:02.545 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:02 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:48:02.545 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:02 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:48:02.545 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:02 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:48:02.546 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:02 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-16T11:48:02.546 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:02 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-16T11:48:02.546 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:02 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:48:02.546 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:02 smithi154 ceph-mon[36320]: Deploying daemon mds.k on smithi152 2024-03-16T11:48:02.546 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:02 smithi154 ceph-mon[36320]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T11:48:02.546 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:02 smithi154 ceph-mon[36320]: from='client.? 172.21.15.29:0/2096890464' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T11:48:03.874 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:03 smithi152 ceph-mon[36291]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T11:48:03.982 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T11:48:04.010 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:03 smithi029 ceph-mon[31975]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T11:48:04.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:03 smithi154 ceph-mon[36320]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T11:48:04.097 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:04.097+0000 7ff06f145640 1 -- 172.21.15.29:0/3374995821 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff068100700 msgr2=0x7ff068100b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:48:04.097 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:04.097+0000 7ff06f145640 1 --2- 172.21.15.29:0/3374995821 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff068100700 0x7ff068100b20 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7ff060009ca0 tx=0x7ff06002c0f0 comp rx=0 tx=0).stop 2024-03-16T11:48:04.097 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:04.098+0000 7ff06f145640 1 -- 172.21.15.29:0/3374995821 shutdown_connections 2024-03-16T11:48:04.097 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:04.098+0000 7ff06f145640 1 --2- 172.21.15.29:0/3374995821 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff0681022b0 0x7ff068108b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:04.097 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:04.098+0000 7ff06f145640 1 --2- 172.21.15.29:0/3374995821 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff0681018f0 0x7ff068101d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:04.097 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:04.098+0000 7ff06f145640 1 --2- 172.21.15.29:0/3374995821 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff068100700 0x7ff068100b20 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:04.097 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:04.098+0000 7ff06f145640 1 -- 172.21.15.29:0/3374995821 >> 172.21.15.29:0/3374995821 conn(0x7ff0680fbc70 msgr2=0x7ff0680fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:48:04.097 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:04.098+0000 7ff06f145640 1 -- 172.21.15.29:0/3374995821 shutdown_connections 2024-03-16T11:48:04.097 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:04.098+0000 7ff06f145640 1 -- 172.21.15.29:0/3374995821 wait complete. 2024-03-16T11:48:04.098 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:04.098+0000 7ff06f145640 1 Processor -- start 2024-03-16T11:48:04.098 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:04.098+0000 7ff06f145640 1 -- start start 2024-03-16T11:48:04.100 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:04.098+0000 7ff06f145640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff068100700 0x7ff068104690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:48:04.100 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:04.098+0000 7ff06f145640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff0681018f0 0x7ff068102cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:48:04.100 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:04.098+0000 7ff06f145640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff0681022b0 0x7ff068103210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:48:04.100 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:04.098+0000 7ff06f145640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7ff06810c370 con 0x7ff0681018f0 2024-03-16T11:48:04.100 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:04.098+0000 7ff06f145640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7ff06810c1f0 con 0x7ff068100700 2024-03-16T11:48:04.100 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:04.098+0000 7ff06f145640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7ff06810c4f0 con 0x7ff0681022b0 2024-03-16T11:48:04.100 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:04.098+0000 7ff067fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff0681018f0 0x7ff068102cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:48:04.100 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:04.099+0000 7ff067fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff0681018f0 0x7ff068102cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:38334/0 (socket says 172.21.15.29:38334) 2024-03-16T11:48:04.100 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:04.099+0000 7ff067fff640 1 -- 172.21.15.29:0/3578420343 learned_addr learned my addr 172.21.15.29:0/3578420343 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:48:04.100 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:04.099+0000 7ff06cee1640 1 --2- 172.21.15.29:0/3578420343 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff068100700 0x7ff068104690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:48:04.100 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:04.099+0000 7ff06d6e2640 1 --2- 172.21.15.29:0/3578420343 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff0681022b0 0x7ff068103210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:48:04.100 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:04.099+0000 7ff067fff640 1 -- 172.21.15.29:0/3578420343 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff0681022b0 msgr2=0x7ff068103210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:48:04.100 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:04.099+0000 7ff067fff640 1 --2- 172.21.15.29:0/3578420343 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff0681022b0 0x7ff068103210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:04.101 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:04.099+0000 7ff067fff640 1 -- 172.21.15.29:0/3578420343 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff068100700 msgr2=0x7ff068104690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:48:04.101 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:04.099+0000 7ff067fff640 1 --2- 172.21.15.29:0/3578420343 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff068100700 0x7ff068104690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:04.101 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:04.099+0000 7ff067fff640 1 -- 172.21.15.29:0/3578420343 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff068103aa0 con 0x7ff0681018f0 2024-03-16T11:48:04.101 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:04.099+0000 7ff06cee1640 1 --2- 172.21.15.29:0/3578420343 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff068100700 0x7ff068104690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T11:48:04.101 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:04.099+0000 7ff06d6e2640 1 --2- 172.21.15.29:0/3578420343 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff0681022b0 0x7ff068103210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-16T11:48:04.101 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:04.099+0000 7ff067fff640 1 --2- 172.21.15.29:0/3578420343 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff0681018f0 0x7ff068102cd0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7ff05c00d770 tx=0x7ff05c00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:48:04.101 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:04.099+0000 7ff065ffb640 1 -- 172.21.15.29:0/3578420343 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff05c017070 con 0x7ff0681018f0 2024-03-16T11:48:04.101 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:04.099+0000 7ff06f145640 1 -- 172.21.15.29:0/3578420343 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff0681a1c70 con 0x7ff0681018f0 2024-03-16T11:48:04.101 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:04.099+0000 7ff065ffb640 1 -- 172.21.15.29:0/3578420343 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff05c005500 con 0x7ff0681018f0 2024-03-16T11:48:04.101 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:04.099+0000 7ff065ffb640 1 -- 172.21.15.29:0/3578420343 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff05c004e00 con 0x7ff0681018f0 2024-03-16T11:48:04.101 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:04.099+0000 7ff06f145640 1 -- 172.21.15.29:0/3578420343 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff0681a21c0 con 0x7ff0681018f0 2024-03-16T11:48:04.101 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:04.100+0000 7ff06f145640 1 -- 172.21.15.29:0/3578420343 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff0681a2390 con 0x7ff0681018f0 2024-03-16T11:48:04.102 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:04.101+0000 7ff065ffb640 1 -- 172.21.15.29:0/3578420343 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7ff05c005000 con 0x7ff0681018f0 2024-03-16T11:48:04.102 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:04.102+0000 7ff065ffb640 1 --2- 172.21.15.29:0/3578420343 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7ff0500a2800 0x7ff0500a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:48:04.102 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:04.102+0000 7ff06cee1640 1 --2- 172.21.15.29:0/3578420343 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7ff0500a2800 0x7ff0500a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:48:04.102 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:04.102+0000 7ff065ffb640 1 -- 172.21.15.29:0/3578420343 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7ff05c0ce0b0 con 0x7ff0681018f0 2024-03-16T11:48:04.102 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:04.103+0000 7ff06cee1640 1 --2- 172.21.15.29:0/3578420343 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7ff0500a2800 0x7ff0500a4cc0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7ff06002c700 tx=0x7ff0600325b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:48:04.106 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:04.107+0000 7ff065ffb640 1 -- 172.21.15.29:0/3578420343 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7ff05c09b5f0 con 0x7ff0681018f0 2024-03-16T11:48:04.128 INFO:journalctl@ceph.mds.k.smithi152.stdout:Mar 16 11:48:04 smithi152 podman[66289]: 2024-03-16 11:48:03.993003838 +0000 UTC m=+0.017752448 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-16T11:48:04.327 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:04.328+0000 7ff06f145640 1 -- 172.21.15.29:0/3578420343 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff0681a2560 con 0x7ff0681018f0 2024-03-16T11:48:04.328 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:04.329+0000 7ff065ffb640 1 -- 172.21.15.29:0/3578420343 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) ==== 93+0+1396 (secure 0 0 0) 0x7ff05c09aef0 con 0x7ff0681018f0 2024-03-16T11:48:04.328 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-16T11:48:04.328 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24421,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6833/2677426282","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":2677426282},{"type":"v1","addr":"172.21.15.154:6833","nonce":2677426282}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-03-16T11:48:04.329 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 2 2024-03-16T11:48:04.330 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:04.331+0000 7ff06f145640 1 -- 172.21.15.29:0/3578420343 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7ff0500a2800 msgr2=0x7ff0500a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:48:04.331 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:04.331+0000 7ff06f145640 1 --2- 172.21.15.29:0/3578420343 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7ff0500a2800 0x7ff0500a4cc0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7ff06002c700 tx=0x7ff0600325b0 comp rx=0 tx=0).stop 2024-03-16T11:48:04.331 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:04.331+0000 7ff06f145640 1 -- 172.21.15.29:0/3578420343 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff0681018f0 msgr2=0x7ff068102cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:48:04.331 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:04.331+0000 7ff06f145640 1 --2- 172.21.15.29:0/3578420343 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff0681018f0 0x7ff068102cd0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7ff05c00d770 tx=0x7ff05c00dc30 comp rx=0 tx=0).stop 2024-03-16T11:48:04.331 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:04.331+0000 7ff06f145640 1 -- 172.21.15.29:0/3578420343 shutdown_connections 2024-03-16T11:48:04.331 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:04.331+0000 7ff06f145640 1 --2- 172.21.15.29:0/3578420343 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7ff0500a2800 0x7ff0500a4cc0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:04.331 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:04.331+0000 7ff06f145640 1 --2- 172.21.15.29:0/3578420343 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff0681022b0 0x7ff068103210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:04.331 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:04.331+0000 7ff06f145640 1 --2- 172.21.15.29:0/3578420343 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff0681018f0 0x7ff068102cd0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:04.331 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:04.331+0000 7ff06f145640 1 --2- 172.21.15.29:0/3578420343 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff068100700 0x7ff068104690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:04.331 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:04.331+0000 7ff06f145640 1 -- 172.21.15.29:0/3578420343 >> 172.21.15.29:0/3578420343 conn(0x7ff0680fbc70 msgr2=0x7ff0680fd750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:48:04.331 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:04.331+0000 7ff06f145640 1 -- 172.21.15.29:0/3578420343 shutdown_connections 2024-03-16T11:48:04.331 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:04.331+0000 7ff06f145640 1 -- 172.21.15.29:0/3578420343 wait complete. 2024-03-16T11:48:04.407 INFO:journalctl@ceph.mds.k.smithi152.stdout:Mar 16 11:48:04 smithi152 podman[66289]: 2024-03-16 11:48:04.13807058 +0000 UTC m=+0.162819206 container create a35e36fe4008d594250a7847588998bc2d7b8f212639a86a45552c604d974a38 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d1887f28-e389-11ee-95c9-87774f69a715-mds-k, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.build-date=20240307) 2024-03-16T11:48:04.407 INFO:journalctl@ceph.mds.k.smithi152.stdout:Mar 16 11:48:04 smithi152 podman[66289]: 2024-03-16 11:48:04.38296571 +0000 UTC m=+0.407714320 container init a35e36fe4008d594250a7847588998bc2d7b8f212639a86a45552c604d974a38 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d1887f28-e389-11ee-95c9-87774f69a715-mds-k, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-16T11:48:04.407 INFO:journalctl@ceph.mds.k.smithi152.stdout:Mar 16 11:48:04 smithi152 podman[66289]: 2024-03-16 11:48:04.385506753 +0000 UTC m=+0.410255354 container start a35e36fe4008d594250a7847588998bc2d7b8f212639a86a45552c604d974a38 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d1887f28-e389-11ee-95c9-87774f69a715-mds-k, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-03-16T11:48:04.682 INFO:journalctl@ceph.mds.k.smithi152.stdout:Mar 16 11:48:04 smithi152 ceph-mds[66309]: -- 172.21.15.152:0/4084013274 <== mon.2 v2:172.21.15.152:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55fd9cab1a40 con 0x55fd9ca85000 2024-03-16T11:48:04.682 INFO:journalctl@ceph.mds.k.smithi152.stdout:Mar 16 11:48:04 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mds-k[66305]: starting mds.k at 2024-03-16T11:48:04.682 INFO:journalctl@ceph.mds.k.smithi152.stdout:Mar 16 11:48:04 smithi152 bash[66289]: a35e36fe4008d594250a7847588998bc2d7b8f212639a86a45552c604d974a38 2024-03-16T11:48:04.682 INFO:journalctl@ceph.mds.k.smithi152.stdout:Mar 16 11:48:04 smithi152 systemd[1]: Started Ceph mds.k for d1887f28-e389-11ee-95c9-87774f69a715. 2024-03-16T11:48:05.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:04 smithi154 ceph-mon[36320]: from='client.? 172.21.15.29:0/3578420343' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T11:48:05.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:04 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:48:05.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:04 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:48:05.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:04 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:48:05.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:04 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-16T11:48:05.046 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:04 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-16T11:48:05.046 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:04 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:48:05.046 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:04 smithi154 ceph-mon[36320]: Deploying daemon mds.j on smithi029 2024-03-16T11:48:05.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:04 smithi029 ceph-mon[31975]: from='client.? 172.21.15.29:0/3578420343' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T11:48:05.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:04 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:48:05.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:04 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:48:05.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:04 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:48:05.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:04 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-16T11:48:05.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:04 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-16T11:48:05.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:04 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:48:05.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:04 smithi029 ceph-mon[31975]: Deploying daemon mds.j on smithi029 2024-03-16T11:48:05.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:04 smithi152 ceph-mon[36291]: from='client.? 172.21.15.29:0/3578420343' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T11:48:05.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:04 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:48:05.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:04 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:48:05.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:04 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:48:05.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:04 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-16T11:48:05.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:04 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-16T11:48:05.128 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:04 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:48:05.128 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:04 smithi152 ceph-mon[36291]: Deploying daemon mds.j on smithi029 2024-03-16T11:48:05.803 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:05 smithi152 ceph-mon[36291]: mds.? [v2:172.21.15.152:6832/3136024628,v1:172.21.15.152:6833/3136024628] up:boot 2024-03-16T11:48:05.803 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:05 smithi152 ceph-mon[36291]: fsmap 2 up:standby 2024-03-16T11:48:05.803 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:05 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-16T11:48:05.804 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:05 smithi152 ceph-mon[36291]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T11:48:06.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:05 smithi154 ceph-mon[36320]: mds.? [v2:172.21.15.152:6832/3136024628,v1:172.21.15.152:6833/3136024628] up:boot 2024-03-16T11:48:06.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:05 smithi154 ceph-mon[36320]: fsmap 2 up:standby 2024-03-16T11:48:06.046 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:05 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-16T11:48:06.046 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:05 smithi154 ceph-mon[36320]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T11:48:06.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:05 smithi029 ceph-mon[31975]: mds.? [v2:172.21.15.152:6832/3136024628,v1:172.21.15.152:6833/3136024628] up:boot 2024-03-16T11:48:06.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:05 smithi029 ceph-mon[31975]: fsmap 2 up:standby 2024-03-16T11:48:06.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:05 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-16T11:48:06.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:05 smithi029 ceph-mon[31975]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T11:48:06.342 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T11:48:06.459 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:06.460+0000 7f1e51e3f640 1 -- 172.21.15.29:0/2134817600 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1e4c102a00 msgr2=0x7f1e4c102e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:48:06.459 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:06.460+0000 7f1e51e3f640 1 --2- 172.21.15.29:0/2134817600 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1e4c102a00 0x7f1e4c102e80 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f1e40009900 tx=0x7f1e4002bc30 comp rx=0 tx=0).stop 2024-03-16T11:48:06.459 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:06.460+0000 7f1e51e3f640 1 -- 172.21.15.29:0/2134817600 shutdown_connections 2024-03-16T11:48:06.459 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:06.460+0000 7f1e51e3f640 1 --2- 172.21.15.29:0/2134817600 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1e4c1033c0 0x7f1e4c109c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:06.459 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:06.460+0000 7f1e51e3f640 1 --2- 172.21.15.29:0/2134817600 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1e4c102a00 0x7f1e4c102e80 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:06.460 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:06.460+0000 7f1e51e3f640 1 --2- 172.21.15.29:0/2134817600 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1e4c101810 0x7f1e4c101c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:06.460 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:06.460+0000 7f1e51e3f640 1 -- 172.21.15.29:0/2134817600 >> 172.21.15.29:0/2134817600 conn(0x7f1e4c0fcd80 msgr2=0x7f1e4c0ff1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:48:06.460 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:06.460+0000 7f1e51e3f640 1 -- 172.21.15.29:0/2134817600 shutdown_connections 2024-03-16T11:48:06.460 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:06.460+0000 7f1e51e3f640 1 -- 172.21.15.29:0/2134817600 wait complete. 2024-03-16T11:48:06.460 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:06.460+0000 7f1e51e3f640 1 Processor -- start 2024-03-16T11:48:06.462 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:06.461+0000 7f1e51e3f640 1 -- start start 2024-03-16T11:48:06.462 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:06.461+0000 7f1e51e3f640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1e4c101810 0x7f1e4c197b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:48:06.462 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:06.461+0000 7f1e51e3f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1e4c102a00 0x7f1e4c1980a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:48:06.462 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:06.461+0000 7f1e51e3f640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1e4c1033c0 0x7f1e4c19f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:48:06.462 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:06.461+0000 7f1e51e3f640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f1e4c10d490 con 0x7f1e4c102a00 2024-03-16T11:48:06.462 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:06.461+0000 7f1e51e3f640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f1e4c10d310 con 0x7f1e4c101810 2024-03-16T11:48:06.462 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:06.461+0000 7f1e51e3f640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f1e4c10d610 con 0x7f1e4c1033c0 2024-03-16T11:48:06.462 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:06.461+0000 7f1e4affd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1e4c102a00 0x7f1e4c1980a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:48:06.463 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:06.461+0000 7f1e4affd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1e4c102a00 0x7f1e4c1980a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:38346/0 (socket says 172.21.15.29:38346) 2024-03-16T11:48:06.463 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:06.461+0000 7f1e4affd640 1 -- 172.21.15.29:0/2006337890 learned_addr learned my addr 172.21.15.29:0/2006337890 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:48:06.463 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:06.461+0000 7f1e4b7fe640 1 --2- 172.21.15.29:0/2006337890 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1e4c101810 0x7f1e4c197b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:48:06.463 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:06.461+0000 7f1e4bfff640 1 --2- 172.21.15.29:0/2006337890 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1e4c1033c0 0x7f1e4c19f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:48:06.463 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:06.461+0000 7f1e4affd640 1 -- 172.21.15.29:0/2006337890 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1e4c1033c0 msgr2=0x7f1e4c19f110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:48:06.463 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:06.461+0000 7f1e4affd640 1 --2- 172.21.15.29:0/2006337890 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1e4c1033c0 0x7f1e4c19f110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:06.463 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:06.461+0000 7f1e4affd640 1 -- 172.21.15.29:0/2006337890 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1e4c101810 msgr2=0x7f1e4c197b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:48:06.463 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:06.461+0000 7f1e4affd640 1 --2- 172.21.15.29:0/2006337890 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1e4c101810 0x7f1e4c197b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:06.463 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:06.461+0000 7f1e4affd640 1 -- 172.21.15.29:0/2006337890 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e4c19f780 con 0x7f1e4c102a00 2024-03-16T11:48:06.463 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:06.461+0000 7f1e4bfff640 1 --2- 172.21.15.29:0/2006337890 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1e4c1033c0 0x7f1e4c19f110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T11:48:06.463 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:06.461+0000 7f1e4b7fe640 1 --2- 172.21.15.29:0/2006337890 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1e4c101810 0x7f1e4c197b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:48:06.463 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:06.461+0000 7f1e4affd640 1 --2- 172.21.15.29:0/2006337890 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1e4c102a00 0x7f1e4c1980a0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f1e40009230 tx=0x7f1e40005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:48:06.463 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:06.461+0000 7f1e48ff9640 1 -- 172.21.15.29:0/2006337890 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e40041070 con 0x7f1e4c102a00 2024-03-16T11:48:06.463 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:06.462+0000 7f1e48ff9640 1 -- 172.21.15.29:0/2006337890 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1e400044d0 con 0x7f1e4c102a00 2024-03-16T11:48:06.463 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:06.462+0000 7f1e51e3f640 1 -- 172.21.15.29:0/2006337890 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1e4c19fa10 con 0x7f1e4c102a00 2024-03-16T11:48:06.464 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:06.462+0000 7f1e48ff9640 1 -- 172.21.15.29:0/2006337890 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e40034440 con 0x7f1e4c102a00 2024-03-16T11:48:06.464 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:06.462+0000 7f1e51e3f640 1 -- 172.21.15.29:0/2006337890 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1e4c19ff50 con 0x7f1e4c102a00 2024-03-16T11:48:06.464 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:06.463+0000 7f1e51e3f640 1 -- 172.21.15.29:0/2006337890 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1e4c108bd0 con 0x7f1e4c102a00 2024-03-16T11:48:06.464 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:06.464+0000 7f1e48ff9640 1 -- 172.21.15.29:0/2006337890 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f1e40044070 con 0x7f1e4c102a00 2024-03-16T11:48:06.464 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:06.464+0000 7f1e48ff9640 1 --2- 172.21.15.29:0/2006337890 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f1e340a2880 0x7f1e340a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:48:06.464 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:06.464+0000 7f1e4b7fe640 1 --2- 172.21.15.29:0/2006337890 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f1e340a2880 0x7f1e340a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:48:06.464 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:06.464+0000 7f1e48ff9640 1 -- 172.21.15.29:0/2006337890 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f1e400f01d0 con 0x7f1e4c102a00 2024-03-16T11:48:06.465 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:06.465+0000 7f1e4b7fe640 1 --2- 172.21.15.29:0/2006337890 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f1e340a2880 0x7f1e340a4d40 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f1e3c005fd0 tx=0x7f1e3c00b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:48:06.468 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:06.469+0000 7f1e48ff9640 1 -- 172.21.15.29:0/2006337890 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f1e400bd710 con 0x7f1e4c102a00 2024-03-16T11:48:06.699 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:06.700+0000 7f1e51e3f640 1 -- 172.21.15.29:0/2006337890 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1e4c02d400 con 0x7f1e4c102a00 2024-03-16T11:48:06.700 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:06.701+0000 7f1e48ff9640 1 -- 172.21.15.29:0/2006337890 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) ==== 93+0+2224 (secure 0 0 0) 0x7f1e400bd010 con 0x7f1e4c102a00 2024-03-16T11:48:06.700 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-16T11:48:06.700 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24407,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6833/3136024628","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6832","nonce":3136024628},{"type":"v1","addr":"172.21.15.152:6833","nonce":3136024628}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24421,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6833/2677426282","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":2677426282},{"type":"v1","addr":"172.21.15.154:6833","nonce":2677426282}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-03-16T11:48:06.700 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 3 2024-03-16T11:48:06.702 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:06.703+0000 7f1e51e3f640 1 -- 172.21.15.29:0/2006337890 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f1e340a2880 msgr2=0x7f1e340a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:48:06.702 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:06.703+0000 7f1e51e3f640 1 --2- 172.21.15.29:0/2006337890 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f1e340a2880 0x7f1e340a4d40 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f1e3c005fd0 tx=0x7f1e3c00b390 comp rx=0 tx=0).stop 2024-03-16T11:48:06.702 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:06.703+0000 7f1e51e3f640 1 -- 172.21.15.29:0/2006337890 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1e4c102a00 msgr2=0x7f1e4c1980a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:48:06.703 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:06.703+0000 7f1e51e3f640 1 --2- 172.21.15.29:0/2006337890 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1e4c102a00 0x7f1e4c1980a0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f1e40009230 tx=0x7f1e40005d50 comp rx=0 tx=0).stop 2024-03-16T11:48:06.703 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:06.703+0000 7f1e51e3f640 1 -- 172.21.15.29:0/2006337890 shutdown_connections 2024-03-16T11:48:06.703 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:06.703+0000 7f1e51e3f640 1 --2- 172.21.15.29:0/2006337890 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f1e340a2880 0x7f1e340a4d40 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:06.703 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:06.703+0000 7f1e51e3f640 1 --2- 172.21.15.29:0/2006337890 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1e4c1033c0 0x7f1e4c19f110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:06.703 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:06.703+0000 7f1e51e3f640 1 --2- 172.21.15.29:0/2006337890 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1e4c102a00 0x7f1e4c1980a0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:06.703 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:06.703+0000 7f1e51e3f640 1 --2- 172.21.15.29:0/2006337890 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1e4c101810 0x7f1e4c197b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:06.703 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:06.703+0000 7f1e51e3f640 1 -- 172.21.15.29:0/2006337890 >> 172.21.15.29:0/2006337890 conn(0x7f1e4c0fcd80 msgr2=0x7f1e4c105da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:48:06.703 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:06.703+0000 7f1e51e3f640 1 -- 172.21.15.29:0/2006337890 shutdown_connections 2024-03-16T11:48:06.703 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:06.703+0000 7f1e51e3f640 1 -- 172.21.15.29:0/2006337890 wait complete. 2024-03-16T11:48:07.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:06 smithi154 ceph-mon[36320]: from='client.? 172.21.15.29:0/2006337890' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T11:48:07.081 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:06 smithi029 ceph-mon[31975]: from='client.? 172.21.15.29:0/2006337890' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T11:48:07.081 INFO:journalctl@ceph.mds.j.smithi029.stdout:Mar 16 11:48:06 smithi029 systemd[1]: Starting Ceph mds.j for d1887f28-e389-11ee-95c9-87774f69a715... 2024-03-16T11:48:07.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:06 smithi152 ceph-mon[36291]: from='client.? 172.21.15.29:0/2006337890' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T11:48:07.338 INFO:journalctl@ceph.mds.j.smithi029.stdout:Mar 16 11:48:07 smithi029 podman[72855]: 2024-03-16 11:48:06.984983464 +0000 UTC m=+0.017030925 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-16T11:48:07.338 INFO:journalctl@ceph.mds.j.smithi029.stdout:Mar 16 11:48:07 smithi029 podman[72855]: 2024-03-16 11:48:07.151615234 +0000 UTC m=+0.183662692 container create edc59c83e563e2e5f4d7002bd75413a1c46a25d1c3acec0ff93ed46c1504bce1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d1887f28-e389-11ee-95c9-87774f69a715-mds-j, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-03-16T11:48:07.838 INFO:journalctl@ceph.mds.j.smithi029.stdout:Mar 16 11:48:07 smithi029 podman[72855]: 2024-03-16 11:48:07.425401258 +0000 UTC m=+0.457448722 container init edc59c83e563e2e5f4d7002bd75413a1c46a25d1c3acec0ff93ed46c1504bce1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d1887f28-e389-11ee-95c9-87774f69a715-mds-j, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True) 2024-03-16T11:48:07.838 INFO:journalctl@ceph.mds.j.smithi029.stdout:Mar 16 11:48:07 smithi029 podman[72855]: 2024-03-16 11:48:07.42830477 +0000 UTC m=+0.460352233 container start edc59c83e563e2e5f4d7002bd75413a1c46a25d1c3acec0ff93ed46c1504bce1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d1887f28-e389-11ee-95c9-87774f69a715-mds-j, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-03-16T11:48:07.838 INFO:journalctl@ceph.mds.j.smithi029.stdout:Mar 16 11:48:07 smithi029 ceph-mds[72874]: -- 172.21.15.29:0/4171089797 <== mon.0 v2:172.21.15.29:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55dc54ae1a40 con 0x55dc54ab4800 2024-03-16T11:48:07.838 INFO:journalctl@ceph.mds.j.smithi029.stdout:Mar 16 11:48:07 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mds-j[72870]: starting mds.j at 2024-03-16T11:48:07.838 INFO:journalctl@ceph.mds.j.smithi029.stdout:Mar 16 11:48:07 smithi029 bash[72855]: edc59c83e563e2e5f4d7002bd75413a1c46a25d1c3acec0ff93ed46c1504bce1 2024-03-16T11:48:07.838 INFO:journalctl@ceph.mds.j.smithi029.stdout:Mar 16 11:48:07 smithi029 systemd[1]: Started Ceph mds.j for d1887f28-e389-11ee-95c9-87774f69a715. 2024-03-16T11:48:08.715 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T11:48:08.821 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:08.821+0000 7ff3c9e3f640 1 -- 172.21.15.29:0/1756150060 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff3c4101800 msgr2=0x7ff3c4101c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:48:08.821 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:08.821+0000 7ff3c9e3f640 1 --2- 172.21.15.29:0/1756150060 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff3c4101800 0x7ff3c4101c20 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7ff3ac009900 tx=0x7ff3ac02bc30 comp rx=0 tx=0).stop 2024-03-16T11:48:08.821 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:08.821+0000 7ff3c9e3f640 1 -- 172.21.15.29:0/1756150060 shutdown_connections 2024-03-16T11:48:08.821 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:08.821+0000 7ff3c9e3f640 1 --2- 172.21.15.29:0/1756150060 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff3c41033b0 0x7ff3c4109c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:08.821 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:08.821+0000 7ff3c9e3f640 1 --2- 172.21.15.29:0/1756150060 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff3c41029f0 0x7ff3c4102e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:08.821 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:08.821+0000 7ff3c9e3f640 1 --2- 172.21.15.29:0/1756150060 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff3c4101800 0x7ff3c4101c20 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:08.821 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:08.821+0000 7ff3c9e3f640 1 -- 172.21.15.29:0/1756150060 >> 172.21.15.29:0/1756150060 conn(0x7ff3c40fcd90 msgr2=0x7ff3c40ff1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:48:08.821 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:08.821+0000 7ff3c9e3f640 1 -- 172.21.15.29:0/1756150060 shutdown_connections 2024-03-16T11:48:08.821 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:08.821+0000 7ff3c9e3f640 1 -- 172.21.15.29:0/1756150060 wait complete. 2024-03-16T11:48:08.821 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:08.822+0000 7ff3c9e3f640 1 Processor -- start 2024-03-16T11:48:08.824 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:08.822+0000 7ff3c9e3f640 1 -- start start 2024-03-16T11:48:08.824 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:08.822+0000 7ff3c9e3f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff3c4101800 0x7ff3c4197b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:48:08.824 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:08.822+0000 7ff3c9e3f640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff3c41029f0 0x7ff3c4198060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:48:08.824 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:08.822+0000 7ff3c9e3f640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff3c41033b0 0x7ff3c419f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:48:08.824 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:08.822+0000 7ff3c9e3f640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7ff3c410d470 con 0x7ff3c4101800 2024-03-16T11:48:08.824 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:08.822+0000 7ff3c9e3f640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7ff3c410d2f0 con 0x7ff3c41029f0 2024-03-16T11:48:08.824 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:08.822+0000 7ff3c9e3f640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7ff3c410d5f0 con 0x7ff3c41033b0 2024-03-16T11:48:08.824 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:08.822+0000 7ff3c37fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff3c4101800 0x7ff3c4197b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:48:08.825 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:08.822+0000 7ff3c37fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff3c4101800 0x7ff3c4197b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:38372/0 (socket says 172.21.15.29:38372) 2024-03-16T11:48:08.825 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:08.822+0000 7ff3c37fe640 1 -- 172.21.15.29:0/710070672 learned_addr learned my addr 172.21.15.29:0/710070672 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:48:08.825 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:08.823+0000 7ff3c3fff640 1 --2- 172.21.15.29:0/710070672 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff3c41033b0 0x7ff3c419f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:48:08.825 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:08.823+0000 7ff3c2ffd640 1 --2- 172.21.15.29:0/710070672 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff3c41029f0 0x7ff3c4198060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:48:08.825 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:08.823+0000 7ff3c37fe640 1 -- 172.21.15.29:0/710070672 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff3c41033b0 msgr2=0x7ff3c419f0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:48:08.825 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:08.823+0000 7ff3c37fe640 1 --2- 172.21.15.29:0/710070672 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff3c41033b0 0x7ff3c419f0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:08.825 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:08.823+0000 7ff3c37fe640 1 -- 172.21.15.29:0/710070672 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff3c41029f0 msgr2=0x7ff3c4198060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:48:08.825 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:08.823+0000 7ff3c37fe640 1 --2- 172.21.15.29:0/710070672 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff3c41029f0 0x7ff3c4198060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:08.825 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:08.823+0000 7ff3c37fe640 1 -- 172.21.15.29:0/710070672 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3c419f740 con 0x7ff3c4101800 2024-03-16T11:48:08.825 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:08.823+0000 7ff3c3fff640 1 --2- 172.21.15.29:0/710070672 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff3c41033b0 0x7ff3c419f0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T11:48:08.825 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:08.823+0000 7ff3c37fe640 1 --2- 172.21.15.29:0/710070672 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff3c4101800 0x7ff3c4197b20 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7ff3ac005ab0 tx=0x7ff3ac005dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:48:08.825 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:08.823+0000 7ff3c0ff9640 1 -- 172.21.15.29:0/710070672 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3ac041070 con 0x7ff3c4101800 2024-03-16T11:48:08.825 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:08.823+0000 7ff3c9e3f640 1 -- 172.21.15.29:0/710070672 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff3c419f9d0 con 0x7ff3c4101800 2024-03-16T11:48:08.826 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:08.823+0000 7ff3c0ff9640 1 -- 172.21.15.29:0/710070672 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff3ac0054e0 con 0x7ff3c4101800 2024-03-16T11:48:08.826 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:08.823+0000 7ff3c0ff9640 1 -- 172.21.15.29:0/710070672 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3ac034440 con 0x7ff3c4101800 2024-03-16T11:48:08.826 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:08.823+0000 7ff3c9e3f640 1 -- 172.21.15.29:0/710070672 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff3c419fec0 con 0x7ff3c4101800 2024-03-16T11:48:08.826 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:08.824+0000 7ff3c9e3f640 1 -- 172.21.15.29:0/710070672 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff3c4108b40 con 0x7ff3c4101800 2024-03-16T11:48:08.827 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:08.825+0000 7ff3c0ff9640 1 -- 172.21.15.29:0/710070672 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7ff3ac044070 con 0x7ff3c4101800 2024-03-16T11:48:08.827 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:08.826+0000 7ff3c0ff9640 1 --2- 172.21.15.29:0/710070672 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7ff3b00a2800 0x7ff3b00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:48:08.827 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:08.826+0000 7ff3c2ffd640 1 --2- 172.21.15.29:0/710070672 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7ff3b00a2800 0x7ff3b00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:48:08.827 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:08.826+0000 7ff3c0ff9640 1 -- 172.21.15.29:0/710070672 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7ff3ac0ef900 con 0x7ff3c4101800 2024-03-16T11:48:08.827 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:08.826+0000 7ff3c2ffd640 1 --2- 172.21.15.29:0/710070672 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7ff3b00a2800 0x7ff3b00a4cc0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7ff3b4005fd0 tx=0x7ff3b4005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:48:08.829 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:08.830+0000 7ff3c0ff9640 1 -- 172.21.15.29:0/710070672 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7ff3ac0bce40 con 0x7ff3c4101800 2024-03-16T11:48:08.837 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:08 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:48:08.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:08 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:48:08.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:08 smithi029 ceph-mon[31975]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T11:48:08.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:08 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:48:08.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:08 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-16T11:48:08.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:08 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-16T11:48:08.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:08 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:48:08.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:08 smithi029 ceph-mon[31975]: Deploying daemon mds.d on smithi029 2024-03-16T11:48:08.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:08 smithi029 ceph-mon[31975]: mds.? [v2:172.21.15.29:6834/3505811955,v1:172.21.15.29:6835/3505811955] up:boot 2024-03-16T11:48:08.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:08 smithi029 ceph-mon[31975]: fsmap 3 up:standby 2024-03-16T11:48:08.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:08 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-16T11:48:08.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:08 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:48:08.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:08 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:48:08.878 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:08 smithi152 ceph-mon[36291]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T11:48:08.878 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:08 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:48:08.878 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:08 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-16T11:48:08.878 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:08 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-16T11:48:08.878 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:08 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:48:08.878 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:08 smithi152 ceph-mon[36291]: Deploying daemon mds.d on smithi029 2024-03-16T11:48:08.878 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:08 smithi152 ceph-mon[36291]: mds.? [v2:172.21.15.29:6834/3505811955,v1:172.21.15.29:6835/3505811955] up:boot 2024-03-16T11:48:08.878 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:08 smithi152 ceph-mon[36291]: fsmap 3 up:standby 2024-03-16T11:48:08.878 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:08 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-16T11:48:08.961 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:08 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:48:08.961 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:08 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:48:08.961 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:08 smithi154 ceph-mon[36320]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T11:48:08.961 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:08 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:48:08.961 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:08 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-16T11:48:08.961 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:08 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-16T11:48:08.962 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:08 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:48:08.962 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:08 smithi154 ceph-mon[36320]: Deploying daemon mds.d on smithi029 2024-03-16T11:48:08.962 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:08 smithi154 ceph-mon[36320]: mds.? [v2:172.21.15.29:6834/3505811955,v1:172.21.15.29:6835/3505811955] up:boot 2024-03-16T11:48:08.962 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:08 smithi154 ceph-mon[36320]: fsmap 3 up:standby 2024-03-16T11:48:08.962 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:08 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-16T11:48:09.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:09.055+0000 7ff3c9e3f640 1 -- 172.21.15.29:0/710070672 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff3c402d1b0 con 0x7ff3c4101800 2024-03-16T11:48:09.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:09.055+0000 7ff3c0ff9640 1 -- 172.21.15.29:0/710070672 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 4 v4) ==== 93+0+3049 (secure 0 0 0) 0x7ff3ac0bc740 con 0x7ff3c4101800 2024-03-16T11:48:09.055 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-16T11:48:09.055 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":4,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14574,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/3505811955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":3505811955},{"type":"v1","addr":"172.21.15.29:6835","nonce":3505811955}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24407,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6833/3136024628","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6832","nonce":3136024628},{"type":"v1","addr":"172.21.15.152:6833","nonce":3136024628}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24421,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6833/2677426282","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":2677426282},{"type":"v1","addr":"172.21.15.154:6833","nonce":2677426282}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-03-16T11:48:09.055 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 4 2024-03-16T11:48:09.057 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:09.058+0000 7ff3c9e3f640 1 -- 172.21.15.29:0/710070672 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7ff3b00a2800 msgr2=0x7ff3b00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:48:09.057 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:09.058+0000 7ff3c9e3f640 1 --2- 172.21.15.29:0/710070672 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7ff3b00a2800 0x7ff3b00a4cc0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7ff3b4005fd0 tx=0x7ff3b4005850 comp rx=0 tx=0).stop 2024-03-16T11:48:09.057 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:09.058+0000 7ff3c9e3f640 1 -- 172.21.15.29:0/710070672 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff3c4101800 msgr2=0x7ff3c4197b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:48:09.057 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:09.058+0000 7ff3c9e3f640 1 --2- 172.21.15.29:0/710070672 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff3c4101800 0x7ff3c4197b20 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7ff3ac005ab0 tx=0x7ff3ac005dc0 comp rx=0 tx=0).stop 2024-03-16T11:48:09.057 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:09.058+0000 7ff3c9e3f640 1 -- 172.21.15.29:0/710070672 shutdown_connections 2024-03-16T11:48:09.057 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:09.058+0000 7ff3c9e3f640 1 --2- 172.21.15.29:0/710070672 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7ff3b00a2800 0x7ff3b00a4cc0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:09.058 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:09.058+0000 7ff3c9e3f640 1 --2- 172.21.15.29:0/710070672 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff3c41033b0 0x7ff3c419f0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:09.058 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:09.058+0000 7ff3c9e3f640 1 --2- 172.21.15.29:0/710070672 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff3c41029f0 0x7ff3c4198060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:09.058 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:09.058+0000 7ff3c9e3f640 1 --2- 172.21.15.29:0/710070672 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff3c4101800 0x7ff3c4197b20 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:09.058 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:09.058+0000 7ff3c9e3f640 1 -- 172.21.15.29:0/710070672 >> 172.21.15.29:0/710070672 conn(0x7ff3c40fcd90 msgr2=0x7ff3c4105d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:48:09.058 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:09.058+0000 7ff3c9e3f640 1 -- 172.21.15.29:0/710070672 shutdown_connections 2024-03-16T11:48:09.058 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:09.058+0000 7ff3c9e3f640 1 -- 172.21.15.29:0/710070672 wait complete. 2024-03-16T11:48:09.710 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:09 smithi029 ceph-mon[31975]: from='client.? 172.21.15.29:0/710070672' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T11:48:09.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:09 smithi152 ceph-mon[36291]: from='client.? 172.21.15.29:0/710070672' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T11:48:09.961 INFO:journalctl@ceph.mds.d.smithi029.stdout:Mar 16 11:48:09 smithi029 systemd[1]: Starting Ceph mds.d for d1887f28-e389-11ee-95c9-87774f69a715... 2024-03-16T11:48:10.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:09 smithi154 ceph-mon[36320]: from='client.? 172.21.15.29:0/710070672' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T11:48:10.255 INFO:journalctl@ceph.mds.d.smithi029.stdout:Mar 16 11:48:10 smithi029 podman[73206]: 2024-03-16 11:48:10.011713998 +0000 UTC m=+0.017641743 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-16T11:48:10.255 INFO:journalctl@ceph.mds.d.smithi029.stdout:Mar 16 11:48:10 smithi029 podman[73206]: 2024-03-16 11:48:10.132865995 +0000 UTC m=+0.138793773 container create 49650e1408b88f1fd89fd899d4f7ebab3603c9d10466d9f86c075fa5a348975b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d1887f28-e389-11ee-95c9-87774f69a715-mds-d, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5) 2024-03-16T11:48:10.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:10 smithi029 ceph-mon[31975]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T11:48:10.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:10 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:48:10.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:10 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:48:10.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:10 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:48:10.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:10 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-16T11:48:10.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:10 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-16T11:48:10.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:10 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:48:10.839 INFO:journalctl@ceph.mds.d.smithi029.stdout:Mar 16 11:48:10 smithi029 podman[73206]: 2024-03-16 11:48:10.379116036 +0000 UTC m=+0.385043786 container init 49650e1408b88f1fd89fd899d4f7ebab3603c9d10466d9f86c075fa5a348975b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d1887f28-e389-11ee-95c9-87774f69a715-mds-d, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True) 2024-03-16T11:48:10.839 INFO:journalctl@ceph.mds.d.smithi029.stdout:Mar 16 11:48:10 smithi029 podman[73206]: 2024-03-16 11:48:10.381838947 +0000 UTC m=+0.387766684 container start 49650e1408b88f1fd89fd899d4f7ebab3603c9d10466d9f86c075fa5a348975b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d1887f28-e389-11ee-95c9-87774f69a715-mds-d, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b) 2024-03-16T11:48:10.839 INFO:journalctl@ceph.mds.d.smithi029.stdout:Mar 16 11:48:10 smithi029 ceph-mds[73225]: -- 172.21.15.29:0/2365144314 <== mon.0 v2:172.21.15.29:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55b970bcba40 con 0x55b970b9f000 2024-03-16T11:48:10.839 INFO:journalctl@ceph.mds.d.smithi029.stdout:Mar 16 11:48:10 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mds-d[73221]: starting mds.d at 2024-03-16T11:48:10.839 INFO:journalctl@ceph.mds.d.smithi029.stdout:Mar 16 11:48:10 smithi029 bash[73206]: 49650e1408b88f1fd89fd899d4f7ebab3603c9d10466d9f86c075fa5a348975b 2024-03-16T11:48:10.839 INFO:journalctl@ceph.mds.d.smithi029.stdout:Mar 16 11:48:10 smithi029 systemd[1]: Started Ceph mds.d for d1887f28-e389-11ee-95c9-87774f69a715. 2024-03-16T11:48:10.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:10 smithi152 ceph-mon[36291]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T11:48:10.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:10 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:48:10.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:10 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:48:10.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:10 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:48:10.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:10 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-16T11:48:10.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:10 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-16T11:48:10.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:10 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:48:11.046 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:10 smithi154 ceph-mon[36320]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T11:48:11.046 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:10 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:48:11.046 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:10 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:48:11.046 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:10 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:48:11.046 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:10 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-16T11:48:11.046 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:10 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-16T11:48:11.046 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:10 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:48:11.069 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T11:48:11.180 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:11.178+0000 7f798c0ac640 1 -- 172.21.15.29:0/1431002170 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7984102330 msgr2=0x7f7984108bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:48:11.180 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:11.178+0000 7f798c0ac640 1 --2- 172.21.15.29:0/1431002170 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7984102330 0x7f7984108bc0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f797c009ca0 tx=0x7f797c02c0f0 comp rx=0 tx=0).stop 2024-03-16T11:48:11.180 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:11.178+0000 7f798c0ac640 1 -- 172.21.15.29:0/1431002170 shutdown_connections 2024-03-16T11:48:11.181 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:11.178+0000 7f798c0ac640 1 --2- 172.21.15.29:0/1431002170 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7984102330 0x7f7984108bc0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:11.181 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:11.178+0000 7f798c0ac640 1 --2- 172.21.15.29:0/1431002170 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7984101970 0x7f7984101df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:11.181 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:11.178+0000 7f798c0ac640 1 --2- 172.21.15.29:0/1431002170 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7984100780 0x7f7984100ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:11.181 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:11.178+0000 7f798c0ac640 1 -- 172.21.15.29:0/1431002170 >> 172.21.15.29:0/1431002170 conn(0x7f79840fbd10 msgr2=0x7f79840fe130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:48:11.181 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:11.178+0000 7f798c0ac640 1 -- 172.21.15.29:0/1431002170 shutdown_connections 2024-03-16T11:48:11.181 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:11.178+0000 7f798c0ac640 1 -- 172.21.15.29:0/1431002170 wait complete. 2024-03-16T11:48:11.181 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:11.179+0000 7f798c0ac640 1 Processor -- start 2024-03-16T11:48:11.181 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:11.179+0000 7f798c0ac640 1 -- start start 2024-03-16T11:48:11.181 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:11.179+0000 7f798c0ac640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7984100780 0x7f7984197b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:48:11.181 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:11.179+0000 7f798c0ac640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7984101970 0x7f79841980a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:48:11.181 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:11.179+0000 7f798c0ac640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7984102330 0x7f798419f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:48:11.182 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:11.179+0000 7f798c0ac640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f798410c390 con 0x7f7984100780 2024-03-16T11:48:11.182 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:11.179+0000 7f798c0ac640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f798410c210 con 0x7f7984102330 2024-03-16T11:48:11.182 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:11.179+0000 7f798c0ac640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f798410c510 con 0x7f7984101970 2024-03-16T11:48:11.182 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:11.179+0000 7f7989e48640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7984100780 0x7f7984197b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:48:11.182 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:11.179+0000 7f7989e48640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7984100780 0x7f7984197b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42810/0 (socket says 172.21.15.29:42810) 2024-03-16T11:48:11.182 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:11.179+0000 7f7989e48640 1 -- 172.21.15.29:0/2810953239 learned_addr learned my addr 172.21.15.29:0/2810953239 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:48:11.182 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:11.179+0000 7f7989647640 1 --2- 172.21.15.29:0/2810953239 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7984101970 0x7f79841980a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:48:11.182 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:11.180+0000 7f7989e48640 1 -- 172.21.15.29:0/2810953239 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7984101970 msgr2=0x7f79841980a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:48:11.182 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:11.180+0000 7f7989e48640 1 --2- 172.21.15.29:0/2810953239 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7984101970 0x7f79841980a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:11.182 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:11.180+0000 7f798a649640 1 --2- 172.21.15.29:0/2810953239 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7984102330 0x7f798419f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:48:11.182 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:11.180+0000 7f7989e48640 1 -- 172.21.15.29:0/2810953239 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7984102330 msgr2=0x7f798419f110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:48:11.183 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:11.180+0000 7f7989e48640 1 --2- 172.21.15.29:0/2810953239 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7984102330 0x7f798419f110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:11.183 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:11.180+0000 7f7989e48640 1 -- 172.21.15.29:0/2810953239 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f798419f780 con 0x7f7984100780 2024-03-16T11:48:11.183 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:11.180+0000 7f7989647640 1 --2- 172.21.15.29:0/2810953239 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7984101970 0x7f79841980a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T11:48:11.183 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:11.180+0000 7f7989e48640 1 --2- 172.21.15.29:0/2810953239 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7984100780 0x7f7984197b60 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f798000d950 tx=0x7f798000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:48:11.183 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:11.180+0000 7f797affd640 1 -- 172.21.15.29:0/2810953239 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7980017020 con 0x7f7984100780 2024-03-16T11:48:11.183 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:11.180+0000 7f797affd640 1 -- 172.21.15.29:0/2810953239 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7980009c30 con 0x7f7984100780 2024-03-16T11:48:11.183 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:11.180+0000 7f798c0ac640 1 -- 172.21.15.29:0/2810953239 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f798419fa70 con 0x7f7984100780 2024-03-16T11:48:11.183 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:11.180+0000 7f797affd640 1 -- 172.21.15.29:0/2810953239 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f798000f740 con 0x7f7984100780 2024-03-16T11:48:11.183 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:11.180+0000 7f798c0ac640 1 -- 172.21.15.29:0/2810953239 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f798419ffc0 con 0x7f7984100780 2024-03-16T11:48:11.183 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:11.181+0000 7f798c0ac640 1 -- 172.21.15.29:0/2810953239 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f79841a0250 con 0x7f7984100780 2024-03-16T11:48:11.184 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:11.182+0000 7f797affd640 1 -- 172.21.15.29:0/2810953239 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f798001a020 con 0x7f7984100780 2024-03-16T11:48:11.184 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:11.183+0000 7f797affd640 1 --2- 172.21.15.29:0/2810953239 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f79700a2800 0x7f79700a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:48:11.185 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:11.183+0000 7f7989647640 1 --2- 172.21.15.29:0/2810953239 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f79700a2800 0x7f79700a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:48:11.185 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:11.183+0000 7f797affd640 1 -- 172.21.15.29:0/2810953239 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f79800ce490 con 0x7f7984100780 2024-03-16T11:48:11.185 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:11.183+0000 7f7989647640 1 --2- 172.21.15.29:0/2810953239 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f79700a2800 0x7f79700a4cc0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f7974005fd0 tx=0x7f797400b5d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:48:11.185 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:11.187+0000 7f797affd640 1 -- 172.21.15.29:0/2810953239 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f7980021070 con 0x7f7984100780 2024-03-16T11:48:11.408 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:11.409+0000 7f798c0ac640 1 -- 172.21.15.29:0/2810953239 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f79841a0420 con 0x7f7984100780 2024-03-16T11:48:11.409 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:11.409+0000 7f797affd640 1 -- 172.21.15.29:0/2810953239 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 5 v5) ==== 93+0+3874 (secure 0 0 0) 0x7f7980009de0 con 0x7f7984100780 2024-03-16T11:48:11.409 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-16T11:48:11.409 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":5,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14574,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/3505811955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":3505811955},{"type":"v1","addr":"172.21.15.29:6835","nonce":3505811955}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14586,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/1901337868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":1901337868},{"type":"v1","addr":"172.21.15.29:6837","nonce":1901337868}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24407,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6833/3136024628","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6832","nonce":3136024628},{"type":"v1","addr":"172.21.15.152:6833","nonce":3136024628}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24421,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6833/2677426282","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":2677426282},{"type":"v1","addr":"172.21.15.154:6833","nonce":2677426282}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-03-16T11:48:11.409 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 5 2024-03-16T11:48:11.411 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:11.411+0000 7f798c0ac640 1 -- 172.21.15.29:0/2810953239 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f79700a2800 msgr2=0x7f79700a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:48:11.411 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:11.411+0000 7f798c0ac640 1 --2- 172.21.15.29:0/2810953239 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f79700a2800 0x7f79700a4cc0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f7974005fd0 tx=0x7f797400b5d0 comp rx=0 tx=0).stop 2024-03-16T11:48:11.411 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:11.411+0000 7f798c0ac640 1 -- 172.21.15.29:0/2810953239 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7984100780 msgr2=0x7f7984197b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:48:11.411 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:11.411+0000 7f798c0ac640 1 --2- 172.21.15.29:0/2810953239 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7984100780 0x7f7984197b60 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f798000d950 tx=0x7f798000de10 comp rx=0 tx=0).stop 2024-03-16T11:48:11.411 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:11.411+0000 7f798c0ac640 1 -- 172.21.15.29:0/2810953239 shutdown_connections 2024-03-16T11:48:11.411 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:11.411+0000 7f798c0ac640 1 --2- 172.21.15.29:0/2810953239 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f79700a2800 0x7f79700a4cc0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:11.411 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:11.411+0000 7f798c0ac640 1 --2- 172.21.15.29:0/2810953239 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7984102330 0x7f798419f110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:11.411 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:11.411+0000 7f798c0ac640 1 --2- 172.21.15.29:0/2810953239 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7984101970 0x7f79841980a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:11.411 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:11.411+0000 7f798c0ac640 1 --2- 172.21.15.29:0/2810953239 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7984100780 0x7f7984197b60 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:11.411 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:11.411+0000 7f798c0ac640 1 -- 172.21.15.29:0/2810953239 >> 172.21.15.29:0/2810953239 conn(0x7f79840fbd10 msgr2=0x7f79840fd740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:48:11.411 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:11.411+0000 7f798c0ac640 1 -- 172.21.15.29:0/2810953239 shutdown_connections 2024-03-16T11:48:11.411 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:11.411+0000 7f798c0ac640 1 -- 172.21.15.29:0/2810953239 wait complete. 2024-03-16T11:48:11.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:11 smithi029 ceph-mon[31975]: Deploying daemon mds.e on smithi152 2024-03-16T11:48:11.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:11 smithi029 ceph-mon[31975]: mds.? [v2:172.21.15.29:6836/1901337868,v1:172.21.15.29:6837/1901337868] up:boot 2024-03-16T11:48:11.839 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:11 smithi029 ceph-mon[31975]: fsmap 4 up:standby 2024-03-16T11:48:11.839 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:11 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-16T11:48:11.839 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:11 smithi029 ceph-mon[31975]: from='client.? 172.21.15.29:0/2810953239' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T11:48:11.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:11 smithi152 ceph-mon[36291]: Deploying daemon mds.e on smithi152 2024-03-16T11:48:11.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:11 smithi152 ceph-mon[36291]: mds.? [v2:172.21.15.29:6836/1901337868,v1:172.21.15.29:6837/1901337868] up:boot 2024-03-16T11:48:11.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:11 smithi152 ceph-mon[36291]: fsmap 4 up:standby 2024-03-16T11:48:11.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:11 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-16T11:48:11.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:11 smithi152 ceph-mon[36291]: from='client.? 172.21.15.29:0/2810953239' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T11:48:12.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:11 smithi154 ceph-mon[36320]: Deploying daemon mds.e on smithi152 2024-03-16T11:48:12.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:11 smithi154 ceph-mon[36320]: mds.? [v2:172.21.15.29:6836/1901337868,v1:172.21.15.29:6837/1901337868] up:boot 2024-03-16T11:48:12.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:11 smithi154 ceph-mon[36320]: fsmap 4 up:standby 2024-03-16T11:48:12.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:11 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-16T11:48:12.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:11 smithi154 ceph-mon[36320]: from='client.? 172.21.15.29:0/2810953239' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T11:48:12.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:12 smithi152 ceph-mon[36291]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T11:48:12.878 INFO:journalctl@ceph.mds.e.smithi152.stdout:Mar 16 11:48:12 smithi152 systemd[1]: Starting Ceph mds.e for d1887f28-e389-11ee-95c9-87774f69a715... 2024-03-16T11:48:13.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:12 smithi154 ceph-mon[36320]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T11:48:13.087 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:12 smithi029 ceph-mon[31975]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T11:48:13.265 INFO:journalctl@ceph.mds.e.smithi152.stdout:Mar 16 11:48:12 smithi152 podman[66604]: 2024-03-16 11:48:12.865455521 +0000 UTC m=+0.017850576 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-16T11:48:13.265 INFO:journalctl@ceph.mds.e.smithi152.stdout:Mar 16 11:48:12 smithi152 podman[66604]: 2024-03-16 11:48:12.996131259 +0000 UTC m=+0.148526303 container create 822bc521a8b8be03443426a1111409f3034b08f2308590e2bde6d3b2c1606c40 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d1887f28-e389-11ee-95c9-87774f69a715-mds-e, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-03-16T11:48:13.265 INFO:journalctl@ceph.mds.e.smithi152.stdout:Mar 16 11:48:13 smithi152 podman[66604]: 2024-03-16 11:48:13.26606715 +0000 UTC m=+0.418462214 container init 822bc521a8b8be03443426a1111409f3034b08f2308590e2bde6d3b2c1606c40 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d1887f28-e389-11ee-95c9-87774f69a715-mds-e, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_BRANCH=HEAD) 2024-03-16T11:48:13.422 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T11:48:13.540 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:13.539+0000 7f6002868640 1 -- 172.21.15.29:0/3012001200 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5ffc100720 msgr2=0x7f5ffc100b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:48:13.540 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:13.539+0000 7f6002868640 1 --2- 172.21.15.29:0/3012001200 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5ffc100720 0x7f5ffc100b40 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f5fec009900 tx=0x7f5fec02bc30 comp rx=0 tx=0).stop 2024-03-16T11:48:13.540 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:13.539+0000 7f6002868640 1 -- 172.21.15.29:0/3012001200 shutdown_connections 2024-03-16T11:48:13.540 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:13.539+0000 7f6002868640 1 --2- 172.21.15.29:0/3012001200 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5ffc1022d0 0x7f5ffc108b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:13.540 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:13.539+0000 7f6002868640 1 --2- 172.21.15.29:0/3012001200 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5ffc101910 0x7f5ffc101d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:13.540 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:13.539+0000 7f6002868640 1 --2- 172.21.15.29:0/3012001200 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5ffc100720 0x7f5ffc100b40 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:13.540 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:13.539+0000 7f6002868640 1 -- 172.21.15.29:0/3012001200 >> 172.21.15.29:0/3012001200 conn(0x7f5ffc0fbc90 msgr2=0x7f5ffc0fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:48:13.541 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:13.539+0000 7f6002868640 1 -- 172.21.15.29:0/3012001200 shutdown_connections 2024-03-16T11:48:13.541 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:13.539+0000 7f6002868640 1 -- 172.21.15.29:0/3012001200 wait complete. 2024-03-16T11:48:13.541 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:13.539+0000 7f6002868640 1 Processor -- start 2024-03-16T11:48:13.541 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:13.539+0000 7f6002868640 1 -- start start 2024-03-16T11:48:13.541 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:13.540+0000 7f6002868640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5ffc100720 0x7f5ffc197ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:48:13.541 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:13.540+0000 7f6002868640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5ffc101910 0x7f5ffc198020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:48:13.541 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:13.540+0000 7f6002868640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5ffc1022d0 0x7f5ffc19f090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:48:13.541 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:13.540+0000 7f6002868640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f5ffc10d4c0 con 0x7f5ffc1022d0 2024-03-16T11:48:13.541 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:13.540+0000 7f6002868640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f5ffc10d340 con 0x7f5ffc100720 2024-03-16T11:48:13.541 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:13.540+0000 7f6002868640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f5ffc10d640 con 0x7f5ffc101910 2024-03-16T11:48:13.541 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:13.540+0000 7f6000e05640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5ffc1022d0 0x7f5ffc19f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:48:13.541 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:13.540+0000 7f5ffbfff640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5ffc100720 0x7f5ffc197ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:48:13.541 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:13.540+0000 7f5ffb7fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5ffc101910 0x7f5ffc198020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:48:13.542 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:13.540+0000 7f6000e05640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5ffc1022d0 0x7f5ffc19f090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42816/0 (socket says 172.21.15.29:42816) 2024-03-16T11:48:13.542 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:13.540+0000 7f5ffbfff640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5ffc100720 0x7f5ffc197ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.29:59838/0 (socket says 172.21.15.29:59838) 2024-03-16T11:48:13.542 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:13.540+0000 7f6000e05640 1 -- 172.21.15.29:0/237431914 learned_addr learned my addr 172.21.15.29:0/237431914 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:48:13.542 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:13.540+0000 7f6000e05640 1 -- 172.21.15.29:0/237431914 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5ffc101910 msgr2=0x7f5ffc198020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:48:13.542 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:13.540+0000 7f6000e05640 1 --2- 172.21.15.29:0/237431914 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5ffc101910 0x7f5ffc198020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:13.542 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:13.540+0000 7f6000e05640 1 -- 172.21.15.29:0/237431914 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5ffc100720 msgr2=0x7f5ffc197ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:48:13.542 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:13.540+0000 7f6000e05640 1 --2- 172.21.15.29:0/237431914 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5ffc100720 0x7f5ffc197ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:13.542 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:13.540+0000 7f6000e05640 1 -- 172.21.15.29:0/237431914 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ffc19f700 con 0x7f5ffc1022d0 2024-03-16T11:48:13.542 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:13.540+0000 7f6000e05640 1 --2- 172.21.15.29:0/237431914 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5ffc1022d0 0x7f5ffc19f090 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f5ff400eb10 tx=0x7f5ff400ef10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:48:13.542 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:13.541+0000 7f5ff97fa640 1 -- 172.21.15.29:0/237431914 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ff400ce20 con 0x7f5ffc1022d0 2024-03-16T11:48:13.542 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:13.541+0000 7f6002868640 1 -- 172.21.15.29:0/237431914 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5ffc19f9f0 con 0x7f5ffc1022d0 2024-03-16T11:48:13.542 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:13.541+0000 7f5ff97fa640 1 -- 172.21.15.29:0/237431914 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5ff4005020 con 0x7f5ffc1022d0 2024-03-16T11:48:13.543 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:13.541+0000 7f5ff97fa640 1 -- 172.21.15.29:0/237431914 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ff4010690 con 0x7f5ffc1022d0 2024-03-16T11:48:13.543 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:13.541+0000 7f6002868640 1 -- 172.21.15.29:0/237431914 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5ffc19ff40 con 0x7f5ffc1022d0 2024-03-16T11:48:13.543 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:13.542+0000 7f6002868640 1 -- 172.21.15.29:0/237431914 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5ffc107a40 con 0x7f5ffc1022d0 2024-03-16T11:48:13.544 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:13.543+0000 7f5ff97fa640 1 -- 172.21.15.29:0/237431914 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f5ff4005540 con 0x7f5ffc1022d0 2024-03-16T11:48:13.544 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:13.543+0000 7f5ff97fa640 1 --2- 172.21.15.29:0/237431914 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f5fe80a2800 0x7f5fe80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:48:13.544 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:13.543+0000 7f5ffbfff640 1 --2- 172.21.15.29:0/237431914 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f5fe80a2800 0x7f5fe80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:48:13.544 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:13.543+0000 7f5ff97fa640 1 -- 172.21.15.29:0/237431914 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f5ff40cecb0 con 0x7f5ffc1022d0 2024-03-16T11:48:13.544 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:13.544+0000 7f5ffbfff640 1 --2- 172.21.15.29:0/237431914 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f5fe80a2800 0x7f5fe80a4cc0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f5fec000c00 tx=0x7f5fec005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:48:13.547 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:13.548+0000 7f5ff97fa640 1 -- 172.21.15.29:0/237431914 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f5ff409c1f0 con 0x7f5ffc1022d0 2024-03-16T11:48:13.627 INFO:journalctl@ceph.mds.e.smithi152.stdout:Mar 16 11:48:13 smithi152 podman[66604]: 2024-03-16 11:48:13.269129981 +0000 UTC m=+0.421525035 container start 822bc521a8b8be03443426a1111409f3034b08f2308590e2bde6d3b2c1606c40 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d1887f28-e389-11ee-95c9-87774f69a715-mds-e, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_BRANCH=HEAD) 2024-03-16T11:48:13.627 INFO:journalctl@ceph.mds.e.smithi152.stdout:Mar 16 11:48:13 smithi152 ceph-mds[66623]: -- 172.21.15.152:0/2647910879 <== mon.2 v2:172.21.15.152:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55ff92cdba40 con 0x55ff92caec00 2024-03-16T11:48:13.628 INFO:journalctl@ceph.mds.e.smithi152.stdout:Mar 16 11:48:13 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mds-e[66619]: starting mds.e at 2024-03-16T11:48:13.628 INFO:journalctl@ceph.mds.e.smithi152.stdout:Mar 16 11:48:13 smithi152 bash[66604]: 822bc521a8b8be03443426a1111409f3034b08f2308590e2bde6d3b2c1606c40 2024-03-16T11:48:13.628 INFO:journalctl@ceph.mds.e.smithi152.stdout:Mar 16 11:48:13 smithi152 systemd[1]: Started Ceph mds.e for d1887f28-e389-11ee-95c9-87774f69a715. 2024-03-16T11:48:13.773 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:13.774+0000 7f6002868640 1 -- 172.21.15.29:0/237431914 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5ffc02d390 con 0x7f5ffc1022d0 2024-03-16T11:48:13.774 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:13.775+0000 7f5ff97fa640 1 -- 172.21.15.29:0/237431914 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) ==== 93+0+4702 (secure 0 0 0) 0x7f5ff409ce20 con 0x7f5ffc1022d0 2024-03-16T11:48:13.774 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-16T11:48:13.774 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":6,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14574,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/3505811955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":3505811955},{"type":"v1","addr":"172.21.15.29:6835","nonce":3505811955}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14586,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/1901337868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":1901337868},{"type":"v1","addr":"172.21.15.29:6837","nonce":1901337868}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24407,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6833/3136024628","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6832","nonce":3136024628},{"type":"v1","addr":"172.21.15.152:6833","nonce":3136024628}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24421,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6833/2677426282","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":2677426282},{"type":"v1","addr":"172.21.15.154:6833","nonce":2677426282}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24434,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6835/2919113178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6834","nonce":2919113178},{"type":"v1","addr":"172.21.15.152:6835","nonce":2919113178}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[]} 2024-03-16T11:48:13.774 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 6 2024-03-16T11:48:13.776 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:13.777+0000 7f6002868640 1 -- 172.21.15.29:0/237431914 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f5fe80a2800 msgr2=0x7f5fe80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:48:13.776 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:13.777+0000 7f6002868640 1 --2- 172.21.15.29:0/237431914 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f5fe80a2800 0x7f5fe80a4cc0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f5fec000c00 tx=0x7f5fec005fd0 comp rx=0 tx=0).stop 2024-03-16T11:48:13.777 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:13.777+0000 7f6002868640 1 -- 172.21.15.29:0/237431914 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5ffc1022d0 msgr2=0x7f5ffc19f090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:48:13.777 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:13.777+0000 7f6002868640 1 --2- 172.21.15.29:0/237431914 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5ffc1022d0 0x7f5ffc19f090 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f5ff400eb10 tx=0x7f5ff400ef10 comp rx=0 tx=0).stop 2024-03-16T11:48:13.777 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:13.777+0000 7f6002868640 1 -- 172.21.15.29:0/237431914 shutdown_connections 2024-03-16T11:48:13.777 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:13.777+0000 7f6002868640 1 --2- 172.21.15.29:0/237431914 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f5fe80a2800 0x7f5fe80a4cc0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:13.777 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:13.777+0000 7f6002868640 1 --2- 172.21.15.29:0/237431914 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5ffc1022d0 0x7f5ffc19f090 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:13.777 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:13.777+0000 7f6002868640 1 --2- 172.21.15.29:0/237431914 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5ffc101910 0x7f5ffc198020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:13.777 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:13.777+0000 7f6002868640 1 --2- 172.21.15.29:0/237431914 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5ffc100720 0x7f5ffc197ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:13.777 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:13.777+0000 7f6002868640 1 -- 172.21.15.29:0/237431914 >> 172.21.15.29:0/237431914 conn(0x7f5ffc0fbc90 msgr2=0x7f5ffc104c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:48:13.777 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:13.777+0000 7f6002868640 1 -- 172.21.15.29:0/237431914 shutdown_connections 2024-03-16T11:48:13.777 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:13.777+0000 7f6002868640 1 -- 172.21.15.29:0/237431914 wait complete. 2024-03-16T11:48:14.774 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:14 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:48:14.775 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:14 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:48:14.775 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:14 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:48:14.775 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:14 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-16T11:48:14.775 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:14 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-16T11:48:14.775 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:14 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:48:14.775 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:14 smithi154 ceph-mon[36320]: Deploying daemon mds.f on smithi154 2024-03-16T11:48:14.775 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:14 smithi154 ceph-mon[36320]: mds.? [v2:172.21.15.152:6834/2919113178,v1:172.21.15.152:6835/2919113178] up:boot 2024-03-16T11:48:14.775 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:14 smithi154 ceph-mon[36320]: fsmap 5 up:standby 2024-03-16T11:48:14.775 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:14 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-16T11:48:14.775 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:14 smithi154 ceph-mon[36320]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T11:48:14.775 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:14 smithi154 ceph-mon[36320]: from='client.? 172.21.15.29:0/237431914' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T11:48:14.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:14 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:48:14.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:14 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:48:14.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:14 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:48:14.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:14 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-16T11:48:14.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:14 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-16T11:48:14.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:14 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:48:14.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:14 smithi029 ceph-mon[31975]: Deploying daemon mds.f on smithi154 2024-03-16T11:48:14.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:14 smithi029 ceph-mon[31975]: mds.? [v2:172.21.15.152:6834/2919113178,v1:172.21.15.152:6835/2919113178] up:boot 2024-03-16T11:48:14.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:14 smithi029 ceph-mon[31975]: fsmap 5 up:standby 2024-03-16T11:48:14.839 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:14 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-16T11:48:14.839 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:14 smithi029 ceph-mon[31975]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T11:48:14.839 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:14 smithi029 ceph-mon[31975]: from='client.? 172.21.15.29:0/237431914' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T11:48:14.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:14 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:48:14.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:14 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:48:14.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:14 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:48:14.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:14 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-16T11:48:14.878 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:14 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-16T11:48:14.878 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:14 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:48:14.878 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:14 smithi152 ceph-mon[36291]: Deploying daemon mds.f on smithi154 2024-03-16T11:48:14.878 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:14 smithi152 ceph-mon[36291]: mds.? [v2:172.21.15.152:6834/2919113178,v1:172.21.15.152:6835/2919113178] up:boot 2024-03-16T11:48:14.878 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:14 smithi152 ceph-mon[36291]: fsmap 5 up:standby 2024-03-16T11:48:14.878 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:14 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-16T11:48:14.878 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:14 smithi152 ceph-mon[36291]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T11:48:14.878 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:14 smithi152 ceph-mon[36291]: from='client.? 172.21.15.29:0/237431914' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T11:48:15.681 INFO:journalctl@ceph.mds.f.smithi154.stdout:Mar 16 11:48:15 smithi154 systemd[1]: Starting Ceph mds.f for d1887f28-e389-11ee-95c9-87774f69a715... 2024-03-16T11:48:15.788 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T11:48:15.904 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:15.903+0000 7f13dc612640 1 -- 172.21.15.29:0/1922927372 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f13d4102360 msgr2=0x7f13d4104750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:48:15.905 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:15.903+0000 7f13dc612640 1 --2- 172.21.15.29:0/1922927372 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f13d4102360 0x7f13d4104750 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f13cc009900 tx=0x7f13cc02bc50 comp rx=0 tx=0).stop 2024-03-16T11:48:15.905 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:15.903+0000 7f13dc612640 1 -- 172.21.15.29:0/1922927372 shutdown_connections 2024-03-16T11:48:15.905 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:15.903+0000 7f13dc612640 1 --2- 172.21.15.29:0/1922927372 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f13d41075c0 0x7f13d41099b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:15.905 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:15.903+0000 7f13dc612640 1 --2- 172.21.15.29:0/1922927372 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f13d4104c90 0x7f13d4107080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:15.905 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:15.903+0000 7f13dc612640 1 --2- 172.21.15.29:0/1922927372 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f13d4102360 0x7f13d4104750 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:15.905 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:15.903+0000 7f13dc612640 1 -- 172.21.15.29:0/1922927372 >> 172.21.15.29:0/1922927372 conn(0x7f13d40fbd10 msgr2=0x7f13d40fe130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:48:15.905 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:15.903+0000 7f13dc612640 1 -- 172.21.15.29:0/1922927372 shutdown_connections 2024-03-16T11:48:15.905 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:15.903+0000 7f13dc612640 1 -- 172.21.15.29:0/1922927372 wait complete. 2024-03-16T11:48:15.905 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:15.904+0000 7f13dc612640 1 Processor -- start 2024-03-16T11:48:15.905 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:15.904+0000 7f13dc612640 1 -- start start 2024-03-16T11:48:15.905 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:15.904+0000 7f13dc612640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f13d4102360 0x7f13d4197c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:48:15.905 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:15.904+0000 7f13dc612640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f13d4104c90 0x7f13d41981b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:48:15.906 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:15.904+0000 7f13dc612640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f13d41075c0 0x7f13d419f1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:48:15.906 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:15.904+0000 7f13dc612640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f13d410d500 con 0x7f13d41075c0 2024-03-16T11:48:15.906 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:15.904+0000 7f13dc612640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f13d410d380 con 0x7f13d4104c90 2024-03-16T11:48:15.906 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:15.904+0000 7f13dc612640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f13d410d680 con 0x7f13d4102360 2024-03-16T11:48:15.906 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:15.904+0000 7f13dabaf640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f13d41075c0 0x7f13d419f1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:48:15.906 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:15.904+0000 7f13dabaf640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f13d41075c0 0x7f13d419f1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42840/0 (socket says 172.21.15.29:42840) 2024-03-16T11:48:15.906 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:15.904+0000 7f13dabaf640 1 -- 172.21.15.29:0/2455923908 learned_addr learned my addr 172.21.15.29:0/2455923908 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:48:15.906 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:15.904+0000 7f13da3ae640 1 --2- 172.21.15.29:0/2455923908 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f13d4102360 0x7f13d4197c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:48:15.906 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:15.904+0000 7f13d9bad640 1 --2- 172.21.15.29:0/2455923908 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f13d4104c90 0x7f13d41981b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:48:15.906 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:15.904+0000 7f13dabaf640 1 -- 172.21.15.29:0/2455923908 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f13d4102360 msgr2=0x7f13d4197c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:48:15.906 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:15.904+0000 7f13dabaf640 1 --2- 172.21.15.29:0/2455923908 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f13d4102360 0x7f13d4197c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:15.906 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:15.904+0000 7f13dabaf640 1 -- 172.21.15.29:0/2455923908 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f13d4104c90 msgr2=0x7f13d41981b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:48:15.906 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:15.904+0000 7f13dabaf640 1 --2- 172.21.15.29:0/2455923908 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f13d4104c90 0x7f13d41981b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:15.906 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:15.904+0000 7f13dabaf640 1 -- 172.21.15.29:0/2455923908 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13d419f880 con 0x7f13d41075c0 2024-03-16T11:48:15.907 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:15.904+0000 7f13dabaf640 1 --2- 172.21.15.29:0/2455923908 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f13d41075c0 0x7f13d419f1e0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f13d000eaa0 tx=0x7f13d000ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:48:15.907 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:15.905+0000 7f13cb7fe640 1 -- 172.21.15.29:0/2455923908 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13d000cdb0 con 0x7f13d41075c0 2024-03-16T11:48:15.907 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:15.905+0000 7f13cb7fe640 1 -- 172.21.15.29:0/2455923908 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f13d0005020 con 0x7f13d41075c0 2024-03-16T11:48:15.907 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:15.905+0000 7f13dc612640 1 -- 172.21.15.29:0/2455923908 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f13d419fb70 con 0x7f13d41075c0 2024-03-16T11:48:15.907 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:15.905+0000 7f13cb7fe640 1 -- 172.21.15.29:0/2455923908 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13d00116b0 con 0x7f13d41075c0 2024-03-16T11:48:15.907 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:15.905+0000 7f13dc612640 1 -- 172.21.15.29:0/2455923908 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f13d41a0070 con 0x7f13d41075c0 2024-03-16T11:48:15.907 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:15.906+0000 7f13dc612640 1 -- 172.21.15.29:0/2455923908 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f13d40fd800 con 0x7f13d41075c0 2024-03-16T11:48:15.908 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:15.907+0000 7f13cb7fe640 1 -- 172.21.15.29:0/2455923908 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f13d0005560 con 0x7f13d41075c0 2024-03-16T11:48:15.908 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:15.907+0000 7f13cb7fe640 1 --2- 172.21.15.29:0/2455923908 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f13c00a2800 0x7f13c00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:48:15.908 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:15.907+0000 7f13da3ae640 1 --2- 172.21.15.29:0/2455923908 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f13c00a2800 0x7f13c00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:48:15.908 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:15.907+0000 7f13cb7fe640 1 -- 172.21.15.29:0/2455923908 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f13d00cecc0 con 0x7f13d41075c0 2024-03-16T11:48:15.908 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:15.908+0000 7f13da3ae640 1 --2- 172.21.15.29:0/2455923908 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f13c00a2800 0x7f13c00a4cc0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f13d4199140 tx=0x7f13cc005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:48:15.910 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:15.911+0000 7f13cb7fe640 1 -- 172.21.15.29:0/2455923908 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f13d0016030 con 0x7f13d41075c0 2024-03-16T11:48:16.023 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:15 smithi154 ceph-mon[36320]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T11:48:16.023 INFO:journalctl@ceph.mds.f.smithi154.stdout:Mar 16 11:48:15 smithi154 podman[66406]: 2024-03-16 11:48:15.651200799 +0000 UTC m=+0.017764860 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-16T11:48:16.023 INFO:journalctl@ceph.mds.f.smithi154.stdout:Mar 16 11:48:15 smithi154 podman[66406]: 2024-03-16 11:48:15.770873862 +0000 UTC m=+0.137437914 container create 3e849fc2b38e390e5e52a846bcefee2dfab94f38b0d914e6b2d19b0b1761baf7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d1887f28-e389-11ee-95c9-87774f69a715-mds-f, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-03-16T11:48:16.023 INFO:journalctl@ceph.mds.f.smithi154.stdout:Mar 16 11:48:16 smithi154 podman[66406]: 2024-03-16 11:48:16.023916663 +0000 UTC m=+0.390480720 container init 3e849fc2b38e390e5e52a846bcefee2dfab94f38b0d914e6b2d19b0b1761baf7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d1887f28-e389-11ee-95c9-87774f69a715-mds-f, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-03-16T11:48:16.087 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:15 smithi029 ceph-mon[31975]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T11:48:16.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:15 smithi152 ceph-mon[36291]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T11:48:16.132 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:16.133+0000 7f13dc612640 1 -- 172.21.15.29:0/2455923908 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f13d402d1b0 con 0x7f13d41075c0 2024-03-16T11:48:16.132 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:16.133+0000 7f13cb7fe640 1 -- 172.21.15.29:0/2455923908 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) ==== 93+0+4702 (secure 0 0 0) 0x7f13d009c200 con 0x7f13d41075c0 2024-03-16T11:48:16.133 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-16T11:48:16.133 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":6,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14574,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/3505811955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":3505811955},{"type":"v1","addr":"172.21.15.29:6835","nonce":3505811955}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14586,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/1901337868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":1901337868},{"type":"v1","addr":"172.21.15.29:6837","nonce":1901337868}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24407,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6833/3136024628","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6832","nonce":3136024628},{"type":"v1","addr":"172.21.15.152:6833","nonce":3136024628}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24421,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6833/2677426282","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":2677426282},{"type":"v1","addr":"172.21.15.154:6833","nonce":2677426282}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24434,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6835/2919113178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6834","nonce":2919113178},{"type":"v1","addr":"172.21.15.152:6835","nonce":2919113178}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[]} 2024-03-16T11:48:16.133 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 6 2024-03-16T11:48:16.135 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:16.135+0000 7f13dc612640 1 -- 172.21.15.29:0/2455923908 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f13c00a2800 msgr2=0x7f13c00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:48:16.135 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:16.135+0000 7f13dc612640 1 --2- 172.21.15.29:0/2455923908 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f13c00a2800 0x7f13c00a4cc0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f13d4199140 tx=0x7f13cc005fd0 comp rx=0 tx=0).stop 2024-03-16T11:48:16.135 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:16.135+0000 7f13dc612640 1 -- 172.21.15.29:0/2455923908 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f13d41075c0 msgr2=0x7f13d419f1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:48:16.135 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:16.135+0000 7f13dc612640 1 --2- 172.21.15.29:0/2455923908 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f13d41075c0 0x7f13d419f1e0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f13d000eaa0 tx=0x7f13d000ef60 comp rx=0 tx=0).stop 2024-03-16T11:48:16.135 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:16.135+0000 7f13dc612640 1 -- 172.21.15.29:0/2455923908 shutdown_connections 2024-03-16T11:48:16.135 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:16.135+0000 7f13dc612640 1 --2- 172.21.15.29:0/2455923908 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f13c00a2800 0x7f13c00a4cc0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:16.135 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:16.135+0000 7f13dc612640 1 --2- 172.21.15.29:0/2455923908 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f13d41075c0 0x7f13d419f1e0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:16.135 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:16.135+0000 7f13dc612640 1 --2- 172.21.15.29:0/2455923908 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f13d4104c90 0x7f13d41981b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:16.135 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:16.135+0000 7f13dc612640 1 --2- 172.21.15.29:0/2455923908 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f13d4102360 0x7f13d4197c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:16.135 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:16.135+0000 7f13dc612640 1 -- 172.21.15.29:0/2455923908 >> 172.21.15.29:0/2455923908 conn(0x7f13d40fbd10 msgr2=0x7f13d4102e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:48:16.135 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:16.135+0000 7f13dc612640 1 -- 172.21.15.29:0/2455923908 shutdown_connections 2024-03-16T11:48:16.136 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:16.135+0000 7f13dc612640 1 -- 172.21.15.29:0/2455923908 wait complete. 2024-03-16T11:48:16.295 INFO:journalctl@ceph.mds.f.smithi154.stdout:Mar 16 11:48:16 smithi154 podman[66406]: 2024-03-16 11:48:16.026604309 +0000 UTC m=+0.393168379 container start 3e849fc2b38e390e5e52a846bcefee2dfab94f38b0d914e6b2d19b0b1761baf7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d1887f28-e389-11ee-95c9-87774f69a715-mds-f, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-03-16T11:48:16.295 INFO:journalctl@ceph.mds.f.smithi154.stdout:Mar 16 11:48:16 smithi154 ceph-mds[66425]: -- 172.21.15.154:0/1372986365 <== mon.1 v2:172.21.15.154:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55f332881a40 con 0x55f332854800 2024-03-16T11:48:16.295 INFO:journalctl@ceph.mds.f.smithi154.stdout:Mar 16 11:48:16 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mds-f[66421]: starting mds.f at 2024-03-16T11:48:16.295 INFO:journalctl@ceph.mds.f.smithi154.stdout:Mar 16 11:48:16 smithi154 bash[66406]: 3e849fc2b38e390e5e52a846bcefee2dfab94f38b0d914e6b2d19b0b1761baf7 2024-03-16T11:48:16.295 INFO:journalctl@ceph.mds.f.smithi154.stdout:Mar 16 11:48:16 smithi154 systemd[1]: Started Ceph mds.f for d1887f28-e389-11ee-95c9-87774f69a715. 2024-03-16T11:48:17.002 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:16 smithi154 ceph-mon[36320]: from='client.? 172.21.15.29:0/2455923908' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T11:48:17.002 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:16 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:48:17.002 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:16 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:48:17.003 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:16 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:48:17.003 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:16 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-16T11:48:17.003 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:16 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-16T11:48:17.003 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:16 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:48:17.003 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:16 smithi154 ceph-mon[36320]: Deploying daemon mds.c on smithi154 2024-03-16T11:48:17.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:16 smithi029 ceph-mon[31975]: from='client.? 172.21.15.29:0/2455923908' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T11:48:17.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:16 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:48:17.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:16 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:48:17.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:16 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:48:17.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:16 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-16T11:48:17.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:16 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-16T11:48:17.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:16 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:48:17.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:16 smithi029 ceph-mon[31975]: Deploying daemon mds.c on smithi154 2024-03-16T11:48:17.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:16 smithi152 ceph-mon[36291]: from='client.? 172.21.15.29:0/2455923908' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T11:48:17.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:16 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:48:17.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:16 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:48:17.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:16 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:48:17.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:16 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-16T11:48:17.128 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:16 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-16T11:48:17.128 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:16 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:48:17.128 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:16 smithi152 ceph-mon[36291]: Deploying daemon mds.c on smithi154 2024-03-16T11:48:18.005 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:17 smithi154 ceph-mon[36320]: mds.? [v2:172.21.15.154:6834/4165274681,v1:172.21.15.154:6835/4165274681] up:boot 2024-03-16T11:48:18.005 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:17 smithi154 ceph-mon[36320]: fsmap 6 up:standby 2024-03-16T11:48:18.005 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:17 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-16T11:48:18.005 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:17 smithi154 ceph-mon[36320]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T11:48:18.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:17 smithi029 ceph-mon[31975]: mds.? [v2:172.21.15.154:6834/4165274681,v1:172.21.15.154:6835/4165274681] up:boot 2024-03-16T11:48:18.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:17 smithi029 ceph-mon[31975]: fsmap 6 up:standby 2024-03-16T11:48:18.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:17 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-16T11:48:18.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:17 smithi029 ceph-mon[31975]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T11:48:18.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:17 smithi152 ceph-mon[36291]: mds.? [v2:172.21.15.154:6834/4165274681,v1:172.21.15.154:6835/4165274681] up:boot 2024-03-16T11:48:18.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:17 smithi152 ceph-mon[36291]: fsmap 6 up:standby 2024-03-16T11:48:18.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:17 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-16T11:48:18.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:17 smithi152 ceph-mon[36291]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T11:48:18.146 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T11:48:18.263 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:18.261+0000 7f0c16481640 1 -- 172.21.15.29:0/1856643281 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0c10100780 msgr2=0x7f0c10100ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:48:18.263 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:18.261+0000 7f0c16481640 1 --2- 172.21.15.29:0/1856643281 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0c10100780 0x7f0c10100ba0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f0c00009900 tx=0x7f0c0002bc30 comp rx=0 tx=0).stop 2024-03-16T11:48:18.263 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:18.261+0000 7f0c16481640 1 -- 172.21.15.29:0/1856643281 shutdown_connections 2024-03-16T11:48:18.263 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:18.261+0000 7f0c16481640 1 --2- 172.21.15.29:0/1856643281 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0c10102330 0x7f0c10108bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:18.263 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:18.261+0000 7f0c16481640 1 --2- 172.21.15.29:0/1856643281 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0c10101970 0x7f0c10101df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:18.263 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:18.261+0000 7f0c16481640 1 --2- 172.21.15.29:0/1856643281 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0c10100780 0x7f0c10100ba0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:18.263 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:18.261+0000 7f0c16481640 1 -- 172.21.15.29:0/1856643281 >> 172.21.15.29:0/1856643281 conn(0x7f0c100fbd10 msgr2=0x7f0c100fe130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:48:18.263 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:18.261+0000 7f0c16481640 1 -- 172.21.15.29:0/1856643281 shutdown_connections 2024-03-16T11:48:18.263 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:18.261+0000 7f0c16481640 1 -- 172.21.15.29:0/1856643281 wait complete. 2024-03-16T11:48:18.263 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:18.262+0000 7f0c16481640 1 Processor -- start 2024-03-16T11:48:18.263 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:18.262+0000 7f0c16481640 1 -- start start 2024-03-16T11:48:18.264 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:18.262+0000 7f0c16481640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0c10100780 0x7f0c1006a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:48:18.264 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:18.262+0000 7f0c16481640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0c10101970 0x7f0c1019c7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:48:18.264 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:18.262+0000 7f0c16481640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0c10102330 0x7f0c1019ee60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:48:18.264 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:18.262+0000 7f0c16481640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f0c1010c360 con 0x7f0c10101970 2024-03-16T11:48:18.264 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:18.262+0000 7f0c16481640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f0c1010c1e0 con 0x7f0c10102330 2024-03-16T11:48:18.264 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:18.262+0000 7f0c16481640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f0c1010c4e0 con 0x7f0c10100780 2024-03-16T11:48:18.264 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:18.262+0000 7f0c0f7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0c10101970 0x7f0c1019c7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:48:18.264 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:18.262+0000 7f0c0f7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0c10101970 0x7f0c1019c7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42862/0 (socket says 172.21.15.29:42862) 2024-03-16T11:48:18.264 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:18.262+0000 7f0c0f7fe640 1 -- 172.21.15.29:0/2211989696 learned_addr learned my addr 172.21.15.29:0/2211989696 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:48:18.264 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:18.262+0000 7f0c0ffff640 1 --2- 172.21.15.29:0/2211989696 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0c10100780 0x7f0c1006a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:48:18.264 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:18.263+0000 7f0c14a1e640 1 --2- 172.21.15.29:0/2211989696 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0c10102330 0x7f0c1019ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:48:18.264 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:18.263+0000 7f0c0f7fe640 1 -- 172.21.15.29:0/2211989696 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0c10100780 msgr2=0x7f0c1006a9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:48:18.264 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:18.263+0000 7f0c0f7fe640 1 --2- 172.21.15.29:0/2211989696 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0c10100780 0x7f0c1006a9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:18.264 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:18.263+0000 7f0c0f7fe640 1 -- 172.21.15.29:0/2211989696 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0c10102330 msgr2=0x7f0c1019ee60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:48:18.265 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:18.263+0000 7f0c0f7fe640 1 --2- 172.21.15.29:0/2211989696 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0c10102330 0x7f0c1019ee60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:18.265 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:18.263+0000 7f0c0f7fe640 1 -- 172.21.15.29:0/2211989696 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c1019f720 con 0x7f0c10101970 2024-03-16T11:48:18.265 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:18.263+0000 7f0c14a1e640 1 --2- 172.21.15.29:0/2211989696 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0c10102330 0x7f0c1019ee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-16T11:48:18.265 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:18.263+0000 7f0c0f7fe640 1 --2- 172.21.15.29:0/2211989696 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0c10101970 0x7f0c1019c7f0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f0c0400d950 tx=0x7f0c0400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:48:18.265 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:18.263+0000 7f0c0d7fa640 1 -- 172.21.15.29:0/2211989696 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c04017070 con 0x7f0c10101970 2024-03-16T11:48:18.265 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:18.263+0000 7f0c0d7fa640 1 -- 172.21.15.29:0/2211989696 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0c04009c30 con 0x7f0c10101970 2024-03-16T11:48:18.265 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:18.263+0000 7f0c16481640 1 -- 172.21.15.29:0/2211989696 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0c1019fa10 con 0x7f0c10101970 2024-03-16T11:48:18.265 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:18.263+0000 7f0c0d7fa640 1 -- 172.21.15.29:0/2211989696 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c0400f6b0 con 0x7f0c10101970 2024-03-16T11:48:18.265 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:18.263+0000 7f0c16481640 1 -- 172.21.15.29:0/2211989696 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0c1019ff60 con 0x7f0c10101970 2024-03-16T11:48:18.266 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:18.264+0000 7f0c16481640 1 -- 172.21.15.29:0/2211989696 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0c10107b20 con 0x7f0c10101970 2024-03-16T11:48:18.266 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:18.265+0000 7f0c0d7fa640 1 -- 172.21.15.29:0/2211989696 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f0c040109d0 con 0x7f0c10101970 2024-03-16T11:48:18.266 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:18.266+0000 7f0c0d7fa640 1 --2- 172.21.15.29:0/2211989696 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f0bf80a2800 0x7f0bf80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:48:18.266 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:18.266+0000 7f0c0ffff640 1 --2- 172.21.15.29:0/2211989696 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f0bf80a2800 0x7f0bf80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:48:18.266 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:18.266+0000 7f0c0d7fa640 1 -- 172.21.15.29:0/2211989696 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f0c040ce1e0 con 0x7f0c10101970 2024-03-16T11:48:18.266 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:18.266+0000 7f0c0ffff640 1 --2- 172.21.15.29:0/2211989696 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f0bf80a2800 0x7f0bf80a4cc0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f0c00000c00 tx=0x7f0c00005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:48:18.269 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:18.270+0000 7f0c0d7fa640 1 -- 172.21.15.29:0/2211989696 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f0c04009de0 con 0x7f0c10101970 2024-03-16T11:48:18.490 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:18.491+0000 7f0c16481640 1 -- 172.21.15.29:0/2211989696 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0c1002d1b0 con 0x7f0c10101970 2024-03-16T11:48:18.491 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:18.492+0000 7f0c0d7fa640 1 -- 172.21.15.29:0/2211989696 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) ==== 93+0+5530 (secure 0 0 0) 0x7f0c0409b720 con 0x7f0c10101970 2024-03-16T11:48:18.491 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-16T11:48:18.491 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":7,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14574,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/3505811955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":3505811955},{"type":"v1","addr":"172.21.15.29:6835","nonce":3505811955}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14586,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/1901337868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":1901337868},{"type":"v1","addr":"172.21.15.29:6837","nonce":1901337868}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24407,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6833/3136024628","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6832","nonce":3136024628},{"type":"v1","addr":"172.21.15.152:6833","nonce":3136024628}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24421,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6833/2677426282","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":2677426282},{"type":"v1","addr":"172.21.15.154:6833","nonce":2677426282}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24434,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6835/2919113178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6834","nonce":2919113178},{"type":"v1","addr":"172.21.15.152:6835","nonce":2919113178}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24466,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/4165274681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":4165274681},{"type":"v1","addr":"172.21.15.154:6835","nonce":4165274681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7}],"filesystems":[]} 2024-03-16T11:48:18.491 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 7 2024-03-16T11:48:18.493 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:18.494+0000 7f0c16481640 1 -- 172.21.15.29:0/2211989696 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f0bf80a2800 msgr2=0x7f0bf80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:48:18.493 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:18.494+0000 7f0c16481640 1 --2- 172.21.15.29:0/2211989696 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f0bf80a2800 0x7f0bf80a4cc0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f0c00000c00 tx=0x7f0c00005fd0 comp rx=0 tx=0).stop 2024-03-16T11:48:18.493 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:18.494+0000 7f0c16481640 1 -- 172.21.15.29:0/2211989696 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0c10101970 msgr2=0x7f0c1019c7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:48:18.493 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:18.494+0000 7f0c16481640 1 --2- 172.21.15.29:0/2211989696 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0c10101970 0x7f0c1019c7f0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f0c0400d950 tx=0x7f0c0400de10 comp rx=0 tx=0).stop 2024-03-16T11:48:18.493 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:18.494+0000 7f0c16481640 1 -- 172.21.15.29:0/2211989696 shutdown_connections 2024-03-16T11:48:18.493 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:18.494+0000 7f0c16481640 1 --2- 172.21.15.29:0/2211989696 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f0bf80a2800 0x7f0bf80a4cc0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:18.493 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:18.494+0000 7f0c16481640 1 --2- 172.21.15.29:0/2211989696 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0c10102330 0x7f0c1019ee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:18.493 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:18.494+0000 7f0c16481640 1 --2- 172.21.15.29:0/2211989696 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0c10101970 0x7f0c1019c7f0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:18.493 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:18.494+0000 7f0c16481640 1 --2- 172.21.15.29:0/2211989696 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0c10100780 0x7f0c1006a9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:18.494 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:18.494+0000 7f0c16481640 1 -- 172.21.15.29:0/2211989696 >> 172.21.15.29:0/2211989696 conn(0x7f0c100fbd10 msgr2=0x7f0c10104cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:48:18.494 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:18.494+0000 7f0c16481640 1 -- 172.21.15.29:0/2211989696 shutdown_connections 2024-03-16T11:48:18.494 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:18.494+0000 7f0c16481640 1 -- 172.21.15.29:0/2211989696 wait complete. 2024-03-16T11:48:19.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:18 smithi154 ceph-mon[36320]: from='client.? 172.21.15.29:0/2211989696' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T11:48:19.295 INFO:journalctl@ceph.mds.c.smithi154.stdout:Mar 16 11:48:19 smithi154 systemd[1]: Started Ceph mds.c for d1887f28-e389-11ee-95c9-87774f69a715. 2024-03-16T11:48:19.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:18 smithi029 ceph-mon[31975]: from='client.? 172.21.15.29:0/2211989696' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T11:48:19.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:18 smithi152 ceph-mon[36291]: from='client.? 172.21.15.29:0/2211989696' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T11:48:20.002 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:19 smithi152 ceph-mon[36291]: mds.? [v2:172.21.15.154:6836/1468560096,v1:172.21.15.154:6837/1468560096] up:boot 2024-03-16T11:48:20.002 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:19 smithi152 ceph-mon[36291]: fsmap 7 up:standby 2024-03-16T11:48:20.002 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:19 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-16T11:48:20.003 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:19 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:48:20.003 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:19 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:48:20.003 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:19 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:48:20.003 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:19 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-16T11:48:20.003 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:19 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-16T11:48:20.003 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:19 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:48:20.003 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:19 smithi152 ceph-mon[36291]: Deploying daemon mds.h on smithi152 2024-03-16T11:48:20.003 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:19 smithi152 ceph-mon[36291]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T11:48:20.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:19 smithi154 ceph-mon[36320]: mds.? [v2:172.21.15.154:6836/1468560096,v1:172.21.15.154:6837/1468560096] up:boot 2024-03-16T11:48:20.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:19 smithi154 ceph-mon[36320]: fsmap 7 up:standby 2024-03-16T11:48:20.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:19 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-16T11:48:20.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:19 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:48:20.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:19 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:48:20.296 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:19 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:48:20.296 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:19 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-16T11:48:20.296 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:19 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-16T11:48:20.296 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:19 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:48:20.296 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:19 smithi154 ceph-mon[36320]: Deploying daemon mds.h on smithi152 2024-03-16T11:48:20.296 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:19 smithi154 ceph-mon[36320]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T11:48:20.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:19 smithi029 ceph-mon[31975]: mds.? [v2:172.21.15.154:6836/1468560096,v1:172.21.15.154:6837/1468560096] up:boot 2024-03-16T11:48:20.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:19 smithi029 ceph-mon[31975]: fsmap 7 up:standby 2024-03-16T11:48:20.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:19 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-16T11:48:20.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:19 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:48:20.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:19 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:48:20.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:19 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:48:20.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:19 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-16T11:48:20.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:19 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-16T11:48:20.339 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:19 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:48:20.339 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:19 smithi029 ceph-mon[31975]: Deploying daemon mds.h on smithi152 2024-03-16T11:48:20.339 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:19 smithi029 ceph-mon[31975]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T11:48:20.505 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T11:48:20.623 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:20.621+0000 7fb952b10640 1 -- 172.21.15.29:0/2384078762 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb94c0ffad0 msgr2=0x7fb94c0ffef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:48:20.623 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:20.621+0000 7fb952b10640 1 --2- 172.21.15.29:0/2384078762 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb94c0ffad0 0x7fb94c0ffef0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fb944009cd0 tx=0x7fb94402c0f0 comp rx=0 tx=0).stop 2024-03-16T11:48:20.623 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:20.621+0000 7fb952b10640 1 -- 172.21.15.29:0/2384078762 shutdown_connections 2024-03-16T11:48:20.623 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:20.621+0000 7fb952b10640 1 --2- 172.21.15.29:0/2384078762 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb94c103460 0x7fb94c109cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:20.623 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:20.621+0000 7fb952b10640 1 --2- 172.21.15.29:0/2384078762 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb94c102aa0 0x7fb94c102f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:20.623 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:20.621+0000 7fb952b10640 1 --2- 172.21.15.29:0/2384078762 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb94c0ffad0 0x7fb94c0ffef0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:20.623 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:20.621+0000 7fb952b10640 1 -- 172.21.15.29:0/2384078762 >> 172.21.15.29:0/2384078762 conn(0x7fb94c0fb400 msgr2=0x7fb94c0fd840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:48:20.623 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:20.621+0000 7fb952b10640 1 -- 172.21.15.29:0/2384078762 shutdown_connections 2024-03-16T11:48:20.623 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:20.621+0000 7fb952b10640 1 -- 172.21.15.29:0/2384078762 wait complete. 2024-03-16T11:48:20.623 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:20.622+0000 7fb952b10640 1 Processor -- start 2024-03-16T11:48:20.624 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:20.622+0000 7fb952b10640 1 -- start start 2024-03-16T11:48:20.624 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:20.622+0000 7fb952b10640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb94c0ffad0 0x7fb94c197ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:48:20.624 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:20.622+0000 7fb952b10640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb94c102aa0 0x7fb94c1980e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:48:20.624 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:20.622+0000 7fb952b10640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb94c103460 0x7fb94c19f150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:48:20.624 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:20.622+0000 7fb952b10640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fb94c10d4a0 con 0x7fb94c102aa0 2024-03-16T11:48:20.624 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:20.622+0000 7fb952b10640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fb94c10d320 con 0x7fb94c0ffad0 2024-03-16T11:48:20.624 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:20.622+0000 7fb952b10640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fb94c10d620 con 0x7fb94c103460 2024-03-16T11:48:20.624 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:20.622+0000 7fb94bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb94c102aa0 0x7fb94c1980e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:48:20.624 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:20.622+0000 7fb94bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb94c102aa0 0x7fb94c1980e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45944/0 (socket says 172.21.15.29:45944) 2024-03-16T11:48:20.624 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:20.622+0000 7fb94bfff640 1 -- 172.21.15.29:0/3887178807 learned_addr learned my addr 172.21.15.29:0/3887178807 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:48:20.624 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:20.622+0000 7fb9510ad640 1 --2- 172.21.15.29:0/3887178807 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb94c103460 0x7fb94c19f150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:48:20.624 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:20.622+0000 7fb9508ac640 1 --2- 172.21.15.29:0/3887178807 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb94c0ffad0 0x7fb94c197ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:48:20.624 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:20.622+0000 7fb94bfff640 1 -- 172.21.15.29:0/3887178807 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb94c103460 msgr2=0x7fb94c19f150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:48:20.624 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:20.622+0000 7fb94bfff640 1 --2- 172.21.15.29:0/3887178807 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb94c103460 0x7fb94c19f150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:20.625 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:20.622+0000 7fb94bfff640 1 -- 172.21.15.29:0/3887178807 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb94c0ffad0 msgr2=0x7fb94c197ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:48:20.625 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:20.622+0000 7fb94bfff640 1 --2- 172.21.15.29:0/3887178807 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb94c0ffad0 0x7fb94c197ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:20.625 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:20.622+0000 7fb94bfff640 1 -- 172.21.15.29:0/3887178807 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb94c19f7c0 con 0x7fb94c102aa0 2024-03-16T11:48:20.625 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:20.622+0000 7fb94bfff640 1 --2- 172.21.15.29:0/3887178807 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb94c102aa0 0x7fb94c1980e0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fb93400d770 tx=0x7fb93400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:48:20.625 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:20.623+0000 7fb949ffb640 1 -- 172.21.15.29:0/3887178807 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb934017070 con 0x7fb94c102aa0 2024-03-16T11:48:20.625 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:20.623+0000 7fb949ffb640 1 -- 172.21.15.29:0/3887178807 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb934005500 con 0x7fb94c102aa0 2024-03-16T11:48:20.625 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:20.623+0000 7fb952b10640 1 -- 172.21.15.29:0/3887178807 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb94c19fab0 con 0x7fb94c102aa0 2024-03-16T11:48:20.625 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:20.623+0000 7fb949ffb640 1 -- 172.21.15.29:0/3887178807 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb934004e00 con 0x7fb94c102aa0 2024-03-16T11:48:20.625 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:20.623+0000 7fb952b10640 1 -- 172.21.15.29:0/3887178807 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb94c19ffb0 con 0x7fb94c102aa0 2024-03-16T11:48:20.626 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:20.624+0000 7fb952b10640 1 -- 172.21.15.29:0/3887178807 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb94c108c40 con 0x7fb94c102aa0 2024-03-16T11:48:20.626 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:20.625+0000 7fb949ffb640 1 -- 172.21.15.29:0/3887178807 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fb934004fa0 con 0x7fb94c102aa0 2024-03-16T11:48:20.626 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:20.626+0000 7fb949ffb640 1 --2- 172.21.15.29:0/3887178807 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fb9380a2800 0x7fb9380a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:48:20.626 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:20.626+0000 7fb9508ac640 1 --2- 172.21.15.29:0/3887178807 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fb9380a2800 0x7fb9380a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:48:20.626 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:20.626+0000 7fb949ffb640 1 -- 172.21.15.29:0/3887178807 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fb9340cdf50 con 0x7fb94c102aa0 2024-03-16T11:48:20.626 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:20.626+0000 7fb9508ac640 1 --2- 172.21.15.29:0/3887178807 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fb9380a2800 0x7fb9380a4cc0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fb94402c700 tx=0x7fb944032590 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:48:20.629 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:20.630+0000 7fb949ffb640 1 -- 172.21.15.29:0/3887178807 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fb93409ae40 con 0x7fb94c102aa0 2024-03-16T11:48:20.852 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:20.853+0000 7fb952b10640 1 -- 172.21.15.29:0/3887178807 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb94c02d360 con 0x7fb94c102aa0 2024-03-16T11:48:20.853 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:20.854+0000 7fb949ffb640 1 -- 172.21.15.29:0/3887178807 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+6358 (secure 0 0 0) 0x7fb93409eee0 con 0x7fb94c102aa0 2024-03-16T11:48:20.853 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-16T11:48:20.853 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":8,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14574,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/3505811955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":3505811955},{"type":"v1","addr":"172.21.15.29:6835","nonce":3505811955}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14586,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/1901337868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":1901337868},{"type":"v1","addr":"172.21.15.29:6837","nonce":1901337868}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24407,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6833/3136024628","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6832","nonce":3136024628},{"type":"v1","addr":"172.21.15.152:6833","nonce":3136024628}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24421,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6833/2677426282","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":2677426282},{"type":"v1","addr":"172.21.15.154:6833","nonce":2677426282}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24434,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6835/2919113178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6834","nonce":2919113178},{"type":"v1","addr":"172.21.15.152:6835","nonce":2919113178}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24466,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/4165274681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":4165274681},{"type":"v1","addr":"172.21.15.154:6835","nonce":4165274681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24478,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6837/1468560096","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":1468560096},{"type":"v1","addr":"172.21.15.154:6837","nonce":1468560096}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[]} 2024-03-16T11:48:20.853 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 8 2024-03-16T11:48:20.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:20.855+0000 7fb952b10640 1 -- 172.21.15.29:0/3887178807 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fb9380a2800 msgr2=0x7fb9380a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:48:20.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:20.855+0000 7fb952b10640 1 --2- 172.21.15.29:0/3887178807 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fb9380a2800 0x7fb9380a4cc0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fb94402c700 tx=0x7fb944032590 comp rx=0 tx=0).stop 2024-03-16T11:48:20.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:20.856+0000 7fb952b10640 1 -- 172.21.15.29:0/3887178807 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb94c102aa0 msgr2=0x7fb94c1980e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:48:20.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:20.856+0000 7fb952b10640 1 --2- 172.21.15.29:0/3887178807 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb94c102aa0 0x7fb94c1980e0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fb93400d770 tx=0x7fb93400dc30 comp rx=0 tx=0).stop 2024-03-16T11:48:20.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:20.856+0000 7fb952b10640 1 -- 172.21.15.29:0/3887178807 shutdown_connections 2024-03-16T11:48:20.856 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:20.856+0000 7fb952b10640 1 --2- 172.21.15.29:0/3887178807 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fb9380a2800 0x7fb9380a4cc0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:20.856 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:20.856+0000 7fb952b10640 1 --2- 172.21.15.29:0/3887178807 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb94c103460 0x7fb94c19f150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:20.856 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:20.856+0000 7fb952b10640 1 --2- 172.21.15.29:0/3887178807 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb94c102aa0 0x7fb94c1980e0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:20.856 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:20.856+0000 7fb952b10640 1 --2- 172.21.15.29:0/3887178807 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb94c0ffad0 0x7fb94c197ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:20.856 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:20.856+0000 7fb952b10640 1 -- 172.21.15.29:0/3887178807 >> 172.21.15.29:0/3887178807 conn(0x7fb94c0fb400 msgr2=0x7fb94c105e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:48:20.856 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:20.856+0000 7fb952b10640 1 -- 172.21.15.29:0/3887178807 shutdown_connections 2024-03-16T11:48:20.856 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:20.856+0000 7fb952b10640 1 -- 172.21.15.29:0/3887178807 wait complete. 2024-03-16T11:48:21.337 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:21 smithi029 ceph-mon[31975]: from='client.? 172.21.15.29:0/3887178807' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T11:48:21.378 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:21 smithi152 ceph-mon[36291]: from='client.? 172.21.15.29:0/3887178807' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T11:48:21.378 INFO:journalctl@ceph.mds.h.smithi152.stdout:Mar 16 11:48:21 smithi152 systemd[1]: Starting Ceph mds.h for d1887f28-e389-11ee-95c9-87774f69a715... 2024-03-16T11:48:21.545 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:21 smithi154 ceph-mon[36320]: from='client.? 172.21.15.29:0/3887178807' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T11:48:21.789 INFO:journalctl@ceph.mds.h.smithi152.stdout:Mar 16 11:48:21 smithi152 podman[66915]: 2024-03-16 11:48:21.415196706 +0000 UTC m=+0.018029085 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-16T11:48:21.789 INFO:journalctl@ceph.mds.h.smithi152.stdout:Mar 16 11:48:21 smithi152 podman[66915]: 2024-03-16 11:48:21.537178897 +0000 UTC m=+0.140011262 container create 873b7794f4d43f8f59db7a9c4cad30c1cb499bf11c72872ce483c5fb1538315a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d1887f28-e389-11ee-95c9-87774f69a715-mds-h, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-16T11:48:21.789 INFO:journalctl@ceph.mds.h.smithi152.stdout:Mar 16 11:48:21 smithi152 podman[66915]: 2024-03-16 11:48:21.790243651 +0000 UTC m=+0.393076025 container init 873b7794f4d43f8f59db7a9c4cad30c1cb499bf11c72872ce483c5fb1538315a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d1887f28-e389-11ee-95c9-87774f69a715-mds-h, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-03-16T11:48:22.117 INFO:journalctl@ceph.mds.h.smithi152.stdout:Mar 16 11:48:21 smithi152 podman[66915]: 2024-03-16 11:48:21.792827445 +0000 UTC m=+0.395659803 container start 873b7794f4d43f8f59db7a9c4cad30c1cb499bf11c72872ce483c5fb1538315a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d1887f28-e389-11ee-95c9-87774f69a715-mds-h, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_CLEAN=True, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-16T11:48:22.117 INFO:journalctl@ceph.mds.h.smithi152.stdout:Mar 16 11:48:21 smithi152 ceph-mds[66936]: -- 172.21.15.152:0/1959432081 <== mon.2 v2:172.21.15.152:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x56244c3efa40 con 0x56244c3c2c00 2024-03-16T11:48:22.117 INFO:journalctl@ceph.mds.h.smithi152.stdout:Mar 16 11:48:21 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mds-h[66932]: starting mds.h at 2024-03-16T11:48:22.117 INFO:journalctl@ceph.mds.h.smithi152.stdout:Mar 16 11:48:21 smithi152 bash[66915]: 873b7794f4d43f8f59db7a9c4cad30c1cb499bf11c72872ce483c5fb1538315a 2024-03-16T11:48:22.117 INFO:journalctl@ceph.mds.h.smithi152.stdout:Mar 16 11:48:21 smithi152 systemd[1]: Started Ceph mds.h for d1887f28-e389-11ee-95c9-87774f69a715. 2024-03-16T11:48:22.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:22 smithi152 ceph-mon[36291]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T11:48:22.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:22 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:48:22.378 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:22 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:48:22.378 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:22 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:48:22.378 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:22 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-16T11:48:22.378 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:22 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-16T11:48:22.378 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:22 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:48:22.545 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:22 smithi154 ceph-mon[36320]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T11:48:22.545 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:22 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:48:22.545 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:22 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:48:22.545 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:22 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:48:22.545 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:22 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-16T11:48:22.546 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:22 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-16T11:48:22.546 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:22 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:48:22.588 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:22 smithi029 ceph-mon[31975]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T11:48:22.588 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:22 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:48:22.588 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:22 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:48:22.588 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:22 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:48:22.588 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:22 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-16T11:48:22.588 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:22 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-16T11:48:22.588 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:22 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:48:22.867 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T11:48:22.984 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:22.982+0000 7fa183c4b640 1 -- 172.21.15.29:0/1822165173 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa17c1023a0 msgr2=0x7fa17c104790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:48:22.984 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:22.982+0000 7fa183c4b640 1 --2- 172.21.15.29:0/1822165173 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa17c1023a0 0x7fa17c104790 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fa178009900 tx=0x7fa17802bc30 comp rx=0 tx=0).stop 2024-03-16T11:48:22.985 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:22.982+0000 7fa183c4b640 1 -- 172.21.15.29:0/1822165173 shutdown_connections 2024-03-16T11:48:22.985 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:22.982+0000 7fa183c4b640 1 --2- 172.21.15.29:0/1822165173 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa17c107600 0x7fa17c1099f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:22.985 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:22.982+0000 7fa183c4b640 1 --2- 172.21.15.29:0/1822165173 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa17c104cd0 0x7fa17c1070c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:22.985 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:22.982+0000 7fa183c4b640 1 --2- 172.21.15.29:0/1822165173 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa17c1023a0 0x7fa17c104790 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:22.985 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:22.982+0000 7fa183c4b640 1 -- 172.21.15.29:0/1822165173 >> 172.21.15.29:0/1822165173 conn(0x7fa17c0fbd10 msgr2=0x7fa17c0fe130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:48:22.985 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:22.982+0000 7fa183c4b640 1 -- 172.21.15.29:0/1822165173 shutdown_connections 2024-03-16T11:48:22.985 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:22.982+0000 7fa183c4b640 1 -- 172.21.15.29:0/1822165173 wait complete. 2024-03-16T11:48:22.985 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:22.983+0000 7fa183c4b640 1 Processor -- start 2024-03-16T11:48:22.985 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:22.983+0000 7fa183c4b640 1 -- start start 2024-03-16T11:48:22.985 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:22.983+0000 7fa183c4b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa17c1023a0 0x7fa17c197bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:48:22.985 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:22.983+0000 7fa183c4b640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa17c104cd0 0x7fa17c1980f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:48:22.985 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:22.983+0000 7fa183c4b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa17c107600 0x7fa17c19f160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:48:22.985 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:22.983+0000 7fa183c4b640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fa17c10d500 con 0x7fa17c1023a0 2024-03-16T11:48:22.985 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:22.983+0000 7fa183c4b640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fa17c10d380 con 0x7fa17c104cd0 2024-03-16T11:48:22.986 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:22.983+0000 7fa183c4b640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fa17c10d680 con 0x7fa17c107600 2024-03-16T11:48:22.986 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:22.983+0000 7fa1819e7640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa17c1023a0 0x7fa17c197bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:48:22.986 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:22.983+0000 7fa1819e7640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa17c1023a0 0x7fa17c197bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45950/0 (socket says 172.21.15.29:45950) 2024-03-16T11:48:22.986 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:22.983+0000 7fa1819e7640 1 -- 172.21.15.29:0/440296 learned_addr learned my addr 172.21.15.29:0/440296 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:48:22.986 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:22.983+0000 7fa1821e8640 1 --2- 172.21.15.29:0/440296 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa17c107600 0x7fa17c19f160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:48:22.986 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:22.983+0000 7fa1811e6640 1 --2- 172.21.15.29:0/440296 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa17c104cd0 0x7fa17c1980f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:48:22.986 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:22.983+0000 7fa1819e7640 1 -- 172.21.15.29:0/440296 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa17c107600 msgr2=0x7fa17c19f160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:48:22.986 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:22.983+0000 7fa1819e7640 1 --2- 172.21.15.29:0/440296 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa17c107600 0x7fa17c19f160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:22.986 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:22.983+0000 7fa1819e7640 1 -- 172.21.15.29:0/440296 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa17c104cd0 msgr2=0x7fa17c1980f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:48:22.986 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:22.983+0000 7fa1819e7640 1 --2- 172.21.15.29:0/440296 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa17c104cd0 0x7fa17c1980f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:22.986 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:22.983+0000 7fa1819e7640 1 -- 172.21.15.29:0/440296 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa17c19f7d0 con 0x7fa17c1023a0 2024-03-16T11:48:22.986 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:22.983+0000 7fa1821e8640 1 --2- 172.21.15.29:0/440296 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa17c107600 0x7fa17c19f160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T11:48:22.986 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:22.984+0000 7fa1819e7640 1 --2- 172.21.15.29:0/440296 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa17c1023a0 0x7fa17c197bb0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fa178009230 tx=0x7fa178005dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:48:22.986 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:22.984+0000 7fa172ffd640 1 -- 172.21.15.29:0/440296 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa178041070 con 0x7fa17c1023a0 2024-03-16T11:48:22.986 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:22.984+0000 7fa172ffd640 1 -- 172.21.15.29:0/440296 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa1780054e0 con 0x7fa17c1023a0 2024-03-16T11:48:22.987 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:22.984+0000 7fa183c4b640 1 -- 172.21.15.29:0/440296 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa17c19fa60 con 0x7fa17c1023a0 2024-03-16T11:48:22.987 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:22.984+0000 7fa172ffd640 1 -- 172.21.15.29:0/440296 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa178034440 con 0x7fa17c1023a0 2024-03-16T11:48:22.987 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:22.984+0000 7fa183c4b640 1 -- 172.21.15.29:0/440296 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa17c19ff50 con 0x7fa17c1023a0 2024-03-16T11:48:22.987 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:22.985+0000 7fa183c4b640 1 -- 172.21.15.29:0/440296 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa17c0fd800 con 0x7fa17c1023a0 2024-03-16T11:48:22.987 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:22.986+0000 7fa172ffd640 1 -- 172.21.15.29:0/440296 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fa178044070 con 0x7fa17c1023a0 2024-03-16T11:48:22.988 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:22.986+0000 7fa172ffd640 1 --2- 172.21.15.29:0/440296 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fa1680a2800 0x7fa1680a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:48:22.988 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:22.986+0000 7fa1811e6640 1 --2- 172.21.15.29:0/440296 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fa1680a2800 0x7fa1680a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:48:22.988 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:22.986+0000 7fa172ffd640 1 -- 172.21.15.29:0/440296 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fa1780f0370 con 0x7fa17c1023a0 2024-03-16T11:48:22.988 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:22.987+0000 7fa1811e6640 1 --2- 172.21.15.29:0/440296 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fa1680a2800 0x7fa1680a4cc0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fa17c1990c0 tx=0x7fa16c008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:48:22.990 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:22.991+0000 7fa172ffd640 1 -- 172.21.15.29:0/440296 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fa1780bd8b0 con 0x7fa17c1023a0 2024-03-16T11:48:23.214 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:23.215+0000 7fa183c4b640 1 -- 172.21.15.29:0/440296 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa17c02d1b0 con 0x7fa17c1023a0 2024-03-16T11:48:23.215 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:23.216+0000 7fa172ffd640 1 -- 172.21.15.29:0/440296 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+7186 (secure 0 0 0) 0x7fa1780bd1b0 con 0x7fa17c1023a0 2024-03-16T11:48:23.215 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-16T11:48:23.215 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":9,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14574,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/3505811955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":3505811955},{"type":"v1","addr":"172.21.15.29:6835","nonce":3505811955}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14586,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/1901337868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":1901337868},{"type":"v1","addr":"172.21.15.29:6837","nonce":1901337868}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24407,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6833/3136024628","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6832","nonce":3136024628},{"type":"v1","addr":"172.21.15.152:6833","nonce":3136024628}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24421,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6833/2677426282","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":2677426282},{"type":"v1","addr":"172.21.15.154:6833","nonce":2677426282}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24434,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6835/2919113178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6834","nonce":2919113178},{"type":"v1","addr":"172.21.15.152:6835","nonce":2919113178}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24464,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6837/1323866722","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6836","nonce":1323866722},{"type":"v1","addr":"172.21.15.152:6837","nonce":1323866722}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24466,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/4165274681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":4165274681},{"type":"v1","addr":"172.21.15.154:6835","nonce":4165274681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24478,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6837/1468560096","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":1468560096},{"type":"v1","addr":"172.21.15.154:6837","nonce":1468560096}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[]} 2024-03-16T11:48:23.215 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 9 2024-03-16T11:48:23.217 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:23.218+0000 7fa183c4b640 1 -- 172.21.15.29:0/440296 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fa1680a2800 msgr2=0x7fa1680a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:48:23.217 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:23.218+0000 7fa183c4b640 1 --2- 172.21.15.29:0/440296 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fa1680a2800 0x7fa1680a4cc0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fa17c1990c0 tx=0x7fa16c008040 comp rx=0 tx=0).stop 2024-03-16T11:48:23.217 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:23.218+0000 7fa183c4b640 1 -- 172.21.15.29:0/440296 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa17c1023a0 msgr2=0x7fa17c197bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:48:23.217 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:23.218+0000 7fa183c4b640 1 --2- 172.21.15.29:0/440296 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa17c1023a0 0x7fa17c197bb0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fa178009230 tx=0x7fa178005dd0 comp rx=0 tx=0).stop 2024-03-16T11:48:23.217 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:23.218+0000 7fa183c4b640 1 -- 172.21.15.29:0/440296 shutdown_connections 2024-03-16T11:48:23.217 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:23.218+0000 7fa183c4b640 1 --2- 172.21.15.29:0/440296 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fa1680a2800 0x7fa1680a4cc0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:23.218 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:23.218+0000 7fa183c4b640 1 --2- 172.21.15.29:0/440296 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa17c107600 0x7fa17c19f160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:23.218 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:23.218+0000 7fa183c4b640 1 --2- 172.21.15.29:0/440296 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa17c104cd0 0x7fa17c1980f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:23.218 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:23.218+0000 7fa183c4b640 1 --2- 172.21.15.29:0/440296 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa17c1023a0 0x7fa17c197bb0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:23.218 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:23.218+0000 7fa183c4b640 1 -- 172.21.15.29:0/440296 >> 172.21.15.29:0/440296 conn(0x7fa17c0fbd10 msgr2=0x7fa17c102dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:48:23.218 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:23.218+0000 7fa183c4b640 1 -- 172.21.15.29:0/440296 shutdown_connections 2024-03-16T11:48:23.218 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:23.218+0000 7fa183c4b640 1 -- 172.21.15.29:0/440296 wait complete. 2024-03-16T11:48:23.226 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:23 smithi029 ceph-mon[31975]: Deploying daemon mds.i on smithi154 2024-03-16T11:48:23.227 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:23 smithi029 ceph-mon[31975]: mds.? [v2:172.21.15.152:6836/1323866722,v1:172.21.15.152:6837/1323866722] up:boot 2024-03-16T11:48:23.227 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:23 smithi029 ceph-mon[31975]: fsmap 8 up:standby 2024-03-16T11:48:23.227 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:23 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-16T11:48:23.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:23 smithi154 ceph-mon[36320]: Deploying daemon mds.i on smithi154 2024-03-16T11:48:23.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:23 smithi154 ceph-mon[36320]: mds.? [v2:172.21.15.152:6836/1323866722,v1:172.21.15.152:6837/1323866722] up:boot 2024-03-16T11:48:23.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:23 smithi154 ceph-mon[36320]: fsmap 8 up:standby 2024-03-16T11:48:23.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:23 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-16T11:48:23.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:23 smithi152 ceph-mon[36291]: Deploying daemon mds.i on smithi154 2024-03-16T11:48:23.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:23 smithi152 ceph-mon[36291]: mds.? [v2:172.21.15.152:6836/1323866722,v1:172.21.15.152:6837/1323866722] up:boot 2024-03-16T11:48:23.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:23 smithi152 ceph-mon[36291]: fsmap 8 up:standby 2024-03-16T11:48:23.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:23 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-16T11:48:24.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:24 smithi154 ceph-mon[36320]: from='client.? 172.21.15.29:0/440296' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T11:48:24.296 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:24 smithi154 ceph-mon[36320]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T11:48:24.296 INFO:journalctl@ceph.mds.i.smithi154.stdout:Mar 16 11:48:24 smithi154 systemd[1]: Starting Ceph mds.i for d1887f28-e389-11ee-95c9-87774f69a715... 2024-03-16T11:48:24.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:24 smithi152 ceph-mon[36291]: from='client.? 172.21.15.29:0/440296' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T11:48:24.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:24 smithi152 ceph-mon[36291]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T11:48:24.588 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:24 smithi029 ceph-mon[31975]: from='client.? 172.21.15.29:0/440296' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T11:48:24.588 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:24 smithi029 ceph-mon[31975]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T11:48:24.643 INFO:journalctl@ceph.mds.i.smithi154.stdout:Mar 16 11:48:24 smithi154 podman[67016]: 2024-03-16 11:48:24.274193869 +0000 UTC m=+0.017946209 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-16T11:48:24.643 INFO:journalctl@ceph.mds.i.smithi154.stdout:Mar 16 11:48:24 smithi154 podman[67016]: 2024-03-16 11:48:24.382638662 +0000 UTC m=+0.126390995 container create 31362b9e990cfc058e042686dd711c139329910e217a0f0b652db955ba522b8e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d1887f28-e389-11ee-95c9-87774f69a715-mds-i, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-16T11:48:24.643 INFO:journalctl@ceph.mds.i.smithi154.stdout:Mar 16 11:48:24 smithi154 podman[67016]: 2024-03-16 11:48:24.618765464 +0000 UTC m=+0.362517808 container init 31362b9e990cfc058e042686dd711c139329910e217a0f0b652db955ba522b8e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d1887f28-e389-11ee-95c9-87774f69a715-mds-i, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-03-16T11:48:24.643 INFO:journalctl@ceph.mds.i.smithi154.stdout:Mar 16 11:48:24 smithi154 podman[67016]: 2024-03-16 11:48:24.621381379 +0000 UTC m=+0.365133719 container start 31362b9e990cfc058e042686dd711c139329910e217a0f0b652db955ba522b8e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d1887f28-e389-11ee-95c9-87774f69a715-mds-i, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.vendor=CentOS) 2024-03-16T11:48:24.643 INFO:journalctl@ceph.mds.i.smithi154.stdout:Mar 16 11:48:24 smithi154 ceph-mds[67035]: -- 172.21.15.154:0/1175680206 <== mon.1 v2:172.21.15.154:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55c639f95a40 con 0x55c639f68c00 2024-03-16T11:48:25.045 INFO:journalctl@ceph.mds.i.smithi154.stdout:Mar 16 11:48:24 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mds-i[67031]: starting mds.i at 2024-03-16T11:48:25.045 INFO:journalctl@ceph.mds.i.smithi154.stdout:Mar 16 11:48:24 smithi154 bash[67016]: 31362b9e990cfc058e042686dd711c139329910e217a0f0b652db955ba522b8e 2024-03-16T11:48:25.045 INFO:journalctl@ceph.mds.i.smithi154.stdout:Mar 16 11:48:24 smithi154 systemd[1]: Started Ceph mds.i for d1887f28-e389-11ee-95c9-87774f69a715. 2024-03-16T11:48:25.228 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T11:48:25.345 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:25.343+0000 7f8213c80640 1 -- 172.21.15.29:0/840900480 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f820c100700 msgr2=0x7f820c100b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:48:25.345 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:25.343+0000 7f8213c80640 1 --2- 172.21.15.29:0/840900480 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f820c100700 0x7f820c100b20 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f8208009900 tx=0x7f820802bc30 comp rx=0 tx=0).stop 2024-03-16T11:48:25.345 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:25.344+0000 7f8213c80640 1 -- 172.21.15.29:0/840900480 shutdown_connections 2024-03-16T11:48:25.345 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:25.344+0000 7f8213c80640 1 --2- 172.21.15.29:0/840900480 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f820c1022b0 0x7f820c108b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:25.346 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:25.344+0000 7f8213c80640 1 --2- 172.21.15.29:0/840900480 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f820c1018f0 0x7f820c101d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:25.346 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:25.344+0000 7f8213c80640 1 --2- 172.21.15.29:0/840900480 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f820c100700 0x7f820c100b20 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:25.346 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:25.344+0000 7f8213c80640 1 -- 172.21.15.29:0/840900480 >> 172.21.15.29:0/840900480 conn(0x7f820c0fbc90 msgr2=0x7f820c0fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:48:25.346 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:25.344+0000 7f8213c80640 1 -- 172.21.15.29:0/840900480 shutdown_connections 2024-03-16T11:48:25.346 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:25.344+0000 7f8213c80640 1 -- 172.21.15.29:0/840900480 wait complete. 2024-03-16T11:48:25.346 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:25.344+0000 7f8213c80640 1 Processor -- start 2024-03-16T11:48:25.346 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:25.344+0000 7f8213c80640 1 -- start start 2024-03-16T11:48:25.346 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:25.344+0000 7f8213c80640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f820c100700 0x7f820c195900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:48:25.346 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:25.344+0000 7f8213c80640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f820c1018f0 0x7f820c195e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:48:25.346 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:25.344+0000 7f8213c80640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f820c1022b0 0x7f820c19ceb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:48:25.346 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:25.344+0000 7f8213c80640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f820c10b280 con 0x7f820c1022b0 2024-03-16T11:48:25.346 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:25.344+0000 7f8213c80640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f820c10b100 con 0x7f820c100700 2024-03-16T11:48:25.346 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:25.345+0000 7f8213c80640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f820c10b400 con 0x7f820c1018f0 2024-03-16T11:48:25.347 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:25.345+0000 7f821221d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f820c1022b0 0x7f820c19ceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:48:25.347 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:25.345+0000 7f821221d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f820c1022b0 0x7f820c19ceb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45970/0 (socket says 172.21.15.29:45970) 2024-03-16T11:48:25.347 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:25.345+0000 7f821221d640 1 -- 172.21.15.29:0/3441627266 learned_addr learned my addr 172.21.15.29:0/3441627266 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:48:25.347 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:25.345+0000 7f821121b640 1 --2- 172.21.15.29:0/3441627266 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f820c1018f0 0x7f820c195e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:48:25.347 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:25.345+0000 7f8211a1c640 1 --2- 172.21.15.29:0/3441627266 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f820c100700 0x7f820c195900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:48:25.347 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:25.345+0000 7f821221d640 1 -- 172.21.15.29:0/3441627266 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f820c1018f0 msgr2=0x7f820c195e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:48:25.347 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:25.345+0000 7f821221d640 1 --2- 172.21.15.29:0/3441627266 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f820c1018f0 0x7f820c195e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:25.347 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:25.345+0000 7f821221d640 1 -- 172.21.15.29:0/3441627266 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f820c100700 msgr2=0x7f820c195900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:48:25.347 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:25.345+0000 7f821221d640 1 --2- 172.21.15.29:0/3441627266 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f820c100700 0x7f820c195900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:25.347 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:25.345+0000 7f821221d640 1 -- 172.21.15.29:0/3441627266 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f820c19d520 con 0x7f820c1022b0 2024-03-16T11:48:25.347 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:25.345+0000 7f821221d640 1 --2- 172.21.15.29:0/3441627266 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f820c1022b0 0x7f820c19ceb0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f820400eb10 tx=0x7f820400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:48:25.347 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:25.345+0000 7f8202ffd640 1 -- 172.21.15.29:0/3441627266 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8204004b20 con 0x7f820c1022b0 2024-03-16T11:48:25.347 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:25.345+0000 7f8213c80640 1 -- 172.21.15.29:0/3441627266 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f820c19d810 con 0x7f820c1022b0 2024-03-16T11:48:25.347 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:25.345+0000 7f8202ffd640 1 -- 172.21.15.29:0/3441627266 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8204004cc0 con 0x7f820c1022b0 2024-03-16T11:48:25.348 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:25.345+0000 7f8202ffd640 1 -- 172.21.15.29:0/3441627266 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8204010760 con 0x7f820c1022b0 2024-03-16T11:48:25.348 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:25.345+0000 7f8213c80640 1 -- 172.21.15.29:0/3441627266 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f820c19dd60 con 0x7f820c1022b0 2024-03-16T11:48:25.348 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:25.346+0000 7f8213c80640 1 -- 172.21.15.29:0/3441627266 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f820c107a20 con 0x7f820c1022b0 2024-03-16T11:48:25.348 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:25.348+0000 7f8202ffd640 1 -- 172.21.15.29:0/3441627266 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f8204005000 con 0x7f820c1022b0 2024-03-16T11:48:25.349 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:25.348+0000 7f8202ffd640 1 --2- 172.21.15.29:0/3441627266 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f81f80a4a10 0x7f81f80a6ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:48:25.349 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:25.348+0000 7f8211a1c640 1 --2- 172.21.15.29:0/3441627266 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f81f80a4a10 0x7f81f80a6ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:48:25.349 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:25.348+0000 7f8202ffd640 1 -- 172.21.15.29:0/3441627266 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f82040cee10 con 0x7f820c1022b0 2024-03-16T11:48:25.349 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:25.349+0000 7f8211a1c640 1 --2- 172.21.15.29:0/3441627266 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f81f80a4a10 0x7f81f80a6ed0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f820802c7b0 tx=0x7f8208005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:48:25.355 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:25.356+0000 7f8202ffd640 1 -- 172.21.15.29:0/3441627266 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f8204053740 con 0x7f820c1022b0 2024-03-16T11:48:25.585 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:25.585+0000 7f8213c80640 1 -- 172.21.15.29:0/3441627266 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f820c02d360 con 0x7f820c1022b0 2024-03-16T11:48:25.585 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:25.586+0000 7f8202ffd640 1 -- 172.21.15.29:0/3441627266 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) ==== 94+0+8013 (secure 0 0 0) 0x7f820409c350 con 0x7f820c1022b0 2024-03-16T11:48:25.585 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-16T11:48:25.585 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":10,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14574,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/3505811955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":3505811955},{"type":"v1","addr":"172.21.15.29:6835","nonce":3505811955}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14586,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/1901337868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":1901337868},{"type":"v1","addr":"172.21.15.29:6837","nonce":1901337868}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24407,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6833/3136024628","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6832","nonce":3136024628},{"type":"v1","addr":"172.21.15.152:6833","nonce":3136024628}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24421,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6833/2677426282","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":2677426282},{"type":"v1","addr":"172.21.15.154:6833","nonce":2677426282}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24434,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6835/2919113178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6834","nonce":2919113178},{"type":"v1","addr":"172.21.15.152:6835","nonce":2919113178}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24464,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6837/1323866722","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6836","nonce":1323866722},{"type":"v1","addr":"172.21.15.152:6837","nonce":1323866722}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24466,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/4165274681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":4165274681},{"type":"v1","addr":"172.21.15.154:6835","nonce":4165274681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24478,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6837/1468560096","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":1468560096},{"type":"v1","addr":"172.21.15.154:6837","nonce":1468560096}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24499,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/685736862","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":685736862},{"type":"v1","addr":"172.21.15.154:6839","nonce":685736862}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-03-16T11:48:25.586 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 10 2024-03-16T11:48:25.587 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:25.588+0000 7f8213c80640 1 -- 172.21.15.29:0/3441627266 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f81f80a4a10 msgr2=0x7f81f80a6ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:48:25.587 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:25.588+0000 7f8213c80640 1 --2- 172.21.15.29:0/3441627266 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f81f80a4a10 0x7f81f80a6ed0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f820802c7b0 tx=0x7f8208005fd0 comp rx=0 tx=0).stop 2024-03-16T11:48:25.587 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:25.588+0000 7f8213c80640 1 -- 172.21.15.29:0/3441627266 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f820c1022b0 msgr2=0x7f820c19ceb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:48:25.587 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:25.588+0000 7f8213c80640 1 --2- 172.21.15.29:0/3441627266 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f820c1022b0 0x7f820c19ceb0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f820400eb10 tx=0x7f820400c2d0 comp rx=0 tx=0).stop 2024-03-16T11:48:25.587 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:25.588+0000 7f8213c80640 1 -- 172.21.15.29:0/3441627266 shutdown_connections 2024-03-16T11:48:25.588 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:25.588+0000 7f8213c80640 1 --2- 172.21.15.29:0/3441627266 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f81f80a4a10 0x7f81f80a6ed0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:25.588 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:25.588+0000 7f8213c80640 1 --2- 172.21.15.29:0/3441627266 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f820c1022b0 0x7f820c19ceb0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:25.588 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:25.588+0000 7f8213c80640 1 --2- 172.21.15.29:0/3441627266 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f820c1018f0 0x7f820c195e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:25.588 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:25.588+0000 7f8213c80640 1 --2- 172.21.15.29:0/3441627266 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f820c100700 0x7f820c195900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:25.588 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:25.588+0000 7f8213c80640 1 -- 172.21.15.29:0/3441627266 >> 172.21.15.29:0/3441627266 conn(0x7f820c0fbc90 msgr2=0x7f820c104bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:48:25.588 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:25.588+0000 7f8213c80640 1 -- 172.21.15.29:0/3441627266 shutdown_connections 2024-03-16T11:48:25.588 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:25.588+0000 7f8213c80640 1 -- 172.21.15.29:0/3441627266 wait complete. 2024-03-16T11:48:26.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:25 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:48:26.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:25 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:48:26.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:25 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:48:26.046 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:25 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-16T11:48:26.046 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:25 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-16T11:48:26.046 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:25 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:48:26.046 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:25 smithi154 ceph-mon[36320]: Deploying daemon mds.g on smithi029 2024-03-16T11:48:26.046 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:25 smithi154 ceph-mon[36320]: mds.? [v2:172.21.15.154:6838/685736862,v1:172.21.15.154:6839/685736862] up:boot 2024-03-16T11:48:26.046 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:25 smithi154 ceph-mon[36320]: fsmap 9 up:standby 2024-03-16T11:48:26.046 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:25 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-16T11:48:26.046 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:25 smithi154 ceph-mon[36320]: from='client.? 172.21.15.29:0/3441627266' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T11:48:26.046 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:25 smithi154 ceph-mon[36320]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T11:48:26.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:25 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:48:26.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:25 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:48:26.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:25 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:48:26.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:25 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-16T11:48:26.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:25 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-16T11:48:26.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:25 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:48:26.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:25 smithi029 ceph-mon[31975]: Deploying daemon mds.g on smithi029 2024-03-16T11:48:26.089 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:25 smithi029 ceph-mon[31975]: mds.? [v2:172.21.15.154:6838/685736862,v1:172.21.15.154:6839/685736862] up:boot 2024-03-16T11:48:26.089 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:25 smithi029 ceph-mon[31975]: fsmap 9 up:standby 2024-03-16T11:48:26.089 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:25 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-16T11:48:26.089 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:25 smithi029 ceph-mon[31975]: from='client.? 172.21.15.29:0/3441627266' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T11:48:26.089 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:25 smithi029 ceph-mon[31975]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T11:48:26.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:25 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:48:26.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:25 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:48:26.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:25 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:48:26.128 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:25 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-16T11:48:26.128 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:25 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-16T11:48:26.128 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:25 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:48:26.128 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:25 smithi152 ceph-mon[36291]: Deploying daemon mds.g on smithi029 2024-03-16T11:48:26.128 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:25 smithi152 ceph-mon[36291]: mds.? [v2:172.21.15.154:6838/685736862,v1:172.21.15.154:6839/685736862] up:boot 2024-03-16T11:48:26.128 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:25 smithi152 ceph-mon[36291]: fsmap 9 up:standby 2024-03-16T11:48:26.128 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:25 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-16T11:48:26.128 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:25 smithi152 ceph-mon[36291]: from='client.? 172.21.15.29:0/3441627266' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T11:48:26.128 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:25 smithi152 ceph-mon[36291]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T11:48:27.521 INFO:journalctl@ceph.mds.g.smithi029.stdout:Mar 16 11:48:27 smithi029 systemd[1]: Starting Ceph mds.g for d1887f28-e389-11ee-95c9-87774f69a715... 2024-03-16T11:48:27.598 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T11:48:27.708 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:27.708+0000 7fbe1ab3b640 1 -- 172.21.15.29:0/1941575749 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbe14102340 msgr2=0x7fbe14104730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:48:27.708 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:27.708+0000 7fbe1ab3b640 1 --2- 172.21.15.29:0/1941575749 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbe14102340 0x7fbe14104730 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fbe0c009ca0 tx=0x7fbe0c02c0f0 comp rx=0 tx=0).stop 2024-03-16T11:48:27.708 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:27.708+0000 7fbe1ab3b640 1 -- 172.21.15.29:0/1941575749 shutdown_connections 2024-03-16T11:48:27.708 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:27.708+0000 7fbe1ab3b640 1 --2- 172.21.15.29:0/1941575749 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbe141075a0 0x7fbe14109990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:27.708 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:27.708+0000 7fbe1ab3b640 1 --2- 172.21.15.29:0/1941575749 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbe14104c70 0x7fbe14107060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:27.708 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:27.708+0000 7fbe1ab3b640 1 --2- 172.21.15.29:0/1941575749 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbe14102340 0x7fbe14104730 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:27.708 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:27.708+0000 7fbe1ab3b640 1 -- 172.21.15.29:0/1941575749 >> 172.21.15.29:0/1941575749 conn(0x7fbe140fbc90 msgr2=0x7fbe140fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:48:27.708 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:27.708+0000 7fbe1ab3b640 1 -- 172.21.15.29:0/1941575749 shutdown_connections 2024-03-16T11:48:27.708 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:27.708+0000 7fbe1ab3b640 1 -- 172.21.15.29:0/1941575749 wait complete. 2024-03-16T11:48:27.709 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:27.709+0000 7fbe1ab3b640 1 Processor -- start 2024-03-16T11:48:27.709 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:27.709+0000 7fbe1ab3b640 1 -- start start 2024-03-16T11:48:27.709 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:27.709+0000 7fbe1ab3b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbe14102340 0x7fbe14197b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:48:27.709 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:27.709+0000 7fbe1ab3b640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbe14104c70 0x7fbe141980a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:48:27.710 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:27.709+0000 7fbe1ab3b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbe141075a0 0x7fbe1419f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:48:27.710 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:27.709+0000 7fbe1ab3b640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fbe1410d4c0 con 0x7fbe141075a0 2024-03-16T11:48:27.710 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:27.709+0000 7fbe1ab3b640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fbe1410d340 con 0x7fbe14104c70 2024-03-16T11:48:27.710 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:27.709+0000 7fbe1ab3b640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fbe1410d640 con 0x7fbe14102340 2024-03-16T11:48:27.712 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:27.709+0000 7fbe190d8640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbe141075a0 0x7fbe1419f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:48:27.712 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:27.709+0000 7fbe190d8640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbe141075a0 0x7fbe1419f110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45988/0 (socket says 172.21.15.29:45988) 2024-03-16T11:48:27.712 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:27.709+0000 7fbe190d8640 1 -- 172.21.15.29:0/1741036840 learned_addr learned my addr 172.21.15.29:0/1741036840 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:48:27.712 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:27.710+0000 7fbe188d7640 1 --2- 172.21.15.29:0/1741036840 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbe14102340 0x7fbe14197b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:48:27.719 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:27.710+0000 7fbe190d8640 1 -- 172.21.15.29:0/1741036840 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbe14102340 msgr2=0x7fbe14197b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:48:27.719 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:27.710+0000 7fbe190d8640 1 --2- 172.21.15.29:0/1741036840 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbe14102340 0x7fbe14197b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:27.719 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:27.710+0000 7fbe190d8640 1 -- 172.21.15.29:0/1741036840 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbe14104c70 msgr2=0x7fbe141980a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:48:27.719 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:27.710+0000 7fbe190d8640 1 --2- 172.21.15.29:0/1741036840 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbe14104c70 0x7fbe141980a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:27.719 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:27.710+0000 7fbe190d8640 1 -- 172.21.15.29:0/1741036840 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe1419f780 con 0x7fbe141075a0 2024-03-16T11:48:27.720 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:27.710+0000 7fbe190d8640 1 --2- 172.21.15.29:0/1741036840 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbe141075a0 0x7fbe1419f110 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fbdfc00d750 tx=0x7fbdfc00dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:48:27.720 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:27.710+0000 7fbe11ffb640 1 -- 172.21.15.29:0/1741036840 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdfc017070 con 0x7fbe141075a0 2024-03-16T11:48:27.720 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:27.710+0000 7fbe11ffb640 1 -- 172.21.15.29:0/1741036840 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbdfc005500 con 0x7fbe141075a0 2024-03-16T11:48:27.720 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:27.710+0000 7fbe11ffb640 1 -- 172.21.15.29:0/1741036840 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdfc004d90 con 0x7fbe141075a0 2024-03-16T11:48:27.720 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:27.710+0000 7fbe1ab3b640 1 -- 172.21.15.29:0/1741036840 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbe1419fa70 con 0x7fbe141075a0 2024-03-16T11:48:27.720 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:27.710+0000 7fbe1ab3b640 1 -- 172.21.15.29:0/1741036840 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbe1419ff70 con 0x7fbe141075a0 2024-03-16T11:48:27.720 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:27.711+0000 7fbe1ab3b640 1 -- 172.21.15.29:0/1741036840 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbe140fd780 con 0x7fbe141075a0 2024-03-16T11:48:27.720 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:27.712+0000 7fbe11ffb640 1 -- 172.21.15.29:0/1741036840 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fbdfc004f30 con 0x7fbe141075a0 2024-03-16T11:48:27.720 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:27.713+0000 7fbe11ffb640 1 --2- 172.21.15.29:0/1741036840 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fbe000a2800 0x7fbe000a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:48:27.720 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:27.713+0000 7fbe11ffb640 1 -- 172.21.15.29:0/1741036840 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fbdfc0cdfa0 con 0x7fbe141075a0 2024-03-16T11:48:27.720 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:27.713+0000 7fbe188d7640 1 --2- 172.21.15.29:0/1741036840 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fbe000a2800 0x7fbe000a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:48:27.720 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:27.713+0000 7fbe188d7640 1 --2- 172.21.15.29:0/1741036840 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fbe000a2800 0x7fbe000a4cc0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fbe0c006010 tx=0x7fbe0c032590 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:48:27.720 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:27.717+0000 7fbe11ffb640 1 -- 172.21.15.29:0/1741036840 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fbdfc09b4e0 con 0x7fbe141075a0 2024-03-16T11:48:27.823 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:27 smithi029 ceph-mon[31975]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T11:48:27.824 INFO:journalctl@ceph.mds.g.smithi029.stdout:Mar 16 11:48:27 smithi029 podman[73800]: 2024-03-16 11:48:27.425449271 +0000 UTC m=+0.018627746 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-16T11:48:27.824 INFO:journalctl@ceph.mds.g.smithi029.stdout:Mar 16 11:48:27 smithi029 podman[73800]: 2024-03-16 11:48:27.537162395 +0000 UTC m=+0.130340853 container create 37fc43af5b3fae320d6d8fa8c1726415d83f134d94b06a6987d793a746a684d2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d1887f28-e389-11ee-95c9-87774f69a715-mds-g, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-16T11:48:27.824 INFO:journalctl@ceph.mds.g.smithi029.stdout:Mar 16 11:48:27 smithi029 podman[73800]: 2024-03-16 11:48:27.825472204 +0000 UTC m=+0.418650660 container init 37fc43af5b3fae320d6d8fa8c1726415d83f134d94b06a6987d793a746a684d2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d1887f28-e389-11ee-95c9-87774f69a715-mds-g, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5) 2024-03-16T11:48:27.945 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:27.946+0000 7fbe1ab3b640 1 -- 172.21.15.29:0/1741036840 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbe1402d360 con 0x7fbe141075a0 2024-03-16T11:48:27.946 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:27.947+0000 7fbe11ffb640 1 -- 172.21.15.29:0/1741036840 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) ==== 94+0+8013 (secure 0 0 0) 0x7fbdfc09ade0 con 0x7fbe141075a0 2024-03-16T11:48:27.946 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-16T11:48:27.946 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":10,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14574,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/3505811955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":3505811955},{"type":"v1","addr":"172.21.15.29:6835","nonce":3505811955}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14586,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/1901337868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":1901337868},{"type":"v1","addr":"172.21.15.29:6837","nonce":1901337868}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24407,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6833/3136024628","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6832","nonce":3136024628},{"type":"v1","addr":"172.21.15.152:6833","nonce":3136024628}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24421,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6833/2677426282","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":2677426282},{"type":"v1","addr":"172.21.15.154:6833","nonce":2677426282}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24434,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6835/2919113178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6834","nonce":2919113178},{"type":"v1","addr":"172.21.15.152:6835","nonce":2919113178}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24464,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6837/1323866722","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6836","nonce":1323866722},{"type":"v1","addr":"172.21.15.152:6837","nonce":1323866722}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24466,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/4165274681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":4165274681},{"type":"v1","addr":"172.21.15.154:6835","nonce":4165274681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24478,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6837/1468560096","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":1468560096},{"type":"v1","addr":"172.21.15.154:6837","nonce":1468560096}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24499,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/685736862","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":685736862},{"type":"v1","addr":"172.21.15.154:6839","nonce":685736862}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-03-16T11:48:27.947 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 10 2024-03-16T11:48:27.948 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:27.949+0000 7fbe1ab3b640 1 -- 172.21.15.29:0/1741036840 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fbe000a2800 msgr2=0x7fbe000a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:48:27.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:27.949+0000 7fbe1ab3b640 1 --2- 172.21.15.29:0/1741036840 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fbe000a2800 0x7fbe000a4cc0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fbe0c006010 tx=0x7fbe0c032590 comp rx=0 tx=0).stop 2024-03-16T11:48:27.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:27.949+0000 7fbe1ab3b640 1 -- 172.21.15.29:0/1741036840 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbe141075a0 msgr2=0x7fbe1419f110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:48:27.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:27.949+0000 7fbe1ab3b640 1 --2- 172.21.15.29:0/1741036840 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbe141075a0 0x7fbe1419f110 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fbdfc00d750 tx=0x7fbdfc00dc10 comp rx=0 tx=0).stop 2024-03-16T11:48:27.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:27.949+0000 7fbe1ab3b640 1 -- 172.21.15.29:0/1741036840 shutdown_connections 2024-03-16T11:48:27.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:27.949+0000 7fbe1ab3b640 1 --2- 172.21.15.29:0/1741036840 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fbe000a2800 0x7fbe000a4cc0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:27.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:27.949+0000 7fbe1ab3b640 1 --2- 172.21.15.29:0/1741036840 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbe141075a0 0x7fbe1419f110 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:27.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:27.949+0000 7fbe1ab3b640 1 --2- 172.21.15.29:0/1741036840 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbe14104c70 0x7fbe141980a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:27.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:27.949+0000 7fbe1ab3b640 1 --2- 172.21.15.29:0/1741036840 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbe14102340 0x7fbe14197b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:27.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:27.949+0000 7fbe1ab3b640 1 -- 172.21.15.29:0/1741036840 >> 172.21.15.29:0/1741036840 conn(0x7fbe140fbc90 msgr2=0x7fbe14102da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:48:27.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:27.949+0000 7fbe1ab3b640 1 -- 172.21.15.29:0/1741036840 shutdown_connections 2024-03-16T11:48:27.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:27.949+0000 7fbe1ab3b640 1 -- 172.21.15.29:0/1741036840 wait complete. 2024-03-16T11:48:28.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:27 smithi154 ceph-mon[36320]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T11:48:28.088 INFO:journalctl@ceph.mds.g.smithi029.stdout:Mar 16 11:48:27 smithi029 podman[73800]: 2024-03-16 11:48:27.828773114 +0000 UTC m=+0.421951578 container start 37fc43af5b3fae320d6d8fa8c1726415d83f134d94b06a6987d793a746a684d2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d1887f28-e389-11ee-95c9-87774f69a715-mds-g, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.vendor=CentOS) 2024-03-16T11:48:28.088 INFO:journalctl@ceph.mds.g.smithi029.stdout:Mar 16 11:48:27 smithi029 ceph-mds[73859]: -- 172.21.15.29:0/3207892098 <== mon.0 v2:172.21.15.29:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55c0dc171a40 con 0x55c0dc145000 2024-03-16T11:48:28.088 INFO:journalctl@ceph.mds.g.smithi029.stdout:Mar 16 11:48:27 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mds-g[73836]: starting mds.g at 2024-03-16T11:48:28.088 INFO:journalctl@ceph.mds.g.smithi029.stdout:Mar 16 11:48:27 smithi029 bash[73800]: 37fc43af5b3fae320d6d8fa8c1726415d83f134d94b06a6987d793a746a684d2 2024-03-16T11:48:28.088 INFO:journalctl@ceph.mds.g.smithi029.stdout:Mar 16 11:48:27 smithi029 systemd[1]: Started Ceph mds.g for d1887f28-e389-11ee-95c9-87774f69a715. 2024-03-16T11:48:28.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:27 smithi152 ceph-mon[36291]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T11:48:28.949 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:28 smithi029 ceph-mon[31975]: from='client.? 172.21.15.29:0/1741036840' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T11:48:28.949 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:28 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:48:28.949 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:28 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:48:28.949 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:28 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:48:28.949 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:28 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-16T11:48:28.949 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:28 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-16T11:48:28.949 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:28 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:48:28.949 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:28 smithi029 ceph-mon[31975]: Deploying daemon mds.a on smithi029 2024-03-16T11:48:29.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:28 smithi154 ceph-mon[36320]: from='client.? 172.21.15.29:0/1741036840' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T11:48:29.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:28 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:48:29.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:28 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:48:29.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:28 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:48:29.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:28 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-16T11:48:29.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:28 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-16T11:48:29.046 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:28 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:48:29.046 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:28 smithi154 ceph-mon[36320]: Deploying daemon mds.a on smithi029 2024-03-16T11:48:29.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:28 smithi152 ceph-mon[36291]: from='client.? 172.21.15.29:0/1741036840' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T11:48:29.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:28 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:48:29.128 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:28 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:48:29.128 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:28 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:48:29.128 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:28 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-16T11:48:29.128 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:28 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-16T11:48:29.128 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:28 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:48:29.128 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:28 smithi152 ceph-mon[36291]: Deploying daemon mds.a on smithi029 2024-03-16T11:48:29.962 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T11:48:29.988 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:29 smithi029 ceph-mon[31975]: mds.? [v2:172.21.15.29:6838/544966534,v1:172.21.15.29:6839/544966534] up:boot 2024-03-16T11:48:29.988 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:29 smithi029 ceph-mon[31975]: fsmap 10 up:standby 2024-03-16T11:48:29.988 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:29 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-16T11:48:29.988 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:29 smithi029 ceph-mon[31975]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T11:48:30.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:29 smithi154 ceph-mon[36320]: mds.? [v2:172.21.15.29:6838/544966534,v1:172.21.15.29:6839/544966534] up:boot 2024-03-16T11:48:30.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:29 smithi154 ceph-mon[36320]: fsmap 10 up:standby 2024-03-16T11:48:30.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:29 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-16T11:48:30.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:29 smithi154 ceph-mon[36320]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T11:48:30.097 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:30.098+0000 7f90ea977640 1 -- 172.21.15.29:0/2735297148 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f90e4101820 msgr2=0x7f90e4101c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:48:30.098 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:30.098+0000 7f90ea977640 1 --2- 172.21.15.29:0/2735297148 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f90e4101820 0x7f90e4101c40 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f90d4009900 tx=0x7f90d402bc30 comp rx=0 tx=0).stop 2024-03-16T11:48:30.098 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:30.098+0000 7f90ea977640 1 -- 172.21.15.29:0/2735297148 shutdown_connections 2024-03-16T11:48:30.098 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:30.098+0000 7f90ea977640 1 --2- 172.21.15.29:0/2735297148 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f90e41033d0 0x7f90e4109c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:30.098 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:30.098+0000 7f90ea977640 1 --2- 172.21.15.29:0/2735297148 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f90e4102a10 0x7f90e4102e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:30.098 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:30.098+0000 7f90ea977640 1 --2- 172.21.15.29:0/2735297148 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f90e4101820 0x7f90e4101c40 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:30.098 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:30.098+0000 7f90ea977640 1 -- 172.21.15.29:0/2735297148 >> 172.21.15.29:0/2735297148 conn(0x7f90e40fcd90 msgr2=0x7f90e40ff1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:48:30.098 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:30.098+0000 7f90ea977640 1 -- 172.21.15.29:0/2735297148 shutdown_connections 2024-03-16T11:48:30.098 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:30.098+0000 7f90ea977640 1 -- 172.21.15.29:0/2735297148 wait complete. 2024-03-16T11:48:30.100 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:30.099+0000 7f90ea977640 1 Processor -- start 2024-03-16T11:48:30.100 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:30.100+0000 7f90ea977640 1 -- start start 2024-03-16T11:48:30.100 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:30.100+0000 7f90ea977640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f90e4101820 0x7f90e4197af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:48:30.101 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:30.100+0000 7f90ea977640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f90e4102a10 0x7f90e4198030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:48:30.101 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:30.100+0000 7f90ea977640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f90e41033d0 0x7f90e419f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:48:30.101 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:30.100+0000 7f90ea977640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f90e410d480 con 0x7f90e41033d0 2024-03-16T11:48:30.101 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:30.100+0000 7f90ea977640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f90e410d300 con 0x7f90e4102a10 2024-03-16T11:48:30.101 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:30.100+0000 7f90ea977640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f90e410d600 con 0x7f90e4101820 2024-03-16T11:48:30.101 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:30.100+0000 7f90e8f14640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f90e41033d0 0x7f90e419f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:48:30.101 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:30.100+0000 7f90e8f14640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f90e41033d0 0x7f90e419f0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60358/0 (socket says 172.21.15.29:60358) 2024-03-16T11:48:30.101 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:30.100+0000 7f90e8f14640 1 -- 172.21.15.29:0/1251132671 learned_addr learned my addr 172.21.15.29:0/1251132671 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:48:30.101 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:30.100+0000 7f90e3fff640 1 --2- 172.21.15.29:0/1251132671 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f90e4101820 0x7f90e4197af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:48:30.101 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:30.100+0000 7f90e37fe640 1 --2- 172.21.15.29:0/1251132671 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f90e4102a10 0x7f90e4198030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:48:30.101 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:30.100+0000 7f90e8f14640 1 -- 172.21.15.29:0/1251132671 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f90e4101820 msgr2=0x7f90e4197af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:48:30.101 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:30.100+0000 7f90e8f14640 1 --2- 172.21.15.29:0/1251132671 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f90e4101820 0x7f90e4197af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:30.102 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:30.100+0000 7f90e8f14640 1 -- 172.21.15.29:0/1251132671 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f90e4102a10 msgr2=0x7f90e4198030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:48:30.102 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:30.100+0000 7f90e8f14640 1 --2- 172.21.15.29:0/1251132671 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f90e4102a10 0x7f90e4198030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:30.102 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:30.100+0000 7f90e8f14640 1 -- 172.21.15.29:0/1251132671 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90e419f710 con 0x7f90e41033d0 2024-03-16T11:48:30.102 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:30.101+0000 7f90e8f14640 1 --2- 172.21.15.29:0/1251132671 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f90e41033d0 0x7f90e419f0a0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f90dc00cce0 tx=0x7f90dc00a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:48:30.102 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:30.101+0000 7f90e17fa640 1 -- 172.21.15.29:0/1251132671 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90dc00f450 con 0x7f90e41033d0 2024-03-16T11:48:30.102 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:30.101+0000 7f90ea977640 1 -- 172.21.15.29:0/1251132671 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f90e419fa00 con 0x7f90e41033d0 2024-03-16T11:48:30.102 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:30.101+0000 7f90e17fa640 1 -- 172.21.15.29:0/1251132671 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f90dc00fad0 con 0x7f90e41033d0 2024-03-16T11:48:30.102 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:30.101+0000 7f90ea977640 1 -- 172.21.15.29:0/1251132671 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f90e419ff50 con 0x7f90e41033d0 2024-03-16T11:48:30.102 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:30.101+0000 7f90e17fa640 1 -- 172.21.15.29:0/1251132671 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90dc018c10 con 0x7f90e41033d0 2024-03-16T11:48:30.103 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:30.102+0000 7f90ea977640 1 -- 172.21.15.29:0/1251132671 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f90e4108b40 con 0x7f90e41033d0 2024-03-16T11:48:30.103 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:30.104+0000 7f90e17fa640 1 -- 172.21.15.29:0/1251132671 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f90dc00f5f0 con 0x7f90e41033d0 2024-03-16T11:48:30.103 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:30.104+0000 7f90e17fa640 1 --2- 172.21.15.29:0/1251132671 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f90cc0a2800 0x7f90cc0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:48:30.104 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:30.105+0000 7f90e17fa640 1 -- 172.21.15.29:0/1251132671 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f90dc0ceca0 con 0x7f90e41033d0 2024-03-16T11:48:30.104 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:30.105+0000 7f90e3fff640 1 --2- 172.21.15.29:0/1251132671 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f90cc0a2800 0x7f90cc0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:48:30.104 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:30.105+0000 7f90e3fff640 1 --2- 172.21.15.29:0/1251132671 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f90cc0a2800 0x7f90cc0a4cc0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f90d4000c00 tx=0x7f90d4005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:48:30.108 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:30.109+0000 7f90e17fa640 1 -- 172.21.15.29:0/1251132671 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f90dc014030 con 0x7f90e41033d0 2024-03-16T11:48:30.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:29 smithi152 ceph-mon[36291]: mds.? [v2:172.21.15.29:6838/544966534,v1:172.21.15.29:6839/544966534] up:boot 2024-03-16T11:48:30.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:29 smithi152 ceph-mon[36291]: fsmap 10 up:standby 2024-03-16T11:48:30.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:29 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-16T11:48:30.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:29 smithi152 ceph-mon[36291]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T11:48:30.317 INFO:journalctl@ceph.mds.a.smithi029.stdout:Mar 16 11:48:30 smithi029 systemd[1]: Starting Ceph mds.a for d1887f28-e389-11ee-95c9-87774f69a715... 2024-03-16T11:48:30.356 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:30.357+0000 7f90ea977640 1 -- 172.21.15.29:0/1251132671 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f90e40623e0 con 0x7f90e41033d0 2024-03-16T11:48:30.357 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:30.358+0000 7f90e17fa640 1 -- 172.21.15.29:0/1251132671 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) ==== 94+0+8836 (secure 0 0 0) 0x7f90dc0afe20 con 0x7f90e41033d0 2024-03-16T11:48:30.358 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-16T11:48:30.358 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":11,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14574,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/3505811955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":3505811955},{"type":"v1","addr":"172.21.15.29:6835","nonce":3505811955}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14586,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/1901337868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":1901337868},{"type":"v1","addr":"172.21.15.29:6837","nonce":1901337868}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14652,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6839/544966534","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6838","nonce":544966534},{"type":"v1","addr":"172.21.15.29:6839","nonce":544966534}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24407,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6833/3136024628","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6832","nonce":3136024628},{"type":"v1","addr":"172.21.15.152:6833","nonce":3136024628}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24421,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6833/2677426282","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":2677426282},{"type":"v1","addr":"172.21.15.154:6833","nonce":2677426282}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24434,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6835/2919113178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6834","nonce":2919113178},{"type":"v1","addr":"172.21.15.152:6835","nonce":2919113178}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24464,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6837/1323866722","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6836","nonce":1323866722},{"type":"v1","addr":"172.21.15.152:6837","nonce":1323866722}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24466,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/4165274681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":4165274681},{"type":"v1","addr":"172.21.15.154:6835","nonce":4165274681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24478,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6837/1468560096","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":1468560096},{"type":"v1","addr":"172.21.15.154:6837","nonce":1468560096}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24499,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/685736862","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":685736862},{"type":"v1","addr":"172.21.15.154:6839","nonce":685736862}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-03-16T11:48:30.358 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 11 2024-03-16T11:48:30.360 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:30.360+0000 7f90ea977640 1 -- 172.21.15.29:0/1251132671 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f90cc0a2800 msgr2=0x7f90cc0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:48:30.360 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:30.360+0000 7f90ea977640 1 --2- 172.21.15.29:0/1251132671 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f90cc0a2800 0x7f90cc0a4cc0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f90d4000c00 tx=0x7f90d4005fd0 comp rx=0 tx=0).stop 2024-03-16T11:48:30.360 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:30.361+0000 7f90ea977640 1 -- 172.21.15.29:0/1251132671 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f90e41033d0 msgr2=0x7f90e419f0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:48:30.360 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:30.361+0000 7f90ea977640 1 --2- 172.21.15.29:0/1251132671 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f90e41033d0 0x7f90e419f0a0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f90dc00cce0 tx=0x7f90dc00a620 comp rx=0 tx=0).stop 2024-03-16T11:48:30.360 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:30.361+0000 7f90ea977640 1 -- 172.21.15.29:0/1251132671 shutdown_connections 2024-03-16T11:48:30.360 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:30.361+0000 7f90ea977640 1 --2- 172.21.15.29:0/1251132671 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f90cc0a2800 0x7f90cc0a4cc0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:30.360 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:30.361+0000 7f90ea977640 1 --2- 172.21.15.29:0/1251132671 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f90e41033d0 0x7f90e419f0a0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:30.361 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:30.361+0000 7f90ea977640 1 --2- 172.21.15.29:0/1251132671 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f90e4102a10 0x7f90e4198030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:30.361 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:30.361+0000 7f90ea977640 1 --2- 172.21.15.29:0/1251132671 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f90e4101820 0x7f90e4197af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:30.361 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:30.361+0000 7f90ea977640 1 -- 172.21.15.29:0/1251132671 >> 172.21.15.29:0/1251132671 conn(0x7f90e40fcd90 msgr2=0x7f90e4105d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:48:30.361 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:30.361+0000 7f90ea977640 1 -- 172.21.15.29:0/1251132671 shutdown_connections 2024-03-16T11:48:30.361 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:30.361+0000 7f90ea977640 1 -- 172.21.15.29:0/1251132671 wait complete. 2024-03-16T11:48:30.588 INFO:journalctl@ceph.mds.a.smithi029.stdout:Mar 16 11:48:30 smithi029 podman[74187]: 2024-03-16 11:48:30.354897374 +0000 UTC m=+0.021034413 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-16T11:48:30.588 INFO:journalctl@ceph.mds.a.smithi029.stdout:Mar 16 11:48:30 smithi029 podman[74187]: 2024-03-16 11:48:30.474677601 +0000 UTC m=+0.140814628 container create e452e9b5df2287e9087cf48fe7d7bda47c831086bf4c87e0d8a4dd1ca16e441e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d1887f28-e389-11ee-95c9-87774f69a715-mds-a, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-03-16T11:48:30.870 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:30 smithi029 ceph-mon[31975]: from='client.? 172.21.15.29:0/1251132671' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T11:48:30.871 INFO:journalctl@ceph.mds.a.smithi029.stdout:Mar 16 11:48:30 smithi029 podman[74187]: 2024-03-16 11:48:30.762727598 +0000 UTC m=+0.428864637 container init e452e9b5df2287e9087cf48fe7d7bda47c831086bf4c87e0d8a4dd1ca16e441e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d1887f28-e389-11ee-95c9-87774f69a715-mds-a, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-03-16T11:48:30.871 INFO:journalctl@ceph.mds.a.smithi029.stdout:Mar 16 11:48:30 smithi029 podman[74187]: 2024-03-16 11:48:30.765616167 +0000 UTC m=+0.431753194 container start e452e9b5df2287e9087cf48fe7d7bda47c831086bf4c87e0d8a4dd1ca16e441e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d1887f28-e389-11ee-95c9-87774f69a715-mds-a, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5) 2024-03-16T11:48:30.871 INFO:journalctl@ceph.mds.a.smithi029.stdout:Mar 16 11:48:30 smithi029 ceph-mds[74218]: -- 172.21.15.29:0/2122568408 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x5601dccc84e0 con 0x5601dcc9e800 2024-03-16T11:48:30.871 INFO:journalctl@ceph.mds.a.smithi029.stdout:Mar 16 11:48:30 smithi029 ceph-mds[74218]: -- 172.21.15.29:0/2122568408 <== mon.0 v2:172.21.15.29:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5601dcccba40 con 0x5601dcc9e800 2024-03-16T11:48:30.871 INFO:journalctl@ceph.mds.a.smithi029.stdout:Mar 16 11:48:30 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mds-a[74212]: starting mds.a at 2024-03-16T11:48:30.871 INFO:journalctl@ceph.mds.a.smithi029.stdout:Mar 16 11:48:30 smithi029 bash[74187]: e452e9b5df2287e9087cf48fe7d7bda47c831086bf4c87e0d8a4dd1ca16e441e 2024-03-16T11:48:31.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:30 smithi154 ceph-mon[36320]: from='client.? 172.21.15.29:0/1251132671' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T11:48:31.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:30 smithi152 ceph-mon[36291]: from='client.? 172.21.15.29:0/1251132671' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T11:48:31.317 INFO:journalctl@ceph.mds.a.smithi029.stdout:Mar 16 11:48:30 smithi029 systemd[1]: Started Ceph mds.a for d1887f28-e389-11ee-95c9-87774f69a715. 2024-03-16T11:48:32.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:31 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:48:32.128 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:31 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:48:32.128 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:31 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:48:32.128 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:31 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-16T11:48:32.128 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:31 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-16T11:48:32.128 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:31 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:48:32.128 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:31 smithi152 ceph-mon[36291]: Deploying daemon mds.b on smithi152 2024-03-16T11:48:32.128 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:31 smithi152 ceph-mon[36291]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T11:48:32.128 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:31 smithi152 ceph-mon[36291]: mds.? [v2:172.21.15.29:6840/540705290,v1:172.21.15.29:6841/540705290] up:boot 2024-03-16T11:48:32.128 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:31 smithi152 ceph-mon[36291]: fsmap 11 up:standby 2024-03-16T11:48:32.128 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:31 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-16T11:48:32.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:31 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:48:32.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:31 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:48:32.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:31 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:48:32.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:31 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-16T11:48:32.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:31 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-16T11:48:32.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:31 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:48:32.296 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:31 smithi154 ceph-mon[36320]: Deploying daemon mds.b on smithi152 2024-03-16T11:48:32.296 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:31 smithi154 ceph-mon[36320]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T11:48:32.296 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:31 smithi154 ceph-mon[36320]: mds.? [v2:172.21.15.29:6840/540705290,v1:172.21.15.29:6841/540705290] up:boot 2024-03-16T11:48:32.296 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:31 smithi154 ceph-mon[36320]: fsmap 11 up:standby 2024-03-16T11:48:32.296 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:31 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-16T11:48:32.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:31 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:48:32.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:31 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:48:32.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:31 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:48:32.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:31 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-16T11:48:32.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:31 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-16T11:48:32.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:31 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:48:32.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:31 smithi029 ceph-mon[31975]: Deploying daemon mds.b on smithi152 2024-03-16T11:48:32.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:31 smithi029 ceph-mon[31975]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T11:48:32.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:31 smithi029 ceph-mon[31975]: mds.? [v2:172.21.15.29:6840/540705290,v1:172.21.15.29:6841/540705290] up:boot 2024-03-16T11:48:32.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:31 smithi029 ceph-mon[31975]: fsmap 11 up:standby 2024-03-16T11:48:32.339 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:31 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-16T11:48:32.372 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T11:48:32.492 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:32.490+0000 7f36ec2dc640 1 -- 172.21.15.29:0/2273814816 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f36e4104cf0 msgr2=0x7f36e41070e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:48:32.492 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:32.490+0000 7f36ec2dc640 1 --2- 172.21.15.29:0/2273814816 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f36e4104cf0 0x7f36e41070e0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f36d4009900 tx=0x7f36d402bc30 comp rx=0 tx=0).stop 2024-03-16T11:48:32.492 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:32.490+0000 7f36ec2dc640 1 -- 172.21.15.29:0/2273814816 shutdown_connections 2024-03-16T11:48:32.492 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:32.490+0000 7f36ec2dc640 1 --2- 172.21.15.29:0/2273814816 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f36e4107620 0x7f36e4109a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:32.492 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:32.490+0000 7f36ec2dc640 1 --2- 172.21.15.29:0/2273814816 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f36e4104cf0 0x7f36e41070e0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:32.493 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:32.490+0000 7f36ec2dc640 1 --2- 172.21.15.29:0/2273814816 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f36e41023c0 0x7f36e41047b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:32.493 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:32.490+0000 7f36ec2dc640 1 -- 172.21.15.29:0/2273814816 >> 172.21.15.29:0/2273814816 conn(0x7f36e40fbd10 msgr2=0x7f36e40fe130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:48:32.493 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:32.490+0000 7f36ec2dc640 1 -- 172.21.15.29:0/2273814816 shutdown_connections 2024-03-16T11:48:32.493 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:32.491+0000 7f36ec2dc640 1 -- 172.21.15.29:0/2273814816 wait complete. 2024-03-16T11:48:32.493 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:32.491+0000 7f36ec2dc640 1 Processor -- start 2024-03-16T11:48:32.493 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:32.491+0000 7f36ec2dc640 1 -- start start 2024-03-16T11:48:32.493 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:32.491+0000 7f36ec2dc640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f36e41023c0 0x7f36e4197ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:48:32.493 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:32.491+0000 7f36ec2dc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f36e4104cf0 0x7f36e4198010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:48:32.493 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:32.491+0000 7f36ec2dc640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f36e4107620 0x7f36e419f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:48:32.493 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:32.491+0000 7f36ec2dc640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f36e410d4c0 con 0x7f36e4104cf0 2024-03-16T11:48:32.493 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:32.491+0000 7f36ec2dc640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f36e410d340 con 0x7f36e41023c0 2024-03-16T11:48:32.493 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:32.491+0000 7f36ec2dc640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f36e410d640 con 0x7f36e4107620 2024-03-16T11:48:32.493 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:32.491+0000 7f36e9877640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f36e4104cf0 0x7f36e4198010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:48:32.493 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:32.491+0000 7f36e9877640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f36e4104cf0 0x7f36e4198010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60400/0 (socket says 172.21.15.29:60400) 2024-03-16T11:48:32.494 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:32.491+0000 7f36e9877640 1 -- 172.21.15.29:0/2934070270 learned_addr learned my addr 172.21.15.29:0/2934070270 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:48:32.494 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:32.491+0000 7f36ea078640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f36e41023c0 0x7f36e4197ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:48:32.494 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:32.492+0000 7f36ea879640 1 --2- 172.21.15.29:0/2934070270 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f36e4107620 0x7f36e419f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:48:32.494 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:32.492+0000 7f36e9877640 1 -- 172.21.15.29:0/2934070270 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f36e4107620 msgr2=0x7f36e419f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:48:32.494 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:32.492+0000 7f36e9877640 1 --2- 172.21.15.29:0/2934070270 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f36e4107620 0x7f36e419f080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:32.494 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:32.492+0000 7f36e9877640 1 -- 172.21.15.29:0/2934070270 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f36e41023c0 msgr2=0x7f36e4197ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:48:32.494 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:32.492+0000 7f36e9877640 1 --2- 172.21.15.29:0/2934070270 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f36e41023c0 0x7f36e4197ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:32.494 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:32.492+0000 7f36e9877640 1 -- 172.21.15.29:0/2934070270 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36e419f6f0 con 0x7f36e4104cf0 2024-03-16T11:48:32.494 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:32.492+0000 7f36ea879640 1 --2- 172.21.15.29:0/2934070270 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f36e4107620 0x7f36e419f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-16T11:48:32.494 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:32.492+0000 7f36e9877640 1 --2- 172.21.15.29:0/2934070270 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f36e4104cf0 0x7f36e4198010 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f36d4000c00 tx=0x7f36d4005dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:48:32.494 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:32.492+0000 7f36db7fe640 1 -- 172.21.15.29:0/2934070270 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36d4043020 con 0x7f36e4104cf0 2024-03-16T11:48:32.494 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:32.492+0000 7f36db7fe640 1 -- 172.21.15.29:0/2934070270 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f36d40054e0 con 0x7f36e4104cf0 2024-03-16T11:48:32.494 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:32.492+0000 7f36ec2dc640 1 -- 172.21.15.29:0/2934070270 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f36e419f980 con 0x7f36e4104cf0 2024-03-16T11:48:32.495 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:32.492+0000 7f36db7fe640 1 -- 172.21.15.29:0/2934070270 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36d4003e60 con 0x7f36e4104cf0 2024-03-16T11:48:32.495 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:32.492+0000 7f36ec2dc640 1 -- 172.21.15.29:0/2934070270 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f36e419fed0 con 0x7f36e4104cf0 2024-03-16T11:48:32.495 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:32.493+0000 7f36ec2dc640 1 -- 172.21.15.29:0/2934070270 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f36e40fd800 con 0x7f36e4104cf0 2024-03-16T11:48:32.495 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:32.494+0000 7f36db7fe640 1 -- 172.21.15.29:0/2934070270 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f36d4046020 con 0x7f36e4104cf0 2024-03-16T11:48:32.496 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:32.495+0000 7f36db7fe640 1 --2- 172.21.15.29:0/2934070270 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f36d00a2800 0x7f36d00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:48:32.496 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:32.495+0000 7f36ea078640 1 --2- 172.21.15.29:0/2934070270 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f36d00a2800 0x7f36d00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:48:32.496 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:32.495+0000 7f36db7fe640 1 -- 172.21.15.29:0/2934070270 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f36d40f0c20 con 0x7f36e4104cf0 2024-03-16T11:48:32.496 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:32.495+0000 7f36ea078640 1 --2- 172.21.15.29:0/2934070270 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f36d00a2800 0x7f36d00a4cc0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f36dc00bbe0 tx=0x7f36dc00b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:48:32.497 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:32.498+0000 7f36db7fe640 1 -- 172.21.15.29:0/2934070270 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f36d404d020 con 0x7f36e4104cf0 2024-03-16T11:48:32.721 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:32.722+0000 7f36ec2dc640 1 -- 172.21.15.29:0/2934070270 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f36e402d1b0 con 0x7f36e4104cf0 2024-03-16T11:48:32.721 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:32.722+0000 7f36db7fe640 1 -- 172.21.15.29:0/2934070270 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) ==== 94+0+9659 (secure 0 0 0) 0x7f36d40be0c0 con 0x7f36e4104cf0 2024-03-16T11:48:32.722 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-16T11:48:32.722 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":12,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14574,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/3505811955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":3505811955},{"type":"v1","addr":"172.21.15.29:6835","nonce":3505811955}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14586,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/1901337868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":1901337868},{"type":"v1","addr":"172.21.15.29:6837","nonce":1901337868}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14652,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6839/544966534","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6838","nonce":544966534},{"type":"v1","addr":"172.21.15.29:6839","nonce":544966534}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14664,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6841/540705290","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6840","nonce":540705290},{"type":"v1","addr":"172.21.15.29:6841","nonce":540705290}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24407,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6833/3136024628","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6832","nonce":3136024628},{"type":"v1","addr":"172.21.15.152:6833","nonce":3136024628}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24421,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6833/2677426282","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":2677426282},{"type":"v1","addr":"172.21.15.154:6833","nonce":2677426282}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24434,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6835/2919113178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6834","nonce":2919113178},{"type":"v1","addr":"172.21.15.152:6835","nonce":2919113178}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24464,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6837/1323866722","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6836","nonce":1323866722},{"type":"v1","addr":"172.21.15.152:6837","nonce":1323866722}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24466,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/4165274681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":4165274681},{"type":"v1","addr":"172.21.15.154:6835","nonce":4165274681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24478,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6837/1468560096","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":1468560096},{"type":"v1","addr":"172.21.15.154:6837","nonce":1468560096}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24499,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/685736862","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":685736862},{"type":"v1","addr":"172.21.15.154:6839","nonce":685736862}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-03-16T11:48:32.722 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 12 2024-03-16T11:48:32.724 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:32.724+0000 7f36ec2dc640 1 -- 172.21.15.29:0/2934070270 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f36d00a2800 msgr2=0x7f36d00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:48:32.724 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:32.724+0000 7f36ec2dc640 1 --2- 172.21.15.29:0/2934070270 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f36d00a2800 0x7f36d00a4cc0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f36dc00bbe0 tx=0x7f36dc00b390 comp rx=0 tx=0).stop 2024-03-16T11:48:32.724 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:32.724+0000 7f36ec2dc640 1 -- 172.21.15.29:0/2934070270 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f36e4104cf0 msgr2=0x7f36e4198010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:48:32.724 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:32.724+0000 7f36ec2dc640 1 --2- 172.21.15.29:0/2934070270 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f36e4104cf0 0x7f36e4198010 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f36d4000c00 tx=0x7f36d4005dd0 comp rx=0 tx=0).stop 2024-03-16T11:48:32.724 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:32.724+0000 7f36ec2dc640 1 -- 172.21.15.29:0/2934070270 shutdown_connections 2024-03-16T11:48:32.724 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:32.724+0000 7f36ec2dc640 1 --2- 172.21.15.29:0/2934070270 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f36d00a2800 0x7f36d00a4cc0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:32.724 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:32.724+0000 7f36ec2dc640 1 --2- 172.21.15.29:0/2934070270 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f36e4107620 0x7f36e419f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:32.724 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:32.724+0000 7f36ec2dc640 1 --2- 172.21.15.29:0/2934070270 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f36e4104cf0 0x7f36e4198010 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:32.724 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:32.724+0000 7f36ec2dc640 1 --2- 172.21.15.29:0/2934070270 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f36e41023c0 0x7f36e4197ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:32.724 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:32.724+0000 7f36ec2dc640 1 -- 172.21.15.29:0/2934070270 >> 172.21.15.29:0/2934070270 conn(0x7f36e40fbd10 msgr2=0x7f36e4102d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:48:32.724 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:32.724+0000 7f36ec2dc640 1 -- 172.21.15.29:0/2934070270 shutdown_connections 2024-03-16T11:48:32.724 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:32.724+0000 7f36ec2dc640 1 -- 172.21.15.29:0/2934070270 wait complete. 2024-03-16T11:48:33.262 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:32 smithi152 ceph-mon[36291]: from='client.? 172.21.15.29:0/2934070270' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T11:48:33.263 INFO:journalctl@ceph.mds.b.smithi152.stdout:Mar 16 11:48:33 smithi152 systemd[1]: Starting Ceph mds.b for d1887f28-e389-11ee-95c9-87774f69a715... 2024-03-16T11:48:33.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:32 smithi154 ceph-mon[36320]: from='client.? 172.21.15.29:0/2934070270' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T11:48:33.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:32 smithi029 ceph-mon[31975]: from='client.? 172.21.15.29:0/2934070270' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T11:48:33.627 INFO:journalctl@ceph.mds.b.smithi152.stdout:Mar 16 11:48:33 smithi152 podman[67227]: 2024-03-16 11:48:33.382310568 +0000 UTC m=+0.017565663 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-16T11:48:33.627 INFO:journalctl@ceph.mds.b.smithi152.stdout:Mar 16 11:48:33 smithi152 podman[67227]: 2024-03-16 11:48:33.503332026 +0000 UTC m=+0.138587105 container create a2a2b2c4d0e786b7df5056eadc30a9f614b2ed272f78b64fa3863b7e3514d204 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d1887f28-e389-11ee-95c9-87774f69a715-mds-b, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307) 2024-03-16T11:48:33.991 INFO:journalctl@ceph.mds.b.smithi152.stdout:Mar 16 11:48:33 smithi152 podman[67227]: 2024-03-16 11:48:33.773745942 +0000 UTC m=+0.409001028 container init a2a2b2c4d0e786b7df5056eadc30a9f614b2ed272f78b64fa3863b7e3514d204 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d1887f28-e389-11ee-95c9-87774f69a715-mds-b, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-03-16T11:48:33.991 INFO:journalctl@ceph.mds.b.smithi152.stdout:Mar 16 11:48:33 smithi152 podman[67227]: 2024-03-16 11:48:33.776324327 +0000 UTC m=+0.411579404 container start a2a2b2c4d0e786b7df5056eadc30a9f614b2ed272f78b64fa3863b7e3514d204 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d1887f28-e389-11ee-95c9-87774f69a715-mds-b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_CLEAN=True, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True) 2024-03-16T11:48:33.991 INFO:journalctl@ceph.mds.b.smithi152.stdout:Mar 16 11:48:33 smithi152 ceph-mds[67246]: -- 172.21.15.152:0/3054019829 <== mon.2 v2:172.21.15.152:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x565048663a40 con 0x565048637000 2024-03-16T11:48:33.991 INFO:journalctl@ceph.mds.b.smithi152.stdout:Mar 16 11:48:33 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mds-b[67242]: starting mds.b at 2024-03-16T11:48:33.991 INFO:journalctl@ceph.mds.b.smithi152.stdout:Mar 16 11:48:33 smithi152 bash[67227]: a2a2b2c4d0e786b7df5056eadc30a9f614b2ed272f78b64fa3863b7e3514d204 2024-03-16T11:48:33.991 INFO:journalctl@ceph.mds.b.smithi152.stdout:Mar 16 11:48:33 smithi152 systemd[1]: Started Ceph mds.b for d1887f28-e389-11ee-95c9-87774f69a715. 2024-03-16T11:48:34.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:33 smithi154 ceph-mon[36320]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T11:48:34.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:33 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:48:34.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:33 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:48:34.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:33 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:48:34.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:33 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:48:34.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:33 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:48:34.337 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:33 smithi029 ceph-mon[31975]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T11:48:34.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:33 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:48:34.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:33 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:48:34.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:33 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:48:34.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:33 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:48:34.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:33 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:48:34.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:33 smithi152 ceph-mon[36291]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T11:48:34.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:33 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:48:34.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:33 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:48:34.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:33 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:48:34.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:33 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:48:34.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:33 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:48:34.735 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T11:48:34.846 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:34.844+0000 7f307a827640 1 -- 172.21.15.29:0/4264800148 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f30741022a0 msgr2=0x7f3074108b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:48:34.846 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:34.844+0000 7f307a827640 1 --2- 172.21.15.29:0/4264800148 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f30741022a0 0x7f3074108b30 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f306c009ca0 tx=0x7f306c02c0f0 comp rx=0 tx=0).stop 2024-03-16T11:48:34.846 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:34.844+0000 7f307a827640 1 -- 172.21.15.29:0/4264800148 shutdown_connections 2024-03-16T11:48:34.846 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:34.844+0000 7f307a827640 1 --2- 172.21.15.29:0/4264800148 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f30741022a0 0x7f3074108b30 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:34.846 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:34.844+0000 7f307a827640 1 --2- 172.21.15.29:0/4264800148 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f30741018e0 0x7f3074101d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:34.846 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:34.844+0000 7f307a827640 1 --2- 172.21.15.29:0/4264800148 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f30741006f0 0x7f3074100b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:34.846 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:34.844+0000 7f307a827640 1 -- 172.21.15.29:0/4264800148 >> 172.21.15.29:0/4264800148 conn(0x7f30740fbc80 msgr2=0x7f30740fe0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:48:34.847 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:34.844+0000 7f307a827640 1 -- 172.21.15.29:0/4264800148 shutdown_connections 2024-03-16T11:48:34.847 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:34.844+0000 7f307a827640 1 -- 172.21.15.29:0/4264800148 wait complete. 2024-03-16T11:48:34.847 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:34.845+0000 7f307a827640 1 Processor -- start 2024-03-16T11:48:34.847 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:34.845+0000 7f307a827640 1 -- start start 2024-03-16T11:48:34.847 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:34.845+0000 7f307a827640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f30741006f0 0x7f3074197b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:48:34.847 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:34.845+0000 7f307a827640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f30741018e0 0x7f3074198040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:48:34.847 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:34.845+0000 7f307a827640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f30741022a0 0x7f307419f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:48:34.847 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:34.845+0000 7f307a827640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f307410c370 con 0x7f30741006f0 2024-03-16T11:48:34.847 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:34.845+0000 7f307a827640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f307410c1f0 con 0x7f30741022a0 2024-03-16T11:48:34.847 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:34.845+0000 7f307a827640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f307410c4f0 con 0x7f30741018e0 2024-03-16T11:48:34.847 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:34.845+0000 7f3073fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f30741006f0 0x7f3074197b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:48:34.848 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:34.845+0000 7f3073fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f30741006f0 0x7f3074197b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60418/0 (socket says 172.21.15.29:60418) 2024-03-16T11:48:34.848 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:34.845+0000 7f3073fff640 1 -- 172.21.15.29:0/1369510760 learned_addr learned my addr 172.21.15.29:0/1369510760 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:48:34.848 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:34.845+0000 7f30737fe640 1 --2- 172.21.15.29:0/1369510760 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f30741018e0 0x7f3074198040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:48:34.848 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:34.845+0000 7f3073fff640 1 -- 172.21.15.29:0/1369510760 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f30741018e0 msgr2=0x7f3074198040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:48:34.848 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:34.845+0000 7f3073fff640 1 --2- 172.21.15.29:0/1369510760 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f30741018e0 0x7f3074198040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:34.848 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:34.845+0000 7f3073fff640 1 -- 172.21.15.29:0/1369510760 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f30741022a0 msgr2=0x7f307419f0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:48:34.848 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:34.845+0000 7f3073fff640 1 --2- 172.21.15.29:0/1369510760 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f30741022a0 0x7f307419f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:34.848 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:34.845+0000 7f3073fff640 1 -- 172.21.15.29:0/1369510760 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f307419f720 con 0x7f30741006f0 2024-03-16T11:48:34.848 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:34.845+0000 7f3073fff640 1 --2- 172.21.15.29:0/1369510760 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f30741006f0 0x7f3074197b00 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f306400d770 tx=0x7f306400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:48:34.848 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:34.846+0000 7f30717fa640 1 -- 172.21.15.29:0/1369510760 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3064017070 con 0x7f30741006f0 2024-03-16T11:48:34.848 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:34.846+0000 7f30717fa640 1 -- 172.21.15.29:0/1369510760 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3064009c30 con 0x7f30741006f0 2024-03-16T11:48:34.848 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:34.846+0000 7f307a827640 1 -- 172.21.15.29:0/1369510760 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f307419fa10 con 0x7f30741006f0 2024-03-16T11:48:34.849 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:34.846+0000 7f30717fa640 1 -- 172.21.15.29:0/1369510760 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3064003dd0 con 0x7f30741006f0 2024-03-16T11:48:34.849 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:34.846+0000 7f307a827640 1 -- 172.21.15.29:0/1369510760 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f307419ffb0 con 0x7f30741006f0 2024-03-16T11:48:34.849 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:34.847+0000 7f307a827640 1 -- 172.21.15.29:0/1369510760 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f30741a01e0 con 0x7f30741006f0 2024-03-16T11:48:34.849 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:34.848+0000 7f30717fa640 1 -- 172.21.15.29:0/1369510760 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f3064003fb0 con 0x7f30741006f0 2024-03-16T11:48:34.850 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:34.848+0000 7f30717fa640 1 --2- 172.21.15.29:0/1369510760 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f305c0a2880 0x7f305c0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:48:34.850 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:34.848+0000 7f30737fe640 1 --2- 172.21.15.29:0/1369510760 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f305c0a2880 0x7f305c0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:48:34.850 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:34.848+0000 7f30717fa640 1 -- 172.21.15.29:0/1369510760 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f30640cdf90 con 0x7f30741006f0 2024-03-16T11:48:34.850 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:34.849+0000 7f30737fe640 1 --2- 172.21.15.29:0/1369510760 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f305c0a2880 0x7f305c0a4d40 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f3074199010 tx=0x7f3068005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:48:34.852 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:34.853+0000 7f30717fa640 1 -- 172.21.15.29:0/1369510760 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f3064009de0 con 0x7f30741006f0 2024-03-16T11:48:35.078 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:35.079+0000 7f307a827640 1 -- 172.21.15.29:0/1369510760 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f30741a03b0 con 0x7f30741006f0 2024-03-16T11:48:35.079 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:35.080+0000 7f30717fa640 1 -- 172.21.15.29:0/1369510760 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) ==== 94+0+10488 (secure 0 0 0) 0x7f30640b2910 con 0x7f30741006f0 2024-03-16T11:48:35.079 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-16T11:48:35.079 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":13,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14574,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/3505811955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":3505811955},{"type":"v1","addr":"172.21.15.29:6835","nonce":3505811955}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14586,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/1901337868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":1901337868},{"type":"v1","addr":"172.21.15.29:6837","nonce":1901337868}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14652,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6839/544966534","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6838","nonce":544966534},{"type":"v1","addr":"172.21.15.29:6839","nonce":544966534}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14664,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6841/540705290","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6840","nonce":540705290},{"type":"v1","addr":"172.21.15.29:6841","nonce":540705290}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24407,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6833/3136024628","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6832","nonce":3136024628},{"type":"v1","addr":"172.21.15.152:6833","nonce":3136024628}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24421,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6833/2677426282","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":2677426282},{"type":"v1","addr":"172.21.15.154:6833","nonce":2677426282}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24434,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6835/2919113178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6834","nonce":2919113178},{"type":"v1","addr":"172.21.15.152:6835","nonce":2919113178}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24464,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6837/1323866722","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6836","nonce":1323866722},{"type":"v1","addr":"172.21.15.152:6837","nonce":1323866722}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24466,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/4165274681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":4165274681},{"type":"v1","addr":"172.21.15.154:6835","nonce":4165274681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24478,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6837/1468560096","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":1468560096},{"type":"v1","addr":"172.21.15.154:6837","nonce":1468560096}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24499,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/685736862","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":685736862},{"type":"v1","addr":"172.21.15.154:6839","nonce":685736862}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24503,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6839/1548731833","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6838","nonce":1548731833},{"type":"v1","addr":"172.21.15.152:6839","nonce":1548731833}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13}],"filesystems":[]} 2024-03-16T11:48:35.080 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 13 2024-03-16T11:48:35.081 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:35.082+0000 7f307a827640 1 -- 172.21.15.29:0/1369510760 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f305c0a2880 msgr2=0x7f305c0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:48:35.081 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:35.082+0000 7f307a827640 1 --2- 172.21.15.29:0/1369510760 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f305c0a2880 0x7f305c0a4d40 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f3074199010 tx=0x7f3068005f90 comp rx=0 tx=0).stop 2024-03-16T11:48:35.081 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:35.082+0000 7f307a827640 1 -- 172.21.15.29:0/1369510760 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f30741006f0 msgr2=0x7f3074197b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:48:35.082 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:35.082+0000 7f307a827640 1 --2- 172.21.15.29:0/1369510760 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f30741006f0 0x7f3074197b00 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f306400d770 tx=0x7f306400dc30 comp rx=0 tx=0).stop 2024-03-16T11:48:35.082 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:35.082+0000 7f307a827640 1 -- 172.21.15.29:0/1369510760 shutdown_connections 2024-03-16T11:48:35.082 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:35.082+0000 7f307a827640 1 --2- 172.21.15.29:0/1369510760 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f305c0a2880 0x7f305c0a4d40 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:35.082 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:35.082+0000 7f307a827640 1 --2- 172.21.15.29:0/1369510760 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f30741022a0 0x7f307419f0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:35.082 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:35.082+0000 7f307a827640 1 --2- 172.21.15.29:0/1369510760 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f30741018e0 0x7f3074198040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:35.082 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:35.082+0000 7f307a827640 1 --2- 172.21.15.29:0/1369510760 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f30741006f0 0x7f3074197b00 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:35.082 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:35.082+0000 7f307a827640 1 -- 172.21.15.29:0/1369510760 >> 172.21.15.29:0/1369510760 conn(0x7f30740fbc80 msgr2=0x7f30740fd6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:48:35.082 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:35.082+0000 7f307a827640 1 -- 172.21.15.29:0/1369510760 shutdown_connections 2024-03-16T11:48:35.082 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:35.082+0000 7f307a827640 1 -- 172.21.15.29:0/1369510760 wait complete. 2024-03-16T11:48:35.087 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:34 smithi029 ceph-mon[31975]: mds.? [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] up:boot 2024-03-16T11:48:35.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:34 smithi029 ceph-mon[31975]: fsmap 12 up:standby 2024-03-16T11:48:35.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:34 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-16T11:48:35.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:34 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T11:48:35.092 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-16T11:48:35.200 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:35.198+0000 7f0bff7a9640 1 -- 172.21.15.29:0/174479419 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0bf8100700 msgr2=0x7f0bf8100b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:48:35.200 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:35.198+0000 7f0bff7a9640 1 --2- 172.21.15.29:0/174479419 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0bf8100700 0x7f0bf8100b20 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f0bf4009900 tx=0x7f0bf402bc30 comp rx=0 tx=0).stop 2024-03-16T11:48:35.200 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:35.198+0000 7f0bff7a9640 1 -- 172.21.15.29:0/174479419 shutdown_connections 2024-03-16T11:48:35.201 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:35.198+0000 7f0bff7a9640 1 --2- 172.21.15.29:0/174479419 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0bf81022b0 0x7f0bf8108b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:35.201 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:35.198+0000 7f0bff7a9640 1 --2- 172.21.15.29:0/174479419 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0bf81018f0 0x7f0bf8101d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:35.201 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:35.198+0000 7f0bff7a9640 1 --2- 172.21.15.29:0/174479419 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0bf8100700 0x7f0bf8100b20 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:35.201 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:35.198+0000 7f0bff7a9640 1 -- 172.21.15.29:0/174479419 >> 172.21.15.29:0/174479419 conn(0x7f0bf80fbc70 msgr2=0x7f0bf80fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:48:35.201 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:35.199+0000 7f0bff7a9640 1 -- 172.21.15.29:0/174479419 shutdown_connections 2024-03-16T11:48:35.201 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:35.199+0000 7f0bff7a9640 1 -- 172.21.15.29:0/174479419 wait complete. 2024-03-16T11:48:35.201 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:35.199+0000 7f0bff7a9640 1 Processor -- start 2024-03-16T11:48:35.201 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:35.199+0000 7f0bff7a9640 1 -- start start 2024-03-16T11:48:35.201 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:35.199+0000 7f0bff7a9640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0bf8100700 0x7f0bf8196a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:48:35.201 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:35.199+0000 7f0bff7a9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0bf81018f0 0x7f0bf8196f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:48:35.201 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:35.199+0000 7f0bff7a9640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0bf81022b0 0x7f0bf819e000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:48:35.201 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:35.199+0000 7f0bff7a9640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f0bf810c350 con 0x7f0bf81018f0 2024-03-16T11:48:35.201 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:35.199+0000 7f0bff7a9640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f0bf810c1d0 con 0x7f0bf81022b0 2024-03-16T11:48:35.202 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:35.199+0000 7f0bff7a9640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f0bf810c4d0 con 0x7f0bf8100700 2024-03-16T11:48:35.202 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:35.199+0000 7f0bfcd44640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0bf81018f0 0x7f0bf8196f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:48:35.202 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:35.199+0000 7f0bfcd44640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0bf81018f0 0x7f0bf8196f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60434/0 (socket says 172.21.15.29:60434) 2024-03-16T11:48:35.202 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:35.199+0000 7f0bfcd44640 1 -- 172.21.15.29:0/67479729 learned_addr learned my addr 172.21.15.29:0/67479729 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:48:35.202 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:35.199+0000 7f0bfd545640 1 --2- 172.21.15.29:0/67479729 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0bf8100700 0x7f0bf8196a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:48:35.202 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:35.200+0000 7f0bfdd46640 1 --2- 172.21.15.29:0/67479729 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0bf81022b0 0x7f0bf819e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:48:35.202 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:35.200+0000 7f0bfcd44640 1 -- 172.21.15.29:0/67479729 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0bf8100700 msgr2=0x7f0bf8196a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:48:35.202 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:35.200+0000 7f0bfcd44640 1 --2- 172.21.15.29:0/67479729 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0bf8100700 0x7f0bf8196a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:35.202 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:35.200+0000 7f0bfcd44640 1 -- 172.21.15.29:0/67479729 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0bf81022b0 msgr2=0x7f0bf819e000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:48:35.202 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:35.200+0000 7f0bfcd44640 1 --2- 172.21.15.29:0/67479729 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0bf81022b0 0x7f0bf819e000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:35.202 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:35.200+0000 7f0bfcd44640 1 -- 172.21.15.29:0/67479729 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0bf819e670 con 0x7f0bf81018f0 2024-03-16T11:48:35.202 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:35.200+0000 7f0bfdd46640 1 --2- 172.21.15.29:0/67479729 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0bf81022b0 0x7f0bf819e000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-16T11:48:35.202 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:35.200+0000 7f0bfcd44640 1 --2- 172.21.15.29:0/67479729 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0bf81018f0 0x7f0bf8196f90 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f0be800c890 tx=0x7f0be800cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:48:35.203 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:35.200+0000 7f0bee7fc640 1 -- 172.21.15.29:0/67479729 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0be8004220 con 0x7f0bf81018f0 2024-03-16T11:48:35.203 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:35.200+0000 7f0bee7fc640 1 -- 172.21.15.29:0/67479729 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0be80043c0 con 0x7f0bf81018f0 2024-03-16T11:48:35.203 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:35.200+0000 7f0bff7a9640 1 -- 172.21.15.29:0/67479729 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0bf819e960 con 0x7f0bf81018f0 2024-03-16T11:48:35.203 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:35.200+0000 7f0bee7fc640 1 -- 172.21.15.29:0/67479729 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0be80049e0 con 0x7f0bf81018f0 2024-03-16T11:48:35.203 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:35.200+0000 7f0bff7a9640 1 -- 172.21.15.29:0/67479729 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0bf819eeb0 con 0x7f0bf81018f0 2024-03-16T11:48:35.203 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:35.201+0000 7f0bff7a9640 1 -- 172.21.15.29:0/67479729 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0bf8107aa0 con 0x7f0bf81018f0 2024-03-16T11:48:35.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:35.202+0000 7f0bee7fc640 1 -- 172.21.15.29:0/67479729 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f0be80079b0 con 0x7f0bf81018f0 2024-03-16T11:48:35.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:35.202+0000 7f0bee7fc640 1 --2- 172.21.15.29:0/67479729 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f0be40a2800 0x7f0be40a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:48:35.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:35.203+0000 7f0bfd545640 1 --2- 172.21.15.29:0/67479729 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f0be40a2800 0x7f0be40a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:48:35.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:35.203+0000 7f0bee7fc640 1 -- 172.21.15.29:0/67479729 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f0be80ce6c0 con 0x7f0bf81018f0 2024-03-16T11:48:35.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:35.203+0000 7f0bfd545640 1 --2- 172.21.15.29:0/67479729 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f0be40a2800 0x7f0be40a4cc0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f0bf4000c00 tx=0x7f0bf4005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:48:35.206 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:35.207+0000 7f0bee7fc640 1 -- 172.21.15.29:0/67479729 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f0be8014030 con 0x7f0bf81018f0 2024-03-16T11:48:35.251 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:34 smithi152 ceph-mon[36291]: mds.? [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] up:boot 2024-03-16T11:48:35.251 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:34 smithi152 ceph-mon[36291]: fsmap 12 up:standby 2024-03-16T11:48:35.251 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:34 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-16T11:48:35.251 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:34 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T11:48:35.287 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:34 smithi154 ceph-mon[36320]: mds.? [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] up:boot 2024-03-16T11:48:35.287 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:34 smithi154 ceph-mon[36320]: fsmap 12 up:standby 2024-03-16T11:48:35.288 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:34 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-16T11:48:35.288 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:34 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T11:48:35.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:35.394+0000 7f0bff7a9640 1 -- 172.21.15.29:0/67479729 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f0bf8062410 con 0x7f0bf81018f0 2024-03-16T11:48:35.394 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:35.395+0000 7f0bee7fc640 1 -- 172.21.15.29:0/67479729 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) ==== 74+0+18470 (secure 0 0 0) 0x7f0be809bb60 con 0x7f0bf81018f0 2024-03-16T11:48:35.395 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-16T11:48:35.395 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":81,"fsid":"d1887f28-e389-11ee-95c9-87774f69a715","created":"2024-03-16T11:40:16.475063+0000","modified":"2024-03-16T11:47:45.354317+0000","last_up_change":"2024-03-16T11:47:44.349348+0000","last_in_change":"2024-03-16T11:47:23.672432+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-16T11:43:51.892652+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"60f5608f-a408-45ba-bdf6-5b53758f88fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":2748947913},{"type":"v1","addr":"172.21.15.29:6803","nonce":2748947913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":2748947913},{"type":"v1","addr":"172.21.15.29:6805","nonce":2748947913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":2748947913},{"type":"v1","addr":"172.21.15.29:6809","nonce":2748947913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":2748947913},{"type":"v1","addr":"172.21.15.29:6807","nonce":2748947913}]},"public_addr":"172.21.15.29:6803/2748947913","cluster_addr":"172.21.15.29:6805/2748947913","heartbeat_back_addr":"172.21.15.29:6809/2748947913","heartbeat_front_addr":"172.21.15.29:6807/2748947913","state":["exists","up"]},{"osd":1,"uuid":"e70b53d6-ee78-4e56-9a2c-263ae0501b43","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6800","nonce":2339408422},{"type":"v1","addr":"172.21.15.152:6801","nonce":2339408422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6802","nonce":2339408422},{"type":"v1","addr":"172.21.15.152:6803","nonce":2339408422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6806","nonce":2339408422},{"type":"v1","addr":"172.21.15.152:6807","nonce":2339408422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6804","nonce":2339408422},{"type":"v1","addr":"172.21.15.152:6805","nonce":2339408422}]},"public_addr":"172.21.15.152:6801/2339408422","cluster_addr":"172.21.15.152:6803/2339408422","heartbeat_back_addr":"172.21.15.152:6807/2339408422","heartbeat_front_addr":"172.21.15.152:6805/2339408422","state":["exists","up"]},{"osd":2,"uuid":"ff39389a-dfac-44b5-80ff-36abca316e78","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6800","nonce":4288325280},{"type":"v1","addr":"172.21.15.154:6801","nonce":4288325280}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6802","nonce":4288325280},{"type":"v1","addr":"172.21.15.154:6803","nonce":4288325280}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6806","nonce":4288325280},{"type":"v1","addr":"172.21.15.154:6807","nonce":4288325280}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":4288325280},{"type":"v1","addr":"172.21.15.154:6805","nonce":4288325280}]},"public_addr":"172.21.15.154:6801/4288325280","cluster_addr":"172.21.15.154:6803/4288325280","heartbeat_back_addr":"172.21.15.154:6807/4288325280","heartbeat_front_addr":"172.21.15.154:6805/4288325280","state":["exists","up"]},{"osd":3,"uuid":"069e1795-55bc-4324-b94c-eb24ef2334c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":1538750397},{"type":"v1","addr":"172.21.15.29:6811","nonce":1538750397}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":1538750397},{"type":"v1","addr":"172.21.15.29:6813","nonce":1538750397}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":1538750397},{"type":"v1","addr":"172.21.15.29:6817","nonce":1538750397}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":1538750397},{"type":"v1","addr":"172.21.15.29:6815","nonce":1538750397}]},"public_addr":"172.21.15.29:6811/1538750397","cluster_addr":"172.21.15.29:6813/1538750397","heartbeat_back_addr":"172.21.15.29:6817/1538750397","heartbeat_front_addr":"172.21.15.29:6815/1538750397","state":["exists","up"]},{"osd":4,"uuid":"5a9d43a1-e8f5-4ab5-a441-afbc6ed4066e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6808","nonce":2795081750},{"type":"v1","addr":"172.21.15.152:6809","nonce":2795081750}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6810","nonce":2795081750},{"type":"v1","addr":"172.21.15.152:6811","nonce":2795081750}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6814","nonce":2795081750},{"type":"v1","addr":"172.21.15.152:6815","nonce":2795081750}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6812","nonce":2795081750},{"type":"v1","addr":"172.21.15.152:6813","nonce":2795081750}]},"public_addr":"172.21.15.152:6809/2795081750","cluster_addr":"172.21.15.152:6811/2795081750","heartbeat_back_addr":"172.21.15.152:6815/2795081750","heartbeat_front_addr":"172.21.15.152:6813/2795081750","state":["exists","up"]},{"osd":5,"uuid":"234de428-4ac7-44be-9a54-7b58331fdf8c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":1472686560},{"type":"v1","addr":"172.21.15.154:6809","nonce":1472686560}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6810","nonce":1472686560},{"type":"v1","addr":"172.21.15.154:6811","nonce":1472686560}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6814","nonce":1472686560},{"type":"v1","addr":"172.21.15.154:6815","nonce":1472686560}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6812","nonce":1472686560},{"type":"v1","addr":"172.21.15.154:6813","nonce":1472686560}]},"public_addr":"172.21.15.154:6809/1472686560","cluster_addr":"172.21.15.154:6811/1472686560","heartbeat_back_addr":"172.21.15.154:6815/1472686560","heartbeat_front_addr":"172.21.15.154:6813/1472686560","state":["exists","up"]},{"osd":6,"uuid":"ae45b8d9-7883-40c9-a9cd-4c92b601d4f2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":816506244},{"type":"v1","addr":"172.21.15.29:6819","nonce":816506244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":816506244},{"type":"v1","addr":"172.21.15.29:6821","nonce":816506244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":816506244},{"type":"v1","addr":"172.21.15.29:6825","nonce":816506244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":816506244},{"type":"v1","addr":"172.21.15.29:6823","nonce":816506244}]},"public_addr":"172.21.15.29:6819/816506244","cluster_addr":"172.21.15.29:6821/816506244","heartbeat_back_addr":"172.21.15.29:6825/816506244","heartbeat_front_addr":"172.21.15.29:6823/816506244","state":["exists","up"]},{"osd":7,"uuid":"6f687b28-4fce-4d4a-8b4e-294173d9d6c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6816","nonce":2924651787},{"type":"v1","addr":"172.21.15.152:6817","nonce":2924651787}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6818","nonce":2924651787},{"type":"v1","addr":"172.21.15.152:6819","nonce":2924651787}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6822","nonce":2924651787},{"type":"v1","addr":"172.21.15.152:6823","nonce":2924651787}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6820","nonce":2924651787},{"type":"v1","addr":"172.21.15.152:6821","nonce":2924651787}]},"public_addr":"172.21.15.152:6817/2924651787","cluster_addr":"172.21.15.152:6819/2924651787","heartbeat_back_addr":"172.21.15.152:6823/2924651787","heartbeat_front_addr":"172.21.15.152:6821/2924651787","state":["exists","up"]},{"osd":8,"uuid":"a25fb741-caa7-429a-b540-e394d034b446","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6816","nonce":129974156},{"type":"v1","addr":"172.21.15.154:6817","nonce":129974156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6818","nonce":129974156},{"type":"v1","addr":"172.21.15.154:6819","nonce":129974156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6822","nonce":129974156},{"type":"v1","addr":"172.21.15.154:6823","nonce":129974156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6820","nonce":129974156},{"type":"v1","addr":"172.21.15.154:6821","nonce":129974156}]},"public_addr":"172.21.15.154:6817/129974156","cluster_addr":"172.21.15.154:6819/129974156","heartbeat_back_addr":"172.21.15.154:6823/129974156","heartbeat_front_addr":"172.21.15.154:6821/129974156","state":["exists","up"]},{"osd":9,"uuid":"1071cb0e-8c5f-4cd8-9805-0045e220d946","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6826","nonce":2951414243},{"type":"v1","addr":"172.21.15.29:6827","nonce":2951414243}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":2951414243},{"type":"v1","addr":"172.21.15.29:6829","nonce":2951414243}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":2951414243},{"type":"v1","addr":"172.21.15.29:6833","nonce":2951414243}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":2951414243},{"type":"v1","addr":"172.21.15.29:6831","nonce":2951414243}]},"public_addr":"172.21.15.29:6827/2951414243","cluster_addr":"172.21.15.29:6829/2951414243","heartbeat_back_addr":"172.21.15.29:6833/2951414243","heartbeat_front_addr":"172.21.15.29:6831/2951414243","state":["exists","up"]},{"osd":10,"uuid":"ee7c7d74-3f39-4cd4-be80-843fb1dd9ecc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6824","nonce":3412525480},{"type":"v1","addr":"172.21.15.152:6825","nonce":3412525480}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6826","nonce":3412525480},{"type":"v1","addr":"172.21.15.152:6827","nonce":3412525480}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6830","nonce":3412525480},{"type":"v1","addr":"172.21.15.152:6831","nonce":3412525480}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6828","nonce":3412525480},{"type":"v1","addr":"172.21.15.152:6829","nonce":3412525480}]},"public_addr":"172.21.15.152:6825/3412525480","cluster_addr":"172.21.15.152:6827/3412525480","heartbeat_back_addr":"172.21.15.152:6831/3412525480","heartbeat_front_addr":"172.21.15.152:6829/3412525480","state":["exists","up"]},{"osd":11,"uuid":"4598c803-984a-4de6-ac22-066502c17889","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6824","nonce":1160927966},{"type":"v1","addr":"172.21.15.154:6825","nonce":1160927966}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6826","nonce":1160927966},{"type":"v1","addr":"172.21.15.154:6827","nonce":1160927966}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6830","nonce":1160927966},{"type":"v1","addr":"172.21.15.154:6831","nonce":1160927966}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6828","nonce":1160927966},{"type":"v1","addr":"172.21.15.154:6829","nonce":1160927966}]},"public_addr":"172.21.15.154:6825/1160927966","cluster_addr":"172.21.15.154:6827/1160927966","heartbeat_back_addr":"172.21.15.154:6831/1160927966","heartbeat_front_addr":"172.21.15.154:6829/1160927966","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:43:02.343393+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:43:25.209151+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:43:48.946164+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:44:14.588424+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:44:39.700025+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:45:04.061197+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:45:31.214508+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:45:56.937296+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:46:22.768915+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:46:50.101011+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:47:16.298041+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:47:42.402095+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.29:0/1853086886":"2024-03-17T11:41:27.531724+0000","172.21.15.29:0/1211659364":"2024-03-17T11:41:27.531724+0000","172.21.15.29:6800/2798831128":"2024-03-17T11:41:27.531724+0000","172.21.15.29:0/1281028285":"2024-03-17T11:41:00.505678+0000","172.21.15.29:6800/2275315654":"2024-03-17T11:41:00.505678+0000","172.21.15.29:6801/2798831128":"2024-03-17T11:41:27.531724+0000","172.21.15.29:6801/2275315654":"2024-03-17T11:41:00.505678+0000","172.21.15.29:0/1330100009":"2024-03-17T11:41:00.505678+0000","172.21.15.29:0/1784249039":"2024-03-17T11:41:00.505678+0000","172.21.15.29:0/1116486932":"2024-03-17T11:41:27.531724+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-16T11:48:35.397 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:35.397+0000 7f0bff7a9640 1 -- 172.21.15.29:0/67479729 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f0be40a2800 msgr2=0x7f0be40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:48:35.397 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:35.397+0000 7f0bff7a9640 1 --2- 172.21.15.29:0/67479729 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f0be40a2800 0x7f0be40a4cc0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f0bf4000c00 tx=0x7f0bf4005fd0 comp rx=0 tx=0).stop 2024-03-16T11:48:35.397 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:35.397+0000 7f0bff7a9640 1 -- 172.21.15.29:0/67479729 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0bf81018f0 msgr2=0x7f0bf8196f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:48:35.397 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:35.397+0000 7f0bff7a9640 1 --2- 172.21.15.29:0/67479729 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0bf81018f0 0x7f0bf8196f90 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f0be800c890 tx=0x7f0be800cd50 comp rx=0 tx=0).stop 2024-03-16T11:48:35.397 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:35.398+0000 7f0bff7a9640 1 -- 172.21.15.29:0/67479729 shutdown_connections 2024-03-16T11:48:35.397 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:35.398+0000 7f0bff7a9640 1 --2- 172.21.15.29:0/67479729 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f0be40a2800 0x7f0be40a4cc0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:35.397 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:35.398+0000 7f0bff7a9640 1 --2- 172.21.15.29:0/67479729 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0bf81022b0 0x7f0bf819e000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:35.398 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:35.398+0000 7f0bff7a9640 1 --2- 172.21.15.29:0/67479729 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0bf81018f0 0x7f0bf8196f90 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:35.398 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:35.398+0000 7f0bff7a9640 1 --2- 172.21.15.29:0/67479729 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0bf8100700 0x7f0bf8196a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:35.398 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:35.398+0000 7f0bff7a9640 1 -- 172.21.15.29:0/67479729 >> 172.21.15.29:0/67479729 conn(0x7f0bf80fbc70 msgr2=0x7f0bf8104c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:48:35.398 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:35.398+0000 7f0bff7a9640 1 -- 172.21.15.29:0/67479729 shutdown_connections 2024-03-16T11:48:35.398 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:35.398+0000 7f0bff7a9640 1 -- 172.21.15.29:0/67479729 wait complete. 2024-03-16T11:48:35.409 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-16T11:43:51.892652+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-16T11:48:35.410 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-16T11:48:35.525 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:35.526+0000 7f4cfdd87640 1 -- 172.21.15.29:0/3997212303 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4cf81006f0 msgr2=0x7f4cf8100b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:48:35.525 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:35.526+0000 7f4cfdd87640 1 --2- 172.21.15.29:0/3997212303 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4cf81006f0 0x7f4cf8100b10 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f4ce8009900 tx=0x7f4ce802bc30 comp rx=0 tx=0).stop 2024-03-16T11:48:35.525 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:35.526+0000 7f4cfdd87640 1 -- 172.21.15.29:0/3997212303 shutdown_connections 2024-03-16T11:48:35.526 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:35.526+0000 7f4cfdd87640 1 --2- 172.21.15.29:0/3997212303 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4cf81022a0 0x7f4cf8108b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:35.526 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:35.526+0000 7f4cfdd87640 1 --2- 172.21.15.29:0/3997212303 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4cf81018e0 0x7f4cf8101d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:35.526 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:35.526+0000 7f4cfdd87640 1 --2- 172.21.15.29:0/3997212303 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4cf81006f0 0x7f4cf8100b10 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:35.526 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:35.526+0000 7f4cfdd87640 1 -- 172.21.15.29:0/3997212303 >> 172.21.15.29:0/3997212303 conn(0x7f4cf80fbc80 msgr2=0x7f4cf80fe0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:48:35.526 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:35.526+0000 7f4cfdd87640 1 -- 172.21.15.29:0/3997212303 shutdown_connections 2024-03-16T11:48:35.526 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:35.526+0000 7f4cfdd87640 1 -- 172.21.15.29:0/3997212303 wait complete. 2024-03-16T11:48:35.526 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:35.526+0000 7f4cfdd87640 1 Processor -- start 2024-03-16T11:48:35.526 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:35.527+0000 7f4cfdd87640 1 -- start start 2024-03-16T11:48:35.528 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:35.527+0000 7f4cfdd87640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4cf81006f0 0x7f4cf806a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:48:35.528 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:35.527+0000 7f4cfdd87640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4cf81018e0 0x7f4cf819c860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:48:35.528 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:35.527+0000 7f4cfdd87640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4cf81022a0 0x7f4cf819ee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:48:35.528 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:35.527+0000 7f4cfdd87640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f4cf810c370 con 0x7f4cf81022a0 2024-03-16T11:48:35.528 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:35.527+0000 7f4cfdd87640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f4cf810c1f0 con 0x7f4cf81018e0 2024-03-16T11:48:35.528 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:35.527+0000 7f4cfdd87640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f4cf810c4f0 con 0x7f4cf81006f0 2024-03-16T11:48:35.528 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:35.527+0000 7f4cf7fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4cf81022a0 0x7f4cf819ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:48:35.528 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:35.527+0000 7f4cf77fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4cf81006f0 0x7f4cf806a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:48:35.529 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:35.527+0000 7f4cf7fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4cf81022a0 0x7f4cf819ee80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60452/0 (socket says 172.21.15.29:60452) 2024-03-16T11:48:35.529 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:35.527+0000 7f4cf7fff640 1 -- 172.21.15.29:0/1672046312 learned_addr learned my addr 172.21.15.29:0/1672046312 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:48:35.529 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:35.527+0000 7f4cf6ffd640 1 --2- 172.21.15.29:0/1672046312 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4cf81018e0 0x7f4cf819c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:48:35.529 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:35.527+0000 7f4cf7fff640 1 -- 172.21.15.29:0/1672046312 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4cf81006f0 msgr2=0x7f4cf806a8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:48:35.529 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:35.527+0000 7f4cf7fff640 1 --2- 172.21.15.29:0/1672046312 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4cf81006f0 0x7f4cf806a8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:35.529 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:35.527+0000 7f4cf7fff640 1 -- 172.21.15.29:0/1672046312 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4cf81018e0 msgr2=0x7f4cf819c860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:48:35.529 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:35.527+0000 7f4cf7fff640 1 --2- 172.21.15.29:0/1672046312 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4cf81018e0 0x7f4cf819c860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:35.529 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:35.527+0000 7f4cf7fff640 1 -- 172.21.15.29:0/1672046312 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4cf819f790 con 0x7f4cf81022a0 2024-03-16T11:48:35.529 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:35.527+0000 7f4cf6ffd640 1 --2- 172.21.15.29:0/1672046312 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4cf81018e0 0x7f4cf819c860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:48:35.529 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:35.527+0000 7f4cf7fff640 1 --2- 172.21.15.29:0/1672046312 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4cf81022a0 0x7f4cf819ee80 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f4cf000eb10 tx=0x7f4cf000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:48:35.529 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:35.528+0000 7f4cf4ff9640 1 -- 172.21.15.29:0/1672046312 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cf0004b20 con 0x7f4cf81022a0 2024-03-16T11:48:35.529 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:35.528+0000 7f4cfdd87640 1 -- 172.21.15.29:0/1672046312 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4cf819fa20 con 0x7f4cf81022a0 2024-03-16T11:48:35.529 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:35.528+0000 7f4cf4ff9640 1 -- 172.21.15.29:0/1672046312 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4cf0004cc0 con 0x7f4cf81022a0 2024-03-16T11:48:35.529 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:35.528+0000 7f4cf4ff9640 1 -- 172.21.15.29:0/1672046312 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cf0010760 con 0x7f4cf81022a0 2024-03-16T11:48:35.529 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:35.528+0000 7f4cfdd87640 1 -- 172.21.15.29:0/1672046312 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4cf819ff70 con 0x7f4cf81022a0 2024-03-16T11:48:35.530 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:35.529+0000 7f4cfdd87640 1 -- 172.21.15.29:0/1672046312 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4cf8107a90 con 0x7f4cf81022a0 2024-03-16T11:48:35.530 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:35.530+0000 7f4cf4ff9640 1 -- 172.21.15.29:0/1672046312 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f4cf0005020 con 0x7f4cf81022a0 2024-03-16T11:48:35.531 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:35.531+0000 7f4cf4ff9640 1 --2- 172.21.15.29:0/1672046312 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f4ce40a2800 0x7f4ce40a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:48:35.531 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:35.531+0000 7f4cf77fe640 1 --2- 172.21.15.29:0/1672046312 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f4ce40a2800 0x7f4ce40a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:48:35.531 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:35.531+0000 7f4cf4ff9640 1 -- 172.21.15.29:0/1672046312 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f4cf00ceca0 con 0x7f4cf81022a0 2024-03-16T11:48:35.531 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:35.532+0000 7f4cf77fe640 1 --2- 172.21.15.29:0/1672046312 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f4ce40a2800 0x7f4ce40a4cc0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f4ce8000c00 tx=0x7f4ce8005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:48:35.535 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:35.536+0000 7f4cf4ff9640 1 -- 172.21.15.29:0/1672046312 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f4cf009c1e0 con 0x7f4cf81022a0 2024-03-16T11:48:35.754 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:35.755+0000 7f4cfdd87640 1 -- 172.21.15.29:0/1672046312 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f4cf802d400 con 0x7f4cf81022a0 2024-03-16T11:48:35.795 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:35.755+0000 7f4cf4ff9640 1 -- 172.21.15.29:0/1672046312 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v81) ==== 93+0+10 (secure 0 0 0) 0x7f4cf009bae0 con 0x7f4cf81022a0 2024-03-16T11:48:35.796 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2024-03-16T11:48:35.797 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:35.757+0000 7f4cfdd87640 1 -- 172.21.15.29:0/1672046312 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f4ce40a2800 msgr2=0x7f4ce40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:48:35.797 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:35.757+0000 7f4cfdd87640 1 --2- 172.21.15.29:0/1672046312 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f4ce40a2800 0x7f4ce40a4cc0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f4ce8000c00 tx=0x7f4ce8005fd0 comp rx=0 tx=0).stop 2024-03-16T11:48:35.797 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:35.757+0000 7f4cfdd87640 1 -- 172.21.15.29:0/1672046312 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4cf81022a0 msgr2=0x7f4cf819ee80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:48:35.797 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:35.757+0000 7f4cfdd87640 1 --2- 172.21.15.29:0/1672046312 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4cf81022a0 0x7f4cf819ee80 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f4cf000eb10 tx=0x7f4cf000c2d0 comp rx=0 tx=0).stop 2024-03-16T11:48:35.798 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:35.757+0000 7f4cfdd87640 1 -- 172.21.15.29:0/1672046312 shutdown_connections 2024-03-16T11:48:35.798 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:35.757+0000 7f4cfdd87640 1 --2- 172.21.15.29:0/1672046312 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f4ce40a2800 0x7f4ce40a4cc0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:35.798 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:35.757+0000 7f4cfdd87640 1 --2- 172.21.15.29:0/1672046312 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4cf81022a0 0x7f4cf819ee80 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:35.798 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:35.757+0000 7f4cfdd87640 1 --2- 172.21.15.29:0/1672046312 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4cf81018e0 0x7f4cf819c860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:35.798 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:35.757+0000 7f4cfdd87640 1 --2- 172.21.15.29:0/1672046312 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4cf81006f0 0x7f4cf806a8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:35.798 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:35.757+0000 7f4cfdd87640 1 -- 172.21.15.29:0/1672046312 >> 172.21.15.29:0/1672046312 conn(0x7f4cf80fbc80 msgr2=0x7f4cf8104c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:48:35.798 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:35.757+0000 7f4cfdd87640 1 -- 172.21.15.29:0/1672046312 shutdown_connections 2024-03-16T11:48:35.798 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:35.757+0000 7f4cfdd87640 1 -- 172.21.15.29:0/1672046312 wait complete. 2024-03-16T11:48:35.798 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-16T11:48:35.907 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:35.907+0000 7fcf98df3640 1 -- 172.21.15.29:0/501537637 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcf941075a0 msgr2=0x7fcf94109990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:48:35.907 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:35.907+0000 7fcf98df3640 1 --2- 172.21.15.29:0/501537637 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcf941075a0 0x7fcf94109990 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fcf8c009ca0 tx=0x7fcf8c02c0f0 comp rx=0 tx=0).stop 2024-03-16T11:48:35.907 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:35.907+0000 7fcf98df3640 1 -- 172.21.15.29:0/501537637 shutdown_connections 2024-03-16T11:48:35.907 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:35.907+0000 7fcf98df3640 1 --2- 172.21.15.29:0/501537637 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcf941075a0 0x7fcf94109990 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:35.907 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:35.907+0000 7fcf98df3640 1 --2- 172.21.15.29:0/501537637 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcf94104c70 0x7fcf94107060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:35.907 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:35.908+0000 7fcf98df3640 1 --2- 172.21.15.29:0/501537637 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fcf94102340 0x7fcf94104730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:35.907 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:35.908+0000 7fcf98df3640 1 -- 172.21.15.29:0/501537637 >> 172.21.15.29:0/501537637 conn(0x7fcf940fbc90 msgr2=0x7fcf940fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:48:35.907 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:35.908+0000 7fcf98df3640 1 -- 172.21.15.29:0/501537637 shutdown_connections 2024-03-16T11:48:35.907 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:35.908+0000 7fcf98df3640 1 -- 172.21.15.29:0/501537637 wait complete. 2024-03-16T11:48:35.910 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:35.908+0000 7fcf98df3640 1 Processor -- start 2024-03-16T11:48:35.910 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:35.908+0000 7fcf98df3640 1 -- start start 2024-03-16T11:48:35.910 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:35.908+0000 7fcf98df3640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcf94102340 0x7fcf94101cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:48:35.910 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:35.908+0000 7fcf98df3640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fcf94104c70 0x7fcf941002f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:48:35.910 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:35.908+0000 7fcf98df3640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcf941075a0 0x7fcf94100830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:48:35.910 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:35.908+0000 7fcf98df3640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fcf9410d4a0 con 0x7fcf94102340 2024-03-16T11:48:35.910 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:35.908+0000 7fcf98df3640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fcf9410d320 con 0x7fcf94104c70 2024-03-16T11:48:35.910 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:35.908+0000 7fcf98df3640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fcf9410d620 con 0x7fcf941075a0 2024-03-16T11:48:35.910 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:35.909+0000 7fcf9259c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcf94102340 0x7fcf94101cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:48:35.910 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:35.909+0000 7fcf9259c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcf94102340 0x7fcf94101cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60468/0 (socket says 172.21.15.29:60468) 2024-03-16T11:48:35.910 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:35.909+0000 7fcf9259c640 1 -- 172.21.15.29:0/996949508 learned_addr learned my addr 172.21.15.29:0/996949508 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:48:35.910 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:35.909+0000 7fcf8bfff640 1 --2- 172.21.15.29:0/996949508 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fcf94104c70 0x7fcf941002f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:48:35.911 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:35.909+0000 7fcf92d9d640 1 --2- 172.21.15.29:0/996949508 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcf941075a0 0x7fcf94100830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:48:35.911 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:35.909+0000 7fcf9259c640 1 -- 172.21.15.29:0/996949508 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcf941075a0 msgr2=0x7fcf94100830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:48:35.911 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:35.909+0000 7fcf9259c640 1 --2- 172.21.15.29:0/996949508 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcf941075a0 0x7fcf94100830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:35.911 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:35.909+0000 7fcf9259c640 1 -- 172.21.15.29:0/996949508 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fcf94104c70 msgr2=0x7fcf941002f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:48:35.911 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:35.909+0000 7fcf9259c640 1 --2- 172.21.15.29:0/996949508 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fcf94104c70 0x7fcf941002f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:35.911 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:35.909+0000 7fcf9259c640 1 -- 172.21.15.29:0/996949508 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf941010c0 con 0x7fcf94102340 2024-03-16T11:48:35.911 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:35.909+0000 7fcf9259c640 1 --2- 172.21.15.29:0/996949508 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcf94102340 0x7fcf94101cb0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fcf8400d950 tx=0x7fcf8400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:48:35.911 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:35.909+0000 7fcf8b7fe640 1 -- 172.21.15.29:0/996949508 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf84017070 con 0x7fcf94102340 2024-03-16T11:48:35.911 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:35.909+0000 7fcf8b7fe640 1 -- 172.21.15.29:0/996949508 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcf84009c30 con 0x7fcf94102340 2024-03-16T11:48:35.911 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:35.909+0000 7fcf8b7fe640 1 -- 172.21.15.29:0/996949508 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf8400f6b0 con 0x7fcf94102340 2024-03-16T11:48:35.911 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:35.909+0000 7fcf98df3640 1 -- 172.21.15.29:0/996949508 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcf941a1c30 con 0x7fcf94102340 2024-03-16T11:48:35.911 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:35.909+0000 7fcf98df3640 1 -- 172.21.15.29:0/996949508 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcf941a2180 con 0x7fcf94102340 2024-03-16T11:48:35.911 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:35.910+0000 7fcf98df3640 1 -- 172.21.15.29:0/996949508 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcf940fd780 con 0x7fcf94102340 2024-03-16T11:48:35.912 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:35.911+0000 7fcf8b7fe640 1 -- 172.21.15.29:0/996949508 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fcf840109d0 con 0x7fcf94102340 2024-03-16T11:48:35.912 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:35.912+0000 7fcf8b7fe640 1 --2- 172.21.15.29:0/996949508 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fcf800a2800 0x7fcf800a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:48:35.913 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:35.912+0000 7fcf8bfff640 1 --2- 172.21.15.29:0/996949508 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fcf800a2800 0x7fcf800a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:48:35.913 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:35.912+0000 7fcf8b7fe640 1 -- 172.21.15.29:0/996949508 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fcf840ce1c0 con 0x7fcf94102340 2024-03-16T11:48:35.913 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:35.913+0000 7fcf8bfff640 1 --2- 172.21.15.29:0/996949508 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fcf800a2800 0x7fcf800a4cc0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fcf94101a20 tx=0x7fcf7c00b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:48:35.916 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:35.917+0000 7fcf8b7fe640 1 -- 172.21.15.29:0/996949508 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fcf84009de0 con 0x7fcf94102340 2024-03-16T11:48:36.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:35 smithi029 ceph-mon[31975]: from='client.? 172.21.15.29:0/1369510760' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T11:48:36.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:35 smithi029 ceph-mon[31975]: from='client.? 172.21.15.29:0/67479729' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T11:48:36.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:35 smithi029 ceph-mon[31975]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T11:48:36.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:35 smithi029 ceph-mon[31975]: from='client.? 172.21.15.29:0/1672046312' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-16T11:48:36.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:35 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:48:36.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:35 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:48:36.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:35 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:48:36.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:35 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:48:36.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:35 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:48:36.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:35 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:48:36.105 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:36.106+0000 7fcf98df3640 1 -- 172.21.15.29:0/996949508 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7fcf9402d3a0 con 0x7fcf94102340 2024-03-16T11:48:36.106 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:36.107+0000 7fcf8b7fe640 1 -- 172.21.15.29:0/996949508 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v81) ==== 84+0+65 (secure 0 0 0) 0x7fcf8409b0b0 con 0x7fcf94102340 2024-03-16T11:48:36.106 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-16T11:48:36.106 INFO:teuthology.orchestra.run.smithi029.stdout:[ 2024-03-16T11:48:36.106 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-16T11:48:36.106 INFO:teuthology.orchestra.run.smithi029.stdout: "poolnum": 1, 2024-03-16T11:48:36.106 INFO:teuthology.orchestra.run.smithi029.stdout: "poolname": ".mgr" 2024-03-16T11:48:36.107 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T11:48:36.107 INFO:teuthology.orchestra.run.smithi029.stdout:] 2024-03-16T11:48:36.108 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:36.109+0000 7fcf98df3640 1 -- 172.21.15.29:0/996949508 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fcf800a2800 msgr2=0x7fcf800a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:48:36.108 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:36.109+0000 7fcf98df3640 1 --2- 172.21.15.29:0/996949508 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fcf800a2800 0x7fcf800a4cc0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fcf94101a20 tx=0x7fcf7c00b410 comp rx=0 tx=0).stop 2024-03-16T11:48:36.108 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:36.109+0000 7fcf98df3640 1 -- 172.21.15.29:0/996949508 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcf94102340 msgr2=0x7fcf94101cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:48:36.108 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:36.109+0000 7fcf98df3640 1 --2- 172.21.15.29:0/996949508 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcf94102340 0x7fcf94101cb0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fcf8400d950 tx=0x7fcf8400de10 comp rx=0 tx=0).stop 2024-03-16T11:48:36.108 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:36.109+0000 7fcf98df3640 1 -- 172.21.15.29:0/996949508 shutdown_connections 2024-03-16T11:48:36.109 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:36.109+0000 7fcf98df3640 1 --2- 172.21.15.29:0/996949508 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fcf800a2800 0x7fcf800a4cc0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:36.109 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:36.109+0000 7fcf98df3640 1 --2- 172.21.15.29:0/996949508 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcf941075a0 0x7fcf94100830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:36.109 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:36.109+0000 7fcf98df3640 1 --2- 172.21.15.29:0/996949508 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fcf94104c70 0x7fcf941002f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:36.109 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:36.109+0000 7fcf98df3640 1 --2- 172.21.15.29:0/996949508 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcf94102340 0x7fcf94101cb0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:36.109 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:36.109+0000 7fcf98df3640 1 -- 172.21.15.29:0/996949508 >> 172.21.15.29:0/996949508 conn(0x7fcf940fbc90 msgr2=0x7fcf94102c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:48:36.109 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:36.109+0000 7fcf98df3640 1 -- 172.21.15.29:0/996949508 shutdown_connections 2024-03-16T11:48:36.109 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:36.109+0000 7fcf98df3640 1 -- 172.21.15.29:0/996949508 wait complete. 2024-03-16T11:48:36.119 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-16T11:48:36.119 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-16T11:48:36.227 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:36.228+0000 7f61ef06e640 1 -- 172.21.15.29:0/3362766177 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f61e81074e0 msgr2=0x7f61e8109930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:48:36.227 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:36.228+0000 7f61ef06e640 1 --2- 172.21.15.29:0/3362766177 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f61e81074e0 0x7f61e8109930 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f61e0009ca0 tx=0x7f61e002c0f0 comp rx=0 tx=0).stop 2024-03-16T11:48:36.227 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:36.228+0000 7f61ef06e640 1 -- 172.21.15.29:0/3362766177 shutdown_connections 2024-03-16T11:48:36.227 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:36.228+0000 7f61ef06e640 1 --2- 172.21.15.29:0/3362766177 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f61e81074e0 0x7f61e8109930 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:36.227 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:36.228+0000 7f61ef06e640 1 --2- 172.21.15.29:0/3362766177 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f61e8104bb0 0x7f61e8106fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:36.227 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:36.228+0000 7f61ef06e640 1 --2- 172.21.15.29:0/3362766177 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f61e8102280 0x7f61e8104670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:36.228 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:36.228+0000 7f61ef06e640 1 -- 172.21.15.29:0/3362766177 >> 172.21.15.29:0/3362766177 conn(0x7f61e80fbc90 msgr2=0x7f61e80fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:48:36.228 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:36.228+0000 7f61ef06e640 1 -- 172.21.15.29:0/3362766177 shutdown_connections 2024-03-16T11:48:36.228 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:36.228+0000 7f61ef06e640 1 -- 172.21.15.29:0/3362766177 wait complete. 2024-03-16T11:48:36.230 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:36.229+0000 7f61ef06e640 1 Processor -- start 2024-03-16T11:48:36.230 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:36.229+0000 7f61ef06e640 1 -- start start 2024-03-16T11:48:36.230 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:36.229+0000 7f61ef06e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f61e8102280 0x7f61e8197ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:48:36.230 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:36.229+0000 7f61ef06e640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f61e8104bb0 0x7f61e8198010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:48:36.230 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:36.229+0000 7f61ef06e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f61e81074e0 0x7f61e819f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:48:36.230 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:36.229+0000 7f61ef06e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f61e810d480 con 0x7f61e8102280 2024-03-16T11:48:36.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:36.229+0000 7f61ef06e640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f61e810d300 con 0x7f61e8104bb0 2024-03-16T11:48:36.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:36.229+0000 7f61ef06e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f61e810d600 con 0x7f61e81074e0 2024-03-16T11:48:36.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:36.229+0000 7f61ece0a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f61e8102280 0x7f61e8197ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:48:36.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:36.229+0000 7f61ece0a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f61e8102280 0x7f61e8197ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60474/0 (socket says 172.21.15.29:60474) 2024-03-16T11:48:36.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:36.229+0000 7f61ece0a640 1 -- 172.21.15.29:0/1178882583 learned_addr learned my addr 172.21.15.29:0/1178882583 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:48:36.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:36.229+0000 7f61e7fff640 1 --2- 172.21.15.29:0/1178882583 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f61e8104bb0 0x7f61e8198010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:48:36.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:36.229+0000 7f61ed60b640 1 --2- 172.21.15.29:0/1178882583 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f61e81074e0 0x7f61e819f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:48:36.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:36.229+0000 7f61ece0a640 1 -- 172.21.15.29:0/1178882583 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f61e81074e0 msgr2=0x7f61e819f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:48:36.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:36.229+0000 7f61ece0a640 1 --2- 172.21.15.29:0/1178882583 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f61e81074e0 0x7f61e819f080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:36.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:36.229+0000 7f61ece0a640 1 -- 172.21.15.29:0/1178882583 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f61e8104bb0 msgr2=0x7f61e8198010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:48:36.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:36.229+0000 7f61ece0a640 1 --2- 172.21.15.29:0/1178882583 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f61e8104bb0 0x7f61e8198010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:36.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:36.229+0000 7f61ece0a640 1 -- 172.21.15.29:0/1178882583 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61e819f6f0 con 0x7f61e8102280 2024-03-16T11:48:36.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:36.229+0000 7f61ed60b640 1 --2- 172.21.15.29:0/1178882583 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f61e81074e0 0x7f61e819f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T11:48:36.232 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:36.229+0000 7f61ece0a640 1 --2- 172.21.15.29:0/1178882583 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f61e8102280 0x7f61e8197ad0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f61dc00e9f0 tx=0x7f61dc00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:48:36.232 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:36.230+0000 7f61e5ffb640 1 -- 172.21.15.29:0/1178882583 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61dc00cd80 con 0x7f61e8102280 2024-03-16T11:48:36.232 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:36.230+0000 7f61e5ffb640 1 -- 172.21.15.29:0/1178882583 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f61dc010910 con 0x7f61e8102280 2024-03-16T11:48:36.232 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:36.230+0000 7f61ef06e640 1 -- 172.21.15.29:0/1178882583 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f61e819f9e0 con 0x7f61e8102280 2024-03-16T11:48:36.232 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:36.230+0000 7f61e5ffb640 1 -- 172.21.15.29:0/1178882583 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61dc018990 con 0x7f61e8102280 2024-03-16T11:48:36.232 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:36.230+0000 7f61ef06e640 1 -- 172.21.15.29:0/1178882583 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f61e819ff80 con 0x7f61e8102280 2024-03-16T11:48:36.232 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:36.231+0000 7f61ef06e640 1 -- 172.21.15.29:0/1178882583 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f61e80fd780 con 0x7f61e8102280 2024-03-16T11:48:36.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:36.232+0000 7f61e5ffb640 1 -- 172.21.15.29:0/1178882583 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f61dc010ac0 con 0x7f61e8102280 2024-03-16T11:48:36.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:36.233+0000 7f61e5ffb640 1 --2- 172.21.15.29:0/1178882583 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f61d80a2880 0x7f61d80a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:48:36.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:36.233+0000 7f61e7fff640 1 --2- 172.21.15.29:0/1178882583 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f61d80a2880 0x7f61d80a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:48:36.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:36.233+0000 7f61e5ffb640 1 -- 172.21.15.29:0/1178882583 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f61dc0ce9c0 con 0x7f61e8102280 2024-03-16T11:48:36.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:36.233+0000 7f61e7fff640 1 --2- 172.21.15.29:0/1178882583 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f61d80a2880 0x7f61d80a4d40 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f61e8198fe0 tx=0x7f61d4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:48:36.236 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:36.237+0000 7f61e5ffb640 1 -- 172.21.15.29:0/1178882583 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f61dc09be60 con 0x7f61e8102280 2024-03-16T11:48:36.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:35 smithi154 ceph-mon[36320]: from='client.? 172.21.15.29:0/1369510760' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T11:48:36.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:35 smithi154 ceph-mon[36320]: from='client.? 172.21.15.29:0/67479729' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T11:48:36.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:35 smithi154 ceph-mon[36320]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T11:48:36.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:35 smithi154 ceph-mon[36320]: from='client.? 172.21.15.29:0/1672046312' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-16T11:48:36.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:35 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:48:36.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:35 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:48:36.296 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:35 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:48:36.296 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:35 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:48:36.296 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:35 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:48:36.296 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:35 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:48:36.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:35 smithi152 ceph-mon[36291]: from='client.? 172.21.15.29:0/1369510760' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T11:48:36.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:35 smithi152 ceph-mon[36291]: from='client.? 172.21.15.29:0/67479729' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T11:48:36.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:35 smithi152 ceph-mon[36291]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T11:48:36.378 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:35 smithi152 ceph-mon[36291]: from='client.? 172.21.15.29:0/1672046312' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-16T11:48:36.378 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:35 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:48:36.378 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:35 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:48:36.378 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:35 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:48:36.378 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:35 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:48:36.378 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:35 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:48:36.378 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:35 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:48:36.460 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:36.461+0000 7f61ef06e640 1 -- 172.21.15.29:0/1178882583 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f61e802d1b0 con 0x7f61e8102280 2024-03-16T11:48:37.001 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:37.002+0000 7f61e5ffb640 1 -- 172.21.15.29:0/1178882583 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v82) ==== 138+0+0 (secure 0 0 0) 0x7f61e802d1b0 con 0x7f61e8102280 2024-03-16T11:48:37.002 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'cephfs_metadata' created 2024-03-16T11:48:37.004 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:37.004+0000 7f61ef06e640 1 -- 172.21.15.29:0/1178882583 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f61d80a2880 msgr2=0x7f61d80a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:48:37.004 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:37.004+0000 7f61ef06e640 1 --2- 172.21.15.29:0/1178882583 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f61d80a2880 0x7f61d80a4d40 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f61e8198fe0 tx=0x7f61d4005f90 comp rx=0 tx=0).stop 2024-03-16T11:48:37.004 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:37.005+0000 7f61ef06e640 1 -- 172.21.15.29:0/1178882583 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f61e8102280 msgr2=0x7f61e8197ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:48:37.004 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:37.005+0000 7f61ef06e640 1 --2- 172.21.15.29:0/1178882583 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f61e8102280 0x7f61e8197ad0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f61dc00e9f0 tx=0x7f61dc00eeb0 comp rx=0 tx=0).stop 2024-03-16T11:48:37.004 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:37.005+0000 7f61ef06e640 1 -- 172.21.15.29:0/1178882583 shutdown_connections 2024-03-16T11:48:37.004 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:37.005+0000 7f61ef06e640 1 --2- 172.21.15.29:0/1178882583 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f61d80a2880 0x7f61d80a4d40 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:37.004 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:37.005+0000 7f61ef06e640 1 --2- 172.21.15.29:0/1178882583 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f61e81074e0 0x7f61e819f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:37.005 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:37.005+0000 7f61ef06e640 1 --2- 172.21.15.29:0/1178882583 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f61e8104bb0 0x7f61e8198010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:37.005 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:37.005+0000 7f61ef06e640 1 --2- 172.21.15.29:0/1178882583 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f61e8102280 0x7f61e8197ad0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:37.005 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:37.005+0000 7f61ef06e640 1 -- 172.21.15.29:0/1178882583 >> 172.21.15.29:0/1178882583 conn(0x7f61e80fbc90 msgr2=0x7f61e8102b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:48:37.005 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:37.005+0000 7f61ef06e640 1 -- 172.21.15.29:0/1178882583 shutdown_connections 2024-03-16T11:48:37.005 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:37.005+0000 7f61ef06e640 1 -- 172.21.15.29:0/1178882583 wait complete. 2024-03-16T11:48:37.019 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-16T11:48:37.130 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:37.130+0000 7f69f4f9d640 1 -- 172.21.15.29:0/2288535069 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f69f01005f0 msgr2=0x7f69f0100a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:48:37.130 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:37.130+0000 7f69f4f9d640 1 --2- 172.21.15.29:0/2288535069 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f69f01005f0 0x7f69f0100a10 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f69e0009900 tx=0x7f69e002bc30 comp rx=0 tx=0).stop 2024-03-16T11:48:37.130 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:37.130+0000 7f69f4f9d640 1 -- 172.21.15.29:0/2288535069 shutdown_connections 2024-03-16T11:48:37.130 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:37.130+0000 7f69f4f9d640 1 --2- 172.21.15.29:0/2288535069 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f69f01021a0 0x7f69f0108a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:37.130 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:37.130+0000 7f69f4f9d640 1 --2- 172.21.15.29:0/2288535069 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f69f01017e0 0x7f69f0101c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:37.130 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:37.130+0000 7f69f4f9d640 1 --2- 172.21.15.29:0/2288535069 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f69f01005f0 0x7f69f0100a10 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:37.130 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:37.130+0000 7f69f4f9d640 1 -- 172.21.15.29:0/2288535069 >> 172.21.15.29:0/2288535069 conn(0x7f69f00fbb60 msgr2=0x7f69f00fdfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:48:37.130 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:37.131+0000 7f69f4f9d640 1 -- 172.21.15.29:0/2288535069 shutdown_connections 2024-03-16T11:48:37.130 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:37.131+0000 7f69f4f9d640 1 -- 172.21.15.29:0/2288535069 wait complete. 2024-03-16T11:48:37.133 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:37.131+0000 7f69f4f9d640 1 Processor -- start 2024-03-16T11:48:37.133 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:37.131+0000 7f69f4f9d640 1 -- start start 2024-03-16T11:48:37.133 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:37.131+0000 7f69f4f9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f69f01005f0 0x7f69f01968d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:48:37.133 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:37.132+0000 7f69f4f9d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f69f01017e0 0x7f69f0196e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:48:37.133 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:37.132+0000 7f69f4f9d640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f69f01021a0 0x7f69f019de80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:48:37.133 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:37.132+0000 7f69f4f9d640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f69f010c260 con 0x7f69f01005f0 2024-03-16T11:48:37.133 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:37.132+0000 7f69f4f9d640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f69f010c0e0 con 0x7f69f01021a0 2024-03-16T11:48:37.133 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:37.132+0000 7f69f4f9d640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f69f010c3e0 con 0x7f69f01017e0 2024-03-16T11:48:37.133 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:37.132+0000 7f69ef7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f69f01005f0 0x7f69f01968d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:48:37.133 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:37.132+0000 7f69ef7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f69f01005f0 0x7f69f01968d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60482/0 (socket says 172.21.15.29:60482) 2024-03-16T11:48:37.134 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:37.132+0000 7f69ef7fe640 1 -- 172.21.15.29:0/1835495197 learned_addr learned my addr 172.21.15.29:0/1835495197 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:48:37.134 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:37.132+0000 7f69e7fff640 1 --2- 172.21.15.29:0/1835495197 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f69f01017e0 0x7f69f0196e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:48:37.134 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:37.132+0000 7f69effff640 1 --2- 172.21.15.29:0/1835495197 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f69f01021a0 0x7f69f019de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:48:37.134 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:37.132+0000 7f69ef7fe640 1 -- 172.21.15.29:0/1835495197 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f69f01017e0 msgr2=0x7f69f0196e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:48:37.134 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:37.132+0000 7f69ef7fe640 1 --2- 172.21.15.29:0/1835495197 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f69f01017e0 0x7f69f0196e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:37.134 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:37.132+0000 7f69ef7fe640 1 -- 172.21.15.29:0/1835495197 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f69f01021a0 msgr2=0x7f69f019de80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:48:37.134 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:37.132+0000 7f69ef7fe640 1 --2- 172.21.15.29:0/1835495197 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f69f01021a0 0x7f69f019de80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:37.134 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:37.132+0000 7f69ef7fe640 1 -- 172.21.15.29:0/1835495197 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69f019e4f0 con 0x7f69f01005f0 2024-03-16T11:48:37.134 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:37.132+0000 7f69ef7fe640 1 --2- 172.21.15.29:0/1835495197 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f69f01005f0 0x7f69f01968d0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f69e0000c00 tx=0x7f69e0005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:48:37.134 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:37.132+0000 7f69ed7fa640 1 -- 172.21.15.29:0/1835495197 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69e0041070 con 0x7f69f01005f0 2024-03-16T11:48:37.134 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:37.132+0000 7f69f4f9d640 1 -- 172.21.15.29:0/1835495197 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f69f019e780 con 0x7f69f01005f0 2024-03-16T11:48:37.134 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:37.132+0000 7f69ed7fa640 1 -- 172.21.15.29:0/1835495197 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f69e00044d0 con 0x7f69f01005f0 2024-03-16T11:48:37.134 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:37.132+0000 7f69ed7fa640 1 -- 172.21.15.29:0/1835495197 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69e0034440 con 0x7f69f01005f0 2024-03-16T11:48:37.134 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:37.132+0000 7f69f4f9d640 1 -- 172.21.15.29:0/1835495197 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f69f019ecc0 con 0x7f69f01005f0 2024-03-16T11:48:37.135 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:37.133+0000 7f69f4f9d640 1 -- 172.21.15.29:0/1835495197 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f69f0107910 con 0x7f69f01005f0 2024-03-16T11:48:37.135 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:37.135+0000 7f69ed7fa640 1 -- 172.21.15.29:0/1835495197 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f69e0044070 con 0x7f69f01005f0 2024-03-16T11:48:37.135 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:37.135+0000 7f69ed7fa640 1 --2- 172.21.15.29:0/1835495197 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f69dc0a2800 0x7f69dc0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:48:37.135 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:37.136+0000 7f69e7fff640 1 --2- 172.21.15.29:0/1835495197 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f69dc0a2800 0x7f69dc0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:48:37.136 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:37.136+0000 7f69ed7fa640 1 -- 172.21.15.29:0/1835495197 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7497+0+0 (secure 0 0 0) 0x7f69e00f04a0 con 0x7f69f01005f0 2024-03-16T11:48:37.136 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:37.136+0000 7f69e7fff640 1 --2- 172.21.15.29:0/1835495197 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f69dc0a2800 0x7f69dc0a4cc0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f69f0197de0 tx=0x7f69d800b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:48:37.138 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:37.139+0000 7f69ed7fa640 1 -- 172.21.15.29:0/1835495197 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f69e00bd1c0 con 0x7f69f01005f0 2024-03-16T11:48:37.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:36 smithi154 ceph-mon[36320]: from='client.? 172.21.15.29:0/996949508' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-03-16T11:48:37.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:36 smithi154 ceph-mon[36320]: from='client.? 172.21.15.29:0/1178882583' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-03-16T11:48:37.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:36 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:48:37.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:36 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T11:48:37.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:36 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:48:37.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:36 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T11:48:37.324 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:37.325+0000 7f69f4f9d640 1 -- 172.21.15.29:0/1835495197 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f69f002d1b0 con 0x7f69f01005f0 2024-03-16T11:48:37.337 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:36 smithi029 ceph-mon[31975]: from='client.? 172.21.15.29:0/996949508' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-03-16T11:48:37.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:36 smithi029 ceph-mon[31975]: from='client.? 172.21.15.29:0/1178882583' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-03-16T11:48:37.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:36 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:48:37.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:36 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T11:48:37.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:36 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:48:37.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:36 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T11:48:37.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:36 smithi152 ceph-mon[36291]: from='client.? 172.21.15.29:0/996949508' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-03-16T11:48:37.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:36 smithi152 ceph-mon[36291]: from='client.? 172.21.15.29:0/1178882583' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-03-16T11:48:37.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:36 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:48:37.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:36 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T11:48:37.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:36 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:48:37.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:36 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T11:48:38.007 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:38.007+0000 7f69ed7fa640 1 -- 172.21.15.29:0/1835495197 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v83) ==== 170+0+0 (secure 0 0 0) 0x7f69e00c1260 con 0x7f69f01005f0 2024-03-16T11:48:38.007 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'cephfs_data' created 2024-03-16T11:48:38.010 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:38.011+0000 7f69f4f9d640 1 -- 172.21.15.29:0/1835495197 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f69dc0a2800 msgr2=0x7f69dc0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:48:38.010 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:38.011+0000 7f69f4f9d640 1 --2- 172.21.15.29:0/1835495197 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f69dc0a2800 0x7f69dc0a4cc0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f69f0197de0 tx=0x7f69d800b410 comp rx=0 tx=0).stop 2024-03-16T11:48:38.010 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:38.011+0000 7f69f4f9d640 1 -- 172.21.15.29:0/1835495197 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f69f01005f0 msgr2=0x7f69f01968d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:48:38.010 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:38.011+0000 7f69f4f9d640 1 --2- 172.21.15.29:0/1835495197 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f69f01005f0 0x7f69f01968d0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f69e0000c00 tx=0x7f69e0005d50 comp rx=0 tx=0).stop 2024-03-16T11:48:38.010 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:38.011+0000 7f69f4f9d640 1 -- 172.21.15.29:0/1835495197 shutdown_connections 2024-03-16T11:48:38.011 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:38.011+0000 7f69f4f9d640 1 --2- 172.21.15.29:0/1835495197 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f69dc0a2800 0x7f69dc0a4cc0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:38.011 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:38.011+0000 7f69f4f9d640 1 --2- 172.21.15.29:0/1835495197 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f69f01021a0 0x7f69f019de80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:38.011 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:38.011+0000 7f69f4f9d640 1 --2- 172.21.15.29:0/1835495197 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f69f01017e0 0x7f69f0196e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:38.011 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:38.011+0000 7f69f4f9d640 1 --2- 172.21.15.29:0/1835495197 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f69f01005f0 0x7f69f01968d0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:38.011 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:38.011+0000 7f69f4f9d640 1 -- 172.21.15.29:0/1835495197 >> 172.21.15.29:0/1835495197 conn(0x7f69f00fbb60 msgr2=0x7f69f0104ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:48:38.011 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:38.011+0000 7f69f4f9d640 1 -- 172.21.15.29:0/1835495197 shutdown_connections 2024-03-16T11:48:38.011 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:38.011+0000 7f69f4f9d640 1 -- 172.21.15.29:0/1835495197 wait complete. 2024-03-16T11:48:38.030 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-16T11:48:38.139 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:38.138+0000 7f8c9eb90640 1 -- 172.21.15.29:0/961083494 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8c981033a0 msgr2=0x7f8c98109c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:48:38.140 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:38.138+0000 7f8c9eb90640 1 --2- 172.21.15.29:0/961083494 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8c981033a0 0x7f8c98109c30 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f8c88009900 tx=0x7f8c8802bc30 comp rx=0 tx=0).stop 2024-03-16T11:48:38.140 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:38.138+0000 7f8c9eb90640 1 -- 172.21.15.29:0/961083494 shutdown_connections 2024-03-16T11:48:38.140 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:38.138+0000 7f8c9eb90640 1 --2- 172.21.15.29:0/961083494 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8c981033a0 0x7f8c98109c30 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:38.140 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:38.138+0000 7f8c9eb90640 1 --2- 172.21.15.29:0/961083494 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8c981029e0 0x7f8c98102e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:38.140 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:38.138+0000 7f8c9eb90640 1 --2- 172.21.15.29:0/961083494 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8c981017f0 0x7f8c98101c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:38.140 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:38.138+0000 7f8c9eb90640 1 -- 172.21.15.29:0/961083494 >> 172.21.15.29:0/961083494 conn(0x7f8c980fcd80 msgr2=0x7f8c980ff1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:48:38.140 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:38.138+0000 7f8c9eb90640 1 -- 172.21.15.29:0/961083494 shutdown_connections 2024-03-16T11:48:38.140 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:38.138+0000 7f8c9eb90640 1 -- 172.21.15.29:0/961083494 wait complete. 2024-03-16T11:48:38.140 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:38.138+0000 7f8c9eb90640 1 Processor -- start 2024-03-16T11:48:38.140 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:38.138+0000 7f8c9eb90640 1 -- start start 2024-03-16T11:48:38.140 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:38.139+0000 7f8c9eb90640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8c981017f0 0x7f8c98197ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:48:38.140 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:38.139+0000 7f8c9eb90640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8c981029e0 0x7f8c98197ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:48:38.141 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:38.139+0000 7f8c9eb90640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8c981033a0 0x7f8c9819f060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:48:38.141 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:38.139+0000 7f8c9eb90640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f8c9810d470 con 0x7f8c981029e0 2024-03-16T11:48:38.141 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:38.139+0000 7f8c9eb90640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f8c9810d2f0 con 0x7f8c981033a0 2024-03-16T11:48:38.141 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:38.139+0000 7f8c9eb90640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f8c9810d5f0 con 0x7f8c981017f0 2024-03-16T11:48:38.141 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:38.139+0000 7f8c97fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8c981029e0 0x7f8c98197ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:48:38.141 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:38.139+0000 7f8c97fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8c981029e0 0x7f8c98197ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60498/0 (socket says 172.21.15.29:60498) 2024-03-16T11:48:38.141 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:38.139+0000 7f8c97fff640 1 -- 172.21.15.29:0/2925241401 learned_addr learned my addr 172.21.15.29:0/2925241401 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:48:38.141 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:38.139+0000 7f8c9c92c640 1 --2- 172.21.15.29:0/2925241401 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8c981017f0 0x7f8c98197ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:48:38.141 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:38.139+0000 7f8c9d12d640 1 --2- 172.21.15.29:0/2925241401 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8c981033a0 0x7f8c9819f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:48:38.141 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:38.139+0000 7f8c97fff640 1 -- 172.21.15.29:0/2925241401 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8c981017f0 msgr2=0x7f8c98197ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:48:38.141 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:38.139+0000 7f8c97fff640 1 --2- 172.21.15.29:0/2925241401 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8c981017f0 0x7f8c98197ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:38.142 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:38.139+0000 7f8c97fff640 1 -- 172.21.15.29:0/2925241401 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8c981033a0 msgr2=0x7f8c9819f060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:48:38.142 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:38.139+0000 7f8c97fff640 1 --2- 172.21.15.29:0/2925241401 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8c981033a0 0x7f8c9819f060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:38.142 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:38.139+0000 7f8c97fff640 1 -- 172.21.15.29:0/2925241401 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c9819f6d0 con 0x7f8c981029e0 2024-03-16T11:48:38.142 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:38.139+0000 7f8c97fff640 1 --2- 172.21.15.29:0/2925241401 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8c981029e0 0x7f8c98197ff0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f8c8000e9f0 tx=0x7f8c8000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:48:38.142 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:38.139+0000 7f8c95ffb640 1 -- 172.21.15.29:0/2925241401 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c8000cd80 con 0x7f8c981029e0 2024-03-16T11:48:38.142 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:38.139+0000 7f8c9eb90640 1 -- 172.21.15.29:0/2925241401 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8c9819f9c0 con 0x7f8c981029e0 2024-03-16T11:48:38.142 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:38.139+0000 7f8c95ffb640 1 -- 172.21.15.29:0/2925241401 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8c80005500 con 0x7f8c981029e0 2024-03-16T11:48:38.142 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:38.139+0000 7f8c95ffb640 1 -- 172.21.15.29:0/2925241401 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c80010690 con 0x7f8c981029e0 2024-03-16T11:48:38.142 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:38.140+0000 7f8c9eb90640 1 -- 172.21.15.29:0/2925241401 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8c9819ff10 con 0x7f8c981029e0 2024-03-16T11:48:38.142 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:38.141+0000 7f8c9eb90640 1 -- 172.21.15.29:0/2925241401 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8c98108b10 con 0x7f8c981029e0 2024-03-16T11:48:38.143 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:38.141+0000 7f8c95ffb640 1 -- 172.21.15.29:0/2925241401 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f8c800046c0 con 0x7f8c981029e0 2024-03-16T11:48:38.143 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:38.142+0000 7f8c95ffb640 1 --2- 172.21.15.29:0/2925241401 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f8c840a2800 0x7f8c840a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:48:38.143 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:38.142+0000 7f8c95ffb640 1 -- 172.21.15.29:0/2925241401 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7879+0+0 (secure 0 0 0) 0x7f8c800cec90 con 0x7f8c981029e0 2024-03-16T11:48:38.143 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:38.142+0000 7f8c9c92c640 1 --2- 172.21.15.29:0/2925241401 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f8c840a2800 0x7f8c840a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:48:38.143 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:38.143+0000 7f8c9c92c640 1 --2- 172.21.15.29:0/2925241401 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f8c840a2800 0x7f8c840a4cc0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f8c98102840 tx=0x7f8c90009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:48:38.146 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:38.147+0000 7f8c95ffb640 1 -- 172.21.15.29:0/2925241401 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f8c8009bee0 con 0x7f8c981029e0 2024-03-16T11:48:38.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:38 smithi154 ceph-mon[36320]: from='client.? 172.21.15.29:0/1178882583' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-03-16T11:48:38.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:38 smithi154 ceph-mon[36320]: osdmap e82: 12 total, 12 up, 12 in 2024-03-16T11:48:38.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:38 smithi154 ceph-mon[36320]: from='client.? 172.21.15.29:0/1835495197' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-03-16T11:48:38.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:38 smithi154 ceph-mon[36320]: pgmap v287: 33 pgs: 3 creating+peering, 1 active+clean, 29 unknown; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T11:48:38.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:38 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:48:38.337 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:38 smithi029 ceph-mon[31975]: from='client.? 172.21.15.29:0/1178882583' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-03-16T11:48:38.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:38 smithi029 ceph-mon[31975]: osdmap e82: 12 total, 12 up, 12 in 2024-03-16T11:48:38.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:38 smithi029 ceph-mon[31975]: from='client.? 172.21.15.29:0/1835495197' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-03-16T11:48:38.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:38 smithi029 ceph-mon[31975]: pgmap v287: 33 pgs: 3 creating+peering, 1 active+clean, 29 unknown; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T11:48:38.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:38 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:48:38.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:38 smithi152 ceph-mon[36291]: from='client.? 172.21.15.29:0/1178882583' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-03-16T11:48:38.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:38 smithi152 ceph-mon[36291]: osdmap e82: 12 total, 12 up, 12 in 2024-03-16T11:48:38.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:38 smithi152 ceph-mon[36291]: from='client.? 172.21.15.29:0/1835495197' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-03-16T11:48:38.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:38 smithi152 ceph-mon[36291]: pgmap v287: 33 pgs: 3 creating+peering, 1 active+clean, 29 unknown; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T11:48:38.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:38 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:48:38.428 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:38.429+0000 7f8c9eb90640 1 -- 172.21.15.29:0/2925241401 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f8c98062450 con 0x7f8c981029e0 2024-03-16T11:48:39.008 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:39.009+0000 7f8c95ffb640 1 -- 172.21.15.29:0/2925241401 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v14) ==== 172+0+0 (secure 0 0 0) 0x7f8c8009b7e0 con 0x7f8c981029e0 2024-03-16T11:48:39.008 INFO:teuthology.orchestra.run.smithi029.stderr:new fs with metadata pool 2 and data pool 3 2024-03-16T11:48:39.011 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:39.011+0000 7f8c9eb90640 1 -- 172.21.15.29:0/2925241401 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f8c840a2800 msgr2=0x7f8c840a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:48:39.011 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:39.011+0000 7f8c9eb90640 1 --2- 172.21.15.29:0/2925241401 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f8c840a2800 0x7f8c840a4cc0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f8c98102840 tx=0x7f8c90009500 comp rx=0 tx=0).stop 2024-03-16T11:48:39.011 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:39.011+0000 7f8c9eb90640 1 -- 172.21.15.29:0/2925241401 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8c981029e0 msgr2=0x7f8c98197ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:48:39.011 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:39.011+0000 7f8c9eb90640 1 --2- 172.21.15.29:0/2925241401 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8c981029e0 0x7f8c98197ff0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f8c8000e9f0 tx=0x7f8c8000edf0 comp rx=0 tx=0).stop 2024-03-16T11:48:39.011 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:39.011+0000 7f8c9eb90640 1 -- 172.21.15.29:0/2925241401 shutdown_connections 2024-03-16T11:48:39.011 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:39.011+0000 7f8c9eb90640 1 --2- 172.21.15.29:0/2925241401 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f8c840a2800 0x7f8c840a4cc0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:39.011 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:39.011+0000 7f8c9eb90640 1 --2- 172.21.15.29:0/2925241401 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8c981033a0 0x7f8c9819f060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:39.011 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:39.011+0000 7f8c9eb90640 1 --2- 172.21.15.29:0/2925241401 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8c981029e0 0x7f8c98197ff0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:39.011 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:39.011+0000 7f8c9eb90640 1 --2- 172.21.15.29:0/2925241401 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8c981017f0 0x7f8c98197ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:39.011 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:39.011+0000 7f8c9eb90640 1 -- 172.21.15.29:0/2925241401 >> 172.21.15.29:0/2925241401 conn(0x7f8c980fcd80 msgr2=0x7f8c98105ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:48:39.011 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:39.011+0000 7f8c9eb90640 1 -- 172.21.15.29:0/2925241401 shutdown_connections 2024-03-16T11:48:39.011 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:39.011+0000 7f8c9eb90640 1 -- 172.21.15.29:0/2925241401 wait complete. 2024-03-16T11:48:39.025 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-16T11:48:39.131 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:39.130+0000 7fb34ffff640 1 -- 172.21.15.29:0/1740077995 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb350102810 msgr2=0x7fb350102c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:48:39.131 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:39.130+0000 7fb34ffff640 1 --2- 172.21.15.29:0/1740077995 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb350102810 0x7fb350102c90 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fb338009900 tx=0x7fb33802bc30 comp rx=0 tx=0).stop 2024-03-16T11:48:39.131 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:39.130+0000 7fb34ffff640 1 -- 172.21.15.29:0/1740077995 shutdown_connections 2024-03-16T11:48:39.131 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:39.130+0000 7fb34ffff640 1 --2- 172.21.15.29:0/1740077995 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb3501031d0 0x7fb350109a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:39.132 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:39.130+0000 7fb34ffff640 1 --2- 172.21.15.29:0/1740077995 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb350102810 0x7fb350102c90 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:39.132 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:39.130+0000 7fb34ffff640 1 --2- 172.21.15.29:0/1740077995 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb350101620 0x7fb350101a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:39.132 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:39.130+0000 7fb34ffff640 1 -- 172.21.15.29:0/1740077995 >> 172.21.15.29:0/1740077995 conn(0x7fb3500fcb70 msgr2=0x7fb3500fefd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:48:39.132 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:39.130+0000 7fb34ffff640 1 -- 172.21.15.29:0/1740077995 shutdown_connections 2024-03-16T11:48:39.132 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:39.130+0000 7fb34ffff640 1 -- 172.21.15.29:0/1740077995 wait complete. 2024-03-16T11:48:39.132 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:39.130+0000 7fb34ffff640 1 Processor -- start 2024-03-16T11:48:39.132 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:39.131+0000 7fb34ffff640 1 -- start start 2024-03-16T11:48:39.132 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:39.131+0000 7fb34ffff640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb350101620 0x7fb3501979e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:48:39.132 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:39.131+0000 7fb34ffff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb350102810 0x7fb350197f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:48:39.132 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:39.131+0000 7fb34ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb3501031d0 0x7fb35019eef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:48:39.132 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:39.131+0000 7fb34ffff640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fb35010d2b0 con 0x7fb3501031d0 2024-03-16T11:48:39.132 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:39.131+0000 7fb34ffff640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fb35010d130 con 0x7fb350101620 2024-03-16T11:48:39.133 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:39.131+0000 7fb34ffff640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fb35010d430 con 0x7fb350102810 2024-03-16T11:48:39.133 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:39.131+0000 7fb34f7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb3501031d0 0x7fb35019eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:48:39.133 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:39.131+0000 7fb34f7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb3501031d0 0x7fb35019eef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60510/0 (socket says 172.21.15.29:60510) 2024-03-16T11:48:39.133 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:39.131+0000 7fb34f7fe640 1 -- 172.21.15.29:0/3698788447 learned_addr learned my addr 172.21.15.29:0/3698788447 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:48:39.133 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:39.131+0000 7fb34e7fc640 1 --2- 172.21.15.29:0/3698788447 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb350102810 0x7fb350197f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:48:39.133 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:39.131+0000 7fb34effd640 1 --2- 172.21.15.29:0/3698788447 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb350101620 0x7fb3501979e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:48:39.133 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:39.131+0000 7fb34f7fe640 1 -- 172.21.15.29:0/3698788447 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb350102810 msgr2=0x7fb350197f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:48:39.133 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:39.131+0000 7fb34f7fe640 1 --2- 172.21.15.29:0/3698788447 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb350102810 0x7fb350197f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:39.133 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:39.131+0000 7fb34f7fe640 1 -- 172.21.15.29:0/3698788447 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb350101620 msgr2=0x7fb3501979e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:48:39.133 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:39.131+0000 7fb34f7fe640 1 --2- 172.21.15.29:0/3698788447 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb350101620 0x7fb3501979e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:39.133 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:39.131+0000 7fb34f7fe640 1 -- 172.21.15.29:0/3698788447 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb35019f560 con 0x7fb3501031d0 2024-03-16T11:48:39.133 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:39.131+0000 7fb34e7fc640 1 --2- 172.21.15.29:0/3698788447 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb350102810 0x7fb350197f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T11:48:39.133 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:39.131+0000 7fb34f7fe640 1 --2- 172.21.15.29:0/3698788447 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb3501031d0 0x7fb35019eef0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fb34000e970 tx=0x7fb34000ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:48:39.134 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:39.131+0000 7fb347fff640 1 -- 172.21.15.29:0/3698788447 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb34000ccc0 con 0x7fb3501031d0 2024-03-16T11:48:39.134 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:39.132+0000 7fb347fff640 1 -- 172.21.15.29:0/3698788447 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb34000ce60 con 0x7fb3501031d0 2024-03-16T11:48:39.134 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:39.132+0000 7fb34ffff640 1 -- 172.21.15.29:0/3698788447 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb35019f850 con 0x7fb3501031d0 2024-03-16T11:48:39.134 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:39.132+0000 7fb347fff640 1 -- 172.21.15.29:0/3698788447 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb340010490 con 0x7fb3501031d0 2024-03-16T11:48:39.134 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:39.132+0000 7fb34ffff640 1 -- 172.21.15.29:0/3698788447 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb35019fd70 con 0x7fb3501031d0 2024-03-16T11:48:39.135 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:39.133+0000 7fb34ffff640 1 -- 172.21.15.29:0/3698788447 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb350108940 con 0x7fb3501031d0 2024-03-16T11:48:39.135 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:39.134+0000 7fb347fff640 1 -- 172.21.15.29:0/3698788447 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fb340005020 con 0x7fb3501031d0 2024-03-16T11:48:39.135 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:39.134+0000 7fb347fff640 1 --2- 172.21.15.29:0/3698788447 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fb33c0a2800 0x7fb33c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:48:39.135 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:39.135+0000 7fb34effd640 1 --2- 172.21.15.29:0/3698788447 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fb33c0a2800 0x7fb33c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:48:39.135 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:39.135+0000 7fb347fff640 1 -- 172.21.15.29:0/3698788447 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 7979+0+0 (secure 0 0 0) 0x7fb3400cedb0 con 0x7fb3501031d0 2024-03-16T11:48:39.135 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:39.135+0000 7fb34effd640 1 --2- 172.21.15.29:0/3698788447 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fb33c0a2800 0x7fb33c0a4cc0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fb350102670 tx=0x7fb348009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:48:39.138 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:39.139+0000 7fb347fff640 1 -- 172.21.15.29:0/3698788447 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fb34009bfa0 con 0x7fb3501031d0 2024-03-16T11:48:39.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:39 smithi154 ceph-mon[36320]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-16T11:48:39.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:39 smithi154 ceph-mon[36320]: from='client.? 172.21.15.29:0/1835495197' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-03-16T11:48:39.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:39 smithi154 ceph-mon[36320]: osdmap e83: 12 total, 12 up, 12 in 2024-03-16T11:48:39.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:39 smithi154 ceph-mon[36320]: from='client.? 172.21.15.29:0/2925241401' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-03-16T11:48:39.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:39 smithi154 ceph-mon[36320]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-03-16T11:48:39.296 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:39 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:48:39.296 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:39 smithi154 ceph-mon[36320]: osdmap e84: 12 total, 12 up, 12 in 2024-03-16T11:48:39.296 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:39 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:48:39.296 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:39 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:48:39.296 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:39 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:48:39.296 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:39 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:48:39.296 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:39 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:48:39.296 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:39 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:48:39.296 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:39 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T11:48:39.296 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:39 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:48:39.322 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:39.323+0000 7fb34ffff640 1 -- 172.21.15.29:0/3698788447 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb3500623e0 con 0x7fb3501031d0 2024-03-16T11:48:39.323 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:39.324+0000 7fb347fff640 1 -- 172.21.15.29:0/3698788447 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v84) ==== 74+0+22246 (secure 0 0 0) 0x7fb34009b8a0 con 0x7fb3501031d0 2024-03-16T11:48:39.324 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-16T11:48:39.324 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":84,"fsid":"d1887f28-e389-11ee-95c9-87774f69a715","created":"2024-03-16T11:40:16.475063+0000","modified":"2024-03-16T11:48:38.431116+0000","last_up_change":"2024-03-16T11:47:44.349348+0000","last_in_change":"2024-03-16T11:47:23.672432+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-16T11:43:51.892652+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-16T11:48:36.709732+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-16T11:48:37.586140+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"60f5608f-a408-45ba-bdf6-5b53758f88fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":2748947913},{"type":"v1","addr":"172.21.15.29:6803","nonce":2748947913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":2748947913},{"type":"v1","addr":"172.21.15.29:6805","nonce":2748947913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":2748947913},{"type":"v1","addr":"172.21.15.29:6809","nonce":2748947913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":2748947913},{"type":"v1","addr":"172.21.15.29:6807","nonce":2748947913}]},"public_addr":"172.21.15.29:6803/2748947913","cluster_addr":"172.21.15.29:6805/2748947913","heartbeat_back_addr":"172.21.15.29:6809/2748947913","heartbeat_front_addr":"172.21.15.29:6807/2748947913","state":["exists","up"]},{"osd":1,"uuid":"e70b53d6-ee78-4e56-9a2c-263ae0501b43","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6800","nonce":2339408422},{"type":"v1","addr":"172.21.15.152:6801","nonce":2339408422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6802","nonce":2339408422},{"type":"v1","addr":"172.21.15.152:6803","nonce":2339408422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6806","nonce":2339408422},{"type":"v1","addr":"172.21.15.152:6807","nonce":2339408422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6804","nonce":2339408422},{"type":"v1","addr":"172.21.15.152:6805","nonce":2339408422}]},"public_addr":"172.21.15.152:6801/2339408422","cluster_addr":"172.21.15.152:6803/2339408422","heartbeat_back_addr":"172.21.15.152:6807/2339408422","heartbeat_front_addr":"172.21.15.152:6805/2339408422","state":["exists","up"]},{"osd":2,"uuid":"ff39389a-dfac-44b5-80ff-36abca316e78","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6800","nonce":4288325280},{"type":"v1","addr":"172.21.15.154:6801","nonce":4288325280}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6802","nonce":4288325280},{"type":"v1","addr":"172.21.15.154:6803","nonce":4288325280}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6806","nonce":4288325280},{"type":"v1","addr":"172.21.15.154:6807","nonce":4288325280}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":4288325280},{"type":"v1","addr":"172.21.15.154:6805","nonce":4288325280}]},"public_addr":"172.21.15.154:6801/4288325280","cluster_addr":"172.21.15.154:6803/4288325280","heartbeat_back_addr":"172.21.15.154:6807/4288325280","heartbeat_front_addr":"172.21.15.154:6805/4288325280","state":["exists","up"]},{"osd":3,"uuid":"069e1795-55bc-4324-b94c-eb24ef2334c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":1538750397},{"type":"v1","addr":"172.21.15.29:6811","nonce":1538750397}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":1538750397},{"type":"v1","addr":"172.21.15.29:6813","nonce":1538750397}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":1538750397},{"type":"v1","addr":"172.21.15.29:6817","nonce":1538750397}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":1538750397},{"type":"v1","addr":"172.21.15.29:6815","nonce":1538750397}]},"public_addr":"172.21.15.29:6811/1538750397","cluster_addr":"172.21.15.29:6813/1538750397","heartbeat_back_addr":"172.21.15.29:6817/1538750397","heartbeat_front_addr":"172.21.15.29:6815/1538750397","state":["exists","up"]},{"osd":4,"uuid":"5a9d43a1-e8f5-4ab5-a441-afbc6ed4066e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6808","nonce":2795081750},{"type":"v1","addr":"172.21.15.152:6809","nonce":2795081750}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6810","nonce":2795081750},{"type":"v1","addr":"172.21.15.152:6811","nonce":2795081750}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6814","nonce":2795081750},{"type":"v1","addr":"172.21.15.152:6815","nonce":2795081750}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6812","nonce":2795081750},{"type":"v1","addr":"172.21.15.152:6813","nonce":2795081750}]},"public_addr":"172.21.15.152:6809/2795081750","cluster_addr":"172.21.15.152:6811/2795081750","heartbeat_back_addr":"172.21.15.152:6815/2795081750","heartbeat_front_addr":"172.21.15.152:6813/2795081750","state":["exists","up"]},{"osd":5,"uuid":"234de428-4ac7-44be-9a54-7b58331fdf8c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":1472686560},{"type":"v1","addr":"172.21.15.154:6809","nonce":1472686560}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6810","nonce":1472686560},{"type":"v1","addr":"172.21.15.154:6811","nonce":1472686560}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6814","nonce":1472686560},{"type":"v1","addr":"172.21.15.154:6815","nonce":1472686560}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6812","nonce":1472686560},{"type":"v1","addr":"172.21.15.154:6813","nonce":1472686560}]},"public_addr":"172.21.15.154:6809/1472686560","cluster_addr":"172.21.15.154:6811/1472686560","heartbeat_back_addr":"172.21.15.154:6815/1472686560","heartbeat_front_addr":"172.21.15.154:6813/1472686560","state":["exists","up"]},{"osd":6,"uuid":"ae45b8d9-7883-40c9-a9cd-4c92b601d4f2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":816506244},{"type":"v1","addr":"172.21.15.29:6819","nonce":816506244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":816506244},{"type":"v1","addr":"172.21.15.29:6821","nonce":816506244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":816506244},{"type":"v1","addr":"172.21.15.29:6825","nonce":816506244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":816506244},{"type":"v1","addr":"172.21.15.29:6823","nonce":816506244}]},"public_addr":"172.21.15.29:6819/816506244","cluster_addr":"172.21.15.29:6821/816506244","heartbeat_back_addr":"172.21.15.29:6825/816506244","heartbeat_front_addr":"172.21.15.29:6823/816506244","state":["exists","up"]},{"osd":7,"uuid":"6f687b28-4fce-4d4a-8b4e-294173d9d6c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6816","nonce":2924651787},{"type":"v1","addr":"172.21.15.152:6817","nonce":2924651787}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6818","nonce":2924651787},{"type":"v1","addr":"172.21.15.152:6819","nonce":2924651787}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6822","nonce":2924651787},{"type":"v1","addr":"172.21.15.152:6823","nonce":2924651787}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6820","nonce":2924651787},{"type":"v1","addr":"172.21.15.152:6821","nonce":2924651787}]},"public_addr":"172.21.15.152:6817/2924651787","cluster_addr":"172.21.15.152:6819/2924651787","heartbeat_back_addr":"172.21.15.152:6823/2924651787","heartbeat_front_addr":"172.21.15.152:6821/2924651787","state":["exists","up"]},{"osd":8,"uuid":"a25fb741-caa7-429a-b540-e394d034b446","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6816","nonce":129974156},{"type":"v1","addr":"172.21.15.154:6817","nonce":129974156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6818","nonce":129974156},{"type":"v1","addr":"172.21.15.154:6819","nonce":129974156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6822","nonce":129974156},{"type":"v1","addr":"172.21.15.154:6823","nonce":129974156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6820","nonce":129974156},{"type":"v1","addr":"172.21.15.154:6821","nonce":129974156}]},"public_addr":"172.21.15.154:6817/129974156","cluster_addr":"172.21.15.154:6819/129974156","heartbeat_back_addr":"172.21.15.154:6823/129974156","heartbeat_front_addr":"172.21.15.154:6821/129974156","state":["exists","up"]},{"osd":9,"uuid":"1071cb0e-8c5f-4cd8-9805-0045e220d946","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6826","nonce":2951414243},{"type":"v1","addr":"172.21.15.29:6827","nonce":2951414243}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":2951414243},{"type":"v1","addr":"172.21.15.29:6829","nonce":2951414243}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":2951414243},{"type":"v1","addr":"172.21.15.29:6833","nonce":2951414243}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":2951414243},{"type":"v1","addr":"172.21.15.29:6831","nonce":2951414243}]},"public_addr":"172.21.15.29:6827/2951414243","cluster_addr":"172.21.15.29:6829/2951414243","heartbeat_back_addr":"172.21.15.29:6833/2951414243","heartbeat_front_addr":"172.21.15.29:6831/2951414243","state":["exists","up"]},{"osd":10,"uuid":"ee7c7d74-3f39-4cd4-be80-843fb1dd9ecc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6824","nonce":3412525480},{"type":"v1","addr":"172.21.15.152:6825","nonce":3412525480}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6826","nonce":3412525480},{"type":"v1","addr":"172.21.15.152:6827","nonce":3412525480}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6830","nonce":3412525480},{"type":"v1","addr":"172.21.15.152:6831","nonce":3412525480}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6828","nonce":3412525480},{"type":"v1","addr":"172.21.15.152:6829","nonce":3412525480}]},"public_addr":"172.21.15.152:6825/3412525480","cluster_addr":"172.21.15.152:6827/3412525480","heartbeat_back_addr":"172.21.15.152:6831/3412525480","heartbeat_front_addr":"172.21.15.152:6829/3412525480","state":["exists","up"]},{"osd":11,"uuid":"4598c803-984a-4de6-ac22-066502c17889","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6824","nonce":1160927966},{"type":"v1","addr":"172.21.15.154:6825","nonce":1160927966}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6826","nonce":1160927966},{"type":"v1","addr":"172.21.15.154:6827","nonce":1160927966}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6830","nonce":1160927966},{"type":"v1","addr":"172.21.15.154:6831","nonce":1160927966}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6828","nonce":1160927966},{"type":"v1","addr":"172.21.15.154:6829","nonce":1160927966}]},"public_addr":"172.21.15.154:6825/1160927966","cluster_addr":"172.21.15.154:6827/1160927966","heartbeat_back_addr":"172.21.15.154:6831/1160927966","heartbeat_front_addr":"172.21.15.154:6829/1160927966","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:43:02.343393+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:43:25.209151+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:43:48.946164+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:44:14.588424+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:44:39.700025+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:45:04.061197+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:45:31.214508+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:45:56.937296+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:46:22.768915+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:46:50.101011+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:47:16.298041+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:47:42.402095+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.29:0/1853086886":"2024-03-17T11:41:27.531724+0000","172.21.15.29:0/1211659364":"2024-03-17T11:41:27.531724+0000","172.21.15.29:6800/2798831128":"2024-03-17T11:41:27.531724+0000","172.21.15.29:0/1281028285":"2024-03-17T11:41:00.505678+0000","172.21.15.29:6800/2275315654":"2024-03-17T11:41:00.505678+0000","172.21.15.29:6801/2798831128":"2024-03-17T11:41:27.531724+0000","172.21.15.29:6801/2275315654":"2024-03-17T11:41:00.505678+0000","172.21.15.29:0/1330100009":"2024-03-17T11:41:00.505678+0000","172.21.15.29:0/1784249039":"2024-03-17T11:41:00.505678+0000","172.21.15.29:0/1116486932":"2024-03-17T11:41:27.531724+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-16T11:48:39.326 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:39.326+0000 7fb34ffff640 1 -- 172.21.15.29:0/3698788447 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fb33c0a2800 msgr2=0x7fb33c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:48:39.326 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:39.326+0000 7fb34ffff640 1 --2- 172.21.15.29:0/3698788447 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fb33c0a2800 0x7fb33c0a4cc0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fb350102670 tx=0x7fb348009500 comp rx=0 tx=0).stop 2024-03-16T11:48:39.326 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:39.326+0000 7fb34ffff640 1 -- 172.21.15.29:0/3698788447 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb3501031d0 msgr2=0x7fb35019eef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:48:39.326 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:39.326+0000 7fb34ffff640 1 --2- 172.21.15.29:0/3698788447 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb3501031d0 0x7fb35019eef0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fb34000e970 tx=0x7fb34000ed70 comp rx=0 tx=0).stop 2024-03-16T11:48:39.326 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:39.326+0000 7fb34ffff640 1 -- 172.21.15.29:0/3698788447 shutdown_connections 2024-03-16T11:48:39.326 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:39.326+0000 7fb34ffff640 1 --2- 172.21.15.29:0/3698788447 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fb33c0a2800 0x7fb33c0a4cc0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:39.326 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:39.326+0000 7fb34ffff640 1 --2- 172.21.15.29:0/3698788447 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb3501031d0 0x7fb35019eef0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:39.326 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:39.326+0000 7fb34ffff640 1 --2- 172.21.15.29:0/3698788447 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb350102810 0x7fb350197f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:39.326 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:39.326+0000 7fb34ffff640 1 --2- 172.21.15.29:0/3698788447 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb350101620 0x7fb3501979e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:39.326 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:39.326+0000 7fb34ffff640 1 -- 172.21.15.29:0/3698788447 >> 172.21.15.29:0/3698788447 conn(0x7fb3500fcb70 msgr2=0x7fb350105b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:48:39.326 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:39.326+0000 7fb34ffff640 1 -- 172.21.15.29:0/3698788447 shutdown_connections 2024-03-16T11:48:39.327 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:39.327+0000 7fb34ffff640 1 -- 172.21.15.29:0/3698788447 wait complete. 2024-03-16T11:48:39.336 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:39 smithi029 ceph-mon[31975]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-16T11:48:39.336 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:39 smithi029 ceph-mon[31975]: from='client.? 172.21.15.29:0/1835495197' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-03-16T11:48:39.336 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:39 smithi029 ceph-mon[31975]: osdmap e83: 12 total, 12 up, 12 in 2024-03-16T11:48:39.336 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:39 smithi029 ceph-mon[31975]: from='client.? 172.21.15.29:0/2925241401' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-03-16T11:48:39.336 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:39 smithi029 ceph-mon[31975]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-03-16T11:48:39.336 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:39 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:48:39.336 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:39 smithi029 ceph-mon[31975]: osdmap e84: 12 total, 12 up, 12 in 2024-03-16T11:48:39.336 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:39 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:48:39.336 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:39 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:48:39.336 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:39 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:48:39.336 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:39 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:48:39.336 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:39 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:48:39.336 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:39 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:48:39.336 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:39 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T11:48:39.337 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:39 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:48:39.338 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-16T11:48:39.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:39 smithi152 ceph-mon[36291]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-16T11:48:39.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:39 smithi152 ceph-mon[36291]: from='client.? 172.21.15.29:0/1835495197' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-03-16T11:48:39.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:39 smithi152 ceph-mon[36291]: osdmap e83: 12 total, 12 up, 12 in 2024-03-16T11:48:39.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:39 smithi152 ceph-mon[36291]: from='client.? 172.21.15.29:0/2925241401' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-03-16T11:48:39.378 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:39 smithi152 ceph-mon[36291]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-03-16T11:48:39.378 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:39 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:48:39.378 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:39 smithi152 ceph-mon[36291]: osdmap e84: 12 total, 12 up, 12 in 2024-03-16T11:48:39.378 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:39 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:48:39.378 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:39 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:48:39.378 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:39 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:48:39.378 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:39 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:48:39.378 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:39 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:48:39.378 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:39 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:48:39.378 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:39 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T11:48:39.378 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:39 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:48:39.443 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:39.441+0000 7f829ac85640 1 -- 172.21.15.29:0/2647312681 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f82940ffad0 msgr2=0x7f82940ffef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:48:39.443 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:39.441+0000 7f829ac85640 1 --2- 172.21.15.29:0/2647312681 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f82940ffad0 0x7f82940ffef0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f8288009930 tx=0x7f828802bc30 comp rx=0 tx=0).stop 2024-03-16T11:48:39.443 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:39.441+0000 7f829ac85640 1 -- 172.21.15.29:0/2647312681 shutdown_connections 2024-03-16T11:48:39.443 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:39.441+0000 7f829ac85640 1 --2- 172.21.15.29:0/2647312681 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8294103460 0x7f8294109cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:39.443 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:39.441+0000 7f829ac85640 1 --2- 172.21.15.29:0/2647312681 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8294102aa0 0x7f8294102f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:39.443 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:39.441+0000 7f829ac85640 1 --2- 172.21.15.29:0/2647312681 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f82940ffad0 0x7f82940ffef0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:39.443 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:39.441+0000 7f829ac85640 1 -- 172.21.15.29:0/2647312681 >> 172.21.15.29:0/2647312681 conn(0x7f82940fb400 msgr2=0x7f82940fd840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:48:39.443 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:39.441+0000 7f829ac85640 1 -- 172.21.15.29:0/2647312681 shutdown_connections 2024-03-16T11:48:39.443 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:39.441+0000 7f829ac85640 1 -- 172.21.15.29:0/2647312681 wait complete. 2024-03-16T11:48:39.444 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:39.442+0000 7f829ac85640 1 Processor -- start 2024-03-16T11:48:39.444 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:39.442+0000 7f829ac85640 1 -- start start 2024-03-16T11:48:39.444 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:39.442+0000 7f829ac85640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f82940ffad0 0x7f8294197b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:48:39.444 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:39.442+0000 7f829ac85640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8294102aa0 0x7f82941980b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:48:39.444 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:39.442+0000 7f829ac85640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8294103460 0x7f829419f120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:48:39.444 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:39.442+0000 7f829ac85640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f829410d4a0 con 0x7f8294103460 2024-03-16T11:48:39.444 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:39.442+0000 7f829ac85640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f829410d320 con 0x7f82940ffad0 2024-03-16T11:48:39.444 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:39.442+0000 7f829ac85640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f829410d620 con 0x7f8294102aa0 2024-03-16T11:48:39.444 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:39.442+0000 7f8299222640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8294103460 0x7f829419f120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:48:39.444 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:39.442+0000 7f8293fff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8294102aa0 0x7f82941980b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:48:39.444 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:39.442+0000 7f8299222640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8294103460 0x7f829419f120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60522/0 (socket says 172.21.15.29:60522) 2024-03-16T11:48:39.444 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:39.442+0000 7f8299222640 1 -- 172.21.15.29:0/3746006667 learned_addr learned my addr 172.21.15.29:0/3746006667 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:48:39.444 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:39.442+0000 7f8298a21640 1 --2- 172.21.15.29:0/3746006667 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f82940ffad0 0x7f8294197b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:48:39.444 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:39.442+0000 7f8299222640 1 -- 172.21.15.29:0/3746006667 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8294102aa0 msgr2=0x7f82941980b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:48:39.445 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:39.442+0000 7f8299222640 1 --2- 172.21.15.29:0/3746006667 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8294102aa0 0x7f82941980b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:39.445 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:39.442+0000 7f8299222640 1 -- 172.21.15.29:0/3746006667 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f82940ffad0 msgr2=0x7f8294197b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:48:39.445 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:39.442+0000 7f8299222640 1 --2- 172.21.15.29:0/3746006667 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f82940ffad0 0x7f8294197b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:39.445 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:39.442+0000 7f8299222640 1 -- 172.21.15.29:0/3746006667 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f829419f790 con 0x7f8294103460 2024-03-16T11:48:39.445 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:39.442+0000 7f8293fff640 1 --2- 172.21.15.29:0/3746006667 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8294102aa0 0x7f82941980b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:48:39.445 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:39.442+0000 7f8298a21640 1 --2- 172.21.15.29:0/3746006667 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f82940ffad0 0x7f8294197b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T11:48:39.445 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:39.442+0000 7f8299222640 1 --2- 172.21.15.29:0/3746006667 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8294103460 0x7f829419f120 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f828c00eaa0 tx=0x7f828c00ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:48:39.445 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:39.443+0000 7f8291ffb640 1 -- 172.21.15.29:0/3746006667 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f828c00cdb0 con 0x7f8294103460 2024-03-16T11:48:39.445 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:39.443+0000 7f829ac85640 1 -- 172.21.15.29:0/3746006667 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f829419fa80 con 0x7f8294103460 2024-03-16T11:48:39.445 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:39.443+0000 7f8291ffb640 1 -- 172.21.15.29:0/3746006667 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f828c005020 con 0x7f8294103460 2024-03-16T11:48:39.445 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:39.443+0000 7f8291ffb640 1 -- 172.21.15.29:0/3746006667 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f828c0116b0 con 0x7f8294103460 2024-03-16T11:48:39.445 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:39.443+0000 7f829ac85640 1 -- 172.21.15.29:0/3746006667 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f829419ff80 con 0x7f8294103460 2024-03-16T11:48:39.445 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:39.444+0000 7f829ac85640 1 -- 172.21.15.29:0/3746006667 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8294108c20 con 0x7f8294103460 2024-03-16T11:48:39.446 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:39.445+0000 7f8291ffb640 1 -- 172.21.15.29:0/3746006667 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f828c005560 con 0x7f8294103460 2024-03-16T11:48:39.446 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:39.445+0000 7f8291ffb640 1 --2- 172.21.15.29:0/3746006667 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f82840a2800 0x7f82840a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:48:39.447 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:39.445+0000 7f8291ffb640 1 -- 172.21.15.29:0/3746006667 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 7979+0+0 (secure 0 0 0) 0x7f828c0ceb80 con 0x7f8294103460 2024-03-16T11:48:39.447 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:39.445+0000 7f8298a21640 1 --2- 172.21.15.29:0/3746006667 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f82840a2800 0x7f82840a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:48:39.447 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:39.446+0000 7f8298a21640 1 --2- 172.21.15.29:0/3746006667 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f82840a2800 0x7f82840a4cc0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f828802c7b0 tx=0x7f8288005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:48:39.449 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:39.450+0000 7f8291ffb640 1 -- 172.21.15.29:0/3746006667 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f828c016030 con 0x7f8294103460 2024-03-16T11:48:39.637 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:39.637+0000 7f829ac85640 1 -- 172.21.15.29:0/3746006667 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f82940623e0 con 0x7f8294103460 2024-03-16T11:48:39.638 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:39.639+0000 7f8291ffb640 1 -- 172.21.15.29:0/3746006667 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) ==== 74+0+22233 (secure 0 0 0) 0x7f828c09bd70 con 0x7f8294103460 2024-03-16T11:48:39.638 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-16T11:48:39.638 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":85,"fsid":"d1887f28-e389-11ee-95c9-87774f69a715","created":"2024-03-16T11:40:16.475063+0000","modified":"2024-03-16T11:48:39.554402+0000","last_up_change":"2024-03-16T11:47:44.349348+0000","last_in_change":"2024-03-16T11:47:23.672432+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-16T11:43:51.892652+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-16T11:48:36.709732+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-16T11:48:37.586140+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"60f5608f-a408-45ba-bdf6-5b53758f88fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":2748947913},{"type":"v1","addr":"172.21.15.29:6803","nonce":2748947913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":2748947913},{"type":"v1","addr":"172.21.15.29:6805","nonce":2748947913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":2748947913},{"type":"v1","addr":"172.21.15.29:6809","nonce":2748947913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":2748947913},{"type":"v1","addr":"172.21.15.29:6807","nonce":2748947913}]},"public_addr":"172.21.15.29:6803/2748947913","cluster_addr":"172.21.15.29:6805/2748947913","heartbeat_back_addr":"172.21.15.29:6809/2748947913","heartbeat_front_addr":"172.21.15.29:6807/2748947913","state":["exists","up"]},{"osd":1,"uuid":"e70b53d6-ee78-4e56-9a2c-263ae0501b43","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6800","nonce":2339408422},{"type":"v1","addr":"172.21.15.152:6801","nonce":2339408422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6802","nonce":2339408422},{"type":"v1","addr":"172.21.15.152:6803","nonce":2339408422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6806","nonce":2339408422},{"type":"v1","addr":"172.21.15.152:6807","nonce":2339408422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6804","nonce":2339408422},{"type":"v1","addr":"172.21.15.152:6805","nonce":2339408422}]},"public_addr":"172.21.15.152:6801/2339408422","cluster_addr":"172.21.15.152:6803/2339408422","heartbeat_back_addr":"172.21.15.152:6807/2339408422","heartbeat_front_addr":"172.21.15.152:6805/2339408422","state":["exists","up"]},{"osd":2,"uuid":"ff39389a-dfac-44b5-80ff-36abca316e78","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6800","nonce":4288325280},{"type":"v1","addr":"172.21.15.154:6801","nonce":4288325280}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6802","nonce":4288325280},{"type":"v1","addr":"172.21.15.154:6803","nonce":4288325280}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6806","nonce":4288325280},{"type":"v1","addr":"172.21.15.154:6807","nonce":4288325280}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":4288325280},{"type":"v1","addr":"172.21.15.154:6805","nonce":4288325280}]},"public_addr":"172.21.15.154:6801/4288325280","cluster_addr":"172.21.15.154:6803/4288325280","heartbeat_back_addr":"172.21.15.154:6807/4288325280","heartbeat_front_addr":"172.21.15.154:6805/4288325280","state":["exists","up"]},{"osd":3,"uuid":"069e1795-55bc-4324-b94c-eb24ef2334c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":1538750397},{"type":"v1","addr":"172.21.15.29:6811","nonce":1538750397}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":1538750397},{"type":"v1","addr":"172.21.15.29:6813","nonce":1538750397}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":1538750397},{"type":"v1","addr":"172.21.15.29:6817","nonce":1538750397}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":1538750397},{"type":"v1","addr":"172.21.15.29:6815","nonce":1538750397}]},"public_addr":"172.21.15.29:6811/1538750397","cluster_addr":"172.21.15.29:6813/1538750397","heartbeat_back_addr":"172.21.15.29:6817/1538750397","heartbeat_front_addr":"172.21.15.29:6815/1538750397","state":["exists","up"]},{"osd":4,"uuid":"5a9d43a1-e8f5-4ab5-a441-afbc6ed4066e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6808","nonce":2795081750},{"type":"v1","addr":"172.21.15.152:6809","nonce":2795081750}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6810","nonce":2795081750},{"type":"v1","addr":"172.21.15.152:6811","nonce":2795081750}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6814","nonce":2795081750},{"type":"v1","addr":"172.21.15.152:6815","nonce":2795081750}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6812","nonce":2795081750},{"type":"v1","addr":"172.21.15.152:6813","nonce":2795081750}]},"public_addr":"172.21.15.152:6809/2795081750","cluster_addr":"172.21.15.152:6811/2795081750","heartbeat_back_addr":"172.21.15.152:6815/2795081750","heartbeat_front_addr":"172.21.15.152:6813/2795081750","state":["exists","up"]},{"osd":5,"uuid":"234de428-4ac7-44be-9a54-7b58331fdf8c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":1472686560},{"type":"v1","addr":"172.21.15.154:6809","nonce":1472686560}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6810","nonce":1472686560},{"type":"v1","addr":"172.21.15.154:6811","nonce":1472686560}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6814","nonce":1472686560},{"type":"v1","addr":"172.21.15.154:6815","nonce":1472686560}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6812","nonce":1472686560},{"type":"v1","addr":"172.21.15.154:6813","nonce":1472686560}]},"public_addr":"172.21.15.154:6809/1472686560","cluster_addr":"172.21.15.154:6811/1472686560","heartbeat_back_addr":"172.21.15.154:6815/1472686560","heartbeat_front_addr":"172.21.15.154:6813/1472686560","state":["exists","up"]},{"osd":6,"uuid":"ae45b8d9-7883-40c9-a9cd-4c92b601d4f2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":816506244},{"type":"v1","addr":"172.21.15.29:6819","nonce":816506244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":816506244},{"type":"v1","addr":"172.21.15.29:6821","nonce":816506244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":816506244},{"type":"v1","addr":"172.21.15.29:6825","nonce":816506244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":816506244},{"type":"v1","addr":"172.21.15.29:6823","nonce":816506244}]},"public_addr":"172.21.15.29:6819/816506244","cluster_addr":"172.21.15.29:6821/816506244","heartbeat_back_addr":"172.21.15.29:6825/816506244","heartbeat_front_addr":"172.21.15.29:6823/816506244","state":["exists","up"]},{"osd":7,"uuid":"6f687b28-4fce-4d4a-8b4e-294173d9d6c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6816","nonce":2924651787},{"type":"v1","addr":"172.21.15.152:6817","nonce":2924651787}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6818","nonce":2924651787},{"type":"v1","addr":"172.21.15.152:6819","nonce":2924651787}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6822","nonce":2924651787},{"type":"v1","addr":"172.21.15.152:6823","nonce":2924651787}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6820","nonce":2924651787},{"type":"v1","addr":"172.21.15.152:6821","nonce":2924651787}]},"public_addr":"172.21.15.152:6817/2924651787","cluster_addr":"172.21.15.152:6819/2924651787","heartbeat_back_addr":"172.21.15.152:6823/2924651787","heartbeat_front_addr":"172.21.15.152:6821/2924651787","state":["exists","up"]},{"osd":8,"uuid":"a25fb741-caa7-429a-b540-e394d034b446","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6816","nonce":129974156},{"type":"v1","addr":"172.21.15.154:6817","nonce":129974156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6818","nonce":129974156},{"type":"v1","addr":"172.21.15.154:6819","nonce":129974156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6822","nonce":129974156},{"type":"v1","addr":"172.21.15.154:6823","nonce":129974156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6820","nonce":129974156},{"type":"v1","addr":"172.21.15.154:6821","nonce":129974156}]},"public_addr":"172.21.15.154:6817/129974156","cluster_addr":"172.21.15.154:6819/129974156","heartbeat_back_addr":"172.21.15.154:6823/129974156","heartbeat_front_addr":"172.21.15.154:6821/129974156","state":["exists","up"]},{"osd":9,"uuid":"1071cb0e-8c5f-4cd8-9805-0045e220d946","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6826","nonce":2951414243},{"type":"v1","addr":"172.21.15.29:6827","nonce":2951414243}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":2951414243},{"type":"v1","addr":"172.21.15.29:6829","nonce":2951414243}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":2951414243},{"type":"v1","addr":"172.21.15.29:6833","nonce":2951414243}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":2951414243},{"type":"v1","addr":"172.21.15.29:6831","nonce":2951414243}]},"public_addr":"172.21.15.29:6827/2951414243","cluster_addr":"172.21.15.29:6829/2951414243","heartbeat_back_addr":"172.21.15.29:6833/2951414243","heartbeat_front_addr":"172.21.15.29:6831/2951414243","state":["exists","up"]},{"osd":10,"uuid":"ee7c7d74-3f39-4cd4-be80-843fb1dd9ecc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6824","nonce":3412525480},{"type":"v1","addr":"172.21.15.152:6825","nonce":3412525480}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6826","nonce":3412525480},{"type":"v1","addr":"172.21.15.152:6827","nonce":3412525480}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6830","nonce":3412525480},{"type":"v1","addr":"172.21.15.152:6831","nonce":3412525480}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6828","nonce":3412525480},{"type":"v1","addr":"172.21.15.152:6829","nonce":3412525480}]},"public_addr":"172.21.15.152:6825/3412525480","cluster_addr":"172.21.15.152:6827/3412525480","heartbeat_back_addr":"172.21.15.152:6831/3412525480","heartbeat_front_addr":"172.21.15.152:6829/3412525480","state":["exists","up"]},{"osd":11,"uuid":"4598c803-984a-4de6-ac22-066502c17889","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6824","nonce":1160927966},{"type":"v1","addr":"172.21.15.154:6825","nonce":1160927966}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6826","nonce":1160927966},{"type":"v1","addr":"172.21.15.154:6827","nonce":1160927966}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6830","nonce":1160927966},{"type":"v1","addr":"172.21.15.154:6831","nonce":1160927966}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6828","nonce":1160927966},{"type":"v1","addr":"172.21.15.154:6829","nonce":1160927966}]},"public_addr":"172.21.15.154:6825/1160927966","cluster_addr":"172.21.15.154:6827/1160927966","heartbeat_back_addr":"172.21.15.154:6831/1160927966","heartbeat_front_addr":"172.21.15.154:6829/1160927966","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:43:02.343393+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:43:25.209151+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:43:48.946164+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:44:14.588424+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:44:39.700025+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:45:04.061197+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:45:31.214508+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:45:56.937296+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:46:22.768915+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:46:50.101011+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:47:16.298041+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:47:42.402095+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.29:0/1853086886":"2024-03-17T11:41:27.531724+0000","172.21.15.29:0/1211659364":"2024-03-17T11:41:27.531724+0000","172.21.15.29:6800/2798831128":"2024-03-17T11:41:27.531724+0000","172.21.15.29:0/1281028285":"2024-03-17T11:41:00.505678+0000","172.21.15.29:6800/2275315654":"2024-03-17T11:41:00.505678+0000","172.21.15.29:6801/2798831128":"2024-03-17T11:41:27.531724+0000","172.21.15.29:6801/2275315654":"2024-03-17T11:41:00.505678+0000","172.21.15.29:0/1330100009":"2024-03-17T11:41:00.505678+0000","172.21.15.29:0/1784249039":"2024-03-17T11:41:00.505678+0000","172.21.15.29:0/1116486932":"2024-03-17T11:41:27.531724+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-16T11:48:39.640 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:39.641+0000 7f829ac85640 1 -- 172.21.15.29:0/3746006667 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f82840a2800 msgr2=0x7f82840a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:48:39.640 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:39.641+0000 7f829ac85640 1 --2- 172.21.15.29:0/3746006667 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f82840a2800 0x7f82840a4cc0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f828802c7b0 tx=0x7f8288005fd0 comp rx=0 tx=0).stop 2024-03-16T11:48:39.640 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:39.641+0000 7f829ac85640 1 -- 172.21.15.29:0/3746006667 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8294103460 msgr2=0x7f829419f120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:48:39.640 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:39.641+0000 7f829ac85640 1 --2- 172.21.15.29:0/3746006667 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8294103460 0x7f829419f120 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f828c00eaa0 tx=0x7f828c00ef60 comp rx=0 tx=0).stop 2024-03-16T11:48:39.641 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:39.641+0000 7f829ac85640 1 -- 172.21.15.29:0/3746006667 shutdown_connections 2024-03-16T11:48:39.641 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:39.641+0000 7f829ac85640 1 --2- 172.21.15.29:0/3746006667 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f82840a2800 0x7f82840a4cc0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:39.641 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:39.641+0000 7f829ac85640 1 --2- 172.21.15.29:0/3746006667 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8294103460 0x7f829419f120 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:39.641 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:39.641+0000 7f829ac85640 1 --2- 172.21.15.29:0/3746006667 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8294102aa0 0x7f82941980b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:39.641 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:39.641+0000 7f829ac85640 1 --2- 172.21.15.29:0/3746006667 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f82940ffad0 0x7f8294197b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:39.641 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:39.641+0000 7f829ac85640 1 -- 172.21.15.29:0/3746006667 >> 172.21.15.29:0/3746006667 conn(0x7f82940fb400 msgr2=0x7f8294105df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:48:39.641 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:39.641+0000 7f829ac85640 1 -- 172.21.15.29:0/3746006667 shutdown_connections 2024-03-16T11:48:39.641 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:39.641+0000 7f829ac85640 1 -- 172.21.15.29:0/3746006667 wait complete. 2024-03-16T11:48:39.651 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-16T11:48:39.759 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:39.757+0000 7f62ed1c4640 1 -- 172.21.15.29:0/3035261442 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f62e81018d0 msgr2=0x7f62e8101d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:48:39.759 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:39.757+0000 7f62ed1c4640 1 --2- 172.21.15.29:0/3035261442 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f62e81018d0 0x7f62e8101d50 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f62d0009900 tx=0x7f62d002bc30 comp rx=0 tx=0).stop 2024-03-16T11:48:39.759 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:39.757+0000 7f62ed1c4640 1 -- 172.21.15.29:0/3035261442 shutdown_connections 2024-03-16T11:48:39.759 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:39.757+0000 7f62ed1c4640 1 --2- 172.21.15.29:0/3035261442 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f62e8102290 0x7f62e8108b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:39.759 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:39.757+0000 7f62ed1c4640 1 --2- 172.21.15.29:0/3035261442 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f62e81018d0 0x7f62e8101d50 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:39.759 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:39.757+0000 7f62ed1c4640 1 --2- 172.21.15.29:0/3035261442 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f62e81006e0 0x7f62e8100b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:39.759 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:39.757+0000 7f62ed1c4640 1 -- 172.21.15.29:0/3035261442 >> 172.21.15.29:0/3035261442 conn(0x7f62e80fbc70 msgr2=0x7f62e80fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:48:39.759 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:39.757+0000 7f62ed1c4640 1 -- 172.21.15.29:0/3035261442 shutdown_connections 2024-03-16T11:48:39.759 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:39.757+0000 7f62ed1c4640 1 -- 172.21.15.29:0/3035261442 wait complete. 2024-03-16T11:48:39.760 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:39.758+0000 7f62ed1c4640 1 Processor -- start 2024-03-16T11:48:39.760 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:39.758+0000 7f62ed1c4640 1 -- start start 2024-03-16T11:48:39.760 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:39.758+0000 7f62ed1c4640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f62e81006e0 0x7f62e8197ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:48:39.760 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:39.758+0000 7f62ed1c4640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f62e81018d0 0x7f62e81980e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:48:39.760 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:39.758+0000 7f62ed1c4640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f62e8102290 0x7f62e819f150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:48:39.760 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:39.758+0000 7f62ed1c4640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f62e810d4c0 con 0x7f62e81006e0 2024-03-16T11:48:39.760 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:39.758+0000 7f62ed1c4640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f62e810d340 con 0x7f62e8102290 2024-03-16T11:48:39.760 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:39.758+0000 7f62ed1c4640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f62e810d640 con 0x7f62e81018d0 2024-03-16T11:48:39.760 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:39.758+0000 7f62e6d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f62e81006e0 0x7f62e8197ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:48:39.760 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:39.758+0000 7f62e6d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f62e81006e0 0x7f62e8197ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49024/0 (socket says 172.21.15.29:49024) 2024-03-16T11:48:39.760 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:39.758+0000 7f62e6d9d640 1 -- 172.21.15.29:0/3044224480 learned_addr learned my addr 172.21.15.29:0/3044224480 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:48:39.760 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:39.759+0000 7f62e659c640 1 --2- 172.21.15.29:0/3044224480 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f62e81018d0 0x7f62e81980e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:48:39.760 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:39.759+0000 7f62e759e640 1 --2- 172.21.15.29:0/3044224480 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f62e8102290 0x7f62e819f150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:48:39.760 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:39.759+0000 7f62e6d9d640 1 -- 172.21.15.29:0/3044224480 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f62e81018d0 msgr2=0x7f62e81980e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:48:39.761 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:39.759+0000 7f62e6d9d640 1 --2- 172.21.15.29:0/3044224480 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f62e81018d0 0x7f62e81980e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:39.761 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:39.759+0000 7f62e6d9d640 1 -- 172.21.15.29:0/3044224480 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f62e8102290 msgr2=0x7f62e819f150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:48:39.761 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:39.759+0000 7f62e6d9d640 1 --2- 172.21.15.29:0/3044224480 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f62e8102290 0x7f62e819f150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:39.761 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:39.759+0000 7f62e6d9d640 1 -- 172.21.15.29:0/3044224480 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62e819f7c0 con 0x7f62e81006e0 2024-03-16T11:48:39.761 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:39.759+0000 7f62e759e640 1 --2- 172.21.15.29:0/3044224480 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f62e8102290 0x7f62e819f150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-16T11:48:39.761 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:39.759+0000 7f62e6d9d640 1 --2- 172.21.15.29:0/3044224480 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f62e81006e0 0x7f62e8197ba0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f62e000d950 tx=0x7f62e000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:48:39.761 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:39.759+0000 7f62dffff640 1 -- 172.21.15.29:0/3044224480 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62e0017020 con 0x7f62e81006e0 2024-03-16T11:48:39.761 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:39.759+0000 7f62dffff640 1 -- 172.21.15.29:0/3044224480 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f62e0009c30 con 0x7f62e81006e0 2024-03-16T11:48:39.761 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:39.759+0000 7f62ed1c4640 1 -- 172.21.15.29:0/3044224480 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f62e819fab0 con 0x7f62e81006e0 2024-03-16T11:48:39.761 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:39.759+0000 7f62dffff640 1 -- 172.21.15.29:0/3044224480 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62e000f720 con 0x7f62e81006e0 2024-03-16T11:48:39.761 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:39.759+0000 7f62ed1c4640 1 -- 172.21.15.29:0/3044224480 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f62e819ffd0 con 0x7f62e81006e0 2024-03-16T11:48:39.761 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:39.760+0000 7f62ed1c4640 1 -- 172.21.15.29:0/3044224480 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f62e8107a80 con 0x7f62e81006e0 2024-03-16T11:48:39.762 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:39.761+0000 7f62dffff640 1 -- 172.21.15.29:0/3044224480 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f62e001a020 con 0x7f62e81006e0 2024-03-16T11:48:39.762 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:39.762+0000 7f62dffff640 1 --2- 172.21.15.29:0/3044224480 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f62d40a2800 0x7f62d40a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:48:39.762 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:39.762+0000 7f62e659c640 1 --2- 172.21.15.29:0/3044224480 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f62d40a2800 0x7f62d40a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:48:39.762 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:39.762+0000 7f62dffff640 1 -- 172.21.15.29:0/3044224480 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f62e00cec50 con 0x7f62e81006e0 2024-03-16T11:48:39.762 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:39.762+0000 7f62e659c640 1 --2- 172.21.15.29:0/3044224480 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f62d40a2800 0x7f62d40a4cc0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f62d002c7b0 tx=0x7f62d0005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:48:39.764 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:39.765+0000 7f62dffff640 1 -- 172.21.15.29:0/3044224480 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f62e0021020 con 0x7f62e81006e0 2024-03-16T11:48:39.987 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:39.988+0000 7f62ed1c4640 1 -- 172.21.15.29:0/3044224480 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f62e802d3d0 con 0x7f62e81006e0 2024-03-16T11:48:40.013 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:40.014+0000 7f62dffff640 1 -- 172.21.15.29:0/3044224480 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v15) ==== 118+0+0 (secure 0 0 0) 0x7f62e0009de0 con 0x7f62e81006e0 2024-03-16T11:48:40.016 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:40.016+0000 7f62ed1c4640 1 -- 172.21.15.29:0/3044224480 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f62d40a2800 msgr2=0x7f62d40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:48:40.016 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:40.016+0000 7f62ed1c4640 1 --2- 172.21.15.29:0/3044224480 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f62d40a2800 0x7f62d40a4cc0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f62d002c7b0 tx=0x7f62d0005fd0 comp rx=0 tx=0).stop 2024-03-16T11:48:40.016 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:40.016+0000 7f62ed1c4640 1 -- 172.21.15.29:0/3044224480 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f62e81006e0 msgr2=0x7f62e8197ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:48:40.016 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:40.016+0000 7f62ed1c4640 1 --2- 172.21.15.29:0/3044224480 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f62e81006e0 0x7f62e8197ba0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f62e000d950 tx=0x7f62e000de10 comp rx=0 tx=0).stop 2024-03-16T11:48:40.016 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:40.016+0000 7f62ed1c4640 1 -- 172.21.15.29:0/3044224480 shutdown_connections 2024-03-16T11:48:40.016 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:40.016+0000 7f62ed1c4640 1 --2- 172.21.15.29:0/3044224480 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f62d40a2800 0x7f62d40a4cc0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:40.016 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:40.016+0000 7f62ed1c4640 1 --2- 172.21.15.29:0/3044224480 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f62e8102290 0x7f62e819f150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:40.016 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:40.016+0000 7f62ed1c4640 1 --2- 172.21.15.29:0/3044224480 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f62e81018d0 0x7f62e81980e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:40.016 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:40.016+0000 7f62ed1c4640 1 --2- 172.21.15.29:0/3044224480 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f62e81006e0 0x7f62e8197ba0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:40.016 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:40.016+0000 7f62ed1c4640 1 -- 172.21.15.29:0/3044224480 >> 172.21.15.29:0/3044224480 conn(0x7f62e80fbc70 msgr2=0x7f62e8104c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:48:40.016 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:40.016+0000 7f62ed1c4640 1 -- 172.21.15.29:0/3044224480 shutdown_connections 2024-03-16T11:48:40.016 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:40.016+0000 7f62ed1c4640 1 -- 172.21.15.29:0/3044224480 wait complete. 2024-03-16T11:48:40.027 DEBUG:tasks.cephfs.filesystem:fs_config: {'max_mds': 3, 'session_timeout': 300, 'standby_replay': True, 'subvols': {'create': 2}} 2024-03-16T11:48:40.027 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 3 2024-03-16T11:48:40.134 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:40.133+0000 7fe055913640 1 -- 172.21.15.29:0/634481775 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe0501018f0 msgr2=0x7fe050101d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:48:40.134 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:40.133+0000 7fe055913640 1 --2- 172.21.15.29:0/634481775 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe0501018f0 0x7fe050101d70 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fe038009900 tx=0x7fe03802bc30 comp rx=0 tx=0).stop 2024-03-16T11:48:40.134 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:40.133+0000 7fe055913640 1 -- 172.21.15.29:0/634481775 shutdown_connections 2024-03-16T11:48:40.134 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:40.133+0000 7fe055913640 1 --2- 172.21.15.29:0/634481775 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe0501022b0 0x7fe050108b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:40.134 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:40.133+0000 7fe055913640 1 --2- 172.21.15.29:0/634481775 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe0501018f0 0x7fe050101d70 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:40.134 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:40.133+0000 7fe055913640 1 --2- 172.21.15.29:0/634481775 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe050100700 0x7fe050100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:40.134 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:40.133+0000 7fe055913640 1 -- 172.21.15.29:0/634481775 >> 172.21.15.29:0/634481775 conn(0x7fe0500fbc90 msgr2=0x7fe0500fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:48:40.134 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:40.133+0000 7fe055913640 1 -- 172.21.15.29:0/634481775 shutdown_connections 2024-03-16T11:48:40.135 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:40.133+0000 7fe055913640 1 -- 172.21.15.29:0/634481775 wait complete. 2024-03-16T11:48:40.135 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:40.133+0000 7fe055913640 1 Processor -- start 2024-03-16T11:48:40.135 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:40.133+0000 7fe055913640 1 -- start start 2024-03-16T11:48:40.135 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:40.134+0000 7fe055913640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe050100700 0x7fe05006a980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:48:40.135 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:40.134+0000 7fe055913640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe0501018f0 0x7fe05019c800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:48:40.135 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:40.134+0000 7fe055913640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe0501022b0 0x7fe05019ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:48:40.135 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:40.134+0000 7fe055913640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fe05010c360 con 0x7fe0501022b0 2024-03-16T11:48:40.135 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:40.134+0000 7fe055913640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fe05010c1e0 con 0x7fe0501018f0 2024-03-16T11:48:40.135 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:40.134+0000 7fe055913640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fe05010c4e0 con 0x7fe050100700 2024-03-16T11:48:40.135 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:40.134+0000 7fe04f7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe0501022b0 0x7fe05019ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:48:40.135 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:40.134+0000 7fe04effd640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe050100700 0x7fe05006a980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:48:40.135 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:40.134+0000 7fe04f7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe0501022b0 0x7fe05019ee70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49040/0 (socket says 172.21.15.29:49040) 2024-03-16T11:48:40.135 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:40.134+0000 7fe04f7fe640 1 -- 172.21.15.29:0/1104968250 learned_addr learned my addr 172.21.15.29:0/1104968250 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:48:40.135 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:40.134+0000 7fe0467fc640 1 --2- 172.21.15.29:0/1104968250 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe0501018f0 0x7fe05019c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:48:40.136 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:40.134+0000 7fe04f7fe640 1 -- 172.21.15.29:0/1104968250 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe050100700 msgr2=0x7fe05006a980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:48:40.136 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:40.134+0000 7fe04f7fe640 1 --2- 172.21.15.29:0/1104968250 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe050100700 0x7fe05006a980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:40.136 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:40.134+0000 7fe04f7fe640 1 -- 172.21.15.29:0/1104968250 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe0501018f0 msgr2=0x7fe05019c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:48:40.136 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:40.134+0000 7fe04f7fe640 1 --2- 172.21.15.29:0/1104968250 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe0501018f0 0x7fe05019c800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:40.136 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:40.134+0000 7fe04f7fe640 1 -- 172.21.15.29:0/1104968250 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe05019f730 con 0x7fe0501022b0 2024-03-16T11:48:40.136 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:40.134+0000 7fe0467fc640 1 --2- 172.21.15.29:0/1104968250 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe0501018f0 0x7fe05019c800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T11:48:40.136 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:40.134+0000 7fe04f7fe640 1 --2- 172.21.15.29:0/1104968250 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe0501022b0 0x7fe05019ee70 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fe04800eb10 tx=0x7fe04800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:48:40.136 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:40.135+0000 7fe054911640 1 -- 172.21.15.29:0/1104968250 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe048004b20 con 0x7fe0501022b0 2024-03-16T11:48:40.136 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:40.135+0000 7fe054911640 1 -- 172.21.15.29:0/1104968250 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe048004cc0 con 0x7fe0501022b0 2024-03-16T11:48:40.136 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:40.135+0000 7fe055913640 1 -- 172.21.15.29:0/1104968250 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe05019fa20 con 0x7fe0501022b0 2024-03-16T11:48:40.136 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:40.135+0000 7fe054911640 1 -- 172.21.15.29:0/1104968250 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe048010760 con 0x7fe0501022b0 2024-03-16T11:48:40.136 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:40.135+0000 7fe055913640 1 -- 172.21.15.29:0/1104968250 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe05019fef0 con 0x7fe0501022b0 2024-03-16T11:48:40.137 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:40.136+0000 7fe055913640 1 -- 172.21.15.29:0/1104968250 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe050107a20 con 0x7fe0501022b0 2024-03-16T11:48:40.137 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:40.137+0000 7fe054911640 1 -- 172.21.15.29:0/1104968250 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fe048005000 con 0x7fe0501022b0 2024-03-16T11:48:40.137 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:40.137+0000 7fe054911640 1 --2- 172.21.15.29:0/1104968250 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fe03c0a2800 0x7fe03c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:48:40.137 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:40.137+0000 7fe04effd640 1 --2- 172.21.15.29:0/1104968250 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fe03c0a2800 0x7fe03c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:48:40.138 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:40.138+0000 7fe054911640 1 -- 172.21.15.29:0/1104968250 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7fe0480cf200 con 0x7fe0501022b0 2024-03-16T11:48:40.138 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:40.138+0000 7fe04effd640 1 --2- 172.21.15.29:0/1104968250 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fe03c0a2800 0x7fe03c0a4cc0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fe050101750 tx=0x7fe040005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:48:40.140 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:40.141+0000 7fe054911640 1 -- 172.21.15.29:0/1104968250 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fe048053740 con 0x7fe0501022b0 2024-03-16T11:48:40.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:40 smithi154 ceph-mon[36320]: from='client.? 172.21.15.29:0/2925241401' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-03-16T11:48:40.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:40 smithi154 ceph-mon[36320]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-03-16T11:48:40.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:40 smithi154 ceph-mon[36320]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-03-16T11:48:40.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:40 smithi154 ceph-mon[36320]: from='client.? 172.21.15.29:0/3698788447' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T11:48:40.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:40 smithi154 ceph-mon[36320]: osdmap e85: 12 total, 12 up, 12 in 2024-03-16T11:48:40.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:40 smithi154 ceph-mon[36320]: pgmap v291: 97 pgs: 13 creating+peering, 22 active+clean, 62 unknown; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T11:48:40.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:40 smithi154 ceph-mon[36320]: from='client.? 172.21.15.29:0/3746006667' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T11:48:40.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:40 smithi154 ceph-mon[36320]: from='client.? 172.21.15.29:0/3044224480' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-03-16T11:48:40.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:40 smithi029 ceph-mon[31975]: from='client.? 172.21.15.29:0/2925241401' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-03-16T11:48:40.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:40 smithi029 ceph-mon[31975]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-03-16T11:48:40.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:40 smithi029 ceph-mon[31975]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-03-16T11:48:40.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:40 smithi029 ceph-mon[31975]: from='client.? 172.21.15.29:0/3698788447' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T11:48:40.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:40 smithi029 ceph-mon[31975]: osdmap e85: 12 total, 12 up, 12 in 2024-03-16T11:48:40.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:40 smithi029 ceph-mon[31975]: pgmap v291: 97 pgs: 13 creating+peering, 22 active+clean, 62 unknown; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T11:48:40.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:40 smithi029 ceph-mon[31975]: from='client.? 172.21.15.29:0/3746006667' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T11:48:40.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:40 smithi029 ceph-mon[31975]: from='client.? 172.21.15.29:0/3044224480' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-03-16T11:48:40.363 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:40.364+0000 7fe055913640 1 -- 172.21.15.29:0/1104968250 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"} v 0) -- 0x7fe05002d1b0 con 0x7fe0501022b0 2024-03-16T11:48:40.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:40 smithi152 ceph-mon[36291]: from='client.? 172.21.15.29:0/2925241401' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-03-16T11:48:40.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:40 smithi152 ceph-mon[36291]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-03-16T11:48:40.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:40 smithi152 ceph-mon[36291]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-03-16T11:48:40.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:40 smithi152 ceph-mon[36291]: from='client.? 172.21.15.29:0/3698788447' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T11:48:40.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:40 smithi152 ceph-mon[36291]: osdmap e85: 12 total, 12 up, 12 in 2024-03-16T11:48:40.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:40 smithi152 ceph-mon[36291]: pgmap v291: 97 pgs: 13 creating+peering, 22 active+clean, 62 unknown; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T11:48:40.378 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:40 smithi152 ceph-mon[36291]: from='client.? 172.21.15.29:0/3746006667' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T11:48:40.378 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:40 smithi152 ceph-mon[36291]: from='client.? 172.21.15.29:0/3044224480' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-03-16T11:48:41.027 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:41.028+0000 7fe054911640 1 -- 172.21.15.29:0/1104968250 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]=0 v16) ==== 105+0+0 (secure 0 0 0) 0x7fe04809c350 con 0x7fe0501022b0 2024-03-16T11:48:41.030 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:41.030+0000 7fe055913640 1 -- 172.21.15.29:0/1104968250 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fe03c0a2800 msgr2=0x7fe03c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:48:41.030 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:41.030+0000 7fe055913640 1 --2- 172.21.15.29:0/1104968250 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fe03c0a2800 0x7fe03c0a4cc0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fe050101750 tx=0x7fe040005f90 comp rx=0 tx=0).stop 2024-03-16T11:48:41.030 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:41.030+0000 7fe055913640 1 -- 172.21.15.29:0/1104968250 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe0501022b0 msgr2=0x7fe05019ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:48:41.030 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:41.030+0000 7fe055913640 1 --2- 172.21.15.29:0/1104968250 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe0501022b0 0x7fe05019ee70 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fe04800eb10 tx=0x7fe04800c2d0 comp rx=0 tx=0).stop 2024-03-16T11:48:41.030 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:41.031+0000 7fe055913640 1 -- 172.21.15.29:0/1104968250 shutdown_connections 2024-03-16T11:48:41.030 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:41.031+0000 7fe055913640 1 --2- 172.21.15.29:0/1104968250 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fe03c0a2800 0x7fe03c0a4cc0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:41.030 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:41.031+0000 7fe055913640 1 --2- 172.21.15.29:0/1104968250 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe0501022b0 0x7fe05019ee70 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:41.030 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:41.031+0000 7fe055913640 1 --2- 172.21.15.29:0/1104968250 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe0501018f0 0x7fe05019c800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:41.030 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:41.031+0000 7fe055913640 1 --2- 172.21.15.29:0/1104968250 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe050100700 0x7fe05006a980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:41.030 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:41.031+0000 7fe055913640 1 -- 172.21.15.29:0/1104968250 >> 172.21.15.29:0/1104968250 conn(0x7fe0500fbc90 msgr2=0x7fe050104bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:48:41.031 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:41.031+0000 7fe055913640 1 -- 172.21.15.29:0/1104968250 shutdown_connections 2024-03-16T11:48:41.031 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:41.031+0000 7fe055913640 1 -- 172.21.15.29:0/1104968250 wait complete. 2024-03-16T11:48:41.044 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay true 2024-03-16T11:48:41.044 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:41 smithi029 ceph-mon[31975]: from='client.? 172.21.15.29:0/3044224480' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-03-16T11:48:41.045 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:41 smithi029 ceph-mon[31975]: mds.? [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] up:active 2024-03-16T11:48:41.045 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:41 smithi029 ceph-mon[31975]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-16T11:48:41.045 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:41 smithi029 ceph-mon[31975]: from='client.? 172.21.15.29:0/1104968250' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-03-16T11:48:41.045 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:41 smithi029 ceph-mon[31975]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-16T11:48:41.045 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:41 smithi029 ceph-mon[31975]: Cluster is now healthy 2024-03-16T11:48:41.156 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:41.154+0000 7fbe79075640 1 -- 172.21.15.29:0/3581437856 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbe741075a0 msgr2=0x7fbe74109990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:48:41.156 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:41.154+0000 7fbe79075640 1 --2- 172.21.15.29:0/3581437856 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbe741075a0 0x7fbe74109990 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fbe64009ca0 tx=0x7fbe6402c0f0 comp rx=0 tx=0).stop 2024-03-16T11:48:41.156 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:41.154+0000 7fbe79075640 1 -- 172.21.15.29:0/3581437856 shutdown_connections 2024-03-16T11:48:41.156 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:41.154+0000 7fbe79075640 1 --2- 172.21.15.29:0/3581437856 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbe741075a0 0x7fbe74109990 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:41.156 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:41.154+0000 7fbe79075640 1 --2- 172.21.15.29:0/3581437856 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbe74104c70 0x7fbe74107060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:41.156 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:41.154+0000 7fbe79075640 1 --2- 172.21.15.29:0/3581437856 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbe74102340 0x7fbe74104730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:41.156 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:41.154+0000 7fbe79075640 1 -- 172.21.15.29:0/3581437856 >> 172.21.15.29:0/3581437856 conn(0x7fbe740fbc90 msgr2=0x7fbe740fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:48:41.156 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:41.154+0000 7fbe79075640 1 -- 172.21.15.29:0/3581437856 shutdown_connections 2024-03-16T11:48:41.157 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:41.154+0000 7fbe79075640 1 -- 172.21.15.29:0/3581437856 wait complete. 2024-03-16T11:48:41.157 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:41.155+0000 7fbe79075640 1 Processor -- start 2024-03-16T11:48:41.157 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:41.155+0000 7fbe79075640 1 -- start start 2024-03-16T11:48:41.157 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:41.155+0000 7fbe79075640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbe74102340 0x7fbe74197ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:48:41.157 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:41.155+0000 7fbe79075640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbe74104c70 0x7fbe74198010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:48:41.157 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:41.155+0000 7fbe79075640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbe741075a0 0x7fbe7419f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:48:41.157 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:41.155+0000 7fbe79075640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fbe7410d490 con 0x7fbe74102340 2024-03-16T11:48:41.157 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:41.155+0000 7fbe79075640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fbe7410d310 con 0x7fbe741075a0 2024-03-16T11:48:41.157 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:41.155+0000 7fbe79075640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fbe7410d610 con 0x7fbe74104c70 2024-03-16T11:48:41.157 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:41.155+0000 7fbe72d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbe74102340 0x7fbe74197ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:48:41.157 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:41.155+0000 7fbe72d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbe74102340 0x7fbe74197ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49058/0 (socket says 172.21.15.29:49058) 2024-03-16T11:48:41.157 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:41.155+0000 7fbe72d9d640 1 -- 172.21.15.29:0/1750482126 learned_addr learned my addr 172.21.15.29:0/1750482126 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:48:41.158 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:41.155+0000 7fbe7359e640 1 --2- 172.21.15.29:0/1750482126 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbe741075a0 0x7fbe7419f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:48:41.158 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:41.155+0000 7fbe7259c640 1 --2- 172.21.15.29:0/1750482126 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbe74104c70 0x7fbe74198010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:48:41.158 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:41.156+0000 7fbe72d9d640 1 -- 172.21.15.29:0/1750482126 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbe74104c70 msgr2=0x7fbe74198010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:48:41.158 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:41.156+0000 7fbe72d9d640 1 --2- 172.21.15.29:0/1750482126 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbe74104c70 0x7fbe74198010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:41.158 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:41.156+0000 7fbe72d9d640 1 -- 172.21.15.29:0/1750482126 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbe741075a0 msgr2=0x7fbe7419f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:48:41.158 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:41.156+0000 7fbe72d9d640 1 --2- 172.21.15.29:0/1750482126 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbe741075a0 0x7fbe7419f080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:41.158 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:41.156+0000 7fbe72d9d640 1 -- 172.21.15.29:0/1750482126 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe7419f6f0 con 0x7fbe74102340 2024-03-16T11:48:41.158 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:41.156+0000 7fbe72d9d640 1 --2- 172.21.15.29:0/1750482126 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbe74102340 0x7fbe74197ad0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fbe6c00e9f0 tx=0x7fbe6c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:48:41.158 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:41.156+0000 7fbe5bfff640 1 -- 172.21.15.29:0/1750482126 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe6c00cd80 con 0x7fbe74102340 2024-03-16T11:48:41.158 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:41.156+0000 7fbe79075640 1 -- 172.21.15.29:0/1750482126 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbe7419f9e0 con 0x7fbe74102340 2024-03-16T11:48:41.158 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:41.156+0000 7fbe5bfff640 1 -- 172.21.15.29:0/1750482126 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbe6c010910 con 0x7fbe74102340 2024-03-16T11:48:41.158 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:41.156+0000 7fbe5bfff640 1 -- 172.21.15.29:0/1750482126 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe6c0188f0 con 0x7fbe74102340 2024-03-16T11:48:41.158 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:41.156+0000 7fbe79075640 1 -- 172.21.15.29:0/1750482126 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbe7419ff30 con 0x7fbe74102340 2024-03-16T11:48:41.159 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:41.157+0000 7fbe79075640 1 -- 172.21.15.29:0/1750482126 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbe74068a20 con 0x7fbe74102340 2024-03-16T11:48:41.159 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:41.158+0000 7fbe5bfff640 1 -- 172.21.15.29:0/1750482126 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fbe6c010ac0 con 0x7fbe74102340 2024-03-16T11:48:41.159 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:41.158+0000 7fbe5bfff640 1 --2- 172.21.15.29:0/1750482126 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fbe5c0a2800 0x7fbe5c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:48:41.159 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:41.159+0000 7fbe5bfff640 1 -- 172.21.15.29:0/1750482126 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7fbe6c0ced70 con 0x7fbe74102340 2024-03-16T11:48:41.159 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:41.159+0000 7fbe7259c640 1 --2- 172.21.15.29:0/1750482126 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fbe5c0a2800 0x7fbe5c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:48:41.159 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:41.159+0000 7fbe7259c640 1 --2- 172.21.15.29:0/1750482126 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fbe5c0a2800 0x7fbe5c0a4cc0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fbe68005fd0 tx=0x7fbe68005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:48:41.162 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:41.163+0000 7fbe5bfff640 1 -- 172.21.15.29:0/1750482126 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fbe6c09bf60 con 0x7fbe74102340 2024-03-16T11:48:41.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:41 smithi154 ceph-mon[36320]: from='client.? 172.21.15.29:0/3044224480' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-03-16T11:48:41.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:41 smithi154 ceph-mon[36320]: mds.? [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] up:active 2024-03-16T11:48:41.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:41 smithi154 ceph-mon[36320]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-16T11:48:41.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:41 smithi154 ceph-mon[36320]: from='client.? 172.21.15.29:0/1104968250' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-03-16T11:48:41.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:41 smithi154 ceph-mon[36320]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-16T11:48:41.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:41 smithi154 ceph-mon[36320]: Cluster is now healthy 2024-03-16T11:48:41.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:41 smithi152 ceph-mon[36291]: from='client.? 172.21.15.29:0/3044224480' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-03-16T11:48:41.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:41 smithi152 ceph-mon[36291]: mds.? [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] up:active 2024-03-16T11:48:41.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:41 smithi152 ceph-mon[36291]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-16T11:48:41.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:41 smithi152 ceph-mon[36291]: from='client.? 172.21.15.29:0/1104968250' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-03-16T11:48:41.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:41 smithi152 ceph-mon[36291]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-16T11:48:41.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:41 smithi152 ceph-mon[36291]: Cluster is now healthy 2024-03-16T11:48:41.387 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:41.388+0000 7fbe79075640 1 -- 172.21.15.29:0/1750482126 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} v 0) -- 0x7fbe740fd780 con 0x7fbe74102340 2024-03-16T11:48:41.956 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:41.956+0000 7fbe5bfff640 1 -- 172.21.15.29:0/1750482126 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]=0 v18) ==== 121+0+0 (secure 0 0 0) 0x7fbe6c09b860 con 0x7fbe74102340 2024-03-16T11:48:41.959 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:41.959+0000 7fbe79075640 1 -- 172.21.15.29:0/1750482126 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fbe5c0a2800 msgr2=0x7fbe5c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:48:41.959 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:41.959+0000 7fbe79075640 1 --2- 172.21.15.29:0/1750482126 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fbe5c0a2800 0x7fbe5c0a4cc0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fbe68005fd0 tx=0x7fbe68005850 comp rx=0 tx=0).stop 2024-03-16T11:48:41.959 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:41.959+0000 7fbe79075640 1 -- 172.21.15.29:0/1750482126 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbe74102340 msgr2=0x7fbe74197ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:48:41.959 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:41.959+0000 7fbe79075640 1 --2- 172.21.15.29:0/1750482126 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbe74102340 0x7fbe74197ad0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fbe6c00e9f0 tx=0x7fbe6c00eeb0 comp rx=0 tx=0).stop 2024-03-16T11:48:41.959 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:41.959+0000 7fbe79075640 1 -- 172.21.15.29:0/1750482126 shutdown_connections 2024-03-16T11:48:41.959 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:41.959+0000 7fbe79075640 1 --2- 172.21.15.29:0/1750482126 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fbe5c0a2800 0x7fbe5c0a4cc0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:41.959 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:41.959+0000 7fbe79075640 1 --2- 172.21.15.29:0/1750482126 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbe741075a0 0x7fbe7419f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:41.959 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:41.959+0000 7fbe79075640 1 --2- 172.21.15.29:0/1750482126 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbe74104c70 0x7fbe74198010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:41.959 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:41.959+0000 7fbe79075640 1 --2- 172.21.15.29:0/1750482126 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbe74102340 0x7fbe74197ad0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:41.959 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:41.960+0000 7fbe79075640 1 -- 172.21.15.29:0/1750482126 >> 172.21.15.29:0/1750482126 conn(0x7fbe740fbc90 msgr2=0x7fbe74102ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:48:41.959 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:41.960+0000 7fbe79075640 1 -- 172.21.15.29:0/1750482126 shutdown_connections 2024-03-16T11:48:41.960 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:41.960+0000 7fbe79075640 1 -- 172.21.15.29:0/1750482126 wait complete. 2024-03-16T11:48:41.978 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs session_timeout 300 2024-03-16T11:48:42.090 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:42.090+0000 7fd5c921f640 1 -- 172.21.15.29:0/2989853461 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd5c41029f0 msgr2=0x7fd5c4102e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:48:42.090 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:42.090+0000 7fd5c921f640 1 --2- 172.21.15.29:0/2989853461 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd5c41029f0 0x7fd5c4102e70 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fd5ac009900 tx=0x7fd5ac02bc30 comp rx=0 tx=0).stop 2024-03-16T11:48:42.090 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:42.090+0000 7fd5c921f640 1 -- 172.21.15.29:0/2989853461 shutdown_connections 2024-03-16T11:48:42.090 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:42.090+0000 7fd5c921f640 1 --2- 172.21.15.29:0/2989853461 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd5c41033b0 0x7fd5c4109c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:42.090 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:42.090+0000 7fd5c921f640 1 --2- 172.21.15.29:0/2989853461 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd5c41029f0 0x7fd5c4102e70 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:42.090 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:42.090+0000 7fd5c921f640 1 --2- 172.21.15.29:0/2989853461 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd5c4101800 0x7fd5c4101c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:42.090 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:42.090+0000 7fd5c921f640 1 -- 172.21.15.29:0/2989853461 >> 172.21.15.29:0/2989853461 conn(0x7fd5c40fcd70 msgr2=0x7fd5c40ff1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:48:42.090 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:42.090+0000 7fd5c921f640 1 -- 172.21.15.29:0/2989853461 shutdown_connections 2024-03-16T11:48:42.090 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:42.090+0000 7fd5c921f640 1 -- 172.21.15.29:0/2989853461 wait complete. 2024-03-16T11:48:42.090 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:42.091+0000 7fd5c921f640 1 Processor -- start 2024-03-16T11:48:42.093 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:42.091+0000 7fd5c921f640 1 -- start start 2024-03-16T11:48:42.093 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:42.091+0000 7fd5c921f640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd5c4101800 0x7fd5c4197ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:48:42.093 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:42.091+0000 7fd5c921f640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd5c41029f0 0x7fd5c4198010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:48:42.093 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:42.091+0000 7fd5c921f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd5c41033b0 0x7fd5c419f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:48:42.093 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:42.091+0000 7fd5c921f640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fd5c410d460 con 0x7fd5c41033b0 2024-03-16T11:48:42.093 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:42.091+0000 7fd5c921f640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fd5c410d2e0 con 0x7fd5c41029f0 2024-03-16T11:48:42.093 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:42.091+0000 7fd5c921f640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fd5c410d5e0 con 0x7fd5c4101800 2024-03-16T11:48:42.093 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:42.091+0000 7fd5c359e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd5c41033b0 0x7fd5c419f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:48:42.093 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:42.091+0000 7fd5c359e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd5c41033b0 0x7fd5c419f080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49086/0 (socket says 172.21.15.29:49086) 2024-03-16T11:48:42.093 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:42.091+0000 7fd5c359e640 1 -- 172.21.15.29:0/913975390 learned_addr learned my addr 172.21.15.29:0/913975390 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:48:42.093 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:42.091+0000 7fd5c2d9d640 1 --2- 172.21.15.29:0/913975390 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd5c4101800 0x7fd5c4197ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:48:42.094 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:42.091+0000 7fd5ba59c640 1 --2- 172.21.15.29:0/913975390 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd5c41029f0 0x7fd5c4198010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:48:42.094 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:42.092+0000 7fd5c359e640 1 -- 172.21.15.29:0/913975390 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd5c4101800 msgr2=0x7fd5c4197ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:48:42.094 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:42.092+0000 7fd5c359e640 1 --2- 172.21.15.29:0/913975390 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd5c4101800 0x7fd5c4197ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:42.094 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:42.092+0000 7fd5c359e640 1 -- 172.21.15.29:0/913975390 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd5c41029f0 msgr2=0x7fd5c4198010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:48:42.094 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:42.092+0000 7fd5c359e640 1 --2- 172.21.15.29:0/913975390 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd5c41029f0 0x7fd5c4198010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:42.094 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:42.092+0000 7fd5c359e640 1 -- 172.21.15.29:0/913975390 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5c419f6f0 con 0x7fd5c41033b0 2024-03-16T11:48:42.094 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:42.092+0000 7fd5ba59c640 1 --2- 172.21.15.29:0/913975390 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd5c41029f0 0x7fd5c4198010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T11:48:42.094 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:42.092+0000 7fd5c359e640 1 --2- 172.21.15.29:0/913975390 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd5c41033b0 0x7fd5c419f080 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fd5bc00eb10 tx=0x7fd5bc00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:48:42.094 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:42.092+0000 7fd5c0d99640 1 -- 172.21.15.29:0/913975390 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5bc004b20 con 0x7fd5c41033b0 2024-03-16T11:48:42.094 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:42.092+0000 7fd5c0d99640 1 -- 172.21.15.29:0/913975390 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd5bc004cc0 con 0x7fd5c41033b0 2024-03-16T11:48:42.094 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:42.092+0000 7fd5c921f640 1 -- 172.21.15.29:0/913975390 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd5c419f9e0 con 0x7fd5c41033b0 2024-03-16T11:48:42.095 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:42.092+0000 7fd5c0d99640 1 -- 172.21.15.29:0/913975390 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5bc010760 con 0x7fd5c41033b0 2024-03-16T11:48:42.095 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:42.092+0000 7fd5c921f640 1 -- 172.21.15.29:0/913975390 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5c419ff00 con 0x7fd5c41033b0 2024-03-16T11:48:42.095 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:42.093+0000 7fd5c921f640 1 -- 172.21.15.29:0/913975390 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd5c4108b20 con 0x7fd5c41033b0 2024-03-16T11:48:42.095 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:42.094+0000 7fd5c0d99640 1 -- 172.21.15.29:0/913975390 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fd5bc005020 con 0x7fd5c41033b0 2024-03-16T11:48:42.095 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:42.095+0000 7fd5c0d99640 1 --2- 172.21.15.29:0/913975390 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fd5b00a2880 0x7fd5b00a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:48:42.096 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:42.095+0000 7fd5c2d9d640 1 --2- 172.21.15.29:0/913975390 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fd5b00a2880 0x7fd5b00a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:48:42.096 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:42.095+0000 7fd5c0d99640 1 -- 172.21.15.29:0/913975390 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7fd5bc0cef40 con 0x7fd5c41033b0 2024-03-16T11:48:42.096 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:42.095+0000 7fd5c2d9d640 1 --2- 172.21.15.29:0/913975390 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fd5b00a2880 0x7fd5b00a4d40 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fd5c4102850 tx=0x7fd5b4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:48:42.098 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:42.099+0000 7fd5c0d99640 1 -- 172.21.15.29:0/913975390 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fd5bc09c090 con 0x7fd5c41033b0 2024-03-16T11:48:42.279 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:42 smithi152 ceph-mon[36291]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-03-16T11:48:42.279 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:42 smithi152 ceph-mon[36291]: from='client.? 172.21.15.29:0/1104968250' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2024-03-16T11:48:42.279 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:42 smithi152 ceph-mon[36291]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-03-16T11:48:42.279 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:42 smithi152 ceph-mon[36291]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-16T11:48:42.279 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:42 smithi152 ceph-mon[36291]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2024-03-16T11:48:42.279 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:42 smithi152 ceph-mon[36291]: daemon mds.i is now active in filesystem cephfs as rank 1 2024-03-16T11:48:42.279 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:42 smithi152 ceph-mon[36291]: from='client.? 172.21.15.29:0/1750482126' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-16T11:48:42.279 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:42 smithi152 ceph-mon[36291]: pgmap v292: 97 pgs: 13 creating+peering, 67 active+clean, 17 unknown; 578 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s wr, 7 op/s 2024-03-16T11:48:42.279 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:42 smithi152 ceph-mon[36291]: from='client.? 172.21.15.29:0/1750482126' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-03-16T11:48:42.279 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:42 smithi152 ceph-mon[36291]: mds.? [v2:172.21.15.154:6838/685736862,v1:172.21.15.154:6839/685736862] up:active 2024-03-16T11:48:42.279 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:42 smithi152 ceph-mon[36291]: daemon mds.h assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-03-16T11:48:42.280 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:42 smithi152 ceph-mon[36291]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-03-16T11:48:42.280 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:42 smithi152 ceph-mon[36291]: Cluster is now healthy 2024-03-16T11:48:42.280 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:42 smithi152 ceph-mon[36291]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 2 up:standby-replay 8 up:standby 2024-03-16T11:48:42.280 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:42 smithi152 ceph-mon[36291]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:creating} 2 up:standby-replay 7 up:standby 2024-03-16T11:48:42.280 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:42 smithi152 ceph-mon[36291]: daemon mds.h is now active in filesystem cephfs as rank 2 2024-03-16T11:48:42.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:42 smithi154 ceph-mon[36320]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-03-16T11:48:42.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:42 smithi154 ceph-mon[36320]: from='client.? 172.21.15.29:0/1104968250' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2024-03-16T11:48:42.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:42 smithi154 ceph-mon[36320]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-03-16T11:48:42.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:42 smithi154 ceph-mon[36320]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-16T11:48:42.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:42 smithi154 ceph-mon[36320]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2024-03-16T11:48:42.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:42 smithi154 ceph-mon[36320]: daemon mds.i is now active in filesystem cephfs as rank 1 2024-03-16T11:48:42.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:42 smithi154 ceph-mon[36320]: from='client.? 172.21.15.29:0/1750482126' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-16T11:48:42.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:42 smithi154 ceph-mon[36320]: pgmap v292: 97 pgs: 13 creating+peering, 67 active+clean, 17 unknown; 578 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s wr, 7 op/s 2024-03-16T11:48:42.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:42 smithi154 ceph-mon[36320]: from='client.? 172.21.15.29:0/1750482126' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-03-16T11:48:42.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:42 smithi154 ceph-mon[36320]: mds.? [v2:172.21.15.154:6838/685736862,v1:172.21.15.154:6839/685736862] up:active 2024-03-16T11:48:42.296 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:42 smithi154 ceph-mon[36320]: daemon mds.h assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-03-16T11:48:42.296 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:42 smithi154 ceph-mon[36320]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-03-16T11:48:42.296 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:42 smithi154 ceph-mon[36320]: Cluster is now healthy 2024-03-16T11:48:42.296 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:42 smithi154 ceph-mon[36320]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 2 up:standby-replay 8 up:standby 2024-03-16T11:48:42.296 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:42 smithi154 ceph-mon[36320]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:creating} 2 up:standby-replay 7 up:standby 2024-03-16T11:48:42.296 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:42 smithi154 ceph-mon[36320]: daemon mds.h is now active in filesystem cephfs as rank 2 2024-03-16T11:48:42.321 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:42.322+0000 7fd5c921f640 1 -- 172.21.15.29:0/913975390 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) -- 0x7fd5c402d200 con 0x7fd5c41033b0 2024-03-16T11:48:42.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:42 smithi029 ceph-mon[31975]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-03-16T11:48:42.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:42 smithi029 ceph-mon[31975]: from='client.? 172.21.15.29:0/1104968250' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2024-03-16T11:48:42.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:42 smithi029 ceph-mon[31975]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-03-16T11:48:42.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:42 smithi029 ceph-mon[31975]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-16T11:48:42.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:42 smithi029 ceph-mon[31975]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2024-03-16T11:48:42.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:42 smithi029 ceph-mon[31975]: daemon mds.i is now active in filesystem cephfs as rank 1 2024-03-16T11:48:42.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:42 smithi029 ceph-mon[31975]: from='client.? 172.21.15.29:0/1750482126' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-16T11:48:42.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:42 smithi029 ceph-mon[31975]: pgmap v292: 97 pgs: 13 creating+peering, 67 active+clean, 17 unknown; 578 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s wr, 7 op/s 2024-03-16T11:48:42.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:42 smithi029 ceph-mon[31975]: from='client.? 172.21.15.29:0/1750482126' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-03-16T11:48:42.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:42 smithi029 ceph-mon[31975]: mds.? [v2:172.21.15.154:6838/685736862,v1:172.21.15.154:6839/685736862] up:active 2024-03-16T11:48:42.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:42 smithi029 ceph-mon[31975]: daemon mds.h assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-03-16T11:48:42.339 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:42 smithi029 ceph-mon[31975]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-03-16T11:48:42.339 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:42 smithi029 ceph-mon[31975]: Cluster is now healthy 2024-03-16T11:48:42.339 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:42 smithi029 ceph-mon[31975]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 2 up:standby-replay 8 up:standby 2024-03-16T11:48:42.339 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:42 smithi029 ceph-mon[31975]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:creating} 2 up:standby-replay 7 up:standby 2024-03-16T11:48:42.339 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:42 smithi029 ceph-mon[31975]: daemon mds.h is now active in filesystem cephfs as rank 2 2024-03-16T11:48:42.963 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:42.964+0000 7fd5c0d99640 1 -- 172.21.15.29:0/913975390 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v20) ==== 115+0+0 (secure 0 0 0) 0x7fd5bc09b990 con 0x7fd5c41033b0 2024-03-16T11:48:42.966 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:42.967+0000 7fd5c921f640 1 -- 172.21.15.29:0/913975390 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fd5b00a2880 msgr2=0x7fd5b00a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:48:42.967 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:42.967+0000 7fd5c921f640 1 --2- 172.21.15.29:0/913975390 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fd5b00a2880 0x7fd5b00a4d40 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fd5c4102850 tx=0x7fd5b4005f90 comp rx=0 tx=0).stop 2024-03-16T11:48:42.967 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:42.967+0000 7fd5c921f640 1 -- 172.21.15.29:0/913975390 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd5c41033b0 msgr2=0x7fd5c419f080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:48:42.967 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:42.967+0000 7fd5c921f640 1 --2- 172.21.15.29:0/913975390 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd5c41033b0 0x7fd5c419f080 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fd5bc00eb10 tx=0x7fd5bc00c2d0 comp rx=0 tx=0).stop 2024-03-16T11:48:42.967 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:42.967+0000 7fd5c921f640 1 -- 172.21.15.29:0/913975390 shutdown_connections 2024-03-16T11:48:42.967 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:42.967+0000 7fd5c921f640 1 --2- 172.21.15.29:0/913975390 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fd5b00a2880 0x7fd5b00a4d40 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:42.967 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:42.967+0000 7fd5c921f640 1 --2- 172.21.15.29:0/913975390 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd5c41033b0 0x7fd5c419f080 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:42.967 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:42.967+0000 7fd5c921f640 1 --2- 172.21.15.29:0/913975390 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd5c41029f0 0x7fd5c4198010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:42.967 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:42.967+0000 7fd5c921f640 1 --2- 172.21.15.29:0/913975390 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd5c4101800 0x7fd5c4197ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:42.967 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:42.967+0000 7fd5c921f640 1 -- 172.21.15.29:0/913975390 >> 172.21.15.29:0/913975390 conn(0x7fd5c40fcd70 msgr2=0x7fd5c4105cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:48:42.967 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:42.967+0000 7fd5c921f640 1 -- 172.21.15.29:0/913975390 shutdown_connections 2024-03-16T11:48:42.967 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:42.967+0000 7fd5c921f640 1 -- 172.21.15.29:0/913975390 wait complete. 2024-03-16T11:48:42.980 DEBUG:tasks.cephfs.filesystem:Creating {'create': 2} subvols for filesystem 'cephfs' 2024-03-16T11:48:42.981 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolumegroup create cephfs qa 2024-03-16T11:48:43.095 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:43.094+0000 7f2f0b5c6640 1 -- 172.21.15.29:0/1462299013 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2f04100fd0 msgr2=0x7f2f04101450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:48:43.095 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:43.094+0000 7f2f0b5c6640 1 --2- 172.21.15.29:0/1462299013 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2f04100fd0 0x7f2f04101450 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f2ef4009900 tx=0x7f2ef402bc30 comp rx=0 tx=0).stop 2024-03-16T11:48:43.096 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:43.094+0000 7f2f0b5c6640 1 -- 172.21.15.29:0/1462299013 shutdown_connections 2024-03-16T11:48:43.096 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:43.094+0000 7f2f0b5c6640 1 --2- 172.21.15.29:0/1462299013 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2f04101990 0x7f2f041082b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:43.096 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:43.094+0000 7f2f0b5c6640 1 --2- 172.21.15.29:0/1462299013 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2f04100fd0 0x7f2f04101450 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:43.096 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:43.094+0000 7f2f0b5c6640 1 --2- 172.21.15.29:0/1462299013 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2f040ffe80 0x7f2f041002a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:43.096 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:43.094+0000 7f2f0b5c6640 1 -- 172.21.15.29:0/1462299013 >> 172.21.15.29:0/1462299013 conn(0x7f2f040fb410 msgr2=0x7f2f040fd830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:48:43.096 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:43.094+0000 7f2f0b5c6640 1 -- 172.21.15.29:0/1462299013 shutdown_connections 2024-03-16T11:48:43.096 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:43.094+0000 7f2f0b5c6640 1 -- 172.21.15.29:0/1462299013 wait complete. 2024-03-16T11:48:43.096 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:43.095+0000 7f2f0b5c6640 1 Processor -- start 2024-03-16T11:48:43.096 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:43.095+0000 7f2f0b5c6640 1 -- start start 2024-03-16T11:48:43.096 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:43.095+0000 7f2f0b5c6640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2f040ffe80 0x7f2f04197b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:48:43.096 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:43.095+0000 7f2f0b5c6640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2f04100fd0 0x7f2f04198070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:48:43.096 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:43.095+0000 7f2f0b5c6640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2f04101990 0x7f2f0419f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:48:43.096 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:43.095+0000 7f2f0b5c6640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f2f0410b2a0 con 0x7f2f040ffe80 2024-03-16T11:48:43.096 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:43.095+0000 7f2f0b5c6640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f2f0410b120 con 0x7f2f04100fd0 2024-03-16T11:48:43.097 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:43.095+0000 7f2f0b5c6640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f2f0410b420 con 0x7f2f04101990 2024-03-16T11:48:43.097 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:43.095+0000 7f2f09362640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2f040ffe80 0x7f2f04197b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:48:43.097 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:43.095+0000 7f2f09362640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2f040ffe80 0x7f2f04197b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49116/0 (socket says 172.21.15.29:49116) 2024-03-16T11:48:43.097 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:43.095+0000 7f2f09362640 1 -- 172.21.15.29:0/708051456 learned_addr learned my addr 172.21.15.29:0/708051456 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:48:43.097 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:43.095+0000 7f2f08b61640 1 --2- 172.21.15.29:0/708051456 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2f04100fd0 0x7f2f04198070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:48:43.097 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:43.095+0000 7f2f09b63640 1 --2- 172.21.15.29:0/708051456 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2f04101990 0x7f2f0419f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:48:43.097 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:43.095+0000 7f2f09362640 1 -- 172.21.15.29:0/708051456 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2f04101990 msgr2=0x7f2f0419f0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:48:43.097 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:43.095+0000 7f2f09362640 1 --2- 172.21.15.29:0/708051456 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2f04101990 0x7f2f0419f0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:43.097 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:43.095+0000 7f2f09362640 1 -- 172.21.15.29:0/708051456 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2f04100fd0 msgr2=0x7f2f04198070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:48:43.097 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:43.095+0000 7f2f09362640 1 --2- 172.21.15.29:0/708051456 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2f04100fd0 0x7f2f04198070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:43.097 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:43.095+0000 7f2f09362640 1 -- 172.21.15.29:0/708051456 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f0419f750 con 0x7f2f040ffe80 2024-03-16T11:48:43.097 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:43.095+0000 7f2f09362640 1 --2- 172.21.15.29:0/708051456 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2f040ffe80 0x7f2f04197b30 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f2f0000ea40 tx=0x7f2f0000ef00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:48:43.097 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:43.095+0000 7f2efa7fc640 1 -- 172.21.15.29:0/708051456 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f0000cd40 con 0x7f2f040ffe80 2024-03-16T11:48:43.097 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:43.096+0000 7f2efa7fc640 1 -- 172.21.15.29:0/708051456 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2f00005500 con 0x7f2f040ffe80 2024-03-16T11:48:43.098 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:43.096+0000 7f2f0b5c6640 1 -- 172.21.15.29:0/708051456 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2f0419fa40 con 0x7f2f040ffe80 2024-03-16T11:48:43.098 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:43.096+0000 7f2efa7fc640 1 -- 172.21.15.29:0/708051456 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f00016670 con 0x7f2f040ffe80 2024-03-16T11:48:43.098 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:43.096+0000 7f2f0b5c6640 1 -- 172.21.15.29:0/708051456 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2f0419ff10 con 0x7f2f040ffe80 2024-03-16T11:48:43.098 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:43.096+0000 7f2f0b5c6640 1 -- 172.21.15.29:0/708051456 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2f04107210 con 0x7f2f040ffe80 2024-03-16T11:48:43.099 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:43.098+0000 7f2efa7fc640 1 -- 172.21.15.29:0/708051456 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f2f000046c0 con 0x7f2f040ffe80 2024-03-16T11:48:43.099 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:43.098+0000 7f2efa7fc640 1 --2- 172.21.15.29:0/708051456 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f2ef00a2800 0x7f2ef00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:48:43.099 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:43.098+0000 7f2efa7fc640 1 -- 172.21.15.29:0/708051456 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f2f000cef20 con 0x7f2f040ffe80 2024-03-16T11:48:43.099 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:43.098+0000 7f2f08b61640 1 --2- 172.21.15.29:0/708051456 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f2ef00a2800 0x7f2ef00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:48:43.099 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:43.099+0000 7f2f08b61640 1 --2- 172.21.15.29:0/708051456 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f2ef00a2800 0x7f2ef00a4cc0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f2ef4000c00 tx=0x7f2ef4005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:48:43.101 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:43.102+0000 7f2efa7fc640 1 -- 172.21.15.29:0/708051456 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f2f0009bac0 con 0x7f2f040ffe80 2024-03-16T11:48:43.288 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:43 smithi152 ceph-mon[36291]: from='client.? 172.21.15.29:0/913975390' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-03-16T11:48:43.289 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:43 smithi152 ceph-mon[36291]: from='client.? 172.21.15.29:0/913975390' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-03-16T11:48:43.289 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:43 smithi152 ceph-mon[36291]: mds.? [v2:172.21.15.152:6836/1323866722,v1:172.21.15.152:6837/1323866722] up:active 2024-03-16T11:48:43.289 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:43 smithi152 ceph-mon[36291]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 2 up:standby-replay 7 up:standby 2024-03-16T11:48:43.289 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:43 smithi152 ceph-mon[36291]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-03-16T11:48:43.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:43 smithi154 ceph-mon[36320]: from='client.? 172.21.15.29:0/913975390' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-03-16T11:48:43.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:43 smithi154 ceph-mon[36320]: from='client.? 172.21.15.29:0/913975390' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-03-16T11:48:43.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:43 smithi154 ceph-mon[36320]: mds.? [v2:172.21.15.152:6836/1323866722,v1:172.21.15.152:6837/1323866722] up:active 2024-03-16T11:48:43.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:43 smithi154 ceph-mon[36320]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 2 up:standby-replay 7 up:standby 2024-03-16T11:48:43.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:43 smithi154 ceph-mon[36320]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-03-16T11:48:43.310 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:43.311+0000 7f2f0b5c6640 1 -- 172.21.15.29:0/708051456 --> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7f2f04061c20 con 0x7f2ef00a2800 2024-03-16T11:48:43.316 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:43 smithi029 ceph-mon[31975]: from='client.? 172.21.15.29:0/913975390' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-03-16T11:48:43.316 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:43 smithi029 ceph-mon[31975]: from='client.? 172.21.15.29:0/913975390' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-03-16T11:48:43.316 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:43 smithi029 ceph-mon[31975]: mds.? [v2:172.21.15.152:6836/1323866722,v1:172.21.15.152:6837/1323866722] up:active 2024-03-16T11:48:43.316 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:43 smithi029 ceph-mon[31975]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 2 up:standby-replay 7 up:standby 2024-03-16T11:48:43.316 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:43 smithi029 ceph-mon[31975]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-03-16T11:48:43.336 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:43.337+0000 7f2efa7fc640 1 -- 172.21.15.29:0/708051456 <== mgr.14152 v2:172.21.15.29:6800/4192064640 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f2f04061c20 con 0x7f2ef00a2800 2024-03-16T11:48:43.339 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:43.339+0000 7f2f0b5c6640 1 -- 172.21.15.29:0/708051456 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f2ef00a2800 msgr2=0x7f2ef00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:48:43.339 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:43.339+0000 7f2f0b5c6640 1 --2- 172.21.15.29:0/708051456 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f2ef00a2800 0x7f2ef00a4cc0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f2ef4000c00 tx=0x7f2ef4005fd0 comp rx=0 tx=0).stop 2024-03-16T11:48:43.339 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:43.339+0000 7f2f0b5c6640 1 -- 172.21.15.29:0/708051456 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2f040ffe80 msgr2=0x7f2f04197b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:48:43.339 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:43.339+0000 7f2f0b5c6640 1 --2- 172.21.15.29:0/708051456 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2f040ffe80 0x7f2f04197b30 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f2f0000ea40 tx=0x7f2f0000ef00 comp rx=0 tx=0).stop 2024-03-16T11:48:43.339 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:43.339+0000 7f2f0b5c6640 1 -- 172.21.15.29:0/708051456 shutdown_connections 2024-03-16T11:48:43.339 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:43.339+0000 7f2f0b5c6640 1 --2- 172.21.15.29:0/708051456 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f2ef00a2800 0x7f2ef00a4cc0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:43.339 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:43.339+0000 7f2f0b5c6640 1 --2- 172.21.15.29:0/708051456 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2f04101990 0x7f2f0419f0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:43.339 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:43.339+0000 7f2f0b5c6640 1 --2- 172.21.15.29:0/708051456 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2f04100fd0 0x7f2f04198070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:43.339 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:43.339+0000 7f2f0b5c6640 1 --2- 172.21.15.29:0/708051456 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2f040ffe80 0x7f2f04197b30 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:43.339 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:43.339+0000 7f2f0b5c6640 1 -- 172.21.15.29:0/708051456 >> 172.21.15.29:0/708051456 conn(0x7f2f040fb410 msgr2=0x7f2f041043e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:48:43.339 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:43.339+0000 7f2f0b5c6640 1 -- 172.21.15.29:0/708051456 shutdown_connections 2024-03-16T11:48:43.339 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:43.339+0000 7f2f0b5c6640 1 -- 172.21.15.29:0/708051456 wait complete. 2024-03-16T11:48:43.349 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_0 --group_name qa 2024-03-16T11:48:43.460 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:43.458+0000 7fbdf5790640 1 -- 172.21.15.29:0/1861938984 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbdf01018d0 msgr2=0x7fbdf0101d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:48:43.460 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:43.458+0000 7fbdf5790640 1 --2- 172.21.15.29:0/1861938984 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbdf01018d0 0x7fbdf0101d50 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fbdd8009900 tx=0x7fbdd802bc30 comp rx=0 tx=0).stop 2024-03-16T11:48:43.460 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:43.458+0000 7fbdf5790640 1 -- 172.21.15.29:0/1861938984 shutdown_connections 2024-03-16T11:48:43.460 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:43.458+0000 7fbdf5790640 1 --2- 172.21.15.29:0/1861938984 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbdf0102290 0x7fbdf0108b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:43.460 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:43.458+0000 7fbdf5790640 1 --2- 172.21.15.29:0/1861938984 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbdf01018d0 0x7fbdf0101d50 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:43.460 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:43.458+0000 7fbdf5790640 1 --2- 172.21.15.29:0/1861938984 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbdf01006e0 0x7fbdf0100b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:43.460 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:43.458+0000 7fbdf5790640 1 -- 172.21.15.29:0/1861938984 >> 172.21.15.29:0/1861938984 conn(0x7fbdf00fbc70 msgr2=0x7fbdf00fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:48:43.460 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:43.458+0000 7fbdf5790640 1 -- 172.21.15.29:0/1861938984 shutdown_connections 2024-03-16T11:48:43.460 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:43.458+0000 7fbdf5790640 1 -- 172.21.15.29:0/1861938984 wait complete. 2024-03-16T11:48:43.460 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:43.459+0000 7fbdf5790640 1 Processor -- start 2024-03-16T11:48:43.460 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:43.459+0000 7fbdf5790640 1 -- start start 2024-03-16T11:48:43.460 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:43.459+0000 7fbdf5790640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbdf01006e0 0x7fbdf0197b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:48:43.460 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:43.459+0000 7fbdf5790640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbdf01018d0 0x7fbdf0198080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:48:43.461 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:43.459+0000 7fbdf5790640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbdf0102290 0x7fbdf019f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:48:43.461 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:43.459+0000 7fbdf5790640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fbdf010c340 con 0x7fbdf01006e0 2024-03-16T11:48:43.461 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:43.459+0000 7fbdf5790640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fbdf010c1c0 con 0x7fbdf0102290 2024-03-16T11:48:43.461 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:43.459+0000 7fbdf5790640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fbdf010c4c0 con 0x7fbdf01018d0 2024-03-16T11:48:43.461 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:43.459+0000 7fbdeeffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbdf01006e0 0x7fbdf0197b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:48:43.461 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:43.459+0000 7fbdeeffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbdf01006e0 0x7fbdf0197b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49154/0 (socket says 172.21.15.29:49154) 2024-03-16T11:48:43.461 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:43.459+0000 7fbdeeffd640 1 -- 172.21.15.29:0/3390303273 learned_addr learned my addr 172.21.15.29:0/3390303273 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:48:43.461 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:43.459+0000 7fbdef7fe640 1 --2- 172.21.15.29:0/3390303273 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbdf0102290 0x7fbdf019f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:48:43.461 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:43.459+0000 7fbde7fff640 1 --2- 172.21.15.29:0/3390303273 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbdf01018d0 0x7fbdf0198080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:48:43.461 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:43.459+0000 7fbdeeffd640 1 -- 172.21.15.29:0/3390303273 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbdf01018d0 msgr2=0x7fbdf0198080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:48:43.461 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:43.459+0000 7fbdeeffd640 1 --2- 172.21.15.29:0/3390303273 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbdf01018d0 0x7fbdf0198080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:43.461 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:43.459+0000 7fbdeeffd640 1 -- 172.21.15.29:0/3390303273 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbdf0102290 msgr2=0x7fbdf019f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:48:43.461 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:43.459+0000 7fbdeeffd640 1 --2- 172.21.15.29:0/3390303273 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbdf0102290 0x7fbdf019f0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:43.462 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:43.459+0000 7fbdeeffd640 1 -- 172.21.15.29:0/3390303273 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbdf019f760 con 0x7fbdf01006e0 2024-03-16T11:48:43.462 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:43.459+0000 7fbdeeffd640 1 --2- 172.21.15.29:0/3390303273 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbdf01006e0 0x7fbdf0197b40 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fbde000d770 tx=0x7fbde000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:48:43.462 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:43.460+0000 7fbdecff9640 1 -- 172.21.15.29:0/3390303273 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbde0017070 con 0x7fbdf01006e0 2024-03-16T11:48:43.462 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:43.460+0000 7fbdecff9640 1 -- 172.21.15.29:0/3390303273 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbde0009c30 con 0x7fbdf01006e0 2024-03-16T11:48:43.462 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:43.460+0000 7fbdf5790640 1 -- 172.21.15.29:0/3390303273 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbdf019fa50 con 0x7fbdf01006e0 2024-03-16T11:48:43.462 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:43.460+0000 7fbdecff9640 1 -- 172.21.15.29:0/3390303273 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbde0003dd0 con 0x7fbdf01006e0 2024-03-16T11:48:43.462 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:43.460+0000 7fbdf5790640 1 -- 172.21.15.29:0/3390303273 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbdf019ffc0 con 0x7fbdf01006e0 2024-03-16T11:48:43.462 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:43.461+0000 7fbdf5790640 1 -- 172.21.15.29:0/3390303273 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbdf01a01f0 con 0x7fbdf01006e0 2024-03-16T11:48:43.463 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:43.462+0000 7fbdecff9640 1 -- 172.21.15.29:0/3390303273 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fbde0003fb0 con 0x7fbdf01006e0 2024-03-16T11:48:43.463 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:43.462+0000 7fbdecff9640 1 --2- 172.21.15.29:0/3390303273 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fbddc0a2880 0x7fbddc0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:48:43.463 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:43.462+0000 7fbdecff9640 1 -- 172.21.15.29:0/3390303273 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7fbde00ce180 con 0x7fbdf01006e0 2024-03-16T11:48:43.463 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:43.462+0000 7fbde7fff640 1 --2- 172.21.15.29:0/3390303273 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fbddc0a2880 0x7fbddc0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:48:43.463 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:43.463+0000 7fbde7fff640 1 --2- 172.21.15.29:0/3390303273 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fbddc0a2880 0x7fbddc0a4d40 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fbdd802c7b0 tx=0x7fbdd8005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:48:43.466 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:43.467+0000 7fbdecff9640 1 -- 172.21.15.29:0/3390303273 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fbde0009de0 con 0x7fbdf01006e0 2024-03-16T11:48:43.588 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 16 11:48:43 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-x[32210]: 2024-03-16T11:48:43.316+0000 7f7e0371a700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-16T11:48:43.588 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 16 11:48:43 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-x[32210]: 2024-03-16T11:48:43.316+0000 7f7e0371a700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-16T11:48:43.588 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 16 11:48:43 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-x[32210]: 2024-03-16T11:48:43.316+0000 7f7e0371a700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-16T11:48:43.588 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 16 11:48:43 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-x[32210]: 2024-03-16T11:48:43.316+0000 7f7e0371a700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-16T11:48:43.588 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 16 11:48:43 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-x[32210]: 2024-03-16T11:48:43.316+0000 7f7e0371a700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-16T11:48:43.678 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:43.679+0000 7fbdf5790640 1 -- 172.21.15.29:0/3390303273 --> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7fbdf0069050 con 0x7fbddc0a2880 2024-03-16T11:48:43.704 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:43.705+0000 7fbdecff9640 1 -- 172.21.15.29:0/3390303273 <== mgr.14152 v2:172.21.15.29:6800/4192064640 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fbdf0069050 con 0x7fbddc0a2880 2024-03-16T11:48:43.706 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:43.707+0000 7fbdf5790640 1 -- 172.21.15.29:0/3390303273 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fbddc0a2880 msgr2=0x7fbddc0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:48:43.706 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:43.707+0000 7fbdf5790640 1 --2- 172.21.15.29:0/3390303273 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fbddc0a2880 0x7fbddc0a4d40 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fbdd802c7b0 tx=0x7fbdd8005fd0 comp rx=0 tx=0).stop 2024-03-16T11:48:43.706 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:43.707+0000 7fbdf5790640 1 -- 172.21.15.29:0/3390303273 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbdf01006e0 msgr2=0x7fbdf0197b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:48:43.707 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:43.707+0000 7fbdf5790640 1 --2- 172.21.15.29:0/3390303273 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbdf01006e0 0x7fbdf0197b40 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fbde000d770 tx=0x7fbde000dc30 comp rx=0 tx=0).stop 2024-03-16T11:48:43.707 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:43.707+0000 7fbdf5790640 1 -- 172.21.15.29:0/3390303273 shutdown_connections 2024-03-16T11:48:43.707 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:43.707+0000 7fbdf5790640 1 --2- 172.21.15.29:0/3390303273 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fbddc0a2880 0x7fbddc0a4d40 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:43.707 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:43.707+0000 7fbdf5790640 1 --2- 172.21.15.29:0/3390303273 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbdf0102290 0x7fbdf019f0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:43.707 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:43.707+0000 7fbdf5790640 1 --2- 172.21.15.29:0/3390303273 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbdf01018d0 0x7fbdf0198080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:43.707 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:43.707+0000 7fbdf5790640 1 --2- 172.21.15.29:0/3390303273 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbdf01006e0 0x7fbdf0197b40 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:43.707 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:43.707+0000 7fbdf5790640 1 -- 172.21.15.29:0/3390303273 >> 172.21.15.29:0/3390303273 conn(0x7fbdf00fbc70 msgr2=0x7fbdf0068940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:48:43.707 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:43.707+0000 7fbdf5790640 1 -- 172.21.15.29:0/3390303273 shutdown_connections 2024-03-16T11:48:43.707 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:43.707+0000 7fbdf5790640 1 -- 172.21.15.29:0/3390303273 wait complete. 2024-03-16T11:48:43.717 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_0 2024-03-16T11:48:43.826 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:43.825+0000 7f71bfdf4640 1 -- 172.21.15.29:0/2123151612 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f71b81018f0 msgr2=0x7f71b8101d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:48:43.826 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:43.825+0000 7f71bfdf4640 1 --2- 172.21.15.29:0/2123151612 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f71b81018f0 0x7f71b8101d70 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f71a8009900 tx=0x7f71a802bc30 comp rx=0 tx=0).stop 2024-03-16T11:48:43.826 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:43.825+0000 7f71bfdf4640 1 -- 172.21.15.29:0/2123151612 shutdown_connections 2024-03-16T11:48:43.826 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:43.825+0000 7f71bfdf4640 1 --2- 172.21.15.29:0/2123151612 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f71b81022b0 0x7f71b8108b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:43.826 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:43.825+0000 7f71bfdf4640 1 --2- 172.21.15.29:0/2123151612 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f71b81018f0 0x7f71b8101d70 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:43.826 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:43.825+0000 7f71bfdf4640 1 --2- 172.21.15.29:0/2123151612 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f71b8100700 0x7f71b8100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:43.826 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:43.825+0000 7f71bfdf4640 1 -- 172.21.15.29:0/2123151612 >> 172.21.15.29:0/2123151612 conn(0x7f71b80fbc90 msgr2=0x7f71b80fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:48:43.826 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:43.825+0000 7f71bfdf4640 1 -- 172.21.15.29:0/2123151612 shutdown_connections 2024-03-16T11:48:43.826 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:43.825+0000 7f71bfdf4640 1 -- 172.21.15.29:0/2123151612 wait complete. 2024-03-16T11:48:43.827 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:43.825+0000 7f71bfdf4640 1 Processor -- start 2024-03-16T11:48:43.827 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:43.825+0000 7f71bfdf4640 1 -- start start 2024-03-16T11:48:43.827 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:43.826+0000 7f71bfdf4640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f71b8100700 0x7f71b806a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:48:43.827 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:43.826+0000 7f71bfdf4640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f71b81018f0 0x7f71b819c890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:48:43.827 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:43.826+0000 7f71bfdf4640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f71b81022b0 0x7f71b819eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:48:43.827 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:43.826+0000 7f71bfdf4640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f71b810c380 con 0x7f71b81018f0 2024-03-16T11:48:43.827 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:43.826+0000 7f71bfdf4640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f71b810c200 con 0x7f71b81022b0 2024-03-16T11:48:43.827 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:43.826+0000 7f71bfdf4640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f71b810c500 con 0x7f71b8100700 2024-03-16T11:48:43.827 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:43.826+0000 7f71bd38f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f71b81018f0 0x7f71b819c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:48:43.827 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:43.826+0000 7f71bd38f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f71b81018f0 0x7f71b819c890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49170/0 (socket says 172.21.15.29:49170) 2024-03-16T11:48:43.827 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:43.826+0000 7f71bd38f640 1 -- 172.21.15.29:0/1116872625 learned_addr learned my addr 172.21.15.29:0/1116872625 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:48:43.827 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:43.826+0000 7f71bdb90640 1 --2- 172.21.15.29:0/1116872625 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f71b8100700 0x7f71b806a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:48:43.827 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:43.826+0000 7f71be391640 1 --2- 172.21.15.29:0/1116872625 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f71b81022b0 0x7f71b819eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:48:43.827 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:43.826+0000 7f71bd38f640 1 -- 172.21.15.29:0/1116872625 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f71b8100700 msgr2=0x7f71b806a940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:48:43.827 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:43.826+0000 7f71bd38f640 1 --2- 172.21.15.29:0/1116872625 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f71b8100700 0x7f71b806a940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:43.828 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:43.826+0000 7f71bd38f640 1 -- 172.21.15.29:0/1116872625 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f71b81022b0 msgr2=0x7f71b819eed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:48:43.828 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:43.826+0000 7f71bd38f640 1 --2- 172.21.15.29:0/1116872625 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f71b81022b0 0x7f71b819eed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:43.828 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:43.826+0000 7f71bd38f640 1 -- 172.21.15.29:0/1116872625 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71b819f790 con 0x7f71b81018f0 2024-03-16T11:48:43.828 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:43.826+0000 7f71bd38f640 1 --2- 172.21.15.29:0/1116872625 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f71b81018f0 0x7f71b819c890 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f71a8009230 tx=0x7f71a8005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:48:43.828 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:43.826+0000 7f71aeffd640 1 -- 172.21.15.29:0/1116872625 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71a8041070 con 0x7f71b81018f0 2024-03-16T11:48:43.828 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:43.826+0000 7f71aeffd640 1 -- 172.21.15.29:0/1116872625 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f71a80044d0 con 0x7f71b81018f0 2024-03-16T11:48:43.828 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:43.826+0000 7f71bfdf4640 1 -- 172.21.15.29:0/1116872625 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f71b819fa20 con 0x7f71b81018f0 2024-03-16T11:48:43.828 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:43.826+0000 7f71aeffd640 1 -- 172.21.15.29:0/1116872625 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71a8034440 con 0x7f71b81018f0 2024-03-16T11:48:43.828 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:43.826+0000 7f71bfdf4640 1 -- 172.21.15.29:0/1116872625 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f71b819ff60 con 0x7f71b81018f0 2024-03-16T11:48:43.829 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:43.827+0000 7f71bfdf4640 1 -- 172.21.15.29:0/1116872625 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f71b8107a40 con 0x7f71b81018f0 2024-03-16T11:48:43.829 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:43.828+0000 7f71aeffd640 1 -- 172.21.15.29:0/1116872625 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f71a8044070 con 0x7f71b81018f0 2024-03-16T11:48:43.829 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:43.829+0000 7f71aeffd640 1 --2- 172.21.15.29:0/1116872625 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f71a40a2880 0x7f71a40a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:48:43.829 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:43.829+0000 7f71bdb90640 1 --2- 172.21.15.29:0/1116872625 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f71a40a2880 0x7f71a40a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:48:43.829 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:43.829+0000 7f71aeffd640 1 -- 172.21.15.29:0/1116872625 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f71a80f0490 con 0x7f71b81018f0 2024-03-16T11:48:43.829 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:43.829+0000 7f71bdb90640 1 --2- 172.21.15.29:0/1116872625 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f71a40a2880 0x7f71a40a4d40 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f71b8101750 tx=0x7f71b4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:48:43.832 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:43.833+0000 7f71aeffd640 1 -- 172.21.15.29:0/1116872625 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f71a80bd030 con 0x7f71b81018f0 2024-03-16T11:48:44.047 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:44.048+0000 7f71bfdf4640 1 -- 172.21.15.29:0/1116872625 --> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}) -- 0x7f71b8061cb0 con 0x7f71a40a2880 2024-03-16T11:48:44.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:44.052+0000 7f71aeffd640 1 -- 172.21.15.29:0/1116872625 <== mgr.14152 v2:172.21.15.29:6800/4192064640 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7f71b8061cb0 con 0x7f71a40a2880 2024-03-16T11:48:44.051 INFO:teuthology.orchestra.run.smithi029.stdout:/volumes/qa/sv_0/44e28983-19cb-4b50-8274-b6210680e300 2024-03-16T11:48:44.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:44.054+0000 7f71bfdf4640 1 -- 172.21.15.29:0/1116872625 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f71a40a2880 msgr2=0x7f71a40a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:48:44.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:44.054+0000 7f71bfdf4640 1 --2- 172.21.15.29:0/1116872625 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f71a40a2880 0x7f71a40a4d40 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f71b8101750 tx=0x7f71b4005f90 comp rx=0 tx=0).stop 2024-03-16T11:48:44.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:44.054+0000 7f71bfdf4640 1 -- 172.21.15.29:0/1116872625 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f71b81018f0 msgr2=0x7f71b819c890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:48:44.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:44.054+0000 7f71bfdf4640 1 --2- 172.21.15.29:0/1116872625 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f71b81018f0 0x7f71b819c890 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f71a8009230 tx=0x7f71a8005d50 comp rx=0 tx=0).stop 2024-03-16T11:48:44.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:44.054+0000 7f71bfdf4640 1 -- 172.21.15.29:0/1116872625 shutdown_connections 2024-03-16T11:48:44.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:44.054+0000 7f71bfdf4640 1 --2- 172.21.15.29:0/1116872625 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f71a40a2880 0x7f71a40a4d40 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:44.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:44.054+0000 7f71bfdf4640 1 --2- 172.21.15.29:0/1116872625 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f71b81022b0 0x7f71b819eed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:44.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:44.054+0000 7f71bfdf4640 1 --2- 172.21.15.29:0/1116872625 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f71b81018f0 0x7f71b819c890 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:44.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:44.054+0000 7f71bfdf4640 1 --2- 172.21.15.29:0/1116872625 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f71b8100700 0x7f71b806a940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:44.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:44.054+0000 7f71bfdf4640 1 -- 172.21.15.29:0/1116872625 >> 172.21.15.29:0/1116872625 conn(0x7f71b80fbc90 msgr2=0x7f71b8104c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:48:44.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:44.054+0000 7f71bfdf4640 1 -- 172.21.15.29:0/1116872625 shutdown_connections 2024-03-16T11:48:44.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:44.054+0000 7f71bfdf4640 1 -- 172.21.15.29:0/1116872625 wait complete. 2024-03-16T11:48:44.064 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_1 --group_name qa 2024-03-16T11:48:44.173 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:44.172+0000 7efcea953640 1 -- 172.21.15.29:0/1342587075 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efce41018f0 msgr2=0x7efce4101d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:48:44.174 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:44.172+0000 7efcea953640 1 --2- 172.21.15.29:0/1342587075 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efce41018f0 0x7efce4101d70 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7efcd4009900 tx=0x7efcd402bc30 comp rx=0 tx=0).stop 2024-03-16T11:48:44.174 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:44.172+0000 7efcea953640 1 -- 172.21.15.29:0/1342587075 shutdown_connections 2024-03-16T11:48:44.174 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:44.172+0000 7efcea953640 1 --2- 172.21.15.29:0/1342587075 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7efce41022b0 0x7efce4108b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:44.174 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:44.172+0000 7efcea953640 1 --2- 172.21.15.29:0/1342587075 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efce41018f0 0x7efce4101d70 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:44.174 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:44.172+0000 7efcea953640 1 --2- 172.21.15.29:0/1342587075 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efce4100700 0x7efce4100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:44.174 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:44.172+0000 7efcea953640 1 -- 172.21.15.29:0/1342587075 >> 172.21.15.29:0/1342587075 conn(0x7efce40fbc90 msgr2=0x7efce40fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:48:44.174 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:44.172+0000 7efcea953640 1 -- 172.21.15.29:0/1342587075 shutdown_connections 2024-03-16T11:48:44.174 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:44.172+0000 7efcea953640 1 -- 172.21.15.29:0/1342587075 wait complete. 2024-03-16T11:48:44.174 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:44.173+0000 7efcea953640 1 Processor -- start 2024-03-16T11:48:44.174 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:44.173+0000 7efcea953640 1 -- start start 2024-03-16T11:48:44.174 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:44.173+0000 7efcea953640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efce4100700 0x7efce406a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:48:44.174 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:44.173+0000 7efcea953640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7efce41018f0 0x7efce419c860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:48:44.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:44.173+0000 7efcea953640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efce41022b0 0x7efce419eea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:48:44.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:44.173+0000 7efcea953640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7efce410c360 con 0x7efce41022b0 2024-03-16T11:48:44.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:44.173+0000 7efcea953640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7efce410c1e0 con 0x7efce41018f0 2024-03-16T11:48:44.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:44.173+0000 7efcea953640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7efce410c4e0 con 0x7efce4100700 2024-03-16T11:48:44.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:44.173+0000 7efce8ef0640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efce41022b0 0x7efce419eea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:48:44.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:44.173+0000 7efce8ef0640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efce41022b0 0x7efce419eea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49180/0 (socket says 172.21.15.29:49180) 2024-03-16T11:48:44.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:44.173+0000 7efce8ef0640 1 -- 172.21.15.29:0/99175973 learned_addr learned my addr 172.21.15.29:0/99175973 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:48:44.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:44.173+0000 7efce3fff640 1 --2- 172.21.15.29:0/99175973 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efce4100700 0x7efce406a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:48:44.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:44.173+0000 7efce37fe640 1 --2- 172.21.15.29:0/99175973 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7efce41018f0 0x7efce419c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:48:44.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:44.173+0000 7efce8ef0640 1 -- 172.21.15.29:0/99175973 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efce4100700 msgr2=0x7efce406a940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:48:44.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:44.173+0000 7efce8ef0640 1 --2- 172.21.15.29:0/99175973 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efce4100700 0x7efce406a940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:44.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:44.173+0000 7efce8ef0640 1 -- 172.21.15.29:0/99175973 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7efce41018f0 msgr2=0x7efce419c860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:48:44.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:44.173+0000 7efce8ef0640 1 --2- 172.21.15.29:0/99175973 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7efce41018f0 0x7efce419c860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:44.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:44.173+0000 7efce8ef0640 1 -- 172.21.15.29:0/99175973 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efce419f760 con 0x7efce41022b0 2024-03-16T11:48:44.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:44.174+0000 7efce8ef0640 1 --2- 172.21.15.29:0/99175973 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efce41022b0 0x7efce419eea0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7efcdc00eb10 tx=0x7efcdc00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:48:44.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:44.174+0000 7efce17fa640 1 -- 172.21.15.29:0/99175973 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcdc004b20 con 0x7efce41022b0 2024-03-16T11:48:44.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:44.174+0000 7efce17fa640 1 -- 172.21.15.29:0/99175973 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efcdc004cc0 con 0x7efce41022b0 2024-03-16T11:48:44.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:44.174+0000 7efcea953640 1 -- 172.21.15.29:0/99175973 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efce419fa50 con 0x7efce41022b0 2024-03-16T11:48:44.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:44.174+0000 7efce17fa640 1 -- 172.21.15.29:0/99175973 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcdc010760 con 0x7efce41022b0 2024-03-16T11:48:44.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:44.174+0000 7efcea953640 1 -- 172.21.15.29:0/99175973 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efce419ff20 con 0x7efce41022b0 2024-03-16T11:48:44.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:44.175+0000 7efcea953640 1 -- 172.21.15.29:0/99175973 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efce4107a70 con 0x7efce41022b0 2024-03-16T11:48:44.177 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:44.176+0000 7efce17fa640 1 -- 172.21.15.29:0/99175973 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7efcdc005000 con 0x7efce41022b0 2024-03-16T11:48:44.177 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:44.176+0000 7efce17fa640 1 --2- 172.21.15.29:0/99175973 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7efccc0a2800 0x7efccc0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:48:44.177 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:44.176+0000 7efce17fa640 1 -- 172.21.15.29:0/99175973 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7efcdc0cf0b0 con 0x7efce41022b0 2024-03-16T11:48:44.177 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:44.176+0000 7efce3fff640 1 --2- 172.21.15.29:0/99175973 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7efccc0a2800 0x7efccc0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:48:44.177 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:44.177+0000 7efce3fff640 1 --2- 172.21.15.29:0/99175973 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7efccc0a2800 0x7efccc0a4cc0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7efce4101750 tx=0x7efcd800b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:48:44.179 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:44.180+0000 7efce17fa640 1 -- 172.21.15.29:0/99175973 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7efcdc053740 con 0x7efce41022b0 2024-03-16T11:48:44.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:44 smithi154 ceph-mon[36320]: from='client.14754 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T11:48:44.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:44 smithi154 ceph-mon[36320]: pgmap v293: 97 pgs: 97 active+clean; 579 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 682 B/s rd, 4.0 KiB/s wr, 13 op/s 2024-03-16T11:48:44.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:44 smithi029 ceph-mon[31975]: from='client.14754 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T11:48:44.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:44 smithi029 ceph-mon[31975]: pgmap v293: 97 pgs: 97 active+clean; 579 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 682 B/s rd, 4.0 KiB/s wr, 13 op/s 2024-03-16T11:48:44.378 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:44 smithi152 ceph-mon[36291]: from='client.14754 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T11:48:44.378 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:44 smithi152 ceph-mon[36291]: pgmap v293: 97 pgs: 97 active+clean; 579 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 682 B/s rd, 4.0 KiB/s wr, 13 op/s 2024-03-16T11:48:44.378 INFO:journalctl@ceph.mds.b.smithi152.stdout:Mar 16 11:48:44 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mds-b[67242]: 2024-03-16T11:48:43.998+0000 7f848d242700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-16T11:48:44.378 INFO:journalctl@ceph.mds.b.smithi152.stdout:Mar 16 11:48:44 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mds-b[67242]: 2024-03-16T11:48:43.998+0000 7f848d242700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-03-16T11:48:44.378 INFO:journalctl@ceph.mds.b.smithi152.stdout:Mar 16 11:48:44 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mds-b[67242]: 2024-03-16T11:48:43.998+0000 7f848d242700 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2024-03-16T11:48:44.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:44.393+0000 7efcea953640 1 -- 172.21.15.29:0/99175973 --> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7efce4061c20 con 0x7efccc0a2800 2024-03-16T11:48:44.414 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:44.415+0000 7efce17fa640 1 -- 172.21.15.29:0/99175973 <== mgr.14152 v2:172.21.15.29:6800/4192064640 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7efce4061c20 con 0x7efccc0a2800 2024-03-16T11:48:44.417 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:44.418+0000 7efcea953640 1 -- 172.21.15.29:0/99175973 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7efccc0a2800 msgr2=0x7efccc0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:48:44.417 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:44.418+0000 7efcea953640 1 --2- 172.21.15.29:0/99175973 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7efccc0a2800 0x7efccc0a4cc0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7efce4101750 tx=0x7efcd800b390 comp rx=0 tx=0).stop 2024-03-16T11:48:44.417 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:44.418+0000 7efcea953640 1 -- 172.21.15.29:0/99175973 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efce41022b0 msgr2=0x7efce419eea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:48:44.417 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:44.418+0000 7efcea953640 1 --2- 172.21.15.29:0/99175973 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efce41022b0 0x7efce419eea0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7efcdc00eb10 tx=0x7efcdc00c2d0 comp rx=0 tx=0).stop 2024-03-16T11:48:44.417 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:44.418+0000 7efcea953640 1 -- 172.21.15.29:0/99175973 shutdown_connections 2024-03-16T11:48:44.417 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:44.418+0000 7efcea953640 1 --2- 172.21.15.29:0/99175973 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7efccc0a2800 0x7efccc0a4cc0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:44.418 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:44.418+0000 7efcea953640 1 --2- 172.21.15.29:0/99175973 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efce41022b0 0x7efce419eea0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:44.418 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:44.418+0000 7efcea953640 1 --2- 172.21.15.29:0/99175973 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7efce41018f0 0x7efce419c860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:44.418 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:44.418+0000 7efcea953640 1 --2- 172.21.15.29:0/99175973 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efce4100700 0x7efce406a940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:44.418 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:44.418+0000 7efcea953640 1 -- 172.21.15.29:0/99175973 >> 172.21.15.29:0/99175973 conn(0x7efce40fbc90 msgr2=0x7efce4104c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:48:44.418 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:44.418+0000 7efcea953640 1 -- 172.21.15.29:0/99175973 shutdown_connections 2024-03-16T11:48:44.418 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:44.418+0000 7efcea953640 1 -- 172.21.15.29:0/99175973 wait complete. 2024-03-16T11:48:44.428 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_1 2024-03-16T11:48:44.538 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:44.537+0000 7f3b8f8b8640 1 -- 172.21.15.29:0/967889315 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3b881018d0 msgr2=0x7f3b88101d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:48:44.538 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:44.537+0000 7f3b8f8b8640 1 --2- 172.21.15.29:0/967889315 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3b881018d0 0x7f3b88101d50 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f3b78009900 tx=0x7f3b7802bc30 comp rx=0 tx=0).stop 2024-03-16T11:48:44.538 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:44.537+0000 7f3b8f8b8640 1 -- 172.21.15.29:0/967889315 shutdown_connections 2024-03-16T11:48:44.538 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:44.537+0000 7f3b8f8b8640 1 --2- 172.21.15.29:0/967889315 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3b88102290 0x7f3b88108b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:44.538 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:44.537+0000 7f3b8f8b8640 1 --2- 172.21.15.29:0/967889315 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3b881018d0 0x7f3b88101d50 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:44.538 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:44.537+0000 7f3b8f8b8640 1 --2- 172.21.15.29:0/967889315 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3b881006e0 0x7f3b88100b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:44.538 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:44.537+0000 7f3b8f8b8640 1 -- 172.21.15.29:0/967889315 >> 172.21.15.29:0/967889315 conn(0x7f3b880fbc70 msgr2=0x7f3b880fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:48:44.538 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:44.537+0000 7f3b8f8b8640 1 -- 172.21.15.29:0/967889315 shutdown_connections 2024-03-16T11:48:44.538 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:44.537+0000 7f3b8f8b8640 1 -- 172.21.15.29:0/967889315 wait complete. 2024-03-16T11:48:44.539 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:44.537+0000 7f3b8f8b8640 1 Processor -- start 2024-03-16T11:48:44.539 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:44.537+0000 7f3b8f8b8640 1 -- start start 2024-03-16T11:48:44.539 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:44.537+0000 7f3b8f8b8640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3b881006e0 0x7f3b880694d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:48:44.539 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:44.537+0000 7f3b8f8b8640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3b881018d0 0x7f3b8819c890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:48:44.539 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:44.537+0000 7f3b8f8b8640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3b88102290 0x7f3b8819eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:48:44.539 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:44.537+0000 7f3b8f8b8640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f3b8810c340 con 0x7f3b881018d0 2024-03-16T11:48:44.539 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:44.537+0000 7f3b8f8b8640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f3b8810c1c0 con 0x7f3b881006e0 2024-03-16T11:48:44.539 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:44.537+0000 7f3b8f8b8640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f3b8810c4c0 con 0x7f3b88102290 2024-03-16T11:48:44.539 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:44.537+0000 7f3b8ce53640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3b881018d0 0x7f3b8819c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:48:44.539 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:44.537+0000 7f3b8ce53640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3b881018d0 0x7f3b8819c890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49190/0 (socket says 172.21.15.29:49190) 2024-03-16T11:48:44.539 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:44.537+0000 7f3b8ce53640 1 -- 172.21.15.29:0/3322871799 learned_addr learned my addr 172.21.15.29:0/3322871799 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:48:44.539 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:44.538+0000 7f3b8d654640 1 --2- 172.21.15.29:0/3322871799 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3b881006e0 0x7f3b880694d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:48:44.539 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:44.538+0000 7f3b8de55640 1 --2- 172.21.15.29:0/3322871799 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3b88102290 0x7f3b8819eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:48:44.539 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:44.538+0000 7f3b8ce53640 1 -- 172.21.15.29:0/3322871799 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3b88102290 msgr2=0x7f3b8819eed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:48:44.540 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:44.538+0000 7f3b8ce53640 1 --2- 172.21.15.29:0/3322871799 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3b88102290 0x7f3b8819eed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:44.540 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:44.538+0000 7f3b8ce53640 1 -- 172.21.15.29:0/3322871799 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3b881006e0 msgr2=0x7f3b880694d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:48:44.540 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:44.538+0000 7f3b8ce53640 1 --2- 172.21.15.29:0/3322871799 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3b881006e0 0x7f3b880694d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:44.540 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:44.538+0000 7f3b8ce53640 1 -- 172.21.15.29:0/3322871799 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b8819f790 con 0x7f3b881018d0 2024-03-16T11:48:44.540 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:44.538+0000 7f3b8ce53640 1 --2- 172.21.15.29:0/3322871799 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3b881018d0 0x7f3b8819c890 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f3b78009230 tx=0x7f3b7802c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:48:44.540 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:44.538+0000 7f3b7e7fc640 1 -- 172.21.15.29:0/3322871799 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b78041070 con 0x7f3b881018d0 2024-03-16T11:48:44.540 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:44.538+0000 7f3b8f8b8640 1 -- 172.21.15.29:0/3322871799 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3b8819fa20 con 0x7f3b881018d0 2024-03-16T11:48:44.540 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:44.538+0000 7f3b7e7fc640 1 -- 172.21.15.29:0/3322871799 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3b78004450 con 0x7f3b881018d0 2024-03-16T11:48:44.540 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:44.538+0000 7f3b7e7fc640 1 -- 172.21.15.29:0/3322871799 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b78003db0 con 0x7f3b881018d0 2024-03-16T11:48:44.540 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:44.538+0000 7f3b8f8b8640 1 -- 172.21.15.29:0/3322871799 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3b8819ff10 con 0x7f3b881018d0 2024-03-16T11:48:44.541 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:44.539+0000 7f3b8f8b8640 1 -- 172.21.15.29:0/3322871799 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3b88107b20 con 0x7f3b881018d0 2024-03-16T11:48:44.541 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:44.540+0000 7f3b7e7fc640 1 -- 172.21.15.29:0/3322871799 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f3b78044070 con 0x7f3b881018d0 2024-03-16T11:48:44.541 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:44.541+0000 7f3b7e7fc640 1 --2- 172.21.15.29:0/3322871799 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f3b740a2800 0x7f3b740a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:48:44.541 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:44.541+0000 7f3b8d654640 1 --2- 172.21.15.29:0/3322871799 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f3b740a2800 0x7f3b740a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:48:44.541 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:44.541+0000 7f3b7e7fc640 1 -- 172.21.15.29:0/3322871799 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f3b780f04c0 con 0x7f3b881018d0 2024-03-16T11:48:44.541 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:44.541+0000 7f3b8d654640 1 --2- 172.21.15.29:0/3322871799 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f3b740a2800 0x7f3b740a4cc0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f3b88101730 tx=0x7f3b84005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:48:44.544 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:44.545+0000 7f3b7e7fc640 1 -- 172.21.15.29:0/3322871799 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f3b780bd6b0 con 0x7f3b881018d0 2024-03-16T11:48:44.756 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:44.757+0000 7f3b8f8b8640 1 -- 172.21.15.29:0/3322871799 --> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}) -- 0x7f3b88061cb0 con 0x7f3b740a2800 2024-03-16T11:48:44.760 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:44.761+0000 7f3b7e7fc640 1 -- 172.21.15.29:0/3322871799 <== mgr.14152 v2:172.21.15.29:6800/4192064640 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7f3b88061cb0 con 0x7f3b740a2800 2024-03-16T11:48:44.760 INFO:teuthology.orchestra.run.smithi029.stdout:/volumes/qa/sv_1/376ddc2b-fccb-40c2-9473-a26f8d0685a7 2024-03-16T11:48:44.763 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:44.763+0000 7f3b8f8b8640 1 -- 172.21.15.29:0/3322871799 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f3b740a2800 msgr2=0x7f3b740a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:48:44.763 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:44.763+0000 7f3b8f8b8640 1 --2- 172.21.15.29:0/3322871799 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f3b740a2800 0x7f3b740a4cc0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f3b88101730 tx=0x7f3b84005f90 comp rx=0 tx=0).stop 2024-03-16T11:48:44.763 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:44.763+0000 7f3b8f8b8640 1 -- 172.21.15.29:0/3322871799 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3b881018d0 msgr2=0x7f3b8819c890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:48:44.763 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:44.763+0000 7f3b8f8b8640 1 --2- 172.21.15.29:0/3322871799 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3b881018d0 0x7f3b8819c890 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f3b78009230 tx=0x7f3b7802c650 comp rx=0 tx=0).stop 2024-03-16T11:48:44.763 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:44.763+0000 7f3b8f8b8640 1 -- 172.21.15.29:0/3322871799 shutdown_connections 2024-03-16T11:48:44.763 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:44.763+0000 7f3b8f8b8640 1 --2- 172.21.15.29:0/3322871799 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f3b740a2800 0x7f3b740a4cc0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:44.763 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:44.763+0000 7f3b8f8b8640 1 --2- 172.21.15.29:0/3322871799 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3b88102290 0x7f3b8819eed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:44.763 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:44.763+0000 7f3b8f8b8640 1 --2- 172.21.15.29:0/3322871799 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3b881018d0 0x7f3b8819c890 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:44.763 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:44.763+0000 7f3b8f8b8640 1 --2- 172.21.15.29:0/3322871799 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3b881006e0 0x7f3b880694d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:44.763 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:44.763+0000 7f3b8f8b8640 1 -- 172.21.15.29:0/3322871799 >> 172.21.15.29:0/3322871799 conn(0x7f3b880fbc70 msgr2=0x7f3b88104cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:48:44.763 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:44.763+0000 7f3b8f8b8640 1 -- 172.21.15.29:0/3322871799 shutdown_connections 2024-03-16T11:48:44.763 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:44.763+0000 7f3b8f8b8640 1 -- 172.21.15.29:0/3322871799 wait complete. 2024-03-16T11:48:44.773 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T11:48:44.877 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:44.876+0000 7f4dbc3b3640 1 -- 172.21.15.29:0/1195711078 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4db41017f0 msgr2=0x7f4db4101c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:48:44.878 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:44.876+0000 7f4dbc3b3640 1 --2- 172.21.15.29:0/1195711078 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4db41017f0 0x7f4db4101c10 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f4dac009900 tx=0x7f4dac02bc30 comp rx=0 tx=0).stop 2024-03-16T11:48:44.878 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:44.876+0000 7f4dbc3b3640 1 -- 172.21.15.29:0/1195711078 shutdown_connections 2024-03-16T11:48:44.878 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:44.876+0000 7f4dbc3b3640 1 --2- 172.21.15.29:0/1195711078 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4db41033a0 0x7f4db4109c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:44.878 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:44.876+0000 7f4dbc3b3640 1 --2- 172.21.15.29:0/1195711078 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4db41029e0 0x7f4db4102e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:44.878 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:44.876+0000 7f4dbc3b3640 1 --2- 172.21.15.29:0/1195711078 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4db41017f0 0x7f4db4101c10 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:44.878 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:44.876+0000 7f4dbc3b3640 1 -- 172.21.15.29:0/1195711078 >> 172.21.15.29:0/1195711078 conn(0x7f4db40fcdc0 msgr2=0x7f4db40ff1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:48:44.878 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:44.876+0000 7f4dbc3b3640 1 -- 172.21.15.29:0/1195711078 shutdown_connections 2024-03-16T11:48:44.878 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:44.876+0000 7f4dbc3b3640 1 -- 172.21.15.29:0/1195711078 wait complete. 2024-03-16T11:48:44.878 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:44.877+0000 7f4dbc3b3640 1 Processor -- start 2024-03-16T11:48:44.878 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:44.877+0000 7f4dbc3b3640 1 -- start start 2024-03-16T11:48:44.878 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:44.877+0000 7f4dbc3b3640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4db41017f0 0x7f4db4075590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:48:44.878 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:44.877+0000 7f4dbc3b3640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4db41029e0 0x7f4db4073bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:48:44.879 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:44.877+0000 7f4dbc3b3640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4db41033a0 0x7f4db4074110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:48:44.879 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:44.877+0000 7f4dbc3b3640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f4db410d470 con 0x7f4db41017f0 2024-03-16T11:48:44.879 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:44.877+0000 7f4dbc3b3640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f4db410d2f0 con 0x7f4db41033a0 2024-03-16T11:48:44.879 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:44.877+0000 7f4dbc3b3640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f4db410d5f0 con 0x7f4db41029e0 2024-03-16T11:48:44.879 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:44.877+0000 7f4dba14f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4db41017f0 0x7f4db4075590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:48:44.879 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:44.877+0000 7f4dba14f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4db41017f0 0x7f4db4075590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49220/0 (socket says 172.21.15.29:49220) 2024-03-16T11:48:44.879 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:44.877+0000 7f4dba14f640 1 -- 172.21.15.29:0/3936659366 learned_addr learned my addr 172.21.15.29:0/3936659366 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:48:44.879 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:44.877+0000 7f4db994e640 1 --2- 172.21.15.29:0/3936659366 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4db41029e0 0x7f4db4073bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:48:44.879 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:44.877+0000 7f4dba950640 1 --2- 172.21.15.29:0/3936659366 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4db41033a0 0x7f4db4074110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:48:44.879 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:44.877+0000 7f4dba14f640 1 -- 172.21.15.29:0/3936659366 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4db41029e0 msgr2=0x7f4db4073bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:48:44.879 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:44.877+0000 7f4dba14f640 1 --2- 172.21.15.29:0/3936659366 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4db41029e0 0x7f4db4073bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:44.879 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:44.877+0000 7f4dba14f640 1 -- 172.21.15.29:0/3936659366 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4db41033a0 msgr2=0x7f4db4074110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:48:44.879 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:44.877+0000 7f4dba14f640 1 --2- 172.21.15.29:0/3936659366 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4db41033a0 0x7f4db4074110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:44.879 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:44.877+0000 7f4dba14f640 1 -- 172.21.15.29:0/3936659366 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4db40749a0 con 0x7f4db41017f0 2024-03-16T11:48:44.879 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:44.877+0000 7f4dba950640 1 --2- 172.21.15.29:0/3936659366 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4db41033a0 0x7f4db4074110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T11:48:44.880 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:44.878+0000 7f4dba14f640 1 --2- 172.21.15.29:0/3936659366 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4db41017f0 0x7f4db4075590 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f4dac000c00 tx=0x7f4dac02c640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:48:44.880 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:44.878+0000 7f4db37fe640 1 -- 172.21.15.29:0/3936659366 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4dac041070 con 0x7f4db41017f0 2024-03-16T11:48:44.880 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:44.878+0000 7f4dbc3b3640 1 -- 172.21.15.29:0/3936659366 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4db41a1bd0 con 0x7f4db41017f0 2024-03-16T11:48:44.880 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:44.878+0000 7f4db37fe640 1 -- 172.21.15.29:0/3936659366 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4dac004450 con 0x7f4db41017f0 2024-03-16T11:48:44.880 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:44.878+0000 7f4db37fe640 1 -- 172.21.15.29:0/3936659366 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4dac034440 con 0x7f4db41017f0 2024-03-16T11:48:44.880 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:44.878+0000 7f4dbc3b3640 1 -- 172.21.15.29:0/3936659366 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4db41a2170 con 0x7f4db41017f0 2024-03-16T11:48:44.881 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:44.879+0000 7f4dbc3b3640 1 -- 172.21.15.29:0/3936659366 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4db4108b10 con 0x7f4db41017f0 2024-03-16T11:48:44.881 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:44.880+0000 7f4db37fe640 1 -- 172.21.15.29:0/3936659366 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f4dac044070 con 0x7f4db41017f0 2024-03-16T11:48:44.881 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:44.880+0000 7f4db37fe640 1 --2- 172.21.15.29:0/3936659366 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f4d9c0a2800 0x7f4d9c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:48:44.881 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:44.881+0000 7f4db994e640 1 --2- 172.21.15.29:0/3936659366 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f4d9c0a2800 0x7f4d9c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:48:44.881 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:44.881+0000 7f4db37fe640 1 -- 172.21.15.29:0/3936659366 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f4dac0f0470 con 0x7f4db41017f0 2024-03-16T11:48:44.881 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:44.881+0000 7f4db994e640 1 --2- 172.21.15.29:0/3936659366 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f4d9c0a2800 0x7f4d9c0a4cc0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f4db4075300 tx=0x7f4da8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:48:44.884 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:44.885+0000 7f4db37fe640 1 -- 172.21.15.29:0/3936659366 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f4dac0bd6f0 con 0x7f4db41017f0 2024-03-16T11:48:45.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:45.055+0000 7f4db37fe640 1 -- 172.21.15.29:0/3936659366 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f4dac0b9660 con 0x7f4db41017f0 2024-03-16T11:48:45.111 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:45.112+0000 7f4dbc3b3640 1 -- 172.21.15.29:0/3936659366 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4db402d1b0 con 0x7f4db41017f0 2024-03-16T11:48:45.111 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:45.112+0000 7f4db37fe640 1 -- 172.21.15.29:0/3936659366 <== mon.0 v2:172.21.15.29:3300/0 8 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+11875 (secure 0 0 0) 0x7f4dac0ad930 con 0x7f4db41017f0 2024-03-16T11:48:45.112 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-16T11:48:45.112 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14574,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/3505811955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":3505811955},{"type":"v1","addr":"172.21.15.29:6835","nonce":3505811955}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14586,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/1901337868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":1901337868},{"type":"v1","addr":"172.21.15.29:6837","nonce":1901337868}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14652,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6839/544966534","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6838","nonce":544966534},{"type":"v1","addr":"172.21.15.29:6839","nonce":544966534}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14664,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6841/540705290","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6840","nonce":540705290},{"type":"v1","addr":"172.21.15.29:6841","nonce":540705290}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24407,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6833/3136024628","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6832","nonce":3136024628},{"type":"v1","addr":"172.21.15.152:6833","nonce":3136024628}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24421,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6833/2677426282","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":2677426282},{"type":"v1","addr":"172.21.15.154:6833","nonce":2677426282}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":21,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-16T11:48:38.431509+0000","modified":"2024-03-16T11:48:42.969084+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24503,"mds_1":24499,"mds_2":24464},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24434":{"gid":24434,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.152:6835/2919113178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6834","nonce":2919113178},{"type":"v1","addr":"172.21.15.152:6835","nonce":2919113178}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24464":{"gid":24464,"name":"h","rank":2,"incarnation":19,"state":"up:active","state_seq":7,"addr":"172.21.15.152:6837/1323866722","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6836","nonce":1323866722},{"type":"v1","addr":"172.21.15.152:6837","nonce":1323866722}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.154:6835/4165274681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":4165274681},{"type":"v1","addr":"172.21.15.154:6835","nonce":4165274681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.154:6837/1468560096","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":1468560096},{"type":"v1","addr":"172.21.15.154:6837","nonce":1468560096}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":1,"incarnation":17,"state":"up:active","state_seq":6,"addr":"172.21.15.154:6839/685736862","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":685736862},{"type":"v1","addr":"172.21.15.154:6839","nonce":685736862}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.152:6839/1548731833","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6838","nonce":1548731833},{"type":"v1","addr":"172.21.15.152:6839","nonce":1548731833}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24503,"qdb_cluster":[24503,24464,24499]},"id":1}]} 2024-03-16T11:48:45.112 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 21 2024-03-16T11:48:45.114 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:45.114+0000 7f4dbc3b3640 1 -- 172.21.15.29:0/3936659366 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f4d9c0a2800 msgr2=0x7f4d9c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:48:45.114 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:45.114+0000 7f4dbc3b3640 1 --2- 172.21.15.29:0/3936659366 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f4d9c0a2800 0x7f4d9c0a4cc0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f4db4075300 tx=0x7f4da8005f90 comp rx=0 tx=0).stop 2024-03-16T11:48:45.114 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:45.114+0000 7f4dbc3b3640 1 -- 172.21.15.29:0/3936659366 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4db41017f0 msgr2=0x7f4db4075590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:48:45.114 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:45.114+0000 7f4dbc3b3640 1 --2- 172.21.15.29:0/3936659366 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4db41017f0 0x7f4db4075590 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f4dac000c00 tx=0x7f4dac02c640 comp rx=0 tx=0).stop 2024-03-16T11:48:45.114 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:45.115+0000 7f4dbc3b3640 1 -- 172.21.15.29:0/3936659366 shutdown_connections 2024-03-16T11:48:45.114 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:45.115+0000 7f4dbc3b3640 1 --2- 172.21.15.29:0/3936659366 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f4d9c0a2800 0x7f4d9c0a4cc0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:45.115 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:45.115+0000 7f4dbc3b3640 1 --2- 172.21.15.29:0/3936659366 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4db41033a0 0x7f4db4074110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:45.115 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:45.115+0000 7f4dbc3b3640 1 --2- 172.21.15.29:0/3936659366 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4db41029e0 0x7f4db4073bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:45.115 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:45.115+0000 7f4dbc3b3640 1 --2- 172.21.15.29:0/3936659366 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4db41017f0 0x7f4db4075590 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:45.115 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:45.115+0000 7f4dbc3b3640 1 -- 172.21.15.29:0/3936659366 >> 172.21.15.29:0/3936659366 conn(0x7f4db40fcdc0 msgr2=0x7f4db4105ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:48:45.115 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:45.115+0000 7f4dbc3b3640 1 -- 172.21.15.29:0/3936659366 shutdown_connections 2024-03-16T11:48:45.115 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:45.115+0000 7f4dbc3b3640 1 -- 172.21.15.29:0/3936659366 wait complete. 2024-03-16T11:48:45.125 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-16T11:48:45.229 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:45.228+0000 7f5866e97640 1 -- 172.21.15.29:0/3446622976 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f58600ffac0 msgr2=0x7f58600ffee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:48:45.229 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:45.228+0000 7f5866e97640 1 --2- 172.21.15.29:0/3446622976 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f58600ffac0 0x7f58600ffee0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f5858009900 tx=0x7f585802bc30 comp rx=0 tx=0).stop 2024-03-16T11:48:45.229 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:45.228+0000 7f5866e97640 1 -- 172.21.15.29:0/3446622976 shutdown_connections 2024-03-16T11:48:45.229 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:45.228+0000 7f5866e97640 1 --2- 172.21.15.29:0/3446622976 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5860103450 0x7f5860109ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:45.229 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:45.228+0000 7f5866e97640 1 --2- 172.21.15.29:0/3446622976 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5860102a90 0x7f5860102f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:45.230 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:45.228+0000 7f5866e97640 1 --2- 172.21.15.29:0/3446622976 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f58600ffac0 0x7f58600ffee0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:45.230 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:45.228+0000 7f5866e97640 1 -- 172.21.15.29:0/3446622976 >> 172.21.15.29:0/3446622976 conn(0x7f58600fb410 msgr2=0x7f58600fd830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:48:45.230 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:45.228+0000 7f5866e97640 1 -- 172.21.15.29:0/3446622976 shutdown_connections 2024-03-16T11:48:45.230 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:45.228+0000 7f5866e97640 1 -- 172.21.15.29:0/3446622976 wait complete. 2024-03-16T11:48:45.230 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:45.229+0000 7f5866e97640 1 Processor -- start 2024-03-16T11:48:45.230 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:45.229+0000 7f5866e97640 1 -- start start 2024-03-16T11:48:45.230 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:45.229+0000 7f5866e97640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f58600ffac0 0x7f5860197bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:48:45.230 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:45.229+0000 7f5866e97640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5860102a90 0x7f5860198100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:48:45.230 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:45.229+0000 7f5866e97640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5860103450 0x7f586019f120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:48:45.230 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:45.229+0000 7f5866e97640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f586010d490 con 0x7f5860103450 2024-03-16T11:48:45.230 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:45.229+0000 7f5866e97640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f586010d310 con 0x7f5860102a90 2024-03-16T11:48:45.230 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:45.229+0000 7f5866e97640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f586010d610 con 0x7f58600ffac0 2024-03-16T11:48:45.230 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:45.229+0000 7f5865434640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5860103450 0x7f586019f120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:48:45.230 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:45.229+0000 7f5864c33640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f58600ffac0 0x7f5860197bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:48:45.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:45.229+0000 7f5865434640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5860103450 0x7f586019f120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49236/0 (socket says 172.21.15.29:49236) 2024-03-16T11:48:45.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:45.229+0000 7f5864c33640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f58600ffac0 0x7f5860197bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.29:38830/0 (socket says 172.21.15.29:38830) 2024-03-16T11:48:45.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:45.229+0000 7f5865434640 1 -- 172.21.15.29:0/2144194365 learned_addr learned my addr 172.21.15.29:0/2144194365 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:48:45.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:45.229+0000 7f585ffff640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5860102a90 0x7f5860198100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:48:45.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:45.229+0000 7f5865434640 1 -- 172.21.15.29:0/2144194365 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f58600ffac0 msgr2=0x7f5860197bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:48:45.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:45.229+0000 7f5865434640 1 --2- 172.21.15.29:0/2144194365 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f58600ffac0 0x7f5860197bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:45.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:45.229+0000 7f5865434640 1 -- 172.21.15.29:0/2144194365 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5860102a90 msgr2=0x7f5860198100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:48:45.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:45.229+0000 7f5865434640 1 --2- 172.21.15.29:0/2144194365 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5860102a90 0x7f5860198100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:45.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:45.229+0000 7f5865434640 1 -- 172.21.15.29:0/2144194365 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f586019f790 con 0x7f5860103450 2024-03-16T11:48:45.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:45.229+0000 7f5864c33640 1 --2- 172.21.15.29:0/2144194365 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f58600ffac0 0x7f5860197bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:48:45.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:45.229+0000 7f585ffff640 1 --2- 172.21.15.29:0/2144194365 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5860102a90 0x7f5860198100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:48:45.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:45.229+0000 7f5865434640 1 --2- 172.21.15.29:0/2144194365 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5860103450 0x7f586019f120 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f584800e9b0 tx=0x7f584800ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:48:45.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:45.230+0000 7f585dffb640 1 -- 172.21.15.29:0/2144194365 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f584800cd80 con 0x7f5860103450 2024-03-16T11:48:45.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:45.230+0000 7f5866e97640 1 -- 172.21.15.29:0/2144194365 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f586019fa80 con 0x7f5860103450 2024-03-16T11:48:45.232 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:45.230+0000 7f585dffb640 1 -- 172.21.15.29:0/2144194365 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f58480035f0 con 0x7f5860103450 2024-03-16T11:48:45.232 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:45.230+0000 7f585dffb640 1 -- 172.21.15.29:0/2144194365 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5848010670 con 0x7f5860103450 2024-03-16T11:48:45.232 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:45.230+0000 7f5866e97640 1 -- 172.21.15.29:0/2144194365 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f586019ff80 con 0x7f5860103450 2024-03-16T11:48:45.232 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:45.231+0000 7f5866e97640 1 -- 172.21.15.29:0/2144194365 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5860108ce0 con 0x7f5860103450 2024-03-16T11:48:45.232 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:45.232+0000 7f585dffb640 1 -- 172.21.15.29:0/2144194365 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f58480037a0 con 0x7f5860103450 2024-03-16T11:48:45.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:45.232+0000 7f585dffb640 1 --2- 172.21.15.29:0/2144194365 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f584c0a2890 0x7f584c0a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:48:45.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:45.232+0000 7f585dffb640 1 -- 172.21.15.29:0/2144194365 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f58480ced00 con 0x7f5860103450 2024-03-16T11:48:45.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:45.232+0000 7f5864c33640 1 --2- 172.21.15.29:0/2144194365 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f584c0a2890 0x7f584c0a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:48:45.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:45.233+0000 7f5864c33640 1 --2- 172.21.15.29:0/2144194365 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f584c0a2890 0x7f584c0a4d50 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f5858000c00 tx=0x7f5858005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:48:45.236 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:45.237+0000 7f585dffb640 1 -- 172.21.15.29:0/2144194365 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f58480d2030 con 0x7f5860103450 2024-03-16T11:48:45.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:45 smithi154 ceph-mon[36320]: from='client.14766 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T11:48:45.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:45 smithi154 ceph-mon[36320]: from='client.14772 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T11:48:45.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:45 smithi154 ceph-mon[36320]: from='client.14778 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T11:48:45.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:45 smithi029 ceph-mon[31975]: from='client.14766 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T11:48:45.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:45 smithi029 ceph-mon[31975]: from='client.14772 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T11:48:45.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:45 smithi029 ceph-mon[31975]: from='client.14778 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T11:48:45.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:45 smithi152 ceph-mon[36291]: from='client.14766 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T11:48:45.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:45 smithi152 ceph-mon[36291]: from='client.14772 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T11:48:45.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:45 smithi152 ceph-mon[36291]: from='client.14778 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T11:48:45.427 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:45.428+0000 7f5866e97640 1 -- 172.21.15.29:0/2144194365 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f58600623e0 con 0x7f5860103450 2024-03-16T11:48:45.428 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:45.429+0000 7f585dffb640 1 -- 172.21.15.29:0/2144194365 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) ==== 74+0+22233 (secure 0 0 0) 0x7f584809bdf0 con 0x7f5860103450 2024-03-16T11:48:45.429 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-16T11:48:45.429 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":85,"fsid":"d1887f28-e389-11ee-95c9-87774f69a715","created":"2024-03-16T11:40:16.475063+0000","modified":"2024-03-16T11:48:39.554402+0000","last_up_change":"2024-03-16T11:47:44.349348+0000","last_in_change":"2024-03-16T11:47:23.672432+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-16T11:43:51.892652+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-16T11:48:36.709732+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-16T11:48:37.586140+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"60f5608f-a408-45ba-bdf6-5b53758f88fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":2748947913},{"type":"v1","addr":"172.21.15.29:6803","nonce":2748947913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":2748947913},{"type":"v1","addr":"172.21.15.29:6805","nonce":2748947913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":2748947913},{"type":"v1","addr":"172.21.15.29:6809","nonce":2748947913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":2748947913},{"type":"v1","addr":"172.21.15.29:6807","nonce":2748947913}]},"public_addr":"172.21.15.29:6803/2748947913","cluster_addr":"172.21.15.29:6805/2748947913","heartbeat_back_addr":"172.21.15.29:6809/2748947913","heartbeat_front_addr":"172.21.15.29:6807/2748947913","state":["exists","up"]},{"osd":1,"uuid":"e70b53d6-ee78-4e56-9a2c-263ae0501b43","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6800","nonce":2339408422},{"type":"v1","addr":"172.21.15.152:6801","nonce":2339408422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6802","nonce":2339408422},{"type":"v1","addr":"172.21.15.152:6803","nonce":2339408422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6806","nonce":2339408422},{"type":"v1","addr":"172.21.15.152:6807","nonce":2339408422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6804","nonce":2339408422},{"type":"v1","addr":"172.21.15.152:6805","nonce":2339408422}]},"public_addr":"172.21.15.152:6801/2339408422","cluster_addr":"172.21.15.152:6803/2339408422","heartbeat_back_addr":"172.21.15.152:6807/2339408422","heartbeat_front_addr":"172.21.15.152:6805/2339408422","state":["exists","up"]},{"osd":2,"uuid":"ff39389a-dfac-44b5-80ff-36abca316e78","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6800","nonce":4288325280},{"type":"v1","addr":"172.21.15.154:6801","nonce":4288325280}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6802","nonce":4288325280},{"type":"v1","addr":"172.21.15.154:6803","nonce":4288325280}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6806","nonce":4288325280},{"type":"v1","addr":"172.21.15.154:6807","nonce":4288325280}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":4288325280},{"type":"v1","addr":"172.21.15.154:6805","nonce":4288325280}]},"public_addr":"172.21.15.154:6801/4288325280","cluster_addr":"172.21.15.154:6803/4288325280","heartbeat_back_addr":"172.21.15.154:6807/4288325280","heartbeat_front_addr":"172.21.15.154:6805/4288325280","state":["exists","up"]},{"osd":3,"uuid":"069e1795-55bc-4324-b94c-eb24ef2334c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":1538750397},{"type":"v1","addr":"172.21.15.29:6811","nonce":1538750397}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":1538750397},{"type":"v1","addr":"172.21.15.29:6813","nonce":1538750397}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":1538750397},{"type":"v1","addr":"172.21.15.29:6817","nonce":1538750397}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":1538750397},{"type":"v1","addr":"172.21.15.29:6815","nonce":1538750397}]},"public_addr":"172.21.15.29:6811/1538750397","cluster_addr":"172.21.15.29:6813/1538750397","heartbeat_back_addr":"172.21.15.29:6817/1538750397","heartbeat_front_addr":"172.21.15.29:6815/1538750397","state":["exists","up"]},{"osd":4,"uuid":"5a9d43a1-e8f5-4ab5-a441-afbc6ed4066e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6808","nonce":2795081750},{"type":"v1","addr":"172.21.15.152:6809","nonce":2795081750}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6810","nonce":2795081750},{"type":"v1","addr":"172.21.15.152:6811","nonce":2795081750}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6814","nonce":2795081750},{"type":"v1","addr":"172.21.15.152:6815","nonce":2795081750}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6812","nonce":2795081750},{"type":"v1","addr":"172.21.15.152:6813","nonce":2795081750}]},"public_addr":"172.21.15.152:6809/2795081750","cluster_addr":"172.21.15.152:6811/2795081750","heartbeat_back_addr":"172.21.15.152:6815/2795081750","heartbeat_front_addr":"172.21.15.152:6813/2795081750","state":["exists","up"]},{"osd":5,"uuid":"234de428-4ac7-44be-9a54-7b58331fdf8c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":1472686560},{"type":"v1","addr":"172.21.15.154:6809","nonce":1472686560}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6810","nonce":1472686560},{"type":"v1","addr":"172.21.15.154:6811","nonce":1472686560}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6814","nonce":1472686560},{"type":"v1","addr":"172.21.15.154:6815","nonce":1472686560}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6812","nonce":1472686560},{"type":"v1","addr":"172.21.15.154:6813","nonce":1472686560}]},"public_addr":"172.21.15.154:6809/1472686560","cluster_addr":"172.21.15.154:6811/1472686560","heartbeat_back_addr":"172.21.15.154:6815/1472686560","heartbeat_front_addr":"172.21.15.154:6813/1472686560","state":["exists","up"]},{"osd":6,"uuid":"ae45b8d9-7883-40c9-a9cd-4c92b601d4f2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":816506244},{"type":"v1","addr":"172.21.15.29:6819","nonce":816506244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":816506244},{"type":"v1","addr":"172.21.15.29:6821","nonce":816506244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":816506244},{"type":"v1","addr":"172.21.15.29:6825","nonce":816506244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":816506244},{"type":"v1","addr":"172.21.15.29:6823","nonce":816506244}]},"public_addr":"172.21.15.29:6819/816506244","cluster_addr":"172.21.15.29:6821/816506244","heartbeat_back_addr":"172.21.15.29:6825/816506244","heartbeat_front_addr":"172.21.15.29:6823/816506244","state":["exists","up"]},{"osd":7,"uuid":"6f687b28-4fce-4d4a-8b4e-294173d9d6c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6816","nonce":2924651787},{"type":"v1","addr":"172.21.15.152:6817","nonce":2924651787}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6818","nonce":2924651787},{"type":"v1","addr":"172.21.15.152:6819","nonce":2924651787}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6822","nonce":2924651787},{"type":"v1","addr":"172.21.15.152:6823","nonce":2924651787}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6820","nonce":2924651787},{"type":"v1","addr":"172.21.15.152:6821","nonce":2924651787}]},"public_addr":"172.21.15.152:6817/2924651787","cluster_addr":"172.21.15.152:6819/2924651787","heartbeat_back_addr":"172.21.15.152:6823/2924651787","heartbeat_front_addr":"172.21.15.152:6821/2924651787","state":["exists","up"]},{"osd":8,"uuid":"a25fb741-caa7-429a-b540-e394d034b446","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6816","nonce":129974156},{"type":"v1","addr":"172.21.15.154:6817","nonce":129974156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6818","nonce":129974156},{"type":"v1","addr":"172.21.15.154:6819","nonce":129974156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6822","nonce":129974156},{"type":"v1","addr":"172.21.15.154:6823","nonce":129974156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6820","nonce":129974156},{"type":"v1","addr":"172.21.15.154:6821","nonce":129974156}]},"public_addr":"172.21.15.154:6817/129974156","cluster_addr":"172.21.15.154:6819/129974156","heartbeat_back_addr":"172.21.15.154:6823/129974156","heartbeat_front_addr":"172.21.15.154:6821/129974156","state":["exists","up"]},{"osd":9,"uuid":"1071cb0e-8c5f-4cd8-9805-0045e220d946","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6826","nonce":2951414243},{"type":"v1","addr":"172.21.15.29:6827","nonce":2951414243}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":2951414243},{"type":"v1","addr":"172.21.15.29:6829","nonce":2951414243}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":2951414243},{"type":"v1","addr":"172.21.15.29:6833","nonce":2951414243}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":2951414243},{"type":"v1","addr":"172.21.15.29:6831","nonce":2951414243}]},"public_addr":"172.21.15.29:6827/2951414243","cluster_addr":"172.21.15.29:6829/2951414243","heartbeat_back_addr":"172.21.15.29:6833/2951414243","heartbeat_front_addr":"172.21.15.29:6831/2951414243","state":["exists","up"]},{"osd":10,"uuid":"ee7c7d74-3f39-4cd4-be80-843fb1dd9ecc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6824","nonce":3412525480},{"type":"v1","addr":"172.21.15.152:6825","nonce":3412525480}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6826","nonce":3412525480},{"type":"v1","addr":"172.21.15.152:6827","nonce":3412525480}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6830","nonce":3412525480},{"type":"v1","addr":"172.21.15.152:6831","nonce":3412525480}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6828","nonce":3412525480},{"type":"v1","addr":"172.21.15.152:6829","nonce":3412525480}]},"public_addr":"172.21.15.152:6825/3412525480","cluster_addr":"172.21.15.152:6827/3412525480","heartbeat_back_addr":"172.21.15.152:6831/3412525480","heartbeat_front_addr":"172.21.15.152:6829/3412525480","state":["exists","up"]},{"osd":11,"uuid":"4598c803-984a-4de6-ac22-066502c17889","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6824","nonce":1160927966},{"type":"v1","addr":"172.21.15.154:6825","nonce":1160927966}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6826","nonce":1160927966},{"type":"v1","addr":"172.21.15.154:6827","nonce":1160927966}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6830","nonce":1160927966},{"type":"v1","addr":"172.21.15.154:6831","nonce":1160927966}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6828","nonce":1160927966},{"type":"v1","addr":"172.21.15.154:6829","nonce":1160927966}]},"public_addr":"172.21.15.154:6825/1160927966","cluster_addr":"172.21.15.154:6827/1160927966","heartbeat_back_addr":"172.21.15.154:6831/1160927966","heartbeat_front_addr":"172.21.15.154:6829/1160927966","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:43:02.343393+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:43:25.209151+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:43:48.946164+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:44:14.588424+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:44:39.700025+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:45:04.061197+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:45:31.214508+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:45:56.937296+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:46:22.768915+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:46:50.101011+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:47:16.298041+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:47:42.402095+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.29:0/1853086886":"2024-03-17T11:41:27.531724+0000","172.21.15.29:0/1211659364":"2024-03-17T11:41:27.531724+0000","172.21.15.29:6800/2798831128":"2024-03-17T11:41:27.531724+0000","172.21.15.29:0/1281028285":"2024-03-17T11:41:00.505678+0000","172.21.15.29:6800/2275315654":"2024-03-17T11:41:00.505678+0000","172.21.15.29:6801/2798831128":"2024-03-17T11:41:27.531724+0000","172.21.15.29:6801/2275315654":"2024-03-17T11:41:00.505678+0000","172.21.15.29:0/1330100009":"2024-03-17T11:41:00.505678+0000","172.21.15.29:0/1784249039":"2024-03-17T11:41:00.505678+0000","172.21.15.29:0/1116486932":"2024-03-17T11:41:27.531724+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-16T11:48:45.431 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:45.431+0000 7f5866e97640 1 -- 172.21.15.29:0/2144194365 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f584c0a2890 msgr2=0x7f584c0a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:48:45.431 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:45.431+0000 7f5866e97640 1 --2- 172.21.15.29:0/2144194365 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f584c0a2890 0x7f584c0a4d50 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f5858000c00 tx=0x7f5858005fd0 comp rx=0 tx=0).stop 2024-03-16T11:48:45.431 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:45.431+0000 7f5866e97640 1 -- 172.21.15.29:0/2144194365 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5860103450 msgr2=0x7f586019f120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:48:45.431 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:45.431+0000 7f5866e97640 1 --2- 172.21.15.29:0/2144194365 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5860103450 0x7f586019f120 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f584800e9b0 tx=0x7f584800ee70 comp rx=0 tx=0).stop 2024-03-16T11:48:45.431 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:45.431+0000 7f5866e97640 1 -- 172.21.15.29:0/2144194365 shutdown_connections 2024-03-16T11:48:45.431 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:45.431+0000 7f5866e97640 1 --2- 172.21.15.29:0/2144194365 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f584c0a2890 0x7f584c0a4d50 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:45.431 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:45.431+0000 7f5866e97640 1 --2- 172.21.15.29:0/2144194365 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5860103450 0x7f586019f120 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:45.431 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:45.431+0000 7f5866e97640 1 --2- 172.21.15.29:0/2144194365 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5860102a90 0x7f5860198100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:45.431 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:45.431+0000 7f5866e97640 1 --2- 172.21.15.29:0/2144194365 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f58600ffac0 0x7f5860197bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:45.431 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:45.431+0000 7f5866e97640 1 -- 172.21.15.29:0/2144194365 >> 172.21.15.29:0/2144194365 conn(0x7f58600fb410 msgr2=0x7f5860105eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:48:45.431 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:45.431+0000 7f5866e97640 1 -- 172.21.15.29:0/2144194365 shutdown_connections 2024-03-16T11:48:45.431 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:45.432+0000 7f5866e97640 1 -- 172.21.15.29:0/2144194365 wait complete. 2024-03-16T11:48:45.442 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-16T11:48:45.442 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-16T11:48:45.547 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:45.546+0000 7ff0a243f640 1 -- 172.21.15.29:0/304663740 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff09c1033b0 msgr2=0x7ff09c109c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:48:45.547 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:45.546+0000 7ff0a243f640 1 --2- 172.21.15.29:0/304663740 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff09c1033b0 0x7ff09c109c40 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7ff094009ca0 tx=0x7ff09402c0f0 comp rx=0 tx=0).stop 2024-03-16T11:48:45.547 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:45.546+0000 7ff0a243f640 1 -- 172.21.15.29:0/304663740 shutdown_connections 2024-03-16T11:48:45.548 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:45.546+0000 7ff0a243f640 1 --2- 172.21.15.29:0/304663740 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff09c1033b0 0x7ff09c109c40 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:45.548 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:45.546+0000 7ff0a243f640 1 --2- 172.21.15.29:0/304663740 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff09c1029f0 0x7ff09c102e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:45.548 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:45.546+0000 7ff0a243f640 1 --2- 172.21.15.29:0/304663740 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff09c101800 0x7ff09c101c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:45.548 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:45.546+0000 7ff0a243f640 1 -- 172.21.15.29:0/304663740 >> 172.21.15.29:0/304663740 conn(0x7ff09c0fcd90 msgr2=0x7ff09c0ff1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:48:45.548 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:45.546+0000 7ff0a243f640 1 -- 172.21.15.29:0/304663740 shutdown_connections 2024-03-16T11:48:45.548 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:45.546+0000 7ff0a243f640 1 -- 172.21.15.29:0/304663740 wait complete. 2024-03-16T11:48:45.548 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:45.546+0000 7ff0a243f640 1 Processor -- start 2024-03-16T11:48:45.548 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:45.546+0000 7ff0a243f640 1 -- start start 2024-03-16T11:48:45.548 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:45.547+0000 7ff0a243f640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff09c101800 0x7ff09c199cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:48:45.548 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:45.547+0000 7ff0a243f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff09c1029f0 0x7ff09c19a230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:48:45.548 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:45.547+0000 7ff0a243f640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff09c1033b0 0x7ff09c1a1260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:48:45.548 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:45.547+0000 7ff0a243f640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7ff09c074020 con 0x7ff09c1029f0 2024-03-16T11:48:45.548 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:45.547+0000 7ff0a243f640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7ff09c073ea0 con 0x7ff09c1033b0 2024-03-16T11:48:45.548 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:45.547+0000 7ff0a243f640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7ff09c0741a0 con 0x7ff09c101800 2024-03-16T11:48:45.548 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:45.547+0000 7ff09b7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff09c1029f0 0x7ff09c19a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:48:45.549 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:45.547+0000 7ff09b7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff09c1029f0 0x7ff09c19a230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49248/0 (socket says 172.21.15.29:49248) 2024-03-16T11:48:45.549 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:45.547+0000 7ff09b7fe640 1 -- 172.21.15.29:0/2436549081 learned_addr learned my addr 172.21.15.29:0/2436549081 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:48:45.549 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:45.547+0000 7ff09bfff640 1 --2- 172.21.15.29:0/2436549081 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff09c101800 0x7ff09c199cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:48:45.549 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:45.547+0000 7ff0a09dc640 1 --2- 172.21.15.29:0/2436549081 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff09c1033b0 0x7ff09c1a1260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:48:45.549 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:45.547+0000 7ff09b7fe640 1 -- 172.21.15.29:0/2436549081 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff09c101800 msgr2=0x7ff09c199cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:48:45.549 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:45.547+0000 7ff09b7fe640 1 --2- 172.21.15.29:0/2436549081 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff09c101800 0x7ff09c199cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:45.549 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:45.547+0000 7ff09b7fe640 1 -- 172.21.15.29:0/2436549081 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff09c1033b0 msgr2=0x7ff09c1a1260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:48:45.549 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:45.547+0000 7ff09b7fe640 1 --2- 172.21.15.29:0/2436549081 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff09c1033b0 0x7ff09c1a1260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:45.549 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:45.547+0000 7ff09b7fe640 1 -- 172.21.15.29:0/2436549081 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff09c1a1900 con 0x7ff09c1029f0 2024-03-16T11:48:45.549 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:45.547+0000 7ff09bfff640 1 --2- 172.21.15.29:0/2436549081 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff09c101800 0x7ff09c199cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:48:45.549 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:45.547+0000 7ff09b7fe640 1 --2- 172.21.15.29:0/2436549081 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff09c1029f0 0x7ff09c19a230 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7ff08c00ea60 tx=0x7ff08c00ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:48:45.549 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:45.547+0000 7ff0997fa640 1 -- 172.21.15.29:0/2436549081 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff08c00cde0 con 0x7ff09c1029f0 2024-03-16T11:48:45.549 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:45.547+0000 7ff0997fa640 1 -- 172.21.15.29:0/2436549081 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff08c0035f0 con 0x7ff09c1029f0 2024-03-16T11:48:45.550 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:45.548+0000 7ff0a243f640 1 -- 172.21.15.29:0/2436549081 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff09c1a1bf0 con 0x7ff09c1029f0 2024-03-16T11:48:45.550 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:45.548+0000 7ff0997fa640 1 -- 172.21.15.29:0/2436549081 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff08c010670 con 0x7ff09c1029f0 2024-03-16T11:48:45.550 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:45.548+0000 7ff0a243f640 1 -- 172.21.15.29:0/2436549081 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff09c1a2140 con 0x7ff09c1029f0 2024-03-16T11:48:45.551 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:45.549+0000 7ff0a243f640 1 -- 172.21.15.29:0/2436549081 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff09c1a2370 con 0x7ff09c1029f0 2024-03-16T11:48:45.551 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:45.550+0000 7ff0997fa640 1 -- 172.21.15.29:0/2436549081 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7ff08c0037a0 con 0x7ff09c1029f0 2024-03-16T11:48:45.551 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:45.550+0000 7ff0997fa640 1 --2- 172.21.15.29:0/2436549081 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7ff0880a2890 0x7ff0880a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:48:45.551 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:45.550+0000 7ff0997fa640 1 -- 172.21.15.29:0/2436549081 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7ff08c0cedc0 con 0x7ff09c1029f0 2024-03-16T11:48:45.551 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:45.550+0000 7ff09bfff640 1 --2- 172.21.15.29:0/2436549081 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7ff0880a2890 0x7ff0880a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:48:45.551 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:45.551+0000 7ff09bfff640 1 --2- 172.21.15.29:0/2436549081 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7ff0880a2890 0x7ff0880a4d50 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7ff09c19b1c0 tx=0x7ff084005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:48:45.554 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:45.555+0000 7ff0997fa640 1 -- 172.21.15.29:0/2436549081 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7ff08c09beb0 con 0x7ff09c1029f0 2024-03-16T11:48:45.738 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:45.739+0000 7ff0a243f640 1 -- 172.21.15.29:0/2436549081 --> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7ff09c1a2540 con 0x7ff0880a2890 2024-03-16T11:48:45.745 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:45.746+0000 7ff0997fa640 1 -- 172.21.15.29:0/2436549081 <== mgr.14152 v2:172.21.15.29:6800/4192064640 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258890 (secure 0 0 0) 0x7ff09c1a2540 con 0x7ff0880a2890 2024-03-16T11:48:45.749 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-16T11:48:45.750 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2024-03-16T11:48:45.752 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:45.749+0000 7ff0a243f640 1 -- 172.21.15.29:0/2436549081 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7ff0880a2890 msgr2=0x7ff0880a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:48:45.752 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:45.749+0000 7ff0a243f640 1 --2- 172.21.15.29:0/2436549081 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7ff0880a2890 0x7ff0880a4d50 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7ff09c19b1c0 tx=0x7ff084005f90 comp rx=0 tx=0).stop 2024-03-16T11:48:45.753 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:45.749+0000 7ff0a243f640 1 -- 172.21.15.29:0/2436549081 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff09c1029f0 msgr2=0x7ff09c19a230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:48:45.753 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:45.749+0000 7ff0a243f640 1 --2- 172.21.15.29:0/2436549081 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff09c1029f0 0x7ff09c19a230 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7ff08c00ea60 tx=0x7ff08c00ef20 comp rx=0 tx=0).stop 2024-03-16T11:48:45.753 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:45.749+0000 7ff0a243f640 1 -- 172.21.15.29:0/2436549081 shutdown_connections 2024-03-16T11:48:45.753 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:45.749+0000 7ff0a243f640 1 --2- 172.21.15.29:0/2436549081 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7ff0880a2890 0x7ff0880a4d50 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:45.753 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:45.749+0000 7ff0a243f640 1 --2- 172.21.15.29:0/2436549081 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff09c1033b0 0x7ff09c1a1260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:45.753 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:45.749+0000 7ff0a243f640 1 --2- 172.21.15.29:0/2436549081 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff09c1029f0 0x7ff09c19a230 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:45.753 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:45.749+0000 7ff0a243f640 1 --2- 172.21.15.29:0/2436549081 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff09c101800 0x7ff09c199cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:45.753 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:45.749+0000 7ff0a243f640 1 -- 172.21.15.29:0/2436549081 >> 172.21.15.29:0/2436549081 conn(0x7ff09c0fcd90 msgr2=0x7ff09c0fe6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:48:45.753 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:45.749+0000 7ff0a243f640 1 -- 172.21.15.29:0/2436549081 shutdown_connections 2024-03-16T11:48:45.753 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:45.749+0000 7ff0a243f640 1 -- 172.21.15.29:0/2436549081 wait complete. 2024-03-16T11:48:45.759 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":294,"stamp":"2024-03-16T11:48:45.617605+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":618889,"num_objects":51,"num_object_clones":0,"num_object_copies":153,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51,"num_whiteouts":0,"num_read":149,"num_read_kb":130,"num_write":292,"num_write_kb":2993,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":32,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":235,"ondisk_log_size":235,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":231,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":372692,"kb_used_data":11004,"kb_used_omap":18,"kb_used_meta":361517,"kb_avail":1124569132,"statfs":{"total":1151940427776,"available":1151558791168,"internally_reserved":0,"allocated":11268096,"data_stored":7028817,"data_compressed":108354,"data_compressed_allocated":135168,"data_compressed_original":1769472,"omap_allocated":19143,"internal_metadata":370193721},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":28521,"num_objects":49,"num_object_clones":0,"num_object_copies":147,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49,"num_whiteouts":0,"num_read":13,"num_read_kb":11,"num_write":111,"num_write_kb":55,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":32,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.608452"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T11:48:39.942383+0000","last_change":"2024-03-16T11:48:38.574718+0000","last_active":"2024-03-16T11:48:39.942383+0000","last_peered":"2024-03-16T11:48:39.942383+0000","last_clean":"2024-03-16T11:48:39.942383+0000","last_became_active":"2024-03-16T11:48:38.574420+0000","last_became_peered":"2024-03-16T11:48:38.574420+0000","last_unstale":"2024-03-16T11:48:39.942383+0000","last_undegraded":"2024-03-16T11:48:39.942383+0000","last_fullsized":"2024-03-16T11:48:39.942383+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:43:56.705135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T11:48:41.044196+0000","last_change":"2024-03-16T11:48:38.572722+0000","last_active":"2024-03-16T11:48:41.044196+0000","last_peered":"2024-03-16T11:48:41.044196+0000","last_clean":"2024-03-16T11:48:41.044196+0000","last_became_active":"2024-03-16T11:48:38.572431+0000","last_became_peered":"2024-03-16T11:48:38.572431+0000","last_unstale":"2024-03-16T11:48:41.044196+0000","last_undegraded":"2024-03-16T11:48:41.044196+0000","last_fullsized":"2024-03-16T11:48:41.044196+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:34:45.070975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T11:48:41.044515+0000","last_change":"2024-03-16T11:48:38.585504+0000","last_active":"2024-03-16T11:48:41.044515+0000","last_peered":"2024-03-16T11:48:41.044515+0000","last_clean":"2024-03-16T11:48:41.044515+0000","last_became_active":"2024-03-16T11:48:38.583879+0000","last_became_peered":"2024-03-16T11:48:38.583879+0000","last_unstale":"2024-03-16T11:48:41.044515+0000","last_undegraded":"2024-03-16T11:48:41.044515+0000","last_fullsized":"2024-03-16T11:48:41.044515+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:08:41.119349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T11:48:41.045678+0000","last_change":"2024-03-16T11:48:38.575773+0000","last_active":"2024-03-16T11:48:41.045678+0000","last_peered":"2024-03-16T11:48:41.045678+0000","last_clean":"2024-03-16T11:48:41.045678+0000","last_became_active":"2024-03-16T11:48:38.574441+0000","last_became_peered":"2024-03-16T11:48:38.574441+0000","last_unstale":"2024-03-16T11:48:41.045678+0000","last_undegraded":"2024-03-16T11:48:41.045678+0000","last_fullsized":"2024-03-16T11:48:41.045678+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:22:56.148164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-16T11:48:38.577525+0000","last_change":"2024-03-16T11:48:38.575505+0000","last_active":"2024-03-16T11:48:38.577525+0000","last_peered":"2024-03-16T11:48:38.577525+0000","last_clean":"2024-03-16T11:48:38.577525+0000","last_became_active":"2024-03-16T11:48:38.575188+0000","last_became_peered":"2024-03-16T11:48:38.575188+0000","last_unstale":"2024-03-16T11:48:38.577525+0000","last_undegraded":"2024-03-16T11:48:38.577525+0000","last_fullsized":"2024-03-16T11:48:38.577525+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:44:08.971947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T11:48:39.811366+0000","last_change":"2024-03-16T11:48:38.572044+0000","last_active":"2024-03-16T11:48:39.811366+0000","last_peered":"2024-03-16T11:48:39.811366+0000","last_clean":"2024-03-16T11:48:39.811366+0000","last_became_active":"2024-03-16T11:48:38.571725+0000","last_became_peered":"2024-03-16T11:48:38.571725+0000","last_unstale":"2024-03-16T11:48:39.811366+0000","last_undegraded":"2024-03-16T11:48:39.811366+0000","last_fullsized":"2024-03-16T11:48:39.811366+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:05:37.060342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T11:48:41.043715+0000","last_change":"2024-03-16T11:48:38.570332+0000","last_active":"2024-03-16T11:48:41.043715+0000","last_peered":"2024-03-16T11:48:41.043715+0000","last_clean":"2024-03-16T11:48:41.043715+0000","last_became_active":"2024-03-16T11:48:38.570009+0000","last_became_peered":"2024-03-16T11:48:38.570009+0000","last_unstale":"2024-03-16T11:48:41.043715+0000","last_undegraded":"2024-03-16T11:48:41.043715+0000","last_fullsized":"2024-03-16T11:48:41.043715+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:42:31.045749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T11:48:41.016200+0000","last_change":"2024-03-16T11:48:38.571741+0000","last_active":"2024-03-16T11:48:41.016200+0000","last_peered":"2024-03-16T11:48:41.016200+0000","last_clean":"2024-03-16T11:48:41.016200+0000","last_became_active":"2024-03-16T11:48:38.569582+0000","last_became_peered":"2024-03-16T11:48:38.569582+0000","last_unstale":"2024-03-16T11:48:41.016200+0000","last_undegraded":"2024-03-16T11:48:41.016200+0000","last_fullsized":"2024-03-16T11:48:41.016200+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:05:36.302223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T11:48:39.560942+0000","last_change":"2024-03-16T11:48:38.576038+0000","last_active":"2024-03-16T11:48:39.560942+0000","last_peered":"2024-03-16T11:48:39.560942+0000","last_clean":"2024-03-16T11:48:39.560942+0000","last_became_active":"2024-03-16T11:48:38.574223+0000","last_became_peered":"2024-03-16T11:48:38.574223+0000","last_unstale":"2024-03-16T11:48:39.560942+0000","last_undegraded":"2024-03-16T11:48:39.560942+0000","last_fullsized":"2024-03-16T11:48:39.560942+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:35:17.236721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T11:48:39.811735+0000","last_change":"2024-03-16T11:48:38.572797+0000","last_active":"2024-03-16T11:48:39.811735+0000","last_peered":"2024-03-16T11:48:39.811735+0000","last_clean":"2024-03-16T11:48:39.811735+0000","last_became_active":"2024-03-16T11:48:38.572497+0000","last_became_peered":"2024-03-16T11:48:38.572497+0000","last_unstale":"2024-03-16T11:48:39.811735+0000","last_undegraded":"2024-03-16T11:48:39.811735+0000","last_fullsized":"2024-03-16T11:48:39.811735+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:25:49.138897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T11:48:41.044595+0000","last_change":"2024-03-16T11:48:38.570443+0000","last_active":"2024-03-16T11:48:41.044595+0000","last_peered":"2024-03-16T11:48:41.044595+0000","last_clean":"2024-03-16T11:48:41.044595+0000","last_became_active":"2024-03-16T11:48:38.569984+0000","last_became_peered":"2024-03-16T11:48:38.569984+0000","last_unstale":"2024-03-16T11:48:41.044595+0000","last_undegraded":"2024-03-16T11:48:41.044595+0000","last_fullsized":"2024-03-16T11:48:41.044595+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:47:42.539130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T11:48:41.045524+0000","last_change":"2024-03-16T11:48:38.576106+0000","last_active":"2024-03-16T11:48:41.045524+0000","last_peered":"2024-03-16T11:48:41.045524+0000","last_clean":"2024-03-16T11:48:41.045524+0000","last_became_active":"2024-03-16T11:48:38.575409+0000","last_became_peered":"2024-03-16T11:48:38.575409+0000","last_unstale":"2024-03-16T11:48:41.045524+0000","last_undegraded":"2024-03-16T11:48:41.045524+0000","last_fullsized":"2024-03-16T11:48:41.045524+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:38:33.277722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T11:48:39.557764+0000","last_change":"2024-03-16T11:48:38.559132+0000","last_active":"2024-03-16T11:48:39.557764+0000","last_peered":"2024-03-16T11:48:39.557764+0000","last_clean":"2024-03-16T11:48:39.557764+0000","last_became_active":"2024-03-16T11:48:38.558826+0000","last_became_peered":"2024-03-16T11:48:38.558826+0000","last_unstale":"2024-03-16T11:48:39.557764+0000","last_undegraded":"2024-03-16T11:48:39.557764+0000","last_fullsized":"2024-03-16T11:48:39.557764+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:27:12.562018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T11:48:41.043879+0000","last_change":"2024-03-16T11:48:38.583598+0000","last_active":"2024-03-16T11:48:41.043879+0000","last_peered":"2024-03-16T11:48:41.043879+0000","last_clean":"2024-03-16T11:48:41.043879+0000","last_became_active":"2024-03-16T11:48:38.583275+0000","last_became_peered":"2024-03-16T11:48:38.583275+0000","last_unstale":"2024-03-16T11:48:41.043879+0000","last_undegraded":"2024-03-16T11:48:41.043879+0000","last_fullsized":"2024-03-16T11:48:41.043879+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:58:48.667075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T11:48:41.043772+0000","last_change":"2024-03-16T11:48:38.580240+0000","last_active":"2024-03-16T11:48:41.043772+0000","last_peered":"2024-03-16T11:48:41.043772+0000","last_clean":"2024-03-16T11:48:41.043772+0000","last_became_active":"2024-03-16T11:48:38.579928+0000","last_became_peered":"2024-03-16T11:48:38.579928+0000","last_unstale":"2024-03-16T11:48:41.043772+0000","last_undegraded":"2024-03-16T11:48:41.043772+0000","last_fullsized":"2024-03-16T11:48:41.043772+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:05:14.006915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T11:48:39.560784+0000","last_change":"2024-03-16T11:48:38.574433+0000","last_active":"2024-03-16T11:48:39.560784+0000","last_peered":"2024-03-16T11:48:39.560784+0000","last_clean":"2024-03-16T11:48:39.560784+0000","last_became_active":"2024-03-16T11:48:38.574104+0000","last_became_peered":"2024-03-16T11:48:38.574104+0000","last_unstale":"2024-03-16T11:48:39.560784+0000","last_undegraded":"2024-03-16T11:48:39.560784+0000","last_fullsized":"2024-03-16T11:48:39.560784+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:42:20.588457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T11:48:39.942412+0000","last_change":"2024-03-16T11:48:38.575505+0000","last_active":"2024-03-16T11:48:39.942412+0000","last_peered":"2024-03-16T11:48:39.942412+0000","last_clean":"2024-03-16T11:48:39.942412+0000","last_became_active":"2024-03-16T11:48:38.575187+0000","last_became_peered":"2024-03-16T11:48:38.575187+0000","last_unstale":"2024-03-16T11:48:39.942412+0000","last_undegraded":"2024-03-16T11:48:39.942412+0000","last_fullsized":"2024-03-16T11:48:39.942412+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:13:06.260535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T11:48:40.507277+0000","last_change":"2024-03-16T11:48:38.568146+0000","last_active":"2024-03-16T11:48:40.507277+0000","last_peered":"2024-03-16T11:48:40.507277+0000","last_clean":"2024-03-16T11:48:40.507277+0000","last_became_active":"2024-03-16T11:48:38.567826+0000","last_became_peered":"2024-03-16T11:48:38.567826+0000","last_unstale":"2024-03-16T11:48:40.507277+0000","last_undegraded":"2024-03-16T11:48:40.507277+0000","last_fullsized":"2024-03-16T11:48:40.507277+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:46:32.993012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T11:48:39.558371+0000","last_change":"2024-03-16T11:48:38.578861+0000","last_active":"2024-03-16T11:48:39.558371+0000","last_peered":"2024-03-16T11:48:39.558371+0000","last_clean":"2024-03-16T11:48:39.558371+0000","last_became_active":"2024-03-16T11:48:38.577028+0000","last_became_peered":"2024-03-16T11:48:38.577028+0000","last_unstale":"2024-03-16T11:48:39.558371+0000","last_undegraded":"2024-03-16T11:48:39.558371+0000","last_fullsized":"2024-03-16T11:48:39.558371+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:55:34.520032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"85'9","reported_seq":25,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T11:48:41.978211+0000","last_change":"2024-03-16T11:48:38.026407+0000","last_active":"2024-03-16T11:48:41.978211+0000","last_peered":"2024-03-16T11:48:41.978211+0000","last_clean":"2024-03-16T11:48:41.978211+0000","last_became_active":"2024-03-16T11:48:38.026083+0000","last_became_peered":"2024-03-16T11:48:38.026083+0000","last_unstale":"2024-03-16T11:48:41.978211+0000","last_undegraded":"2024-03-16T11:48:41.978211+0000","last_fullsized":"2024-03-16T11:48:41.978211+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_clean_scrub_stamp":"2024-03-16T11:48:36.997377+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:01:23.751657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T11:48:41.016366+0000","last_change":"2024-03-16T11:48:38.573658+0000","last_active":"2024-03-16T11:48:41.016366+0000","last_peered":"2024-03-16T11:48:41.016366+0000","last_clean":"2024-03-16T11:48:41.016366+0000","last_became_active":"2024-03-16T11:48:38.572747+0000","last_became_peered":"2024-03-16T11:48:38.572747+0000","last_unstale":"2024-03-16T11:48:41.016366+0000","last_undegraded":"2024-03-16T11:48:41.016366+0000","last_fullsized":"2024-03-16T11:48:41.016366+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:09:11.128269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"84'1","reported_seq":16,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T11:48:41.016438+0000","last_change":"2024-03-16T11:48:38.027088+0000","last_active":"2024-03-16T11:48:41.016438+0000","last_peered":"2024-03-16T11:48:41.016438+0000","last_clean":"2024-03-16T11:48:41.016438+0000","last_became_active":"2024-03-16T11:48:38.026730+0000","last_became_peered":"2024-03-16T11:48:38.026730+0000","last_unstale":"2024-03-16T11:48:41.016438+0000","last_undegraded":"2024-03-16T11:48:41.016438+0000","last_fullsized":"2024-03-16T11:48:41.016438+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_clean_scrub_stamp":"2024-03-16T11:48:36.997377+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:23:28.987731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T11:48:41.045297+0000","last_change":"2024-03-16T11:48:38.583730+0000","last_active":"2024-03-16T11:48:41.045297+0000","last_peered":"2024-03-16T11:48:41.045297+0000","last_clean":"2024-03-16T11:48:41.045297+0000","last_became_active":"2024-03-16T11:48:38.583323+0000","last_became_peered":"2024-03-16T11:48:38.583323+0000","last_unstale":"2024-03-16T11:48:41.045297+0000","last_undegraded":"2024-03-16T11:48:41.045297+0000","last_fullsized":"2024-03-16T11:48:41.045297+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:35:20.923176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"84'2","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-16T11:48:39.020665+0000","last_change":"2024-03-16T11:48:38.027310+0000","last_active":"2024-03-16T11:48:39.020665+0000","last_peered":"2024-03-16T11:48:39.020665+0000","last_clean":"2024-03-16T11:48:39.020665+0000","last_became_active":"2024-03-16T11:48:38.026991+0000","last_became_peered":"2024-03-16T11:48:38.026991+0000","last_unstale":"2024-03-16T11:48:39.020665+0000","last_undegraded":"2024-03-16T11:48:39.020665+0000","last_fullsized":"2024-03-16T11:48:39.020665+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_clean_scrub_stamp":"2024-03-16T11:48:36.997377+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:19:24.922203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T11:48:39.560691+0000","last_change":"2024-03-16T11:48:38.581452+0000","last_active":"2024-03-16T11:48:39.560691+0000","last_peered":"2024-03-16T11:48:39.560691+0000","last_clean":"2024-03-16T11:48:39.560691+0000","last_became_active":"2024-03-16T11:48:38.579305+0000","last_became_peered":"2024-03-16T11:48:38.579305+0000","last_unstale":"2024-03-16T11:48:39.560691+0000","last_undegraded":"2024-03-16T11:48:39.560691+0000","last_fullsized":"2024-03-16T11:48:39.560691+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:57:43.812783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-16T11:48:39.019344+0000","last_change":"2024-03-16T11:48:38.023747+0000","last_active":"2024-03-16T11:48:39.019344+0000","last_peered":"2024-03-16T11:48:39.019344+0000","last_clean":"2024-03-16T11:48:39.019344+0000","last_became_active":"2024-03-16T11:48:38.023396+0000","last_became_peered":"2024-03-16T11:48:38.023396+0000","last_unstale":"2024-03-16T11:48:39.019344+0000","last_undegraded":"2024-03-16T11:48:39.019344+0000","last_fullsized":"2024-03-16T11:48:39.019344+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_clean_scrub_stamp":"2024-03-16T11:48:36.997377+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:42:22.456357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T11:48:41.045817+0000","last_change":"2024-03-16T11:48:38.572157+0000","last_active":"2024-03-16T11:48:41.045817+0000","last_peered":"2024-03-16T11:48:41.045817+0000","last_clean":"2024-03-16T11:48:41.045817+0000","last_became_active":"2024-03-16T11:48:38.571629+0000","last_became_peered":"2024-03-16T11:48:38.571629+0000","last_unstale":"2024-03-16T11:48:41.045817+0000","last_undegraded":"2024-03-16T11:48:41.045817+0000","last_fullsized":"2024-03-16T11:48:41.045817+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:47:39.353965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"85'5","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T11:48:41.049457+0000","last_change":"2024-03-16T11:48:38.027883+0000","last_active":"2024-03-16T11:48:41.049457+0000","last_peered":"2024-03-16T11:48:41.049457+0000","last_clean":"2024-03-16T11:48:41.049457+0000","last_became_active":"2024-03-16T11:48:38.027561+0000","last_became_peered":"2024-03-16T11:48:38.027561+0000","last_unstale":"2024-03-16T11:48:41.049457+0000","last_undegraded":"2024-03-16T11:48:41.049457+0000","last_fullsized":"2024-03-16T11:48:41.049457+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_clean_scrub_stamp":"2024-03-16T11:48:36.997377+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T11:58:56.063422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-16T11:48:38.577551+0000","last_change":"2024-03-16T11:48:38.574227+0000","last_active":"2024-03-16T11:48:38.577551+0000","last_peered":"2024-03-16T11:48:38.577551+0000","last_clean":"2024-03-16T11:48:38.577551+0000","last_became_active":"2024-03-16T11:48:38.571938+0000","last_became_peered":"2024-03-16T11:48:38.571938+0000","last_unstale":"2024-03-16T11:48:38.577551+0000","last_undegraded":"2024-03-16T11:48:38.577551+0000","last_fullsized":"2024-03-16T11:48:38.577551+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:48:25.028283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"85'3","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T11:48:41.974511+0000","last_change":"2024-03-16T11:48:38.024448+0000","last_active":"2024-03-16T11:48:41.974511+0000","last_peered":"2024-03-16T11:48:41.974511+0000","last_clean":"2024-03-16T11:48:41.974511+0000","last_became_active":"2024-03-16T11:48:38.024109+0000","last_became_peered":"2024-03-16T11:48:38.024109+0000","last_unstale":"2024-03-16T11:48:41.974511+0000","last_undegraded":"2024-03-16T11:48:41.974511+0000","last_fullsized":"2024-03-16T11:48:41.974511+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_clean_scrub_stamp":"2024-03-16T11:48:36.997377+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:37:49.521297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T11:48:41.044579+0000","last_change":"2024-03-16T11:48:38.577722+0000","last_active":"2024-03-16T11:48:41.044579+0000","last_peered":"2024-03-16T11:48:41.044579+0000","last_clean":"2024-03-16T11:48:41.044579+0000","last_became_active":"2024-03-16T11:48:38.577223+0000","last_became_peered":"2024-03-16T11:48:38.577223+0000","last_unstale":"2024-03-16T11:48:41.044579+0000","last_undegraded":"2024-03-16T11:48:41.044579+0000","last_fullsized":"2024-03-16T11:48:41.044579+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:24:05.892850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"85'2","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T11:48:41.977680+0000","last_change":"2024-03-16T11:48:38.029681+0000","last_active":"2024-03-16T11:48:41.977680+0000","last_peered":"2024-03-16T11:48:41.977680+0000","last_clean":"2024-03-16T11:48:41.977680+0000","last_became_active":"2024-03-16T11:48:38.029261+0000","last_became_peered":"2024-03-16T11:48:38.029261+0000","last_unstale":"2024-03-16T11:48:41.977680+0000","last_undegraded":"2024-03-16T11:48:41.977680+0000","last_fullsized":"2024-03-16T11:48:41.977680+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_clean_scrub_stamp":"2024-03-16T11:48:36.997377+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:57:51.155337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T11:48:41.015999+0000","last_change":"2024-03-16T11:48:38.570655+0000","last_active":"2024-03-16T11:48:41.015999+0000","last_peered":"2024-03-16T11:48:41.015999+0000","last_clean":"2024-03-16T11:48:41.015999+0000","last_became_active":"2024-03-16T11:48:38.569143+0000","last_became_peered":"2024-03-16T11:48:38.569143+0000","last_unstale":"2024-03-16T11:48:41.015999+0000","last_undegraded":"2024-03-16T11:48:41.015999+0000","last_fullsized":"2024-03-16T11:48:41.015999+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:51:51.449168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"85'1","reported_seq":16,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T11:48:41.048005+0000","last_change":"2024-03-16T11:48:38.027074+0000","last_active":"2024-03-16T11:48:41.048005+0000","last_peered":"2024-03-16T11:48:41.048005+0000","last_clean":"2024-03-16T11:48:41.048005+0000","last_became_active":"2024-03-16T11:48:38.026749+0000","last_became_peered":"2024-03-16T11:48:38.026749+0000","last_unstale":"2024-03-16T11:48:41.048005+0000","last_undegraded":"2024-03-16T11:48:41.048005+0000","last_fullsized":"2024-03-16T11:48:41.048005+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_clean_scrub_stamp":"2024-03-16T11:48:36.997377+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:14:26.801688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T11:48:40.507244+0000","last_change":"2024-03-16T11:48:38.575699+0000","last_active":"2024-03-16T11:48:40.507244+0000","last_peered":"2024-03-16T11:48:40.507244+0000","last_clean":"2024-03-16T11:48:40.507244+0000","last_became_active":"2024-03-16T11:48:38.575401+0000","last_became_peered":"2024-03-16T11:48:38.575401+0000","last_unstale":"2024-03-16T11:48:40.507244+0000","last_undegraded":"2024-03-16T11:48:40.507244+0000","last_fullsized":"2024-03-16T11:48:40.507244+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:48:06.289779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"85'3","reported_seq":21,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T11:48:42.973069+0000","last_change":"2024-03-16T11:48:38.026488+0000","last_active":"2024-03-16T11:48:42.973069+0000","last_peered":"2024-03-16T11:48:42.973069+0000","last_clean":"2024-03-16T11:48:42.973069+0000","last_became_active":"2024-03-16T11:48:38.026101+0000","last_became_peered":"2024-03-16T11:48:38.026101+0000","last_unstale":"2024-03-16T11:48:42.973069+0000","last_undegraded":"2024-03-16T11:48:42.973069+0000","last_fullsized":"2024-03-16T11:48:42.973069+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_clean_scrub_stamp":"2024-03-16T11:48:36.997377+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:37:29.788307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T11:48:39.560914+0000","last_change":"2024-03-16T11:48:38.578081+0000","last_active":"2024-03-16T11:48:39.560914+0000","last_peered":"2024-03-16T11:48:39.560914+0000","last_clean":"2024-03-16T11:48:39.560914+0000","last_became_active":"2024-03-16T11:48:38.577742+0000","last_became_peered":"2024-03-16T11:48:38.577742+0000","last_unstale":"2024-03-16T11:48:39.560914+0000","last_undegraded":"2024-03-16T11:48:39.560914+0000","last_fullsized":"2024-03-16T11:48:39.560914+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:58:41.479658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"85'2","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T11:48:41.047455+0000","last_change":"2024-03-16T11:48:38.026891+0000","last_active":"2024-03-16T11:48:41.047455+0000","last_peered":"2024-03-16T11:48:41.047455+0000","last_clean":"2024-03-16T11:48:41.047455+0000","last_became_active":"2024-03-16T11:48:38.026397+0000","last_became_peered":"2024-03-16T11:48:38.026397+0000","last_unstale":"2024-03-16T11:48:41.047455+0000","last_undegraded":"2024-03-16T11:48:41.047455+0000","last_fullsized":"2024-03-16T11:48:41.047455+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_clean_scrub_stamp":"2024-03-16T11:48:36.997377+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:02:01.018103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T11:48:41.045244+0000","last_change":"2024-03-16T11:48:38.579205+0000","last_active":"2024-03-16T11:48:41.045244+0000","last_peered":"2024-03-16T11:48:41.045244+0000","last_clean":"2024-03-16T11:48:41.045244+0000","last_became_active":"2024-03-16T11:48:38.578910+0000","last_became_peered":"2024-03-16T11:48:38.578910+0000","last_unstale":"2024-03-16T11:48:41.045244+0000","last_undegraded":"2024-03-16T11:48:41.045244+0000","last_fullsized":"2024-03-16T11:48:41.045244+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:22:06.498974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"85'4","reported_seq":19,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T11:48:41.048250+0000","last_change":"2024-03-16T11:48:38.025103+0000","last_active":"2024-03-16T11:48:41.048250+0000","last_peered":"2024-03-16T11:48:41.048250+0000","last_clean":"2024-03-16T11:48:41.048250+0000","last_became_active":"2024-03-16T11:48:38.024754+0000","last_became_peered":"2024-03-16T11:48:38.024754+0000","last_unstale":"2024-03-16T11:48:41.048250+0000","last_undegraded":"2024-03-16T11:48:41.048250+0000","last_fullsized":"2024-03-16T11:48:41.048250+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_clean_scrub_stamp":"2024-03-16T11:48:36.997377+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:40:10.335339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T11:48:39.560755+0000","last_change":"2024-03-16T11:48:38.578305+0000","last_active":"2024-03-16T11:48:39.560755+0000","last_peered":"2024-03-16T11:48:39.560755+0000","last_clean":"2024-03-16T11:48:39.560755+0000","last_became_active":"2024-03-16T11:48:38.577971+0000","last_became_peered":"2024-03-16T11:48:38.577971+0000","last_unstale":"2024-03-16T11:48:39.560755+0000","last_undegraded":"2024-03-16T11:48:39.560755+0000","last_fullsized":"2024-03-16T11:48:39.560755+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:04:34.259456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"85'16","reported_seq":37,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T11:48:43.701456+0000","last_change":"2024-03-16T11:48:38.026013+0000","last_active":"2024-03-16T11:48:43.701456+0000","last_peered":"2024-03-16T11:48:43.701456+0000","last_clean":"2024-03-16T11:48:43.701456+0000","last_became_active":"2024-03-16T11:48:38.025680+0000","last_became_peered":"2024-03-16T11:48:38.025680+0000","last_unstale":"2024-03-16T11:48:43.701456+0000","last_undegraded":"2024-03-16T11:48:43.701456+0000","last_fullsized":"2024-03-16T11:48:43.701456+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_clean_scrub_stamp":"2024-03-16T11:48:36.997377+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:02:02.375157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25510,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":17,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-16T11:48:38.577620+0000","last_change":"2024-03-16T11:48:38.574404+0000","last_active":"2024-03-16T11:48:38.577620+0000","last_peered":"2024-03-16T11:48:38.577620+0000","last_clean":"2024-03-16T11:48:38.577620+0000","last_became_active":"2024-03-16T11:48:38.572334+0000","last_became_peered":"2024-03-16T11:48:38.572334+0000","last_unstale":"2024-03-16T11:48:38.577620+0000","last_undegraded":"2024-03-16T11:48:38.577620+0000","last_fullsized":"2024-03-16T11:48:38.577620+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:47:04.912409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"85'4","reported_seq":19,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T11:48:41.047418+0000","last_change":"2024-03-16T11:48:38.025287+0000","last_active":"2024-03-16T11:48:41.047418+0000","last_peered":"2024-03-16T11:48:41.047418+0000","last_clean":"2024-03-16T11:48:41.047418+0000","last_became_active":"2024-03-16T11:48:38.022433+0000","last_became_peered":"2024-03-16T11:48:38.022433+0000","last_unstale":"2024-03-16T11:48:41.047418+0000","last_undegraded":"2024-03-16T11:48:41.047418+0000","last_fullsized":"2024-03-16T11:48:41.047418+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_clean_scrub_stamp":"2024-03-16T11:48:36.997377+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:27:20.573925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T11:48:39.557388+0000","last_change":"2024-03-16T11:48:38.574715+0000","last_active":"2024-03-16T11:48:39.557388+0000","last_peered":"2024-03-16T11:48:39.557388+0000","last_clean":"2024-03-16T11:48:39.557388+0000","last_became_active":"2024-03-16T11:48:38.574456+0000","last_became_peered":"2024-03-16T11:48:38.574456+0000","last_unstale":"2024-03-16T11:48:39.557388+0000","last_undegraded":"2024-03-16T11:48:39.557388+0000","last_fullsized":"2024-03-16T11:48:39.557388+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:57:00.123583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"85'3","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T11:48:41.048362+0000","last_change":"2024-03-16T11:48:38.029557+0000","last_active":"2024-03-16T11:48:41.048362+0000","last_peered":"2024-03-16T11:48:41.048362+0000","last_clean":"2024-03-16T11:48:41.048362+0000","last_became_active":"2024-03-16T11:48:38.029249+0000","last_became_peered":"2024-03-16T11:48:38.029249+0000","last_unstale":"2024-03-16T11:48:41.048362+0000","last_undegraded":"2024-03-16T11:48:41.048362+0000","last_fullsized":"2024-03-16T11:48:41.048362+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_clean_scrub_stamp":"2024-03-16T11:48:36.997377+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:32:41.364467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T11:48:41.045083+0000","last_change":"2024-03-16T11:48:38.575579+0000","last_active":"2024-03-16T11:48:41.045083+0000","last_peered":"2024-03-16T11:48:41.045083+0000","last_clean":"2024-03-16T11:48:41.045083+0000","last_became_active":"2024-03-16T11:48:38.575263+0000","last_became_peered":"2024-03-16T11:48:38.575263+0000","last_unstale":"2024-03-16T11:48:41.045083+0000","last_undegraded":"2024-03-16T11:48:41.045083+0000","last_fullsized":"2024-03-16T11:48:41.045083+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:59:38.983419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"85'3","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T11:48:41.972993+0000","last_change":"2024-03-16T11:48:38.027047+0000","last_active":"2024-03-16T11:48:41.972993+0000","last_peered":"2024-03-16T11:48:41.972993+0000","last_clean":"2024-03-16T11:48:41.972993+0000","last_became_active":"2024-03-16T11:48:38.026733+0000","last_became_peered":"2024-03-16T11:48:38.026733+0000","last_unstale":"2024-03-16T11:48:41.972993+0000","last_undegraded":"2024-03-16T11:48:41.972993+0000","last_fullsized":"2024-03-16T11:48:41.972993+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_clean_scrub_stamp":"2024-03-16T11:48:36.997377+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:27:23.763104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T11:48:41.043421+0000","last_change":"2024-03-16T11:48:38.568802+0000","last_active":"2024-03-16T11:48:41.043421+0000","last_peered":"2024-03-16T11:48:41.043421+0000","last_clean":"2024-03-16T11:48:41.043421+0000","last_became_active":"2024-03-16T11:48:38.568213+0000","last_became_peered":"2024-03-16T11:48:38.568213+0000","last_unstale":"2024-03-16T11:48:41.043421+0000","last_undegraded":"2024-03-16T11:48:41.043421+0000","last_fullsized":"2024-03-16T11:48:41.043421+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:38:23.233563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"84'8","reported_seq":21,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-16T11:48:39.021485+0000","last_change":"2024-03-16T11:48:38.027596+0000","last_active":"2024-03-16T11:48:39.021485+0000","last_peered":"2024-03-16T11:48:39.021485+0000","last_clean":"2024-03-16T11:48:39.021485+0000","last_became_active":"2024-03-16T11:48:38.027280+0000","last_became_peered":"2024-03-16T11:48:38.027280+0000","last_unstale":"2024-03-16T11:48:39.021485+0000","last_undegraded":"2024-03-16T11:48:39.021485+0000","last_fullsized":"2024-03-16T11:48:39.021485+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_clean_scrub_stamp":"2024-03-16T11:48:36.997377+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:05:26.409613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T11:48:41.044594+0000","last_change":"2024-03-16T11:48:38.571308+0000","last_active":"2024-03-16T11:48:41.044594+0000","last_peered":"2024-03-16T11:48:41.044594+0000","last_clean":"2024-03-16T11:48:41.044594+0000","last_became_active":"2024-03-16T11:48:38.570927+0000","last_became_peered":"2024-03-16T11:48:38.570927+0000","last_unstale":"2024-03-16T11:48:41.044594+0000","last_undegraded":"2024-03-16T11:48:41.044594+0000","last_fullsized":"2024-03-16T11:48:41.044594+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:15:18.190361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"84'1","reported_seq":14,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-16T11:48:39.019684+0000","last_change":"2024-03-16T11:48:38.024783+0000","last_active":"2024-03-16T11:48:39.019684+0000","last_peered":"2024-03-16T11:48:39.019684+0000","last_clean":"2024-03-16T11:48:39.019684+0000","last_became_active":"2024-03-16T11:48:38.023990+0000","last_became_peered":"2024-03-16T11:48:38.023990+0000","last_unstale":"2024-03-16T11:48:39.019684+0000","last_undegraded":"2024-03-16T11:48:39.019684+0000","last_fullsized":"2024-03-16T11:48:39.019684+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_clean_scrub_stamp":"2024-03-16T11:48:36.997377+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:42:49.375788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T11:48:39.942341+0000","last_change":"2024-03-16T11:48:38.580019+0000","last_active":"2024-03-16T11:48:39.942341+0000","last_peered":"2024-03-16T11:48:39.942341+0000","last_clean":"2024-03-16T11:48:39.942341+0000","last_became_active":"2024-03-16T11:48:38.579686+0000","last_became_peered":"2024-03-16T11:48:38.579686+0000","last_unstale":"2024-03-16T11:48:39.942341+0000","last_undegraded":"2024-03-16T11:48:39.942341+0000","last_fullsized":"2024-03-16T11:48:39.942341+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:00:21.752048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T11:48:40.507110+0000","last_change":"2024-03-16T11:48:38.017631+0000","last_active":"2024-03-16T11:48:40.507110+0000","last_peered":"2024-03-16T11:48:40.507110+0000","last_clean":"2024-03-16T11:48:40.507110+0000","last_became_active":"2024-03-16T11:48:38.017218+0000","last_became_peered":"2024-03-16T11:48:38.017218+0000","last_unstale":"2024-03-16T11:48:40.507110+0000","last_undegraded":"2024-03-16T11:48:40.507110+0000","last_fullsized":"2024-03-16T11:48:40.507110+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_clean_scrub_stamp":"2024-03-16T11:48:36.997377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:31:41.378757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"85'1","reported_seq":14,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T11:48:39.811681+0000","last_change":"2024-03-16T11:48:38.576751+0000","last_active":"2024-03-16T11:48:39.811681+0000","last_peered":"2024-03-16T11:48:39.811681+0000","last_clean":"2024-03-16T11:48:39.811681+0000","last_became_active":"2024-03-16T11:48:38.576442+0000","last_became_peered":"2024-03-16T11:48:38.576442+0000","last_unstale":"2024-03-16T11:48:39.811681+0000","last_undegraded":"2024-03-16T11:48:39.811681+0000","last_fullsized":"2024-03-16T11:48:39.811681+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:49:59.475427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"85'1","reported_seq":16,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T11:48:41.044303+0000","last_change":"2024-03-16T11:48:38.030159+0000","last_active":"2024-03-16T11:48:41.044303+0000","last_peered":"2024-03-16T11:48:41.044303+0000","last_clean":"2024-03-16T11:48:41.044303+0000","last_became_active":"2024-03-16T11:48:38.029808+0000","last_became_peered":"2024-03-16T11:48:38.029808+0000","last_unstale":"2024-03-16T11:48:41.044303+0000","last_undegraded":"2024-03-16T11:48:41.044303+0000","last_fullsized":"2024-03-16T11:48:41.044303+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_clean_scrub_stamp":"2024-03-16T11:48:36.997377+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:46:54.692837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T11:48:39.811447+0000","last_change":"2024-03-16T11:48:38.574080+0000","last_active":"2024-03-16T11:48:39.811447+0000","last_peered":"2024-03-16T11:48:39.811447+0000","last_clean":"2024-03-16T11:48:39.811447+0000","last_became_active":"2024-03-16T11:48:38.573745+0000","last_became_peered":"2024-03-16T11:48:38.573745+0000","last_unstale":"2024-03-16T11:48:39.811447+0000","last_undegraded":"2024-03-16T11:48:39.811447+0000","last_fullsized":"2024-03-16T11:48:39.811447+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:12:56.154804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T11:48:41.044232+0000","last_change":"2024-03-16T11:48:38.571750+0000","last_active":"2024-03-16T11:48:41.044232+0000","last_peered":"2024-03-16T11:48:41.044232+0000","last_clean":"2024-03-16T11:48:41.044232+0000","last_became_active":"2024-03-16T11:48:38.571273+0000","last_became_peered":"2024-03-16T11:48:38.571273+0000","last_unstale":"2024-03-16T11:48:41.044232+0000","last_undegraded":"2024-03-16T11:48:41.044232+0000","last_fullsized":"2024-03-16T11:48:41.044232+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:59:21.952927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"85'4","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T11:48:41.977354+0000","last_change":"2024-03-16T11:48:38.034121+0000","last_active":"2024-03-16T11:48:41.977354+0000","last_peered":"2024-03-16T11:48:41.977354+0000","last_clean":"2024-03-16T11:48:41.977354+0000","last_became_active":"2024-03-16T11:48:38.033777+0000","last_became_peered":"2024-03-16T11:48:38.033777+0000","last_unstale":"2024-03-16T11:48:41.977354+0000","last_undegraded":"2024-03-16T11:48:41.977354+0000","last_fullsized":"2024-03-16T11:48:41.977354+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_clean_scrub_stamp":"2024-03-16T11:48:36.997377+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:51:54.637120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T11:48:41.016427+0000","last_change":"2024-03-16T11:48:38.580050+0000","last_active":"2024-03-16T11:48:41.016427+0000","last_peered":"2024-03-16T11:48:41.016427+0000","last_clean":"2024-03-16T11:48:41.016427+0000","last_became_active":"2024-03-16T11:48:38.578950+0000","last_became_peered":"2024-03-16T11:48:38.578950+0000","last_unstale":"2024-03-16T11:48:41.016427+0000","last_undegraded":"2024-03-16T11:48:41.016427+0000","last_fullsized":"2024-03-16T11:48:41.016427+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:10:21.717916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"85'3","reported_seq":19,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T11:48:41.964761+0000","last_change":"2024-03-16T11:48:38.023925+0000","last_active":"2024-03-16T11:48:41.964761+0000","last_peered":"2024-03-16T11:48:41.964761+0000","last_clean":"2024-03-16T11:48:41.964761+0000","last_became_active":"2024-03-16T11:48:38.023424+0000","last_became_peered":"2024-03-16T11:48:38.023424+0000","last_unstale":"2024-03-16T11:48:41.964761+0000","last_undegraded":"2024-03-16T11:48:41.964761+0000","last_fullsized":"2024-03-16T11:48:41.964761+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_clean_scrub_stamp":"2024-03-16T11:48:36.997377+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:01:53.085039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":349,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T11:48:41.043439+0000","last_change":"2024-03-16T11:47:22.001149+0000","last_active":"2024-03-16T11:48:41.043439+0000","last_peered":"2024-03-16T11:48:41.043439+0000","last_clean":"2024-03-16T11:48:41.043439+0000","last_became_active":"2024-03-16T11:47:22.000506+0000","last_became_peered":"2024-03-16T11:47:22.000506+0000","last_unstale":"2024-03-16T11:48:41.043439+0000","last_undegraded":"2024-03-16T11:48:41.043439+0000","last_fullsized":"2024-03-16T11:48:41.043439+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:43:52.376552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:43:52.376552+0000","last_clean_scrub_stamp":"2024-03-16T11:43:52.376552+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:21:18.466389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T11:48:39.557252+0000","last_change":"2024-03-16T11:48:38.577530+0000","last_active":"2024-03-16T11:48:39.557252+0000","last_peered":"2024-03-16T11:48:39.557252+0000","last_clean":"2024-03-16T11:48:39.557252+0000","last_became_active":"2024-03-16T11:48:38.577252+0000","last_became_peered":"2024-03-16T11:48:38.577252+0000","last_unstale":"2024-03-16T11:48:39.557252+0000","last_undegraded":"2024-03-16T11:48:39.557252+0000","last_fullsized":"2024-03-16T11:48:39.557252+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:07:47.809075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"85'3","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T11:48:41.973618+0000","last_change":"2024-03-16T11:48:38.027630+0000","last_active":"2024-03-16T11:48:41.973618+0000","last_peered":"2024-03-16T11:48:41.973618+0000","last_clean":"2024-03-16T11:48:41.973618+0000","last_became_active":"2024-03-16T11:48:38.027105+0000","last_became_peered":"2024-03-16T11:48:38.027105+0000","last_unstale":"2024-03-16T11:48:41.973618+0000","last_undegraded":"2024-03-16T11:48:41.973618+0000","last_fullsized":"2024-03-16T11:48:41.973618+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_clean_scrub_stamp":"2024-03-16T11:48:36.997377+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:24:20.605988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":13,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-16T11:48:38.564200+0000","last_change":"2024-03-16T11:48:38.032419+0000","last_active":"2024-03-16T11:48:38.564200+0000","last_peered":"2024-03-16T11:48:38.564200+0000","last_clean":"2024-03-16T11:48:38.564200+0000","last_became_active":"2024-03-16T11:48:38.032087+0000","last_became_peered":"2024-03-16T11:48:38.032087+0000","last_unstale":"2024-03-16T11:48:38.564200+0000","last_undegraded":"2024-03-16T11:48:38.564200+0000","last_fullsized":"2024-03-16T11:48:38.564200+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_clean_scrub_stamp":"2024-03-16T11:48:36.997377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:18:54.459321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T11:48:39.557576+0000","last_change":"2024-03-16T11:48:38.568768+0000","last_active":"2024-03-16T11:48:39.557576+0000","last_peered":"2024-03-16T11:48:39.557576+0000","last_clean":"2024-03-16T11:48:39.557576+0000","last_became_active":"2024-03-16T11:48:38.568408+0000","last_became_peered":"2024-03-16T11:48:38.568408+0000","last_unstale":"2024-03-16T11:48:39.557576+0000","last_undegraded":"2024-03-16T11:48:39.557576+0000","last_fullsized":"2024-03-16T11:48:39.557576+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:10:33.293464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"85'3","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T11:48:41.977856+0000","last_change":"2024-03-16T11:48:38.023387+0000","last_active":"2024-03-16T11:48:41.977856+0000","last_peered":"2024-03-16T11:48:41.977856+0000","last_clean":"2024-03-16T11:48:41.977856+0000","last_became_active":"2024-03-16T11:48:38.022916+0000","last_became_peered":"2024-03-16T11:48:38.022916+0000","last_unstale":"2024-03-16T11:48:41.977856+0000","last_undegraded":"2024-03-16T11:48:41.977856+0000","last_fullsized":"2024-03-16T11:48:41.977856+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_clean_scrub_stamp":"2024-03-16T11:48:36.997377+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:25:41.478737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T11:48:41.016406+0000","last_change":"2024-03-16T11:48:38.572243+0000","last_active":"2024-03-16T11:48:41.016406+0000","last_peered":"2024-03-16T11:48:41.016406+0000","last_clean":"2024-03-16T11:48:41.016406+0000","last_became_active":"2024-03-16T11:48:38.571918+0000","last_became_peered":"2024-03-16T11:48:38.571918+0000","last_unstale":"2024-03-16T11:48:41.016406+0000","last_undegraded":"2024-03-16T11:48:41.016406+0000","last_fullsized":"2024-03-16T11:48:41.016406+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:21:45.843298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"85'4","reported_seq":19,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T11:48:41.047711+0000","last_change":"2024-03-16T11:48:38.022825+0000","last_active":"2024-03-16T11:48:41.047711+0000","last_peered":"2024-03-16T11:48:41.047711+0000","last_clean":"2024-03-16T11:48:41.047711+0000","last_became_active":"2024-03-16T11:48:38.022499+0000","last_became_peered":"2024-03-16T11:48:38.022499+0000","last_unstale":"2024-03-16T11:48:41.047711+0000","last_undegraded":"2024-03-16T11:48:41.047711+0000","last_fullsized":"2024-03-16T11:48:41.047711+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_clean_scrub_stamp":"2024-03-16T11:48:36.997377+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:52:51.810960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T11:48:41.044367+0000","last_change":"2024-03-16T11:48:38.577473+0000","last_active":"2024-03-16T11:48:41.044367+0000","last_peered":"2024-03-16T11:48:41.044367+0000","last_clean":"2024-03-16T11:48:41.044367+0000","last_became_active":"2024-03-16T11:48:38.577121+0000","last_became_peered":"2024-03-16T11:48:38.577121+0000","last_unstale":"2024-03-16T11:48:41.044367+0000","last_undegraded":"2024-03-16T11:48:41.044367+0000","last_fullsized":"2024-03-16T11:48:41.044367+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:21:06.905785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"85'2","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T11:48:41.046675+0000","last_change":"2024-03-16T11:48:38.020324+0000","last_active":"2024-03-16T11:48:41.046675+0000","last_peered":"2024-03-16T11:48:41.046675+0000","last_clean":"2024-03-16T11:48:41.046675+0000","last_became_active":"2024-03-16T11:48:38.019692+0000","last_became_peered":"2024-03-16T11:48:38.019692+0000","last_unstale":"2024-03-16T11:48:41.046675+0000","last_undegraded":"2024-03-16T11:48:41.046675+0000","last_fullsized":"2024-03-16T11:48:41.046675+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_clean_scrub_stamp":"2024-03-16T11:48:36.997377+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:42:51.709046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-16T11:48:38.573748+0000","last_change":"2024-03-16T11:48:38.570911+0000","last_active":"2024-03-16T11:48:38.573748+0000","last_peered":"2024-03-16T11:48:38.573748+0000","last_clean":"2024-03-16T11:48:38.573748+0000","last_became_active":"2024-03-16T11:48:38.570288+0000","last_became_peered":"2024-03-16T11:48:38.570288+0000","last_unstale":"2024-03-16T11:48:38.573748+0000","last_undegraded":"2024-03-16T11:48:38.573748+0000","last_fullsized":"2024-03-16T11:48:38.573748+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:27:21.274856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"85'3","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T11:48:41.046519+0000","last_change":"2024-03-16T11:48:38.021817+0000","last_active":"2024-03-16T11:48:41.046519+0000","last_peered":"2024-03-16T11:48:41.046519+0000","last_clean":"2024-03-16T11:48:41.046519+0000","last_became_active":"2024-03-16T11:48:38.020906+0000","last_became_peered":"2024-03-16T11:48:38.020906+0000","last_unstale":"2024-03-16T11:48:41.046519+0000","last_undegraded":"2024-03-16T11:48:41.046519+0000","last_fullsized":"2024-03-16T11:48:41.046519+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_clean_scrub_stamp":"2024-03-16T11:48:36.997377+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:09:53.372698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T11:48:39.942085+0000","last_change":"2024-03-16T11:48:38.573536+0000","last_active":"2024-03-16T11:48:39.942085+0000","last_peered":"2024-03-16T11:48:39.942085+0000","last_clean":"2024-03-16T11:48:39.942085+0000","last_became_active":"2024-03-16T11:48:38.573211+0000","last_became_peered":"2024-03-16T11:48:38.573211+0000","last_unstale":"2024-03-16T11:48:39.942085+0000","last_undegraded":"2024-03-16T11:48:39.942085+0000","last_fullsized":"2024-03-16T11:48:39.942085+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:53:02.607175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":15,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T11:48:39.558004+0000","last_change":"2024-03-16T11:48:38.022830+0000","last_active":"2024-03-16T11:48:39.558004+0000","last_peered":"2024-03-16T11:48:39.558004+0000","last_clean":"2024-03-16T11:48:39.558004+0000","last_became_active":"2024-03-16T11:48:38.021159+0000","last_became_peered":"2024-03-16T11:48:38.021159+0000","last_unstale":"2024-03-16T11:48:39.558004+0000","last_undegraded":"2024-03-16T11:48:39.558004+0000","last_fullsized":"2024-03-16T11:48:39.558004+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_clean_scrub_stamp":"2024-03-16T11:48:36.997377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:06:32.910622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T11:48:41.016373+0000","last_change":"2024-03-16T11:48:38.575965+0000","last_active":"2024-03-16T11:48:41.016373+0000","last_peered":"2024-03-16T11:48:41.016373+0000","last_clean":"2024-03-16T11:48:41.016373+0000","last_became_active":"2024-03-16T11:48:38.575627+0000","last_became_peered":"2024-03-16T11:48:38.575627+0000","last_unstale":"2024-03-16T11:48:41.016373+0000","last_undegraded":"2024-03-16T11:48:41.016373+0000","last_fullsized":"2024-03-16T11:48:41.016373+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:00:37.768897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"84'2","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-16T11:48:39.020331+0000","last_change":"2024-03-16T11:48:38.027516+0000","last_active":"2024-03-16T11:48:39.020331+0000","last_peered":"2024-03-16T11:48:39.020331+0000","last_clean":"2024-03-16T11:48:39.020331+0000","last_became_active":"2024-03-16T11:48:38.026865+0000","last_became_peered":"2024-03-16T11:48:38.026865+0000","last_unstale":"2024-03-16T11:48:39.020331+0000","last_undegraded":"2024-03-16T11:48:39.020331+0000","last_fullsized":"2024-03-16T11:48:39.020331+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_clean_scrub_stamp":"2024-03-16T11:48:36.997377+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:50:50.240973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T11:48:41.044384+0000","last_change":"2024-03-16T11:48:38.571242+0000","last_active":"2024-03-16T11:48:41.044384+0000","last_peered":"2024-03-16T11:48:41.044384+0000","last_clean":"2024-03-16T11:48:41.044384+0000","last_became_active":"2024-03-16T11:48:38.570917+0000","last_became_peered":"2024-03-16T11:48:38.570917+0000","last_unstale":"2024-03-16T11:48:41.044384+0000","last_undegraded":"2024-03-16T11:48:41.044384+0000","last_fullsized":"2024-03-16T11:48:41.044384+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:30:12.601982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":15,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T11:48:39.558335+0000","last_change":"2024-03-16T11:48:38.023970+0000","last_active":"2024-03-16T11:48:39.558335+0000","last_peered":"2024-03-16T11:48:39.558335+0000","last_clean":"2024-03-16T11:48:39.558335+0000","last_became_active":"2024-03-16T11:48:38.023646+0000","last_became_peered":"2024-03-16T11:48:38.023646+0000","last_unstale":"2024-03-16T11:48:39.558335+0000","last_undegraded":"2024-03-16T11:48:39.558335+0000","last_fullsized":"2024-03-16T11:48:39.558335+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_clean_scrub_stamp":"2024-03-16T11:48:36.997377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:48:35.549588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T11:48:41.044754+0000","last_change":"2024-03-16T11:48:38.567376+0000","last_active":"2024-03-16T11:48:41.044754+0000","last_peered":"2024-03-16T11:48:41.044754+0000","last_clean":"2024-03-16T11:48:41.044754+0000","last_became_active":"2024-03-16T11:48:38.567048+0000","last_became_peered":"2024-03-16T11:48:38.567048+0000","last_unstale":"2024-03-16T11:48:41.044754+0000","last_undegraded":"2024-03-16T11:48:41.044754+0000","last_fullsized":"2024-03-16T11:48:41.044754+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:45:50.097013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"84'1","reported_seq":16,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T11:48:41.045756+0000","last_change":"2024-03-16T11:48:38.029192+0000","last_active":"2024-03-16T11:48:41.045756+0000","last_peered":"2024-03-16T11:48:41.045756+0000","last_clean":"2024-03-16T11:48:41.045756+0000","last_became_active":"2024-03-16T11:48:38.028866+0000","last_became_peered":"2024-03-16T11:48:38.028866+0000","last_unstale":"2024-03-16T11:48:41.045756+0000","last_undegraded":"2024-03-16T11:48:41.045756+0000","last_fullsized":"2024-03-16T11:48:41.045756+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_clean_scrub_stamp":"2024-03-16T11:48:36.997377+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:52:31.329863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T11:48:41.043999+0000","last_change":"2024-03-16T11:48:38.576538+0000","last_active":"2024-03-16T11:48:41.043999+0000","last_peered":"2024-03-16T11:48:41.043999+0000","last_clean":"2024-03-16T11:48:41.043999+0000","last_became_active":"2024-03-16T11:48:38.576222+0000","last_became_peered":"2024-03-16T11:48:38.576222+0000","last_unstale":"2024-03-16T11:48:41.043999+0000","last_undegraded":"2024-03-16T11:48:41.043999+0000","last_fullsized":"2024-03-16T11:48:41.043999+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:34:13.186406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T11:48:39.557621+0000","last_change":"2024-03-16T11:48:38.031002+0000","last_active":"2024-03-16T11:48:39.557621+0000","last_peered":"2024-03-16T11:48:39.557621+0000","last_clean":"2024-03-16T11:48:39.557621+0000","last_became_active":"2024-03-16T11:48:38.030653+0000","last_became_peered":"2024-03-16T11:48:38.030653+0000","last_unstale":"2024-03-16T11:48:39.557621+0000","last_undegraded":"2024-03-16T11:48:39.557621+0000","last_fullsized":"2024-03-16T11:48:39.557621+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_clean_scrub_stamp":"2024-03-16T11:48:36.997377+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:42:30.148594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T11:48:41.044271+0000","last_change":"2024-03-16T11:48:38.571942+0000","last_active":"2024-03-16T11:48:41.044271+0000","last_peered":"2024-03-16T11:48:41.044271+0000","last_clean":"2024-03-16T11:48:41.044271+0000","last_became_active":"2024-03-16T11:48:38.571606+0000","last_became_peered":"2024-03-16T11:48:38.571606+0000","last_unstale":"2024-03-16T11:48:41.044271+0000","last_undegraded":"2024-03-16T11:48:41.044271+0000","last_fullsized":"2024-03-16T11:48:41.044271+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:29:13.797054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T11:48:39.942194+0000","last_change":"2024-03-16T11:48:38.573469+0000","last_active":"2024-03-16T11:48:39.942194+0000","last_peered":"2024-03-16T11:48:39.942194+0000","last_clean":"2024-03-16T11:48:39.942194+0000","last_became_active":"2024-03-16T11:48:38.573155+0000","last_became_peered":"2024-03-16T11:48:38.573155+0000","last_unstale":"2024-03-16T11:48:39.942194+0000","last_undegraded":"2024-03-16T11:48:39.942194+0000","last_fullsized":"2024-03-16T11:48:39.942194+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:28:50.180821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T11:48:39.558166+0000","last_change":"2024-03-16T11:48:38.580913+0000","last_active":"2024-03-16T11:48:39.558166+0000","last_peered":"2024-03-16T11:48:39.558166+0000","last_clean":"2024-03-16T11:48:39.558166+0000","last_became_active":"2024-03-16T11:48:38.580573+0000","last_became_peered":"2024-03-16T11:48:38.580573+0000","last_unstale":"2024-03-16T11:48:39.558166+0000","last_undegraded":"2024-03-16T11:48:39.558166+0000","last_fullsized":"2024-03-16T11:48:39.558166+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:44:58.080502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T11:48:39.558089+0000","last_change":"2024-03-16T11:48:38.577168+0000","last_active":"2024-03-16T11:48:39.558089+0000","last_peered":"2024-03-16T11:48:39.558089+0000","last_clean":"2024-03-16T11:48:39.558089+0000","last_became_active":"2024-03-16T11:48:38.576845+0000","last_became_peered":"2024-03-16T11:48:38.576845+0000","last_unstale":"2024-03-16T11:48:39.558089+0000","last_undegraded":"2024-03-16T11:48:39.558089+0000","last_fullsized":"2024-03-16T11:48:39.558089+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:13:02.145610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T11:48:41.044247+0000","last_change":"2024-03-16T11:48:38.577477+0000","last_active":"2024-03-16T11:48:41.044247+0000","last_peered":"2024-03-16T11:48:41.044247+0000","last_clean":"2024-03-16T11:48:41.044247+0000","last_became_active":"2024-03-16T11:48:38.577126+0000","last_became_peered":"2024-03-16T11:48:38.577126+0000","last_unstale":"2024-03-16T11:48:41.044247+0000","last_undegraded":"2024-03-16T11:48:41.044247+0000","last_fullsized":"2024-03-16T11:48:41.044247+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:45:27.246827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T11:48:39.557457+0000","last_change":"2024-03-16T11:48:38.578508+0000","last_active":"2024-03-16T11:48:39.557457+0000","last_peered":"2024-03-16T11:48:39.557457+0000","last_clean":"2024-03-16T11:48:39.557457+0000","last_became_active":"2024-03-16T11:48:38.578125+0000","last_became_peered":"2024-03-16T11:48:38.578125+0000","last_unstale":"2024-03-16T11:48:39.557457+0000","last_undegraded":"2024-03-16T11:48:39.557457+0000","last_fullsized":"2024-03-16T11:48:39.557457+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:13:36.996405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T11:48:39.942338+0000","last_change":"2024-03-16T11:48:38.570450+0000","last_active":"2024-03-16T11:48:39.942338+0000","last_peered":"2024-03-16T11:48:39.942338+0000","last_clean":"2024-03-16T11:48:39.942338+0000","last_became_active":"2024-03-16T11:48:38.569952+0000","last_became_peered":"2024-03-16T11:48:38.569952+0000","last_unstale":"2024-03-16T11:48:39.942338+0000","last_undegraded":"2024-03-16T11:48:39.942338+0000","last_fullsized":"2024-03-16T11:48:39.942338+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:34:39.265969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T11:48:39.558576+0000","last_change":"2024-03-16T11:48:38.569568+0000","last_active":"2024-03-16T11:48:39.558576+0000","last_peered":"2024-03-16T11:48:39.558576+0000","last_clean":"2024-03-16T11:48:39.558576+0000","last_became_active":"2024-03-16T11:48:38.569260+0000","last_became_peered":"2024-03-16T11:48:38.569260+0000","last_unstale":"2024-03-16T11:48:39.558576+0000","last_undegraded":"2024-03-16T11:48:39.558576+0000","last_fullsized":"2024-03-16T11:48:39.558576+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:52:52.315980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T11:48:41.016231+0000","last_change":"2024-03-16T11:48:38.584562+0000","last_active":"2024-03-16T11:48:41.016231+0000","last_peered":"2024-03-16T11:48:41.016231+0000","last_clean":"2024-03-16T11:48:41.016231+0000","last_became_active":"2024-03-16T11:48:38.584096+0000","last_became_peered":"2024-03-16T11:48:38.584096+0000","last_unstale":"2024-03-16T11:48:41.016231+0000","last_undegraded":"2024-03-16T11:48:41.016231+0000","last_fullsized":"2024-03-16T11:48:41.016231+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:43:45.933252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T11:48:41.044218+0000","last_change":"2024-03-16T11:48:38.570174+0000","last_active":"2024-03-16T11:48:41.044218+0000","last_peered":"2024-03-16T11:48:41.044218+0000","last_clean":"2024-03-16T11:48:41.044218+0000","last_became_active":"2024-03-16T11:48:38.569865+0000","last_became_peered":"2024-03-16T11:48:38.569865+0000","last_unstale":"2024-03-16T11:48:41.044218+0000","last_undegraded":"2024-03-16T11:48:41.044218+0000","last_fullsized":"2024-03-16T11:48:41.044218+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T11:53:01.012481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T11:48:41.016256+0000","last_change":"2024-03-16T11:48:38.574868+0000","last_active":"2024-03-16T11:48:41.016256+0000","last_peered":"2024-03-16T11:48:41.016256+0000","last_clean":"2024-03-16T11:48:41.016256+0000","last_became_active":"2024-03-16T11:48:38.574434+0000","last_became_peered":"2024-03-16T11:48:38.574434+0000","last_unstale":"2024-03-16T11:48:41.016256+0000","last_undegraded":"2024-03-16T11:48:41.016256+0000","last_fullsized":"2024-03-16T11:48:41.016256+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:09:57.102253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T11:48:39.560883+0000","last_change":"2024-03-16T11:48:38.577189+0000","last_active":"2024-03-16T11:48:39.560883+0000","last_peered":"2024-03-16T11:48:39.560883+0000","last_clean":"2024-03-16T11:48:39.560883+0000","last_became_active":"2024-03-16T11:48:38.576884+0000","last_became_peered":"2024-03-16T11:48:38.576884+0000","last_unstale":"2024-03-16T11:48:39.560883+0000","last_undegraded":"2024-03-16T11:48:39.560883+0000","last_fullsized":"2024-03-16T11:48:39.560883+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:11:51.558517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T11:48:39.557803+0000","last_change":"2024-03-16T11:48:38.576238+0000","last_active":"2024-03-16T11:48:39.557803+0000","last_peered":"2024-03-16T11:48:39.557803+0000","last_clean":"2024-03-16T11:48:39.557803+0000","last_became_active":"2024-03-16T11:48:38.575951+0000","last_became_peered":"2024-03-16T11:48:38.575951+0000","last_unstale":"2024-03-16T11:48:39.557803+0000","last_undegraded":"2024-03-16T11:48:39.557803+0000","last_fullsized":"2024-03-16T11:48:39.557803+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:47:32.571580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T11:48:39.942117+0000","last_change":"2024-03-16T11:48:38.580025+0000","last_active":"2024-03-16T11:48:39.942117+0000","last_peered":"2024-03-16T11:48:39.942117+0000","last_clean":"2024-03-16T11:48:39.942117+0000","last_became_active":"2024-03-16T11:48:38.579689+0000","last_became_peered":"2024-03-16T11:48:38.579689+0000","last_unstale":"2024-03-16T11:48:39.942117+0000","last_undegraded":"2024-03-16T11:48:39.942117+0000","last_fullsized":"2024-03-16T11:48:39.942117+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:07:50.418306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":284036,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":28404,"num_objects":48,"num_object_clones":0,"num_object_copies":144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":0,"num_read":13,"num_read_kb":11,"num_write":110,"num_write_kb":54,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":32,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":241664,"data_stored":39819,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383695,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32072,"kb_used_data":896,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713080,"statfs":{"total":95995035648,"available":95962193920,"internally_reserved":0,"allocated":917504,"data_stored":439103,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645331,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28004,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717148,"statfs":{"total":95995035648,"available":95966359552,"internally_reserved":0,"allocated":1007616,"data_stored":1032079,"data_compressed":36118,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808857,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27932,"kb_used_data":912,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717220,"statfs":{"total":95995035648,"available":95966433280,"internally_reserved":0,"allocated":933888,"data_stored":439339,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070494,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32072,"kb_used_data":888,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713080,"statfs":{"total":95995035648,"available":95962193920,"internally_reserved":0,"allocated":909312,"data_stored":430555,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332131,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27972,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717180,"statfs":{"total":95995035648,"available":95966392320,"internally_reserved":0,"allocated":978944,"data_stored":1031072,"data_compressed":36118,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495657,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36232,"kb_used_data":896,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708920,"statfs":{"total":95995035648,"available":95957934080,"internally_reserved":0,"allocated":917504,"data_stored":438611,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574914,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32080,"kb_used_data":904,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713072,"statfs":{"total":95995035648,"available":95962185728,"internally_reserved":0,"allocated":925696,"data_stored":440133,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738439,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27940,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717212,"statfs":{"total":95995035648,"available":95966425088,"internally_reserved":0,"allocated":942080,"data_stored":439999,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411389,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32100,"kb_used_data":908,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713052,"statfs":{"total":95995035648,"available":95962165248,"internally_reserved":0,"allocated":929792,"data_stored":440167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182455,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36244,"kb_used_data":904,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708908,"statfs":{"total":95995035648,"available":95957921792,"internally_reserved":0,"allocated":925696,"data_stored":438715,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920819,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27916,"kb_used_data":896,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717236,"statfs":{"total":95995035648,"available":95966449664,"internally_reserved":0,"allocated":917504,"data_stored":438543,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757293,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32128,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713024,"statfs":{"total":95995035648,"available":95962136576,"internally_reserved":0,"allocated":962560,"data_stored":1020501,"data_compressed":36118,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34482,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34482,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34482,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36118,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36118,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36118,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36118,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36118,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2268,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":27479,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-16T11:48:45.763 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-16T11:48:45.904 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:45.903+0000 7fd0b06d1640 1 -- 172.21.15.29:0/3837868659 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd0a8100700 msgr2=0x7fd0a8100b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:48:45.904 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:45.903+0000 7fd0b06d1640 1 --2- 172.21.15.29:0/3837868659 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd0a8100700 0x7fd0a8100b20 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fd0a4009900 tx=0x7fd0a402bc30 comp rx=0 tx=0).stop 2024-03-16T11:48:45.905 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:45.904+0000 7fd0b06d1640 1 -- 172.21.15.29:0/3837868659 shutdown_connections 2024-03-16T11:48:45.905 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:45.904+0000 7fd0b06d1640 1 --2- 172.21.15.29:0/3837868659 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd0a81022b0 0x7fd0a8108b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:45.905 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:45.904+0000 7fd0b06d1640 1 --2- 172.21.15.29:0/3837868659 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd0a81018f0 0x7fd0a8101d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:45.905 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:45.904+0000 7fd0b06d1640 1 --2- 172.21.15.29:0/3837868659 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd0a8100700 0x7fd0a8100b20 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:45.905 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:45.904+0000 7fd0b06d1640 1 -- 172.21.15.29:0/3837868659 >> 172.21.15.29:0/3837868659 conn(0x7fd0a80fbc70 msgr2=0x7fd0a80fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:48:45.905 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:45.904+0000 7fd0b06d1640 1 -- 172.21.15.29:0/3837868659 shutdown_connections 2024-03-16T11:48:45.905 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:45.904+0000 7fd0b06d1640 1 -- 172.21.15.29:0/3837868659 wait complete. 2024-03-16T11:48:45.905 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:45.904+0000 7fd0b06d1640 1 Processor -- start 2024-03-16T11:48:45.905 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:45.904+0000 7fd0b06d1640 1 -- start start 2024-03-16T11:48:45.905 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:45.904+0000 7fd0b06d1640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd0a8100700 0x7fd0a80694b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:48:45.905 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:45.904+0000 7fd0b06d1640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd0a81018f0 0x7fd0a819c870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:48:45.905 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:45.904+0000 7fd0b06d1640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd0a81022b0 0x7fd0a819ee90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:48:45.905 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:45.904+0000 7fd0b06d1640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fd0a810c380 con 0x7fd0a81022b0 2024-03-16T11:48:45.905 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:45.904+0000 7fd0b06d1640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fd0a810c200 con 0x7fd0a8100700 2024-03-16T11:48:45.905 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:45.904+0000 7fd0b06d1640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fd0a810c500 con 0x7fd0a81018f0 2024-03-16T11:48:45.906 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:45.904+0000 7fd0aec6e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd0a81022b0 0x7fd0a819ee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:48:45.906 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:45.904+0000 7fd0aec6e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd0a81022b0 0x7fd0a819ee90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49270/0 (socket says 172.21.15.29:49270) 2024-03-16T11:48:45.906 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:45.904+0000 7fd0aec6e640 1 -- 172.21.15.29:0/2178982848 learned_addr learned my addr 172.21.15.29:0/2178982848 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:48:45.906 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:45.904+0000 7fd0ae46d640 1 --2- 172.21.15.29:0/2178982848 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd0a8100700 0x7fd0a80694b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:48:45.906 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:45.905+0000 7fd0adc6c640 1 --2- 172.21.15.29:0/2178982848 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd0a81018f0 0x7fd0a819c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:48:45.906 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:45.905+0000 7fd0aec6e640 1 -- 172.21.15.29:0/2178982848 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd0a81018f0 msgr2=0x7fd0a819c870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:48:45.906 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:45.905+0000 7fd0aec6e640 1 --2- 172.21.15.29:0/2178982848 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd0a81018f0 0x7fd0a819c870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:45.906 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:45.905+0000 7fd0aec6e640 1 -- 172.21.15.29:0/2178982848 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd0a8100700 msgr2=0x7fd0a80694b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:48:45.906 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:45.905+0000 7fd0aec6e640 1 --2- 172.21.15.29:0/2178982848 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd0a8100700 0x7fd0a80694b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:45.906 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:45.905+0000 7fd0aec6e640 1 -- 172.21.15.29:0/2178982848 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0a819f7a0 con 0x7fd0a81022b0 2024-03-16T11:48:45.906 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:45.905+0000 7fd0aec6e640 1 --2- 172.21.15.29:0/2178982848 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd0a81022b0 0x7fd0a819ee90 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fd0a000eb10 tx=0x7fd0a000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:48:45.906 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:45.905+0000 7fd09f7fe640 1 -- 172.21.15.29:0/2178982848 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0a0004b20 con 0x7fd0a81022b0 2024-03-16T11:48:45.906 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:45.905+0000 7fd09f7fe640 1 -- 172.21.15.29:0/2178982848 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd0a0004cc0 con 0x7fd0a81022b0 2024-03-16T11:48:45.906 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:45.905+0000 7fd0b06d1640 1 -- 172.21.15.29:0/2178982848 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd0a819fa30 con 0x7fd0a81022b0 2024-03-16T11:48:45.906 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:45.905+0000 7fd09f7fe640 1 -- 172.21.15.29:0/2178982848 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0a0010760 con 0x7fd0a81022b0 2024-03-16T11:48:45.907 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:45.905+0000 7fd0b06d1640 1 -- 172.21.15.29:0/2178982848 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd0a819ff80 con 0x7fd0a81022b0 2024-03-16T11:48:45.907 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:45.906+0000 7fd0b06d1640 1 -- 172.21.15.29:0/2178982848 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd0a8107a20 con 0x7fd0a81022b0 2024-03-16T11:48:45.907 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:45.907+0000 7fd09f7fe640 1 -- 172.21.15.29:0/2178982848 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7fd0a0005020 con 0x7fd0a81022b0 2024-03-16T11:48:45.907 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:45.908+0000 7fd09f7fe640 1 --2- 172.21.15.29:0/2178982848 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fd0940a2890 0x7fd0940a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:48:45.908 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:45.908+0000 7fd0ae46d640 1 --2- 172.21.15.29:0/2178982848 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fd0940a2890 0x7fd0940a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:48:45.908 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:45.908+0000 7fd09f7fe640 1 -- 172.21.15.29:0/2178982848 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7fd0a00cf220 con 0x7fd0a81022b0 2024-03-16T11:48:45.908 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:45.908+0000 7fd0ae46d640 1 --2- 172.21.15.29:0/2178982848 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fd0940a2890 0x7fd0940a4d50 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fd0a4000c00 tx=0x7fd0a4005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:48:45.911 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:45.912+0000 7fd09f7fe640 1 -- 172.21.15.29:0/2178982848 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fd0a009c210 con 0x7fd0a81022b0 2024-03-16T11:48:46.092 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:46.092+0000 7fd0b06d1640 1 -- 172.21.15.29:0/2178982848 --> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fd0a8061c20 con 0x7fd0940a2890 2024-03-16T11:48:46.099 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:46.100+0000 7fd09f7fe640 1 -- 172.21.15.29:0/2178982848 <== mgr.14152 v2:172.21.15.29:6800/4192064640 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258890 (secure 0 0 0) 0x7fd0a8061c20 con 0x7fd0940a2890 2024-03-16T11:48:46.101 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-16T11:48:46.102 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2024-03-16T11:48:46.104 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:46.103+0000 7fd0b06d1640 1 -- 172.21.15.29:0/2178982848 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fd0940a2890 msgr2=0x7fd0940a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:48:46.104 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:46.103+0000 7fd0b06d1640 1 --2- 172.21.15.29:0/2178982848 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fd0940a2890 0x7fd0940a4d50 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fd0a4000c00 tx=0x7fd0a4005fd0 comp rx=0 tx=0).stop 2024-03-16T11:48:46.104 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:46.103+0000 7fd0b06d1640 1 -- 172.21.15.29:0/2178982848 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd0a81022b0 msgr2=0x7fd0a819ee90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:48:46.104 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:46.103+0000 7fd0b06d1640 1 --2- 172.21.15.29:0/2178982848 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd0a81022b0 0x7fd0a819ee90 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fd0a000eb10 tx=0x7fd0a000c2d0 comp rx=0 tx=0).stop 2024-03-16T11:48:46.104 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:46.103+0000 7fd0b06d1640 1 -- 172.21.15.29:0/2178982848 shutdown_connections 2024-03-16T11:48:46.104 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:46.103+0000 7fd0b06d1640 1 --2- 172.21.15.29:0/2178982848 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fd0940a2890 0x7fd0940a4d50 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:46.105 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:46.103+0000 7fd0b06d1640 1 --2- 172.21.15.29:0/2178982848 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd0a81022b0 0x7fd0a819ee90 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:46.105 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:46.103+0000 7fd0b06d1640 1 --2- 172.21.15.29:0/2178982848 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd0a81018f0 0x7fd0a819c870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:46.105 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:46.103+0000 7fd0b06d1640 1 --2- 172.21.15.29:0/2178982848 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd0a8100700 0x7fd0a80694b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:46.105 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:46.103+0000 7fd0b06d1640 1 -- 172.21.15.29:0/2178982848 >> 172.21.15.29:0/2178982848 conn(0x7fd0a80fbc70 msgr2=0x7fd0a8104bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:48:46.105 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:46.103+0000 7fd0b06d1640 1 -- 172.21.15.29:0/2178982848 shutdown_connections 2024-03-16T11:48:46.105 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:46.103+0000 7fd0b06d1640 1 -- 172.21.15.29:0/2178982848 wait complete. 2024-03-16T11:48:46.113 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":294,"stamp":"2024-03-16T11:48:45.617605+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":618889,"num_objects":51,"num_object_clones":0,"num_object_copies":153,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51,"num_whiteouts":0,"num_read":149,"num_read_kb":130,"num_write":292,"num_write_kb":2993,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":32,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":235,"ondisk_log_size":235,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":231,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":372692,"kb_used_data":11004,"kb_used_omap":18,"kb_used_meta":361517,"kb_avail":1124569132,"statfs":{"total":1151940427776,"available":1151558791168,"internally_reserved":0,"allocated":11268096,"data_stored":7028817,"data_compressed":108354,"data_compressed_allocated":135168,"data_compressed_original":1769472,"omap_allocated":19143,"internal_metadata":370193721},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":28521,"num_objects":49,"num_object_clones":0,"num_object_copies":147,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49,"num_whiteouts":0,"num_read":13,"num_read_kb":11,"num_write":111,"num_write_kb":55,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":32,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.608452"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T11:48:39.942383+0000","last_change":"2024-03-16T11:48:38.574718+0000","last_active":"2024-03-16T11:48:39.942383+0000","last_peered":"2024-03-16T11:48:39.942383+0000","last_clean":"2024-03-16T11:48:39.942383+0000","last_became_active":"2024-03-16T11:48:38.574420+0000","last_became_peered":"2024-03-16T11:48:38.574420+0000","last_unstale":"2024-03-16T11:48:39.942383+0000","last_undegraded":"2024-03-16T11:48:39.942383+0000","last_fullsized":"2024-03-16T11:48:39.942383+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:43:56.705135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T11:48:41.044196+0000","last_change":"2024-03-16T11:48:38.572722+0000","last_active":"2024-03-16T11:48:41.044196+0000","last_peered":"2024-03-16T11:48:41.044196+0000","last_clean":"2024-03-16T11:48:41.044196+0000","last_became_active":"2024-03-16T11:48:38.572431+0000","last_became_peered":"2024-03-16T11:48:38.572431+0000","last_unstale":"2024-03-16T11:48:41.044196+0000","last_undegraded":"2024-03-16T11:48:41.044196+0000","last_fullsized":"2024-03-16T11:48:41.044196+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:34:45.070975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T11:48:41.044515+0000","last_change":"2024-03-16T11:48:38.585504+0000","last_active":"2024-03-16T11:48:41.044515+0000","last_peered":"2024-03-16T11:48:41.044515+0000","last_clean":"2024-03-16T11:48:41.044515+0000","last_became_active":"2024-03-16T11:48:38.583879+0000","last_became_peered":"2024-03-16T11:48:38.583879+0000","last_unstale":"2024-03-16T11:48:41.044515+0000","last_undegraded":"2024-03-16T11:48:41.044515+0000","last_fullsized":"2024-03-16T11:48:41.044515+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:08:41.119349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T11:48:41.045678+0000","last_change":"2024-03-16T11:48:38.575773+0000","last_active":"2024-03-16T11:48:41.045678+0000","last_peered":"2024-03-16T11:48:41.045678+0000","last_clean":"2024-03-16T11:48:41.045678+0000","last_became_active":"2024-03-16T11:48:38.574441+0000","last_became_peered":"2024-03-16T11:48:38.574441+0000","last_unstale":"2024-03-16T11:48:41.045678+0000","last_undegraded":"2024-03-16T11:48:41.045678+0000","last_fullsized":"2024-03-16T11:48:41.045678+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:22:56.148164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-16T11:48:38.577525+0000","last_change":"2024-03-16T11:48:38.575505+0000","last_active":"2024-03-16T11:48:38.577525+0000","last_peered":"2024-03-16T11:48:38.577525+0000","last_clean":"2024-03-16T11:48:38.577525+0000","last_became_active":"2024-03-16T11:48:38.575188+0000","last_became_peered":"2024-03-16T11:48:38.575188+0000","last_unstale":"2024-03-16T11:48:38.577525+0000","last_undegraded":"2024-03-16T11:48:38.577525+0000","last_fullsized":"2024-03-16T11:48:38.577525+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:44:08.971947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T11:48:39.811366+0000","last_change":"2024-03-16T11:48:38.572044+0000","last_active":"2024-03-16T11:48:39.811366+0000","last_peered":"2024-03-16T11:48:39.811366+0000","last_clean":"2024-03-16T11:48:39.811366+0000","last_became_active":"2024-03-16T11:48:38.571725+0000","last_became_peered":"2024-03-16T11:48:38.571725+0000","last_unstale":"2024-03-16T11:48:39.811366+0000","last_undegraded":"2024-03-16T11:48:39.811366+0000","last_fullsized":"2024-03-16T11:48:39.811366+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:05:37.060342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T11:48:41.043715+0000","last_change":"2024-03-16T11:48:38.570332+0000","last_active":"2024-03-16T11:48:41.043715+0000","last_peered":"2024-03-16T11:48:41.043715+0000","last_clean":"2024-03-16T11:48:41.043715+0000","last_became_active":"2024-03-16T11:48:38.570009+0000","last_became_peered":"2024-03-16T11:48:38.570009+0000","last_unstale":"2024-03-16T11:48:41.043715+0000","last_undegraded":"2024-03-16T11:48:41.043715+0000","last_fullsized":"2024-03-16T11:48:41.043715+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:42:31.045749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T11:48:41.016200+0000","last_change":"2024-03-16T11:48:38.571741+0000","last_active":"2024-03-16T11:48:41.016200+0000","last_peered":"2024-03-16T11:48:41.016200+0000","last_clean":"2024-03-16T11:48:41.016200+0000","last_became_active":"2024-03-16T11:48:38.569582+0000","last_became_peered":"2024-03-16T11:48:38.569582+0000","last_unstale":"2024-03-16T11:48:41.016200+0000","last_undegraded":"2024-03-16T11:48:41.016200+0000","last_fullsized":"2024-03-16T11:48:41.016200+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:05:36.302223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T11:48:39.560942+0000","last_change":"2024-03-16T11:48:38.576038+0000","last_active":"2024-03-16T11:48:39.560942+0000","last_peered":"2024-03-16T11:48:39.560942+0000","last_clean":"2024-03-16T11:48:39.560942+0000","last_became_active":"2024-03-16T11:48:38.574223+0000","last_became_peered":"2024-03-16T11:48:38.574223+0000","last_unstale":"2024-03-16T11:48:39.560942+0000","last_undegraded":"2024-03-16T11:48:39.560942+0000","last_fullsized":"2024-03-16T11:48:39.560942+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:35:17.236721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T11:48:39.811735+0000","last_change":"2024-03-16T11:48:38.572797+0000","last_active":"2024-03-16T11:48:39.811735+0000","last_peered":"2024-03-16T11:48:39.811735+0000","last_clean":"2024-03-16T11:48:39.811735+0000","last_became_active":"2024-03-16T11:48:38.572497+0000","last_became_peered":"2024-03-16T11:48:38.572497+0000","last_unstale":"2024-03-16T11:48:39.811735+0000","last_undegraded":"2024-03-16T11:48:39.811735+0000","last_fullsized":"2024-03-16T11:48:39.811735+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:25:49.138897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T11:48:41.044595+0000","last_change":"2024-03-16T11:48:38.570443+0000","last_active":"2024-03-16T11:48:41.044595+0000","last_peered":"2024-03-16T11:48:41.044595+0000","last_clean":"2024-03-16T11:48:41.044595+0000","last_became_active":"2024-03-16T11:48:38.569984+0000","last_became_peered":"2024-03-16T11:48:38.569984+0000","last_unstale":"2024-03-16T11:48:41.044595+0000","last_undegraded":"2024-03-16T11:48:41.044595+0000","last_fullsized":"2024-03-16T11:48:41.044595+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:47:42.539130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T11:48:41.045524+0000","last_change":"2024-03-16T11:48:38.576106+0000","last_active":"2024-03-16T11:48:41.045524+0000","last_peered":"2024-03-16T11:48:41.045524+0000","last_clean":"2024-03-16T11:48:41.045524+0000","last_became_active":"2024-03-16T11:48:38.575409+0000","last_became_peered":"2024-03-16T11:48:38.575409+0000","last_unstale":"2024-03-16T11:48:41.045524+0000","last_undegraded":"2024-03-16T11:48:41.045524+0000","last_fullsized":"2024-03-16T11:48:41.045524+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:38:33.277722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T11:48:39.557764+0000","last_change":"2024-03-16T11:48:38.559132+0000","last_active":"2024-03-16T11:48:39.557764+0000","last_peered":"2024-03-16T11:48:39.557764+0000","last_clean":"2024-03-16T11:48:39.557764+0000","last_became_active":"2024-03-16T11:48:38.558826+0000","last_became_peered":"2024-03-16T11:48:38.558826+0000","last_unstale":"2024-03-16T11:48:39.557764+0000","last_undegraded":"2024-03-16T11:48:39.557764+0000","last_fullsized":"2024-03-16T11:48:39.557764+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:27:12.562018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T11:48:41.043879+0000","last_change":"2024-03-16T11:48:38.583598+0000","last_active":"2024-03-16T11:48:41.043879+0000","last_peered":"2024-03-16T11:48:41.043879+0000","last_clean":"2024-03-16T11:48:41.043879+0000","last_became_active":"2024-03-16T11:48:38.583275+0000","last_became_peered":"2024-03-16T11:48:38.583275+0000","last_unstale":"2024-03-16T11:48:41.043879+0000","last_undegraded":"2024-03-16T11:48:41.043879+0000","last_fullsized":"2024-03-16T11:48:41.043879+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:58:48.667075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T11:48:41.043772+0000","last_change":"2024-03-16T11:48:38.580240+0000","last_active":"2024-03-16T11:48:41.043772+0000","last_peered":"2024-03-16T11:48:41.043772+0000","last_clean":"2024-03-16T11:48:41.043772+0000","last_became_active":"2024-03-16T11:48:38.579928+0000","last_became_peered":"2024-03-16T11:48:38.579928+0000","last_unstale":"2024-03-16T11:48:41.043772+0000","last_undegraded":"2024-03-16T11:48:41.043772+0000","last_fullsized":"2024-03-16T11:48:41.043772+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:05:14.006915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T11:48:39.560784+0000","last_change":"2024-03-16T11:48:38.574433+0000","last_active":"2024-03-16T11:48:39.560784+0000","last_peered":"2024-03-16T11:48:39.560784+0000","last_clean":"2024-03-16T11:48:39.560784+0000","last_became_active":"2024-03-16T11:48:38.574104+0000","last_became_peered":"2024-03-16T11:48:38.574104+0000","last_unstale":"2024-03-16T11:48:39.560784+0000","last_undegraded":"2024-03-16T11:48:39.560784+0000","last_fullsized":"2024-03-16T11:48:39.560784+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:42:20.588457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T11:48:39.942412+0000","last_change":"2024-03-16T11:48:38.575505+0000","last_active":"2024-03-16T11:48:39.942412+0000","last_peered":"2024-03-16T11:48:39.942412+0000","last_clean":"2024-03-16T11:48:39.942412+0000","last_became_active":"2024-03-16T11:48:38.575187+0000","last_became_peered":"2024-03-16T11:48:38.575187+0000","last_unstale":"2024-03-16T11:48:39.942412+0000","last_undegraded":"2024-03-16T11:48:39.942412+0000","last_fullsized":"2024-03-16T11:48:39.942412+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:13:06.260535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T11:48:40.507277+0000","last_change":"2024-03-16T11:48:38.568146+0000","last_active":"2024-03-16T11:48:40.507277+0000","last_peered":"2024-03-16T11:48:40.507277+0000","last_clean":"2024-03-16T11:48:40.507277+0000","last_became_active":"2024-03-16T11:48:38.567826+0000","last_became_peered":"2024-03-16T11:48:38.567826+0000","last_unstale":"2024-03-16T11:48:40.507277+0000","last_undegraded":"2024-03-16T11:48:40.507277+0000","last_fullsized":"2024-03-16T11:48:40.507277+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:46:32.993012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T11:48:39.558371+0000","last_change":"2024-03-16T11:48:38.578861+0000","last_active":"2024-03-16T11:48:39.558371+0000","last_peered":"2024-03-16T11:48:39.558371+0000","last_clean":"2024-03-16T11:48:39.558371+0000","last_became_active":"2024-03-16T11:48:38.577028+0000","last_became_peered":"2024-03-16T11:48:38.577028+0000","last_unstale":"2024-03-16T11:48:39.558371+0000","last_undegraded":"2024-03-16T11:48:39.558371+0000","last_fullsized":"2024-03-16T11:48:39.558371+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:55:34.520032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"85'9","reported_seq":25,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T11:48:41.978211+0000","last_change":"2024-03-16T11:48:38.026407+0000","last_active":"2024-03-16T11:48:41.978211+0000","last_peered":"2024-03-16T11:48:41.978211+0000","last_clean":"2024-03-16T11:48:41.978211+0000","last_became_active":"2024-03-16T11:48:38.026083+0000","last_became_peered":"2024-03-16T11:48:38.026083+0000","last_unstale":"2024-03-16T11:48:41.978211+0000","last_undegraded":"2024-03-16T11:48:41.978211+0000","last_fullsized":"2024-03-16T11:48:41.978211+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_clean_scrub_stamp":"2024-03-16T11:48:36.997377+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:01:23.751657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T11:48:41.016366+0000","last_change":"2024-03-16T11:48:38.573658+0000","last_active":"2024-03-16T11:48:41.016366+0000","last_peered":"2024-03-16T11:48:41.016366+0000","last_clean":"2024-03-16T11:48:41.016366+0000","last_became_active":"2024-03-16T11:48:38.572747+0000","last_became_peered":"2024-03-16T11:48:38.572747+0000","last_unstale":"2024-03-16T11:48:41.016366+0000","last_undegraded":"2024-03-16T11:48:41.016366+0000","last_fullsized":"2024-03-16T11:48:41.016366+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:09:11.128269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"84'1","reported_seq":16,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T11:48:41.016438+0000","last_change":"2024-03-16T11:48:38.027088+0000","last_active":"2024-03-16T11:48:41.016438+0000","last_peered":"2024-03-16T11:48:41.016438+0000","last_clean":"2024-03-16T11:48:41.016438+0000","last_became_active":"2024-03-16T11:48:38.026730+0000","last_became_peered":"2024-03-16T11:48:38.026730+0000","last_unstale":"2024-03-16T11:48:41.016438+0000","last_undegraded":"2024-03-16T11:48:41.016438+0000","last_fullsized":"2024-03-16T11:48:41.016438+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_clean_scrub_stamp":"2024-03-16T11:48:36.997377+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:23:28.987731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T11:48:41.045297+0000","last_change":"2024-03-16T11:48:38.583730+0000","last_active":"2024-03-16T11:48:41.045297+0000","last_peered":"2024-03-16T11:48:41.045297+0000","last_clean":"2024-03-16T11:48:41.045297+0000","last_became_active":"2024-03-16T11:48:38.583323+0000","last_became_peered":"2024-03-16T11:48:38.583323+0000","last_unstale":"2024-03-16T11:48:41.045297+0000","last_undegraded":"2024-03-16T11:48:41.045297+0000","last_fullsized":"2024-03-16T11:48:41.045297+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:35:20.923176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"84'2","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-16T11:48:39.020665+0000","last_change":"2024-03-16T11:48:38.027310+0000","last_active":"2024-03-16T11:48:39.020665+0000","last_peered":"2024-03-16T11:48:39.020665+0000","last_clean":"2024-03-16T11:48:39.020665+0000","last_became_active":"2024-03-16T11:48:38.026991+0000","last_became_peered":"2024-03-16T11:48:38.026991+0000","last_unstale":"2024-03-16T11:48:39.020665+0000","last_undegraded":"2024-03-16T11:48:39.020665+0000","last_fullsized":"2024-03-16T11:48:39.020665+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_clean_scrub_stamp":"2024-03-16T11:48:36.997377+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:19:24.922203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T11:48:39.560691+0000","last_change":"2024-03-16T11:48:38.581452+0000","last_active":"2024-03-16T11:48:39.560691+0000","last_peered":"2024-03-16T11:48:39.560691+0000","last_clean":"2024-03-16T11:48:39.560691+0000","last_became_active":"2024-03-16T11:48:38.579305+0000","last_became_peered":"2024-03-16T11:48:38.579305+0000","last_unstale":"2024-03-16T11:48:39.560691+0000","last_undegraded":"2024-03-16T11:48:39.560691+0000","last_fullsized":"2024-03-16T11:48:39.560691+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:57:43.812783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-16T11:48:39.019344+0000","last_change":"2024-03-16T11:48:38.023747+0000","last_active":"2024-03-16T11:48:39.019344+0000","last_peered":"2024-03-16T11:48:39.019344+0000","last_clean":"2024-03-16T11:48:39.019344+0000","last_became_active":"2024-03-16T11:48:38.023396+0000","last_became_peered":"2024-03-16T11:48:38.023396+0000","last_unstale":"2024-03-16T11:48:39.019344+0000","last_undegraded":"2024-03-16T11:48:39.019344+0000","last_fullsized":"2024-03-16T11:48:39.019344+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_clean_scrub_stamp":"2024-03-16T11:48:36.997377+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:42:22.456357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T11:48:41.045817+0000","last_change":"2024-03-16T11:48:38.572157+0000","last_active":"2024-03-16T11:48:41.045817+0000","last_peered":"2024-03-16T11:48:41.045817+0000","last_clean":"2024-03-16T11:48:41.045817+0000","last_became_active":"2024-03-16T11:48:38.571629+0000","last_became_peered":"2024-03-16T11:48:38.571629+0000","last_unstale":"2024-03-16T11:48:41.045817+0000","last_undegraded":"2024-03-16T11:48:41.045817+0000","last_fullsized":"2024-03-16T11:48:41.045817+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:47:39.353965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"85'5","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T11:48:41.049457+0000","last_change":"2024-03-16T11:48:38.027883+0000","last_active":"2024-03-16T11:48:41.049457+0000","last_peered":"2024-03-16T11:48:41.049457+0000","last_clean":"2024-03-16T11:48:41.049457+0000","last_became_active":"2024-03-16T11:48:38.027561+0000","last_became_peered":"2024-03-16T11:48:38.027561+0000","last_unstale":"2024-03-16T11:48:41.049457+0000","last_undegraded":"2024-03-16T11:48:41.049457+0000","last_fullsized":"2024-03-16T11:48:41.049457+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_clean_scrub_stamp":"2024-03-16T11:48:36.997377+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T11:58:56.063422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-16T11:48:38.577551+0000","last_change":"2024-03-16T11:48:38.574227+0000","last_active":"2024-03-16T11:48:38.577551+0000","last_peered":"2024-03-16T11:48:38.577551+0000","last_clean":"2024-03-16T11:48:38.577551+0000","last_became_active":"2024-03-16T11:48:38.571938+0000","last_became_peered":"2024-03-16T11:48:38.571938+0000","last_unstale":"2024-03-16T11:48:38.577551+0000","last_undegraded":"2024-03-16T11:48:38.577551+0000","last_fullsized":"2024-03-16T11:48:38.577551+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:48:25.028283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"85'3","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T11:48:41.974511+0000","last_change":"2024-03-16T11:48:38.024448+0000","last_active":"2024-03-16T11:48:41.974511+0000","last_peered":"2024-03-16T11:48:41.974511+0000","last_clean":"2024-03-16T11:48:41.974511+0000","last_became_active":"2024-03-16T11:48:38.024109+0000","last_became_peered":"2024-03-16T11:48:38.024109+0000","last_unstale":"2024-03-16T11:48:41.974511+0000","last_undegraded":"2024-03-16T11:48:41.974511+0000","last_fullsized":"2024-03-16T11:48:41.974511+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_clean_scrub_stamp":"2024-03-16T11:48:36.997377+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:37:49.521297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T11:48:41.044579+0000","last_change":"2024-03-16T11:48:38.577722+0000","last_active":"2024-03-16T11:48:41.044579+0000","last_peered":"2024-03-16T11:48:41.044579+0000","last_clean":"2024-03-16T11:48:41.044579+0000","last_became_active":"2024-03-16T11:48:38.577223+0000","last_became_peered":"2024-03-16T11:48:38.577223+0000","last_unstale":"2024-03-16T11:48:41.044579+0000","last_undegraded":"2024-03-16T11:48:41.044579+0000","last_fullsized":"2024-03-16T11:48:41.044579+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:24:05.892850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"85'2","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T11:48:41.977680+0000","last_change":"2024-03-16T11:48:38.029681+0000","last_active":"2024-03-16T11:48:41.977680+0000","last_peered":"2024-03-16T11:48:41.977680+0000","last_clean":"2024-03-16T11:48:41.977680+0000","last_became_active":"2024-03-16T11:48:38.029261+0000","last_became_peered":"2024-03-16T11:48:38.029261+0000","last_unstale":"2024-03-16T11:48:41.977680+0000","last_undegraded":"2024-03-16T11:48:41.977680+0000","last_fullsized":"2024-03-16T11:48:41.977680+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_clean_scrub_stamp":"2024-03-16T11:48:36.997377+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:57:51.155337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T11:48:41.015999+0000","last_change":"2024-03-16T11:48:38.570655+0000","last_active":"2024-03-16T11:48:41.015999+0000","last_peered":"2024-03-16T11:48:41.015999+0000","last_clean":"2024-03-16T11:48:41.015999+0000","last_became_active":"2024-03-16T11:48:38.569143+0000","last_became_peered":"2024-03-16T11:48:38.569143+0000","last_unstale":"2024-03-16T11:48:41.015999+0000","last_undegraded":"2024-03-16T11:48:41.015999+0000","last_fullsized":"2024-03-16T11:48:41.015999+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:51:51.449168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"85'1","reported_seq":16,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T11:48:41.048005+0000","last_change":"2024-03-16T11:48:38.027074+0000","last_active":"2024-03-16T11:48:41.048005+0000","last_peered":"2024-03-16T11:48:41.048005+0000","last_clean":"2024-03-16T11:48:41.048005+0000","last_became_active":"2024-03-16T11:48:38.026749+0000","last_became_peered":"2024-03-16T11:48:38.026749+0000","last_unstale":"2024-03-16T11:48:41.048005+0000","last_undegraded":"2024-03-16T11:48:41.048005+0000","last_fullsized":"2024-03-16T11:48:41.048005+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_clean_scrub_stamp":"2024-03-16T11:48:36.997377+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:14:26.801688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T11:48:40.507244+0000","last_change":"2024-03-16T11:48:38.575699+0000","last_active":"2024-03-16T11:48:40.507244+0000","last_peered":"2024-03-16T11:48:40.507244+0000","last_clean":"2024-03-16T11:48:40.507244+0000","last_became_active":"2024-03-16T11:48:38.575401+0000","last_became_peered":"2024-03-16T11:48:38.575401+0000","last_unstale":"2024-03-16T11:48:40.507244+0000","last_undegraded":"2024-03-16T11:48:40.507244+0000","last_fullsized":"2024-03-16T11:48:40.507244+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:48:06.289779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"85'3","reported_seq":21,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T11:48:42.973069+0000","last_change":"2024-03-16T11:48:38.026488+0000","last_active":"2024-03-16T11:48:42.973069+0000","last_peered":"2024-03-16T11:48:42.973069+0000","last_clean":"2024-03-16T11:48:42.973069+0000","last_became_active":"2024-03-16T11:48:38.026101+0000","last_became_peered":"2024-03-16T11:48:38.026101+0000","last_unstale":"2024-03-16T11:48:42.973069+0000","last_undegraded":"2024-03-16T11:48:42.973069+0000","last_fullsized":"2024-03-16T11:48:42.973069+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_clean_scrub_stamp":"2024-03-16T11:48:36.997377+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:37:29.788307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T11:48:39.560914+0000","last_change":"2024-03-16T11:48:38.578081+0000","last_active":"2024-03-16T11:48:39.560914+0000","last_peered":"2024-03-16T11:48:39.560914+0000","last_clean":"2024-03-16T11:48:39.560914+0000","last_became_active":"2024-03-16T11:48:38.577742+0000","last_became_peered":"2024-03-16T11:48:38.577742+0000","last_unstale":"2024-03-16T11:48:39.560914+0000","last_undegraded":"2024-03-16T11:48:39.560914+0000","last_fullsized":"2024-03-16T11:48:39.560914+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:58:41.479658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"85'2","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T11:48:41.047455+0000","last_change":"2024-03-16T11:48:38.026891+0000","last_active":"2024-03-16T11:48:41.047455+0000","last_peered":"2024-03-16T11:48:41.047455+0000","last_clean":"2024-03-16T11:48:41.047455+0000","last_became_active":"2024-03-16T11:48:38.026397+0000","last_became_peered":"2024-03-16T11:48:38.026397+0000","last_unstale":"2024-03-16T11:48:41.047455+0000","last_undegraded":"2024-03-16T11:48:41.047455+0000","last_fullsized":"2024-03-16T11:48:41.047455+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_clean_scrub_stamp":"2024-03-16T11:48:36.997377+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:02:01.018103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T11:48:41.045244+0000","last_change":"2024-03-16T11:48:38.579205+0000","last_active":"2024-03-16T11:48:41.045244+0000","last_peered":"2024-03-16T11:48:41.045244+0000","last_clean":"2024-03-16T11:48:41.045244+0000","last_became_active":"2024-03-16T11:48:38.578910+0000","last_became_peered":"2024-03-16T11:48:38.578910+0000","last_unstale":"2024-03-16T11:48:41.045244+0000","last_undegraded":"2024-03-16T11:48:41.045244+0000","last_fullsized":"2024-03-16T11:48:41.045244+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:22:06.498974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"85'4","reported_seq":19,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T11:48:41.048250+0000","last_change":"2024-03-16T11:48:38.025103+0000","last_active":"2024-03-16T11:48:41.048250+0000","last_peered":"2024-03-16T11:48:41.048250+0000","last_clean":"2024-03-16T11:48:41.048250+0000","last_became_active":"2024-03-16T11:48:38.024754+0000","last_became_peered":"2024-03-16T11:48:38.024754+0000","last_unstale":"2024-03-16T11:48:41.048250+0000","last_undegraded":"2024-03-16T11:48:41.048250+0000","last_fullsized":"2024-03-16T11:48:41.048250+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_clean_scrub_stamp":"2024-03-16T11:48:36.997377+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:40:10.335339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T11:48:39.560755+0000","last_change":"2024-03-16T11:48:38.578305+0000","last_active":"2024-03-16T11:48:39.560755+0000","last_peered":"2024-03-16T11:48:39.560755+0000","last_clean":"2024-03-16T11:48:39.560755+0000","last_became_active":"2024-03-16T11:48:38.577971+0000","last_became_peered":"2024-03-16T11:48:38.577971+0000","last_unstale":"2024-03-16T11:48:39.560755+0000","last_undegraded":"2024-03-16T11:48:39.560755+0000","last_fullsized":"2024-03-16T11:48:39.560755+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:04:34.259456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"85'16","reported_seq":37,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T11:48:43.701456+0000","last_change":"2024-03-16T11:48:38.026013+0000","last_active":"2024-03-16T11:48:43.701456+0000","last_peered":"2024-03-16T11:48:43.701456+0000","last_clean":"2024-03-16T11:48:43.701456+0000","last_became_active":"2024-03-16T11:48:38.025680+0000","last_became_peered":"2024-03-16T11:48:38.025680+0000","last_unstale":"2024-03-16T11:48:43.701456+0000","last_undegraded":"2024-03-16T11:48:43.701456+0000","last_fullsized":"2024-03-16T11:48:43.701456+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_clean_scrub_stamp":"2024-03-16T11:48:36.997377+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:02:02.375157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25510,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":17,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-16T11:48:38.577620+0000","last_change":"2024-03-16T11:48:38.574404+0000","last_active":"2024-03-16T11:48:38.577620+0000","last_peered":"2024-03-16T11:48:38.577620+0000","last_clean":"2024-03-16T11:48:38.577620+0000","last_became_active":"2024-03-16T11:48:38.572334+0000","last_became_peered":"2024-03-16T11:48:38.572334+0000","last_unstale":"2024-03-16T11:48:38.577620+0000","last_undegraded":"2024-03-16T11:48:38.577620+0000","last_fullsized":"2024-03-16T11:48:38.577620+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:47:04.912409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"85'4","reported_seq":19,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T11:48:41.047418+0000","last_change":"2024-03-16T11:48:38.025287+0000","last_active":"2024-03-16T11:48:41.047418+0000","last_peered":"2024-03-16T11:48:41.047418+0000","last_clean":"2024-03-16T11:48:41.047418+0000","last_became_active":"2024-03-16T11:48:38.022433+0000","last_became_peered":"2024-03-16T11:48:38.022433+0000","last_unstale":"2024-03-16T11:48:41.047418+0000","last_undegraded":"2024-03-16T11:48:41.047418+0000","last_fullsized":"2024-03-16T11:48:41.047418+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_clean_scrub_stamp":"2024-03-16T11:48:36.997377+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:27:20.573925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T11:48:39.557388+0000","last_change":"2024-03-16T11:48:38.574715+0000","last_active":"2024-03-16T11:48:39.557388+0000","last_peered":"2024-03-16T11:48:39.557388+0000","last_clean":"2024-03-16T11:48:39.557388+0000","last_became_active":"2024-03-16T11:48:38.574456+0000","last_became_peered":"2024-03-16T11:48:38.574456+0000","last_unstale":"2024-03-16T11:48:39.557388+0000","last_undegraded":"2024-03-16T11:48:39.557388+0000","last_fullsized":"2024-03-16T11:48:39.557388+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:57:00.123583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"85'3","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T11:48:41.048362+0000","last_change":"2024-03-16T11:48:38.029557+0000","last_active":"2024-03-16T11:48:41.048362+0000","last_peered":"2024-03-16T11:48:41.048362+0000","last_clean":"2024-03-16T11:48:41.048362+0000","last_became_active":"2024-03-16T11:48:38.029249+0000","last_became_peered":"2024-03-16T11:48:38.029249+0000","last_unstale":"2024-03-16T11:48:41.048362+0000","last_undegraded":"2024-03-16T11:48:41.048362+0000","last_fullsized":"2024-03-16T11:48:41.048362+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_clean_scrub_stamp":"2024-03-16T11:48:36.997377+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:32:41.364467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T11:48:41.045083+0000","last_change":"2024-03-16T11:48:38.575579+0000","last_active":"2024-03-16T11:48:41.045083+0000","last_peered":"2024-03-16T11:48:41.045083+0000","last_clean":"2024-03-16T11:48:41.045083+0000","last_became_active":"2024-03-16T11:48:38.575263+0000","last_became_peered":"2024-03-16T11:48:38.575263+0000","last_unstale":"2024-03-16T11:48:41.045083+0000","last_undegraded":"2024-03-16T11:48:41.045083+0000","last_fullsized":"2024-03-16T11:48:41.045083+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:59:38.983419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"85'3","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T11:48:41.972993+0000","last_change":"2024-03-16T11:48:38.027047+0000","last_active":"2024-03-16T11:48:41.972993+0000","last_peered":"2024-03-16T11:48:41.972993+0000","last_clean":"2024-03-16T11:48:41.972993+0000","last_became_active":"2024-03-16T11:48:38.026733+0000","last_became_peered":"2024-03-16T11:48:38.026733+0000","last_unstale":"2024-03-16T11:48:41.972993+0000","last_undegraded":"2024-03-16T11:48:41.972993+0000","last_fullsized":"2024-03-16T11:48:41.972993+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_clean_scrub_stamp":"2024-03-16T11:48:36.997377+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:27:23.763104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T11:48:41.043421+0000","last_change":"2024-03-16T11:48:38.568802+0000","last_active":"2024-03-16T11:48:41.043421+0000","last_peered":"2024-03-16T11:48:41.043421+0000","last_clean":"2024-03-16T11:48:41.043421+0000","last_became_active":"2024-03-16T11:48:38.568213+0000","last_became_peered":"2024-03-16T11:48:38.568213+0000","last_unstale":"2024-03-16T11:48:41.043421+0000","last_undegraded":"2024-03-16T11:48:41.043421+0000","last_fullsized":"2024-03-16T11:48:41.043421+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:38:23.233563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"84'8","reported_seq":21,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-16T11:48:39.021485+0000","last_change":"2024-03-16T11:48:38.027596+0000","last_active":"2024-03-16T11:48:39.021485+0000","last_peered":"2024-03-16T11:48:39.021485+0000","last_clean":"2024-03-16T11:48:39.021485+0000","last_became_active":"2024-03-16T11:48:38.027280+0000","last_became_peered":"2024-03-16T11:48:38.027280+0000","last_unstale":"2024-03-16T11:48:39.021485+0000","last_undegraded":"2024-03-16T11:48:39.021485+0000","last_fullsized":"2024-03-16T11:48:39.021485+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_clean_scrub_stamp":"2024-03-16T11:48:36.997377+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:05:26.409613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T11:48:41.044594+0000","last_change":"2024-03-16T11:48:38.571308+0000","last_active":"2024-03-16T11:48:41.044594+0000","last_peered":"2024-03-16T11:48:41.044594+0000","last_clean":"2024-03-16T11:48:41.044594+0000","last_became_active":"2024-03-16T11:48:38.570927+0000","last_became_peered":"2024-03-16T11:48:38.570927+0000","last_unstale":"2024-03-16T11:48:41.044594+0000","last_undegraded":"2024-03-16T11:48:41.044594+0000","last_fullsized":"2024-03-16T11:48:41.044594+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:15:18.190361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"84'1","reported_seq":14,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-16T11:48:39.019684+0000","last_change":"2024-03-16T11:48:38.024783+0000","last_active":"2024-03-16T11:48:39.019684+0000","last_peered":"2024-03-16T11:48:39.019684+0000","last_clean":"2024-03-16T11:48:39.019684+0000","last_became_active":"2024-03-16T11:48:38.023990+0000","last_became_peered":"2024-03-16T11:48:38.023990+0000","last_unstale":"2024-03-16T11:48:39.019684+0000","last_undegraded":"2024-03-16T11:48:39.019684+0000","last_fullsized":"2024-03-16T11:48:39.019684+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_clean_scrub_stamp":"2024-03-16T11:48:36.997377+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:42:49.375788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T11:48:39.942341+0000","last_change":"2024-03-16T11:48:38.580019+0000","last_active":"2024-03-16T11:48:39.942341+0000","last_peered":"2024-03-16T11:48:39.942341+0000","last_clean":"2024-03-16T11:48:39.942341+0000","last_became_active":"2024-03-16T11:48:38.579686+0000","last_became_peered":"2024-03-16T11:48:38.579686+0000","last_unstale":"2024-03-16T11:48:39.942341+0000","last_undegraded":"2024-03-16T11:48:39.942341+0000","last_fullsized":"2024-03-16T11:48:39.942341+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:00:21.752048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T11:48:40.507110+0000","last_change":"2024-03-16T11:48:38.017631+0000","last_active":"2024-03-16T11:48:40.507110+0000","last_peered":"2024-03-16T11:48:40.507110+0000","last_clean":"2024-03-16T11:48:40.507110+0000","last_became_active":"2024-03-16T11:48:38.017218+0000","last_became_peered":"2024-03-16T11:48:38.017218+0000","last_unstale":"2024-03-16T11:48:40.507110+0000","last_undegraded":"2024-03-16T11:48:40.507110+0000","last_fullsized":"2024-03-16T11:48:40.507110+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_clean_scrub_stamp":"2024-03-16T11:48:36.997377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:31:41.378757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"85'1","reported_seq":14,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T11:48:39.811681+0000","last_change":"2024-03-16T11:48:38.576751+0000","last_active":"2024-03-16T11:48:39.811681+0000","last_peered":"2024-03-16T11:48:39.811681+0000","last_clean":"2024-03-16T11:48:39.811681+0000","last_became_active":"2024-03-16T11:48:38.576442+0000","last_became_peered":"2024-03-16T11:48:38.576442+0000","last_unstale":"2024-03-16T11:48:39.811681+0000","last_undegraded":"2024-03-16T11:48:39.811681+0000","last_fullsized":"2024-03-16T11:48:39.811681+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:49:59.475427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"85'1","reported_seq":16,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T11:48:41.044303+0000","last_change":"2024-03-16T11:48:38.030159+0000","last_active":"2024-03-16T11:48:41.044303+0000","last_peered":"2024-03-16T11:48:41.044303+0000","last_clean":"2024-03-16T11:48:41.044303+0000","last_became_active":"2024-03-16T11:48:38.029808+0000","last_became_peered":"2024-03-16T11:48:38.029808+0000","last_unstale":"2024-03-16T11:48:41.044303+0000","last_undegraded":"2024-03-16T11:48:41.044303+0000","last_fullsized":"2024-03-16T11:48:41.044303+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_clean_scrub_stamp":"2024-03-16T11:48:36.997377+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:46:54.692837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T11:48:39.811447+0000","last_change":"2024-03-16T11:48:38.574080+0000","last_active":"2024-03-16T11:48:39.811447+0000","last_peered":"2024-03-16T11:48:39.811447+0000","last_clean":"2024-03-16T11:48:39.811447+0000","last_became_active":"2024-03-16T11:48:38.573745+0000","last_became_peered":"2024-03-16T11:48:38.573745+0000","last_unstale":"2024-03-16T11:48:39.811447+0000","last_undegraded":"2024-03-16T11:48:39.811447+0000","last_fullsized":"2024-03-16T11:48:39.811447+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:12:56.154804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T11:48:41.044232+0000","last_change":"2024-03-16T11:48:38.571750+0000","last_active":"2024-03-16T11:48:41.044232+0000","last_peered":"2024-03-16T11:48:41.044232+0000","last_clean":"2024-03-16T11:48:41.044232+0000","last_became_active":"2024-03-16T11:48:38.571273+0000","last_became_peered":"2024-03-16T11:48:38.571273+0000","last_unstale":"2024-03-16T11:48:41.044232+0000","last_undegraded":"2024-03-16T11:48:41.044232+0000","last_fullsized":"2024-03-16T11:48:41.044232+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:59:21.952927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"85'4","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T11:48:41.977354+0000","last_change":"2024-03-16T11:48:38.034121+0000","last_active":"2024-03-16T11:48:41.977354+0000","last_peered":"2024-03-16T11:48:41.977354+0000","last_clean":"2024-03-16T11:48:41.977354+0000","last_became_active":"2024-03-16T11:48:38.033777+0000","last_became_peered":"2024-03-16T11:48:38.033777+0000","last_unstale":"2024-03-16T11:48:41.977354+0000","last_undegraded":"2024-03-16T11:48:41.977354+0000","last_fullsized":"2024-03-16T11:48:41.977354+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_clean_scrub_stamp":"2024-03-16T11:48:36.997377+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:51:54.637120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T11:48:41.016427+0000","last_change":"2024-03-16T11:48:38.580050+0000","last_active":"2024-03-16T11:48:41.016427+0000","last_peered":"2024-03-16T11:48:41.016427+0000","last_clean":"2024-03-16T11:48:41.016427+0000","last_became_active":"2024-03-16T11:48:38.578950+0000","last_became_peered":"2024-03-16T11:48:38.578950+0000","last_unstale":"2024-03-16T11:48:41.016427+0000","last_undegraded":"2024-03-16T11:48:41.016427+0000","last_fullsized":"2024-03-16T11:48:41.016427+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:10:21.717916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"85'3","reported_seq":19,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T11:48:41.964761+0000","last_change":"2024-03-16T11:48:38.023925+0000","last_active":"2024-03-16T11:48:41.964761+0000","last_peered":"2024-03-16T11:48:41.964761+0000","last_clean":"2024-03-16T11:48:41.964761+0000","last_became_active":"2024-03-16T11:48:38.023424+0000","last_became_peered":"2024-03-16T11:48:38.023424+0000","last_unstale":"2024-03-16T11:48:41.964761+0000","last_undegraded":"2024-03-16T11:48:41.964761+0000","last_fullsized":"2024-03-16T11:48:41.964761+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_clean_scrub_stamp":"2024-03-16T11:48:36.997377+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:01:53.085039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":349,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T11:48:41.043439+0000","last_change":"2024-03-16T11:47:22.001149+0000","last_active":"2024-03-16T11:48:41.043439+0000","last_peered":"2024-03-16T11:48:41.043439+0000","last_clean":"2024-03-16T11:48:41.043439+0000","last_became_active":"2024-03-16T11:47:22.000506+0000","last_became_peered":"2024-03-16T11:47:22.000506+0000","last_unstale":"2024-03-16T11:48:41.043439+0000","last_undegraded":"2024-03-16T11:48:41.043439+0000","last_fullsized":"2024-03-16T11:48:41.043439+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:43:52.376552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:43:52.376552+0000","last_clean_scrub_stamp":"2024-03-16T11:43:52.376552+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:21:18.466389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T11:48:39.557252+0000","last_change":"2024-03-16T11:48:38.577530+0000","last_active":"2024-03-16T11:48:39.557252+0000","last_peered":"2024-03-16T11:48:39.557252+0000","last_clean":"2024-03-16T11:48:39.557252+0000","last_became_active":"2024-03-16T11:48:38.577252+0000","last_became_peered":"2024-03-16T11:48:38.577252+0000","last_unstale":"2024-03-16T11:48:39.557252+0000","last_undegraded":"2024-03-16T11:48:39.557252+0000","last_fullsized":"2024-03-16T11:48:39.557252+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:07:47.809075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"85'3","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T11:48:41.973618+0000","last_change":"2024-03-16T11:48:38.027630+0000","last_active":"2024-03-16T11:48:41.973618+0000","last_peered":"2024-03-16T11:48:41.973618+0000","last_clean":"2024-03-16T11:48:41.973618+0000","last_became_active":"2024-03-16T11:48:38.027105+0000","last_became_peered":"2024-03-16T11:48:38.027105+0000","last_unstale":"2024-03-16T11:48:41.973618+0000","last_undegraded":"2024-03-16T11:48:41.973618+0000","last_fullsized":"2024-03-16T11:48:41.973618+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_clean_scrub_stamp":"2024-03-16T11:48:36.997377+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:24:20.605988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":13,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-16T11:48:38.564200+0000","last_change":"2024-03-16T11:48:38.032419+0000","last_active":"2024-03-16T11:48:38.564200+0000","last_peered":"2024-03-16T11:48:38.564200+0000","last_clean":"2024-03-16T11:48:38.564200+0000","last_became_active":"2024-03-16T11:48:38.032087+0000","last_became_peered":"2024-03-16T11:48:38.032087+0000","last_unstale":"2024-03-16T11:48:38.564200+0000","last_undegraded":"2024-03-16T11:48:38.564200+0000","last_fullsized":"2024-03-16T11:48:38.564200+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_clean_scrub_stamp":"2024-03-16T11:48:36.997377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:18:54.459321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T11:48:39.557576+0000","last_change":"2024-03-16T11:48:38.568768+0000","last_active":"2024-03-16T11:48:39.557576+0000","last_peered":"2024-03-16T11:48:39.557576+0000","last_clean":"2024-03-16T11:48:39.557576+0000","last_became_active":"2024-03-16T11:48:38.568408+0000","last_became_peered":"2024-03-16T11:48:38.568408+0000","last_unstale":"2024-03-16T11:48:39.557576+0000","last_undegraded":"2024-03-16T11:48:39.557576+0000","last_fullsized":"2024-03-16T11:48:39.557576+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:10:33.293464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"85'3","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T11:48:41.977856+0000","last_change":"2024-03-16T11:48:38.023387+0000","last_active":"2024-03-16T11:48:41.977856+0000","last_peered":"2024-03-16T11:48:41.977856+0000","last_clean":"2024-03-16T11:48:41.977856+0000","last_became_active":"2024-03-16T11:48:38.022916+0000","last_became_peered":"2024-03-16T11:48:38.022916+0000","last_unstale":"2024-03-16T11:48:41.977856+0000","last_undegraded":"2024-03-16T11:48:41.977856+0000","last_fullsized":"2024-03-16T11:48:41.977856+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_clean_scrub_stamp":"2024-03-16T11:48:36.997377+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:25:41.478737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T11:48:41.016406+0000","last_change":"2024-03-16T11:48:38.572243+0000","last_active":"2024-03-16T11:48:41.016406+0000","last_peered":"2024-03-16T11:48:41.016406+0000","last_clean":"2024-03-16T11:48:41.016406+0000","last_became_active":"2024-03-16T11:48:38.571918+0000","last_became_peered":"2024-03-16T11:48:38.571918+0000","last_unstale":"2024-03-16T11:48:41.016406+0000","last_undegraded":"2024-03-16T11:48:41.016406+0000","last_fullsized":"2024-03-16T11:48:41.016406+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:21:45.843298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"85'4","reported_seq":19,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T11:48:41.047711+0000","last_change":"2024-03-16T11:48:38.022825+0000","last_active":"2024-03-16T11:48:41.047711+0000","last_peered":"2024-03-16T11:48:41.047711+0000","last_clean":"2024-03-16T11:48:41.047711+0000","last_became_active":"2024-03-16T11:48:38.022499+0000","last_became_peered":"2024-03-16T11:48:38.022499+0000","last_unstale":"2024-03-16T11:48:41.047711+0000","last_undegraded":"2024-03-16T11:48:41.047711+0000","last_fullsized":"2024-03-16T11:48:41.047711+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_clean_scrub_stamp":"2024-03-16T11:48:36.997377+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:52:51.810960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T11:48:41.044367+0000","last_change":"2024-03-16T11:48:38.577473+0000","last_active":"2024-03-16T11:48:41.044367+0000","last_peered":"2024-03-16T11:48:41.044367+0000","last_clean":"2024-03-16T11:48:41.044367+0000","last_became_active":"2024-03-16T11:48:38.577121+0000","last_became_peered":"2024-03-16T11:48:38.577121+0000","last_unstale":"2024-03-16T11:48:41.044367+0000","last_undegraded":"2024-03-16T11:48:41.044367+0000","last_fullsized":"2024-03-16T11:48:41.044367+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:21:06.905785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"85'2","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T11:48:41.046675+0000","last_change":"2024-03-16T11:48:38.020324+0000","last_active":"2024-03-16T11:48:41.046675+0000","last_peered":"2024-03-16T11:48:41.046675+0000","last_clean":"2024-03-16T11:48:41.046675+0000","last_became_active":"2024-03-16T11:48:38.019692+0000","last_became_peered":"2024-03-16T11:48:38.019692+0000","last_unstale":"2024-03-16T11:48:41.046675+0000","last_undegraded":"2024-03-16T11:48:41.046675+0000","last_fullsized":"2024-03-16T11:48:41.046675+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_clean_scrub_stamp":"2024-03-16T11:48:36.997377+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:42:51.709046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-16T11:48:38.573748+0000","last_change":"2024-03-16T11:48:38.570911+0000","last_active":"2024-03-16T11:48:38.573748+0000","last_peered":"2024-03-16T11:48:38.573748+0000","last_clean":"2024-03-16T11:48:38.573748+0000","last_became_active":"2024-03-16T11:48:38.570288+0000","last_became_peered":"2024-03-16T11:48:38.570288+0000","last_unstale":"2024-03-16T11:48:38.573748+0000","last_undegraded":"2024-03-16T11:48:38.573748+0000","last_fullsized":"2024-03-16T11:48:38.573748+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:27:21.274856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"85'3","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T11:48:41.046519+0000","last_change":"2024-03-16T11:48:38.021817+0000","last_active":"2024-03-16T11:48:41.046519+0000","last_peered":"2024-03-16T11:48:41.046519+0000","last_clean":"2024-03-16T11:48:41.046519+0000","last_became_active":"2024-03-16T11:48:38.020906+0000","last_became_peered":"2024-03-16T11:48:38.020906+0000","last_unstale":"2024-03-16T11:48:41.046519+0000","last_undegraded":"2024-03-16T11:48:41.046519+0000","last_fullsized":"2024-03-16T11:48:41.046519+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_clean_scrub_stamp":"2024-03-16T11:48:36.997377+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:09:53.372698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T11:48:39.942085+0000","last_change":"2024-03-16T11:48:38.573536+0000","last_active":"2024-03-16T11:48:39.942085+0000","last_peered":"2024-03-16T11:48:39.942085+0000","last_clean":"2024-03-16T11:48:39.942085+0000","last_became_active":"2024-03-16T11:48:38.573211+0000","last_became_peered":"2024-03-16T11:48:38.573211+0000","last_unstale":"2024-03-16T11:48:39.942085+0000","last_undegraded":"2024-03-16T11:48:39.942085+0000","last_fullsized":"2024-03-16T11:48:39.942085+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:53:02.607175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":15,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T11:48:39.558004+0000","last_change":"2024-03-16T11:48:38.022830+0000","last_active":"2024-03-16T11:48:39.558004+0000","last_peered":"2024-03-16T11:48:39.558004+0000","last_clean":"2024-03-16T11:48:39.558004+0000","last_became_active":"2024-03-16T11:48:38.021159+0000","last_became_peered":"2024-03-16T11:48:38.021159+0000","last_unstale":"2024-03-16T11:48:39.558004+0000","last_undegraded":"2024-03-16T11:48:39.558004+0000","last_fullsized":"2024-03-16T11:48:39.558004+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_clean_scrub_stamp":"2024-03-16T11:48:36.997377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:06:32.910622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T11:48:41.016373+0000","last_change":"2024-03-16T11:48:38.575965+0000","last_active":"2024-03-16T11:48:41.016373+0000","last_peered":"2024-03-16T11:48:41.016373+0000","last_clean":"2024-03-16T11:48:41.016373+0000","last_became_active":"2024-03-16T11:48:38.575627+0000","last_became_peered":"2024-03-16T11:48:38.575627+0000","last_unstale":"2024-03-16T11:48:41.016373+0000","last_undegraded":"2024-03-16T11:48:41.016373+0000","last_fullsized":"2024-03-16T11:48:41.016373+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:00:37.768897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"84'2","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-16T11:48:39.020331+0000","last_change":"2024-03-16T11:48:38.027516+0000","last_active":"2024-03-16T11:48:39.020331+0000","last_peered":"2024-03-16T11:48:39.020331+0000","last_clean":"2024-03-16T11:48:39.020331+0000","last_became_active":"2024-03-16T11:48:38.026865+0000","last_became_peered":"2024-03-16T11:48:38.026865+0000","last_unstale":"2024-03-16T11:48:39.020331+0000","last_undegraded":"2024-03-16T11:48:39.020331+0000","last_fullsized":"2024-03-16T11:48:39.020331+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_clean_scrub_stamp":"2024-03-16T11:48:36.997377+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:50:50.240973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T11:48:41.044384+0000","last_change":"2024-03-16T11:48:38.571242+0000","last_active":"2024-03-16T11:48:41.044384+0000","last_peered":"2024-03-16T11:48:41.044384+0000","last_clean":"2024-03-16T11:48:41.044384+0000","last_became_active":"2024-03-16T11:48:38.570917+0000","last_became_peered":"2024-03-16T11:48:38.570917+0000","last_unstale":"2024-03-16T11:48:41.044384+0000","last_undegraded":"2024-03-16T11:48:41.044384+0000","last_fullsized":"2024-03-16T11:48:41.044384+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:30:12.601982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":15,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T11:48:39.558335+0000","last_change":"2024-03-16T11:48:38.023970+0000","last_active":"2024-03-16T11:48:39.558335+0000","last_peered":"2024-03-16T11:48:39.558335+0000","last_clean":"2024-03-16T11:48:39.558335+0000","last_became_active":"2024-03-16T11:48:38.023646+0000","last_became_peered":"2024-03-16T11:48:38.023646+0000","last_unstale":"2024-03-16T11:48:39.558335+0000","last_undegraded":"2024-03-16T11:48:39.558335+0000","last_fullsized":"2024-03-16T11:48:39.558335+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_clean_scrub_stamp":"2024-03-16T11:48:36.997377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:48:35.549588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T11:48:41.044754+0000","last_change":"2024-03-16T11:48:38.567376+0000","last_active":"2024-03-16T11:48:41.044754+0000","last_peered":"2024-03-16T11:48:41.044754+0000","last_clean":"2024-03-16T11:48:41.044754+0000","last_became_active":"2024-03-16T11:48:38.567048+0000","last_became_peered":"2024-03-16T11:48:38.567048+0000","last_unstale":"2024-03-16T11:48:41.044754+0000","last_undegraded":"2024-03-16T11:48:41.044754+0000","last_fullsized":"2024-03-16T11:48:41.044754+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:45:50.097013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"84'1","reported_seq":16,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T11:48:41.045756+0000","last_change":"2024-03-16T11:48:38.029192+0000","last_active":"2024-03-16T11:48:41.045756+0000","last_peered":"2024-03-16T11:48:41.045756+0000","last_clean":"2024-03-16T11:48:41.045756+0000","last_became_active":"2024-03-16T11:48:38.028866+0000","last_became_peered":"2024-03-16T11:48:38.028866+0000","last_unstale":"2024-03-16T11:48:41.045756+0000","last_undegraded":"2024-03-16T11:48:41.045756+0000","last_fullsized":"2024-03-16T11:48:41.045756+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_clean_scrub_stamp":"2024-03-16T11:48:36.997377+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:52:31.329863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T11:48:41.043999+0000","last_change":"2024-03-16T11:48:38.576538+0000","last_active":"2024-03-16T11:48:41.043999+0000","last_peered":"2024-03-16T11:48:41.043999+0000","last_clean":"2024-03-16T11:48:41.043999+0000","last_became_active":"2024-03-16T11:48:38.576222+0000","last_became_peered":"2024-03-16T11:48:38.576222+0000","last_unstale":"2024-03-16T11:48:41.043999+0000","last_undegraded":"2024-03-16T11:48:41.043999+0000","last_fullsized":"2024-03-16T11:48:41.043999+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:34:13.186406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T11:48:39.557621+0000","last_change":"2024-03-16T11:48:38.031002+0000","last_active":"2024-03-16T11:48:39.557621+0000","last_peered":"2024-03-16T11:48:39.557621+0000","last_clean":"2024-03-16T11:48:39.557621+0000","last_became_active":"2024-03-16T11:48:38.030653+0000","last_became_peered":"2024-03-16T11:48:38.030653+0000","last_unstale":"2024-03-16T11:48:39.557621+0000","last_undegraded":"2024-03-16T11:48:39.557621+0000","last_fullsized":"2024-03-16T11:48:39.557621+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_clean_scrub_stamp":"2024-03-16T11:48:36.997377+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:42:30.148594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T11:48:41.044271+0000","last_change":"2024-03-16T11:48:38.571942+0000","last_active":"2024-03-16T11:48:41.044271+0000","last_peered":"2024-03-16T11:48:41.044271+0000","last_clean":"2024-03-16T11:48:41.044271+0000","last_became_active":"2024-03-16T11:48:38.571606+0000","last_became_peered":"2024-03-16T11:48:38.571606+0000","last_unstale":"2024-03-16T11:48:41.044271+0000","last_undegraded":"2024-03-16T11:48:41.044271+0000","last_fullsized":"2024-03-16T11:48:41.044271+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:29:13.797054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T11:48:39.942194+0000","last_change":"2024-03-16T11:48:38.573469+0000","last_active":"2024-03-16T11:48:39.942194+0000","last_peered":"2024-03-16T11:48:39.942194+0000","last_clean":"2024-03-16T11:48:39.942194+0000","last_became_active":"2024-03-16T11:48:38.573155+0000","last_became_peered":"2024-03-16T11:48:38.573155+0000","last_unstale":"2024-03-16T11:48:39.942194+0000","last_undegraded":"2024-03-16T11:48:39.942194+0000","last_fullsized":"2024-03-16T11:48:39.942194+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:28:50.180821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T11:48:39.558166+0000","last_change":"2024-03-16T11:48:38.580913+0000","last_active":"2024-03-16T11:48:39.558166+0000","last_peered":"2024-03-16T11:48:39.558166+0000","last_clean":"2024-03-16T11:48:39.558166+0000","last_became_active":"2024-03-16T11:48:38.580573+0000","last_became_peered":"2024-03-16T11:48:38.580573+0000","last_unstale":"2024-03-16T11:48:39.558166+0000","last_undegraded":"2024-03-16T11:48:39.558166+0000","last_fullsized":"2024-03-16T11:48:39.558166+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:44:58.080502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T11:48:39.558089+0000","last_change":"2024-03-16T11:48:38.577168+0000","last_active":"2024-03-16T11:48:39.558089+0000","last_peered":"2024-03-16T11:48:39.558089+0000","last_clean":"2024-03-16T11:48:39.558089+0000","last_became_active":"2024-03-16T11:48:38.576845+0000","last_became_peered":"2024-03-16T11:48:38.576845+0000","last_unstale":"2024-03-16T11:48:39.558089+0000","last_undegraded":"2024-03-16T11:48:39.558089+0000","last_fullsized":"2024-03-16T11:48:39.558089+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:13:02.145610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T11:48:41.044247+0000","last_change":"2024-03-16T11:48:38.577477+0000","last_active":"2024-03-16T11:48:41.044247+0000","last_peered":"2024-03-16T11:48:41.044247+0000","last_clean":"2024-03-16T11:48:41.044247+0000","last_became_active":"2024-03-16T11:48:38.577126+0000","last_became_peered":"2024-03-16T11:48:38.577126+0000","last_unstale":"2024-03-16T11:48:41.044247+0000","last_undegraded":"2024-03-16T11:48:41.044247+0000","last_fullsized":"2024-03-16T11:48:41.044247+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:45:27.246827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T11:48:39.557457+0000","last_change":"2024-03-16T11:48:38.578508+0000","last_active":"2024-03-16T11:48:39.557457+0000","last_peered":"2024-03-16T11:48:39.557457+0000","last_clean":"2024-03-16T11:48:39.557457+0000","last_became_active":"2024-03-16T11:48:38.578125+0000","last_became_peered":"2024-03-16T11:48:38.578125+0000","last_unstale":"2024-03-16T11:48:39.557457+0000","last_undegraded":"2024-03-16T11:48:39.557457+0000","last_fullsized":"2024-03-16T11:48:39.557457+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:13:36.996405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T11:48:39.942338+0000","last_change":"2024-03-16T11:48:38.570450+0000","last_active":"2024-03-16T11:48:39.942338+0000","last_peered":"2024-03-16T11:48:39.942338+0000","last_clean":"2024-03-16T11:48:39.942338+0000","last_became_active":"2024-03-16T11:48:38.569952+0000","last_became_peered":"2024-03-16T11:48:38.569952+0000","last_unstale":"2024-03-16T11:48:39.942338+0000","last_undegraded":"2024-03-16T11:48:39.942338+0000","last_fullsized":"2024-03-16T11:48:39.942338+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:34:39.265969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T11:48:39.558576+0000","last_change":"2024-03-16T11:48:38.569568+0000","last_active":"2024-03-16T11:48:39.558576+0000","last_peered":"2024-03-16T11:48:39.558576+0000","last_clean":"2024-03-16T11:48:39.558576+0000","last_became_active":"2024-03-16T11:48:38.569260+0000","last_became_peered":"2024-03-16T11:48:38.569260+0000","last_unstale":"2024-03-16T11:48:39.558576+0000","last_undegraded":"2024-03-16T11:48:39.558576+0000","last_fullsized":"2024-03-16T11:48:39.558576+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:52:52.315980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T11:48:41.016231+0000","last_change":"2024-03-16T11:48:38.584562+0000","last_active":"2024-03-16T11:48:41.016231+0000","last_peered":"2024-03-16T11:48:41.016231+0000","last_clean":"2024-03-16T11:48:41.016231+0000","last_became_active":"2024-03-16T11:48:38.584096+0000","last_became_peered":"2024-03-16T11:48:38.584096+0000","last_unstale":"2024-03-16T11:48:41.016231+0000","last_undegraded":"2024-03-16T11:48:41.016231+0000","last_fullsized":"2024-03-16T11:48:41.016231+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:43:45.933252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T11:48:41.044218+0000","last_change":"2024-03-16T11:48:38.570174+0000","last_active":"2024-03-16T11:48:41.044218+0000","last_peered":"2024-03-16T11:48:41.044218+0000","last_clean":"2024-03-16T11:48:41.044218+0000","last_became_active":"2024-03-16T11:48:38.569865+0000","last_became_peered":"2024-03-16T11:48:38.569865+0000","last_unstale":"2024-03-16T11:48:41.044218+0000","last_undegraded":"2024-03-16T11:48:41.044218+0000","last_fullsized":"2024-03-16T11:48:41.044218+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T11:53:01.012481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T11:48:41.016256+0000","last_change":"2024-03-16T11:48:38.574868+0000","last_active":"2024-03-16T11:48:41.016256+0000","last_peered":"2024-03-16T11:48:41.016256+0000","last_clean":"2024-03-16T11:48:41.016256+0000","last_became_active":"2024-03-16T11:48:38.574434+0000","last_became_peered":"2024-03-16T11:48:38.574434+0000","last_unstale":"2024-03-16T11:48:41.016256+0000","last_undegraded":"2024-03-16T11:48:41.016256+0000","last_fullsized":"2024-03-16T11:48:41.016256+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:09:57.102253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T11:48:39.560883+0000","last_change":"2024-03-16T11:48:38.577189+0000","last_active":"2024-03-16T11:48:39.560883+0000","last_peered":"2024-03-16T11:48:39.560883+0000","last_clean":"2024-03-16T11:48:39.560883+0000","last_became_active":"2024-03-16T11:48:38.576884+0000","last_became_peered":"2024-03-16T11:48:38.576884+0000","last_unstale":"2024-03-16T11:48:39.560883+0000","last_undegraded":"2024-03-16T11:48:39.560883+0000","last_fullsized":"2024-03-16T11:48:39.560883+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:11:51.558517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T11:48:39.557803+0000","last_change":"2024-03-16T11:48:38.576238+0000","last_active":"2024-03-16T11:48:39.557803+0000","last_peered":"2024-03-16T11:48:39.557803+0000","last_clean":"2024-03-16T11:48:39.557803+0000","last_became_active":"2024-03-16T11:48:38.575951+0000","last_became_peered":"2024-03-16T11:48:38.575951+0000","last_unstale":"2024-03-16T11:48:39.557803+0000","last_undegraded":"2024-03-16T11:48:39.557803+0000","last_fullsized":"2024-03-16T11:48:39.557803+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:47:32.571580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-16T11:48:39.942117+0000","last_change":"2024-03-16T11:48:38.580025+0000","last_active":"2024-03-16T11:48:39.942117+0000","last_peered":"2024-03-16T11:48:39.942117+0000","last_clean":"2024-03-16T11:48:39.942117+0000","last_became_active":"2024-03-16T11:48:38.579689+0000","last_became_peered":"2024-03-16T11:48:38.579689+0000","last_unstale":"2024-03-16T11:48:39.942117+0000","last_undegraded":"2024-03-16T11:48:39.942117+0000","last_fullsized":"2024-03-16T11:48:39.942117+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:07:50.418306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":284036,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":28404,"num_objects":48,"num_object_clones":0,"num_object_copies":144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":0,"num_read":13,"num_read_kb":11,"num_write":110,"num_write_kb":54,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":32,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":241664,"data_stored":39819,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383695,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32072,"kb_used_data":896,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713080,"statfs":{"total":95995035648,"available":95962193920,"internally_reserved":0,"allocated":917504,"data_stored":439103,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645331,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28004,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717148,"statfs":{"total":95995035648,"available":95966359552,"internally_reserved":0,"allocated":1007616,"data_stored":1032079,"data_compressed":36118,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808857,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27932,"kb_used_data":912,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717220,"statfs":{"total":95995035648,"available":95966433280,"internally_reserved":0,"allocated":933888,"data_stored":439339,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070494,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32072,"kb_used_data":888,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713080,"statfs":{"total":95995035648,"available":95962193920,"internally_reserved":0,"allocated":909312,"data_stored":430555,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332131,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27972,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717180,"statfs":{"total":95995035648,"available":95966392320,"internally_reserved":0,"allocated":978944,"data_stored":1031072,"data_compressed":36118,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495657,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36232,"kb_used_data":896,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708920,"statfs":{"total":95995035648,"available":95957934080,"internally_reserved":0,"allocated":917504,"data_stored":438611,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574914,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32080,"kb_used_data":904,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713072,"statfs":{"total":95995035648,"available":95962185728,"internally_reserved":0,"allocated":925696,"data_stored":440133,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738439,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27940,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717212,"statfs":{"total":95995035648,"available":95966425088,"internally_reserved":0,"allocated":942080,"data_stored":439999,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411389,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32100,"kb_used_data":908,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713052,"statfs":{"total":95995035648,"available":95962165248,"internally_reserved":0,"allocated":929792,"data_stored":440167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182455,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36244,"kb_used_data":904,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708908,"statfs":{"total":95995035648,"available":95957921792,"internally_reserved":0,"allocated":925696,"data_stored":438715,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920819,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27916,"kb_used_data":896,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717236,"statfs":{"total":95995035648,"available":95966449664,"internally_reserved":0,"allocated":917504,"data_stored":438543,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757293,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32128,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713024,"statfs":{"total":95995035648,"available":95962136576,"internally_reserved":0,"allocated":962560,"data_stored":1020501,"data_compressed":36118,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34482,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34482,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34482,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36118,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36118,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36118,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36118,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36118,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2268,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":27479,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-16T11:48:46.118 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-16T11:48:46.118 INFO:tasks.cephadm:Setting up client nodes... 2024-03-16T11:48:46.119 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d1887f28-e389-11ee-95c9-87774f69a715 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-03-16T11:48:46.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:46 smithi029 ceph-mon[31975]: from='client.14784 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T11:48:46.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:46 smithi029 ceph-mon[31975]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-03-16T11:48:46.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:46 smithi029 ceph-mon[31975]: from='client.? 172.21.15.29:0/3936659366' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T11:48:46.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:46 smithi029 ceph-mon[31975]: from='client.? 172.21.15.29:0/2144194365' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T11:48:46.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:46 smithi029 ceph-mon[31975]: pgmap v294: 97 pgs: 97 active+clean; 604 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 7.2 KiB/s wr, 16 op/s 2024-03-16T11:48:46.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:46 smithi152 ceph-mon[36291]: from='client.14784 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T11:48:46.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:46 smithi152 ceph-mon[36291]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-03-16T11:48:46.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:46 smithi152 ceph-mon[36291]: from='client.? 172.21.15.29:0/3936659366' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T11:48:46.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:46 smithi152 ceph-mon[36291]: from='client.? 172.21.15.29:0/2144194365' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T11:48:46.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:46 smithi152 ceph-mon[36291]: pgmap v294: 97 pgs: 97 active+clean; 604 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 7.2 KiB/s wr, 16 op/s 2024-03-16T11:48:46.392 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/d1887f28-e389-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T11:48:46.545 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:46 smithi154 ceph-mon[36320]: from='client.14784 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T11:48:46.545 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:46 smithi154 ceph-mon[36320]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-03-16T11:48:46.545 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:46 smithi154 ceph-mon[36320]: from='client.? 172.21.15.29:0/3936659366' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T11:48:46.545 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:46 smithi154 ceph-mon[36320]: from='client.? 172.21.15.29:0/2144194365' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T11:48:46.545 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:46 smithi154 ceph-mon[36320]: pgmap v294: 97 pgs: 97 active+clean; 604 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 7.2 KiB/s wr, 16 op/s 2024-03-16T11:48:47.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:47 smithi029 ceph-mon[31975]: from='client.14802 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-16T11:48:47.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:47 smithi029 ceph-mon[31975]: from='client.14808 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-16T11:48:47.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:47 smithi152 ceph-mon[36291]: from='client.14802 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-16T11:48:47.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:47 smithi152 ceph-mon[36291]: from='client.14808 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-16T11:48:47.545 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:47 smithi154 ceph-mon[36320]: from='client.14802 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-16T11:48:47.545 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:47 smithi154 ceph-mon[36320]: from='client.14808 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-16T11:48:48.193 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:48.189+0000 7f9416451700 1 -- 172.21.15.29:0/2032076978 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9410103f50 msgr2=0x7f941010a800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:48:48.193 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:48.189+0000 7f9416451700 1 --2- 172.21.15.29:0/2032076978 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9410103f50 0x7f941010a800 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f940800b0a0 tx=0x7f940800b3b0 comp rx=0 tx=0).stop 2024-03-16T11:48:48.193 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:48.189+0000 7f9416451700 1 -- 172.21.15.29:0/2032076978 shutdown_connections 2024-03-16T11:48:48.193 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:48.189+0000 7f9416451700 1 --2- 172.21.15.29:0/2032076978 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9410103f50 0x7f941010a800 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:48.193 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:48.189+0000 7f9416451700 1 --2- 172.21.15.29:0/2032076978 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f94101005d0 0x7f94101009f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:48.193 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:48.189+0000 7f9416451700 1 --2- 172.21.15.29:0/2032076978 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f94101035b0 0x7f9410103a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:48.193 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:48.189+0000 7f9416451700 1 -- 172.21.15.29:0/2032076978 >> 172.21.15.29:0/2032076978 conn(0x7f94100fbf70 msgr2=0x7f94100fe390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:48:48.193 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:48.189+0000 7f9416451700 1 -- 172.21.15.29:0/2032076978 shutdown_connections 2024-03-16T11:48:48.193 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:48.189+0000 7f9416451700 1 -- 172.21.15.29:0/2032076978 wait complete. 2024-03-16T11:48:48.194 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:48.190+0000 7f9416451700 1 Processor -- start 2024-03-16T11:48:48.194 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:48.190+0000 7f9416451700 1 -- start start 2024-03-16T11:48:48.194 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:48.190+0000 7f9416451700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f94101005d0 0x7f94101983b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:48:48.194 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:48.190+0000 7f9416451700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f94101035b0 0x7f94101988f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:48:48.194 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:48.190+0000 7f9416451700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9410103f50 0x7f941019f970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:48:48.194 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:48.190+0000 7f9416451700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f941010dec0 con 0x7f94101005d0 2024-03-16T11:48:48.194 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:48.190+0000 7f9416451700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f941010dd40 con 0x7f9410103f50 2024-03-16T11:48:48.194 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:48.190+0000 7f9416451700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f941010e040 con 0x7f94101035b0 2024-03-16T11:48:48.194 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:48.190+0000 7f940ffff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f94101005d0 0x7f94101983b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:48:48.194 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:48.190+0000 7f940ffff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f94101005d0 0x7f94101983b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49286/0 (socket says 172.21.15.29:49286) 2024-03-16T11:48:48.194 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:48.190+0000 7f940ffff700 1 -- 172.21.15.29:0/2476408380 learned_addr learned my addr 172.21.15.29:0/2476408380 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:48:48.194 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:48.190+0000 7f9407fff700 1 --2- 172.21.15.29:0/2476408380 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f94101035b0 0x7f94101988f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:48:48.195 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:48.190+0000 7f94149ee700 1 --2- 172.21.15.29:0/2476408380 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9410103f50 0x7f941019f970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:48:48.195 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:48.191+0000 7f940ffff700 1 -- 172.21.15.29:0/2476408380 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f94101035b0 msgr2=0x7f94101988f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:48:48.195 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:48.191+0000 7f940ffff700 1 --2- 172.21.15.29:0/2476408380 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f94101035b0 0x7f94101988f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:48.195 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:48.191+0000 7f940ffff700 1 -- 172.21.15.29:0/2476408380 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9410103f50 msgr2=0x7f941019f970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:48:48.195 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:48.191+0000 7f940ffff700 1 --2- 172.21.15.29:0/2476408380 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9410103f50 0x7f941019f970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:48.195 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:48.191+0000 7f940ffff700 1 -- 172.21.15.29:0/2476408380 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94101a0010 con 0x7f94101005d0 2024-03-16T11:48:48.195 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:48.191+0000 7f94149ee700 1 --2- 172.21.15.29:0/2476408380 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9410103f50 0x7f941019f970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T11:48:48.195 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:48.191+0000 7f940ffff700 1 --2- 172.21.15.29:0/2476408380 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f94101005d0 0x7f94101983b0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f940000e9a0 tx=0x7f940000ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:48:48.195 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:48.191+0000 7f940dffb700 1 -- 172.21.15.29:0/2476408380 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f940000cbd0 con 0x7f94101005d0 2024-03-16T11:48:48.195 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:48.191+0000 7f940dffb700 1 -- 172.21.15.29:0/2476408380 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9400004500 con 0x7f94101005d0 2024-03-16T11:48:48.195 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:48.191+0000 7f9416451700 1 -- 172.21.15.29:0/2476408380 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f94101a0300 con 0x7f94101005d0 2024-03-16T11:48:48.195 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:48.191+0000 7f940dffb700 1 -- 172.21.15.29:0/2476408380 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9400003e00 con 0x7f94101005d0 2024-03-16T11:48:48.195 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:48.191+0000 7f9416451700 1 -- 172.21.15.29:0/2476408380 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f94101a0890 con 0x7f94101005d0 2024-03-16T11:48:48.195 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:48.192+0000 7f9416451700 1 -- 172.21.15.29:0/2476408380 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f94100507e0 con 0x7f94101005d0 2024-03-16T11:48:48.196 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:48.193+0000 7f940dffb700 1 -- 172.21.15.29:0/2476408380 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f9400004000 con 0x7f94101005d0 2024-03-16T11:48:48.196 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:48.193+0000 7f940dffb700 1 --2- 172.21.15.29:0/2476408380 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f93fc0a2a50 0x7f93fc0a4f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:48:48.196 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:48.194+0000 7f9407fff700 1 --2- 172.21.15.29:0/2476408380 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f93fc0a2a50 0x7f93fc0a4f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:48:48.196 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:48.194+0000 7f940dffb700 1 -- 172.21.15.29:0/2476408380 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f94000cecd0 con 0x7f94101005d0 2024-03-16T11:48:48.196 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:48.194+0000 7f9407fff700 1 --2- 172.21.15.29:0/2476408380 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f93fc0a2a50 0x7f93fc0a4f10 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f93f8009c60 tx=0x7f93f8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:48:48.198 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:48.198+0000 7f940dffb700 1 -- 172.21.15.29:0/2476408380 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f9400016330 con 0x7f94101005d0 2024-03-16T11:48:48.337 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:48 smithi029 ceph-mon[31975]: pgmap v295: 97 pgs: 97 active+clean; 606 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 7.4 KiB/s wr, 19 op/s 2024-03-16T11:48:48.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:48 smithi152 ceph-mon[36291]: pgmap v295: 97 pgs: 97 active+clean; 606 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 7.4 KiB/s wr, 19 op/s 2024-03-16T11:48:48.489 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:48.489+0000 7f9416451700 1 -- 172.21.15.29:0/2476408380 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) -- 0x7f9410062ee0 con 0x7f94101005d0 2024-03-16T11:48:48.491 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:48.491+0000 7f940dffb700 1 -- 172.21.15.29:0/2476408380 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v32) ==== 170+0+59 (secure 0 0 0) 0x7f940009bdc0 con 0x7f94101005d0 2024-03-16T11:48:48.491 INFO:teuthology.orchestra.run.smithi029.stdout:[client.0] 2024-03-16T11:48:48.491 INFO:teuthology.orchestra.run.smithi029.stdout: key = AQAgh/VlzvRKHRAADFb1H4FmJT4e6SuAcjV/9A== 2024-03-16T11:48:48.493 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:48.493+0000 7f9416451700 1 -- 172.21.15.29:0/2476408380 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f93fc0a2a50 msgr2=0x7f93fc0a4f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:48:48.493 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:48.493+0000 7f9416451700 1 --2- 172.21.15.29:0/2476408380 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f93fc0a2a50 0x7f93fc0a4f10 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f93f8009c60 tx=0x7f93f8009260 comp rx=0 tx=0).stop 2024-03-16T11:48:48.493 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:48.493+0000 7f9416451700 1 -- 172.21.15.29:0/2476408380 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f94101005d0 msgr2=0x7f94101983b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:48:48.493 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:48.493+0000 7f9416451700 1 --2- 172.21.15.29:0/2476408380 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f94101005d0 0x7f94101983b0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f940000e9a0 tx=0x7f940000ecb0 comp rx=0 tx=0).stop 2024-03-16T11:48:48.494 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:48.493+0000 7f9416451700 1 -- 172.21.15.29:0/2476408380 shutdown_connections 2024-03-16T11:48:48.494 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:48.493+0000 7f9416451700 1 --2- 172.21.15.29:0/2476408380 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9410103f50 0x7f941019f970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:48.494 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:48.493+0000 7f9416451700 1 --2- 172.21.15.29:0/2476408380 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f94101005d0 0x7f94101983b0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:48.494 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:48.493+0000 7f9416451700 1 --2- 172.21.15.29:0/2476408380 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f93fc0a2a50 0x7f93fc0a4f10 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:48.494 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:48.493+0000 7f9416451700 1 --2- 172.21.15.29:0/2476408380 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f94101035b0 0x7f94101988f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:48.494 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:48.493+0000 7f9416451700 1 -- 172.21.15.29:0/2476408380 >> 172.21.15.29:0/2476408380 conn(0x7f94100fbf70 msgr2=0x7f94101067e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:48:48.494 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:48.493+0000 7f9416451700 1 -- 172.21.15.29:0/2476408380 shutdown_connections 2024-03-16T11:48:48.494 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:48.493+0000 7f9416451700 1 -- 172.21.15.29:0/2476408380 wait complete. 2024-03-16T11:48:48.545 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:48 smithi154 ceph-mon[36320]: pgmap v295: 97 pgs: 97 active+clean; 606 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 7.4 KiB/s wr, 19 op/s 2024-03-16T11:48:49.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:49 smithi029 ceph-mon[31975]: from='client.? 172.21.15.29:0/2476408380' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-16T11:48:49.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:49 smithi029 ceph-mon[31975]: from='client.? 172.21.15.29:0/2476408380' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-16T11:48:49.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:49 smithi152 ceph-mon[36291]: from='client.? 172.21.15.29:0/2476408380' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-16T11:48:49.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:49 smithi152 ceph-mon[36291]: from='client.? 172.21.15.29:0/2476408380' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-16T11:48:49.545 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:49 smithi154 ceph-mon[36320]: from='client.? 172.21.15.29:0/2476408380' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-16T11:48:49.545 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:49 smithi154 ceph-mon[36320]: from='client.? 172.21.15.29:0/2476408380' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-16T11:48:49.562 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-03-16T11:48:49.562 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-03-16T11:48:49.562 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-16T11:48:49.608 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-03-16T11:48:49.608 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-03-16T11:48:49.608 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid d1887f28-e389-11ee-95c9-87774f69a715 -- ceph mgr dump --format=json 2024-03-16T11:48:49.895 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/d1887f28-e389-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T11:48:50.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:50 smithi154 ceph-mon[36320]: pgmap v296: 97 pgs: 97 active+clean; 627 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 9.4 KiB/s wr, 20 op/s 2024-03-16T11:48:50.337 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:50 smithi029 ceph-mon[31975]: pgmap v296: 97 pgs: 97 active+clean; 627 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 9.4 KiB/s wr, 20 op/s 2024-03-16T11:48:50.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:50 smithi152 ceph-mon[36291]: pgmap v296: 97 pgs: 97 active+clean; 627 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 9.4 KiB/s wr, 20 op/s 2024-03-16T11:48:51.763 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:51 smithi029 ceph-mon[31975]: pgmap v297: 97 pgs: 97 active+clean; 627 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 7.9 KiB/s wr, 17 op/s 2024-03-16T11:48:51.869 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:51.868+0000 7f16886d7700 1 -- 172.21.15.29:0/3919789427 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1680107640 msgr2=0x7f168010f1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:48:51.869 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:51.868+0000 7f16886d7700 1 --2- 172.21.15.29:0/3919789427 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1680107640 0x7f168010f1c0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f167800b0d0 tx=0x7f167800b3e0 comp rx=0 tx=0).stop 2024-03-16T11:48:51.869 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:51.868+0000 7f16886d7700 1 -- 172.21.15.29:0/3919789427 shutdown_connections 2024-03-16T11:48:51.869 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:51.869+0000 7f16886d7700 1 --2- 172.21.15.29:0/3919789427 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1680107640 0x7f168010f1c0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:51.870 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:51.869+0000 7f16886d7700 1 --2- 172.21.15.29:0/3919789427 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f168006abd0 0x7f168006afb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:51.870 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:51.869+0000 7f16886d7700 1 --2- 172.21.15.29:0/3919789427 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f168006b580 0x7f1680106f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:51.870 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:51.869+0000 7f16886d7700 1 -- 172.21.15.29:0/3919789427 >> 172.21.15.29:0/3919789427 conn(0x7f1680078140 msgr2=0x7f1680078550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:48:51.870 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:51.869+0000 7f16886d7700 1 -- 172.21.15.29:0/3919789427 shutdown_connections 2024-03-16T11:48:51.870 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:51.869+0000 7f16886d7700 1 -- 172.21.15.29:0/3919789427 wait complete. 2024-03-16T11:48:51.870 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:51.869+0000 7f16886d7700 1 Processor -- start 2024-03-16T11:48:51.870 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:51.869+0000 7f16886d7700 1 -- start start 2024-03-16T11:48:51.872 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:51.869+0000 7f16886d7700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f168006abd0 0x7f168019e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:48:51.872 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:51.869+0000 7f16886d7700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f168006b580 0x7f168019ec10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:48:51.872 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:51.869+0000 7f16886d7700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1680107640 0x7f16801988e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:48:51.872 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:51.869+0000 7f16886d7700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f1680112600 con 0x7f168006b580 2024-03-16T11:48:51.872 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:51.869+0000 7f16886d7700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f1680112480 con 0x7f168006abd0 2024-03-16T11:48:51.872 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:51.869+0000 7f16886d7700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f1680112780 con 0x7f1680107640 2024-03-16T11:48:51.872 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:51.870+0000 7f1685c72700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f168006b580 0x7f168019ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:48:51.872 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:51.870+0000 7f1685c72700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f168006b580 0x7f168019ec10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57132/0 (socket says 172.21.15.29:57132) 2024-03-16T11:48:51.872 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:51.870+0000 7f1685c72700 1 -- 172.21.15.29:0/3464666659 learned_addr learned my addr 172.21.15.29:0/3464666659 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:48:51.872 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:51.870+0000 7f1686c74700 1 --2- 172.21.15.29:0/3464666659 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1680107640 0x7f16801988e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:48:51.873 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:51.870+0000 7f1686473700 1 --2- 172.21.15.29:0/3464666659 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f168006abd0 0x7f168019e6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:48:51.873 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:51.870+0000 7f1685c72700 1 -- 172.21.15.29:0/3464666659 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1680107640 msgr2=0x7f16801988e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:48:51.873 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:51.870+0000 7f1685c72700 1 --2- 172.21.15.29:0/3464666659 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1680107640 0x7f16801988e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:51.873 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:51.870+0000 7f1685c72700 1 -- 172.21.15.29:0/3464666659 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f168006abd0 msgr2=0x7f168019e6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:48:51.873 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:51.870+0000 7f1685c72700 1 --2- 172.21.15.29:0/3464666659 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f168006abd0 0x7f168019e6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:51.873 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:51.870+0000 7f1685c72700 1 -- 172.21.15.29:0/3464666659 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1680199040 con 0x7f168006b580 2024-03-16T11:48:51.873 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:51.870+0000 7f1686c74700 1 --2- 172.21.15.29:0/3464666659 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1680107640 0x7f16801988e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:48:51.873 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:51.870+0000 7f1685c72700 1 --2- 172.21.15.29:0/3464666659 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f168006b580 0x7f168019ec10 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f167000e9c0 tx=0x7f167000ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:48:51.873 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:51.870+0000 7f16777fe700 1 -- 172.21.15.29:0/3464666659 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1670004a30 con 0x7f168006b580 2024-03-16T11:48:51.873 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:51.870+0000 7f16777fe700 1 -- 172.21.15.29:0/3464666659 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f167000cca0 con 0x7f168006b580 2024-03-16T11:48:51.873 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:51.870+0000 7f16886d7700 1 -- 172.21.15.29:0/3464666659 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1680199330 con 0x7f168006b580 2024-03-16T11:48:51.873 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:51.870+0000 7f16777fe700 1 -- 172.21.15.29:0/3464666659 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1670005690 con 0x7f168006b580 2024-03-16T11:48:51.874 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:51.870+0000 7f16886d7700 1 -- 172.21.15.29:0/3464666659 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f16801a55a0 con 0x7f168006b580 2024-03-16T11:48:51.874 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:51.871+0000 7f16886d7700 1 -- 172.21.15.29:0/3464666659 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f16801039f0 con 0x7f168006b580 2024-03-16T11:48:51.874 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:51.873+0000 7f16777fe700 1 -- 172.21.15.29:0/3464666659 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f1670004cf0 con 0x7f168006b580 2024-03-16T11:48:51.874 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:51.873+0000 7f16777fe700 1 --2- 172.21.15.29:0/3464666659 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f166c0a29a0 0x7f166c0a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:48:51.874 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:51.873+0000 7f1686473700 1 --2- 172.21.15.29:0/3464666659 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f166c0a29a0 0x7f166c0a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:48:51.875 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:51.873+0000 7f16777fe700 1 -- 172.21.15.29:0/3464666659 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f16700cecf0 con 0x7f168006b580 2024-03-16T11:48:51.875 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:51.874+0000 7f1686473700 1 --2- 172.21.15.29:0/3464666659 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f166c0a29a0 0x7f166c0a4e60 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f1680107180 tx=0x7f167c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:48:51.877 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:51.877+0000 7f16777fe700 1 -- 172.21.15.29:0/3464666659 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f1670053740 con 0x7f168006b580 2024-03-16T11:48:52.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:51 smithi154 ceph-mon[36320]: pgmap v297: 97 pgs: 97 active+clean; 627 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 7.9 KiB/s wr, 17 op/s 2024-03-16T11:48:52.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:51 smithi152 ceph-mon[36291]: pgmap v297: 97 pgs: 97 active+clean; 627 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 7.9 KiB/s wr, 17 op/s 2024-03-16T11:48:52.147 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:52.147+0000 7f16886d7700 1 -- 172.21.15.29:0/3464666659 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7f168002db70 con 0x7f168006b580 2024-03-16T11:48:52.152 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:52.152+0000 7f16777fe700 1 -- 172.21.15.29:0/3464666659 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v17) ==== 74+0+262659 (secure 0 0 0) 0x7f167009bde0 con 0x7f168006b580 2024-03-16T11:48:52.155 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-16T11:48:52.157 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:52.156+0000 7f16886d7700 1 -- 172.21.15.29:0/3464666659 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f166c0a29a0 msgr2=0x7f166c0a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:48:52.158 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:52.156+0000 7f16886d7700 1 --2- 172.21.15.29:0/3464666659 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f166c0a29a0 0x7f166c0a4e60 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f1680107180 tx=0x7f167c009260 comp rx=0 tx=0).stop 2024-03-16T11:48:52.158 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:52.156+0000 7f16886d7700 1 -- 172.21.15.29:0/3464666659 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f168006b580 msgr2=0x7f168019ec10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:48:52.158 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:52.156+0000 7f16886d7700 1 --2- 172.21.15.29:0/3464666659 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f168006b580 0x7f168019ec10 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f167000e9c0 tx=0x7f167000ecd0 comp rx=0 tx=0).stop 2024-03-16T11:48:52.158 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:52.156+0000 7f16886d7700 1 -- 172.21.15.29:0/3464666659 shutdown_connections 2024-03-16T11:48:52.158 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:52.156+0000 7f16886d7700 1 --2- 172.21.15.29:0/3464666659 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f166c0a29a0 0x7f166c0a4e60 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:52.158 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:52.156+0000 7f16886d7700 1 --2- 172.21.15.29:0/3464666659 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1680107640 0x7f16801988e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:52.158 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:52.156+0000 7f16886d7700 1 --2- 172.21.15.29:0/3464666659 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f168006abd0 0x7f168019e6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:52.158 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:52.156+0000 7f16886d7700 1 --2- 172.21.15.29:0/3464666659 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f168006b580 0x7f168019ec10 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:52.158 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:52.156+0000 7f16886d7700 1 -- 172.21.15.29:0/3464666659 >> 172.21.15.29:0/3464666659 conn(0x7f1680078140 msgr2=0x7f16800ff3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:48:52.158 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:52.156+0000 7f16886d7700 1 -- 172.21.15.29:0/3464666659 shutdown_connections 2024-03-16T11:48:52.158 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:52.156+0000 7f16886d7700 1 -- 172.21.15.29:0/3464666659 wait complete. 2024-03-16T11:48:52.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:52 smithi029 ceph-mon[31975]: from='client.? 172.21.15.29:0/3464666659' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-16T11:48:53.045 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":17,"flags":0,"active_gid":14152,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":4192064640},{"type":"v1","addr":"172.21.15.29:6801","nonce":4192064640}]},"active_addr":"172.21.15.29:6801/4192064640","active_change":"2024-03-16T11:41:27.531846+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":24109,"name":"z","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]},{"gid":24110,"name":"y","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.29:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.29:0","nonce":340047334}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.29:0","nonce":2374713213}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.29:0","nonce":1114950131}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.29:0","nonce":623985488}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.29:0","nonce":3134170177}]}]} 2024-03-16T11:48:53.050 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-03-16T11:48:53.050 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-03-16T11:48:53.050 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid d1887f28-e389-11ee-95c9-87774f69a715 -- ceph osd dump --format=json 2024-03-16T11:48:53.051 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:52 smithi154 ceph-mon[36320]: from='client.? 172.21.15.29:0/3464666659' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-16T11:48:53.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:52 smithi152 ceph-mon[36291]: from='client.? 172.21.15.29:0/3464666659' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-16T11:48:53.296 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/d1887f28-e389-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T11:48:54.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:53 smithi154 ceph-mon[36320]: pgmap v298: 97 pgs: 97 active+clean; 627 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 7.4 KiB/s wr, 16 op/s 2024-03-16T11:48:54.067 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:53 smithi152 ceph-mon[36291]: pgmap v298: 97 pgs: 97 active+clean; 627 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 7.4 KiB/s wr, 16 op/s 2024-03-16T11:48:54.087 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:53 smithi029 ceph-mon[31975]: pgmap v298: 97 pgs: 97 active+clean; 627 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 7.4 KiB/s wr, 16 op/s 2024-03-16T11:48:55.214 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:55.214+0000 7f947b98b700 1 -- 172.21.15.29:0/964921326 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9474102ce0 msgr2=0x7f947410f120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:48:55.214 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:55.214+0000 7f947b98b700 1 --2- 172.21.15.29:0/964921326 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9474102ce0 0x7f947410f120 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f946c00b0d0 tx=0x7f946c00b3e0 comp rx=0 tx=0).stop 2024-03-16T11:48:55.214 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:55.214+0000 7f947b98b700 1 -- 172.21.15.29:0/964921326 shutdown_connections 2024-03-16T11:48:55.214 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:55.214+0000 7f947b98b700 1 --2- 172.21.15.29:0/964921326 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9474102ce0 0x7f947410f120 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:55.215 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:55.214+0000 7f947b98b700 1 --2- 172.21.15.29:0/964921326 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9474108300 0x7f94741086e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:55.215 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:55.214+0000 7f947b98b700 1 --2- 172.21.15.29:0/964921326 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9474102320 0x7f94741027a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:55.215 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:55.214+0000 7f947b98b700 1 -- 172.21.15.29:0/964921326 >> 172.21.15.29:0/964921326 conn(0x7f9474079400 msgr2=0x7f9474079810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:48:55.215 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:55.214+0000 7f947b98b700 1 -- 172.21.15.29:0/964921326 shutdown_connections 2024-03-16T11:48:55.215 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:55.214+0000 7f947b98b700 1 -- 172.21.15.29:0/964921326 wait complete. 2024-03-16T11:48:55.216 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:55.215+0000 7f947b98b700 1 Processor -- start 2024-03-16T11:48:55.216 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:55.215+0000 7f947b98b700 1 -- start start 2024-03-16T11:48:55.217 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:55.215+0000 7f947b98b700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9474102320 0x7f947419e600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:48:55.217 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:55.215+0000 7f947b98b700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9474102ce0 0x7f947419eb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:48:55.217 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:55.215+0000 7f947b98b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9474108300 0x7f9474198780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:48:55.217 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:55.215+0000 7f947b98b700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f94741125a0 con 0x7f9474108300 2024-03-16T11:48:55.217 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:55.215+0000 7f947b98b700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f9474112420 con 0x7f9474102ce0 2024-03-16T11:48:55.217 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:55.215+0000 7f947b98b700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f9474112720 con 0x7f9474102320 2024-03-16T11:48:55.217 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:55.215+0000 7f9479f28700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9474108300 0x7f9474198780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:48:55.217 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:55.215+0000 7f9479727700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9474102320 0x7f947419e600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:48:55.217 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:55.215+0000 7f9478f26700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9474102ce0 0x7f947419eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:48:55.217 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:55.215+0000 7f9478f26700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9474102ce0 0x7f947419eb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.29:60558/0 (socket says 172.21.15.29:60558) 2024-03-16T11:48:55.217 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:55.215+0000 7f9479f28700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9474108300 0x7f9474198780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57150/0 (socket says 172.21.15.29:57150) 2024-03-16T11:48:55.217 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:55.215+0000 7f9478f26700 1 -- 172.21.15.29:0/3585188251 learned_addr learned my addr 172.21.15.29:0/3585188251 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:48:55.217 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:55.215+0000 7f9479727700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9474102320 0x7f947419e600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.29:41940/0 (socket says 172.21.15.29:41940) 2024-03-16T11:48:55.217 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:55.215+0000 7f9479f28700 1 -- 172.21.15.29:0/3585188251 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9474102320 msgr2=0x7f947419e600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:48:55.218 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:55.215+0000 7f9479f28700 1 --2- 172.21.15.29:0/3585188251 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9474102320 0x7f947419e600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:55.218 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:55.215+0000 7f9479f28700 1 -- 172.21.15.29:0/3585188251 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9474102ce0 msgr2=0x7f947419eb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:48:55.218 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:55.215+0000 7f9479f28700 1 --2- 172.21.15.29:0/3585188251 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9474102ce0 0x7f947419eb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:55.218 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:55.215+0000 7f9479f28700 1 -- 172.21.15.29:0/3585188251 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9474198f60 con 0x7f9474108300 2024-03-16T11:48:55.218 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:55.215+0000 7f9479f28700 1 --2- 172.21.15.29:0/3585188251 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9474108300 0x7f9474198780 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f946c00b770 tx=0x7f946c007a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:48:55.218 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:55.216+0000 7f946a7fc700 1 -- 172.21.15.29:0/3585188251 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f946c021020 con 0x7f9474108300 2024-03-16T11:48:55.218 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:55.216+0000 7f946a7fc700 1 -- 172.21.15.29:0/3585188251 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f946c004c50 con 0x7f9474108300 2024-03-16T11:48:55.218 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:55.216+0000 7f947b98b700 1 -- 172.21.15.29:0/3585188251 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f94741991f0 con 0x7f9474108300 2024-03-16T11:48:55.218 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:55.216+0000 7f946a7fc700 1 -- 172.21.15.29:0/3585188251 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f946c0052b0 con 0x7f9474108300 2024-03-16T11:48:55.218 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:55.216+0000 7f947b98b700 1 -- 172.21.15.29:0/3585188251 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f94741a5460 con 0x7f9474108300 2024-03-16T11:48:55.218 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:55.217+0000 7f947b98b700 1 -- 172.21.15.29:0/3585188251 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f94741039b0 con 0x7f9474108300 2024-03-16T11:48:55.219 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:55.218+0000 7f946a7fc700 1 -- 172.21.15.29:0/3585188251 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f946c024020 con 0x7f9474108300 2024-03-16T11:48:55.219 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:55.219+0000 7f946a7fc700 1 --2- 172.21.15.29:0/3585188251 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f94600a2920 0x7f94600a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:48:55.219 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:55.219+0000 7f9479727700 1 --2- 172.21.15.29:0/3585188251 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f94600a2920 0x7f94600a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:48:55.219 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:55.219+0000 7f946a7fc700 1 -- 172.21.15.29:0/3585188251 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f946c00f020 con 0x7f9474108300 2024-03-16T11:48:55.220 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:55.220+0000 7f9479727700 1 --2- 172.21.15.29:0/3585188251 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f94600a2920 0x7f94600a4de0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f9470009c60 tx=0x7f9470009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:48:55.223 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:55.223+0000 7f946a7fc700 1 -- 172.21.15.29:0/3585188251 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f946c01de20 con 0x7f9474108300 2024-03-16T11:48:55.427 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:55.427+0000 7f947b98b700 1 -- 172.21.15.29:0/3585188251 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f947402db70 con 0x7f9474108300 2024-03-16T11:48:55.429 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:55.429+0000 7f946a7fc700 1 -- 172.21.15.29:0/3585188251 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) ==== 74+0+22233 (secure 0 0 0) 0x7f946c09cfb0 con 0x7f9474108300 2024-03-16T11:48:55.429 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-16T11:48:55.429 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":85,"fsid":"d1887f28-e389-11ee-95c9-87774f69a715","created":"2024-03-16T11:40:16.475063+0000","modified":"2024-03-16T11:48:39.554402+0000","last_up_change":"2024-03-16T11:47:44.349348+0000","last_in_change":"2024-03-16T11:47:23.672432+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-16T11:43:51.892652+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-16T11:48:36.709732+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-16T11:48:37.586140+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"60f5608f-a408-45ba-bdf6-5b53758f88fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":2748947913},{"type":"v1","addr":"172.21.15.29:6803","nonce":2748947913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":2748947913},{"type":"v1","addr":"172.21.15.29:6805","nonce":2748947913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":2748947913},{"type":"v1","addr":"172.21.15.29:6809","nonce":2748947913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":2748947913},{"type":"v1","addr":"172.21.15.29:6807","nonce":2748947913}]},"public_addr":"172.21.15.29:6803/2748947913","cluster_addr":"172.21.15.29:6805/2748947913","heartbeat_back_addr":"172.21.15.29:6809/2748947913","heartbeat_front_addr":"172.21.15.29:6807/2748947913","state":["exists","up"]},{"osd":1,"uuid":"e70b53d6-ee78-4e56-9a2c-263ae0501b43","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6800","nonce":2339408422},{"type":"v1","addr":"172.21.15.152:6801","nonce":2339408422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6802","nonce":2339408422},{"type":"v1","addr":"172.21.15.152:6803","nonce":2339408422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6806","nonce":2339408422},{"type":"v1","addr":"172.21.15.152:6807","nonce":2339408422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6804","nonce":2339408422},{"type":"v1","addr":"172.21.15.152:6805","nonce":2339408422}]},"public_addr":"172.21.15.152:6801/2339408422","cluster_addr":"172.21.15.152:6803/2339408422","heartbeat_back_addr":"172.21.15.152:6807/2339408422","heartbeat_front_addr":"172.21.15.152:6805/2339408422","state":["exists","up"]},{"osd":2,"uuid":"ff39389a-dfac-44b5-80ff-36abca316e78","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6800","nonce":4288325280},{"type":"v1","addr":"172.21.15.154:6801","nonce":4288325280}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6802","nonce":4288325280},{"type":"v1","addr":"172.21.15.154:6803","nonce":4288325280}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6806","nonce":4288325280},{"type":"v1","addr":"172.21.15.154:6807","nonce":4288325280}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":4288325280},{"type":"v1","addr":"172.21.15.154:6805","nonce":4288325280}]},"public_addr":"172.21.15.154:6801/4288325280","cluster_addr":"172.21.15.154:6803/4288325280","heartbeat_back_addr":"172.21.15.154:6807/4288325280","heartbeat_front_addr":"172.21.15.154:6805/4288325280","state":["exists","up"]},{"osd":3,"uuid":"069e1795-55bc-4324-b94c-eb24ef2334c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":1538750397},{"type":"v1","addr":"172.21.15.29:6811","nonce":1538750397}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":1538750397},{"type":"v1","addr":"172.21.15.29:6813","nonce":1538750397}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":1538750397},{"type":"v1","addr":"172.21.15.29:6817","nonce":1538750397}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":1538750397},{"type":"v1","addr":"172.21.15.29:6815","nonce":1538750397}]},"public_addr":"172.21.15.29:6811/1538750397","cluster_addr":"172.21.15.29:6813/1538750397","heartbeat_back_addr":"172.21.15.29:6817/1538750397","heartbeat_front_addr":"172.21.15.29:6815/1538750397","state":["exists","up"]},{"osd":4,"uuid":"5a9d43a1-e8f5-4ab5-a441-afbc6ed4066e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6808","nonce":2795081750},{"type":"v1","addr":"172.21.15.152:6809","nonce":2795081750}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6810","nonce":2795081750},{"type":"v1","addr":"172.21.15.152:6811","nonce":2795081750}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6814","nonce":2795081750},{"type":"v1","addr":"172.21.15.152:6815","nonce":2795081750}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6812","nonce":2795081750},{"type":"v1","addr":"172.21.15.152:6813","nonce":2795081750}]},"public_addr":"172.21.15.152:6809/2795081750","cluster_addr":"172.21.15.152:6811/2795081750","heartbeat_back_addr":"172.21.15.152:6815/2795081750","heartbeat_front_addr":"172.21.15.152:6813/2795081750","state":["exists","up"]},{"osd":5,"uuid":"234de428-4ac7-44be-9a54-7b58331fdf8c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":1472686560},{"type":"v1","addr":"172.21.15.154:6809","nonce":1472686560}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6810","nonce":1472686560},{"type":"v1","addr":"172.21.15.154:6811","nonce":1472686560}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6814","nonce":1472686560},{"type":"v1","addr":"172.21.15.154:6815","nonce":1472686560}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6812","nonce":1472686560},{"type":"v1","addr":"172.21.15.154:6813","nonce":1472686560}]},"public_addr":"172.21.15.154:6809/1472686560","cluster_addr":"172.21.15.154:6811/1472686560","heartbeat_back_addr":"172.21.15.154:6815/1472686560","heartbeat_front_addr":"172.21.15.154:6813/1472686560","state":["exists","up"]},{"osd":6,"uuid":"ae45b8d9-7883-40c9-a9cd-4c92b601d4f2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":816506244},{"type":"v1","addr":"172.21.15.29:6819","nonce":816506244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":816506244},{"type":"v1","addr":"172.21.15.29:6821","nonce":816506244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":816506244},{"type":"v1","addr":"172.21.15.29:6825","nonce":816506244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":816506244},{"type":"v1","addr":"172.21.15.29:6823","nonce":816506244}]},"public_addr":"172.21.15.29:6819/816506244","cluster_addr":"172.21.15.29:6821/816506244","heartbeat_back_addr":"172.21.15.29:6825/816506244","heartbeat_front_addr":"172.21.15.29:6823/816506244","state":["exists","up"]},{"osd":7,"uuid":"6f687b28-4fce-4d4a-8b4e-294173d9d6c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6816","nonce":2924651787},{"type":"v1","addr":"172.21.15.152:6817","nonce":2924651787}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6818","nonce":2924651787},{"type":"v1","addr":"172.21.15.152:6819","nonce":2924651787}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6822","nonce":2924651787},{"type":"v1","addr":"172.21.15.152:6823","nonce":2924651787}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6820","nonce":2924651787},{"type":"v1","addr":"172.21.15.152:6821","nonce":2924651787}]},"public_addr":"172.21.15.152:6817/2924651787","cluster_addr":"172.21.15.152:6819/2924651787","heartbeat_back_addr":"172.21.15.152:6823/2924651787","heartbeat_front_addr":"172.21.15.152:6821/2924651787","state":["exists","up"]},{"osd":8,"uuid":"a25fb741-caa7-429a-b540-e394d034b446","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6816","nonce":129974156},{"type":"v1","addr":"172.21.15.154:6817","nonce":129974156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6818","nonce":129974156},{"type":"v1","addr":"172.21.15.154:6819","nonce":129974156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6822","nonce":129974156},{"type":"v1","addr":"172.21.15.154:6823","nonce":129974156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6820","nonce":129974156},{"type":"v1","addr":"172.21.15.154:6821","nonce":129974156}]},"public_addr":"172.21.15.154:6817/129974156","cluster_addr":"172.21.15.154:6819/129974156","heartbeat_back_addr":"172.21.15.154:6823/129974156","heartbeat_front_addr":"172.21.15.154:6821/129974156","state":["exists","up"]},{"osd":9,"uuid":"1071cb0e-8c5f-4cd8-9805-0045e220d946","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6826","nonce":2951414243},{"type":"v1","addr":"172.21.15.29:6827","nonce":2951414243}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":2951414243},{"type":"v1","addr":"172.21.15.29:6829","nonce":2951414243}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":2951414243},{"type":"v1","addr":"172.21.15.29:6833","nonce":2951414243}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":2951414243},{"type":"v1","addr":"172.21.15.29:6831","nonce":2951414243}]},"public_addr":"172.21.15.29:6827/2951414243","cluster_addr":"172.21.15.29:6829/2951414243","heartbeat_back_addr":"172.21.15.29:6833/2951414243","heartbeat_front_addr":"172.21.15.29:6831/2951414243","state":["exists","up"]},{"osd":10,"uuid":"ee7c7d74-3f39-4cd4-be80-843fb1dd9ecc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6824","nonce":3412525480},{"type":"v1","addr":"172.21.15.152:6825","nonce":3412525480}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6826","nonce":3412525480},{"type":"v1","addr":"172.21.15.152:6827","nonce":3412525480}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6830","nonce":3412525480},{"type":"v1","addr":"172.21.15.152:6831","nonce":3412525480}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6828","nonce":3412525480},{"type":"v1","addr":"172.21.15.152:6829","nonce":3412525480}]},"public_addr":"172.21.15.152:6825/3412525480","cluster_addr":"172.21.15.152:6827/3412525480","heartbeat_back_addr":"172.21.15.152:6831/3412525480","heartbeat_front_addr":"172.21.15.152:6829/3412525480","state":["exists","up"]},{"osd":11,"uuid":"4598c803-984a-4de6-ac22-066502c17889","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6824","nonce":1160927966},{"type":"v1","addr":"172.21.15.154:6825","nonce":1160927966}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6826","nonce":1160927966},{"type":"v1","addr":"172.21.15.154:6827","nonce":1160927966}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6830","nonce":1160927966},{"type":"v1","addr":"172.21.15.154:6831","nonce":1160927966}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6828","nonce":1160927966},{"type":"v1","addr":"172.21.15.154:6829","nonce":1160927966}]},"public_addr":"172.21.15.154:6825/1160927966","cluster_addr":"172.21.15.154:6827/1160927966","heartbeat_back_addr":"172.21.15.154:6831/1160927966","heartbeat_front_addr":"172.21.15.154:6829/1160927966","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:43:02.343393+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:43:25.209151+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:43:48.946164+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:44:14.588424+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:44:39.700025+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:45:04.061197+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:45:31.214508+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:45:56.937296+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:46:22.768915+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:46:50.101011+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:47:16.298041+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:47:42.402095+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.29:0/1853086886":"2024-03-17T11:41:27.531724+0000","172.21.15.29:0/1211659364":"2024-03-17T11:41:27.531724+0000","172.21.15.29:6800/2798831128":"2024-03-17T11:41:27.531724+0000","172.21.15.29:0/1281028285":"2024-03-17T11:41:00.505678+0000","172.21.15.29:6800/2275315654":"2024-03-17T11:41:00.505678+0000","172.21.15.29:6801/2798831128":"2024-03-17T11:41:27.531724+0000","172.21.15.29:6801/2275315654":"2024-03-17T11:41:00.505678+0000","172.21.15.29:0/1330100009":"2024-03-17T11:41:00.505678+0000","172.21.15.29:0/1784249039":"2024-03-17T11:41:00.505678+0000","172.21.15.29:0/1116486932":"2024-03-17T11:41:27.531724+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-16T11:48:55.431 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:55.431+0000 7f947b98b700 1 -- 172.21.15.29:0/3585188251 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f94600a2920 msgr2=0x7f94600a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:48:55.431 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:55.431+0000 7f947b98b700 1 --2- 172.21.15.29:0/3585188251 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f94600a2920 0x7f94600a4de0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f9470009c60 tx=0x7f9470009260 comp rx=0 tx=0).stop 2024-03-16T11:48:55.431 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:55.431+0000 7f947b98b700 1 -- 172.21.15.29:0/3585188251 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9474108300 msgr2=0x7f9474198780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:48:55.431 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:55.431+0000 7f947b98b700 1 --2- 172.21.15.29:0/3585188251 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9474108300 0x7f9474198780 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f946c00b770 tx=0x7f946c007a20 comp rx=0 tx=0).stop 2024-03-16T11:48:55.431 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:55.431+0000 7f947b98b700 1 -- 172.21.15.29:0/3585188251 shutdown_connections 2024-03-16T11:48:55.432 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:55.431+0000 7f947b98b700 1 --2- 172.21.15.29:0/3585188251 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9474108300 0x7f9474198780 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:55.432 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:55.431+0000 7f947b98b700 1 --2- 172.21.15.29:0/3585188251 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f94600a2920 0x7f94600a4de0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:55.432 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:55.431+0000 7f947b98b700 1 --2- 172.21.15.29:0/3585188251 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9474102320 0x7f947419e600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:55.432 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:55.431+0000 7f947b98b700 1 --2- 172.21.15.29:0/3585188251 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9474102ce0 0x7f947419eb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:55.432 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:55.431+0000 7f947b98b700 1 -- 172.21.15.29:0/3585188251 >> 172.21.15.29:0/3585188251 conn(0x7f9474079400 msgr2=0x7f9474100ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:48:55.432 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:55.431+0000 7f947b98b700 1 -- 172.21.15.29:0/3585188251 shutdown_connections 2024-03-16T11:48:55.432 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:55.431+0000 7f947b98b700 1 -- 172.21.15.29:0/3585188251 wait complete. 2024-03-16T11:48:55.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:55 smithi154 ceph-mon[36320]: from='client.? 172.21.15.29:0/3585188251' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T11:48:55.834 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:55 smithi029 ceph-mon[31975]: from='client.? 172.21.15.29:0/3585188251' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T11:48:55.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:55 smithi152 ceph-mon[36291]: from='client.? 172.21.15.29:0/3585188251' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T11:48:56.203 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-03-16T11:48:56.203 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid d1887f28-e389-11ee-95c9-87774f69a715 -- ceph osd dump --format=json 2024-03-16T11:48:56.452 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/d1887f28-e389-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T11:48:56.738 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:56 smithi029 ceph-mon[31975]: pgmap v299: 97 pgs: 97 active+clean; 636 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 6.7 KiB/s wr, 13 op/s 2024-03-16T11:48:56.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:56 smithi154 ceph-mon[36320]: pgmap v299: 97 pgs: 97 active+clean; 636 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 6.7 KiB/s wr, 13 op/s 2024-03-16T11:48:56.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:56 smithi152 ceph-mon[36291]: pgmap v299: 97 pgs: 97 active+clean; 636 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 6.7 KiB/s wr, 13 op/s 2024-03-16T11:48:58.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:57 smithi154 ceph-mon[36320]: pgmap v300: 97 pgs: 97 active+clean; 636 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 4.2 KiB/s wr, 11 op/s 2024-03-16T11:48:58.058 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:57 smithi029 ceph-mon[31975]: pgmap v300: 97 pgs: 97 active+clean; 636 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 4.2 KiB/s wr, 11 op/s 2024-03-16T11:48:58.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:57 smithi152 ceph-mon[36291]: pgmap v300: 97 pgs: 97 active+clean; 636 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 4.2 KiB/s wr, 11 op/s 2024-03-16T11:48:58.293 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:58.290+0000 7f337aae9700 1 -- 172.21.15.29:0/592186534 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f33741083a0 msgr2=0x7f3374108780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:48:58.293 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:58.290+0000 7f337aae9700 1 --2- 172.21.15.29:0/592186534 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f33741083a0 0x7f3374108780 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f3368009990 tx=0x7f3368009ca0 comp rx=0 tx=0).stop 2024-03-16T11:48:58.293 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:58.290+0000 7f337aae9700 1 -- 172.21.15.29:0/592186534 shutdown_connections 2024-03-16T11:48:58.293 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:58.290+0000 7f337aae9700 1 --2- 172.21.15.29:0/592186534 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3374102d10 0x7f337410f1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:58.294 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:58.290+0000 7f337aae9700 1 --2- 172.21.15.29:0/592186534 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f33741083a0 0x7f3374108780 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:58.294 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:58.290+0000 7f337aae9700 1 --2- 172.21.15.29:0/592186534 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3374102350 0x7f33741027d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:58.294 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:58.290+0000 7f337aae9700 1 -- 172.21.15.29:0/592186534 >> 172.21.15.29:0/592186534 conn(0x7f3374079420 msgr2=0x7f3374079830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:48:58.294 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:58.290+0000 7f337aae9700 1 -- 172.21.15.29:0/592186534 shutdown_connections 2024-03-16T11:48:58.294 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:58.290+0000 7f337aae9700 1 -- 172.21.15.29:0/592186534 wait complete. 2024-03-16T11:48:58.294 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:58.291+0000 7f337aae9700 1 Processor -- start 2024-03-16T11:48:58.294 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:58.291+0000 7f337aae9700 1 -- start start 2024-03-16T11:48:58.295 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:58.291+0000 7f337aae9700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3374102350 0x7f33740745d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:48:58.295 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:58.291+0000 7f337aae9700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3374102d10 0x7f3374074b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:48:58.295 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:58.291+0000 7f337aae9700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f33741083a0 0x7f3374075050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:48:58.295 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:58.291+0000 7f337aae9700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f33741125e0 con 0x7f33741083a0 2024-03-16T11:48:58.295 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:58.291+0000 7f337aae9700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f3374112460 con 0x7f3374102350 2024-03-16T11:48:58.295 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:58.292+0000 7f337aae9700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f3374112760 con 0x7f3374102d10 2024-03-16T11:48:58.295 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:58.292+0000 7f3379086700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f33741083a0 0x7f3374075050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:48:58.295 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:58.292+0000 7f3373fff700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3374102d10 0x7f3374074b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:48:58.295 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:58.292+0000 7f3378885700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3374102350 0x7f33740745d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:48:58.295 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:58.292+0000 7f3379086700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f33741083a0 0x7f3374075050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57158/0 (socket says 172.21.15.29:57158) 2024-03-16T11:48:58.296 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:58.292+0000 7f3379086700 1 -- 172.21.15.29:0/1097302845 learned_addr learned my addr 172.21.15.29:0/1097302845 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:48:58.296 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:58.292+0000 7f3379086700 1 -- 172.21.15.29:0/1097302845 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3374102d10 msgr2=0x7f3374074b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:48:58.296 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:58.292+0000 7f3379086700 1 --2- 172.21.15.29:0/1097302845 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3374102d10 0x7f3374074b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:58.296 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:58.292+0000 7f3379086700 1 -- 172.21.15.29:0/1097302845 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3374102350 msgr2=0x7f33740745d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:48:58.296 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:58.292+0000 7f3379086700 1 --2- 172.21.15.29:0/1097302845 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3374102350 0x7f33740745d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:58.296 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:58.292+0000 7f3379086700 1 -- 172.21.15.29:0/1097302845 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33741a6fe0 con 0x7f33741083a0 2024-03-16T11:48:58.296 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:58.292+0000 7f3373fff700 1 --2- 172.21.15.29:0/1097302845 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3374102d10 0x7f3374074b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:48:58.296 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:58.292+0000 7f3378885700 1 --2- 172.21.15.29:0/1097302845 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3374102350 0x7f33740745d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:48:58.296 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:58.292+0000 7f3379086700 1 --2- 172.21.15.29:0/1097302845 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f33741083a0 0x7f3374075050 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f336c007f10 tx=0x7f336c00d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:48:58.296 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:58.292+0000 7f3371ffb700 1 -- 172.21.15.29:0/1097302845 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f336c00dd20 con 0x7f33741083a0 2024-03-16T11:48:58.296 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:58.292+0000 7f3371ffb700 1 -- 172.21.15.29:0/1097302845 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f336c004020 con 0x7f33741083a0 2024-03-16T11:48:58.296 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:58.292+0000 7f3371ffb700 1 -- 172.21.15.29:0/1097302845 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f336c015640 con 0x7f33741083a0 2024-03-16T11:48:58.296 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:58.292+0000 7f337aae9700 1 -- 172.21.15.29:0/1097302845 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f33741a72d0 con 0x7f33741083a0 2024-03-16T11:48:58.297 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:58.293+0000 7f337aae9700 1 -- 172.21.15.29:0/1097302845 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f33741a77c0 con 0x7f33741083a0 2024-03-16T11:48:58.297 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:58.294+0000 7f337aae9700 1 -- 172.21.15.29:0/1097302845 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f337410a860 con 0x7f33741083a0 2024-03-16T11:48:58.297 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:58.295+0000 7f3371ffb700 1 -- 172.21.15.29:0/1097302845 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f336c0041d0 con 0x7f33741083a0 2024-03-16T11:48:58.298 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:58.295+0000 7f3371ffb700 1 --2- 172.21.15.29:0/1097302845 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f33640a2920 0x7f33640a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:48:58.298 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:58.295+0000 7f3378885700 1 --2- 172.21.15.29:0/1097302845 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f33640a2920 0x7f33640a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:48:58.298 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:58.295+0000 7f3371ffb700 1 -- 172.21.15.29:0/1097302845 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f336c0ce2e0 con 0x7f33741083a0 2024-03-16T11:48:58.298 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:58.296+0000 7f3378885700 1 --2- 172.21.15.29:0/1097302845 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f33640a2920 0x7f33640a4de0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f336800b2e0 tx=0x7f336801a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:48:58.300 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:58.300+0000 7f3371ffb700 1 -- 172.21.15.29:0/1097302845 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f336c010070 con 0x7f33741083a0 2024-03-16T11:48:58.512 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:58.511+0000 7f337aae9700 1 -- 172.21.15.29:0/1097302845 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f3374075b00 con 0x7f33741083a0 2024-03-16T11:48:58.513 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:58.513+0000 7f3371ffb700 1 -- 172.21.15.29:0/1097302845 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) ==== 74+0+22233 (secure 0 0 0) 0x7f336c09b3d0 con 0x7f33741083a0 2024-03-16T11:48:58.514 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-16T11:48:58.514 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":85,"fsid":"d1887f28-e389-11ee-95c9-87774f69a715","created":"2024-03-16T11:40:16.475063+0000","modified":"2024-03-16T11:48:39.554402+0000","last_up_change":"2024-03-16T11:47:44.349348+0000","last_in_change":"2024-03-16T11:47:23.672432+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-16T11:43:51.892652+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-16T11:48:36.709732+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-16T11:48:37.586140+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"60f5608f-a408-45ba-bdf6-5b53758f88fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":2748947913},{"type":"v1","addr":"172.21.15.29:6803","nonce":2748947913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":2748947913},{"type":"v1","addr":"172.21.15.29:6805","nonce":2748947913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":2748947913},{"type":"v1","addr":"172.21.15.29:6809","nonce":2748947913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":2748947913},{"type":"v1","addr":"172.21.15.29:6807","nonce":2748947913}]},"public_addr":"172.21.15.29:6803/2748947913","cluster_addr":"172.21.15.29:6805/2748947913","heartbeat_back_addr":"172.21.15.29:6809/2748947913","heartbeat_front_addr":"172.21.15.29:6807/2748947913","state":["exists","up"]},{"osd":1,"uuid":"e70b53d6-ee78-4e56-9a2c-263ae0501b43","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6800","nonce":2339408422},{"type":"v1","addr":"172.21.15.152:6801","nonce":2339408422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6802","nonce":2339408422},{"type":"v1","addr":"172.21.15.152:6803","nonce":2339408422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6806","nonce":2339408422},{"type":"v1","addr":"172.21.15.152:6807","nonce":2339408422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6804","nonce":2339408422},{"type":"v1","addr":"172.21.15.152:6805","nonce":2339408422}]},"public_addr":"172.21.15.152:6801/2339408422","cluster_addr":"172.21.15.152:6803/2339408422","heartbeat_back_addr":"172.21.15.152:6807/2339408422","heartbeat_front_addr":"172.21.15.152:6805/2339408422","state":["exists","up"]},{"osd":2,"uuid":"ff39389a-dfac-44b5-80ff-36abca316e78","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6800","nonce":4288325280},{"type":"v1","addr":"172.21.15.154:6801","nonce":4288325280}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6802","nonce":4288325280},{"type":"v1","addr":"172.21.15.154:6803","nonce":4288325280}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6806","nonce":4288325280},{"type":"v1","addr":"172.21.15.154:6807","nonce":4288325280}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":4288325280},{"type":"v1","addr":"172.21.15.154:6805","nonce":4288325280}]},"public_addr":"172.21.15.154:6801/4288325280","cluster_addr":"172.21.15.154:6803/4288325280","heartbeat_back_addr":"172.21.15.154:6807/4288325280","heartbeat_front_addr":"172.21.15.154:6805/4288325280","state":["exists","up"]},{"osd":3,"uuid":"069e1795-55bc-4324-b94c-eb24ef2334c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":1538750397},{"type":"v1","addr":"172.21.15.29:6811","nonce":1538750397}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":1538750397},{"type":"v1","addr":"172.21.15.29:6813","nonce":1538750397}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":1538750397},{"type":"v1","addr":"172.21.15.29:6817","nonce":1538750397}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":1538750397},{"type":"v1","addr":"172.21.15.29:6815","nonce":1538750397}]},"public_addr":"172.21.15.29:6811/1538750397","cluster_addr":"172.21.15.29:6813/1538750397","heartbeat_back_addr":"172.21.15.29:6817/1538750397","heartbeat_front_addr":"172.21.15.29:6815/1538750397","state":["exists","up"]},{"osd":4,"uuid":"5a9d43a1-e8f5-4ab5-a441-afbc6ed4066e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6808","nonce":2795081750},{"type":"v1","addr":"172.21.15.152:6809","nonce":2795081750}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6810","nonce":2795081750},{"type":"v1","addr":"172.21.15.152:6811","nonce":2795081750}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6814","nonce":2795081750},{"type":"v1","addr":"172.21.15.152:6815","nonce":2795081750}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6812","nonce":2795081750},{"type":"v1","addr":"172.21.15.152:6813","nonce":2795081750}]},"public_addr":"172.21.15.152:6809/2795081750","cluster_addr":"172.21.15.152:6811/2795081750","heartbeat_back_addr":"172.21.15.152:6815/2795081750","heartbeat_front_addr":"172.21.15.152:6813/2795081750","state":["exists","up"]},{"osd":5,"uuid":"234de428-4ac7-44be-9a54-7b58331fdf8c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":1472686560},{"type":"v1","addr":"172.21.15.154:6809","nonce":1472686560}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6810","nonce":1472686560},{"type":"v1","addr":"172.21.15.154:6811","nonce":1472686560}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6814","nonce":1472686560},{"type":"v1","addr":"172.21.15.154:6815","nonce":1472686560}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6812","nonce":1472686560},{"type":"v1","addr":"172.21.15.154:6813","nonce":1472686560}]},"public_addr":"172.21.15.154:6809/1472686560","cluster_addr":"172.21.15.154:6811/1472686560","heartbeat_back_addr":"172.21.15.154:6815/1472686560","heartbeat_front_addr":"172.21.15.154:6813/1472686560","state":["exists","up"]},{"osd":6,"uuid":"ae45b8d9-7883-40c9-a9cd-4c92b601d4f2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":816506244},{"type":"v1","addr":"172.21.15.29:6819","nonce":816506244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":816506244},{"type":"v1","addr":"172.21.15.29:6821","nonce":816506244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":816506244},{"type":"v1","addr":"172.21.15.29:6825","nonce":816506244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":816506244},{"type":"v1","addr":"172.21.15.29:6823","nonce":816506244}]},"public_addr":"172.21.15.29:6819/816506244","cluster_addr":"172.21.15.29:6821/816506244","heartbeat_back_addr":"172.21.15.29:6825/816506244","heartbeat_front_addr":"172.21.15.29:6823/816506244","state":["exists","up"]},{"osd":7,"uuid":"6f687b28-4fce-4d4a-8b4e-294173d9d6c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6816","nonce":2924651787},{"type":"v1","addr":"172.21.15.152:6817","nonce":2924651787}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6818","nonce":2924651787},{"type":"v1","addr":"172.21.15.152:6819","nonce":2924651787}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6822","nonce":2924651787},{"type":"v1","addr":"172.21.15.152:6823","nonce":2924651787}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6820","nonce":2924651787},{"type":"v1","addr":"172.21.15.152:6821","nonce":2924651787}]},"public_addr":"172.21.15.152:6817/2924651787","cluster_addr":"172.21.15.152:6819/2924651787","heartbeat_back_addr":"172.21.15.152:6823/2924651787","heartbeat_front_addr":"172.21.15.152:6821/2924651787","state":["exists","up"]},{"osd":8,"uuid":"a25fb741-caa7-429a-b540-e394d034b446","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6816","nonce":129974156},{"type":"v1","addr":"172.21.15.154:6817","nonce":129974156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6818","nonce":129974156},{"type":"v1","addr":"172.21.15.154:6819","nonce":129974156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6822","nonce":129974156},{"type":"v1","addr":"172.21.15.154:6823","nonce":129974156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6820","nonce":129974156},{"type":"v1","addr":"172.21.15.154:6821","nonce":129974156}]},"public_addr":"172.21.15.154:6817/129974156","cluster_addr":"172.21.15.154:6819/129974156","heartbeat_back_addr":"172.21.15.154:6823/129974156","heartbeat_front_addr":"172.21.15.154:6821/129974156","state":["exists","up"]},{"osd":9,"uuid":"1071cb0e-8c5f-4cd8-9805-0045e220d946","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6826","nonce":2951414243},{"type":"v1","addr":"172.21.15.29:6827","nonce":2951414243}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":2951414243},{"type":"v1","addr":"172.21.15.29:6829","nonce":2951414243}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":2951414243},{"type":"v1","addr":"172.21.15.29:6833","nonce":2951414243}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":2951414243},{"type":"v1","addr":"172.21.15.29:6831","nonce":2951414243}]},"public_addr":"172.21.15.29:6827/2951414243","cluster_addr":"172.21.15.29:6829/2951414243","heartbeat_back_addr":"172.21.15.29:6833/2951414243","heartbeat_front_addr":"172.21.15.29:6831/2951414243","state":["exists","up"]},{"osd":10,"uuid":"ee7c7d74-3f39-4cd4-be80-843fb1dd9ecc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6824","nonce":3412525480},{"type":"v1","addr":"172.21.15.152:6825","nonce":3412525480}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6826","nonce":3412525480},{"type":"v1","addr":"172.21.15.152:6827","nonce":3412525480}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6830","nonce":3412525480},{"type":"v1","addr":"172.21.15.152:6831","nonce":3412525480}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6828","nonce":3412525480},{"type":"v1","addr":"172.21.15.152:6829","nonce":3412525480}]},"public_addr":"172.21.15.152:6825/3412525480","cluster_addr":"172.21.15.152:6827/3412525480","heartbeat_back_addr":"172.21.15.152:6831/3412525480","heartbeat_front_addr":"172.21.15.152:6829/3412525480","state":["exists","up"]},{"osd":11,"uuid":"4598c803-984a-4de6-ac22-066502c17889","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6824","nonce":1160927966},{"type":"v1","addr":"172.21.15.154:6825","nonce":1160927966}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6826","nonce":1160927966},{"type":"v1","addr":"172.21.15.154:6827","nonce":1160927966}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6830","nonce":1160927966},{"type":"v1","addr":"172.21.15.154:6831","nonce":1160927966}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6828","nonce":1160927966},{"type":"v1","addr":"172.21.15.154:6829","nonce":1160927966}]},"public_addr":"172.21.15.154:6825/1160927966","cluster_addr":"172.21.15.154:6827/1160927966","heartbeat_back_addr":"172.21.15.154:6831/1160927966","heartbeat_front_addr":"172.21.15.154:6829/1160927966","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:43:02.343393+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:43:25.209151+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:43:48.946164+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:44:14.588424+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:44:39.700025+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:45:04.061197+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:45:31.214508+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:45:56.937296+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:46:22.768915+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:46:50.101011+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:47:16.298041+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:47:42.402095+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.29:0/1853086886":"2024-03-17T11:41:27.531724+0000","172.21.15.29:0/1211659364":"2024-03-17T11:41:27.531724+0000","172.21.15.29:6800/2798831128":"2024-03-17T11:41:27.531724+0000","172.21.15.29:0/1281028285":"2024-03-17T11:41:00.505678+0000","172.21.15.29:6800/2275315654":"2024-03-17T11:41:00.505678+0000","172.21.15.29:6801/2798831128":"2024-03-17T11:41:27.531724+0000","172.21.15.29:6801/2275315654":"2024-03-17T11:41:00.505678+0000","172.21.15.29:0/1330100009":"2024-03-17T11:41:00.505678+0000","172.21.15.29:0/1784249039":"2024-03-17T11:41:00.505678+0000","172.21.15.29:0/1116486932":"2024-03-17T11:41:27.531724+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-16T11:48:58.516 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:58.515+0000 7f337aae9700 1 -- 172.21.15.29:0/1097302845 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f33640a2920 msgr2=0x7f33640a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:48:58.516 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:58.515+0000 7f337aae9700 1 --2- 172.21.15.29:0/1097302845 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f33640a2920 0x7f33640a4de0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f336800b2e0 tx=0x7f336801a040 comp rx=0 tx=0).stop 2024-03-16T11:48:58.516 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:58.515+0000 7f337aae9700 1 -- 172.21.15.29:0/1097302845 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f33741083a0 msgr2=0x7f3374075050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:48:58.516 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:58.515+0000 7f337aae9700 1 --2- 172.21.15.29:0/1097302845 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f33741083a0 0x7f3374075050 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f336c007f10 tx=0x7f336c00d3b0 comp rx=0 tx=0).stop 2024-03-16T11:48:58.516 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:58.515+0000 7f337aae9700 1 -- 172.21.15.29:0/1097302845 shutdown_connections 2024-03-16T11:48:58.516 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:58.515+0000 7f337aae9700 1 --2- 172.21.15.29:0/1097302845 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f33741083a0 0x7f3374075050 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:58.516 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:58.515+0000 7f337aae9700 1 --2- 172.21.15.29:0/1097302845 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3374102350 0x7f33740745d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:58.516 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:58.515+0000 7f337aae9700 1 --2- 172.21.15.29:0/1097302845 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f33640a2920 0x7f33640a4de0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:58.517 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:58.515+0000 7f337aae9700 1 --2- 172.21.15.29:0/1097302845 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3374102d10 0x7f3374074b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:48:58.517 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:58.515+0000 7f337aae9700 1 -- 172.21.15.29:0/1097302845 >> 172.21.15.29:0/1097302845 conn(0x7f3374079420 msgr2=0x7f3374100fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:48:58.517 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:58.516+0000 7f337aae9700 1 -- 172.21.15.29:0/1097302845 shutdown_connections 2024-03-16T11:48:58.517 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:48:58.516+0000 7f337aae9700 1 -- 172.21.15.29:0/1097302845 wait complete. 2024-03-16T11:48:59.011 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:58 smithi029 ceph-mon[31975]: from='client.? 172.21.15.29:0/1097302845' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T11:48:59.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:58 smithi154 ceph-mon[36320]: from='client.? 172.21.15.29:0/1097302845' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T11:48:59.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:58 smithi152 ceph-mon[36291]: from='client.? 172.21.15.29:0/1097302845' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T11:48:59.468 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid d1887f28-e389-11ee-95c9-87774f69a715 -- ceph tell osd.0 flush_pg_stats 2024-03-16T11:48:59.468 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid d1887f28-e389-11ee-95c9-87774f69a715 -- ceph tell osd.1 flush_pg_stats 2024-03-16T11:48:59.468 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid d1887f28-e389-11ee-95c9-87774f69a715 -- ceph tell osd.2 flush_pg_stats 2024-03-16T11:48:59.469 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid d1887f28-e389-11ee-95c9-87774f69a715 -- ceph tell osd.3 flush_pg_stats 2024-03-16T11:48:59.469 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid d1887f28-e389-11ee-95c9-87774f69a715 -- ceph tell osd.4 flush_pg_stats 2024-03-16T11:48:59.470 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid d1887f28-e389-11ee-95c9-87774f69a715 -- ceph tell osd.5 flush_pg_stats 2024-03-16T11:48:59.470 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid d1887f28-e389-11ee-95c9-87774f69a715 -- ceph tell osd.6 flush_pg_stats 2024-03-16T11:48:59.471 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid d1887f28-e389-11ee-95c9-87774f69a715 -- ceph tell osd.7 flush_pg_stats 2024-03-16T11:48:59.471 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid d1887f28-e389-11ee-95c9-87774f69a715 -- ceph tell osd.8 flush_pg_stats 2024-03-16T11:48:59.471 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid d1887f28-e389-11ee-95c9-87774f69a715 -- ceph tell osd.9 flush_pg_stats 2024-03-16T11:48:59.472 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid d1887f28-e389-11ee-95c9-87774f69a715 -- ceph tell osd.10 flush_pg_stats 2024-03-16T11:48:59.472 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid d1887f28-e389-11ee-95c9-87774f69a715 -- ceph tell osd.11 flush_pg_stats 2024-03-16T11:48:59.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:48:59 smithi029 ceph-mon[31975]: pgmap v301: 97 pgs: 97 active+clean; 638 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 3.6 KiB/s wr, 8 op/s 2024-03-16T11:49:00.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:48:59 smithi154 ceph-mon[36320]: pgmap v301: 97 pgs: 97 active+clean; 638 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 3.6 KiB/s wr, 8 op/s 2024-03-16T11:49:00.119 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/d1887f28-e389-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T11:49:00.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:48:59 smithi152 ceph-mon[36291]: pgmap v301: 97 pgs: 97 active+clean; 638 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 3.6 KiB/s wr, 8 op/s 2024-03-16T11:49:00.246 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/d1887f28-e389-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T11:49:00.247 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/d1887f28-e389-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T11:49:00.247 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/d1887f28-e389-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T11:49:00.247 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/d1887f28-e389-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T11:49:00.248 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/d1887f28-e389-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T11:49:00.250 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/d1887f28-e389-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T11:49:00.251 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/d1887f28-e389-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T11:49:02.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:49:01 smithi154 ceph-mon[36320]: pgmap v302: 97 pgs: 97 active+clean; 638 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 1.2 KiB/s wr, 6 op/s 2024-03-16T11:49:02.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:49:01 smithi029 ceph-mon[31975]: pgmap v302: 97 pgs: 97 active+clean; 638 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 1.2 KiB/s wr, 6 op/s 2024-03-16T11:49:02.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:49:01 smithi152 ceph-mon[36291]: pgmap v302: 97 pgs: 97 active+clean; 638 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 1.2 KiB/s wr, 6 op/s 2024-03-16T11:49:03.953 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:49:03 smithi029 ceph-mon[31975]: pgmap v303: 97 pgs: 97 active+clean; 638 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 1.2 KiB/s wr, 6 op/s 2024-03-16T11:49:04.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:49:03 smithi154 ceph-mon[36320]: pgmap v303: 97 pgs: 97 active+clean; 638 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 1.2 KiB/s wr, 6 op/s 2024-03-16T11:49:04.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:49:03 smithi152 ceph-mon[36291]: pgmap v303: 97 pgs: 97 active+clean; 638 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 1.2 KiB/s wr, 6 op/s 2024-03-16T11:49:06.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:49:05 smithi154 ceph-mon[36320]: pgmap v304: 97 pgs: 97 active+clean; 638 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 1.2 KiB/s wr, 6 op/s 2024-03-16T11:49:06.062 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:49:05 smithi029 ceph-mon[31975]: pgmap v304: 97 pgs: 97 active+clean; 638 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 1.2 KiB/s wr, 6 op/s 2024-03-16T11:49:06.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:49:05 smithi152 ceph-mon[36291]: pgmap v304: 97 pgs: 97 active+clean; 638 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 1.2 KiB/s wr, 6 op/s 2024-03-16T11:49:06.431 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/d1887f28-e389-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T11:49:06.431 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/d1887f28-e389-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T11:49:06.432 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/d1887f28-e389-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T11:49:06.433 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/d1887f28-e389-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T11:49:07.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:49:07 smithi029 ceph-mon[31975]: pgmap v305: 97 pgs: 97 active+clean; 638 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 426 B/s wr, 6 op/s 2024-03-16T11:49:08.044 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:49:07 smithi154 ceph-mon[36320]: pgmap v305: 97 pgs: 97 active+clean; 638 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 426 B/s wr, 6 op/s 2024-03-16T11:49:08.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:49:07 smithi152 ceph-mon[36291]: pgmap v305: 97 pgs: 97 active+clean; 638 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 426 B/s wr, 6 op/s 2024-03-16T11:49:09.826 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:49:09 smithi029 ceph-mon[31975]: pgmap v306: 97 pgs: 97 active+clean; 640 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 597 B/s wr, 7 op/s 2024-03-16T11:49:10.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:49:09 smithi154 ceph-mon[36320]: pgmap v306: 97 pgs: 97 active+clean; 640 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 597 B/s wr, 7 op/s 2024-03-16T11:49:10.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:49:09 smithi152 ceph-mon[36291]: pgmap v306: 97 pgs: 97 active+clean; 640 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 597 B/s wr, 7 op/s 2024-03-16T11:49:11.200 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:11.164+0000 7fe41753a700 1 -- 172.21.15.29:0/3718049247 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe410101820 msgr2=0x7fe410106f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:49:11.201 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:11.164+0000 7fe41753a700 1 --2- 172.21.15.29:0/3718049247 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe410101820 0x7fe410106f60 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fe40c0099c0 tx=0x7fe40c009cd0 comp rx=0 tx=0).stop 2024-03-16T11:49:11.201 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:11.164+0000 7fe41753a700 1 -- 172.21.15.29:0/3718049247 shutdown_connections 2024-03-16T11:49:11.201 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:11.164+0000 7fe41753a700 1 --2- 172.21.15.29:0/3718049247 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe410107690 0x7fe41010f210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:49:11.201 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:11.164+0000 7fe41753a700 1 --2- 172.21.15.29:0/3718049247 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe410100e70 0x7fe410101250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:49:11.201 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:11.164+0000 7fe41753a700 1 --2- 172.21.15.29:0/3718049247 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe410101820 0x7fe410106f60 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:49:11.201 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:11.164+0000 7fe41753a700 1 -- 172.21.15.29:0/3718049247 >> 172.21.15.29:0/3718049247 conn(0x7fe410076530 msgr2=0x7fe410076940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:49:11.201 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:11.164+0000 7fe41753a700 1 -- 172.21.15.29:0/3718049247 shutdown_connections 2024-03-16T11:49:11.201 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:11.164+0000 7fe41753a700 1 -- 172.21.15.29:0/3718049247 wait complete. 2024-03-16T11:49:11.201 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:11.165+0000 7fe41753a700 1 Processor -- start 2024-03-16T11:49:11.201 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:11.165+0000 7fe41753a700 1 -- start start 2024-03-16T11:49:11.201 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:11.165+0000 7fe41753a700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe410100e70 0x7fe41019e740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:49:11.201 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:11.165+0000 7fe41753a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe410101820 0x7fe41019ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:49:11.201 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:11.165+0000 7fe41753a700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe410107690 0x7fe410198900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:49:11.202 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:11.165+0000 7fe41753a700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fe410112600 con 0x7fe410101820 2024-03-16T11:49:11.202 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:11.165+0000 7fe41753a700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fe410112480 con 0x7fe410107690 2024-03-16T11:49:11.202 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:11.165+0000 7fe41753a700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fe410112780 con 0x7fe410100e70 2024-03-16T11:49:11.202 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:11.165+0000 7fe414ad5700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe410101820 0x7fe41019ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:49:11.202 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:11.165+0000 7fe4152d6700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe410100e70 0x7fe41019e740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:49:11.202 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:11.165+0000 7fe414ad5700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe410101820 0x7fe41019ec80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:37374/0 (socket says 172.21.15.29:37374) 2024-03-16T11:49:11.202 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:11.165+0000 7fe414ad5700 1 -- 172.21.15.29:0/4138627099 learned_addr learned my addr 172.21.15.29:0/4138627099 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:49:11.202 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:11.165+0000 7fe415ad7700 1 --2- 172.21.15.29:0/4138627099 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe410107690 0x7fe410198900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:49:11.202 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:11.165+0000 7fe414ad5700 1 -- 172.21.15.29:0/4138627099 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe410100e70 msgr2=0x7fe41019e740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:49:11.202 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:11.165+0000 7fe414ad5700 1 --2- 172.21.15.29:0/4138627099 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe410100e70 0x7fe41019e740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:49:11.202 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:11.165+0000 7fe414ad5700 1 -- 172.21.15.29:0/4138627099 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe410107690 msgr2=0x7fe410198900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:49:11.202 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:11.165+0000 7fe414ad5700 1 --2- 172.21.15.29:0/4138627099 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe410107690 0x7fe410198900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:49:11.202 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:11.165+0000 7fe414ad5700 1 -- 172.21.15.29:0/4138627099 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe410199060 con 0x7fe410101820 2024-03-16T11:49:11.202 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:11.166+0000 7fe414ad5700 1 --2- 172.21.15.29:0/4138627099 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe410101820 0x7fe41019ec80 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fe40c00b580 tx=0x7fe40c0038b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:49:11.203 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:11.166+0000 7fe4067fc700 1 -- 172.21.15.29:0/4138627099 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe40c026070 con 0x7fe410101820 2024-03-16T11:49:11.203 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:11.166+0000 7fe41753a700 1 -- 172.21.15.29:0/4138627099 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe4101992f0 con 0x7fe410101820 2024-03-16T11:49:11.203 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:11.166+0000 7fe4067fc700 1 -- 172.21.15.29:0/4138627099 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe40c003960 con 0x7fe410101820 2024-03-16T11:49:11.203 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:11.166+0000 7fe4067fc700 1 -- 172.21.15.29:0/4138627099 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe40c0047e0 con 0x7fe410101820 2024-03-16T11:49:11.203 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:11.166+0000 7fe41753a700 1 -- 172.21.15.29:0/4138627099 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe4101a5610 con 0x7fe410101820 2024-03-16T11:49:11.203 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:11.167+0000 7fe41753a700 1 -- 172.21.15.29:0/4138627099 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fe410062a60 con 0x7fe410101820 2024-03-16T11:49:11.203 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:11.168+0000 7fe4067fc700 1 -- 172.21.15.29:0/4138627099 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7fe40c029070 con 0x7fe410101820 2024-03-16T11:49:11.203 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:11.169+0000 7fe4067fc700 1 --2- 172.21.15.29:0/4138627099 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fe3fc0a2920 0x7fe3fc0a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:49:11.203 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:11.169+0000 7fe4152d6700 1 --2- 172.21.15.29:0/4138627099 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fe3fc0a2920 0x7fe3fc0a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:49:11.203 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:11.169+0000 7fe4067fc700 1 -- 172.21.15.29:0/4138627099 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7fe40c0cfb20 con 0x7fe410101820 2024-03-16T11:49:11.203 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:11.169+0000 7fe4067fc700 1 --2- 172.21.15.29:0/4138627099 >> [v2:172.21.15.154:6816/129974156,v1:172.21.15.154:6817/129974156] conn(0x7fe3fc0ac100 0x7fe3fc0ae520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:49:11.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:11.169+0000 7fe4067fc700 1 -- 172.21.15.29:0/4138627099 --> [v2:172.21.15.154:6816/129974156,v1:172.21.15.154:6817/129974156] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe40c028e50 con 0x7fe3fc0ac100 2024-03-16T11:49:11.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:11.169+0000 7fe4067fc700 1 -- 172.21.15.29:0/4138627099 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) ==== 24+0+0 (secure 0 0 0) 0x7fe40c0d0000 con 0x7fe410101820 2024-03-16T11:49:11.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:11.169+0000 7fe415ad7700 1 --2- 172.21.15.29:0/4138627099 >> [v2:172.21.15.154:6816/129974156,v1:172.21.15.154:6817/129974156] conn(0x7fe3fc0ac100 0x7fe3fc0ae520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:49:11.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:11.169+0000 7fe4152d6700 1 --2- 172.21.15.29:0/4138627099 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fe3fc0a2920 0x7fe3fc0a4de0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fe410199a40 tx=0x7fe4000073d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:49:11.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:11.169+0000 7fe415ad7700 1 --2- 172.21.15.29:0/4138627099 >> [v2:172.21.15.154:6816/129974156,v1:172.21.15.154:6817/129974156] conn(0x7fe3fc0ac100 0x7fe3fc0ae520 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:49:11.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:11.170+0000 7fe4067fc700 1 -- 172.21.15.29:0/4138627099 <== osd.8 v2:172.21.15.154:6816/129974156 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fe40c028e50 con 0x7fe3fc0ac100 2024-03-16T11:49:11.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:11.186+0000 7fe41753a700 1 -- 172.21.15.29:0/4138627099 --> [v2:172.21.15.154:6816/129974156,v1:172.21.15.154:6817/129974156] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fe410062ee0 con 0x7fe3fc0ac100 2024-03-16T11:49:11.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:11.187+0000 7fe4067fc700 1 -- 172.21.15.29:0/4138627099 <== osd.8 v2:172.21.15.154:6816/129974156 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fe410062ee0 con 0x7fe3fc0ac100 2024-03-16T11:49:11.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:11.187+0000 7fe41753a700 1 -- 172.21.15.29:0/4138627099 >> [v2:172.21.15.154:6816/129974156,v1:172.21.15.154:6817/129974156] conn(0x7fe3fc0ac100 msgr2=0x7fe3fc0ae520 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:49:11.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:11.187+0000 7fe41753a700 1 --2- 172.21.15.29:0/4138627099 >> [v2:172.21.15.154:6816/129974156,v1:172.21.15.154:6817/129974156] conn(0x7fe3fc0ac100 0x7fe3fc0ae520 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:49:11.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:11.187+0000 7fe41753a700 1 -- 172.21.15.29:0/4138627099 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fe3fc0a2920 msgr2=0x7fe3fc0a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:49:11.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:11.187+0000 7fe41753a700 1 --2- 172.21.15.29:0/4138627099 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fe3fc0a2920 0x7fe3fc0a4de0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fe410199a40 tx=0x7fe4000073d0 comp rx=0 tx=0).stop 2024-03-16T11:49:11.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:11.187+0000 7fe41753a700 1 -- 172.21.15.29:0/4138627099 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe410101820 msgr2=0x7fe41019ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:49:11.205 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:11.187+0000 7fe41753a700 1 --2- 172.21.15.29:0/4138627099 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe410101820 0x7fe41019ec80 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fe40c00b580 tx=0x7fe40c0038b0 comp rx=0 tx=0).stop 2024-03-16T11:49:11.205 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:11.187+0000 7fe415ad7700 1 -- 172.21.15.29:0/4138627099 reap_dead start 2024-03-16T11:49:11.205 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:11.187+0000 7fe41753a700 1 -- 172.21.15.29:0/4138627099 shutdown_connections 2024-03-16T11:49:11.205 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:11.187+0000 7fe41753a700 1 -- 172.21.15.29:0/4138627099 >> 172.21.15.29:0/4138627099 conn(0x7fe410076530 msgr2=0x7fe410100390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:49:11.205 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:11.188+0000 7fe41753a700 1 -- 172.21.15.29:0/4138627099 shutdown_connections 2024-03-16T11:49:11.205 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:11.188+0000 7fe41753a700 1 -- 172.21.15.29:0/4138627099 wait complete. 2024-03-16T11:49:11.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:49:11 smithi029 ceph-mon[31975]: pgmap v307: 97 pgs: 97 active+clean; 640 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 170 B/s wr, 5 op/s 2024-03-16T11:49:12.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:49:11 smithi154 ceph-mon[36320]: pgmap v307: 97 pgs: 97 active+clean; 640 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 170 B/s wr, 5 op/s 2024-03-16T11:49:12.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:49:11 smithi152 ceph-mon[36291]: pgmap v307: 97 pgs: 97 active+clean; 640 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 170 B/s wr, 5 op/s 2024-03-16T11:49:12.149 INFO:teuthology.orchestra.run.smithi029.stdout:253403070501 2024-03-16T11:49:12.149 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid d1887f28-e389-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.8 2024-03-16T11:49:13.014 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:13.007+0000 7f68d98e1700 1 -- 172.21.15.29:0/2053801306 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f68d4101e00 msgr2=0x7f68d4105e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:49:13.014 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:13.007+0000 7f68d98e1700 1 --2- 172.21.15.29:0/2053801306 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f68d4101e00 0x7f68d4105e50 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f68bc0099c0 tx=0x7f68bc009cd0 comp rx=0 tx=0).stop 2024-03-16T11:49:13.014 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:13.007+0000 7f68d98e1700 1 -- 172.21.15.29:0/2053801306 shutdown_connections 2024-03-16T11:49:13.014 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:13.007+0000 7f68d98e1700 1 --2- 172.21.15.29:0/2053801306 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f68d4106530 0x7f68d410e100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:49:13.014 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:13.007+0000 7f68d98e1700 1 --2- 172.21.15.29:0/2053801306 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f68d4101450 0x7f68d4101830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:49:13.014 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:13.007+0000 7f68d98e1700 1 --2- 172.21.15.29:0/2053801306 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f68d4101e00 0x7f68d4105e50 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:49:13.014 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:13.007+0000 7f68d98e1700 1 -- 172.21.15.29:0/2053801306 >> 172.21.15.29:0/2053801306 conn(0x7f68d4078230 msgr2=0x7f68d4078640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:49:13.015 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:13.007+0000 7f68d98e1700 1 -- 172.21.15.29:0/2053801306 shutdown_connections 2024-03-16T11:49:13.015 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:13.007+0000 7f68d98e1700 1 -- 172.21.15.29:0/2053801306 wait complete. 2024-03-16T11:49:13.015 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:13.008+0000 7f68d98e1700 1 Processor -- start 2024-03-16T11:49:13.015 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:13.008+0000 7f68d98e1700 1 -- start start 2024-03-16T11:49:13.015 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:13.008+0000 7f68d98e1700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f68d4101450 0x7f68d419d690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:49:13.015 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:13.008+0000 7f68d98e1700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f68d4101e00 0x7f68d419dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:49:13.015 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:13.008+0000 7f68d98e1700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f68d4106530 0x7f68d4197810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:49:13.015 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:13.009+0000 7f68d98e1700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f68d41114f0 con 0x7f68d4101450 2024-03-16T11:49:13.015 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:13.009+0000 7f68d98e1700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f68d4111370 con 0x7f68d4106530 2024-03-16T11:49:13.015 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:13.009+0000 7f68d98e1700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f68d4111670 con 0x7f68d4101e00 2024-03-16T11:49:13.015 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:13.009+0000 7f68d2ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f68d4101450 0x7f68d419d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:49:13.015 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:13.009+0000 7f68d2ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f68d4101450 0x7f68d419d690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:37386/0 (socket says 172.21.15.29:37386) 2024-03-16T11:49:13.015 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:13.009+0000 7f68d2ffd700 1 -- 172.21.15.29:0/596772964 learned_addr learned my addr 172.21.15.29:0/596772964 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:49:13.015 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:13.009+0000 7f68d37fe700 1 --2- 172.21.15.29:0/596772964 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f68d4106530 0x7f68d4197810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:49:13.016 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:13.009+0000 7f68d27fc700 1 --2- 172.21.15.29:0/596772964 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f68d4101e00 0x7f68d419dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:49:13.016 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:13.009+0000 7f68d2ffd700 1 -- 172.21.15.29:0/596772964 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f68d4101e00 msgr2=0x7f68d419dbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:49:13.016 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:13.009+0000 7f68d2ffd700 1 --2- 172.21.15.29:0/596772964 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f68d4101e00 0x7f68d419dbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:49:13.016 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:13.009+0000 7f68d2ffd700 1 -- 172.21.15.29:0/596772964 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f68d4106530 msgr2=0x7f68d4197810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:49:13.016 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:13.009+0000 7f68d2ffd700 1 --2- 172.21.15.29:0/596772964 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f68d4106530 0x7f68d4197810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:49:13.016 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:13.009+0000 7f68d2ffd700 1 -- 172.21.15.29:0/596772964 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68d4197ff0 con 0x7f68d4101450 2024-03-16T11:49:13.016 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:13.009+0000 7f68d2ffd700 1 --2- 172.21.15.29:0/596772964 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f68d4101450 0x7f68d419d690 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f68cc00ef10 tx=0x7f68cc00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:49:13.016 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:13.009+0000 7f68d88df700 1 -- 172.21.15.29:0/596772964 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68cc00c9c0 con 0x7f68d4101450 2024-03-16T11:49:13.016 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:13.009+0000 7f68d88df700 1 -- 172.21.15.29:0/596772964 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f68cc019070 con 0x7f68d4101450 2024-03-16T11:49:13.016 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:13.009+0000 7f68d98e1700 1 -- 172.21.15.29:0/596772964 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f68d4198280 con 0x7f68d4101450 2024-03-16T11:49:13.016 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:13.009+0000 7f68d88df700 1 -- 172.21.15.29:0/596772964 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68cc007e60 con 0x7f68d4101450 2024-03-16T11:49:13.016 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:13.009+0000 7f68d98e1700 1 -- 172.21.15.29:0/596772964 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f68d406aa10 con 0x7f68d4101450 2024-03-16T11:49:13.016 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:13.010+0000 7f68d98e1700 1 -- 172.21.15.29:0/596772964 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f68d406af00 con 0x7f68d4101450 2024-03-16T11:49:13.016 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:13.011+0000 7f68d88df700 1 -- 172.21.15.29:0/596772964 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f68cc012050 con 0x7f68d4101450 2024-03-16T11:49:13.016 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:13.012+0000 7f68d88df700 1 --2- 172.21.15.29:0/596772964 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f68c00a2920 0x7f68c00a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:49:13.017 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:13.012+0000 7f68d27fc700 1 --2- 172.21.15.29:0/596772964 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f68c00a2920 0x7f68c00a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:49:13.017 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:13.012+0000 7f68d88df700 1 -- 172.21.15.29:0/596772964 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f68cc0cf470 con 0x7f68d4101450 2024-03-16T11:49:13.017 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:13.012+0000 7f68d88df700 1 --2- 172.21.15.29:0/596772964 >> [v2:172.21.15.29:6810/1538750397,v1:172.21.15.29:6811/1538750397] conn(0x7f68c00ac100 0x7f68c00ae520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:49:13.017 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:13.012+0000 7f68d88df700 1 -- 172.21.15.29:0/596772964 --> [v2:172.21.15.29:6810/1538750397,v1:172.21.15.29:6811/1538750397] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f68cc00be50 con 0x7f68c00ac100 2024-03-16T11:49:13.017 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:13.012+0000 7f68d88df700 1 -- 172.21.15.29:0/596772964 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) ==== 24+0+0 (secure 0 0 0) 0x7f68cc0cf950 con 0x7f68d4101450 2024-03-16T11:49:13.017 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:13.012+0000 7f68d37fe700 1 --2- 172.21.15.29:0/596772964 >> [v2:172.21.15.29:6810/1538750397,v1:172.21.15.29:6811/1538750397] conn(0x7f68c00ac100 0x7f68c00ae520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:49:13.017 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:13.012+0000 7f68d27fc700 1 --2- 172.21.15.29:0/596772964 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f68c00a2920 0x7f68c00a4de0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f68bc000c00 tx=0x7f68bc005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:49:13.017 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:13.013+0000 7f68d37fe700 1 --2- 172.21.15.29:0/596772964 >> [v2:172.21.15.29:6810/1538750397,v1:172.21.15.29:6811/1538750397] conn(0x7f68c00ac100 0x7f68c00ae520 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:49:13.017 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:13.013+0000 7f68d88df700 1 -- 172.21.15.29:0/596772964 <== osd.3 v2:172.21.15.29:6810/1538750397 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f68cc00be50 con 0x7f68c00ac100 2024-03-16T11:49:13.029 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:13.030+0000 7f68d98e1700 1 -- 172.21.15.29:0/596772964 --> [v2:172.21.15.29:6810/1538750397,v1:172.21.15.29:6811/1538750397] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f68d406b380 con 0x7f68c00ac100 2024-03-16T11:49:13.029 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:13.030+0000 7f68d88df700 1 -- 172.21.15.29:0/596772964 <== osd.3 v2:172.21.15.29:6810/1538750397 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f68d406b380 con 0x7f68c00ac100 2024-03-16T11:49:13.030 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:13.030+0000 7f68d98e1700 1 -- 172.21.15.29:0/596772964 >> [v2:172.21.15.29:6810/1538750397,v1:172.21.15.29:6811/1538750397] conn(0x7f68c00ac100 msgr2=0x7f68c00ae520 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:49:13.030 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:13.030+0000 7f68d98e1700 1 --2- 172.21.15.29:0/596772964 >> [v2:172.21.15.29:6810/1538750397,v1:172.21.15.29:6811/1538750397] conn(0x7f68c00ac100 0x7f68c00ae520 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:49:13.030 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:13.031+0000 7f68d98e1700 1 -- 172.21.15.29:0/596772964 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f68c00a2920 msgr2=0x7f68c00a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:49:13.030 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:13.031+0000 7f68d98e1700 1 --2- 172.21.15.29:0/596772964 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f68c00a2920 0x7f68c00a4de0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f68bc000c00 tx=0x7f68bc005fb0 comp rx=0 tx=0).stop 2024-03-16T11:49:13.030 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:13.031+0000 7f68d98e1700 1 -- 172.21.15.29:0/596772964 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f68d4101450 msgr2=0x7f68d419d690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:49:13.030 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:13.031+0000 7f68d98e1700 1 --2- 172.21.15.29:0/596772964 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f68d4101450 0x7f68d419d690 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f68cc00ef10 tx=0x7f68cc00c3a0 comp rx=0 tx=0).stop 2024-03-16T11:49:13.031 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:13.031+0000 7f68d37fe700 1 -- 172.21.15.29:0/596772964 reap_dead start 2024-03-16T11:49:13.031 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:13.031+0000 7f68d98e1700 1 -- 172.21.15.29:0/596772964 shutdown_connections 2024-03-16T11:49:13.031 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:13.031+0000 7f68d98e1700 1 -- 172.21.15.29:0/596772964 >> 172.21.15.29:0/596772964 conn(0x7f68d4078230 msgr2=0x7f68d40ff260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:49:13.031 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:13.031+0000 7f68d98e1700 1 -- 172.21.15.29:0/596772964 shutdown_connections 2024-03-16T11:49:13.031 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:13.031+0000 7f68d98e1700 1 -- 172.21.15.29:0/596772964 wait complete. 2024-03-16T11:49:13.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:49:13 smithi029 ceph-mon[31975]: pgmap v308: 97 pgs: 97 active+clean; 640 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 170 B/s wr, 5 op/s 2024-03-16T11:49:14.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:49:13 smithi154 ceph-mon[36320]: pgmap v308: 97 pgs: 97 active+clean; 640 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 170 B/s wr, 5 op/s 2024-03-16T11:49:14.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:49:13 smithi152 ceph-mon[36291]: pgmap v308: 97 pgs: 97 active+clean; 640 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 170 B/s wr, 5 op/s 2024-03-16T11:49:15.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:49:15 smithi029 ceph-mon[31975]: pgmap v309: 97 pgs: 97 active+clean; 640 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 170 B/s wr, 6 op/s 2024-03-16T11:49:16.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:49:15 smithi154 ceph-mon[36320]: pgmap v309: 97 pgs: 97 active+clean; 640 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 170 B/s wr, 6 op/s 2024-03-16T11:49:16.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:49:15 smithi152 ceph-mon[36291]: pgmap v309: 97 pgs: 97 active+clean; 640 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 170 B/s wr, 6 op/s 2024-03-16T11:49:16.427 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:16.425+0000 7f1d6cecc700 1 -- 172.21.15.29:0/4239389654 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1d68106140 msgr2=0x7f1d68106520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:49:16.427 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:16.425+0000 7f1d6cecc700 1 --2- 172.21.15.29:0/4239389654 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1d68106140 0x7f1d68106520 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f1d58009990 tx=0x7f1d58009ca0 comp rx=0 tx=0).stop 2024-03-16T11:49:16.427 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:16.425+0000 7f1d6cecc700 1 -- 172.21.15.29:0/4239389654 shutdown_connections 2024-03-16T11:49:16.427 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:16.425+0000 7f1d6cecc700 1 --2- 172.21.15.29:0/4239389654 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1d68101ec0 0x7f1d6810dfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:49:16.427 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:16.425+0000 7f1d6cecc700 1 --2- 172.21.15.29:0/4239389654 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1d68106140 0x7f1d68106520 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:49:16.427 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:16.425+0000 7f1d6cecc700 1 --2- 172.21.15.29:0/4239389654 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1d68101500 0x7f1d68101980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:49:16.427 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:16.425+0000 7f1d6cecc700 1 -- 172.21.15.29:0/4239389654 >> 172.21.15.29:0/4239389654 conn(0x7f1d68077f80 msgr2=0x7f1d68078390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:49:16.427 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:16.425+0000 7f1d6cecc700 1 -- 172.21.15.29:0/4239389654 shutdown_connections 2024-03-16T11:49:16.427 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:16.425+0000 7f1d6cecc700 1 -- 172.21.15.29:0/4239389654 wait complete. 2024-03-16T11:49:16.427 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:16.426+0000 7f1d6cecc700 1 Processor -- start 2024-03-16T11:49:16.427 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:16.426+0000 7f1d6cecc700 1 -- start start 2024-03-16T11:49:16.427 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:16.426+0000 7f1d6cecc700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1d68101500 0x7f1d681998c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:49:16.428 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:16.426+0000 7f1d6cecc700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1d68101ec0 0x7f1d68199e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:49:16.428 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:16.426+0000 7f1d6cecc700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1d68106140 0x7f1d681a4940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:49:16.428 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:16.426+0000 7f1d6cecc700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f1d68111540 con 0x7f1d68101ec0 2024-03-16T11:49:16.428 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:16.426+0000 7f1d6cecc700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f1d681113c0 con 0x7f1d68106140 2024-03-16T11:49:16.428 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:16.426+0000 7f1d6cecc700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f1d681116c0 con 0x7f1d68101500 2024-03-16T11:49:16.428 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:16.426+0000 7f1d65d9b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1d68101ec0 0x7f1d68199e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:49:16.428 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:16.426+0000 7f1d65d9b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1d68101ec0 0x7f1d68199e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:37408/0 (socket says 172.21.15.29:37408) 2024-03-16T11:49:16.428 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:16.426+0000 7f1d65d9b700 1 -- 172.21.15.29:0/3003339949 learned_addr learned my addr 172.21.15.29:0/3003339949 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:49:16.428 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:16.426+0000 7f1d6659c700 1 --2- 172.21.15.29:0/3003339949 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1d68101500 0x7f1d681998c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:49:16.428 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:16.426+0000 7f1d66d9d700 1 --2- 172.21.15.29:0/3003339949 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1d68106140 0x7f1d681a4940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:49:16.428 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:16.427+0000 7f1d65d9b700 1 -- 172.21.15.29:0/3003339949 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1d68101500 msgr2=0x7f1d681998c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:49:16.428 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:16.427+0000 7f1d65d9b700 1 --2- 172.21.15.29:0/3003339949 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1d68101500 0x7f1d681998c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:49:16.429 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:16.427+0000 7f1d65d9b700 1 -- 172.21.15.29:0/3003339949 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1d68106140 msgr2=0x7f1d681a4940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:49:16.429 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:16.427+0000 7f1d65d9b700 1 --2- 172.21.15.29:0/3003339949 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1d68106140 0x7f1d681a4940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:49:16.429 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:16.427+0000 7f1d65d9b700 1 -- 172.21.15.29:0/3003339949 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d681a4e80 con 0x7f1d68101ec0 2024-03-16T11:49:16.429 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:16.427+0000 7f1d65d9b700 1 --2- 172.21.15.29:0/3003339949 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1d68101ec0 0x7f1d68199e00 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f1d5c00e9a0 tx=0x7f1d5c00ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:49:16.429 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:16.427+0000 7f1d577fe700 1 -- 172.21.15.29:0/3003339949 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d5c00ca10 con 0x7f1d68101ec0 2024-03-16T11:49:16.429 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:16.427+0000 7f1d577fe700 1 -- 172.21.15.29:0/3003339949 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1d5c00cbb0 con 0x7f1d68101ec0 2024-03-16T11:49:16.429 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:16.427+0000 7f1d577fe700 1 -- 172.21.15.29:0/3003339949 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d5c018680 con 0x7f1d68101ec0 2024-03-16T11:49:16.429 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:16.427+0000 7f1d6cecc700 1 -- 172.21.15.29:0/3003339949 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1d681a5050 con 0x7f1d68101ec0 2024-03-16T11:49:16.429 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:16.427+0000 7f1d6cecc700 1 -- 172.21.15.29:0/3003339949 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1d681a5450 con 0x7f1d68101ec0 2024-03-16T11:49:16.430 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:16.428+0000 7f1d6cecc700 1 -- 172.21.15.29:0/3003339949 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f1d680629e0 con 0x7f1d68101ec0 2024-03-16T11:49:16.430 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:16.429+0000 7f1d577fe700 1 -- 172.21.15.29:0/3003339949 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f1d5c010430 con 0x7f1d68101ec0 2024-03-16T11:49:16.431 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:16.430+0000 7f1d577fe700 1 --2- 172.21.15.29:0/3003339949 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f1d500a2920 0x7f1d500a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:49:16.431 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:16.430+0000 7f1d6659c700 1 --2- 172.21.15.29:0/3003339949 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f1d500a2920 0x7f1d500a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:49:16.431 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:16.430+0000 7f1d577fe700 1 -- 172.21.15.29:0/3003339949 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f1d5c0cf490 con 0x7f1d68101ec0 2024-03-16T11:49:16.431 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:16.430+0000 7f1d577fe700 1 --2- 172.21.15.29:0/3003339949 >> [v2:172.21.15.29:6802/2748947913,v1:172.21.15.29:6803/2748947913] conn(0x7f1d500ac100 0x7f1d500ae520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:49:16.431 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:16.430+0000 7f1d577fe700 1 -- 172.21.15.29:0/3003339949 --> [v2:172.21.15.29:6802/2748947913,v1:172.21.15.29:6803/2748947913] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f1d5c01bec0 con 0x7f1d500ac100 2024-03-16T11:49:16.431 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:16.430+0000 7f1d577fe700 1 -- 172.21.15.29:0/3003339949 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) ==== 24+0+0 (secure 0 0 0) 0x7f1d5c0cf970 con 0x7f1d68101ec0 2024-03-16T11:49:16.431 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:16.430+0000 7f1d6659c700 1 --2- 172.21.15.29:0/3003339949 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f1d500a2920 0x7f1d500a4de0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f1d5800b2e0 tx=0x7f1d58005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:49:16.431 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:16.430+0000 7f1d66d9d700 1 --2- 172.21.15.29:0/3003339949 >> [v2:172.21.15.29:6802/2748947913,v1:172.21.15.29:6803/2748947913] conn(0x7f1d500ac100 0x7f1d500ae520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:49:16.431 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:16.431+0000 7f1d66d9d700 1 --2- 172.21.15.29:0/3003339949 >> [v2:172.21.15.29:6802/2748947913,v1:172.21.15.29:6803/2748947913] conn(0x7f1d500ac100 0x7f1d500ae520 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:49:16.431 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:16.431+0000 7f1d577fe700 1 -- 172.21.15.29:0/3003339949 <== osd.0 v2:172.21.15.29:6802/2748947913 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f1d5c01bec0 con 0x7f1d500ac100 2024-03-16T11:49:16.447 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:16.448+0000 7f1d6cecc700 1 -- 172.21.15.29:0/3003339949 --> [v2:172.21.15.29:6802/2748947913,v1:172.21.15.29:6803/2748947913] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f1d68062de0 con 0x7f1d500ac100 2024-03-16T11:49:16.447 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:16.448+0000 7f1d577fe700 1 -- 172.21.15.29:0/3003339949 <== osd.0 v2:172.21.15.29:6802/2748947913 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f1d68062de0 con 0x7f1d500ac100 2024-03-16T11:49:16.448 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:16.448+0000 7f1d6cecc700 1 -- 172.21.15.29:0/3003339949 >> [v2:172.21.15.29:6802/2748947913,v1:172.21.15.29:6803/2748947913] conn(0x7f1d500ac100 msgr2=0x7f1d500ae520 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:49:16.448 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:16.448+0000 7f1d6cecc700 1 --2- 172.21.15.29:0/3003339949 >> [v2:172.21.15.29:6802/2748947913,v1:172.21.15.29:6803/2748947913] conn(0x7f1d500ac100 0x7f1d500ae520 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:49:16.448 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:16.448+0000 7f1d6cecc700 1 -- 172.21.15.29:0/3003339949 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f1d500a2920 msgr2=0x7f1d500a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:49:16.448 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:16.448+0000 7f1d6cecc700 1 --2- 172.21.15.29:0/3003339949 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f1d500a2920 0x7f1d500a4de0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f1d5800b2e0 tx=0x7f1d58005960 comp rx=0 tx=0).stop 2024-03-16T11:49:16.448 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:16.448+0000 7f1d6cecc700 1 -- 172.21.15.29:0/3003339949 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1d68101ec0 msgr2=0x7f1d68199e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:49:16.448 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:16.448+0000 7f1d6cecc700 1 --2- 172.21.15.29:0/3003339949 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1d68101ec0 0x7f1d68199e00 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f1d5c00e9a0 tx=0x7f1d5c00ecb0 comp rx=0 tx=0).stop 2024-03-16T11:49:16.449 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:16.448+0000 7f1d66d9d700 1 -- 172.21.15.29:0/3003339949 reap_dead start 2024-03-16T11:49:16.449 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:16.449+0000 7f1d6cecc700 1 -- 172.21.15.29:0/3003339949 shutdown_connections 2024-03-16T11:49:16.449 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:16.449+0000 7f1d6cecc700 1 -- 172.21.15.29:0/3003339949 >> 172.21.15.29:0/3003339949 conn(0x7f1d68077f80 msgr2=0x7f1d680ff4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:49:16.449 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:16.449+0000 7f1d6cecc700 1 -- 172.21.15.29:0/3003339949 shutdown_connections 2024-03-16T11:49:16.449 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:16.449+0000 7f1d6cecc700 1 -- 172.21.15.29:0/3003339949 wait complete. 2024-03-16T11:49:16.800 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:16.790+0000 7fa255dc9700 1 -- 172.21.15.29:0/2590124512 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa2501083a0 msgr2=0x7fa250108780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:49:16.800 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:16.790+0000 7fa255dc9700 1 --2- 172.21.15.29:0/2590124512 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa2501083a0 0x7fa250108780 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fa244009990 tx=0x7fa244009ca0 comp rx=0 tx=0).stop 2024-03-16T11:49:16.800 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:16.790+0000 7fa255dc9700 1 -- 172.21.15.29:0/2590124512 shutdown_connections 2024-03-16T11:49:16.800 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:16.790+0000 7fa255dc9700 1 --2- 172.21.15.29:0/2590124512 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa250102d10 0x7fa25010f170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:49:16.800 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:16.790+0000 7fa255dc9700 1 --2- 172.21.15.29:0/2590124512 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa2501083a0 0x7fa250108780 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:49:16.800 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:16.790+0000 7fa255dc9700 1 --2- 172.21.15.29:0/2590124512 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa250102350 0x7fa2501027d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:49:16.800 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:16.790+0000 7fa255dc9700 1 -- 172.21.15.29:0/2590124512 >> 172.21.15.29:0/2590124512 conn(0x7fa250079390 msgr2=0x7fa2500797a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:49:16.800 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:16.790+0000 7fa255dc9700 1 -- 172.21.15.29:0/2590124512 shutdown_connections 2024-03-16T11:49:16.800 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:16.790+0000 7fa255dc9700 1 -- 172.21.15.29:0/2590124512 wait complete. 2024-03-16T11:49:16.800 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:16.791+0000 7fa255dc9700 1 Processor -- start 2024-03-16T11:49:16.800 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:16.791+0000 7fa255dc9700 1 -- start start 2024-03-16T11:49:16.800 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:16.791+0000 7fa255dc9700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa250102350 0x7fa25019e660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:49:16.801 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:16.791+0000 7fa255dc9700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa250102d10 0x7fa25019eba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:49:16.801 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:16.791+0000 7fa255dc9700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa2501083a0 0x7fa2501987e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:49:16.801 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:16.791+0000 7fa255dc9700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fa250112580 con 0x7fa2501083a0 2024-03-16T11:49:16.801 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:16.791+0000 7fa255dc9700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fa250112400 con 0x7fa250102d10 2024-03-16T11:49:16.801 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:16.791+0000 7fa255dc9700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fa250112700 con 0x7fa250102350 2024-03-16T11:49:16.801 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:16.791+0000 7fa24ffff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa2501083a0 0x7fa2501987e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:49:16.801 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:16.791+0000 7fa24ffff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa2501083a0 0x7fa2501987e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:37434/0 (socket says 172.21.15.29:37434) 2024-03-16T11:49:16.801 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:16.791+0000 7fa24ffff700 1 -- 172.21.15.29:0/2369436108 learned_addr learned my addr 172.21.15.29:0/2369436108 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:49:16.801 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:16.791+0000 7fa24f7fe700 1 --2- 172.21.15.29:0/2369436108 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa250102350 0x7fa25019e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:49:16.801 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:16.791+0000 7fa24effd700 1 --2- 172.21.15.29:0/2369436108 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa250102d10 0x7fa25019eba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:49:16.801 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:16.791+0000 7fa24ffff700 1 -- 172.21.15.29:0/2369436108 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa250102350 msgr2=0x7fa25019e660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:49:16.801 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:16.791+0000 7fa24ffff700 1 --2- 172.21.15.29:0/2369436108 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa250102350 0x7fa25019e660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:49:16.801 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:16.791+0000 7fa24ffff700 1 -- 172.21.15.29:0/2369436108 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa250102d10 msgr2=0x7fa25019eba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:49:16.801 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:16.791+0000 7fa24ffff700 1 --2- 172.21.15.29:0/2369436108 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa250102d10 0x7fa25019eba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:49:16.802 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:16.791+0000 7fa24ffff700 1 -- 172.21.15.29:0/2369436108 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa250198fc0 con 0x7fa2501083a0 2024-03-16T11:49:16.802 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:16.792+0000 7fa24ffff700 1 --2- 172.21.15.29:0/2369436108 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa2501083a0 0x7fa2501987e0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fa24800ef30 tx=0x7fa24800c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:49:16.802 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:16.792+0000 7fa24cff9700 1 -- 172.21.15.29:0/2369436108 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2480092e0 con 0x7fa2501083a0 2024-03-16T11:49:16.802 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:16.792+0000 7fa255dc9700 1 -- 172.21.15.29:0/2369436108 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa250199250 con 0x7fa2501083a0 2024-03-16T11:49:16.802 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:16.792+0000 7fa24cff9700 1 -- 172.21.15.29:0/2369436108 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa248004bb0 con 0x7fa2501083a0 2024-03-16T11:49:16.802 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:16.792+0000 7fa24cff9700 1 -- 172.21.15.29:0/2369436108 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa248004280 con 0x7fa2501083a0 2024-03-16T11:49:16.802 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:16.792+0000 7fa255dc9700 1 -- 172.21.15.29:0/2369436108 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa2501a54c0 con 0x7fa2501083a0 2024-03-16T11:49:16.802 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:16.792+0000 7fa255dc9700 1 -- 172.21.15.29:0/2369436108 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fa250062a60 con 0x7fa2501083a0 2024-03-16T11:49:16.802 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:16.794+0000 7fa24cff9700 1 -- 172.21.15.29:0/2369436108 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7fa24801b030 con 0x7fa2501083a0 2024-03-16T11:49:16.802 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:16.794+0000 7fa24cff9700 1 --2- 172.21.15.29:0/2369436108 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fa2380a2920 0x7fa2380a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:49:16.802 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:16.794+0000 7fa24f7fe700 1 --2- 172.21.15.29:0/2369436108 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fa2380a2920 0x7fa2380a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:49:16.802 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:16.795+0000 7fa24cff9700 1 -- 172.21.15.29:0/2369436108 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7fa2480cedf0 con 0x7fa2501083a0 2024-03-16T11:49:16.802 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:16.795+0000 7fa24cff9700 1 --2- 172.21.15.29:0/2369436108 >> [v2:172.21.15.29:6818/816506244,v1:172.21.15.29:6819/816506244] conn(0x7fa2380ac100 0x7fa2380ae520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:49:16.802 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:16.795+0000 7fa24cff9700 1 -- 172.21.15.29:0/2369436108 --> [v2:172.21.15.29:6818/816506244,v1:172.21.15.29:6819/816506244] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa24801fe50 con 0x7fa2380ac100 2024-03-16T11:49:16.803 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:16.795+0000 7fa24cff9700 1 -- 172.21.15.29:0/2369436108 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) ==== 24+0+0 (secure 0 0 0) 0x7fa2480cf2d0 con 0x7fa2501083a0 2024-03-16T11:49:16.803 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:16.795+0000 7fa24effd700 1 --2- 172.21.15.29:0/2369436108 >> [v2:172.21.15.29:6818/816506244,v1:172.21.15.29:6819/816506244] conn(0x7fa2380ac100 0x7fa2380ae520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:49:16.803 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:16.795+0000 7fa24f7fe700 1 --2- 172.21.15.29:0/2369436108 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fa2380a2920 0x7fa2380a4de0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fa24400b2e0 tx=0x7fa244005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:49:16.803 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:16.795+0000 7fa24effd700 1 --2- 172.21.15.29:0/2369436108 >> [v2:172.21.15.29:6818/816506244,v1:172.21.15.29:6819/816506244] conn(0x7fa2380ac100 0x7fa2380ae520 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:49:16.803 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:16.796+0000 7fa24cff9700 1 -- 172.21.15.29:0/2369436108 <== osd.6 v2:172.21.15.29:6818/816506244 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fa24801fe50 con 0x7fa2380ac100 2024-03-16T11:49:16.811 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:16.812+0000 7fa255dc9700 1 -- 172.21.15.29:0/2369436108 --> [v2:172.21.15.29:6818/816506244,v1:172.21.15.29:6819/816506244] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fa250062e60 con 0x7fa2380ac100 2024-03-16T11:49:16.812 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:16.813+0000 7fa24cff9700 1 -- 172.21.15.29:0/2369436108 <== osd.6 v2:172.21.15.29:6818/816506244 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fa250062e60 con 0x7fa2380ac100 2024-03-16T11:49:16.813 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:16.813+0000 7fa255dc9700 1 -- 172.21.15.29:0/2369436108 >> [v2:172.21.15.29:6818/816506244,v1:172.21.15.29:6819/816506244] conn(0x7fa2380ac100 msgr2=0x7fa2380ae520 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:49:16.813 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:16.813+0000 7fa255dc9700 1 --2- 172.21.15.29:0/2369436108 >> [v2:172.21.15.29:6818/816506244,v1:172.21.15.29:6819/816506244] conn(0x7fa2380ac100 0x7fa2380ae520 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:49:16.813 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:16.813+0000 7fa255dc9700 1 -- 172.21.15.29:0/2369436108 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fa2380a2920 msgr2=0x7fa2380a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:49:16.813 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:16.813+0000 7fa255dc9700 1 --2- 172.21.15.29:0/2369436108 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fa2380a2920 0x7fa2380a4de0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fa24400b2e0 tx=0x7fa244005960 comp rx=0 tx=0).stop 2024-03-16T11:49:16.813 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:16.813+0000 7fa255dc9700 1 -- 172.21.15.29:0/2369436108 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa2501083a0 msgr2=0x7fa2501987e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:49:16.813 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:16.813+0000 7fa255dc9700 1 --2- 172.21.15.29:0/2369436108 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa2501083a0 0x7fa2501987e0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fa24800ef30 tx=0x7fa24800c3a0 comp rx=0 tx=0).stop 2024-03-16T11:49:16.813 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:16.813+0000 7fa24ffff700 1 -- 172.21.15.29:0/2369436108 reap_dead start 2024-03-16T11:49:16.813 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:16.813+0000 7fa255dc9700 1 -- 172.21.15.29:0/2369436108 shutdown_connections 2024-03-16T11:49:16.813 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:16.813+0000 7fa255dc9700 1 -- 172.21.15.29:0/2369436108 >> 172.21.15.29:0/2369436108 conn(0x7fa250079390 msgr2=0x7fa2501010c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:49:16.813 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:16.814+0000 7fa255dc9700 1 -- 172.21.15.29:0/2369436108 shutdown_connections 2024-03-16T11:49:16.813 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:16.814+0000 7fa255dc9700 1 -- 172.21.15.29:0/2369436108 wait complete. 2024-03-16T11:49:17.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:49:17 smithi029 ceph-mon[31975]: pgmap v310: 97 pgs: 97 active+clean; 640 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 170 B/s wr, 6 op/s 2024-03-16T11:49:17.873 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:17.871+0000 7f2db59cc700 1 -- 172.21.15.29:0/124261707 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2db0069f80 msgr2=0x7f2db006a3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:49:17.873 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:17.871+0000 7f2db59cc700 1 --2- 172.21.15.29:0/124261707 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2db0069f80 0x7f2db006a3e0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f2d980099c0 tx=0x7f2d98009cd0 comp rx=0 tx=0).stop 2024-03-16T11:49:17.873 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:17.871+0000 7f2db59cc700 1 -- 172.21.15.29:0/124261707 shutdown_connections 2024-03-16T11:49:17.873 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:17.871+0000 7f2db59cc700 1 --2- 172.21.15.29:0/124261707 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2db006a920 0x7f2db010f160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:49:17.873 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:17.871+0000 7f2db59cc700 1 --2- 172.21.15.29:0/124261707 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2db0108320 0x7f2db0108700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:49:17.873 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:17.871+0000 7f2db59cc700 1 --2- 172.21.15.29:0/124261707 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2db0069f80 0x7f2db006a3e0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:49:17.874 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:17.871+0000 7f2db59cc700 1 -- 172.21.15.29:0/124261707 >> 172.21.15.29:0/124261707 conn(0x7f2db0078100 msgr2=0x7f2db0078510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:49:17.874 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:17.871+0000 7f2db59cc700 1 -- 172.21.15.29:0/124261707 shutdown_connections 2024-03-16T11:49:17.874 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:17.871+0000 7f2db59cc700 1 -- 172.21.15.29:0/124261707 wait complete. 2024-03-16T11:49:17.874 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:17.872+0000 7f2db59cc700 1 Processor -- start 2024-03-16T11:49:17.874 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:17.872+0000 7f2db59cc700 1 -- start start 2024-03-16T11:49:17.874 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:17.872+0000 7f2db59cc700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2db0069f80 0x7f2db019e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:49:17.874 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:17.872+0000 7f2db59cc700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2db006a920 0x7f2db019ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:49:17.874 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:17.872+0000 7f2db59cc700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2db0108320 0x7f2db0198860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:49:17.874 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:17.872+0000 7f2db59cc700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f2db01125a0 con 0x7f2db006a920 2024-03-16T11:49:17.874 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:17.872+0000 7f2db59cc700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f2db0112420 con 0x7f2db0108320 2024-03-16T11:49:17.874 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:17.872+0000 7f2db59cc700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f2db0112720 con 0x7f2db0069f80 2024-03-16T11:49:17.874 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:17.872+0000 7f2dae7fc700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2db006a920 0x7f2db019ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:49:17.874 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:17.872+0000 7f2dae7fc700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2db006a920 0x7f2db019ebe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:37450/0 (socket says 172.21.15.29:37450) 2024-03-16T11:49:17.875 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:17.872+0000 7f2dae7fc700 1 -- 172.21.15.29:0/1332745288 learned_addr learned my addr 172.21.15.29:0/1332745288 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:49:17.875 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:17.872+0000 7f2daeffd700 1 --2- 172.21.15.29:0/1332745288 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2db0069f80 0x7f2db019e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:49:17.875 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:17.873+0000 7f2daf7fe700 1 --2- 172.21.15.29:0/1332745288 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2db0108320 0x7f2db0198860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:49:17.875 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:17.873+0000 7f2dae7fc700 1 -- 172.21.15.29:0/1332745288 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2db0069f80 msgr2=0x7f2db019e6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:49:17.875 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:17.873+0000 7f2dae7fc700 1 --2- 172.21.15.29:0/1332745288 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2db0069f80 0x7f2db019e6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:49:17.875 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:17.873+0000 7f2dae7fc700 1 -- 172.21.15.29:0/1332745288 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2db0108320 msgr2=0x7f2db0198860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:49:17.875 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:17.873+0000 7f2dae7fc700 1 --2- 172.21.15.29:0/1332745288 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2db0108320 0x7f2db0198860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:49:17.875 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:17.873+0000 7f2dae7fc700 1 -- 172.21.15.29:0/1332745288 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2db0198fc0 con 0x7f2db006a920 2024-03-16T11:49:17.875 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:17.873+0000 7f2dae7fc700 1 --2- 172.21.15.29:0/1332745288 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2db006a920 0x7f2db019ebe0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f2d98006010 tx=0x7f2d9800f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:49:17.875 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:17.873+0000 7f2db49ca700 1 -- 172.21.15.29:0/1332745288 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d98003680 con 0x7f2db006a920 2024-03-16T11:49:17.875 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:17.873+0000 7f2db49ca700 1 -- 172.21.15.29:0/1332745288 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2d980050d0 con 0x7f2db006a920 2024-03-16T11:49:17.875 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:17.873+0000 7f2db59cc700 1 -- 172.21.15.29:0/1332745288 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2db0199250 con 0x7f2db006a920 2024-03-16T11:49:17.875 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:17.873+0000 7f2db49ca700 1 -- 172.21.15.29:0/1332745288 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d98003db0 con 0x7f2db006a920 2024-03-16T11:49:17.875 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:17.873+0000 7f2db59cc700 1 -- 172.21.15.29:0/1332745288 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2db01a5570 con 0x7f2db006a920 2024-03-16T11:49:17.876 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:17.874+0000 7f2db59cc700 1 -- 172.21.15.29:0/1332745288 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f2db0062a60 con 0x7f2db006a920 2024-03-16T11:49:17.877 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:17.875+0000 7f2db49ca700 1 -- 172.21.15.29:0/1332745288 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f2d98005280 con 0x7f2db006a920 2024-03-16T11:49:17.877 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:17.876+0000 7f2db49ca700 1 --2- 172.21.15.29:0/1332745288 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f2d9c0a2b40 0x7f2d9c0a5000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:49:17.877 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:17.876+0000 7f2daeffd700 1 --2- 172.21.15.29:0/1332745288 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f2d9c0a2b40 0x7f2d9c0a5000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:49:17.877 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:17.876+0000 7f2db49ca700 1 -- 172.21.15.29:0/1332745288 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f2d980cfb60 con 0x7f2db006a920 2024-03-16T11:49:17.877 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:17.876+0000 7f2db49ca700 1 --2- 172.21.15.29:0/1332745288 >> [v2:172.21.15.154:6808/1472686560,v1:172.21.15.154:6809/1472686560] conn(0x7f2d9c0ac320 0x7f2d9c0ae740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:49:17.877 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:17.876+0000 7f2db49ca700 1 -- 172.21.15.29:0/1332745288 --> [v2:172.21.15.154:6808/1472686560,v1:172.21.15.154:6809/1472686560] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2d98003680 con 0x7f2d9c0ac320 2024-03-16T11:49:17.877 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:17.876+0000 7f2db49ca700 1 -- 172.21.15.29:0/1332745288 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) ==== 24+0+0 (secure 0 0 0) 0x7f2d980cff10 con 0x7f2db006a920 2024-03-16T11:49:17.877 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:17.876+0000 7f2daf7fe700 1 --2- 172.21.15.29:0/1332745288 >> [v2:172.21.15.154:6808/1472686560,v1:172.21.15.154:6809/1472686560] conn(0x7f2d9c0ac320 0x7f2d9c0ae740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:49:17.877 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:17.876+0000 7f2daeffd700 1 --2- 172.21.15.29:0/1332745288 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f2d9c0a2b40 0x7f2d9c0a5000 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f2db01999a0 tx=0x7f2da800b040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:49:17.877 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:17.877+0000 7f2daf7fe700 1 --2- 172.21.15.29:0/1332745288 >> [v2:172.21.15.154:6808/1472686560,v1:172.21.15.154:6809/1472686560] conn(0x7f2d9c0ac320 0x7f2d9c0ae740 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:49:17.877 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:17.877+0000 7f2db49ca700 1 -- 172.21.15.29:0/1332745288 <== osd.5 v2:172.21.15.154:6808/1472686560 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f2d98003680 con 0x7f2d9c0ac320 2024-03-16T11:49:17.893 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:17.894+0000 7f2db59cc700 1 -- 172.21.15.29:0/1332745288 --> [v2:172.21.15.154:6808/1472686560,v1:172.21.15.154:6809/1472686560] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f2db0062e60 con 0x7f2d9c0ac320 2024-03-16T11:49:17.893 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:17.894+0000 7f2db49ca700 1 -- 172.21.15.29:0/1332745288 <== osd.5 v2:172.21.15.154:6808/1472686560 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f2db0062e60 con 0x7f2d9c0ac320 2024-03-16T11:49:17.894 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:17.895+0000 7f2db59cc700 1 -- 172.21.15.29:0/1332745288 >> [v2:172.21.15.154:6808/1472686560,v1:172.21.15.154:6809/1472686560] conn(0x7f2d9c0ac320 msgr2=0x7f2d9c0ae740 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:49:17.894 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:17.895+0000 7f2db59cc700 1 --2- 172.21.15.29:0/1332745288 >> [v2:172.21.15.154:6808/1472686560,v1:172.21.15.154:6809/1472686560] conn(0x7f2d9c0ac320 0x7f2d9c0ae740 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:49:17.894 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:17.895+0000 7f2db59cc700 1 -- 172.21.15.29:0/1332745288 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f2d9c0a2b40 msgr2=0x7f2d9c0a5000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:49:17.895 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:17.895+0000 7f2db59cc700 1 --2- 172.21.15.29:0/1332745288 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f2d9c0a2b40 0x7f2d9c0a5000 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f2db01999a0 tx=0x7f2da800b040 comp rx=0 tx=0).stop 2024-03-16T11:49:17.895 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:17.895+0000 7f2db59cc700 1 -- 172.21.15.29:0/1332745288 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2db006a920 msgr2=0x7f2db019ebe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:49:17.895 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:17.895+0000 7f2db59cc700 1 --2- 172.21.15.29:0/1332745288 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2db006a920 0x7f2db019ebe0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f2d98006010 tx=0x7f2d9800f690 comp rx=0 tx=0).stop 2024-03-16T11:49:17.895 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:17.895+0000 7f2daf7fe700 1 -- 172.21.15.29:0/1332745288 reap_dead start 2024-03-16T11:49:17.895 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:17.895+0000 7f2db59cc700 1 -- 172.21.15.29:0/1332745288 shutdown_connections 2024-03-16T11:49:17.895 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:17.895+0000 7f2db59cc700 1 -- 172.21.15.29:0/1332745288 >> 172.21.15.29:0/1332745288 conn(0x7f2db0078100 msgr2=0x7f2db00ffee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:49:17.895 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:17.895+0000 7f2db59cc700 1 -- 172.21.15.29:0/1332745288 shutdown_connections 2024-03-16T11:49:17.895 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:17.895+0000 7f2db59cc700 1 -- 172.21.15.29:0/1332745288 wait complete. 2024-03-16T11:49:18.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:49:17 smithi154 ceph-mon[36320]: pgmap v310: 97 pgs: 97 active+clean; 640 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 170 B/s wr, 6 op/s 2024-03-16T11:49:18.102 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:18.097+0000 7f6aeed79700 1 -- 172.21.15.29:0/3072246406 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ae8102580 msgr2=0x7f6ae8102960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:49:18.102 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:18.097+0000 7f6aeed79700 1 --2- 172.21.15.29:0/3072246406 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ae8102580 0x7f6ae8102960 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f6adc0099c0 tx=0x7f6adc009cd0 comp rx=0 tx=0).stop 2024-03-16T11:49:18.103 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:18.097+0000 7f6aeed79700 1 -- 172.21.15.29:0/3072246406 shutdown_connections 2024-03-16T11:49:18.103 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:18.097+0000 7f6aeed79700 1 --2- 172.21.15.29:0/3072246406 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6ae81075a0 0x7f6ae810f170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:49:18.103 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:18.097+0000 7f6aeed79700 1 --2- 172.21.15.29:0/3072246406 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ae8102580 0x7f6ae8102960 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:49:18.103 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:18.097+0000 7f6aeed79700 1 --2- 172.21.15.29:0/3072246406 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6ae8102f30 0x7f6ae8106ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:49:18.103 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:18.097+0000 7f6aeed79700 1 -- 172.21.15.29:0/3072246406 >> 172.21.15.29:0/3072246406 conn(0x7f6ae8079490 msgr2=0x7f6ae80798a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:49:18.103 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:18.097+0000 7f6aeed79700 1 -- 172.21.15.29:0/3072246406 shutdown_connections 2024-03-16T11:49:18.103 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:18.097+0000 7f6aeed79700 1 -- 172.21.15.29:0/3072246406 wait complete. 2024-03-16T11:49:18.103 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:18.098+0000 7f6aeed79700 1 Processor -- start 2024-03-16T11:49:18.103 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:18.098+0000 7f6aeed79700 1 -- start start 2024-03-16T11:49:18.103 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:18.098+0000 7f6aeed79700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6ae8102580 0x7f6ae819e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:49:18.103 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:18.098+0000 7f6aeed79700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ae8102f30 0x7f6ae819ec10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:49:18.103 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:18.098+0000 7f6aeed79700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6ae81075a0 0x7f6ae81988a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:49:18.104 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:18.098+0000 7f6aeed79700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f6ae8112580 con 0x7f6ae8102f30 2024-03-16T11:49:18.104 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:18.098+0000 7f6aeed79700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f6ae8112400 con 0x7f6ae81075a0 2024-03-16T11:49:18.104 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:18.098+0000 7f6aeed79700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f6ae8112700 con 0x7f6ae8102580 2024-03-16T11:49:18.104 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:18.098+0000 7f6ae7fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ae8102f30 0x7f6ae819ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:49:18.104 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:18.098+0000 7f6ae7fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ae8102f30 0x7f6ae819ec10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:37462/0 (socket says 172.21.15.29:37462) 2024-03-16T11:49:18.104 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:18.098+0000 7f6aecb15700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6ae8102580 0x7f6ae819e6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:49:18.104 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:18.098+0000 7f6ae7fff700 1 -- 172.21.15.29:0/2531723478 learned_addr learned my addr 172.21.15.29:0/2531723478 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:49:18.104 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:18.098+0000 7f6aed316700 1 --2- 172.21.15.29:0/2531723478 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6ae81075a0 0x7f6ae81988a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:49:18.104 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:18.098+0000 7f6ae7fff700 1 -- 172.21.15.29:0/2531723478 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6ae8102580 msgr2=0x7f6ae819e6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:49:18.104 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:18.098+0000 7f6ae7fff700 1 --2- 172.21.15.29:0/2531723478 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6ae8102580 0x7f6ae819e6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:49:18.104 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:18.098+0000 7f6ae7fff700 1 -- 172.21.15.29:0/2531723478 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6ae81075a0 msgr2=0x7f6ae81988a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:49:18.104 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:18.098+0000 7f6ae7fff700 1 --2- 172.21.15.29:0/2531723478 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6ae81075a0 0x7f6ae81988a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:49:18.104 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:18.098+0000 7f6ae7fff700 1 -- 172.21.15.29:0/2531723478 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ae8199080 con 0x7f6ae8102f30 2024-03-16T11:49:18.105 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:18.098+0000 7f6aecb15700 1 --2- 172.21.15.29:0/2531723478 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6ae8102580 0x7f6ae819e6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:49:18.105 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:18.098+0000 7f6ae7fff700 1 --2- 172.21.15.29:0/2531723478 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ae8102f30 0x7f6ae819ec10 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f6ad400e970 tx=0x7f6ad400ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:49:18.105 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:18.099+0000 7f6ae5ffb700 1 -- 172.21.15.29:0/2531723478 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ad4004a30 con 0x7f6ae8102f30 2024-03-16T11:49:18.105 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:18.099+0000 7f6aeed79700 1 -- 172.21.15.29:0/2531723478 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6ae8199370 con 0x7f6ae8102f30 2024-03-16T11:49:18.105 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:18.099+0000 7f6ae5ffb700 1 -- 172.21.15.29:0/2531723478 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6ad400ccc0 con 0x7f6ae8102f30 2024-03-16T11:49:18.105 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:18.099+0000 7f6ae5ffb700 1 -- 172.21.15.29:0/2531723478 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ad4005650 con 0x7f6ae8102f30 2024-03-16T11:49:18.105 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:18.099+0000 7f6aeed79700 1 -- 172.21.15.29:0/2531723478 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6ae81a5530 con 0x7f6ae8102f30 2024-03-16T11:49:18.105 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:18.100+0000 7f6aeed79700 1 -- 172.21.15.29:0/2531723478 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f6ae8062a60 con 0x7f6ae8102f30 2024-03-16T11:49:18.105 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:18.101+0000 7f6ae5ffb700 1 -- 172.21.15.29:0/2531723478 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f6ad4004bd0 con 0x7f6ae8102f30 2024-03-16T11:49:18.105 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:18.101+0000 7f6ae5ffb700 1 --2- 172.21.15.29:0/2531723478 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f6ad80a2920 0x7f6ad80a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:49:18.105 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:18.101+0000 7f6aecb15700 1 --2- 172.21.15.29:0/2531723478 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f6ad80a2920 0x7f6ad80a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:49:18.105 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:18.101+0000 7f6ae5ffb700 1 -- 172.21.15.29:0/2531723478 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f6ad40cec30 con 0x7f6ae8102f30 2024-03-16T11:49:18.105 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:18.101+0000 7f6ae5ffb700 1 --2- 172.21.15.29:0/2531723478 >> [v2:172.21.15.152:6824/3412525480,v1:172.21.15.152:6825/3412525480] conn(0x7f6ad80ac100 0x7f6ad80ae520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:49:18.106 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:18.101+0000 7f6ae5ffb700 1 -- 172.21.15.29:0/2531723478 --> [v2:172.21.15.152:6824/3412525480,v1:172.21.15.152:6825/3412525480] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f6ad4021e50 con 0x7f6ad80ac100 2024-03-16T11:49:18.106 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:18.101+0000 7f6ae5ffb700 1 -- 172.21.15.29:0/2531723478 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) ==== 24+0+0 (secure 0 0 0) 0x7f6ad40d1380 con 0x7f6ae8102f30 2024-03-16T11:49:18.106 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:18.102+0000 7f6aecb15700 1 --2- 172.21.15.29:0/2531723478 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f6ad80a2920 0x7f6ad80a4de0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f6adc000c00 tx=0x7f6adc005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:49:18.106 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:18.102+0000 7f6aed316700 1 --2- 172.21.15.29:0/2531723478 >> [v2:172.21.15.152:6824/3412525480,v1:172.21.15.152:6825/3412525480] conn(0x7f6ad80ac100 0x7f6ad80ae520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:49:18.106 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:18.102+0000 7f6aed316700 1 --2- 172.21.15.29:0/2531723478 >> [v2:172.21.15.152:6824/3412525480,v1:172.21.15.152:6825/3412525480] conn(0x7f6ad80ac100 0x7f6ad80ae520 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.10 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:49:18.106 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:18.103+0000 7f6ae5ffb700 1 -- 172.21.15.29:0/2531723478 <== osd.10 v2:172.21.15.152:6824/3412525480 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f6ad4021e50 con 0x7f6ad80ac100 2024-03-16T11:49:18.122 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:18.123+0000 7f6aeed79700 1 -- 172.21.15.29:0/2531723478 --> [v2:172.21.15.152:6824/3412525480,v1:172.21.15.152:6825/3412525480] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f6ae8062e60 con 0x7f6ad80ac100 2024-03-16T11:49:18.122 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:18.123+0000 7f6ae5ffb700 1 -- 172.21.15.29:0/2531723478 <== osd.10 v2:172.21.15.152:6824/3412525480 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f6ae8062e60 con 0x7f6ad80ac100 2024-03-16T11:49:18.123 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:18.124+0000 7f6aeed79700 1 -- 172.21.15.29:0/2531723478 >> [v2:172.21.15.152:6824/3412525480,v1:172.21.15.152:6825/3412525480] conn(0x7f6ad80ac100 msgr2=0x7f6ad80ae520 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:49:18.123 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:18.124+0000 7f6aeed79700 1 --2- 172.21.15.29:0/2531723478 >> [v2:172.21.15.152:6824/3412525480,v1:172.21.15.152:6825/3412525480] conn(0x7f6ad80ac100 0x7f6ad80ae520 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:49:18.123 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:18.124+0000 7f6aeed79700 1 -- 172.21.15.29:0/2531723478 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f6ad80a2920 msgr2=0x7f6ad80a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:49:18.124 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:18.124+0000 7f6aeed79700 1 --2- 172.21.15.29:0/2531723478 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f6ad80a2920 0x7f6ad80a4de0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f6adc000c00 tx=0x7f6adc005fb0 comp rx=0 tx=0).stop 2024-03-16T11:49:18.124 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:18.124+0000 7f6aeed79700 1 -- 172.21.15.29:0/2531723478 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ae8102f30 msgr2=0x7f6ae819ec10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:49:18.124 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:18.124+0000 7f6aeed79700 1 --2- 172.21.15.29:0/2531723478 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ae8102f30 0x7f6ae819ec10 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f6ad400e970 tx=0x7f6ad400ec80 comp rx=0 tx=0).stop 2024-03-16T11:49:18.124 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:18.124+0000 7f6aed316700 1 -- 172.21.15.29:0/2531723478 reap_dead start 2024-03-16T11:49:18.124 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:18.124+0000 7f6aeed79700 1 -- 172.21.15.29:0/2531723478 shutdown_connections 2024-03-16T11:49:18.124 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:18.124+0000 7f6aeed79700 1 -- 172.21.15.29:0/2531723478 >> 172.21.15.29:0/2531723478 conn(0x7f6ae8079490 msgr2=0x7f6ae8106730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:49:18.124 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:18.124+0000 7f6aeed79700 1 -- 172.21.15.29:0/2531723478 shutdown_connections 2024-03-16T11:49:18.124 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:18.124+0000 7f6aeed79700 1 -- 172.21.15.29:0/2531723478 wait complete. 2024-03-16T11:49:18.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:49:17 smithi152 ceph-mon[36291]: pgmap v310: 97 pgs: 97 active+clean; 640 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 170 B/s wr, 6 op/s 2024-03-16T11:49:19.987 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:49:19 smithi029 ceph-mon[31975]: pgmap v311: 97 pgs: 97 active+clean; 640 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 170 B/s wr, 6 op/s 2024-03-16T11:49:20.019 INFO:teuthology.orchestra.run.smithi029.stdout:107374182462 2024-03-16T11:49:20.019 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid d1887f28-e389-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.3 2024-03-16T11:49:20.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:49:19 smithi154 ceph-mon[36320]: pgmap v311: 97 pgs: 97 active+clean; 640 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 170 B/s wr, 6 op/s 2024-03-16T11:49:20.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:49:19 smithi152 ceph-mon[36291]: pgmap v311: 97 pgs: 97 active+clean; 640 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 170 B/s wr, 6 op/s 2024-03-16T11:49:20.265 INFO:teuthology.orchestra.run.smithi029.stdout:163208757301 2024-03-16T11:49:20.265 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid d1887f28-e389-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.5 2024-03-16T11:49:20.520 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:20.509+0000 7fa0d3620700 1 -- 172.21.15.29:0/1205130172 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa0cc102d10 msgr2=0x7fa0cc10f170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:49:20.521 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:20.509+0000 7fa0d3620700 1 --2- 172.21.15.29:0/1205130172 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa0cc102d10 0x7fa0cc10f170 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fa0c400b0a0 tx=0x7fa0c400b3b0 comp rx=0 tx=0).stop 2024-03-16T11:49:20.521 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:20.509+0000 7fa0d3620700 1 -- 172.21.15.29:0/1205130172 shutdown_connections 2024-03-16T11:49:20.521 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:20.509+0000 7fa0d3620700 1 --2- 172.21.15.29:0/1205130172 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa0cc102d10 0x7fa0cc10f170 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:49:20.521 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:20.509+0000 7fa0d3620700 1 --2- 172.21.15.29:0/1205130172 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa0cc1083a0 0x7fa0cc108780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:49:20.521 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:20.509+0000 7fa0d3620700 1 --2- 172.21.15.29:0/1205130172 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa0cc102350 0x7fa0cc1027d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:49:20.521 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:20.509+0000 7fa0d3620700 1 -- 172.21.15.29:0/1205130172 >> 172.21.15.29:0/1205130172 conn(0x7fa0cc079390 msgr2=0x7fa0cc0797a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:49:20.521 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:20.509+0000 7fa0d3620700 1 -- 172.21.15.29:0/1205130172 shutdown_connections 2024-03-16T11:49:20.521 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:20.509+0000 7fa0d3620700 1 -- 172.21.15.29:0/1205130172 wait complete. 2024-03-16T11:49:20.521 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:20.510+0000 7fa0d3620700 1 Processor -- start 2024-03-16T11:49:20.521 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:20.510+0000 7fa0d3620700 1 -- start start 2024-03-16T11:49:20.521 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:20.510+0000 7fa0d3620700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa0cc102350 0x7fa0cc19e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:49:20.522 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:20.510+0000 7fa0d3620700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa0cc102d10 0x7fa0cc19ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:49:20.522 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:20.510+0000 7fa0d3620700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa0cc1083a0 0x7fa0cc198860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:49:20.522 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:20.510+0000 7fa0d3620700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fa0cc1125a0 con 0x7fa0cc1083a0 2024-03-16T11:49:20.522 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:20.510+0000 7fa0d3620700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fa0cc112420 con 0x7fa0cc102350 2024-03-16T11:49:20.522 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:20.510+0000 7fa0d3620700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fa0cc112720 con 0x7fa0cc102d10 2024-03-16T11:49:20.522 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:20.510+0000 7fa0d1bbd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa0cc1083a0 0x7fa0cc198860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:49:20.522 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:20.510+0000 7fa0d1bbd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa0cc1083a0 0x7fa0cc198860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43824/0 (socket says 172.21.15.29:43824) 2024-03-16T11:49:20.522 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:20.510+0000 7fa0d13bc700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa0cc102350 0x7fa0cc19e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:49:20.522 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:20.510+0000 7fa0d0bbb700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa0cc102d10 0x7fa0cc19ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:49:20.522 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:20.510+0000 7fa0d1bbd700 1 -- 172.21.15.29:0/1217485756 learned_addr learned my addr 172.21.15.29:0/1217485756 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:49:20.522 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:20.510+0000 7fa0d1bbd700 1 -- 172.21.15.29:0/1217485756 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa0cc102d10 msgr2=0x7fa0cc19ebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:49:20.522 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:20.510+0000 7fa0d1bbd700 1 --2- 172.21.15.29:0/1217485756 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa0cc102d10 0x7fa0cc19ebe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:49:20.522 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:20.510+0000 7fa0d1bbd700 1 -- 172.21.15.29:0/1217485756 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa0cc102350 msgr2=0x7fa0cc19e6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:49:20.523 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:20.510+0000 7fa0d1bbd700 1 --2- 172.21.15.29:0/1217485756 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa0cc102350 0x7fa0cc19e6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:49:20.523 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:20.510+0000 7fa0d1bbd700 1 -- 172.21.15.29:0/1217485756 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0cc198fc0 con 0x7fa0cc1083a0 2024-03-16T11:49:20.523 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:20.510+0000 7fa0d13bc700 1 --2- 172.21.15.29:0/1217485756 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa0cc102350 0x7fa0cc19e6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:49:20.523 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:20.510+0000 7fa0d1bbd700 1 --2- 172.21.15.29:0/1217485756 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa0cc1083a0 0x7fa0cc198860 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fa0c400b780 tx=0x7fa0c4004d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:49:20.523 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:20.511+0000 7fa0c27fc700 1 -- 172.21.15.29:0/1217485756 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0c4021070 con 0x7fa0cc1083a0 2024-03-16T11:49:20.523 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:20.511+0000 7fa0d3620700 1 -- 172.21.15.29:0/1217485756 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa0cc199250 con 0x7fa0cc1083a0 2024-03-16T11:49:20.523 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:20.511+0000 7fa0c27fc700 1 -- 172.21.15.29:0/1217485756 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa0c4007b80 con 0x7fa0cc1083a0 2024-03-16T11:49:20.523 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:20.511+0000 7fa0c27fc700 1 -- 172.21.15.29:0/1217485756 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0c40052a0 con 0x7fa0cc1083a0 2024-03-16T11:49:20.523 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:20.511+0000 7fa0d3620700 1 -- 172.21.15.29:0/1217485756 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa0cc074500 con 0x7fa0cc1083a0 2024-03-16T11:49:20.523 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:20.511+0000 7fa0d3620700 1 -- 172.21.15.29:0/1217485756 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fa0cc0629e0 con 0x7fa0cc1083a0 2024-03-16T11:49:20.523 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:20.513+0000 7fa0c27fc700 1 -- 172.21.15.29:0/1217485756 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7fa0c4024070 con 0x7fa0cc1083a0 2024-03-16T11:49:20.523 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:20.513+0000 7fa0c27fc700 1 --2- 172.21.15.29:0/1217485756 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fa0b80a2920 0x7fa0b80a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:49:20.523 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:20.513+0000 7fa0d13bc700 1 --2- 172.21.15.29:0/1217485756 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fa0b80a2920 0x7fa0b80a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:49:20.523 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:20.513+0000 7fa0c27fc700 1 -- 172.21.15.29:0/1217485756 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7fa0c400f020 con 0x7fa0cc1083a0 2024-03-16T11:49:20.523 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:20.513+0000 7fa0c27fc700 1 --2- 172.21.15.29:0/1217485756 >> [v2:172.21.15.152:6800/2339408422,v1:172.21.15.152:6801/2339408422] conn(0x7fa0b80abfd0 0x7fa0b80ae3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:49:20.524 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:20.513+0000 7fa0c27fc700 1 -- 172.21.15.29:0/1217485756 --> [v2:172.21.15.152:6800/2339408422,v1:172.21.15.152:6801/2339408422] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa0c4030ea0 con 0x7fa0b80abfd0 2024-03-16T11:49:20.524 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:20.513+0000 7fa0c27fc700 1 -- 172.21.15.29:0/1217485756 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) ==== 24+0+0 (secure 0 0 0) 0x7fa0c40d0670 con 0x7fa0cc1083a0 2024-03-16T11:49:20.524 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:20.514+0000 7fa0d13bc700 1 --2- 172.21.15.29:0/1217485756 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fa0b80a2920 0x7fa0b80a4de0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fa0cc1999a0 tx=0x7fa0c8012040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:49:20.524 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:20.514+0000 7fa0d0bbb700 1 --2- 172.21.15.29:0/1217485756 >> [v2:172.21.15.152:6800/2339408422,v1:172.21.15.152:6801/2339408422] conn(0x7fa0b80abfd0 0x7fa0b80ae3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:49:20.524 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:20.514+0000 7fa0d0bbb700 1 --2- 172.21.15.29:0/1217485756 >> [v2:172.21.15.152:6800/2339408422,v1:172.21.15.152:6801/2339408422] conn(0x7fa0b80abfd0 0x7fa0b80ae3f0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:49:20.524 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:20.515+0000 7fa0c27fc700 1 -- 172.21.15.29:0/1217485756 <== osd.1 v2:172.21.15.152:6800/2339408422 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fa0c4030ea0 con 0x7fa0b80abfd0 2024-03-16T11:49:20.533 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:20.534+0000 7fa0d3620700 1 -- 172.21.15.29:0/1217485756 --> [v2:172.21.15.152:6800/2339408422,v1:172.21.15.152:6801/2339408422] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fa0cc062e10 con 0x7fa0b80abfd0 2024-03-16T11:49:20.533 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:20.534+0000 7fa0c27fc700 1 -- 172.21.15.29:0/1217485756 <== osd.1 v2:172.21.15.152:6800/2339408422 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fa0cc062e10 con 0x7fa0b80abfd0 2024-03-16T11:49:20.534 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:20.534+0000 7fa0d3620700 1 -- 172.21.15.29:0/1217485756 >> [v2:172.21.15.152:6800/2339408422,v1:172.21.15.152:6801/2339408422] conn(0x7fa0b80abfd0 msgr2=0x7fa0b80ae3f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:49:20.534 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:20.534+0000 7fa0d3620700 1 --2- 172.21.15.29:0/1217485756 >> [v2:172.21.15.152:6800/2339408422,v1:172.21.15.152:6801/2339408422] conn(0x7fa0b80abfd0 0x7fa0b80ae3f0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:49:20.534 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:20.534+0000 7fa0d3620700 1 -- 172.21.15.29:0/1217485756 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fa0b80a2920 msgr2=0x7fa0b80a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:49:20.534 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:20.534+0000 7fa0d3620700 1 --2- 172.21.15.29:0/1217485756 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fa0b80a2920 0x7fa0b80a4de0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fa0cc1999a0 tx=0x7fa0c8012040 comp rx=0 tx=0).stop 2024-03-16T11:49:20.534 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:20.535+0000 7fa0d3620700 1 -- 172.21.15.29:0/1217485756 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa0cc1083a0 msgr2=0x7fa0cc198860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:49:20.534 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:20.535+0000 7fa0d3620700 1 --2- 172.21.15.29:0/1217485756 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa0cc1083a0 0x7fa0cc198860 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fa0c400b780 tx=0x7fa0c4004d00 comp rx=0 tx=0).stop 2024-03-16T11:49:20.534 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:20.535+0000 7fa0d1bbd700 1 -- 172.21.15.29:0/1217485756 reap_dead start 2024-03-16T11:49:20.534 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:20.535+0000 7fa0d3620700 1 -- 172.21.15.29:0/1217485756 shutdown_connections 2024-03-16T11:49:20.535 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:20.535+0000 7fa0d3620700 1 -- 172.21.15.29:0/1217485756 >> 172.21.15.29:0/1217485756 conn(0x7fa0cc079390 msgr2=0x7fa0cc100fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:49:20.535 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:20.535+0000 7fa0d3620700 1 -- 172.21.15.29:0/1217485756 shutdown_connections 2024-03-16T11:49:20.535 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:20.535+0000 7fa0d3620700 1 -- 172.21.15.29:0/1217485756 wait complete. 2024-03-16T11:49:21.735 INFO:teuthology.orchestra.run.smithi029.stdout:197568495664 2024-03-16T11:49:21.736 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid d1887f28-e389-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.6 2024-03-16T11:49:22.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:49:21 smithi154 ceph-mon[36320]: pgmap v312: 97 pgs: 97 active+clean; 640 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-16T11:49:22.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:49:21 smithi029 ceph-mon[31975]: pgmap v312: 97 pgs: 97 active+clean; 640 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-16T11:49:22.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:49:21 smithi152 ceph-mon[36291]: pgmap v312: 97 pgs: 97 active+clean; 640 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-16T11:49:22.968 INFO:teuthology.orchestra.run.smithi029.stdout:309237645338 2024-03-16T11:49:22.969 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid d1887f28-e389-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.10 2024-03-16T11:49:23.641 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:23.641+0000 7faf2a3ed700 1 -- 172.21.15.29:0/3843005755 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faf24069f80 msgr2=0x7faf2406a3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:49:23.641 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:23.641+0000 7faf2a3ed700 1 --2- 172.21.15.29:0/3843005755 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faf24069f80 0x7faf2406a3e0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7faf180099c0 tx=0x7faf18009cd0 comp rx=0 tx=0).stop 2024-03-16T11:49:23.641 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:23.641+0000 7faf2a3ed700 1 -- 172.21.15.29:0/3843005755 shutdown_connections 2024-03-16T11:49:23.641 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:23.641+0000 7faf2a3ed700 1 --2- 172.21.15.29:0/3843005755 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7faf2406a920 0x7faf2410f160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:49:23.641 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:23.641+0000 7faf2a3ed700 1 --2- 172.21.15.29:0/3843005755 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faf24108320 0x7faf24108700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:49:23.641 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:23.641+0000 7faf2a3ed700 1 --2- 172.21.15.29:0/3843005755 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faf24069f80 0x7faf2406a3e0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:49:23.641 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:23.641+0000 7faf2a3ed700 1 -- 172.21.15.29:0/3843005755 >> 172.21.15.29:0/3843005755 conn(0x7faf24078100 msgr2=0x7faf24078510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:49:23.641 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:23.641+0000 7faf2a3ed700 1 -- 172.21.15.29:0/3843005755 shutdown_connections 2024-03-16T11:49:23.641 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:23.641+0000 7faf2a3ed700 1 -- 172.21.15.29:0/3843005755 wait complete. 2024-03-16T11:49:23.642 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:23.642+0000 7faf2a3ed700 1 Processor -- start 2024-03-16T11:49:23.642 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:23.642+0000 7faf2a3ed700 1 -- start start 2024-03-16T11:49:23.643 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:23.642+0000 7faf2a3ed700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faf24069f80 0x7faf2419e740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:49:23.643 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:23.642+0000 7faf2a3ed700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faf2406a920 0x7faf2419ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:49:23.643 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:23.642+0000 7faf2a3ed700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7faf24108320 0x7faf24198900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:49:23.643 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:23.642+0000 7faf2a3ed700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7faf24112580 con 0x7faf24069f80 2024-03-16T11:49:23.643 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:23.642+0000 7faf2a3ed700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7faf24112400 con 0x7faf24108320 2024-03-16T11:49:23.644 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:23.642+0000 7faf2a3ed700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7faf24112700 con 0x7faf2406a920 2024-03-16T11:49:23.644 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:23.642+0000 7faf23fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faf24069f80 0x7faf2419e740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:49:23.644 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:23.642+0000 7faf23fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faf24069f80 0x7faf2419e740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43840/0 (socket says 172.21.15.29:43840) 2024-03-16T11:49:23.644 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:23.642+0000 7faf23fff700 1 -- 172.21.15.29:0/2990335304 learned_addr learned my addr 172.21.15.29:0/2990335304 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:49:23.644 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:23.642+0000 7faf237fe700 1 --2- 172.21.15.29:0/2990335304 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faf2406a920 0x7faf2419ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:49:23.644 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:23.643+0000 7faf2898a700 1 --2- 172.21.15.29:0/2990335304 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7faf24108320 0x7faf24198900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:49:23.644 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:23.643+0000 7faf23fff700 1 -- 172.21.15.29:0/2990335304 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faf2406a920 msgr2=0x7faf2419ec80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:49:23.644 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:23.643+0000 7faf23fff700 1 --2- 172.21.15.29:0/2990335304 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faf2406a920 0x7faf2419ec80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:49:23.644 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:23.643+0000 7faf23fff700 1 -- 172.21.15.29:0/2990335304 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7faf24108320 msgr2=0x7faf24198900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:49:23.644 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:23.643+0000 7faf23fff700 1 --2- 172.21.15.29:0/2990335304 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7faf24108320 0x7faf24198900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:49:23.644 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:23.643+0000 7faf23fff700 1 -- 172.21.15.29:0/2990335304 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf24199060 con 0x7faf24069f80 2024-03-16T11:49:23.644 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:23.643+0000 7faf23fff700 1 --2- 172.21.15.29:0/2990335304 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faf24069f80 0x7faf2419e740 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7faf1400d980 tx=0x7faf1400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:49:23.644 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:23.643+0000 7faf217fa700 1 -- 172.21.15.29:0/2990335304 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf14016070 con 0x7faf24069f80 2024-03-16T11:49:23.645 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:23.643+0000 7faf2a3ed700 1 -- 172.21.15.29:0/2990335304 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faf24199350 con 0x7faf24069f80 2024-03-16T11:49:23.645 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:23.643+0000 7faf2a3ed700 1 -- 172.21.15.29:0/2990335304 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faf241a5610 con 0x7faf24069f80 2024-03-16T11:49:23.645 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:23.643+0000 7faf217fa700 1 -- 172.21.15.29:0/2990335304 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faf14004d10 con 0x7faf24069f80 2024-03-16T11:49:23.645 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:23.643+0000 7faf217fa700 1 -- 172.21.15.29:0/2990335304 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf14005670 con 0x7faf24069f80 2024-03-16T11:49:23.645 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:23.644+0000 7faf2a3ed700 1 -- 172.21.15.29:0/2990335304 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7faf240629e0 con 0x7faf24069f80 2024-03-16T11:49:23.646 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:23.645+0000 7faf217fa700 1 -- 172.21.15.29:0/2990335304 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7faf14004750 con 0x7faf24069f80 2024-03-16T11:49:23.646 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:23.646+0000 7faf217fa700 1 --2- 172.21.15.29:0/2990335304 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7faf0c0a2920 0x7faf0c0a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:49:23.646 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:23.646+0000 7faf237fe700 1 --2- 172.21.15.29:0/2990335304 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7faf0c0a2920 0x7faf0c0a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:49:23.646 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:23.646+0000 7faf217fa700 1 -- 172.21.15.29:0/2990335304 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7faf140cee40 con 0x7faf24069f80 2024-03-16T11:49:23.646 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:23.646+0000 7faf217fa700 1 --2- 172.21.15.29:0/2990335304 >> [v2:172.21.15.152:6816/2924651787,v1:172.21.15.152:6817/2924651787] conn(0x7faf0c0ac100 0x7faf0c0ae520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:49:23.646 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:23.646+0000 7faf217fa700 1 -- 172.21.15.29:0/2990335304 --> [v2:172.21.15.152:6816/2924651787,v1:172.21.15.152:6817/2924651787] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7faf14015ec0 con 0x7faf0c0ac100 2024-03-16T11:49:23.646 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:23.646+0000 7faf217fa700 1 -- 172.21.15.29:0/2990335304 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) ==== 24+0+0 (secure 0 0 0) 0x7faf140cf320 con 0x7faf24069f80 2024-03-16T11:49:23.646 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:23.646+0000 7faf2898a700 1 --2- 172.21.15.29:0/2990335304 >> [v2:172.21.15.152:6816/2924651787,v1:172.21.15.152:6817/2924651787] conn(0x7faf0c0ac100 0x7faf0c0ae520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:49:23.646 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:23.647+0000 7faf237fe700 1 --2- 172.21.15.29:0/2990335304 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7faf0c0a2920 0x7faf0c0a4de0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7faf240787c0 tx=0x7faf18005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:49:23.647 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:23.647+0000 7faf2898a700 1 --2- 172.21.15.29:0/2990335304 >> [v2:172.21.15.152:6816/2924651787,v1:172.21.15.152:6817/2924651787] conn(0x7faf0c0ac100 0x7faf0c0ae520 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:49:23.647 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:23.648+0000 7faf217fa700 1 -- 172.21.15.29:0/2990335304 <== osd.7 v2:172.21.15.152:6816/2924651787 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7faf14015ec0 con 0x7faf0c0ac100 2024-03-16T11:49:23.663 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:23.664+0000 7faf2a3ed700 1 -- 172.21.15.29:0/2990335304 --> [v2:172.21.15.152:6816/2924651787,v1:172.21.15.152:6817/2924651787] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7faf24062de0 con 0x7faf0c0ac100 2024-03-16T11:49:23.664 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:23.664+0000 7faf217fa700 1 -- 172.21.15.29:0/2990335304 <== osd.7 v2:172.21.15.152:6816/2924651787 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7faf24062de0 con 0x7faf0c0ac100 2024-03-16T11:49:23.664 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:23.665+0000 7faf2a3ed700 1 -- 172.21.15.29:0/2990335304 >> [v2:172.21.15.152:6816/2924651787,v1:172.21.15.152:6817/2924651787] conn(0x7faf0c0ac100 msgr2=0x7faf0c0ae520 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:49:23.664 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:23.665+0000 7faf2a3ed700 1 --2- 172.21.15.29:0/2990335304 >> [v2:172.21.15.152:6816/2924651787,v1:172.21.15.152:6817/2924651787] conn(0x7faf0c0ac100 0x7faf0c0ae520 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:49:23.664 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:23.665+0000 7faf2a3ed700 1 -- 172.21.15.29:0/2990335304 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7faf0c0a2920 msgr2=0x7faf0c0a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:49:23.664 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:23.665+0000 7faf2a3ed700 1 --2- 172.21.15.29:0/2990335304 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7faf0c0a2920 0x7faf0c0a4de0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7faf240787c0 tx=0x7faf18005fb0 comp rx=0 tx=0).stop 2024-03-16T11:49:23.664 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:23.665+0000 7faf2a3ed700 1 -- 172.21.15.29:0/2990335304 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faf24069f80 msgr2=0x7faf2419e740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:49:23.664 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:23.665+0000 7faf2a3ed700 1 --2- 172.21.15.29:0/2990335304 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faf24069f80 0x7faf2419e740 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7faf1400d980 tx=0x7faf1400dc90 comp rx=0 tx=0).stop 2024-03-16T11:49:23.664 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:23.665+0000 7faf2898a700 1 -- 172.21.15.29:0/2990335304 reap_dead start 2024-03-16T11:49:23.665 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:23.665+0000 7faf2a3ed700 1 -- 172.21.15.29:0/2990335304 shutdown_connections 2024-03-16T11:49:23.665 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:23.665+0000 7faf2a3ed700 1 -- 172.21.15.29:0/2990335304 >> 172.21.15.29:0/2990335304 conn(0x7faf24078100 msgr2=0x7faf240fff70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:49:23.665 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:23.665+0000 7faf2a3ed700 1 -- 172.21.15.29:0/2990335304 shutdown_connections 2024-03-16T11:49:23.665 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:23.665+0000 7faf2a3ed700 1 -- 172.21.15.29:0/2990335304 wait complete. 2024-03-16T11:49:24.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:49:23 smithi154 ceph-mon[36320]: pgmap v313: 97 pgs: 97 active+clean; 640 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 5 op/s 2024-03-16T11:49:24.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:49:23 smithi029 ceph-mon[31975]: pgmap v313: 97 pgs: 97 active+clean; 640 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 5 op/s 2024-03-16T11:49:24.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:49:23 smithi152 ceph-mon[36291]: pgmap v313: 97 pgs: 97 active+clean; 640 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 5 op/s 2024-03-16T11:49:24.627 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:23.886+0000 7fe8dec4a700 1 -- 172.21.15.29:0/590211470 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe8d81011b0 msgr2=0x7fe8d8101630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:49:24.627 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:23.886+0000 7fe8dec4a700 1 --2- 172.21.15.29:0/590211470 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe8d81011b0 0x7fe8d8101630 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fe8c40098f0 tx=0x7fe8c4009c00 comp rx=0 tx=0).stop 2024-03-16T11:49:24.627 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:23.886+0000 7fe8dec4a700 1 -- 172.21.15.29:0/590211470 shutdown_connections 2024-03-16T11:49:24.628 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:23.886+0000 7fe8dec4a700 1 --2- 172.21.15.29:0/590211470 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe8d8101b70 0x7fe8d810e010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:49:24.628 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:23.886+0000 7fe8dec4a700 1 --2- 172.21.15.29:0/590211470 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe8d8107260 0x7fe8d8107640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:49:24.628 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:23.886+0000 7fe8dec4a700 1 --2- 172.21.15.29:0/590211470 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe8d81011b0 0x7fe8d8101630 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:49:24.628 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:23.886+0000 7fe8dec4a700 1 -- 172.21.15.29:0/590211470 >> 172.21.15.29:0/590211470 conn(0x7fe8d80781e0 msgr2=0x7fe8d80785f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:49:24.628 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:23.886+0000 7fe8dec4a700 1 -- 172.21.15.29:0/590211470 shutdown_connections 2024-03-16T11:49:24.628 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:23.886+0000 7fe8dec4a700 1 -- 172.21.15.29:0/590211470 wait complete. 2024-03-16T11:49:24.628 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:23.886+0000 7fe8dec4a700 1 Processor -- start 2024-03-16T11:49:24.628 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:23.886+0000 7fe8dec4a700 1 -- start start 2024-03-16T11:49:24.628 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:23.887+0000 7fe8dec4a700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe8d81011b0 0x7fe8d819e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:49:24.628 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:23.887+0000 7fe8dec4a700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe8d8101b70 0x7fe8d819ebf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:49:24.628 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:23.887+0000 7fe8dec4a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe8d8107260 0x7fe8d8198830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:49:24.628 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:23.887+0000 7fe8dec4a700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fe8d8111470 con 0x7fe8d8107260 2024-03-16T11:49:24.628 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:23.887+0000 7fe8dec4a700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fe8d81112f0 con 0x7fe8d8101b70 2024-03-16T11:49:24.628 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:23.887+0000 7fe8dec4a700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fe8d81115f0 con 0x7fe8d81011b0 2024-03-16T11:49:24.629 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:23.887+0000 7fe8dd1e7700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe8d8107260 0x7fe8d8198830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:49:24.629 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:23.887+0000 7fe8dd1e7700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe8d8107260 0x7fe8d8198830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43850/0 (socket says 172.21.15.29:43850) 2024-03-16T11:49:24.629 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:23.887+0000 7fe8dd1e7700 1 -- 172.21.15.29:0/2950813652 learned_addr learned my addr 172.21.15.29:0/2950813652 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:49:24.629 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:23.887+0000 7fe8d7fff700 1 --2- 172.21.15.29:0/2950813652 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe8d8101b70 0x7fe8d819ebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:49:24.629 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:23.887+0000 7fe8dc9e6700 1 --2- 172.21.15.29:0/2950813652 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe8d81011b0 0x7fe8d819e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:49:24.629 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:23.887+0000 7fe8dd1e7700 1 -- 172.21.15.29:0/2950813652 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe8d81011b0 msgr2=0x7fe8d819e6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:49:24.629 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:23.887+0000 7fe8dd1e7700 1 --2- 172.21.15.29:0/2950813652 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe8d81011b0 0x7fe8d819e6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:49:24.629 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:23.887+0000 7fe8dd1e7700 1 -- 172.21.15.29:0/2950813652 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe8d8101b70 msgr2=0x7fe8d819ebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:49:24.629 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:23.887+0000 7fe8dd1e7700 1 --2- 172.21.15.29:0/2950813652 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe8d8101b70 0x7fe8d819ebf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:49:24.630 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:23.887+0000 7fe8dd1e7700 1 -- 172.21.15.29:0/2950813652 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8d8199010 con 0x7fe8d8107260 2024-03-16T11:49:24.630 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:23.887+0000 7fe8d7fff700 1 --2- 172.21.15.29:0/2950813652 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe8d8101b70 0x7fe8d819ebf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:49:24.630 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:23.887+0000 7fe8dd1e7700 1 --2- 172.21.15.29:0/2950813652 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe8d8107260 0x7fe8d8198830 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fe8d000e3c0 tx=0x7fe8d000e6d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:49:24.630 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:23.887+0000 7fe8d5ffb700 1 -- 172.21.15.29:0/2950813652 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8d00090d0 con 0x7fe8d8107260 2024-03-16T11:49:24.630 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:23.887+0000 7fe8dec4a700 1 -- 172.21.15.29:0/2950813652 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe8d8199300 con 0x7fe8d8107260 2024-03-16T11:49:24.630 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:23.888+0000 7fe8d5ffb700 1 -- 172.21.15.29:0/2950813652 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe8d0009270 con 0x7fe8d8107260 2024-03-16T11:49:24.630 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:23.888+0000 7fe8d5ffb700 1 -- 172.21.15.29:0/2950813652 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8d0004030 con 0x7fe8d8107260 2024-03-16T11:49:24.630 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:23.888+0000 7fe8dec4a700 1 -- 172.21.15.29:0/2950813652 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe8d81a5510 con 0x7fe8d8107260 2024-03-16T11:49:24.630 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:23.888+0000 7fe8dec4a700 1 -- 172.21.15.29:0/2950813652 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fe8d8100090 con 0x7fe8d8107260 2024-03-16T11:49:24.630 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:23.890+0000 7fe8d5ffb700 1 -- 172.21.15.29:0/2950813652 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7fe8d0012050 con 0x7fe8d8107260 2024-03-16T11:49:24.630 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:23.890+0000 7fe8d5ffb700 1 --2- 172.21.15.29:0/2950813652 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fe8c80a2920 0x7fe8c80a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:49:24.630 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:23.890+0000 7fe8d5ffb700 1 -- 172.21.15.29:0/2950813652 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7fe8d00cec50 con 0x7fe8d8107260 2024-03-16T11:49:24.630 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:23.890+0000 7fe8dc9e6700 1 --2- 172.21.15.29:0/2950813652 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fe8c80a2920 0x7fe8c80a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:49:24.630 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:23.890+0000 7fe8d5ffb700 1 --2- 172.21.15.29:0/2950813652 >> [v2:172.21.15.152:6808/2795081750,v1:172.21.15.152:6809/2795081750] conn(0x7fe8c80ac160 0x7fe8c80ae580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:49:24.631 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:23.890+0000 7fe8d5ffb700 1 -- 172.21.15.29:0/2950813652 --> [v2:172.21.15.152:6808/2795081750,v1:172.21.15.152:6809/2795081750] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe8c80aec30 con 0x7fe8c80ac160 2024-03-16T11:49:24.631 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:23.890+0000 7fe8d5ffb700 1 -- 172.21.15.29:0/2950813652 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) ==== 24+0+0 (secure 0 0 0) 0x7fe8d00cf130 con 0x7fe8d8107260 2024-03-16T11:49:24.631 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:23.891+0000 7fe8d7fff700 1 --2- 172.21.15.29:0/2950813652 >> [v2:172.21.15.152:6808/2795081750,v1:172.21.15.152:6809/2795081750] conn(0x7fe8c80ac160 0x7fe8c80ae580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:49:24.631 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:23.891+0000 7fe8dc9e6700 1 --2- 172.21.15.29:0/2950813652 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fe8c80a2920 0x7fe8c80a4de0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fe8cc009680 tx=0x7fe8cc009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:49:24.631 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:23.891+0000 7fe8d7fff700 1 --2- 172.21.15.29:0/2950813652 >> [v2:172.21.15.152:6808/2795081750,v1:172.21.15.152:6809/2795081750] conn(0x7fe8c80ac160 0x7fe8c80ae580 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:49:24.631 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:23.892+0000 7fe8d5ffb700 1 -- 172.21.15.29:0/2950813652 <== osd.4 v2:172.21.15.152:6808/2795081750 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fe8c80aec30 con 0x7fe8c80ac160 2024-03-16T11:49:24.631 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:23.909+0000 7fe8dec4a700 1 -- 172.21.15.29:0/2950813652 --> [v2:172.21.15.152:6808/2795081750,v1:172.21.15.152:6809/2795081750] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fe8d81028a0 con 0x7fe8c80ac160 2024-03-16T11:49:24.631 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:23.909+0000 7fe8d5ffb700 1 -- 172.21.15.29:0/2950813652 <== osd.4 v2:172.21.15.152:6808/2795081750 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fe8d81028a0 con 0x7fe8c80ac160 2024-03-16T11:49:24.631 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:23.909+0000 7fe8dec4a700 1 -- 172.21.15.29:0/2950813652 >> [v2:172.21.15.152:6808/2795081750,v1:172.21.15.152:6809/2795081750] conn(0x7fe8c80ac160 msgr2=0x7fe8c80ae580 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:49:24.631 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:23.909+0000 7fe8dec4a700 1 --2- 172.21.15.29:0/2950813652 >> [v2:172.21.15.152:6808/2795081750,v1:172.21.15.152:6809/2795081750] conn(0x7fe8c80ac160 0x7fe8c80ae580 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:49:24.631 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:23.909+0000 7fe8dec4a700 1 -- 172.21.15.29:0/2950813652 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fe8c80a2920 msgr2=0x7fe8c80a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:49:24.631 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:23.909+0000 7fe8dec4a700 1 --2- 172.21.15.29:0/2950813652 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fe8c80a2920 0x7fe8c80a4de0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fe8cc009680 tx=0x7fe8cc009260 comp rx=0 tx=0).stop 2024-03-16T11:49:24.631 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:23.909+0000 7fe8dec4a700 1 -- 172.21.15.29:0/2950813652 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe8d8107260 msgr2=0x7fe8d8198830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:49:24.631 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:23.909+0000 7fe8dec4a700 1 --2- 172.21.15.29:0/2950813652 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe8d8107260 0x7fe8d8198830 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fe8d000e3c0 tx=0x7fe8d000e6d0 comp rx=0 tx=0).stop 2024-03-16T11:49:24.632 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:23.910+0000 7fe8dd1e7700 1 -- 172.21.15.29:0/2950813652 reap_dead start 2024-03-16T11:49:24.632 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:23.910+0000 7fe8dec4a700 1 -- 172.21.15.29:0/2950813652 shutdown_connections 2024-03-16T11:49:24.632 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:23.910+0000 7fe8dec4a700 1 -- 172.21.15.29:0/2950813652 >> 172.21.15.29:0/2950813652 conn(0x7fe8d80781e0 msgr2=0x7fe8d8105670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:49:24.632 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:23.910+0000 7fe8dec4a700 1 -- 172.21.15.29:0/2950813652 shutdown_connections 2024-03-16T11:49:24.632 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:23.910+0000 7fe8dec4a700 1 -- 172.21.15.29:0/2950813652 wait complete. 2024-03-16T11:49:25.349 INFO:teuthology.orchestra.run.smithi029.stdout:141733920828 2024-03-16T11:49:25.349 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid d1887f28-e389-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.4 2024-03-16T11:49:26.005 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:49:25 smithi029 ceph-mon[31975]: pgmap v314: 97 pgs: 97 active+clean; 640 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2024-03-16T11:49:26.044 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:49:25 smithi154 ceph-mon[36320]: pgmap v314: 97 pgs: 97 active+clean; 640 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2024-03-16T11:49:26.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:49:25 smithi152 ceph-mon[36291]: pgmap v314: 97 pgs: 97 active+clean; 640 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2024-03-16T11:49:26.718 INFO:teuthology.orchestra.run.smithi029.stdout:219043332139 2024-03-16T11:49:26.719 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid d1887f28-e389-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.7 2024-03-16T11:49:26.919 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:26.915+0000 7f53da569700 1 -- 172.21.15.29:0/3589144811 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f53d4101e10 msgr2=0x7f53d4105dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:49:26.919 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:26.915+0000 7f53da569700 1 --2- 172.21.15.29:0/3589144811 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f53d4101e10 0x7f53d4105dd0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f53bc0099c0 tx=0x7f53bc009cd0 comp rx=0 tx=0).stop 2024-03-16T11:49:26.919 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:26.915+0000 7f53da569700 1 -- 172.21.15.29:0/3589144811 shutdown_connections 2024-03-16T11:49:26.919 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:26.915+0000 7f53da569700 1 --2- 172.21.15.29:0/3589144811 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f53d41064b0 0x7f53d410e080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:49:26.919 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:26.915+0000 7f53da569700 1 --2- 172.21.15.29:0/3589144811 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f53d4101460 0x7f53d4101840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:49:26.919 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:26.915+0000 7f53da569700 1 --2- 172.21.15.29:0/3589144811 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f53d4101e10 0x7f53d4105dd0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:49:26.919 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:26.915+0000 7f53da569700 1 -- 172.21.15.29:0/3589144811 >> 172.21.15.29:0/3589144811 conn(0x7f53d4077220 msgr2=0x7f53d4077630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:49:26.920 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:26.915+0000 7f53da569700 1 -- 172.21.15.29:0/3589144811 shutdown_connections 2024-03-16T11:49:26.920 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:26.915+0000 7f53da569700 1 -- 172.21.15.29:0/3589144811 wait complete. 2024-03-16T11:49:26.920 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:26.916+0000 7f53da569700 1 Processor -- start 2024-03-16T11:49:26.920 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:26.916+0000 7f53da569700 1 -- start start 2024-03-16T11:49:26.920 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:26.916+0000 7f53da569700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f53d4101460 0x7f53d41998b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:49:26.920 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:26.916+0000 7f53da569700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f53d4101e10 0x7f53d4199df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:49:26.920 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:26.916+0000 7f53da569700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f53d41064b0 0x7f53d41a4930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:49:26.920 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:26.916+0000 7f53da569700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f53d4111490 con 0x7f53d4101460 2024-03-16T11:49:26.920 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:26.916+0000 7f53da569700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f53d4111310 con 0x7f53d41064b0 2024-03-16T11:49:26.920 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:26.916+0000 7f53da569700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f53d4111610 con 0x7f53d4101e10 2024-03-16T11:49:26.920 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:26.916+0000 7f53d3fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f53d4101460 0x7f53d41998b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:49:26.920 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:26.916+0000 7f53d3fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f53d4101460 0x7f53d41998b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43854/0 (socket says 172.21.15.29:43854) 2024-03-16T11:49:26.920 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:26.916+0000 7f53d3fff700 1 -- 172.21.15.29:0/18528698 learned_addr learned my addr 172.21.15.29:0/18528698 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:49:26.920 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:26.916+0000 7f53cbfff700 1 --2- 172.21.15.29:0/18528698 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f53d4101e10 0x7f53d4199df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:49:26.921 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:26.916+0000 7f53d8b06700 1 --2- 172.21.15.29:0/18528698 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f53d41064b0 0x7f53d41a4930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:49:26.921 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:26.917+0000 7f53d3fff700 1 -- 172.21.15.29:0/18528698 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f53d4101e10 msgr2=0x7f53d4199df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:49:26.921 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:26.917+0000 7f53d3fff700 1 --2- 172.21.15.29:0/18528698 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f53d4101e10 0x7f53d4199df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:49:26.921 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:26.917+0000 7f53d3fff700 1 -- 172.21.15.29:0/18528698 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f53d41064b0 msgr2=0x7f53d41a4930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:49:26.921 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:26.917+0000 7f53d3fff700 1 --2- 172.21.15.29:0/18528698 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f53d41064b0 0x7f53d41a4930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:49:26.921 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:26.917+0000 7f53d3fff700 1 -- 172.21.15.29:0/18528698 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53d41a4e70 con 0x7f53d4101460 2024-03-16T11:49:26.921 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:26.917+0000 7f53d3fff700 1 --2- 172.21.15.29:0/18528698 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f53d4101460 0x7f53d41998b0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f53c400d980 tx=0x7f53c400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:49:26.921 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:26.917+0000 7f53d1ffb700 1 -- 172.21.15.29:0/18528698 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53c4017070 con 0x7f53d4101460 2024-03-16T11:49:26.921 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:26.917+0000 7f53d1ffb700 1 -- 172.21.15.29:0/18528698 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f53c4005020 con 0x7f53d4101460 2024-03-16T11:49:26.921 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:26.917+0000 7f53da569700 1 -- 172.21.15.29:0/18528698 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f53d41a50a0 con 0x7f53d4101460 2024-03-16T11:49:26.921 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:26.917+0000 7f53d1ffb700 1 -- 172.21.15.29:0/18528698 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53c400fbf0 con 0x7f53d4101460 2024-03-16T11:49:26.921 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:26.917+0000 7f53da569700 1 -- 172.21.15.29:0/18528698 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f53d41a55b0 con 0x7f53d4101460 2024-03-16T11:49:26.921 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:26.918+0000 7f53da569700 1 -- 172.21.15.29:0/18528698 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f53d40629a0 con 0x7f53d4101460 2024-03-16T11:49:26.921 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:26.919+0000 7f53d1ffb700 1 -- 172.21.15.29:0/18528698 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f53c40051d0 con 0x7f53d4101460 2024-03-16T11:49:26.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:26.919+0000 7f53d1ffb700 1 --2- 172.21.15.29:0/18528698 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f53c00a29a0 0x7f53c00a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:49:26.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:26.920+0000 7f53cbfff700 1 --2- 172.21.15.29:0/18528698 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f53c00a29a0 0x7f53c00a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:49:26.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:26.920+0000 7f53d1ffb700 1 -- 172.21.15.29:0/18528698 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f53c40ced90 con 0x7f53d4101460 2024-03-16T11:49:26.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:26.920+0000 7f53d1ffb700 1 --2- 172.21.15.29:0/18528698 >> [v2:172.21.15.29:6826/2951414243,v1:172.21.15.29:6827/2951414243] conn(0x7f53c00ac180 0x7f53c00ae5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:49:26.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:26.920+0000 7f53d1ffb700 1 -- 172.21.15.29:0/18528698 --> [v2:172.21.15.29:6826/2951414243,v1:172.21.15.29:6827/2951414243] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f53c00aec50 con 0x7f53c00ac180 2024-03-16T11:49:26.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:26.920+0000 7f53d1ffb700 1 -- 172.21.15.29:0/18528698 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) ==== 24+0+0 (secure 0 0 0) 0x7f53c40cf270 con 0x7f53d4101460 2024-03-16T11:49:26.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:26.920+0000 7f53d8b06700 1 --2- 172.21.15.29:0/18528698 >> [v2:172.21.15.29:6826/2951414243,v1:172.21.15.29:6827/2951414243] conn(0x7f53c00ac180 0x7f53c00ae5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:49:26.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:26.920+0000 7f53cbfff700 1 --2- 172.21.15.29:0/18528698 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f53c00a29a0 0x7f53c00a4e60 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f53bc000c00 tx=0x7f53bc005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:49:26.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:26.920+0000 7f53d8b06700 1 --2- 172.21.15.29:0/18528698 >> [v2:172.21.15.29:6826/2951414243,v1:172.21.15.29:6827/2951414243] conn(0x7f53c00ac180 0x7f53c00ae5a0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.9 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:49:26.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:26.921+0000 7f53d1ffb700 1 -- 172.21.15.29:0/18528698 <== osd.9 v2:172.21.15.29:6826/2951414243 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f53c00aec50 con 0x7f53c00ac180 2024-03-16T11:49:26.937 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:26.938+0000 7f53da569700 1 -- 172.21.15.29:0/18528698 --> [v2:172.21.15.29:6826/2951414243,v1:172.21.15.29:6827/2951414243] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f53d4062e20 con 0x7f53c00ac180 2024-03-16T11:49:26.938 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:26.938+0000 7f53d1ffb700 1 -- 172.21.15.29:0/18528698 <== osd.9 v2:172.21.15.29:6826/2951414243 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f53d4062e20 con 0x7f53c00ac180 2024-03-16T11:49:26.938 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:26.939+0000 7f53da569700 1 -- 172.21.15.29:0/18528698 >> [v2:172.21.15.29:6826/2951414243,v1:172.21.15.29:6827/2951414243] conn(0x7f53c00ac180 msgr2=0x7f53c00ae5a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:49:26.938 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:26.939+0000 7f53da569700 1 --2- 172.21.15.29:0/18528698 >> [v2:172.21.15.29:6826/2951414243,v1:172.21.15.29:6827/2951414243] conn(0x7f53c00ac180 0x7f53c00ae5a0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:49:26.939 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:26.939+0000 7f53da569700 1 -- 172.21.15.29:0/18528698 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f53c00a29a0 msgr2=0x7f53c00a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:49:26.939 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:26.939+0000 7f53da569700 1 --2- 172.21.15.29:0/18528698 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f53c00a29a0 0x7f53c00a4e60 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f53bc000c00 tx=0x7f53bc005fb0 comp rx=0 tx=0).stop 2024-03-16T11:49:26.939 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:26.939+0000 7f53da569700 1 -- 172.21.15.29:0/18528698 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f53d4101460 msgr2=0x7f53d41998b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:49:26.939 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:26.939+0000 7f53da569700 1 --2- 172.21.15.29:0/18528698 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f53d4101460 0x7f53d41998b0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f53c400d980 tx=0x7f53c400dc90 comp rx=0 tx=0).stop 2024-03-16T11:49:26.939 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:26.939+0000 7f53d8b06700 1 -- 172.21.15.29:0/18528698 reap_dead start 2024-03-16T11:49:26.939 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:26.939+0000 7f53da569700 1 -- 172.21.15.29:0/18528698 shutdown_connections 2024-03-16T11:49:26.939 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:26.939+0000 7f53da569700 1 -- 172.21.15.29:0/18528698 >> 172.21.15.29:0/18528698 conn(0x7f53d4077220 msgr2=0x7f53d40ff250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:49:26.939 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:26.939+0000 7f53da569700 1 -- 172.21.15.29:0/18528698 shutdown_connections 2024-03-16T11:49:26.939 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:26.939+0000 7f53da569700 1 -- 172.21.15.29:0/18528698 wait complete. 2024-03-16T11:49:27.483 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:27.483+0000 7f87eff2b700 1 -- 172.21.15.29:0/492729439 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f87e8101ee0 msgr2=0x7f87e810dfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:49:27.483 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:27.483+0000 7f87eff2b700 1 --2- 172.21.15.29:0/492729439 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f87e8101ee0 0x7f87e810dfe0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f87e000b0a0 tx=0x7f87e000b3b0 comp rx=0 tx=0).stop 2024-03-16T11:49:27.483 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:27.483+0000 7f87eff2b700 1 -- 172.21.15.29:0/492729439 shutdown_connections 2024-03-16T11:49:27.483 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:27.483+0000 7f87eff2b700 1 --2- 172.21.15.29:0/492729439 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f87e8101ee0 0x7f87e810dfe0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:49:27.483 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:27.483+0000 7f87eff2b700 1 --2- 172.21.15.29:0/492729439 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f87e8106160 0x7f87e8106540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:49:27.483 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:27.483+0000 7f87eff2b700 1 --2- 172.21.15.29:0/492729439 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f87e8101520 0x7f87e81019a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:49:27.483 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:27.483+0000 7f87eff2b700 1 -- 172.21.15.29:0/492729439 >> 172.21.15.29:0/492729439 conn(0x7f87e8077f80 msgr2=0x7f87e8078390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:49:27.483 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:27.484+0000 7f87eff2b700 1 -- 172.21.15.29:0/492729439 shutdown_connections 2024-03-16T11:49:27.483 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:27.484+0000 7f87eff2b700 1 -- 172.21.15.29:0/492729439 wait complete. 2024-03-16T11:49:27.485 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:27.484+0000 7f87eff2b700 1 Processor -- start 2024-03-16T11:49:27.485 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:27.484+0000 7f87eff2b700 1 -- start start 2024-03-16T11:49:27.485 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:27.484+0000 7f87eff2b700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f87e8101520 0x7f87e819e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:49:27.485 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:27.484+0000 7f87eff2b700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f87e8101ee0 0x7f87e819ec10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:49:27.485 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:27.484+0000 7f87eff2b700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f87e8106160 0x7f87e8198850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:49:27.485 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:27.484+0000 7f87eff2b700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f87e8111580 con 0x7f87e8101520 2024-03-16T11:49:27.485 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:27.484+0000 7f87eff2b700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f87e8111400 con 0x7f87e8101ee0 2024-03-16T11:49:27.485 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:27.484+0000 7f87eff2b700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f87e8111700 con 0x7f87e8106160 2024-03-16T11:49:27.485 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:27.485+0000 7f87edcc7700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f87e8101520 0x7f87e819e6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:49:27.485 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:27.485+0000 7f87edcc7700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f87e8101520 0x7f87e819e6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43874/0 (socket says 172.21.15.29:43874) 2024-03-16T11:49:27.486 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:27.485+0000 7f87edcc7700 1 -- 172.21.15.29:0/2049815553 learned_addr learned my addr 172.21.15.29:0/2049815553 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:49:27.486 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:27.485+0000 7f87ed4c6700 1 --2- 172.21.15.29:0/2049815553 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f87e8101ee0 0x7f87e819ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:49:27.486 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:27.485+0000 7f87ee4c8700 1 --2- 172.21.15.29:0/2049815553 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f87e8106160 0x7f87e8198850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:49:27.486 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:27.485+0000 7f87edcc7700 1 -- 172.21.15.29:0/2049815553 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f87e8106160 msgr2=0x7f87e8198850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:49:27.486 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:27.485+0000 7f87edcc7700 1 --2- 172.21.15.29:0/2049815553 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f87e8106160 0x7f87e8198850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:49:27.486 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:27.485+0000 7f87edcc7700 1 -- 172.21.15.29:0/2049815553 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f87e8101ee0 msgr2=0x7f87e819ec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:49:27.486 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:27.485+0000 7f87edcc7700 1 --2- 172.21.15.29:0/2049815553 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f87e8101ee0 0x7f87e819ec10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:49:27.486 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:27.485+0000 7f87edcc7700 1 -- 172.21.15.29:0/2049815553 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87e8199030 con 0x7f87e8101520 2024-03-16T11:49:27.486 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:27.485+0000 7f87edcc7700 1 --2- 172.21.15.29:0/2049815553 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f87e8101520 0x7f87e819e6d0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f87e400e9a0 tx=0x7f87e400ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:49:27.486 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:27.485+0000 7f87deffd700 1 -- 172.21.15.29:0/2049815553 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87e400ca10 con 0x7f87e8101520 2024-03-16T11:49:27.486 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:27.485+0000 7f87deffd700 1 -- 172.21.15.29:0/2049815553 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f87e400cbb0 con 0x7f87e8101520 2024-03-16T11:49:27.486 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:27.485+0000 7f87eff2b700 1 -- 172.21.15.29:0/2049815553 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f87e8199320 con 0x7f87e8101520 2024-03-16T11:49:27.486 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:27.485+0000 7f87deffd700 1 -- 172.21.15.29:0/2049815553 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87e4018680 con 0x7f87e8101520 2024-03-16T11:49:27.486 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:27.485+0000 7f87eff2b700 1 -- 172.21.15.29:0/2049815553 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f87e81a5530 con 0x7f87e8101520 2024-03-16T11:49:27.487 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:27.486+0000 7f87eff2b700 1 -- 172.21.15.29:0/2049815553 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f87e80629e0 con 0x7f87e8101520 2024-03-16T11:49:27.487 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:27.487+0000 7f87deffd700 1 -- 172.21.15.29:0/2049815553 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f87e4010430 con 0x7f87e8101520 2024-03-16T11:49:27.487 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:27.488+0000 7f87deffd700 1 --2- 172.21.15.29:0/2049815553 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f87d40a29a0 0x7f87d40a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:49:27.487 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:27.488+0000 7f87ed4c6700 1 --2- 172.21.15.29:0/2049815553 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f87d40a29a0 0x7f87d40a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:49:27.487 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:27.488+0000 7f87deffd700 1 -- 172.21.15.29:0/2049815553 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f87e40cf550 con 0x7f87e8101520 2024-03-16T11:49:27.488 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:27.489+0000 7f87deffd700 1 --2- 172.21.15.29:0/2049815553 >> [v2:172.21.15.154:6824/1160927966,v1:172.21.15.154:6825/1160927966] conn(0x7f87d40ac180 0x7f87d40ae5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:49:27.488 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:27.489+0000 7f87deffd700 1 -- 172.21.15.29:0/2049815553 --> [v2:172.21.15.154:6824/1160927966,v1:172.21.15.154:6825/1160927966] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f87e401bec0 con 0x7f87d40ac180 2024-03-16T11:49:27.488 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:27.489+0000 7f87deffd700 1 -- 172.21.15.29:0/2049815553 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) ==== 24+0+0 (secure 0 0 0) 0x7f87e40cf900 con 0x7f87e8101520 2024-03-16T11:49:27.488 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:27.489+0000 7f87ee4c8700 1 --2- 172.21.15.29:0/2049815553 >> [v2:172.21.15.154:6824/1160927966,v1:172.21.15.154:6825/1160927966] conn(0x7f87d40ac180 0x7f87d40ae5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:49:27.489 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:27.489+0000 7f87ed4c6700 1 --2- 172.21.15.29:0/2049815553 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f87d40a29a0 0x7f87d40a4e60 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f87e80785d0 tx=0x7f87d8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:49:27.489 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:27.489+0000 7f87ee4c8700 1 --2- 172.21.15.29:0/2049815553 >> [v2:172.21.15.154:6824/1160927966,v1:172.21.15.154:6825/1160927966] conn(0x7f87d40ac180 0x7f87d40ae5a0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.11 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:49:27.489 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:27.490+0000 7f87deffd700 1 -- 172.21.15.29:0/2049815553 <== osd.11 v2:172.21.15.154:6824/1160927966 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f87e401bec0 con 0x7f87d40ac180 2024-03-16T11:49:27.507 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:27.508+0000 7f87eff2b700 1 -- 172.21.15.29:0/2049815553 --> [v2:172.21.15.154:6824/1160927966,v1:172.21.15.154:6825/1160927966] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f87e8062de0 con 0x7f87d40ac180 2024-03-16T11:49:27.507 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:27.508+0000 7f87deffd700 1 -- 172.21.15.29:0/2049815553 <== osd.11 v2:172.21.15.154:6824/1160927966 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f87e8062de0 con 0x7f87d40ac180 2024-03-16T11:49:27.508 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:27.508+0000 7f87eff2b700 1 -- 172.21.15.29:0/2049815553 >> [v2:172.21.15.154:6824/1160927966,v1:172.21.15.154:6825/1160927966] conn(0x7f87d40ac180 msgr2=0x7f87d40ae5a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:49:27.508 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:27.508+0000 7f87eff2b700 1 --2- 172.21.15.29:0/2049815553 >> [v2:172.21.15.154:6824/1160927966,v1:172.21.15.154:6825/1160927966] conn(0x7f87d40ac180 0x7f87d40ae5a0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:49:27.508 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:27.508+0000 7f87eff2b700 1 -- 172.21.15.29:0/2049815553 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f87d40a29a0 msgr2=0x7f87d40a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:49:27.508 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:27.508+0000 7f87eff2b700 1 --2- 172.21.15.29:0/2049815553 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f87d40a29a0 0x7f87d40a4e60 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f87e80785d0 tx=0x7f87d8009260 comp rx=0 tx=0).stop 2024-03-16T11:49:27.508 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:27.508+0000 7f87eff2b700 1 -- 172.21.15.29:0/2049815553 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f87e8101520 msgr2=0x7f87e819e6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:49:27.508 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:27.508+0000 7f87eff2b700 1 --2- 172.21.15.29:0/2049815553 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f87e8101520 0x7f87e819e6d0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f87e400e9a0 tx=0x7f87e400ecb0 comp rx=0 tx=0).stop 2024-03-16T11:49:27.508 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:27.509+0000 7f87ee4c8700 1 -- 172.21.15.29:0/2049815553 reap_dead start 2024-03-16T11:49:27.508 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:27.509+0000 7f87eff2b700 1 -- 172.21.15.29:0/2049815553 shutdown_connections 2024-03-16T11:49:27.508 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:27.509+0000 7f87eff2b700 1 -- 172.21.15.29:0/2049815553 >> 172.21.15.29:0/2049815553 conn(0x7f87e8077f80 msgr2=0x7f87e806a200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:49:27.508 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:27.509+0000 7f87eff2b700 1 -- 172.21.15.29:0/2049815553 shutdown_connections 2024-03-16T11:49:27.508 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:27.509+0000 7f87eff2b700 1 -- 172.21.15.29:0/2049815553 wait complete. 2024-03-16T11:49:27.754 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:27.722+0000 7f90076fd700 1 -- 172.21.15.29:0/2226048935 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9000101430 msgr2=0x7f9000101810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:49:27.754 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:27.722+0000 7f90076fd700 1 --2- 172.21.15.29:0/2226048935 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9000101430 0x7f9000101810 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f8ffc0099c0 tx=0x7f8ffc009cd0 comp rx=0 tx=0).stop 2024-03-16T11:49:27.754 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:27.722+0000 7f90076fd700 1 -- 172.21.15.29:0/2226048935 shutdown_connections 2024-03-16T11:49:27.754 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:27.722+0000 7f90076fd700 1 --2- 172.21.15.29:0/2226048935 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9000106510 0x7f900010e0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:49:27.754 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:27.722+0000 7f90076fd700 1 --2- 172.21.15.29:0/2226048935 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9000101430 0x7f9000101810 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:49:27.754 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:27.722+0000 7f90076fd700 1 --2- 172.21.15.29:0/2226048935 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9000101de0 0x7f9000105e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:49:27.755 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:27.722+0000 7f90076fd700 1 -- 172.21.15.29:0/2226048935 >> 172.21.15.29:0/2226048935 conn(0x7f90000782a0 msgr2=0x7f90000786b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:49:27.755 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:27.722+0000 7f90076fd700 1 -- 172.21.15.29:0/2226048935 shutdown_connections 2024-03-16T11:49:27.755 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:27.722+0000 7f90076fd700 1 -- 172.21.15.29:0/2226048935 wait complete. 2024-03-16T11:49:27.755 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:27.723+0000 7f90076fd700 1 Processor -- start 2024-03-16T11:49:27.755 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:27.723+0000 7f90076fd700 1 -- start start 2024-03-16T11:49:27.755 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:27.723+0000 7f90076fd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9000101430 0x7f900019e730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:49:27.755 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:27.723+0000 7f90076fd700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9000101de0 0x7f900019ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:49:27.755 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:27.723+0000 7f90076fd700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9000106510 0x7f90001988f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:49:27.755 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:27.723+0000 7f90076fd700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f9000111570 con 0x7f9000101430 2024-03-16T11:49:27.755 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:27.723+0000 7f90076fd700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f90001113f0 con 0x7f9000101de0 2024-03-16T11:49:27.755 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:27.723+0000 7f90076fd700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f90001116f0 con 0x7f9000106510 2024-03-16T11:49:27.755 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:27.723+0000 7f9005499700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9000101430 0x7f900019e730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:49:27.755 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:27.723+0000 7f9005499700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9000101430 0x7f900019e730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43888/0 (socket says 172.21.15.29:43888) 2024-03-16T11:49:27.755 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:27.723+0000 7f9005499700 1 -- 172.21.15.29:0/3094367256 learned_addr learned my addr 172.21.15.29:0/3094367256 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:49:27.755 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:27.723+0000 7f9004c98700 1 --2- 172.21.15.29:0/3094367256 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9000101de0 0x7f900019ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:49:27.755 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:27.723+0000 7f9005c9a700 1 --2- 172.21.15.29:0/3094367256 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9000106510 0x7f90001988f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:49:27.756 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:27.724+0000 7f9005499700 1 -- 172.21.15.29:0/3094367256 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9000106510 msgr2=0x7f90001988f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:49:27.756 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:27.724+0000 7f9005499700 1 --2- 172.21.15.29:0/3094367256 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9000106510 0x7f90001988f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:49:27.756 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:27.724+0000 7f9005499700 1 -- 172.21.15.29:0/3094367256 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9000101de0 msgr2=0x7f900019ec70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:49:27.756 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:27.724+0000 7f9005499700 1 --2- 172.21.15.29:0/3094367256 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9000101de0 0x7f900019ec70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:49:27.756 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:27.724+0000 7f9005499700 1 -- 172.21.15.29:0/3094367256 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9000199050 con 0x7f9000101430 2024-03-16T11:49:27.756 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:27.724+0000 7f9004c98700 1 --2- 172.21.15.29:0/3094367256 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9000101de0 0x7f900019ec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:49:27.756 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:27.724+0000 7f9005499700 1 --2- 172.21.15.29:0/3094367256 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9000101430 0x7f900019e730 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f8ffc005910 tx=0x7f8ffc00f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:49:27.756 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:27.724+0000 7f8ff67fc700 1 -- 172.21.15.29:0/3094367256 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ffc027070 con 0x7f9000101430 2024-03-16T11:49:27.756 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:27.724+0000 7f8ff67fc700 1 -- 172.21.15.29:0/3094367256 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8ffc003680 con 0x7f9000101430 2024-03-16T11:49:27.756 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:27.724+0000 7f8ff67fc700 1 -- 172.21.15.29:0/3094367256 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ffc0173f0 con 0x7f9000101430 2024-03-16T11:49:27.757 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:27.724+0000 7f90076fd700 1 -- 172.21.15.29:0/3094367256 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f90001992e0 con 0x7f9000101430 2024-03-16T11:49:27.757 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:27.724+0000 7f90076fd700 1 -- 172.21.15.29:0/3094367256 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f90001a5600 con 0x7f9000101430 2024-03-16T11:49:27.757 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:27.725+0000 7f90076fd700 1 -- 172.21.15.29:0/3094367256 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f9000062a20 con 0x7f9000101430 2024-03-16T11:49:27.757 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:27.726+0000 7f8ff67fc700 1 -- 172.21.15.29:0/3094367256 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f8ffc022590 con 0x7f9000101430 2024-03-16T11:49:27.757 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:27.727+0000 7f8ff67fc700 1 --2- 172.21.15.29:0/3094367256 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f8fec0a29a0 0x7f8fec0a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:49:27.757 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:27.727+0000 7f9004c98700 1 --2- 172.21.15.29:0/3094367256 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f8fec0a29a0 0x7f8fec0a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:49:27.757 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:27.727+0000 7f8ff67fc700 1 -- 172.21.15.29:0/3094367256 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 8211+0+0 (secure 0 0 0) 0x7f8ffc0cfc00 con 0x7f9000101430 2024-03-16T11:49:27.757 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:27.727+0000 7f8ff67fc700 1 --2- 172.21.15.29:0/3094367256 >> [v2:172.21.15.154:6800/4288325280,v1:172.21.15.154:6801/4288325280] conn(0x7f8fec0ac3b0 0x7f8fec0ae850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:49:27.757 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:27.727+0000 7f8ff67fc700 1 -- 172.21.15.29:0/3094367256 --> [v2:172.21.15.154:6800/4288325280,v1:172.21.15.154:6801/4288325280] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f8ffc00b0b0 con 0x7f8fec0ac3b0 2024-03-16T11:49:27.757 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:27.727+0000 7f8ff67fc700 1 -- 172.21.15.29:0/3094367256 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) ==== 24+0+0 (secure 0 0 0) 0x7f8ffc0cffb0 con 0x7f9000101430 2024-03-16T11:49:27.757 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:27.727+0000 7f9004c98700 1 --2- 172.21.15.29:0/3094367256 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f8fec0a29a0 0x7f8fec0a4e60 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f9000107ab0 tx=0x7f8ff0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:49:27.757 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:27.727+0000 7f9005c9a700 1 --2- 172.21.15.29:0/3094367256 >> [v2:172.21.15.154:6800/4288325280,v1:172.21.15.154:6801/4288325280] conn(0x7f8fec0ac3b0 0x7f8fec0ae850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:49:27.757 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:27.727+0000 7f9005c9a700 1 --2- 172.21.15.29:0/3094367256 >> [v2:172.21.15.154:6800/4288325280,v1:172.21.15.154:6801/4288325280] conn(0x7f8fec0ac3b0 0x7f8fec0ae850 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:49:27.757 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:27.728+0000 7f8ff67fc700 1 -- 172.21.15.29:0/3094367256 <== osd.2 v2:172.21.15.154:6800/4288325280 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f8ffc00b0b0 con 0x7f8fec0ac3b0 2024-03-16T11:49:27.757 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:27.745+0000 7f90076fd700 1 -- 172.21.15.29:0/3094367256 --> [v2:172.21.15.154:6800/4288325280,v1:172.21.15.154:6801/4288325280] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f9000062e20 con 0x7f8fec0ac3b0 2024-03-16T11:49:27.757 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:27.745+0000 7f8ff67fc700 1 -- 172.21.15.29:0/3094367256 <== osd.2 v2:172.21.15.154:6800/4288325280 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f9000062e20 con 0x7f8fec0ac3b0 2024-03-16T11:49:27.758 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:27.745+0000 7f90076fd700 1 -- 172.21.15.29:0/3094367256 >> [v2:172.21.15.154:6800/4288325280,v1:172.21.15.154:6801/4288325280] conn(0x7f8fec0ac3b0 msgr2=0x7f8fec0ae850 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:49:27.758 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:27.745+0000 7f90076fd700 1 --2- 172.21.15.29:0/3094367256 >> [v2:172.21.15.154:6800/4288325280,v1:172.21.15.154:6801/4288325280] conn(0x7f8fec0ac3b0 0x7f8fec0ae850 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:49:27.758 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:27.745+0000 7f90076fd700 1 -- 172.21.15.29:0/3094367256 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f8fec0a29a0 msgr2=0x7f8fec0a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:49:27.758 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:27.745+0000 7f90076fd700 1 --2- 172.21.15.29:0/3094367256 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f8fec0a29a0 0x7f8fec0a4e60 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f9000107ab0 tx=0x7f8ff0009260 comp rx=0 tx=0).stop 2024-03-16T11:49:27.758 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:27.745+0000 7f90076fd700 1 -- 172.21.15.29:0/3094367256 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9000101430 msgr2=0x7f900019e730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:49:27.758 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:27.745+0000 7f90076fd700 1 --2- 172.21.15.29:0/3094367256 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9000101430 0x7f900019e730 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f8ffc005910 tx=0x7f8ffc00f690 comp rx=0 tx=0).stop 2024-03-16T11:49:27.758 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:27.745+0000 7f9005c9a700 1 -- 172.21.15.29:0/3094367256 reap_dead start 2024-03-16T11:49:27.758 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:27.745+0000 7f90076fd700 1 -- 172.21.15.29:0/3094367256 shutdown_connections 2024-03-16T11:49:27.758 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:27.745+0000 7f90076fd700 1 -- 172.21.15.29:0/3094367256 >> 172.21.15.29:0/3094367256 conn(0x7f90000782a0 msgr2=0x7f900006acd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:49:27.758 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:27.745+0000 7f90076fd700 1 -- 172.21.15.29:0/3094367256 shutdown_connections 2024-03-16T11:49:27.758 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:27.745+0000 7f90076fd700 1 -- 172.21.15.29:0/3094367256 wait complete. 2024-03-16T11:49:28.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:49:27 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]: dispatch 2024-03-16T11:49:28.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:49:27 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.14", "id": [2, 4]}]: dispatch 2024-03-16T11:49:28.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:49:27 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [2, 1]}]: dispatch 2024-03-16T11:49:28.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:49:27 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [8, 11]}]: dispatch 2024-03-16T11:49:28.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:49:27 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.13", "id": [4, 9]}]: dispatch 2024-03-16T11:49:28.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:49:27 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 8]}]: dispatch 2024-03-16T11:49:28.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:49:27 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2c", "id": [4, 9]}]: dispatch 2024-03-16T11:49:28.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:49:27 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.33", "id": [5, 9]}]: dispatch 2024-03-16T11:49:28.087 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:49:27 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]: dispatch 2024-03-16T11:49:28.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:49:27 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.14", "id": [2, 4]}]: dispatch 2024-03-16T11:49:28.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:49:27 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [2, 1]}]: dispatch 2024-03-16T11:49:28.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:49:27 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [8, 11]}]: dispatch 2024-03-16T11:49:28.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:49:27 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.13", "id": [4, 9]}]: dispatch 2024-03-16T11:49:28.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:49:27 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 8]}]: dispatch 2024-03-16T11:49:28.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:49:27 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2c", "id": [4, 9]}]: dispatch 2024-03-16T11:49:28.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:49:27 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.33", "id": [5, 9]}]: dispatch 2024-03-16T11:49:28.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:49:27 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]: dispatch 2024-03-16T11:49:28.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:49:27 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.14", "id": [2, 4]}]: dispatch 2024-03-16T11:49:28.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:49:27 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [2, 1]}]: dispatch 2024-03-16T11:49:28.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:49:27 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [8, 11]}]: dispatch 2024-03-16T11:49:28.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:49:27 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.13", "id": [4, 9]}]: dispatch 2024-03-16T11:49:28.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:49:27 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 8]}]: dispatch 2024-03-16T11:49:28.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:49:27 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2c", "id": [4, 9]}]: dispatch 2024-03-16T11:49:28.128 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:49:27 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.33", "id": [5, 9]}]: dispatch 2024-03-16T11:49:28.648 INFO:teuthology.orchestra.run.smithi029.stdout:34359738446 2024-03-16T11:49:28.648 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid d1887f28-e389-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.0 2024-03-16T11:49:28.991 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:49:28 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]': finished 2024-03-16T11:49:28.991 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:49:28 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.14", "id": [2, 4]}]': finished 2024-03-16T11:49:28.991 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:49:28 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [2, 1]}]': finished 2024-03-16T11:49:28.991 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:49:28 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [8, 11]}]': finished 2024-03-16T11:49:28.991 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:49:28 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.13", "id": [4, 9]}]': finished 2024-03-16T11:49:28.991 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:49:28 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 8]}]': finished 2024-03-16T11:49:28.991 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:49:28 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2c", "id": [4, 9]}]': finished 2024-03-16T11:49:28.991 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:49:28 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.33", "id": [5, 9]}]': finished 2024-03-16T11:49:28.991 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:49:28 smithi029 ceph-mon[31975]: pgmap v316: 97 pgs: 97 active+clean; 640 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-16T11:49:28.991 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:49:28 smithi029 ceph-mon[31975]: osdmap e86: 12 total, 12 up, 12 in 2024-03-16T11:49:28.991 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:49:28 smithi029 ceph-mon[31975]: osdmap e87: 12 total, 12 up, 12 in 2024-03-16T11:49:29.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:49:28 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]': finished 2024-03-16T11:49:29.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:49:28 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.14", "id": [2, 4]}]': finished 2024-03-16T11:49:29.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:49:28 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [2, 1]}]': finished 2024-03-16T11:49:29.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:49:28 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [8, 11]}]': finished 2024-03-16T11:49:29.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:49:28 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.13", "id": [4, 9]}]': finished 2024-03-16T11:49:29.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:49:28 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 8]}]': finished 2024-03-16T11:49:29.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:49:28 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2c", "id": [4, 9]}]': finished 2024-03-16T11:49:29.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:49:28 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.33", "id": [5, 9]}]': finished 2024-03-16T11:49:29.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:49:28 smithi154 ceph-mon[36320]: pgmap v316: 97 pgs: 97 active+clean; 640 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-16T11:49:29.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:49:28 smithi154 ceph-mon[36320]: osdmap e86: 12 total, 12 up, 12 in 2024-03-16T11:49:29.046 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:49:28 smithi154 ceph-mon[36320]: osdmap e87: 12 total, 12 up, 12 in 2024-03-16T11:49:29.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:49:28 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]': finished 2024-03-16T11:49:29.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:49:28 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.14", "id": [2, 4]}]': finished 2024-03-16T11:49:29.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:49:28 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [2, 1]}]': finished 2024-03-16T11:49:29.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:49:28 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [8, 11]}]': finished 2024-03-16T11:49:29.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:49:28 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.13", "id": [4, 9]}]': finished 2024-03-16T11:49:29.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:49:28 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 8]}]': finished 2024-03-16T11:49:29.128 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:49:28 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2c", "id": [4, 9]}]': finished 2024-03-16T11:49:29.128 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:49:28 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.33", "id": [5, 9]}]': finished 2024-03-16T11:49:29.128 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:49:28 smithi152 ceph-mon[36291]: pgmap v316: 97 pgs: 97 active+clean; 640 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-16T11:49:29.128 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:49:28 smithi152 ceph-mon[36291]: osdmap e86: 12 total, 12 up, 12 in 2024-03-16T11:49:29.128 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:49:28 smithi152 ceph-mon[36291]: osdmap e87: 12 total, 12 up, 12 in 2024-03-16T11:49:29.341 INFO:teuthology.orchestra.run.smithi029.stdout:55834574922 2024-03-16T11:49:29.342 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid d1887f28-e389-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.1 2024-03-16T11:49:29.724 INFO:teuthology.orchestra.run.smithi029.stdout:77309411399 2024-03-16T11:49:29.724 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid d1887f28-e389-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.2 2024-03-16T11:49:30.196 INFO:teuthology.orchestra.run.smithi029.stdout:287762808866 2024-03-16T11:49:30.197 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid d1887f28-e389-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.9 2024-03-16T11:49:30.712 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/d1887f28-e389-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T11:49:30.713 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/d1887f28-e389-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T11:49:30.714 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/d1887f28-e389-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T11:49:30.714 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/d1887f28-e389-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T11:49:30.779 INFO:teuthology.orchestra.run.smithi029.stdout:343597383704 2024-03-16T11:49:30.780 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid d1887f28-e389-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.11 2024-03-16T11:49:30.880 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:49:30 smithi029 ceph-mon[31975]: pgmap v318: 97 pgs: 1 peering, 96 active+clean; 640 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 6 op/s; 7.3 KiB/s, 0 objects/s recovering 2024-03-16T11:49:31.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:49:30 smithi154 ceph-mon[36320]: pgmap v318: 97 pgs: 1 peering, 96 active+clean; 640 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 6 op/s; 7.3 KiB/s, 0 objects/s recovering 2024-03-16T11:49:31.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:49:30 smithi152 ceph-mon[36291]: pgmap v318: 97 pgs: 1 peering, 96 active+clean; 640 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 6 op/s; 7.3 KiB/s, 0 objects/s recovering 2024-03-16T11:49:31.317 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/d1887f28-e389-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T11:49:31.318 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/d1887f28-e389-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T11:49:31.322 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/d1887f28-e389-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T11:49:31.322 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/d1887f28-e389-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T11:49:32.837 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:49:32 smithi029 ceph-mon[31975]: pgmap v319: 97 pgs: 1 peering, 96 active+clean; 640 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s; 7.4 KiB/s, 1 keys/s, 1 objects/s recovering 2024-03-16T11:49:33.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:49:32 smithi154 ceph-mon[36320]: pgmap v319: 97 pgs: 1 peering, 96 active+clean; 640 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s; 7.4 KiB/s, 1 keys/s, 1 objects/s recovering 2024-03-16T11:49:33.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:49:32 smithi152 ceph-mon[36291]: pgmap v319: 97 pgs: 1 peering, 96 active+clean; 640 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s; 7.4 KiB/s, 1 keys/s, 1 objects/s recovering 2024-03-16T11:49:34.859 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:49:34 smithi029 ceph-mon[31975]: pgmap v320: 97 pgs: 97 active+clean; 640 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s; 7.4 KiB/s, 1 keys/s, 1 objects/s recovering 2024-03-16T11:49:35.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:49:34 smithi154 ceph-mon[36320]: pgmap v320: 97 pgs: 97 active+clean; 640 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s; 7.4 KiB/s, 1 keys/s, 1 objects/s recovering 2024-03-16T11:49:35.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:49:34 smithi152 ceph-mon[36291]: pgmap v320: 97 pgs: 97 active+clean; 640 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s; 7.4 KiB/s, 1 keys/s, 1 objects/s recovering 2024-03-16T11:49:36.988 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:49:36 smithi029 ceph-mon[31975]: pgmap v321: 97 pgs: 97 active+clean; 640 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 5 op/s; 7.4 KiB/s, 1 keys/s, 1 objects/s recovering 2024-03-16T11:49:37.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:49:36 smithi154 ceph-mon[36320]: pgmap v321: 97 pgs: 97 active+clean; 640 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 5 op/s; 7.4 KiB/s, 1 keys/s, 1 objects/s recovering 2024-03-16T11:49:37.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:49:36 smithi152 ceph-mon[36291]: pgmap v321: 97 pgs: 97 active+clean; 640 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 5 op/s; 7.4 KiB/s, 1 keys/s, 1 objects/s recovering 2024-03-16T11:49:37.671 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/d1887f28-e389-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T11:49:37.673 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/d1887f28-e389-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T11:49:39.006 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:49:38 smithi029 ceph-mon[31975]: pgmap v322: 97 pgs: 97 active+clean; 640 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s; 6.0 KiB/s, 0 keys/s, 0 objects/s recovering 2024-03-16T11:49:39.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:49:38 smithi154 ceph-mon[36320]: pgmap v322: 97 pgs: 97 active+clean; 640 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s; 6.0 KiB/s, 0 keys/s, 0 objects/s recovering 2024-03-16T11:49:39.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:49:38 smithi152 ceph-mon[36291]: pgmap v322: 97 pgs: 97 active+clean; 640 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s; 6.0 KiB/s, 0 keys/s, 0 objects/s recovering 2024-03-16T11:49:39.599 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:39.599+0000 7fa6b7421700 1 -- 172.21.15.29:0/3019277211 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa6b0108300 msgr2=0x7fa6b01086e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:49:39.599 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:39.599+0000 7fa6b7421700 1 --2- 172.21.15.29:0/3019277211 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa6b0108300 0x7fa6b01086e0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fa6a000b0d0 tx=0x7fa6a000b3e0 comp rx=0 tx=0).stop 2024-03-16T11:49:39.599 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:39.600+0000 7fa6b7421700 1 -- 172.21.15.29:0/3019277211 shutdown_connections 2024-03-16T11:49:39.599 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:39.600+0000 7fa6b7421700 1 --2- 172.21.15.29:0/3019277211 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa6b0102ce0 0x7fa6b010f120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:49:39.599 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:39.600+0000 7fa6b7421700 1 --2- 172.21.15.29:0/3019277211 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa6b0108300 0x7fa6b01086e0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:49:39.599 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:39.600+0000 7fa6b7421700 1 --2- 172.21.15.29:0/3019277211 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa6b0102320 0x7fa6b01027a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:49:39.599 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:39.600+0000 7fa6b7421700 1 -- 172.21.15.29:0/3019277211 >> 172.21.15.29:0/3019277211 conn(0x7fa6b0079400 msgr2=0x7fa6b0079810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:49:39.600 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:39.600+0000 7fa6b7421700 1 -- 172.21.15.29:0/3019277211 shutdown_connections 2024-03-16T11:49:39.600 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:39.600+0000 7fa6b7421700 1 -- 172.21.15.29:0/3019277211 wait complete. 2024-03-16T11:49:39.601 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:39.600+0000 7fa6b7421700 1 Processor -- start 2024-03-16T11:49:39.601 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:39.600+0000 7fa6b7421700 1 -- start start 2024-03-16T11:49:39.601 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:39.601+0000 7fa6b7421700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa6b0102320 0x7fa6b00745d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:49:39.602 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:39.601+0000 7fa6b7421700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa6b0102ce0 0x7fa6b0074b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:49:39.602 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:39.601+0000 7fa6b7421700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa6b0108300 0x7fa6b0075050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:49:39.602 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:39.601+0000 7fa6b7421700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fa6b0112580 con 0x7fa6b0102ce0 2024-03-16T11:49:39.602 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:39.601+0000 7fa6b7421700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fa6b0112400 con 0x7fa6b0102320 2024-03-16T11:49:39.602 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:39.601+0000 7fa6b7421700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fa6b0112700 con 0x7fa6b0108300 2024-03-16T11:49:39.602 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:39.601+0000 7fa6b49bc700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa6b0102ce0 0x7fa6b0074b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:49:39.602 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:39.601+0000 7fa6b49bc700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa6b0102ce0 0x7fa6b0074b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57014/0 (socket says 172.21.15.29:57014) 2024-03-16T11:49:39.602 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:39.601+0000 7fa6b49bc700 1 -- 172.21.15.29:0/3702457586 learned_addr learned my addr 172.21.15.29:0/3702457586 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:49:39.602 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:39.601+0000 7fa6b51bd700 1 --2- 172.21.15.29:0/3702457586 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa6b0102320 0x7fa6b00745d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:49:39.602 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:39.601+0000 7fa6b59be700 1 --2- 172.21.15.29:0/3702457586 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa6b0108300 0x7fa6b0075050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:49:39.602 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:39.601+0000 7fa6b51bd700 1 -- 172.21.15.29:0/3702457586 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa6b0108300 msgr2=0x7fa6b0075050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:49:39.602 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:39.601+0000 7fa6b51bd700 1 --2- 172.21.15.29:0/3702457586 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa6b0108300 0x7fa6b0075050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:49:39.602 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:39.601+0000 7fa6b51bd700 1 -- 172.21.15.29:0/3702457586 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa6b0102ce0 msgr2=0x7fa6b0074b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:49:39.603 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:39.601+0000 7fa6b51bd700 1 --2- 172.21.15.29:0/3702457586 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa6b0102ce0 0x7fa6b0074b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:49:39.603 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:39.601+0000 7fa6b51bd700 1 -- 172.21.15.29:0/3702457586 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6b01a6ee0 con 0x7fa6b0102320 2024-03-16T11:49:39.603 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:39.601+0000 7fa6b59be700 1 --2- 172.21.15.29:0/3702457586 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa6b0108300 0x7fa6b0075050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:49:39.603 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:39.602+0000 7fa6b51bd700 1 --2- 172.21.15.29:0/3702457586 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa6b0102320 0x7fa6b00745d0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fa6a000d010 tx=0x7fa6a0007a20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:49:39.603 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:39.602+0000 7fa6ae7fc700 1 -- 172.21.15.29:0/3702457586 <== mon.2 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6a0003c00 con 0x7fa6b0102320 2024-03-16T11:49:39.603 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:39.602+0000 7fa6b7421700 1 -- 172.21.15.29:0/3702457586 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa6b01a7170 con 0x7fa6b0102320 2024-03-16T11:49:39.603 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:39.602+0000 7fa6b7421700 1 -- 172.21.15.29:0/3702457586 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa6b01a7620 con 0x7fa6b0102320 2024-03-16T11:49:39.603 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:39.602+0000 7fa6ae7fc700 1 -- 172.21.15.29:0/3702457586 <== mon.2 v2:172.21.15.152:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa6a0022070 con 0x7fa6b0102320 2024-03-16T11:49:39.603 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:39.602+0000 7fa6ae7fc700 1 -- 172.21.15.29:0/3702457586 <== mon.2 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6a001b6c0 con 0x7fa6b0102320 2024-03-16T11:49:39.603 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:39.603+0000 7fa6b7421700 1 -- 172.21.15.29:0/3702457586 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa6b0068f90 con 0x7fa6b0102320 2024-03-16T11:49:39.604 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:39.604+0000 7fa6ae7fc700 1 -- 172.21.15.29:0/3702457586 <== mon.2 v2:172.21.15.152:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7fa6a001d470 con 0x7fa6b0102320 2024-03-16T11:49:39.604 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:39.605+0000 7fa6ae7fc700 1 --2- 172.21.15.29:0/3702457586 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fa6980a29a0 0x7fa6980a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:49:39.604 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:39.605+0000 7fa6b49bc700 1 --2- 172.21.15.29:0/3702457586 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fa6980a29a0 0x7fa6980a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:49:39.604 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:39.605+0000 7fa6ae7fc700 1 -- 172.21.15.29:0/3702457586 <== mon.2 v2:172.21.15.152:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8211+0+0 (secure 0 0 0) 0x7fa6a000f020 con 0x7fa6b0102320 2024-03-16T11:49:39.604 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:39.605+0000 7fa6b49bc700 1 --2- 172.21.15.29:0/3702457586 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fa6980a29a0 0x7fa6980a4e60 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fa6b0079990 tx=0x7fa6a4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:49:39.609 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:39.610+0000 7fa6ae7fc700 1 -- 172.21.15.29:0/3702457586 <== mon.2 v2:172.21.15.152:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fa6a009c7d0 con 0x7fa6b0102320 2024-03-16T11:49:39.817 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:39.818+0000 7fa6b7421700 1 -- 172.21.15.29:0/3702457586 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7fa6b002db70 con 0x7fa6b0102320 2024-03-16T11:49:39.818 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:39.819+0000 7fa6ae7fc700 1 -- 172.21.15.29:0/3702457586 <== mon.2 v2:172.21.15.152:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fa6a009c0d0 con 0x7fa6b0102320 2024-03-16T11:49:39.818 INFO:teuthology.orchestra.run.smithi029.stdout:141733920831 2024-03-16T11:49:39.820 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:39.821+0000 7fa6b7421700 1 -- 172.21.15.29:0/3702457586 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fa6980a29a0 msgr2=0x7fa6980a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:49:39.821 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:39.821+0000 7fa6b7421700 1 --2- 172.21.15.29:0/3702457586 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fa6980a29a0 0x7fa6980a4e60 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fa6b0079990 tx=0x7fa6a4009260 comp rx=0 tx=0).stop 2024-03-16T11:49:39.821 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:39.821+0000 7fa6b7421700 1 -- 172.21.15.29:0/3702457586 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa6b0102320 msgr2=0x7fa6b00745d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:49:39.821 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:39.821+0000 7fa6b7421700 1 --2- 172.21.15.29:0/3702457586 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa6b0102320 0x7fa6b00745d0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fa6a000d010 tx=0x7fa6a0007a20 comp rx=0 tx=0).stop 2024-03-16T11:49:39.821 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:39.821+0000 7fa6b7421700 1 -- 172.21.15.29:0/3702457586 shutdown_connections 2024-03-16T11:49:39.821 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:39.821+0000 7fa6b7421700 1 --2- 172.21.15.29:0/3702457586 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fa6980a29a0 0x7fa6980a4e60 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:49:39.821 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:39.821+0000 7fa6b7421700 1 --2- 172.21.15.29:0/3702457586 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa6b0108300 0x7fa6b0075050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:49:39.821 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:39.821+0000 7fa6b7421700 1 --2- 172.21.15.29:0/3702457586 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa6b0102320 0x7fa6b00745d0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:49:39.821 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:39.821+0000 7fa6b7421700 1 --2- 172.21.15.29:0/3702457586 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa6b0102ce0 0x7fa6b0074b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:49:39.821 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:39.821+0000 7fa6b7421700 1 -- 172.21.15.29:0/3702457586 >> 172.21.15.29:0/3702457586 conn(0x7fa6b0079400 msgr2=0x7fa6b0100ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:49:39.821 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:39.821+0000 7fa6b7421700 1 -- 172.21.15.29:0/3702457586 shutdown_connections 2024-03-16T11:49:39.821 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:39.821+0000 7fa6b7421700 1 -- 172.21.15.29:0/3702457586 wait complete. 2024-03-16T11:49:39.854 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:39.825+0000 7fb26179e700 1 -- 172.21.15.29:0/929694072 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb25c102db0 msgr2=0x7fb25c106e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:49:39.854 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:39.825+0000 7fb26179e700 1 --2- 172.21.15.29:0/929694072 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb25c102db0 0x7fb25c106e00 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fb2500099c0 tx=0x7fb250009cd0 comp rx=0 tx=0).stop 2024-03-16T11:49:39.854 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:39.825+0000 7fb26179e700 1 -- 172.21.15.29:0/929694072 shutdown_connections 2024-03-16T11:49:39.854 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:39.825+0000 7fb26179e700 1 --2- 172.21.15.29:0/929694072 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb25c1074e0 0x7fb25c10f0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:49:39.854 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:39.825+0000 7fb26179e700 1 --2- 172.21.15.29:0/929694072 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb25c102400 0x7fb25c1027e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:49:39.854 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:39.825+0000 7fb26179e700 1 --2- 172.21.15.29:0/929694072 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb25c102db0 0x7fb25c106e00 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:49:39.854 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:39.825+0000 7fb26179e700 1 -- 172.21.15.29:0/929694072 >> 172.21.15.29:0/929694072 conn(0x7fb25c0fdcf0 msgr2=0x7fb25c100110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:49:39.854 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:39.825+0000 7fb26179e700 1 -- 172.21.15.29:0/929694072 shutdown_connections 2024-03-16T11:49:39.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:39.825+0000 7fb26179e700 1 -- 172.21.15.29:0/929694072 wait complete. 2024-03-16T11:49:39.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:39.826+0000 7fb26179e700 1 Processor -- start 2024-03-16T11:49:39.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:39.826+0000 7fb26179e700 1 -- start start 2024-03-16T11:49:39.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:39.826+0000 7fb26179e700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb25c102400 0x7fb25c0745d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:49:39.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:39.826+0000 7fb26179e700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb25c102db0 0x7fb25c074b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:49:39.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:39.826+0000 7fb26179e700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb25c1074e0 0x7fb25c075050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:49:39.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:39.826+0000 7fb26179e700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fb25c1124a0 con 0x7fb25c1074e0 2024-03-16T11:49:39.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:39.826+0000 7fb26179e700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fb25c112320 con 0x7fb25c102400 2024-03-16T11:49:39.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:39.826+0000 7fb26179e700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fb25c112620 con 0x7fb25c102db0 2024-03-16T11:49:39.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:39.827+0000 7fb260f9d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb25c1074e0 0x7fb25c075050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:49:39.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:39.827+0000 7fb260f9d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb25c1074e0 0x7fb25c075050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57036/0 (socket says 172.21.15.29:57036) 2024-03-16T11:49:39.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:39.827+0000 7fb25bfff700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb25c102400 0x7fb25c0745d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:49:39.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:39.827+0000 7fb260f9d700 1 -- 172.21.15.29:0/3399477805 learned_addr learned my addr 172.21.15.29:0/3399477805 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:49:39.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:39.827+0000 7fb25b7fe700 1 --2- 172.21.15.29:0/3399477805 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb25c102db0 0x7fb25c074b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:49:39.856 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:39.827+0000 7fb260f9d700 1 -- 172.21.15.29:0/3399477805 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb25c102db0 msgr2=0x7fb25c074b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:49:39.856 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:39.827+0000 7fb260f9d700 1 --2- 172.21.15.29:0/3399477805 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb25c102db0 0x7fb25c074b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:49:39.856 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:39.827+0000 7fb260f9d700 1 -- 172.21.15.29:0/3399477805 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb25c102400 msgr2=0x7fb25c0745d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:49:39.856 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:39.827+0000 7fb260f9d700 1 --2- 172.21.15.29:0/3399477805 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb25c102400 0x7fb25c0745d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:49:39.856 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:39.827+0000 7fb260f9d700 1 -- 172.21.15.29:0/3399477805 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb25c1a6e70 con 0x7fb25c1074e0 2024-03-16T11:49:39.856 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:39.827+0000 7fb25bfff700 1 --2- 172.21.15.29:0/3399477805 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb25c102400 0x7fb25c0745d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:49:39.856 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:39.827+0000 7fb260f9d700 1 --2- 172.21.15.29:0/3399477805 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb25c1074e0 0x7fb25c075050 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fb25400ef30 tx=0x7fb25400c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:49:39.856 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:39.827+0000 7fb2597fa700 1 -- 172.21.15.29:0/3399477805 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2540092e0 con 0x7fb25c1074e0 2024-03-16T11:49:39.856 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:39.827+0000 7fb26179e700 1 -- 172.21.15.29:0/3399477805 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb25c1a7100 con 0x7fb25c1074e0 2024-03-16T11:49:39.856 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:39.827+0000 7fb2597fa700 1 -- 172.21.15.29:0/3399477805 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb254004bb0 con 0x7fb25c1074e0 2024-03-16T11:49:39.856 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:39.827+0000 7fb2597fa700 1 -- 172.21.15.29:0/3399477805 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb254004280 con 0x7fb25c1074e0 2024-03-16T11:49:39.856 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:39.827+0000 7fb26179e700 1 -- 172.21.15.29:0/3399477805 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb25c1a75c0 con 0x7fb25c1074e0 2024-03-16T11:49:39.856 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:39.828+0000 7fb26179e700 1 -- 172.21.15.29:0/3399477805 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb25c1038e0 con 0x7fb25c1074e0 2024-03-16T11:49:39.856 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:39.830+0000 7fb2597fa700 1 -- 172.21.15.29:0/3399477805 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7fb25401b030 con 0x7fb25c1074e0 2024-03-16T11:49:39.857 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:39.831+0000 7fb2597fa700 1 --2- 172.21.15.29:0/3399477805 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fb2440a2920 0x7fb2440a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:49:39.857 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:39.831+0000 7fb25bfff700 1 --2- 172.21.15.29:0/3399477805 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fb2440a2920 0x7fb2440a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:49:39.857 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:39.831+0000 7fb2597fa700 1 -- 172.21.15.29:0/3399477805 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8211+0+0 (secure 0 0 0) 0x7fb2540ceff0 con 0x7fb25c1074e0 2024-03-16T11:49:39.857 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:39.831+0000 7fb25bfff700 1 --2- 172.21.15.29:0/3399477805 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fb2440a2920 0x7fb2440a4de0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fb24c009680 tx=0x7fb24c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:49:39.857 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:39.835+0000 7fb2597fa700 1 -- 172.21.15.29:0/3399477805 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fb254022020 con 0x7fb25c1074e0 2024-03-16T11:49:39.857 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:49:39 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T11:49:39.946 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:49:39 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T11:49:40.044 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:40.044+0000 7fb26179e700 1 -- 172.21.15.29:0/3399477805 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7fb25c068f90 con 0x7fb25c1074e0 2024-03-16T11:49:40.044 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:40.045+0000 7fb2597fa700 1 -- 172.21.15.29:0/3399477805 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fb254010040 con 0x7fb25c1074e0 2024-03-16T11:49:40.044 INFO:teuthology.orchestra.run.smithi029.stdout:55834574926 2024-03-16T11:49:40.047 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:40.048+0000 7fb26179e700 1 -- 172.21.15.29:0/3399477805 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fb2440a2920 msgr2=0x7fb2440a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:49:40.048 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:40.048+0000 7fb26179e700 1 --2- 172.21.15.29:0/3399477805 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fb2440a2920 0x7fb2440a4de0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fb24c009680 tx=0x7fb24c009260 comp rx=0 tx=0).stop 2024-03-16T11:49:40.048 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:40.048+0000 7fb26179e700 1 -- 172.21.15.29:0/3399477805 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb25c1074e0 msgr2=0x7fb25c075050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:49:40.048 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:40.048+0000 7fb26179e700 1 --2- 172.21.15.29:0/3399477805 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb25c1074e0 0x7fb25c075050 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fb25400ef30 tx=0x7fb25400c3a0 comp rx=0 tx=0).stop 2024-03-16T11:49:40.048 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:40.048+0000 7fb26179e700 1 -- 172.21.15.29:0/3399477805 shutdown_connections 2024-03-16T11:49:40.048 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:40.048+0000 7fb26179e700 1 --2- 172.21.15.29:0/3399477805 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb25c1074e0 0x7fb25c075050 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:49:40.048 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:40.048+0000 7fb26179e700 1 --2- 172.21.15.29:0/3399477805 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb25c102400 0x7fb25c0745d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:49:40.048 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:40.048+0000 7fb26179e700 1 --2- 172.21.15.29:0/3399477805 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fb2440a2920 0x7fb2440a4de0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:49:40.048 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:40.048+0000 7fb26179e700 1 --2- 172.21.15.29:0/3399477805 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb25c102db0 0x7fb25c074b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:49:40.048 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:40.048+0000 7fb26179e700 1 -- 172.21.15.29:0/3399477805 >> 172.21.15.29:0/3399477805 conn(0x7fb25c0fdcf0 msgr2=0x7fb25c100060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:49:40.048 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:40.048+0000 7fb26179e700 1 -- 172.21.15.29:0/3399477805 shutdown_connections 2024-03-16T11:49:40.048 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:40.048+0000 7fb26179e700 1 -- 172.21.15.29:0/3399477805 wait complete. 2024-03-16T11:49:40.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:49:39 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T11:49:40.837 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:49:40 smithi029 ceph-mon[31975]: pgmap v323: 97 pgs: 97 active+clean; 640 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s; 5.4 KiB/s, 0 keys/s, 0 objects/s recovering 2024-03-16T11:49:40.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:49:40 smithi029 ceph-mon[31975]: from='client.? 172.21.15.29:0/3702457586' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-16T11:49:40.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:49:40 smithi029 ceph-mon[31975]: from='client.? 172.21.15.29:0/3399477805' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-16T11:49:40.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:49:40 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:49:40.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:49:40 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:49:41.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:49:40 smithi154 ceph-mon[36320]: pgmap v323: 97 pgs: 97 active+clean; 640 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s; 5.4 KiB/s, 0 keys/s, 0 objects/s recovering 2024-03-16T11:49:41.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:49:40 smithi154 ceph-mon[36320]: from='client.? 172.21.15.29:0/3702457586' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-16T11:49:41.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:49:40 smithi154 ceph-mon[36320]: from='client.? 172.21.15.29:0/3399477805' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-16T11:49:41.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:49:40 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:49:41.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:49:40 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:49:41.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:49:40 smithi152 ceph-mon[36291]: pgmap v323: 97 pgs: 97 active+clean; 640 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s; 5.4 KiB/s, 0 keys/s, 0 objects/s recovering 2024-03-16T11:49:41.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:49:40 smithi152 ceph-mon[36291]: from='client.? 172.21.15.29:0/3702457586' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-16T11:49:41.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:49:40 smithi152 ceph-mon[36291]: from='client.? 172.21.15.29:0/3399477805' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-16T11:49:41.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:49:40 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:49:41.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:49:40 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:49:41.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:49:41 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:49:41.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:49:41 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:49:41.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:49:41 smithi029 ceph-mon[31975]: pgmap v324: 97 pgs: 97 active+clean; 640 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s; 84 B/s, 0 keys/s, 0 objects/s recovering 2024-03-16T11:49:42.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:49:41 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:49:42.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:49:41 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:49:42.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:49:41 smithi154 ceph-mon[36320]: pgmap v324: 97 pgs: 97 active+clean; 640 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s; 84 B/s, 0 keys/s, 0 objects/s recovering 2024-03-16T11:49:42.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:49:41 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:49:42.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:49:41 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:49:42.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:49:41 smithi152 ceph-mon[36291]: pgmap v324: 97 pgs: 97 active+clean; 640 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s; 84 B/s, 0 keys/s, 0 objects/s recovering 2024-03-16T11:49:43.511 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:49:43 smithi029 ceph-mon[31975]: from='client.? 172.21.15.29:0/2869771738' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-03-16T11:49:43.545 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:49:43 smithi154 ceph-mon[36320]: from='client.? 172.21.15.29:0/2869771738' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-03-16T11:49:43.627 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:49:43 smithi152 ceph-mon[36291]: from='client.? 172.21.15.29:0/2869771738' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-03-16T11:49:44.493 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:49:44 smithi029 ceph-mon[31975]: pgmap v325: 97 pgs: 97 active+clean; 640 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-16T11:49:44.544 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:49:44 smithi154 ceph-mon[36320]: pgmap v325: 97 pgs: 97 active+clean; 640 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-16T11:49:44.627 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:49:44 smithi152 ceph-mon[36291]: pgmap v325: 97 pgs: 97 active+clean; 640 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-16T11:49:44.736 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:44.722+0000 7ff1f2224700 1 -- 172.21.15.29:0/4112625403 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff1ec102d10 msgr2=0x7ff1ec10f170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:49:44.736 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:44.722+0000 7ff1f2224700 1 --2- 172.21.15.29:0/4112625403 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff1ec102d10 0x7ff1ec10f170 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7ff1e400b0a0 tx=0x7ff1e400b3b0 comp rx=0 tx=0).stop 2024-03-16T11:49:44.736 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:44.722+0000 7ff1f2224700 1 -- 172.21.15.29:0/4112625403 shutdown_connections 2024-03-16T11:49:44.736 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:44.722+0000 7ff1f2224700 1 --2- 172.21.15.29:0/4112625403 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff1ec102d10 0x7ff1ec10f170 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:49:44.736 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:44.722+0000 7ff1f2224700 1 --2- 172.21.15.29:0/4112625403 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff1ec1083a0 0x7ff1ec108780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:49:44.736 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:44.722+0000 7ff1f2224700 1 --2- 172.21.15.29:0/4112625403 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff1ec102350 0x7ff1ec1027d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:49:44.736 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:44.723+0000 7ff1f2224700 1 -- 172.21.15.29:0/4112625403 >> 172.21.15.29:0/4112625403 conn(0x7ff1ec079390 msgr2=0x7ff1ec0797a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:49:44.736 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:44.723+0000 7ff1f2224700 1 -- 172.21.15.29:0/4112625403 shutdown_connections 2024-03-16T11:49:44.737 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:44.723+0000 7ff1f2224700 1 -- 172.21.15.29:0/4112625403 wait complete. 2024-03-16T11:49:44.737 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:44.723+0000 7ff1f2224700 1 Processor -- start 2024-03-16T11:49:44.737 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:44.724+0000 7ff1f2224700 1 -- start start 2024-03-16T11:49:44.737 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:44.724+0000 7ff1f2224700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff1ec102350 0x7ff1ec19e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:49:44.737 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:44.724+0000 7ff1f2224700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff1ec102d10 0x7ff1ec19ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:49:44.737 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:44.724+0000 7ff1f2224700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff1ec1083a0 0x7ff1ec198820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:49:44.737 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:44.724+0000 7ff1f2224700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7ff1ec1125a0 con 0x7ff1ec102d10 2024-03-16T11:49:44.737 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:44.724+0000 7ff1f2224700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7ff1ec112420 con 0x7ff1ec102350 2024-03-16T11:49:44.737 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:44.724+0000 7ff1f2224700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7ff1ec112720 con 0x7ff1ec1083a0 2024-03-16T11:49:44.737 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:44.724+0000 7ff1eaffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff1ec102d10 0x7ff1ec19ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:49:44.737 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:44.724+0000 7ff1eaffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff1ec102d10 0x7ff1ec19ebe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57104/0 (socket says 172.21.15.29:57104) 2024-03-16T11:49:44.737 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:44.724+0000 7ff1eaffd700 1 -- 172.21.15.29:0/3460902300 learned_addr learned my addr 172.21.15.29:0/3460902300 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:49:44.737 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:44.724+0000 7ff1eb7fe700 1 --2- 172.21.15.29:0/3460902300 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff1ec102350 0x7ff1ec19e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:49:44.738 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:44.724+0000 7ff1ebfff700 1 --2- 172.21.15.29:0/3460902300 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff1ec1083a0 0x7ff1ec198820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:49:44.738 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:44.724+0000 7ff1eaffd700 1 -- 172.21.15.29:0/3460902300 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff1ec1083a0 msgr2=0x7ff1ec198820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:49:44.738 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:44.724+0000 7ff1eaffd700 1 --2- 172.21.15.29:0/3460902300 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff1ec1083a0 0x7ff1ec198820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:49:44.738 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:44.724+0000 7ff1eaffd700 1 -- 172.21.15.29:0/3460902300 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff1ec102350 msgr2=0x7ff1ec19e6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:49:44.738 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:44.724+0000 7ff1eaffd700 1 --2- 172.21.15.29:0/3460902300 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff1ec102350 0x7ff1ec19e6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:49:44.738 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:44.724+0000 7ff1eaffd700 1 -- 172.21.15.29:0/3460902300 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1ec199000 con 0x7ff1ec102d10 2024-03-16T11:49:44.738 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:44.724+0000 7ff1eb7fe700 1 --2- 172.21.15.29:0/3460902300 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff1ec102350 0x7ff1ec19e6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:49:44.738 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:44.724+0000 7ff1eaffd700 1 --2- 172.21.15.29:0/3460902300 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff1ec102d10 0x7ff1ec19ebe0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7ff1e000e9c0 tx=0x7ff1e000ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:49:44.738 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:44.724+0000 7ff1e8ff9700 1 -- 172.21.15.29:0/3460902300 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1e000cbd0 con 0x7ff1ec102d10 2024-03-16T11:49:44.738 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:44.725+0000 7ff1f2224700 1 -- 172.21.15.29:0/3460902300 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff1ec1992f0 con 0x7ff1ec102d10 2024-03-16T11:49:44.738 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:44.725+0000 7ff1e8ff9700 1 -- 172.21.15.29:0/3460902300 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff1e000ee60 con 0x7ff1ec102d10 2024-03-16T11:49:44.738 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:44.725+0000 7ff1e8ff9700 1 -- 172.21.15.29:0/3460902300 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1e0005630 con 0x7ff1ec102d10 2024-03-16T11:49:44.738 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:44.725+0000 7ff1f2224700 1 -- 172.21.15.29:0/3460902300 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff1ec1a5500 con 0x7ff1ec102d10 2024-03-16T11:49:44.738 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:44.726+0000 7ff1f2224700 1 -- 172.21.15.29:0/3460902300 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff1ec103a10 con 0x7ff1ec102d10 2024-03-16T11:49:44.739 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:44.727+0000 7ff1e8ff9700 1 -- 172.21.15.29:0/3460902300 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7ff1e0004750 con 0x7ff1ec102d10 2024-03-16T11:49:44.739 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:44.727+0000 7ff1e8ff9700 1 --2- 172.21.15.29:0/3460902300 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7ff1d40a2920 0x7ff1d40a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:49:44.739 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:44.728+0000 7ff1eb7fe700 1 --2- 172.21.15.29:0/3460902300 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7ff1d40a2920 0x7ff1d40a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:49:44.739 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:44.728+0000 7ff1e8ff9700 1 -- 172.21.15.29:0/3460902300 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8211+0+0 (secure 0 0 0) 0x7ff1e00cf0d0 con 0x7ff1ec102d10 2024-03-16T11:49:44.739 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:44.728+0000 7ff1eb7fe700 1 --2- 172.21.15.29:0/3460902300 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7ff1d40a2920 0x7ff1d40a4de0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7ff1dc009fd0 tx=0x7ff1dc009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:49:44.739 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:44.731+0000 7ff1e8ff9700 1 -- 172.21.15.29:0/3460902300 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7ff1e0053740 con 0x7ff1ec102d10 2024-03-16T11:49:44.845 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:43.008+0000 7f42fcf41700 1 -- 172.21.15.29:0/2357285124 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f42f8108300 msgr2=0x7f42f81086e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:49:44.845 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:43.008+0000 7f42fcf41700 1 --2- 172.21.15.29:0/2357285124 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f42f8108300 0x7f42f81086e0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f42e80099c0 tx=0x7f42e8009cd0 comp rx=0 tx=0).stop 2024-03-16T11:49:44.845 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:43.008+0000 7f42fcf41700 1 -- 172.21.15.29:0/2357285124 shutdown_connections 2024-03-16T11:49:44.845 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:43.008+0000 7f42fcf41700 1 --2- 172.21.15.29:0/2357285124 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f42f8102ce0 0x7f42f810f120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:49:44.845 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:43.008+0000 7f42fcf41700 1 --2- 172.21.15.29:0/2357285124 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f42f8108300 0x7f42f81086e0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:49:44.845 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:43.008+0000 7f42fcf41700 1 --2- 172.21.15.29:0/2357285124 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f42f8102320 0x7f42f81027a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:49:44.845 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:43.008+0000 7f42fcf41700 1 -- 172.21.15.29:0/2357285124 >> 172.21.15.29:0/2357285124 conn(0x7f42f8079400 msgr2=0x7f42f8079810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:49:44.846 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:43.008+0000 7f42fcf41700 1 -- 172.21.15.29:0/2357285124 shutdown_connections 2024-03-16T11:49:44.846 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:43.008+0000 7f42fcf41700 1 -- 172.21.15.29:0/2357285124 wait complete. 2024-03-16T11:49:44.846 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:43.009+0000 7f42fcf41700 1 Processor -- start 2024-03-16T11:49:44.846 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:43.009+0000 7f42fcf41700 1 -- start start 2024-03-16T11:49:44.846 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:43.009+0000 7f42fcf41700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f42f8102320 0x7f42f819e640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:49:44.846 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:43.009+0000 7f42fcf41700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f42f8102ce0 0x7f42f819eb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:49:44.846 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:43.009+0000 7f42fcf41700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f42f8108300 0x7f42f81987c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:49:44.846 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:43.009+0000 7f42fcf41700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f42f8112580 con 0x7f42f8102320 2024-03-16T11:49:44.846 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:43.009+0000 7f42fcf41700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f42f8112400 con 0x7f42f8108300 2024-03-16T11:49:44.846 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:43.009+0000 7f42fcf41700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f42f8112700 con 0x7f42f8102ce0 2024-03-16T11:49:44.846 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:43.009+0000 7f42f659c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f42f8102320 0x7f42f819e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:49:44.846 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:43.009+0000 7f42f659c700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f42f8102320 0x7f42f819e640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57058/0 (socket says 172.21.15.29:57058) 2024-03-16T11:49:44.846 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:43.009+0000 7f42f659c700 1 -- 172.21.15.29:0/2869771738 learned_addr learned my addr 172.21.15.29:0/2869771738 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:49:44.846 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:43.009+0000 7f42f6d9d700 1 --2- 172.21.15.29:0/2869771738 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f42f8108300 0x7f42f81987c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:49:44.847 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:43.009+0000 7f42effff700 1 --2- 172.21.15.29:0/2869771738 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f42f8102ce0 0x7f42f819eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:49:44.847 INFO:teuthology.orchestra.run.smithi029.stdout:309237645343 2024-03-16T11:49:44.847 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:43.009+0000 7f42f659c700 1 -- 172.21.15.29:0/2869771738 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f42f8102ce0 msgr2=0x7f42f819eb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:49:44.847 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:43.009+0000 7f42f659c700 1 --2- 172.21.15.29:0/2869771738 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f42f8102ce0 0x7f42f819eb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:49:44.847 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:43.009+0000 7f42f659c700 1 -- 172.21.15.29:0/2869771738 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f42f8108300 msgr2=0x7f42f81987c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:49:44.847 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:43.009+0000 7f42f659c700 1 --2- 172.21.15.29:0/2869771738 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f42f8108300 0x7f42f81987c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:49:44.847 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:43.009+0000 7f42f659c700 1 -- 172.21.15.29:0/2869771738 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42f8198fa0 con 0x7f42f8102320 2024-03-16T11:49:44.847 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:43.010+0000 7f42f659c700 1 --2- 172.21.15.29:0/2869771738 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f42f8102320 0x7f42f819e640 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f42e8006010 tx=0x7f42e800f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:49:44.847 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:43.010+0000 7f42ef7fe700 1 -- 172.21.15.29:0/2869771738 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42e8003680 con 0x7f42f8102320 2024-03-16T11:49:44.848 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:43.010+0000 7f42ef7fe700 1 -- 172.21.15.29:0/2869771738 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f42e80050d0 con 0x7f42f8102320 2024-03-16T11:49:44.848 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:43.010+0000 7f42fcf41700 1 -- 172.21.15.29:0/2869771738 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f42f8199230 con 0x7f42f8102320 2024-03-16T11:49:44.848 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:43.010+0000 7f42ef7fe700 1 -- 172.21.15.29:0/2869771738 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42e8003db0 con 0x7f42f8102320 2024-03-16T11:49:44.848 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:43.010+0000 7f42fcf41700 1 -- 172.21.15.29:0/2869771738 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f42f81a54a0 con 0x7f42f8102320 2024-03-16T11:49:44.848 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:43.011+0000 7f42fcf41700 1 -- 172.21.15.29:0/2869771738 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f42f81039b0 con 0x7f42f8102320 2024-03-16T11:49:44.848 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:43.012+0000 7f42ef7fe700 1 -- 172.21.15.29:0/2869771738 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f42e8005280 con 0x7f42f8102320 2024-03-16T11:49:44.848 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:43.012+0000 7f42ef7fe700 1 --2- 172.21.15.29:0/2869771738 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f42e40a2ac0 0x7f42e40a4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:49:44.848 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:43.012+0000 7f42effff700 1 --2- 172.21.15.29:0/2869771738 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f42e40a2ac0 0x7f42e40a4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:49:44.848 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:43.012+0000 7f42ef7fe700 1 -- 172.21.15.29:0/2869771738 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8211+0+0 (secure 0 0 0) 0x7f42e80cf8b0 con 0x7f42f8102320 2024-03-16T11:49:44.848 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:43.013+0000 7f42effff700 1 --2- 172.21.15.29:0/2869771738 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f42e40a2ac0 0x7f42e40a4f80 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f42f8079990 tx=0x7f42e0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:49:44.848 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:43.017+0000 7f42ef7fe700 1 -- 172.21.15.29:0/2869771738 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f42e8013070 con 0x7f42f8102320 2024-03-16T11:49:44.848 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:43.222+0000 7f42fcf41700 1 -- 172.21.15.29:0/2869771738 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 10} v 0) -- 0x7f42f8199cf0 con 0x7f42f8102320 2024-03-16T11:49:44.848 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:43.222+0000 7f42ef7fe700 1 -- 172.21.15.29:0/2869771738 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 10}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7f42e809c890 con 0x7f42f8102320 2024-03-16T11:49:44.848 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:43.224+0000 7f42fcf41700 1 -- 172.21.15.29:0/2869771738 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f42e40a2ac0 msgr2=0x7f42e40a4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:49:44.848 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:43.224+0000 7f42fcf41700 1 --2- 172.21.15.29:0/2869771738 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f42e40a2ac0 0x7f42e40a4f80 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f42f8079990 tx=0x7f42e0009260 comp rx=0 tx=0).stop 2024-03-16T11:49:44.849 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:43.224+0000 7f42fcf41700 1 -- 172.21.15.29:0/2869771738 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f42f8102320 msgr2=0x7f42f819e640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:49:44.849 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:43.224+0000 7f42fcf41700 1 --2- 172.21.15.29:0/2869771738 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f42f8102320 0x7f42f819e640 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f42e8006010 tx=0x7f42e800f690 comp rx=0 tx=0).stop 2024-03-16T11:49:44.849 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:43.224+0000 7f42fcf41700 1 -- 172.21.15.29:0/2869771738 shutdown_connections 2024-03-16T11:49:44.849 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:43.224+0000 7f42fcf41700 1 --2- 172.21.15.29:0/2869771738 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f42f8108300 0x7f42f81987c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:49:44.849 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:43.224+0000 7f42fcf41700 1 --2- 172.21.15.29:0/2869771738 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f42f8102320 0x7f42f819e640 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:49:44.849 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:43.224+0000 7f42fcf41700 1 --2- 172.21.15.29:0/2869771738 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f42e40a2ac0 0x7f42e40a4f80 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:49:44.849 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:43.224+0000 7f42fcf41700 1 --2- 172.21.15.29:0/2869771738 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f42f8102ce0 0x7f42f819eb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:49:44.849 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:43.224+0000 7f42fcf41700 1 -- 172.21.15.29:0/2869771738 >> 172.21.15.29:0/2869771738 conn(0x7f42f8079400 msgr2=0x7f42f8100ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:49:44.849 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:43.224+0000 7f42fcf41700 1 -- 172.21.15.29:0/2869771738 shutdown_connections 2024-03-16T11:49:44.849 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:43.224+0000 7f42fcf41700 1 -- 172.21.15.29:0/2869771738 wait complete. 2024-03-16T11:49:44.941 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:44.942+0000 7ff1f2224700 1 -- 172.21.15.29:0/3460902300 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7ff1ec02db70 con 0x7ff1ec102d10 2024-03-16T11:49:44.941 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:44.942+0000 7ff1e8ff9700 1 -- 172.21.15.29:0/3460902300 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7ff1e009bfc0 con 0x7ff1ec102d10 2024-03-16T11:49:44.942 INFO:teuthology.orchestra.run.smithi029.stdout:107374182468 2024-03-16T11:49:44.944 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:44.944+0000 7ff1f2224700 1 -- 172.21.15.29:0/3460902300 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7ff1d40a2920 msgr2=0x7ff1d40a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:49:44.944 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:44.944+0000 7ff1f2224700 1 --2- 172.21.15.29:0/3460902300 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7ff1d40a2920 0x7ff1d40a4de0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7ff1dc009fd0 tx=0x7ff1dc009260 comp rx=0 tx=0).stop 2024-03-16T11:49:44.944 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:44.944+0000 7ff1f2224700 1 -- 172.21.15.29:0/3460902300 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff1ec102d10 msgr2=0x7ff1ec19ebe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:49:44.944 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:44.944+0000 7ff1f2224700 1 --2- 172.21.15.29:0/3460902300 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff1ec102d10 0x7ff1ec19ebe0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7ff1e000e9c0 tx=0x7ff1e000ecd0 comp rx=0 tx=0).stop 2024-03-16T11:49:44.944 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:44.944+0000 7ff1f2224700 1 -- 172.21.15.29:0/3460902300 shutdown_connections 2024-03-16T11:49:44.944 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:44.944+0000 7ff1f2224700 1 --2- 172.21.15.29:0/3460902300 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7ff1d40a2920 0x7ff1d40a4de0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:49:44.944 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:44.944+0000 7ff1f2224700 1 --2- 172.21.15.29:0/3460902300 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff1ec1083a0 0x7ff1ec198820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:49:44.944 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:44.944+0000 7ff1f2224700 1 --2- 172.21.15.29:0/3460902300 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff1ec102350 0x7ff1ec19e6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:49:44.944 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:44.944+0000 7ff1f2224700 1 --2- 172.21.15.29:0/3460902300 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff1ec102d10 0x7ff1ec19ebe0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:49:44.944 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:44.944+0000 7ff1f2224700 1 -- 172.21.15.29:0/3460902300 >> 172.21.15.29:0/3460902300 conn(0x7ff1ec079390 msgr2=0x7ff1ec101020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:49:44.944 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:44.945+0000 7ff1f2224700 1 -- 172.21.15.29:0/3460902300 shutdown_connections 2024-03-16T11:49:44.944 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:44.945+0000 7ff1f2224700 1 -- 172.21.15.29:0/3460902300 wait complete. 2024-03-16T11:49:45.537 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:49:45 smithi029 ceph-mon[31975]: from='client.? 172.21.15.29:0/1027021516' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-16T11:49:45.537 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:49:45 smithi029 ceph-mon[31975]: from='client.? 172.21.15.29:0/3460902300' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-16T11:49:45.545 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:49:45 smithi154 ceph-mon[36320]: from='client.? 172.21.15.29:0/1027021516' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-16T11:49:45.545 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:49:45 smithi154 ceph-mon[36320]: from='client.? 172.21.15.29:0/3460902300' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-16T11:49:45.627 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:49:45 smithi152 ceph-mon[36291]: from='client.? 172.21.15.29:0/1027021516' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-16T11:49:45.627 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:49:45 smithi152 ceph-mon[36291]: from='client.? 172.21.15.29:0/3460902300' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-16T11:49:46.371 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:45.642+0000 7f1166317700 1 -- 172.21.15.29:0/3249803740 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1160102ec0 msgr2=0x7f1160106f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:49:46.371 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:45.642+0000 7f1166317700 1 --2- 172.21.15.29:0/3249803740 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1160102ec0 0x7f1160106f10 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f11540099c0 tx=0x7f1154009cd0 comp rx=0 tx=0).stop 2024-03-16T11:49:46.371 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:45.642+0000 7f1166317700 1 -- 172.21.15.29:0/3249803740 shutdown_connections 2024-03-16T11:49:46.371 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:45.642+0000 7f1166317700 1 --2- 172.21.15.29:0/3249803740 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f11601075f0 0x7f116010f1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:49:46.371 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:45.642+0000 7f1166317700 1 --2- 172.21.15.29:0/3249803740 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1160102510 0x7f11601028f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:49:46.372 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:45.642+0000 7f1166317700 1 --2- 172.21.15.29:0/3249803740 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1160102ec0 0x7f1160106f10 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:49:46.372 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:45.642+0000 7f1166317700 1 -- 172.21.15.29:0/3249803740 >> 172.21.15.29:0/3249803740 conn(0x7f11600fdd80 msgr2=0x7f11601001a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:49:46.372 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:45.642+0000 7f1166317700 1 -- 172.21.15.29:0/3249803740 shutdown_connections 2024-03-16T11:49:46.372 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:45.642+0000 7f1166317700 1 -- 172.21.15.29:0/3249803740 wait complete. 2024-03-16T11:49:46.372 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:45.643+0000 7f1166317700 1 Processor -- start 2024-03-16T11:49:46.372 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:45.643+0000 7f1166317700 1 -- start start 2024-03-16T11:49:46.372 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:45.643+0000 7f1166317700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1160102510 0x7f116019e700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:49:46.372 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:45.643+0000 7f1166317700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1160102ec0 0x7f116019ec40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:49:46.372 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:45.643+0000 7f1166317700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f11601075f0 0x7f1160198910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:49:46.372 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:45.643+0000 7f1166317700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f11601125d0 con 0x7f11601075f0 2024-03-16T11:49:46.372 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:45.643+0000 7f1166317700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f1160112450 con 0x7f1160102510 2024-03-16T11:49:46.372 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:45.643+0000 7f1166317700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f1160112750 con 0x7f1160102ec0 2024-03-16T11:49:46.372 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:45.643+0000 7f11648b4700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f11601075f0 0x7f1160198910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:49:46.372 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:45.643+0000 7f115ffff700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1160102510 0x7f116019e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:49:46.373 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:45.643+0000 7f11648b4700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f11601075f0 0x7f1160198910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57114/0 (socket says 172.21.15.29:57114) 2024-03-16T11:49:46.373 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:45.643+0000 7f11648b4700 1 -- 172.21.15.29:0/2987985866 learned_addr learned my addr 172.21.15.29:0/2987985866 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:49:46.373 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:45.643+0000 7f115f7fe700 1 --2- 172.21.15.29:0/2987985866 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1160102ec0 0x7f116019ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:49:46.373 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:45.644+0000 7f11648b4700 1 -- 172.21.15.29:0/2987985866 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1160102ec0 msgr2=0x7f116019ec40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:49:46.373 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:45.644+0000 7f11648b4700 1 --2- 172.21.15.29:0/2987985866 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1160102ec0 0x7f116019ec40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:49:46.373 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:45.644+0000 7f11648b4700 1 -- 172.21.15.29:0/2987985866 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1160102510 msgr2=0x7f116019e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:49:46.373 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:45.644+0000 7f11648b4700 1 --2- 172.21.15.29:0/2987985866 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1160102510 0x7f116019e700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:49:46.373 INFO:teuthology.orchestra.run.smithi029.stdout:34359738451 2024-03-16T11:49:46.374 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:45.644+0000 7f11648b4700 1 -- 172.21.15.29:0/2987985866 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1160199070 con 0x7f11601075f0 2024-03-16T11:49:46.374 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:45.644+0000 7f11648b4700 1 --2- 172.21.15.29:0/2987985866 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f11601075f0 0x7f1160198910 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f115800ef30 tx=0x7f115800c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:49:46.374 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:45.644+0000 7f115d7fa700 1 -- 172.21.15.29:0/2987985866 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11580092e0 con 0x7f11601075f0 2024-03-16T11:49:46.374 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:45.644+0000 7f115d7fa700 1 -- 172.21.15.29:0/2987985866 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1158004bb0 con 0x7f11601075f0 2024-03-16T11:49:46.374 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:45.644+0000 7f1166317700 1 -- 172.21.15.29:0/2987985866 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1160199300 con 0x7f11601075f0 2024-03-16T11:49:46.374 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:45.644+0000 7f115d7fa700 1 -- 172.21.15.29:0/2987985866 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1158004280 con 0x7f11601075f0 2024-03-16T11:49:46.374 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:45.644+0000 7f1166317700 1 -- 172.21.15.29:0/2987985866 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11601a55d0 con 0x7f11601075f0 2024-03-16T11:49:46.374 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:45.645+0000 7f1166317700 1 -- 172.21.15.29:0/2987985866 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f11601039f0 con 0x7f11601075f0 2024-03-16T11:49:46.374 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:45.646+0000 7f115d7fa700 1 -- 172.21.15.29:0/2987985866 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f115801b030 con 0x7f11601075f0 2024-03-16T11:49:46.374 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:45.647+0000 7f115d7fa700 1 --2- 172.21.15.29:0/2987985866 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f11480a2920 0x7f11480a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:49:46.374 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:45.647+0000 7f115ffff700 1 --2- 172.21.15.29:0/2987985866 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f11480a2920 0x7f11480a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:49:46.374 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:45.647+0000 7f115d7fa700 1 -- 172.21.15.29:0/2987985866 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8211+0+0 (secure 0 0 0) 0x7f11580ce640 con 0x7f11601075f0 2024-03-16T11:49:46.374 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:45.647+0000 7f115ffff700 1 --2- 172.21.15.29:0/2987985866 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f11480a2920 0x7f11480a4de0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f11600fe220 tx=0x7f11500073d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:49:46.374 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:45.651+0000 7f115d7fa700 1 -- 172.21.15.29:0/2987985866 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f1158022020 con 0x7f11601075f0 2024-03-16T11:49:46.375 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:45.855+0000 7f1166317700 1 -- 172.21.15.29:0/2987985866 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7f116002db70 con 0x7f11601075f0 2024-03-16T11:49:46.375 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:45.855+0000 7f115d7fa700 1 -- 172.21.15.29:0/2987985866 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f1158010040 con 0x7f11601075f0 2024-03-16T11:49:46.375 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:45.857+0000 7f1166317700 1 -- 172.21.15.29:0/2987985866 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f11480a2920 msgr2=0x7f11480a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:49:46.375 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:45.857+0000 7f1166317700 1 --2- 172.21.15.29:0/2987985866 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f11480a2920 0x7f11480a4de0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f11600fe220 tx=0x7f11500073d0 comp rx=0 tx=0).stop 2024-03-16T11:49:46.375 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:45.857+0000 7f1166317700 1 -- 172.21.15.29:0/2987985866 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f11601075f0 msgr2=0x7f1160198910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:49:46.375 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:45.857+0000 7f1166317700 1 --2- 172.21.15.29:0/2987985866 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f11601075f0 0x7f1160198910 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f115800ef30 tx=0x7f115800c3a0 comp rx=0 tx=0).stop 2024-03-16T11:49:46.375 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:45.858+0000 7f1166317700 1 -- 172.21.15.29:0/2987985866 shutdown_connections 2024-03-16T11:49:46.375 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:45.858+0000 7f1166317700 1 --2- 172.21.15.29:0/2987985866 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f11601075f0 0x7f1160198910 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:49:46.375 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:45.858+0000 7f1166317700 1 --2- 172.21.15.29:0/2987985866 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1160102510 0x7f116019e700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:49:46.375 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:45.858+0000 7f1166317700 1 --2- 172.21.15.29:0/2987985866 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f11480a2920 0x7f11480a4de0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:49:46.375 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:45.858+0000 7f1166317700 1 --2- 172.21.15.29:0/2987985866 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1160102ec0 0x7f116019ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:49:46.375 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:45.858+0000 7f1166317700 1 -- 172.21.15.29:0/2987985866 >> 172.21.15.29:0/2987985866 conn(0x7f11600fdd80 msgr2=0x7f1160100050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:49:46.375 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:45.858+0000 7f1166317700 1 -- 172.21.15.29:0/2987985866 shutdown_connections 2024-03-16T11:49:46.375 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:45.858+0000 7f1166317700 1 -- 172.21.15.29:0/2987985866 wait complete. 2024-03-16T11:49:46.545 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:49:46 smithi154 ceph-mon[36320]: pgmap v326: 97 pgs: 97 active+clean; 640 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-16T11:49:46.545 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:49:46 smithi154 ceph-mon[36320]: from='client.? 172.21.15.29:0/2987985866' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-16T11:49:46.587 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:49:46 smithi029 ceph-mon[31975]: pgmap v326: 97 pgs: 97 active+clean; 640 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-16T11:49:46.588 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:49:46 smithi029 ceph-mon[31975]: from='client.? 172.21.15.29:0/2987985866' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-16T11:49:46.627 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:49:46 smithi152 ceph-mon[36291]: pgmap v326: 97 pgs: 97 active+clean; 640 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-16T11:49:46.627 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:49:46 smithi152 ceph-mon[36291]: from='client.? 172.21.15.29:0/2987985866' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-16T11:49:47.437 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:44.596+0000 7f6f77a81700 1 -- 172.21.15.29:0/3901549144 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6f701076b0 msgr2=0x7f6f7010f230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:49:47.438 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:44.596+0000 7f6f77a81700 1 --2- 172.21.15.29:0/3901549144 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6f701076b0 0x7f6f7010f230 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f6f68009990 tx=0x7f6f68009ca0 comp rx=0 tx=0).stop 2024-03-16T11:49:47.438 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:44.596+0000 7f6f77a81700 1 -- 172.21.15.29:0/3901549144 shutdown_connections 2024-03-16T11:49:47.438 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:44.596+0000 7f6f77a81700 1 --2- 172.21.15.29:0/3901549144 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6f701076b0 0x7f6f7010f230 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:49:47.438 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:44.596+0000 7f6f77a81700 1 --2- 172.21.15.29:0/3901549144 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6f7006abd0 0x7f6f7006afb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:49:47.438 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:44.596+0000 7f6f77a81700 1 --2- 172.21.15.29:0/3901549144 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6f7006b580 0x7f6f70106f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:49:47.438 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:44.596+0000 7f6f77a81700 1 -- 172.21.15.29:0/3901549144 >> 172.21.15.29:0/3901549144 conn(0x7f6f70078090 msgr2=0x7f6f700784a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:49:47.438 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:44.596+0000 7f6f77a81700 1 -- 172.21.15.29:0/3901549144 shutdown_connections 2024-03-16T11:49:47.438 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:44.596+0000 7f6f77a81700 1 -- 172.21.15.29:0/3901549144 wait complete. 2024-03-16T11:49:47.438 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:44.597+0000 7f6f77a81700 1 Processor -- start 2024-03-16T11:49:47.438 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:44.597+0000 7f6f77a81700 1 -- start start 2024-03-16T11:49:47.439 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:44.597+0000 7f6f77a81700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6f7006abd0 0x7f6f7019e670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:49:47.439 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:44.597+0000 7f6f77a81700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6f7006b580 0x7f6f7019ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:49:47.439 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:44.597+0000 7f6f77a81700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6f701076b0 0x7f6f701987f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:49:47.439 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:44.597+0000 7f6f77a81700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f6f701125f0 con 0x7f6f701076b0 2024-03-16T11:49:47.439 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:44.597+0000 7f6f77a81700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f6f70112470 con 0x7f6f7006b580 2024-03-16T11:49:47.439 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:44.597+0000 7f6f77a81700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f6f70112770 con 0x7f6f7006abd0 2024-03-16T11:49:47.439 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:44.597+0000 7f6f7601e700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6f701076b0 0x7f6f701987f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:49:47.439 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:44.597+0000 7f6f7581d700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6f7006abd0 0x7f6f7019e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:49:47.439 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:44.597+0000 7f6f7601e700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6f701076b0 0x7f6f701987f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57086/0 (socket says 172.21.15.29:57086) 2024-03-16T11:49:47.439 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:44.597+0000 7f6f7581d700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6f7006abd0 0x7f6f7019e670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.29:47242/0 (socket says 172.21.15.29:47242) 2024-03-16T11:49:47.439 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:44.597+0000 7f6f7601e700 1 -- 172.21.15.29:0/1027021516 learned_addr learned my addr 172.21.15.29:0/1027021516 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:49:47.439 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:44.597+0000 7f6f7501c700 1 --2- 172.21.15.29:0/1027021516 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6f7006b580 0x7f6f7019ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:49:47.440 INFO:teuthology.orchestra.run.smithi029.stdout:197568495670 2024-03-16T11:49:47.440 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:44.597+0000 7f6f7601e700 1 -- 172.21.15.29:0/1027021516 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6f7006abd0 msgr2=0x7f6f7019e670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:49:47.440 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:44.597+0000 7f6f7601e700 1 --2- 172.21.15.29:0/1027021516 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6f7006abd0 0x7f6f7019e670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:49:47.440 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:44.597+0000 7f6f7601e700 1 -- 172.21.15.29:0/1027021516 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6f7006b580 msgr2=0x7f6f7019ebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:49:47.440 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:44.597+0000 7f6f7601e700 1 --2- 172.21.15.29:0/1027021516 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6f7006b580 0x7f6f7019ebb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:49:47.440 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:44.597+0000 7f6f7601e700 1 -- 172.21.15.29:0/1027021516 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f70198fd0 con 0x7f6f701076b0 2024-03-16T11:49:47.441 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:44.598+0000 7f6f7601e700 1 --2- 172.21.15.29:0/1027021516 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6f701076b0 0x7f6f701987f0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f6f6800bab0 tx=0x7f6f6800bae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:49:47.441 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:44.598+0000 7f6f66ffd700 1 -- 172.21.15.29:0/1027021516 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f68027070 con 0x7f6f701076b0 2024-03-16T11:49:47.441 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:44.598+0000 7f6f77a81700 1 -- 172.21.15.29:0/1027021516 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6f70199260 con 0x7f6f701076b0 2024-03-16T11:49:47.441 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:44.598+0000 7f6f66ffd700 1 -- 172.21.15.29:0/1027021516 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6f68003b60 con 0x7f6f701076b0 2024-03-16T11:49:47.441 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:44.598+0000 7f6f66ffd700 1 -- 172.21.15.29:0/1027021516 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f68022da0 con 0x7f6f701076b0 2024-03-16T11:49:47.441 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:44.598+0000 7f6f77a81700 1 -- 172.21.15.29:0/1027021516 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6f701a54d0 con 0x7f6f701076b0 2024-03-16T11:49:47.441 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:44.599+0000 7f6f77a81700 1 -- 172.21.15.29:0/1027021516 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6f70103a30 con 0x7f6f701076b0 2024-03-16T11:49:47.441 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:44.600+0000 7f6f66ffd700 1 -- 172.21.15.29:0/1027021516 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f6f6802a070 con 0x7f6f701076b0 2024-03-16T11:49:47.441 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:44.600+0000 7f6f66ffd700 1 --2- 172.21.15.29:0/1027021516 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f6f5c0a2920 0x7f6f5c0a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:49:47.441 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:44.600+0000 7f6f7581d700 1 --2- 172.21.15.29:0/1027021516 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f6f5c0a2920 0x7f6f5c0a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:49:47.441 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:44.600+0000 7f6f66ffd700 1 -- 172.21.15.29:0/1027021516 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8211+0+0 (secure 0 0 0) 0x7f6f680cf920 con 0x7f6f701076b0 2024-03-16T11:49:47.441 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:44.601+0000 7f6f7581d700 1 --2- 172.21.15.29:0/1027021516 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f6f5c0a2920 0x7f6f5c0a4de0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f6f6c009550 tx=0x7f6f6c0091f0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:49:47.441 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:44.605+0000 7f6f66ffd700 1 -- 172.21.15.29:0/1027021516 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f6f6805e740 con 0x7f6f701076b0 2024-03-16T11:49:47.441 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:44.813+0000 7f6f77a81700 1 -- 172.21.15.29:0/1027021516 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7f6f70199d20 con 0x7f6f701076b0 2024-03-16T11:49:47.441 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:44.813+0000 7f6f66ffd700 1 -- 172.21.15.29:0/1027021516 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f6f68013070 con 0x7f6f701076b0 2024-03-16T11:49:47.441 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:44.815+0000 7f6f77a81700 1 -- 172.21.15.29:0/1027021516 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f6f5c0a2920 msgr2=0x7f6f5c0a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:49:47.442 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:44.815+0000 7f6f77a81700 1 --2- 172.21.15.29:0/1027021516 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f6f5c0a2920 0x7f6f5c0a4de0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f6f6c009550 tx=0x7f6f6c0091f0 comp rx=0 tx=0).stop 2024-03-16T11:49:47.442 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:44.815+0000 7f6f77a81700 1 -- 172.21.15.29:0/1027021516 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6f701076b0 msgr2=0x7f6f701987f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:49:47.442 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:44.815+0000 7f6f77a81700 1 --2- 172.21.15.29:0/1027021516 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6f701076b0 0x7f6f701987f0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f6f6800bab0 tx=0x7f6f6800bae0 comp rx=0 tx=0).stop 2024-03-16T11:49:47.442 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:44.815+0000 7f6f77a81700 1 -- 172.21.15.29:0/1027021516 shutdown_connections 2024-03-16T11:49:47.442 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:44.815+0000 7f6f77a81700 1 --2- 172.21.15.29:0/1027021516 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6f701076b0 0x7f6f701987f0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:49:47.442 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:44.815+0000 7f6f77a81700 1 --2- 172.21.15.29:0/1027021516 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f6f5c0a2920 0x7f6f5c0a4de0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:49:47.442 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:44.815+0000 7f6f77a81700 1 --2- 172.21.15.29:0/1027021516 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6f7006abd0 0x7f6f7019e670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:49:47.442 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:44.815+0000 7f6f77a81700 1 --2- 172.21.15.29:0/1027021516 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6f7006b580 0x7f6f7019ebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:49:47.442 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:44.815+0000 7f6f77a81700 1 -- 172.21.15.29:0/1027021516 >> 172.21.15.29:0/1027021516 conn(0x7f6f70078090 msgr2=0x7f6f700ff460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:49:47.442 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:44.816+0000 7f6f77a81700 1 -- 172.21.15.29:0/1027021516 shutdown_connections 2024-03-16T11:49:47.442 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:44.816+0000 7f6f77a81700 1 -- 172.21.15.29:0/1027021516 wait complete. 2024-03-16T11:49:48.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:49:47 smithi154 ceph-mon[36320]: pgmap v327: 97 pgs: 97 active+clean; 640 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-16T11:49:48.072 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:49:47 smithi029 ceph-mon[31975]: pgmap v327: 97 pgs: 97 active+clean; 640 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-16T11:49:48.106 INFO:tasks.cephadm.ceph_manager.ceph:need seq 197568495664 got 197568495670 for osd.6 2024-03-16T11:49:48.106 DEBUG:teuthology.parallel:result is None 2024-03-16T11:49:48.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:49:47 smithi152 ceph-mon[36291]: pgmap v327: 97 pgs: 97 active+clean; 640 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-16T11:49:48.539 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738446 got 34359738451 for osd.0 2024-03-16T11:49:48.540 DEBUG:teuthology.parallel:result is None 2024-03-16T11:49:49.341 INFO:tasks.cephadm.ceph_manager.ceph:need seq 107374182462 got 107374182468 for osd.3 2024-03-16T11:49:49.342 DEBUG:teuthology.parallel:result is None 2024-03-16T11:49:49.775 INFO:tasks.cephadm.ceph_manager.ceph:need seq 309237645338 got 309237645343 for osd.10 2024-03-16T11:49:49.775 DEBUG:teuthology.parallel:result is None 2024-03-16T11:49:50.012 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/d1887f28-e389-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T11:49:50.016 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:49:49 smithi029 ceph-mon[31975]: pgmap v328: 97 pgs: 97 active+clean; 640 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-03-16T11:49:50.022 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/d1887f28-e389-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T11:49:50.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:49:49 smithi154 ceph-mon[36320]: pgmap v328: 97 pgs: 97 active+clean; 640 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-03-16T11:49:50.048 INFO:tasks.cephadm.ceph_manager.ceph:need seq 141733920828 got 141733920831 for osd.4 2024-03-16T11:49:50.048 DEBUG:teuthology.parallel:result is None 2024-03-16T11:49:50.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:49:49 smithi152 ceph-mon[36291]: pgmap v328: 97 pgs: 97 active+clean; 640 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-03-16T11:49:50.836 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574922 got 55834574926 for osd.1 2024-03-16T11:49:50.836 DEBUG:teuthology.parallel:result is None 2024-03-16T11:49:51.067 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:51.027+0000 7f30ca12a700 1 -- 172.21.15.29:0/275500415 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f30c4107210 msgr2=0x7f30c41075f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:49:51.067 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:51.027+0000 7f30ca12a700 1 --2- 172.21.15.29:0/275500415 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f30c4107210 0x7f30c41075f0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f30b4009990 tx=0x7f30b4009ca0 comp rx=0 tx=0).stop 2024-03-16T11:49:51.067 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:51.027+0000 7f30ca12a700 1 -- 172.21.15.29:0/275500415 shutdown_connections 2024-03-16T11:49:51.067 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:51.027+0000 7f30ca12a700 1 --2- 172.21.15.29:0/275500415 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f30c4101b70 0x7f30c410e050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:49:51.067 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:51.027+0000 7f30ca12a700 1 --2- 172.21.15.29:0/275500415 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f30c4107210 0x7f30c41075f0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:49:51.067 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:51.027+0000 7f30ca12a700 1 --2- 172.21.15.29:0/275500415 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f30c41011d0 0x7f30c4101630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:49:51.068 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:51.027+0000 7f30ca12a700 1 -- 172.21.15.29:0/275500415 >> 172.21.15.29:0/275500415 conn(0x7f30c4078100 msgr2=0x7f30c4078510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:49:51.068 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:51.027+0000 7f30ca12a700 1 -- 172.21.15.29:0/275500415 shutdown_connections 2024-03-16T11:49:51.068 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:51.027+0000 7f30ca12a700 1 -- 172.21.15.29:0/275500415 wait complete. 2024-03-16T11:49:51.068 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:51.027+0000 7f30ca12a700 1 Processor -- start 2024-03-16T11:49:51.068 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:51.027+0000 7f30ca12a700 1 -- start start 2024-03-16T11:49:51.068 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:51.028+0000 7f30ca12a700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f30c41011d0 0x7f30c419e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:49:51.068 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:51.028+0000 7f30ca12a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f30c4101b70 0x7f30c419ec10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:49:51.068 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:51.028+0000 7f30ca12a700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f30c4107210 0x7f30c41988a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:49:51.068 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:51.028+0000 7f30ca12a700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f30c4111450 con 0x7f30c4101b70 2024-03-16T11:49:51.068 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:51.028+0000 7f30ca12a700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f30c41112d0 con 0x7f30c4107210 2024-03-16T11:49:51.068 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:51.028+0000 7f30ca12a700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f30c41115d0 con 0x7f30c41011d0 2024-03-16T11:49:51.068 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:51.028+0000 7f30c2ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f30c4101b70 0x7f30c419ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:49:51.068 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:51.028+0000 7f30c2ffd700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f30c4101b70 0x7f30c419ec10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60840/0 (socket says 172.21.15.29:60840) 2024-03-16T11:49:51.068 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:51.028+0000 7f30c2ffd700 1 -- 172.21.15.29:0/1455529577 learned_addr learned my addr 172.21.15.29:0/1455529577 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:49:51.069 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:51.028+0000 7f30c37fe700 1 --2- 172.21.15.29:0/1455529577 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f30c41011d0 0x7f30c419e6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:49:51.069 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:51.028+0000 7f30c3fff700 1 --2- 172.21.15.29:0/1455529577 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f30c4107210 0x7f30c41988a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:49:51.069 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:51.028+0000 7f30c2ffd700 1 -- 172.21.15.29:0/1455529577 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f30c41011d0 msgr2=0x7f30c419e6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:49:51.069 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:51.028+0000 7f30c2ffd700 1 --2- 172.21.15.29:0/1455529577 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f30c41011d0 0x7f30c419e6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:49:51.069 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:51.028+0000 7f30c2ffd700 1 -- 172.21.15.29:0/1455529577 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f30c4107210 msgr2=0x7f30c41988a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:49:51.069 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:51.028+0000 7f30c2ffd700 1 --2- 172.21.15.29:0/1455529577 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f30c4107210 0x7f30c41988a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:49:51.069 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:51.028+0000 7f30c2ffd700 1 -- 172.21.15.29:0/1455529577 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30c4199080 con 0x7f30c4101b70 2024-03-16T11:49:51.069 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:51.028+0000 7f30c2ffd700 1 --2- 172.21.15.29:0/1455529577 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f30c4101b70 0x7f30c419ec10 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f30b800d950 tx=0x7f30b800dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:49:51.069 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:51.028+0000 7f30c0ff9700 1 -- 172.21.15.29:0/1455529577 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30b8016070 con 0x7f30c4101b70 2024-03-16T11:49:51.069 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:51.028+0000 7f30ca12a700 1 -- 172.21.15.29:0/1455529577 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f30c4199370 con 0x7f30c4101b70 2024-03-16T11:49:51.069 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:51.028+0000 7f30c0ff9700 1 -- 172.21.15.29:0/1455529577 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f30b800fd70 con 0x7f30c4101b70 2024-03-16T11:49:51.069 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:51.029+0000 7f30c0ff9700 1 -- 172.21.15.29:0/1455529577 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30b800e610 con 0x7f30c4101b70 2024-03-16T11:49:51.069 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:51.029+0000 7f30ca12a700 1 -- 172.21.15.29:0/1455529577 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f30c41a5530 con 0x7f30c4101b70 2024-03-16T11:49:51.069 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:51.029+0000 7f30ca12a700 1 -- 172.21.15.29:0/1455529577 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f30c41028c0 con 0x7f30c4101b70 2024-03-16T11:49:51.069 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:51.030+0000 7f30c0ff9700 1 -- 172.21.15.29:0/1455529577 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f30b8019070 con 0x7f30c4101b70 2024-03-16T11:49:51.069 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:51.031+0000 7f30c0ff9700 1 --2- 172.21.15.29:0/1455529577 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f30ac0a2920 0x7f30ac0a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:49:51.070 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:51.031+0000 7f30c37fe700 1 --2- 172.21.15.29:0/1455529577 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f30ac0a2920 0x7f30ac0a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:49:51.070 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:51.031+0000 7f30c0ff9700 1 -- 172.21.15.29:0/1455529577 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8211+0+0 (secure 0 0 0) 0x7f30b80cea50 con 0x7f30c4101b70 2024-03-16T11:49:51.070 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:51.031+0000 7f30c37fe700 1 --2- 172.21.15.29:0/1455529577 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f30ac0a2920 0x7f30ac0a4de0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f30b400b5c0 tx=0x7f30b4005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:49:51.070 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:51.036+0000 7f30c0ff9700 1 -- 172.21.15.29:0/1455529577 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f30b809b9a0 con 0x7f30c4101b70 2024-03-16T11:49:51.243 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:51.242+0000 7f30ca12a700 1 -- 172.21.15.29:0/1455529577 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 8} v 0) -- 0x7f30c41a5790 con 0x7f30c4101b70 2024-03-16T11:49:51.243 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:51.243+0000 7f30c0ff9700 1 -- 172.21.15.29:0/1455529577 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 8}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f30b809b2a0 con 0x7f30c4101b70 2024-03-16T11:49:51.243 INFO:teuthology.orchestra.run.smithi029.stdout:253403070508 2024-03-16T11:49:51.245 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:51.245+0000 7f30ca12a700 1 -- 172.21.15.29:0/1455529577 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f30ac0a2920 msgr2=0x7f30ac0a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:49:51.245 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:51.245+0000 7f30ca12a700 1 --2- 172.21.15.29:0/1455529577 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f30ac0a2920 0x7f30ac0a4de0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f30b400b5c0 tx=0x7f30b4005960 comp rx=0 tx=0).stop 2024-03-16T11:49:51.245 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:51.245+0000 7f30ca12a700 1 -- 172.21.15.29:0/1455529577 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f30c4101b70 msgr2=0x7f30c419ec10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:49:51.246 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:51.245+0000 7f30ca12a700 1 --2- 172.21.15.29:0/1455529577 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f30c4101b70 0x7f30c419ec10 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f30b800d950 tx=0x7f30b800dc60 comp rx=0 tx=0).stop 2024-03-16T11:49:51.246 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:51.245+0000 7f30ca12a700 1 -- 172.21.15.29:0/1455529577 shutdown_connections 2024-03-16T11:49:51.246 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:51.245+0000 7f30ca12a700 1 --2- 172.21.15.29:0/1455529577 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f30c4107210 0x7f30c41988a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:49:51.246 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:51.245+0000 7f30ca12a700 1 --2- 172.21.15.29:0/1455529577 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f30ac0a2920 0x7f30ac0a4de0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:49:51.246 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:51.245+0000 7f30ca12a700 1 --2- 172.21.15.29:0/1455529577 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f30c41011d0 0x7f30c419e6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:49:51.246 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:51.245+0000 7f30ca12a700 1 --2- 172.21.15.29:0/1455529577 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f30c4101b70 0x7f30c419ec10 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:49:51.246 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:51.245+0000 7f30ca12a700 1 -- 172.21.15.29:0/1455529577 >> 172.21.15.29:0/1455529577 conn(0x7f30c4078100 msgr2=0x7f30c4105470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:49:51.246 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:51.245+0000 7f30ca12a700 1 -- 172.21.15.29:0/1455529577 shutdown_connections 2024-03-16T11:49:51.246 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:51.245+0000 7f30ca12a700 1 -- 172.21.15.29:0/1455529577 wait complete. 2024-03-16T11:49:51.299 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:51.287+0000 7f22c4fd3700 1 -- 172.21.15.29:0/1875084041 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f22c0074e20 msgr2=0x7f22c010e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:49:51.300 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:51.287+0000 7f22c4fd3700 1 --2- 172.21.15.29:0/1875084041 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f22c0074e20 0x7f22c010e7c0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f22a80098f0 tx=0x7f22a8009c00 comp rx=0 tx=0).stop 2024-03-16T11:49:51.300 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:51.287+0000 7f22c4fd3700 1 -- 172.21.15.29:0/1875084041 shutdown_connections 2024-03-16T11:49:51.300 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:51.287+0000 7f22c4fd3700 1 --2- 172.21.15.29:0/1875084041 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f22c010ed00 0x7f22c01110f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:49:51.300 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:51.287+0000 7f22c4fd3700 1 --2- 172.21.15.29:0/1875084041 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f22c0074500 0x7f22c00748e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:49:51.300 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:51.287+0000 7f22c4fd3700 1 --2- 172.21.15.29:0/1875084041 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f22c0074e20 0x7f22c010e7c0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:49:51.300 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:51.287+0000 7f22c4fd3700 1 -- 172.21.15.29:0/1875084041 >> 172.21.15.29:0/1875084041 conn(0x7f22c0079390 msgr2=0x7f22c00797a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:49:51.300 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:51.287+0000 7f22c4fd3700 1 -- 172.21.15.29:0/1875084041 shutdown_connections 2024-03-16T11:49:51.300 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:51.287+0000 7f22c4fd3700 1 -- 172.21.15.29:0/1875084041 wait complete. 2024-03-16T11:49:51.300 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:51.288+0000 7f22c4fd3700 1 Processor -- start 2024-03-16T11:49:51.300 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:51.288+0000 7f22c4fd3700 1 -- start start 2024-03-16T11:49:51.300 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:51.288+0000 7f22c4fd3700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f22c0074500 0x7f22c01a0a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:49:51.300 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:51.288+0000 7f22c4fd3700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f22c0074e20 0x7f22c01a0f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:49:51.301 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:51.288+0000 7f22c4fd3700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f22c010ed00 0x7f22c019ab10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:49:51.301 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:51.288+0000 7f22c4fd3700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f22c01147b0 con 0x7f22c010ed00 2024-03-16T11:49:51.301 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:51.288+0000 7f22c4fd3700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f22c0114630 con 0x7f22c0074e20 2024-03-16T11:49:51.301 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:51.288+0000 7f22c4fd3700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f22c0114930 con 0x7f22c0074500 2024-03-16T11:49:51.301 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:51.288+0000 7f22bed9d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f22c010ed00 0x7f22c019ab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:49:51.301 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:51.288+0000 7f22bed9d700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f22c010ed00 0x7f22c019ab10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60854/0 (socket says 172.21.15.29:60854) 2024-03-16T11:49:51.301 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:51.288+0000 7f22bed9d700 1 -- 172.21.15.29:0/3025671236 learned_addr learned my addr 172.21.15.29:0/3025671236 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:49:51.301 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:51.288+0000 7f22be59c700 1 --2- 172.21.15.29:0/3025671236 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f22c0074500 0x7f22c01a0a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:49:51.301 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:51.288+0000 7f22bdd9b700 1 --2- 172.21.15.29:0/3025671236 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f22c0074e20 0x7f22c01a0f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:49:51.301 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:51.289+0000 7f22bed9d700 1 -- 172.21.15.29:0/3025671236 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f22c0074500 msgr2=0x7f22c01a0a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:49:51.301 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:51.289+0000 7f22bed9d700 1 --2- 172.21.15.29:0/3025671236 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f22c0074500 0x7f22c01a0a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:49:51.301 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:51.289+0000 7f22bed9d700 1 -- 172.21.15.29:0/3025671236 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f22c0074e20 msgr2=0x7f22c01a0f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:49:51.301 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:51.289+0000 7f22bed9d700 1 --2- 172.21.15.29:0/3025671236 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f22c0074e20 0x7f22c01a0f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:49:51.301 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:51.289+0000 7f22bed9d700 1 -- 172.21.15.29:0/3025671236 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22c019b3f0 con 0x7f22c010ed00 2024-03-16T11:49:51.302 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:51.289+0000 7f22bed9d700 1 --2- 172.21.15.29:0/3025671236 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f22c010ed00 0x7f22c019ab10 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f22b800ebe0 tx=0x7f22b800eef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:49:51.302 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:51.289+0000 7f22b77fe700 1 -- 172.21.15.29:0/3025671236 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22b800cc90 con 0x7f22c010ed00 2024-03-16T11:49:51.302 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:51.289+0000 7f22c4fd3700 1 -- 172.21.15.29:0/3025671236 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f22c019b6e0 con 0x7f22c010ed00 2024-03-16T11:49:51.302 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:51.289+0000 7f22b77fe700 1 -- 172.21.15.29:0/3025671236 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f22b800ce30 con 0x7f22c010ed00 2024-03-16T11:49:51.302 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:51.289+0000 7f22b77fe700 1 -- 172.21.15.29:0/3025671236 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22b8010430 con 0x7f22c010ed00 2024-03-16T11:49:51.302 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:51.289+0000 7f22c4fd3700 1 -- 172.21.15.29:0/3025671236 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f22c01a79c0 con 0x7f22c010ed00 2024-03-16T11:49:51.302 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:51.290+0000 7f22c4fd3700 1 -- 172.21.15.29:0/3025671236 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f22c0075b90 con 0x7f22c010ed00 2024-03-16T11:49:51.302 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:51.291+0000 7f22b77fe700 1 -- 172.21.15.29:0/3025671236 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f22b8003fa0 con 0x7f22c010ed00 2024-03-16T11:49:51.302 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:51.292+0000 7f22b77fe700 1 --2- 172.21.15.29:0/3025671236 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f22ac0a2920 0x7f22ac0a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:49:51.302 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:51.292+0000 7f22be59c700 1 --2- 172.21.15.29:0/3025671236 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f22ac0a2920 0x7f22ac0a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:49:51.302 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:51.292+0000 7f22b77fe700 1 -- 172.21.15.29:0/3025671236 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8211+0+0 (secure 0 0 0) 0x7f22b80ceee0 con 0x7f22c010ed00 2024-03-16T11:49:51.302 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:51.292+0000 7f22be59c700 1 --2- 172.21.15.29:0/3025671236 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f22ac0a2920 0x7f22ac0a4de0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f22b000f540 tx=0x7f22b000b040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:49:51.303 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:51.295+0000 7f22b77fe700 1 -- 172.21.15.29:0/3025671236 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f22b809bec0 con 0x7f22c010ed00 2024-03-16T11:49:51.503 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:51.503+0000 7f22c4fd3700 1 -- 172.21.15.29:0/3025671236 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 9} v 0) -- 0x7f22c019b910 con 0x7f22c010ed00 2024-03-16T11:49:51.504 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:49:51 smithi029 ceph-mon[31975]: from='client.? 172.21.15.29:0/1455529577' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-03-16T11:49:51.504 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:51.504+0000 7f22b77fe700 1 -- 172.21.15.29:0/3025671236 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 9}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f22b809b7c0 con 0x7f22c010ed00 2024-03-16T11:49:51.504 INFO:teuthology.orchestra.run.smithi029.stdout:287762808871 2024-03-16T11:49:51.506 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:51.506+0000 7f22c4fd3700 1 -- 172.21.15.29:0/3025671236 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f22ac0a2920 msgr2=0x7f22ac0a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:49:51.506 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:51.506+0000 7f22c4fd3700 1 --2- 172.21.15.29:0/3025671236 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f22ac0a2920 0x7f22ac0a4de0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f22b000f540 tx=0x7f22b000b040 comp rx=0 tx=0).stop 2024-03-16T11:49:51.506 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:51.506+0000 7f22c4fd3700 1 -- 172.21.15.29:0/3025671236 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f22c010ed00 msgr2=0x7f22c019ab10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:49:51.506 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:51.506+0000 7f22c4fd3700 1 --2- 172.21.15.29:0/3025671236 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f22c010ed00 0x7f22c019ab10 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f22b800ebe0 tx=0x7f22b800eef0 comp rx=0 tx=0).stop 2024-03-16T11:49:51.506 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:51.506+0000 7f22c4fd3700 1 -- 172.21.15.29:0/3025671236 shutdown_connections 2024-03-16T11:49:51.506 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:51.506+0000 7f22c4fd3700 1 --2- 172.21.15.29:0/3025671236 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f22c010ed00 0x7f22c019ab10 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:49:51.507 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:51.506+0000 7f22c4fd3700 1 --2- 172.21.15.29:0/3025671236 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f22ac0a2920 0x7f22ac0a4de0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:49:51.507 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:51.506+0000 7f22c4fd3700 1 --2- 172.21.15.29:0/3025671236 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f22c0074500 0x7f22c01a0a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:49:51.507 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:51.506+0000 7f22c4fd3700 1 --2- 172.21.15.29:0/3025671236 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f22c0074e20 0x7f22c01a0f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:49:51.507 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:51.506+0000 7f22c4fd3700 1 -- 172.21.15.29:0/3025671236 >> 172.21.15.29:0/3025671236 conn(0x7f22c0079390 msgr2=0x7f22c00ff530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:49:51.507 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:51.506+0000 7f22c4fd3700 1 -- 172.21.15.29:0/3025671236 shutdown_connections 2024-03-16T11:49:51.507 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:51.506+0000 7f22c4fd3700 1 -- 172.21.15.29:0/3025671236 wait complete. 2024-03-16T11:49:51.627 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:49:51 smithi152 ceph-mon[36291]: from='client.? 172.21.15.29:0/1455529577' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-03-16T11:49:51.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:49:51 smithi154 ceph-mon[36320]: from='client.? 172.21.15.29:0/1455529577' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-03-16T11:49:52.588 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:49:52 smithi029 ceph-mon[31975]: from='client.? 172.21.15.29:0/3025671236' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-03-16T11:49:52.588 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:49:52 smithi029 ceph-mon[31975]: pgmap v329: 97 pgs: 97 active+clean; 640 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-16T11:49:52.627 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:49:52 smithi152 ceph-mon[36291]: from='client.? 172.21.15.29:0/3025671236' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-03-16T11:49:52.627 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:49:52 smithi152 ceph-mon[36291]: pgmap v329: 97 pgs: 97 active+clean; 640 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-16T11:49:52.739 INFO:tasks.cephadm.ceph_manager.ceph:need seq 253403070501 got 253403070508 for osd.8 2024-03-16T11:49:52.739 DEBUG:teuthology.parallel:result is None 2024-03-16T11:49:52.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:49:52 smithi154 ceph-mon[36320]: from='client.? 172.21.15.29:0/3025671236' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-03-16T11:49:52.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:49:52 smithi154 ceph-mon[36320]: pgmap v329: 97 pgs: 97 active+clean; 640 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-16T11:49:53.983 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:49:53 smithi029 ceph-mon[31975]: pgmap v330: 97 pgs: 97 active+clean; 640 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-16T11:49:54.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:49:53 smithi154 ceph-mon[36320]: pgmap v330: 97 pgs: 97 active+clean; 640 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-16T11:49:54.092 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:54.090+0000 7f20ce328700 1 -- 172.21.15.29:0/618891149 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f20c810ede0 msgr2=0x7f20c81111d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:49:54.092 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:54.090+0000 7f20ce328700 1 --2- 172.21.15.29:0/618891149 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f20c810ede0 0x7f20c81111d0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f20c000b0a0 tx=0x7f20c000b3b0 comp rx=0 tx=0).stop 2024-03-16T11:49:54.092 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:54.090+0000 7f20ce328700 1 -- 172.21.15.29:0/618891149 shutdown_connections 2024-03-16T11:49:54.092 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:54.090+0000 7f20ce328700 1 --2- 172.21.15.29:0/618891149 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f20c810ede0 0x7f20c81111d0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:49:54.092 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:54.090+0000 7f20ce328700 1 --2- 172.21.15.29:0/618891149 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f20c806a9d0 0x7f20c806adb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:49:54.092 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:54.090+0000 7f20ce328700 1 --2- 172.21.15.29:0/618891149 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f20c806b2f0 0x7f20c810e8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:49:54.092 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:54.090+0000 7f20ce328700 1 -- 172.21.15.29:0/618891149 >> 172.21.15.29:0/618891149 conn(0x7f20c8077220 msgr2=0x7f20c8077630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:49:54.092 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:54.090+0000 7f20ce328700 1 -- 172.21.15.29:0/618891149 shutdown_connections 2024-03-16T11:49:54.092 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:54.090+0000 7f20ce328700 1 -- 172.21.15.29:0/618891149 wait complete. 2024-03-16T11:49:54.093 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:54.090+0000 7f20ce328700 1 Processor -- start 2024-03-16T11:49:54.093 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:54.090+0000 7f20ce328700 1 -- start start 2024-03-16T11:49:54.093 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:54.091+0000 7f20ce328700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f20c806a9d0 0x7f20c81a0820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:49:54.093 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:54.091+0000 7f20ce328700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f20c806b2f0 0x7f20c81a0d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:49:54.093 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:54.091+0000 7f20ce328700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f20c810ede0 0x7f20c819a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:49:54.093 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:54.091+0000 7f20ce328700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f20c81147c0 con 0x7f20c806a9d0 2024-03-16T11:49:54.093 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:54.091+0000 7f20ce328700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f20c8114640 con 0x7f20c806b2f0 2024-03-16T11:49:54.093 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:54.091+0000 7f20ce328700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f20c8114940 con 0x7f20c810ede0 2024-03-16T11:49:54.093 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:54.091+0000 7f20c7fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f20c806a9d0 0x7f20c81a0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:49:54.093 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:54.091+0000 7f20c77fe700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f20c806b2f0 0x7f20c81a0d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:49:54.093 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:54.091+0000 7f20c77fe700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f20c806b2f0 0x7f20c81a0d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.29:42076/0 (socket says 172.21.15.29:42076) 2024-03-16T11:49:54.093 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:54.091+0000 7f20c7fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f20c806a9d0 0x7f20c81a0820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60882/0 (socket says 172.21.15.29:60882) 2024-03-16T11:49:54.093 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:54.091+0000 7f20cc8c5700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f20c810ede0 0x7f20c819a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:49:54.093 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:54.091+0000 7f20c77fe700 1 -- 172.21.15.29:0/1387357590 learned_addr learned my addr 172.21.15.29:0/1387357590 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:49:54.093 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:54.091+0000 7f20cc8c5700 1 -- 172.21.15.29:0/1387357590 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f20c806b2f0 msgr2=0x7f20c81a0d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:49:54.094 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:54.091+0000 7f20cc8c5700 1 --2- 172.21.15.29:0/1387357590 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f20c806b2f0 0x7f20c81a0d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:49:54.094 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:54.091+0000 7f20cc8c5700 1 -- 172.21.15.29:0/1387357590 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f20c806a9d0 msgr2=0x7f20c81a0820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:49:54.094 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:54.091+0000 7f20cc8c5700 1 --2- 172.21.15.29:0/1387357590 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f20c806a9d0 0x7f20c81a0820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:49:54.094 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:54.091+0000 7f20cc8c5700 1 -- 172.21.15.29:0/1387357590 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20c819b140 con 0x7f20c810ede0 2024-03-16T11:49:54.094 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:54.091+0000 7f20c77fe700 1 --2- 172.21.15.29:0/1387357590 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f20c806b2f0 0x7f20c81a0d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:49:54.094 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:54.091+0000 7f20c7fff700 1 --2- 172.21.15.29:0/1387357590 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f20c806a9d0 0x7f20c81a0820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:49:54.094 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:54.091+0000 7f20cc8c5700 1 --2- 172.21.15.29:0/1387357590 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f20c810ede0 0x7f20c819a9e0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f20c000b0a0 tx=0x7f20c0004d00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:49:54.094 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:54.092+0000 7f20c57fa700 1 -- 172.21.15.29:0/1387357590 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20c0021070 con 0x7f20c810ede0 2024-03-16T11:49:54.094 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:54.092+0000 7f20ce328700 1 -- 172.21.15.29:0/1387357590 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f20c819b3d0 con 0x7f20c810ede0 2024-03-16T11:49:54.094 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:54.092+0000 7f20c57fa700 1 -- 172.21.15.29:0/1387357590 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f20c0007b80 con 0x7f20c810ede0 2024-03-16T11:49:54.094 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:54.092+0000 7f20c57fa700 1 -- 172.21.15.29:0/1387357590 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20c0005040 con 0x7f20c810ede0 2024-03-16T11:49:54.094 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:54.092+0000 7f20ce328700 1 -- 172.21.15.29:0/1387357590 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f20c81a76f0 con 0x7f20c810ede0 2024-03-16T11:49:54.095 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:54.093+0000 7f20ce328700 1 -- 172.21.15.29:0/1387357590 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f20c8068f90 con 0x7f20c810ede0 2024-03-16T11:49:54.095 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:54.094+0000 7f20c57fa700 1 -- 172.21.15.29:0/1387357590 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f20c0024070 con 0x7f20c810ede0 2024-03-16T11:49:54.096 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:54.094+0000 7f20c57fa700 1 --2- 172.21.15.29:0/1387357590 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f20b00a2920 0x7f20b00a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:49:54.096 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:54.095+0000 7f20c7fff700 1 --2- 172.21.15.29:0/1387357590 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f20b00a2920 0x7f20b00a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:49:54.096 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:54.095+0000 7f20c57fa700 1 -- 172.21.15.29:0/1387357590 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8211+0+0 (secure 0 0 0) 0x7f20c000f020 con 0x7f20c810ede0 2024-03-16T11:49:54.096 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:54.095+0000 7f20c7fff700 1 --2- 172.21.15.29:0/1387357590 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f20b00a2920 0x7f20b00a4de0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f20c819bb20 tx=0x7f20b8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:49:54.098 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:54.098+0000 7f20c57fa700 1 -- 172.21.15.29:0/1387357590 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f20c002b020 con 0x7f20c810ede0 2024-03-16T11:49:54.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:49:53 smithi152 ceph-mon[36291]: pgmap v330: 97 pgs: 97 active+clean; 640 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-16T11:49:54.307 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:54.307+0000 7f20ce328700 1 -- 172.21.15.29:0/1387357590 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7f20c802db70 con 0x7f20c810ede0 2024-03-16T11:49:54.308 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:54.307+0000 7f20c57fa700 1 -- 172.21.15.29:0/1387357590 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f20c0007d30 con 0x7f20c810ede0 2024-03-16T11:49:54.308 INFO:teuthology.orchestra.run.smithi029.stdout:219043332145 2024-03-16T11:49:54.311 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:54.311+0000 7f20ce328700 1 -- 172.21.15.29:0/1387357590 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f20b00a2920 msgr2=0x7f20b00a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:49:54.311 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:54.311+0000 7f20ce328700 1 --2- 172.21.15.29:0/1387357590 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f20b00a2920 0x7f20b00a4de0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f20c819bb20 tx=0x7f20b8009260 comp rx=0 tx=0).stop 2024-03-16T11:49:54.311 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:54.311+0000 7f20ce328700 1 -- 172.21.15.29:0/1387357590 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f20c810ede0 msgr2=0x7f20c819a9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:49:54.311 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:54.311+0000 7f20ce328700 1 --2- 172.21.15.29:0/1387357590 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f20c810ede0 0x7f20c819a9e0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f20c000b0a0 tx=0x7f20c0004d00 comp rx=0 tx=0).stop 2024-03-16T11:49:54.311 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:54.311+0000 7f20ce328700 1 -- 172.21.15.29:0/1387357590 shutdown_connections 2024-03-16T11:49:54.311 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:54.311+0000 7f20ce328700 1 --2- 172.21.15.29:0/1387357590 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f20b00a2920 0x7f20b00a4de0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:49:54.311 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:54.311+0000 7f20ce328700 1 --2- 172.21.15.29:0/1387357590 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f20c810ede0 0x7f20c819a9e0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:49:54.312 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:54.311+0000 7f20ce328700 1 --2- 172.21.15.29:0/1387357590 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f20c806a9d0 0x7f20c81a0820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:49:54.312 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:54.311+0000 7f20ce328700 1 --2- 172.21.15.29:0/1387357590 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f20c806b2f0 0x7f20c81a0d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:49:54.312 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:54.311+0000 7f20ce328700 1 -- 172.21.15.29:0/1387357590 >> 172.21.15.29:0/1387357590 conn(0x7f20c8077220 msgr2=0x7f20c81101f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:49:54.312 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:54.311+0000 7f20ce328700 1 -- 172.21.15.29:0/1387357590 shutdown_connections 2024-03-16T11:49:54.312 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:54.311+0000 7f20ce328700 1 -- 172.21.15.29:0/1387357590 wait complete. 2024-03-16T11:49:54.344 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:54.330+0000 7f1b5a37a700 1 -- 172.21.15.29:0/1900725533 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1b541025e0 msgr2=0x7f1b54105e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:49:54.345 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:54.330+0000 7f1b5a37a700 1 --2- 172.21.15.29:0/1900725533 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1b541025e0 0x7f1b54105e70 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f1b480098f0 tx=0x7f1b48009c00 comp rx=0 tx=0).stop 2024-03-16T11:49:54.345 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:54.330+0000 7f1b5a37a700 1 -- 172.21.15.29:0/1900725533 shutdown_connections 2024-03-16T11:49:54.345 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:54.330+0000 7f1b5a37a700 1 --2- 172.21.15.29:0/1900725533 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1b54106550 0x7f1b5410e120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:49:54.345 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:54.330+0000 7f1b5a37a700 1 --2- 172.21.15.29:0/1900725533 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1b54101c30 0x7f1b54102010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:49:54.345 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:54.330+0000 7f1b5a37a700 1 --2- 172.21.15.29:0/1900725533 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1b541025e0 0x7f1b54105e70 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:49:54.345 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:54.330+0000 7f1b5a37a700 1 -- 172.21.15.29:0/1900725533 >> 172.21.15.29:0/1900725533 conn(0x7f1b540777d0 msgr2=0x7f1b540febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:49:54.345 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:54.330+0000 7f1b5a37a700 1 -- 172.21.15.29:0/1900725533 shutdown_connections 2024-03-16T11:49:54.345 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:54.331+0000 7f1b5a37a700 1 -- 172.21.15.29:0/1900725533 wait complete. 2024-03-16T11:49:54.345 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:54.331+0000 7f1b5a37a700 1 Processor -- start 2024-03-16T11:49:54.345 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:54.331+0000 7f1b5a37a700 1 -- start start 2024-03-16T11:49:54.345 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:54.331+0000 7f1b5a37a700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1b54101c30 0x7f1b54197ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:49:54.345 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:54.332+0000 7f1b5a37a700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1b541025e0 0x7f1b54198000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:49:54.345 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:54.332+0000 7f1b5a37a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1b54106550 0x7f1b5419c3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:49:54.346 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:54.332+0000 7f1b5a37a700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f1b54111510 con 0x7f1b54106550 2024-03-16T11:49:54.346 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:54.332+0000 7f1b5a37a700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f1b54111390 con 0x7f1b54101c30 2024-03-16T11:49:54.346 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:54.332+0000 7f1b5a37a700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f1b54111690 con 0x7f1b541025e0 2024-03-16T11:49:54.346 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:54.332+0000 7f1b58917700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1b54106550 0x7f1b5419c3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:49:54.346 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:54.332+0000 7f1b53fff700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1b54101c30 0x7f1b54197ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:49:54.346 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:54.332+0000 7f1b58917700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1b54106550 0x7f1b5419c3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60906/0 (socket says 172.21.15.29:60906) 2024-03-16T11:49:54.346 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:54.332+0000 7f1b58917700 1 -- 172.21.15.29:0/548855204 learned_addr learned my addr 172.21.15.29:0/548855204 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:49:54.346 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:54.332+0000 7f1b537fe700 1 --2- 172.21.15.29:0/548855204 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1b541025e0 0x7f1b54198000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:49:54.346 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:54.332+0000 7f1b53fff700 1 -- 172.21.15.29:0/548855204 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1b541025e0 msgr2=0x7f1b54198000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:49:54.346 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:54.332+0000 7f1b53fff700 1 --2- 172.21.15.29:0/548855204 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1b541025e0 0x7f1b54198000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:49:54.346 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:54.332+0000 7f1b53fff700 1 -- 172.21.15.29:0/548855204 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1b54106550 msgr2=0x7f1b5419c3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:49:54.346 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:54.332+0000 7f1b53fff700 1 --2- 172.21.15.29:0/548855204 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1b54106550 0x7f1b5419c3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:49:54.346 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:54.332+0000 7f1b53fff700 1 -- 172.21.15.29:0/548855204 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b5419cac0 con 0x7f1b54101c30 2024-03-16T11:49:54.346 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:54.332+0000 7f1b53fff700 1 --2- 172.21.15.29:0/548855204 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1b54101c30 0x7f1b54197ac0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f1b4400e9f0 tx=0x7f1b4400ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:49:54.346 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:54.332+0000 7f1b517fa700 1 -- 172.21.15.29:0/548855204 <== mon.2 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b4400cb70 con 0x7f1b54101c30 2024-03-16T11:49:54.347 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:54.332+0000 7f1b517fa700 1 -- 172.21.15.29:0/548855204 <== mon.2 v2:172.21.15.152:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1b4400ee90 con 0x7f1b54101c30 2024-03-16T11:49:54.347 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:54.332+0000 7f1b5a37a700 1 -- 172.21.15.29:0/548855204 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1b5419cdb0 con 0x7f1b54101c30 2024-03-16T11:49:54.347 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:54.332+0000 7f1b517fa700 1 -- 172.21.15.29:0/548855204 <== mon.2 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b44003dc0 con 0x7f1b54101c30 2024-03-16T11:49:54.347 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:54.333+0000 7f1b5a37a700 1 -- 172.21.15.29:0/548855204 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1b541a4670 con 0x7f1b54101c30 2024-03-16T11:49:54.347 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:54.333+0000 7f1b5a37a700 1 -- 172.21.15.29:0/548855204 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1b54068f90 con 0x7f1b54101c30 2024-03-16T11:49:54.347 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:54.335+0000 7f1b517fa700 1 -- 172.21.15.29:0/548855204 <== mon.2 v2:172.21.15.152:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f1b44004000 con 0x7f1b54101c30 2024-03-16T11:49:54.347 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:54.335+0000 7f1b517fa700 1 --2- 172.21.15.29:0/548855204 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f1b3c0a2920 0x7f1b3c0a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:49:54.347 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:54.335+0000 7f1b537fe700 1 --2- 172.21.15.29:0/548855204 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f1b3c0a2920 0x7f1b3c0a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:49:54.347 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:54.335+0000 7f1b517fa700 1 -- 172.21.15.29:0/548855204 <== mon.2 v2:172.21.15.152:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8211+0+0 (secure 0 0 0) 0x7f1b440ced10 con 0x7f1b54101c30 2024-03-16T11:49:54.347 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:54.336+0000 7f1b537fe700 1 --2- 172.21.15.29:0/548855204 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f1b3c0a2920 0x7f1b3c0a4de0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f1b4800c010 tx=0x7f1b4801a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:49:54.347 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:54.339+0000 7f1b517fa700 1 -- 172.21.15.29:0/548855204 <== mon.2 v2:172.21.15.152:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f1b4400f3c0 con 0x7f1b54101c30 2024-03-16T11:49:54.549 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:54.549+0000 7f1b5a37a700 1 -- 172.21.15.29:0/548855204 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7f1b5402db70 con 0x7f1b54101c30 2024-03-16T11:49:54.550 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:54.550+0000 7f1b517fa700 1 -- 172.21.15.29:0/548855204 <== mon.2 v2:172.21.15.152:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f1b4409bc60 con 0x7f1b54101c30 2024-03-16T11:49:54.550 INFO:teuthology.orchestra.run.smithi029.stdout:163208757308 2024-03-16T11:49:54.552 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:54.552+0000 7f1b5a37a700 1 -- 172.21.15.29:0/548855204 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f1b3c0a2920 msgr2=0x7f1b3c0a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:49:54.552 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:54.552+0000 7f1b5a37a700 1 --2- 172.21.15.29:0/548855204 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f1b3c0a2920 0x7f1b3c0a4de0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f1b4800c010 tx=0x7f1b4801a040 comp rx=0 tx=0).stop 2024-03-16T11:49:54.552 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:54.552+0000 7f1b5a37a700 1 -- 172.21.15.29:0/548855204 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1b54101c30 msgr2=0x7f1b54197ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:49:54.552 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:54.552+0000 7f1b5a37a700 1 --2- 172.21.15.29:0/548855204 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1b54101c30 0x7f1b54197ac0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f1b4400e9f0 tx=0x7f1b4400ed00 comp rx=0 tx=0).stop 2024-03-16T11:49:54.552 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:54.552+0000 7f1b5a37a700 1 -- 172.21.15.29:0/548855204 shutdown_connections 2024-03-16T11:49:54.552 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:54.552+0000 7f1b5a37a700 1 --2- 172.21.15.29:0/548855204 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1b54106550 0x7f1b5419c3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:49:54.552 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:54.552+0000 7f1b5a37a700 1 --2- 172.21.15.29:0/548855204 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1b54101c30 0x7f1b54197ac0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:49:54.553 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:54.552+0000 7f1b5a37a700 1 --2- 172.21.15.29:0/548855204 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f1b3c0a2920 0x7f1b3c0a4de0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:49:54.553 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:54.552+0000 7f1b5a37a700 1 --2- 172.21.15.29:0/548855204 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1b541025e0 0x7f1b54198000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:49:54.553 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:54.552+0000 7f1b5a37a700 1 -- 172.21.15.29:0/548855204 >> 172.21.15.29:0/548855204 conn(0x7f1b540777d0 msgr2=0x7f1b540febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:49:54.553 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:54.552+0000 7f1b5a37a700 1 -- 172.21.15.29:0/548855204 shutdown_connections 2024-03-16T11:49:54.553 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:54.552+0000 7f1b5a37a700 1 -- 172.21.15.29:0/548855204 wait complete. 2024-03-16T11:49:54.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:49:54 smithi029 ceph-mon[31975]: from='client.? 172.21.15.29:0/1387357590' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-16T11:49:54.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:49:54 smithi029 ceph-mon[31975]: from='client.? 172.21.15.29:0/548855204' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-16T11:49:55.040 INFO:tasks.cephadm.ceph_manager.ceph:need seq 219043332139 got 219043332145 for osd.7 2024-03-16T11:49:55.040 DEBUG:teuthology.parallel:result is None 2024-03-16T11:49:55.042 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:49:54 smithi152 ceph-mon[36291]: from='client.? 172.21.15.29:0/1387357590' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-16T11:49:55.043 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:49:54 smithi152 ceph-mon[36291]: from='client.? 172.21.15.29:0/548855204' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-16T11:49:55.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:49:54 smithi154 ceph-mon[36320]: from='client.? 172.21.15.29:0/1387357590' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-16T11:49:55.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:49:54 smithi154 ceph-mon[36320]: from='client.? 172.21.15.29:0/548855204' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-16T11:49:55.515 INFO:tasks.cephadm.ceph_manager.ceph:need seq 287762808866 got 287762808871 for osd.9 2024-03-16T11:49:55.516 DEBUG:teuthology.parallel:result is None 2024-03-16T11:49:55.808 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:49:55 smithi029 ceph-mon[31975]: pgmap v331: 97 pgs: 97 active+clean; 640 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-16T11:49:56.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:49:55 smithi154 ceph-mon[36320]: pgmap v331: 97 pgs: 97 active+clean; 640 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-16T11:49:56.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:49:55 smithi152 ceph-mon[36291]: pgmap v331: 97 pgs: 97 active+clean; 640 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-16T11:49:56.151 INFO:tasks.cephadm.ceph_manager.ceph:need seq 163208757301 got 163208757308 for osd.5 2024-03-16T11:49:56.151 DEBUG:teuthology.parallel:result is None 2024-03-16T11:49:56.913 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:56.857+0000 7fdc66c7e700 1 -- 172.21.15.29:0/3222703660 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdc6006b5e0 msgr2=0x7fdc60106f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:49:56.913 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:56.857+0000 7fdc66c7e700 1 --2- 172.21.15.29:0/3222703660 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdc6006b5e0 0x7fdc60106f20 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fdc4c0099c0 tx=0x7fdc4c009cd0 comp rx=0 tx=0).stop 2024-03-16T11:49:56.913 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:56.857+0000 7fdc66c7e700 1 -- 172.21.15.29:0/3222703660 shutdown_connections 2024-03-16T11:49:56.913 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:56.857+0000 7fdc66c7e700 1 --2- 172.21.15.29:0/3222703660 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdc60107650 0x7fdc6010f1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:49:56.913 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:56.857+0000 7fdc66c7e700 1 --2- 172.21.15.29:0/3222703660 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fdc6006ac30 0x7fdc6006b010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:49:56.913 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:56.857+0000 7fdc66c7e700 1 --2- 172.21.15.29:0/3222703660 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdc6006b5e0 0x7fdc60106f20 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:49:56.913 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:56.857+0000 7fdc66c7e700 1 -- 172.21.15.29:0/3222703660 >> 172.21.15.29:0/3222703660 conn(0x7fdc600771f0 msgr2=0x7fdc60077600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:49:56.913 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:56.857+0000 7fdc66c7e700 1 -- 172.21.15.29:0/3222703660 shutdown_connections 2024-03-16T11:49:56.913 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:56.857+0000 7fdc66c7e700 1 -- 172.21.15.29:0/3222703660 wait complete. 2024-03-16T11:49:56.914 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:56.858+0000 7fdc66c7e700 1 Processor -- start 2024-03-16T11:49:56.914 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:56.858+0000 7fdc66c7e700 1 -- start start 2024-03-16T11:49:56.914 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:56.858+0000 7fdc66c7e700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdc6006ac30 0x7fdc6019e750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:49:56.914 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:56.858+0000 7fdc66c7e700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdc6006b5e0 0x7fdc6019ec90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:49:56.914 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:56.858+0000 7fdc66c7e700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fdc60107650 0x7fdc60198910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:49:56.914 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:56.858+0000 7fdc66c7e700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fdc601125f0 con 0x7fdc6006b5e0 2024-03-16T11:49:56.914 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:56.858+0000 7fdc66c7e700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fdc60112470 con 0x7fdc60107650 2024-03-16T11:49:56.914 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:56.858+0000 7fdc66c7e700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fdc60112770 con 0x7fdc6006ac30 2024-03-16T11:49:56.914 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:56.858+0000 7fdc5ffff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdc6006b5e0 0x7fdc6019ec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:49:56.914 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:56.858+0000 7fdc5ffff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdc6006b5e0 0x7fdc6019ec90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60924/0 (socket says 172.21.15.29:60924) 2024-03-16T11:49:56.914 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:56.858+0000 7fdc5ffff700 1 -- 172.21.15.29:0/2189000925 learned_addr learned my addr 172.21.15.29:0/2189000925 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:49:56.914 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:56.858+0000 7fdc64a1a700 1 --2- 172.21.15.29:0/2189000925 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdc6006ac30 0x7fdc6019e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:49:56.914 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:56.858+0000 7fdc6521b700 1 --2- 172.21.15.29:0/2189000925 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fdc60107650 0x7fdc60198910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:49:56.914 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:56.858+0000 7fdc5ffff700 1 -- 172.21.15.29:0/2189000925 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdc6006ac30 msgr2=0x7fdc6019e750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:49:56.915 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:56.858+0000 7fdc5ffff700 1 --2- 172.21.15.29:0/2189000925 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdc6006ac30 0x7fdc6019e750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:49:56.915 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:56.858+0000 7fdc5ffff700 1 -- 172.21.15.29:0/2189000925 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fdc60107650 msgr2=0x7fdc60198910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:49:56.915 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:56.858+0000 7fdc5ffff700 1 --2- 172.21.15.29:0/2189000925 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fdc60107650 0x7fdc60198910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:49:56.915 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:56.858+0000 7fdc5ffff700 1 -- 172.21.15.29:0/2189000925 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc60199070 con 0x7fdc6006b5e0 2024-03-16T11:49:56.915 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:56.858+0000 7fdc5ffff700 1 --2- 172.21.15.29:0/2189000925 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdc6006b5e0 0x7fdc6019ec90 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fdc4c0098e0 tx=0x7fdc4c003e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:49:56.915 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:56.859+0000 7fdc5dffb700 1 -- 172.21.15.29:0/2189000925 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc4c009e60 con 0x7fdc6006b5e0 2024-03-16T11:49:56.915 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:56.859+0000 7fdc5dffb700 1 -- 172.21.15.29:0/2189000925 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdc4c003680 con 0x7fdc6006b5e0 2024-03-16T11:49:56.915 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:56.859+0000 7fdc66c7e700 1 -- 172.21.15.29:0/2189000925 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdc60199300 con 0x7fdc6006b5e0 2024-03-16T11:49:56.915 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:56.859+0000 7fdc5dffb700 1 -- 172.21.15.29:0/2189000925 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc4c00fd80 con 0x7fdc6006b5e0 2024-03-16T11:49:56.915 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:56.859+0000 7fdc66c7e700 1 -- 172.21.15.29:0/2189000925 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdc601a5620 con 0x7fdc6006b5e0 2024-03-16T11:49:56.915 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:56.860+0000 7fdc66c7e700 1 -- 172.21.15.29:0/2189000925 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdc60103a00 con 0x7fdc6006b5e0 2024-03-16T11:49:56.915 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:56.861+0000 7fdc5dffb700 1 -- 172.21.15.29:0/2189000925 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7fdc4c00f460 con 0x7fdc6006b5e0 2024-03-16T11:49:56.915 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:56.862+0000 7fdc5dffb700 1 --2- 172.21.15.29:0/2189000925 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fdc500a29a0 0x7fdc500a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:49:56.915 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:56.862+0000 7fdc64a1a700 1 --2- 172.21.15.29:0/2189000925 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fdc500a29a0 0x7fdc500a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:49:56.916 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:56.862+0000 7fdc5dffb700 1 -- 172.21.15.29:0/2189000925 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8211+0+0 (secure 0 0 0) 0x7fdc4c0cf990 con 0x7fdc6006b5e0 2024-03-16T11:49:56.916 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:56.862+0000 7fdc64a1a700 1 --2- 172.21.15.29:0/2189000925 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fdc500a29a0 0x7fdc500a4e60 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fdc60199a50 tx=0x7fdc54009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:49:56.916 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:56.867+0000 7fdc5dffb700 1 -- 172.21.15.29:0/2189000925 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fdc4c013070 con 0x7fdc6006b5e0 2024-03-16T11:49:57.073 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:57.073+0000 7fdc66c7e700 1 -- 172.21.15.29:0/2189000925 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7fdc6002db70 con 0x7fdc6006b5e0 2024-03-16T11:49:57.073 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:57.073+0000 7fdc5dffb700 1 -- 172.21.15.29:0/2189000925 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fdc4c09c970 con 0x7fdc6006b5e0 2024-03-16T11:49:57.073 INFO:teuthology.orchestra.run.smithi029.stdout:77309411404 2024-03-16T11:49:57.075 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:57.076+0000 7fdc66c7e700 1 -- 172.21.15.29:0/2189000925 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fdc500a29a0 msgr2=0x7fdc500a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:49:57.075 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:57.076+0000 7fdc66c7e700 1 --2- 172.21.15.29:0/2189000925 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fdc500a29a0 0x7fdc500a4e60 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fdc60199a50 tx=0x7fdc54009260 comp rx=0 tx=0).stop 2024-03-16T11:49:57.075 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:57.076+0000 7fdc66c7e700 1 -- 172.21.15.29:0/2189000925 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdc6006b5e0 msgr2=0x7fdc6019ec90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:49:57.075 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:57.076+0000 7fdc66c7e700 1 --2- 172.21.15.29:0/2189000925 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdc6006b5e0 0x7fdc6019ec90 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fdc4c0098e0 tx=0x7fdc4c003e00 comp rx=0 tx=0).stop 2024-03-16T11:49:57.076 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:57.076+0000 7fdc66c7e700 1 -- 172.21.15.29:0/2189000925 shutdown_connections 2024-03-16T11:49:57.076 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:57.076+0000 7fdc66c7e700 1 --2- 172.21.15.29:0/2189000925 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fdc60107650 0x7fdc60198910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:49:57.076 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:57.076+0000 7fdc66c7e700 1 --2- 172.21.15.29:0/2189000925 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fdc500a29a0 0x7fdc500a4e60 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:49:57.076 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:57.076+0000 7fdc66c7e700 1 --2- 172.21.15.29:0/2189000925 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdc6006ac30 0x7fdc6019e750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:49:57.076 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:57.076+0000 7fdc66c7e700 1 --2- 172.21.15.29:0/2189000925 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdc6006b5e0 0x7fdc6019ec90 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:49:57.076 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:57.076+0000 7fdc66c7e700 1 -- 172.21.15.29:0/2189000925 >> 172.21.15.29:0/2189000925 conn(0x7fdc600771f0 msgr2=0x7fdc600ff5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:49:57.076 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:57.076+0000 7fdc66c7e700 1 -- 172.21.15.29:0/2189000925 shutdown_connections 2024-03-16T11:49:57.076 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:57.076+0000 7fdc66c7e700 1 -- 172.21.15.29:0/2189000925 wait complete. 2024-03-16T11:49:57.337 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:49:57 smithi029 ceph-mon[31975]: from='client.? 172.21.15.29:0/2189000925' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-16T11:49:57.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:49:57 smithi152 ceph-mon[36291]: from='client.? 172.21.15.29:0/2189000925' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-16T11:49:57.545 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:49:57 smithi154 ceph-mon[36320]: from='client.? 172.21.15.29:0/2189000925' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-16T11:49:58.089 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411399 got 77309411404 for osd.2 2024-03-16T11:49:58.089 DEBUG:teuthology.parallel:result is None 2024-03-16T11:49:58.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:49:58 smithi029 ceph-mon[31975]: pgmap v332: 97 pgs: 97 active+clean; 640 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-16T11:49:58.545 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:49:58 smithi154 ceph-mon[36320]: pgmap v332: 97 pgs: 97 active+clean; 640 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-16T11:49:58.627 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:49:58 smithi152 ceph-mon[36291]: pgmap v332: 97 pgs: 97 active+clean; 640 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-16T11:49:59.180 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:59.181+0000 7fa5b855f700 1 -- 172.21.15.29:0/4051315596 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa5b0108300 msgr2=0x7fa5b01086e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:49:59.180 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:59.181+0000 7fa5b855f700 1 --2- 172.21.15.29:0/4051315596 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa5b0108300 0x7fa5b01086e0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fa5ac0099c0 tx=0x7fa5ac009cd0 comp rx=0 tx=0).stop 2024-03-16T11:49:59.180 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:59.181+0000 7fa5b855f700 1 -- 172.21.15.29:0/4051315596 shutdown_connections 2024-03-16T11:49:59.180 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:59.181+0000 7fa5b855f700 1 --2- 172.21.15.29:0/4051315596 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa5b0102ce0 0x7fa5b010f120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:49:59.181 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:59.181+0000 7fa5b855f700 1 --2- 172.21.15.29:0/4051315596 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa5b0108300 0x7fa5b01086e0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:49:59.181 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:59.181+0000 7fa5b855f700 1 --2- 172.21.15.29:0/4051315596 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa5b0102320 0x7fa5b01027a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:49:59.181 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:59.181+0000 7fa5b855f700 1 -- 172.21.15.29:0/4051315596 >> 172.21.15.29:0/4051315596 conn(0x7fa5b0079400 msgr2=0x7fa5b0079810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:49:59.181 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:59.181+0000 7fa5b855f700 1 -- 172.21.15.29:0/4051315596 shutdown_connections 2024-03-16T11:49:59.181 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:59.181+0000 7fa5b855f700 1 -- 172.21.15.29:0/4051315596 wait complete. 2024-03-16T11:49:59.183 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:59.181+0000 7fa5b855f700 1 Processor -- start 2024-03-16T11:49:59.183 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:59.182+0000 7fa5b855f700 1 -- start start 2024-03-16T11:49:59.183 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:59.182+0000 7fa5b855f700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa5b0102320 0x7fa5b00745d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:49:59.183 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:59.182+0000 7fa5b855f700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa5b0102ce0 0x7fa5b0074b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:49:59.183 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:59.182+0000 7fa5b855f700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa5b0108300 0x7fa5b0075050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:49:59.183 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:59.182+0000 7fa5b855f700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fa5b01125b0 con 0x7fa5b0108300 2024-03-16T11:49:59.183 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:59.182+0000 7fa5b855f700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fa5b0112430 con 0x7fa5b0102320 2024-03-16T11:49:59.183 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:59.182+0000 7fa5b855f700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fa5b0112730 con 0x7fa5b0102ce0 2024-03-16T11:49:59.183 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:59.182+0000 7fa5b6afc700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa5b0108300 0x7fa5b0075050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:49:59.183 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:59.182+0000 7fa5b62fb700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa5b0102320 0x7fa5b00745d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:49:59.183 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:59.182+0000 7fa5b6afc700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa5b0108300 0x7fa5b0075050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60944/0 (socket says 172.21.15.29:60944) 2024-03-16T11:49:59.183 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:59.182+0000 7fa5b62fb700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa5b0102320 0x7fa5b00745d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.29:42124/0 (socket says 172.21.15.29:42124) 2024-03-16T11:49:59.184 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:59.182+0000 7fa5b6afc700 1 -- 172.21.15.29:0/789101752 learned_addr learned my addr 172.21.15.29:0/789101752 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:49:59.184 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:59.182+0000 7fa5b5afa700 1 --2- 172.21.15.29:0/789101752 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa5b0102ce0 0x7fa5b0074b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:49:59.184 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:59.182+0000 7fa5b5afa700 1 -- 172.21.15.29:0/789101752 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa5b0102320 msgr2=0x7fa5b00745d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:49:59.184 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:59.182+0000 7fa5b5afa700 1 --2- 172.21.15.29:0/789101752 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa5b0102320 0x7fa5b00745d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:49:59.184 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:59.182+0000 7fa5b5afa700 1 -- 172.21.15.29:0/789101752 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa5b0108300 msgr2=0x7fa5b0075050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:49:59.184 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:59.182+0000 7fa5b5afa700 1 --2- 172.21.15.29:0/789101752 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa5b0108300 0x7fa5b0075050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:49:59.184 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:59.182+0000 7fa5b5afa700 1 -- 172.21.15.29:0/789101752 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5b01a6f90 con 0x7fa5b0102ce0 2024-03-16T11:49:59.184 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:59.182+0000 7fa5b6afc700 1 --2- 172.21.15.29:0/789101752 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa5b0108300 0x7fa5b0075050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:49:59.184 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:59.182+0000 7fa5b62fb700 1 --2- 172.21.15.29:0/789101752 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa5b0102320 0x7fa5b00745d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:49:59.184 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:59.183+0000 7fa5b5afa700 1 --2- 172.21.15.29:0/789101752 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa5b0102ce0 0x7fa5b0074b10 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fa5a000e9f0 tx=0x7fa5a000ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:49:59.184 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:59.183+0000 7fa5a77fe700 1 -- 172.21.15.29:0/789101752 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5a000c9c0 con 0x7fa5b0102ce0 2024-03-16T11:49:59.184 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:59.183+0000 7fa5b855f700 1 -- 172.21.15.29:0/789101752 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa5b01a7280 con 0x7fa5b0102ce0 2024-03-16T11:49:59.184 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:59.183+0000 7fa5a77fe700 1 -- 172.21.15.29:0/789101752 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa5a000cb60 con 0x7fa5b0102ce0 2024-03-16T11:49:59.185 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:59.183+0000 7fa5a77fe700 1 -- 172.21.15.29:0/789101752 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5a0018680 con 0x7fa5b0102ce0 2024-03-16T11:49:59.185 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:59.183+0000 7fa5b855f700 1 -- 172.21.15.29:0/789101752 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa5b01a7810 con 0x7fa5b0102ce0 2024-03-16T11:49:59.185 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:59.184+0000 7fa5b855f700 1 -- 172.21.15.29:0/789101752 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa5b0068f90 con 0x7fa5b0102ce0 2024-03-16T11:49:59.185 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:59.185+0000 7fa5a77fe700 1 -- 172.21.15.29:0/789101752 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fa5a0047a60 con 0x7fa5b0102ce0 2024-03-16T11:49:59.186 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:59.185+0000 7fa5a77fe700 1 --2- 172.21.15.29:0/789101752 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fa59c0a2890 0x7fa59c0a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:49:59.186 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:59.186+0000 7fa5b62fb700 1 --2- 172.21.15.29:0/789101752 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fa59c0a2890 0x7fa59c0a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:49:59.186 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:59.186+0000 7fa5a77fe700 1 -- 172.21.15.29:0/789101752 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8211+0+0 (secure 0 0 0) 0x7fa5a00ceba0 con 0x7fa5b0102ce0 2024-03-16T11:49:59.186 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:59.186+0000 7fa5b62fb700 1 --2- 172.21.15.29:0/789101752 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fa59c0a2890 0x7fa59c0a4d50 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fa5ac000c00 tx=0x7fa5ac005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:49:59.188 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:59.189+0000 7fa5a77fe700 1 -- 172.21.15.29:0/789101752 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fa5a009bbf0 con 0x7fa5b0102ce0 2024-03-16T11:49:59.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:59.394+0000 7fa5b855f700 1 -- 172.21.15.29:0/789101752 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 11} v 0) -- 0x7fa5b002db70 con 0x7fa5b0102ce0 2024-03-16T11:49:59.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:59.394+0000 7fa5a77fe700 1 -- 172.21.15.29:0/789101752 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 11}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7fa5a009b4f0 con 0x7fa5b0102ce0 2024-03-16T11:49:59.394 INFO:teuthology.orchestra.run.smithi029.stdout:343597383710 2024-03-16T11:49:59.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:59.396+0000 7fa5b855f700 1 -- 172.21.15.29:0/789101752 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fa59c0a2890 msgr2=0x7fa59c0a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:49:59.396 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:59.396+0000 7fa5b855f700 1 --2- 172.21.15.29:0/789101752 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fa59c0a2890 0x7fa59c0a4d50 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fa5ac000c00 tx=0x7fa5ac005fb0 comp rx=0 tx=0).stop 2024-03-16T11:49:59.396 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:59.396+0000 7fa5b855f700 1 -- 172.21.15.29:0/789101752 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa5b0102ce0 msgr2=0x7fa5b0074b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:49:59.396 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:59.396+0000 7fa5b855f700 1 --2- 172.21.15.29:0/789101752 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa5b0102ce0 0x7fa5b0074b10 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fa5a000e9f0 tx=0x7fa5a000ed00 comp rx=0 tx=0).stop 2024-03-16T11:49:59.396 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:59.396+0000 7fa5b855f700 1 -- 172.21.15.29:0/789101752 shutdown_connections 2024-03-16T11:49:59.396 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:59.396+0000 7fa5b855f700 1 --2- 172.21.15.29:0/789101752 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa5b0108300 0x7fa5b0075050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:49:59.396 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:59.396+0000 7fa5b855f700 1 --2- 172.21.15.29:0/789101752 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa5b0102320 0x7fa5b00745d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:49:59.396 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:59.396+0000 7fa5b855f700 1 --2- 172.21.15.29:0/789101752 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fa59c0a2890 0x7fa59c0a4d50 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:49:59.396 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:59.396+0000 7fa5b855f700 1 --2- 172.21.15.29:0/789101752 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa5b0102ce0 0x7fa5b0074b10 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:49:59.396 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:59.396+0000 7fa5b855f700 1 -- 172.21.15.29:0/789101752 >> 172.21.15.29:0/789101752 conn(0x7fa5b0079400 msgr2=0x7fa5b0101020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:49:59.396 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:59.397+0000 7fa5b855f700 1 -- 172.21.15.29:0/789101752 shutdown_connections 2024-03-16T11:49:59.396 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:49:59.397+0000 7fa5b855f700 1 -- 172.21.15.29:0/789101752 wait complete. 2024-03-16T11:50:00.325 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:50:00 smithi029 ceph-mon[31975]: mgrmap e18: x(active, since 8m), standbys: z, y 2024-03-16T11:50:00.326 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:50:00 smithi029 ceph-mon[31975]: from='client.? 172.21.15.29:0/789101752' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-03-16T11:50:00.326 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:50:00 smithi029 ceph-mon[31975]: pgmap v333: 97 pgs: 97 active+clean; 640 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 426 B/s wr, 7 op/s 2024-03-16T11:50:00.326 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:50:00 smithi029 ceph-mon[31975]: overall HEALTH_OK 2024-03-16T11:50:00.545 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:50:00 smithi154 ceph-mon[36320]: mgrmap e18: x(active, since 8m), standbys: z, y 2024-03-16T11:50:00.545 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:50:00 smithi154 ceph-mon[36320]: from='client.? 172.21.15.29:0/789101752' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-03-16T11:50:00.545 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:50:00 smithi154 ceph-mon[36320]: pgmap v333: 97 pgs: 97 active+clean; 640 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 426 B/s wr, 7 op/s 2024-03-16T11:50:00.545 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:50:00 smithi154 ceph-mon[36320]: overall HEALTH_OK 2024-03-16T11:50:00.627 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:50:00 smithi152 ceph-mon[36291]: mgrmap e18: x(active, since 8m), standbys: z, y 2024-03-16T11:50:00.627 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:50:00 smithi152 ceph-mon[36291]: from='client.? 172.21.15.29:0/789101752' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-03-16T11:50:00.627 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:50:00 smithi152 ceph-mon[36291]: pgmap v333: 97 pgs: 97 active+clean; 640 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 426 B/s wr, 7 op/s 2024-03-16T11:50:00.627 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:50:00 smithi152 ceph-mon[36291]: overall HEALTH_OK 2024-03-16T11:50:00.654 INFO:tasks.cephadm.ceph_manager.ceph:need seq 343597383704 got 343597383710 for osd.11 2024-03-16T11:50:00.654 DEBUG:teuthology.parallel:result is None 2024-03-16T11:50:00.654 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-03-16T11:50:00.654 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid d1887f28-e389-11ee-95c9-87774f69a715 -- ceph pg dump --format=json 2024-03-16T11:50:00.930 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/d1887f28-e389-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T11:50:02.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:50:01 smithi154 ceph-mon[36320]: pgmap v334: 97 pgs: 97 active+clean; 640 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 511 B/s wr, 6 op/s 2024-03-16T11:50:02.083 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:50:01 smithi029 ceph-mon[31975]: pgmap v334: 97 pgs: 97 active+clean; 640 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 511 B/s wr, 6 op/s 2024-03-16T11:50:02.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:50:01 smithi152 ceph-mon[36291]: pgmap v334: 97 pgs: 97 active+clean; 640 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 511 B/s wr, 6 op/s 2024-03-16T11:50:02.795 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:02.774+0000 7fbe300c1700 1 -- 172.21.15.29:0/655770208 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbe28106940 msgr2=0x7fbe28106d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:02.795 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:02.774+0000 7fbe300c1700 1 --2- 172.21.15.29:0/655770208 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbe28106940 0x7fbe28106d20 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fbe24009990 tx=0x7fbe24009ca0 comp rx=0 tx=0).stop 2024-03-16T11:50:02.795 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:02.774+0000 7fbe300c1700 1 -- 172.21.15.29:0/655770208 shutdown_connections 2024-03-16T11:50:02.795 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:02.774+0000 7fbe300c1700 1 --2- 172.21.15.29:0/655770208 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbe28101350 0x7fbe2810d810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:02.795 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:02.774+0000 7fbe300c1700 1 --2- 172.21.15.29:0/655770208 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbe28106940 0x7fbe28106d20 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:02.795 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:02.774+0000 7fbe300c1700 1 --2- 172.21.15.29:0/655770208 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbe28100990 0x7fbe28100e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:02.795 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:02.774+0000 7fbe300c1700 1 -- 172.21.15.29:0/655770208 >> 172.21.15.29:0/655770208 conn(0x7fbe280777d0 msgr2=0x7fbe280febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:50:02.795 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:02.774+0000 7fbe300c1700 1 -- 172.21.15.29:0/655770208 shutdown_connections 2024-03-16T11:50:02.795 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:02.774+0000 7fbe300c1700 1 -- 172.21.15.29:0/655770208 wait complete. 2024-03-16T11:50:02.796 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:02.775+0000 7fbe300c1700 1 Processor -- start 2024-03-16T11:50:02.796 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:02.775+0000 7fbe300c1700 1 -- start start 2024-03-16T11:50:02.796 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:02.775+0000 7fbe300c1700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbe28100990 0x7fbe280fccb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:02.796 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:02.775+0000 7fbe300c1700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbe28101350 0x7fbe280fd1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:02.796 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:02.775+0000 7fbe300c1700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbe28106940 0x7fbe280fd730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:02.796 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:02.775+0000 7fbe300c1700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fbe281103e0 con 0x7fbe28106940 2024-03-16T11:50:02.796 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:02.775+0000 7fbe300c1700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fbe28110260 con 0x7fbe28100990 2024-03-16T11:50:02.796 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:02.775+0000 7fbe300c1700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fbe28110560 con 0x7fbe28101350 2024-03-16T11:50:02.796 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:02.775+0000 7fbe2e65e700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbe28106940 0x7fbe280fd730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:02.796 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:02.775+0000 7fbe2d65c700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbe28101350 0x7fbe280fd1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:02.796 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:02.775+0000 7fbe2de5d700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbe28100990 0x7fbe280fccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:02.797 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:02.775+0000 7fbe2d65c700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbe28101350 0x7fbe280fd1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.29:40114/0 (socket says 172.21.15.29:40114) 2024-03-16T11:50:02.797 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:02.775+0000 7fbe2d65c700 1 -- 172.21.15.29:0/1649508010 learned_addr learned my addr 172.21.15.29:0/1649508010 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:50:02.797 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:02.775+0000 7fbe2de5d700 1 -- 172.21.15.29:0/1649508010 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbe28101350 msgr2=0x7fbe280fd1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:02.797 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:02.775+0000 7fbe2de5d700 1 --2- 172.21.15.29:0/1649508010 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbe28101350 0x7fbe280fd1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:02.797 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:02.775+0000 7fbe2de5d700 1 -- 172.21.15.29:0/1649508010 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbe28106940 msgr2=0x7fbe280fd730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:02.797 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:02.775+0000 7fbe2de5d700 1 --2- 172.21.15.29:0/1649508010 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbe28106940 0x7fbe280fd730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:02.797 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:02.775+0000 7fbe2de5d700 1 -- 172.21.15.29:0/1649508010 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe281a6f50 con 0x7fbe28100990 2024-03-16T11:50:02.797 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:02.775+0000 7fbe2d65c700 1 --2- 172.21.15.29:0/1649508010 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbe28101350 0x7fbe280fd1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:50:02.797 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:02.776+0000 7fbe2de5d700 1 --2- 172.21.15.29:0/1649508010 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbe28100990 0x7fbe280fccb0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fbe240059d0 tx=0x7fbe2400bab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:50:02.797 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:02.776+0000 7fbe1effd700 1 -- 172.21.15.29:0/1649508010 <== mon.2 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe24027070 con 0x7fbe28100990 2024-03-16T11:50:02.797 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:02.776+0000 7fbe300c1700 1 -- 172.21.15.29:0/1649508010 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbe281a71e0 con 0x7fbe28100990 2024-03-16T11:50:02.797 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:02.776+0000 7fbe1effd700 1 -- 172.21.15.29:0/1649508010 <== mon.2 v2:172.21.15.152:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbe24003d70 con 0x7fbe28100990 2024-03-16T11:50:02.797 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:02.776+0000 7fbe1effd700 1 -- 172.21.15.29:0/1649508010 <== mon.2 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe24022e30 con 0x7fbe28100990 2024-03-16T11:50:02.798 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:02.776+0000 7fbe300c1700 1 -- 172.21.15.29:0/1649508010 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbe281a7690 con 0x7fbe28100990 2024-03-16T11:50:02.798 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:02.777+0000 7fbe300c1700 1 -- 172.21.15.29:0/1649508010 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbe28068f90 con 0x7fbe28100990 2024-03-16T11:50:02.798 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:02.778+0000 7fbe1effd700 1 -- 172.21.15.29:0/1649508010 <== mon.2 v2:172.21.15.152:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fbe2402a070 con 0x7fbe28100990 2024-03-16T11:50:02.798 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:02.778+0000 7fbe1effd700 1 --2- 172.21.15.29:0/1649508010 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fbe140a2910 0x7fbe140a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:02.798 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:02.778+0000 7fbe2d65c700 1 --2- 172.21.15.29:0/1649508010 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fbe140a2910 0x7fbe140a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:02.798 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:02.778+0000 7fbe1effd700 1 -- 172.21.15.29:0/1649508010 <== mon.2 v2:172.21.15.152:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8211+0+0 (secure 0 0 0) 0x7fbe240cf740 con 0x7fbe28100990 2024-03-16T11:50:02.798 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:02.779+0000 7fbe2d65c700 1 --2- 172.21.15.29:0/1649508010 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fbe140a2910 0x7fbe140a4dd0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fbe28105100 tx=0x7fbe18009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:50:02.798 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:02.787+0000 7fbe1effd700 1 -- 172.21.15.29:0/1649508010 <== mon.2 v2:172.21.15.152:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fbe24013070 con 0x7fbe28100990 2024-03-16T11:50:02.986 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:02.987+0000 7fbe300c1700 1 -- 172.21.15.29:0/1649508010 --> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fbe28062710 con 0x7fbe140a2910 2024-03-16T11:50:02.993 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:02.994+0000 7fbe1effd700 1 -- 172.21.15.29:0/1649508010 <== mgr.14152 v2:172.21.15.29:6800/4192064640 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+259156 (secure 0 0 0) 0x7fbe28062710 con 0x7fbe140a2910 2024-03-16T11:50:02.997 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-16T11:50:02.999 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:02.997+0000 7fbe300c1700 1 -- 172.21.15.29:0/1649508010 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fbe140a2910 msgr2=0x7fbe140a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:02.999 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:02.997+0000 7fbe300c1700 1 --2- 172.21.15.29:0/1649508010 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fbe140a2910 0x7fbe140a4dd0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fbe28105100 tx=0x7fbe18009260 comp rx=0 tx=0).stop 2024-03-16T11:50:02.999 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:02.998+0000 7fbe300c1700 1 -- 172.21.15.29:0/1649508010 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbe28100990 msgr2=0x7fbe280fccb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:03.000 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:02.998+0000 7fbe300c1700 1 --2- 172.21.15.29:0/1649508010 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbe28100990 0x7fbe280fccb0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fbe240059d0 tx=0x7fbe2400bab0 comp rx=0 tx=0).stop 2024-03-16T11:50:03.000 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:02.998+0000 7fbe300c1700 1 -- 172.21.15.29:0/1649508010 shutdown_connections 2024-03-16T11:50:03.000 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:02.998+0000 7fbe300c1700 1 --2- 172.21.15.29:0/1649508010 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbe28106940 0x7fbe280fd730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:03.000 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:02.998+0000 7fbe300c1700 1 --2- 172.21.15.29:0/1649508010 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbe28100990 0x7fbe280fccb0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:03.000 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:02.998+0000 7fbe300c1700 1 --2- 172.21.15.29:0/1649508010 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fbe140a2910 0x7fbe140a4dd0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:03.000 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:02.998+0000 7fbe300c1700 1 --2- 172.21.15.29:0/1649508010 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbe28101350 0x7fbe280fd1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:03.000 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:02.998+0000 7fbe300c1700 1 -- 172.21.15.29:0/1649508010 >> 172.21.15.29:0/1649508010 conn(0x7fbe280777d0 msgr2=0x7fbe280ff130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:50:03.000 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:02.998+0000 7fbe300c1700 1 -- 172.21.15.29:0/1649508010 shutdown_connections 2024-03-16T11:50:03.000 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:02.998+0000 7fbe300c1700 1 -- 172.21.15.29:0/1649508010 wait complete. 2024-03-16T11:50:03.000 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2024-03-16T11:50:03.714 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":334,"stamp":"2024-03-16T11:50:01.638143+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":655407,"num_objects":64,"num_object_clones":0,"num_object_copies":192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":624,"num_read_kb":432,"num_write":347,"num_write_kb":3056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":1241772,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":725,"ondisk_log_size":725,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":373176,"kb_used_data":11524,"kb_used_omap":18,"kb_used_meta":361517,"kb_avail":1124568648,"statfs":{"total":1151940427776,"available":1151558295552,"internally_reserved":0,"allocated":11800576,"data_stored":7433340,"data_compressed":117936,"data_compressed_allocated":147456,"data_compressed_original":1826816,"omap_allocated":19143,"internal_metadata":370193721},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":390,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":67,"num_read_kb":32,"num_write":6,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003384"},"pg_stats":[{"pgid":"3.3f","version":"87'8","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T11:49:57.668966+0000","last_change":"2024-03-16T11:48:38.574718+0000","last_active":"2024-03-16T11:49:57.668966+0000","last_peered":"2024-03-16T11:49:57.668966+0000","last_clean":"2024-03-16T11:49:57.668966+0000","last_became_active":"2024-03-16T11:48:38.574420+0000","last_became_peered":"2024-03-16T11:48:38.574420+0000","last_unstale":"2024-03-16T11:49:57.668966+0000","last_undegraded":"2024-03-16T11:49:57.668966+0000","last_fullsized":"2024-03-16T11:49:57.668966+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:43:56.705135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"87'4","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T11:49:57.664700+0000","last_change":"2024-03-16T11:48:38.572722+0000","last_active":"2024-03-16T11:49:57.664700+0000","last_peered":"2024-03-16T11:49:57.664700+0000","last_clean":"2024-03-16T11:49:57.664700+0000","last_became_active":"2024-03-16T11:48:38.572431+0000","last_became_peered":"2024-03-16T11:48:38.572431+0000","last_unstale":"2024-03-16T11:49:57.664700+0000","last_undegraded":"2024-03-16T11:49:57.664700+0000","last_fullsized":"2024-03-16T11:49:57.664700+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:34:45.070975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"87'7","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T11:49:57.666931+0000","last_change":"2024-03-16T11:48:38.585504+0000","last_active":"2024-03-16T11:49:57.666931+0000","last_peered":"2024-03-16T11:49:57.666931+0000","last_clean":"2024-03-16T11:49:57.666931+0000","last_became_active":"2024-03-16T11:48:38.583879+0000","last_became_peered":"2024-03-16T11:48:38.583879+0000","last_unstale":"2024-03-16T11:49:57.666931+0000","last_undegraded":"2024-03-16T11:49:57.666931+0000","last_fullsized":"2024-03-16T11:49:57.666931+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:08:41.119349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T11:49:28.731858+0000","last_change":"2024-03-16T11:48:38.575773+0000","last_active":"2024-03-16T11:49:28.731858+0000","last_peered":"2024-03-16T11:49:28.731858+0000","last_clean":"2024-03-16T11:49:28.731858+0000","last_became_active":"2024-03-16T11:48:38.574441+0000","last_became_peered":"2024-03-16T11:48:38.574441+0000","last_unstale":"2024-03-16T11:49:28.731858+0000","last_undegraded":"2024-03-16T11:49:28.731858+0000","last_fullsized":"2024-03-16T11:49:28.731858+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:22:56.148164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"87'4","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T11:49:57.649692+0000","last_change":"2024-03-16T11:48:38.575505+0000","last_active":"2024-03-16T11:49:57.649692+0000","last_peered":"2024-03-16T11:49:57.649692+0000","last_clean":"2024-03-16T11:49:57.649692+0000","last_became_active":"2024-03-16T11:48:38.575188+0000","last_became_peered":"2024-03-16T11:48:38.575188+0000","last_unstale":"2024-03-16T11:49:57.649692+0000","last_undegraded":"2024-03-16T11:49:57.649692+0000","last_fullsized":"2024-03-16T11:49:57.649692+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:55:33.625249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"87'6","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T11:49:57.659117+0000","last_change":"2024-03-16T11:48:38.572044+0000","last_active":"2024-03-16T11:49:57.659117+0000","last_peered":"2024-03-16T11:49:57.659117+0000","last_clean":"2024-03-16T11:49:57.659117+0000","last_became_active":"2024-03-16T11:48:38.571725+0000","last_became_peered":"2024-03-16T11:48:38.571725+0000","last_unstale":"2024-03-16T11:49:57.659117+0000","last_undegraded":"2024-03-16T11:49:57.659117+0000","last_fullsized":"2024-03-16T11:49:57.659117+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:05:37.060342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"87'5","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T11:49:57.664109+0000","last_change":"2024-03-16T11:48:38.570332+0000","last_active":"2024-03-16T11:49:57.664109+0000","last_peered":"2024-03-16T11:49:57.664109+0000","last_clean":"2024-03-16T11:49:57.664109+0000","last_became_active":"2024-03-16T11:48:38.570009+0000","last_became_peered":"2024-03-16T11:48:38.570009+0000","last_unstale":"2024-03-16T11:49:57.664109+0000","last_undegraded":"2024-03-16T11:49:57.664109+0000","last_fullsized":"2024-03-16T11:49:57.664109+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:42:31.045749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"87'9","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T11:49:57.670699+0000","last_change":"2024-03-16T11:48:38.571741+0000","last_active":"2024-03-16T11:49:57.670699+0000","last_peered":"2024-03-16T11:49:57.670699+0000","last_clean":"2024-03-16T11:49:57.670699+0000","last_became_active":"2024-03-16T11:48:38.569582+0000","last_became_peered":"2024-03-16T11:48:38.569582+0000","last_unstale":"2024-03-16T11:49:57.670699+0000","last_undegraded":"2024-03-16T11:49:57.670699+0000","last_fullsized":"2024-03-16T11:49:57.670699+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:05:36.302223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"87'11","reported_seq":28,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T11:49:57.666578+0000","last_change":"2024-03-16T11:48:38.576038+0000","last_active":"2024-03-16T11:49:57.666578+0000","last_peered":"2024-03-16T11:49:57.666578+0000","last_clean":"2024-03-16T11:49:57.666578+0000","last_became_active":"2024-03-16T11:48:38.574223+0000","last_became_peered":"2024-03-16T11:48:38.574223+0000","last_unstale":"2024-03-16T11:49:57.666578+0000","last_undegraded":"2024-03-16T11:49:57.666578+0000","last_fullsized":"2024-03-16T11:49:57.666578+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:35:17.236721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"87'16","reported_seq":33,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T11:49:57.658419+0000","last_change":"2024-03-16T11:48:38.572797+0000","last_active":"2024-03-16T11:49:57.658419+0000","last_peered":"2024-03-16T11:49:57.658419+0000","last_clean":"2024-03-16T11:49:57.658419+0000","last_became_active":"2024-03-16T11:48:38.572497+0000","last_became_peered":"2024-03-16T11:48:38.572497+0000","last_unstale":"2024-03-16T11:49:57.658419+0000","last_undegraded":"2024-03-16T11:49:57.658419+0000","last_fullsized":"2024-03-16T11:49:57.658419+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:25:49.138897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"87'6","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T11:49:57.676428+0000","last_change":"2024-03-16T11:48:38.570443+0000","last_active":"2024-03-16T11:49:57.676428+0000","last_peered":"2024-03-16T11:49:57.676428+0000","last_clean":"2024-03-16T11:49:57.676428+0000","last_became_active":"2024-03-16T11:48:38.569984+0000","last_became_peered":"2024-03-16T11:48:38.569984+0000","last_unstale":"2024-03-16T11:49:57.676428+0000","last_undegraded":"2024-03-16T11:49:57.676428+0000","last_fullsized":"2024-03-16T11:49:57.676428+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:47:42.539130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T11:49:28.731859+0000","last_change":"2024-03-16T11:48:38.576106+0000","last_active":"2024-03-16T11:49:28.731859+0000","last_peered":"2024-03-16T11:49:28.731859+0000","last_clean":"2024-03-16T11:49:28.731859+0000","last_became_active":"2024-03-16T11:48:38.575409+0000","last_became_peered":"2024-03-16T11:48:38.575409+0000","last_unstale":"2024-03-16T11:49:28.731859+0000","last_undegraded":"2024-03-16T11:49:28.731859+0000","last_fullsized":"2024-03-16T11:49:28.731859+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:38:33.277722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"87'10","reported_seq":33,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T11:49:57.662256+0000","last_change":"2024-03-16T11:49:28.732730+0000","last_active":"2024-03-16T11:49:57.662256+0000","last_peered":"2024-03-16T11:49:57.662256+0000","last_clean":"2024-03-16T11:49:57.662256+0000","last_became_active":"2024-03-16T11:49:28.732332+0000","last_became_peered":"2024-03-16T11:49:28.732332+0000","last_unstale":"2024-03-16T11:49:57.662256+0000","last_undegraded":"2024-03-16T11:49:57.662256+0000","last_fullsized":"2024-03-16T11:49:57.662256+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:32:23.882235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,9],"acting":[6,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"87'7","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T11:49:57.669216+0000","last_change":"2024-03-16T11:48:38.583598+0000","last_active":"2024-03-16T11:49:57.669216+0000","last_peered":"2024-03-16T11:49:57.669216+0000","last_clean":"2024-03-16T11:49:57.669216+0000","last_became_active":"2024-03-16T11:48:38.583275+0000","last_became_peered":"2024-03-16T11:48:38.583275+0000","last_unstale":"2024-03-16T11:49:57.669216+0000","last_undegraded":"2024-03-16T11:49:57.669216+0000","last_fullsized":"2024-03-16T11:49:57.669216+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:58:48.667075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"87'7","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T11:49:57.667881+0000","last_change":"2024-03-16T11:48:38.580240+0000","last_active":"2024-03-16T11:49:57.667881+0000","last_peered":"2024-03-16T11:49:57.667881+0000","last_clean":"2024-03-16T11:49:57.667881+0000","last_became_active":"2024-03-16T11:48:38.579928+0000","last_became_peered":"2024-03-16T11:48:38.579928+0000","last_unstale":"2024-03-16T11:49:57.667881+0000","last_undegraded":"2024-03-16T11:49:57.667881+0000","last_fullsized":"2024-03-16T11:49:57.667881+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:05:14.006915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"87'9","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T11:49:57.673861+0000","last_change":"2024-03-16T11:48:38.574433+0000","last_active":"2024-03-16T11:49:57.673861+0000","last_peered":"2024-03-16T11:49:57.673861+0000","last_clean":"2024-03-16T11:49:57.673861+0000","last_became_active":"2024-03-16T11:48:38.574104+0000","last_became_peered":"2024-03-16T11:48:38.574104+0000","last_unstale":"2024-03-16T11:49:57.673861+0000","last_undegraded":"2024-03-16T11:49:57.673861+0000","last_fullsized":"2024-03-16T11:49:57.673861+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:42:20.588457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"87'8","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T11:49:57.669023+0000","last_change":"2024-03-16T11:48:38.575505+0000","last_active":"2024-03-16T11:49:57.669023+0000","last_peered":"2024-03-16T11:49:57.669023+0000","last_clean":"2024-03-16T11:49:57.669023+0000","last_became_active":"2024-03-16T11:48:38.575187+0000","last_became_peered":"2024-03-16T11:48:38.575187+0000","last_unstale":"2024-03-16T11:49:57.669023+0000","last_undegraded":"2024-03-16T11:49:57.669023+0000","last_fullsized":"2024-03-16T11:49:57.669023+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:13:06.260535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"87'10","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T11:49:57.646315+0000","last_change":"2024-03-16T11:48:38.568146+0000","last_active":"2024-03-16T11:49:57.646315+0000","last_peered":"2024-03-16T11:49:57.646315+0000","last_clean":"2024-03-16T11:49:57.646315+0000","last_became_active":"2024-03-16T11:48:38.567826+0000","last_became_peered":"2024-03-16T11:48:38.567826+0000","last_unstale":"2024-03-16T11:49:57.646315+0000","last_undegraded":"2024-03-16T11:49:57.646315+0000","last_fullsized":"2024-03-16T11:49:57.646315+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:46:32.993012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T11:49:28.636374+0000","last_change":"2024-03-16T11:48:38.578861+0000","last_active":"2024-03-16T11:49:28.636374+0000","last_peered":"2024-03-16T11:49:28.636374+0000","last_clean":"2024-03-16T11:49:28.636374+0000","last_became_active":"2024-03-16T11:48:38.577028+0000","last_became_peered":"2024-03-16T11:48:38.577028+0000","last_unstale":"2024-03-16T11:49:28.636374+0000","last_undegraded":"2024-03-16T11:49:28.636374+0000","last_fullsized":"2024-03-16T11:49:28.636374+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:55:34.520032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"85'9","reported_seq":29,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T11:49:28.731814+0000","last_change":"2024-03-16T11:48:38.026407+0000","last_active":"2024-03-16T11:49:28.731814+0000","last_peered":"2024-03-16T11:49:28.731814+0000","last_clean":"2024-03-16T11:49:28.731814+0000","last_became_active":"2024-03-16T11:48:38.026083+0000","last_became_peered":"2024-03-16T11:48:38.026083+0000","last_unstale":"2024-03-16T11:49:28.731814+0000","last_undegraded":"2024-03-16T11:49:28.731814+0000","last_fullsized":"2024-03-16T11:49:28.731814+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_clean_scrub_stamp":"2024-03-16T11:48:36.997377+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:01:23.751657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"87'5","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T11:49:57.668917+0000","last_change":"2024-03-16T11:48:38.573658+0000","last_active":"2024-03-16T11:49:57.668917+0000","last_peered":"2024-03-16T11:49:57.668917+0000","last_clean":"2024-03-16T11:49:57.668917+0000","last_became_active":"2024-03-16T11:48:38.572747+0000","last_became_peered":"2024-03-16T11:48:38.572747+0000","last_unstale":"2024-03-16T11:49:57.668917+0000","last_undegraded":"2024-03-16T11:49:57.668917+0000","last_fullsized":"2024-03-16T11:49:57.668917+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:09:11.128269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"85'2","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T11:49:28.636450+0000","last_change":"2024-03-16T11:48:38.027088+0000","last_active":"2024-03-16T11:49:28.636450+0000","last_peered":"2024-03-16T11:49:28.636450+0000","last_clean":"2024-03-16T11:49:28.636450+0000","last_became_active":"2024-03-16T11:48:38.026730+0000","last_became_peered":"2024-03-16T11:48:38.026730+0000","last_unstale":"2024-03-16T11:49:28.636450+0000","last_undegraded":"2024-03-16T11:49:28.636450+0000","last_fullsized":"2024-03-16T11:49:28.636450+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_clean_scrub_stamp":"2024-03-16T11:48:36.997377+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:23:28.987731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"87'6","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T11:49:57.667462+0000","last_change":"2024-03-16T11:48:38.583730+0000","last_active":"2024-03-16T11:49:57.667462+0000","last_peered":"2024-03-16T11:49:57.667462+0000","last_clean":"2024-03-16T11:49:57.667462+0000","last_became_active":"2024-03-16T11:48:38.583323+0000","last_became_peered":"2024-03-16T11:48:38.583323+0000","last_unstale":"2024-03-16T11:49:57.667462+0000","last_undegraded":"2024-03-16T11:49:57.667462+0000","last_fullsized":"2024-03-16T11:49:57.667462+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:35:20.923176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"85'4","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T11:49:28.643436+0000","last_change":"2024-03-16T11:49:28.642975+0000","last_active":"2024-03-16T11:49:28.643436+0000","last_peered":"2024-03-16T11:49:28.643436+0000","last_clean":"2024-03-16T11:49:28.643436+0000","last_became_active":"2024-03-16T11:49:28.637997+0000","last_became_peered":"2024-03-16T11:49:28.637997+0000","last_unstale":"2024-03-16T11:49:28.643436+0000","last_undegraded":"2024-03-16T11:49:28.643436+0000","last_fullsized":"2024-03-16T11:49:28.643436+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_clean_scrub_stamp":"2024-03-16T11:48:36.997377+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:24:15.401309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1c","version":"87'9","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T11:49:57.677119+0000","last_change":"2024-03-16T11:48:38.581452+0000","last_active":"2024-03-16T11:49:57.677119+0000","last_peered":"2024-03-16T11:49:57.677119+0000","last_clean":"2024-03-16T11:49:57.677119+0000","last_became_active":"2024-03-16T11:48:38.579305+0000","last_became_peered":"2024-03-16T11:48:38.579305+0000","last_unstale":"2024-03-16T11:49:57.677119+0000","last_undegraded":"2024-03-16T11:49:57.677119+0000","last_fullsized":"2024-03-16T11:49:57.677119+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:57:43.812783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T11:49:28.631814+0000","last_change":"2024-03-16T11:48:38.023747+0000","last_active":"2024-03-16T11:49:28.631814+0000","last_peered":"2024-03-16T11:49:28.631814+0000","last_clean":"2024-03-16T11:49:28.631814+0000","last_became_active":"2024-03-16T11:48:38.023396+0000","last_became_peered":"2024-03-16T11:48:38.023396+0000","last_unstale":"2024-03-16T11:49:28.631814+0000","last_undegraded":"2024-03-16T11:49:28.631814+0000","last_fullsized":"2024-03-16T11:49:28.631814+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_clean_scrub_stamp":"2024-03-16T11:48:36.997377+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:42:22.456357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"87'15","reported_seq":32,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T11:49:57.663227+0000","last_change":"2024-03-16T11:48:38.572157+0000","last_active":"2024-03-16T11:49:57.663227+0000","last_peered":"2024-03-16T11:49:57.663227+0000","last_clean":"2024-03-16T11:49:57.663227+0000","last_became_active":"2024-03-16T11:48:38.571629+0000","last_became_peered":"2024-03-16T11:48:38.571629+0000","last_unstale":"2024-03-16T11:49:57.663227+0000","last_undegraded":"2024-03-16T11:49:57.663227+0000","last_fullsized":"2024-03-16T11:49:57.663227+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:47:39.353965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"87'7","reported_seq":121,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T11:49:59.674211+0000","last_change":"2024-03-16T11:49:28.644872+0000","last_active":"2024-03-16T11:49:59.674211+0000","last_peered":"2024-03-16T11:49:59.674211+0000","last_clean":"2024-03-16T11:49:59.674211+0000","last_became_active":"2024-03-16T11:49:28.638836+0000","last_became_peered":"2024-03-16T11:49:28.638836+0000","last_unstale":"2024-03-16T11:49:59.674211+0000","last_undegraded":"2024-03-16T11:49:59.674211+0000","last_fullsized":"2024-03-16T11:49:59.674211+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_clean_scrub_stamp":"2024-03-16T11:48:36.997377+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:38:42.689838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1154,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":82,"num_read_kb":4,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1013,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"87'7","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T11:49:57.662375+0000","last_change":"2024-03-16T11:48:38.574227+0000","last_active":"2024-03-16T11:49:57.662375+0000","last_peered":"2024-03-16T11:49:57.662375+0000","last_clean":"2024-03-16T11:49:57.662375+0000","last_became_active":"2024-03-16T11:48:38.571938+0000","last_became_peered":"2024-03-16T11:48:38.571938+0000","last_unstale":"2024-03-16T11:49:57.662375+0000","last_undegraded":"2024-03-16T11:49:57.662375+0000","last_fullsized":"2024-03-16T11:49:57.662375+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:50:35.992068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"85'3","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T11:49:28.634815+0000","last_change":"2024-03-16T11:48:38.024448+0000","last_active":"2024-03-16T11:49:28.634815+0000","last_peered":"2024-03-16T11:49:28.634815+0000","last_clean":"2024-03-16T11:49:28.634815+0000","last_became_active":"2024-03-16T11:48:38.024109+0000","last_became_peered":"2024-03-16T11:48:38.024109+0000","last_unstale":"2024-03-16T11:49:28.634815+0000","last_undegraded":"2024-03-16T11:49:28.634815+0000","last_fullsized":"2024-03-16T11:49:28.634815+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_clean_scrub_stamp":"2024-03-16T11:48:36.997377+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:37:49.521297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"87'8","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T11:49:57.665301+0000","last_change":"2024-03-16T11:48:38.577722+0000","last_active":"2024-03-16T11:49:57.665301+0000","last_peered":"2024-03-16T11:49:57.665301+0000","last_clean":"2024-03-16T11:49:57.665301+0000","last_became_active":"2024-03-16T11:48:38.577223+0000","last_became_peered":"2024-03-16T11:48:38.577223+0000","last_unstale":"2024-03-16T11:49:57.665301+0000","last_undegraded":"2024-03-16T11:49:57.665301+0000","last_fullsized":"2024-03-16T11:49:57.665301+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:24:05.892850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"85'2","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T11:49:28.635314+0000","last_change":"2024-03-16T11:48:38.029681+0000","last_active":"2024-03-16T11:49:28.635314+0000","last_peered":"2024-03-16T11:49:28.635314+0000","last_clean":"2024-03-16T11:49:28.635314+0000","last_became_active":"2024-03-16T11:48:38.029261+0000","last_became_peered":"2024-03-16T11:48:38.029261+0000","last_unstale":"2024-03-16T11:49:28.635314+0000","last_undegraded":"2024-03-16T11:49:28.635314+0000","last_fullsized":"2024-03-16T11:49:28.635314+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_clean_scrub_stamp":"2024-03-16T11:48:36.997377+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:57:51.155337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"87'8","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T11:49:57.671216+0000","last_change":"2024-03-16T11:48:38.570655+0000","last_active":"2024-03-16T11:49:57.671216+0000","last_peered":"2024-03-16T11:49:57.671216+0000","last_clean":"2024-03-16T11:49:57.671216+0000","last_became_active":"2024-03-16T11:48:38.569143+0000","last_became_peered":"2024-03-16T11:48:38.569143+0000","last_unstale":"2024-03-16T11:49:57.671216+0000","last_undegraded":"2024-03-16T11:49:57.671216+0000","last_fullsized":"2024-03-16T11:49:57.671216+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:51:51.449168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"85'1","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T11:49:28.634042+0000","last_change":"2024-03-16T11:48:38.027074+0000","last_active":"2024-03-16T11:49:28.634042+0000","last_peered":"2024-03-16T11:49:28.634042+0000","last_clean":"2024-03-16T11:49:28.634042+0000","last_became_active":"2024-03-16T11:48:38.026749+0000","last_became_peered":"2024-03-16T11:48:38.026749+0000","last_unstale":"2024-03-16T11:49:28.634042+0000","last_undegraded":"2024-03-16T11:49:28.634042+0000","last_fullsized":"2024-03-16T11:49:28.634042+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_clean_scrub_stamp":"2024-03-16T11:48:36.997377+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:14:26.801688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"87'11","reported_seq":28,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T11:49:57.654587+0000","last_change":"2024-03-16T11:48:38.575699+0000","last_active":"2024-03-16T11:49:57.654587+0000","last_peered":"2024-03-16T11:49:57.654587+0000","last_clean":"2024-03-16T11:49:57.654587+0000","last_became_active":"2024-03-16T11:48:38.575401+0000","last_became_peered":"2024-03-16T11:48:38.575401+0000","last_unstale":"2024-03-16T11:49:57.654587+0000","last_undegraded":"2024-03-16T11:49:57.654587+0000","last_fullsized":"2024-03-16T11:49:57.654587+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:48:06.289779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"87'4","reported_seq":100,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T11:49:57.670996+0000","last_change":"2024-03-16T11:48:38.026488+0000","last_active":"2024-03-16T11:49:57.670996+0000","last_peered":"2024-03-16T11:49:57.670996+0000","last_clean":"2024-03-16T11:49:57.670996+0000","last_became_active":"2024-03-16T11:48:38.026101+0000","last_became_peered":"2024-03-16T11:48:38.026101+0000","last_unstale":"2024-03-16T11:49:57.670996+0000","last_undegraded":"2024-03-16T11:49:57.670996+0000","last_fullsized":"2024-03-16T11:49:57.670996+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_clean_scrub_stamp":"2024-03-16T11:48:36.997377+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:37:29.788307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":77,"num_read_kb":77,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"87'10","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T11:49:57.675842+0000","last_change":"2024-03-16T11:48:38.578081+0000","last_active":"2024-03-16T11:49:57.675842+0000","last_peered":"2024-03-16T11:49:57.675842+0000","last_clean":"2024-03-16T11:49:57.675842+0000","last_became_active":"2024-03-16T11:48:38.577742+0000","last_became_peered":"2024-03-16T11:48:38.577742+0000","last_unstale":"2024-03-16T11:49:57.675842+0000","last_undegraded":"2024-03-16T11:49:57.675842+0000","last_fullsized":"2024-03-16T11:49:57.675842+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:58:41.479658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"85'2","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T11:49:28.632342+0000","last_change":"2024-03-16T11:48:38.026891+0000","last_active":"2024-03-16T11:49:28.632342+0000","last_peered":"2024-03-16T11:49:28.632342+0000","last_clean":"2024-03-16T11:49:28.632342+0000","last_became_active":"2024-03-16T11:48:38.026397+0000","last_became_peered":"2024-03-16T11:48:38.026397+0000","last_unstale":"2024-03-16T11:49:28.632342+0000","last_undegraded":"2024-03-16T11:49:28.632342+0000","last_fullsized":"2024-03-16T11:49:28.632342+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_clean_scrub_stamp":"2024-03-16T11:48:36.997377+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:02:01.018103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"87'9","reported_seq":32,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T11:49:57.668593+0000","last_change":"2024-03-16T11:49:28.638309+0000","last_active":"2024-03-16T11:49:57.668593+0000","last_peered":"2024-03-16T11:49:57.668593+0000","last_clean":"2024-03-16T11:49:57.668593+0000","last_became_active":"2024-03-16T11:49:28.637786+0000","last_became_peered":"2024-03-16T11:49:28.637786+0000","last_unstale":"2024-03-16T11:49:57.668593+0000","last_undegraded":"2024-03-16T11:49:57.668593+0000","last_fullsized":"2024-03-16T11:49:57.668593+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:32:43.043447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,5],"acting":[0,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"85'4","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T11:49:28.634965+0000","last_change":"2024-03-16T11:48:38.025103+0000","last_active":"2024-03-16T11:49:28.634965+0000","last_peered":"2024-03-16T11:49:28.634965+0000","last_clean":"2024-03-16T11:49:28.634965+0000","last_became_active":"2024-03-16T11:48:38.024754+0000","last_became_peered":"2024-03-16T11:48:38.024754+0000","last_unstale":"2024-03-16T11:49:28.634965+0000","last_undegraded":"2024-03-16T11:49:28.634965+0000","last_fullsized":"2024-03-16T11:49:28.634965+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_clean_scrub_stamp":"2024-03-16T11:48:36.997377+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:40:10.335339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"87'11","reported_seq":28,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T11:49:57.676305+0000","last_change":"2024-03-16T11:48:38.578305+0000","last_active":"2024-03-16T11:49:57.676305+0000","last_peered":"2024-03-16T11:49:57.676305+0000","last_clean":"2024-03-16T11:49:57.676305+0000","last_became_active":"2024-03-16T11:48:38.577971+0000","last_became_peered":"2024-03-16T11:48:38.577971+0000","last_unstale":"2024-03-16T11:49:57.676305+0000","last_undegraded":"2024-03-16T11:49:57.676305+0000","last_fullsized":"2024-03-16T11:49:57.676305+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:04:34.259456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"87'32","reported_seq":232,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T11:49:58.766166+0000","last_change":"2024-03-16T11:49:28.739284+0000","last_active":"2024-03-16T11:49:58.766166+0000","last_peered":"2024-03-16T11:49:58.766166+0000","last_clean":"2024-03-16T11:49:58.766166+0000","last_became_active":"2024-03-16T11:49:28.732510+0000","last_became_peered":"2024-03-16T11:49:28.732510+0000","last_unstale":"2024-03-16T11:49:58.766166+0000","last_undegraded":"2024-03-16T11:49:58.766166+0000","last_fullsized":"2024-03-16T11:49:58.766166+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_clean_scrub_stamp":"2024-03-16T11:48:36.997377+0000","objects_scrubbed":0,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:12:16.286605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":60272,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":161,"num_read_kb":139,"num_write":33,"num_write_kb":73,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":60023,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,4],"acting":[10,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"87'6","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T11:49:57.654869+0000","last_change":"2024-03-16T11:48:38.574404+0000","last_active":"2024-03-16T11:49:57.654869+0000","last_peered":"2024-03-16T11:49:57.654869+0000","last_clean":"2024-03-16T11:49:57.654869+0000","last_became_active":"2024-03-16T11:48:38.572334+0000","last_became_peered":"2024-03-16T11:48:38.572334+0000","last_unstale":"2024-03-16T11:49:57.654869+0000","last_undegraded":"2024-03-16T11:49:57.654869+0000","last_fullsized":"2024-03-16T11:49:57.654869+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:25:37.442152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"85'6","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T11:49:28.636242+0000","last_change":"2024-03-16T11:48:38.025287+0000","last_active":"2024-03-16T11:49:28.636242+0000","last_peered":"2024-03-16T11:49:28.636242+0000","last_clean":"2024-03-16T11:49:28.636242+0000","last_became_active":"2024-03-16T11:48:38.022433+0000","last_became_peered":"2024-03-16T11:48:38.022433+0000","last_unstale":"2024-03-16T11:49:28.636242+0000","last_undegraded":"2024-03-16T11:49:28.636242+0000","last_fullsized":"2024-03-16T11:49:28.636242+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_clean_scrub_stamp":"2024-03-16T11:48:36.997377+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:27:20.573925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"87'9","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T11:49:57.666290+0000","last_change":"2024-03-16T11:48:38.574715+0000","last_active":"2024-03-16T11:49:57.666290+0000","last_peered":"2024-03-16T11:49:57.666290+0000","last_clean":"2024-03-16T11:49:57.666290+0000","last_became_active":"2024-03-16T11:48:38.574456+0000","last_became_peered":"2024-03-16T11:48:38.574456+0000","last_unstale":"2024-03-16T11:49:57.666290+0000","last_undegraded":"2024-03-16T11:49:57.666290+0000","last_fullsized":"2024-03-16T11:49:57.666290+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:57:00.123583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"85'3","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T11:49:28.634578+0000","last_change":"2024-03-16T11:48:38.029557+0000","last_active":"2024-03-16T11:49:28.634578+0000","last_peered":"2024-03-16T11:49:28.634578+0000","last_clean":"2024-03-16T11:49:28.634578+0000","last_became_active":"2024-03-16T11:48:38.029249+0000","last_became_peered":"2024-03-16T11:48:38.029249+0000","last_unstale":"2024-03-16T11:49:28.634578+0000","last_undegraded":"2024-03-16T11:49:28.634578+0000","last_fullsized":"2024-03-16T11:49:28.634578+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_clean_scrub_stamp":"2024-03-16T11:48:36.997377+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:32:41.364467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"87'15","reported_seq":38,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T11:49:57.666085+0000","last_change":"2024-03-16T11:49:28.732430+0000","last_active":"2024-03-16T11:49:57.666085+0000","last_peered":"2024-03-16T11:49:57.666085+0000","last_clean":"2024-03-16T11:49:57.666085+0000","last_became_active":"2024-03-16T11:49:28.732041+0000","last_became_peered":"2024-03-16T11:49:28.732041+0000","last_unstale":"2024-03-16T11:49:57.666085+0000","last_undegraded":"2024-03-16T11:49:57.666085+0000","last_fullsized":"2024-03-16T11:49:57.666085+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:39:20.346214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,6],"acting":[7,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"85'4","reported_seq":101,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T11:49:56.065575+0000","last_change":"2024-03-16T11:48:38.027047+0000","last_active":"2024-03-16T11:49:56.065575+0000","last_peered":"2024-03-16T11:49:56.065575+0000","last_clean":"2024-03-16T11:49:56.065575+0000","last_became_active":"2024-03-16T11:48:38.026733+0000","last_became_peered":"2024-03-16T11:48:38.026733+0000","last_unstale":"2024-03-16T11:49:56.065575+0000","last_undegraded":"2024-03-16T11:49:56.065575+0000","last_fullsized":"2024-03-16T11:49:56.065575+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_clean_scrub_stamp":"2024-03-16T11:48:36.997377+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:27:23.763104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":979,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":79,"num_read_kb":4,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"87'6","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T11:49:57.662127+0000","last_change":"2024-03-16T11:48:38.568802+0000","last_active":"2024-03-16T11:49:57.662127+0000","last_peered":"2024-03-16T11:49:57.662127+0000","last_clean":"2024-03-16T11:49:57.662127+0000","last_became_active":"2024-03-16T11:48:38.568213+0000","last_became_peered":"2024-03-16T11:48:38.568213+0000","last_unstale":"2024-03-16T11:49:57.662127+0000","last_undegraded":"2024-03-16T11:49:57.662127+0000","last_fullsized":"2024-03-16T11:49:57.662127+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:38:23.233563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"85'15","reported_seq":35,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T11:49:28.631457+0000","last_change":"2024-03-16T11:48:38.027596+0000","last_active":"2024-03-16T11:49:28.631457+0000","last_peered":"2024-03-16T11:49:28.631457+0000","last_clean":"2024-03-16T11:49:28.631457+0000","last_became_active":"2024-03-16T11:48:38.027280+0000","last_became_peered":"2024-03-16T11:48:38.027280+0000","last_unstale":"2024-03-16T11:49:28.631457+0000","last_undegraded":"2024-03-16T11:49:28.631457+0000","last_fullsized":"2024-03-16T11:49:28.631457+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_clean_scrub_stamp":"2024-03-16T11:48:36.997377+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:05:26.409613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"87'6","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T11:49:57.662685+0000","last_change":"2024-03-16T11:48:38.571308+0000","last_active":"2024-03-16T11:49:57.662685+0000","last_peered":"2024-03-16T11:49:57.662685+0000","last_clean":"2024-03-16T11:49:57.662685+0000","last_became_active":"2024-03-16T11:48:38.570927+0000","last_became_peered":"2024-03-16T11:48:38.570927+0000","last_unstale":"2024-03-16T11:49:57.662685+0000","last_undegraded":"2024-03-16T11:49:57.662685+0000","last_fullsized":"2024-03-16T11:49:57.662685+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:15:18.190361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"85'2","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T11:49:28.631944+0000","last_change":"2024-03-16T11:48:38.024783+0000","last_active":"2024-03-16T11:49:28.631944+0000","last_peered":"2024-03-16T11:49:28.631944+0000","last_clean":"2024-03-16T11:49:28.631944+0000","last_became_active":"2024-03-16T11:48:38.023990+0000","last_became_peered":"2024-03-16T11:48:38.023990+0000","last_unstale":"2024-03-16T11:49:28.631944+0000","last_undegraded":"2024-03-16T11:49:28.631944+0000","last_fullsized":"2024-03-16T11:49:28.631944+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_clean_scrub_stamp":"2024-03-16T11:48:36.997377+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:42:49.375788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"87'5","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T11:49:57.660582+0000","last_change":"2024-03-16T11:48:38.580019+0000","last_active":"2024-03-16T11:49:57.660582+0000","last_peered":"2024-03-16T11:49:57.660582+0000","last_clean":"2024-03-16T11:49:57.660582+0000","last_became_active":"2024-03-16T11:48:38.579686+0000","last_became_peered":"2024-03-16T11:48:38.579686+0000","last_unstale":"2024-03-16T11:49:57.660582+0000","last_undegraded":"2024-03-16T11:49:57.660582+0000","last_fullsized":"2024-03-16T11:49:57.660582+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:00:21.752048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":19,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T11:49:28.731593+0000","last_change":"2024-03-16T11:48:38.017631+0000","last_active":"2024-03-16T11:49:28.731593+0000","last_peered":"2024-03-16T11:49:28.731593+0000","last_clean":"2024-03-16T11:49:28.731593+0000","last_became_active":"2024-03-16T11:48:38.017218+0000","last_became_peered":"2024-03-16T11:48:38.017218+0000","last_unstale":"2024-03-16T11:49:28.731593+0000","last_undegraded":"2024-03-16T11:49:28.731593+0000","last_fullsized":"2024-03-16T11:49:28.731593+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_clean_scrub_stamp":"2024-03-16T11:48:36.997377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:31:41.378757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"87'5","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T11:49:57.649050+0000","last_change":"2024-03-16T11:48:38.576751+0000","last_active":"2024-03-16T11:49:57.649050+0000","last_peered":"2024-03-16T11:49:57.649050+0000","last_clean":"2024-03-16T11:49:57.649050+0000","last_became_active":"2024-03-16T11:48:38.576442+0000","last_became_peered":"2024-03-16T11:48:38.576442+0000","last_unstale":"2024-03-16T11:49:57.649050+0000","last_undegraded":"2024-03-16T11:49:57.649050+0000","last_fullsized":"2024-03-16T11:49:57.649050+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:49:59.475427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"85'1","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T11:49:28.632243+0000","last_change":"2024-03-16T11:48:38.030159+0000","last_active":"2024-03-16T11:49:28.632243+0000","last_peered":"2024-03-16T11:49:28.632243+0000","last_clean":"2024-03-16T11:49:28.632243+0000","last_became_active":"2024-03-16T11:48:38.029808+0000","last_became_peered":"2024-03-16T11:48:38.029808+0000","last_unstale":"2024-03-16T11:49:28.632243+0000","last_undegraded":"2024-03-16T11:49:28.632243+0000","last_fullsized":"2024-03-16T11:49:28.632243+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_clean_scrub_stamp":"2024-03-16T11:48:36.997377+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:46:54.692837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"87'7","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T11:49:57.651285+0000","last_change":"2024-03-16T11:48:38.574080+0000","last_active":"2024-03-16T11:49:57.651285+0000","last_peered":"2024-03-16T11:49:57.651285+0000","last_clean":"2024-03-16T11:49:57.651285+0000","last_became_active":"2024-03-16T11:48:38.573745+0000","last_became_peered":"2024-03-16T11:48:38.573745+0000","last_unstale":"2024-03-16T11:49:57.651285+0000","last_undegraded":"2024-03-16T11:49:57.651285+0000","last_fullsized":"2024-03-16T11:49:57.651285+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:12:56.154804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"87'8","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T11:49:57.662455+0000","last_change":"2024-03-16T11:48:38.571750+0000","last_active":"2024-03-16T11:49:57.662455+0000","last_peered":"2024-03-16T11:49:57.662455+0000","last_clean":"2024-03-16T11:49:57.662455+0000","last_became_active":"2024-03-16T11:48:38.571273+0000","last_became_peered":"2024-03-16T11:48:38.571273+0000","last_unstale":"2024-03-16T11:49:57.662455+0000","last_undegraded":"2024-03-16T11:49:57.662455+0000","last_fullsized":"2024-03-16T11:49:57.662455+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:59:21.952927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"87'5","reported_seq":100,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T11:49:57.620875+0000","last_change":"2024-03-16T11:48:38.034121+0000","last_active":"2024-03-16T11:49:57.620875+0000","last_peered":"2024-03-16T11:49:57.620875+0000","last_clean":"2024-03-16T11:49:57.620875+0000","last_became_active":"2024-03-16T11:48:38.033777+0000","last_became_peered":"2024-03-16T11:48:38.033777+0000","last_unstale":"2024-03-16T11:49:57.620875+0000","last_undegraded":"2024-03-16T11:49:57.620875+0000","last_fullsized":"2024-03-16T11:49:57.620875+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_clean_scrub_stamp":"2024-03-16T11:48:36.997377+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:51:54.637120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":76,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"87'8","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T11:49:57.670058+0000","last_change":"2024-03-16T11:48:38.580050+0000","last_active":"2024-03-16T11:49:57.670058+0000","last_peered":"2024-03-16T11:49:57.670058+0000","last_clean":"2024-03-16T11:49:57.670058+0000","last_became_active":"2024-03-16T11:48:38.578950+0000","last_became_peered":"2024-03-16T11:48:38.578950+0000","last_unstale":"2024-03-16T11:49:57.670058+0000","last_undegraded":"2024-03-16T11:49:57.670058+0000","last_fullsized":"2024-03-16T11:49:57.670058+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:10:21.717916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"85'3","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T11:49:28.731215+0000","last_change":"2024-03-16T11:48:38.023925+0000","last_active":"2024-03-16T11:49:28.731215+0000","last_peered":"2024-03-16T11:49:28.731215+0000","last_clean":"2024-03-16T11:49:28.731215+0000","last_became_active":"2024-03-16T11:48:38.023424+0000","last_became_peered":"2024-03-16T11:48:38.023424+0000","last_unstale":"2024-03-16T11:49:28.731215+0000","last_undegraded":"2024-03-16T11:49:28.731215+0000","last_fullsized":"2024-03-16T11:49:28.731215+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_clean_scrub_stamp":"2024-03-16T11:48:36.997377+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:01:53.085039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":353,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T11:49:28.632178+0000","last_change":"2024-03-16T11:47:22.001149+0000","last_active":"2024-03-16T11:49:28.632178+0000","last_peered":"2024-03-16T11:49:28.632178+0000","last_clean":"2024-03-16T11:49:28.632178+0000","last_became_active":"2024-03-16T11:47:22.000506+0000","last_became_peered":"2024-03-16T11:47:22.000506+0000","last_unstale":"2024-03-16T11:49:28.632178+0000","last_undegraded":"2024-03-16T11:49:28.632178+0000","last_fullsized":"2024-03-16T11:49:28.632178+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:43:52.376552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:43:52.376552+0000","last_clean_scrub_stamp":"2024-03-16T11:43:52.376552+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:21:18.466389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"87'5","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T11:49:57.659224+0000","last_change":"2024-03-16T11:48:38.577530+0000","last_active":"2024-03-16T11:49:57.659224+0000","last_peered":"2024-03-16T11:49:57.659224+0000","last_clean":"2024-03-16T11:49:57.659224+0000","last_became_active":"2024-03-16T11:48:38.577252+0000","last_became_peered":"2024-03-16T11:48:38.577252+0000","last_unstale":"2024-03-16T11:49:57.659224+0000","last_undegraded":"2024-03-16T11:49:57.659224+0000","last_fullsized":"2024-03-16T11:49:57.659224+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:07:47.809075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"85'3","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T11:49:28.632293+0000","last_change":"2024-03-16T11:48:38.027630+0000","last_active":"2024-03-16T11:49:28.632293+0000","last_peered":"2024-03-16T11:49:28.632293+0000","last_clean":"2024-03-16T11:49:28.632293+0000","last_became_active":"2024-03-16T11:48:38.027105+0000","last_became_peered":"2024-03-16T11:48:38.027105+0000","last_unstale":"2024-03-16T11:49:28.632293+0000","last_undegraded":"2024-03-16T11:49:28.632293+0000","last_fullsized":"2024-03-16T11:49:28.632293+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_clean_scrub_stamp":"2024-03-16T11:48:36.997377+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:24:20.605988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"85'1","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T11:49:28.631706+0000","last_change":"2024-03-16T11:48:38.032419+0000","last_active":"2024-03-16T11:49:28.631706+0000","last_peered":"2024-03-16T11:49:28.631706+0000","last_clean":"2024-03-16T11:49:28.631706+0000","last_became_active":"2024-03-16T11:48:38.032087+0000","last_became_peered":"2024-03-16T11:48:38.032087+0000","last_unstale":"2024-03-16T11:49:28.631706+0000","last_undegraded":"2024-03-16T11:49:28.631706+0000","last_fullsized":"2024-03-16T11:49:28.631706+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_clean_scrub_stamp":"2024-03-16T11:48:36.997377+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:18:54.459321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"87'9","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T11:49:57.661272+0000","last_change":"2024-03-16T11:48:38.568768+0000","last_active":"2024-03-16T11:49:57.661272+0000","last_peered":"2024-03-16T11:49:57.661272+0000","last_clean":"2024-03-16T11:49:57.661272+0000","last_became_active":"2024-03-16T11:48:38.568408+0000","last_became_peered":"2024-03-16T11:48:38.568408+0000","last_unstale":"2024-03-16T11:49:57.661272+0000","last_undegraded":"2024-03-16T11:49:57.661272+0000","last_fullsized":"2024-03-16T11:49:57.661272+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:10:33.293464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"85'8","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T11:49:28.731733+0000","last_change":"2024-03-16T11:48:38.023387+0000","last_active":"2024-03-16T11:49:28.731733+0000","last_peered":"2024-03-16T11:49:28.731733+0000","last_clean":"2024-03-16T11:49:28.731733+0000","last_became_active":"2024-03-16T11:48:38.022916+0000","last_became_peered":"2024-03-16T11:48:38.022916+0000","last_unstale":"2024-03-16T11:49:28.731733+0000","last_undegraded":"2024-03-16T11:49:28.731733+0000","last_fullsized":"2024-03-16T11:49:28.731733+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_clean_scrub_stamp":"2024-03-16T11:48:36.997377+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:25:41.478737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"87'5","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T11:49:57.661696+0000","last_change":"2024-03-16T11:48:38.572243+0000","last_active":"2024-03-16T11:49:57.661696+0000","last_peered":"2024-03-16T11:49:57.661696+0000","last_clean":"2024-03-16T11:49:57.661696+0000","last_became_active":"2024-03-16T11:48:38.571918+0000","last_became_peered":"2024-03-16T11:48:38.571918+0000","last_unstale":"2024-03-16T11:49:57.661696+0000","last_undegraded":"2024-03-16T11:49:57.661696+0000","last_fullsized":"2024-03-16T11:49:57.661696+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:21:45.843298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"85'4","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T11:49:28.731270+0000","last_change":"2024-03-16T11:48:38.022825+0000","last_active":"2024-03-16T11:49:28.731270+0000","last_peered":"2024-03-16T11:49:28.731270+0000","last_clean":"2024-03-16T11:49:28.731270+0000","last_became_active":"2024-03-16T11:48:38.022499+0000","last_became_peered":"2024-03-16T11:48:38.022499+0000","last_unstale":"2024-03-16T11:49:28.731270+0000","last_undegraded":"2024-03-16T11:49:28.731270+0000","last_fullsized":"2024-03-16T11:49:28.731270+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_clean_scrub_stamp":"2024-03-16T11:48:36.997377+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:52:51.810960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"87'7","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T11:49:57.668985+0000","last_change":"2024-03-16T11:48:38.577473+0000","last_active":"2024-03-16T11:49:57.668985+0000","last_peered":"2024-03-16T11:49:57.668985+0000","last_clean":"2024-03-16T11:49:57.668985+0000","last_became_active":"2024-03-16T11:48:38.577121+0000","last_became_peered":"2024-03-16T11:48:38.577121+0000","last_unstale":"2024-03-16T11:49:57.668985+0000","last_undegraded":"2024-03-16T11:49:57.668985+0000","last_fullsized":"2024-03-16T11:49:57.668985+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:21:06.905785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"85'2","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T11:49:28.637444+0000","last_change":"2024-03-16T11:48:38.020324+0000","last_active":"2024-03-16T11:49:28.637444+0000","last_peered":"2024-03-16T11:49:28.637444+0000","last_clean":"2024-03-16T11:49:28.637444+0000","last_became_active":"2024-03-16T11:48:38.019692+0000","last_became_peered":"2024-03-16T11:48:38.019692+0000","last_unstale":"2024-03-16T11:49:28.637444+0000","last_undegraded":"2024-03-16T11:49:28.637444+0000","last_fullsized":"2024-03-16T11:49:28.637444+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_clean_scrub_stamp":"2024-03-16T11:48:36.997377+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:42:51.709046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"87'2","reported_seq":19,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T11:49:57.623612+0000","last_change":"2024-03-16T11:48:38.570911+0000","last_active":"2024-03-16T11:49:57.623612+0000","last_peered":"2024-03-16T11:49:57.623612+0000","last_clean":"2024-03-16T11:49:57.623612+0000","last_became_active":"2024-03-16T11:48:38.570288+0000","last_became_peered":"2024-03-16T11:48:38.570288+0000","last_unstale":"2024-03-16T11:49:57.623612+0000","last_undegraded":"2024-03-16T11:49:57.623612+0000","last_fullsized":"2024-03-16T11:49:57.623612+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:34:20.516104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"85'3","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T11:49:28.635756+0000","last_change":"2024-03-16T11:48:38.021817+0000","last_active":"2024-03-16T11:49:28.635756+0000","last_peered":"2024-03-16T11:49:28.635756+0000","last_clean":"2024-03-16T11:49:28.635756+0000","last_became_active":"2024-03-16T11:48:38.020906+0000","last_became_peered":"2024-03-16T11:48:38.020906+0000","last_unstale":"2024-03-16T11:49:28.635756+0000","last_undegraded":"2024-03-16T11:49:28.635756+0000","last_fullsized":"2024-03-16T11:49:28.635756+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_clean_scrub_stamp":"2024-03-16T11:48:36.997377+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:09:53.372698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"87'9","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T11:49:57.668407+0000","last_change":"2024-03-16T11:48:38.573536+0000","last_active":"2024-03-16T11:49:57.668407+0000","last_peered":"2024-03-16T11:49:57.668407+0000","last_clean":"2024-03-16T11:49:57.668407+0000","last_became_active":"2024-03-16T11:48:38.573211+0000","last_became_peered":"2024-03-16T11:48:38.573211+0000","last_unstale":"2024-03-16T11:49:57.668407+0000","last_undegraded":"2024-03-16T11:49:57.668407+0000","last_fullsized":"2024-03-16T11:49:57.668407+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:53:02.607175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"85'1","reported_seq":15,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T11:49:28.645934+0000","last_change":"2024-03-16T11:49:28.645457+0000","last_active":"2024-03-16T11:49:28.645934+0000","last_peered":"2024-03-16T11:49:28.645934+0000","last_clean":"2024-03-16T11:49:28.645934+0000","last_became_active":"2024-03-16T11:49:28.637962+0000","last_became_peered":"2024-03-16T11:49:28.637962+0000","last_unstale":"2024-03-16T11:49:28.645934+0000","last_undegraded":"2024-03-16T11:49:28.645934+0000","last_fullsized":"2024-03-16T11:49:28.645934+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_clean_scrub_stamp":"2024-03-16T11:48:36.997377+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:53:38.317556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,8],"acting":[11,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.b","version":"87'8","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T11:49:57.662515+0000","last_change":"2024-03-16T11:48:38.575965+0000","last_active":"2024-03-16T11:49:57.662515+0000","last_peered":"2024-03-16T11:49:57.662515+0000","last_clean":"2024-03-16T11:49:57.662515+0000","last_became_active":"2024-03-16T11:48:38.575627+0000","last_became_peered":"2024-03-16T11:48:38.575627+0000","last_unstale":"2024-03-16T11:49:57.662515+0000","last_undegraded":"2024-03-16T11:49:57.662515+0000","last_fullsized":"2024-03-16T11:49:57.662515+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:00:37.768897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"85'4","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T11:49:28.631560+0000","last_change":"2024-03-16T11:48:38.027516+0000","last_active":"2024-03-16T11:49:28.631560+0000","last_peered":"2024-03-16T11:49:28.631560+0000","last_clean":"2024-03-16T11:49:28.631560+0000","last_became_active":"2024-03-16T11:48:38.026865+0000","last_became_peered":"2024-03-16T11:48:38.026865+0000","last_unstale":"2024-03-16T11:49:28.631560+0000","last_undegraded":"2024-03-16T11:49:28.631560+0000","last_fullsized":"2024-03-16T11:49:28.631560+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_clean_scrub_stamp":"2024-03-16T11:48:36.997377+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:50:50.240973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"87'5","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T11:49:57.676209+0000","last_change":"2024-03-16T11:48:38.571242+0000","last_active":"2024-03-16T11:49:57.676209+0000","last_peered":"2024-03-16T11:49:57.676209+0000","last_clean":"2024-03-16T11:49:57.676209+0000","last_became_active":"2024-03-16T11:48:38.570917+0000","last_became_peered":"2024-03-16T11:48:38.570917+0000","last_unstale":"2024-03-16T11:49:57.676209+0000","last_undegraded":"2024-03-16T11:49:57.676209+0000","last_fullsized":"2024-03-16T11:49:57.676209+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:30:12.601982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"85'3","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T11:49:28.636508+0000","last_change":"2024-03-16T11:48:38.023970+0000","last_active":"2024-03-16T11:49:28.636508+0000","last_peered":"2024-03-16T11:49:28.636508+0000","last_clean":"2024-03-16T11:49:28.636508+0000","last_became_active":"2024-03-16T11:48:38.023646+0000","last_became_peered":"2024-03-16T11:48:38.023646+0000","last_unstale":"2024-03-16T11:49:28.636508+0000","last_undegraded":"2024-03-16T11:49:28.636508+0000","last_fullsized":"2024-03-16T11:49:28.636508+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_clean_scrub_stamp":"2024-03-16T11:48:36.997377+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:48:35.549588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"87'7","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T11:49:57.676353+0000","last_change":"2024-03-16T11:48:38.567376+0000","last_active":"2024-03-16T11:49:57.676353+0000","last_peered":"2024-03-16T11:49:57.676353+0000","last_clean":"2024-03-16T11:49:57.676353+0000","last_became_active":"2024-03-16T11:48:38.567048+0000","last_became_peered":"2024-03-16T11:48:38.567048+0000","last_unstale":"2024-03-16T11:49:57.676353+0000","last_undegraded":"2024-03-16T11:49:57.676353+0000","last_fullsized":"2024-03-16T11:49:57.676353+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:45:50.097013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"84'1","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T11:49:28.634840+0000","last_change":"2024-03-16T11:48:38.029192+0000","last_active":"2024-03-16T11:49:28.634840+0000","last_peered":"2024-03-16T11:49:28.634840+0000","last_clean":"2024-03-16T11:49:28.634840+0000","last_became_active":"2024-03-16T11:48:38.028866+0000","last_became_peered":"2024-03-16T11:48:38.028866+0000","last_unstale":"2024-03-16T11:49:28.634840+0000","last_undegraded":"2024-03-16T11:49:28.634840+0000","last_fullsized":"2024-03-16T11:49:28.634840+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_clean_scrub_stamp":"2024-03-16T11:48:36.997377+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:52:31.329863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"87'7","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T11:49:57.648434+0000","last_change":"2024-03-16T11:48:38.576538+0000","last_active":"2024-03-16T11:49:57.648434+0000","last_peered":"2024-03-16T11:49:57.648434+0000","last_clean":"2024-03-16T11:49:57.648434+0000","last_became_active":"2024-03-16T11:48:38.576222+0000","last_became_peered":"2024-03-16T11:48:38.576222+0000","last_unstale":"2024-03-16T11:49:57.648434+0000","last_undegraded":"2024-03-16T11:49:57.648434+0000","last_fullsized":"2024-03-16T11:49:57.648434+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:34:13.186406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T11:49:28.635452+0000","last_change":"2024-03-16T11:48:38.031002+0000","last_active":"2024-03-16T11:49:28.635452+0000","last_peered":"2024-03-16T11:49:28.635452+0000","last_clean":"2024-03-16T11:49:28.635452+0000","last_became_active":"2024-03-16T11:48:38.030653+0000","last_became_peered":"2024-03-16T11:48:38.030653+0000","last_unstale":"2024-03-16T11:49:28.635452+0000","last_undegraded":"2024-03-16T11:49:28.635452+0000","last_fullsized":"2024-03-16T11:49:28.635452+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_clean_scrub_stamp":"2024-03-16T11:48:36.997377+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:42:30.148594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"87'9","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T11:49:57.660225+0000","last_change":"2024-03-16T11:48:38.571942+0000","last_active":"2024-03-16T11:49:57.660225+0000","last_peered":"2024-03-16T11:49:57.660225+0000","last_clean":"2024-03-16T11:49:57.660225+0000","last_became_active":"2024-03-16T11:48:38.571606+0000","last_became_peered":"2024-03-16T11:48:38.571606+0000","last_unstale":"2024-03-16T11:49:57.660225+0000","last_undegraded":"2024-03-16T11:49:57.660225+0000","last_fullsized":"2024-03-16T11:49:57.660225+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:29:13.797054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"87'9","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T11:49:57.659937+0000","last_change":"2024-03-16T11:48:38.573469+0000","last_active":"2024-03-16T11:49:57.659937+0000","last_peered":"2024-03-16T11:49:57.659937+0000","last_clean":"2024-03-16T11:49:57.659937+0000","last_became_active":"2024-03-16T11:48:38.573155+0000","last_became_peered":"2024-03-16T11:48:38.573155+0000","last_unstale":"2024-03-16T11:49:57.659937+0000","last_undegraded":"2024-03-16T11:49:57.659937+0000","last_fullsized":"2024-03-16T11:49:57.659937+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:28:50.180821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T11:49:28.635977+0000","last_change":"2024-03-16T11:48:38.580913+0000","last_active":"2024-03-16T11:49:28.635977+0000","last_peered":"2024-03-16T11:49:28.635977+0000","last_clean":"2024-03-16T11:49:28.635977+0000","last_became_active":"2024-03-16T11:48:38.580573+0000","last_became_peered":"2024-03-16T11:48:38.580573+0000","last_unstale":"2024-03-16T11:49:28.635977+0000","last_undegraded":"2024-03-16T11:49:28.635977+0000","last_fullsized":"2024-03-16T11:49:28.635977+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:44:58.080502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T11:49:28.635942+0000","last_change":"2024-03-16T11:48:38.577168+0000","last_active":"2024-03-16T11:49:28.635942+0000","last_peered":"2024-03-16T11:49:28.635942+0000","last_clean":"2024-03-16T11:49:28.635942+0000","last_became_active":"2024-03-16T11:48:38.576845+0000","last_became_peered":"2024-03-16T11:48:38.576845+0000","last_unstale":"2024-03-16T11:49:28.635942+0000","last_undegraded":"2024-03-16T11:49:28.635942+0000","last_fullsized":"2024-03-16T11:49:28.635942+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:13:02.145610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"87'7","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T11:49:57.668392+0000","last_change":"2024-03-16T11:48:38.577477+0000","last_active":"2024-03-16T11:49:57.668392+0000","last_peered":"2024-03-16T11:49:57.668392+0000","last_clean":"2024-03-16T11:49:57.668392+0000","last_became_active":"2024-03-16T11:48:38.577126+0000","last_became_peered":"2024-03-16T11:48:38.577126+0000","last_unstale":"2024-03-16T11:49:57.668392+0000","last_undegraded":"2024-03-16T11:49:57.668392+0000","last_fullsized":"2024-03-16T11:49:57.668392+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:45:27.246827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"87'4","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T11:49:57.661494+0000","last_change":"2024-03-16T11:48:38.578508+0000","last_active":"2024-03-16T11:49:57.661494+0000","last_peered":"2024-03-16T11:49:57.661494+0000","last_clean":"2024-03-16T11:49:57.661494+0000","last_became_active":"2024-03-16T11:48:38.578125+0000","last_became_peered":"2024-03-16T11:48:38.578125+0000","last_unstale":"2024-03-16T11:49:57.661494+0000","last_undegraded":"2024-03-16T11:49:57.661494+0000","last_fullsized":"2024-03-16T11:49:57.661494+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:13:36.996405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"87'6","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T11:49:57.659578+0000","last_change":"2024-03-16T11:48:38.570450+0000","last_active":"2024-03-16T11:49:57.659578+0000","last_peered":"2024-03-16T11:49:57.659578+0000","last_clean":"2024-03-16T11:49:57.659578+0000","last_became_active":"2024-03-16T11:48:38.569952+0000","last_became_peered":"2024-03-16T11:48:38.569952+0000","last_unstale":"2024-03-16T11:49:57.659578+0000","last_undegraded":"2024-03-16T11:49:57.659578+0000","last_fullsized":"2024-03-16T11:49:57.659578+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:34:39.265969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T11:49:28.636406+0000","last_change":"2024-03-16T11:48:38.569568+0000","last_active":"2024-03-16T11:49:28.636406+0000","last_peered":"2024-03-16T11:49:28.636406+0000","last_clean":"2024-03-16T11:49:28.636406+0000","last_became_active":"2024-03-16T11:48:38.569260+0000","last_became_peered":"2024-03-16T11:48:38.569260+0000","last_unstale":"2024-03-16T11:49:28.636406+0000","last_undegraded":"2024-03-16T11:49:28.636406+0000","last_fullsized":"2024-03-16T11:49:28.636406+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:52:52.315980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"87'2","reported_seq":19,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T11:49:57.665231+0000","last_change":"2024-03-16T11:48:38.584562+0000","last_active":"2024-03-16T11:49:57.665231+0000","last_peered":"2024-03-16T11:49:57.665231+0000","last_clean":"2024-03-16T11:49:57.665231+0000","last_became_active":"2024-03-16T11:48:38.584096+0000","last_became_peered":"2024-03-16T11:48:38.584096+0000","last_unstale":"2024-03-16T11:49:57.665231+0000","last_undegraded":"2024-03-16T11:49:57.665231+0000","last_fullsized":"2024-03-16T11:49:57.665231+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:43:45.933252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"87'6","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T11:49:57.665200+0000","last_change":"2024-03-16T11:48:38.570174+0000","last_active":"2024-03-16T11:49:57.665200+0000","last_peered":"2024-03-16T11:49:57.665200+0000","last_clean":"2024-03-16T11:49:57.665200+0000","last_became_active":"2024-03-16T11:48:38.569865+0000","last_became_peered":"2024-03-16T11:48:38.569865+0000","last_unstale":"2024-03-16T11:49:57.665200+0000","last_undegraded":"2024-03-16T11:49:57.665200+0000","last_fullsized":"2024-03-16T11:49:57.665200+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T11:53:01.012481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"87'8","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T11:49:57.665777+0000","last_change":"2024-03-16T11:48:38.574868+0000","last_active":"2024-03-16T11:49:57.665777+0000","last_peered":"2024-03-16T11:49:57.665777+0000","last_clean":"2024-03-16T11:49:57.665777+0000","last_became_active":"2024-03-16T11:48:38.574434+0000","last_became_peered":"2024-03-16T11:48:38.574434+0000","last_unstale":"2024-03-16T11:49:57.665777+0000","last_undegraded":"2024-03-16T11:49:57.665777+0000","last_fullsized":"2024-03-16T11:49:57.665777+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:09:57.102253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"87'10","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T11:49:57.674127+0000","last_change":"2024-03-16T11:48:38.577189+0000","last_active":"2024-03-16T11:49:57.674127+0000","last_peered":"2024-03-16T11:49:57.674127+0000","last_clean":"2024-03-16T11:49:57.674127+0000","last_became_active":"2024-03-16T11:48:38.576884+0000","last_became_peered":"2024-03-16T11:48:38.576884+0000","last_unstale":"2024-03-16T11:49:57.674127+0000","last_undegraded":"2024-03-16T11:49:57.674127+0000","last_fullsized":"2024-03-16T11:49:57.674127+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:11:51.558517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"87'11","reported_seq":28,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T11:49:57.657680+0000","last_change":"2024-03-16T11:48:38.576238+0000","last_active":"2024-03-16T11:49:57.657680+0000","last_peered":"2024-03-16T11:49:57.657680+0000","last_clean":"2024-03-16T11:49:57.657680+0000","last_became_active":"2024-03-16T11:48:38.575951+0000","last_became_peered":"2024-03-16T11:48:38.575951+0000","last_unstale":"2024-03-16T11:49:57.657680+0000","last_undegraded":"2024-03-16T11:49:57.657680+0000","last_fullsized":"2024-03-16T11:49:57.657680+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:47:32.571580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"87'8","reported_seq":31,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T11:49:57.662748+0000","last_change":"2024-03-16T11:49:28.732348+0000","last_active":"2024-03-16T11:49:57.662748+0000","last_peered":"2024-03-16T11:49:57.662748+0000","last_clean":"2024-03-16T11:49:57.662748+0000","last_became_active":"2024-03-16T11:49:28.732043+0000","last_became_peered":"2024-03-16T11:49:28.732043+0000","last_unstale":"2024-03-16T11:49:57.662748+0000","last_undegraded":"2024-03-16T11:49:57.662748+0000","last_fullsized":"2024-03-16T11:49:57.662748+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:21:32.831045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,8],"acting":[11,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":445,"ondisk_log_size":445,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":284036,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":64805,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":488,"num_read_kb":313,"num_write":164,"num_write_kb":116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":61036,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":372736,"data_stored":194448,"data_compressed":9582,"data_compressed_allocated":12288,"data_compressed_original":57344,"omap_allocated":0,"internal_metadata":0},"log_size":143,"ondisk_log_size":143,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383711,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32096,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713056,"statfs":{"total":95995035648,"available":95962169344,"internally_reserved":0,"allocated":942080,"data_stored":456205,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645347,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28084,"kb_used_data":1068,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717068,"statfs":{"total":95995035648,"available":95966277632,"internally_reserved":0,"allocated":1093632,"data_stored":1108490,"data_compressed":36118,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808873,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27956,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717196,"statfs":{"total":95995035648,"available":95966408704,"internally_reserved":0,"allocated":958464,"data_stored":456441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070511,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32112,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713040,"statfs":{"total":95995035648,"available":95962152960,"internally_reserved":0,"allocated":958464,"data_stored":456361,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332147,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28000,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717152,"statfs":{"total":95995035648,"available":95966363648,"internally_reserved":0,"allocated":1007616,"data_stored":1048964,"data_compressed":36118,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495673,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36260,"kb_used_data":924,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708892,"statfs":{"total":95995035648,"available":95957905408,"internally_reserved":0,"allocated":946176,"data_stored":455830,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574930,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32172,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712980,"statfs":{"total":95995035648,"available":95962091520,"internally_reserved":0,"allocated":1015808,"data_stored":516659,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738455,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27960,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717192,"statfs":{"total":95995035648,"available":95966404608,"internally_reserved":0,"allocated":966656,"data_stored":457101,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411405,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32112,"kb_used_data":932,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713040,"statfs":{"total":95995035648,"available":95962152960,"internally_reserved":0,"allocated":954368,"data_stored":456351,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182472,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36276,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708876,"statfs":{"total":95995035648,"available":95957889024,"internally_reserved":0,"allocated":962560,"data_stored":456141,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920835,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27976,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717176,"statfs":{"total":95995035648,"available":95966388224,"internally_reserved":0,"allocated":974848,"data_stored":516681,"data_compressed":9582,"data_compressed_allocated":12288,"data_compressed_original":57344,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757310,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32172,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712980,"statfs":{"total":95995035648,"available":95962091520,"internally_reserved":0,"allocated":1019904,"data_stored":1048116,"data_compressed":36118,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34482,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34482,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34482,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36118,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36118,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36118,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36118,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36118,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":61426,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":61058,"data_compressed":9582,"data_compressed_allocated":12288,"data_compressed_original":57344,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":62382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-16T11:50:03.718 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid d1887f28-e389-11ee-95c9-87774f69a715 -- ceph pg dump --format=json 2024-03-16T11:50:03.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:50:03 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:50:03.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:50:03 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:50:03.968 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/d1887f28-e389-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T11:50:04.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:50:03 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:50:04.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:50:03 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:50:04.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:50:03 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:50:04.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:50:03 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:50:04.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:50:04 smithi029 ceph-mon[31975]: from='client.24800 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-16T11:50:04.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:50:04 smithi029 ceph-mon[31975]: pgmap v335: 97 pgs: 97 active+clean; 640 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 597 B/s wr, 6 op/s 2024-03-16T11:50:04.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:50:04 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:50:04.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:50:04 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T11:50:04.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:50:04 smithi029 ceph-mon[31975]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:50:04.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:50:04 smithi152 ceph-mon[36291]: from='client.24800 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-16T11:50:04.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:50:04 smithi152 ceph-mon[36291]: pgmap v335: 97 pgs: 97 active+clean; 640 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 597 B/s wr, 6 op/s 2024-03-16T11:50:04.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:50:04 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:50:04.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:50:04 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T11:50:04.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:50:04 smithi152 ceph-mon[36291]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:50:05.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:50:04 smithi154 ceph-mon[36320]: from='client.24800 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-16T11:50:05.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:50:04 smithi154 ceph-mon[36320]: pgmap v335: 97 pgs: 97 active+clean; 640 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 597 B/s wr, 6 op/s 2024-03-16T11:50:05.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:50:04 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:50:05.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:50:04 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T11:50:05.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:50:04 smithi154 ceph-mon[36320]: from='mgr.14152 172.21.15.29:0/3517263510' entity='mgr.x' 2024-03-16T11:50:05.780 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:05.779+0000 7fba943e5700 1 -- 172.21.15.29:0/3827987289 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fba8c1025b0 msgr2=0x7fba8c102990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:05.780 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:05.779+0000 7fba943e5700 1 --2- 172.21.15.29:0/3827987289 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fba8c1025b0 0x7fba8c102990 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fba88009990 tx=0x7fba88009ca0 comp rx=0 tx=0).stop 2024-03-16T11:50:05.781 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:05.779+0000 7fba943e5700 1 -- 172.21.15.29:0/3827987289 shutdown_connections 2024-03-16T11:50:05.781 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:05.779+0000 7fba943e5700 1 --2- 172.21.15.29:0/3827987289 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fba8c1075e0 0x7fba8c10f1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:05.781 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:05.779+0000 7fba943e5700 1 --2- 172.21.15.29:0/3827987289 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fba8c1025b0 0x7fba8c102990 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:05.781 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:05.779+0000 7fba943e5700 1 --2- 172.21.15.29:0/3827987289 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fba8c102f60 0x7fba8c106f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:05.781 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:05.779+0000 7fba943e5700 1 -- 172.21.15.29:0/3827987289 >> 172.21.15.29:0/3827987289 conn(0x7fba8c079360 msgr2=0x7fba8c079770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:50:05.781 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:05.779+0000 7fba943e5700 1 -- 172.21.15.29:0/3827987289 shutdown_connections 2024-03-16T11:50:05.781 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:05.779+0000 7fba943e5700 1 -- 172.21.15.29:0/3827987289 wait complete. 2024-03-16T11:50:05.781 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:05.780+0000 7fba943e5700 1 Processor -- start 2024-03-16T11:50:05.781 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:05.780+0000 7fba943e5700 1 -- start start 2024-03-16T11:50:05.781 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:05.780+0000 7fba943e5700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fba8c1025b0 0x7fba8c19e700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:05.781 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:05.780+0000 7fba943e5700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fba8c102f60 0x7fba8c19ec40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:05.781 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:05.780+0000 7fba943e5700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fba8c1075e0 0x7fba8c1988c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:05.781 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:05.780+0000 7fba943e5700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fba8c1125c0 con 0x7fba8c1025b0 2024-03-16T11:50:05.782 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:05.780+0000 7fba943e5700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fba8c112440 con 0x7fba8c102f60 2024-03-16T11:50:05.782 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:05.780+0000 7fba943e5700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fba8c112740 con 0x7fba8c1075e0 2024-03-16T11:50:05.782 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:05.780+0000 7fba92181700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fba8c1025b0 0x7fba8c19e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:05.782 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:05.780+0000 7fba92181700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fba8c1025b0 0x7fba8c19e700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49126/0 (socket says 172.21.15.29:49126) 2024-03-16T11:50:05.782 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:05.780+0000 7fba92181700 1 -- 172.21.15.29:0/1456767194 learned_addr learned my addr 172.21.15.29:0/1456767194 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:50:05.782 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:05.780+0000 7fba91980700 1 --2- 172.21.15.29:0/1456767194 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fba8c102f60 0x7fba8c19ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:05.782 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:05.780+0000 7fba92982700 1 --2- 172.21.15.29:0/1456767194 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fba8c1075e0 0x7fba8c1988c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:05.782 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:05.781+0000 7fba92181700 1 -- 172.21.15.29:0/1456767194 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fba8c1075e0 msgr2=0x7fba8c1988c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:05.782 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:05.781+0000 7fba92181700 1 --2- 172.21.15.29:0/1456767194 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fba8c1075e0 0x7fba8c1988c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:05.782 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:05.781+0000 7fba92181700 1 -- 172.21.15.29:0/1456767194 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fba8c102f60 msgr2=0x7fba8c19ec40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:05.782 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:05.781+0000 7fba92181700 1 --2- 172.21.15.29:0/1456767194 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fba8c102f60 0x7fba8c19ec40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:05.782 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:05.781+0000 7fba92181700 1 -- 172.21.15.29:0/1456767194 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba8c199020 con 0x7fba8c1025b0 2024-03-16T11:50:05.783 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:05.781+0000 7fba92181700 1 --2- 172.21.15.29:0/1456767194 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fba8c1025b0 0x7fba8c19e700 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fba88000c00 tx=0x7fba8800bab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:50:05.783 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:05.781+0000 7fba837fe700 1 -- 172.21.15.29:0/1456767194 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba88027070 con 0x7fba8c1025b0 2024-03-16T11:50:05.783 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:05.781+0000 7fba837fe700 1 -- 172.21.15.29:0/1456767194 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fba88003d70 con 0x7fba8c1025b0 2024-03-16T11:50:05.783 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:05.781+0000 7fba943e5700 1 -- 172.21.15.29:0/1456767194 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fba8c1992b0 con 0x7fba8c1025b0 2024-03-16T11:50:05.783 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:05.781+0000 7fba837fe700 1 -- 172.21.15.29:0/1456767194 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba8800f460 con 0x7fba8c1025b0 2024-03-16T11:50:05.783 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:05.781+0000 7fba943e5700 1 -- 172.21.15.29:0/1456767194 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fba8c1a55d0 con 0x7fba8c1025b0 2024-03-16T11:50:05.784 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:05.782+0000 7fba943e5700 1 -- 172.21.15.29:0/1456767194 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fba8c10a800 con 0x7fba8c1025b0 2024-03-16T11:50:05.784 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:05.783+0000 7fba837fe700 1 -- 172.21.15.29:0/1456767194 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fba8802a070 con 0x7fba8c1025b0 2024-03-16T11:50:05.784 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:05.784+0000 7fba837fe700 1 --2- 172.21.15.29:0/1456767194 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fba780a2890 0x7fba780a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:05.784 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:05.784+0000 7fba91980700 1 --2- 172.21.15.29:0/1456767194 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fba780a2890 0x7fba780a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:05.784 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:05.784+0000 7fba837fe700 1 -- 172.21.15.29:0/1456767194 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8211+0+0 (secure 0 0 0) 0x7fba880cf7d0 con 0x7fba8c1025b0 2024-03-16T11:50:05.784 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:05.784+0000 7fba91980700 1 --2- 172.21.15.29:0/1456767194 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fba780a2890 0x7fba780a4d50 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fba8c101f20 tx=0x7fba7c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:50:05.787 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:05.788+0000 7fba837fe700 1 -- 172.21.15.29:0/1456767194 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fba88013070 con 0x7fba8c1025b0 2024-03-16T11:50:05.987 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:05.988+0000 7fba943e5700 1 -- 172.21.15.29:0/1456767194 --> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fba8c199df0 con 0x7fba780a2890 2024-03-16T11:50:05.988 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:50:05 smithi029 ceph-mon[31975]: pgmap v336: 97 pgs: 97 active+clean; 640 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 597 B/s wr, 7 op/s 2024-03-16T11:50:05.994 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:05.996+0000 7fba837fe700 1 -- 172.21.15.29:0/1456767194 <== mgr.14152 v2:172.21.15.29:6800/4192064640 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+259109 (secure 0 0 0) 0x7fba8c199df0 con 0x7fba780a2890 2024-03-16T11:50:05.998 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-16T11:50:06.000 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:05.998+0000 7fba943e5700 1 -- 172.21.15.29:0/1456767194 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fba780a2890 msgr2=0x7fba780a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:06.000 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:05.998+0000 7fba943e5700 1 --2- 172.21.15.29:0/1456767194 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fba780a2890 0x7fba780a4d50 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fba8c101f20 tx=0x7fba7c009260 comp rx=0 tx=0).stop 2024-03-16T11:50:06.000 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:05.998+0000 7fba943e5700 1 -- 172.21.15.29:0/1456767194 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fba8c1025b0 msgr2=0x7fba8c19e700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:06.000 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:05.998+0000 7fba943e5700 1 --2- 172.21.15.29:0/1456767194 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fba8c1025b0 0x7fba8c19e700 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fba88000c00 tx=0x7fba8800bab0 comp rx=0 tx=0).stop 2024-03-16T11:50:06.000 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:05.998+0000 7fba943e5700 1 -- 172.21.15.29:0/1456767194 shutdown_connections 2024-03-16T11:50:06.000 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:05.998+0000 7fba943e5700 1 --2- 172.21.15.29:0/1456767194 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fba780a2890 0x7fba780a4d50 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:06.000 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:05.998+0000 7fba943e5700 1 --2- 172.21.15.29:0/1456767194 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fba8c1075e0 0x7fba8c1988c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:06.000 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:05.998+0000 7fba943e5700 1 --2- 172.21.15.29:0/1456767194 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fba8c1025b0 0x7fba8c19e700 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:06.000 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:05.998+0000 7fba943e5700 1 --2- 172.21.15.29:0/1456767194 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fba8c102f60 0x7fba8c19ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:06.000 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:05.998+0000 7fba943e5700 1 -- 172.21.15.29:0/1456767194 >> 172.21.15.29:0/1456767194 conn(0x7fba8c079360 msgr2=0x7fba8c108760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:50:06.001 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:05.998+0000 7fba943e5700 1 -- 172.21.15.29:0/1456767194 shutdown_connections 2024-03-16T11:50:06.001 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:05.998+0000 7fba943e5700 1 -- 172.21.15.29:0/1456767194 wait complete. 2024-03-16T11:50:06.001 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2024-03-16T11:50:06.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:50:05 smithi154 ceph-mon[36320]: pgmap v336: 97 pgs: 97 active+clean; 640 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 597 B/s wr, 7 op/s 2024-03-16T11:50:06.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:50:05 smithi152 ceph-mon[36291]: pgmap v336: 97 pgs: 97 active+clean; 640 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 597 B/s wr, 7 op/s 2024-03-16T11:50:06.916 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":336,"stamp":"2024-03-16T11:50:05.639136+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":655548,"num_objects":64,"num_object_clones":0,"num_object_copies":192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":650,"num_read_kb":448,"num_write":350,"num_write_kb":3057,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":1241772,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":784,"ondisk_log_size":784,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":373156,"kb_used_data":11524,"kb_used_omap":18,"kb_used_meta":361517,"kb_avail":1124568668,"statfs":{"total":1151940427776,"available":1151558316032,"internally_reserved":0,"allocated":11800576,"data_stored":7434120,"data_compressed":117936,"data_compressed_allocated":147456,"data_compressed_original":1826816,"omap_allocated":19143,"internal_metadata":370193721},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":531,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":78,"num_read_kb":43,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003260"},"pg_stats":[{"pgid":"3.3f","version":"87'8","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T11:49:57.668966+0000","last_change":"2024-03-16T11:48:38.574718+0000","last_active":"2024-03-16T11:49:57.668966+0000","last_peered":"2024-03-16T11:49:57.668966+0000","last_clean":"2024-03-16T11:49:57.668966+0000","last_became_active":"2024-03-16T11:48:38.574420+0000","last_became_peered":"2024-03-16T11:48:38.574420+0000","last_unstale":"2024-03-16T11:49:57.668966+0000","last_undegraded":"2024-03-16T11:49:57.668966+0000","last_fullsized":"2024-03-16T11:49:57.668966+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:43:56.705135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"87'4","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T11:49:57.664700+0000","last_change":"2024-03-16T11:48:38.572722+0000","last_active":"2024-03-16T11:49:57.664700+0000","last_peered":"2024-03-16T11:49:57.664700+0000","last_clean":"2024-03-16T11:49:57.664700+0000","last_became_active":"2024-03-16T11:48:38.572431+0000","last_became_peered":"2024-03-16T11:48:38.572431+0000","last_unstale":"2024-03-16T11:49:57.664700+0000","last_undegraded":"2024-03-16T11:49:57.664700+0000","last_fullsized":"2024-03-16T11:49:57.664700+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:34:45.070975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"87'7","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T11:49:57.666931+0000","last_change":"2024-03-16T11:48:38.585504+0000","last_active":"2024-03-16T11:49:57.666931+0000","last_peered":"2024-03-16T11:49:57.666931+0000","last_clean":"2024-03-16T11:49:57.666931+0000","last_became_active":"2024-03-16T11:48:38.583879+0000","last_became_peered":"2024-03-16T11:48:38.583879+0000","last_unstale":"2024-03-16T11:49:57.666931+0000","last_undegraded":"2024-03-16T11:49:57.666931+0000","last_fullsized":"2024-03-16T11:49:57.666931+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:08:41.119349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"87'8","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T11:49:57.662856+0000","last_change":"2024-03-16T11:48:38.575773+0000","last_active":"2024-03-16T11:49:57.662856+0000","last_peered":"2024-03-16T11:49:57.662856+0000","last_clean":"2024-03-16T11:49:57.662856+0000","last_became_active":"2024-03-16T11:48:38.574441+0000","last_became_peered":"2024-03-16T11:48:38.574441+0000","last_unstale":"2024-03-16T11:49:57.662856+0000","last_undegraded":"2024-03-16T11:49:57.662856+0000","last_fullsized":"2024-03-16T11:49:57.662856+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:22:56.148164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"87'4","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T11:49:57.649692+0000","last_change":"2024-03-16T11:48:38.575505+0000","last_active":"2024-03-16T11:49:57.649692+0000","last_peered":"2024-03-16T11:49:57.649692+0000","last_clean":"2024-03-16T11:49:57.649692+0000","last_became_active":"2024-03-16T11:48:38.575188+0000","last_became_peered":"2024-03-16T11:48:38.575188+0000","last_unstale":"2024-03-16T11:49:57.649692+0000","last_undegraded":"2024-03-16T11:49:57.649692+0000","last_fullsized":"2024-03-16T11:49:57.649692+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:55:33.625249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"87'6","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T11:49:57.659117+0000","last_change":"2024-03-16T11:48:38.572044+0000","last_active":"2024-03-16T11:49:57.659117+0000","last_peered":"2024-03-16T11:49:57.659117+0000","last_clean":"2024-03-16T11:49:57.659117+0000","last_became_active":"2024-03-16T11:48:38.571725+0000","last_became_peered":"2024-03-16T11:48:38.571725+0000","last_unstale":"2024-03-16T11:49:57.659117+0000","last_undegraded":"2024-03-16T11:49:57.659117+0000","last_fullsized":"2024-03-16T11:49:57.659117+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:05:37.060342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"87'5","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T11:49:57.664109+0000","last_change":"2024-03-16T11:48:38.570332+0000","last_active":"2024-03-16T11:49:57.664109+0000","last_peered":"2024-03-16T11:49:57.664109+0000","last_clean":"2024-03-16T11:49:57.664109+0000","last_became_active":"2024-03-16T11:48:38.570009+0000","last_became_peered":"2024-03-16T11:48:38.570009+0000","last_unstale":"2024-03-16T11:49:57.664109+0000","last_undegraded":"2024-03-16T11:49:57.664109+0000","last_fullsized":"2024-03-16T11:49:57.664109+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:42:31.045749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"87'9","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T11:49:57.670699+0000","last_change":"2024-03-16T11:48:38.571741+0000","last_active":"2024-03-16T11:49:57.670699+0000","last_peered":"2024-03-16T11:49:57.670699+0000","last_clean":"2024-03-16T11:49:57.670699+0000","last_became_active":"2024-03-16T11:48:38.569582+0000","last_became_peered":"2024-03-16T11:48:38.569582+0000","last_unstale":"2024-03-16T11:49:57.670699+0000","last_undegraded":"2024-03-16T11:49:57.670699+0000","last_fullsized":"2024-03-16T11:49:57.670699+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:05:36.302223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"87'11","reported_seq":28,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T11:49:57.666578+0000","last_change":"2024-03-16T11:48:38.576038+0000","last_active":"2024-03-16T11:49:57.666578+0000","last_peered":"2024-03-16T11:49:57.666578+0000","last_clean":"2024-03-16T11:49:57.666578+0000","last_became_active":"2024-03-16T11:48:38.574223+0000","last_became_peered":"2024-03-16T11:48:38.574223+0000","last_unstale":"2024-03-16T11:49:57.666578+0000","last_undegraded":"2024-03-16T11:49:57.666578+0000","last_fullsized":"2024-03-16T11:49:57.666578+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:35:17.236721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"87'16","reported_seq":33,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T11:49:57.658419+0000","last_change":"2024-03-16T11:48:38.572797+0000","last_active":"2024-03-16T11:49:57.658419+0000","last_peered":"2024-03-16T11:49:57.658419+0000","last_clean":"2024-03-16T11:49:57.658419+0000","last_became_active":"2024-03-16T11:48:38.572497+0000","last_became_peered":"2024-03-16T11:48:38.572497+0000","last_unstale":"2024-03-16T11:49:57.658419+0000","last_undegraded":"2024-03-16T11:49:57.658419+0000","last_fullsized":"2024-03-16T11:49:57.658419+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:25:49.138897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"87'6","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T11:49:57.676428+0000","last_change":"2024-03-16T11:48:38.570443+0000","last_active":"2024-03-16T11:49:57.676428+0000","last_peered":"2024-03-16T11:49:57.676428+0000","last_clean":"2024-03-16T11:49:57.676428+0000","last_became_active":"2024-03-16T11:48:38.569984+0000","last_became_peered":"2024-03-16T11:48:38.569984+0000","last_unstale":"2024-03-16T11:49:57.676428+0000","last_undegraded":"2024-03-16T11:49:57.676428+0000","last_fullsized":"2024-03-16T11:49:57.676428+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:47:42.539130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"87'7","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T11:49:57.662735+0000","last_change":"2024-03-16T11:48:38.576106+0000","last_active":"2024-03-16T11:49:57.662735+0000","last_peered":"2024-03-16T11:49:57.662735+0000","last_clean":"2024-03-16T11:49:57.662735+0000","last_became_active":"2024-03-16T11:48:38.575409+0000","last_became_peered":"2024-03-16T11:48:38.575409+0000","last_unstale":"2024-03-16T11:49:57.662735+0000","last_undegraded":"2024-03-16T11:49:57.662735+0000","last_fullsized":"2024-03-16T11:49:57.662735+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:38:33.277722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"87'10","reported_seq":33,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T11:49:57.662256+0000","last_change":"2024-03-16T11:49:28.732730+0000","last_active":"2024-03-16T11:49:57.662256+0000","last_peered":"2024-03-16T11:49:57.662256+0000","last_clean":"2024-03-16T11:49:57.662256+0000","last_became_active":"2024-03-16T11:49:28.732332+0000","last_became_peered":"2024-03-16T11:49:28.732332+0000","last_unstale":"2024-03-16T11:49:57.662256+0000","last_undegraded":"2024-03-16T11:49:57.662256+0000","last_fullsized":"2024-03-16T11:49:57.662256+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:32:23.882235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,9],"acting":[6,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"87'7","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T11:49:57.669216+0000","last_change":"2024-03-16T11:48:38.583598+0000","last_active":"2024-03-16T11:49:57.669216+0000","last_peered":"2024-03-16T11:49:57.669216+0000","last_clean":"2024-03-16T11:49:57.669216+0000","last_became_active":"2024-03-16T11:48:38.583275+0000","last_became_peered":"2024-03-16T11:48:38.583275+0000","last_unstale":"2024-03-16T11:49:57.669216+0000","last_undegraded":"2024-03-16T11:49:57.669216+0000","last_fullsized":"2024-03-16T11:49:57.669216+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:58:48.667075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"87'7","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T11:49:57.667881+0000","last_change":"2024-03-16T11:48:38.580240+0000","last_active":"2024-03-16T11:49:57.667881+0000","last_peered":"2024-03-16T11:49:57.667881+0000","last_clean":"2024-03-16T11:49:57.667881+0000","last_became_active":"2024-03-16T11:48:38.579928+0000","last_became_peered":"2024-03-16T11:48:38.579928+0000","last_unstale":"2024-03-16T11:49:57.667881+0000","last_undegraded":"2024-03-16T11:49:57.667881+0000","last_fullsized":"2024-03-16T11:49:57.667881+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:05:14.006915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"87'9","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T11:49:57.673861+0000","last_change":"2024-03-16T11:48:38.574433+0000","last_active":"2024-03-16T11:49:57.673861+0000","last_peered":"2024-03-16T11:49:57.673861+0000","last_clean":"2024-03-16T11:49:57.673861+0000","last_became_active":"2024-03-16T11:48:38.574104+0000","last_became_peered":"2024-03-16T11:48:38.574104+0000","last_unstale":"2024-03-16T11:49:57.673861+0000","last_undegraded":"2024-03-16T11:49:57.673861+0000","last_fullsized":"2024-03-16T11:49:57.673861+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:42:20.588457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"87'8","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T11:49:57.669023+0000","last_change":"2024-03-16T11:48:38.575505+0000","last_active":"2024-03-16T11:49:57.669023+0000","last_peered":"2024-03-16T11:49:57.669023+0000","last_clean":"2024-03-16T11:49:57.669023+0000","last_became_active":"2024-03-16T11:48:38.575187+0000","last_became_peered":"2024-03-16T11:48:38.575187+0000","last_unstale":"2024-03-16T11:49:57.669023+0000","last_undegraded":"2024-03-16T11:49:57.669023+0000","last_fullsized":"2024-03-16T11:49:57.669023+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:13:06.260535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"87'10","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T11:49:57.673944+0000","last_change":"2024-03-16T11:48:38.568146+0000","last_active":"2024-03-16T11:49:57.673944+0000","last_peered":"2024-03-16T11:49:57.673944+0000","last_clean":"2024-03-16T11:49:57.673944+0000","last_became_active":"2024-03-16T11:48:38.567826+0000","last_became_peered":"2024-03-16T11:48:38.567826+0000","last_unstale":"2024-03-16T11:49:57.673944+0000","last_undegraded":"2024-03-16T11:49:57.673944+0000","last_fullsized":"2024-03-16T11:49:57.673944+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:46:32.993012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"87'7","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T11:49:57.668493+0000","last_change":"2024-03-16T11:48:38.578861+0000","last_active":"2024-03-16T11:49:57.668493+0000","last_peered":"2024-03-16T11:49:57.668493+0000","last_clean":"2024-03-16T11:49:57.668493+0000","last_became_active":"2024-03-16T11:48:38.577028+0000","last_became_peered":"2024-03-16T11:48:38.577028+0000","last_unstale":"2024-03-16T11:49:57.668493+0000","last_undegraded":"2024-03-16T11:49:57.668493+0000","last_fullsized":"2024-03-16T11:49:57.668493+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:55:34.520032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"85'9","reported_seq":29,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T11:49:28.731814+0000","last_change":"2024-03-16T11:48:38.026407+0000","last_active":"2024-03-16T11:49:28.731814+0000","last_peered":"2024-03-16T11:49:28.731814+0000","last_clean":"2024-03-16T11:49:28.731814+0000","last_became_active":"2024-03-16T11:48:38.026083+0000","last_became_peered":"2024-03-16T11:48:38.026083+0000","last_unstale":"2024-03-16T11:49:28.731814+0000","last_undegraded":"2024-03-16T11:49:28.731814+0000","last_fullsized":"2024-03-16T11:49:28.731814+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_clean_scrub_stamp":"2024-03-16T11:48:36.997377+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:01:23.751657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"87'5","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T11:49:57.668917+0000","last_change":"2024-03-16T11:48:38.573658+0000","last_active":"2024-03-16T11:49:57.668917+0000","last_peered":"2024-03-16T11:49:57.668917+0000","last_clean":"2024-03-16T11:49:57.668917+0000","last_became_active":"2024-03-16T11:48:38.572747+0000","last_became_peered":"2024-03-16T11:48:38.572747+0000","last_unstale":"2024-03-16T11:49:57.668917+0000","last_undegraded":"2024-03-16T11:49:57.668917+0000","last_fullsized":"2024-03-16T11:49:57.668917+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:09:11.128269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"85'2","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T11:49:28.636450+0000","last_change":"2024-03-16T11:48:38.027088+0000","last_active":"2024-03-16T11:49:28.636450+0000","last_peered":"2024-03-16T11:49:28.636450+0000","last_clean":"2024-03-16T11:49:28.636450+0000","last_became_active":"2024-03-16T11:48:38.026730+0000","last_became_peered":"2024-03-16T11:48:38.026730+0000","last_unstale":"2024-03-16T11:49:28.636450+0000","last_undegraded":"2024-03-16T11:49:28.636450+0000","last_fullsized":"2024-03-16T11:49:28.636450+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_clean_scrub_stamp":"2024-03-16T11:48:36.997377+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:23:28.987731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"87'6","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T11:49:57.667462+0000","last_change":"2024-03-16T11:48:38.583730+0000","last_active":"2024-03-16T11:49:57.667462+0000","last_peered":"2024-03-16T11:49:57.667462+0000","last_clean":"2024-03-16T11:49:57.667462+0000","last_became_active":"2024-03-16T11:48:38.583323+0000","last_became_peered":"2024-03-16T11:48:38.583323+0000","last_unstale":"2024-03-16T11:49:57.667462+0000","last_undegraded":"2024-03-16T11:49:57.667462+0000","last_fullsized":"2024-03-16T11:49:57.667462+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:35:20.923176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"85'4","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T11:49:28.643436+0000","last_change":"2024-03-16T11:49:28.642975+0000","last_active":"2024-03-16T11:49:28.643436+0000","last_peered":"2024-03-16T11:49:28.643436+0000","last_clean":"2024-03-16T11:49:28.643436+0000","last_became_active":"2024-03-16T11:49:28.637997+0000","last_became_peered":"2024-03-16T11:49:28.637997+0000","last_unstale":"2024-03-16T11:49:28.643436+0000","last_undegraded":"2024-03-16T11:49:28.643436+0000","last_fullsized":"2024-03-16T11:49:28.643436+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_clean_scrub_stamp":"2024-03-16T11:48:36.997377+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:24:15.401309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1c","version":"87'9","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T11:49:57.677119+0000","last_change":"2024-03-16T11:48:38.581452+0000","last_active":"2024-03-16T11:49:57.677119+0000","last_peered":"2024-03-16T11:49:57.677119+0000","last_clean":"2024-03-16T11:49:57.677119+0000","last_became_active":"2024-03-16T11:48:38.579305+0000","last_became_peered":"2024-03-16T11:48:38.579305+0000","last_unstale":"2024-03-16T11:49:57.677119+0000","last_undegraded":"2024-03-16T11:49:57.677119+0000","last_fullsized":"2024-03-16T11:49:57.677119+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:57:43.812783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T11:49:28.631814+0000","last_change":"2024-03-16T11:48:38.023747+0000","last_active":"2024-03-16T11:49:28.631814+0000","last_peered":"2024-03-16T11:49:28.631814+0000","last_clean":"2024-03-16T11:49:28.631814+0000","last_became_active":"2024-03-16T11:48:38.023396+0000","last_became_peered":"2024-03-16T11:48:38.023396+0000","last_unstale":"2024-03-16T11:49:28.631814+0000","last_undegraded":"2024-03-16T11:49:28.631814+0000","last_fullsized":"2024-03-16T11:49:28.631814+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_clean_scrub_stamp":"2024-03-16T11:48:36.997377+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:42:22.456357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"87'15","reported_seq":32,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T11:49:57.663227+0000","last_change":"2024-03-16T11:48:38.572157+0000","last_active":"2024-03-16T11:49:57.663227+0000","last_peered":"2024-03-16T11:49:57.663227+0000","last_clean":"2024-03-16T11:49:57.663227+0000","last_became_active":"2024-03-16T11:48:38.571629+0000","last_became_peered":"2024-03-16T11:48:38.571629+0000","last_unstale":"2024-03-16T11:49:57.663227+0000","last_undegraded":"2024-03-16T11:49:57.663227+0000","last_fullsized":"2024-03-16T11:49:57.663227+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:47:39.353965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"87'7","reported_seq":121,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T11:49:59.674211+0000","last_change":"2024-03-16T11:49:28.644872+0000","last_active":"2024-03-16T11:49:59.674211+0000","last_peered":"2024-03-16T11:49:59.674211+0000","last_clean":"2024-03-16T11:49:59.674211+0000","last_became_active":"2024-03-16T11:49:28.638836+0000","last_became_peered":"2024-03-16T11:49:28.638836+0000","last_unstale":"2024-03-16T11:49:59.674211+0000","last_undegraded":"2024-03-16T11:49:59.674211+0000","last_fullsized":"2024-03-16T11:49:59.674211+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_clean_scrub_stamp":"2024-03-16T11:48:36.997377+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:38:42.689838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1154,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":82,"num_read_kb":4,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1013,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"87'7","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T11:49:57.662375+0000","last_change":"2024-03-16T11:48:38.574227+0000","last_active":"2024-03-16T11:49:57.662375+0000","last_peered":"2024-03-16T11:49:57.662375+0000","last_clean":"2024-03-16T11:49:57.662375+0000","last_became_active":"2024-03-16T11:48:38.571938+0000","last_became_peered":"2024-03-16T11:48:38.571938+0000","last_unstale":"2024-03-16T11:49:57.662375+0000","last_undegraded":"2024-03-16T11:49:57.662375+0000","last_fullsized":"2024-03-16T11:49:57.662375+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:50:35.992068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"85'3","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T11:49:28.634815+0000","last_change":"2024-03-16T11:48:38.024448+0000","last_active":"2024-03-16T11:49:28.634815+0000","last_peered":"2024-03-16T11:49:28.634815+0000","last_clean":"2024-03-16T11:49:28.634815+0000","last_became_active":"2024-03-16T11:48:38.024109+0000","last_became_peered":"2024-03-16T11:48:38.024109+0000","last_unstale":"2024-03-16T11:49:28.634815+0000","last_undegraded":"2024-03-16T11:49:28.634815+0000","last_fullsized":"2024-03-16T11:49:28.634815+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_clean_scrub_stamp":"2024-03-16T11:48:36.997377+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:37:49.521297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"87'8","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T11:49:57.665301+0000","last_change":"2024-03-16T11:48:38.577722+0000","last_active":"2024-03-16T11:49:57.665301+0000","last_peered":"2024-03-16T11:49:57.665301+0000","last_clean":"2024-03-16T11:49:57.665301+0000","last_became_active":"2024-03-16T11:48:38.577223+0000","last_became_peered":"2024-03-16T11:48:38.577223+0000","last_unstale":"2024-03-16T11:49:57.665301+0000","last_undegraded":"2024-03-16T11:49:57.665301+0000","last_fullsized":"2024-03-16T11:49:57.665301+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:24:05.892850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"85'2","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T11:49:28.635314+0000","last_change":"2024-03-16T11:48:38.029681+0000","last_active":"2024-03-16T11:49:28.635314+0000","last_peered":"2024-03-16T11:49:28.635314+0000","last_clean":"2024-03-16T11:49:28.635314+0000","last_became_active":"2024-03-16T11:48:38.029261+0000","last_became_peered":"2024-03-16T11:48:38.029261+0000","last_unstale":"2024-03-16T11:49:28.635314+0000","last_undegraded":"2024-03-16T11:49:28.635314+0000","last_fullsized":"2024-03-16T11:49:28.635314+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_clean_scrub_stamp":"2024-03-16T11:48:36.997377+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:57:51.155337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"87'8","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T11:49:57.671216+0000","last_change":"2024-03-16T11:48:38.570655+0000","last_active":"2024-03-16T11:49:57.671216+0000","last_peered":"2024-03-16T11:49:57.671216+0000","last_clean":"2024-03-16T11:49:57.671216+0000","last_became_active":"2024-03-16T11:48:38.569143+0000","last_became_peered":"2024-03-16T11:48:38.569143+0000","last_unstale":"2024-03-16T11:49:57.671216+0000","last_undegraded":"2024-03-16T11:49:57.671216+0000","last_fullsized":"2024-03-16T11:49:57.671216+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:51:51.449168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"85'1","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T11:49:28.634042+0000","last_change":"2024-03-16T11:48:38.027074+0000","last_active":"2024-03-16T11:49:28.634042+0000","last_peered":"2024-03-16T11:49:28.634042+0000","last_clean":"2024-03-16T11:49:28.634042+0000","last_became_active":"2024-03-16T11:48:38.026749+0000","last_became_peered":"2024-03-16T11:48:38.026749+0000","last_unstale":"2024-03-16T11:49:28.634042+0000","last_undegraded":"2024-03-16T11:49:28.634042+0000","last_fullsized":"2024-03-16T11:49:28.634042+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_clean_scrub_stamp":"2024-03-16T11:48:36.997377+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:14:26.801688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"87'11","reported_seq":28,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T11:49:57.654587+0000","last_change":"2024-03-16T11:48:38.575699+0000","last_active":"2024-03-16T11:49:57.654587+0000","last_peered":"2024-03-16T11:49:57.654587+0000","last_clean":"2024-03-16T11:49:57.654587+0000","last_became_active":"2024-03-16T11:48:38.575401+0000","last_became_peered":"2024-03-16T11:48:38.575401+0000","last_unstale":"2024-03-16T11:49:57.654587+0000","last_undegraded":"2024-03-16T11:49:57.654587+0000","last_fullsized":"2024-03-16T11:49:57.654587+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:48:06.289779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"87'4","reported_seq":105,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T11:50:02.677050+0000","last_change":"2024-03-16T11:48:38.026488+0000","last_active":"2024-03-16T11:50:02.677050+0000","last_peered":"2024-03-16T11:50:02.677050+0000","last_clean":"2024-03-16T11:50:02.677050+0000","last_became_active":"2024-03-16T11:48:38.026101+0000","last_became_peered":"2024-03-16T11:48:38.026101+0000","last_unstale":"2024-03-16T11:50:02.677050+0000","last_undegraded":"2024-03-16T11:50:02.677050+0000","last_fullsized":"2024-03-16T11:50:02.677050+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_clean_scrub_stamp":"2024-03-16T11:48:36.997377+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:37:29.788307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":82,"num_read_kb":82,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"87'10","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T11:49:57.675842+0000","last_change":"2024-03-16T11:48:38.578081+0000","last_active":"2024-03-16T11:49:57.675842+0000","last_peered":"2024-03-16T11:49:57.675842+0000","last_clean":"2024-03-16T11:49:57.675842+0000","last_became_active":"2024-03-16T11:48:38.577742+0000","last_became_peered":"2024-03-16T11:48:38.577742+0000","last_unstale":"2024-03-16T11:49:57.675842+0000","last_undegraded":"2024-03-16T11:49:57.675842+0000","last_fullsized":"2024-03-16T11:49:57.675842+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:58:41.479658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"85'2","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T11:49:28.632342+0000","last_change":"2024-03-16T11:48:38.026891+0000","last_active":"2024-03-16T11:49:28.632342+0000","last_peered":"2024-03-16T11:49:28.632342+0000","last_clean":"2024-03-16T11:49:28.632342+0000","last_became_active":"2024-03-16T11:48:38.026397+0000","last_became_peered":"2024-03-16T11:48:38.026397+0000","last_unstale":"2024-03-16T11:49:28.632342+0000","last_undegraded":"2024-03-16T11:49:28.632342+0000","last_fullsized":"2024-03-16T11:49:28.632342+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_clean_scrub_stamp":"2024-03-16T11:48:36.997377+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:02:01.018103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"87'9","reported_seq":32,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T11:49:57.668593+0000","last_change":"2024-03-16T11:49:28.638309+0000","last_active":"2024-03-16T11:49:57.668593+0000","last_peered":"2024-03-16T11:49:57.668593+0000","last_clean":"2024-03-16T11:49:57.668593+0000","last_became_active":"2024-03-16T11:49:28.637786+0000","last_became_peered":"2024-03-16T11:49:28.637786+0000","last_unstale":"2024-03-16T11:49:57.668593+0000","last_undegraded":"2024-03-16T11:49:57.668593+0000","last_fullsized":"2024-03-16T11:49:57.668593+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:32:43.043447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,5],"acting":[0,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"85'4","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T11:49:28.634965+0000","last_change":"2024-03-16T11:48:38.025103+0000","last_active":"2024-03-16T11:49:28.634965+0000","last_peered":"2024-03-16T11:49:28.634965+0000","last_clean":"2024-03-16T11:49:28.634965+0000","last_became_active":"2024-03-16T11:48:38.024754+0000","last_became_peered":"2024-03-16T11:48:38.024754+0000","last_unstale":"2024-03-16T11:49:28.634965+0000","last_undegraded":"2024-03-16T11:49:28.634965+0000","last_fullsized":"2024-03-16T11:49:28.634965+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_clean_scrub_stamp":"2024-03-16T11:48:36.997377+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:40:10.335339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"87'11","reported_seq":28,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T11:49:57.676305+0000","last_change":"2024-03-16T11:48:38.578305+0000","last_active":"2024-03-16T11:49:57.676305+0000","last_peered":"2024-03-16T11:49:57.676305+0000","last_clean":"2024-03-16T11:49:57.676305+0000","last_became_active":"2024-03-16T11:48:38.577971+0000","last_became_peered":"2024-03-16T11:48:38.577971+0000","last_unstale":"2024-03-16T11:49:57.676305+0000","last_undegraded":"2024-03-16T11:49:57.676305+0000","last_fullsized":"2024-03-16T11:49:57.676305+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:04:34.259456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"87'32","reported_seq":242,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T11:50:03.771191+0000","last_change":"2024-03-16T11:49:28.739284+0000","last_active":"2024-03-16T11:50:03.771191+0000","last_peered":"2024-03-16T11:50:03.771191+0000","last_clean":"2024-03-16T11:50:03.771191+0000","last_became_active":"2024-03-16T11:49:28.732510+0000","last_became_peered":"2024-03-16T11:49:28.732510+0000","last_unstale":"2024-03-16T11:50:03.771191+0000","last_undegraded":"2024-03-16T11:50:03.771191+0000","last_fullsized":"2024-03-16T11:50:03.771191+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_clean_scrub_stamp":"2024-03-16T11:48:36.997377+0000","objects_scrubbed":0,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:12:16.286605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":60272,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":171,"num_read_kb":144,"num_write":33,"num_write_kb":73,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":60023,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,4],"acting":[10,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"87'6","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T11:49:57.654869+0000","last_change":"2024-03-16T11:48:38.574404+0000","last_active":"2024-03-16T11:49:57.654869+0000","last_peered":"2024-03-16T11:49:57.654869+0000","last_clean":"2024-03-16T11:49:57.654869+0000","last_became_active":"2024-03-16T11:48:38.572334+0000","last_became_peered":"2024-03-16T11:48:38.572334+0000","last_unstale":"2024-03-16T11:49:57.654869+0000","last_undegraded":"2024-03-16T11:49:57.654869+0000","last_fullsized":"2024-03-16T11:49:57.654869+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:25:37.442152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"85'6","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T11:49:28.636242+0000","last_change":"2024-03-16T11:48:38.025287+0000","last_active":"2024-03-16T11:49:28.636242+0000","last_peered":"2024-03-16T11:49:28.636242+0000","last_clean":"2024-03-16T11:49:28.636242+0000","last_became_active":"2024-03-16T11:48:38.022433+0000","last_became_peered":"2024-03-16T11:48:38.022433+0000","last_unstale":"2024-03-16T11:49:28.636242+0000","last_undegraded":"2024-03-16T11:49:28.636242+0000","last_fullsized":"2024-03-16T11:49:28.636242+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_clean_scrub_stamp":"2024-03-16T11:48:36.997377+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:27:20.573925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"87'9","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T11:49:57.666290+0000","last_change":"2024-03-16T11:48:38.574715+0000","last_active":"2024-03-16T11:49:57.666290+0000","last_peered":"2024-03-16T11:49:57.666290+0000","last_clean":"2024-03-16T11:49:57.666290+0000","last_became_active":"2024-03-16T11:48:38.574456+0000","last_became_peered":"2024-03-16T11:48:38.574456+0000","last_unstale":"2024-03-16T11:49:57.666290+0000","last_undegraded":"2024-03-16T11:49:57.666290+0000","last_fullsized":"2024-03-16T11:49:57.666290+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:57:00.123583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"85'3","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T11:49:28.634578+0000","last_change":"2024-03-16T11:48:38.029557+0000","last_active":"2024-03-16T11:49:28.634578+0000","last_peered":"2024-03-16T11:49:28.634578+0000","last_clean":"2024-03-16T11:49:28.634578+0000","last_became_active":"2024-03-16T11:48:38.029249+0000","last_became_peered":"2024-03-16T11:48:38.029249+0000","last_unstale":"2024-03-16T11:49:28.634578+0000","last_undegraded":"2024-03-16T11:49:28.634578+0000","last_fullsized":"2024-03-16T11:49:28.634578+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_clean_scrub_stamp":"2024-03-16T11:48:36.997377+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:32:41.364467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"87'15","reported_seq":38,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T11:49:57.666085+0000","last_change":"2024-03-16T11:49:28.732430+0000","last_active":"2024-03-16T11:49:57.666085+0000","last_peered":"2024-03-16T11:49:57.666085+0000","last_clean":"2024-03-16T11:49:57.666085+0000","last_became_active":"2024-03-16T11:49:28.732041+0000","last_became_peered":"2024-03-16T11:49:28.732041+0000","last_unstale":"2024-03-16T11:49:57.666085+0000","last_undegraded":"2024-03-16T11:49:57.666085+0000","last_fullsized":"2024-03-16T11:49:57.666085+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:39:20.346214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,6],"acting":[7,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"87'5","reported_seq":108,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T11:50:01.072230+0000","last_change":"2024-03-16T11:48:38.027047+0000","last_active":"2024-03-16T11:50:01.072230+0000","last_peered":"2024-03-16T11:50:01.072230+0000","last_clean":"2024-03-16T11:50:01.072230+0000","last_became_active":"2024-03-16T11:48:38.026733+0000","last_became_peered":"2024-03-16T11:48:38.026733+0000","last_unstale":"2024-03-16T11:50:01.072230+0000","last_undegraded":"2024-03-16T11:50:01.072230+0000","last_fullsized":"2024-03-16T11:50:01.072230+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_clean_scrub_stamp":"2024-03-16T11:48:36.997377+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:27:23.763104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1120,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":85,"num_read_kb":5,"num_write":4,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"87'6","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T11:49:57.662127+0000","last_change":"2024-03-16T11:48:38.568802+0000","last_active":"2024-03-16T11:49:57.662127+0000","last_peered":"2024-03-16T11:49:57.662127+0000","last_clean":"2024-03-16T11:49:57.662127+0000","last_became_active":"2024-03-16T11:48:38.568213+0000","last_became_peered":"2024-03-16T11:48:38.568213+0000","last_unstale":"2024-03-16T11:49:57.662127+0000","last_undegraded":"2024-03-16T11:49:57.662127+0000","last_fullsized":"2024-03-16T11:49:57.662127+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:38:23.233563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"85'15","reported_seq":35,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T11:49:28.631457+0000","last_change":"2024-03-16T11:48:38.027596+0000","last_active":"2024-03-16T11:49:28.631457+0000","last_peered":"2024-03-16T11:49:28.631457+0000","last_clean":"2024-03-16T11:49:28.631457+0000","last_became_active":"2024-03-16T11:48:38.027280+0000","last_became_peered":"2024-03-16T11:48:38.027280+0000","last_unstale":"2024-03-16T11:49:28.631457+0000","last_undegraded":"2024-03-16T11:49:28.631457+0000","last_fullsized":"2024-03-16T11:49:28.631457+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_clean_scrub_stamp":"2024-03-16T11:48:36.997377+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:05:26.409613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"87'6","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T11:49:57.662685+0000","last_change":"2024-03-16T11:48:38.571308+0000","last_active":"2024-03-16T11:49:57.662685+0000","last_peered":"2024-03-16T11:49:57.662685+0000","last_clean":"2024-03-16T11:49:57.662685+0000","last_became_active":"2024-03-16T11:48:38.570927+0000","last_became_peered":"2024-03-16T11:48:38.570927+0000","last_unstale":"2024-03-16T11:49:57.662685+0000","last_undegraded":"2024-03-16T11:49:57.662685+0000","last_fullsized":"2024-03-16T11:49:57.662685+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:15:18.190361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"85'2","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T11:49:28.631944+0000","last_change":"2024-03-16T11:48:38.024783+0000","last_active":"2024-03-16T11:49:28.631944+0000","last_peered":"2024-03-16T11:49:28.631944+0000","last_clean":"2024-03-16T11:49:28.631944+0000","last_became_active":"2024-03-16T11:48:38.023990+0000","last_became_peered":"2024-03-16T11:48:38.023990+0000","last_unstale":"2024-03-16T11:49:28.631944+0000","last_undegraded":"2024-03-16T11:49:28.631944+0000","last_fullsized":"2024-03-16T11:49:28.631944+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_clean_scrub_stamp":"2024-03-16T11:48:36.997377+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:42:49.375788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"87'5","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T11:49:57.660582+0000","last_change":"2024-03-16T11:48:38.580019+0000","last_active":"2024-03-16T11:49:57.660582+0000","last_peered":"2024-03-16T11:49:57.660582+0000","last_clean":"2024-03-16T11:49:57.660582+0000","last_became_active":"2024-03-16T11:48:38.579686+0000","last_became_peered":"2024-03-16T11:48:38.579686+0000","last_unstale":"2024-03-16T11:49:57.660582+0000","last_undegraded":"2024-03-16T11:49:57.660582+0000","last_fullsized":"2024-03-16T11:49:57.660582+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:00:21.752048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":19,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T11:49:28.731593+0000","last_change":"2024-03-16T11:48:38.017631+0000","last_active":"2024-03-16T11:49:28.731593+0000","last_peered":"2024-03-16T11:49:28.731593+0000","last_clean":"2024-03-16T11:49:28.731593+0000","last_became_active":"2024-03-16T11:48:38.017218+0000","last_became_peered":"2024-03-16T11:48:38.017218+0000","last_unstale":"2024-03-16T11:49:28.731593+0000","last_undegraded":"2024-03-16T11:49:28.731593+0000","last_fullsized":"2024-03-16T11:49:28.731593+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_clean_scrub_stamp":"2024-03-16T11:48:36.997377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:31:41.378757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"87'5","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T11:49:57.649050+0000","last_change":"2024-03-16T11:48:38.576751+0000","last_active":"2024-03-16T11:49:57.649050+0000","last_peered":"2024-03-16T11:49:57.649050+0000","last_clean":"2024-03-16T11:49:57.649050+0000","last_became_active":"2024-03-16T11:48:38.576442+0000","last_became_peered":"2024-03-16T11:48:38.576442+0000","last_unstale":"2024-03-16T11:49:57.649050+0000","last_undegraded":"2024-03-16T11:49:57.649050+0000","last_fullsized":"2024-03-16T11:49:57.649050+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:49:59.475427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"85'1","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T11:49:28.632243+0000","last_change":"2024-03-16T11:48:38.030159+0000","last_active":"2024-03-16T11:49:28.632243+0000","last_peered":"2024-03-16T11:49:28.632243+0000","last_clean":"2024-03-16T11:49:28.632243+0000","last_became_active":"2024-03-16T11:48:38.029808+0000","last_became_peered":"2024-03-16T11:48:38.029808+0000","last_unstale":"2024-03-16T11:49:28.632243+0000","last_undegraded":"2024-03-16T11:49:28.632243+0000","last_fullsized":"2024-03-16T11:49:28.632243+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_clean_scrub_stamp":"2024-03-16T11:48:36.997377+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:46:54.692837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"87'7","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T11:49:57.651285+0000","last_change":"2024-03-16T11:48:38.574080+0000","last_active":"2024-03-16T11:49:57.651285+0000","last_peered":"2024-03-16T11:49:57.651285+0000","last_clean":"2024-03-16T11:49:57.651285+0000","last_became_active":"2024-03-16T11:48:38.573745+0000","last_became_peered":"2024-03-16T11:48:38.573745+0000","last_unstale":"2024-03-16T11:49:57.651285+0000","last_undegraded":"2024-03-16T11:49:57.651285+0000","last_fullsized":"2024-03-16T11:49:57.651285+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:12:56.154804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"87'8","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T11:49:57.662455+0000","last_change":"2024-03-16T11:48:38.571750+0000","last_active":"2024-03-16T11:49:57.662455+0000","last_peered":"2024-03-16T11:49:57.662455+0000","last_clean":"2024-03-16T11:49:57.662455+0000","last_became_active":"2024-03-16T11:48:38.571273+0000","last_became_peered":"2024-03-16T11:48:38.571273+0000","last_unstale":"2024-03-16T11:49:57.662455+0000","last_undegraded":"2024-03-16T11:49:57.662455+0000","last_fullsized":"2024-03-16T11:49:57.662455+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:59:21.952927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"87'5","reported_seq":105,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T11:50:02.072963+0000","last_change":"2024-03-16T11:48:38.034121+0000","last_active":"2024-03-16T11:50:02.072963+0000","last_peered":"2024-03-16T11:50:02.072963+0000","last_clean":"2024-03-16T11:50:02.072963+0000","last_became_active":"2024-03-16T11:48:38.033777+0000","last_became_peered":"2024-03-16T11:48:38.033777+0000","last_unstale":"2024-03-16T11:50:02.072963+0000","last_undegraded":"2024-03-16T11:50:02.072963+0000","last_fullsized":"2024-03-16T11:50:02.072963+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_clean_scrub_stamp":"2024-03-16T11:48:36.997377+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:51:54.637120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":81,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"87'8","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T11:49:57.670058+0000","last_change":"2024-03-16T11:48:38.580050+0000","last_active":"2024-03-16T11:49:57.670058+0000","last_peered":"2024-03-16T11:49:57.670058+0000","last_clean":"2024-03-16T11:49:57.670058+0000","last_became_active":"2024-03-16T11:48:38.578950+0000","last_became_peered":"2024-03-16T11:48:38.578950+0000","last_unstale":"2024-03-16T11:49:57.670058+0000","last_undegraded":"2024-03-16T11:49:57.670058+0000","last_fullsized":"2024-03-16T11:49:57.670058+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:10:21.717916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"85'3","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T11:49:28.731215+0000","last_change":"2024-03-16T11:48:38.023925+0000","last_active":"2024-03-16T11:49:28.731215+0000","last_peered":"2024-03-16T11:49:28.731215+0000","last_clean":"2024-03-16T11:49:28.731215+0000","last_became_active":"2024-03-16T11:48:38.023424+0000","last_became_peered":"2024-03-16T11:48:38.023424+0000","last_unstale":"2024-03-16T11:49:28.731215+0000","last_undegraded":"2024-03-16T11:49:28.731215+0000","last_fullsized":"2024-03-16T11:49:28.731215+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_clean_scrub_stamp":"2024-03-16T11:48:36.997377+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:01:53.085039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":354,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T11:49:28.632545+0000","last_change":"2024-03-16T11:47:22.001149+0000","last_active":"2024-03-16T11:49:28.632545+0000","last_peered":"2024-03-16T11:49:28.632545+0000","last_clean":"2024-03-16T11:49:28.632545+0000","last_became_active":"2024-03-16T11:47:22.000506+0000","last_became_peered":"2024-03-16T11:47:22.000506+0000","last_unstale":"2024-03-16T11:49:28.632545+0000","last_undegraded":"2024-03-16T11:49:28.632545+0000","last_fullsized":"2024-03-16T11:49:28.632545+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:43:52.376552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:43:52.376552+0000","last_clean_scrub_stamp":"2024-03-16T11:43:52.376552+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:21:18.466389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"87'5","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T11:49:57.659224+0000","last_change":"2024-03-16T11:48:38.577530+0000","last_active":"2024-03-16T11:49:57.659224+0000","last_peered":"2024-03-16T11:49:57.659224+0000","last_clean":"2024-03-16T11:49:57.659224+0000","last_became_active":"2024-03-16T11:48:38.577252+0000","last_became_peered":"2024-03-16T11:48:38.577252+0000","last_unstale":"2024-03-16T11:49:57.659224+0000","last_undegraded":"2024-03-16T11:49:57.659224+0000","last_fullsized":"2024-03-16T11:49:57.659224+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:07:47.809075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"85'3","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T11:49:28.632293+0000","last_change":"2024-03-16T11:48:38.027630+0000","last_active":"2024-03-16T11:49:28.632293+0000","last_peered":"2024-03-16T11:49:28.632293+0000","last_clean":"2024-03-16T11:49:28.632293+0000","last_became_active":"2024-03-16T11:48:38.027105+0000","last_became_peered":"2024-03-16T11:48:38.027105+0000","last_unstale":"2024-03-16T11:49:28.632293+0000","last_undegraded":"2024-03-16T11:49:28.632293+0000","last_fullsized":"2024-03-16T11:49:28.632293+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_clean_scrub_stamp":"2024-03-16T11:48:36.997377+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T12:24:20.605988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"85'1","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T11:49:28.631706+0000","last_change":"2024-03-16T11:48:38.032419+0000","last_active":"2024-03-16T11:49:28.631706+0000","last_peered":"2024-03-16T11:49:28.631706+0000","last_clean":"2024-03-16T11:49:28.631706+0000","last_became_active":"2024-03-16T11:48:38.032087+0000","last_became_peered":"2024-03-16T11:48:38.032087+0000","last_unstale":"2024-03-16T11:49:28.631706+0000","last_undegraded":"2024-03-16T11:49:28.631706+0000","last_fullsized":"2024-03-16T11:49:28.631706+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_clean_scrub_stamp":"2024-03-16T11:48:36.997377+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:18:54.459321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"87'9","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T11:49:57.661272+0000","last_change":"2024-03-16T11:48:38.568768+0000","last_active":"2024-03-16T11:49:57.661272+0000","last_peered":"2024-03-16T11:49:57.661272+0000","last_clean":"2024-03-16T11:49:57.661272+0000","last_became_active":"2024-03-16T11:48:38.568408+0000","last_became_peered":"2024-03-16T11:48:38.568408+0000","last_unstale":"2024-03-16T11:49:57.661272+0000","last_undegraded":"2024-03-16T11:49:57.661272+0000","last_fullsized":"2024-03-16T11:49:57.661272+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:10:33.293464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"87'9","reported_seq":28,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T11:49:28.731790+0000","last_change":"2024-03-16T11:48:38.023387+0000","last_active":"2024-03-16T11:49:28.731790+0000","last_peered":"2024-03-16T11:49:28.731790+0000","last_clean":"2024-03-16T11:49:28.731790+0000","last_became_active":"2024-03-16T11:48:38.022916+0000","last_became_peered":"2024-03-16T11:48:38.022916+0000","last_unstale":"2024-03-16T11:49:28.731790+0000","last_undegraded":"2024-03-16T11:49:28.731790+0000","last_fullsized":"2024-03-16T11:49:28.731790+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_clean_scrub_stamp":"2024-03-16T11:48:36.997377+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:25:41.478737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"87'5","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T11:49:57.661696+0000","last_change":"2024-03-16T11:48:38.572243+0000","last_active":"2024-03-16T11:49:57.661696+0000","last_peered":"2024-03-16T11:49:57.661696+0000","last_clean":"2024-03-16T11:49:57.661696+0000","last_became_active":"2024-03-16T11:48:38.571918+0000","last_became_peered":"2024-03-16T11:48:38.571918+0000","last_unstale":"2024-03-16T11:49:57.661696+0000","last_undegraded":"2024-03-16T11:49:57.661696+0000","last_fullsized":"2024-03-16T11:49:57.661696+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:21:45.843298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"85'4","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T11:49:28.731270+0000","last_change":"2024-03-16T11:48:38.022825+0000","last_active":"2024-03-16T11:49:28.731270+0000","last_peered":"2024-03-16T11:49:28.731270+0000","last_clean":"2024-03-16T11:49:28.731270+0000","last_became_active":"2024-03-16T11:48:38.022499+0000","last_became_peered":"2024-03-16T11:48:38.022499+0000","last_unstale":"2024-03-16T11:49:28.731270+0000","last_undegraded":"2024-03-16T11:49:28.731270+0000","last_fullsized":"2024-03-16T11:49:28.731270+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_clean_scrub_stamp":"2024-03-16T11:48:36.997377+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:52:51.810960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"87'7","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T11:49:57.668985+0000","last_change":"2024-03-16T11:48:38.577473+0000","last_active":"2024-03-16T11:49:57.668985+0000","last_peered":"2024-03-16T11:49:57.668985+0000","last_clean":"2024-03-16T11:49:57.668985+0000","last_became_active":"2024-03-16T11:48:38.577121+0000","last_became_peered":"2024-03-16T11:48:38.577121+0000","last_unstale":"2024-03-16T11:49:57.668985+0000","last_undegraded":"2024-03-16T11:49:57.668985+0000","last_fullsized":"2024-03-16T11:49:57.668985+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:21:06.905785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"85'2","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T11:49:28.637444+0000","last_change":"2024-03-16T11:48:38.020324+0000","last_active":"2024-03-16T11:49:28.637444+0000","last_peered":"2024-03-16T11:49:28.637444+0000","last_clean":"2024-03-16T11:49:28.637444+0000","last_became_active":"2024-03-16T11:48:38.019692+0000","last_became_peered":"2024-03-16T11:48:38.019692+0000","last_unstale":"2024-03-16T11:49:28.637444+0000","last_undegraded":"2024-03-16T11:49:28.637444+0000","last_fullsized":"2024-03-16T11:49:28.637444+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_clean_scrub_stamp":"2024-03-16T11:48:36.997377+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:42:51.709046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"87'2","reported_seq":19,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T11:49:57.623612+0000","last_change":"2024-03-16T11:48:38.570911+0000","last_active":"2024-03-16T11:49:57.623612+0000","last_peered":"2024-03-16T11:49:57.623612+0000","last_clean":"2024-03-16T11:49:57.623612+0000","last_became_active":"2024-03-16T11:48:38.570288+0000","last_became_peered":"2024-03-16T11:48:38.570288+0000","last_unstale":"2024-03-16T11:49:57.623612+0000","last_undegraded":"2024-03-16T11:49:57.623612+0000","last_fullsized":"2024-03-16T11:49:57.623612+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:34:20.516104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"85'3","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T11:49:28.635756+0000","last_change":"2024-03-16T11:48:38.021817+0000","last_active":"2024-03-16T11:49:28.635756+0000","last_peered":"2024-03-16T11:49:28.635756+0000","last_clean":"2024-03-16T11:49:28.635756+0000","last_became_active":"2024-03-16T11:48:38.020906+0000","last_became_peered":"2024-03-16T11:48:38.020906+0000","last_unstale":"2024-03-16T11:49:28.635756+0000","last_undegraded":"2024-03-16T11:49:28.635756+0000","last_fullsized":"2024-03-16T11:49:28.635756+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_clean_scrub_stamp":"2024-03-16T11:48:36.997377+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:09:53.372698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"87'9","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T11:49:57.668407+0000","last_change":"2024-03-16T11:48:38.573536+0000","last_active":"2024-03-16T11:49:57.668407+0000","last_peered":"2024-03-16T11:49:57.668407+0000","last_clean":"2024-03-16T11:49:57.668407+0000","last_became_active":"2024-03-16T11:48:38.573211+0000","last_became_peered":"2024-03-16T11:48:38.573211+0000","last_unstale":"2024-03-16T11:49:57.668407+0000","last_undegraded":"2024-03-16T11:49:57.668407+0000","last_fullsized":"2024-03-16T11:49:57.668407+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:53:02.607175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"85'1","reported_seq":15,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T11:49:28.645934+0000","last_change":"2024-03-16T11:49:28.645457+0000","last_active":"2024-03-16T11:49:28.645934+0000","last_peered":"2024-03-16T11:49:28.645934+0000","last_clean":"2024-03-16T11:49:28.645934+0000","last_became_active":"2024-03-16T11:49:28.637962+0000","last_became_peered":"2024-03-16T11:49:28.637962+0000","last_unstale":"2024-03-16T11:49:28.645934+0000","last_undegraded":"2024-03-16T11:49:28.645934+0000","last_fullsized":"2024-03-16T11:49:28.645934+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_clean_scrub_stamp":"2024-03-16T11:48:36.997377+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:53:38.317556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,8],"acting":[11,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.b","version":"87'8","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T11:49:57.662515+0000","last_change":"2024-03-16T11:48:38.575965+0000","last_active":"2024-03-16T11:49:57.662515+0000","last_peered":"2024-03-16T11:49:57.662515+0000","last_clean":"2024-03-16T11:49:57.662515+0000","last_became_active":"2024-03-16T11:48:38.575627+0000","last_became_peered":"2024-03-16T11:48:38.575627+0000","last_unstale":"2024-03-16T11:49:57.662515+0000","last_undegraded":"2024-03-16T11:49:57.662515+0000","last_fullsized":"2024-03-16T11:49:57.662515+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:00:37.768897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"85'4","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T11:49:28.631560+0000","last_change":"2024-03-16T11:48:38.027516+0000","last_active":"2024-03-16T11:49:28.631560+0000","last_peered":"2024-03-16T11:49:28.631560+0000","last_clean":"2024-03-16T11:49:28.631560+0000","last_became_active":"2024-03-16T11:48:38.026865+0000","last_became_peered":"2024-03-16T11:48:38.026865+0000","last_unstale":"2024-03-16T11:49:28.631560+0000","last_undegraded":"2024-03-16T11:49:28.631560+0000","last_fullsized":"2024-03-16T11:49:28.631560+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_clean_scrub_stamp":"2024-03-16T11:48:36.997377+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:50:50.240973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"87'5","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T11:49:57.676209+0000","last_change":"2024-03-16T11:48:38.571242+0000","last_active":"2024-03-16T11:49:57.676209+0000","last_peered":"2024-03-16T11:49:57.676209+0000","last_clean":"2024-03-16T11:49:57.676209+0000","last_became_active":"2024-03-16T11:48:38.570917+0000","last_became_peered":"2024-03-16T11:48:38.570917+0000","last_unstale":"2024-03-16T11:49:57.676209+0000","last_undegraded":"2024-03-16T11:49:57.676209+0000","last_fullsized":"2024-03-16T11:49:57.676209+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:30:12.601982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"85'3","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T11:49:28.636508+0000","last_change":"2024-03-16T11:48:38.023970+0000","last_active":"2024-03-16T11:49:28.636508+0000","last_peered":"2024-03-16T11:49:28.636508+0000","last_clean":"2024-03-16T11:49:28.636508+0000","last_became_active":"2024-03-16T11:48:38.023646+0000","last_became_peered":"2024-03-16T11:48:38.023646+0000","last_unstale":"2024-03-16T11:49:28.636508+0000","last_undegraded":"2024-03-16T11:49:28.636508+0000","last_fullsized":"2024-03-16T11:49:28.636508+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_clean_scrub_stamp":"2024-03-16T11:48:36.997377+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:48:35.549588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"87'7","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T11:49:57.676353+0000","last_change":"2024-03-16T11:48:38.567376+0000","last_active":"2024-03-16T11:49:57.676353+0000","last_peered":"2024-03-16T11:49:57.676353+0000","last_clean":"2024-03-16T11:49:57.676353+0000","last_became_active":"2024-03-16T11:48:38.567048+0000","last_became_peered":"2024-03-16T11:48:38.567048+0000","last_unstale":"2024-03-16T11:49:57.676353+0000","last_undegraded":"2024-03-16T11:49:57.676353+0000","last_fullsized":"2024-03-16T11:49:57.676353+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:45:50.097013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"84'1","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T11:49:28.634840+0000","last_change":"2024-03-16T11:48:38.029192+0000","last_active":"2024-03-16T11:49:28.634840+0000","last_peered":"2024-03-16T11:49:28.634840+0000","last_clean":"2024-03-16T11:49:28.634840+0000","last_became_active":"2024-03-16T11:48:38.028866+0000","last_became_peered":"2024-03-16T11:48:38.028866+0000","last_unstale":"2024-03-16T11:49:28.634840+0000","last_undegraded":"2024-03-16T11:49:28.634840+0000","last_fullsized":"2024-03-16T11:49:28.634840+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_clean_scrub_stamp":"2024-03-16T11:48:36.997377+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:52:31.329863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"87'7","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T11:49:57.648434+0000","last_change":"2024-03-16T11:48:38.576538+0000","last_active":"2024-03-16T11:49:57.648434+0000","last_peered":"2024-03-16T11:49:57.648434+0000","last_clean":"2024-03-16T11:49:57.648434+0000","last_became_active":"2024-03-16T11:48:38.576222+0000","last_became_peered":"2024-03-16T11:48:38.576222+0000","last_unstale":"2024-03-16T11:49:57.648434+0000","last_undegraded":"2024-03-16T11:49:57.648434+0000","last_fullsized":"2024-03-16T11:49:57.648434+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:34:13.186406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T11:49:28.635452+0000","last_change":"2024-03-16T11:48:38.031002+0000","last_active":"2024-03-16T11:49:28.635452+0000","last_peered":"2024-03-16T11:49:28.635452+0000","last_clean":"2024-03-16T11:49:28.635452+0000","last_became_active":"2024-03-16T11:48:38.030653+0000","last_became_peered":"2024-03-16T11:48:38.030653+0000","last_unstale":"2024-03-16T11:49:28.635452+0000","last_undegraded":"2024-03-16T11:49:28.635452+0000","last_fullsized":"2024-03-16T11:49:28.635452+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:36.997377+0000","last_clean_scrub_stamp":"2024-03-16T11:48:36.997377+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:42:30.148594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"87'9","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T11:49:57.660225+0000","last_change":"2024-03-16T11:48:38.571942+0000","last_active":"2024-03-16T11:49:57.660225+0000","last_peered":"2024-03-16T11:49:57.660225+0000","last_clean":"2024-03-16T11:49:57.660225+0000","last_became_active":"2024-03-16T11:48:38.571606+0000","last_became_peered":"2024-03-16T11:48:38.571606+0000","last_unstale":"2024-03-16T11:49:57.660225+0000","last_undegraded":"2024-03-16T11:49:57.660225+0000","last_fullsized":"2024-03-16T11:49:57.660225+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:29:13.797054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"87'9","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T11:49:57.659937+0000","last_change":"2024-03-16T11:48:38.573469+0000","last_active":"2024-03-16T11:49:57.659937+0000","last_peered":"2024-03-16T11:49:57.659937+0000","last_clean":"2024-03-16T11:49:57.659937+0000","last_became_active":"2024-03-16T11:48:38.573155+0000","last_became_peered":"2024-03-16T11:48:38.573155+0000","last_unstale":"2024-03-16T11:49:57.659937+0000","last_undegraded":"2024-03-16T11:49:57.659937+0000","last_fullsized":"2024-03-16T11:49:57.659937+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:28:50.180821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"87'12","reported_seq":29,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T11:49:57.663085+0000","last_change":"2024-03-16T11:48:38.580913+0000","last_active":"2024-03-16T11:49:57.663085+0000","last_peered":"2024-03-16T11:49:57.663085+0000","last_clean":"2024-03-16T11:49:57.663085+0000","last_became_active":"2024-03-16T11:48:38.580573+0000","last_became_peered":"2024-03-16T11:48:38.580573+0000","last_unstale":"2024-03-16T11:49:57.663085+0000","last_undegraded":"2024-03-16T11:49:57.663085+0000","last_fullsized":"2024-03-16T11:49:57.663085+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:44:58.080502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"87'8","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T11:49:57.658872+0000","last_change":"2024-03-16T11:48:38.577168+0000","last_active":"2024-03-16T11:49:57.658872+0000","last_peered":"2024-03-16T11:49:57.658872+0000","last_clean":"2024-03-16T11:49:57.658872+0000","last_became_active":"2024-03-16T11:48:38.576845+0000","last_became_peered":"2024-03-16T11:48:38.576845+0000","last_unstale":"2024-03-16T11:49:57.658872+0000","last_undegraded":"2024-03-16T11:49:57.658872+0000","last_fullsized":"2024-03-16T11:49:57.658872+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:13:02.145610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"87'7","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T11:49:57.668392+0000","last_change":"2024-03-16T11:48:38.577477+0000","last_active":"2024-03-16T11:49:57.668392+0000","last_peered":"2024-03-16T11:49:57.668392+0000","last_clean":"2024-03-16T11:49:57.668392+0000","last_became_active":"2024-03-16T11:48:38.577126+0000","last_became_peered":"2024-03-16T11:48:38.577126+0000","last_unstale":"2024-03-16T11:49:57.668392+0000","last_undegraded":"2024-03-16T11:49:57.668392+0000","last_fullsized":"2024-03-16T11:49:57.668392+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:45:27.246827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"87'4","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T11:49:57.661494+0000","last_change":"2024-03-16T11:48:38.578508+0000","last_active":"2024-03-16T11:49:57.661494+0000","last_peered":"2024-03-16T11:49:57.661494+0000","last_clean":"2024-03-16T11:49:57.661494+0000","last_became_active":"2024-03-16T11:48:38.578125+0000","last_became_peered":"2024-03-16T11:48:38.578125+0000","last_unstale":"2024-03-16T11:49:57.661494+0000","last_undegraded":"2024-03-16T11:49:57.661494+0000","last_fullsized":"2024-03-16T11:49:57.661494+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:13:36.996405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"87'6","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T11:49:57.659578+0000","last_change":"2024-03-16T11:48:38.570450+0000","last_active":"2024-03-16T11:49:57.659578+0000","last_peered":"2024-03-16T11:49:57.659578+0000","last_clean":"2024-03-16T11:49:57.659578+0000","last_became_active":"2024-03-16T11:48:38.569952+0000","last_became_peered":"2024-03-16T11:48:38.569952+0000","last_unstale":"2024-03-16T11:49:57.659578+0000","last_undegraded":"2024-03-16T11:49:57.659578+0000","last_fullsized":"2024-03-16T11:49:57.659578+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:34:39.265969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"87'15","reported_seq":32,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T11:49:57.652053+0000","last_change":"2024-03-16T11:48:38.569568+0000","last_active":"2024-03-16T11:49:57.652053+0000","last_peered":"2024-03-16T11:49:57.652053+0000","last_clean":"2024-03-16T11:49:57.652053+0000","last_became_active":"2024-03-16T11:48:38.569260+0000","last_became_peered":"2024-03-16T11:48:38.569260+0000","last_unstale":"2024-03-16T11:49:57.652053+0000","last_undegraded":"2024-03-16T11:49:57.652053+0000","last_fullsized":"2024-03-16T11:49:57.652053+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:52:52.315980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"87'2","reported_seq":19,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T11:49:57.665231+0000","last_change":"2024-03-16T11:48:38.584562+0000","last_active":"2024-03-16T11:49:57.665231+0000","last_peered":"2024-03-16T11:49:57.665231+0000","last_clean":"2024-03-16T11:49:57.665231+0000","last_became_active":"2024-03-16T11:48:38.584096+0000","last_became_peered":"2024-03-16T11:48:38.584096+0000","last_unstale":"2024-03-16T11:49:57.665231+0000","last_undegraded":"2024-03-16T11:49:57.665231+0000","last_fullsized":"2024-03-16T11:49:57.665231+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T13:43:45.933252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"87'6","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T11:49:57.665200+0000","last_change":"2024-03-16T11:48:38.570174+0000","last_active":"2024-03-16T11:49:57.665200+0000","last_peered":"2024-03-16T11:49:57.665200+0000","last_clean":"2024-03-16T11:49:57.665200+0000","last_became_active":"2024-03-16T11:48:38.569865+0000","last_became_peered":"2024-03-16T11:48:38.569865+0000","last_unstale":"2024-03-16T11:49:57.665200+0000","last_undegraded":"2024-03-16T11:49:57.665200+0000","last_fullsized":"2024-03-16T11:49:57.665200+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T11:53:01.012481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"87'8","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T11:49:57.665777+0000","last_change":"2024-03-16T11:48:38.574868+0000","last_active":"2024-03-16T11:49:57.665777+0000","last_peered":"2024-03-16T11:49:57.665777+0000","last_clean":"2024-03-16T11:49:57.665777+0000","last_became_active":"2024-03-16T11:48:38.574434+0000","last_became_peered":"2024-03-16T11:48:38.574434+0000","last_unstale":"2024-03-16T11:49:57.665777+0000","last_undegraded":"2024-03-16T11:49:57.665777+0000","last_fullsized":"2024-03-16T11:49:57.665777+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:09:57.102253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"87'10","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T11:49:57.674127+0000","last_change":"2024-03-16T11:48:38.577189+0000","last_active":"2024-03-16T11:49:57.674127+0000","last_peered":"2024-03-16T11:49:57.674127+0000","last_clean":"2024-03-16T11:49:57.674127+0000","last_became_active":"2024-03-16T11:48:38.576884+0000","last_became_peered":"2024-03-16T11:48:38.576884+0000","last_unstale":"2024-03-16T11:49:57.674127+0000","last_undegraded":"2024-03-16T11:49:57.674127+0000","last_fullsized":"2024-03-16T11:49:57.674127+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:11:51.558517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"87'11","reported_seq":28,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T11:49:57.657680+0000","last_change":"2024-03-16T11:48:38.576238+0000","last_active":"2024-03-16T11:49:57.657680+0000","last_peered":"2024-03-16T11:49:57.657680+0000","last_clean":"2024-03-16T11:49:57.657680+0000","last_became_active":"2024-03-16T11:48:38.575951+0000","last_became_peered":"2024-03-16T11:48:38.575951+0000","last_unstale":"2024-03-16T11:49:57.657680+0000","last_undegraded":"2024-03-16T11:49:57.657680+0000","last_fullsized":"2024-03-16T11:49:57.657680+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:47:32.571580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"87'8","reported_seq":31,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-16T11:49:57.662748+0000","last_change":"2024-03-16T11:49:28.732348+0000","last_active":"2024-03-16T11:49:57.662748+0000","last_peered":"2024-03-16T11:49:57.662748+0000","last_clean":"2024-03-16T11:49:57.662748+0000","last_became_active":"2024-03-16T11:49:28.732043+0000","last_became_peered":"2024-03-16T11:49:28.732043+0000","last_unstale":"2024-03-16T11:49:57.662748+0000","last_undegraded":"2024-03-16T11:49:57.662748+0000","last_fullsized":"2024-03-16T11:49:57.662748+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T11:48:38.003829+0000","last_clean_scrub_stamp":"2024-03-16T11:48:38.003829+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:21:32.831045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,8],"acting":[11,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":284036,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":64946,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":514,"num_read_kb":329,"num_write":167,"num_write_kb":117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":61036,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":372736,"data_stored":194838,"data_compressed":9582,"data_compressed_allocated":12288,"data_compressed_original":57344,"omap_allocated":0,"internal_metadata":0},"log_size":145,"ondisk_log_size":145,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383712,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32096,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713056,"statfs":{"total":95995035648,"available":95962169344,"internally_reserved":0,"allocated":942080,"data_stored":456205,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645348,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28084,"kb_used_data":1068,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717068,"statfs":{"total":95995035648,"available":95966277632,"internally_reserved":0,"allocated":1093632,"data_stored":1108490,"data_compressed":36118,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808874,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27952,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717200,"statfs":{"total":95995035648,"available":95966412800,"internally_reserved":0,"allocated":958464,"data_stored":456441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070511,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32112,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713040,"statfs":{"total":95995035648,"available":95962152960,"internally_reserved":0,"allocated":958464,"data_stored":456361,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332148,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28000,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717152,"statfs":{"total":95995035648,"available":95966363648,"internally_reserved":0,"allocated":1007616,"data_stored":1048964,"data_compressed":36118,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495674,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36260,"kb_used_data":924,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708892,"statfs":{"total":95995035648,"available":95957905408,"internally_reserved":0,"allocated":946176,"data_stored":455830,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574931,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32168,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712984,"statfs":{"total":95995035648,"available":95962095616,"internally_reserved":0,"allocated":1015808,"data_stored":517049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738456,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27960,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717192,"statfs":{"total":95995035648,"available":95966404608,"internally_reserved":0,"allocated":966656,"data_stored":457101,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411406,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32108,"kb_used_data":932,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713044,"statfs":{"total":95995035648,"available":95962157056,"internally_reserved":0,"allocated":954368,"data_stored":456351,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182472,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36276,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708876,"statfs":{"total":95995035648,"available":95957889024,"internally_reserved":0,"allocated":962560,"data_stored":456141,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920836,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27968,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717184,"statfs":{"total":95995035648,"available":95966396416,"internally_reserved":0,"allocated":974848,"data_stored":517071,"data_compressed":9582,"data_compressed_allocated":12288,"data_compressed_original":57344,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757310,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32172,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712980,"statfs":{"total":95995035648,"available":95962091520,"internally_reserved":0,"allocated":1019904,"data_stored":1048116,"data_compressed":36118,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34482,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34482,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34482,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36118,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36118,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36118,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36118,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36118,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":61426,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":61448,"data_compressed":9582,"data_compressed_allocated":12288,"data_compressed_original":57344,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":62382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-16T11:50:06.920 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-03-16T11:50:06.921 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-03-16T11:50:06.921 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-03-16T11:50:06.921 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid d1887f28-e389-11ee-95c9-87774f69a715 -- ceph health --format=json 2024-03-16T11:50:07.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:50:06 smithi154 ceph-mon[36320]: from='client.14988 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-16T11:50:07.087 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:50:06 smithi029 ceph-mon[31975]: from='client.14988 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-16T11:50:07.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:50:06 smithi152 ceph-mon[36291]: from='client.14988 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-16T11:50:07.174 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/d1887f28-e389-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T11:50:07.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:50:07 smithi029 ceph-mon[31975]: pgmap v337: 97 pgs: 97 active+clean; 640 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 597 B/s wr, 6 op/s 2024-03-16T11:50:08.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:50:07 smithi154 ceph-mon[36320]: pgmap v337: 97 pgs: 97 active+clean; 640 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 597 B/s wr, 6 op/s 2024-03-16T11:50:08.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:50:07 smithi152 ceph-mon[36291]: pgmap v337: 97 pgs: 97 active+clean; 640 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 597 B/s wr, 6 op/s 2024-03-16T11:50:08.983 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:08.976+0000 7f7a9fc9a700 1 -- 172.21.15.29:0/3817955629 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7a98101820 msgr2=0x7f7a98106f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:08.983 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:08.976+0000 7f7a9fc9a700 1 --2- 172.21.15.29:0/3817955629 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7a98101820 0x7f7a98106f60 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f7a940099c0 tx=0x7f7a94009cd0 comp rx=0 tx=0).stop 2024-03-16T11:50:08.983 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:08.976+0000 7f7a9fc9a700 1 -- 172.21.15.29:0/3817955629 shutdown_connections 2024-03-16T11:50:08.983 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:08.976+0000 7f7a9fc9a700 1 --2- 172.21.15.29:0/3817955629 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7a98107690 0x7f7a9810f210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:08.984 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:08.976+0000 7f7a9fc9a700 1 --2- 172.21.15.29:0/3817955629 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7a98100e70 0x7f7a98101250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:08.984 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:08.976+0000 7f7a9fc9a700 1 --2- 172.21.15.29:0/3817955629 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7a98101820 0x7f7a98106f60 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:08.984 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:08.976+0000 7f7a9fc9a700 1 -- 172.21.15.29:0/3817955629 >> 172.21.15.29:0/3817955629 conn(0x7f7a98076510 msgr2=0x7f7a98076920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:50:08.984 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:08.977+0000 7f7a9fc9a700 1 -- 172.21.15.29:0/3817955629 shutdown_connections 2024-03-16T11:50:08.984 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:08.977+0000 7f7a9fc9a700 1 -- 172.21.15.29:0/3817955629 wait complete. 2024-03-16T11:50:08.984 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:08.977+0000 7f7a9fc9a700 1 Processor -- start 2024-03-16T11:50:08.984 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:08.977+0000 7f7a9fc9a700 1 -- start start 2024-03-16T11:50:08.984 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:08.977+0000 7f7a9fc9a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7a98100e70 0x7f7a981a0910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:08.984 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:08.977+0000 7f7a9fc9a700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7a98101820 0x7f7a981a0e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:08.984 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:08.977+0000 7f7a9fc9a700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7a98107690 0x7f7a9819aad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:08.984 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:08.977+0000 7f7a9fc9a700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f7a98112600 con 0x7f7a98100e70 2024-03-16T11:50:08.984 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:08.977+0000 7f7a9fc9a700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f7a98112480 con 0x7f7a98101820 2024-03-16T11:50:08.984 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:08.977+0000 7f7a9fc9a700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f7a98112780 con 0x7f7a98107690 2024-03-16T11:50:08.985 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:08.977+0000 7f7a9da36700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7a98100e70 0x7f7a981a0910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:08.985 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:08.977+0000 7f7a9da36700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7a98100e70 0x7f7a981a0910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49150/0 (socket says 172.21.15.29:49150) 2024-03-16T11:50:08.985 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:08.977+0000 7f7a9da36700 1 -- 172.21.15.29:0/331709896 learned_addr learned my addr 172.21.15.29:0/331709896 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:50:08.985 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:08.978+0000 7f7a9e237700 1 --2- 172.21.15.29:0/331709896 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7a98107690 0x7f7a9819aad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:08.985 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:08.978+0000 7f7a9d235700 1 --2- 172.21.15.29:0/331709896 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7a98101820 0x7f7a981a0e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:08.985 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:08.978+0000 7f7a9da36700 1 -- 172.21.15.29:0/331709896 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7a98107690 msgr2=0x7f7a9819aad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:08.985 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:08.978+0000 7f7a9da36700 1 --2- 172.21.15.29:0/331709896 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7a98107690 0x7f7a9819aad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:08.985 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:08.978+0000 7f7a9da36700 1 -- 172.21.15.29:0/331709896 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7a98101820 msgr2=0x7f7a981a0e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:08.985 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:08.978+0000 7f7a9da36700 1 --2- 172.21.15.29:0/331709896 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7a98101820 0x7f7a981a0e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:08.985 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:08.978+0000 7f7a9da36700 1 -- 172.21.15.29:0/331709896 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a9819b230 con 0x7f7a98100e70 2024-03-16T11:50:08.985 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:08.978+0000 7f7a9da36700 1 --2- 172.21.15.29:0/331709896 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7a98100e70 0x7f7a981a0910 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f7a8800d980 tx=0x7f7a8800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:50:08.985 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:08.978+0000 7f7a8effd700 1 -- 172.21.15.29:0/331709896 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a88016070 con 0x7f7a98100e70 2024-03-16T11:50:08.985 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:08.978+0000 7f7a9fc9a700 1 -- 172.21.15.29:0/331709896 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7a9819b520 con 0x7f7a98100e70 2024-03-16T11:50:08.985 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:08.978+0000 7f7a8effd700 1 -- 172.21.15.29:0/331709896 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7a88004d10 con 0x7f7a98100e70 2024-03-16T11:50:08.986 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:08.978+0000 7f7a8effd700 1 -- 172.21.15.29:0/331709896 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a88005670 con 0x7f7a98100e70 2024-03-16T11:50:08.986 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:08.978+0000 7f7a9fc9a700 1 -- 172.21.15.29:0/331709896 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7a981a77e0 con 0x7f7a98100e70 2024-03-16T11:50:08.986 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:08.979+0000 7f7a9fc9a700 1 -- 172.21.15.29:0/331709896 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7a98068f90 con 0x7f7a98100e70 2024-03-16T11:50:08.986 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:08.980+0000 7f7a8effd700 1 -- 172.21.15.29:0/331709896 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f7a88004750 con 0x7f7a98100e70 2024-03-16T11:50:08.986 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:08.981+0000 7f7a8effd700 1 --2- 172.21.15.29:0/331709896 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f7a840a2890 0x7f7a840a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:08.986 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:08.981+0000 7f7a9d235700 1 --2- 172.21.15.29:0/331709896 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f7a840a2890 0x7f7a840a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:08.986 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:08.981+0000 7f7a8effd700 1 -- 172.21.15.29:0/331709896 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8211+0+0 (secure 0 0 0) 0x7f7a880ced50 con 0x7f7a98100e70 2024-03-16T11:50:08.986 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:08.981+0000 7f7a9d235700 1 --2- 172.21.15.29:0/331709896 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f7a840a2890 0x7f7a840a4d50 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f7a94009990 tx=0x7f7a94005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:50:08.986 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:08.985+0000 7f7a8effd700 1 -- 172.21.15.29:0/331709896 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f7a8801f020 con 0x7f7a98100e70 2024-03-16T11:50:09.239 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:09.239+0000 7f7a9fc9a700 1 -- 172.21.15.29:0/331709896 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f7a9802db70 con 0x7f7a98100e70 2024-03-16T11:50:09.239 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:09.240+0000 7f7a8effd700 1 -- 172.21.15.29:0/331709896 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7f7a8809bde0 con 0x7f7a98100e70 2024-03-16T11:50:09.239 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-16T11:50:09.239 INFO:teuthology.orchestra.run.smithi029.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-03-16T11:50:09.241 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:09.242+0000 7f7a9fc9a700 1 -- 172.21.15.29:0/331709896 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f7a840a2890 msgr2=0x7f7a840a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:09.241 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:09.242+0000 7f7a9fc9a700 1 --2- 172.21.15.29:0/331709896 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f7a840a2890 0x7f7a840a4d50 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f7a94009990 tx=0x7f7a94005fb0 comp rx=0 tx=0).stop 2024-03-16T11:50:09.241 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:09.242+0000 7f7a9fc9a700 1 -- 172.21.15.29:0/331709896 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7a98100e70 msgr2=0x7f7a981a0910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:09.241 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:09.242+0000 7f7a9fc9a700 1 --2- 172.21.15.29:0/331709896 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7a98100e70 0x7f7a981a0910 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f7a8800d980 tx=0x7f7a8800dc90 comp rx=0 tx=0).stop 2024-03-16T11:50:09.241 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:09.242+0000 7f7a9fc9a700 1 -- 172.21.15.29:0/331709896 shutdown_connections 2024-03-16T11:50:09.241 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:09.242+0000 7f7a9fc9a700 1 --2- 172.21.15.29:0/331709896 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f7a840a2890 0x7f7a840a4d50 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:09.241 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:09.242+0000 7f7a9fc9a700 1 --2- 172.21.15.29:0/331709896 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7a98107690 0x7f7a9819aad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:09.242 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:09.242+0000 7f7a9fc9a700 1 --2- 172.21.15.29:0/331709896 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7a98100e70 0x7f7a981a0910 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:09.242 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:09.242+0000 7f7a9fc9a700 1 --2- 172.21.15.29:0/331709896 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7a98101820 0x7f7a981a0e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:09.242 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:09.242+0000 7f7a9fc9a700 1 -- 172.21.15.29:0/331709896 >> 172.21.15.29:0/331709896 conn(0x7f7a98076510 msgr2=0x7f7a98100420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:50:09.242 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:09.242+0000 7f7a9fc9a700 1 -- 172.21.15.29:0/331709896 shutdown_connections 2024-03-16T11:50:09.242 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:09.242+0000 7f7a9fc9a700 1 -- 172.21.15.29:0/331709896 wait complete. 2024-03-16T11:50:09.545 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:50:09 smithi154 ceph-mon[36320]: from='client.? 172.21.15.29:0/331709896' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-16T11:50:09.546 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:50:09 smithi029 ceph-mon[31975]: from='client.? 172.21.15.29:0/331709896' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-16T11:50:09.627 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:50:09 smithi152 ceph-mon[36291]: from='client.? 172.21.15.29:0/331709896' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-16T11:50:09.894 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-03-16T11:50:09.895 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-16T11:50:10.003 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:10.002+0000 7f982ff62640 1 -- 172.21.15.29:0/1257577513 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f98281018f0 msgr2=0x7f9828101d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:10.003 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:10.002+0000 7f982ff62640 1 --2- 172.21.15.29:0/1257577513 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f98281018f0 0x7f9828101d70 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f981c009900 tx=0x7f981c02bc30 comp rx=0 tx=0).stop 2024-03-16T11:50:10.003 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:10.002+0000 7f982ff62640 1 -- 172.21.15.29:0/1257577513 shutdown_connections 2024-03-16T11:50:10.003 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:10.002+0000 7f982ff62640 1 --2- 172.21.15.29:0/1257577513 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f98281022b0 0x7f9828108b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:10.003 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:10.002+0000 7f982ff62640 1 --2- 172.21.15.29:0/1257577513 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f98281018f0 0x7f9828101d70 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:10.003 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:10.002+0000 7f982ff62640 1 --2- 172.21.15.29:0/1257577513 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9828100700 0x7f9828100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:10.003 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:10.002+0000 7f982ff62640 1 -- 172.21.15.29:0/1257577513 >> 172.21.15.29:0/1257577513 conn(0x7f98280fbc90 msgr2=0x7f98280fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:50:10.003 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:10.002+0000 7f982ff62640 1 -- 172.21.15.29:0/1257577513 shutdown_connections 2024-03-16T11:50:10.004 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:10.002+0000 7f982ff62640 1 -- 172.21.15.29:0/1257577513 wait complete. 2024-03-16T11:50:10.004 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:10.003+0000 7f982ff62640 1 Processor -- start 2024-03-16T11:50:10.004 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:10.003+0000 7f982ff62640 1 -- start start 2024-03-16T11:50:10.004 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:10.003+0000 7f982ff62640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9828100700 0x7f9828197b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:10.004 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:10.003+0000 7f982ff62640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f98281018f0 0x7f9828198040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:10.004 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:10.003+0000 7f982ff62640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f98281022b0 0x7f982819f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:10.004 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:10.003+0000 7f982ff62640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f982810c360 con 0x7f9828100700 2024-03-16T11:50:10.004 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:10.003+0000 7f982ff62640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f982810c1e0 con 0x7f98281022b0 2024-03-16T11:50:10.004 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:10.003+0000 7f982ff62640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f982810c4e0 con 0x7f98281018f0 2024-03-16T11:50:10.004 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:10.003+0000 7f982dcfe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9828100700 0x7f9828197b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:10.004 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:10.003+0000 7f982dcfe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9828100700 0x7f9828197b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47208/0 (socket says 172.21.15.29:47208) 2024-03-16T11:50:10.004 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:10.003+0000 7f982dcfe640 1 -- 172.21.15.29:0/2828491680 learned_addr learned my addr 172.21.15.29:0/2828491680 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:50:10.004 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:10.003+0000 7f982d4fd640 1 --2- 172.21.15.29:0/2828491680 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f98281018f0 0x7f9828198040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:10.004 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:10.003+0000 7f982e4ff640 1 --2- 172.21.15.29:0/2828491680 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f98281022b0 0x7f982819f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:10.004 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:10.003+0000 7f982dcfe640 1 -- 172.21.15.29:0/2828491680 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f98281018f0 msgr2=0x7f9828198040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:10.005 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:10.003+0000 7f982dcfe640 1 --2- 172.21.15.29:0/2828491680 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f98281018f0 0x7f9828198040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:10.005 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:10.003+0000 7f982dcfe640 1 -- 172.21.15.29:0/2828491680 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f98281022b0 msgr2=0x7f982819f0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:10.005 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:10.003+0000 7f982dcfe640 1 --2- 172.21.15.29:0/2828491680 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f98281022b0 0x7f982819f0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:10.005 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:10.003+0000 7f982dcfe640 1 -- 172.21.15.29:0/2828491680 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f982819f720 con 0x7f9828100700 2024-03-16T11:50:10.005 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:10.003+0000 7f982dcfe640 1 --2- 172.21.15.29:0/2828491680 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9828100700 0x7f9828197b00 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f982000e9c0 tx=0x7f982000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:50:10.005 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:10.004+0000 7f9826ffd640 1 -- 172.21.15.29:0/2828491680 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f982000cd80 con 0x7f9828100700 2024-03-16T11:50:10.005 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:10.004+0000 7f9826ffd640 1 -- 172.21.15.29:0/2828491680 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f98200035f0 con 0x7f9828100700 2024-03-16T11:50:10.005 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:10.004+0000 7f982ff62640 1 -- 172.21.15.29:0/2828491680 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f982819fa10 con 0x7f9828100700 2024-03-16T11:50:10.005 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:10.004+0000 7f9826ffd640 1 -- 172.21.15.29:0/2828491680 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9820010690 con 0x7f9828100700 2024-03-16T11:50:10.005 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:10.004+0000 7f982ff62640 1 -- 172.21.15.29:0/2828491680 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f982819ff30 con 0x7f9828100700 2024-03-16T11:50:10.006 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:10.004+0000 7f982ff62640 1 -- 172.21.15.29:0/2828491680 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9828068a20 con 0x7f9828100700 2024-03-16T11:50:10.006 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:10.006+0000 7f9826ffd640 1 -- 172.21.15.29:0/2828491680 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f98200037a0 con 0x7f9828100700 2024-03-16T11:50:10.006 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:10.006+0000 7f9826ffd640 1 --2- 172.21.15.29:0/2828491680 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f98100a2800 0x7f98100a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:10.006 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:10.006+0000 7f9826ffd640 1 -- 172.21.15.29:0/2828491680 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8211+0+0 (secure 0 0 0) 0x7f98200ceee0 con 0x7f9828100700 2024-03-16T11:50:10.006 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:10.006+0000 7f982d4fd640 1 --2- 172.21.15.29:0/2828491680 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f98100a2800 0x7f98100a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:10.006 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:10.007+0000 7f982d4fd640 1 --2- 172.21.15.29:0/2828491680 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f98100a2800 0x7f98100a4cc0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f981c02c7b0 tx=0x7f981c005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:50:10.009 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:10.010+0000 7f9826ffd640 1 -- 172.21.15.29:0/2828491680 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f982009c020 con 0x7f9828100700 2024-03-16T11:50:10.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:10.192+0000 7f982ff62640 1 -- 172.21.15.29:0/2828491680 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f98281a0160 con 0x7f9828100700 2024-03-16T11:50:10.193 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:10.194+0000 7f9826ffd640 1 -- 172.21.15.29:0/2828491680 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) ==== 74+0+22668 (secure 0 0 0) 0x7f982009b920 con 0x7f9828100700 2024-03-16T11:50:10.193 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-16T11:50:10.193 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":87,"fsid":"d1887f28-e389-11ee-95c9-87774f69a715","created":"2024-03-16T11:40:16.475063+0000","modified":"2024-03-16T11:49:28.628389+0000","last_up_change":"2024-03-16T11:47:44.349348+0000","last_in_change":"2024-03-16T11:47:23.672432+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-16T11:43:51.892652+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-16T11:48:36.709732+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-16T11:48:37.586140+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"60f5608f-a408-45ba-bdf6-5b53758f88fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":2748947913},{"type":"v1","addr":"172.21.15.29:6803","nonce":2748947913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":2748947913},{"type":"v1","addr":"172.21.15.29:6805","nonce":2748947913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":2748947913},{"type":"v1","addr":"172.21.15.29:6809","nonce":2748947913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":2748947913},{"type":"v1","addr":"172.21.15.29:6807","nonce":2748947913}]},"public_addr":"172.21.15.29:6803/2748947913","cluster_addr":"172.21.15.29:6805/2748947913","heartbeat_back_addr":"172.21.15.29:6809/2748947913","heartbeat_front_addr":"172.21.15.29:6807/2748947913","state":["exists","up"]},{"osd":1,"uuid":"e70b53d6-ee78-4e56-9a2c-263ae0501b43","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6800","nonce":2339408422},{"type":"v1","addr":"172.21.15.152:6801","nonce":2339408422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6802","nonce":2339408422},{"type":"v1","addr":"172.21.15.152:6803","nonce":2339408422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6806","nonce":2339408422},{"type":"v1","addr":"172.21.15.152:6807","nonce":2339408422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6804","nonce":2339408422},{"type":"v1","addr":"172.21.15.152:6805","nonce":2339408422}]},"public_addr":"172.21.15.152:6801/2339408422","cluster_addr":"172.21.15.152:6803/2339408422","heartbeat_back_addr":"172.21.15.152:6807/2339408422","heartbeat_front_addr":"172.21.15.152:6805/2339408422","state":["exists","up"]},{"osd":2,"uuid":"ff39389a-dfac-44b5-80ff-36abca316e78","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6800","nonce":4288325280},{"type":"v1","addr":"172.21.15.154:6801","nonce":4288325280}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6802","nonce":4288325280},{"type":"v1","addr":"172.21.15.154:6803","nonce":4288325280}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6806","nonce":4288325280},{"type":"v1","addr":"172.21.15.154:6807","nonce":4288325280}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":4288325280},{"type":"v1","addr":"172.21.15.154:6805","nonce":4288325280}]},"public_addr":"172.21.15.154:6801/4288325280","cluster_addr":"172.21.15.154:6803/4288325280","heartbeat_back_addr":"172.21.15.154:6807/4288325280","heartbeat_front_addr":"172.21.15.154:6805/4288325280","state":["exists","up"]},{"osd":3,"uuid":"069e1795-55bc-4324-b94c-eb24ef2334c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":1538750397},{"type":"v1","addr":"172.21.15.29:6811","nonce":1538750397}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":1538750397},{"type":"v1","addr":"172.21.15.29:6813","nonce":1538750397}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":1538750397},{"type":"v1","addr":"172.21.15.29:6817","nonce":1538750397}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":1538750397},{"type":"v1","addr":"172.21.15.29:6815","nonce":1538750397}]},"public_addr":"172.21.15.29:6811/1538750397","cluster_addr":"172.21.15.29:6813/1538750397","heartbeat_back_addr":"172.21.15.29:6817/1538750397","heartbeat_front_addr":"172.21.15.29:6815/1538750397","state":["exists","up"]},{"osd":4,"uuid":"5a9d43a1-e8f5-4ab5-a441-afbc6ed4066e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6808","nonce":2795081750},{"type":"v1","addr":"172.21.15.152:6809","nonce":2795081750}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6810","nonce":2795081750},{"type":"v1","addr":"172.21.15.152:6811","nonce":2795081750}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6814","nonce":2795081750},{"type":"v1","addr":"172.21.15.152:6815","nonce":2795081750}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6812","nonce":2795081750},{"type":"v1","addr":"172.21.15.152:6813","nonce":2795081750}]},"public_addr":"172.21.15.152:6809/2795081750","cluster_addr":"172.21.15.152:6811/2795081750","heartbeat_back_addr":"172.21.15.152:6815/2795081750","heartbeat_front_addr":"172.21.15.152:6813/2795081750","state":["exists","up"]},{"osd":5,"uuid":"234de428-4ac7-44be-9a54-7b58331fdf8c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":1472686560},{"type":"v1","addr":"172.21.15.154:6809","nonce":1472686560}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6810","nonce":1472686560},{"type":"v1","addr":"172.21.15.154:6811","nonce":1472686560}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6814","nonce":1472686560},{"type":"v1","addr":"172.21.15.154:6815","nonce":1472686560}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6812","nonce":1472686560},{"type":"v1","addr":"172.21.15.154:6813","nonce":1472686560}]},"public_addr":"172.21.15.154:6809/1472686560","cluster_addr":"172.21.15.154:6811/1472686560","heartbeat_back_addr":"172.21.15.154:6815/1472686560","heartbeat_front_addr":"172.21.15.154:6813/1472686560","state":["exists","up"]},{"osd":6,"uuid":"ae45b8d9-7883-40c9-a9cd-4c92b601d4f2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":816506244},{"type":"v1","addr":"172.21.15.29:6819","nonce":816506244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":816506244},{"type":"v1","addr":"172.21.15.29:6821","nonce":816506244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":816506244},{"type":"v1","addr":"172.21.15.29:6825","nonce":816506244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":816506244},{"type":"v1","addr":"172.21.15.29:6823","nonce":816506244}]},"public_addr":"172.21.15.29:6819/816506244","cluster_addr":"172.21.15.29:6821/816506244","heartbeat_back_addr":"172.21.15.29:6825/816506244","heartbeat_front_addr":"172.21.15.29:6823/816506244","state":["exists","up"]},{"osd":7,"uuid":"6f687b28-4fce-4d4a-8b4e-294173d9d6c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6816","nonce":2924651787},{"type":"v1","addr":"172.21.15.152:6817","nonce":2924651787}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6818","nonce":2924651787},{"type":"v1","addr":"172.21.15.152:6819","nonce":2924651787}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6822","nonce":2924651787},{"type":"v1","addr":"172.21.15.152:6823","nonce":2924651787}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6820","nonce":2924651787},{"type":"v1","addr":"172.21.15.152:6821","nonce":2924651787}]},"public_addr":"172.21.15.152:6817/2924651787","cluster_addr":"172.21.15.152:6819/2924651787","heartbeat_back_addr":"172.21.15.152:6823/2924651787","heartbeat_front_addr":"172.21.15.152:6821/2924651787","state":["exists","up"]},{"osd":8,"uuid":"a25fb741-caa7-429a-b540-e394d034b446","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6816","nonce":129974156},{"type":"v1","addr":"172.21.15.154:6817","nonce":129974156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6818","nonce":129974156},{"type":"v1","addr":"172.21.15.154:6819","nonce":129974156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6822","nonce":129974156},{"type":"v1","addr":"172.21.15.154:6823","nonce":129974156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6820","nonce":129974156},{"type":"v1","addr":"172.21.15.154:6821","nonce":129974156}]},"public_addr":"172.21.15.154:6817/129974156","cluster_addr":"172.21.15.154:6819/129974156","heartbeat_back_addr":"172.21.15.154:6823/129974156","heartbeat_front_addr":"172.21.15.154:6821/129974156","state":["exists","up"]},{"osd":9,"uuid":"1071cb0e-8c5f-4cd8-9805-0045e220d946","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6826","nonce":2951414243},{"type":"v1","addr":"172.21.15.29:6827","nonce":2951414243}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":2951414243},{"type":"v1","addr":"172.21.15.29:6829","nonce":2951414243}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":2951414243},{"type":"v1","addr":"172.21.15.29:6833","nonce":2951414243}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":2951414243},{"type":"v1","addr":"172.21.15.29:6831","nonce":2951414243}]},"public_addr":"172.21.15.29:6827/2951414243","cluster_addr":"172.21.15.29:6829/2951414243","heartbeat_back_addr":"172.21.15.29:6833/2951414243","heartbeat_front_addr":"172.21.15.29:6831/2951414243","state":["exists","up"]},{"osd":10,"uuid":"ee7c7d74-3f39-4cd4-be80-843fb1dd9ecc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6824","nonce":3412525480},{"type":"v1","addr":"172.21.15.152:6825","nonce":3412525480}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6826","nonce":3412525480},{"type":"v1","addr":"172.21.15.152:6827","nonce":3412525480}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6830","nonce":3412525480},{"type":"v1","addr":"172.21.15.152:6831","nonce":3412525480}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6828","nonce":3412525480},{"type":"v1","addr":"172.21.15.152:6829","nonce":3412525480}]},"public_addr":"172.21.15.152:6825/3412525480","cluster_addr":"172.21.15.152:6827/3412525480","heartbeat_back_addr":"172.21.15.152:6831/3412525480","heartbeat_front_addr":"172.21.15.152:6829/3412525480","state":["exists","up"]},{"osd":11,"uuid":"4598c803-984a-4de6-ac22-066502c17889","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6824","nonce":1160927966},{"type":"v1","addr":"172.21.15.154:6825","nonce":1160927966}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6826","nonce":1160927966},{"type":"v1","addr":"172.21.15.154:6827","nonce":1160927966}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6830","nonce":1160927966},{"type":"v1","addr":"172.21.15.154:6831","nonce":1160927966}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6828","nonce":1160927966},{"type":"v1","addr":"172.21.15.154:6829","nonce":1160927966}]},"public_addr":"172.21.15.154:6825/1160927966","cluster_addr":"172.21.15.154:6827/1160927966","heartbeat_back_addr":"172.21.15.154:6831/1160927966","heartbeat_front_addr":"172.21.15.154:6829/1160927966","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:43:02.343393+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:43:25.209151+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:43:48.946164+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:44:14.588424+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:44:39.700025+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:45:04.061197+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:45:31.214508+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:45:56.937296+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:46:22.768915+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:46:50.101011+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:47:16.298041+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:47:42.402095+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"2.1b","mappings":[{"from":2,"to":1}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"3.33","mappings":[{"from":5,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.29:0/1853086886":"2024-03-17T11:41:27.531724+0000","172.21.15.29:0/1211659364":"2024-03-17T11:41:27.531724+0000","172.21.15.29:6800/2798831128":"2024-03-17T11:41:27.531724+0000","172.21.15.29:0/1281028285":"2024-03-17T11:41:00.505678+0000","172.21.15.29:6800/2275315654":"2024-03-17T11:41:00.505678+0000","172.21.15.29:6801/2798831128":"2024-03-17T11:41:27.531724+0000","172.21.15.29:6801/2275315654":"2024-03-17T11:41:00.505678+0000","172.21.15.29:0/1330100009":"2024-03-17T11:41:00.505678+0000","172.21.15.29:0/1784249039":"2024-03-17T11:41:00.505678+0000","172.21.15.29:0/1116486932":"2024-03-17T11:41:27.531724+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-16T11:50:10.195 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:10.196+0000 7f982ff62640 1 -- 172.21.15.29:0/2828491680 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f98100a2800 msgr2=0x7f98100a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:10.196 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:10.196+0000 7f982ff62640 1 --2- 172.21.15.29:0/2828491680 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f98100a2800 0x7f98100a4cc0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f981c02c7b0 tx=0x7f981c005fd0 comp rx=0 tx=0).stop 2024-03-16T11:50:10.196 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:10.196+0000 7f982ff62640 1 -- 172.21.15.29:0/2828491680 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9828100700 msgr2=0x7f9828197b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:10.196 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:10.196+0000 7f982ff62640 1 --2- 172.21.15.29:0/2828491680 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9828100700 0x7f9828197b00 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f982000e9c0 tx=0x7f982000ee80 comp rx=0 tx=0).stop 2024-03-16T11:50:10.196 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:10.196+0000 7f982ff62640 1 -- 172.21.15.29:0/2828491680 shutdown_connections 2024-03-16T11:50:10.196 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:10.196+0000 7f982ff62640 1 --2- 172.21.15.29:0/2828491680 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f98100a2800 0x7f98100a4cc0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:10.196 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:10.196+0000 7f982ff62640 1 --2- 172.21.15.29:0/2828491680 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f98281022b0 0x7f982819f0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:10.196 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:10.196+0000 7f982ff62640 1 --2- 172.21.15.29:0/2828491680 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f98281018f0 0x7f9828198040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:10.196 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:10.196+0000 7f982ff62640 1 --2- 172.21.15.29:0/2828491680 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9828100700 0x7f9828197b00 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:10.196 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:10.196+0000 7f982ff62640 1 -- 172.21.15.29:0/2828491680 >> 172.21.15.29:0/2828491680 conn(0x7f98280fbc90 msgr2=0x7f98280fd710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:50:10.196 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:10.196+0000 7f982ff62640 1 -- 172.21.15.29:0/2828491680 shutdown_connections 2024-03-16T11:50:10.196 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:10.196+0000 7f982ff62640 1 -- 172.21.15.29:0/2828491680 wait complete. 2024-03-16T11:50:10.207 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-16T11:43:51.892652+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-16T11:48:36.709732+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-16T11:48:37.586140+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-16T11:50:10.207 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-16T11:50:10.315 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:10.314+0000 7ff9a15da640 1 -- 172.21.15.29:0/487296944 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff99c102260 msgr2=0x7ff99c104650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:10.315 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:10.314+0000 7ff9a15da640 1 --2- 172.21.15.29:0/487296944 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff99c102260 0x7ff99c104650 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7ff98c009900 tx=0x7ff98c02bc30 comp rx=0 tx=0).stop 2024-03-16T11:50:10.315 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:10.314+0000 7ff9a15da640 1 -- 172.21.15.29:0/487296944 shutdown_connections 2024-03-16T11:50:10.315 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:10.314+0000 7ff9a15da640 1 --2- 172.21.15.29:0/487296944 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff99c1074c0 0x7ff99c109910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:10.315 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:10.314+0000 7ff9a15da640 1 --2- 172.21.15.29:0/487296944 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff99c104b90 0x7ff99c106f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:10.315 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:10.314+0000 7ff9a15da640 1 --2- 172.21.15.29:0/487296944 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff99c102260 0x7ff99c104650 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:10.316 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:10.314+0000 7ff9a15da640 1 -- 172.21.15.29:0/487296944 >> 172.21.15.29:0/487296944 conn(0x7ff99c0fbc70 msgr2=0x7ff99c0fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:50:10.316 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:10.314+0000 7ff9a15da640 1 -- 172.21.15.29:0/487296944 shutdown_connections 2024-03-16T11:50:10.316 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:10.314+0000 7ff9a15da640 1 -- 172.21.15.29:0/487296944 wait complete. 2024-03-16T11:50:10.316 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:10.315+0000 7ff9a15da640 1 Processor -- start 2024-03-16T11:50:10.316 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:10.315+0000 7ff9a15da640 1 -- start start 2024-03-16T11:50:10.316 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:10.315+0000 7ff9a15da640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff99c102260 0x7ff99c197b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:10.316 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:10.315+0000 7ff9a15da640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff99c104b90 0x7ff99c1980d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:10.316 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:10.315+0000 7ff9a15da640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff99c1074c0 0x7ff99c19f140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:10.316 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:10.315+0000 7ff9a15da640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7ff99c10d490 con 0x7ff99c104b90 2024-03-16T11:50:10.316 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:10.315+0000 7ff9a15da640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7ff99c10d310 con 0x7ff99c102260 2024-03-16T11:50:10.316 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:10.315+0000 7ff9a15da640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7ff99c10d610 con 0x7ff99c1074c0 2024-03-16T11:50:10.316 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:10.315+0000 7ff9927fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff99c104b90 0x7ff99c1980d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:10.316 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:10.315+0000 7ff9927fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff99c104b90 0x7ff99c1980d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47216/0 (socket says 172.21.15.29:47216) 2024-03-16T11:50:10.316 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:10.315+0000 7ff9927fc640 1 -- 172.21.15.29:0/1235505682 learned_addr learned my addr 172.21.15.29:0/1235505682 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:50:10.316 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:10.315+0000 7ff99affd640 1 --2- 172.21.15.29:0/1235505682 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff99c102260 0x7ff99c197b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:10.317 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:10.315+0000 7ff99b7fe640 1 --2- 172.21.15.29:0/1235505682 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff99c1074c0 0x7ff99c19f140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:10.317 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:10.316+0000 7ff9927fc640 1 -- 172.21.15.29:0/1235505682 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff99c1074c0 msgr2=0x7ff99c19f140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:10.317 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:10.316+0000 7ff9927fc640 1 --2- 172.21.15.29:0/1235505682 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff99c1074c0 0x7ff99c19f140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:10.317 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:10.316+0000 7ff9927fc640 1 -- 172.21.15.29:0/1235505682 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff99c102260 msgr2=0x7ff99c197b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:10.317 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:10.316+0000 7ff9927fc640 1 --2- 172.21.15.29:0/1235505682 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff99c102260 0x7ff99c197b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:10.317 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:10.316+0000 7ff9927fc640 1 -- 172.21.15.29:0/1235505682 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff99c19f7b0 con 0x7ff99c104b90 2024-03-16T11:50:10.317 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:10.316+0000 7ff9927fc640 1 --2- 172.21.15.29:0/1235505682 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff99c104b90 0x7ff99c1980d0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7ff98400e9f0 tx=0x7ff98400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:50:10.317 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:10.316+0000 7ff998ff9640 1 -- 172.21.15.29:0/1235505682 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff98400ccc0 con 0x7ff99c104b90 2024-03-16T11:50:10.317 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:10.316+0000 7ff9a15da640 1 -- 172.21.15.29:0/1235505682 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff99c19faa0 con 0x7ff99c104b90 2024-03-16T11:50:10.317 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:10.316+0000 7ff998ff9640 1 -- 172.21.15.29:0/1235505682 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff984005500 con 0x7ff99c104b90 2024-03-16T11:50:10.317 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:10.316+0000 7ff998ff9640 1 -- 172.21.15.29:0/1235505682 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff984010690 con 0x7ff99c104b90 2024-03-16T11:50:10.317 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:10.316+0000 7ff9a15da640 1 -- 172.21.15.29:0/1235505682 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff99c19fff0 con 0x7ff99c104b90 2024-03-16T11:50:10.318 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:10.317+0000 7ff9a15da640 1 -- 172.21.15.29:0/1235505682 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff99c0fd760 con 0x7ff99c104b90 2024-03-16T11:50:10.318 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:10.318+0000 7ff998ff9640 1 -- 172.21.15.29:0/1235505682 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7ff9840046c0 con 0x7ff99c104b90 2024-03-16T11:50:10.318 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:10.319+0000 7ff998ff9640 1 --2- 172.21.15.29:0/1235505682 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7ff9880a2800 0x7ff9880a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:10.319 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:10.319+0000 7ff998ff9640 1 -- 172.21.15.29:0/1235505682 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8211+0+0 (secure 0 0 0) 0x7ff9840cef90 con 0x7ff99c104b90 2024-03-16T11:50:10.319 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:10.319+0000 7ff99affd640 1 --2- 172.21.15.29:0/1235505682 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7ff9880a2800 0x7ff9880a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:10.319 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:10.319+0000 7ff99affd640 1 --2- 172.21.15.29:0/1235505682 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7ff9880a2800 0x7ff9880a4cc0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7ff98c02c7b0 tx=0x7ff98c005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:50:10.321 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:10.322+0000 7ff998ff9640 1 -- 172.21.15.29:0/1235505682 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7ff984053740 con 0x7ff99c104b90 2024-03-16T11:50:10.505 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:10.506+0000 7ff9a15da640 1 -- 172.21.15.29:0/1235505682 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7ff99c02d370 con 0x7ff99c104b90 2024-03-16T11:50:10.505 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:10.506+0000 7ff998ff9640 1 -- 172.21.15.29:0/1235505682 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v87) ==== 93+0+10 (secure 0 0 0) 0x7ff98409c040 con 0x7ff99c104b90 2024-03-16T11:50:10.505 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2024-03-16T11:50:10.507 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:10.508+0000 7ff9a15da640 1 -- 172.21.15.29:0/1235505682 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7ff9880a2800 msgr2=0x7ff9880a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:10.507 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:10.508+0000 7ff9a15da640 1 --2- 172.21.15.29:0/1235505682 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7ff9880a2800 0x7ff9880a4cc0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7ff98c02c7b0 tx=0x7ff98c005fd0 comp rx=0 tx=0).stop 2024-03-16T11:50:10.508 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:10.508+0000 7ff9a15da640 1 -- 172.21.15.29:0/1235505682 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff99c104b90 msgr2=0x7ff99c1980d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:10.508 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:10.508+0000 7ff9a15da640 1 --2- 172.21.15.29:0/1235505682 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff99c104b90 0x7ff99c1980d0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7ff98400e9f0 tx=0x7ff98400eeb0 comp rx=0 tx=0).stop 2024-03-16T11:50:10.508 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:10.508+0000 7ff9a15da640 1 -- 172.21.15.29:0/1235505682 shutdown_connections 2024-03-16T11:50:10.508 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:10.508+0000 7ff9a15da640 1 --2- 172.21.15.29:0/1235505682 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7ff9880a2800 0x7ff9880a4cc0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:10.508 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:10.508+0000 7ff9a15da640 1 --2- 172.21.15.29:0/1235505682 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff99c1074c0 0x7ff99c19f140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:10.508 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:10.508+0000 7ff9a15da640 1 --2- 172.21.15.29:0/1235505682 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff99c104b90 0x7ff99c1980d0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:10.508 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:10.508+0000 7ff9a15da640 1 --2- 172.21.15.29:0/1235505682 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff99c102260 0x7ff99c197b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:10.508 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:10.508+0000 7ff9a15da640 1 -- 172.21.15.29:0/1235505682 >> 172.21.15.29:0/1235505682 conn(0x7ff99c0fbc70 msgr2=0x7ff99c102d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:50:10.508 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:10.508+0000 7ff9a15da640 1 -- 172.21.15.29:0/1235505682 shutdown_connections 2024-03-16T11:50:10.508 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:10.508+0000 7ff9a15da640 1 -- 172.21.15.29:0/1235505682 wait complete. 2024-03-16T11:50:10.517 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:50:10 smithi029 ceph-mon[31975]: pgmap v338: 97 pgs: 97 active+clean; 640 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 597 B/s wr, 7 op/s 2024-03-16T11:50:10.518 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:50:10 smithi029 ceph-mon[31975]: from='client.? 172.21.15.29:0/2828491680' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T11:50:10.518 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-16T11:50:10.625 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:10.624+0000 7fbe7b6dc640 1 -- 172.21.15.29:0/3438627477 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbe741033a0 msgr2=0x7fbe74109c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:10.625 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:10.624+0000 7fbe7b6dc640 1 --2- 172.21.15.29:0/3438627477 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbe741033a0 0x7fbe74109c30 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fbe6c009ca0 tx=0x7fbe6c02c0f0 comp rx=0 tx=0).stop 2024-03-16T11:50:10.625 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:10.624+0000 7fbe7b6dc640 1 -- 172.21.15.29:0/3438627477 shutdown_connections 2024-03-16T11:50:10.625 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:10.624+0000 7fbe7b6dc640 1 --2- 172.21.15.29:0/3438627477 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbe741033a0 0x7fbe74109c30 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:10.626 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:10.624+0000 7fbe7b6dc640 1 --2- 172.21.15.29:0/3438627477 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbe741029e0 0x7fbe74102e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:10.626 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:10.624+0000 7fbe7b6dc640 1 --2- 172.21.15.29:0/3438627477 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbe741017f0 0x7fbe74101c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:10.626 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:10.624+0000 7fbe7b6dc640 1 -- 172.21.15.29:0/3438627477 >> 172.21.15.29:0/3438627477 conn(0x7fbe740fcd80 msgr2=0x7fbe740ff1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:50:10.626 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:10.624+0000 7fbe7b6dc640 1 -- 172.21.15.29:0/3438627477 shutdown_connections 2024-03-16T11:50:10.626 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:10.624+0000 7fbe7b6dc640 1 -- 172.21.15.29:0/3438627477 wait complete. 2024-03-16T11:50:10.626 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:10.625+0000 7fbe7b6dc640 1 Processor -- start 2024-03-16T11:50:10.626 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:10.625+0000 7fbe7b6dc640 1 -- start start 2024-03-16T11:50:10.626 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:10.625+0000 7fbe7b6dc640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbe741017f0 0x7fbe74197ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:10.626 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:10.625+0000 7fbe7b6dc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbe741029e0 0x7fbe74197ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:10.626 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:10.625+0000 7fbe7b6dc640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbe741033a0 0x7fbe7419f060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:10.626 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:10.625+0000 7fbe7b6dc640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fbe7410d470 con 0x7fbe741029e0 2024-03-16T11:50:10.626 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:10.625+0000 7fbe7b6dc640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fbe7410d2f0 con 0x7fbe741017f0 2024-03-16T11:50:10.626 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:10.625+0000 7fbe7b6dc640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fbe7410d5f0 con 0x7fbe741033a0 2024-03-16T11:50:10.626 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:10.625+0000 7fbe78c77640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbe741029e0 0x7fbe74197ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:10.626 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:10.625+0000 7fbe78c77640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbe741029e0 0x7fbe74197ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47234/0 (socket says 172.21.15.29:47234) 2024-03-16T11:50:10.627 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:10.625+0000 7fbe78c77640 1 -- 172.21.15.29:0/2863023860 learned_addr learned my addr 172.21.15.29:0/2863023860 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:50:10.627 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:10.625+0000 7fbe79478640 1 --2- 172.21.15.29:0/2863023860 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbe741017f0 0x7fbe74197ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:10.627 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:10.625+0000 7fbe79c79640 1 --2- 172.21.15.29:0/2863023860 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbe741033a0 0x7fbe7419f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:10.627 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:10.625+0000 7fbe78c77640 1 -- 172.21.15.29:0/2863023860 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbe741033a0 msgr2=0x7fbe7419f060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:10.627 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:10.625+0000 7fbe78c77640 1 --2- 172.21.15.29:0/2863023860 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbe741033a0 0x7fbe7419f060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:10.627 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:10.625+0000 7fbe78c77640 1 -- 172.21.15.29:0/2863023860 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbe741017f0 msgr2=0x7fbe74197ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:10.627 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:10.625+0000 7fbe78c77640 1 --2- 172.21.15.29:0/2863023860 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbe741017f0 0x7fbe74197ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:10.627 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:10.625+0000 7fbe78c77640 1 -- 172.21.15.29:0/2863023860 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe7419f6d0 con 0x7fbe741029e0 2024-03-16T11:50:10.627 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:10.626+0000 7fbe78c77640 1 --2- 172.21.15.29:0/2863023860 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbe741029e0 0x7fbe74197ff0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fbe6400e9f0 tx=0x7fbe6400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:50:10.627 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:10.626+0000 7fbe6a7fc640 1 -- 172.21.15.29:0/2863023860 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe6400cd80 con 0x7fbe741029e0 2024-03-16T11:50:10.627 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:10.626+0000 7fbe6a7fc640 1 -- 172.21.15.29:0/2863023860 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbe64005500 con 0x7fbe741029e0 2024-03-16T11:50:10.627 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:10.626+0000 7fbe7b6dc640 1 -- 172.21.15.29:0/2863023860 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbe7419f9c0 con 0x7fbe741029e0 2024-03-16T11:50:10.627 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:10.626+0000 7fbe6a7fc640 1 -- 172.21.15.29:0/2863023860 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe64010690 con 0x7fbe741029e0 2024-03-16T11:50:10.627 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:10.626+0000 7fbe7b6dc640 1 -- 172.21.15.29:0/2863023860 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbe7419fec0 con 0x7fbe741029e0 2024-03-16T11:50:10.628 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:50:10 smithi152 ceph-mon[36291]: pgmap v338: 97 pgs: 97 active+clean; 640 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 597 B/s wr, 7 op/s 2024-03-16T11:50:10.628 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:50:10 smithi152 ceph-mon[36291]: from='client.? 172.21.15.29:0/2828491680' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T11:50:10.629 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:10.627+0000 7fbe7b6dc640 1 -- 172.21.15.29:0/2863023860 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbe74108b10 con 0x7fbe741029e0 2024-03-16T11:50:10.629 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:10.628+0000 7fbe6a7fc640 1 -- 172.21.15.29:0/2863023860 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fbe640046c0 con 0x7fbe741029e0 2024-03-16T11:50:10.629 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:10.628+0000 7fbe6a7fc640 1 --2- 172.21.15.29:0/2863023860 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fbe600a2800 0x7fbe600a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:10.629 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:10.628+0000 7fbe79478640 1 --2- 172.21.15.29:0/2863023860 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fbe600a2800 0x7fbe600a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:10.629 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:10.628+0000 7fbe6a7fc640 1 -- 172.21.15.29:0/2863023860 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8211+0+0 (secure 0 0 0) 0x7fbe640cee40 con 0x7fbe741029e0 2024-03-16T11:50:10.629 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:10.629+0000 7fbe79478640 1 --2- 172.21.15.29:0/2863023860 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fbe600a2800 0x7fbe600a4cc0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fbe74102840 tx=0x7fbe70005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:50:10.632 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:10.633+0000 7fbe6a7fc640 1 -- 172.21.15.29:0/2863023860 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fbe6409bef0 con 0x7fbe741029e0 2024-03-16T11:50:10.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:50:10 smithi154 ceph-mon[36320]: pgmap v338: 97 pgs: 97 active+clean; 640 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 597 B/s wr, 7 op/s 2024-03-16T11:50:10.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:50:10 smithi154 ceph-mon[36320]: from='client.? 172.21.15.29:0/2828491680' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T11:50:10.816 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:10.817+0000 7fbe7b6dc640 1 -- 172.21.15.29:0/2863023860 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fbe740623e0 con 0x7fbe741029e0 2024-03-16T11:50:10.816 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:10.817+0000 7fbe6a7fc640 1 -- 172.21.15.29:0/2863023860 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v87) ==== 104+0+11 (secure 0 0 0) 0x7fbe6409b7f0 con 0x7fbe741029e0 2024-03-16T11:50:10.816 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 32 2024-03-16T11:50:10.818 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:10.819+0000 7fbe7b6dc640 1 -- 172.21.15.29:0/2863023860 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fbe600a2800 msgr2=0x7fbe600a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:10.819 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:10.819+0000 7fbe7b6dc640 1 --2- 172.21.15.29:0/2863023860 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fbe600a2800 0x7fbe600a4cc0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fbe74102840 tx=0x7fbe70005f90 comp rx=0 tx=0).stop 2024-03-16T11:50:10.819 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:10.819+0000 7fbe7b6dc640 1 -- 172.21.15.29:0/2863023860 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbe741029e0 msgr2=0x7fbe74197ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:10.819 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:10.819+0000 7fbe7b6dc640 1 --2- 172.21.15.29:0/2863023860 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbe741029e0 0x7fbe74197ff0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fbe6400e9f0 tx=0x7fbe6400edf0 comp rx=0 tx=0).stop 2024-03-16T11:50:10.819 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:10.819+0000 7fbe7b6dc640 1 -- 172.21.15.29:0/2863023860 shutdown_connections 2024-03-16T11:50:10.819 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:10.819+0000 7fbe7b6dc640 1 --2- 172.21.15.29:0/2863023860 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fbe600a2800 0x7fbe600a4cc0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:10.819 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:10.819+0000 7fbe7b6dc640 1 --2- 172.21.15.29:0/2863023860 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbe741033a0 0x7fbe7419f060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:10.819 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:10.819+0000 7fbe7b6dc640 1 --2- 172.21.15.29:0/2863023860 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbe741029e0 0x7fbe74197ff0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:10.819 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:10.819+0000 7fbe7b6dc640 1 --2- 172.21.15.29:0/2863023860 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbe741017f0 0x7fbe74197ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:10.819 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:10.819+0000 7fbe7b6dc640 1 -- 172.21.15.29:0/2863023860 >> 172.21.15.29:0/2863023860 conn(0x7fbe740fcd80 msgr2=0x7fbe74105ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:50:10.819 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:10.819+0000 7fbe7b6dc640 1 -- 172.21.15.29:0/2863023860 shutdown_connections 2024-03-16T11:50:10.819 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:10.819+0000 7fbe7b6dc640 1 -- 172.21.15.29:0/2863023860 wait complete. 2024-03-16T11:50:10.829 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-16T11:50:10.936 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:10.935+0000 7f829b8c6640 1 -- 172.21.15.29:0/1717776178 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8294102290 msgr2=0x7f8294108b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:10.936 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:10.935+0000 7f829b8c6640 1 --2- 172.21.15.29:0/1717776178 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8294102290 0x7f8294108b20 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f828c009ca0 tx=0x7f828c02c0f0 comp rx=0 tx=0).stop 2024-03-16T11:50:10.936 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:10.935+0000 7f829b8c6640 1 -- 172.21.15.29:0/1717776178 shutdown_connections 2024-03-16T11:50:10.936 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:10.935+0000 7f829b8c6640 1 --2- 172.21.15.29:0/1717776178 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8294102290 0x7f8294108b20 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:10.936 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:10.935+0000 7f829b8c6640 1 --2- 172.21.15.29:0/1717776178 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f82941018d0 0x7f8294101d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:10.936 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:10.935+0000 7f829b8c6640 1 --2- 172.21.15.29:0/1717776178 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f82941006e0 0x7f8294100b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:10.937 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:10.935+0000 7f829b8c6640 1 -- 172.21.15.29:0/1717776178 >> 172.21.15.29:0/1717776178 conn(0x7f82940fbc90 msgr2=0x7f82940fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:50:10.937 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:10.935+0000 7f829b8c6640 1 -- 172.21.15.29:0/1717776178 shutdown_connections 2024-03-16T11:50:10.937 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:10.935+0000 7f829b8c6640 1 -- 172.21.15.29:0/1717776178 wait complete. 2024-03-16T11:50:10.937 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:10.936+0000 7f829b8c6640 1 Processor -- start 2024-03-16T11:50:10.937 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:10.936+0000 7f829b8c6640 1 -- start start 2024-03-16T11:50:10.937 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:10.936+0000 7f829b8c6640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f82941006e0 0x7f829406a980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:10.937 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:10.936+0000 7f829b8c6640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f82941018d0 0x7f829419c820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:10.937 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:10.936+0000 7f829b8c6640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8294102290 0x7f829419ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:10.937 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:10.936+0000 7f829b8c6640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f829410c360 con 0x7f8294102290 2024-03-16T11:50:10.937 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:10.936+0000 7f829b8c6640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f829410c1e0 con 0x7f82941006e0 2024-03-16T11:50:10.937 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:10.936+0000 7f829b8c6640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f829410c4e0 con 0x7f82941018d0 2024-03-16T11:50:10.937 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:10.936+0000 7f8299e63640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8294102290 0x7f829419ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:10.937 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:10.936+0000 7f8299e63640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8294102290 0x7f829419ee40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47238/0 (socket says 172.21.15.29:47238) 2024-03-16T11:50:10.937 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:10.936+0000 7f8299e63640 1 -- 172.21.15.29:0/2464034538 learned_addr learned my addr 172.21.15.29:0/2464034538 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:50:10.937 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:10.936+0000 7f8299662640 1 --2- 172.21.15.29:0/2464034538 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f82941006e0 0x7f829406a980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:10.937 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:10.936+0000 7f8298e61640 1 --2- 172.21.15.29:0/2464034538 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f82941018d0 0x7f829419c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:10.938 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:10.936+0000 7f8299e63640 1 -- 172.21.15.29:0/2464034538 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f82941018d0 msgr2=0x7f829419c820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:10.938 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:10.936+0000 7f8299e63640 1 --2- 172.21.15.29:0/2464034538 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f82941018d0 0x7f829419c820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:10.938 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:10.936+0000 7f8299e63640 1 -- 172.21.15.29:0/2464034538 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f82941006e0 msgr2=0x7f829406a980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:10.938 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:10.936+0000 7f8299e63640 1 --2- 172.21.15.29:0/2464034538 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f82941006e0 0x7f829406a980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:10.938 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:10.936+0000 7f8299e63640 1 -- 172.21.15.29:0/2464034538 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f829419f750 con 0x7f8294102290 2024-03-16T11:50:10.938 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:10.937+0000 7f8299e63640 1 --2- 172.21.15.29:0/2464034538 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8294102290 0x7f829419ee40 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f828c037040 tx=0x7f828c032f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:50:10.938 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:10.937+0000 7f828a7fc640 1 -- 172.21.15.29:0/2464034538 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f828c041020 con 0x7f8294102290 2024-03-16T11:50:10.938 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:10.937+0000 7f829b8c6640 1 -- 172.21.15.29:0/2464034538 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f829419f980 con 0x7f8294102290 2024-03-16T11:50:10.938 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:10.937+0000 7f828a7fc640 1 -- 172.21.15.29:0/2464034538 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f828c034cd0 con 0x7f8294102290 2024-03-16T11:50:10.938 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:10.937+0000 7f828a7fc640 1 -- 172.21.15.29:0/2464034538 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f828c039040 con 0x7f8294102290 2024-03-16T11:50:10.938 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:10.937+0000 7f829b8c6640 1 -- 172.21.15.29:0/2464034538 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f829419fe70 con 0x7f8294102290 2024-03-16T11:50:10.939 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:10.938+0000 7f829b8c6640 1 -- 172.21.15.29:0/2464034538 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8294107a00 con 0x7f8294102290 2024-03-16T11:50:10.939 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:10.939+0000 7f828a7fc640 1 -- 172.21.15.29:0/2464034538 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f828c044020 con 0x7f8294102290 2024-03-16T11:50:10.939 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:10.939+0000 7f828a7fc640 1 --2- 172.21.15.29:0/2464034538 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f82800a2800 0x7f82800a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:10.939 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:10.940+0000 7f8299662640 1 --2- 172.21.15.29:0/2464034538 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f82800a2800 0x7f82800a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:10.939 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:10.940+0000 7f828a7fc640 1 -- 172.21.15.29:0/2464034538 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8211+0+0 (secure 0 0 0) 0x7f828c0f0cb0 con 0x7f8294102290 2024-03-16T11:50:10.939 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:10.940+0000 7f8299662640 1 --2- 172.21.15.29:0/2464034538 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f82800a2800 0x7f82800a4cc0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f82900095f0 tx=0x7f8290005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:50:10.942 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:10.943+0000 7f828a7fc640 1 -- 172.21.15.29:0/2464034538 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f828c04b020 con 0x7f8294102290 2024-03-16T11:50:11.128 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:11.129+0000 7f829b8c6640 1 -- 172.21.15.29:0/2464034538 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f829402d340 con 0x7f8294102290 2024-03-16T11:50:11.128 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:11.129+0000 7f828a7fc640 1 -- 172.21.15.29:0/2464034538 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v87) ==== 100+0+11 (secure 0 0 0) 0x7f828c0bddf0 con 0x7f8294102290 2024-03-16T11:50:11.129 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 64 2024-03-16T11:50:11.131 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:11.131+0000 7f829b8c6640 1 -- 172.21.15.29:0/2464034538 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f82800a2800 msgr2=0x7f82800a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:11.131 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:11.131+0000 7f829b8c6640 1 --2- 172.21.15.29:0/2464034538 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f82800a2800 0x7f82800a4cc0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f82900095f0 tx=0x7f8290005f90 comp rx=0 tx=0).stop 2024-03-16T11:50:11.131 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:11.131+0000 7f829b8c6640 1 -- 172.21.15.29:0/2464034538 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8294102290 msgr2=0x7f829419ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:11.131 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:11.131+0000 7f829b8c6640 1 --2- 172.21.15.29:0/2464034538 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8294102290 0x7f829419ee40 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f828c037040 tx=0x7f828c032f40 comp rx=0 tx=0).stop 2024-03-16T11:50:11.131 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:11.131+0000 7f829b8c6640 1 -- 172.21.15.29:0/2464034538 shutdown_connections 2024-03-16T11:50:11.131 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:11.131+0000 7f829b8c6640 1 --2- 172.21.15.29:0/2464034538 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f82800a2800 0x7f82800a4cc0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:11.131 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:11.131+0000 7f829b8c6640 1 --2- 172.21.15.29:0/2464034538 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8294102290 0x7f829419ee40 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:11.131 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:11.131+0000 7f829b8c6640 1 --2- 172.21.15.29:0/2464034538 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f82941018d0 0x7f829419c820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:11.131 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:11.131+0000 7f829b8c6640 1 --2- 172.21.15.29:0/2464034538 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f82941006e0 0x7f829406a980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:11.131 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:11.131+0000 7f829b8c6640 1 -- 172.21.15.29:0/2464034538 >> 172.21.15.29:0/2464034538 conn(0x7f82940fbc90 msgr2=0x7f8294104bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:50:11.131 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:11.131+0000 7f829b8c6640 1 -- 172.21.15.29:0/2464034538 shutdown_connections 2024-03-16T11:50:11.131 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:11.131+0000 7f829b8c6640 1 -- 172.21.15.29:0/2464034538 wait complete. 2024-03-16T11:50:11.141 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T11:50:11.246 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:11.245+0000 7f25272e8640 1 -- 172.21.15.29:0/1375662762 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2520100780 msgr2=0x7f2520100ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:11.246 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:11.245+0000 7f25272e8640 1 --2- 172.21.15.29:0/1375662762 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2520100780 0x7f2520100ba0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f251c009900 tx=0x7f251c02bc30 comp rx=0 tx=0).stop 2024-03-16T11:50:11.246 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:11.245+0000 7f25272e8640 1 -- 172.21.15.29:0/1375662762 shutdown_connections 2024-03-16T11:50:11.246 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:11.245+0000 7f25272e8640 1 --2- 172.21.15.29:0/1375662762 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2520102330 0x7f2520108bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:11.246 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:11.245+0000 7f25272e8640 1 --2- 172.21.15.29:0/1375662762 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2520101970 0x7f2520101df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:11.246 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:11.245+0000 7f25272e8640 1 --2- 172.21.15.29:0/1375662762 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2520100780 0x7f2520100ba0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:11.246 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:11.245+0000 7f25272e8640 1 -- 172.21.15.29:0/1375662762 >> 172.21.15.29:0/1375662762 conn(0x7f25200fbd10 msgr2=0x7f25200fe130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:50:11.246 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:11.245+0000 7f25272e8640 1 -- 172.21.15.29:0/1375662762 shutdown_connections 2024-03-16T11:50:11.247 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:11.245+0000 7f25272e8640 1 -- 172.21.15.29:0/1375662762 wait complete. 2024-03-16T11:50:11.247 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:11.246+0000 7f25272e8640 1 Processor -- start 2024-03-16T11:50:11.247 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:11.246+0000 7f25272e8640 1 -- start start 2024-03-16T11:50:11.247 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:11.246+0000 7f25272e8640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2520100780 0x7f25201046a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:11.247 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:11.246+0000 7f25272e8640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2520101970 0x7f2520102d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:11.247 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:11.246+0000 7f25272e8640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2520102330 0x7f2520103290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:11.247 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:11.246+0000 7f25272e8640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f252010c360 con 0x7f2520101970 2024-03-16T11:50:11.247 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:11.246+0000 7f25272e8640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f252010c1e0 con 0x7f2520100780 2024-03-16T11:50:11.247 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:11.246+0000 7f25272e8640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f252010c4e0 con 0x7f2520102330 2024-03-16T11:50:11.247 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:11.246+0000 7f2524883640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2520101970 0x7f2520102d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:11.247 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:11.246+0000 7f2524883640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2520101970 0x7f2520102d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47262/0 (socket says 172.21.15.29:47262) 2024-03-16T11:50:11.247 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:11.246+0000 7f2524883640 1 -- 172.21.15.29:0/2054424790 learned_addr learned my addr 172.21.15.29:0/2054424790 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:50:11.248 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:11.246+0000 7f2525084640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2520100780 0x7f25201046a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:11.248 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:11.246+0000 7f2525885640 1 --2- 172.21.15.29:0/2054424790 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2520102330 0x7f2520103290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:11.248 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:11.246+0000 7f2524883640 1 -- 172.21.15.29:0/2054424790 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2520102330 msgr2=0x7f2520103290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:11.248 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:11.246+0000 7f2524883640 1 --2- 172.21.15.29:0/2054424790 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2520102330 0x7f2520103290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:11.248 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:11.246+0000 7f2524883640 1 -- 172.21.15.29:0/2054424790 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2520100780 msgr2=0x7f25201046a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:11.248 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:11.246+0000 7f2524883640 1 --2- 172.21.15.29:0/2054424790 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2520100780 0x7f25201046a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:11.248 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:11.246+0000 7f2524883640 1 -- 172.21.15.29:0/2054424790 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2520103ab0 con 0x7f2520101970 2024-03-16T11:50:11.248 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:11.246+0000 7f2524883640 1 --2- 172.21.15.29:0/2054424790 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2520101970 0x7f2520102d50 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f251000ea60 tx=0x7f251000ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:50:11.248 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:11.246+0000 7f25167fc640 1 -- 172.21.15.29:0/2054424790 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f251000cde0 con 0x7f2520101970 2024-03-16T11:50:11.248 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:11.246+0000 7f25167fc640 1 -- 172.21.15.29:0/2054424790 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f25100035f0 con 0x7f2520101970 2024-03-16T11:50:11.248 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:11.246+0000 7f25272e8640 1 -- 172.21.15.29:0/2054424790 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f25201a1c70 con 0x7f2520101970 2024-03-16T11:50:11.248 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:11.246+0000 7f25167fc640 1 -- 172.21.15.29:0/2054424790 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2510010670 con 0x7f2520101970 2024-03-16T11:50:11.248 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:11.247+0000 7f25272e8640 1 -- 172.21.15.29:0/2054424790 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f25201a21c0 con 0x7f2520101970 2024-03-16T11:50:11.249 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:11.247+0000 7f25272e8640 1 -- 172.21.15.29:0/2054424790 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f25201a23f0 con 0x7f2520101970 2024-03-16T11:50:11.249 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:11.248+0000 7f25167fc640 1 -- 172.21.15.29:0/2054424790 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f25100037a0 con 0x7f2520101970 2024-03-16T11:50:11.249 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:11.249+0000 7f25167fc640 1 --2- 172.21.15.29:0/2054424790 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f250c0a2800 0x7f250c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:11.249 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:11.249+0000 7f2525084640 1 --2- 172.21.15.29:0/2054424790 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f250c0a2800 0x7f250c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:11.249 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:11.250+0000 7f25167fc640 1 -- 172.21.15.29:0/2054424790 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8211+0+0 (secure 0 0 0) 0x7f25100cee50 con 0x7f2520101970 2024-03-16T11:50:11.250 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:11.250+0000 7f2525084640 1 --2- 172.21.15.29:0/2054424790 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f250c0a2800 0x7f250c0a4cc0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f251c000c00 tx=0x7f251c005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:50:11.252 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:11.253+0000 7f25167fc640 1 -- 172.21.15.29:0/2054424790 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f251009bf30 con 0x7f2520101970 2024-03-16T11:50:11.475 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:11.476+0000 7f25272e8640 1 -- 172.21.15.29:0/2054424790 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f25201a26e0 con 0x7f2520101970 2024-03-16T11:50:11.475 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:11.477+0000 7f25167fc640 1 -- 172.21.15.29:0/2054424790 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+11875 (secure 0 0 0) 0x7f251009b830 con 0x7f2520101970 2024-03-16T11:50:11.476 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-16T11:50:11.476 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14574,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/3505811955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":3505811955},{"type":"v1","addr":"172.21.15.29:6835","nonce":3505811955}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14586,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/1901337868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":1901337868},{"type":"v1","addr":"172.21.15.29:6837","nonce":1901337868}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14652,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6839/544966534","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6838","nonce":544966534},{"type":"v1","addr":"172.21.15.29:6839","nonce":544966534}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14664,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6841/540705290","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6840","nonce":540705290},{"type":"v1","addr":"172.21.15.29:6841","nonce":540705290}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24407,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6833/3136024628","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6832","nonce":3136024628},{"type":"v1","addr":"172.21.15.152:6833","nonce":3136024628}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24421,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6833/2677426282","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":2677426282},{"type":"v1","addr":"172.21.15.154:6833","nonce":2677426282}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":21,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-16T11:48:38.431509+0000","modified":"2024-03-16T11:48:42.969084+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24503,"mds_1":24499,"mds_2":24464},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24434":{"gid":24434,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.152:6835/2919113178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6834","nonce":2919113178},{"type":"v1","addr":"172.21.15.152:6835","nonce":2919113178}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24464":{"gid":24464,"name":"h","rank":2,"incarnation":19,"state":"up:active","state_seq":7,"addr":"172.21.15.152:6837/1323866722","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6836","nonce":1323866722},{"type":"v1","addr":"172.21.15.152:6837","nonce":1323866722}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.154:6835/4165274681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":4165274681},{"type":"v1","addr":"172.21.15.154:6835","nonce":4165274681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.154:6837/1468560096","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":1468560096},{"type":"v1","addr":"172.21.15.154:6837","nonce":1468560096}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":1,"incarnation":17,"state":"up:active","state_seq":6,"addr":"172.21.15.154:6839/685736862","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":685736862},{"type":"v1","addr":"172.21.15.154:6839","nonce":685736862}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.152:6839/1548731833","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6838","nonce":1548731833},{"type":"v1","addr":"172.21.15.152:6839","nonce":1548731833}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24503,"qdb_cluster":[24503,24464,24499]},"id":1}]} 2024-03-16T11:50:11.476 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 21 2024-03-16T11:50:11.478 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:11.479+0000 7f25272e8640 1 -- 172.21.15.29:0/2054424790 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f250c0a2800 msgr2=0x7f250c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:11.478 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:11.479+0000 7f25272e8640 1 --2- 172.21.15.29:0/2054424790 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f250c0a2800 0x7f250c0a4cc0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f251c000c00 tx=0x7f251c005fd0 comp rx=0 tx=0).stop 2024-03-16T11:50:11.478 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:11.479+0000 7f25272e8640 1 -- 172.21.15.29:0/2054424790 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2520101970 msgr2=0x7f2520102d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:11.478 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:11.479+0000 7f25272e8640 1 --2- 172.21.15.29:0/2054424790 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2520101970 0x7f2520102d50 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f251000ea60 tx=0x7f251000ef20 comp rx=0 tx=0).stop 2024-03-16T11:50:11.479 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:11.479+0000 7f25272e8640 1 -- 172.21.15.29:0/2054424790 shutdown_connections 2024-03-16T11:50:11.479 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:11.479+0000 7f25272e8640 1 --2- 172.21.15.29:0/2054424790 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f250c0a2800 0x7f250c0a4cc0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:11.479 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:11.479+0000 7f25272e8640 1 --2- 172.21.15.29:0/2054424790 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2520102330 0x7f2520103290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:11.479 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:11.479+0000 7f25272e8640 1 --2- 172.21.15.29:0/2054424790 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2520101970 0x7f2520102d50 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:11.479 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:11.479+0000 7f25272e8640 1 --2- 172.21.15.29:0/2054424790 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2520100780 0x7f25201046a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:11.479 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:11.479+0000 7f25272e8640 1 -- 172.21.15.29:0/2054424790 >> 172.21.15.29:0/2054424790 conn(0x7f25200fbd10 msgr2=0x7f25200fd7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:50:11.479 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:11.479+0000 7f25272e8640 1 -- 172.21.15.29:0/2054424790 shutdown_connections 2024-03-16T11:50:11.479 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:11.479+0000 7f25272e8640 1 -- 172.21.15.29:0/2054424790 wait complete. 2024-03-16T11:50:11.488 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:50:11 smithi029 ceph-mon[31975]: from='client.? 172.21.15.29:0/1235505682' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-16T11:50:11.488 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:50:11 smithi029 ceph-mon[31975]: from='client.? 172.21.15.29:0/2863023860' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-16T11:50:11.488 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:50:11 smithi029 ceph-mon[31975]: from='client.? 172.21.15.29:0/2464034538' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-16T11:50:11.489 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-16T11:50:11.592 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:11.592+0000 7f8096475640 1 -- 172.21.15.29:0/2578582303 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8090101910 msgr2=0x7f8090101d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:11.592 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:11.592+0000 7f8096475640 1 --2- 172.21.15.29:0/2578582303 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8090101910 0x7f8090101d90 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f8080009900 tx=0x7f808002bc30 comp rx=0 tx=0).stop 2024-03-16T11:50:11.592 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:11.592+0000 7f8096475640 1 -- 172.21.15.29:0/2578582303 shutdown_connections 2024-03-16T11:50:11.592 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:11.592+0000 7f8096475640 1 --2- 172.21.15.29:0/2578582303 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f80901022d0 0x7f8090108b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:11.592 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:11.592+0000 7f8096475640 1 --2- 172.21.15.29:0/2578582303 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8090101910 0x7f8090101d90 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:11.592 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:11.592+0000 7f8096475640 1 --2- 172.21.15.29:0/2578582303 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8090100720 0x7f8090100b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:11.592 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:11.592+0000 7f8096475640 1 -- 172.21.15.29:0/2578582303 >> 172.21.15.29:0/2578582303 conn(0x7f80900fbc90 msgr2=0x7f80900fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:50:11.592 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:11.592+0000 7f8096475640 1 -- 172.21.15.29:0/2578582303 shutdown_connections 2024-03-16T11:50:11.592 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:11.592+0000 7f8096475640 1 -- 172.21.15.29:0/2578582303 wait complete. 2024-03-16T11:50:11.592 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:11.593+0000 7f8096475640 1 Processor -- start 2024-03-16T11:50:11.592 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:11.593+0000 7f8096475640 1 -- start start 2024-03-16T11:50:11.595 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:11.593+0000 7f8096475640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8090100720 0x7f8090197b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:11.595 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:11.593+0000 7f8096475640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8090101910 0x7f80901980c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:11.595 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:11.593+0000 7f8096475640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f80901022d0 0x7f809019f130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:11.595 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:11.593+0000 7f8096475640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f809010c380 con 0x7f8090101910 2024-03-16T11:50:11.595 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:11.593+0000 7f8096475640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f809010c200 con 0x7f8090100720 2024-03-16T11:50:11.595 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:11.593+0000 7f8096475640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f809010c500 con 0x7f80901022d0 2024-03-16T11:50:11.595 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:11.593+0000 7f808f7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8090101910 0x7f80901980c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:11.595 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:11.593+0000 7f808f7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8090101910 0x7f80901980c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47266/0 (socket says 172.21.15.29:47266) 2024-03-16T11:50:11.595 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:11.593+0000 7f808f7fe640 1 -- 172.21.15.29:0/614719963 learned_addr learned my addr 172.21.15.29:0/614719963 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:50:11.595 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:11.593+0000 7f8094a12640 1 --2- 172.21.15.29:0/614719963 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f80901022d0 0x7f809019f130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:11.595 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:11.593+0000 7f808ffff640 1 --2- 172.21.15.29:0/614719963 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8090100720 0x7f8090197b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:11.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:11.594+0000 7f808f7fe640 1 -- 172.21.15.29:0/614719963 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f80901022d0 msgr2=0x7f809019f130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:11.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:11.594+0000 7f808f7fe640 1 --2- 172.21.15.29:0/614719963 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f80901022d0 0x7f809019f130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:11.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:11.594+0000 7f808f7fe640 1 -- 172.21.15.29:0/614719963 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8090100720 msgr2=0x7f8090197b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:11.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:11.594+0000 7f808f7fe640 1 --2- 172.21.15.29:0/614719963 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8090100720 0x7f8090197b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:11.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:11.594+0000 7f808f7fe640 1 -- 172.21.15.29:0/614719963 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f809019f7a0 con 0x7f8090101910 2024-03-16T11:50:11.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:11.594+0000 7f8094a12640 1 --2- 172.21.15.29:0/614719963 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f80901022d0 0x7f809019f130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:50:11.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:11.594+0000 7f808f7fe640 1 --2- 172.21.15.29:0/614719963 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8090101910 0x7f80901980c0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f8080009230 tx=0x7f808002c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:50:11.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:11.594+0000 7f808d7fa640 1 -- 172.21.15.29:0/614719963 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8080041070 con 0x7f8090101910 2024-03-16T11:50:11.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:11.594+0000 7f808d7fa640 1 -- 172.21.15.29:0/614719963 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8080004450 con 0x7f8090101910 2024-03-16T11:50:11.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:11.594+0000 7f8096475640 1 -- 172.21.15.29:0/614719963 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f809019fa30 con 0x7f8090101910 2024-03-16T11:50:11.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:11.594+0000 7f808d7fa640 1 -- 172.21.15.29:0/614719963 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8080003db0 con 0x7f8090101910 2024-03-16T11:50:11.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:11.594+0000 7f8096475640 1 -- 172.21.15.29:0/614719963 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f809019ff70 con 0x7f8090101910 2024-03-16T11:50:11.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:11.595+0000 7f8096475640 1 -- 172.21.15.29:0/614719963 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f80901a01a0 con 0x7f8090101910 2024-03-16T11:50:11.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:11.596+0000 7f808d7fa640 1 -- 172.21.15.29:0/614719963 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f8080044070 con 0x7f8090101910 2024-03-16T11:50:11.597 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:11.596+0000 7f808d7fa640 1 --2- 172.21.15.29:0/614719963 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f807c0a2880 0x7f807c0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:11.597 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:11.597+0000 7f808ffff640 1 --2- 172.21.15.29:0/614719963 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f807c0a2880 0x7f807c0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:11.597 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:11.597+0000 7f808d7fa640 1 -- 172.21.15.29:0/614719963 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8211+0+0 (secure 0 0 0) 0x7f80800f0520 con 0x7f8090101910 2024-03-16T11:50:11.597 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:11.597+0000 7f808ffff640 1 --2- 172.21.15.29:0/614719963 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f807c0a2880 0x7f807c0a4d40 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f8090101770 tx=0x7f8078005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:50:11.600 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:11.601+0000 7f808d7fa640 1 -- 172.21.15.29:0/614719963 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f80800bd5d0 con 0x7f8090101910 2024-03-16T11:50:11.627 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:50:11 smithi152 ceph-mon[36291]: from='client.? 172.21.15.29:0/1235505682' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-16T11:50:11.627 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:50:11 smithi152 ceph-mon[36291]: from='client.? 172.21.15.29:0/2863023860' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-16T11:50:11.627 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:50:11 smithi152 ceph-mon[36291]: from='client.? 172.21.15.29:0/2464034538' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-16T11:50:11.785 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:11.786+0000 7f8096475640 1 -- 172.21.15.29:0/614719963 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f80901a0930 con 0x7f8090101910 2024-03-16T11:50:11.787 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:11.788+0000 7f808d7fa640 1 -- 172.21.15.29:0/614719963 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) ==== 74+0+22668 (secure 0 0 0) 0x7f80800bced0 con 0x7f8090101910 2024-03-16T11:50:11.787 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-16T11:50:11.787 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":87,"fsid":"d1887f28-e389-11ee-95c9-87774f69a715","created":"2024-03-16T11:40:16.475063+0000","modified":"2024-03-16T11:49:28.628389+0000","last_up_change":"2024-03-16T11:47:44.349348+0000","last_in_change":"2024-03-16T11:47:23.672432+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-16T11:43:51.892652+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-16T11:48:36.709732+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-16T11:48:37.586140+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"60f5608f-a408-45ba-bdf6-5b53758f88fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":2748947913},{"type":"v1","addr":"172.21.15.29:6803","nonce":2748947913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":2748947913},{"type":"v1","addr":"172.21.15.29:6805","nonce":2748947913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":2748947913},{"type":"v1","addr":"172.21.15.29:6809","nonce":2748947913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":2748947913},{"type":"v1","addr":"172.21.15.29:6807","nonce":2748947913}]},"public_addr":"172.21.15.29:6803/2748947913","cluster_addr":"172.21.15.29:6805/2748947913","heartbeat_back_addr":"172.21.15.29:6809/2748947913","heartbeat_front_addr":"172.21.15.29:6807/2748947913","state":["exists","up"]},{"osd":1,"uuid":"e70b53d6-ee78-4e56-9a2c-263ae0501b43","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6800","nonce":2339408422},{"type":"v1","addr":"172.21.15.152:6801","nonce":2339408422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6802","nonce":2339408422},{"type":"v1","addr":"172.21.15.152:6803","nonce":2339408422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6806","nonce":2339408422},{"type":"v1","addr":"172.21.15.152:6807","nonce":2339408422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6804","nonce":2339408422},{"type":"v1","addr":"172.21.15.152:6805","nonce":2339408422}]},"public_addr":"172.21.15.152:6801/2339408422","cluster_addr":"172.21.15.152:6803/2339408422","heartbeat_back_addr":"172.21.15.152:6807/2339408422","heartbeat_front_addr":"172.21.15.152:6805/2339408422","state":["exists","up"]},{"osd":2,"uuid":"ff39389a-dfac-44b5-80ff-36abca316e78","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6800","nonce":4288325280},{"type":"v1","addr":"172.21.15.154:6801","nonce":4288325280}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6802","nonce":4288325280},{"type":"v1","addr":"172.21.15.154:6803","nonce":4288325280}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6806","nonce":4288325280},{"type":"v1","addr":"172.21.15.154:6807","nonce":4288325280}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":4288325280},{"type":"v1","addr":"172.21.15.154:6805","nonce":4288325280}]},"public_addr":"172.21.15.154:6801/4288325280","cluster_addr":"172.21.15.154:6803/4288325280","heartbeat_back_addr":"172.21.15.154:6807/4288325280","heartbeat_front_addr":"172.21.15.154:6805/4288325280","state":["exists","up"]},{"osd":3,"uuid":"069e1795-55bc-4324-b94c-eb24ef2334c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":1538750397},{"type":"v1","addr":"172.21.15.29:6811","nonce":1538750397}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":1538750397},{"type":"v1","addr":"172.21.15.29:6813","nonce":1538750397}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":1538750397},{"type":"v1","addr":"172.21.15.29:6817","nonce":1538750397}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":1538750397},{"type":"v1","addr":"172.21.15.29:6815","nonce":1538750397}]},"public_addr":"172.21.15.29:6811/1538750397","cluster_addr":"172.21.15.29:6813/1538750397","heartbeat_back_addr":"172.21.15.29:6817/1538750397","heartbeat_front_addr":"172.21.15.29:6815/1538750397","state":["exists","up"]},{"osd":4,"uuid":"5a9d43a1-e8f5-4ab5-a441-afbc6ed4066e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6808","nonce":2795081750},{"type":"v1","addr":"172.21.15.152:6809","nonce":2795081750}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6810","nonce":2795081750},{"type":"v1","addr":"172.21.15.152:6811","nonce":2795081750}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6814","nonce":2795081750},{"type":"v1","addr":"172.21.15.152:6815","nonce":2795081750}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6812","nonce":2795081750},{"type":"v1","addr":"172.21.15.152:6813","nonce":2795081750}]},"public_addr":"172.21.15.152:6809/2795081750","cluster_addr":"172.21.15.152:6811/2795081750","heartbeat_back_addr":"172.21.15.152:6815/2795081750","heartbeat_front_addr":"172.21.15.152:6813/2795081750","state":["exists","up"]},{"osd":5,"uuid":"234de428-4ac7-44be-9a54-7b58331fdf8c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":1472686560},{"type":"v1","addr":"172.21.15.154:6809","nonce":1472686560}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6810","nonce":1472686560},{"type":"v1","addr":"172.21.15.154:6811","nonce":1472686560}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6814","nonce":1472686560},{"type":"v1","addr":"172.21.15.154:6815","nonce":1472686560}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6812","nonce":1472686560},{"type":"v1","addr":"172.21.15.154:6813","nonce":1472686560}]},"public_addr":"172.21.15.154:6809/1472686560","cluster_addr":"172.21.15.154:6811/1472686560","heartbeat_back_addr":"172.21.15.154:6815/1472686560","heartbeat_front_addr":"172.21.15.154:6813/1472686560","state":["exists","up"]},{"osd":6,"uuid":"ae45b8d9-7883-40c9-a9cd-4c92b601d4f2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":816506244},{"type":"v1","addr":"172.21.15.29:6819","nonce":816506244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":816506244},{"type":"v1","addr":"172.21.15.29:6821","nonce":816506244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":816506244},{"type":"v1","addr":"172.21.15.29:6825","nonce":816506244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":816506244},{"type":"v1","addr":"172.21.15.29:6823","nonce":816506244}]},"public_addr":"172.21.15.29:6819/816506244","cluster_addr":"172.21.15.29:6821/816506244","heartbeat_back_addr":"172.21.15.29:6825/816506244","heartbeat_front_addr":"172.21.15.29:6823/816506244","state":["exists","up"]},{"osd":7,"uuid":"6f687b28-4fce-4d4a-8b4e-294173d9d6c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6816","nonce":2924651787},{"type":"v1","addr":"172.21.15.152:6817","nonce":2924651787}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6818","nonce":2924651787},{"type":"v1","addr":"172.21.15.152:6819","nonce":2924651787}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6822","nonce":2924651787},{"type":"v1","addr":"172.21.15.152:6823","nonce":2924651787}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6820","nonce":2924651787},{"type":"v1","addr":"172.21.15.152:6821","nonce":2924651787}]},"public_addr":"172.21.15.152:6817/2924651787","cluster_addr":"172.21.15.152:6819/2924651787","heartbeat_back_addr":"172.21.15.152:6823/2924651787","heartbeat_front_addr":"172.21.15.152:6821/2924651787","state":["exists","up"]},{"osd":8,"uuid":"a25fb741-caa7-429a-b540-e394d034b446","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6816","nonce":129974156},{"type":"v1","addr":"172.21.15.154:6817","nonce":129974156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6818","nonce":129974156},{"type":"v1","addr":"172.21.15.154:6819","nonce":129974156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6822","nonce":129974156},{"type":"v1","addr":"172.21.15.154:6823","nonce":129974156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6820","nonce":129974156},{"type":"v1","addr":"172.21.15.154:6821","nonce":129974156}]},"public_addr":"172.21.15.154:6817/129974156","cluster_addr":"172.21.15.154:6819/129974156","heartbeat_back_addr":"172.21.15.154:6823/129974156","heartbeat_front_addr":"172.21.15.154:6821/129974156","state":["exists","up"]},{"osd":9,"uuid":"1071cb0e-8c5f-4cd8-9805-0045e220d946","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6826","nonce":2951414243},{"type":"v1","addr":"172.21.15.29:6827","nonce":2951414243}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":2951414243},{"type":"v1","addr":"172.21.15.29:6829","nonce":2951414243}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":2951414243},{"type":"v1","addr":"172.21.15.29:6833","nonce":2951414243}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":2951414243},{"type":"v1","addr":"172.21.15.29:6831","nonce":2951414243}]},"public_addr":"172.21.15.29:6827/2951414243","cluster_addr":"172.21.15.29:6829/2951414243","heartbeat_back_addr":"172.21.15.29:6833/2951414243","heartbeat_front_addr":"172.21.15.29:6831/2951414243","state":["exists","up"]},{"osd":10,"uuid":"ee7c7d74-3f39-4cd4-be80-843fb1dd9ecc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6824","nonce":3412525480},{"type":"v1","addr":"172.21.15.152:6825","nonce":3412525480}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6826","nonce":3412525480},{"type":"v1","addr":"172.21.15.152:6827","nonce":3412525480}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6830","nonce":3412525480},{"type":"v1","addr":"172.21.15.152:6831","nonce":3412525480}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6828","nonce":3412525480},{"type":"v1","addr":"172.21.15.152:6829","nonce":3412525480}]},"public_addr":"172.21.15.152:6825/3412525480","cluster_addr":"172.21.15.152:6827/3412525480","heartbeat_back_addr":"172.21.15.152:6831/3412525480","heartbeat_front_addr":"172.21.15.152:6829/3412525480","state":["exists","up"]},{"osd":11,"uuid":"4598c803-984a-4de6-ac22-066502c17889","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6824","nonce":1160927966},{"type":"v1","addr":"172.21.15.154:6825","nonce":1160927966}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6826","nonce":1160927966},{"type":"v1","addr":"172.21.15.154:6827","nonce":1160927966}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6830","nonce":1160927966},{"type":"v1","addr":"172.21.15.154:6831","nonce":1160927966}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6828","nonce":1160927966},{"type":"v1","addr":"172.21.15.154:6829","nonce":1160927966}]},"public_addr":"172.21.15.154:6825/1160927966","cluster_addr":"172.21.15.154:6827/1160927966","heartbeat_back_addr":"172.21.15.154:6831/1160927966","heartbeat_front_addr":"172.21.15.154:6829/1160927966","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:43:02.343393+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:43:25.209151+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:43:48.946164+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:44:14.588424+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:44:39.700025+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:45:04.061197+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:45:31.214508+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:45:56.937296+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:46:22.768915+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:46:50.101011+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:47:16.298041+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:47:42.402095+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"2.1b","mappings":[{"from":2,"to":1}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"3.33","mappings":[{"from":5,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.29:0/1853086886":"2024-03-17T11:41:27.531724+0000","172.21.15.29:0/1211659364":"2024-03-17T11:41:27.531724+0000","172.21.15.29:6800/2798831128":"2024-03-17T11:41:27.531724+0000","172.21.15.29:0/1281028285":"2024-03-17T11:41:00.505678+0000","172.21.15.29:6800/2275315654":"2024-03-17T11:41:00.505678+0000","172.21.15.29:6801/2798831128":"2024-03-17T11:41:27.531724+0000","172.21.15.29:6801/2275315654":"2024-03-17T11:41:00.505678+0000","172.21.15.29:0/1330100009":"2024-03-17T11:41:00.505678+0000","172.21.15.29:0/1784249039":"2024-03-17T11:41:00.505678+0000","172.21.15.29:0/1116486932":"2024-03-17T11:41:27.531724+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-16T11:50:11.789 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:11.790+0000 7f8096475640 1 -- 172.21.15.29:0/614719963 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f807c0a2880 msgr2=0x7f807c0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:11.789 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:11.790+0000 7f8096475640 1 --2- 172.21.15.29:0/614719963 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f807c0a2880 0x7f807c0a4d40 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f8090101770 tx=0x7f8078005f90 comp rx=0 tx=0).stop 2024-03-16T11:50:11.789 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:11.790+0000 7f8096475640 1 -- 172.21.15.29:0/614719963 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8090101910 msgr2=0x7f80901980c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:11.789 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:11.790+0000 7f8096475640 1 --2- 172.21.15.29:0/614719963 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8090101910 0x7f80901980c0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f8080009230 tx=0x7f808002c650 comp rx=0 tx=0).stop 2024-03-16T11:50:11.789 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:11.790+0000 7f8096475640 1 -- 172.21.15.29:0/614719963 shutdown_connections 2024-03-16T11:50:11.789 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:11.790+0000 7f8096475640 1 --2- 172.21.15.29:0/614719963 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f807c0a2880 0x7f807c0a4d40 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:11.790 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:11.790+0000 7f8096475640 1 --2- 172.21.15.29:0/614719963 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f80901022d0 0x7f809019f130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:11.790 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:11.790+0000 7f8096475640 1 --2- 172.21.15.29:0/614719963 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8090101910 0x7f80901980c0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:11.790 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:11.790+0000 7f8096475640 1 --2- 172.21.15.29:0/614719963 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8090100720 0x7f8090197b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:11.790 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:11.790+0000 7f8096475640 1 -- 172.21.15.29:0/614719963 >> 172.21.15.29:0/614719963 conn(0x7f80900fbc90 msgr2=0x7f8090069f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:50:11.790 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:11.790+0000 7f8096475640 1 -- 172.21.15.29:0/614719963 shutdown_connections 2024-03-16T11:50:11.790 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:11.790+0000 7f8096475640 1 -- 172.21.15.29:0/614719963 wait complete. 2024-03-16T11:50:11.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:50:11 smithi154 ceph-mon[36320]: from='client.? 172.21.15.29:0/1235505682' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-16T11:50:11.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:50:11 smithi154 ceph-mon[36320]: from='client.? 172.21.15.29:0/2863023860' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-16T11:50:11.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:50:11 smithi154 ceph-mon[36320]: from='client.? 172.21.15.29:0/2464034538' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-16T11:50:11.800 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-16T11:43:51.892652+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-16T11:48:36.709732+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-16T11:48:37.586140+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-16T11:50:11.801 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-16T11:50:11.906 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:11.907+0000 7f9bedd59640 1 -- 172.21.15.29:0/587898883 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9be8102290 msgr2=0x7f9be8108b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:11.906 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:11.907+0000 7f9bedd59640 1 --2- 172.21.15.29:0/587898883 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9be8102290 0x7f9be8108b20 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f9be0009ca0 tx=0x7f9be002c0f0 comp rx=0 tx=0).stop 2024-03-16T11:50:11.906 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:11.907+0000 7f9bedd59640 1 -- 172.21.15.29:0/587898883 shutdown_connections 2024-03-16T11:50:11.906 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:11.907+0000 7f9bedd59640 1 --2- 172.21.15.29:0/587898883 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9be8102290 0x7f9be8108b20 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:11.906 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:11.907+0000 7f9bedd59640 1 --2- 172.21.15.29:0/587898883 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9be81018d0 0x7f9be8101d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:11.906 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:11.907+0000 7f9bedd59640 1 --2- 172.21.15.29:0/587898883 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9be81006e0 0x7f9be8100b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:11.907 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:11.907+0000 7f9bedd59640 1 -- 172.21.15.29:0/587898883 >> 172.21.15.29:0/587898883 conn(0x7f9be80fbc70 msgr2=0x7f9be80fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:50:11.907 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:11.907+0000 7f9bedd59640 1 -- 172.21.15.29:0/587898883 shutdown_connections 2024-03-16T11:50:11.907 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:11.907+0000 7f9bedd59640 1 -- 172.21.15.29:0/587898883 wait complete. 2024-03-16T11:50:11.907 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:11.907+0000 7f9bedd59640 1 Processor -- start 2024-03-16T11:50:11.908 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:11.907+0000 7f9bedd59640 1 -- start start 2024-03-16T11:50:11.908 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:11.907+0000 7f9bedd59640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9be81006e0 0x7f9be80694b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:11.909 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:11.908+0000 7f9bedd59640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9be81018d0 0x7f9be819c800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:11.909 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:11.908+0000 7f9bedd59640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9be8102290 0x7f9be819ee20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:11.909 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:11.908+0000 7f9bedd59640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f9be810c340 con 0x7f9be81018d0 2024-03-16T11:50:11.909 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:11.908+0000 7f9bedd59640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f9be810c1c0 con 0x7f9be8102290 2024-03-16T11:50:11.909 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:11.908+0000 7f9bedd59640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f9be810c4c0 con 0x7f9be81006e0 2024-03-16T11:50:11.909 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:11.908+0000 7f9be6ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9be81018d0 0x7f9be819c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:11.909 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:11.908+0000 7f9be77fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9be81006e0 0x7f9be80694b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:11.909 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:11.908+0000 7f9be77fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9be81006e0 0x7f9be80694b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.29:33766/0 (socket says 172.21.15.29:33766) 2024-03-16T11:50:11.909 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:11.908+0000 7f9be6ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9be81018d0 0x7f9be819c800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47278/0 (socket says 172.21.15.29:47278) 2024-03-16T11:50:11.909 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:11.908+0000 7f9be77fe640 1 -- 172.21.15.29:0/2328829966 learned_addr learned my addr 172.21.15.29:0/2328829966 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:50:11.909 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:11.908+0000 7f9be7fff640 1 --2- 172.21.15.29:0/2328829966 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9be8102290 0x7f9be819ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:11.909 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:11.908+0000 7f9be77fe640 1 -- 172.21.15.29:0/2328829966 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9be8102290 msgr2=0x7f9be819ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:11.909 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:11.908+0000 7f9be77fe640 1 --2- 172.21.15.29:0/2328829966 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9be8102290 0x7f9be819ee20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:11.909 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:11.908+0000 7f9be77fe640 1 -- 172.21.15.29:0/2328829966 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9be81018d0 msgr2=0x7f9be819c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:11.909 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:11.908+0000 7f9be77fe640 1 --2- 172.21.15.29:0/2328829966 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9be81018d0 0x7f9be819c800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:11.910 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:11.908+0000 7f9be77fe640 1 -- 172.21.15.29:0/2328829966 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9be819f730 con 0x7f9be81006e0 2024-03-16T11:50:11.910 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:11.908+0000 7f9be77fe640 1 --2- 172.21.15.29:0/2328829966 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9be81006e0 0x7f9be80694b0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f9bd800e9c0 tx=0x7f9bd800ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:50:11.910 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:11.908+0000 7f9be4ff9640 1 -- 172.21.15.29:0/2328829966 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bd800cde0 con 0x7f9be81006e0 2024-03-16T11:50:11.910 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:11.909+0000 7f9be4ff9640 1 -- 172.21.15.29:0/2328829966 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9bd8010910 con 0x7f9be81006e0 2024-03-16T11:50:11.910 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:11.909+0000 7f9bedd59640 1 -- 172.21.15.29:0/2328829966 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9be819f9c0 con 0x7f9be81006e0 2024-03-16T11:50:11.910 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:11.909+0000 7f9be4ff9640 1 -- 172.21.15.29:0/2328829966 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bd8018a00 con 0x7f9be81006e0 2024-03-16T11:50:11.910 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:11.909+0000 7f9bedd59640 1 -- 172.21.15.29:0/2328829966 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9be819ff60 con 0x7f9be81006e0 2024-03-16T11:50:11.910 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:11.910+0000 7f9bedd59640 1 -- 172.21.15.29:0/2328829966 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9be8107a00 con 0x7f9be81006e0 2024-03-16T11:50:11.911 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:11.911+0000 7f9be4ff9640 1 -- 172.21.15.29:0/2328829966 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f9bd801f020 con 0x7f9be81006e0 2024-03-16T11:50:11.911 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:11.911+0000 7f9be4ff9640 1 --2- 172.21.15.29:0/2328829966 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f9bd00a2880 0x7f9bd00a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:11.911 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:11.911+0000 7f9be6ffd640 1 --2- 172.21.15.29:0/2328829966 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f9bd00a2880 0x7f9bd00a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:11.911 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:11.911+0000 7f9be4ff9640 1 -- 172.21.15.29:0/2328829966 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8211+0+0 (secure 0 0 0) 0x7f9bd80ced60 con 0x7f9be81006e0 2024-03-16T11:50:11.911 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:11.911+0000 7f9be6ffd640 1 --2- 172.21.15.29:0/2328829966 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f9bd00a2880 0x7f9bd00a4d40 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f9bdc0095f0 tx=0x7f9bdc005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:50:11.914 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:11.915+0000 7f9be4ff9640 1 -- 172.21.15.29:0/2328829966 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f9bd809be10 con 0x7f9be81006e0 2024-03-16T11:50:12.095 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:12.096+0000 7f9bedd59640 1 -- 172.21.15.29:0/2328829966 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f9be80623e0 con 0x7f9be81006e0 2024-03-16T11:50:12.096 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:12.097+0000 7f9be4ff9640 1 -- 172.21.15.29:0/2328829966 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v87) ==== 93+0+10 (secure 0 0 0) 0x7f9bd809b710 con 0x7f9be81006e0 2024-03-16T11:50:12.096 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2024-03-16T11:50:12.098 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:12.099+0000 7f9bedd59640 1 -- 172.21.15.29:0/2328829966 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f9bd00a2880 msgr2=0x7f9bd00a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:12.098 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:12.099+0000 7f9bedd59640 1 --2- 172.21.15.29:0/2328829966 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f9bd00a2880 0x7f9bd00a4d40 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f9bdc0095f0 tx=0x7f9bdc005f90 comp rx=0 tx=0).stop 2024-03-16T11:50:12.098 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:12.099+0000 7f9bedd59640 1 -- 172.21.15.29:0/2328829966 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9be81006e0 msgr2=0x7f9be80694b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:12.098 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:12.099+0000 7f9bedd59640 1 --2- 172.21.15.29:0/2328829966 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9be81006e0 0x7f9be80694b0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f9bd800e9c0 tx=0x7f9bd800ee80 comp rx=0 tx=0).stop 2024-03-16T11:50:12.098 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:12.099+0000 7f9bedd59640 1 -- 172.21.15.29:0/2328829966 shutdown_connections 2024-03-16T11:50:12.099 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:12.099+0000 7f9bedd59640 1 --2- 172.21.15.29:0/2328829966 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f9bd00a2880 0x7f9bd00a4d40 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:12.099 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:12.099+0000 7f9bedd59640 1 --2- 172.21.15.29:0/2328829966 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9be8102290 0x7f9be819ee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:12.099 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:12.099+0000 7f9bedd59640 1 --2- 172.21.15.29:0/2328829966 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9be81018d0 0x7f9be819c800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:12.099 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:12.099+0000 7f9bedd59640 1 --2- 172.21.15.29:0/2328829966 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9be81006e0 0x7f9be80694b0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:12.099 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:12.099+0000 7f9bedd59640 1 -- 172.21.15.29:0/2328829966 >> 172.21.15.29:0/2328829966 conn(0x7f9be80fbc70 msgr2=0x7f9be8104bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:50:12.099 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:12.099+0000 7f9bedd59640 1 -- 172.21.15.29:0/2328829966 shutdown_connections 2024-03-16T11:50:12.099 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:12.099+0000 7f9bedd59640 1 -- 172.21.15.29:0/2328829966 wait complete. 2024-03-16T11:50:12.109 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-16T11:50:12.216 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:12.215+0000 7f6946285640 1 -- 172.21.15.29:0/197597343 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f69401022b0 msgr2=0x7f6940108b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:12.216 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:12.215+0000 7f6946285640 1 --2- 172.21.15.29:0/197597343 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f69401022b0 0x7f6940108b40 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f6938009ca0 tx=0x7f693802c0f0 comp rx=0 tx=0).stop 2024-03-16T11:50:12.216 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:12.215+0000 7f6946285640 1 -- 172.21.15.29:0/197597343 shutdown_connections 2024-03-16T11:50:12.216 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:12.215+0000 7f6946285640 1 --2- 172.21.15.29:0/197597343 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f69401022b0 0x7f6940108b40 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:12.216 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:12.215+0000 7f6946285640 1 --2- 172.21.15.29:0/197597343 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f69401018f0 0x7f6940101d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:12.216 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:12.215+0000 7f6946285640 1 --2- 172.21.15.29:0/197597343 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6940100700 0x7f6940100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:12.216 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:12.215+0000 7f6946285640 1 -- 172.21.15.29:0/197597343 >> 172.21.15.29:0/197597343 conn(0x7f69400fbc90 msgr2=0x7f69400fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:50:12.216 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:12.215+0000 7f6946285640 1 -- 172.21.15.29:0/197597343 shutdown_connections 2024-03-16T11:50:12.216 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:12.215+0000 7f6946285640 1 -- 172.21.15.29:0/197597343 wait complete. 2024-03-16T11:50:12.216 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:12.215+0000 7f6946285640 1 Processor -- start 2024-03-16T11:50:12.216 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:12.215+0000 7f6946285640 1 -- start start 2024-03-16T11:50:12.217 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:12.216+0000 7f6946285640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6940100700 0x7f69401969a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:12.217 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:12.216+0000 7f6946285640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f69401018f0 0x7f6940196ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:12.217 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:12.216+0000 7f6946285640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f69401022b0 0x7f694019df50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:12.217 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:12.216+0000 7f6946285640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f694010c380 con 0x7f6940100700 2024-03-16T11:50:12.217 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:12.216+0000 7f6946285640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f694010c200 con 0x7f69401022b0 2024-03-16T11:50:12.217 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:12.216+0000 7f6946285640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f694010c500 con 0x7f69401018f0 2024-03-16T11:50:12.217 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:12.216+0000 7f693ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6940100700 0x7f69401969a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:12.217 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:12.216+0000 7f693ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6940100700 0x7f69401969a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47302/0 (socket says 172.21.15.29:47302) 2024-03-16T11:50:12.217 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:12.216+0000 7f693ffff640 1 -- 172.21.15.29:0/3188069105 learned_addr learned my addr 172.21.15.29:0/3188069105 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:50:12.217 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:12.216+0000 7f6944822640 1 --2- 172.21.15.29:0/3188069105 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f69401022b0 0x7f694019df50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:12.217 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:12.216+0000 7f693f7fe640 1 --2- 172.21.15.29:0/3188069105 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f69401018f0 0x7f6940196ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:12.217 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:12.216+0000 7f693ffff640 1 -- 172.21.15.29:0/3188069105 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f69401018f0 msgr2=0x7f6940196ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:12.217 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:12.216+0000 7f693ffff640 1 --2- 172.21.15.29:0/3188069105 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f69401018f0 0x7f6940196ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:12.217 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:12.216+0000 7f693ffff640 1 -- 172.21.15.29:0/3188069105 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f69401022b0 msgr2=0x7f694019df50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:12.217 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:12.216+0000 7f693ffff640 1 --2- 172.21.15.29:0/3188069105 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f69401022b0 0x7f694019df50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:12.217 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:12.216+0000 7f693ffff640 1 -- 172.21.15.29:0/3188069105 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f694019e5c0 con 0x7f6940100700 2024-03-16T11:50:12.218 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:12.216+0000 7f693ffff640 1 --2- 172.21.15.29:0/3188069105 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6940100700 0x7f69401969a0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f693400e9c0 tx=0x7f693400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:50:12.218 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:12.216+0000 7f693d7fa640 1 -- 172.21.15.29:0/3188069105 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f693400cde0 con 0x7f6940100700 2024-03-16T11:50:12.218 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:12.216+0000 7f693d7fa640 1 -- 172.21.15.29:0/3188069105 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6934010910 con 0x7f6940100700 2024-03-16T11:50:12.218 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:12.216+0000 7f6946285640 1 -- 172.21.15.29:0/3188069105 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f694019e8b0 con 0x7f6940100700 2024-03-16T11:50:12.218 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:12.216+0000 7f693d7fa640 1 -- 172.21.15.29:0/3188069105 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6934018a00 con 0x7f6940100700 2024-03-16T11:50:12.218 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:12.216+0000 7f6946285640 1 -- 172.21.15.29:0/3188069105 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f694019ee50 con 0x7f6940100700 2024-03-16T11:50:12.219 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:12.217+0000 7f6946285640 1 -- 172.21.15.29:0/3188069105 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6940107a20 con 0x7f6940100700 2024-03-16T11:50:12.219 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:12.218+0000 7f693d7fa640 1 -- 172.21.15.29:0/3188069105 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f693401f020 con 0x7f6940100700 2024-03-16T11:50:12.219 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:12.219+0000 7f693d7fa640 1 --2- 172.21.15.29:0/3188069105 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f69280a2880 0x7f69280a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:12.219 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:12.219+0000 7f693f7fe640 1 --2- 172.21.15.29:0/3188069105 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f69280a2880 0x7f69280a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:12.219 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:12.219+0000 7f693d7fa640 1 -- 172.21.15.29:0/3188069105 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8211+0+0 (secure 0 0 0) 0x7f69340cebb0 con 0x7f6940100700 2024-03-16T11:50:12.219 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:12.219+0000 7f693f7fe640 1 --2- 172.21.15.29:0/3188069105 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f69280a2880 0x7f69280a4d40 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f6940197eb0 tx=0x7f6930005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:50:12.222 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:12.224+0000 7f693d7fa640 1 -- 172.21.15.29:0/3188069105 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f693409bc60 con 0x7f6940100700 2024-03-16T11:50:12.407 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:12.408+0000 7f6946285640 1 -- 172.21.15.29:0/3188069105 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f69400623e0 con 0x7f6940100700 2024-03-16T11:50:12.407 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:12.408+0000 7f693d7fa640 1 -- 172.21.15.29:0/3188069105 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v87) ==== 104+0+11 (secure 0 0 0) 0x7f693409b560 con 0x7f6940100700 2024-03-16T11:50:12.408 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 32 2024-03-16T11:50:12.410 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:12.411+0000 7f6946285640 1 -- 172.21.15.29:0/3188069105 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f69280a2880 msgr2=0x7f69280a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:12.410 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:12.411+0000 7f6946285640 1 --2- 172.21.15.29:0/3188069105 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f69280a2880 0x7f69280a4d40 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f6940197eb0 tx=0x7f6930005f90 comp rx=0 tx=0).stop 2024-03-16T11:50:12.410 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:12.411+0000 7f6946285640 1 -- 172.21.15.29:0/3188069105 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6940100700 msgr2=0x7f69401969a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:12.410 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:12.411+0000 7f6946285640 1 --2- 172.21.15.29:0/3188069105 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6940100700 0x7f69401969a0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f693400e9c0 tx=0x7f693400ee80 comp rx=0 tx=0).stop 2024-03-16T11:50:12.410 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:12.411+0000 7f6946285640 1 -- 172.21.15.29:0/3188069105 shutdown_connections 2024-03-16T11:50:12.410 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:12.411+0000 7f6946285640 1 --2- 172.21.15.29:0/3188069105 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f69280a2880 0x7f69280a4d40 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:12.410 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:12.411+0000 7f6946285640 1 --2- 172.21.15.29:0/3188069105 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f69401022b0 0x7f694019df50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:12.410 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:12.411+0000 7f6946285640 1 --2- 172.21.15.29:0/3188069105 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f69401018f0 0x7f6940196ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:12.411 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:12.411+0000 7f6946285640 1 --2- 172.21.15.29:0/3188069105 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6940100700 0x7f69401969a0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:12.411 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:12.411+0000 7f6946285640 1 -- 172.21.15.29:0/3188069105 >> 172.21.15.29:0/3188069105 conn(0x7f69400fbc90 msgr2=0x7f6940104bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:50:12.411 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:12.411+0000 7f6946285640 1 -- 172.21.15.29:0/3188069105 shutdown_connections 2024-03-16T11:50:12.411 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:12.411+0000 7f6946285640 1 -- 172.21.15.29:0/3188069105 wait complete. 2024-03-16T11:50:12.420 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:50:12 smithi029 ceph-mon[31975]: from='client.? 172.21.15.29:0/2054424790' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T11:50:12.420 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:50:12 smithi029 ceph-mon[31975]: pgmap v339: 97 pgs: 97 active+clean; 640 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 170 B/s wr, 5 op/s 2024-03-16T11:50:12.420 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:50:12 smithi029 ceph-mon[31975]: from='client.? 172.21.15.29:0/614719963' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T11:50:12.420 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:50:12 smithi029 ceph-mon[31975]: from='client.? 172.21.15.29:0/2328829966' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-16T11:50:12.421 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-16T11:50:12.528 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:12.527+0000 7f9257df3640 1 -- 172.21.15.29:0/1577950752 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f92501018d0 msgr2=0x7f9250101d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:12.528 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:12.527+0000 7f9257df3640 1 --2- 172.21.15.29:0/1577950752 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f92501018d0 0x7f9250101d50 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f9240009900 tx=0x7f924002bc30 comp rx=0 tx=0).stop 2024-03-16T11:50:12.528 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:12.527+0000 7f9257df3640 1 -- 172.21.15.29:0/1577950752 shutdown_connections 2024-03-16T11:50:12.528 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:12.527+0000 7f9257df3640 1 --2- 172.21.15.29:0/1577950752 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9250102290 0x7f9250108b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:12.529 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:12.527+0000 7f9257df3640 1 --2- 172.21.15.29:0/1577950752 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f92501018d0 0x7f9250101d50 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:12.529 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:12.527+0000 7f9257df3640 1 --2- 172.21.15.29:0/1577950752 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f92501006e0 0x7f9250100b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:12.529 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:12.527+0000 7f9257df3640 1 -- 172.21.15.29:0/1577950752 >> 172.21.15.29:0/1577950752 conn(0x7f92500fbc70 msgr2=0x7f92500fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:50:12.529 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:12.527+0000 7f9257df3640 1 -- 172.21.15.29:0/1577950752 shutdown_connections 2024-03-16T11:50:12.529 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:12.527+0000 7f9257df3640 1 -- 172.21.15.29:0/1577950752 wait complete. 2024-03-16T11:50:12.529 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:12.528+0000 7f9257df3640 1 Processor -- start 2024-03-16T11:50:12.529 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:12.528+0000 7f9257df3640 1 -- start start 2024-03-16T11:50:12.529 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:12.528+0000 7f9257df3640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f92501006e0 0x7f9250104670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:12.529 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:12.528+0000 7f9257df3640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f92501018d0 0x7f9250102cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:12.529 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:12.528+0000 7f9257df3640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9250102290 0x7f92501031f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:12.529 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:12.528+0000 7f9257df3640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f925010c390 con 0x7f9250102290 2024-03-16T11:50:12.529 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:12.528+0000 7f9257df3640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f925010c210 con 0x7f92501006e0 2024-03-16T11:50:12.529 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:12.528+0000 7f9257df3640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f925010c510 con 0x7f92501018d0 2024-03-16T11:50:12.529 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:12.528+0000 7f9256390640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9250102290 0x7f92501031f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:12.530 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:12.528+0000 7f9256390640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9250102290 0x7f92501031f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47320/0 (socket says 172.21.15.29:47320) 2024-03-16T11:50:12.530 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:12.528+0000 7f9255b8f640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f92501006e0 0x7f9250104670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:12.530 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:12.528+0000 7f9256390640 1 -- 172.21.15.29:0/1802217765 learned_addr learned my addr 172.21.15.29:0/1802217765 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:50:12.530 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:12.528+0000 7f925538e640 1 --2- 172.21.15.29:0/1802217765 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f92501018d0 0x7f9250102cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:12.530 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:12.528+0000 7f9256390640 1 -- 172.21.15.29:0/1802217765 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f92501018d0 msgr2=0x7f9250102cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:12.530 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:12.528+0000 7f9256390640 1 --2- 172.21.15.29:0/1802217765 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f92501018d0 0x7f9250102cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:12.530 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:12.528+0000 7f9256390640 1 -- 172.21.15.29:0/1802217765 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f92501006e0 msgr2=0x7f9250104670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:12.530 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:12.528+0000 7f9256390640 1 --2- 172.21.15.29:0/1802217765 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f92501006e0 0x7f9250104670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:12.530 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:12.528+0000 7f9256390640 1 -- 172.21.15.29:0/1802217765 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9250103a80 con 0x7f9250102290 2024-03-16T11:50:12.530 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:12.528+0000 7f9256390640 1 --2- 172.21.15.29:0/1802217765 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9250102290 0x7f92501031f0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f924800eb10 tx=0x7f924800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:50:12.530 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:12.528+0000 7f9246ffd640 1 -- 172.21.15.29:0/1802217765 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9248004b20 con 0x7f9250102290 2024-03-16T11:50:12.530 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:12.529+0000 7f9246ffd640 1 -- 172.21.15.29:0/1802217765 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9248004cc0 con 0x7f9250102290 2024-03-16T11:50:12.530 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:12.529+0000 7f9257df3640 1 -- 172.21.15.29:0/1802217765 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f92501a1c20 con 0x7f9250102290 2024-03-16T11:50:12.531 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:12.529+0000 7f9246ffd640 1 -- 172.21.15.29:0/1802217765 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9248010760 con 0x7f9250102290 2024-03-16T11:50:12.531 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:12.529+0000 7f9257df3640 1 -- 172.21.15.29:0/1802217765 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f92501a20f0 con 0x7f9250102290 2024-03-16T11:50:12.531 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:12.529+0000 7f9257df3640 1 -- 172.21.15.29:0/1802217765 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f92500685d0 con 0x7f9250102290 2024-03-16T11:50:12.531 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:12.531+0000 7f9246ffd640 1 -- 172.21.15.29:0/1802217765 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f9248005020 con 0x7f9250102290 2024-03-16T11:50:12.531 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:12.531+0000 7f9246ffd640 1 --2- 172.21.15.29:0/1802217765 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f923c0a2800 0x7f923c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:12.532 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:12.531+0000 7f9255b8f640 1 --2- 172.21.15.29:0/1802217765 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f923c0a2800 0x7f923c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:12.532 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:12.532+0000 7f9246ffd640 1 -- 172.21.15.29:0/1802217765 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8211+0+0 (secure 0 0 0) 0x7f92480cef90 con 0x7f9250102290 2024-03-16T11:50:12.532 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:12.532+0000 7f9255b8f640 1 --2- 172.21.15.29:0/1802217765 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f923c0a2800 0x7f923c0a4cc0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f9250101730 tx=0x7f924c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:50:12.535 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:12.536+0000 7f9246ffd640 1 -- 172.21.15.29:0/1802217765 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f924809c0e0 con 0x7f9250102290 2024-03-16T11:50:12.627 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:50:12 smithi152 ceph-mon[36291]: from='client.? 172.21.15.29:0/2054424790' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T11:50:12.627 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:50:12 smithi152 ceph-mon[36291]: pgmap v339: 97 pgs: 97 active+clean; 640 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 170 B/s wr, 5 op/s 2024-03-16T11:50:12.627 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:50:12 smithi152 ceph-mon[36291]: from='client.? 172.21.15.29:0/614719963' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T11:50:12.627 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:50:12 smithi152 ceph-mon[36291]: from='client.? 172.21.15.29:0/2328829966' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-16T11:50:12.719 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:12.720+0000 7f9257df3640 1 -- 172.21.15.29:0/1802217765 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f92501a2570 con 0x7f9250102290 2024-03-16T11:50:12.719 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:12.720+0000 7f9246ffd640 1 -- 172.21.15.29:0/1802217765 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v87) ==== 100+0+11 (secure 0 0 0) 0x7f924809b9e0 con 0x7f9250102290 2024-03-16T11:50:12.719 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 64 2024-03-16T11:50:12.721 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:12.722+0000 7f9257df3640 1 -- 172.21.15.29:0/1802217765 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f923c0a2800 msgr2=0x7f923c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:12.721 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:12.722+0000 7f9257df3640 1 --2- 172.21.15.29:0/1802217765 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f923c0a2800 0x7f923c0a4cc0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f9250101730 tx=0x7f924c005f90 comp rx=0 tx=0).stop 2024-03-16T11:50:12.722 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:12.722+0000 7f9257df3640 1 -- 172.21.15.29:0/1802217765 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9250102290 msgr2=0x7f92501031f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:12.722 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:12.722+0000 7f9257df3640 1 --2- 172.21.15.29:0/1802217765 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9250102290 0x7f92501031f0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f924800eb10 tx=0x7f924800c2d0 comp rx=0 tx=0).stop 2024-03-16T11:50:12.722 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:12.722+0000 7f9257df3640 1 -- 172.21.15.29:0/1802217765 shutdown_connections 2024-03-16T11:50:12.722 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:12.722+0000 7f9257df3640 1 --2- 172.21.15.29:0/1802217765 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f923c0a2800 0x7f923c0a4cc0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:12.722 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:12.722+0000 7f9257df3640 1 --2- 172.21.15.29:0/1802217765 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9250102290 0x7f92501031f0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:12.722 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:12.722+0000 7f9257df3640 1 --2- 172.21.15.29:0/1802217765 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f92501018d0 0x7f9250102cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:12.722 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:12.722+0000 7f9257df3640 1 --2- 172.21.15.29:0/1802217765 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f92501006e0 0x7f9250104670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:12.722 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:12.722+0000 7f9257df3640 1 -- 172.21.15.29:0/1802217765 >> 172.21.15.29:0/1802217765 conn(0x7f92500fbc70 msgr2=0x7f92500fd700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:50:12.722 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:12.722+0000 7f9257df3640 1 -- 172.21.15.29:0/1802217765 shutdown_connections 2024-03-16T11:50:12.722 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:12.722+0000 7f9257df3640 1 -- 172.21.15.29:0/1802217765 wait complete. 2024-03-16T11:50:12.732 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T11:50:12.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:50:12 smithi154 ceph-mon[36320]: from='client.? 172.21.15.29:0/2054424790' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T11:50:12.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:50:12 smithi154 ceph-mon[36320]: pgmap v339: 97 pgs: 97 active+clean; 640 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 170 B/s wr, 5 op/s 2024-03-16T11:50:12.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:50:12 smithi154 ceph-mon[36320]: from='client.? 172.21.15.29:0/614719963' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T11:50:12.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:50:12 smithi154 ceph-mon[36320]: from='client.? 172.21.15.29:0/2328829966' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-16T11:50:12.835 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:12.836+0000 7f04f24b2640 1 -- 172.21.15.29:0/1946053736 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f04ec1006f0 msgr2=0x7f04ec100b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:12.835 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:12.836+0000 7f04f24b2640 1 --2- 172.21.15.29:0/1946053736 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f04ec1006f0 0x7f04ec100b10 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f04dc009900 tx=0x7f04dc02bc30 comp rx=0 tx=0).stop 2024-03-16T11:50:12.835 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:12.836+0000 7f04f24b2640 1 -- 172.21.15.29:0/1946053736 shutdown_connections 2024-03-16T11:50:12.835 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:12.836+0000 7f04f24b2640 1 --2- 172.21.15.29:0/1946053736 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f04ec1022a0 0x7f04ec108b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:12.836 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:12.836+0000 7f04f24b2640 1 --2- 172.21.15.29:0/1946053736 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f04ec1018e0 0x7f04ec101d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:12.836 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:12.836+0000 7f04f24b2640 1 --2- 172.21.15.29:0/1946053736 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f04ec1006f0 0x7f04ec100b10 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:12.836 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:12.836+0000 7f04f24b2640 1 -- 172.21.15.29:0/1946053736 >> 172.21.15.29:0/1946053736 conn(0x7f04ec0fbc80 msgr2=0x7f04ec0fe0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:50:12.836 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:12.836+0000 7f04f24b2640 1 -- 172.21.15.29:0/1946053736 shutdown_connections 2024-03-16T11:50:12.836 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:12.836+0000 7f04f24b2640 1 -- 172.21.15.29:0/1946053736 wait complete. 2024-03-16T11:50:12.836 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:12.836+0000 7f04f24b2640 1 Processor -- start 2024-03-16T11:50:12.836 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:12.836+0000 7f04f24b2640 1 -- start start 2024-03-16T11:50:12.838 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:12.837+0000 7f04f24b2640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f04ec1006f0 0x7f04ec199d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:12.838 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:12.837+0000 7f04f24b2640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f04ec1018e0 0x7f04ec19a290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:12.838 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:12.837+0000 7f04f24b2640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f04ec1022a0 0x7f04ec1a12c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:12.838 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:12.837+0000 7f04f24b2640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f04ec10c340 con 0x7f04ec1018e0 2024-03-16T11:50:12.838 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:12.837+0000 7f04f24b2640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f04ec10c1c0 con 0x7f04ec1006f0 2024-03-16T11:50:12.838 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:12.837+0000 7f04f24b2640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f04ec10c4c0 con 0x7f04ec1022a0 2024-03-16T11:50:12.838 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:12.837+0000 7f04eb7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f04ec1018e0 0x7f04ec19a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:12.838 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:12.837+0000 7f04eb7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f04ec1018e0 0x7f04ec19a290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47340/0 (socket says 172.21.15.29:47340) 2024-03-16T11:50:12.838 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:12.837+0000 7f04eb7fe640 1 -- 172.21.15.29:0/4101459434 learned_addr learned my addr 172.21.15.29:0/4101459434 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:50:12.839 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:12.837+0000 7f04ebfff640 1 --2- 172.21.15.29:0/4101459434 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f04ec1006f0 0x7f04ec199d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:12.839 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:12.837+0000 7f04f0a4f640 1 --2- 172.21.15.29:0/4101459434 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f04ec1022a0 0x7f04ec1a12c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:12.839 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:12.837+0000 7f04eb7fe640 1 -- 172.21.15.29:0/4101459434 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f04ec1022a0 msgr2=0x7f04ec1a12c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:12.839 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:12.837+0000 7f04eb7fe640 1 --2- 172.21.15.29:0/4101459434 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f04ec1022a0 0x7f04ec1a12c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:12.839 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:12.837+0000 7f04eb7fe640 1 -- 172.21.15.29:0/4101459434 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f04ec1006f0 msgr2=0x7f04ec199d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:12.839 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:12.837+0000 7f04eb7fe640 1 --2- 172.21.15.29:0/4101459434 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f04ec1006f0 0x7f04ec199d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:12.839 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:12.837+0000 7f04eb7fe640 1 -- 172.21.15.29:0/4101459434 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04ec1a1960 con 0x7f04ec1018e0 2024-03-16T11:50:12.839 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:12.837+0000 7f04eb7fe640 1 --2- 172.21.15.29:0/4101459434 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f04ec1018e0 0x7f04ec19a290 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f04e000e9f0 tx=0x7f04e000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:50:12.839 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:12.837+0000 7f04e97fa640 1 -- 172.21.15.29:0/4101459434 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04e000cd80 con 0x7f04ec1018e0 2024-03-16T11:50:12.839 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:12.837+0000 7f04f24b2640 1 -- 172.21.15.29:0/4101459434 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f04ec1a1c50 con 0x7f04ec1018e0 2024-03-16T11:50:12.839 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:12.837+0000 7f04e97fa640 1 -- 172.21.15.29:0/4101459434 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f04e0010910 con 0x7f04ec1018e0 2024-03-16T11:50:12.839 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:12.838+0000 7f04e97fa640 1 -- 172.21.15.29:0/4101459434 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04e0018990 con 0x7f04ec1018e0 2024-03-16T11:50:12.839 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:12.838+0000 7f04f24b2640 1 -- 172.21.15.29:0/4101459434 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f04ec1a2170 con 0x7f04ec1018e0 2024-03-16T11:50:12.839 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:12.838+0000 7f04f24b2640 1 -- 172.21.15.29:0/4101459434 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f04ec1a2340 con 0x7f04ec1018e0 2024-03-16T11:50:12.840 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:12.839+0000 7f04e97fa640 1 -- 172.21.15.29:0/4101459434 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f04e0010ac0 con 0x7f04ec1018e0 2024-03-16T11:50:12.840 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:12.840+0000 7f04e97fa640 1 --2- 172.21.15.29:0/4101459434 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f04d40a2800 0x7f04d40a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:12.840 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:12.840+0000 7f04ebfff640 1 --2- 172.21.15.29:0/4101459434 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f04d40a2800 0x7f04d40a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:12.840 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:12.840+0000 7f04e97fa640 1 -- 172.21.15.29:0/4101459434 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8211+0+0 (secure 0 0 0) 0x7f04e00cee30 con 0x7f04ec1018e0 2024-03-16T11:50:12.840 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:12.840+0000 7f04ebfff640 1 --2- 172.21.15.29:0/4101459434 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f04d40a2800 0x7f04d40a4cc0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f04dc000c00 tx=0x7f04dc005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:50:12.843 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:12.844+0000 7f04e97fa640 1 -- 172.21.15.29:0/4101459434 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f04e009b890 con 0x7f04ec1018e0 2024-03-16T11:50:13.066 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.067+0000 7f04f24b2640 1 -- 172.21.15.29:0/4101459434 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f04ec1a2510 con 0x7f04ec1018e0 2024-03-16T11:50:13.067 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.068+0000 7f04e97fa640 1 -- 172.21.15.29:0/4101459434 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+11875 (secure 0 0 0) 0x7f04e009f930 con 0x7f04ec1018e0 2024-03-16T11:50:13.067 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-16T11:50:13.067 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14574,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/3505811955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":3505811955},{"type":"v1","addr":"172.21.15.29:6835","nonce":3505811955}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14586,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/1901337868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":1901337868},{"type":"v1","addr":"172.21.15.29:6837","nonce":1901337868}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14652,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6839/544966534","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6838","nonce":544966534},{"type":"v1","addr":"172.21.15.29:6839","nonce":544966534}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14664,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6841/540705290","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6840","nonce":540705290},{"type":"v1","addr":"172.21.15.29:6841","nonce":540705290}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24407,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6833/3136024628","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6832","nonce":3136024628},{"type":"v1","addr":"172.21.15.152:6833","nonce":3136024628}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24421,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6833/2677426282","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":2677426282},{"type":"v1","addr":"172.21.15.154:6833","nonce":2677426282}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":21,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-16T11:48:38.431509+0000","modified":"2024-03-16T11:48:42.969084+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24503,"mds_1":24499,"mds_2":24464},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24434":{"gid":24434,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.152:6835/2919113178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6834","nonce":2919113178},{"type":"v1","addr":"172.21.15.152:6835","nonce":2919113178}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24464":{"gid":24464,"name":"h","rank":2,"incarnation":19,"state":"up:active","state_seq":7,"addr":"172.21.15.152:6837/1323866722","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6836","nonce":1323866722},{"type":"v1","addr":"172.21.15.152:6837","nonce":1323866722}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.154:6835/4165274681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":4165274681},{"type":"v1","addr":"172.21.15.154:6835","nonce":4165274681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.154:6837/1468560096","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":1468560096},{"type":"v1","addr":"172.21.15.154:6837","nonce":1468560096}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":1,"incarnation":17,"state":"up:active","state_seq":6,"addr":"172.21.15.154:6839/685736862","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":685736862},{"type":"v1","addr":"172.21.15.154:6839","nonce":685736862}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.152:6839/1548731833","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6838","nonce":1548731833},{"type":"v1","addr":"172.21.15.152:6839","nonce":1548731833}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24503,"qdb_cluster":[24503,24464,24499]},"id":1}]} 2024-03-16T11:50:13.068 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 21 2024-03-16T11:50:13.070 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.070+0000 7f04f24b2640 1 -- 172.21.15.29:0/4101459434 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f04d40a2800 msgr2=0x7f04d40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:13.070 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.070+0000 7f04f24b2640 1 --2- 172.21.15.29:0/4101459434 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f04d40a2800 0x7f04d40a4cc0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f04dc000c00 tx=0x7f04dc005fd0 comp rx=0 tx=0).stop 2024-03-16T11:50:13.070 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.070+0000 7f04f24b2640 1 -- 172.21.15.29:0/4101459434 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f04ec1018e0 msgr2=0x7f04ec19a290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:13.070 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.070+0000 7f04f24b2640 1 --2- 172.21.15.29:0/4101459434 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f04ec1018e0 0x7f04ec19a290 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f04e000e9f0 tx=0x7f04e000eeb0 comp rx=0 tx=0).stop 2024-03-16T11:50:13.070 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.070+0000 7f04f24b2640 1 -- 172.21.15.29:0/4101459434 shutdown_connections 2024-03-16T11:50:13.070 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.070+0000 7f04f24b2640 1 --2- 172.21.15.29:0/4101459434 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f04d40a2800 0x7f04d40a4cc0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:13.070 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.070+0000 7f04f24b2640 1 --2- 172.21.15.29:0/4101459434 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f04ec1022a0 0x7f04ec1a12c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:13.070 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.070+0000 7f04f24b2640 1 --2- 172.21.15.29:0/4101459434 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f04ec1018e0 0x7f04ec19a290 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:13.070 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.070+0000 7f04f24b2640 1 --2- 172.21.15.29:0/4101459434 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f04ec1006f0 0x7f04ec199d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:13.070 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.070+0000 7f04f24b2640 1 -- 172.21.15.29:0/4101459434 >> 172.21.15.29:0/4101459434 conn(0x7f04ec0fbc80 msgr2=0x7f04ec0fd750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:50:13.070 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.070+0000 7f04f24b2640 1 -- 172.21.15.29:0/4101459434 shutdown_connections 2024-03-16T11:50:13.070 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.070+0000 7f04f24b2640 1 -- 172.21.15.29:0/4101459434 wait complete. 2024-03-16T11:50:13.081 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-16T11:50:13.184 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.184+0000 7f30d25c4640 1 -- 172.21.15.29:0/3609105086 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f30cc104c30 msgr2=0x7f30cc107020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:13.184 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.184+0000 7f30d25c4640 1 --2- 172.21.15.29:0/3609105086 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f30cc104c30 0x7f30cc107020 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f30bc009900 tx=0x7f30bc02bc30 comp rx=0 tx=0).stop 2024-03-16T11:50:13.184 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.184+0000 7f30d25c4640 1 -- 172.21.15.29:0/3609105086 shutdown_connections 2024-03-16T11:50:13.184 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.184+0000 7f30d25c4640 1 --2- 172.21.15.29:0/3609105086 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f30cc107560 0x7f30cc1099b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:13.184 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.184+0000 7f30d25c4640 1 --2- 172.21.15.29:0/3609105086 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f30cc104c30 0x7f30cc107020 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:13.184 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.184+0000 7f30d25c4640 1 --2- 172.21.15.29:0/3609105086 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f30cc102300 0x7f30cc1046f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:13.184 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.184+0000 7f30d25c4640 1 -- 172.21.15.29:0/3609105086 >> 172.21.15.29:0/3609105086 conn(0x7f30cc0fbd10 msgr2=0x7f30cc0fe130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:50:13.184 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.184+0000 7f30d25c4640 1 -- 172.21.15.29:0/3609105086 shutdown_connections 2024-03-16T11:50:13.184 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.184+0000 7f30d25c4640 1 -- 172.21.15.29:0/3609105086 wait complete. 2024-03-16T11:50:13.184 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.185+0000 7f30d25c4640 1 Processor -- start 2024-03-16T11:50:13.184 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.185+0000 7f30d25c4640 1 -- start start 2024-03-16T11:50:13.186 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.185+0000 7f30d25c4640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f30cc102300 0x7f30cc197b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:13.186 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.185+0000 7f30d25c4640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f30cc104c30 0x7f30cc198070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:13.186 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.185+0000 7f30d25c4640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f30cc107560 0x7f30cc19f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:13.186 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.185+0000 7f30d25c4640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f30cc10d480 con 0x7f30cc107560 2024-03-16T11:50:13.186 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.185+0000 7f30d25c4640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f30cc10d300 con 0x7f30cc104c30 2024-03-16T11:50:13.186 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.185+0000 7f30d25c4640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f30cc10d600 con 0x7f30cc102300 2024-03-16T11:50:13.186 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.185+0000 7f30d0b61640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f30cc107560 0x7f30cc19f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:13.186 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.185+0000 7f30d0b61640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f30cc107560 0x7f30cc19f0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47370/0 (socket says 172.21.15.29:47370) 2024-03-16T11:50:13.187 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.185+0000 7f30d0b61640 1 -- 172.21.15.29:0/2787263113 learned_addr learned my addr 172.21.15.29:0/2787263113 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:50:13.187 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.185+0000 7f30cbfff640 1 --2- 172.21.15.29:0/2787263113 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f30cc102300 0x7f30cc197b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:13.187 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.185+0000 7f30cb7fe640 1 --2- 172.21.15.29:0/2787263113 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f30cc104c30 0x7f30cc198070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:13.187 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.185+0000 7f30d0b61640 1 -- 172.21.15.29:0/2787263113 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f30cc102300 msgr2=0x7f30cc197b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:13.187 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.185+0000 7f30d0b61640 1 --2- 172.21.15.29:0/2787263113 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f30cc102300 0x7f30cc197b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:13.187 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.185+0000 7f30d0b61640 1 -- 172.21.15.29:0/2787263113 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f30cc104c30 msgr2=0x7f30cc198070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:13.187 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.185+0000 7f30d0b61640 1 --2- 172.21.15.29:0/2787263113 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f30cc104c30 0x7f30cc198070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:13.187 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.185+0000 7f30d0b61640 1 -- 172.21.15.29:0/2787263113 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30cc19f750 con 0x7f30cc107560 2024-03-16T11:50:13.187 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.186+0000 7f30d0b61640 1 --2- 172.21.15.29:0/2787263113 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f30cc107560 0x7f30cc19f0e0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f30c400eb10 tx=0x7f30c400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:50:13.187 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.186+0000 7f30c97fa640 1 -- 172.21.15.29:0/2787263113 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30c4004b20 con 0x7f30cc107560 2024-03-16T11:50:13.187 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.186+0000 7f30d25c4640 1 -- 172.21.15.29:0/2787263113 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f30cc19fa40 con 0x7f30cc107560 2024-03-16T11:50:13.187 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.186+0000 7f30c97fa640 1 -- 172.21.15.29:0/2787263113 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f30c4004cc0 con 0x7f30cc107560 2024-03-16T11:50:13.187 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.186+0000 7f30c97fa640 1 -- 172.21.15.29:0/2787263113 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30c4010760 con 0x7f30cc107560 2024-03-16T11:50:13.188 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.186+0000 7f30d25c4640 1 -- 172.21.15.29:0/2787263113 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f30cc19ff10 con 0x7f30cc107560 2024-03-16T11:50:13.188 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.187+0000 7f30d25c4640 1 -- 172.21.15.29:0/2787263113 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f30cc0fd800 con 0x7f30cc107560 2024-03-16T11:50:13.188 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.188+0000 7f30c97fa640 1 -- 172.21.15.29:0/2787263113 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f30c4005000 con 0x7f30cc107560 2024-03-16T11:50:13.189 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.189+0000 7f30c97fa640 1 --2- 172.21.15.29:0/2787263113 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f30b80a2800 0x7f30b80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:13.189 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.189+0000 7f30cbfff640 1 --2- 172.21.15.29:0/2787263113 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f30b80a2800 0x7f30b80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:13.189 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.189+0000 7f30c97fa640 1 -- 172.21.15.29:0/2787263113 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8211+0+0 (secure 0 0 0) 0x7f30c40cf1f0 con 0x7f30cc107560 2024-03-16T11:50:13.189 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.189+0000 7f30cbfff640 1 --2- 172.21.15.29:0/2787263113 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f30b80a2800 0x7f30b80a4cc0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f30b40095f0 tx=0x7f30b4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:50:13.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.193+0000 7f30c97fa640 1 -- 172.21.15.29:0/2787263113 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f30c4053740 con 0x7f30cc107560 2024-03-16T11:50:13.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:50:13 smithi029 ceph-mon[31975]: from='client.? 172.21.15.29:0/3188069105' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-16T11:50:13.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:50:13 smithi029 ceph-mon[31975]: from='client.? 172.21.15.29:0/1802217765' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-16T11:50:13.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:50:13 smithi029 ceph-mon[31975]: from='client.? 172.21.15.29:0/4101459434' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T11:50:13.377 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.378+0000 7f30d25c4640 1 -- 172.21.15.29:0/2787263113 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f30cc0623e0 con 0x7f30cc107560 2024-03-16T11:50:13.379 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.380+0000 7f30c97fa640 1 -- 172.21.15.29:0/2787263113 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) ==== 74+0+22668 (secure 0 0 0) 0x7f30c409c2a0 con 0x7f30cc107560 2024-03-16T11:50:13.379 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-16T11:50:13.379 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":87,"fsid":"d1887f28-e389-11ee-95c9-87774f69a715","created":"2024-03-16T11:40:16.475063+0000","modified":"2024-03-16T11:49:28.628389+0000","last_up_change":"2024-03-16T11:47:44.349348+0000","last_in_change":"2024-03-16T11:47:23.672432+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-16T11:43:51.892652+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-16T11:48:36.709732+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-16T11:48:37.586140+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"60f5608f-a408-45ba-bdf6-5b53758f88fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":2748947913},{"type":"v1","addr":"172.21.15.29:6803","nonce":2748947913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":2748947913},{"type":"v1","addr":"172.21.15.29:6805","nonce":2748947913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":2748947913},{"type":"v1","addr":"172.21.15.29:6809","nonce":2748947913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":2748947913},{"type":"v1","addr":"172.21.15.29:6807","nonce":2748947913}]},"public_addr":"172.21.15.29:6803/2748947913","cluster_addr":"172.21.15.29:6805/2748947913","heartbeat_back_addr":"172.21.15.29:6809/2748947913","heartbeat_front_addr":"172.21.15.29:6807/2748947913","state":["exists","up"]},{"osd":1,"uuid":"e70b53d6-ee78-4e56-9a2c-263ae0501b43","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6800","nonce":2339408422},{"type":"v1","addr":"172.21.15.152:6801","nonce":2339408422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6802","nonce":2339408422},{"type":"v1","addr":"172.21.15.152:6803","nonce":2339408422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6806","nonce":2339408422},{"type":"v1","addr":"172.21.15.152:6807","nonce":2339408422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6804","nonce":2339408422},{"type":"v1","addr":"172.21.15.152:6805","nonce":2339408422}]},"public_addr":"172.21.15.152:6801/2339408422","cluster_addr":"172.21.15.152:6803/2339408422","heartbeat_back_addr":"172.21.15.152:6807/2339408422","heartbeat_front_addr":"172.21.15.152:6805/2339408422","state":["exists","up"]},{"osd":2,"uuid":"ff39389a-dfac-44b5-80ff-36abca316e78","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6800","nonce":4288325280},{"type":"v1","addr":"172.21.15.154:6801","nonce":4288325280}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6802","nonce":4288325280},{"type":"v1","addr":"172.21.15.154:6803","nonce":4288325280}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6806","nonce":4288325280},{"type":"v1","addr":"172.21.15.154:6807","nonce":4288325280}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":4288325280},{"type":"v1","addr":"172.21.15.154:6805","nonce":4288325280}]},"public_addr":"172.21.15.154:6801/4288325280","cluster_addr":"172.21.15.154:6803/4288325280","heartbeat_back_addr":"172.21.15.154:6807/4288325280","heartbeat_front_addr":"172.21.15.154:6805/4288325280","state":["exists","up"]},{"osd":3,"uuid":"069e1795-55bc-4324-b94c-eb24ef2334c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":1538750397},{"type":"v1","addr":"172.21.15.29:6811","nonce":1538750397}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":1538750397},{"type":"v1","addr":"172.21.15.29:6813","nonce":1538750397}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":1538750397},{"type":"v1","addr":"172.21.15.29:6817","nonce":1538750397}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":1538750397},{"type":"v1","addr":"172.21.15.29:6815","nonce":1538750397}]},"public_addr":"172.21.15.29:6811/1538750397","cluster_addr":"172.21.15.29:6813/1538750397","heartbeat_back_addr":"172.21.15.29:6817/1538750397","heartbeat_front_addr":"172.21.15.29:6815/1538750397","state":["exists","up"]},{"osd":4,"uuid":"5a9d43a1-e8f5-4ab5-a441-afbc6ed4066e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6808","nonce":2795081750},{"type":"v1","addr":"172.21.15.152:6809","nonce":2795081750}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6810","nonce":2795081750},{"type":"v1","addr":"172.21.15.152:6811","nonce":2795081750}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6814","nonce":2795081750},{"type":"v1","addr":"172.21.15.152:6815","nonce":2795081750}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6812","nonce":2795081750},{"type":"v1","addr":"172.21.15.152:6813","nonce":2795081750}]},"public_addr":"172.21.15.152:6809/2795081750","cluster_addr":"172.21.15.152:6811/2795081750","heartbeat_back_addr":"172.21.15.152:6815/2795081750","heartbeat_front_addr":"172.21.15.152:6813/2795081750","state":["exists","up"]},{"osd":5,"uuid":"234de428-4ac7-44be-9a54-7b58331fdf8c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":1472686560},{"type":"v1","addr":"172.21.15.154:6809","nonce":1472686560}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6810","nonce":1472686560},{"type":"v1","addr":"172.21.15.154:6811","nonce":1472686560}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6814","nonce":1472686560},{"type":"v1","addr":"172.21.15.154:6815","nonce":1472686560}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6812","nonce":1472686560},{"type":"v1","addr":"172.21.15.154:6813","nonce":1472686560}]},"public_addr":"172.21.15.154:6809/1472686560","cluster_addr":"172.21.15.154:6811/1472686560","heartbeat_back_addr":"172.21.15.154:6815/1472686560","heartbeat_front_addr":"172.21.15.154:6813/1472686560","state":["exists","up"]},{"osd":6,"uuid":"ae45b8d9-7883-40c9-a9cd-4c92b601d4f2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":816506244},{"type":"v1","addr":"172.21.15.29:6819","nonce":816506244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":816506244},{"type":"v1","addr":"172.21.15.29:6821","nonce":816506244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":816506244},{"type":"v1","addr":"172.21.15.29:6825","nonce":816506244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":816506244},{"type":"v1","addr":"172.21.15.29:6823","nonce":816506244}]},"public_addr":"172.21.15.29:6819/816506244","cluster_addr":"172.21.15.29:6821/816506244","heartbeat_back_addr":"172.21.15.29:6825/816506244","heartbeat_front_addr":"172.21.15.29:6823/816506244","state":["exists","up"]},{"osd":7,"uuid":"6f687b28-4fce-4d4a-8b4e-294173d9d6c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6816","nonce":2924651787},{"type":"v1","addr":"172.21.15.152:6817","nonce":2924651787}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6818","nonce":2924651787},{"type":"v1","addr":"172.21.15.152:6819","nonce":2924651787}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6822","nonce":2924651787},{"type":"v1","addr":"172.21.15.152:6823","nonce":2924651787}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6820","nonce":2924651787},{"type":"v1","addr":"172.21.15.152:6821","nonce":2924651787}]},"public_addr":"172.21.15.152:6817/2924651787","cluster_addr":"172.21.15.152:6819/2924651787","heartbeat_back_addr":"172.21.15.152:6823/2924651787","heartbeat_front_addr":"172.21.15.152:6821/2924651787","state":["exists","up"]},{"osd":8,"uuid":"a25fb741-caa7-429a-b540-e394d034b446","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6816","nonce":129974156},{"type":"v1","addr":"172.21.15.154:6817","nonce":129974156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6818","nonce":129974156},{"type":"v1","addr":"172.21.15.154:6819","nonce":129974156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6822","nonce":129974156},{"type":"v1","addr":"172.21.15.154:6823","nonce":129974156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6820","nonce":129974156},{"type":"v1","addr":"172.21.15.154:6821","nonce":129974156}]},"public_addr":"172.21.15.154:6817/129974156","cluster_addr":"172.21.15.154:6819/129974156","heartbeat_back_addr":"172.21.15.154:6823/129974156","heartbeat_front_addr":"172.21.15.154:6821/129974156","state":["exists","up"]},{"osd":9,"uuid":"1071cb0e-8c5f-4cd8-9805-0045e220d946","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6826","nonce":2951414243},{"type":"v1","addr":"172.21.15.29:6827","nonce":2951414243}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":2951414243},{"type":"v1","addr":"172.21.15.29:6829","nonce":2951414243}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":2951414243},{"type":"v1","addr":"172.21.15.29:6833","nonce":2951414243}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":2951414243},{"type":"v1","addr":"172.21.15.29:6831","nonce":2951414243}]},"public_addr":"172.21.15.29:6827/2951414243","cluster_addr":"172.21.15.29:6829/2951414243","heartbeat_back_addr":"172.21.15.29:6833/2951414243","heartbeat_front_addr":"172.21.15.29:6831/2951414243","state":["exists","up"]},{"osd":10,"uuid":"ee7c7d74-3f39-4cd4-be80-843fb1dd9ecc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6824","nonce":3412525480},{"type":"v1","addr":"172.21.15.152:6825","nonce":3412525480}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6826","nonce":3412525480},{"type":"v1","addr":"172.21.15.152:6827","nonce":3412525480}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6830","nonce":3412525480},{"type":"v1","addr":"172.21.15.152:6831","nonce":3412525480}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6828","nonce":3412525480},{"type":"v1","addr":"172.21.15.152:6829","nonce":3412525480}]},"public_addr":"172.21.15.152:6825/3412525480","cluster_addr":"172.21.15.152:6827/3412525480","heartbeat_back_addr":"172.21.15.152:6831/3412525480","heartbeat_front_addr":"172.21.15.152:6829/3412525480","state":["exists","up"]},{"osd":11,"uuid":"4598c803-984a-4de6-ac22-066502c17889","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6824","nonce":1160927966},{"type":"v1","addr":"172.21.15.154:6825","nonce":1160927966}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6826","nonce":1160927966},{"type":"v1","addr":"172.21.15.154:6827","nonce":1160927966}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6830","nonce":1160927966},{"type":"v1","addr":"172.21.15.154:6831","nonce":1160927966}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6828","nonce":1160927966},{"type":"v1","addr":"172.21.15.154:6829","nonce":1160927966}]},"public_addr":"172.21.15.154:6825/1160927966","cluster_addr":"172.21.15.154:6827/1160927966","heartbeat_back_addr":"172.21.15.154:6831/1160927966","heartbeat_front_addr":"172.21.15.154:6829/1160927966","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:43:02.343393+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:43:25.209151+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:43:48.946164+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:44:14.588424+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:44:39.700025+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:45:04.061197+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:45:31.214508+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:45:56.937296+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:46:22.768915+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:46:50.101011+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:47:16.298041+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:47:42.402095+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"2.1b","mappings":[{"from":2,"to":1}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"3.33","mappings":[{"from":5,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.29:0/1853086886":"2024-03-17T11:41:27.531724+0000","172.21.15.29:0/1211659364":"2024-03-17T11:41:27.531724+0000","172.21.15.29:6800/2798831128":"2024-03-17T11:41:27.531724+0000","172.21.15.29:0/1281028285":"2024-03-17T11:41:00.505678+0000","172.21.15.29:6800/2275315654":"2024-03-17T11:41:00.505678+0000","172.21.15.29:6801/2798831128":"2024-03-17T11:41:27.531724+0000","172.21.15.29:6801/2275315654":"2024-03-17T11:41:00.505678+0000","172.21.15.29:0/1330100009":"2024-03-17T11:41:00.505678+0000","172.21.15.29:0/1784249039":"2024-03-17T11:41:00.505678+0000","172.21.15.29:0/1116486932":"2024-03-17T11:41:27.531724+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-16T11:50:13.381 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.382+0000 7f30d25c4640 1 -- 172.21.15.29:0/2787263113 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f30b80a2800 msgr2=0x7f30b80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:13.381 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.382+0000 7f30d25c4640 1 --2- 172.21.15.29:0/2787263113 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f30b80a2800 0x7f30b80a4cc0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f30b40095f0 tx=0x7f30b4005f90 comp rx=0 tx=0).stop 2024-03-16T11:50:13.381 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.382+0000 7f30d25c4640 1 -- 172.21.15.29:0/2787263113 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f30cc107560 msgr2=0x7f30cc19f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:13.381 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.382+0000 7f30d25c4640 1 --2- 172.21.15.29:0/2787263113 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f30cc107560 0x7f30cc19f0e0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f30c400eb10 tx=0x7f30c400c2d0 comp rx=0 tx=0).stop 2024-03-16T11:50:13.381 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.382+0000 7f30d25c4640 1 -- 172.21.15.29:0/2787263113 shutdown_connections 2024-03-16T11:50:13.382 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.382+0000 7f30d25c4640 1 --2- 172.21.15.29:0/2787263113 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f30b80a2800 0x7f30b80a4cc0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:13.382 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.382+0000 7f30d25c4640 1 --2- 172.21.15.29:0/2787263113 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f30cc107560 0x7f30cc19f0e0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:13.382 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.382+0000 7f30d25c4640 1 --2- 172.21.15.29:0/2787263113 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f30cc104c30 0x7f30cc198070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:13.382 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.382+0000 7f30d25c4640 1 --2- 172.21.15.29:0/2787263113 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f30cc102300 0x7f30cc197b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:13.382 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.382+0000 7f30d25c4640 1 -- 172.21.15.29:0/2787263113 >> 172.21.15.29:0/2787263113 conn(0x7f30cc0fbd10 msgr2=0x7f30cc102ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:50:13.382 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.382+0000 7f30d25c4640 1 -- 172.21.15.29:0/2787263113 shutdown_connections 2024-03-16T11:50:13.382 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.382+0000 7f30d25c4640 1 -- 172.21.15.29:0/2787263113 wait complete. 2024-03-16T11:50:13.393 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T11:50:13.498 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.496+0000 7f67c2c36640 1 -- 172.21.15.29:0/936918802 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f67bc100780 msgr2=0x7f67bc100ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:13.498 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.496+0000 7f67c2c36640 1 --2- 172.21.15.29:0/936918802 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f67bc100780 0x7f67bc100ba0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f67b0009900 tx=0x7f67b002bc30 comp rx=0 tx=0).stop 2024-03-16T11:50:13.498 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.496+0000 7f67c2c36640 1 -- 172.21.15.29:0/936918802 shutdown_connections 2024-03-16T11:50:13.498 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.496+0000 7f67c2c36640 1 --2- 172.21.15.29:0/936918802 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f67bc102330 0x7f67bc108bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:13.498 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.496+0000 7f67c2c36640 1 --2- 172.21.15.29:0/936918802 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f67bc101970 0x7f67bc101df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:13.498 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.496+0000 7f67c2c36640 1 --2- 172.21.15.29:0/936918802 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f67bc100780 0x7f67bc100ba0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:13.498 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.496+0000 7f67c2c36640 1 -- 172.21.15.29:0/936918802 >> 172.21.15.29:0/936918802 conn(0x7f67bc0fbd10 msgr2=0x7f67bc0fe130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:50:13.498 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.497+0000 7f67c2c36640 1 -- 172.21.15.29:0/936918802 shutdown_connections 2024-03-16T11:50:13.498 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.497+0000 7f67c2c36640 1 -- 172.21.15.29:0/936918802 wait complete. 2024-03-16T11:50:13.498 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.497+0000 7f67c2c36640 1 Processor -- start 2024-03-16T11:50:13.499 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.497+0000 7f67c2c36640 1 -- start start 2024-03-16T11:50:13.499 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.497+0000 7f67c2c36640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f67bc100780 0x7f67bc196a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:13.499 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.497+0000 7f67c2c36640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f67bc101970 0x7f67bc196f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:13.499 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.497+0000 7f67c2c36640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f67bc102330 0x7f67bc19dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:13.499 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.497+0000 7f67c2c36640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f67bc10c380 con 0x7f67bc101970 2024-03-16T11:50:13.499 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.497+0000 7f67c2c36640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f67bc10c200 con 0x7f67bc102330 2024-03-16T11:50:13.499 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.497+0000 7f67c2c36640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f67bc10c500 con 0x7f67bc100780 2024-03-16T11:50:13.499 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.497+0000 7f67bbfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f67bc101970 0x7f67bc196f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:13.499 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.497+0000 7f67bbfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f67bc101970 0x7f67bc196f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47386/0 (socket says 172.21.15.29:47386) 2024-03-16T11:50:13.499 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.497+0000 7f67bbfff640 1 -- 172.21.15.29:0/2075665980 learned_addr learned my addr 172.21.15.29:0/2075665980 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:50:13.499 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.497+0000 7f67c09d2640 1 --2- 172.21.15.29:0/2075665980 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f67bc100780 0x7f67bc196a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:13.499 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.497+0000 7f67c11d3640 1 --2- 172.21.15.29:0/2075665980 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f67bc102330 0x7f67bc19dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:13.499 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.498+0000 7f67bbfff640 1 -- 172.21.15.29:0/2075665980 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f67bc100780 msgr2=0x7f67bc196a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:13.499 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.498+0000 7f67bbfff640 1 --2- 172.21.15.29:0/2075665980 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f67bc100780 0x7f67bc196a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:13.499 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.498+0000 7f67bbfff640 1 -- 172.21.15.29:0/2075665980 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f67bc102330 msgr2=0x7f67bc19dfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:13.500 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.498+0000 7f67bbfff640 1 --2- 172.21.15.29:0/2075665980 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f67bc102330 0x7f67bc19dfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:13.500 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.498+0000 7f67bbfff640 1 -- 172.21.15.29:0/2075665980 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67bc19e650 con 0x7f67bc101970 2024-03-16T11:50:13.500 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.498+0000 7f67bbfff640 1 --2- 172.21.15.29:0/2075665980 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f67bc101970 0x7f67bc196f70 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f67a800e9f0 tx=0x7f67a800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:50:13.500 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.498+0000 7f67b9ffb640 1 -- 172.21.15.29:0/2075665980 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67a800cd80 con 0x7f67bc101970 2024-03-16T11:50:13.500 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.498+0000 7f67b9ffb640 1 -- 172.21.15.29:0/2075665980 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f67a8005500 con 0x7f67bc101970 2024-03-16T11:50:13.500 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.498+0000 7f67c2c36640 1 -- 172.21.15.29:0/2075665980 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f67bc19e940 con 0x7f67bc101970 2024-03-16T11:50:13.500 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.498+0000 7f67b9ffb640 1 -- 172.21.15.29:0/2075665980 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67a8010690 con 0x7f67bc101970 2024-03-16T11:50:13.500 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.498+0000 7f67c2c36640 1 -- 172.21.15.29:0/2075665980 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f67bc19ee40 con 0x7f67bc101970 2024-03-16T11:50:13.501 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.499+0000 7f67c2c36640 1 -- 172.21.15.29:0/2075665980 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f67bc107b10 con 0x7f67bc101970 2024-03-16T11:50:13.501 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.500+0000 7f67b9ffb640 1 -- 172.21.15.29:0/2075665980 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f67a80046c0 con 0x7f67bc101970 2024-03-16T11:50:13.501 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.500+0000 7f67b9ffb640 1 --2- 172.21.15.29:0/2075665980 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f67ac0a2800 0x7f67ac0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:13.501 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.501+0000 7f67b9ffb640 1 -- 172.21.15.29:0/2075665980 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8211+0+0 (secure 0 0 0) 0x7f67a80cee30 con 0x7f67bc101970 2024-03-16T11:50:13.501 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.501+0000 7f67c09d2640 1 --2- 172.21.15.29:0/2075665980 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f67ac0a2800 0x7f67ac0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:13.501 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.501+0000 7f67c09d2640 1 --2- 172.21.15.29:0/2075665980 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f67ac0a2800 0x7f67ac0a4cc0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f67b0000c00 tx=0x7f67b0005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:50:13.504 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.505+0000 7f67b9ffb640 1 -- 172.21.15.29:0/2075665980 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f67a809bee0 con 0x7f67bc101970 2024-03-16T11:50:13.627 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:50:13 smithi152 ceph-mon[36291]: from='client.? 172.21.15.29:0/3188069105' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-16T11:50:13.627 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:50:13 smithi152 ceph-mon[36291]: from='client.? 172.21.15.29:0/1802217765' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-16T11:50:13.627 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:50:13 smithi152 ceph-mon[36291]: from='client.? 172.21.15.29:0/4101459434' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T11:50:13.730 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.731+0000 7f67c2c36640 1 -- 172.21.15.29:0/2075665980 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f67bc02d1b0 con 0x7f67bc101970 2024-03-16T11:50:13.731 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.731+0000 7f67b9ffb640 1 -- 172.21.15.29:0/2075665980 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+11875 (secure 0 0 0) 0x7f67a809b7e0 con 0x7f67bc101970 2024-03-16T11:50:13.731 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-16T11:50:13.731 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14574,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/3505811955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":3505811955},{"type":"v1","addr":"172.21.15.29:6835","nonce":3505811955}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14586,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/1901337868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":1901337868},{"type":"v1","addr":"172.21.15.29:6837","nonce":1901337868}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14652,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6839/544966534","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6838","nonce":544966534},{"type":"v1","addr":"172.21.15.29:6839","nonce":544966534}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14664,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6841/540705290","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6840","nonce":540705290},{"type":"v1","addr":"172.21.15.29:6841","nonce":540705290}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24407,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6833/3136024628","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6832","nonce":3136024628},{"type":"v1","addr":"172.21.15.152:6833","nonce":3136024628}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24421,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6833/2677426282","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":2677426282},{"type":"v1","addr":"172.21.15.154:6833","nonce":2677426282}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":21,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-16T11:48:38.431509+0000","modified":"2024-03-16T11:48:42.969084+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24503,"mds_1":24499,"mds_2":24464},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24434":{"gid":24434,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.152:6835/2919113178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6834","nonce":2919113178},{"type":"v1","addr":"172.21.15.152:6835","nonce":2919113178}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24464":{"gid":24464,"name":"h","rank":2,"incarnation":19,"state":"up:active","state_seq":7,"addr":"172.21.15.152:6837/1323866722","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6836","nonce":1323866722},{"type":"v1","addr":"172.21.15.152:6837","nonce":1323866722}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.154:6835/4165274681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":4165274681},{"type":"v1","addr":"172.21.15.154:6835","nonce":4165274681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.154:6837/1468560096","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":1468560096},{"type":"v1","addr":"172.21.15.154:6837","nonce":1468560096}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":1,"incarnation":17,"state":"up:active","state_seq":6,"addr":"172.21.15.154:6839/685736862","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":685736862},{"type":"v1","addr":"172.21.15.154:6839","nonce":685736862}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.152:6839/1548731833","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6838","nonce":1548731833},{"type":"v1","addr":"172.21.15.152:6839","nonce":1548731833}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24503,"qdb_cluster":[24503,24464,24499]},"id":1}]} 2024-03-16T11:50:13.731 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 21 2024-03-16T11:50:13.733 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.733+0000 7f67c2c36640 1 -- 172.21.15.29:0/2075665980 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f67ac0a2800 msgr2=0x7f67ac0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:13.733 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.733+0000 7f67c2c36640 1 --2- 172.21.15.29:0/2075665980 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f67ac0a2800 0x7f67ac0a4cc0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f67b0000c00 tx=0x7f67b0005fd0 comp rx=0 tx=0).stop 2024-03-16T11:50:13.733 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.733+0000 7f67c2c36640 1 -- 172.21.15.29:0/2075665980 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f67bc101970 msgr2=0x7f67bc196f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:13.733 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.733+0000 7f67c2c36640 1 --2- 172.21.15.29:0/2075665980 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f67bc101970 0x7f67bc196f70 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f67a800e9f0 tx=0x7f67a800edf0 comp rx=0 tx=0).stop 2024-03-16T11:50:13.733 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.734+0000 7f67c2c36640 1 -- 172.21.15.29:0/2075665980 shutdown_connections 2024-03-16T11:50:13.733 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.734+0000 7f67c2c36640 1 --2- 172.21.15.29:0/2075665980 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f67ac0a2800 0x7f67ac0a4cc0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:13.733 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.734+0000 7f67c2c36640 1 --2- 172.21.15.29:0/2075665980 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f67bc102330 0x7f67bc19dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:13.733 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.734+0000 7f67c2c36640 1 --2- 172.21.15.29:0/2075665980 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f67bc101970 0x7f67bc196f70 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:13.733 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.734+0000 7f67c2c36640 1 --2- 172.21.15.29:0/2075665980 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f67bc100780 0x7f67bc196a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:13.733 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.734+0000 7f67c2c36640 1 -- 172.21.15.29:0/2075665980 >> 172.21.15.29:0/2075665980 conn(0x7f67bc0fbd10 msgr2=0x7f67bc104ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:50:13.733 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.734+0000 7f67c2c36640 1 -- 172.21.15.29:0/2075665980 shutdown_connections 2024-03-16T11:50:13.734 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.734+0000 7f67c2c36640 1 -- 172.21.15.29:0/2075665980 wait complete. 2024-03-16T11:50:13.744 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 21, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-03-16T11:48:38.431509+0000', 'modified': '2024-03-16T11:48:42.969084+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 24503, 'mds_1': 24499, 'mds_2': 24464}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24434': {'gid': 24434, 'name': 'e', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.152:6835/2919113178', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.152:6834', 'nonce': 2919113178}, {'type': 'v1', 'addr': '172.21.15.152:6835', 'nonce': 2919113178}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24464': {'gid': 24464, 'name': 'h', 'rank': 2, 'incarnation': 19, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.152:6837/1323866722', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.152:6836', 'nonce': 1323866722}, {'type': 'v1', 'addr': '172.21.15.152:6837', 'nonce': 1323866722}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24466': {'gid': 24466, 'name': 'f', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.154:6835/4165274681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 4165274681}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 4165274681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24478': {'gid': 24478, 'name': 'c', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.154:6837/1468560096', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 1468560096}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 1468560096}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24499': {'gid': 24499, 'name': 'i', 'rank': 1, 'incarnation': 17, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.154:6839/685736862', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 685736862}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 685736862}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24503': {'gid': 24503, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.152:6839/1548731833', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.152:6838', 'nonce': 1548731833}, {'type': 'v1', 'addr': '172.21.15.152:6839', 'nonce': 1548731833}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24503, 'qdb_cluster': [24503, 24464, 24499]} 2024-03-16T11:50:13.744 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 3/3 2024-03-16T11:50:13.744 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.h status 2024-03-16T11:50:13.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:50:13 smithi154 ceph-mon[36320]: from='client.? 172.21.15.29:0/3188069105' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-16T11:50:13.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:50:13 smithi154 ceph-mon[36320]: from='client.? 172.21.15.29:0/1802217765' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-16T11:50:13.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:50:13 smithi154 ceph-mon[36320]: from='client.? 172.21.15.29:0/4101459434' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T11:50:13.849 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.848+0000 7f1fff3c4640 1 -- 172.21.15.29:0/528887125 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1ff81022a0 msgr2=0x7f1ff8108b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:13.850 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.848+0000 7f1fff3c4640 1 --2- 172.21.15.29:0/528887125 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1ff81022a0 0x7f1ff8108b30 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f1fe8009ca0 tx=0x7f1fe802c0f0 comp rx=0 tx=0).stop 2024-03-16T11:50:13.850 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.848+0000 7f1fff3c4640 1 -- 172.21.15.29:0/528887125 shutdown_connections 2024-03-16T11:50:13.850 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.848+0000 7f1fff3c4640 1 --2- 172.21.15.29:0/528887125 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1ff81022a0 0x7f1ff8108b30 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:13.850 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.848+0000 7f1fff3c4640 1 --2- 172.21.15.29:0/528887125 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1ff81018e0 0x7f1ff8101d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:13.850 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.848+0000 7f1fff3c4640 1 --2- 172.21.15.29:0/528887125 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1ff81006f0 0x7f1ff8100b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:13.850 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.848+0000 7f1fff3c4640 1 -- 172.21.15.29:0/528887125 >> 172.21.15.29:0/528887125 conn(0x7f1ff80fbc80 msgr2=0x7f1ff80fe0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:50:13.850 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.848+0000 7f1fff3c4640 1 -- 172.21.15.29:0/528887125 shutdown_connections 2024-03-16T11:50:13.850 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.848+0000 7f1fff3c4640 1 -- 172.21.15.29:0/528887125 wait complete. 2024-03-16T11:50:13.850 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.849+0000 7f1fff3c4640 1 Processor -- start 2024-03-16T11:50:13.850 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.849+0000 7f1fff3c4640 1 -- start start 2024-03-16T11:50:13.850 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.849+0000 7f1fff3c4640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1ff81006f0 0x7f1ff8197ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:13.850 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.849+0000 7f1fff3c4640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1ff81018e0 0x7f1ff8198010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:13.850 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.849+0000 7f1fff3c4640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1ff81022a0 0x7f1ff819f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:13.851 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.849+0000 7f1fff3c4640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f1ff810c370 con 0x7f1ff81018e0 2024-03-16T11:50:13.851 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.849+0000 7f1fff3c4640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f1ff810c1f0 con 0x7f1ff81006f0 2024-03-16T11:50:13.851 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.849+0000 7f1fff3c4640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f1ff810c4f0 con 0x7f1ff81022a0 2024-03-16T11:50:13.851 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.849+0000 7f1ffc95f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1ff81018e0 0x7f1ff8198010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:13.851 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.849+0000 7f1ffc95f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1ff81018e0 0x7f1ff8198010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47412/0 (socket says 172.21.15.29:47412) 2024-03-16T11:50:13.851 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.849+0000 7f1ffc95f640 1 -- 172.21.15.29:0/4105892757 learned_addr learned my addr 172.21.15.29:0/4105892757 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:50:13.851 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.849+0000 7f1ffd160640 1 --2- 172.21.15.29:0/4105892757 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1ff81006f0 0x7f1ff8197ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:13.851 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.849+0000 7f1ffd961640 1 --2- 172.21.15.29:0/4105892757 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1ff81022a0 0x7f1ff819f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:13.851 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.849+0000 7f1ffc95f640 1 -- 172.21.15.29:0/4105892757 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1ff81022a0 msgr2=0x7f1ff819f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:13.851 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.849+0000 7f1ffc95f640 1 --2- 172.21.15.29:0/4105892757 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1ff81022a0 0x7f1ff819f080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:13.851 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.849+0000 7f1ffc95f640 1 -- 172.21.15.29:0/4105892757 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1ff81006f0 msgr2=0x7f1ff8197ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:13.851 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.849+0000 7f1ffc95f640 1 --2- 172.21.15.29:0/4105892757 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1ff81006f0 0x7f1ff8197ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:13.851 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.849+0000 7f1ffc95f640 1 -- 172.21.15.29:0/4105892757 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ff819f6f0 con 0x7f1ff81018e0 2024-03-16T11:50:13.851 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.850+0000 7f1ffc95f640 1 --2- 172.21.15.29:0/4105892757 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1ff81018e0 0x7f1ff8198010 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f1fec00e9f0 tx=0x7f1fec00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:50:13.851 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.850+0000 7f1ff67fc640 1 -- 172.21.15.29:0/4105892757 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fec00cd80 con 0x7f1ff81018e0 2024-03-16T11:50:13.851 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.850+0000 7f1ff67fc640 1 -- 172.21.15.29:0/4105892757 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1fec010910 con 0x7f1ff81018e0 2024-03-16T11:50:13.852 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.850+0000 7f1fff3c4640 1 -- 172.21.15.29:0/4105892757 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1ff819f9e0 con 0x7f1ff81018e0 2024-03-16T11:50:13.852 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.850+0000 7f1ff67fc640 1 -- 172.21.15.29:0/4105892757 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fec018990 con 0x7f1ff81018e0 2024-03-16T11:50:13.852 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.850+0000 7f1fff3c4640 1 -- 172.21.15.29:0/4105892757 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1ff819ff80 con 0x7f1ff81018e0 2024-03-16T11:50:13.858 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.852+0000 7f1ff67fc640 1 -- 172.21.15.29:0/4105892757 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f1fec010ac0 con 0x7f1ff81018e0 2024-03-16T11:50:13.858 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.852+0000 7f2001fa9740 1 Processor -- start 2024-03-16T11:50:13.858 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.852+0000 7f2001fa9740 1 -- start start 2024-03-16T11:50:13.858 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.852+0000 7f2001fa9740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556b94918bd0 0x556b94918ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:13.858 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.852+0000 7f2001fa9740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556b949195c0 0x556b9491e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:13.858 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.852+0000 7f2001fa9740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556b9491e730 0x556b94920bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:13.858 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.852+0000 7f1ff67fc640 1 --2- 172.21.15.29:0/4105892757 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f1fe00a2880 0x7f1fe00a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:13.858 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.852+0000 7f2001fa9740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x556b9487cc80 con 0x556b94918bd0 2024-03-16T11:50:13.858 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.852+0000 7f2001fa9740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x556b94825aa0 con 0x556b949195c0 2024-03-16T11:50:13.858 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.852+0000 7f2001fa9740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x556b94878d20 con 0x556b9491e730 2024-03-16T11:50:13.859 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.853+0000 7f1ffd961640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556b94918bd0 0x556b94918ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:13.859 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.853+0000 7f1ffd961640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556b94918bd0 0x556b94918ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47428/0 (socket says 172.21.15.29:47428) 2024-03-16T11:50:13.859 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.853+0000 7f1ffd961640 1 -- 172.21.15.29:0/1077175800 learned_addr learned my addr 172.21.15.29:0/1077175800 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:50:13.859 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.853+0000 7f1ffc95f640 1 --2- 172.21.15.29:0/1077175800 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556b9491e730 0x556b94920bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:13.859 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.853+0000 7f1ffd160640 1 --2- 172.21.15.29:0/1077175800 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556b949195c0 0x556b9491e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:13.859 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.853+0000 7f1ffd961640 1 --2- 172.21.15.29:0/4105892757 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f1fe00a2880 0x7f1fe00a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:13.859 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.853+0000 7f1ff67fc640 1 -- 172.21.15.29:0/4105892757 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8211+0+0 (secure 0 0 0) 0x7f1fec0ced00 con 0x7f1ff81018e0 2024-03-16T11:50:13.859 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.853+0000 7f1ffd961640 1 -- 172.21.15.29:0/1077175800 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556b9491e730 msgr2=0x556b94920bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:50:13.859 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.853+0000 7f1ffd961640 1 --2- 172.21.15.29:0/1077175800 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556b9491e730 0x556b94920bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:13.859 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.853+0000 7f1ffd961640 1 -- 172.21.15.29:0/1077175800 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556b949195c0 msgr2=0x556b9491e0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:50:13.859 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.853+0000 7f1ffd961640 1 --2- 172.21.15.29:0/1077175800 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556b949195c0 0x556b9491e0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:13.859 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.853+0000 7f1ffd961640 1 -- 172.21.15.29:0/1077175800 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556b94847ff0 con 0x556b94918bd0 2024-03-16T11:50:13.859 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.853+0000 7f1ffd160640 1 --2- 172.21.15.29:0/1077175800 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556b949195c0 0x556b9491e0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:50:13.859 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.853+0000 7f1ffd961640 1 --2- 172.21.15.29:0/1077175800 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556b94918bd0 0x556b94918ff0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f1fe8032420 tx=0x7f1fe8032f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89d4fed7d5feda4 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:50:13.859 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.853+0000 7f1ffd961640 1 --2- 172.21.15.29:0/4105892757 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f1fe00a2880 0x7f1fe00a4d40 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f1fe8009430 tx=0x7f1fe8042000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:50:13.860 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.853+0000 7f1fdf7fe640 1 -- 172.21.15.29:0/1077175800 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fe8051070 con 0x556b94918bd0 2024-03-16T11:50:13.860 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.853+0000 7f1fdf7fe640 1 -- 172.21.15.29:0/1077175800 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1fe8004b40 con 0x556b94918bd0 2024-03-16T11:50:13.860 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.853+0000 7f1fdf7fe640 1 -- 172.21.15.29:0/1077175800 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fe8039040 con 0x556b94918bd0 2024-03-16T11:50:13.860 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.854+0000 7f2001fa9740 1 -- 172.21.15.29:0/1077175800 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556b94918bd0 msgr2=0x556b94918ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:13.860 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.854+0000 7f2001fa9740 1 --2- 172.21.15.29:0/1077175800 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556b94918bd0 0x556b94918ff0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f1fe8032420 tx=0x7f1fe8032f60 comp rx=0 tx=0).stop 2024-03-16T11:50:13.860 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.854+0000 7f2001fa9740 1 -- 172.21.15.29:0/1077175800 shutdown_connections 2024-03-16T11:50:13.860 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.854+0000 7f2001fa9740 1 --2- 172.21.15.29:0/1077175800 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556b9491e730 0x556b94920bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:13.860 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.854+0000 7f2001fa9740 1 --2- 172.21.15.29:0/1077175800 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556b949195c0 0x556b9491e0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:13.860 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.854+0000 7f2001fa9740 1 --2- 172.21.15.29:0/1077175800 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556b94918bd0 0x556b94918ff0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:13.860 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.854+0000 7f2001fa9740 1 -- 172.21.15.29:0/1077175800 >> 172.21.15.29:0/1077175800 conn(0x556b94914140 msgr2=0x556b94914540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:50:13.860 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.854+0000 7f2001fa9740 1 -- 172.21.15.29:0/1077175800 shutdown_connections 2024-03-16T11:50:13.860 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.854+0000 7f2001fa9740 1 -- 172.21.15.29:0/1077175800 wait complete. 2024-03-16T11:50:13.860 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.854+0000 7f2001fa9740 1 -- start start 2024-03-16T11:50:13.860 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.855+0000 7f2001fa9740 1 Processor -- start 2024-03-16T11:50:13.860 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.855+0000 7f2001fa9740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556b94918bd0 0x556b94922df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:13.861 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.855+0000 7f2001fa9740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556b949195c0 0x556b94923330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:13.861 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.855+0000 7f2001fa9740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556b9491e730 0x556b9492a370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:13.861 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.855+0000 7f2001fa9740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x556b94921680 con 0x556b949195c0 2024-03-16T11:50:13.861 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.855+0000 7f2001fa9740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x556b94838ef0 con 0x556b94918bd0 2024-03-16T11:50:13.861 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.855+0000 7f2001fa9740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x556b94921800 con 0x556b9491e730 2024-03-16T11:50:13.861 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.855+0000 7f2001fa9740 10 client.0 _ll_register_callbacks cb 0x556b94833dc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T11:50:13.861 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.855+0000 7f1ffc95f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556b949195c0 0x556b94923330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:13.861 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.855+0000 7f1ffc95f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556b949195c0 0x556b94923330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47444/0 (socket says 172.21.15.29:47444) 2024-03-16T11:50:13.861 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.855+0000 7f1ffc95f640 1 -- 172.21.15.29:0/1806502261 learned_addr learned my addr 172.21.15.29:0/1806502261 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:50:13.861 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.855+0000 7f1ffd961640 1 --2- 172.21.15.29:0/1806502261 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556b9491e730 0x556b9492a370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:13.861 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.855+0000 7f1ffd160640 1 --2- 172.21.15.29:0/1806502261 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556b94918bd0 0x556b94922df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:13.861 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.855+0000 7f1ffc95f640 1 -- 172.21.15.29:0/1806502261 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556b9491e730 msgr2=0x556b9492a370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:50:13.861 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.855+0000 7f1ffc95f640 1 --2- 172.21.15.29:0/1806502261 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556b9491e730 0x556b9492a370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:13.861 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.855+0000 7f1ffc95f640 1 -- 172.21.15.29:0/1806502261 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556b94918bd0 msgr2=0x556b94922df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:50:13.861 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.855+0000 7f1ffc95f640 1 --2- 172.21.15.29:0/1806502261 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556b94918bd0 0x556b94922df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:13.861 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.855+0000 7f1ffc95f640 1 -- 172.21.15.29:0/1806502261 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556b94845a30 con 0x556b949195c0 2024-03-16T11:50:13.862 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.855+0000 7f1ffc95f640 1 --2- 172.21.15.29:0/1806502261 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556b949195c0 0x556b94923330 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f1fec00cfd0 tx=0x7f1fec014820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4c8738b55148f74 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:50:13.862 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.855+0000 7f1fdcff9640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-16T11:50:13.862 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.856+0000 7f1fdcff9640 1 -- 172.21.15.29:0/1806502261 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fec026040 con 0x556b949195c0 2024-03-16T11:50:13.862 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.856+0000 7f1fdcff9640 1 -- 172.21.15.29:0/1806502261 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8211+0+0 (secure 0 0 0) 0x7f1fec01cd30 con 0x556b949195c0 2024-03-16T11:50:13.862 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.856+0000 7f2001fa9740 1 -- 172.21.15.29:0/1806502261 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1fe8004b40 con 0x556b949195c0 2024-03-16T11:50:13.862 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.856+0000 7f1fdcff9640 1 -- 172.21.15.29:0/1806502261 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f1fec09f800 con 0x556b949195c0 2024-03-16T11:50:13.862 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.856+0000 7f2001fa9740 10 client.15078 fetch_fsmap learned FSMap version 21 2024-03-16T11:50:13.862 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.856+0000 7f2001fa9740 1 -- 172.21.15.29:0/1806502261 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x556b945912d0 con 0x556b949195c0 2024-03-16T11:50:13.862 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.856+0000 7f1fdcff9640 1 -- 172.21.15.29:0/1806502261 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1fe8004b40 con 0x556b949195c0 2024-03-16T11:50:13.862 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.856+0000 7f1fdcff9640 1 -- 172.21.15.29:0/1806502261 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f1fec09abb0 con 0x556b949195c0 2024-03-16T11:50:13.862 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.856+0000 7f2001fa9740 10 client.15078 fetch_fsmap finished waiting for FSMap version 21 2024-03-16T11:50:13.862 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.856+0000 7f2001fa9740 10 client.15078 resolve_mds: resolved name 'h' to daemon mds.h 2024-03-16T11:50:13.862 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.856+0000 7f2001fa9740 20 client.15078 populate_metadata read hostname 'smithi029' 2024-03-16T11:50:13.862 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.856+0000 7f2001fa9740 1 --2- 172.21.15.29:0/1806502261 >> [v2:172.21.15.152:6836/1323866722,v1:172.21.15.152:6837/1323866722] conn(0x556b9492f850 0x556b94930cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:13.862 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.856+0000 7f2001fa9740 4 client.15078 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T11:50:13.862 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.856+0000 7f2001fa9740 1 -- 172.21.15.29:0/1806502261 --> [v2:172.21.15.152:6836/1323866722,v1:172.21.15.152:6837/1323866722] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556b9488a1c0 con 0x556b9492f850 2024-03-16T11:50:13.863 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.856+0000 7f1ffd160640 1 --2- 172.21.15.29:0/1806502261 >> [v2:172.21.15.152:6836/1323866722,v1:172.21.15.152:6837/1323866722] conn(0x556b9492f850 0x556b94930cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:13.863 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.857+0000 7f1fdcff9640 10 client.15078 ms_handle_connect on v2:172.21.15.152:6836/1323866722 2024-03-16T11:50:13.863 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.857+0000 7f1ffd160640 1 --2- 172.21.15.29:0/1806502261 >> [v2:172.21.15.152:6836/1323866722,v1:172.21.15.152:6837/1323866722] conn(0x556b9492f850 0x556b94930cc0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=7a3b6f35bf188d3c server_cookie=75edbef3bf71cfbe in_seq=0 out_seq=0 2024-03-16T11:50:13.863 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.857+0000 7f1fdcff9640 1 -- 172.21.15.29:0/1806502261 <== mds.2 v2:172.21.15.152:6836/1323866722 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x556b9488a1c0 con 0x556b9492f850 2024-03-16T11:50:13.863 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.857+0000 7f1fdcff9640 10 client.15078 handle_command_reply: tid=0 2024-03-16T11:50:13.863 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.857+0000 7f2001fa9740 1 client.15078 shutdown 2024-03-16T11:50:13.863 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.857+0000 7f2001fa9740 1 -- 172.21.15.29:0/1806502261 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556b949195c0 msgr2=0x556b94923330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:13.863 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.857+0000 7f2001fa9740 1 --2- 172.21.15.29:0/1806502261 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556b949195c0 0x556b94923330 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f1fec00cfd0 tx=0x7f1fec014820 comp rx=0 tx=0).stop 2024-03-16T11:50:13.863 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.858+0000 7f2001fa9740 1 -- 172.21.15.29:0/1806502261 shutdown_connections 2024-03-16T11:50:13.863 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.858+0000 7f2001fa9740 1 --2- 172.21.15.29:0/1806502261 >> [v2:172.21.15.152:6836/1323866722,v1:172.21.15.152:6837/1323866722] conn(0x556b9492f850 0x556b94930cc0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:13.863 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.858+0000 7f2001fa9740 1 --2- 172.21.15.29:0/1806502261 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556b9491e730 0x556b9492a370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:13.863 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.858+0000 7f2001fa9740 1 --2- 172.21.15.29:0/1806502261 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556b949195c0 0x556b94923330 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:13.863 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.858+0000 7f1fdcff9640 0 client.15078 ms_handle_reset on v2:172.21.15.152:6836/1323866722 2024-03-16T11:50:13.863 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.858+0000 7f2001fa9740 1 --2- 172.21.15.29:0/1806502261 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556b94918bd0 0x556b94922df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:13.864 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.858+0000 7f2001fa9740 1 -- 172.21.15.29:0/1806502261 >> 172.21.15.29:0/1806502261 conn(0x556b94914140 msgr2=0x556b94913380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:50:13.864 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.858+0000 7f2001fa9740 1 -- 172.21.15.29:0/1806502261 shutdown_connections 2024-03-16T11:50:13.864 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.858+0000 7f2001fa9740 1 -- 172.21.15.29:0/1806502261 wait complete. 2024-03-16T11:50:13.864 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.858+0000 7f2001fa9740 20 client.15078 trim_cache size 0 max 16384 2024-03-16T11:50:13.877 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.876+0000 7f2001fa9740 1 Processor -- start 2024-03-16T11:50:13.877 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.876+0000 7f2001fa9740 1 -- start start 2024-03-16T11:50:13.877 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.876+0000 7f2001fa9740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556b94914140 0x556b94923110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:13.877 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.876+0000 7f2001fa9740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556b94918bd0 0x556b9492a330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:13.877 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.876+0000 7f2001fa9740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556b949195c0 0x556b94912af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:13.877 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.876+0000 7f2001fa9740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x556b9492f000 con 0x556b94914140 2024-03-16T11:50:13.877 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.876+0000 7f2001fa9740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x556b9492f6d0 con 0x556b949195c0 2024-03-16T11:50:13.877 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.876+0000 7f2001fa9740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x556b9492fda0 con 0x556b94918bd0 2024-03-16T11:50:13.877 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.876+0000 7f1ffd160640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556b94914140 0x556b94923110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:13.877 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.876+0000 7f1ffd160640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556b94914140 0x556b94923110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47448/0 (socket says 172.21.15.29:47448) 2024-03-16T11:50:13.877 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.876+0000 7f1ffd160640 1 -- 172.21.15.29:0/1363396944 learned_addr learned my addr 172.21.15.29:0/1363396944 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:50:13.878 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.876+0000 7f1ffc95f640 1 --2- 172.21.15.29:0/1363396944 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556b94918bd0 0x556b9492a330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:13.878 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.876+0000 7f1ffd961640 1 --2- 172.21.15.29:0/1363396944 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556b949195c0 0x556b94912af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:13.878 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.876+0000 7f1ffd160640 1 -- 172.21.15.29:0/1363396944 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556b94918bd0 msgr2=0x556b9492a330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:50:13.878 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.876+0000 7f1ffd160640 1 --2- 172.21.15.29:0/1363396944 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556b94918bd0 0x556b9492a330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:13.878 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.876+0000 7f1ffd160640 1 -- 172.21.15.29:0/1363396944 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556b949195c0 msgr2=0x556b94912af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:50:13.878 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.876+0000 7f1ffd160640 1 --2- 172.21.15.29:0/1363396944 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556b949195c0 0x556b94912af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:13.878 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.876+0000 7f1ffd160640 1 -- 172.21.15.29:0/1363396944 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556b9465b6b0 con 0x556b94914140 2024-03-16T11:50:13.878 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.877+0000 7f1ffd160640 1 --2- 172.21.15.29:0/1363396944 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556b94914140 0x556b94923110 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x556b9491a030 tx=0x7f1ff0005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf3b8d284075588e server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:50:13.878 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.877+0000 7f1fdeffd640 1 -- 172.21.15.29:0/1363396944 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ff0025070 con 0x556b94914140 2024-03-16T11:50:13.878 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.877+0000 7f1fdeffd640 1 -- 172.21.15.29:0/1363396944 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1ff00205d0 con 0x556b94914140 2024-03-16T11:50:13.878 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.877+0000 7f1fdeffd640 1 -- 172.21.15.29:0/1363396944 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ff0010bf0 con 0x556b94914140 2024-03-16T11:50:13.878 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.877+0000 7f2001fa9740 1 -- 172.21.15.29:0/1363396944 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556b94914140 msgr2=0x556b94923110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:13.878 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.877+0000 7f2001fa9740 1 --2- 172.21.15.29:0/1363396944 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556b94914140 0x556b94923110 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x556b9491a030 tx=0x7f1ff0005f90 comp rx=0 tx=0).stop 2024-03-16T11:50:13.878 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.878+0000 7f2001fa9740 1 -- 172.21.15.29:0/1363396944 shutdown_connections 2024-03-16T11:50:13.879 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.878+0000 7f2001fa9740 1 --2- 172.21.15.29:0/1363396944 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556b949195c0 0x556b94912af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:13.879 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.878+0000 7f2001fa9740 1 --2- 172.21.15.29:0/1363396944 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556b94918bd0 0x556b9492a330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:13.879 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.878+0000 7f2001fa9740 1 --2- 172.21.15.29:0/1363396944 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556b94914140 0x556b94923110 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:13.879 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.878+0000 7f2001fa9740 1 -- 172.21.15.29:0/1363396944 >> 172.21.15.29:0/1363396944 conn(0x556b9491e160 msgr2=0x556b94922a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:50:13.879 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.878+0000 7f2001fa9740 1 -- 172.21.15.29:0/1363396944 shutdown_connections 2024-03-16T11:50:13.879 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.878+0000 7f2001fa9740 1 -- 172.21.15.29:0/1363396944 wait complete. 2024-03-16T11:50:13.879 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.878+0000 7f2001fa9740 1 -- start start 2024-03-16T11:50:13.879 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.878+0000 7f2001fa9740 1 Processor -- start 2024-03-16T11:50:13.883 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.878+0000 7f2001fa9740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556b94914140 0x556b94926560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:13.883 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.878+0000 7f2001fa9740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556b94918bd0 0x556b94926aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:13.883 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.878+0000 7f2001fa9740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556b949195c0 0x556b94929460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:13.883 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.878+0000 7f2001fa9740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x556b94930b40 con 0x556b94914140 2024-03-16T11:50:13.883 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.878+0000 7f2001fa9740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x556b94930470 con 0x556b949195c0 2024-03-16T11:50:13.883 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.878+0000 7f2001fa9740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x556b94931210 con 0x556b94918bd0 2024-03-16T11:50:13.883 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.878+0000 7f2001fa9740 10 client.0 _ll_register_callbacks cb 0x7f1fec097370 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T11:50:13.883 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.878+0000 7f1ffd160640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556b94914140 0x556b94926560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:13.883 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.878+0000 7f1ffd160640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556b94914140 0x556b94926560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47460/0 (socket says 172.21.15.29:47460) 2024-03-16T11:50:13.883 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.878+0000 7f1ffd160640 1 -- 172.21.15.29:0/1898435059 learned_addr learned my addr 172.21.15.29:0/1898435059 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:50:13.883 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.878+0000 7f1ffc95f640 1 --2- 172.21.15.29:0/1898435059 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556b94918bd0 0x556b94926aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:13.883 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.878+0000 7f1ffd961640 1 --2- 172.21.15.29:0/1898435059 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556b949195c0 0x556b94929460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:13.884 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.879+0000 7f1ffd160640 1 -- 172.21.15.29:0/1898435059 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556b94918bd0 msgr2=0x556b94926aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:50:13.884 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.879+0000 7f1ffd160640 1 --2- 172.21.15.29:0/1898435059 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556b94918bd0 0x556b94926aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:13.884 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.879+0000 7f1ffd160640 1 -- 172.21.15.29:0/1898435059 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556b949195c0 msgr2=0x556b94929460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:50:13.884 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.879+0000 7f1ffd160640 1 --2- 172.21.15.29:0/1898435059 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556b949195c0 0x556b94929460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:13.884 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.879+0000 7f1ffd160640 1 -- 172.21.15.29:0/1898435059 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556b94929aa0 con 0x556b94914140 2024-03-16T11:50:13.884 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.879+0000 7f1ffc95f640 1 --2- 172.21.15.29:0/1898435059 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556b94918bd0 0x556b94926aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:50:13.884 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.879+0000 7f1ffd160640 1 --2- 172.21.15.29:0/1898435059 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556b94914140 0x556b94926560 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f1ff0004650 tx=0x7f1ff0009c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3a87445a234f8a8 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:50:13.884 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.879+0000 7f1fdcff9640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-16T11:50:13.884 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.879+0000 7f1fdcff9640 1 -- 172.21.15.29:0/1898435059 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ff0031060 con 0x556b94914140 2024-03-16T11:50:13.884 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.879+0000 7f1fdcff9640 1 -- 172.21.15.29:0/1898435059 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8211+0+0 (secure 0 0 0) 0x7f1ff00200f0 con 0x556b94914140 2024-03-16T11:50:13.884 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.879+0000 7f2001fa9740 1 -- 172.21.15.29:0/1898435059 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1ff00205d0 con 0x556b94914140 2024-03-16T11:50:13.884 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.879+0000 7f1fdcff9640 1 -- 172.21.15.29:0/1898435059 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f1ff0025070 con 0x556b94914140 2024-03-16T11:50:13.884 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.879+0000 7f2001fa9740 10 client.15084 fetch_fsmap learned FSMap version 21 2024-03-16T11:50:13.884 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.879+0000 7f2001fa9740 1 -- 172.21.15.29:0/1898435059 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x556b94995a60 con 0x556b94914140 2024-03-16T11:50:13.884 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.879+0000 7f1fdcff9640 1 -- 172.21.15.29:0/1898435059 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1ff00205d0 con 0x556b94914140 2024-03-16T11:50:13.885 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.879+0000 7f1fdcff9640 1 -- 172.21.15.29:0/1898435059 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f1ff00046c0 con 0x556b94914140 2024-03-16T11:50:13.885 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.879+0000 7f2001fa9740 10 client.15084 fetch_fsmap finished waiting for FSMap version 21 2024-03-16T11:50:13.885 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.880+0000 7f2001fa9740 10 client.15084 resolve_mds: resolved name 'h' to daemon mds.h 2024-03-16T11:50:13.885 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.880+0000 7f2001fa9740 20 client.15084 populate_metadata read hostname 'smithi029' 2024-03-16T11:50:13.885 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.880+0000 7f2001fa9740 1 --2- 172.21.15.29:0/1898435059 >> [v2:172.21.15.152:6836/1323866722,v1:172.21.15.152:6837/1323866722] conn(0x556b9491e730 0x556b94999020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:13.885 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.880+0000 7f2001fa9740 4 client.15084 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-16T11:50:13.885 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.880+0000 7f2001fa9740 1 -- 172.21.15.29:0/1898435059 --> [v2:172.21.15.152:6836/1323866722,v1:172.21.15.152:6837/1323866722] -- command(tid 0: {"prefix": "status"}) -- 0x556b94884510 con 0x556b9491e730 2024-03-16T11:50:13.885 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.880+0000 7f1ffc95f640 1 --2- 172.21.15.29:0/1898435059 >> [v2:172.21.15.152:6836/1323866722,v1:172.21.15.152:6837/1323866722] conn(0x556b9491e730 0x556b94999020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:13.885 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.880+0000 7f1fdcff9640 10 client.15084 ms_handle_connect on v2:172.21.15.152:6836/1323866722 2024-03-16T11:50:13.885 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.880+0000 7f1ffc95f640 1 --2- 172.21.15.29:0/1898435059 >> [v2:172.21.15.152:6836/1323866722,v1:172.21.15.152:6837/1323866722] conn(0x556b9491e730 0x556b94999020 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=c6cb776394ebc9aa server_cookie=5c3fca324ee9af52 in_seq=0 out_seq=0 2024-03-16T11:50:13.885 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.880+0000 7f1fdcff9640 1 -- 172.21.15.29:0/1898435059 <== mds.2 v2:172.21.15.152:6836/1323866722 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x556b94884510 con 0x556b9491e730 2024-03-16T11:50:13.885 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.880+0000 7f1fdcff9640 10 client.15084 handle_command_reply: tid=0 2024-03-16T11:50:13.885 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.880+0000 7f2001fa9740 1 client.15084 shutdown 2024-03-16T11:50:13.885 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.880+0000 7f2001fa9740 1 -- 172.21.15.29:0/1898435059 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556b94914140 msgr2=0x556b94926560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:13.886 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.880+0000 7f2001fa9740 1 --2- 172.21.15.29:0/1898435059 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556b94914140 0x556b94926560 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f1ff0004650 tx=0x7f1ff0009c20 comp rx=0 tx=0).stop 2024-03-16T11:50:13.886 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.880+0000 7f2001fa9740 1 -- 172.21.15.29:0/1898435059 shutdown_connections 2024-03-16T11:50:13.886 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.880+0000 7f2001fa9740 1 --2- 172.21.15.29:0/1898435059 >> [v2:172.21.15.152:6836/1323866722,v1:172.21.15.152:6837/1323866722] conn(0x556b9491e730 0x556b94999020 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:13.886 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.880+0000 7f2001fa9740 1 --2- 172.21.15.29:0/1898435059 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556b949195c0 0x556b94929460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:13.886 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.880+0000 7f1fdcff9640 0 client.15084 ms_handle_reset on v2:172.21.15.152:6836/1323866722 2024-03-16T11:50:13.886 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.880+0000 7f2001fa9740 1 --2- 172.21.15.29:0/1898435059 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556b94918bd0 0x556b94926aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:13.886 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.880+0000 7f2001fa9740 1 --2- 172.21.15.29:0/1898435059 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556b94914140 0x556b94926560 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:13.886 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.880+0000 7f2001fa9740 1 -- 172.21.15.29:0/1898435059 >> 172.21.15.29:0/1898435059 conn(0x556b9491e160 msgr2=0x556b94922a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:50:13.886 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.881+0000 7f2001fa9740 1 -- 172.21.15.29:0/1898435059 shutdown_connections 2024-03-16T11:50:13.886 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.881+0000 7f2001fa9740 1 -- 172.21.15.29:0/1898435059 wait complete. 2024-03-16T11:50:13.886 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.881+0000 7f2001fa9740 20 client.15084 trim_cache size 0 max 16384 2024-03-16T11:50:13.887 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-03-16T11:50:13.887 INFO:teuthology.orchestra.run.smithi029.stdout: "cluster_fsid": "d1887f28-e389-11ee-95c9-87774f69a715", 2024-03-16T11:50:13.887 INFO:teuthology.orchestra.run.smithi029.stdout: "whoami": 2, 2024-03-16T11:50:13.887 INFO:teuthology.orchestra.run.smithi029.stdout: "id": 24464, 2024-03-16T11:50:13.887 INFO:teuthology.orchestra.run.smithi029.stdout: "want_state": "up:active", 2024-03-16T11:50:13.887 INFO:teuthology.orchestra.run.smithi029.stdout: "state": "up:active", 2024-03-16T11:50:13.887 INFO:teuthology.orchestra.run.smithi029.stdout: "fs_name": "cephfs", 2024-03-16T11:50:13.887 INFO:teuthology.orchestra.run.smithi029.stdout: "rank_uptime": 91.916689305000006, 2024-03-16T11:50:13.887 INFO:teuthology.orchestra.run.smithi029.stdout: "mdsmap_epoch": 21, 2024-03-16T11:50:13.887 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_epoch": 87, 2024-03-16T11:50:13.887 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_epoch_barrier": 85, 2024-03-16T11:50:13.887 INFO:teuthology.orchestra.run.smithi029.stdout: "uptime": 112.063374583 2024-03-16T11:50:13.887 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-03-16T11:50:13.888 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.881+0000 7f1fff3c4640 1 -- 172.21.15.29:0/4105892757 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f1fe00a2880 msgr2=0x7f1fe00a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:13.888 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.881+0000 7f1fff3c4640 1 --2- 172.21.15.29:0/4105892757 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f1fe00a2880 0x7f1fe00a4d40 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f1fe8009430 tx=0x7f1fe8042000 comp rx=0 tx=0).stop 2024-03-16T11:50:13.888 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.881+0000 7f1fff3c4640 1 -- 172.21.15.29:0/4105892757 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1ff81018e0 msgr2=0x7f1ff8198010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:13.888 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.881+0000 7f1fff3c4640 1 --2- 172.21.15.29:0/4105892757 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1ff81018e0 0x7f1ff8198010 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f1fec00e9f0 tx=0x7f1fec00eeb0 comp rx=0 tx=0).stop 2024-03-16T11:50:13.888 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.881+0000 7f1fff3c4640 1 -- 172.21.15.29:0/4105892757 shutdown_connections 2024-03-16T11:50:13.888 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.881+0000 7f1fff3c4640 1 --2- 172.21.15.29:0/4105892757 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f1fe00a2880 0x7f1fe00a4d40 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:13.889 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.881+0000 7f1fff3c4640 1 --2- 172.21.15.29:0/4105892757 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1ff81022a0 0x7f1ff819f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:13.889 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.881+0000 7f1fff3c4640 1 --2- 172.21.15.29:0/4105892757 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1ff81018e0 0x7f1ff8198010 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:13.889 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.881+0000 7f1fff3c4640 1 --2- 172.21.15.29:0/4105892757 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1ff81006f0 0x7f1ff8197ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:13.889 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.881+0000 7f1fff3c4640 1 -- 172.21.15.29:0/4105892757 >> 172.21.15.29:0/4105892757 conn(0x7f1ff80fbc80 msgr2=0x7f1ff80fd6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:50:13.889 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.881+0000 7f1fff3c4640 1 -- 172.21.15.29:0/4105892757 shutdown_connections 2024-03-16T11:50:13.889 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.881+0000 7f1fff3c4640 1 -- 172.21.15.29:0/4105892757 wait complete. 2024-03-16T11:50:13.891 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.i status 2024-03-16T11:50:13.994 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.995+0000 7f744909a640 1 -- 172.21.15.29:0/4077558278 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f74441006e0 msgr2=0x7f7444100b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:13.994 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.995+0000 7f744909a640 1 --2- 172.21.15.29:0/4077558278 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f74441006e0 0x7f7444100b00 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f7434009860 tx=0x7f743402bb70 comp rx=0 tx=0).stop 2024-03-16T11:50:13.995 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.995+0000 7f744909a640 1 -- 172.21.15.29:0/4077558278 shutdown_connections 2024-03-16T11:50:13.995 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.995+0000 7f744909a640 1 --2- 172.21.15.29:0/4077558278 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7444102290 0x7f7444108b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:13.995 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.995+0000 7f744909a640 1 --2- 172.21.15.29:0/4077558278 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f74441018d0 0x7f7444101d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:13.995 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.995+0000 7f744909a640 1 --2- 172.21.15.29:0/4077558278 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f74441006e0 0x7f7444100b00 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:13.995 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.995+0000 7f744909a640 1 -- 172.21.15.29:0/4077558278 >> 172.21.15.29:0/4077558278 conn(0x7f74440fbc70 msgr2=0x7f74440fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:50:13.995 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.995+0000 7f744909a640 1 -- 172.21.15.29:0/4077558278 shutdown_connections 2024-03-16T11:50:13.995 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.995+0000 7f744909a640 1 -- 172.21.15.29:0/4077558278 wait complete. 2024-03-16T11:50:13.997 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.996+0000 7f744909a640 1 Processor -- start 2024-03-16T11:50:13.997 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.996+0000 7f744909a640 1 -- start start 2024-03-16T11:50:13.997 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.996+0000 7f744909a640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f74441006e0 0x7f74440694d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:13.997 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.996+0000 7f744909a640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f74441018d0 0x7f744419c880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:13.997 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.996+0000 7f744909a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7444102290 0x7f744419eec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:13.997 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.996+0000 7f744909a640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f744410c350 con 0x7f7444102290 2024-03-16T11:50:13.997 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.996+0000 7f744909a640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f744410c1d0 con 0x7f74441006e0 2024-03-16T11:50:13.997 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.996+0000 7f744909a640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f744410c4d0 con 0x7f74441018d0 2024-03-16T11:50:13.997 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.996+0000 7f744359e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7444102290 0x7f744419eec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:13.997 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.996+0000 7f744359e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7444102290 0x7f744419eec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47476/0 (socket says 172.21.15.29:47476) 2024-03-16T11:50:13.998 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.996+0000 7f744359e640 1 -- 172.21.15.29:0/322979187 learned_addr learned my addr 172.21.15.29:0/322979187 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:50:13.998 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.996+0000 7f7442d9d640 1 --2- 172.21.15.29:0/322979187 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f74441006e0 0x7f74440694d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:13.998 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.997+0000 7f744259c640 1 --2- 172.21.15.29:0/322979187 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f74441018d0 0x7f744419c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:13.998 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.997+0000 7f744359e640 1 -- 172.21.15.29:0/322979187 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f74441018d0 msgr2=0x7f744419c880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:13.998 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.997+0000 7f744359e640 1 --2- 172.21.15.29:0/322979187 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f74441018d0 0x7f744419c880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:13.998 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.997+0000 7f744359e640 1 -- 172.21.15.29:0/322979187 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f74441006e0 msgr2=0x7f74440694d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:13.998 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.997+0000 7f744359e640 1 --2- 172.21.15.29:0/322979187 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f74441006e0 0x7f74440694d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:13.998 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.997+0000 7f744359e640 1 -- 172.21.15.29:0/322979187 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f744419f780 con 0x7f7444102290 2024-03-16T11:50:13.998 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.997+0000 7f744259c640 1 --2- 172.21.15.29:0/322979187 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f74441018d0 0x7f744419c880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:50:13.998 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.997+0000 7f7442d9d640 1 --2- 172.21.15.29:0/322979187 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f74441006e0 0x7f74440694d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:50:13.998 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.997+0000 7f744359e640 1 --2- 172.21.15.29:0/322979187 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7444102290 0x7f744419eec0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f743c009fd0 tx=0x7f743c00c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:50:13.998 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.997+0000 7f742bfff640 1 -- 172.21.15.29:0/322979187 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f743c004b20 con 0x7f7444102290 2024-03-16T11:50:13.998 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.997+0000 7f744909a640 1 -- 172.21.15.29:0/322979187 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f744419fa70 con 0x7f7444102290 2024-03-16T11:50:13.998 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.997+0000 7f742bfff640 1 -- 172.21.15.29:0/322979187 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f743c003e50 con 0x7f7444102290 2024-03-16T11:50:13.999 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.997+0000 7f742bfff640 1 -- 172.21.15.29:0/322979187 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f743c00b630 con 0x7f7444102290 2024-03-16T11:50:13.999 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:13.997+0000 7f744909a640 1 -- 172.21.15.29:0/322979187 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f744419ffc0 con 0x7f7444102290 2024-03-16T11:50:14.005 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.000+0000 7f742bfff640 1 -- 172.21.15.29:0/322979187 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f743c01d020 con 0x7f7444102290 2024-03-16T11:50:14.005 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.000+0000 7f744bd65740 1 Processor -- start 2024-03-16T11:50:14.005 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.000+0000 7f744bd65740 1 -- start start 2024-03-16T11:50:14.005 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.000+0000 7f744bd65740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56315a88fbd0 0x56315a88fff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:14.005 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.000+0000 7f744bd65740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56315a8905c0 0x56315a8950c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:14.005 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.000+0000 7f744bd65740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56315a895730 0x56315a897bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:14.005 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.000+0000 7f744bd65740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x56315a7f3c80 con 0x56315a895730 2024-03-16T11:50:14.005 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.000+0000 7f744bd65740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x56315a79caa0 con 0x56315a88fbd0 2024-03-16T11:50:14.005 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.000+0000 7f744bd65740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x56315a7efd20 con 0x56315a8905c0 2024-03-16T11:50:14.006 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.000+0000 7f744259c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56315a895730 0x56315a897bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:14.006 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.000+0000 7f744259c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56315a895730 0x56315a897bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47478/0 (socket says 172.21.15.29:47478) 2024-03-16T11:50:14.006 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.000+0000 7f742bfff640 1 --2- 172.21.15.29:0/322979187 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f742c0a2800 0x7f742c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:14.006 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.000+0000 7f744259c640 1 -- 172.21.15.29:0/508299350 learned_addr learned my addr 172.21.15.29:0/508299350 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:50:14.006 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.000+0000 7f7442d9d640 1 --2- 172.21.15.29:0/508299350 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56315a8905c0 0x56315a8950c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:14.006 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.000+0000 7f744259c640 1 --2- 172.21.15.29:0/508299350 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56315a88fbd0 0x56315a88fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:14.006 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.001+0000 7f744359e640 1 --2- 172.21.15.29:0/322979187 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f742c0a2800 0x7f742c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:14.006 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.001+0000 7f744259c640 1 -- 172.21.15.29:0/508299350 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56315a8905c0 msgr2=0x56315a8950c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:50:14.006 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.001+0000 7f744259c640 1 --2- 172.21.15.29:0/508299350 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56315a8905c0 0x56315a8950c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:14.006 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.001+0000 7f744259c640 1 -- 172.21.15.29:0/508299350 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56315a88fbd0 msgr2=0x56315a88fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:50:14.006 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.001+0000 7f744259c640 1 --2- 172.21.15.29:0/508299350 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56315a88fbd0 0x56315a88fff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:14.006 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.001+0000 7f744259c640 1 -- 172.21.15.29:0/508299350 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56315a7beff0 con 0x56315a895730 2024-03-16T11:50:14.006 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.001+0000 7f742bfff640 1 -- 172.21.15.29:0/322979187 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8211+0+0 (secure 0 0 0) 0x7f743c0ce9e0 con 0x7f7444102290 2024-03-16T11:50:14.006 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.001+0000 7f744259c640 1 --2- 172.21.15.29:0/508299350 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56315a895730 0x56315a897bc0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f7438009fd0 tx=0x7f7438009b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ddd10f06192a95d server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:50:14.007 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.001+0000 7f744359e640 1 --2- 172.21.15.29:0/322979187 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f742c0a2800 0x7f742c0a4cc0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f743c007c90 tx=0x7f743c06d010 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:50:14.007 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.001+0000 7f74297fa640 1 -- 172.21.15.29:0/508299350 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f743800fc20 con 0x56315a895730 2024-03-16T11:50:14.007 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.001+0000 7f74297fa640 1 -- 172.21.15.29:0/508299350 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f743800fdc0 con 0x56315a895730 2024-03-16T11:50:14.007 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.001+0000 7f74297fa640 1 -- 172.21.15.29:0/508299350 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f743800e6a0 con 0x56315a895730 2024-03-16T11:50:14.007 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.002+0000 7f744bd65740 1 -- 172.21.15.29:0/508299350 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56315a895730 msgr2=0x56315a897bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:14.007 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.002+0000 7f744bd65740 1 --2- 172.21.15.29:0/508299350 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56315a895730 0x56315a897bc0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f7438009fd0 tx=0x7f7438009b80 comp rx=0 tx=0).stop 2024-03-16T11:50:14.007 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.002+0000 7f744bd65740 1 -- 172.21.15.29:0/508299350 shutdown_connections 2024-03-16T11:50:14.007 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.002+0000 7f744bd65740 1 --2- 172.21.15.29:0/508299350 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56315a895730 0x56315a897bc0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:14.007 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.002+0000 7f744bd65740 1 --2- 172.21.15.29:0/508299350 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56315a8905c0 0x56315a8950c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:14.007 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.002+0000 7f744bd65740 1 --2- 172.21.15.29:0/508299350 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56315a88fbd0 0x56315a88fff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:14.007 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.002+0000 7f744bd65740 1 -- 172.21.15.29:0/508299350 >> 172.21.15.29:0/508299350 conn(0x56315a88b140 msgr2=0x56315a88b540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:50:14.007 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.002+0000 7f744bd65740 1 -- 172.21.15.29:0/508299350 shutdown_connections 2024-03-16T11:50:14.007 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.002+0000 7f744bd65740 1 -- 172.21.15.29:0/508299350 wait complete. 2024-03-16T11:50:14.007 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.002+0000 7f744bd65740 1 -- start start 2024-03-16T11:50:14.007 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.002+0000 7f744bd65740 1 Processor -- start 2024-03-16T11:50:14.007 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.002+0000 7f744bd65740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56315a88fbd0 0x56315a899db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:14.008 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.002+0000 7f744bd65740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56315a8905c0 0x56315a89a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:14.008 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.002+0000 7f744bd65740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56315a895730 0x56315a8a1330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:14.008 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.002+0000 7f744bd65740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x56315a8986e0 con 0x56315a895730 2024-03-16T11:50:14.008 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.002+0000 7f744bd65740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x56315a7afef0 con 0x56315a88fbd0 2024-03-16T11:50:14.008 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.002+0000 7f744bd65740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x56315a898860 con 0x56315a8905c0 2024-03-16T11:50:14.008 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.002+0000 7f744bd65740 10 client.0 _ll_register_callbacks cb 0x56315a7aadc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T11:50:14.008 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.003+0000 7f744259c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56315a895730 0x56315a8a1330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:14.008 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.003+0000 7f744259c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56315a895730 0x56315a8a1330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47492/0 (socket says 172.21.15.29:47492) 2024-03-16T11:50:14.008 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.003+0000 7f744259c640 1 -- 172.21.15.29:0/752154760 learned_addr learned my addr 172.21.15.29:0/752154760 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:50:14.008 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.003+0000 7f7442d9d640 1 --2- 172.21.15.29:0/752154760 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56315a8905c0 0x56315a89a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:14.008 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.003+0000 7f744259c640 1 --2- 172.21.15.29:0/752154760 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56315a88fbd0 0x56315a899db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:14.008 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.003+0000 7f744259c640 1 -- 172.21.15.29:0/752154760 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56315a8905c0 msgr2=0x56315a89a2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:50:14.008 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.003+0000 7f744259c640 1 --2- 172.21.15.29:0/752154760 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56315a8905c0 0x56315a89a2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:14.008 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.003+0000 7f744259c640 1 -- 172.21.15.29:0/752154760 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56315a88fbd0 msgr2=0x56315a899db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:50:14.008 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.003+0000 7f744259c640 1 --2- 172.21.15.29:0/752154760 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56315a88fbd0 0x56315a899db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:14.009 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.003+0000 7f744259c640 1 -- 172.21.15.29:0/752154760 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56315a7bca30 con 0x56315a895730 2024-03-16T11:50:14.009 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.003+0000 7f744259c640 1 --2- 172.21.15.29:0/752154760 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56315a895730 0x56315a8a1330 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f7438005600 tx=0x7f74380096a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec5befacde3c9397 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:50:14.009 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.003+0000 7f7426ff5640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-16T11:50:14.009 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.003+0000 7f7426ff5640 1 -- 172.21.15.29:0/752154760 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7438004050 con 0x56315a895730 2024-03-16T11:50:14.009 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.003+0000 7f7426ff5640 1 -- 172.21.15.29:0/752154760 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8211+0+0 (secure 0 0 0) 0x7f7438028040 con 0x56315a895730 2024-03-16T11:50:14.009 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.003+0000 7f744bd65740 1 -- 172.21.15.29:0/752154760 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f743800fdc0 con 0x56315a895730 2024-03-16T11:50:14.009 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.003+0000 7f7426ff5640 1 -- 172.21.15.29:0/752154760 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f74380041f0 con 0x56315a895730 2024-03-16T11:50:14.009 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.003+0000 7f744bd65740 10 client.15096 fetch_fsmap learned FSMap version 21 2024-03-16T11:50:14.009 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.003+0000 7f744bd65740 1 -- 172.21.15.29:0/752154760 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x56315a5082d0 con 0x56315a895730 2024-03-16T11:50:14.009 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.003+0000 7f7426ff5640 1 -- 172.21.15.29:0/752154760 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f743800fdc0 con 0x56315a895730 2024-03-16T11:50:14.009 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.004+0000 7f7426ff5640 1 -- 172.21.15.29:0/752154760 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f743802b020 con 0x56315a895730 2024-03-16T11:50:14.009 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.004+0000 7f744bd65740 10 client.15096 fetch_fsmap finished waiting for FSMap version 21 2024-03-16T11:50:14.009 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.004+0000 7f744bd65740 10 client.15096 resolve_mds: resolved name 'i' to daemon mds.i 2024-03-16T11:50:14.009 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.004+0000 7f744bd65740 20 client.15096 populate_metadata read hostname 'smithi029' 2024-03-16T11:50:14.009 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.004+0000 7f744bd65740 1 --2- 172.21.15.29:0/752154760 >> [v2:172.21.15.154:6838/685736862,v1:172.21.15.154:6839/685736862] conn(0x56315a8a6490 0x56315a8a7900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:14.009 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.004+0000 7f744bd65740 4 client.15096 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T11:50:14.010 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.004+0000 7f744bd65740 1 -- 172.21.15.29:0/752154760 --> [v2:172.21.15.154:6838/685736862,v1:172.21.15.154:6839/685736862] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56315a8011c0 con 0x56315a8a6490 2024-03-16T11:50:14.010 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.004+0000 7f7442d9d640 1 --2- 172.21.15.29:0/752154760 >> [v2:172.21.15.154:6838/685736862,v1:172.21.15.154:6839/685736862] conn(0x56315a8a6490 0x56315a8a7900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:14.010 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.004+0000 7f7442d9d640 1 --2- 172.21.15.29:0/752154760 >> [v2:172.21.15.154:6838/685736862,v1:172.21.15.154:6839/685736862] conn(0x56315a8a6490 0x56315a8a7900 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=a81790a1a5d52ccb server_cookie=ca786c9426627ef4 in_seq=0 out_seq=0 2024-03-16T11:50:14.010 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.004+0000 7f7426ff5640 10 client.15096 ms_handle_connect on v2:172.21.15.154:6838/685736862 2024-03-16T11:50:14.010 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.005+0000 7f7426ff5640 1 -- 172.21.15.29:0/752154760 <== mds.1 v2:172.21.15.154:6838/685736862 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x56315a8011c0 con 0x56315a8a6490 2024-03-16T11:50:14.010 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.005+0000 7f7426ff5640 10 client.15096 handle_command_reply: tid=0 2024-03-16T11:50:14.010 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.005+0000 7f744bd65740 1 client.15096 shutdown 2024-03-16T11:50:14.010 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.005+0000 7f744bd65740 1 -- 172.21.15.29:0/752154760 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56315a895730 msgr2=0x56315a8a1330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:14.010 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.005+0000 7f744bd65740 1 --2- 172.21.15.29:0/752154760 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56315a895730 0x56315a8a1330 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f7438005600 tx=0x7f74380096a0 comp rx=0 tx=0).stop 2024-03-16T11:50:14.010 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.005+0000 7f744bd65740 1 -- 172.21.15.29:0/752154760 shutdown_connections 2024-03-16T11:50:14.010 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.005+0000 7f744bd65740 1 --2- 172.21.15.29:0/752154760 >> [v2:172.21.15.154:6838/685736862,v1:172.21.15.154:6839/685736862] conn(0x56315a8a6490 0x56315a8a7900 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:14.010 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.005+0000 7f744bd65740 1 --2- 172.21.15.29:0/752154760 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56315a895730 0x56315a8a1330 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:14.010 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.005+0000 7f744bd65740 1 --2- 172.21.15.29:0/752154760 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56315a8905c0 0x56315a89a2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:14.010 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.005+0000 7f744bd65740 1 --2- 172.21.15.29:0/752154760 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56315a88fbd0 0x56315a899db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:14.010 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.005+0000 7f7426ff5640 0 client.15096 ms_handle_reset on v2:172.21.15.154:6838/685736862 2024-03-16T11:50:14.011 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.005+0000 7f744bd65740 1 -- 172.21.15.29:0/752154760 >> 172.21.15.29:0/752154760 conn(0x56315a88b140 msgr2=0x56315a88a380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:50:14.011 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.005+0000 7f744bd65740 1 -- 172.21.15.29:0/752154760 shutdown_connections 2024-03-16T11:50:14.011 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.005+0000 7f744bd65740 1 -- 172.21.15.29:0/752154760 wait complete. 2024-03-16T11:50:14.011 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.005+0000 7f744bd65740 20 client.15096 trim_cache size 0 max 16384 2024-03-16T11:50:14.028 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.024+0000 7f744bd65740 1 Processor -- start 2024-03-16T11:50:14.028 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.024+0000 7f744bd65740 1 -- start start 2024-03-16T11:50:14.028 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.024+0000 7f744bd65740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56315a88fbd0 0x56315a89a250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:14.028 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.024+0000 7f744bd65740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56315a8905c0 0x56315a894dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:14.028 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.024+0000 7f744bd65740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56315a895730 0x56315a88b520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:14.028 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.024+0000 7f744bd65740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x56315a8a5c40 con 0x56315a895730 2024-03-16T11:50:14.028 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.024+0000 7f744bd65740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x56315a8a6310 con 0x56315a88fbd0 2024-03-16T11:50:14.028 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.024+0000 7f744bd65740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x56315a8a69e0 con 0x56315a8905c0 2024-03-16T11:50:14.028 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.024+0000 7f744259c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56315a895730 0x56315a88b520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:14.029 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.024+0000 7f744259c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56315a895730 0x56315a88b520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47496/0 (socket says 172.21.15.29:47496) 2024-03-16T11:50:14.029 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.024+0000 7f744259c640 1 -- 172.21.15.29:0/4278434432 learned_addr learned my addr 172.21.15.29:0/4278434432 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:50:14.029 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.024+0000 7f7442d9d640 1 --2- 172.21.15.29:0/4278434432 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56315a8905c0 0x56315a894dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:14.029 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.024+0000 7f744259c640 1 --2- 172.21.15.29:0/4278434432 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56315a88fbd0 0x56315a89a250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:14.029 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.024+0000 7f744259c640 1 -- 172.21.15.29:0/4278434432 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56315a8905c0 msgr2=0x56315a894dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:50:14.029 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.024+0000 7f744259c640 1 --2- 172.21.15.29:0/4278434432 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56315a8905c0 0x56315a894dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:14.029 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.024+0000 7f744259c640 1 -- 172.21.15.29:0/4278434432 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56315a88fbd0 msgr2=0x56315a89a250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:50:14.029 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.024+0000 7f744259c640 1 --2- 172.21.15.29:0/4278434432 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56315a88fbd0 0x56315a89a250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:14.029 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.024+0000 7f744259c640 1 -- 172.21.15.29:0/4278434432 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56315a5d26b0 con 0x56315a895730 2024-03-16T11:50:14.029 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.024+0000 7f744259c640 1 --2- 172.21.15.29:0/4278434432 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56315a895730 0x56315a88b520 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f7438001eb0 tx=0x7f74380118c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb64394b944d9371 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:50:14.029 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.024+0000 7f7428ff9640 1 -- 172.21.15.29:0/4278434432 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74380114c0 con 0x56315a895730 2024-03-16T11:50:14.029 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.025+0000 7f7428ff9640 1 -- 172.21.15.29:0/4278434432 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7438004a00 con 0x56315a895730 2024-03-16T11:50:14.029 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.025+0000 7f7428ff9640 1 -- 172.21.15.29:0/4278434432 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7438004f60 con 0x56315a895730 2024-03-16T11:50:14.029 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.025+0000 7f744bd65740 1 -- 172.21.15.29:0/4278434432 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56315a895730 msgr2=0x56315a88b520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:14.029 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.025+0000 7f744bd65740 1 --2- 172.21.15.29:0/4278434432 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56315a895730 0x56315a88b520 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f7438001eb0 tx=0x7f74380118c0 comp rx=0 tx=0).stop 2024-03-16T11:50:14.029 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.025+0000 7f744bd65740 1 -- 172.21.15.29:0/4278434432 shutdown_connections 2024-03-16T11:50:14.030 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.025+0000 7f744bd65740 1 --2- 172.21.15.29:0/4278434432 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56315a895730 0x56315a88b520 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:14.030 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.025+0000 7f744bd65740 1 --2- 172.21.15.29:0/4278434432 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56315a8905c0 0x56315a894dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:14.030 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.025+0000 7f744bd65740 1 --2- 172.21.15.29:0/4278434432 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56315a88fbd0 0x56315a89a250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:14.030 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.025+0000 7f744bd65740 1 -- 172.21.15.29:0/4278434432 >> 172.21.15.29:0/4278434432 conn(0x56315a88b140 msgr2=0x56315a8a5470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:50:14.030 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.025+0000 7f744bd65740 1 -- 172.21.15.29:0/4278434432 shutdown_connections 2024-03-16T11:50:14.030 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.025+0000 7f744bd65740 1 -- 172.21.15.29:0/4278434432 wait complete. 2024-03-16T11:50:14.030 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.025+0000 7f744bd65740 1 -- start start 2024-03-16T11:50:14.030 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.026+0000 7f744bd65740 1 Processor -- start 2024-03-16T11:50:14.030 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.026+0000 7f744bd65740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56315a88fbd0 0x56315a8930d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:14.030 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.026+0000 7f744bd65740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56315a8905c0 0x56315a893610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:14.030 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.026+0000 7f744bd65740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56315a895730 0x56315a90e210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:14.030 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.026+0000 7f744bd65740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x56315a8a7780 con 0x56315a895730 2024-03-16T11:50:14.030 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.026+0000 7f744bd65740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x56315a8a70b0 con 0x56315a8905c0 2024-03-16T11:50:14.030 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.026+0000 7f744bd65740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x56315a8a7e50 con 0x56315a88fbd0 2024-03-16T11:50:14.030 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.026+0000 7f744bd65740 10 client.0 _ll_register_callbacks cb 0x7f74380116d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T11:50:14.030 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.026+0000 7f744259c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56315a895730 0x56315a90e210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:14.031 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.026+0000 7f744259c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56315a895730 0x56315a90e210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47502/0 (socket says 172.21.15.29:47502) 2024-03-16T11:50:14.031 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.026+0000 7f744259c640 1 -- 172.21.15.29:0/2464461147 learned_addr learned my addr 172.21.15.29:0/2464461147 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:50:14.031 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.026+0000 7f7442d9d640 1 --2- 172.21.15.29:0/2464461147 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56315a8905c0 0x56315a893610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:14.031 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.026+0000 7f744259c640 1 --2- 172.21.15.29:0/2464461147 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56315a88fbd0 0x56315a8930d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:14.031 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.026+0000 7f744259c640 1 -- 172.21.15.29:0/2464461147 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56315a88fbd0 msgr2=0x56315a8930d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:50:14.031 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.026+0000 7f744259c640 1 --2- 172.21.15.29:0/2464461147 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56315a88fbd0 0x56315a8930d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:14.031 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.026+0000 7f744259c640 1 -- 172.21.15.29:0/2464461147 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56315a8905c0 msgr2=0x56315a893610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:50:14.031 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.026+0000 7f744259c640 1 --2- 172.21.15.29:0/2464461147 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56315a8905c0 0x56315a893610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:14.031 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.026+0000 7f744259c640 1 -- 172.21.15.29:0/2464461147 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56315a90e750 con 0x56315a895730 2024-03-16T11:50:14.031 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.026+0000 7f744259c640 1 --2- 172.21.15.29:0/2464461147 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56315a895730 0x56315a90e210 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f7438005600 tx=0x7f7438005970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=705e44b8b86e92e server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:50:14.031 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.026+0000 7f7426ff5640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-16T11:50:14.031 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.026+0000 7f7426ff5640 1 -- 172.21.15.29:0/2464461147 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7438004f10 con 0x56315a895730 2024-03-16T11:50:14.031 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.026+0000 7f744bd65740 1 -- 172.21.15.29:0/2464461147 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7438004a00 con 0x56315a895730 2024-03-16T11:50:14.031 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.026+0000 7f7426ff5640 1 -- 172.21.15.29:0/2464461147 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8211+0+0 (secure 0 0 0) 0x7f7438038450 con 0x56315a895730 2024-03-16T11:50:14.031 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.027+0000 7f7426ff5640 1 -- 172.21.15.29:0/2464461147 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f7438038650 con 0x56315a895730 2024-03-16T11:50:14.031 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.027+0000 7f744bd65740 10 client.15102 fetch_fsmap learned FSMap version 21 2024-03-16T11:50:14.031 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.027+0000 7f744bd65740 1 -- 172.21.15.29:0/2464461147 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x56315a90f9f0 con 0x56315a895730 2024-03-16T11:50:14.032 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.027+0000 7f7426ff5640 1 -- 172.21.15.29:0/2464461147 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7438004a00 con 0x56315a895730 2024-03-16T11:50:14.032 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.027+0000 7f7426ff5640 1 -- 172.21.15.29:0/2464461147 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f7438045050 con 0x56315a895730 2024-03-16T11:50:14.032 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.027+0000 7f744bd65740 10 client.15102 fetch_fsmap finished waiting for FSMap version 21 2024-03-16T11:50:14.032 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.027+0000 7f744bd65740 10 client.15102 resolve_mds: resolved name 'i' to daemon mds.i 2024-03-16T11:50:14.032 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.027+0000 7f744bd65740 20 client.15102 populate_metadata read hostname 'smithi029' 2024-03-16T11:50:14.032 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.027+0000 7f744bd65740 1 --2- 172.21.15.29:0/2464461147 >> [v2:172.21.15.154:6838/685736862,v1:172.21.15.154:6839/685736862] conn(0x56315a914410 0x56315a913120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:14.032 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.027+0000 7f744bd65740 4 client.15102 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-16T11:50:14.032 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.027+0000 7f744bd65740 1 -- 172.21.15.29:0/2464461147 --> [v2:172.21.15.154:6838/685736862,v1:172.21.15.154:6839/685736862] -- command(tid 0: {"prefix": "status"}) -- 0x56315a7fb510 con 0x56315a914410 2024-03-16T11:50:14.032 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.027+0000 7f7442d9d640 1 --2- 172.21.15.29:0/2464461147 >> [v2:172.21.15.154:6838/685736862,v1:172.21.15.154:6839/685736862] conn(0x56315a914410 0x56315a913120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:14.032 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.028+0000 7f7426ff5640 10 client.15102 ms_handle_connect on v2:172.21.15.154:6838/685736862 2024-03-16T11:50:14.032 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.028+0000 7f7442d9d640 1 --2- 172.21.15.29:0/2464461147 >> [v2:172.21.15.154:6838/685736862,v1:172.21.15.154:6839/685736862] conn(0x56315a914410 0x56315a913120 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=a80fd80f3026daea server_cookie=6366b4b64a2b0cd0 in_seq=0 out_seq=0 2024-03-16T11:50:14.032 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.028+0000 7f7426ff5640 1 -- 172.21.15.29:0/2464461147 <== mds.1 v2:172.21.15.154:6838/685736862 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x56315a7fb510 con 0x56315a914410 2024-03-16T11:50:14.032 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.028+0000 7f7426ff5640 10 client.15102 handle_command_reply: tid=0 2024-03-16T11:50:14.032 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.028+0000 7f744bd65740 1 client.15102 shutdown 2024-03-16T11:50:14.032 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.028+0000 7f744bd65740 1 -- 172.21.15.29:0/2464461147 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56315a895730 msgr2=0x56315a90e210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:14.032 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.028+0000 7f744bd65740 1 --2- 172.21.15.29:0/2464461147 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56315a895730 0x56315a90e210 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f7438005600 tx=0x7f7438005970 comp rx=0 tx=0).stop 2024-03-16T11:50:14.033 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.028+0000 7f744bd65740 1 -- 172.21.15.29:0/2464461147 shutdown_connections 2024-03-16T11:50:14.033 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.028+0000 7f744bd65740 1 --2- 172.21.15.29:0/2464461147 >> [v2:172.21.15.154:6838/685736862,v1:172.21.15.154:6839/685736862] conn(0x56315a914410 0x56315a913120 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:14.033 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.028+0000 7f744bd65740 1 --2- 172.21.15.29:0/2464461147 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56315a895730 0x56315a90e210 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:14.033 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.028+0000 7f7426ff5640 0 client.15102 ms_handle_reset on v2:172.21.15.154:6838/685736862 2024-03-16T11:50:14.033 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.028+0000 7f744bd65740 1 --2- 172.21.15.29:0/2464461147 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56315a8905c0 0x56315a893610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:14.033 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.028+0000 7f744bd65740 1 --2- 172.21.15.29:0/2464461147 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56315a88fbd0 0x56315a8930d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:14.033 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.028+0000 7f744bd65740 1 -- 172.21.15.29:0/2464461147 >> 172.21.15.29:0/2464461147 conn(0x56315a88b140 msgr2=0x56315a88b520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:50:14.033 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.028+0000 7f744bd65740 1 -- 172.21.15.29:0/2464461147 shutdown_connections 2024-03-16T11:50:14.033 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.028+0000 7f744bd65740 1 -- 172.21.15.29:0/2464461147 wait complete. 2024-03-16T11:50:14.033 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.028+0000 7f744bd65740 20 client.15102 trim_cache size 0 max 16384 2024-03-16T11:50:14.033 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-03-16T11:50:14.033 INFO:teuthology.orchestra.run.smithi029.stdout: "cluster_fsid": "d1887f28-e389-11ee-95c9-87774f69a715", 2024-03-16T11:50:14.033 INFO:teuthology.orchestra.run.smithi029.stdout: "whoami": 1, 2024-03-16T11:50:14.034 INFO:teuthology.orchestra.run.smithi029.stdout: "id": 24499, 2024-03-16T11:50:14.034 INFO:teuthology.orchestra.run.smithi029.stdout: "want_state": "up:active", 2024-03-16T11:50:14.034 INFO:teuthology.orchestra.run.smithi029.stdout: "state": "up:active", 2024-03-16T11:50:14.034 INFO:teuthology.orchestra.run.smithi029.stdout: "fs_name": "cephfs", 2024-03-16T11:50:14.034 INFO:teuthology.orchestra.run.smithi029.stdout: "rank_uptime": 92.990732054000006, 2024-03-16T11:50:14.034 INFO:teuthology.orchestra.run.smithi029.stdout: "mdsmap_epoch": 21, 2024-03-16T11:50:14.034 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_epoch": 87, 2024-03-16T11:50:14.034 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_epoch_barrier": 85, 2024-03-16T11:50:14.034 INFO:teuthology.orchestra.run.smithi029.stdout: "uptime": 109.38244437500001 2024-03-16T11:50:14.034 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-03-16T11:50:14.035 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.029+0000 7f744909a640 1 -- 172.21.15.29:0/322979187 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f742c0a2800 msgr2=0x7f742c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:14.035 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.029+0000 7f744909a640 1 --2- 172.21.15.29:0/322979187 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f742c0a2800 0x7f742c0a4cc0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f743c007c90 tx=0x7f743c06d010 comp rx=0 tx=0).stop 2024-03-16T11:50:14.035 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.029+0000 7f744909a640 1 -- 172.21.15.29:0/322979187 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7444102290 msgr2=0x7f744419eec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:14.035 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.029+0000 7f744909a640 1 --2- 172.21.15.29:0/322979187 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7444102290 0x7f744419eec0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f743c009fd0 tx=0x7f743c00c800 comp rx=0 tx=0).stop 2024-03-16T11:50:14.035 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.029+0000 7f744909a640 1 -- 172.21.15.29:0/322979187 shutdown_connections 2024-03-16T11:50:14.035 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.029+0000 7f744909a640 1 --2- 172.21.15.29:0/322979187 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f742c0a2800 0x7f742c0a4cc0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:14.035 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.029+0000 7f744909a640 1 --2- 172.21.15.29:0/322979187 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7444102290 0x7f744419eec0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:14.035 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.029+0000 7f744909a640 1 --2- 172.21.15.29:0/322979187 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f74441018d0 0x7f744419c880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:14.035 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.029+0000 7f744909a640 1 --2- 172.21.15.29:0/322979187 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f74441006e0 0x7f74440694d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:14.036 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.029+0000 7f744909a640 1 -- 172.21.15.29:0/322979187 >> 172.21.15.29:0/322979187 conn(0x7f74440fbc70 msgr2=0x7f7444104ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:50:14.036 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.029+0000 7f744909a640 1 -- 172.21.15.29:0/322979187 shutdown_connections 2024-03-16T11:50:14.036 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.029+0000 7f744909a640 1 -- 172.21.15.29:0/322979187 wait complete. 2024-03-16T11:50:14.039 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-03-16T11:50:14.145 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.144+0000 7fbbcb259640 1 -- 172.21.15.29:0/647049509 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbbc41033a0 msgr2=0x7fbbc4109c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:14.145 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.144+0000 7fbbcb259640 1 --2- 172.21.15.29:0/647049509 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbbc41033a0 0x7fbbc4109c30 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fbbbc009ca0 tx=0x7fbbbc02c0f0 comp rx=0 tx=0).stop 2024-03-16T11:50:14.145 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.144+0000 7fbbcb259640 1 -- 172.21.15.29:0/647049509 shutdown_connections 2024-03-16T11:50:14.145 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.144+0000 7fbbcb259640 1 --2- 172.21.15.29:0/647049509 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbbc41033a0 0x7fbbc4109c30 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:14.145 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.144+0000 7fbbcb259640 1 --2- 172.21.15.29:0/647049509 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbbc41029e0 0x7fbbc4102e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:14.145 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.144+0000 7fbbcb259640 1 --2- 172.21.15.29:0/647049509 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbbc41017f0 0x7fbbc4101c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:14.145 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.144+0000 7fbbcb259640 1 -- 172.21.15.29:0/647049509 >> 172.21.15.29:0/647049509 conn(0x7fbbc40fcd80 msgr2=0x7fbbc40ff1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:50:14.145 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.144+0000 7fbbcb259640 1 -- 172.21.15.29:0/647049509 shutdown_connections 2024-03-16T11:50:14.145 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.144+0000 7fbbcb259640 1 -- 172.21.15.29:0/647049509 wait complete. 2024-03-16T11:50:14.146 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.144+0000 7fbbcb259640 1 Processor -- start 2024-03-16T11:50:14.146 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.144+0000 7fbbcb259640 1 -- start start 2024-03-16T11:50:14.146 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.145+0000 7fbbcb259640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbbc41017f0 0x7fbbc4197aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:14.146 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.145+0000 7fbbcb259640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbbc41029e0 0x7fbbc4197fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:14.146 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.145+0000 7fbbcb259640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbbc41033a0 0x7fbbc419f050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:14.146 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.145+0000 7fbbcb259640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fbbc410d450 con 0x7fbbc41033a0 2024-03-16T11:50:14.146 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.145+0000 7fbbcb259640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fbbc410d2d0 con 0x7fbbc41017f0 2024-03-16T11:50:14.146 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.145+0000 7fbbcb259640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fbbc410d5d0 con 0x7fbbc41029e0 2024-03-16T11:50:14.146 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.145+0000 7fbbc97f6640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbbc41033a0 0x7fbbc419f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:14.146 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.145+0000 7fbbc97f6640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbbc41033a0 0x7fbbc419f050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47532/0 (socket says 172.21.15.29:47532) 2024-03-16T11:50:14.146 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.145+0000 7fbbc97f6640 1 -- 172.21.15.29:0/4179124394 learned_addr learned my addr 172.21.15.29:0/4179124394 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:50:14.146 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.145+0000 7fbbc8ff5640 1 --2- 172.21.15.29:0/4179124394 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbbc41017f0 0x7fbbc4197aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:14.146 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.145+0000 7fbbc3fff640 1 --2- 172.21.15.29:0/4179124394 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbbc41029e0 0x7fbbc4197fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:14.146 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.145+0000 7fbbc97f6640 1 -- 172.21.15.29:0/4179124394 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbbc41029e0 msgr2=0x7fbbc4197fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:14.146 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.145+0000 7fbbc97f6640 1 --2- 172.21.15.29:0/4179124394 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbbc41029e0 0x7fbbc4197fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:14.147 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.145+0000 7fbbc97f6640 1 -- 172.21.15.29:0/4179124394 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbbc41017f0 msgr2=0x7fbbc4197aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:14.147 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.145+0000 7fbbc97f6640 1 --2- 172.21.15.29:0/4179124394 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbbc41017f0 0x7fbbc4197aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:14.147 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.145+0000 7fbbc97f6640 1 -- 172.21.15.29:0/4179124394 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbbc419f6c0 con 0x7fbbc41033a0 2024-03-16T11:50:14.147 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.145+0000 7fbbc97f6640 1 --2- 172.21.15.29:0/4179124394 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbbc41033a0 0x7fbbc419f050 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fbbbc037040 tx=0x7fbbbc032f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:50:14.147 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.145+0000 7fbbc1ffb640 1 -- 172.21.15.29:0/4179124394 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbbc041020 con 0x7fbbc41033a0 2024-03-16T11:50:14.147 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.145+0000 7fbbc1ffb640 1 -- 172.21.15.29:0/4179124394 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbbbc034cd0 con 0x7fbbc41033a0 2024-03-16T11:50:14.147 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.145+0000 7fbbcb259640 1 -- 172.21.15.29:0/4179124394 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbbc419f950 con 0x7fbbc41033a0 2024-03-16T11:50:14.147 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.145+0000 7fbbc1ffb640 1 -- 172.21.15.29:0/4179124394 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbbc039040 con 0x7fbbc41033a0 2024-03-16T11:50:14.147 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.146+0000 7fbbcb259640 1 -- 172.21.15.29:0/4179124394 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbbc419fe40 con 0x7fbbc41033a0 2024-03-16T11:50:14.153 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.148+0000 7fbbc1ffb640 1 -- 172.21.15.29:0/4179124394 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fbbbc044020 con 0x7fbbc41033a0 2024-03-16T11:50:14.153 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.148+0000 7fbbcde40740 1 Processor -- start 2024-03-16T11:50:14.153 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.148+0000 7fbbcde40740 1 -- start start 2024-03-16T11:50:14.154 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.148+0000 7fbbc1ffb640 1 --2- 172.21.15.29:0/4179124394 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fbbb40a2800 0x7fbbb40a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:14.154 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.148+0000 7fbbcde40740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5644bc9ecfe0 0x5644bc9ed400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:14.154 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.148+0000 7fbbcde40740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5644bc9ed9d0 0x5644bc9f24d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:14.154 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.148+0000 7fbbcde40740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5644bc9f2b40 0x5644bc9f4fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:14.154 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.149+0000 7fbbcde40740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x5644bc94cc80 con 0x5644bc9f2b40 2024-03-16T11:50:14.154 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.149+0000 7fbbcde40740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x5644bc8f5aa0 con 0x5644bc9ecfe0 2024-03-16T11:50:14.154 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.149+0000 7fbbcde40740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5644bc948d20 con 0x5644bc9ed9d0 2024-03-16T11:50:14.154 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.149+0000 7fbbc1ffb640 1 -- 172.21.15.29:0/4179124394 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8211+0+0 (secure 0 0 0) 0x7fbbbc0f0940 con 0x7fbbc41033a0 2024-03-16T11:50:14.154 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.149+0000 7fbbc3fff640 1 --2- 172.21.15.29:0/4179124394 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fbbb40a2800 0x7fbbb40a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:14.154 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.149+0000 7fbbc3fff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5644bc9ed9d0 0x5644bc9f24d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:14.154 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.149+0000 7fbbc8ff5640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5644bc9ecfe0 0x5644bc9ed400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:14.154 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.149+0000 7fbbc97f6640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5644bc9f2b40 0x5644bc9f4fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:14.154 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.149+0000 7fbbc3fff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5644bc9ed9d0 0x5644bc9f24d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.29:33974/0 (socket says 172.21.15.29:33974) 2024-03-16T11:50:14.154 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.149+0000 7fbbc8ff5640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5644bc9ecfe0 0x5644bc9ed400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.29:49458/0 (socket says 172.21.15.29:49458) 2024-03-16T11:50:14.155 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.149+0000 7fbbc3fff640 1 -- 172.21.15.29:0/1025880711 learned_addr learned my addr 172.21.15.29:0/1025880711 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:50:14.155 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.149+0000 7fbbc97f6640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5644bc9f2b40 0x5644bc9f4fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47542/0 (socket says 172.21.15.29:47542) 2024-03-16T11:50:14.155 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.149+0000 7fbbc8ff5640 1 -- 172.21.15.29:0/1025880711 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5644bc9ed9d0 msgr2=0x5644bc9f24d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:50:14.155 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.149+0000 7fbbc8ff5640 1 --2- 172.21.15.29:0/1025880711 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5644bc9ed9d0 0x5644bc9f24d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:14.155 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.149+0000 7fbbc8ff5640 1 -- 172.21.15.29:0/1025880711 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5644bc9f2b40 msgr2=0x5644bc9f4fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:50:14.155 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.149+0000 7fbbc8ff5640 1 --2- 172.21.15.29:0/1025880711 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5644bc9f2b40 0x5644bc9f4fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:14.155 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.149+0000 7fbbc8ff5640 1 -- 172.21.15.29:0/1025880711 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5644bc917ff0 con 0x5644bc9ecfe0 2024-03-16T11:50:14.155 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.149+0000 7fbbc97f6640 1 --2- 172.21.15.29:0/1025880711 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5644bc9f2b40 0x5644bc9f4fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:50:14.155 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.149+0000 7fbbc3fff640 1 --2- 172.21.15.29:0/4179124394 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fbbb40a2800 0x7fbbb40a4cc0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fbbb00119d0 tx=0x7fbbb0009c10 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:50:14.155 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.149+0000 7fbbc8ff5640 1 --2- 172.21.15.29:0/1025880711 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5644bc9ecfe0 0x5644bc9ed400 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fbbb8009bf0 tx=0x7fbbb8011440 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8e7ee10a1dd78899 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:50:14.155 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.149+0000 7fbbaeffd640 1 -- 172.21.15.29:0/1025880711 <== mon.2 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbb8004310 con 0x5644bc9ecfe0 2024-03-16T11:50:14.155 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.149+0000 7fbbaeffd640 1 -- 172.21.15.29:0/1025880711 <== mon.2 v2:172.21.15.152:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbbb80044b0 con 0x5644bc9ecfe0 2024-03-16T11:50:14.155 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.149+0000 7fbbaeffd640 1 -- 172.21.15.29:0/1025880711 <== mon.2 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbb80035f0 con 0x5644bc9ecfe0 2024-03-16T11:50:14.155 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.150+0000 7fbbcde40740 1 -- 172.21.15.29:0/1025880711 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5644bc9ecfe0 msgr2=0x5644bc9ed400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:14.155 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.150+0000 7fbbcde40740 1 --2- 172.21.15.29:0/1025880711 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5644bc9ecfe0 0x5644bc9ed400 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fbbb8009bf0 tx=0x7fbbb8011440 comp rx=0 tx=0).stop 2024-03-16T11:50:14.155 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.150+0000 7fbbcde40740 1 -- 172.21.15.29:0/1025880711 shutdown_connections 2024-03-16T11:50:14.156 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.150+0000 7fbbcde40740 1 --2- 172.21.15.29:0/1025880711 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5644bc9f2b40 0x5644bc9f4fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:14.156 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.150+0000 7fbbcde40740 1 --2- 172.21.15.29:0/1025880711 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5644bc9ed9d0 0x5644bc9f24d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:14.156 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.150+0000 7fbbcde40740 1 --2- 172.21.15.29:0/1025880711 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5644bc9ecfe0 0x5644bc9ed400 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:14.156 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.150+0000 7fbbcde40740 1 -- 172.21.15.29:0/1025880711 >> 172.21.15.29:0/1025880711 conn(0x5644bc9e4140 msgr2=0x5644bc9e4540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:50:14.156 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.150+0000 7fbbcde40740 1 -- 172.21.15.29:0/1025880711 shutdown_connections 2024-03-16T11:50:14.156 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.150+0000 7fbbcde40740 1 -- 172.21.15.29:0/1025880711 wait complete. 2024-03-16T11:50:14.156 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.150+0000 7fbbcde40740 1 -- start start 2024-03-16T11:50:14.156 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.150+0000 7fbbcde40740 1 Processor -- start 2024-03-16T11:50:14.156 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.151+0000 7fbbcde40740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5644bc9ecfe0 0x5644bc9f71a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:14.156 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.151+0000 7fbbcde40740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5644bc9ed9d0 0x5644bc9f76e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:14.156 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.151+0000 7fbbcde40740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5644bc9f2b40 0x5644bc9fe720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:14.156 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.151+0000 7fbbcde40740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x5644bc9f5af0 con 0x5644bc9ecfe0 2024-03-16T11:50:14.156 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.151+0000 7fbbcde40740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x5644bc908ef0 con 0x5644bc9ed9d0 2024-03-16T11:50:14.156 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.151+0000 7fbbcde40740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5644bc9f5c70 con 0x5644bc9f2b40 2024-03-16T11:50:14.157 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.151+0000 7fbbcde40740 10 client.0 _ll_register_callbacks cb 0x5644bc903dc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T11:50:14.157 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.151+0000 7fbbc8ff5640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5644bc9ecfe0 0x5644bc9f71a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:14.157 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.151+0000 7fbbc8ff5640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5644bc9ecfe0 0x5644bc9f71a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47552/0 (socket says 172.21.15.29:47552) 2024-03-16T11:50:14.157 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.151+0000 7fbbc8ff5640 1 -- 172.21.15.29:0/839232662 learned_addr learned my addr 172.21.15.29:0/839232662 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:50:14.157 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.151+0000 7fbbc97f6640 1 --2- 172.21.15.29:0/839232662 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5644bc9f2b40 0x5644bc9fe720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:14.157 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.151+0000 7fbbc3fff640 1 --2- 172.21.15.29:0/839232662 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5644bc9ed9d0 0x5644bc9f76e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:14.157 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.151+0000 7fbbc8ff5640 1 -- 172.21.15.29:0/839232662 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5644bc9f2b40 msgr2=0x5644bc9fe720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:50:14.157 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.151+0000 7fbbc8ff5640 1 --2- 172.21.15.29:0/839232662 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5644bc9f2b40 0x5644bc9fe720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:14.157 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.151+0000 7fbbc8ff5640 1 -- 172.21.15.29:0/839232662 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5644bc9ed9d0 msgr2=0x5644bc9f76e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:50:14.157 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.151+0000 7fbbc8ff5640 1 --2- 172.21.15.29:0/839232662 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5644bc9ed9d0 0x5644bc9f76e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:14.157 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.151+0000 7fbbc8ff5640 1 -- 172.21.15.29:0/839232662 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5644bc915a30 con 0x5644bc9ecfe0 2024-03-16T11:50:14.157 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.151+0000 7fbbc3fff640 1 --2- 172.21.15.29:0/839232662 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5644bc9ed9d0 0x5644bc9f76e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T11:50:14.157 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.151+0000 7fbbc8ff5640 1 --2- 172.21.15.29:0/839232662 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5644bc9ecfe0 0x5644bc9f71a0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fbbb8005e00 tx=0x7fbbb8004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e49b446b88f34b7 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:50:14.157 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.151+0000 7fbba3fff640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-16T11:50:14.157 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.152+0000 7fbba3fff640 1 -- 172.21.15.29:0/839232662 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbb8027020 con 0x5644bc9ecfe0 2024-03-16T11:50:14.158 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.152+0000 7fbbcde40740 1 -- 172.21.15.29:0/839232662 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbbb80044b0 con 0x5644bc9ecfe0 2024-03-16T11:50:14.158 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.152+0000 7fbba3fff640 1 -- 172.21.15.29:0/839232662 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8211+0+0 (secure 0 0 0) 0x7fbbb8003d40 con 0x5644bc9ecfe0 2024-03-16T11:50:14.158 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.152+0000 7fbba3fff640 1 -- 172.21.15.29:0/839232662 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fbbb801e040 con 0x5644bc9ecfe0 2024-03-16T11:50:14.158 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.152+0000 7fbbcde40740 10 client.15114 fetch_fsmap learned FSMap version 21 2024-03-16T11:50:14.158 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.152+0000 7fbbcde40740 1 -- 172.21.15.29:0/839232662 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x5644bc6612d0 con 0x5644bc9ecfe0 2024-03-16T11:50:14.158 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.152+0000 7fbba3fff640 1 -- 172.21.15.29:0/839232662 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbbb80044b0 con 0x5644bc9ecfe0 2024-03-16T11:50:14.158 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.152+0000 7fbba3fff640 1 -- 172.21.15.29:0/839232662 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7fbbb8018070 con 0x5644bc9ecfe0 2024-03-16T11:50:14.158 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.152+0000 7fbbcde40740 10 client.15114 fetch_fsmap finished waiting for FSMap version 21 2024-03-16T11:50:14.158 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.152+0000 7fbbcde40740 10 client.15114 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-16T11:50:14.158 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.152+0000 7fbbcde40740 20 client.15114 populate_metadata read hostname 'smithi029' 2024-03-16T11:50:14.158 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.152+0000 7fbbcde40740 1 --2- 172.21.15.29:0/839232662 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x5644bca03990 0x5644bca04e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:14.158 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.152+0000 7fbbcde40740 4 client.15114 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T11:50:14.158 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.152+0000 7fbbcde40740 1 -- 172.21.15.29:0/839232662 --> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5644bc95a1c0 con 0x5644bca03990 2024-03-16T11:50:14.158 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.152+0000 7fbbc3fff640 1 --2- 172.21.15.29:0/839232662 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x5644bca03990 0x5644bca04e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:14.158 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.153+0000 7fbbc3fff640 1 --2- 172.21.15.29:0/839232662 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x5644bca03990 0x5644bca04e00 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=719b0ed3af6a4069 server_cookie=d96bfac59a8d3ebf in_seq=0 out_seq=0 2024-03-16T11:50:14.159 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.153+0000 7fbba3fff640 10 client.15114 ms_handle_connect on v2:172.21.15.152:6838/1548731833 2024-03-16T11:50:14.159 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.154+0000 7fbba3fff640 1 -- 172.21.15.29:0/839232662 <== mds.0 v2:172.21.15.152:6838/1548731833 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5644bc95a1c0 con 0x5644bca03990 2024-03-16T11:50:14.159 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.154+0000 7fbba3fff640 10 client.15114 handle_command_reply: tid=0 2024-03-16T11:50:14.159 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.154+0000 7fbbcde40740 1 client.15114 shutdown 2024-03-16T11:50:14.159 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.154+0000 7fbbcde40740 1 -- 172.21.15.29:0/839232662 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5644bc9ecfe0 msgr2=0x5644bc9f71a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:14.159 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.154+0000 7fbbcde40740 1 --2- 172.21.15.29:0/839232662 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5644bc9ecfe0 0x5644bc9f71a0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fbbb8005e00 tx=0x7fbbb8004ad0 comp rx=0 tx=0).stop 2024-03-16T11:50:14.159 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.154+0000 7fbbcde40740 1 -- 172.21.15.29:0/839232662 shutdown_connections 2024-03-16T11:50:14.159 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.154+0000 7fbbcde40740 1 --2- 172.21.15.29:0/839232662 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5644bc9f2b40 0x5644bc9fe720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:14.159 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.154+0000 7fbbcde40740 1 --2- 172.21.15.29:0/839232662 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x5644bca03990 0x5644bca04e00 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:14.159 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.154+0000 7fbbcde40740 1 --2- 172.21.15.29:0/839232662 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5644bc9ed9d0 0x5644bc9f76e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:14.159 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.154+0000 7fbba3fff640 0 client.15114 ms_handle_reset on v2:172.21.15.152:6838/1548731833 2024-03-16T11:50:14.159 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.154+0000 7fbbcde40740 1 --2- 172.21.15.29:0/839232662 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5644bc9ecfe0 0x5644bc9f71a0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:14.159 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.154+0000 7fbbcde40740 1 -- 172.21.15.29:0/839232662 >> 172.21.15.29:0/839232662 conn(0x5644bc9e4140 msgr2=0x5644bc9e3380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:50:14.159 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.154+0000 7fbbcde40740 1 -- 172.21.15.29:0/839232662 shutdown_connections 2024-03-16T11:50:14.159 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.154+0000 7fbbcde40740 1 -- 172.21.15.29:0/839232662 wait complete. 2024-03-16T11:50:14.159 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.154+0000 7fbbcde40740 20 client.15114 trim_cache size 0 max 16384 2024-03-16T11:50:14.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.172+0000 7fbbcde40740 1 Processor -- start 2024-03-16T11:50:14.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.172+0000 7fbbcde40740 1 -- start start 2024-03-16T11:50:14.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.172+0000 7fbbcde40740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5644bc9ecfe0 0x5644bc9e2af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:14.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.172+0000 7fbbcde40740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5644bc9ed9d0 0x5644bc9fe6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:14.177 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.172+0000 7fbbcde40740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5644bc9f2b40 0x5644bc9e3030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:14.177 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.172+0000 7fbbcde40740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x5644bca03140 con 0x5644bc9f2b40 2024-03-16T11:50:14.177 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.172+0000 7fbbcde40740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x5644bca03810 con 0x5644bc9ed9d0 2024-03-16T11:50:14.177 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.172+0000 7fbbcde40740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5644bca03ee0 con 0x5644bc9ecfe0 2024-03-16T11:50:14.177 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.172+0000 7fbbc97f6640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5644bc9f2b40 0x5644bc9e3030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:14.177 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.172+0000 7fbbc97f6640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5644bc9f2b40 0x5644bc9e3030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47566/0 (socket says 172.21.15.29:47566) 2024-03-16T11:50:14.177 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.172+0000 7fbbc97f6640 1 -- 172.21.15.29:0/1407561450 learned_addr learned my addr 172.21.15.29:0/1407561450 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:50:14.177 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.172+0000 7fbbc8ff5640 1 --2- 172.21.15.29:0/1407561450 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5644bc9ecfe0 0x5644bc9e2af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:14.177 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.172+0000 7fbbc3fff640 1 --2- 172.21.15.29:0/1407561450 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5644bc9ed9d0 0x5644bc9fe6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:14.177 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.173+0000 7fbbc97f6640 1 -- 172.21.15.29:0/1407561450 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5644bc9ecfe0 msgr2=0x5644bc9e2af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:50:14.177 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.173+0000 7fbbc97f6640 1 --2- 172.21.15.29:0/1407561450 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5644bc9ecfe0 0x5644bc9e2af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:14.177 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.173+0000 7fbbc97f6640 1 -- 172.21.15.29:0/1407561450 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5644bc9ed9d0 msgr2=0x5644bc9fe6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:50:14.177 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.173+0000 7fbbc97f6640 1 --2- 172.21.15.29:0/1407561450 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5644bc9ed9d0 0x5644bc9fe6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:14.177 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.173+0000 7fbbc97f6640 1 -- 172.21.15.29:0/1407561450 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5644bc72b6b0 con 0x5644bc9f2b40 2024-03-16T11:50:14.177 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.173+0000 7fbbc97f6640 1 --2- 172.21.15.29:0/1407561450 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5644bc9f2b40 0x5644bc9e3030 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fbbbc0ee450 tx=0x7fbbbc02c0f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e654c961f7931aba server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:50:14.178 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.173+0000 7fbbae7fc640 1 -- 172.21.15.29:0/1407561450 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbbc0bda80 con 0x5644bc9f2b40 2024-03-16T11:50:14.178 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.173+0000 7fbbae7fc640 1 -- 172.21.15.29:0/1407561450 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbbbc0bd380 con 0x5644bc9f2b40 2024-03-16T11:50:14.178 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.173+0000 7fbbae7fc640 1 -- 172.21.15.29:0/1407561450 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbbc0c1160 con 0x5644bc9f2b40 2024-03-16T11:50:14.178 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.173+0000 7fbbcde40740 1 -- 172.21.15.29:0/1407561450 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5644bc9f2b40 msgr2=0x5644bc9e3030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:14.178 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.173+0000 7fbbcde40740 1 --2- 172.21.15.29:0/1407561450 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5644bc9f2b40 0x5644bc9e3030 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fbbbc0ee450 tx=0x7fbbbc02c0f0 comp rx=0 tx=0).stop 2024-03-16T11:50:14.178 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.173+0000 7fbbcde40740 1 -- 172.21.15.29:0/1407561450 shutdown_connections 2024-03-16T11:50:14.178 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.173+0000 7fbbcde40740 1 --2- 172.21.15.29:0/1407561450 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5644bc9f2b40 0x5644bc9e3030 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:14.178 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.173+0000 7fbbcde40740 1 --2- 172.21.15.29:0/1407561450 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5644bc9ed9d0 0x5644bc9fe6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:14.178 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.173+0000 7fbbcde40740 1 --2- 172.21.15.29:0/1407561450 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5644bc9ecfe0 0x5644bc9e2af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:14.178 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.174+0000 7fbbcde40740 1 -- 172.21.15.29:0/1407561450 >> 172.21.15.29:0/1407561450 conn(0x5644bc9e4140 msgr2=0x5644bca028e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:50:14.178 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.174+0000 7fbbcde40740 1 -- 172.21.15.29:0/1407561450 shutdown_connections 2024-03-16T11:50:14.178 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.174+0000 7fbbcde40740 1 -- 172.21.15.29:0/1407561450 wait complete. 2024-03-16T11:50:14.178 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.174+0000 7fbbcde40740 1 -- start start 2024-03-16T11:50:14.178 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.174+0000 7fbbcde40740 1 Processor -- start 2024-03-16T11:50:14.179 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.174+0000 7fbbcde40740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5644bc9ecfe0 0x5644bc9f9800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:14.179 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.174+0000 7fbbcde40740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5644bc9ed9d0 0x5644bc9f9d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:14.179 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.174+0000 7fbbcde40740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5644bc9f2b40 0x5644bc9fcd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:14.179 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.174+0000 7fbbcde40740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x5644bca04c80 con 0x5644bc9f2b40 2024-03-16T11:50:14.179 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.174+0000 7fbbcde40740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x5644bca045b0 con 0x5644bc9ed9d0 2024-03-16T11:50:14.179 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.174+0000 7fbbcde40740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5644bca05350 con 0x5644bc9ecfe0 2024-03-16T11:50:14.179 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.174+0000 7fbbcde40740 10 client.0 _ll_register_callbacks cb 0x7fbbb0011520 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T11:50:14.179 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.174+0000 7fbbc97f6640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5644bc9f2b40 0x5644bc9fcd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:14.179 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.174+0000 7fbbc8ff5640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5644bc9ecfe0 0x5644bc9f9800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:14.179 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.174+0000 7fbbc8ff5640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5644bc9ecfe0 0x5644bc9f9800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.29:34000/0 (socket says 172.21.15.29:34000) 2024-03-16T11:50:14.179 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.174+0000 7fbbc97f6640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5644bc9f2b40 0x5644bc9fcd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47576/0 (socket says 172.21.15.29:47576) 2024-03-16T11:50:14.179 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.174+0000 7fbbc8ff5640 1 -- 172.21.15.29:0/3012761116 learned_addr learned my addr 172.21.15.29:0/3012761116 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:50:14.179 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.174+0000 7fbbc3fff640 1 --2- 172.21.15.29:0/3012761116 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5644bc9ed9d0 0x5644bc9f9d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:14.179 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.175+0000 7fbbc97f6640 1 -- 172.21.15.29:0/3012761116 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5644bc9ecfe0 msgr2=0x5644bc9f9800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:50:14.180 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.175+0000 7fbbc97f6640 1 --2- 172.21.15.29:0/3012761116 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5644bc9ecfe0 0x5644bc9f9800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:14.180 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.175+0000 7fbbc97f6640 1 -- 172.21.15.29:0/3012761116 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5644bc9ed9d0 msgr2=0x5644bc9f9d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:50:14.180 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.175+0000 7fbbc97f6640 1 --2- 172.21.15.29:0/3012761116 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5644bc9ed9d0 0x5644bc9f9d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:14.180 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.175+0000 7fbbc97f6640 1 -- 172.21.15.29:0/3012761116 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5644bc9fd350 con 0x5644bc9f2b40 2024-03-16T11:50:14.180 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.175+0000 7fbbc97f6640 1 --2- 172.21.15.29:0/3012761116 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5644bc9f2b40 0x5644bc9fcd60 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fbbbc0c5560 tx=0x7fbbbc0c15a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e27152e97233b1c server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:50:14.180 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.175+0000 7fbba3fff640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-16T11:50:14.180 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.175+0000 7fbba3fff640 1 -- 172.21.15.29:0/3012761116 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbbc0d5020 con 0x5644bc9f2b40 2024-03-16T11:50:14.180 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.175+0000 7fbba3fff640 1 -- 172.21.15.29:0/3012761116 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8211+0+0 (secure 0 0 0) 0x7fbbbc0b99f0 con 0x5644bc9f2b40 2024-03-16T11:50:14.180 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.175+0000 7fbbcde40740 1 -- 172.21.15.29:0/3012761116 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbbbc0bd380 con 0x5644bc9f2b40 2024-03-16T11:50:14.180 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.175+0000 7fbba3fff640 1 -- 172.21.15.29:0/3012761116 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fbbbc0bda80 con 0x5644bc9f2b40 2024-03-16T11:50:14.180 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.175+0000 7fbbcde40740 10 client.15120 fetch_fsmap learned FSMap version 21 2024-03-16T11:50:14.180 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.175+0000 7fbbcde40740 1 -- 172.21.15.29:0/3012761116 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x5644bc9fe650 con 0x5644bc9f2b40 2024-03-16T11:50:14.180 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.175+0000 7fbba3fff640 1 -- 172.21.15.29:0/3012761116 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbbbc0bd380 con 0x5644bc9f2b40 2024-03-16T11:50:14.180 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.175+0000 7fbba3fff640 1 -- 172.21.15.29:0/3012761116 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7fbbbc09f020 con 0x5644bc9f2b40 2024-03-16T11:50:14.180 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.175+0000 7fbbcde40740 10 client.15120 fetch_fsmap finished waiting for FSMap version 21 2024-03-16T11:50:14.181 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.176+0000 7fbbcde40740 10 client.15120 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-16T11:50:14.181 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.176+0000 7fbbcde40740 20 client.15120 populate_metadata read hostname 'smithi029' 2024-03-16T11:50:14.181 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.176+0000 7fbbcde40740 1 --2- 172.21.15.29:0/3012761116 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x5644bca03990 0x5644bca6c800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:14.181 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.176+0000 7fbbcde40740 4 client.15120 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-16T11:50:14.181 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.176+0000 7fbbcde40740 1 -- 172.21.15.29:0/3012761116 --> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] -- command(tid 0: {"prefix": "status"}) -- 0x5644bc954510 con 0x5644bca03990 2024-03-16T11:50:14.181 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.176+0000 7fbbc8ff5640 1 --2- 172.21.15.29:0/3012761116 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x5644bca03990 0x5644bca6c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:14.181 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.176+0000 7fbbc8ff5640 1 --2- 172.21.15.29:0/3012761116 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x5644bca03990 0x5644bca6c800 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=100b42e3e67c5404 server_cookie=646df423cbb86c05 in_seq=0 out_seq=0 2024-03-16T11:50:14.181 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.176+0000 7fbba3fff640 10 client.15120 ms_handle_connect on v2:172.21.15.152:6838/1548731833 2024-03-16T11:50:14.181 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.176+0000 7fbba3fff640 1 -- 172.21.15.29:0/3012761116 <== mds.0 v2:172.21.15.152:6838/1548731833 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x5644bc954510 con 0x5644bca03990 2024-03-16T11:50:14.181 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.176+0000 7fbba3fff640 10 client.15120 handle_command_reply: tid=0 2024-03-16T11:50:14.181 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.176+0000 7fbbcde40740 1 client.15120 shutdown 2024-03-16T11:50:14.181 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.176+0000 7fbbcde40740 1 -- 172.21.15.29:0/3012761116 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5644bc9f2b40 msgr2=0x5644bc9fcd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:14.181 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.176+0000 7fbbcde40740 1 --2- 172.21.15.29:0/3012761116 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5644bc9f2b40 0x5644bc9fcd60 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fbbbc0c5560 tx=0x7fbbbc0c15a0 comp rx=0 tx=0).stop 2024-03-16T11:50:14.181 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.176+0000 7fbbcde40740 1 -- 172.21.15.29:0/3012761116 shutdown_connections 2024-03-16T11:50:14.181 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.176+0000 7fbbcde40740 1 --2- 172.21.15.29:0/3012761116 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5644bc9f2b40 0x5644bc9fcd60 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:14.181 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.176+0000 7fbbcde40740 1 --2- 172.21.15.29:0/3012761116 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x5644bca03990 0x5644bca6c800 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:14.182 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.176+0000 7fbbcde40740 1 --2- 172.21.15.29:0/3012761116 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5644bc9ed9d0 0x5644bc9f9d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:14.182 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.176+0000 7fbbcde40740 1 --2- 172.21.15.29:0/3012761116 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5644bc9ecfe0 0x5644bc9f9800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:14.182 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.176+0000 7fbba3fff640 0 client.15120 ms_handle_reset on v2:172.21.15.152:6838/1548731833 2024-03-16T11:50:14.182 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.176+0000 7fbbcde40740 1 -- 172.21.15.29:0/3012761116 >> 172.21.15.29:0/3012761116 conn(0x5644bc9e4140 msgr2=0x5644bc9e73a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:50:14.182 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.177+0000 7fbbcde40740 1 -- 172.21.15.29:0/3012761116 shutdown_connections 2024-03-16T11:50:14.182 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.177+0000 7fbbcde40740 1 -- 172.21.15.29:0/3012761116 wait complete. 2024-03-16T11:50:14.182 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.177+0000 7fbbcde40740 20 client.15120 trim_cache size 0 max 16384 2024-03-16T11:50:14.182 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-03-16T11:50:14.182 INFO:teuthology.orchestra.run.smithi029.stdout: "cluster_fsid": "d1887f28-e389-11ee-95c9-87774f69a715", 2024-03-16T11:50:14.182 INFO:teuthology.orchestra.run.smithi029.stdout: "whoami": 0, 2024-03-16T11:50:14.182 INFO:teuthology.orchestra.run.smithi029.stdout: "id": 24503, 2024-03-16T11:50:14.183 INFO:teuthology.orchestra.run.smithi029.stdout: "want_state": "up:active", 2024-03-16T11:50:14.183 INFO:teuthology.orchestra.run.smithi029.stdout: "state": "up:active", 2024-03-16T11:50:14.183 INFO:teuthology.orchestra.run.smithi029.stdout: "fs_name": "cephfs", 2024-03-16T11:50:14.183 INFO:teuthology.orchestra.run.smithi029.stdout: "rank_uptime": 95.166211227000005, 2024-03-16T11:50:14.183 INFO:teuthology.orchestra.run.smithi029.stdout: "mdsmap_epoch": 21, 2024-03-16T11:50:14.183 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_epoch": 87, 2024-03-16T11:50:14.183 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_epoch_barrier": 85, 2024-03-16T11:50:14.183 INFO:teuthology.orchestra.run.smithi029.stdout: "uptime": 100.375372494 2024-03-16T11:50:14.183 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-03-16T11:50:14.184 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.177+0000 7fbbcb259640 1 -- 172.21.15.29:0/4179124394 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fbbb40a2800 msgr2=0x7fbbb40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:14.184 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.177+0000 7fbbcb259640 1 --2- 172.21.15.29:0/4179124394 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fbbb40a2800 0x7fbbb40a4cc0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fbbb00119d0 tx=0x7fbbb0009c10 comp rx=0 tx=0).stop 2024-03-16T11:50:14.184 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.177+0000 7fbbcb259640 1 -- 172.21.15.29:0/4179124394 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbbc41033a0 msgr2=0x7fbbc419f050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:14.184 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.177+0000 7fbbcb259640 1 --2- 172.21.15.29:0/4179124394 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbbc41033a0 0x7fbbc419f050 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fbbbc037040 tx=0x7fbbbc032f40 comp rx=0 tx=0).stop 2024-03-16T11:50:14.184 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.177+0000 7fbbcb259640 1 -- 172.21.15.29:0/4179124394 shutdown_connections 2024-03-16T11:50:14.184 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.177+0000 7fbbcb259640 1 --2- 172.21.15.29:0/4179124394 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fbbb40a2800 0x7fbbb40a4cc0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:14.184 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.177+0000 7fbbcb259640 1 --2- 172.21.15.29:0/4179124394 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbbc41033a0 0x7fbbc419f050 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:14.184 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.177+0000 7fbbcb259640 1 --2- 172.21.15.29:0/4179124394 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbbc41029e0 0x7fbbc4197fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:14.184 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.177+0000 7fbbcb259640 1 --2- 172.21.15.29:0/4179124394 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbbc41017f0 0x7fbbc4197aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:14.184 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.177+0000 7fbbcb259640 1 -- 172.21.15.29:0/4179124394 >> 172.21.15.29:0/4179124394 conn(0x7fbbc40fcd80 msgr2=0x7fbbc4105ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:50:14.184 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.177+0000 7fbbcb259640 1 -- 172.21.15.29:0/4179124394 shutdown_connections 2024-03-16T11:50:14.184 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:14.177+0000 7fbbcb259640 1 -- 172.21.15.29:0/4179124394 wait complete. 2024-03-16T11:50:14.187 INFO:tasks.cephadm:Setup complete, yielding 2024-03-16T11:50:14.187 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-16T11:50:14.196 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi029.front.sepia.ceph.com 2024-03-16T11:50:14.196 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d1887f28-e389-11ee-95c9-87774f69a715 -- bash -c 'ceph orch status' 2024-03-16T11:50:14.439 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/d1887f28-e389-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T11:50:14.560 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:50:14 smithi029 ceph-mon[31975]: from='client.? 172.21.15.29:0/2787263113' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T11:50:14.561 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:50:14 smithi029 ceph-mon[31975]: pgmap v340: 97 pgs: 97 active+clean; 640 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 85 B/s wr, 6 op/s 2024-03-16T11:50:14.561 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:50:14 smithi029 ceph-mon[31975]: from='client.? 172.21.15.29:0/2075665980' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T11:50:14.627 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:50:14 smithi152 ceph-mon[36291]: from='client.? 172.21.15.29:0/2787263113' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T11:50:14.627 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:50:14 smithi152 ceph-mon[36291]: pgmap v340: 97 pgs: 97 active+clean; 640 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 85 B/s wr, 6 op/s 2024-03-16T11:50:14.627 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:50:14 smithi152 ceph-mon[36291]: from='client.? 172.21.15.29:0/2075665980' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T11:50:14.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:50:14 smithi154 ceph-mon[36320]: from='client.? 172.21.15.29:0/2787263113' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T11:50:14.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:50:14 smithi154 ceph-mon[36320]: pgmap v340: 97 pgs: 97 active+clean; 640 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 85 B/s wr, 6 op/s 2024-03-16T11:50:14.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:50:14 smithi154 ceph-mon[36320]: from='client.? 172.21.15.29:0/2075665980' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T11:50:16.042 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:50:15 smithi029 ceph-mon[31975]: pgmap v341: 97 pgs: 97 active+clean; 640 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-16T11:50:16.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:50:15 smithi154 ceph-mon[36320]: pgmap v341: 97 pgs: 97 active+clean; 640 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-16T11:50:16.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:50:15 smithi152 ceph-mon[36291]: pgmap v341: 97 pgs: 97 active+clean; 640 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-16T11:50:16.485 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:16.486+0000 7f115faa8700 1 -- 172.21.15.29:0/2434722935 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1158108090 msgr2=0x7f115810a480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:16.485 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:16.486+0000 7f115faa8700 1 --2- 172.21.15.29:0/2434722935 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1158108090 0x7f115810a480 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f115000b0a0 tx=0x7f115000b3b0 comp rx=0 tx=0).stop 2024-03-16T11:50:16.485 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:16.486+0000 7f115faa8700 1 -- 172.21.15.29:0/2434722935 shutdown_connections 2024-03-16T11:50:16.486 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:16.486+0000 7f115faa8700 1 --2- 172.21.15.29:0/2434722935 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1158108090 0x7f115810a480 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:16.486 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:16.486+0000 7f115faa8700 1 --2- 172.21.15.29:0/2434722935 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1158102e30 0x7f1158105220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:16.486 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:16.486+0000 7f115faa8700 1 --2- 172.21.15.29:0/2434722935 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1158105760 0x7f1158107b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:16.486 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:16.486+0000 7f115faa8700 1 -- 172.21.15.29:0/2434722935 >> 172.21.15.29:0/2434722935 conn(0x7f11580fc7a0 msgr2=0x7f11580febe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:50:16.486 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:16.486+0000 7f115faa8700 1 -- 172.21.15.29:0/2434722935 shutdown_connections 2024-03-16T11:50:16.486 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:16.486+0000 7f115faa8700 1 -- 172.21.15.29:0/2434722935 wait complete. 2024-03-16T11:50:16.488 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:16.487+0000 7f115faa8700 1 Processor -- start 2024-03-16T11:50:16.488 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:16.487+0000 7f115faa8700 1 -- start start 2024-03-16T11:50:16.489 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:16.487+0000 7f115faa8700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1158102e30 0x7f1158198510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:16.489 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:16.487+0000 7f115faa8700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1158105760 0x7f1158198a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:16.489 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:16.487+0000 7f115faa8700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1158108090 0x7f115819fad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:16.489 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:16.487+0000 7f115faa8700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f115810ded0 con 0x7f1158102e30 2024-03-16T11:50:16.489 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:16.487+0000 7f115faa8700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f115810dd50 con 0x7f1158105760 2024-03-16T11:50:16.489 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:16.487+0000 7f115d844700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1158102e30 0x7f1158198510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:16.489 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:16.487+0000 7f115faa8700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f115810e050 con 0x7f1158108090 2024-03-16T11:50:16.489 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:16.487+0000 7f115d844700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1158102e30 0x7f1158198510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47588/0 (socket says 172.21.15.29:47588) 2024-03-16T11:50:16.489 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:16.487+0000 7f115d844700 1 -- 172.21.15.29:0/954885901 learned_addr learned my addr 172.21.15.29:0/954885901 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:50:16.489 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:16.487+0000 7f115d043700 1 --2- 172.21.15.29:0/954885901 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1158105760 0x7f1158198a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:16.489 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:16.487+0000 7f115e045700 1 --2- 172.21.15.29:0/954885901 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1158108090 0x7f115819fad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:16.489 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:16.487+0000 7f115d844700 1 -- 172.21.15.29:0/954885901 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1158108090 msgr2=0x7f115819fad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:16.489 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:16.487+0000 7f115d844700 1 --2- 172.21.15.29:0/954885901 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1158108090 0x7f115819fad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:16.489 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:16.487+0000 7f115d844700 1 -- 172.21.15.29:0/954885901 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1158105760 msgr2=0x7f1158198a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:16.490 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:16.487+0000 7f115d844700 1 --2- 172.21.15.29:0/954885901 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1158105760 0x7f1158198a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:16.490 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:16.487+0000 7f115d844700 1 -- 172.21.15.29:0/954885901 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11581a0170 con 0x7f1158102e30 2024-03-16T11:50:16.490 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:16.487+0000 7f115e045700 1 --2- 172.21.15.29:0/954885901 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1158108090 0x7f115819fad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:50:16.490 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:16.487+0000 7f115d043700 1 --2- 172.21.15.29:0/954885901 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1158105760 0x7f1158198a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:50:16.490 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:16.487+0000 7f115d844700 1 --2- 172.21.15.29:0/954885901 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1158102e30 0x7f1158198510 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f115400e9d0 tx=0x7f115400ece0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:50:16.490 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:16.488+0000 7f114effd700 1 -- 172.21.15.29:0/954885901 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f115400cbd0 con 0x7f1158102e30 2024-03-16T11:50:16.490 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:16.488+0000 7f114effd700 1 -- 172.21.15.29:0/954885901 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f115400ee70 con 0x7f1158102e30 2024-03-16T11:50:16.490 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:16.488+0000 7f115faa8700 1 -- 172.21.15.29:0/954885901 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f11581a0460 con 0x7f1158102e30 2024-03-16T11:50:16.490 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:16.488+0000 7f114effd700 1 -- 172.21.15.29:0/954885901 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1154018660 con 0x7f1158102e30 2024-03-16T11:50:16.490 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:16.488+0000 7f115faa8700 1 -- 172.21.15.29:0/954885901 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11581a09f0 con 0x7f1158102e30 2024-03-16T11:50:16.490 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:16.489+0000 7f115faa8700 1 -- 172.21.15.29:0/954885901 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1158068f90 con 0x7f1158102e30 2024-03-16T11:50:16.491 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:16.490+0000 7f114effd700 1 -- 172.21.15.29:0/954885901 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f1154010430 con 0x7f1158102e30 2024-03-16T11:50:16.491 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:16.491+0000 7f114effd700 1 --2- 172.21.15.29:0/954885901 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f11440a2770 0x7f11440a4c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:16.491 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:16.491+0000 7f115d043700 1 --2- 172.21.15.29:0/954885901 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f11440a2770 0x7f11440a4c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:16.491 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:16.491+0000 7f114effd700 1 -- 172.21.15.29:0/954885901 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8211+0+0 (secure 0 0 0) 0x7f11540cec00 con 0x7f1158102e30 2024-03-16T11:50:16.491 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:16.491+0000 7f115d043700 1 --2- 172.21.15.29:0/954885901 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f11440a2770 0x7f11440a4c30 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f1148011490 tx=0x7f1148009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:50:16.494 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:16.496+0000 7f114effd700 1 -- 172.21.15.29:0/954885901 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f11540d5080 con 0x7f1158102e30 2024-03-16T11:50:16.718 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:16.719+0000 7f115faa8700 1 -- 172.21.15.29:0/954885901 --> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) -- 0x7f115802dfb0 con 0x7f11440a2770 2024-03-16T11:50:16.719 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:16.720+0000 7f114effd700 1 -- 172.21.15.29:0/954885901 <== mgr.14152 v2:172.21.15.29:6800/4192064640 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+43 (secure 0 0 0) 0x7f115802dfb0 con 0x7f11440a2770 2024-03-16T11:50:16.719 INFO:teuthology.orchestra.run.smithi029.stdout:Backend: cephadm 2024-03-16T11:50:16.719 INFO:teuthology.orchestra.run.smithi029.stdout:Available: Yes 2024-03-16T11:50:16.719 INFO:teuthology.orchestra.run.smithi029.stdout:Paused: No 2024-03-16T11:50:16.721 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:16.722+0000 7f115faa8700 1 -- 172.21.15.29:0/954885901 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f11440a2770 msgr2=0x7f11440a4c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:16.721 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:16.722+0000 7f115faa8700 1 --2- 172.21.15.29:0/954885901 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f11440a2770 0x7f11440a4c30 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f1148011490 tx=0x7f1148009260 comp rx=0 tx=0).stop 2024-03-16T11:50:16.722 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:16.722+0000 7f115faa8700 1 -- 172.21.15.29:0/954885901 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1158102e30 msgr2=0x7f1158198510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:16.722 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:16.722+0000 7f115faa8700 1 --2- 172.21.15.29:0/954885901 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1158102e30 0x7f1158198510 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f115400e9d0 tx=0x7f115400ece0 comp rx=0 tx=0).stop 2024-03-16T11:50:16.722 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:16.722+0000 7f115faa8700 1 -- 172.21.15.29:0/954885901 shutdown_connections 2024-03-16T11:50:16.722 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:16.722+0000 7f115faa8700 1 --2- 172.21.15.29:0/954885901 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f11440a2770 0x7f11440a4c30 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:16.722 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:16.722+0000 7f115faa8700 1 --2- 172.21.15.29:0/954885901 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1158108090 0x7f115819fad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:16.722 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:16.722+0000 7f115faa8700 1 --2- 172.21.15.29:0/954885901 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1158102e30 0x7f1158198510 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:16.722 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:16.722+0000 7f115faa8700 1 --2- 172.21.15.29:0/954885901 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1158105760 0x7f1158198a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:16.722 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:16.722+0000 7f115faa8700 1 -- 172.21.15.29:0/954885901 >> 172.21.15.29:0/954885901 conn(0x7f11580fc7a0 msgr2=0x7f11581036d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:50:16.722 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:16.722+0000 7f115faa8700 1 -- 172.21.15.29:0/954885901 shutdown_connections 2024-03-16T11:50:16.722 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:16.722+0000 7f115faa8700 1 -- 172.21.15.29:0/954885901 wait complete. 2024-03-16T11:50:17.564 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d1887f28-e389-11ee-95c9-87774f69a715 -- bash -c 'ceph orch ps' 2024-03-16T11:50:17.816 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/d1887f28-e389-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T11:50:17.837 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:50:17 smithi029 ceph-mon[31975]: from='client.15126 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T11:50:17.837 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:50:17 smithi029 ceph-mon[31975]: pgmap v342: 97 pgs: 97 active+clean; 640 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-16T11:50:18.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:50:17 smithi154 ceph-mon[36320]: from='client.15126 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T11:50:18.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:50:17 smithi154 ceph-mon[36320]: pgmap v342: 97 pgs: 97 active+clean; 640 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-16T11:50:18.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:50:17 smithi152 ceph-mon[36291]: from='client.15126 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T11:50:18.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:50:17 smithi152 ceph-mon[36291]: pgmap v342: 97 pgs: 97 active+clean; 640 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-16T11:50:19.572 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:19.566+0000 7f6b96327700 1 -- 172.21.15.29:0/2405229177 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6b90100b80 msgr2=0x7f6b90100fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:19.572 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:19.566+0000 7f6b96327700 1 --2- 172.21.15.29:0/2405229177 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6b90100b80 0x7f6b90100fa0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f6b80009990 tx=0x7f6b80009ca0 comp rx=0 tx=0).stop 2024-03-16T11:50:19.572 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:19.566+0000 7f6b96327700 1 -- 172.21.15.29:0/2405229177 shutdown_connections 2024-03-16T11:50:19.572 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:19.566+0000 7f6b96327700 1 --2- 172.21.15.29:0/2405229177 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6b90101ea0 0x7f6b90102320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:19.572 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:19.566+0000 7f6b96327700 1 --2- 172.21.15.29:0/2405229177 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6b90100b80 0x7f6b90100fa0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:19.572 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:19.566+0000 7f6b96327700 1 --2- 172.21.15.29:0/2405229177 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6b901014e0 0x7f6b90101960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:19.572 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:19.566+0000 7f6b96327700 1 -- 172.21.15.29:0/2405229177 >> 172.21.15.29:0/2405229177 conn(0x7f6b900fc7c0 msgr2=0x7f6b900febe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:50:19.572 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:19.566+0000 7f6b96327700 1 -- 172.21.15.29:0/2405229177 shutdown_connections 2024-03-16T11:50:19.572 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:19.566+0000 7f6b96327700 1 -- 172.21.15.29:0/2405229177 wait complete. 2024-03-16T11:50:19.572 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:19.567+0000 7f6b96327700 1 Processor -- start 2024-03-16T11:50:19.573 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:19.567+0000 7f6b96327700 1 -- start start 2024-03-16T11:50:19.573 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:19.567+0000 7f6b96327700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6b90100b80 0x7f6b9019a770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:19.573 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:19.567+0000 7f6b96327700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6b901014e0 0x7f6b9019acb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:19.573 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:19.567+0000 7f6b96327700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6b90101ea0 0x7f6b901a1d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:19.573 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:19.567+0000 7f6b96327700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f6b90110100 con 0x7f6b90101ea0 2024-03-16T11:50:19.573 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:19.567+0000 7f6b96327700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f6b9010ff80 con 0x7f6b90100b80 2024-03-16T11:50:19.573 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:19.567+0000 7f6b96327700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f6b90110280 con 0x7f6b901014e0 2024-03-16T11:50:19.573 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:19.567+0000 7f6b948c4700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6b90101ea0 0x7f6b901a1d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:19.573 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:19.567+0000 7f6b948c4700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6b90101ea0 0x7f6b901a1d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47578/0 (socket says 172.21.15.29:47578) 2024-03-16T11:50:19.573 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:19.567+0000 7f6b8ffff700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6b90100b80 0x7f6b9019a770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:19.573 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:19.567+0000 7f6b8f7fe700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6b901014e0 0x7f6b9019acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:19.573 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:19.567+0000 7f6b948c4700 1 -- 172.21.15.29:0/3712503081 learned_addr learned my addr 172.21.15.29:0/3712503081 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:50:19.573 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:19.567+0000 7f6b8f7fe700 1 -- 172.21.15.29:0/3712503081 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6b90100b80 msgr2=0x7f6b9019a770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:19.573 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:19.567+0000 7f6b8f7fe700 1 --2- 172.21.15.29:0/3712503081 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6b90100b80 0x7f6b9019a770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:19.574 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:19.567+0000 7f6b8f7fe700 1 -- 172.21.15.29:0/3712503081 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6b90101ea0 msgr2=0x7f6b901a1d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:19.574 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:19.567+0000 7f6b8f7fe700 1 --2- 172.21.15.29:0/3712503081 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6b90101ea0 0x7f6b901a1d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:19.574 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:19.567+0000 7f6b8f7fe700 1 -- 172.21.15.29:0/3712503081 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b901a23d0 con 0x7f6b901014e0 2024-03-16T11:50:19.574 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:19.568+0000 7f6b948c4700 1 --2- 172.21.15.29:0/3712503081 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6b90101ea0 0x7f6b901a1d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:50:19.574 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:19.568+0000 7f6b8f7fe700 1 --2- 172.21.15.29:0/3712503081 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6b901014e0 0x7f6b9019acb0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f6b7800ea20 tx=0x7f6b7800ed30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:50:19.574 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:19.568+0000 7f6b8d7fa700 1 -- 172.21.15.29:0/3712503081 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b7800cbd0 con 0x7f6b901014e0 2024-03-16T11:50:19.574 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:19.568+0000 7f6b96327700 1 -- 172.21.15.29:0/3712503081 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6b901a26c0 con 0x7f6b901014e0 2024-03-16T11:50:19.574 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:19.568+0000 7f6b8d7fa700 1 -- 172.21.15.29:0/3712503081 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6b7800ee60 con 0x7f6b901014e0 2024-03-16T11:50:19.574 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:19.568+0000 7f6b8d7fa700 1 -- 172.21.15.29:0/3712503081 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b78018680 con 0x7f6b901014e0 2024-03-16T11:50:19.574 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:19.568+0000 7f6b96327700 1 -- 172.21.15.29:0/3712503081 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6b901a2c50 con 0x7f6b901014e0 2024-03-16T11:50:19.574 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:19.569+0000 7f6b96327700 1 -- 172.21.15.29:0/3712503081 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6b90068f90 con 0x7f6b901014e0 2024-03-16T11:50:19.574 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:19.570+0000 7f6b8d7fa700 1 -- 172.21.15.29:0/3712503081 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f6b78018820 con 0x7f6b901014e0 2024-03-16T11:50:19.574 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:19.571+0000 7f6b8d7fa700 1 --2- 172.21.15.29:0/3712503081 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f6b7c0a2890 0x7f6b7c0a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:19.574 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:19.571+0000 7f6b8ffff700 1 --2- 172.21.15.29:0/3712503081 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f6b7c0a2890 0x7f6b7c0a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:19.574 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:19.571+0000 7f6b8d7fa700 1 -- 172.21.15.29:0/3712503081 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8211+0+0 (secure 0 0 0) 0x7f6b780ceda0 con 0x7f6b901014e0 2024-03-16T11:50:19.575 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:19.571+0000 7f6b8ffff700 1 --2- 172.21.15.29:0/3712503081 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f6b7c0a2890 0x7f6b7c0a4d50 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f6b8000b2e0 tx=0x7f6b80005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:50:19.575 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:19.575+0000 7f6b8d7fa700 1 -- 172.21.15.29:0/3712503081 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f6b7809bdf0 con 0x7f6b901014e0 2024-03-16T11:50:19.789 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:19.790+0000 7f6b96327700 1 -- 172.21.15.29:0/3712503081 --> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) -- 0x7f6b90062710 con 0x7f6b7c0a2890 2024-03-16T11:50:19.802 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:19.803+0000 7f6b8d7fa700 1 -- 172.21.15.29:0/3712503081 <== mgr.14152 v2:172.21.15.29:6800/4192064640 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+4185 (secure 0 0 0) 0x7f6b90062710 con 0x7f6b7c0a2890 2024-03-16T11:50:19.803 INFO:teuthology.orchestra.run.smithi029.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-16T11:50:19.803 INFO:teuthology.orchestra.run.smithi029.stdout:mds.a smithi029 running (109s) 17s ago 108s 15.2M - 19.0.0-2162-g3d78f5ba be86dc8cb22b e452e9b5df22 2024-03-16T11:50:19.803 INFO:teuthology.orchestra.run.smithi029.stdout:mds.b smithi152 running (106s) 39s ago 105s 25.7M - 19.0.0-2162-g3d78f5ba be86dc8cb22b a2a2b2c4d0e7 2024-03-16T11:50:19.803 INFO:teuthology.orchestra.run.smithi029.stdout:mds.c smithi154 running (2m) 39s ago 2m 20.1M - 19.0.0-2162-g3d78f5ba be86dc8cb22b 36cab96ff533 2024-03-16T11:50:19.803 INFO:teuthology.orchestra.run.smithi029.stdout:mds.d smithi029 running (2m) 17s ago 2m 16.3M - 19.0.0-2162-g3d78f5ba be86dc8cb22b 49650e1408b8 2024-03-16T11:50:19.803 INFO:teuthology.orchestra.run.smithi029.stdout:mds.e smithi152 running (2m) 39s ago 2m 18.2M - 19.0.0-2162-g3d78f5ba be86dc8cb22b 822bc521a8b8 2024-03-16T11:50:19.803 INFO:teuthology.orchestra.run.smithi029.stdout:mds.f smithi154 running (2m) 39s ago 2m 18.5M - 19.0.0-2162-g3d78f5ba be86dc8cb22b 3e849fc2b38e 2024-03-16T11:50:19.803 INFO:teuthology.orchestra.run.smithi029.stdout:mds.g smithi029 running (112s) 17s ago 111s 13.9M - 19.0.0-2162-g3d78f5ba be86dc8cb22b 37fc43af5b3f 2024-03-16T11:50:19.803 INFO:teuthology.orchestra.run.smithi029.stdout:mds.h smithi152 running (118s) 39s ago 117s 20.2M - 19.0.0-2162-g3d78f5ba be86dc8cb22b 873b7794f4d4 2024-03-16T11:50:19.803 INFO:teuthology.orchestra.run.smithi029.stdout:mds.i smithi154 running (115s) 39s ago 115s 18.0M - 19.0.0-2162-g3d78f5ba be86dc8cb22b 31362b9e990c 2024-03-16T11:50:19.803 INFO:teuthology.orchestra.run.smithi029.stdout:mds.j smithi029 running (2m) 17s ago 2m 15.3M - 19.0.0-2162-g3d78f5ba be86dc8cb22b edc59c83e563 2024-03-16T11:50:19.804 INFO:teuthology.orchestra.run.smithi029.stdout:mds.k smithi152 running (2m) 39s ago 2m 15.4M - 19.0.0-2162-g3d78f5ba be86dc8cb22b a35e36fe4008 2024-03-16T11:50:19.804 INFO:teuthology.orchestra.run.smithi029.stdout:mds.l smithi154 running (2m) 39s ago 2m 14.0M - 19.0.0-2162-g3d78f5ba be86dc8cb22b 5a6f0df597be 2024-03-16T11:50:19.804 INFO:teuthology.orchestra.run.smithi029.stdout:mgr.x smithi029 *:9283,8765 running (9m) 17s ago 9m 513M - 19.0.0-2162-g3d78f5ba be86dc8cb22b 4ae154094330 2024-03-16T11:50:19.804 INFO:teuthology.orchestra.run.smithi029.stdout:mgr.y smithi152 *:8443,8765 running (7m) 39s ago 7m 453M - 19.0.0-2162-g3d78f5ba be86dc8cb22b eec9dd63927a 2024-03-16T11:50:19.804 INFO:teuthology.orchestra.run.smithi029.stdout:mgr.z smithi154 *:8443,8765 running (7m) 39s ago 7m 454M - 19.0.0-2162-g3d78f5ba be86dc8cb22b d145605c62c8 2024-03-16T11:50:19.804 INFO:teuthology.orchestra.run.smithi029.stdout:mon.a smithi029 running (9m) 17s ago 10m 50.4M 2048M 19.0.0-2162-g3d78f5ba be86dc8cb22b 4277a6990573 2024-03-16T11:50:19.804 INFO:teuthology.orchestra.run.smithi029.stdout:mon.b smithi152 running (7m) 39s ago 7m 38.8M 2048M 19.0.0-2162-g3d78f5ba be86dc8cb22b 315b45918146 2024-03-16T11:50:19.804 INFO:teuthology.orchestra.run.smithi029.stdout:mon.c smithi154 running (8m) 39s ago 8m 39.6M 2048M 19.0.0-2162-g3d78f5ba be86dc8cb22b 79732bdfb11e 2024-03-16T11:50:19.804 INFO:teuthology.orchestra.run.smithi029.stdout:osd.0 smithi029 running (7m) 17s ago 7m 45.1M 4273M 19.0.0-2162-g3d78f5ba be86dc8cb22b e866dfb0c19b 2024-03-16T11:50:19.804 INFO:teuthology.orchestra.run.smithi029.stdout:osd.1 smithi152 running (6m) 39s ago 6m 56.9M 4270M 19.0.0-2162-g3d78f5ba be86dc8cb22b e2be6b67d15a 2024-03-16T11:50:19.804 INFO:teuthology.orchestra.run.smithi029.stdout:osd.2 smithi154 running (6m) 39s ago 6m 56.6M 4270M 19.0.0-2162-g3d78f5ba be86dc8cb22b cb30524b6745 2024-03-16T11:50:19.804 INFO:teuthology.orchestra.run.smithi029.stdout:osd.3 smithi029 running (6m) 17s ago 6m 53.3M 4273M 19.0.0-2162-g3d78f5ba be86dc8cb22b ae3a6b97b39c 2024-03-16T11:50:19.804 INFO:teuthology.orchestra.run.smithi029.stdout:osd.4 smithi152 running (5m) 39s ago 5m 42.9M 4270M 19.0.0-2162-g3d78f5ba be86dc8cb22b 78822bb59b2b 2024-03-16T11:50:19.804 INFO:teuthology.orchestra.run.smithi029.stdout:osd.5 smithi154 running (5m) 39s ago 5m 55.4M 4270M 19.0.0-2162-g3d78f5ba be86dc8cb22b 87e3c930ba34 2024-03-16T11:50:19.804 INFO:teuthology.orchestra.run.smithi029.stdout:osd.6 smithi029 running (4m) 17s ago 4m 51.2M 4273M 19.0.0-2162-g3d78f5ba be86dc8cb22b 3a976a17c65d 2024-03-16T11:50:19.805 INFO:teuthology.orchestra.run.smithi029.stdout:osd.7 smithi152 running (4m) 39s ago 4m 44.2M 4270M 19.0.0-2162-g3d78f5ba be86dc8cb22b 492ffb94294e 2024-03-16T11:50:19.805 INFO:teuthology.orchestra.run.smithi029.stdout:osd.8 smithi154 running (4m) 39s ago 4m 54.1M 4270M 19.0.0-2162-g3d78f5ba be86dc8cb22b 0b9a3c71e8c8 2024-03-16T11:50:19.805 INFO:teuthology.orchestra.run.smithi029.stdout:osd.9 smithi029 running (3m) 17s ago 3m 44.0M 4273M 19.0.0-2162-g3d78f5ba be86dc8cb22b 2f4710eef03c 2024-03-16T11:50:19.805 INFO:teuthology.orchestra.run.smithi029.stdout:osd.10 smithi152 running (3m) 39s ago 3m 41.8M 4270M 19.0.0-2162-g3d78f5ba be86dc8cb22b 826ecf362519 2024-03-16T11:50:19.805 INFO:teuthology.orchestra.run.smithi029.stdout:osd.11 smithi154 running (2m) 39s ago 2m 51.8M 4270M 19.0.0-2162-g3d78f5ba be86dc8cb22b 50699c793913 2024-03-16T11:50:19.805 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:19.806+0000 7f6b96327700 1 -- 172.21.15.29:0/3712503081 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f6b7c0a2890 msgr2=0x7f6b7c0a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:19.806 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:19.806+0000 7f6b96327700 1 --2- 172.21.15.29:0/3712503081 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f6b7c0a2890 0x7f6b7c0a4d50 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f6b8000b2e0 tx=0x7f6b80005fd0 comp rx=0 tx=0).stop 2024-03-16T11:50:19.806 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:19.806+0000 7f6b96327700 1 -- 172.21.15.29:0/3712503081 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6b901014e0 msgr2=0x7f6b9019acb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:19.806 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:19.806+0000 7f6b96327700 1 --2- 172.21.15.29:0/3712503081 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6b901014e0 0x7f6b9019acb0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f6b7800ea20 tx=0x7f6b7800ed30 comp rx=0 tx=0).stop 2024-03-16T11:50:19.806 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:19.806+0000 7f6b96327700 1 -- 172.21.15.29:0/3712503081 shutdown_connections 2024-03-16T11:50:19.806 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:19.806+0000 7f6b96327700 1 --2- 172.21.15.29:0/3712503081 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6b90101ea0 0x7f6b901a1d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:19.806 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:19.806+0000 7f6b96327700 1 --2- 172.21.15.29:0/3712503081 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6b90100b80 0x7f6b9019a770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:19.806 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:19.806+0000 7f6b96327700 1 --2- 172.21.15.29:0/3712503081 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f6b7c0a2890 0x7f6b7c0a4d50 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:19.806 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:19.806+0000 7f6b96327700 1 --2- 172.21.15.29:0/3712503081 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6b901014e0 0x7f6b9019acb0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:19.806 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:19.806+0000 7f6b96327700 1 -- 172.21.15.29:0/3712503081 >> 172.21.15.29:0/3712503081 conn(0x7f6b900fc7c0 msgr2=0x7f6b900fe2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:50:19.806 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:19.806+0000 7f6b96327700 1 -- 172.21.15.29:0/3712503081 shutdown_connections 2024-03-16T11:50:19.806 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:19.807+0000 7f6b96327700 1 -- 172.21.15.29:0/3712503081 wait complete. 2024-03-16T11:50:20.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:50:19 smithi154 ceph-mon[36320]: pgmap v343: 97 pgs: 97 active+clean; 640 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-03-16T11:50:20.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:50:19 smithi029 ceph-mon[31975]: pgmap v343: 97 pgs: 97 active+clean; 640 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-03-16T11:50:20.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:50:19 smithi152 ceph-mon[36291]: pgmap v343: 97 pgs: 97 active+clean; 640 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-03-16T11:50:20.648 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d1887f28-e389-11ee-95c9-87774f69a715 -- bash -c 'ceph orch ls' 2024-03-16T11:50:20.897 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/d1887f28-e389-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T11:50:21.018 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:50:20 smithi029 ceph-mon[31975]: from='client.24985 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T11:50:21.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:50:20 smithi154 ceph-mon[36320]: from='client.24985 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T11:50:21.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:50:20 smithi152 ceph-mon[36291]: from='client.24985 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T11:50:22.046 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:50:21 smithi154 ceph-mon[36320]: pgmap v344: 97 pgs: 97 active+clean; 640 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-16T11:50:22.047 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:50:21 smithi029 ceph-mon[31975]: pgmap v344: 97 pgs: 97 active+clean; 640 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-16T11:50:22.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:50:21 smithi152 ceph-mon[36291]: pgmap v344: 97 pgs: 97 active+clean; 640 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-16T11:50:22.810 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:22.791+0000 7fb1aeb3a700 1 -- 172.21.15.29:0/3224255838 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb1a81011d0 msgr2=0x7fb1a81015f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:22.810 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:22.791+0000 7fb1aeb3a700 1 --2- 172.21.15.29:0/3224255838 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb1a81011d0 0x7fb1a81015f0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fb19c0099c0 tx=0x7fb19c009cd0 comp rx=0 tx=0).stop 2024-03-16T11:50:22.810 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:22.791+0000 7fb1aeb3a700 1 -- 172.21.15.29:0/3224255838 shutdown_connections 2024-03-16T11:50:22.810 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:22.791+0000 7fb1aeb3a700 1 --2- 172.21.15.29:0/3224255838 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb1a8102d70 0x7fb1a8109620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:22.810 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:22.791+0000 7fb1aeb3a700 1 --2- 172.21.15.29:0/3224255838 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb1a81011d0 0x7fb1a81015f0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:22.810 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:22.791+0000 7fb1aeb3a700 1 --2- 172.21.15.29:0/3224255838 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb1a81023d0 0x7fb1a8102830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:22.811 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:22.791+0000 7fb1aeb3a700 1 -- 172.21.15.29:0/3224255838 >> 172.21.15.29:0/3224255838 conn(0x7fb1a80fc770 msgr2=0x7fb1a80febb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:50:22.811 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:22.791+0000 7fb1aeb3a700 1 -- 172.21.15.29:0/3224255838 shutdown_connections 2024-03-16T11:50:22.811 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:22.791+0000 7fb1aeb3a700 1 -- 172.21.15.29:0/3224255838 wait complete. 2024-03-16T11:50:22.811 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:22.792+0000 7fb1aeb3a700 1 Processor -- start 2024-03-16T11:50:22.811 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:22.792+0000 7fb1aeb3a700 1 -- start start 2024-03-16T11:50:22.811 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:22.792+0000 7fb1aeb3a700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb1a81011d0 0x7fb1a8198580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:22.811 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:22.792+0000 7fb1aeb3a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb1a81023d0 0x7fb1a8198ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:22.811 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:22.792+0000 7fb1aeb3a700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb1a8102d70 0x7fb1a819fb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:22.811 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:22.792+0000 7fb1aeb3a700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fb1a810cdc0 con 0x7fb1a81023d0 2024-03-16T11:50:22.811 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:22.792+0000 7fb1aeb3a700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fb1a810cc40 con 0x7fb1a81011d0 2024-03-16T11:50:22.811 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:22.792+0000 7fb1aeb3a700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fb1a810cf40 con 0x7fb1a8102d70 2024-03-16T11:50:22.811 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:22.792+0000 7fb1a7fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb1a81023d0 0x7fb1a8198ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:22.811 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:22.792+0000 7fb1a7fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb1a81023d0 0x7fb1a8198ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47592/0 (socket says 172.21.15.29:47592) 2024-03-16T11:50:22.811 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:22.792+0000 7fb1a7fff700 1 -- 172.21.15.29:0/1573530735 learned_addr learned my addr 172.21.15.29:0/1573530735 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:50:22.811 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:22.792+0000 7fb1ac8d6700 1 --2- 172.21.15.29:0/1573530735 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb1a81011d0 0x7fb1a8198580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:22.812 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:22.792+0000 7fb1ad0d7700 1 --2- 172.21.15.29:0/1573530735 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb1a8102d70 0x7fb1a819fb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:22.812 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:22.792+0000 7fb1a7fff700 1 -- 172.21.15.29:0/1573530735 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb1a8102d70 msgr2=0x7fb1a819fb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:22.812 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:22.792+0000 7fb1a7fff700 1 --2- 172.21.15.29:0/1573530735 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb1a8102d70 0x7fb1a819fb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:22.812 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:22.792+0000 7fb1a7fff700 1 -- 172.21.15.29:0/1573530735 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb1a81011d0 msgr2=0x7fb1a8198580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:22.812 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:22.792+0000 7fb1a7fff700 1 --2- 172.21.15.29:0/1573530735 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb1a81011d0 0x7fb1a8198580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:22.812 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:22.792+0000 7fb1a7fff700 1 -- 172.21.15.29:0/1573530735 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1a81a01e0 con 0x7fb1a81023d0 2024-03-16T11:50:22.812 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:22.793+0000 7fb1a7fff700 1 --2- 172.21.15.29:0/1573530735 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb1a81023d0 0x7fb1a8198ac0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fb19400e9d0 tx=0x7fb19400ece0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:50:22.812 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:22.793+0000 7fb1a5ffb700 1 -- 172.21.15.29:0/1573530735 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb19400cbd0 con 0x7fb1a81023d0 2024-03-16T11:50:22.812 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:22.793+0000 7fb1aeb3a700 1 -- 172.21.15.29:0/1573530735 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb1a81a04d0 con 0x7fb1a81023d0 2024-03-16T11:50:22.812 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:22.793+0000 7fb1a5ffb700 1 -- 172.21.15.29:0/1573530735 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb19400ee70 con 0x7fb1a81023d0 2024-03-16T11:50:22.812 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:22.793+0000 7fb1a5ffb700 1 -- 172.21.15.29:0/1573530735 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb194018680 con 0x7fb1a81023d0 2024-03-16T11:50:22.812 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:22.793+0000 7fb1aeb3a700 1 -- 172.21.15.29:0/1573530735 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1a81a0a60 con 0x7fb1a81023d0 2024-03-16T11:50:22.813 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:22.794+0000 7fb1aeb3a700 1 -- 172.21.15.29:0/1573530735 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb1a8068f90 con 0x7fb1a81023d0 2024-03-16T11:50:22.813 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:22.795+0000 7fb1a5ffb700 1 -- 172.21.15.29:0/1573530735 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fb1940188e0 con 0x7fb1a81023d0 2024-03-16T11:50:22.813 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:22.796+0000 7fb1a5ffb700 1 --2- 172.21.15.29:0/1573530735 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fb1980a2890 0x7fb1980a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:22.813 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:22.796+0000 7fb1ac8d6700 1 --2- 172.21.15.29:0/1573530735 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fb1980a2890 0x7fb1980a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:22.813 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:22.796+0000 7fb1a5ffb700 1 -- 172.21.15.29:0/1573530735 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8211+0+0 (secure 0 0 0) 0x7fb1940ceca0 con 0x7fb1a81023d0 2024-03-16T11:50:22.813 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:22.796+0000 7fb1ac8d6700 1 --2- 172.21.15.29:0/1573530735 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fb1980a2890 0x7fb1980a4d50 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fb19c000c00 tx=0x7fb19c005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:50:22.813 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:22.800+0000 7fb1a5ffb700 1 -- 172.21.15.29:0/1573530735 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fb19409bcf0 con 0x7fb1a81023d0 2024-03-16T11:50:23.011 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:23.012+0000 7fb1aeb3a700 1 -- 172.21.15.29:0/1573530735 --> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) -- 0x7fb1a80fe970 con 0x7fb1980a2890 2024-03-16T11:50:23.017 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:23.018+0000 7fb1a5ffb700 1 -- 172.21.15.29:0/1573530735 <== mgr.14152 v2:172.21.15.29:6800/4192064640 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+980 (secure 0 0 0) 0x7fb1a80fe970 con 0x7fb1980a2890 2024-03-16T11:50:23.018 INFO:teuthology.orchestra.run.smithi029.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-03-16T11:50:23.018 INFO:teuthology.orchestra.run.smithi029.stdout:mds.all 12/12 42s ago 2m smithi029=a;smithi029=d;smithi029=g;smithi029=j;smithi152=b;smithi152=e;smithi152=h;smithi152=k;smithi154=c;smithi154=f;smithi154=i;smithi154=l;count:12 2024-03-16T11:50:23.018 INFO:teuthology.orchestra.run.smithi029.stdout:mgr 3/3 42s ago 7m smithi029=x;smithi152=y;smithi154=z;count:3 2024-03-16T11:50:23.018 INFO:teuthology.orchestra.run.smithi029.stdout:mon 3/3 42s ago 8m smithi029:172.21.15.29=a;smithi152:172.21.15.152=b;smithi154:172.21.15.154=c;count:3 2024-03-16T11:50:23.018 INFO:teuthology.orchestra.run.smithi029.stdout:osd 12 42s ago - 2024-03-16T11:50:23.020 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:23.020+0000 7fb1aeb3a700 1 -- 172.21.15.29:0/1573530735 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fb1980a2890 msgr2=0x7fb1980a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:23.020 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:23.020+0000 7fb1aeb3a700 1 --2- 172.21.15.29:0/1573530735 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fb1980a2890 0x7fb1980a4d50 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fb19c000c00 tx=0x7fb19c005fb0 comp rx=0 tx=0).stop 2024-03-16T11:50:23.020 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:23.020+0000 7fb1aeb3a700 1 -- 172.21.15.29:0/1573530735 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb1a81023d0 msgr2=0x7fb1a8198ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:23.020 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:23.020+0000 7fb1aeb3a700 1 --2- 172.21.15.29:0/1573530735 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb1a81023d0 0x7fb1a8198ac0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fb19400e9d0 tx=0x7fb19400ece0 comp rx=0 tx=0).stop 2024-03-16T11:50:23.020 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:23.021+0000 7fb1aeb3a700 1 -- 172.21.15.29:0/1573530735 shutdown_connections 2024-03-16T11:50:23.020 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:23.021+0000 7fb1aeb3a700 1 --2- 172.21.15.29:0/1573530735 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fb1980a2890 0x7fb1980a4d50 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:23.020 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:23.021+0000 7fb1aeb3a700 1 --2- 172.21.15.29:0/1573530735 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb1a8102d70 0x7fb1a819fb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:23.020 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:23.021+0000 7fb1aeb3a700 1 --2- 172.21.15.29:0/1573530735 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb1a81011d0 0x7fb1a8198580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:23.020 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:23.021+0000 7fb1aeb3a700 1 --2- 172.21.15.29:0/1573530735 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb1a81023d0 0x7fb1a8198ac0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:23.020 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:23.021+0000 7fb1aeb3a700 1 -- 172.21.15.29:0/1573530735 >> 172.21.15.29:0/1573530735 conn(0x7fb1a80fc770 msgr2=0x7fb1a80fe260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:50:23.020 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:23.021+0000 7fb1aeb3a700 1 -- 172.21.15.29:0/1573530735 shutdown_connections 2024-03-16T11:50:23.020 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:23.021+0000 7fb1aeb3a700 1 -- 172.21.15.29:0/1573530735 wait complete. 2024-03-16T11:50:23.926 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d1887f28-e389-11ee-95c9-87774f69a715 -- bash -c 'ceph orch host ls' 2024-03-16T11:50:24.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:50:23 smithi154 ceph-mon[36320]: from='client.15138 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T11:50:24.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:50:23 smithi154 ceph-mon[36320]: pgmap v345: 97 pgs: 97 active+clean; 640 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-16T11:50:24.087 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:50:23 smithi029 ceph-mon[31975]: from='client.15138 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T11:50:24.087 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:50:23 smithi029 ceph-mon[31975]: pgmap v345: 97 pgs: 97 active+clean; 640 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-16T11:50:24.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:50:23 smithi152 ceph-mon[36291]: from='client.15138 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T11:50:24.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:50:23 smithi152 ceph-mon[36291]: pgmap v345: 97 pgs: 97 active+clean; 640 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-16T11:50:24.173 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/d1887f28-e389-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T11:50:25.928 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:50:25 smithi029 ceph-mon[31975]: pgmap v346: 97 pgs: 97 active+clean; 640 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-16T11:50:26.037 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:26.036+0000 7f937791a700 1 -- 172.21.15.29:0/2578959236 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9370103ea0 msgr2=0x7f937010a750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:26.038 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:26.036+0000 7f937791a700 1 --2- 172.21.15.29:0/2578959236 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9370103ea0 0x7f937010a750 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f936800b0d0 tx=0x7f936800b3e0 comp rx=0 tx=0).stop 2024-03-16T11:50:26.038 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:26.036+0000 7f937791a700 1 -- 172.21.15.29:0/2578959236 shutdown_connections 2024-03-16T11:50:26.038 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:26.036+0000 7f937791a700 1 --2- 172.21.15.29:0/2578959236 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9370103ea0 0x7f937010a750 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:26.038 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:26.036+0000 7f937791a700 1 --2- 172.21.15.29:0/2578959236 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9370102300 0x7f9370102720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:26.038 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:26.036+0000 7f937791a700 1 --2- 172.21.15.29:0/2578959236 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9370103500 0x7f9370103960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:26.038 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:26.036+0000 7f937791a700 1 -- 172.21.15.29:0/2578959236 >> 172.21.15.29:0/2578959236 conn(0x7f93700fd8e0 msgr2=0x7f93700ffd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:50:26.038 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:26.037+0000 7f937791a700 1 -- 172.21.15.29:0/2578959236 shutdown_connections 2024-03-16T11:50:26.038 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:26.037+0000 7f937791a700 1 -- 172.21.15.29:0/2578959236 wait complete. 2024-03-16T11:50:26.038 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:26.037+0000 7f937791a700 1 Processor -- start 2024-03-16T11:50:26.038 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:26.037+0000 7f937791a700 1 -- start start 2024-03-16T11:50:26.038 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:26.037+0000 7f937791a700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9370102300 0x7f9370198480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:26.038 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:26.037+0000 7f937791a700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9370103500 0x7f93701989c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:26.038 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:26.037+0000 7f937791a700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9370103ea0 0x7f937019fa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:26.039 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:26.037+0000 7f937791a700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f937010dec0 con 0x7f9370103500 2024-03-16T11:50:26.039 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:26.037+0000 7f937791a700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f937010dd40 con 0x7f9370103ea0 2024-03-16T11:50:26.039 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:26.037+0000 7f937791a700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f937010e040 con 0x7f9370102300 2024-03-16T11:50:26.039 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:26.037+0000 7f9374eb5700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9370103500 0x7f93701989c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:26.039 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:26.038+0000 7f9374eb5700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9370103500 0x7f93701989c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47618/0 (socket says 172.21.15.29:47618) 2024-03-16T11:50:26.039 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:26.038+0000 7f9374eb5700 1 -- 172.21.15.29:0/335865337 learned_addr learned my addr 172.21.15.29:0/335865337 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:50:26.039 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:26.038+0000 7f93756b6700 1 --2- 172.21.15.29:0/335865337 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9370102300 0x7f9370198480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:26.039 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:26.038+0000 7f9375eb7700 1 --2- 172.21.15.29:0/335865337 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9370103ea0 0x7f937019fa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:26.039 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:26.038+0000 7f9374eb5700 1 -- 172.21.15.29:0/335865337 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9370102300 msgr2=0x7f9370198480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:26.039 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:26.038+0000 7f9374eb5700 1 --2- 172.21.15.29:0/335865337 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9370102300 0x7f9370198480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:26.039 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:26.038+0000 7f9374eb5700 1 -- 172.21.15.29:0/335865337 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9370103ea0 msgr2=0x7f937019fa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:26.039 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:26.038+0000 7f9374eb5700 1 --2- 172.21.15.29:0/335865337 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9370103ea0 0x7f937019fa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:26.039 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:26.038+0000 7f9374eb5700 1 -- 172.21.15.29:0/335865337 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93701a00e0 con 0x7f9370103500 2024-03-16T11:50:26.039 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:26.038+0000 7f93756b6700 1 --2- 172.21.15.29:0/335865337 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9370102300 0x7f9370198480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:50:26.039 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:26.038+0000 7f9374eb5700 1 --2- 172.21.15.29:0/335865337 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9370103500 0x7f93701989c0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f936000e9d0 tx=0x7f936000ece0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:50:26.040 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:26.038+0000 7f93667fc700 1 -- 172.21.15.29:0/335865337 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f936000cbd0 con 0x7f9370103500 2024-03-16T11:50:26.040 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:26.038+0000 7f93667fc700 1 -- 172.21.15.29:0/335865337 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f936000ee70 con 0x7f9370103500 2024-03-16T11:50:26.040 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:26.038+0000 7f937791a700 1 -- 172.21.15.29:0/335865337 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f93701a03d0 con 0x7f9370103500 2024-03-16T11:50:26.040 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:26.038+0000 7f93667fc700 1 -- 172.21.15.29:0/335865337 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9360018660 con 0x7f9370103500 2024-03-16T11:50:26.040 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:26.038+0000 7f937791a700 1 -- 172.21.15.29:0/335865337 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f93701a0960 con 0x7f9370103500 2024-03-16T11:50:26.041 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:26.039+0000 7f937791a700 1 -- 172.21.15.29:0/335865337 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9370068f90 con 0x7f9370103500 2024-03-16T11:50:26.041 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:26.040+0000 7f93667fc700 1 -- 172.21.15.29:0/335865337 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f9360010430 con 0x7f9370103500 2024-03-16T11:50:26.041 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:26.041+0000 7f93667fc700 1 --2- 172.21.15.29:0/335865337 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f935c0a2910 0x7f935c0a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:26.041 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:26.041+0000 7f93756b6700 1 --2- 172.21.15.29:0/335865337 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f935c0a2910 0x7f935c0a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:26.041 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:26.041+0000 7f93667fc700 1 -- 172.21.15.29:0/335865337 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8211+0+0 (secure 0 0 0) 0x7f93600cf560 con 0x7f9370103500 2024-03-16T11:50:26.041 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:26.041+0000 7f93756b6700 1 --2- 172.21.15.29:0/335865337 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f935c0a2910 0x7f935c0a4dd0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f9370103360 tx=0x7f936c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:50:26.044 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:26.045+0000 7f93667fc700 1 -- 172.21.15.29:0/335865337 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f9360026070 con 0x7f9370103500 2024-03-16T11:50:26.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:50:25 smithi154 ceph-mon[36320]: pgmap v346: 97 pgs: 97 active+clean; 640 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-16T11:50:26.129 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:50:25 smithi152 ceph-mon[36291]: pgmap v346: 97 pgs: 97 active+clean; 640 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-16T11:50:26.255 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:26.256+0000 7f937791a700 1 -- 172.21.15.29:0/335865337 --> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) -- 0x7f9370062710 con 0x7f935c0a2910 2024-03-16T11:50:26.256 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:26.257+0000 7f93667fc700 1 -- 172.21.15.29:0/335865337 <== mgr.14152 v2:172.21.15.29:6800/4192064640 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+191 (secure 0 0 0) 0x7f9370062710 con 0x7f935c0a2910 2024-03-16T11:50:26.257 INFO:teuthology.orchestra.run.smithi029.stdout:HOST ADDR LABELS STATUS 2024-03-16T11:50:26.257 INFO:teuthology.orchestra.run.smithi029.stdout:smithi029 172.21.15.29 2024-03-16T11:50:26.257 INFO:teuthology.orchestra.run.smithi029.stdout:smithi152 172.21.15.152 2024-03-16T11:50:26.257 INFO:teuthology.orchestra.run.smithi029.stdout:smithi154 172.21.15.154 2024-03-16T11:50:26.257 INFO:teuthology.orchestra.run.smithi029.stdout:3 hosts in cluster 2024-03-16T11:50:26.259 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:26.260+0000 7f937791a700 1 -- 172.21.15.29:0/335865337 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f935c0a2910 msgr2=0x7f935c0a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:26.259 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:26.260+0000 7f937791a700 1 --2- 172.21.15.29:0/335865337 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f935c0a2910 0x7f935c0a4dd0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f9370103360 tx=0x7f936c009260 comp rx=0 tx=0).stop 2024-03-16T11:50:26.259 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:26.260+0000 7f937791a700 1 -- 172.21.15.29:0/335865337 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9370103500 msgr2=0x7f93701989c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:26.259 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:26.260+0000 7f937791a700 1 --2- 172.21.15.29:0/335865337 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9370103500 0x7f93701989c0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f936000e9d0 tx=0x7f936000ece0 comp rx=0 tx=0).stop 2024-03-16T11:50:26.259 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:26.260+0000 7f937791a700 1 -- 172.21.15.29:0/335865337 shutdown_connections 2024-03-16T11:50:26.259 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:26.260+0000 7f937791a700 1 --2- 172.21.15.29:0/335865337 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9370103ea0 0x7f937019fa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:26.259 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:26.260+0000 7f937791a700 1 --2- 172.21.15.29:0/335865337 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f935c0a2910 0x7f935c0a4dd0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:26.259 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:26.260+0000 7f937791a700 1 --2- 172.21.15.29:0/335865337 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9370102300 0x7f9370198480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:26.259 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:26.260+0000 7f937791a700 1 --2- 172.21.15.29:0/335865337 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9370103500 0x7f93701989c0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:26.260 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:26.260+0000 7f937791a700 1 -- 172.21.15.29:0/335865337 >> 172.21.15.29:0/335865337 conn(0x7f93700fd8e0 msgr2=0x7f9370106800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:50:26.260 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:26.260+0000 7f937791a700 1 -- 172.21.15.29:0/335865337 shutdown_connections 2024-03-16T11:50:26.260 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:26.260+0000 7f937791a700 1 -- 172.21.15.29:0/335865337 wait complete. 2024-03-16T11:50:26.988 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:50:26 smithi029 ceph-mon[31975]: from='client.15144 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T11:50:27.020 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d1887f28-e389-11ee-95c9-87774f69a715 -- bash -c 'ceph orch device ls' 2024-03-16T11:50:27.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:50:26 smithi154 ceph-mon[36320]: from='client.15144 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T11:50:27.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:50:26 smithi152 ceph-mon[36291]: from='client.15144 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T11:50:27.269 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/d1887f28-e389-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T11:50:28.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:50:27 smithi154 ceph-mon[36320]: pgmap v347: 97 pgs: 97 active+clean; 640 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-16T11:50:28.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:50:27 smithi029 ceph-mon[31975]: pgmap v347: 97 pgs: 97 active+clean; 640 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-16T11:50:28.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:50:27 smithi152 ceph-mon[36291]: pgmap v347: 97 pgs: 97 active+clean; 640 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-16T11:50:29.290 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:29.286+0000 7f4b49ff9700 1 -- 172.21.15.29:0/3589880334 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4b44102da0 msgr2=0x7f4b44109650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:29.290 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:29.286+0000 7f4b49ff9700 1 --2- 172.21.15.29:0/3589880334 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4b44102da0 0x7f4b44109650 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f4b3c00b0d0 tx=0x7f4b3c00b3e0 comp rx=0 tx=0).stop 2024-03-16T11:50:29.290 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:29.286+0000 7f4b49ff9700 1 -- 172.21.15.29:0/3589880334 shutdown_connections 2024-03-16T11:50:29.291 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:29.286+0000 7f4b49ff9700 1 --2- 172.21.15.29:0/3589880334 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4b44102da0 0x7f4b44109650 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:29.291 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:29.286+0000 7f4b49ff9700 1 --2- 172.21.15.29:0/3589880334 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4b44101200 0x7f4b44101620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:29.291 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:29.286+0000 7f4b49ff9700 1 --2- 172.21.15.29:0/3589880334 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4b44102400 0x7f4b44102860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:29.291 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:29.286+0000 7f4b49ff9700 1 -- 172.21.15.29:0/3589880334 >> 172.21.15.29:0/3589880334 conn(0x7f4b440fc7a0 msgr2=0x7f4b440febe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:50:29.291 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:29.286+0000 7f4b49ff9700 1 -- 172.21.15.29:0/3589880334 shutdown_connections 2024-03-16T11:50:29.291 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:29.286+0000 7f4b49ff9700 1 -- 172.21.15.29:0/3589880334 wait complete. 2024-03-16T11:50:29.291 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:29.286+0000 7f4b49ff9700 1 Processor -- start 2024-03-16T11:50:29.291 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:29.286+0000 7f4b49ff9700 1 -- start start 2024-03-16T11:50:29.291 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:29.286+0000 7f4b49ff9700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4b44101200 0x7f4b44105190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:29.291 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:29.286+0000 7f4b49ff9700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4b44102400 0x7f4b441037e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:29.291 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:29.287+0000 7f4b49ff9700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4b44102da0 0x7f4b44103d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:29.291 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:29.287+0000 7f4b49ff9700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f4b4410cde0 con 0x7f4b44102da0 2024-03-16T11:50:29.292 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:29.287+0000 7f4b49ff9700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f4b4410cc60 con 0x7f4b44101200 2024-03-16T11:50:29.292 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:29.287+0000 7f4b49ff9700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f4b4410cf60 con 0x7f4b44102400 2024-03-16T11:50:29.292 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:29.287+0000 7f4b43fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4b44102da0 0x7f4b44103d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:29.292 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:29.287+0000 7f4b3bfff700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4b44102400 0x7f4b441037e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:29.292 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:29.287+0000 7f4b437fe700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4b44101200 0x7f4b44105190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:29.292 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:29.287+0000 7f4b3bfff700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4b44102400 0x7f4b441037e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.29:38240/0 (socket says 172.21.15.29:38240) 2024-03-16T11:50:29.292 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:29.287+0000 7f4b3bfff700 1 -- 172.21.15.29:0/1889896343 learned_addr learned my addr 172.21.15.29:0/1889896343 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:50:29.292 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:29.287+0000 7f4b43fff700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4b44102da0 0x7f4b44103d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47622/0 (socket says 172.21.15.29:47622) 2024-03-16T11:50:29.292 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:29.287+0000 7f4b437fe700 1 -- 172.21.15.29:0/1889896343 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4b44102400 msgr2=0x7f4b441037e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:29.292 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:29.287+0000 7f4b437fe700 1 --2- 172.21.15.29:0/1889896343 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4b44102400 0x7f4b441037e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:29.292 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:29.287+0000 7f4b437fe700 1 -- 172.21.15.29:0/1889896343 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4b44102da0 msgr2=0x7f4b44103d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:29.292 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:29.287+0000 7f4b437fe700 1 --2- 172.21.15.29:0/1889896343 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4b44102da0 0x7f4b44103d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:29.292 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:29.287+0000 7f4b437fe700 1 -- 172.21.15.29:0/1889896343 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b441045d0 con 0x7f4b44101200 2024-03-16T11:50:29.293 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:29.287+0000 7f4b43fff700 1 --2- 172.21.15.29:0/1889896343 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4b44102da0 0x7f4b44103d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:50:29.293 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:29.287+0000 7f4b437fe700 1 --2- 172.21.15.29:0/1889896343 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4b44101200 0x7f4b44105190 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f4b3400e9c0 tx=0x7f4b3400ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:50:29.293 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:29.287+0000 7f4b417fa700 1 -- 172.21.15.29:0/1889896343 <== mon.2 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b34004220 con 0x7f4b44101200 2024-03-16T11:50:29.293 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:29.287+0000 7f4b417fa700 1 -- 172.21.15.29:0/1889896343 <== mon.2 v2:172.21.15.152:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4b340043c0 con 0x7f4b44101200 2024-03-16T11:50:29.293 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:29.287+0000 7f4b49ff9700 1 -- 172.21.15.29:0/1889896343 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4b441a2680 con 0x7f4b44101200 2024-03-16T11:50:29.293 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:29.287+0000 7f4b417fa700 1 -- 172.21.15.29:0/1889896343 <== mon.2 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b34003cc0 con 0x7f4b44101200 2024-03-16T11:50:29.293 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:29.288+0000 7f4b49ff9700 1 -- 172.21.15.29:0/1889896343 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4b441a2b90 con 0x7f4b44101200 2024-03-16T11:50:29.293 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:29.288+0000 7f4b49ff9700 1 -- 172.21.15.29:0/1889896343 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4b44068f90 con 0x7f4b44101200 2024-03-16T11:50:29.293 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:29.290+0000 7f4b417fa700 1 -- 172.21.15.29:0/1889896343 <== mon.2 v2:172.21.15.152:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f4b3400c7c0 con 0x7f4b44101200 2024-03-16T11:50:29.293 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:29.290+0000 7f4b417fa700 1 --2- 172.21.15.29:0/1889896343 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f4b300a2890 0x7f4b300a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:29.293 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:29.290+0000 7f4b417fa700 1 -- 172.21.15.29:0/1889896343 <== mon.2 v2:172.21.15.152:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8211+0+0 (secure 0 0 0) 0x7f4b340cedc0 con 0x7f4b44101200 2024-03-16T11:50:29.293 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:29.291+0000 7f4b3bfff700 1 --2- 172.21.15.29:0/1889896343 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f4b300a2890 0x7f4b300a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:29.293 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:29.291+0000 7f4b3bfff700 1 --2- 172.21.15.29:0/1889896343 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f4b300a2890 0x7f4b300a4d50 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f4b2c009d20 tx=0x7f4b2c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:50:29.294 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:29.294+0000 7f4b417fa700 1 -- 172.21.15.29:0/1889896343 <== mon.2 v2:172.21.15.152:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f4b3409bea0 con 0x7f4b44101200 2024-03-16T11:50:29.507 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:29.508+0000 7f4b49ff9700 1 -- 172.21.15.29:0/1889896343 --> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) -- 0x7f4b441a2e50 con 0x7f4b300a2890 2024-03-16T11:50:29.509 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:29.510+0000 7f4b417fa700 1 -- 172.21.15.29:0/1889896343 <== mgr.14152 v2:172.21.15.29:6800/4192064640 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+532 (secure 0 0 0) 0x7f4b441a2e50 con 0x7f4b300a2890 2024-03-16T11:50:29.509 INFO:teuthology.orchestra.run.smithi029.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-03-16T11:50:29.509 INFO:teuthology.orchestra.run.smithi029.stdout:smithi029 /dev/nvme0n1 ssd INTEL_SSDPED1D480GA_PHMB7512009B480DGN 447G No 3m ago Has a FileSystem, LVM detected 2024-03-16T11:50:29.509 INFO:teuthology.orchestra.run.smithi029.stdout:smithi152 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT623300JT400BGN 372G No 3m ago Has a FileSystem, LVM detected 2024-03-16T11:50:29.509 INFO:teuthology.orchestra.run.smithi029.stdout:smithi154 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT6233003B400BGN 372G No 2m ago Has a FileSystem, LVM detected 2024-03-16T11:50:29.512 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:29.512+0000 7f4b49ff9700 1 -- 172.21.15.29:0/1889896343 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f4b300a2890 msgr2=0x7f4b300a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:29.512 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:29.512+0000 7f4b49ff9700 1 --2- 172.21.15.29:0/1889896343 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f4b300a2890 0x7f4b300a4d50 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f4b2c009d20 tx=0x7f4b2c009260 comp rx=0 tx=0).stop 2024-03-16T11:50:29.512 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:29.512+0000 7f4b49ff9700 1 -- 172.21.15.29:0/1889896343 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4b44101200 msgr2=0x7f4b44105190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:29.512 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:29.512+0000 7f4b49ff9700 1 --2- 172.21.15.29:0/1889896343 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4b44101200 0x7f4b44105190 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f4b3400e9c0 tx=0x7f4b3400ecd0 comp rx=0 tx=0).stop 2024-03-16T11:50:29.512 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:29.512+0000 7f4b49ff9700 1 -- 172.21.15.29:0/1889896343 shutdown_connections 2024-03-16T11:50:29.512 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:29.512+0000 7f4b49ff9700 1 --2- 172.21.15.29:0/1889896343 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4b44102da0 0x7f4b44103d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:29.512 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:29.512+0000 7f4b49ff9700 1 --2- 172.21.15.29:0/1889896343 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4b44101200 0x7f4b44105190 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:29.512 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:29.512+0000 7f4b49ff9700 1 --2- 172.21.15.29:0/1889896343 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f4b300a2890 0x7f4b300a4d50 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:29.512 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:29.512+0000 7f4b49ff9700 1 --2- 172.21.15.29:0/1889896343 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4b44102400 0x7f4b441037e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:29.512 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:29.512+0000 7f4b49ff9700 1 -- 172.21.15.29:0/1889896343 >> 172.21.15.29:0/1889896343 conn(0x7f4b440fc7a0 msgr2=0x7f4b440fe250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:50:29.512 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:29.512+0000 7f4b49ff9700 1 -- 172.21.15.29:0/1889896343 shutdown_connections 2024-03-16T11:50:29.512 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:29.512+0000 7f4b49ff9700 1 -- 172.21.15.29:0/1889896343 wait complete. 2024-03-16T11:50:30.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:50:29 smithi154 ceph-mon[36320]: from='client.24968 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T11:50:30.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:50:29 smithi154 ceph-mon[36320]: pgmap v348: 97 pgs: 97 active+clean; 640 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-03-16T11:50:30.087 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:50:29 smithi029 ceph-mon[31975]: from='client.24968 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T11:50:30.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:50:29 smithi029 ceph-mon[31975]: pgmap v348: 97 pgs: 97 active+clean; 640 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-03-16T11:50:30.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:50:29 smithi152 ceph-mon[36291]: from='client.24968 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T11:50:30.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:50:29 smithi152 ceph-mon[36291]: pgmap v348: 97 pgs: 97 active+clean; 640 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-03-16T11:50:30.334 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-16T11:50:30.343 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi029.front.sepia.ceph.com 2024-03-16T11:50:30.343 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d1887f28-e389-11ee-95c9-87774f69a715 -- bash -c 'ceph fs dump' 2024-03-16T11:50:30.591 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/d1887f28-e389-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T11:50:32.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:50:31 smithi154 ceph-mon[36320]: pgmap v349: 97 pgs: 97 active+clean; 640 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-16T11:50:32.047 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:50:31 smithi029 ceph-mon[31975]: pgmap v349: 97 pgs: 97 active+clean; 640 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-16T11:50:32.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:50:31 smithi152 ceph-mon[36291]: pgmap v349: 97 pgs: 97 active+clean; 640 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-16T11:50:32.475 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:32.474+0000 7f7a780f8700 1 -- 172.21.15.29:0/1286992277 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7a70102470 msgr2=0x7f7a701028d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:32.476 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:32.474+0000 7f7a780f8700 1 --2- 172.21.15.29:0/1286992277 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7a70102470 0x7f7a701028d0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f7a600099c0 tx=0x7f7a60009cd0 comp rx=0 tx=0).stop 2024-03-16T11:50:32.476 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:32.474+0000 7f7a780f8700 1 -- 172.21.15.29:0/1286992277 shutdown_connections 2024-03-16T11:50:32.476 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:32.474+0000 7f7a780f8700 1 --2- 172.21.15.29:0/1286992277 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7a70102e10 0x7f7a701096c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:32.476 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:32.474+0000 7f7a780f8700 1 --2- 172.21.15.29:0/1286992277 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7a70101270 0x7f7a70101690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:32.476 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:32.474+0000 7f7a780f8700 1 --2- 172.21.15.29:0/1286992277 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7a70102470 0x7f7a701028d0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:32.476 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:32.474+0000 7f7a780f8700 1 -- 172.21.15.29:0/1286992277 >> 172.21.15.29:0/1286992277 conn(0x7f7a700fc830 msgr2=0x7f7a700fec50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:50:32.476 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:32.474+0000 7f7a780f8700 1 -- 172.21.15.29:0/1286992277 shutdown_connections 2024-03-16T11:50:32.476 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:32.474+0000 7f7a780f8700 1 -- 172.21.15.29:0/1286992277 wait complete. 2024-03-16T11:50:32.476 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:32.475+0000 7f7a780f8700 1 Processor -- start 2024-03-16T11:50:32.476 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:32.475+0000 7f7a780f8700 1 -- start start 2024-03-16T11:50:32.476 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:32.475+0000 7f7a780f8700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7a70101270 0x7f7a701984f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:32.476 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:32.475+0000 7f7a780f8700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7a70102470 0x7f7a70198a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:32.477 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:32.475+0000 7f7a780f8700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7a70102e10 0x7f7a7019fab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:32.477 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:32.475+0000 7f7a780f8700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f7a7010cdc0 con 0x7f7a70102e10 2024-03-16T11:50:32.477 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:32.475+0000 7f7a780f8700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f7a7010cc40 con 0x7f7a70101270 2024-03-16T11:50:32.477 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:32.475+0000 7f7a780f8700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f7a7010cf40 con 0x7f7a70102470 2024-03-16T11:50:32.477 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:32.475+0000 7f7a76695700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7a70102e10 0x7f7a7019fab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:32.477 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:32.475+0000 7f7a75693700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7a70102470 0x7f7a70198a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:32.477 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:32.475+0000 7f7a76695700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7a70102e10 0x7f7a7019fab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59280/0 (socket says 172.21.15.29:59280) 2024-03-16T11:50:32.477 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:32.475+0000 7f7a76695700 1 -- 172.21.15.29:0/3284245376 learned_addr learned my addr 172.21.15.29:0/3284245376 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:50:32.477 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:32.475+0000 7f7a75e94700 1 --2- 172.21.15.29:0/3284245376 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7a70101270 0x7f7a701984f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:32.477 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:32.475+0000 7f7a76695700 1 -- 172.21.15.29:0/3284245376 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7a70102470 msgr2=0x7f7a70198a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:32.477 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:32.475+0000 7f7a76695700 1 --2- 172.21.15.29:0/3284245376 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7a70102470 0x7f7a70198a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:32.477 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:32.475+0000 7f7a76695700 1 -- 172.21.15.29:0/3284245376 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7a70101270 msgr2=0x7f7a701984f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:32.477 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:32.475+0000 7f7a76695700 1 --2- 172.21.15.29:0/3284245376 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7a70101270 0x7f7a701984f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:32.477 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:32.475+0000 7f7a76695700 1 -- 172.21.15.29:0/3284245376 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a701a0150 con 0x7f7a70102e10 2024-03-16T11:50:32.478 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:32.476+0000 7f7a76695700 1 --2- 172.21.15.29:0/3284245376 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7a70102e10 0x7f7a7019fab0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f7a6800ef60 tx=0x7f7a6800ef90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:50:32.478 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:32.476+0000 7f7a66ffd700 1 -- 172.21.15.29:0/3284245376 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a680092e0 con 0x7f7a70102e10 2024-03-16T11:50:32.478 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:32.476+0000 7f7a780f8700 1 -- 172.21.15.29:0/3284245376 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7a701a03e0 con 0x7f7a70102e10 2024-03-16T11:50:32.478 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:32.476+0000 7f7a66ffd700 1 -- 172.21.15.29:0/3284245376 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7a68004bb0 con 0x7f7a70102e10 2024-03-16T11:50:32.478 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:32.476+0000 7f7a66ffd700 1 -- 172.21.15.29:0/3284245376 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a68004280 con 0x7f7a70102e10 2024-03-16T11:50:32.478 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:32.476+0000 7f7a780f8700 1 -- 172.21.15.29:0/3284245376 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7a701a0970 con 0x7f7a70102e10 2024-03-16T11:50:32.479 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:32.477+0000 7f7a780f8700 1 -- 172.21.15.29:0/3284245376 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7a70068f90 con 0x7f7a70102e10 2024-03-16T11:50:32.479 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:32.478+0000 7f7a66ffd700 1 -- 172.21.15.29:0/3284245376 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f7a6801b030 con 0x7f7a70102e10 2024-03-16T11:50:32.479 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:32.478+0000 7f7a66ffd700 1 --2- 172.21.15.29:0/3284245376 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f7a5c0a2910 0x7f7a5c0a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:32.479 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:32.478+0000 7f7a75e94700 1 --2- 172.21.15.29:0/3284245376 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f7a5c0a2910 0x7f7a5c0a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:32.479 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:32.479+0000 7f7a66ffd700 1 -- 172.21.15.29:0/3284245376 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8211+0+0 (secure 0 0 0) 0x7f7a680ce3d0 con 0x7f7a70102e10 2024-03-16T11:50:32.479 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:32.479+0000 7f7a75e94700 1 --2- 172.21.15.29:0/3284245376 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f7a5c0a2910 0x7f7a5c0a4dd0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f7a701022d0 tx=0x7f7a6c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:50:32.482 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:32.483+0000 7f7a66ffd700 1 -- 172.21.15.29:0/3284245376 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f7a68022020 con 0x7f7a70102e10 2024-03-16T11:50:32.734 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:32.734+0000 7f7a780f8700 1 -- 172.21.15.29:0/3284245376 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump"} v 0) -- 0x7f7a701a12b0 con 0x7f7a70102e10 2024-03-16T11:50:32.734 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:32.735+0000 7f7a66ffd700 1 -- 172.21.15.29:0/3284245376 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 21 v21) ==== 76+0+3052 (secure 0 0 0) 0x7f7a68010040 con 0x7f7a70102e10 2024-03-16T11:50:32.734 INFO:teuthology.orchestra.run.smithi029.stdout:e21 2024-03-16T11:50:32.734 INFO:teuthology.orchestra.run.smithi029.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-16T11:50:32.734 INFO:teuthology.orchestra.run.smithi029.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-03-16T11:50:32.735 INFO:teuthology.orchestra.run.smithi029.stdout:legacy client fscid: 1 2024-03-16T11:50:32.735 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-16T11:50:32.735 INFO:teuthology.orchestra.run.smithi029.stdout:Filesystem 'cephfs' (1) 2024-03-16T11:50:32.735 INFO:teuthology.orchestra.run.smithi029.stdout:fs_name cephfs 2024-03-16T11:50:32.735 INFO:teuthology.orchestra.run.smithi029.stdout:epoch 21 2024-03-16T11:50:32.735 INFO:teuthology.orchestra.run.smithi029.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-03-16T11:50:32.735 INFO:teuthology.orchestra.run.smithi029.stdout:created 2024-03-16T11:48:38.431509+0000 2024-03-16T11:50:32.735 INFO:teuthology.orchestra.run.smithi029.stdout:modified 2024-03-16T11:48:42.969084+0000 2024-03-16T11:50:32.735 INFO:teuthology.orchestra.run.smithi029.stdout:tableserver 0 2024-03-16T11:50:32.735 INFO:teuthology.orchestra.run.smithi029.stdout:root 0 2024-03-16T11:50:32.735 INFO:teuthology.orchestra.run.smithi029.stdout:session_timeout 300 2024-03-16T11:50:32.735 INFO:teuthology.orchestra.run.smithi029.stdout:session_autoclose 300 2024-03-16T11:50:32.735 INFO:teuthology.orchestra.run.smithi029.stdout:max_file_size 1099511627776 2024-03-16T11:50:32.735 INFO:teuthology.orchestra.run.smithi029.stdout:max_xattr_size 65536 2024-03-16T11:50:32.735 INFO:teuthology.orchestra.run.smithi029.stdout:required_client_features {} 2024-03-16T11:50:32.736 INFO:teuthology.orchestra.run.smithi029.stdout:last_failure 0 2024-03-16T11:50:32.736 INFO:teuthology.orchestra.run.smithi029.stdout:last_failure_osd_epoch 0 2024-03-16T11:50:32.736 INFO:teuthology.orchestra.run.smithi029.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-03-16T11:50:32.736 INFO:teuthology.orchestra.run.smithi029.stdout:max_mds 3 2024-03-16T11:50:32.736 INFO:teuthology.orchestra.run.smithi029.stdout:in 0,1,2 2024-03-16T11:50:32.736 INFO:teuthology.orchestra.run.smithi029.stdout:up {0=24503,1=24499,2=24464} 2024-03-16T11:50:32.736 INFO:teuthology.orchestra.run.smithi029.stdout:failed 2024-03-16T11:50:32.736 INFO:teuthology.orchestra.run.smithi029.stdout:damaged 2024-03-16T11:50:32.736 INFO:teuthology.orchestra.run.smithi029.stdout:stopped 2024-03-16T11:50:32.736 INFO:teuthology.orchestra.run.smithi029.stdout:data_pools [3] 2024-03-16T11:50:32.736 INFO:teuthology.orchestra.run.smithi029.stdout:metadata_pool 2 2024-03-16T11:50:32.736 INFO:teuthology.orchestra.run.smithi029.stdout:inline_data disabled 2024-03-16T11:50:32.736 INFO:teuthology.orchestra.run.smithi029.stdout:balancer 2024-03-16T11:50:32.736 INFO:teuthology.orchestra.run.smithi029.stdout:bal_rank_mask -1 2024-03-16T11:50:32.736 INFO:teuthology.orchestra.run.smithi029.stdout:standby_count_wanted 0 2024-03-16T11:50:32.737 INFO:teuthology.orchestra.run.smithi029.stdout:qdb_cluster leader: 24503 members: 24503,24464,24499 2024-03-16T11:50:32.737 INFO:teuthology.orchestra.run.smithi029.stdout:[mds.b{0:24503} state up:active seq 3 addr [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] compat {c=[1],r=[1],i=[1fff]}] 2024-03-16T11:50:32.737 INFO:teuthology.orchestra.run.smithi029.stdout:[mds.c{0:24478} state up:standby-replay seq 1 addr [v2:172.21.15.154:6836/1468560096,v1:172.21.15.154:6837/1468560096] compat {c=[1],r=[1],i=[1fff]}] 2024-03-16T11:50:32.737 INFO:teuthology.orchestra.run.smithi029.stdout:[mds.i{1:24499} state up:active seq 6 addr [v2:172.21.15.154:6838/685736862,v1:172.21.15.154:6839/685736862] compat {c=[1],r=[1],i=[1fff]}] 2024-03-16T11:50:32.737 INFO:teuthology.orchestra.run.smithi029.stdout:[mds.f{1:24466} state up:standby-replay seq 1 addr [v2:172.21.15.154:6834/4165274681,v1:172.21.15.154:6835/4165274681] compat {c=[1],r=[1],i=[1fff]}] 2024-03-16T11:50:32.737 INFO:teuthology.orchestra.run.smithi029.stdout:[mds.h{2:24464} state up:active seq 7 addr [v2:172.21.15.152:6836/1323866722,v1:172.21.15.152:6837/1323866722] compat {c=[1],r=[1],i=[1fff]}] 2024-03-16T11:50:32.737 INFO:teuthology.orchestra.run.smithi029.stdout:[mds.e{2:24434} state up:standby-replay seq 1 addr [v2:172.21.15.152:6834/2919113178,v1:172.21.15.152:6835/2919113178] compat {c=[1],r=[1],i=[1fff]}] 2024-03-16T11:50:32.737 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-16T11:50:32.737 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-16T11:50:32.737 INFO:teuthology.orchestra.run.smithi029.stdout:Standby daemons: 2024-03-16T11:50:32.737 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-16T11:50:32.737 INFO:teuthology.orchestra.run.smithi029.stdout:[mds.j{-1:14574} state up:standby seq 1 addr [v2:172.21.15.29:6834/3505811955,v1:172.21.15.29:6835/3505811955] compat {c=[1],r=[1],i=[1fff]}] 2024-03-16T11:50:32.737 INFO:teuthology.orchestra.run.smithi029.stdout:[mds.d{-1:14586} state up:standby seq 1 addr [v2:172.21.15.29:6836/1901337868,v1:172.21.15.29:6837/1901337868] compat {c=[1],r=[1],i=[1fff]}] 2024-03-16T11:50:32.737 INFO:teuthology.orchestra.run.smithi029.stdout:[mds.g{-1:14652} state up:standby seq 1 addr [v2:172.21.15.29:6838/544966534,v1:172.21.15.29:6839/544966534] compat {c=[1],r=[1],i=[1fff]}] 2024-03-16T11:50:32.738 INFO:teuthology.orchestra.run.smithi029.stdout:[mds.a{-1:14664} state up:standby seq 1 addr [v2:172.21.15.29:6840/540705290,v1:172.21.15.29:6841/540705290] compat {c=[1],r=[1],i=[1fff]}] 2024-03-16T11:50:32.738 INFO:teuthology.orchestra.run.smithi029.stdout:[mds.k{-1:24407} state up:standby seq 1 addr [v2:172.21.15.152:6832/3136024628,v1:172.21.15.152:6833/3136024628] compat {c=[1],r=[1],i=[1fff]}] 2024-03-16T11:50:32.738 INFO:teuthology.orchestra.run.smithi029.stdout:[mds.l{-1:24421} state up:standby seq 1 addr [v2:172.21.15.154:6832/2677426282,v1:172.21.15.154:6833/2677426282] compat {c=[1],r=[1],i=[1fff]}] 2024-03-16T11:50:32.738 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:32.737+0000 7f7a780f8700 1 -- 172.21.15.29:0/3284245376 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f7a5c0a2910 msgr2=0x7f7a5c0a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:32.738 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:32.737+0000 7f7a780f8700 1 --2- 172.21.15.29:0/3284245376 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f7a5c0a2910 0x7f7a5c0a4dd0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f7a701022d0 tx=0x7f7a6c009260 comp rx=0 tx=0).stop 2024-03-16T11:50:32.739 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:32.737+0000 7f7a780f8700 1 -- 172.21.15.29:0/3284245376 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7a70102e10 msgr2=0x7f7a7019fab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:32.739 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:32.737+0000 7f7a780f8700 1 --2- 172.21.15.29:0/3284245376 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7a70102e10 0x7f7a7019fab0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f7a6800ef60 tx=0x7f7a6800ef90 comp rx=0 tx=0).stop 2024-03-16T11:50:32.739 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:32.737+0000 7f7a780f8700 1 -- 172.21.15.29:0/3284245376 shutdown_connections 2024-03-16T11:50:32.739 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:32.737+0000 7f7a780f8700 1 --2- 172.21.15.29:0/3284245376 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7a70102e10 0x7f7a7019fab0 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:32.739 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:32.737+0000 7f7a780f8700 1 --2- 172.21.15.29:0/3284245376 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7a70101270 0x7f7a701984f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:32.739 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:32.737+0000 7f7a780f8700 1 --2- 172.21.15.29:0/3284245376 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f7a5c0a2910 0x7f7a5c0a4dd0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:32.739 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:32.737+0000 7f7a780f8700 1 --2- 172.21.15.29:0/3284245376 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7a70102470 0x7f7a70198a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:32.739 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:32.737+0000 7f7a780f8700 1 -- 172.21.15.29:0/3284245376 >> 172.21.15.29:0/3284245376 conn(0x7f7a700fc830 msgr2=0x7f7a700fe2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:50:32.739 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:32.737+0000 7f7a780f8700 1 -- 172.21.15.29:0/3284245376 shutdown_connections 2024-03-16T11:50:32.739 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:32.737+0000 7f7a780f8700 1 -- 172.21.15.29:0/3284245376 wait complete. 2024-03-16T11:50:32.739 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 21 2024-03-16T11:50:33.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:50:32 smithi154 ceph-mon[36320]: from='client.? 172.21.15.29:0/3284245376' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-16T11:50:33.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:50:32 smithi029 ceph-mon[31975]: from='client.? 172.21.15.29:0/3284245376' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-16T11:50:33.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:50:32 smithi152 ceph-mon[36291]: from='client.? 172.21.15.29:0/3284245376' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-16T11:50:33.487 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d1887f28-e389-11ee-95c9-87774f69a715 -- bash -c 'ceph osd dump' 2024-03-16T11:50:33.732 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/d1887f28-e389-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T11:50:34.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:50:33 smithi154 ceph-mon[36320]: pgmap v350: 97 pgs: 97 active+clean; 640 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-16T11:50:34.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:50:33 smithi029 ceph-mon[31975]: pgmap v350: 97 pgs: 97 active+clean; 640 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-16T11:50:34.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:50:33 smithi152 ceph-mon[36291]: pgmap v350: 97 pgs: 97 active+clean; 640 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-16T11:50:35.703 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:35.696+0000 7f6166f05700 1 -- 172.21.15.29:0/2152603416 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6160102400 msgr2=0x7f6160102860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:35.703 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:35.696+0000 7f6166f05700 1 --2- 172.21.15.29:0/2152603416 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6160102400 0x7f6160102860 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f614c0099c0 tx=0x7f614c009cd0 comp rx=0 tx=0).stop 2024-03-16T11:50:35.703 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:35.696+0000 7f6166f05700 1 -- 172.21.15.29:0/2152603416 shutdown_connections 2024-03-16T11:50:35.703 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:35.696+0000 7f6166f05700 1 --2- 172.21.15.29:0/2152603416 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6160102da0 0x7f6160109650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:35.703 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:35.696+0000 7f6166f05700 1 --2- 172.21.15.29:0/2152603416 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6160101200 0x7f6160101620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:35.703 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:35.696+0000 7f6166f05700 1 --2- 172.21.15.29:0/2152603416 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6160102400 0x7f6160102860 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:35.703 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:35.696+0000 7f6166f05700 1 -- 172.21.15.29:0/2152603416 >> 172.21.15.29:0/2152603416 conn(0x7f61600fc7a0 msgr2=0x7f61600febe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:50:35.703 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:35.696+0000 7f6166f05700 1 -- 172.21.15.29:0/2152603416 shutdown_connections 2024-03-16T11:50:35.703 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:35.696+0000 7f6166f05700 1 -- 172.21.15.29:0/2152603416 wait complete. 2024-03-16T11:50:35.704 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:35.697+0000 7f6166f05700 1 Processor -- start 2024-03-16T11:50:35.704 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:35.697+0000 7f6166f05700 1 -- start start 2024-03-16T11:50:35.704 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:35.697+0000 7f6166f05700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6160101200 0x7f6160198530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:35.704 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:35.697+0000 7f6166f05700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6160102400 0x7f6160198a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:35.704 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:35.697+0000 7f6166f05700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6160102da0 0x7f616019faf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:35.704 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:35.697+0000 7f6166f05700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f616010ce20 con 0x7f6160102da0 2024-03-16T11:50:35.704 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:35.697+0000 7f6166f05700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f616010cca0 con 0x7f6160102400 2024-03-16T11:50:35.704 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:35.697+0000 7f6166f05700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f616010cfa0 con 0x7f6160101200 2024-03-16T11:50:35.704 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:35.697+0000 7f61654a2700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6160102da0 0x7f616019faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:35.704 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:35.697+0000 7f61654a2700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6160102da0 0x7f616019faf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59306/0 (socket says 172.21.15.29:59306) 2024-03-16T11:50:35.704 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:35.697+0000 7f61654a2700 1 -- 172.21.15.29:0/319450910 learned_addr learned my addr 172.21.15.29:0/319450910 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:50:35.704 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:35.697+0000 7f615ffff700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6160102400 0x7f6160198a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:35.704 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:35.697+0000 7f6164ca1700 1 --2- 172.21.15.29:0/319450910 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6160101200 0x7f6160198530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:35.704 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:35.698+0000 7f61654a2700 1 -- 172.21.15.29:0/319450910 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6160101200 msgr2=0x7f6160198530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:35.704 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:35.698+0000 7f61654a2700 1 --2- 172.21.15.29:0/319450910 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6160101200 0x7f6160198530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:35.704 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:35.698+0000 7f61654a2700 1 -- 172.21.15.29:0/319450910 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6160102400 msgr2=0x7f6160198a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:35.705 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:35.698+0000 7f61654a2700 1 --2- 172.21.15.29:0/319450910 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6160102400 0x7f6160198a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:35.705 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:35.698+0000 7f61654a2700 1 -- 172.21.15.29:0/319450910 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6160103620 con 0x7f6160102da0 2024-03-16T11:50:35.705 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:35.698+0000 7f615ffff700 1 --2- 172.21.15.29:0/319450910 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6160102400 0x7f6160198a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:50:35.705 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:35.698+0000 7f61654a2700 1 --2- 172.21.15.29:0/319450910 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6160102da0 0x7f616019faf0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f615800bed0 tx=0x7f6158016330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:50:35.705 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:35.700+0000 7f615dffb700 1 -- 172.21.15.29:0/319450910 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61580092e0 con 0x7f6160102da0 2024-03-16T11:50:35.705 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:35.700+0000 7f6166f05700 1 -- 172.21.15.29:0/319450910 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6160103910 con 0x7f6160102da0 2024-03-16T11:50:35.705 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:35.700+0000 7f615dffb700 1 -- 172.21.15.29:0/319450910 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6158016d30 con 0x7f6160102da0 2024-03-16T11:50:35.705 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:35.700+0000 7f615dffb700 1 -- 172.21.15.29:0/319450910 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6158007a40 con 0x7f6160102da0 2024-03-16T11:50:35.705 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:35.700+0000 7f6166f05700 1 -- 172.21.15.29:0/319450910 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6160103e50 con 0x7f6160102da0 2024-03-16T11:50:35.705 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:35.701+0000 7f6166f05700 1 -- 172.21.15.29:0/319450910 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6160068f90 con 0x7f6160102da0 2024-03-16T11:50:35.705 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:35.702+0000 7f615dffb700 1 -- 172.21.15.29:0/319450910 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f6158004ad0 con 0x7f6160102da0 2024-03-16T11:50:35.705 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:35.703+0000 7f615dffb700 1 --2- 172.21.15.29:0/319450910 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f61500a2910 0x7f61500a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:35.705 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:35.703+0000 7f6164ca1700 1 --2- 172.21.15.29:0/319450910 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f61500a2910 0x7f61500a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:35.705 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:35.703+0000 7f615dffb700 1 -- 172.21.15.29:0/319450910 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8211+0+0 (secure 0 0 0) 0x7f61580ce4c0 con 0x7f6160102da0 2024-03-16T11:50:35.705 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:35.703+0000 7f6164ca1700 1 --2- 172.21.15.29:0/319450910 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f61500a2910 0x7f61500a4dd0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f6160102260 tx=0x7f6154009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:50:35.706 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:35.707+0000 7f615dffb700 1 -- 172.21.15.29:0/319450910 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f615801a020 con 0x7f6160102da0 2024-03-16T11:50:35.914 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:35.915+0000 7f6166f05700 1 -- 172.21.15.29:0/319450910 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump"} v 0) -- 0x7f61601047f0 con 0x7f6160102da0 2024-03-16T11:50:35.915 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:35.916+0000 7f615dffb700 1 -- 172.21.15.29:0/319450910 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump"}]=0 v87) ==== 56+0+5291 (secure 0 0 0) 0x7f615809b510 con 0x7f6160102da0 2024-03-16T11:50:35.915 INFO:teuthology.orchestra.run.smithi029.stdout:epoch 87 2024-03-16T11:50:35.915 INFO:teuthology.orchestra.run.smithi029.stdout:fsid d1887f28-e389-11ee-95c9-87774f69a715 2024-03-16T11:50:35.916 INFO:teuthology.orchestra.run.smithi029.stdout:created 2024-03-16T11:40:16.475063+0000 2024-03-16T11:50:35.916 INFO:teuthology.orchestra.run.smithi029.stdout:modified 2024-03-16T11:49:28.628389+0000 2024-03-16T11:50:35.916 INFO:teuthology.orchestra.run.smithi029.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-03-16T11:50:35.916 INFO:teuthology.orchestra.run.smithi029.stdout:crush_version 26 2024-03-16T11:50:35.916 INFO:teuthology.orchestra.run.smithi029.stdout:full_ratio 0.95 2024-03-16T11:50:35.916 INFO:teuthology.orchestra.run.smithi029.stdout:backfillfull_ratio 0.9 2024-03-16T11:50:35.916 INFO:teuthology.orchestra.run.smithi029.stdout:nearfull_ratio 0.85 2024-03-16T11:50:35.916 INFO:teuthology.orchestra.run.smithi029.stdout:require_min_compat_client luminous 2024-03-16T11:50:35.916 INFO:teuthology.orchestra.run.smithi029.stdout:min_compat_client luminous 2024-03-16T11:50:35.916 INFO:teuthology.orchestra.run.smithi029.stdout:require_osd_release squid 2024-03-16T11:50:35.916 INFO:teuthology.orchestra.run.smithi029.stdout:stretch_mode_enabled false 2024-03-16T11:50:35.916 INFO:teuthology.orchestra.run.smithi029.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 21 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 12.00 2024-03-16T11:50:35.916 INFO:teuthology.orchestra.run.smithi029.stdout:pool 2 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 84 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 1.88 2024-03-16T11:50:35.916 INFO:teuthology.orchestra.run.smithi029.stdout:pool 3 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode off last_change 85 flags hashpspool stripe_width 0 pg_num_min 64 target_size_ratio 0.9 application cephfs read_balance_score 1.50 2024-03-16T11:50:35.917 INFO:teuthology.orchestra.run.smithi029.stdout:max_osd 12 2024-03-16T11:50:35.917 INFO:teuthology.orchestra.run.smithi029.stdout:osd.0 up in weight 1 up_from 8 up_thru 86 down_at 0 last_clean_interval [0,0) [v2:172.21.15.29:6802/2748947913,v1:172.21.15.29:6803/2748947913] [v2:172.21.15.29:6804/2748947913,v1:172.21.15.29:6805/2748947913] exists,up 60f5608f-a408-45ba-bdf6-5b53758f88fc 2024-03-16T11:50:35.917 INFO:teuthology.orchestra.run.smithi029.stdout:osd.1 up in weight 1 up_from 13 up_thru 83 down_at 0 last_clean_interval [0,0) [v2:172.21.15.152:6800/2339408422,v1:172.21.15.152:6801/2339408422] [v2:172.21.15.152:6802/2339408422,v1:172.21.15.152:6803/2339408422] exists,up e70b53d6-ee78-4e56-9a2c-263ae0501b43 2024-03-16T11:50:35.917 INFO:teuthology.orchestra.run.smithi029.stdout:osd.2 up in weight 1 up_from 18 up_thru 83 down_at 0 last_clean_interval [0,0) [v2:172.21.15.154:6800/4288325280,v1:172.21.15.154:6801/4288325280] [v2:172.21.15.154:6802/4288325280,v1:172.21.15.154:6803/4288325280] exists,up ff39389a-dfac-44b5-80ff-36abca316e78 2024-03-16T11:50:35.917 INFO:teuthology.orchestra.run.smithi029.stdout:osd.3 up in weight 1 up_from 25 up_thru 83 down_at 0 last_clean_interval [0,0) [v2:172.21.15.29:6810/1538750397,v1:172.21.15.29:6811/1538750397] [v2:172.21.15.29:6812/1538750397,v1:172.21.15.29:6813/1538750397] exists,up 069e1795-55bc-4324-b94c-eb24ef2334c0 2024-03-16T11:50:35.917 INFO:teuthology.orchestra.run.smithi029.stdout:osd.4 up in weight 1 up_from 33 up_thru 83 down_at 0 last_clean_interval [0,0) [v2:172.21.15.152:6808/2795081750,v1:172.21.15.152:6809/2795081750] [v2:172.21.15.152:6810/2795081750,v1:172.21.15.152:6811/2795081750] exists,up 5a9d43a1-e8f5-4ab5-a441-afbc6ed4066e 2024-03-16T11:50:35.917 INFO:teuthology.orchestra.run.smithi029.stdout:osd.5 up in weight 1 up_from 38 up_thru 86 down_at 0 last_clean_interval [0,0) [v2:172.21.15.154:6808/1472686560,v1:172.21.15.154:6809/1472686560] [v2:172.21.15.154:6810/1472686560,v1:172.21.15.154:6811/1472686560] exists,up 234de428-4ac7-44be-9a54-7b58331fdf8c 2024-03-16T11:50:35.917 INFO:teuthology.orchestra.run.smithi029.stdout:osd.6 up in weight 1 up_from 46 up_thru 86 down_at 0 last_clean_interval [0,0) [v2:172.21.15.29:6818/816506244,v1:172.21.15.29:6819/816506244] [v2:172.21.15.29:6820/816506244,v1:172.21.15.29:6821/816506244] exists,up ae45b8d9-7883-40c9-a9cd-4c92b601d4f2 2024-03-16T11:50:35.917 INFO:teuthology.orchestra.run.smithi029.stdout:osd.7 up in weight 1 up_from 51 up_thru 86 down_at 0 last_clean_interval [0,0) [v2:172.21.15.152:6816/2924651787,v1:172.21.15.152:6817/2924651787] [v2:172.21.15.152:6818/2924651787,v1:172.21.15.152:6819/2924651787] exists,up 6f687b28-4fce-4d4a-8b4e-294173d9d6c5 2024-03-16T11:50:35.917 INFO:teuthology.orchestra.run.smithi029.stdout:osd.8 up in weight 1 up_from 59 up_thru 83 down_at 0 last_clean_interval [0,0) [v2:172.21.15.154:6816/129974156,v1:172.21.15.154:6817/129974156] [v2:172.21.15.154:6818/129974156,v1:172.21.15.154:6819/129974156] exists,up a25fb741-caa7-429a-b540-e394d034b446 2024-03-16T11:50:35.917 INFO:teuthology.orchestra.run.smithi029.stdout:osd.9 up in weight 1 up_from 67 up_thru 83 down_at 0 last_clean_interval [0,0) [v2:172.21.15.29:6826/2951414243,v1:172.21.15.29:6827/2951414243] [v2:172.21.15.29:6828/2951414243,v1:172.21.15.29:6829/2951414243] exists,up 1071cb0e-8c5f-4cd8-9805-0045e220d946 2024-03-16T11:50:35.917 INFO:teuthology.orchestra.run.smithi029.stdout:osd.10 up in weight 1 up_from 72 up_thru 86 down_at 0 last_clean_interval [0,0) [v2:172.21.15.152:6824/3412525480,v1:172.21.15.152:6825/3412525480] [v2:172.21.15.152:6826/3412525480,v1:172.21.15.152:6827/3412525480] exists,up ee7c7d74-3f39-4cd4-be80-843fb1dd9ecc 2024-03-16T11:50:35.917 INFO:teuthology.orchestra.run.smithi029.stdout:osd.11 up in weight 1 up_from 80 up_thru 86 down_at 0 last_clean_interval [0,0) [v2:172.21.15.154:6824/1160927966,v1:172.21.15.154:6825/1160927966] [v2:172.21.15.154:6826/1160927966,v1:172.21.15.154:6827/1160927966] exists,up 4598c803-984a-4de6-ac22-066502c17889 2024-03-16T11:50:35.917 INFO:teuthology.orchestra.run.smithi029.stdout:pg_upmap_items 2.a [2,11] 2024-03-16T11:50:35.918 INFO:teuthology.orchestra.run.smithi029.stdout:pg_upmap_items 2.14 [2,4] 2024-03-16T11:50:35.918 INFO:teuthology.orchestra.run.smithi029.stdout:pg_upmap_items 2.1b [2,1] 2024-03-16T11:50:35.918 INFO:teuthology.orchestra.run.smithi029.stdout:pg_upmap_items 2.1d [8,11] 2024-03-16T11:50:35.918 INFO:teuthology.orchestra.run.smithi029.stdout:pg_upmap_items 3.13 [4,9] 2024-03-16T11:50:35.918 INFO:teuthology.orchestra.run.smithi029.stdout:pg_upmap_items 3.17 [4,8] 2024-03-16T11:50:35.918 INFO:teuthology.orchestra.run.smithi029.stdout:pg_upmap_items 3.2c [4,9] 2024-03-16T11:50:35.918 INFO:teuthology.orchestra.run.smithi029.stdout:pg_upmap_items 3.33 [5,9] 2024-03-16T11:50:35.918 INFO:teuthology.orchestra.run.smithi029.stdout:blocklist 172.21.15.29:0/1853086886 expires 2024-03-17T11:41:27.531724+0000 2024-03-16T11:50:35.918 INFO:teuthology.orchestra.run.smithi029.stdout:blocklist 172.21.15.29:0/1211659364 expires 2024-03-17T11:41:27.531724+0000 2024-03-16T11:50:35.918 INFO:teuthology.orchestra.run.smithi029.stdout:blocklist 172.21.15.29:6800/2798831128 expires 2024-03-17T11:41:27.531724+0000 2024-03-16T11:50:35.918 INFO:teuthology.orchestra.run.smithi029.stdout:blocklist 172.21.15.29:0/1281028285 expires 2024-03-17T11:41:00.505678+0000 2024-03-16T11:50:35.918 INFO:teuthology.orchestra.run.smithi029.stdout:blocklist 172.21.15.29:6800/2275315654 expires 2024-03-17T11:41:00.505678+0000 2024-03-16T11:50:35.918 INFO:teuthology.orchestra.run.smithi029.stdout:blocklist 172.21.15.29:6801/2798831128 expires 2024-03-17T11:41:27.531724+0000 2024-03-16T11:50:35.918 INFO:teuthology.orchestra.run.smithi029.stdout:blocklist 172.21.15.29:6801/2275315654 expires 2024-03-17T11:41:00.505678+0000 2024-03-16T11:50:35.918 INFO:teuthology.orchestra.run.smithi029.stdout:blocklist 172.21.15.29:0/1330100009 expires 2024-03-17T11:41:00.505678+0000 2024-03-16T11:50:35.919 INFO:teuthology.orchestra.run.smithi029.stdout:blocklist 172.21.15.29:0/1784249039 expires 2024-03-17T11:41:00.505678+0000 2024-03-16T11:50:35.919 INFO:teuthology.orchestra.run.smithi029.stdout:blocklist 172.21.15.29:0/1116486932 expires 2024-03-17T11:41:27.531724+0000 2024-03-16T11:50:35.919 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:35.918+0000 7f6166f05700 1 -- 172.21.15.29:0/319450910 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f61500a2910 msgr2=0x7f61500a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:35.919 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:35.918+0000 7f6166f05700 1 --2- 172.21.15.29:0/319450910 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f61500a2910 0x7f61500a4dd0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f6160102260 tx=0x7f6154009260 comp rx=0 tx=0).stop 2024-03-16T11:50:35.919 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:35.918+0000 7f6166f05700 1 -- 172.21.15.29:0/319450910 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6160102da0 msgr2=0x7f616019faf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:35.919 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:35.918+0000 7f6166f05700 1 --2- 172.21.15.29:0/319450910 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6160102da0 0x7f616019faf0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f615800bed0 tx=0x7f6158016330 comp rx=0 tx=0).stop 2024-03-16T11:50:35.919 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:35.918+0000 7f6166f05700 1 -- 172.21.15.29:0/319450910 shutdown_connections 2024-03-16T11:50:35.920 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:35.918+0000 7f6166f05700 1 --2- 172.21.15.29:0/319450910 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6160102da0 0x7f616019faf0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:35.920 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:35.918+0000 7f6166f05700 1 --2- 172.21.15.29:0/319450910 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f61500a2910 0x7f61500a4dd0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:35.920 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:35.918+0000 7f6166f05700 1 --2- 172.21.15.29:0/319450910 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6160101200 0x7f6160198530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:35.920 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:35.918+0000 7f6166f05700 1 --2- 172.21.15.29:0/319450910 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6160102400 0x7f6160198a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:35.920 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:35.918+0000 7f6166f05700 1 -- 172.21.15.29:0/319450910 >> 172.21.15.29:0/319450910 conn(0x7f61600fc7a0 msgr2=0x7f61600fe220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:50:35.920 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:35.918+0000 7f6166f05700 1 -- 172.21.15.29:0/319450910 shutdown_connections 2024-03-16T11:50:35.920 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:35.918+0000 7f6166f05700 1 -- 172.21.15.29:0/319450910 wait complete. 2024-03-16T11:50:36.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:50:35 smithi154 ceph-mon[36320]: pgmap v351: 97 pgs: 97 active+clean; 640 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-16T11:50:36.087 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:50:35 smithi029 ceph-mon[31975]: pgmap v351: 97 pgs: 97 active+clean; 640 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-16T11:50:36.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:50:35 smithi152 ceph-mon[36291]: pgmap v351: 97 pgs: 97 active+clean; 640 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-16T11:50:36.600 INFO:teuthology.run_tasks:Running task fs.ready... 2024-03-16T11:50:36.613 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-16T11:50:36.725 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:36.723+0000 7f1f79aff640 1 -- 172.21.15.29:0/1046157877 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1f741022e0 msgr2=0x7f1f741046d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:36.725 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:36.723+0000 7f1f79aff640 1 --2- 172.21.15.29:0/1046157877 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1f741022e0 0x7f1f741046d0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f1f64009900 tx=0x7f1f6402bc30 comp rx=0 tx=0).stop 2024-03-16T11:50:36.725 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:36.723+0000 7f1f79aff640 1 -- 172.21.15.29:0/1046157877 shutdown_connections 2024-03-16T11:50:36.725 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:36.723+0000 7f1f79aff640 1 --2- 172.21.15.29:0/1046157877 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1f74107540 0x7f1f74109930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:36.725 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:36.723+0000 7f1f79aff640 1 --2- 172.21.15.29:0/1046157877 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1f74104c10 0x7f1f74107000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:36.725 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:36.723+0000 7f1f79aff640 1 --2- 172.21.15.29:0/1046157877 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1f741022e0 0x7f1f741046d0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:36.725 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:36.723+0000 7f1f79aff640 1 -- 172.21.15.29:0/1046157877 >> 172.21.15.29:0/1046157877 conn(0x7f1f740fbc70 msgr2=0x7f1f740fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:50:36.725 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:36.723+0000 7f1f79aff640 1 -- 172.21.15.29:0/1046157877 shutdown_connections 2024-03-16T11:50:36.725 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:36.723+0000 7f1f79aff640 1 -- 172.21.15.29:0/1046157877 wait complete. 2024-03-16T11:50:36.725 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:36.724+0000 7f1f79aff640 1 Processor -- start 2024-03-16T11:50:36.725 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:36.724+0000 7f1f79aff640 1 -- start start 2024-03-16T11:50:36.725 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:36.724+0000 7f1f79aff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1f741022e0 0x7f1f74197b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:36.726 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:36.724+0000 7f1f79aff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1f74104c10 0x7f1f74198080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:36.726 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:36.724+0000 7f1f79aff640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1f74107540 0x7f1f7419f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:36.726 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:36.724+0000 7f1f79aff640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f1f7410d4a0 con 0x7f1f74104c10 2024-03-16T11:50:36.726 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:36.724+0000 7f1f79aff640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f1f7410d320 con 0x7f1f74107540 2024-03-16T11:50:36.726 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:36.724+0000 7f1f79aff640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f1f7410d620 con 0x7f1f741022e0 2024-03-16T11:50:36.726 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:36.724+0000 7f1f72ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1f74104c10 0x7f1f74198080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:36.726 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:36.724+0000 7f1f72ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1f74104c10 0x7f1f74198080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59318/0 (socket says 172.21.15.29:59318) 2024-03-16T11:50:36.726 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:36.724+0000 7f1f72ffd640 1 -- 172.21.15.29:0/2542244855 learned_addr learned my addr 172.21.15.29:0/2542244855 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:50:36.726 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:36.724+0000 7f1f737fe640 1 --2- 172.21.15.29:0/2542244855 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1f741022e0 0x7f1f74197b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:36.726 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:36.725+0000 7f1f73fff640 1 --2- 172.21.15.29:0/2542244855 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1f74107540 0x7f1f7419f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:36.726 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:36.725+0000 7f1f72ffd640 1 -- 172.21.15.29:0/2542244855 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1f741022e0 msgr2=0x7f1f74197b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:36.726 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:36.725+0000 7f1f72ffd640 1 --2- 172.21.15.29:0/2542244855 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1f741022e0 0x7f1f74197b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:36.726 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:36.725+0000 7f1f72ffd640 1 -- 172.21.15.29:0/2542244855 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1f74107540 msgr2=0x7f1f7419f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:36.727 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:36.725+0000 7f1f72ffd640 1 --2- 172.21.15.29:0/2542244855 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1f74107540 0x7f1f7419f0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:36.727 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:36.725+0000 7f1f72ffd640 1 -- 172.21.15.29:0/2542244855 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f7419f760 con 0x7f1f74104c10 2024-03-16T11:50:36.727 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:36.725+0000 7f1f72ffd640 1 --2- 172.21.15.29:0/2542244855 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1f74104c10 0x7f1f74198080 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f1f6800e9c0 tx=0x7f1f6800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:50:36.727 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:36.725+0000 7f1f70ff9640 1 -- 172.21.15.29:0/2542244855 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f6800cd80 con 0x7f1f74104c10 2024-03-16T11:50:36.727 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:36.725+0000 7f1f79aff640 1 -- 172.21.15.29:0/2542244855 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1f7419fa50 con 0x7f1f74104c10 2024-03-16T11:50:36.727 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:36.725+0000 7f1f70ff9640 1 -- 172.21.15.29:0/2542244855 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1f68005500 con 0x7f1f74104c10 2024-03-16T11:50:36.727 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:36.725+0000 7f1f70ff9640 1 -- 172.21.15.29:0/2542244855 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f68010690 con 0x7f1f74104c10 2024-03-16T11:50:36.727 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:36.725+0000 7f1f79aff640 1 -- 172.21.15.29:0/2542244855 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1f7419fff0 con 0x7f1f74104c10 2024-03-16T11:50:36.728 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:36.726+0000 7f1f79aff640 1 -- 172.21.15.29:0/2542244855 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1f740fd760 con 0x7f1f74104c10 2024-03-16T11:50:36.728 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:36.727+0000 7f1f70ff9640 1 -- 172.21.15.29:0/2542244855 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f1f680046c0 con 0x7f1f74104c10 2024-03-16T11:50:36.728 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:36.728+0000 7f1f70ff9640 1 --2- 172.21.15.29:0/2542244855 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f1f5c0a2800 0x7f1f5c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:36.728 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:36.728+0000 7f1f737fe640 1 --2- 172.21.15.29:0/2542244855 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f1f5c0a2800 0x7f1f5c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:36.728 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:36.728+0000 7f1f70ff9640 1 -- 172.21.15.29:0/2542244855 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8211+0+0 (secure 0 0 0) 0x7f1f680cede0 con 0x7f1f74104c10 2024-03-16T11:50:36.728 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:36.728+0000 7f1f737fe640 1 --2- 172.21.15.29:0/2542244855 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f1f5c0a2800 0x7f1f5c0a4cc0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f1f6402c7b0 tx=0x7f1f64005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:50:36.731 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:36.732+0000 7f1f70ff9640 1 -- 172.21.15.29:0/2542244855 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f1f6809be90 con 0x7f1f74104c10 2024-03-16T11:50:36.837 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:50:36 smithi029 ceph-mon[31975]: from='client.? 172.21.15.29:0/319450910' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-16T11:50:36.915 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:36.916+0000 7f1f79aff640 1 -- 172.21.15.29:0/2542244855 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f1f740623e0 con 0x7f1f74104c10 2024-03-16T11:50:36.916 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:36.917+0000 7f1f70ff9640 1 -- 172.21.15.29:0/2542244855 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) ==== 74+0+22668 (secure 0 0 0) 0x7f1f6809b790 con 0x7f1f74104c10 2024-03-16T11:50:36.917 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-16T11:50:36.917 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":87,"fsid":"d1887f28-e389-11ee-95c9-87774f69a715","created":"2024-03-16T11:40:16.475063+0000","modified":"2024-03-16T11:49:28.628389+0000","last_up_change":"2024-03-16T11:47:44.349348+0000","last_in_change":"2024-03-16T11:47:23.672432+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-16T11:43:51.892652+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-16T11:48:36.709732+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-16T11:48:37.586140+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"60f5608f-a408-45ba-bdf6-5b53758f88fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":2748947913},{"type":"v1","addr":"172.21.15.29:6803","nonce":2748947913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":2748947913},{"type":"v1","addr":"172.21.15.29:6805","nonce":2748947913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":2748947913},{"type":"v1","addr":"172.21.15.29:6809","nonce":2748947913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":2748947913},{"type":"v1","addr":"172.21.15.29:6807","nonce":2748947913}]},"public_addr":"172.21.15.29:6803/2748947913","cluster_addr":"172.21.15.29:6805/2748947913","heartbeat_back_addr":"172.21.15.29:6809/2748947913","heartbeat_front_addr":"172.21.15.29:6807/2748947913","state":["exists","up"]},{"osd":1,"uuid":"e70b53d6-ee78-4e56-9a2c-263ae0501b43","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6800","nonce":2339408422},{"type":"v1","addr":"172.21.15.152:6801","nonce":2339408422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6802","nonce":2339408422},{"type":"v1","addr":"172.21.15.152:6803","nonce":2339408422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6806","nonce":2339408422},{"type":"v1","addr":"172.21.15.152:6807","nonce":2339408422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6804","nonce":2339408422},{"type":"v1","addr":"172.21.15.152:6805","nonce":2339408422}]},"public_addr":"172.21.15.152:6801/2339408422","cluster_addr":"172.21.15.152:6803/2339408422","heartbeat_back_addr":"172.21.15.152:6807/2339408422","heartbeat_front_addr":"172.21.15.152:6805/2339408422","state":["exists","up"]},{"osd":2,"uuid":"ff39389a-dfac-44b5-80ff-36abca316e78","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6800","nonce":4288325280},{"type":"v1","addr":"172.21.15.154:6801","nonce":4288325280}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6802","nonce":4288325280},{"type":"v1","addr":"172.21.15.154:6803","nonce":4288325280}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6806","nonce":4288325280},{"type":"v1","addr":"172.21.15.154:6807","nonce":4288325280}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":4288325280},{"type":"v1","addr":"172.21.15.154:6805","nonce":4288325280}]},"public_addr":"172.21.15.154:6801/4288325280","cluster_addr":"172.21.15.154:6803/4288325280","heartbeat_back_addr":"172.21.15.154:6807/4288325280","heartbeat_front_addr":"172.21.15.154:6805/4288325280","state":["exists","up"]},{"osd":3,"uuid":"069e1795-55bc-4324-b94c-eb24ef2334c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":1538750397},{"type":"v1","addr":"172.21.15.29:6811","nonce":1538750397}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":1538750397},{"type":"v1","addr":"172.21.15.29:6813","nonce":1538750397}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":1538750397},{"type":"v1","addr":"172.21.15.29:6817","nonce":1538750397}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":1538750397},{"type":"v1","addr":"172.21.15.29:6815","nonce":1538750397}]},"public_addr":"172.21.15.29:6811/1538750397","cluster_addr":"172.21.15.29:6813/1538750397","heartbeat_back_addr":"172.21.15.29:6817/1538750397","heartbeat_front_addr":"172.21.15.29:6815/1538750397","state":["exists","up"]},{"osd":4,"uuid":"5a9d43a1-e8f5-4ab5-a441-afbc6ed4066e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6808","nonce":2795081750},{"type":"v1","addr":"172.21.15.152:6809","nonce":2795081750}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6810","nonce":2795081750},{"type":"v1","addr":"172.21.15.152:6811","nonce":2795081750}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6814","nonce":2795081750},{"type":"v1","addr":"172.21.15.152:6815","nonce":2795081750}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6812","nonce":2795081750},{"type":"v1","addr":"172.21.15.152:6813","nonce":2795081750}]},"public_addr":"172.21.15.152:6809/2795081750","cluster_addr":"172.21.15.152:6811/2795081750","heartbeat_back_addr":"172.21.15.152:6815/2795081750","heartbeat_front_addr":"172.21.15.152:6813/2795081750","state":["exists","up"]},{"osd":5,"uuid":"234de428-4ac7-44be-9a54-7b58331fdf8c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":1472686560},{"type":"v1","addr":"172.21.15.154:6809","nonce":1472686560}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6810","nonce":1472686560},{"type":"v1","addr":"172.21.15.154:6811","nonce":1472686560}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6814","nonce":1472686560},{"type":"v1","addr":"172.21.15.154:6815","nonce":1472686560}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6812","nonce":1472686560},{"type":"v1","addr":"172.21.15.154:6813","nonce":1472686560}]},"public_addr":"172.21.15.154:6809/1472686560","cluster_addr":"172.21.15.154:6811/1472686560","heartbeat_back_addr":"172.21.15.154:6815/1472686560","heartbeat_front_addr":"172.21.15.154:6813/1472686560","state":["exists","up"]},{"osd":6,"uuid":"ae45b8d9-7883-40c9-a9cd-4c92b601d4f2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":816506244},{"type":"v1","addr":"172.21.15.29:6819","nonce":816506244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":816506244},{"type":"v1","addr":"172.21.15.29:6821","nonce":816506244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":816506244},{"type":"v1","addr":"172.21.15.29:6825","nonce":816506244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":816506244},{"type":"v1","addr":"172.21.15.29:6823","nonce":816506244}]},"public_addr":"172.21.15.29:6819/816506244","cluster_addr":"172.21.15.29:6821/816506244","heartbeat_back_addr":"172.21.15.29:6825/816506244","heartbeat_front_addr":"172.21.15.29:6823/816506244","state":["exists","up"]},{"osd":7,"uuid":"6f687b28-4fce-4d4a-8b4e-294173d9d6c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6816","nonce":2924651787},{"type":"v1","addr":"172.21.15.152:6817","nonce":2924651787}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6818","nonce":2924651787},{"type":"v1","addr":"172.21.15.152:6819","nonce":2924651787}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6822","nonce":2924651787},{"type":"v1","addr":"172.21.15.152:6823","nonce":2924651787}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6820","nonce":2924651787},{"type":"v1","addr":"172.21.15.152:6821","nonce":2924651787}]},"public_addr":"172.21.15.152:6817/2924651787","cluster_addr":"172.21.15.152:6819/2924651787","heartbeat_back_addr":"172.21.15.152:6823/2924651787","heartbeat_front_addr":"172.21.15.152:6821/2924651787","state":["exists","up"]},{"osd":8,"uuid":"a25fb741-caa7-429a-b540-e394d034b446","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6816","nonce":129974156},{"type":"v1","addr":"172.21.15.154:6817","nonce":129974156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6818","nonce":129974156},{"type":"v1","addr":"172.21.15.154:6819","nonce":129974156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6822","nonce":129974156},{"type":"v1","addr":"172.21.15.154:6823","nonce":129974156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6820","nonce":129974156},{"type":"v1","addr":"172.21.15.154:6821","nonce":129974156}]},"public_addr":"172.21.15.154:6817/129974156","cluster_addr":"172.21.15.154:6819/129974156","heartbeat_back_addr":"172.21.15.154:6823/129974156","heartbeat_front_addr":"172.21.15.154:6821/129974156","state":["exists","up"]},{"osd":9,"uuid":"1071cb0e-8c5f-4cd8-9805-0045e220d946","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6826","nonce":2951414243},{"type":"v1","addr":"172.21.15.29:6827","nonce":2951414243}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":2951414243},{"type":"v1","addr":"172.21.15.29:6829","nonce":2951414243}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":2951414243},{"type":"v1","addr":"172.21.15.29:6833","nonce":2951414243}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":2951414243},{"type":"v1","addr":"172.21.15.29:6831","nonce":2951414243}]},"public_addr":"172.21.15.29:6827/2951414243","cluster_addr":"172.21.15.29:6829/2951414243","heartbeat_back_addr":"172.21.15.29:6833/2951414243","heartbeat_front_addr":"172.21.15.29:6831/2951414243","state":["exists","up"]},{"osd":10,"uuid":"ee7c7d74-3f39-4cd4-be80-843fb1dd9ecc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6824","nonce":3412525480},{"type":"v1","addr":"172.21.15.152:6825","nonce":3412525480}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6826","nonce":3412525480},{"type":"v1","addr":"172.21.15.152:6827","nonce":3412525480}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6830","nonce":3412525480},{"type":"v1","addr":"172.21.15.152:6831","nonce":3412525480}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6828","nonce":3412525480},{"type":"v1","addr":"172.21.15.152:6829","nonce":3412525480}]},"public_addr":"172.21.15.152:6825/3412525480","cluster_addr":"172.21.15.152:6827/3412525480","heartbeat_back_addr":"172.21.15.152:6831/3412525480","heartbeat_front_addr":"172.21.15.152:6829/3412525480","state":["exists","up"]},{"osd":11,"uuid":"4598c803-984a-4de6-ac22-066502c17889","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6824","nonce":1160927966},{"type":"v1","addr":"172.21.15.154:6825","nonce":1160927966}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6826","nonce":1160927966},{"type":"v1","addr":"172.21.15.154:6827","nonce":1160927966}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6830","nonce":1160927966},{"type":"v1","addr":"172.21.15.154:6831","nonce":1160927966}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6828","nonce":1160927966},{"type":"v1","addr":"172.21.15.154:6829","nonce":1160927966}]},"public_addr":"172.21.15.154:6825/1160927966","cluster_addr":"172.21.15.154:6827/1160927966","heartbeat_back_addr":"172.21.15.154:6831/1160927966","heartbeat_front_addr":"172.21.15.154:6829/1160927966","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:43:02.343393+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:43:25.209151+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:43:48.946164+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:44:14.588424+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:44:39.700025+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:45:04.061197+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:45:31.214508+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:45:56.937296+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:46:22.768915+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:46:50.101011+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:47:16.298041+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:47:42.402095+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"2.1b","mappings":[{"from":2,"to":1}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"3.33","mappings":[{"from":5,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.29:0/1853086886":"2024-03-17T11:41:27.531724+0000","172.21.15.29:0/1211659364":"2024-03-17T11:41:27.531724+0000","172.21.15.29:6800/2798831128":"2024-03-17T11:41:27.531724+0000","172.21.15.29:0/1281028285":"2024-03-17T11:41:00.505678+0000","172.21.15.29:6800/2275315654":"2024-03-17T11:41:00.505678+0000","172.21.15.29:6801/2798831128":"2024-03-17T11:41:27.531724+0000","172.21.15.29:6801/2275315654":"2024-03-17T11:41:00.505678+0000","172.21.15.29:0/1330100009":"2024-03-17T11:41:00.505678+0000","172.21.15.29:0/1784249039":"2024-03-17T11:41:00.505678+0000","172.21.15.29:0/1116486932":"2024-03-17T11:41:27.531724+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-16T11:50:36.919 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:36.919+0000 7f1f79aff640 1 -- 172.21.15.29:0/2542244855 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f1f5c0a2800 msgr2=0x7f1f5c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:36.919 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:36.919+0000 7f1f79aff640 1 --2- 172.21.15.29:0/2542244855 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f1f5c0a2800 0x7f1f5c0a4cc0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f1f6402c7b0 tx=0x7f1f64005fd0 comp rx=0 tx=0).stop 2024-03-16T11:50:36.919 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:36.919+0000 7f1f79aff640 1 -- 172.21.15.29:0/2542244855 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1f74104c10 msgr2=0x7f1f74198080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:36.919 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:36.919+0000 7f1f79aff640 1 --2- 172.21.15.29:0/2542244855 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1f74104c10 0x7f1f74198080 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f1f6800e9c0 tx=0x7f1f6800ee80 comp rx=0 tx=0).stop 2024-03-16T11:50:36.919 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:36.919+0000 7f1f79aff640 1 -- 172.21.15.29:0/2542244855 shutdown_connections 2024-03-16T11:50:36.919 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:36.919+0000 7f1f79aff640 1 --2- 172.21.15.29:0/2542244855 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f1f5c0a2800 0x7f1f5c0a4cc0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:36.919 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:36.919+0000 7f1f79aff640 1 --2- 172.21.15.29:0/2542244855 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1f74107540 0x7f1f7419f0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:36.919 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:36.919+0000 7f1f79aff640 1 --2- 172.21.15.29:0/2542244855 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1f74104c10 0x7f1f74198080 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:36.919 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:36.919+0000 7f1f79aff640 1 --2- 172.21.15.29:0/2542244855 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1f741022e0 0x7f1f74197b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:36.919 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:36.919+0000 7f1f79aff640 1 -- 172.21.15.29:0/2542244855 >> 172.21.15.29:0/2542244855 conn(0x7f1f740fbc70 msgr2=0x7f1f74102b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:50:36.919 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:36.919+0000 7f1f79aff640 1 -- 172.21.15.29:0/2542244855 shutdown_connections 2024-03-16T11:50:36.919 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:36.919+0000 7f1f79aff640 1 -- 172.21.15.29:0/2542244855 wait complete. 2024-03-16T11:50:36.930 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-16T11:43:51.892652+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-16T11:48:36.709732+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-16T11:48:37.586140+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-16T11:50:36.930 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-16T11:50:37.038 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:37.037+0000 7fea0a7b9640 1 -- 172.21.15.29:0/48990485 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fea04100700 msgr2=0x7fea04100b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:37.039 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:37.037+0000 7fea0a7b9640 1 --2- 172.21.15.29:0/48990485 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fea04100700 0x7fea04100b20 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fe9ec009900 tx=0x7fe9ec02bc30 comp rx=0 tx=0).stop 2024-03-16T11:50:37.039 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:37.037+0000 7fea0a7b9640 1 -- 172.21.15.29:0/48990485 shutdown_connections 2024-03-16T11:50:37.039 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:37.037+0000 7fea0a7b9640 1 --2- 172.21.15.29:0/48990485 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fea041022b0 0x7fea04108b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:37.039 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:37.037+0000 7fea0a7b9640 1 --2- 172.21.15.29:0/48990485 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fea041018f0 0x7fea04101d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:37.039 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:37.037+0000 7fea0a7b9640 1 --2- 172.21.15.29:0/48990485 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fea04100700 0x7fea04100b20 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:37.039 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:37.037+0000 7fea0a7b9640 1 -- 172.21.15.29:0/48990485 >> 172.21.15.29:0/48990485 conn(0x7fea040fbc90 msgr2=0x7fea040fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:50:37.039 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:37.037+0000 7fea0a7b9640 1 -- 172.21.15.29:0/48990485 shutdown_connections 2024-03-16T11:50:37.039 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:37.037+0000 7fea0a7b9640 1 -- 172.21.15.29:0/48990485 wait complete. 2024-03-16T11:50:37.039 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:37.038+0000 7fea0a7b9640 1 Processor -- start 2024-03-16T11:50:37.039 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:37.038+0000 7fea0a7b9640 1 -- start start 2024-03-16T11:50:37.039 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:37.038+0000 7fea0a7b9640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fea04100700 0x7fea04104690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:37.039 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:37.038+0000 7fea0a7b9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fea041018f0 0x7fea04102cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:37.039 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:37.038+0000 7fea0a7b9640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fea041022b0 0x7fea04103210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:37.040 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:37.038+0000 7fea0a7b9640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fea0410c370 con 0x7fea041018f0 2024-03-16T11:50:37.040 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:37.038+0000 7fea0a7b9640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fea0410c1f0 con 0x7fea04100700 2024-03-16T11:50:37.040 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:37.038+0000 7fea0a7b9640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fea0410c4f0 con 0x7fea041022b0 2024-03-16T11:50:37.040 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:37.038+0000 7fea037fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fea041018f0 0x7fea04102cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:37.040 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:37.038+0000 7fea037fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fea041018f0 0x7fea04102cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59340/0 (socket says 172.21.15.29:59340) 2024-03-16T11:50:37.040 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:37.038+0000 7fea037fe640 1 -- 172.21.15.29:0/3536610773 learned_addr learned my addr 172.21.15.29:0/3536610773 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:50:37.040 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:37.038+0000 7fea03fff640 1 --2- 172.21.15.29:0/3536610773 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fea04100700 0x7fea04104690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:37.040 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:37.038+0000 7fea08d56640 1 --2- 172.21.15.29:0/3536610773 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fea041022b0 0x7fea04103210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:37.040 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:37.039+0000 7fea037fe640 1 -- 172.21.15.29:0/3536610773 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fea041022b0 msgr2=0x7fea04103210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:37.040 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:37.039+0000 7fea037fe640 1 --2- 172.21.15.29:0/3536610773 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fea041022b0 0x7fea04103210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:37.040 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:37.039+0000 7fea037fe640 1 -- 172.21.15.29:0/3536610773 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fea04100700 msgr2=0x7fea04104690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:37.040 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:37.039+0000 7fea037fe640 1 --2- 172.21.15.29:0/3536610773 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fea04100700 0x7fea04104690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:37.040 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:37.039+0000 7fea037fe640 1 -- 172.21.15.29:0/3536610773 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea04103aa0 con 0x7fea041018f0 2024-03-16T11:50:37.041 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:37.039+0000 7fea037fe640 1 --2- 172.21.15.29:0/3536610773 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fea041018f0 0x7fea04102cd0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fe9f400e9f0 tx=0x7fe9f400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:50:37.041 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:37.039+0000 7fea017fa640 1 -- 172.21.15.29:0/3536610773 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9f400cd80 con 0x7fea041018f0 2024-03-16T11:50:37.041 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:37.039+0000 7fea017fa640 1 -- 172.21.15.29:0/3536610773 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe9f4005500 con 0x7fea041018f0 2024-03-16T11:50:37.041 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:37.039+0000 7fea0a7b9640 1 -- 172.21.15.29:0/3536610773 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fea041a1cb0 con 0x7fea041018f0 2024-03-16T11:50:37.041 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:37.039+0000 7fea017fa640 1 -- 172.21.15.29:0/3536610773 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9f4010690 con 0x7fea041018f0 2024-03-16T11:50:37.041 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:37.039+0000 7fea0a7b9640 1 -- 172.21.15.29:0/3536610773 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fea041a21b0 con 0x7fea041018f0 2024-03-16T11:50:37.042 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:37.040+0000 7fea0a7b9640 1 -- 172.21.15.29:0/3536610773 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fea041a23e0 con 0x7fea041018f0 2024-03-16T11:50:37.042 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:37.041+0000 7fea017fa640 1 -- 172.21.15.29:0/3536610773 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fe9f40046c0 con 0x7fea041018f0 2024-03-16T11:50:37.042 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:37.041+0000 7fea017fa640 1 --2- 172.21.15.29:0/3536610773 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fe9f00a2800 0x7fe9f00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:37.042 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:37.042+0000 7fea03fff640 1 --2- 172.21.15.29:0/3536610773 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fe9f00a2800 0x7fe9f00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:37.042 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:37.042+0000 7fea017fa640 1 -- 172.21.15.29:0/3536610773 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8211+0+0 (secure 0 0 0) 0x7fe9f40cef30 con 0x7fea041018f0 2024-03-16T11:50:37.042 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:37.042+0000 7fea03fff640 1 --2- 172.21.15.29:0/3536610773 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fe9f00a2800 0x7fe9f00a4cc0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fe9ec000c00 tx=0x7fe9ec005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:50:37.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:50:36 smithi154 ceph-mon[36320]: from='client.? 172.21.15.29:0/319450910' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-16T11:50:37.045 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:37.046+0000 7fea017fa640 1 -- 172.21.15.29:0/3536610773 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fe9f409bf80 con 0x7fea041018f0 2024-03-16T11:50:37.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:50:36 smithi152 ceph-mon[36291]: from='client.? 172.21.15.29:0/319450910' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-16T11:50:37.228 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:37.229+0000 7fea0a7b9640 1 -- 172.21.15.29:0/3536610773 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fea041a2770 con 0x7fea041018f0 2024-03-16T11:50:37.228 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:37.229+0000 7fea017fa640 1 -- 172.21.15.29:0/3536610773 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v87) ==== 93+0+10 (secure 0 0 0) 0x7fe9f409b880 con 0x7fea041018f0 2024-03-16T11:50:37.228 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2024-03-16T11:50:37.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:37.231+0000 7fea0a7b9640 1 -- 172.21.15.29:0/3536610773 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fe9f00a2800 msgr2=0x7fe9f00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:37.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:37.231+0000 7fea0a7b9640 1 --2- 172.21.15.29:0/3536610773 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fe9f00a2800 0x7fe9f00a4cc0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fe9ec000c00 tx=0x7fe9ec005fd0 comp rx=0 tx=0).stop 2024-03-16T11:50:37.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:37.231+0000 7fea0a7b9640 1 -- 172.21.15.29:0/3536610773 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fea041018f0 msgr2=0x7fea04102cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:37.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:37.231+0000 7fea0a7b9640 1 --2- 172.21.15.29:0/3536610773 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fea041018f0 0x7fea04102cd0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fe9f400e9f0 tx=0x7fe9f400edf0 comp rx=0 tx=0).stop 2024-03-16T11:50:37.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:37.231+0000 7fea0a7b9640 1 -- 172.21.15.29:0/3536610773 shutdown_connections 2024-03-16T11:50:37.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:37.231+0000 7fea0a7b9640 1 --2- 172.21.15.29:0/3536610773 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fe9f00a2800 0x7fe9f00a4cc0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:37.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:37.231+0000 7fea0a7b9640 1 --2- 172.21.15.29:0/3536610773 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fea041022b0 0x7fea04103210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:37.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:37.231+0000 7fea0a7b9640 1 --2- 172.21.15.29:0/3536610773 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fea041018f0 0x7fea04102cd0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:37.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:37.231+0000 7fea0a7b9640 1 --2- 172.21.15.29:0/3536610773 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fea04100700 0x7fea04104690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:37.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:37.231+0000 7fea0a7b9640 1 -- 172.21.15.29:0/3536610773 >> 172.21.15.29:0/3536610773 conn(0x7fea040fbc90 msgr2=0x7fea040fd790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:50:37.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:37.231+0000 7fea0a7b9640 1 -- 172.21.15.29:0/3536610773 shutdown_connections 2024-03-16T11:50:37.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:37.232+0000 7fea0a7b9640 1 -- 172.21.15.29:0/3536610773 wait complete. 2024-03-16T11:50:37.242 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-16T11:50:37.349 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:37.348+0000 7f3d11e85640 1 -- 172.21.15.29:0/4160923479 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3d0c102290 msgr2=0x7f3d0c108b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:37.349 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:37.348+0000 7f3d11e85640 1 --2- 172.21.15.29:0/4160923479 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3d0c102290 0x7f3d0c108b20 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f3d04009ca0 tx=0x7f3d0402c0f0 comp rx=0 tx=0).stop 2024-03-16T11:50:37.349 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:37.348+0000 7f3d11e85640 1 -- 172.21.15.29:0/4160923479 shutdown_connections 2024-03-16T11:50:37.349 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:37.348+0000 7f3d11e85640 1 --2- 172.21.15.29:0/4160923479 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3d0c102290 0x7f3d0c108b20 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:37.349 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:37.348+0000 7f3d11e85640 1 --2- 172.21.15.29:0/4160923479 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3d0c1018d0 0x7f3d0c101d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:37.349 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:37.348+0000 7f3d11e85640 1 --2- 172.21.15.29:0/4160923479 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3d0c1006e0 0x7f3d0c100b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:37.349 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:37.348+0000 7f3d11e85640 1 -- 172.21.15.29:0/4160923479 >> 172.21.15.29:0/4160923479 conn(0x7f3d0c0fbc70 msgr2=0x7f3d0c0fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:50:37.350 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:37.348+0000 7f3d11e85640 1 -- 172.21.15.29:0/4160923479 shutdown_connections 2024-03-16T11:50:37.350 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:37.348+0000 7f3d11e85640 1 -- 172.21.15.29:0/4160923479 wait complete. 2024-03-16T11:50:37.350 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:37.348+0000 7f3d11e85640 1 Processor -- start 2024-03-16T11:50:37.350 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:37.349+0000 7f3d11e85640 1 -- start start 2024-03-16T11:50:37.350 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:37.349+0000 7f3d11e85640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3d0c1006e0 0x7f3d0c197a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:37.350 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:37.349+0000 7f3d11e85640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3d0c1018d0 0x7f3d0c197fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:37.350 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:37.349+0000 7f3d11e85640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3d0c102290 0x7f3d0c19f020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:37.350 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:37.349+0000 7f3d11e85640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f3d0c10d480 con 0x7f3d0c1018d0 2024-03-16T11:50:37.350 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:37.349+0000 7f3d11e85640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f3d0c10d300 con 0x7f3d0c102290 2024-03-16T11:50:37.350 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:37.349+0000 7f3d11e85640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f3d0c10d600 con 0x7f3d0c1006e0 2024-03-16T11:50:37.350 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:37.349+0000 7f3d0affd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3d0c1018d0 0x7f3d0c197fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:37.350 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:37.349+0000 7f3d0b7fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3d0c1006e0 0x7f3d0c197a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:37.350 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:37.349+0000 7f3d0b7fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3d0c1006e0 0x7f3d0c197a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.29:36786/0 (socket says 172.21.15.29:36786) 2024-03-16T11:50:37.350 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:37.349+0000 7f3d0affd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3d0c1018d0 0x7f3d0c197fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59360/0 (socket says 172.21.15.29:59360) 2024-03-16T11:50:37.350 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:37.349+0000 7f3d0b7fe640 1 -- 172.21.15.29:0/3579887518 learned_addr learned my addr 172.21.15.29:0/3579887518 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:50:37.351 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:37.349+0000 7f3d0bfff640 1 --2- 172.21.15.29:0/3579887518 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3d0c102290 0x7f3d0c19f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:37.351 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:37.349+0000 7f3d0affd640 1 -- 172.21.15.29:0/3579887518 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3d0c1006e0 msgr2=0x7f3d0c197a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:37.351 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:37.349+0000 7f3d0affd640 1 --2- 172.21.15.29:0/3579887518 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3d0c1006e0 0x7f3d0c197a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:37.351 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:37.349+0000 7f3d0affd640 1 -- 172.21.15.29:0/3579887518 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3d0c102290 msgr2=0x7f3d0c19f020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:37.351 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:37.349+0000 7f3d0affd640 1 --2- 172.21.15.29:0/3579887518 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3d0c102290 0x7f3d0c19f020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:37.351 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:37.349+0000 7f3d0affd640 1 -- 172.21.15.29:0/3579887518 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d0c19f690 con 0x7f3d0c1018d0 2024-03-16T11:50:37.351 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:37.349+0000 7f3d0b7fe640 1 --2- 172.21.15.29:0/3579887518 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3d0c1006e0 0x7f3d0c197a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:50:37.351 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:37.349+0000 7f3d0bfff640 1 --2- 172.21.15.29:0/3579887518 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3d0c102290 0x7f3d0c19f020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:50:37.351 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:37.349+0000 7f3d0affd640 1 --2- 172.21.15.29:0/3579887518 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3d0c1018d0 0x7f3d0c197fb0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f3d0000e9f0 tx=0x7f3d0000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:50:37.351 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:37.349+0000 7f3d08ff9640 1 -- 172.21.15.29:0/3579887518 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d0000ccc0 con 0x7f3d0c1018d0 2024-03-16T11:50:37.351 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:37.349+0000 7f3d11e85640 1 -- 172.21.15.29:0/3579887518 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3d0c19f980 con 0x7f3d0c1018d0 2024-03-16T11:50:37.351 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:37.350+0000 7f3d08ff9640 1 -- 172.21.15.29:0/3579887518 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3d00005020 con 0x7f3d0c1018d0 2024-03-16T11:50:37.351 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:37.350+0000 7f3d08ff9640 1 -- 172.21.15.29:0/3579887518 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d000049e0 con 0x7f3d0c1018d0 2024-03-16T11:50:37.351 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:37.350+0000 7f3d11e85640 1 -- 172.21.15.29:0/3579887518 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3d0c19fed0 con 0x7f3d0c1018d0 2024-03-16T11:50:37.352 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:37.350+0000 7f3d11e85640 1 -- 172.21.15.29:0/3579887518 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3d0c107a00 con 0x7f3d0c1018d0 2024-03-16T11:50:37.352 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:37.351+0000 7f3d08ff9640 1 -- 172.21.15.29:0/3579887518 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f3d00005560 con 0x7f3d0c1018d0 2024-03-16T11:50:37.352 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:37.352+0000 7f3d08ff9640 1 --2- 172.21.15.29:0/3579887518 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f3cf40a2800 0x7f3cf40a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:37.352 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:37.352+0000 7f3d0b7fe640 1 --2- 172.21.15.29:0/3579887518 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f3cf40a2800 0x7f3cf40a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:37.352 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:37.352+0000 7f3d08ff9640 1 -- 172.21.15.29:0/3579887518 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8211+0+0 (secure 0 0 0) 0x7f3d000cea10 con 0x7f3d0c1018d0 2024-03-16T11:50:37.353 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:37.352+0000 7f3d0b7fe640 1 --2- 172.21.15.29:0/3579887518 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f3cf40a2800 0x7f3cf40a4cc0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f3cfc0095f0 tx=0x7f3cfc005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:50:37.355 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:37.357+0000 7f3d08ff9640 1 -- 172.21.15.29:0/3579887518 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f3d000051d0 con 0x7f3d0c1018d0 2024-03-16T11:50:37.553 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:37.554+0000 7f3d11e85640 1 -- 172.21.15.29:0/3579887518 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f3d0c0623e0 con 0x7f3d0c1018d0 2024-03-16T11:50:37.553 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:37.554+0000 7f3d08ff9640 1 -- 172.21.15.29:0/3579887518 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v87) ==== 104+0+11 (secure 0 0 0) 0x7f3d0009bac0 con 0x7f3d0c1018d0 2024-03-16T11:50:37.554 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 32 2024-03-16T11:50:37.556 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:37.557+0000 7f3d11e85640 1 -- 172.21.15.29:0/3579887518 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f3cf40a2800 msgr2=0x7f3cf40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:37.556 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:37.557+0000 7f3d11e85640 1 --2- 172.21.15.29:0/3579887518 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f3cf40a2800 0x7f3cf40a4cc0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f3cfc0095f0 tx=0x7f3cfc005f90 comp rx=0 tx=0).stop 2024-03-16T11:50:37.556 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:37.557+0000 7f3d11e85640 1 -- 172.21.15.29:0/3579887518 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3d0c1018d0 msgr2=0x7f3d0c197fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:37.557 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:37.557+0000 7f3d11e85640 1 --2- 172.21.15.29:0/3579887518 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3d0c1018d0 0x7f3d0c197fb0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f3d0000e9f0 tx=0x7f3d0000eeb0 comp rx=0 tx=0).stop 2024-03-16T11:50:37.557 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:37.557+0000 7f3d11e85640 1 -- 172.21.15.29:0/3579887518 shutdown_connections 2024-03-16T11:50:37.557 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:37.557+0000 7f3d11e85640 1 --2- 172.21.15.29:0/3579887518 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f3cf40a2800 0x7f3cf40a4cc0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:37.557 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:37.557+0000 7f3d11e85640 1 --2- 172.21.15.29:0/3579887518 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3d0c102290 0x7f3d0c19f020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:37.557 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:37.557+0000 7f3d11e85640 1 --2- 172.21.15.29:0/3579887518 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3d0c1018d0 0x7f3d0c197fb0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:37.557 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:37.557+0000 7f3d11e85640 1 --2- 172.21.15.29:0/3579887518 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3d0c1006e0 0x7f3d0c197a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:37.557 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:37.557+0000 7f3d11e85640 1 -- 172.21.15.29:0/3579887518 >> 172.21.15.29:0/3579887518 conn(0x7f3d0c0fbc70 msgr2=0x7f3d0c104bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:50:37.557 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:37.557+0000 7f3d11e85640 1 -- 172.21.15.29:0/3579887518 shutdown_connections 2024-03-16T11:50:37.557 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:37.557+0000 7f3d11e85640 1 -- 172.21.15.29:0/3579887518 wait complete. 2024-03-16T11:50:37.567 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-16T11:50:37.675 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:37.674+0000 7fae27d87640 1 -- 172.21.15.29:0/3209045150 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae201029e0 msgr2=0x7fae20102e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:37.675 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:37.674+0000 7fae27d87640 1 --2- 172.21.15.29:0/3209045150 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae201029e0 0x7fae20102e60 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fae10009900 tx=0x7fae1002bc30 comp rx=0 tx=0).stop 2024-03-16T11:50:37.675 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:37.674+0000 7fae27d87640 1 -- 172.21.15.29:0/3209045150 shutdown_connections 2024-03-16T11:50:37.675 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:37.674+0000 7fae27d87640 1 --2- 172.21.15.29:0/3209045150 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fae201033a0 0x7fae20109c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:37.675 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:37.674+0000 7fae27d87640 1 --2- 172.21.15.29:0/3209045150 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae201029e0 0x7fae20102e60 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:37.675 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:37.674+0000 7fae27d87640 1 --2- 172.21.15.29:0/3209045150 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fae201017f0 0x7fae20101c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:37.675 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:37.674+0000 7fae27d87640 1 -- 172.21.15.29:0/3209045150 >> 172.21.15.29:0/3209045150 conn(0x7fae200fcd80 msgr2=0x7fae200ff1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:50:37.676 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:37.674+0000 7fae27d87640 1 -- 172.21.15.29:0/3209045150 shutdown_connections 2024-03-16T11:50:37.676 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:37.674+0000 7fae27d87640 1 -- 172.21.15.29:0/3209045150 wait complete. 2024-03-16T11:50:37.676 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:37.674+0000 7fae27d87640 1 Processor -- start 2024-03-16T11:50:37.676 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:37.674+0000 7fae27d87640 1 -- start start 2024-03-16T11:50:37.676 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:37.675+0000 7fae27d87640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fae201017f0 0x7fae20197af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:37.676 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:37.675+0000 7fae27d87640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fae201029e0 0x7fae20198030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:37.676 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:37.675+0000 7fae27d87640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae201033a0 0x7fae2019f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:37.676 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:37.675+0000 7fae27d87640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fae2010d450 con 0x7fae201033a0 2024-03-16T11:50:37.676 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:37.675+0000 7fae27d87640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fae2010d2d0 con 0x7fae201029e0 2024-03-16T11:50:37.676 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:37.675+0000 7fae27d87640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fae2010d5d0 con 0x7fae201017f0 2024-03-16T11:50:37.676 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:37.675+0000 7fae26324640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae201033a0 0x7fae2019f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:37.676 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:37.675+0000 7fae26324640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae201033a0 0x7fae2019f0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59378/0 (socket says 172.21.15.29:59378) 2024-03-16T11:50:37.676 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:37.675+0000 7fae26324640 1 -- 172.21.15.29:0/4183524737 learned_addr learned my addr 172.21.15.29:0/4183524737 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:50:37.676 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:37.675+0000 7fae25322640 1 --2- 172.21.15.29:0/4183524737 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fae201029e0 0x7fae20198030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:37.676 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:37.675+0000 7fae25b23640 1 --2- 172.21.15.29:0/4183524737 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fae201017f0 0x7fae20197af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:37.676 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:37.675+0000 7fae26324640 1 -- 172.21.15.29:0/4183524737 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fae201017f0 msgr2=0x7fae20197af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:37.677 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:37.675+0000 7fae26324640 1 --2- 172.21.15.29:0/4183524737 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fae201017f0 0x7fae20197af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:37.677 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:37.675+0000 7fae26324640 1 -- 172.21.15.29:0/4183524737 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fae201029e0 msgr2=0x7fae20198030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:37.677 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:37.675+0000 7fae26324640 1 --2- 172.21.15.29:0/4183524737 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fae201029e0 0x7fae20198030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:37.677 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:37.675+0000 7fae26324640 1 -- 172.21.15.29:0/4183524737 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae2019f710 con 0x7fae201033a0 2024-03-16T11:50:37.677 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:37.675+0000 7fae26324640 1 --2- 172.21.15.29:0/4183524737 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae201033a0 0x7fae2019f0a0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fae1800eb10 tx=0x7fae1800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:50:37.677 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:37.675+0000 7fae16ffd640 1 -- 172.21.15.29:0/4183524737 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae18004b20 con 0x7fae201033a0 2024-03-16T11:50:37.677 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:37.675+0000 7fae16ffd640 1 -- 172.21.15.29:0/4183524737 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fae18004cc0 con 0x7fae201033a0 2024-03-16T11:50:37.677 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:37.675+0000 7fae27d87640 1 -- 172.21.15.29:0/4183524737 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fae2019fa00 con 0x7fae201033a0 2024-03-16T11:50:37.677 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:37.675+0000 7fae16ffd640 1 -- 172.21.15.29:0/4183524737 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae18010760 con 0x7fae201033a0 2024-03-16T11:50:37.677 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:37.675+0000 7fae27d87640 1 -- 172.21.15.29:0/4183524737 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fae2019fed0 con 0x7fae201033a0 2024-03-16T11:50:37.677 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:37.676+0000 7fae27d87640 1 -- 172.21.15.29:0/4183524737 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fae20108b90 con 0x7fae201033a0 2024-03-16T11:50:37.678 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:37.677+0000 7fae16ffd640 1 -- 172.21.15.29:0/4183524737 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fae18005000 con 0x7fae201033a0 2024-03-16T11:50:37.678 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:37.678+0000 7fae16ffd640 1 --2- 172.21.15.29:0/4183524737 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fae0c0a2800 0x7fae0c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:37.678 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:37.678+0000 7fae25b23640 1 --2- 172.21.15.29:0/4183524737 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fae0c0a2800 0x7fae0c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:37.678 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:37.678+0000 7fae16ffd640 1 -- 172.21.15.29:0/4183524737 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8211+0+0 (secure 0 0 0) 0x7fae180cf250 con 0x7fae201033a0 2024-03-16T11:50:37.678 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:37.679+0000 7fae25b23640 1 --2- 172.21.15.29:0/4183524737 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fae0c0a2800 0x7fae0c0a4cc0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fae1c0095f0 tx=0x7fae1c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:50:37.681 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:37.682+0000 7fae16ffd640 1 -- 172.21.15.29:0/4183524737 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fae18053740 con 0x7fae201033a0 2024-03-16T11:50:37.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:50:37 smithi029 ceph-mon[31975]: from='client.? 172.21.15.29:0/2542244855' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T11:50:37.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:50:37 smithi029 ceph-mon[31975]: from='client.? 172.21.15.29:0/3536610773' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-16T11:50:37.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:50:37 smithi029 ceph-mon[31975]: from='client.? 172.21.15.29:0/3579887518' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-16T11:50:37.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:50:37 smithi029 ceph-mon[31975]: pgmap v352: 97 pgs: 97 active+clean; 640 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-16T11:50:37.865 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:37.865+0000 7fae27d87640 1 -- 172.21.15.29:0/4183524737 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fae2002d1b0 con 0x7fae201033a0 2024-03-16T11:50:37.865 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:37.866+0000 7fae16ffd640 1 -- 172.21.15.29:0/4183524737 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v87) ==== 100+0+11 (secure 0 0 0) 0x7fae1809c2a0 con 0x7fae201033a0 2024-03-16T11:50:37.865 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 64 2024-03-16T11:50:37.867 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:37.868+0000 7fae27d87640 1 -- 172.21.15.29:0/4183524737 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fae0c0a2800 msgr2=0x7fae0c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:37.867 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:37.868+0000 7fae27d87640 1 --2- 172.21.15.29:0/4183524737 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fae0c0a2800 0x7fae0c0a4cc0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fae1c0095f0 tx=0x7fae1c005f90 comp rx=0 tx=0).stop 2024-03-16T11:50:37.867 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:37.868+0000 7fae27d87640 1 -- 172.21.15.29:0/4183524737 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae201033a0 msgr2=0x7fae2019f0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:37.867 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:37.868+0000 7fae27d87640 1 --2- 172.21.15.29:0/4183524737 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae201033a0 0x7fae2019f0a0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fae1800eb10 tx=0x7fae1800c2d0 comp rx=0 tx=0).stop 2024-03-16T11:50:37.867 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:37.868+0000 7fae27d87640 1 -- 172.21.15.29:0/4183524737 shutdown_connections 2024-03-16T11:50:37.867 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:37.868+0000 7fae27d87640 1 --2- 172.21.15.29:0/4183524737 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fae0c0a2800 0x7fae0c0a4cc0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:37.867 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:37.868+0000 7fae27d87640 1 --2- 172.21.15.29:0/4183524737 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae201033a0 0x7fae2019f0a0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:37.867 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:37.868+0000 7fae27d87640 1 --2- 172.21.15.29:0/4183524737 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fae201029e0 0x7fae20198030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:37.868 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:37.868+0000 7fae27d87640 1 --2- 172.21.15.29:0/4183524737 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fae201017f0 0x7fae20197af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:37.868 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:37.868+0000 7fae27d87640 1 -- 172.21.15.29:0/4183524737 >> 172.21.15.29:0/4183524737 conn(0x7fae200fcd80 msgr2=0x7fae20105d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:50:37.868 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:37.868+0000 7fae27d87640 1 -- 172.21.15.29:0/4183524737 shutdown_connections 2024-03-16T11:50:37.868 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:37.868+0000 7fae27d87640 1 -- 172.21.15.29:0/4183524737 wait complete. 2024-03-16T11:50:37.878 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T11:50:37.982 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:37.981+0000 7fce1e073640 1 -- 172.21.15.29:0/1723308145 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fce18101800 msgr2=0x7fce18101c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:37.983 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:37.981+0000 7fce1e073640 1 --2- 172.21.15.29:0/1723308145 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fce18101800 0x7fce18101c20 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fce08009900 tx=0x7fce0802bc30 comp rx=0 tx=0).stop 2024-03-16T11:50:37.983 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:37.981+0000 7fce1e073640 1 -- 172.21.15.29:0/1723308145 shutdown_connections 2024-03-16T11:50:37.983 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:37.981+0000 7fce1e073640 1 --2- 172.21.15.29:0/1723308145 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fce181033b0 0x7fce18109c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:37.983 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:37.981+0000 7fce1e073640 1 --2- 172.21.15.29:0/1723308145 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fce181029f0 0x7fce18102e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:37.983 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:37.981+0000 7fce1e073640 1 --2- 172.21.15.29:0/1723308145 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fce18101800 0x7fce18101c20 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:37.983 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:37.981+0000 7fce1e073640 1 -- 172.21.15.29:0/1723308145 >> 172.21.15.29:0/1723308145 conn(0x7fce180fcd90 msgr2=0x7fce180ff1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:50:37.983 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:37.981+0000 7fce1e073640 1 -- 172.21.15.29:0/1723308145 shutdown_connections 2024-03-16T11:50:37.983 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:37.981+0000 7fce1e073640 1 -- 172.21.15.29:0/1723308145 wait complete. 2024-03-16T11:50:37.983 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:37.982+0000 7fce1e073640 1 Processor -- start 2024-03-16T11:50:37.983 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:37.982+0000 7fce1e073640 1 -- start start 2024-03-16T11:50:37.983 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:37.982+0000 7fce1e073640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fce18101800 0x7fce18197af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:37.983 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:37.982+0000 7fce1e073640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fce181029f0 0x7fce18198030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:37.983 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:37.982+0000 7fce1e073640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fce181033b0 0x7fce1819f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:37.983 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:37.982+0000 7fce1e073640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fce1810d480 con 0x7fce181029f0 2024-03-16T11:50:37.984 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:37.982+0000 7fce1e073640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fce1810d300 con 0x7fce181033b0 2024-03-16T11:50:37.984 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:37.982+0000 7fce1e073640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fce1810d600 con 0x7fce18101800 2024-03-16T11:50:37.984 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:37.982+0000 7fce16ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fce181029f0 0x7fce18198030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:37.984 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:37.982+0000 7fce16ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fce181029f0 0x7fce18198030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59398/0 (socket says 172.21.15.29:59398) 2024-03-16T11:50:37.984 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:37.982+0000 7fce16ffd640 1 -- 172.21.15.29:0/3500465737 learned_addr learned my addr 172.21.15.29:0/3500465737 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:50:37.984 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:37.982+0000 7fce177fe640 1 --2- 172.21.15.29:0/3500465737 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fce18101800 0x7fce18197af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:37.984 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:37.982+0000 7fce17fff640 1 --2- 172.21.15.29:0/3500465737 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fce181033b0 0x7fce1819f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:37.984 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:37.982+0000 7fce16ffd640 1 -- 172.21.15.29:0/3500465737 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fce18101800 msgr2=0x7fce18197af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:37.984 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:37.982+0000 7fce16ffd640 1 --2- 172.21.15.29:0/3500465737 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fce18101800 0x7fce18197af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:37.984 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:37.983+0000 7fce16ffd640 1 -- 172.21.15.29:0/3500465737 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fce181033b0 msgr2=0x7fce1819f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:37.984 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:37.983+0000 7fce16ffd640 1 --2- 172.21.15.29:0/3500465737 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fce181033b0 0x7fce1819f0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:37.984 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:37.983+0000 7fce16ffd640 1 -- 172.21.15.29:0/3500465737 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce1819f710 con 0x7fce181029f0 2024-03-16T11:50:37.984 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:37.983+0000 7fce16ffd640 1 --2- 172.21.15.29:0/3500465737 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fce181029f0 0x7fce18198030 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fce0c00e9f0 tx=0x7fce0c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:50:37.985 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:37.983+0000 7fce14ff9640 1 -- 172.21.15.29:0/3500465737 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce0c00cd80 con 0x7fce181029f0 2024-03-16T11:50:37.985 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:37.983+0000 7fce14ff9640 1 -- 172.21.15.29:0/3500465737 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fce0c010910 con 0x7fce181029f0 2024-03-16T11:50:37.985 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:37.983+0000 7fce1e073640 1 -- 172.21.15.29:0/3500465737 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fce1819fa00 con 0x7fce181029f0 2024-03-16T11:50:37.985 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:37.983+0000 7fce14ff9640 1 -- 172.21.15.29:0/3500465737 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce0c018990 con 0x7fce181029f0 2024-03-16T11:50:37.985 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:37.983+0000 7fce1e073640 1 -- 172.21.15.29:0/3500465737 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fce1819ff50 con 0x7fce181029f0 2024-03-16T11:50:37.986 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:37.984+0000 7fce1e073640 1 -- 172.21.15.29:0/3500465737 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fce18108b20 con 0x7fce181029f0 2024-03-16T11:50:37.986 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:37.985+0000 7fce14ff9640 1 -- 172.21.15.29:0/3500465737 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fce0c010ac0 con 0x7fce181029f0 2024-03-16T11:50:37.986 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:37.985+0000 7fce14ff9640 1 --2- 172.21.15.29:0/3500465737 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fce000a2800 0x7fce000a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:37.986 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:37.986+0000 7fce177fe640 1 --2- 172.21.15.29:0/3500465737 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fce000a2800 0x7fce000a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:37.986 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:37.986+0000 7fce14ff9640 1 -- 172.21.15.29:0/3500465737 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8211+0+0 (secure 0 0 0) 0x7fce0c0ced80 con 0x7fce181029f0 2024-03-16T11:50:37.986 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:37.986+0000 7fce177fe640 1 --2- 172.21.15.29:0/3500465737 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fce000a2800 0x7fce000a4cc0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fce08000c00 tx=0x7fce08005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:50:37.989 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:37.990+0000 7fce14ff9640 1 -- 172.21.15.29:0/3500465737 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fce0c09b7e0 con 0x7fce181029f0 2024-03-16T11:50:38.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:50:37 smithi154 ceph-mon[36320]: from='client.? 172.21.15.29:0/2542244855' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T11:50:38.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:50:37 smithi154 ceph-mon[36320]: from='client.? 172.21.15.29:0/3536610773' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-16T11:50:38.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:50:37 smithi154 ceph-mon[36320]: from='client.? 172.21.15.29:0/3579887518' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-16T11:50:38.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:50:37 smithi154 ceph-mon[36320]: pgmap v352: 97 pgs: 97 active+clean; 640 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-16T11:50:38.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:50:37 smithi152 ceph-mon[36291]: from='client.? 172.21.15.29:0/2542244855' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T11:50:38.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:50:37 smithi152 ceph-mon[36291]: from='client.? 172.21.15.29:0/3536610773' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-16T11:50:38.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:50:37 smithi152 ceph-mon[36291]: from='client.? 172.21.15.29:0/3579887518' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-16T11:50:38.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:50:37 smithi152 ceph-mon[36291]: pgmap v352: 97 pgs: 97 active+clean; 640 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-16T11:50:38.215 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:38.217+0000 7fce1e073640 1 -- 172.21.15.29:0/3500465737 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fce1802d1b0 con 0x7fce181029f0 2024-03-16T11:50:38.216 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:38.217+0000 7fce14ff9640 1 -- 172.21.15.29:0/3500465737 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+11875 (secure 0 0 0) 0x7fce0c09f880 con 0x7fce181029f0 2024-03-16T11:50:38.217 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-16T11:50:38.217 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14574,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/3505811955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":3505811955},{"type":"v1","addr":"172.21.15.29:6835","nonce":3505811955}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14586,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/1901337868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":1901337868},{"type":"v1","addr":"172.21.15.29:6837","nonce":1901337868}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14652,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6839/544966534","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6838","nonce":544966534},{"type":"v1","addr":"172.21.15.29:6839","nonce":544966534}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14664,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6841/540705290","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6840","nonce":540705290},{"type":"v1","addr":"172.21.15.29:6841","nonce":540705290}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24407,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6833/3136024628","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6832","nonce":3136024628},{"type":"v1","addr":"172.21.15.152:6833","nonce":3136024628}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24421,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6833/2677426282","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":2677426282},{"type":"v1","addr":"172.21.15.154:6833","nonce":2677426282}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":21,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-16T11:48:38.431509+0000","modified":"2024-03-16T11:48:42.969084+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24503,"mds_1":24499,"mds_2":24464},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24434":{"gid":24434,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.152:6835/2919113178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6834","nonce":2919113178},{"type":"v1","addr":"172.21.15.152:6835","nonce":2919113178}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24464":{"gid":24464,"name":"h","rank":2,"incarnation":19,"state":"up:active","state_seq":7,"addr":"172.21.15.152:6837/1323866722","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6836","nonce":1323866722},{"type":"v1","addr":"172.21.15.152:6837","nonce":1323866722}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.154:6835/4165274681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":4165274681},{"type":"v1","addr":"172.21.15.154:6835","nonce":4165274681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.154:6837/1468560096","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":1468560096},{"type":"v1","addr":"172.21.15.154:6837","nonce":1468560096}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":1,"incarnation":17,"state":"up:active","state_seq":6,"addr":"172.21.15.154:6839/685736862","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":685736862},{"type":"v1","addr":"172.21.15.154:6839","nonce":685736862}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.152:6839/1548731833","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6838","nonce":1548731833},{"type":"v1","addr":"172.21.15.152:6839","nonce":1548731833}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24503,"qdb_cluster":[24503,24464,24499]},"id":1}]} 2024-03-16T11:50:38.217 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 21 2024-03-16T11:50:38.219 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:38.219+0000 7fce1e073640 1 -- 172.21.15.29:0/3500465737 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fce000a2800 msgr2=0x7fce000a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:38.219 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:38.219+0000 7fce1e073640 1 --2- 172.21.15.29:0/3500465737 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fce000a2800 0x7fce000a4cc0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fce08000c00 tx=0x7fce08005fd0 comp rx=0 tx=0).stop 2024-03-16T11:50:38.219 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:38.219+0000 7fce1e073640 1 -- 172.21.15.29:0/3500465737 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fce181029f0 msgr2=0x7fce18198030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:38.219 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:38.219+0000 7fce1e073640 1 --2- 172.21.15.29:0/3500465737 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fce181029f0 0x7fce18198030 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fce0c00e9f0 tx=0x7fce0c00eeb0 comp rx=0 tx=0).stop 2024-03-16T11:50:38.219 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:38.220+0000 7fce1e073640 1 -- 172.21.15.29:0/3500465737 shutdown_connections 2024-03-16T11:50:38.219 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:38.220+0000 7fce1e073640 1 --2- 172.21.15.29:0/3500465737 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fce000a2800 0x7fce000a4cc0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:38.219 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:38.220+0000 7fce1e073640 1 --2- 172.21.15.29:0/3500465737 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fce181033b0 0x7fce1819f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:38.219 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:38.220+0000 7fce1e073640 1 --2- 172.21.15.29:0/3500465737 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fce181029f0 0x7fce18198030 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:38.219 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:38.220+0000 7fce1e073640 1 --2- 172.21.15.29:0/3500465737 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fce18101800 0x7fce18197af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:38.219 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:38.220+0000 7fce1e073640 1 -- 172.21.15.29:0/3500465737 >> 172.21.15.29:0/3500465737 conn(0x7fce180fcd90 msgr2=0x7fce18105cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:50:38.219 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:38.220+0000 7fce1e073640 1 -- 172.21.15.29:0/3500465737 shutdown_connections 2024-03-16T11:50:38.220 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:38.220+0000 7fce1e073640 1 -- 172.21.15.29:0/3500465737 wait complete. 2024-03-16T11:50:38.230 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-16T11:50:38.335 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:38.333+0000 7fc7d795b640 1 -- 172.21.15.29:0/1124043702 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc7d0101370 msgr2=0x7fc7d01017f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:38.335 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:38.333+0000 7fc7d795b640 1 --2- 172.21.15.29:0/1124043702 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc7d0101370 0x7fc7d01017f0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fc7c8009ca0 tx=0x7fc7c802c0f0 comp rx=0 tx=0).stop 2024-03-16T11:50:38.335 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:38.333+0000 7fc7d795b640 1 -- 172.21.15.29:0/1124043702 shutdown_connections 2024-03-16T11:50:38.335 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:38.333+0000 7fc7d795b640 1 --2- 172.21.15.29:0/1124043702 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc7d0101370 0x7fc7d01017f0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:38.335 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:38.333+0000 7fc7d795b640 1 --2- 172.21.15.29:0/1124043702 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc7d01009d0 0x7fc7d0100e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:38.335 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:38.333+0000 7fc7d795b640 1 --2- 172.21.15.29:0/1124043702 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc7d0100070 0x7fc7d0100490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:38.335 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:38.333+0000 7fc7d795b640 1 -- 172.21.15.29:0/1124043702 >> 172.21.15.29:0/1124043702 conn(0x7fc7d00fbc70 msgr2=0x7fc7d00fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:50:38.335 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:38.333+0000 7fc7d795b640 1 -- 172.21.15.29:0/1124043702 shutdown_connections 2024-03-16T11:50:38.335 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:38.333+0000 7fc7d795b640 1 -- 172.21.15.29:0/1124043702 wait complete. 2024-03-16T11:50:38.335 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:38.334+0000 7fc7d795b640 1 Processor -- start 2024-03-16T11:50:38.335 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:38.334+0000 7fc7d795b640 1 -- start start 2024-03-16T11:50:38.335 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:38.334+0000 7fc7d795b640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc7d0100070 0x7fc7d0199c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:38.335 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:38.334+0000 7fc7d795b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc7d01009d0 0x7fc7d019a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:38.335 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:38.334+0000 7fc7d795b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc7d0101370 0x7fc7d01a11f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:38.336 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:38.334+0000 7fc7d795b640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fc7d010f650 con 0x7fc7d01009d0 2024-03-16T11:50:38.336 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:38.334+0000 7fc7d795b640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fc7d010f4d0 con 0x7fc7d0100070 2024-03-16T11:50:38.336 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:38.334+0000 7fc7d795b640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fc7d010f7d0 con 0x7fc7d0101370 2024-03-16T11:50:38.336 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:38.334+0000 7fc7d4ef6640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc7d01009d0 0x7fc7d019a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:38.336 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:38.334+0000 7fc7d4ef6640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc7d01009d0 0x7fc7d019a1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59414/0 (socket says 172.21.15.29:59414) 2024-03-16T11:50:38.336 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:38.334+0000 7fc7d4ef6640 1 -- 172.21.15.29:0/2790331527 learned_addr learned my addr 172.21.15.29:0/2790331527 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:50:38.336 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:38.334+0000 7fc7d56f7640 1 --2- 172.21.15.29:0/2790331527 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc7d0100070 0x7fc7d0199c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:38.336 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:38.334+0000 7fc7d5ef8640 1 --2- 172.21.15.29:0/2790331527 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc7d0101370 0x7fc7d01a11f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:38.336 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:38.335+0000 7fc7d4ef6640 1 -- 172.21.15.29:0/2790331527 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc7d0101370 msgr2=0x7fc7d01a11f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:38.336 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:38.335+0000 7fc7d4ef6640 1 --2- 172.21.15.29:0/2790331527 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc7d0101370 0x7fc7d01a11f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:38.336 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:38.335+0000 7fc7d4ef6640 1 -- 172.21.15.29:0/2790331527 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc7d0100070 msgr2=0x7fc7d0199c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:38.336 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:38.335+0000 7fc7d4ef6640 1 --2- 172.21.15.29:0/2790331527 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc7d0100070 0x7fc7d0199c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:38.336 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:38.335+0000 7fc7d4ef6640 1 -- 172.21.15.29:0/2790331527 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7d01a1890 con 0x7fc7d01009d0 2024-03-16T11:50:38.336 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:38.335+0000 7fc7d56f7640 1 --2- 172.21.15.29:0/2790331527 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc7d0100070 0x7fc7d0199c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:50:38.336 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:38.335+0000 7fc7d4ef6640 1 --2- 172.21.15.29:0/2790331527 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc7d01009d0 0x7fc7d019a1c0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fc7c000e9f0 tx=0x7fc7c000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:50:38.336 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:38.335+0000 7fc7c67fc640 1 -- 172.21.15.29:0/2790331527 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7c000cd80 con 0x7fc7d01009d0 2024-03-16T11:50:38.337 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:38.335+0000 7fc7c67fc640 1 -- 172.21.15.29:0/2790331527 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc7c0010910 con 0x7fc7d01009d0 2024-03-16T11:50:38.337 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:38.335+0000 7fc7d795b640 1 -- 172.21.15.29:0/2790331527 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc7d01a1b80 con 0x7fc7d01009d0 2024-03-16T11:50:38.337 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:38.335+0000 7fc7c67fc640 1 -- 172.21.15.29:0/2790331527 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7c0018990 con 0x7fc7d01009d0 2024-03-16T11:50:38.337 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:38.335+0000 7fc7d795b640 1 -- 172.21.15.29:0/2790331527 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc7d01a20d0 con 0x7fc7d01009d0 2024-03-16T11:50:38.337 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:38.336+0000 7fc7d795b640 1 -- 172.21.15.29:0/2790331527 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc7d010af00 con 0x7fc7d01009d0 2024-03-16T11:50:38.337 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:38.337+0000 7fc7c67fc640 1 -- 172.21.15.29:0/2790331527 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fc7c0010ac0 con 0x7fc7d01009d0 2024-03-16T11:50:38.338 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:38.338+0000 7fc7c67fc640 1 --2- 172.21.15.29:0/2790331527 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fc7bc0a2800 0x7fc7bc0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:38.338 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:38.338+0000 7fc7d56f7640 1 --2- 172.21.15.29:0/2790331527 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fc7bc0a2800 0x7fc7bc0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:38.338 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:38.338+0000 7fc7c67fc640 1 -- 172.21.15.29:0/2790331527 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8211+0+0 (secure 0 0 0) 0x7fc7c00cee90 con 0x7fc7d01009d0 2024-03-16T11:50:38.338 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:38.338+0000 7fc7d56f7640 1 --2- 172.21.15.29:0/2790331527 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fc7bc0a2800 0x7fc7bc0a4cc0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fc7d019b150 tx=0x7fc7cc005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:50:38.341 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:38.342+0000 7fc7c67fc640 1 -- 172.21.15.29:0/2790331527 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fc7c009b890 con 0x7fc7d01009d0 2024-03-16T11:50:38.526 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:38.527+0000 7fc7d795b640 1 -- 172.21.15.29:0/2790331527 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc7d002d3d0 con 0x7fc7d01009d0 2024-03-16T11:50:38.527 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:38.528+0000 7fc7c67fc640 1 -- 172.21.15.29:0/2790331527 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) ==== 74+0+22668 (secure 0 0 0) 0x7fc7c009f930 con 0x7fc7d01009d0 2024-03-16T11:50:38.528 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-16T11:50:38.528 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":87,"fsid":"d1887f28-e389-11ee-95c9-87774f69a715","created":"2024-03-16T11:40:16.475063+0000","modified":"2024-03-16T11:49:28.628389+0000","last_up_change":"2024-03-16T11:47:44.349348+0000","last_in_change":"2024-03-16T11:47:23.672432+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-16T11:43:51.892652+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-16T11:48:36.709732+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-16T11:48:37.586140+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"60f5608f-a408-45ba-bdf6-5b53758f88fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":2748947913},{"type":"v1","addr":"172.21.15.29:6803","nonce":2748947913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":2748947913},{"type":"v1","addr":"172.21.15.29:6805","nonce":2748947913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":2748947913},{"type":"v1","addr":"172.21.15.29:6809","nonce":2748947913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":2748947913},{"type":"v1","addr":"172.21.15.29:6807","nonce":2748947913}]},"public_addr":"172.21.15.29:6803/2748947913","cluster_addr":"172.21.15.29:6805/2748947913","heartbeat_back_addr":"172.21.15.29:6809/2748947913","heartbeat_front_addr":"172.21.15.29:6807/2748947913","state":["exists","up"]},{"osd":1,"uuid":"e70b53d6-ee78-4e56-9a2c-263ae0501b43","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6800","nonce":2339408422},{"type":"v1","addr":"172.21.15.152:6801","nonce":2339408422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6802","nonce":2339408422},{"type":"v1","addr":"172.21.15.152:6803","nonce":2339408422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6806","nonce":2339408422},{"type":"v1","addr":"172.21.15.152:6807","nonce":2339408422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6804","nonce":2339408422},{"type":"v1","addr":"172.21.15.152:6805","nonce":2339408422}]},"public_addr":"172.21.15.152:6801/2339408422","cluster_addr":"172.21.15.152:6803/2339408422","heartbeat_back_addr":"172.21.15.152:6807/2339408422","heartbeat_front_addr":"172.21.15.152:6805/2339408422","state":["exists","up"]},{"osd":2,"uuid":"ff39389a-dfac-44b5-80ff-36abca316e78","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6800","nonce":4288325280},{"type":"v1","addr":"172.21.15.154:6801","nonce":4288325280}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6802","nonce":4288325280},{"type":"v1","addr":"172.21.15.154:6803","nonce":4288325280}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6806","nonce":4288325280},{"type":"v1","addr":"172.21.15.154:6807","nonce":4288325280}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":4288325280},{"type":"v1","addr":"172.21.15.154:6805","nonce":4288325280}]},"public_addr":"172.21.15.154:6801/4288325280","cluster_addr":"172.21.15.154:6803/4288325280","heartbeat_back_addr":"172.21.15.154:6807/4288325280","heartbeat_front_addr":"172.21.15.154:6805/4288325280","state":["exists","up"]},{"osd":3,"uuid":"069e1795-55bc-4324-b94c-eb24ef2334c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":1538750397},{"type":"v1","addr":"172.21.15.29:6811","nonce":1538750397}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":1538750397},{"type":"v1","addr":"172.21.15.29:6813","nonce":1538750397}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":1538750397},{"type":"v1","addr":"172.21.15.29:6817","nonce":1538750397}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":1538750397},{"type":"v1","addr":"172.21.15.29:6815","nonce":1538750397}]},"public_addr":"172.21.15.29:6811/1538750397","cluster_addr":"172.21.15.29:6813/1538750397","heartbeat_back_addr":"172.21.15.29:6817/1538750397","heartbeat_front_addr":"172.21.15.29:6815/1538750397","state":["exists","up"]},{"osd":4,"uuid":"5a9d43a1-e8f5-4ab5-a441-afbc6ed4066e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6808","nonce":2795081750},{"type":"v1","addr":"172.21.15.152:6809","nonce":2795081750}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6810","nonce":2795081750},{"type":"v1","addr":"172.21.15.152:6811","nonce":2795081750}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6814","nonce":2795081750},{"type":"v1","addr":"172.21.15.152:6815","nonce":2795081750}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6812","nonce":2795081750},{"type":"v1","addr":"172.21.15.152:6813","nonce":2795081750}]},"public_addr":"172.21.15.152:6809/2795081750","cluster_addr":"172.21.15.152:6811/2795081750","heartbeat_back_addr":"172.21.15.152:6815/2795081750","heartbeat_front_addr":"172.21.15.152:6813/2795081750","state":["exists","up"]},{"osd":5,"uuid":"234de428-4ac7-44be-9a54-7b58331fdf8c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":1472686560},{"type":"v1","addr":"172.21.15.154:6809","nonce":1472686560}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6810","nonce":1472686560},{"type":"v1","addr":"172.21.15.154:6811","nonce":1472686560}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6814","nonce":1472686560},{"type":"v1","addr":"172.21.15.154:6815","nonce":1472686560}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6812","nonce":1472686560},{"type":"v1","addr":"172.21.15.154:6813","nonce":1472686560}]},"public_addr":"172.21.15.154:6809/1472686560","cluster_addr":"172.21.15.154:6811/1472686560","heartbeat_back_addr":"172.21.15.154:6815/1472686560","heartbeat_front_addr":"172.21.15.154:6813/1472686560","state":["exists","up"]},{"osd":6,"uuid":"ae45b8d9-7883-40c9-a9cd-4c92b601d4f2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":816506244},{"type":"v1","addr":"172.21.15.29:6819","nonce":816506244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":816506244},{"type":"v1","addr":"172.21.15.29:6821","nonce":816506244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":816506244},{"type":"v1","addr":"172.21.15.29:6825","nonce":816506244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":816506244},{"type":"v1","addr":"172.21.15.29:6823","nonce":816506244}]},"public_addr":"172.21.15.29:6819/816506244","cluster_addr":"172.21.15.29:6821/816506244","heartbeat_back_addr":"172.21.15.29:6825/816506244","heartbeat_front_addr":"172.21.15.29:6823/816506244","state":["exists","up"]},{"osd":7,"uuid":"6f687b28-4fce-4d4a-8b4e-294173d9d6c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6816","nonce":2924651787},{"type":"v1","addr":"172.21.15.152:6817","nonce":2924651787}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6818","nonce":2924651787},{"type":"v1","addr":"172.21.15.152:6819","nonce":2924651787}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6822","nonce":2924651787},{"type":"v1","addr":"172.21.15.152:6823","nonce":2924651787}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6820","nonce":2924651787},{"type":"v1","addr":"172.21.15.152:6821","nonce":2924651787}]},"public_addr":"172.21.15.152:6817/2924651787","cluster_addr":"172.21.15.152:6819/2924651787","heartbeat_back_addr":"172.21.15.152:6823/2924651787","heartbeat_front_addr":"172.21.15.152:6821/2924651787","state":["exists","up"]},{"osd":8,"uuid":"a25fb741-caa7-429a-b540-e394d034b446","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6816","nonce":129974156},{"type":"v1","addr":"172.21.15.154:6817","nonce":129974156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6818","nonce":129974156},{"type":"v1","addr":"172.21.15.154:6819","nonce":129974156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6822","nonce":129974156},{"type":"v1","addr":"172.21.15.154:6823","nonce":129974156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6820","nonce":129974156},{"type":"v1","addr":"172.21.15.154:6821","nonce":129974156}]},"public_addr":"172.21.15.154:6817/129974156","cluster_addr":"172.21.15.154:6819/129974156","heartbeat_back_addr":"172.21.15.154:6823/129974156","heartbeat_front_addr":"172.21.15.154:6821/129974156","state":["exists","up"]},{"osd":9,"uuid":"1071cb0e-8c5f-4cd8-9805-0045e220d946","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6826","nonce":2951414243},{"type":"v1","addr":"172.21.15.29:6827","nonce":2951414243}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":2951414243},{"type":"v1","addr":"172.21.15.29:6829","nonce":2951414243}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":2951414243},{"type":"v1","addr":"172.21.15.29:6833","nonce":2951414243}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":2951414243},{"type":"v1","addr":"172.21.15.29:6831","nonce":2951414243}]},"public_addr":"172.21.15.29:6827/2951414243","cluster_addr":"172.21.15.29:6829/2951414243","heartbeat_back_addr":"172.21.15.29:6833/2951414243","heartbeat_front_addr":"172.21.15.29:6831/2951414243","state":["exists","up"]},{"osd":10,"uuid":"ee7c7d74-3f39-4cd4-be80-843fb1dd9ecc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6824","nonce":3412525480},{"type":"v1","addr":"172.21.15.152:6825","nonce":3412525480}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6826","nonce":3412525480},{"type":"v1","addr":"172.21.15.152:6827","nonce":3412525480}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6830","nonce":3412525480},{"type":"v1","addr":"172.21.15.152:6831","nonce":3412525480}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6828","nonce":3412525480},{"type":"v1","addr":"172.21.15.152:6829","nonce":3412525480}]},"public_addr":"172.21.15.152:6825/3412525480","cluster_addr":"172.21.15.152:6827/3412525480","heartbeat_back_addr":"172.21.15.152:6831/3412525480","heartbeat_front_addr":"172.21.15.152:6829/3412525480","state":["exists","up"]},{"osd":11,"uuid":"4598c803-984a-4de6-ac22-066502c17889","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6824","nonce":1160927966},{"type":"v1","addr":"172.21.15.154:6825","nonce":1160927966}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6826","nonce":1160927966},{"type":"v1","addr":"172.21.15.154:6827","nonce":1160927966}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6830","nonce":1160927966},{"type":"v1","addr":"172.21.15.154:6831","nonce":1160927966}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6828","nonce":1160927966},{"type":"v1","addr":"172.21.15.154:6829","nonce":1160927966}]},"public_addr":"172.21.15.154:6825/1160927966","cluster_addr":"172.21.15.154:6827/1160927966","heartbeat_back_addr":"172.21.15.154:6831/1160927966","heartbeat_front_addr":"172.21.15.154:6829/1160927966","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:43:02.343393+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:43:25.209151+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:43:48.946164+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:44:14.588424+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:44:39.700025+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:45:04.061197+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:45:31.214508+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:45:56.937296+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:46:22.768915+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:46:50.101011+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:47:16.298041+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:47:42.402095+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"2.1b","mappings":[{"from":2,"to":1}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"3.33","mappings":[{"from":5,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.29:0/1853086886":"2024-03-17T11:41:27.531724+0000","172.21.15.29:0/1211659364":"2024-03-17T11:41:27.531724+0000","172.21.15.29:6800/2798831128":"2024-03-17T11:41:27.531724+0000","172.21.15.29:0/1281028285":"2024-03-17T11:41:00.505678+0000","172.21.15.29:6800/2275315654":"2024-03-17T11:41:00.505678+0000","172.21.15.29:6801/2798831128":"2024-03-17T11:41:27.531724+0000","172.21.15.29:6801/2275315654":"2024-03-17T11:41:00.505678+0000","172.21.15.29:0/1330100009":"2024-03-17T11:41:00.505678+0000","172.21.15.29:0/1784249039":"2024-03-17T11:41:00.505678+0000","172.21.15.29:0/1116486932":"2024-03-17T11:41:27.531724+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-16T11:50:38.529 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:38.530+0000 7fc7d795b640 1 -- 172.21.15.29:0/2790331527 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fc7bc0a2800 msgr2=0x7fc7bc0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:38.529 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:38.530+0000 7fc7d795b640 1 --2- 172.21.15.29:0/2790331527 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fc7bc0a2800 0x7fc7bc0a4cc0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fc7d019b150 tx=0x7fc7cc005f90 comp rx=0 tx=0).stop 2024-03-16T11:50:38.530 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:38.530+0000 7fc7d795b640 1 -- 172.21.15.29:0/2790331527 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc7d01009d0 msgr2=0x7fc7d019a1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:38.530 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:38.530+0000 7fc7d795b640 1 --2- 172.21.15.29:0/2790331527 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc7d01009d0 0x7fc7d019a1c0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fc7c000e9f0 tx=0x7fc7c000eeb0 comp rx=0 tx=0).stop 2024-03-16T11:50:38.530 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:38.530+0000 7fc7d795b640 1 -- 172.21.15.29:0/2790331527 shutdown_connections 2024-03-16T11:50:38.530 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:38.530+0000 7fc7d795b640 1 --2- 172.21.15.29:0/2790331527 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fc7bc0a2800 0x7fc7bc0a4cc0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:38.530 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:38.530+0000 7fc7d795b640 1 --2- 172.21.15.29:0/2790331527 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc7d0101370 0x7fc7d01a11f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:38.530 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:38.530+0000 7fc7d795b640 1 --2- 172.21.15.29:0/2790331527 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc7d01009d0 0x7fc7d019a1c0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:38.530 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:38.530+0000 7fc7d795b640 1 --2- 172.21.15.29:0/2790331527 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc7d0100070 0x7fc7d0199c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:38.530 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:38.530+0000 7fc7d795b640 1 -- 172.21.15.29:0/2790331527 >> 172.21.15.29:0/2790331527 conn(0x7fc7d00fbc70 msgr2=0x7fc7d00fd340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:50:38.530 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:38.530+0000 7fc7d795b640 1 -- 172.21.15.29:0/2790331527 shutdown_connections 2024-03-16T11:50:38.530 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:38.530+0000 7fc7d795b640 1 -- 172.21.15.29:0/2790331527 wait complete. 2024-03-16T11:50:38.541 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-16T11:43:51.892652+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-16T11:48:36.709732+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-16T11:48:37.586140+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-16T11:50:38.541 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-16T11:50:38.650 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:38.649+0000 7f9224797640 1 -- 172.21.15.29:0/2803748418 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f921c1022b0 msgr2=0x7f921c108b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:38.650 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:38.649+0000 7f9224797640 1 --2- 172.21.15.29:0/2803748418 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f921c1022b0 0x7f921c108b40 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f920c009ca0 tx=0x7f920c02c0f0 comp rx=0 tx=0).stop 2024-03-16T11:50:38.651 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:38.649+0000 7f9224797640 1 -- 172.21.15.29:0/2803748418 shutdown_connections 2024-03-16T11:50:38.651 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:38.649+0000 7f9224797640 1 --2- 172.21.15.29:0/2803748418 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f921c1022b0 0x7f921c108b40 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:38.651 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:38.649+0000 7f9224797640 1 --2- 172.21.15.29:0/2803748418 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f921c1018f0 0x7f921c101d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:38.651 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:38.649+0000 7f9224797640 1 --2- 172.21.15.29:0/2803748418 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f921c100700 0x7f921c100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:38.651 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:38.649+0000 7f9224797640 1 -- 172.21.15.29:0/2803748418 >> 172.21.15.29:0/2803748418 conn(0x7f921c0fbc90 msgr2=0x7f921c0fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:50:38.651 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:38.649+0000 7f9224797640 1 -- 172.21.15.29:0/2803748418 shutdown_connections 2024-03-16T11:50:38.651 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:38.649+0000 7f9224797640 1 -- 172.21.15.29:0/2803748418 wait complete. 2024-03-16T11:50:38.651 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:38.650+0000 7f9224797640 1 Processor -- start 2024-03-16T11:50:38.651 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:38.650+0000 7f9224797640 1 -- start start 2024-03-16T11:50:38.651 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:38.650+0000 7f9224797640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f921c100700 0x7f921c104690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:38.651 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:38.650+0000 7f9224797640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f921c1018f0 0x7f921c102cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:38.651 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:38.650+0000 7f9224797640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f921c1022b0 0x7f921c103210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:38.651 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:38.650+0000 7f9224797640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f921c10c360 con 0x7f921c1022b0 2024-03-16T11:50:38.651 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:38.650+0000 7f9224797640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f921c10c1e0 con 0x7f921c1018f0 2024-03-16T11:50:38.651 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:38.650+0000 7f9224797640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f921c10c4e0 con 0x7f921c100700 2024-03-16T11:50:38.651 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:38.650+0000 7f9222d34640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f921c1022b0 0x7f921c103210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:38.652 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:38.650+0000 7f9222d34640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f921c1022b0 0x7f921c103210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59436/0 (socket says 172.21.15.29:59436) 2024-03-16T11:50:38.652 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:38.650+0000 7f9222d34640 1 -- 172.21.15.29:0/139782518 learned_addr learned my addr 172.21.15.29:0/139782518 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:50:38.652 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:38.650+0000 7f9222533640 1 --2- 172.21.15.29:0/139782518 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f921c100700 0x7f921c104690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:38.652 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:38.650+0000 7f9221d32640 1 --2- 172.21.15.29:0/139782518 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f921c1018f0 0x7f921c102cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:38.652 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:38.650+0000 7f9222d34640 1 -- 172.21.15.29:0/139782518 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f921c100700 msgr2=0x7f921c104690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:38.652 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:38.650+0000 7f9222d34640 1 --2- 172.21.15.29:0/139782518 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f921c100700 0x7f921c104690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:38.652 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:38.650+0000 7f9222d34640 1 -- 172.21.15.29:0/139782518 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f921c1018f0 msgr2=0x7f921c102cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:38.652 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:38.650+0000 7f9222d34640 1 --2- 172.21.15.29:0/139782518 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f921c1018f0 0x7f921c102cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:38.652 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:38.650+0000 7f9222d34640 1 -- 172.21.15.29:0/139782518 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f921c103aa0 con 0x7f921c1022b0 2024-03-16T11:50:38.652 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:38.651+0000 7f9221d32640 1 --2- 172.21.15.29:0/139782518 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f921c1018f0 0x7f921c102cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T11:50:38.652 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:38.651+0000 7f9222d34640 1 --2- 172.21.15.29:0/139782518 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f921c1022b0 0x7f921c103210 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f920c032530 tx=0x7f920c032f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:50:38.652 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:38.651+0000 7f921b7fe640 1 -- 172.21.15.29:0/139782518 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f920c041020 con 0x7f921c1022b0 2024-03-16T11:50:38.652 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:38.651+0000 7f921b7fe640 1 -- 172.21.15.29:0/139782518 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f920c034cd0 con 0x7f921c1022b0 2024-03-16T11:50:38.652 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:38.651+0000 7f9224797640 1 -- 172.21.15.29:0/139782518 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f921c1a1bf0 con 0x7f921c1022b0 2024-03-16T11:50:38.652 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:38.651+0000 7f921b7fe640 1 -- 172.21.15.29:0/139782518 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f920c039040 con 0x7f921c1022b0 2024-03-16T11:50:38.652 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:38.651+0000 7f9224797640 1 -- 172.21.15.29:0/139782518 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f921c1a2110 con 0x7f921c1022b0 2024-03-16T11:50:38.653 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:38.652+0000 7f9224797640 1 -- 172.21.15.29:0/139782518 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f921c1a22e0 con 0x7f921c1022b0 2024-03-16T11:50:38.653 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:38.653+0000 7f921b7fe640 1 -- 172.21.15.29:0/139782518 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f920c044020 con 0x7f921c1022b0 2024-03-16T11:50:38.654 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:38.654+0000 7f921b7fe640 1 --2- 172.21.15.29:0/139782518 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f92040a2800 0x7f92040a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:38.654 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:38.654+0000 7f9222533640 1 --2- 172.21.15.29:0/139782518 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f92040a2800 0x7f92040a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:38.654 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:38.654+0000 7f921b7fe640 1 -- 172.21.15.29:0/139782518 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8211+0+0 (secure 0 0 0) 0x7f920c0f0d50 con 0x7f921c1022b0 2024-03-16T11:50:38.654 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:38.655+0000 7f9222533640 1 --2- 172.21.15.29:0/139782518 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f92040a2800 0x7f92040a4cc0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f921c101750 tx=0x7f9214005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:50:38.657 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:38.658+0000 7f921b7fe640 1 -- 172.21.15.29:0/139782518 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f920c04b020 con 0x7f921c1022b0 2024-03-16T11:50:38.837 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:50:38 smithi029 ceph-mon[31975]: from='client.? 172.21.15.29:0/4183524737' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-16T11:50:38.837 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:50:38 smithi029 ceph-mon[31975]: from='client.? 172.21.15.29:0/3500465737' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T11:50:38.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:50:38 smithi029 ceph-mon[31975]: from='client.? 172.21.15.29:0/2790331527' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T11:50:38.840 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:38.841+0000 7f9224797640 1 -- 172.21.15.29:0/139782518 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f921c1a2670 con 0x7f921c1022b0 2024-03-16T11:50:38.840 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:38.841+0000 7f921b7fe640 1 -- 172.21.15.29:0/139782518 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v87) ==== 93+0+10 (secure 0 0 0) 0x7f920c0bdda0 con 0x7f921c1022b0 2024-03-16T11:50:38.840 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2024-03-16T11:50:38.843 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:38.843+0000 7f9224797640 1 -- 172.21.15.29:0/139782518 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f92040a2800 msgr2=0x7f92040a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:38.843 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:38.843+0000 7f9224797640 1 --2- 172.21.15.29:0/139782518 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f92040a2800 0x7f92040a4cc0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f921c101750 tx=0x7f9214005f90 comp rx=0 tx=0).stop 2024-03-16T11:50:38.843 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:38.843+0000 7f9224797640 1 -- 172.21.15.29:0/139782518 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f921c1022b0 msgr2=0x7f921c103210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:38.843 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:38.843+0000 7f9224797640 1 --2- 172.21.15.29:0/139782518 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f921c1022b0 0x7f921c103210 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f920c032530 tx=0x7f920c032f20 comp rx=0 tx=0).stop 2024-03-16T11:50:38.843 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:38.843+0000 7f9224797640 1 -- 172.21.15.29:0/139782518 shutdown_connections 2024-03-16T11:50:38.843 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:38.843+0000 7f9224797640 1 --2- 172.21.15.29:0/139782518 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f92040a2800 0x7f92040a4cc0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:38.843 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:38.843+0000 7f9224797640 1 --2- 172.21.15.29:0/139782518 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f921c1022b0 0x7f921c103210 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:38.843 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:38.843+0000 7f9224797640 1 --2- 172.21.15.29:0/139782518 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f921c1018f0 0x7f921c102cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:38.843 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:38.843+0000 7f9224797640 1 --2- 172.21.15.29:0/139782518 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f921c100700 0x7f921c104690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:38.843 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:38.843+0000 7f9224797640 1 -- 172.21.15.29:0/139782518 >> 172.21.15.29:0/139782518 conn(0x7f921c0fbc90 msgr2=0x7f921c0fd6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:50:38.844 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:38.844+0000 7f9224797640 1 -- 172.21.15.29:0/139782518 shutdown_connections 2024-03-16T11:50:38.844 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:38.844+0000 7f9224797640 1 -- 172.21.15.29:0/139782518 wait complete. 2024-03-16T11:50:38.853 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-16T11:50:38.961 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:38.960+0000 7fb229dc4640 1 -- 172.21.15.29:0/2147564696 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb224102290 msgr2=0x7fb224108b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:38.961 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:38.960+0000 7fb229dc4640 1 --2- 172.21.15.29:0/2147564696 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb224102290 0x7fb224108b20 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fb21c009ca0 tx=0x7fb21c02c0f0 comp rx=0 tx=0).stop 2024-03-16T11:50:38.961 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:38.960+0000 7fb229dc4640 1 -- 172.21.15.29:0/2147564696 shutdown_connections 2024-03-16T11:50:38.961 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:38.960+0000 7fb229dc4640 1 --2- 172.21.15.29:0/2147564696 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb224102290 0x7fb224108b20 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:38.961 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:38.960+0000 7fb229dc4640 1 --2- 172.21.15.29:0/2147564696 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb2241018d0 0x7fb224101d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:38.961 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:38.960+0000 7fb229dc4640 1 --2- 172.21.15.29:0/2147564696 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb2241006e0 0x7fb224100b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:38.961 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:38.960+0000 7fb229dc4640 1 -- 172.21.15.29:0/2147564696 >> 172.21.15.29:0/2147564696 conn(0x7fb2240fbc70 msgr2=0x7fb2240fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:50:38.961 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:38.960+0000 7fb229dc4640 1 -- 172.21.15.29:0/2147564696 shutdown_connections 2024-03-16T11:50:38.961 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:38.960+0000 7fb229dc4640 1 -- 172.21.15.29:0/2147564696 wait complete. 2024-03-16T11:50:38.962 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:38.960+0000 7fb229dc4640 1 Processor -- start 2024-03-16T11:50:38.962 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:38.960+0000 7fb229dc4640 1 -- start start 2024-03-16T11:50:38.962 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:38.960+0000 7fb229dc4640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb2241006e0 0x7fb224197aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:38.962 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:38.960+0000 7fb229dc4640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb2241018d0 0x7fb224197fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:38.962 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:38.960+0000 7fb229dc4640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb224102290 0x7fb22419f050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:38.962 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:38.960+0000 7fb229dc4640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fb22410d480 con 0x7fb224102290 2024-03-16T11:50:38.962 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:38.960+0000 7fb229dc4640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fb22410d300 con 0x7fb2241018d0 2024-03-16T11:50:38.962 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:38.961+0000 7fb229dc4640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fb22410d600 con 0x7fb2241006e0 2024-03-16T11:50:38.962 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:38.961+0000 7fb223fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb224102290 0x7fb22419f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:38.962 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:38.961+0000 7fb223fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb224102290 0x7fb22419f050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59456/0 (socket says 172.21.15.29:59456) 2024-03-16T11:50:38.962 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:38.961+0000 7fb223fff640 1 -- 172.21.15.29:0/3268742104 learned_addr learned my addr 172.21.15.29:0/3268742104 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:50:38.962 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:38.961+0000 7fb2237fe640 1 --2- 172.21.15.29:0/3268742104 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb2241006e0 0x7fb224197aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:38.962 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:38.961+0000 7fb222ffd640 1 --2- 172.21.15.29:0/3268742104 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb2241018d0 0x7fb224197fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:38.962 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:38.961+0000 7fb223fff640 1 -- 172.21.15.29:0/3268742104 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb2241006e0 msgr2=0x7fb224197aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:38.963 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:38.961+0000 7fb223fff640 1 --2- 172.21.15.29:0/3268742104 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb2241006e0 0x7fb224197aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:38.963 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:38.961+0000 7fb223fff640 1 -- 172.21.15.29:0/3268742104 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb2241018d0 msgr2=0x7fb224197fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:38.963 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:38.961+0000 7fb223fff640 1 --2- 172.21.15.29:0/3268742104 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb2241018d0 0x7fb224197fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:38.963 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:38.961+0000 7fb223fff640 1 -- 172.21.15.29:0/3268742104 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb22419f6c0 con 0x7fb224102290 2024-03-16T11:50:38.963 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:38.961+0000 7fb223fff640 1 --2- 172.21.15.29:0/3268742104 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb224102290 0x7fb22419f050 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fb21c037040 tx=0x7fb21c032f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:50:38.963 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:38.961+0000 7fb220ff9640 1 -- 172.21.15.29:0/3268742104 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb21c041020 con 0x7fb224102290 2024-03-16T11:50:38.963 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:38.961+0000 7fb220ff9640 1 -- 172.21.15.29:0/3268742104 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb21c034cd0 con 0x7fb224102290 2024-03-16T11:50:38.963 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:38.961+0000 7fb229dc4640 1 -- 172.21.15.29:0/3268742104 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb22419f950 con 0x7fb224102290 2024-03-16T11:50:38.963 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:38.961+0000 7fb220ff9640 1 -- 172.21.15.29:0/3268742104 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb21c039040 con 0x7fb224102290 2024-03-16T11:50:38.963 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:38.961+0000 7fb229dc4640 1 -- 172.21.15.29:0/3268742104 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb22419fe90 con 0x7fb224102290 2024-03-16T11:50:38.964 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:38.962+0000 7fb229dc4640 1 -- 172.21.15.29:0/3268742104 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb224107a00 con 0x7fb224102290 2024-03-16T11:50:38.964 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:38.964+0000 7fb220ff9640 1 -- 172.21.15.29:0/3268742104 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fb21c044020 con 0x7fb224102290 2024-03-16T11:50:38.964 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:38.964+0000 7fb220ff9640 1 --2- 172.21.15.29:0/3268742104 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fb20c0a2880 0x7fb20c0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:38.964 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:38.964+0000 7fb2237fe640 1 --2- 172.21.15.29:0/3268742104 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fb20c0a2880 0x7fb20c0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:38.964 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:38.965+0000 7fb220ff9640 1 -- 172.21.15.29:0/3268742104 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8211+0+0 (secure 0 0 0) 0x7fb21c0f0c40 con 0x7fb224102290 2024-03-16T11:50:38.964 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:38.965+0000 7fb2237fe640 1 --2- 172.21.15.29:0/3268742104 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fb20c0a2880 0x7fb20c0a4d40 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fb224101730 tx=0x7fb214005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:50:38.968 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:38.969+0000 7fb220ff9640 1 -- 172.21.15.29:0/3268742104 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fb21c04b020 con 0x7fb224102290 2024-03-16T11:50:39.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:50:38 smithi154 ceph-mon[36320]: from='client.? 172.21.15.29:0/4183524737' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-16T11:50:39.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:50:38 smithi154 ceph-mon[36320]: from='client.? 172.21.15.29:0/3500465737' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T11:50:39.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:50:38 smithi154 ceph-mon[36320]: from='client.? 172.21.15.29:0/2790331527' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T11:50:39.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:50:38 smithi152 ceph-mon[36291]: from='client.? 172.21.15.29:0/4183524737' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-16T11:50:39.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:50:38 smithi152 ceph-mon[36291]: from='client.? 172.21.15.29:0/3500465737' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T11:50:39.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:50:38 smithi152 ceph-mon[36291]: from='client.? 172.21.15.29:0/2790331527' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T11:50:39.153 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:39.154+0000 7fb229dc4640 1 -- 172.21.15.29:0/3268742104 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fb22402d1b0 con 0x7fb224102290 2024-03-16T11:50:39.154 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:39.155+0000 7fb220ff9640 1 -- 172.21.15.29:0/3268742104 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v87) ==== 104+0+11 (secure 0 0 0) 0x7fb21c0bdd20 con 0x7fb224102290 2024-03-16T11:50:39.154 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 32 2024-03-16T11:50:39.156 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:39.157+0000 7fb229dc4640 1 -- 172.21.15.29:0/3268742104 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fb20c0a2880 msgr2=0x7fb20c0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:39.156 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:39.157+0000 7fb229dc4640 1 --2- 172.21.15.29:0/3268742104 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fb20c0a2880 0x7fb20c0a4d40 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fb224101730 tx=0x7fb214005f90 comp rx=0 tx=0).stop 2024-03-16T11:50:39.156 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:39.157+0000 7fb229dc4640 1 -- 172.21.15.29:0/3268742104 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb224102290 msgr2=0x7fb22419f050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:39.157 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:39.157+0000 7fb229dc4640 1 --2- 172.21.15.29:0/3268742104 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb224102290 0x7fb22419f050 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fb21c037040 tx=0x7fb21c032f40 comp rx=0 tx=0).stop 2024-03-16T11:50:39.157 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:39.157+0000 7fb229dc4640 1 -- 172.21.15.29:0/3268742104 shutdown_connections 2024-03-16T11:50:39.157 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:39.157+0000 7fb229dc4640 1 --2- 172.21.15.29:0/3268742104 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fb20c0a2880 0x7fb20c0a4d40 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:39.157 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:39.157+0000 7fb229dc4640 1 --2- 172.21.15.29:0/3268742104 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb224102290 0x7fb22419f050 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:39.157 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:39.157+0000 7fb229dc4640 1 --2- 172.21.15.29:0/3268742104 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb2241018d0 0x7fb224197fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:39.157 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:39.157+0000 7fb229dc4640 1 --2- 172.21.15.29:0/3268742104 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb2241006e0 0x7fb224197aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:39.157 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:39.157+0000 7fb229dc4640 1 -- 172.21.15.29:0/3268742104 >> 172.21.15.29:0/3268742104 conn(0x7fb2240fbc70 msgr2=0x7fb224104bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:50:39.157 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:39.157+0000 7fb229dc4640 1 -- 172.21.15.29:0/3268742104 shutdown_connections 2024-03-16T11:50:39.157 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:39.157+0000 7fb229dc4640 1 -- 172.21.15.29:0/3268742104 wait complete. 2024-03-16T11:50:39.167 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-16T11:50:39.275 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:39.273+0000 7f9c09913640 1 -- 172.21.15.29:0/169398661 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9c041018d0 msgr2=0x7f9c04101d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:39.275 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:39.273+0000 7f9c09913640 1 --2- 172.21.15.29:0/169398661 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9c041018d0 0x7f9c04101d50 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f9bec009900 tx=0x7f9bec02bc30 comp rx=0 tx=0).stop 2024-03-16T11:50:39.275 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:39.273+0000 7f9c09913640 1 -- 172.21.15.29:0/169398661 shutdown_connections 2024-03-16T11:50:39.275 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:39.273+0000 7f9c09913640 1 --2- 172.21.15.29:0/169398661 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9c04102290 0x7f9c04108b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:39.275 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:39.273+0000 7f9c09913640 1 --2- 172.21.15.29:0/169398661 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9c041018d0 0x7f9c04101d50 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:39.275 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:39.273+0000 7f9c09913640 1 --2- 172.21.15.29:0/169398661 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9c041006e0 0x7f9c04100b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:39.275 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:39.273+0000 7f9c09913640 1 -- 172.21.15.29:0/169398661 >> 172.21.15.29:0/169398661 conn(0x7f9c040fbc70 msgr2=0x7f9c040fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:50:39.275 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:39.273+0000 7f9c09913640 1 -- 172.21.15.29:0/169398661 shutdown_connections 2024-03-16T11:50:39.275 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:39.273+0000 7f9c09913640 1 -- 172.21.15.29:0/169398661 wait complete. 2024-03-16T11:50:39.275 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:39.274+0000 7f9c09913640 1 Processor -- start 2024-03-16T11:50:39.275 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:39.274+0000 7f9c09913640 1 -- start start 2024-03-16T11:50:39.276 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:39.274+0000 7f9c09913640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9c041006e0 0x7f9c040694b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:39.276 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:39.274+0000 7f9c09913640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9c041018d0 0x7f9c0419c870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:39.276 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:39.274+0000 7f9c09913640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9c04102290 0x7f9c0419ee90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:39.276 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:39.274+0000 7f9c09913640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f9c0410c360 con 0x7f9c04102290 2024-03-16T11:50:39.276 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:39.274+0000 7f9c09913640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f9c0410c1e0 con 0x7f9c041006e0 2024-03-16T11:50:39.276 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:39.274+0000 7f9c09913640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f9c0410c4e0 con 0x7f9c041018d0 2024-03-16T11:50:39.276 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:39.274+0000 7f9c037fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9c04102290 0x7f9c0419ee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:39.276 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:39.274+0000 7f9c027fc640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9c041018d0 0x7f9c0419c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:39.276 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:39.274+0000 7f9c037fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9c04102290 0x7f9c0419ee90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59466/0 (socket says 172.21.15.29:59466) 2024-03-16T11:50:39.276 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:39.274+0000 7f9c037fe640 1 -- 172.21.15.29:0/3967230018 learned_addr learned my addr 172.21.15.29:0/3967230018 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:50:39.276 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:39.274+0000 7f9c02ffd640 1 --2- 172.21.15.29:0/3967230018 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9c041006e0 0x7f9c040694b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:39.276 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:39.275+0000 7f9c037fe640 1 -- 172.21.15.29:0/3967230018 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9c041018d0 msgr2=0x7f9c0419c870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:39.276 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:39.275+0000 7f9c037fe640 1 --2- 172.21.15.29:0/3967230018 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9c041018d0 0x7f9c0419c870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:39.276 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:39.275+0000 7f9c037fe640 1 -- 172.21.15.29:0/3967230018 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9c041006e0 msgr2=0x7f9c040694b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:39.276 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:39.275+0000 7f9c037fe640 1 --2- 172.21.15.29:0/3967230018 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9c041006e0 0x7f9c040694b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:39.277 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:39.275+0000 7f9c037fe640 1 -- 172.21.15.29:0/3967230018 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c0419f7a0 con 0x7f9c04102290 2024-03-16T11:50:39.277 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:39.275+0000 7f9c02ffd640 1 --2- 172.21.15.29:0/3967230018 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9c041006e0 0x7f9c040694b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:50:39.277 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:39.275+0000 7f9c037fe640 1 --2- 172.21.15.29:0/3967230018 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9c04102290 0x7f9c0419ee90 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f9bf400e970 tx=0x7f9bf400ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:50:39.277 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:39.275+0000 7f9c08911640 1 -- 172.21.15.29:0/3967230018 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bf400ccc0 con 0x7f9c04102290 2024-03-16T11:50:39.277 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:39.275+0000 7f9c08911640 1 -- 172.21.15.29:0/3967230018 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9bf400ce60 con 0x7f9c04102290 2024-03-16T11:50:39.277 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:39.275+0000 7f9c08911640 1 -- 172.21.15.29:0/3967230018 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bf4010490 con 0x7f9c04102290 2024-03-16T11:50:39.277 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:39.275+0000 7f9c09913640 1 -- 172.21.15.29:0/3967230018 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9c0419fa30 con 0x7f9c04102290 2024-03-16T11:50:39.277 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:39.275+0000 7f9c09913640 1 -- 172.21.15.29:0/3967230018 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9c0419ff50 con 0x7f9c04102290 2024-03-16T11:50:39.277 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:39.276+0000 7f9c09913640 1 -- 172.21.15.29:0/3967230018 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9c04107ad0 con 0x7f9c04102290 2024-03-16T11:50:39.278 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:39.277+0000 7f9c08911640 1 -- 172.21.15.29:0/3967230018 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f9bf4005020 con 0x7f9c04102290 2024-03-16T11:50:39.278 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:39.278+0000 7f9c08911640 1 --2- 172.21.15.29:0/3967230018 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f9bf00a2880 0x7f9bf00a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:39.278 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:39.278+0000 7f9c02ffd640 1 --2- 172.21.15.29:0/3967230018 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f9bf00a2880 0x7f9bf00a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:39.278 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:39.278+0000 7f9c08911640 1 -- 172.21.15.29:0/3967230018 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8211+0+0 (secure 0 0 0) 0x7f9bf40cee90 con 0x7f9c04102290 2024-03-16T11:50:39.278 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:39.278+0000 7f9c02ffd640 1 --2- 172.21.15.29:0/3967230018 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f9bf00a2880 0x7f9bf00a4d40 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f9c04101730 tx=0x7f9bfc009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:50:39.281 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:39.282+0000 7f9c08911640 1 -- 172.21.15.29:0/3967230018 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f9bf409bee0 con 0x7f9c04102290 2024-03-16T11:50:39.463 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:39.464+0000 7f9c09913640 1 -- 172.21.15.29:0/3967230018 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f9c04062480 con 0x7f9c04102290 2024-03-16T11:50:39.463 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:39.464+0000 7f9c08911640 1 -- 172.21.15.29:0/3967230018 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v87) ==== 100+0+11 (secure 0 0 0) 0x7f9bf409b7e0 con 0x7f9c04102290 2024-03-16T11:50:39.463 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 64 2024-03-16T11:50:39.465 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:39.466+0000 7f9c09913640 1 -- 172.21.15.29:0/3967230018 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f9bf00a2880 msgr2=0x7f9bf00a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:39.465 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:39.466+0000 7f9c09913640 1 --2- 172.21.15.29:0/3967230018 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f9bf00a2880 0x7f9bf00a4d40 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f9c04101730 tx=0x7f9bfc009500 comp rx=0 tx=0).stop 2024-03-16T11:50:39.465 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:39.466+0000 7f9c09913640 1 -- 172.21.15.29:0/3967230018 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9c04102290 msgr2=0x7f9c0419ee90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:39.465 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:39.466+0000 7f9c09913640 1 --2- 172.21.15.29:0/3967230018 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9c04102290 0x7f9c0419ee90 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f9bf400e970 tx=0x7f9bf400ed70 comp rx=0 tx=0).stop 2024-03-16T11:50:39.466 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:39.466+0000 7f9c09913640 1 -- 172.21.15.29:0/3967230018 shutdown_connections 2024-03-16T11:50:39.466 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:39.466+0000 7f9c09913640 1 --2- 172.21.15.29:0/3967230018 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f9bf00a2880 0x7f9bf00a4d40 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:39.466 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:39.466+0000 7f9c09913640 1 --2- 172.21.15.29:0/3967230018 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9c04102290 0x7f9c0419ee90 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:39.466 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:39.466+0000 7f9c09913640 1 --2- 172.21.15.29:0/3967230018 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9c041018d0 0x7f9c0419c870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:39.466 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:39.466+0000 7f9c09913640 1 --2- 172.21.15.29:0/3967230018 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9c041006e0 0x7f9c040694b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:39.466 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:39.466+0000 7f9c09913640 1 -- 172.21.15.29:0/3967230018 >> 172.21.15.29:0/3967230018 conn(0x7f9c040fbc70 msgr2=0x7f9c04104ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:50:39.466 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:39.466+0000 7f9c09913640 1 -- 172.21.15.29:0/3967230018 shutdown_connections 2024-03-16T11:50:39.466 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:39.466+0000 7f9c09913640 1 -- 172.21.15.29:0/3967230018 wait complete. 2024-03-16T11:50:39.476 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T11:50:39.581 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:39.580+0000 7fe21c63f640 1 -- 172.21.15.29:0/1423315589 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe214100700 msgr2=0x7fe214100b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:39.581 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:39.580+0000 7fe21c63f640 1 --2- 172.21.15.29:0/1423315589 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe214100700 0x7fe214100b20 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fe20c009900 tx=0x7fe20c02bc30 comp rx=0 tx=0).stop 2024-03-16T11:50:39.581 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:39.580+0000 7fe21c63f640 1 -- 172.21.15.29:0/1423315589 shutdown_connections 2024-03-16T11:50:39.581 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:39.580+0000 7fe21c63f640 1 --2- 172.21.15.29:0/1423315589 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe2141022b0 0x7fe214108b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:39.581 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:39.580+0000 7fe21c63f640 1 --2- 172.21.15.29:0/1423315589 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe2141018f0 0x7fe214101d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:39.581 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:39.580+0000 7fe21c63f640 1 --2- 172.21.15.29:0/1423315589 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe214100700 0x7fe214100b20 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:39.581 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:39.580+0000 7fe21c63f640 1 -- 172.21.15.29:0/1423315589 >> 172.21.15.29:0/1423315589 conn(0x7fe2140fbc70 msgr2=0x7fe2140fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:50:39.581 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:39.580+0000 7fe21c63f640 1 -- 172.21.15.29:0/1423315589 shutdown_connections 2024-03-16T11:50:39.581 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:39.580+0000 7fe21c63f640 1 -- 172.21.15.29:0/1423315589 wait complete. 2024-03-16T11:50:39.581 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:39.580+0000 7fe21c63f640 1 Processor -- start 2024-03-16T11:50:39.581 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:39.580+0000 7fe21c63f640 1 -- start start 2024-03-16T11:50:39.581 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:39.580+0000 7fe21c63f640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe214100700 0x7fe214197b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:39.581 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:39.580+0000 7fe21c63f640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe2141018f0 0x7fe214198050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:39.581 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:39.580+0000 7fe21c63f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe2141022b0 0x7fe21419f0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:39.582 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:39.580+0000 7fe21c63f640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fe2140694d0 con 0x7fe2141022b0 2024-03-16T11:50:39.582 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:39.580+0000 7fe21c63f640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fe214069350 con 0x7fe2141018f0 2024-03-16T11:50:39.582 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:39.580+0000 7fe21c63f640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fe214069650 con 0x7fe214100700 2024-03-16T11:50:39.582 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:39.581+0000 7fe21abdc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe2141022b0 0x7fe21419f0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:39.582 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:39.581+0000 7fe21a3db640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe214100700 0x7fe214197b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:39.582 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:39.581+0000 7fe21abdc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe2141022b0 0x7fe21419f0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39640/0 (socket says 172.21.15.29:39640) 2024-03-16T11:50:39.582 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:39.581+0000 7fe219bda640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe2141018f0 0x7fe214198050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:39.582 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:39.581+0000 7fe21a3db640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe214100700 0x7fe214197b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.29:60812/0 (socket says 172.21.15.29:60812) 2024-03-16T11:50:39.582 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:39.581+0000 7fe21abdc640 1 -- 172.21.15.29:0/4247670357 learned_addr learned my addr 172.21.15.29:0/4247670357 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:50:39.582 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:39.581+0000 7fe21abdc640 1 -- 172.21.15.29:0/4247670357 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe214100700 msgr2=0x7fe214197b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:39.582 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:39.581+0000 7fe21abdc640 1 --2- 172.21.15.29:0/4247670357 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe214100700 0x7fe214197b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:39.582 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:39.581+0000 7fe21abdc640 1 -- 172.21.15.29:0/4247670357 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe2141018f0 msgr2=0x7fe214198050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:39.582 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:39.581+0000 7fe21abdc640 1 --2- 172.21.15.29:0/4247670357 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe2141018f0 0x7fe214198050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:39.582 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:39.581+0000 7fe21abdc640 1 -- 172.21.15.29:0/4247670357 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe21419f730 con 0x7fe2141022b0 2024-03-16T11:50:39.583 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:39.581+0000 7fe219bda640 1 --2- 172.21.15.29:0/4247670357 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe2141018f0 0x7fe214198050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:50:39.583 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:39.581+0000 7fe21abdc640 1 --2- 172.21.15.29:0/4247670357 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe2141022b0 0x7fe21419f0c0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fe20400eb10 tx=0x7fe20400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:50:39.583 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:39.581+0000 7fe2137fe640 1 -- 172.21.15.29:0/4247670357 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe204004b20 con 0x7fe2141022b0 2024-03-16T11:50:39.583 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:39.581+0000 7fe21c63f640 1 -- 172.21.15.29:0/4247670357 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe21419fa20 con 0x7fe2141022b0 2024-03-16T11:50:39.583 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:39.581+0000 7fe2137fe640 1 -- 172.21.15.29:0/4247670357 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe204004cc0 con 0x7fe2141022b0 2024-03-16T11:50:39.583 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:39.581+0000 7fe2137fe640 1 -- 172.21.15.29:0/4247670357 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe204010760 con 0x7fe2141022b0 2024-03-16T11:50:39.583 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:39.581+0000 7fe21c63f640 1 -- 172.21.15.29:0/4247670357 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe21419ff20 con 0x7fe2141022b0 2024-03-16T11:50:39.584 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:39.582+0000 7fe21c63f640 1 -- 172.21.15.29:0/4247670357 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe214107a20 con 0x7fe2141022b0 2024-03-16T11:50:39.584 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:39.583+0000 7fe2137fe640 1 -- 172.21.15.29:0/4247670357 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fe204005020 con 0x7fe2141022b0 2024-03-16T11:50:39.584 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:39.584+0000 7fe2137fe640 1 --2- 172.21.15.29:0/4247670357 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fe1fc0a2800 0x7fe1fc0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:39.584 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:39.584+0000 7fe2137fe640 1 -- 172.21.15.29:0/4247670357 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8211+0+0 (secure 0 0 0) 0x7fe2040cefb0 con 0x7fe2141022b0 2024-03-16T11:50:39.584 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:39.584+0000 7fe21a3db640 1 --2- 172.21.15.29:0/4247670357 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fe1fc0a2800 0x7fe1fc0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:39.584 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:39.585+0000 7fe21a3db640 1 --2- 172.21.15.29:0/4247670357 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fe1fc0a2800 0x7fe1fc0a4cc0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fe20c000c00 tx=0x7fe20c005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:50:39.588 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:39.589+0000 7fe2137fe640 1 -- 172.21.15.29:0/4247670357 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fe20409c060 con 0x7fe2141022b0 2024-03-16T11:50:39.786 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:50:39 smithi029 ceph-mon[31975]: from='client.? 172.21.15.29:0/139782518' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-16T11:50:39.786 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:50:39 smithi029 ceph-mon[31975]: from='client.? 172.21.15.29:0/3268742104' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-16T11:50:39.786 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:50:39 smithi029 ceph-mon[31975]: from='client.? 172.21.15.29:0/3967230018' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-16T11:50:39.786 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:50:39 smithi029 ceph-mon[31975]: pgmap v353: 97 pgs: 97 active+clean; 640 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-03-16T11:50:39.810 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:39.811+0000 7fe21c63f640 1 -- 172.21.15.29:0/4247670357 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe2140623e0 con 0x7fe2141022b0 2024-03-16T11:50:39.811 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:39.812+0000 7fe2137fe640 1 -- 172.21.15.29:0/4247670357 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+11875 (secure 0 0 0) 0x7fe20409b960 con 0x7fe2141022b0 2024-03-16T11:50:39.811 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-16T11:50:39.811 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14574,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/3505811955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":3505811955},{"type":"v1","addr":"172.21.15.29:6835","nonce":3505811955}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14586,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/1901337868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":1901337868},{"type":"v1","addr":"172.21.15.29:6837","nonce":1901337868}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14652,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6839/544966534","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6838","nonce":544966534},{"type":"v1","addr":"172.21.15.29:6839","nonce":544966534}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14664,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6841/540705290","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6840","nonce":540705290},{"type":"v1","addr":"172.21.15.29:6841","nonce":540705290}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24407,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6833/3136024628","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6832","nonce":3136024628},{"type":"v1","addr":"172.21.15.152:6833","nonce":3136024628}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24421,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6833/2677426282","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":2677426282},{"type":"v1","addr":"172.21.15.154:6833","nonce":2677426282}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":21,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-16T11:48:38.431509+0000","modified":"2024-03-16T11:48:42.969084+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24503,"mds_1":24499,"mds_2":24464},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24434":{"gid":24434,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.152:6835/2919113178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6834","nonce":2919113178},{"type":"v1","addr":"172.21.15.152:6835","nonce":2919113178}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24464":{"gid":24464,"name":"h","rank":2,"incarnation":19,"state":"up:active","state_seq":7,"addr":"172.21.15.152:6837/1323866722","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6836","nonce":1323866722},{"type":"v1","addr":"172.21.15.152:6837","nonce":1323866722}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.154:6835/4165274681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":4165274681},{"type":"v1","addr":"172.21.15.154:6835","nonce":4165274681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.154:6837/1468560096","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":1468560096},{"type":"v1","addr":"172.21.15.154:6837","nonce":1468560096}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":1,"incarnation":17,"state":"up:active","state_seq":6,"addr":"172.21.15.154:6839/685736862","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":685736862},{"type":"v1","addr":"172.21.15.154:6839","nonce":685736862}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.152:6839/1548731833","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6838","nonce":1548731833},{"type":"v1","addr":"172.21.15.152:6839","nonce":1548731833}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24503,"qdb_cluster":[24503,24464,24499]},"id":1}]} 2024-03-16T11:50:39.811 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 21 2024-03-16T11:50:39.813 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:39.814+0000 7fe21c63f640 1 -- 172.21.15.29:0/4247670357 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fe1fc0a2800 msgr2=0x7fe1fc0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:39.813 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:39.814+0000 7fe21c63f640 1 --2- 172.21.15.29:0/4247670357 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fe1fc0a2800 0x7fe1fc0a4cc0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fe20c000c00 tx=0x7fe20c005fd0 comp rx=0 tx=0).stop 2024-03-16T11:50:39.813 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:39.814+0000 7fe21c63f640 1 -- 172.21.15.29:0/4247670357 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe2141022b0 msgr2=0x7fe21419f0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:39.813 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:39.814+0000 7fe21c63f640 1 --2- 172.21.15.29:0/4247670357 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe2141022b0 0x7fe21419f0c0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fe20400eb10 tx=0x7fe20400c2d0 comp rx=0 tx=0).stop 2024-03-16T11:50:39.813 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:39.814+0000 7fe21c63f640 1 -- 172.21.15.29:0/4247670357 shutdown_connections 2024-03-16T11:50:39.813 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:39.814+0000 7fe21c63f640 1 --2- 172.21.15.29:0/4247670357 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fe1fc0a2800 0x7fe1fc0a4cc0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:39.813 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:39.814+0000 7fe21c63f640 1 --2- 172.21.15.29:0/4247670357 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe2141022b0 0x7fe21419f0c0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:39.814 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:39.814+0000 7fe21c63f640 1 --2- 172.21.15.29:0/4247670357 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe2141018f0 0x7fe214198050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:39.814 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:39.814+0000 7fe21c63f640 1 --2- 172.21.15.29:0/4247670357 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe214100700 0x7fe214197b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:39.814 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:39.814+0000 7fe21c63f640 1 -- 172.21.15.29:0/4247670357 >> 172.21.15.29:0/4247670357 conn(0x7fe2140fbc70 msgr2=0x7fe214104bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:50:39.814 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:39.814+0000 7fe21c63f640 1 -- 172.21.15.29:0/4247670357 shutdown_connections 2024-03-16T11:50:39.814 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:39.814+0000 7fe21c63f640 1 -- 172.21.15.29:0/4247670357 wait complete. 2024-03-16T11:50:39.824 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-16T11:50:39.928 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:39.927+0000 7f16d1d97640 1 -- 172.21.15.29:0/1120873548 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f16cc1033b0 msgr2=0x7f16cc109c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:39.929 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:39.927+0000 7f16d1d97640 1 --2- 172.21.15.29:0/1120873548 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f16cc1033b0 0x7f16cc109c40 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f16c4009ca0 tx=0x7f16c402c0f0 comp rx=0 tx=0).stop 2024-03-16T11:50:39.929 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:39.927+0000 7f16d1d97640 1 -- 172.21.15.29:0/1120873548 shutdown_connections 2024-03-16T11:50:39.929 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:39.927+0000 7f16d1d97640 1 --2- 172.21.15.29:0/1120873548 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f16cc1033b0 0x7f16cc109c40 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:39.929 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:39.927+0000 7f16d1d97640 1 --2- 172.21.15.29:0/1120873548 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f16cc1029f0 0x7f16cc102e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:39.929 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:39.927+0000 7f16d1d97640 1 --2- 172.21.15.29:0/1120873548 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f16cc101800 0x7f16cc101c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:39.929 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:39.927+0000 7f16d1d97640 1 -- 172.21.15.29:0/1120873548 >> 172.21.15.29:0/1120873548 conn(0x7f16cc0fcd90 msgr2=0x7f16cc0ff1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:50:39.929 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:39.928+0000 7f16d1d97640 1 -- 172.21.15.29:0/1120873548 shutdown_connections 2024-03-16T11:50:39.929 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:39.928+0000 7f16d1d97640 1 -- 172.21.15.29:0/1120873548 wait complete. 2024-03-16T11:50:39.929 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:39.928+0000 7f16d1d97640 1 Processor -- start 2024-03-16T11:50:39.929 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:39.928+0000 7f16d1d97640 1 -- start start 2024-03-16T11:50:39.929 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:39.928+0000 7f16d1d97640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f16cc101800 0x7f16cc197a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:39.929 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:39.928+0000 7f16d1d97640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f16cc1029f0 0x7f16cc197fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:39.929 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:39.928+0000 7f16d1d97640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f16cc1033b0 0x7f16cc19f040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:39.930 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:39.928+0000 7f16d1d97640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f16cc10d480 con 0x7f16cc1033b0 2024-03-16T11:50:39.930 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:39.928+0000 7f16d1d97640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f16cc10d300 con 0x7f16cc101800 2024-03-16T11:50:39.930 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:39.928+0000 7f16d1d97640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f16cc10d600 con 0x7f16cc1029f0 2024-03-16T11:50:39.930 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:39.928+0000 7f16cbfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f16cc1033b0 0x7f16cc19f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:39.930 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:39.928+0000 7f16cbfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f16cc1033b0 0x7f16cc19f040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39650/0 (socket says 172.21.15.29:39650) 2024-03-16T11:50:39.930 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:39.928+0000 7f16cbfff640 1 -- 172.21.15.29:0/3957378813 learned_addr learned my addr 172.21.15.29:0/3957378813 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:50:39.930 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:39.929+0000 7f16cb7fe640 1 --2- 172.21.15.29:0/3957378813 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f16cc101800 0x7f16cc197a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:39.930 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:39.929+0000 7f16caffd640 1 --2- 172.21.15.29:0/3957378813 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f16cc1029f0 0x7f16cc197fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:39.930 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:39.929+0000 7f16cbfff640 1 -- 172.21.15.29:0/3957378813 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f16cc1029f0 msgr2=0x7f16cc197fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:39.930 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:39.929+0000 7f16cbfff640 1 --2- 172.21.15.29:0/3957378813 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f16cc1029f0 0x7f16cc197fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:39.930 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:39.929+0000 7f16cbfff640 1 -- 172.21.15.29:0/3957378813 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f16cc101800 msgr2=0x7f16cc197a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:39.930 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:39.929+0000 7f16cbfff640 1 --2- 172.21.15.29:0/3957378813 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f16cc101800 0x7f16cc197a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:39.930 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:39.929+0000 7f16cbfff640 1 -- 172.21.15.29:0/3957378813 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16cc19f6b0 con 0x7f16cc1033b0 2024-03-16T11:50:39.930 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:39.929+0000 7f16cbfff640 1 --2- 172.21.15.29:0/3957378813 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f16cc1033b0 0x7f16cc19f040 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f16c4037040 tx=0x7f16c4032f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:50:39.931 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:39.929+0000 7f16c8ff9640 1 -- 172.21.15.29:0/3957378813 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16c4041020 con 0x7f16cc1033b0 2024-03-16T11:50:39.931 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:39.929+0000 7f16d1d97640 1 -- 172.21.15.29:0/3957378813 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f16cc19f940 con 0x7f16cc1033b0 2024-03-16T11:50:39.931 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:39.929+0000 7f16c8ff9640 1 -- 172.21.15.29:0/3957378813 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f16c4034cd0 con 0x7f16cc1033b0 2024-03-16T11:50:39.931 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:39.929+0000 7f16c8ff9640 1 -- 172.21.15.29:0/3957378813 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16c4039040 con 0x7f16cc1033b0 2024-03-16T11:50:39.931 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:39.929+0000 7f16d1d97640 1 -- 172.21.15.29:0/3957378813 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f16cc19fe80 con 0x7f16cc1033b0 2024-03-16T11:50:39.932 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:39.930+0000 7f16d1d97640 1 -- 172.21.15.29:0/3957378813 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f16cc108b20 con 0x7f16cc1033b0 2024-03-16T11:50:39.932 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:39.931+0000 7f16c8ff9640 1 -- 172.21.15.29:0/3957378813 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f16c4044020 con 0x7f16cc1033b0 2024-03-16T11:50:39.932 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:39.932+0000 7f16c8ff9640 1 --2- 172.21.15.29:0/3957378813 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f16b40a2800 0x7f16b40a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:39.932 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:39.932+0000 7f16cb7fe640 1 --2- 172.21.15.29:0/3957378813 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f16b40a2800 0x7f16b40a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:39.932 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:39.932+0000 7f16c8ff9640 1 -- 172.21.15.29:0/3957378813 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8211+0+0 (secure 0 0 0) 0x7f16c40f08f0 con 0x7f16cc1033b0 2024-03-16T11:50:39.932 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:39.932+0000 7f16cb7fe640 1 --2- 172.21.15.29:0/3957378813 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f16b40a2800 0x7f16b40a4cc0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f16cc102850 tx=0x7f16bc005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:50:39.935 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:39.936+0000 7f16c8ff9640 1 -- 172.21.15.29:0/3957378813 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f16c404b020 con 0x7f16cc1033b0 2024-03-16T11:50:40.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:50:39 smithi154 ceph-mon[36320]: from='client.? 172.21.15.29:0/139782518' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-16T11:50:40.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:50:39 smithi154 ceph-mon[36320]: from='client.? 172.21.15.29:0/3268742104' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-16T11:50:40.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:50:39 smithi154 ceph-mon[36320]: from='client.? 172.21.15.29:0/3967230018' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-16T11:50:40.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:50:39 smithi154 ceph-mon[36320]: pgmap v353: 97 pgs: 97 active+clean; 640 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-03-16T11:50:40.118 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.119+0000 7f16d1d97640 1 -- 172.21.15.29:0/3957378813 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f16cc0623e0 con 0x7f16cc1033b0 2024-03-16T11:50:40.120 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.120+0000 7f16c8ff9640 1 -- 172.21.15.29:0/3957378813 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) ==== 74+0+22668 (secure 0 0 0) 0x7f16c40bda30 con 0x7f16cc1033b0 2024-03-16T11:50:40.120 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-16T11:50:40.120 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":87,"fsid":"d1887f28-e389-11ee-95c9-87774f69a715","created":"2024-03-16T11:40:16.475063+0000","modified":"2024-03-16T11:49:28.628389+0000","last_up_change":"2024-03-16T11:47:44.349348+0000","last_in_change":"2024-03-16T11:47:23.672432+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-16T11:43:51.892652+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-16T11:48:36.709732+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-16T11:48:37.586140+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"60f5608f-a408-45ba-bdf6-5b53758f88fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":2748947913},{"type":"v1","addr":"172.21.15.29:6803","nonce":2748947913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":2748947913},{"type":"v1","addr":"172.21.15.29:6805","nonce":2748947913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":2748947913},{"type":"v1","addr":"172.21.15.29:6809","nonce":2748947913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":2748947913},{"type":"v1","addr":"172.21.15.29:6807","nonce":2748947913}]},"public_addr":"172.21.15.29:6803/2748947913","cluster_addr":"172.21.15.29:6805/2748947913","heartbeat_back_addr":"172.21.15.29:6809/2748947913","heartbeat_front_addr":"172.21.15.29:6807/2748947913","state":["exists","up"]},{"osd":1,"uuid":"e70b53d6-ee78-4e56-9a2c-263ae0501b43","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6800","nonce":2339408422},{"type":"v1","addr":"172.21.15.152:6801","nonce":2339408422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6802","nonce":2339408422},{"type":"v1","addr":"172.21.15.152:6803","nonce":2339408422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6806","nonce":2339408422},{"type":"v1","addr":"172.21.15.152:6807","nonce":2339408422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6804","nonce":2339408422},{"type":"v1","addr":"172.21.15.152:6805","nonce":2339408422}]},"public_addr":"172.21.15.152:6801/2339408422","cluster_addr":"172.21.15.152:6803/2339408422","heartbeat_back_addr":"172.21.15.152:6807/2339408422","heartbeat_front_addr":"172.21.15.152:6805/2339408422","state":["exists","up"]},{"osd":2,"uuid":"ff39389a-dfac-44b5-80ff-36abca316e78","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6800","nonce":4288325280},{"type":"v1","addr":"172.21.15.154:6801","nonce":4288325280}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6802","nonce":4288325280},{"type":"v1","addr":"172.21.15.154:6803","nonce":4288325280}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6806","nonce":4288325280},{"type":"v1","addr":"172.21.15.154:6807","nonce":4288325280}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":4288325280},{"type":"v1","addr":"172.21.15.154:6805","nonce":4288325280}]},"public_addr":"172.21.15.154:6801/4288325280","cluster_addr":"172.21.15.154:6803/4288325280","heartbeat_back_addr":"172.21.15.154:6807/4288325280","heartbeat_front_addr":"172.21.15.154:6805/4288325280","state":["exists","up"]},{"osd":3,"uuid":"069e1795-55bc-4324-b94c-eb24ef2334c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":1538750397},{"type":"v1","addr":"172.21.15.29:6811","nonce":1538750397}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":1538750397},{"type":"v1","addr":"172.21.15.29:6813","nonce":1538750397}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":1538750397},{"type":"v1","addr":"172.21.15.29:6817","nonce":1538750397}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":1538750397},{"type":"v1","addr":"172.21.15.29:6815","nonce":1538750397}]},"public_addr":"172.21.15.29:6811/1538750397","cluster_addr":"172.21.15.29:6813/1538750397","heartbeat_back_addr":"172.21.15.29:6817/1538750397","heartbeat_front_addr":"172.21.15.29:6815/1538750397","state":["exists","up"]},{"osd":4,"uuid":"5a9d43a1-e8f5-4ab5-a441-afbc6ed4066e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6808","nonce":2795081750},{"type":"v1","addr":"172.21.15.152:6809","nonce":2795081750}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6810","nonce":2795081750},{"type":"v1","addr":"172.21.15.152:6811","nonce":2795081750}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6814","nonce":2795081750},{"type":"v1","addr":"172.21.15.152:6815","nonce":2795081750}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6812","nonce":2795081750},{"type":"v1","addr":"172.21.15.152:6813","nonce":2795081750}]},"public_addr":"172.21.15.152:6809/2795081750","cluster_addr":"172.21.15.152:6811/2795081750","heartbeat_back_addr":"172.21.15.152:6815/2795081750","heartbeat_front_addr":"172.21.15.152:6813/2795081750","state":["exists","up"]},{"osd":5,"uuid":"234de428-4ac7-44be-9a54-7b58331fdf8c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":1472686560},{"type":"v1","addr":"172.21.15.154:6809","nonce":1472686560}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6810","nonce":1472686560},{"type":"v1","addr":"172.21.15.154:6811","nonce":1472686560}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6814","nonce":1472686560},{"type":"v1","addr":"172.21.15.154:6815","nonce":1472686560}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6812","nonce":1472686560},{"type":"v1","addr":"172.21.15.154:6813","nonce":1472686560}]},"public_addr":"172.21.15.154:6809/1472686560","cluster_addr":"172.21.15.154:6811/1472686560","heartbeat_back_addr":"172.21.15.154:6815/1472686560","heartbeat_front_addr":"172.21.15.154:6813/1472686560","state":["exists","up"]},{"osd":6,"uuid":"ae45b8d9-7883-40c9-a9cd-4c92b601d4f2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":816506244},{"type":"v1","addr":"172.21.15.29:6819","nonce":816506244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":816506244},{"type":"v1","addr":"172.21.15.29:6821","nonce":816506244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":816506244},{"type":"v1","addr":"172.21.15.29:6825","nonce":816506244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":816506244},{"type":"v1","addr":"172.21.15.29:6823","nonce":816506244}]},"public_addr":"172.21.15.29:6819/816506244","cluster_addr":"172.21.15.29:6821/816506244","heartbeat_back_addr":"172.21.15.29:6825/816506244","heartbeat_front_addr":"172.21.15.29:6823/816506244","state":["exists","up"]},{"osd":7,"uuid":"6f687b28-4fce-4d4a-8b4e-294173d9d6c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6816","nonce":2924651787},{"type":"v1","addr":"172.21.15.152:6817","nonce":2924651787}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6818","nonce":2924651787},{"type":"v1","addr":"172.21.15.152:6819","nonce":2924651787}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6822","nonce":2924651787},{"type":"v1","addr":"172.21.15.152:6823","nonce":2924651787}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6820","nonce":2924651787},{"type":"v1","addr":"172.21.15.152:6821","nonce":2924651787}]},"public_addr":"172.21.15.152:6817/2924651787","cluster_addr":"172.21.15.152:6819/2924651787","heartbeat_back_addr":"172.21.15.152:6823/2924651787","heartbeat_front_addr":"172.21.15.152:6821/2924651787","state":["exists","up"]},{"osd":8,"uuid":"a25fb741-caa7-429a-b540-e394d034b446","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6816","nonce":129974156},{"type":"v1","addr":"172.21.15.154:6817","nonce":129974156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6818","nonce":129974156},{"type":"v1","addr":"172.21.15.154:6819","nonce":129974156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6822","nonce":129974156},{"type":"v1","addr":"172.21.15.154:6823","nonce":129974156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6820","nonce":129974156},{"type":"v1","addr":"172.21.15.154:6821","nonce":129974156}]},"public_addr":"172.21.15.154:6817/129974156","cluster_addr":"172.21.15.154:6819/129974156","heartbeat_back_addr":"172.21.15.154:6823/129974156","heartbeat_front_addr":"172.21.15.154:6821/129974156","state":["exists","up"]},{"osd":9,"uuid":"1071cb0e-8c5f-4cd8-9805-0045e220d946","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6826","nonce":2951414243},{"type":"v1","addr":"172.21.15.29:6827","nonce":2951414243}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":2951414243},{"type":"v1","addr":"172.21.15.29:6829","nonce":2951414243}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":2951414243},{"type":"v1","addr":"172.21.15.29:6833","nonce":2951414243}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":2951414243},{"type":"v1","addr":"172.21.15.29:6831","nonce":2951414243}]},"public_addr":"172.21.15.29:6827/2951414243","cluster_addr":"172.21.15.29:6829/2951414243","heartbeat_back_addr":"172.21.15.29:6833/2951414243","heartbeat_front_addr":"172.21.15.29:6831/2951414243","state":["exists","up"]},{"osd":10,"uuid":"ee7c7d74-3f39-4cd4-be80-843fb1dd9ecc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6824","nonce":3412525480},{"type":"v1","addr":"172.21.15.152:6825","nonce":3412525480}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6826","nonce":3412525480},{"type":"v1","addr":"172.21.15.152:6827","nonce":3412525480}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6830","nonce":3412525480},{"type":"v1","addr":"172.21.15.152:6831","nonce":3412525480}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6828","nonce":3412525480},{"type":"v1","addr":"172.21.15.152:6829","nonce":3412525480}]},"public_addr":"172.21.15.152:6825/3412525480","cluster_addr":"172.21.15.152:6827/3412525480","heartbeat_back_addr":"172.21.15.152:6831/3412525480","heartbeat_front_addr":"172.21.15.152:6829/3412525480","state":["exists","up"]},{"osd":11,"uuid":"4598c803-984a-4de6-ac22-066502c17889","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6824","nonce":1160927966},{"type":"v1","addr":"172.21.15.154:6825","nonce":1160927966}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6826","nonce":1160927966},{"type":"v1","addr":"172.21.15.154:6827","nonce":1160927966}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6830","nonce":1160927966},{"type":"v1","addr":"172.21.15.154:6831","nonce":1160927966}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6828","nonce":1160927966},{"type":"v1","addr":"172.21.15.154:6829","nonce":1160927966}]},"public_addr":"172.21.15.154:6825/1160927966","cluster_addr":"172.21.15.154:6827/1160927966","heartbeat_back_addr":"172.21.15.154:6831/1160927966","heartbeat_front_addr":"172.21.15.154:6829/1160927966","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:43:02.343393+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:43:25.209151+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:43:48.946164+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:44:14.588424+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:44:39.700025+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:45:04.061197+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:45:31.214508+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:45:56.937296+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:46:22.768915+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:46:50.101011+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:47:16.298041+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:47:42.402095+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"2.1b","mappings":[{"from":2,"to":1}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"3.33","mappings":[{"from":5,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.29:0/1853086886":"2024-03-17T11:41:27.531724+0000","172.21.15.29:0/1211659364":"2024-03-17T11:41:27.531724+0000","172.21.15.29:6800/2798831128":"2024-03-17T11:41:27.531724+0000","172.21.15.29:0/1281028285":"2024-03-17T11:41:00.505678+0000","172.21.15.29:6800/2275315654":"2024-03-17T11:41:00.505678+0000","172.21.15.29:6801/2798831128":"2024-03-17T11:41:27.531724+0000","172.21.15.29:6801/2275315654":"2024-03-17T11:41:00.505678+0000","172.21.15.29:0/1330100009":"2024-03-17T11:41:00.505678+0000","172.21.15.29:0/1784249039":"2024-03-17T11:41:00.505678+0000","172.21.15.29:0/1116486932":"2024-03-17T11:41:27.531724+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-16T11:50:40.122 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.122+0000 7f16d1d97640 1 -- 172.21.15.29:0/3957378813 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f16b40a2800 msgr2=0x7f16b40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:40.122 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.123+0000 7f16d1d97640 1 --2- 172.21.15.29:0/3957378813 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f16b40a2800 0x7f16b40a4cc0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f16cc102850 tx=0x7f16bc005f90 comp rx=0 tx=0).stop 2024-03-16T11:50:40.122 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.123+0000 7f16d1d97640 1 -- 172.21.15.29:0/3957378813 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f16cc1033b0 msgr2=0x7f16cc19f040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:40.123 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.123+0000 7f16d1d97640 1 --2- 172.21.15.29:0/3957378813 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f16cc1033b0 0x7f16cc19f040 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f16c4037040 tx=0x7f16c4032f40 comp rx=0 tx=0).stop 2024-03-16T11:50:40.123 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.123+0000 7f16d1d97640 1 -- 172.21.15.29:0/3957378813 shutdown_connections 2024-03-16T11:50:40.123 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.123+0000 7f16d1d97640 1 --2- 172.21.15.29:0/3957378813 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f16b40a2800 0x7f16b40a4cc0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:40.123 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.123+0000 7f16d1d97640 1 --2- 172.21.15.29:0/3957378813 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f16cc1033b0 0x7f16cc19f040 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:40.123 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.123+0000 7f16d1d97640 1 --2- 172.21.15.29:0/3957378813 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f16cc1029f0 0x7f16cc197fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:40.123 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.123+0000 7f16d1d97640 1 --2- 172.21.15.29:0/3957378813 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f16cc101800 0x7f16cc197a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:40.123 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.123+0000 7f16d1d97640 1 -- 172.21.15.29:0/3957378813 >> 172.21.15.29:0/3957378813 conn(0x7f16cc0fcd90 msgr2=0x7f16cc105cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:50:40.123 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.123+0000 7f16d1d97640 1 -- 172.21.15.29:0/3957378813 shutdown_connections 2024-03-16T11:50:40.123 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.123+0000 7f16d1d97640 1 -- 172.21.15.29:0/3957378813 wait complete. 2024-03-16T11:50:40.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:50:39 smithi152 ceph-mon[36291]: from='client.? 172.21.15.29:0/139782518' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-16T11:50:40.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:50:39 smithi152 ceph-mon[36291]: from='client.? 172.21.15.29:0/3268742104' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-16T11:50:40.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:50:39 smithi152 ceph-mon[36291]: from='client.? 172.21.15.29:0/3967230018' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-16T11:50:40.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:50:39 smithi152 ceph-mon[36291]: pgmap v353: 97 pgs: 97 active+clean; 640 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-03-16T11:50:40.133 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 21, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-03-16T11:48:38.431509+0000', 'modified': '2024-03-16T11:48:42.969084+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 24503, 'mds_1': 24499, 'mds_2': 24464}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24434': {'gid': 24434, 'name': 'e', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.152:6835/2919113178', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.152:6834', 'nonce': 2919113178}, {'type': 'v1', 'addr': '172.21.15.152:6835', 'nonce': 2919113178}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24464': {'gid': 24464, 'name': 'h', 'rank': 2, 'incarnation': 19, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.152:6837/1323866722', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.152:6836', 'nonce': 1323866722}, {'type': 'v1', 'addr': '172.21.15.152:6837', 'nonce': 1323866722}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24466': {'gid': 24466, 'name': 'f', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.154:6835/4165274681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 4165274681}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 4165274681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24478': {'gid': 24478, 'name': 'c', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.154:6837/1468560096', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 1468560096}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 1468560096}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24499': {'gid': 24499, 'name': 'i', 'rank': 1, 'incarnation': 17, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.154:6839/685736862', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 685736862}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 685736862}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24503': {'gid': 24503, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.152:6839/1548731833', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.152:6838', 'nonce': 1548731833}, {'type': 'v1', 'addr': '172.21.15.152:6839', 'nonce': 1548731833}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24503, 'qdb_cluster': [24503, 24464, 24499]} 2024-03-16T11:50:40.134 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 3/3 2024-03-16T11:50:40.134 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.h status 2024-03-16T11:50:40.239 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.238+0000 7f2595c68640 1 -- 172.21.15.29:0/1652732539 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f25901022a0 msgr2=0x7f2590108b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:40.239 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.238+0000 7f2595c68640 1 --2- 172.21.15.29:0/1652732539 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f25901022a0 0x7f2590108b30 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f2588009be0 tx=0x7f258802bef0 comp rx=0 tx=0).stop 2024-03-16T11:50:40.239 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.238+0000 7f2595c68640 1 -- 172.21.15.29:0/1652732539 shutdown_connections 2024-03-16T11:50:40.239 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.238+0000 7f2595c68640 1 --2- 172.21.15.29:0/1652732539 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f25901022a0 0x7f2590108b30 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:40.239 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.238+0000 7f2595c68640 1 --2- 172.21.15.29:0/1652732539 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f25901018e0 0x7f2590101d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:40.239 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.238+0000 7f2595c68640 1 --2- 172.21.15.29:0/1652732539 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f25901006f0 0x7f2590100b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:40.239 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.238+0000 7f2595c68640 1 -- 172.21.15.29:0/1652732539 >> 172.21.15.29:0/1652732539 conn(0x7f25900fbc80 msgr2=0x7f25900fe0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:50:40.239 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.238+0000 7f2595c68640 1 -- 172.21.15.29:0/1652732539 shutdown_connections 2024-03-16T11:50:40.239 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.238+0000 7f2595c68640 1 -- 172.21.15.29:0/1652732539 wait complete. 2024-03-16T11:50:40.239 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.238+0000 7f2595c68640 1 Processor -- start 2024-03-16T11:50:40.239 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.238+0000 7f2595c68640 1 -- start start 2024-03-16T11:50:40.239 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.238+0000 7f2595c68640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f25901006f0 0x7f259006a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:40.240 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.239+0000 7f2595c68640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f25901018e0 0x7f259019c800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:40.240 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.239+0000 7f2595c68640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f25901022a0 0x7f259019ee20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:40.240 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.239+0000 7f2595c68640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f259010c370 con 0x7f25901006f0 2024-03-16T11:50:40.240 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.239+0000 7f2595c68640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f259010c1f0 con 0x7f25901018e0 2024-03-16T11:50:40.240 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.239+0000 7f2595c68640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f259010c4f0 con 0x7f25901022a0 2024-03-16T11:50:40.240 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.239+0000 7f258f7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f25901006f0 0x7f259006a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:40.240 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.239+0000 7f258f7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f25901006f0 0x7f259006a940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39666/0 (socket says 172.21.15.29:39666) 2024-03-16T11:50:40.240 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.239+0000 7f258f7fe640 1 -- 172.21.15.29:0/265838325 learned_addr learned my addr 172.21.15.29:0/265838325 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:50:40.240 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.239+0000 7f258ffff640 1 --2- 172.21.15.29:0/265838325 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f25901022a0 0x7f259019ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:40.240 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.239+0000 7f258effd640 1 --2- 172.21.15.29:0/265838325 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f25901018e0 0x7f259019c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:40.240 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.239+0000 7f258f7fe640 1 -- 172.21.15.29:0/265838325 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f25901022a0 msgr2=0x7f259019ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:40.240 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.239+0000 7f258f7fe640 1 --2- 172.21.15.29:0/265838325 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f25901022a0 0x7f259019ee20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:40.240 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.239+0000 7f258f7fe640 1 -- 172.21.15.29:0/265838325 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f25901018e0 msgr2=0x7f259019c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:40.240 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.239+0000 7f258f7fe640 1 --2- 172.21.15.29:0/265838325 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f25901018e0 0x7f259019c800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:40.240 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.239+0000 7f258f7fe640 1 -- 172.21.15.29:0/265838325 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f259019f730 con 0x7f25901006f0 2024-03-16T11:50:40.241 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.239+0000 7f258f7fe640 1 --2- 172.21.15.29:0/265838325 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f25901006f0 0x7f259006a940 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f258000e9c0 tx=0x7f258000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:50:40.241 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.239+0000 7f258cff9640 1 -- 172.21.15.29:0/265838325 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f258000cd80 con 0x7f25901006f0 2024-03-16T11:50:40.241 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.239+0000 7f2595c68640 1 -- 172.21.15.29:0/265838325 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f259019f9c0 con 0x7f25901006f0 2024-03-16T11:50:40.241 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.239+0000 7f258cff9640 1 -- 172.21.15.29:0/265838325 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f25800035f0 con 0x7f25901006f0 2024-03-16T11:50:40.241 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.239+0000 7f258cff9640 1 -- 172.21.15.29:0/265838325 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2580010690 con 0x7f25901006f0 2024-03-16T11:50:40.241 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.239+0000 7f2595c68640 1 -- 172.21.15.29:0/265838325 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f259019fec0 con 0x7f25901006f0 2024-03-16T11:50:40.247 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.241+0000 7f258cff9640 1 -- 172.21.15.29:0/265838325 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f25800037a0 con 0x7f25901006f0 2024-03-16T11:50:40.247 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.242+0000 7f259884e740 1 Processor -- start 2024-03-16T11:50:40.247 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.242+0000 7f259884e740 1 -- start start 2024-03-16T11:50:40.247 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.242+0000 7f259884e740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561d5530ebd0 0x561d5530eff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:40.247 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.242+0000 7f259884e740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561d5530f5c0 0x561d553140c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:40.247 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.242+0000 7f259884e740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561d55314730 0x561d55316bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:40.247 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.242+0000 7f259884e740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x561d55272c80 con 0x561d55314730 2024-03-16T11:50:40.247 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.242+0000 7f259884e740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x561d5521baa0 con 0x561d5530ebd0 2024-03-16T11:50:40.247 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.242+0000 7f258cff9640 1 --2- 172.21.15.29:0/265838325 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f25780a2800 0x7f25780a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:40.247 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.242+0000 7f259884e740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x561d5526ed20 con 0x561d5530f5c0 2024-03-16T11:50:40.248 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.242+0000 7f258effd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561d55314730 0x561d55316bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:40.248 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.242+0000 7f258ffff640 1 --2- 172.21.15.29:0/265838325 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f25780a2800 0x7f25780a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:40.248 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.242+0000 7f258effd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561d55314730 0x561d55316bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39668/0 (socket says 172.21.15.29:39668) 2024-03-16T11:50:40.248 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.242+0000 7f258effd640 1 -- 172.21.15.29:0/3861476869 learned_addr learned my addr 172.21.15.29:0/3861476869 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:50:40.248 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.242+0000 7f258ffff640 1 --2- 172.21.15.29:0/3861476869 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561d5530ebd0 0x561d5530eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:40.248 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.242+0000 7f258f7fe640 1 --2- 172.21.15.29:0/3861476869 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561d5530f5c0 0x561d553140c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:40.248 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.242+0000 7f258cff9640 1 -- 172.21.15.29:0/265838325 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8211+0+0 (secure 0 0 0) 0x7f25800cee20 con 0x7f25901006f0 2024-03-16T11:50:40.248 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.242+0000 7f258effd640 1 -- 172.21.15.29:0/3861476869 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561d5530f5c0 msgr2=0x561d553140c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:50:40.248 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.242+0000 7f258effd640 1 --2- 172.21.15.29:0/3861476869 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561d5530f5c0 0x561d553140c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:40.248 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.242+0000 7f258effd640 1 -- 172.21.15.29:0/3861476869 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561d5530ebd0 msgr2=0x561d5530eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:50:40.248 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.242+0000 7f258effd640 1 --2- 172.21.15.29:0/3861476869 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561d5530ebd0 0x561d5530eff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:40.248 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.242+0000 7f258effd640 1 -- 172.21.15.29:0/3861476869 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561d5523dff0 con 0x561d55314730 2024-03-16T11:50:40.248 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.243+0000 7f258ffff640 1 --2- 172.21.15.29:0/3861476869 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561d5530ebd0 0x561d5530eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:50:40.248 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.243+0000 7f258effd640 1 --2- 172.21.15.29:0/3861476869 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561d55314730 0x561d55316bc0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f2584009c80 tx=0x7f2584011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d11e1cf7db8aaeae server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:50:40.249 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.243+0000 7f258ffff640 1 --2- 172.21.15.29:0/265838325 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f25780a2800 0x7f25780a4cc0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f2588004ab0 tx=0x7f2588005e30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:50:40.249 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.243+0000 7f2575ffb640 1 -- 172.21.15.29:0/3861476869 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2584004220 con 0x561d55314730 2024-03-16T11:50:40.249 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.243+0000 7f2575ffb640 1 -- 172.21.15.29:0/3861476869 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2584009e30 con 0x561d55314730 2024-03-16T11:50:40.249 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.243+0000 7f2575ffb640 1 -- 172.21.15.29:0/3861476869 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2584019850 con 0x561d55314730 2024-03-16T11:50:40.249 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.244+0000 7f259884e740 1 -- 172.21.15.29:0/3861476869 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561d55314730 msgr2=0x561d55316bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:40.249 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.244+0000 7f259884e740 1 --2- 172.21.15.29:0/3861476869 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561d55314730 0x561d55316bc0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f2584009c80 tx=0x7f2584011810 comp rx=0 tx=0).stop 2024-03-16T11:50:40.249 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.244+0000 7f259884e740 1 -- 172.21.15.29:0/3861476869 shutdown_connections 2024-03-16T11:50:40.249 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.244+0000 7f259884e740 1 --2- 172.21.15.29:0/3861476869 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561d55314730 0x561d55316bc0 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:40.249 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.244+0000 7f259884e740 1 --2- 172.21.15.29:0/3861476869 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561d5530f5c0 0x561d553140c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:40.249 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.244+0000 7f259884e740 1 --2- 172.21.15.29:0/3861476869 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561d5530ebd0 0x561d5530eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:40.249 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.244+0000 7f259884e740 1 -- 172.21.15.29:0/3861476869 >> 172.21.15.29:0/3861476869 conn(0x561d5530a140 msgr2=0x561d5530a540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:50:40.249 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.244+0000 7f259884e740 1 -- 172.21.15.29:0/3861476869 shutdown_connections 2024-03-16T11:50:40.249 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.244+0000 7f259884e740 1 -- 172.21.15.29:0/3861476869 wait complete. 2024-03-16T11:50:40.249 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.244+0000 7f259884e740 1 -- start start 2024-03-16T11:50:40.249 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.244+0000 7f259884e740 1 Processor -- start 2024-03-16T11:50:40.250 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.244+0000 7f259884e740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561d5530ebd0 0x561d55318cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:40.250 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.244+0000 7f259884e740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561d5530f5c0 0x561d55319210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:40.250 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.244+0000 7f259884e740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561d55314730 0x561d55320250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:40.250 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.244+0000 7f259884e740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x561d553176e0 con 0x561d55314730 2024-03-16T11:50:40.250 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.244+0000 7f259884e740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x561d5522eef0 con 0x561d5530ebd0 2024-03-16T11:50:40.250 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.244+0000 7f259884e740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x561d55317860 con 0x561d5530f5c0 2024-03-16T11:50:40.250 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.244+0000 7f259884e740 10 client.0 _ll_register_callbacks cb 0x561d55229dc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T11:50:40.250 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.244+0000 7f258ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561d55314730 0x561d55320250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:40.250 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.244+0000 7f258ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561d55314730 0x561d55320250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39684/0 (socket says 172.21.15.29:39684) 2024-03-16T11:50:40.250 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.245+0000 7f258ffff640 1 -- 172.21.15.29:0/3039405083 learned_addr learned my addr 172.21.15.29:0/3039405083 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:50:40.250 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.245+0000 7f258effd640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561d5530f5c0 0x561d55319210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:40.250 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.245+0000 7f258f7fe640 1 --2- 172.21.15.29:0/3039405083 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561d5530ebd0 0x561d55318cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:40.250 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.245+0000 7f258ffff640 1 -- 172.21.15.29:0/3039405083 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561d5530f5c0 msgr2=0x561d55319210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:50:40.251 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.245+0000 7f258ffff640 1 --2- 172.21.15.29:0/3039405083 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561d5530f5c0 0x561d55319210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:40.251 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.245+0000 7f258ffff640 1 -- 172.21.15.29:0/3039405083 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561d5530ebd0 msgr2=0x561d55318cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:50:40.251 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.245+0000 7f258ffff640 1 --2- 172.21.15.29:0/3039405083 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561d5530ebd0 0x561d55318cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:40.251 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.245+0000 7f258ffff640 1 -- 172.21.15.29:0/3039405083 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561d5523ba30 con 0x561d55314730 2024-03-16T11:50:40.251 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.245+0000 7f258ffff640 1 --2- 172.21.15.29:0/3039405083 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561d55314730 0x561d55320250 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f2588035000 tx=0x7f258802bef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7615b0e93bce615e server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:50:40.251 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.245+0000 7f256effd640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-16T11:50:40.251 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.245+0000 7f256effd640 1 -- 172.21.15.29:0/3039405083 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f258802ce10 con 0x561d55314730 2024-03-16T11:50:40.251 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.245+0000 7f256effd640 1 -- 172.21.15.29:0/3039405083 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8211+0+0 (secure 0 0 0) 0x7f2588004040 con 0x561d55314730 2024-03-16T11:50:40.251 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.245+0000 7f259884e740 1 -- 172.21.15.29:0/3039405083 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2584009e30 con 0x561d55314730 2024-03-16T11:50:40.251 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.245+0000 7f256effd640 1 -- 172.21.15.29:0/3039405083 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f2588004240 con 0x561d55314730 2024-03-16T11:50:40.251 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.245+0000 7f259884e740 10 client.15240 fetch_fsmap learned FSMap version 21 2024-03-16T11:50:40.251 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.245+0000 7f259884e740 1 -- 172.21.15.29:0/3039405083 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x561d54f872d0 con 0x561d55314730 2024-03-16T11:50:40.251 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.245+0000 7f256effd640 1 -- 172.21.15.29:0/3039405083 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2584009e30 con 0x561d55314730 2024-03-16T11:50:40.251 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.246+0000 7f256effd640 1 -- 172.21.15.29:0/3039405083 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f25880349d0 con 0x561d55314730 2024-03-16T11:50:40.251 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.246+0000 7f259884e740 10 client.15240 fetch_fsmap finished waiting for FSMap version 21 2024-03-16T11:50:40.252 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.246+0000 7f259884e740 10 client.15240 resolve_mds: resolved name 'h' to daemon mds.h 2024-03-16T11:50:40.252 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.246+0000 7f259884e740 20 client.15240 populate_metadata read hostname 'smithi029' 2024-03-16T11:50:40.252 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.246+0000 7f259884e740 1 --2- 172.21.15.29:0/3039405083 >> [v2:172.21.15.152:6836/1323866722,v1:172.21.15.152:6837/1323866722] conn(0x561d55325870 0x561d55326ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:40.252 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.246+0000 7f259884e740 4 client.15240 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T11:50:40.252 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.246+0000 7f259884e740 1 -- 172.21.15.29:0/3039405083 --> [v2:172.21.15.152:6836/1323866722,v1:172.21.15.152:6837/1323866722] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561d552801c0 con 0x561d55325870 2024-03-16T11:50:40.252 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.246+0000 7f258f7fe640 1 --2- 172.21.15.29:0/3039405083 >> [v2:172.21.15.152:6836/1323866722,v1:172.21.15.152:6837/1323866722] conn(0x561d55325870 0x561d55326ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:40.252 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.246+0000 7f258f7fe640 1 --2- 172.21.15.29:0/3039405083 >> [v2:172.21.15.152:6836/1323866722,v1:172.21.15.152:6837/1323866722] conn(0x561d55325870 0x561d55326ce0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=9d344486b02f50d2 server_cookie=cf2963e710197ac2 in_seq=0 out_seq=0 2024-03-16T11:50:40.252 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.246+0000 7f256effd640 10 client.15240 ms_handle_connect on v2:172.21.15.152:6836/1323866722 2024-03-16T11:50:40.252 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.247+0000 7f256effd640 1 -- 172.21.15.29:0/3039405083 <== mds.2 v2:172.21.15.152:6836/1323866722 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x561d552801c0 con 0x561d55325870 2024-03-16T11:50:40.252 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.247+0000 7f256effd640 10 client.15240 handle_command_reply: tid=0 2024-03-16T11:50:40.252 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.247+0000 7f259884e740 1 client.15240 shutdown 2024-03-16T11:50:40.252 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.247+0000 7f259884e740 1 -- 172.21.15.29:0/3039405083 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561d55314730 msgr2=0x561d55320250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:40.252 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.247+0000 7f259884e740 1 --2- 172.21.15.29:0/3039405083 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561d55314730 0x561d55320250 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f2588035000 tx=0x7f258802bef0 comp rx=0 tx=0).stop 2024-03-16T11:50:40.252 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.247+0000 7f259884e740 1 -- 172.21.15.29:0/3039405083 shutdown_connections 2024-03-16T11:50:40.253 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.247+0000 7f259884e740 1 --2- 172.21.15.29:0/3039405083 >> [v2:172.21.15.152:6836/1323866722,v1:172.21.15.152:6837/1323866722] conn(0x561d55325870 0x561d55326ce0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:40.253 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.247+0000 7f259884e740 1 --2- 172.21.15.29:0/3039405083 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561d55314730 0x561d55320250 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:40.253 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.247+0000 7f256effd640 0 client.15240 ms_handle_reset on v2:172.21.15.152:6836/1323866722 2024-03-16T11:50:40.253 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.247+0000 7f259884e740 1 --2- 172.21.15.29:0/3039405083 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561d5530f5c0 0x561d55319210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:40.253 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.247+0000 7f259884e740 1 --2- 172.21.15.29:0/3039405083 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561d5530ebd0 0x561d55318cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:40.253 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.247+0000 7f259884e740 1 -- 172.21.15.29:0/3039405083 >> 172.21.15.29:0/3039405083 conn(0x561d5530a140 msgr2=0x561d55309380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:50:40.253 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.247+0000 7f259884e740 1 -- 172.21.15.29:0/3039405083 shutdown_connections 2024-03-16T11:50:40.253 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.247+0000 7f259884e740 1 -- 172.21.15.29:0/3039405083 wait complete. 2024-03-16T11:50:40.253 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.247+0000 7f259884e740 20 client.15240 trim_cache size 0 max 16384 2024-03-16T11:50:40.270 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.265+0000 7f259884e740 1 Processor -- start 2024-03-16T11:50:40.270 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.265+0000 7f259884e740 1 -- start start 2024-03-16T11:50:40.270 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.265+0000 7f259884e740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561d5530ebd0 0x561d55318cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:40.270 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.265+0000 7f259884e740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561d5530f5c0 0x561d55319210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:40.270 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.265+0000 7f259884e740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561d55314730 0x561d55320170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:40.270 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.265+0000 7f259884e740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x561d55325020 con 0x561d5530f5c0 2024-03-16T11:50:40.270 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.265+0000 7f259884e740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x561d553256f0 con 0x561d55314730 2024-03-16T11:50:40.270 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.265+0000 7f259884e740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x561d55325dc0 con 0x561d5530ebd0 2024-03-16T11:50:40.271 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.265+0000 7f258effd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561d5530f5c0 0x561d55319210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:40.271 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.265+0000 7f258effd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561d5530f5c0 0x561d55319210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39688/0 (socket says 172.21.15.29:39688) 2024-03-16T11:50:40.271 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.265+0000 7f258effd640 1 -- 172.21.15.29:0/395731837 learned_addr learned my addr 172.21.15.29:0/395731837 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:50:40.271 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.266+0000 7f258f7fe640 1 --2- 172.21.15.29:0/395731837 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561d5530ebd0 0x561d55318cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:40.271 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.266+0000 7f258ffff640 1 --2- 172.21.15.29:0/395731837 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561d55314730 0x561d55320170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:40.271 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.266+0000 7f258effd640 1 -- 172.21.15.29:0/395731837 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561d5530ebd0 msgr2=0x561d55318cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:50:40.271 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.266+0000 7f258effd640 1 --2- 172.21.15.29:0/395731837 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561d5530ebd0 0x561d55318cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:40.271 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.266+0000 7f258effd640 1 -- 172.21.15.29:0/395731837 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561d55314730 msgr2=0x561d55320170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:50:40.271 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.266+0000 7f258effd640 1 --2- 172.21.15.29:0/395731837 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561d55314730 0x561d55320170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:40.271 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.266+0000 7f258effd640 1 -- 172.21.15.29:0/395731837 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561d550516b0 con 0x561d5530f5c0 2024-03-16T11:50:40.271 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.266+0000 7f258effd640 1 --2- 172.21.15.29:0/395731837 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561d5530f5c0 0x561d55319210 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f2584000c00 tx=0x7f2584005a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12689112d476fd07 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:50:40.271 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.266+0000 7f25757fa640 1 -- 172.21.15.29:0/395731837 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2584005040 con 0x561d5530f5c0 2024-03-16T11:50:40.271 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.266+0000 7f25757fa640 1 -- 172.21.15.29:0/395731837 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2584004660 con 0x561d5530f5c0 2024-03-16T11:50:40.271 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.266+0000 7f25757fa640 1 -- 172.21.15.29:0/395731837 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2584025690 con 0x561d5530f5c0 2024-03-16T11:50:40.272 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.267+0000 7f259884e740 1 -- 172.21.15.29:0/395731837 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561d5530f5c0 msgr2=0x561d55319210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:40.272 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.267+0000 7f259884e740 1 --2- 172.21.15.29:0/395731837 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561d5530f5c0 0x561d55319210 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f2584000c00 tx=0x7f2584005a00 comp rx=0 tx=0).stop 2024-03-16T11:50:40.272 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.267+0000 7f259884e740 1 -- 172.21.15.29:0/395731837 shutdown_connections 2024-03-16T11:50:40.272 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.267+0000 7f259884e740 1 --2- 172.21.15.29:0/395731837 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561d55314730 0x561d55320170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:40.272 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.267+0000 7f259884e740 1 --2- 172.21.15.29:0/395731837 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561d5530f5c0 0x561d55319210 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:40.272 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.267+0000 7f259884e740 1 --2- 172.21.15.29:0/395731837 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561d5530ebd0 0x561d55318cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:40.272 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.267+0000 7f259884e740 1 -- 172.21.15.29:0/395731837 >> 172.21.15.29:0/395731837 conn(0x561d5530a140 msgr2=0x561d553245e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:50:40.272 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.267+0000 7f259884e740 1 -- 172.21.15.29:0/395731837 shutdown_connections 2024-03-16T11:50:40.272 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.267+0000 7f259884e740 1 -- 172.21.15.29:0/395731837 wait complete. 2024-03-16T11:50:40.272 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.267+0000 7f259884e740 1 -- start start 2024-03-16T11:50:40.272 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.267+0000 7f259884e740 1 Processor -- start 2024-03-16T11:50:40.272 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.267+0000 7f259884e740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561d5530ebd0 0x561d5531b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:40.272 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.267+0000 7f259884e740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561d5530f5c0 0x561d5531bec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:40.272 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.267+0000 7f259884e740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561d55314730 0x561d5531ee90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:40.273 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.267+0000 7f259884e740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x561d55326b60 con 0x561d5530ebd0 2024-03-16T11:50:40.273 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.267+0000 7f259884e740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x561d55326490 con 0x561d55314730 2024-03-16T11:50:40.273 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.267+0000 7f259884e740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x561d55327230 con 0x561d5530f5c0 2024-03-16T11:50:40.273 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.267+0000 7f259884e740 10 client.0 _ll_register_callbacks cb 0x7f2588009a60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T11:50:40.273 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.267+0000 7f258f7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561d5530ebd0 0x561d5531b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:40.273 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.267+0000 7f258f7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561d5530ebd0 0x561d5531b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39692/0 (socket says 172.21.15.29:39692) 2024-03-16T11:50:40.273 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.267+0000 7f258effd640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561d5530f5c0 0x561d5531bec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:40.273 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.267+0000 7f258f7fe640 1 -- 172.21.15.29:0/1050116020 learned_addr learned my addr 172.21.15.29:0/1050116020 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:50:40.273 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.267+0000 7f258ffff640 1 --2- 172.21.15.29:0/1050116020 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561d55314730 0x561d5531ee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:40.273 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.268+0000 7f258f7fe640 1 -- 172.21.15.29:0/1050116020 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561d5530f5c0 msgr2=0x561d5531bec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:50:40.273 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.268+0000 7f258f7fe640 1 --2- 172.21.15.29:0/1050116020 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561d5530f5c0 0x561d5531bec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:40.273 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.268+0000 7f258f7fe640 1 -- 172.21.15.29:0/1050116020 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561d55314730 msgr2=0x561d5531ee90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:50:40.273 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.268+0000 7f258f7fe640 1 --2- 172.21.15.29:0/1050116020 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561d55314730 0x561d5531ee90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:40.274 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.268+0000 7f258f7fe640 1 -- 172.21.15.29:0/1050116020 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561d5531f480 con 0x561d5530ebd0 2024-03-16T11:50:40.274 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.268+0000 7f258effd640 1 --2- 172.21.15.29:0/1050116020 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561d5530f5c0 0x561d5531bec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:50:40.274 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.268+0000 7f258f7fe640 1 --2- 172.21.15.29:0/1050116020 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561d5530ebd0 0x561d5531b980 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f25800cc8d0 tx=0x7f2580007850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e84a70f73d70300 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:50:40.274 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.268+0000 7f256effd640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-16T11:50:40.274 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.268+0000 7f256effd640 1 -- 172.21.15.29:0/1050116020 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f258009f8c0 con 0x561d5530ebd0 2024-03-16T11:50:40.274 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.268+0000 7f256effd640 1 -- 172.21.15.29:0/1050116020 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8211+0+0 (secure 0 0 0) 0x7f25800860b0 con 0x561d5530ebd0 2024-03-16T11:50:40.274 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.268+0000 7f259884e740 1 -- 172.21.15.29:0/1050116020 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2584004660 con 0x561d5530ebd0 2024-03-16T11:50:40.274 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.268+0000 7f256effd640 1 -- 172.21.15.29:0/1050116020 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f25800a3f10 con 0x561d5530ebd0 2024-03-16T11:50:40.274 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.268+0000 7f259884e740 10 client.15246 fetch_fsmap learned FSMap version 21 2024-03-16T11:50:40.274 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.268+0000 7f259884e740 1 -- 172.21.15.29:0/1050116020 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x561d5538b5e0 con 0x561d5530ebd0 2024-03-16T11:50:40.274 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.268+0000 7f256effd640 1 -- 172.21.15.29:0/1050116020 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2584004660 con 0x561d5530ebd0 2024-03-16T11:50:40.274 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.268+0000 7f256effd640 1 -- 172.21.15.29:0/1050116020 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f25800a3a80 con 0x561d5530ebd0 2024-03-16T11:50:40.274 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.268+0000 7f259884e740 10 client.15246 fetch_fsmap finished waiting for FSMap version 21 2024-03-16T11:50:40.275 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.268+0000 7f259884e740 10 client.15246 resolve_mds: resolved name 'h' to daemon mds.h 2024-03-16T11:50:40.275 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.268+0000 7f259884e740 20 client.15246 populate_metadata read hostname 'smithi029' 2024-03-16T11:50:40.275 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.268+0000 7f259884e740 1 --2- 172.21.15.29:0/1050116020 >> [v2:172.21.15.152:6836/1323866722,v1:172.21.15.152:6837/1323866722] conn(0x561d55390180 0x561d5538c5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:40.275 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.269+0000 7f259884e740 4 client.15246 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-16T11:50:40.275 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.269+0000 7f259884e740 1 -- 172.21.15.29:0/1050116020 --> [v2:172.21.15.152:6836/1323866722,v1:172.21.15.152:6837/1323866722] -- command(tid 0: {"prefix": "status"}) -- 0x561d5527a510 con 0x561d55390180 2024-03-16T11:50:40.275 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.269+0000 7f258effd640 1 --2- 172.21.15.29:0/1050116020 >> [v2:172.21.15.152:6836/1323866722,v1:172.21.15.152:6837/1323866722] conn(0x561d55390180 0x561d5538c5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:40.275 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.269+0000 7f256effd640 10 client.15246 ms_handle_connect on v2:172.21.15.152:6836/1323866722 2024-03-16T11:50:40.275 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.269+0000 7f258effd640 1 --2- 172.21.15.29:0/1050116020 >> [v2:172.21.15.152:6836/1323866722,v1:172.21.15.152:6837/1323866722] conn(0x561d55390180 0x561d5538c5c0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=79b8ce524101a073 server_cookie=337492e886e028c9 in_seq=0 out_seq=0 2024-03-16T11:50:40.275 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.269+0000 7f256effd640 1 -- 172.21.15.29:0/1050116020 <== mds.2 v2:172.21.15.152:6836/1323866722 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x561d5527a510 con 0x561d55390180 2024-03-16T11:50:40.275 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.269+0000 7f256effd640 10 client.15246 handle_command_reply: tid=0 2024-03-16T11:50:40.275 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.269+0000 7f259884e740 1 client.15246 shutdown 2024-03-16T11:50:40.275 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.269+0000 7f259884e740 1 -- 172.21.15.29:0/1050116020 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561d5530ebd0 msgr2=0x561d5531b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:40.275 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.269+0000 7f259884e740 1 --2- 172.21.15.29:0/1050116020 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561d5530ebd0 0x561d5531b980 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f25800cc8d0 tx=0x7f2580007850 comp rx=0 tx=0).stop 2024-03-16T11:50:40.275 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.269+0000 7f259884e740 1 -- 172.21.15.29:0/1050116020 shutdown_connections 2024-03-16T11:50:40.275 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.269+0000 7f259884e740 1 --2- 172.21.15.29:0/1050116020 >> [v2:172.21.15.152:6836/1323866722,v1:172.21.15.152:6837/1323866722] conn(0x561d55390180 0x561d5538c5c0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:40.276 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.269+0000 7f256effd640 0 client.15246 ms_handle_reset on v2:172.21.15.152:6836/1323866722 2024-03-16T11:50:40.276 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.269+0000 7f259884e740 1 --2- 172.21.15.29:0/1050116020 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561d55314730 0x561d5531ee90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:40.276 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.269+0000 7f259884e740 1 --2- 172.21.15.29:0/1050116020 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561d5530f5c0 0x561d5531bec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:40.276 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.269+0000 7f259884e740 1 --2- 172.21.15.29:0/1050116020 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561d5530ebd0 0x561d5531b980 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:40.276 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.269+0000 7f259884e740 1 -- 172.21.15.29:0/1050116020 >> 172.21.15.29:0/1050116020 conn(0x561d5530a140 msgr2=0x561d55318cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:50:40.276 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.270+0000 7f259884e740 1 -- 172.21.15.29:0/1050116020 shutdown_connections 2024-03-16T11:50:40.276 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.270+0000 7f259884e740 1 -- 172.21.15.29:0/1050116020 wait complete. 2024-03-16T11:50:40.276 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.270+0000 7f259884e740 20 client.15246 trim_cache size 0 max 16384 2024-03-16T11:50:40.276 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-03-16T11:50:40.276 INFO:teuthology.orchestra.run.smithi029.stdout: "cluster_fsid": "d1887f28-e389-11ee-95c9-87774f69a715", 2024-03-16T11:50:40.276 INFO:teuthology.orchestra.run.smithi029.stdout: "whoami": 2, 2024-03-16T11:50:40.276 INFO:teuthology.orchestra.run.smithi029.stdout: "id": 24464, 2024-03-16T11:50:40.277 INFO:teuthology.orchestra.run.smithi029.stdout: "want_state": "up:active", 2024-03-16T11:50:40.277 INFO:teuthology.orchestra.run.smithi029.stdout: "state": "up:active", 2024-03-16T11:50:40.277 INFO:teuthology.orchestra.run.smithi029.stdout: "fs_name": "cephfs", 2024-03-16T11:50:40.277 INFO:teuthology.orchestra.run.smithi029.stdout: "rank_uptime": 118.305635541, 2024-03-16T11:50:40.277 INFO:teuthology.orchestra.run.smithi029.stdout: "mdsmap_epoch": 21, 2024-03-16T11:50:40.277 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_epoch": 87, 2024-03-16T11:50:40.277 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_epoch_barrier": 85, 2024-03-16T11:50:40.277 INFO:teuthology.orchestra.run.smithi029.stdout: "uptime": 138.45232243000001 2024-03-16T11:50:40.277 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-03-16T11:50:40.278 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.270+0000 7f2595c68640 1 -- 172.21.15.29:0/265838325 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f25780a2800 msgr2=0x7f25780a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:40.278 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.270+0000 7f2595c68640 1 --2- 172.21.15.29:0/265838325 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f25780a2800 0x7f25780a4cc0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f2588004ab0 tx=0x7f2588005e30 comp rx=0 tx=0).stop 2024-03-16T11:50:40.278 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.270+0000 7f2595c68640 1 -- 172.21.15.29:0/265838325 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f25901006f0 msgr2=0x7f259006a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:40.278 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.270+0000 7f2595c68640 1 --2- 172.21.15.29:0/265838325 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f25901006f0 0x7f259006a940 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f258000e9c0 tx=0x7f258000ee80 comp rx=0 tx=0).stop 2024-03-16T11:50:40.278 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.270+0000 7f2595c68640 1 -- 172.21.15.29:0/265838325 shutdown_connections 2024-03-16T11:50:40.278 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.270+0000 7f2595c68640 1 --2- 172.21.15.29:0/265838325 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f25780a2800 0x7f25780a4cc0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:40.278 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.270+0000 7f2595c68640 1 --2- 172.21.15.29:0/265838325 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f25901022a0 0x7f259019ee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:40.278 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.270+0000 7f2595c68640 1 --2- 172.21.15.29:0/265838325 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f25901018e0 0x7f259019c800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:40.279 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.270+0000 7f2595c68640 1 --2- 172.21.15.29:0/265838325 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f25901006f0 0x7f259006a940 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:40.279 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.270+0000 7f2595c68640 1 -- 172.21.15.29:0/265838325 >> 172.21.15.29:0/265838325 conn(0x7f25900fbc80 msgr2=0x7f2590104c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:50:40.279 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.270+0000 7f2595c68640 1 -- 172.21.15.29:0/265838325 shutdown_connections 2024-03-16T11:50:40.279 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.270+0000 7f2595c68640 1 -- 172.21.15.29:0/265838325 wait complete. 2024-03-16T11:50:40.280 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.i status 2024-03-16T11:50:40.388 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.387+0000 7fd3f7177640 1 -- 172.21.15.29:0/346444130 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd3f01022a0 msgr2=0x7fd3f0108b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:40.388 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.387+0000 7fd3f7177640 1 --2- 172.21.15.29:0/346444130 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd3f01022a0 0x7fd3f0108b30 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fd3d8009900 tx=0x7fd3d802bc30 comp rx=0 tx=0).stop 2024-03-16T11:50:40.388 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.387+0000 7fd3f7177640 1 -- 172.21.15.29:0/346444130 shutdown_connections 2024-03-16T11:50:40.388 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.387+0000 7fd3f7177640 1 --2- 172.21.15.29:0/346444130 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd3f01022a0 0x7fd3f0108b30 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:40.388 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.387+0000 7fd3f7177640 1 --2- 172.21.15.29:0/346444130 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd3f01018e0 0x7fd3f0101d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:40.388 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.387+0000 7fd3f7177640 1 --2- 172.21.15.29:0/346444130 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd3f01006f0 0x7fd3f0100b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:40.388 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.387+0000 7fd3f7177640 1 -- 172.21.15.29:0/346444130 >> 172.21.15.29:0/346444130 conn(0x7fd3f00fbc80 msgr2=0x7fd3f00fe0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:50:40.388 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.387+0000 7fd3f7177640 1 -- 172.21.15.29:0/346444130 shutdown_connections 2024-03-16T11:50:40.388 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.387+0000 7fd3f7177640 1 -- 172.21.15.29:0/346444130 wait complete. 2024-03-16T11:50:40.388 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.387+0000 7fd3f7177640 1 Processor -- start 2024-03-16T11:50:40.388 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.387+0000 7fd3f7177640 1 -- start start 2024-03-16T11:50:40.388 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.387+0000 7fd3f7177640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd3f01006f0 0x7fd3f0197b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:40.388 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.387+0000 7fd3f7177640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd3f01018e0 0x7fd3f0198040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:40.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.387+0000 7fd3f7177640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd3f01022a0 0x7fd3f019f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:40.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.387+0000 7fd3f7177640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fd3f010c370 con 0x7fd3f01006f0 2024-03-16T11:50:40.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.387+0000 7fd3f7177640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fd3f010c1f0 con 0x7fd3f01022a0 2024-03-16T11:50:40.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.387+0000 7fd3f7177640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fd3f010c4f0 con 0x7fd3f01018e0 2024-03-16T11:50:40.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.388+0000 7fd3f4f13640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd3f01006f0 0x7fd3f0197b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:40.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.388+0000 7fd3f4f13640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd3f01006f0 0x7fd3f0197b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39710/0 (socket says 172.21.15.29:39710) 2024-03-16T11:50:40.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.388+0000 7fd3f4f13640 1 -- 172.21.15.29:0/1917173480 learned_addr learned my addr 172.21.15.29:0/1917173480 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:50:40.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.388+0000 7fd3effff640 1 --2- 172.21.15.29:0/1917173480 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd3f01018e0 0x7fd3f0198040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:40.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.388+0000 7fd3f5714640 1 --2- 172.21.15.29:0/1917173480 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd3f01022a0 0x7fd3f019f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:40.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.388+0000 7fd3f4f13640 1 -- 172.21.15.29:0/1917173480 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd3f01018e0 msgr2=0x7fd3f0198040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:40.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.388+0000 7fd3f4f13640 1 --2- 172.21.15.29:0/1917173480 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd3f01018e0 0x7fd3f0198040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:40.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.388+0000 7fd3f4f13640 1 -- 172.21.15.29:0/1917173480 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd3f01022a0 msgr2=0x7fd3f019f0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:40.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.388+0000 7fd3f4f13640 1 --2- 172.21.15.29:0/1917173480 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd3f01022a0 0x7fd3f019f0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:40.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.388+0000 7fd3f4f13640 1 -- 172.21.15.29:0/1917173480 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3f019f720 con 0x7fd3f01006f0 2024-03-16T11:50:40.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.388+0000 7fd3f5714640 1 --2- 172.21.15.29:0/1917173480 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd3f01022a0 0x7fd3f019f0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T11:50:40.390 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.388+0000 7fd3f4f13640 1 --2- 172.21.15.29:0/1917173480 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd3f01006f0 0x7fd3f0197b00 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fd3e800e9c0 tx=0x7fd3e800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:50:40.390 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.388+0000 7fd3edffb640 1 -- 172.21.15.29:0/1917173480 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3e800cd80 con 0x7fd3f01006f0 2024-03-16T11:50:40.390 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.388+0000 7fd3edffb640 1 -- 172.21.15.29:0/1917173480 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd3e80035f0 con 0x7fd3f01006f0 2024-03-16T11:50:40.390 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.388+0000 7fd3f7177640 1 -- 172.21.15.29:0/1917173480 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd3f019fa10 con 0x7fd3f01006f0 2024-03-16T11:50:40.390 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.388+0000 7fd3edffb640 1 -- 172.21.15.29:0/1917173480 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3e8010690 con 0x7fd3f01006f0 2024-03-16T11:50:40.390 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.388+0000 7fd3f7177640 1 -- 172.21.15.29:0/1917173480 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd3f019ff10 con 0x7fd3f01006f0 2024-03-16T11:50:40.396 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.390+0000 7fd3edffb640 1 -- 172.21.15.29:0/1917173480 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fd3e80037a0 con 0x7fd3f01006f0 2024-03-16T11:50:40.396 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.391+0000 7fd3f9e38740 1 Processor -- start 2024-03-16T11:50:40.396 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.391+0000 7fd3f9e38740 1 -- start start 2024-03-16T11:50:40.396 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.391+0000 7fd3edffb640 1 --2- 172.21.15.29:0/1917173480 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fd3dc0a2800 0x7fd3dc0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:40.396 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.391+0000 7fd3f9e38740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b95fef6bd0 0x55b95fef6ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:40.397 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.391+0000 7fd3f9e38740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b95fef75c0 0x55b95fefc0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:40.397 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.391+0000 7fd3f9e38740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b95fefc730 0x55b95fefebc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:40.397 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.391+0000 7fd3f5714640 1 --2- 172.21.15.29:0/1917173480 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fd3dc0a2800 0x7fd3dc0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:40.397 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.391+0000 7fd3f9e38740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55b95fe5ac80 con 0x55b95fefc730 2024-03-16T11:50:40.397 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.391+0000 7fd3f9e38740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x55b95fe03aa0 con 0x55b95fef75c0 2024-03-16T11:50:40.397 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.391+0000 7fd3f9e38740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55b95fe56d20 con 0x55b95fef6bd0 2024-03-16T11:50:40.397 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.391+0000 7fd3edffb640 1 -- 172.21.15.29:0/1917173480 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8211+0+0 (secure 0 0 0) 0x7fd3e80cee60 con 0x7fd3f01006f0 2024-03-16T11:50:40.397 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.391+0000 7fd3f4f13640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b95fef6bd0 0x55b95fef6ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:40.397 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.391+0000 7fd3f5714640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b95fefc730 0x55b95fefebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:40.397 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.391+0000 7fd3f4f13640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b95fef6bd0 0x55b95fef6ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.29:60878/0 (socket says 172.21.15.29:60878) 2024-03-16T11:50:40.397 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.391+0000 7fd3f4f13640 1 -- 172.21.15.29:0/195333587 learned_addr learned my addr 172.21.15.29:0/195333587 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:50:40.397 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.391+0000 7fd3effff640 1 --2- 172.21.15.29:0/195333587 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b95fef75c0 0x55b95fefc0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:40.397 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.391+0000 7fd3f4f13640 1 -- 172.21.15.29:0/195333587 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b95fef75c0 msgr2=0x55b95fefc0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:50:40.397 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.391+0000 7fd3f4f13640 1 --2- 172.21.15.29:0/195333587 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b95fef75c0 0x55b95fefc0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:40.397 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.391+0000 7fd3f4f13640 1 -- 172.21.15.29:0/195333587 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b95fefc730 msgr2=0x55b95fefebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:50:40.398 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.391+0000 7fd3f4f13640 1 --2- 172.21.15.29:0/195333587 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b95fefc730 0x55b95fefebc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:40.398 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.392+0000 7fd3f4f13640 1 -- 172.21.15.29:0/195333587 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b95fe25ff0 con 0x55b95fef6bd0 2024-03-16T11:50:40.398 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.392+0000 7fd3f5714640 1 --2- 172.21.15.29:0/1917173480 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fd3dc0a2800 0x7fd3dc0a4cc0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fd3d8006010 tx=0x7fd3d8005dd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:50:40.398 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.392+0000 7fd3f4f13640 1 --2- 172.21.15.29:0/195333587 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b95fef6bd0 0x55b95fef6ff0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fd3e80981e0 tx=0x7fd3e809b710 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a611fb0334a852c server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:50:40.398 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.392+0000 7fd3e6ffd640 1 -- 172.21.15.29:0/195333587 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3e804a1d0 con 0x55b95fef6bd0 2024-03-16T11:50:40.398 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.392+0000 7fd3e6ffd640 1 -- 172.21.15.29:0/195333587 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd3e809bf10 con 0x55b95fef6bd0 2024-03-16T11:50:40.398 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.392+0000 7fd3e6ffd640 1 -- 172.21.15.29:0/195333587 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3e8086110 con 0x55b95fef6bd0 2024-03-16T11:50:40.398 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.393+0000 7fd3f9e38740 1 -- 172.21.15.29:0/195333587 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b95fef6bd0 msgr2=0x55b95fef6ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:40.398 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.393+0000 7fd3f9e38740 1 --2- 172.21.15.29:0/195333587 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b95fef6bd0 0x55b95fef6ff0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fd3e80981e0 tx=0x7fd3e809b710 comp rx=0 tx=0).stop 2024-03-16T11:50:40.398 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.393+0000 7fd3f9e38740 1 -- 172.21.15.29:0/195333587 shutdown_connections 2024-03-16T11:50:40.398 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.393+0000 7fd3f9e38740 1 --2- 172.21.15.29:0/195333587 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b95fefc730 0x55b95fefebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:40.398 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.393+0000 7fd3f9e38740 1 --2- 172.21.15.29:0/195333587 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b95fef75c0 0x55b95fefc0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:40.398 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.393+0000 7fd3f9e38740 1 --2- 172.21.15.29:0/195333587 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b95fef6bd0 0x55b95fef6ff0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:40.398 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.393+0000 7fd3f9e38740 1 -- 172.21.15.29:0/195333587 >> 172.21.15.29:0/195333587 conn(0x55b95fef2140 msgr2=0x55b95fef2540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:50:40.399 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.393+0000 7fd3f9e38740 1 -- 172.21.15.29:0/195333587 shutdown_connections 2024-03-16T11:50:40.399 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.393+0000 7fd3f9e38740 1 -- 172.21.15.29:0/195333587 wait complete. 2024-03-16T11:50:40.399 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.393+0000 7fd3f9e38740 1 -- start start 2024-03-16T11:50:40.399 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.393+0000 7fd3f9e38740 1 Processor -- start 2024-03-16T11:50:40.399 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.393+0000 7fd3f9e38740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b95fef6bd0 0x55b95ff00e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:40.399 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.393+0000 7fd3f9e38740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b95fef75c0 0x55b95ff01340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:40.399 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.393+0000 7fd3f9e38740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b95fefc730 0x55b95ff08380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:40.399 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.393+0000 7fd3f9e38740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55b95feff6e0 con 0x55b95fef6bd0 2024-03-16T11:50:40.399 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.393+0000 7fd3f9e38740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x55b95fe16ef0 con 0x55b95fef75c0 2024-03-16T11:50:40.399 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.393+0000 7fd3f9e38740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55b95feff860 con 0x55b95fefc730 2024-03-16T11:50:40.399 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.393+0000 7fd3f9e38740 10 client.0 _ll_register_callbacks cb 0x55b95fe11dc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T11:50:40.399 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.393+0000 7fd3f4f13640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b95fef6bd0 0x55b95ff00e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:40.399 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.394+0000 7fd3f4f13640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b95fef6bd0 0x55b95ff00e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39724/0 (socket says 172.21.15.29:39724) 2024-03-16T11:50:40.399 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.394+0000 7fd3f4f13640 1 -- 172.21.15.29:0/3021540904 learned_addr learned my addr 172.21.15.29:0/3021540904 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:50:40.399 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.394+0000 7fd3f5714640 1 --2- 172.21.15.29:0/3021540904 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b95fefc730 0x55b95ff08380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:40.400 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.394+0000 7fd3effff640 1 --2- 172.21.15.29:0/3021540904 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b95fef75c0 0x55b95ff01340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:40.400 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.394+0000 7fd3f4f13640 1 -- 172.21.15.29:0/3021540904 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b95fefc730 msgr2=0x55b95ff08380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:50:40.400 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.394+0000 7fd3f4f13640 1 --2- 172.21.15.29:0/3021540904 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b95fefc730 0x55b95ff08380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:40.400 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.394+0000 7fd3f4f13640 1 -- 172.21.15.29:0/3021540904 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b95fef75c0 msgr2=0x55b95ff01340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:50:40.400 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.394+0000 7fd3f4f13640 1 --2- 172.21.15.29:0/3021540904 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b95fef75c0 0x55b95ff01340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:40.400 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.394+0000 7fd3f4f13640 1 -- 172.21.15.29:0/3021540904 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b95fe23a30 con 0x55b95fef6bd0 2024-03-16T11:50:40.400 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.394+0000 7fd3f5714640 1 --2- 172.21.15.29:0/3021540904 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b95fefc730 0x55b95ff08380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:50:40.400 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.394+0000 7fd3f4f13640 1 --2- 172.21.15.29:0/3021540904 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b95fef6bd0 0x55b95ff00e00 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fd3e800c6a0 tx=0x7fd3e8099b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df45dc16c9d4d929 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:50:40.400 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.394+0000 7fd3cffff640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-16T11:50:40.400 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.394+0000 7fd3cffff640 1 -- 172.21.15.29:0/3021540904 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3e809b810 con 0x55b95fef6bd0 2024-03-16T11:50:40.400 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.394+0000 7fd3cffff640 1 -- 172.21.15.29:0/3021540904 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8211+0+0 (secure 0 0 0) 0x7fd3e804a530 con 0x55b95fef6bd0 2024-03-16T11:50:40.400 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.394+0000 7fd3f9e38740 1 -- 172.21.15.29:0/3021540904 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd3e809bf10 con 0x55b95fef6bd0 2024-03-16T11:50:40.400 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.394+0000 7fd3cffff640 1 -- 172.21.15.29:0/3021540904 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fd3e80b9050 con 0x55b95fef6bd0 2024-03-16T11:50:40.400 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.394+0000 7fd3f9e38740 10 client.15258 fetch_fsmap learned FSMap version 21 2024-03-16T11:50:40.400 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.394+0000 7fd3f9e38740 1 -- 172.21.15.29:0/3021540904 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55b95fb6f2d0 con 0x55b95fef6bd0 2024-03-16T11:50:40.400 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.394+0000 7fd3cffff640 1 -- 172.21.15.29:0/3021540904 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd3e809bf10 con 0x55b95fef6bd0 2024-03-16T11:50:40.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.395+0000 7fd3cffff640 1 -- 172.21.15.29:0/3021540904 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7fd3e809cb40 con 0x55b95fef6bd0 2024-03-16T11:50:40.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.395+0000 7fd3f9e38740 10 client.15258 fetch_fsmap finished waiting for FSMap version 21 2024-03-16T11:50:40.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.395+0000 7fd3f9e38740 10 client.15258 resolve_mds: resolved name 'i' to daemon mds.i 2024-03-16T11:50:40.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.395+0000 7fd3f9e38740 20 client.15258 populate_metadata read hostname 'smithi029' 2024-03-16T11:50:40.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.395+0000 7fd3f9e38740 1 --2- 172.21.15.29:0/3021540904 >> [v2:172.21.15.154:6838/685736862,v1:172.21.15.154:6839/685736862] conn(0x55b95ff0d360 0x55b95ff0e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:40.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.395+0000 7fd3f9e38740 4 client.15258 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T11:50:40.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.395+0000 7fd3f9e38740 1 -- 172.21.15.29:0/3021540904 --> [v2:172.21.15.154:6838/685736862,v1:172.21.15.154:6839/685736862] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b95fe681c0 con 0x55b95ff0d360 2024-03-16T11:50:40.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.395+0000 7fd3effff640 1 --2- 172.21.15.29:0/3021540904 >> [v2:172.21.15.154:6838/685736862,v1:172.21.15.154:6839/685736862] conn(0x55b95ff0d360 0x55b95ff0e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:40.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.395+0000 7fd3effff640 1 --2- 172.21.15.29:0/3021540904 >> [v2:172.21.15.154:6838/685736862,v1:172.21.15.154:6839/685736862] conn(0x55b95ff0d360 0x55b95ff0e7d0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=89e2540d2c714bbc server_cookie=b39d5500edfcdab8 in_seq=0 out_seq=0 2024-03-16T11:50:40.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.395+0000 7fd3cffff640 10 client.15258 ms_handle_connect on v2:172.21.15.154:6838/685736862 2024-03-16T11:50:40.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.396+0000 7fd3cffff640 1 -- 172.21.15.29:0/3021540904 <== mds.1 v2:172.21.15.154:6838/685736862 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55b95fe681c0 con 0x55b95ff0d360 2024-03-16T11:50:40.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.396+0000 7fd3cffff640 10 client.15258 handle_command_reply: tid=0 2024-03-16T11:50:40.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.396+0000 7fd3f9e38740 1 client.15258 shutdown 2024-03-16T11:50:40.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.396+0000 7fd3f9e38740 1 -- 172.21.15.29:0/3021540904 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b95fef6bd0 msgr2=0x55b95ff00e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:40.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.396+0000 7fd3f9e38740 1 --2- 172.21.15.29:0/3021540904 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b95fef6bd0 0x55b95ff00e00 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fd3e800c6a0 tx=0x7fd3e8099b80 comp rx=0 tx=0).stop 2024-03-16T11:50:40.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.396+0000 7fd3f9e38740 1 -- 172.21.15.29:0/3021540904 shutdown_connections 2024-03-16T11:50:40.402 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.396+0000 7fd3f9e38740 1 --2- 172.21.15.29:0/3021540904 >> [v2:172.21.15.154:6838/685736862,v1:172.21.15.154:6839/685736862] conn(0x55b95ff0d360 0x55b95ff0e7d0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:40.402 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.396+0000 7fd3f9e38740 1 --2- 172.21.15.29:0/3021540904 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b95fefc730 0x55b95ff08380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:40.402 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.396+0000 7fd3f9e38740 1 --2- 172.21.15.29:0/3021540904 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b95fef75c0 0x55b95ff01340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:40.402 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.396+0000 7fd3f9e38740 1 --2- 172.21.15.29:0/3021540904 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b95fef6bd0 0x55b95ff00e00 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:40.402 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.396+0000 7fd3f9e38740 1 -- 172.21.15.29:0/3021540904 >> 172.21.15.29:0/3021540904 conn(0x55b95fef2140 msgr2=0x55b95fef1380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:50:40.402 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.396+0000 7fd3cffff640 0 client.15258 ms_handle_reset on v2:172.21.15.154:6838/685736862 2024-03-16T11:50:40.402 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.396+0000 7fd3f9e38740 1 -- 172.21.15.29:0/3021540904 shutdown_connections 2024-03-16T11:50:40.402 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.396+0000 7fd3f9e38740 1 -- 172.21.15.29:0/3021540904 wait complete. 2024-03-16T11:50:40.402 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.396+0000 7fd3f9e38740 20 client.15258 trim_cache size 0 max 16384 2024-03-16T11:50:40.418 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.414+0000 7fd3f9e38740 1 Processor -- start 2024-03-16T11:50:40.418 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.414+0000 7fd3f9e38740 1 -- start start 2024-03-16T11:50:40.419 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.414+0000 7fd3f9e38740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b95fef6bd0 0x55b95feff5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:40.419 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.414+0000 7fd3f9e38740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b95fef75c0 0x55b95ff08340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:40.419 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.414+0000 7fd3f9e38740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b95fefc730 0x55b95fef2520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:40.419 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.414+0000 7fd3f9e38740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55b95ff0cb10 con 0x55b95fef75c0 2024-03-16T11:50:40.419 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.414+0000 7fd3f9e38740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x55b95ff0d1e0 con 0x55b95fefc730 2024-03-16T11:50:40.419 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.414+0000 7fd3f9e38740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55b95ff0d8b0 con 0x55b95fef6bd0 2024-03-16T11:50:40.419 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.415+0000 7fd3effff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b95fef75c0 0x55b95ff08340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:40.419 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.415+0000 7fd3effff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b95fef75c0 0x55b95ff08340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39726/0 (socket says 172.21.15.29:39726) 2024-03-16T11:50:40.419 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.415+0000 7fd3effff640 1 -- 172.21.15.29:0/4205443276 learned_addr learned my addr 172.21.15.29:0/4205443276 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:50:40.419 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.415+0000 7fd3f5714640 1 --2- 172.21.15.29:0/4205443276 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b95fefc730 0x55b95fef2520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:40.419 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.415+0000 7fd3f4f13640 1 --2- 172.21.15.29:0/4205443276 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b95fef6bd0 0x55b95feff5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:40.419 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.415+0000 7fd3effff640 1 -- 172.21.15.29:0/4205443276 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b95fef6bd0 msgr2=0x55b95feff5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:50:40.419 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.415+0000 7fd3effff640 1 --2- 172.21.15.29:0/4205443276 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b95fef6bd0 0x55b95feff5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:40.419 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.415+0000 7fd3effff640 1 -- 172.21.15.29:0/4205443276 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b95fefc730 msgr2=0x55b95fef2520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:50:40.419 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.415+0000 7fd3effff640 1 --2- 172.21.15.29:0/4205443276 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b95fefc730 0x55b95fef2520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:40.420 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.415+0000 7fd3effff640 1 -- 172.21.15.29:0/4205443276 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b95fc396b0 con 0x55b95fef75c0 2024-03-16T11:50:40.420 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.415+0000 7fd3effff640 1 --2- 172.21.15.29:0/4205443276 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b95fef75c0 0x55b95ff08340 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x55b95fef8030 tx=0x7fd3e0009500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dee14701b47801b0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:50:40.420 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.415+0000 7fd3e67fc640 1 -- 172.21.15.29:0/4205443276 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3e0023070 con 0x55b95fef75c0 2024-03-16T11:50:40.420 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.415+0000 7fd3e67fc640 1 -- 172.21.15.29:0/4205443276 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd3e000fd50 con 0x55b95fef75c0 2024-03-16T11:50:40.420 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.415+0000 7fd3e67fc640 1 -- 172.21.15.29:0/4205443276 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3e001e5d0 con 0x55b95fef75c0 2024-03-16T11:50:40.420 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.416+0000 7fd3f9e38740 1 -- 172.21.15.29:0/4205443276 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b95fef75c0 msgr2=0x55b95ff08340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:40.420 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.416+0000 7fd3f9e38740 1 --2- 172.21.15.29:0/4205443276 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b95fef75c0 0x55b95ff08340 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x55b95fef8030 tx=0x7fd3e0009500 comp rx=0 tx=0).stop 2024-03-16T11:50:40.420 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.416+0000 7fd3f9e38740 1 -- 172.21.15.29:0/4205443276 shutdown_connections 2024-03-16T11:50:40.420 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.416+0000 7fd3f9e38740 1 --2- 172.21.15.29:0/4205443276 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b95fefc730 0x55b95fef2520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:40.420 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.416+0000 7fd3f9e38740 1 --2- 172.21.15.29:0/4205443276 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b95fef75c0 0x55b95ff08340 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:40.420 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.416+0000 7fd3f9e38740 1 --2- 172.21.15.29:0/4205443276 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b95fef6bd0 0x55b95feff5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:40.420 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.416+0000 7fd3f9e38740 1 -- 172.21.15.29:0/4205443276 >> 172.21.15.29:0/4205443276 conn(0x55b95fef2140 msgr2=0x55b95fef5c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:50:40.420 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.416+0000 7fd3f9e38740 1 -- 172.21.15.29:0/4205443276 shutdown_connections 2024-03-16T11:50:40.420 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.416+0000 7fd3f9e38740 1 -- 172.21.15.29:0/4205443276 wait complete. 2024-03-16T11:50:40.421 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.416+0000 7fd3f9e38740 1 -- start start 2024-03-16T11:50:40.421 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.416+0000 7fd3f9e38740 1 Processor -- start 2024-03-16T11:50:40.421 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.416+0000 7fd3f9e38740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b95fef6bd0 0x55b95fefa860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:40.421 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.416+0000 7fd3f9e38740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b95fef75c0 0x55b95fefada0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:40.421 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.416+0000 7fd3f9e38740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b95fefc730 0x55b95ff710c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:40.421 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.416+0000 7fd3f9e38740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55b95ff0e650 con 0x55b95fef6bd0 2024-03-16T11:50:40.421 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.416+0000 7fd3f9e38740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x55b95ff0df80 con 0x55b95fef75c0 2024-03-16T11:50:40.421 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.416+0000 7fd3f9e38740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55b95ff0ed20 con 0x55b95fefc730 2024-03-16T11:50:40.421 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.416+0000 7fd3f9e38740 10 client.0 _ll_register_callbacks cb 0x7fd3e00093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T11:50:40.421 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.417+0000 7fd3f4f13640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b95fef6bd0 0x55b95fefa860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:40.421 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.417+0000 7fd3f4f13640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b95fef6bd0 0x55b95fefa860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39734/0 (socket says 172.21.15.29:39734) 2024-03-16T11:50:40.421 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.417+0000 7fd3effff640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b95fef75c0 0x55b95fefada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:40.421 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.417+0000 7fd3f4f13640 1 -- 172.21.15.29:0/3248057772 learned_addr learned my addr 172.21.15.29:0/3248057772 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:50:40.421 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.417+0000 7fd3f5714640 1 --2- 172.21.15.29:0/3248057772 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b95fefc730 0x55b95ff710c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:40.422 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.417+0000 7fd3f4f13640 1 -- 172.21.15.29:0/3248057772 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b95fefc730 msgr2=0x55b95ff710c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:50:40.422 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.417+0000 7fd3f4f13640 1 --2- 172.21.15.29:0/3248057772 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b95fefc730 0x55b95ff710c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:40.422 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.417+0000 7fd3f4f13640 1 -- 172.21.15.29:0/3248057772 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b95fef75c0 msgr2=0x55b95fefada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:50:40.422 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.417+0000 7fd3f4f13640 1 --2- 172.21.15.29:0/3248057772 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b95fef75c0 0x55b95fefada0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:40.422 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.417+0000 7fd3f4f13640 1 -- 172.21.15.29:0/3248057772 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b95ff71680 con 0x55b95fef6bd0 2024-03-16T11:50:40.422 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.417+0000 7fd3f4f13640 1 --2- 172.21.15.29:0/3248057772 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b95fef6bd0 0x55b95fefa860 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fd3e809b7a0 tx=0x7fd3e8086320 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0f0e042e865df0e server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:50:40.422 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.417+0000 7fd3cffff640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-16T11:50:40.422 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.417+0000 7fd3cffff640 1 -- 172.21.15.29:0/3248057772 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3e8099bf0 con 0x55b95fef6bd0 2024-03-16T11:50:40.422 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.417+0000 7fd3cffff640 1 -- 172.21.15.29:0/3248057772 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8211+0+0 (secure 0 0 0) 0x7fd3e80a3d20 con 0x55b95fef6bd0 2024-03-16T11:50:40.422 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.417+0000 7fd3f9e38740 1 -- 172.21.15.29:0/3248057772 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd3e000fd50 con 0x55b95fef6bd0 2024-03-16T11:50:40.422 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.417+0000 7fd3cffff640 1 -- 172.21.15.29:0/3248057772 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fd3e809cd50 con 0x55b95fef6bd0 2024-03-16T11:50:40.422 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.417+0000 7fd3f9e38740 10 client.15264 fetch_fsmap learned FSMap version 21 2024-03-16T11:50:40.422 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.417+0000 7fd3f9e38740 1 -- 172.21.15.29:0/3248057772 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55b95ff71680 con 0x55b95fef6bd0 2024-03-16T11:50:40.422 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.417+0000 7fd3cffff640 1 -- 172.21.15.29:0/3248057772 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd3e000fd50 con 0x55b95fef6bd0 2024-03-16T11:50:40.423 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.418+0000 7fd3cffff640 1 -- 172.21.15.29:0/3248057772 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7fd3e804a470 con 0x55b95fef6bd0 2024-03-16T11:50:40.423 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.418+0000 7fd3f9e38740 10 client.15264 fetch_fsmap finished waiting for FSMap version 21 2024-03-16T11:50:40.423 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.418+0000 7fd3f9e38740 10 client.15264 resolve_mds: resolved name 'i' to daemon mds.i 2024-03-16T11:50:40.423 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.418+0000 7fd3f9e38740 20 client.15264 populate_metadata read hostname 'smithi029' 2024-03-16T11:50:40.423 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.418+0000 7fd3f9e38740 1 --2- 172.21.15.29:0/3248057772 >> [v2:172.21.15.154:6838/685736862,v1:172.21.15.154:6839/685736862] conn(0x55b95ff77200 0x55b95ff75f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:40.423 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.418+0000 7fd3f9e38740 4 client.15264 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-16T11:50:40.423 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.418+0000 7fd3f9e38740 1 -- 172.21.15.29:0/3248057772 --> [v2:172.21.15.154:6838/685736862,v1:172.21.15.154:6839/685736862] -- command(tid 0: {"prefix": "status"}) -- 0x55b95fe62510 con 0x55b95ff77200 2024-03-16T11:50:40.423 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.418+0000 7fd3effff640 1 --2- 172.21.15.29:0/3248057772 >> [v2:172.21.15.154:6838/685736862,v1:172.21.15.154:6839/685736862] conn(0x55b95ff77200 0x55b95ff75f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:40.423 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.418+0000 7fd3cffff640 10 client.15264 ms_handle_connect on v2:172.21.15.154:6838/685736862 2024-03-16T11:50:40.423 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.418+0000 7fd3effff640 1 --2- 172.21.15.29:0/3248057772 >> [v2:172.21.15.154:6838/685736862,v1:172.21.15.154:6839/685736862] conn(0x55b95ff77200 0x55b95ff75f10 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=9572aa8060676f7c server_cookie=e752186a603d0c0d in_seq=0 out_seq=0 2024-03-16T11:50:40.423 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.418+0000 7fd3cffff640 1 -- 172.21.15.29:0/3248057772 <== mds.1 v2:172.21.15.154:6838/685736862 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x55b95fe62510 con 0x55b95ff77200 2024-03-16T11:50:40.423 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.418+0000 7fd3cffff640 10 client.15264 handle_command_reply: tid=0 2024-03-16T11:50:40.423 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.418+0000 7fd3f9e38740 1 client.15264 shutdown 2024-03-16T11:50:40.423 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.418+0000 7fd3f9e38740 1 -- 172.21.15.29:0/3248057772 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b95fef6bd0 msgr2=0x55b95fefa860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:40.423 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.418+0000 7fd3f9e38740 1 --2- 172.21.15.29:0/3248057772 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b95fef6bd0 0x55b95fefa860 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fd3e809b7a0 tx=0x7fd3e8086320 comp rx=0 tx=0).stop 2024-03-16T11:50:40.424 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.418+0000 7fd3f9e38740 1 -- 172.21.15.29:0/3248057772 shutdown_connections 2024-03-16T11:50:40.424 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.418+0000 7fd3f9e38740 1 --2- 172.21.15.29:0/3248057772 >> [v2:172.21.15.154:6838/685736862,v1:172.21.15.154:6839/685736862] conn(0x55b95ff77200 0x55b95ff75f10 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:40.424 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.418+0000 7fd3f9e38740 1 --2- 172.21.15.29:0/3248057772 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b95fefc730 0x55b95ff710c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:40.424 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.418+0000 7fd3f9e38740 1 --2- 172.21.15.29:0/3248057772 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b95fef75c0 0x55b95fefada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:40.424 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.418+0000 7fd3f9e38740 1 --2- 172.21.15.29:0/3248057772 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b95fef6bd0 0x55b95fefa860 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:40.424 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.418+0000 7fd3cffff640 0 client.15264 ms_handle_reset on v2:172.21.15.154:6838/685736862 2024-03-16T11:50:40.424 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.418+0000 7fd3f9e38740 1 -- 172.21.15.29:0/3248057772 >> 172.21.15.29:0/3248057772 conn(0x55b95fef2140 msgr2=0x55b95fef5c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:50:40.424 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.419+0000 7fd3f9e38740 1 -- 172.21.15.29:0/3248057772 shutdown_connections 2024-03-16T11:50:40.424 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.419+0000 7fd3f9e38740 1 -- 172.21.15.29:0/3248057772 wait complete. 2024-03-16T11:50:40.424 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.419+0000 7fd3f9e38740 20 client.15264 trim_cache size 0 max 16384 2024-03-16T11:50:40.424 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-03-16T11:50:40.425 INFO:teuthology.orchestra.run.smithi029.stdout: "cluster_fsid": "d1887f28-e389-11ee-95c9-87774f69a715", 2024-03-16T11:50:40.425 INFO:teuthology.orchestra.run.smithi029.stdout: "whoami": 1, 2024-03-16T11:50:40.425 INFO:teuthology.orchestra.run.smithi029.stdout: "id": 24499, 2024-03-16T11:50:40.425 INFO:teuthology.orchestra.run.smithi029.stdout: "want_state": "up:active", 2024-03-16T11:50:40.425 INFO:teuthology.orchestra.run.smithi029.stdout: "state": "up:active", 2024-03-16T11:50:40.425 INFO:teuthology.orchestra.run.smithi029.stdout: "fs_name": "cephfs", 2024-03-16T11:50:40.425 INFO:teuthology.orchestra.run.smithi029.stdout: "rank_uptime": 119.381069939, 2024-03-16T11:50:40.425 INFO:teuthology.orchestra.run.smithi029.stdout: "mdsmap_epoch": 21, 2024-03-16T11:50:40.425 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_epoch": 87, 2024-03-16T11:50:40.425 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_epoch_barrier": 85, 2024-03-16T11:50:40.425 INFO:teuthology.orchestra.run.smithi029.stdout: "uptime": 135.77278318500001 2024-03-16T11:50:40.425 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-03-16T11:50:40.426 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.419+0000 7fd3f7177640 1 -- 172.21.15.29:0/1917173480 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fd3dc0a2800 msgr2=0x7fd3dc0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:40.426 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.419+0000 7fd3f7177640 1 --2- 172.21.15.29:0/1917173480 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fd3dc0a2800 0x7fd3dc0a4cc0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fd3d8006010 tx=0x7fd3d8005dd0 comp rx=0 tx=0).stop 2024-03-16T11:50:40.426 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.419+0000 7fd3f7177640 1 -- 172.21.15.29:0/1917173480 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd3f01006f0 msgr2=0x7fd3f0197b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:40.426 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.419+0000 7fd3f7177640 1 --2- 172.21.15.29:0/1917173480 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd3f01006f0 0x7fd3f0197b00 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fd3e800e9c0 tx=0x7fd3e800ee80 comp rx=0 tx=0).stop 2024-03-16T11:50:40.426 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.419+0000 7fd3f7177640 1 -- 172.21.15.29:0/1917173480 shutdown_connections 2024-03-16T11:50:40.426 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.419+0000 7fd3f7177640 1 --2- 172.21.15.29:0/1917173480 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fd3dc0a2800 0x7fd3dc0a4cc0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:40.427 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.419+0000 7fd3f7177640 1 --2- 172.21.15.29:0/1917173480 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd3f01022a0 0x7fd3f019f0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:40.427 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.419+0000 7fd3f7177640 1 --2- 172.21.15.29:0/1917173480 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd3f01018e0 0x7fd3f0198040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:40.427 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.419+0000 7fd3f7177640 1 --2- 172.21.15.29:0/1917173480 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd3f01006f0 0x7fd3f0197b00 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:40.427 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.419+0000 7fd3f7177640 1 -- 172.21.15.29:0/1917173480 >> 172.21.15.29:0/1917173480 conn(0x7fd3f00fbc80 msgr2=0x7fd3f00fd6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:50:40.427 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.419+0000 7fd3f7177640 1 -- 172.21.15.29:0/1917173480 shutdown_connections 2024-03-16T11:50:40.427 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.419+0000 7fd3f7177640 1 -- 172.21.15.29:0/1917173480 wait complete. 2024-03-16T11:50:40.429 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-03-16T11:50:40.535 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.533+0000 7f78b1e44640 1 -- 172.21.15.29:0/2933774237 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f78ac1006e0 msgr2=0x7f78ac100b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:40.535 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.534+0000 7f78b1e44640 1 --2- 172.21.15.29:0/2933774237 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f78ac1006e0 0x7f78ac100b00 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f789c009ca0 tx=0x7f789c02c0f0 comp rx=0 tx=0).stop 2024-03-16T11:50:40.535 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.534+0000 7f78b1e44640 1 -- 172.21.15.29:0/2933774237 shutdown_connections 2024-03-16T11:50:40.535 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.534+0000 7f78b1e44640 1 --2- 172.21.15.29:0/2933774237 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f78ac102290 0x7f78ac108b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:40.535 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.534+0000 7f78b1e44640 1 --2- 172.21.15.29:0/2933774237 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f78ac1018d0 0x7f78ac101d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:40.535 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.534+0000 7f78b1e44640 1 --2- 172.21.15.29:0/2933774237 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f78ac1006e0 0x7f78ac100b00 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:40.535 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.534+0000 7f78b1e44640 1 -- 172.21.15.29:0/2933774237 >> 172.21.15.29:0/2933774237 conn(0x7f78ac0fbc70 msgr2=0x7f78ac0fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:50:40.535 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.534+0000 7f78b1e44640 1 -- 172.21.15.29:0/2933774237 shutdown_connections 2024-03-16T11:50:40.535 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.534+0000 7f78b1e44640 1 -- 172.21.15.29:0/2933774237 wait complete. 2024-03-16T11:50:40.535 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.534+0000 7f78b1e44640 1 Processor -- start 2024-03-16T11:50:40.535 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.534+0000 7f78b1e44640 1 -- start start 2024-03-16T11:50:40.536 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.534+0000 7f78b1e44640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f78ac1006e0 0x7f78ac10b060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:40.536 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.534+0000 7f78b1e44640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f78ac1018d0 0x7f78ac1096a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:40.536 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.534+0000 7f78b1e44640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f78ac102290 0x7f78ac109be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:40.536 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.534+0000 7f78b1e44640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f78ac10d480 con 0x7f78ac1018d0 2024-03-16T11:50:40.536 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.534+0000 7f78b1e44640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f78ac10d300 con 0x7f78ac1006e0 2024-03-16T11:50:40.536 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.534+0000 7f78b1e44640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f78ac10d600 con 0x7f78ac102290 2024-03-16T11:50:40.536 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.535+0000 7f78aaffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f78ac1018d0 0x7f78ac1096a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:40.536 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.535+0000 7f78aaffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f78ac1018d0 0x7f78ac1096a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39762/0 (socket says 172.21.15.29:39762) 2024-03-16T11:50:40.536 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.535+0000 7f78aaffd640 1 -- 172.21.15.29:0/2805978406 learned_addr learned my addr 172.21.15.29:0/2805978406 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:50:40.536 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.535+0000 7f78ab7fe640 1 --2- 172.21.15.29:0/2805978406 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f78ac1006e0 0x7f78ac10b060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:40.536 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.535+0000 7f78abfff640 1 --2- 172.21.15.29:0/2805978406 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f78ac102290 0x7f78ac109be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:40.536 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.535+0000 7f78aaffd640 1 -- 172.21.15.29:0/2805978406 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f78ac102290 msgr2=0x7f78ac109be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:40.536 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.535+0000 7f78aaffd640 1 --2- 172.21.15.29:0/2805978406 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f78ac102290 0x7f78ac109be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:40.536 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.535+0000 7f78aaffd640 1 -- 172.21.15.29:0/2805978406 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f78ac1006e0 msgr2=0x7f78ac10b060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:40.536 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.535+0000 7f78aaffd640 1 --2- 172.21.15.29:0/2805978406 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f78ac1006e0 0x7f78ac10b060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:40.537 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.535+0000 7f78aaffd640 1 -- 172.21.15.29:0/2805978406 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78ac10a470 con 0x7f78ac1018d0 2024-03-16T11:50:40.537 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.535+0000 7f78aaffd640 1 --2- 172.21.15.29:0/2805978406 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f78ac1018d0 0x7f78ac1096a0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f789400ea60 tx=0x7f789400ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:50:40.537 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.535+0000 7f78a8ff9640 1 -- 172.21.15.29:0/2805978406 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f789400cde0 con 0x7f78ac1018d0 2024-03-16T11:50:40.537 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.535+0000 7f78a8ff9640 1 -- 172.21.15.29:0/2805978406 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f78940035f0 con 0x7f78ac1018d0 2024-03-16T11:50:40.537 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.535+0000 7f78b1e44640 1 -- 172.21.15.29:0/2805978406 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f78ac1a1c60 con 0x7f78ac1018d0 2024-03-16T11:50:40.537 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.535+0000 7f78a8ff9640 1 -- 172.21.15.29:0/2805978406 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7894010670 con 0x7f78ac1018d0 2024-03-16T11:50:40.537 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.535+0000 7f78b1e44640 1 -- 172.21.15.29:0/2805978406 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f78ac1a21b0 con 0x7f78ac1018d0 2024-03-16T11:50:40.543 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.538+0000 7f78a8ff9640 1 -- 172.21.15.29:0/2805978406 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f78940037a0 con 0x7f78ac1018d0 2024-03-16T11:50:40.543 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.538+0000 7f78b4a2e740 1 Processor -- start 2024-03-16T11:50:40.544 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.538+0000 7f78b4a2e740 1 -- start start 2024-03-16T11:50:40.544 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.538+0000 7f78b4a2e740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c4164e1bd0 0x55c4164e1ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:40.544 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.538+0000 7f78b4a2e740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c4164e25c0 0x55c4164e70c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:40.544 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.538+0000 7f78b4a2e740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c4164e7730 0x55c4164e9bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:40.544 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.538+0000 7f78b4a2e740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55c416445c80 con 0x55c4164e25c0 2024-03-16T11:50:40.544 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.538+0000 7f78b4a2e740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x55c4163eeaa0 con 0x55c4164e7730 2024-03-16T11:50:40.544 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.538+0000 7f78b4a2e740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55c416441d20 con 0x55c4164e1bd0 2024-03-16T11:50:40.544 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.538+0000 7f78ab7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c4164e25c0 0x55c4164e70c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:40.544 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.538+0000 7f78abfff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c4164e1bd0 0x55c4164e1ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:40.544 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.538+0000 7f78ab7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c4164e25c0 0x55c4164e70c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39776/0 (socket says 172.21.15.29:39776) 2024-03-16T11:50:40.544 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.538+0000 7f78abfff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c4164e1bd0 0x55c4164e1ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.29:60958/0 (socket says 172.21.15.29:60958) 2024-03-16T11:50:40.544 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.538+0000 7f78ab7fe640 1 -- 172.21.15.29:0/123335295 learned_addr learned my addr 172.21.15.29:0/123335295 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:50:40.544 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.538+0000 7f78aaffd640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c4164e7730 0x55c4164e9bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:40.544 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.538+0000 7f78a8ff9640 1 --2- 172.21.15.29:0/2805978406 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f78980a2800 0x7f78980a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:40.544 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.539+0000 7f78abfff640 1 -- 172.21.15.29:0/123335295 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c4164e7730 msgr2=0x55c4164e9bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:50:40.545 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.539+0000 7f78abfff640 1 --2- 172.21.15.29:0/123335295 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c4164e7730 0x55c4164e9bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:40.545 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.539+0000 7f78abfff640 1 -- 172.21.15.29:0/123335295 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c4164e25c0 msgr2=0x55c4164e70c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:50:40.545 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.539+0000 7f78abfff640 1 --2- 172.21.15.29:0/123335295 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c4164e25c0 0x55c4164e70c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:40.545 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.539+0000 7f78abfff640 1 -- 172.21.15.29:0/123335295 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c416410ff0 con 0x55c4164e1bd0 2024-03-16T11:50:40.545 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.539+0000 7f78ab7fe640 1 --2- 172.21.15.29:0/123335295 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c4164e25c0 0x55c4164e70c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:50:40.545 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.539+0000 7f78a8ff9640 1 -- 172.21.15.29:0/2805978406 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8211+0+0 (secure 0 0 0) 0x7f78940cedf0 con 0x7f78ac1018d0 2024-03-16T11:50:40.545 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.539+0000 7f78abfff640 1 --2- 172.21.15.29:0/2805978406 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f78980a2800 0x7f78980a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:40.545 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.539+0000 7f78abfff640 1 --2- 172.21.15.29:0/123335295 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c4164e1bd0 0x55c4164e1ff0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f78a4009ca0 tx=0x7f78a4011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=76afb1ceb6c814d3 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:50:40.545 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.539+0000 7f78a1ffb640 1 -- 172.21.15.29:0/123335295 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78a4017050 con 0x55c4164e1bd0 2024-03-16T11:50:40.545 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.539+0000 7f78abfff640 1 --2- 172.21.15.29:0/2805978406 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f78980a2800 0x7f78980a4cc0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f78a4023000 tx=0x7f78a4024000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:50:40.545 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.539+0000 7f78a1ffb640 1 -- 172.21.15.29:0/123335295 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f78a4003c40 con 0x55c4164e1bd0 2024-03-16T11:50:40.545 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.539+0000 7f78a1ffb640 1 -- 172.21.15.29:0/123335295 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78a400f3c0 con 0x55c4164e1bd0 2024-03-16T11:50:40.545 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.540+0000 7f78b4a2e740 1 -- 172.21.15.29:0/123335295 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c4164e1bd0 msgr2=0x55c4164e1ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:40.545 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.540+0000 7f78b4a2e740 1 --2- 172.21.15.29:0/123335295 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c4164e1bd0 0x55c4164e1ff0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f78a4009ca0 tx=0x7f78a4011810 comp rx=0 tx=0).stop 2024-03-16T11:50:40.545 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.540+0000 7f78b4a2e740 1 -- 172.21.15.29:0/123335295 shutdown_connections 2024-03-16T11:50:40.545 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.540+0000 7f78b4a2e740 1 --2- 172.21.15.29:0/123335295 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c4164e7730 0x55c4164e9bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:40.546 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.540+0000 7f78b4a2e740 1 --2- 172.21.15.29:0/123335295 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c4164e25c0 0x55c4164e70c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:40.546 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.540+0000 7f78b4a2e740 1 --2- 172.21.15.29:0/123335295 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c4164e1bd0 0x55c4164e1ff0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:40.546 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.540+0000 7f78b4a2e740 1 -- 172.21.15.29:0/123335295 >> 172.21.15.29:0/123335295 conn(0x55c4164dd140 msgr2=0x55c4164dd540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:50:40.546 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.540+0000 7f78b4a2e740 1 -- 172.21.15.29:0/123335295 shutdown_connections 2024-03-16T11:50:40.546 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.540+0000 7f78b4a2e740 1 -- 172.21.15.29:0/123335295 wait complete. 2024-03-16T11:50:40.546 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.540+0000 7f78b4a2e740 1 -- start start 2024-03-16T11:50:40.546 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.540+0000 7f78b4a2e740 1 Processor -- start 2024-03-16T11:50:40.546 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.540+0000 7f78b4a2e740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c4164e1bd0 0x55c4164ebde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:40.546 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.540+0000 7f78b4a2e740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c4164e25c0 0x55c4164ec320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:40.546 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.541+0000 7f78b4a2e740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c4164e7730 0x55c4164f3360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:40.546 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.541+0000 7f78b4a2e740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55c4164ea6e0 con 0x55c4164e1bd0 2024-03-16T11:50:40.546 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.541+0000 7f78b4a2e740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x55c416401ef0 con 0x55c4164e7730 2024-03-16T11:50:40.546 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.541+0000 7f78b4a2e740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55c4164ea860 con 0x55c4164e25c0 2024-03-16T11:50:40.546 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.541+0000 7f78b4a2e740 10 client.0 _ll_register_callbacks cb 0x55c4163fcdc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T11:50:40.546 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.541+0000 7f78ab7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c4164e1bd0 0x55c4164ebde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:40.547 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.541+0000 7f78ab7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c4164e1bd0 0x55c4164ebde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39792/0 (socket says 172.21.15.29:39792) 2024-03-16T11:50:40.547 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.541+0000 7f78ab7fe640 1 -- 172.21.15.29:0/3255685226 learned_addr learned my addr 172.21.15.29:0/3255685226 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:50:40.547 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.541+0000 7f78aaffd640 1 --2- 172.21.15.29:0/3255685226 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c4164e25c0 0x55c4164ec320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:40.547 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.541+0000 7f78abfff640 1 --2- 172.21.15.29:0/3255685226 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c4164e7730 0x55c4164f3360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:40.547 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.541+0000 7f78ab7fe640 1 -- 172.21.15.29:0/3255685226 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c4164e25c0 msgr2=0x55c4164ec320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:50:40.547 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.541+0000 7f78ab7fe640 1 --2- 172.21.15.29:0/3255685226 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c4164e25c0 0x55c4164ec320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:40.547 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.541+0000 7f78ab7fe640 1 -- 172.21.15.29:0/3255685226 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c4164e7730 msgr2=0x55c4164f3360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:50:40.547 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.541+0000 7f78ab7fe640 1 --2- 172.21.15.29:0/3255685226 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c4164e7730 0x55c4164f3360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:40.547 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.541+0000 7f78ab7fe640 1 -- 172.21.15.29:0/3255685226 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c41640ea30 con 0x55c4164e1bd0 2024-03-16T11:50:40.547 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.541+0000 7f78ab7fe640 1 --2- 172.21.15.29:0/3255685226 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c4164e1bd0 0x55c4164ebde0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f78ac101730 tx=0x7f789c005fc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8638c904db49be05 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:50:40.547 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.541+0000 7f788affd640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-16T11:50:40.547 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.541+0000 7f788affd640 1 -- 172.21.15.29:0/3255685226 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f789c044070 con 0x55c4164e1bd0 2024-03-16T11:50:40.547 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.541+0000 7f788affd640 1 -- 172.21.15.29:0/3255685226 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8211+0+0 (secure 0 0 0) 0x7f789c032690 con 0x55c4164e1bd0 2024-03-16T11:50:40.547 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.541+0000 7f78b4a2e740 1 -- 172.21.15.29:0/3255685226 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f78a4003c40 con 0x55c4164e1bd0 2024-03-16T11:50:40.548 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.541+0000 7f788affd640 1 -- 172.21.15.29:0/3255685226 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f789c050050 con 0x55c4164e1bd0 2024-03-16T11:50:40.548 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.541+0000 7f78b4a2e740 10 client.15276 fetch_fsmap learned FSMap version 21 2024-03-16T11:50:40.548 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.541+0000 7f78b4a2e740 1 -- 172.21.15.29:0/3255685226 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55c41615a2d0 con 0x55c4164e1bd0 2024-03-16T11:50:40.548 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.542+0000 7f788affd640 1 -- 172.21.15.29:0/3255685226 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f78a4003c40 con 0x55c4164e1bd0 2024-03-16T11:50:40.548 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.542+0000 7f788affd640 1 -- 172.21.15.29:0/3255685226 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f789c004270 con 0x55c4164e1bd0 2024-03-16T11:50:40.548 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.542+0000 7f78b4a2e740 10 client.15276 fetch_fsmap finished waiting for FSMap version 21 2024-03-16T11:50:40.548 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.542+0000 7f78b4a2e740 10 client.15276 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-16T11:50:40.548 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.542+0000 7f78b4a2e740 20 client.15276 populate_metadata read hostname 'smithi029' 2024-03-16T11:50:40.548 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.542+0000 7f78b4a2e740 1 --2- 172.21.15.29:0/3255685226 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55c4164f8860 0x55c4164f9cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:40.548 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.542+0000 7f78b4a2e740 4 client.15276 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T11:50:40.548 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.542+0000 7f78b4a2e740 1 -- 172.21.15.29:0/3255685226 --> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c4164531c0 con 0x55c4164f8860 2024-03-16T11:50:40.548 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.542+0000 7f78aaffd640 1 --2- 172.21.15.29:0/3255685226 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55c4164f8860 0x55c4164f9cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:40.548 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.542+0000 7f788affd640 10 client.15276 ms_handle_connect on v2:172.21.15.152:6838/1548731833 2024-03-16T11:50:40.548 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.542+0000 7f78aaffd640 1 --2- 172.21.15.29:0/3255685226 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55c4164f8860 0x55c4164f9cd0 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=680955cad99ca8f server_cookie=2224911214cecb86 in_seq=0 out_seq=0 2024-03-16T11:50:40.548 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.543+0000 7f788affd640 1 -- 172.21.15.29:0/3255685226 <== mds.0 v2:172.21.15.152:6838/1548731833 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55c4164531c0 con 0x55c4164f8860 2024-03-16T11:50:40.548 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.543+0000 7f788affd640 10 client.15276 handle_command_reply: tid=0 2024-03-16T11:50:40.549 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.543+0000 7f78b4a2e740 1 client.15276 shutdown 2024-03-16T11:50:40.549 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.544+0000 7f78b4a2e740 1 -- 172.21.15.29:0/3255685226 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c4164e1bd0 msgr2=0x55c4164ebde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:40.549 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.544+0000 7f78b4a2e740 1 --2- 172.21.15.29:0/3255685226 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c4164e1bd0 0x55c4164ebde0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f78ac101730 tx=0x7f789c005fc0 comp rx=0 tx=0).stop 2024-03-16T11:50:40.549 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.544+0000 7f78b4a2e740 1 -- 172.21.15.29:0/3255685226 shutdown_connections 2024-03-16T11:50:40.549 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.544+0000 7f78b4a2e740 1 --2- 172.21.15.29:0/3255685226 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55c4164f8860 0x55c4164f9cd0 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:40.549 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.544+0000 7f78b4a2e740 1 --2- 172.21.15.29:0/3255685226 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c4164e7730 0x55c4164f3360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:40.549 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.544+0000 7f788affd640 0 client.15276 ms_handle_reset on v2:172.21.15.152:6838/1548731833 2024-03-16T11:50:40.549 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.544+0000 7f78b4a2e740 1 --2- 172.21.15.29:0/3255685226 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c4164e25c0 0x55c4164ec320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:40.549 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.544+0000 7f78b4a2e740 1 --2- 172.21.15.29:0/3255685226 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c4164e1bd0 0x55c4164ebde0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:40.549 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.544+0000 7f78b4a2e740 1 -- 172.21.15.29:0/3255685226 >> 172.21.15.29:0/3255685226 conn(0x55c4164dd140 msgr2=0x55c4164dc380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:50:40.549 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.544+0000 7f78b4a2e740 1 -- 172.21.15.29:0/3255685226 shutdown_connections 2024-03-16T11:50:40.549 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.544+0000 7f78b4a2e740 1 -- 172.21.15.29:0/3255685226 wait complete. 2024-03-16T11:50:40.549 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.544+0000 7f78b4a2e740 20 client.15276 trim_cache size 0 max 16384 2024-03-16T11:50:40.566 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.562+0000 7f78b4a2e740 1 Processor -- start 2024-03-16T11:50:40.566 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.562+0000 7f78b4a2e740 1 -- start start 2024-03-16T11:50:40.566 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.562+0000 7f78b4a2e740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c4164e1bd0 0x55c4164e6da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:40.566 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.562+0000 7f78b4a2e740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c4164e25c0 0x55c4164f3320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:40.566 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.562+0000 7f78b4a2e740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c4164e7730 0x55c4164dd520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:40.567 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.562+0000 7f78b4a2e740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55c4164f8010 con 0x55c4164e1bd0 2024-03-16T11:50:40.567 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.562+0000 7f78b4a2e740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x55c4164f86e0 con 0x55c4164e25c0 2024-03-16T11:50:40.567 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.562+0000 7f78b4a2e740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55c4164f8db0 con 0x55c4164e7730 2024-03-16T11:50:40.567 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.562+0000 7f78ab7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c4164e1bd0 0x55c4164e6da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:40.567 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.562+0000 7f78ab7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c4164e1bd0 0x55c4164e6da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39806/0 (socket says 172.21.15.29:39806) 2024-03-16T11:50:40.567 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.562+0000 7f78ab7fe640 1 -- 172.21.15.29:0/4011719180 learned_addr learned my addr 172.21.15.29:0/4011719180 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:50:40.567 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.562+0000 7f78aaffd640 1 --2- 172.21.15.29:0/4011719180 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c4164e25c0 0x55c4164f3320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:40.567 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.562+0000 7f78abfff640 1 --2- 172.21.15.29:0/4011719180 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c4164e7730 0x55c4164dd520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:40.567 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.562+0000 7f78ab7fe640 1 -- 172.21.15.29:0/4011719180 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c4164e7730 msgr2=0x55c4164dd520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:50:40.567 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.562+0000 7f78ab7fe640 1 --2- 172.21.15.29:0/4011719180 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c4164e7730 0x55c4164dd520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:40.567 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.562+0000 7f78ab7fe640 1 -- 172.21.15.29:0/4011719180 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c4164e25c0 msgr2=0x55c4164f3320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:50:40.567 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.562+0000 7f78ab7fe640 1 --2- 172.21.15.29:0/4011719180 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c4164e25c0 0x55c4164f3320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:40.567 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.562+0000 7f78ab7fe640 1 -- 172.21.15.29:0/4011719180 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c4162246b0 con 0x55c4164e1bd0 2024-03-16T11:50:40.567 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.562+0000 7f78ab7fe640 1 --2- 172.21.15.29:0/4011719180 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c4164e1bd0 0x55c4164e6da0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f789c02c630 tx=0x7f789c005d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab217a4899432f2f server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:50:40.567 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.562+0000 7f78a17fa640 1 -- 172.21.15.29:0/4011719180 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f789c04d020 con 0x55c4164e1bd0 2024-03-16T11:50:40.568 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.562+0000 7f78a17fa640 1 -- 172.21.15.29:0/4011719180 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f789c032c90 con 0x55c4164e1bd0 2024-03-16T11:50:40.568 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.562+0000 7f78a17fa640 1 -- 172.21.15.29:0/4011719180 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f789c044070 con 0x55c4164e1bd0 2024-03-16T11:50:40.568 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.563+0000 7f78b4a2e740 1 -- 172.21.15.29:0/4011719180 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c4164e1bd0 msgr2=0x55c4164e6da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:40.568 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.563+0000 7f78b4a2e740 1 --2- 172.21.15.29:0/4011719180 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c4164e1bd0 0x55c4164e6da0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f789c02c630 tx=0x7f789c005d80 comp rx=0 tx=0).stop 2024-03-16T11:50:40.568 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.563+0000 7f78b4a2e740 1 -- 172.21.15.29:0/4011719180 shutdown_connections 2024-03-16T11:50:40.568 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.563+0000 7f78b4a2e740 1 --2- 172.21.15.29:0/4011719180 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c4164e7730 0x55c4164dd520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:40.568 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.563+0000 7f78b4a2e740 1 --2- 172.21.15.29:0/4011719180 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c4164e25c0 0x55c4164f3320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:40.568 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.563+0000 7f78b4a2e740 1 --2- 172.21.15.29:0/4011719180 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c4164e1bd0 0x55c4164e6da0 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:40.568 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.563+0000 7f78b4a2e740 1 -- 172.21.15.29:0/4011719180 >> 172.21.15.29:0/4011719180 conn(0x55c4164dd140 msgr2=0x55c4164dbaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:50:40.568 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.563+0000 7f78b4a2e740 1 -- 172.21.15.29:0/4011719180 shutdown_connections 2024-03-16T11:50:40.568 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.563+0000 7f78b4a2e740 1 -- 172.21.15.29:0/4011719180 wait complete. 2024-03-16T11:50:40.568 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.563+0000 7f78b4a2e740 1 -- start start 2024-03-16T11:50:40.568 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.563+0000 7f78b4a2e740 1 Processor -- start 2024-03-16T11:50:40.568 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.564+0000 7f78b4a2e740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c4164e1bd0 0x55c4164eebd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:40.568 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.564+0000 7f78b4a2e740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c4164e25c0 0x55c4164ef110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:40.569 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.564+0000 7f78b4a2e740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c4164e7730 0x55c4164f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:40.569 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.564+0000 7f78b4a2e740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55c4164f9b50 con 0x55c4164e25c0 2024-03-16T11:50:40.569 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.564+0000 7f78b4a2e740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x55c4164f9480 con 0x55c4164e7730 2024-03-16T11:50:40.569 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.564+0000 7f78b4a2e740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55c4164fa220 con 0x55c4164e1bd0 2024-03-16T11:50:40.569 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.564+0000 7f78b4a2e740 10 client.0 _ll_register_callbacks cb 0x7f7894005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T11:50:40.569 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.564+0000 7f78aaffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c4164e25c0 0x55c4164ef110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:40.569 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.564+0000 7f78aaffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c4164e25c0 0x55c4164ef110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39808/0 (socket says 172.21.15.29:39808) 2024-03-16T11:50:40.569 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.564+0000 7f78aaffd640 1 -- 172.21.15.29:0/1057916232 learned_addr learned my addr 172.21.15.29:0/1057916232 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:50:40.569 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.564+0000 7f78abfff640 1 --2- 172.21.15.29:0/1057916232 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c4164e7730 0x55c4164f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:40.569 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.564+0000 7f78ab7fe640 1 --2- 172.21.15.29:0/1057916232 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c4164e1bd0 0x55c4164eebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:40.569 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.564+0000 7f78aaffd640 1 -- 172.21.15.29:0/1057916232 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c4164e1bd0 msgr2=0x55c4164eebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:50:40.569 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.564+0000 7f78aaffd640 1 --2- 172.21.15.29:0/1057916232 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c4164e1bd0 0x55c4164eebd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:40.569 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.564+0000 7f78aaffd640 1 -- 172.21.15.29:0/1057916232 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c4164e7730 msgr2=0x55c4164f20e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:50:40.569 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.564+0000 7f78aaffd640 1 --2- 172.21.15.29:0/1057916232 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c4164e7730 0x55c4164f20e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:40.569 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.564+0000 7f78aaffd640 1 -- 172.21.15.29:0/1057916232 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c4164f26d0 con 0x55c4164e25c0 2024-03-16T11:50:40.570 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.564+0000 7f78abfff640 1 --2- 172.21.15.29:0/1057916232 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c4164e7730 0x55c4164f20e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:50:40.570 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.564+0000 7f78aaffd640 1 --2- 172.21.15.29:0/1057916232 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c4164e25c0 0x55c4164ef110 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f789400d010 tx=0x7f78940096c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58c4e930ec40dcd1 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:50:40.570 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.564+0000 7f788affd640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-16T11:50:40.570 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.564+0000 7f788affd640 1 -- 172.21.15.29:0/1057916232 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78940b3070 con 0x55c4164e25c0 2024-03-16T11:50:40.570 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.564+0000 7f788affd640 1 -- 172.21.15.29:0/1057916232 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8211+0+0 (secure 0 0 0) 0x7f78940860b0 con 0x55c4164e25c0 2024-03-16T11:50:40.570 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.564+0000 7f78b4a2e740 1 -- 172.21.15.29:0/1057916232 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f789c032c90 con 0x55c4164e25c0 2024-03-16T11:50:40.570 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.565+0000 7f788affd640 1 -- 172.21.15.29:0/1057916232 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f789409f890 con 0x55c4164e25c0 2024-03-16T11:50:40.570 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.565+0000 7f78b4a2e740 10 client.15282 fetch_fsmap learned FSMap version 21 2024-03-16T11:50:40.570 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.565+0000 7f78b4a2e740 1 -- 172.21.15.29:0/1057916232 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55c41655e700 con 0x55c4164e25c0 2024-03-16T11:50:40.570 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.565+0000 7f788affd640 1 -- 172.21.15.29:0/1057916232 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f789c032c90 con 0x55c4164e25c0 2024-03-16T11:50:40.570 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.565+0000 7f788affd640 1 -- 172.21.15.29:0/1057916232 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f7894097db0 con 0x55c4164e25c0 2024-03-16T11:50:40.570 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.565+0000 7f78b4a2e740 10 client.15282 fetch_fsmap finished waiting for FSMap version 21 2024-03-16T11:50:40.570 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.565+0000 7f78b4a2e740 10 client.15282 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-16T11:50:40.570 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.565+0000 7f78b4a2e740 20 client.15282 populate_metadata read hostname 'smithi029' 2024-03-16T11:50:40.570 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.565+0000 7f78b4a2e740 1 --2- 172.21.15.29:0/1057916232 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55c4165632d0 0x55c416561fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:40.570 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.565+0000 7f78b4a2e740 4 client.15282 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-16T11:50:40.571 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.565+0000 7f78b4a2e740 1 -- 172.21.15.29:0/1057916232 --> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] -- command(tid 0: {"prefix": "status"}) -- 0x55c41644d510 con 0x55c4165632d0 2024-03-16T11:50:40.571 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.565+0000 7f78ab7fe640 1 --2- 172.21.15.29:0/1057916232 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55c4165632d0 0x55c416561fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:40.571 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.565+0000 7f788affd640 10 client.15282 ms_handle_connect on v2:172.21.15.152:6838/1548731833 2024-03-16T11:50:40.571 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.565+0000 7f78ab7fe640 1 --2- 172.21.15.29:0/1057916232 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55c4165632d0 0x55c416561fe0 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d5902d90830188d2 server_cookie=f0d928443e647a7d in_seq=0 out_seq=0 2024-03-16T11:50:40.571 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.566+0000 7f788affd640 1 -- 172.21.15.29:0/1057916232 <== mds.0 v2:172.21.15.152:6838/1548731833 1 ==== command_reply(tid 0: 0 ) ==== 8+0+322 (crc 0 0 0) 0x55c41644d510 con 0x55c4165632d0 2024-03-16T11:50:40.571 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.566+0000 7f788affd640 10 client.15282 handle_command_reply: tid=0 2024-03-16T11:50:40.571 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.566+0000 7f78b4a2e740 1 client.15282 shutdown 2024-03-16T11:50:40.571 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.566+0000 7f78b4a2e740 1 -- 172.21.15.29:0/1057916232 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c4164e25c0 msgr2=0x55c4164ef110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:40.571 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.566+0000 7f78b4a2e740 1 --2- 172.21.15.29:0/1057916232 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c4164e25c0 0x55c4164ef110 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f789400d010 tx=0x7f78940096c0 comp rx=0 tx=0).stop 2024-03-16T11:50:40.571 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.566+0000 7f78b4a2e740 1 -- 172.21.15.29:0/1057916232 shutdown_connections 2024-03-16T11:50:40.571 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.566+0000 7f78b4a2e740 1 --2- 172.21.15.29:0/1057916232 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55c4165632d0 0x55c416561fe0 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:40.571 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.566+0000 7f78b4a2e740 1 --2- 172.21.15.29:0/1057916232 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c4164e7730 0x55c4164f20e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:40.571 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.566+0000 7f788affd640 0 client.15282 ms_handle_reset on v2:172.21.15.152:6838/1548731833 2024-03-16T11:50:40.571 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.566+0000 7f78b4a2e740 1 --2- 172.21.15.29:0/1057916232 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c4164e25c0 0x55c4164ef110 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:40.571 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.566+0000 7f78b4a2e740 1 --2- 172.21.15.29:0/1057916232 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c4164e1bd0 0x55c4164eebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:40.572 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.566+0000 7f78b4a2e740 1 -- 172.21.15.29:0/1057916232 >> 172.21.15.29:0/1057916232 conn(0x55c4164dd140 msgr2=0x55c4164dbaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:50:40.572 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.566+0000 7f78b4a2e740 1 -- 172.21.15.29:0/1057916232 shutdown_connections 2024-03-16T11:50:40.572 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.566+0000 7f78b4a2e740 1 -- 172.21.15.29:0/1057916232 wait complete. 2024-03-16T11:50:40.572 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.566+0000 7f78b4a2e740 20 client.15282 trim_cache size 0 max 16384 2024-03-16T11:50:40.572 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-03-16T11:50:40.572 INFO:teuthology.orchestra.run.smithi029.stdout: "cluster_fsid": "d1887f28-e389-11ee-95c9-87774f69a715", 2024-03-16T11:50:40.572 INFO:teuthology.orchestra.run.smithi029.stdout: "whoami": 0, 2024-03-16T11:50:40.572 INFO:teuthology.orchestra.run.smithi029.stdout: "id": 24503, 2024-03-16T11:50:40.572 INFO:teuthology.orchestra.run.smithi029.stdout: "want_state": "up:active", 2024-03-16T11:50:40.572 INFO:teuthology.orchestra.run.smithi029.stdout: "state": "up:active", 2024-03-16T11:50:40.572 INFO:teuthology.orchestra.run.smithi029.stdout: "fs_name": "cephfs", 2024-03-16T11:50:40.572 INFO:teuthology.orchestra.run.smithi029.stdout: "rank_uptime": 121.555634248, 2024-03-16T11:50:40.573 INFO:teuthology.orchestra.run.smithi029.stdout: "mdsmap_epoch": 21, 2024-03-16T11:50:40.573 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_epoch": 87, 2024-03-16T11:50:40.573 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_epoch_barrier": 85, 2024-03-16T11:50:40.573 INFO:teuthology.orchestra.run.smithi029.stdout: "uptime": 126.764795112 2024-03-16T11:50:40.573 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-03-16T11:50:40.574 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.566+0000 7f78b1e44640 1 -- 172.21.15.29:0/2805978406 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f78980a2800 msgr2=0x7f78980a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:40.574 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.566+0000 7f78b1e44640 1 --2- 172.21.15.29:0/2805978406 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f78980a2800 0x7f78980a4cc0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f78a4023000 tx=0x7f78a4024000 comp rx=0 tx=0).stop 2024-03-16T11:50:40.574 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.566+0000 7f78b1e44640 1 -- 172.21.15.29:0/2805978406 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f78ac1018d0 msgr2=0x7f78ac1096a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:40.574 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.566+0000 7f78b1e44640 1 --2- 172.21.15.29:0/2805978406 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f78ac1018d0 0x7f78ac1096a0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f789400ea60 tx=0x7f789400ef20 comp rx=0 tx=0).stop 2024-03-16T11:50:40.574 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.566+0000 7f78b1e44640 1 -- 172.21.15.29:0/2805978406 shutdown_connections 2024-03-16T11:50:40.574 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.566+0000 7f78b1e44640 1 --2- 172.21.15.29:0/2805978406 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f78980a2800 0x7f78980a4cc0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:40.574 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.566+0000 7f78b1e44640 1 --2- 172.21.15.29:0/2805978406 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f78ac102290 0x7f78ac109be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:40.574 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.567+0000 7f78b1e44640 1 --2- 172.21.15.29:0/2805978406 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f78ac1018d0 0x7f78ac1096a0 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:40.574 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.567+0000 7f78b1e44640 1 --2- 172.21.15.29:0/2805978406 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f78ac1006e0 0x7f78ac10b060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:40.574 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.567+0000 7f78b1e44640 1 -- 172.21.15.29:0/2805978406 >> 172.21.15.29:0/2805978406 conn(0x7f78ac0fbc70 msgr2=0x7f78ac104ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:50:40.574 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.567+0000 7f78b1e44640 1 -- 172.21.15.29:0/2805978406 shutdown_connections 2024-03-16T11:50:40.574 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:40.567+0000 7f78b1e44640 1 -- 172.21.15.29:0/2805978406 wait complete. 2024-03-16T11:50:40.576 INFO:teuthology.run_tasks:Running task kclient... 2024-03-16T11:50:40.598 INFO:tasks.kclient:Mounting kernel clients... 2024-03-16T11:50:40.598 INFO:tasks.kclient:config is {'client.0': {}} 2024-03-16T11:50:40.598 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-16T11:50:40.598 DEBUG:teuthology.orchestra.run.smithi029:> ip netns list 2024-03-16T11:50:40.613 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-16T11:50:40.613 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip link delete ceph-brx 2024-03-16T11:50:40.681 INFO:teuthology.orchestra.run.smithi029.stderr:Cannot find device "ceph-brx" 2024-03-16T11:50:40.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-16T11:50:40.683 INFO:tasks.kclient:client.0 config is {'mntopts': ['ms_mode=legacy', 'nowsync'], 'syntax': 'v1', 'mount_subvol_num': 1} 2024-03-16T11:50:40.683 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-16T11:50:40.683 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-16T11:50:40.683 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid d1887f28-e389-11ee-95c9-87774f69a715 -- ceph fs ls 2024-03-16T11:50:40.965 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/d1887f28-e389-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T11:50:41.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:50:40 smithi154 ceph-mon[36320]: from='client.? 172.21.15.29:0/4247670357' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T11:50:41.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:50:40 smithi154 ceph-mon[36320]: from='client.? 172.21.15.29:0/3957378813' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T11:50:41.086 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:50:40 smithi029 ceph-mon[31975]: from='client.? 172.21.15.29:0/4247670357' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T11:50:41.087 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:50:40 smithi029 ceph-mon[31975]: from='client.? 172.21.15.29:0/3957378813' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T11:50:41.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:50:40 smithi152 ceph-mon[36291]: from='client.? 172.21.15.29:0/4247670357' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T11:50:41.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:50:40 smithi152 ceph-mon[36291]: from='client.? 172.21.15.29:0/3957378813' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T11:50:42.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:50:41 smithi154 ceph-mon[36320]: pgmap v354: 97 pgs: 97 active+clean; 640 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-16T11:50:42.087 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:50:41 smithi029 ceph-mon[31975]: pgmap v354: 97 pgs: 97 active+clean; 640 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-16T11:50:42.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:50:41 smithi152 ceph-mon[36291]: pgmap v354: 97 pgs: 97 active+clean; 640 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-16T11:50:42.815 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:42.814+0000 7f57e3230700 1 -- 172.21.15.29:0/1204882184 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f57dc1075e0 msgr2=0x7f57dc10f1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:42.815 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:42.814+0000 7f57e3230700 1 --2- 172.21.15.29:0/1204882184 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f57dc1075e0 0x7f57dc10f1b0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f57d400b0a0 tx=0x7f57d400b3b0 comp rx=0 tx=0).stop 2024-03-16T11:50:42.816 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:42.814+0000 7f57e3230700 1 -- 172.21.15.29:0/1204882184 shutdown_connections 2024-03-16T11:50:42.816 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:42.814+0000 7f57e3230700 1 --2- 172.21.15.29:0/1204882184 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f57dc1075e0 0x7f57dc10f1b0 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:42.816 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:42.814+0000 7f57e3230700 1 --2- 172.21.15.29:0/1204882184 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f57dc1025b0 0x7f57dc102990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:42.816 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:42.814+0000 7f57e3230700 1 --2- 172.21.15.29:0/1204882184 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f57dc102f60 0x7f57dc106f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:42.816 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:42.814+0000 7f57e3230700 1 -- 172.21.15.29:0/1204882184 >> 172.21.15.29:0/1204882184 conn(0x7f57dc079360 msgr2=0x7f57dc079770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:50:42.816 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:42.814+0000 7f57e3230700 1 -- 172.21.15.29:0/1204882184 shutdown_connections 2024-03-16T11:50:42.816 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:42.814+0000 7f57e3230700 1 -- 172.21.15.29:0/1204882184 wait complete. 2024-03-16T11:50:42.816 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:42.814+0000 7f57e3230700 1 Processor -- start 2024-03-16T11:50:42.816 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:42.815+0000 7f57e3230700 1 -- start start 2024-03-16T11:50:42.816 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:42.815+0000 7f57e3230700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f57dc1025b0 0x7f57dc19e680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:42.816 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:42.815+0000 7f57e3230700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f57dc102f60 0x7f57dc19ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:42.816 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:42.815+0000 7f57e3230700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f57dc1075e0 0x7f57dc198800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:42.816 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:42.815+0000 7f57e3230700 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f57dc1125c0 con 0x7f57dc1025b0 2024-03-16T11:50:42.817 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:42.815+0000 7f57e3230700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f57dc112440 con 0x7f57dc102f60 2024-03-16T11:50:42.817 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:42.815+0000 7f57e3230700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f57dc112740 con 0x7f57dc1075e0 2024-03-16T11:50:42.817 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:42.815+0000 7f57e0fcc700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f57dc1025b0 0x7f57dc19e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:42.817 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:42.815+0000 7f57e0fcc700 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f57dc1025b0 0x7f57dc19e680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39824/0 (socket says 172.21.15.29:39824) 2024-03-16T11:50:42.817 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:42.815+0000 7f57e0fcc700 1 -- 172.21.15.29:0/675906267 learned_addr learned my addr 172.21.15.29:0/675906267 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:50:42.817 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:42.815+0000 7f57e17cd700 1 --2- 172.21.15.29:0/675906267 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f57dc1075e0 0x7f57dc198800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:42.817 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:42.815+0000 7f57dbfff700 1 --2- 172.21.15.29:0/675906267 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f57dc102f60 0x7f57dc19ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:42.817 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:42.815+0000 7f57e0fcc700 1 -- 172.21.15.29:0/675906267 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f57dc1075e0 msgr2=0x7f57dc198800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:42.817 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:42.815+0000 7f57e0fcc700 1 --2- 172.21.15.29:0/675906267 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f57dc1075e0 0x7f57dc198800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:42.817 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:42.815+0000 7f57e0fcc700 1 -- 172.21.15.29:0/675906267 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f57dc102f60 msgr2=0x7f57dc19ebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:42.817 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:42.815+0000 7f57e0fcc700 1 --2- 172.21.15.29:0/675906267 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f57dc102f60 0x7f57dc19ebc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:42.817 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:42.815+0000 7f57e0fcc700 1 -- 172.21.15.29:0/675906267 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57dc198fe0 con 0x7f57dc1025b0 2024-03-16T11:50:42.818 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:42.815+0000 7f57e0fcc700 1 --2- 172.21.15.29:0/675906267 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f57dc1025b0 0x7f57dc19e680 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f57cc00e9a0 tx=0x7f57cc00ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:50:42.818 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:42.815+0000 7f57d9ffb700 1 -- 172.21.15.29:0/675906267 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57cc00ca10 con 0x7f57dc1025b0 2024-03-16T11:50:42.818 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:42.816+0000 7f57d9ffb700 1 -- 172.21.15.29:0/675906267 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f57cc00cbb0 con 0x7f57dc1025b0 2024-03-16T11:50:42.818 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:42.816+0000 7f57e3230700 1 -- 172.21.15.29:0/675906267 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f57dc1992d0 con 0x7f57dc1025b0 2024-03-16T11:50:42.818 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:42.816+0000 7f57d9ffb700 1 -- 172.21.15.29:0/675906267 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57cc018680 con 0x7f57dc1025b0 2024-03-16T11:50:42.818 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:42.816+0000 7f57e3230700 1 -- 172.21.15.29:0/675906267 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f57dc1a54e0 con 0x7f57dc1025b0 2024-03-16T11:50:42.819 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:42.817+0000 7f57e3230700 1 -- 172.21.15.29:0/675906267 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f57dc1039e0 con 0x7f57dc1025b0 2024-03-16T11:50:42.819 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:42.818+0000 7f57d9ffb700 1 -- 172.21.15.29:0/675906267 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f57cc010430 con 0x7f57dc1025b0 2024-03-16T11:50:42.819 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:42.818+0000 7f57d9ffb700 1 --2- 172.21.15.29:0/675906267 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f57c80a2910 0x7f57c80a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:42.819 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:42.819+0000 7f57dbfff700 1 --2- 172.21.15.29:0/675906267 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f57c80a2910 0x7f57c80a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:42.819 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:42.819+0000 7f57d9ffb700 1 -- 172.21.15.29:0/675906267 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8211+0+0 (secure 0 0 0) 0x7f57cc0cec70 con 0x7f57dc1025b0 2024-03-16T11:50:42.819 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:42.819+0000 7f57dbfff700 1 --2- 172.21.15.29:0/675906267 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f57c80a2910 0x7f57c80a4dd0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f57dc108960 tx=0x7f57c4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:50:42.822 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:42.823+0000 7f57d9ffb700 1 -- 172.21.15.29:0/675906267 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f57cc09bcc0 con 0x7f57dc1025b0 2024-03-16T11:50:43.073 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:43.074+0000 7f57e3230700 1 -- 172.21.15.29:0/675906267 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f57dc199d30 con 0x7f57dc1025b0 2024-03-16T11:50:43.073 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:43.074+0000 7f57d9ffb700 1 -- 172.21.15.29:0/675906267 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v21) ==== 53+0+73 (secure 0 0 0) 0x7f57cc09b5c0 con 0x7f57dc1025b0 2024-03-16T11:50:43.073 INFO:teuthology.orchestra.run.smithi029.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-16T11:50:43.075 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:43.076+0000 7f57e3230700 1 -- 172.21.15.29:0/675906267 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f57c80a2910 msgr2=0x7f57c80a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:43.075 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:43.076+0000 7f57e3230700 1 --2- 172.21.15.29:0/675906267 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f57c80a2910 0x7f57c80a4dd0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f57dc108960 tx=0x7f57c4009260 comp rx=0 tx=0).stop 2024-03-16T11:50:43.076 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:43.076+0000 7f57e3230700 1 -- 172.21.15.29:0/675906267 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f57dc1025b0 msgr2=0x7f57dc19e680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:43.076 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:43.076+0000 7f57e3230700 1 --2- 172.21.15.29:0/675906267 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f57dc1025b0 0x7f57dc19e680 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f57cc00e9a0 tx=0x7f57cc00ecb0 comp rx=0 tx=0).stop 2024-03-16T11:50:43.076 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:43.076+0000 7f57e3230700 1 -- 172.21.15.29:0/675906267 shutdown_connections 2024-03-16T11:50:43.076 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:43.076+0000 7f57e3230700 1 --2- 172.21.15.29:0/675906267 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f57c80a2910 0x7f57c80a4dd0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:43.076 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:43.076+0000 7f57e3230700 1 --2- 172.21.15.29:0/675906267 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f57dc1075e0 0x7f57dc198800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:43.076 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:43.076+0000 7f57e3230700 1 --2- 172.21.15.29:0/675906267 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f57dc1025b0 0x7f57dc19e680 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:43.076 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:43.076+0000 7f57e3230700 1 --2- 172.21.15.29:0/675906267 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f57dc102f60 0x7f57dc19ebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:43.076 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:43.076+0000 7f57e3230700 1 -- 172.21.15.29:0/675906267 >> 172.21.15.29:0/675906267 conn(0x7f57dc079360 msgr2=0x7f57dc100360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:50:43.076 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:43.076+0000 7f57e3230700 1 -- 172.21.15.29:0/675906267 shutdown_connections 2024-03-16T11:50:43.076 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:43.076+0000 7f57e3230700 1 -- 172.21.15.29:0/675906267 wait complete. 2024-03-16T11:50:43.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:50:43 smithi029 ceph-mon[31975]: from='client.? 172.21.15.29:0/675906267' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-16T11:50:43.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:50:43 smithi152 ceph-mon[36291]: from='client.? 172.21.15.29:0/675906267' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-16T11:50:43.545 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:50:43 smithi154 ceph-mon[36320]: from='client.? 172.21.15.29:0/675906267' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-16T11:50:43.809 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-16T11:50:43.831 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-16T11:50:43.832 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi029.front.sepia.ceph.com 2024-03-16T11:50:43.832 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-16T11:50:43.832 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-16T11:50:43.832 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-16T11:50:43.832 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-16T11:50:43.832 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-16T11:50:43.832 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-16T11:50:43.832 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-16T11:50:43.832 DEBUG:teuthology.orchestra.run.smithi029:> ip addr 2024-03-16T11:50:43.849 INFO:teuthology.orchestra.run.smithi029.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-16T11:50:43.849 INFO:teuthology.orchestra.run.smithi029.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-16T11:50:43.849 INFO:teuthology.orchestra.run.smithi029.stdout: inet 127.0.0.1/8 scope host lo 2024-03-16T11:50:43.849 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2024-03-16T11:50:43.849 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 ::1/128 scope host 2024-03-16T11:50:43.849 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2024-03-16T11:50:43.849 INFO:teuthology.orchestra.run.smithi029.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-16T11:50:43.849 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:6b:fd:a0 brd ff:ff:ff:ff:ff:ff 2024-03-16T11:50:43.849 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp7s0f0 2024-03-16T11:50:43.849 INFO:teuthology.orchestra.run.smithi029.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-16T11:50:43.849 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:6b:fd:a1 brd ff:ff:ff:ff:ff:ff 2024-03-16T11:50:43.849 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp7s0f1 2024-03-16T11:50:43.849 INFO:teuthology.orchestra.run.smithi029.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-16T11:50:43.850 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:bd:12:5a brd ff:ff:ff:ff:ff:ff 2024-03-16T11:50:43.850 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp1s0f0 2024-03-16T11:50:43.850 INFO:teuthology.orchestra.run.smithi029.stdout: inet 172.21.15.29/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-03-16T11:50:43.850 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft 42007sec preferred_lft 42007sec 2024-03-16T11:50:43.850 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 fe80::ec4:7aff:febd:125a/64 scope link noprefixroute 2024-03-16T11:50:43.850 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2024-03-16T11:50:43.850 INFO:teuthology.orchestra.run.smithi029.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-16T11:50:43.850 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:bd:12:5b brd ff:ff:ff:ff:ff:ff 2024-03-16T11:50:43.850 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp1s0f1 2024-03-16T11:50:43.850 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-03-16T11:50:43.851 DEBUG:teuthology.orchestra.run.smithi029:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-16T11:50:43.851 DEBUG:teuthology.orchestra.run.smithi029:> set -e 2024-03-16T11:50:43.851 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip link add name ceph-brx type bridge 2024-03-16T11:50:43.851 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip addr flush dev ceph-brx 2024-03-16T11:50:43.851 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip link set ceph-brx up 2024-03-16T11:50:43.851 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-03-16T11:50:43.851 DEBUG:teuthology.orchestra.run.smithi029:> ') 2024-03-16T11:50:43.937 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:43 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-16T11:50:44.081 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:44 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-16T11:50:44.087 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-16T11:50:44.088 DEBUG:teuthology.orchestra.run.smithi029:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-16T11:50:44.115 INFO:teuthology.orchestra.run.smithi029.stdout:1 2024-03-16T11:50:44.116 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-16T11:50:44.116 DEBUG:teuthology.orchestra.run.smithi029:> route 2024-03-16T11:50:44.185 INFO:teuthology.orchestra.run.smithi029.stdout:Kernel IP routing table 2024-03-16T11:50:44.185 INFO:teuthology.orchestra.run.smithi029.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-16T11:50:44.185 INFO:teuthology.orchestra.run.smithi029.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-03-16T11:50:44.185 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-03-16T11:50:44.185 INFO:teuthology.orchestra.run.smithi029.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-16T11:50:44.185 DEBUG:teuthology.orchestra.run.smithi029:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-16T11:50:44.186 DEBUG:teuthology.orchestra.run.smithi029:> set -e 2024-03-16T11:50:44.186 DEBUG:teuthology.orchestra.run.smithi029:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-03-16T11:50:44.186 DEBUG:teuthology.orchestra.run.smithi029:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-03-16T11:50:44.186 DEBUG:teuthology.orchestra.run.smithi029:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-03-16T11:50:44.186 DEBUG:teuthology.orchestra.run.smithi029:> ') 2024-03-16T11:50:44.272 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:44 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-16T11:50:44.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:50:44 smithi029 ceph-mon[31975]: pgmap v355: 97 pgs: 97 active+clean; 640 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-16T11:50:44.515 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:44 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-16T11:50:44.521 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-16T11:50:44.522 DEBUG:teuthology.orchestra.run.smithi029:> ip netns list 2024-03-16T11:50:44.536 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-16T11:50:44.536 DEBUG:teuthology.orchestra.run.smithi029:> ip netns list-id 2024-03-16T11:50:44.544 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:50:44 smithi154 ceph-mon[36320]: pgmap v355: 97 pgs: 97 active+clean; 640 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-16T11:50:44.590 DEBUG:teuthology.orchestra.run.smithi029:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-16T11:50:44.590 DEBUG:teuthology.orchestra.run.smithi029:> set -e 2024-03-16T11:50:44.590 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-16T11:50:44.591 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-03-16T11:50:44.591 DEBUG:teuthology.orchestra.run.smithi029:> ') 2024-03-16T11:50:44.627 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:50:44 smithi152 ceph-mon[36291]: pgmap v355: 97 pgs: 97 active+clean; 640 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-16T11:50:44.678 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:44 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-16T11:50:44.707 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:44 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-16T11:50:44.713 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-03-16T11:50:44.713 DEBUG:teuthology.orchestra.run.smithi029:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-16T11:50:44.713 DEBUG:teuthology.orchestra.run.smithi029:> set -e 2024-03-16T11:50:44.713 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-03-16T11:50:44.713 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-03-16T11:50:44.713 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-03-16T11:50:44.714 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-03-16T11:50:44.714 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-03-16T11:50:44.714 DEBUG:teuthology.orchestra.run.smithi029:> ') 2024-03-16T11:50:44.761 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:44 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-16T11:50:44.848 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:44 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-16T11:50:44.854 DEBUG:teuthology.orchestra.run.smithi029:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-16T11:50:44.854 DEBUG:teuthology.orchestra.run.smithi029:> set -e 2024-03-16T11:50:44.854 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip link set brx.0 up 2024-03-16T11:50:44.854 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip link set dev brx.0 master ceph-brx 2024-03-16T11:50:44.854 DEBUG:teuthology.orchestra.run.smithi029:> ') 2024-03-16T11:50:44.902 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:44 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-16T11:50:44.929 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:44 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-16T11:50:44.935 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-16T11:50:44.935 DEBUG:teuthology.orchestra.run.smithi029:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-16T11:50:44.950 INFO:teuthology.orchestra.run.smithi029.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-16T11:50:44.950 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-16T11:50:44.950 DEBUG:teuthology.orchestra.run.smithi029:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-16T11:50:45.004 WARNING:tasks.cephfs.mount:You cannot specify both: cephfs_mntpt and mount_subvol_num 2024-03-16T11:50:45.004 INFO:tasks.cephfs.mount:Mounting subvol 1 for now 2024-03-16T11:50:45.004 INFO:tasks.cephfs.kernel_mount:mounting using device: :/volumes/qa/sv_1/376ddc2b-fccb-40c2-9473-a26f8d0685a7 2024-03-16T11:50:45.004 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-16T11:50:45.004 DEBUG:teuthology.orchestra.run.smithi029:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/volumes/qa/sv_1/376ddc2b-fccb-40c2-9473-a26f8d0685a7 /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs,ms_mode=legacy,nowsync 2024-03-16T11:50:45.087 INFO:teuthology.orchestra.run.smithi029.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs,ms_mode=legacy,nowsync 2024-03-16T11:50:45.088 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,ms_mode=legacy,nowsync". 2024-03-16T11:50:45.088 INFO:teuthology.orchestra.run.smithi029.stdout:invalid new device string format 2024-03-16T11:50:45.208 INFO:teuthology.orchestra.run.smithi029.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs,ms_mode=legacy,nowsync 2024-03-16T11:50:45.209 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,ms_mode=legacy,nowsync". 2024-03-16T11:50:45.209 INFO:teuthology.orchestra.run.smithi029.stdout:invalid new device string format 2024-03-16T11:50:45.209 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: resolved to: "172.21.15.29:6789,172.21.15.152:6789,172.21.15.154:6789" 2024-03-16T11:50:45.209 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.29:6789,172.21.15.152:6789,172.21.15.154:6789:/volumes/qa/sv_1/376ddc2b-fccb-40c2-9473-a26f8d0685a7 2024-03-16T11:50:45.209 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,ms_mode=legacy,nowsync,key=0,fsid=d1887f28-e389-11ee-95c9-87774f69a715" will pass to kernel 2024-03-16T11:50:45.209 INFO:teuthology.orchestra.run.smithi029.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-03-16T11:50:45.209 INFO:teuthology.orchestra.run.smithi029.stdout: You just mounted a file system that supports labels which does not 2024-03-16T11:50:45.209 INFO:teuthology.orchestra.run.smithi029.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-03-16T11:50:45.210 INFO:teuthology.orchestra.run.smithi029.stdout: applications will generate AVC messages and not be allowed access to 2024-03-16T11:50:45.210 INFO:teuthology.orchestra.run.smithi029.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-03-16T11:50:45.211 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-03-16T11:50:45.212 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-16T11:50:45.212 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-16T11:50:45.282 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-16T11:50:45.371 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:45 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-03-16T11:50:45.373 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:45 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-16T11:50:45.379 INFO:teuthology.orchestra.run.smithi029.stdout:client15291 2024-03-16T11:50:45.379 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-03-16T11:50:45.379 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-16T11:50:45.395 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-16T11:50:45.483 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:45 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-03-16T11:50:45.485 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:45 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-16T11:50:45.490 INFO:teuthology.orchestra.run.smithi029.stdout:client15291 2024-03-16T11:50:45.490 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-16T11:50:45.538 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:45 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-03-16T11:50:45.539 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:45 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-16T11:50:45.545 INFO:teuthology.orchestra.run.smithi029.stdout:client15291 2024-03-16T11:50:45.545 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=300 -- bash -c 'sudo dd if=/sys/kernel/debug/ceph/d1887f28-e389-11ee-95c9-87774f69a715.client15291/status') 2024-03-16T11:50:45.593 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:45 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-16T11:50:45.606 INFO:teuthology.orchestra.run.smithi029.stdout:instance: client.15291 (3)192.168.0.1:0/3782913819 2024-03-16T11:50:45.606 INFO:teuthology.orchestra.run.smithi029.stdout:blocklisted: false 2024-03-16T11:50:45.606 INFO:teuthology.orchestra.run.smithi029.stderr:0+1 records in 2024-03-16T11:50:45.606 INFO:teuthology.orchestra.run.smithi029.stderr:0+1 records out 2024-03-16T11:50:45.606 INFO:teuthology.orchestra.run.smithi029.stderr:70 bytes copied, 2.0825e-05 s, 3.4 MB/s 2024-03-16T11:50:45.607 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:45 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-16T11:50:45.613 INFO:teuthology.run_tasks:Running task exec... 2024-03-16T11:50:45.623 INFO:teuthology.task.exec:Executing custom commands... 2024-03-16T11:50:45.623 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi029.front.sepia.ceph.com 2024-03-16T11:50:45.624 DEBUG:teuthology.orchestra.run.smithi029:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs set cephfs balance_automate false' 2024-03-16T11:50:45.728 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:45.728+0000 7f718119d640 1 -- 172.21.15.29:0/4190647072 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f717c102840 msgr2=0x7f717c102cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:45.729 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:45.728+0000 7f718119d640 1 --2- 172.21.15.29:0/4190647072 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f717c102840 0x7f717c102cc0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f716c009900 tx=0x7f716c02bc30 comp rx=0 tx=0).stop 2024-03-16T11:50:45.729 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:45.728+0000 7f718119d640 1 -- 172.21.15.29:0/4190647072 shutdown_connections 2024-03-16T11:50:45.729 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:45.728+0000 7f718119d640 1 --2- 172.21.15.29:0/4190647072 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f717c103200 0x7f717c109a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:45.729 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:45.728+0000 7f718119d640 1 --2- 172.21.15.29:0/4190647072 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f717c102840 0x7f717c102cc0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:45.729 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:45.728+0000 7f718119d640 1 --2- 172.21.15.29:0/4190647072 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f717c101650 0x7f717c101a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:45.729 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:45.728+0000 7f718119d640 1 -- 172.21.15.29:0/4190647072 >> 172.21.15.29:0/4190647072 conn(0x7f717c0fcbc0 msgr2=0x7f717c0ff000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:50:45.729 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:45.728+0000 7f718119d640 1 -- 172.21.15.29:0/4190647072 shutdown_connections 2024-03-16T11:50:45.729 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:45.728+0000 7f718119d640 1 -- 172.21.15.29:0/4190647072 wait complete. 2024-03-16T11:50:45.729 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:45.729+0000 7f718119d640 1 Processor -- start 2024-03-16T11:50:45.729 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:45.729+0000 7f718119d640 1 -- start start 2024-03-16T11:50:45.731 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:45.729+0000 7f718119d640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f717c101650 0x7f717c197980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:45.731 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:45.729+0000 7f718119d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f717c102840 0x7f717c197ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:45.732 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:45.729+0000 7f718119d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f717c103200 0x7f717c19ef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:45.732 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:45.729+0000 7f718119d640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f717c10d2e0 con 0x7f717c102840 2024-03-16T11:50:45.732 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:45.729+0000 7f718119d640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f717c10d160 con 0x7f717c101650 2024-03-16T11:50:45.732 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:45.729+0000 7f718119d640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f717c10d460 con 0x7f717c103200 2024-03-16T11:50:45.732 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:45.729+0000 7f717b7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f717c102840 0x7f717c197ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:45.732 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:45.729+0000 7f717b7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f717c102840 0x7f717c197ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39844/0 (socket says 172.21.15.29:39844) 2024-03-16T11:50:45.732 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:45.729+0000 7f717b7fe640 1 -- 172.21.15.29:0/3768823820 learned_addr learned my addr 172.21.15.29:0/3768823820 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:50:45.732 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:45.729+0000 7f718099c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f717c103200 0x7f717c19ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:45.732 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:45.729+0000 7f717bfff640 1 --2- 172.21.15.29:0/3768823820 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f717c101650 0x7f717c197980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:45.732 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:45.729+0000 7f717b7fe640 1 -- 172.21.15.29:0/3768823820 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f717c103200 msgr2=0x7f717c19ef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:45.732 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:45.729+0000 7f717b7fe640 1 --2- 172.21.15.29:0/3768823820 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f717c103200 0x7f717c19ef30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:45.732 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:45.729+0000 7f717b7fe640 1 -- 172.21.15.29:0/3768823820 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f717c101650 msgr2=0x7f717c197980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:45.732 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:45.729+0000 7f717b7fe640 1 --2- 172.21.15.29:0/3768823820 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f717c101650 0x7f717c197980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:45.732 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:45.729+0000 7f717b7fe640 1 -- 172.21.15.29:0/3768823820 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f717c19f5a0 con 0x7f717c102840 2024-03-16T11:50:45.733 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:45.729+0000 7f717bfff640 1 --2- 172.21.15.29:0/3768823820 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f717c101650 0x7f717c197980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:50:45.733 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:45.730+0000 7f717b7fe640 1 --2- 172.21.15.29:0/3768823820 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f717c102840 0x7f717c197ec0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f716c009230 tx=0x7f716c02c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:50:45.733 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:45.730+0000 7f71797fa640 1 -- 172.21.15.29:0/3768823820 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f716c041070 con 0x7f717c102840 2024-03-16T11:50:45.733 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:45.730+0000 7f71797fa640 1 -- 172.21.15.29:0/3768823820 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f716c004450 con 0x7f717c102840 2024-03-16T11:50:45.733 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:45.730+0000 7f718119d640 1 -- 172.21.15.29:0/3768823820 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f717c19f830 con 0x7f717c102840 2024-03-16T11:50:45.733 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:45.730+0000 7f71797fa640 1 -- 172.21.15.29:0/3768823820 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f716c003e20 con 0x7f717c102840 2024-03-16T11:50:45.733 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:45.730+0000 7f718119d640 1 -- 172.21.15.29:0/3768823820 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f717c19fd70 con 0x7f717c102840 2024-03-16T11:50:45.733 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:45.731+0000 7f718119d640 1 -- 172.21.15.29:0/3768823820 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f717c108970 con 0x7f717c102840 2024-03-16T11:50:45.734 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:45.732+0000 7f71797fa640 1 -- 172.21.15.29:0/3768823820 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f716c044070 con 0x7f717c102840 2024-03-16T11:50:45.734 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:45.733+0000 7f71797fa640 1 --2- 172.21.15.29:0/3768823820 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f71680a2870 0x7f71680a4d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:45.734 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:45.733+0000 7f717bfff640 1 --2- 172.21.15.29:0/3768823820 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f71680a2870 0x7f71680a4d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:45.734 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:45.733+0000 7f71797fa640 1 -- 172.21.15.29:0/3768823820 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8211+0+0 (secure 0 0 0) 0x7f716c0f0560 con 0x7f717c102840 2024-03-16T11:50:45.734 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:45.733+0000 7f717bfff640 1 --2- 172.21.15.29:0/3768823820 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f71680a2870 0x7f71680a4d30 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f717c1026a0 tx=0x7f7164005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:50:45.737 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:45.737+0000 7f71797fa640 1 -- 172.21.15.29:0/3768823820 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f716c0bcfc0 con 0x7f717c102840 2024-03-16T11:50:45.960 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:45.960+0000 7f718119d640 1 -- 172.21.15.29:0/3768823820 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"} v 0) -- 0x7f717c0623e0 con 0x7f717c102840 2024-03-16T11:50:46.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:50:45 smithi154 ceph-mon[36320]: pgmap v356: 97 pgs: 97 active+clean; 640 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-16T11:50:46.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:50:45 smithi029 ceph-mon[31975]: pgmap v356: 97 pgs: 97 active+clean; 640 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-16T11:50:46.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:50:45 smithi152 ceph-mon[36291]: pgmap v356: 97 pgs: 97 active+clean; 640 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-16T11:50:46.718 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:46.718+0000 7f71797fa640 1 -- 172.21.15.29:0/3768823820 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]=0 v22) ==== 118+0+0 (secure 0 0 0) 0x7f716c0c1060 con 0x7f717c102840 2024-03-16T11:50:46.722 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:46.722+0000 7f718119d640 1 -- 172.21.15.29:0/3768823820 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f71680a2870 msgr2=0x7f71680a4d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:46.722 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:46.722+0000 7f718119d640 1 --2- 172.21.15.29:0/3768823820 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f71680a2870 0x7f71680a4d30 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f717c1026a0 tx=0x7f7164005f90 comp rx=0 tx=0).stop 2024-03-16T11:50:46.723 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:46.722+0000 7f718119d640 1 -- 172.21.15.29:0/3768823820 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f717c102840 msgr2=0x7f717c197ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:46.723 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:46.722+0000 7f718119d640 1 --2- 172.21.15.29:0/3768823820 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f717c102840 0x7f717c197ec0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f716c009230 tx=0x7f716c02c650 comp rx=0 tx=0).stop 2024-03-16T11:50:46.723 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:46.722+0000 7f718119d640 1 -- 172.21.15.29:0/3768823820 shutdown_connections 2024-03-16T11:50:46.723 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:46.722+0000 7f718119d640 1 --2- 172.21.15.29:0/3768823820 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f71680a2870 0x7f71680a4d30 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:46.723 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:46.722+0000 7f718119d640 1 --2- 172.21.15.29:0/3768823820 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f717c103200 0x7f717c19ef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:46.723 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:46.722+0000 7f718119d640 1 --2- 172.21.15.29:0/3768823820 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f717c102840 0x7f717c197ec0 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:46.723 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:46.722+0000 7f718119d640 1 --2- 172.21.15.29:0/3768823820 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f717c101650 0x7f717c197980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:46.723 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:46.722+0000 7f718119d640 1 -- 172.21.15.29:0/3768823820 >> 172.21.15.29:0/3768823820 conn(0x7f717c0fcbc0 msgr2=0x7f717c105b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:50:46.723 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:46.722+0000 7f718119d640 1 -- 172.21.15.29:0/3768823820 shutdown_connections 2024-03-16T11:50:46.723 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:46.722+0000 7f718119d640 1 -- 172.21.15.29:0/3768823820 wait complete. 2024-03-16T11:50:46.736 DEBUG:teuthology.orchestra.run.smithi029:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs subvolumegroup pin cephfs qa random 0.10' 2024-03-16T11:50:46.848 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:46.848+0000 7fdb4f190640 1 -- 172.21.15.29:0/3672442082 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdb48104bf0 msgr2=0x7fdb48106fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:46.849 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:46.848+0000 7fdb4f190640 1 --2- 172.21.15.29:0/3672442082 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdb48104bf0 0x7fdb48106fe0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fdb34009900 tx=0x7fdb3402bc30 comp rx=0 tx=0).stop 2024-03-16T11:50:46.849 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:46.848+0000 7fdb4f190640 1 -- 172.21.15.29:0/3672442082 shutdown_connections 2024-03-16T11:50:46.849 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:46.848+0000 7fdb4f190640 1 --2- 172.21.15.29:0/3672442082 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fdb48107520 0x7fdb48109930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:46.849 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:46.848+0000 7fdb4f190640 1 --2- 172.21.15.29:0/3672442082 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdb48104bf0 0x7fdb48106fe0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:46.849 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:46.848+0000 7fdb4f190640 1 --2- 172.21.15.29:0/3672442082 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdb481022c0 0x7fdb481046b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:46.849 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:46.848+0000 7fdb4f190640 1 -- 172.21.15.29:0/3672442082 >> 172.21.15.29:0/3672442082 conn(0x7fdb480fbc70 msgr2=0x7fdb480fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:50:46.849 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:46.848+0000 7fdb4f190640 1 -- 172.21.15.29:0/3672442082 shutdown_connections 2024-03-16T11:50:46.849 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:46.848+0000 7fdb4f190640 1 -- 172.21.15.29:0/3672442082 wait complete. 2024-03-16T11:50:46.849 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:46.848+0000 7fdb4f190640 1 Processor -- start 2024-03-16T11:50:46.849 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:46.848+0000 7fdb4f190640 1 -- start start 2024-03-16T11:50:46.849 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:46.848+0000 7fdb4f190640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdb481022c0 0x7fdb48197ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:46.849 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:46.848+0000 7fdb4f190640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdb48104bf0 0x7fdb48198010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:46.849 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:46.848+0000 7fdb4f190640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fdb48107520 0x7fdb4819f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:46.851 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:46.848+0000 7fdb4f190640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fdb4810d480 con 0x7fdb481022c0 2024-03-16T11:50:46.851 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:46.848+0000 7fdb4f190640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fdb4810d300 con 0x7fdb48107520 2024-03-16T11:50:46.851 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:46.848+0000 7fdb4f190640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fdb4810d600 con 0x7fdb48104bf0 2024-03-16T11:50:46.851 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:46.849+0000 7fdb4cf2c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdb481022c0 0x7fdb48197ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:46.852 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:46.849+0000 7fdb4cf2c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdb481022c0 0x7fdb48197ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39864/0 (socket says 172.21.15.29:39864) 2024-03-16T11:50:46.852 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:46.849+0000 7fdb4cf2c640 1 -- 172.21.15.29:0/120592845 learned_addr learned my addr 172.21.15.29:0/120592845 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:50:46.852 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:46.849+0000 7fdb4d72d640 1 --2- 172.21.15.29:0/120592845 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fdb48107520 0x7fdb4819f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:46.852 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:46.849+0000 7fdb47fff640 1 --2- 172.21.15.29:0/120592845 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdb48104bf0 0x7fdb48198010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:46.852 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:46.849+0000 7fdb4cf2c640 1 -- 172.21.15.29:0/120592845 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdb48104bf0 msgr2=0x7fdb48198010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:46.852 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:46.849+0000 7fdb4cf2c640 1 --2- 172.21.15.29:0/120592845 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdb48104bf0 0x7fdb48198010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:46.852 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:46.849+0000 7fdb4cf2c640 1 -- 172.21.15.29:0/120592845 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fdb48107520 msgr2=0x7fdb4819f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:46.852 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:46.849+0000 7fdb4cf2c640 1 --2- 172.21.15.29:0/120592845 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fdb48107520 0x7fdb4819f080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:46.852 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:46.849+0000 7fdb4cf2c640 1 -- 172.21.15.29:0/120592845 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb4819f6f0 con 0x7fdb481022c0 2024-03-16T11:50:46.852 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:46.849+0000 7fdb4cf2c640 1 --2- 172.21.15.29:0/120592845 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdb481022c0 0x7fdb48197ad0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fdb3c00e9f0 tx=0x7fdb3c00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:50:46.852 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:46.849+0000 7fdb45ffb640 1 -- 172.21.15.29:0/120592845 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb3c00cd80 con 0x7fdb481022c0 2024-03-16T11:50:46.852 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:46.849+0000 7fdb45ffb640 1 -- 172.21.15.29:0/120592845 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdb3c005500 con 0x7fdb481022c0 2024-03-16T11:50:46.852 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:46.849+0000 7fdb4f190640 1 -- 172.21.15.29:0/120592845 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdb4819f9e0 con 0x7fdb481022c0 2024-03-16T11:50:46.852 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:46.849+0000 7fdb45ffb640 1 -- 172.21.15.29:0/120592845 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb3c010690 con 0x7fdb481022c0 2024-03-16T11:50:46.853 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:46.849+0000 7fdb4f190640 1 -- 172.21.15.29:0/120592845 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdb4819feb0 con 0x7fdb481022c0 2024-03-16T11:50:46.853 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:46.850+0000 7fdb4f190640 1 -- 172.21.15.29:0/120592845 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdb480fd760 con 0x7fdb481022c0 2024-03-16T11:50:46.853 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:46.851+0000 7fdb45ffb640 1 -- 172.21.15.29:0/120592845 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fdb3c0046c0 con 0x7fdb481022c0 2024-03-16T11:50:46.853 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:46.852+0000 7fdb45ffb640 1 --2- 172.21.15.29:0/120592845 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fdb380a28a0 0x7fdb380a4d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:46.853 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:46.852+0000 7fdb47fff640 1 --2- 172.21.15.29:0/120592845 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fdb380a28a0 0x7fdb380a4d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:46.854 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:46.852+0000 7fdb45ffb640 1 -- 172.21.15.29:0/120592845 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8211+0+0 (secure 0 0 0) 0x7fdb3c0cee90 con 0x7fdb481022c0 2024-03-16T11:50:46.854 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:46.852+0000 7fdb47fff640 1 --2- 172.21.15.29:0/120592845 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fdb380a28a0 0x7fdb380a4d60 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fdb34000c00 tx=0x7fdb34005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:50:46.856 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:46.856+0000 7fdb45ffb640 1 -- 172.21.15.29:0/120592845 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fdb3c09bee0 con 0x7fdb481022c0 2024-03-16T11:50:47.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:50:46 smithi154 ceph-mon[36320]: from='client.? 172.21.15.29:0/3768823820' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-03-16T11:50:47.066 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:47.066+0000 7fdb4f190640 1 -- 172.21.15.29:0/120592845 --> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}) -- 0x7fdb48061cc0 con 0x7fdb380a28a0 2024-03-16T11:50:47.072 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:50:46 smithi029 ceph-mon[31975]: from='client.? 172.21.15.29:0/3768823820' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-03-16T11:50:47.082 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:47.082+0000 7fdb45ffb640 1 -- 172.21.15.29:0/120592845 <== mgr.14152 v2:172.21.15.29:6800/4192064640 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+3 (secure 0 0 0) 0x7fdb48061cc0 con 0x7fdb380a28a0 2024-03-16T11:50:47.083 INFO:teuthology.orchestra.run.smithi029.stdout:{} 2024-03-16T11:50:47.085 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:47.084+0000 7fdb4f190640 1 -- 172.21.15.29:0/120592845 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fdb380a28a0 msgr2=0x7fdb380a4d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:47.085 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:47.084+0000 7fdb4f190640 1 --2- 172.21.15.29:0/120592845 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fdb380a28a0 0x7fdb380a4d60 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fdb34000c00 tx=0x7fdb34005fd0 comp rx=0 tx=0).stop 2024-03-16T11:50:47.085 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:47.084+0000 7fdb4f190640 1 -- 172.21.15.29:0/120592845 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdb481022c0 msgr2=0x7fdb48197ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:47.086 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:47.084+0000 7fdb4f190640 1 --2- 172.21.15.29:0/120592845 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdb481022c0 0x7fdb48197ad0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fdb3c00e9f0 tx=0x7fdb3c00edf0 comp rx=0 tx=0).stop 2024-03-16T11:50:47.086 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:47.084+0000 7fdb4f190640 1 -- 172.21.15.29:0/120592845 shutdown_connections 2024-03-16T11:50:47.086 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:47.084+0000 7fdb4f190640 1 --2- 172.21.15.29:0/120592845 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fdb380a28a0 0x7fdb380a4d60 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:47.086 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:47.084+0000 7fdb4f190640 1 --2- 172.21.15.29:0/120592845 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fdb48107520 0x7fdb4819f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:47.086 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:47.084+0000 7fdb4f190640 1 --2- 172.21.15.29:0/120592845 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdb48104bf0 0x7fdb48198010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:47.087 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:47.084+0000 7fdb4f190640 1 --2- 172.21.15.29:0/120592845 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdb481022c0 0x7fdb48197ad0 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:47.087 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:47.084+0000 7fdb4f190640 1 -- 172.21.15.29:0/120592845 >> 172.21.15.29:0/120592845 conn(0x7fdb480fbc70 msgr2=0x7fdb48102c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:50:47.087 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:47.084+0000 7fdb4f190640 1 -- 172.21.15.29:0/120592845 shutdown_connections 2024-03-16T11:50:47.087 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:47.084+0000 7fdb4f190640 1 -- 172.21.15.29:0/120592845 wait complete. 2024-03-16T11:50:47.095 INFO:teuthology.run_tasks:Running task check-counter... 2024-03-16T11:50:47.106 INFO:teuthology.run_tasks:Running task exec... 2024-03-16T11:50:47.124 INFO:teuthology.task.exec:Executing custom commands... 2024-03-16T11:50:47.124 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi029.front.sepia.ceph.com 2024-03-16T11:50:47.124 DEBUG:teuthology.orchestra.run.smithi029:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph mgr module enable snap_schedule' 2024-03-16T11:50:47.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:50:46 smithi152 ceph-mon[36291]: from='client.? 172.21.15.29:0/3768823820' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-03-16T11:50:47.229 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:47.227+0000 7fdaa31df640 1 -- 172.21.15.29:0/806184788 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fda9c1022b0 msgr2=0x7fda9c108b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:47.229 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:47.227+0000 7fdaa31df640 1 --2- 172.21.15.29:0/806184788 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fda9c1022b0 0x7fda9c108b40 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fda8c009860 tx=0x7fda8c02bb70 comp rx=0 tx=0).stop 2024-03-16T11:50:47.229 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:47.227+0000 7fdaa31df640 1 -- 172.21.15.29:0/806184788 shutdown_connections 2024-03-16T11:50:47.229 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:47.227+0000 7fdaa31df640 1 --2- 172.21.15.29:0/806184788 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fda9c1022b0 0x7fda9c108b40 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:47.229 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:47.227+0000 7fdaa31df640 1 --2- 172.21.15.29:0/806184788 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fda9c1018f0 0x7fda9c101d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:47.229 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:47.227+0000 7fdaa31df640 1 --2- 172.21.15.29:0/806184788 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fda9c100700 0x7fda9c100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:47.229 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:47.227+0000 7fdaa31df640 1 -- 172.21.15.29:0/806184788 >> 172.21.15.29:0/806184788 conn(0x7fda9c0fbc70 msgr2=0x7fda9c0fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:50:47.230 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:47.227+0000 7fdaa31df640 1 -- 172.21.15.29:0/806184788 shutdown_connections 2024-03-16T11:50:47.230 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:47.227+0000 7fdaa31df640 1 -- 172.21.15.29:0/806184788 wait complete. 2024-03-16T11:50:47.230 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:47.227+0000 7fdaa31df640 1 Processor -- start 2024-03-16T11:50:47.230 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:47.227+0000 7fdaa31df640 1 -- start start 2024-03-16T11:50:47.230 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:47.228+0000 7fdaa31df640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fda9c100700 0x7fda9c199c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:47.230 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:47.228+0000 7fdaa31df640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fda9c1018f0 0x7fda9c19a160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:47.230 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:47.228+0000 7fdaa31df640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fda9c1022b0 0x7fda9c1a1190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:47.230 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:47.228+0000 7fdaa31df640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fda9c10c360 con 0x7fda9c1022b0 2024-03-16T11:50:47.230 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:47.228+0000 7fdaa31df640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fda9c10c1e0 con 0x7fda9c100700 2024-03-16T11:50:47.230 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:47.228+0000 7fdaa31df640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fda9c10c4e0 con 0x7fda9c1018f0 2024-03-16T11:50:47.230 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:47.228+0000 7fdaa177c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fda9c1022b0 0x7fda9c1a1190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:47.230 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:47.228+0000 7fdaa177c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fda9c1022b0 0x7fda9c1a1190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39918/0 (socket says 172.21.15.29:39918) 2024-03-16T11:50:47.230 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:47.228+0000 7fdaa177c640 1 -- 172.21.15.29:0/903019515 learned_addr learned my addr 172.21.15.29:0/903019515 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:50:47.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:47.228+0000 7fda9bfff640 1 --2- 172.21.15.29:0/903019515 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fda9c1018f0 0x7fda9c19a160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:47.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:47.228+0000 7fdaa0f7b640 1 --2- 172.21.15.29:0/903019515 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fda9c100700 0x7fda9c199c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:47.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:47.228+0000 7fdaa177c640 1 -- 172.21.15.29:0/903019515 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fda9c1018f0 msgr2=0x7fda9c19a160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:47.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:47.228+0000 7fdaa177c640 1 --2- 172.21.15.29:0/903019515 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fda9c1018f0 0x7fda9c19a160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:47.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:47.228+0000 7fdaa177c640 1 -- 172.21.15.29:0/903019515 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fda9c100700 msgr2=0x7fda9c199c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:47.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:47.228+0000 7fdaa177c640 1 --2- 172.21.15.29:0/903019515 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fda9c100700 0x7fda9c199c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:47.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:47.228+0000 7fdaa177c640 1 -- 172.21.15.29:0/903019515 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda9c1a1830 con 0x7fda9c1022b0 2024-03-16T11:50:47.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:47.228+0000 7fdaa177c640 1 --2- 172.21.15.29:0/903019515 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fda9c1022b0 0x7fda9c1a1190 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fda8c034b10 tx=0x7fda8c032470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:50:47.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:47.228+0000 7fda99ffb640 1 -- 172.21.15.29:0/903019515 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda8c041030 con 0x7fda9c1022b0 2024-03-16T11:50:47.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:47.228+0000 7fdaa31df640 1 -- 172.21.15.29:0/903019515 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fda9c1a1ac0 con 0x7fda9c1022b0 2024-03-16T11:50:47.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:47.228+0000 7fda99ffb640 1 -- 172.21.15.29:0/903019515 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fda8c02c510 con 0x7fda9c1022b0 2024-03-16T11:50:47.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:47.228+0000 7fda99ffb640 1 -- 172.21.15.29:0/903019515 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda8c039040 con 0x7fda9c1022b0 2024-03-16T11:50:47.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:47.229+0000 7fdaa31df640 1 -- 172.21.15.29:0/903019515 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fda9c1a2000 con 0x7fda9c1022b0 2024-03-16T11:50:47.232 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:47.230+0000 7fdaa31df640 1 -- 172.21.15.29:0/903019515 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fda9c1a21d0 con 0x7fda9c1022b0 2024-03-16T11:50:47.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:47.231+0000 7fda99ffb640 1 -- 172.21.15.29:0/903019515 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fda8c044070 con 0x7fda9c1022b0 2024-03-16T11:50:47.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:47.231+0000 7fda99ffb640 1 --2- 172.21.15.29:0/903019515 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fda840a28a0 0x7fda840a4d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:47.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:47.232+0000 7fdaa0f7b640 1 --2- 172.21.15.29:0/903019515 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fda840a28a0 0x7fda840a4d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:47.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:47.232+0000 7fda99ffb640 1 -- 172.21.15.29:0/903019515 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8211+0+0 (secure 0 0 0) 0x7fda8c0eff60 con 0x7fda9c1022b0 2024-03-16T11:50:47.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:47.232+0000 7fdaa0f7b640 1 --2- 172.21.15.29:0/903019515 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fda840a28a0 0x7fda840a4d60 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fda9c19b0f0 tx=0x7fda94009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:50:47.236 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:47.236+0000 7fda99ffb640 1 -- 172.21.15.29:0/903019515 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fda8c0bd010 con 0x7fda9c1022b0 2024-03-16T11:50:47.338 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 16 11:50:47 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-x[32210]: 2024-03-16T11:50:47.071+0000 7f7e0371a700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-16T11:50:47.338 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 16 11:50:47 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-x[32210]: 2024-03-16T11:50:47.071+0000 7f7e0371a700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-16T11:50:47.338 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 16 11:50:47 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-x[32210]: 2024-03-16T11:50:47.071+0000 7f7e0371a700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-16T11:50:47.338 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 16 11:50:47 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-x[32210]: 2024-03-16T11:50:47.071+0000 7f7e0371a700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-16T11:50:47.338 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 16 11:50:47 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-x[32210]: 2024-03-16T11:50:47.071+0000 7f7e0371a700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-16T11:50:47.468 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:47.468+0000 7fdaa31df640 1 -- 172.21.15.29:0/903019515 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) -- 0x7fda9c1a23a0 con 0x7fda9c1022b0 2024-03-16T11:50:47.724 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:47.724+0000 7fda99ffb640 1 -- 172.21.15.29:0/903019515 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v19) ==== 92+0+0 (secure 0 0 0) 0x7fda8c0bc910 con 0x7fda9c1022b0 2024-03-16T11:50:47.727 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:47.726+0000 7fdaa31df640 1 -- 172.21.15.29:0/903019515 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fda840a28a0 msgr2=0x7fda840a4d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:47.727 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:47.726+0000 7fdaa31df640 1 --2- 172.21.15.29:0/903019515 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fda840a28a0 0x7fda840a4d60 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fda9c19b0f0 tx=0x7fda94009500 comp rx=0 tx=0).stop 2024-03-16T11:50:47.727 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:47.726+0000 7fdaa31df640 1 -- 172.21.15.29:0/903019515 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fda9c1022b0 msgr2=0x7fda9c1a1190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:47.727 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:47.726+0000 7fdaa31df640 1 --2- 172.21.15.29:0/903019515 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fda9c1022b0 0x7fda9c1a1190 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fda8c034b10 tx=0x7fda8c032470 comp rx=0 tx=0).stop 2024-03-16T11:50:47.727 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:47.727+0000 7fdaa31df640 1 -- 172.21.15.29:0/903019515 shutdown_connections 2024-03-16T11:50:47.727 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:47.727+0000 7fdaa31df640 1 --2- 172.21.15.29:0/903019515 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fda840a28a0 0x7fda840a4d60 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:47.727 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:47.727+0000 7fdaa31df640 1 --2- 172.21.15.29:0/903019515 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fda9c1022b0 0x7fda9c1a1190 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:47.727 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:47.727+0000 7fdaa31df640 1 --2- 172.21.15.29:0/903019515 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fda9c1018f0 0x7fda9c19a160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:47.728 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:47.727+0000 7fdaa31df640 1 --2- 172.21.15.29:0/903019515 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fda9c100700 0x7fda9c199c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:47.728 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:47.727+0000 7fdaa31df640 1 -- 172.21.15.29:0/903019515 >> 172.21.15.29:0/903019515 conn(0x7fda9c0fbc70 msgr2=0x7fda9c0fd630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:50:47.728 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:47.727+0000 7fdaa31df640 1 -- 172.21.15.29:0/903019515 shutdown_connections 2024-03-16T11:50:47.728 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:47.727+0000 7fdaa31df640 1 -- 172.21.15.29:0/903019515 wait complete. 2024-03-16T11:50:47.744 DEBUG:teuthology.orchestra.run.smithi029:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph config set mgr mgr/snap_schedule/allow_m_granularity true' 2024-03-16T11:50:47.856 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:47.853+0000 7fea6071b640 1 -- 172.21.15.29:0/280251332 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fea58074200 msgr2=0x7fea58074680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:47.856 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:47.853+0000 7fea6071b640 1 --2- 172.21.15.29:0/280251332 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fea58074200 0x7fea58074680 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fea54009900 tx=0x7fea5402bc30 comp rx=0 tx=0).stop 2024-03-16T11:50:47.856 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:47.854+0000 7fea6071b640 1 -- 172.21.15.29:0/280251332 shutdown_connections 2024-03-16T11:50:47.856 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:47.854+0000 7fea6071b640 1 --2- 172.21.15.29:0/280251332 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fea581096e0 0x7fea5810bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:47.856 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:47.854+0000 7fea6071b640 1 --2- 172.21.15.29:0/280251332 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fea58074200 0x7fea58074680 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:47.856 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:47.854+0000 7fea6071b640 1 --2- 172.21.15.29:0/280251332 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fea580757e0 0x7fea58073c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:47.856 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:47.854+0000 7fea6071b640 1 -- 172.21.15.29:0/280251332 >> 172.21.15.29:0/280251332 conn(0x7fea580fcda0 msgr2=0x7fea580ff1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:50:47.856 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:47.854+0000 7fea6071b640 1 -- 172.21.15.29:0/280251332 shutdown_connections 2024-03-16T11:50:47.856 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:47.854+0000 7fea6071b640 1 -- 172.21.15.29:0/280251332 wait complete. 2024-03-16T11:50:47.856 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:47.854+0000 7fea6071b640 1 Processor -- start 2024-03-16T11:50:47.856 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:47.854+0000 7fea6071b640 1 -- start start 2024-03-16T11:50:47.856 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:47.854+0000 7fea6071b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fea58074200 0x7fea58199d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:47.856 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:47.854+0000 7fea6071b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fea580757e0 0x7fea5819a250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:47.857 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:47.854+0000 7fea6071b640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fea581096e0 0x7fea581a1210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:47.857 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:47.854+0000 7fea6071b640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fea5810f690 con 0x7fea580757e0 2024-03-16T11:50:47.857 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:47.854+0000 7fea6071b640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fea5810f510 con 0x7fea581096e0 2024-03-16T11:50:47.857 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:47.854+0000 7fea6071b640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fea5810f810 con 0x7fea58074200 2024-03-16T11:50:47.857 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:47.855+0000 7fea5dcb6640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fea580757e0 0x7fea5819a250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:47.857 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:47.855+0000 7fea5dcb6640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fea580757e0 0x7fea5819a250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39950/0 (socket says 172.21.15.29:39950) 2024-03-16T11:50:47.857 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:47.855+0000 7fea5dcb6640 1 -- 172.21.15.29:0/3002082110 learned_addr learned my addr 172.21.15.29:0/3002082110 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:50:47.857 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:47.855+0000 7fea5e4b7640 1 --2- 172.21.15.29:0/3002082110 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fea58074200 0x7fea58199d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:47.857 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:47.855+0000 7fea5ecb8640 1 --2- 172.21.15.29:0/3002082110 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fea581096e0 0x7fea581a1210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:47.857 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:47.855+0000 7fea5dcb6640 1 -- 172.21.15.29:0/3002082110 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fea58074200 msgr2=0x7fea58199d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:47.857 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:47.855+0000 7fea5dcb6640 1 --2- 172.21.15.29:0/3002082110 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fea58074200 0x7fea58199d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:47.857 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:47.855+0000 7fea5dcb6640 1 -- 172.21.15.29:0/3002082110 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fea581096e0 msgr2=0x7fea581a1210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:47.857 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:47.855+0000 7fea5dcb6640 1 --2- 172.21.15.29:0/3002082110 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fea581096e0 0x7fea581a1210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:47.857 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:47.855+0000 7fea5dcb6640 1 -- 172.21.15.29:0/3002082110 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea581a18b0 con 0x7fea580757e0 2024-03-16T11:50:47.857 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:47.855+0000 7fea5dcb6640 1 --2- 172.21.15.29:0/3002082110 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fea580757e0 0x7fea5819a250 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fea54009230 tx=0x7fea5402c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:50:47.858 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:47.855+0000 7fea4b7fe640 1 -- 172.21.15.29:0/3002082110 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea54041070 con 0x7fea580757e0 2024-03-16T11:50:47.858 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:47.855+0000 7fea4b7fe640 1 -- 172.21.15.29:0/3002082110 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fea54004450 con 0x7fea580757e0 2024-03-16T11:50:47.858 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:47.855+0000 7fea6071b640 1 -- 172.21.15.29:0/3002082110 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fea581a1b40 con 0x7fea580757e0 2024-03-16T11:50:47.858 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:47.855+0000 7fea4b7fe640 1 -- 172.21.15.29:0/3002082110 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea54003e20 con 0x7fea580757e0 2024-03-16T11:50:47.858 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:47.855+0000 7fea6071b640 1 -- 172.21.15.29:0/3002082110 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fea581a2030 con 0x7fea580757e0 2024-03-16T11:50:47.859 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:47.856+0000 7fea6071b640 1 -- 172.21.15.29:0/3002082110 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fea58108460 con 0x7fea580757e0 2024-03-16T11:50:47.859 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:47.857+0000 7fea4b7fe640 1 -- 172.21.15.29:0/3002082110 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 19) ==== 137307+0+0 (secure 0 0 0) 0x7fea54044070 con 0x7fea580757e0 2024-03-16T11:50:47.859 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:47.858+0000 7fea4b7fe640 1 --2- 172.21.15.29:0/3002082110 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fea440a28f0 0x7fea440a4db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:47.859 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:47.858+0000 7fea5e4b7640 1 -- 172.21.15.29:0/3002082110 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fea440a28f0 msgr2=0x7fea440a4db0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6800/4192064640 2024-03-16T11:50:47.859 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:47.858+0000 7fea4b7fe640 1 -- 172.21.15.29:0/3002082110 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8211+0+0 (secure 0 0 0) 0x7fea540f0630 con 0x7fea580757e0 2024-03-16T11:50:47.859 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:47.858+0000 7fea5e4b7640 1 --2- 172.21.15.29:0/3002082110 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fea440a28f0 0x7fea440a4db0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-16T11:50:47.862 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:47.863+0000 7fea4b7fe640 1 -- 172.21.15.29:0/3002082110 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fea540bd090 con 0x7fea580757e0 2024-03-16T11:50:48.045 INFO:journalctl@ceph.mgr.z.smithi154.stdout:Mar 16 11:50:47 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-z[38182]: ignoring --setuser ceph since I am not root 2024-03-16T11:50:48.045 INFO:journalctl@ceph.mgr.z.smithi154.stdout:Mar 16 11:50:47 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-z[38182]: ignoring --setgroup ceph since I am not root 2024-03-16T11:50:48.045 INFO:journalctl@ceph.mgr.z.smithi154.stdout:Mar 16 11:50:47 smithi154 ceph-mgr[38186]: -- 172.21.15.154:0/3840828631 <== mon.1 v2:172.21.15.154:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55ebab8ac5a0 con 0x55ebab881000 2024-03-16T11:50:48.046 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:50:47 smithi154 ceph-mon[36320]: from='client.? 172.21.15.29:0/3768823820' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2024-03-16T11:50:48.046 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:50:47 smithi154 ceph-mon[36320]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-03-16T11:50:48.046 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:50:47 smithi154 ceph-mon[36320]: from='client.15303 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T11:50:48.046 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:50:47 smithi154 ceph-mon[36320]: from='client.? 172.21.15.29:0/903019515' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-03-16T11:50:48.046 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:50:47 smithi154 ceph-mon[36320]: pgmap v357: 97 pgs: 97 active+clean; 641 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 170 B/s wr, 6 op/s 2024-03-16T11:50:48.048 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:48.048+0000 7fea6071b640 1 -- 172.21.15.29:0/3002082110 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) -- 0x7fea58062480 con 0x7fea580757e0 2024-03-16T11:50:48.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:48.052+0000 7fea4b7fe640 1 -- 172.21.15.29:0/3002082110 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v30)=0 v30) ==== 138+0+0 (secure 0 0 0) 0x7fea540c1130 con 0x7fea580757e0 2024-03-16T11:50:48.055 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:48.054+0000 7fea6071b640 1 -- 172.21.15.29:0/3002082110 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fea440a28f0 msgr2=0x7fea440a4db0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-16T11:50:48.055 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:48.054+0000 7fea6071b640 1 --2- 172.21.15.29:0/3002082110 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fea440a28f0 0x7fea440a4db0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:48.055 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:48.054+0000 7fea6071b640 1 -- 172.21.15.29:0/3002082110 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fea580757e0 msgr2=0x7fea5819a250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:48.055 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:48.054+0000 7fea6071b640 1 --2- 172.21.15.29:0/3002082110 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fea580757e0 0x7fea5819a250 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fea54009230 tx=0x7fea5402c650 comp rx=0 tx=0).stop 2024-03-16T11:50:48.055 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:48.055+0000 7fea6071b640 1 -- 172.21.15.29:0/3002082110 shutdown_connections 2024-03-16T11:50:48.055 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:48.055+0000 7fea6071b640 1 --2- 172.21.15.29:0/3002082110 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fea440a28f0 0x7fea440a4db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:48.055 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:48.055+0000 7fea6071b640 1 --2- 172.21.15.29:0/3002082110 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fea581096e0 0x7fea581a1210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:48.055 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:48.055+0000 7fea6071b640 1 --2- 172.21.15.29:0/3002082110 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fea580757e0 0x7fea5819a250 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:48.055 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:48.055+0000 7fea6071b640 1 --2- 172.21.15.29:0/3002082110 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fea58074200 0x7fea58199d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:48.056 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:48.055+0000 7fea6071b640 1 -- 172.21.15.29:0/3002082110 >> 172.21.15.29:0/3002082110 conn(0x7fea580fcda0 msgr2=0x7fea58105630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:50:48.056 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:48.055+0000 7fea6071b640 1 -- 172.21.15.29:0/3002082110 shutdown_connections 2024-03-16T11:50:48.056 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:48.055+0000 7fea6071b640 1 -- 172.21.15.29:0/3002082110 wait complete. 2024-03-16T11:50:48.065 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 16 11:50:47 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-x[32210]: ignoring --setuser ceph since I am not root 2024-03-16T11:50:48.065 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 16 11:50:47 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-x[32210]: ignoring --setgroup ceph since I am not root 2024-03-16T11:50:48.065 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:50:47 smithi029 ceph-mon[31975]: from='client.? 172.21.15.29:0/3768823820' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2024-03-16T11:50:48.065 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:50:47 smithi029 ceph-mon[31975]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-03-16T11:50:48.065 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:50:47 smithi029 ceph-mon[31975]: from='client.15303 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T11:50:48.065 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:50:47 smithi029 ceph-mon[31975]: from='client.? 172.21.15.29:0/903019515' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-03-16T11:50:48.065 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:50:47 smithi029 ceph-mon[31975]: pgmap v357: 97 pgs: 97 active+clean; 641 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 170 B/s wr, 6 op/s 2024-03-16T11:50:48.066 DEBUG:teuthology.orchestra.run.smithi029:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph config set mgr mgr/snap_schedule/dump_on_update true' 2024-03-16T11:50:48.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:50:47 smithi152 ceph-mon[36291]: from='client.? 172.21.15.29:0/3768823820' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2024-03-16T11:50:48.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:50:47 smithi152 ceph-mon[36291]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-03-16T11:50:48.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:50:47 smithi152 ceph-mon[36291]: from='client.15303 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T11:50:48.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:50:47 smithi152 ceph-mon[36291]: from='client.? 172.21.15.29:0/903019515' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-03-16T11:50:48.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:50:47 smithi152 ceph-mon[36291]: pgmap v357: 97 pgs: 97 active+clean; 641 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 170 B/s wr, 6 op/s 2024-03-16T11:50:48.128 INFO:journalctl@ceph.mgr.y.smithi152.stdout:Mar 16 11:50:47 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-y[37668]: ignoring --setuser ceph since I am not root 2024-03-16T11:50:48.128 INFO:journalctl@ceph.mgr.y.smithi152.stdout:Mar 16 11:50:47 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-y[37668]: ignoring --setgroup ceph since I am not root 2024-03-16T11:50:48.128 INFO:journalctl@ceph.mgr.y.smithi152.stdout:Mar 16 11:50:47 smithi152 ceph-mgr[37672]: -- 172.21.15.152:0/2719013839 <== mon.2 v2:172.21.15.152:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55814f7d65a0 con 0x55814f7aac00 2024-03-16T11:50:48.173 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:48.171+0000 7f6412ac6640 1 -- 172.21.15.29:0/1445860824 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f640c1033e0 msgr2=0x7f640c109c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:48.173 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:48.171+0000 7f6412ac6640 1 --2- 172.21.15.29:0/1445860824 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f640c1033e0 0x7f640c109c70 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f6404009ca0 tx=0x7f640402c0f0 comp rx=0 tx=0).stop 2024-03-16T11:50:48.174 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:48.171+0000 7f6412ac6640 1 -- 172.21.15.29:0/1445860824 shutdown_connections 2024-03-16T11:50:48.174 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:48.171+0000 7f6412ac6640 1 --2- 172.21.15.29:0/1445860824 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f640c1033e0 0x7f640c109c70 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:48.174 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:48.171+0000 7f6412ac6640 1 --2- 172.21.15.29:0/1445860824 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f640c102a20 0x7f640c102ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:48.174 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:48.171+0000 7f6412ac6640 1 --2- 172.21.15.29:0/1445860824 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f640c101830 0x7f640c101c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:48.174 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:48.171+0000 7f6412ac6640 1 -- 172.21.15.29:0/1445860824 >> 172.21.15.29:0/1445860824 conn(0x7f640c0fcdc0 msgr2=0x7f640c0ff1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:50:48.174 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:48.172+0000 7f6412ac6640 1 -- 172.21.15.29:0/1445860824 shutdown_connections 2024-03-16T11:50:48.174 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:48.172+0000 7f6412ac6640 1 -- 172.21.15.29:0/1445860824 wait complete. 2024-03-16T11:50:48.174 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:48.172+0000 7f6412ac6640 1 Processor -- start 2024-03-16T11:50:48.174 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:48.172+0000 7f6412ac6640 1 -- start start 2024-03-16T11:50:48.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:48.172+0000 7f6412ac6640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f640c101830 0x7f640c075590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:48.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:48.172+0000 7f6412ac6640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f640c102a20 0x7f640c073bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:48.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:48.172+0000 7f6412ac6640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f640c1033e0 0x7f640c074110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:48.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:48.172+0000 7f6412ac6640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f640c10d4b0 con 0x7f640c101830 2024-03-16T11:50:48.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:48.172+0000 7f6412ac6640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f640c10d330 con 0x7f640c102a20 2024-03-16T11:50:48.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:48.172+0000 7f6412ac6640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f640c10d630 con 0x7f640c1033e0 2024-03-16T11:50:48.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:48.172+0000 7f6410862640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f640c101830 0x7f640c075590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:48.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:48.172+0000 7f6410862640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f640c101830 0x7f640c075590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39966/0 (socket says 172.21.15.29:39966) 2024-03-16T11:50:48.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:48.172+0000 7f6410862640 1 -- 172.21.15.29:0/3703485186 learned_addr learned my addr 172.21.15.29:0/3703485186 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:50:48.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:48.172+0000 7f6411063640 1 --2- 172.21.15.29:0/3703485186 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f640c1033e0 0x7f640c074110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:48.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:48.172+0000 7f640bfff640 1 --2- 172.21.15.29:0/3703485186 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f640c102a20 0x7f640c073bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:48.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:48.173+0000 7f6410862640 1 -- 172.21.15.29:0/3703485186 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f640c1033e0 msgr2=0x7f640c074110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:48.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:48.173+0000 7f6410862640 1 --2- 172.21.15.29:0/3703485186 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f640c1033e0 0x7f640c074110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:48.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:48.173+0000 7f6410862640 1 -- 172.21.15.29:0/3703485186 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f640c102a20 msgr2=0x7f640c073bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:48.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:48.173+0000 7f6410862640 1 --2- 172.21.15.29:0/3703485186 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f640c102a20 0x7f640c073bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:48.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:48.173+0000 7f6410862640 1 -- 172.21.15.29:0/3703485186 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f640c0749a0 con 0x7f640c101830 2024-03-16T11:50:48.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:48.173+0000 7f6410862640 1 --2- 172.21.15.29:0/3703485186 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f640c101830 0x7f640c075590 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f640000e9c0 tx=0x7f640000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:50:48.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:48.173+0000 7f6409ffb640 1 -- 172.21.15.29:0/3703485186 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f640000cd80 con 0x7f640c101830 2024-03-16T11:50:48.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:48.173+0000 7f6409ffb640 1 -- 172.21.15.29:0/3703485186 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f64000035f0 con 0x7f640c101830 2024-03-16T11:50:48.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:48.173+0000 7f6412ac6640 1 -- 172.21.15.29:0/3703485186 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f640c1a1b30 con 0x7f640c101830 2024-03-16T11:50:48.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:48.173+0000 7f6409ffb640 1 -- 172.21.15.29:0/3703485186 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6400010690 con 0x7f640c101830 2024-03-16T11:50:48.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:48.173+0000 7f6412ac6640 1 -- 172.21.15.29:0/3703485186 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f640c1a2030 con 0x7f640c101830 2024-03-16T11:50:48.177 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:48.175+0000 7f6412ac6640 1 -- 172.21.15.29:0/3703485186 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f640c108af0 con 0x7f640c101830 2024-03-16T11:50:48.177 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:48.175+0000 7f6409ffb640 1 -- 172.21.15.29:0/3703485186 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 19) ==== 137307+0+0 (secure 0 0 0) 0x7f64000037a0 con 0x7f640c101830 2024-03-16T11:50:48.177 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:48.176+0000 7f6409ffb640 1 --2- 172.21.15.29:0/3703485186 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f63fc0a28f0 0x7f63fc0a4db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:48.177 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:48.176+0000 7f640bfff640 1 -- 172.21.15.29:0/3703485186 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f63fc0a28f0 msgr2=0x7f63fc0a4db0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6800/4192064640 2024-03-16T11:50:48.177 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:48.176+0000 7f6409ffb640 1 -- 172.21.15.29:0/3703485186 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8211+0+0 (secure 0 0 0) 0x7f64000cee90 con 0x7f640c101830 2024-03-16T11:50:48.177 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:48.176+0000 7f640bfff640 1 --2- 172.21.15.29:0/3703485186 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f63fc0a28f0 0x7f63fc0a4db0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-16T11:50:48.180 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:48.181+0000 7f6409ffb640 1 -- 172.21.15.29:0/3703485186 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f640009bf40 con 0x7f640c101830 2024-03-16T11:50:48.377 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:48.377+0000 7f640bfff640 1 -- 172.21.15.29:0/3703485186 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f63fc0a28f0 msgr2=0x7f63fc0a4db0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6800/4192064640 2024-03-16T11:50:48.377 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:48.377+0000 7f640bfff640 1 --2- 172.21.15.29:0/3703485186 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f63fc0a28f0 0x7f63fc0a4db0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-16T11:50:48.381 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:48.381+0000 7f6412ac6640 1 -- 172.21.15.29:0/3703485186 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) -- 0x7f640c02d1b0 con 0x7f640c101830 2024-03-16T11:50:48.384 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:48.384+0000 7f6409ffb640 1 -- 172.21.15.29:0/3703485186 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v31)=0 v31) ==== 133+0+0 (secure 0 0 0) 0x7f640009b840 con 0x7f640c101830 2024-03-16T11:50:48.387 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:48.387+0000 7f6412ac6640 1 -- 172.21.15.29:0/3703485186 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f63fc0a28f0 msgr2=0x7f63fc0a4db0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-16T11:50:48.387 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:48.387+0000 7f6412ac6640 1 --2- 172.21.15.29:0/3703485186 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f63fc0a28f0 0x7f63fc0a4db0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:48.387 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:48.387+0000 7f6412ac6640 1 -- 172.21.15.29:0/3703485186 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f640c101830 msgr2=0x7f640c075590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:48.387 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:48.387+0000 7f6412ac6640 1 --2- 172.21.15.29:0/3703485186 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f640c101830 0x7f640c075590 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f640000e9c0 tx=0x7f640000ee80 comp rx=0 tx=0).stop 2024-03-16T11:50:48.387 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:48.387+0000 7f6412ac6640 1 -- 172.21.15.29:0/3703485186 shutdown_connections 2024-03-16T11:50:48.387 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:48.387+0000 7f6412ac6640 1 --2- 172.21.15.29:0/3703485186 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7f63fc0a28f0 0x7f63fc0a4db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:48.387 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:48.387+0000 7f6412ac6640 1 --2- 172.21.15.29:0/3703485186 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f640c1033e0 0x7f640c074110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:48.387 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:48.387+0000 7f6412ac6640 1 --2- 172.21.15.29:0/3703485186 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f640c102a20 0x7f640c073bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:48.387 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:48.387+0000 7f6412ac6640 1 --2- 172.21.15.29:0/3703485186 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f640c101830 0x7f640c075590 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:48.388 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:48.387+0000 7f6412ac6640 1 -- 172.21.15.29:0/3703485186 >> 172.21.15.29:0/3703485186 conn(0x7f640c0fcdc0 msgr2=0x7f640c105cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:50:48.388 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:48.387+0000 7f6412ac6640 1 -- 172.21.15.29:0/3703485186 shutdown_connections 2024-03-16T11:50:48.388 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:48.387+0000 7f6412ac6640 1 -- 172.21.15.29:0/3703485186 wait complete. 2024-03-16T11:50:48.398 DEBUG:teuthology.orchestra.run.smithi029:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule add --fs=cephfs --path=/ --snap_schedule=1m' 2024-03-16T11:50:48.505 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:48.505+0000 7fae24beb640 1 -- 172.21.15.29:0/1874666891 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae20101900 msgr2=0x7fae20101d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:48.505 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:48.505+0000 7fae24beb640 1 --2- 172.21.15.29:0/1874666891 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae20101900 0x7fae20101d80 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fae14009900 tx=0x7fae1402bc30 comp rx=0 tx=0).stop 2024-03-16T11:50:48.505 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:48.505+0000 7fae24beb640 1 -- 172.21.15.29:0/1874666891 shutdown_connections 2024-03-16T11:50:48.505 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:48.505+0000 7fae24beb640 1 --2- 172.21.15.29:0/1874666891 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fae201022c0 0x7fae20108b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:48.505 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:48.505+0000 7fae24beb640 1 --2- 172.21.15.29:0/1874666891 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae20101900 0x7fae20101d80 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:48.506 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:48.505+0000 7fae24beb640 1 --2- 172.21.15.29:0/1874666891 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fae20100710 0x7fae20100b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:48.506 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:48.505+0000 7fae24beb640 1 -- 172.21.15.29:0/1874666891 >> 172.21.15.29:0/1874666891 conn(0x7fae200fbca0 msgr2=0x7fae200fe0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:50:48.506 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:48.505+0000 7fae24beb640 1 -- 172.21.15.29:0/1874666891 shutdown_connections 2024-03-16T11:50:48.506 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:48.505+0000 7fae24beb640 1 -- 172.21.15.29:0/1874666891 wait complete. 2024-03-16T11:50:48.508 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:48.506+0000 7fae24beb640 1 Processor -- start 2024-03-16T11:50:48.508 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:48.506+0000 7fae24beb640 1 -- start start 2024-03-16T11:50:48.508 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:48.506+0000 7fae24beb640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae20100710 0x7fae20197a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:48.508 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:48.506+0000 7fae24beb640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fae20101900 0x7fae20197f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:48.508 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:48.506+0000 7fae24beb640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fae201022c0 0x7fae2019f000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:48.508 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:48.506+0000 7fae24beb640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fae2010d4b0 con 0x7fae20100710 2024-03-16T11:50:48.508 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:48.506+0000 7fae24beb640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fae2010d330 con 0x7fae201022c0 2024-03-16T11:50:48.509 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:48.506+0000 7fae24beb640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fae2010d630 con 0x7fae20101900 2024-03-16T11:50:48.509 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:48.506+0000 7fae1e59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae20100710 0x7fae20197a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:48.509 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:48.506+0000 7fae1e59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae20100710 0x7fae20197a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39992/0 (socket says 172.21.15.29:39992) 2024-03-16T11:50:48.509 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:48.506+0000 7fae1e59c640 1 -- 172.21.15.29:0/1441403041 learned_addr learned my addr 172.21.15.29:0/1441403041 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:50:48.509 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:48.506+0000 7fae1ed9d640 1 --2- 172.21.15.29:0/1441403041 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fae201022c0 0x7fae2019f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:48.509 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:48.506+0000 7fae1dd9b640 1 --2- 172.21.15.29:0/1441403041 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fae20101900 0x7fae20197f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:48.509 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:48.506+0000 7fae1e59c640 1 -- 172.21.15.29:0/1441403041 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fae20101900 msgr2=0x7fae20197f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:48.509 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:48.506+0000 7fae1e59c640 1 --2- 172.21.15.29:0/1441403041 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fae20101900 0x7fae20197f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:48.509 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:48.506+0000 7fae1e59c640 1 -- 172.21.15.29:0/1441403041 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fae201022c0 msgr2=0x7fae2019f000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:48.509 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:48.506+0000 7fae1e59c640 1 --2- 172.21.15.29:0/1441403041 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fae201022c0 0x7fae2019f000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:48.509 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:48.506+0000 7fae1e59c640 1 -- 172.21.15.29:0/1441403041 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae2019f670 con 0x7fae20100710 2024-03-16T11:50:48.509 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:48.506+0000 7fae1e59c640 1 --2- 172.21.15.29:0/1441403041 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae20100710 0x7fae20197a50 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fae1000e9c0 tx=0x7fae1000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:50:48.509 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:48.507+0000 7fae0f7fe640 1 -- 172.21.15.29:0/1441403041 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae1000cd80 con 0x7fae20100710 2024-03-16T11:50:48.509 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:48.507+0000 7fae0f7fe640 1 -- 172.21.15.29:0/1441403041 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fae100035f0 con 0x7fae20100710 2024-03-16T11:50:48.509 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:48.507+0000 7fae24beb640 1 -- 172.21.15.29:0/1441403041 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fae2019f960 con 0x7fae20100710 2024-03-16T11:50:48.510 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:48.507+0000 7fae0f7fe640 1 -- 172.21.15.29:0/1441403041 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae10010690 con 0x7fae20100710 2024-03-16T11:50:48.510 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:48.507+0000 7fae24beb640 1 -- 172.21.15.29:0/1441403041 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fae2019fed0 con 0x7fae20100710 2024-03-16T11:50:48.510 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:48.508+0000 7fae24beb640 1 -- 172.21.15.29:0/1441403041 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fae2004f540 con 0x7fae20100710 2024-03-16T11:50:48.510 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:48.509+0000 7fae0f7fe640 1 -- 172.21.15.29:0/1441403041 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 19) ==== 137307+0+0 (secure 0 0 0) 0x7fae100037a0 con 0x7fae20100710 2024-03-16T11:50:48.511 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:48.509+0000 7fae0f7fe640 1 --2- 172.21.15.29:0/1441403041 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fae080a2970 0x7fae080a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:48.511 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:48.509+0000 7fae1dd9b640 1 -- 172.21.15.29:0/1441403041 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fae080a2970 msgr2=0x7fae080a4e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6800/4192064640 2024-03-16T11:50:48.511 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:48.509+0000 7fae0f7fe640 1 -- 172.21.15.29:0/1441403041 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8211+0+0 (secure 0 0 0) 0x7fae100cede0 con 0x7fae20100710 2024-03-16T11:50:48.511 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:48.509+0000 7fae1dd9b640 1 --2- 172.21.15.29:0/1441403041 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fae080a2970 0x7fae080a4e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-16T11:50:48.514 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:48.514+0000 7fae0f7fe640 1 -- 172.21.15.29:0/1441403041 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fae1009be90 con 0x7fae20100710 2024-03-16T11:50:48.587 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 16 11:50:48 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-x[32210]: 2024-03-16T11:50:48.335+0000 7fa209d7d200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-16T11:50:48.588 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 16 11:50:48 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-x[32210]: 2024-03-16T11:50:48.435+0000 7fa209d7d200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-16T11:50:48.627 INFO:journalctl@ceph.mgr.y.smithi152.stdout:Mar 16 11:50:48 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-y[37668]: 2024-03-16T11:50:48.350+0000 7f2a359eb200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-16T11:50:48.627 INFO:journalctl@ceph.mgr.y.smithi152.stdout:Mar 16 11:50:48 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-y[37668]: 2024-03-16T11:50:48.445+0000 7f2a359eb200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-16T11:50:48.710 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:48.710+0000 7fae1dd9b640 1 -- 172.21.15.29:0/1441403041 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fae080a2970 msgr2=0x7fae080a4e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6800/4192064640 2024-03-16T11:50:48.710 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:48.710+0000 7fae1dd9b640 1 --2- 172.21.15.29:0/1441403041 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fae080a2970 0x7fae080a4e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-16T11:50:48.721 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:48.721+0000 7fae24beb640 1 -- 172.21.15.29:0/1441403041 --> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) -- 0x7fae20061c20 con 0x7fae080a2970 2024-03-16T11:50:48.725 INFO:journalctl@ceph.mgr.z.smithi154.stdout:Mar 16 11:50:48 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-z[38182]: 2024-03-16T11:50:48.336+0000 7f526d7ef200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-16T11:50:48.725 INFO:journalctl@ceph.mgr.z.smithi154.stdout:Mar 16 11:50:48 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-z[38182]: 2024-03-16T11:50:48.432+0000 7f526d7ef200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-16T11:50:49.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:50:48 smithi154 ceph-mon[36320]: from='client.? 172.21.15.29:0/903019515' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-03-16T11:50:49.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:50:48 smithi154 ceph-mon[36320]: mgrmap e19: x(active, since 9m), standbys: z, y 2024-03-16T11:50:49.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:50:48 smithi154 ceph-mon[36320]: from='client.? 172.21.15.29:0/3002082110' entity='client.admin' 2024-03-16T11:50:49.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:50:48 smithi154 ceph-mon[36320]: from='client.? 172.21.15.29:0/3703485186' entity='client.admin' 2024-03-16T11:50:49.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:50:48 smithi029 ceph-mon[31975]: from='client.? 172.21.15.29:0/903019515' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-03-16T11:50:49.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:50:48 smithi029 ceph-mon[31975]: mgrmap e19: x(active, since 9m), standbys: z, y 2024-03-16T11:50:49.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:50:48 smithi029 ceph-mon[31975]: from='client.? 172.21.15.29:0/3002082110' entity='client.admin' 2024-03-16T11:50:49.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:50:48 smithi029 ceph-mon[31975]: from='client.? 172.21.15.29:0/3703485186' entity='client.admin' 2024-03-16T11:50:49.110 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:49.111+0000 7fae1dd9b640 1 -- 172.21.15.29:0/1441403041 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fae080a2970 msgr2=0x7fae080a4e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6800/4192064640 2024-03-16T11:50:49.111 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:49.111+0000 7fae1dd9b640 1 --2- 172.21.15.29:0/1441403041 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fae080a2970 0x7fae080a4e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-16T11:50:49.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:50:48 smithi152 ceph-mon[36291]: from='client.? 172.21.15.29:0/903019515' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-03-16T11:50:49.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:50:48 smithi152 ceph-mon[36291]: mgrmap e19: x(active, since 9m), standbys: z, y 2024-03-16T11:50:49.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:50:48 smithi152 ceph-mon[36291]: from='client.? 172.21.15.29:0/3002082110' entity='client.admin' 2024-03-16T11:50:49.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:50:48 smithi152 ceph-mon[36291]: from='client.? 172.21.15.29:0/3703485186' entity='client.admin' 2024-03-16T11:50:49.388 INFO:journalctl@ceph.mgr.z.smithi154.stdout:Mar 16 11:50:49 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-z[38182]: 2024-03-16T11:50:49.089+0000 7f526d7ef200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-16T11:50:49.388 INFO:journalctl@ceph.mgr.z.smithi154.stdout:Mar 16 11:50:49 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-z[38182]: 2024-03-16T11:50:49.224+0000 7f526d7ef200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-16T11:50:49.444 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 16 11:50:49 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-x[32210]: 2024-03-16T11:50:49.104+0000 7fa209d7d200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-16T11:50:49.444 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 16 11:50:49 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-x[32210]: 2024-03-16T11:50:49.200+0000 7fa209d7d200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-16T11:50:49.445 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 16 11:50:49 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-x[32210]: 2024-03-16T11:50:49.344+0000 7fa209d7d200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-16T11:50:49.445 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 16 11:50:49 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-x[32210]: 2024-03-16T11:50:49.444+0000 7fa209d7d200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-16T11:50:49.556 INFO:journalctl@ceph.mgr.y.smithi152.stdout:Mar 16 11:50:49 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-y[37668]: 2024-03-16T11:50:49.255+0000 7f2a359eb200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-16T11:50:49.556 INFO:journalctl@ceph.mgr.y.smithi152.stdout:Mar 16 11:50:49 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-y[37668]: 2024-03-16T11:50:49.381+0000 7f2a359eb200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-16T11:50:49.556 INFO:journalctl@ceph.mgr.y.smithi152.stdout:Mar 16 11:50:49 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-y[37668]: 2024-03-16T11:50:49.556+0000 7f2a359eb200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-16T11:50:49.795 INFO:journalctl@ceph.mgr.z.smithi154.stdout:Mar 16 11:50:49 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-z[38182]: 2024-03-16T11:50:49.389+0000 7f526d7ef200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-16T11:50:49.795 INFO:journalctl@ceph.mgr.z.smithi154.stdout:Mar 16 11:50:49 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-z[38182]: 2024-03-16T11:50:49.486+0000 7f526d7ef200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-16T11:50:49.795 INFO:journalctl@ceph.mgr.z.smithi154.stdout:Mar 16 11:50:49 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-z[38182]: 2024-03-16T11:50:49.570+0000 7f526d7ef200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-16T11:50:49.802 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 16 11:50:49 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-x[32210]: 2024-03-16T11:50:49.530+0000 7fa209d7d200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-16T11:50:49.877 INFO:journalctl@ceph.mgr.y.smithi152.stdout:Mar 16 11:50:49 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-y[37668]: 2024-03-16T11:50:49.675+0000 7f2a359eb200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-16T11:50:49.877 INFO:journalctl@ceph.mgr.y.smithi152.stdout:Mar 16 11:50:49 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-y[37668]: 2024-03-16T11:50:49.758+0000 7f2a359eb200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-16T11:50:49.912 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:49.912+0000 7fae1dd9b640 1 -- 172.21.15.29:0/1441403041 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fae080a2970 msgr2=0x7fae080a4e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6800/4192064640 2024-03-16T11:50:49.912 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:49.912+0000 7fae1dd9b640 1 --2- 172.21.15.29:0/1441403041 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fae080a2970 0x7fae080a4e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-16T11:50:50.088 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 16 11:50:49 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-x[32210]: 2024-03-16T11:50:49.802+0000 7fa209d7d200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-16T11:50:50.088 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 16 11:50:49 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-x[32210]: 2024-03-16T11:50:49.915+0000 7fa209d7d200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-16T11:50:50.088 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 16 11:50:50 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-x[32210]: 2024-03-16T11:50:50.003+0000 7fa209d7d200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-16T11:50:50.206 INFO:journalctl@ceph.mgr.z.smithi154.stdout:Mar 16 11:50:49 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-z[38182]: 2024-03-16T11:50:49.860+0000 7f526d7ef200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-16T11:50:50.206 INFO:journalctl@ceph.mgr.z.smithi154.stdout:Mar 16 11:50:49 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-z[38182]: 2024-03-16T11:50:49.982+0000 7f526d7ef200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-16T11:50:50.206 INFO:journalctl@ceph.mgr.z.smithi154.stdout:Mar 16 11:50:50 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-z[38182]: 2024-03-16T11:50:50.093+0000 7f526d7ef200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-16T11:50:50.206 INFO:journalctl@ceph.mgr.z.smithi154.stdout:Mar 16 11:50:50 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-z[38182]: 2024-03-16T11:50:50.206+0000 7f526d7ef200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-16T11:50:50.355 INFO:journalctl@ceph.mgr.y.smithi152.stdout:Mar 16 11:50:50 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-y[37668]: 2024-03-16T11:50:50.031+0000 7f2a359eb200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-16T11:50:50.355 INFO:journalctl@ceph.mgr.y.smithi152.stdout:Mar 16 11:50:50 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-y[37668]: 2024-03-16T11:50:50.172+0000 7f2a359eb200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-16T11:50:50.355 INFO:journalctl@ceph.mgr.y.smithi152.stdout:Mar 16 11:50:50 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-y[37668]: 2024-03-16T11:50:50.259+0000 7f2a359eb200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-16T11:50:50.355 INFO:journalctl@ceph.mgr.y.smithi152.stdout:Mar 16 11:50:50 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-y[37668]: 2024-03-16T11:50:50.355+0000 7f2a359eb200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-16T11:50:50.545 INFO:journalctl@ceph.mgr.z.smithi154.stdout:Mar 16 11:50:50 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-z[38182]: 2024-03-16T11:50:50.444+0000 7f526d7ef200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-16T11:50:50.587 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 16 11:50:50 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-x[32210]: 2024-03-16T11:50:50.101+0000 7fa209d7d200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-16T11:50:50.587 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 16 11:50:50 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-x[32210]: 2024-03-16T11:50:50.280+0000 7fa209d7d200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-16T11:50:50.627 INFO:journalctl@ceph.mgr.y.smithi152.stdout:Mar 16 11:50:50 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-y[37668]: 2024-03-16T11:50:50.534+0000 7f2a359eb200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-16T11:50:51.513 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:51.513+0000 7fae1dd9b640 1 -- 172.21.15.29:0/1441403041 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fae080a2970 msgr2=0x7fae080a4e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6800/4192064640 2024-03-16T11:50:51.514 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:51.513+0000 7fae1dd9b640 1 --2- 172.21.15.29:0/1441403041 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fae080a2970 0x7fae080a4e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-16T11:50:51.587 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 16 11:50:51 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-x[32210]: 2024-03-16T11:50:51.273+0000 7fa209d7d200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-16T11:50:51.588 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 16 11:50:51 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-x[32210]: 2024-03-16T11:50:51.407+0000 7fa209d7d200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-16T11:50:51.795 INFO:journalctl@ceph.mgr.z.smithi154.stdout:Mar 16 11:50:51 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-z[38182]: 2024-03-16T11:50:51.486+0000 7f526d7ef200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-16T11:50:51.795 INFO:journalctl@ceph.mgr.z.smithi154.stdout:Mar 16 11:50:51 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-z[38182]: 2024-03-16T11:50:51.620+0000 7f526d7ef200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-16T11:50:52.050 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 16 11:50:51 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-x[32210]: 2024-03-16T11:50:51.750+0000 7fa209d7d200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-16T11:50:52.050 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 16 11:50:51 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-x[32210]: 2024-03-16T11:50:51.843+0000 7fa209d7d200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-16T11:50:52.123 INFO:journalctl@ceph.mgr.y.smithi152.stdout:Mar 16 11:50:51 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-y[37668]: 2024-03-16T11:50:51.651+0000 7f2a359eb200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-16T11:50:52.123 INFO:journalctl@ceph.mgr.y.smithi152.stdout:Mar 16 11:50:51 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-y[37668]: 2024-03-16T11:50:51.783+0000 7f2a359eb200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-16T11:50:52.123 INFO:journalctl@ceph.mgr.y.smithi152.stdout:Mar 16 11:50:52 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-y[37668]: 2024-03-16T11:50:52.123+0000 7f2a359eb200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-16T11:50:52.279 INFO:journalctl@ceph.mgr.z.smithi154.stdout:Mar 16 11:50:51 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-z[38182]: 2024-03-16T11:50:51.963+0000 7f526d7ef200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-16T11:50:52.279 INFO:journalctl@ceph.mgr.z.smithi154.stdout:Mar 16 11:50:52 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-z[38182]: 2024-03-16T11:50:52.054+0000 7f526d7ef200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-16T11:50:52.279 INFO:journalctl@ceph.mgr.z.smithi154.stdout:Mar 16 11:50:52 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-z[38182]: 2024-03-16T11:50:52.280+0000 7f526d7ef200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-16T11:50:52.337 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 16 11:50:52 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-x[32210]: 2024-03-16T11:50:52.048+0000 7fa209d7d200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-16T11:50:52.337 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 16 11:50:52 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-x[32210]: 2024-03-16T11:50:52.241+0000 7fa209d7d200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-16T11:50:52.377 INFO:journalctl@ceph.mgr.y.smithi152.stdout:Mar 16 11:50:52 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-y[37668]: 2024-03-16T11:50:52.214+0000 7f2a359eb200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-16T11:50:52.545 INFO:journalctl@ceph.mgr.z.smithi154.stdout:Mar 16 11:50:52 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-z[38182]: 2024-03-16T11:50:52.491+0000 7f526d7ef200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-16T11:50:52.802 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 16 11:50:52 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-x[32210]: 2024-03-16T11:50:52.539+0000 7fa209d7d200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-16T11:50:52.877 INFO:journalctl@ceph.mgr.y.smithi152.stdout:Mar 16 11:50:52 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-y[37668]: 2024-03-16T11:50:52.412+0000 7f2a359eb200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-16T11:50:52.877 INFO:journalctl@ceph.mgr.y.smithi152.stdout:Mar 16 11:50:52 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-y[37668]: 2024-03-16T11:50:52.601+0000 7f2a359eb200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-16T11:50:53.045 INFO:journalctl@ceph.mgr.z.smithi154.stdout:Mar 16 11:50:52 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-z[38182]: 2024-03-16T11:50:52.794+0000 7f526d7ef200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-16T11:50:53.088 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 16 11:50:52 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-x[32210]: 2024-03-16T11:50:52.802+0000 7fa209d7d200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-16T11:50:53.088 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 16 11:50:53 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-x[32210]: 2024-03-16T11:50:53.048+0000 7fa209d7d200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-16T11:50:53.155 INFO:journalctl@ceph.mgr.y.smithi152.stdout:Mar 16 11:50:52 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-y[37668]: 2024-03-16T11:50:52.890+0000 7f2a359eb200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-16T11:50:53.356 INFO:journalctl@ceph.mgr.z.smithi154.stdout:Mar 16 11:50:53 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-z[38182]: 2024-03-16T11:50:53.099+0000 7f526d7ef200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-16T11:50:53.459 INFO:journalctl@ceph.mgr.y.smithi152.stdout:Mar 16 11:50:53 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-y[37668]: 2024-03-16T11:50:53.155+0000 7f2a359eb200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-16T11:50:53.795 INFO:journalctl@ceph.mgr.z.smithi154.stdout:Mar 16 11:50:53 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-z[38182]: 2024-03-16T11:50:53.356+0000 7f526d7ef200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-16T11:50:53.877 INFO:journalctl@ceph.mgr.y.smithi152.stdout:Mar 16 11:50:53 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-y[37668]: 2024-03-16T11:50:53.460+0000 7f2a359eb200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-16T11:50:54.089 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 16 11:50:53 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-x[32210]: 2024-03-16T11:50:53.891+0000 7fa209d7d200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-16T11:50:54.545 INFO:journalctl@ceph.mgr.z.smithi154.stdout:Mar 16 11:50:54 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-z[38182]: 2024-03-16T11:50:54.200+0000 7f526d7ef200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-16T11:50:54.627 INFO:journalctl@ceph.mgr.y.smithi152.stdout:Mar 16 11:50:54 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-y[37668]: 2024-03-16T11:50:54.362+0000 7f2a359eb200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-16T11:50:54.717 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:54.717+0000 7fae1dd9b640 1 -- 172.21.15.29:0/1441403041 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fae080a2970 msgr2=0x7fae080a4e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6800/4192064640 2024-03-16T11:50:54.717 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:54.717+0000 7fae1dd9b640 1 --2- 172.21.15.29:0/1441403041 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fae080a2970 0x7fae080a4e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 6.400000 2024-03-16T11:50:54.836 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 16 11:50:54 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-x[32210]: 2024-03-16T11:50:54.502+0000 7fa209d7d200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-16T11:50:54.836 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 16 11:50:54 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-x[32210]: 2024-03-16T11:50:54.607+0000 7fa209d7d200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-16T11:50:54.836 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 16 11:50:54 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-x[32210]: 2024-03-16T11:50:54.835+0000 7fa209d7d200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-16T11:50:54.860 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:54.859+0000 7fae0f7fe640 1 -- 172.21.15.29:0/1441403041 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mgrmap(e 20) ==== 136975+0+0 (secure 0 0 0) 0x7fae10097e00 con 0x7fae20100710 2024-03-16T11:50:54.860 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:54.860+0000 7fae0f7fe640 1 -- 172.21.15.29:0/1441403041 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fae080a2970 msgr2=0x7fae080a4e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-16T11:50:54.860 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:54.860+0000 7fae0f7fe640 1 --2- 172.21.15.29:0/1441403041 >> [v2:172.21.15.29:6800/4192064640,v1:172.21.15.29:6801/4192064640] conn(0x7fae080a2970 0x7fae080a4e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:55.045 INFO:journalctl@ceph.mgr.z.smithi154.stdout:Mar 16 11:50:54 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-z[38182]: 2024-03-16T11:50:54.788+0000 7f526d7ef200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-16T11:50:55.045 INFO:journalctl@ceph.mgr.z.smithi154.stdout:Mar 16 11:50:54 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-z[38182]: 2024-03-16T11:50:54.892+0000 7f526d7ef200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-16T11:50:55.046 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:50:54 smithi154 ceph-mon[36320]: Active manager daemon x restarted 2024-03-16T11:50:55.046 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:50:54 smithi154 ceph-mon[36320]: Activating manager daemon x 2024-03-16T11:50:55.046 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:50:54 smithi154 ceph-mon[36320]: osdmap e88: 12 total, 12 up, 12 in 2024-03-16T11:50:55.046 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:50:54 smithi154 ceph-mon[36320]: mgrmap e20: x(active, starting, since 0.0116245s), standbys: z, y 2024-03-16T11:50:55.046 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:50:54 smithi154 ceph-mon[36320]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-16T11:50:55.047 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:50:54 smithi154 ceph-mon[36320]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T11:50:55.047 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:50:54 smithi154 ceph-mon[36320]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-16T11:50:55.047 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:50:54 smithi154 ceph-mon[36320]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-16T11:50:55.047 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:50:54 smithi154 ceph-mon[36320]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-16T11:50:55.047 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:50:54 smithi154 ceph-mon[36320]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-16T11:50:55.047 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:50:54 smithi154 ceph-mon[36320]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-16T11:50:55.047 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:50:54 smithi154 ceph-mon[36320]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-16T11:50:55.047 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:50:54 smithi154 ceph-mon[36320]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-16T11:50:55.047 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:50:54 smithi154 ceph-mon[36320]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-16T11:50:55.047 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:50:54 smithi154 ceph-mon[36320]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-16T11:50:55.047 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:50:54 smithi154 ceph-mon[36320]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-16T11:50:55.048 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:50:54 smithi154 ceph-mon[36320]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-16T11:50:55.048 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:50:54 smithi154 ceph-mon[36320]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-16T11:50:55.048 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:50:54 smithi154 ceph-mon[36320]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-16T11:50:55.048 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:50:54 smithi154 ceph-mon[36320]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-16T11:50:55.048 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:50:54 smithi154 ceph-mon[36320]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-16T11:50:55.048 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:50:54 smithi154 ceph-mon[36320]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-16T11:50:55.048 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:50:54 smithi154 ceph-mon[36320]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-16T11:50:55.048 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:50:54 smithi154 ceph-mon[36320]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-16T11:50:55.048 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:50:54 smithi154 ceph-mon[36320]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-16T11:50:55.048 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:50:54 smithi154 ceph-mon[36320]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-16T11:50:55.048 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:50:54 smithi154 ceph-mon[36320]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-16T11:50:55.048 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:50:54 smithi154 ceph-mon[36320]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-16T11:50:55.049 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:50:54 smithi154 ceph-mon[36320]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-16T11:50:55.049 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:50:54 smithi154 ceph-mon[36320]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-16T11:50:55.049 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:50:54 smithi154 ceph-mon[36320]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-16T11:50:55.049 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:50:54 smithi154 ceph-mon[36320]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-16T11:50:55.049 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:50:54 smithi154 ceph-mon[36320]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-16T11:50:55.049 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:50:54 smithi154 ceph-mon[36320]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-16T11:50:55.049 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:50:54 smithi154 ceph-mon[36320]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-16T11:50:55.049 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:50:54 smithi154 ceph-mon[36320]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-16T11:50:55.049 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:50:54 smithi154 ceph-mon[36320]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-16T11:50:55.049 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:50:54 smithi154 ceph-mon[36320]: Manager daemon x is now available 2024-03-16T11:50:55.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:50:54 smithi029 ceph-mon[31975]: Active manager daemon x restarted 2024-03-16T11:50:55.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:50:54 smithi029 ceph-mon[31975]: Activating manager daemon x 2024-03-16T11:50:55.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:50:54 smithi029 ceph-mon[31975]: osdmap e88: 12 total, 12 up, 12 in 2024-03-16T11:50:55.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:50:54 smithi029 ceph-mon[31975]: mgrmap e20: x(active, starting, since 0.0116245s), standbys: z, y 2024-03-16T11:50:55.089 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:50:54 smithi029 ceph-mon[31975]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-16T11:50:55.089 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:50:54 smithi029 ceph-mon[31975]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T11:50:55.089 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:50:54 smithi029 ceph-mon[31975]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-16T11:50:55.089 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:50:54 smithi029 ceph-mon[31975]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-16T11:50:55.089 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:50:54 smithi029 ceph-mon[31975]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-16T11:50:55.089 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:50:54 smithi029 ceph-mon[31975]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-16T11:50:55.089 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:50:54 smithi029 ceph-mon[31975]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-16T11:50:55.089 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:50:54 smithi029 ceph-mon[31975]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-16T11:50:55.089 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:50:54 smithi029 ceph-mon[31975]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-16T11:50:55.089 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:50:54 smithi029 ceph-mon[31975]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-16T11:50:55.089 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:50:54 smithi029 ceph-mon[31975]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-16T11:50:55.089 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:50:54 smithi029 ceph-mon[31975]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-16T11:50:55.089 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:50:54 smithi029 ceph-mon[31975]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-16T11:50:55.089 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:50:54 smithi029 ceph-mon[31975]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-16T11:50:55.090 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:50:54 smithi029 ceph-mon[31975]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-16T11:50:55.090 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:50:54 smithi029 ceph-mon[31975]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-16T11:50:55.090 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:50:54 smithi029 ceph-mon[31975]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-16T11:50:55.090 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:50:54 smithi029 ceph-mon[31975]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-16T11:50:55.090 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:50:54 smithi029 ceph-mon[31975]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-16T11:50:55.090 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:50:54 smithi029 ceph-mon[31975]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-16T11:50:55.090 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:50:54 smithi029 ceph-mon[31975]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-16T11:50:55.090 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:50:54 smithi029 ceph-mon[31975]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-16T11:50:55.090 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:50:54 smithi029 ceph-mon[31975]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-16T11:50:55.090 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:50:54 smithi029 ceph-mon[31975]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-16T11:50:55.090 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:50:54 smithi029 ceph-mon[31975]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-16T11:50:55.090 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:50:54 smithi029 ceph-mon[31975]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-16T11:50:55.090 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:50:54 smithi029 ceph-mon[31975]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-16T11:50:55.091 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:50:54 smithi029 ceph-mon[31975]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-16T11:50:55.091 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:50:54 smithi029 ceph-mon[31975]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-16T11:50:55.091 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:50:54 smithi029 ceph-mon[31975]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-16T11:50:55.091 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:50:54 smithi029 ceph-mon[31975]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-16T11:50:55.091 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:50:54 smithi029 ceph-mon[31975]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-16T11:50:55.091 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:50:54 smithi029 ceph-mon[31975]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-16T11:50:55.091 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:50:54 smithi029 ceph-mon[31975]: Manager daemon x is now available 2024-03-16T11:50:55.155 INFO:journalctl@ceph.mgr.y.smithi152.stdout:Mar 16 11:50:55 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-y[37668]: 2024-03-16T11:50:55.051+0000 7f2a359eb200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-16T11:50:55.155 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:50:54 smithi152 ceph-mon[36291]: Active manager daemon x restarted 2024-03-16T11:50:55.155 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:50:54 smithi152 ceph-mon[36291]: Activating manager daemon x 2024-03-16T11:50:55.156 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:50:54 smithi152 ceph-mon[36291]: osdmap e88: 12 total, 12 up, 12 in 2024-03-16T11:50:55.156 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:50:54 smithi152 ceph-mon[36291]: mgrmap e20: x(active, starting, since 0.0116245s), standbys: z, y 2024-03-16T11:50:55.156 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:50:54 smithi152 ceph-mon[36291]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-16T11:50:55.156 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:50:54 smithi152 ceph-mon[36291]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T11:50:55.156 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:50:54 smithi152 ceph-mon[36291]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-16T11:50:55.156 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:50:54 smithi152 ceph-mon[36291]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-16T11:50:55.156 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:50:54 smithi152 ceph-mon[36291]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-16T11:50:55.156 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:50:54 smithi152 ceph-mon[36291]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-16T11:50:55.156 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:50:54 smithi152 ceph-mon[36291]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-16T11:50:55.156 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:50:54 smithi152 ceph-mon[36291]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-16T11:50:55.156 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:50:54 smithi152 ceph-mon[36291]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-16T11:50:55.156 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:50:54 smithi152 ceph-mon[36291]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-16T11:50:55.157 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:50:54 smithi152 ceph-mon[36291]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-16T11:50:55.157 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:50:54 smithi152 ceph-mon[36291]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-16T11:50:55.157 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:50:54 smithi152 ceph-mon[36291]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-16T11:50:55.157 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:50:54 smithi152 ceph-mon[36291]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-16T11:50:55.157 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:50:54 smithi152 ceph-mon[36291]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-16T11:50:55.157 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:50:54 smithi152 ceph-mon[36291]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-16T11:50:55.157 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:50:54 smithi152 ceph-mon[36291]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-16T11:50:55.157 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:50:54 smithi152 ceph-mon[36291]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-16T11:50:55.157 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:50:54 smithi152 ceph-mon[36291]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-16T11:50:55.157 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:50:54 smithi152 ceph-mon[36291]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-16T11:50:55.157 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:50:54 smithi152 ceph-mon[36291]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-16T11:50:55.157 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:50:54 smithi152 ceph-mon[36291]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-16T11:50:55.157 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:50:54 smithi152 ceph-mon[36291]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-16T11:50:55.157 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:50:54 smithi152 ceph-mon[36291]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-16T11:50:55.158 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:50:54 smithi152 ceph-mon[36291]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-16T11:50:55.158 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:50:54 smithi152 ceph-mon[36291]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-16T11:50:55.158 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:50:54 smithi152 ceph-mon[36291]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-16T11:50:55.158 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:50:54 smithi152 ceph-mon[36291]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-16T11:50:55.158 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:50:54 smithi152 ceph-mon[36291]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-16T11:50:55.158 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:50:54 smithi152 ceph-mon[36291]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-16T11:50:55.158 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:50:54 smithi152 ceph-mon[36291]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-16T11:50:55.158 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:50:54 smithi152 ceph-mon[36291]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-16T11:50:55.158 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:50:54 smithi152 ceph-mon[36291]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-16T11:50:55.158 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:50:54 smithi152 ceph-mon[36291]: Manager daemon x is now available 2024-03-16T11:50:55.451 INFO:journalctl@ceph.mgr.y.smithi152.stdout:Mar 16 11:50:55 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-y[37668]: 2024-03-16T11:50:55.211+0000 7f2a359eb200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-16T11:50:55.545 INFO:journalctl@ceph.mgr.z.smithi154.stdout:Mar 16 11:50:55 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-z[38182]: 2024-03-16T11:50:55.133+0000 7f526d7ef200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-16T11:50:55.588 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 16 11:50:55 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-x[32210]: 2024-03-16T11:50:55.102+0000 7fa168f0e700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-16T11:50:55.588 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 16 11:50:55 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-x[32210]: 2024-03-16T11:50:55.102+0000 7fa168f0e700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-16T11:50:55.588 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 16 11:50:55 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-x[32210]: 2024-03-16T11:50:55.102+0000 7fa168f0e700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-16T11:50:55.588 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 16 11:50:55 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-x[32210]: 2024-03-16T11:50:55.102+0000 7fa168f0e700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-16T11:50:55.589 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 16 11:50:55 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-x[32210]: 2024-03-16T11:50:55.102+0000 7fa168f0e700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-16T11:50:55.589 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 16 11:50:55 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-x[32210]: 2024-03-16T11:50:55.104+0000 7fa16bf14700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-16T11:50:55.589 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 16 11:50:55 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-x[32210]: 2024-03-16T11:50:55.104+0000 7fa16bf14700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-16T11:50:55.589 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 16 11:50:55 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-x[32210]: 2024-03-16T11:50:55.104+0000 7fa16bf14700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-16T11:50:55.589 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 16 11:50:55 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-x[32210]: 2024-03-16T11:50:55.104+0000 7fa16bf14700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-16T11:50:55.589 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 16 11:50:55 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-x[32210]: 2024-03-16T11:50:55.104+0000 7fa16bf14700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-16T11:50:55.870 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:55.868+0000 7fae0f7fe640 1 -- 172.21.15.29:0/1441403041 <== mon.0 v2:172.21.15.29:3300/0 8 ==== mgrmap(e 21) ==== 137102+0+0 (secure 0 0 0) 0x7fae100994a0 con 0x7fae20100710 2024-03-16T11:50:55.870 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:55.869+0000 7fae0f7fe640 1 --2- 172.21.15.29:0/1441403041 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7fae080511d0 0x7fae080515b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:55.870 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:55.869+0000 7fae0f7fe640 1 -- 172.21.15.29:0/1441403041 --> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) -- 0x7fae100bb740 con 0x7fae080511d0 2024-03-16T11:50:55.871 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:55.872+0000 7fae1dd9b640 1 --2- 172.21.15.29:0/1441403041 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7fae080511d0 0x7fae080515b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:55.873 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:55.874+0000 7fae1dd9b640 1 --2- 172.21.15.29:0/1441403041 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7fae080511d0 0x7fae080515b0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fae1402c7b0 tx=0x7fae14005fd0 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:50:55.877 INFO:journalctl@ceph.mgr.y.smithi152.stdout:Mar 16 11:50:55 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-y[37668]: 2024-03-16T11:50:55.452+0000 7f2a359eb200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-16T11:50:55.944 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:55.945+0000 7fae0f7fe640 1 -- 172.21.15.29:0/1441403041 <== mgr.15333 v2:172.21.15.29:6800/1188678802 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7fae100bb740 con 0x7fae080511d0 2024-03-16T11:50:55.944 INFO:teuthology.orchestra.run.smithi029.stdout:Schedule set for path / 2024-03-16T11:50:55.946 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:55.947+0000 7fae24beb640 1 -- 172.21.15.29:0/1441403041 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7fae080511d0 msgr2=0x7fae080515b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:55.946 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:55.947+0000 7fae24beb640 1 --2- 172.21.15.29:0/1441403041 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7fae080511d0 0x7fae080515b0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fae1402c7b0 tx=0x7fae14005fd0 comp rx=0 tx=0).stop 2024-03-16T11:50:55.946 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:55.947+0000 7fae24beb640 1 -- 172.21.15.29:0/1441403041 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae20100710 msgr2=0x7fae20197a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:55.946 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:55.947+0000 7fae24beb640 1 --2- 172.21.15.29:0/1441403041 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae20100710 0x7fae20197a50 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fae1000e9c0 tx=0x7fae1000ee80 comp rx=0 tx=0).stop 2024-03-16T11:50:55.946 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:55.947+0000 7fae1ed9d640 1 -- 172.21.15.29:0/1441403041 reap_dead start 2024-03-16T11:50:55.947 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:55.947+0000 7fae24beb640 1 -- 172.21.15.29:0/1441403041 shutdown_connections 2024-03-16T11:50:55.947 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:55.947+0000 7fae24beb640 1 -- 172.21.15.29:0/1441403041 >> 172.21.15.29:0/1441403041 conn(0x7fae200fbca0 msgr2=0x7fae20104ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:50:55.947 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:55.947+0000 7fae24beb640 1 -- 172.21.15.29:0/1441403041 shutdown_connections 2024-03-16T11:50:55.947 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:55.947+0000 7fae24beb640 1 -- 172.21.15.29:0/1441403041 wait complete. 2024-03-16T11:50:55.957 DEBUG:teuthology.orchestra.run.smithi029:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule retention add --fs=cephfs --path=/ --retention-spec-or-period=6m3h' 2024-03-16T11:50:56.067 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:56.065+0000 7f367988c640 1 -- 172.21.15.29:0/3958579935 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f36741022b0 msgr2=0x7f3674108b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:56.067 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:56.065+0000 7f367988c640 1 --2- 172.21.15.29:0/3958579935 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f36741022b0 0x7f3674108b40 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f366c009ca0 tx=0x7f366c02c0f0 comp rx=0 tx=0).stop 2024-03-16T11:50:56.067 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:56.065+0000 7f367988c640 1 -- 172.21.15.29:0/3958579935 shutdown_connections 2024-03-16T11:50:56.067 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:56.065+0000 7f367988c640 1 --2- 172.21.15.29:0/3958579935 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f36741022b0 0x7f3674108b40 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:56.067 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:56.065+0000 7f367988c640 1 --2- 172.21.15.29:0/3958579935 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f36741018f0 0x7f3674101d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:56.067 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:56.065+0000 7f367988c640 1 --2- 172.21.15.29:0/3958579935 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3674100700 0x7f3674100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:56.067 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:56.065+0000 7f367988c640 1 -- 172.21.15.29:0/3958579935 >> 172.21.15.29:0/3958579935 conn(0x7f36740fbc70 msgr2=0x7f36740fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:50:56.067 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:56.066+0000 7f367988c640 1 -- 172.21.15.29:0/3958579935 shutdown_connections 2024-03-16T11:50:56.067 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:56.066+0000 7f367988c640 1 -- 172.21.15.29:0/3958579935 wait complete. 2024-03-16T11:50:56.067 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:56.066+0000 7f367988c640 1 Processor -- start 2024-03-16T11:50:56.067 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:56.066+0000 7f367988c640 1 -- start start 2024-03-16T11:50:56.067 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:56.066+0000 7f367988c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3674100700 0x7f36740699d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:56.068 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:56.066+0000 7f367988c640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f36741018f0 0x7f367419c790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:56.068 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:56.066+0000 7f367988c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f36741022b0 0x7f367419edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:56.068 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:56.066+0000 7f367988c640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f367410c380 con 0x7f36741022b0 2024-03-16T11:50:56.068 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:56.066+0000 7f367988c640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f367410c200 con 0x7f36741018f0 2024-03-16T11:50:56.068 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:56.066+0000 7f367988c640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f367410c500 con 0x7f3674100700 2024-03-16T11:50:56.068 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:56.066+0000 7f36737fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f36741022b0 0x7f367419edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:56.068 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:56.067+0000 7f36727fc640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f36741018f0 0x7f367419c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:56.068 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:56.067+0000 7f36727fc640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f36741018f0 0x7f367419c790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.29:50416/0 (socket says 172.21.15.29:50416) 2024-03-16T11:50:56.068 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:56.067+0000 7f36737fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f36741022b0 0x7f367419edb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39716/0 (socket says 172.21.15.29:39716) 2024-03-16T11:50:56.068 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:56.067+0000 7f3672ffd640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3674100700 0x7f36740699d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:56.068 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:56.067+0000 7f36727fc640 1 -- 172.21.15.29:0/4050989225 learned_addr learned my addr 172.21.15.29:0/4050989225 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:50:56.068 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:56.067+0000 7f36737fe640 1 -- 172.21.15.29:0/4050989225 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3674100700 msgr2=0x7f36740699d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:56.068 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:56.067+0000 7f36737fe640 1 --2- 172.21.15.29:0/4050989225 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3674100700 0x7f36740699d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:56.068 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:56.067+0000 7f36737fe640 1 -- 172.21.15.29:0/4050989225 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f36741018f0 msgr2=0x7f367419c790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:56.068 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:56.067+0000 7f36737fe640 1 --2- 172.21.15.29:0/4050989225 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f36741018f0 0x7f367419c790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:56.068 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:56.067+0000 7f36737fe640 1 -- 172.21.15.29:0/4050989225 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f367419f6c0 con 0x7f36741022b0 2024-03-16T11:50:56.068 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:56.067+0000 7f36727fc640 1 --2- 172.21.15.29:0/4050989225 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f36741018f0 0x7f367419c790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:50:56.069 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:56.067+0000 7f36737fe640 1 --2- 172.21.15.29:0/4050989225 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f36741022b0 0x7f367419edb0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f366c02c630 tx=0x7f366c032f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:50:56.069 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:56.067+0000 7f367888a640 1 -- 172.21.15.29:0/4050989225 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f366c041020 con 0x7f36741022b0 2024-03-16T11:50:56.069 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:56.067+0000 7f367888a640 1 -- 172.21.15.29:0/4050989225 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f366c034cd0 con 0x7f36741022b0 2024-03-16T11:50:56.069 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:56.067+0000 7f367888a640 1 -- 172.21.15.29:0/4050989225 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f366c039040 con 0x7f36741022b0 2024-03-16T11:50:56.069 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:56.067+0000 7f367988c640 1 -- 172.21.15.29:0/4050989225 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f367419f8f0 con 0x7f36741022b0 2024-03-16T11:50:56.069 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:56.067+0000 7f367988c640 1 -- 172.21.15.29:0/4050989225 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f367419fdb0 con 0x7f36741022b0 2024-03-16T11:50:56.070 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:56.068+0000 7f367988c640 1 -- 172.21.15.29:0/4050989225 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3674107a20 con 0x7f36741022b0 2024-03-16T11:50:56.070 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:56.069+0000 7f367888a640 1 -- 172.21.15.29:0/4050989225 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 21) ==== 137102+0+0 (secure 0 0 0) 0x7f366c044020 con 0x7f36741022b0 2024-03-16T11:50:56.070 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:56.070+0000 7f367888a640 1 --2- 172.21.15.29:0/4050989225 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f36600a26b0 0x7f36600a4b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:56.070 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:56.070+0000 7f3672ffd640 1 --2- 172.21.15.29:0/4050989225 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f36600a26b0 0x7f36600a4b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:56.070 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:56.070+0000 7f367888a640 1 -- 172.21.15.29:0/4050989225 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8469+0+0 (secure 0 0 0) 0x7f366c0f0530 con 0x7f36741022b0 2024-03-16T11:50:56.070 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:56.070+0000 7f3672ffd640 1 --2- 172.21.15.29:0/4050989225 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f36600a26b0 0x7f36600a4b70 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f3674101750 tx=0x7f3664005f90 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:50:56.073 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:56.075+0000 7f367888a640 1 -- 172.21.15.29:0/4050989225 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f366c04b020 con 0x7f36741022b0 2024-03-16T11:50:56.164 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:50:55 smithi029 ceph-mon[31975]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T11:50:56.164 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:50:55 smithi029 ceph-mon[31975]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T11:50:56.164 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:50:55 smithi029 ceph-mon[31975]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-16T11:50:56.164 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:50:55 smithi029 ceph-mon[31975]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-16T11:50:56.164 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:50:55 smithi029 ceph-mon[31975]: Standby manager daemon z restarted 2024-03-16T11:50:56.164 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:50:55 smithi029 ceph-mon[31975]: Standby manager daemon z started 2024-03-16T11:50:56.164 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:50:55 smithi029 ceph-mon[31975]: from='mgr.? 172.21.15.154:0/2686489766' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-16T11:50:56.164 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:50:55 smithi029 ceph-mon[31975]: from='mgr.? 172.21.15.154:0/2686489766' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-16T11:50:56.164 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:50:55 smithi029 ceph-mon[31975]: from='mgr.? 172.21.15.154:0/2686489766' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-16T11:50:56.165 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:50:55 smithi029 ceph-mon[31975]: from='mgr.? 172.21.15.154:0/2686489766' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-16T11:50:56.165 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:50:55 smithi029 ceph-mon[31975]: Standby manager daemon y restarted 2024-03-16T11:50:56.165 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:50:55 smithi029 ceph-mon[31975]: Standby manager daemon y started 2024-03-16T11:50:56.165 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:50:55 smithi029 ceph-mon[31975]: from='mgr.? 172.21.15.152:0/3823946931' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-16T11:50:56.165 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:50:55 smithi029 ceph-mon[31975]: from='mgr.? 172.21.15.152:0/3823946931' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-16T11:50:56.165 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:50:55 smithi029 ceph-mon[31975]: from='mgr.? 172.21.15.152:0/3823946931' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-16T11:50:56.165 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:50:55 smithi029 ceph-mon[31975]: from='mgr.? 172.21.15.152:0/3823946931' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-16T11:50:56.165 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:50:55 smithi029 ceph-mon[31975]: mgrmap e21: x(active, since 1.01975s), standbys: y, z 2024-03-16T11:50:56.250 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:50:55 smithi152 ceph-mon[36291]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T11:50:56.250 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:50:55 smithi152 ceph-mon[36291]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T11:50:56.250 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:50:55 smithi152 ceph-mon[36291]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-16T11:50:56.250 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:50:55 smithi152 ceph-mon[36291]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-16T11:50:56.250 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:50:55 smithi152 ceph-mon[36291]: Standby manager daemon z restarted 2024-03-16T11:50:56.250 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:50:55 smithi152 ceph-mon[36291]: Standby manager daemon z started 2024-03-16T11:50:56.250 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:50:55 smithi152 ceph-mon[36291]: from='mgr.? 172.21.15.154:0/2686489766' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-16T11:50:56.250 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:50:55 smithi152 ceph-mon[36291]: from='mgr.? 172.21.15.154:0/2686489766' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-16T11:50:56.250 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:50:55 smithi152 ceph-mon[36291]: from='mgr.? 172.21.15.154:0/2686489766' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-16T11:50:56.250 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:50:55 smithi152 ceph-mon[36291]: from='mgr.? 172.21.15.154:0/2686489766' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-16T11:50:56.250 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:50:55 smithi152 ceph-mon[36291]: Standby manager daemon y restarted 2024-03-16T11:50:56.251 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:50:55 smithi152 ceph-mon[36291]: Standby manager daemon y started 2024-03-16T11:50:56.251 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:50:55 smithi152 ceph-mon[36291]: from='mgr.? 172.21.15.152:0/3823946931' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-16T11:50:56.251 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:50:55 smithi152 ceph-mon[36291]: from='mgr.? 172.21.15.152:0/3823946931' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-16T11:50:56.251 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:50:55 smithi152 ceph-mon[36291]: from='mgr.? 172.21.15.152:0/3823946931' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-16T11:50:56.251 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:50:55 smithi152 ceph-mon[36291]: from='mgr.? 172.21.15.152:0/3823946931' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-16T11:50:56.251 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:50:55 smithi152 ceph-mon[36291]: mgrmap e21: x(active, since 1.01975s), standbys: y, z 2024-03-16T11:50:56.286 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:56.287+0000 7f367988c640 1 -- 172.21.15.29:0/4050989225 --> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] -- mgr_command(tid 0: {"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}) -- 0x7f3674061d80 con 0x7f36600a26b0 2024-03-16T11:50:56.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:50:55 smithi154 ceph-mon[36320]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T11:50:56.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:50:55 smithi154 ceph-mon[36320]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T11:50:56.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:50:55 smithi154 ceph-mon[36320]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-16T11:50:56.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:50:55 smithi154 ceph-mon[36320]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-16T11:50:56.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:50:55 smithi154 ceph-mon[36320]: Standby manager daemon z restarted 2024-03-16T11:50:56.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:50:55 smithi154 ceph-mon[36320]: Standby manager daemon z started 2024-03-16T11:50:56.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:50:55 smithi154 ceph-mon[36320]: from='mgr.? 172.21.15.154:0/2686489766' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-16T11:50:56.296 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:50:55 smithi154 ceph-mon[36320]: from='mgr.? 172.21.15.154:0/2686489766' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-16T11:50:56.296 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:50:55 smithi154 ceph-mon[36320]: from='mgr.? 172.21.15.154:0/2686489766' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-16T11:50:56.296 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:50:55 smithi154 ceph-mon[36320]: from='mgr.? 172.21.15.154:0/2686489766' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-16T11:50:56.296 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:50:55 smithi154 ceph-mon[36320]: Standby manager daemon y restarted 2024-03-16T11:50:56.296 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:50:55 smithi154 ceph-mon[36320]: Standby manager daemon y started 2024-03-16T11:50:56.296 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:50:55 smithi154 ceph-mon[36320]: from='mgr.? 172.21.15.152:0/3823946931' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-16T11:50:56.296 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:50:55 smithi154 ceph-mon[36320]: from='mgr.? 172.21.15.152:0/3823946931' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-16T11:50:56.296 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:50:55 smithi154 ceph-mon[36320]: from='mgr.? 172.21.15.152:0/3823946931' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-16T11:50:56.296 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:50:55 smithi154 ceph-mon[36320]: from='mgr.? 172.21.15.152:0/3823946931' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-16T11:50:56.296 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:50:55 smithi154 ceph-mon[36320]: mgrmap e21: x(active, since 1.01975s), standbys: y, z 2024-03-16T11:50:56.340 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:56.341+0000 7f367888a640 1 -- 172.21.15.29:0/4050989225 <== mgr.15333 v2:172.21.15.29:6800/1188678802 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+26 (secure 0 0 0) 0x7f3674061d80 con 0x7f36600a26b0 2024-03-16T11:50:56.340 INFO:teuthology.orchestra.run.smithi029.stdout:Retention added to path / 2024-03-16T11:50:56.342 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:56.343+0000 7f367988c640 1 -- 172.21.15.29:0/4050989225 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f36600a26b0 msgr2=0x7f36600a4b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:56.342 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:56.343+0000 7f367988c640 1 --2- 172.21.15.29:0/4050989225 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f36600a26b0 0x7f36600a4b70 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f3674101750 tx=0x7f3664005f90 comp rx=0 tx=0).stop 2024-03-16T11:50:56.342 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:56.343+0000 7f367988c640 1 -- 172.21.15.29:0/4050989225 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f36741022b0 msgr2=0x7f367419edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:56.342 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:56.343+0000 7f367988c640 1 --2- 172.21.15.29:0/4050989225 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f36741022b0 0x7f367419edb0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f366c02c630 tx=0x7f366c032f20 comp rx=0 tx=0).stop 2024-03-16T11:50:56.343 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:56.343+0000 7f367988c640 1 -- 172.21.15.29:0/4050989225 shutdown_connections 2024-03-16T11:50:56.343 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:56.343+0000 7f367988c640 1 --2- 172.21.15.29:0/4050989225 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f36600a26b0 0x7f36600a4b70 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:56.343 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:56.343+0000 7f367988c640 1 --2- 172.21.15.29:0/4050989225 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f36741022b0 0x7f367419edb0 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:56.343 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:56.343+0000 7f367988c640 1 --2- 172.21.15.29:0/4050989225 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f36741018f0 0x7f367419c790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:56.343 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:56.343+0000 7f367988c640 1 --2- 172.21.15.29:0/4050989225 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3674100700 0x7f36740699d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:56.343 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:56.343+0000 7f367988c640 1 -- 172.21.15.29:0/4050989225 >> 172.21.15.29:0/4050989225 conn(0x7f36740fbc70 msgr2=0x7f3674104bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:50:56.343 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:56.343+0000 7f367988c640 1 -- 172.21.15.29:0/4050989225 shutdown_connections 2024-03-16T11:50:56.343 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:56.343+0000 7f367988c640 1 -- 172.21.15.29:0/4050989225 wait complete. 2024-03-16T11:50:56.353 DEBUG:teuthology.orchestra.run.smithi029:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule status --fs=cephfs --path=/' 2024-03-16T11:50:56.461 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:56.460+0000 7f966fb16640 1 -- 172.21.15.29:0/1593377717 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9668100720 msgr2=0x7f9668100b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:56.461 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:56.460+0000 7f966fb16640 1 --2- 172.21.15.29:0/1593377717 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9668100720 0x7f9668100b40 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f9658009ca0 tx=0x7f965802c0f0 comp rx=0 tx=0).stop 2024-03-16T11:50:56.461 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:56.460+0000 7f966fb16640 1 -- 172.21.15.29:0/1593377717 shutdown_connections 2024-03-16T11:50:56.462 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:56.460+0000 7f966fb16640 1 --2- 172.21.15.29:0/1593377717 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f96681022d0 0x7f9668108b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:56.462 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:56.460+0000 7f966fb16640 1 --2- 172.21.15.29:0/1593377717 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9668101910 0x7f9668101d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:56.462 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:56.460+0000 7f966fb16640 1 --2- 172.21.15.29:0/1593377717 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9668100720 0x7f9668100b40 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:56.462 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:56.460+0000 7f966fb16640 1 -- 172.21.15.29:0/1593377717 >> 172.21.15.29:0/1593377717 conn(0x7f96680fbc90 msgr2=0x7f96680fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:50:56.462 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:56.460+0000 7f966fb16640 1 -- 172.21.15.29:0/1593377717 shutdown_connections 2024-03-16T11:50:56.462 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:56.460+0000 7f966fb16640 1 -- 172.21.15.29:0/1593377717 wait complete. 2024-03-16T11:50:56.462 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:56.461+0000 7f966fb16640 1 Processor -- start 2024-03-16T11:50:56.462 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:56.461+0000 7f966fb16640 1 -- start start 2024-03-16T11:50:56.462 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:56.461+0000 7f966fb16640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9668100720 0x7f9668197a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:56.462 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:56.461+0000 7f966fb16640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9668101910 0x7f9668197fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:56.462 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:56.461+0000 7f966fb16640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f96681022d0 0x7f966819f020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:56.462 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:56.461+0000 7f966fb16640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f966810c370 con 0x7f9668100720 2024-03-16T11:50:56.462 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:56.461+0000 7f966fb16640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f966810c1f0 con 0x7f9668101910 2024-03-16T11:50:56.463 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:56.461+0000 7f966fb16640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f966810c4f0 con 0x7f96681022d0 2024-03-16T11:50:56.463 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:56.461+0000 7f966d8b2640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9668100720 0x7f9668197a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:56.463 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:56.461+0000 7f966d8b2640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9668100720 0x7f9668197a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39732/0 (socket says 172.21.15.29:39732) 2024-03-16T11:50:56.463 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:56.461+0000 7f966d8b2640 1 -- 172.21.15.29:0/1548251595 learned_addr learned my addr 172.21.15.29:0/1548251595 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:50:56.463 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:56.461+0000 7f966d0b1640 1 --2- 172.21.15.29:0/1548251595 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9668101910 0x7f9668197fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:56.463 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:56.461+0000 7f966e0b3640 1 --2- 172.21.15.29:0/1548251595 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f96681022d0 0x7f966819f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:56.463 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:56.461+0000 7f966d8b2640 1 -- 172.21.15.29:0/1548251595 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f96681022d0 msgr2=0x7f966819f020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:56.463 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:56.461+0000 7f966d8b2640 1 --2- 172.21.15.29:0/1548251595 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f96681022d0 0x7f966819f020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:56.463 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:56.461+0000 7f966d8b2640 1 -- 172.21.15.29:0/1548251595 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9668101910 msgr2=0x7f9668197fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:56.463 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:56.461+0000 7f966d8b2640 1 --2- 172.21.15.29:0/1548251595 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9668101910 0x7f9668197fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:56.463 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:56.461+0000 7f966d8b2640 1 -- 172.21.15.29:0/1548251595 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f966819f690 con 0x7f9668100720 2024-03-16T11:50:56.463 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:56.461+0000 7f966d8b2640 1 --2- 172.21.15.29:0/1548251595 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9668100720 0x7f9668197a70 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f96580324c0 tx=0x7f9658032f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:50:56.463 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:56.462+0000 7f9666ffd640 1 -- 172.21.15.29:0/1548251595 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9658041020 con 0x7f9668100720 2024-03-16T11:50:56.463 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:56.462+0000 7f9666ffd640 1 -- 172.21.15.29:0/1548251595 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9658034cd0 con 0x7f9668100720 2024-03-16T11:50:56.463 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:56.462+0000 7f966fb16640 1 -- 172.21.15.29:0/1548251595 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f966819f920 con 0x7f9668100720 2024-03-16T11:50:56.464 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:56.462+0000 7f9666ffd640 1 -- 172.21.15.29:0/1548251595 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9658039040 con 0x7f9668100720 2024-03-16T11:50:56.464 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:56.462+0000 7f966fb16640 1 -- 172.21.15.29:0/1548251595 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f966819fe10 con 0x7f9668100720 2024-03-16T11:50:56.464 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:56.462+0000 7f966fb16640 1 -- 172.21.15.29:0/1548251595 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f96681a0040 con 0x7f9668100720 2024-03-16T11:50:56.464 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:56.464+0000 7f9666ffd640 1 -- 172.21.15.29:0/1548251595 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 21) ==== 137102+0+0 (secure 0 0 0) 0x7f9658044020 con 0x7f9668100720 2024-03-16T11:50:56.465 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:56.464+0000 7f9666ffd640 1 --2- 172.21.15.29:0/1548251595 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f96500a26b0 0x7f96500a4b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:56.465 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:56.464+0000 7f9666ffd640 1 -- 172.21.15.29:0/1548251595 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8469+0+0 (secure 0 0 0) 0x7f96580f09e0 con 0x7f9668100720 2024-03-16T11:50:56.465 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:56.464+0000 7f966d0b1640 1 --2- 172.21.15.29:0/1548251595 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f96500a26b0 0x7f96500a4b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:56.465 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:56.465+0000 7f966d0b1640 1 --2- 172.21.15.29:0/1548251595 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f96500a26b0 0x7f96500a4b70 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f965c0095f0 tx=0x7f965c005f90 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:50:56.467 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:56.468+0000 7f9666ffd640 1 -- 172.21.15.29:0/1548251595 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f965804b020 con 0x7f9668100720 2024-03-16T11:50:56.686 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:56.687+0000 7f966fb16640 1 -- 172.21.15.29:0/1548251595 --> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}) -- 0x7f96680fdda0 con 0x7f96500a26b0 2024-03-16T11:50:56.694 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:56.696+0000 7f9666ffd640 1 -- 172.21.15.29:0/1548251595 <== mgr.15333 v2:172.21.15.29:6800/1188678802 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+298 (secure 0 0 0) 0x7f96680fdda0 con 0x7f96500a26b0 2024-03-16T11:50:56.695 INFO:teuthology.orchestra.run.smithi029.stdout:{"fs": "cephfs", "subvol": null, "group": null, "path": "/", "rel_path": "/", "schedule": "1m", "retention": {"m": 6, "h": 3}, "start": "2024-03-16T00:00:00", "created": "2024-03-16T11:50:55", "first": null, "last": null, "last_pruned": null, "created_count": 0, "pruned_count": 0, "active": true} 2024-03-16T11:50:56.698 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:56.698+0000 7f966fb16640 1 -- 172.21.15.29:0/1548251595 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f96500a26b0 msgr2=0x7f96500a4b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:56.698 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:56.698+0000 7f966fb16640 1 --2- 172.21.15.29:0/1548251595 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f96500a26b0 0x7f96500a4b70 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f965c0095f0 tx=0x7f965c005f90 comp rx=0 tx=0).stop 2024-03-16T11:50:56.698 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:56.698+0000 7f966fb16640 1 -- 172.21.15.29:0/1548251595 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9668100720 msgr2=0x7f9668197a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:56.698 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:56.698+0000 7f966fb16640 1 --2- 172.21.15.29:0/1548251595 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9668100720 0x7f9668197a70 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f96580324c0 tx=0x7f9658032f40 comp rx=0 tx=0).stop 2024-03-16T11:50:56.698 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:56.699+0000 7f966fb16640 1 -- 172.21.15.29:0/1548251595 shutdown_connections 2024-03-16T11:50:56.698 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:56.699+0000 7f966fb16640 1 --2- 172.21.15.29:0/1548251595 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f96500a26b0 0x7f96500a4b70 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:56.698 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:56.699+0000 7f966fb16640 1 --2- 172.21.15.29:0/1548251595 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f96681022d0 0x7f966819f020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:56.698 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:56.699+0000 7f966fb16640 1 --2- 172.21.15.29:0/1548251595 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9668101910 0x7f9668197fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:56.698 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:56.699+0000 7f966fb16640 1 --2- 172.21.15.29:0/1548251595 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9668100720 0x7f9668197a70 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:56.699 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:56.699+0000 7f966fb16640 1 -- 172.21.15.29:0/1548251595 >> 172.21.15.29:0/1548251595 conn(0x7f96680fbc90 msgr2=0x7f96680fd690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:50:56.699 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:56.699+0000 7f966fb16640 1 -- 172.21.15.29:0/1548251595 shutdown_connections 2024-03-16T11:50:56.699 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:56.699+0000 7f966fb16640 1 -- 172.21.15.29:0/1548251595 wait complete. 2024-03-16T11:50:56.710 DEBUG:teuthology.orchestra.run.smithi029:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true' 2024-03-16T11:50:56.823 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:56.822+0000 7fa5732d1640 1 -- 172.21.15.29:0/2053807589 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa56c1018f0 msgr2=0x7fa56c101d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:56.823 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:56.822+0000 7fa5732d1640 1 --2- 172.21.15.29:0/2053807589 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa56c1018f0 0x7fa56c101d70 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fa55c00b880 tx=0x7fa55c005f90 comp rx=0 tx=0).stop 2024-03-16T11:50:56.823 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:56.822+0000 7fa5732d1640 1 -- 172.21.15.29:0/2053807589 shutdown_connections 2024-03-16T11:50:56.823 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:56.822+0000 7fa5732d1640 1 --2- 172.21.15.29:0/2053807589 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa56c1022b0 0x7fa56c108b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:56.823 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:56.822+0000 7fa5732d1640 1 --2- 172.21.15.29:0/2053807589 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa56c1018f0 0x7fa56c101d70 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:56.824 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:56.822+0000 7fa5732d1640 1 --2- 172.21.15.29:0/2053807589 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa56c100700 0x7fa56c100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:56.824 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:56.822+0000 7fa5732d1640 1 -- 172.21.15.29:0/2053807589 >> 172.21.15.29:0/2053807589 conn(0x7fa56c0fbc70 msgr2=0x7fa56c0fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:50:56.824 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:56.822+0000 7fa5732d1640 1 -- 172.21.15.29:0/2053807589 shutdown_connections 2024-03-16T11:50:56.824 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:56.822+0000 7fa5732d1640 1 -- 172.21.15.29:0/2053807589 wait complete. 2024-03-16T11:50:56.824 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:56.823+0000 7fa5732d1640 1 Processor -- start 2024-03-16T11:50:56.824 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:56.823+0000 7fa5732d1640 1 -- start start 2024-03-16T11:50:56.824 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:56.823+0000 7fa5732d1640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa56c100700 0x7fa56c195880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:56.824 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:56.823+0000 7fa5732d1640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa56c1018f0 0x7fa56c195dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:56.824 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:56.823+0000 7fa5732d1640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa56c1022b0 0x7fa56c19ce30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:56.824 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:56.823+0000 7fa5732d1640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fa56c10b280 con 0x7fa56c100700 2024-03-16T11:50:56.824 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:56.823+0000 7fa5732d1640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fa56c10b100 con 0x7fa56c1018f0 2024-03-16T11:50:56.824 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:56.823+0000 7fa5732d1640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fa56c10b400 con 0x7fa56c1022b0 2024-03-16T11:50:56.824 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:56.823+0000 7fa57106d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa56c100700 0x7fa56c195880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:56.824 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:56.823+0000 7fa57106d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa56c100700 0x7fa56c195880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39748/0 (socket says 172.21.15.29:39748) 2024-03-16T11:50:56.824 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:56.823+0000 7fa57086c640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa56c1018f0 0x7fa56c195dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:56.825 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:56.823+0000 7fa57106d640 1 -- 172.21.15.29:0/85911154 learned_addr learned my addr 172.21.15.29:0/85911154 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:50:56.825 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:56.823+0000 7fa57186e640 1 --2- 172.21.15.29:0/85911154 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa56c1022b0 0x7fa56c19ce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:56.825 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:56.823+0000 7fa57106d640 1 -- 172.21.15.29:0/85911154 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa56c1022b0 msgr2=0x7fa56c19ce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:56.825 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:56.823+0000 7fa57106d640 1 --2- 172.21.15.29:0/85911154 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa56c1022b0 0x7fa56c19ce30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:56.825 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:56.823+0000 7fa57106d640 1 -- 172.21.15.29:0/85911154 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa56c1018f0 msgr2=0x7fa56c195dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:56.825 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:56.823+0000 7fa57106d640 1 --2- 172.21.15.29:0/85911154 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa56c1018f0 0x7fa56c195dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:56.825 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:56.823+0000 7fa57106d640 1 -- 172.21.15.29:0/85911154 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa56c19d4a0 con 0x7fa56c100700 2024-03-16T11:50:56.825 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:56.824+0000 7fa57106d640 1 --2- 172.21.15.29:0/85911154 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa56c100700 0x7fa56c195880 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fa56800e9f0 tx=0x7fa56800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:50:56.825 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:56.824+0000 7fa5627fc640 1 -- 172.21.15.29:0/85911154 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa56800ccc0 con 0x7fa56c100700 2024-03-16T11:50:56.825 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:56.824+0000 7fa5732d1640 1 -- 172.21.15.29:0/85911154 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa56c19d790 con 0x7fa56c100700 2024-03-16T11:50:56.825 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:56.824+0000 7fa5627fc640 1 -- 172.21.15.29:0/85911154 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa5680035f0 con 0x7fa56c100700 2024-03-16T11:50:56.825 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:56.824+0000 7fa5627fc640 1 -- 172.21.15.29:0/85911154 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa56800ce60 con 0x7fa56c100700 2024-03-16T11:50:56.825 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:56.824+0000 7fa5732d1640 1 -- 172.21.15.29:0/85911154 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa56c19dd00 con 0x7fa56c100700 2024-03-16T11:50:56.826 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:56.825+0000 7fa5732d1640 1 -- 172.21.15.29:0/85911154 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa56c04f540 con 0x7fa56c100700 2024-03-16T11:50:56.826 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:56.826+0000 7fa5627fc640 1 -- 172.21.15.29:0/85911154 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 21) ==== 137102+0+0 (secure 0 0 0) 0x7fa5680037a0 con 0x7fa56c100700 2024-03-16T11:50:56.827 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:56.827+0000 7fa5627fc640 1 --2- 172.21.15.29:0/85911154 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7fa5580a26b0 0x7fa5580a4b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:50:56.827 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:56.827+0000 7fa5627fc640 1 -- 172.21.15.29:0/85911154 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8469+0+0 (secure 0 0 0) 0x7fa5680ce9b0 con 0x7fa56c100700 2024-03-16T11:50:56.827 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:56.827+0000 7fa57086c640 1 --2- 172.21.15.29:0/85911154 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7fa5580a26b0 0x7fa5580a4b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:50:56.827 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:56.827+0000 7fa57086c640 1 --2- 172.21.15.29:0/85911154 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7fa5580a26b0 0x7fa5580a4b70 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fa55c02bcf0 tx=0x7fa55c00a830 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:50:56.831 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:56.833+0000 7fa5627fc640 1 -- 172.21.15.29:0/85911154 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fa56809bc50 con 0x7fa56c100700 2024-03-16T11:50:57.055 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:57.056+0000 7fa5732d1640 1 -- 172.21.15.29:0/85911154 --> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}) -- 0x7fa56c061c20 con 0x7fa5580a26b0 2024-03-16T11:50:57.069 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:57.070+0000 7fa5627fc640 1 -- 172.21.15.29:0/85911154 <== mgr.15333 v2:172.21.15.29:6800/1188678802 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+10 (secure 0 0 0) 0x7fa56c061c20 con 0x7fa5580a26b0 2024-03-16T11:50:57.070 INFO:teuthology.orchestra.run.smithi029.stdout:/ 1m 6m3h 2024-03-16T11:50:57.071 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:57.072+0000 7fa5732d1640 1 -- 172.21.15.29:0/85911154 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7fa5580a26b0 msgr2=0x7fa5580a4b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:57.071 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:57.072+0000 7fa5732d1640 1 --2- 172.21.15.29:0/85911154 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7fa5580a26b0 0x7fa5580a4b70 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fa55c02bcf0 tx=0x7fa55c00a830 comp rx=0 tx=0).stop 2024-03-16T11:50:57.072 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:57.072+0000 7fa5732d1640 1 -- 172.21.15.29:0/85911154 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa56c100700 msgr2=0x7fa56c195880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:50:57.072 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:57.072+0000 7fa5732d1640 1 --2- 172.21.15.29:0/85911154 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa56c100700 0x7fa56c195880 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fa56800e9f0 tx=0x7fa56800eeb0 comp rx=0 tx=0).stop 2024-03-16T11:50:57.072 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:57.072+0000 7fa5732d1640 1 -- 172.21.15.29:0/85911154 shutdown_connections 2024-03-16T11:50:57.072 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:57.072+0000 7fa5732d1640 1 --2- 172.21.15.29:0/85911154 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7fa5580a26b0 0x7fa5580a4b70 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:57.072 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:57.072+0000 7fa5732d1640 1 --2- 172.21.15.29:0/85911154 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa56c1022b0 0x7fa56c19ce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:57.072 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:57.072+0000 7fa5732d1640 1 --2- 172.21.15.29:0/85911154 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa56c1018f0 0x7fa56c195dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:57.072 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:57.072+0000 7fa5732d1640 1 --2- 172.21.15.29:0/85911154 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa56c100700 0x7fa56c195880 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:50:57.072 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:57.072+0000 7fa5732d1640 1 -- 172.21.15.29:0/85911154 >> 172.21.15.29:0/85911154 conn(0x7fa56c0fbc70 msgr2=0x7fa56c104bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:50:57.072 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:57.072+0000 7fa5732d1640 1 -- 172.21.15.29:0/85911154 shutdown_connections 2024-03-16T11:50:57.072 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T11:50:57.072+0000 7fa5732d1640 1 -- 172.21.15.29:0/85911154 wait complete. 2024-03-16T11:50:57.083 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:50:56 smithi029 ceph-mon[31975]: [16/Mar/2024:11:50:56] ENGINE Bus STARTING 2024-03-16T11:50:57.083 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:50:56 smithi029 ceph-mon[31975]: [16/Mar/2024:11:50:56] ENGINE Serving on http://172.21.15.29:8765 2024-03-16T11:50:57.083 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:50:56 smithi029 ceph-mon[31975]: [16/Mar/2024:11:50:56] ENGINE Serving on https://172.21.15.29:7150 2024-03-16T11:50:57.083 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:50:56 smithi029 ceph-mon[31975]: [16/Mar/2024:11:50:56] ENGINE Bus STARTED 2024-03-16T11:50:57.083 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:50:56 smithi029 ceph-mon[31975]: from='client.15390 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T11:50:57.083 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:50:56 smithi029 ceph-mon[31975]: from='client.15396 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T11:50:57.084 DEBUG:teuthology.orchestra.run.smithi029:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'date +%s > START_TIME' 2024-03-16T11:50:57.112 INFO:teuthology.run_tasks:Running task full_sequential_finally... 2024-03-16T11:50:57.122 INFO:teuthology.run_tasks:Running task background_exec... 2024-03-16T11:50:57.130 INFO:teuthology.task.background_exec:Running background command on role mon.a host ubuntu@smithi029.front.sepia.ceph.com 2024-03-16T11:50:57.130 DEBUG:teuthology.orchestra.run.smithi029:> sudo TESTDIR=/home/ubuntu/cephtest daemon-helper kill --kill-group bash -c 'while sleep 13; do ceph tell mds.cephfs:0 flush journal; done' 2024-03-16T11:50:57.156 INFO:teuthology.run_tasks:Running task workunit... 2024-03-16T11:50:57.167 INFO:tasks.workunit:Pulling workunits from ref 3d78f5ba07bf18880172c07d4d7b4ac00c9d30af 2024-03-16T11:50:57.167 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-03-16T11:50:57.167 INFO:tasks.workunit:timeout=3h 2024-03-16T11:50:57.167 INFO:tasks.workunit:cleanup=True 2024-03-16T11:50:57.168 DEBUG:teuthology.orchestra.run.smithi029:> stat -- /home/ubuntu/cephtest/mnt.0 2024-03-16T11:50:57.183 INFO:teuthology.orchestra.run.smithi029.stdout: File: /home/ubuntu/cephtest/mnt.0 2024-03-16T11:50:57.183 INFO:teuthology.orchestra.run.smithi029.stdout: Size: 0 Blocks: 0 IO Block: 65536 directory 2024-03-16T11:50:57.183 INFO:teuthology.orchestra.run.smithi029.stdout:Device: 77h/119d Inode: 1099511628283 Links: 2 2024-03-16T11:50:57.183 INFO:teuthology.orchestra.run.smithi029.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-16T11:50:57.183 INFO:teuthology.orchestra.run.smithi029.stdout:Context: system_u:object_r:unlabeled_t:s0 2024-03-16T11:50:57.183 INFO:teuthology.orchestra.run.smithi029.stdout:Access: 2024-03-16 11:48:44.397967730 +0000 2024-03-16T11:50:57.183 INFO:teuthology.orchestra.run.smithi029.stdout:Modify: 2024-03-16 11:48:44.397967730 +0000 2024-03-16T11:50:57.183 INFO:teuthology.orchestra.run.smithi029.stdout:Change: 2024-03-16 11:50:45.278967236 +0000 2024-03-16T11:50:57.183 INFO:teuthology.orchestra.run.smithi029.stdout: Birth: 2024-03-16 11:48:44.397967730 +0000 2024-03-16T11:50:57.184 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2024-03-16T11:50:57.184 DEBUG:teuthology.orchestra.run.smithi029:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2024-03-16T11:50:57.258 DEBUG:teuthology.orchestra.run.smithi029:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 3d78f5ba07bf18880172c07d4d7b4ac00c9d30af 2024-03-16T11:50:57.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:50:56 smithi154 ceph-mon[36320]: [16/Mar/2024:11:50:56] ENGINE Bus STARTING 2024-03-16T11:50:57.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:50:56 smithi154 ceph-mon[36320]: [16/Mar/2024:11:50:56] ENGINE Serving on http://172.21.15.29:8765 2024-03-16T11:50:57.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:50:56 smithi154 ceph-mon[36320]: [16/Mar/2024:11:50:56] ENGINE Serving on https://172.21.15.29:7150 2024-03-16T11:50:57.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:50:56 smithi154 ceph-mon[36320]: [16/Mar/2024:11:50:56] ENGINE Bus STARTED 2024-03-16T11:50:57.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:50:56 smithi154 ceph-mon[36320]: from='client.15390 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T11:50:57.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:50:56 smithi154 ceph-mon[36320]: from='client.15396 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T11:50:57.313 INFO:tasks.workunit.client.0.smithi029.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-03-16T11:50:57.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:50:56 smithi152 ceph-mon[36291]: [16/Mar/2024:11:50:56] ENGINE Bus STARTING 2024-03-16T11:50:57.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:50:56 smithi152 ceph-mon[36291]: [16/Mar/2024:11:50:56] ENGINE Serving on http://172.21.15.29:8765 2024-03-16T11:50:57.378 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:50:56 smithi152 ceph-mon[36291]: [16/Mar/2024:11:50:56] ENGINE Serving on https://172.21.15.29:7150 2024-03-16T11:50:57.378 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:50:56 smithi152 ceph-mon[36291]: [16/Mar/2024:11:50:56] ENGINE Bus STARTED 2024-03-16T11:50:57.378 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:50:56 smithi152 ceph-mon[36291]: from='client.15390 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T11:50:57.378 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:50:56 smithi152 ceph-mon[36291]: from='client.15396 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T11:50:58.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:50:58 smithi029 ceph-mon[31975]: pgmap v4: 97 pgs: 97 active+clean; 1.1 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T11:50:58.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:50:58 smithi029 ceph-mon[31975]: from='client.15402 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T11:50:58.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:50:58 smithi029 ceph-mon[31975]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' 2024-03-16T11:50:58.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:50:58 smithi029 ceph-mon[31975]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' 2024-03-16T11:50:58.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:50:58 smithi029 ceph-mon[31975]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' 2024-03-16T11:50:58.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:50:58 smithi029 ceph-mon[31975]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' 2024-03-16T11:50:58.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:50:58 smithi029 ceph-mon[31975]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' 2024-03-16T11:50:58.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:50:58 smithi029 ceph-mon[31975]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' 2024-03-16T11:50:58.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:50:58 smithi029 ceph-mon[31975]: mgrmap e22: x(active, since 3s), standbys: y, z 2024-03-16T11:50:58.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:50:58 smithi152 ceph-mon[36291]: pgmap v4: 97 pgs: 97 active+clean; 1.1 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T11:50:58.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:50:58 smithi152 ceph-mon[36291]: from='client.15402 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T11:50:58.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:50:58 smithi152 ceph-mon[36291]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' 2024-03-16T11:50:58.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:50:58 smithi152 ceph-mon[36291]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' 2024-03-16T11:50:58.378 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:50:58 smithi152 ceph-mon[36291]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' 2024-03-16T11:50:58.378 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:50:58 smithi152 ceph-mon[36291]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' 2024-03-16T11:50:58.378 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:50:58 smithi152 ceph-mon[36291]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' 2024-03-16T11:50:58.378 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:50:58 smithi152 ceph-mon[36291]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' 2024-03-16T11:50:58.378 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:50:58 smithi152 ceph-mon[36291]: mgrmap e22: x(active, since 3s), standbys: y, z 2024-03-16T11:50:58.545 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:50:58 smithi154 ceph-mon[36320]: pgmap v4: 97 pgs: 97 active+clean; 1.1 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T11:50:58.545 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:50:58 smithi154 ceph-mon[36320]: from='client.15402 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T11:50:58.545 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:50:58 smithi154 ceph-mon[36320]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' 2024-03-16T11:50:58.545 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:50:58 smithi154 ceph-mon[36320]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' 2024-03-16T11:50:58.545 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:50:58 smithi154 ceph-mon[36320]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' 2024-03-16T11:50:58.545 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:50:58 smithi154 ceph-mon[36320]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' 2024-03-16T11:50:58.545 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:50:58 smithi154 ceph-mon[36320]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' 2024-03-16T11:50:58.545 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:50:58 smithi154 ceph-mon[36320]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' 2024-03-16T11:50:58.545 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:50:58 smithi154 ceph-mon[36320]: mgrmap e22: x(active, since 3s), standbys: y, z 2024-03-16T11:50:59.545 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:50:59 smithi154 ceph-mon[36320]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' 2024-03-16T11:50:59.546 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:50:59 smithi154 ceph-mon[36320]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' 2024-03-16T11:50:59.546 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:50:59 smithi154 ceph-mon[36320]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-16T11:50:59.546 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:50:59 smithi154 ceph-mon[36320]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-16T11:50:59.546 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:50:59 smithi154 ceph-mon[36320]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-16T11:50:59.546 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:50:59 smithi154 ceph-mon[36320]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-16T11:50:59.546 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:50:59 smithi154 ceph-mon[36320]: Adjusting osd_memory_target on smithi029 to 177.2M 2024-03-16T11:50:59.546 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:50:59 smithi154 ceph-mon[36320]: Unable to set osd_memory_target on smithi029 to 185894912: error parsing value: Value '185894912' is below minimum 939524096 2024-03-16T11:50:59.546 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:50:59 smithi154 ceph-mon[36320]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' 2024-03-16T11:50:59.546 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:50:59 smithi154 ceph-mon[36320]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' 2024-03-16T11:50:59.546 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:50:59 smithi154 ceph-mon[36320]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-16T11:50:59.546 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:50:59 smithi154 ceph-mon[36320]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-16T11:50:59.546 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:50:59 smithi154 ceph-mon[36320]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-16T11:50:59.546 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:50:59 smithi154 ceph-mon[36320]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-16T11:50:59.547 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:50:59 smithi154 ceph-mon[36320]: Adjusting osd_memory_target on smithi154 to 174.1M 2024-03-16T11:50:59.547 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:50:59 smithi154 ceph-mon[36320]: Unable to set osd_memory_target on smithi154 to 182636339: error parsing value: Value '182636339' is below minimum 939524096 2024-03-16T11:50:59.547 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:50:59 smithi154 ceph-mon[36320]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' 2024-03-16T11:50:59.547 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:50:59 smithi154 ceph-mon[36320]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' 2024-03-16T11:50:59.547 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:50:59 smithi154 ceph-mon[36320]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-16T11:50:59.547 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:50:59 smithi154 ceph-mon[36320]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-16T11:50:59.547 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:50:59 smithi154 ceph-mon[36320]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-16T11:50:59.547 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:50:59 smithi154 ceph-mon[36320]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-16T11:50:59.547 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:50:59 smithi154 ceph-mon[36320]: Adjusting osd_memory_target on smithi152 to 174.1M 2024-03-16T11:50:59.547 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:50:59 smithi154 ceph-mon[36320]: Unable to set osd_memory_target on smithi152 to 182635622: error parsing value: Value '182635622' is below minimum 939524096 2024-03-16T11:50:59.547 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:50:59 smithi154 ceph-mon[36320]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:50:59.547 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:50:59 smithi154 ceph-mon[36320]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T11:50:59.547 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:50:59 smithi154 ceph-mon[36320]: Updating smithi029:/etc/ceph/ceph.conf 2024-03-16T11:50:59.547 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:50:59 smithi154 ceph-mon[36320]: Updating smithi152:/etc/ceph/ceph.conf 2024-03-16T11:50:59.547 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:50:59 smithi154 ceph-mon[36320]: Updating smithi154:/etc/ceph/ceph.conf 2024-03-16T11:50:59.547 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:50:59 smithi154 ceph-mon[36320]: Updating smithi152:/var/lib/ceph/d1887f28-e389-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-16T11:50:59.547 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:50:59 smithi154 ceph-mon[36320]: Updating smithi154:/var/lib/ceph/d1887f28-e389-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-16T11:50:59.548 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:50:59 smithi154 ceph-mon[36320]: Updating smithi029:/var/lib/ceph/d1887f28-e389-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-16T11:50:59.588 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:50:59 smithi029 ceph-mon[31975]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' 2024-03-16T11:50:59.588 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:50:59 smithi029 ceph-mon[31975]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' 2024-03-16T11:50:59.588 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:50:59 smithi029 ceph-mon[31975]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-16T11:50:59.588 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:50:59 smithi029 ceph-mon[31975]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-16T11:50:59.588 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:50:59 smithi029 ceph-mon[31975]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-16T11:50:59.588 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:50:59 smithi029 ceph-mon[31975]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-16T11:50:59.588 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:50:59 smithi029 ceph-mon[31975]: Adjusting osd_memory_target on smithi029 to 177.2M 2024-03-16T11:50:59.588 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:50:59 smithi029 ceph-mon[31975]: Unable to set osd_memory_target on smithi029 to 185894912: error parsing value: Value '185894912' is below minimum 939524096 2024-03-16T11:50:59.588 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:50:59 smithi029 ceph-mon[31975]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' 2024-03-16T11:50:59.589 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:50:59 smithi029 ceph-mon[31975]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' 2024-03-16T11:50:59.589 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:50:59 smithi029 ceph-mon[31975]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-16T11:50:59.589 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:50:59 smithi029 ceph-mon[31975]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-16T11:50:59.589 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:50:59 smithi029 ceph-mon[31975]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-16T11:50:59.589 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:50:59 smithi029 ceph-mon[31975]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-16T11:50:59.589 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:50:59 smithi029 ceph-mon[31975]: Adjusting osd_memory_target on smithi154 to 174.1M 2024-03-16T11:50:59.589 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:50:59 smithi029 ceph-mon[31975]: Unable to set osd_memory_target on smithi154 to 182636339: error parsing value: Value '182636339' is below minimum 939524096 2024-03-16T11:50:59.589 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:50:59 smithi029 ceph-mon[31975]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' 2024-03-16T11:50:59.589 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:50:59 smithi029 ceph-mon[31975]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' 2024-03-16T11:50:59.589 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:50:59 smithi029 ceph-mon[31975]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-16T11:50:59.589 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:50:59 smithi029 ceph-mon[31975]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-16T11:50:59.589 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:50:59 smithi029 ceph-mon[31975]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-16T11:50:59.589 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:50:59 smithi029 ceph-mon[31975]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-16T11:50:59.589 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:50:59 smithi029 ceph-mon[31975]: Adjusting osd_memory_target on smithi152 to 174.1M 2024-03-16T11:50:59.589 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:50:59 smithi029 ceph-mon[31975]: Unable to set osd_memory_target on smithi152 to 182635622: error parsing value: Value '182635622' is below minimum 939524096 2024-03-16T11:50:59.589 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:50:59 smithi029 ceph-mon[31975]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:50:59.590 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:50:59 smithi029 ceph-mon[31975]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T11:50:59.590 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:50:59 smithi029 ceph-mon[31975]: Updating smithi029:/etc/ceph/ceph.conf 2024-03-16T11:50:59.590 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:50:59 smithi029 ceph-mon[31975]: Updating smithi152:/etc/ceph/ceph.conf 2024-03-16T11:50:59.590 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:50:59 smithi029 ceph-mon[31975]: Updating smithi154:/etc/ceph/ceph.conf 2024-03-16T11:50:59.590 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:50:59 smithi029 ceph-mon[31975]: Updating smithi152:/var/lib/ceph/d1887f28-e389-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-16T11:50:59.590 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:50:59 smithi029 ceph-mon[31975]: Updating smithi154:/var/lib/ceph/d1887f28-e389-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-16T11:50:59.590 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:50:59 smithi029 ceph-mon[31975]: Updating smithi029:/var/lib/ceph/d1887f28-e389-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-16T11:50:59.628 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:50:59 smithi152 ceph-mon[36291]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' 2024-03-16T11:50:59.628 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:50:59 smithi152 ceph-mon[36291]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' 2024-03-16T11:50:59.628 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:50:59 smithi152 ceph-mon[36291]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-16T11:50:59.628 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:50:59 smithi152 ceph-mon[36291]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-16T11:50:59.628 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:50:59 smithi152 ceph-mon[36291]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-16T11:50:59.628 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:50:59 smithi152 ceph-mon[36291]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-16T11:50:59.628 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:50:59 smithi152 ceph-mon[36291]: Adjusting osd_memory_target on smithi029 to 177.2M 2024-03-16T11:50:59.628 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:50:59 smithi152 ceph-mon[36291]: Unable to set osd_memory_target on smithi029 to 185894912: error parsing value: Value '185894912' is below minimum 939524096 2024-03-16T11:50:59.628 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:50:59 smithi152 ceph-mon[36291]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' 2024-03-16T11:50:59.628 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:50:59 smithi152 ceph-mon[36291]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' 2024-03-16T11:50:59.628 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:50:59 smithi152 ceph-mon[36291]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-16T11:50:59.629 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:50:59 smithi152 ceph-mon[36291]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-16T11:50:59.629 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:50:59 smithi152 ceph-mon[36291]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-16T11:50:59.629 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:50:59 smithi152 ceph-mon[36291]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-16T11:50:59.629 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:50:59 smithi152 ceph-mon[36291]: Adjusting osd_memory_target on smithi154 to 174.1M 2024-03-16T11:50:59.629 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:50:59 smithi152 ceph-mon[36291]: Unable to set osd_memory_target on smithi154 to 182636339: error parsing value: Value '182636339' is below minimum 939524096 2024-03-16T11:50:59.629 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:50:59 smithi152 ceph-mon[36291]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' 2024-03-16T11:50:59.629 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:50:59 smithi152 ceph-mon[36291]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' 2024-03-16T11:50:59.629 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:50:59 smithi152 ceph-mon[36291]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-16T11:50:59.629 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:50:59 smithi152 ceph-mon[36291]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-16T11:50:59.629 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:50:59 smithi152 ceph-mon[36291]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-16T11:50:59.629 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:50:59 smithi152 ceph-mon[36291]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-16T11:50:59.629 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:50:59 smithi152 ceph-mon[36291]: Adjusting osd_memory_target on smithi152 to 174.1M 2024-03-16T11:50:59.629 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:50:59 smithi152 ceph-mon[36291]: Unable to set osd_memory_target on smithi152 to 182635622: error parsing value: Value '182635622' is below minimum 939524096 2024-03-16T11:50:59.629 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:50:59 smithi152 ceph-mon[36291]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:50:59.630 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:50:59 smithi152 ceph-mon[36291]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T11:50:59.630 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:50:59 smithi152 ceph-mon[36291]: Updating smithi029:/etc/ceph/ceph.conf 2024-03-16T11:50:59.630 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:50:59 smithi152 ceph-mon[36291]: Updating smithi152:/etc/ceph/ceph.conf 2024-03-16T11:50:59.630 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:50:59 smithi152 ceph-mon[36291]: Updating smithi154:/etc/ceph/ceph.conf 2024-03-16T11:50:59.630 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:50:59 smithi152 ceph-mon[36291]: Updating smithi152:/var/lib/ceph/d1887f28-e389-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-16T11:50:59.630 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:50:59 smithi152 ceph-mon[36291]: Updating smithi154:/var/lib/ceph/d1887f28-e389-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-16T11:50:59.630 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:50:59 smithi152 ceph-mon[36291]: Updating smithi029:/var/lib/ceph/d1887f28-e389-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-16T11:51:00.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:51:00 smithi154 ceph-mon[36320]: pgmap v5: 97 pgs: 97 active+clean; 1.1 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T11:51:00.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:51:00 smithi154 ceph-mon[36320]: Updating smithi029:/etc/ceph/ceph.client.admin.keyring 2024-03-16T11:51:00.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:51:00 smithi154 ceph-mon[36320]: Updating smithi154:/etc/ceph/ceph.client.admin.keyring 2024-03-16T11:51:00.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:51:00 smithi154 ceph-mon[36320]: Updating smithi152:/etc/ceph/ceph.client.admin.keyring 2024-03-16T11:51:00.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:51:00 smithi154 ceph-mon[36320]: Updating smithi029:/var/lib/ceph/d1887f28-e389-11ee-95c9-87774f69a715/config/ceph.client.admin.keyring 2024-03-16T11:51:00.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:51:00 smithi154 ceph-mon[36320]: Updating smithi154:/var/lib/ceph/d1887f28-e389-11ee-95c9-87774f69a715/config/ceph.client.admin.keyring 2024-03-16T11:51:00.796 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:51:00 smithi154 ceph-mon[36320]: Updating smithi152:/var/lib/ceph/d1887f28-e389-11ee-95c9-87774f69a715/config/ceph.client.admin.keyring 2024-03-16T11:51:00.796 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:51:00 smithi154 ceph-mon[36320]: mgrmap e23: x(active, since 4s), standbys: y, z 2024-03-16T11:51:00.796 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:51:00 smithi154 ceph-mon[36320]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' 2024-03-16T11:51:00.796 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:51:00 smithi154 ceph-mon[36320]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' 2024-03-16T11:51:00.796 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:51:00 smithi154 ceph-mon[36320]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' 2024-03-16T11:51:00.796 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:51:00 smithi154 ceph-mon[36320]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' 2024-03-16T11:51:00.796 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:51:00 smithi154 ceph-mon[36320]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' 2024-03-16T11:51:00.796 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:51:00 smithi154 ceph-mon[36320]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' 2024-03-16T11:51:00.796 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:51:00 smithi154 ceph-mon[36320]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' 2024-03-16T11:51:00.796 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:51:00 smithi154 ceph-mon[36320]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T11:51:00.796 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:51:00 smithi154 ceph-mon[36320]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:51:00.796 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:51:00 smithi154 ceph-mon[36320]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T11:51:00.796 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:51:00 smithi154 ceph-mon[36320]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' 2024-03-16T11:51:00.838 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 16 11:51:00 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-x[32210]: 2024-03-16T11:51:00.352+0000 7fa14a351700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-16T11:51:00.838 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 16 11:51:00 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-x[32210]: 2024-03-16T11:51:00.352+0000 7fa14a351700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-16T11:51:00.838 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 16 11:51:00 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-x[32210]: 2024-03-16T11:51:00.352+0000 7fa14a351700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-16T11:51:00.838 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 16 11:51:00 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-x[32210]: 2024-03-16T11:51:00.352+0000 7fa14a351700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-16T11:51:00.838 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 16 11:51:00 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-x[32210]: 2024-03-16T11:51:00.352+0000 7fa14a351700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-16T11:51:00.839 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:51:00 smithi029 ceph-mon[31975]: pgmap v5: 97 pgs: 97 active+clean; 1.1 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T11:51:00.839 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:51:00 smithi029 ceph-mon[31975]: Updating smithi029:/etc/ceph/ceph.client.admin.keyring 2024-03-16T11:51:00.839 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:51:00 smithi029 ceph-mon[31975]: Updating smithi154:/etc/ceph/ceph.client.admin.keyring 2024-03-16T11:51:00.839 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:51:00 smithi029 ceph-mon[31975]: Updating smithi152:/etc/ceph/ceph.client.admin.keyring 2024-03-16T11:51:00.839 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:51:00 smithi029 ceph-mon[31975]: Updating smithi029:/var/lib/ceph/d1887f28-e389-11ee-95c9-87774f69a715/config/ceph.client.admin.keyring 2024-03-16T11:51:00.839 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:51:00 smithi029 ceph-mon[31975]: Updating smithi154:/var/lib/ceph/d1887f28-e389-11ee-95c9-87774f69a715/config/ceph.client.admin.keyring 2024-03-16T11:51:00.839 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:51:00 smithi029 ceph-mon[31975]: Updating smithi152:/var/lib/ceph/d1887f28-e389-11ee-95c9-87774f69a715/config/ceph.client.admin.keyring 2024-03-16T11:51:00.839 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:51:00 smithi029 ceph-mon[31975]: mgrmap e23: x(active, since 4s), standbys: y, z 2024-03-16T11:51:00.839 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:51:00 smithi029 ceph-mon[31975]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' 2024-03-16T11:51:00.839 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:51:00 smithi029 ceph-mon[31975]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' 2024-03-16T11:51:00.839 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:51:00 smithi029 ceph-mon[31975]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' 2024-03-16T11:51:00.839 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:51:00 smithi029 ceph-mon[31975]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' 2024-03-16T11:51:00.839 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:51:00 smithi029 ceph-mon[31975]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' 2024-03-16T11:51:00.839 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:51:00 smithi029 ceph-mon[31975]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' 2024-03-16T11:51:00.840 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:51:00 smithi029 ceph-mon[31975]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' 2024-03-16T11:51:00.840 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:51:00 smithi029 ceph-mon[31975]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T11:51:00.840 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:51:00 smithi029 ceph-mon[31975]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:51:00.840 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:51:00 smithi029 ceph-mon[31975]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T11:51:00.840 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:51:00 smithi029 ceph-mon[31975]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' 2024-03-16T11:51:00.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:51:00 smithi152 ceph-mon[36291]: pgmap v5: 97 pgs: 97 active+clean; 1.1 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T11:51:00.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:51:00 smithi152 ceph-mon[36291]: Updating smithi029:/etc/ceph/ceph.client.admin.keyring 2024-03-16T11:51:00.878 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:51:00 smithi152 ceph-mon[36291]: Updating smithi154:/etc/ceph/ceph.client.admin.keyring 2024-03-16T11:51:00.878 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:51:00 smithi152 ceph-mon[36291]: Updating smithi152:/etc/ceph/ceph.client.admin.keyring 2024-03-16T11:51:00.878 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:51:00 smithi152 ceph-mon[36291]: Updating smithi029:/var/lib/ceph/d1887f28-e389-11ee-95c9-87774f69a715/config/ceph.client.admin.keyring 2024-03-16T11:51:00.878 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:51:00 smithi152 ceph-mon[36291]: Updating smithi154:/var/lib/ceph/d1887f28-e389-11ee-95c9-87774f69a715/config/ceph.client.admin.keyring 2024-03-16T11:51:00.878 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:51:00 smithi152 ceph-mon[36291]: Updating smithi152:/var/lib/ceph/d1887f28-e389-11ee-95c9-87774f69a715/config/ceph.client.admin.keyring 2024-03-16T11:51:00.878 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:51:00 smithi152 ceph-mon[36291]: mgrmap e23: x(active, since 4s), standbys: y, z 2024-03-16T11:51:00.879 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:51:00 smithi152 ceph-mon[36291]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' 2024-03-16T11:51:00.879 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:51:00 smithi152 ceph-mon[36291]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' 2024-03-16T11:51:00.879 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:51:00 smithi152 ceph-mon[36291]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' 2024-03-16T11:51:00.879 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:51:00 smithi152 ceph-mon[36291]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' 2024-03-16T11:51:00.879 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:51:00 smithi152 ceph-mon[36291]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' 2024-03-16T11:51:00.879 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:51:00 smithi152 ceph-mon[36291]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' 2024-03-16T11:51:00.879 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:51:00 smithi152 ceph-mon[36291]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' 2024-03-16T11:51:00.879 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:51:00 smithi152 ceph-mon[36291]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T11:51:00.879 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:51:00 smithi152 ceph-mon[36291]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:51:00.879 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:51:00 smithi152 ceph-mon[36291]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T11:51:00.879 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:51:00 smithi152 ceph-mon[36291]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' 2024-03-16T11:51:02.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:51:02 smithi154 ceph-mon[36320]: pgmap v6: 97 pgs: 97 active+clean; 1.1 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T11:51:02.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:51:02 smithi154 ceph-mon[36320]: mgrmap e24: x(active, since 6s), standbys: y, z 2024-03-16T11:51:02.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:51:02 smithi029 ceph-mon[31975]: pgmap v6: 97 pgs: 97 active+clean; 1.1 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T11:51:02.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:51:02 smithi029 ceph-mon[31975]: mgrmap e24: x(active, since 6s), standbys: y, z 2024-03-16T11:51:02.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:51:02 smithi152 ceph-mon[36291]: pgmap v6: 97 pgs: 97 active+clean; 1.1 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T11:51:02.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:51:02 smithi152 ceph-mon[36291]: mgrmap e24: x(active, since 6s), standbys: y, z 2024-03-16T11:51:04.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:51:04 smithi154 ceph-mon[36320]: pgmap v7: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 170 KiB/s wr, 103 op/s 2024-03-16T11:51:04.837 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:51:04 smithi029 ceph-mon[31975]: pgmap v7: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 170 KiB/s wr, 103 op/s 2024-03-16T11:51:04.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:51:04 smithi152 ceph-mon[36291]: pgmap v7: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 170 KiB/s wr, 103 op/s 2024-03-16T11:51:06.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:51:06 smithi154 ceph-mon[36320]: pgmap v8: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 133 KiB/s wr, 80 op/s 2024-03-16T11:51:06.837 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:51:06 smithi029 ceph-mon[31975]: pgmap v8: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 133 KiB/s wr, 80 op/s 2024-03-16T11:51:06.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:51:06 smithi152 ceph-mon[36291]: pgmap v8: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 133 KiB/s wr, 80 op/s 2024-03-16T11:51:08.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:51:08 smithi154 ceph-mon[36320]: pgmap v9: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 109 KiB/s wr, 68 op/s 2024-03-16T11:51:08.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:51:08 smithi029 ceph-mon[31975]: pgmap v9: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 109 KiB/s wr, 68 op/s 2024-03-16T11:51:08.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:51:08 smithi152 ceph-mon[36291]: pgmap v9: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 109 KiB/s wr, 68 op/s 2024-03-16T11:51:10.299 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:10.300+0000 7f7b5ddd4640 1 -- 172.21.15.29:0/1588402906 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7b581029e0 msgr2=0x7f7b58102dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:51:10.299 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:10.300+0000 7f7b5ddd4640 1 --2- 172.21.15.29:0/1588402906 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7b581029e0 0x7f7b58102dc0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f7b48009ca0 tx=0x7f7b4802c0f0 comp rx=0 tx=0).stop 2024-03-16T11:51:10.299 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:10.300+0000 7f7b5ddd4640 1 -- 172.21.15.29:0/1588402906 shutdown_connections 2024-03-16T11:51:10.299 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:10.300+0000 7f7b5ddd4640 1 --2- 172.21.15.29:0/1588402906 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7b5810be50 0x7f7b5810e2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:51:10.300 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:10.300+0000 7f7b5ddd4640 1 --2- 172.21.15.29:0/1588402906 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7b58103300 0x7f7b5810b7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:51:10.300 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:10.300+0000 7f7b5ddd4640 1 --2- 172.21.15.29:0/1588402906 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7b581029e0 0x7f7b58102dc0 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:51:10.300 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:10.300+0000 7f7b5ddd4640 1 -- 172.21.15.29:0/1588402906 >> 172.21.15.29:0/1588402906 conn(0x7f7b58077810 msgr2=0x7f7b58077c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:51:10.300 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:10.300+0000 7f7b5ddd4640 1 -- 172.21.15.29:0/1588402906 shutdown_connections 2024-03-16T11:51:10.300 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:10.300+0000 7f7b5ddd4640 1 -- 172.21.15.29:0/1588402906 wait complete. 2024-03-16T11:51:10.300 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:10.300+0000 7f7b5ddd4640 1 Processor -- start 2024-03-16T11:51:10.302 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:10.301+0000 7f7b5ddd4640 1 -- start start 2024-03-16T11:51:10.302 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:10.301+0000 7f7b5ddd4640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7b581029e0 0x7f7b58197f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:51:10.302 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:10.301+0000 7f7b5ddd4640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7b58103300 0x7f7b581984d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:51:10.302 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:10.301+0000 7f7b5ddd4640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7b5810be50 0x7f7b5819c850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:51:10.302 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:10.301+0000 7f7b5ddd4640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f7b58111bf0 con 0x7f7b5810be50 2024-03-16T11:51:10.302 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:10.301+0000 7f7b5ddd4640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f7b58111a70 con 0x7f7b58103300 2024-03-16T11:51:10.302 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:10.301+0000 7f7b5ddd4640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f7b58111d70 con 0x7f7b581029e0 2024-03-16T11:51:10.302 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:10.301+0000 7f7b57fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7b5810be50 0x7f7b5819c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:51:10.303 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:10.301+0000 7f7b577fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7b581029e0 0x7f7b58197f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:51:10.303 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:10.301+0000 7f7b57fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7b5810be50 0x7f7b5819c850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:40740/0 (socket says 172.21.15.29:40740) 2024-03-16T11:51:10.303 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:10.301+0000 7f7b57fff640 1 -- 172.21.15.29:0/1687128581 learned_addr learned my addr 172.21.15.29:0/1687128581 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:51:10.303 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:10.301+0000 7f7b56ffd640 1 --2- 172.21.15.29:0/1687128581 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7b58103300 0x7f7b581984d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:51:10.303 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:10.301+0000 7f7b57fff640 1 -- 172.21.15.29:0/1687128581 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7b581029e0 msgr2=0x7f7b58197f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:51:10.303 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:10.301+0000 7f7b57fff640 1 --2- 172.21.15.29:0/1687128581 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7b581029e0 0x7f7b58197f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:51:10.303 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:10.301+0000 7f7b57fff640 1 -- 172.21.15.29:0/1687128581 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7b58103300 msgr2=0x7f7b581984d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:51:10.303 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:10.301+0000 7f7b57fff640 1 --2- 172.21.15.29:0/1687128581 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7b58103300 0x7f7b581984d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:51:10.303 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:10.301+0000 7f7b57fff640 1 -- 172.21.15.29:0/1687128581 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b5819cf80 con 0x7f7b5810be50 2024-03-16T11:51:10.303 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:10.301+0000 7f7b56ffd640 1 --2- 172.21.15.29:0/1687128581 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7b58103300 0x7f7b581984d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T11:51:10.303 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:10.301+0000 7f7b577fe640 1 --2- 172.21.15.29:0/1687128581 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7b581029e0 0x7f7b58197f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:51:10.303 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:10.301+0000 7f7b57fff640 1 --2- 172.21.15.29:0/1687128581 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7b5810be50 0x7f7b5819c850 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f7b5000e970 tx=0x7f7b5000ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:51:10.303 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:10.302+0000 7f7b54ff9640 1 -- 172.21.15.29:0/1687128581 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b5000ccc0 con 0x7f7b5810be50 2024-03-16T11:51:10.303 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:10.302+0000 7f7b54ff9640 1 -- 172.21.15.29:0/1687128581 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7b5000ce60 con 0x7f7b5810be50 2024-03-16T11:51:10.303 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:10.302+0000 7f7b5ddd4640 1 -- 172.21.15.29:0/1687128581 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7b5819d270 con 0x7f7b5810be50 2024-03-16T11:51:10.304 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:10.302+0000 7f7b54ff9640 1 -- 172.21.15.29:0/1687128581 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b50010490 con 0x7f7b5810be50 2024-03-16T11:51:10.304 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:10.302+0000 7f7b5ddd4640 1 -- 172.21.15.29:0/1687128581 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7b581a4b00 con 0x7f7b5810be50 2024-03-16T11:51:10.310 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:10.304+0000 7f7b54ff9640 1 -- 172.21.15.29:0/1687128581 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 24) ==== 137523+0+0 (secure 0 0 0) 0x7f7b50005020 con 0x7f7b5810be50 2024-03-16T11:51:10.310 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:10.304+0000 7f7b609c4740 1 Processor -- start 2024-03-16T11:51:10.311 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:10.304+0000 7f7b609c4740 1 -- start start 2024-03-16T11:51:10.311 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:10.304+0000 7f7b609c4740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bc070b3130 0x55bc070b3510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:51:10.311 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:10.304+0000 7f7b609c4740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55bc070b3a50 0x55bc070bc2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:51:10.311 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:10.304+0000 7f7b609c4740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bc070bc820 0x55bc070bec10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:51:10.311 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:10.304+0000 7f7b609c4740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55bc07017a40 con 0x55bc070b3130 2024-03-16T11:51:10.311 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:10.304+0000 7f7b609c4740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x55bc06fc0920 con 0x55bc070b3a50 2024-03-16T11:51:10.311 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:10.304+0000 7f7b609c4740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55bc07013ae0 con 0x55bc070bc820 2024-03-16T11:51:10.311 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:10.305+0000 7f7b54ff9640 1 --2- 172.21.15.29:0/1687128581 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f7b400a2b30 0x7f7b400a4ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:51:10.311 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:10.305+0000 7f7b56ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bc070b3130 0x55bc070b3510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:51:10.311 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:10.305+0000 7f7b56ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bc070b3130 0x55bc070b3510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:40754/0 (socket says 172.21.15.29:40754) 2024-03-16T11:51:10.311 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:10.305+0000 7f7b56ffd640 1 -- 172.21.15.29:0/994047079 learned_addr learned my addr 172.21.15.29:0/994047079 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:51:10.311 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:10.305+0000 7f7b577fe640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55bc070b3a50 0x55bc070bc2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:51:10.311 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:10.305+0000 7f7b57fff640 1 --2- 172.21.15.29:0/1687128581 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f7b400a2b30 0x7f7b400a4ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:51:10.311 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:10.305+0000 7f7b56ffd640 1 --2- 172.21.15.29:0/994047079 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bc070bc820 0x55bc070bec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:51:10.312 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:10.305+0000 7f7b54ff9640 1 -- 172.21.15.29:0/1687128581 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8469+0+0 (secure 0 0 0) 0x7f7b500cf620 con 0x7f7b5810be50 2024-03-16T11:51:10.312 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:10.305+0000 7f7b577fe640 1 -- 172.21.15.29:0/994047079 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bc070bc820 msgr2=0x55bc070bec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:51:10.312 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:10.305+0000 7f7b577fe640 1 --2- 172.21.15.29:0/994047079 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bc070bc820 0x55bc070bec10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:51:10.312 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:10.305+0000 7f7b577fe640 1 -- 172.21.15.29:0/994047079 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bc070b3130 msgr2=0x55bc070b3510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:51:10.312 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:10.305+0000 7f7b577fe640 1 --2- 172.21.15.29:0/994047079 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bc070b3130 0x55bc070b3510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:51:10.312 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:10.305+0000 7f7b577fe640 1 -- 172.21.15.29:0/994047079 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bc06fd8460 con 0x55bc070b3a50 2024-03-16T11:51:10.312 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:10.305+0000 7f7b57fff640 1 --2- 172.21.15.29:0/1687128581 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f7b400a2b30 0x7f7b400a4ff0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f7b500ccc80 tx=0x7f7b50004a50 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:51:10.312 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:10.305+0000 7f7b577fe640 1 --2- 172.21.15.29:0/994047079 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55bc070b3a50 0x55bc070bc2e0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f7b48006010 tx=0x7f7b480098c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=593044017ac1cf04 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:51:10.312 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:10.306+0000 7f7b3dffb640 1 -- 172.21.15.29:0/994047079 <== mon.2 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b48043070 con 0x55bc070b3a50 2024-03-16T11:51:10.312 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:10.306+0000 7f7b3dffb640 1 -- 172.21.15.29:0/994047079 <== mon.2 v2:172.21.15.152:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7b48032c90 con 0x55bc070b3a50 2024-03-16T11:51:10.312 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:10.306+0000 7f7b3dffb640 1 -- 172.21.15.29:0/994047079 <== mon.2 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b48039040 con 0x55bc070b3a50 2024-03-16T11:51:10.312 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:10.306+0000 7f7b609c4740 1 -- 172.21.15.29:0/994047079 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55bc070b3a50 msgr2=0x55bc070bc2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:51:10.312 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:10.306+0000 7f7b609c4740 1 --2- 172.21.15.29:0/994047079 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55bc070b3a50 0x55bc070bc2e0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f7b48006010 tx=0x7f7b480098c0 comp rx=0 tx=0).stop 2024-03-16T11:51:10.312 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:10.306+0000 7f7b609c4740 1 -- 172.21.15.29:0/994047079 shutdown_connections 2024-03-16T11:51:10.313 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:10.306+0000 7f7b609c4740 1 --2- 172.21.15.29:0/994047079 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bc070bc820 0x55bc070bec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:51:10.313 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:10.306+0000 7f7b609c4740 1 --2- 172.21.15.29:0/994047079 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55bc070b3a50 0x55bc070bc2e0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:51:10.313 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:10.306+0000 7f7b609c4740 1 --2- 172.21.15.29:0/994047079 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bc070b3130 0x55bc070b3510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:51:10.313 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:10.306+0000 7f7b609c4740 1 -- 172.21.15.29:0/994047079 >> 172.21.15.29:0/994047079 conn(0x55bc070ada50 msgr2=0x55bc070ade30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:51:10.313 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:10.306+0000 7f7b609c4740 1 -- 172.21.15.29:0/994047079 shutdown_connections 2024-03-16T11:51:10.313 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:10.306+0000 7f7b609c4740 1 -- 172.21.15.29:0/994047079 wait complete. 2024-03-16T11:51:10.313 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:10.307+0000 7f7b609c4740 1 -- start start 2024-03-16T11:51:10.313 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:10.307+0000 7f7b609c4740 1 Processor -- start 2024-03-16T11:51:10.313 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:10.307+0000 7f7b609c4740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bc070b3130 0x55bc070b9860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:51:10.313 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:10.307+0000 7f7b609c4740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bc070b3a50 0x55bc070b9da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:51:10.313 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:10.307+0000 7f7b609c4740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55bc070bc820 0x55bc070ba2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:51:10.313 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:10.307+0000 7f7b609c4740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55bc070bf3f0 con 0x55bc070b3a50 2024-03-16T11:51:10.313 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:10.307+0000 7f7b609c4740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x55bc06fd3d70 con 0x55bc070bc820 2024-03-16T11:51:10.313 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:10.307+0000 7f7b609c4740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55bc070bf570 con 0x55bc070b3130 2024-03-16T11:51:10.313 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:10.307+0000 7f7b609c4740 10 client.0 _ll_register_callbacks cb 0x55bc06d2ba00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T11:51:10.314 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:10.307+0000 7f7b577fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bc070b3a50 0x55bc070b9da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:51:10.314 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:10.307+0000 7f7b577fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bc070b3a50 0x55bc070b9da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:40764/0 (socket says 172.21.15.29:40764) 2024-03-16T11:51:10.314 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:10.307+0000 7f7b577fe640 1 -- 172.21.15.29:0/3397227782 learned_addr learned my addr 172.21.15.29:0/3397227782 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:51:10.314 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:10.307+0000 7f7b56ffd640 1 --2- 172.21.15.29:0/3397227782 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bc070b3130 0x55bc070b9860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:51:10.314 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:10.307+0000 7f7b56ffd640 1 --2- 172.21.15.29:0/3397227782 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55bc070bc820 0x55bc070ba2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:51:10.314 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:10.308+0000 7f7b577fe640 1 -- 172.21.15.29:0/3397227782 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bc070b3130 msgr2=0x55bc070b9860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:51:10.314 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:10.308+0000 7f7b577fe640 1 --2- 172.21.15.29:0/3397227782 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bc070b3130 0x55bc070b9860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:51:10.314 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:10.308+0000 7f7b577fe640 1 -- 172.21.15.29:0/3397227782 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55bc070bc820 msgr2=0x55bc070ba2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:51:10.314 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:10.308+0000 7f7b577fe640 1 --2- 172.21.15.29:0/3397227782 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55bc070bc820 0x55bc070ba2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:51:10.314 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:10.308+0000 7f7b577fe640 1 -- 172.21.15.29:0/3397227782 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bc06d2c210 con 0x55bc070b3a50 2024-03-16T11:51:10.314 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:10.308+0000 7f7b56ffd640 1 --2- 172.21.15.29:0/3397227782 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bc070b3130 0x55bc070b9860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:51:10.314 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:10.308+0000 7f7b577fe640 1 --2- 172.21.15.29:0/3397227782 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bc070b3a50 0x55bc070b9da0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f7b4802c630 tx=0x7f7b48032990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d53c734b127c1fe5 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:51:10.314 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:10.308+0000 7f7b36ffd640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-16T11:51:10.314 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:10.308+0000 7f7b36ffd640 1 -- 172.21.15.29:0/3397227782 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b4804c030 con 0x55bc070b3a50 2024-03-16T11:51:10.315 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:10.308+0000 7f7b36ffd640 1 -- 172.21.15.29:0/3397227782 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8469+0+0 (secure 0 0 0) 0x7f7b480347f0 con 0x55bc070b3a50 2024-03-16T11:51:10.315 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:10.308+0000 7f7b609c4740 1 -- 172.21.15.29:0/3397227782 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7b48032c90 con 0x55bc070b3a50 2024-03-16T11:51:10.315 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:10.308+0000 7f7b36ffd640 1 -- 172.21.15.29:0/3397227782 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f7b48039070 con 0x55bc070b3a50 2024-03-16T11:51:10.315 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:10.308+0000 7f7b609c4740 10 client.15426 fetch_fsmap learned FSMap version 22 2024-03-16T11:51:10.315 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:10.308+0000 7f7b609c4740 1 -- 172.21.15.29:0/3397227782 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x55bc06df8620 con 0x55bc070b3a50 2024-03-16T11:51:10.315 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:10.308+0000 7f7b36ffd640 1 -- 172.21.15.29:0/3397227782 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7b48032c90 con 0x55bc070b3a50 2024-03-16T11:51:10.315 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:10.309+0000 7f7b36ffd640 1 -- 172.21.15.29:0/3397227782 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7f7b4805a860 con 0x55bc070b3a50 2024-03-16T11:51:10.315 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:10.309+0000 7f7b609c4740 10 client.15426 fetch_fsmap finished waiting for FSMap version 22 2024-03-16T11:51:10.315 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:10.309+0000 7f7b609c4740 10 client.15426 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T11:51:10.315 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:10.309+0000 7f7b609c4740 20 client.15426 populate_metadata read hostname 'smithi029' 2024-03-16T11:51:10.315 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:10.309+0000 7f7b609c4740 1 --2- 172.21.15.29:0/3397227782 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55bc070cc7d0 0x55bc070cebc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:51:10.315 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:10.309+0000 7f7b609c4740 4 client.15426 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T11:51:10.315 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:10.309+0000 7f7b609c4740 1 -- 172.21.15.29:0/3397227782 --> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bc07024f80 con 0x55bc070cc7d0 2024-03-16T11:51:10.315 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:10.309+0000 7f7b56ffd640 1 --2- 172.21.15.29:0/3397227782 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55bc070cc7d0 0x55bc070cebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:51:10.315 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:10.309+0000 7f7b36ffd640 10 client.15426 ms_handle_connect on v2:172.21.15.152:6838/1548731833 2024-03-16T11:51:10.316 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:10.309+0000 7f7b56ffd640 1 --2- 172.21.15.29:0/3397227782 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55bc070cc7d0 0x55bc070cebc0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4ac67eedb0558c73 server_cookie=aa63d5f6575bb8b9 in_seq=0 out_seq=0 2024-03-16T11:51:10.316 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:10.310+0000 7f7b36ffd640 1 -- 172.21.15.29:0/3397227782 <== mds.0 v2:172.21.15.152:6838/1548731833 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55bc07024f80 con 0x55bc070cc7d0 2024-03-16T11:51:10.316 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:10.310+0000 7f7b36ffd640 10 client.15426 handle_command_reply: tid=0 2024-03-16T11:51:10.316 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:10.310+0000 7f7b609c4740 1 client.15426 shutdown 2024-03-16T11:51:10.316 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:10.310+0000 7f7b609c4740 1 -- 172.21.15.29:0/3397227782 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bc070b3a50 msgr2=0x55bc070b9da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:51:10.316 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:10.310+0000 7f7b609c4740 1 --2- 172.21.15.29:0/3397227782 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bc070b3a50 0x55bc070b9da0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f7b4802c630 tx=0x7f7b48032990 comp rx=0 tx=0).stop 2024-03-16T11:51:10.316 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:10.310+0000 7f7b609c4740 1 -- 172.21.15.29:0/3397227782 shutdown_connections 2024-03-16T11:51:10.316 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:10.310+0000 7f7b609c4740 1 --2- 172.21.15.29:0/3397227782 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55bc070cc7d0 0x55bc070cebc0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:51:10.316 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:10.310+0000 7f7b609c4740 1 --2- 172.21.15.29:0/3397227782 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55bc070bc820 0x55bc070ba2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:51:10.316 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:10.310+0000 7f7b609c4740 1 --2- 172.21.15.29:0/3397227782 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bc070b3a50 0x55bc070b9da0 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:51:10.316 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:10.310+0000 7f7b36ffd640 0 client.15426 ms_handle_reset on v2:172.21.15.152:6838/1548731833 2024-03-16T11:51:10.316 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:10.310+0000 7f7b609c4740 1 --2- 172.21.15.29:0/3397227782 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bc070b3130 0x55bc070b9860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:51:10.316 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:10.310+0000 7f7b609c4740 1 -- 172.21.15.29:0/3397227782 >> 172.21.15.29:0/3397227782 conn(0x55bc070ada50 msgr2=0x55bc070af7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:51:10.316 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:10.310+0000 7f7b609c4740 1 -- 172.21.15.29:0/3397227782 shutdown_connections 2024-03-16T11:51:10.317 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:10.310+0000 7f7b609c4740 1 -- 172.21.15.29:0/3397227782 wait complete. 2024-03-16T11:51:10.317 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:10.310+0000 7f7b609c4740 20 client.15426 trim_cache size 0 max 16384 2024-03-16T11:51:10.329 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:10.325+0000 7f7b609c4740 1 Processor -- start 2024-03-16T11:51:10.329 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:10.325+0000 7f7b609c4740 1 -- start start 2024-03-16T11:51:10.329 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:10.325+0000 7f7b609c4740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55bc070b3130 0x55bc070c4ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:51:10.329 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:10.326+0000 7f7b609c4740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bc070b3a50 0x55bc070c8190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:51:10.330 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:10.326+0000 7f7b609c4740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bc070bc820 0x55bc070d4190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:51:10.330 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:10.326+0000 7f7b609c4740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f7b581afbc0 con 0x55bc070b3a50 2024-03-16T11:51:10.330 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:10.326+0000 7f7b609c4740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f7b581ad490 con 0x55bc070b3130 2024-03-16T11:51:10.330 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:10.326+0000 7f7b609c4740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f7b581ab2e0 con 0x55bc070bc820 2024-03-16T11:51:10.330 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:10.326+0000 7f7b577fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bc070b3a50 0x55bc070c8190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:51:10.330 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:10.326+0000 7f7b577fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bc070b3a50 0x55bc070c8190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:40776/0 (socket says 172.21.15.29:40776) 2024-03-16T11:51:10.330 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:10.326+0000 7f7b577fe640 1 -- 172.21.15.29:0/3743462410 learned_addr learned my addr 172.21.15.29:0/3743462410 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:51:10.330 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:10.326+0000 7f7b56ffd640 1 --2- 172.21.15.29:0/3743462410 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55bc070b3130 0x55bc070c4ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:51:10.330 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:10.326+0000 7f7b56ffd640 1 --2- 172.21.15.29:0/3743462410 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bc070bc820 0x55bc070d4190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:51:10.330 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:10.326+0000 7f7b577fe640 1 -- 172.21.15.29:0/3743462410 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bc070bc820 msgr2=0x55bc070d4190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:51:10.330 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:10.326+0000 7f7b577fe640 1 --2- 172.21.15.29:0/3743462410 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bc070bc820 0x55bc070d4190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:51:10.330 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:10.326+0000 7f7b577fe640 1 -- 172.21.15.29:0/3743462410 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55bc070b3130 msgr2=0x55bc070c4ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:51:10.330 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:10.326+0000 7f7b577fe640 1 --2- 172.21.15.29:0/3743462410 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55bc070b3130 0x55bc070c4ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:51:10.331 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:10.326+0000 7f7b577fe640 1 -- 172.21.15.29:0/3743462410 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bc070bec10 con 0x55bc070b3a50 2024-03-16T11:51:10.331 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:10.326+0000 7f7b56ffd640 1 --2- 172.21.15.29:0/3743462410 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bc070bc820 0x55bc070d4190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T11:51:10.331 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:10.326+0000 7f7b577fe640 1 --2- 172.21.15.29:0/3743462410 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bc070b3a50 0x55bc070c8190 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f7b4802c700 tx=0x7f7b48032920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33fcdb05eb2c1a5c server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:51:10.331 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:10.326+0000 7f7b3d7fa640 1 -- 172.21.15.29:0/3743462410 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b4804c070 con 0x55bc070b3a50 2024-03-16T11:51:10.331 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:10.326+0000 7f7b3d7fa640 1 -- 172.21.15.29:0/3743462410 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7b4805ae50 con 0x55bc070b3a50 2024-03-16T11:51:10.331 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:10.326+0000 7f7b3d7fa640 1 -- 172.21.15.29:0/3743462410 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b48043040 con 0x55bc070b3a50 2024-03-16T11:51:10.331 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:10.327+0000 7f7b609c4740 1 -- 172.21.15.29:0/3743462410 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bc070b3a50 msgr2=0x55bc070c8190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:51:10.331 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:10.327+0000 7f7b609c4740 1 --2- 172.21.15.29:0/3743462410 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bc070b3a50 0x55bc070c8190 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f7b4802c700 tx=0x7f7b48032920 comp rx=0 tx=0).stop 2024-03-16T11:51:10.331 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:10.327+0000 7f7b609c4740 1 -- 172.21.15.29:0/3743462410 shutdown_connections 2024-03-16T11:51:10.331 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:10.327+0000 7f7b609c4740 1 --2- 172.21.15.29:0/3743462410 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bc070bc820 0x55bc070d4190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:51:10.331 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:10.327+0000 7f7b609c4740 1 --2- 172.21.15.29:0/3743462410 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bc070b3a50 0x55bc070c8190 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:51:10.331 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:10.327+0000 7f7b609c4740 1 --2- 172.21.15.29:0/3743462410 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55bc070b3130 0x55bc070c4ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:51:10.331 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:10.327+0000 7f7b609c4740 1 -- 172.21.15.29:0/3743462410 >> 172.21.15.29:0/3743462410 conn(0x55bc070ada50 msgr2=0x55bc070b4b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:51:10.331 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:10.327+0000 7f7b609c4740 1 -- 172.21.15.29:0/3743462410 shutdown_connections 2024-03-16T11:51:10.332 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:10.327+0000 7f7b609c4740 1 -- 172.21.15.29:0/3743462410 wait complete. 2024-03-16T11:51:10.332 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:10.327+0000 7f7b609c4740 1 -- start start 2024-03-16T11:51:10.332 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:10.328+0000 7f7b609c4740 1 Processor -- start 2024-03-16T11:51:10.332 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:10.328+0000 7f7b609c4740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bc070b3130 0x55bc070b67c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:51:10.332 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:10.328+0000 7f7b609c4740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bc070b3a50 0x55bc070b6d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:51:10.332 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:10.328+0000 7f7b609c4740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55bc070bc820 0x55bc0712da40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:51:10.332 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:10.328+0000 7f7b609c4740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f7b581ab950 con 0x55bc070b3130 2024-03-16T11:51:10.332 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:10.328+0000 7f7b609c4740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f7b581acdc0 con 0x55bc070bc820 2024-03-16T11:51:10.332 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:10.328+0000 7f7b609c4740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f7b581ac020 con 0x55bc070b3a50 2024-03-16T11:51:10.332 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:10.328+0000 7f7b609c4740 10 client.0 _ll_register_callbacks cb 0x7f7b4c009b10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T11:51:10.332 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:10.328+0000 7f7b56ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bc070b3130 0x55bc070b67c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:51:10.332 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:10.328+0000 7f7b56ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bc070b3130 0x55bc070b67c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:40780/0 (socket says 172.21.15.29:40780) 2024-03-16T11:51:10.332 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:10.328+0000 7f7b56ffd640 1 -- 172.21.15.29:0/2504459014 learned_addr learned my addr 172.21.15.29:0/2504459014 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:51:10.332 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:10.328+0000 7f7b577fe640 1 --2- 172.21.15.29:0/2504459014 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bc070b3a50 0x55bc070b6d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:51:10.333 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:10.328+0000 7f7b56ffd640 1 --2- 172.21.15.29:0/2504459014 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55bc070bc820 0x55bc0712da40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:51:10.333 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:10.328+0000 7f7b56ffd640 1 -- 172.21.15.29:0/2504459014 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bc070b3a50 msgr2=0x55bc070b6d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:51:10.333 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:10.328+0000 7f7b56ffd640 1 --2- 172.21.15.29:0/2504459014 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bc070b3a50 0x55bc070b6d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:51:10.333 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:10.328+0000 7f7b56ffd640 1 -- 172.21.15.29:0/2504459014 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55bc070bc820 msgr2=0x55bc0712da40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:51:10.333 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:10.328+0000 7f7b56ffd640 1 --2- 172.21.15.29:0/2504459014 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55bc070bc820 0x55bc0712da40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:51:10.333 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:10.328+0000 7f7b56ffd640 1 -- 172.21.15.29:0/2504459014 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bc0712e0b0 con 0x55bc070b3130 2024-03-16T11:51:10.333 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:10.328+0000 7f7b56ffd640 1 --2- 172.21.15.29:0/2504459014 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bc070b3130 0x55bc070b67c0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x55bc070cf130 tx=0x7f7b4c015040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a60cfb0f243bf6da server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:51:10.333 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:10.328+0000 7f7b36ffd640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-16T11:51:10.333 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:10.328+0000 7f7b36ffd640 1 -- 172.21.15.29:0/2504459014 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b4c02f020 con 0x55bc070b3130 2024-03-16T11:51:10.333 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:10.329+0000 7f7b36ffd640 1 -- 172.21.15.29:0/2504459014 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8469+0+0 (secure 0 0 0) 0x7f7b4c028720 con 0x55bc070b3130 2024-03-16T11:51:10.333 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:10.329+0000 7f7b609c4740 1 -- 172.21.15.29:0/2504459014 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7b4805ae50 con 0x55bc070b3130 2024-03-16T11:51:10.333 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:10.329+0000 7f7b36ffd640 1 -- 172.21.15.29:0/2504459014 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f7b4c03a050 con 0x55bc070b3130 2024-03-16T11:51:10.333 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:10.329+0000 7f7b609c4740 10 client.15432 fetch_fsmap learned FSMap version 22 2024-03-16T11:51:10.333 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:10.329+0000 7f7b609c4740 1 -- 172.21.15.29:0/2504459014 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x55bc071366c0 con 0x55bc070b3130 2024-03-16T11:51:10.334 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:10.329+0000 7f7b36ffd640 1 -- 172.21.15.29:0/2504459014 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7b4805ae50 con 0x55bc070b3130 2024-03-16T11:51:10.334 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:10.329+0000 7f7b36ffd640 1 -- 172.21.15.29:0/2504459014 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7f7b4c022040 con 0x55bc070b3130 2024-03-16T11:51:10.334 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:10.329+0000 7f7b609c4740 10 client.15432 fetch_fsmap finished waiting for FSMap version 22 2024-03-16T11:51:10.334 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:10.329+0000 7f7b609c4740 10 client.15432 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T11:51:10.334 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:10.329+0000 7f7b609c4740 20 client.15432 populate_metadata read hostname 'smithi029' 2024-03-16T11:51:10.334 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:10.329+0000 7f7b609c4740 1 --2- 172.21.15.29:0/2504459014 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55bc07136e70 0x55bc07139290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:51:10.334 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:10.329+0000 7f7b609c4740 4 client.15432 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-16T11:51:10.334 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:10.329+0000 7f7b609c4740 1 -- 172.21.15.29:0/2504459014 --> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55bc0701f2d0 con 0x55bc07136e70 2024-03-16T11:51:10.334 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:10.329+0000 7f7b577fe640 1 --2- 172.21.15.29:0/2504459014 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55bc07136e70 0x55bc07139290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:51:10.334 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:10.330+0000 7f7b577fe640 1 --2- 172.21.15.29:0/2504459014 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55bc07136e70 0x55bc07139290 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=219b2c169e6ae7c6 server_cookie=a93590a15280ec4b in_seq=0 out_seq=0 2024-03-16T11:51:10.334 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:10.330+0000 7f7b36ffd640 10 client.15432 ms_handle_connect on v2:172.21.15.152:6838/1548731833 2024-03-16T11:51:10.341 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:10.341+0000 7f7b36ffd640 1 -- 172.21.15.29:0/2504459014 <== mds.0 v2:172.21.15.152:6838/1548731833 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55bc0701f2d0 con 0x55bc07136e70 2024-03-16T11:51:10.341 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:10.341+0000 7f7b36ffd640 10 client.15432 handle_command_reply: tid=0 2024-03-16T11:51:10.341 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:10.341+0000 7f7b609c4740 1 client.15432 shutdown 2024-03-16T11:51:10.341 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:10.342+0000 7f7b609c4740 1 -- 172.21.15.29:0/2504459014 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bc070b3130 msgr2=0x55bc070b67c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:51:10.341 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:10.342+0000 7f7b609c4740 1 --2- 172.21.15.29:0/2504459014 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bc070b3130 0x55bc070b67c0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x55bc070cf130 tx=0x7f7b4c015040 comp rx=0 tx=0).stop 2024-03-16T11:51:10.341 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:10.342+0000 7f7b609c4740 1 -- 172.21.15.29:0/2504459014 shutdown_connections 2024-03-16T11:51:10.342 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:10.342+0000 7f7b609c4740 1 --2- 172.21.15.29:0/2504459014 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55bc07136e70 0x55bc07139290 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:51:10.342 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:10.342+0000 7f7b609c4740 1 --2- 172.21.15.29:0/2504459014 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55bc070bc820 0x55bc0712da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:51:10.342 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:10.342+0000 7f7b609c4740 1 --2- 172.21.15.29:0/2504459014 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bc070b3a50 0x55bc070b6d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:51:10.342 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:10.342+0000 7f7b36ffd640 0 client.15432 ms_handle_reset on v2:172.21.15.152:6838/1548731833 2024-03-16T11:51:10.342 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:10.342+0000 7f7b609c4740 1 --2- 172.21.15.29:0/2504459014 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bc070b3130 0x55bc070b67c0 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:51:10.342 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:10.342+0000 7f7b609c4740 1 -- 172.21.15.29:0/2504459014 >> 172.21.15.29:0/2504459014 conn(0x55bc070ada50 msgr2=0x55bc070aeea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:51:10.342 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:10.342+0000 7f7b609c4740 1 -- 172.21.15.29:0/2504459014 shutdown_connections 2024-03-16T11:51:10.342 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:10.342+0000 7f7b609c4740 1 -- 172.21.15.29:0/2504459014 wait complete. 2024-03-16T11:51:10.342 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:10.342+0000 7f7b609c4740 20 client.15432 trim_cache size 0 max 16384 2024-03-16T11:51:10.342 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:{ 2024-03-16T11:51:10.343 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "message": "", 2024-03-16T11:51:10.343 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "return_code": 0 2024-03-16T11:51:10.343 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:} 2024-03-16T11:51:10.343 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:10.342+0000 7f7b5ddd4640 1 -- 172.21.15.29:0/1687128581 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f7b400a2b30 msgr2=0x7f7b400a4ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:51:10.343 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:10.342+0000 7f7b5ddd4640 1 --2- 172.21.15.29:0/1687128581 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f7b400a2b30 0x7f7b400a4ff0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f7b500ccc80 tx=0x7f7b50004a50 comp rx=0 tx=0).stop 2024-03-16T11:51:10.344 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:10.342+0000 7f7b5ddd4640 1 -- 172.21.15.29:0/1687128581 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7b5810be50 msgr2=0x7f7b5819c850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:51:10.344 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:10.342+0000 7f7b5ddd4640 1 --2- 172.21.15.29:0/1687128581 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7b5810be50 0x7f7b5819c850 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f7b5000e970 tx=0x7f7b5000ed70 comp rx=0 tx=0).stop 2024-03-16T11:51:10.344 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:10.342+0000 7f7b5ddd4640 1 -- 172.21.15.29:0/1687128581 shutdown_connections 2024-03-16T11:51:10.344 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:10.342+0000 7f7b5ddd4640 1 --2- 172.21.15.29:0/1687128581 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f7b400a2b30 0x7f7b400a4ff0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:51:10.344 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:10.342+0000 7f7b5ddd4640 1 --2- 172.21.15.29:0/1687128581 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7b5810be50 0x7f7b5819c850 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:51:10.344 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:10.342+0000 7f7b5ddd4640 1 --2- 172.21.15.29:0/1687128581 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7b58103300 0x7f7b581984d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:51:10.344 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:10.342+0000 7f7b5ddd4640 1 --2- 172.21.15.29:0/1687128581 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7b581029e0 0x7f7b58197f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:51:10.344 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:10.343+0000 7f7b5ddd4640 1 -- 172.21.15.29:0/1687128581 >> 172.21.15.29:0/1687128581 conn(0x7f7b58077810 msgr2=0x7f7b580fd5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:51:10.344 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:10.343+0000 7f7b5ddd4640 1 -- 172.21.15.29:0/1687128581 shutdown_connections 2024-03-16T11:51:10.344 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:10.343+0000 7f7b5ddd4640 1 -- 172.21.15.29:0/1687128581 wait complete. 2024-03-16T11:51:10.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:51:10 smithi154 ceph-mon[36320]: pgmap v10: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 100 KiB/s wr, 62 op/s 2024-03-16T11:51:10.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:51:10 smithi029 ceph-mon[31975]: pgmap v10: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 100 KiB/s wr, 62 op/s 2024-03-16T11:51:10.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:51:10 smithi152 ceph-mon[36291]: pgmap v10: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 100 KiB/s wr, 62 op/s 2024-03-16T11:51:12.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:51:12 smithi154 ceph-mon[36320]: pgmap v11: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 100 KiB/s wr, 62 op/s 2024-03-16T11:51:12.837 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:51:12 smithi029 ceph-mon[31975]: pgmap v11: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 100 KiB/s wr, 62 op/s 2024-03-16T11:51:12.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:51:12 smithi152 ceph-mon[36291]: pgmap v11: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 100 KiB/s wr, 62 op/s 2024-03-16T11:51:14.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:51:14 smithi154 ceph-mon[36320]: pgmap v12: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 102 KiB/s wr, 69 op/s 2024-03-16T11:51:14.837 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:51:14 smithi029 ceph-mon[31975]: pgmap v12: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 102 KiB/s wr, 69 op/s 2024-03-16T11:51:14.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:51:14 smithi152 ceph-mon[36291]: pgmap v12: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 102 KiB/s wr, 69 op/s 2024-03-16T11:51:16.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:51:16 smithi154 ceph-mon[36320]: pgmap v13: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 2.8 KiB/s wr, 9 op/s 2024-03-16T11:51:16.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:51:16 smithi029 ceph-mon[31975]: pgmap v13: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 2.8 KiB/s wr, 9 op/s 2024-03-16T11:51:16.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:51:16 smithi152 ceph-mon[36291]: pgmap v13: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 2.8 KiB/s wr, 9 op/s 2024-03-16T11:51:18.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:51:18 smithi154 ceph-mon[36320]: pgmap v14: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 2.8 KiB/s wr, 12 op/s 2024-03-16T11:51:18.837 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:51:18 smithi029 ceph-mon[31975]: pgmap v14: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 2.8 KiB/s wr, 12 op/s 2024-03-16T11:51:18.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:51:18 smithi152 ceph-mon[36291]: pgmap v14: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 2.8 KiB/s wr, 12 op/s 2024-03-16T11:51:20.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:51:20 smithi154 ceph-mon[36320]: pgmap v15: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 2.2 KiB/s wr, 9 op/s 2024-03-16T11:51:20.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:51:20 smithi029 ceph-mon[31975]: pgmap v15: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 2.2 KiB/s wr, 9 op/s 2024-03-16T11:51:20.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:51:20 smithi152 ceph-mon[36291]: pgmap v15: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 2.2 KiB/s wr, 9 op/s 2024-03-16T11:51:22.545 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:51:22 smithi154 ceph-mon[36320]: pgmap v16: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 2.2 KiB/s wr, 9 op/s 2024-03-16T11:51:22.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:51:22 smithi029 ceph-mon[31975]: pgmap v16: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 2.2 KiB/s wr, 9 op/s 2024-03-16T11:51:22.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:51:22 smithi152 ceph-mon[36291]: pgmap v16: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 2.2 KiB/s wr, 9 op/s 2024-03-16T11:51:23.442 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:23.440+0000 7f8fb1f37640 1 -- 172.21.15.29:0/3688355481 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8fac102a50 msgr2=0x7f8fac102e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:51:23.442 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:23.440+0000 7f8fb1f37640 1 --2- 172.21.15.29:0/3688355481 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8fac102a50 0x7f8fac102e30 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f8f9c009900 tx=0x7f8f9c02bc30 comp rx=0 tx=0).stop 2024-03-16T11:51:23.442 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:23.440+0000 7f8fb1f37640 1 -- 172.21.15.29:0/3688355481 shutdown_connections 2024-03-16T11:51:23.442 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:23.440+0000 7f8fb1f37640 1 --2- 172.21.15.29:0/3688355481 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8fac10bf60 0x7f8fac10e3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:51:23.442 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:23.440+0000 7f8fb1f37640 1 --2- 172.21.15.29:0/3688355481 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8fac103400 0x7f8fac10b850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:51:23.442 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:23.440+0000 7f8fb1f37640 1 --2- 172.21.15.29:0/3688355481 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8fac102a50 0x7f8fac102e30 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:51:23.443 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:23.440+0000 7f8fb1f37640 1 -- 172.21.15.29:0/3688355481 >> 172.21.15.29:0/3688355481 conn(0x7f8fac077800 msgr2=0x7f8fac077c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:51:23.443 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:23.440+0000 7f8fb1f37640 1 -- 172.21.15.29:0/3688355481 shutdown_connections 2024-03-16T11:51:23.443 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:23.441+0000 7f8fb1f37640 1 -- 172.21.15.29:0/3688355481 wait complete. 2024-03-16T11:51:23.443 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:23.441+0000 7f8fb1f37640 1 Processor -- start 2024-03-16T11:51:23.443 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:23.441+0000 7f8fb1f37640 1 -- start start 2024-03-16T11:51:23.443 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:23.441+0000 7f8fb1f37640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8fac102a50 0x7f8fac1981d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:51:23.443 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:23.441+0000 7f8fb1f37640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8fac10bf60 0x7f8fac198710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:51:23.443 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:23.441+0000 7f8fb1f37640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8fac19ca40 0x7f8fac19cef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:51:23.443 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:23.441+0000 7f8fb1f37640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f8fac111d70 con 0x7f8fac19ca40 2024-03-16T11:51:23.443 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:23.441+0000 7f8fb1f37640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f8fac111bf0 con 0x7f8fac102a50 2024-03-16T11:51:23.443 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:23.441+0000 7f8fb1f37640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f8fac111ef0 con 0x7f8fac10bf60 2024-03-16T11:51:23.443 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:23.442+0000 7f8fabfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8fac19ca40 0x7f8fac19cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:51:23.444 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:23.442+0000 7f8fabfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8fac19ca40 0x7f8fac19cef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33634/0 (socket says 172.21.15.29:33634) 2024-03-16T11:51:23.444 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:23.442+0000 7f8fabfff640 1 -- 172.21.15.29:0/2608965090 learned_addr learned my addr 172.21.15.29:0/2608965090 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:51:23.444 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:23.442+0000 7f8fab7fe640 1 --2- 172.21.15.29:0/2608965090 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8fac102a50 0x7f8fac1981d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:51:23.444 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:23.442+0000 7f8faaffd640 1 --2- 172.21.15.29:0/2608965090 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8fac10bf60 0x7f8fac198710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:51:23.444 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:23.442+0000 7f8fabfff640 1 -- 172.21.15.29:0/2608965090 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8fac10bf60 msgr2=0x7f8fac198710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:51:23.444 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:23.442+0000 7f8fabfff640 1 --2- 172.21.15.29:0/2608965090 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8fac10bf60 0x7f8fac198710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:51:23.444 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:23.442+0000 7f8fabfff640 1 -- 172.21.15.29:0/2608965090 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8fac102a50 msgr2=0x7f8fac1981d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:51:23.444 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:23.442+0000 7f8fabfff640 1 --2- 172.21.15.29:0/2608965090 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8fac102a50 0x7f8fac1981d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:51:23.444 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:23.442+0000 7f8fabfff640 1 -- 172.21.15.29:0/2608965090 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8fac19d530 con 0x7f8fac19ca40 2024-03-16T11:51:23.444 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:23.442+0000 7f8fabfff640 1 --2- 172.21.15.29:0/2608965090 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8fac19ca40 0x7f8fac19cef0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f8fa400eb10 tx=0x7f8fa400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:51:23.444 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:23.442+0000 7f8fa8ff9640 1 -- 172.21.15.29:0/2608965090 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fa4004b20 con 0x7f8fac19ca40 2024-03-16T11:51:23.444 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:23.442+0000 7f8fa8ff9640 1 -- 172.21.15.29:0/2608965090 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8fa4004cc0 con 0x7f8fac19ca40 2024-03-16T11:51:23.445 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:23.442+0000 7f8fb1f37640 1 -- 172.21.15.29:0/2608965090 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8fac19d820 con 0x7f8fac19ca40 2024-03-16T11:51:23.445 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:23.442+0000 7f8fa8ff9640 1 -- 172.21.15.29:0/2608965090 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fa4010760 con 0x7f8fac19ca40 2024-03-16T11:51:23.445 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:23.442+0000 7f8fb1f37640 1 -- 172.21.15.29:0/2608965090 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8fac1a5010 con 0x7f8fac19ca40 2024-03-16T11:51:23.452 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:23.445+0000 7f8fa8ff9640 1 -- 172.21.15.29:0/2608965090 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 24) ==== 137523+0+0 (secure 0 0 0) 0x7f8fa4005020 con 0x7f8fac19ca40 2024-03-16T11:51:23.452 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:23.445+0000 7f8fb4bfa740 1 Processor -- start 2024-03-16T11:51:23.452 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:23.445+0000 7f8fb4bfa740 1 -- start start 2024-03-16T11:51:23.452 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:23.445+0000 7f8fb4bfa740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561f6ec38130 0x561f6ec38510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:51:23.452 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:23.445+0000 7f8fb4bfa740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561f6ec38a50 0x561f6ec412e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:51:23.453 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:23.445+0000 7f8fb4bfa740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561f6ec41820 0x561f6ec43c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:51:23.453 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:23.445+0000 7f8fb4bfa740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x561f6eb9ca40 con 0x561f6ec38130 2024-03-16T11:51:23.453 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:23.445+0000 7f8fb4bfa740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x561f6eb45920 con 0x561f6ec38a50 2024-03-16T11:51:23.453 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:23.445+0000 7f8fb4bfa740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x561f6eb98ae0 con 0x561f6ec41820 2024-03-16T11:51:23.453 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:23.445+0000 7f8fa8ff9640 1 --2- 172.21.15.29:0/2608965090 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f8f940a2b30 0x7f8f940a4ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:51:23.453 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:23.445+0000 7f8faaffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561f6ec38130 0x561f6ec38510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:51:23.453 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:23.445+0000 7f8faaffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561f6ec38130 0x561f6ec38510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33646/0 (socket says 172.21.15.29:33646) 2024-03-16T11:51:23.453 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:23.445+0000 7f8faaffd640 1 -- 172.21.15.29:0/3013486315 learned_addr learned my addr 172.21.15.29:0/3013486315 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:51:23.454 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:23.445+0000 7f8fab7fe640 1 --2- 172.21.15.29:0/3013486315 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561f6ec38a50 0x561f6ec412e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:51:23.454 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:23.445+0000 7f8fabfff640 1 --2- 172.21.15.29:0/2608965090 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f8f940a2b30 0x7f8f940a4ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:51:23.454 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:23.446+0000 7f8faaffd640 1 --2- 172.21.15.29:0/3013486315 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561f6ec41820 0x561f6ec43c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:51:23.454 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:23.446+0000 7f8faaffd640 1 -- 172.21.15.29:0/3013486315 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561f6ec41820 msgr2=0x561f6ec43c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:51:23.454 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:23.446+0000 7f8faaffd640 1 --2- 172.21.15.29:0/3013486315 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561f6ec41820 0x561f6ec43c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:51:23.454 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:23.446+0000 7f8faaffd640 1 -- 172.21.15.29:0/3013486315 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561f6ec38a50 msgr2=0x561f6ec412e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:51:23.454 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:23.446+0000 7f8faaffd640 1 --2- 172.21.15.29:0/3013486315 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561f6ec38a50 0x561f6ec412e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:51:23.454 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:23.446+0000 7f8faaffd640 1 -- 172.21.15.29:0/3013486315 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561f6eb5d460 con 0x561f6ec38130 2024-03-16T11:51:23.454 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:23.446+0000 7f8fab7fe640 1 --2- 172.21.15.29:0/3013486315 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561f6ec38a50 0x561f6ec412e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:51:23.454 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:23.446+0000 7f8fa8ff9640 1 -- 172.21.15.29:0/2608965090 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8469+0+0 (secure 0 0 0) 0x7f8fa40cf740 con 0x7f8fac19ca40 2024-03-16T11:51:23.454 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:23.446+0000 7f8faaffd640 1 --2- 172.21.15.29:0/3013486315 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561f6ec38130 0x561f6ec38510 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f8fa00058c0 tx=0x7f8fa0005850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=421cc0c0d2f3838e server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:51:23.454 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:23.446+0000 7f8fabfff640 1 --2- 172.21.15.29:0/2608965090 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f8f940a2b30 0x7f8f940a4ff0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f8fa40ccda0 tx=0x7f8fa40057d0 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:51:23.455 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:23.446+0000 7f8f91ffb640 1 -- 172.21.15.29:0/3013486315 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fa0004050 con 0x561f6ec38130 2024-03-16T11:51:23.455 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:23.446+0000 7f8f91ffb640 1 -- 172.21.15.29:0/3013486315 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8fa00041f0 con 0x561f6ec38130 2024-03-16T11:51:23.455 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:23.446+0000 7f8f91ffb640 1 -- 172.21.15.29:0/3013486315 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fa0010750 con 0x561f6ec38130 2024-03-16T11:51:23.455 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:23.447+0000 7f8fb4bfa740 1 -- 172.21.15.29:0/3013486315 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561f6ec38130 msgr2=0x561f6ec38510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:51:23.455 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:23.447+0000 7f8fb4bfa740 1 --2- 172.21.15.29:0/3013486315 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561f6ec38130 0x561f6ec38510 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f8fa00058c0 tx=0x7f8fa0005850 comp rx=0 tx=0).stop 2024-03-16T11:51:23.455 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:23.447+0000 7f8fb4bfa740 1 -- 172.21.15.29:0/3013486315 shutdown_connections 2024-03-16T11:51:23.455 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:23.447+0000 7f8fb4bfa740 1 --2- 172.21.15.29:0/3013486315 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561f6ec41820 0x561f6ec43c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:51:23.455 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:23.447+0000 7f8fb4bfa740 1 --2- 172.21.15.29:0/3013486315 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561f6ec38a50 0x561f6ec412e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:51:23.455 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:23.447+0000 7f8fb4bfa740 1 --2- 172.21.15.29:0/3013486315 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561f6ec38130 0x561f6ec38510 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:51:23.455 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:23.447+0000 7f8fb4bfa740 1 -- 172.21.15.29:0/3013486315 >> 172.21.15.29:0/3013486315 conn(0x561f6ec32a50 msgr2=0x561f6ec32e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:51:23.455 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:23.447+0000 7f8fb4bfa740 1 -- 172.21.15.29:0/3013486315 shutdown_connections 2024-03-16T11:51:23.455 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:23.447+0000 7f8fb4bfa740 1 -- 172.21.15.29:0/3013486315 wait complete. 2024-03-16T11:51:23.456 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:23.447+0000 7f8fb4bfa740 1 -- start start 2024-03-16T11:51:23.456 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:23.448+0000 7f8fb4bfa740 1 Processor -- start 2024-03-16T11:51:23.456 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:23.448+0000 7f8fb4bfa740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561f6ec38130 0x561f6ec3e810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:51:23.456 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:23.448+0000 7f8fb4bfa740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561f6ec38a50 0x561f6ec3ed50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:51:23.456 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:23.448+0000 7f8fb4bfa740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561f6ec41820 0x561f6ec3f290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:51:23.456 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:23.448+0000 7f8fb4bfa740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x561f6ec44450 con 0x561f6ec38130 2024-03-16T11:51:23.456 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:23.448+0000 7f8fb4bfa740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x561f6eb58d70 con 0x561f6ec38a50 2024-03-16T11:51:23.456 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:23.448+0000 7f8fb4bfa740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x561f6ec445d0 con 0x561f6ec41820 2024-03-16T11:51:23.456 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:23.448+0000 7f8fb4bfa740 10 client.0 _ll_register_callbacks cb 0x561f6e8b0a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T11:51:23.456 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:23.448+0000 7f8fab7fe640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561f6ec38a50 0x561f6ec3ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:51:23.456 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:23.448+0000 7f8faaffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561f6ec38130 0x561f6ec3e810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:51:23.456 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:23.448+0000 7f8fab7fe640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561f6ec38a50 0x561f6ec3ed50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.29:57534/0 (socket says 172.21.15.29:57534) 2024-03-16T11:51:23.457 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:23.448+0000 7f8fab7fe640 1 -- 172.21.15.29:0/901610644 learned_addr learned my addr 172.21.15.29:0/901610644 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:51:23.457 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:23.448+0000 7f8faaffd640 1 --2- 172.21.15.29:0/901610644 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561f6ec41820 0x561f6ec3f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:51:23.457 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:23.448+0000 7f8fab7fe640 1 -- 172.21.15.29:0/901610644 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561f6ec41820 msgr2=0x561f6ec3f290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:51:23.457 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:23.448+0000 7f8fab7fe640 1 --2- 172.21.15.29:0/901610644 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561f6ec41820 0x561f6ec3f290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:51:23.457 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:23.448+0000 7f8fab7fe640 1 -- 172.21.15.29:0/901610644 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561f6ec38130 msgr2=0x561f6ec3e810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:51:23.457 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:23.448+0000 7f8fab7fe640 1 --2- 172.21.15.29:0/901610644 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561f6ec38130 0x561f6ec3e810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:51:23.457 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:23.448+0000 7f8fab7fe640 1 -- 172.21.15.29:0/901610644 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561f6e8b1210 con 0x561f6ec38a50 2024-03-16T11:51:23.457 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:23.448+0000 7f8faaffd640 1 --2- 172.21.15.29:0/901610644 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561f6ec41820 0x561f6ec3f290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T11:51:23.457 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:23.449+0000 7f8fab7fe640 1 --2- 172.21.15.29:0/901610644 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561f6ec38a50 0x561f6ec3ed50 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f8f9c02cfd0 tx=0x7f8f9c009560 comp rx=0 tx=0).ready entity=mon.2 client_cookie=67bcc04b92f7eafa server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:51:23.457 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:23.449+0000 7f8f8affd640 10 client.0 ms_handle_connect on v2:172.21.15.152:3300/0 2024-03-16T11:51:23.457 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:23.449+0000 7f8f8affd640 1 -- 172.21.15.29:0/901610644 <== mon.2 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f9c004240 con 0x561f6ec38a50 2024-03-16T11:51:23.457 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:23.449+0000 7f8fb4bfa740 1 -- 172.21.15.29:0/901610644 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8fa00041f0 con 0x561f6ec38a50 2024-03-16T11:51:23.458 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:23.449+0000 7f8f8affd640 1 -- 172.21.15.29:0/901610644 <== mon.2 v2:172.21.15.152:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8469+0+0 (secure 0 0 0) 0x7f8f9c0043e0 con 0x561f6ec38a50 2024-03-16T11:51:23.458 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:23.449+0000 7f8f8affd640 1 -- 172.21.15.29:0/901610644 <== mon.2 v2:172.21.15.152:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f8f9c0046c0 con 0x561f6ec38a50 2024-03-16T11:51:23.458 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:23.449+0000 7f8fb4bfa740 10 client.25259 fetch_fsmap learned FSMap version 22 2024-03-16T11:51:23.458 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:23.449+0000 7f8fb4bfa740 1 -- 172.21.15.29:0/901610644 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x561f6e97d620 con 0x561f6ec38a50 2024-03-16T11:51:23.458 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:23.449+0000 7f8f8affd640 1 -- 172.21.15.29:0/901610644 <== mon.2 v2:172.21.15.152:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8fa00041f0 con 0x561f6ec38a50 2024-03-16T11:51:23.458 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:23.449+0000 7f8f8affd640 1 -- 172.21.15.29:0/901610644 <== mon.2 v2:172.21.15.152:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7f8f9c0035f0 con 0x561f6ec38a50 2024-03-16T11:51:23.458 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:23.449+0000 7f8fb4bfa740 10 client.25259 fetch_fsmap finished waiting for FSMap version 22 2024-03-16T11:51:23.458 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:23.449+0000 7f8fb4bfa740 10 client.25259 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T11:51:23.458 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:23.449+0000 7f8fb4bfa740 20 client.25259 populate_metadata read hostname 'smithi029' 2024-03-16T11:51:23.458 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:23.449+0000 7f8fb4bfa740 1 --2- 172.21.15.29:0/901610644 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x561f6ec51950 0x561f6ec53d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:51:23.458 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:23.449+0000 7f8fb4bfa740 4 client.25259 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T11:51:23.459 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:23.449+0000 7f8fb4bfa740 1 -- 172.21.15.29:0/901610644 --> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561f6eba9f80 con 0x561f6ec51950 2024-03-16T11:51:23.459 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:23.450+0000 7f8faaffd640 1 --2- 172.21.15.29:0/901610644 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x561f6ec51950 0x561f6ec53d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:51:23.459 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:23.450+0000 7f8f8affd640 10 client.25259 ms_handle_connect on v2:172.21.15.152:6838/1548731833 2024-03-16T11:51:23.459 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:23.450+0000 7f8faaffd640 1 --2- 172.21.15.29:0/901610644 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x561f6ec51950 0x561f6ec53d60 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7ba50d74339a4840 server_cookie=b49c6ce6c0f4763a in_seq=0 out_seq=0 2024-03-16T11:51:23.459 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:23.451+0000 7f8f8affd640 1 -- 172.21.15.29:0/901610644 <== mds.0 v2:172.21.15.152:6838/1548731833 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x561f6eba9f80 con 0x561f6ec51950 2024-03-16T11:51:23.459 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:23.451+0000 7f8f8affd640 10 client.25259 handle_command_reply: tid=0 2024-03-16T11:51:23.459 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:23.451+0000 7f8fb4bfa740 1 client.25259 shutdown 2024-03-16T11:51:23.459 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:23.451+0000 7f8fb4bfa740 1 -- 172.21.15.29:0/901610644 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561f6ec38a50 msgr2=0x561f6ec3ed50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:51:23.459 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:23.451+0000 7f8fb4bfa740 1 --2- 172.21.15.29:0/901610644 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561f6ec38a50 0x561f6ec3ed50 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f8f9c02cfd0 tx=0x7f8f9c009560 comp rx=0 tx=0).stop 2024-03-16T11:51:23.459 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:23.451+0000 7f8fb4bfa740 1 -- 172.21.15.29:0/901610644 shutdown_connections 2024-03-16T11:51:23.459 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:23.451+0000 7f8fb4bfa740 1 --2- 172.21.15.29:0/901610644 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561f6ec41820 0x561f6ec3f290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:51:23.459 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:23.451+0000 7f8fb4bfa740 1 --2- 172.21.15.29:0/901610644 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x561f6ec51950 0x561f6ec53d60 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:51:23.460 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:23.451+0000 7f8fb4bfa740 1 --2- 172.21.15.29:0/901610644 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561f6ec38a50 0x561f6ec3ed50 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:51:23.460 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:23.451+0000 7f8fb4bfa740 1 --2- 172.21.15.29:0/901610644 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561f6ec38130 0x561f6ec3e810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:51:23.460 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:23.451+0000 7f8f8affd640 0 client.25259 ms_handle_reset on v2:172.21.15.152:6838/1548731833 2024-03-16T11:51:23.460 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:23.451+0000 7f8fb4bfa740 1 -- 172.21.15.29:0/901610644 >> 172.21.15.29:0/901610644 conn(0x561f6ec32a50 msgr2=0x561f6ec347a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:51:23.460 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:23.451+0000 7f8fb4bfa740 1 -- 172.21.15.29:0/901610644 shutdown_connections 2024-03-16T11:51:23.460 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:23.451+0000 7f8fb4bfa740 1 -- 172.21.15.29:0/901610644 wait complete. 2024-03-16T11:51:23.460 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:23.451+0000 7f8fb4bfa740 20 client.25259 trim_cache size 0 max 16384 2024-03-16T11:51:23.467 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:23.466+0000 7f8fb4bfa740 1 Processor -- start 2024-03-16T11:51:23.467 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:23.466+0000 7f8fb4bfa740 1 -- start start 2024-03-16T11:51:23.467 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:23.466+0000 7f8fb4bfa740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561f6ec38130 0x561f6ec40ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:51:23.467 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:23.466+0000 7f8fb4bfa740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561f6ec38a50 0x561f6ec4d1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:51:23.467 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:23.467+0000 7f8fb4bfa740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561f6ec41820 0x561f6ec59670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:51:23.467 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:23.467+0000 7f8fb4bfa740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f8fac1afc80 con 0x561f6ec41820 2024-03-16T11:51:23.467 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:23.467+0000 7f8fb4bfa740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f8fac1ad550 con 0x561f6ec38a50 2024-03-16T11:51:23.467 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:23.467+0000 7f8fb4bfa740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f8fac1ab3a0 con 0x561f6ec38130 2024-03-16T11:51:23.467 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:23.467+0000 7f8faaffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561f6ec41820 0x561f6ec59670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:51:23.467 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:23.467+0000 7f8fab7fe640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561f6ec38a50 0x561f6ec4d1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:51:23.467 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:23.467+0000 7f8fab7fe640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561f6ec38a50 0x561f6ec4d1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.29:57550/0 (socket says 172.21.15.29:57550) 2024-03-16T11:51:23.468 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:23.467+0000 7f8fab7fe640 1 -- 172.21.15.29:0/494740826 learned_addr learned my addr 172.21.15.29:0/494740826 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:51:23.468 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:23.467+0000 7f8faaffd640 1 --2- 172.21.15.29:0/494740826 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561f6ec38130 0x561f6ec40ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:51:23.468 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:23.467+0000 7f8faaffd640 1 -- 172.21.15.29:0/494740826 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561f6ec38130 msgr2=0x561f6ec40ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:51:23.468 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:23.467+0000 7f8faaffd640 1 --2- 172.21.15.29:0/494740826 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561f6ec38130 0x561f6ec40ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:51:23.468 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:23.467+0000 7f8faaffd640 1 -- 172.21.15.29:0/494740826 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561f6ec38a50 msgr2=0x561f6ec4d1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:51:23.468 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:23.467+0000 7f8faaffd640 1 --2- 172.21.15.29:0/494740826 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561f6ec38a50 0x561f6ec4d1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:51:23.468 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:23.467+0000 7f8faaffd640 1 -- 172.21.15.29:0/494740826 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561f6ec32880 con 0x561f6ec41820 2024-03-16T11:51:23.468 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:23.467+0000 7f8faaffd640 1 --2- 172.21.15.29:0/494740826 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561f6ec41820 0x561f6ec59670 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f8fa0003f60 tx=0x7f8fa0009640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c31c2f8352791d7d server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:51:23.472 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:23.468+0000 7f8f917fa640 1 -- 172.21.15.29:0/494740826 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fa0028040 con 0x561f6ec41820 2024-03-16T11:51:23.473 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:23.468+0000 7f8f917fa640 1 -- 172.21.15.29:0/494740826 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8fa0032600 con 0x561f6ec41820 2024-03-16T11:51:23.473 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:23.468+0000 7f8f917fa640 1 -- 172.21.15.29:0/494740826 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fa0003bc0 con 0x561f6ec41820 2024-03-16T11:51:23.473 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:23.469+0000 7f8fb4bfa740 1 -- 172.21.15.29:0/494740826 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561f6ec41820 msgr2=0x561f6ec59670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:51:23.473 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:23.469+0000 7f8fb4bfa740 1 --2- 172.21.15.29:0/494740826 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561f6ec41820 0x561f6ec59670 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f8fa0003f60 tx=0x7f8fa0009640 comp rx=0 tx=0).stop 2024-03-16T11:51:23.473 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:23.469+0000 7f8fb4bfa740 1 -- 172.21.15.29:0/494740826 shutdown_connections 2024-03-16T11:51:23.473 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:23.469+0000 7f8fb4bfa740 1 --2- 172.21.15.29:0/494740826 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561f6ec41820 0x561f6ec59670 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:51:23.473 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:23.469+0000 7f8fb4bfa740 1 --2- 172.21.15.29:0/494740826 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561f6ec38a50 0x561f6ec4d1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:51:23.473 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:23.469+0000 7f8fb4bfa740 1 --2- 172.21.15.29:0/494740826 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561f6ec38130 0x561f6ec40ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:51:23.473 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:23.469+0000 7f8fb4bfa740 1 -- 172.21.15.29:0/494740826 >> 172.21.15.29:0/494740826 conn(0x561f6ec32a50 msgr2=0x561f6ec3e7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:51:23.473 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:23.469+0000 7f8fb4bfa740 1 -- 172.21.15.29:0/494740826 shutdown_connections 2024-03-16T11:51:23.473 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:23.469+0000 7f8fb4bfa740 1 -- 172.21.15.29:0/494740826 wait complete. 2024-03-16T11:51:23.473 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:23.469+0000 7f8fb4bfa740 1 -- start start 2024-03-16T11:51:23.473 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:23.470+0000 7f8fb4bfa740 1 Processor -- start 2024-03-16T11:51:23.474 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:23.470+0000 7f8fb4bfa740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561f6ec38130 0x561f6ec45d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:51:23.474 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:23.470+0000 7f8fb4bfa740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561f6ec38a50 0x561f6ec462b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:51:23.474 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:23.470+0000 7f8fb4bfa740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561f6ec41820 0x561f6ecafc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:51:23.474 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:23.470+0000 7f8fb4bfa740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f8fac1aba10 con 0x561f6ec38130 2024-03-16T11:51:23.474 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:23.470+0000 7f8fb4bfa740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f8fac1ace80 con 0x561f6ec38a50 2024-03-16T11:51:23.474 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:23.470+0000 7f8fb4bfa740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f8fac1ac0e0 con 0x561f6ec41820 2024-03-16T11:51:23.474 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:23.470+0000 7f8fb4bfa740 10 client.0 _ll_register_callbacks cb 0x7f8f9c009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T11:51:23.474 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:23.470+0000 7f8faaffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561f6ec38130 0x561f6ec45d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:51:23.474 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:23.470+0000 7f8faaffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561f6ec38130 0x561f6ec45d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33680/0 (socket says 172.21.15.29:33680) 2024-03-16T11:51:23.474 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:23.470+0000 7f8fab7fe640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561f6ec38a50 0x561f6ec462b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:51:23.474 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:23.470+0000 7f8faaffd640 1 -- 172.21.15.29:0/976547289 learned_addr learned my addr 172.21.15.29:0/976547289 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:51:23.474 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:23.470+0000 7f8faaffd640 1 --2- 172.21.15.29:0/976547289 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561f6ec41820 0x561f6ecafc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:51:23.475 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:23.470+0000 7f8faaffd640 1 -- 172.21.15.29:0/976547289 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561f6ec41820 msgr2=0x561f6ecafc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:51:23.475 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:23.470+0000 7f8faaffd640 1 --2- 172.21.15.29:0/976547289 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561f6ec41820 0x561f6ecafc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:51:23.475 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:23.470+0000 7f8faaffd640 1 -- 172.21.15.29:0/976547289 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561f6ec38a50 msgr2=0x561f6ec462b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:51:23.475 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:23.470+0000 7f8faaffd640 1 --2- 172.21.15.29:0/976547289 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561f6ec38a50 0x561f6ec462b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:51:23.475 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:23.470+0000 7f8faaffd640 1 -- 172.21.15.29:0/976547289 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561f6ec4d730 con 0x561f6ec38130 2024-03-16T11:51:23.475 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:23.470+0000 7f8fab7fe640 1 --2- 172.21.15.29:0/976547289 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561f6ec38a50 0x561f6ec462b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:51:23.475 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:23.470+0000 7f8faaffd640 1 --2- 172.21.15.29:0/976547289 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561f6ec38130 0x561f6ec45d70 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f8fa0013e50 tx=0x7f8fa0005990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96678f7078586dd7 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:51:23.475 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:23.470+0000 7f8f8affd640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-16T11:51:23.475 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:23.471+0000 7f8f8affd640 1 -- 172.21.15.29:0/976547289 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fa0028040 con 0x561f6ec38130 2024-03-16T11:51:23.475 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:23.471+0000 7f8f8affd640 1 -- 172.21.15.29:0/976547289 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8469+0+0 (secure 0 0 0) 0x7f8fa0003940 con 0x561f6ec38130 2024-03-16T11:51:23.475 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:23.471+0000 7f8fb4bfa740 1 -- 172.21.15.29:0/976547289 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8fa0032600 con 0x561f6ec38130 2024-03-16T11:51:23.475 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:23.471+0000 7f8f8affd640 1 -- 172.21.15.29:0/976547289 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f8fa0003b40 con 0x561f6ec38130 2024-03-16T11:51:23.475 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:23.471+0000 7f8fb4bfa740 10 client.15450 fetch_fsmap learned FSMap version 22 2024-03-16T11:51:23.476 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:23.471+0000 7f8fb4bfa740 1 -- 172.21.15.29:0/976547289 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x561f6ecb85d0 con 0x561f6ec38130 2024-03-16T11:51:23.476 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:23.471+0000 7f8f8affd640 1 -- 172.21.15.29:0/976547289 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8fa0032600 con 0x561f6ec38130 2024-03-16T11:51:23.476 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:23.471+0000 7f8f8affd640 1 -- 172.21.15.29:0/976547289 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7f8fa002e030 con 0x561f6ec38130 2024-03-16T11:51:23.476 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:23.471+0000 7f8fb4bfa740 10 client.15450 fetch_fsmap finished waiting for FSMap version 22 2024-03-16T11:51:23.476 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:23.471+0000 7f8fb4bfa740 10 client.15450 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T11:51:23.476 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:23.471+0000 7f8fb4bfa740 20 client.15450 populate_metadata read hostname 'smithi029' 2024-03-16T11:51:23.476 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:23.471+0000 7f8fb4bfa740 1 --2- 172.21.15.29:0/976547289 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x561f6ecb8a40 0x561f6ecbae30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:51:23.476 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:23.471+0000 7f8fb4bfa740 4 client.15450 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-16T11:51:23.476 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:23.471+0000 7f8fb4bfa740 1 -- 172.21.15.29:0/976547289 --> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] -- command(tid 0: {"prefix": "flush journal"}) -- 0x561f6eba42d0 con 0x561f6ecb8a40 2024-03-16T11:51:23.476 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:23.471+0000 7f8fab7fe640 1 --2- 172.21.15.29:0/976547289 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x561f6ecb8a40 0x561f6ecbae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:51:23.476 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:23.472+0000 7f8f8affd640 10 client.15450 ms_handle_connect on v2:172.21.15.152:6838/1548731833 2024-03-16T11:51:23.476 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:23.472+0000 7f8fab7fe640 1 --2- 172.21.15.29:0/976547289 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x561f6ecb8a40 0x561f6ecbae30 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=52368e3f2e90b623 server_cookie=363bb46aba95a2dc in_seq=0 out_seq=0 2024-03-16T11:51:23.478 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:23.478+0000 7f8f8affd640 1 -- 172.21.15.29:0/976547289 <== mds.0 v2:172.21.15.152:6838/1548731833 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x561f6eba42d0 con 0x561f6ecb8a40 2024-03-16T11:51:23.478 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:23.478+0000 7f8f8affd640 10 client.15450 handle_command_reply: tid=0 2024-03-16T11:51:23.478 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:23.478+0000 7f8fb4bfa740 1 client.15450 shutdown 2024-03-16T11:51:23.478 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:23.478+0000 7f8fb4bfa740 1 -- 172.21.15.29:0/976547289 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561f6ec38130 msgr2=0x561f6ec45d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:51:23.478 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:23.478+0000 7f8fb4bfa740 1 --2- 172.21.15.29:0/976547289 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561f6ec38130 0x561f6ec45d70 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f8fa0013e50 tx=0x7f8fa0005990 comp rx=0 tx=0).stop 2024-03-16T11:51:23.478 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:23.478+0000 7f8fb4bfa740 1 -- 172.21.15.29:0/976547289 shutdown_connections 2024-03-16T11:51:23.478 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:23.478+0000 7f8fb4bfa740 1 --2- 172.21.15.29:0/976547289 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561f6ec41820 0x561f6ecafc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:51:23.478 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:23.478+0000 7f8fb4bfa740 1 --2- 172.21.15.29:0/976547289 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x561f6ecb8a40 0x561f6ecbae30 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:51:23.479 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:23.478+0000 7f8fb4bfa740 1 --2- 172.21.15.29:0/976547289 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561f6ec38a50 0x561f6ec462b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:51:23.479 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:23.478+0000 7f8fb4bfa740 1 --2- 172.21.15.29:0/976547289 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561f6ec38130 0x561f6ec45d70 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:51:23.479 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:23.478+0000 7f8f8affd640 0 client.15450 ms_handle_reset on v2:172.21.15.152:6838/1548731833 2024-03-16T11:51:23.479 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:23.478+0000 7f8fb4bfa740 1 -- 172.21.15.29:0/976547289 >> 172.21.15.29:0/976547289 conn(0x561f6ec32a50 msgr2=0x561f6ec31bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:51:23.479 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:23.478+0000 7f8fb4bfa740 1 -- 172.21.15.29:0/976547289 shutdown_connections 2024-03-16T11:51:23.479 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:23.478+0000 7f8fb4bfa740 1 -- 172.21.15.29:0/976547289 wait complete. 2024-03-16T11:51:23.479 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:23.479+0000 7f8fb4bfa740 20 client.15450 trim_cache size 0 max 16384 2024-03-16T11:51:23.479 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:{ 2024-03-16T11:51:23.479 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "message": "", 2024-03-16T11:51:23.480 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "return_code": 0 2024-03-16T11:51:23.480 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:} 2024-03-16T11:51:23.480 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:23.479+0000 7f8fb1f37640 1 -- 172.21.15.29:0/2608965090 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f8f940a2b30 msgr2=0x7f8f940a4ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:51:23.480 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:23.479+0000 7f8fb1f37640 1 --2- 172.21.15.29:0/2608965090 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f8f940a2b30 0x7f8f940a4ff0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f8fa40ccda0 tx=0x7f8fa40057d0 comp rx=0 tx=0).stop 2024-03-16T11:51:23.481 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:23.479+0000 7f8fb1f37640 1 -- 172.21.15.29:0/2608965090 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8fac19ca40 msgr2=0x7f8fac19cef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:51:23.481 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:23.479+0000 7f8fb1f37640 1 --2- 172.21.15.29:0/2608965090 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8fac19ca40 0x7f8fac19cef0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f8fa400eb10 tx=0x7f8fa400c2d0 comp rx=0 tx=0).stop 2024-03-16T11:51:23.481 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:23.479+0000 7f8fb1f37640 1 -- 172.21.15.29:0/2608965090 shutdown_connections 2024-03-16T11:51:23.481 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:23.479+0000 7f8fb1f37640 1 --2- 172.21.15.29:0/2608965090 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f8f940a2b30 0x7f8f940a4ff0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:51:23.481 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:23.479+0000 7f8fb1f37640 1 --2- 172.21.15.29:0/2608965090 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8fac19ca40 0x7f8fac19cef0 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:51:23.481 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:23.479+0000 7f8fb1f37640 1 --2- 172.21.15.29:0/2608965090 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8fac10bf60 0x7f8fac198710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:51:23.481 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:23.479+0000 7f8fb1f37640 1 --2- 172.21.15.29:0/2608965090 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8fac102a50 0x7f8fac1981d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:51:23.481 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:23.479+0000 7f8fb1f37640 1 -- 172.21.15.29:0/2608965090 >> 172.21.15.29:0/2608965090 conn(0x7f8fac077800 msgr2=0x7f8fac0fd790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:51:23.481 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:23.479+0000 7f8fb1f37640 1 -- 172.21.15.29:0/2608965090 shutdown_connections 2024-03-16T11:51:23.481 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:23.479+0000 7f8fb1f37640 1 -- 172.21.15.29:0/2608965090 wait complete. 2024-03-16T11:51:24.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:51:24 smithi154 ceph-mon[36320]: pgmap v17: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 2.2 KiB/s wr, 11 op/s 2024-03-16T11:51:24.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:51:24 smithi029 ceph-mon[31975]: pgmap v17: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 2.2 KiB/s wr, 11 op/s 2024-03-16T11:51:24.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:51:24 smithi152 ceph-mon[36291]: pgmap v17: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 2.2 KiB/s wr, 11 op/s 2024-03-16T11:51:26.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:51:26 smithi154 ceph-mon[36320]: pgmap v18: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-16T11:51:26.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:51:26 smithi029 ceph-mon[31975]: pgmap v18: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-16T11:51:26.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:51:26 smithi152 ceph-mon[36291]: pgmap v18: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-16T11:51:28.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:51:28 smithi154 ceph-mon[36320]: pgmap v19: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 597 B/s wr, 8 op/s 2024-03-16T11:51:28.837 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:51:28 smithi029 ceph-mon[31975]: pgmap v19: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 597 B/s wr, 8 op/s 2024-03-16T11:51:28.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:51:28 smithi152 ceph-mon[36291]: pgmap v19: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 597 B/s wr, 8 op/s 2024-03-16T11:51:30.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:51:30 smithi154 ceph-mon[36320]: pgmap v20: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 597 B/s wr, 6 op/s 2024-03-16T11:51:30.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:51:30 smithi029 ceph-mon[31975]: pgmap v20: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 597 B/s wr, 6 op/s 2024-03-16T11:51:30.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:51:30 smithi152 ceph-mon[36291]: pgmap v20: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 597 B/s wr, 6 op/s 2024-03-16T11:51:32.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:51:32 smithi154 ceph-mon[36320]: pgmap v21: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 682 B/s wr, 6 op/s 2024-03-16T11:51:32.837 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:51:32 smithi029 ceph-mon[31975]: pgmap v21: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 682 B/s wr, 6 op/s 2024-03-16T11:51:32.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:51:32 smithi152 ceph-mon[36291]: pgmap v21: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 682 B/s wr, 6 op/s 2024-03-16T11:51:34.797 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:51:34 smithi154 ceph-mon[36320]: pgmap v22: 97 pgs: 97 active+clean; 1.4 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 1.9 KiB/s wr, 8 op/s 2024-03-16T11:51:34.837 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:51:34 smithi029 ceph-mon[31975]: pgmap v22: 97 pgs: 97 active+clean; 1.4 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 1.9 KiB/s wr, 8 op/s 2024-03-16T11:51:34.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:51:34 smithi152 ceph-mon[36291]: pgmap v22: 97 pgs: 97 active+clean; 1.4 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 1.9 KiB/s wr, 8 op/s 2024-03-16T11:51:36.569 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:36.567+0000 7fc8bf0ff640 1 -- 172.21.15.29:0/2363049255 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc8b8102150 msgr2=0x7fc8b810e660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:51:36.569 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:36.567+0000 7fc8bf0ff640 1 --2- 172.21.15.29:0/2363049255 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc8b8102150 0x7fc8b810e660 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fc8b0009cd0 tx=0x7fc8b002c0f0 comp rx=0 tx=0).stop 2024-03-16T11:51:36.569 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:36.567+0000 7fc8bf0ff640 1 -- 172.21.15.29:0/2363049255 shutdown_connections 2024-03-16T11:51:36.569 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:36.567+0000 7fc8bf0ff640 1 --2- 172.21.15.29:0/2363049255 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc8b8102150 0x7fc8b810e660 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:51:36.569 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:36.567+0000 7fc8bf0ff640 1 --2- 172.21.15.29:0/2363049255 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc8b8101790 0x7fc8b8101c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:51:36.569 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:36.567+0000 7fc8bf0ff640 1 --2- 172.21.15.29:0/2363049255 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc8b81077a0 0x7fc8b8107b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:51:36.570 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:36.567+0000 7fc8bf0ff640 1 -- 172.21.15.29:0/2363049255 >> 172.21.15.29:0/2363049255 conn(0x7fc8b80fd2a0 msgr2=0x7fc8b80ff6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:51:36.570 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:36.567+0000 7fc8bf0ff640 1 -- 172.21.15.29:0/2363049255 shutdown_connections 2024-03-16T11:51:36.570 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:36.567+0000 7fc8bf0ff640 1 -- 172.21.15.29:0/2363049255 wait complete. 2024-03-16T11:51:36.570 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:36.567+0000 7fc8bf0ff640 1 Processor -- start 2024-03-16T11:51:36.570 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:36.567+0000 7fc8bf0ff640 1 -- start start 2024-03-16T11:51:36.570 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:36.568+0000 7fc8bf0ff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc8b8101790 0x7fc8b8197fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:51:36.570 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:36.568+0000 7fc8bf0ff640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc8b8102150 0x7fc8b81984f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:51:36.570 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:36.568+0000 7fc8bf0ff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc8b81077a0 0x7fc8b819c820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:51:36.570 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:36.568+0000 7fc8bf0ff640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fc8b8111bb0 con 0x7fc8b8101790 2024-03-16T11:51:36.571 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:36.568+0000 7fc8bf0ff640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fc8b8111a30 con 0x7fc8b8102150 2024-03-16T11:51:36.571 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:36.568+0000 7fc8bf0ff640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fc8b8111d30 con 0x7fc8b81077a0 2024-03-16T11:51:36.571 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:36.568+0000 7fc8bce9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc8b8101790 0x7fc8b8197fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:51:36.571 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:36.568+0000 7fc8bce9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc8b8101790 0x7fc8b8197fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50996/0 (socket says 172.21.15.29:50996) 2024-03-16T11:51:36.571 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:36.568+0000 7fc8bce9b640 1 -- 172.21.15.29:0/1379040248 learned_addr learned my addr 172.21.15.29:0/1379040248 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:51:36.571 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:36.568+0000 7fc8b7fff640 1 --2- 172.21.15.29:0/1379040248 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc8b8102150 0x7fc8b81984f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:51:36.571 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:36.568+0000 7fc8bd69c640 1 --2- 172.21.15.29:0/1379040248 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc8b81077a0 0x7fc8b819c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:51:36.571 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:36.568+0000 7fc8bce9b640 1 -- 172.21.15.29:0/1379040248 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc8b81077a0 msgr2=0x7fc8b819c820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:51:36.572 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:36.568+0000 7fc8bce9b640 1 --2- 172.21.15.29:0/1379040248 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc8b81077a0 0x7fc8b819c820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:51:36.572 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:36.568+0000 7fc8bce9b640 1 -- 172.21.15.29:0/1379040248 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc8b8102150 msgr2=0x7fc8b81984f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:51:36.572 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:36.568+0000 7fc8bce9b640 1 --2- 172.21.15.29:0/1379040248 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc8b8102150 0x7fc8b81984f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:51:36.572 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:36.568+0000 7fc8bce9b640 1 -- 172.21.15.29:0/1379040248 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8b819cf50 con 0x7fc8b8101790 2024-03-16T11:51:36.572 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:36.568+0000 7fc8b7fff640 1 --2- 172.21.15.29:0/1379040248 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc8b8102150 0x7fc8b81984f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:51:36.572 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:36.568+0000 7fc8bce9b640 1 --2- 172.21.15.29:0/1379040248 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc8b8101790 0x7fc8b8197fb0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fc8a800e9f0 tx=0x7fc8a800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:51:36.572 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:36.568+0000 7fc8b5ffb640 1 -- 172.21.15.29:0/1379040248 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8a800cd80 con 0x7fc8b8101790 2024-03-16T11:51:36.572 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:36.568+0000 7fc8b5ffb640 1 -- 172.21.15.29:0/1379040248 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc8a8005500 con 0x7fc8b8101790 2024-03-16T11:51:36.572 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:36.568+0000 7fc8bf0ff640 1 -- 172.21.15.29:0/1379040248 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc8b819d240 con 0x7fc8b8101790 2024-03-16T11:51:36.573 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:36.568+0000 7fc8b5ffb640 1 -- 172.21.15.29:0/1379040248 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8a8010690 con 0x7fc8b8101790 2024-03-16T11:51:36.573 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:36.569+0000 7fc8bf0ff640 1 -- 172.21.15.29:0/1379040248 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc8b81a4ad0 con 0x7fc8b8101790 2024-03-16T11:51:36.583 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:36.571+0000 7fc8b5ffb640 1 -- 172.21.15.29:0/1379040248 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 24) ==== 137523+0+0 (secure 0 0 0) 0x7fc8a80046c0 con 0x7fc8b8101790 2024-03-16T11:51:36.583 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:36.571+0000 7fc8c1dc6740 1 Processor -- start 2024-03-16T11:51:36.583 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:36.571+0000 7fc8c1dc6740 1 -- start start 2024-03-16T11:51:36.584 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:36.571+0000 7fc8c1dc6740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563289755130 0x563289755510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:51:36.584 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:36.571+0000 7fc8c1dc6740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563289755a50 0x56328975e2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:51:36.584 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:36.571+0000 7fc8c1dc6740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56328975e820 0x563289760c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:51:36.584 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:36.571+0000 7fc8c1dc6740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x5632896b9a40 con 0x56328975e820 2024-03-16T11:51:36.584 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:36.571+0000 7fc8c1dc6740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x563289662920 con 0x563289755a50 2024-03-16T11:51:36.584 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:36.571+0000 7fc8c1dc6740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5632896b5ae0 con 0x563289755130 2024-03-16T11:51:36.584 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:36.572+0000 7fc8b5ffb640 1 --2- 172.21.15.29:0/1379040248 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7fc8a00a2b30 0x7fc8a00a4ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:51:36.584 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:36.572+0000 7fc8b7fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56328975e820 0x563289760c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:51:36.584 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:36.572+0000 7fc8b7fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56328975e820 0x563289760c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51006/0 (socket says 172.21.15.29:51006) 2024-03-16T11:51:36.584 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:36.572+0000 7fc8b7fff640 1 -- 172.21.15.29:0/3083564827 learned_addr learned my addr 172.21.15.29:0/3083564827 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:51:36.584 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:36.572+0000 7fc8bce9b640 1 --2- 172.21.15.29:0/3083564827 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563289755a50 0x56328975e2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:51:36.584 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:36.572+0000 7fc8bd69c640 1 --2- 172.21.15.29:0/3083564827 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563289755130 0x563289755510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:51:36.584 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:36.572+0000 7fc8bd69c640 1 --2- 172.21.15.29:0/1379040248 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7fc8a00a2b30 0x7fc8a00a4ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:51:36.584 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:36.572+0000 7fc8b5ffb640 1 -- 172.21.15.29:0/1379040248 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8469+0+0 (secure 0 0 0) 0x7fc8a80cf480 con 0x7fc8b8101790 2024-03-16T11:51:36.585 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:36.572+0000 7fc8b7fff640 1 -- 172.21.15.29:0/3083564827 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563289755130 msgr2=0x563289755510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:51:36.585 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:36.572+0000 7fc8b7fff640 1 --2- 172.21.15.29:0/3083564827 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563289755130 0x563289755510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:51:36.585 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:36.572+0000 7fc8b7fff640 1 -- 172.21.15.29:0/3083564827 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563289755a50 msgr2=0x56328975e2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:51:36.585 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:36.572+0000 7fc8b7fff640 1 --2- 172.21.15.29:0/3083564827 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563289755a50 0x56328975e2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:51:36.585 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:36.572+0000 7fc8b7fff640 1 -- 172.21.15.29:0/3083564827 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56328967a460 con 0x56328975e820 2024-03-16T11:51:36.585 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:36.572+0000 7fc8bd69c640 1 --2- 172.21.15.29:0/3083564827 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563289755130 0x563289755510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:51:36.585 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:36.572+0000 7fc8bd69c640 1 --2- 172.21.15.29:0/1379040248 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7fc8a00a2b30 0x7fc8a00a4ff0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fc8b0005cc0 tx=0x7fc8b0005c50 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:51:36.585 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:36.572+0000 7fc8b7fff640 1 --2- 172.21.15.29:0/3083564827 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56328975e820 0x563289760c10 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fc8ac0114f0 tx=0x7fc8ac0119b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17549b1c09d483ad server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:51:36.585 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:36.572+0000 7fc89effd640 1 -- 172.21.15.29:0/3083564827 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8ac01e070 con 0x56328975e820 2024-03-16T11:51:36.585 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:36.572+0000 7fc89effd640 1 -- 172.21.15.29:0/3083564827 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc8ac0035f0 con 0x56328975e820 2024-03-16T11:51:36.585 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:36.572+0000 7fc89effd640 1 -- 172.21.15.29:0/3083564827 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8ac00f800 con 0x56328975e820 2024-03-16T11:51:36.585 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:36.573+0000 7fc8c1dc6740 1 -- 172.21.15.29:0/3083564827 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56328975e820 msgr2=0x563289760c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:51:36.585 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:36.573+0000 7fc8c1dc6740 1 --2- 172.21.15.29:0/3083564827 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56328975e820 0x563289760c10 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fc8ac0114f0 tx=0x7fc8ac0119b0 comp rx=0 tx=0).stop 2024-03-16T11:51:36.585 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:36.573+0000 7fc8c1dc6740 1 -- 172.21.15.29:0/3083564827 shutdown_connections 2024-03-16T11:51:36.586 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:36.573+0000 7fc8c1dc6740 1 --2- 172.21.15.29:0/3083564827 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56328975e820 0x563289760c10 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:51:36.586 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:36.573+0000 7fc8c1dc6740 1 --2- 172.21.15.29:0/3083564827 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563289755a50 0x56328975e2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:51:36.586 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:36.573+0000 7fc8c1dc6740 1 --2- 172.21.15.29:0/3083564827 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563289755130 0x563289755510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:51:36.586 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:36.573+0000 7fc8c1dc6740 1 -- 172.21.15.29:0/3083564827 >> 172.21.15.29:0/3083564827 conn(0x56328974fa50 msgr2=0x56328974fe30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:51:36.586 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:36.573+0000 7fc8c1dc6740 1 -- 172.21.15.29:0/3083564827 shutdown_connections 2024-03-16T11:51:36.586 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:36.573+0000 7fc8c1dc6740 1 -- 172.21.15.29:0/3083564827 wait complete. 2024-03-16T11:51:36.586 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:36.573+0000 7fc8c1dc6740 1 -- start start 2024-03-16T11:51:36.586 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:36.574+0000 7fc8c1dc6740 1 Processor -- start 2024-03-16T11:51:36.586 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:36.574+0000 7fc8c1dc6740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563289755130 0x56328975b850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:51:36.586 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:36.574+0000 7fc8c1dc6740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563289755a50 0x56328975bd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:51:36.586 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:36.574+0000 7fc8c1dc6740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56328975e820 0x56328975c2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:51:36.587 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:36.574+0000 7fc8c1dc6740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x563289761450 con 0x563289755a50 2024-03-16T11:51:36.587 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:36.574+0000 7fc8c1dc6740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x563289675d70 con 0x56328975e820 2024-03-16T11:51:36.587 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:36.574+0000 7fc8c1dc6740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5632897615d0 con 0x563289755130 2024-03-16T11:51:36.587 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:36.574+0000 7fc8c1dc6740 10 client.0 _ll_register_callbacks cb 0x5632893cda00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T11:51:36.587 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:36.574+0000 7fc8b7fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563289755a50 0x56328975bd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:51:36.587 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:36.574+0000 7fc8b7fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563289755a50 0x56328975bd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51020/0 (socket says 172.21.15.29:51020) 2024-03-16T11:51:36.587 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:36.574+0000 7fc8b7fff640 1 -- 172.21.15.29:0/602427276 learned_addr learned my addr 172.21.15.29:0/602427276 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:51:36.587 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:36.574+0000 7fc8bce9b640 1 --2- 172.21.15.29:0/602427276 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563289755130 0x56328975b850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:51:36.587 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:36.574+0000 7fc8bd69c640 1 --2- 172.21.15.29:0/602427276 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56328975e820 0x56328975c2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:51:36.587 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:36.574+0000 7fc8b7fff640 1 -- 172.21.15.29:0/602427276 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563289755130 msgr2=0x56328975b850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:51:36.587 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:36.574+0000 7fc8b7fff640 1 --2- 172.21.15.29:0/602427276 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563289755130 0x56328975b850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:51:36.587 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:36.574+0000 7fc8b7fff640 1 -- 172.21.15.29:0/602427276 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56328975e820 msgr2=0x56328975c2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:51:36.587 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:36.574+0000 7fc8b7fff640 1 --2- 172.21.15.29:0/602427276 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56328975e820 0x56328975c2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:51:36.587 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:36.574+0000 7fc8b7fff640 1 -- 172.21.15.29:0/602427276 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5632893ce210 con 0x563289755a50 2024-03-16T11:51:36.588 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:36.575+0000 7fc8b7fff640 1 --2- 172.21.15.29:0/602427276 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563289755a50 0x56328975bd90 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fc8ac011eb0 tx=0x7fc8ac003a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8bf63505724f7f33 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:51:36.588 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:36.575+0000 7fc897fff640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-16T11:51:36.588 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:36.575+0000 7fc897fff640 1 -- 172.21.15.29:0/602427276 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8ac026020 con 0x563289755a50 2024-03-16T11:51:36.588 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:36.575+0000 7fc8c1dc6740 1 -- 172.21.15.29:0/602427276 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc8ac0035f0 con 0x563289755a50 2024-03-16T11:51:36.588 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:36.575+0000 7fc897fff640 1 -- 172.21.15.29:0/602427276 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8469+0+0 (secure 0 0 0) 0x7fc8ac005040 con 0x563289755a50 2024-03-16T11:51:36.588 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:36.575+0000 7fc897fff640 1 -- 172.21.15.29:0/602427276 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7fc8ac031050 con 0x563289755a50 2024-03-16T11:51:36.588 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:36.575+0000 7fc8c1dc6740 10 client.15462 fetch_fsmap learned FSMap version 22 2024-03-16T11:51:36.588 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:36.575+0000 7fc8c1dc6740 1 -- 172.21.15.29:0/602427276 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x56328949a620 con 0x563289755a50 2024-03-16T11:51:36.588 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:36.575+0000 7fc897fff640 1 -- 172.21.15.29:0/602427276 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc8ac0035f0 con 0x563289755a50 2024-03-16T11:51:36.588 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:36.575+0000 7fc897fff640 1 -- 172.21.15.29:0/602427276 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7fc8ac021030 con 0x563289755a50 2024-03-16T11:51:36.588 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:36.575+0000 7fc8c1dc6740 10 client.15462 fetch_fsmap finished waiting for FSMap version 22 2024-03-16T11:51:36.588 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:36.575+0000 7fc8c1dc6740 10 client.15462 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T11:51:36.588 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:36.575+0000 7fc8c1dc6740 20 client.15462 populate_metadata read hostname 'smithi029' 2024-03-16T11:51:36.588 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:36.575+0000 7fc8c1dc6740 1 --2- 172.21.15.29:0/602427276 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x56328976e890 0x563289770c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:51:36.589 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:36.575+0000 7fc8c1dc6740 4 client.15462 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T11:51:36.589 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:36.575+0000 7fc8c1dc6740 1 -- 172.21.15.29:0/602427276 --> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5632896c6f80 con 0x56328976e890 2024-03-16T11:51:36.589 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:36.575+0000 7fc8bce9b640 1 --2- 172.21.15.29:0/602427276 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x56328976e890 0x563289770c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:51:36.589 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:36.576+0000 7fc8bce9b640 1 --2- 172.21.15.29:0/602427276 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x56328976e890 0x563289770c80 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2ccf45de84abbf14 server_cookie=f803196dc6ec7455 in_seq=0 out_seq=0 2024-03-16T11:51:36.589 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:36.576+0000 7fc897fff640 10 client.15462 ms_handle_connect on v2:172.21.15.152:6838/1548731833 2024-03-16T11:51:36.589 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:36.577+0000 7fc897fff640 1 -- 172.21.15.29:0/602427276 <== mds.0 v2:172.21.15.152:6838/1548731833 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5632896c6f80 con 0x56328976e890 2024-03-16T11:51:36.589 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:36.577+0000 7fc897fff640 10 client.15462 handle_command_reply: tid=0 2024-03-16T11:51:36.589 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:36.577+0000 7fc8c1dc6740 1 client.15462 shutdown 2024-03-16T11:51:36.589 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:36.577+0000 7fc8c1dc6740 1 -- 172.21.15.29:0/602427276 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563289755a50 msgr2=0x56328975bd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:51:36.589 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:36.577+0000 7fc8c1dc6740 1 --2- 172.21.15.29:0/602427276 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563289755a50 0x56328975bd90 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fc8ac011eb0 tx=0x7fc8ac003a50 comp rx=0 tx=0).stop 2024-03-16T11:51:36.589 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:36.577+0000 7fc8c1dc6740 1 -- 172.21.15.29:0/602427276 shutdown_connections 2024-03-16T11:51:36.589 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:36.577+0000 7fc8c1dc6740 1 --2- 172.21.15.29:0/602427276 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x56328976e890 0x563289770c80 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:51:36.589 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:36.577+0000 7fc8c1dc6740 1 --2- 172.21.15.29:0/602427276 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56328975e820 0x56328975c2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:51:36.589 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:36.577+0000 7fc8c1dc6740 1 --2- 172.21.15.29:0/602427276 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563289755a50 0x56328975bd90 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:51:36.590 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:36.577+0000 7fc8c1dc6740 1 --2- 172.21.15.29:0/602427276 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563289755130 0x56328975b850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:51:36.590 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:36.577+0000 7fc897fff640 0 client.15462 ms_handle_reset on v2:172.21.15.152:6838/1548731833 2024-03-16T11:51:36.590 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:36.577+0000 7fc8c1dc6740 1 -- 172.21.15.29:0/602427276 >> 172.21.15.29:0/602427276 conn(0x56328974fa50 msgr2=0x5632897517a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:51:36.590 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:36.577+0000 7fc8c1dc6740 1 -- 172.21.15.29:0/602427276 shutdown_connections 2024-03-16T11:51:36.590 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:36.577+0000 7fc8c1dc6740 1 -- 172.21.15.29:0/602427276 wait complete. 2024-03-16T11:51:36.590 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:36.577+0000 7fc8c1dc6740 20 client.15462 trim_cache size 0 max 16384 2024-03-16T11:51:36.596 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:36.592+0000 7fc8c1dc6740 1 Processor -- start 2024-03-16T11:51:36.596 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:36.592+0000 7fc8c1dc6740 1 -- start start 2024-03-16T11:51:36.596 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:36.592+0000 7fc8c1dc6740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563289755130 0x56328975df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:51:36.596 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:36.592+0000 7fc8c1dc6740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563289755a50 0x56328976a1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:51:36.596 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:36.592+0000 7fc8c1dc6740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56328975e820 0x56328974ea30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:51:36.596 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:36.592+0000 7fc8c1dc6740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fc8b81ada10 con 0x563289755a50 2024-03-16T11:51:36.596 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:36.592+0000 7fc8c1dc6740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fc8b81ab2e0 con 0x563289755130 2024-03-16T11:51:36.596 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:36.592+0000 7fc8c1dc6740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fc8b81a9150 con 0x56328975e820 2024-03-16T11:51:36.596 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:36.592+0000 7fc8b7fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563289755a50 0x56328976a1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:51:36.596 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:36.592+0000 7fc8b7fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563289755a50 0x56328976a1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51034/0 (socket says 172.21.15.29:51034) 2024-03-16T11:51:36.596 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:36.592+0000 7fc8b7fff640 1 -- 172.21.15.29:0/484152092 learned_addr learned my addr 172.21.15.29:0/484152092 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:51:36.596 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:36.592+0000 7fc8bce9b640 1 --2- 172.21.15.29:0/484152092 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563289755130 0x56328975df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:51:36.596 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:36.592+0000 7fc8bd69c640 1 --2- 172.21.15.29:0/484152092 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56328975e820 0x56328974ea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:51:36.596 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:36.592+0000 7fc8b7fff640 1 -- 172.21.15.29:0/484152092 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56328975e820 msgr2=0x56328974ea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:51:36.597 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:36.592+0000 7fc8b7fff640 1 --2- 172.21.15.29:0/484152092 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56328975e820 0x56328974ea30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:51:36.597 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:36.592+0000 7fc8b7fff640 1 -- 172.21.15.29:0/484152092 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563289755130 msgr2=0x56328975df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:51:36.597 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:36.592+0000 7fc8b7fff640 1 --2- 172.21.15.29:0/484152092 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563289755130 0x56328975df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:51:36.597 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:36.592+0000 7fc8b7fff640 1 -- 172.21.15.29:0/484152092 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5632897548f0 con 0x563289755a50 2024-03-16T11:51:36.597 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:36.592+0000 7fc8bce9b640 1 --2- 172.21.15.29:0/484152092 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563289755130 0x56328975df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T11:51:36.597 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:36.592+0000 7fc8b7fff640 1 --2- 172.21.15.29:0/484152092 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563289755a50 0x56328976a1f0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fc8ac009730 tx=0x7fc8ac005e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da5e735ff45aea3b server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:51:36.597 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:36.593+0000 7fc89e7fc640 1 -- 172.21.15.29:0/484152092 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8ac026070 con 0x563289755a50 2024-03-16T11:51:36.597 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:36.593+0000 7fc89e7fc640 1 -- 172.21.15.29:0/484152092 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc8ac016070 con 0x563289755a50 2024-03-16T11:51:36.597 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:36.593+0000 7fc89e7fc640 1 -- 172.21.15.29:0/484152092 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8ac01e040 con 0x563289755a50 2024-03-16T11:51:36.597 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:36.593+0000 7fc8c1dc6740 1 -- 172.21.15.29:0/484152092 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563289755a50 msgr2=0x56328976a1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:51:36.597 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:36.593+0000 7fc8c1dc6740 1 --2- 172.21.15.29:0/484152092 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563289755a50 0x56328976a1f0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fc8ac009730 tx=0x7fc8ac005e70 comp rx=0 tx=0).stop 2024-03-16T11:51:36.597 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:36.593+0000 7fc8c1dc6740 1 -- 172.21.15.29:0/484152092 shutdown_connections 2024-03-16T11:51:36.597 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:36.593+0000 7fc8c1dc6740 1 --2- 172.21.15.29:0/484152092 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56328975e820 0x56328974ea30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:51:36.597 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:36.593+0000 7fc8c1dc6740 1 --2- 172.21.15.29:0/484152092 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563289755a50 0x56328976a1f0 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:51:36.598 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:36.593+0000 7fc8c1dc6740 1 --2- 172.21.15.29:0/484152092 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563289755130 0x56328975df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:51:36.598 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:36.593+0000 7fc8c1dc6740 1 -- 172.21.15.29:0/484152092 >> 172.21.15.29:0/484152092 conn(0x56328974fa50 msgr2=0x56328974e320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:51:36.598 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:36.593+0000 7fc8c1dc6740 1 -- 172.21.15.29:0/484152092 shutdown_connections 2024-03-16T11:51:36.598 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:36.593+0000 7fc8c1dc6740 1 -- 172.21.15.29:0/484152092 wait complete. 2024-03-16T11:51:36.598 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:36.593+0000 7fc8c1dc6740 1 -- start start 2024-03-16T11:51:36.598 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:36.593+0000 7fc8c1dc6740 1 Processor -- start 2024-03-16T11:51:36.598 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:36.594+0000 7fc8c1dc6740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563289755130 0x563289762090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:51:36.598 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:36.594+0000 7fc8c1dc6740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563289755a50 0x5632897625d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:51:36.598 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:36.594+0000 7fc8c1dc6740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56328975e820 0x563289762b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:51:36.598 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:36.594+0000 7fc8c1dc6740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fc8b81a97a0 con 0x563289755a50 2024-03-16T11:51:36.598 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:36.594+0000 7fc8c1dc6740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fc8b81aac10 con 0x563289755130 2024-03-16T11:51:36.598 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:36.594+0000 7fc8c1dc6740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fc8b81a9e70 con 0x56328975e820 2024-03-16T11:51:36.598 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:36.594+0000 7fc8c1dc6740 10 client.0 _ll_register_callbacks cb 0x7fc8ac01c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T11:51:36.598 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:36.594+0000 7fc8b7fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563289755a50 0x5632897625d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:51:36.599 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:36.594+0000 7fc8bce9b640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563289755130 0x563289762090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:51:36.599 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:36.594+0000 7fc8b7fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563289755a50 0x5632897625d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51048/0 (socket says 172.21.15.29:51048) 2024-03-16T11:51:36.599 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:36.594+0000 7fc8b7fff640 1 -- 172.21.15.29:0/2052669512 learned_addr learned my addr 172.21.15.29:0/2052669512 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:51:36.599 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:36.594+0000 7fc8bd69c640 1 --2- 172.21.15.29:0/2052669512 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56328975e820 0x563289762b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:51:36.599 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:36.594+0000 7fc8b7fff640 1 -- 172.21.15.29:0/2052669512 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56328975e820 msgr2=0x563289762b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:51:36.599 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:36.594+0000 7fc8b7fff640 1 --2- 172.21.15.29:0/2052669512 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56328975e820 0x563289762b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:51:36.599 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:36.594+0000 7fc8b7fff640 1 -- 172.21.15.29:0/2052669512 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563289755130 msgr2=0x563289762090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:51:36.599 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:36.594+0000 7fc8b7fff640 1 --2- 172.21.15.29:0/2052669512 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563289755130 0x563289762090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:51:36.599 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:36.594+0000 7fc8b7fff640 1 -- 172.21.15.29:0/2052669512 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5632897d3a20 con 0x563289755a50 2024-03-16T11:51:36.599 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:36.594+0000 7fc8bce9b640 1 --2- 172.21.15.29:0/2052669512 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563289755130 0x563289762090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:51:36.599 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:36.594+0000 7fc8b7fff640 1 --2- 172.21.15.29:0/2052669512 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563289755a50 0x5632897625d0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7fc8ac012ec0 tx=0x7fc8ac009d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=383c83d038f288c1 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:51:36.599 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:36.594+0000 7fc897fff640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-16T11:51:36.599 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:36.595+0000 7fc897fff640 1 -- 172.21.15.29:0/2052669512 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8ac01e030 con 0x563289755a50 2024-03-16T11:51:36.599 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:36.595+0000 7fc897fff640 1 -- 172.21.15.29:0/2052669512 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8469+0+0 (secure 0 0 0) 0x7fc8ac004550 con 0x563289755a50 2024-03-16T11:51:36.600 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:36.595+0000 7fc8c1dc6740 1 -- 172.21.15.29:0/2052669512 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc8ac016070 con 0x563289755a50 2024-03-16T11:51:36.600 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:36.595+0000 7fc897fff640 1 -- 172.21.15.29:0/2052669512 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7fc8ac026040 con 0x563289755a50 2024-03-16T11:51:36.600 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:36.595+0000 7fc8c1dc6740 10 client.15468 fetch_fsmap learned FSMap version 22 2024-03-16T11:51:36.600 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:36.595+0000 7fc8c1dc6740 1 -- 172.21.15.29:0/2052669512 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x5632897d4c90 con 0x563289755a50 2024-03-16T11:51:36.600 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:36.595+0000 7fc897fff640 1 -- 172.21.15.29:0/2052669512 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc8ac016070 con 0x563289755a50 2024-03-16T11:51:36.600 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:36.595+0000 7fc897fff640 1 -- 172.21.15.29:0/2052669512 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7fc8ac021050 con 0x563289755a50 2024-03-16T11:51:36.600 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:36.595+0000 7fc8c1dc6740 10 client.15468 fetch_fsmap finished waiting for FSMap version 22 2024-03-16T11:51:36.600 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:36.595+0000 7fc8c1dc6740 10 client.15468 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T11:51:36.600 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:36.595+0000 7fc8c1dc6740 20 client.15468 populate_metadata read hostname 'smithi029' 2024-03-16T11:51:36.600 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:36.595+0000 7fc8c1dc6740 1 --2- 172.21.15.29:0/2052669512 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x5632897d5350 0x5632897d7770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:51:36.600 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:36.595+0000 7fc8c1dc6740 4 client.15468 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-16T11:51:36.600 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:36.595+0000 7fc8c1dc6740 1 -- 172.21.15.29:0/2052669512 --> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5632896c12d0 con 0x5632897d5350 2024-03-16T11:51:36.600 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:36.595+0000 7fc8bce9b640 1 --2- 172.21.15.29:0/2052669512 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x5632897d5350 0x5632897d7770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:51:36.600 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:36.596+0000 7fc8bce9b640 1 --2- 172.21.15.29:0/2052669512 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x5632897d5350 0x5632897d7770 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=227ba7323e4d911f server_cookie=4cb0e3542f77078b in_seq=0 out_seq=0 2024-03-16T11:51:36.600 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:36.596+0000 7fc897fff640 10 client.15468 ms_handle_connect on v2:172.21.15.152:6838/1548731833 2024-03-16T11:51:36.602 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:36.602+0000 7fc897fff640 1 -- 172.21.15.29:0/2052669512 <== mds.0 v2:172.21.15.152:6838/1548731833 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5632896c12d0 con 0x5632897d5350 2024-03-16T11:51:36.602 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:36.602+0000 7fc897fff640 10 client.15468 handle_command_reply: tid=0 2024-03-16T11:51:36.602 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:36.602+0000 7fc8c1dc6740 1 client.15468 shutdown 2024-03-16T11:51:36.602 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:36.602+0000 7fc8c1dc6740 1 -- 172.21.15.29:0/2052669512 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563289755a50 msgr2=0x5632897625d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:51:36.602 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:36.602+0000 7fc8c1dc6740 1 --2- 172.21.15.29:0/2052669512 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563289755a50 0x5632897625d0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7fc8ac012ec0 tx=0x7fc8ac009d60 comp rx=0 tx=0).stop 2024-03-16T11:51:36.602 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:36.602+0000 7fc8c1dc6740 1 -- 172.21.15.29:0/2052669512 shutdown_connections 2024-03-16T11:51:36.602 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:36.602+0000 7fc8c1dc6740 1 --2- 172.21.15.29:0/2052669512 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56328975e820 0x563289762b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:51:36.602 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:36.602+0000 7fc8c1dc6740 1 --2- 172.21.15.29:0/2052669512 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563289755a50 0x5632897625d0 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:51:36.602 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:36.602+0000 7fc8c1dc6740 1 --2- 172.21.15.29:0/2052669512 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x5632897d5350 0x5632897d7770 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:51:36.602 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:36.602+0000 7fc8c1dc6740 1 --2- 172.21.15.29:0/2052669512 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563289755130 0x563289762090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:51:36.603 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:36.602+0000 7fc897fff640 0 client.15468 ms_handle_reset on v2:172.21.15.152:6838/1548731833 2024-03-16T11:51:36.603 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:36.602+0000 7fc8c1dc6740 1 -- 172.21.15.29:0/2052669512 >> 172.21.15.29:0/2052669512 conn(0x56328974fa50 msgr2=0x56328974ebb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:51:36.603 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:36.602+0000 7fc8c1dc6740 1 -- 172.21.15.29:0/2052669512 shutdown_connections 2024-03-16T11:51:36.603 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:36.602+0000 7fc8c1dc6740 1 -- 172.21.15.29:0/2052669512 wait complete. 2024-03-16T11:51:36.603 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:36.602+0000 7fc8c1dc6740 20 client.15468 trim_cache size 0 max 16384 2024-03-16T11:51:36.603 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:{ 2024-03-16T11:51:36.603 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "message": "", 2024-03-16T11:51:36.603 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "return_code": 0 2024-03-16T11:51:36.603 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:} 2024-03-16T11:51:36.604 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:36.603+0000 7fc8bf0ff640 1 -- 172.21.15.29:0/1379040248 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7fc8a00a2b30 msgr2=0x7fc8a00a4ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:51:36.604 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:36.603+0000 7fc8bf0ff640 1 --2- 172.21.15.29:0/1379040248 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7fc8a00a2b30 0x7fc8a00a4ff0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fc8b0005cc0 tx=0x7fc8b0005c50 comp rx=0 tx=0).stop 2024-03-16T11:51:36.604 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:36.603+0000 7fc8bf0ff640 1 -- 172.21.15.29:0/1379040248 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc8b8101790 msgr2=0x7fc8b8197fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:51:36.604 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:36.603+0000 7fc8bf0ff640 1 --2- 172.21.15.29:0/1379040248 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc8b8101790 0x7fc8b8197fb0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fc8a800e9f0 tx=0x7fc8a800edf0 comp rx=0 tx=0).stop 2024-03-16T11:51:36.604 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:36.603+0000 7fc8bf0ff640 1 -- 172.21.15.29:0/1379040248 shutdown_connections 2024-03-16T11:51:36.604 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:36.603+0000 7fc8bf0ff640 1 --2- 172.21.15.29:0/1379040248 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7fc8a00a2b30 0x7fc8a00a4ff0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:51:36.604 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:36.603+0000 7fc8bf0ff640 1 --2- 172.21.15.29:0/1379040248 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc8b81077a0 0x7fc8b819c820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:51:36.605 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:36.603+0000 7fc8bf0ff640 1 --2- 172.21.15.29:0/1379040248 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc8b8102150 0x7fc8b81984f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:51:36.605 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:36.603+0000 7fc8bf0ff640 1 --2- 172.21.15.29:0/1379040248 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc8b8101790 0x7fc8b8197fb0 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:51:36.605 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:36.603+0000 7fc8bf0ff640 1 -- 172.21.15.29:0/1379040248 >> 172.21.15.29:0/1379040248 conn(0x7fc8b80fd2a0 msgr2=0x7fc8b80fd7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:51:36.605 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:36.603+0000 7fc8bf0ff640 1 -- 172.21.15.29:0/1379040248 shutdown_connections 2024-03-16T11:51:36.605 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:36.603+0000 7fc8bf0ff640 1 -- 172.21.15.29:0/1379040248 wait complete. 2024-03-16T11:51:36.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:51:36 smithi154 ceph-mon[36320]: pgmap v23: 97 pgs: 97 active+clean; 1.4 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 1.9 KiB/s wr, 6 op/s 2024-03-16T11:51:36.837 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:51:36 smithi029 ceph-mon[31975]: pgmap v23: 97 pgs: 97 active+clean; 1.4 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 1.9 KiB/s wr, 6 op/s 2024-03-16T11:51:36.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:51:36 smithi152 ceph-mon[36291]: pgmap v23: 97 pgs: 97 active+clean; 1.4 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 1.9 KiB/s wr, 6 op/s 2024-03-16T11:51:38.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:51:38 smithi154 ceph-mon[36320]: pgmap v24: 97 pgs: 97 active+clean; 1.4 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 1.9 KiB/s wr, 8 op/s 2024-03-16T11:51:38.837 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:51:38 smithi029 ceph-mon[31975]: pgmap v24: 97 pgs: 97 active+clean; 1.4 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 1.9 KiB/s wr, 8 op/s 2024-03-16T11:51:38.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:51:38 smithi152 ceph-mon[36291]: pgmap v24: 97 pgs: 97 active+clean; 1.4 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 1.9 KiB/s wr, 8 op/s 2024-03-16T11:51:40.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:51:40 smithi154 ceph-mon[36320]: pgmap v25: 97 pgs: 97 active+clean; 1.4 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 1.3 KiB/s wr, 5 op/s 2024-03-16T11:51:40.837 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:51:40 smithi029 ceph-mon[31975]: pgmap v25: 97 pgs: 97 active+clean; 1.4 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 1.3 KiB/s wr, 5 op/s 2024-03-16T11:51:40.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:51:40 smithi152 ceph-mon[36291]: pgmap v25: 97 pgs: 97 active+clean; 1.4 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 1.3 KiB/s wr, 5 op/s 2024-03-16T11:51:42.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:51:42 smithi154 ceph-mon[36320]: pgmap v26: 97 pgs: 97 active+clean; 1.4 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 2.2 KiB/s wr, 7 op/s 2024-03-16T11:51:42.837 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:51:42 smithi029 ceph-mon[31975]: pgmap v26: 97 pgs: 97 active+clean; 1.4 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 2.2 KiB/s wr, 7 op/s 2024-03-16T11:51:42.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:51:42 smithi152 ceph-mon[36291]: pgmap v26: 97 pgs: 97 active+clean; 1.4 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 2.2 KiB/s wr, 7 op/s 2024-03-16T11:51:44.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:51:44 smithi154 ceph-mon[36320]: pgmap v27: 97 pgs: 97 active+clean; 1.4 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 2.4 KiB/s wr, 9 op/s 2024-03-16T11:51:44.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:51:44 smithi029 ceph-mon[31975]: pgmap v27: 97 pgs: 97 active+clean; 1.4 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 2.4 KiB/s wr, 9 op/s 2024-03-16T11:51:44.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:51:44 smithi152 ceph-mon[36291]: pgmap v27: 97 pgs: 97 active+clean; 1.4 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 2.4 KiB/s wr, 9 op/s 2024-03-16T11:51:46.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:51:46 smithi154 ceph-mon[36320]: pgmap v28: 97 pgs: 97 active+clean; 1.4 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 1.2 KiB/s wr, 6 op/s 2024-03-16T11:51:46.837 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:51:46 smithi029 ceph-mon[31975]: pgmap v28: 97 pgs: 97 active+clean; 1.4 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 1.2 KiB/s wr, 6 op/s 2024-03-16T11:51:46.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:51:46 smithi152 ceph-mon[36291]: pgmap v28: 97 pgs: 97 active+clean; 1.4 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 1.2 KiB/s wr, 6 op/s 2024-03-16T11:51:48.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:51:48 smithi154 ceph-mon[36320]: pgmap v29: 97 pgs: 97 active+clean; 1.4 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 1.2 KiB/s wr, 9 op/s 2024-03-16T11:51:48.837 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:51:48 smithi029 ceph-mon[31975]: pgmap v29: 97 pgs: 97 active+clean; 1.4 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 1.2 KiB/s wr, 9 op/s 2024-03-16T11:51:48.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:51:48 smithi152 ceph-mon[36291]: pgmap v29: 97 pgs: 97 active+clean; 1.4 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 1.2 KiB/s wr, 9 op/s 2024-03-16T11:51:49.692 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:49.692+0000 7f6c32953640 1 -- 172.21.15.29:0/2378031610 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6c2c069780 msgr2=0x7f6c2c10dfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:51:49.693 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:49.692+0000 7f6c32953640 1 --2- 172.21.15.29:0/2378031610 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6c2c069780 0x7f6c2c10dfa0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f6c20009900 tx=0x7f6c2002bcd0 comp rx=0 tx=0).stop 2024-03-16T11:51:49.693 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:49.692+0000 7f6c32953640 1 -- 172.21.15.29:0/2378031610 shutdown_connections 2024-03-16T11:51:49.693 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:49.692+0000 7f6c32953640 1 --2- 172.21.15.29:0/2378031610 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6c2c10e4e0 0x7f6c2c1108d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:51:49.693 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:49.692+0000 7f6c32953640 1 --2- 172.21.15.29:0/2378031610 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6c2c069780 0x7f6c2c10dfa0 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:51:49.693 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:49.692+0000 7f6c32953640 1 --2- 172.21.15.29:0/2378031610 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6c2c068e60 0x7f6c2c069240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:51:49.693 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:49.692+0000 7f6c32953640 1 -- 172.21.15.29:0/2378031610 >> 172.21.15.29:0/2378031610 conn(0x7f6c2c077530 msgr2=0x7f6c2c077940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:51:49.693 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:49.692+0000 7f6c32953640 1 -- 172.21.15.29:0/2378031610 shutdown_connections 2024-03-16T11:51:49.693 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:49.692+0000 7f6c32953640 1 -- 172.21.15.29:0/2378031610 wait complete. 2024-03-16T11:51:49.693 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:49.693+0000 7f6c32953640 1 Processor -- start 2024-03-16T11:51:49.693 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:49.693+0000 7f6c32953640 1 -- start start 2024-03-16T11:51:49.695 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:49.693+0000 7f6c32953640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6c2c068e60 0x7f6c2c19a220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:51:49.696 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:49.693+0000 7f6c32953640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6c2c069780 0x7f6c2c19a760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:51:49.696 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:49.693+0000 7f6c32953640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6c2c10e4e0 0x7f6c2c19ea90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:51:49.696 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:49.693+0000 7f6c32953640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f6c2c113e00 con 0x7f6c2c069780 2024-03-16T11:51:49.696 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:49.693+0000 7f6c32953640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f6c2c113c80 con 0x7f6c2c10e4e0 2024-03-16T11:51:49.696 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:49.693+0000 7f6c32953640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f6c2c113f80 con 0x7f6c2c068e60 2024-03-16T11:51:49.696 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:49.693+0000 7f6c2b7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6c2c069780 0x7f6c2c19a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:51:49.696 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:49.693+0000 7f6c2b7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6c2c069780 0x7f6c2c19a760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36516/0 (socket says 172.21.15.29:36516) 2024-03-16T11:51:49.696 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:49.693+0000 7f6c2b7fe640 1 -- 172.21.15.29:0/3737075302 learned_addr learned my addr 172.21.15.29:0/3737075302 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:51:49.696 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:49.693+0000 7f6c2bfff640 1 --2- 172.21.15.29:0/3737075302 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6c2c068e60 0x7f6c2c19a220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:51:49.696 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:49.693+0000 7f6c30ef0640 1 --2- 172.21.15.29:0/3737075302 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6c2c10e4e0 0x7f6c2c19ea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:51:49.696 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:49.694+0000 7f6c2b7fe640 1 -- 172.21.15.29:0/3737075302 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6c2c068e60 msgr2=0x7f6c2c19a220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:51:49.696 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:49.694+0000 7f6c2b7fe640 1 --2- 172.21.15.29:0/3737075302 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6c2c068e60 0x7f6c2c19a220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:51:49.696 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:49.694+0000 7f6c2b7fe640 1 -- 172.21.15.29:0/3737075302 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6c2c10e4e0 msgr2=0x7f6c2c19ea90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:51:49.696 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:49.694+0000 7f6c2b7fe640 1 --2- 172.21.15.29:0/3737075302 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6c2c10e4e0 0x7f6c2c19ea90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:51:49.697 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:49.694+0000 7f6c2b7fe640 1 -- 172.21.15.29:0/3737075302 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c2c19f1c0 con 0x7f6c2c069780 2024-03-16T11:51:49.697 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:49.694+0000 7f6c2b7fe640 1 --2- 172.21.15.29:0/3737075302 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6c2c069780 0x7f6c2c19a760 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f6c20000c00 tx=0x7f6c20004360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:51:49.697 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:49.694+0000 7f6c297fa640 1 -- 172.21.15.29:0/3737075302 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c20042020 con 0x7f6c2c069780 2024-03-16T11:51:49.697 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:49.694+0000 7f6c297fa640 1 -- 172.21.15.29:0/3737075302 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6c200054e0 con 0x7f6c2c069780 2024-03-16T11:51:49.697 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:49.694+0000 7f6c297fa640 1 -- 172.21.15.29:0/3737075302 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c20003b20 con 0x7f6c2c069780 2024-03-16T11:51:49.697 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:49.694+0000 7f6c32953640 1 -- 172.21.15.29:0/3737075302 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6c2c19f450 con 0x7f6c2c069780 2024-03-16T11:51:49.697 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:49.694+0000 7f6c32953640 1 -- 172.21.15.29:0/3737075302 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6c2c1a6d40 con 0x7f6c2c069780 2024-03-16T11:51:49.699 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:49.697+0000 7f6c35618740 1 Processor -- start 2024-03-16T11:51:49.699 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:49.697+0000 7f6c35618740 1 -- start start 2024-03-16T11:51:49.699 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:49.697+0000 7f6c35618740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564ea36bc130 0x564ea36bc510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:51:49.699 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:49.697+0000 7f6c35618740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564ea36bca50 0x564ea36c52e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:51:49.699 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:49.697+0000 7f6c35618740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564ea36c5820 0x564ea36c7c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:51:49.699 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:49.697+0000 7f6c35618740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x564ea3620a40 con 0x564ea36bc130 2024-03-16T11:51:49.699 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:49.697+0000 7f6c35618740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x564ea35c9920 con 0x564ea36bca50 2024-03-16T11:51:49.699 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:49.697+0000 7f6c35618740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x564ea361cae0 con 0x564ea36c5820 2024-03-16T11:51:49.699 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:49.697+0000 7f6c297fa640 1 -- 172.21.15.29:0/3737075302 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 24) ==== 137523+0+0 (secure 0 0 0) 0x7f6c20045020 con 0x7f6c2c069780 2024-03-16T11:51:49.700 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:49.698+0000 7f6c30ef0640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564ea36bc130 0x564ea36bc510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:51:49.700 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:49.698+0000 7f6c2bfff640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564ea36bca50 0x564ea36c52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:51:49.700 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:49.698+0000 7f6c30ef0640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564ea36bc130 0x564ea36bc510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36528/0 (socket says 172.21.15.29:36528) 2024-03-16T11:51:49.700 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:49.698+0000 7f6c2bfff640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564ea36bca50 0x564ea36c52e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.29:38396/0 (socket says 172.21.15.29:38396) 2024-03-16T11:51:49.700 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:49.698+0000 7f6c30ef0640 1 -- 172.21.15.29:0/1216045785 learned_addr learned my addr 172.21.15.29:0/1216045785 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:51:49.700 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:49.698+0000 7f6c2b7fe640 1 --2- 172.21.15.29:0/1216045785 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564ea36c5820 0x564ea36c7c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:51:49.700 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:49.698+0000 7f6c30ef0640 1 -- 172.21.15.29:0/1216045785 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564ea36c5820 msgr2=0x564ea36c7c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:51:49.700 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:49.698+0000 7f6c30ef0640 1 --2- 172.21.15.29:0/1216045785 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564ea36c5820 0x564ea36c7c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:51:49.700 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:49.698+0000 7f6c30ef0640 1 -- 172.21.15.29:0/1216045785 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564ea36bca50 msgr2=0x564ea36c52e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:51:49.700 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:49.698+0000 7f6c30ef0640 1 --2- 172.21.15.29:0/1216045785 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564ea36bca50 0x564ea36c52e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:51:49.700 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:49.698+0000 7f6c30ef0640 1 -- 172.21.15.29:0/1216045785 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564ea35e1460 con 0x564ea36bc130 2024-03-16T11:51:49.700 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:49.698+0000 7f6c297fa640 1 --2- 172.21.15.29:0/3737075302 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f6c140a2bb0 0x7f6c140a5070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:51:49.700 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:49.698+0000 7f6c30ef0640 1 --2- 172.21.15.29:0/1216045785 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564ea36bc130 0x564ea36bc510 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f6c24009f80 tx=0x7f6c240118d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8172da6230f596a7 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:51:49.700 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:49.698+0000 7f6c297fa640 1 -- 172.21.15.29:0/3737075302 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8469+0+0 (secure 0 0 0) 0x7f6c200f0b90 con 0x7f6c2c069780 2024-03-16T11:51:49.701 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:49.698+0000 7f6c2bfff640 1 --2- 172.21.15.29:0/3737075302 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f6c140a2bb0 0x7f6c140a5070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:51:49.701 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:49.699+0000 7f6c127fc640 1 -- 172.21.15.29:0/1216045785 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c2401e070 con 0x564ea36bc130 2024-03-16T11:51:49.701 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:49.699+0000 7f6c127fc640 1 -- 172.21.15.29:0/1216045785 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6c24003c30 con 0x564ea36bc130 2024-03-16T11:51:49.701 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:49.699+0000 7f6c127fc640 1 -- 172.21.15.29:0/1216045785 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c2400fad0 con 0x564ea36bc130 2024-03-16T11:51:49.705 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:49.699+0000 7f6c35618740 1 -- 172.21.15.29:0/1216045785 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564ea36bc130 msgr2=0x564ea36bc510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:51:49.705 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:49.699+0000 7f6c2bfff640 1 --2- 172.21.15.29:0/3737075302 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f6c140a2bb0 0x7f6c140a5070 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x564ea36bbbe0 tx=0x7f6c1c005f90 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:51:49.705 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:49.699+0000 7f6c35618740 1 --2- 172.21.15.29:0/1216045785 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564ea36bc130 0x564ea36bc510 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f6c24009f80 tx=0x7f6c240118d0 comp rx=0 tx=0).stop 2024-03-16T11:51:49.705 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:49.700+0000 7f6c35618740 1 -- 172.21.15.29:0/1216045785 shutdown_connections 2024-03-16T11:51:49.705 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:49.700+0000 7f6c35618740 1 --2- 172.21.15.29:0/1216045785 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564ea36c5820 0x564ea36c7c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:51:49.705 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:49.700+0000 7f6c35618740 1 --2- 172.21.15.29:0/1216045785 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564ea36bca50 0x564ea36c52e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:51:49.705 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:49.700+0000 7f6c35618740 1 --2- 172.21.15.29:0/1216045785 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564ea36bc130 0x564ea36bc510 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:51:49.705 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:49.700+0000 7f6c35618740 1 -- 172.21.15.29:0/1216045785 >> 172.21.15.29:0/1216045785 conn(0x564ea36b6a50 msgr2=0x564ea36b6e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:51:49.706 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:49.700+0000 7f6c35618740 1 -- 172.21.15.29:0/1216045785 shutdown_connections 2024-03-16T11:51:49.706 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:49.700+0000 7f6c35618740 1 -- 172.21.15.29:0/1216045785 wait complete. 2024-03-16T11:51:49.706 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:49.700+0000 7f6c35618740 1 -- start start 2024-03-16T11:51:49.706 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:49.700+0000 7f6c35618740 1 Processor -- start 2024-03-16T11:51:49.706 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:49.700+0000 7f6c35618740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564ea36bc130 0x564ea36c28c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:51:49.706 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:49.701+0000 7f6c35618740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564ea36bca50 0x564ea36c2e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:51:49.706 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:49.701+0000 7f6c35618740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564ea36c5820 0x564ea36c3340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:51:49.706 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:49.701+0000 7f6c35618740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x564ea36c8450 con 0x564ea36bca50 2024-03-16T11:51:49.706 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:49.701+0000 7f6c35618740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x564ea35dcd70 con 0x564ea36c5820 2024-03-16T11:51:49.706 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:49.701+0000 7f6c35618740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x564ea36c85d0 con 0x564ea36bc130 2024-03-16T11:51:49.706 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:49.701+0000 7f6c2b7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564ea36bca50 0x564ea36c2e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:51:49.706 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:49.701+0000 7f6c2b7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564ea36bca50 0x564ea36c2e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36534/0 (socket says 172.21.15.29:36534) 2024-03-16T11:51:49.706 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:49.701+0000 7f6c2b7fe640 1 -- 172.21.15.29:0/1337418204 learned_addr learned my addr 172.21.15.29:0/1337418204 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:51:49.707 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:49.701+0000 7f6c35618740 10 client.0 _ll_register_callbacks cb 0x564ea3334a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T11:51:49.707 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:49.701+0000 7f6c2bfff640 1 --2- 172.21.15.29:0/1337418204 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564ea36bc130 0x564ea36c28c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:51:49.707 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:49.701+0000 7f6c30ef0640 1 --2- 172.21.15.29:0/1337418204 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564ea36c5820 0x564ea36c3340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:51:49.707 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:49.701+0000 7f6c2b7fe640 1 -- 172.21.15.29:0/1337418204 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564ea36bc130 msgr2=0x564ea36c28c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:51:49.707 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:49.701+0000 7f6c2b7fe640 1 --2- 172.21.15.29:0/1337418204 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564ea36bc130 0x564ea36c28c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:51:49.707 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:49.701+0000 7f6c2b7fe640 1 -- 172.21.15.29:0/1337418204 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564ea36c5820 msgr2=0x564ea36c3340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:51:49.707 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:49.701+0000 7f6c2b7fe640 1 --2- 172.21.15.29:0/1337418204 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564ea36c5820 0x564ea36c3340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:51:49.707 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:49.701+0000 7f6c2b7fe640 1 -- 172.21.15.29:0/1337418204 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564ea3335210 con 0x564ea36bca50 2024-03-16T11:51:49.707 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:49.701+0000 7f6c2b7fe640 1 --2- 172.21.15.29:0/1337418204 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564ea36bca50 0x564ea36c2e00 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f6c20006010 tx=0x7f6c20004610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7805e329b51cb1a server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:51:49.707 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:49.701+0000 7f6c0b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-16T11:51:49.707 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:49.701+0000 7f6c0b7fe640 1 -- 172.21.15.29:0/1337418204 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c20005690 con 0x564ea36bca50 2024-03-16T11:51:49.707 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:49.701+0000 7f6c35618740 1 -- 172.21.15.29:0/1337418204 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6c24003c30 con 0x564ea36bca50 2024-03-16T11:51:49.707 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:49.701+0000 7f6c0b7fe640 1 -- 172.21.15.29:0/1337418204 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8469+0+0 (secure 0 0 0) 0x7f6c20003cc0 con 0x564ea36bca50 2024-03-16T11:51:49.707 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:49.702+0000 7f6c0b7fe640 1 -- 172.21.15.29:0/1337418204 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f6c200bd780 con 0x564ea36bca50 2024-03-16T11:51:49.708 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:49.702+0000 7f6c35618740 10 client.15480 fetch_fsmap learned FSMap version 22 2024-03-16T11:51:49.708 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:49.702+0000 7f6c35618740 1 -- 172.21.15.29:0/1337418204 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x564ea3401620 con 0x564ea36bca50 2024-03-16T11:51:49.708 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:49.702+0000 7f6c0b7fe640 1 -- 172.21.15.29:0/1337418204 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6c24003c30 con 0x564ea36bca50 2024-03-16T11:51:49.708 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:49.702+0000 7f6c0b7fe640 1 -- 172.21.15.29:0/1337418204 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7f6c2002cc50 con 0x564ea36bca50 2024-03-16T11:51:49.708 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:49.702+0000 7f6c35618740 10 client.15480 fetch_fsmap finished waiting for FSMap version 22 2024-03-16T11:51:49.708 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:49.702+0000 7f6c35618740 10 client.15480 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T11:51:49.708 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:49.702+0000 7f6c35618740 20 client.15480 populate_metadata read hostname 'smithi029' 2024-03-16T11:51:49.708 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:49.702+0000 7f6c35618740 1 --2- 172.21.15.29:0/1337418204 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x564ea36d5930 0x564ea36d7d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:51:49.708 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:49.702+0000 7f6c35618740 4 client.15480 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T11:51:49.708 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:49.702+0000 7f6c35618740 1 -- 172.21.15.29:0/1337418204 --> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564ea362df80 con 0x564ea36d5930 2024-03-16T11:51:49.708 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:49.702+0000 7f6c2bfff640 1 --2- 172.21.15.29:0/1337418204 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x564ea36d5930 0x564ea36d7d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:51:49.708 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:49.703+0000 7f6c0b7fe640 10 client.15480 ms_handle_connect on v2:172.21.15.152:6838/1548731833 2024-03-16T11:51:49.708 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:49.703+0000 7f6c2bfff640 1 --2- 172.21.15.29:0/1337418204 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x564ea36d5930 0x564ea36d7d20 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=22907521e9b2a6ef server_cookie=4ba323b3d86ec237 in_seq=0 out_seq=0 2024-03-16T11:51:49.708 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:49.703+0000 7f6c0b7fe640 1 -- 172.21.15.29:0/1337418204 <== mds.0 v2:172.21.15.152:6838/1548731833 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x564ea362df80 con 0x564ea36d5930 2024-03-16T11:51:49.709 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:49.704+0000 7f6c0b7fe640 10 client.15480 handle_command_reply: tid=0 2024-03-16T11:51:49.709 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:49.704+0000 7f6c35618740 1 client.15480 shutdown 2024-03-16T11:51:49.709 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:49.704+0000 7f6c35618740 1 -- 172.21.15.29:0/1337418204 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564ea36bca50 msgr2=0x564ea36c2e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:51:49.709 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:49.704+0000 7f6c35618740 1 --2- 172.21.15.29:0/1337418204 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564ea36bca50 0x564ea36c2e00 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f6c20006010 tx=0x7f6c20004610 comp rx=0 tx=0).stop 2024-03-16T11:51:49.709 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:49.704+0000 7f6c35618740 1 -- 172.21.15.29:0/1337418204 shutdown_connections 2024-03-16T11:51:49.709 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:49.704+0000 7f6c35618740 1 --2- 172.21.15.29:0/1337418204 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x564ea36d5930 0x564ea36d7d20 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:51:49.709 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:49.704+0000 7f6c0b7fe640 0 client.15480 ms_handle_reset on v2:172.21.15.152:6838/1548731833 2024-03-16T11:51:49.709 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:49.704+0000 7f6c35618740 1 --2- 172.21.15.29:0/1337418204 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564ea36c5820 0x564ea36c3340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:51:49.709 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:49.704+0000 7f6c35618740 1 --2- 172.21.15.29:0/1337418204 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564ea36bca50 0x564ea36c2e00 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:51:49.709 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:49.704+0000 7f6c35618740 1 --2- 172.21.15.29:0/1337418204 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564ea36bc130 0x564ea36c28c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:51:49.709 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:49.704+0000 7f6c35618740 1 -- 172.21.15.29:0/1337418204 >> 172.21.15.29:0/1337418204 conn(0x564ea36b6a50 msgr2=0x564ea36b87a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:51:49.709 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:49.704+0000 7f6c35618740 1 -- 172.21.15.29:0/1337418204 shutdown_connections 2024-03-16T11:51:49.709 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:49.704+0000 7f6c35618740 1 -- 172.21.15.29:0/1337418204 wait complete. 2024-03-16T11:51:49.709 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:49.704+0000 7f6c35618740 20 client.15480 trim_cache size 0 max 16384 2024-03-16T11:51:49.720 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:49.719+0000 7f6c35618740 1 Processor -- start 2024-03-16T11:51:49.720 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:49.719+0000 7f6c35618740 1 -- start start 2024-03-16T11:51:49.720 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:49.719+0000 7f6c35618740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564ea36bc130 0x564ea36c4ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:51:49.721 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:49.719+0000 7f6c35618740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564ea36bca50 0x564ea36d11f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:51:49.721 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:49.719+0000 7f6c35618740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564ea36c5820 0x564ea36dd630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:51:49.721 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:49.719+0000 7f6c35618740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f6c2c1b1e00 con 0x564ea36bc130 2024-03-16T11:51:49.721 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:49.719+0000 7f6c2bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564ea36bc130 0x564ea36c4ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:51:49.721 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:49.719+0000 7f6c2bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564ea36bc130 0x564ea36c4ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36548/0 (socket says 172.21.15.29:36548) 2024-03-16T11:51:49.721 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:49.719+0000 7f6c2bfff640 1 -- 172.21.15.29:0/3074578178 learned_addr learned my addr 172.21.15.29:0/3074578178 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:51:49.721 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:49.719+0000 7f6c2b7fe640 1 --2- 172.21.15.29:0/3074578178 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564ea36bca50 0x564ea36d11f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:51:49.721 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:49.719+0000 7f6c30ef0640 1 --2- 172.21.15.29:0/3074578178 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564ea36c5820 0x564ea36dd630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:51:49.721 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:49.719+0000 7f6c35618740 1 -- 172.21.15.29:0/3074578178 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f6c2c1af6d0 con 0x564ea36c5820 2024-03-16T11:51:49.721 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:49.719+0000 7f6c35618740 1 -- 172.21.15.29:0/3074578178 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f6c2c1ad520 con 0x564ea36bca50 2024-03-16T11:51:49.721 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:49.720+0000 7f6c2bfff640 1 -- 172.21.15.29:0/3074578178 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564ea36bca50 msgr2=0x564ea36d11f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:51:49.721 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:49.720+0000 7f6c2bfff640 1 --2- 172.21.15.29:0/3074578178 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564ea36bca50 0x564ea36d11f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:51:49.721 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:49.720+0000 7f6c2bfff640 1 -- 172.21.15.29:0/3074578178 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564ea36c5820 msgr2=0x564ea36dd630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:51:49.721 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:49.720+0000 7f6c2bfff640 1 --2- 172.21.15.29:0/3074578178 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564ea36c5820 0x564ea36dd630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:51:49.722 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:49.720+0000 7f6c2bfff640 1 -- 172.21.15.29:0/3074578178 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564ea36c0e50 con 0x564ea36bc130 2024-03-16T11:51:49.722 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:49.720+0000 7f6c2b7fe640 1 --2- 172.21.15.29:0/3074578178 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564ea36bca50 0x564ea36d11f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:51:49.722 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:49.720+0000 7f6c2bfff640 1 --2- 172.21.15.29:0/3074578178 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564ea36bc130 0x564ea36c4ff0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f6c1c012cb0 tx=0x7f6c1c01b000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95c5d68e07fd1438 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:51:49.722 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:49.720+0000 7f6c11ffb640 1 -- 172.21.15.29:0/3074578178 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c1c02bb90 con 0x564ea36bc130 2024-03-16T11:51:49.722 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:49.720+0000 7f6c11ffb640 1 -- 172.21.15.29:0/3074578178 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6c1c02bd30 con 0x564ea36bc130 2024-03-16T11:51:49.722 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:49.720+0000 7f6c11ffb640 1 -- 172.21.15.29:0/3074578178 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c1c01fbc0 con 0x564ea36bc130 2024-03-16T11:51:49.725 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:49.721+0000 7f6c35618740 1 -- 172.21.15.29:0/3074578178 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564ea36bc130 msgr2=0x564ea36c4ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:51:49.725 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:49.721+0000 7f6c35618740 1 --2- 172.21.15.29:0/3074578178 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564ea36bc130 0x564ea36c4ff0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f6c1c012cb0 tx=0x7f6c1c01b000 comp rx=0 tx=0).stop 2024-03-16T11:51:49.725 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:49.721+0000 7f6c35618740 1 -- 172.21.15.29:0/3074578178 shutdown_connections 2024-03-16T11:51:49.725 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:49.721+0000 7f6c35618740 1 --2- 172.21.15.29:0/3074578178 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564ea36c5820 0x564ea36dd630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:51:49.726 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:49.721+0000 7f6c35618740 1 --2- 172.21.15.29:0/3074578178 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564ea36bca50 0x564ea36d11f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:51:49.726 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:49.721+0000 7f6c35618740 1 --2- 172.21.15.29:0/3074578178 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564ea36bc130 0x564ea36c4ff0 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:51:49.726 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:49.721+0000 7f6c35618740 1 -- 172.21.15.29:0/3074578178 >> 172.21.15.29:0/3074578178 conn(0x564ea36b6a50 msgr2=0x564ea36b5320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:51:49.726 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:49.721+0000 7f6c35618740 1 -- 172.21.15.29:0/3074578178 shutdown_connections 2024-03-16T11:51:49.726 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:49.721+0000 7f6c35618740 1 -- 172.21.15.29:0/3074578178 wait complete. 2024-03-16T11:51:49.726 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:49.721+0000 7f6c35618740 1 -- start start 2024-03-16T11:51:49.726 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:49.722+0000 7f6c35618740 1 Processor -- start 2024-03-16T11:51:49.726 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:49.722+0000 7f6c35618740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564ea36bca50 0x564ea36cd820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:51:49.726 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:49.722+0000 7f6c35618740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564ea36c5820 0x564ea36ca7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:51:49.726 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:49.722+0000 7f6c35618740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564ea36d5930 0x564ea36cad30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:51:49.726 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:49.722+0000 7f6c35618740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f6c2c1adb90 con 0x564ea36c5820 2024-03-16T11:51:49.726 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:49.722+0000 7f6c35618740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f6c2c1af000 con 0x564ea36d5930 2024-03-16T11:51:49.726 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:49.722+0000 7f6c35618740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f6c2c1ae260 con 0x564ea36bca50 2024-03-16T11:51:49.726 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:49.722+0000 7f6c35618740 10 client.0 _ll_register_callbacks cb 0x7f6c20009560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T11:51:49.727 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:49.722+0000 7f6c2b7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564ea36c5820 0x564ea36ca7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:51:49.727 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:49.722+0000 7f6c2bfff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564ea36bca50 0x564ea36cd820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:51:49.727 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:49.722+0000 7f6c2b7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564ea36c5820 0x564ea36ca7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36558/0 (socket says 172.21.15.29:36558) 2024-03-16T11:51:49.727 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:49.722+0000 7f6c2b7fe640 1 -- 172.21.15.29:0/2770892958 learned_addr learned my addr 172.21.15.29:0/2770892958 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:51:49.727 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:49.722+0000 7f6c30ef0640 1 --2- 172.21.15.29:0/2770892958 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564ea36d5930 0x564ea36cad30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:51:49.727 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:49.722+0000 7f6c2b7fe640 1 -- 172.21.15.29:0/2770892958 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564ea36bca50 msgr2=0x564ea36cd820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:51:49.727 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:49.722+0000 7f6c2b7fe640 1 --2- 172.21.15.29:0/2770892958 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564ea36bca50 0x564ea36cd820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:51:49.727 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:49.722+0000 7f6c2b7fe640 1 -- 172.21.15.29:0/2770892958 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564ea36d5930 msgr2=0x564ea36cad30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:51:49.727 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:49.722+0000 7f6c2b7fe640 1 --2- 172.21.15.29:0/2770892958 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564ea36d5930 0x564ea36cad30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:51:49.727 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:49.722+0000 7f6c2b7fe640 1 -- 172.21.15.29:0/2770892958 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564ea36cb270 con 0x564ea36c5820 2024-03-16T11:51:49.727 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:49.722+0000 7f6c30ef0640 1 --2- 172.21.15.29:0/2770892958 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564ea36d5930 0x564ea36cad30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:51:49.727 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:49.723+0000 7f6c2b7fe640 1 --2- 172.21.15.29:0/2770892958 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564ea36c5820 0x564ea36ca7f0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f6c200bd010 tx=0x7f6c200c5550 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3940225a5e59b53 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:51:49.727 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:49.723+0000 7f6c0b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-16T11:51:49.727 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:49.723+0000 7f6c0b7fe640 1 -- 172.21.15.29:0/2770892958 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c200dbdd0 con 0x564ea36c5820 2024-03-16T11:51:49.728 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:49.723+0000 7f6c0b7fe640 1 -- 172.21.15.29:0/2770892958 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8469+0+0 (secure 0 0 0) 0x7f6c200bc4d0 con 0x564ea36c5820 2024-03-16T11:51:49.728 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:49.723+0000 7f6c35618740 1 -- 172.21.15.29:0/2770892958 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6c1c02bd30 con 0x564ea36c5820 2024-03-16T11:51:49.728 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:49.723+0000 7f6c0b7fe640 1 -- 172.21.15.29:0/2770892958 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f6c200ee180 con 0x564ea36c5820 2024-03-16T11:51:49.728 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:49.723+0000 7f6c35618740 10 client.15486 fetch_fsmap learned FSMap version 22 2024-03-16T11:51:49.728 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:49.723+0000 7f6c35618740 1 -- 172.21.15.29:0/2770892958 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x564ea373c980 con 0x564ea36c5820 2024-03-16T11:51:49.728 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:49.723+0000 7f6c0b7fe640 1 -- 172.21.15.29:0/2770892958 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6c1c02bd30 con 0x564ea36c5820 2024-03-16T11:51:49.728 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:49.723+0000 7f6c0b7fe640 1 -- 172.21.15.29:0/2770892958 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7f6c200bbf40 con 0x564ea36c5820 2024-03-16T11:51:49.728 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:49.723+0000 7f6c35618740 10 client.15486 fetch_fsmap finished waiting for FSMap version 22 2024-03-16T11:51:49.728 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:49.723+0000 7f6c35618740 10 client.15486 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T11:51:49.728 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:49.723+0000 7f6c35618740 20 client.15486 populate_metadata read hostname 'smithi029' 2024-03-16T11:51:49.728 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:49.723+0000 7f6c35618740 1 --2- 172.21.15.29:0/2770892958 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x564ea373cfc0 0x564ea373f3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:51:49.728 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:49.724+0000 7f6c35618740 4 client.15486 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-16T11:51:49.728 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:49.724+0000 7f6c35618740 1 -- 172.21.15.29:0/2770892958 --> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] -- command(tid 0: {"prefix": "flush journal"}) -- 0x564ea36282d0 con 0x564ea373cfc0 2024-03-16T11:51:49.728 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:49.724+0000 7f6c2bfff640 1 --2- 172.21.15.29:0/2770892958 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x564ea373cfc0 0x564ea373f3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:51:49.729 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:49.724+0000 7f6c0b7fe640 10 client.15486 ms_handle_connect on v2:172.21.15.152:6838/1548731833 2024-03-16T11:51:49.729 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:49.724+0000 7f6c2bfff640 1 --2- 172.21.15.29:0/2770892958 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x564ea373cfc0 0x564ea373f3e0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d7bd4e0c3bab9e0 server_cookie=f9887167f37cd228 in_seq=0 out_seq=0 2024-03-16T11:51:49.730 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:49.727+0000 7f6c0b7fe640 1 -- 172.21.15.29:0/2770892958 <== mds.0 v2:172.21.15.152:6838/1548731833 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x564ea36282d0 con 0x564ea373cfc0 2024-03-16T11:51:49.730 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:49.727+0000 7f6c0b7fe640 10 client.15486 handle_command_reply: tid=0 2024-03-16T11:51:49.730 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:49.727+0000 7f6c35618740 1 client.15486 shutdown 2024-03-16T11:51:49.730 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:49.728+0000 7f6c35618740 1 -- 172.21.15.29:0/2770892958 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564ea36c5820 msgr2=0x564ea36ca7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:51:49.730 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:49.728+0000 7f6c35618740 1 --2- 172.21.15.29:0/2770892958 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564ea36c5820 0x564ea36ca7f0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f6c200bd010 tx=0x7f6c200c5550 comp rx=0 tx=0).stop 2024-03-16T11:51:49.730 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:49.728+0000 7f6c35618740 1 -- 172.21.15.29:0/2770892958 shutdown_connections 2024-03-16T11:51:49.730 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:49.728+0000 7f6c35618740 1 --2- 172.21.15.29:0/2770892958 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x564ea373cfc0 0x564ea373f3e0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:51:49.730 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:49.728+0000 7f6c0b7fe640 0 client.15486 ms_handle_reset on v2:172.21.15.152:6838/1548731833 2024-03-16T11:51:49.730 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:49.728+0000 7f6c35618740 1 --2- 172.21.15.29:0/2770892958 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564ea36d5930 0x564ea36cad30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:51:49.730 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:49.728+0000 7f6c35618740 1 --2- 172.21.15.29:0/2770892958 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564ea36c5820 0x564ea36ca7f0 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:51:49.730 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:49.728+0000 7f6c35618740 1 --2- 172.21.15.29:0/2770892958 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564ea36bca50 0x564ea36cd820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:51:49.731 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:49.728+0000 7f6c35618740 1 -- 172.21.15.29:0/2770892958 >> 172.21.15.29:0/2770892958 conn(0x564ea36b6a50 msgr2=0x564ea36c3110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:51:49.731 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:49.728+0000 7f6c35618740 1 -- 172.21.15.29:0/2770892958 shutdown_connections 2024-03-16T11:51:49.731 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:49.728+0000 7f6c35618740 1 -- 172.21.15.29:0/2770892958 wait complete. 2024-03-16T11:51:49.731 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:49.728+0000 7f6c35618740 20 client.15486 trim_cache size 0 max 16384 2024-03-16T11:51:49.731 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:{ 2024-03-16T11:51:49.731 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "message": "", 2024-03-16T11:51:49.731 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "return_code": 0 2024-03-16T11:51:49.731 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:} 2024-03-16T11:51:49.734 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:49.728+0000 7f6c32953640 1 -- 172.21.15.29:0/3737075302 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f6c140a2bb0 msgr2=0x7f6c140a5070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:51:49.734 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:49.728+0000 7f6c32953640 1 --2- 172.21.15.29:0/3737075302 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f6c140a2bb0 0x7f6c140a5070 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x564ea36bbbe0 tx=0x7f6c1c005f90 comp rx=0 tx=0).stop 2024-03-16T11:51:49.734 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:49.728+0000 7f6c32953640 1 -- 172.21.15.29:0/3737075302 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6c2c069780 msgr2=0x7f6c2c19a760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:51:49.734 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:49.728+0000 7f6c32953640 1 --2- 172.21.15.29:0/3737075302 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6c2c069780 0x7f6c2c19a760 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f6c20000c00 tx=0x7f6c20004360 comp rx=0 tx=0).stop 2024-03-16T11:51:49.734 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:49.729+0000 7f6c32953640 1 -- 172.21.15.29:0/3737075302 shutdown_connections 2024-03-16T11:51:49.734 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:49.729+0000 7f6c32953640 1 --2- 172.21.15.29:0/3737075302 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f6c140a2bb0 0x7f6c140a5070 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:51:49.734 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:49.729+0000 7f6c32953640 1 --2- 172.21.15.29:0/3737075302 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6c2c10e4e0 0x7f6c2c19ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:51:49.734 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:49.729+0000 7f6c32953640 1 --2- 172.21.15.29:0/3737075302 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6c2c069780 0x7f6c2c19a760 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:51:49.734 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:49.729+0000 7f6c32953640 1 --2- 172.21.15.29:0/3737075302 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6c2c068e60 0x7f6c2c19a220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:51:49.735 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:49.729+0000 7f6c32953640 1 -- 172.21.15.29:0/3737075302 >> 172.21.15.29:0/3737075302 conn(0x7f6c2c077530 msgr2=0x7f6c2c10f890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:51:49.735 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:49.729+0000 7f6c32953640 1 -- 172.21.15.29:0/3737075302 shutdown_connections 2024-03-16T11:51:49.735 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:51:49.729+0000 7f6c32953640 1 -- 172.21.15.29:0/3737075302 wait complete. 2024-03-16T11:51:50.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:51:50 smithi154 ceph-mon[36320]: pgmap v30: 97 pgs: 97 active+clean; 1.4 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 1.2 KiB/s wr, 6 op/s 2024-03-16T11:51:50.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:51:50 smithi029 ceph-mon[31975]: pgmap v30: 97 pgs: 97 active+clean; 1.4 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 1.2 KiB/s wr, 6 op/s 2024-03-16T11:51:50.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:51:50 smithi152 ceph-mon[36291]: pgmap v30: 97 pgs: 97 active+clean; 1.4 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 1.2 KiB/s wr, 6 op/s 2024-03-16T11:51:52.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:51:52 smithi154 ceph-mon[36320]: pgmap v31: 97 pgs: 97 active+clean; 1.4 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 1.4 KiB/s wr, 9 op/s 2024-03-16T11:51:52.837 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:51:52 smithi029 ceph-mon[31975]: pgmap v31: 97 pgs: 97 active+clean; 1.4 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 1.4 KiB/s wr, 9 op/s 2024-03-16T11:51:52.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:51:52 smithi152 ceph-mon[36291]: pgmap v31: 97 pgs: 97 active+clean; 1.4 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 1.4 KiB/s wr, 9 op/s 2024-03-16T11:51:54.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:51:54 smithi154 ceph-mon[36320]: pgmap v32: 97 pgs: 97 active+clean; 1.4 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 597 B/s wr, 6 op/s 2024-03-16T11:51:54.837 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:51:54 smithi029 ceph-mon[31975]: pgmap v32: 97 pgs: 97 active+clean; 1.4 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 597 B/s wr, 6 op/s 2024-03-16T11:51:54.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:51:54 smithi152 ceph-mon[36291]: pgmap v32: 97 pgs: 97 active+clean; 1.4 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 597 B/s wr, 6 op/s 2024-03-16T11:51:56.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:51:56 smithi154 ceph-mon[36320]: pgmap v33: 97 pgs: 97 active+clean; 1.4 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 255 B/s wr, 5 op/s 2024-03-16T11:51:56.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:51:56 smithi029 ceph-mon[31975]: pgmap v33: 97 pgs: 97 active+clean; 1.4 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 255 B/s wr, 5 op/s 2024-03-16T11:51:56.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:51:56 smithi152 ceph-mon[36291]: pgmap v33: 97 pgs: 97 active+clean; 1.4 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 255 B/s wr, 5 op/s 2024-03-16T11:51:58.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:51:58 smithi154 ceph-mon[36320]: pgmap v34: 97 pgs: 97 active+clean; 1.4 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 255 B/s wr, 7 op/s 2024-03-16T11:51:58.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:51:58 smithi029 ceph-mon[31975]: pgmap v34: 97 pgs: 97 active+clean; 1.4 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 255 B/s wr, 7 op/s 2024-03-16T11:51:58.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:51:58 smithi152 ceph-mon[36291]: pgmap v34: 97 pgs: 97 active+clean; 1.4 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 255 B/s wr, 7 op/s 2024-03-16T11:52:00.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:52:00 smithi154 ceph-mon[36320]: pgmap v35: 97 pgs: 97 active+clean; 1.4 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-03-16T11:52:00.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:52:00 smithi154 ceph-mon[36320]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T11:52:00.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:52:00 smithi154 ceph-mon[36320]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' 2024-03-16T11:52:00.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:52:00 smithi154 ceph-mon[36320]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' 2024-03-16T11:52:00.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:52:00 smithi154 ceph-mon[36320]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' 2024-03-16T11:52:00.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:52:00 smithi154 ceph-mon[36320]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' 2024-03-16T11:52:00.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:52:00 smithi029 ceph-mon[31975]: pgmap v35: 97 pgs: 97 active+clean; 1.4 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-03-16T11:52:00.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:52:00 smithi029 ceph-mon[31975]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T11:52:00.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:52:00 smithi029 ceph-mon[31975]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' 2024-03-16T11:52:00.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:52:00 smithi029 ceph-mon[31975]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' 2024-03-16T11:52:00.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:52:00 smithi029 ceph-mon[31975]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' 2024-03-16T11:52:00.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:52:00 smithi029 ceph-mon[31975]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' 2024-03-16T11:52:00.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:52:00 smithi152 ceph-mon[36291]: pgmap v35: 97 pgs: 97 active+clean; 1.4 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-03-16T11:52:00.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:52:00 smithi152 ceph-mon[36291]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T11:52:00.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:52:00 smithi152 ceph-mon[36291]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' 2024-03-16T11:52:00.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:52:00 smithi152 ceph-mon[36291]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' 2024-03-16T11:52:00.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:52:00 smithi152 ceph-mon[36291]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' 2024-03-16T11:52:00.878 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:52:00 smithi152 ceph-mon[36291]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' 2024-03-16T11:52:01.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:52:01 smithi154 ceph-mon[36320]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:52:01.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:52:01 smithi154 ceph-mon[36320]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T11:52:01.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:52:01 smithi154 ceph-mon[36320]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' 2024-03-16T11:52:01.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:52:01 smithi029 ceph-mon[31975]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:52:01.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:52:01 smithi029 ceph-mon[31975]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T11:52:01.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:52:01 smithi029 ceph-mon[31975]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' 2024-03-16T11:52:01.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:52:01 smithi152 ceph-mon[36291]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:52:01.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:52:01 smithi152 ceph-mon[36291]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T11:52:01.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:52:01 smithi152 ceph-mon[36291]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' 2024-03-16T11:52:02.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:52:02 smithi154 ceph-mon[36320]: pgmap v36: 97 pgs: 97 active+clean; 1.4 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-03-16T11:52:02.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:52:02 smithi029 ceph-mon[31975]: pgmap v36: 97 pgs: 97 active+clean; 1.4 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-03-16T11:52:02.867 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:02.867+0000 7efc7ceb2640 1 -- 172.21.15.29:0/2300320820 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efc781012f0 msgr2=0x7efc7810d510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:52:02.867 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:02.867+0000 7efc7ceb2640 1 --2- 172.21.15.29:0/2300320820 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efc781012f0 0x7efc7810d510 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7efc70009cd0 tx=0x7efc7002c0f0 comp rx=0 tx=0).stop 2024-03-16T11:52:02.867 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:02.868+0000 7efc7ceb2640 1 -- 172.21.15.29:0/2300320820 shutdown_connections 2024-03-16T11:52:02.867 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:02.868+0000 7efc7ceb2640 1 --2- 172.21.15.29:0/2300320820 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efc781012f0 0x7efc7810d510 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:52:02.868 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:02.868+0000 7efc7ceb2640 1 --2- 172.21.15.29:0/2300320820 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efc78100930 0x7efc78100db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:52:02.868 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:02.868+0000 7efc7ceb2640 1 --2- 172.21.15.29:0/2300320820 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7efc78105680 0x7efc78105a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:52:02.868 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:02.868+0000 7efc7ceb2640 1 -- 172.21.15.29:0/2300320820 >> 172.21.15.29:0/2300320820 conn(0x7efc78077530 msgr2=0x7efc78077940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:52:02.868 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:02.868+0000 7efc7ceb2640 1 -- 172.21.15.29:0/2300320820 shutdown_connections 2024-03-16T11:52:02.868 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:02.868+0000 7efc7ceb2640 1 -- 172.21.15.29:0/2300320820 wait complete. 2024-03-16T11:52:02.870 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:02.869+0000 7efc7ceb2640 1 Processor -- start 2024-03-16T11:52:02.870 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:02.869+0000 7efc7ceb2640 1 -- start start 2024-03-16T11:52:02.870 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:02.869+0000 7efc7ceb2640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efc78100930 0x7efc781980b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:52:02.870 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:02.870+0000 7efc7ceb2640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7efc781012f0 0x7efc781985f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:52:02.870 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:02.870+0000 7efc7ceb2640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efc78105680 0x7efc7819c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:52:02.870 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:02.870+0000 7efc7659c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efc78100930 0x7efc781980b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:52:02.870 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:02.870+0000 7efc7659c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efc78100930 0x7efc781980b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.29:43906/0 (socket says 172.21.15.29:43906) 2024-03-16T11:52:02.870 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:02.870+0000 7efc7659c640 1 -- 172.21.15.29:0/3848268831 learned_addr learned my addr 172.21.15.29:0/3848268831 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:52:02.870 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:02.870+0000 7efc7ceb2640 1 -- 172.21.15.29:0/3848268831 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7efc78111c50 con 0x7efc78105680 2024-03-16T11:52:02.871 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:02.870+0000 7efc7ceb2640 1 -- 172.21.15.29:0/3848268831 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7efc78111ad0 con 0x7efc781012f0 2024-03-16T11:52:02.871 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:02.870+0000 7efc75d9b640 1 --2- 172.21.15.29:0/3848268831 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7efc781012f0 0x7efc781985f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:52:02.871 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:02.870+0000 7efc7ceb2640 1 -- 172.21.15.29:0/3848268831 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7efc78111dd0 con 0x7efc78100930 2024-03-16T11:52:02.871 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:02.870+0000 7efc76d9d640 1 --2- 172.21.15.29:0/3848268831 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efc78105680 0x7efc7819c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:52:02.871 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:02.870+0000 7efc75d9b640 1 -- 172.21.15.29:0/3848268831 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efc78100930 msgr2=0x7efc781980b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:52:02.871 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:02.870+0000 7efc75d9b640 1 --2- 172.21.15.29:0/3848268831 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efc78100930 0x7efc781980b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:52:02.871 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:02.870+0000 7efc75d9b640 1 -- 172.21.15.29:0/3848268831 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efc78105680 msgr2=0x7efc7819c920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:52:02.871 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:02.870+0000 7efc75d9b640 1 --2- 172.21.15.29:0/3848268831 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efc78105680 0x7efc7819c920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:52:02.871 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:02.870+0000 7efc75d9b640 1 -- 172.21.15.29:0/3848268831 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc7819d050 con 0x7efc781012f0 2024-03-16T11:52:02.871 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:02.870+0000 7efc7659c640 1 --2- 172.21.15.29:0/3848268831 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efc78100930 0x7efc781980b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:52:02.871 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:02.870+0000 7efc75d9b640 1 --2- 172.21.15.29:0/3848268831 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7efc781012f0 0x7efc781985f0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7efc6c00e9f0 tx=0x7efc6c00eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:52:02.871 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:02.871+0000 7efc677fe640 1 -- 172.21.15.29:0/3848268831 <== mon.2 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc6c00ccc0 con 0x7efc781012f0 2024-03-16T11:52:02.871 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:02.871+0000 7efc7ceb2640 1 -- 172.21.15.29:0/3848268831 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efc7819d340 con 0x7efc781012f0 2024-03-16T11:52:02.871 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:02.871+0000 7efc677fe640 1 -- 172.21.15.29:0/3848268831 <== mon.2 v2:172.21.15.152:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efc6c010910 con 0x7efc781012f0 2024-03-16T11:52:02.872 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:02.871+0000 7efc7ceb2640 1 -- 172.21.15.29:0/3848268831 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efc781a4bd0 con 0x7efc781012f0 2024-03-16T11:52:02.872 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:02.872+0000 7efc677fe640 1 -- 172.21.15.29:0/3848268831 <== mon.2 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc6c018960 con 0x7efc781012f0 2024-03-16T11:52:02.872 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:02.874+0000 7efc677fe640 1 -- 172.21.15.29:0/3848268831 <== mon.2 v2:172.21.15.152:3300/0 4 ==== mgrmap(e 24) ==== 137523+0+0 (secure 0 0 0) 0x7efc6c010ac0 con 0x7efc781012f0 2024-03-16T11:52:02.873 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:02.874+0000 7efc7fb7a740 1 Processor -- start 2024-03-16T11:52:02.874 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:02.874+0000 7efc7fb7a740 1 -- start start 2024-03-16T11:52:02.874 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:02.875+0000 7efc7fb7a740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f9ec442130 0x55f9ec442510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:52:02.874 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:02.875+0000 7efc7fb7a740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f9ec442a50 0x55f9ec44b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:52:02.874 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:02.875+0000 7efc7fb7a740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f9ec44b820 0x55f9ec44dc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:52:02.874 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:02.875+0000 7efc7fb7a740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55f9ec3a6a40 con 0x55f9ec44b820 2024-03-16T11:52:02.874 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:02.875+0000 7efc7fb7a740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x55f9ec34f920 con 0x55f9ec442130 2024-03-16T11:52:02.874 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:02.875+0000 7efc7fb7a740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55f9ec3a2ae0 con 0x55f9ec442a50 2024-03-16T11:52:02.874 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:02.875+0000 7efc76d9d640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f9ec442130 0x55f9ec442510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:52:02.876 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:02.875+0000 7efc75d9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f9ec44b820 0x55f9ec44dc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:52:02.876 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:02.875+0000 7efc7659c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f9ec442a50 0x55f9ec44b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:52:02.876 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:02.875+0000 7efc76d9d640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f9ec442130 0x55f9ec442510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.29:39236/0 (socket says 172.21.15.29:39236) 2024-03-16T11:52:02.876 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:02.875+0000 7efc76d9d640 1 -- 172.21.15.29:0/1508084839 learned_addr learned my addr 172.21.15.29:0/1508084839 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:52:02.876 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:02.875+0000 7efc7659c640 1 -- 172.21.15.29:0/1508084839 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f9ec442130 msgr2=0x55f9ec442510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:52:02.876 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:02.875+0000 7efc7659c640 1 --2- 172.21.15.29:0/1508084839 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f9ec442130 0x55f9ec442510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:52:02.876 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:02.875+0000 7efc7659c640 1 -- 172.21.15.29:0/1508084839 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f9ec44b820 msgr2=0x55f9ec44dc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:52:02.876 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:02.876+0000 7efc7659c640 1 --2- 172.21.15.29:0/1508084839 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f9ec44b820 0x55f9ec44dc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:52:02.876 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:02.876+0000 7efc7659c640 1 -- 172.21.15.29:0/1508084839 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f9ec367460 con 0x55f9ec442a50 2024-03-16T11:52:02.876 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:02.876+0000 7efc76d9d640 1 --2- 172.21.15.29:0/1508084839 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f9ec442130 0x55f9ec442510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T11:52:02.876 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:02.876+0000 7efc7659c640 1 --2- 172.21.15.29:0/1508084839 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f9ec442a50 0x55f9ec44b2e0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7efc68011440 tx=0x7efc68011900 comp rx=0 tx=0).ready entity=mon.1 client_cookie=48cdab254a08b8f8 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:52:02.876 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:02.876+0000 7efc64ff9640 1 -- 172.21.15.29:0/1508084839 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc68004030 con 0x55f9ec442a50 2024-03-16T11:52:02.876 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:02.876+0000 7efc64ff9640 1 -- 172.21.15.29:0/1508084839 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efc680041d0 con 0x55f9ec442a50 2024-03-16T11:52:02.877 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:02.876+0000 7efc64ff9640 1 -- 172.21.15.29:0/1508084839 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc6800f8f0 con 0x55f9ec442a50 2024-03-16T11:52:02.877 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:02.876+0000 7efc677fe640 1 --2- 172.21.15.29:0/3848268831 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7efc600a2bb0 0x7efc600a5070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:52:02.877 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:02.877+0000 7efc76d9d640 1 --2- 172.21.15.29:0/3848268831 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7efc600a2bb0 0x7efc600a5070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:52:02.877 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:02.877+0000 7efc677fe640 1 -- 172.21.15.29:0/3848268831 <== mon.2 v2:172.21.15.152:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8469+0+0 (secure 0 0 0) 0x7efc6c0cf2c0 con 0x7efc781012f0 2024-03-16T11:52:02.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:52:02 smithi152 ceph-mon[36291]: pgmap v36: 97 pgs: 97 active+clean; 1.4 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-03-16T11:52:02.880 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:02.877+0000 7efc7fb7a740 1 -- 172.21.15.29:0/1508084839 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f9ec442a50 msgr2=0x55f9ec44b2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:52:02.880 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:02.877+0000 7efc7fb7a740 1 --2- 172.21.15.29:0/1508084839 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f9ec442a50 0x55f9ec44b2e0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7efc68011440 tx=0x7efc68011900 comp rx=0 tx=0).stop 2024-03-16T11:52:02.880 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:02.877+0000 7efc76d9d640 1 --2- 172.21.15.29:0/3848268831 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7efc600a2bb0 0x7efc600a5070 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7efc70009a60 tx=0x7efc7002e040 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:52:02.881 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:02.877+0000 7efc7fb7a740 1 -- 172.21.15.29:0/1508084839 shutdown_connections 2024-03-16T11:52:02.881 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:02.877+0000 7efc7fb7a740 1 --2- 172.21.15.29:0/1508084839 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f9ec44b820 0x55f9ec44dc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:52:02.881 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:02.878+0000 7efc7fb7a740 1 --2- 172.21.15.29:0/1508084839 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f9ec442a50 0x55f9ec44b2e0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:52:02.881 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:02.878+0000 7efc7fb7a740 1 --2- 172.21.15.29:0/1508084839 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f9ec442130 0x55f9ec442510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:52:02.881 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:02.878+0000 7efc7fb7a740 1 -- 172.21.15.29:0/1508084839 >> 172.21.15.29:0/1508084839 conn(0x55f9ec43ca50 msgr2=0x55f9ec43ce30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:52:02.881 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:02.878+0000 7efc7fb7a740 1 -- 172.21.15.29:0/1508084839 shutdown_connections 2024-03-16T11:52:02.881 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:02.878+0000 7efc7fb7a740 1 -- 172.21.15.29:0/1508084839 wait complete. 2024-03-16T11:52:02.881 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:02.878+0000 7efc7fb7a740 1 -- start start 2024-03-16T11:52:02.881 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:02.878+0000 7efc7fb7a740 1 Processor -- start 2024-03-16T11:52:02.881 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:02.878+0000 7efc7fb7a740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f9ec442130 0x55f9ec448790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:52:02.881 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:02.878+0000 7efc7fb7a740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f9ec442a50 0x55f9ec448cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:52:02.881 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:02.879+0000 7efc7fb7a740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f9ec44b820 0x55f9ec449210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:52:02.881 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:02.879+0000 7efc7fb7a740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55f9ec452860 con 0x55f9ec442130 2024-03-16T11:52:02.882 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:02.879+0000 7efc7fb7a740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x55f9ec362d70 con 0x55f9ec442a50 2024-03-16T11:52:02.882 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:02.879+0000 7efc7fb7a740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55f9ec4529e0 con 0x55f9ec44b820 2024-03-16T11:52:02.882 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:02.879+0000 7efc75d9b640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f9ec442a50 0x55f9ec448cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:52:02.882 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:02.879+0000 7efc75d9b640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f9ec442a50 0x55f9ec448cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.29:39240/0 (socket says 172.21.15.29:39240) 2024-03-16T11:52:02.882 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:02.879+0000 7efc75d9b640 1 -- 172.21.15.29:0/244784339 learned_addr learned my addr 172.21.15.29:0/244784339 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:52:02.882 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:02.879+0000 7efc76d9d640 1 --2- 172.21.15.29:0/244784339 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f9ec44b820 0x55f9ec449210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:52:02.882 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:02.879+0000 7efc7fb7a740 10 client.0 _ll_register_callbacks cb 0x55f9ec0baa00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T11:52:02.882 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:02.879+0000 7efc7659c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f9ec442130 0x55f9ec448790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:52:02.882 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:02.879+0000 7efc75d9b640 1 -- 172.21.15.29:0/244784339 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f9ec44b820 msgr2=0x55f9ec449210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:52:02.882 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:02.879+0000 7efc75d9b640 1 --2- 172.21.15.29:0/244784339 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f9ec44b820 0x55f9ec449210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:52:02.882 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:02.879+0000 7efc75d9b640 1 -- 172.21.15.29:0/244784339 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f9ec442130 msgr2=0x55f9ec448790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:52:02.882 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:02.879+0000 7efc75d9b640 1 --2- 172.21.15.29:0/244784339 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f9ec442130 0x55f9ec448790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:52:02.882 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:02.879+0000 7efc75d9b640 1 -- 172.21.15.29:0/244784339 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f9ec0bb210 con 0x55f9ec442a50 2024-03-16T11:52:02.882 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:02.879+0000 7efc7659c640 1 --2- 172.21.15.29:0/244784339 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f9ec442130 0x55f9ec448790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:52:02.883 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:02.879+0000 7efc76d9d640 1 --2- 172.21.15.29:0/244784339 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f9ec44b820 0x55f9ec449210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:52:02.883 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:02.880+0000 7efc75d9b640 1 --2- 172.21.15.29:0/244784339 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f9ec442a50 0x55f9ec448cd0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7efc6c00c6a0 tx=0x7efc6c09b3f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d4a0954322cfd8f5 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:52:02.883 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:02.880+0000 7efc5dffb640 10 client.0 ms_handle_connect on v2:172.21.15.152:3300/0 2024-03-16T11:52:02.883 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:02.880+0000 7efc5dffb640 1 -- 172.21.15.29:0/244784339 <== mon.2 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc6c09b7b0 con 0x55f9ec442a50 2024-03-16T11:52:02.883 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:02.880+0000 7efc7fb7a740 1 -- 172.21.15.29:0/244784339 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efc680041d0 con 0x55f9ec442a50 2024-03-16T11:52:02.883 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:02.880+0000 7efc5dffb640 1 -- 172.21.15.29:0/244784339 <== mon.2 v2:172.21.15.152:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8469+0+0 (secure 0 0 0) 0x7efc6c0a3a10 con 0x55f9ec442a50 2024-03-16T11:52:02.883 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:02.880+0000 7efc5dffb640 1 -- 172.21.15.29:0/244784339 <== mon.2 v2:172.21.15.152:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7efc6c09f850 con 0x55f9ec442a50 2024-03-16T11:52:02.883 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:02.880+0000 7efc7fb7a740 10 client.25313 fetch_fsmap learned FSMap version 22 2024-03-16T11:52:02.883 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:02.880+0000 7efc7fb7a740 1 -- 172.21.15.29:0/244784339 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x55f9ec187620 con 0x55f9ec442a50 2024-03-16T11:52:02.883 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:02.880+0000 7efc5dffb640 1 -- 172.21.15.29:0/244784339 <== mon.2 v2:172.21.15.152:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efc680041d0 con 0x55f9ec442a50 2024-03-16T11:52:02.883 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:02.881+0000 7efc5dffb640 1 -- 172.21.15.29:0/244784339 <== mon.2 v2:172.21.15.152:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7efc6c09ac00 con 0x55f9ec442a50 2024-03-16T11:52:02.883 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:02.881+0000 7efc7fb7a740 10 client.25313 fetch_fsmap finished waiting for FSMap version 22 2024-03-16T11:52:02.884 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:02.881+0000 7efc7fb7a740 10 client.25313 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T11:52:02.884 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:02.881+0000 7efc7fb7a740 20 client.25313 populate_metadata read hostname 'smithi029' 2024-03-16T11:52:02.884 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:02.881+0000 7efc7fb7a740 1 --2- 172.21.15.29:0/244784339 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55f9ec460190 0x55f9ec4625d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:52:02.884 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:02.881+0000 7efc7fb7a740 4 client.25313 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T11:52:02.884 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:02.881+0000 7efc7fb7a740 1 -- 172.21.15.29:0/244784339 --> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f9ec3b3f80 con 0x55f9ec460190 2024-03-16T11:52:02.884 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:02.881+0000 7efc7659c640 1 --2- 172.21.15.29:0/244784339 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55f9ec460190 0x55f9ec4625d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:52:02.885 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:02.882+0000 7efc5dffb640 10 client.25313 ms_handle_connect on v2:172.21.15.152:6838/1548731833 2024-03-16T11:52:02.885 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:02.882+0000 7efc7659c640 1 --2- 172.21.15.29:0/244784339 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55f9ec460190 0x55f9ec4625d0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1b47cffba230707b server_cookie=92b09ee8c8618c5d in_seq=0 out_seq=0 2024-03-16T11:52:02.885 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:02.882+0000 7efc5dffb640 1 -- 172.21.15.29:0/244784339 <== mds.0 v2:172.21.15.152:6838/1548731833 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55f9ec3b3f80 con 0x55f9ec460190 2024-03-16T11:52:02.885 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:02.882+0000 7efc5dffb640 10 client.25313 handle_command_reply: tid=0 2024-03-16T11:52:02.885 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:02.883+0000 7efc7fb7a740 1 client.25313 shutdown 2024-03-16T11:52:02.885 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:02.883+0000 7efc7fb7a740 1 -- 172.21.15.29:0/244784339 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f9ec442a50 msgr2=0x55f9ec448cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:52:02.886 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:02.883+0000 7efc7fb7a740 1 --2- 172.21.15.29:0/244784339 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f9ec442a50 0x55f9ec448cd0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7efc6c00c6a0 tx=0x7efc6c09b3f0 comp rx=0 tx=0).stop 2024-03-16T11:52:02.886 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:02.883+0000 7efc7fb7a740 1 -- 172.21.15.29:0/244784339 shutdown_connections 2024-03-16T11:52:02.886 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:02.883+0000 7efc7fb7a740 1 --2- 172.21.15.29:0/244784339 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f9ec44b820 0x55f9ec449210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:52:02.886 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:02.883+0000 7efc7fb7a740 1 --2- 172.21.15.29:0/244784339 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55f9ec460190 0x55f9ec4625d0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:52:02.886 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:02.883+0000 7efc7fb7a740 1 --2- 172.21.15.29:0/244784339 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f9ec442a50 0x55f9ec448cd0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:52:02.886 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:02.883+0000 7efc7fb7a740 1 --2- 172.21.15.29:0/244784339 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f9ec442130 0x55f9ec448790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:52:02.886 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:02.883+0000 7efc7fb7a740 1 -- 172.21.15.29:0/244784339 >> 172.21.15.29:0/244784339 conn(0x55f9ec43ca50 msgr2=0x55f9ec43e7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:52:02.886 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:02.883+0000 7efc5dffb640 0 client.25313 ms_handle_reset on v2:172.21.15.152:6838/1548731833 2024-03-16T11:52:02.886 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:02.883+0000 7efc7fb7a740 1 -- 172.21.15.29:0/244784339 shutdown_connections 2024-03-16T11:52:02.886 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:02.883+0000 7efc7fb7a740 1 -- 172.21.15.29:0/244784339 wait complete. 2024-03-16T11:52:02.886 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:02.883+0000 7efc7fb7a740 20 client.25313 trim_cache size 0 max 16384 2024-03-16T11:52:02.914 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:02.913+0000 7efc7fb7a740 1 Processor -- start 2024-03-16T11:52:02.914 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:02.913+0000 7efc7fb7a740 1 -- start start 2024-03-16T11:52:02.914 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:02.913+0000 7efc7fb7a740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f9ec442130 0x55f9ec44aec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:52:02.914 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:02.913+0000 7efc7fb7a740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f9ec442a50 0x55f9ec45b600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:52:02.914 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:02.913+0000 7efc7fb7a740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f9ec44b820 0x55f9ec467ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:52:02.914 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:02.913+0000 7efc7fb7a740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7efc58010f90 con 0x55f9ec442a50 2024-03-16T11:52:02.914 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:02.913+0000 7efc7fb7a740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7efc5800e860 con 0x55f9ec44b820 2024-03-16T11:52:02.914 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:02.913+0000 7efc7fb7a740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7efc5800c6b0 con 0x55f9ec442130 2024-03-16T11:52:02.914 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:02.914+0000 7efc75d9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f9ec442a50 0x55f9ec45b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:52:02.914 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:02.914+0000 7efc75d9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f9ec442a50 0x55f9ec45b600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45152/0 (socket says 172.21.15.29:45152) 2024-03-16T11:52:02.914 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:02.914+0000 7efc75d9b640 1 -- 172.21.15.29:0/305882903 learned_addr learned my addr 172.21.15.29:0/305882903 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:52:02.915 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:02.914+0000 7efc76d9d640 1 --2- 172.21.15.29:0/305882903 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f9ec44b820 0x55f9ec467ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:52:02.915 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:02.914+0000 7efc7659c640 1 --2- 172.21.15.29:0/305882903 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f9ec442130 0x55f9ec44aec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:52:02.915 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:02.914+0000 7efc75d9b640 1 -- 172.21.15.29:0/305882903 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f9ec442130 msgr2=0x55f9ec44aec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:52:02.915 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:02.914+0000 7efc75d9b640 1 --2- 172.21.15.29:0/305882903 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f9ec442130 0x55f9ec44aec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:52:02.915 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:02.914+0000 7efc75d9b640 1 -- 172.21.15.29:0/305882903 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f9ec44b820 msgr2=0x55f9ec467ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:52:02.915 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:02.914+0000 7efc75d9b640 1 --2- 172.21.15.29:0/305882903 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f9ec44b820 0x55f9ec467ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:52:02.915 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:02.914+0000 7efc75d9b640 1 -- 172.21.15.29:0/305882903 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f9ec444390 con 0x55f9ec442a50 2024-03-16T11:52:02.915 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:02.914+0000 7efc75d9b640 1 --2- 172.21.15.29:0/305882903 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f9ec442a50 0x55f9ec45b600 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7efc6c0d2b70 tx=0x7efc6c0cc9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ead273d5921f7f38 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:52:02.915 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:02.914+0000 7efc5ffff640 1 -- 172.21.15.29:0/305882903 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc6c09b580 con 0x55f9ec442a50 2024-03-16T11:52:02.915 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:02.915+0000 7efc5ffff640 1 -- 172.21.15.29:0/305882903 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efc6c0a3ea0 con 0x55f9ec442a50 2024-03-16T11:52:02.915 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:02.915+0000 7efc5ffff640 1 -- 172.21.15.29:0/305882903 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc6c00ce60 con 0x55f9ec442a50 2024-03-16T11:52:02.919 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:02.915+0000 7efc7fb7a740 1 -- 172.21.15.29:0/305882903 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f9ec442a50 msgr2=0x55f9ec45b600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:52:02.919 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:02.915+0000 7efc7fb7a740 1 --2- 172.21.15.29:0/305882903 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f9ec442a50 0x55f9ec45b600 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7efc6c0d2b70 tx=0x7efc6c0cc9d0 comp rx=0 tx=0).stop 2024-03-16T11:52:02.919 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:02.916+0000 7efc7fb7a740 1 -- 172.21.15.29:0/305882903 shutdown_connections 2024-03-16T11:52:02.920 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:02.916+0000 7efc7fb7a740 1 --2- 172.21.15.29:0/305882903 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f9ec44b820 0x55f9ec467ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:52:02.920 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:02.916+0000 7efc7fb7a740 1 --2- 172.21.15.29:0/305882903 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f9ec442a50 0x55f9ec45b600 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:52:02.920 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:02.916+0000 7efc7fb7a740 1 --2- 172.21.15.29:0/305882903 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f9ec442130 0x55f9ec44aec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:52:02.920 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:02.916+0000 7efc7fb7a740 1 -- 172.21.15.29:0/305882903 >> 172.21.15.29:0/305882903 conn(0x55f9ec43ca50 msgr2=0x55f9ec43b320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:52:02.920 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:02.916+0000 7efc7fb7a740 1 -- 172.21.15.29:0/305882903 shutdown_connections 2024-03-16T11:52:02.920 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:02.916+0000 7efc7fb7a740 1 -- 172.21.15.29:0/305882903 wait complete. 2024-03-16T11:52:02.920 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:02.916+0000 7efc7fb7a740 1 -- start start 2024-03-16T11:52:02.920 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:02.916+0000 7efc7fb7a740 1 Processor -- start 2024-03-16T11:52:02.920 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:02.917+0000 7efc7fb7a740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f9ec442130 0x55f9ec457c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:52:02.920 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:02.917+0000 7efc7fb7a740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f9ec442a50 0x55f9ec444d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:52:02.920 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:02.917+0000 7efc7fb7a740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f9ec44b820 0x55f9ec4452b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:52:02.920 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:02.917+0000 7efc7fb7a740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7efc5800cd20 con 0x55f9ec44b820 2024-03-16T11:52:02.920 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:02.917+0000 7efc7fb7a740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7efc5800e190 con 0x55f9ec442130 2024-03-16T11:52:02.920 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:02.917+0000 7efc7fb7a740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7efc5800d3f0 con 0x55f9ec442a50 2024-03-16T11:52:02.921 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:02.917+0000 7efc7fb7a740 10 client.0 _ll_register_callbacks cb 0x7efc6c0d2920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T11:52:02.921 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:02.917+0000 7efc7659c640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f9ec442130 0x55f9ec457c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:52:02.921 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:02.917+0000 7efc7659c640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f9ec442130 0x55f9ec457c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.29:39256/0 (socket says 172.21.15.29:39256) 2024-03-16T11:52:02.921 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:02.917+0000 7efc7659c640 1 -- 172.21.15.29:0/4073331070 learned_addr learned my addr 172.21.15.29:0/4073331070 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:52:02.921 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:02.917+0000 7efc75d9b640 1 --2- 172.21.15.29:0/4073331070 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f9ec442a50 0x55f9ec444d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:52:02.921 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:02.917+0000 7efc76d9d640 1 --2- 172.21.15.29:0/4073331070 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f9ec44b820 0x55f9ec4452b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:52:02.921 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:02.917+0000 7efc75d9b640 1 -- 172.21.15.29:0/4073331070 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f9ec442130 msgr2=0x55f9ec457c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:52:02.921 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:02.917+0000 7efc75d9b640 1 --2- 172.21.15.29:0/4073331070 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f9ec442130 0x55f9ec457c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:52:02.921 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:02.917+0000 7efc75d9b640 1 -- 172.21.15.29:0/4073331070 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f9ec44b820 msgr2=0x55f9ec4452b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:52:02.921 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:02.917+0000 7efc75d9b640 1 --2- 172.21.15.29:0/4073331070 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f9ec44b820 0x55f9ec4452b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:52:02.921 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:02.917+0000 7efc75d9b640 1 -- 172.21.15.29:0/4073331070 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f9ec445820 con 0x55f9ec442a50 2024-03-16T11:52:02.921 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:02.918+0000 7efc75d9b640 1 --2- 172.21.15.29:0/4073331070 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f9ec442a50 0x55f9ec444d70 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x55f9ec441bb0 tx=0x7efc6c0d2b00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=99f9daa75d21e4f server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:52:02.921 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:02.918+0000 7efc5dffb640 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-16T11:52:02.921 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:02.918+0000 7efc5dffb640 1 -- 172.21.15.29:0/4073331070 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc6c00ce60 con 0x55f9ec442a50 2024-03-16T11:52:02.921 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:02.918+0000 7efc5dffb640 1 -- 172.21.15.29:0/4073331070 <== mon.1 v2:172.21.15.154:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8469+0+0 (secure 0 0 0) 0x7efc6c018d00 con 0x55f9ec442a50 2024-03-16T11:52:02.922 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:02.918+0000 7efc7fb7a740 1 -- 172.21.15.29:0/4073331070 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efc6c0a3ea0 con 0x55f9ec442a50 2024-03-16T11:52:02.922 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:02.918+0000 7efc5dffb640 1 -- 172.21.15.29:0/4073331070 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7efc6c09b580 con 0x55f9ec442a50 2024-03-16T11:52:02.922 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:02.918+0000 7efc7fb7a740 10 client.25351 fetch_fsmap learned FSMap version 22 2024-03-16T11:52:02.922 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:02.918+0000 7efc7fb7a740 1 -- 172.21.15.29:0/4073331070 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x55f9ec4c62c0 con 0x55f9ec442a50 2024-03-16T11:52:02.922 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:02.918+0000 7efc5dffb640 1 -- 172.21.15.29:0/4073331070 <== mon.1 v2:172.21.15.154:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efc6c0a3ea0 con 0x55f9ec442a50 2024-03-16T11:52:02.922 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:02.918+0000 7efc5dffb640 1 -- 172.21.15.29:0/4073331070 <== mon.1 v2:172.21.15.154:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7efc6c0afc70 con 0x55f9ec442a50 2024-03-16T11:52:02.922 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:02.919+0000 7efc7fb7a740 10 client.25351 fetch_fsmap finished waiting for FSMap version 22 2024-03-16T11:52:02.922 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:02.919+0000 7efc7fb7a740 10 client.25351 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T11:52:02.922 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:02.919+0000 7efc7fb7a740 20 client.25351 populate_metadata read hostname 'smithi029' 2024-03-16T11:52:02.922 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:02.919+0000 7efc7fb7a740 1 --2- 172.21.15.29:0/4073331070 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55f9ec4c6740 0x55f9ec4c8b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:52:02.922 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:02.919+0000 7efc7fb7a740 4 client.25351 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-16T11:52:02.922 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:02.919+0000 7efc7fb7a740 1 -- 172.21.15.29:0/4073331070 --> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55f9ec3ae2d0 con 0x55f9ec4c6740 2024-03-16T11:52:02.922 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:02.919+0000 7efc7659c640 1 --2- 172.21.15.29:0/4073331070 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55f9ec4c6740 0x55f9ec4c8b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:52:02.922 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:02.919+0000 7efc5dffb640 10 client.25351 ms_handle_connect on v2:172.21.15.152:6838/1548731833 2024-03-16T11:52:02.922 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:02.919+0000 7efc7659c640 1 --2- 172.21.15.29:0/4073331070 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55f9ec4c6740 0x55f9ec4c8b30 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=25305eb98074f791 server_cookie=7c6412402a460ffc in_seq=0 out_seq=0 2024-03-16T11:52:03.809 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:03.808+0000 7efc5dffb640 1 -- 172.21.15.29:0/4073331070 <== mds.0 v2:172.21.15.152:6838/1548731833 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55f9ec3ae2d0 con 0x55f9ec4c6740 2024-03-16T11:52:03.809 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:03.808+0000 7efc5dffb640 10 client.25351 handle_command_reply: tid=0 2024-03-16T11:52:03.809 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:03.808+0000 7efc7fb7a740 1 client.25351 shutdown 2024-03-16T11:52:03.809 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:03.808+0000 7efc7fb7a740 1 -- 172.21.15.29:0/4073331070 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f9ec442a50 msgr2=0x55f9ec444d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:52:03.809 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:03.808+0000 7efc7fb7a740 1 --2- 172.21.15.29:0/4073331070 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f9ec442a50 0x55f9ec444d70 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x55f9ec441bb0 tx=0x7efc6c0d2b00 comp rx=0 tx=0).stop 2024-03-16T11:52:03.809 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:03.808+0000 7efc7fb7a740 1 -- 172.21.15.29:0/4073331070 shutdown_connections 2024-03-16T11:52:03.809 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:03.808+0000 7efc7fb7a740 1 --2- 172.21.15.29:0/4073331070 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f9ec44b820 0x55f9ec4452b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:52:03.809 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:03.808+0000 7efc7fb7a740 1 --2- 172.21.15.29:0/4073331070 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f9ec442a50 0x55f9ec444d70 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:52:03.809 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:03.809+0000 7efc7fb7a740 1 --2- 172.21.15.29:0/4073331070 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55f9ec4c6740 0x55f9ec4c8b30 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:52:03.810 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:03.809+0000 7efc7fb7a740 1 --2- 172.21.15.29:0/4073331070 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f9ec442130 0x55f9ec457c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:52:03.810 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:03.809+0000 7efc7fb7a740 1 -- 172.21.15.29:0/4073331070 >> 172.21.15.29:0/4073331070 conn(0x55f9ec43ca50 msgr2=0x55f9ec448fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:52:03.810 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:03.809+0000 7efc5dffb640 0 client.25351 ms_handle_reset on v2:172.21.15.152:6838/1548731833 2024-03-16T11:52:03.810 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:03.809+0000 7efc7fb7a740 1 -- 172.21.15.29:0/4073331070 shutdown_connections 2024-03-16T11:52:03.810 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:03.809+0000 7efc7fb7a740 1 -- 172.21.15.29:0/4073331070 wait complete. 2024-03-16T11:52:03.810 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:03.809+0000 7efc7fb7a740 20 client.25351 trim_cache size 0 max 16384 2024-03-16T11:52:03.810 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:{ 2024-03-16T11:52:03.810 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "message": "", 2024-03-16T11:52:03.810 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "return_code": 0 2024-03-16T11:52:03.810 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:} 2024-03-16T11:52:03.811 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:03.809+0000 7efc7ceb2640 1 -- 172.21.15.29:0/3848268831 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7efc600a2bb0 msgr2=0x7efc600a5070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:52:03.811 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:03.809+0000 7efc7ceb2640 1 --2- 172.21.15.29:0/3848268831 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7efc600a2bb0 0x7efc600a5070 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7efc70009a60 tx=0x7efc7002e040 comp rx=0 tx=0).stop 2024-03-16T11:52:03.811 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:03.809+0000 7efc7ceb2640 1 -- 172.21.15.29:0/3848268831 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7efc781012f0 msgr2=0x7efc781985f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:52:03.811 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:03.809+0000 7efc7ceb2640 1 --2- 172.21.15.29:0/3848268831 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7efc781012f0 0x7efc781985f0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7efc6c00e9f0 tx=0x7efc6c00eeb0 comp rx=0 tx=0).stop 2024-03-16T11:52:03.812 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:03.810+0000 7efc7ceb2640 1 -- 172.21.15.29:0/3848268831 shutdown_connections 2024-03-16T11:52:03.812 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:03.810+0000 7efc7ceb2640 1 --2- 172.21.15.29:0/3848268831 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7efc600a2bb0 0x7efc600a5070 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:52:03.812 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:03.810+0000 7efc7ceb2640 1 --2- 172.21.15.29:0/3848268831 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efc78105680 0x7efc7819c920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:52:03.812 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:03.810+0000 7efc7ceb2640 1 --2- 172.21.15.29:0/3848268831 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7efc781012f0 0x7efc781985f0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:52:03.812 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:03.810+0000 7efc7ceb2640 1 --2- 172.21.15.29:0/3848268831 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efc78100930 0x7efc781980b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:52:03.812 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:03.810+0000 7efc7ceb2640 1 -- 172.21.15.29:0/3848268831 >> 172.21.15.29:0/3848268831 conn(0x7efc78077530 msgr2=0x7efc780fe980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:52:03.812 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:03.810+0000 7efc7ceb2640 1 -- 172.21.15.29:0/3848268831 shutdown_connections 2024-03-16T11:52:03.812 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:03.810+0000 7efc7ceb2640 1 -- 172.21.15.29:0/3848268831 wait complete. 2024-03-16T11:52:04.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:52:04 smithi154 ceph-mon[36320]: pgmap v37: 97 pgs: 97 active+clean; 1.4 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-03-16T11:52:04.837 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:52:04 smithi029 ceph-mon[31975]: pgmap v37: 97 pgs: 97 active+clean; 1.4 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-03-16T11:52:04.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:52:04 smithi152 ceph-mon[36291]: pgmap v37: 97 pgs: 97 active+clean; 1.4 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-03-16T11:52:06.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:52:06 smithi154 ceph-mon[36320]: pgmap v38: 97 pgs: 97 active+clean; 1.4 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 19 op/s 2024-03-16T11:52:06.837 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:52:06 smithi029 ceph-mon[31975]: pgmap v38: 97 pgs: 97 active+clean; 1.4 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 19 op/s 2024-03-16T11:52:06.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:52:06 smithi152 ceph-mon[36291]: pgmap v38: 97 pgs: 97 active+clean; 1.4 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 19 op/s 2024-03-16T11:52:08.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:52:08 smithi154 ceph-mon[36320]: pgmap v39: 97 pgs: 97 active+clean; 1.4 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 23 KiB/s wr, 23 op/s 2024-03-16T11:52:08.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:52:08 smithi029 ceph-mon[31975]: pgmap v39: 97 pgs: 97 active+clean; 1.4 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 23 KiB/s wr, 23 op/s 2024-03-16T11:52:08.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:52:08 smithi152 ceph-mon[36291]: pgmap v39: 97 pgs: 97 active+clean; 1.4 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 23 KiB/s wr, 23 op/s 2024-03-16T11:52:10.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:52:10 smithi154 ceph-mon[36320]: pgmap v40: 97 pgs: 97 active+clean; 1.4 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 23 KiB/s wr, 20 op/s 2024-03-16T11:52:10.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:52:10 smithi029 ceph-mon[31975]: pgmap v40: 97 pgs: 97 active+clean; 1.4 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 23 KiB/s wr, 20 op/s 2024-03-16T11:52:10.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:52:10 smithi152 ceph-mon[36291]: pgmap v40: 97 pgs: 97 active+clean; 1.4 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 23 KiB/s wr, 20 op/s 2024-03-16T11:52:12.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:52:12 smithi154 ceph-mon[36320]: pgmap v41: 97 pgs: 97 active+clean; 1.4 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 23 KiB/s wr, 23 op/s 2024-03-16T11:52:12.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:52:12 smithi029 ceph-mon[31975]: pgmap v41: 97 pgs: 97 active+clean; 1.4 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 23 KiB/s wr, 23 op/s 2024-03-16T11:52:12.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:52:12 smithi152 ceph-mon[36291]: pgmap v41: 97 pgs: 97 active+clean; 1.4 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 23 KiB/s wr, 23 op/s 2024-03-16T11:52:13.549 INFO:tasks.workunit.client.0.smithi029.stderr:Updating files: 95% (11661/12271) Updating files: 96% (11781/12271) Updating files: 97% (11903/12271) Updating files: 98% (12026/12271) Updating files: 99% (12149/12271) Updating files: 100% (12271/12271) Updating files: 100% (12271/12271), done. 2024-03-16T11:52:14.160 INFO:tasks.workunit.client.0.smithi029.stderr:Note: switching to '3d78f5ba07bf18880172c07d4d7b4ac00c9d30af'. 2024-03-16T11:52:14.160 INFO:tasks.workunit.client.0.smithi029.stderr: 2024-03-16T11:52:14.160 INFO:tasks.workunit.client.0.smithi029.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-03-16T11:52:14.160 INFO:tasks.workunit.client.0.smithi029.stderr:changes and commit them, and you can discard any commits you make in this 2024-03-16T11:52:14.161 INFO:tasks.workunit.client.0.smithi029.stderr:state without impacting any branches by switching back to a branch. 2024-03-16T11:52:14.161 INFO:tasks.workunit.client.0.smithi029.stderr: 2024-03-16T11:52:14.161 INFO:tasks.workunit.client.0.smithi029.stderr:If you want to create a new branch to retain commits you create, you may 2024-03-16T11:52:14.161 INFO:tasks.workunit.client.0.smithi029.stderr:do so (now or later) by using -c with the switch command. Example: 2024-03-16T11:52:14.161 INFO:tasks.workunit.client.0.smithi029.stderr: 2024-03-16T11:52:14.161 INFO:tasks.workunit.client.0.smithi029.stderr: git switch -c 2024-03-16T11:52:14.161 INFO:tasks.workunit.client.0.smithi029.stderr: 2024-03-16T11:52:14.161 INFO:tasks.workunit.client.0.smithi029.stderr:Or undo this operation with: 2024-03-16T11:52:14.161 INFO:tasks.workunit.client.0.smithi029.stderr: 2024-03-16T11:52:14.161 INFO:tasks.workunit.client.0.smithi029.stderr: git switch - 2024-03-16T11:52:14.161 INFO:tasks.workunit.client.0.smithi029.stderr: 2024-03-16T11:52:14.161 INFO:tasks.workunit.client.0.smithi029.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-03-16T11:52:14.161 INFO:tasks.workunit.client.0.smithi029.stderr: 2024-03-16T11:52:14.161 INFO:tasks.workunit.client.0.smithi029.stderr:HEAD is now at 3d78f5ba07b Merge PR #54581 into wip-batrick-testing-20240314.183243 2024-03-16T11:52:14.165 DEBUG:teuthology.orchestra.run.smithi029:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-03-16T11:52:14.234 INFO:tasks.workunit.client.0.smithi029.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-03-16T11:52:14.236 INFO:tasks.workunit.client.0.smithi029.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-03-16T11:52:14.237 INFO:tasks.workunit.client.0.smithi029.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-03-16T11:52:14.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:52:14 smithi154 ceph-mon[36320]: pgmap v42: 97 pgs: 97 active+clean; 1.4 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 682 B/s wr, 6 op/s 2024-03-16T11:52:14.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:52:14 smithi029 ceph-mon[31975]: pgmap v42: 97 pgs: 97 active+clean; 1.4 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 682 B/s wr, 6 op/s 2024-03-16T11:52:14.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:52:14 smithi152 ceph-mon[36291]: pgmap v42: 97 pgs: 97 active+clean; 1.4 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 682 B/s wr, 6 op/s 2024-03-16T11:52:15.364 INFO:tasks.workunit.client.0.smithi029.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-03-16T11:52:15.457 INFO:tasks.workunit.client.0.smithi029.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-03-16T11:52:15.498 INFO:tasks.workunit.client.0.smithi029.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-03-16T11:52:15.500 INFO:tasks.workunit.client.0.smithi029.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-03-16T11:52:15.500 INFO:tasks.workunit.client.0.smithi029.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-03-16T11:52:15.540 INFO:tasks.workunit.client.0.smithi029.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-03-16T11:52:15.543 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-03-16T11:52:15.543 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-03-16T11:52:15.598 INFO:tasks.workunit:Running workunits matching suites/dbench.sh on client.0... 2024-03-16T11:52:15.599 INFO:tasks.workunit:Running workunit suites/dbench.sh... 2024-03-16T11:52:15.600 DEBUG:teuthology.orchestra.run.smithi029:workunit test suites/dbench.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=3d78f5ba07bf18880172c07d4d7b4ac00c9d30af TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh 2024-03-16T11:52:15.660 INFO:tasks.workunit.client.0.smithi029.stderr:+ dbench 1 2024-03-16T11:52:15.667 INFO:tasks.workunit.client.0.smithi029.stdout:dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 2024-03-16T11:52:15.667 INFO:tasks.workunit.client.0.smithi029.stdout: 2024-03-16T11:52:15.668 INFO:tasks.workunit.client.0.smithi029.stdout:Running for 600 seconds with load '/usr/share/dbench/client.txt' and minimum warmup 120 secs 2024-03-16T11:52:15.673 INFO:tasks.workunit.client.0.smithi029.stdout:0 of 1 processes prepared for launch 0 sec 2024-03-16T11:52:15.773 INFO:tasks.workunit.client.0.smithi029.stdout:1 of 1 processes prepared for launch 0 sec 2024-03-16T11:52:15.774 INFO:tasks.workunit.client.0.smithi029.stdout:releasing clients 2024-03-16T11:52:16.773 INFO:tasks.workunit.client.0.smithi029.stdout: 1 2902 32.82 MB/sec warmup 1 sec latency 7.855 ms 2024-03-16T11:52:16.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:52:16 smithi154 ceph-mon[36320]: pgmap v43: 97 pgs: 97 active+clean; 1.4 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 682 B/s wr, 5 op/s 2024-03-16T11:52:16.837 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:52:16 smithi029 ceph-mon[31975]: pgmap v43: 97 pgs: 97 active+clean; 1.4 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 682 B/s wr, 5 op/s 2024-03-16T11:52:16.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:52:16 smithi152 ceph-mon[36291]: pgmap v43: 97 pgs: 97 active+clean; 1.4 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 682 B/s wr, 5 op/s 2024-03-16T11:52:16.904 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:16.903+0000 7f0a5b612640 1 -- 172.21.15.29:0/2895345364 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0a54105680 msgr2=0x7f0a54105a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:52:16.904 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:16.903+0000 7f0a5b612640 1 --2- 172.21.15.29:0/2895345364 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0a54105680 0x7f0a54105a60 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f0a50009930 tx=0x7f0a5002bc30 comp rx=0 tx=0).stop 2024-03-16T11:52:16.904 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:16.903+0000 7f0a5b612640 1 -- 172.21.15.29:0/2895345364 shutdown_connections 2024-03-16T11:52:16.904 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:16.903+0000 7f0a5b612640 1 --2- 172.21.15.29:0/2895345364 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0a541012f0 0x7f0a5410d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:52:16.904 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:16.903+0000 7f0a5b612640 1 --2- 172.21.15.29:0/2895345364 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0a54100930 0x7f0a54100db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:52:16.905 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:16.903+0000 7f0a5b612640 1 --2- 172.21.15.29:0/2895345364 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0a54105680 0x7f0a54105a60 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:52:16.905 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:16.903+0000 7f0a5b612640 1 -- 172.21.15.29:0/2895345364 >> 172.21.15.29:0/2895345364 conn(0x7f0a54077550 msgr2=0x7f0a54077960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:52:16.905 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:16.903+0000 7f0a5b612640 1 -- 172.21.15.29:0/2895345364 shutdown_connections 2024-03-16T11:52:16.905 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:16.903+0000 7f0a5b612640 1 -- 172.21.15.29:0/2895345364 wait complete. 2024-03-16T11:52:16.905 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:16.904+0000 7f0a5b612640 1 Processor -- start 2024-03-16T11:52:16.905 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:16.904+0000 7f0a5b612640 1 -- start start 2024-03-16T11:52:16.905 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:16.904+0000 7f0a5b612640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0a54100930 0x7f0a54198070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:52:16.905 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:16.904+0000 7f0a5b612640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0a541012f0 0x7f0a541985b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:52:16.905 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:16.904+0000 7f0a5b612640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0a54105680 0x7f0a5419c930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:52:16.905 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:16.904+0000 7f0a5b612640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f0a54110b90 con 0x7f0a54100930 2024-03-16T11:52:16.905 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:16.904+0000 7f0a5b612640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f0a54110a10 con 0x7f0a541012f0 2024-03-16T11:52:16.905 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:16.904+0000 7f0a5b612640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f0a54110d10 con 0x7f0a54105680 2024-03-16T11:52:16.906 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:16.904+0000 7f0a593ae640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0a54100930 0x7f0a54198070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:52:16.906 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:16.904+0000 7f0a593ae640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0a54100930 0x7f0a54198070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49634/0 (socket says 172.21.15.29:49634) 2024-03-16T11:52:16.906 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:16.904+0000 7f0a593ae640 1 -- 172.21.15.29:0/734178501 learned_addr learned my addr 172.21.15.29:0/734178501 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:52:16.906 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:16.904+0000 7f0a58bad640 1 --2- 172.21.15.29:0/734178501 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0a541012f0 0x7f0a541985b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:52:16.906 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:16.904+0000 7f0a59baf640 1 --2- 172.21.15.29:0/734178501 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0a54105680 0x7f0a5419c930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:52:16.906 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:16.904+0000 7f0a593ae640 1 -- 172.21.15.29:0/734178501 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0a54105680 msgr2=0x7f0a5419c930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:52:16.906 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:16.904+0000 7f0a593ae640 1 --2- 172.21.15.29:0/734178501 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0a54105680 0x7f0a5419c930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:52:16.906 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:16.904+0000 7f0a593ae640 1 -- 172.21.15.29:0/734178501 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0a541012f0 msgr2=0x7f0a541985b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:52:16.906 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:16.904+0000 7f0a593ae640 1 --2- 172.21.15.29:0/734178501 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0a541012f0 0x7f0a541985b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:52:16.906 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:16.904+0000 7f0a593ae640 1 -- 172.21.15.29:0/734178501 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a5419d060 con 0x7f0a54100930 2024-03-16T11:52:16.906 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:16.904+0000 7f0a593ae640 1 --2- 172.21.15.29:0/734178501 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0a54100930 0x7f0a54198070 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f0a50009230 tx=0x7f0a50004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:52:16.906 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:16.905+0000 7f0a4a7fc640 1 -- 172.21.15.29:0/734178501 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a50041070 con 0x7f0a54100930 2024-03-16T11:52:16.906 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:16.905+0000 7f0a5b612640 1 -- 172.21.15.29:0/734178501 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0a5419d2f0 con 0x7f0a54100930 2024-03-16T11:52:16.906 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:16.905+0000 7f0a4a7fc640 1 -- 172.21.15.29:0/734178501 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0a500054e0 con 0x7f0a54100930 2024-03-16T11:52:16.907 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:16.905+0000 7f0a4a7fc640 1 -- 172.21.15.29:0/734178501 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a50003c90 con 0x7f0a54100930 2024-03-16T11:52:16.907 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:16.905+0000 7f0a5b612640 1 -- 172.21.15.29:0/734178501 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0a541a4be0 con 0x7f0a54100930 2024-03-16T11:52:16.913 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:16.907+0000 7f0a4a7fc640 1 -- 172.21.15.29:0/734178501 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 24) ==== 137523+0+0 (secure 0 0 0) 0x7f0a50044070 con 0x7f0a54100930 2024-03-16T11:52:16.914 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:16.908+0000 7f0a5e200740 1 Processor -- start 2024-03-16T11:52:16.914 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:16.908+0000 7f0a5e200740 1 -- start start 2024-03-16T11:52:16.914 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:16.908+0000 7f0a5e200740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a5141efc50 0x55a5141f0030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:52:16.914 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:16.908+0000 7f0a5e200740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a5141f0570 0x55a5141fd6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:52:16.914 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:16.908+0000 7f0a5e200740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a5141fdc30 0x55a514200020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:52:16.914 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:16.908+0000 7f0a5e200740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55a514154a40 con 0x55a5141f0570 2024-03-16T11:52:16.914 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:16.908+0000 7f0a4a7fc640 1 --2- 172.21.15.29:0/734178501 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f0a400a2b30 0x7f0a400a4ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:52:16.914 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:16.908+0000 7f0a5e200740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x55a5140fd920 con 0x55a5141fdc30 2024-03-16T11:52:16.914 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:16.908+0000 7f0a5e200740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55a514150ae0 con 0x55a5141efc50 2024-03-16T11:52:16.914 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:16.908+0000 7f0a593ae640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a5141f0570 0x55a5141fd6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:52:16.914 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:16.908+0000 7f0a59baf640 1 --2- 172.21.15.29:0/734178501 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f0a400a2b30 0x7f0a400a4ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:52:16.914 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:16.908+0000 7f0a58bad640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a5141fdc30 0x55a514200020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:52:16.914 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:16.908+0000 7f0a58bad640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a5141fdc30 0x55a514200020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.29:53512/0 (socket says 172.21.15.29:53512) 2024-03-16T11:52:16.915 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:16.908+0000 7f0a58bad640 1 -- 172.21.15.29:0/350976659 learned_addr learned my addr 172.21.15.29:0/350976659 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:52:16.915 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:16.908+0000 7f0a593ae640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a5141f0570 0x55a5141fd6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49650/0 (socket says 172.21.15.29:49650) 2024-03-16T11:52:16.915 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:16.908+0000 7f0a59baf640 1 --2- 172.21.15.29:0/350976659 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a5141efc50 0x55a5141f0030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:52:16.915 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:16.908+0000 7f0a4a7fc640 1 -- 172.21.15.29:0/734178501 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8469+0+0 (secure 0 0 0) 0x7f0a500f0c10 con 0x7f0a54100930 2024-03-16T11:52:16.915 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:16.909+0000 7f0a58bad640 1 -- 172.21.15.29:0/350976659 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a5141efc50 msgr2=0x55a5141f0030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:52:16.915 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:16.909+0000 7f0a58bad640 1 --2- 172.21.15.29:0/350976659 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a5141efc50 0x55a5141f0030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:52:16.915 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:16.909+0000 7f0a58bad640 1 -- 172.21.15.29:0/350976659 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a5141f0570 msgr2=0x55a5141fd6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:52:16.915 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:16.909+0000 7f0a58bad640 1 --2- 172.21.15.29:0/350976659 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a5141f0570 0x55a5141fd6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:52:16.915 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:16.909+0000 7f0a58bad640 1 -- 172.21.15.29:0/350976659 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a514115460 con 0x55a5141fdc30 2024-03-16T11:52:16.915 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:16.909+0000 7f0a59baf640 1 --2- 172.21.15.29:0/734178501 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f0a400a2b30 0x7f0a400a4ff0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f0a4c005fd0 tx=0x7f0a4c009bd0 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:52:16.915 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:16.909+0000 7f0a58bad640 1 --2- 172.21.15.29:0/350976659 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a5141fdc30 0x55a514200020 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f0a44009ca0 tx=0x7f0a44011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=30a4b315574079e9 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:52:16.915 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:16.909+0000 7f0a3f7fe640 1 -- 172.21.15.29:0/350976659 <== mon.2 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a4401e070 con 0x55a5141fdc30 2024-03-16T11:52:16.915 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:16.909+0000 7f0a3f7fe640 1 -- 172.21.15.29:0/350976659 <== mon.2 v2:172.21.15.152:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0a44019b10 con 0x55a5141fdc30 2024-03-16T11:52:16.915 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:16.909+0000 7f0a3f7fe640 1 -- 172.21.15.29:0/350976659 <== mon.2 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a4400ea30 con 0x55a5141fdc30 2024-03-16T11:52:16.916 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:16.910+0000 7f0a5e200740 1 -- 172.21.15.29:0/350976659 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a5141fdc30 msgr2=0x55a514200020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:52:16.916 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:16.910+0000 7f0a5e200740 1 --2- 172.21.15.29:0/350976659 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a5141fdc30 0x55a514200020 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f0a44009ca0 tx=0x7f0a44011810 comp rx=0 tx=0).stop 2024-03-16T11:52:16.916 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:16.910+0000 7f0a5e200740 1 -- 172.21.15.29:0/350976659 shutdown_connections 2024-03-16T11:52:16.916 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:16.910+0000 7f0a5e200740 1 --2- 172.21.15.29:0/350976659 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a5141fdc30 0x55a514200020 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:52:16.916 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:16.910+0000 7f0a5e200740 1 --2- 172.21.15.29:0/350976659 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a5141f0570 0x55a5141fd6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:52:16.916 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:16.910+0000 7f0a5e200740 1 --2- 172.21.15.29:0/350976659 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a5141efc50 0x55a5141f0030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:52:16.916 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:16.910+0000 7f0a5e200740 1 -- 172.21.15.29:0/350976659 >> 172.21.15.29:0/350976659 conn(0x55a5141eaa50 msgr2=0x55a5141eae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:52:16.916 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:16.910+0000 7f0a5e200740 1 -- 172.21.15.29:0/350976659 shutdown_connections 2024-03-16T11:52:16.916 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:16.910+0000 7f0a5e200740 1 -- 172.21.15.29:0/350976659 wait complete. 2024-03-16T11:52:16.916 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:16.910+0000 7f0a5e200740 1 -- start start 2024-03-16T11:52:16.916 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:16.910+0000 7f0a5e200740 1 Processor -- start 2024-03-16T11:52:16.916 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:16.911+0000 7f0a5e200740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a5141efc50 0x55a5141fac30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:52:16.916 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:16.911+0000 7f0a5e200740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a5141f0570 0x55a5141fb170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:52:16.916 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:16.911+0000 7f0a5e200740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a5141fdc30 0x55a5141fb6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:52:16.917 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:16.911+0000 7f0a5e200740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55a5142007f0 con 0x55a5141fdc30 2024-03-16T11:52:16.917 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:16.911+0000 7f0a5e200740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x55a514110d70 con 0x55a5141f0570 2024-03-16T11:52:16.917 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:16.911+0000 7f0a5e200740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55a514200970 con 0x55a5141efc50 2024-03-16T11:52:16.917 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:16.911+0000 7f0a5e200740 10 client.0 _ll_register_callbacks cb 0x55a513e68a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T11:52:16.917 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:16.911+0000 7f0a59baf640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a5141fdc30 0x55a5141fb6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:52:16.917 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:16.911+0000 7f0a59baf640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a5141fdc30 0x55a5141fb6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49664/0 (socket says 172.21.15.29:49664) 2024-03-16T11:52:16.917 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:16.911+0000 7f0a593ae640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a5141efc50 0x55a5141fac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:52:16.917 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:16.911+0000 7f0a59baf640 1 -- 172.21.15.29:0/2089500253 learned_addr learned my addr 172.21.15.29:0/2089500253 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:52:16.917 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:16.911+0000 7f0a58bad640 1 --2- 172.21.15.29:0/2089500253 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a5141f0570 0x55a5141fb170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:52:16.917 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:16.911+0000 7f0a59baf640 1 -- 172.21.15.29:0/2089500253 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a5141efc50 msgr2=0x55a5141fac30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:52:16.917 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:16.911+0000 7f0a59baf640 1 --2- 172.21.15.29:0/2089500253 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a5141efc50 0x55a5141fac30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:52:16.917 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:16.911+0000 7f0a59baf640 1 -- 172.21.15.29:0/2089500253 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a5141f0570 msgr2=0x55a5141fb170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:52:16.917 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:16.911+0000 7f0a59baf640 1 --2- 172.21.15.29:0/2089500253 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a5141f0570 0x55a5141fb170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:52:16.917 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:16.911+0000 7f0a59baf640 1 -- 172.21.15.29:0/2089500253 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a513e69210 con 0x55a5141fdc30 2024-03-16T11:52:16.918 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:16.911+0000 7f0a593ae640 1 --2- 172.21.15.29:0/2089500253 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a5141efc50 0x55a5141fac30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:52:16.918 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:16.911+0000 7f0a59baf640 1 --2- 172.21.15.29:0/2089500253 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a5141fdc30 0x55a5141fb6b0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f0a4c005c00 tx=0x7f0a4c004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6eea39cc6b4a14a1 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:52:16.918 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:16.911+0000 7f0a3cff9640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-16T11:52:16.918 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:16.911+0000 7f0a3cff9640 1 -- 172.21.15.29:0/2089500253 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a4c026070 con 0x55a5141fdc30 2024-03-16T11:52:16.918 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:16.911+0000 7f0a3cff9640 1 -- 172.21.15.29:0/2089500253 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8469+0+0 (secure 0 0 0) 0x7f0a4c01fcd0 con 0x55a5141fdc30 2024-03-16T11:52:16.918 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:16.911+0000 7f0a5e200740 1 -- 172.21.15.29:0/2089500253 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0a44019b10 con 0x55a5141fdc30 2024-03-16T11:52:16.918 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:16.912+0000 7f0a3cff9640 1 -- 172.21.15.29:0/2089500253 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f0a4c00e930 con 0x55a5141fdc30 2024-03-16T11:52:16.918 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:16.912+0000 7f0a5e200740 10 client.15516 fetch_fsmap learned FSMap version 22 2024-03-16T11:52:16.918 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:16.912+0000 7f0a5e200740 1 -- 172.21.15.29:0/2089500253 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x55a513f35620 con 0x55a5141fdc30 2024-03-16T11:52:16.918 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:16.912+0000 7f0a3cff9640 1 -- 172.21.15.29:0/2089500253 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0a44019b10 con 0x55a5141fdc30 2024-03-16T11:52:16.918 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:16.912+0000 7f0a3cff9640 1 -- 172.21.15.29:0/2089500253 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7f0a4c01d040 con 0x55a5141fdc30 2024-03-16T11:52:16.918 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:16.912+0000 7f0a5e200740 10 client.15516 fetch_fsmap finished waiting for FSMap version 22 2024-03-16T11:52:16.918 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:16.912+0000 7f0a5e200740 10 client.15516 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T11:52:16.919 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:16.912+0000 7f0a5e200740 20 client.15516 populate_metadata read hostname 'smithi029' 2024-03-16T11:52:16.919 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:16.912+0000 7f0a5e200740 1 --2- 172.21.15.29:0/2089500253 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55a51420ddf0 0x55a514210200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:52:16.919 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:16.912+0000 7f0a5e200740 4 client.15516 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T11:52:16.919 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:16.912+0000 7f0a5e200740 1 -- 172.21.15.29:0/2089500253 --> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a514161f80 con 0x55a51420ddf0 2024-03-16T11:52:16.919 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:16.912+0000 7f0a593ae640 1 --2- 172.21.15.29:0/2089500253 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55a51420ddf0 0x55a514210200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:52:16.919 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:16.913+0000 7f0a593ae640 1 --2- 172.21.15.29:0/2089500253 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55a51420ddf0 0x55a514210200 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c2a8a704923a281 server_cookie=38fb0b87ef78ab0f in_seq=0 out_seq=0 2024-03-16T11:52:16.919 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:16.913+0000 7f0a3cff9640 10 client.15516 ms_handle_connect on v2:172.21.15.152:6838/1548731833 2024-03-16T11:52:16.919 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:16.914+0000 7f0a3cff9640 1 -- 172.21.15.29:0/2089500253 <== mds.0 v2:172.21.15.152:6838/1548731833 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55a514161f80 con 0x55a51420ddf0 2024-03-16T11:52:16.919 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:16.914+0000 7f0a3cff9640 10 client.15516 handle_command_reply: tid=0 2024-03-16T11:52:16.919 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:16.914+0000 7f0a5e200740 1 client.15516 shutdown 2024-03-16T11:52:16.919 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:16.914+0000 7f0a5e200740 1 -- 172.21.15.29:0/2089500253 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a5141fdc30 msgr2=0x55a5141fb6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:52:16.919 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:16.914+0000 7f0a5e200740 1 --2- 172.21.15.29:0/2089500253 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a5141fdc30 0x55a5141fb6b0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f0a4c005c00 tx=0x7f0a4c004ab0 comp rx=0 tx=0).stop 2024-03-16T11:52:16.919 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:16.914+0000 7f0a5e200740 1 -- 172.21.15.29:0/2089500253 shutdown_connections 2024-03-16T11:52:16.919 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:16.914+0000 7f0a5e200740 1 --2- 172.21.15.29:0/2089500253 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a5141fdc30 0x55a5141fb6b0 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:52:16.920 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:16.914+0000 7f0a5e200740 1 --2- 172.21.15.29:0/2089500253 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55a51420ddf0 0x55a514210200 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:52:16.920 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:16.914+0000 7f0a5e200740 1 --2- 172.21.15.29:0/2089500253 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a5141f0570 0x55a5141fb170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:52:16.920 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:16.914+0000 7f0a5e200740 1 --2- 172.21.15.29:0/2089500253 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a5141efc50 0x55a5141fac30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:52:16.920 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:16.914+0000 7f0a3cff9640 0 client.15516 ms_handle_reset on v2:172.21.15.152:6838/1548731833 2024-03-16T11:52:16.920 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:16.914+0000 7f0a5e200740 1 -- 172.21.15.29:0/2089500253 >> 172.21.15.29:0/2089500253 conn(0x55a5141eaa50 msgr2=0x55a5141ec7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:52:16.920 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:16.914+0000 7f0a5e200740 1 -- 172.21.15.29:0/2089500253 shutdown_connections 2024-03-16T11:52:16.920 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:16.914+0000 7f0a5e200740 1 -- 172.21.15.29:0/2089500253 wait complete. 2024-03-16T11:52:16.920 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:16.914+0000 7f0a5e200740 20 client.15516 trim_cache size 0 max 16384 2024-03-16T11:52:16.933 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:16.929+0000 7f0a5e200740 1 Processor -- start 2024-03-16T11:52:16.933 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:16.929+0000 7f0a5e200740 1 -- start start 2024-03-16T11:52:16.933 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:16.930+0000 7f0a5e200740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a5141efc50 0x55a5141fd360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:52:16.933 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:16.930+0000 7f0a5e200740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a5141f0570 0x55a5142095d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:52:16.933 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:16.930+0000 7f0a5e200740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a5141fdc30 0x55a5141ec5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:52:16.934 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:16.930+0000 7f0a5e200740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f0a541ad9d0 con 0x55a5141f0570 2024-03-16T11:52:16.934 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:16.930+0000 7f0a5e200740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f0a541ab2a0 con 0x55a5141fdc30 2024-03-16T11:52:16.934 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:16.930+0000 7f0a5e200740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f0a541a90f0 con 0x55a5141efc50 2024-03-16T11:52:16.934 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:16.930+0000 7f0a58bad640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a5141f0570 0x55a5142095d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:52:16.934 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:16.930+0000 7f0a58bad640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a5141f0570 0x55a5142095d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49676/0 (socket says 172.21.15.29:49676) 2024-03-16T11:52:16.934 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:16.930+0000 7f0a58bad640 1 -- 172.21.15.29:0/775320012 learned_addr learned my addr 172.21.15.29:0/775320012 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:52:16.934 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:16.930+0000 7f0a593ae640 1 --2- 172.21.15.29:0/775320012 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a5141efc50 0x55a5141fd360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:52:16.934 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:16.930+0000 7f0a59baf640 1 --2- 172.21.15.29:0/775320012 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a5141fdc30 0x55a5141ec5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:52:16.934 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:16.930+0000 7f0a58bad640 1 -- 172.21.15.29:0/775320012 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a5141efc50 msgr2=0x55a5141fd360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:52:16.934 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:16.930+0000 7f0a58bad640 1 --2- 172.21.15.29:0/775320012 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a5141efc50 0x55a5141fd360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:52:16.934 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:16.930+0000 7f0a58bad640 1 -- 172.21.15.29:0/775320012 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a5141fdc30 msgr2=0x55a5141ec5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:52:16.934 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:16.930+0000 7f0a58bad640 1 --2- 172.21.15.29:0/775320012 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a5141fdc30 0x55a5141ec5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:52:16.934 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:16.930+0000 7f0a58bad640 1 -- 172.21.15.29:0/775320012 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a5141ea880 con 0x55a5141f0570 2024-03-16T11:52:16.934 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:16.930+0000 7f0a59baf640 1 --2- 172.21.15.29:0/775320012 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a5141fdc30 0x55a5141ec5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T11:52:16.935 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:16.930+0000 7f0a58bad640 1 --2- 172.21.15.29:0/775320012 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a5141f0570 0x55a5142095d0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f0a44009bc0 tx=0x7f0a4400e5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7136543386cc6dc2 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:52:16.935 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:16.930+0000 7f0a3effd640 1 -- 172.21.15.29:0/775320012 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a44028030 con 0x55a5141f0570 2024-03-16T11:52:16.935 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:16.930+0000 7f0a3effd640 1 -- 172.21.15.29:0/775320012 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0a44005520 con 0x55a5141f0570 2024-03-16T11:52:16.935 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:16.930+0000 7f0a3effd640 1 -- 172.21.15.29:0/775320012 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a4401e040 con 0x55a5141f0570 2024-03-16T11:52:16.935 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:16.931+0000 7f0a5e200740 1 -- 172.21.15.29:0/775320012 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a5141f0570 msgr2=0x55a5142095d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:52:16.935 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:16.931+0000 7f0a5e200740 1 --2- 172.21.15.29:0/775320012 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a5141f0570 0x55a5142095d0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f0a44009bc0 tx=0x7f0a4400e5d0 comp rx=0 tx=0).stop 2024-03-16T11:52:16.935 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:16.931+0000 7f0a5e200740 1 -- 172.21.15.29:0/775320012 shutdown_connections 2024-03-16T11:52:16.935 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:16.931+0000 7f0a5e200740 1 --2- 172.21.15.29:0/775320012 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a5141fdc30 0x55a5141ec5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:52:16.935 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:16.931+0000 7f0a5e200740 1 --2- 172.21.15.29:0/775320012 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a5141f0570 0x55a5142095d0 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:52:16.935 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:16.931+0000 7f0a5e200740 1 --2- 172.21.15.29:0/775320012 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a5141efc50 0x55a5141fd360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:52:16.935 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:16.931+0000 7f0a5e200740 1 -- 172.21.15.29:0/775320012 >> 172.21.15.29:0/775320012 conn(0x55a5141eaa50 msgr2=0x55a5141ebea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:52:16.935 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:16.931+0000 7f0a5e200740 1 -- 172.21.15.29:0/775320012 shutdown_connections 2024-03-16T11:52:16.935 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:16.931+0000 7f0a5e200740 1 -- 172.21.15.29:0/775320012 wait complete. 2024-03-16T11:52:16.935 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:16.931+0000 7f0a5e200740 1 -- start start 2024-03-16T11:52:16.935 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:16.931+0000 7f0a5e200740 1 Processor -- start 2024-03-16T11:52:16.936 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:16.932+0000 7f0a5e200740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a5141efc50 0x55a514201b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:52:16.936 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:16.932+0000 7f0a5e200740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a5141f0570 0x55a514202080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:52:16.936 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:16.932+0000 7f0a5e200740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a5141fdc30 0x55a514273190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:52:16.936 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:16.932+0000 7f0a5e200740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f0a541a9760 con 0x55a5141fdc30 2024-03-16T11:52:16.936 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:16.932+0000 7f0a5e200740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f0a541aabd0 con 0x55a5141efc50 2024-03-16T11:52:16.936 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:16.932+0000 7f0a5e200740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f0a541a9e30 con 0x55a5141f0570 2024-03-16T11:52:16.936 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:16.932+0000 7f0a5e200740 10 client.0 _ll_register_callbacks cb 0x7f0a4c009d80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T11:52:16.936 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:16.932+0000 7f0a59baf640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a5141fdc30 0x55a514273190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:52:16.936 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:16.932+0000 7f0a59baf640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a5141fdc30 0x55a514273190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49682/0 (socket says 172.21.15.29:49682) 2024-03-16T11:52:16.936 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:16.932+0000 7f0a59baf640 1 -- 172.21.15.29:0/4113427208 learned_addr learned my addr 172.21.15.29:0/4113427208 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:52:16.936 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:16.932+0000 7f0a593ae640 1 --2- 172.21.15.29:0/4113427208 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a5141efc50 0x55a514201b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:52:16.936 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:16.932+0000 7f0a58bad640 1 --2- 172.21.15.29:0/4113427208 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a5141f0570 0x55a514202080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:52:16.936 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:16.932+0000 7f0a59baf640 1 -- 172.21.15.29:0/4113427208 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a5141f0570 msgr2=0x55a514202080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:52:16.936 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:16.932+0000 7f0a59baf640 1 --2- 172.21.15.29:0/4113427208 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a5141f0570 0x55a514202080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:52:16.937 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:16.932+0000 7f0a59baf640 1 -- 172.21.15.29:0/4113427208 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a5141efc50 msgr2=0x55a514201b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:52:16.937 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:16.932+0000 7f0a59baf640 1 --2- 172.21.15.29:0/4113427208 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a5141efc50 0x55a514201b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:52:16.937 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:16.932+0000 7f0a59baf640 1 -- 172.21.15.29:0/4113427208 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a514200020 con 0x55a5141fdc30 2024-03-16T11:52:16.937 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:16.932+0000 7f0a59baf640 1 --2- 172.21.15.29:0/4113427208 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a5141fdc30 0x55a514273190 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f0a4c000f80 tx=0x7f0a4c01ff60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=335a9ceab67e54e8 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:52:16.937 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:16.932+0000 7f0a3cff9640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-16T11:52:16.937 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:16.932+0000 7f0a3cff9640 1 -- 172.21.15.29:0/4113427208 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a4c026040 con 0x55a5141fdc30 2024-03-16T11:52:16.937 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:16.933+0000 7f0a3cff9640 1 -- 172.21.15.29:0/4113427208 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8469+0+0 (secure 0 0 0) 0x7f0a4c004b20 con 0x55a5141fdc30 2024-03-16T11:52:16.937 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:16.933+0000 7f0a5e200740 1 -- 172.21.15.29:0/4113427208 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0a44005520 con 0x55a5141fdc30 2024-03-16T11:52:16.937 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:16.933+0000 7f0a3cff9640 1 -- 172.21.15.29:0/4113427208 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f0a4c00e780 con 0x55a5141fdc30 2024-03-16T11:52:16.937 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:16.933+0000 7f0a5e200740 10 client.15522 fetch_fsmap learned FSMap version 22 2024-03-16T11:52:16.937 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:16.933+0000 7f0a5e200740 1 -- 172.21.15.29:0/4113427208 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x55a514274710 con 0x55a5141fdc30 2024-03-16T11:52:16.937 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:16.933+0000 7f0a3cff9640 1 -- 172.21.15.29:0/4113427208 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0a44005520 con 0x55a5141fdc30 2024-03-16T11:52:16.937 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:16.933+0000 7f0a3cff9640 1 -- 172.21.15.29:0/4113427208 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7f0a4c03c030 con 0x55a5141fdc30 2024-03-16T11:52:16.938 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:16.933+0000 7f0a5e200740 10 client.15522 fetch_fsmap finished waiting for FSMap version 22 2024-03-16T11:52:16.938 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:16.933+0000 7f0a5e200740 10 client.15522 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T11:52:16.938 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:16.933+0000 7f0a5e200740 20 client.15522 populate_metadata read hostname 'smithi029' 2024-03-16T11:52:16.938 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:16.933+0000 7f0a5e200740 1 --2- 172.21.15.29:0/4113427208 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55a514274e50 0x55a514277270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:52:16.938 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:16.933+0000 7f0a5e200740 4 client.15522 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-16T11:52:16.938 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:16.933+0000 7f0a5e200740 1 -- 172.21.15.29:0/4113427208 --> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55a51415c2d0 con 0x55a514274e50 2024-03-16T11:52:16.938 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:16.933+0000 7f0a593ae640 1 --2- 172.21.15.29:0/4113427208 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55a514274e50 0x55a514277270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:52:16.938 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:16.934+0000 7f0a3cff9640 10 client.15522 ms_handle_connect on v2:172.21.15.152:6838/1548731833 2024-03-16T11:52:16.938 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:16.934+0000 7f0a593ae640 1 --2- 172.21.15.29:0/4113427208 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55a514274e50 0x55a514277270 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2485f8d19d0dc9d2 server_cookie=289dfa4494bd5a9 in_seq=0 out_seq=0 2024-03-16T11:52:16.996 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:16.996+0000 7f0a3cff9640 1 -- 172.21.15.29:0/4113427208 <== mds.0 v2:172.21.15.152:6838/1548731833 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55a51415c2d0 con 0x55a514274e50 2024-03-16T11:52:16.996 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:16.996+0000 7f0a3cff9640 10 client.15522 handle_command_reply: tid=0 2024-03-16T11:52:16.996 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:16.996+0000 7f0a5e200740 1 client.15522 shutdown 2024-03-16T11:52:16.996 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:16.996+0000 7f0a5e200740 1 -- 172.21.15.29:0/4113427208 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a5141fdc30 msgr2=0x55a514273190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:52:16.996 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:16.996+0000 7f0a5e200740 1 --2- 172.21.15.29:0/4113427208 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a5141fdc30 0x55a514273190 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f0a4c000f80 tx=0x7f0a4c01ff60 comp rx=0 tx=0).stop 2024-03-16T11:52:16.996 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:16.996+0000 7f0a5e200740 1 -- 172.21.15.29:0/4113427208 shutdown_connections 2024-03-16T11:52:16.996 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:16.996+0000 7f0a5e200740 1 --2- 172.21.15.29:0/4113427208 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a5141fdc30 0x55a514273190 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:52:16.996 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:16.996+0000 7f0a5e200740 1 --2- 172.21.15.29:0/4113427208 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a5141f0570 0x55a514202080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:52:16.996 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:16.996+0000 7f0a5e200740 1 --2- 172.21.15.29:0/4113427208 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55a514274e50 0x55a514277270 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:52:16.996 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:16.996+0000 7f0a5e200740 1 --2- 172.21.15.29:0/4113427208 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a5141efc50 0x55a514201b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:52:16.996 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:16.996+0000 7f0a3cff9640 0 client.15522 ms_handle_reset on v2:172.21.15.152:6838/1548731833 2024-03-16T11:52:16.996 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:16.996+0000 7f0a5e200740 1 -- 172.21.15.29:0/4113427208 >> 172.21.15.29:0/4113427208 conn(0x55a5141eaa50 msgr2=0x55a5141e9bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:52:16.997 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:16.996+0000 7f0a5e200740 1 -- 172.21.15.29:0/4113427208 shutdown_connections 2024-03-16T11:52:16.997 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:16.996+0000 7f0a5e200740 1 -- 172.21.15.29:0/4113427208 wait complete. 2024-03-16T11:52:16.997 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:16.996+0000 7f0a5e200740 20 client.15522 trim_cache size 0 max 16384 2024-03-16T11:52:16.997 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:{ 2024-03-16T11:52:16.997 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "message": "", 2024-03-16T11:52:16.997 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "return_code": 0 2024-03-16T11:52:16.997 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:} 2024-03-16T11:52:16.998 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:16.997+0000 7f0a5b612640 1 -- 172.21.15.29:0/734178501 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f0a400a2b30 msgr2=0x7f0a400a4ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:52:16.998 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:16.997+0000 7f0a5b612640 1 --2- 172.21.15.29:0/734178501 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f0a400a2b30 0x7f0a400a4ff0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f0a4c005fd0 tx=0x7f0a4c009bd0 comp rx=0 tx=0).stop 2024-03-16T11:52:16.998 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:16.997+0000 7f0a5b612640 1 -- 172.21.15.29:0/734178501 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0a54100930 msgr2=0x7f0a54198070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:52:16.998 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:16.997+0000 7f0a5b612640 1 --2- 172.21.15.29:0/734178501 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0a54100930 0x7f0a54198070 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f0a50009230 tx=0x7f0a50004270 comp rx=0 tx=0).stop 2024-03-16T11:52:16.998 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:16.997+0000 7f0a5b612640 1 -- 172.21.15.29:0/734178501 shutdown_connections 2024-03-16T11:52:16.998 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:16.997+0000 7f0a5b612640 1 --2- 172.21.15.29:0/734178501 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f0a400a2b30 0x7f0a400a4ff0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:52:16.998 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:16.997+0000 7f0a5b612640 1 --2- 172.21.15.29:0/734178501 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0a54105680 0x7f0a5419c930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:52:16.998 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:16.997+0000 7f0a5b612640 1 --2- 172.21.15.29:0/734178501 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0a541012f0 0x7f0a541985b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:52:16.999 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:16.997+0000 7f0a5b612640 1 --2- 172.21.15.29:0/734178501 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0a54100930 0x7f0a54198070 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:52:16.999 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:16.997+0000 7f0a5b612640 1 -- 172.21.15.29:0/734178501 >> 172.21.15.29:0/734178501 conn(0x7f0a54077550 msgr2=0x7f0a5406a240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:52:16.999 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:16.997+0000 7f0a5b612640 1 -- 172.21.15.29:0/734178501 shutdown_connections 2024-03-16T11:52:16.999 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:16.997+0000 7f0a5b612640 1 -- 172.21.15.29:0/734178501 wait complete. 2024-03-16T11:52:17.773 INFO:tasks.workunit.client.0.smithi029.stdout: 1 4884 21.62 MB/sec warmup 2 sec latency 28.269 ms 2024-03-16T11:52:18.774 INFO:tasks.workunit.client.0.smithi029.stdout: 1 7562 19.73 MB/sec warmup 3 sec latency 11.936 ms 2024-03-16T11:52:18.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:52:18 smithi154 ceph-mon[36320]: pgmap v44: 97 pgs: 97 active+clean; 1.4 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 6.9 KiB/s wr, 9 op/s 2024-03-16T11:52:18.837 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:52:18 smithi029 ceph-mon[31975]: pgmap v44: 97 pgs: 97 active+clean; 1.4 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 6.9 KiB/s wr, 9 op/s 2024-03-16T11:52:18.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:52:18 smithi152 ceph-mon[36291]: pgmap v44: 97 pgs: 97 active+clean; 1.4 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 6.9 KiB/s wr, 9 op/s 2024-03-16T11:52:19.774 INFO:tasks.workunit.client.0.smithi029.stdout: 1 10465 18.42 MB/sec warmup 4 sec latency 16.321 ms 2024-03-16T11:52:20.774 INFO:tasks.workunit.client.0.smithi029.stdout: 1 12608 16.76 MB/sec warmup 5 sec latency 19.732 ms 2024-03-16T11:52:20.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:52:20 smithi154 ceph-mon[36320]: pgmap v45: 97 pgs: 97 active+clean; 1.4 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 6.2 KiB/s wr, 6 op/s 2024-03-16T11:52:20.837 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:52:20 smithi029 ceph-mon[31975]: pgmap v45: 97 pgs: 97 active+clean; 1.4 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 6.2 KiB/s wr, 6 op/s 2024-03-16T11:52:20.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:52:20 smithi152 ceph-mon[36291]: pgmap v45: 97 pgs: 97 active+clean; 1.4 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 6.2 KiB/s wr, 6 op/s 2024-03-16T11:52:21.774 INFO:tasks.workunit.client.0.smithi029.stdout: 1 14928 16.30 MB/sec warmup 6 sec latency 22.453 ms 2024-03-16T11:52:22.774 INFO:tasks.workunit.client.0.smithi029.stdout: 1 17962 16.61 MB/sec warmup 7 sec latency 19.091 ms 2024-03-16T11:52:22.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:52:22 smithi154 ceph-mon[36320]: pgmap v46: 97 pgs: 97 active+clean; 8.0 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 253 KiB/s rd, 667 KiB/s wr, 113 op/s 2024-03-16T11:52:22.837 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:52:22 smithi029 ceph-mon[31975]: pgmap v46: 97 pgs: 97 active+clean; 8.0 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 253 KiB/s rd, 667 KiB/s wr, 113 op/s 2024-03-16T11:52:22.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:52:22 smithi152 ceph-mon[36291]: pgmap v46: 97 pgs: 97 active+clean; 8.0 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 253 KiB/s rd, 667 KiB/s wr, 113 op/s 2024-03-16T11:52:23.774 INFO:tasks.workunit.client.0.smithi029.stdout: 1 20276 15.55 MB/sec warmup 8 sec latency 27.103 ms 2024-03-16T11:52:24.774 INFO:tasks.workunit.client.0.smithi029.stdout: 1 22298 15.18 MB/sec warmup 9 sec latency 19.544 ms 2024-03-16T11:52:24.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:52:24 smithi154 ceph-mon[36320]: pgmap v47: 97 pgs: 97 active+clean; 8.0 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 252 KiB/s rd, 667 KiB/s wr, 110 op/s 2024-03-16T11:52:24.837 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:52:24 smithi029 ceph-mon[31975]: pgmap v47: 97 pgs: 97 active+clean; 8.0 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 252 KiB/s rd, 667 KiB/s wr, 110 op/s 2024-03-16T11:52:24.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:52:24 smithi152 ceph-mon[36291]: pgmap v47: 97 pgs: 97 active+clean; 8.0 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 252 KiB/s rd, 667 KiB/s wr, 110 op/s 2024-03-16T11:52:25.774 INFO:tasks.workunit.client.0.smithi029.stdout: 1 25193 15.45 MB/sec warmup 10 sec latency 7.807 ms 2024-03-16T11:52:26.774 INFO:tasks.workunit.client.0.smithi029.stdout: 1 27844 15.22 MB/sec warmup 11 sec latency 18.673 ms 2024-03-16T11:52:26.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:52:26 smithi154 ceph-mon[36320]: pgmap v48: 97 pgs: 97 active+clean; 8.0 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 252 KiB/s rd, 667 KiB/s wr, 110 op/s 2024-03-16T11:52:26.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:52:26 smithi154 ceph-mon[36320]: mgrmap e25: x(active, since 90s), standbys: y, z 2024-03-16T11:52:26.837 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:52:26 smithi029 ceph-mon[31975]: pgmap v48: 97 pgs: 97 active+clean; 8.0 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 252 KiB/s rd, 667 KiB/s wr, 110 op/s 2024-03-16T11:52:26.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:52:26 smithi029 ceph-mon[31975]: mgrmap e25: x(active, since 90s), standbys: y, z 2024-03-16T11:52:26.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:52:26 smithi152 ceph-mon[36291]: pgmap v48: 97 pgs: 97 active+clean; 8.0 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 252 KiB/s rd, 667 KiB/s wr, 110 op/s 2024-03-16T11:52:26.878 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:52:26 smithi152 ceph-mon[36291]: mgrmap e25: x(active, since 90s), standbys: y, z 2024-03-16T11:52:27.774 INFO:tasks.workunit.client.0.smithi029.stdout: 1 29923 14.92 MB/sec warmup 12 sec latency 21.356 ms 2024-03-16T11:52:28.774 INFO:tasks.workunit.client.0.smithi029.stdout: 1 32584 14.85 MB/sec warmup 13 sec latency 13.824 ms 2024-03-16T11:52:28.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:52:28 smithi154 ceph-mon[36320]: pgmap v49: 97 pgs: 97 active+clean; 13 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 587 KiB/s rd, 1.2 MiB/s wr, 185 op/s 2024-03-16T11:52:28.837 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:52:28 smithi029 ceph-mon[31975]: pgmap v49: 97 pgs: 97 active+clean; 13 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 587 KiB/s rd, 1.2 MiB/s wr, 185 op/s 2024-03-16T11:52:28.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:52:28 smithi152 ceph-mon[36291]: pgmap v49: 97 pgs: 97 active+clean; 13 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 587 KiB/s rd, 1.2 MiB/s wr, 185 op/s 2024-03-16T11:52:29.774 INFO:tasks.workunit.client.0.smithi029.stdout: 1 35672 15.07 MB/sec warmup 14 sec latency 14.750 ms 2024-03-16T11:52:30.088 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:30.086+0000 7f960864b640 1 -- 172.21.15.29:0/823069475 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f960010be30 msgr2=0x7f960010e2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:52:30.088 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:30.086+0000 7f960864b640 1 --2- 172.21.15.29:0/823069475 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f960010be30 0x7f960010e2a0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f95f8009cc0 tx=0x7f95f8030510 comp rx=0 tx=0).stop 2024-03-16T11:52:30.088 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:30.086+0000 7f960864b640 1 -- 172.21.15.29:0/823069475 shutdown_connections 2024-03-16T11:52:30.088 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:30.086+0000 7f960864b640 1 --2- 172.21.15.29:0/823069475 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f960010be30 0x7f960010e2a0 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:52:30.088 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:30.086+0000 7f960864b640 1 --2- 172.21.15.29:0/823069475 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f96001032e0 0x7f960010b7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:52:30.088 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:30.086+0000 7f960864b640 1 --2- 172.21.15.29:0/823069475 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f96001029c0 0x7f9600102da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:52:30.088 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:30.086+0000 7f960864b640 1 -- 172.21.15.29:0/823069475 >> 172.21.15.29:0/823069475 conn(0x7f9600077810 msgr2=0x7f9600077c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:52:30.088 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:30.086+0000 7f960864b640 1 -- 172.21.15.29:0/823069475 shutdown_connections 2024-03-16T11:52:30.088 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:30.086+0000 7f960864b640 1 -- 172.21.15.29:0/823069475 wait complete. 2024-03-16T11:52:30.088 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:30.087+0000 7f960864b640 1 Processor -- start 2024-03-16T11:52:30.089 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:30.087+0000 7f960864b640 1 -- start start 2024-03-16T11:52:30.089 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:30.087+0000 7f960864b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f96001029c0 0x7f9600195d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:52:30.089 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:30.087+0000 7f960864b640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f96001032e0 0x7f96001962d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:52:30.089 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:30.087+0000 7f960864b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f960010be30 0x7f960019a600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:52:30.089 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:30.087+0000 7f960864b640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f960010f9f0 con 0x7f960010be30 2024-03-16T11:52:30.089 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:30.087+0000 7f960864b640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f960010f870 con 0x7f96001032e0 2024-03-16T11:52:30.089 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:30.087+0000 7f960864b640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f960010fb70 con 0x7f96001029c0 2024-03-16T11:52:30.089 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:30.087+0000 7f9606be8640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f960010be30 0x7f960019a600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:52:30.089 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:30.087+0000 7f9606be8640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f960010be30 0x7f960019a600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39270/0 (socket says 172.21.15.29:39270) 2024-03-16T11:52:30.089 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:30.087+0000 7f9606be8640 1 -- 172.21.15.29:0/3295556984 learned_addr learned my addr 172.21.15.29:0/3295556984 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:52:30.089 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:30.087+0000 7f96063e7640 1 --2- 172.21.15.29:0/3295556984 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f96001029c0 0x7f9600195d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:52:30.089 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:30.087+0000 7f9605be6640 1 --2- 172.21.15.29:0/3295556984 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f96001032e0 0x7f96001962d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:52:30.089 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:30.088+0000 7f9606be8640 1 -- 172.21.15.29:0/3295556984 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f96001029c0 msgr2=0x7f9600195d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:52:30.089 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:30.088+0000 7f9606be8640 1 --2- 172.21.15.29:0/3295556984 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f96001029c0 0x7f9600195d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:52:30.089 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:30.088+0000 7f9606be8640 1 -- 172.21.15.29:0/3295556984 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f96001032e0 msgr2=0x7f96001962d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:52:30.090 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:30.088+0000 7f9606be8640 1 --2- 172.21.15.29:0/3295556984 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f96001032e0 0x7f96001962d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:52:30.090 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:30.088+0000 7f9606be8640 1 -- 172.21.15.29:0/3295556984 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f960019ad30 con 0x7f960010be30 2024-03-16T11:52:30.090 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:30.088+0000 7f9606be8640 1 --2- 172.21.15.29:0/3295556984 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f960010be30 0x7f960019a600 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f95f8006010 tx=0x7f95f8003f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:52:30.090 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:30.088+0000 7f95f77fe640 1 -- 172.21.15.29:0/3295556984 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95f8046070 con 0x7f960010be30 2024-03-16T11:52:30.090 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:30.088+0000 7f960864b640 1 -- 172.21.15.29:0/3295556984 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f960019afc0 con 0x7f960010be30 2024-03-16T11:52:30.090 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:30.088+0000 7f95f77fe640 1 -- 172.21.15.29:0/3295556984 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f95f80359f0 con 0x7f960010be30 2024-03-16T11:52:30.090 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:30.088+0000 7f95f77fe640 1 -- 172.21.15.29:0/3295556984 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95f80045e0 con 0x7f960010be30 2024-03-16T11:52:30.090 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:30.088+0000 7f960864b640 1 -- 172.21.15.29:0/3295556984 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f96001a28b0 con 0x7f960010be30 2024-03-16T11:52:30.096 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:30.090+0000 7f95f77fe640 1 -- 172.21.15.29:0/3295556984 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7f95f803f040 con 0x7f960010be30 2024-03-16T11:52:30.096 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:30.090+0000 7f960b237740 1 Processor -- start 2024-03-16T11:52:30.096 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:30.091+0000 7f960b237740 1 -- start start 2024-03-16T11:52:30.096 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:30.091+0000 7f960b237740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563781cc1130 0x563781cc1510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:52:30.097 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:30.091+0000 7f960b237740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563781cc1a50 0x563781cca2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:52:30.097 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:30.091+0000 7f95f77fe640 1 --2- 172.21.15.29:0/3295556984 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f95ec0a2a10 0x7f95ec0a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:52:30.097 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:30.091+0000 7f960b237740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563781cca820 0x563781cccc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:52:30.097 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:30.091+0000 7f960b237740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x563781c25a40 con 0x563781cc1a50 2024-03-16T11:52:30.097 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:30.091+0000 7f960b237740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x563781bce920 con 0x563781cca820 2024-03-16T11:52:30.097 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:30.091+0000 7f960b237740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x563781c21ae0 con 0x563781cc1130 2024-03-16T11:52:30.097 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:30.091+0000 7f96063e7640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563781cc1a50 0x563781cca2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:52:30.097 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:30.091+0000 7f96063e7640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563781cc1a50 0x563781cca2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39280/0 (socket says 172.21.15.29:39280) 2024-03-16T11:52:30.097 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:30.091+0000 7f96063e7640 1 -- 172.21.15.29:0/2148656420 learned_addr learned my addr 172.21.15.29:0/2148656420 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:52:30.097 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:30.091+0000 7f9606be8640 1 --2- 172.21.15.29:0/2148656420 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563781cca820 0x563781cccc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:52:30.097 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:30.091+0000 7f9605be6640 1 --2- 172.21.15.29:0/2148656420 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563781cc1130 0x563781cc1510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:52:30.097 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:30.091+0000 7f9605be6640 1 --2- 172.21.15.29:0/3295556984 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f95ec0a2a10 0x7f95ec0a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:52:30.097 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:30.091+0000 7f95f77fe640 1 -- 172.21.15.29:0/3295556984 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8469+0+0 (secure 0 0 0) 0x7f95f80f6030 con 0x7f960010be30 2024-03-16T11:52:30.097 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:30.091+0000 7f96063e7640 1 -- 172.21.15.29:0/2148656420 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563781cc1130 msgr2=0x563781cc1510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:52:30.097 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:30.091+0000 7f96063e7640 1 --2- 172.21.15.29:0/2148656420 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563781cc1130 0x563781cc1510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:52:30.098 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:30.091+0000 7f96063e7640 1 -- 172.21.15.29:0/2148656420 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563781cca820 msgr2=0x563781cccc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:52:30.098 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:30.091+0000 7f96063e7640 1 --2- 172.21.15.29:0/2148656420 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563781cca820 0x563781cccc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:52:30.098 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:30.091+0000 7f96063e7640 1 -- 172.21.15.29:0/2148656420 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563781be6460 con 0x563781cc1a50 2024-03-16T11:52:30.098 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:30.091+0000 7f9606be8640 1 --2- 172.21.15.29:0/2148656420 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563781cca820 0x563781cccc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:52:30.098 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:30.092+0000 7f96063e7640 1 --2- 172.21.15.29:0/2148656420 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563781cc1a50 0x563781cca2e0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f95fc00bd50 tx=0x7f95fc0107a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74b8b57f42467554 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:52:30.098 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:30.092+0000 7f9605be6640 1 --2- 172.21.15.29:0/3295556984 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f95ec0a2a10 0x7f95ec0a4ed0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f95f000b6c0 tx=0x7f95f000b410 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:52:30.098 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:30.092+0000 7f95f4bfe640 1 -- 172.21.15.29:0/2148656420 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95fc005220 con 0x563781cc1a50 2024-03-16T11:52:30.098 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:30.092+0000 7f95f4bfe640 1 -- 172.21.15.29:0/2148656420 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f95fc00be40 con 0x563781cc1a50 2024-03-16T11:52:30.098 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:30.092+0000 7f95f4bfe640 1 -- 172.21.15.29:0/2148656420 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95fc011860 con 0x563781cc1a50 2024-03-16T11:52:30.098 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:30.092+0000 7f960b237740 1 -- 172.21.15.29:0/2148656420 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563781cc1a50 msgr2=0x563781cca2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:52:30.098 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:30.093+0000 7f960b237740 1 --2- 172.21.15.29:0/2148656420 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563781cc1a50 0x563781cca2e0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f95fc00bd50 tx=0x7f95fc0107a0 comp rx=0 tx=0).stop 2024-03-16T11:52:30.098 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:30.093+0000 7f960b237740 1 -- 172.21.15.29:0/2148656420 shutdown_connections 2024-03-16T11:52:30.098 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:30.093+0000 7f960b237740 1 --2- 172.21.15.29:0/2148656420 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563781cca820 0x563781cccc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:52:30.098 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:30.093+0000 7f960b237740 1 --2- 172.21.15.29:0/2148656420 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563781cc1a50 0x563781cca2e0 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:52:30.099 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:30.093+0000 7f960b237740 1 --2- 172.21.15.29:0/2148656420 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563781cc1130 0x563781cc1510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:52:30.099 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:30.093+0000 7f960b237740 1 -- 172.21.15.29:0/2148656420 >> 172.21.15.29:0/2148656420 conn(0x563781cbba50 msgr2=0x563781cbbe30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:52:30.099 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:30.093+0000 7f960b237740 1 -- 172.21.15.29:0/2148656420 shutdown_connections 2024-03-16T11:52:30.099 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:30.093+0000 7f960b237740 1 -- 172.21.15.29:0/2148656420 wait complete. 2024-03-16T11:52:30.099 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:30.093+0000 7f960b237740 1 -- start start 2024-03-16T11:52:30.099 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:30.093+0000 7f960b237740 1 Processor -- start 2024-03-16T11:52:30.099 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:30.093+0000 7f960b237740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563781cc1130 0x563781cc7890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:52:30.099 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:30.093+0000 7f960b237740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563781cc1a50 0x563781cc7dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:52:30.099 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:30.093+0000 7f960b237740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563781cca820 0x563781cc8310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:52:30.099 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:30.093+0000 7f960b237740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x563781ccd450 con 0x563781cc1a50 2024-03-16T11:52:30.099 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:30.093+0000 7f960b237740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x563781be1d70 con 0x563781cca820 2024-03-16T11:52:30.099 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:30.093+0000 7f960b237740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x563781ccd5d0 con 0x563781cc1130 2024-03-16T11:52:30.099 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:30.093+0000 7f960b237740 10 client.0 _ll_register_callbacks cb 0x563781939a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T11:52:30.099 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:30.093+0000 7f9605be6640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563781cc1a50 0x563781cc7dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:52:30.099 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:30.093+0000 7f9605be6640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563781cc1a50 0x563781cc7dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39284/0 (socket says 172.21.15.29:39284) 2024-03-16T11:52:30.100 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:30.093+0000 7f9605be6640 1 -- 172.21.15.29:0/2644173812 learned_addr learned my addr 172.21.15.29:0/2644173812 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:52:30.100 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:30.093+0000 7f96063e7640 1 --2- 172.21.15.29:0/2644173812 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563781cc1130 0x563781cc7890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:52:30.100 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:30.094+0000 7f9606be8640 1 --2- 172.21.15.29:0/2644173812 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563781cca820 0x563781cc8310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:52:30.100 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:30.094+0000 7f9605be6640 1 -- 172.21.15.29:0/2644173812 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563781cc1130 msgr2=0x563781cc7890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:52:30.100 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:30.094+0000 7f9605be6640 1 --2- 172.21.15.29:0/2644173812 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563781cc1130 0x563781cc7890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:52:30.100 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:30.094+0000 7f9605be6640 1 -- 172.21.15.29:0/2644173812 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563781cca820 msgr2=0x563781cc8310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:52:30.100 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:30.094+0000 7f9605be6640 1 --2- 172.21.15.29:0/2644173812 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563781cca820 0x563781cc8310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:52:30.100 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:30.094+0000 7f9605be6640 1 -- 172.21.15.29:0/2644173812 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56378193a210 con 0x563781cc1a50 2024-03-16T11:52:30.100 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:30.094+0000 7f9605be6640 1 --2- 172.21.15.29:0/2644173812 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563781cc1a50 0x563781cc7dd0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f95f000bae0 tx=0x7f95f0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa4d5d25f10b31de server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:52:30.100 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:30.094+0000 7f95e9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-16T11:52:30.100 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:30.094+0000 7f95e9ffb640 1 -- 172.21.15.29:0/2644173812 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95f0010960 con 0x563781cc1a50 2024-03-16T11:52:30.100 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:30.094+0000 7f95e9ffb640 1 -- 172.21.15.29:0/2644173812 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8469+0+0 (secure 0 0 0) 0x7f95f0010b00 con 0x563781cc1a50 2024-03-16T11:52:30.100 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:30.094+0000 7f960b237740 1 -- 172.21.15.29:0/2644173812 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x563781c32f80 con 0x563781cc1a50 2024-03-16T11:52:30.100 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:30.094+0000 7f95e9ffb640 1 -- 172.21.15.29:0/2644173812 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f95f0010d00 con 0x563781cc1a50 2024-03-16T11:52:30.101 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:30.094+0000 7f960b237740 10 client.15534 fetch_fsmap learned FSMap version 22 2024-03-16T11:52:30.101 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:30.094+0000 7f960b237740 1 -- 172.21.15.29:0/2644173812 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x563781a06620 con 0x563781cc1a50 2024-03-16T11:52:30.101 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:30.094+0000 7f95e9ffb640 1 -- 172.21.15.29:0/2644173812 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x563781c32f80 con 0x563781cc1a50 2024-03-16T11:52:30.101 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:30.094+0000 7f95e9ffb640 1 -- 172.21.15.29:0/2644173812 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7f95f0005470 con 0x563781cc1a50 2024-03-16T11:52:30.101 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:30.095+0000 7f960b237740 10 client.15534 fetch_fsmap finished waiting for FSMap version 22 2024-03-16T11:52:30.101 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:30.095+0000 7f960b237740 10 client.15534 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T11:52:30.101 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:30.095+0000 7f960b237740 20 client.15534 populate_metadata read hostname 'smithi029' 2024-03-16T11:52:30.101 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:30.095+0000 7f960b237740 1 --2- 172.21.15.29:0/2644173812 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x563781cdab10 0x563781cdcf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:52:30.101 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:30.095+0000 7f960b237740 4 client.15534 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T11:52:30.101 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:30.095+0000 7f960b237740 1 -- 172.21.15.29:0/2644173812 --> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563781c2d2d0 con 0x563781cdab10 2024-03-16T11:52:30.101 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:30.095+0000 7f96063e7640 1 --2- 172.21.15.29:0/2644173812 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x563781cdab10 0x563781cdcf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:52:30.101 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:30.095+0000 7f96063e7640 1 --2- 172.21.15.29:0/2644173812 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x563781cdab10 0x563781cdcf00 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4d79296b653951a4 server_cookie=99a3a4b226975c3d in_seq=0 out_seq=0 2024-03-16T11:52:30.101 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:30.095+0000 7f95e9ffb640 10 client.15534 ms_handle_connect on v2:172.21.15.152:6838/1548731833 2024-03-16T11:52:30.101 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:30.096+0000 7f95e9ffb640 1 -- 172.21.15.29:0/2644173812 <== mds.0 v2:172.21.15.152:6838/1548731833 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x563781c2d2d0 con 0x563781cdab10 2024-03-16T11:52:30.101 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:30.096+0000 7f95e9ffb640 10 client.15534 handle_command_reply: tid=0 2024-03-16T11:52:30.102 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:30.097+0000 7f960b237740 1 client.15534 shutdown 2024-03-16T11:52:30.102 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:30.097+0000 7f960b237740 1 -- 172.21.15.29:0/2644173812 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563781cc1a50 msgr2=0x563781cc7dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:52:30.102 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:30.097+0000 7f960b237740 1 --2- 172.21.15.29:0/2644173812 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563781cc1a50 0x563781cc7dd0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f95f000bae0 tx=0x7f95f0004730 comp rx=0 tx=0).stop 2024-03-16T11:52:30.102 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:30.097+0000 7f960b237740 1 -- 172.21.15.29:0/2644173812 shutdown_connections 2024-03-16T11:52:30.102 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:30.097+0000 7f960b237740 1 --2- 172.21.15.29:0/2644173812 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x563781cdab10 0x563781cdcf00 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:52:30.102 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:30.097+0000 7f960b237740 1 --2- 172.21.15.29:0/2644173812 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563781cca820 0x563781cc8310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:52:30.102 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:30.097+0000 7f95e9ffb640 0 client.15534 ms_handle_reset on v2:172.21.15.152:6838/1548731833 2024-03-16T11:52:30.102 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:30.097+0000 7f960b237740 1 --2- 172.21.15.29:0/2644173812 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563781cc1a50 0x563781cc7dd0 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:52:30.102 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:30.097+0000 7f960b237740 1 --2- 172.21.15.29:0/2644173812 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563781cc1130 0x563781cc7890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:52:30.102 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:30.097+0000 7f960b237740 1 -- 172.21.15.29:0/2644173812 >> 172.21.15.29:0/2644173812 conn(0x563781cbba50 msgr2=0x563781cbd7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:52:30.102 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:30.097+0000 7f960b237740 1 -- 172.21.15.29:0/2644173812 shutdown_connections 2024-03-16T11:52:30.102 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:30.097+0000 7f960b237740 1 -- 172.21.15.29:0/2644173812 wait complete. 2024-03-16T11:52:30.102 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:30.097+0000 7f960b237740 20 client.15534 trim_cache size 0 max 16384 2024-03-16T11:52:30.115 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:30.112+0000 7f960b237740 1 Processor -- start 2024-03-16T11:52:30.117 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:30.112+0000 7f960b237740 1 -- start start 2024-03-16T11:52:30.117 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:30.112+0000 7f960b237740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563781cc1130 0x563781cc9fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:52:30.117 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:30.112+0000 7f960b237740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563781cc1a50 0x563781cd61f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:52:30.117 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:30.112+0000 7f960b237740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563781cca820 0x563781ce27f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:52:30.117 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:30.112+0000 7f960b237740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f96001ad970 con 0x563781cc1130 2024-03-16T11:52:30.117 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:30.112+0000 7f960b237740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f96001ab240 con 0x563781cc1a50 2024-03-16T11:52:30.117 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:30.112+0000 7f960b237740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f96001a9090 con 0x563781cca820 2024-03-16T11:52:30.118 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:30.112+0000 7f96063e7640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563781cc1130 0x563781cc9fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:52:30.118 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:30.112+0000 7f96063e7640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563781cc1130 0x563781cc9fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39290/0 (socket says 172.21.15.29:39290) 2024-03-16T11:52:30.118 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:30.112+0000 7f96063e7640 1 -- 172.21.15.29:0/1909356173 learned_addr learned my addr 172.21.15.29:0/1909356173 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:52:30.118 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:30.112+0000 7f9605be6640 1 --2- 172.21.15.29:0/1909356173 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563781cc1a50 0x563781cd61f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:52:30.118 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:30.112+0000 7f9606be8640 1 --2- 172.21.15.29:0/1909356173 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563781cca820 0x563781ce27f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:52:30.118 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:30.112+0000 7f96063e7640 1 -- 172.21.15.29:0/1909356173 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563781cca820 msgr2=0x563781ce27f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:52:30.118 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:30.112+0000 7f96063e7640 1 --2- 172.21.15.29:0/1909356173 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563781cca820 0x563781ce27f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:52:30.118 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:30.112+0000 7f96063e7640 1 -- 172.21.15.29:0/1909356173 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563781cc1a50 msgr2=0x563781cd61f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:52:30.118 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:30.112+0000 7f96063e7640 1 --2- 172.21.15.29:0/1909356173 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563781cc1a50 0x563781cd61f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:52:30.118 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:30.112+0000 7f96063e7640 1 -- 172.21.15.29:0/1909356173 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563781cc1880 con 0x563781cc1130 2024-03-16T11:52:30.118 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:30.113+0000 7f96063e7640 1 --2- 172.21.15.29:0/1909356173 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563781cc1130 0x563781cc9fc0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f95fc005d00 tx=0x7f95fc010fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b77743b0101f6a41 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:52:30.118 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:30.113+0000 7f95ebfff640 1 -- 172.21.15.29:0/1909356173 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95fc005380 con 0x563781cc1130 2024-03-16T11:52:30.118 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:30.113+0000 7f95ebfff640 1 -- 172.21.15.29:0/1909356173 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f95fc005520 con 0x563781cc1130 2024-03-16T11:52:30.119 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:30.113+0000 7f95ebfff640 1 -- 172.21.15.29:0/1909356173 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95fc004660 con 0x563781cc1130 2024-03-16T11:52:30.119 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:30.113+0000 7f960b237740 1 -- 172.21.15.29:0/1909356173 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563781cc1130 msgr2=0x563781cc9fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:52:30.119 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:30.113+0000 7f960b237740 1 --2- 172.21.15.29:0/1909356173 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563781cc1130 0x563781cc9fc0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f95fc005d00 tx=0x7f95fc010fd0 comp rx=0 tx=0).stop 2024-03-16T11:52:30.119 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:30.113+0000 7f960b237740 1 -- 172.21.15.29:0/1909356173 shutdown_connections 2024-03-16T11:52:30.119 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:30.113+0000 7f960b237740 1 --2- 172.21.15.29:0/1909356173 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563781cca820 0x563781ce27f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:52:30.119 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:30.113+0000 7f960b237740 1 --2- 172.21.15.29:0/1909356173 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563781cc1a50 0x563781cd61f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:52:30.119 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:30.113+0000 7f960b237740 1 --2- 172.21.15.29:0/1909356173 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563781cc1130 0x563781cc9fc0 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:52:30.119 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:30.113+0000 7f960b237740 1 -- 172.21.15.29:0/1909356173 >> 172.21.15.29:0/1909356173 conn(0x563781cbba50 msgr2=0x563781cb6a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:52:30.119 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:30.113+0000 7f960b237740 1 -- 172.21.15.29:0/1909356173 shutdown_connections 2024-03-16T11:52:30.119 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:30.114+0000 7f960b237740 1 -- 172.21.15.29:0/1909356173 wait complete. 2024-03-16T11:52:30.119 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:30.114+0000 7f960b237740 1 -- start start 2024-03-16T11:52:30.119 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:30.114+0000 7f960b237740 1 Processor -- start 2024-03-16T11:52:30.119 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:30.114+0000 7f960b237740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563781cc1130 0x563781cd2740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:52:30.119 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:30.114+0000 7f960b237740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563781cc1a50 0x563781cd2c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:52:30.120 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:30.114+0000 7f960b237740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563781cca820 0x563781ccddf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:52:30.120 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:30.114+0000 7f960b237740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f96001a9700 con 0x563781cc1a50 2024-03-16T11:52:30.120 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:30.114+0000 7f960b237740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f96001aab70 con 0x563781cca820 2024-03-16T11:52:30.120 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:30.114+0000 7f960b237740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f96001a9dd0 con 0x563781cc1130 2024-03-16T11:52:30.120 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:30.114+0000 7f960b237740 10 client.0 _ll_register_callbacks cb 0x7f95f0012510 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T11:52:30.120 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:30.114+0000 7f9605be6640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563781cc1a50 0x563781cd2c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:52:30.120 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:30.114+0000 7f9605be6640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563781cc1a50 0x563781cd2c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39302/0 (socket says 172.21.15.29:39302) 2024-03-16T11:52:30.120 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:30.114+0000 7f96063e7640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563781cc1130 0x563781cd2740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:52:30.120 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:30.114+0000 7f9605be6640 1 -- 172.21.15.29:0/1564844243 learned_addr learned my addr 172.21.15.29:0/1564844243 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:52:30.120 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:30.114+0000 7f9606be8640 1 --2- 172.21.15.29:0/1564844243 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563781cca820 0x563781ccddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:52:30.120 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:30.114+0000 7f96063e7640 1 -- 172.21.15.29:0/1564844243 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563781cca820 msgr2=0x563781ccddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:52:30.120 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:30.114+0000 7f96063e7640 1 --2- 172.21.15.29:0/1564844243 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563781cca820 0x563781ccddf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:52:30.121 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:30.115+0000 7f96063e7640 1 -- 172.21.15.29:0/1564844243 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563781cc1a50 msgr2=0x563781cd2c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:52:30.121 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:30.115+0000 7f96063e7640 1 --2- 172.21.15.29:0/1564844243 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563781cc1a50 0x563781cd2c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:52:30.121 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:30.115+0000 7f96063e7640 1 -- 172.21.15.29:0/1564844243 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563781cce5f0 con 0x563781cc1130 2024-03-16T11:52:30.121 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:30.115+0000 7f9605be6640 1 --2- 172.21.15.29:0/1564844243 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563781cc1a50 0x563781cd2c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:52:30.121 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:30.115+0000 7f96063e7640 1 --2- 172.21.15.29:0/1564844243 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563781cc1130 0x563781cd2740 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f95fc000c00 tx=0x7f95fc005c60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=53381699c1c8b27a server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:52:30.121 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:30.115+0000 7f95e9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-16T11:52:30.121 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:30.115+0000 7f95e9ffb640 1 -- 172.21.15.29:0/1564844243 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95fc0282d0 con 0x563781cc1130 2024-03-16T11:52:30.121 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:30.115+0000 7f95e9ffb640 1 -- 172.21.15.29:0/1564844243 <== mon.1 v2:172.21.15.154:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8469+0+0 (secure 0 0 0) 0x7f95fc0285a0 con 0x563781cc1130 2024-03-16T11:52:30.121 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:30.115+0000 7f960b237740 1 -- 172.21.15.29:0/1564844243 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f95fc005520 con 0x563781cc1130 2024-03-16T11:52:30.121 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:30.115+0000 7f95e9ffb640 1 -- 172.21.15.29:0/1564844243 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f95fc0287a0 con 0x563781cc1130 2024-03-16T11:52:30.121 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:30.115+0000 7f960b237740 10 client.25387 fetch_fsmap learned FSMap version 22 2024-03-16T11:52:30.121 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:30.115+0000 7f960b237740 1 -- 172.21.15.29:0/1564844243 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x563781d40ac0 con 0x563781cc1130 2024-03-16T11:52:30.121 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:30.115+0000 7f95e9ffb640 1 -- 172.21.15.29:0/1564844243 <== mon.1 v2:172.21.15.154:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f95fc005520 con 0x563781cc1130 2024-03-16T11:52:30.121 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:30.115+0000 7f95e9ffb640 1 -- 172.21.15.29:0/1564844243 <== mon.1 v2:172.21.15.154:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7f95fc031070 con 0x563781cc1130 2024-03-16T11:52:30.122 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:30.115+0000 7f960b237740 10 client.25387 fetch_fsmap finished waiting for FSMap version 22 2024-03-16T11:52:30.122 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:30.115+0000 7f960b237740 10 client.25387 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T11:52:30.122 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:30.115+0000 7f960b237740 20 client.25387 populate_metadata read hostname 'smithi029' 2024-03-16T11:52:30.122 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:30.115+0000 7f960b237740 1 --2- 172.21.15.29:0/1564844243 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x563781d40f90 0x563781d43380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:52:30.122 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:30.115+0000 7f960b237740 4 client.25387 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-16T11:52:30.122 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:30.115+0000 7f960b237740 1 -- 172.21.15.29:0/1564844243 --> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] -- command(tid 0: {"prefix": "flush journal"}) -- 0x563781bab770 con 0x563781d40f90 2024-03-16T11:52:30.122 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:30.116+0000 7f9605be6640 1 --2- 172.21.15.29:0/1564844243 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x563781d40f90 0x563781d43380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:52:30.122 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:30.116+0000 7f95e9ffb640 10 client.25387 ms_handle_connect on v2:172.21.15.152:6838/1548731833 2024-03-16T11:52:30.122 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:30.116+0000 7f9605be6640 1 --2- 172.21.15.29:0/1564844243 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x563781d40f90 0x563781d43380 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=edac64bfeb67256e server_cookie=a09219382d25a415 in_seq=0 out_seq=0 2024-03-16T11:52:30.361 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:30.361+0000 7f95e9ffb640 1 -- 172.21.15.29:0/1564844243 <== mds.0 v2:172.21.15.152:6838/1548731833 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x563781bab770 con 0x563781d40f90 2024-03-16T11:52:30.361 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:30.361+0000 7f95e9ffb640 10 client.25387 handle_command_reply: tid=0 2024-03-16T11:52:30.361 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:30.361+0000 7f960b237740 1 client.25387 shutdown 2024-03-16T11:52:30.361 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:30.361+0000 7f960b237740 1 -- 172.21.15.29:0/1564844243 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563781cc1130 msgr2=0x563781cd2740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:52:30.361 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:30.361+0000 7f960b237740 1 --2- 172.21.15.29:0/1564844243 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563781cc1130 0x563781cd2740 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f95fc000c00 tx=0x7f95fc005c60 comp rx=0 tx=0).stop 2024-03-16T11:52:30.361 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:30.361+0000 7f960b237740 1 -- 172.21.15.29:0/1564844243 shutdown_connections 2024-03-16T11:52:30.362 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:30.361+0000 7f960b237740 1 --2- 172.21.15.29:0/1564844243 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x563781d40f90 0x563781d43380 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:52:30.362 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:30.361+0000 7f960b237740 1 --2- 172.21.15.29:0/1564844243 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563781cca820 0x563781ccddf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:52:30.362 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:30.361+0000 7f95e9ffb640 0 client.25387 ms_handle_reset on v2:172.21.15.152:6838/1548731833 2024-03-16T11:52:30.362 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:30.361+0000 7f960b237740 1 --2- 172.21.15.29:0/1564844243 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563781cc1a50 0x563781cd2c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:52:30.362 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:30.361+0000 7f960b237740 1 --2- 172.21.15.29:0/1564844243 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563781cc1130 0x563781cd2740 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:52:30.362 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:30.361+0000 7f960b237740 1 -- 172.21.15.29:0/1564844243 >> 172.21.15.29:0/1564844243 conn(0x563781cbba50 msgr2=0x563781cbcd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:52:30.362 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:30.361+0000 7f960b237740 1 -- 172.21.15.29:0/1564844243 shutdown_connections 2024-03-16T11:52:30.362 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:30.361+0000 7f960b237740 1 -- 172.21.15.29:0/1564844243 wait complete. 2024-03-16T11:52:30.362 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:30.362+0000 7f960b237740 20 client.25387 trim_cache size 0 max 16384 2024-03-16T11:52:30.362 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:{ 2024-03-16T11:52:30.363 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "message": "", 2024-03-16T11:52:30.363 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "return_code": 0 2024-03-16T11:52:30.363 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:} 2024-03-16T11:52:30.363 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:30.362+0000 7f960864b640 1 -- 172.21.15.29:0/3295556984 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f95ec0a2a10 msgr2=0x7f95ec0a4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:52:30.364 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:30.362+0000 7f960864b640 1 --2- 172.21.15.29:0/3295556984 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f95ec0a2a10 0x7f95ec0a4ed0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f95f000b6c0 tx=0x7f95f000b410 comp rx=0 tx=0).stop 2024-03-16T11:52:30.364 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:30.362+0000 7f960864b640 1 -- 172.21.15.29:0/3295556984 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f960010be30 msgr2=0x7f960019a600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:52:30.364 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:30.362+0000 7f960864b640 1 --2- 172.21.15.29:0/3295556984 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f960010be30 0x7f960019a600 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f95f8006010 tx=0x7f95f8003f10 comp rx=0 tx=0).stop 2024-03-16T11:52:30.364 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:30.362+0000 7f960864b640 1 -- 172.21.15.29:0/3295556984 shutdown_connections 2024-03-16T11:52:30.364 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:30.362+0000 7f960864b640 1 --2- 172.21.15.29:0/3295556984 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f95ec0a2a10 0x7f95ec0a4ed0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:52:30.364 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:30.362+0000 7f960864b640 1 --2- 172.21.15.29:0/3295556984 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f960010be30 0x7f960019a600 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:52:30.364 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:30.362+0000 7f960864b640 1 --2- 172.21.15.29:0/3295556984 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f96001032e0 0x7f96001962d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:52:30.364 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:30.362+0000 7f960864b640 1 --2- 172.21.15.29:0/3295556984 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f96001029c0 0x7f9600195d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:52:30.364 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:30.362+0000 7f960864b640 1 -- 172.21.15.29:0/3295556984 >> 172.21.15.29:0/3295556984 conn(0x7f9600077810 msgr2=0x7f96000fd5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:52:30.364 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:30.362+0000 7f960864b640 1 -- 172.21.15.29:0/3295556984 shutdown_connections 2024-03-16T11:52:30.364 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:30.362+0000 7f960864b640 1 -- 172.21.15.29:0/3295556984 wait complete. 2024-03-16T11:52:30.774 INFO:tasks.workunit.client.0.smithi029.stdout: 1 37402 14.51 MB/sec warmup 15 sec latency 58.111 ms 2024-03-16T11:52:30.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:52:30 smithi154 ceph-mon[36320]: pgmap v50: 97 pgs: 97 active+clean; 13 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 586 KiB/s rd, 1.2 MiB/s wr, 181 op/s 2024-03-16T11:52:30.837 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:52:30 smithi029 ceph-mon[31975]: pgmap v50: 97 pgs: 97 active+clean; 13 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 586 KiB/s rd, 1.2 MiB/s wr, 181 op/s 2024-03-16T11:52:30.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:52:30 smithi152 ceph-mon[36291]: pgmap v50: 97 pgs: 97 active+clean; 13 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 586 KiB/s rd, 1.2 MiB/s wr, 181 op/s 2024-03-16T11:52:31.774 INFO:tasks.workunit.client.0.smithi029.stdout: 1 39810 14.48 MB/sec warmup 16 sec latency 17.715 ms 2024-03-16T11:52:32.774 INFO:tasks.workunit.client.0.smithi029.stdout: 1 42846 14.71 MB/sec warmup 17 sec latency 18.468 ms 2024-03-16T11:52:32.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:52:32 smithi154 ceph-mon[36320]: pgmap v51: 97 pgs: 97 active+clean; 11 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 922 KiB/s rd, 1.9 MiB/s wr, 353 op/s 2024-03-16T11:52:32.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:52:32 smithi029 ceph-mon[31975]: pgmap v51: 97 pgs: 97 active+clean; 11 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 922 KiB/s rd, 1.9 MiB/s wr, 353 op/s 2024-03-16T11:52:32.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:52:32 smithi152 ceph-mon[36291]: pgmap v51: 97 pgs: 97 active+clean; 11 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 922 KiB/s rd, 1.9 MiB/s wr, 353 op/s 2024-03-16T11:52:33.774 INFO:tasks.workunit.client.0.smithi029.stdout: 1 45088 14.34 MB/sec warmup 18 sec latency 31.783 ms 2024-03-16T11:52:34.774 INFO:tasks.workunit.client.0.smithi029.stdout: 1 47137 14.24 MB/sec warmup 19 sec latency 17.908 ms 2024-03-16T11:52:34.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:52:34 smithi154 ceph-mon[36320]: pgmap v52: 97 pgs: 97 active+clean; 11 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 672 KiB/s rd, 1.2 MiB/s wr, 246 op/s 2024-03-16T11:52:34.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:52:34 smithi029 ceph-mon[31975]: pgmap v52: 97 pgs: 97 active+clean; 11 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 672 KiB/s rd, 1.2 MiB/s wr, 246 op/s 2024-03-16T11:52:34.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:52:34 smithi152 ceph-mon[36291]: pgmap v52: 97 pgs: 97 active+clean; 11 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 672 KiB/s rd, 1.2 MiB/s wr, 246 op/s 2024-03-16T11:52:35.774 INFO:tasks.workunit.client.0.smithi029.stdout: 1 49999 14.42 MB/sec warmup 20 sec latency 21.409 ms 2024-03-16T11:52:36.774 INFO:tasks.workunit.client.0.smithi029.stdout: 1 52683 14.35 MB/sec warmup 21 sec latency 14.484 ms 2024-03-16T11:52:36.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:52:36 smithi029 ceph-mon[31975]: pgmap v53: 97 pgs: 97 active+clean; 11 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 672 KiB/s rd, 1.2 MiB/s wr, 246 op/s 2024-03-16T11:52:36.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:52:36 smithi152 ceph-mon[36291]: pgmap v53: 97 pgs: 97 active+clean; 11 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 672 KiB/s rd, 1.2 MiB/s wr, 246 op/s 2024-03-16T11:52:37.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:52:36 smithi154 ceph-mon[36320]: pgmap v53: 97 pgs: 97 active+clean; 11 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 672 KiB/s rd, 1.2 MiB/s wr, 246 op/s 2024-03-16T11:52:37.774 INFO:tasks.workunit.client.0.smithi029.stdout: 1 54755 14.22 MB/sec warmup 22 sec latency 25.511 ms 2024-03-16T11:52:38.775 INFO:tasks.workunit.client.0.smithi029.stdout: 1 57382 14.21 MB/sec warmup 23 sec latency 7.875 ms 2024-03-16T11:52:38.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:52:38 smithi029 ceph-mon[31975]: pgmap v54: 97 pgs: 97 active+clean; 16 MiB data, 446 MiB used, 1.0 TiB / 1.0 TiB avail; 984 KiB/s rd, 1.8 MiB/s wr, 327 op/s 2024-03-16T11:52:38.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:52:38 smithi152 ceph-mon[36291]: pgmap v54: 97 pgs: 97 active+clean; 16 MiB data, 446 MiB used, 1.0 TiB / 1.0 TiB avail; 984 KiB/s rd, 1.8 MiB/s wr, 327 op/s 2024-03-16T11:52:39.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:52:38 smithi154 ceph-mon[36320]: pgmap v54: 97 pgs: 97 active+clean; 16 MiB data, 446 MiB used, 1.0 TiB / 1.0 TiB avail; 984 KiB/s rd, 1.8 MiB/s wr, 327 op/s 2024-03-16T11:52:39.775 INFO:tasks.workunit.client.0.smithi029.stdout: 1 60397 14.36 MB/sec warmup 24 sec latency 21.639 ms 2024-03-16T11:52:40.778 INFO:tasks.workunit.client.0.smithi029.stdout: 1 62707 14.12 MB/sec warmup 25 sec latency 16.385 ms 2024-03-16T11:52:40.837 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:52:40 smithi029 ceph-mon[31975]: pgmap v55: 97 pgs: 97 active+clean; 16 MiB data, 446 MiB used, 1.0 TiB / 1.0 TiB avail; 648 KiB/s rd, 1.2 MiB/s wr, 252 op/s 2024-03-16T11:52:40.846 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:52:40 smithi154 ceph-mon[36320]: pgmap v55: 97 pgs: 97 active+clean; 16 MiB data, 446 MiB used, 1.0 TiB / 1.0 TiB avail; 648 KiB/s rd, 1.2 MiB/s wr, 252 op/s 2024-03-16T11:52:40.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:52:40 smithi152 ceph-mon[36291]: pgmap v55: 97 pgs: 97 active+clean; 16 MiB data, 446 MiB used, 1.0 TiB / 1.0 TiB avail; 648 KiB/s rd, 1.2 MiB/s wr, 252 op/s 2024-03-16T11:52:41.775 INFO:tasks.workunit.client.0.smithi029.stdout: 1 64660 14.06 MB/sec warmup 26 sec latency 22.486 ms 2024-03-16T11:52:42.775 INFO:tasks.workunit.client.0.smithi029.stdout: 1 67682 14.22 MB/sec warmup 27 sec latency 9.482 ms 2024-03-16T11:52:42.837 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:52:42 smithi029 ceph-mon[31975]: pgmap v56: 97 pgs: 97 active+clean; 23 MiB data, 472 MiB used, 1.0 TiB / 1.0 TiB avail; 967 KiB/s rd, 1.9 MiB/s wr, 328 op/s 2024-03-16T11:52:42.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:52:42 smithi152 ceph-mon[36291]: pgmap v56: 97 pgs: 97 active+clean; 23 MiB data, 472 MiB used, 1.0 TiB / 1.0 TiB avail; 967 KiB/s rd, 1.9 MiB/s wr, 328 op/s 2024-03-16T11:52:43.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:52:42 smithi154 ceph-mon[36320]: pgmap v56: 97 pgs: 97 active+clean; 23 MiB data, 472 MiB used, 1.0 TiB / 1.0 TiB avail; 967 KiB/s rd, 1.9 MiB/s wr, 328 op/s 2024-03-16T11:52:43.455 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:43.453+0000 7f80581f3640 1 -- 172.21.15.29:0/3962377847 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f80501078b0 msgr2=0x7f8050107c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:52:43.455 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:43.453+0000 7f80581f3640 1 --2- 172.21.15.29:0/3962377847 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f80501078b0 0x7f8050107c90 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f8048009930 tx=0x7f804802bc30 comp rx=0 tx=0).stop 2024-03-16T11:52:43.455 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:43.453+0000 7f80581f3640 1 -- 172.21.15.29:0/3962377847 shutdown_connections 2024-03-16T11:52:43.455 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:43.453+0000 7f80581f3640 1 --2- 172.21.15.29:0/3962377847 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8050069800 0x7f805010e670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:52:43.455 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:43.453+0000 7f80581f3640 1 --2- 172.21.15.29:0/3962377847 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8050068e60 0x7f80500692c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:52:43.455 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:43.453+0000 7f80581f3640 1 --2- 172.21.15.29:0/3962377847 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f80501078b0 0x7f8050107c90 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:52:43.455 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:43.453+0000 7f80581f3640 1 -- 172.21.15.29:0/3962377847 >> 172.21.15.29:0/3962377847 conn(0x7f8050077530 msgr2=0x7f8050077940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:52:43.456 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:43.453+0000 7f80581f3640 1 -- 172.21.15.29:0/3962377847 shutdown_connections 2024-03-16T11:52:43.456 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:43.453+0000 7f80581f3640 1 -- 172.21.15.29:0/3962377847 wait complete. 2024-03-16T11:52:43.456 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:43.453+0000 7f80581f3640 1 Processor -- start 2024-03-16T11:52:43.456 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:43.453+0000 7f80581f3640 1 -- start start 2024-03-16T11:52:43.456 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:43.453+0000 7f80581f3640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8050068e60 0x7f8050197f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:52:43.456 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:43.453+0000 7f80581f3640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8050069800 0x7f80501984a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:52:43.456 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:43.454+0000 7f80581f3640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f80501078b0 0x7f805019c7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:52:43.456 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:43.454+0000 7f80581f3640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f8050111ba0 con 0x7f80501078b0 2024-03-16T11:52:43.456 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:43.454+0000 7f80581f3640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f8050111a20 con 0x7f8050068e60 2024-03-16T11:52:43.456 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:43.454+0000 7f80581f3640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f8050111d20 con 0x7f8050069800 2024-03-16T11:52:43.456 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:43.454+0000 7f8055f8f640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8050068e60 0x7f8050197f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:52:43.456 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:43.454+0000 7f8055f8f640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8050068e60 0x7f8050197f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.29:43320/0 (socket says 172.21.15.29:43320) 2024-03-16T11:52:43.456 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:43.454+0000 7f8055f8f640 1 -- 172.21.15.29:0/1398466949 learned_addr learned my addr 172.21.15.29:0/1398466949 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:52:43.457 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:43.454+0000 7f8056790640 1 --2- 172.21.15.29:0/1398466949 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f80501078b0 0x7f805019c7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:52:43.457 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:43.454+0000 7f805578e640 1 --2- 172.21.15.29:0/1398466949 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8050069800 0x7f80501984a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:52:43.457 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:43.454+0000 7f8056790640 1 -- 172.21.15.29:0/1398466949 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8050069800 msgr2=0x7f80501984a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:52:43.457 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:43.454+0000 7f8056790640 1 --2- 172.21.15.29:0/1398466949 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8050069800 0x7f80501984a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:52:43.457 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:43.454+0000 7f8056790640 1 -- 172.21.15.29:0/1398466949 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8050068e60 msgr2=0x7f8050197f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:52:43.457 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:43.454+0000 7f8056790640 1 --2- 172.21.15.29:0/1398466949 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8050068e60 0x7f8050197f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:52:43.457 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:43.454+0000 7f8056790640 1 -- 172.21.15.29:0/1398466949 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f805019cf00 con 0x7f80501078b0 2024-03-16T11:52:43.457 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:43.454+0000 7f8056790640 1 --2- 172.21.15.29:0/1398466949 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f80501078b0 0x7f805019c7d0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f8040005fd0 tx=0x7f804000ee60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:52:43.457 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:43.454+0000 7f804effd640 1 -- 172.21.15.29:0/1398466949 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f804000cd80 con 0x7f80501078b0 2024-03-16T11:52:43.457 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:43.454+0000 7f804effd640 1 -- 172.21.15.29:0/1398466949 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8040005020 con 0x7f80501078b0 2024-03-16T11:52:43.458 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:43.454+0000 7f80581f3640 1 -- 172.21.15.29:0/1398466949 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f805019d1f0 con 0x7f80501078b0 2024-03-16T11:52:43.458 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:43.454+0000 7f804effd640 1 -- 172.21.15.29:0/1398466949 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8040010720 con 0x7f80501078b0 2024-03-16T11:52:43.458 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:43.454+0000 7f80581f3640 1 -- 172.21.15.29:0/1398466949 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f80501a4a80 con 0x7f80501078b0 2024-03-16T11:52:43.463 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:43.456+0000 7f804effd640 1 -- 172.21.15.29:0/1398466949 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7f80400051d0 con 0x7f80501078b0 2024-03-16T11:52:43.463 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:43.457+0000 7f805add7740 1 Processor -- start 2024-03-16T11:52:43.463 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:43.457+0000 7f804effd640 1 --2- 172.21.15.29:0/1398466949 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f80380a2a10 0x7f80380a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:52:43.463 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:43.457+0000 7f805add7740 1 -- start start 2024-03-16T11:52:43.463 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:43.457+0000 7f805578e640 1 --2- 172.21.15.29:0/1398466949 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f80380a2a10 0x7f80380a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:52:43.463 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:43.457+0000 7f805add7740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b11cea1130 0x55b11cea1510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:52:43.463 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:43.457+0000 7f805add7740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b11cea1a50 0x55b11ceaa2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:52:43.463 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:43.457+0000 7f804effd640 1 -- 172.21.15.29:0/1398466949 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8469+0+0 (secure 0 0 0) 0x7f80400cf400 con 0x7f80501078b0 2024-03-16T11:52:43.463 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:43.457+0000 7f805add7740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b11ceaec30 0x55b11ceb1020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:52:43.464 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:43.457+0000 7f805add7740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55b11ce05a40 con 0x55b11cea1130 2024-03-16T11:52:43.464 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:43.457+0000 7f805add7740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x55b11cdae920 con 0x55b11cea1a50 2024-03-16T11:52:43.464 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:43.457+0000 7f805add7740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55b11ce01ae0 con 0x55b11ceaec30 2024-03-16T11:52:43.464 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:43.457+0000 7f8055f8f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b11cea1130 0x55b11cea1510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:52:43.464 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:43.458+0000 7f8055f8f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b11cea1130 0x55b11cea1510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:44728/0 (socket says 172.21.15.29:44728) 2024-03-16T11:52:43.464 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:43.458+0000 7f8055f8f640 1 -- 172.21.15.29:0/2070030244 learned_addr learned my addr 172.21.15.29:0/2070030244 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:52:43.464 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:43.458+0000 7f8056790640 1 --2- 172.21.15.29:0/2070030244 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b11ceaec30 0x55b11ceb1020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:52:43.464 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:43.458+0000 7f805578e640 1 --2- 172.21.15.29:0/2070030244 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b11cea1a50 0x55b11ceaa2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:52:43.464 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:43.458+0000 7f8055f8f640 1 -- 172.21.15.29:0/2070030244 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b11ceaec30 msgr2=0x55b11ceb1020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:52:43.464 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:43.458+0000 7f8055f8f640 1 --2- 172.21.15.29:0/2070030244 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b11ceaec30 0x55b11ceb1020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:52:43.464 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:43.458+0000 7f8055f8f640 1 -- 172.21.15.29:0/2070030244 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b11cea1a50 msgr2=0x55b11ceaa2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:52:43.464 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:43.458+0000 7f8055f8f640 1 --2- 172.21.15.29:0/2070030244 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b11cea1a50 0x55b11ceaa2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:52:43.464 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:43.458+0000 7f8055f8f640 1 -- 172.21.15.29:0/2070030244 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b11cdc6460 con 0x55b11cea1130 2024-03-16T11:52:43.464 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:43.458+0000 7f8056790640 1 --2- 172.21.15.29:0/2070030244 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b11ceaec30 0x55b11ceb1020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:52:43.464 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:43.458+0000 7f805578e640 1 --2- 172.21.15.29:0/1398466949 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f80380a2a10 0x7f80380a4ed0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f8050077a20 tx=0x7f8044011920 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:52:43.465 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:43.458+0000 7f8055f8f640 1 --2- 172.21.15.29:0/2070030244 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b11cea1130 0x55b11cea1510 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f8048009430 tx=0x7f8048009660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9055be12190e86cc server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:52:43.465 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:43.458+0000 7f8037fff640 1 -- 172.21.15.29:0/2070030244 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8048042070 con 0x55b11cea1130 2024-03-16T11:52:43.465 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:43.458+0000 7f8037fff640 1 -- 172.21.15.29:0/2070030244 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f804802c8b0 con 0x55b11cea1130 2024-03-16T11:52:43.465 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:43.458+0000 7f8037fff640 1 -- 172.21.15.29:0/2070030244 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8048004c20 con 0x55b11cea1130 2024-03-16T11:52:43.465 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:43.459+0000 7f805add7740 1 -- 172.21.15.29:0/2070030244 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b11cea1130 msgr2=0x55b11cea1510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:52:43.465 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:43.459+0000 7f805add7740 1 --2- 172.21.15.29:0/2070030244 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b11cea1130 0x55b11cea1510 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f8048009430 tx=0x7f8048009660 comp rx=0 tx=0).stop 2024-03-16T11:52:43.465 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:43.459+0000 7f805add7740 1 -- 172.21.15.29:0/2070030244 shutdown_connections 2024-03-16T11:52:43.465 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:43.459+0000 7f805add7740 1 --2- 172.21.15.29:0/2070030244 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b11ceaec30 0x55b11ceb1020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:52:43.465 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:43.459+0000 7f805add7740 1 --2- 172.21.15.29:0/2070030244 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b11cea1a50 0x55b11ceaa2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:52:43.465 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:43.459+0000 7f805add7740 1 --2- 172.21.15.29:0/2070030244 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b11cea1130 0x55b11cea1510 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:52:43.465 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:43.459+0000 7f805add7740 1 -- 172.21.15.29:0/2070030244 >> 172.21.15.29:0/2070030244 conn(0x55b11ce9ba50 msgr2=0x55b11ce9be30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:52:43.465 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:43.459+0000 7f805add7740 1 -- 172.21.15.29:0/2070030244 shutdown_connections 2024-03-16T11:52:43.465 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:43.459+0000 7f805add7740 1 -- 172.21.15.29:0/2070030244 wait complete. 2024-03-16T11:52:43.465 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:43.459+0000 7f805add7740 1 -- start start 2024-03-16T11:52:43.465 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:43.459+0000 7f805add7740 1 Processor -- start 2024-03-16T11:52:43.466 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:43.460+0000 7f805add7740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b11cea1130 0x55b11cea7820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:52:43.466 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:43.460+0000 7f805add7740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b11cea1a50 0x55b11cea7d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:52:43.466 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:43.460+0000 7f805add7740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b11ceaec30 0x55b11cea82a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:52:43.466 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:43.460+0000 7f805add7740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55b11ceb1800 con 0x55b11cea1130 2024-03-16T11:52:43.466 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:43.460+0000 7f805add7740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x55b11cdc1d70 con 0x55b11cea1a50 2024-03-16T11:52:43.466 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:43.460+0000 7f805add7740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55b11ceb1980 con 0x55b11ceaec30 2024-03-16T11:52:43.466 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:43.460+0000 7f805add7740 10 client.0 _ll_register_callbacks cb 0x55b11cb19a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T11:52:43.466 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:43.460+0000 7f8055f8f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b11cea1130 0x55b11cea7820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:52:43.466 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:43.460+0000 7f8055f8f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b11cea1130 0x55b11cea7820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:44740/0 (socket says 172.21.15.29:44740) 2024-03-16T11:52:43.466 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:43.460+0000 7f8055f8f640 1 -- 172.21.15.29:0/4208794376 learned_addr learned my addr 172.21.15.29:0/4208794376 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:52:43.466 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:43.460+0000 7f8056790640 1 --2- 172.21.15.29:0/4208794376 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b11ceaec30 0x55b11cea82a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:52:43.466 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:43.460+0000 7f805578e640 1 --2- 172.21.15.29:0/4208794376 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b11cea1a50 0x55b11cea7d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:52:43.466 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:43.460+0000 7f8055f8f640 1 -- 172.21.15.29:0/4208794376 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b11ceaec30 msgr2=0x55b11cea82a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:52:43.467 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:43.460+0000 7f8055f8f640 1 --2- 172.21.15.29:0/4208794376 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b11ceaec30 0x55b11cea82a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:52:43.467 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:43.460+0000 7f8055f8f640 1 -- 172.21.15.29:0/4208794376 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b11cea1a50 msgr2=0x55b11cea7d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:52:43.467 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:43.460+0000 7f8055f8f640 1 --2- 172.21.15.29:0/4208794376 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b11cea1a50 0x55b11cea7d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:52:43.467 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:43.460+0000 7f8055f8f640 1 -- 172.21.15.29:0/4208794376 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b11cb1a210 con 0x55b11cea1130 2024-03-16T11:52:43.467 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:43.460+0000 7f8055f8f640 1 --2- 172.21.15.29:0/4208794376 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b11cea1130 0x55b11cea7820 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f8048004700 tx=0x7f804802c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5cf8bb33a7b31eff server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:52:43.467 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:43.460+0000 7f80357fa640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-16T11:52:43.467 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:43.461+0000 7f80357fa640 1 -- 172.21.15.29:0/4208794376 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8048042040 con 0x55b11cea1130 2024-03-16T11:52:43.467 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:43.461+0000 7f80357fa640 1 -- 172.21.15.29:0/4208794376 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8469+0+0 (secure 0 0 0) 0x7f8048004240 con 0x55b11cea1130 2024-03-16T11:52:43.467 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:43.461+0000 7f805add7740 1 -- 172.21.15.29:0/4208794376 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f804802c8b0 con 0x55b11cea1130 2024-03-16T11:52:43.467 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:43.461+0000 7f80357fa640 1 -- 172.21.15.29:0/4208794376 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f8048004c70 con 0x55b11cea1130 2024-03-16T11:52:43.467 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:43.461+0000 7f805add7740 10 client.15552 fetch_fsmap learned FSMap version 22 2024-03-16T11:52:43.467 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:43.461+0000 7f805add7740 1 -- 172.21.15.29:0/4208794376 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x55b11cbe6620 con 0x55b11cea1130 2024-03-16T11:52:43.467 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:43.461+0000 7f80357fa640 1 -- 172.21.15.29:0/4208794376 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f804802c8b0 con 0x55b11cea1130 2024-03-16T11:52:43.467 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:43.461+0000 7f80357fa640 1 -- 172.21.15.29:0/4208794376 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7f8048003950 con 0x55b11cea1130 2024-03-16T11:52:43.467 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:43.461+0000 7f805add7740 10 client.15552 fetch_fsmap finished waiting for FSMap version 22 2024-03-16T11:52:43.468 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:43.461+0000 7f805add7740 10 client.15552 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T11:52:43.468 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:43.461+0000 7f805add7740 20 client.15552 populate_metadata read hostname 'smithi029' 2024-03-16T11:52:43.468 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:43.461+0000 7f805add7740 1 --2- 172.21.15.29:0/4208794376 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55b11cebeba0 0x55b11cec0f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:52:43.468 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:43.461+0000 7f805add7740 4 client.15552 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T11:52:43.468 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:43.461+0000 7f805add7740 1 -- 172.21.15.29:0/4208794376 --> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b11ce12f80 con 0x55b11cebeba0 2024-03-16T11:52:43.468 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:43.461+0000 7f805578e640 1 --2- 172.21.15.29:0/4208794376 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55b11cebeba0 0x55b11cec0f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:52:43.468 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:43.462+0000 7f805578e640 1 --2- 172.21.15.29:0/4208794376 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55b11cebeba0 0x55b11cec0f90 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=91b7348bdedb357b server_cookie=c803b0783461c857 in_seq=0 out_seq=0 2024-03-16T11:52:43.468 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:43.462+0000 7f80357fa640 10 client.15552 ms_handle_connect on v2:172.21.15.152:6838/1548731833 2024-03-16T11:52:43.469 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:43.465+0000 7f80357fa640 1 -- 172.21.15.29:0/4208794376 <== mds.0 v2:172.21.15.152:6838/1548731833 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55b11ce12f80 con 0x55b11cebeba0 2024-03-16T11:52:43.469 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:43.465+0000 7f80357fa640 10 client.15552 handle_command_reply: tid=0 2024-03-16T11:52:43.469 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:43.465+0000 7f805add7740 1 client.15552 shutdown 2024-03-16T11:52:43.469 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:43.465+0000 7f805add7740 1 -- 172.21.15.29:0/4208794376 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b11cea1130 msgr2=0x55b11cea7820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:52:43.469 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:43.465+0000 7f805add7740 1 --2- 172.21.15.29:0/4208794376 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b11cea1130 0x55b11cea7820 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f8048004700 tx=0x7f804802c800 comp rx=0 tx=0).stop 2024-03-16T11:52:43.469 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:43.465+0000 7f805add7740 1 -- 172.21.15.29:0/4208794376 shutdown_connections 2024-03-16T11:52:43.470 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:43.465+0000 7f805add7740 1 --2- 172.21.15.29:0/4208794376 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b11ceaec30 0x55b11cea82a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:52:43.470 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:43.465+0000 7f805add7740 1 --2- 172.21.15.29:0/4208794376 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55b11cebeba0 0x55b11cec0f90 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:52:43.470 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:43.465+0000 7f805add7740 1 --2- 172.21.15.29:0/4208794376 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b11cea1a50 0x55b11cea7d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:52:43.470 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:43.465+0000 7f805add7740 1 --2- 172.21.15.29:0/4208794376 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b11cea1130 0x55b11cea7820 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:52:43.470 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:43.465+0000 7f80357fa640 0 client.15552 ms_handle_reset on v2:172.21.15.152:6838/1548731833 2024-03-16T11:52:43.470 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:43.465+0000 7f805add7740 1 -- 172.21.15.29:0/4208794376 >> 172.21.15.29:0/4208794376 conn(0x55b11ce9ba50 msgr2=0x55b11ce9d7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:52:43.470 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:43.465+0000 7f805add7740 1 -- 172.21.15.29:0/4208794376 shutdown_connections 2024-03-16T11:52:43.470 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:43.465+0000 7f805add7740 1 -- 172.21.15.29:0/4208794376 wait complete. 2024-03-16T11:52:43.470 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:43.465+0000 7f805add7740 20 client.15552 trim_cache size 0 max 16384 2024-03-16T11:52:43.485 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:43.481+0000 7f805add7740 1 Processor -- start 2024-03-16T11:52:43.485 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:43.481+0000 7f805add7740 1 -- start start 2024-03-16T11:52:43.485 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:43.481+0000 7f805add7740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b11cea1130 0x55b11cea9f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:52:43.485 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:43.481+0000 7f805add7740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b11cea1a50 0x55b11ce96a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:52:43.485 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:43.481+0000 7f805add7740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b11ceaec30 0x55b11ceba5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:52:43.485 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:43.481+0000 7f805add7740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f80501ad990 con 0x55b11cea1a50 2024-03-16T11:52:43.485 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:43.481+0000 7f805add7740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f80501ab260 con 0x55b11cea1130 2024-03-16T11:52:43.486 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:43.481+0000 7f805add7740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f80501a90b0 con 0x55b11ceaec30 2024-03-16T11:52:43.486 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:43.481+0000 7f805578e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b11cea1a50 0x55b11ce96a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:52:43.486 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:43.481+0000 7f805578e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b11cea1a50 0x55b11ce96a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:44744/0 (socket says 172.21.15.29:44744) 2024-03-16T11:52:43.486 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:43.481+0000 7f805578e640 1 -- 172.21.15.29:0/795836026 learned_addr learned my addr 172.21.15.29:0/795836026 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:52:43.486 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:43.481+0000 7f8056790640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b11ceaec30 0x55b11ceba5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:52:43.486 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:43.481+0000 7f8055f8f640 1 --2- 172.21.15.29:0/795836026 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b11cea1130 0x55b11cea9f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:52:43.486 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:43.481+0000 7f805578e640 1 -- 172.21.15.29:0/795836026 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b11ceaec30 msgr2=0x55b11ceba5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:52:43.486 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:43.481+0000 7f805578e640 1 --2- 172.21.15.29:0/795836026 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b11ceaec30 0x55b11ceba5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:52:43.486 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:43.481+0000 7f805578e640 1 -- 172.21.15.29:0/795836026 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b11cea1130 msgr2=0x55b11cea9f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:52:43.486 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:43.481+0000 7f805578e640 1 --2- 172.21.15.29:0/795836026 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b11cea1130 0x55b11cea9f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:52:43.486 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:43.481+0000 7f805578e640 1 -- 172.21.15.29:0/795836026 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b11cea67d0 con 0x55b11cea1a50 2024-03-16T11:52:43.486 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:43.481+0000 7f805578e640 1 --2- 172.21.15.29:0/795836026 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b11cea1a50 0x55b11ce96a40 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f8044013010 tx=0x7f804401d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e728243adc2827a7 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:52:43.486 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:43.482+0000 7f80377fe640 1 -- 172.21.15.29:0/795836026 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8044032070 con 0x55b11cea1a50 2024-03-16T11:52:43.487 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:43.482+0000 7f80377fe640 1 -- 172.21.15.29:0/795836026 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f804402c7b0 con 0x55b11cea1a50 2024-03-16T11:52:43.487 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:43.482+0000 7f80377fe640 1 -- 172.21.15.29:0/795836026 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f804401a040 con 0x55b11cea1a50 2024-03-16T11:52:43.487 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:43.482+0000 7f805add7740 1 -- 172.21.15.29:0/795836026 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b11cea1a50 msgr2=0x55b11ce96a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:52:43.487 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:43.482+0000 7f805add7740 1 --2- 172.21.15.29:0/795836026 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b11cea1a50 0x55b11ce96a40 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f8044013010 tx=0x7f804401d000 comp rx=0 tx=0).stop 2024-03-16T11:52:43.487 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:43.482+0000 7f805add7740 1 -- 172.21.15.29:0/795836026 shutdown_connections 2024-03-16T11:52:43.487 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:43.482+0000 7f805add7740 1 --2- 172.21.15.29:0/795836026 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b11ceaec30 0x55b11ceba5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:52:43.487 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:43.482+0000 7f805add7740 1 --2- 172.21.15.29:0/795836026 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b11cea1a50 0x55b11ce96a40 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:52:43.487 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:43.482+0000 7f805add7740 1 --2- 172.21.15.29:0/795836026 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b11cea1130 0x55b11cea9f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:52:43.487 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:43.482+0000 7f805add7740 1 -- 172.21.15.29:0/795836026 >> 172.21.15.29:0/795836026 conn(0x55b11ce9ba50 msgr2=0x55b11ce9abb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:52:43.487 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:43.482+0000 7f805add7740 1 -- 172.21.15.29:0/795836026 shutdown_connections 2024-03-16T11:52:43.487 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:43.482+0000 7f805add7740 1 -- 172.21.15.29:0/795836026 wait complete. 2024-03-16T11:52:43.487 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:43.483+0000 7f805add7740 1 -- start start 2024-03-16T11:52:43.488 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:43.483+0000 7f805add7740 1 Processor -- start 2024-03-16T11:52:43.488 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:43.483+0000 7f805add7740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b11cea1130 0x55b11ceb26a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:52:43.488 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:43.483+0000 7f805add7740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b11cea1a50 0x55b11ceb2be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:52:43.488 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:43.483+0000 7f805add7740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b11ceaec30 0x55b11ceb3120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:52:43.488 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:43.483+0000 7f805add7740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f80501a9720 con 0x55b11cea1a50 2024-03-16T11:52:43.488 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:43.483+0000 7f805add7740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f80501aab90 con 0x55b11ceaec30 2024-03-16T11:52:43.488 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:43.483+0000 7f805add7740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f80501a9df0 con 0x55b11cea1130 2024-03-16T11:52:43.488 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:43.483+0000 7f805add7740 10 client.0 _ll_register_callbacks cb 0x7f804400f5c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T11:52:43.488 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:43.483+0000 7f805578e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b11cea1a50 0x55b11ceb2be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:52:43.488 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:43.483+0000 7f8055f8f640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b11cea1130 0x55b11ceb26a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:52:43.488 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:43.483+0000 7f805578e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b11cea1a50 0x55b11ceb2be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:44760/0 (socket says 172.21.15.29:44760) 2024-03-16T11:52:43.489 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:43.483+0000 7f805578e640 1 -- 172.21.15.29:0/4165183710 learned_addr learned my addr 172.21.15.29:0/4165183710 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:52:43.489 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:43.483+0000 7f8056790640 1 --2- 172.21.15.29:0/4165183710 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b11ceaec30 0x55b11ceb3120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:52:43.489 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:43.484+0000 7f805578e640 1 -- 172.21.15.29:0/4165183710 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b11cea1130 msgr2=0x55b11ceb26a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:52:43.489 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:43.484+0000 7f805578e640 1 --2- 172.21.15.29:0/4165183710 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b11cea1130 0x55b11ceb26a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:52:43.489 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:43.484+0000 7f805578e640 1 -- 172.21.15.29:0/4165183710 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b11ceaec30 msgr2=0x55b11ceb3120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:52:43.489 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:43.484+0000 7f805578e640 1 --2- 172.21.15.29:0/4165183710 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b11ceaec30 0x55b11ceb3120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:52:43.489 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:43.484+0000 7f805578e640 1 -- 172.21.15.29:0/4165183710 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b11cf23d90 con 0x55b11cea1a50 2024-03-16T11:52:43.489 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:43.484+0000 7f8055f8f640 1 --2- 172.21.15.29:0/4165183710 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b11cea1130 0x55b11ceb26a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:52:43.489 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:43.484+0000 7f805578e640 1 --2- 172.21.15.29:0/4165183710 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b11cea1a50 0x55b11ceb2be0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f8044003ae0 tx=0x7f8044009a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d46cfb612c38279 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:52:43.489 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:43.484+0000 7f80357fa640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-16T11:52:43.489 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:43.484+0000 7f80357fa640 1 -- 172.21.15.29:0/4165183710 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f804403b060 con 0x55b11cea1a50 2024-03-16T11:52:43.489 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:43.484+0000 7f80357fa640 1 -- 172.21.15.29:0/4165183710 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8469+0+0 (secure 0 0 0) 0x7f8044003b40 con 0x55b11cea1a50 2024-03-16T11:52:43.489 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:43.484+0000 7f805add7740 1 -- 172.21.15.29:0/4165183710 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f804402c7b0 con 0x55b11cea1a50 2024-03-16T11:52:43.489 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:43.484+0000 7f80357fa640 1 -- 172.21.15.29:0/4165183710 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f8044032040 con 0x55b11cea1a50 2024-03-16T11:52:43.490 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:43.484+0000 7f805add7740 10 client.15558 fetch_fsmap learned FSMap version 22 2024-03-16T11:52:43.490 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:43.484+0000 7f805add7740 1 -- 172.21.15.29:0/4165183710 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x55b11cf25060 con 0x55b11cea1a50 2024-03-16T11:52:43.490 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:43.484+0000 7f80357fa640 1 -- 172.21.15.29:0/4165183710 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f804402c7b0 con 0x55b11cea1a50 2024-03-16T11:52:43.490 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:43.484+0000 7f80357fa640 1 -- 172.21.15.29:0/4165183710 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7f8044005160 con 0x55b11cea1a50 2024-03-16T11:52:43.490 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:43.484+0000 7f805add7740 10 client.15558 fetch_fsmap finished waiting for FSMap version 22 2024-03-16T11:52:43.490 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:43.484+0000 7f805add7740 10 client.15558 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T11:52:43.490 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:43.484+0000 7f805add7740 20 client.15558 populate_metadata read hostname 'smithi029' 2024-03-16T11:52:43.490 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:43.484+0000 7f805add7740 1 --2- 172.21.15.29:0/4165183710 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55b11cf256f0 0x55b11cf27b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:52:43.490 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:43.484+0000 7f805add7740 4 client.15558 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-16T11:52:43.490 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:43.484+0000 7f805add7740 1 -- 172.21.15.29:0/4165183710 --> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55b11ce0d2d0 con 0x55b11cf256f0 2024-03-16T11:52:43.490 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:43.485+0000 7f8055f8f640 1 --2- 172.21.15.29:0/4165183710 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55b11cf256f0 0x55b11cf27b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:52:43.490 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:43.485+0000 7f80357fa640 10 client.15558 ms_handle_connect on v2:172.21.15.152:6838/1548731833 2024-03-16T11:52:43.490 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:43.485+0000 7f8055f8f640 1 --2- 172.21.15.29:0/4165183710 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55b11cf256f0 0x55b11cf27b10 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=56e1a96f96f06872 server_cookie=66315daf4d3752a7 in_seq=0 out_seq=0 2024-03-16T11:52:43.644 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:43.644+0000 7f80357fa640 1 -- 172.21.15.29:0/4165183710 <== mds.0 v2:172.21.15.152:6838/1548731833 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55b11ce0d2d0 con 0x55b11cf256f0 2024-03-16T11:52:43.644 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:43.644+0000 7f80357fa640 10 client.15558 handle_command_reply: tid=0 2024-03-16T11:52:43.645 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:43.644+0000 7f805add7740 1 client.15558 shutdown 2024-03-16T11:52:43.645 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:43.644+0000 7f805add7740 1 -- 172.21.15.29:0/4165183710 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b11cea1a50 msgr2=0x55b11ceb2be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:52:43.645 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:43.644+0000 7f805add7740 1 --2- 172.21.15.29:0/4165183710 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b11cea1a50 0x55b11ceb2be0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f8044003ae0 tx=0x7f8044009a90 comp rx=0 tx=0).stop 2024-03-16T11:52:43.645 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:43.644+0000 7f805add7740 1 -- 172.21.15.29:0/4165183710 shutdown_connections 2024-03-16T11:52:43.645 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:43.644+0000 7f805add7740 1 --2- 172.21.15.29:0/4165183710 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55b11cf256f0 0x55b11cf27b10 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:52:43.645 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:43.644+0000 7f805add7740 1 --2- 172.21.15.29:0/4165183710 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b11ceaec30 0x55b11ceb3120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:52:43.645 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:43.644+0000 7f80357fa640 0 client.15558 ms_handle_reset on v2:172.21.15.152:6838/1548731833 2024-03-16T11:52:43.645 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:43.644+0000 7f805add7740 1 --2- 172.21.15.29:0/4165183710 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b11cea1a50 0x55b11ceb2be0 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:52:43.645 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:43.644+0000 7f805add7740 1 --2- 172.21.15.29:0/4165183710 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b11cea1130 0x55b11ceb26a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:52:43.645 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:43.644+0000 7f805add7740 1 -- 172.21.15.29:0/4165183710 >> 172.21.15.29:0/4165183710 conn(0x55b11ce9ba50 msgr2=0x55b11cea77c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:52:43.645 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:43.644+0000 7f805add7740 1 -- 172.21.15.29:0/4165183710 shutdown_connections 2024-03-16T11:52:43.645 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:43.644+0000 7f805add7740 1 -- 172.21.15.29:0/4165183710 wait complete. 2024-03-16T11:52:43.645 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:43.644+0000 7f805add7740 20 client.15558 trim_cache size 0 max 16384 2024-03-16T11:52:43.646 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:{ 2024-03-16T11:52:43.646 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "message": "", 2024-03-16T11:52:43.646 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "return_code": 0 2024-03-16T11:52:43.646 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:} 2024-03-16T11:52:43.647 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:43.645+0000 7f80581f3640 1 -- 172.21.15.29:0/1398466949 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f80380a2a10 msgr2=0x7f80380a4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:52:43.647 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:43.645+0000 7f80581f3640 1 --2- 172.21.15.29:0/1398466949 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f80380a2a10 0x7f80380a4ed0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f8050077a20 tx=0x7f8044011920 comp rx=0 tx=0).stop 2024-03-16T11:52:43.647 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:43.645+0000 7f80581f3640 1 -- 172.21.15.29:0/1398466949 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f80501078b0 msgr2=0x7f805019c7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:52:43.647 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:43.645+0000 7f80581f3640 1 --2- 172.21.15.29:0/1398466949 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f80501078b0 0x7f805019c7d0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f8040005fd0 tx=0x7f804000ee60 comp rx=0 tx=0).stop 2024-03-16T11:52:43.647 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:43.645+0000 7f80581f3640 1 -- 172.21.15.29:0/1398466949 shutdown_connections 2024-03-16T11:52:43.647 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:43.645+0000 7f80581f3640 1 --2- 172.21.15.29:0/1398466949 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f80380a2a10 0x7f80380a4ed0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:52:43.647 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:43.645+0000 7f80581f3640 1 --2- 172.21.15.29:0/1398466949 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f80501078b0 0x7f805019c7d0 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:52:43.647 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:43.645+0000 7f80581f3640 1 --2- 172.21.15.29:0/1398466949 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8050069800 0x7f80501984a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:52:43.647 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:43.645+0000 7f80581f3640 1 --2- 172.21.15.29:0/1398466949 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8050068e60 0x7f8050197f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:52:43.647 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:43.645+0000 7f80581f3640 1 -- 172.21.15.29:0/1398466949 >> 172.21.15.29:0/1398466949 conn(0x7f8050077530 msgr2=0x7f80500fe8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:52:43.647 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:43.645+0000 7f80581f3640 1 -- 172.21.15.29:0/1398466949 shutdown_connections 2024-03-16T11:52:43.647 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:43.645+0000 7f80581f3640 1 -- 172.21.15.29:0/1398466949 wait complete. 2024-03-16T11:52:43.775 INFO:tasks.workunit.client.0.smithi029.stdout: 1 69797 13.97 MB/sec warmup 28 sec latency 55.415 ms 2024-03-16T11:52:44.775 INFO:tasks.workunit.client.0.smithi029.stdout: 1 71789 13.93 MB/sec warmup 29 sec latency 19.301 ms 2024-03-16T11:52:44.837 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:52:44 smithi029 ceph-mon[31975]: pgmap v57: 97 pgs: 97 active+clean; 23 MiB data, 472 MiB used, 1.0 TiB / 1.0 TiB avail; 632 KiB/s rd, 1.2 MiB/s wr, 157 op/s 2024-03-16T11:52:44.879 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:52:44 smithi152 ceph-mon[36291]: pgmap v57: 97 pgs: 97 active+clean; 23 MiB data, 472 MiB used, 1.0 TiB / 1.0 TiB avail; 632 KiB/s rd, 1.2 MiB/s wr, 157 op/s 2024-03-16T11:52:45.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:52:44 smithi154 ceph-mon[36320]: pgmap v57: 97 pgs: 97 active+clean; 23 MiB data, 472 MiB used, 1.0 TiB / 1.0 TiB avail; 632 KiB/s rd, 1.2 MiB/s wr, 157 op/s 2024-03-16T11:52:45.775 INFO:tasks.workunit.client.0.smithi029.stdout: 1 74844 14.08 MB/sec warmup 30 sec latency 10.628 ms 2024-03-16T11:52:46.775 INFO:tasks.workunit.client.0.smithi029.stdout: 1 76961 13.86 MB/sec warmup 31 sec latency 126.530 ms 2024-03-16T11:52:46.837 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:52:46 smithi029 ceph-mon[31975]: pgmap v58: 97 pgs: 97 active+clean; 23 MiB data, 472 MiB used, 1.0 TiB / 1.0 TiB avail; 632 KiB/s rd, 1.2 MiB/s wr, 157 op/s 2024-03-16T11:52:46.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:52:46 smithi152 ceph-mon[36291]: pgmap v58: 97 pgs: 97 active+clean; 23 MiB data, 472 MiB used, 1.0 TiB / 1.0 TiB avail; 632 KiB/s rd, 1.2 MiB/s wr, 157 op/s 2024-03-16T11:52:47.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:52:46 smithi154 ceph-mon[36320]: pgmap v58: 97 pgs: 97 active+clean; 23 MiB data, 472 MiB used, 1.0 TiB / 1.0 TiB avail; 632 KiB/s rd, 1.2 MiB/s wr, 157 op/s 2024-03-16T11:52:47.775 INFO:tasks.workunit.client.0.smithi029.stdout: 1 77459 13.60 MB/sec warmup 32 sec latency 502.091 ms 2024-03-16T11:52:48.775 INFO:tasks.workunit.client.0.smithi029.stdout: 1 78809 13.39 MB/sec warmup 33 sec latency 39.443 ms 2024-03-16T11:52:48.837 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:52:48 smithi029 ceph-mon[31975]: pgmap v59: 97 pgs: 97 active+clean; 20 MiB data, 456 MiB used, 1.0 TiB / 1.0 TiB avail; 876 KiB/s rd, 1.8 MiB/s wr, 357 op/s 2024-03-16T11:52:48.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:52:48 smithi152 ceph-mon[36291]: pgmap v59: 97 pgs: 97 active+clean; 20 MiB data, 456 MiB used, 1.0 TiB / 1.0 TiB avail; 876 KiB/s rd, 1.8 MiB/s wr, 357 op/s 2024-03-16T11:52:49.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:52:48 smithi154 ceph-mon[36320]: pgmap v59: 97 pgs: 97 active+clean; 20 MiB data, 456 MiB used, 1.0 TiB / 1.0 TiB avail; 876 KiB/s rd, 1.8 MiB/s wr, 357 op/s 2024-03-16T11:52:49.775 INFO:tasks.workunit.client.0.smithi029.stdout: 1 81859 13.54 MB/sec warmup 34 sec latency 14.205 ms 2024-03-16T11:52:50.775 INFO:tasks.workunit.client.0.smithi029.stdout: 1 83971 13.36 MB/sec warmup 35 sec latency 84.195 ms 2024-03-16T11:52:50.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:52:50 smithi029 ceph-mon[31975]: pgmap v60: 97 pgs: 97 active+clean; 20 MiB data, 456 MiB used, 1.0 TiB / 1.0 TiB avail; 564 KiB/s rd, 1.2 MiB/s wr, 276 op/s 2024-03-16T11:52:50.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:52:50 smithi152 ceph-mon[36291]: pgmap v60: 97 pgs: 97 active+clean; 20 MiB data, 456 MiB used, 1.0 TiB / 1.0 TiB avail; 564 KiB/s rd, 1.2 MiB/s wr, 276 op/s 2024-03-16T11:52:51.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:52:50 smithi154 ceph-mon[36320]: pgmap v60: 97 pgs: 97 active+clean; 20 MiB data, 456 MiB used, 1.0 TiB / 1.0 TiB avail; 564 KiB/s rd, 1.2 MiB/s wr, 276 op/s 2024-03-16T11:52:51.775 INFO:tasks.workunit.client.0.smithi029.stdout: 1 85853 13.34 MB/sec warmup 36 sec latency 17.557 ms 2024-03-16T11:52:52.775 INFO:tasks.workunit.client.0.smithi029.stdout: 1 88811 13.45 MB/sec warmup 37 sec latency 13.568 ms 2024-03-16T11:52:52.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:52:52 smithi029 ceph-mon[31975]: pgmap v61: 97 pgs: 97 active+clean; 34 MiB data, 502 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 8.4 MiB/s wr, 523 op/s 2024-03-16T11:52:52.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:52:52 smithi152 ceph-mon[36291]: pgmap v61: 97 pgs: 97 active+clean; 34 MiB data, 502 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 8.4 MiB/s wr, 523 op/s 2024-03-16T11:52:53.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:52:52 smithi154 ceph-mon[36320]: pgmap v61: 97 pgs: 97 active+clean; 34 MiB data, 502 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 8.4 MiB/s wr, 523 op/s 2024-03-16T11:52:53.775 INFO:tasks.workunit.client.0.smithi029.stdout: 1 90817 13.28 MB/sec warmup 38 sec latency 31.260 ms 2024-03-16T11:52:54.775 INFO:tasks.workunit.client.0.smithi029.stdout: 1 92934 13.29 MB/sec warmup 39 sec latency 22.066 ms 2024-03-16T11:52:54.837 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:52:54 smithi029 ceph-mon[31975]: pgmap v62: 97 pgs: 97 active+clean; 34 MiB data, 502 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 7.8 MiB/s wr, 447 op/s 2024-03-16T11:52:54.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:52:54 smithi152 ceph-mon[36291]: pgmap v62: 97 pgs: 97 active+clean; 34 MiB data, 502 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 7.8 MiB/s wr, 447 op/s 2024-03-16T11:52:55.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:52:54 smithi154 ceph-mon[36320]: pgmap v62: 97 pgs: 97 active+clean; 34 MiB data, 502 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 7.8 MiB/s wr, 447 op/s 2024-03-16T11:52:55.775 INFO:tasks.workunit.client.0.smithi029.stdout: 1 95877 13.40 MB/sec warmup 40 sec latency 17.283 ms 2024-03-16T11:52:56.737 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:56.736+0000 7f27195cd640 1 -- 172.21.15.29:0/1645979582 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f27141067c0 msgr2=0x7f2714106ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:52:56.737 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:56.736+0000 7f27195cd640 1 --2- 172.21.15.29:0/1645979582 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f27141067c0 0x7f2714106ba0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f2704009930 tx=0x7f270402bc30 comp rx=0 tx=0).stop 2024-03-16T11:52:56.738 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:56.736+0000 7f27195cd640 1 -- 172.21.15.29:0/1645979582 shutdown_connections 2024-03-16T11:52:56.738 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:56.736+0000 7f27195cd640 1 --2- 172.21.15.29:0/1645979582 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f27141006a0 0x7f271410e670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:52:56.738 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:56.736+0000 7f27195cd640 1 --2- 172.21.15.29:0/1645979582 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f27140ffce0 0x7f2714100160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:52:56.738 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:56.736+0000 7f27195cd640 1 --2- 172.21.15.29:0/1645979582 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f27141067c0 0x7f2714106ba0 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:52:56.738 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:56.736+0000 7f27195cd640 1 -- 172.21.15.29:0/1645979582 >> 172.21.15.29:0/1645979582 conn(0x7f2714076c70 msgr2=0x7f27140fdff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:52:56.738 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:56.736+0000 7f27195cd640 1 -- 172.21.15.29:0/1645979582 shutdown_connections 2024-03-16T11:52:56.738 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:56.736+0000 7f27195cd640 1 -- 172.21.15.29:0/1645979582 wait complete. 2024-03-16T11:52:56.741 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:56.737+0000 7f27195cd640 1 Processor -- start 2024-03-16T11:52:56.741 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:56.737+0000 7f27195cd640 1 -- start start 2024-03-16T11:52:56.741 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:56.737+0000 7f27195cd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f27140ffce0 0x7f27141980b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:52:56.741 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:56.737+0000 7f27195cd640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f27141006a0 0x7f27141985f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:52:56.741 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:56.737+0000 7f27195cd640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f27141067c0 0x7f271419c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:52:56.741 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:56.737+0000 7f27195cd640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f2714111cd0 con 0x7f27140ffce0 2024-03-16T11:52:56.741 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:56.737+0000 7f27195cd640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f2714111b50 con 0x7f27141006a0 2024-03-16T11:52:56.742 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:56.737+0000 7f27195cd640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f2714111e50 con 0x7f27141067c0 2024-03-16T11:52:56.742 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:56.737+0000 7f2712ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f27140ffce0 0x7f27141980b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:52:56.742 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:56.737+0000 7f2712ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f27140ffce0 0x7f27141980b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:40584/0 (socket says 172.21.15.29:40584) 2024-03-16T11:52:56.742 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:56.737+0000 7f2712ffd640 1 -- 172.21.15.29:0/106249506 learned_addr learned my addr 172.21.15.29:0/106249506 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:52:56.742 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:56.737+0000 7f27137fe640 1 --2- 172.21.15.29:0/106249506 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f27141067c0 0x7f271419c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:52:56.742 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:56.737+0000 7f270bfff640 1 --2- 172.21.15.29:0/106249506 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f27141006a0 0x7f27141985f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:52:56.742 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:56.737+0000 7f2712ffd640 1 -- 172.21.15.29:0/106249506 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f27141067c0 msgr2=0x7f271419c920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:52:56.742 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:56.737+0000 7f2712ffd640 1 --2- 172.21.15.29:0/106249506 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f27141067c0 0x7f271419c920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:52:56.742 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:56.737+0000 7f2712ffd640 1 -- 172.21.15.29:0/106249506 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f27141006a0 msgr2=0x7f27141985f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:52:56.742 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:56.737+0000 7f2712ffd640 1 --2- 172.21.15.29:0/106249506 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f27141006a0 0x7f27141985f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:52:56.742 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:56.737+0000 7f2712ffd640 1 -- 172.21.15.29:0/106249506 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f271419d050 con 0x7f27140ffce0 2024-03-16T11:52:56.742 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:56.737+0000 7f2712ffd640 1 --2- 172.21.15.29:0/106249506 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f27140ffce0 0x7f27141980b0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f2704009230 tx=0x7f270402cb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:52:56.742 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:56.738+0000 7f2710ff9640 1 -- 172.21.15.29:0/106249506 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2704041070 con 0x7f27140ffce0 2024-03-16T11:52:56.742 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:56.738+0000 7f27195cd640 1 -- 172.21.15.29:0/106249506 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f271419d2e0 con 0x7f27140ffce0 2024-03-16T11:52:56.743 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:56.738+0000 7f2710ff9640 1 -- 172.21.15.29:0/106249506 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2704004450 con 0x7f27140ffce0 2024-03-16T11:52:56.743 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:56.738+0000 7f2710ff9640 1 -- 172.21.15.29:0/106249506 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2704003e50 con 0x7f27140ffce0 2024-03-16T11:52:56.743 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:56.738+0000 7f27195cd640 1 -- 172.21.15.29:0/106249506 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f27141a4bd0 con 0x7f27140ffce0 2024-03-16T11:52:56.749 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:56.740+0000 7f2710ff9640 1 -- 172.21.15.29:0/106249506 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7f2704044070 con 0x7f27140ffce0 2024-03-16T11:52:56.750 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:56.740+0000 7f2710ff9640 1 --2- 172.21.15.29:0/106249506 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f27000a2bb0 0x7f27000a5070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:52:56.750 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:56.740+0000 7f270bfff640 1 --2- 172.21.15.29:0/106249506 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f27000a2bb0 0x7f27000a5070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:52:56.750 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:56.740+0000 7f2710ff9640 1 -- 172.21.15.29:0/106249506 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8469+0+0 (secure 0 0 0) 0x7f27040f05e0 con 0x7f27140ffce0 2024-03-16T11:52:56.750 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:56.740+0000 7f271c1bc740 1 Processor -- start 2024-03-16T11:52:56.750 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:56.741+0000 7f271c1bc740 1 -- start start 2024-03-16T11:52:56.750 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:56.741+0000 7f270bfff640 1 --2- 172.21.15.29:0/106249506 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f27000a2bb0 0x7f27000a5070 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f27140ff230 tx=0x7f26fc005f90 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:52:56.750 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:56.741+0000 7f271c1bc740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5583a4f1b130 0x5583a4f1b510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:52:56.750 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:56.741+0000 7f271c1bc740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5583a4f1ba50 0x5583a4f242e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:52:56.750 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:56.741+0000 7f271c1bc740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5583a4f24820 0x5583a4f26c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:52:56.750 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:56.741+0000 7f271c1bc740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x5583a4e7fa40 con 0x5583a4f24820 2024-03-16T11:52:56.750 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:56.741+0000 7f271c1bc740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x5583a4e28920 con 0x5583a4f1b130 2024-03-16T11:52:56.750 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:56.741+0000 7f271c1bc740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5583a4e7bae0 con 0x5583a4f1ba50 2024-03-16T11:52:56.751 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:56.741+0000 7f27137fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5583a4f24820 0x5583a4f26c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:52:56.751 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:56.741+0000 7f2712ffd640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5583a4f1b130 0x5583a4f1b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:52:56.751 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:56.741+0000 7f270bfff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5583a4f1ba50 0x5583a4f242e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:52:56.751 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:56.741+0000 7f27137fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5583a4f24820 0x5583a4f26c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:40592/0 (socket says 172.21.15.29:40592) 2024-03-16T11:52:56.751 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:56.741+0000 7f2712ffd640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5583a4f1b130 0x5583a4f1b510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.29:58730/0 (socket says 172.21.15.29:58730) 2024-03-16T11:52:56.751 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:56.741+0000 7f27137fe640 1 -- 172.21.15.29:0/3123106643 learned_addr learned my addr 172.21.15.29:0/3123106643 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:52:56.751 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:56.741+0000 7f270bfff640 1 -- 172.21.15.29:0/3123106643 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5583a4f1b130 msgr2=0x5583a4f1b510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:52:56.751 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:56.741+0000 7f270bfff640 1 --2- 172.21.15.29:0/3123106643 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5583a4f1b130 0x5583a4f1b510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:52:56.751 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:56.741+0000 7f270bfff640 1 -- 172.21.15.29:0/3123106643 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5583a4f24820 msgr2=0x5583a4f26c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:52:56.751 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:56.741+0000 7f270bfff640 1 --2- 172.21.15.29:0/3123106643 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5583a4f24820 0x5583a4f26c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:52:56.751 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:56.741+0000 7f270bfff640 1 -- 172.21.15.29:0/3123106643 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5583a4e40460 con 0x5583a4f1ba50 2024-03-16T11:52:56.751 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:56.741+0000 7f2712ffd640 1 --2- 172.21.15.29:0/3123106643 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5583a4f1b130 0x5583a4f1b510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:52:56.751 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:56.742+0000 7f270bfff640 1 --2- 172.21.15.29:0/3123106643 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5583a4f1ba50 0x5583a4f242e0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f26fc0118d0 tx=0x7f26fc014000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c96f09f38c33d9ff server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:52:56.751 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:56.742+0000 7f27097fa640 1 -- 172.21.15.29:0/3123106643 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26fc004030 con 0x5583a4f1ba50 2024-03-16T11:52:56.751 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:56.742+0000 7f27097fa640 1 -- 172.21.15.29:0/3123106643 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f26fc0041d0 con 0x5583a4f1ba50 2024-03-16T11:52:56.752 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:56.742+0000 7f27097fa640 1 -- 172.21.15.29:0/3123106643 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26fc026760 con 0x5583a4f1ba50 2024-03-16T11:52:56.752 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:56.742+0000 7f271c1bc740 1 -- 172.21.15.29:0/3123106643 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5583a4f1ba50 msgr2=0x5583a4f242e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:52:56.752 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:56.742+0000 7f271c1bc740 1 --2- 172.21.15.29:0/3123106643 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5583a4f1ba50 0x5583a4f242e0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f26fc0118d0 tx=0x7f26fc014000 comp rx=0 tx=0).stop 2024-03-16T11:52:56.752 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:56.743+0000 7f271c1bc740 1 -- 172.21.15.29:0/3123106643 shutdown_connections 2024-03-16T11:52:56.752 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:56.743+0000 7f271c1bc740 1 --2- 172.21.15.29:0/3123106643 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5583a4f24820 0x5583a4f26c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:52:56.752 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:56.743+0000 7f271c1bc740 1 --2- 172.21.15.29:0/3123106643 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5583a4f1ba50 0x5583a4f242e0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:52:56.752 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:56.743+0000 7f271c1bc740 1 --2- 172.21.15.29:0/3123106643 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5583a4f1b130 0x5583a4f1b510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:52:56.752 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:56.743+0000 7f271c1bc740 1 -- 172.21.15.29:0/3123106643 >> 172.21.15.29:0/3123106643 conn(0x5583a4f15a50 msgr2=0x5583a4f15e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:52:56.752 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:56.743+0000 7f271c1bc740 1 -- 172.21.15.29:0/3123106643 shutdown_connections 2024-03-16T11:52:56.752 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:56.743+0000 7f271c1bc740 1 -- 172.21.15.29:0/3123106643 wait complete. 2024-03-16T11:52:56.752 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:56.743+0000 7f271c1bc740 1 -- start start 2024-03-16T11:52:56.752 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:56.743+0000 7f271c1bc740 1 Processor -- start 2024-03-16T11:52:56.752 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:56.743+0000 7f271c1bc740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5583a4f1b130 0x5583a4f218b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:52:56.752 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:56.743+0000 7f271c1bc740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5583a4f1ba50 0x5583a4f21df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:52:56.753 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:56.743+0000 7f271c1bc740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5583a4f24820 0x5583a4f22330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:52:56.753 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:56.743+0000 7f271c1bc740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x5583a4f27450 con 0x5583a4f1b130 2024-03-16T11:52:56.753 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:56.743+0000 7f271c1bc740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x5583a4e3bd70 con 0x5583a4f1ba50 2024-03-16T11:52:56.753 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:56.743+0000 7f271c1bc740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5583a4f275d0 con 0x5583a4f24820 2024-03-16T11:52:56.753 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:56.743+0000 7f271c1bc740 10 client.0 _ll_register_callbacks cb 0x5583a4b93a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T11:52:56.753 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:56.743+0000 7f2712ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5583a4f1b130 0x5583a4f218b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:52:56.753 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:56.743+0000 7f2712ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5583a4f1b130 0x5583a4f218b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:40602/0 (socket says 172.21.15.29:40602) 2024-03-16T11:52:56.753 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:56.743+0000 7f2712ffd640 1 -- 172.21.15.29:0/453209943 learned_addr learned my addr 172.21.15.29:0/453209943 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:52:56.753 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:56.743+0000 7f270bfff640 1 --2- 172.21.15.29:0/453209943 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5583a4f1ba50 0x5583a4f21df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:52:56.753 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:56.743+0000 7f27137fe640 1 --2- 172.21.15.29:0/453209943 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5583a4f24820 0x5583a4f22330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:52:56.753 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:56.744+0000 7f2712ffd640 1 -- 172.21.15.29:0/453209943 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5583a4f24820 msgr2=0x5583a4f22330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:52:56.753 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:56.744+0000 7f2712ffd640 1 --2- 172.21.15.29:0/453209943 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5583a4f24820 0x5583a4f22330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:52:56.753 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:56.744+0000 7f2712ffd640 1 -- 172.21.15.29:0/453209943 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5583a4f1ba50 msgr2=0x5583a4f21df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:52:56.754 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:56.744+0000 7f2712ffd640 1 --2- 172.21.15.29:0/453209943 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5583a4f1ba50 0x5583a4f21df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:52:56.754 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:56.744+0000 7f2712ffd640 1 -- 172.21.15.29:0/453209943 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5583a4b94210 con 0x5583a4f1b130 2024-03-16T11:52:56.754 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:56.744+0000 7f2712ffd640 1 --2- 172.21.15.29:0/453209943 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5583a4f1b130 0x5583a4f218b0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f2704009430 tx=0x7f27040bc850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e646ea56da76fa3 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:52:56.754 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:56.744+0000 7f26fa7fc640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-16T11:52:56.754 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:56.744+0000 7f26fa7fc640 1 -- 172.21.15.29:0/453209943 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f270404b040 con 0x5583a4f1b130 2024-03-16T11:52:56.754 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:56.744+0000 7f26fa7fc640 1 -- 172.21.15.29:0/453209943 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8469+0+0 (secure 0 0 0) 0x7f27040c5090 con 0x5583a4f1b130 2024-03-16T11:52:56.754 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:56.744+0000 7f271c1bc740 1 -- 172.21.15.29:0/453209943 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f26fc0041d0 con 0x5583a4f1b130 2024-03-16T11:52:56.754 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:56.744+0000 7f26fa7fc640 1 -- 172.21.15.29:0/453209943 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f27040bce30 con 0x5583a4f1b130 2024-03-16T11:52:56.754 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:56.744+0000 7f271c1bc740 10 client.15570 fetch_fsmap learned FSMap version 22 2024-03-16T11:52:56.754 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:56.744+0000 7f271c1bc740 1 -- 172.21.15.29:0/453209943 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x5583a4c60620 con 0x5583a4f1b130 2024-03-16T11:52:56.754 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:56.744+0000 7f26fa7fc640 1 -- 172.21.15.29:0/453209943 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f26fc0041d0 con 0x5583a4f1b130 2024-03-16T11:52:56.754 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:56.744+0000 7f26fa7fc640 1 -- 172.21.15.29:0/453209943 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7f27040bc280 con 0x5583a4f1b130 2024-03-16T11:52:56.754 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:56.744+0000 7f271c1bc740 10 client.15570 fetch_fsmap finished waiting for FSMap version 22 2024-03-16T11:52:56.754 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:56.744+0000 7f271c1bc740 10 client.15570 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T11:52:56.754 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:56.745+0000 7f271c1bc740 20 client.15570 populate_metadata read hostname 'smithi029' 2024-03-16T11:52:56.755 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:56.745+0000 7f271c1bc740 1 --2- 172.21.15.29:0/453209943 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x5583a4f34a30 0x5583a4f36e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:52:56.755 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:56.745+0000 7f271c1bc740 4 client.15570 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T11:52:56.755 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:56.745+0000 7f271c1bc740 1 -- 172.21.15.29:0/453209943 --> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5583a4e8cf80 con 0x5583a4f34a30 2024-03-16T11:52:56.755 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:56.745+0000 7f270bfff640 1 --2- 172.21.15.29:0/453209943 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x5583a4f34a30 0x5583a4f36e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:52:56.755 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:56.745+0000 7f26fa7fc640 10 client.15570 ms_handle_connect on v2:172.21.15.152:6838/1548731833 2024-03-16T11:52:56.755 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:56.745+0000 7f270bfff640 1 --2- 172.21.15.29:0/453209943 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x5583a4f34a30 0x5583a4f36e80 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=32b7ad2aa6b7d40 server_cookie=a1e5d849ea20d500 in_seq=0 out_seq=0 2024-03-16T11:52:56.755 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:56.747+0000 7f26fa7fc640 1 -- 172.21.15.29:0/453209943 <== mds.0 v2:172.21.15.152:6838/1548731833 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5583a4e8cf80 con 0x5583a4f34a30 2024-03-16T11:52:56.755 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:56.747+0000 7f26fa7fc640 10 client.15570 handle_command_reply: tid=0 2024-03-16T11:52:56.755 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:56.747+0000 7f271c1bc740 1 client.15570 shutdown 2024-03-16T11:52:56.755 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:56.747+0000 7f271c1bc740 1 -- 172.21.15.29:0/453209943 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5583a4f1b130 msgr2=0x5583a4f218b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:52:56.755 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:56.747+0000 7f271c1bc740 1 --2- 172.21.15.29:0/453209943 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5583a4f1b130 0x5583a4f218b0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f2704009430 tx=0x7f27040bc850 comp rx=0 tx=0).stop 2024-03-16T11:52:56.755 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:56.747+0000 7f271c1bc740 1 -- 172.21.15.29:0/453209943 shutdown_connections 2024-03-16T11:52:56.755 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:56.747+0000 7f271c1bc740 1 --2- 172.21.15.29:0/453209943 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5583a4f24820 0x5583a4f22330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:52:56.755 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:56.747+0000 7f271c1bc740 1 --2- 172.21.15.29:0/453209943 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x5583a4f34a30 0x5583a4f36e80 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:52:56.755 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:56.747+0000 7f271c1bc740 1 --2- 172.21.15.29:0/453209943 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5583a4f1ba50 0x5583a4f21df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:52:56.756 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:56.747+0000 7f26fa7fc640 0 client.15570 ms_handle_reset on v2:172.21.15.152:6838/1548731833 2024-03-16T11:52:56.756 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:56.747+0000 7f271c1bc740 1 --2- 172.21.15.29:0/453209943 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5583a4f1b130 0x5583a4f218b0 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:52:56.756 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:56.747+0000 7f271c1bc740 1 -- 172.21.15.29:0/453209943 >> 172.21.15.29:0/453209943 conn(0x5583a4f15a50 msgr2=0x5583a4f177a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:52:56.756 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:56.747+0000 7f271c1bc740 1 -- 172.21.15.29:0/453209943 shutdown_connections 2024-03-16T11:52:56.756 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:56.747+0000 7f271c1bc740 1 -- 172.21.15.29:0/453209943 wait complete. 2024-03-16T11:52:56.756 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:56.747+0000 7f271c1bc740 20 client.15570 trim_cache size 0 max 16384 2024-03-16T11:52:56.767 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:56.763+0000 7f271c1bc740 1 Processor -- start 2024-03-16T11:52:56.767 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:56.763+0000 7f271c1bc740 1 -- start start 2024-03-16T11:52:56.767 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:56.763+0000 7f271c1bc740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5583a4f1b130 0x5583a4f23fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:52:56.767 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:56.763+0000 7f271c1bc740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5583a4f1ba50 0x5583a4f301f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:52:56.768 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:56.763+0000 7f271c1bc740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5583a4f24820 0x5583a4f175b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:52:56.768 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:56.763+0000 7f271c1bc740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f27141afc10 con 0x5583a4f1b130 2024-03-16T11:52:56.768 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:56.763+0000 7f271c1bc740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f27141ad4e0 con 0x5583a4f24820 2024-03-16T11:52:56.768 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:56.763+0000 7f271c1bc740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f27141ab330 con 0x5583a4f1ba50 2024-03-16T11:52:56.768 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:56.763+0000 7f2712ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5583a4f1b130 0x5583a4f23fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:52:56.768 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:56.763+0000 7f2712ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5583a4f1b130 0x5583a4f23fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:40612/0 (socket says 172.21.15.29:40612) 2024-03-16T11:52:56.768 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:56.763+0000 7f2712ffd640 1 -- 172.21.15.29:0/3202535934 learned_addr learned my addr 172.21.15.29:0/3202535934 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:52:56.768 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:56.763+0000 7f27137fe640 1 --2- 172.21.15.29:0/3202535934 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5583a4f24820 0x5583a4f175b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:52:56.768 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:56.763+0000 7f270bfff640 1 --2- 172.21.15.29:0/3202535934 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5583a4f1ba50 0x5583a4f301f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:52:56.768 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:56.764+0000 7f2712ffd640 1 -- 172.21.15.29:0/3202535934 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5583a4f1ba50 msgr2=0x5583a4f301f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:52:56.768 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:56.764+0000 7f2712ffd640 1 --2- 172.21.15.29:0/3202535934 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5583a4f1ba50 0x5583a4f301f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:52:56.768 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:56.764+0000 7f2712ffd640 1 -- 172.21.15.29:0/3202535934 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5583a4f24820 msgr2=0x5583a4f175b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:52:56.768 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:56.764+0000 7f2712ffd640 1 --2- 172.21.15.29:0/3202535934 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5583a4f24820 0x5583a4f175b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:52:56.768 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:56.764+0000 7f2712ffd640 1 -- 172.21.15.29:0/3202535934 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5583a4f1fe00 con 0x5583a4f1b130 2024-03-16T11:52:56.769 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:56.764+0000 7f2712ffd640 1 --2- 172.21.15.29:0/3202535934 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5583a4f1b130 0x5583a4f23fe0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f27040b9290 tx=0x7f27040bc8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20408aa1599919eb server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:52:56.769 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:56.764+0000 7f2708ff9640 1 -- 172.21.15.29:0/3202535934 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f270404b070 con 0x5583a4f1b130 2024-03-16T11:52:56.769 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:56.764+0000 7f2708ff9640 1 -- 172.21.15.29:0/3202535934 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f27040c0ed0 con 0x5583a4f1b130 2024-03-16T11:52:56.769 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:56.764+0000 7f2708ff9640 1 -- 172.21.15.29:0/3202535934 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27040b94a0 con 0x5583a4f1b130 2024-03-16T11:52:56.769 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:56.765+0000 7f271c1bc740 1 -- 172.21.15.29:0/3202535934 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5583a4f1b130 msgr2=0x5583a4f23fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:52:56.769 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:56.765+0000 7f271c1bc740 1 --2- 172.21.15.29:0/3202535934 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5583a4f1b130 0x5583a4f23fe0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f27040b9290 tx=0x7f27040bc8c0 comp rx=0 tx=0).stop 2024-03-16T11:52:56.769 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:56.765+0000 7f271c1bc740 1 -- 172.21.15.29:0/3202535934 shutdown_connections 2024-03-16T11:52:56.769 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:56.765+0000 7f271c1bc740 1 --2- 172.21.15.29:0/3202535934 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5583a4f24820 0x5583a4f175b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:52:56.769 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:56.765+0000 7f271c1bc740 1 --2- 172.21.15.29:0/3202535934 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5583a4f1ba50 0x5583a4f301f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:52:56.769 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:56.765+0000 7f271c1bc740 1 --2- 172.21.15.29:0/3202535934 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5583a4f1b130 0x5583a4f23fe0 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:52:56.769 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:56.765+0000 7f271c1bc740 1 -- 172.21.15.29:0/3202535934 >> 172.21.15.29:0/3202535934 conn(0x5583a4f15a50 msgr2=0x5583a4f16ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:52:56.769 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:56.765+0000 7f271c1bc740 1 -- 172.21.15.29:0/3202535934 shutdown_connections 2024-03-16T11:52:56.769 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:56.765+0000 7f271c1bc740 1 -- 172.21.15.29:0/3202535934 wait complete. 2024-03-16T11:52:56.769 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:56.765+0000 7f271c1bc740 1 -- start start 2024-03-16T11:52:56.769 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:56.765+0000 7f271c1bc740 1 Processor -- start 2024-03-16T11:52:56.770 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:56.765+0000 7f271c1bc740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5583a4f1b130 0x5583a4f28a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:52:56.770 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:56.765+0000 7f271c1bc740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5583a4f1ba50 0x5583a4f28f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:52:56.770 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:56.765+0000 7f271c1bc740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5583a4f24820 0x5583a4f93dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:52:56.770 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:56.765+0000 7f271c1bc740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f27141ab9a0 con 0x5583a4f24820 2024-03-16T11:52:56.770 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:56.765+0000 7f271c1bc740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f27141ace10 con 0x5583a4f1ba50 2024-03-16T11:52:56.770 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:56.765+0000 7f271c1bc740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f27141ac070 con 0x5583a4f1b130 2024-03-16T11:52:56.770 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:56.765+0000 7f271c1bc740 10 client.0 _ll_register_callbacks cb 0x7f27040095c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T11:52:56.770 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:56.766+0000 7f2712ffd640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5583a4f1b130 0x5583a4f28a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:52:56.770 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:56.766+0000 7f27137fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5583a4f24820 0x5583a4f93dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:52:56.770 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:56.766+0000 7f2712ffd640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5583a4f1b130 0x5583a4f28a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.29:45804/0 (socket says 172.21.15.29:45804) 2024-03-16T11:52:56.770 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:56.766+0000 7f2712ffd640 1 -- 172.21.15.29:0/1969590041 learned_addr learned my addr 172.21.15.29:0/1969590041 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:52:56.770 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:56.766+0000 7f270bfff640 1 --2- 172.21.15.29:0/1969590041 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5583a4f1ba50 0x5583a4f28f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:52:56.770 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:56.766+0000 7f2712ffd640 1 -- 172.21.15.29:0/1969590041 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5583a4f1ba50 msgr2=0x5583a4f28f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:52:56.770 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:56.766+0000 7f2712ffd640 1 --2- 172.21.15.29:0/1969590041 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5583a4f1ba50 0x5583a4f28f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:52:56.771 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:56.766+0000 7f2712ffd640 1 -- 172.21.15.29:0/1969590041 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5583a4f24820 msgr2=0x5583a4f93dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:52:56.771 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:56.766+0000 7f2712ffd640 1 --2- 172.21.15.29:0/1969590041 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5583a4f24820 0x5583a4f93dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:52:56.771 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:56.766+0000 7f2712ffd640 1 -- 172.21.15.29:0/1969590041 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5583a4f94440 con 0x5583a4f1b130 2024-03-16T11:52:56.771 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:56.766+0000 7f27137fe640 1 --2- 172.21.15.29:0/1969590041 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5583a4f24820 0x5583a4f93dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:52:56.771 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:56.766+0000 7f2712ffd640 1 --2- 172.21.15.29:0/1969590041 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5583a4f1b130 0x5583a4f28a00 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f2704009430 tx=0x7f27040bc850 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a7561b04eeba1748 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:52:56.771 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:56.766+0000 7f26fa7fc640 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-16T11:52:56.771 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:56.766+0000 7f26fa7fc640 1 -- 172.21.15.29:0/1969590041 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f270404b070 con 0x5583a4f1b130 2024-03-16T11:52:56.771 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:56.766+0000 7f26fa7fc640 1 -- 172.21.15.29:0/1969590041 <== mon.1 v2:172.21.15.154:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8469+0+0 (secure 0 0 0) 0x7f27040c5090 con 0x5583a4f1b130 2024-03-16T11:52:56.771 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:56.766+0000 7f271c1bc740 1 -- 172.21.15.29:0/1969590041 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f27040c0ed0 con 0x5583a4f1b130 2024-03-16T11:52:56.771 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:56.766+0000 7f26fa7fc640 1 -- 172.21.15.29:0/1969590041 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f27040bcc00 con 0x5583a4f1b130 2024-03-16T11:52:56.771 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:56.766+0000 7f271c1bc740 10 client.25423 fetch_fsmap learned FSMap version 22 2024-03-16T11:52:56.771 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:56.766+0000 7f271c1bc740 1 -- 172.21.15.29:0/1969590041 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x5583a4f94440 con 0x5583a4f1b130 2024-03-16T11:52:56.771 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:56.766+0000 7f26fa7fc640 1 -- 172.21.15.29:0/1969590041 <== mon.1 v2:172.21.15.154:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f27040c0ed0 con 0x5583a4f1b130 2024-03-16T11:52:56.771 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:56.767+0000 7f26fa7fc640 1 -- 172.21.15.29:0/1969590041 <== mon.1 v2:172.21.15.154:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7f27040b8c10 con 0x5583a4f1b130 2024-03-16T11:52:56.772 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:56.767+0000 7f271c1bc740 10 client.25423 fetch_fsmap finished waiting for FSMap version 22 2024-03-16T11:52:56.772 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:56.767+0000 7f271c1bc740 10 client.25423 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T11:52:56.772 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:56.767+0000 7f271c1bc740 20 client.25423 populate_metadata read hostname 'smithi029' 2024-03-16T11:52:56.772 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:56.767+0000 7f271c1bc740 1 --2- 172.21.15.29:0/1969590041 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x5583a4f34a30 0x5583a4f9db00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:52:56.772 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:56.767+0000 7f271c1bc740 4 client.25423 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-16T11:52:56.772 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:56.767+0000 7f271c1bc740 1 -- 172.21.15.29:0/1969590041 --> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5583a4e872d0 con 0x5583a4f34a30 2024-03-16T11:52:56.772 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:56.767+0000 7f270bfff640 1 --2- 172.21.15.29:0/1969590041 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x5583a4f34a30 0x5583a4f9db00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:52:56.772 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:56.767+0000 7f270bfff640 1 --2- 172.21.15.29:0/1969590041 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x5583a4f34a30 0x5583a4f9db00 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ca5e1e6f28caf927 server_cookie=976dbad7ec56f14f in_seq=0 out_seq=0 2024-03-16T11:52:56.772 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:56.767+0000 7f26fa7fc640 10 client.25423 ms_handle_connect on v2:172.21.15.152:6838/1548731833 2024-03-16T11:52:56.775 INFO:tasks.workunit.client.0.smithi029.stdout: 1 97724 13.24 MB/sec warmup 41 sec latency 26.463 ms 2024-03-16T11:52:56.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:52:56 smithi029 ceph-mon[31975]: pgmap v63: 97 pgs: 97 active+clean; 34 MiB data, 502 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 7.8 MiB/s wr, 447 op/s 2024-03-16T11:52:56.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:52:56 smithi152 ceph-mon[36291]: pgmap v63: 97 pgs: 97 active+clean; 34 MiB data, 502 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 7.8 MiB/s wr, 447 op/s 2024-03-16T11:52:56.899 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:56.899+0000 7f26fa7fc640 1 -- 172.21.15.29:0/1969590041 <== mds.0 v2:172.21.15.152:6838/1548731833 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5583a4e872d0 con 0x5583a4f34a30 2024-03-16T11:52:56.899 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:56.899+0000 7f26fa7fc640 10 client.25423 handle_command_reply: tid=0 2024-03-16T11:52:56.899 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:56.899+0000 7f271c1bc740 1 client.25423 shutdown 2024-03-16T11:52:56.899 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:56.899+0000 7f271c1bc740 1 -- 172.21.15.29:0/1969590041 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5583a4f1b130 msgr2=0x5583a4f28a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:52:56.899 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:56.899+0000 7f271c1bc740 1 --2- 172.21.15.29:0/1969590041 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5583a4f1b130 0x5583a4f28a00 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f2704009430 tx=0x7f27040bc850 comp rx=0 tx=0).stop 2024-03-16T11:52:56.899 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:56.899+0000 7f271c1bc740 1 -- 172.21.15.29:0/1969590041 shutdown_connections 2024-03-16T11:52:56.899 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:56.899+0000 7f271c1bc740 1 --2- 172.21.15.29:0/1969590041 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5583a4f24820 0x5583a4f93dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:52:56.899 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:56.899+0000 7f271c1bc740 1 --2- 172.21.15.29:0/1969590041 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x5583a4f34a30 0x5583a4f9db00 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:52:56.900 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:56.899+0000 7f271c1bc740 1 --2- 172.21.15.29:0/1969590041 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5583a4f1ba50 0x5583a4f28f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:52:56.900 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:56.899+0000 7f271c1bc740 1 --2- 172.21.15.29:0/1969590041 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5583a4f1b130 0x5583a4f28a00 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:52:56.900 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:56.899+0000 7f271c1bc740 1 -- 172.21.15.29:0/1969590041 >> 172.21.15.29:0/1969590041 conn(0x5583a4f15a50 msgr2=0x5583a4f17730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:52:56.900 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:56.900+0000 7f26fa7fc640 0 client.25423 ms_handle_reset on v2:172.21.15.152:6838/1548731833 2024-03-16T11:52:56.900 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:56.900+0000 7f271c1bc740 1 -- 172.21.15.29:0/1969590041 shutdown_connections 2024-03-16T11:52:56.900 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:56.900+0000 7f271c1bc740 1 -- 172.21.15.29:0/1969590041 wait complete. 2024-03-16T11:52:56.900 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:56.900+0000 7f271c1bc740 20 client.25423 trim_cache size 0 max 16384 2024-03-16T11:52:56.901 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:{ 2024-03-16T11:52:56.901 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "message": "", 2024-03-16T11:52:56.901 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "return_code": 0 2024-03-16T11:52:56.901 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:} 2024-03-16T11:52:56.902 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:56.900+0000 7f27195cd640 1 -- 172.21.15.29:0/106249506 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f27000a2bb0 msgr2=0x7f27000a5070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:52:56.902 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:56.900+0000 7f27195cd640 1 --2- 172.21.15.29:0/106249506 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f27000a2bb0 0x7f27000a5070 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f27140ff230 tx=0x7f26fc005f90 comp rx=0 tx=0).stop 2024-03-16T11:52:56.902 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:56.900+0000 7f27195cd640 1 -- 172.21.15.29:0/106249506 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f27140ffce0 msgr2=0x7f27141980b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:52:56.902 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:56.900+0000 7f27195cd640 1 --2- 172.21.15.29:0/106249506 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f27140ffce0 0x7f27141980b0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f2704009230 tx=0x7f270402cb30 comp rx=0 tx=0).stop 2024-03-16T11:52:56.902 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:56.901+0000 7f27195cd640 1 -- 172.21.15.29:0/106249506 shutdown_connections 2024-03-16T11:52:56.902 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:56.901+0000 7f27195cd640 1 --2- 172.21.15.29:0/106249506 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f27000a2bb0 0x7f27000a5070 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:52:56.902 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:56.901+0000 7f27195cd640 1 --2- 172.21.15.29:0/106249506 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f27141067c0 0x7f271419c920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:52:56.902 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:56.901+0000 7f27195cd640 1 --2- 172.21.15.29:0/106249506 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f27141006a0 0x7f27141985f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:52:56.902 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:56.901+0000 7f27195cd640 1 --2- 172.21.15.29:0/106249506 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f27140ffce0 0x7f27141980b0 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:52:56.902 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:56.901+0000 7f27195cd640 1 -- 172.21.15.29:0/106249506 >> 172.21.15.29:0/106249506 conn(0x7f2714076c70 msgr2=0x7f27141047b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:52:56.902 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:56.901+0000 7f27195cd640 1 -- 172.21.15.29:0/106249506 shutdown_connections 2024-03-16T11:52:56.902 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:52:56.901+0000 7f27195cd640 1 -- 172.21.15.29:0/106249506 wait complete. 2024-03-16T11:52:57.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:52:56 smithi154 ceph-mon[36320]: pgmap v63: 97 pgs: 97 active+clean; 34 MiB data, 502 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 7.8 MiB/s wr, 447 op/s 2024-03-16T11:52:57.776 INFO:tasks.workunit.client.0.smithi029.stdout: 1 99826 13.25 MB/sec warmup 42 sec latency 36.718 ms 2024-03-16T11:52:58.776 INFO:tasks.workunit.client.0.smithi029.stdout: 1 102554 13.25 MB/sec warmup 43 sec latency 14.644 ms 2024-03-16T11:52:58.837 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:52:58 smithi029 ceph-mon[31975]: pgmap v64: 97 pgs: 97 active+clean; 39 MiB data, 484 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 8.3 MiB/s wr, 529 op/s 2024-03-16T11:52:58.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:52:58 smithi152 ceph-mon[36291]: pgmap v64: 97 pgs: 97 active+clean; 39 MiB data, 484 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 8.3 MiB/s wr, 529 op/s 2024-03-16T11:52:59.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:52:58 smithi154 ceph-mon[36320]: pgmap v64: 97 pgs: 97 active+clean; 39 MiB data, 484 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 8.3 MiB/s wr, 529 op/s 2024-03-16T11:52:59.776 INFO:tasks.workunit.client.0.smithi029.stdout: 1 104398 13.20 MB/sec warmup 44 sec latency 14.594 ms 2024-03-16T11:53:00.776 INFO:tasks.workunit.client.0.smithi029.stdout: 1 106976 13.22 MB/sec warmup 45 sec latency 14.837 ms 2024-03-16T11:53:00.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:53:00 smithi029 ceph-mon[31975]: pgmap v65: 97 pgs: 97 active+clean; 39 MiB data, 484 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 7.7 MiB/s wr, 329 op/s 2024-03-16T11:53:00.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:53:00 smithi152 ceph-mon[36291]: pgmap v65: 97 pgs: 97 active+clean; 39 MiB data, 484 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 7.7 MiB/s wr, 329 op/s 2024-03-16T11:53:01.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:53:00 smithi154 ceph-mon[36320]: pgmap v65: 97 pgs: 97 active+clean; 39 MiB data, 484 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 7.7 MiB/s wr, 329 op/s 2024-03-16T11:53:01.776 INFO:tasks.workunit.client.0.smithi029.stdout: 1 108835 13.08 MB/sec warmup 46 sec latency 27.283 ms 2024-03-16T11:53:01.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:53:01 smithi029 ceph-mon[31975]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T11:53:01.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:53:01 smithi152 ceph-mon[36291]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T11:53:02.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:53:01 smithi154 ceph-mon[36320]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T11:53:02.776 INFO:tasks.workunit.client.0.smithi029.stdout: 1 110620 13.06 MB/sec warmup 47 sec latency 19.743 ms 2024-03-16T11:53:02.837 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:53:02 smithi029 ceph-mon[31975]: pgmap v66: 97 pgs: 97 active+clean; 33 MiB data, 451 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 8.3 MiB/s wr, 491 op/s 2024-03-16T11:53:02.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:53:02 smithi029 ceph-mon[31975]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' 2024-03-16T11:53:02.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:53:02 smithi029 ceph-mon[31975]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' 2024-03-16T11:53:02.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:53:02 smithi152 ceph-mon[36291]: pgmap v66: 97 pgs: 97 active+clean; 33 MiB data, 451 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 8.3 MiB/s wr, 491 op/s 2024-03-16T11:53:02.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:53:02 smithi152 ceph-mon[36291]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' 2024-03-16T11:53:02.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:53:02 smithi152 ceph-mon[36291]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' 2024-03-16T11:53:03.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:53:02 smithi154 ceph-mon[36320]: pgmap v66: 97 pgs: 97 active+clean; 33 MiB data, 451 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 8.3 MiB/s wr, 491 op/s 2024-03-16T11:53:03.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:53:02 smithi154 ceph-mon[36320]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' 2024-03-16T11:53:03.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:53:02 smithi154 ceph-mon[36320]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' 2024-03-16T11:53:03.776 INFO:tasks.workunit.client.0.smithi029.stdout: 1 113604 13.16 MB/sec warmup 48 sec latency 22.128 ms 2024-03-16T11:53:03.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:53:03 smithi029 ceph-mon[31975]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:53:03.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:53:03 smithi029 ceph-mon[31975]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T11:53:03.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:53:03 smithi029 ceph-mon[31975]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' 2024-03-16T11:53:03.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:53:03 smithi152 ceph-mon[36291]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:53:03.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:53:03 smithi152 ceph-mon[36291]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T11:53:03.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:53:03 smithi152 ceph-mon[36291]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' 2024-03-16T11:53:04.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:53:03 smithi154 ceph-mon[36320]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:53:04.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:53:03 smithi154 ceph-mon[36320]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T11:53:04.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:53:03 smithi154 ceph-mon[36320]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' 2024-03-16T11:53:04.776 INFO:tasks.workunit.client.0.smithi029.stdout: 1 115318 13.03 MB/sec warmup 49 sec latency 21.066 ms 2024-03-16T11:53:04.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:53:04 smithi152 ceph-mon[36291]: pgmap v67: 97 pgs: 97 active+clean; 33 MiB data, 451 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 1.2 MiB/s wr, 243 op/s 2024-03-16T11:53:05.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:53:04 smithi154 ceph-mon[36320]: pgmap v67: 97 pgs: 97 active+clean; 33 MiB data, 451 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 1.2 MiB/s wr, 243 op/s 2024-03-16T11:53:05.087 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:53:04 smithi029 ceph-mon[31975]: pgmap v67: 97 pgs: 97 active+clean; 33 MiB data, 451 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 1.2 MiB/s wr, 243 op/s 2024-03-16T11:53:05.776 INFO:tasks.workunit.client.0.smithi029.stdout: 1 117652 13.04 MB/sec warmup 50 sec latency 17.035 ms 2024-03-16T11:53:06.776 INFO:tasks.workunit.client.0.smithi029.stdout: 1 120462 13.07 MB/sec warmup 51 sec latency 22.021 ms 2024-03-16T11:53:06.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:53:06 smithi152 ceph-mon[36291]: pgmap v68: 97 pgs: 97 active+clean; 33 MiB data, 451 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 1.2 MiB/s wr, 243 op/s 2024-03-16T11:53:07.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:53:06 smithi154 ceph-mon[36320]: pgmap v68: 97 pgs: 97 active+clean; 33 MiB data, 451 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 1.2 MiB/s wr, 243 op/s 2024-03-16T11:53:07.087 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:53:06 smithi029 ceph-mon[31975]: pgmap v68: 97 pgs: 97 active+clean; 33 MiB data, 451 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 1.2 MiB/s wr, 243 op/s 2024-03-16T11:53:07.776 INFO:tasks.workunit.client.0.smithi029.stdout: 1 122255 13.01 MB/sec warmup 52 sec latency 21.257 ms 2024-03-16T11:53:08.776 INFO:tasks.workunit.client.0.smithi029.stdout: 1 124649 13.02 MB/sec warmup 53 sec latency 18.400 ms 2024-03-16T11:53:08.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:53:08 smithi152 ceph-mon[36291]: pgmap v69: 97 pgs: 97 active+clean; 38 MiB data, 515 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 1.8 MiB/s wr, 337 op/s 2024-03-16T11:53:09.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:53:08 smithi154 ceph-mon[36320]: pgmap v69: 97 pgs: 97 active+clean; 38 MiB data, 515 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 1.8 MiB/s wr, 337 op/s 2024-03-16T11:53:09.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:53:08 smithi029 ceph-mon[31975]: pgmap v69: 97 pgs: 97 active+clean; 38 MiB data, 515 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 1.8 MiB/s wr, 337 op/s 2024-03-16T11:53:09.776 INFO:tasks.workunit.client.0.smithi029.stdout: 1 127556 13.05 MB/sec warmup 54 sec latency 18.883 ms 2024-03-16T11:53:09.989 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:09.990+0000 7f9d81159640 1 -- 172.21.15.29:0/4281953976 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9d7c06a9e0 msgr2=0x7f9d7c10e660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:53:09.990 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:09.990+0000 7f9d81159640 1 --2- 172.21.15.29:0/4281953976 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9d7c06a9e0 0x7f9d7c10e660 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f9d74009930 tx=0x7f9d7402bc30 comp rx=0 tx=0).stop 2024-03-16T11:53:09.990 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:09.990+0000 7f9d81159640 1 -- 172.21.15.29:0/4281953976 shutdown_connections 2024-03-16T11:53:09.990 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:09.990+0000 7f9d81159640 1 --2- 172.21.15.29:0/4281953976 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9d7c06a9e0 0x7f9d7c10e660 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:53:09.990 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:09.990+0000 7f9d81159640 1 --2- 172.21.15.29:0/4281953976 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9d7c06a020 0x7f9d7c06a4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:53:09.990 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:09.990+0000 7f9d81159640 1 --2- 172.21.15.29:0/4281953976 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9d7c1077a0 0x7f9d7c107b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:53:09.990 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:09.990+0000 7f9d81159640 1 -- 172.21.15.29:0/4281953976 >> 172.21.15.29:0/4281953976 conn(0x7f9d7c077400 msgr2=0x7f9d7c077810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:53:09.990 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:09.990+0000 7f9d81159640 1 -- 172.21.15.29:0/4281953976 shutdown_connections 2024-03-16T11:53:09.990 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:09.990+0000 7f9d81159640 1 -- 172.21.15.29:0/4281953976 wait complete. 2024-03-16T11:53:09.990 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:09.991+0000 7f9d81159640 1 Processor -- start 2024-03-16T11:53:09.992 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:09.991+0000 7f9d81159640 1 -- start start 2024-03-16T11:53:09.992 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:09.991+0000 7f9d81159640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9d7c06a020 0x7f9d7c197f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:53:09.992 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:09.991+0000 7f9d81159640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9d7c06a9e0 0x7f9d7c1984b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:53:09.992 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:09.991+0000 7f9d81159640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9d7c1077a0 0x7f9d7c19c830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:53:09.992 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:09.991+0000 7f9d81159640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f9d7c111bb0 con 0x7f9d7c06a020 2024-03-16T11:53:09.992 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:09.991+0000 7f9d81159640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f9d7c111a30 con 0x7f9d7c1077a0 2024-03-16T11:53:09.993 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:09.991+0000 7f9d81159640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f9d7c111d30 con 0x7f9d7c06a9e0 2024-03-16T11:53:09.993 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:09.991+0000 7f9d7ad9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9d7c06a020 0x7f9d7c197f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:53:09.993 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:09.991+0000 7f9d7ad9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9d7c06a020 0x7f9d7c197f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53458/0 (socket says 172.21.15.29:53458) 2024-03-16T11:53:09.993 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:09.991+0000 7f9d7ad9d640 1 -- 172.21.15.29:0/843891525 learned_addr learned my addr 172.21.15.29:0/843891525 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:53:09.993 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:09.991+0000 7f9d7a59c640 1 --2- 172.21.15.29:0/843891525 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9d7c06a9e0 0x7f9d7c1984b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:53:09.993 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:09.991+0000 7f9d7b59e640 1 --2- 172.21.15.29:0/843891525 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9d7c1077a0 0x7f9d7c19c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:53:09.993 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:09.991+0000 7f9d7ad9d640 1 -- 172.21.15.29:0/843891525 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9d7c06a9e0 msgr2=0x7f9d7c1984b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:53:09.993 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:09.991+0000 7f9d7ad9d640 1 --2- 172.21.15.29:0/843891525 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9d7c06a9e0 0x7f9d7c1984b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:53:09.993 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:09.991+0000 7f9d7ad9d640 1 -- 172.21.15.29:0/843891525 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9d7c1077a0 msgr2=0x7f9d7c19c830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:53:09.993 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:09.991+0000 7f9d7ad9d640 1 --2- 172.21.15.29:0/843891525 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9d7c1077a0 0x7f9d7c19c830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:53:09.993 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:09.991+0000 7f9d7ad9d640 1 -- 172.21.15.29:0/843891525 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d7c19cf60 con 0x7f9d7c06a020 2024-03-16T11:53:09.993 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:09.991+0000 7f9d7a59c640 1 --2- 172.21.15.29:0/843891525 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9d7c06a9e0 0x7f9d7c1984b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:53:09.993 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:09.992+0000 7f9d7ad9d640 1 --2- 172.21.15.29:0/843891525 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9d7c06a020 0x7f9d7c197f70 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f9d6c00ebc0 tx=0x7f9d6c00c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:53:09.993 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:09.992+0000 7f9d63fff640 1 -- 172.21.15.29:0/843891525 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d6c010630 con 0x7f9d7c06a020 2024-03-16T11:53:09.994 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:09.992+0000 7f9d81159640 1 -- 172.21.15.29:0/843891525 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9d7c19d250 con 0x7f9d7c06a020 2024-03-16T11:53:09.994 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:09.992+0000 7f9d63fff640 1 -- 172.21.15.29:0/843891525 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9d6c010cb0 con 0x7f9d7c06a020 2024-03-16T11:53:09.994 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:09.992+0000 7f9d63fff640 1 -- 172.21.15.29:0/843891525 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d6c018bd0 con 0x7f9d7c06a020 2024-03-16T11:53:09.994 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:09.992+0000 7f9d81159640 1 -- 172.21.15.29:0/843891525 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9d7c1a4ae0 con 0x7f9d7c06a020 2024-03-16T11:53:10.000 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:09.994+0000 7f9d63fff640 1 -- 172.21.15.29:0/843891525 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7f9d6c0107d0 con 0x7f9d7c06a020 2024-03-16T11:53:10.000 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:09.995+0000 7f9d83e16740 1 Processor -- start 2024-03-16T11:53:10.000 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:09.995+0000 7f9d63fff640 1 --2- 172.21.15.29:0/843891525 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f9d640a2a10 0x7f9d640a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:53:10.000 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:09.995+0000 7f9d83e16740 1 -- start start 2024-03-16T11:53:10.000 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:09.995+0000 7f9d83e16740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563ddc6c6130 0x563ddc6c6510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:53:10.001 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:09.995+0000 7f9d83e16740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563ddc6c6a50 0x563ddc6cf2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:53:10.001 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:09.995+0000 7f9d7b59e640 1 --2- 172.21.15.29:0/843891525 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f9d640a2a10 0x7f9d640a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:53:10.001 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:09.995+0000 7f9d83e16740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563ddc6cf820 0x563ddc6d1c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:53:10.001 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:09.995+0000 7f9d83e16740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x563ddc62aa40 con 0x563ddc6c6a50 2024-03-16T11:53:10.001 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:09.995+0000 7f9d83e16740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x563ddc5d3920 con 0x563ddc6c6130 2024-03-16T11:53:10.001 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:09.995+0000 7f9d83e16740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x563ddc626ae0 con 0x563ddc6cf820 2024-03-16T11:53:10.001 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:09.995+0000 7f9d63fff640 1 -- 172.21.15.29:0/843891525 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8469+0+0 (secure 0 0 0) 0x7f9d6c0cf430 con 0x7f9d7c06a020 2024-03-16T11:53:10.001 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:09.995+0000 7f9d7a59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563ddc6c6a50 0x563ddc6cf2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:53:10.001 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:09.995+0000 7f9d7ad9d640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563ddc6c6130 0x563ddc6c6510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:53:10.001 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:09.995+0000 7f9d7a59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563ddc6c6a50 0x563ddc6cf2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53462/0 (socket says 172.21.15.29:53462) 2024-03-16T11:53:10.001 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:09.995+0000 7f9d7a59c640 1 -- 172.21.15.29:0/543215784 learned_addr learned my addr 172.21.15.29:0/543215784 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:53:10.002 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:09.995+0000 7f9d7b59e640 1 --2- 172.21.15.29:0/543215784 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563ddc6cf820 0x563ddc6d1c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:53:10.002 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:09.995+0000 7f9d7a59c640 1 -- 172.21.15.29:0/543215784 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563ddc6cf820 msgr2=0x563ddc6d1c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:53:10.002 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:09.995+0000 7f9d7a59c640 1 --2- 172.21.15.29:0/543215784 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563ddc6cf820 0x563ddc6d1c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:53:10.002 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:09.995+0000 7f9d7a59c640 1 -- 172.21.15.29:0/543215784 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563ddc6c6130 msgr2=0x563ddc6c6510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:53:10.002 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:09.995+0000 7f9d7a59c640 1 --2- 172.21.15.29:0/543215784 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563ddc6c6130 0x563ddc6c6510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:53:10.002 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:09.995+0000 7f9d7a59c640 1 -- 172.21.15.29:0/543215784 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563ddc5eb460 con 0x563ddc6c6a50 2024-03-16T11:53:10.002 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:09.995+0000 7f9d7ad9d640 1 --2- 172.21.15.29:0/543215784 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563ddc6c6130 0x563ddc6c6510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:53:10.002 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:09.995+0000 7f9d7b59e640 1 --2- 172.21.15.29:0/843891525 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f9d640a2a10 0x7f9d640a4ed0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f9d74005940 tx=0x7f9d7402c980 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:53:10.002 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:09.995+0000 7f9d7a59c640 1 --2- 172.21.15.29:0/543215784 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563ddc6c6a50 0x563ddc6cf2e0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f9d700114f0 tx=0x7f9d700119b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c59964d49cf33e1a server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:53:10.002 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:09.996+0000 7f9d617fa640 1 -- 172.21.15.29:0/543215784 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d70004020 con 0x563ddc6c6a50 2024-03-16T11:53:10.002 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:09.996+0000 7f9d617fa640 1 -- 172.21.15.29:0/543215784 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9d700041c0 con 0x563ddc6c6a50 2024-03-16T11:53:10.002 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:09.996+0000 7f9d617fa640 1 -- 172.21.15.29:0/543215784 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d7000f8b0 con 0x563ddc6c6a50 2024-03-16T11:53:10.003 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:09.996+0000 7f9d83e16740 1 -- 172.21.15.29:0/543215784 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563ddc6c6a50 msgr2=0x563ddc6cf2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:53:10.003 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:09.996+0000 7f9d83e16740 1 --2- 172.21.15.29:0/543215784 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563ddc6c6a50 0x563ddc6cf2e0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f9d700114f0 tx=0x7f9d700119b0 comp rx=0 tx=0).stop 2024-03-16T11:53:10.003 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:09.996+0000 7f9d83e16740 1 -- 172.21.15.29:0/543215784 shutdown_connections 2024-03-16T11:53:10.003 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:09.996+0000 7f9d83e16740 1 --2- 172.21.15.29:0/543215784 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563ddc6cf820 0x563ddc6d1c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:53:10.003 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:09.996+0000 7f9d83e16740 1 --2- 172.21.15.29:0/543215784 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563ddc6c6a50 0x563ddc6cf2e0 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:53:10.003 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:09.996+0000 7f9d83e16740 1 --2- 172.21.15.29:0/543215784 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563ddc6c6130 0x563ddc6c6510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:53:10.003 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:09.996+0000 7f9d83e16740 1 -- 172.21.15.29:0/543215784 >> 172.21.15.29:0/543215784 conn(0x563ddc6c0a50 msgr2=0x563ddc6c0e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:53:10.003 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:09.996+0000 7f9d83e16740 1 -- 172.21.15.29:0/543215784 shutdown_connections 2024-03-16T11:53:10.003 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:09.996+0000 7f9d83e16740 1 -- 172.21.15.29:0/543215784 wait complete. 2024-03-16T11:53:10.003 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:09.997+0000 7f9d83e16740 1 -- start start 2024-03-16T11:53:10.003 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:09.997+0000 7f9d83e16740 1 Processor -- start 2024-03-16T11:53:10.004 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:09.997+0000 7f9d83e16740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563ddc6c6130 0x563ddc6cc830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:53:10.004 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:09.997+0000 7f9d83e16740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563ddc6c6a50 0x563ddc6ccd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:53:10.004 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:09.997+0000 7f9d83e16740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563ddc6cf820 0x563ddc6cd2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:53:10.004 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:09.997+0000 7f9d83e16740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x563ddc6d6860 con 0x563ddc6c6130 2024-03-16T11:53:10.004 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:09.997+0000 7f9d83e16740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x563ddc5e6d70 con 0x563ddc6cf820 2024-03-16T11:53:10.004 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:09.997+0000 7f9d83e16740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x563ddc6d69e0 con 0x563ddc6c6a50 2024-03-16T11:53:10.004 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:09.997+0000 7f9d83e16740 10 client.0 _ll_register_callbacks cb 0x563ddc33ea00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T11:53:10.004 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:09.997+0000 7f9d7ad9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563ddc6c6130 0x563ddc6cc830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:53:10.004 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:09.997+0000 7f9d7ad9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563ddc6c6130 0x563ddc6cc830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53474/0 (socket says 172.21.15.29:53474) 2024-03-16T11:53:10.004 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:09.997+0000 7f9d7ad9d640 1 -- 172.21.15.29:0/3629890919 learned_addr learned my addr 172.21.15.29:0/3629890919 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:53:10.004 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:09.997+0000 7f9d7a59c640 1 --2- 172.21.15.29:0/3629890919 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563ddc6c6a50 0x563ddc6ccd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:53:10.004 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:09.997+0000 7f9d7b59e640 1 --2- 172.21.15.29:0/3629890919 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563ddc6cf820 0x563ddc6cd2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:53:10.005 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:09.997+0000 7f9d7ad9d640 1 -- 172.21.15.29:0/3629890919 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563ddc6c6a50 msgr2=0x563ddc6ccd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:53:10.005 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:09.997+0000 7f9d7ad9d640 1 --2- 172.21.15.29:0/3629890919 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563ddc6c6a50 0x563ddc6ccd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:53:10.005 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:09.997+0000 7f9d7ad9d640 1 -- 172.21.15.29:0/3629890919 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563ddc6cf820 msgr2=0x563ddc6cd2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:53:10.005 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:09.997+0000 7f9d7ad9d640 1 --2- 172.21.15.29:0/3629890919 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563ddc6cf820 0x563ddc6cd2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:53:10.005 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:09.997+0000 7f9d7ad9d640 1 -- 172.21.15.29:0/3629890919 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563ddc33f210 con 0x563ddc6c6130 2024-03-16T11:53:10.005 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:09.998+0000 7f9d7ad9d640 1 --2- 172.21.15.29:0/3629890919 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563ddc6c6130 0x563ddc6cc830 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f9d6c00eb90 tx=0x7f9d6c09b770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7cdaddc816c37dc6 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:53:10.005 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:09.998+0000 7f9d5eff5640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-16T11:53:10.005 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:09.998+0000 7f9d5eff5640 1 -- 172.21.15.29:0/3629890919 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d6c010e60 con 0x563ddc6c6130 2024-03-16T11:53:10.005 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:09.998+0000 7f9d5eff5640 1 -- 172.21.15.29:0/3629890919 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8469+0+0 (secure 0 0 0) 0x7f9d6c0ccdd0 con 0x563ddc6c6130 2024-03-16T11:53:10.005 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:09.998+0000 7f9d83e16740 1 -- 172.21.15.29:0/3629890919 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9d700041c0 con 0x563ddc6c6130 2024-03-16T11:53:10.005 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:09.998+0000 7f9d5eff5640 1 -- 172.21.15.29:0/3629890919 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f9d6c0ccfd0 con 0x563ddc6c6130 2024-03-16T11:53:10.005 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:09.998+0000 7f9d83e16740 10 client.15588 fetch_fsmap learned FSMap version 22 2024-03-16T11:53:10.005 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:09.998+0000 7f9d83e16740 1 -- 172.21.15.29:0/3629890919 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x563ddc40b620 con 0x563ddc6c6130 2024-03-16T11:53:10.005 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:09.998+0000 7f9d5eff5640 1 -- 172.21.15.29:0/3629890919 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9d700041c0 con 0x563ddc6c6130 2024-03-16T11:53:10.005 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:09.998+0000 7f9d5eff5640 1 -- 172.21.15.29:0/3629890919 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7f9d6c05fce0 con 0x563ddc6c6130 2024-03-16T11:53:10.006 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:09.998+0000 7f9d83e16740 10 client.15588 fetch_fsmap finished waiting for FSMap version 22 2024-03-16T11:53:10.006 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:09.998+0000 7f9d83e16740 10 client.15588 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T11:53:10.006 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:09.998+0000 7f9d83e16740 20 client.15588 populate_metadata read hostname 'smithi029' 2024-03-16T11:53:10.006 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:09.998+0000 7f9d83e16740 1 --2- 172.21.15.29:0/3629890919 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x563ddc6e3d90 0x563ddc6e6180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:53:10.006 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:09.998+0000 7f9d83e16740 4 client.15588 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T11:53:10.006 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:09.998+0000 7f9d83e16740 1 -- 172.21.15.29:0/3629890919 --> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563ddc637f80 con 0x563ddc6e3d90 2024-03-16T11:53:10.006 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:09.999+0000 7f9d7a59c640 1 --2- 172.21.15.29:0/3629890919 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x563ddc6e3d90 0x563ddc6e6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:53:10.006 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:09.999+0000 7f9d5eff5640 10 client.15588 ms_handle_connect on v2:172.21.15.152:6838/1548731833 2024-03-16T11:53:10.006 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:09.999+0000 7f9d7a59c640 1 --2- 172.21.15.29:0/3629890919 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x563ddc6e3d90 0x563ddc6e6180 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c071d8e562e93715 server_cookie=eae2f58b0944de40 in_seq=0 out_seq=0 2024-03-16T11:53:10.006 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:10.000+0000 7f9d5eff5640 1 -- 172.21.15.29:0/3629890919 <== mds.0 v2:172.21.15.152:6838/1548731833 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x563ddc637f80 con 0x563ddc6e3d90 2024-03-16T11:53:10.006 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:10.000+0000 7f9d5eff5640 10 client.15588 handle_command_reply: tid=0 2024-03-16T11:53:10.006 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:10.000+0000 7f9d83e16740 1 client.15588 shutdown 2024-03-16T11:53:10.006 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:10.000+0000 7f9d83e16740 1 -- 172.21.15.29:0/3629890919 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563ddc6c6130 msgr2=0x563ddc6cc830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:53:10.007 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:10.000+0000 7f9d83e16740 1 --2- 172.21.15.29:0/3629890919 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563ddc6c6130 0x563ddc6cc830 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f9d6c00eb90 tx=0x7f9d6c09b770 comp rx=0 tx=0).stop 2024-03-16T11:53:10.007 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:10.000+0000 7f9d83e16740 1 -- 172.21.15.29:0/3629890919 shutdown_connections 2024-03-16T11:53:10.007 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:10.000+0000 7f9d83e16740 1 --2- 172.21.15.29:0/3629890919 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x563ddc6e3d90 0x563ddc6e6180 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:53:10.007 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:10.000+0000 7f9d83e16740 1 --2- 172.21.15.29:0/3629890919 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563ddc6cf820 0x563ddc6cd2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:53:10.007 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:10.000+0000 7f9d83e16740 1 --2- 172.21.15.29:0/3629890919 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563ddc6c6a50 0x563ddc6ccd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:53:10.007 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:10.000+0000 7f9d5eff5640 0 client.15588 ms_handle_reset on v2:172.21.15.152:6838/1548731833 2024-03-16T11:53:10.007 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:10.000+0000 7f9d83e16740 1 --2- 172.21.15.29:0/3629890919 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563ddc6c6130 0x563ddc6cc830 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:53:10.007 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:10.000+0000 7f9d83e16740 1 -- 172.21.15.29:0/3629890919 >> 172.21.15.29:0/3629890919 conn(0x563ddc6c0a50 msgr2=0x563ddc6c27a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:53:10.007 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:10.000+0000 7f9d83e16740 1 -- 172.21.15.29:0/3629890919 shutdown_connections 2024-03-16T11:53:10.008 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:10.000+0000 7f9d83e16740 1 -- 172.21.15.29:0/3629890919 wait complete. 2024-03-16T11:53:10.008 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:10.001+0000 7f9d83e16740 20 client.15588 trim_cache size 0 max 16384 2024-03-16T11:53:10.019 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:10.015+0000 7f9d83e16740 1 Processor -- start 2024-03-16T11:53:10.019 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:10.015+0000 7f9d83e16740 1 -- start start 2024-03-16T11:53:10.020 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:10.015+0000 7f9d83e16740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563ddc6c6130 0x563ddc6cef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:53:10.020 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:10.015+0000 7f9d83e16740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563ddc6c6a50 0x563ddc6df600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:53:10.020 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:10.015+0000 7f9d83e16740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563ddc6cf820 0x563ddc6eba90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:53:10.020 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:10.015+0000 7f9d83e16740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f9d7c1ada10 con 0x563ddc6cf820 2024-03-16T11:53:10.020 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:10.015+0000 7f9d83e16740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f9d7c1ab2e0 con 0x563ddc6c6a50 2024-03-16T11:53:10.020 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:10.015+0000 7f9d83e16740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f9d7c1a9130 con 0x563ddc6c6130 2024-03-16T11:53:10.020 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:10.016+0000 7f9d7b59e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563ddc6cf820 0x563ddc6eba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:53:10.020 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:10.016+0000 7f9d7b59e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563ddc6cf820 0x563ddc6eba90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53476/0 (socket says 172.21.15.29:53476) 2024-03-16T11:53:10.020 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:10.016+0000 7f9d7b59e640 1 -- 172.21.15.29:0/2350599105 learned_addr learned my addr 172.21.15.29:0/2350599105 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:53:10.020 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:10.016+0000 7f9d7ad9d640 1 --2- 172.21.15.29:0/2350599105 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563ddc6c6130 0x563ddc6cef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:53:10.020 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:10.016+0000 7f9d7a59c640 1 --2- 172.21.15.29:0/2350599105 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563ddc6c6a50 0x563ddc6df600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:53:10.020 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:10.016+0000 7f9d7b59e640 1 -- 172.21.15.29:0/2350599105 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563ddc6c6130 msgr2=0x563ddc6cef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:53:10.021 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:10.016+0000 7f9d7b59e640 1 --2- 172.21.15.29:0/2350599105 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563ddc6c6130 0x563ddc6cef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:53:10.021 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:10.016+0000 7f9d7b59e640 1 -- 172.21.15.29:0/2350599105 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563ddc6c6a50 msgr2=0x563ddc6df600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:53:10.021 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:10.016+0000 7f9d7b59e640 1 --2- 172.21.15.29:0/2350599105 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563ddc6c6a50 0x563ddc6df600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:53:10.021 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:10.016+0000 7f9d7b59e640 1 -- 172.21.15.29:0/2350599105 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563ddc6dfb40 con 0x563ddc6cf820 2024-03-16T11:53:10.021 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:10.016+0000 7f9d7b59e640 1 --2- 172.21.15.29:0/2350599105 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563ddc6cf820 0x563ddc6eba90 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f9d74009930 tx=0x7f9d7402e050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a7531743e2f5661 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:53:10.021 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:10.016+0000 7f9d60ff9640 1 -- 172.21.15.29:0/2350599105 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d740048c0 con 0x563ddc6cf820 2024-03-16T11:53:10.021 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:10.016+0000 7f9d60ff9640 1 -- 172.21.15.29:0/2350599105 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9d74051070 con 0x563ddc6cf820 2024-03-16T11:53:10.021 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:10.016+0000 7f9d60ff9640 1 -- 172.21.15.29:0/2350599105 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d74003850 con 0x563ddc6cf820 2024-03-16T11:53:10.021 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:10.017+0000 7f9d83e16740 1 -- 172.21.15.29:0/2350599105 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563ddc6cf820 msgr2=0x563ddc6eba90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:53:10.021 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:10.017+0000 7f9d83e16740 1 --2- 172.21.15.29:0/2350599105 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563ddc6cf820 0x563ddc6eba90 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f9d74009930 tx=0x7f9d7402e050 comp rx=0 tx=0).stop 2024-03-16T11:53:10.021 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:10.017+0000 7f9d83e16740 1 -- 172.21.15.29:0/2350599105 shutdown_connections 2024-03-16T11:53:10.021 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:10.017+0000 7f9d83e16740 1 --2- 172.21.15.29:0/2350599105 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563ddc6cf820 0x563ddc6eba90 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:53:10.021 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:10.017+0000 7f9d83e16740 1 --2- 172.21.15.29:0/2350599105 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563ddc6c6a50 0x563ddc6df600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:53:10.021 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:10.017+0000 7f9d83e16740 1 --2- 172.21.15.29:0/2350599105 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563ddc6c6130 0x563ddc6cef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:53:10.022 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:10.017+0000 7f9d83e16740 1 -- 172.21.15.29:0/2350599105 >> 172.21.15.29:0/2350599105 conn(0x563ddc6c0a50 msgr2=0x563ddc6cc7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:53:10.022 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:10.017+0000 7f9d83e16740 1 -- 172.21.15.29:0/2350599105 shutdown_connections 2024-03-16T11:53:10.022 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:10.017+0000 7f9d83e16740 1 -- 172.21.15.29:0/2350599105 wait complete. 2024-03-16T11:53:10.022 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:10.017+0000 7f9d83e16740 1 -- start start 2024-03-16T11:53:10.022 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:10.017+0000 7f9d83e16740 1 Processor -- start 2024-03-16T11:53:10.022 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:10.018+0000 7f9d83e16740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563ddc6c6130 0x563ddc6dbc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:53:10.022 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:10.018+0000 7f9d83e16740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563ddc6c6a50 0x563ddc6d8c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:53:10.022 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:10.018+0000 7f9d83e16740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563ddc6cf820 0x563ddc6d9140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:53:10.022 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:10.018+0000 7f9d83e16740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f9d7c1a97a0 con 0x563ddc6c6a50 2024-03-16T11:53:10.022 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:10.018+0000 7f9d83e16740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f9d7c1aac10 con 0x563ddc6c6130 2024-03-16T11:53:10.022 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:10.018+0000 7f9d83e16740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f9d7c1a9e70 con 0x563ddc6cf820 2024-03-16T11:53:10.022 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:10.018+0000 7f9d83e16740 10 client.0 _ll_register_callbacks cb 0x7f9d6c061050 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T11:53:10.022 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:10.018+0000 7f9d7a59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563ddc6c6a50 0x563ddc6d8c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:53:10.023 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:10.018+0000 7f9d7ad9d640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563ddc6c6130 0x563ddc6dbc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:53:10.023 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:10.018+0000 7f9d7b59e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563ddc6cf820 0x563ddc6d9140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:53:10.023 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:10.018+0000 7f9d7a59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563ddc6c6a50 0x563ddc6d8c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53492/0 (socket says 172.21.15.29:53492) 2024-03-16T11:53:10.023 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:10.018+0000 7f9d7ad9d640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563ddc6c6130 0x563ddc6dbc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.29:47794/0 (socket says 172.21.15.29:47794) 2024-03-16T11:53:10.023 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:10.018+0000 7f9d7a59c640 1 -- 172.21.15.29:0/1183799954 learned_addr learned my addr 172.21.15.29:0/1183799954 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:53:10.023 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:10.018+0000 7f9d7a59c640 1 -- 172.21.15.29:0/1183799954 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563ddc6cf820 msgr2=0x563ddc6d9140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:53:10.023 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:10.018+0000 7f9d7a59c640 1 --2- 172.21.15.29:0/1183799954 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563ddc6cf820 0x563ddc6d9140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:53:10.023 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:10.018+0000 7f9d7a59c640 1 -- 172.21.15.29:0/1183799954 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563ddc6c6130 msgr2=0x563ddc6dbc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:53:10.023 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:10.018+0000 7f9d7a59c640 1 --2- 172.21.15.29:0/1183799954 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563ddc6c6130 0x563ddc6dbc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:53:10.023 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:10.018+0000 7f9d7a59c640 1 -- 172.21.15.29:0/1183799954 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563ddc6dc340 con 0x563ddc6c6a50 2024-03-16T11:53:10.023 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:10.018+0000 7f9d7b59e640 1 --2- 172.21.15.29:0/1183799954 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563ddc6cf820 0x563ddc6d9140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:53:10.023 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:10.018+0000 7f9d7a59c640 1 --2- 172.21.15.29:0/1183799954 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563ddc6c6a50 0x563ddc6d8c00 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f9d700114f0 tx=0x7f9d70005700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4976d82a0cf94c18 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:53:10.023 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:10.018+0000 7f9d5eff5640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-16T11:53:10.024 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:10.018+0000 7f9d5eff5640 1 -- 172.21.15.29:0/1183799954 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d70009b30 con 0x563ddc6c6a50 2024-03-16T11:53:10.024 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:10.018+0000 7f9d5eff5640 1 -- 172.21.15.29:0/1183799954 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8469+0+0 (secure 0 0 0) 0x7f9d7001f040 con 0x563ddc6c6a50 2024-03-16T11:53:10.024 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:10.018+0000 7f9d83e16740 1 -- 172.21.15.29:0/1183799954 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9d74051070 con 0x563ddc6c6a50 2024-03-16T11:53:10.024 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:10.019+0000 7f9d5eff5640 1 -- 172.21.15.29:0/1183799954 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f9d70011040 con 0x563ddc6c6a50 2024-03-16T11:53:10.024 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:10.019+0000 7f9d83e16740 10 client.15594 fetch_fsmap learned FSMap version 22 2024-03-16T11:53:10.024 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:10.019+0000 7f9d83e16740 1 -- 172.21.15.29:0/1183799954 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x563ddc74abf0 con 0x563ddc6c6a50 2024-03-16T11:53:10.024 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:10.019+0000 7f9d5eff5640 1 -- 172.21.15.29:0/1183799954 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9d74051070 con 0x563ddc6c6a50 2024-03-16T11:53:10.024 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:10.019+0000 7f9d5eff5640 1 -- 172.21.15.29:0/1183799954 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7f9d7000f540 con 0x563ddc6c6a50 2024-03-16T11:53:10.024 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:10.019+0000 7f9d83e16740 10 client.15594 fetch_fsmap finished waiting for FSMap version 22 2024-03-16T11:53:10.024 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:10.019+0000 7f9d83e16740 10 client.15594 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T11:53:10.024 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:10.019+0000 7f9d83e16740 20 client.15594 populate_metadata read hostname 'smithi029' 2024-03-16T11:53:10.024 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:10.019+0000 7f9d83e16740 1 --2- 172.21.15.29:0/1183799954 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x563ddc74b210 0x563ddc74d630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:53:10.024 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:10.019+0000 7f9d83e16740 4 client.15594 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-16T11:53:10.025 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:10.019+0000 7f9d83e16740 1 -- 172.21.15.29:0/1183799954 --> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] -- command(tid 0: {"prefix": "flush journal"}) -- 0x563ddc6322d0 con 0x563ddc74b210 2024-03-16T11:53:10.025 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:10.019+0000 7f9d7ad9d640 1 --2- 172.21.15.29:0/1183799954 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x563ddc74b210 0x563ddc74d630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:53:10.025 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:10.020+0000 7f9d7ad9d640 1 --2- 172.21.15.29:0/1183799954 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x563ddc74b210 0x563ddc74d630 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=aff80be0cbca53bb server_cookie=d94af6c02bf48cd in_seq=0 out_seq=0 2024-03-16T11:53:10.025 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:10.020+0000 7f9d5eff5640 10 client.15594 ms_handle_connect on v2:172.21.15.152:6838/1548731833 2024-03-16T11:53:10.159 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:10.160+0000 7f9d5eff5640 1 -- 172.21.15.29:0/1183799954 <== mds.0 v2:172.21.15.152:6838/1548731833 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x563ddc6322d0 con 0x563ddc74b210 2024-03-16T11:53:10.160 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:10.160+0000 7f9d5eff5640 10 client.15594 handle_command_reply: tid=0 2024-03-16T11:53:10.160 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:10.160+0000 7f9d83e16740 1 client.15594 shutdown 2024-03-16T11:53:10.160 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:10.160+0000 7f9d83e16740 1 -- 172.21.15.29:0/1183799954 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563ddc6c6a50 msgr2=0x563ddc6d8c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:53:10.160 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:10.160+0000 7f9d83e16740 1 --2- 172.21.15.29:0/1183799954 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563ddc6c6a50 0x563ddc6d8c00 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f9d700114f0 tx=0x7f9d70005700 comp rx=0 tx=0).stop 2024-03-16T11:53:10.160 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:10.160+0000 7f9d83e16740 1 -- 172.21.15.29:0/1183799954 shutdown_connections 2024-03-16T11:53:10.160 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:10.160+0000 7f9d83e16740 1 --2- 172.21.15.29:0/1183799954 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563ddc6cf820 0x563ddc6d9140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:53:10.160 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:10.160+0000 7f9d83e16740 1 --2- 172.21.15.29:0/1183799954 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563ddc6c6a50 0x563ddc6d8c00 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:53:10.160 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:10.160+0000 7f9d83e16740 1 --2- 172.21.15.29:0/1183799954 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x563ddc74b210 0x563ddc74d630 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:53:10.160 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:10.160+0000 7f9d83e16740 1 --2- 172.21.15.29:0/1183799954 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563ddc6c6130 0x563ddc6dbc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:53:10.160 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:10.160+0000 7f9d5eff5640 0 client.15594 ms_handle_reset on v2:172.21.15.152:6838/1548731833 2024-03-16T11:53:10.160 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:10.160+0000 7f9d83e16740 1 -- 172.21.15.29:0/1183799954 >> 172.21.15.29:0/1183799954 conn(0x563ddc6c0a50 msgr2=0x563ddc6bfbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:53:10.160 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:10.160+0000 7f9d83e16740 1 -- 172.21.15.29:0/1183799954 shutdown_connections 2024-03-16T11:53:10.160 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:10.160+0000 7f9d83e16740 1 -- 172.21.15.29:0/1183799954 wait complete. 2024-03-16T11:53:10.161 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:10.160+0000 7f9d83e16740 20 client.15594 trim_cache size 0 max 16384 2024-03-16T11:53:10.161 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:{ 2024-03-16T11:53:10.161 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "message": "", 2024-03-16T11:53:10.161 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "return_code": 0 2024-03-16T11:53:10.161 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:} 2024-03-16T11:53:10.162 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:10.161+0000 7f9d81159640 1 -- 172.21.15.29:0/843891525 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f9d640a2a10 msgr2=0x7f9d640a4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:53:10.162 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:10.161+0000 7f9d81159640 1 --2- 172.21.15.29:0/843891525 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f9d640a2a10 0x7f9d640a4ed0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f9d74005940 tx=0x7f9d7402c980 comp rx=0 tx=0).stop 2024-03-16T11:53:10.162 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:10.161+0000 7f9d81159640 1 -- 172.21.15.29:0/843891525 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9d7c06a020 msgr2=0x7f9d7c197f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:53:10.162 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:10.161+0000 7f9d81159640 1 --2- 172.21.15.29:0/843891525 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9d7c06a020 0x7f9d7c197f70 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f9d6c00ebc0 tx=0x7f9d6c00c6a0 comp rx=0 tx=0).stop 2024-03-16T11:53:10.162 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:10.161+0000 7f9d81159640 1 -- 172.21.15.29:0/843891525 shutdown_connections 2024-03-16T11:53:10.162 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:10.161+0000 7f9d81159640 1 --2- 172.21.15.29:0/843891525 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f9d640a2a10 0x7f9d640a4ed0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:53:10.162 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:10.161+0000 7f9d81159640 1 --2- 172.21.15.29:0/843891525 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9d7c1077a0 0x7f9d7c19c830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:53:10.162 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:10.161+0000 7f9d81159640 1 --2- 172.21.15.29:0/843891525 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9d7c06a9e0 0x7f9d7c1984b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:53:10.163 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:10.161+0000 7f9d81159640 1 --2- 172.21.15.29:0/843891525 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9d7c06a020 0x7f9d7c197f70 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:53:10.163 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:10.161+0000 7f9d81159640 1 -- 172.21.15.29:0/843891525 >> 172.21.15.29:0/843891525 conn(0x7f9d7c077400 msgr2=0x7f9d7c0fe3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:53:10.163 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:10.161+0000 7f9d81159640 1 -- 172.21.15.29:0/843891525 shutdown_connections 2024-03-16T11:53:10.163 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:10.161+0000 7f9d81159640 1 -- 172.21.15.29:0/843891525 wait complete. 2024-03-16T11:53:10.776 INFO:tasks.workunit.client.0.smithi029.stdout: 1 128848 12.93 MB/sec warmup 55 sec latency 60.719 ms 2024-03-16T11:53:10.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:53:10 smithi152 ceph-mon[36291]: pgmap v70: 97 pgs: 97 active+clean; 38 MiB data, 515 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 1.3 MiB/s wr, 256 op/s 2024-03-16T11:53:11.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:53:10 smithi154 ceph-mon[36320]: pgmap v70: 97 pgs: 97 active+clean; 38 MiB data, 515 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 1.3 MiB/s wr, 256 op/s 2024-03-16T11:53:11.087 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:53:10 smithi029 ceph-mon[31975]: pgmap v70: 97 pgs: 97 active+clean; 38 MiB data, 515 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 1.3 MiB/s wr, 256 op/s 2024-03-16T11:53:11.776 INFO:tasks.workunit.client.0.smithi029.stdout: 1 131644 13.01 MB/sec warmup 56 sec latency 12.751 ms 2024-03-16T11:53:12.776 INFO:tasks.workunit.client.0.smithi029.stdout: 1 134036 12.97 MB/sec warmup 57 sec latency 15.798 ms 2024-03-16T11:53:12.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:53:12 smithi152 ceph-mon[36291]: pgmap v71: 97 pgs: 97 active+clean; 33 MiB data, 536 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 2.0 MiB/s wr, 439 op/s 2024-03-16T11:53:13.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:53:12 smithi154 ceph-mon[36320]: pgmap v71: 97 pgs: 97 active+clean; 33 MiB data, 536 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 2.0 MiB/s wr, 439 op/s 2024-03-16T11:53:13.087 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:53:12 smithi029 ceph-mon[31975]: pgmap v71: 97 pgs: 97 active+clean; 33 MiB data, 536 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 2.0 MiB/s wr, 439 op/s 2024-03-16T11:53:13.776 INFO:tasks.workunit.client.0.smithi029.stdout: 1 135861 12.92 MB/sec warmup 58 sec latency 19.646 ms 2024-03-16T11:53:14.776 INFO:tasks.workunit.client.0.smithi029.stdout: 1 138589 12.99 MB/sec warmup 59 sec latency 30.001 ms 2024-03-16T11:53:14.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:53:14 smithi152 ceph-mon[36291]: pgmap v72: 97 pgs: 97 active+clean; 33 MiB data, 536 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 1.4 MiB/s wr, 277 op/s 2024-03-16T11:53:15.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:53:14 smithi154 ceph-mon[36320]: pgmap v72: 97 pgs: 97 active+clean; 33 MiB data, 536 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 1.4 MiB/s wr, 277 op/s 2024-03-16T11:53:15.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:53:14 smithi029 ceph-mon[31975]: pgmap v72: 97 pgs: 97 active+clean; 33 MiB data, 536 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 1.4 MiB/s wr, 277 op/s 2024-03-16T11:53:15.777 INFO:tasks.workunit.client.0.smithi029.stdout: 1 140797 12.90 MB/sec warmup 60 sec latency 20.446 ms 2024-03-16T11:53:16.777 INFO:tasks.workunit.client.0.smithi029.stdout: 1 142577 12.89 MB/sec warmup 61 sec latency 64.710 ms 2024-03-16T11:53:16.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:53:16 smithi152 ceph-mon[36291]: pgmap v73: 97 pgs: 97 active+clean; 33 MiB data, 536 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 1.4 MiB/s wr, 277 op/s 2024-03-16T11:53:17.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:53:16 smithi154 ceph-mon[36320]: pgmap v73: 97 pgs: 97 active+clean; 33 MiB data, 536 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 1.4 MiB/s wr, 277 op/s 2024-03-16T11:53:17.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:53:16 smithi029 ceph-mon[31975]: pgmap v73: 97 pgs: 97 active+clean; 33 MiB data, 536 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 1.4 MiB/s wr, 277 op/s 2024-03-16T11:53:17.776 INFO:tasks.workunit.client.0.smithi029.stdout: 1 145533 12.96 MB/sec warmup 62 sec latency 23.385 ms 2024-03-16T11:53:18.776 INFO:tasks.workunit.client.0.smithi029.stdout: 1 147286 12.86 MB/sec warmup 63 sec latency 24.820 ms 2024-03-16T11:53:18.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:53:18 smithi152 ceph-mon[36291]: pgmap v74: 97 pgs: 97 active+clean; 42 MiB data, 546 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 2.2 MiB/s wr, 360 op/s 2024-03-16T11:53:19.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:53:18 smithi154 ceph-mon[36320]: pgmap v74: 97 pgs: 97 active+clean; 42 MiB data, 546 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 2.2 MiB/s wr, 360 op/s 2024-03-16T11:53:19.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:53:18 smithi029 ceph-mon[31975]: pgmap v74: 97 pgs: 97 active+clean; 42 MiB data, 546 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 2.2 MiB/s wr, 360 op/s 2024-03-16T11:53:19.777 INFO:tasks.workunit.client.0.smithi029.stdout: 1 149528 12.88 MB/sec warmup 64 sec latency 9.894 ms 2024-03-16T11:53:20.777 INFO:tasks.workunit.client.0.smithi029.stdout: 1 152301 12.90 MB/sec warmup 65 sec latency 20.853 ms 2024-03-16T11:53:20.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:53:20 smithi152 ceph-mon[36291]: pgmap v75: 97 pgs: 97 active+clean; 42 MiB data, 546 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 1.5 MiB/s wr, 266 op/s 2024-03-16T11:53:21.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:53:20 smithi154 ceph-mon[36320]: pgmap v75: 97 pgs: 97 active+clean; 42 MiB data, 546 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 1.5 MiB/s wr, 266 op/s 2024-03-16T11:53:21.087 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:53:20 smithi029 ceph-mon[31975]: pgmap v75: 97 pgs: 97 active+clean; 42 MiB data, 546 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 1.5 MiB/s wr, 266 op/s 2024-03-16T11:53:21.777 INFO:tasks.workunit.client.0.smithi029.stdout: 1 153801 12.81 MB/sec warmup 66 sec latency 35.233 ms 2024-03-16T11:53:22.777 INFO:tasks.workunit.client.0.smithi029.stdout: 1 156250 12.85 MB/sec warmup 67 sec latency 90.037 ms 2024-03-16T11:53:22.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:53:22 smithi152 ceph-mon[36291]: pgmap v76: 97 pgs: 97 active+clean; 48 MiB data, 591 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 2.1 MiB/s wr, 344 op/s 2024-03-16T11:53:23.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:53:22 smithi154 ceph-mon[36320]: pgmap v76: 97 pgs: 97 active+clean; 48 MiB data, 591 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 2.1 MiB/s wr, 344 op/s 2024-03-16T11:53:23.087 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:53:22 smithi029 ceph-mon[31975]: pgmap v76: 97 pgs: 97 active+clean; 48 MiB data, 591 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 2.1 MiB/s wr, 344 op/s 2024-03-16T11:53:23.248 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:23.249+0000 7f337caff640 1 -- 172.21.15.29:0/1493930440 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f33781099b0 msgr2=0x7f3378109d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:53:23.248 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:23.249+0000 7f337caff640 1 --2- 172.21.15.29:0/1493930440 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f33781099b0 0x7f3378109d90 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f3368009930 tx=0x7f336802bc30 comp rx=0 tx=0).stop 2024-03-16T11:53:23.248 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:23.249+0000 7f337caff640 1 -- 172.21.15.29:0/1493930440 shutdown_connections 2024-03-16T11:53:23.248 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:23.249+0000 7f337caff640 1 --2- 172.21.15.29:0/1493930440 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3378104360 0x7f3378110870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:53:23.248 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:23.249+0000 7f337caff640 1 --2- 172.21.15.29:0/1493930440 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f33781039a0 0x7f3378103e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:53:23.249 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:23.249+0000 7f337caff640 1 --2- 172.21.15.29:0/1493930440 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f33781099b0 0x7f3378109d90 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:53:23.249 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:23.249+0000 7f337caff640 1 -- 172.21.15.29:0/1493930440 >> 172.21.15.29:0/1493930440 conn(0x7f33780ff4d0 msgr2=0x7f33781018f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:53:23.249 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:23.249+0000 7f337caff640 1 -- 172.21.15.29:0/1493930440 shutdown_connections 2024-03-16T11:53:23.249 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:23.249+0000 7f337caff640 1 -- 172.21.15.29:0/1493930440 wait complete. 2024-03-16T11:53:23.251 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:23.250+0000 7f337caff640 1 Processor -- start 2024-03-16T11:53:23.251 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:23.250+0000 7f337caff640 1 -- start start 2024-03-16T11:53:23.251 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:23.250+0000 7f337caff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f33781039a0 0x7f337819a220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:53:23.251 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:23.250+0000 7f337caff640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3378104360 0x7f337819a760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:53:23.251 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:23.250+0000 7f337caff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f33781099b0 0x7f337819eae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:53:23.251 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:23.250+0000 7f337caff640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f3378113dc0 con 0x7f33781099b0 2024-03-16T11:53:23.251 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:23.250+0000 7f337caff640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f3378113c40 con 0x7f3378104360 2024-03-16T11:53:23.251 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:23.250+0000 7f337caff640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f3378113f40 con 0x7f33781039a0 2024-03-16T11:53:23.251 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:23.250+0000 7f3376d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f33781099b0 0x7f337819eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:53:23.251 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:23.250+0000 7f337659c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f33781039a0 0x7f337819a220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:53:23.251 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:23.250+0000 7f337659c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f33781039a0 0x7f337819a220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.29:33042/0 (socket says 172.21.15.29:33042) 2024-03-16T11:53:23.251 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:23.250+0000 7f337659c640 1 -- 172.21.15.29:0/4122171794 learned_addr learned my addr 172.21.15.29:0/4122171794 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:53:23.251 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:23.250+0000 7f336ffff640 1 --2- 172.21.15.29:0/4122171794 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3378104360 0x7f337819a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:53:23.251 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:23.250+0000 7f3376d9d640 1 -- 172.21.15.29:0/4122171794 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f33781039a0 msgr2=0x7f337819a220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:53:23.252 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:23.250+0000 7f3376d9d640 1 --2- 172.21.15.29:0/4122171794 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f33781039a0 0x7f337819a220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:53:23.252 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:23.250+0000 7f3376d9d640 1 -- 172.21.15.29:0/4122171794 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3378104360 msgr2=0x7f337819a760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:53:23.252 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:23.250+0000 7f3376d9d640 1 --2- 172.21.15.29:0/4122171794 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3378104360 0x7f337819a760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:53:23.252 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:23.250+0000 7f3376d9d640 1 -- 172.21.15.29:0/4122171794 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f337819f210 con 0x7f33781099b0 2024-03-16T11:53:23.252 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:23.250+0000 7f337659c640 1 --2- 172.21.15.29:0/4122171794 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f33781039a0 0x7f337819a220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:53:23.252 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:23.251+0000 7f3376d9d640 1 --2- 172.21.15.29:0/4122171794 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f33781099b0 0x7f337819eae0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f3370009fd0 tx=0x7f337000ef10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:53:23.252 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:23.251+0000 7f336f7fe640 1 -- 172.21.15.29:0/4122171794 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f337000ce20 con 0x7f33781099b0 2024-03-16T11:53:23.252 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:23.251+0000 7f336f7fe640 1 -- 172.21.15.29:0/4122171794 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3370005020 con 0x7f33781099b0 2024-03-16T11:53:23.252 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:23.251+0000 7f337caff640 1 -- 172.21.15.29:0/4122171794 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f337819f500 con 0x7f33781099b0 2024-03-16T11:53:23.252 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:23.251+0000 7f336f7fe640 1 -- 172.21.15.29:0/4122171794 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3370010720 con 0x7f33781099b0 2024-03-16T11:53:23.252 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:23.251+0000 7f337caff640 1 -- 172.21.15.29:0/4122171794 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f33781a6d90 con 0x7f33781099b0 2024-03-16T11:53:23.258 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:23.253+0000 7f336f7fe640 1 -- 172.21.15.29:0/4122171794 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7f3370005540 con 0x7f33781099b0 2024-03-16T11:53:23.258 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:23.253+0000 7f337f7c7740 1 Processor -- start 2024-03-16T11:53:23.258 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:23.253+0000 7f337f7c7740 1 -- start start 2024-03-16T11:53:23.258 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:23.253+0000 7f336f7fe640 1 --2- 172.21.15.29:0/4122171794 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f33640a2a10 0x7f33640a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:53:23.258 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:23.253+0000 7f337f7c7740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55eaf8ac0130 0x55eaf8ac0510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:53:23.258 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:23.253+0000 7f337f7c7740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55eaf8ac0a50 0x55eaf8ac92e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:53:23.258 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:23.254+0000 7f337f7c7740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55eaf8ac9820 0x55eaf8acbc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:53:23.259 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:23.254+0000 7f337f7c7740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55eaf8a24a40 con 0x55eaf8ac9820 2024-03-16T11:53:23.259 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:23.254+0000 7f337f7c7740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x55eaf89cd920 con 0x55eaf8ac0a50 2024-03-16T11:53:23.259 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:23.254+0000 7f337f7c7740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55eaf8a20ae0 con 0x55eaf8ac0130 2024-03-16T11:53:23.259 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:23.254+0000 7f336ffff640 1 --2- 172.21.15.29:0/4122171794 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f33640a2a10 0x7f33640a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:53:23.259 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:23.254+0000 7f3376d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55eaf8ac9820 0x55eaf8acbc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:53:23.259 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:23.254+0000 7f337659c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55eaf8ac0130 0x55eaf8ac0510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:53:23.259 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:23.254+0000 7f337659c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55eaf8ac0130 0x55eaf8ac0510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.29:33056/0 (socket says 172.21.15.29:33056) 2024-03-16T11:53:23.259 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:23.254+0000 7f336f7fe640 1 -- 172.21.15.29:0/4122171794 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8469+0+0 (secure 0 0 0) 0x7f33700cf3f0 con 0x7f33781099b0 2024-03-16T11:53:23.259 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:23.254+0000 7f337659c640 1 -- 172.21.15.29:0/100222342 learned_addr learned my addr 172.21.15.29:0/100222342 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:53:23.259 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:23.254+0000 7f336ffff640 1 --2- 172.21.15.29:0/100222342 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55eaf8ac0a50 0x55eaf8ac92e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:53:23.259 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:23.254+0000 7f3376d9d640 1 -- 172.21.15.29:0/100222342 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55eaf8ac0130 msgr2=0x55eaf8ac0510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:53:23.259 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:23.254+0000 7f3376d9d640 1 --2- 172.21.15.29:0/100222342 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55eaf8ac0130 0x55eaf8ac0510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:53:23.259 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:23.254+0000 7f3376d9d640 1 -- 172.21.15.29:0/100222342 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55eaf8ac0a50 msgr2=0x55eaf8ac92e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:53:23.259 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:23.254+0000 7f3376d9d640 1 --2- 172.21.15.29:0/100222342 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55eaf8ac0a50 0x55eaf8ac92e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:53:23.259 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:23.254+0000 7f3376d9d640 1 -- 172.21.15.29:0/100222342 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55eaf89e5460 con 0x55eaf8ac9820 2024-03-16T11:53:23.260 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:23.254+0000 7f337659c640 1 --2- 172.21.15.29:0/100222342 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55eaf8ac0130 0x55eaf8ac0510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:53:23.260 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:23.254+0000 7f3376d9d640 1 --2- 172.21.15.29:0/100222342 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55eaf8ac9820 0x55eaf8acbc10 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f33700ccb70 tx=0x7f3370005940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26be9e5bbe2343fd server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:53:23.260 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:23.254+0000 7f336ffff640 1 --2- 172.21.15.29:0/4122171794 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f33640a2a10 0x7f33640a4ed0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f336000bfd0 tx=0x7f336000bb40 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:53:23.260 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:23.254+0000 7f336cff9640 1 -- 172.21.15.29:0/100222342 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f337006fb20 con 0x55eaf8ac9820 2024-03-16T11:53:23.260 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:23.254+0000 7f336cff9640 1 -- 172.21.15.29:0/100222342 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3370066ee0 con 0x55eaf8ac9820 2024-03-16T11:53:23.260 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:23.254+0000 7f336cff9640 1 -- 172.21.15.29:0/100222342 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33700ccdf0 con 0x55eaf8ac9820 2024-03-16T11:53:23.260 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:23.255+0000 7f337f7c7740 1 -- 172.21.15.29:0/100222342 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55eaf8ac9820 msgr2=0x55eaf8acbc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:53:23.260 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:23.255+0000 7f337f7c7740 1 --2- 172.21.15.29:0/100222342 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55eaf8ac9820 0x55eaf8acbc10 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f33700ccb70 tx=0x7f3370005940 comp rx=0 tx=0).stop 2024-03-16T11:53:23.260 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:23.255+0000 7f337f7c7740 1 -- 172.21.15.29:0/100222342 shutdown_connections 2024-03-16T11:53:23.260 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:23.255+0000 7f337f7c7740 1 --2- 172.21.15.29:0/100222342 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55eaf8ac9820 0x55eaf8acbc10 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:53:23.260 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:23.255+0000 7f337f7c7740 1 --2- 172.21.15.29:0/100222342 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55eaf8ac0a50 0x55eaf8ac92e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:53:23.260 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:23.255+0000 7f337f7c7740 1 --2- 172.21.15.29:0/100222342 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55eaf8ac0130 0x55eaf8ac0510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:53:23.260 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:23.255+0000 7f337f7c7740 1 -- 172.21.15.29:0/100222342 >> 172.21.15.29:0/100222342 conn(0x55eaf8abaa50 msgr2=0x55eaf8abae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:53:23.260 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:23.255+0000 7f337f7c7740 1 -- 172.21.15.29:0/100222342 shutdown_connections 2024-03-16T11:53:23.261 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:23.255+0000 7f337f7c7740 1 -- 172.21.15.29:0/100222342 wait complete. 2024-03-16T11:53:23.261 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:23.256+0000 7f337f7c7740 1 -- start start 2024-03-16T11:53:23.261 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:23.256+0000 7f337f7c7740 1 Processor -- start 2024-03-16T11:53:23.261 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:23.256+0000 7f337f7c7740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55eaf8ac0130 0x55eaf8ac68c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:53:23.261 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:23.256+0000 7f337f7c7740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55eaf8ac0a50 0x55eaf8ac6e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:53:23.261 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:23.256+0000 7f337f7c7740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55eaf8ac9820 0x55eaf8ac7340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:53:23.261 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:23.256+0000 7f337f7c7740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55eaf8acc3f0 con 0x55eaf8ac0130 2024-03-16T11:53:23.261 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:23.256+0000 7f337f7c7740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x55eaf89e0d70 con 0x55eaf8ac9820 2024-03-16T11:53:23.261 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:23.256+0000 7f337f7c7740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55eaf8acc570 con 0x55eaf8ac0a50 2024-03-16T11:53:23.261 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:23.256+0000 7f337f7c7740 10 client.0 _ll_register_callbacks cb 0x55eaf8738a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T11:53:23.261 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:23.256+0000 7f337659c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55eaf8ac0130 0x55eaf8ac68c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:53:23.261 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:23.256+0000 7f337659c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55eaf8ac0130 0x55eaf8ac68c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:37596/0 (socket says 172.21.15.29:37596) 2024-03-16T11:53:23.261 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:23.256+0000 7f337659c640 1 -- 172.21.15.29:0/2595842037 learned_addr learned my addr 172.21.15.29:0/2595842037 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:53:23.261 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:23.256+0000 7f336ffff640 1 --2- 172.21.15.29:0/2595842037 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55eaf8ac0a50 0x55eaf8ac6e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:53:23.261 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:23.256+0000 7f3376d9d640 1 --2- 172.21.15.29:0/2595842037 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55eaf8ac9820 0x55eaf8ac7340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:53:23.262 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:23.256+0000 7f337659c640 1 -- 172.21.15.29:0/2595842037 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55eaf8ac0a50 msgr2=0x55eaf8ac6e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:53:23.262 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:23.256+0000 7f337659c640 1 --2- 172.21.15.29:0/2595842037 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55eaf8ac0a50 0x55eaf8ac6e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:53:23.262 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:23.256+0000 7f337659c640 1 -- 172.21.15.29:0/2595842037 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55eaf8ac9820 msgr2=0x55eaf8ac7340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:53:23.262 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:23.256+0000 7f337659c640 1 --2- 172.21.15.29:0/2595842037 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55eaf8ac9820 0x55eaf8ac7340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:53:23.262 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:23.256+0000 7f337659c640 1 -- 172.21.15.29:0/2595842037 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55eaf8739210 con 0x55eaf8ac0130 2024-03-16T11:53:23.262 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:23.256+0000 7f337659c640 1 --2- 172.21.15.29:0/2595842037 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55eaf8ac0130 0x55eaf8ac68c0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f3368005940 tx=0x7f336802c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25c25f8561125c2 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:53:23.262 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:23.256+0000 7f335dffb640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-16T11:53:23.262 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:23.257+0000 7f335dffb640 1 -- 172.21.15.29:0/2595842037 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3368004040 con 0x55eaf8ac0130 2024-03-16T11:53:23.262 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:23.257+0000 7f335dffb640 1 -- 172.21.15.29:0/2595842037 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8469+0+0 (secure 0 0 0) 0x7f33680041e0 con 0x55eaf8ac0130 2024-03-16T11:53:23.262 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:23.257+0000 7f337f7c7740 1 -- 172.21.15.29:0/2595842037 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3370066ee0 con 0x55eaf8ac0130 2024-03-16T11:53:23.262 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:23.257+0000 7f335dffb640 1 -- 172.21.15.29:0/2595842037 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f33680043e0 con 0x55eaf8ac0130 2024-03-16T11:53:23.262 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:23.257+0000 7f337f7c7740 10 client.15606 fetch_fsmap learned FSMap version 22 2024-03-16T11:53:23.262 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:23.257+0000 7f337f7c7740 1 -- 172.21.15.29:0/2595842037 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x55eaf8805620 con 0x55eaf8ac0130 2024-03-16T11:53:23.262 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:23.257+0000 7f335dffb640 1 -- 172.21.15.29:0/2595842037 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3370066ee0 con 0x55eaf8ac0130 2024-03-16T11:53:23.262 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:23.257+0000 7f335dffb640 1 -- 172.21.15.29:0/2595842037 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7f33680039d0 con 0x55eaf8ac0130 2024-03-16T11:53:23.263 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:23.257+0000 7f337f7c7740 10 client.15606 fetch_fsmap finished waiting for FSMap version 22 2024-03-16T11:53:23.263 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:23.257+0000 7f337f7c7740 10 client.15606 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T11:53:23.263 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:23.257+0000 7f337f7c7740 20 client.15606 populate_metadata read hostname 'smithi029' 2024-03-16T11:53:23.263 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:23.257+0000 7f337f7c7740 1 --2- 172.21.15.29:0/2595842037 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55eaf8ad99f0 0x55eaf8adbe00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:53:23.263 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:23.257+0000 7f337f7c7740 4 client.15606 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T11:53:23.263 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:23.257+0000 7f337f7c7740 1 -- 172.21.15.29:0/2595842037 --> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55eaf8a31f80 con 0x55eaf8ad99f0 2024-03-16T11:53:23.263 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:23.257+0000 7f336ffff640 1 --2- 172.21.15.29:0/2595842037 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55eaf8ad99f0 0x55eaf8adbe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:53:23.263 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:23.258+0000 7f335dffb640 10 client.15606 ms_handle_connect on v2:172.21.15.152:6838/1548731833 2024-03-16T11:53:23.263 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:23.258+0000 7f336ffff640 1 --2- 172.21.15.29:0/2595842037 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55eaf8ad99f0 0x55eaf8adbe00 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=38d310c46e9823e3 server_cookie=c9b8196d043d7fe9 in_seq=0 out_seq=0 2024-03-16T11:53:23.264 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:23.260+0000 7f335dffb640 1 -- 172.21.15.29:0/2595842037 <== mds.0 v2:172.21.15.152:6838/1548731833 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55eaf8a31f80 con 0x55eaf8ad99f0 2024-03-16T11:53:23.264 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:23.260+0000 7f335dffb640 10 client.15606 handle_command_reply: tid=0 2024-03-16T11:53:23.264 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:23.260+0000 7f337f7c7740 1 client.15606 shutdown 2024-03-16T11:53:23.264 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:23.260+0000 7f337f7c7740 1 -- 172.21.15.29:0/2595842037 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55eaf8ac0130 msgr2=0x55eaf8ac68c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:53:23.264 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:23.260+0000 7f337f7c7740 1 --2- 172.21.15.29:0/2595842037 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55eaf8ac0130 0x55eaf8ac68c0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f3368005940 tx=0x7f336802c880 comp rx=0 tx=0).stop 2024-03-16T11:53:23.264 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:23.260+0000 7f337f7c7740 1 -- 172.21.15.29:0/2595842037 shutdown_connections 2024-03-16T11:53:23.265 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:23.260+0000 7f337f7c7740 1 --2- 172.21.15.29:0/2595842037 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55eaf8ad99f0 0x55eaf8adbe00 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:53:23.265 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:23.260+0000 7f337f7c7740 1 --2- 172.21.15.29:0/2595842037 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55eaf8ac9820 0x55eaf8ac7340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:53:23.265 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:23.260+0000 7f335dffb640 0 client.15606 ms_handle_reset on v2:172.21.15.152:6838/1548731833 2024-03-16T11:53:23.265 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:23.260+0000 7f337f7c7740 1 --2- 172.21.15.29:0/2595842037 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55eaf8ac0a50 0x55eaf8ac6e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:53:23.265 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:23.260+0000 7f337f7c7740 1 --2- 172.21.15.29:0/2595842037 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55eaf8ac0130 0x55eaf8ac68c0 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:53:23.265 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:23.260+0000 7f337f7c7740 1 -- 172.21.15.29:0/2595842037 >> 172.21.15.29:0/2595842037 conn(0x55eaf8abaa50 msgr2=0x55eaf8abc7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:53:23.265 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:23.260+0000 7f337f7c7740 1 -- 172.21.15.29:0/2595842037 shutdown_connections 2024-03-16T11:53:23.265 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:23.260+0000 7f337f7c7740 1 -- 172.21.15.29:0/2595842037 wait complete. 2024-03-16T11:53:23.265 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:23.260+0000 7f337f7c7740 20 client.15606 trim_cache size 0 max 16384 2024-03-16T11:53:23.276 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:23.276+0000 7f337f7c7740 1 Processor -- start 2024-03-16T11:53:23.277 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:23.276+0000 7f337f7c7740 1 -- start start 2024-03-16T11:53:23.277 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:23.276+0000 7f337f7c7740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55eaf8ac0130 0x55eaf8ac8ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:53:23.277 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:23.276+0000 7f337f7c7740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55eaf8ac0a50 0x55eaf8ad5190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:53:23.277 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:23.277+0000 7f337f7c7740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55eaf8ac9820 0x55eaf8abc580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:53:23.277 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:23.277+0000 7f337f7c7740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f33781b1f40 con 0x55eaf8ac9820 2024-03-16T11:53:23.277 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:23.277+0000 7f337f7c7740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f33781af810 con 0x55eaf8ac0a50 2024-03-16T11:53:23.277 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:23.277+0000 7f337f7c7740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f33781ad660 con 0x55eaf8ac0130 2024-03-16T11:53:23.277 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:23.277+0000 7f3376d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55eaf8ac9820 0x55eaf8abc580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:53:23.277 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:23.277+0000 7f337659c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55eaf8ac0130 0x55eaf8ac8ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:53:23.277 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:23.277+0000 7f3376d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55eaf8ac9820 0x55eaf8abc580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:37604/0 (socket says 172.21.15.29:37604) 2024-03-16T11:53:23.277 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:23.277+0000 7f337659c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55eaf8ac0130 0x55eaf8ac8ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.29:33068/0 (socket says 172.21.15.29:33068) 2024-03-16T11:53:23.277 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:23.277+0000 7f3376d9d640 1 -- 172.21.15.29:0/3152463900 learned_addr learned my addr 172.21.15.29:0/3152463900 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:53:23.277 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:23.277+0000 7f336ffff640 1 --2- 172.21.15.29:0/3152463900 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55eaf8ac0a50 0x55eaf8ad5190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:53:23.278 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:23.277+0000 7f337659c640 1 -- 172.21.15.29:0/3152463900 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55eaf8ac0a50 msgr2=0x55eaf8ad5190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:53:23.278 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:23.277+0000 7f337659c640 1 --2- 172.21.15.29:0/3152463900 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55eaf8ac0a50 0x55eaf8ad5190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:53:23.278 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:23.277+0000 7f337659c640 1 -- 172.21.15.29:0/3152463900 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55eaf8ac9820 msgr2=0x55eaf8abc580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:53:23.278 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:23.277+0000 7f337659c640 1 --2- 172.21.15.29:0/3152463900 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55eaf8ac9820 0x55eaf8abc580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:53:23.278 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:23.277+0000 7f337659c640 1 -- 172.21.15.29:0/3152463900 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55eaf8ab6310 con 0x55eaf8ac0130 2024-03-16T11:53:23.278 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:23.277+0000 7f3376d9d640 1 --2- 172.21.15.29:0/3152463900 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55eaf8ac9820 0x55eaf8abc580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:53:23.278 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:23.277+0000 7f337659c640 1 --2- 172.21.15.29:0/3152463900 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55eaf8ac0130 0x55eaf8ac8ff0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f3368003e70 tx=0x7f3368005b70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a2aa8fc7cce35a36 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:53:23.278 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:23.277+0000 7f335ffff640 1 -- 172.21.15.29:0/3152463900 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3368054070 con 0x55eaf8ac0130 2024-03-16T11:53:23.278 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:23.277+0000 7f335ffff640 1 -- 172.21.15.29:0/3152463900 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f336802c8f0 con 0x55eaf8ac0130 2024-03-16T11:53:23.278 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:23.277+0000 7f335ffff640 1 -- 172.21.15.29:0/3152463900 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3368003f90 con 0x55eaf8ac0130 2024-03-16T11:53:23.281 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:23.278+0000 7f337f7c7740 1 -- 172.21.15.29:0/3152463900 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55eaf8ac0130 msgr2=0x55eaf8ac8ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:53:23.282 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:23.278+0000 7f337f7c7740 1 --2- 172.21.15.29:0/3152463900 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55eaf8ac0130 0x55eaf8ac8ff0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f3368003e70 tx=0x7f3368005b70 comp rx=0 tx=0).stop 2024-03-16T11:53:23.282 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:23.278+0000 7f337f7c7740 1 -- 172.21.15.29:0/3152463900 shutdown_connections 2024-03-16T11:53:23.282 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:23.278+0000 7f337f7c7740 1 --2- 172.21.15.29:0/3152463900 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55eaf8ac9820 0x55eaf8abc580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:53:23.282 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:23.278+0000 7f337f7c7740 1 --2- 172.21.15.29:0/3152463900 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55eaf8ac0a50 0x55eaf8ad5190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:53:23.282 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:23.278+0000 7f337f7c7740 1 --2- 172.21.15.29:0/3152463900 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55eaf8ac0130 0x55eaf8ac8ff0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:53:23.282 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:23.278+0000 7f337f7c7740 1 -- 172.21.15.29:0/3152463900 >> 172.21.15.29:0/3152463900 conn(0x55eaf8abaa50 msgr2=0x55eaf8abbe70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:53:23.282 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:23.278+0000 7f337f7c7740 1 -- 172.21.15.29:0/3152463900 shutdown_connections 2024-03-16T11:53:23.282 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:23.278+0000 7f337f7c7740 1 -- 172.21.15.29:0/3152463900 wait complete. 2024-03-16T11:53:23.282 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:23.278+0000 7f337f7c7740 1 -- start start 2024-03-16T11:53:23.282 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:23.279+0000 7f337f7c7740 1 Processor -- start 2024-03-16T11:53:23.282 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:23.279+0000 7f337f7c7740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55eaf8ac0130 0x55eaf8ad19b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:53:23.282 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:23.279+0000 7f337f7c7740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55eaf8ac0a50 0x55eaf8ace790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:53:23.282 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:23.279+0000 7f337f7c7740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55eaf8ac9820 0x55eaf8acecd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:53:23.283 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:23.279+0000 7f337f7c7740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f33781adcd0 con 0x55eaf8ac0a50 2024-03-16T11:53:23.283 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:23.279+0000 7f337f7c7740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f33781af140 con 0x55eaf8ac9820 2024-03-16T11:53:23.283 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:23.279+0000 7f337f7c7740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f33781ae3a0 con 0x55eaf8ac0130 2024-03-16T11:53:23.283 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:23.279+0000 7f337f7c7740 10 client.0 _ll_register_callbacks cb 0x7f3368009880 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T11:53:23.283 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:23.279+0000 7f336ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55eaf8ac0a50 0x55eaf8ace790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:53:23.283 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:23.279+0000 7f337659c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55eaf8ac0130 0x55eaf8ad19b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:53:23.283 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:23.279+0000 7f336ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55eaf8ac0a50 0x55eaf8ace790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:37618/0 (socket says 172.21.15.29:37618) 2024-03-16T11:53:23.283 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:23.279+0000 7f336ffff640 1 -- 172.21.15.29:0/1290412282 learned_addr learned my addr 172.21.15.29:0/1290412282 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:53:23.283 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:23.279+0000 7f3376d9d640 1 --2- 172.21.15.29:0/1290412282 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55eaf8ac9820 0x55eaf8acecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:53:23.283 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:23.279+0000 7f336ffff640 1 -- 172.21.15.29:0/1290412282 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55eaf8ac0130 msgr2=0x55eaf8ad19b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:53:23.283 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:23.279+0000 7f336ffff640 1 --2- 172.21.15.29:0/1290412282 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55eaf8ac0130 0x55eaf8ad19b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:53:23.283 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:23.279+0000 7f336ffff640 1 -- 172.21.15.29:0/1290412282 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55eaf8ac9820 msgr2=0x55eaf8acecd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:53:23.284 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:23.279+0000 7f336ffff640 1 --2- 172.21.15.29:0/1290412282 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55eaf8ac9820 0x55eaf8acecd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:53:23.284 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:23.279+0000 7f336ffff640 1 -- 172.21.15.29:0/1290412282 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55eaf8acf210 con 0x55eaf8ac0a50 2024-03-16T11:53:23.284 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:23.279+0000 7f336ffff640 1 --2- 172.21.15.29:0/1290412282 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55eaf8ac0a50 0x55eaf8ace790 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f3360005fd0 tx=0x7f336001c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf607dcf32485dfa server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:53:23.284 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:23.279+0000 7f335dffb640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-16T11:53:23.284 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:23.279+0000 7f335dffb640 1 -- 172.21.15.29:0/1290412282 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3360030060 con 0x55eaf8ac0a50 2024-03-16T11:53:23.284 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:23.280+0000 7f337f7c7740 1 -- 172.21.15.29:0/1290412282 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f336802c8f0 con 0x55eaf8ac0a50 2024-03-16T11:53:23.284 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:23.280+0000 7f335dffb640 1 -- 172.21.15.29:0/1290412282 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8469+0+0 (secure 0 0 0) 0x7f336002a480 con 0x55eaf8ac0a50 2024-03-16T11:53:23.284 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:23.280+0000 7f335dffb640 1 -- 172.21.15.29:0/1290412282 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f336002a680 con 0x55eaf8ac0a50 2024-03-16T11:53:23.284 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:23.280+0000 7f337f7c7740 10 client.15612 fetch_fsmap learned FSMap version 22 2024-03-16T11:53:23.284 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:23.280+0000 7f337f7c7740 1 -- 172.21.15.29:0/1290412282 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x55eaf8b40ad0 con 0x55eaf8ac0a50 2024-03-16T11:53:23.284 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:23.280+0000 7f335dffb640 1 -- 172.21.15.29:0/1290412282 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f336802c8f0 con 0x55eaf8ac0a50 2024-03-16T11:53:23.284 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:23.280+0000 7f335dffb640 1 -- 172.21.15.29:0/1290412282 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7f336001f070 con 0x55eaf8ac0a50 2024-03-16T11:53:23.284 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:23.280+0000 7f337f7c7740 10 client.15612 fetch_fsmap finished waiting for FSMap version 22 2024-03-16T11:53:23.284 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:23.280+0000 7f337f7c7740 10 client.15612 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T11:53:23.285 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:23.280+0000 7f337f7c7740 20 client.15612 populate_metadata read hostname 'smithi029' 2024-03-16T11:53:23.285 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:23.280+0000 7f337f7c7740 1 --2- 172.21.15.29:0/1290412282 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55eaf8b41230 0x55eaf8b43650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:53:23.285 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:23.280+0000 7f337f7c7740 4 client.15612 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-16T11:53:23.285 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:23.280+0000 7f337f7c7740 1 -- 172.21.15.29:0/1290412282 --> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55eaf8a2c2d0 con 0x55eaf8b41230 2024-03-16T11:53:23.285 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:23.280+0000 7f337659c640 1 --2- 172.21.15.29:0/1290412282 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55eaf8b41230 0x55eaf8b43650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:53:23.285 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:23.281+0000 7f335dffb640 10 client.15612 ms_handle_connect on v2:172.21.15.152:6838/1548731833 2024-03-16T11:53:23.285 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:23.281+0000 7f337659c640 1 --2- 172.21.15.29:0/1290412282 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55eaf8b41230 0x55eaf8b43650 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d816a7718e3ee973 server_cookie=b70fb24cf2c9ba39 in_seq=0 out_seq=0 2024-03-16T11:53:23.453 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:23.453+0000 7f335dffb640 1 -- 172.21.15.29:0/1290412282 <== mds.0 v2:172.21.15.152:6838/1548731833 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55eaf8a2c2d0 con 0x55eaf8b41230 2024-03-16T11:53:23.454 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:23.454+0000 7f335dffb640 10 client.15612 handle_command_reply: tid=0 2024-03-16T11:53:23.454 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:23.454+0000 7f337f7c7740 1 client.15612 shutdown 2024-03-16T11:53:23.454 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:23.454+0000 7f337f7c7740 1 -- 172.21.15.29:0/1290412282 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55eaf8ac0a50 msgr2=0x55eaf8ace790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:53:23.454 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:23.454+0000 7f337f7c7740 1 --2- 172.21.15.29:0/1290412282 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55eaf8ac0a50 0x55eaf8ace790 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f3360005fd0 tx=0x7f336001c000 comp rx=0 tx=0).stop 2024-03-16T11:53:23.454 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:23.454+0000 7f337f7c7740 1 -- 172.21.15.29:0/1290412282 shutdown_connections 2024-03-16T11:53:23.454 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:23.454+0000 7f337f7c7740 1 --2- 172.21.15.29:0/1290412282 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55eaf8b41230 0x55eaf8b43650 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:53:23.454 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:23.454+0000 7f335dffb640 0 client.15612 ms_handle_reset on v2:172.21.15.152:6838/1548731833 2024-03-16T11:53:23.454 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:23.454+0000 7f337f7c7740 1 --2- 172.21.15.29:0/1290412282 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55eaf8ac9820 0x55eaf8acecd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:53:23.454 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:23.454+0000 7f337f7c7740 1 --2- 172.21.15.29:0/1290412282 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55eaf8ac0a50 0x55eaf8ace790 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:53:23.454 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:23.454+0000 7f337f7c7740 1 --2- 172.21.15.29:0/1290412282 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55eaf8ac0130 0x55eaf8ad19b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:53:23.454 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:23.454+0000 7f337f7c7740 1 -- 172.21.15.29:0/1290412282 >> 172.21.15.29:0/1290412282 conn(0x55eaf8abaa50 msgr2=0x55eaf8abc700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:53:23.454 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:23.454+0000 7f337f7c7740 1 -- 172.21.15.29:0/1290412282 shutdown_connections 2024-03-16T11:53:23.454 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:23.454+0000 7f337f7c7740 1 -- 172.21.15.29:0/1290412282 wait complete. 2024-03-16T11:53:23.455 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:23.454+0000 7f337f7c7740 20 client.15612 trim_cache size 0 max 16384 2024-03-16T11:53:23.455 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:{ 2024-03-16T11:53:23.455 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "message": "", 2024-03-16T11:53:23.455 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "return_code": 0 2024-03-16T11:53:23.455 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:} 2024-03-16T11:53:23.456 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:23.455+0000 7f337caff640 1 -- 172.21.15.29:0/4122171794 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f33640a2a10 msgr2=0x7f33640a4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:53:23.456 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:23.455+0000 7f337caff640 1 --2- 172.21.15.29:0/4122171794 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f33640a2a10 0x7f33640a4ed0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f336000bfd0 tx=0x7f336000bb40 comp rx=0 tx=0).stop 2024-03-16T11:53:23.456 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:23.455+0000 7f337caff640 1 -- 172.21.15.29:0/4122171794 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f33781099b0 msgr2=0x7f337819eae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:53:23.456 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:23.455+0000 7f337caff640 1 --2- 172.21.15.29:0/4122171794 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f33781099b0 0x7f337819eae0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f3370009fd0 tx=0x7f337000ef10 comp rx=0 tx=0).stop 2024-03-16T11:53:23.456 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:23.455+0000 7f337caff640 1 -- 172.21.15.29:0/4122171794 shutdown_connections 2024-03-16T11:53:23.456 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:23.455+0000 7f337caff640 1 --2- 172.21.15.29:0/4122171794 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f33640a2a10 0x7f33640a4ed0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:53:23.456 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:23.455+0000 7f337caff640 1 --2- 172.21.15.29:0/4122171794 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f33781099b0 0x7f337819eae0 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:53:23.456 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:23.455+0000 7f337caff640 1 --2- 172.21.15.29:0/4122171794 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3378104360 0x7f337819a760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:53:23.456 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:23.455+0000 7f337caff640 1 --2- 172.21.15.29:0/4122171794 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f33781039a0 0x7f337819a220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:53:23.457 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:23.455+0000 7f337caff640 1 -- 172.21.15.29:0/4122171794 >> 172.21.15.29:0/4122171794 conn(0x7f33780ff4d0 msgr2=0x7f33780ffa90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:53:23.457 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:23.455+0000 7f337caff640 1 -- 172.21.15.29:0/4122171794 shutdown_connections 2024-03-16T11:53:23.457 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:23.455+0000 7f337caff640 1 -- 172.21.15.29:0/4122171794 wait complete. 2024-03-16T11:53:23.777 INFO:tasks.workunit.client.0.smithi029.stdout: 1 157733 12.76 MB/sec warmup 68 sec latency 75.457 ms 2024-03-16T11:53:24.777 INFO:tasks.workunit.client.0.smithi029.stdout: 1 160109 12.78 MB/sec warmup 69 sec latency 21.915 ms 2024-03-16T11:53:24.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:53:24 smithi152 ceph-mon[36291]: pgmap v77: 97 pgs: 97 active+clean; 48 MiB data, 591 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 1.4 MiB/s wr, 161 op/s 2024-03-16T11:53:25.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:53:24 smithi154 ceph-mon[36320]: pgmap v77: 97 pgs: 97 active+clean; 48 MiB data, 591 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 1.4 MiB/s wr, 161 op/s 2024-03-16T11:53:25.087 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:53:24 smithi029 ceph-mon[31975]: pgmap v77: 97 pgs: 97 active+clean; 48 MiB data, 591 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 1.4 MiB/s wr, 161 op/s 2024-03-16T11:53:25.777 INFO:tasks.workunit.client.0.smithi029.stdout: 1 162843 12.78 MB/sec warmup 70 sec latency 25.587 ms 2024-03-16T11:53:26.777 INFO:tasks.workunit.client.0.smithi029.stdout: 1 164571 12.76 MB/sec warmup 71 sec latency 23.066 ms 2024-03-16T11:53:26.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:53:26 smithi152 ceph-mon[36291]: pgmap v78: 97 pgs: 97 active+clean; 48 MiB data, 591 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 1.4 MiB/s wr, 161 op/s 2024-03-16T11:53:27.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:53:26 smithi154 ceph-mon[36320]: pgmap v78: 97 pgs: 97 active+clean; 48 MiB data, 591 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 1.4 MiB/s wr, 161 op/s 2024-03-16T11:53:27.087 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:53:26 smithi029 ceph-mon[31975]: pgmap v78: 97 pgs: 97 active+clean; 48 MiB data, 591 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 1.4 MiB/s wr, 161 op/s 2024-03-16T11:53:27.777 INFO:tasks.workunit.client.0.smithi029.stdout: 1 167031 12.77 MB/sec warmup 72 sec latency 60.731 ms 2024-03-16T11:53:28.777 INFO:tasks.workunit.client.0.smithi029.stdout: 1 169174 12.70 MB/sec warmup 73 sec latency 28.714 ms 2024-03-16T11:53:28.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:53:28 smithi152 ceph-mon[36291]: pgmap v79: 97 pgs: 97 active+clean; 41 MiB data, 607 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 2.9 MiB/s wr, 358 op/s 2024-03-16T11:53:28.889 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:53:28 smithi029 ceph-mon[31975]: pgmap v79: 97 pgs: 97 active+clean; 41 MiB data, 607 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 2.9 MiB/s wr, 358 op/s 2024-03-16T11:53:29.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:53:28 smithi154 ceph-mon[36320]: pgmap v79: 97 pgs: 97 active+clean; 41 MiB data, 607 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 2.9 MiB/s wr, 358 op/s 2024-03-16T11:53:29.777 INFO:tasks.workunit.client.0.smithi029.stdout: 1 171063 12.69 MB/sec warmup 74 sec latency 19.066 ms 2024-03-16T11:53:30.777 INFO:tasks.workunit.client.0.smithi029.stdout: 1 174030 12.77 MB/sec warmup 75 sec latency 23.835 ms 2024-03-16T11:53:30.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:53:30 smithi152 ceph-mon[36291]: pgmap v80: 97 pgs: 97 active+clean; 41 MiB data, 607 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 2.1 MiB/s wr, 275 op/s 2024-03-16T11:53:31.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:53:30 smithi154 ceph-mon[36320]: pgmap v80: 97 pgs: 97 active+clean; 41 MiB data, 607 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 2.1 MiB/s wr, 275 op/s 2024-03-16T11:53:31.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:53:30 smithi029 ceph-mon[31975]: pgmap v80: 97 pgs: 97 active+clean; 41 MiB data, 607 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 2.1 MiB/s wr, 275 op/s 2024-03-16T11:53:31.778 INFO:tasks.workunit.client.0.smithi029.stdout: 1 176223 12.70 MB/sec warmup 76 sec latency 13.673 ms 2024-03-16T11:53:32.777 INFO:tasks.workunit.client.0.smithi029.stdout: 1 178026 12.69 MB/sec warmup 77 sec latency 33.472 ms 2024-03-16T11:53:32.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:53:32 smithi152 ceph-mon[36291]: pgmap v81: 97 pgs: 97 active+clean; 45 MiB data, 637 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 3.5 MiB/s wr, 378 op/s 2024-03-16T11:53:33.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:53:32 smithi154 ceph-mon[36320]: pgmap v81: 97 pgs: 97 active+clean; 45 MiB data, 637 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 3.5 MiB/s wr, 378 op/s 2024-03-16T11:53:33.087 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:53:32 smithi029 ceph-mon[31975]: pgmap v81: 97 pgs: 97 active+clean; 45 MiB data, 637 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 3.5 MiB/s wr, 378 op/s 2024-03-16T11:53:33.777 INFO:tasks.workunit.client.0.smithi029.stdout: 1 180867 12.75 MB/sec warmup 78 sec latency 69.962 ms 2024-03-16T11:53:34.777 INFO:tasks.workunit.client.0.smithi029.stdout: 1 182347 12.67 MB/sec warmup 79 sec latency 74.971 ms 2024-03-16T11:53:34.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:53:34 smithi152 ceph-mon[36291]: pgmap v82: 97 pgs: 97 active+clean; 45 MiB data, 637 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 2.9 MiB/s wr, 300 op/s 2024-03-16T11:53:35.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:53:34 smithi154 ceph-mon[36320]: pgmap v82: 97 pgs: 97 active+clean; 45 MiB data, 637 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 2.9 MiB/s wr, 300 op/s 2024-03-16T11:53:35.087 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:53:34 smithi029 ceph-mon[31975]: pgmap v82: 97 pgs: 97 active+clean; 45 MiB data, 637 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 2.9 MiB/s wr, 300 op/s 2024-03-16T11:53:35.777 INFO:tasks.workunit.client.0.smithi029.stdout: 1 184854 12.69 MB/sec warmup 80 sec latency 17.442 ms 2024-03-16T11:53:36.549 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:36.547+0000 7ffa6feb8640 1 -- 172.21.15.29:0/2961510281 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffa681077c0 msgr2=0x7ffa68107ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:53:36.549 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:36.547+0000 7ffa6feb8640 1 --2- 172.21.15.29:0/2961510281 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffa681077c0 0x7ffa68107ba0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7ffa64009930 tx=0x7ffa6402bc30 comp rx=0 tx=0).stop 2024-03-16T11:53:36.550 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:36.547+0000 7ffa6feb8640 1 -- 172.21.15.29:0/2961510281 shutdown_connections 2024-03-16T11:53:36.550 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:36.547+0000 7ffa6feb8640 1 --2- 172.21.15.29:0/2961510281 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffa68102170 0x7ffa6810e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:53:36.550 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:36.547+0000 7ffa6feb8640 1 --2- 172.21.15.29:0/2961510281 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ffa681017b0 0x7ffa68101c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:53:36.550 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:36.547+0000 7ffa6feb8640 1 --2- 172.21.15.29:0/2961510281 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffa681077c0 0x7ffa68107ba0 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:53:36.550 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:36.547+0000 7ffa6feb8640 1 -- 172.21.15.29:0/2961510281 >> 172.21.15.29:0/2961510281 conn(0x7ffa680fd2c0 msgr2=0x7ffa680ff6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:53:36.550 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:36.547+0000 7ffa6feb8640 1 -- 172.21.15.29:0/2961510281 shutdown_connections 2024-03-16T11:53:36.550 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:36.547+0000 7ffa6feb8640 1 -- 172.21.15.29:0/2961510281 wait complete. 2024-03-16T11:53:36.550 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:36.548+0000 7ffa6feb8640 1 Processor -- start 2024-03-16T11:53:36.550 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:36.548+0000 7ffa6feb8640 1 -- start start 2024-03-16T11:53:36.550 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:36.548+0000 7ffa6feb8640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffa681017b0 0x7ffa68198030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:53:36.550 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:36.548+0000 7ffa6feb8640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ffa68102170 0x7ffa68198570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:53:36.550 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:36.548+0000 7ffa6feb8640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffa681077c0 0x7ffa6819c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:53:36.550 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:36.548+0000 7ffa6feb8640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7ffa68111bd0 con 0x7ffa681017b0 2024-03-16T11:53:36.551 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:36.548+0000 7ffa6feb8640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7ffa68111a50 con 0x7ffa68102170 2024-03-16T11:53:36.551 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:36.548+0000 7ffa6feb8640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7ffa68111d50 con 0x7ffa681077c0 2024-03-16T11:53:36.551 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:36.548+0000 7ffa6dc54640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffa681017b0 0x7ffa68198030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:53:36.551 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:36.548+0000 7ffa6dc54640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffa681017b0 0x7ffa68198030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:35182/0 (socket says 172.21.15.29:35182) 2024-03-16T11:53:36.551 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:36.548+0000 7ffa6dc54640 1 -- 172.21.15.29:0/1079873252 learned_addr learned my addr 172.21.15.29:0/1079873252 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:53:36.551 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:36.548+0000 7ffa6d453640 1 --2- 172.21.15.29:0/1079873252 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ffa68102170 0x7ffa68198570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:53:36.551 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:36.548+0000 7ffa6e455640 1 --2- 172.21.15.29:0/1079873252 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffa681077c0 0x7ffa6819c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:53:36.551 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:36.548+0000 7ffa6dc54640 1 -- 172.21.15.29:0/1079873252 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffa681077c0 msgr2=0x7ffa6819c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:53:36.551 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:36.548+0000 7ffa6dc54640 1 --2- 172.21.15.29:0/1079873252 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffa681077c0 0x7ffa6819c8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:53:36.551 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:36.548+0000 7ffa6dc54640 1 -- 172.21.15.29:0/1079873252 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ffa68102170 msgr2=0x7ffa68198570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:53:36.551 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:36.548+0000 7ffa6dc54640 1 --2- 172.21.15.29:0/1079873252 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ffa68102170 0x7ffa68198570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:53:36.551 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:36.548+0000 7ffa6dc54640 1 -- 172.21.15.29:0/1079873252 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa6819d020 con 0x7ffa681017b0 2024-03-16T11:53:36.551 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:36.548+0000 7ffa6dc54640 1 --2- 172.21.15.29:0/1079873252 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffa681017b0 0x7ffa68198030 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7ffa64009230 tx=0x7ffa6402cb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:53:36.551 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:36.548+0000 7ffa5effd640 1 -- 172.21.15.29:0/1079873252 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa64041070 con 0x7ffa681017b0 2024-03-16T11:53:36.552 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:36.549+0000 7ffa5effd640 1 -- 172.21.15.29:0/1079873252 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffa64004450 con 0x7ffa681017b0 2024-03-16T11:53:36.552 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:36.549+0000 7ffa6feb8640 1 -- 172.21.15.29:0/1079873252 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffa6819d2b0 con 0x7ffa681017b0 2024-03-16T11:53:36.552 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:36.549+0000 7ffa5effd640 1 -- 172.21.15.29:0/1079873252 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa64003e50 con 0x7ffa681017b0 2024-03-16T11:53:36.552 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:36.549+0000 7ffa6feb8640 1 -- 172.21.15.29:0/1079873252 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffa681a4ba0 con 0x7ffa681017b0 2024-03-16T11:53:36.559 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:36.551+0000 7ffa5effd640 1 -- 172.21.15.29:0/1079873252 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7ffa64044070 con 0x7ffa681017b0 2024-03-16T11:53:36.559 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:36.551+0000 7ffa72b78740 1 Processor -- start 2024-03-16T11:53:36.559 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:36.551+0000 7ffa72b78740 1 -- start start 2024-03-16T11:53:36.559 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:36.551+0000 7ffa5effd640 1 --2- 172.21.15.29:0/1079873252 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7ffa540a2bb0 0x7ffa540a5070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:53:36.559 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:36.551+0000 7ffa72b78740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557ee5494130 0x557ee5494510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:53:36.559 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:36.551+0000 7ffa72b78740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557ee5494a50 0x557ee549d2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:53:36.559 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:36.552+0000 7ffa72b78740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557ee549d820 0x557ee549fc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:53:36.559 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:36.552+0000 7ffa5effd640 1 -- 172.21.15.29:0/1079873252 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8469+0+0 (secure 0 0 0) 0x7ffa640f05e0 con 0x7ffa681017b0 2024-03-16T11:53:36.559 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:36.552+0000 7ffa72b78740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x557ee53f8a40 con 0x557ee549d820 2024-03-16T11:53:36.559 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:36.552+0000 7ffa72b78740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x557ee53a1920 con 0x557ee5494130 2024-03-16T11:53:36.559 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:36.552+0000 7ffa72b78740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x557ee53f4ae0 con 0x557ee5494a50 2024-03-16T11:53:36.559 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:36.552+0000 7ffa6e455640 1 --2- 172.21.15.29:0/1079873252 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7ffa540a2bb0 0x7ffa540a5070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:53:36.560 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:36.552+0000 7ffa6d453640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557ee5494a50 0x557ee549d2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:53:36.560 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:36.552+0000 7ffa6d453640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557ee5494a50 0x557ee549d2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.29:33562/0 (socket says 172.21.15.29:33562) 2024-03-16T11:53:36.560 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:36.552+0000 7ffa6dc54640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557ee5494130 0x557ee5494510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:53:36.560 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:36.552+0000 7ffa6d453640 1 -- 172.21.15.29:0/1584474718 learned_addr learned my addr 172.21.15.29:0/1584474718 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:53:36.560 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:36.552+0000 7ffa6e455640 1 --2- 172.21.15.29:0/1584474718 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557ee549d820 0x557ee549fc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:53:36.560 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:36.552+0000 7ffa6d453640 1 -- 172.21.15.29:0/1584474718 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557ee5494130 msgr2=0x557ee5494510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:53:36.560 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:36.552+0000 7ffa6d453640 1 --2- 172.21.15.29:0/1584474718 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557ee5494130 0x557ee5494510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:53:36.560 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:36.552+0000 7ffa6d453640 1 -- 172.21.15.29:0/1584474718 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557ee549d820 msgr2=0x557ee549fc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:53:36.560 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:36.552+0000 7ffa6d453640 1 --2- 172.21.15.29:0/1584474718 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557ee549d820 0x557ee549fc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:53:36.560 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:36.552+0000 7ffa6d453640 1 -- 172.21.15.29:0/1584474718 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557ee53b9460 con 0x557ee5494a50 2024-03-16T11:53:36.560 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:36.552+0000 7ffa6e455640 1 --2- 172.21.15.29:0/1079873252 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7ffa540a2bb0 0x7ffa540a5070 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7ffa60005fd0 tx=0x7ffa60013ae0 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:53:36.560 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:36.552+0000 7ffa6d453640 1 --2- 172.21.15.29:0/1584474718 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557ee5494a50 0x557ee549d2e0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7ffa58009c90 tx=0x7ffa58011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a6c61ab994f35e61 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:53:36.560 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:36.552+0000 7ffa53fff640 1 -- 172.21.15.29:0/1584474718 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa5801e070 con 0x557ee5494a50 2024-03-16T11:53:36.560 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:36.552+0000 7ffa53fff640 1 -- 172.21.15.29:0/1584474718 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffa58019b10 con 0x557ee5494a50 2024-03-16T11:53:36.561 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:36.552+0000 7ffa53fff640 1 -- 172.21.15.29:0/1584474718 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa5800ea00 con 0x557ee5494a50 2024-03-16T11:53:36.561 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:36.553+0000 7ffa72b78740 1 -- 172.21.15.29:0/1584474718 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557ee5494a50 msgr2=0x557ee549d2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:53:36.561 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:36.553+0000 7ffa72b78740 1 --2- 172.21.15.29:0/1584474718 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557ee5494a50 0x557ee549d2e0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7ffa58009c90 tx=0x7ffa58011810 comp rx=0 tx=0).stop 2024-03-16T11:53:36.561 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:36.553+0000 7ffa72b78740 1 -- 172.21.15.29:0/1584474718 shutdown_connections 2024-03-16T11:53:36.561 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:36.553+0000 7ffa72b78740 1 --2- 172.21.15.29:0/1584474718 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557ee549d820 0x557ee549fc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:53:36.561 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:36.553+0000 7ffa72b78740 1 --2- 172.21.15.29:0/1584474718 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557ee5494a50 0x557ee549d2e0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:53:36.561 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:36.553+0000 7ffa72b78740 1 --2- 172.21.15.29:0/1584474718 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557ee5494130 0x557ee5494510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:53:36.561 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:36.553+0000 7ffa72b78740 1 -- 172.21.15.29:0/1584474718 >> 172.21.15.29:0/1584474718 conn(0x557ee548ea50 msgr2=0x557ee548ee30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:53:36.561 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:36.553+0000 7ffa72b78740 1 -- 172.21.15.29:0/1584474718 shutdown_connections 2024-03-16T11:53:36.561 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:36.553+0000 7ffa72b78740 1 -- 172.21.15.29:0/1584474718 wait complete. 2024-03-16T11:53:36.561 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:36.554+0000 7ffa72b78740 1 -- start start 2024-03-16T11:53:36.561 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:36.554+0000 7ffa72b78740 1 Processor -- start 2024-03-16T11:53:36.561 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:36.554+0000 7ffa72b78740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557ee5494130 0x557ee549a830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:53:36.561 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:36.554+0000 7ffa72b78740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557ee5494a50 0x557ee549ad70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:53:36.562 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:36.554+0000 7ffa72b78740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557ee549d820 0x557ee549b2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:53:36.562 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:36.554+0000 7ffa72b78740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x557ee54a4860 con 0x557ee5494130 2024-03-16T11:53:36.562 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:36.554+0000 7ffa72b78740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x557ee53b4d70 con 0x557ee5494a50 2024-03-16T11:53:36.562 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:36.554+0000 7ffa72b78740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x557ee54a49e0 con 0x557ee549d820 2024-03-16T11:53:36.562 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:36.554+0000 7ffa72b78740 10 client.0 _ll_register_callbacks cb 0x557ee510ca00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T11:53:36.562 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:36.554+0000 7ffa6dc54640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557ee5494130 0x557ee549a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:53:36.562 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:36.554+0000 7ffa6d453640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557ee5494a50 0x557ee549ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:53:36.562 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:36.554+0000 7ffa6dc54640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557ee5494130 0x557ee549a830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:35188/0 (socket says 172.21.15.29:35188) 2024-03-16T11:53:36.562 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:36.554+0000 7ffa6e455640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557ee549d820 0x557ee549b2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:53:36.562 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:36.554+0000 7ffa6dc54640 1 -- 172.21.15.29:0/2279945704 learned_addr learned my addr 172.21.15.29:0/2279945704 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:53:36.562 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:36.554+0000 7ffa6dc54640 1 -- 172.21.15.29:0/2279945704 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557ee549d820 msgr2=0x557ee549b2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:53:36.562 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:36.554+0000 7ffa6dc54640 1 --2- 172.21.15.29:0/2279945704 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557ee549d820 0x557ee549b2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:53:36.562 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:36.554+0000 7ffa6dc54640 1 -- 172.21.15.29:0/2279945704 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557ee5494a50 msgr2=0x557ee549ad70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:53:36.563 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:36.554+0000 7ffa6dc54640 1 --2- 172.21.15.29:0/2279945704 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557ee5494a50 0x557ee549ad70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:53:36.563 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:36.554+0000 7ffa6dc54640 1 -- 172.21.15.29:0/2279945704 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557ee510d210 con 0x557ee5494130 2024-03-16T11:53:36.563 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:36.554+0000 7ffa6e455640 1 --2- 172.21.15.29:0/2279945704 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557ee549d820 0x557ee549b2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:53:36.563 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:36.555+0000 7ffa6dc54640 1 --2- 172.21.15.29:0/2279945704 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557ee5494130 0x557ee549a830 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7ffa640bcdc0 tx=0x7ffa640bca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1c5b8c3f0b62623 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:53:36.563 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:36.555+0000 7ffa517fa640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-16T11:53:36.563 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:36.555+0000 7ffa517fa640 1 -- 172.21.15.29:0/2279945704 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa6404b040 con 0x557ee5494130 2024-03-16T11:53:36.563 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:36.555+0000 7ffa517fa640 1 -- 172.21.15.29:0/2279945704 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8469+0+0 (secure 0 0 0) 0x7ffa640c5090 con 0x557ee5494130 2024-03-16T11:53:36.563 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:36.555+0000 7ffa72b78740 1 -- 172.21.15.29:0/2279945704 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ffa58019b10 con 0x557ee5494130 2024-03-16T11:53:36.563 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:36.555+0000 7ffa517fa640 1 -- 172.21.15.29:0/2279945704 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7ffa640bce30 con 0x557ee5494130 2024-03-16T11:53:36.563 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:36.555+0000 7ffa72b78740 10 client.15624 fetch_fsmap learned FSMap version 22 2024-03-16T11:53:36.563 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:36.555+0000 7ffa72b78740 1 -- 172.21.15.29:0/2279945704 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x557ee51d9620 con 0x557ee5494130 2024-03-16T11:53:36.563 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:36.555+0000 7ffa517fa640 1 -- 172.21.15.29:0/2279945704 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffa58019b10 con 0x557ee5494130 2024-03-16T11:53:36.564 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:36.555+0000 7ffa517fa640 1 -- 172.21.15.29:0/2279945704 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7ffa640d5030 con 0x557ee5494130 2024-03-16T11:53:36.564 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:36.555+0000 7ffa72b78740 10 client.15624 fetch_fsmap finished waiting for FSMap version 22 2024-03-16T11:53:36.564 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:36.555+0000 7ffa72b78740 10 client.15624 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T11:53:36.564 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:36.555+0000 7ffa72b78740 20 client.15624 populate_metadata read hostname 'smithi029' 2024-03-16T11:53:36.564 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:36.555+0000 7ffa72b78740 1 --2- 172.21.15.29:0/2279945704 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x557ee54b1df0 0x557ee54b4220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:53:36.564 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:36.555+0000 7ffa72b78740 4 client.15624 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T11:53:36.564 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:36.555+0000 7ffa72b78740 1 -- 172.21.15.29:0/2279945704 --> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557ee5405f80 con 0x557ee54b1df0 2024-03-16T11:53:36.564 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:36.556+0000 7ffa6d453640 1 --2- 172.21.15.29:0/2279945704 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x557ee54b1df0 0x557ee54b4220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:53:36.564 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:36.556+0000 7ffa6d453640 1 --2- 172.21.15.29:0/2279945704 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x557ee54b1df0 0x557ee54b4220 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bc7f87373f114b16 server_cookie=9a56080622fc0f3c in_seq=0 out_seq=0 2024-03-16T11:53:36.564 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:36.556+0000 7ffa517fa640 10 client.15624 ms_handle_connect on v2:172.21.15.152:6838/1548731833 2024-03-16T11:53:36.564 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:36.557+0000 7ffa517fa640 1 -- 172.21.15.29:0/2279945704 <== mds.0 v2:172.21.15.152:6838/1548731833 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x557ee5405f80 con 0x557ee54b1df0 2024-03-16T11:53:36.564 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:36.557+0000 7ffa517fa640 10 client.15624 handle_command_reply: tid=0 2024-03-16T11:53:36.564 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:36.557+0000 7ffa72b78740 1 client.15624 shutdown 2024-03-16T11:53:36.564 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:36.557+0000 7ffa72b78740 1 -- 172.21.15.29:0/2279945704 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557ee5494130 msgr2=0x557ee549a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:53:36.564 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:36.557+0000 7ffa72b78740 1 --2- 172.21.15.29:0/2279945704 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557ee5494130 0x557ee549a830 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7ffa640bcdc0 tx=0x7ffa640bca30 comp rx=0 tx=0).stop 2024-03-16T11:53:36.565 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:36.557+0000 7ffa72b78740 1 -- 172.21.15.29:0/2279945704 shutdown_connections 2024-03-16T11:53:36.565 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:36.557+0000 7ffa72b78740 1 --2- 172.21.15.29:0/2279945704 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557ee549d820 0x557ee549b2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:53:36.565 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:36.557+0000 7ffa72b78740 1 --2- 172.21.15.29:0/2279945704 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x557ee54b1df0 0x557ee54b4220 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:53:36.565 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:36.557+0000 7ffa72b78740 1 --2- 172.21.15.29:0/2279945704 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557ee5494a50 0x557ee549ad70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:53:36.565 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:36.557+0000 7ffa72b78740 1 --2- 172.21.15.29:0/2279945704 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557ee5494130 0x557ee549a830 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:53:36.565 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:36.557+0000 7ffa72b78740 1 -- 172.21.15.29:0/2279945704 >> 172.21.15.29:0/2279945704 conn(0x557ee548ea50 msgr2=0x557ee54907a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:53:36.565 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:36.557+0000 7ffa517fa640 0 client.15624 ms_handle_reset on v2:172.21.15.152:6838/1548731833 2024-03-16T11:53:36.565 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:36.557+0000 7ffa72b78740 1 -- 172.21.15.29:0/2279945704 shutdown_connections 2024-03-16T11:53:36.565 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:36.557+0000 7ffa72b78740 1 -- 172.21.15.29:0/2279945704 wait complete. 2024-03-16T11:53:36.565 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:36.557+0000 7ffa72b78740 20 client.15624 trim_cache size 0 max 16384 2024-03-16T11:53:36.578 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:36.572+0000 7ffa72b78740 1 Processor -- start 2024-03-16T11:53:36.579 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:36.572+0000 7ffa72b78740 1 -- start start 2024-03-16T11:53:36.579 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:36.572+0000 7ffa72b78740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557ee5494130 0x557ee549cf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:53:36.579 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:36.572+0000 7ffa72b78740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557ee5494a50 0x557ee5495060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:53:36.579 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:36.572+0000 7ffa72b78740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557ee549d820 0x557ee54ad630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:53:36.579 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:36.572+0000 7ffa72b78740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7ffa681ada90 con 0x557ee5494a50 2024-03-16T11:53:36.579 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:36.572+0000 7ffa72b78740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7ffa681ab360 con 0x557ee549d820 2024-03-16T11:53:36.579 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:36.572+0000 7ffa72b78740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7ffa681a91b0 con 0x557ee5494130 2024-03-16T11:53:36.579 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:36.573+0000 7ffa6d453640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557ee5494a50 0x557ee5495060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:53:36.579 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:36.573+0000 7ffa6dc54640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557ee5494130 0x557ee549cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:53:36.579 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:36.573+0000 7ffa6d453640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557ee5494a50 0x557ee5495060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:35190/0 (socket says 172.21.15.29:35190) 2024-03-16T11:53:36.580 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:36.573+0000 7ffa6d453640 1 -- 172.21.15.29:0/3751420315 learned_addr learned my addr 172.21.15.29:0/3751420315 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:53:36.580 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:36.573+0000 7ffa6e455640 1 --2- 172.21.15.29:0/3751420315 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557ee549d820 0x557ee54ad630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:53:36.580 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:36.573+0000 7ffa6d453640 1 -- 172.21.15.29:0/3751420315 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557ee5494130 msgr2=0x557ee549cf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:53:36.580 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:36.573+0000 7ffa6d453640 1 --2- 172.21.15.29:0/3751420315 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557ee5494130 0x557ee549cf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:53:36.581 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:36.573+0000 7ffa6d453640 1 -- 172.21.15.29:0/3751420315 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557ee549d820 msgr2=0x557ee54ad630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:53:36.581 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:36.573+0000 7ffa6d453640 1 --2- 172.21.15.29:0/3751420315 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557ee549d820 0x557ee54ad630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:53:36.581 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:36.573+0000 7ffa6d453640 1 -- 172.21.15.29:0/3751420315 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557ee548e580 con 0x557ee5494a50 2024-03-16T11:53:36.581 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:36.573+0000 7ffa6e455640 1 --2- 172.21.15.29:0/3751420315 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557ee549d820 0x557ee54ad630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T11:53:36.581 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:36.573+0000 7ffa6d453640 1 --2- 172.21.15.29:0/3751420315 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557ee5494a50 0x557ee5495060 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7ffa5800e610 tx=0x7ffa5800e560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b779be3f558e463f server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:53:36.581 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:36.573+0000 7ffa537fe640 1 -- 172.21.15.29:0/3751420315 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa5801e040 con 0x557ee5494a50 2024-03-16T11:53:36.581 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:36.573+0000 7ffa537fe640 1 -- 172.21.15.29:0/3751420315 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffa5801f040 con 0x557ee5494a50 2024-03-16T11:53:36.581 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:36.573+0000 7ffa537fe640 1 -- 172.21.15.29:0/3751420315 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa580052a0 con 0x557ee5494a50 2024-03-16T11:53:36.581 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:36.574+0000 7ffa72b78740 1 -- 172.21.15.29:0/3751420315 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557ee5494a50 msgr2=0x557ee5495060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:53:36.582 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:36.574+0000 7ffa72b78740 1 --2- 172.21.15.29:0/3751420315 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557ee5494a50 0x557ee5495060 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7ffa5800e610 tx=0x7ffa5800e560 comp rx=0 tx=0).stop 2024-03-16T11:53:36.582 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:36.574+0000 7ffa72b78740 1 -- 172.21.15.29:0/3751420315 shutdown_connections 2024-03-16T11:53:36.582 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:36.574+0000 7ffa72b78740 1 --2- 172.21.15.29:0/3751420315 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557ee549d820 0x557ee54ad630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:53:36.582 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:36.574+0000 7ffa72b78740 1 --2- 172.21.15.29:0/3751420315 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557ee5494a50 0x557ee5495060 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:53:36.582 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:36.574+0000 7ffa72b78740 1 --2- 172.21.15.29:0/3751420315 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557ee5494130 0x557ee549cf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:53:36.583 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:36.574+0000 7ffa72b78740 1 -- 172.21.15.29:0/3751420315 >> 172.21.15.29:0/3751420315 conn(0x557ee548ea50 msgr2=0x557ee548fea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:53:36.583 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:36.574+0000 7ffa72b78740 1 -- 172.21.15.29:0/3751420315 shutdown_connections 2024-03-16T11:53:36.583 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:36.574+0000 7ffa72b78740 1 -- 172.21.15.29:0/3751420315 wait complete. 2024-03-16T11:53:36.583 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:36.574+0000 7ffa72b78740 1 -- start start 2024-03-16T11:53:36.583 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:36.575+0000 7ffa72b78740 1 Processor -- start 2024-03-16T11:53:36.583 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:36.575+0000 7ffa72b78740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557ee5494130 0x557ee54a9c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:53:36.583 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:36.575+0000 7ffa72b78740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557ee5494a50 0x557ee54a6c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:53:36.583 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:36.575+0000 7ffa72b78740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557ee549d820 0x557ee54a7170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:53:36.583 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:36.575+0000 7ffa72b78740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7ffa681a9820 con 0x557ee5494130 2024-03-16T11:53:36.583 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:36.575+0000 7ffa72b78740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7ffa681aac90 con 0x557ee5494a50 2024-03-16T11:53:36.583 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:36.575+0000 7ffa72b78740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7ffa681a9ef0 con 0x557ee549d820 2024-03-16T11:53:36.583 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:36.575+0000 7ffa72b78740 10 client.0 _ll_register_callbacks cb 0x7ffa64009880 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T11:53:36.584 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:36.575+0000 7ffa6dc54640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557ee5494130 0x557ee54a9c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:53:36.584 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:36.575+0000 7ffa6dc54640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557ee5494130 0x557ee54a9c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:35196/0 (socket says 172.21.15.29:35196) 2024-03-16T11:53:36.584 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:36.575+0000 7ffa6dc54640 1 -- 172.21.15.29:0/1696991379 learned_addr learned my addr 172.21.15.29:0/1696991379 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:53:36.584 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:36.575+0000 7ffa6e455640 1 --2- 172.21.15.29:0/1696991379 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557ee549d820 0x557ee54a7170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:53:36.584 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:36.575+0000 7ffa6d453640 1 --2- 172.21.15.29:0/1696991379 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557ee5494a50 0x557ee54a6c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:53:36.584 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:36.575+0000 7ffa6dc54640 1 -- 172.21.15.29:0/1696991379 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557ee549d820 msgr2=0x557ee54a7170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:53:36.584 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:36.575+0000 7ffa6dc54640 1 --2- 172.21.15.29:0/1696991379 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557ee549d820 0x557ee54a7170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:53:36.584 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:36.575+0000 7ffa6dc54640 1 -- 172.21.15.29:0/1696991379 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557ee5494a50 msgr2=0x557ee54a6c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:53:36.584 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:36.575+0000 7ffa6dc54640 1 --2- 172.21.15.29:0/1696991379 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557ee5494a50 0x557ee54a6c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:53:36.584 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:36.575+0000 7ffa6dc54640 1 -- 172.21.15.29:0/1696991379 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557ee54aa410 con 0x557ee5494130 2024-03-16T11:53:36.584 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:36.576+0000 7ffa6dc54640 1 --2- 172.21.15.29:0/1696991379 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557ee5494130 0x557ee54a9c60 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7ffa640bc3f0 tx=0x7ffa64000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fbeeadedf602e55a server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:53:36.584 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:36.576+0000 7ffa517fa640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-16T11:53:36.584 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:36.576+0000 7ffa517fa640 1 -- 172.21.15.29:0/1696991379 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa6404b040 con 0x557ee5494130 2024-03-16T11:53:36.584 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:36.576+0000 7ffa72b78740 1 -- 172.21.15.29:0/1696991379 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ffa5801f040 con 0x557ee5494130 2024-03-16T11:53:36.585 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:36.576+0000 7ffa517fa640 1 -- 172.21.15.29:0/1696991379 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8469+0+0 (secure 0 0 0) 0x7ffa640a60b0 con 0x557ee5494130 2024-03-16T11:53:36.585 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:36.576+0000 7ffa517fa640 1 -- 172.21.15.29:0/1696991379 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7ffa640cc050 con 0x557ee5494130 2024-03-16T11:53:36.585 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:36.576+0000 7ffa72b78740 10 client.15630 fetch_fsmap learned FSMap version 22 2024-03-16T11:53:36.585 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:36.576+0000 7ffa72b78740 1 -- 172.21.15.29:0/1696991379 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x557ee54aa410 con 0x557ee5494130 2024-03-16T11:53:36.585 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:36.576+0000 7ffa517fa640 1 -- 172.21.15.29:0/1696991379 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffa5801f040 con 0x557ee5494130 2024-03-16T11:53:36.585 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:36.576+0000 7ffa517fa640 1 -- 172.21.15.29:0/1696991379 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7ffa640a2020 con 0x557ee5494130 2024-03-16T11:53:36.585 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:36.576+0000 7ffa72b78740 10 client.15630 fetch_fsmap finished waiting for FSMap version 22 2024-03-16T11:53:36.585 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:36.576+0000 7ffa72b78740 10 client.15630 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T11:53:36.585 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:36.576+0000 7ffa72b78740 20 client.15630 populate_metadata read hostname 'smithi029' 2024-03-16T11:53:36.585 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:36.576+0000 7ffa72b78740 1 --2- 172.21.15.29:0/1696991379 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x557ee5519010 0x557ee551b400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:53:36.585 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:36.576+0000 7ffa72b78740 4 client.15630 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-16T11:53:36.585 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:36.576+0000 7ffa72b78740 1 -- 172.21.15.29:0/1696991379 --> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] -- command(tid 0: {"prefix": "flush journal"}) -- 0x557ee54002d0 con 0x557ee5519010 2024-03-16T11:53:36.585 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:36.577+0000 7ffa6d453640 1 --2- 172.21.15.29:0/1696991379 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x557ee5519010 0x557ee551b400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:53:36.585 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:36.577+0000 7ffa6d453640 1 --2- 172.21.15.29:0/1696991379 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x557ee5519010 0x557ee551b400 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c2e394a3cadbfa28 server_cookie=6392e18971e03ca1 in_seq=0 out_seq=0 2024-03-16T11:53:36.585 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:36.577+0000 7ffa517fa640 10 client.15630 ms_handle_connect on v2:172.21.15.152:6838/1548731833 2024-03-16T11:53:36.714 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:36.713+0000 7ffa517fa640 1 -- 172.21.15.29:0/1696991379 <== mds.0 v2:172.21.15.152:6838/1548731833 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x557ee54002d0 con 0x557ee5519010 2024-03-16T11:53:36.714 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:36.713+0000 7ffa517fa640 10 client.15630 handle_command_reply: tid=0 2024-03-16T11:53:36.714 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:36.713+0000 7ffa72b78740 1 client.15630 shutdown 2024-03-16T11:53:36.714 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:36.713+0000 7ffa72b78740 1 -- 172.21.15.29:0/1696991379 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557ee5494130 msgr2=0x557ee54a9c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:53:36.714 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:36.713+0000 7ffa72b78740 1 --2- 172.21.15.29:0/1696991379 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557ee5494130 0x557ee54a9c60 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7ffa640bc3f0 tx=0x7ffa64000bc0 comp rx=0 tx=0).stop 2024-03-16T11:53:36.715 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:36.713+0000 7ffa72b78740 1 -- 172.21.15.29:0/1696991379 shutdown_connections 2024-03-16T11:53:36.715 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:36.713+0000 7ffa72b78740 1 --2- 172.21.15.29:0/1696991379 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557ee549d820 0x557ee54a7170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:53:36.715 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:36.713+0000 7ffa72b78740 1 --2- 172.21.15.29:0/1696991379 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x557ee5519010 0x557ee551b400 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:53:36.715 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:36.713+0000 7ffa72b78740 1 --2- 172.21.15.29:0/1696991379 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557ee5494a50 0x557ee54a6c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:53:36.715 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:36.713+0000 7ffa517fa640 0 client.15630 ms_handle_reset on v2:172.21.15.152:6838/1548731833 2024-03-16T11:53:36.715 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:36.713+0000 7ffa72b78740 1 --2- 172.21.15.29:0/1696991379 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557ee5494130 0x557ee54a9c60 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:53:36.715 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:36.713+0000 7ffa72b78740 1 -- 172.21.15.29:0/1696991379 >> 172.21.15.29:0/1696991379 conn(0x557ee548ea50 msgr2=0x557ee548dbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:53:36.715 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:36.713+0000 7ffa72b78740 1 -- 172.21.15.29:0/1696991379 shutdown_connections 2024-03-16T11:53:36.715 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:36.713+0000 7ffa72b78740 1 -- 172.21.15.29:0/1696991379 wait complete. 2024-03-16T11:53:36.715 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:36.713+0000 7ffa72b78740 20 client.15630 trim_cache size 0 max 16384 2024-03-16T11:53:36.715 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:{ 2024-03-16T11:53:36.716 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "message": "", 2024-03-16T11:53:36.716 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "return_code": 0 2024-03-16T11:53:36.716 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:} 2024-03-16T11:53:36.717 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:36.714+0000 7ffa6feb8640 1 -- 172.21.15.29:0/1079873252 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7ffa540a2bb0 msgr2=0x7ffa540a5070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:53:36.717 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:36.714+0000 7ffa6feb8640 1 --2- 172.21.15.29:0/1079873252 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7ffa540a2bb0 0x7ffa540a5070 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7ffa60005fd0 tx=0x7ffa60013ae0 comp rx=0 tx=0).stop 2024-03-16T11:53:36.717 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:36.714+0000 7ffa6feb8640 1 -- 172.21.15.29:0/1079873252 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffa681017b0 msgr2=0x7ffa68198030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:53:36.717 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:36.714+0000 7ffa6feb8640 1 --2- 172.21.15.29:0/1079873252 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffa681017b0 0x7ffa68198030 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7ffa64009230 tx=0x7ffa6402cb30 comp rx=0 tx=0).stop 2024-03-16T11:53:36.717 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:36.714+0000 7ffa6feb8640 1 -- 172.21.15.29:0/1079873252 shutdown_connections 2024-03-16T11:53:36.717 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:36.714+0000 7ffa6feb8640 1 --2- 172.21.15.29:0/1079873252 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7ffa540a2bb0 0x7ffa540a5070 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:53:36.717 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:36.714+0000 7ffa6feb8640 1 --2- 172.21.15.29:0/1079873252 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffa681077c0 0x7ffa6819c8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:53:36.717 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:36.714+0000 7ffa6feb8640 1 --2- 172.21.15.29:0/1079873252 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ffa68102170 0x7ffa68198570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:53:36.717 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:36.714+0000 7ffa6feb8640 1 --2- 172.21.15.29:0/1079873252 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffa681017b0 0x7ffa68198030 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:53:36.717 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:36.714+0000 7ffa6feb8640 1 -- 172.21.15.29:0/1079873252 >> 172.21.15.29:0/1079873252 conn(0x7ffa680fd2c0 msgr2=0x7ffa680fd880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:53:36.717 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:36.715+0000 7ffa6feb8640 1 -- 172.21.15.29:0/1079873252 shutdown_connections 2024-03-16T11:53:36.717 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:36.715+0000 7ffa6feb8640 1 -- 172.21.15.29:0/1079873252 wait complete. 2024-03-16T11:53:36.777 INFO:tasks.workunit.client.0.smithi029.stdout: 1 186946 12.63 MB/sec warmup 81 sec latency 78.363 ms 2024-03-16T11:53:37.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:53:36 smithi154 ceph-mon[36320]: pgmap v83: 97 pgs: 97 active+clean; 45 MiB data, 637 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 2.9 MiB/s wr, 300 op/s 2024-03-16T11:53:37.087 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:53:36 smithi029 ceph-mon[31975]: pgmap v83: 97 pgs: 97 active+clean; 45 MiB data, 637 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 2.9 MiB/s wr, 300 op/s 2024-03-16T11:53:37.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:53:36 smithi152 ceph-mon[36291]: pgmap v83: 97 pgs: 97 active+clean; 45 MiB data, 637 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 2.9 MiB/s wr, 300 op/s 2024-03-16T11:53:37.777 INFO:tasks.workunit.client.0.smithi029.stdout: 1 188919 12.62 MB/sec warmup 82 sec latency 22.353 ms 2024-03-16T11:53:38.777 INFO:tasks.workunit.client.0.smithi029.stdout: 1 191556 12.67 MB/sec warmup 83 sec latency 66.978 ms 2024-03-16T11:53:39.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:53:38 smithi154 ceph-mon[36320]: pgmap v84: 97 pgs: 97 active+clean; 50 MiB data, 643 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 4.1 MiB/s wr, 402 op/s 2024-03-16T11:53:39.087 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:53:38 smithi029 ceph-mon[31975]: pgmap v84: 97 pgs: 97 active+clean; 50 MiB data, 643 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 4.1 MiB/s wr, 402 op/s 2024-03-16T11:53:39.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:53:38 smithi152 ceph-mon[36291]: pgmap v84: 97 pgs: 97 active+clean; 50 MiB data, 643 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 4.1 MiB/s wr, 402 op/s 2024-03-16T11:53:39.778 INFO:tasks.workunit.client.0.smithi029.stdout: 1 193242 12.61 MB/sec warmup 84 sec latency 14.700 ms 2024-03-16T11:53:40.777 INFO:tasks.workunit.client.0.smithi029.stdout: 1 195660 12.62 MB/sec warmup 85 sec latency 9.737 ms 2024-03-16T11:53:41.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:53:40 smithi154 ceph-mon[36320]: pgmap v85: 97 pgs: 97 active+clean; 50 MiB data, 643 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 2.6 MiB/s wr, 205 op/s 2024-03-16T11:53:41.087 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:53:40 smithi029 ceph-mon[31975]: pgmap v85: 97 pgs: 97 active+clean; 50 MiB data, 643 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 2.6 MiB/s wr, 205 op/s 2024-03-16T11:53:41.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:53:40 smithi152 ceph-mon[36291]: pgmap v85: 97 pgs: 97 active+clean; 50 MiB data, 643 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 2.6 MiB/s wr, 205 op/s 2024-03-16T11:53:41.778 INFO:tasks.workunit.client.0.smithi029.stdout: 1 198469 12.64 MB/sec warmup 86 sec latency 16.225 ms 2024-03-16T11:53:42.778 INFO:tasks.workunit.client.0.smithi029.stdout: 1 200135 12.61 MB/sec warmup 87 sec latency 25.745 ms 2024-03-16T11:53:43.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:53:42 smithi154 ceph-mon[36320]: pgmap v86: 97 pgs: 97 active+clean; 42 MiB data, 667 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 4.0 MiB/s wr, 403 op/s 2024-03-16T11:53:43.087 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:53:42 smithi029 ceph-mon[31975]: pgmap v86: 97 pgs: 97 active+clean; 42 MiB data, 667 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 4.0 MiB/s wr, 403 op/s 2024-03-16T11:53:43.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:53:42 smithi152 ceph-mon[36291]: pgmap v86: 97 pgs: 97 active+clean; 42 MiB data, 667 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 4.0 MiB/s wr, 403 op/s 2024-03-16T11:53:43.778 INFO:tasks.workunit.client.0.smithi029.stdout: 1 202403 12.62 MB/sec warmup 88 sec latency 66.887 ms 2024-03-16T11:53:44.778 INFO:tasks.workunit.client.0.smithi029.stdout: 1 204576 12.56 MB/sec warmup 89 sec latency 22.344 ms 2024-03-16T11:53:44.929 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:53:44 smithi152 ceph-mon[36291]: pgmap v87: 97 pgs: 97 active+clean; 42 MiB data, 667 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 2.5 MiB/s wr, 300 op/s 2024-03-16T11:53:45.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:53:44 smithi154 ceph-mon[36320]: pgmap v87: 97 pgs: 97 active+clean; 42 MiB data, 667 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 2.5 MiB/s wr, 300 op/s 2024-03-16T11:53:45.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:53:44 smithi029 ceph-mon[31975]: pgmap v87: 97 pgs: 97 active+clean; 42 MiB data, 667 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 2.5 MiB/s wr, 300 op/s 2024-03-16T11:53:45.778 INFO:tasks.workunit.client.0.smithi029.stdout: 1 206471 12.56 MB/sec warmup 90 sec latency 18.952 ms 2024-03-16T11:53:46.778 INFO:tasks.workunit.client.0.smithi029.stdout: 1 209407 12.61 MB/sec warmup 91 sec latency 17.597 ms 2024-03-16T11:53:47.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:53:46 smithi154 ceph-mon[36320]: pgmap v88: 97 pgs: 97 active+clean; 42 MiB data, 667 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 2.5 MiB/s wr, 300 op/s 2024-03-16T11:53:47.087 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:53:46 smithi029 ceph-mon[31975]: pgmap v88: 97 pgs: 97 active+clean; 42 MiB data, 667 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 2.5 MiB/s wr, 300 op/s 2024-03-16T11:53:47.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:53:46 smithi152 ceph-mon[36291]: pgmap v88: 97 pgs: 97 active+clean; 42 MiB data, 667 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 2.5 MiB/s wr, 300 op/s 2024-03-16T11:53:47.778 INFO:tasks.workunit.client.0.smithi029.stdout: 1 211668 12.57 MB/sec warmup 92 sec latency 22.267 ms 2024-03-16T11:53:48.778 INFO:tasks.workunit.client.0.smithi029.stdout: 1 213291 12.56 MB/sec warmup 93 sec latency 98.221 ms 2024-03-16T11:53:49.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:53:48 smithi154 ceph-mon[36320]: pgmap v89: 97 pgs: 97 active+clean; 46 MiB data, 673 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 4.0 MiB/s wr, 407 op/s 2024-03-16T11:53:49.087 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:53:48 smithi029 ceph-mon[31975]: pgmap v89: 97 pgs: 97 active+clean; 46 MiB data, 673 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 4.0 MiB/s wr, 407 op/s 2024-03-16T11:53:49.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:53:48 smithi152 ceph-mon[36291]: pgmap v89: 97 pgs: 97 active+clean; 46 MiB data, 673 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 4.0 MiB/s wr, 407 op/s 2024-03-16T11:53:49.778 INFO:tasks.workunit.client.0.smithi029.stdout: 1 216211 12.58 MB/sec warmup 94 sec latency 16.008 ms 2024-03-16T11:53:49.806 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:49.805+0000 7eff03fcd640 1 -- 172.21.15.29:0/2732227439 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efefc107850 msgr2=0x7efefc107c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:53:49.806 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:49.805+0000 7eff03fcd640 1 --2- 172.21.15.29:0/2732227439 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efefc107850 0x7efefc107c30 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7efef8009900 tx=0x7efef802bc30 comp rx=0 tx=0).stop 2024-03-16T11:53:49.806 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:49.805+0000 7eff03fcd640 1 -- 172.21.15.29:0/2732227439 shutdown_connections 2024-03-16T11:53:49.806 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:49.805+0000 7eff03fcd640 1 --2- 172.21.15.29:0/2732227439 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efefc069800 0x7efefc10e630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:53:49.806 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:49.805+0000 7eff03fcd640 1 --2- 172.21.15.29:0/2732227439 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7efefc068e60 0x7efefc0692c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:53:49.806 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:49.805+0000 7eff03fcd640 1 --2- 172.21.15.29:0/2732227439 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efefc107850 0x7efefc107c30 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:53:49.807 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:49.805+0000 7eff03fcd640 1 -- 172.21.15.29:0/2732227439 >> 172.21.15.29:0/2732227439 conn(0x7efefc077530 msgr2=0x7efefc077940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:53:49.807 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:49.805+0000 7eff03fcd640 1 -- 172.21.15.29:0/2732227439 shutdown_connections 2024-03-16T11:53:49.807 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:49.805+0000 7eff03fcd640 1 -- 172.21.15.29:0/2732227439 wait complete. 2024-03-16T11:53:49.807 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:49.806+0000 7eff03fcd640 1 Processor -- start 2024-03-16T11:53:49.807 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:49.806+0000 7eff03fcd640 1 -- start start 2024-03-16T11:53:49.807 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:49.806+0000 7eff03fcd640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7efefc068e60 0x7efefc197f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:53:49.807 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:49.806+0000 7eff03fcd640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efefc069800 0x7efefc1984b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:53:49.807 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:49.806+0000 7eff03fcd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efefc107850 0x7efefc19c830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:53:49.807 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:49.806+0000 7eff03fcd640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7efefc111b70 con 0x7efefc107850 2024-03-16T11:53:49.807 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:49.806+0000 7eff03fcd640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7efefc1119f0 con 0x7efefc068e60 2024-03-16T11:53:49.807 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:49.806+0000 7eff03fcd640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7efefc111cf0 con 0x7efefc069800 2024-03-16T11:53:49.807 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:49.806+0000 7eff0256a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efefc107850 0x7efefc19c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:53:49.807 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:49.806+0000 7eff0256a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efefc107850 0x7efefc19c830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47042/0 (socket says 172.21.15.29:47042) 2024-03-16T11:53:49.807 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:49.806+0000 7eff0256a640 1 -- 172.21.15.29:0/2718757983 learned_addr learned my addr 172.21.15.29:0/2718757983 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:53:49.808 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:49.806+0000 7eff01d69640 1 --2- 172.21.15.29:0/2718757983 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7efefc068e60 0x7efefc197f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:53:49.808 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:49.806+0000 7eff01568640 1 --2- 172.21.15.29:0/2718757983 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efefc069800 0x7efefc1984b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:53:49.808 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:49.806+0000 7eff0256a640 1 -- 172.21.15.29:0/2718757983 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efefc069800 msgr2=0x7efefc1984b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:53:49.808 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:49.806+0000 7eff0256a640 1 --2- 172.21.15.29:0/2718757983 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efefc069800 0x7efefc1984b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:53:49.808 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:49.806+0000 7eff0256a640 1 -- 172.21.15.29:0/2718757983 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7efefc068e60 msgr2=0x7efefc197f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:53:49.808 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:49.806+0000 7eff0256a640 1 --2- 172.21.15.29:0/2718757983 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7efefc068e60 0x7efefc197f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:53:49.808 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:49.806+0000 7eff0256a640 1 -- 172.21.15.29:0/2718757983 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efefc19cf60 con 0x7efefc107850 2024-03-16T11:53:49.808 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:49.806+0000 7eff0256a640 1 --2- 172.21.15.29:0/2718757983 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efefc107850 0x7efefc19c830 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7efef400ea60 tx=0x7efef400ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:53:49.808 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:49.807+0000 7efef2ffd640 1 -- 172.21.15.29:0/2718757983 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efef400ce40 con 0x7efefc107850 2024-03-16T11:53:49.808 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:49.807+0000 7efef2ffd640 1 -- 172.21.15.29:0/2718757983 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efef4005020 con 0x7efefc107850 2024-03-16T11:53:49.808 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:49.807+0000 7eff03fcd640 1 -- 172.21.15.29:0/2718757983 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efefc19d250 con 0x7efefc107850 2024-03-16T11:53:49.808 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:49.807+0000 7efef2ffd640 1 -- 172.21.15.29:0/2718757983 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efef4010690 con 0x7efefc107850 2024-03-16T11:53:49.808 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:49.807+0000 7eff03fcd640 1 -- 172.21.15.29:0/2718757983 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efefc1a4ae0 con 0x7efefc107850 2024-03-16T11:53:49.814 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:49.809+0000 7efef2ffd640 1 -- 172.21.15.29:0/2718757983 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7efef4005520 con 0x7efefc107850 2024-03-16T11:53:49.814 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:49.809+0000 7eff06bb8740 1 Processor -- start 2024-03-16T11:53:49.814 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:49.809+0000 7eff06bb8740 1 -- start start 2024-03-16T11:53:49.814 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:49.809+0000 7efef2ffd640 1 --2- 172.21.15.29:0/2718757983 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7efee80a2a10 0x7efee80a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:53:49.814 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:49.809+0000 7eff06bb8740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55605e871130 0x55605e871510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:53:49.814 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:49.809+0000 7eff06bb8740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55605e871a50 0x55605e87a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:53:49.815 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:49.809+0000 7eff06bb8740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55605e87a820 0x55605e87cc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:53:49.815 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:49.809+0000 7eff06bb8740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55605e7d5a40 con 0x55605e871a50 2024-03-16T11:53:49.815 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:49.809+0000 7eff06bb8740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x55605e77e920 con 0x55605e87a820 2024-03-16T11:53:49.815 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:49.809+0000 7eff06bb8740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55605e7d1ae0 con 0x55605e871130 2024-03-16T11:53:49.815 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:49.810+0000 7efef2ffd640 1 -- 172.21.15.29:0/2718757983 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8469+0+0 (secure 0 0 0) 0x7efef40cf2d0 con 0x7efefc107850 2024-03-16T11:53:49.815 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:49.810+0000 7eff01568640 1 --2- 172.21.15.29:0/2718757983 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7efee80a2a10 0x7efee80a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:53:49.815 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:49.810+0000 7eff01d69640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55605e871130 0x55605e871510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:53:49.815 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:49.810+0000 7eff01d69640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55605e871130 0x55605e871510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.29:45192/0 (socket says 172.21.15.29:45192) 2024-03-16T11:53:49.815 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:49.810+0000 7eff01d69640 1 -- 172.21.15.29:0/2546674311 learned_addr learned my addr 172.21.15.29:0/2546674311 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:53:49.815 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:49.810+0000 7eff0256a640 1 --2- 172.21.15.29:0/2546674311 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55605e87a820 0x55605e87cc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:53:49.815 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:49.810+0000 7eff01568640 1 --2- 172.21.15.29:0/2546674311 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55605e871a50 0x55605e87a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:53:49.815 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:49.810+0000 7eff01d69640 1 -- 172.21.15.29:0/2546674311 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55605e87a820 msgr2=0x55605e87cc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:53:49.815 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:49.810+0000 7eff01d69640 1 --2- 172.21.15.29:0/2546674311 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55605e87a820 0x55605e87cc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:53:49.815 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:49.810+0000 7eff01d69640 1 -- 172.21.15.29:0/2546674311 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55605e871a50 msgr2=0x55605e87a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:53:49.816 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:49.810+0000 7eff01d69640 1 --2- 172.21.15.29:0/2546674311 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55605e871a50 0x55605e87a2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:53:49.816 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:49.810+0000 7eff01d69640 1 -- 172.21.15.29:0/2546674311 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55605e796460 con 0x55605e871130 2024-03-16T11:53:49.816 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:49.810+0000 7eff01568640 1 --2- 172.21.15.29:0/2718757983 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7efee80a2a10 0x7efee80a4ed0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7efeec00bfd0 tx=0x7efeec012470 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:53:49.816 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:49.810+0000 7eff01d69640 1 --2- 172.21.15.29:0/2546674311 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55605e871130 0x55605e871510 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7efef802cfd0 tx=0x7efef8005c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8697e3b3a16f01fd server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:53:49.816 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:49.810+0000 7efee7fff640 1 -- 172.21.15.29:0/2546674311 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efef8042070 con 0x55605e871130 2024-03-16T11:53:49.816 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:49.810+0000 7efee7fff640 1 -- 172.21.15.29:0/2546674311 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efef802ca80 con 0x55605e871130 2024-03-16T11:53:49.816 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:49.810+0000 7efee7fff640 1 -- 172.21.15.29:0/2546674311 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efef8004f00 con 0x55605e871130 2024-03-16T11:53:49.816 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:49.811+0000 7eff06bb8740 1 -- 172.21.15.29:0/2546674311 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55605e871130 msgr2=0x55605e871510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:53:49.816 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:49.811+0000 7eff06bb8740 1 --2- 172.21.15.29:0/2546674311 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55605e871130 0x55605e871510 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7efef802cfd0 tx=0x7efef8005c00 comp rx=0 tx=0).stop 2024-03-16T11:53:49.816 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:49.811+0000 7eff06bb8740 1 -- 172.21.15.29:0/2546674311 shutdown_connections 2024-03-16T11:53:49.816 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:49.811+0000 7eff06bb8740 1 --2- 172.21.15.29:0/2546674311 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55605e87a820 0x55605e87cc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:53:49.816 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:49.811+0000 7eff06bb8740 1 --2- 172.21.15.29:0/2546674311 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55605e871a50 0x55605e87a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:53:49.816 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:49.811+0000 7eff06bb8740 1 --2- 172.21.15.29:0/2546674311 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55605e871130 0x55605e871510 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:53:49.816 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:49.811+0000 7eff06bb8740 1 -- 172.21.15.29:0/2546674311 >> 172.21.15.29:0/2546674311 conn(0x55605e86ba50 msgr2=0x55605e86be30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:53:49.817 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:49.811+0000 7eff06bb8740 1 -- 172.21.15.29:0/2546674311 shutdown_connections 2024-03-16T11:53:49.817 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:49.811+0000 7eff06bb8740 1 -- 172.21.15.29:0/2546674311 wait complete. 2024-03-16T11:53:49.817 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:49.811+0000 7eff06bb8740 1 -- start start 2024-03-16T11:53:49.817 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:49.812+0000 7eff06bb8740 1 Processor -- start 2024-03-16T11:53:49.817 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:49.812+0000 7eff06bb8740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55605e871130 0x55605e8777f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:53:49.817 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:49.812+0000 7eff06bb8740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55605e871a50 0x55605e877d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:53:49.817 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:49.812+0000 7eff06bb8740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55605e87a820 0x55605e878270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:53:49.817 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:49.812+0000 7eff06bb8740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55605e87d3f0 con 0x55605e87a820 2024-03-16T11:53:49.817 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:49.812+0000 7eff06bb8740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x55605e791d70 con 0x55605e871a50 2024-03-16T11:53:49.817 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:49.812+0000 7eff06bb8740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55605e87d570 con 0x55605e871130 2024-03-16T11:53:49.817 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:49.812+0000 7eff06bb8740 10 client.0 _ll_register_callbacks cb 0x55605e4e9a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T11:53:49.817 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:49.812+0000 7eff0256a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55605e87a820 0x55605e878270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:53:49.817 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:49.812+0000 7eff01d69640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55605e871130 0x55605e8777f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:53:49.817 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:49.812+0000 7eff0256a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55605e87a820 0x55605e878270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47060/0 (socket says 172.21.15.29:47060) 2024-03-16T11:53:49.818 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:49.812+0000 7eff01568640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55605e871a50 0x55605e877d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:53:49.818 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:49.812+0000 7eff0256a640 1 -- 172.21.15.29:0/47573827 learned_addr learned my addr 172.21.15.29:0/47573827 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:53:49.818 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:49.812+0000 7eff01d69640 1 -- 172.21.15.29:0/47573827 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55605e871a50 msgr2=0x55605e877d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:53:49.818 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:49.812+0000 7eff01d69640 1 --2- 172.21.15.29:0/47573827 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55605e871a50 0x55605e877d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:53:49.818 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:49.812+0000 7eff01d69640 1 -- 172.21.15.29:0/47573827 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55605e87a820 msgr2=0x55605e878270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:53:49.818 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:49.812+0000 7eff01d69640 1 --2- 172.21.15.29:0/47573827 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55605e87a820 0x55605e878270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:53:49.818 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:49.812+0000 7eff01d69640 1 -- 172.21.15.29:0/47573827 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55605e4ea210 con 0x55605e871130 2024-03-16T11:53:49.818 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:49.813+0000 7eff01d69640 1 --2- 172.21.15.29:0/47573827 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55605e871130 0x55605e8777f0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7efef8000c00 tx=0x7efef802c3b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=69c741cd8a46e2a6 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:53:49.818 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:49.813+0000 7efee57fa640 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-16T11:53:49.818 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:49.813+0000 7efee57fa640 1 -- 172.21.15.29:0/47573827 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efef8042040 con 0x55605e871130 2024-03-16T11:53:49.818 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:49.813+0000 7eff06bb8740 1 -- 172.21.15.29:0/47573827 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efef802ca80 con 0x55605e871130 2024-03-16T11:53:49.818 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:49.813+0000 7efee57fa640 1 -- 172.21.15.29:0/47573827 <== mon.1 v2:172.21.15.154:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8469+0+0 (secure 0 0 0) 0x7efef8004250 con 0x55605e871130 2024-03-16T11:53:49.818 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:49.813+0000 7efee57fa640 1 -- 172.21.15.29:0/47573827 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7efef8004450 con 0x55605e871130 2024-03-16T11:53:49.818 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:49.813+0000 7eff06bb8740 10 client.25489 fetch_fsmap learned FSMap version 22 2024-03-16T11:53:49.819 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:49.813+0000 7eff06bb8740 1 -- 172.21.15.29:0/47573827 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x55605e5b6620 con 0x55605e871130 2024-03-16T11:53:49.819 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:49.813+0000 7efee57fa640 1 -- 172.21.15.29:0/47573827 <== mon.1 v2:172.21.15.154:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efef802ca80 con 0x55605e871130 2024-03-16T11:53:49.819 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:49.813+0000 7efee57fa640 1 -- 172.21.15.29:0/47573827 <== mon.1 v2:172.21.15.154:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7efef8003a70 con 0x55605e871130 2024-03-16T11:53:49.819 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:49.813+0000 7eff06bb8740 10 client.25489 fetch_fsmap finished waiting for FSMap version 22 2024-03-16T11:53:49.819 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:49.813+0000 7eff06bb8740 10 client.25489 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T11:53:49.819 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:49.813+0000 7eff06bb8740 20 client.25489 populate_metadata read hostname 'smithi029' 2024-03-16T11:53:49.819 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:49.813+0000 7eff06bb8740 1 --2- 172.21.15.29:0/47573827 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55605e88a770 0x55605e88cb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:53:49.819 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:49.813+0000 7eff06bb8740 4 client.25489 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T11:53:49.819 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:49.813+0000 7eff06bb8740 1 -- 172.21.15.29:0/47573827 --> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55605e7e2f80 con 0x55605e88a770 2024-03-16T11:53:49.819 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:49.813+0000 7eff01568640 1 --2- 172.21.15.29:0/47573827 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55605e88a770 0x55605e88cb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:53:49.819 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:49.814+0000 7eff01568640 1 --2- 172.21.15.29:0/47573827 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55605e88a770 0x55605e88cb60 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5c10c695b2798e65 server_cookie=9d5c4b453734bf8c in_seq=0 out_seq=0 2024-03-16T11:53:49.819 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:49.814+0000 7efee57fa640 10 client.25489 ms_handle_connect on v2:172.21.15.152:6838/1548731833 2024-03-16T11:53:49.820 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:49.816+0000 7efee57fa640 1 -- 172.21.15.29:0/47573827 <== mds.0 v2:172.21.15.152:6838/1548731833 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55605e7e2f80 con 0x55605e88a770 2024-03-16T11:53:49.820 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:49.816+0000 7efee57fa640 10 client.25489 handle_command_reply: tid=0 2024-03-16T11:53:49.821 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:49.816+0000 7eff06bb8740 1 client.25489 shutdown 2024-03-16T11:53:49.821 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:49.817+0000 7eff06bb8740 1 -- 172.21.15.29:0/47573827 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55605e871130 msgr2=0x55605e8777f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:53:49.821 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:49.817+0000 7eff06bb8740 1 --2- 172.21.15.29:0/47573827 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55605e871130 0x55605e8777f0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7efef8000c00 tx=0x7efef802c3b0 comp rx=0 tx=0).stop 2024-03-16T11:53:49.821 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:49.817+0000 7eff06bb8740 1 -- 172.21.15.29:0/47573827 shutdown_connections 2024-03-16T11:53:49.821 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:49.817+0000 7eff06bb8740 1 --2- 172.21.15.29:0/47573827 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55605e87a820 0x55605e878270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:53:49.821 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:49.817+0000 7eff06bb8740 1 --2- 172.21.15.29:0/47573827 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55605e88a770 0x55605e88cb60 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:53:49.821 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:49.817+0000 7eff06bb8740 1 --2- 172.21.15.29:0/47573827 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55605e871a50 0x55605e877d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:53:49.821 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:49.817+0000 7eff06bb8740 1 --2- 172.21.15.29:0/47573827 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55605e871130 0x55605e8777f0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:53:49.821 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:49.817+0000 7efee57fa640 0 client.25489 ms_handle_reset on v2:172.21.15.152:6838/1548731833 2024-03-16T11:53:49.821 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:49.817+0000 7eff06bb8740 1 -- 172.21.15.29:0/47573827 >> 172.21.15.29:0/47573827 conn(0x55605e86ba50 msgr2=0x55605e86d7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:53:49.821 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:49.817+0000 7eff06bb8740 1 -- 172.21.15.29:0/47573827 shutdown_connections 2024-03-16T11:53:49.821 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:49.817+0000 7eff06bb8740 1 -- 172.21.15.29:0/47573827 wait complete. 2024-03-16T11:53:49.821 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:49.817+0000 7eff06bb8740 20 client.25489 trim_cache size 0 max 16384 2024-03-16T11:53:49.832 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:49.832+0000 7eff06bb8740 1 Processor -- start 2024-03-16T11:53:49.832 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:49.832+0000 7eff06bb8740 1 -- start start 2024-03-16T11:53:49.832 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:49.832+0000 7eff06bb8740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55605e871130 0x55605e879f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:53:49.832 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:49.832+0000 7eff06bb8740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55605e871a50 0x55605e886190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:53:49.832 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:49.832+0000 7eff06bb8740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55605e87a820 0x55605e892160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:53:49.832 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:49.832+0000 7eff06bb8740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7efefc1afc10 con 0x55605e87a820 2024-03-16T11:53:49.832 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:49.832+0000 7eff06bb8740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7efefc1ad4e0 con 0x55605e871130 2024-03-16T11:53:49.833 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:49.832+0000 7eff06bb8740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7efefc1ab330 con 0x55605e871a50 2024-03-16T11:53:49.833 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:49.832+0000 7eff0256a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55605e87a820 0x55605e892160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:53:49.833 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:49.832+0000 7eff01568640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55605e871a50 0x55605e886190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:53:49.833 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:49.832+0000 7eff0256a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55605e87a820 0x55605e892160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47068/0 (socket says 172.21.15.29:47068) 2024-03-16T11:53:49.833 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:49.832+0000 7eff0256a640 1 -- 172.21.15.29:0/2268262623 learned_addr learned my addr 172.21.15.29:0/2268262623 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:53:49.833 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:49.832+0000 7eff01d69640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55605e871130 0x55605e879f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:53:49.833 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:49.832+0000 7eff0256a640 1 -- 172.21.15.29:0/2268262623 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55605e871a50 msgr2=0x55605e886190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:53:49.833 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:49.832+0000 7eff0256a640 1 --2- 172.21.15.29:0/2268262623 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55605e871a50 0x55605e886190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:53:49.833 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:49.832+0000 7eff0256a640 1 -- 172.21.15.29:0/2268262623 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55605e871130 msgr2=0x55605e879f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:53:49.833 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:49.832+0000 7eff0256a640 1 --2- 172.21.15.29:0/2268262623 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55605e871130 0x55605e879f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:53:49.833 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:49.832+0000 7eff0256a640 1 -- 172.21.15.29:0/2268262623 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55605e871880 con 0x55605e87a820 2024-03-16T11:53:49.833 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:49.833+0000 7eff0256a640 1 --2- 172.21.15.29:0/2268262623 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55605e87a820 0x55605e892160 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7efef400c760 tx=0x7efef409b3f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7308c977bfa462ee server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:53:49.833 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:49.833+0000 7efee77fe640 1 -- 172.21.15.29:0/2268262623 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efef409fab0 con 0x55605e87a820 2024-03-16T11:53:49.833 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:49.833+0000 7efee77fe640 1 -- 172.21.15.29:0/2268262623 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efef40a4100 con 0x55605e87a820 2024-03-16T11:53:49.833 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:49.833+0000 7efee77fe640 1 -- 172.21.15.29:0/2268262623 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efef4097fd0 con 0x55605e87a820 2024-03-16T11:53:49.837 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:49.834+0000 7eff06bb8740 1 -- 172.21.15.29:0/2268262623 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55605e87a820 msgr2=0x55605e892160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:53:49.837 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:49.834+0000 7eff06bb8740 1 --2- 172.21.15.29:0/2268262623 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55605e87a820 0x55605e892160 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7efef400c760 tx=0x7efef409b3f0 comp rx=0 tx=0).stop 2024-03-16T11:53:49.837 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:49.834+0000 7eff06bb8740 1 -- 172.21.15.29:0/2268262623 shutdown_connections 2024-03-16T11:53:49.837 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:49.834+0000 7eff06bb8740 1 --2- 172.21.15.29:0/2268262623 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55605e87a820 0x55605e892160 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:53:49.838 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:49.834+0000 7eff06bb8740 1 --2- 172.21.15.29:0/2268262623 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55605e871a50 0x55605e886190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:53:49.838 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:49.834+0000 7eff06bb8740 1 --2- 172.21.15.29:0/2268262623 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55605e871130 0x55605e879f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:53:49.838 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:49.834+0000 7eff06bb8740 1 -- 172.21.15.29:0/2268262623 >> 172.21.15.29:0/2268262623 conn(0x55605e86ba50 msgr2=0x55605e86abb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:53:49.838 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:49.834+0000 7eff06bb8740 1 -- 172.21.15.29:0/2268262623 shutdown_connections 2024-03-16T11:53:49.838 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:49.834+0000 7eff06bb8740 1 -- 172.21.15.29:0/2268262623 wait complete. 2024-03-16T11:53:49.838 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:49.834+0000 7eff06bb8740 1 -- start start 2024-03-16T11:53:49.838 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:49.834+0000 7eff06bb8740 1 Processor -- start 2024-03-16T11:53:49.838 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:49.834+0000 7eff06bb8740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55605e871130 0x55605e87dfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:53:49.838 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:49.834+0000 7eff06bb8740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55605e871a50 0x55605e87e510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:53:49.838 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:49.834+0000 7eff06bb8740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55605e87a820 0x55605e87ea50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:53:49.838 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:49.834+0000 7eff06bb8740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7efefc1ab9a0 con 0x55605e871a50 2024-03-16T11:53:49.838 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:49.835+0000 7eff06bb8740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7efefc1ace10 con 0x55605e87a820 2024-03-16T11:53:49.839 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:49.835+0000 7eff06bb8740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7efefc1ac070 con 0x55605e871130 2024-03-16T11:53:49.839 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:49.835+0000 7eff06bb8740 10 client.0 _ll_register_callbacks cb 0x7efeec00bf20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T11:53:49.839 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:49.835+0000 7eff01568640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55605e871a50 0x55605e87e510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:53:49.839 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:49.835+0000 7eff01568640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55605e871a50 0x55605e87e510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47082/0 (socket says 172.21.15.29:47082) 2024-03-16T11:53:49.839 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:49.835+0000 7eff01568640 1 -- 172.21.15.29:0/3882758280 learned_addr learned my addr 172.21.15.29:0/3882758280 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:53:49.839 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:49.835+0000 7eff01d69640 1 --2- 172.21.15.29:0/3882758280 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55605e871130 0x55605e87dfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:53:49.839 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:49.835+0000 7eff0256a640 1 --2- 172.21.15.29:0/3882758280 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55605e87a820 0x55605e87ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:53:49.839 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:49.835+0000 7eff01568640 1 -- 172.21.15.29:0/3882758280 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55605e871130 msgr2=0x55605e87dfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:53:49.839 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:49.835+0000 7eff01568640 1 --2- 172.21.15.29:0/3882758280 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55605e871130 0x55605e87dfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:53:49.839 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:49.835+0000 7eff01568640 1 -- 172.21.15.29:0/3882758280 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55605e87a820 msgr2=0x55605e87ea50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:53:49.839 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:49.835+0000 7eff01568640 1 --2- 172.21.15.29:0/3882758280 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55605e87a820 0x55605e87ea50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:53:49.839 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:49.835+0000 7eff01568640 1 -- 172.21.15.29:0/3882758280 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55605e8ef790 con 0x55605e871a50 2024-03-16T11:53:49.839 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:49.835+0000 7eff01568640 1 --2- 172.21.15.29:0/3882758280 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55605e871a50 0x55605e87e510 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7efeec00be90 tx=0x7efeec01c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bdac1108607715b4 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:53:49.839 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:49.835+0000 7efee57fa640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-16T11:53:49.840 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:49.835+0000 7efee57fa640 1 -- 172.21.15.29:0/3882758280 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efeec02d420 con 0x55605e871a50 2024-03-16T11:53:49.840 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:49.835+0000 7efee57fa640 1 -- 172.21.15.29:0/3882758280 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8469+0+0 (secure 0 0 0) 0x7efeec02d5c0 con 0x55605e871a50 2024-03-16T11:53:49.840 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:49.835+0000 7eff06bb8740 1 -- 172.21.15.29:0/3882758280 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efef40a4100 con 0x55605e871a50 2024-03-16T11:53:49.840 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:49.835+0000 7efee57fa640 1 -- 172.21.15.29:0/3882758280 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7efeec03f050 con 0x55605e871a50 2024-03-16T11:53:49.840 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:49.835+0000 7eff06bb8740 10 client.15648 fetch_fsmap learned FSMap version 22 2024-03-16T11:53:49.840 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:49.835+0000 7eff06bb8740 1 -- 172.21.15.29:0/3882758280 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x55605e8f0a30 con 0x55605e871a50 2024-03-16T11:53:49.840 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:49.835+0000 7efee57fa640 1 -- 172.21.15.29:0/3882758280 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efef40a4100 con 0x55605e871a50 2024-03-16T11:53:49.840 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:49.836+0000 7efee57fa640 1 -- 172.21.15.29:0/3882758280 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7efeec01f070 con 0x55605e871a50 2024-03-16T11:53:49.840 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:49.836+0000 7eff06bb8740 10 client.15648 fetch_fsmap finished waiting for FSMap version 22 2024-03-16T11:53:49.840 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:49.836+0000 7eff06bb8740 10 client.15648 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T11:53:49.840 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:49.836+0000 7eff06bb8740 20 client.15648 populate_metadata read hostname 'smithi029' 2024-03-16T11:53:49.840 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:49.836+0000 7eff06bb8740 1 --2- 172.21.15.29:0/3882758280 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55605e8f11e0 0x55605e8f3600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:53:49.840 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:49.836+0000 7eff06bb8740 4 client.15648 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-16T11:53:49.840 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:49.836+0000 7eff06bb8740 1 -- 172.21.15.29:0/3882758280 --> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55605e7dd2d0 con 0x55605e8f11e0 2024-03-16T11:53:49.841 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:49.836+0000 7eff01d69640 1 --2- 172.21.15.29:0/3882758280 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55605e8f11e0 0x55605e8f3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:53:49.841 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:49.836+0000 7eff01d69640 1 --2- 172.21.15.29:0/3882758280 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55605e8f11e0 0x55605e8f3600 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3c0e8a58cd0049a1 server_cookie=8de6e3c6ab4ec7fa in_seq=0 out_seq=0 2024-03-16T11:53:49.841 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:49.836+0000 7efee57fa640 10 client.15648 ms_handle_connect on v2:172.21.15.152:6838/1548731833 2024-03-16T11:53:49.979 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:49.980+0000 7efee57fa640 1 -- 172.21.15.29:0/3882758280 <== mds.0 v2:172.21.15.152:6838/1548731833 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55605e7dd2d0 con 0x55605e8f11e0 2024-03-16T11:53:49.980 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:49.980+0000 7efee57fa640 10 client.15648 handle_command_reply: tid=0 2024-03-16T11:53:49.980 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:49.980+0000 7eff06bb8740 1 client.15648 shutdown 2024-03-16T11:53:49.980 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:49.980+0000 7eff06bb8740 1 -- 172.21.15.29:0/3882758280 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55605e871a50 msgr2=0x55605e87e510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:53:49.980 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:49.980+0000 7eff06bb8740 1 --2- 172.21.15.29:0/3882758280 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55605e871a50 0x55605e87e510 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7efeec00be90 tx=0x7efeec01c000 comp rx=0 tx=0).stop 2024-03-16T11:53:49.980 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:49.980+0000 7eff06bb8740 1 -- 172.21.15.29:0/3882758280 shutdown_connections 2024-03-16T11:53:49.980 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:49.980+0000 7eff06bb8740 1 --2- 172.21.15.29:0/3882758280 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55605e8f11e0 0x55605e8f3600 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:53:49.980 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:49.980+0000 7eff06bb8740 1 --2- 172.21.15.29:0/3882758280 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55605e87a820 0x55605e87ea50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:53:49.980 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:49.980+0000 7eff06bb8740 1 --2- 172.21.15.29:0/3882758280 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55605e871a50 0x55605e87e510 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:53:49.980 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:49.980+0000 7eff06bb8740 1 --2- 172.21.15.29:0/3882758280 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55605e871130 0x55605e87dfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:53:49.980 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:49.980+0000 7efee57fa640 0 client.15648 ms_handle_reset on v2:172.21.15.152:6838/1548731833 2024-03-16T11:53:49.980 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:49.980+0000 7eff06bb8740 1 -- 172.21.15.29:0/3882758280 >> 172.21.15.29:0/3882758280 conn(0x55605e86ba50 msgr2=0x55605e877710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:53:49.980 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:49.980+0000 7eff06bb8740 1 -- 172.21.15.29:0/3882758280 shutdown_connections 2024-03-16T11:53:49.981 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:49.980+0000 7eff06bb8740 1 -- 172.21.15.29:0/3882758280 wait complete. 2024-03-16T11:53:49.981 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:49.980+0000 7eff06bb8740 20 client.15648 trim_cache size 0 max 16384 2024-03-16T11:53:49.981 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:{ 2024-03-16T11:53:49.981 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "message": "", 2024-03-16T11:53:49.981 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "return_code": 0 2024-03-16T11:53:49.981 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:} 2024-03-16T11:53:49.982 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:49.981+0000 7eff03fcd640 1 -- 172.21.15.29:0/2718757983 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7efee80a2a10 msgr2=0x7efee80a4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:53:49.982 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:49.981+0000 7eff03fcd640 1 --2- 172.21.15.29:0/2718757983 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7efee80a2a10 0x7efee80a4ed0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7efeec00bfd0 tx=0x7efeec012470 comp rx=0 tx=0).stop 2024-03-16T11:53:49.982 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:49.981+0000 7eff03fcd640 1 -- 172.21.15.29:0/2718757983 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efefc107850 msgr2=0x7efefc19c830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:53:49.982 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:49.981+0000 7eff03fcd640 1 --2- 172.21.15.29:0/2718757983 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efefc107850 0x7efefc19c830 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7efef400ea60 tx=0x7efef400ef20 comp rx=0 tx=0).stop 2024-03-16T11:53:49.982 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:49.981+0000 7eff03fcd640 1 -- 172.21.15.29:0/2718757983 shutdown_connections 2024-03-16T11:53:49.983 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:49.981+0000 7eff03fcd640 1 --2- 172.21.15.29:0/2718757983 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7efee80a2a10 0x7efee80a4ed0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:53:49.983 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:49.981+0000 7eff03fcd640 1 --2- 172.21.15.29:0/2718757983 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efefc107850 0x7efefc19c830 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:53:49.983 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:49.981+0000 7eff03fcd640 1 --2- 172.21.15.29:0/2718757983 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efefc069800 0x7efefc1984b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:53:49.983 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:49.981+0000 7eff03fcd640 1 --2- 172.21.15.29:0/2718757983 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7efefc068e60 0x7efefc197f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:53:49.983 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:49.981+0000 7eff03fcd640 1 -- 172.21.15.29:0/2718757983 >> 172.21.15.29:0/2718757983 conn(0x7efefc077530 msgr2=0x7efefc0ff3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:53:49.983 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:49.981+0000 7eff03fcd640 1 -- 172.21.15.29:0/2718757983 shutdown_connections 2024-03-16T11:53:49.983 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:53:49.981+0000 7eff03fcd640 1 -- 172.21.15.29:0/2718757983 wait complete. 2024-03-16T11:53:50.778 INFO:tasks.workunit.client.0.smithi029.stdout: 1 217586 12.52 MB/sec warmup 95 sec latency 56.999 ms 2024-03-16T11:53:51.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:53:50 smithi154 ceph-mon[36320]: pgmap v90: 97 pgs: 97 active+clean; 46 MiB data, 673 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 2.9 MiB/s wr, 305 op/s 2024-03-16T11:53:51.046 INFO:journalctl@ceph.mds.c.smithi154.stdout:Mar 16 11:53:50 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mds-c[66727]: ignoring --setuser ceph since I am not root 2024-03-16T11:53:51.046 INFO:journalctl@ceph.mds.c.smithi154.stdout:Mar 16 11:53:50 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mds-c[66727]: ignoring --setgroup ceph since I am not root 2024-03-16T11:53:51.046 INFO:journalctl@ceph.mds.c.smithi154.stdout:Mar 16 11:53:50 smithi154 ceph-mds[66731]: -- 172.21.15.154:0/47263557 <== mon.1 v2:172.21.15.154:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x560c3cb97a40 con 0x560c3cb6ac00 2024-03-16T11:53:51.046 INFO:journalctl@ceph.mds.c.smithi154.stdout:Mar 16 11:53:50 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mds-c[66727]: starting mds.c at 2024-03-16T11:53:51.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:53:50 smithi029 ceph-mon[31975]: pgmap v90: 97 pgs: 97 active+clean; 46 MiB data, 673 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 2.9 MiB/s wr, 305 op/s 2024-03-16T11:53:51.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:53:50 smithi152 ceph-mon[36291]: pgmap v90: 97 pgs: 97 active+clean; 46 MiB data, 673 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 2.9 MiB/s wr, 305 op/s 2024-03-16T11:53:51.778 INFO:tasks.workunit.client.0.smithi029.stdout: 1 220328 12.57 MB/sec warmup 96 sec latency 9.599 ms 2024-03-16T11:53:52.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:53:51 smithi154 ceph-mon[36320]: daemon mds.c restarted 2024-03-16T11:53:52.087 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:53:51 smithi029 ceph-mon[31975]: daemon mds.c restarted 2024-03-16T11:53:52.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:53:51 smithi152 ceph-mon[36291]: daemon mds.c restarted 2024-03-16T11:53:52.778 INFO:tasks.workunit.client.0.smithi029.stdout: 1 222880 12.57 MB/sec warmup 97 sec latency 15.230 ms 2024-03-16T11:53:53.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:53:52 smithi154 ceph-mon[36320]: pgmap v91: 97 pgs: 97 active+clean; 39 MiB data, 661 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 4.3 MiB/s wr, 502 op/s 2024-03-16T11:53:53.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:53:52 smithi154 ceph-mon[36320]: osdmap e89: 12 total, 12 up, 12 in 2024-03-16T11:53:53.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:53:52 smithi154 ceph-mon[36320]: mds.? [v2:172.21.15.154:6836/3659006495,v1:172.21.15.154:6837/3659006495] up:boot 2024-03-16T11:53:53.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:53:52 smithi154 ceph-mon[36320]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 2 up:standby-replay 7 up:standby 2024-03-16T11:53:53.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:53:52 smithi154 ceph-mon[36320]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-16T11:53:53.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:53:52 smithi154 ceph-mon[36320]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-03-16T11:53:53.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:53:52 smithi029 ceph-mon[31975]: pgmap v91: 97 pgs: 97 active+clean; 39 MiB data, 661 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 4.3 MiB/s wr, 502 op/s 2024-03-16T11:53:53.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:53:52 smithi029 ceph-mon[31975]: osdmap e89: 12 total, 12 up, 12 in 2024-03-16T11:53:53.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:53:52 smithi029 ceph-mon[31975]: mds.? [v2:172.21.15.154:6836/3659006495,v1:172.21.15.154:6837/3659006495] up:boot 2024-03-16T11:53:53.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:53:52 smithi029 ceph-mon[31975]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 2 up:standby-replay 7 up:standby 2024-03-16T11:53:53.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:53:52 smithi029 ceph-mon[31975]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-16T11:53:53.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:53:52 smithi029 ceph-mon[31975]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-03-16T11:53:53.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:53:52 smithi152 ceph-mon[36291]: pgmap v91: 97 pgs: 97 active+clean; 39 MiB data, 661 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 4.3 MiB/s wr, 502 op/s 2024-03-16T11:53:53.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:53:52 smithi152 ceph-mon[36291]: osdmap e89: 12 total, 12 up, 12 in 2024-03-16T11:53:53.128 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:53:52 smithi152 ceph-mon[36291]: mds.? [v2:172.21.15.154:6836/3659006495,v1:172.21.15.154:6837/3659006495] up:boot 2024-03-16T11:53:53.128 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:53:52 smithi152 ceph-mon[36291]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 2 up:standby-replay 7 up:standby 2024-03-16T11:53:53.128 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:53:52 smithi152 ceph-mon[36291]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-16T11:53:53.128 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:53:52 smithi152 ceph-mon[36291]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-03-16T11:53:53.778 INFO:tasks.workunit.client.0.smithi029.stdout: 1 224235 12.51 MB/sec warmup 98 sec latency 82.792 ms 2024-03-16T11:53:54.778 INFO:tasks.workunit.client.0.smithi029.stdout: 1 227175 12.56 MB/sec warmup 99 sec latency 8.293 ms 2024-03-16T11:53:55.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:53:54 smithi154 ceph-mon[36320]: pgmap v93: 97 pgs: 97 active+clean; 39 MiB data, 661 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.5 MiB/s wr, 364 op/s 2024-03-16T11:53:55.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:53:54 smithi029 ceph-mon[31975]: pgmap v93: 97 pgs: 97 active+clean; 39 MiB data, 661 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.5 MiB/s wr, 364 op/s 2024-03-16T11:53:55.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:53:54 smithi152 ceph-mon[36291]: pgmap v93: 97 pgs: 97 active+clean; 39 MiB data, 661 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.5 MiB/s wr, 364 op/s 2024-03-16T11:53:55.778 INFO:tasks.workunit.client.0.smithi029.stdout: 1 229050 12.50 MB/sec warmup 100 sec latency 18.309 ms 2024-03-16T11:53:56.778 INFO:tasks.workunit.client.0.smithi029.stdout: 1 231202 12.51 MB/sec warmup 101 sec latency 23.824 ms 2024-03-16T11:53:57.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:53:56 smithi154 ceph-mon[36320]: pgmap v94: 97 pgs: 97 active+clean; 39 MiB data, 661 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.5 MiB/s wr, 364 op/s 2024-03-16T11:53:57.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:53:56 smithi029 ceph-mon[31975]: pgmap v94: 97 pgs: 97 active+clean; 39 MiB data, 661 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.5 MiB/s wr, 364 op/s 2024-03-16T11:53:57.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:53:56 smithi152 ceph-mon[36291]: pgmap v94: 97 pgs: 97 active+clean; 39 MiB data, 661 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.5 MiB/s wr, 364 op/s 2024-03-16T11:53:57.779 INFO:tasks.workunit.client.0.smithi029.stdout: 1 233979 12.53 MB/sec warmup 102 sec latency 20.118 ms 2024-03-16T11:53:58.779 INFO:tasks.workunit.client.0.smithi029.stdout: 1 235206 12.48 MB/sec warmup 103 sec latency 68.947 ms 2024-03-16T11:53:59.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:53:58 smithi154 ceph-mon[36320]: pgmap v95: 97 pgs: 97 active+clean; 43 MiB data, 716 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 3.3 MiB/s wr, 359 op/s 2024-03-16T11:53:59.087 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:53:58 smithi029 ceph-mon[31975]: pgmap v95: 97 pgs: 97 active+clean; 43 MiB data, 716 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 3.3 MiB/s wr, 359 op/s 2024-03-16T11:53:59.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:53:58 smithi152 ceph-mon[36291]: pgmap v95: 97 pgs: 97 active+clean; 43 MiB data, 716 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 3.3 MiB/s wr, 359 op/s 2024-03-16T11:53:59.779 INFO:tasks.workunit.client.0.smithi029.stdout: 1 237958 12.52 MB/sec warmup 104 sec latency 14.470 ms 2024-03-16T11:54:00.779 INFO:tasks.workunit.client.0.smithi029.stdout: 1 239980 12.46 MB/sec warmup 105 sec latency 23.672 ms 2024-03-16T11:54:01.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:54:00 smithi154 ceph-mon[36320]: pgmap v96: 97 pgs: 97 active+clean; 43 MiB data, 716 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 3.3 MiB/s wr, 359 op/s 2024-03-16T11:54:01.087 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:54:00 smithi029 ceph-mon[31975]: pgmap v96: 97 pgs: 97 active+clean; 43 MiB data, 716 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 3.3 MiB/s wr, 359 op/s 2024-03-16T11:54:01.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:54:00 smithi152 ceph-mon[36291]: pgmap v96: 97 pgs: 97 active+clean; 43 MiB data, 716 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 3.3 MiB/s wr, 359 op/s 2024-03-16T11:54:01.779 INFO:tasks.workunit.client.0.smithi029.stdout: 1 241604 12.47 MB/sec warmup 106 sec latency 82.873 ms 2024-03-16T11:54:02.779 INFO:tasks.workunit.client.0.smithi029.stdout: 1 243958 12.45 MB/sec warmup 107 sec latency 25.125 ms 2024-03-16T11:54:03.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:54:02 smithi154 ceph-mon[36320]: pgmap v97: 97 pgs: 97 active+clean; 47 MiB data, 722 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 3.3 MiB/s wr, 278 op/s 2024-03-16T11:54:03.072 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:03.072+0000 7fea44adc640 1 -- 172.21.15.29:0/2073186882 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fea40105680 msgr2=0x7fea40105a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:54:03.072 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:03.072+0000 7fea44adc640 1 --2- 172.21.15.29:0/2073186882 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fea40105680 0x7fea40105a60 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7fea30009900 tx=0x7fea3002bc30 comp rx=0 tx=0).stop 2024-03-16T11:54:03.072 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:03.072+0000 7fea44adc640 1 -- 172.21.15.29:0/2073186882 shutdown_connections 2024-03-16T11:54:03.072 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:03.072+0000 7fea44adc640 1 --2- 172.21.15.29:0/2073186882 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fea401012f0 0x7fea4010d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:54:03.072 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:03.072+0000 7fea44adc640 1 --2- 172.21.15.29:0/2073186882 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fea40100930 0x7fea40100db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:54:03.072 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:03.072+0000 7fea44adc640 1 --2- 172.21.15.29:0/2073186882 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fea40105680 0x7fea40105a60 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:54:03.072 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:03.072+0000 7fea44adc640 1 -- 172.21.15.29:0/2073186882 >> 172.21.15.29:0/2073186882 conn(0x7fea40077530 msgr2=0x7fea40077940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:54:03.072 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:03.072+0000 7fea44adc640 1 -- 172.21.15.29:0/2073186882 shutdown_connections 2024-03-16T11:54:03.072 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:03.072+0000 7fea44adc640 1 -- 172.21.15.29:0/2073186882 wait complete. 2024-03-16T11:54:03.075 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:03.073+0000 7fea44adc640 1 Processor -- start 2024-03-16T11:54:03.075 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:03.073+0000 7fea44adc640 1 -- start start 2024-03-16T11:54:03.075 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:03.073+0000 7fea44adc640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fea40100930 0x7fea40198090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:54:03.075 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:03.073+0000 7fea44adc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fea401012f0 0x7fea401985d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:54:03.075 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:03.073+0000 7fea44adc640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fea40105680 0x7fea4019c950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:54:03.076 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:03.073+0000 7fea44adc640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fea40110b60 con 0x7fea401012f0 2024-03-16T11:54:03.076 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:03.073+0000 7fea44adc640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fea401109e0 con 0x7fea40100930 2024-03-16T11:54:03.076 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:03.073+0000 7fea44adc640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fea40110ce0 con 0x7fea40105680 2024-03-16T11:54:03.076 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:03.074+0000 7fea3dd9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fea401012f0 0x7fea401985d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:54:03.076 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:03.074+0000 7fea3e59c640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fea40100930 0x7fea40198090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:54:03.076 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:03.074+0000 7fea3dd9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fea401012f0 0x7fea401985d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60016/0 (socket says 172.21.15.29:60016) 2024-03-16T11:54:03.076 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:03.074+0000 7fea3dd9b640 1 -- 172.21.15.29:0/3029013668 learned_addr learned my addr 172.21.15.29:0/3029013668 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:54:03.076 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:03.074+0000 7fea3ed9d640 1 --2- 172.21.15.29:0/3029013668 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fea40105680 0x7fea4019c950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:54:03.077 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:03.074+0000 7fea3dd9b640 1 -- 172.21.15.29:0/3029013668 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fea40105680 msgr2=0x7fea4019c950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:54:03.077 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:03.074+0000 7fea3dd9b640 1 --2- 172.21.15.29:0/3029013668 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fea40105680 0x7fea4019c950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:54:03.077 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:03.074+0000 7fea3dd9b640 1 -- 172.21.15.29:0/3029013668 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fea40100930 msgr2=0x7fea40198090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:54:03.077 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:03.074+0000 7fea3dd9b640 1 --2- 172.21.15.29:0/3029013668 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fea40100930 0x7fea40198090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:54:03.077 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:03.074+0000 7fea3dd9b640 1 -- 172.21.15.29:0/3029013668 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea4019d080 con 0x7fea401012f0 2024-03-16T11:54:03.077 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:03.074+0000 7fea3e59c640 1 --2- 172.21.15.29:0/3029013668 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fea40100930 0x7fea40198090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:54:03.077 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:03.074+0000 7fea3dd9b640 1 --2- 172.21.15.29:0/3029013668 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fea401012f0 0x7fea401985d0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7fea3400ea60 tx=0x7fea3400ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:54:03.077 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:03.074+0000 7fea2f7fe640 1 -- 172.21.15.29:0/3029013668 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea3400cde0 con 0x7fea401012f0 2024-03-16T11:54:03.077 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:03.074+0000 7fea44adc640 1 -- 172.21.15.29:0/3029013668 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fea4019d370 con 0x7fea401012f0 2024-03-16T11:54:03.077 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:03.074+0000 7fea2f7fe640 1 -- 172.21.15.29:0/3029013668 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fea340035f0 con 0x7fea401012f0 2024-03-16T11:54:03.077 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:03.074+0000 7fea2f7fe640 1 -- 172.21.15.29:0/3029013668 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea34010670 con 0x7fea401012f0 2024-03-16T11:54:03.077 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:03.074+0000 7fea44adc640 1 -- 172.21.15.29:0/3029013668 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fea401a4c00 con 0x7fea401012f0 2024-03-16T11:54:03.085 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:03.077+0000 7fea2f7fe640 1 -- 172.21.15.29:0/3029013668 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7fea34010810 con 0x7fea401012f0 2024-03-16T11:54:03.085 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:03.077+0000 7fea477a2740 1 Processor -- start 2024-03-16T11:54:03.085 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:03.077+0000 7fea477a2740 1 -- start start 2024-03-16T11:54:03.086 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:03.077+0000 7fea477a2740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56264d393130 0x56264d393510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:54:03.086 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:03.077+0000 7fea477a2740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56264d393a50 0x56264d39c2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:54:03.086 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:03.077+0000 7fea477a2740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56264d39c820 0x56264d39ec10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:54:03.086 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:03.078+0000 7fea477a2740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x56264d2f7a40 con 0x56264d393130 2024-03-16T11:54:03.086 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:03.078+0000 7fea477a2740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x56264d2a0920 con 0x56264d39c820 2024-03-16T11:54:03.086 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:03.078+0000 7fea477a2740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x56264d2f3ae0 con 0x56264d393a50 2024-03-16T11:54:03.086 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:03.078+0000 7fea3ed9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56264d393130 0x56264d393510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:54:03.086 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:03.078+0000 7fea3e59c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56264d393a50 0x56264d39c2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:54:03.087 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:03.078+0000 7fea3ed9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56264d393130 0x56264d393510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60018/0 (socket says 172.21.15.29:60018) 2024-03-16T11:54:03.087 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:03.078+0000 7fea3ed9d640 1 -- 172.21.15.29:0/2871799989 learned_addr learned my addr 172.21.15.29:0/2871799989 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:54:03.087 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:03.078+0000 7fea3dd9b640 1 --2- 172.21.15.29:0/2871799989 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56264d39c820 0x56264d39ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:54:03.087 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:03.078+0000 7fea2f7fe640 1 --2- 172.21.15.29:0/3029013668 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7fea280a2a10 0x7fea280a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:54:03.087 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:03.078+0000 7fea3ed9d640 1 --2- 172.21.15.29:0/3029013668 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7fea280a2a10 0x7fea280a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:54:03.087 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:03.078+0000 7fea3e59c640 1 -- 172.21.15.29:0/2871799989 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56264d39c820 msgr2=0x56264d39ec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:54:03.087 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:03.078+0000 7fea3e59c640 1 --2- 172.21.15.29:0/2871799989 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56264d39c820 0x56264d39ec10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:54:03.087 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:03.078+0000 7fea3e59c640 1 -- 172.21.15.29:0/2871799989 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56264d393130 msgr2=0x56264d393510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:54:03.087 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:03.078+0000 7fea3e59c640 1 --2- 172.21.15.29:0/2871799989 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56264d393130 0x56264d393510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:54:03.087 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:03.078+0000 7fea3e59c640 1 -- 172.21.15.29:0/2871799989 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56264d2b8460 con 0x56264d393a50 2024-03-16T11:54:03.087 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:03.078+0000 7fea3ed9d640 1 --2- 172.21.15.29:0/2871799989 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56264d393130 0x56264d393510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:54:03.087 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:03.078+0000 7fea2f7fe640 1 -- 172.21.15.29:0/3029013668 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7fea340cf0a0 con 0x7fea401012f0 2024-03-16T11:54:03.087 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:03.078+0000 7fea3e59c640 1 --2- 172.21.15.29:0/2871799989 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56264d393a50 0x56264d39c2e0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fea30000c00 tx=0x7fea30005e80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=13556a844c32ed49 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:54:03.087 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:03.078+0000 7fea3ed9d640 1 --2- 172.21.15.29:0/3029013668 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7fea280a2a10 0x7fea280a4ed0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fea38005c40 tx=0x7fea38004ab0 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:54:03.088 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:03.079+0000 7fea2cff9640 1 -- 172.21.15.29:0/2871799989 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea30042070 con 0x56264d393a50 2024-03-16T11:54:03.088 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:03.079+0000 7fea2cff9640 1 -- 172.21.15.29:0/2871799989 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fea30004a70 con 0x56264d393a50 2024-03-16T11:54:03.088 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:03.079+0000 7fea2cff9640 1 -- 172.21.15.29:0/2871799989 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea30005180 con 0x56264d393a50 2024-03-16T11:54:03.088 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:03.079+0000 7fea477a2740 1 -- 172.21.15.29:0/2871799989 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56264d393a50 msgr2=0x56264d39c2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:54:03.088 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:03.079+0000 7fea477a2740 1 --2- 172.21.15.29:0/2871799989 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56264d393a50 0x56264d39c2e0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fea30000c00 tx=0x7fea30005e80 comp rx=0 tx=0).stop 2024-03-16T11:54:03.088 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:03.079+0000 7fea477a2740 1 -- 172.21.15.29:0/2871799989 shutdown_connections 2024-03-16T11:54:03.088 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:03.079+0000 7fea477a2740 1 --2- 172.21.15.29:0/2871799989 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56264d39c820 0x56264d39ec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:54:03.088 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:03.079+0000 7fea477a2740 1 --2- 172.21.15.29:0/2871799989 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56264d393a50 0x56264d39c2e0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:54:03.088 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:03.079+0000 7fea477a2740 1 --2- 172.21.15.29:0/2871799989 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56264d393130 0x56264d393510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:54:03.088 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:03.079+0000 7fea477a2740 1 -- 172.21.15.29:0/2871799989 >> 172.21.15.29:0/2871799989 conn(0x56264d38da50 msgr2=0x56264d38de30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:54:03.088 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:03.080+0000 7fea477a2740 1 -- 172.21.15.29:0/2871799989 shutdown_connections 2024-03-16T11:54:03.088 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:03.080+0000 7fea477a2740 1 -- 172.21.15.29:0/2871799989 wait complete. 2024-03-16T11:54:03.088 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:03.080+0000 7fea477a2740 1 -- start start 2024-03-16T11:54:03.088 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:03.080+0000 7fea477a2740 1 Processor -- start 2024-03-16T11:54:03.089 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:03.080+0000 7fea477a2740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56264d393130 0x56264d399870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:54:03.089 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:03.080+0000 7fea477a2740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56264d393a50 0x56264d399db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:54:03.089 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:03.080+0000 7fea477a2740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56264d39c820 0x56264d39a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:54:03.089 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:03.080+0000 7fea477a2740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x56264d39f3f0 con 0x56264d393130 2024-03-16T11:54:03.089 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:03.080+0000 7fea477a2740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x56264d2b3d70 con 0x56264d393a50 2024-03-16T11:54:03.089 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:03.080+0000 7fea477a2740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x56264d39f570 con 0x56264d39c820 2024-03-16T11:54:03.089 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:03.080+0000 7fea477a2740 10 client.0 _ll_register_callbacks cb 0x56264d00ba00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T11:54:03.089 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:03.080+0000 7fea3e59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56264d393130 0x56264d399870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:54:03.089 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:03.080+0000 7fea3e59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56264d393130 0x56264d399870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60034/0 (socket says 172.21.15.29:60034) 2024-03-16T11:54:03.089 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:03.080+0000 7fea3e59c640 1 -- 172.21.15.29:0/1457713239 learned_addr learned my addr 172.21.15.29:0/1457713239 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:54:03.089 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:03.080+0000 7fea3ed9d640 1 --2- 172.21.15.29:0/1457713239 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56264d39c820 0x56264d39a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:54:03.089 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:03.080+0000 7fea3dd9b640 1 --2- 172.21.15.29:0/1457713239 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56264d393a50 0x56264d399db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:54:03.089 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:03.081+0000 7fea3e59c640 1 -- 172.21.15.29:0/1457713239 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56264d39c820 msgr2=0x56264d39a2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:54:03.089 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:03.081+0000 7fea3e59c640 1 --2- 172.21.15.29:0/1457713239 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56264d39c820 0x56264d39a2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:54:03.090 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:03.081+0000 7fea3e59c640 1 -- 172.21.15.29:0/1457713239 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56264d393a50 msgr2=0x56264d399db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:54:03.090 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:03.081+0000 7fea3e59c640 1 --2- 172.21.15.29:0/1457713239 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56264d393a50 0x56264d399db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:54:03.090 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:03.081+0000 7fea3e59c640 1 -- 172.21.15.29:0/1457713239 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56264d00c210 con 0x56264d393130 2024-03-16T11:54:03.090 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:03.081+0000 7fea3ed9d640 1 --2- 172.21.15.29:0/1457713239 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56264d39c820 0x56264d39a2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:54:03.090 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:03.081+0000 7fea3e59c640 1 --2- 172.21.15.29:0/1457713239 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56264d393130 0x56264d399870 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7fea3002c650 tx=0x7fea3002c680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6ca323cae2a3343 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:54:03.090 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:03.081+0000 7fea25ffb640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-16T11:54:03.090 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:03.081+0000 7fea25ffb640 1 -- 172.21.15.29:0/1457713239 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea3004b070 con 0x56264d393130 2024-03-16T11:54:03.090 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:03.081+0000 7fea25ffb640 1 -- 172.21.15.29:0/1457713239 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7fea30005470 con 0x56264d393130 2024-03-16T11:54:03.090 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:03.081+0000 7fea477a2740 1 -- 172.21.15.29:0/1457713239 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fea30004a70 con 0x56264d393130 2024-03-16T11:54:03.090 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:03.081+0000 7fea25ffb640 1 -- 172.21.15.29:0/1457713239 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fea30057050 con 0x56264d393130 2024-03-16T11:54:03.090 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:03.081+0000 7fea477a2740 10 client.15666 fetch_fsmap learned FSMap version 24 2024-03-16T11:54:03.090 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:03.081+0000 7fea477a2740 1 -- 172.21.15.29:0/1457713239 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x56264d0d8620 con 0x56264d393130 2024-03-16T11:54:03.090 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:03.082+0000 7fea25ffb640 1 -- 172.21.15.29:0/1457713239 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fea30004a70 con 0x56264d393130 2024-03-16T11:54:03.091 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:03.082+0000 7fea25ffb640 1 -- 172.21.15.29:0/1457713239 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7fea300344d0 con 0x56264d393130 2024-03-16T11:54:03.091 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:03.082+0000 7fea477a2740 10 client.15666 fetch_fsmap finished waiting for FSMap version 24 2024-03-16T11:54:03.091 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:03.082+0000 7fea477a2740 10 client.15666 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T11:54:03.091 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:03.082+0000 7fea477a2740 20 client.15666 populate_metadata read hostname 'smithi029' 2024-03-16T11:54:03.091 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:03.082+0000 7fea477a2740 1 --2- 172.21.15.29:0/1457713239 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x56264d3ac7f0 0x56264d3aebe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:54:03.091 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:03.082+0000 7fea477a2740 4 client.15666 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T11:54:03.091 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:03.082+0000 7fea477a2740 1 -- 172.21.15.29:0/1457713239 --> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56264d304f80 con 0x56264d3ac7f0 2024-03-16T11:54:03.091 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:03.082+0000 7fea3dd9b640 1 --2- 172.21.15.29:0/1457713239 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x56264d3ac7f0 0x56264d3aebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:54:03.092 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:03.082+0000 7fea25ffb640 10 client.15666 ms_handle_connect on v2:172.21.15.152:6838/1548731833 2024-03-16T11:54:03.092 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:03.082+0000 7fea3dd9b640 1 --2- 172.21.15.29:0/1457713239 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x56264d3ac7f0 0x56264d3aebe0 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=af97f50d8458d53 server_cookie=2a6198c9b165f584 in_seq=0 out_seq=0 2024-03-16T11:54:03.092 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:03.084+0000 7fea25ffb640 1 -- 172.21.15.29:0/1457713239 <== mds.0 v2:172.21.15.152:6838/1548731833 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x56264d304f80 con 0x56264d3ac7f0 2024-03-16T11:54:03.092 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:03.084+0000 7fea25ffb640 10 client.15666 handle_command_reply: tid=0 2024-03-16T11:54:03.092 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:03.084+0000 7fea477a2740 1 client.15666 shutdown 2024-03-16T11:54:03.092 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:03.084+0000 7fea477a2740 1 -- 172.21.15.29:0/1457713239 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56264d393130 msgr2=0x56264d399870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:54:03.092 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:03.084+0000 7fea477a2740 1 --2- 172.21.15.29:0/1457713239 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56264d393130 0x56264d399870 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7fea3002c650 tx=0x7fea3002c680 comp rx=0 tx=0).stop 2024-03-16T11:54:03.092 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:03.084+0000 7fea477a2740 1 -- 172.21.15.29:0/1457713239 shutdown_connections 2024-03-16T11:54:03.092 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:03.084+0000 7fea477a2740 1 --2- 172.21.15.29:0/1457713239 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56264d39c820 0x56264d39a2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:54:03.093 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:03.084+0000 7fea477a2740 1 --2- 172.21.15.29:0/1457713239 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x56264d3ac7f0 0x56264d3aebe0 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:54:03.093 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:03.084+0000 7fea477a2740 1 --2- 172.21.15.29:0/1457713239 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56264d393a50 0x56264d399db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:54:03.093 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:03.084+0000 7fea477a2740 1 --2- 172.21.15.29:0/1457713239 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56264d393130 0x56264d399870 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:54:03.093 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:03.084+0000 7fea25ffb640 0 client.15666 ms_handle_reset on v2:172.21.15.152:6838/1548731833 2024-03-16T11:54:03.093 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:03.084+0000 7fea477a2740 1 -- 172.21.15.29:0/1457713239 >> 172.21.15.29:0/1457713239 conn(0x56264d38da50 msgr2=0x56264d38f7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:54:03.093 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:03.084+0000 7fea477a2740 1 -- 172.21.15.29:0/1457713239 shutdown_connections 2024-03-16T11:54:03.093 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:03.084+0000 7fea477a2740 1 -- 172.21.15.29:0/1457713239 wait complete. 2024-03-16T11:54:03.093 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:03.084+0000 7fea477a2740 20 client.15666 trim_cache size 0 max 16384 2024-03-16T11:54:03.094 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:54:02 smithi029 ceph-mon[31975]: pgmap v97: 97 pgs: 97 active+clean; 47 MiB data, 722 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 3.3 MiB/s wr, 278 op/s 2024-03-16T11:54:03.105 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:03.100+0000 7fea477a2740 1 Processor -- start 2024-03-16T11:54:03.105 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:03.100+0000 7fea477a2740 1 -- start start 2024-03-16T11:54:03.105 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:03.101+0000 7fea477a2740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56264d393a50 0x56264d39bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:54:03.105 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:03.101+0000 7fea477a2740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56264d39c820 0x56264d3a8190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:54:03.106 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:03.101+0000 7fea477a2740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56264d38fa60 0x56264d3b4270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:54:03.106 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:03.101+0000 7fea477a2740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fea401afb40 con 0x56264d38fa60 2024-03-16T11:54:03.106 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:03.101+0000 7fea477a2740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fea401ad410 con 0x56264d39c820 2024-03-16T11:54:03.106 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:03.101+0000 7fea477a2740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fea401ab260 con 0x56264d393a50 2024-03-16T11:54:03.106 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:03.101+0000 7fea3ed9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56264d38fa60 0x56264d3b4270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:54:03.106 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:03.101+0000 7fea3ed9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56264d38fa60 0x56264d3b4270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60042/0 (socket says 172.21.15.29:60042) 2024-03-16T11:54:03.106 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:03.101+0000 7fea3ed9d640 1 -- 172.21.15.29:0/1466459598 learned_addr learned my addr 172.21.15.29:0/1466459598 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:54:03.106 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:03.101+0000 7fea3e59c640 1 --2- 172.21.15.29:0/1466459598 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56264d393a50 0x56264d39bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:54:03.106 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:03.101+0000 7fea3dd9b640 1 --2- 172.21.15.29:0/1466459598 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56264d39c820 0x56264d3a8190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:54:03.106 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:03.101+0000 7fea3ed9d640 1 -- 172.21.15.29:0/1466459598 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56264d393a50 msgr2=0x56264d39bfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:54:03.106 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:03.101+0000 7fea3ed9d640 1 --2- 172.21.15.29:0/1466459598 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56264d393a50 0x56264d39bfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:54:03.106 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:03.101+0000 7fea3ed9d640 1 -- 172.21.15.29:0/1466459598 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56264d39c820 msgr2=0x56264d3a8190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:54:03.107 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:03.101+0000 7fea3ed9d640 1 --2- 172.21.15.29:0/1466459598 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56264d39c820 0x56264d3a8190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:54:03.107 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:03.101+0000 7fea3ed9d640 1 -- 172.21.15.29:0/1466459598 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56264d387fc0 con 0x56264d38fa60 2024-03-16T11:54:03.107 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:03.101+0000 7fea3ed9d640 1 --2- 172.21.15.29:0/1466459598 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56264d38fa60 0x56264d3b4270 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7fea38004240 tx=0x7fea38009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ee8d6c6f5374210 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:54:03.107 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:03.101+0000 7fea27fff640 1 -- 172.21.15.29:0/1466459598 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea380109b0 con 0x56264d38fa60 2024-03-16T11:54:03.107 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:03.102+0000 7fea27fff640 1 -- 172.21.15.29:0/1466459598 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fea38010b50 con 0x56264d38fa60 2024-03-16T11:54:03.107 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:03.102+0000 7fea27fff640 1 -- 172.21.15.29:0/1466459598 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea3802a640 con 0x56264d38fa60 2024-03-16T11:54:03.107 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:03.102+0000 7fea477a2740 1 -- 172.21.15.29:0/1466459598 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56264d38fa60 msgr2=0x56264d3b4270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:54:03.107 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:03.102+0000 7fea477a2740 1 --2- 172.21.15.29:0/1466459598 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56264d38fa60 0x56264d3b4270 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7fea38004240 tx=0x7fea38009bd0 comp rx=0 tx=0).stop 2024-03-16T11:54:03.107 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:03.102+0000 7fea477a2740 1 -- 172.21.15.29:0/1466459598 shutdown_connections 2024-03-16T11:54:03.107 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:03.102+0000 7fea477a2740 1 --2- 172.21.15.29:0/1466459598 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56264d38fa60 0x56264d3b4270 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:54:03.107 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:03.102+0000 7fea477a2740 1 --2- 172.21.15.29:0/1466459598 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56264d39c820 0x56264d3a8190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:54:03.107 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:03.102+0000 7fea477a2740 1 --2- 172.21.15.29:0/1466459598 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56264d393a50 0x56264d39bfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:54:03.107 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:03.102+0000 7fea477a2740 1 -- 172.21.15.29:0/1466459598 >> 172.21.15.29:0/1466459598 conn(0x56264d38da50 msgr2=0x56264d394ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:54:03.107 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:03.102+0000 7fea477a2740 1 -- 172.21.15.29:0/1466459598 shutdown_connections 2024-03-16T11:54:03.108 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:03.102+0000 7fea477a2740 1 -- 172.21.15.29:0/1466459598 wait complete. 2024-03-16T11:54:03.108 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:03.103+0000 7fea477a2740 1 -- start start 2024-03-16T11:54:03.108 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:03.103+0000 7fea477a2740 1 Processor -- start 2024-03-16T11:54:03.108 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:03.103+0000 7fea477a2740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56264d38fa60 0x56264d395cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:54:03.108 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:03.103+0000 7fea477a2740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56264d393a50 0x56264d396210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:54:03.108 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:03.103+0000 7fea477a2740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56264d39c820 0x56264d396840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:54:03.108 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:03.103+0000 7fea477a2740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fea401ab8d0 con 0x56264d393a50 2024-03-16T11:54:03.108 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:03.103+0000 7fea477a2740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fea401acd40 con 0x56264d39c820 2024-03-16T11:54:03.108 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:03.103+0000 7fea477a2740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fea401abfa0 con 0x56264d38fa60 2024-03-16T11:54:03.108 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:03.103+0000 7fea477a2740 10 client.0 _ll_register_callbacks cb 0x7fea38009b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T11:54:03.108 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:03.103+0000 7fea3dd9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56264d393a50 0x56264d396210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:54:03.108 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:03.103+0000 7fea3e59c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56264d38fa60 0x56264d395cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:54:03.108 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:03.103+0000 7fea3ed9d640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56264d39c820 0x56264d396840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:54:03.108 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:03.103+0000 7fea3dd9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56264d393a50 0x56264d396210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60058/0 (socket says 172.21.15.29:60058) 2024-03-16T11:54:03.108 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:03.103+0000 7fea3dd9b640 1 -- 172.21.15.29:0/17427055 learned_addr learned my addr 172.21.15.29:0/17427055 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:54:03.109 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:03.103+0000 7fea3dd9b640 1 -- 172.21.15.29:0/17427055 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56264d38fa60 msgr2=0x56264d395cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:54:03.109 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:03.103+0000 7fea3dd9b640 1 --2- 172.21.15.29:0/17427055 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56264d38fa60 0x56264d395cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:54:03.109 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:03.103+0000 7fea3dd9b640 1 -- 172.21.15.29:0/17427055 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56264d39c820 msgr2=0x56264d396840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:54:03.109 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:03.103+0000 7fea3dd9b640 1 --2- 172.21.15.29:0/17427055 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56264d39c820 0x56264d396840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:54:03.109 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:03.104+0000 7fea3dd9b640 1 -- 172.21.15.29:0/17427055 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56264d396fe0 con 0x56264d393a50 2024-03-16T11:54:03.109 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:03.104+0000 7fea3dd9b640 1 --2- 172.21.15.29:0/17427055 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56264d393a50 0x56264d396210 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fea34097c30 tx=0x7fea34097f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6726f10cbed9cd8 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:54:03.109 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:03.104+0000 7fea25ffb640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-16T11:54:03.109 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:03.104+0000 7fea25ffb640 1 -- 172.21.15.29:0/17427055 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea3409b130 con 0x56264d393a50 2024-03-16T11:54:03.109 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:03.104+0000 7fea25ffb640 1 -- 172.21.15.29:0/17427055 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7fea34003b30 con 0x56264d393a50 2024-03-16T11:54:03.109 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:03.104+0000 7fea477a2740 1 -- 172.21.15.29:0/17427055 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fea38010b50 con 0x56264d393a50 2024-03-16T11:54:03.109 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:03.104+0000 7fea25ffb640 1 -- 172.21.15.29:0/17427055 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fea3409bed0 con 0x56264d393a50 2024-03-16T11:54:03.109 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:03.104+0000 7fea477a2740 10 client.15672 fetch_fsmap learned FSMap version 24 2024-03-16T11:54:03.109 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:03.104+0000 7fea477a2740 1 -- 172.21.15.29:0/17427055 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x56264d4170f0 con 0x56264d393a50 2024-03-16T11:54:03.109 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:03.104+0000 7fea25ffb640 1 -- 172.21.15.29:0/17427055 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fea38010b50 con 0x56264d393a50 2024-03-16T11:54:03.110 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:03.104+0000 7fea25ffb640 1 -- 172.21.15.29:0/17427055 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7fea340ccb40 con 0x56264d393a50 2024-03-16T11:54:03.110 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:03.104+0000 7fea477a2740 10 client.15672 fetch_fsmap finished waiting for FSMap version 24 2024-03-16T11:54:03.110 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:03.104+0000 7fea477a2740 10 client.15672 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T11:54:03.110 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:03.104+0000 7fea477a2740 20 client.15672 populate_metadata read hostname 'smithi029' 2024-03-16T11:54:03.110 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:03.104+0000 7fea477a2740 1 --2- 172.21.15.29:0/17427055 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x56264d417820 0x56264d419c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:54:03.110 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:03.105+0000 7fea477a2740 4 client.15672 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-16T11:54:03.110 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:03.105+0000 7fea477a2740 1 -- 172.21.15.29:0/17427055 --> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] -- command(tid 0: {"prefix": "flush journal"}) -- 0x56264d2ff2d0 con 0x56264d417820 2024-03-16T11:54:03.110 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:03.105+0000 7fea3e59c640 1 --2- 172.21.15.29:0/17427055 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x56264d417820 0x56264d419c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:54:03.110 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:03.105+0000 7fea3e59c640 1 --2- 172.21.15.29:0/17427055 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x56264d417820 0x56264d419c40 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bfe5fb1a27fbf731 server_cookie=c82a14fb64ea2102 in_seq=0 out_seq=0 2024-03-16T11:54:03.110 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:03.105+0000 7fea25ffb640 10 client.15672 ms_handle_connect on v2:172.21.15.152:6838/1548731833 2024-03-16T11:54:03.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:54:02 smithi152 ceph-mon[36291]: pgmap v97: 97 pgs: 97 active+clean; 47 MiB data, 722 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 3.3 MiB/s wr, 278 op/s 2024-03-16T11:54:03.280 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:03.279+0000 7fea25ffb640 1 -- 172.21.15.29:0/17427055 <== mds.0 v2:172.21.15.152:6838/1548731833 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x56264d2ff2d0 con 0x56264d417820 2024-03-16T11:54:03.280 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:03.279+0000 7fea25ffb640 10 client.15672 handle_command_reply: tid=0 2024-03-16T11:54:03.280 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:03.279+0000 7fea477a2740 1 client.15672 shutdown 2024-03-16T11:54:03.281 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:03.279+0000 7fea477a2740 1 -- 172.21.15.29:0/17427055 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56264d393a50 msgr2=0x56264d396210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:54:03.281 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:03.279+0000 7fea477a2740 1 --2- 172.21.15.29:0/17427055 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56264d393a50 0x56264d396210 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fea34097c30 tx=0x7fea34097f70 comp rx=0 tx=0).stop 2024-03-16T11:54:03.281 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:03.279+0000 7fea477a2740 1 -- 172.21.15.29:0/17427055 shutdown_connections 2024-03-16T11:54:03.281 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:03.279+0000 7fea477a2740 1 --2- 172.21.15.29:0/17427055 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x56264d417820 0x56264d419c40 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:54:03.281 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:03.279+0000 7fea477a2740 1 --2- 172.21.15.29:0/17427055 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56264d39c820 0x56264d396840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:54:03.281 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:03.279+0000 7fea25ffb640 0 client.15672 ms_handle_reset on v2:172.21.15.152:6838/1548731833 2024-03-16T11:54:03.281 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:03.279+0000 7fea477a2740 1 --2- 172.21.15.29:0/17427055 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56264d393a50 0x56264d396210 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:54:03.281 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:03.279+0000 7fea477a2740 1 --2- 172.21.15.29:0/17427055 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56264d38fa60 0x56264d395cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:54:03.281 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:03.279+0000 7fea477a2740 1 -- 172.21.15.29:0/17427055 >> 172.21.15.29:0/17427055 conn(0x56264d38da50 msgr2=0x56264d38c320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:54:03.281 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:03.280+0000 7fea477a2740 1 -- 172.21.15.29:0/17427055 shutdown_connections 2024-03-16T11:54:03.281 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:03.280+0000 7fea477a2740 1 -- 172.21.15.29:0/17427055 wait complete. 2024-03-16T11:54:03.281 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:03.280+0000 7fea477a2740 20 client.15672 trim_cache size 0 max 16384 2024-03-16T11:54:03.282 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:{ 2024-03-16T11:54:03.282 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "message": "", 2024-03-16T11:54:03.282 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "return_code": 0 2024-03-16T11:54:03.282 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:} 2024-03-16T11:54:03.283 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:03.280+0000 7fea44adc640 1 -- 172.21.15.29:0/3029013668 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7fea280a2a10 msgr2=0x7fea280a4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:54:03.283 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:03.280+0000 7fea44adc640 1 --2- 172.21.15.29:0/3029013668 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7fea280a2a10 0x7fea280a4ed0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fea38005c40 tx=0x7fea38004ab0 comp rx=0 tx=0).stop 2024-03-16T11:54:03.283 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:03.280+0000 7fea44adc640 1 -- 172.21.15.29:0/3029013668 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fea401012f0 msgr2=0x7fea401985d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:54:03.283 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:03.280+0000 7fea44adc640 1 --2- 172.21.15.29:0/3029013668 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fea401012f0 0x7fea401985d0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7fea3400ea60 tx=0x7fea3400ef20 comp rx=0 tx=0).stop 2024-03-16T11:54:03.283 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:03.280+0000 7fea44adc640 1 -- 172.21.15.29:0/3029013668 shutdown_connections 2024-03-16T11:54:03.283 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:03.280+0000 7fea44adc640 1 --2- 172.21.15.29:0/3029013668 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7fea280a2a10 0x7fea280a4ed0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:54:03.283 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:03.280+0000 7fea44adc640 1 --2- 172.21.15.29:0/3029013668 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fea40105680 0x7fea4019c950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:54:03.283 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:03.280+0000 7fea44adc640 1 --2- 172.21.15.29:0/3029013668 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fea401012f0 0x7fea401985d0 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:54:03.283 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:03.280+0000 7fea44adc640 1 --2- 172.21.15.29:0/3029013668 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fea40100930 0x7fea40198090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:54:03.283 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:03.280+0000 7fea44adc640 1 -- 172.21.15.29:0/3029013668 >> 172.21.15.29:0/3029013668 conn(0x7fea40077530 msgr2=0x7fea401044f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:54:03.283 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:03.280+0000 7fea44adc640 1 -- 172.21.15.29:0/3029013668 shutdown_connections 2024-03-16T11:54:03.284 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:03.280+0000 7fea44adc640 1 -- 172.21.15.29:0/3029013668 wait complete. 2024-03-16T11:54:03.779 INFO:tasks.workunit.client.0.smithi029.stdout: 1 245179 12.41 MB/sec warmup 108 sec latency 73.432 ms 2024-03-16T11:54:04.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:54:03 smithi154 ceph-mon[36320]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T11:54:04.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:54:03 smithi154 ceph-mon[36320]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:54:04.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:54:03 smithi154 ceph-mon[36320]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T11:54:04.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:54:03 smithi154 ceph-mon[36320]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' 2024-03-16T11:54:04.087 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:54:03 smithi029 ceph-mon[31975]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T11:54:04.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:54:03 smithi029 ceph-mon[31975]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:54:04.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:54:03 smithi029 ceph-mon[31975]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T11:54:04.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:54:03 smithi029 ceph-mon[31975]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' 2024-03-16T11:54:04.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:54:03 smithi152 ceph-mon[36291]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T11:54:04.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:54:03 smithi152 ceph-mon[36291]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:54:04.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:54:03 smithi152 ceph-mon[36291]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T11:54:04.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:54:03 smithi152 ceph-mon[36291]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' 2024-03-16T11:54:04.779 INFO:tasks.workunit.client.0.smithi029.stdout: 1 247686 12.41 MB/sec warmup 109 sec latency 26.943 ms 2024-03-16T11:54:05.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:54:04 smithi154 ceph-mon[36320]: pgmap v98: 97 pgs: 97 active+clean; 47 MiB data, 722 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 2.9 MiB/s wr, 247 op/s 2024-03-16T11:54:05.087 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:54:04 smithi029 ceph-mon[31975]: pgmap v98: 97 pgs: 97 active+clean; 47 MiB data, 722 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 2.9 MiB/s wr, 247 op/s 2024-03-16T11:54:05.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:54:04 smithi152 ceph-mon[36291]: pgmap v98: 97 pgs: 97 active+clean; 47 MiB data, 722 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 2.9 MiB/s wr, 247 op/s 2024-03-16T11:54:05.779 INFO:tasks.workunit.client.0.smithi029.stdout: 1 249471 12.38 MB/sec warmup 110 sec latency 21.471 ms 2024-03-16T11:54:06.779 INFO:tasks.workunit.client.0.smithi029.stdout: 1 252123 12.42 MB/sec warmup 111 sec latency 45.488 ms 2024-03-16T11:54:07.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:54:06 smithi154 ceph-mon[36320]: pgmap v99: 97 pgs: 97 active+clean; 47 MiB data, 722 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 2.8 MiB/s wr, 232 op/s 2024-03-16T11:54:07.087 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:54:06 smithi029 ceph-mon[31975]: pgmap v99: 97 pgs: 97 active+clean; 47 MiB data, 722 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 2.8 MiB/s wr, 232 op/s 2024-03-16T11:54:07.128 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:54:06 smithi152 ceph-mon[36291]: pgmap v99: 97 pgs: 97 active+clean; 47 MiB data, 722 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 2.8 MiB/s wr, 232 op/s 2024-03-16T11:54:07.779 INFO:tasks.workunit.client.0.smithi029.stdout: 1 254309 12.37 MB/sec warmup 112 sec latency 27.286 ms 2024-03-16T11:54:08.779 INFO:tasks.workunit.client.0.smithi029.stdout: 1 256000 12.37 MB/sec warmup 113 sec latency 65.285 ms 2024-03-16T11:54:09.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:54:08 smithi154 ceph-mon[36320]: pgmap v100: 97 pgs: 97 active+clean; 44 MiB data, 720 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 4.2 MiB/s wr, 443 op/s 2024-03-16T11:54:09.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:54:08 smithi029 ceph-mon[31975]: pgmap v100: 97 pgs: 97 active+clean; 44 MiB data, 720 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 4.2 MiB/s wr, 443 op/s 2024-03-16T11:54:09.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:54:08 smithi152 ceph-mon[36291]: pgmap v100: 97 pgs: 97 active+clean; 44 MiB data, 720 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 4.2 MiB/s wr, 443 op/s 2024-03-16T11:54:09.779 INFO:tasks.workunit.client.0.smithi029.stdout: 1 258778 12.39 MB/sec warmup 114 sec latency 21.556 ms 2024-03-16T11:54:10.779 INFO:tasks.workunit.client.0.smithi029.stdout: 1 260498 12.37 MB/sec warmup 115 sec latency 21.606 ms 2024-03-16T11:54:11.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:54:10 smithi154 ceph-mon[36320]: pgmap v101: 97 pgs: 97 active+clean; 44 MiB data, 720 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 2.8 MiB/s wr, 341 op/s 2024-03-16T11:54:11.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:54:10 smithi029 ceph-mon[31975]: pgmap v101: 97 pgs: 97 active+clean; 44 MiB data, 720 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 2.8 MiB/s wr, 341 op/s 2024-03-16T11:54:11.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:54:10 smithi152 ceph-mon[36291]: pgmap v101: 97 pgs: 97 active+clean; 44 MiB data, 720 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 2.8 MiB/s wr, 341 op/s 2024-03-16T11:54:11.780 INFO:tasks.workunit.client.0.smithi029.stdout: 1 262954 12.38 MB/sec warmup 116 sec latency 15.382 ms 2024-03-16T11:54:12.780 INFO:tasks.workunit.client.0.smithi029.stdout: 1 265532 12.38 MB/sec warmup 117 sec latency 27.895 ms 2024-03-16T11:54:13.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:54:12 smithi154 ceph-mon[36320]: pgmap v102: 97 pgs: 97 active+clean; 49 MiB data, 766 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 4.3 MiB/s wr, 449 op/s 2024-03-16T11:54:13.087 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:54:12 smithi029 ceph-mon[31975]: pgmap v102: 97 pgs: 97 active+clean; 49 MiB data, 766 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 4.3 MiB/s wr, 449 op/s 2024-03-16T11:54:13.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:54:12 smithi152 ceph-mon[36291]: pgmap v102: 97 pgs: 97 active+clean; 49 MiB data, 766 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 4.3 MiB/s wr, 449 op/s 2024-03-16T11:54:13.780 INFO:tasks.workunit.client.0.smithi029.stdout: 1 267101 12.35 MB/sec warmup 118 sec latency 47.793 ms 2024-03-16T11:54:14.780 INFO:tasks.workunit.client.0.smithi029.stdout: 1 269826 12.39 MB/sec warmup 119 sec latency 68.910 ms 2024-03-16T11:54:14.928 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:54:14 smithi152 ceph-mon[36291]: pgmap v103: 97 pgs: 97 active+clean; 49 MiB data, 766 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 2.9 MiB/s wr, 319 op/s 2024-03-16T11:54:15.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:54:14 smithi154 ceph-mon[36320]: pgmap v103: 97 pgs: 97 active+clean; 49 MiB data, 766 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 2.9 MiB/s wr, 319 op/s 2024-03-16T11:54:15.087 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:54:14 smithi029 ceph-mon[31975]: pgmap v103: 97 pgs: 97 active+clean; 49 MiB data, 766 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 2.9 MiB/s wr, 319 op/s 2024-03-16T11:54:16.370 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:16.369+0000 7f3233f59640 1 -- 172.21.15.29:0/3060699016 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f322c105680 msgr2=0x7f322c105a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:54:16.370 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:16.369+0000 7f3233f59640 1 --2- 172.21.15.29:0/3060699016 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f322c105680 0x7f322c105a60 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f3228009900 tx=0x7f322802bc30 comp rx=0 tx=0).stop 2024-03-16T11:54:16.370 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:16.369+0000 7f3233f59640 1 -- 172.21.15.29:0/3060699016 shutdown_connections 2024-03-16T11:54:16.370 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:16.369+0000 7f3233f59640 1 --2- 172.21.15.29:0/3060699016 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f322c1012f0 0x7f322c10d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:54:16.370 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:16.369+0000 7f3233f59640 1 --2- 172.21.15.29:0/3060699016 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f322c100930 0x7f322c100db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:54:16.370 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:16.369+0000 7f3233f59640 1 --2- 172.21.15.29:0/3060699016 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f322c105680 0x7f322c105a60 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:54:16.371 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:16.369+0000 7f3233f59640 1 -- 172.21.15.29:0/3060699016 >> 172.21.15.29:0/3060699016 conn(0x7f322c077530 msgr2=0x7f322c077940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:54:16.371 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:16.369+0000 7f3233f59640 1 -- 172.21.15.29:0/3060699016 shutdown_connections 2024-03-16T11:54:16.371 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:16.369+0000 7f3233f59640 1 -- 172.21.15.29:0/3060699016 wait complete. 2024-03-16T11:54:16.371 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:16.369+0000 7f3233f59640 1 Processor -- start 2024-03-16T11:54:16.371 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:16.369+0000 7f3233f59640 1 -- start start 2024-03-16T11:54:16.371 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:16.370+0000 7f3233f59640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f322c100930 0x7f322c198060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:54:16.371 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:16.370+0000 7f3233f59640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f322c1012f0 0x7f322c1985a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:54:16.371 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:16.370+0000 7f3233f59640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f322c105680 0x7f322c19c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:54:16.371 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:16.370+0000 7f3233f59640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f322c110ba0 con 0x7f322c105680 2024-03-16T11:54:16.371 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:16.370+0000 7f3233f59640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f322c110a20 con 0x7f322c100930 2024-03-16T11:54:16.371 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:16.370+0000 7f3233f59640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f322c110d20 con 0x7f322c1012f0 2024-03-16T11:54:16.371 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:16.370+0000 7f32324f6640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f322c105680 0x7f322c19c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:54:16.371 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:16.370+0000 7f32324f6640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f322c105680 0x7f322c19c8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:37958/0 (socket says 172.21.15.29:37958) 2024-03-16T11:54:16.371 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:16.370+0000 7f32324f6640 1 -- 172.21.15.29:0/3663957166 learned_addr learned my addr 172.21.15.29:0/3663957166 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:54:16.371 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:16.370+0000 7f3231cf5640 1 --2- 172.21.15.29:0/3663957166 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f322c100930 0x7f322c198060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:54:16.372 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:16.370+0000 7f32314f4640 1 --2- 172.21.15.29:0/3663957166 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f322c1012f0 0x7f322c1985a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:54:16.372 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:16.370+0000 7f32324f6640 1 -- 172.21.15.29:0/3663957166 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f322c1012f0 msgr2=0x7f322c1985a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:54:16.372 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:16.370+0000 7f32324f6640 1 --2- 172.21.15.29:0/3663957166 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f322c1012f0 0x7f322c1985a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:54:16.372 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:16.370+0000 7f32324f6640 1 -- 172.21.15.29:0/3663957166 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f322c100930 msgr2=0x7f322c198060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:54:16.372 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:16.370+0000 7f32324f6640 1 --2- 172.21.15.29:0/3663957166 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f322c100930 0x7f322c198060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:54:16.372 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:16.370+0000 7f32324f6640 1 -- 172.21.15.29:0/3663957166 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f322c19d000 con 0x7f322c105680 2024-03-16T11:54:16.372 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:16.370+0000 7f32324f6640 1 --2- 172.21.15.29:0/3663957166 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f322c105680 0x7f322c19c8d0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f322400ea60 tx=0x7f322400ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:54:16.372 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:16.370+0000 7f3222ffd640 1 -- 172.21.15.29:0/3663957166 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f322400ce40 con 0x7f322c105680 2024-03-16T11:54:16.372 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:16.370+0000 7f3222ffd640 1 -- 172.21.15.29:0/3663957166 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3224005020 con 0x7f322c105680 2024-03-16T11:54:16.372 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:16.370+0000 7f3233f59640 1 -- 172.21.15.29:0/3663957166 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f322c19d2f0 con 0x7f322c105680 2024-03-16T11:54:16.372 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:16.370+0000 7f3222ffd640 1 -- 172.21.15.29:0/3663957166 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3224010690 con 0x7f322c105680 2024-03-16T11:54:16.372 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:16.371+0000 7f3233f59640 1 -- 172.21.15.29:0/3663957166 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f322c1a4b80 con 0x7f322c105680 2024-03-16T11:54:16.379 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:16.373+0000 7f3222ffd640 1 -- 172.21.15.29:0/3663957166 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7f3224005520 con 0x7f322c105680 2024-03-16T11:54:16.379 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:16.373+0000 7f3236c17740 1 Processor -- start 2024-03-16T11:54:16.379 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:16.373+0000 7f3236c17740 1 -- start start 2024-03-16T11:54:16.379 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:16.374+0000 7f3236c17740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55839ee82130 0x55839ee82510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:54:16.379 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:16.374+0000 7f3236c17740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55839ee82a50 0x55839ee8b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:54:16.379 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:16.374+0000 7f3236c17740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55839ee8b820 0x55839ee8dc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:54:16.380 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:16.374+0000 7f3236c17740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55839ede6a40 con 0x55839ee82130 2024-03-16T11:54:16.380 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:16.374+0000 7f3236c17740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x55839ed8f920 con 0x55839ee82a50 2024-03-16T11:54:16.380 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:16.374+0000 7f3236c17740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55839ede2ae0 con 0x55839ee8b820 2024-03-16T11:54:16.380 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:16.374+0000 7f32314f4640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55839ee82130 0x55839ee82510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:54:16.380 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:16.374+0000 7f3222ffd640 1 --2- 172.21.15.29:0/3663957166 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f32180a2a90 0x7f32180a4f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:54:16.380 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:16.374+0000 7f3231cf5640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55839ee82a50 0x55839ee8b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:54:16.380 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:16.374+0000 7f32314f4640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55839ee82130 0x55839ee82510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:37964/0 (socket says 172.21.15.29:37964) 2024-03-16T11:54:16.380 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:16.374+0000 7f32314f4640 1 -- 172.21.15.29:0/2411923319 learned_addr learned my addr 172.21.15.29:0/2411923319 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:54:16.380 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:16.374+0000 7f32324f6640 1 --2- 172.21.15.29:0/3663957166 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f32180a2a90 0x7f32180a4f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:54:16.380 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:16.374+0000 7f32314f4640 1 --2- 172.21.15.29:0/2411923319 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55839ee8b820 0x55839ee8dc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:54:16.380 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:16.374+0000 7f3222ffd640 1 -- 172.21.15.29:0/3663957166 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7f32240cf1a0 con 0x7f322c105680 2024-03-16T11:54:16.380 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:16.374+0000 7f32314f4640 1 -- 172.21.15.29:0/2411923319 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55839ee8b820 msgr2=0x55839ee8dc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:54:16.380 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:16.374+0000 7f32314f4640 1 --2- 172.21.15.29:0/2411923319 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55839ee8b820 0x55839ee8dc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:54:16.380 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:16.374+0000 7f32314f4640 1 -- 172.21.15.29:0/2411923319 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55839ee82a50 msgr2=0x55839ee8b2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:54:16.380 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:16.374+0000 7f32314f4640 1 --2- 172.21.15.29:0/2411923319 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55839ee82a50 0x55839ee8b2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:54:16.381 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:16.374+0000 7f32314f4640 1 -- 172.21.15.29:0/2411923319 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55839eda7460 con 0x55839ee82130 2024-03-16T11:54:16.381 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:16.374+0000 7f32314f4640 1 --2- 172.21.15.29:0/2411923319 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55839ee82130 0x55839ee82510 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f321c00bfc0 tx=0x7f321c00b8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61b781c32673f7d8 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:54:16.381 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:16.374+0000 7f32324f6640 1 --2- 172.21.15.29:0/3663957166 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f32180a2a90 0x7f32180a4f50 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f32240ccf20 tx=0x7f3224086360 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:54:16.381 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:16.374+0000 7f3217fff640 1 -- 172.21.15.29:0/2411923319 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f321c004220 con 0x55839ee82130 2024-03-16T11:54:16.381 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:16.375+0000 7f3217fff640 1 -- 172.21.15.29:0/2411923319 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f321c0043c0 con 0x55839ee82130 2024-03-16T11:54:16.381 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:16.375+0000 7f3217fff640 1 -- 172.21.15.29:0/2411923319 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f321c010970 con 0x55839ee82130 2024-03-16T11:54:16.381 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:16.375+0000 7f3236c17740 1 -- 172.21.15.29:0/2411923319 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55839ee82130 msgr2=0x55839ee82510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:54:16.381 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:16.375+0000 7f3236c17740 1 --2- 172.21.15.29:0/2411923319 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55839ee82130 0x55839ee82510 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f321c00bfc0 tx=0x7f321c00b8c0 comp rx=0 tx=0).stop 2024-03-16T11:54:16.381 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:16.375+0000 7f3236c17740 1 -- 172.21.15.29:0/2411923319 shutdown_connections 2024-03-16T11:54:16.381 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:16.375+0000 7f3236c17740 1 --2- 172.21.15.29:0/2411923319 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55839ee8b820 0x55839ee8dc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:54:16.381 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:16.375+0000 7f3236c17740 1 --2- 172.21.15.29:0/2411923319 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55839ee82a50 0x55839ee8b2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:54:16.381 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:16.375+0000 7f3236c17740 1 --2- 172.21.15.29:0/2411923319 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55839ee82130 0x55839ee82510 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:54:16.381 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:16.375+0000 7f3236c17740 1 -- 172.21.15.29:0/2411923319 >> 172.21.15.29:0/2411923319 conn(0x55839ee7ca50 msgr2=0x55839ee7ce30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:54:16.381 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:16.375+0000 7f3236c17740 1 -- 172.21.15.29:0/2411923319 shutdown_connections 2024-03-16T11:54:16.381 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:16.375+0000 7f3236c17740 1 -- 172.21.15.29:0/2411923319 wait complete. 2024-03-16T11:54:16.382 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:16.376+0000 7f3236c17740 1 -- start start 2024-03-16T11:54:16.382 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:16.376+0000 7f3236c17740 1 Processor -- start 2024-03-16T11:54:16.382 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:16.376+0000 7f3236c17740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55839ee82130 0x55839ee887e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:54:16.382 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:16.376+0000 7f3236c17740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55839ee82a50 0x55839ee88d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:54:16.382 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:16.376+0000 7f3236c17740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55839ee8b820 0x55839ee89260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:54:16.382 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:16.376+0000 7f3236c17740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55839ee8e450 con 0x55839ee82a50 2024-03-16T11:54:16.382 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:16.376+0000 7f3236c17740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x55839eda2d70 con 0x55839ee82130 2024-03-16T11:54:16.382 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:16.376+0000 7f3236c17740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55839ee8e5d0 con 0x55839ee8b820 2024-03-16T11:54:16.383 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:16.376+0000 7f3236c17740 10 client.0 _ll_register_callbacks cb 0x55839eafaa00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T11:54:16.383 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:16.376+0000 7f3231cf5640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55839ee82a50 0x55839ee88d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:54:16.383 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:16.376+0000 7f3231cf5640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55839ee82a50 0x55839ee88d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:37970/0 (socket says 172.21.15.29:37970) 2024-03-16T11:54:16.383 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:16.376+0000 7f3231cf5640 1 -- 172.21.15.29:0/2686610205 learned_addr learned my addr 172.21.15.29:0/2686610205 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:54:16.383 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:16.376+0000 7f32314f4640 1 --2- 172.21.15.29:0/2686610205 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55839ee82130 0x55839ee887e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:54:16.383 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:16.376+0000 7f32314f4640 1 --2- 172.21.15.29:0/2686610205 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55839ee8b820 0x55839ee89260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:54:16.383 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:16.377+0000 7f3231cf5640 1 -- 172.21.15.29:0/2686610205 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55839ee8b820 msgr2=0x55839ee89260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:54:16.383 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:16.377+0000 7f3231cf5640 1 --2- 172.21.15.29:0/2686610205 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55839ee8b820 0x55839ee89260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:54:16.383 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:16.377+0000 7f3231cf5640 1 -- 172.21.15.29:0/2686610205 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55839ee82130 msgr2=0x55839ee887e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:54:16.384 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:16.377+0000 7f3231cf5640 1 --2- 172.21.15.29:0/2686610205 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55839ee82130 0x55839ee887e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:54:16.384 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:16.377+0000 7f3231cf5640 1 -- 172.21.15.29:0/2686610205 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55839eafb210 con 0x55839ee82a50 2024-03-16T11:54:16.384 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:16.377+0000 7f32314f4640 1 --2- 172.21.15.29:0/2686610205 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55839ee8b820 0x55839ee89260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T11:54:16.384 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:16.377+0000 7f3231cf5640 1 --2- 172.21.15.29:0/2686610205 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55839ee82a50 0x55839ee88d20 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f3228035000 tx=0x7f322802c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bfb0d80e67b0b182 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:54:16.384 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:16.377+0000 7f32157fa640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-16T11:54:16.384 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:16.377+0000 7f32157fa640 1 -- 172.21.15.29:0/2686610205 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32280046c0 con 0x55839ee82a50 2024-03-16T11:54:16.384 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:16.377+0000 7f32157fa640 1 -- 172.21.15.29:0/2686610205 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7f3228004860 con 0x55839ee82a50 2024-03-16T11:54:16.384 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:16.377+0000 7f3236c17740 1 -- 172.21.15.29:0/2686610205 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f321c0043c0 con 0x55839ee82a50 2024-03-16T11:54:16.384 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:16.377+0000 7f32157fa640 1 -- 172.21.15.29:0/2686610205 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f3228004a60 con 0x55839ee82a50 2024-03-16T11:54:16.384 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:16.377+0000 7f3236c17740 10 client.15684 fetch_fsmap learned FSMap version 24 2024-03-16T11:54:16.384 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:16.377+0000 7f3236c17740 1 -- 172.21.15.29:0/2686610205 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55839ebc7620 con 0x55839ee82a50 2024-03-16T11:54:16.385 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:16.377+0000 7f32157fa640 1 -- 172.21.15.29:0/2686610205 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f321c0043c0 con 0x55839ee82a50 2024-03-16T11:54:16.385 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:16.378+0000 7f32157fa640 1 -- 172.21.15.29:0/2686610205 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7f3228046070 con 0x55839ee82a50 2024-03-16T11:54:16.385 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:16.378+0000 7f3236c17740 10 client.15684 fetch_fsmap finished waiting for FSMap version 24 2024-03-16T11:54:16.385 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:16.378+0000 7f3236c17740 10 client.15684 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T11:54:16.385 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:16.378+0000 7f3236c17740 20 client.15684 populate_metadata read hostname 'smithi029' 2024-03-16T11:54:16.385 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:16.378+0000 7f3236c17740 1 --2- 172.21.15.29:0/2686610205 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55839ee9b8d0 0x55839ee9dd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:54:16.385 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:16.378+0000 7f3236c17740 4 client.15684 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T11:54:16.385 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:16.378+0000 7f3236c17740 1 -- 172.21.15.29:0/2686610205 --> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55839edf3f80 con 0x55839ee9b8d0 2024-03-16T11:54:16.385 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:16.378+0000 7f32314f4640 1 --2- 172.21.15.29:0/2686610205 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55839ee9b8d0 0x55839ee9dd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:54:16.385 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:16.378+0000 7f32157fa640 10 client.15684 ms_handle_connect on v2:172.21.15.152:6838/1548731833 2024-03-16T11:54:16.385 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:16.378+0000 7f32314f4640 1 --2- 172.21.15.29:0/2686610205 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55839ee9b8d0 0x55839ee9dd40 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=55e5f14bfef302ad server_cookie=fc626d9c528e99c2 in_seq=0 out_seq=0 2024-03-16T11:54:16.385 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:16.379+0000 7f32157fa640 1 -- 172.21.15.29:0/2686610205 <== mds.0 v2:172.21.15.152:6838/1548731833 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55839edf3f80 con 0x55839ee9b8d0 2024-03-16T11:54:16.385 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:16.379+0000 7f32157fa640 10 client.15684 handle_command_reply: tid=0 2024-03-16T11:54:16.386 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:16.379+0000 7f3236c17740 1 client.15684 shutdown 2024-03-16T11:54:16.386 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:16.380+0000 7f3236c17740 1 -- 172.21.15.29:0/2686610205 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55839ee82a50 msgr2=0x55839ee88d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:54:16.386 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:16.380+0000 7f3236c17740 1 --2- 172.21.15.29:0/2686610205 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55839ee82a50 0x55839ee88d20 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f3228035000 tx=0x7f322802c7e0 comp rx=0 tx=0).stop 2024-03-16T11:54:16.386 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:16.380+0000 7f3236c17740 1 -- 172.21.15.29:0/2686610205 shutdown_connections 2024-03-16T11:54:16.386 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:16.380+0000 7f3236c17740 1 --2- 172.21.15.29:0/2686610205 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55839ee8b820 0x55839ee89260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:54:16.386 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:16.380+0000 7f3236c17740 1 --2- 172.21.15.29:0/2686610205 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55839ee82a50 0x55839ee88d20 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:54:16.386 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:16.380+0000 7f3236c17740 1 --2- 172.21.15.29:0/2686610205 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55839ee9b8d0 0x55839ee9dd40 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:54:16.386 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:16.380+0000 7f3236c17740 1 --2- 172.21.15.29:0/2686610205 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55839ee82130 0x55839ee887e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:54:16.386 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:16.380+0000 7f3236c17740 1 -- 172.21.15.29:0/2686610205 >> 172.21.15.29:0/2686610205 conn(0x55839ee7ca50 msgr2=0x55839ee7e7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:54:16.386 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:16.380+0000 7f32157fa640 0 client.15684 ms_handle_reset on v2:172.21.15.152:6838/1548731833 2024-03-16T11:54:16.386 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:16.380+0000 7f3236c17740 1 -- 172.21.15.29:0/2686610205 shutdown_connections 2024-03-16T11:54:16.386 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:16.380+0000 7f3236c17740 1 -- 172.21.15.29:0/2686610205 wait complete. 2024-03-16T11:54:16.386 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:16.380+0000 7f3236c17740 20 client.15684 trim_cache size 0 max 16384 2024-03-16T11:54:16.399 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:16.395+0000 7f3236c17740 1 Processor -- start 2024-03-16T11:54:16.399 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:16.395+0000 7f3236c17740 1 -- start start 2024-03-16T11:54:16.399 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:16.395+0000 7f3236c17740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55839ee82130 0x55839ee8af10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:54:16.399 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:16.395+0000 7f3236c17740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55839ee82a50 0x55839ee971f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:54:16.399 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:16.395+0000 7f3236c17740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55839ee8b820 0x55839eea3650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:54:16.399 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:16.395+0000 7f3236c17740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f322c1afc60 con 0x55839ee82130 2024-03-16T11:54:16.399 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:16.395+0000 7f3236c17740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f322c1ad530 con 0x55839ee8b820 2024-03-16T11:54:16.399 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:16.395+0000 7f3236c17740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f322c1ab380 con 0x55839ee82a50 2024-03-16T11:54:16.399 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:16.395+0000 7f32314f4640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55839ee82130 0x55839ee8af10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:54:16.399 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:16.395+0000 7f32314f4640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55839ee82130 0x55839ee8af10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:37976/0 (socket says 172.21.15.29:37976) 2024-03-16T11:54:16.399 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:16.395+0000 7f32314f4640 1 -- 172.21.15.29:0/3422056535 learned_addr learned my addr 172.21.15.29:0/3422056535 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:54:16.399 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:16.395+0000 7f3231cf5640 1 --2- 172.21.15.29:0/3422056535 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55839ee82a50 0x55839ee971f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:54:16.399 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:16.395+0000 7f32314f4640 1 --2- 172.21.15.29:0/3422056535 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55839ee8b820 0x55839eea3650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:54:16.399 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:16.395+0000 7f32314f4640 1 -- 172.21.15.29:0/3422056535 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55839ee82a50 msgr2=0x55839ee971f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:54:16.400 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:16.395+0000 7f32314f4640 1 --2- 172.21.15.29:0/3422056535 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55839ee82a50 0x55839ee971f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:54:16.400 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:16.395+0000 7f32314f4640 1 -- 172.21.15.29:0/3422056535 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55839ee8b820 msgr2=0x55839eea3650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:54:16.400 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:16.395+0000 7f32314f4640 1 --2- 172.21.15.29:0/3422056535 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55839ee8b820 0x55839eea3650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:54:16.400 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:16.395+0000 7f32314f4640 1 -- 172.21.15.29:0/3422056535 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55839ee843b0 con 0x55839ee82130 2024-03-16T11:54:16.400 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:16.395+0000 7f3231cf5640 1 --2- 172.21.15.29:0/3422056535 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55839ee82a50 0x55839ee971f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T11:54:16.400 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:16.395+0000 7f32314f4640 1 --2- 172.21.15.29:0/3422056535 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55839ee82130 0x55839ee8af10 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f321c012ec0 tx=0x7f321c010550 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6dc3d74ef1cb62ac server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:54:16.400 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:16.395+0000 7f32177fe640 1 -- 172.21.15.29:0/3422056535 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f321c022040 con 0x55839ee82130 2024-03-16T11:54:16.400 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:16.396+0000 7f32177fe640 1 -- 172.21.15.29:0/3422056535 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f321c0035f0 con 0x55839ee82130 2024-03-16T11:54:16.400 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:16.396+0000 7f32177fe640 1 -- 172.21.15.29:0/3422056535 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f321c0043d0 con 0x55839ee82130 2024-03-16T11:54:16.400 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:16.396+0000 7f3236c17740 1 -- 172.21.15.29:0/3422056535 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55839ee82130 msgr2=0x55839ee8af10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:54:16.400 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:16.396+0000 7f3236c17740 1 --2- 172.21.15.29:0/3422056535 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55839ee82130 0x55839ee8af10 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f321c012ec0 tx=0x7f321c010550 comp rx=0 tx=0).stop 2024-03-16T11:54:16.400 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:16.396+0000 7f3236c17740 1 -- 172.21.15.29:0/3422056535 shutdown_connections 2024-03-16T11:54:16.400 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:16.396+0000 7f3236c17740 1 --2- 172.21.15.29:0/3422056535 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55839ee8b820 0x55839eea3650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:54:16.400 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:16.396+0000 7f3236c17740 1 --2- 172.21.15.29:0/3422056535 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55839ee82a50 0x55839ee971f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:54:16.400 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:16.396+0000 7f3236c17740 1 --2- 172.21.15.29:0/3422056535 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55839ee82130 0x55839ee8af10 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:54:16.401 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:16.396+0000 7f3236c17740 1 -- 172.21.15.29:0/3422056535 >> 172.21.15.29:0/3422056535 conn(0x55839ee7ca50 msgr2=0x55839ee89030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:54:16.401 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:16.396+0000 7f3236c17740 1 -- 172.21.15.29:0/3422056535 shutdown_connections 2024-03-16T11:54:16.401 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:16.396+0000 7f3236c17740 1 -- 172.21.15.29:0/3422056535 wait complete. 2024-03-16T11:54:16.401 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:16.396+0000 7f3236c17740 1 -- start start 2024-03-16T11:54:16.401 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:16.396+0000 7f3236c17740 1 Processor -- start 2024-03-16T11:54:16.401 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:16.397+0000 7f3236c17740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55839ee82130 0x55839ee8f940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:54:16.401 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:16.397+0000 7f3236c17740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55839ee82a50 0x55839ee8fe80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:54:16.401 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:16.397+0000 7f3236c17740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55839ee8b820 0x55839ef00c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:54:16.401 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:16.397+0000 7f3236c17740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f322c1ab9f0 con 0x55839ee8b820 2024-03-16T11:54:16.401 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:16.397+0000 7f3236c17740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f322c1ace60 con 0x55839ee82a50 2024-03-16T11:54:16.401 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:16.397+0000 7f3236c17740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f322c1ac0c0 con 0x55839ee82130 2024-03-16T11:54:16.401 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:16.397+0000 7f3236c17740 10 client.0 _ll_register_callbacks cb 0x7f3228005b50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T11:54:16.401 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:16.397+0000 7f32314f4640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55839ee8b820 0x55839ef00c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:54:16.401 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:16.397+0000 7f32314f4640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55839ee8b820 0x55839ef00c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:37988/0 (socket says 172.21.15.29:37988) 2024-03-16T11:54:16.402 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:16.397+0000 7f32314f4640 1 -- 172.21.15.29:0/920629696 learned_addr learned my addr 172.21.15.29:0/920629696 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:54:16.402 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:16.397+0000 7f3231cf5640 1 --2- 172.21.15.29:0/920629696 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55839ee82a50 0x55839ee8fe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:54:16.402 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:16.397+0000 7f32314f4640 1 --2- 172.21.15.29:0/920629696 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55839ee82130 0x55839ee8f940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:54:16.402 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:16.397+0000 7f32314f4640 1 -- 172.21.15.29:0/920629696 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55839ee82130 msgr2=0x55839ee8f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:54:16.402 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:16.397+0000 7f32314f4640 1 --2- 172.21.15.29:0/920629696 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55839ee82130 0x55839ee8f940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:54:16.402 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:16.397+0000 7f32314f4640 1 -- 172.21.15.29:0/920629696 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55839ee82a50 msgr2=0x55839ee8fe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:54:16.402 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:16.397+0000 7f32314f4640 1 --2- 172.21.15.29:0/920629696 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55839ee82a50 0x55839ee8fe80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:54:16.402 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:16.397+0000 7f32314f4640 1 -- 172.21.15.29:0/920629696 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55839ef01270 con 0x55839ee8b820 2024-03-16T11:54:16.402 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:16.397+0000 7f3231cf5640 1 --2- 172.21.15.29:0/920629696 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55839ee82a50 0x55839ee8fe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:54:16.402 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:16.398+0000 7f32314f4640 1 --2- 172.21.15.29:0/920629696 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55839ee8b820 0x55839ef00c80 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f321c00b510 tx=0x7f321c00b630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7cb5efce264fc55c server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:54:16.402 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:16.398+0000 7f32157fa640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-16T11:54:16.402 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:16.398+0000 7f32157fa640 1 -- 172.21.15.29:0/920629696 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f321c003b40 con 0x55839ee8b820 2024-03-16T11:54:16.402 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:16.398+0000 7f32157fa640 1 -- 172.21.15.29:0/920629696 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7f321c005020 con 0x55839ee8b820 2024-03-16T11:54:16.402 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:16.398+0000 7f3236c17740 1 -- 172.21.15.29:0/920629696 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f321c0035f0 con 0x55839ee8b820 2024-03-16T11:54:16.403 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:16.398+0000 7f32157fa640 1 -- 172.21.15.29:0/920629696 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f321c005220 con 0x55839ee8b820 2024-03-16T11:54:16.403 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:16.398+0000 7f3236c17740 10 client.15690 fetch_fsmap learned FSMap version 24 2024-03-16T11:54:16.403 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:16.398+0000 7f3236c17740 1 -- 172.21.15.29:0/920629696 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55839ef02560 con 0x55839ee8b820 2024-03-16T11:54:16.403 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:16.398+0000 7f32157fa640 1 -- 172.21.15.29:0/920629696 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f321c0035f0 con 0x55839ee8b820 2024-03-16T11:54:16.403 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:16.398+0000 7f32157fa640 1 -- 172.21.15.29:0/920629696 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7f321c028040 con 0x55839ee8b820 2024-03-16T11:54:16.403 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:16.398+0000 7f3236c17740 10 client.15690 fetch_fsmap finished waiting for FSMap version 24 2024-03-16T11:54:16.403 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:16.398+0000 7f3236c17740 10 client.15690 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T11:54:16.403 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:16.398+0000 7f3236c17740 20 client.15690 populate_metadata read hostname 'smithi029' 2024-03-16T11:54:16.403 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:16.398+0000 7f3236c17740 1 --2- 172.21.15.29:0/920629696 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55839ef02930 0x55839ef04d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:54:16.403 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:16.398+0000 7f3236c17740 4 client.15690 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-16T11:54:16.403 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:16.398+0000 7f3236c17740 1 -- 172.21.15.29:0/920629696 --> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55839edee2d0 con 0x55839ef02930 2024-03-16T11:54:16.403 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:16.398+0000 7f3231cf5640 1 --2- 172.21.15.29:0/920629696 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55839ef02930 0x55839ef04d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:54:16.403 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:16.399+0000 7f3231cf5640 1 --2- 172.21.15.29:0/920629696 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55839ef02930 0x55839ef04d20 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=da9082a020a15494 server_cookie=2e99c214a6c34a86 in_seq=0 out_seq=0 2024-03-16T11:54:16.403 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:16.399+0000 7f32157fa640 10 client.15690 ms_handle_connect on v2:172.21.15.152:6838/1548731833 2024-03-16T11:54:16.620 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:16.620+0000 7f32157fa640 1 -- 172.21.15.29:0/920629696 <== mds.0 v2:172.21.15.152:6838/1548731833 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55839edee2d0 con 0x55839ef02930 2024-03-16T11:54:16.620 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:16.620+0000 7f32157fa640 10 client.15690 handle_command_reply: tid=0 2024-03-16T11:54:16.620 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:16.620+0000 7f3236c17740 1 client.15690 shutdown 2024-03-16T11:54:16.620 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:16.620+0000 7f3236c17740 1 -- 172.21.15.29:0/920629696 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55839ee8b820 msgr2=0x55839ef00c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:54:16.620 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:16.620+0000 7f3236c17740 1 --2- 172.21.15.29:0/920629696 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55839ee8b820 0x55839ef00c80 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f321c00b510 tx=0x7f321c00b630 comp rx=0 tx=0).stop 2024-03-16T11:54:16.621 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:16.620+0000 7f3236c17740 1 -- 172.21.15.29:0/920629696 shutdown_connections 2024-03-16T11:54:16.621 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:16.620+0000 7f3236c17740 1 --2- 172.21.15.29:0/920629696 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55839ee8b820 0x55839ef00c80 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:54:16.621 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:16.620+0000 7f3236c17740 1 --2- 172.21.15.29:0/920629696 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55839ef02930 0x55839ef04d20 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:54:16.621 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:16.620+0000 7f3236c17740 1 --2- 172.21.15.29:0/920629696 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55839ee82a50 0x55839ee8fe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:54:16.621 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:16.620+0000 7f32157fa640 0 client.15690 ms_handle_reset on v2:172.21.15.152:6838/1548731833 2024-03-16T11:54:16.621 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:16.620+0000 7f3236c17740 1 --2- 172.21.15.29:0/920629696 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55839ee82130 0x55839ee8f940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:54:16.621 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:16.620+0000 7f3236c17740 1 -- 172.21.15.29:0/920629696 >> 172.21.15.29:0/920629696 conn(0x55839ee7ca50 msgr2=0x55839ee7b320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:54:16.621 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:16.620+0000 7f3236c17740 1 -- 172.21.15.29:0/920629696 shutdown_connections 2024-03-16T11:54:16.621 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:16.620+0000 7f3236c17740 1 -- 172.21.15.29:0/920629696 wait complete. 2024-03-16T11:54:16.621 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:16.620+0000 7f3236c17740 20 client.15690 trim_cache size 0 max 16384 2024-03-16T11:54:16.622 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:{ 2024-03-16T11:54:16.622 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "message": "", 2024-03-16T11:54:16.622 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "return_code": 0 2024-03-16T11:54:16.622 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:} 2024-03-16T11:54:16.623 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:16.621+0000 7f3233f59640 1 -- 172.21.15.29:0/3663957166 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f32180a2a90 msgr2=0x7f32180a4f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:54:16.624 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:16.621+0000 7f3233f59640 1 --2- 172.21.15.29:0/3663957166 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f32180a2a90 0x7f32180a4f50 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f32240ccf20 tx=0x7f3224086360 comp rx=0 tx=0).stop 2024-03-16T11:54:16.624 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:16.621+0000 7f3233f59640 1 -- 172.21.15.29:0/3663957166 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f322c105680 msgr2=0x7f322c19c8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:54:16.624 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:16.621+0000 7f3233f59640 1 --2- 172.21.15.29:0/3663957166 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f322c105680 0x7f322c19c8d0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f322400ea60 tx=0x7f322400ef20 comp rx=0 tx=0).stop 2024-03-16T11:54:16.624 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:16.621+0000 7f3233f59640 1 -- 172.21.15.29:0/3663957166 shutdown_connections 2024-03-16T11:54:16.624 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:16.621+0000 7f3233f59640 1 --2- 172.21.15.29:0/3663957166 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f32180a2a90 0x7f32180a4f50 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:54:16.624 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:16.621+0000 7f3233f59640 1 --2- 172.21.15.29:0/3663957166 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f322c105680 0x7f322c19c8d0 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:54:16.624 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:16.621+0000 7f3233f59640 1 --2- 172.21.15.29:0/3663957166 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f322c1012f0 0x7f322c1985a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:54:16.624 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:16.621+0000 7f3233f59640 1 --2- 172.21.15.29:0/3663957166 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f322c100930 0x7f322c198060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:54:16.624 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:16.621+0000 7f3233f59640 1 -- 172.21.15.29:0/3663957166 >> 172.21.15.29:0/3663957166 conn(0x7f322c077530 msgr2=0x7f322c0690e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:54:16.624 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:16.621+0000 7f3233f59640 1 -- 172.21.15.29:0/3663957166 shutdown_connections 2024-03-16T11:54:16.624 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:16.621+0000 7f3233f59640 1 -- 172.21.15.29:0/3663957166 wait complete. 2024-03-16T11:54:16.780 INFO:tasks.workunit.client.0.smithi029.stdout: 1 273590 12.16 MB/sec execute 1 sec latency 137.310 ms 2024-03-16T11:54:17.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:54:16 smithi154 ceph-mon[36320]: pgmap v104: 97 pgs: 97 active+clean; 49 MiB data, 766 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 2.9 MiB/s wr, 319 op/s 2024-03-16T11:54:17.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:54:16 smithi029 ceph-mon[31975]: pgmap v104: 97 pgs: 97 active+clean; 49 MiB data, 766 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 2.9 MiB/s wr, 319 op/s 2024-03-16T11:54:17.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:54:16 smithi152 ceph-mon[36291]: pgmap v104: 97 pgs: 97 active+clean; 49 MiB data, 766 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 2.9 MiB/s wr, 319 op/s 2024-03-16T11:54:17.780 INFO:tasks.workunit.client.0.smithi029.stdout: 1 276130 12.37 MB/sec execute 2 sec latency 18.306 ms 2024-03-16T11:54:18.780 INFO:tasks.workunit.client.0.smithi029.stdout: 1 277732 11.07 MB/sec execute 3 sec latency 22.828 ms 2024-03-16T11:54:19.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:54:18 smithi154 ceph-mon[36320]: pgmap v105: 97 pgs: 97 active+clean; 55 MiB data, 825 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 4.4 MiB/s wr, 421 op/s 2024-03-16T11:54:19.087 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:54:18 smithi029 ceph-mon[31975]: pgmap v105: 97 pgs: 97 active+clean; 55 MiB data, 825 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 4.4 MiB/s wr, 421 op/s 2024-03-16T11:54:19.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:54:18 smithi152 ceph-mon[36291]: pgmap v105: 97 pgs: 97 active+clean; 55 MiB data, 825 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 4.4 MiB/s wr, 421 op/s 2024-03-16T11:54:19.780 INFO:tasks.workunit.client.0.smithi029.stdout: 1 280347 12.26 MB/sec execute 4 sec latency 63.052 ms 2024-03-16T11:54:20.780 INFO:tasks.workunit.client.0.smithi029.stdout: 1 282491 11.25 MB/sec execute 5 sec latency 26.045 ms 2024-03-16T11:54:21.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:54:20 smithi154 ceph-mon[36320]: pgmap v106: 97 pgs: 97 active+clean; 55 MiB data, 825 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.0 MiB/s wr, 210 op/s 2024-03-16T11:54:21.087 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:54:20 smithi029 ceph-mon[31975]: pgmap v106: 97 pgs: 97 active+clean; 55 MiB data, 825 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.0 MiB/s wr, 210 op/s 2024-03-16T11:54:21.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:54:20 smithi152 ceph-mon[36291]: pgmap v106: 97 pgs: 97 active+clean; 55 MiB data, 825 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.0 MiB/s wr, 210 op/s 2024-03-16T11:54:21.780 INFO:tasks.workunit.client.0.smithi029.stdout: 1 284404 11.59 MB/sec execute 6 sec latency 25.294 ms 2024-03-16T11:54:22.780 INFO:tasks.workunit.client.0.smithi029.stdout: 1 287418 12.46 MB/sec execute 7 sec latency 22.327 ms 2024-03-16T11:54:23.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:54:22 smithi154 ceph-mon[36320]: pgmap v107: 97 pgs: 97 active+clean; 44 MiB data, 802 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 4.4 MiB/s wr, 425 op/s 2024-03-16T11:54:23.087 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:54:22 smithi029 ceph-mon[31975]: pgmap v107: 97 pgs: 97 active+clean; 44 MiB data, 802 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 4.4 MiB/s wr, 425 op/s 2024-03-16T11:54:23.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:54:22 smithi152 ceph-mon[36291]: pgmap v107: 97 pgs: 97 active+clean; 44 MiB data, 802 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 4.4 MiB/s wr, 425 op/s 2024-03-16T11:54:23.780 INFO:tasks.workunit.client.0.smithi029.stdout: 1 289423 11.79 MB/sec execute 8 sec latency 18.557 ms 2024-03-16T11:54:24.780 INFO:tasks.workunit.client.0.smithi029.stdout: 1 291286 11.97 MB/sec execute 9 sec latency 76.012 ms 2024-03-16T11:54:25.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:54:24 smithi154 ceph-mon[36320]: pgmap v108: 97 pgs: 97 active+clean; 44 MiB data, 802 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 2.9 MiB/s wr, 317 op/s 2024-03-16T11:54:25.087 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:54:24 smithi029 ceph-mon[31975]: pgmap v108: 97 pgs: 97 active+clean; 44 MiB data, 802 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 2.9 MiB/s wr, 317 op/s 2024-03-16T11:54:25.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:54:24 smithi152 ceph-mon[36291]: pgmap v108: 97 pgs: 97 active+clean; 44 MiB data, 802 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 2.9 MiB/s wr, 317 op/s 2024-03-16T11:54:25.780 INFO:tasks.workunit.client.0.smithi029.stdout: 1 293659 11.88 MB/sec execute 10 sec latency 20.636 ms 2024-03-16T11:54:26.780 INFO:tasks.workunit.client.0.smithi029.stdout: 1 295404 11.61 MB/sec execute 11 sec latency 17.787 ms 2024-03-16T11:54:27.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:54:26 smithi154 ceph-mon[36320]: pgmap v109: 97 pgs: 97 active+clean; 44 MiB data, 802 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 2.9 MiB/s wr, 317 op/s 2024-03-16T11:54:27.087 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:54:26 smithi029 ceph-mon[31975]: pgmap v109: 97 pgs: 97 active+clean; 44 MiB data, 802 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 2.9 MiB/s wr, 317 op/s 2024-03-16T11:54:27.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:54:26 smithi152 ceph-mon[36291]: pgmap v109: 97 pgs: 97 active+clean; 44 MiB data, 802 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 2.9 MiB/s wr, 317 op/s 2024-03-16T11:54:27.780 INFO:tasks.workunit.client.0.smithi029.stdout: 1 298229 12.14 MB/sec execute 12 sec latency 21.412 ms 2024-03-16T11:54:28.780 INFO:tasks.workunit.client.0.smithi029.stdout: 1 299834 11.65 MB/sec execute 13 sec latency 60.600 ms 2024-03-16T11:54:29.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:54:28 smithi154 ceph-mon[36320]: pgmap v110: 97 pgs: 97 active+clean; 49 MiB data, 865 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 4.4 MiB/s wr, 420 op/s 2024-03-16T11:54:29.087 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:54:28 smithi029 ceph-mon[31975]: pgmap v110: 97 pgs: 97 active+clean; 49 MiB data, 865 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 4.4 MiB/s wr, 420 op/s 2024-03-16T11:54:29.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:54:28 smithi152 ceph-mon[36291]: pgmap v110: 97 pgs: 97 active+clean; 49 MiB data, 865 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 4.4 MiB/s wr, 420 op/s 2024-03-16T11:54:29.729 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:29.730+0000 7f995bb3b640 1 -- 172.21.15.29:0/3851356949 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9954105680 msgr2=0x7f9954105a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:54:29.729 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:29.730+0000 7f995bb3b640 1 --2- 172.21.15.29:0/3851356949 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9954105680 0x7f9954105a60 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f9950009890 tx=0x7f995002bb70 comp rx=0 tx=0).stop 2024-03-16T11:54:29.729 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:29.730+0000 7f995bb3b640 1 -- 172.21.15.29:0/3851356949 shutdown_connections 2024-03-16T11:54:29.730 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:29.730+0000 7f995bb3b640 1 --2- 172.21.15.29:0/3851356949 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f99541012f0 0x7f995410d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:54:29.730 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:29.730+0000 7f995bb3b640 1 --2- 172.21.15.29:0/3851356949 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9954100930 0x7f9954100db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:54:29.730 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:29.730+0000 7f995bb3b640 1 --2- 172.21.15.29:0/3851356949 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9954105680 0x7f9954105a60 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:54:29.730 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:29.730+0000 7f995bb3b640 1 -- 172.21.15.29:0/3851356949 >> 172.21.15.29:0/3851356949 conn(0x7f9954077530 msgr2=0x7f9954077940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:54:29.730 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:29.730+0000 7f995bb3b640 1 -- 172.21.15.29:0/3851356949 shutdown_connections 2024-03-16T11:54:29.730 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:29.730+0000 7f995bb3b640 1 -- 172.21.15.29:0/3851356949 wait complete. 2024-03-16T11:54:29.730 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:29.730+0000 7f995bb3b640 1 Processor -- start 2024-03-16T11:54:29.730 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:29.730+0000 7f995bb3b640 1 -- start start 2024-03-16T11:54:29.732 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:29.731+0000 7f995bb3b640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9954100930 0x7f9954198100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:54:29.732 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:29.731+0000 7f995bb3b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f99541012f0 0x7f9954198640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:54:29.732 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:29.731+0000 7f995bb3b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9954105680 0x7f995419c970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:54:29.732 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:29.731+0000 7f995bb3b640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f9954110b60 con 0x7f9954105680 2024-03-16T11:54:29.732 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:29.731+0000 7f995bb3b640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f99541109e0 con 0x7f9954100930 2024-03-16T11:54:29.732 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:29.731+0000 7f995bb3b640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f9954110ce0 con 0x7f99541012f0 2024-03-16T11:54:29.732 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:29.731+0000 7f995a0d8640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9954105680 0x7f995419c970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:54:29.732 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:29.731+0000 7f99590d6640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f99541012f0 0x7f9954198640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:54:29.732 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:29.731+0000 7f995a0d8640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9954105680 0x7f995419c970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53466/0 (socket says 172.21.15.29:53466) 2024-03-16T11:54:29.732 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:29.731+0000 7f995a0d8640 1 -- 172.21.15.29:0/2823519807 learned_addr learned my addr 172.21.15.29:0/2823519807 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:54:29.733 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:29.731+0000 7f99598d7640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9954100930 0x7f9954198100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:54:29.733 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:29.731+0000 7f99590d6640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f99541012f0 0x7f9954198640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.29:55112/0 (socket says 172.21.15.29:55112) 2024-03-16T11:54:29.733 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:29.731+0000 7f995a0d8640 1 -- 172.21.15.29:0/2823519807 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f99541012f0 msgr2=0x7f9954198640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:54:29.733 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:29.731+0000 7f995a0d8640 1 --2- 172.21.15.29:0/2823519807 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f99541012f0 0x7f9954198640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:54:29.733 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:29.731+0000 7f995a0d8640 1 -- 172.21.15.29:0/2823519807 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9954100930 msgr2=0x7f9954198100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:54:29.733 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:29.731+0000 7f995a0d8640 1 --2- 172.21.15.29:0/2823519807 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9954100930 0x7f9954198100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:54:29.733 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:29.731+0000 7f995a0d8640 1 -- 172.21.15.29:0/2823519807 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f995419d0a0 con 0x7f9954105680 2024-03-16T11:54:29.733 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:29.731+0000 7f99590d6640 1 --2- 172.21.15.29:0/2823519807 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f99541012f0 0x7f9954198640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:54:29.733 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:29.731+0000 7f99598d7640 1 --2- 172.21.15.29:0/2823519807 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9954100930 0x7f9954198100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T11:54:29.733 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:29.731+0000 7f995a0d8640 1 --2- 172.21.15.29:0/2823519807 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9954105680 0x7f995419c970 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f994c00ef00 tx=0x7f994c00c740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:54:29.733 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:29.732+0000 7f994affd640 1 -- 172.21.15.29:0/2823519807 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f994c003970 con 0x7f9954105680 2024-03-16T11:54:29.733 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:29.732+0000 7f995bb3b640 1 -- 172.21.15.29:0/2823519807 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f995419d390 con 0x7f9954105680 2024-03-16T11:54:29.733 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:29.732+0000 7f994affd640 1 -- 172.21.15.29:0/2823519807 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f994c004020 con 0x7f9954105680 2024-03-16T11:54:29.733 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:29.732+0000 7f994affd640 1 -- 172.21.15.29:0/2823519807 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f994c00b910 con 0x7f9954105680 2024-03-16T11:54:29.734 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:29.732+0000 7f995bb3b640 1 -- 172.21.15.29:0/2823519807 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f99541a4c20 con 0x7f9954105680 2024-03-16T11:54:29.734 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:29.735+0000 7f994affd640 1 -- 172.21.15.29:0/2823519807 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7f994c01d020 con 0x7f9954105680 2024-03-16T11:54:29.735 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:29.736+0000 7f995e80a740 1 Processor -- start 2024-03-16T11:54:29.740 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:29.736+0000 7f995e80a740 1 -- start start 2024-03-16T11:54:29.740 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:29.736+0000 7f994affd640 1 --2- 172.21.15.29:0/2823519807 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f99400a2a10 0x7f99400a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:54:29.740 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:29.736+0000 7f995e80a740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558b661fb130 0x558b661fb510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:54:29.741 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:29.737+0000 7f995e80a740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558b661fba50 0x558b662042e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:54:29.741 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:29.737+0000 7f995e80a740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558b66204820 0x558b66206c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:54:29.741 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:29.737+0000 7f995e80a740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x558b6615fa40 con 0x558b661fb130 2024-03-16T11:54:29.741 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:29.737+0000 7f995e80a740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x558b66108920 con 0x558b661fba50 2024-03-16T11:54:29.741 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:29.737+0000 7f995e80a740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x558b6615bae0 con 0x558b66204820 2024-03-16T11:54:29.741 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:29.737+0000 7f99590d6640 1 --2- 172.21.15.29:0/2823519807 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f99400a2a10 0x7f99400a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:54:29.741 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:29.737+0000 7f99598d7640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558b661fb130 0x558b661fb510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:54:29.741 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:29.737+0000 7f994affd640 1 -- 172.21.15.29:0/2823519807 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7f994c0cee80 con 0x7f9954105680 2024-03-16T11:54:29.741 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:29.737+0000 7f99598d7640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558b661fb130 0x558b661fb510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53470/0 (socket says 172.21.15.29:53470) 2024-03-16T11:54:29.741 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:29.737+0000 7f99598d7640 1 -- 172.21.15.29:0/2958406241 learned_addr learned my addr 172.21.15.29:0/2958406241 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:54:29.741 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:29.737+0000 7f995a0d8640 1 --2- 172.21.15.29:0/2958406241 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558b66204820 0x558b66206c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:54:29.741 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:29.737+0000 7f99590d6640 1 --2- 172.21.15.29:0/2958406241 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558b661fba50 0x558b662042e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:54:29.742 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:29.737+0000 7f99598d7640 1 -- 172.21.15.29:0/2958406241 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558b66204820 msgr2=0x558b66206c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:54:29.742 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:29.737+0000 7f99598d7640 1 --2- 172.21.15.29:0/2958406241 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558b66204820 0x558b66206c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:54:29.742 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:29.737+0000 7f99598d7640 1 -- 172.21.15.29:0/2958406241 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558b661fba50 msgr2=0x558b662042e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:54:29.742 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:29.737+0000 7f99598d7640 1 --2- 172.21.15.29:0/2958406241 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558b661fba50 0x558b662042e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:54:29.742 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:29.737+0000 7f99598d7640 1 -- 172.21.15.29:0/2958406241 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558b66120460 con 0x558b661fb130 2024-03-16T11:54:29.742 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:29.737+0000 7f99590d6640 1 --2- 172.21.15.29:0/2823519807 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f99400a2a10 0x7f99400a4ed0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x558b661fabe0 tx=0x7f9944009c30 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:54:29.742 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:29.737+0000 7f99598d7640 1 --2- 172.21.15.29:0/2958406241 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558b661fb130 0x558b661fb510 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f9950036000 tx=0x7f9950032340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f2e23febfc20097e server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:54:29.742 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:29.737+0000 7f993ffff640 1 -- 172.21.15.29:0/2958406241 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9950039040 con 0x558b661fb130 2024-03-16T11:54:29.742 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:29.737+0000 7f993ffff640 1 -- 172.21.15.29:0/2958406241 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9950034cb0 con 0x558b661fb130 2024-03-16T11:54:29.742 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:29.737+0000 7f993ffff640 1 -- 172.21.15.29:0/2958406241 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9950005280 con 0x558b661fb130 2024-03-16T11:54:29.742 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:29.738+0000 7f995e80a740 1 -- 172.21.15.29:0/2958406241 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558b661fb130 msgr2=0x558b661fb510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:54:29.743 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:29.738+0000 7f995e80a740 1 --2- 172.21.15.29:0/2958406241 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558b661fb130 0x558b661fb510 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f9950036000 tx=0x7f9950032340 comp rx=0 tx=0).stop 2024-03-16T11:54:29.743 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:29.738+0000 7f995e80a740 1 -- 172.21.15.29:0/2958406241 shutdown_connections 2024-03-16T11:54:29.743 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:29.738+0000 7f995e80a740 1 --2- 172.21.15.29:0/2958406241 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558b66204820 0x558b66206c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:54:29.743 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:29.738+0000 7f995e80a740 1 --2- 172.21.15.29:0/2958406241 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558b661fba50 0x558b662042e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:54:29.743 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:29.738+0000 7f995e80a740 1 --2- 172.21.15.29:0/2958406241 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558b661fb130 0x558b661fb510 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:54:29.743 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:29.738+0000 7f995e80a740 1 -- 172.21.15.29:0/2958406241 >> 172.21.15.29:0/2958406241 conn(0x558b661f5a50 msgr2=0x558b661f5e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:54:29.743 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:29.738+0000 7f995e80a740 1 -- 172.21.15.29:0/2958406241 shutdown_connections 2024-03-16T11:54:29.743 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:29.738+0000 7f995e80a740 1 -- 172.21.15.29:0/2958406241 wait complete. 2024-03-16T11:54:29.743 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:29.739+0000 7f995e80a740 1 -- start start 2024-03-16T11:54:29.743 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:29.739+0000 7f995e80a740 1 Processor -- start 2024-03-16T11:54:29.743 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:29.739+0000 7f995e80a740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558b661fb130 0x558b662017e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:54:29.743 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:29.739+0000 7f995e80a740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558b661fba50 0x558b66201d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:54:29.744 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:29.739+0000 7f995e80a740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558b66204820 0x558b66202260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:54:29.744 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:29.739+0000 7f995e80a740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x558b6620b800 con 0x558b661fba50 2024-03-16T11:54:29.744 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:29.739+0000 7f995e80a740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x558b6611bd70 con 0x558b661fb130 2024-03-16T11:54:29.744 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:29.739+0000 7f995e80a740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x558b6620b980 con 0x558b66204820 2024-03-16T11:54:29.744 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:29.739+0000 7f995e80a740 10 client.0 _ll_register_callbacks cb 0x558b65e73a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T11:54:29.744 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:29.739+0000 7f99590d6640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558b661fba50 0x558b66201d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:54:29.744 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:29.739+0000 7f99590d6640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558b661fba50 0x558b66201d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53486/0 (socket says 172.21.15.29:53486) 2024-03-16T11:54:29.744 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:29.739+0000 7f99590d6640 1 -- 172.21.15.29:0/1915943751 learned_addr learned my addr 172.21.15.29:0/1915943751 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:54:29.744 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:29.739+0000 7f995a0d8640 1 --2- 172.21.15.29:0/1915943751 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558b66204820 0x558b66202260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:54:29.744 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:29.739+0000 7f99598d7640 1 --2- 172.21.15.29:0/1915943751 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558b661fb130 0x558b662017e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:54:29.744 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:29.740+0000 7f99590d6640 1 -- 172.21.15.29:0/1915943751 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558b66204820 msgr2=0x558b66202260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:54:29.744 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:29.740+0000 7f99590d6640 1 --2- 172.21.15.29:0/1915943751 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558b66204820 0x558b66202260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:54:29.745 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:29.740+0000 7f99590d6640 1 -- 172.21.15.29:0/1915943751 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558b661fb130 msgr2=0x558b662017e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:54:29.745 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:29.740+0000 7f99590d6640 1 --2- 172.21.15.29:0/1915943751 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558b661fb130 0x558b662017e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:54:29.745 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:29.740+0000 7f99590d6640 1 -- 172.21.15.29:0/1915943751 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558b65e74210 con 0x558b661fba50 2024-03-16T11:54:29.745 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:29.740+0000 7f99590d6640 1 --2- 172.21.15.29:0/1915943751 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558b661fba50 0x558b66201d20 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f9944004290 tx=0x7f99440046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=950304e29e61b55d server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:54:29.745 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:29.740+0000 7f993d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-16T11:54:29.745 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:29.740+0000 7f993d7fa640 1 -- 172.21.15.29:0/1915943751 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9944026020 con 0x558b661fba50 2024-03-16T11:54:29.745 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:29.740+0000 7f993d7fa640 1 -- 172.21.15.29:0/1915943751 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7f9944003d70 con 0x558b661fba50 2024-03-16T11:54:29.745 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:29.740+0000 7f995e80a740 1 -- 172.21.15.29:0/1915943751 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9950034cb0 con 0x558b661fba50 2024-03-16T11:54:29.746 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:29.740+0000 7f993d7fa640 1 -- 172.21.15.29:0/1915943751 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f9944031050 con 0x558b661fba50 2024-03-16T11:54:29.746 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:29.740+0000 7f995e80a740 10 client.15702 fetch_fsmap learned FSMap version 24 2024-03-16T11:54:29.746 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:29.740+0000 7f995e80a740 1 -- 172.21.15.29:0/1915943751 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x558b65f40620 con 0x558b661fba50 2024-03-16T11:54:29.746 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:29.740+0000 7f993d7fa640 1 -- 172.21.15.29:0/1915943751 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9950034cb0 con 0x558b661fba50 2024-03-16T11:54:29.746 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:29.740+0000 7f993d7fa640 1 -- 172.21.15.29:0/1915943751 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7f9944010430 con 0x558b661fba50 2024-03-16T11:54:29.746 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:29.740+0000 7f995e80a740 10 client.15702 fetch_fsmap finished waiting for FSMap version 24 2024-03-16T11:54:29.746 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:29.740+0000 7f995e80a740 10 client.15702 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T11:54:29.746 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:29.741+0000 7f995e80a740 20 client.15702 populate_metadata read hostname 'smithi029' 2024-03-16T11:54:29.746 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:29.741+0000 7f995e80a740 1 --2- 172.21.15.29:0/1915943751 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x558b66218d70 0x558b6621b190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:54:29.746 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:29.741+0000 7f995e80a740 4 client.15702 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T11:54:29.746 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:29.741+0000 7f995e80a740 1 -- 172.21.15.29:0/1915943751 --> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558b6616cf80 con 0x558b66218d70 2024-03-16T11:54:29.747 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:29.741+0000 7f99598d7640 1 --2- 172.21.15.29:0/1915943751 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x558b66218d70 0x558b6621b190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:54:29.747 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:29.741+0000 7f99598d7640 1 --2- 172.21.15.29:0/1915943751 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x558b66218d70 0x558b6621b190 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a9a50dcb7fc7da87 server_cookie=7f50a68712894482 in_seq=0 out_seq=0 2024-03-16T11:54:29.747 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:29.741+0000 7f993d7fa640 10 client.15702 ms_handle_connect on v2:172.21.15.152:6838/1548731833 2024-03-16T11:54:29.748 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:29.742+0000 7f993d7fa640 1 -- 172.21.15.29:0/1915943751 <== mds.0 v2:172.21.15.152:6838/1548731833 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x558b6616cf80 con 0x558b66218d70 2024-03-16T11:54:29.748 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:29.742+0000 7f993d7fa640 10 client.15702 handle_command_reply: tid=0 2024-03-16T11:54:29.748 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:29.742+0000 7f995e80a740 1 client.15702 shutdown 2024-03-16T11:54:29.748 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:29.743+0000 7f995e80a740 1 -- 172.21.15.29:0/1915943751 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558b661fba50 msgr2=0x558b66201d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:54:29.748 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:29.743+0000 7f995e80a740 1 --2- 172.21.15.29:0/1915943751 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558b661fba50 0x558b66201d20 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f9944004290 tx=0x7f99440046c0 comp rx=0 tx=0).stop 2024-03-16T11:54:29.748 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:29.743+0000 7f995e80a740 1 -- 172.21.15.29:0/1915943751 shutdown_connections 2024-03-16T11:54:29.748 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:29.743+0000 7f995e80a740 1 --2- 172.21.15.29:0/1915943751 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558b66204820 0x558b66202260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:54:29.748 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:29.743+0000 7f995e80a740 1 --2- 172.21.15.29:0/1915943751 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558b661fba50 0x558b66201d20 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:54:29.748 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:29.743+0000 7f995e80a740 1 --2- 172.21.15.29:0/1915943751 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x558b66218d70 0x558b6621b190 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:54:29.748 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:29.743+0000 7f995e80a740 1 --2- 172.21.15.29:0/1915943751 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558b661fb130 0x558b662017e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:54:29.749 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:29.743+0000 7f995e80a740 1 -- 172.21.15.29:0/1915943751 >> 172.21.15.29:0/1915943751 conn(0x558b661f5a50 msgr2=0x558b661f77a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:54:29.749 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:29.743+0000 7f993d7fa640 0 client.15702 ms_handle_reset on v2:172.21.15.152:6838/1548731833 2024-03-16T11:54:29.749 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:29.743+0000 7f995e80a740 1 -- 172.21.15.29:0/1915943751 shutdown_connections 2024-03-16T11:54:29.749 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:29.743+0000 7f995e80a740 1 -- 172.21.15.29:0/1915943751 wait complete. 2024-03-16T11:54:29.749 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:29.743+0000 7f995e80a740 20 client.15702 trim_cache size 0 max 16384 2024-03-16T11:54:29.760 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:29.759+0000 7f995e80a740 1 Processor -- start 2024-03-16T11:54:29.760 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:29.759+0000 7f995e80a740 1 -- start start 2024-03-16T11:54:29.760 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:29.759+0000 7f995e80a740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558b661fb130 0x558b662145a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:54:29.760 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:29.759+0000 7f995e80a740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558b661fba50 0x558b66220aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:54:29.760 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:29.759+0000 7f995e80a740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558b66204820 0x558b66210bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:54:29.761 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:29.759+0000 7f995e80a740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f99541ada30 con 0x558b661fb130 2024-03-16T11:54:29.761 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:29.759+0000 7f995e80a740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f99541ab300 con 0x558b661fba50 2024-03-16T11:54:29.761 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:29.759+0000 7f995e80a740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f99541a9150 con 0x558b66204820 2024-03-16T11:54:29.761 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:29.759+0000 7f99598d7640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558b661fb130 0x558b662145a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:54:29.761 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:29.759+0000 7f99598d7640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558b661fb130 0x558b662145a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53498/0 (socket says 172.21.15.29:53498) 2024-03-16T11:54:29.761 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:29.759+0000 7f99598d7640 1 -- 172.21.15.29:0/3181427492 learned_addr learned my addr 172.21.15.29:0/3181427492 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:54:29.761 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:29.759+0000 7f99590d6640 1 --2- 172.21.15.29:0/3181427492 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558b661fba50 0x558b66220aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:54:29.761 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:29.759+0000 7f995a0d8640 1 --2- 172.21.15.29:0/3181427492 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558b66204820 0x558b66210bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:54:29.761 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:29.760+0000 7f99598d7640 1 -- 172.21.15.29:0/3181427492 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558b66204820 msgr2=0x558b66210bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:54:29.761 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:29.760+0000 7f99598d7640 1 --2- 172.21.15.29:0/3181427492 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558b66204820 0x558b66210bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:54:29.761 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:29.760+0000 7f99598d7640 1 -- 172.21.15.29:0/3181427492 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558b661fba50 msgr2=0x558b66220aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:54:29.761 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:29.760+0000 7f99598d7640 1 --2- 172.21.15.29:0/3181427492 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558b661fba50 0x558b66220aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:54:29.761 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:29.760+0000 7f99598d7640 1 -- 172.21.15.29:0/3181427492 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558b661f1310 con 0x558b661fb130 2024-03-16T11:54:29.761 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:29.760+0000 7f99598d7640 1 --2- 172.21.15.29:0/3181427492 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558b661fb130 0x558b662145a0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f9950004650 tx=0x7f9950005dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a207f32d67bf0566 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:54:29.762 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:29.760+0000 7f993f7fe640 1 -- 172.21.15.29:0/3181427492 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9950039040 con 0x558b661fb130 2024-03-16T11:54:29.762 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:29.760+0000 7f993f7fe640 1 -- 172.21.15.29:0/3181427492 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9950050d10 con 0x558b661fb130 2024-03-16T11:54:29.762 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:29.760+0000 7f993f7fe640 1 -- 172.21.15.29:0/3181427492 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9950005290 con 0x558b661fb130 2024-03-16T11:54:29.762 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:29.761+0000 7f995e80a740 1 -- 172.21.15.29:0/3181427492 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558b661fb130 msgr2=0x558b662145a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:54:29.762 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:29.761+0000 7f995e80a740 1 --2- 172.21.15.29:0/3181427492 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558b661fb130 0x558b662145a0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f9950004650 tx=0x7f9950005dc0 comp rx=0 tx=0).stop 2024-03-16T11:54:29.762 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:29.761+0000 7f995e80a740 1 -- 172.21.15.29:0/3181427492 shutdown_connections 2024-03-16T11:54:29.762 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:29.761+0000 7f995e80a740 1 --2- 172.21.15.29:0/3181427492 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558b66204820 0x558b66210bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:54:29.762 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:29.761+0000 7f995e80a740 1 --2- 172.21.15.29:0/3181427492 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558b661fba50 0x558b66220aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:54:29.762 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:29.761+0000 7f995e80a740 1 --2- 172.21.15.29:0/3181427492 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558b661fb130 0x558b662145a0 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:54:29.762 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:29.761+0000 7f995e80a740 1 -- 172.21.15.29:0/3181427492 >> 172.21.15.29:0/3181427492 conn(0x558b661f5a50 msgr2=0x558b662017a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:54:29.762 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:29.761+0000 7f995e80a740 1 -- 172.21.15.29:0/3181427492 shutdown_connections 2024-03-16T11:54:29.762 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:29.761+0000 7f995e80a740 1 -- 172.21.15.29:0/3181427492 wait complete. 2024-03-16T11:54:29.762 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:29.761+0000 7f995e80a740 1 -- start start 2024-03-16T11:54:29.763 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:29.761+0000 7f995e80a740 1 Processor -- start 2024-03-16T11:54:29.765 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:29.762+0000 7f995e80a740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558b661fb130 0x558b66210bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:54:29.766 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:29.762+0000 7f995e80a740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558b661fba50 0x558b6620dba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:54:29.766 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:29.762+0000 7f995e80a740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558b66204820 0x558b6620e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:54:29.766 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:29.762+0000 7f995e80a740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f99541a97c0 con 0x558b661fba50 2024-03-16T11:54:29.766 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:29.762+0000 7f995e80a740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f99541aac30 con 0x558b661fb130 2024-03-16T11:54:29.766 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:29.762+0000 7f995e80a740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f99541a9e90 con 0x558b66204820 2024-03-16T11:54:29.766 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:29.762+0000 7f995e80a740 10 client.0 _ll_register_callbacks cb 0x7f9944009e20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T11:54:29.766 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:29.762+0000 7f99590d6640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558b661fba50 0x558b6620dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:54:29.766 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:29.762+0000 7f99590d6640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558b661fba50 0x558b6620dba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53514/0 (socket says 172.21.15.29:53514) 2024-03-16T11:54:29.766 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:29.762+0000 7f99590d6640 1 -- 172.21.15.29:0/1715230985 learned_addr learned my addr 172.21.15.29:0/1715230985 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:54:29.766 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:29.762+0000 7f99598d7640 1 --2- 172.21.15.29:0/1715230985 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558b661fb130 0x558b66210bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:54:29.766 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:29.762+0000 7f995a0d8640 1 --2- 172.21.15.29:0/1715230985 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558b66204820 0x558b6620e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:54:29.766 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:29.762+0000 7f99590d6640 1 -- 172.21.15.29:0/1715230985 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558b66204820 msgr2=0x558b6620e0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:54:29.767 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:29.762+0000 7f99590d6640 1 --2- 172.21.15.29:0/1715230985 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558b66204820 0x558b6620e0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:54:29.767 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:29.762+0000 7f99590d6640 1 -- 172.21.15.29:0/1715230985 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558b661fb130 msgr2=0x558b66210bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:54:29.767 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:29.762+0000 7f99590d6640 1 --2- 172.21.15.29:0/1715230985 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558b661fb130 0x558b66210bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:54:29.767 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:29.762+0000 7f99590d6640 1 -- 172.21.15.29:0/1715230985 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558b66211240 con 0x558b661fba50 2024-03-16T11:54:29.767 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:29.762+0000 7f99598d7640 1 --2- 172.21.15.29:0/1715230985 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558b661fb130 0x558b66210bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:54:29.767 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:29.762+0000 7f99590d6640 1 --2- 172.21.15.29:0/1715230985 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558b661fba50 0x558b6620dba0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f9944004260 tx=0x7f9944013780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f35fb398d2950101 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:54:29.767 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:29.762+0000 7f993d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-16T11:54:29.767 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:29.762+0000 7f993d7fa640 1 -- 172.21.15.29:0/1715230985 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9944026050 con 0x558b661fba50 2024-03-16T11:54:29.767 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:29.763+0000 7f993d7fa640 1 -- 172.21.15.29:0/1715230985 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7f9944010ae0 con 0x558b661fba50 2024-03-16T11:54:29.767 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:29.763+0000 7f995e80a740 1 -- 172.21.15.29:0/1715230985 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9950050d10 con 0x558b661fba50 2024-03-16T11:54:29.767 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:29.763+0000 7f993d7fa640 1 -- 172.21.15.29:0/1715230985 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f9944010ce0 con 0x558b661fba50 2024-03-16T11:54:29.767 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:29.763+0000 7f995e80a740 10 client.15708 fetch_fsmap learned FSMap version 24 2024-03-16T11:54:29.767 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:29.763+0000 7f995e80a740 1 -- 172.21.15.29:0/1715230985 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x558b6627fb10 con 0x558b661fba50 2024-03-16T11:54:29.768 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:29.763+0000 7f993d7fa640 1 -- 172.21.15.29:0/1715230985 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9950050d10 con 0x558b661fba50 2024-03-16T11:54:29.768 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:29.763+0000 7f993d7fa640 1 -- 172.21.15.29:0/1715230985 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7f9944022480 con 0x558b661fba50 2024-03-16T11:54:29.768 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:29.763+0000 7f995e80a740 10 client.15708 fetch_fsmap finished waiting for FSMap version 24 2024-03-16T11:54:29.768 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:29.763+0000 7f995e80a740 10 client.15708 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T11:54:29.768 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:29.763+0000 7f995e80a740 20 client.15708 populate_metadata read hostname 'smithi029' 2024-03-16T11:54:29.768 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:29.763+0000 7f995e80a740 1 --2- 172.21.15.29:0/1715230985 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x558b66280290 0x558b662826b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:54:29.768 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:29.763+0000 7f995e80a740 4 client.15708 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-16T11:54:29.768 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:29.763+0000 7f995e80a740 1 -- 172.21.15.29:0/1715230985 --> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] -- command(tid 0: {"prefix": "flush journal"}) -- 0x558b661672d0 con 0x558b66280290 2024-03-16T11:54:29.768 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:29.763+0000 7f99598d7640 1 --2- 172.21.15.29:0/1715230985 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x558b66280290 0x558b662826b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:54:29.768 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:29.764+0000 7f993d7fa640 10 client.15708 ms_handle_connect on v2:172.21.15.152:6838/1548731833 2024-03-16T11:54:29.768 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:29.764+0000 7f99598d7640 1 --2- 172.21.15.29:0/1715230985 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x558b66280290 0x558b662826b0 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8dee22c862a50496 server_cookie=81b111c3194900f in_seq=0 out_seq=0 2024-03-16T11:54:29.780 INFO:tasks.workunit.client.0.smithi029.stdout: 1 301827 11.77 MB/sec execute 14 sec latency 59.988 ms 2024-03-16T11:54:29.926 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:29.926+0000 7f993d7fa640 1 -- 172.21.15.29:0/1715230985 <== mds.0 v2:172.21.15.152:6838/1548731833 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x558b661672d0 con 0x558b66280290 2024-03-16T11:54:29.926 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:29.926+0000 7f993d7fa640 10 client.15708 handle_command_reply: tid=0 2024-03-16T11:54:29.926 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:29.926+0000 7f995e80a740 1 client.15708 shutdown 2024-03-16T11:54:29.926 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:29.926+0000 7f995e80a740 1 -- 172.21.15.29:0/1715230985 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558b661fba50 msgr2=0x558b6620dba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:54:29.926 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:29.926+0000 7f995e80a740 1 --2- 172.21.15.29:0/1715230985 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558b661fba50 0x558b6620dba0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f9944004260 tx=0x7f9944013780 comp rx=0 tx=0).stop 2024-03-16T11:54:29.926 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:29.926+0000 7f995e80a740 1 -- 172.21.15.29:0/1715230985 shutdown_connections 2024-03-16T11:54:29.926 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:29.926+0000 7f995e80a740 1 --2- 172.21.15.29:0/1715230985 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558b66204820 0x558b6620e0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:54:29.926 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:29.926+0000 7f995e80a740 1 --2- 172.21.15.29:0/1715230985 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558b661fba50 0x558b6620dba0 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:54:29.927 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:29.926+0000 7f995e80a740 1 --2- 172.21.15.29:0/1715230985 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x558b66280290 0x558b662826b0 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:54:29.927 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:29.926+0000 7f995e80a740 1 --2- 172.21.15.29:0/1715230985 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558b661fb130 0x558b66210bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:54:29.927 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:29.926+0000 7f993d7fa640 0 client.15708 ms_handle_reset on v2:172.21.15.152:6838/1548731833 2024-03-16T11:54:29.927 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:29.926+0000 7f995e80a740 1 -- 172.21.15.29:0/1715230985 >> 172.21.15.29:0/1715230985 conn(0x558b661f5a50 msgr2=0x558b662017a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:54:29.927 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:29.927+0000 7f995e80a740 1 -- 172.21.15.29:0/1715230985 shutdown_connections 2024-03-16T11:54:29.927 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:29.927+0000 7f995e80a740 1 -- 172.21.15.29:0/1715230985 wait complete. 2024-03-16T11:54:29.927 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:29.927+0000 7f995e80a740 20 client.15708 trim_cache size 0 max 16384 2024-03-16T11:54:29.927 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:{ 2024-03-16T11:54:29.927 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "message": "", 2024-03-16T11:54:29.927 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "return_code": 0 2024-03-16T11:54:29.927 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:} 2024-03-16T11:54:29.928 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:29.927+0000 7f995bb3b640 1 -- 172.21.15.29:0/2823519807 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f99400a2a10 msgr2=0x7f99400a4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:54:29.928 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:29.927+0000 7f995bb3b640 1 --2- 172.21.15.29:0/2823519807 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f99400a2a10 0x7f99400a4ed0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x558b661fabe0 tx=0x7f9944009c30 comp rx=0 tx=0).stop 2024-03-16T11:54:29.928 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:29.927+0000 7f995bb3b640 1 -- 172.21.15.29:0/2823519807 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9954105680 msgr2=0x7f995419c970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:54:29.929 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:29.927+0000 7f995bb3b640 1 --2- 172.21.15.29:0/2823519807 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9954105680 0x7f995419c970 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f994c00ef00 tx=0x7f994c00c740 comp rx=0 tx=0).stop 2024-03-16T11:54:29.929 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:29.927+0000 7f995bb3b640 1 -- 172.21.15.29:0/2823519807 shutdown_connections 2024-03-16T11:54:29.929 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:29.927+0000 7f995bb3b640 1 --2- 172.21.15.29:0/2823519807 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f99400a2a10 0x7f99400a4ed0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:54:29.929 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:29.927+0000 7f995bb3b640 1 --2- 172.21.15.29:0/2823519807 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9954105680 0x7f995419c970 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:54:29.929 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:29.927+0000 7f995bb3b640 1 --2- 172.21.15.29:0/2823519807 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f99541012f0 0x7f9954198640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:54:29.929 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:29.927+0000 7f995bb3b640 1 --2- 172.21.15.29:0/2823519807 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9954100930 0x7f9954198100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:54:29.929 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:29.927+0000 7f995bb3b640 1 -- 172.21.15.29:0/2823519807 >> 172.21.15.29:0/2823519807 conn(0x7f9954077530 msgr2=0x7f99541045d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:54:29.929 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:29.927+0000 7f995bb3b640 1 -- 172.21.15.29:0/2823519807 shutdown_connections 2024-03-16T11:54:29.929 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:29.927+0000 7f995bb3b640 1 -- 172.21.15.29:0/2823519807 wait complete. 2024-03-16T11:54:30.780 INFO:tasks.workunit.client.0.smithi029.stdout: 1 303722 11.40 MB/sec execute 15 sec latency 64.148 ms 2024-03-16T11:54:31.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:54:30 smithi154 ceph-mon[36320]: pgmap v111: 97 pgs: 97 active+clean; 49 MiB data, 865 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 2.9 MiB/s wr, 318 op/s 2024-03-16T11:54:31.117 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:54:30 smithi029 ceph-mon[31975]: pgmap v111: 97 pgs: 97 active+clean; 49 MiB data, 865 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 2.9 MiB/s wr, 318 op/s 2024-03-16T11:54:31.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:54:30 smithi152 ceph-mon[36291]: pgmap v111: 97 pgs: 97 active+clean; 49 MiB data, 865 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 2.9 MiB/s wr, 318 op/s 2024-03-16T11:54:31.780 INFO:tasks.workunit.client.0.smithi029.stdout: 1 305668 11.52 MB/sec execute 16 sec latency 46.538 ms 2024-03-16T11:54:32.780 INFO:tasks.workunit.client.0.smithi029.stdout: 1 308562 11.86 MB/sec execute 17 sec latency 13.442 ms 2024-03-16T11:54:33.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:54:32 smithi154 ceph-mon[36320]: pgmap v112: 97 pgs: 97 active+clean; 42 MiB data, 928 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 4.3 MiB/s wr, 515 op/s 2024-03-16T11:54:33.089 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:54:32 smithi029 ceph-mon[31975]: pgmap v112: 97 pgs: 97 active+clean; 42 MiB data, 928 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 4.3 MiB/s wr, 515 op/s 2024-03-16T11:54:33.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:54:32 smithi152 ceph-mon[36291]: pgmap v112: 97 pgs: 97 active+clean; 42 MiB data, 928 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 4.3 MiB/s wr, 515 op/s 2024-03-16T11:54:33.780 INFO:tasks.workunit.client.0.smithi029.stdout: 1 310048 11.58 MB/sec execute 18 sec latency 60.519 ms 2024-03-16T11:54:34.781 INFO:tasks.workunit.client.0.smithi029.stdout: 1 312399 11.69 MB/sec execute 19 sec latency 66.233 ms 2024-03-16T11:54:35.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:54:34 smithi154 ceph-mon[36320]: pgmap v113: 97 pgs: 97 active+clean; 42 MiB data, 928 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 2.8 MiB/s wr, 300 op/s 2024-03-16T11:54:35.087 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:54:34 smithi029 ceph-mon[31975]: pgmap v113: 97 pgs: 97 active+clean; 42 MiB data, 928 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 2.8 MiB/s wr, 300 op/s 2024-03-16T11:54:35.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:54:34 smithi152 ceph-mon[36291]: pgmap v113: 97 pgs: 97 active+clean; 42 MiB data, 928 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 2.8 MiB/s wr, 300 op/s 2024-03-16T11:54:35.780 INFO:tasks.workunit.client.0.smithi029.stdout: 1 314447 11.46 MB/sec execute 20 sec latency 36.893 ms 2024-03-16T11:54:36.781 INFO:tasks.workunit.client.0.smithi029.stdout: 1 316281 11.51 MB/sec execute 21 sec latency 19.477 ms 2024-03-16T11:54:37.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:54:36 smithi154 ceph-mon[36320]: pgmap v114: 97 pgs: 97 active+clean; 42 MiB data, 928 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 2.8 MiB/s wr, 300 op/s 2024-03-16T11:54:37.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:54:36 smithi029 ceph-mon[31975]: pgmap v114: 97 pgs: 97 active+clean; 42 MiB data, 928 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 2.8 MiB/s wr, 300 op/s 2024-03-16T11:54:37.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:54:36 smithi152 ceph-mon[36291]: pgmap v114: 97 pgs: 97 active+clean; 42 MiB data, 928 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 2.8 MiB/s wr, 300 op/s 2024-03-16T11:54:37.781 INFO:tasks.workunit.client.0.smithi029.stdout: 1 319061 11.64 MB/sec execute 22 sec latency 26.588 ms 2024-03-16T11:54:38.781 INFO:tasks.workunit.client.0.smithi029.stdout: 1 320699 11.56 MB/sec execute 23 sec latency 18.371 ms 2024-03-16T11:54:39.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:54:38 smithi154 ceph-mon[36320]: pgmap v115: 97 pgs: 97 active+clean; 47 MiB data, 915 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 4.2 MiB/s wr, 402 op/s 2024-03-16T11:54:39.087 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:54:38 smithi029 ceph-mon[31975]: pgmap v115: 97 pgs: 97 active+clean; 47 MiB data, 915 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 4.2 MiB/s wr, 402 op/s 2024-03-16T11:54:39.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:54:38 smithi152 ceph-mon[36291]: pgmap v115: 97 pgs: 97 active+clean; 47 MiB data, 915 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 4.2 MiB/s wr, 402 op/s 2024-03-16T11:54:39.781 INFO:tasks.workunit.client.0.smithi029.stdout: 1 322902 11.60 MB/sec execute 24 sec latency 153.247 ms 2024-03-16T11:54:40.781 INFO:tasks.workunit.client.0.smithi029.stdout: 1 325094 11.47 MB/sec execute 25 sec latency 17.535 ms 2024-03-16T11:54:41.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:54:40 smithi154 ceph-mon[36320]: pgmap v116: 97 pgs: 97 active+clean; 47 MiB data, 915 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 2.7 MiB/s wr, 298 op/s 2024-03-16T11:54:41.087 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:54:40 smithi029 ceph-mon[31975]: pgmap v116: 97 pgs: 97 active+clean; 47 MiB data, 915 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 2.7 MiB/s wr, 298 op/s 2024-03-16T11:54:41.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:54:40 smithi152 ceph-mon[36291]: pgmap v116: 97 pgs: 97 active+clean; 47 MiB data, 915 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 2.7 MiB/s wr, 298 op/s 2024-03-16T11:54:41.781 INFO:tasks.workunit.client.0.smithi029.stdout: 1 327004 11.50 MB/sec execute 26 sec latency 19.001 ms 2024-03-16T11:54:42.781 INFO:tasks.workunit.client.0.smithi029.stdout: 1 329956 11.73 MB/sec execute 27 sec latency 25.747 ms 2024-03-16T11:54:43.017 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.015+0000 7ff0ec9b3640 1 -- 172.21.15.29:0/3443657906 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff0e8104380 msgr2=0x7ff0e8110890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:54:43.017 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.015+0000 7ff0ec9b3640 1 --2- 172.21.15.29:0/3443657906 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff0e8104380 0x7ff0e8110890 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7ff0e0009cd0 tx=0x7ff0e002c0f0 comp rx=0 tx=0).stop 2024-03-16T11:54:43.017 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.015+0000 7ff0ec9b3640 1 -- 172.21.15.29:0/3443657906 shutdown_connections 2024-03-16T11:54:43.018 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.015+0000 7ff0ec9b3640 1 --2- 172.21.15.29:0/3443657906 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff0e8104380 0x7ff0e8110890 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:54:43.018 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.015+0000 7ff0ec9b3640 1 --2- 172.21.15.29:0/3443657906 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff0e81039c0 0x7ff0e8103e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:54:43.018 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.015+0000 7ff0ec9b3640 1 --2- 172.21.15.29:0/3443657906 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff0e81099d0 0x7ff0e8109db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:54:43.018 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.015+0000 7ff0ec9b3640 1 -- 172.21.15.29:0/3443657906 >> 172.21.15.29:0/3443657906 conn(0x7ff0e80ff490 msgr2=0x7ff0e81018b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:54:43.018 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.015+0000 7ff0ec9b3640 1 -- 172.21.15.29:0/3443657906 shutdown_connections 2024-03-16T11:54:43.018 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.015+0000 7ff0ec9b3640 1 -- 172.21.15.29:0/3443657906 wait complete. 2024-03-16T11:54:43.018 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.016+0000 7ff0ec9b3640 1 Processor -- start 2024-03-16T11:54:43.018 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.016+0000 7ff0ec9b3640 1 -- start start 2024-03-16T11:54:43.018 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.016+0000 7ff0ec9b3640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff0e81039c0 0x7ff0e819a1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:54:43.018 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.016+0000 7ff0ec9b3640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff0e8104380 0x7ff0e819a6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:54:43.018 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.016+0000 7ff0ec9b3640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff0e81099d0 0x7ff0e819ea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:54:43.018 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.016+0000 7ff0ec9b3640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7ff0e8113de0 con 0x7ff0e8104380 2024-03-16T11:54:43.018 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.016+0000 7ff0ec9b3640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7ff0e8113c60 con 0x7ff0e81039c0 2024-03-16T11:54:43.019 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.016+0000 7ff0ec9b3640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7ff0e8113f60 con 0x7ff0e81099d0 2024-03-16T11:54:43.019 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.016+0000 7ff0ddd9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff0e8104380 0x7ff0e819a6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:54:43.019 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.016+0000 7ff0e659c640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff0e81039c0 0x7ff0e819a1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:54:43.019 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.016+0000 7ff0ddd9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff0e8104380 0x7ff0e819a6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55970/0 (socket says 172.21.15.29:55970) 2024-03-16T11:54:43.019 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.016+0000 7ff0e659c640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff0e81039c0 0x7ff0e819a1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.29:60096/0 (socket says 172.21.15.29:60096) 2024-03-16T11:54:43.019 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.016+0000 7ff0ddd9b640 1 -- 172.21.15.29:0/2798957597 learned_addr learned my addr 172.21.15.29:0/2798957597 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:54:43.019 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.016+0000 7ff0e6d9d640 1 --2- 172.21.15.29:0/2798957597 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff0e81099d0 0x7ff0e819ea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:54:43.019 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.017+0000 7ff0ddd9b640 1 -- 172.21.15.29:0/2798957597 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff0e81099d0 msgr2=0x7ff0e819ea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:54:43.019 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.017+0000 7ff0ddd9b640 1 --2- 172.21.15.29:0/2798957597 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff0e81099d0 0x7ff0e819ea60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:54:43.019 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.017+0000 7ff0ddd9b640 1 -- 172.21.15.29:0/2798957597 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff0e81039c0 msgr2=0x7ff0e819a1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:54:43.019 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.017+0000 7ff0ddd9b640 1 --2- 172.21.15.29:0/2798957597 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff0e81039c0 0x7ff0e819a1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:54:43.019 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.017+0000 7ff0ddd9b640 1 -- 172.21.15.29:0/2798957597 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0e819f190 con 0x7ff0e8104380 2024-03-16T11:54:43.019 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.017+0000 7ff0e659c640 1 --2- 172.21.15.29:0/2798957597 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff0e81039c0 0x7ff0e819a1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:54:43.019 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.017+0000 7ff0ddd9b640 1 --2- 172.21.15.29:0/2798957597 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff0e8104380 0x7ff0e819a6e0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7ff0d000e9f0 tx=0x7ff0d000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:54:43.019 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.017+0000 7ff0dffff640 1 -- 172.21.15.29:0/2798957597 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0d000cd80 con 0x7ff0e8104380 2024-03-16T11:54:43.020 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.017+0000 7ff0dffff640 1 -- 172.21.15.29:0/2798957597 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff0d0005500 con 0x7ff0e8104380 2024-03-16T11:54:43.020 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.017+0000 7ff0ec9b3640 1 -- 172.21.15.29:0/2798957597 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff0e819f480 con 0x7ff0e8104380 2024-03-16T11:54:43.020 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.017+0000 7ff0dffff640 1 -- 172.21.15.29:0/2798957597 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0d0010690 con 0x7ff0e8104380 2024-03-16T11:54:43.020 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.017+0000 7ff0ec9b3640 1 -- 172.21.15.29:0/2798957597 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff0e81a6d10 con 0x7ff0e8104380 2024-03-16T11:54:43.025 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.019+0000 7ff0dffff640 1 -- 172.21.15.29:0/2798957597 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7ff0d00046c0 con 0x7ff0e8104380 2024-03-16T11:54:43.025 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.019+0000 7ff0ef598740 1 Processor -- start 2024-03-16T11:54:43.025 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.020+0000 7ff0ef598740 1 -- start start 2024-03-16T11:54:43.025 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.020+0000 7ff0dffff640 1 --2- 172.21.15.29:0/2798957597 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7ff0d40a2a10 0x7ff0d40a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:54:43.026 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.020+0000 7ff0ef598740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564e066f2130 0x564e066f2510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:54:43.026 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.020+0000 7ff0ef598740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564e066f2a50 0x564e066fb2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:54:43.026 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.020+0000 7ff0ef598740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564e066fb820 0x564e066fdc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:54:43.026 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.020+0000 7ff0ef598740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x564e06656a40 con 0x564e066f2130 2024-03-16T11:54:43.026 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.020+0000 7ff0ef598740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x564e065ff920 con 0x564e066f2a50 2024-03-16T11:54:43.026 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.020+0000 7ff0ef598740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x564e06652ae0 con 0x564e066fb820 2024-03-16T11:54:43.026 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.020+0000 7ff0e6d9d640 1 --2- 172.21.15.29:0/2798957597 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7ff0d40a2a10 0x7ff0d40a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:54:43.026 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.020+0000 7ff0dffff640 1 -- 172.21.15.29:0/2798957597 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7ff0d00cf220 con 0x7ff0e8104380 2024-03-16T11:54:43.027 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.020+0000 7ff0e659c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564e066f2130 0x564e066f2510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:54:43.027 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.020+0000 7ff0e659c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564e066f2130 0x564e066f2510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55972/0 (socket says 172.21.15.29:55972) 2024-03-16T11:54:43.027 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.020+0000 7ff0e659c640 1 -- 172.21.15.29:0/3815026013 learned_addr learned my addr 172.21.15.29:0/3815026013 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:54:43.027 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.020+0000 7ff0ddd9b640 1 --2- 172.21.15.29:0/3815026013 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564e066f2a50 0x564e066fb2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:54:43.027 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.020+0000 7ff0e6d9d640 1 --2- 172.21.15.29:0/3815026013 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564e066fb820 0x564e066fdc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:54:43.027 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.020+0000 7ff0ddd9b640 1 -- 172.21.15.29:0/3815026013 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564e066fb820 msgr2=0x564e066fdc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:54:43.027 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.020+0000 7ff0ddd9b640 1 --2- 172.21.15.29:0/3815026013 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564e066fb820 0x564e066fdc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:54:43.027 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.020+0000 7ff0ddd9b640 1 -- 172.21.15.29:0/3815026013 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564e066f2130 msgr2=0x564e066f2510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:54:43.027 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.020+0000 7ff0ddd9b640 1 --2- 172.21.15.29:0/3815026013 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564e066f2130 0x564e066f2510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:54:43.027 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.020+0000 7ff0ddd9b640 1 -- 172.21.15.29:0/3815026013 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564e06617460 con 0x564e066f2a50 2024-03-16T11:54:43.027 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.020+0000 7ff0e659c640 1 --2- 172.21.15.29:0/3815026013 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564e066f2130 0x564e066f2510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:54:43.027 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.020+0000 7ff0e6d9d640 1 --2- 172.21.15.29:0/2798957597 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7ff0d40a2a10 0x7ff0d40a4ed0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7ff0e0005cc0 tx=0x7ff0e0032340 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:54:43.027 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.021+0000 7ff0ddd9b640 1 --2- 172.21.15.29:0/3815026013 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564e066f2a50 0x564e066fb2e0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7ff0d00cc9a0 tx=0x7ff0d009b3f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=18c90868096a0123 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:54:43.028 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.021+0000 7ff0dcd99640 1 -- 172.21.15.29:0/3815026013 <== mon.2 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0d007f590 con 0x564e066f2a50 2024-03-16T11:54:43.028 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.021+0000 7ff0dcd99640 1 -- 172.21.15.29:0/3815026013 <== mon.2 v2:172.21.15.152:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff0d009bfb0 con 0x564e066f2a50 2024-03-16T11:54:43.028 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.021+0000 7ff0dcd99640 1 -- 172.21.15.29:0/3815026013 <== mon.2 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0d0099bb0 con 0x564e066f2a50 2024-03-16T11:54:43.028 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.021+0000 7ff0ef598740 1 -- 172.21.15.29:0/3815026013 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564e066f2a50 msgr2=0x564e066fb2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:54:43.028 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.021+0000 7ff0ef598740 1 --2- 172.21.15.29:0/3815026013 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564e066f2a50 0x564e066fb2e0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7ff0d00cc9a0 tx=0x7ff0d009b3f0 comp rx=0 tx=0).stop 2024-03-16T11:54:43.028 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.022+0000 7ff0ef598740 1 -- 172.21.15.29:0/3815026013 shutdown_connections 2024-03-16T11:54:43.028 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.022+0000 7ff0ef598740 1 --2- 172.21.15.29:0/3815026013 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564e066fb820 0x564e066fdc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:54:43.028 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.022+0000 7ff0ef598740 1 --2- 172.21.15.29:0/3815026013 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564e066f2a50 0x564e066fb2e0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:54:43.028 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.022+0000 7ff0ef598740 1 --2- 172.21.15.29:0/3815026013 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564e066f2130 0x564e066f2510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:54:43.028 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.022+0000 7ff0ef598740 1 -- 172.21.15.29:0/3815026013 >> 172.21.15.29:0/3815026013 conn(0x564e066eca50 msgr2=0x564e066ece30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:54:43.028 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.022+0000 7ff0ef598740 1 -- 172.21.15.29:0/3815026013 shutdown_connections 2024-03-16T11:54:43.028 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.022+0000 7ff0ef598740 1 -- 172.21.15.29:0/3815026013 wait complete. 2024-03-16T11:54:43.028 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.022+0000 7ff0ef598740 1 -- start start 2024-03-16T11:54:43.028 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.022+0000 7ff0ef598740 1 Processor -- start 2024-03-16T11:54:43.029 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.022+0000 7ff0ef598740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564e066f2130 0x564e066f8940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:54:43.029 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.022+0000 7ff0ef598740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564e066f2a50 0x564e066f8e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:54:43.029 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.022+0000 7ff0ef598740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564e066fb820 0x564e066f93c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:54:43.029 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.022+0000 7ff0ef598740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x564e066fe3f0 con 0x564e066fb820 2024-03-16T11:54:43.029 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.022+0000 7ff0ef598740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x564e06612d70 con 0x564e066f2a50 2024-03-16T11:54:43.029 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.022+0000 7ff0ef598740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x564e066fe570 con 0x564e066f2130 2024-03-16T11:54:43.029 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.022+0000 7ff0ef598740 10 client.0 _ll_register_callbacks cb 0x564e0636aa00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T11:54:43.029 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.022+0000 7ff0e6d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564e066fb820 0x564e066f93c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:54:43.029 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.022+0000 7ff0e659c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564e066f2130 0x564e066f8940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:54:43.029 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.022+0000 7ff0e6d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564e066fb820 0x564e066f93c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55982/0 (socket says 172.21.15.29:55982) 2024-03-16T11:54:43.029 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.022+0000 7ff0e6d9d640 1 -- 172.21.15.29:0/1310102080 learned_addr learned my addr 172.21.15.29:0/1310102080 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:54:43.029 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.023+0000 7ff0ddd9b640 1 --2- 172.21.15.29:0/1310102080 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564e066f2a50 0x564e066f8e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:54:43.029 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.023+0000 7ff0e6d9d640 1 -- 172.21.15.29:0/1310102080 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564e066f2130 msgr2=0x564e066f8940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:54:43.030 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.023+0000 7ff0e6d9d640 1 --2- 172.21.15.29:0/1310102080 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564e066f2130 0x564e066f8940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:54:43.030 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.023+0000 7ff0e6d9d640 1 -- 172.21.15.29:0/1310102080 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564e066f2a50 msgr2=0x564e066f8e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:54:43.030 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.023+0000 7ff0e6d9d640 1 --2- 172.21.15.29:0/1310102080 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564e066f2a50 0x564e066f8e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:54:43.030 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.023+0000 7ff0e6d9d640 1 -- 172.21.15.29:0/1310102080 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564e0636b210 con 0x564e066fb820 2024-03-16T11:54:43.030 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.023+0000 7ff0ddd9b640 1 --2- 172.21.15.29:0/1310102080 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564e066f2a50 0x564e066f8e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:54:43.030 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.023+0000 7ff0e6d9d640 1 --2- 172.21.15.29:0/1310102080 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564e066fb820 0x564e066f93c0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7ff0e0036000 tx=0x7ff0e002c0f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9029f1af2302d7b server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:54:43.030 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.023+0000 7ff0cdffb640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-16T11:54:43.030 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.023+0000 7ff0cdffb640 1 -- 172.21.15.29:0/1310102080 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0e0039040 con 0x564e066fb820 2024-03-16T11:54:43.030 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.023+0000 7ff0cdffb640 1 -- 172.21.15.29:0/1310102080 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7ff0e0004040 con 0x564e066fb820 2024-03-16T11:54:43.030 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.023+0000 7ff0ef598740 1 -- 172.21.15.29:0/1310102080 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff0d009bfb0 con 0x564e066fb820 2024-03-16T11:54:43.030 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.023+0000 7ff0cdffb640 1 -- 172.21.15.29:0/1310102080 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7ff0e0004240 con 0x564e066fb820 2024-03-16T11:54:43.030 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.023+0000 7ff0ef598740 10 client.15720 fetch_fsmap learned FSMap version 24 2024-03-16T11:54:43.030 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.023+0000 7ff0ef598740 1 -- 172.21.15.29:0/1310102080 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x564e06437620 con 0x564e066fb820 2024-03-16T11:54:43.030 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.023+0000 7ff0cdffb640 1 -- 172.21.15.29:0/1310102080 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff0d009bfb0 con 0x564e066fb820 2024-03-16T11:54:43.031 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.024+0000 7ff0cdffb640 1 -- 172.21.15.29:0/1310102080 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7ff0e003c030 con 0x564e066fb820 2024-03-16T11:54:43.031 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.024+0000 7ff0ef598740 10 client.15720 fetch_fsmap finished waiting for FSMap version 24 2024-03-16T11:54:43.031 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.024+0000 7ff0ef598740 10 client.15720 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T11:54:43.031 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.024+0000 7ff0ef598740 20 client.15720 populate_metadata read hostname 'smithi029' 2024-03-16T11:54:43.031 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.024+0000 7ff0ef598740 1 --2- 172.21.15.29:0/1310102080 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x564e0670ba80 0x564e0670de70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:54:43.031 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.024+0000 7ff0ef598740 4 client.15720 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T11:54:43.031 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.024+0000 7ff0ef598740 1 -- 172.21.15.29:0/1310102080 --> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564e06663f80 con 0x564e0670ba80 2024-03-16T11:54:43.031 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.024+0000 7ff0e659c640 1 --2- 172.21.15.29:0/1310102080 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x564e0670ba80 0x564e0670de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:54:43.031 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.024+0000 7ff0e659c640 1 --2- 172.21.15.29:0/1310102080 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x564e0670ba80 0x564e0670de70 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dd387148d44c8d82 server_cookie=bff0159cd6786874 in_seq=0 out_seq=0 2024-03-16T11:54:43.031 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.024+0000 7ff0cdffb640 10 client.15720 ms_handle_connect on v2:172.21.15.152:6838/1548731833 2024-03-16T11:54:43.032 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.027+0000 7ff0cdffb640 1 -- 172.21.15.29:0/1310102080 <== mds.0 v2:172.21.15.152:6838/1548731833 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x564e06663f80 con 0x564e0670ba80 2024-03-16T11:54:43.032 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.027+0000 7ff0cdffb640 10 client.15720 handle_command_reply: tid=0 2024-03-16T11:54:43.032 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.027+0000 7ff0ef598740 1 client.15720 shutdown 2024-03-16T11:54:43.033 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.027+0000 7ff0ef598740 1 -- 172.21.15.29:0/1310102080 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564e066fb820 msgr2=0x564e066f93c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:54:43.033 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.027+0000 7ff0ef598740 1 --2- 172.21.15.29:0/1310102080 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564e066fb820 0x564e066f93c0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7ff0e0036000 tx=0x7ff0e002c0f0 comp rx=0 tx=0).stop 2024-03-16T11:54:43.033 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.027+0000 7ff0ef598740 1 -- 172.21.15.29:0/1310102080 shutdown_connections 2024-03-16T11:54:43.033 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.027+0000 7ff0ef598740 1 --2- 172.21.15.29:0/1310102080 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564e066fb820 0x564e066f93c0 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:54:43.033 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.027+0000 7ff0ef598740 1 --2- 172.21.15.29:0/1310102080 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x564e0670ba80 0x564e0670de70 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:54:43.033 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.027+0000 7ff0ef598740 1 --2- 172.21.15.29:0/1310102080 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564e066f2a50 0x564e066f8e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:54:43.033 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.027+0000 7ff0ef598740 1 --2- 172.21.15.29:0/1310102080 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564e066f2130 0x564e066f8940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:54:43.033 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.027+0000 7ff0cdffb640 0 client.15720 ms_handle_reset on v2:172.21.15.152:6838/1548731833 2024-03-16T11:54:43.033 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.027+0000 7ff0ef598740 1 -- 172.21.15.29:0/1310102080 >> 172.21.15.29:0/1310102080 conn(0x564e066eca50 msgr2=0x564e066ee7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:54:43.033 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.027+0000 7ff0ef598740 1 -- 172.21.15.29:0/1310102080 shutdown_connections 2024-03-16T11:54:43.033 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.027+0000 7ff0ef598740 1 -- 172.21.15.29:0/1310102080 wait complete. 2024-03-16T11:54:43.033 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.027+0000 7ff0ef598740 20 client.15720 trim_cache size 0 max 16384 2024-03-16T11:54:43.047 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.042+0000 7ff0ef598740 1 Processor -- start 2024-03-16T11:54:43.047 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.042+0000 7ff0ef598740 1 -- start start 2024-03-16T11:54:43.047 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.042+0000 7ff0ef598740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564e066f2130 0x564e06707190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:54:43.047 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.042+0000 7ff0ef598740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564e066f2a50 0x564e06713730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:54:43.047 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.042+0000 7ff0ef598740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564e066fb820 0x564e066eb320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:54:43.047 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.042+0000 7ff0ef598740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7ff0e81b1ed0 con 0x564e066f2a50 2024-03-16T11:54:43.047 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.042+0000 7ff0ef598740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7ff0e81af7a0 con 0x564e066fb820 2024-03-16T11:54:43.047 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.042+0000 7ff0ef598740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7ff0e81ad5f0 con 0x564e066f2130 2024-03-16T11:54:43.047 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.042+0000 7ff0ddd9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564e066f2a50 0x564e06713730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:54:43.047 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.042+0000 7ff0ddd9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564e066f2a50 0x564e06713730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55996/0 (socket says 172.21.15.29:55996) 2024-03-16T11:54:43.047 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.042+0000 7ff0ddd9b640 1 -- 172.21.15.29:0/4246747582 learned_addr learned my addr 172.21.15.29:0/4246747582 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:54:43.048 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.042+0000 7ff0e659c640 1 --2- 172.21.15.29:0/4246747582 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564e066f2130 0x564e06707190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:54:43.048 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.042+0000 7ff0e6d9d640 1 --2- 172.21.15.29:0/4246747582 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564e066fb820 0x564e066eb320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:54:43.048 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.042+0000 7ff0ddd9b640 1 -- 172.21.15.29:0/4246747582 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564e066f2130 msgr2=0x564e06707190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:54:43.048 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.042+0000 7ff0ddd9b640 1 --2- 172.21.15.29:0/4246747582 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564e066f2130 0x564e06707190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:54:43.048 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.042+0000 7ff0ddd9b640 1 -- 172.21.15.29:0/4246747582 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564e066fb820 msgr2=0x564e066eb320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:54:43.048 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.042+0000 7ff0ddd9b640 1 --2- 172.21.15.29:0/4246747582 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564e066fb820 0x564e066eb320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:54:43.048 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.042+0000 7ff0ddd9b640 1 -- 172.21.15.29:0/4246747582 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564e0670e4a0 con 0x564e066f2a50 2024-03-16T11:54:43.048 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.042+0000 7ff0e6d9d640 1 --2- 172.21.15.29:0/4246747582 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564e066fb820 0x564e066eb320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T11:54:43.048 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.043+0000 7ff0ddd9b640 1 --2- 172.21.15.29:0/4246747582 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564e066f2a50 0x564e06713730 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7ff0d0097d10 tx=0x7ff0d0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97575f58f50e8182 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:54:43.048 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.043+0000 7ff0cffff640 1 -- 172.21.15.29:0/4246747582 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0d009fa00 con 0x564e066f2a50 2024-03-16T11:54:43.048 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.043+0000 7ff0cffff640 1 -- 172.21.15.29:0/4246747582 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff0d009bfb0 con 0x564e066f2a50 2024-03-16T11:54:43.048 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.043+0000 7ff0cffff640 1 -- 172.21.15.29:0/4246747582 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0d00980b0 con 0x564e066f2a50 2024-03-16T11:54:43.048 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.043+0000 7ff0ef598740 1 -- 172.21.15.29:0/4246747582 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564e066f2a50 msgr2=0x564e06713730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:54:43.048 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.043+0000 7ff0ef598740 1 --2- 172.21.15.29:0/4246747582 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564e066f2a50 0x564e06713730 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7ff0d0097d10 tx=0x7ff0d0004a40 comp rx=0 tx=0).stop 2024-03-16T11:54:43.048 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.043+0000 7ff0ef598740 1 -- 172.21.15.29:0/4246747582 shutdown_connections 2024-03-16T11:54:43.049 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.043+0000 7ff0ef598740 1 --2- 172.21.15.29:0/4246747582 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564e066fb820 0x564e066eb320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:54:43.049 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.043+0000 7ff0ef598740 1 --2- 172.21.15.29:0/4246747582 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564e066f2a50 0x564e06713730 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:54:43.049 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.043+0000 7ff0ef598740 1 --2- 172.21.15.29:0/4246747582 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564e066f2130 0x564e06707190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:54:43.049 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.043+0000 7ff0ef598740 1 -- 172.21.15.29:0/4246747582 >> 172.21.15.29:0/4246747582 conn(0x564e066eca50 msgr2=0x564e066ece30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:54:43.049 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.044+0000 7ff0ef598740 1 -- 172.21.15.29:0/4246747582 shutdown_connections 2024-03-16T11:54:43.049 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.044+0000 7ff0ef598740 1 -- 172.21.15.29:0/4246747582 wait complete. 2024-03-16T11:54:43.049 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.044+0000 7ff0ef598740 1 -- start start 2024-03-16T11:54:43.049 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.044+0000 7ff0ef598740 1 Processor -- start 2024-03-16T11:54:43.049 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.044+0000 7ff0ef598740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564e066f2130 0x564e066f56f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:54:43.049 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.044+0000 7ff0ef598740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564e066f2a50 0x564e066f5c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:54:43.049 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.044+0000 7ff0ef598740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564e066fb820 0x564e066f6170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:54:43.049 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.044+0000 7ff0ef598740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7ff0e81adc60 con 0x564e066f2130 2024-03-16T11:54:43.049 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.044+0000 7ff0ef598740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7ff0e81af0d0 con 0x564e066fb820 2024-03-16T11:54:43.049 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.044+0000 7ff0ef598740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7ff0e81ae330 con 0x564e066f2a50 2024-03-16T11:54:43.050 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.044+0000 7ff0ef598740 10 client.0 _ll_register_callbacks cb 0x7ff0d0005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T11:54:43.050 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.044+0000 7ff0e659c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564e066f2130 0x564e066f56f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:54:43.050 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.044+0000 7ff0e659c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564e066f2130 0x564e066f56f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56008/0 (socket says 172.21.15.29:56008) 2024-03-16T11:54:43.050 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.044+0000 7ff0e659c640 1 -- 172.21.15.29:0/1955980570 learned_addr learned my addr 172.21.15.29:0/1955980570 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:54:43.050 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.044+0000 7ff0ddd9b640 1 --2- 172.21.15.29:0/1955980570 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564e066f2a50 0x564e066f5c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:54:43.050 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.044+0000 7ff0e6d9d640 1 --2- 172.21.15.29:0/1955980570 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564e066fb820 0x564e066f6170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:54:43.050 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.044+0000 7ff0e659c640 1 -- 172.21.15.29:0/1955980570 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564e066f2a50 msgr2=0x564e066f5c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:54:43.050 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.044+0000 7ff0e659c640 1 --2- 172.21.15.29:0/1955980570 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564e066f2a50 0x564e066f5c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:54:43.050 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.044+0000 7ff0e659c640 1 -- 172.21.15.29:0/1955980570 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564e066fb820 msgr2=0x564e066f6170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:54:43.050 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.044+0000 7ff0e659c640 1 --2- 172.21.15.29:0/1955980570 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564e066fb820 0x564e066f6170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:54:43.050 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.045+0000 7ff0e659c640 1 -- 172.21.15.29:0/1955980570 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564e06770f00 con 0x564e066f2130 2024-03-16T11:54:43.050 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.045+0000 7ff0e659c640 1 --2- 172.21.15.29:0/1955980570 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564e066f2130 0x564e066f56f0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7ff0d8000c00 tx=0x7ff0d8012bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99e3305c3d6db9a2 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:54:43.050 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.045+0000 7ff0cdffb640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-16T11:54:43.051 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.045+0000 7ff0cdffb640 1 -- 172.21.15.29:0/1955980570 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0d8010c10 con 0x564e066f2130 2024-03-16T11:54:43.051 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.045+0000 7ff0cdffb640 1 -- 172.21.15.29:0/1955980570 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7ff0d8010db0 con 0x564e066f2130 2024-03-16T11:54:43.051 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.045+0000 7ff0ef598740 1 -- 172.21.15.29:0/1955980570 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff0d009bfb0 con 0x564e066f2130 2024-03-16T11:54:43.051 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.045+0000 7ff0cdffb640 1 -- 172.21.15.29:0/1955980570 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7ff0d8005020 con 0x564e066f2130 2024-03-16T11:54:43.051 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.045+0000 7ff0ef598740 10 client.15726 fetch_fsmap learned FSMap version 24 2024-03-16T11:54:43.051 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.045+0000 7ff0ef598740 1 -- 172.21.15.29:0/1955980570 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x564e06772280 con 0x564e066f2130 2024-03-16T11:54:43.051 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.045+0000 7ff0cdffb640 1 -- 172.21.15.29:0/1955980570 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff0d009bfb0 con 0x564e066f2130 2024-03-16T11:54:43.051 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.045+0000 7ff0cdffb640 1 -- 172.21.15.29:0/1955980570 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7ff0d8020040 con 0x564e066f2130 2024-03-16T11:54:43.051 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.045+0000 7ff0ef598740 10 client.15726 fetch_fsmap finished waiting for FSMap version 24 2024-03-16T11:54:43.051 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.045+0000 7ff0ef598740 10 client.15726 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T11:54:43.051 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.045+0000 7ff0ef598740 20 client.15726 populate_metadata read hostname 'smithi029' 2024-03-16T11:54:43.051 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.045+0000 7ff0ef598740 1 --2- 172.21.15.29:0/1955980570 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x564e067729e0 0x564e06774e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:54:43.052 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.045+0000 7ff0ef598740 4 client.15726 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-16T11:54:43.052 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.045+0000 7ff0ef598740 1 -- 172.21.15.29:0/1955980570 --> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] -- command(tid 0: {"prefix": "flush journal"}) -- 0x564e0665e2d0 con 0x564e067729e0 2024-03-16T11:54:43.052 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.046+0000 7ff0ddd9b640 1 --2- 172.21.15.29:0/1955980570 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x564e067729e0 0x564e06774e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:54:43.052 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.046+0000 7ff0ddd9b640 1 --2- 172.21.15.29:0/1955980570 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x564e067729e0 0x564e06774e00 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dec57cdf73ffec1a server_cookie=6828fe1e95e4c367 in_seq=0 out_seq=0 2024-03-16T11:54:43.052 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.046+0000 7ff0cdffb640 10 client.15726 ms_handle_connect on v2:172.21.15.152:6838/1548731833 2024-03-16T11:54:43.052 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:54:42 smithi154 ceph-mon[36320]: pgmap v117: 97 pgs: 97 active+clean; 52 MiB data, 937 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 4.1 MiB/s wr, 401 op/s 2024-03-16T11:54:43.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:54:42 smithi029 ceph-mon[31975]: pgmap v117: 97 pgs: 97 active+clean; 52 MiB data, 937 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 4.1 MiB/s wr, 401 op/s 2024-03-16T11:54:43.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:54:42 smithi152 ceph-mon[36291]: pgmap v117: 97 pgs: 97 active+clean; 52 MiB data, 937 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 4.1 MiB/s wr, 401 op/s 2024-03-16T11:54:43.223 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.222+0000 7ff0cdffb640 1 -- 172.21.15.29:0/1955980570 <== mds.0 v2:172.21.15.152:6838/1548731833 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x564e0665e2d0 con 0x564e067729e0 2024-03-16T11:54:43.223 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.222+0000 7ff0cdffb640 10 client.15726 handle_command_reply: tid=0 2024-03-16T11:54:43.223 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.222+0000 7ff0ef598740 1 client.15726 shutdown 2024-03-16T11:54:43.223 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.222+0000 7ff0ef598740 1 -- 172.21.15.29:0/1955980570 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564e066f2130 msgr2=0x564e066f56f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:54:43.223 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.222+0000 7ff0ef598740 1 --2- 172.21.15.29:0/1955980570 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564e066f2130 0x564e066f56f0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7ff0d8000c00 tx=0x7ff0d8012bf0 comp rx=0 tx=0).stop 2024-03-16T11:54:43.223 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.222+0000 7ff0ef598740 1 -- 172.21.15.29:0/1955980570 shutdown_connections 2024-03-16T11:54:43.223 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.222+0000 7ff0ef598740 1 --2- 172.21.15.29:0/1955980570 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x564e067729e0 0x564e06774e00 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:54:43.223 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.222+0000 7ff0ef598740 1 --2- 172.21.15.29:0/1955980570 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564e066fb820 0x564e066f6170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:54:43.223 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.222+0000 7ff0cdffb640 0 client.15726 ms_handle_reset on v2:172.21.15.152:6838/1548731833 2024-03-16T11:54:43.224 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.222+0000 7ff0ef598740 1 --2- 172.21.15.29:0/1955980570 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564e066f2a50 0x564e066f5c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:54:43.224 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.222+0000 7ff0ef598740 1 --2- 172.21.15.29:0/1955980570 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564e066f2130 0x564e066f56f0 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:54:43.224 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.222+0000 7ff0ef598740 1 -- 172.21.15.29:0/1955980570 >> 172.21.15.29:0/1955980570 conn(0x564e066eca50 msgr2=0x564e066f8900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:54:43.224 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.222+0000 7ff0ef598740 1 -- 172.21.15.29:0/1955980570 shutdown_connections 2024-03-16T11:54:43.224 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.222+0000 7ff0ef598740 1 -- 172.21.15.29:0/1955980570 wait complete. 2024-03-16T11:54:43.224 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.222+0000 7ff0ef598740 20 client.15726 trim_cache size 0 max 16384 2024-03-16T11:54:43.224 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:{ 2024-03-16T11:54:43.224 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "message": "", 2024-03-16T11:54:43.224 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "return_code": 0 2024-03-16T11:54:43.224 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:} 2024-03-16T11:54:43.225 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.223+0000 7ff0ec9b3640 1 -- 172.21.15.29:0/2798957597 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7ff0d40a2a10 msgr2=0x7ff0d40a4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:54:43.225 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.223+0000 7ff0ec9b3640 1 --2- 172.21.15.29:0/2798957597 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7ff0d40a2a10 0x7ff0d40a4ed0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7ff0e0005cc0 tx=0x7ff0e0032340 comp rx=0 tx=0).stop 2024-03-16T11:54:43.226 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.223+0000 7ff0ec9b3640 1 -- 172.21.15.29:0/2798957597 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff0e8104380 msgr2=0x7ff0e819a6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:54:43.226 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.223+0000 7ff0ec9b3640 1 --2- 172.21.15.29:0/2798957597 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff0e8104380 0x7ff0e819a6e0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7ff0d000e9f0 tx=0x7ff0d000edf0 comp rx=0 tx=0).stop 2024-03-16T11:54:43.226 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.223+0000 7ff0ec9b3640 1 -- 172.21.15.29:0/2798957597 shutdown_connections 2024-03-16T11:54:43.226 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.223+0000 7ff0ec9b3640 1 --2- 172.21.15.29:0/2798957597 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7ff0d40a2a10 0x7ff0d40a4ed0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:54:43.226 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.223+0000 7ff0ec9b3640 1 --2- 172.21.15.29:0/2798957597 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff0e81099d0 0x7ff0e819ea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:54:43.226 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.223+0000 7ff0ec9b3640 1 --2- 172.21.15.29:0/2798957597 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff0e8104380 0x7ff0e819a6e0 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:54:43.226 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.223+0000 7ff0ec9b3640 1 --2- 172.21.15.29:0/2798957597 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff0e81039c0 0x7ff0e819a1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:54:43.226 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.223+0000 7ff0ec9b3640 1 -- 172.21.15.29:0/2798957597 >> 172.21.15.29:0/2798957597 conn(0x7ff0e80ff490 msgr2=0x7ff0e80ff970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:54:43.226 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.223+0000 7ff0ec9b3640 1 -- 172.21.15.29:0/2798957597 shutdown_connections 2024-03-16T11:54:43.227 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:43.223+0000 7ff0ec9b3640 1 -- 172.21.15.29:0/2798957597 wait complete. 2024-03-16T11:54:43.781 INFO:tasks.workunit.client.0.smithi029.stdout: 1 331144 11.44 MB/sec execute 28 sec latency 99.269 ms 2024-03-16T11:54:44.781 INFO:tasks.workunit.client.0.smithi029.stdout: 1 333657 11.62 MB/sec execute 29 sec latency 72.824 ms 2024-03-16T11:54:45.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:54:44 smithi154 ceph-mon[36320]: pgmap v118: 97 pgs: 97 active+clean; 52 MiB data, 937 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 2.7 MiB/s wr, 204 op/s 2024-03-16T11:54:45.087 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:54:44 smithi029 ceph-mon[31975]: pgmap v118: 97 pgs: 97 active+clean; 52 MiB data, 937 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 2.7 MiB/s wr, 204 op/s 2024-03-16T11:54:45.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:54:44 smithi152 ceph-mon[36291]: pgmap v118: 97 pgs: 97 active+clean; 52 MiB data, 937 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 2.7 MiB/s wr, 204 op/s 2024-03-16T11:54:45.781 INFO:tasks.workunit.client.0.smithi029.stdout: 1 335836 11.48 MB/sec execute 30 sec latency 19.355 ms 2024-03-16T11:54:46.781 INFO:tasks.workunit.client.0.smithi029.stdout: 1 337698 11.50 MB/sec execute 31 sec latency 25.903 ms 2024-03-16T11:54:47.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:54:46 smithi154 ceph-mon[36320]: pgmap v119: 97 pgs: 97 active+clean; 52 MiB data, 937 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 2.7 MiB/s wr, 204 op/s 2024-03-16T11:54:47.087 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:54:46 smithi029 ceph-mon[31975]: pgmap v119: 97 pgs: 97 active+clean; 52 MiB data, 937 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 2.7 MiB/s wr, 204 op/s 2024-03-16T11:54:47.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:54:46 smithi152 ceph-mon[36291]: pgmap v119: 97 pgs: 97 active+clean; 52 MiB data, 937 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 2.7 MiB/s wr, 204 op/s 2024-03-16T11:54:47.781 INFO:tasks.workunit.client.0.smithi029.stdout: 1 340592 11.69 MB/sec execute 32 sec latency 12.924 ms 2024-03-16T11:54:48.781 INFO:tasks.workunit.client.0.smithi029.stdout: 1 342485 11.54 MB/sec execute 33 sec latency 17.347 ms 2024-03-16T11:54:49.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:54:48 smithi154 ceph-mon[36320]: pgmap v120: 97 pgs: 97 active+clean; 45 MiB data, 912 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 4.3 MiB/s wr, 420 op/s 2024-03-16T11:54:49.087 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:54:48 smithi029 ceph-mon[31975]: pgmap v120: 97 pgs: 97 active+clean; 45 MiB data, 912 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 4.3 MiB/s wr, 420 op/s 2024-03-16T11:54:49.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:54:48 smithi152 ceph-mon[36291]: pgmap v120: 97 pgs: 97 active+clean; 45 MiB data, 912 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 4.3 MiB/s wr, 420 op/s 2024-03-16T11:54:49.781 INFO:tasks.workunit.client.0.smithi029.stdout: 1 344434 11.60 MB/sec execute 34 sec latency 56.923 ms 2024-03-16T11:54:50.781 INFO:tasks.workunit.client.0.smithi029.stdout: 1 346965 11.61 MB/sec execute 35 sec latency 13.379 ms 2024-03-16T11:54:51.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:54:50 smithi154 ceph-mon[36320]: pgmap v121: 97 pgs: 97 active+clean; 45 MiB data, 912 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 2.9 MiB/s wr, 318 op/s 2024-03-16T11:54:51.087 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:54:50 smithi029 ceph-mon[31975]: pgmap v121: 97 pgs: 97 active+clean; 45 MiB data, 912 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 2.9 MiB/s wr, 318 op/s 2024-03-16T11:54:51.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:54:50 smithi152 ceph-mon[36291]: pgmap v121: 97 pgs: 97 active+clean; 45 MiB data, 912 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 2.9 MiB/s wr, 318 op/s 2024-03-16T11:54:51.781 INFO:tasks.workunit.client.0.smithi029.stdout: 1 348669 11.54 MB/sec execute 36 sec latency 27.924 ms 2024-03-16T11:54:52.781 INFO:tasks.workunit.client.0.smithi029.stdout: 1 351450 11.69 MB/sec execute 37 sec latency 10.916 ms 2024-03-16T11:54:53.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:54:52 smithi154 ceph-mon[36320]: pgmap v122: 97 pgs: 97 active+clean; 50 MiB data, 948 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 4.3 MiB/s wr, 423 op/s 2024-03-16T11:54:53.087 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:54:52 smithi029 ceph-mon[31975]: pgmap v122: 97 pgs: 97 active+clean; 50 MiB data, 948 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 4.3 MiB/s wr, 423 op/s 2024-03-16T11:54:53.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:54:52 smithi152 ceph-mon[36291]: pgmap v122: 97 pgs: 97 active+clean; 50 MiB data, 948 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 4.3 MiB/s wr, 423 op/s 2024-03-16T11:54:53.781 INFO:tasks.workunit.client.0.smithi029.stdout: 1 353766 11.60 MB/sec execute 38 sec latency 14.042 ms 2024-03-16T11:54:54.781 INFO:tasks.workunit.client.0.smithi029.stdout: 1 355398 11.59 MB/sec execute 39 sec latency 113.778 ms 2024-03-16T11:54:55.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:54:54 smithi154 ceph-mon[36320]: pgmap v123: 97 pgs: 97 active+clean; 50 MiB data, 948 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 2.9 MiB/s wr, 320 op/s 2024-03-16T11:54:55.087 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:54:54 smithi029 ceph-mon[31975]: pgmap v123: 97 pgs: 97 active+clean; 50 MiB data, 948 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 2.9 MiB/s wr, 320 op/s 2024-03-16T11:54:55.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:54:54 smithi152 ceph-mon[36291]: pgmap v123: 97 pgs: 97 active+clean; 50 MiB data, 948 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 2.9 MiB/s wr, 320 op/s 2024-03-16T11:54:55.781 INFO:tasks.workunit.client.0.smithi029.stdout: 1 358252 11.73 MB/sec execute 40 sec latency 40.659 ms 2024-03-16T11:54:56.312 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:56.311+0000 7f631e285640 1 -- 172.21.15.29:0/415937177 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f63180ffce0 msgr2=0x7f6318100160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:54:56.312 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:56.311+0000 7f631e285640 1 --2- 172.21.15.29:0/415937177 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f63180ffce0 0x7f6318100160 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f630c009930 tx=0x7f630c02bc30 comp rx=0 tx=0).stop 2024-03-16T11:54:56.313 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:56.311+0000 7f631e285640 1 -- 172.21.15.29:0/415937177 shutdown_connections 2024-03-16T11:54:56.313 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:56.311+0000 7f631e285640 1 --2- 172.21.15.29:0/415937177 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f63181006a0 0x7f631810e670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:54:56.313 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:56.311+0000 7f631e285640 1 --2- 172.21.15.29:0/415937177 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f63180ffce0 0x7f6318100160 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:54:56.313 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:56.311+0000 7f631e285640 1 --2- 172.21.15.29:0/415937177 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f63181067c0 0x7f6318106ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:54:56.313 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:56.311+0000 7f631e285640 1 -- 172.21.15.29:0/415937177 >> 172.21.15.29:0/415937177 conn(0x7f6318076c70 msgr2=0x7f63180fdff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:54:56.313 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:56.311+0000 7f631e285640 1 -- 172.21.15.29:0/415937177 shutdown_connections 2024-03-16T11:54:56.313 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:56.311+0000 7f631e285640 1 -- 172.21.15.29:0/415937177 wait complete. 2024-03-16T11:54:56.313 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:56.312+0000 7f631e285640 1 Processor -- start 2024-03-16T11:54:56.313 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:56.312+0000 7f631e285640 1 -- start start 2024-03-16T11:54:56.313 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:56.312+0000 7f631e285640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f63180ffce0 0x7f6318100ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:54:56.313 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:56.312+0000 7f631e285640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f63181006a0 0x7f6318101400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:54:56.313 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:56.312+0000 7f631e285640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f63181067c0 0x7f6318101940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:54:56.314 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:56.312+0000 7f631e285640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f6318111cd0 con 0x7f63180ffce0 2024-03-16T11:54:56.314 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:56.312+0000 7f631e285640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f6318111b50 con 0x7f63181067c0 2024-03-16T11:54:56.314 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:56.312+0000 7f631e285640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f6318111e50 con 0x7f63181006a0 2024-03-16T11:54:56.314 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:56.312+0000 7f6317fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f63180ffce0 0x7f6318100ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:54:56.314 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:56.312+0000 7f6317fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f63180ffce0 0x7f6318100ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:38394/0 (socket says 172.21.15.29:38394) 2024-03-16T11:54:56.314 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:56.312+0000 7f6317fff640 1 -- 172.21.15.29:0/180226162 learned_addr learned my addr 172.21.15.29:0/180226162 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:54:56.314 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:56.312+0000 7f631c822640 1 --2- 172.21.15.29:0/180226162 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f63181067c0 0x7f6318101940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:54:56.314 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:56.312+0000 7f63177fe640 1 --2- 172.21.15.29:0/180226162 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f63181006a0 0x7f6318101400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:54:56.314 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:56.312+0000 7f6317fff640 1 -- 172.21.15.29:0/180226162 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f63181006a0 msgr2=0x7f6318101400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:54:56.314 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:56.312+0000 7f6317fff640 1 --2- 172.21.15.29:0/180226162 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f63181006a0 0x7f6318101400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:54:56.314 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:56.312+0000 7f6317fff640 1 -- 172.21.15.29:0/180226162 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f63181067c0 msgr2=0x7f6318101940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:54:56.314 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:56.312+0000 7f6317fff640 1 --2- 172.21.15.29:0/180226162 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f63181067c0 0x7f6318101940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:54:56.314 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:56.312+0000 7f6317fff640 1 -- 172.21.15.29:0/180226162 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63181a6590 con 0x7f63180ffce0 2024-03-16T11:54:56.315 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:56.313+0000 7f6317fff640 1 --2- 172.21.15.29:0/180226162 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f63180ffce0 0x7f6318100ec0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f630800e9f0 tx=0x7f630800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:54:56.315 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:56.313+0000 7f63157fa640 1 -- 172.21.15.29:0/180226162 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f630800cd80 con 0x7f63180ffce0 2024-03-16T11:54:56.315 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:56.313+0000 7f631e285640 1 -- 172.21.15.29:0/180226162 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f63181a6880 con 0x7f63180ffce0 2024-03-16T11:54:56.315 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:56.313+0000 7f63157fa640 1 -- 172.21.15.29:0/180226162 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6308010910 con 0x7f63180ffce0 2024-03-16T11:54:56.315 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:56.313+0000 7f63157fa640 1 -- 172.21.15.29:0/180226162 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63080188f0 con 0x7f63180ffce0 2024-03-16T11:54:56.315 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:56.313+0000 7f631e285640 1 -- 172.21.15.29:0/180226162 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f63181a6df0 con 0x7f63180ffce0 2024-03-16T11:54:56.321 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:56.315+0000 7f63157fa640 1 -- 172.21.15.29:0/180226162 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7f6308010ac0 con 0x7f63180ffce0 2024-03-16T11:54:56.321 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:56.315+0000 7f6320e76740 1 Processor -- start 2024-03-16T11:54:56.322 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:56.315+0000 7f6320e76740 1 -- start start 2024-03-16T11:54:56.322 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:56.316+0000 7f6320e76740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e621bc3130 0x55e621bc3510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:54:56.322 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:56.316+0000 7f6320e76740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e621bc3a50 0x55e621bcc2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:54:56.322 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:56.316+0000 7f63157fa640 1 --2- 172.21.15.29:0/180226162 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f63000a2a10 0x7f63000a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:54:56.322 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:56.316+0000 7f6320e76740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e621bcc820 0x55e621bcec10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:54:56.322 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:56.316+0000 7f6320e76740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55e621b27a40 con 0x55e621bc3130 2024-03-16T11:54:56.322 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:56.316+0000 7f6320e76740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x55e621ad0920 con 0x55e621bc3a50 2024-03-16T11:54:56.322 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:56.316+0000 7f6320e76740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55e621b23ae0 con 0x55e621bcc820 2024-03-16T11:54:56.322 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:56.316+0000 7f631c822640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e621bc3130 0x55e621bc3510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:54:56.322 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:56.316+0000 7f63177fe640 1 --2- 172.21.15.29:0/180226162 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f63000a2a10 0x7f63000a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:54:56.322 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:56.316+0000 7f631c822640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e621bc3130 0x55e621bc3510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:38408/0 (socket says 172.21.15.29:38408) 2024-03-16T11:54:56.322 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:56.316+0000 7f631c822640 1 -- 172.21.15.29:0/4212275947 learned_addr learned my addr 172.21.15.29:0/4212275947 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:54:56.322 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:56.316+0000 7f6317fff640 1 --2- 172.21.15.29:0/4212275947 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e621bc3a50 0x55e621bcc2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:54:56.323 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:56.316+0000 7f631c822640 1 --2- 172.21.15.29:0/4212275947 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e621bcc820 0x55e621bcec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:54:56.323 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:56.316+0000 7f63157fa640 1 -- 172.21.15.29:0/180226162 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7f63080cf0b0 con 0x7f63180ffce0 2024-03-16T11:54:56.323 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:56.316+0000 7f6317fff640 1 -- 172.21.15.29:0/4212275947 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e621bcc820 msgr2=0x55e621bcec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:54:56.323 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:56.316+0000 7f6317fff640 1 --2- 172.21.15.29:0/4212275947 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e621bcc820 0x55e621bcec10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:54:56.323 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:56.316+0000 7f6317fff640 1 -- 172.21.15.29:0/4212275947 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e621bc3130 msgr2=0x55e621bc3510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:54:56.323 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:56.316+0000 7f6317fff640 1 --2- 172.21.15.29:0/4212275947 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e621bc3130 0x55e621bc3510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:54:56.323 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:56.316+0000 7f6317fff640 1 -- 172.21.15.29:0/4212275947 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e621ae8460 con 0x55e621bc3a50 2024-03-16T11:54:56.323 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:56.316+0000 7f631c822640 1 --2- 172.21.15.29:0/4212275947 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e621bc3130 0x55e621bc3510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:54:56.323 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:56.316+0000 7f63177fe640 1 --2- 172.21.15.29:0/180226162 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f63000a2a10 0x7f63000a4ed0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f630c005ab0 tx=0x7f630c005fd0 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:54:56.323 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:56.316+0000 7f6317fff640 1 --2- 172.21.15.29:0/4212275947 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e621bc3a50 0x55e621bcc2e0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f63080cc8d0 tx=0x7f630801ccc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2bb8f39ac81f2a4e server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:54:56.323 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:56.316+0000 7f62fe7fc640 1 -- 172.21.15.29:0/4212275947 <== mon.2 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6308026040 con 0x55e621bc3a50 2024-03-16T11:54:56.323 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:56.317+0000 7f62fe7fc640 1 -- 172.21.15.29:0/4212275947 <== mon.2 v2:172.21.15.152:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f630809b890 con 0x55e621bc3a50 2024-03-16T11:54:56.323 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:56.317+0000 7f62fe7fc640 1 -- 172.21.15.29:0/4212275947 <== mon.2 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63080ccce0 con 0x55e621bc3a50 2024-03-16T11:54:56.323 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:56.317+0000 7f6320e76740 1 -- 172.21.15.29:0/4212275947 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e621bc3a50 msgr2=0x55e621bcc2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:54:56.324 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:56.317+0000 7f6320e76740 1 --2- 172.21.15.29:0/4212275947 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e621bc3a50 0x55e621bcc2e0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f63080cc8d0 tx=0x7f630801ccc0 comp rx=0 tx=0).stop 2024-03-16T11:54:56.324 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:56.317+0000 7f6320e76740 1 -- 172.21.15.29:0/4212275947 shutdown_connections 2024-03-16T11:54:56.324 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:56.317+0000 7f6320e76740 1 --2- 172.21.15.29:0/4212275947 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e621bcc820 0x55e621bcec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:54:56.324 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:56.317+0000 7f6320e76740 1 --2- 172.21.15.29:0/4212275947 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e621bc3a50 0x55e621bcc2e0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:54:56.324 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:56.317+0000 7f6320e76740 1 --2- 172.21.15.29:0/4212275947 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e621bc3130 0x55e621bc3510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:54:56.324 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:56.317+0000 7f6320e76740 1 -- 172.21.15.29:0/4212275947 >> 172.21.15.29:0/4212275947 conn(0x55e621bbda50 msgr2=0x55e621bbde30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:54:56.324 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:56.317+0000 7f6320e76740 1 -- 172.21.15.29:0/4212275947 shutdown_connections 2024-03-16T11:54:56.324 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:56.317+0000 7f6320e76740 1 -- 172.21.15.29:0/4212275947 wait complete. 2024-03-16T11:54:56.324 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:56.318+0000 7f6320e76740 1 -- start start 2024-03-16T11:54:56.324 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:56.318+0000 7f6320e76740 1 Processor -- start 2024-03-16T11:54:56.324 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:56.318+0000 7f6320e76740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e621bc3130 0x55e621bc9950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:54:56.324 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:56.318+0000 7f6320e76740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e621bc3a50 0x55e621bc9e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:54:56.324 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:56.318+0000 7f6320e76740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e621bcc820 0x55e621bca3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:54:56.324 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:56.318+0000 7f6320e76740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55e621bcf390 con 0x55e621bcc820 2024-03-16T11:54:56.325 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:56.318+0000 7f6320e76740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x55e621ae3d70 con 0x55e621bc3a50 2024-03-16T11:54:56.325 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:56.318+0000 7f6320e76740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55e621bcf510 con 0x55e621bc3130 2024-03-16T11:54:56.325 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:56.318+0000 7f6320e76740 10 client.0 _ll_register_callbacks cb 0x55e62183ba00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T11:54:56.325 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:56.318+0000 7f631c822640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e621bcc820 0x55e621bca3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:54:56.325 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:56.318+0000 7f6317fff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e621bc3130 0x55e621bc9950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:54:56.325 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:56.318+0000 7f631c822640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e621bcc820 0x55e621bca3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:38424/0 (socket says 172.21.15.29:38424) 2024-03-16T11:54:56.325 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:56.318+0000 7f6317fff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e621bc3130 0x55e621bc9950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.29:46322/0 (socket says 172.21.15.29:46322) 2024-03-16T11:54:56.325 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:56.318+0000 7f631c822640 1 -- 172.21.15.29:0/134177202 learned_addr learned my addr 172.21.15.29:0/134177202 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:54:56.325 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:56.318+0000 7f63177fe640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e621bc3a50 0x55e621bc9e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:54:56.325 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:56.318+0000 7f631c822640 1 -- 172.21.15.29:0/134177202 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e621bc3130 msgr2=0x55e621bc9950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:54:56.325 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:56.318+0000 7f631c822640 1 --2- 172.21.15.29:0/134177202 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e621bc3130 0x55e621bc9950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:54:56.325 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:56.318+0000 7f631c822640 1 -- 172.21.15.29:0/134177202 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e621bc3a50 msgr2=0x55e621bc9e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:54:56.325 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:56.318+0000 7f631c822640 1 --2- 172.21.15.29:0/134177202 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e621bc3a50 0x55e621bc9e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:54:56.325 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:56.319+0000 7f631c822640 1 -- 172.21.15.29:0/134177202 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e62183c210 con 0x55e621bcc820 2024-03-16T11:54:56.326 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:56.319+0000 7f6317fff640 1 --2- 172.21.15.29:0/134177202 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e621bc3130 0x55e621bc9950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:54:56.326 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:56.319+0000 7f631c822640 1 --2- 172.21.15.29:0/134177202 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e621bcc820 0x55e621bca3d0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f6310013d10 tx=0x7f6310010800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7dcd7a3602f0bee5 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:54:56.326 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:56.319+0000 7f62f77fe640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-16T11:54:56.326 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:56.319+0000 7f62f77fe640 1 -- 172.21.15.29:0/134177202 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6310004d20 con 0x55e621bcc820 2024-03-16T11:54:56.326 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:56.319+0000 7f62f77fe640 1 -- 172.21.15.29:0/134177202 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7f631000f5d0 con 0x55e621bcc820 2024-03-16T11:54:56.326 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:56.319+0000 7f6320e76740 1 -- 172.21.15.29:0/134177202 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f630809b890 con 0x55e621bcc820 2024-03-16T11:54:56.326 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:56.319+0000 7f62f77fe640 1 -- 172.21.15.29:0/134177202 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f6310026050 con 0x55e621bcc820 2024-03-16T11:54:56.326 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:56.319+0000 7f6320e76740 10 client.15738 fetch_fsmap learned FSMap version 24 2024-03-16T11:54:56.326 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:56.319+0000 7f6320e76740 1 -- 172.21.15.29:0/134177202 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55e621908620 con 0x55e621bcc820 2024-03-16T11:54:56.326 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:56.319+0000 7f62f77fe640 1 -- 172.21.15.29:0/134177202 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f630809b890 con 0x55e621bcc820 2024-03-16T11:54:56.326 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:56.319+0000 7f62f77fe640 1 -- 172.21.15.29:0/134177202 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7f631000e890 con 0x55e621bcc820 2024-03-16T11:54:56.326 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:56.319+0000 7f6320e76740 10 client.15738 fetch_fsmap finished waiting for FSMap version 24 2024-03-16T11:54:56.326 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:56.319+0000 7f6320e76740 10 client.15738 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T11:54:56.326 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:56.319+0000 7f6320e76740 20 client.15738 populate_metadata read hostname 'smithi029' 2024-03-16T11:54:56.326 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:56.319+0000 7f6320e76740 1 --2- 172.21.15.29:0/134177202 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55e621bdc9f0 0x55e621bdede0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:54:56.327 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:56.319+0000 7f6320e76740 4 client.15738 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T11:54:56.327 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:56.319+0000 7f6320e76740 1 -- 172.21.15.29:0/134177202 --> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e621b34f80 con 0x55e621bdc9f0 2024-03-16T11:54:56.327 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:56.320+0000 7f6317fff640 1 --2- 172.21.15.29:0/134177202 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55e621bdc9f0 0x55e621bdede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:54:56.327 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:56.320+0000 7f62f77fe640 10 client.15738 ms_handle_connect on v2:172.21.15.152:6838/1548731833 2024-03-16T11:54:56.327 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:56.320+0000 7f6317fff640 1 --2- 172.21.15.29:0/134177202 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55e621bdc9f0 0x55e621bdede0 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e9288925a3e996a3 server_cookie=adc8cbcffc591993 in_seq=0 out_seq=0 2024-03-16T11:54:56.327 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:56.321+0000 7f62f77fe640 1 -- 172.21.15.29:0/134177202 <== mds.0 v2:172.21.15.152:6838/1548731833 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55e621b34f80 con 0x55e621bdc9f0 2024-03-16T11:54:56.327 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:56.321+0000 7f62f77fe640 10 client.15738 handle_command_reply: tid=0 2024-03-16T11:54:56.327 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:56.321+0000 7f6320e76740 1 client.15738 shutdown 2024-03-16T11:54:56.327 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:56.322+0000 7f6320e76740 1 -- 172.21.15.29:0/134177202 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e621bcc820 msgr2=0x55e621bca3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:54:56.327 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:56.322+0000 7f6320e76740 1 --2- 172.21.15.29:0/134177202 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e621bcc820 0x55e621bca3d0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f6310013d10 tx=0x7f6310010800 comp rx=0 tx=0).stop 2024-03-16T11:54:56.327 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:56.322+0000 7f6320e76740 1 -- 172.21.15.29:0/134177202 shutdown_connections 2024-03-16T11:54:56.327 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:56.322+0000 7f6320e76740 1 --2- 172.21.15.29:0/134177202 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e621bcc820 0x55e621bca3d0 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:54:56.327 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:56.322+0000 7f6320e76740 1 --2- 172.21.15.29:0/134177202 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55e621bdc9f0 0x55e621bdede0 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:54:56.327 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:56.322+0000 7f6320e76740 1 --2- 172.21.15.29:0/134177202 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e621bc3a50 0x55e621bc9e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:54:56.328 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:56.322+0000 7f6320e76740 1 --2- 172.21.15.29:0/134177202 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e621bc3130 0x55e621bc9950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:54:56.328 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:56.322+0000 7f62f77fe640 0 client.15738 ms_handle_reset on v2:172.21.15.152:6838/1548731833 2024-03-16T11:54:56.328 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:56.322+0000 7f6320e76740 1 -- 172.21.15.29:0/134177202 >> 172.21.15.29:0/134177202 conn(0x55e621bbda50 msgr2=0x55e621bbf7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:54:56.328 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:56.322+0000 7f6320e76740 1 -- 172.21.15.29:0/134177202 shutdown_connections 2024-03-16T11:54:56.328 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:56.322+0000 7f6320e76740 1 -- 172.21.15.29:0/134177202 wait complete. 2024-03-16T11:54:56.328 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:56.322+0000 7f6320e76740 20 client.15738 trim_cache size 0 max 16384 2024-03-16T11:54:56.341 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:56.337+0000 7f6320e76740 1 Processor -- start 2024-03-16T11:54:56.342 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:56.337+0000 7f6320e76740 1 -- start start 2024-03-16T11:54:56.342 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:56.337+0000 7f6320e76740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e621bc3130 0x55e621bd8130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:54:56.343 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:56.337+0000 7f6320e76740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e621bc3a50 0x55e621bca1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:54:56.343 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:56.337+0000 7f6320e76740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e621bcc820 0x55e621bbca30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:54:56.343 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:56.337+0000 7f6320e76740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f63181b1fb0 con 0x55e621bcc820 2024-03-16T11:54:56.343 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:56.337+0000 7f6320e76740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f63181af880 con 0x55e621bc3130 2024-03-16T11:54:56.343 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:56.337+0000 7f6320e76740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f63181ad6d0 con 0x55e621bc3a50 2024-03-16T11:54:56.343 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:56.338+0000 7f631c822640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e621bcc820 0x55e621bbca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:54:56.343 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:56.338+0000 7f631c822640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e621bcc820 0x55e621bbca30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:38432/0 (socket says 172.21.15.29:38432) 2024-03-16T11:54:56.343 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:56.338+0000 7f631c822640 1 -- 172.21.15.29:0/2561588794 learned_addr learned my addr 172.21.15.29:0/2561588794 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:54:56.343 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:56.338+0000 7f6317fff640 1 --2- 172.21.15.29:0/2561588794 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e621bc3130 0x55e621bd8130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:54:56.343 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:56.338+0000 7f63177fe640 1 --2- 172.21.15.29:0/2561588794 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e621bc3a50 0x55e621bca1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:54:56.344 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:56.338+0000 7f631c822640 1 -- 172.21.15.29:0/2561588794 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e621bc3a50 msgr2=0x55e621bca1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:54:56.344 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:56.338+0000 7f631c822640 1 --2- 172.21.15.29:0/2561588794 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e621bc3a50 0x55e621bca1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:54:56.344 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:56.338+0000 7f631c822640 1 -- 172.21.15.29:0/2561588794 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e621bc3130 msgr2=0x55e621bd8130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:54:56.344 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:56.338+0000 7f631c822640 1 --2- 172.21.15.29:0/2561588794 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e621bc3130 0x55e621bd8130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:54:56.344 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:56.338+0000 7f631c822640 1 -- 172.21.15.29:0/2561588794 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e621bc5530 con 0x55e621bcc820 2024-03-16T11:54:56.344 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:56.338+0000 7f631c822640 1 --2- 172.21.15.29:0/2561588794 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e621bcc820 0x55e621bbca30 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f6310000f80 tx=0x7f6310009fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd23a4a70992702f server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:54:56.344 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:56.338+0000 7f62fdffb640 1 -- 172.21.15.29:0/2561588794 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f631002f070 con 0x55e621bcc820 2024-03-16T11:54:56.344 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:56.338+0000 7f62fdffb640 1 -- 172.21.15.29:0/2561588794 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6310023070 con 0x55e621bcc820 2024-03-16T11:54:56.344 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:56.338+0000 7f62fdffb640 1 -- 172.21.15.29:0/2561588794 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f631000e950 con 0x55e621bcc820 2024-03-16T11:54:56.344 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:56.339+0000 7f6320e76740 1 -- 172.21.15.29:0/2561588794 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e621bcc820 msgr2=0x55e621bbca30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:54:56.344 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:56.339+0000 7f6320e76740 1 --2- 172.21.15.29:0/2561588794 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e621bcc820 0x55e621bbca30 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f6310000f80 tx=0x7f6310009fb0 comp rx=0 tx=0).stop 2024-03-16T11:54:56.344 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:56.339+0000 7f6320e76740 1 -- 172.21.15.29:0/2561588794 shutdown_connections 2024-03-16T11:54:56.344 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:56.339+0000 7f6320e76740 1 --2- 172.21.15.29:0/2561588794 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e621bcc820 0x55e621bbca30 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:54:56.344 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:56.339+0000 7f6320e76740 1 --2- 172.21.15.29:0/2561588794 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e621bc3a50 0x55e621bca1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:54:56.345 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:56.339+0000 7f6320e76740 1 --2- 172.21.15.29:0/2561588794 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e621bc3130 0x55e621bd8130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:54:56.345 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:56.339+0000 7f6320e76740 1 -- 172.21.15.29:0/2561588794 >> 172.21.15.29:0/2561588794 conn(0x55e621bbda50 msgr2=0x55e621bbc320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:54:56.345 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:56.339+0000 7f6320e76740 1 -- 172.21.15.29:0/2561588794 shutdown_connections 2024-03-16T11:54:56.345 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:56.339+0000 7f6320e76740 1 -- 172.21.15.29:0/2561588794 wait complete. 2024-03-16T11:54:56.345 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:56.339+0000 7f6320e76740 1 -- start start 2024-03-16T11:54:56.345 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:56.339+0000 7f6320e76740 1 Processor -- start 2024-03-16T11:54:56.345 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:56.339+0000 7f6320e76740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e621bc3130 0x55e621bd4950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:54:56.345 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:56.339+0000 7f6320e76740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e621bc3a50 0x55e621bc61e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:54:56.345 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:56.339+0000 7f6320e76740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e621bcc820 0x55e621bc6720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:54:56.345 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:56.339+0000 7f6320e76740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f63181add40 con 0x55e621bc3a50 2024-03-16T11:54:56.345 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:56.339+0000 7f6320e76740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f63181af1b0 con 0x55e621bc3130 2024-03-16T11:54:56.345 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:56.339+0000 7f6320e76740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f63181ae410 con 0x55e621bcc820 2024-03-16T11:54:56.346 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:56.339+0000 7f6320e76740 10 client.0 _ll_register_callbacks cb 0x7f6310013990 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T11:54:56.346 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:56.340+0000 7f63177fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e621bc3a50 0x55e621bc61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:54:56.346 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:56.340+0000 7f63177fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e621bc3a50 0x55e621bc61e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:38434/0 (socket says 172.21.15.29:38434) 2024-03-16T11:54:56.346 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:56.340+0000 7f63177fe640 1 -- 172.21.15.29:0/3768276816 learned_addr learned my addr 172.21.15.29:0/3768276816 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:54:56.346 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:56.340+0000 7f6317fff640 1 --2- 172.21.15.29:0/3768276816 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e621bc3130 0x55e621bd4950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:54:56.346 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:56.340+0000 7f631c822640 1 --2- 172.21.15.29:0/3768276816 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e621bcc820 0x55e621bc6720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:54:56.346 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:56.340+0000 7f63177fe640 1 -- 172.21.15.29:0/3768276816 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e621bcc820 msgr2=0x55e621bc6720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:54:56.346 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:56.340+0000 7f63177fe640 1 --2- 172.21.15.29:0/3768276816 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e621bcc820 0x55e621bc6720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:54:56.346 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:56.340+0000 7f63177fe640 1 -- 172.21.15.29:0/3768276816 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e621bc3130 msgr2=0x55e621bd4950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:54:56.346 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:56.340+0000 7f63177fe640 1 --2- 172.21.15.29:0/3768276816 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e621bc3130 0x55e621bd4950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:54:56.346 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:56.340+0000 7f63177fe640 1 -- 172.21.15.29:0/3768276816 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e621bc6f50 con 0x55e621bc3a50 2024-03-16T11:54:56.346 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:56.340+0000 7f63177fe640 1 --2- 172.21.15.29:0/3768276816 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e621bc3a50 0x55e621bc61e0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f630c0096c0 tx=0x7f630c02bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85562ec124d89b1a server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:54:56.346 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:56.340+0000 7f62f77fe640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-16T11:54:56.347 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:56.340+0000 7f62f77fe640 1 -- 172.21.15.29:0/3768276816 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f630c004b50 con 0x55e621bc3a50 2024-03-16T11:54:56.347 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:56.340+0000 7f62f77fe640 1 -- 172.21.15.29:0/3768276816 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7f630c004cf0 con 0x55e621bc3a50 2024-03-16T11:54:56.347 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:56.340+0000 7f6320e76740 1 -- 172.21.15.29:0/3768276816 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6310023070 con 0x55e621bc3a50 2024-03-16T11:54:56.347 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:56.340+0000 7f62f77fe640 1 -- 172.21.15.29:0/3768276816 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f630c004f50 con 0x55e621bc3a50 2024-03-16T11:54:56.347 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:56.340+0000 7f6320e76740 10 client.15744 fetch_fsmap learned FSMap version 24 2024-03-16T11:54:56.347 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:56.340+0000 7f6320e76740 1 -- 172.21.15.29:0/3768276816 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55e621c42bc0 con 0x55e621bc3a50 2024-03-16T11:54:56.347 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:56.341+0000 7f62f77fe640 1 -- 172.21.15.29:0/3768276816 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6310023070 con 0x55e621bc3a50 2024-03-16T11:54:56.347 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:56.341+0000 7f62f77fe640 1 -- 172.21.15.29:0/3768276816 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7f630c053070 con 0x55e621bc3a50 2024-03-16T11:54:56.347 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:56.341+0000 7f6320e76740 10 client.15744 fetch_fsmap finished waiting for FSMap version 24 2024-03-16T11:54:56.347 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:56.341+0000 7f6320e76740 10 client.15744 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T11:54:56.347 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:56.341+0000 7f6320e76740 20 client.15744 populate_metadata read hostname 'smithi029' 2024-03-16T11:54:56.347 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:56.341+0000 7f6320e76740 1 --2- 172.21.15.29:0/3768276816 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55e621c43370 0x55e621c45790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:54:56.347 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:56.341+0000 7f6320e76740 4 client.15744 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-16T11:54:56.348 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:56.341+0000 7f6320e76740 1 -- 172.21.15.29:0/3768276816 --> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55e621b2f2d0 con 0x55e621c43370 2024-03-16T11:54:56.348 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:56.341+0000 7f6317fff640 1 --2- 172.21.15.29:0/3768276816 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55e621c43370 0x55e621c45790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:54:56.348 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:56.341+0000 7f62f77fe640 10 client.15744 ms_handle_connect on v2:172.21.15.152:6838/1548731833 2024-03-16T11:54:56.348 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:56.341+0000 7f6317fff640 1 --2- 172.21.15.29:0/3768276816 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55e621c43370 0x55e621c45790 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3afe007507a8c0a1 server_cookie=70032f50822399e9 in_seq=0 out_seq=0 2024-03-16T11:54:56.600 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:56.601+0000 7f62f77fe640 1 -- 172.21.15.29:0/3768276816 <== mds.0 v2:172.21.15.152:6838/1548731833 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55e621b2f2d0 con 0x55e621c43370 2024-03-16T11:54:56.601 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:56.601+0000 7f62f77fe640 10 client.15744 handle_command_reply: tid=0 2024-03-16T11:54:56.601 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:56.601+0000 7f6320e76740 1 client.15744 shutdown 2024-03-16T11:54:56.601 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:56.601+0000 7f6320e76740 1 -- 172.21.15.29:0/3768276816 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e621bc3a50 msgr2=0x55e621bc61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:54:56.601 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:56.601+0000 7f6320e76740 1 --2- 172.21.15.29:0/3768276816 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e621bc3a50 0x55e621bc61e0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f630c0096c0 tx=0x7f630c02bc30 comp rx=0 tx=0).stop 2024-03-16T11:54:56.601 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:56.601+0000 7f6320e76740 1 -- 172.21.15.29:0/3768276816 shutdown_connections 2024-03-16T11:54:56.601 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:56.601+0000 7f6320e76740 1 --2- 172.21.15.29:0/3768276816 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e621bcc820 0x55e621bc6720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:54:56.601 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:56.601+0000 7f6320e76740 1 --2- 172.21.15.29:0/3768276816 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e621bc3a50 0x55e621bc61e0 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:54:56.601 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:56.601+0000 7f6320e76740 1 --2- 172.21.15.29:0/3768276816 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55e621c43370 0x55e621c45790 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:54:56.601 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:56.601+0000 7f62f77fe640 0 client.15744 ms_handle_reset on v2:172.21.15.152:6838/1548731833 2024-03-16T11:54:56.601 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:56.601+0000 7f6320e76740 1 --2- 172.21.15.29:0/3768276816 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e621bc3130 0x55e621bd4950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:54:56.601 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:56.601+0000 7f6320e76740 1 -- 172.21.15.29:0/3768276816 >> 172.21.15.29:0/3768276816 conn(0x55e621bbda50 msgr2=0x55e621bbcbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:54:56.601 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:56.601+0000 7f6320e76740 1 -- 172.21.15.29:0/3768276816 shutdown_connections 2024-03-16T11:54:56.601 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:56.601+0000 7f6320e76740 1 -- 172.21.15.29:0/3768276816 wait complete. 2024-03-16T11:54:56.602 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:56.602+0000 7f6320e76740 20 client.15744 trim_cache size 0 max 16384 2024-03-16T11:54:56.602 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:{ 2024-03-16T11:54:56.602 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "message": "", 2024-03-16T11:54:56.602 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "return_code": 0 2024-03-16T11:54:56.602 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:} 2024-03-16T11:54:56.603 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:56.602+0000 7f631e285640 1 -- 172.21.15.29:0/180226162 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f63000a2a10 msgr2=0x7f63000a4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:54:56.603 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:56.602+0000 7f631e285640 1 --2- 172.21.15.29:0/180226162 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f63000a2a10 0x7f63000a4ed0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f630c005ab0 tx=0x7f630c005fd0 comp rx=0 tx=0).stop 2024-03-16T11:54:56.603 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:56.602+0000 7f631e285640 1 -- 172.21.15.29:0/180226162 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f63180ffce0 msgr2=0x7f6318100ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:54:56.603 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:56.602+0000 7f631e285640 1 --2- 172.21.15.29:0/180226162 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f63180ffce0 0x7f6318100ec0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f630800e9f0 tx=0x7f630800eeb0 comp rx=0 tx=0).stop 2024-03-16T11:54:56.603 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:56.602+0000 7f631e285640 1 -- 172.21.15.29:0/180226162 shutdown_connections 2024-03-16T11:54:56.603 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:56.603+0000 7f631e285640 1 --2- 172.21.15.29:0/180226162 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f63000a2a10 0x7f63000a4ed0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:54:56.603 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:56.603+0000 7f631e285640 1 --2- 172.21.15.29:0/180226162 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f63181067c0 0x7f6318101940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:54:56.603 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:56.603+0000 7f631e285640 1 --2- 172.21.15.29:0/180226162 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f63181006a0 0x7f6318101400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:54:56.603 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:56.603+0000 7f631e285640 1 --2- 172.21.15.29:0/180226162 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f63180ffce0 0x7f6318100ec0 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:54:56.603 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:56.603+0000 7f631e285640 1 -- 172.21.15.29:0/180226162 >> 172.21.15.29:0/180226162 conn(0x7f6318076c70 msgr2=0x7f63181046c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:54:56.603 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:56.603+0000 7f631e285640 1 -- 172.21.15.29:0/180226162 shutdown_connections 2024-03-16T11:54:56.604 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:54:56.603+0000 7f631e285640 1 -- 172.21.15.29:0/180226162 wait complete. 2024-03-16T11:54:56.782 INFO:tasks.workunit.client.0.smithi029.stdout: 1 359318 11.54 MB/sec execute 41 sec latency 77.317 ms 2024-03-16T11:54:57.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:54:56 smithi154 ceph-mon[36320]: pgmap v124: 97 pgs: 97 active+clean; 50 MiB data, 948 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 2.9 MiB/s wr, 320 op/s 2024-03-16T11:54:57.087 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:54:56 smithi029 ceph-mon[31975]: pgmap v124: 97 pgs: 97 active+clean; 50 MiB data, 948 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 2.9 MiB/s wr, 320 op/s 2024-03-16T11:54:57.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:54:56 smithi152 ceph-mon[36291]: pgmap v124: 97 pgs: 97 active+clean; 50 MiB data, 948 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 2.9 MiB/s wr, 320 op/s 2024-03-16T11:54:57.782 INFO:tasks.workunit.client.0.smithi029.stdout: 1 362072 11.66 MB/sec execute 42 sec latency 22.684 ms 2024-03-16T11:54:58.782 INFO:tasks.workunit.client.0.smithi029.stdout: 1 364277 11.58 MB/sec execute 43 sec latency 21.968 ms 2024-03-16T11:54:59.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:54:58 smithi154 ceph-mon[36320]: pgmap v125: 97 pgs: 97 active+clean; 55 MiB data, 998 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 4.4 MiB/s wr, 426 op/s 2024-03-16T11:54:59.087 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:54:58 smithi029 ceph-mon[31975]: pgmap v125: 97 pgs: 97 active+clean; 55 MiB data, 998 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 4.4 MiB/s wr, 426 op/s 2024-03-16T11:54:59.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:54:58 smithi152 ceph-mon[36291]: pgmap v125: 97 pgs: 97 active+clean; 55 MiB data, 998 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 4.4 MiB/s wr, 426 op/s 2024-03-16T11:54:59.782 INFO:tasks.workunit.client.0.smithi029.stdout: 1 366223 11.59 MB/sec execute 44 sec latency 26.334 ms 2024-03-16T11:55:00.782 INFO:tasks.workunit.client.0.smithi029.stdout: 1 368581 11.62 MB/sec execute 45 sec latency 97.761 ms 2024-03-16T11:55:01.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:55:00 smithi154 ceph-mon[36320]: pgmap v126: 97 pgs: 97 active+clean; 55 MiB data, 998 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 2.8 MiB/s wr, 209 op/s 2024-03-16T11:55:01.087 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:55:00 smithi029 ceph-mon[31975]: pgmap v126: 97 pgs: 97 active+clean; 55 MiB data, 998 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 2.8 MiB/s wr, 209 op/s 2024-03-16T11:55:01.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:55:00 smithi152 ceph-mon[36291]: pgmap v126: 97 pgs: 97 active+clean; 55 MiB data, 998 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 2.8 MiB/s wr, 209 op/s 2024-03-16T11:55:01.782 INFO:tasks.workunit.client.0.smithi029.stdout: 1 369605 11.49 MB/sec execute 46 sec latency 43.857 ms 2024-03-16T11:55:02.782 INFO:tasks.workunit.client.0.smithi029.stdout: 1 372501 11.62 MB/sec execute 47 sec latency 10.985 ms 2024-03-16T11:55:03.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:55:02 smithi154 ceph-mon[36320]: pgmap v127: 97 pgs: 97 active+clean; 43 MiB data, 976 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 4.3 MiB/s wr, 447 op/s 2024-03-16T11:55:03.087 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:55:02 smithi029 ceph-mon[31975]: pgmap v127: 97 pgs: 97 active+clean; 43 MiB data, 976 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 4.3 MiB/s wr, 447 op/s 2024-03-16T11:55:03.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:55:02 smithi152 ceph-mon[36291]: pgmap v127: 97 pgs: 97 active+clean; 43 MiB data, 976 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 4.3 MiB/s wr, 447 op/s 2024-03-16T11:55:03.782 INFO:tasks.workunit.client.0.smithi029.stdout: 1 374099 11.52 MB/sec execute 48 sec latency 22.999 ms 2024-03-16T11:55:04.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:55:03 smithi154 ceph-mon[36320]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T11:55:04.087 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:55:03 smithi029 ceph-mon[31975]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T11:55:04.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:55:03 smithi152 ceph-mon[36291]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T11:55:04.782 INFO:tasks.workunit.client.0.smithi029.stdout: 1 376401 11.56 MB/sec execute 49 sec latency 12.483 ms 2024-03-16T11:55:05.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:55:04 smithi154 ceph-mon[36320]: pgmap v128: 97 pgs: 97 active+clean; 43 MiB data, 976 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 2.9 MiB/s wr, 343 op/s 2024-03-16T11:55:05.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:55:04 smithi154 ceph-mon[36320]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:55:05.046 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:55:04 smithi154 ceph-mon[36320]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T11:55:05.046 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:55:04 smithi154 ceph-mon[36320]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' 2024-03-16T11:55:05.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:55:04 smithi029 ceph-mon[31975]: pgmap v128: 97 pgs: 97 active+clean; 43 MiB data, 976 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 2.9 MiB/s wr, 343 op/s 2024-03-16T11:55:05.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:55:04 smithi029 ceph-mon[31975]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:55:05.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:55:04 smithi029 ceph-mon[31975]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T11:55:05.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:55:04 smithi029 ceph-mon[31975]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' 2024-03-16T11:55:05.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:55:04 smithi152 ceph-mon[36291]: pgmap v128: 97 pgs: 97 active+clean; 43 MiB data, 976 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 2.9 MiB/s wr, 343 op/s 2024-03-16T11:55:05.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:55:04 smithi152 ceph-mon[36291]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:55:05.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:55:04 smithi152 ceph-mon[36291]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T11:55:05.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:55:04 smithi152 ceph-mon[36291]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' 2024-03-16T11:55:05.782 INFO:tasks.workunit.client.0.smithi029.stdout: 1 378421 11.48 MB/sec execute 50 sec latency 81.477 ms 2024-03-16T11:55:06.782 INFO:tasks.workunit.client.0.smithi029.stdout: 1 380239 11.49 MB/sec execute 51 sec latency 20.153 ms 2024-03-16T11:55:07.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:55:06 smithi154 ceph-mon[36320]: pgmap v129: 97 pgs: 97 active+clean; 43 MiB data, 976 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 2.9 MiB/s wr, 343 op/s 2024-03-16T11:55:07.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:55:06 smithi029 ceph-mon[31975]: pgmap v129: 97 pgs: 97 active+clean; 43 MiB data, 976 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 2.9 MiB/s wr, 343 op/s 2024-03-16T11:55:07.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:55:06 smithi152 ceph-mon[36291]: pgmap v129: 97 pgs: 97 active+clean; 43 MiB data, 976 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 2.9 MiB/s wr, 343 op/s 2024-03-16T11:55:07.782 INFO:tasks.workunit.client.0.smithi029.stdout: 1 383121 11.60 MB/sec execute 52 sec latency 9.378 ms 2024-03-16T11:55:08.782 INFO:tasks.workunit.client.0.smithi029.stdout: 1 385021 11.52 MB/sec execute 53 sec latency 19.944 ms 2024-03-16T11:55:09.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:55:08 smithi154 ceph-mon[36320]: pgmap v130: 97 pgs: 97 active+clean; 49 MiB data, 1000 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 4.3 MiB/s wr, 448 op/s 2024-03-16T11:55:09.087 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:55:08 smithi029 ceph-mon[31975]: pgmap v130: 97 pgs: 97 active+clean; 49 MiB data, 1000 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 4.3 MiB/s wr, 448 op/s 2024-03-16T11:55:09.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:55:08 smithi152 ceph-mon[36291]: pgmap v130: 97 pgs: 97 active+clean; 49 MiB data, 1000 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 4.3 MiB/s wr, 448 op/s 2024-03-16T11:55:09.697 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:09.696+0000 7f3e2be63640 1 -- 172.21.15.29:0/945754165 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3e241066d0 msgr2=0x7f3e24106ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:55:09.698 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:09.696+0000 7f3e2be63640 1 --2- 172.21.15.29:0/945754165 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3e241066d0 0x7f3e24106ab0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f3e20009930 tx=0x7f3e2002bc30 comp rx=0 tx=0).stop 2024-03-16T11:55:09.698 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:09.696+0000 7f3e2be63640 1 -- 172.21.15.29:0/945754165 shutdown_connections 2024-03-16T11:55:09.698 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:09.696+0000 7f3e2be63640 1 --2- 172.21.15.29:0/945754165 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3e24101080 0x7f3e2410d590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:55:09.698 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:09.696+0000 7f3e2be63640 1 --2- 172.21.15.29:0/945754165 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3e241006c0 0x7f3e24100b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:55:09.698 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:09.696+0000 7f3e2be63640 1 --2- 172.21.15.29:0/945754165 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3e241066d0 0x7f3e24106ab0 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:55:09.698 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:09.696+0000 7f3e2be63640 1 -- 172.21.15.29:0/945754165 >> 172.21.15.29:0/945754165 conn(0x7f3e24077830 msgr2=0x7f3e24077c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:55:09.698 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:09.696+0000 7f3e2be63640 1 -- 172.21.15.29:0/945754165 shutdown_connections 2024-03-16T11:55:09.698 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:09.696+0000 7f3e2be63640 1 -- 172.21.15.29:0/945754165 wait complete. 2024-03-16T11:55:09.698 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:09.697+0000 7f3e2be63640 1 Processor -- start 2024-03-16T11:55:09.698 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:09.697+0000 7f3e2be63640 1 -- start start 2024-03-16T11:55:09.698 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:09.697+0000 7f3e2be63640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3e241006c0 0x7f3e24198ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:55:09.699 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:09.697+0000 7f3e2be63640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3e24101080 0x7f3e24199400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:55:09.699 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:09.697+0000 7f3e2be63640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3e241066d0 0x7f3e241a3f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:55:09.699 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:09.697+0000 7f3e2be63640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f3e24110ae0 con 0x7f3e24101080 2024-03-16T11:55:09.699 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:09.697+0000 7f3e2be63640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f3e24110960 con 0x7f3e241006c0 2024-03-16T11:55:09.699 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:09.697+0000 7f3e2be63640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f3e24110c60 con 0x7f3e241066d0 2024-03-16T11:55:09.699 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:09.697+0000 7f3e293fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3e24101080 0x7f3e24199400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:55:09.699 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:09.697+0000 7f3e293fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3e24101080 0x7f3e24199400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43036/0 (socket says 172.21.15.29:43036) 2024-03-16T11:55:09.699 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:09.697+0000 7f3e293fe640 1 -- 172.21.15.29:0/4259845438 learned_addr learned my addr 172.21.15.29:0/4259845438 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:55:09.699 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:09.697+0000 7f3e29bff640 1 --2- 172.21.15.29:0/4259845438 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3e241006c0 0x7f3e24198ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:55:09.699 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:09.697+0000 7f3e2a400640 1 --2- 172.21.15.29:0/4259845438 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3e241066d0 0x7f3e241a3f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:55:09.699 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:09.698+0000 7f3e293fe640 1 -- 172.21.15.29:0/4259845438 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3e241066d0 msgr2=0x7f3e241a3f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:55:09.699 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:09.698+0000 7f3e293fe640 1 --2- 172.21.15.29:0/4259845438 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3e241066d0 0x7f3e241a3f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:55:09.699 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:09.698+0000 7f3e293fe640 1 -- 172.21.15.29:0/4259845438 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3e241006c0 msgr2=0x7f3e24198ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:55:09.699 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:09.698+0000 7f3e293fe640 1 --2- 172.21.15.29:0/4259845438 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3e241006c0 0x7f3e24198ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:55:09.699 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:09.698+0000 7f3e293fe640 1 -- 172.21.15.29:0/4259845438 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e241a4480 con 0x7f3e24101080 2024-03-16T11:55:09.700 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:09.698+0000 7f3e293fe640 1 --2- 172.21.15.29:0/4259845438 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3e24101080 0x7f3e24199400 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f3e1400e9f0 tx=0x7f3e1400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:55:09.700 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:09.698+0000 7f3e1affd640 1 -- 172.21.15.29:0/4259845438 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e1400cd80 con 0x7f3e24101080 2024-03-16T11:55:09.700 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:09.698+0000 7f3e1affd640 1 -- 172.21.15.29:0/4259845438 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3e14010910 con 0x7f3e24101080 2024-03-16T11:55:09.700 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:09.698+0000 7f3e2be63640 1 -- 172.21.15.29:0/4259845438 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3e241a4650 con 0x7f3e24101080 2024-03-16T11:55:09.700 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:09.698+0000 7f3e1affd640 1 -- 172.21.15.29:0/4259845438 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e14018990 con 0x7f3e24101080 2024-03-16T11:55:09.700 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:09.698+0000 7f3e2be63640 1 -- 172.21.15.29:0/4259845438 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3e241a4b40 con 0x7f3e24101080 2024-03-16T11:55:09.703 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:09.701+0000 7f3e2ea48740 1 Processor -- start 2024-03-16T11:55:09.703 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:09.701+0000 7f3e2ea48740 1 -- start start 2024-03-16T11:55:09.703 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:09.701+0000 7f3e1affd640 1 -- 172.21.15.29:0/4259845438 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7f3e14018b30 con 0x7f3e24101080 2024-03-16T11:55:09.703 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:09.702+0000 7f3e2ea48740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56520096b130 0x56520096b510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:55:09.703 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:09.702+0000 7f3e2ea48740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56520096ba50 0x5652009742e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:55:09.704 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:09.702+0000 7f3e2ea48740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x565200974820 0x565200976c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:55:09.704 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:09.702+0000 7f3e2ea48740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x5652008cfa40 con 0x56520096b130 2024-03-16T11:55:09.704 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:09.702+0000 7f3e2ea48740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x565200878920 con 0x56520096ba50 2024-03-16T11:55:09.704 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:09.702+0000 7f3e2ea48740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5652008cbae0 con 0x565200974820 2024-03-16T11:55:09.704 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:09.702+0000 7f3e2a400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56520096b130 0x56520096b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:55:09.704 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:09.702+0000 7f3e2a400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56520096b130 0x56520096b510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43048/0 (socket says 172.21.15.29:43048) 2024-03-16T11:55:09.704 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:09.702+0000 7f3e2a400640 1 -- 172.21.15.29:0/4246596463 learned_addr learned my addr 172.21.15.29:0/4246596463 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:55:09.704 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:09.702+0000 7f3e293fe640 1 --2- 172.21.15.29:0/4246596463 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x565200974820 0x565200976c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:55:09.704 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:09.702+0000 7f3e29bff640 1 --2- 172.21.15.29:0/4246596463 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56520096ba50 0x5652009742e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:55:09.704 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:09.702+0000 7f3e1affd640 1 --2- 172.21.15.29:0/4259845438 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f3e100a2a10 0x7f3e100a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:55:09.705 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:09.702+0000 7f3e2a400640 1 -- 172.21.15.29:0/4246596463 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x565200974820 msgr2=0x565200976c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:55:09.705 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:09.702+0000 7f3e2a400640 1 --2- 172.21.15.29:0/4246596463 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x565200974820 0x565200976c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:55:09.705 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:09.702+0000 7f3e2a400640 1 -- 172.21.15.29:0/4246596463 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56520096ba50 msgr2=0x5652009742e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:55:09.705 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:09.702+0000 7f3e2a400640 1 --2- 172.21.15.29:0/4246596463 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56520096ba50 0x5652009742e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:55:09.705 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:09.702+0000 7f3e2a400640 1 -- 172.21.15.29:0/4246596463 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x565200890460 con 0x56520096b130 2024-03-16T11:55:09.706 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:09.702+0000 7f3e293fe640 1 --2- 172.21.15.29:0/4246596463 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x565200974820 0x565200976c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:55:09.706 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:09.702+0000 7f3e1affd640 1 -- 172.21.15.29:0/4259845438 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7f3e140cf120 con 0x7f3e24101080 2024-03-16T11:55:09.706 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:09.702+0000 7f3e2a400640 1 --2- 172.21.15.29:0/4259845438 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f3e100a2a10 0x7f3e100a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:55:09.706 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:09.702+0000 7f3e2a400640 1 --2- 172.21.15.29:0/4246596463 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56520096b130 0x56520096b510 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f3e1c009f50 tx=0x7f3e1c0118d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82507579d76172d5 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:55:09.706 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:09.703+0000 7f3e0ffff640 1 -- 172.21.15.29:0/4246596463 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e1c004250 con 0x56520096b130 2024-03-16T11:55:09.706 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:09.703+0000 7f3e0ffff640 1 -- 172.21.15.29:0/4246596463 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3e1c00fe80 con 0x56520096b130 2024-03-16T11:55:09.706 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:09.703+0000 7f3e0ffff640 1 -- 172.21.15.29:0/4246596463 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e1c01bb20 con 0x56520096b130 2024-03-16T11:55:09.706 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:09.703+0000 7f3e2ea48740 1 -- 172.21.15.29:0/4246596463 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56520096b130 msgr2=0x56520096b510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:55:09.706 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:09.703+0000 7f3e2a400640 1 --2- 172.21.15.29:0/4259845438 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f3e100a2a10 0x7f3e100a4ed0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f3e1c012fd0 tx=0x7f3e1c021000 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:55:09.706 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:09.703+0000 7f3e2ea48740 1 --2- 172.21.15.29:0/4246596463 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56520096b130 0x56520096b510 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f3e1c009f50 tx=0x7f3e1c0118d0 comp rx=0 tx=0).stop 2024-03-16T11:55:09.706 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:09.703+0000 7f3e2ea48740 1 -- 172.21.15.29:0/4246596463 shutdown_connections 2024-03-16T11:55:09.706 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:09.704+0000 7f3e2ea48740 1 --2- 172.21.15.29:0/4246596463 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x565200974820 0x565200976c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:55:09.706 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:09.704+0000 7f3e2ea48740 1 --2- 172.21.15.29:0/4246596463 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56520096ba50 0x5652009742e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:55:09.706 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:09.704+0000 7f3e2ea48740 1 --2- 172.21.15.29:0/4246596463 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56520096b130 0x56520096b510 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:55:09.707 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:09.704+0000 7f3e2ea48740 1 -- 172.21.15.29:0/4246596463 >> 172.21.15.29:0/4246596463 conn(0x565200965a50 msgr2=0x565200965e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:55:09.707 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:09.704+0000 7f3e2ea48740 1 -- 172.21.15.29:0/4246596463 shutdown_connections 2024-03-16T11:55:09.707 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:09.704+0000 7f3e2ea48740 1 -- 172.21.15.29:0/4246596463 wait complete. 2024-03-16T11:55:09.707 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:09.704+0000 7f3e2ea48740 1 -- start start 2024-03-16T11:55:09.707 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:09.704+0000 7f3e2ea48740 1 Processor -- start 2024-03-16T11:55:09.711 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:09.705+0000 7f3e2ea48740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56520096b130 0x5652009718b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:55:09.711 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:09.705+0000 7f3e2ea48740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56520096ba50 0x565200971df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:55:09.711 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:09.705+0000 7f3e2ea48740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x565200974820 0x565200972330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:55:09.711 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:09.705+0000 7f3e2ea48740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x565200977450 con 0x56520096ba50 2024-03-16T11:55:09.711 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:09.705+0000 7f3e2ea48740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x56520088bd70 con 0x56520096b130 2024-03-16T11:55:09.711 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:09.705+0000 7f3e2ea48740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5652009775d0 con 0x565200974820 2024-03-16T11:55:09.711 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:09.705+0000 7f3e2ea48740 10 client.0 _ll_register_callbacks cb 0x5652005e3a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T11:55:09.711 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:09.705+0000 7f3e293fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56520096ba50 0x565200971df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:55:09.711 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:09.705+0000 7f3e293fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56520096ba50 0x565200971df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43054/0 (socket says 172.21.15.29:43054) 2024-03-16T11:55:09.711 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:09.705+0000 7f3e29bff640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56520096b130 0x5652009718b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:55:09.711 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:09.705+0000 7f3e293fe640 1 -- 172.21.15.29:0/2889965294 learned_addr learned my addr 172.21.15.29:0/2889965294 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:55:09.711 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:09.705+0000 7f3e2a400640 1 --2- 172.21.15.29:0/2889965294 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x565200974820 0x565200972330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:55:09.711 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:09.705+0000 7f3e293fe640 1 -- 172.21.15.29:0/2889965294 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x565200974820 msgr2=0x565200972330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:55:09.711 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:09.705+0000 7f3e293fe640 1 --2- 172.21.15.29:0/2889965294 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x565200974820 0x565200972330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:55:09.712 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:09.705+0000 7f3e293fe640 1 -- 172.21.15.29:0/2889965294 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56520096b130 msgr2=0x5652009718b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:55:09.712 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:09.705+0000 7f3e293fe640 1 --2- 172.21.15.29:0/2889965294 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56520096b130 0x5652009718b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:55:09.712 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:09.705+0000 7f3e293fe640 1 -- 172.21.15.29:0/2889965294 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5652005e4210 con 0x56520096ba50 2024-03-16T11:55:09.712 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:09.705+0000 7f3e293fe640 1 --2- 172.21.15.29:0/2889965294 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56520096ba50 0x565200971df0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f3e1400c6a0 tx=0x7f3e14009310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=896a6bd52df38311 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:55:09.712 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:09.705+0000 7f3e0d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-16T11:55:09.712 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:09.705+0000 7f3e0d7fa640 1 -- 172.21.15.29:0/2889965294 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e1409b3a0 con 0x56520096ba50 2024-03-16T11:55:09.712 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:09.705+0000 7f3e0d7fa640 1 -- 172.21.15.29:0/2889965294 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7f3e14097ed0 con 0x56520096ba50 2024-03-16T11:55:09.712 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:09.705+0000 7f3e2ea48740 1 -- 172.21.15.29:0/2889965294 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5652008dcf80 con 0x56520096ba50 2024-03-16T11:55:09.712 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:09.706+0000 7f3e0d7fa640 1 -- 172.21.15.29:0/2889965294 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f3e1409b1c0 con 0x56520096ba50 2024-03-16T11:55:09.712 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:09.706+0000 7f3e2ea48740 10 client.15756 fetch_fsmap learned FSMap version 24 2024-03-16T11:55:09.712 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:09.706+0000 7f3e2ea48740 1 -- 172.21.15.29:0/2889965294 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x5652006b0620 con 0x56520096ba50 2024-03-16T11:55:09.712 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:09.706+0000 7f3e0d7fa640 1 -- 172.21.15.29:0/2889965294 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x5652008dcf80 con 0x56520096ba50 2024-03-16T11:55:09.712 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:09.706+0000 7f3e0d7fa640 1 -- 172.21.15.29:0/2889965294 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7f3e140afb90 con 0x56520096ba50 2024-03-16T11:55:09.712 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:09.706+0000 7f3e2ea48740 10 client.15756 fetch_fsmap finished waiting for FSMap version 24 2024-03-16T11:55:09.712 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:09.706+0000 7f3e2ea48740 10 client.15756 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T11:55:09.713 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:09.706+0000 7f3e2ea48740 20 client.15756 populate_metadata read hostname 'smithi029' 2024-03-16T11:55:09.713 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:09.706+0000 7f3e2ea48740 1 --2- 172.21.15.29:0/2889965294 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x565200984a00 0x565200986df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:55:09.713 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:09.706+0000 7f3e2ea48740 4 client.15756 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T11:55:09.713 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:09.706+0000 7f3e2ea48740 1 -- 172.21.15.29:0/2889965294 --> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5652008d72d0 con 0x565200984a00 2024-03-16T11:55:09.713 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:09.706+0000 7f3e29bff640 1 --2- 172.21.15.29:0/2889965294 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x565200984a00 0x565200986df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:55:09.713 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:09.707+0000 7f3e0d7fa640 10 client.15756 ms_handle_connect on v2:172.21.15.152:6838/1548731833 2024-03-16T11:55:09.713 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:09.707+0000 7f3e29bff640 1 --2- 172.21.15.29:0/2889965294 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x565200984a00 0x565200986df0 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=68e623eb96e9986b server_cookie=2ad3fb5edae8f5b1 in_seq=0 out_seq=0 2024-03-16T11:55:09.713 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:09.708+0000 7f3e0d7fa640 1 -- 172.21.15.29:0/2889965294 <== mds.0 v2:172.21.15.152:6838/1548731833 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5652008d72d0 con 0x565200984a00 2024-03-16T11:55:09.713 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:09.708+0000 7f3e0d7fa640 10 client.15756 handle_command_reply: tid=0 2024-03-16T11:55:09.713 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:09.708+0000 7f3e2ea48740 1 client.15756 shutdown 2024-03-16T11:55:09.713 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:09.708+0000 7f3e2ea48740 1 -- 172.21.15.29:0/2889965294 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56520096ba50 msgr2=0x565200971df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:55:09.713 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:09.708+0000 7f3e2ea48740 1 --2- 172.21.15.29:0/2889965294 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56520096ba50 0x565200971df0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f3e1400c6a0 tx=0x7f3e14009310 comp rx=0 tx=0).stop 2024-03-16T11:55:09.713 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:09.708+0000 7f3e2ea48740 1 -- 172.21.15.29:0/2889965294 shutdown_connections 2024-03-16T11:55:09.713 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:09.708+0000 7f3e2ea48740 1 --2- 172.21.15.29:0/2889965294 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x565200974820 0x565200972330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:55:09.714 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:09.708+0000 7f3e2ea48740 1 --2- 172.21.15.29:0/2889965294 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56520096ba50 0x565200971df0 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:55:09.714 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:09.708+0000 7f3e2ea48740 1 --2- 172.21.15.29:0/2889965294 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x565200984a00 0x565200986df0 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:55:09.714 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:09.708+0000 7f3e2ea48740 1 --2- 172.21.15.29:0/2889965294 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56520096b130 0x5652009718b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:55:09.714 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:09.708+0000 7f3e2ea48740 1 -- 172.21.15.29:0/2889965294 >> 172.21.15.29:0/2889965294 conn(0x565200965a50 msgr2=0x5652009677a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:55:09.714 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:09.708+0000 7f3e0d7fa640 0 client.15756 ms_handle_reset on v2:172.21.15.152:6838/1548731833 2024-03-16T11:55:09.714 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:09.708+0000 7f3e2ea48740 1 -- 172.21.15.29:0/2889965294 shutdown_connections 2024-03-16T11:55:09.714 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:09.708+0000 7f3e2ea48740 1 -- 172.21.15.29:0/2889965294 wait complete. 2024-03-16T11:55:09.714 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:09.709+0000 7f3e2ea48740 20 client.15756 trim_cache size 0 max 16384 2024-03-16T11:55:09.731 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:09.726+0000 7f3e2ea48740 1 Processor -- start 2024-03-16T11:55:09.731 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:09.726+0000 7f3e2ea48740 1 -- start start 2024-03-16T11:55:09.731 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:09.726+0000 7f3e2ea48740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56520096b130 0x565200973fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:55:09.731 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:09.726+0000 7f3e2ea48740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56520096ba50 0x5652009801f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:55:09.731 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:09.726+0000 7f3e2ea48740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x565200974820 0x56520098c700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:55:09.731 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:09.726+0000 7f3e2ea48740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f3e241afd10 con 0x56520096ba50 2024-03-16T11:55:09.731 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:09.726+0000 7f3e2ea48740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f3e241ad5e0 con 0x565200974820 2024-03-16T11:55:09.731 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:09.726+0000 7f3e2ea48740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f3e241ab430 con 0x56520096b130 2024-03-16T11:55:09.732 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:09.726+0000 7f3e293fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56520096ba50 0x5652009801f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:55:09.732 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:09.726+0000 7f3e293fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56520096ba50 0x5652009801f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43056/0 (socket says 172.21.15.29:43056) 2024-03-16T11:55:09.732 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:09.726+0000 7f3e293fe640 1 -- 172.21.15.29:0/2235504312 learned_addr learned my addr 172.21.15.29:0/2235504312 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:55:09.732 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:09.726+0000 7f3e29bff640 1 --2- 172.21.15.29:0/2235504312 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56520096b130 0x565200973fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:55:09.732 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:09.726+0000 7f3e2a400640 1 --2- 172.21.15.29:0/2235504312 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x565200974820 0x56520098c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:55:09.732 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:09.726+0000 7f3e293fe640 1 -- 172.21.15.29:0/2235504312 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56520096b130 msgr2=0x565200973fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:55:09.732 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:09.726+0000 7f3e293fe640 1 --2- 172.21.15.29:0/2235504312 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56520096b130 0x565200973fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:55:09.732 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:09.726+0000 7f3e293fe640 1 -- 172.21.15.29:0/2235504312 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x565200974820 msgr2=0x56520098c700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:55:09.732 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:09.726+0000 7f3e293fe640 1 --2- 172.21.15.29:0/2235504312 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x565200974820 0x56520098c700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:55:09.732 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:09.726+0000 7f3e293fe640 1 -- 172.21.15.29:0/2235504312 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56520095ffc0 con 0x56520096ba50 2024-03-16T11:55:09.732 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:09.727+0000 7f3e2a400640 1 --2- 172.21.15.29:0/2235504312 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x565200974820 0x56520098c700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T11:55:09.732 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:09.727+0000 7f3e293fe640 1 --2- 172.21.15.29:0/2235504312 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56520096ba50 0x5652009801f0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f3e1400e9c0 tx=0x7f3e140cf4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e65580dd55ac1548 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:55:09.733 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:09.727+0000 7f3e0f7fe640 1 -- 172.21.15.29:0/2235504312 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e1409afb0 con 0x56520096ba50 2024-03-16T11:55:09.733 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:09.727+0000 7f3e0f7fe640 1 -- 172.21.15.29:0/2235504312 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3e140d2960 con 0x56520096ba50 2024-03-16T11:55:09.733 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:09.727+0000 7f3e0f7fe640 1 -- 172.21.15.29:0/2235504312 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e140afb90 con 0x56520096ba50 2024-03-16T11:55:09.733 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:09.727+0000 7f3e2ea48740 1 -- 172.21.15.29:0/2235504312 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56520096ba50 msgr2=0x5652009801f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:55:09.733 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:09.727+0000 7f3e2ea48740 1 --2- 172.21.15.29:0/2235504312 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56520096ba50 0x5652009801f0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f3e1400e9c0 tx=0x7f3e140cf4b0 comp rx=0 tx=0).stop 2024-03-16T11:55:09.733 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:09.727+0000 7f3e2ea48740 1 -- 172.21.15.29:0/2235504312 shutdown_connections 2024-03-16T11:55:09.733 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:09.727+0000 7f3e2ea48740 1 --2- 172.21.15.29:0/2235504312 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x565200974820 0x56520098c700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:55:09.733 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:09.727+0000 7f3e2ea48740 1 --2- 172.21.15.29:0/2235504312 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56520096ba50 0x5652009801f0 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:55:09.733 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:09.727+0000 7f3e2ea48740 1 --2- 172.21.15.29:0/2235504312 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56520096b130 0x565200973fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:55:09.733 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:09.727+0000 7f3e2ea48740 1 -- 172.21.15.29:0/2235504312 >> 172.21.15.29:0/2235504312 conn(0x565200965a50 msgr2=0x56520096cba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:55:09.733 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:09.728+0000 7f3e2ea48740 1 -- 172.21.15.29:0/2235504312 shutdown_connections 2024-03-16T11:55:09.733 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:09.728+0000 7f3e2ea48740 1 -- 172.21.15.29:0/2235504312 wait complete. 2024-03-16T11:55:09.734 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:09.728+0000 7f3e2ea48740 1 -- start start 2024-03-16T11:55:09.734 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:09.728+0000 7f3e2ea48740 1 Processor -- start 2024-03-16T11:55:09.734 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:09.728+0000 7f3e2ea48740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56520096b130 0x565200978080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:55:09.734 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:09.728+0000 7f3e2ea48740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56520096ba50 0x5652009785c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:55:09.734 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:09.728+0000 7f3e2ea48740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x565200974820 0x565200978b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:55:09.734 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:09.728+0000 7f3e2ea48740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f3e241abaa0 con 0x56520096b130 2024-03-16T11:55:09.734 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:09.728+0000 7f3e2ea48740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f3e241acf10 con 0x565200974820 2024-03-16T11:55:09.734 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:09.728+0000 7f3e2ea48740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f3e241ac170 con 0x56520096ba50 2024-03-16T11:55:09.734 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:09.728+0000 7f3e2ea48740 10 client.0 _ll_register_callbacks cb 0x7f3e1400cf20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T11:55:09.734 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:09.728+0000 7f3e29bff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56520096b130 0x565200978080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:55:09.734 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:09.728+0000 7f3e29bff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56520096b130 0x565200978080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43068/0 (socket says 172.21.15.29:43068) 2024-03-16T11:55:09.734 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:09.729+0000 7f3e29bff640 1 -- 172.21.15.29:0/1295074916 learned_addr learned my addr 172.21.15.29:0/1295074916 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:55:09.735 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:09.729+0000 7f3e293fe640 1 --2- 172.21.15.29:0/1295074916 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56520096ba50 0x5652009785c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:55:09.735 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:09.729+0000 7f3e2a400640 1 --2- 172.21.15.29:0/1295074916 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x565200974820 0x565200978b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:55:09.735 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:09.729+0000 7f3e29bff640 1 -- 172.21.15.29:0/1295074916 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56520096ba50 msgr2=0x5652009785c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:55:09.735 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:09.729+0000 7f3e29bff640 1 --2- 172.21.15.29:0/1295074916 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56520096ba50 0x5652009785c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:55:09.735 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:09.729+0000 7f3e29bff640 1 -- 172.21.15.29:0/1295074916 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x565200974820 msgr2=0x565200978b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:55:09.735 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:09.729+0000 7f3e29bff640 1 --2- 172.21.15.29:0/1295074916 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x565200974820 0x565200978b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:55:09.735 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:09.729+0000 7f3e29bff640 1 -- 172.21.15.29:0/1295074916 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5652009e9d30 con 0x56520096b130 2024-03-16T11:55:09.735 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:09.729+0000 7f3e29bff640 1 --2- 172.21.15.29:0/1295074916 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56520096b130 0x565200978080 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f3e2002bc70 tx=0x7f3e2002c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b000511b099b87e server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:55:09.735 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:09.729+0000 7f3e0d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-16T11:55:09.735 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:09.729+0000 7f3e0d7fa640 1 -- 172.21.15.29:0/1295074916 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e20049d10 con 0x56520096b130 2024-03-16T11:55:09.735 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:09.729+0000 7f3e0d7fa640 1 -- 172.21.15.29:0/1295074916 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7f3e20004030 con 0x56520096b130 2024-03-16T11:55:09.735 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:09.729+0000 7f3e2ea48740 1 -- 172.21.15.29:0/1295074916 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3e140d2960 con 0x56520096b130 2024-03-16T11:55:09.735 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:09.729+0000 7f3e0d7fa640 1 -- 172.21.15.29:0/1295074916 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f3e20004230 con 0x56520096b130 2024-03-16T11:55:09.736 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:09.729+0000 7f3e2ea48740 10 client.15762 fetch_fsmap learned FSMap version 24 2024-03-16T11:55:09.736 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:09.729+0000 7f3e2ea48740 1 -- 172.21.15.29:0/1295074916 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x5652009eafe0 con 0x56520096b130 2024-03-16T11:55:09.736 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:09.729+0000 7f3e0d7fa640 1 -- 172.21.15.29:0/1295074916 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3e140d2960 con 0x56520096b130 2024-03-16T11:55:09.736 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:09.730+0000 7f3e0d7fa640 1 -- 172.21.15.29:0/1295074916 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7f3e200049c0 con 0x56520096b130 2024-03-16T11:55:09.736 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:09.730+0000 7f3e2ea48740 10 client.15762 fetch_fsmap finished waiting for FSMap version 24 2024-03-16T11:55:09.736 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:09.730+0000 7f3e2ea48740 10 client.15762 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T11:55:09.736 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:09.730+0000 7f3e2ea48740 20 client.15762 populate_metadata read hostname 'smithi029' 2024-03-16T11:55:09.736 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:09.730+0000 7f3e2ea48740 1 --2- 172.21.15.29:0/1295074916 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x5652009eb710 0x5652009edb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:55:09.736 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:09.730+0000 7f3e2ea48740 4 client.15762 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-16T11:55:09.736 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:09.730+0000 7f3e2ea48740 1 -- 172.21.15.29:0/1295074916 --> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] -- command(tid 0: {"prefix": "flush journal"}) -- 0x565200855770 con 0x5652009eb710 2024-03-16T11:55:09.736 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:09.730+0000 7f3e293fe640 1 --2- 172.21.15.29:0/1295074916 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x5652009eb710 0x5652009edb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:55:09.736 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:09.730+0000 7f3e0d7fa640 10 client.15762 ms_handle_connect on v2:172.21.15.152:6838/1548731833 2024-03-16T11:55:09.736 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:09.730+0000 7f3e293fe640 1 --2- 172.21.15.29:0/1295074916 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x5652009eb710 0x5652009edb30 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8857f0c369a1fa69 server_cookie=c9105ce57e4a7bc in_seq=0 out_seq=0 2024-03-16T11:55:09.782 INFO:tasks.workunit.client.0.smithi029.stdout: 1 387246 11.56 MB/sec execute 54 sec latency 16.698 ms 2024-03-16T11:55:10.055 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:10.055+0000 7f3e0d7fa640 1 -- 172.21.15.29:0/1295074916 <== mds.0 v2:172.21.15.152:6838/1548731833 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x565200855770 con 0x5652009eb710 2024-03-16T11:55:10.055 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:10.055+0000 7f3e0d7fa640 10 client.15762 handle_command_reply: tid=0 2024-03-16T11:55:10.055 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:10.055+0000 7f3e2ea48740 1 client.15762 shutdown 2024-03-16T11:55:10.055 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:10.055+0000 7f3e2ea48740 1 -- 172.21.15.29:0/1295074916 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56520096b130 msgr2=0x565200978080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:55:10.055 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:10.055+0000 7f3e2ea48740 1 --2- 172.21.15.29:0/1295074916 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56520096b130 0x565200978080 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f3e2002bc70 tx=0x7f3e2002c9f0 comp rx=0 tx=0).stop 2024-03-16T11:55:10.055 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:10.056+0000 7f3e2ea48740 1 -- 172.21.15.29:0/1295074916 shutdown_connections 2024-03-16T11:55:10.055 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:10.056+0000 7f3e2ea48740 1 --2- 172.21.15.29:0/1295074916 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x5652009eb710 0x5652009edb30 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:55:10.056 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:10.056+0000 7f3e2ea48740 1 --2- 172.21.15.29:0/1295074916 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x565200974820 0x565200978b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:55:10.056 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:10.056+0000 7f3e2ea48740 1 --2- 172.21.15.29:0/1295074916 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56520096ba50 0x5652009785c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:55:10.056 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:10.056+0000 7f3e0d7fa640 0 client.15762 ms_handle_reset on v2:172.21.15.152:6838/1548731833 2024-03-16T11:55:10.056 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:10.056+0000 7f3e2ea48740 1 --2- 172.21.15.29:0/1295074916 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56520096b130 0x565200978080 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:55:10.056 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:10.056+0000 7f3e2ea48740 1 -- 172.21.15.29:0/1295074916 >> 172.21.15.29:0/1295074916 conn(0x565200965a50 msgr2=0x565200971870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:55:10.056 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:10.056+0000 7f3e2ea48740 1 -- 172.21.15.29:0/1295074916 shutdown_connections 2024-03-16T11:55:10.056 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:10.056+0000 7f3e2ea48740 1 -- 172.21.15.29:0/1295074916 wait complete. 2024-03-16T11:55:10.056 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:10.056+0000 7f3e2ea48740 20 client.15762 trim_cache size 0 max 16384 2024-03-16T11:55:10.056 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:{ 2024-03-16T11:55:10.056 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "message": "", 2024-03-16T11:55:10.056 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "return_code": 0 2024-03-16T11:55:10.057 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:} 2024-03-16T11:55:10.057 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:10.057+0000 7f3e2be63640 1 -- 172.21.15.29:0/4259845438 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f3e100a2a10 msgr2=0x7f3e100a4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:55:10.057 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:10.057+0000 7f3e2be63640 1 --2- 172.21.15.29:0/4259845438 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f3e100a2a10 0x7f3e100a4ed0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f3e1c012fd0 tx=0x7f3e1c021000 comp rx=0 tx=0).stop 2024-03-16T11:55:10.057 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:10.057+0000 7f3e2be63640 1 -- 172.21.15.29:0/4259845438 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3e24101080 msgr2=0x7f3e24199400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:55:10.057 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:10.057+0000 7f3e2be63640 1 --2- 172.21.15.29:0/4259845438 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3e24101080 0x7f3e24199400 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f3e1400e9f0 tx=0x7f3e1400eeb0 comp rx=0 tx=0).stop 2024-03-16T11:55:10.057 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:10.057+0000 7f3e2be63640 1 -- 172.21.15.29:0/4259845438 shutdown_connections 2024-03-16T11:55:10.058 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:10.057+0000 7f3e2be63640 1 --2- 172.21.15.29:0/4259845438 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f3e100a2a10 0x7f3e100a4ed0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:55:10.058 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:10.057+0000 7f3e2be63640 1 --2- 172.21.15.29:0/4259845438 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3e241066d0 0x7f3e241a3f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:55:10.058 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:10.057+0000 7f3e2be63640 1 --2- 172.21.15.29:0/4259845438 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3e24101080 0x7f3e24199400 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:55:10.058 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:10.057+0000 7f3e2be63640 1 --2- 172.21.15.29:0/4259845438 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3e241006c0 0x7f3e24198ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:55:10.058 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:10.057+0000 7f3e2be63640 1 -- 172.21.15.29:0/4259845438 >> 172.21.15.29:0/4259845438 conn(0x7f3e24077830 msgr2=0x7f3e24103230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:55:10.058 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:10.057+0000 7f3e2be63640 1 -- 172.21.15.29:0/4259845438 shutdown_connections 2024-03-16T11:55:10.058 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:10.057+0000 7f3e2be63640 1 -- 172.21.15.29:0/4259845438 wait complete. 2024-03-16T11:55:10.782 INFO:tasks.workunit.client.0.smithi029.stdout: 1 389251 11.49 MB/sec execute 55 sec latency 69.948 ms 2024-03-16T11:55:11.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:55:10 smithi154 ceph-mon[36320]: pgmap v131: 97 pgs: 97 active+clean; 49 MiB data, 1000 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 2.9 MiB/s wr, 343 op/s 2024-03-16T11:55:11.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:55:10 smithi029 ceph-mon[31975]: pgmap v131: 97 pgs: 97 active+clean; 49 MiB data, 1000 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 2.9 MiB/s wr, 343 op/s 2024-03-16T11:55:11.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:55:10 smithi152 ceph-mon[36291]: pgmap v131: 97 pgs: 97 active+clean; 49 MiB data, 1000 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 2.9 MiB/s wr, 343 op/s 2024-03-16T11:55:11.782 INFO:tasks.workunit.client.0.smithi029.stdout: 1 391133 11.50 MB/sec execute 56 sec latency 17.979 ms 2024-03-16T11:55:12.782 INFO:tasks.workunit.client.0.smithi029.stdout: 1 393992 11.61 MB/sec execute 57 sec latency 15.349 ms 2024-03-16T11:55:13.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:55:12 smithi154 ceph-mon[36320]: pgmap v132: 97 pgs: 97 active+clean; 43 MiB data, 1009 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 4.4 MiB/s wr, 539 op/s 2024-03-16T11:55:13.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:55:12 smithi029 ceph-mon[31975]: pgmap v132: 97 pgs: 97 active+clean; 43 MiB data, 1009 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 4.4 MiB/s wr, 539 op/s 2024-03-16T11:55:13.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:55:12 smithi152 ceph-mon[36291]: pgmap v132: 97 pgs: 97 active+clean; 43 MiB data, 1009 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 4.4 MiB/s wr, 539 op/s 2024-03-16T11:55:13.782 INFO:tasks.workunit.client.0.smithi029.stdout: 1 396197 11.54 MB/sec execute 58 sec latency 19.156 ms 2024-03-16T11:55:14.782 INFO:tasks.workunit.client.0.smithi029.stdout: 1 397976 11.55 MB/sec execute 59 sec latency 38.567 ms 2024-03-16T11:55:15.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:55:14 smithi154 ceph-mon[36320]: pgmap v133: 97 pgs: 97 active+clean; 43 MiB data, 1009 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 2.9 MiB/s wr, 301 op/s 2024-03-16T11:55:15.087 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:55:14 smithi029 ceph-mon[31975]: pgmap v133: 97 pgs: 97 active+clean; 43 MiB data, 1009 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 2.9 MiB/s wr, 301 op/s 2024-03-16T11:55:15.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:55:14 smithi152 ceph-mon[36291]: pgmap v133: 97 pgs: 97 active+clean; 43 MiB data, 1009 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 2.9 MiB/s wr, 301 op/s 2024-03-16T11:55:15.782 INFO:tasks.workunit.client.0.smithi029.stdout: 1 400476 11.58 MB/sec execute 60 sec latency 54.059 ms 2024-03-16T11:55:16.782 INFO:tasks.workunit.client.0.smithi029.stdout: 1 402117 11.52 MB/sec execute 61 sec latency 22.023 ms 2024-03-16T11:55:17.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:55:16 smithi154 ceph-mon[36320]: pgmap v134: 97 pgs: 97 active+clean; 43 MiB data, 1009 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 2.9 MiB/s wr, 301 op/s 2024-03-16T11:55:17.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:55:16 smithi029 ceph-mon[31975]: pgmap v134: 97 pgs: 97 active+clean; 43 MiB data, 1009 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 2.9 MiB/s wr, 301 op/s 2024-03-16T11:55:17.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:55:16 smithi152 ceph-mon[36291]: pgmap v134: 97 pgs: 97 active+clean; 43 MiB data, 1009 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 2.9 MiB/s wr, 301 op/s 2024-03-16T11:55:17.782 INFO:tasks.workunit.client.0.smithi029.stdout: 1 404771 11.61 MB/sec execute 62 sec latency 15.005 ms 2024-03-16T11:55:18.783 INFO:tasks.workunit.client.0.smithi029.stdout: 1 406982 11.55 MB/sec execute 63 sec latency 17.061 ms 2024-03-16T11:55:19.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:55:18 smithi154 ceph-mon[36320]: pgmap v135: 97 pgs: 97 active+clean; 51 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 4.5 MiB/s wr, 406 op/s 2024-03-16T11:55:19.087 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:55:18 smithi029 ceph-mon[31975]: pgmap v135: 97 pgs: 97 active+clean; 51 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 4.5 MiB/s wr, 406 op/s 2024-03-16T11:55:19.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:55:18 smithi152 ceph-mon[36291]: pgmap v135: 97 pgs: 97 active+clean; 51 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 4.5 MiB/s wr, 406 op/s 2024-03-16T11:55:19.782 INFO:tasks.workunit.client.0.smithi029.stdout: 1 408738 11.55 MB/sec execute 64 sec latency 16.093 ms 2024-03-16T11:55:20.783 INFO:tasks.workunit.client.0.smithi029.stdout: 1 411284 11.59 MB/sec execute 65 sec latency 85.934 ms 2024-03-16T11:55:21.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:55:20 smithi154 ceph-mon[36320]: pgmap v136: 97 pgs: 97 active+clean; 51 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 3.1 MiB/s wr, 301 op/s 2024-03-16T11:55:21.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:55:20 smithi029 ceph-mon[31975]: pgmap v136: 97 pgs: 97 active+clean; 51 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 3.1 MiB/s wr, 301 op/s 2024-03-16T11:55:21.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:55:20 smithi152 ceph-mon[36291]: pgmap v136: 97 pgs: 97 active+clean; 51 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 3.1 MiB/s wr, 301 op/s 2024-03-16T11:55:21.783 INFO:tasks.workunit.client.0.smithi029.stdout: 1 412917 11.56 MB/sec execute 66 sec latency 26.970 ms 2024-03-16T11:55:22.783 INFO:tasks.workunit.client.0.smithi029.stdout: 1 415413 11.60 MB/sec execute 67 sec latency 56.441 ms 2024-03-16T11:55:23.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:55:22 smithi154 ceph-mon[36320]: pgmap v137: 97 pgs: 97 active+clean; 57 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 4.5 MiB/s wr, 404 op/s 2024-03-16T11:55:23.087 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:55:22 smithi029 ceph-mon[31975]: pgmap v137: 97 pgs: 97 active+clean; 57 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 4.5 MiB/s wr, 404 op/s 2024-03-16T11:55:23.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:55:22 smithi152 ceph-mon[36291]: pgmap v137: 97 pgs: 97 active+clean; 57 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 4.5 MiB/s wr, 404 op/s 2024-03-16T11:55:23.154 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:23.153+0000 7efcfca81640 1 -- 172.21.15.29:0/348430007 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efcf81067c0 msgr2=0x7efcf8106ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:55:23.155 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:23.153+0000 7efcfca81640 1 --2- 172.21.15.29:0/348430007 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efcf81067c0 0x7efcf8106ba0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7efce8009930 tx=0x7efce802bc30 comp rx=0 tx=0).stop 2024-03-16T11:55:23.155 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:23.153+0000 7efcfca81640 1 -- 172.21.15.29:0/348430007 shutdown_connections 2024-03-16T11:55:23.155 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:23.153+0000 7efcfca81640 1 --2- 172.21.15.29:0/348430007 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efcf81006a0 0x7efcf810e670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:55:23.155 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:23.153+0000 7efcfca81640 1 --2- 172.21.15.29:0/348430007 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7efcf80ffce0 0x7efcf8100160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:55:23.155 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:23.153+0000 7efcfca81640 1 --2- 172.21.15.29:0/348430007 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efcf81067c0 0x7efcf8106ba0 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:55:23.155 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:23.153+0000 7efcfca81640 1 -- 172.21.15.29:0/348430007 >> 172.21.15.29:0/348430007 conn(0x7efcf8076ca0 msgr2=0x7efcf80fdff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:55:23.155 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:23.153+0000 7efcfca81640 1 -- 172.21.15.29:0/348430007 shutdown_connections 2024-03-16T11:55:23.155 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:23.153+0000 7efcfca81640 1 -- 172.21.15.29:0/348430007 wait complete. 2024-03-16T11:55:23.155 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:23.154+0000 7efcfca81640 1 Processor -- start 2024-03-16T11:55:23.155 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:23.154+0000 7efcfca81640 1 -- start start 2024-03-16T11:55:23.155 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:23.154+0000 7efcfca81640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7efcf80ffce0 0x7efcf8100ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:55:23.155 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:23.154+0000 7efcfca81640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efcf81006a0 0x7efcf8101400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:55:23.155 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:23.154+0000 7efcfca81640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efcf81067c0 0x7efcf8101940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:55:23.156 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:23.154+0000 7efcfca81640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7efcf8111c90 con 0x7efcf81006a0 2024-03-16T11:55:23.156 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:23.154+0000 7efcfca81640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7efcf8111b10 con 0x7efcf80ffce0 2024-03-16T11:55:23.156 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:23.154+0000 7efcfca81640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7efcf8111e10 con 0x7efcf81067c0 2024-03-16T11:55:23.156 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:23.154+0000 7efceffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efcf81006a0 0x7efcf8101400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:55:23.156 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:23.154+0000 7efceffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efcf81006a0 0x7efcf8101400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47344/0 (socket says 172.21.15.29:47344) 2024-03-16T11:55:23.156 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:23.154+0000 7efceffff640 1 -- 172.21.15.29:0/2285544000 learned_addr learned my addr 172.21.15.29:0/2285544000 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:55:23.156 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:23.154+0000 7efcf659c640 1 --2- 172.21.15.29:0/2285544000 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7efcf80ffce0 0x7efcf8100ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:55:23.156 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:23.154+0000 7efcf6d9d640 1 --2- 172.21.15.29:0/2285544000 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efcf81067c0 0x7efcf8101940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:55:23.156 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:23.154+0000 7efceffff640 1 -- 172.21.15.29:0/2285544000 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efcf81067c0 msgr2=0x7efcf8101940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:55:23.156 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:23.155+0000 7efceffff640 1 --2- 172.21.15.29:0/2285544000 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efcf81067c0 0x7efcf8101940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:55:23.156 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:23.155+0000 7efceffff640 1 -- 172.21.15.29:0/2285544000 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7efcf80ffce0 msgr2=0x7efcf8100ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:55:23.156 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:23.155+0000 7efceffff640 1 --2- 172.21.15.29:0/2285544000 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7efcf80ffce0 0x7efcf8100ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:55:23.156 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:23.155+0000 7efceffff640 1 -- 172.21.15.29:0/2285544000 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efcf81a6570 con 0x7efcf81006a0 2024-03-16T11:55:23.156 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:23.155+0000 7efceffff640 1 --2- 172.21.15.29:0/2285544000 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efcf81006a0 0x7efcf8101400 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7efce000e9f0 tx=0x7efce000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:55:23.157 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:23.155+0000 7efcef7fe640 1 -- 172.21.15.29:0/2285544000 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efce000ccc0 con 0x7efcf81006a0 2024-03-16T11:55:23.157 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:23.155+0000 7efcfca81640 1 -- 172.21.15.29:0/2285544000 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efcf81a6860 con 0x7efcf81006a0 2024-03-16T11:55:23.157 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:23.155+0000 7efcef7fe640 1 -- 172.21.15.29:0/2285544000 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efce0010910 con 0x7efcf81006a0 2024-03-16T11:55:23.157 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:23.155+0000 7efcef7fe640 1 -- 172.21.15.29:0/2285544000 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efce00188a0 con 0x7efcf81006a0 2024-03-16T11:55:23.157 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:23.155+0000 7efcfca81640 1 -- 172.21.15.29:0/2285544000 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efcf81a6e00 con 0x7efcf81006a0 2024-03-16T11:55:23.162 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:23.157+0000 7efcef7fe640 1 -- 172.21.15.29:0/2285544000 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7efce0010ac0 con 0x7efcf81006a0 2024-03-16T11:55:23.162 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:23.157+0000 7efcff66d740 1 Processor -- start 2024-03-16T11:55:23.162 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:23.158+0000 7efcff66d740 1 -- start start 2024-03-16T11:55:23.162 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:23.158+0000 7efcff66d740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c936a0f130 0x55c936a0f510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:55:23.163 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:23.158+0000 7efcff66d740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c936a0fa50 0x55c936a182e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:55:23.163 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:23.158+0000 7efcff66d740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c936a18820 0x55c936a1ac10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:55:23.163 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:23.158+0000 7efcff66d740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55c936973a40 con 0x55c936a18820 2024-03-16T11:55:23.163 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:23.158+0000 7efcff66d740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x55c93691c920 con 0x55c936a0fa50 2024-03-16T11:55:23.163 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:23.158+0000 7efcff66d740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55c93696fae0 con 0x55c936a0f130 2024-03-16T11:55:23.163 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:23.158+0000 7efcef7fe640 1 --2- 172.21.15.29:0/2285544000 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7efce40a2a10 0x7efce40a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:55:23.163 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:23.158+0000 7efceffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c936a18820 0x55c936a1ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:55:23.163 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:23.158+0000 7efcf6d9d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c936a0f130 0x55c936a0f510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:55:23.163 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:23.158+0000 7efcf659c640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c936a0fa50 0x55c936a182e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:55:23.163 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:23.158+0000 7efceffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c936a18820 0x55c936a1ac10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47356/0 (socket says 172.21.15.29:47356) 2024-03-16T11:55:23.163 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:23.158+0000 7efceffff640 1 -- 172.21.15.29:0/3342862770 learned_addr learned my addr 172.21.15.29:0/3342862770 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:55:23.163 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:23.158+0000 7efcf6d9d640 1 --2- 172.21.15.29:0/2285544000 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7efce40a2a10 0x7efce40a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:55:23.163 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:23.158+0000 7efcef7fe640 1 -- 172.21.15.29:0/2285544000 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7efce00cf280 con 0x7efcf81006a0 2024-03-16T11:55:23.164 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:23.158+0000 7efcf659c640 1 -- 172.21.15.29:0/3342862770 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c936a0f130 msgr2=0x55c936a0f510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:55:23.164 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:23.158+0000 7efcf659c640 1 --2- 172.21.15.29:0/3342862770 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c936a0f130 0x55c936a0f510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:55:23.164 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:23.158+0000 7efcf659c640 1 -- 172.21.15.29:0/3342862770 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c936a18820 msgr2=0x55c936a1ac10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:55:23.164 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:23.158+0000 7efcf659c640 1 --2- 172.21.15.29:0/3342862770 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c936a18820 0x55c936a1ac10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:55:23.164 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:23.158+0000 7efcf659c640 1 -- 172.21.15.29:0/3342862770 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c936934460 con 0x55c936a0fa50 2024-03-16T11:55:23.164 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:23.159+0000 7efcf6d9d640 1 --2- 172.21.15.29:0/2285544000 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7efce40a2a10 0x7efce40a4ed0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7efcf0009b80 tx=0x7efcf0009500 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:55:23.164 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:23.159+0000 7efcf659c640 1 --2- 172.21.15.29:0/3342862770 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c936a0fa50 0x55c936a182e0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7efce8009230 tx=0x7efce8009610 comp rx=0 tx=0).ready entity=mon.2 client_cookie=634c820ddaac2e80 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:55:23.164 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:23.159+0000 7efcecff9640 1 -- 172.21.15.29:0/3342862770 <== mon.2 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efce8042070 con 0x55c936a0fa50 2024-03-16T11:55:23.164 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:23.159+0000 7efcecff9640 1 -- 172.21.15.29:0/3342862770 <== mon.2 v2:172.21.15.152:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efce802c690 con 0x55c936a0fa50 2024-03-16T11:55:23.164 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:23.159+0000 7efcecff9640 1 -- 172.21.15.29:0/3342862770 <== mon.2 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efce8004b30 con 0x55c936a0fa50 2024-03-16T11:55:23.164 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:23.159+0000 7efcff66d740 1 -- 172.21.15.29:0/3342862770 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c936a0fa50 msgr2=0x55c936a182e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:55:23.164 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:23.159+0000 7efcff66d740 1 --2- 172.21.15.29:0/3342862770 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c936a0fa50 0x55c936a182e0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7efce8009230 tx=0x7efce8009610 comp rx=0 tx=0).stop 2024-03-16T11:55:23.164 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:23.159+0000 7efcff66d740 1 -- 172.21.15.29:0/3342862770 shutdown_connections 2024-03-16T11:55:23.164 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:23.159+0000 7efcff66d740 1 --2- 172.21.15.29:0/3342862770 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c936a18820 0x55c936a1ac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:55:23.164 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:23.160+0000 7efcff66d740 1 --2- 172.21.15.29:0/3342862770 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c936a0fa50 0x55c936a182e0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:55:23.165 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:23.160+0000 7efcff66d740 1 --2- 172.21.15.29:0/3342862770 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c936a0f130 0x55c936a0f510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:55:23.165 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:23.160+0000 7efcff66d740 1 -- 172.21.15.29:0/3342862770 >> 172.21.15.29:0/3342862770 conn(0x55c936a09a50 msgr2=0x55c936a09e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:55:23.165 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:23.160+0000 7efcff66d740 1 -- 172.21.15.29:0/3342862770 shutdown_connections 2024-03-16T11:55:23.165 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:23.160+0000 7efcff66d740 1 -- 172.21.15.29:0/3342862770 wait complete. 2024-03-16T11:55:23.165 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:23.160+0000 7efcff66d740 1 -- start start 2024-03-16T11:55:23.165 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:23.160+0000 7efcff66d740 1 Processor -- start 2024-03-16T11:55:23.165 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:23.160+0000 7efcff66d740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c936a0f130 0x55c936a15850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:55:23.165 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:23.160+0000 7efcff66d740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c936a0fa50 0x55c936a15d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:55:23.165 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:23.160+0000 7efcff66d740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c936a18820 0x55c936a162d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:55:23.165 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:23.160+0000 7efcff66d740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55c936a1b3f0 con 0x55c936a0fa50 2024-03-16T11:55:23.165 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:23.160+0000 7efcff66d740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x55c93692fd70 con 0x55c936a18820 2024-03-16T11:55:23.165 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:23.160+0000 7efcff66d740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55c936a1b570 con 0x55c936a0f130 2024-03-16T11:55:23.165 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:23.160+0000 7efcff66d740 10 client.0 _ll_register_callbacks cb 0x55c936687a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T11:55:23.165 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:23.160+0000 7efceffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c936a0fa50 0x55c936a15d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:55:23.165 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:23.160+0000 7efcf659c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c936a0f130 0x55c936a15850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:55:23.166 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:23.160+0000 7efceffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c936a0fa50 0x55c936a15d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47366/0 (socket says 172.21.15.29:47366) 2024-03-16T11:55:23.166 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:23.160+0000 7efceffff640 1 -- 172.21.15.29:0/242268995 learned_addr learned my addr 172.21.15.29:0/242268995 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:55:23.166 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:23.160+0000 7efcf6d9d640 1 --2- 172.21.15.29:0/242268995 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c936a18820 0x55c936a162d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:55:23.166 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:23.161+0000 7efceffff640 1 -- 172.21.15.29:0/242268995 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c936a0f130 msgr2=0x55c936a15850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:55:23.166 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:23.161+0000 7efceffff640 1 --2- 172.21.15.29:0/242268995 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c936a0f130 0x55c936a15850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:55:23.166 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:23.161+0000 7efceffff640 1 -- 172.21.15.29:0/242268995 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c936a18820 msgr2=0x55c936a162d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:55:23.166 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:23.161+0000 7efceffff640 1 --2- 172.21.15.29:0/242268995 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c936a18820 0x55c936a162d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:55:23.166 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:23.161+0000 7efceffff640 1 -- 172.21.15.29:0/242268995 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c936688210 con 0x55c936a0fa50 2024-03-16T11:55:23.166 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:23.161+0000 7efcf659c640 1 --2- 172.21.15.29:0/242268995 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c936a0f130 0x55c936a15850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:55:23.166 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:23.161+0000 7efceffff640 1 --2- 172.21.15.29:0/242268995 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c936a0fa50 0x55c936a15d90 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7efce0097d70 tx=0x7efce000cef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=985993646053705e server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:55:23.166 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:23.161+0000 7efcddffb640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-16T11:55:23.166 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:23.161+0000 7efcddffb640 1 -- 172.21.15.29:0/242268995 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efce00a3e20 con 0x55c936a0fa50 2024-03-16T11:55:23.166 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:23.161+0000 7efcddffb640 1 -- 172.21.15.29:0/242268995 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7efce0086070 con 0x55c936a0fa50 2024-03-16T11:55:23.167 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:23.161+0000 7efcff66d740 1 -- 172.21.15.29:0/242268995 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55c936980f80 con 0x55c936a0fa50 2024-03-16T11:55:23.167 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:23.161+0000 7efcddffb640 1 -- 172.21.15.29:0/242268995 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7efce009b910 con 0x55c936a0fa50 2024-03-16T11:55:23.167 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:23.161+0000 7efcff66d740 10 client.15774 fetch_fsmap learned FSMap version 24 2024-03-16T11:55:23.167 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:23.161+0000 7efcff66d740 1 -- 172.21.15.29:0/242268995 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55c936754620 con 0x55c936a0fa50 2024-03-16T11:55:23.167 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:23.161+0000 7efcddffb640 1 -- 172.21.15.29:0/242268995 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55c936980f80 con 0x55c936a0fa50 2024-03-16T11:55:23.167 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:23.162+0000 7efcddffb640 1 -- 172.21.15.29:0/242268995 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7efce009ad60 con 0x55c936a0fa50 2024-03-16T11:55:23.167 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:23.162+0000 7efcff66d740 10 client.15774 fetch_fsmap finished waiting for FSMap version 24 2024-03-16T11:55:23.167 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:23.162+0000 7efcff66d740 10 client.15774 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T11:55:23.167 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:23.162+0000 7efcff66d740 20 client.15774 populate_metadata read hostname 'smithi029' 2024-03-16T11:55:23.167 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:23.162+0000 7efcff66d740 1 --2- 172.21.15.29:0/242268995 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55c936a28940 0x55c936a2ad90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:55:23.167 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:23.162+0000 7efcff66d740 4 client.15774 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T11:55:23.167 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:23.162+0000 7efcff66d740 1 -- 172.21.15.29:0/242268995 --> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c93697b2d0 con 0x55c936a28940 2024-03-16T11:55:23.167 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:23.162+0000 7efcf659c640 1 --2- 172.21.15.29:0/242268995 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55c936a28940 0x55c936a2ad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:55:23.167 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:23.162+0000 7efcf659c640 1 --2- 172.21.15.29:0/242268995 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55c936a28940 0x55c936a2ad90 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f234b1dcfa434b76 server_cookie=ac1cd8b7033f07a2 in_seq=0 out_seq=0 2024-03-16T11:55:23.167 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:23.162+0000 7efcddffb640 10 client.15774 ms_handle_connect on v2:172.21.15.152:6838/1548731833 2024-03-16T11:55:23.169 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:23.164+0000 7efcddffb640 1 -- 172.21.15.29:0/242268995 <== mds.0 v2:172.21.15.152:6838/1548731833 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55c93697b2d0 con 0x55c936a28940 2024-03-16T11:55:23.169 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:23.164+0000 7efcddffb640 10 client.15774 handle_command_reply: tid=0 2024-03-16T11:55:23.169 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:23.164+0000 7efcff66d740 1 client.15774 shutdown 2024-03-16T11:55:23.169 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:23.164+0000 7efcff66d740 1 -- 172.21.15.29:0/242268995 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c936a0fa50 msgr2=0x55c936a15d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:55:23.169 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:23.164+0000 7efcff66d740 1 --2- 172.21.15.29:0/242268995 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c936a0fa50 0x55c936a15d90 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7efce0097d70 tx=0x7efce000cef0 comp rx=0 tx=0).stop 2024-03-16T11:55:23.169 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:23.164+0000 7efcff66d740 1 -- 172.21.15.29:0/242268995 shutdown_connections 2024-03-16T11:55:23.169 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:23.164+0000 7efcff66d740 1 --2- 172.21.15.29:0/242268995 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55c936a28940 0x55c936a2ad90 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:55:23.169 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:23.164+0000 7efcff66d740 1 --2- 172.21.15.29:0/242268995 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c936a18820 0x55c936a162d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:55:23.169 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:23.164+0000 7efcff66d740 1 --2- 172.21.15.29:0/242268995 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c936a0fa50 0x55c936a15d90 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:55:23.169 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:23.164+0000 7efcddffb640 0 client.15774 ms_handle_reset on v2:172.21.15.152:6838/1548731833 2024-03-16T11:55:23.169 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:23.164+0000 7efcff66d740 1 --2- 172.21.15.29:0/242268995 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c936a0f130 0x55c936a15850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:55:23.169 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:23.164+0000 7efcff66d740 1 -- 172.21.15.29:0/242268995 >> 172.21.15.29:0/242268995 conn(0x55c936a09a50 msgr2=0x55c936a0b7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:55:23.169 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:23.164+0000 7efcff66d740 1 -- 172.21.15.29:0/242268995 shutdown_connections 2024-03-16T11:55:23.170 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:23.164+0000 7efcff66d740 1 -- 172.21.15.29:0/242268995 wait complete. 2024-03-16T11:55:23.170 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:23.164+0000 7efcff66d740 20 client.15774 trim_cache size 0 max 16384 2024-03-16T11:55:23.185 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:23.181+0000 7efcff66d740 1 Processor -- start 2024-03-16T11:55:23.185 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:23.181+0000 7efcff66d740 1 -- start start 2024-03-16T11:55:23.185 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:23.181+0000 7efcff66d740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c936a0f130 0x55c936a17f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:55:23.185 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:23.181+0000 7efcff66d740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c936a0fa50 0x55c936a04a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:55:23.185 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:23.181+0000 7efcff66d740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c936a18820 0x55c936a24190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:55:23.185 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:23.181+0000 7efcff66d740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7efcf81b1d90 con 0x55c936a0fa50 2024-03-16T11:55:23.186 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:23.181+0000 7efcff66d740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7efcf81af660 con 0x55c936a18820 2024-03-16T11:55:23.186 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:23.181+0000 7efcff66d740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7efcf81ad4b0 con 0x55c936a0f130 2024-03-16T11:55:23.186 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:23.181+0000 7efceffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c936a0fa50 0x55c936a04a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:55:23.186 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:23.182+0000 7efcf659c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c936a0f130 0x55c936a17f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:55:23.186 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:23.182+0000 7efcf6d9d640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c936a18820 0x55c936a24190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:55:23.186 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:23.182+0000 7efceffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c936a0fa50 0x55c936a04a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47372/0 (socket says 172.21.15.29:47372) 2024-03-16T11:55:23.186 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:23.182+0000 7efcf659c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c936a0f130 0x55c936a17f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.29:52276/0 (socket says 172.21.15.29:52276) 2024-03-16T11:55:23.186 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:23.182+0000 7efceffff640 1 -- 172.21.15.29:0/3154199670 learned_addr learned my addr 172.21.15.29:0/3154199670 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:55:23.186 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:23.182+0000 7efceffff640 1 -- 172.21.15.29:0/3154199670 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c936a0f130 msgr2=0x55c936a17f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:55:23.186 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:23.182+0000 7efceffff640 1 --2- 172.21.15.29:0/3154199670 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c936a0f130 0x55c936a17f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:55:23.187 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:23.182+0000 7efceffff640 1 -- 172.21.15.29:0/3154199670 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c936a18820 msgr2=0x55c936a24190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:55:23.187 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:23.182+0000 7efceffff640 1 --2- 172.21.15.29:0/3154199670 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c936a18820 0x55c936a24190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:55:23.187 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:23.182+0000 7efceffff640 1 -- 172.21.15.29:0/3154199670 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c936a14800 con 0x55c936a0fa50 2024-03-16T11:55:23.187 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:23.182+0000 7efcf659c640 1 --2- 172.21.15.29:0/3154199670 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c936a0f130 0x55c936a17f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:55:23.187 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:23.182+0000 7efceffff640 1 --2- 172.21.15.29:0/3154199670 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c936a0fa50 0x55c936a04a40 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7efce0099bc0 tx=0x7efce0099e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=434d7a1619dfd7fe server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:55:23.187 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:23.182+0000 7efcdffff640 1 -- 172.21.15.29:0/3154199670 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efce009af60 con 0x55c936a0fa50 2024-03-16T11:55:23.187 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:23.182+0000 7efcdffff640 1 -- 172.21.15.29:0/3154199670 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efce0084e50 con 0x55c936a0fa50 2024-03-16T11:55:23.187 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:23.182+0000 7efcdffff640 1 -- 172.21.15.29:0/3154199670 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efce00a3e80 con 0x55c936a0fa50 2024-03-16T11:55:23.187 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:23.183+0000 7efcff66d740 1 -- 172.21.15.29:0/3154199670 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c936a0fa50 msgr2=0x55c936a04a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:55:23.187 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:23.183+0000 7efcff66d740 1 --2- 172.21.15.29:0/3154199670 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c936a0fa50 0x55c936a04a40 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7efce0099bc0 tx=0x7efce0099e20 comp rx=0 tx=0).stop 2024-03-16T11:55:23.187 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:23.183+0000 7efcff66d740 1 -- 172.21.15.29:0/3154199670 shutdown_connections 2024-03-16T11:55:23.187 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:23.183+0000 7efcff66d740 1 --2- 172.21.15.29:0/3154199670 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c936a18820 0x55c936a24190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:55:23.187 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:23.183+0000 7efcff66d740 1 --2- 172.21.15.29:0/3154199670 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c936a0fa50 0x55c936a04a40 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:55:23.187 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:23.183+0000 7efcff66d740 1 --2- 172.21.15.29:0/3154199670 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c936a0f130 0x55c936a17f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:55:23.187 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:23.183+0000 7efcff66d740 1 -- 172.21.15.29:0/3154199670 >> 172.21.15.29:0/3154199670 conn(0x55c936a09a50 msgr2=0x55c936a08bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:55:23.188 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:23.183+0000 7efcff66d740 1 -- 172.21.15.29:0/3154199670 shutdown_connections 2024-03-16T11:55:23.188 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:23.183+0000 7efcff66d740 1 -- 172.21.15.29:0/3154199670 wait complete. 2024-03-16T11:55:23.188 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:23.183+0000 7efcff66d740 1 -- start start 2024-03-16T11:55:23.188 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:23.183+0000 7efcff66d740 1 Processor -- start 2024-03-16T11:55:23.188 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:23.183+0000 7efcff66d740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c936a0f130 0x55c936a1c800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:55:23.188 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:23.183+0000 7efcff66d740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c936a0fa50 0x55c936a1cd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:55:23.188 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:23.183+0000 7efcff66d740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c936a18820 0x55c936a8dc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:55:23.188 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:23.183+0000 7efcff66d740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7efcf81adb20 con 0x55c936a0f130 2024-03-16T11:55:23.188 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:23.183+0000 7efcff66d740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7efcf81aef90 con 0x55c936a18820 2024-03-16T11:55:23.188 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:23.183+0000 7efcff66d740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7efcf81ae1f0 con 0x55c936a0fa50 2024-03-16T11:55:23.188 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:23.183+0000 7efcff66d740 10 client.0 _ll_register_callbacks cb 0x7efce000e940 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T11:55:23.188 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:23.184+0000 7efcf659c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c936a0f130 0x55c936a1c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:55:23.188 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:23.184+0000 7efcf659c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c936a0f130 0x55c936a1c800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47376/0 (socket says 172.21.15.29:47376) 2024-03-16T11:55:23.188 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:23.184+0000 7efcf659c640 1 -- 172.21.15.29:0/3925884289 learned_addr learned my addr 172.21.15.29:0/3925884289 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:55:23.189 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:23.184+0000 7efceffff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c936a0fa50 0x55c936a1cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:55:23.189 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:23.184+0000 7efcf6d9d640 1 --2- 172.21.15.29:0/3925884289 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c936a18820 0x55c936a8dc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:55:23.189 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:23.184+0000 7efcf659c640 1 -- 172.21.15.29:0/3925884289 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c936a0fa50 msgr2=0x55c936a1cd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:55:23.189 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:23.184+0000 7efcf659c640 1 --2- 172.21.15.29:0/3925884289 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c936a0fa50 0x55c936a1cd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:55:23.189 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:23.184+0000 7efcf659c640 1 -- 172.21.15.29:0/3925884289 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c936a18820 msgr2=0x55c936a8dc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:55:23.189 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:23.184+0000 7efcf659c640 1 --2- 172.21.15.29:0/3925884289 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c936a18820 0x55c936a8dc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:55:23.189 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:23.184+0000 7efcf659c640 1 -- 172.21.15.29:0/3925884289 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c936a8e1c0 con 0x55c936a0f130 2024-03-16T11:55:23.189 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:23.184+0000 7efcf659c640 1 --2- 172.21.15.29:0/3925884289 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c936a0f130 0x55c936a1c800 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7efce8009930 tx=0x7efce8005b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25fa34c38353f544 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:55:23.189 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:23.184+0000 7efcddffb640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-16T11:55:23.189 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:23.184+0000 7efcddffb640 1 -- 172.21.15.29:0/3925884289 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efce8042040 con 0x55c936a0f130 2024-03-16T11:55:23.189 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:23.184+0000 7efcddffb640 1 -- 172.21.15.29:0/3925884289 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7efce80503b0 con 0x55c936a0f130 2024-03-16T11:55:23.189 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:23.184+0000 7efcff66d740 1 -- 172.21.15.29:0/3925884289 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efce0084e50 con 0x55c936a0f130 2024-03-16T11:55:23.189 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:23.184+0000 7efcddffb640 1 -- 172.21.15.29:0/3925884289 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7efce8005d50 con 0x55c936a0f130 2024-03-16T11:55:23.189 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:23.184+0000 7efcff66d740 10 client.15780 fetch_fsmap learned FSMap version 24 2024-03-16T11:55:23.190 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:23.184+0000 7efcff66d740 1 -- 172.21.15.29:0/3925884289 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55c936a8f4d0 con 0x55c936a0f130 2024-03-16T11:55:23.190 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:23.184+0000 7efcddffb640 1 -- 172.21.15.29:0/3925884289 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efce0084e50 con 0x55c936a0f130 2024-03-16T11:55:23.190 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:23.185+0000 7efcddffb640 1 -- 172.21.15.29:0/3925884289 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7efce8004270 con 0x55c936a0f130 2024-03-16T11:55:23.190 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:23.185+0000 7efcff66d740 10 client.15780 fetch_fsmap finished waiting for FSMap version 24 2024-03-16T11:55:23.190 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:23.185+0000 7efcff66d740 10 client.15780 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T11:55:23.190 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:23.185+0000 7efcff66d740 20 client.15780 populate_metadata read hostname 'smithi029' 2024-03-16T11:55:23.190 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:23.185+0000 7efcff66d740 1 --2- 172.21.15.29:0/3925884289 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55c936a8f830 0x55c936a91c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:55:23.190 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:23.185+0000 7efcff66d740 4 client.15780 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-16T11:55:23.190 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:23.185+0000 7efcff66d740 1 -- 172.21.15.29:0/3925884289 --> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55c9368f9770 con 0x55c936a8f830 2024-03-16T11:55:23.190 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:23.185+0000 7efceffff640 1 --2- 172.21.15.29:0/3925884289 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55c936a8f830 0x55c936a91c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:55:23.190 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:23.185+0000 7efceffff640 1 --2- 172.21.15.29:0/3925884289 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55c936a8f830 0x55c936a91c20 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d9cc60198a4443c4 server_cookie=15caf6b7a8ebd132 in_seq=0 out_seq=0 2024-03-16T11:55:23.190 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:23.185+0000 7efcddffb640 10 client.15780 ms_handle_connect on v2:172.21.15.152:6838/1548731833 2024-03-16T11:55:23.359 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:23.359+0000 7efcddffb640 1 -- 172.21.15.29:0/3925884289 <== mds.0 v2:172.21.15.152:6838/1548731833 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55c9368f9770 con 0x55c936a8f830 2024-03-16T11:55:23.359 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:23.359+0000 7efcddffb640 10 client.15780 handle_command_reply: tid=0 2024-03-16T11:55:23.359 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:23.359+0000 7efcff66d740 1 client.15780 shutdown 2024-03-16T11:55:23.359 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:23.359+0000 7efcff66d740 1 -- 172.21.15.29:0/3925884289 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c936a0f130 msgr2=0x55c936a1c800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:55:23.359 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:23.359+0000 7efcff66d740 1 --2- 172.21.15.29:0/3925884289 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c936a0f130 0x55c936a1c800 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7efce8009930 tx=0x7efce8005b50 comp rx=0 tx=0).stop 2024-03-16T11:55:23.359 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:23.359+0000 7efcff66d740 1 -- 172.21.15.29:0/3925884289 shutdown_connections 2024-03-16T11:55:23.359 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:23.359+0000 7efcff66d740 1 --2- 172.21.15.29:0/3925884289 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55c936a8f830 0x55c936a91c20 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:55:23.359 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:23.359+0000 7efcff66d740 1 --2- 172.21.15.29:0/3925884289 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c936a18820 0x55c936a8dc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:55:23.360 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:23.359+0000 7efcff66d740 1 --2- 172.21.15.29:0/3925884289 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c936a0fa50 0x55c936a1cd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:55:23.360 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:23.359+0000 7efcff66d740 1 --2- 172.21.15.29:0/3925884289 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c936a0f130 0x55c936a1c800 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:55:23.360 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:23.359+0000 7efcddffb640 0 client.15780 ms_handle_reset on v2:172.21.15.152:6838/1548731833 2024-03-16T11:55:23.360 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:23.359+0000 7efcff66d740 1 -- 172.21.15.29:0/3925884289 >> 172.21.15.29:0/3925884289 conn(0x55c936a09a50 msgr2=0x55c936a15810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:55:23.360 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:23.359+0000 7efcff66d740 1 -- 172.21.15.29:0/3925884289 shutdown_connections 2024-03-16T11:55:23.360 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:23.359+0000 7efcff66d740 1 -- 172.21.15.29:0/3925884289 wait complete. 2024-03-16T11:55:23.360 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:23.359+0000 7efcff66d740 20 client.15780 trim_cache size 0 max 16384 2024-03-16T11:55:23.360 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:{ 2024-03-16T11:55:23.360 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "message": "", 2024-03-16T11:55:23.360 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "return_code": 0 2024-03-16T11:55:23.361 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:} 2024-03-16T11:55:23.361 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:23.360+0000 7efcfca81640 1 -- 172.21.15.29:0/2285544000 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7efce40a2a10 msgr2=0x7efce40a4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:55:23.361 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:23.360+0000 7efcfca81640 1 --2- 172.21.15.29:0/2285544000 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7efce40a2a10 0x7efce40a4ed0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7efcf0009b80 tx=0x7efcf0009500 comp rx=0 tx=0).stop 2024-03-16T11:55:23.361 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:23.360+0000 7efcfca81640 1 -- 172.21.15.29:0/2285544000 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efcf81006a0 msgr2=0x7efcf8101400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:55:23.362 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:23.360+0000 7efcfca81640 1 --2- 172.21.15.29:0/2285544000 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efcf81006a0 0x7efcf8101400 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7efce000e9f0 tx=0x7efce000edf0 comp rx=0 tx=0).stop 2024-03-16T11:55:23.362 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:23.360+0000 7efcfca81640 1 -- 172.21.15.29:0/2285544000 shutdown_connections 2024-03-16T11:55:23.362 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:23.360+0000 7efcfca81640 1 --2- 172.21.15.29:0/2285544000 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7efce40a2a10 0x7efce40a4ed0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:55:23.362 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:23.360+0000 7efcfca81640 1 --2- 172.21.15.29:0/2285544000 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efcf81067c0 0x7efcf8101940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:55:23.362 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:23.360+0000 7efcfca81640 1 --2- 172.21.15.29:0/2285544000 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efcf81006a0 0x7efcf8101400 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:55:23.362 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:23.360+0000 7efcfca81640 1 --2- 172.21.15.29:0/2285544000 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7efcf80ffce0 0x7efcf8100ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:55:23.362 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:23.360+0000 7efcfca81640 1 -- 172.21.15.29:0/2285544000 >> 172.21.15.29:0/2285544000 conn(0x7efcf8076ca0 msgr2=0x7efcf81046f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:55:23.362 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:23.360+0000 7efcfca81640 1 -- 172.21.15.29:0/2285544000 shutdown_connections 2024-03-16T11:55:23.362 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:23.360+0000 7efcfca81640 1 -- 172.21.15.29:0/2285544000 wait complete. 2024-03-16T11:55:23.783 INFO:tasks.workunit.client.0.smithi029.stdout: 1 417418 11.53 MB/sec execute 68 sec latency 81.056 ms 2024-03-16T11:55:24.783 INFO:tasks.workunit.client.0.smithi029.stdout: 1 419305 11.54 MB/sec execute 69 sec latency 21.240 ms 2024-03-16T11:55:25.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:55:24 smithi154 ceph-mon[36320]: pgmap v138: 97 pgs: 97 active+clean; 57 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 3.0 MiB/s wr, 208 op/s 2024-03-16T11:55:25.087 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:55:24 smithi029 ceph-mon[31975]: pgmap v138: 97 pgs: 97 active+clean; 57 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 3.0 MiB/s wr, 208 op/s 2024-03-16T11:55:25.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:55:24 smithi152 ceph-mon[36291]: pgmap v138: 97 pgs: 97 active+clean; 57 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 3.0 MiB/s wr, 208 op/s 2024-03-16T11:55:25.783 INFO:tasks.workunit.client.0.smithi029.stdout: 1 421592 11.55 MB/sec execute 70 sec latency 114.418 ms 2024-03-16T11:55:26.783 INFO:tasks.workunit.client.0.smithi029.stdout: 1 423205 11.51 MB/sec execute 71 sec latency 19.461 ms 2024-03-16T11:55:27.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:55:26 smithi154 ceph-mon[36320]: pgmap v139: 97 pgs: 97 active+clean; 57 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 3.0 MiB/s wr, 208 op/s 2024-03-16T11:55:27.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:55:26 smithi029 ceph-mon[31975]: pgmap v139: 97 pgs: 97 active+clean; 57 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 3.0 MiB/s wr, 208 op/s 2024-03-16T11:55:27.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:55:26 smithi152 ceph-mon[36291]: pgmap v139: 97 pgs: 97 active+clean; 57 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 3.0 MiB/s wr, 208 op/s 2024-03-16T11:55:27.783 INFO:tasks.workunit.client.0.smithi029.stdout: 1 425941 11.58 MB/sec execute 72 sec latency 12.786 ms 2024-03-16T11:55:28.783 INFO:tasks.workunit.client.0.smithi029.stdout: 1 428125 11.53 MB/sec execute 73 sec latency 17.849 ms 2024-03-16T11:55:29.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:55:28 smithi154 ceph-mon[36320]: pgmap v140: 97 pgs: 97 active+clean; 54 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 4.5 MiB/s wr, 411 op/s 2024-03-16T11:55:29.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:55:28 smithi029 ceph-mon[31975]: pgmap v140: 97 pgs: 97 active+clean; 54 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 4.5 MiB/s wr, 411 op/s 2024-03-16T11:55:29.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:55:28 smithi152 ceph-mon[36291]: pgmap v140: 97 pgs: 97 active+clean; 54 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 4.5 MiB/s wr, 411 op/s 2024-03-16T11:55:29.783 INFO:tasks.workunit.client.0.smithi029.stdout: 1 430049 11.54 MB/sec execute 74 sec latency 13.122 ms 2024-03-16T11:55:30.783 INFO:tasks.workunit.client.0.smithi029.stdout: 1 432774 11.61 MB/sec execute 75 sec latency 81.259 ms 2024-03-16T11:55:31.046 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:55:30 smithi154 ceph-mon[36320]: pgmap v141: 97 pgs: 97 active+clean; 54 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 2.9 MiB/s wr, 306 op/s 2024-03-16T11:55:31.087 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:55:30 smithi029 ceph-mon[31975]: pgmap v141: 97 pgs: 97 active+clean; 54 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 2.9 MiB/s wr, 306 op/s 2024-03-16T11:55:31.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:55:30 smithi152 ceph-mon[36291]: pgmap v141: 97 pgs: 97 active+clean; 54 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 2.9 MiB/s wr, 306 op/s 2024-03-16T11:55:31.783 INFO:tasks.workunit.client.0.smithi029.stdout: 1 434434 11.55 MB/sec execute 76 sec latency 17.584 ms 2024-03-16T11:55:32.783 INFO:tasks.workunit.client.0.smithi029.stdout: 1 436846 11.58 MB/sec execute 77 sec latency 14.829 ms 2024-03-16T11:55:33.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:55:32 smithi154 ceph-mon[36320]: pgmap v142: 97 pgs: 97 active+clean; 62 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 4.3 MiB/s wr, 406 op/s 2024-03-16T11:55:33.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:55:32 smithi029 ceph-mon[31975]: pgmap v142: 97 pgs: 97 active+clean; 62 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 4.3 MiB/s wr, 406 op/s 2024-03-16T11:55:33.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:55:32 smithi152 ceph-mon[36291]: pgmap v142: 97 pgs: 97 active+clean; 62 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 4.3 MiB/s wr, 406 op/s 2024-03-16T11:55:33.783 INFO:tasks.workunit.client.0.smithi029.stdout: 1 439662 11.62 MB/sec execute 78 sec latency 21.794 ms 2024-03-16T11:55:34.783 INFO:tasks.workunit.client.0.smithi029.stdout: 1 441430 11.60 MB/sec execute 79 sec latency 18.186 ms 2024-03-16T11:55:35.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:55:34 smithi154 ceph-mon[36320]: pgmap v143: 97 pgs: 97 active+clean; 62 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 2.9 MiB/s wr, 303 op/s 2024-03-16T11:55:35.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:55:34 smithi029 ceph-mon[31975]: pgmap v143: 97 pgs: 97 active+clean; 62 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 2.9 MiB/s wr, 303 op/s 2024-03-16T11:55:35.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:55:34 smithi152 ceph-mon[36291]: pgmap v143: 97 pgs: 97 active+clean; 62 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 2.9 MiB/s wr, 303 op/s 2024-03-16T11:55:35.783 INFO:tasks.workunit.client.0.smithi029.stdout: 1 443722 11.62 MB/sec execute 80 sec latency 43.837 ms 2024-03-16T11:55:36.451 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:36.449+0000 7f73b29f3640 1 -- 172.21.15.29:0/844278063 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f73ac107850 msgr2=0x7f73ac107c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:55:36.451 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:36.449+0000 7f73b29f3640 1 --2- 172.21.15.29:0/844278063 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f73ac107850 0x7f73ac107c30 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f739c009900 tx=0x7f739c02bc30 comp rx=0 tx=0).stop 2024-03-16T11:55:36.452 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:36.449+0000 7f73b29f3640 1 -- 172.21.15.29:0/844278063 shutdown_connections 2024-03-16T11:55:36.452 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:36.449+0000 7f73b29f3640 1 --2- 172.21.15.29:0/844278063 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f73ac06a980 0x7f73ac10e630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:55:36.452 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:36.449+0000 7f73b29f3640 1 --2- 172.21.15.29:0/844278063 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f73ac069fe0 0x7f73ac06a440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:55:36.452 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:36.449+0000 7f73b29f3640 1 --2- 172.21.15.29:0/844278063 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f73ac107850 0x7f73ac107c30 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:55:36.452 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:36.449+0000 7f73b29f3640 1 -- 172.21.15.29:0/844278063 >> 172.21.15.29:0/844278063 conn(0x7f73ac0774b0 msgr2=0x7f73ac0778c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:55:36.452 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:36.449+0000 7f73b29f3640 1 -- 172.21.15.29:0/844278063 shutdown_connections 2024-03-16T11:55:36.452 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:36.449+0000 7f73b29f3640 1 -- 172.21.15.29:0/844278063 wait complete. 2024-03-16T11:55:36.452 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:36.449+0000 7f73b29f3640 1 Processor -- start 2024-03-16T11:55:36.452 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:36.449+0000 7f73b29f3640 1 -- start start 2024-03-16T11:55:36.452 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:36.450+0000 7f73b29f3640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f73ac069fe0 0x7f73ac197f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:55:36.452 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:36.450+0000 7f73b29f3640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f73ac06a980 0x7f73ac198450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:55:36.452 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:36.450+0000 7f73b29f3640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f73ac107850 0x7f73ac19c7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:55:36.452 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:36.450+0000 7f73b29f3640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f73ac111b70 con 0x7f73ac107850 2024-03-16T11:55:36.453 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:36.450+0000 7f73b29f3640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f73ac1119f0 con 0x7f73ac06a980 2024-03-16T11:55:36.453 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:36.450+0000 7f73b29f3640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f73ac111cf0 con 0x7f73ac069fe0 2024-03-16T11:55:36.453 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:36.450+0000 7f73b0f90640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f73ac107850 0x7f73ac19c7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:55:36.453 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:36.450+0000 7f73abfff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f73ac069fe0 0x7f73ac197f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:55:36.453 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:36.450+0000 7f73b0f90640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f73ac107850 0x7f73ac19c7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43252/0 (socket says 172.21.15.29:43252) 2024-03-16T11:55:36.453 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:36.450+0000 7f73b0f90640 1 -- 172.21.15.29:0/31923775 learned_addr learned my addr 172.21.15.29:0/31923775 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:55:36.453 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:36.450+0000 7f73ab7fe640 1 --2- 172.21.15.29:0/31923775 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f73ac06a980 0x7f73ac198450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:55:36.453 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:36.450+0000 7f73b0f90640 1 -- 172.21.15.29:0/31923775 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f73ac069fe0 msgr2=0x7f73ac197f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:55:36.453 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:36.450+0000 7f73b0f90640 1 --2- 172.21.15.29:0/31923775 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f73ac069fe0 0x7f73ac197f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:55:36.453 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:36.450+0000 7f73b0f90640 1 -- 172.21.15.29:0/31923775 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f73ac06a980 msgr2=0x7f73ac198450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:55:36.453 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:36.450+0000 7f73b0f90640 1 --2- 172.21.15.29:0/31923775 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f73ac06a980 0x7f73ac198450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:55:36.454 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:36.450+0000 7f73b0f90640 1 -- 172.21.15.29:0/31923775 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73ac19cf00 con 0x7f73ac107850 2024-03-16T11:55:36.454 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:36.450+0000 7f73b0f90640 1 --2- 172.21.15.29:0/31923775 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f73ac107850 0x7f73ac19c7d0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f73a400eb10 tx=0x7f73a400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:55:36.454 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:36.450+0000 7f73a97fa640 1 -- 172.21.15.29:0/31923775 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73a4004b20 con 0x7f73ac107850 2024-03-16T11:55:36.454 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:36.450+0000 7f73a97fa640 1 -- 172.21.15.29:0/31923775 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f73a4004cc0 con 0x7f73ac107850 2024-03-16T11:55:36.454 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:36.450+0000 7f73b29f3640 1 -- 172.21.15.29:0/31923775 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f73ac19d1f0 con 0x7f73ac107850 2024-03-16T11:55:36.454 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:36.450+0000 7f73a97fa640 1 -- 172.21.15.29:0/31923775 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73a4010760 con 0x7f73ac107850 2024-03-16T11:55:36.454 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:36.450+0000 7f73b29f3640 1 -- 172.21.15.29:0/31923775 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f73ac1a4a80 con 0x7f73ac107850 2024-03-16T11:55:36.459 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:36.453+0000 7f73b55db740 1 Processor -- start 2024-03-16T11:55:36.460 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:36.453+0000 7f73a97fa640 1 -- 172.21.15.29:0/31923775 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7f73a4005020 con 0x7f73ac107850 2024-03-16T11:55:36.460 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:36.453+0000 7f73b55db740 1 -- start start 2024-03-16T11:55:36.460 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:36.453+0000 7f73b55db740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b2371abbd0 0x55b2371abfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:55:36.460 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:36.453+0000 7f73b55db740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b2371ac4f0 0x55b2371b96f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:55:36.460 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:36.453+0000 7f73b55db740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b2371b9c30 0x55b2371bc020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:55:36.460 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:36.453+0000 7f73b55db740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55b237110a40 con 0x55b2371b9c30 2024-03-16T11:55:36.460 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:36.453+0000 7f73b55db740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x55b2370b9920 con 0x55b2371abbd0 2024-03-16T11:55:36.460 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:36.453+0000 7f73b55db740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55b23710cae0 con 0x55b2371ac4f0 2024-03-16T11:55:36.460 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:36.454+0000 7f73abfff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b2371ac4f0 0x55b2371b96f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:55:36.460 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:36.454+0000 7f73ab7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b2371b9c30 0x55b2371bc020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:55:36.460 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:36.454+0000 7f73abfff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b2371ac4f0 0x55b2371b96f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.29:34806/0 (socket says 172.21.15.29:34806) 2024-03-16T11:55:36.460 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:36.454+0000 7f73abfff640 1 -- 172.21.15.29:0/2461246712 learned_addr learned my addr 172.21.15.29:0/2461246712 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:55:36.460 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:36.454+0000 7f73a97fa640 1 --2- 172.21.15.29:0/31923775 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f73980a2a90 0x7f73980a4f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:55:36.460 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:36.454+0000 7f73ab7fe640 1 --2- 172.21.15.29:0/2461246712 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b2371abbd0 0x55b2371abfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:55:36.461 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:36.454+0000 7f73b0f90640 1 --2- 172.21.15.29:0/31923775 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f73980a2a90 0x7f73980a4f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:55:36.461 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:36.454+0000 7f73abfff640 1 -- 172.21.15.29:0/2461246712 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b2371abbd0 msgr2=0x55b2371abfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:55:36.461 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:36.454+0000 7f73abfff640 1 --2- 172.21.15.29:0/2461246712 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b2371abbd0 0x55b2371abfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:55:36.461 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:36.454+0000 7f73abfff640 1 -- 172.21.15.29:0/2461246712 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b2371b9c30 msgr2=0x55b2371bc020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:55:36.461 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:36.454+0000 7f73abfff640 1 --2- 172.21.15.29:0/2461246712 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b2371b9c30 0x55b2371bc020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:55:36.461 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:36.454+0000 7f73a97fa640 1 -- 172.21.15.29:0/31923775 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7f73a40cf320 con 0x7f73ac107850 2024-03-16T11:55:36.461 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:36.454+0000 7f73abfff640 1 -- 172.21.15.29:0/2461246712 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b2370d1460 con 0x55b2371ac4f0 2024-03-16T11:55:36.461 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:36.454+0000 7f73abfff640 1 --2- 172.21.15.29:0/2461246712 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b2371ac4f0 0x55b2371b96f0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f739c02cfd0 tx=0x7f739c005c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c3bd7cf88bbca29a server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:55:36.461 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:36.454+0000 7f73b0f90640 1 --2- 172.21.15.29:0/31923775 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f73980a2a90 0x7f73980a4f50 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f73a40cd0a0 tx=0x7f73a406a7c0 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:55:36.461 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:36.454+0000 7f73a27fc640 1 -- 172.21.15.29:0/2461246712 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f739c042070 con 0x55b2371ac4f0 2024-03-16T11:55:36.461 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:36.454+0000 7f73a27fc640 1 -- 172.21.15.29:0/2461246712 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f739c02ca80 con 0x55b2371ac4f0 2024-03-16T11:55:36.461 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:36.454+0000 7f73a27fc640 1 -- 172.21.15.29:0/2461246712 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f739c004f00 con 0x55b2371ac4f0 2024-03-16T11:55:36.461 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:36.455+0000 7f73b55db740 1 -- 172.21.15.29:0/2461246712 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b2371ac4f0 msgr2=0x55b2371b96f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:55:36.462 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:36.455+0000 7f73b55db740 1 --2- 172.21.15.29:0/2461246712 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b2371ac4f0 0x55b2371b96f0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f739c02cfd0 tx=0x7f739c005c00 comp rx=0 tx=0).stop 2024-03-16T11:55:36.462 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:36.455+0000 7f73b55db740 1 -- 172.21.15.29:0/2461246712 shutdown_connections 2024-03-16T11:55:36.462 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:36.455+0000 7f73b55db740 1 --2- 172.21.15.29:0/2461246712 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b2371b9c30 0x55b2371bc020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:55:36.462 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:36.455+0000 7f73b55db740 1 --2- 172.21.15.29:0/2461246712 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b2371ac4f0 0x55b2371b96f0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:55:36.462 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:36.455+0000 7f73b55db740 1 --2- 172.21.15.29:0/2461246712 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b2371abbd0 0x55b2371abfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:55:36.462 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:36.455+0000 7f73b55db740 1 -- 172.21.15.29:0/2461246712 >> 172.21.15.29:0/2461246712 conn(0x55b2371a6a50 msgr2=0x55b2371a6e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:55:36.462 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:36.455+0000 7f73b55db740 1 -- 172.21.15.29:0/2461246712 shutdown_connections 2024-03-16T11:55:36.462 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:36.455+0000 7f73b55db740 1 -- 172.21.15.29:0/2461246712 wait complete. 2024-03-16T11:55:36.462 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:36.455+0000 7f73b55db740 1 -- start start 2024-03-16T11:55:36.462 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:36.456+0000 7f73b55db740 1 Processor -- start 2024-03-16T11:55:36.462 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:36.456+0000 7f73b55db740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b2371abbd0 0x55b2371b6c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:55:36.462 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:36.456+0000 7f73b55db740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b2371ac4f0 0x55b2371b71a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:55:36.462 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:36.456+0000 7f73b55db740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b2371b9c30 0x55b2371b76e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:55:36.462 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:36.456+0000 7f73b55db740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55b2371bc800 con 0x55b2371abbd0 2024-03-16T11:55:36.463 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:36.456+0000 7f73b55db740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x55b2370ccd70 con 0x55b2371ac4f0 2024-03-16T11:55:36.463 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:36.456+0000 7f73b55db740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55b2371bc980 con 0x55b2371b9c30 2024-03-16T11:55:36.463 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:36.456+0000 7f73b55db740 10 client.0 _ll_register_callbacks cb 0x55b236e24a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T11:55:36.463 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:36.456+0000 7f73ab7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b2371abbd0 0x55b2371b6c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:55:36.463 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:36.456+0000 7f73ab7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b2371abbd0 0x55b2371b6c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43280/0 (socket says 172.21.15.29:43280) 2024-03-16T11:55:36.463 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:36.456+0000 7f73ab7fe640 1 -- 172.21.15.29:0/3243919253 learned_addr learned my addr 172.21.15.29:0/3243919253 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:55:36.463 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:36.456+0000 7f73abfff640 1 --2- 172.21.15.29:0/3243919253 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b2371ac4f0 0x55b2371b71a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:55:36.463 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:36.456+0000 7f73ab7fe640 1 --2- 172.21.15.29:0/3243919253 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b2371b9c30 0x55b2371b76e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:55:36.463 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:36.456+0000 7f73abfff640 1 -- 172.21.15.29:0/3243919253 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b2371b9c30 msgr2=0x55b2371b76e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:55:36.463 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:36.456+0000 7f73abfff640 1 --2- 172.21.15.29:0/3243919253 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b2371b9c30 0x55b2371b76e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:55:36.463 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:36.456+0000 7f73abfff640 1 -- 172.21.15.29:0/3243919253 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b2371abbd0 msgr2=0x55b2371b6c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:55:36.463 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:36.456+0000 7f73abfff640 1 --2- 172.21.15.29:0/3243919253 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b2371abbd0 0x55b2371b6c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:55:36.463 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:36.456+0000 7f73abfff640 1 -- 172.21.15.29:0/3243919253 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b236e25210 con 0x55b2371ac4f0 2024-03-16T11:55:36.463 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:36.456+0000 7f73ab7fe640 1 --2- 172.21.15.29:0/3243919253 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b2371abbd0 0x55b2371b6c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:55:36.464 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:36.457+0000 7f73abfff640 1 --2- 172.21.15.29:0/3243919253 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b2371ac4f0 0x55b2371b71a0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f739c009900 tx=0x7f739c02c330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1b4673d56fa419d8 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:55:36.464 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:36.457+0000 7f738b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.152:3300/0 2024-03-16T11:55:36.464 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:36.457+0000 7f738b7fe640 1 -- 172.21.15.29:0/3243919253 <== mon.2 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f739c042040 con 0x55b2371ac4f0 2024-03-16T11:55:36.464 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:36.457+0000 7f73b55db740 1 -- 172.21.15.29:0/3243919253 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f739c02ca80 con 0x55b2371ac4f0 2024-03-16T11:55:36.464 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:36.457+0000 7f738b7fe640 1 -- 172.21.15.29:0/3243919253 <== mon.2 v2:172.21.15.152:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7f739c004250 con 0x55b2371ac4f0 2024-03-16T11:55:36.464 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:36.457+0000 7f738b7fe640 1 -- 172.21.15.29:0/3243919253 <== mon.2 v2:172.21.15.152:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f739c004450 con 0x55b2371ac4f0 2024-03-16T11:55:36.464 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:36.457+0000 7f73b55db740 10 client.25607 fetch_fsmap learned FSMap version 24 2024-03-16T11:55:36.464 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:36.457+0000 7f73b55db740 1 -- 172.21.15.29:0/3243919253 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55b236ef1620 con 0x55b2371ac4f0 2024-03-16T11:55:36.464 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:36.457+0000 7f738b7fe640 1 -- 172.21.15.29:0/3243919253 <== mon.2 v2:172.21.15.152:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f739c02ca80 con 0x55b2371ac4f0 2024-03-16T11:55:36.464 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:36.457+0000 7f738b7fe640 1 -- 172.21.15.29:0/3243919253 <== mon.2 v2:172.21.15.152:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7f739c003820 con 0x55b2371ac4f0 2024-03-16T11:55:36.464 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:36.457+0000 7f73b55db740 10 client.25607 fetch_fsmap finished waiting for FSMap version 24 2024-03-16T11:55:36.464 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:36.457+0000 7f73b55db740 10 client.25607 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T11:55:36.464 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:36.458+0000 7f73b55db740 20 client.25607 populate_metadata read hostname 'smithi029' 2024-03-16T11:55:36.464 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:36.458+0000 7f73b55db740 1 --2- 172.21.15.29:0/3243919253 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55b2371c9c90 0x55b2371cc080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:55:36.465 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:36.458+0000 7f73b55db740 4 client.25607 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T11:55:36.465 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:36.458+0000 7f73b55db740 1 -- 172.21.15.29:0/3243919253 --> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b23711df80 con 0x55b2371c9c90 2024-03-16T11:55:36.465 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:36.458+0000 7f73ab7fe640 1 --2- 172.21.15.29:0/3243919253 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55b2371c9c90 0x55b2371cc080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:55:36.465 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:36.458+0000 7f738b7fe640 10 client.25607 ms_handle_connect on v2:172.21.15.152:6838/1548731833 2024-03-16T11:55:36.465 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:36.458+0000 7f73ab7fe640 1 --2- 172.21.15.29:0/3243919253 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55b2371c9c90 0x55b2371cc080 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7dc3bfbe05b5702b server_cookie=352a1707e7f47a25 in_seq=0 out_seq=0 2024-03-16T11:55:36.466 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:36.461+0000 7f738b7fe640 1 -- 172.21.15.29:0/3243919253 <== mds.0 v2:172.21.15.152:6838/1548731833 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55b23711df80 con 0x55b2371c9c90 2024-03-16T11:55:36.466 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:36.461+0000 7f738b7fe640 10 client.25607 handle_command_reply: tid=0 2024-03-16T11:55:36.466 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:36.461+0000 7f73b55db740 1 client.25607 shutdown 2024-03-16T11:55:36.466 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:36.461+0000 7f73b55db740 1 -- 172.21.15.29:0/3243919253 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b2371ac4f0 msgr2=0x55b2371b71a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:55:36.466 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:36.461+0000 7f73b55db740 1 --2- 172.21.15.29:0/3243919253 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b2371ac4f0 0x55b2371b71a0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f739c009900 tx=0x7f739c02c330 comp rx=0 tx=0).stop 2024-03-16T11:55:36.466 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:36.461+0000 7f73b55db740 1 -- 172.21.15.29:0/3243919253 shutdown_connections 2024-03-16T11:55:36.467 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:36.461+0000 7f73b55db740 1 --2- 172.21.15.29:0/3243919253 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b2371b9c30 0x55b2371b76e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:55:36.467 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:36.461+0000 7f73b55db740 1 --2- 172.21.15.29:0/3243919253 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55b2371c9c90 0x55b2371cc080 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:55:36.467 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:36.461+0000 7f73b55db740 1 --2- 172.21.15.29:0/3243919253 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b2371ac4f0 0x55b2371b71a0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:55:36.467 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:36.461+0000 7f73b55db740 1 --2- 172.21.15.29:0/3243919253 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b2371abbd0 0x55b2371b6c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:55:36.467 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:36.461+0000 7f738b7fe640 0 client.25607 ms_handle_reset on v2:172.21.15.152:6838/1548731833 2024-03-16T11:55:36.467 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:36.461+0000 7f73b55db740 1 -- 172.21.15.29:0/3243919253 >> 172.21.15.29:0/3243919253 conn(0x55b2371a6a50 msgr2=0x55b2371a87a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:55:36.467 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:36.461+0000 7f73b55db740 1 -- 172.21.15.29:0/3243919253 shutdown_connections 2024-03-16T11:55:36.467 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:36.461+0000 7f73b55db740 1 -- 172.21.15.29:0/3243919253 wait complete. 2024-03-16T11:55:36.467 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:36.461+0000 7f73b55db740 20 client.25607 trim_cache size 0 max 16384 2024-03-16T11:55:36.481 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:36.476+0000 7f73b55db740 1 Processor -- start 2024-03-16T11:55:36.481 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:36.476+0000 7f73b55db740 1 -- start start 2024-03-16T11:55:36.481 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:36.476+0000 7f73b55db740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b2371abbd0 0x55b2371b9390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:55:36.481 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:36.476+0000 7f73b55db740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b2371ac4f0 0x55b2371c5610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:55:36.481 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:36.477+0000 7f73b55db740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b2371b9c30 0x55b2371a85b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:55:36.481 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:36.477+0000 7f73b55db740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f73ac1ad910 con 0x55b2371b9c30 2024-03-16T11:55:36.482 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:36.477+0000 7f73b55db740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f73ac1ab1e0 con 0x55b2371abbd0 2024-03-16T11:55:36.482 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:36.477+0000 7f73b55db740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f73ac1a9050 con 0x55b2371ac4f0 2024-03-16T11:55:36.482 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:36.477+0000 7f73abfff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b2371ac4f0 0x55b2371c5610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:55:36.482 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:36.477+0000 7f73ab7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b2371b9c30 0x55b2371a85b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:55:36.482 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:36.477+0000 7f73abfff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b2371ac4f0 0x55b2371c5610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.29:34826/0 (socket says 172.21.15.29:34826) 2024-03-16T11:55:36.482 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:36.477+0000 7f73abfff640 1 -- 172.21.15.29:0/2155336300 learned_addr learned my addr 172.21.15.29:0/2155336300 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:55:36.482 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:36.477+0000 7f73ab7fe640 1 --2- 172.21.15.29:0/2155336300 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b2371abbd0 0x55b2371b9390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:55:36.482 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:36.477+0000 7f73ab7fe640 1 -- 172.21.15.29:0/2155336300 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b2371ac4f0 msgr2=0x55b2371c5610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:55:36.482 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:36.477+0000 7f73ab7fe640 1 --2- 172.21.15.29:0/2155336300 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b2371ac4f0 0x55b2371c5610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:55:36.482 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:36.477+0000 7f73ab7fe640 1 -- 172.21.15.29:0/2155336300 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b2371abbd0 msgr2=0x55b2371b9390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:55:36.482 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:36.477+0000 7f73ab7fe640 1 --2- 172.21.15.29:0/2155336300 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b2371abbd0 0x55b2371b9390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:55:36.482 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:36.477+0000 7f73ab7fe640 1 -- 172.21.15.29:0/2155336300 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b2371ac320 con 0x55b2371b9c30 2024-03-16T11:55:36.482 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:36.477+0000 7f73ab7fe640 1 --2- 172.21.15.29:0/2155336300 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b2371b9c30 0x55b2371a85b0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x55b2371b80c0 tx=0x7f7394004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de977bbe5c1fb561 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:55:36.483 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:36.477+0000 7f73a1ffb640 1 -- 172.21.15.29:0/2155336300 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7394003d50 con 0x55b2371b9c30 2024-03-16T11:55:36.483 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:36.477+0000 7f73a1ffb640 1 -- 172.21.15.29:0/2155336300 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7394005500 con 0x55b2371b9c30 2024-03-16T11:55:36.483 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:36.477+0000 7f73a1ffb640 1 -- 172.21.15.29:0/2155336300 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7394004b90 con 0x55b2371b9c30 2024-03-16T11:55:36.483 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:36.478+0000 7f73b55db740 1 -- 172.21.15.29:0/2155336300 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b2371b9c30 msgr2=0x55b2371a85b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:55:36.483 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:36.478+0000 7f73b55db740 1 --2- 172.21.15.29:0/2155336300 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b2371b9c30 0x55b2371a85b0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x55b2371b80c0 tx=0x7f7394004730 comp rx=0 tx=0).stop 2024-03-16T11:55:36.483 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:36.478+0000 7f73b55db740 1 -- 172.21.15.29:0/2155336300 shutdown_connections 2024-03-16T11:55:36.483 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:36.478+0000 7f73b55db740 1 --2- 172.21.15.29:0/2155336300 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b2371b9c30 0x55b2371a85b0 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:55:36.483 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:36.478+0000 7f73b55db740 1 --2- 172.21.15.29:0/2155336300 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b2371ac4f0 0x55b2371c5610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:55:36.483 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:36.478+0000 7f73b55db740 1 --2- 172.21.15.29:0/2155336300 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b2371abbd0 0x55b2371b9390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:55:36.483 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:36.478+0000 7f73b55db740 1 -- 172.21.15.29:0/2155336300 >> 172.21.15.29:0/2155336300 conn(0x55b2371a6a50 msgr2=0x55b2371a7ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:55:36.483 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:36.478+0000 7f73b55db740 1 -- 172.21.15.29:0/2155336300 shutdown_connections 2024-03-16T11:55:36.483 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:36.478+0000 7f73b55db740 1 -- 172.21.15.29:0/2155336300 wait complete. 2024-03-16T11:55:36.483 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:36.478+0000 7f73b55db740 1 -- start start 2024-03-16T11:55:36.483 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:36.479+0000 7f73b55db740 1 Processor -- start 2024-03-16T11:55:36.483 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:36.479+0000 7f73b55db740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b2371abbd0 0x55b2371b38b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:55:36.484 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:36.479+0000 7f73b55db740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b2371ac4f0 0x55b2371b3df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:55:36.484 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:36.479+0000 7f73b55db740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b2371b9c30 0x55b2371b4330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:55:36.484 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:36.479+0000 7f73b55db740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f73ac1a96a0 con 0x55b2371b9c30 2024-03-16T11:55:36.484 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:36.479+0000 7f73b55db740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f73ac1aab10 con 0x55b2371ac4f0 2024-03-16T11:55:36.484 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:36.479+0000 7f73b55db740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f73ac1a9d70 con 0x55b2371abbd0 2024-03-16T11:55:36.484 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:36.479+0000 7f73b55db740 10 client.0 _ll_register_callbacks cb 0x7f7394009a10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T11:55:36.484 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:36.479+0000 7f73ab7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b2371b9c30 0x55b2371b4330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:55:36.484 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:36.479+0000 7f73ab7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b2371b9c30 0x55b2371b4330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43298/0 (socket says 172.21.15.29:43298) 2024-03-16T11:55:36.484 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:36.479+0000 7f73ab7fe640 1 -- 172.21.15.29:0/1860835095 learned_addr learned my addr 172.21.15.29:0/1860835095 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:55:36.484 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:36.479+0000 7f73abfff640 1 --2- 172.21.15.29:0/1860835095 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b2371ac4f0 0x55b2371b3df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:55:36.484 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:36.479+0000 7f73ab7fe640 1 --2- 172.21.15.29:0/1860835095 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b2371abbd0 0x55b2371b38b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:55:36.484 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:36.479+0000 7f73ab7fe640 1 -- 172.21.15.29:0/1860835095 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b2371abbd0 msgr2=0x55b2371b38b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:55:36.484 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:36.479+0000 7f73ab7fe640 1 --2- 172.21.15.29:0/1860835095 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b2371abbd0 0x55b2371b38b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:55:36.484 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:36.479+0000 7f73ab7fe640 1 -- 172.21.15.29:0/1860835095 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b2371ac4f0 msgr2=0x55b2371b3df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:55:36.485 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:36.479+0000 7f73ab7fe640 1 --2- 172.21.15.29:0/1860835095 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b2371ac4f0 0x55b2371b3df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:55:36.485 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:36.479+0000 7f73ab7fe640 1 -- 172.21.15.29:0/1860835095 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b23722ecd0 con 0x55b2371b9c30 2024-03-16T11:55:36.485 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:36.479+0000 7f73ab7fe640 1 --2- 172.21.15.29:0/1860835095 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b2371b9c30 0x55b2371b4330 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f7394011fd0 tx=0x7f7394013040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d02575915600d68e server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:55:36.485 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:36.479+0000 7f738b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-16T11:55:36.485 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:36.480+0000 7f738b7fe640 1 -- 172.21.15.29:0/1860835095 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73940047a0 con 0x55b2371b9c30 2024-03-16T11:55:36.485 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:36.480+0000 7f738b7fe640 1 -- 172.21.15.29:0/1860835095 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7f7394003d50 con 0x55b2371b9c30 2024-03-16T11:55:36.485 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:36.480+0000 7f73b55db740 1 -- 172.21.15.29:0/1860835095 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7394005500 con 0x55b2371b9c30 2024-03-16T11:55:36.485 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:36.480+0000 7f738b7fe640 1 -- 172.21.15.29:0/1860835095 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f7394011020 con 0x55b2371b9c30 2024-03-16T11:55:36.485 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:36.480+0000 7f73b55db740 10 client.15798 fetch_fsmap learned FSMap version 24 2024-03-16T11:55:36.485 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:36.480+0000 7f73b55db740 1 -- 172.21.15.29:0/1860835095 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55b23722ffc0 con 0x55b2371b9c30 2024-03-16T11:55:36.485 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:36.480+0000 7f738b7fe640 1 -- 172.21.15.29:0/1860835095 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7394005500 con 0x55b2371b9c30 2024-03-16T11:55:36.485 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:36.480+0000 7f738b7fe640 1 -- 172.21.15.29:0/1860835095 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7f7394032040 con 0x55b2371b9c30 2024-03-16T11:55:36.485 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:36.480+0000 7f73b55db740 10 client.15798 fetch_fsmap finished waiting for FSMap version 24 2024-03-16T11:55:36.485 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:36.480+0000 7f73b55db740 10 client.15798 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T11:55:36.485 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:36.480+0000 7f73b55db740 20 client.15798 populate_metadata read hostname 'smithi029' 2024-03-16T11:55:36.486 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:36.480+0000 7f73b55db740 1 --2- 172.21.15.29:0/1860835095 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55b237230640 0x55b237232a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:55:36.486 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:36.480+0000 7f73b55db740 4 client.15798 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-16T11:55:36.486 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:36.480+0000 7f73b55db740 1 -- 172.21.15.29:0/1860835095 --> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55b2371182d0 con 0x55b237230640 2024-03-16T11:55:36.486 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:36.480+0000 7f73abfff640 1 --2- 172.21.15.29:0/1860835095 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55b237230640 0x55b237232a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:55:36.486 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:36.481+0000 7f738b7fe640 10 client.15798 ms_handle_connect on v2:172.21.15.152:6838/1548731833 2024-03-16T11:55:36.486 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:36.481+0000 7f73abfff640 1 --2- 172.21.15.29:0/1860835095 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55b237230640 0x55b237232a60 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bd5dc2a237b639e8 server_cookie=d7c26dd8dc6d3988 in_seq=0 out_seq=0 2024-03-16T11:55:36.652 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:36.651+0000 7f738b7fe640 1 -- 172.21.15.29:0/1860835095 <== mds.0 v2:172.21.15.152:6838/1548731833 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55b2371182d0 con 0x55b237230640 2024-03-16T11:55:36.652 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:36.651+0000 7f738b7fe640 10 client.15798 handle_command_reply: tid=0 2024-03-16T11:55:36.652 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:36.651+0000 7f73b55db740 1 client.15798 shutdown 2024-03-16T11:55:36.652 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:36.651+0000 7f73b55db740 1 -- 172.21.15.29:0/1860835095 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b2371b9c30 msgr2=0x55b2371b4330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:55:36.652 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:36.651+0000 7f73b55db740 1 --2- 172.21.15.29:0/1860835095 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b2371b9c30 0x55b2371b4330 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f7394011fd0 tx=0x7f7394013040 comp rx=0 tx=0).stop 2024-03-16T11:55:36.652 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:36.651+0000 7f73b55db740 1 -- 172.21.15.29:0/1860835095 shutdown_connections 2024-03-16T11:55:36.652 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:36.651+0000 7f73b55db740 1 --2- 172.21.15.29:0/1860835095 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b2371b9c30 0x55b2371b4330 secure :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f7394011fd0 tx=0x7f7394013040 comp rx=0 tx=0).stop 2024-03-16T11:55:36.652 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:36.651+0000 7f73b55db740 1 --2- 172.21.15.29:0/1860835095 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55b237230640 0x55b237232a60 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:55:36.652 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:36.651+0000 7f73b55db740 1 --2- 172.21.15.29:0/1860835095 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b2371ac4f0 0x55b2371b3df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:55:36.652 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:36.651+0000 7f73b55db740 1 --2- 172.21.15.29:0/1860835095 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b2371abbd0 0x55b2371b38b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:55:36.652 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:36.651+0000 7f73b55db740 1 -- 172.21.15.29:0/1860835095 >> 172.21.15.29:0/1860835095 conn(0x55b2371a6a50 msgr2=0x55b2371a8730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:55:36.653 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:36.651+0000 7f738b7fe640 0 client.15798 ms_handle_reset on v2:172.21.15.152:6838/1548731833 2024-03-16T11:55:36.653 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:36.651+0000 7f73b55db740 1 -- 172.21.15.29:0/1860835095 shutdown_connections 2024-03-16T11:55:36.653 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:36.651+0000 7f73b55db740 1 -- 172.21.15.29:0/1860835095 wait complete. 2024-03-16T11:55:36.653 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:36.651+0000 7f73b55db740 20 client.15798 trim_cache size 0 max 16384 2024-03-16T11:55:36.653 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:{ 2024-03-16T11:55:36.653 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "message": "", 2024-03-16T11:55:36.653 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "return_code": 0 2024-03-16T11:55:36.653 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:} 2024-03-16T11:55:36.654 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:36.652+0000 7f73b29f3640 1 -- 172.21.15.29:0/31923775 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f73980a2a90 msgr2=0x7f73980a4f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:55:36.654 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:36.652+0000 7f73b29f3640 1 --2- 172.21.15.29:0/31923775 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f73980a2a90 0x7f73980a4f50 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f73a40cd0a0 tx=0x7f73a406a7c0 comp rx=0 tx=0).stop 2024-03-16T11:55:36.654 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:36.652+0000 7f73b29f3640 1 -- 172.21.15.29:0/31923775 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f73ac107850 msgr2=0x7f73ac19c7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:55:36.654 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:36.652+0000 7f73b29f3640 1 --2- 172.21.15.29:0/31923775 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f73ac107850 0x7f73ac19c7d0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f73a400eb10 tx=0x7f73a400c2d0 comp rx=0 tx=0).stop 2024-03-16T11:55:36.654 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:36.652+0000 7f73b29f3640 1 -- 172.21.15.29:0/31923775 shutdown_connections 2024-03-16T11:55:36.654 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:36.652+0000 7f73b29f3640 1 --2- 172.21.15.29:0/31923775 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f73980a2a90 0x7f73980a4f50 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:55:36.654 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:36.652+0000 7f73b29f3640 1 --2- 172.21.15.29:0/31923775 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f73ac107850 0x7f73ac19c7d0 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:55:36.655 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:36.652+0000 7f73b29f3640 1 --2- 172.21.15.29:0/31923775 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f73ac06a980 0x7f73ac198450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:55:36.655 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:36.652+0000 7f73b29f3640 1 --2- 172.21.15.29:0/31923775 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f73ac069fe0 0x7f73ac197f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:55:36.655 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:36.653+0000 7f73b29f3640 1 -- 172.21.15.29:0/31923775 >> 172.21.15.29:0/31923775 conn(0x7f73ac0774b0 msgr2=0x7f73ac0ff440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:55:36.655 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:36.653+0000 7f73b29f3640 1 -- 172.21.15.29:0/31923775 shutdown_connections 2024-03-16T11:55:36.655 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:36.653+0000 7f73b29f3640 1 -- 172.21.15.29:0/31923775 wait complete. 2024-03-16T11:55:36.783 INFO:tasks.workunit.client.0.smithi029.stdout: 1 445372 11.55 MB/sec execute 81 sec latency 103.864 ms 2024-03-16T11:55:37.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:55:36 smithi154 ceph-mon[36320]: pgmap v144: 97 pgs: 97 active+clean; 62 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 2.9 MiB/s wr, 303 op/s 2024-03-16T11:55:37.087 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:55:36 smithi029 ceph-mon[31975]: pgmap v144: 97 pgs: 97 active+clean; 62 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 2.9 MiB/s wr, 303 op/s 2024-03-16T11:55:37.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:55:36 smithi152 ceph-mon[36291]: pgmap v144: 97 pgs: 97 active+clean; 62 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 2.9 MiB/s wr, 303 op/s 2024-03-16T11:55:37.784 INFO:tasks.workunit.client.0.smithi029.stdout: 1 447507 11.57 MB/sec execute 82 sec latency 8.915 ms 2024-03-16T11:55:38.784 INFO:tasks.workunit.client.0.smithi029.stdout: 1 450168 11.60 MB/sec execute 83 sec latency 21.668 ms 2024-03-16T11:55:39.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:55:38 smithi154 ceph-mon[36320]: pgmap v145: 97 pgs: 97 active+clean; 66 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 4.2 MiB/s wr, 405 op/s 2024-03-16T11:55:39.087 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:55:38 smithi029 ceph-mon[31975]: pgmap v145: 97 pgs: 97 active+clean; 66 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 4.2 MiB/s wr, 405 op/s 2024-03-16T11:55:39.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:55:38 smithi152 ceph-mon[36291]: pgmap v145: 97 pgs: 97 active+clean; 66 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 4.2 MiB/s wr, 405 op/s 2024-03-16T11:55:39.784 INFO:tasks.workunit.client.0.smithi029.stdout: 1 451905 11.59 MB/sec execute 84 sec latency 25.557 ms 2024-03-16T11:55:40.784 INFO:tasks.workunit.client.0.smithi029.stdout: 1 454403 11.62 MB/sec execute 85 sec latency 32.652 ms 2024-03-16T11:55:41.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:55:40 smithi154 ceph-mon[36320]: pgmap v146: 97 pgs: 97 active+clean; 66 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 2.7 MiB/s wr, 201 op/s 2024-03-16T11:55:41.087 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:55:40 smithi029 ceph-mon[31975]: pgmap v146: 97 pgs: 97 active+clean; 66 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 2.7 MiB/s wr, 201 op/s 2024-03-16T11:55:41.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:55:40 smithi152 ceph-mon[36291]: pgmap v146: 97 pgs: 97 active+clean; 66 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 2.7 MiB/s wr, 201 op/s 2024-03-16T11:55:41.784 INFO:tasks.workunit.client.0.smithi029.stdout: 1 456423 11.56 MB/sec execute 86 sec latency 62.990 ms 2024-03-16T11:55:42.784 INFO:tasks.workunit.client.0.smithi029.stdout: 1 458264 11.57 MB/sec execute 87 sec latency 21.519 ms 2024-03-16T11:55:43.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:55:42 smithi154 ceph-mon[36320]: pgmap v147: 97 pgs: 97 active+clean; 57 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 4.1 MiB/s wr, 403 op/s 2024-03-16T11:55:43.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:55:42 smithi029 ceph-mon[31975]: pgmap v147: 97 pgs: 97 active+clean; 57 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 4.1 MiB/s wr, 403 op/s 2024-03-16T11:55:43.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:55:42 smithi152 ceph-mon[36291]: pgmap v147: 97 pgs: 97 active+clean; 57 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 4.1 MiB/s wr, 403 op/s 2024-03-16T11:55:43.784 INFO:tasks.workunit.client.0.smithi029.stdout: 1 460033 11.74 MB/sec execute 88 sec latency 78.652 ms 2024-03-16T11:55:44.784 INFO:tasks.workunit.client.0.smithi029.stdout: 1 462377 11.76 MB/sec execute 89 sec latency 56.565 ms 2024-03-16T11:55:45.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:55:44 smithi154 ceph-mon[36320]: pgmap v148: 97 pgs: 97 active+clean; 57 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 2.7 MiB/s wr, 303 op/s 2024-03-16T11:55:45.087 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:55:44 smithi029 ceph-mon[31975]: pgmap v148: 97 pgs: 97 active+clean; 57 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 2.7 MiB/s wr, 303 op/s 2024-03-16T11:55:45.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:55:44 smithi152 ceph-mon[36291]: pgmap v148: 97 pgs: 97 active+clean; 57 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 2.7 MiB/s wr, 303 op/s 2024-03-16T11:55:45.784 INFO:tasks.workunit.client.0.smithi029.stdout: 1 465096 11.78 MB/sec execute 90 sec latency 21.918 ms 2024-03-16T11:55:46.784 INFO:tasks.workunit.client.0.smithi029.stdout: 1 466353 11.72 MB/sec execute 91 sec latency 130.748 ms 2024-03-16T11:55:47.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:55:46 smithi154 ceph-mon[36320]: pgmap v149: 97 pgs: 97 active+clean; 57 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 2.7 MiB/s wr, 303 op/s 2024-03-16T11:55:47.087 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:55:46 smithi029 ceph-mon[31975]: pgmap v149: 97 pgs: 97 active+clean; 57 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 2.7 MiB/s wr, 303 op/s 2024-03-16T11:55:47.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:55:46 smithi152 ceph-mon[36291]: pgmap v149: 97 pgs: 97 active+clean; 57 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 2.7 MiB/s wr, 303 op/s 2024-03-16T11:55:47.784 INFO:tasks.workunit.client.0.smithi029.stdout: 1 469242 11.79 MB/sec execute 92 sec latency 20.363 ms 2024-03-16T11:55:48.784 INFO:tasks.workunit.client.0.smithi029.stdout: 1 471452 11.74 MB/sec execute 93 sec latency 27.975 ms 2024-03-16T11:55:49.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:55:48 smithi154 ceph-mon[36320]: pgmap v150: 97 pgs: 97 active+clean; 65 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 4.3 MiB/s wr, 398 op/s 2024-03-16T11:55:49.087 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:55:48 smithi029 ceph-mon[31975]: pgmap v150: 97 pgs: 97 active+clean; 65 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 4.3 MiB/s wr, 398 op/s 2024-03-16T11:55:49.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:55:48 smithi152 ceph-mon[36291]: pgmap v150: 97 pgs: 97 active+clean; 65 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 4.3 MiB/s wr, 398 op/s 2024-03-16T11:55:49.745 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.746+0000 7f428dcbf640 1 -- 172.21.15.29:0/2578894351 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4288069800 msgr2=0x7f428810e670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:55:49.745 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.746+0000 7f428dcbf640 1 --2- 172.21.15.29:0/2578894351 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4288069800 0x7f428810e670 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f4280009ca0 tx=0x7f428002c0f0 comp rx=0 tx=0).stop 2024-03-16T11:55:49.746 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.746+0000 7f428dcbf640 1 -- 172.21.15.29:0/2578894351 shutdown_connections 2024-03-16T11:55:49.746 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.746+0000 7f428dcbf640 1 --2- 172.21.15.29:0/2578894351 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4288069800 0x7f428810e670 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:55:49.746 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.746+0000 7f428dcbf640 1 --2- 172.21.15.29:0/2578894351 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4288068e60 0x7f42880692c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:55:49.746 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.746+0000 7f428dcbf640 1 --2- 172.21.15.29:0/2578894351 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f42881078d0 0x7f4288107cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:55:49.746 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.746+0000 7f428dcbf640 1 -- 172.21.15.29:0/2578894351 >> 172.21.15.29:0/2578894351 conn(0x7f4288077530 msgr2=0x7f4288077940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:55:49.746 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.746+0000 7f428dcbf640 1 -- 172.21.15.29:0/2578894351 shutdown_connections 2024-03-16T11:55:49.746 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.746+0000 7f428dcbf640 1 -- 172.21.15.29:0/2578894351 wait complete. 2024-03-16T11:55:49.746 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.746+0000 7f428dcbf640 1 Processor -- start 2024-03-16T11:55:49.748 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.747+0000 7f428dcbf640 1 -- start start 2024-03-16T11:55:49.748 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.747+0000 7f428dcbf640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4288068e60 0x7f4288197fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:55:49.748 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.747+0000 7f428dcbf640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4288069800 0x7f4288198510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:55:49.748 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.747+0000 7f428dcbf640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f42881078d0 0x7f428819c840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:55:49.748 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.747+0000 7f428dcbf640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f4288111c50 con 0x7f4288069800 2024-03-16T11:55:49.749 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.747+0000 7f428dcbf640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f4288111ad0 con 0x7f42881078d0 2024-03-16T11:55:49.749 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.747+0000 7f428dcbf640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f4288111dd0 con 0x7f4288068e60 2024-03-16T11:55:49.749 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.747+0000 7f4286ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4288069800 0x7f4288198510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:55:49.749 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.747+0000 7f4286ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4288069800 0x7f4288198510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49252/0 (socket says 172.21.15.29:49252) 2024-03-16T11:55:49.749 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.747+0000 7f4286ffd640 1 -- 172.21.15.29:0/1639637348 learned_addr learned my addr 172.21.15.29:0/1639637348 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:55:49.749 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.747+0000 7f42877fe640 1 --2- 172.21.15.29:0/1639637348 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4288068e60 0x7f4288197fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:55:49.749 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.747+0000 7f4287fff640 1 --2- 172.21.15.29:0/1639637348 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f42881078d0 0x7f428819c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:55:49.749 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.747+0000 7f4286ffd640 1 -- 172.21.15.29:0/1639637348 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4288068e60 msgr2=0x7f4288197fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:55:49.749 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.747+0000 7f4286ffd640 1 --2- 172.21.15.29:0/1639637348 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4288068e60 0x7f4288197fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:55:49.749 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.747+0000 7f4286ffd640 1 -- 172.21.15.29:0/1639637348 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f42881078d0 msgr2=0x7f428819c840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:55:49.749 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.747+0000 7f4286ffd640 1 --2- 172.21.15.29:0/1639637348 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f42881078d0 0x7f428819c840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:55:49.749 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.747+0000 7f4286ffd640 1 -- 172.21.15.29:0/1639637348 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f428819cf70 con 0x7f4288069800 2024-03-16T11:55:49.749 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.747+0000 7f4287fff640 1 --2- 172.21.15.29:0/1639637348 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f42881078d0 0x7f428819c840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:55:49.749 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.747+0000 7f42877fe640 1 --2- 172.21.15.29:0/1639637348 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4288068e60 0x7f4288197fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:55:49.750 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.748+0000 7f4286ffd640 1 --2- 172.21.15.29:0/1639637348 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4288069800 0x7f4288198510 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f427c00d950 tx=0x7f427c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:55:49.750 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.748+0000 7f4284ff9640 1 -- 172.21.15.29:0/1639637348 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f427c017030 con 0x7f4288069800 2024-03-16T11:55:49.750 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.748+0000 7f428dcbf640 1 -- 172.21.15.29:0/1639637348 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f428819d260 con 0x7f4288069800 2024-03-16T11:55:49.750 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.748+0000 7f4284ff9640 1 -- 172.21.15.29:0/1639637348 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f427c004020 con 0x7f4288069800 2024-03-16T11:55:49.750 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.748+0000 7f4284ff9640 1 -- 172.21.15.29:0/1639637348 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f427c0047a0 con 0x7f4288069800 2024-03-16T11:55:49.750 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.748+0000 7f428dcbf640 1 -- 172.21.15.29:0/1639637348 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f42881a4af0 con 0x7f4288069800 2024-03-16T11:55:49.751 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.750+0000 7f4284ff9640 1 -- 172.21.15.29:0/1639637348 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7f427c0041d0 con 0x7f4288069800 2024-03-16T11:55:49.751 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.751+0000 7f4290981740 1 Processor -- start 2024-03-16T11:55:49.751 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.751+0000 7f4290981740 1 -- start start 2024-03-16T11:55:49.751 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.751+0000 7f4284ff9640 1 --2- 172.21.15.29:0/1639637348 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f42700a2a10 0x7f42700a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:55:49.751 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.751+0000 7f4290981740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562b078f2130 0x562b078f2510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:55:49.751 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.751+0000 7f4290981740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562b078f2a50 0x562b078fb2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:55:49.758 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.751+0000 7f4290981740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562b078fb820 0x562b078fdc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:55:49.758 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.751+0000 7f4284ff9640 1 -- 172.21.15.29:0/1639637348 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7f427c0ce840 con 0x7f4288069800 2024-03-16T11:55:49.758 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.751+0000 7f4290981740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x562b07856a40 con 0x562b078f2130 2024-03-16T11:55:49.758 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.751+0000 7f4290981740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x562b077ff920 con 0x562b078f2a50 2024-03-16T11:55:49.758 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.751+0000 7f4290981740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x562b07852ae0 con 0x562b078fb820 2024-03-16T11:55:49.758 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.751+0000 7f4287fff640 1 --2- 172.21.15.29:0/1639637348 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f42700a2a10 0x7f42700a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:55:49.758 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.751+0000 7f42877fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562b078f2130 0x562b078f2510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:55:49.758 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.751+0000 7f42877fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562b078f2130 0x562b078f2510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49260/0 (socket says 172.21.15.29:49260) 2024-03-16T11:55:49.759 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.751+0000 7f42877fe640 1 -- 172.21.15.29:0/1087451200 learned_addr learned my addr 172.21.15.29:0/1087451200 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:55:49.759 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.751+0000 7f4286ffd640 1 --2- 172.21.15.29:0/1087451200 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562b078f2a50 0x562b078fb2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:55:49.759 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.751+0000 7f4287fff640 1 --2- 172.21.15.29:0/1087451200 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562b078fb820 0x562b078fdc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:55:49.759 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.751+0000 7f42877fe640 1 -- 172.21.15.29:0/1087451200 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562b078fb820 msgr2=0x562b078fdc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:55:49.759 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.751+0000 7f42877fe640 1 --2- 172.21.15.29:0/1087451200 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562b078fb820 0x562b078fdc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:55:49.759 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.751+0000 7f42877fe640 1 -- 172.21.15.29:0/1087451200 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562b078f2a50 msgr2=0x562b078fb2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:55:49.759 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.751+0000 7f42877fe640 1 --2- 172.21.15.29:0/1087451200 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562b078f2a50 0x562b078fb2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:55:49.759 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.751+0000 7f42877fe640 1 -- 172.21.15.29:0/1087451200 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562b07817460 con 0x562b078f2130 2024-03-16T11:55:49.759 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.752+0000 7f4287fff640 1 --2- 172.21.15.29:0/1639637348 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f42700a2a10 0x7f42700a4ed0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f4280004af0 tx=0x7f428002cec0 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:55:49.759 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.752+0000 7f42877fe640 1 --2- 172.21.15.29:0/1087451200 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562b078f2130 0x562b078f2510 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f42780114c0 tx=0x7f4278011980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86569ec5a84b4aee server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:55:49.759 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.752+0000 7f426dffb640 1 -- 172.21.15.29:0/1087451200 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4278004020 con 0x562b078f2130 2024-03-16T11:55:49.759 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.752+0000 7f426dffb640 1 -- 172.21.15.29:0/1087451200 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f42780041c0 con 0x562b078f2130 2024-03-16T11:55:49.759 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.752+0000 7f426dffb640 1 -- 172.21.15.29:0/1087451200 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f427800f900 con 0x562b078f2130 2024-03-16T11:55:49.759 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.753+0000 7f4290981740 1 -- 172.21.15.29:0/1087451200 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562b078f2130 msgr2=0x562b078f2510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:55:49.760 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.753+0000 7f4290981740 1 --2- 172.21.15.29:0/1087451200 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562b078f2130 0x562b078f2510 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f42780114c0 tx=0x7f4278011980 comp rx=0 tx=0).stop 2024-03-16T11:55:49.760 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.753+0000 7f4290981740 1 -- 172.21.15.29:0/1087451200 shutdown_connections 2024-03-16T11:55:49.760 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.753+0000 7f4290981740 1 --2- 172.21.15.29:0/1087451200 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562b078fb820 0x562b078fdc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:55:49.760 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.753+0000 7f4290981740 1 --2- 172.21.15.29:0/1087451200 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562b078f2a50 0x562b078fb2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:55:49.760 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.753+0000 7f4290981740 1 --2- 172.21.15.29:0/1087451200 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562b078f2130 0x562b078f2510 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:55:49.760 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.753+0000 7f4290981740 1 -- 172.21.15.29:0/1087451200 >> 172.21.15.29:0/1087451200 conn(0x562b078eca50 msgr2=0x562b078ece30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:55:49.760 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.753+0000 7f4290981740 1 -- 172.21.15.29:0/1087451200 shutdown_connections 2024-03-16T11:55:49.760 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.753+0000 7f4290981740 1 -- 172.21.15.29:0/1087451200 wait complete. 2024-03-16T11:55:49.760 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.753+0000 7f4290981740 1 -- start start 2024-03-16T11:55:49.760 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.753+0000 7f4290981740 1 Processor -- start 2024-03-16T11:55:49.760 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.753+0000 7f4290981740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562b078f2130 0x562b078f8920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:55:49.760 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.753+0000 7f4290981740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562b078f2a50 0x562b078f8e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:55:49.760 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.754+0000 7f4290981740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562b078fb820 0x562b078f93a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:55:49.760 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.754+0000 7f4290981740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x562b07902860 con 0x562b078f2130 2024-03-16T11:55:49.761 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.754+0000 7f4290981740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x562b07812d70 con 0x562b078f2a50 2024-03-16T11:55:49.761 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.754+0000 7f4290981740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x562b079029e0 con 0x562b078fb820 2024-03-16T11:55:49.761 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.754+0000 7f4290981740 10 client.0 _ll_register_callbacks cb 0x562b0756aa00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T11:55:49.761 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.754+0000 7f42877fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562b078f2130 0x562b078f8920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:55:49.761 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.754+0000 7f42877fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562b078f2130 0x562b078f8920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49270/0 (socket says 172.21.15.29:49270) 2024-03-16T11:55:49.761 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.754+0000 7f42877fe640 1 -- 172.21.15.29:0/3390906403 learned_addr learned my addr 172.21.15.29:0/3390906403 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:55:49.761 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.754+0000 7f4286ffd640 1 --2- 172.21.15.29:0/3390906403 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562b078f2a50 0x562b078f8e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:55:49.761 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.754+0000 7f4287fff640 1 --2- 172.21.15.29:0/3390906403 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562b078fb820 0x562b078f93a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:55:49.761 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.754+0000 7f42877fe640 1 -- 172.21.15.29:0/3390906403 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562b078fb820 msgr2=0x562b078f93a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:55:49.761 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.754+0000 7f42877fe640 1 --2- 172.21.15.29:0/3390906403 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562b078fb820 0x562b078f93a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:55:49.761 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.754+0000 7f42877fe640 1 -- 172.21.15.29:0/3390906403 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562b078f2a50 msgr2=0x562b078f8e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:55:49.761 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.754+0000 7f42877fe640 1 --2- 172.21.15.29:0/3390906403 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562b078f2a50 0x562b078f8e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:55:49.761 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.754+0000 7f42877fe640 1 -- 172.21.15.29:0/3390906403 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562b0756b210 con 0x562b078f2130 2024-03-16T11:55:49.761 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.754+0000 7f4286ffd640 1 --2- 172.21.15.29:0/3390906403 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562b078f2a50 0x562b078f8e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:55:49.762 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.754+0000 7f42877fe640 1 --2- 172.21.15.29:0/3390906403 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562b078f2130 0x562b078f8920 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f4278005f10 tx=0x7f4278005750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7dd918c678ab816 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:55:49.762 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.754+0000 7f4266ffd640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-16T11:55:49.762 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.754+0000 7f4266ffd640 1 -- 172.21.15.29:0/3390906403 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f427801d040 con 0x562b078f2130 2024-03-16T11:55:49.762 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.754+0000 7f4266ffd640 1 -- 172.21.15.29:0/3390906403 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7f4278003f70 con 0x562b078f2130 2024-03-16T11:55:49.762 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.754+0000 7f4290981740 1 -- 172.21.15.29:0/3390906403 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f42780041c0 con 0x562b078f2130 2024-03-16T11:55:49.762 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.755+0000 7f4266ffd640 1 -- 172.21.15.29:0/3390906403 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f4278004370 con 0x562b078f2130 2024-03-16T11:55:49.762 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.755+0000 7f4290981740 10 client.15810 fetch_fsmap learned FSMap version 24 2024-03-16T11:55:49.762 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.755+0000 7f4290981740 1 -- 172.21.15.29:0/3390906403 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x562b07637620 con 0x562b078f2130 2024-03-16T11:55:49.762 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.755+0000 7f4266ffd640 1 -- 172.21.15.29:0/3390906403 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f42780041c0 con 0x562b078f2130 2024-03-16T11:55:49.762 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.755+0000 7f4266ffd640 1 -- 172.21.15.29:0/3390906403 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7f4278020070 con 0x562b078f2130 2024-03-16T11:55:49.762 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.755+0000 7f4290981740 10 client.15810 fetch_fsmap finished waiting for FSMap version 24 2024-03-16T11:55:49.762 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.755+0000 7f4290981740 10 client.15810 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T11:55:49.762 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.755+0000 7f4290981740 20 client.15810 populate_metadata read hostname 'smithi029' 2024-03-16T11:55:49.763 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.755+0000 7f4290981740 1 --2- 172.21.15.29:0/3390906403 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x562b0790fde0 0x562b079121d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:55:49.763 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.755+0000 7f4290981740 4 client.15810 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T11:55:49.763 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.755+0000 7f4290981740 1 -- 172.21.15.29:0/3390906403 --> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562b07863f80 con 0x562b0790fde0 2024-03-16T11:55:49.763 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.755+0000 7f4286ffd640 1 --2- 172.21.15.29:0/3390906403 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x562b0790fde0 0x562b079121d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:55:49.763 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.756+0000 7f4266ffd640 10 client.15810 ms_handle_connect on v2:172.21.15.152:6838/1548731833 2024-03-16T11:55:49.763 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.756+0000 7f4286ffd640 1 --2- 172.21.15.29:0/3390906403 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x562b0790fde0 0x562b079121d0 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=af0de149344e3ada server_cookie=bfb694d658f39b58 in_seq=0 out_seq=0 2024-03-16T11:55:49.763 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.758+0000 7f4266ffd640 1 -- 172.21.15.29:0/3390906403 <== mds.0 v2:172.21.15.152:6838/1548731833 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x562b07863f80 con 0x562b0790fde0 2024-03-16T11:55:49.763 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.758+0000 7f4266ffd640 10 client.15810 handle_command_reply: tid=0 2024-03-16T11:55:49.763 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.758+0000 7f4290981740 1 client.15810 shutdown 2024-03-16T11:55:49.763 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.758+0000 7f4290981740 1 -- 172.21.15.29:0/3390906403 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562b078f2130 msgr2=0x562b078f8920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:55:49.763 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.758+0000 7f4290981740 1 --2- 172.21.15.29:0/3390906403 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562b078f2130 0x562b078f8920 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f4278005f10 tx=0x7f4278005750 comp rx=0 tx=0).stop 2024-03-16T11:55:49.763 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.758+0000 7f4290981740 1 -- 172.21.15.29:0/3390906403 shutdown_connections 2024-03-16T11:55:49.763 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.758+0000 7f4290981740 1 --2- 172.21.15.29:0/3390906403 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562b078fb820 0x562b078f93a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:55:49.763 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.758+0000 7f4290981740 1 --2- 172.21.15.29:0/3390906403 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x562b0790fde0 0x562b079121d0 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:55:49.764 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.758+0000 7f4290981740 1 --2- 172.21.15.29:0/3390906403 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562b078f2a50 0x562b078f8e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:55:49.764 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.758+0000 7f4290981740 1 --2- 172.21.15.29:0/3390906403 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562b078f2130 0x562b078f8920 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:55:49.764 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.758+0000 7f4266ffd640 0 client.15810 ms_handle_reset on v2:172.21.15.152:6838/1548731833 2024-03-16T11:55:49.764 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.758+0000 7f4290981740 1 -- 172.21.15.29:0/3390906403 >> 172.21.15.29:0/3390906403 conn(0x562b078eca50 msgr2=0x562b078ee7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:55:49.764 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.758+0000 7f4290981740 1 -- 172.21.15.29:0/3390906403 shutdown_connections 2024-03-16T11:55:49.764 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.758+0000 7f4290981740 1 -- 172.21.15.29:0/3390906403 wait complete. 2024-03-16T11:55:49.764 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.758+0000 7f4290981740 20 client.15810 trim_cache size 0 max 16384 2024-03-16T11:55:49.775 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.774+0000 7f4290981740 1 Processor -- start 2024-03-16T11:55:49.775 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.774+0000 7f4290981740 1 -- start start 2024-03-16T11:55:49.775 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.774+0000 7f4290981740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562b078f2130 0x562b0790b600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:55:49.775 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.774+0000 7f4290981740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562b078f2a50 0x562b07917960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:55:49.775 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.774+0000 7f4290981740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562b078fb820 0x562b078eb320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:55:49.775 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.774+0000 7f4290981740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f42881ada50 con 0x562b078f2a50 2024-03-16T11:55:49.776 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.774+0000 7f4290981740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f42881ab320 con 0x562b078f2130 2024-03-16T11:55:49.776 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.774+0000 7f4290981740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f42881a9170 con 0x562b078fb820 2024-03-16T11:55:49.776 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.774+0000 7f4286ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562b078f2a50 0x562b07917960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:55:49.776 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.774+0000 7f4286ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562b078f2a50 0x562b07917960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49284/0 (socket says 172.21.15.29:49284) 2024-03-16T11:55:49.776 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.774+0000 7f4286ffd640 1 -- 172.21.15.29:0/3120169917 learned_addr learned my addr 172.21.15.29:0/3120169917 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:55:49.776 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.774+0000 7f4287fff640 1 --2- 172.21.15.29:0/3120169917 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562b078fb820 0x562b078eb320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:55:49.776 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.774+0000 7f42877fe640 1 --2- 172.21.15.29:0/3120169917 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562b078f2130 0x562b0790b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:55:49.776 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.774+0000 7f4287fff640 1 -- 172.21.15.29:0/3120169917 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562b078f2130 msgr2=0x562b0790b600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:55:49.776 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.774+0000 7f4287fff640 1 --2- 172.21.15.29:0/3120169917 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562b078f2130 0x562b0790b600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:55:49.776 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.774+0000 7f4287fff640 1 -- 172.21.15.29:0/3120169917 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562b078f2a50 msgr2=0x562b07917960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:55:49.776 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.774+0000 7f4287fff640 1 --2- 172.21.15.29:0/3120169917 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562b078f2a50 0x562b07917960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:55:49.776 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.774+0000 7f4287fff640 1 -- 172.21.15.29:0/3120169917 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562b078f2880 con 0x562b078fb820 2024-03-16T11:55:49.776 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.774+0000 7f4286ffd640 1 --2- 172.21.15.29:0/3120169917 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562b078f2a50 0x562b07917960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:55:49.776 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.775+0000 7f4287fff640 1 --2- 172.21.15.29:0/3120169917 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562b078fb820 0x562b078eb320 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f428002c700 tx=0x7f428002e050 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7e3e46dadc44560b server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:55:49.777 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.775+0000 7f426d7fa640 1 -- 172.21.15.29:0/3120169917 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4280039040 con 0x562b078fb820 2024-03-16T11:55:49.777 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.775+0000 7f426d7fa640 1 -- 172.21.15.29:0/3120169917 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4280052070 con 0x562b078fb820 2024-03-16T11:55:49.777 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.775+0000 7f426d7fa640 1 -- 172.21.15.29:0/3120169917 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4280004040 con 0x562b078fb820 2024-03-16T11:55:49.777 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.776+0000 7f4290981740 1 -- 172.21.15.29:0/3120169917 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562b078fb820 msgr2=0x562b078eb320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:55:49.777 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.776+0000 7f4290981740 1 --2- 172.21.15.29:0/3120169917 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562b078fb820 0x562b078eb320 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f428002c700 tx=0x7f428002e050 comp rx=0 tx=0).stop 2024-03-16T11:55:49.777 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.776+0000 7f4290981740 1 -- 172.21.15.29:0/3120169917 shutdown_connections 2024-03-16T11:55:49.777 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.776+0000 7f4290981740 1 --2- 172.21.15.29:0/3120169917 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562b078fb820 0x562b078eb320 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:55:49.777 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.776+0000 7f4290981740 1 --2- 172.21.15.29:0/3120169917 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562b078f2a50 0x562b07917960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:55:49.777 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.776+0000 7f4290981740 1 --2- 172.21.15.29:0/3120169917 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562b078f2130 0x562b0790b600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:55:49.777 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.776+0000 7f4290981740 1 -- 172.21.15.29:0/3120169917 >> 172.21.15.29:0/3120169917 conn(0x562b078eca50 msgr2=0x562b078ece30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:55:49.777 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.776+0000 7f4290981740 1 -- 172.21.15.29:0/3120169917 shutdown_connections 2024-03-16T11:55:49.777 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.776+0000 7f4290981740 1 -- 172.21.15.29:0/3120169917 wait complete. 2024-03-16T11:55:49.777 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.776+0000 7f4290981740 1 -- start start 2024-03-16T11:55:49.779 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.776+0000 7f4290981740 1 Processor -- start 2024-03-16T11:55:49.779 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.777+0000 7f4290981740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562b078f2130 0x562b07903570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:55:49.779 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.777+0000 7f4290981740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562b078f2a50 0x562b07903ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:55:49.779 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.777+0000 7f4290981740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562b078fb820 0x562b07903ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:55:49.779 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.777+0000 7f4290981740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f42881a97e0 con 0x562b078fb820 2024-03-16T11:55:49.779 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.777+0000 7f4290981740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f42881aac50 con 0x562b078f2130 2024-03-16T11:55:49.779 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.777+0000 7f4290981740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f42881a9eb0 con 0x562b078f2a50 2024-03-16T11:55:49.779 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.777+0000 7f4290981740 10 client.0 _ll_register_callbacks cb 0x7f427c097170 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T11:55:49.779 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.777+0000 7f4287fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562b078fb820 0x562b07903ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:55:49.779 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.777+0000 7f4286ffd640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562b078f2a50 0x562b07903ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:55:49.779 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.777+0000 7f4286ffd640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562b078f2a50 0x562b07903ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.29:43658/0 (socket says 172.21.15.29:43658) 2024-03-16T11:55:49.780 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.777+0000 7f4287fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562b078fb820 0x562b07903ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49294/0 (socket says 172.21.15.29:49294) 2024-03-16T11:55:49.780 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.777+0000 7f4286ffd640 1 -- 172.21.15.29:0/271391555 learned_addr learned my addr 172.21.15.29:0/271391555 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:55:49.780 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.777+0000 7f42877fe640 1 --2- 172.21.15.29:0/271391555 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562b078f2130 0x562b07903570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:55:49.780 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.777+0000 7f4286ffd640 1 -- 172.21.15.29:0/271391555 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562b078f2130 msgr2=0x562b07903570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:55:49.780 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.777+0000 7f4286ffd640 1 --2- 172.21.15.29:0/271391555 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562b078f2130 0x562b07903570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:55:49.780 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.777+0000 7f4286ffd640 1 -- 172.21.15.29:0/271391555 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562b078fb820 msgr2=0x562b07903ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:55:49.780 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.778+0000 7f4286ffd640 1 --2- 172.21.15.29:0/271391555 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562b078fb820 0x562b07903ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:55:49.780 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.778+0000 7f4286ffd640 1 -- 172.21.15.29:0/271391555 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562b07974fe0 con 0x562b078f2a50 2024-03-16T11:55:49.780 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.778+0000 7f4287fff640 1 --2- 172.21.15.29:0/271391555 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562b078fb820 0x562b07903ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:55:49.780 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.778+0000 7f4286ffd640 1 --2- 172.21.15.29:0/271391555 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562b078f2a50 0x562b07903ab0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f427c0978a0 tx=0x7f427c005200 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e0002838d4a5486 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:55:49.780 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.778+0000 7f4266ffd640 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-16T11:55:49.783 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.782+0000 7f4266ffd640 1 -- 172.21.15.29:0/271391555 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f427c0b7030 con 0x562b078f2a50 2024-03-16T11:55:49.783 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.783+0000 7f4266ffd640 1 -- 172.21.15.29:0/271391555 <== mon.1 v2:172.21.15.154:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8512+0+0 (secure 0 0 0) 0x7f427c097540 con 0x562b078f2a50 2024-03-16T11:55:49.783 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.783+0000 7f4290981740 1 -- 172.21.15.29:0/271391555 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4280052070 con 0x562b078f2a50 2024-03-16T11:55:49.783 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.783+0000 7f4266ffd640 1 -- 172.21.15.29:0/271391555 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f427c099290 con 0x562b078f2a50 2024-03-16T11:55:49.783 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.783+0000 7f4290981740 10 client.25663 fetch_fsmap learned FSMap version 24 2024-03-16T11:55:49.783 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.783+0000 7f4290981740 1 -- 172.21.15.29:0/271391555 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x562b079762d0 con 0x562b078f2a50 2024-03-16T11:55:49.783 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.783+0000 7f4266ffd640 1 -- 172.21.15.29:0/271391555 <== mon.1 v2:172.21.15.154:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4280052070 con 0x562b078f2a50 2024-03-16T11:55:49.783 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.783+0000 7f4266ffd640 1 -- 172.21.15.29:0/271391555 <== mon.1 v2:172.21.15.154:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7f427c099d90 con 0x562b078f2a50 2024-03-16T11:55:49.783 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.783+0000 7f4290981740 10 client.25663 fetch_fsmap finished waiting for FSMap version 24 2024-03-16T11:55:49.783 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.783+0000 7f4290981740 10 client.25663 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T11:55:49.783 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.783+0000 7f4290981740 20 client.25663 populate_metadata read hostname 'smithi029' 2024-03-16T11:55:49.783 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.783+0000 7f4290981740 1 --2- 172.21.15.29:0/271391555 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x562b07976a80 0x562b07978ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:55:49.783 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.783+0000 7f4290981740 4 client.25663 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-16T11:55:49.784 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.783+0000 7f4290981740 1 -- 172.21.15.29:0/271391555 --> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] -- command(tid 0: {"prefix": "flush journal"}) -- 0x562b0785e2d0 con 0x562b07976a80 2024-03-16T11:55:49.784 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.783+0000 7f42877fe640 1 --2- 172.21.15.29:0/271391555 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x562b07976a80 0x562b07978ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:55:49.784 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.785+0000 7f4266ffd640 10 client.25663 ms_handle_connect on v2:172.21.15.152:6838/1548731833 2024-03-16T11:55:49.784 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.785+0000 7f42877fe640 1 --2- 172.21.15.29:0/271391555 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x562b07976a80 0x562b07978ea0 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2740f09993e4c46f server_cookie=16d484c5d9600c4e in_seq=0 out_seq=0 2024-03-16T11:55:49.784 INFO:tasks.workunit.client.0.smithi029.stdout: 1 473519 11.76 MB/sec execute 94 sec latency 12.797 ms 2024-03-16T11:55:49.989 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.989+0000 7f4266ffd640 1 -- 172.21.15.29:0/271391555 <== mds.0 v2:172.21.15.152:6838/1548731833 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x562b0785e2d0 con 0x562b07976a80 2024-03-16T11:55:49.989 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.989+0000 7f4266ffd640 10 client.25663 handle_command_reply: tid=0 2024-03-16T11:55:49.990 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.989+0000 7f4290981740 1 client.25663 shutdown 2024-03-16T11:55:49.990 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.990+0000 7f4290981740 1 -- 172.21.15.29:0/271391555 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562b078f2a50 msgr2=0x562b07903ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:55:49.990 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.990+0000 7f4290981740 1 --2- 172.21.15.29:0/271391555 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562b078f2a50 0x562b07903ab0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f427c0978a0 tx=0x7f427c005200 comp rx=0 tx=0).stop 2024-03-16T11:55:49.990 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.990+0000 7f4290981740 1 -- 172.21.15.29:0/271391555 shutdown_connections 2024-03-16T11:55:49.990 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.990+0000 7f4290981740 1 --2- 172.21.15.29:0/271391555 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562b078fb820 0x562b07903ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:55:49.990 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.990+0000 7f4290981740 1 --2- 172.21.15.29:0/271391555 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562b078f2a50 0x562b07903ab0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:55:49.990 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.990+0000 7f4290981740 1 --2- 172.21.15.29:0/271391555 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x562b07976a80 0x562b07978ea0 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:55:49.990 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.990+0000 7f4290981740 1 --2- 172.21.15.29:0/271391555 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562b078f2130 0x562b07903570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:55:49.990 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.990+0000 7f4266ffd640 0 client.25663 ms_handle_reset on v2:172.21.15.152:6838/1548731833 2024-03-16T11:55:49.990 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.990+0000 7f4290981740 1 -- 172.21.15.29:0/271391555 >> 172.21.15.29:0/271391555 conn(0x562b078eca50 msgr2=0x562b078eb320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:55:49.990 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.990+0000 7f4290981740 1 -- 172.21.15.29:0/271391555 shutdown_connections 2024-03-16T11:55:49.991 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.990+0000 7f4290981740 1 -- 172.21.15.29:0/271391555 wait complete. 2024-03-16T11:55:49.991 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.990+0000 7f4290981740 20 client.25663 trim_cache size 0 max 16384 2024-03-16T11:55:49.991 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:{ 2024-03-16T11:55:49.991 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "message": "", 2024-03-16T11:55:49.991 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "return_code": 0 2024-03-16T11:55:49.991 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:} 2024-03-16T11:55:49.992 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.990+0000 7f428dcbf640 1 -- 172.21.15.29:0/1639637348 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f42700a2a10 msgr2=0x7f42700a4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:55:49.992 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.990+0000 7f428dcbf640 1 --2- 172.21.15.29:0/1639637348 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f42700a2a10 0x7f42700a4ed0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f4280004af0 tx=0x7f428002cec0 comp rx=0 tx=0).stop 2024-03-16T11:55:49.992 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.990+0000 7f428dcbf640 1 -- 172.21.15.29:0/1639637348 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4288069800 msgr2=0x7f4288198510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:55:49.992 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.990+0000 7f428dcbf640 1 --2- 172.21.15.29:0/1639637348 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4288069800 0x7f4288198510 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f427c00d950 tx=0x7f427c00de10 comp rx=0 tx=0).stop 2024-03-16T11:55:49.992 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.990+0000 7f428dcbf640 1 -- 172.21.15.29:0/1639637348 shutdown_connections 2024-03-16T11:55:49.992 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.990+0000 7f428dcbf640 1 --2- 172.21.15.29:0/1639637348 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f42700a2a10 0x7f42700a4ed0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:55:49.992 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.990+0000 7f428dcbf640 1 --2- 172.21.15.29:0/1639637348 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f42881078d0 0x7f428819c840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:55:49.993 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.990+0000 7f428dcbf640 1 --2- 172.21.15.29:0/1639637348 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4288069800 0x7f4288198510 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:55:49.993 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.990+0000 7f428dcbf640 1 --2- 172.21.15.29:0/1639637348 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4288068e60 0x7f4288197fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:55:49.993 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.990+0000 7f428dcbf640 1 -- 172.21.15.29:0/1639637348 >> 172.21.15.29:0/1639637348 conn(0x7f4288077530 msgr2=0x7f42880fe8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:55:49.993 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.991+0000 7f428dcbf640 1 -- 172.21.15.29:0/1639637348 shutdown_connections 2024-03-16T11:55:49.993 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:55:49.991+0000 7f428dcbf640 1 -- 172.21.15.29:0/1639637348 wait complete. 2024-03-16T11:55:50.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:55:49 smithi154 ceph-mon[36320]: evicting unresponsive client smithi029:x (15309), after 301.747 seconds 2024-03-16T11:55:50.046 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:55:49 smithi154 ceph-mon[36320]: Evicting (and blocklisting) client session 15309 (172.21.15.29:0/1361425453) 2024-03-16T11:55:50.046 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:55:49 smithi154 ceph-mon[36320]: from='mds.? [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833]' entity='mds.b' cmd=[{"prefix":"osd blocklist", "blocklistop":"add","addr":"172.21.15.29:0/1361425453"}]: dispatch 2024-03-16T11:55:50.046 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:55:49 smithi154 ceph-mon[36320]: from='mds.? ' entity='mds.b' cmd=[{"prefix":"osd blocklist", "blocklistop":"add","addr":"172.21.15.29:0/1361425453"}]: dispatch 2024-03-16T11:55:50.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:55:49 smithi029 ceph-mon[31975]: evicting unresponsive client smithi029:x (15309), after 301.747 seconds 2024-03-16T11:55:50.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:55:49 smithi029 ceph-mon[31975]: Evicting (and blocklisting) client session 15309 (172.21.15.29:0/1361425453) 2024-03-16T11:55:50.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:55:49 smithi029 ceph-mon[31975]: from='mds.? [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833]' entity='mds.b' cmd=[{"prefix":"osd blocklist", "blocklistop":"add","addr":"172.21.15.29:0/1361425453"}]: dispatch 2024-03-16T11:55:50.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:55:49 smithi029 ceph-mon[31975]: from='mds.? ' entity='mds.b' cmd=[{"prefix":"osd blocklist", "blocklistop":"add","addr":"172.21.15.29:0/1361425453"}]: dispatch 2024-03-16T11:55:50.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:55:49 smithi152 ceph-mon[36291]: evicting unresponsive client smithi029:x (15309), after 301.747 seconds 2024-03-16T11:55:50.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:55:49 smithi152 ceph-mon[36291]: Evicting (and blocklisting) client session 15309 (172.21.15.29:0/1361425453) 2024-03-16T11:55:50.128 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:55:49 smithi152 ceph-mon[36291]: from='mds.? [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833]' entity='mds.b' cmd=[{"prefix":"osd blocklist", "blocklistop":"add","addr":"172.21.15.29:0/1361425453"}]: dispatch 2024-03-16T11:55:50.128 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:55:49 smithi152 ceph-mon[36291]: from='mds.? ' entity='mds.b' cmd=[{"prefix":"osd blocklist", "blocklistop":"add","addr":"172.21.15.29:0/1361425453"}]: dispatch 2024-03-16T11:55:50.784 INFO:tasks.workunit.client.0.smithi029.stdout: 1 476194 11.81 MB/sec execute 95 sec latency 81.862 ms 2024-03-16T11:55:51.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:55:50 smithi154 ceph-mon[36320]: pgmap v151: 97 pgs: 97 active+clean; 65 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 3.0 MiB/s wr, 296 op/s 2024-03-16T11:55:51.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:55:50 smithi154 ceph-mon[36320]: from='mds.? ' entity='mds.b' cmd='[{"prefix":"osd blocklist", "blocklistop":"add","addr":"172.21.15.29:0/1361425453"}]': finished 2024-03-16T11:55:51.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:55:50 smithi154 ceph-mon[36320]: osdmap e90: 12 total, 12 up, 12 in 2024-03-16T11:55:51.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:55:50 smithi029 ceph-mon[31975]: pgmap v151: 97 pgs: 97 active+clean; 65 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 3.0 MiB/s wr, 296 op/s 2024-03-16T11:55:51.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:55:50 smithi029 ceph-mon[31975]: from='mds.? ' entity='mds.b' cmd='[{"prefix":"osd blocklist", "blocklistop":"add","addr":"172.21.15.29:0/1361425453"}]': finished 2024-03-16T11:55:51.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:55:50 smithi029 ceph-mon[31975]: osdmap e90: 12 total, 12 up, 12 in 2024-03-16T11:55:51.129 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:55:50 smithi152 ceph-mon[36291]: pgmap v151: 97 pgs: 97 active+clean; 65 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 3.0 MiB/s wr, 296 op/s 2024-03-16T11:55:51.129 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:55:50 smithi152 ceph-mon[36291]: from='mds.? ' entity='mds.b' cmd='[{"prefix":"osd blocklist", "blocklistop":"add","addr":"172.21.15.29:0/1361425453"}]': finished 2024-03-16T11:55:51.130 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:55:50 smithi152 ceph-mon[36291]: osdmap e90: 12 total, 12 up, 12 in 2024-03-16T11:55:51.784 INFO:tasks.workunit.client.0.smithi029.stdout: 1 477679 11.75 MB/sec execute 96 sec latency 85.022 ms 2024-03-16T11:55:52.784 INFO:tasks.workunit.client.0.smithi029.stdout: 1 480127 11.78 MB/sec execute 97 sec latency 29.346 ms 2024-03-16T11:55:53.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:55:52 smithi154 ceph-mon[36320]: pgmap v153: 97 pgs: 97 active+clean; 48 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.8 MiB/s wr, 360 op/s 2024-03-16T11:55:53.087 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:55:52 smithi029 ceph-mon[31975]: pgmap v153: 97 pgs: 97 active+clean; 48 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.8 MiB/s wr, 360 op/s 2024-03-16T11:55:53.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:55:52 smithi152 ceph-mon[36291]: pgmap v153: 97 pgs: 97 active+clean; 48 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.8 MiB/s wr, 360 op/s 2024-03-16T11:55:53.784 INFO:tasks.workunit.client.0.smithi029.stdout: 1 482692 11.79 MB/sec execute 98 sec latency 18.653 ms 2024-03-16T11:55:54.784 INFO:tasks.workunit.client.0.smithi029.stdout: 1 484621 11.78 MB/sec execute 99 sec latency 28.744 ms 2024-03-16T11:55:55.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:55:54 smithi154 ceph-mon[36320]: pgmap v154: 97 pgs: 97 active+clean; 48 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.8 MiB/s wr, 360 op/s 2024-03-16T11:55:55.087 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:55:54 smithi029 ceph-mon[31975]: pgmap v154: 97 pgs: 97 active+clean; 48 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.8 MiB/s wr, 360 op/s 2024-03-16T11:55:55.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:55:54 smithi152 ceph-mon[36291]: pgmap v154: 97 pgs: 97 active+clean; 48 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.8 MiB/s wr, 360 op/s 2024-03-16T11:55:55.784 INFO:tasks.workunit.client.0.smithi029.stdout: 1 487198 11.81 MB/sec execute 100 sec latency 8.876 ms 2024-03-16T11:55:56.784 INFO:tasks.workunit.client.0.smithi029.stdout: 1 489327 11.77 MB/sec execute 101 sec latency 105.595 ms 2024-03-16T11:55:57.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:55:56 smithi154 ceph-mon[36320]: pgmap v155: 97 pgs: 97 active+clean; 48 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.8 MiB/s wr, 360 op/s 2024-03-16T11:55:57.087 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:55:56 smithi029 ceph-mon[31975]: pgmap v155: 97 pgs: 97 active+clean; 48 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.8 MiB/s wr, 360 op/s 2024-03-16T11:55:57.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:55:56 smithi152 ceph-mon[36291]: pgmap v155: 97 pgs: 97 active+clean; 48 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.8 MiB/s wr, 360 op/s 2024-03-16T11:55:57.784 INFO:tasks.workunit.client.0.smithi029.stdout: 1 491222 11.77 MB/sec execute 102 sec latency 23.860 ms 2024-03-16T11:55:58.785 INFO:tasks.workunit.client.0.smithi029.stdout: 1 494154 11.83 MB/sec execute 103 sec latency 11.320 ms 2024-03-16T11:55:59.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:55:58 smithi154 ceph-mon[36320]: pgmap v156: 97 pgs: 97 active+clean; 53 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.6 MiB/s wr, 367 op/s 2024-03-16T11:55:59.087 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:55:58 smithi029 ceph-mon[31975]: pgmap v156: 97 pgs: 97 active+clean; 53 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.6 MiB/s wr, 367 op/s 2024-03-16T11:55:59.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:55:58 smithi152 ceph-mon[36291]: pgmap v156: 97 pgs: 97 active+clean; 53 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.6 MiB/s wr, 367 op/s 2024-03-16T11:55:59.785 INFO:tasks.workunit.client.0.smithi029.stdout: 1 496281 11.79 MB/sec execute 104 sec latency 65.558 ms 2024-03-16T11:56:00.785 INFO:tasks.workunit.client.0.smithi029.stdout: 1 498271 11.80 MB/sec execute 105 sec latency 14.298 ms 2024-03-16T11:56:01.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:56:00 smithi154 ceph-mon[36320]: pgmap v157: 97 pgs: 97 active+clean; 53 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.6 MiB/s wr, 367 op/s 2024-03-16T11:56:01.087 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:56:00 smithi029 ceph-mon[31975]: pgmap v157: 97 pgs: 97 active+clean; 53 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.6 MiB/s wr, 367 op/s 2024-03-16T11:56:01.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:56:00 smithi152 ceph-mon[36291]: pgmap v157: 97 pgs: 97 active+clean; 53 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.6 MiB/s wr, 367 op/s 2024-03-16T11:56:01.785 INFO:tasks.workunit.client.0.smithi029.stdout: 1 499968 11.75 MB/sec execute 106 sec latency 91.511 ms 2024-03-16T11:56:02.785 INFO:tasks.workunit.client.0.smithi029.stdout: 1 501731 11.75 MB/sec execute 107 sec latency 25.229 ms 2024-03-16T11:56:03.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:56:02 smithi154 ceph-mon[36320]: pgmap v158: 97 pgs: 97 active+clean; 58 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 4.0 MiB/s wr, 451 op/s 2024-03-16T11:56:03.080 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.079+0000 7f5e9824b640 1 -- 172.21.15.29:0/3011921994 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5e900744a0 msgr2=0x7f5e901122a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:56:03.080 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.079+0000 7f5e9824b640 1 --2- 172.21.15.29:0/3011921994 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5e900744a0 0x7f5e901122a0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f5e80009900 tx=0x7f5e8002bcd0 comp rx=0 tx=0).stop 2024-03-16T11:56:03.080 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.079+0000 7f5e9824b640 1 -- 172.21.15.29:0/3011921994 shutdown_connections 2024-03-16T11:56:03.080 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.079+0000 7f5e9824b640 1 --2- 172.21.15.29:0/3011921994 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5e901127e0 0x7f5e90114bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:56:03.080 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.079+0000 7f5e9824b640 1 --2- 172.21.15.29:0/3011921994 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5e900744a0 0x7f5e901122a0 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:56:03.081 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.079+0000 7f5e9824b640 1 --2- 172.21.15.29:0/3011921994 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5e90073b80 0x7f5e90073f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:56:03.081 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.079+0000 7f5e9824b640 1 -- 172.21.15.29:0/3011921994 >> 172.21.15.29:0/3011921994 conn(0x7f5e900fd2a0 msgr2=0x7f5e900ff6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:56:03.081 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.079+0000 7f5e9824b640 1 -- 172.21.15.29:0/3011921994 shutdown_connections 2024-03-16T11:56:03.081 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.079+0000 7f5e9824b640 1 -- 172.21.15.29:0/3011921994 wait complete. 2024-03-16T11:56:03.081 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.080+0000 7f5e9824b640 1 Processor -- start 2024-03-16T11:56:03.081 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.080+0000 7f5e9824b640 1 -- start start 2024-03-16T11:56:03.081 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.080+0000 7f5e9824b640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5e90073b80 0x7f5e9019e670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:56:03.081 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.080+0000 7f5e9824b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5e900744a0 0x7f5e9019ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:56:03.081 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.080+0000 7f5e9824b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5e901127e0 0x7f5e901a2f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:56:03.081 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.080+0000 7f5e9824b640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f5e90118210 con 0x7f5e900744a0 2024-03-16T11:56:03.081 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.080+0000 7f5e9824b640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f5e90118090 con 0x7f5e90073b80 2024-03-16T11:56:03.081 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.080+0000 7f5e9824b640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f5e90118390 con 0x7f5e901127e0 2024-03-16T11:56:03.081 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.080+0000 7f5e957e6640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5e900744a0 0x7f5e9019ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:56:03.082 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.080+0000 7f5e95fe7640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5e90073b80 0x7f5e9019e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:56:03.082 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.080+0000 7f5e957e6640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5e900744a0 0x7f5e9019ebb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:37888/0 (socket says 172.21.15.29:37888) 2024-03-16T11:56:03.082 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.080+0000 7f5e957e6640 1 -- 172.21.15.29:0/2260921387 learned_addr learned my addr 172.21.15.29:0/2260921387 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:56:03.082 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.080+0000 7f5e967e8640 1 --2- 172.21.15.29:0/2260921387 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5e901127e0 0x7f5e901a2f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:56:03.082 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.080+0000 7f5e957e6640 1 -- 172.21.15.29:0/2260921387 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5e901127e0 msgr2=0x7f5e901a2f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:56:03.082 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.080+0000 7f5e957e6640 1 --2- 172.21.15.29:0/2260921387 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5e901127e0 0x7f5e901a2f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:56:03.082 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.081+0000 7f5e957e6640 1 -- 172.21.15.29:0/2260921387 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5e90073b80 msgr2=0x7f5e9019e670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:56:03.082 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.081+0000 7f5e957e6640 1 --2- 172.21.15.29:0/2260921387 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5e90073b80 0x7f5e9019e670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:56:03.082 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.081+0000 7f5e957e6640 1 -- 172.21.15.29:0/2260921387 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e901a3660 con 0x7f5e900744a0 2024-03-16T11:56:03.082 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.081+0000 7f5e957e6640 1 --2- 172.21.15.29:0/2260921387 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5e900744a0 0x7f5e9019ebb0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f5e80000c00 tx=0x7f5e80004360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:56:03.082 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.081+0000 7f5e86ffd640 1 -- 172.21.15.29:0/2260921387 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e80042020 con 0x7f5e900744a0 2024-03-16T11:56:03.082 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.081+0000 7f5e86ffd640 1 -- 172.21.15.29:0/2260921387 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5e800054e0 con 0x7f5e900744a0 2024-03-16T11:56:03.082 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.081+0000 7f5e9824b640 1 -- 172.21.15.29:0/2260921387 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5e901a38f0 con 0x7f5e900744a0 2024-03-16T11:56:03.082 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.081+0000 7f5e86ffd640 1 -- 172.21.15.29:0/2260921387 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e80003b20 con 0x7f5e900744a0 2024-03-16T11:56:03.083 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.081+0000 7f5e9824b640 1 -- 172.21.15.29:0/2260921387 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5e901ab1e0 con 0x7f5e900744a0 2024-03-16T11:56:03.088 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.083+0000 7f5e86ffd640 1 -- 172.21.15.29:0/2260921387 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7f5e80045020 con 0x7f5e900744a0 2024-03-16T11:56:03.089 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.084+0000 7f5e9ae39740 1 Processor -- start 2024-03-16T11:56:03.089 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.084+0000 7f5e9ae39740 1 -- start start 2024-03-16T11:56:03.089 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.084+0000 7f5e9ae39740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563654b0b130 0x563654b0b510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:56:03.089 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.084+0000 7f5e9ae39740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563654b0ba50 0x563654b142e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:56:03.089 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.084+0000 7f5e9ae39740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563654b14820 0x563654b16c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:56:03.089 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.084+0000 7f5e9ae39740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x563654a6fa40 con 0x563654b0ba50 2024-03-16T11:56:03.089 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.084+0000 7f5e9ae39740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x563654a18920 con 0x563654b0b130 2024-03-16T11:56:03.089 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.084+0000 7f5e9ae39740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x563654a6bae0 con 0x563654b14820 2024-03-16T11:56:03.089 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.084+0000 7f5e86ffd640 1 --2- 172.21.15.29:0/2260921387 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f5e7c0a2a10 0x7f5e7c0a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:56:03.089 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.084+0000 7f5e95fe7640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563654b0ba50 0x563654b142e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:56:03.089 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.084+0000 7f5e957e6640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563654b14820 0x563654b16c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:56:03.089 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.084+0000 7f5e95fe7640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563654b0ba50 0x563654b142e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:37894/0 (socket says 172.21.15.29:37894) 2024-03-16T11:56:03.089 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.084+0000 7f5e95fe7640 1 -- 172.21.15.29:0/2775346110 learned_addr learned my addr 172.21.15.29:0/2775346110 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:56:03.090 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.084+0000 7f5e967e8640 1 --2- 172.21.15.29:0/2775346110 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563654b0b130 0x563654b0b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:56:03.090 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.084+0000 7f5e967e8640 1 --2- 172.21.15.29:0/2260921387 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f5e7c0a2a10 0x7f5e7c0a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:56:03.090 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.084+0000 7f5e86ffd640 1 -- 172.21.15.29:0/2260921387 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8512+0+0 (secure 0 0 0) 0x7f5e800f0910 con 0x7f5e900744a0 2024-03-16T11:56:03.090 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.084+0000 7f5e95fe7640 1 -- 172.21.15.29:0/2775346110 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563654b14820 msgr2=0x563654b16c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:56:03.090 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.084+0000 7f5e95fe7640 1 --2- 172.21.15.29:0/2775346110 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563654b14820 0x563654b16c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:56:03.090 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.085+0000 7f5e95fe7640 1 -- 172.21.15.29:0/2775346110 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563654b0b130 msgr2=0x563654b0b510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:56:03.090 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.085+0000 7f5e95fe7640 1 --2- 172.21.15.29:0/2775346110 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563654b0b130 0x563654b0b510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:56:03.090 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.085+0000 7f5e95fe7640 1 -- 172.21.15.29:0/2775346110 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563654a30460 con 0x563654b0ba50 2024-03-16T11:56:03.090 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.085+0000 7f5e967e8640 1 --2- 172.21.15.29:0/2775346110 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563654b0b130 0x563654b0b510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:56:03.090 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.085+0000 7f5e95fe7640 1 --2- 172.21.15.29:0/2775346110 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563654b0ba50 0x563654b142e0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f5e8c009cf0 tx=0x7f5e8c011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e02f1850e711f0dc server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:56:03.090 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.085+0000 7f5e967e8640 1 --2- 172.21.15.29:0/2260921387 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f5e7c0a2a10 0x7f5e7c0a4ed0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f5e9006ae10 tx=0x7f5e88009500 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:56:03.090 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.085+0000 7f5e7bfff640 1 -- 172.21.15.29:0/2775346110 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e8c004230 con 0x563654b0ba50 2024-03-16T11:56:03.090 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.085+0000 7f5e7bfff640 1 -- 172.21.15.29:0/2775346110 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5e8c0043d0 con 0x563654b0ba50 2024-03-16T11:56:03.091 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.085+0000 7f5e7bfff640 1 -- 172.21.15.29:0/2775346110 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e8c0198c0 con 0x563654b0ba50 2024-03-16T11:56:03.091 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.086+0000 7f5e9ae39740 1 -- 172.21.15.29:0/2775346110 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563654b0ba50 msgr2=0x563654b142e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:56:03.091 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.086+0000 7f5e9ae39740 1 --2- 172.21.15.29:0/2775346110 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563654b0ba50 0x563654b142e0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f5e8c009cf0 tx=0x7f5e8c011810 comp rx=0 tx=0).stop 2024-03-16T11:56:03.091 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.086+0000 7f5e9ae39740 1 -- 172.21.15.29:0/2775346110 shutdown_connections 2024-03-16T11:56:03.091 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.086+0000 7f5e9ae39740 1 --2- 172.21.15.29:0/2775346110 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563654b14820 0x563654b16c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:56:03.091 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.086+0000 7f5e9ae39740 1 --2- 172.21.15.29:0/2775346110 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563654b0ba50 0x563654b142e0 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:56:03.091 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.086+0000 7f5e9ae39740 1 --2- 172.21.15.29:0/2775346110 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563654b0b130 0x563654b0b510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:56:03.091 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.086+0000 7f5e9ae39740 1 -- 172.21.15.29:0/2775346110 >> 172.21.15.29:0/2775346110 conn(0x563654b05a50 msgr2=0x563654b05e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:56:03.091 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.086+0000 7f5e9ae39740 1 -- 172.21.15.29:0/2775346110 shutdown_connections 2024-03-16T11:56:03.091 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.086+0000 7f5e9ae39740 1 -- 172.21.15.29:0/2775346110 wait complete. 2024-03-16T11:56:03.091 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.086+0000 7f5e9ae39740 1 -- start start 2024-03-16T11:56:03.091 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.086+0000 7f5e9ae39740 1 Processor -- start 2024-03-16T11:56:03.091 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.087+0000 7f5e9ae39740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563654b0b130 0x563654b11810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:56:03.091 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.087+0000 7f5e9ae39740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563654b0ba50 0x563654b11d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:56:03.091 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.087+0000 7f5e9ae39740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563654b14820 0x563654b12290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:56:03.092 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.087+0000 7f5e9ae39740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x563654b17450 con 0x563654b14820 2024-03-16T11:56:03.092 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.087+0000 7f5e9ae39740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x563654a2bd70 con 0x563654b0ba50 2024-03-16T11:56:03.092 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.087+0000 7f5e9ae39740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x563654b175d0 con 0x563654b0b130 2024-03-16T11:56:03.092 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.087+0000 7f5e9ae39740 10 client.0 _ll_register_callbacks cb 0x563654783a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T11:56:03.092 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.087+0000 7f5e95fe7640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563654b0b130 0x563654b11810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:56:03.092 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.087+0000 7f5e967e8640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563654b14820 0x563654b12290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:56:03.092 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.087+0000 7f5e95fe7640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563654b0b130 0x563654b11810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.29:56708/0 (socket says 172.21.15.29:56708) 2024-03-16T11:56:03.092 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.087+0000 7f5e957e6640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563654b0ba50 0x563654b11d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:56:03.092 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.087+0000 7f5e967e8640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563654b14820 0x563654b12290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:37898/0 (socket says 172.21.15.29:37898) 2024-03-16T11:56:03.092 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.087+0000 7f5e95fe7640 1 -- 172.21.15.29:0/3349209003 learned_addr learned my addr 172.21.15.29:0/3349209003 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:56:03.092 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.087+0000 7f5e95fe7640 1 -- 172.21.15.29:0/3349209003 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563654b0ba50 msgr2=0x563654b11d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:56:03.092 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.087+0000 7f5e95fe7640 1 --2- 172.21.15.29:0/3349209003 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563654b0ba50 0x563654b11d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:56:03.092 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.087+0000 7f5e95fe7640 1 -- 172.21.15.29:0/3349209003 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563654b14820 msgr2=0x563654b12290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:56:03.093 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.087+0000 7f5e95fe7640 1 --2- 172.21.15.29:0/3349209003 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563654b14820 0x563654b12290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:56:03.093 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.087+0000 7f5e95fe7640 1 -- 172.21.15.29:0/3349209003 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563654784210 con 0x563654b0b130 2024-03-16T11:56:03.093 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.087+0000 7f5e967e8640 1 --2- 172.21.15.29:0/3349209003 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563654b14820 0x563654b12290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:56:03.093 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.087+0000 7f5e95fe7640 1 --2- 172.21.15.29:0/3349209003 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563654b0b130 0x563654b11810 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f5e8c0038a0 tx=0x7f5e8c0038d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=57b3f56df09d9c5b server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:56:03.093 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.087+0000 7f5e797fa640 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-16T11:56:03.093 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.087+0000 7f5e797fa640 1 -- 172.21.15.29:0/3349209003 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e8c027070 con 0x563654b0b130 2024-03-16T11:56:03.093 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.087+0000 7f5e9ae39740 1 -- 172.21.15.29:0/3349209003 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5e8c0043d0 con 0x563654b0b130 2024-03-16T11:56:03.093 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.087+0000 7f5e797fa640 1 -- 172.21.15.29:0/3349209003 <== mon.1 v2:172.21.15.154:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8512+0+0 (secure 0 0 0) 0x7f5e8c005450 con 0x563654b0b130 2024-03-16T11:56:03.093 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.088+0000 7f5e797fa640 1 -- 172.21.15.29:0/3349209003 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f5e8c033050 con 0x563654b0b130 2024-03-16T11:56:03.093 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.088+0000 7f5e9ae39740 10 client.25675 fetch_fsmap learned FSMap version 24 2024-03-16T11:56:03.093 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.088+0000 7f5e9ae39740 1 -- 172.21.15.29:0/3349209003 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x563654850620 con 0x563654b0b130 2024-03-16T11:56:03.093 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.088+0000 7f5e797fa640 1 -- 172.21.15.29:0/3349209003 <== mon.1 v2:172.21.15.154:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5e8c0043d0 con 0x563654b0b130 2024-03-16T11:56:03.093 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.088+0000 7f5e797fa640 1 -- 172.21.15.29:0/3349209003 <== mon.1 v2:172.21.15.154:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7f5e8c004bc0 con 0x563654b0b130 2024-03-16T11:56:03.093 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.088+0000 7f5e9ae39740 10 client.25675 fetch_fsmap finished waiting for FSMap version 24 2024-03-16T11:56:03.093 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.088+0000 7f5e9ae39740 10 client.25675 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T11:56:03.094 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.088+0000 7f5e9ae39740 20 client.25675 populate_metadata read hostname 'smithi029' 2024-03-16T11:56:03.094 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.088+0000 7f5e9ae39740 1 --2- 172.21.15.29:0/3349209003 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x563654b24900 0x563654b26cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:56:03.094 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.088+0000 7f5e9ae39740 4 client.25675 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T11:56:03.094 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.088+0000 7f5e9ae39740 1 -- 172.21.15.29:0/3349209003 --> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563654a7cf80 con 0x563654b24900 2024-03-16T11:56:03.094 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.088+0000 7f5e957e6640 1 --2- 172.21.15.29:0/3349209003 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x563654b24900 0x563654b26cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:56:03.094 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:56:02 smithi029 ceph-mon[31975]: pgmap v158: 97 pgs: 97 active+clean; 58 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 4.0 MiB/s wr, 451 op/s 2024-03-16T11:56:03.095 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.089+0000 7f5e797fa640 10 client.25675 ms_handle_connect on v2:172.21.15.152:6838/1548731833 2024-03-16T11:56:03.095 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.089+0000 7f5e957e6640 1 --2- 172.21.15.29:0/3349209003 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x563654b24900 0x563654b26cf0 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=619cd5acf8aeb5bb server_cookie=4e7f907ef7d218d4 in_seq=0 out_seq=0 2024-03-16T11:56:03.095 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.089+0000 7f5e797fa640 1 -- 172.21.15.29:0/3349209003 <== mds.0 v2:172.21.15.152:6838/1548731833 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x563654a7cf80 con 0x563654b24900 2024-03-16T11:56:03.095 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.089+0000 7f5e797fa640 10 client.25675 handle_command_reply: tid=0 2024-03-16T11:56:03.095 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.090+0000 7f5e9ae39740 1 client.25675 shutdown 2024-03-16T11:56:03.096 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.090+0000 7f5e9ae39740 1 -- 172.21.15.29:0/3349209003 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563654b0b130 msgr2=0x563654b11810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:56:03.096 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.090+0000 7f5e9ae39740 1 --2- 172.21.15.29:0/3349209003 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563654b0b130 0x563654b11810 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f5e8c0038a0 tx=0x7f5e8c0038d0 comp rx=0 tx=0).stop 2024-03-16T11:56:03.096 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.090+0000 7f5e9ae39740 1 -- 172.21.15.29:0/3349209003 shutdown_connections 2024-03-16T11:56:03.096 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.090+0000 7f5e9ae39740 1 --2- 172.21.15.29:0/3349209003 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563654b14820 0x563654b12290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:56:03.096 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.090+0000 7f5e9ae39740 1 --2- 172.21.15.29:0/3349209003 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x563654b24900 0x563654b26cf0 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:56:03.096 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.090+0000 7f5e9ae39740 1 --2- 172.21.15.29:0/3349209003 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563654b0ba50 0x563654b11d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:56:03.096 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.090+0000 7f5e9ae39740 1 --2- 172.21.15.29:0/3349209003 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563654b0b130 0x563654b11810 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:56:03.096 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.090+0000 7f5e797fa640 0 client.25675 ms_handle_reset on v2:172.21.15.152:6838/1548731833 2024-03-16T11:56:03.096 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.090+0000 7f5e9ae39740 1 -- 172.21.15.29:0/3349209003 >> 172.21.15.29:0/3349209003 conn(0x563654b05a50 msgr2=0x563654b077a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:56:03.096 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.090+0000 7f5e9ae39740 1 -- 172.21.15.29:0/3349209003 shutdown_connections 2024-03-16T11:56:03.096 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.090+0000 7f5e9ae39740 1 -- 172.21.15.29:0/3349209003 wait complete. 2024-03-16T11:56:03.096 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.090+0000 7f5e9ae39740 20 client.25675 trim_cache size 0 max 16384 2024-03-16T11:56:03.108 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.104+0000 7f5e9ae39740 1 Processor -- start 2024-03-16T11:56:03.108 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.104+0000 7f5e9ae39740 1 -- start start 2024-03-16T11:56:03.109 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.105+0000 7f5e9ae39740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563654b0b130 0x563654b13ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:56:03.109 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.105+0000 7f5e9ae39740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563654b0ba50 0x563654b201f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:56:03.109 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.105+0000 7f5e9ae39740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563654b14820 0x563654b07560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:56:03.109 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.105+0000 7f5e9ae39740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f5e901b6300 con 0x563654b0ba50 2024-03-16T11:56:03.109 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.105+0000 7f5e9ae39740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f5e901b3bd0 con 0x563654b0b130 2024-03-16T11:56:03.109 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.105+0000 7f5e9ae39740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f5e901b1a00 con 0x563654b14820 2024-03-16T11:56:03.109 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.105+0000 7f5e957e6640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563654b0ba50 0x563654b201f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:56:03.109 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.105+0000 7f5e957e6640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563654b0ba50 0x563654b201f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:37900/0 (socket says 172.21.15.29:37900) 2024-03-16T11:56:03.109 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.105+0000 7f5e957e6640 1 -- 172.21.15.29:0/1119018841 learned_addr learned my addr 172.21.15.29:0/1119018841 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:56:03.109 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.105+0000 7f5e95fe7640 1 --2- 172.21.15.29:0/1119018841 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563654b0b130 0x563654b13ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:56:03.109 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.105+0000 7f5e967e8640 1 --2- 172.21.15.29:0/1119018841 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563654b14820 0x563654b07560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:56:03.109 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.105+0000 7f5e957e6640 1 -- 172.21.15.29:0/1119018841 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563654b14820 msgr2=0x563654b07560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:56:03.109 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.105+0000 7f5e957e6640 1 --2- 172.21.15.29:0/1119018841 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563654b14820 0x563654b07560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:56:03.110 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.105+0000 7f5e957e6640 1 -- 172.21.15.29:0/1119018841 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563654b0b130 msgr2=0x563654b13ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:56:03.110 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.105+0000 7f5e957e6640 1 --2- 172.21.15.29:0/1119018841 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563654b0b130 0x563654b13ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:56:03.110 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.105+0000 7f5e957e6640 1 -- 172.21.15.29:0/1119018841 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563654b20730 con 0x563654b0ba50 2024-03-16T11:56:03.110 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.105+0000 7f5e967e8640 1 --2- 172.21.15.29:0/1119018841 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563654b14820 0x563654b07560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T11:56:03.110 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.105+0000 7f5e95fe7640 1 --2- 172.21.15.29:0/1119018841 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563654b0b130 0x563654b13ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T11:56:03.110 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.105+0000 7f5e957e6640 1 --2- 172.21.15.29:0/1119018841 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563654b0ba50 0x563654b201f0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f5e80005ec0 tx=0x7f5e8002cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf77fa79bddead8 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:56:03.110 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.105+0000 7f5e7b7fe640 1 -- 172.21.15.29:0/1119018841 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e80005690 con 0x563654b0ba50 2024-03-16T11:56:03.110 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.105+0000 7f5e7b7fe640 1 -- 172.21.15.29:0/1119018841 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5e800bd790 con 0x563654b0ba50 2024-03-16T11:56:03.110 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.105+0000 7f5e7b7fe640 1 -- 172.21.15.29:0/1119018841 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e800b9700 con 0x563654b0ba50 2024-03-16T11:56:03.110 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.106+0000 7f5e9ae39740 1 -- 172.21.15.29:0/1119018841 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563654b0ba50 msgr2=0x563654b201f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:56:03.110 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.106+0000 7f5e9ae39740 1 --2- 172.21.15.29:0/1119018841 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563654b0ba50 0x563654b201f0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f5e80005ec0 tx=0x7f5e8002cbc0 comp rx=0 tx=0).stop 2024-03-16T11:56:03.110 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.106+0000 7f5e9ae39740 1 -- 172.21.15.29:0/1119018841 shutdown_connections 2024-03-16T11:56:03.110 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.106+0000 7f5e9ae39740 1 --2- 172.21.15.29:0/1119018841 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563654b14820 0x563654b07560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:56:03.110 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.106+0000 7f5e9ae39740 1 --2- 172.21.15.29:0/1119018841 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563654b0ba50 0x563654b201f0 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:56:03.111 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.106+0000 7f5e9ae39740 1 --2- 172.21.15.29:0/1119018841 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563654b0b130 0x563654b13ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:56:03.111 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.106+0000 7f5e9ae39740 1 -- 172.21.15.29:0/1119018841 >> 172.21.15.29:0/1119018841 conn(0x563654b05a50 msgr2=0x563654b06e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:56:03.111 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.106+0000 7f5e9ae39740 1 -- 172.21.15.29:0/1119018841 shutdown_connections 2024-03-16T11:56:03.111 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.106+0000 7f5e9ae39740 1 -- 172.21.15.29:0/1119018841 wait complete. 2024-03-16T11:56:03.111 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.106+0000 7f5e9ae39740 1 -- start start 2024-03-16T11:56:03.111 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.106+0000 7f5e9ae39740 1 Processor -- start 2024-03-16T11:56:03.111 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.107+0000 7f5e9ae39740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563654b0b130 0x563654b18040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:56:03.111 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.107+0000 7f5e9ae39740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563654b0ba50 0x563654b18580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:56:03.111 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.107+0000 7f5e9ae39740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563654b14820 0x563654b18ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:56:03.111 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.107+0000 7f5e9ae39740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f5e901b2090 con 0x563654b0b130 2024-03-16T11:56:03.111 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.107+0000 7f5e9ae39740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f5e901b3500 con 0x563654b0ba50 2024-03-16T11:56:03.111 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.107+0000 7f5e9ae39740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f5e901b2760 con 0x563654b14820 2024-03-16T11:56:03.111 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.107+0000 7f5e9ae39740 10 client.0 _ll_register_callbacks cb 0x7f5e88009ec0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T11:56:03.111 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.107+0000 7f5e95fe7640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563654b0b130 0x563654b18040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:56:03.112 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.107+0000 7f5e95fe7640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563654b0b130 0x563654b18040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:37908/0 (socket says 172.21.15.29:37908) 2024-03-16T11:56:03.112 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.107+0000 7f5e95fe7640 1 -- 172.21.15.29:0/1303265745 learned_addr learned my addr 172.21.15.29:0/1303265745 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:56:03.112 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.107+0000 7f5e967e8640 1 --2- 172.21.15.29:0/1303265745 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563654b14820 0x563654b18ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:56:03.112 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.107+0000 7f5e957e6640 1 --2- 172.21.15.29:0/1303265745 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563654b0ba50 0x563654b18580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:56:03.112 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.107+0000 7f5e95fe7640 1 -- 172.21.15.29:0/1303265745 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563654b14820 msgr2=0x563654b18ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:56:03.112 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.107+0000 7f5e95fe7640 1 --2- 172.21.15.29:0/1303265745 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563654b14820 0x563654b18ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:56:03.112 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.107+0000 7f5e95fe7640 1 -- 172.21.15.29:0/1303265745 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563654b0ba50 msgr2=0x563654b18580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:56:03.112 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.107+0000 7f5e95fe7640 1 --2- 172.21.15.29:0/1303265745 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563654b0ba50 0x563654b18580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:56:03.112 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.107+0000 7f5e95fe7640 1 -- 172.21.15.29:0/1303265745 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563654b89a60 con 0x563654b0b130 2024-03-16T11:56:03.112 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.107+0000 7f5e957e6640 1 --2- 172.21.15.29:0/1303265745 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563654b0ba50 0x563654b18580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T11:56:03.112 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.107+0000 7f5e95fe7640 1 --2- 172.21.15.29:0/1303265745 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563654b0b130 0x563654b18040 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f5e8c019730 tx=0x7f5e8c005da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=576d2f84a4e22bb9 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:56:03.112 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.107+0000 7f5e797fa640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-16T11:56:03.112 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.107+0000 7f5e797fa640 1 -- 172.21.15.29:0/1303265745 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e8c03c060 con 0x563654b0b130 2024-03-16T11:56:03.112 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.107+0000 7f5e797fa640 1 -- 172.21.15.29:0/1303265745 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8512+0+0 (secure 0 0 0) 0x7f5e8c005250 con 0x563654b0b130 2024-03-16T11:56:03.112 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.107+0000 7f5e9ae39740 1 -- 172.21.15.29:0/1303265745 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5e800bd790 con 0x563654b0b130 2024-03-16T11:56:03.113 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.108+0000 7f5e797fa640 1 -- 172.21.15.29:0/1303265745 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f5e8c027070 con 0x563654b0b130 2024-03-16T11:56:03.113 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.108+0000 7f5e9ae39740 10 client.15834 fetch_fsmap learned FSMap version 24 2024-03-16T11:56:03.113 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.108+0000 7f5e9ae39740 1 -- 172.21.15.29:0/1303265745 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x563654b8ac80 con 0x563654b0b130 2024-03-16T11:56:03.113 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.108+0000 7f5e797fa640 1 -- 172.21.15.29:0/1303265745 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5e800bd790 con 0x563654b0b130 2024-03-16T11:56:03.113 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.108+0000 7f5e797fa640 1 -- 172.21.15.29:0/1303265745 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7f5e8c0234f0 con 0x563654b0b130 2024-03-16T11:56:03.113 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.108+0000 7f5e9ae39740 10 client.15834 fetch_fsmap finished waiting for FSMap version 24 2024-03-16T11:56:03.113 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.108+0000 7f5e9ae39740 10 client.15834 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T11:56:03.113 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.108+0000 7f5e9ae39740 20 client.15834 populate_metadata read hostname 'smithi029' 2024-03-16T11:56:03.113 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.108+0000 7f5e9ae39740 1 --2- 172.21.15.29:0/1303265745 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x563654b8b2e0 0x563654b8d700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:56:03.113 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.108+0000 7f5e9ae39740 4 client.15834 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-16T11:56:03.113 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.108+0000 7f5e9ae39740 1 -- 172.21.15.29:0/1303265745 --> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] -- command(tid 0: {"prefix": "flush journal"}) -- 0x563654a772d0 con 0x563654b8b2e0 2024-03-16T11:56:03.113 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.108+0000 7f5e957e6640 1 --2- 172.21.15.29:0/1303265745 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x563654b8b2e0 0x563654b8d700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:56:03.113 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.108+0000 7f5e797fa640 10 client.15834 ms_handle_connect on v2:172.21.15.152:6838/1548731833 2024-03-16T11:56:03.113 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.108+0000 7f5e957e6640 1 --2- 172.21.15.29:0/1303265745 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x563654b8b2e0 0x563654b8d700 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=78b2e123ecec4475 server_cookie=e6bf7a142ed5449b in_seq=0 out_seq=0 2024-03-16T11:56:03.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:56:02 smithi152 ceph-mon[36291]: pgmap v158: 97 pgs: 97 active+clean; 58 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 4.0 MiB/s wr, 451 op/s 2024-03-16T11:56:03.279 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.279+0000 7f5e797fa640 1 -- 172.21.15.29:0/1303265745 <== mds.0 v2:172.21.15.152:6838/1548731833 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x563654a772d0 con 0x563654b8b2e0 2024-03-16T11:56:03.279 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.279+0000 7f5e797fa640 10 client.15834 handle_command_reply: tid=0 2024-03-16T11:56:03.279 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.279+0000 7f5e9ae39740 1 client.15834 shutdown 2024-03-16T11:56:03.279 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.279+0000 7f5e9ae39740 1 -- 172.21.15.29:0/1303265745 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563654b0b130 msgr2=0x563654b18040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:56:03.279 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.279+0000 7f5e9ae39740 1 --2- 172.21.15.29:0/1303265745 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563654b0b130 0x563654b18040 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f5e8c019730 tx=0x7f5e8c005da0 comp rx=0 tx=0).stop 2024-03-16T11:56:03.279 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.279+0000 7f5e9ae39740 1 -- 172.21.15.29:0/1303265745 shutdown_connections 2024-03-16T11:56:03.279 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.279+0000 7f5e9ae39740 1 --2- 172.21.15.29:0/1303265745 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563654b14820 0x563654b18ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:56:03.279 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.279+0000 7f5e9ae39740 1 --2- 172.21.15.29:0/1303265745 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x563654b8b2e0 0x563654b8d700 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:56:03.280 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.279+0000 7f5e9ae39740 1 --2- 172.21.15.29:0/1303265745 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563654b0ba50 0x563654b18580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:56:03.280 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.279+0000 7f5e9ae39740 1 --2- 172.21.15.29:0/1303265745 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563654b0b130 0x563654b18040 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:56:03.280 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.279+0000 7f5e797fa640 0 client.15834 ms_handle_reset on v2:172.21.15.152:6838/1548731833 2024-03-16T11:56:03.280 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.279+0000 7f5e9ae39740 1 -- 172.21.15.29:0/1303265745 >> 172.21.15.29:0/1303265745 conn(0x563654b05a50 msgr2=0x563654b04bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:56:03.280 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.279+0000 7f5e9ae39740 1 -- 172.21.15.29:0/1303265745 shutdown_connections 2024-03-16T11:56:03.280 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.279+0000 7f5e9ae39740 1 -- 172.21.15.29:0/1303265745 wait complete. 2024-03-16T11:56:03.280 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.279+0000 7f5e9ae39740 20 client.15834 trim_cache size 0 max 16384 2024-03-16T11:56:03.280 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:{ 2024-03-16T11:56:03.280 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "message": "", 2024-03-16T11:56:03.280 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "return_code": 0 2024-03-16T11:56:03.280 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:} 2024-03-16T11:56:03.281 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.280+0000 7f5e9824b640 1 -- 172.21.15.29:0/2260921387 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f5e7c0a2a10 msgr2=0x7f5e7c0a4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:56:03.281 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.280+0000 7f5e9824b640 1 --2- 172.21.15.29:0/2260921387 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f5e7c0a2a10 0x7f5e7c0a4ed0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f5e9006ae10 tx=0x7f5e88009500 comp rx=0 tx=0).stop 2024-03-16T11:56:03.281 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.280+0000 7f5e9824b640 1 -- 172.21.15.29:0/2260921387 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5e900744a0 msgr2=0x7f5e9019ebb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:56:03.281 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.280+0000 7f5e9824b640 1 --2- 172.21.15.29:0/2260921387 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5e900744a0 0x7f5e9019ebb0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f5e80000c00 tx=0x7f5e80004360 comp rx=0 tx=0).stop 2024-03-16T11:56:03.281 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.280+0000 7f5e9824b640 1 -- 172.21.15.29:0/2260921387 shutdown_connections 2024-03-16T11:56:03.281 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.280+0000 7f5e9824b640 1 --2- 172.21.15.29:0/2260921387 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f5e7c0a2a10 0x7f5e7c0a4ed0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:56:03.282 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.280+0000 7f5e9824b640 1 --2- 172.21.15.29:0/2260921387 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5e901127e0 0x7f5e901a2f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:56:03.282 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.280+0000 7f5e9824b640 1 --2- 172.21.15.29:0/2260921387 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5e900744a0 0x7f5e9019ebb0 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:56:03.282 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.280+0000 7f5e9824b640 1 --2- 172.21.15.29:0/2260921387 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5e90073b80 0x7f5e9019e670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:56:03.282 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.280+0000 7f5e9824b640 1 -- 172.21.15.29:0/2260921387 >> 172.21.15.29:0/2260921387 conn(0x7f5e900fd2a0 msgr2=0x7f5e900fdf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:56:03.282 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.280+0000 7f5e9824b640 1 -- 172.21.15.29:0/2260921387 shutdown_connections 2024-03-16T11:56:03.282 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:03.280+0000 7f5e9824b640 1 -- 172.21.15.29:0/2260921387 wait complete. 2024-03-16T11:56:03.785 INFO:tasks.workunit.client.0.smithi029.stdout: 1 504283 11.77 MB/sec execute 108 sec latency 48.558 ms 2024-03-16T11:56:04.785 INFO:tasks.workunit.client.0.smithi029.stdout: 1 505916 11.76 MB/sec execute 109 sec latency 22.412 ms 2024-03-16T11:56:05.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:56:04 smithi154 ceph-mon[36320]: pgmap v159: 97 pgs: 97 active+clean; 58 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 887 KiB/s rd, 2.9 MiB/s wr, 215 op/s 2024-03-16T11:56:05.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:56:04 smithi154 ceph-mon[36320]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T11:56:05.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:56:04 smithi029 ceph-mon[31975]: pgmap v159: 97 pgs: 97 active+clean; 58 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 887 KiB/s rd, 2.9 MiB/s wr, 215 op/s 2024-03-16T11:56:05.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:56:04 smithi029 ceph-mon[31975]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T11:56:05.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:56:04 smithi152 ceph-mon[36291]: pgmap v159: 97 pgs: 97 active+clean; 58 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 887 KiB/s rd, 2.9 MiB/s wr, 215 op/s 2024-03-16T11:56:05.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:56:04 smithi152 ceph-mon[36291]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T11:56:05.785 INFO:tasks.workunit.client.0.smithi029.stdout: 1 508423 11.78 MB/sec execute 110 sec latency 8.963 ms 2024-03-16T11:56:06.087 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:56:05 smithi029 ceph-mon[31975]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:56:06.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:56:05 smithi029 ceph-mon[31975]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T11:56:06.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:56:05 smithi029 ceph-mon[31975]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' 2024-03-16T11:56:06.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:56:05 smithi152 ceph-mon[36291]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:56:06.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:56:05 smithi152 ceph-mon[36291]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T11:56:06.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:56:05 smithi152 ceph-mon[36291]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' 2024-03-16T11:56:06.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:56:05 smithi154 ceph-mon[36320]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:56:06.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:56:05 smithi154 ceph-mon[36320]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T11:56:06.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:56:05 smithi154 ceph-mon[36320]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' 2024-03-16T11:56:06.785 INFO:tasks.workunit.client.0.smithi029.stdout: 1 510593 11.74 MB/sec execute 111 sec latency 95.403 ms 2024-03-16T11:56:07.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:56:06 smithi029 ceph-mon[31975]: pgmap v160: 97 pgs: 97 active+clean; 58 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 887 KiB/s rd, 2.9 MiB/s wr, 215 op/s 2024-03-16T11:56:07.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:56:06 smithi152 ceph-mon[36291]: pgmap v160: 97 pgs: 97 active+clean; 58 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 887 KiB/s rd, 2.9 MiB/s wr, 215 op/s 2024-03-16T11:56:07.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:56:06 smithi154 ceph-mon[36320]: pgmap v160: 97 pgs: 97 active+clean; 58 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 887 KiB/s rd, 2.9 MiB/s wr, 215 op/s 2024-03-16T11:56:07.785 INFO:tasks.workunit.client.0.smithi029.stdout: 1 512494 11.74 MB/sec execute 112 sec latency 27.500 ms 2024-03-16T11:56:08.785 INFO:tasks.workunit.client.0.smithi029.stdout: 1 515435 11.80 MB/sec execute 113 sec latency 9.598 ms 2024-03-16T11:56:09.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:56:08 smithi029 ceph-mon[31975]: pgmap v161: 97 pgs: 97 active+clean; 48 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 4.4 MiB/s wr, 429 op/s 2024-03-16T11:56:09.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:56:08 smithi152 ceph-mon[36291]: pgmap v161: 97 pgs: 97 active+clean; 48 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 4.4 MiB/s wr, 429 op/s 2024-03-16T11:56:09.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:56:08 smithi154 ceph-mon[36320]: pgmap v161: 97 pgs: 97 active+clean; 48 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 4.4 MiB/s wr, 429 op/s 2024-03-16T11:56:09.785 INFO:tasks.workunit.client.0.smithi029.stdout: 1 517694 11.77 MB/sec execute 114 sec latency 24.094 ms 2024-03-16T11:56:10.785 INFO:tasks.workunit.client.0.smithi029.stdout: 1 519585 11.77 MB/sec execute 115 sec latency 20.748 ms 2024-03-16T11:56:11.087 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:56:10 smithi029 ceph-mon[31975]: pgmap v162: 97 pgs: 97 active+clean; 48 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 983 KiB/s rd, 3.0 MiB/s wr, 328 op/s 2024-03-16T11:56:11.096 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:56:10 smithi154 ceph-mon[36320]: pgmap v162: 97 pgs: 97 active+clean; 48 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 983 KiB/s rd, 3.0 MiB/s wr, 328 op/s 2024-03-16T11:56:11.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:56:10 smithi152 ceph-mon[36291]: pgmap v162: 97 pgs: 97 active+clean; 48 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 983 KiB/s rd, 3.0 MiB/s wr, 328 op/s 2024-03-16T11:56:11.785 INFO:tasks.workunit.client.0.smithi029.stdout: 1 522313 11.81 MB/sec execute 116 sec latency 79.305 ms 2024-03-16T11:56:12.785 INFO:tasks.workunit.client.0.smithi029.stdout: 1 524040 11.77 MB/sec execute 117 sec latency 24.095 ms 2024-03-16T11:56:13.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:56:12 smithi029 ceph-mon[31975]: pgmap v163: 97 pgs: 97 active+clean; 55 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 4.6 MiB/s wr, 426 op/s 2024-03-16T11:56:13.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:56:12 smithi152 ceph-mon[36291]: pgmap v163: 97 pgs: 97 active+clean; 55 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 4.6 MiB/s wr, 426 op/s 2024-03-16T11:56:13.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:56:12 smithi154 ceph-mon[36320]: pgmap v163: 97 pgs: 97 active+clean; 55 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 4.6 MiB/s wr, 426 op/s 2024-03-16T11:56:13.785 INFO:tasks.workunit.client.0.smithi029.stdout: 1 526370 11.79 MB/sec execute 118 sec latency 8.984 ms 2024-03-16T11:56:14.785 INFO:tasks.workunit.client.0.smithi029.stdout: 1 528937 11.80 MB/sec execute 119 sec latency 59.197 ms 2024-03-16T11:56:15.087 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:56:14 smithi029 ceph-mon[31975]: pgmap v164: 97 pgs: 97 active+clean; 55 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 3.0 MiB/s wr, 312 op/s 2024-03-16T11:56:15.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:56:14 smithi152 ceph-mon[36291]: pgmap v164: 97 pgs: 97 active+clean; 55 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 3.0 MiB/s wr, 312 op/s 2024-03-16T11:56:15.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:56:14 smithi154 ceph-mon[36320]: pgmap v164: 97 pgs: 97 active+clean; 55 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 3.0 MiB/s wr, 312 op/s 2024-03-16T11:56:15.785 INFO:tasks.workunit.client.0.smithi029.stdout: 1 530742 11.79 MB/sec execute 120 sec latency 22.685 ms 2024-03-16T11:56:16.372 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:16.370+0000 7f8959d90640 1 -- 172.21.15.29:0/2872816458 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8954107860 msgr2=0x7f8954107c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:56:16.372 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:16.370+0000 7f8959d90640 1 --2- 172.21.15.29:0/2872816458 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8954107860 0x7f8954107c40 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f8944009930 tx=0x7f894402bc30 comp rx=0 tx=0).stop 2024-03-16T11:56:16.373 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:16.370+0000 7f8959d90640 1 -- 172.21.15.29:0/2872816458 shutdown_connections 2024-03-16T11:56:16.373 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:16.370+0000 7f8959d90640 1 --2- 172.21.15.29:0/2872816458 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f895406a9a0 0x7f895410e670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:56:16.373 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:16.370+0000 7f8959d90640 1 --2- 172.21.15.29:0/2872816458 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8954069fe0 0x7f895406a460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:56:16.373 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:16.370+0000 7f8959d90640 1 --2- 172.21.15.29:0/2872816458 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8954107860 0x7f8954107c40 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:56:16.373 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:16.370+0000 7f8959d90640 1 -- 172.21.15.29:0/2872816458 >> 172.21.15.29:0/2872816458 conn(0x7f89540774b0 msgr2=0x7f89540778c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:56:16.373 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:16.370+0000 7f8959d90640 1 -- 172.21.15.29:0/2872816458 shutdown_connections 2024-03-16T11:56:16.373 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:16.370+0000 7f8959d90640 1 -- 172.21.15.29:0/2872816458 wait complete. 2024-03-16T11:56:16.373 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:16.370+0000 7f8959d90640 1 Processor -- start 2024-03-16T11:56:16.373 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:16.371+0000 7f8959d90640 1 -- start start 2024-03-16T11:56:16.373 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:16.371+0000 7f8959d90640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8954069fe0 0x7f8954197f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:56:16.373 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:16.371+0000 7f8959d90640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f895406a9a0 0x7f89541984a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:56:16.373 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:16.371+0000 7f8959d90640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8954107860 0x7f895419c820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:56:16.373 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:16.371+0000 7f8959d90640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f8954111bc0 con 0x7f8954107860 2024-03-16T11:56:16.373 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:16.371+0000 7f8959d90640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f8954111a40 con 0x7f8954069fe0 2024-03-16T11:56:16.374 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:16.371+0000 7f8959d90640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f8954111d40 con 0x7f895406a9a0 2024-03-16T11:56:16.374 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:16.371+0000 7f8953fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8954107860 0x7f895419c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:56:16.374 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:16.371+0000 7f8953fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8954107860 0x7f895419c820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56556/0 (socket says 172.21.15.29:56556) 2024-03-16T11:56:16.374 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:16.371+0000 7f8953fff640 1 -- 172.21.15.29:0/1666038505 learned_addr learned my addr 172.21.15.29:0/1666038505 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:56:16.374 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:16.371+0000 7f8952ffd640 1 --2- 172.21.15.29:0/1666038505 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f895406a9a0 0x7f89541984a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:56:16.374 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:16.371+0000 7f89537fe640 1 --2- 172.21.15.29:0/1666038505 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8954069fe0 0x7f8954197f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:56:16.374 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:16.371+0000 7f8953fff640 1 -- 172.21.15.29:0/1666038505 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f895406a9a0 msgr2=0x7f89541984a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:56:16.374 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:16.371+0000 7f8953fff640 1 --2- 172.21.15.29:0/1666038505 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f895406a9a0 0x7f89541984a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:56:16.374 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:16.371+0000 7f8953fff640 1 -- 172.21.15.29:0/1666038505 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8954069fe0 msgr2=0x7f8954197f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:56:16.374 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:16.371+0000 7f8953fff640 1 --2- 172.21.15.29:0/1666038505 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8954069fe0 0x7f8954197f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:56:16.374 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:16.371+0000 7f8953fff640 1 -- 172.21.15.29:0/1666038505 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f895419cf50 con 0x7f8954107860 2024-03-16T11:56:16.375 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:16.371+0000 7f8952ffd640 1 --2- 172.21.15.29:0/1666038505 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f895406a9a0 0x7f89541984a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:56:16.375 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:16.371+0000 7f8953fff640 1 --2- 172.21.15.29:0/1666038505 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8954107860 0x7f895419c820 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f894c00eb10 tx=0x7f894c00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:56:16.375 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:16.372+0000 7f8950ff9640 1 -- 172.21.15.29:0/1666038505 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f894c004b20 con 0x7f8954107860 2024-03-16T11:56:16.375 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:16.372+0000 7f8950ff9640 1 -- 172.21.15.29:0/1666038505 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f894c004cc0 con 0x7f8954107860 2024-03-16T11:56:16.375 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:16.372+0000 7f8959d90640 1 -- 172.21.15.29:0/1666038505 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f895419d240 con 0x7f8954107860 2024-03-16T11:56:16.375 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:16.372+0000 7f8950ff9640 1 -- 172.21.15.29:0/1666038505 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f894c010760 con 0x7f8954107860 2024-03-16T11:56:16.375 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:16.372+0000 7f8959d90640 1 -- 172.21.15.29:0/1666038505 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f89541a4ad0 con 0x7f8954107860 2024-03-16T11:56:16.383 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:16.374+0000 7f8950ff9640 1 -- 172.21.15.29:0/1666038505 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7f894c005020 con 0x7f8954107860 2024-03-16T11:56:16.383 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:16.374+0000 7f895c977740 1 Processor -- start 2024-03-16T11:56:16.383 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:16.374+0000 7f895c977740 1 -- start start 2024-03-16T11:56:16.383 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:16.374+0000 7f895c977740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562a5dbacbd0 0x562a5dbacfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:56:16.383 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:16.374+0000 7f895c977740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562a5dbad4f0 0x562a5dbba6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:56:16.383 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:16.374+0000 7f895c977740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562a5dbbac30 0x562a5dbbd020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:56:16.383 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:16.374+0000 7f895c977740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x562a5db11a40 con 0x562a5dbacbd0 2024-03-16T11:56:16.384 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:16.374+0000 7f895c977740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x562a5daba920 con 0x562a5dbbac30 2024-03-16T11:56:16.384 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:16.374+0000 7f895c977740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x562a5db0dae0 con 0x562a5dbad4f0 2024-03-16T11:56:16.384 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:16.374+0000 7f8950ff9640 1 --2- 172.21.15.29:0/1666038505 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f893c0a2a10 0x7f893c0a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:56:16.384 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:16.375+0000 7f8952ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562a5dbacbd0 0x562a5dbacfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:56:16.384 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:16.375+0000 7f8952ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562a5dbacbd0 0x562a5dbacfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56558/0 (socket says 172.21.15.29:56558) 2024-03-16T11:56:16.384 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:16.375+0000 7f8952ffd640 1 -- 172.21.15.29:0/2123134516 learned_addr learned my addr 172.21.15.29:0/2123134516 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:56:16.384 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:16.375+0000 7f89537fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562a5dbad4f0 0x562a5dbba6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:56:16.384 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:16.375+0000 7f8953fff640 1 --2- 172.21.15.29:0/1666038505 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f893c0a2a10 0x7f893c0a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:56:16.384 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:16.375+0000 7f8952ffd640 1 --2- 172.21.15.29:0/2123134516 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562a5dbbac30 0x562a5dbbd020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:56:16.384 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:16.375+0000 7f8950ff9640 1 -- 172.21.15.29:0/1666038505 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8512+0+0 (secure 0 0 0) 0x7f894c0cf3d0 con 0x7f8954107860 2024-03-16T11:56:16.384 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:16.375+0000 7f8952ffd640 1 -- 172.21.15.29:0/2123134516 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562a5dbad4f0 msgr2=0x562a5dbba6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:56:16.384 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:16.375+0000 7f8952ffd640 1 --2- 172.21.15.29:0/2123134516 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562a5dbad4f0 0x562a5dbba6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:56:16.385 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:16.375+0000 7f8952ffd640 1 -- 172.21.15.29:0/2123134516 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562a5dbbac30 msgr2=0x562a5dbbd020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:56:16.385 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:16.375+0000 7f8952ffd640 1 --2- 172.21.15.29:0/2123134516 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562a5dbbac30 0x562a5dbbd020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:56:16.385 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:16.375+0000 7f8952ffd640 1 -- 172.21.15.29:0/2123134516 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562a5dad2460 con 0x562a5dbacbd0 2024-03-16T11:56:16.385 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:16.375+0000 7f89537fe640 1 --2- 172.21.15.29:0/2123134516 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562a5dbad4f0 0x562a5dbba6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:56:16.385 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:16.375+0000 7f8952ffd640 1 --2- 172.21.15.29:0/2123134516 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562a5dbacbd0 0x562a5dbacfb0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f8948013a40 tx=0x7f8948009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61d020f5c708d40a server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:56:16.385 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:16.375+0000 7f8953fff640 1 --2- 172.21.15.29:0/1666038505 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f893c0a2a10 0x7f893c0a4ed0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f894c097ea0 tx=0x7f894c086040 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:56:16.385 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:16.375+0000 7f8939ffb640 1 -- 172.21.15.29:0/2123134516 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8948004050 con 0x562a5dbacbd0 2024-03-16T11:56:16.385 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:16.375+0000 7f8939ffb640 1 -- 172.21.15.29:0/2123134516 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f894800fe60 con 0x562a5dbacbd0 2024-03-16T11:56:16.385 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:16.375+0000 7f8939ffb640 1 -- 172.21.15.29:0/2123134516 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8948003a80 con 0x562a5dbacbd0 2024-03-16T11:56:16.385 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:16.376+0000 7f895c977740 1 -- 172.21.15.29:0/2123134516 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562a5dbacbd0 msgr2=0x562a5dbacfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:56:16.385 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:16.376+0000 7f895c977740 1 --2- 172.21.15.29:0/2123134516 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562a5dbacbd0 0x562a5dbacfb0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f8948013a40 tx=0x7f8948009c00 comp rx=0 tx=0).stop 2024-03-16T11:56:16.386 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:16.376+0000 7f895c977740 1 -- 172.21.15.29:0/2123134516 shutdown_connections 2024-03-16T11:56:16.386 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:16.376+0000 7f895c977740 1 --2- 172.21.15.29:0/2123134516 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562a5dbbac30 0x562a5dbbd020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:56:16.386 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:16.376+0000 7f895c977740 1 --2- 172.21.15.29:0/2123134516 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562a5dbad4f0 0x562a5dbba6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:56:16.386 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:16.376+0000 7f895c977740 1 --2- 172.21.15.29:0/2123134516 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562a5dbacbd0 0x562a5dbacfb0 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:56:16.386 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:16.376+0000 7f895c977740 1 -- 172.21.15.29:0/2123134516 >> 172.21.15.29:0/2123134516 conn(0x562a5dba7a50 msgr2=0x562a5dba7e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:56:16.386 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:16.376+0000 7f895c977740 1 -- 172.21.15.29:0/2123134516 shutdown_connections 2024-03-16T11:56:16.386 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:16.376+0000 7f895c977740 1 -- 172.21.15.29:0/2123134516 wait complete. 2024-03-16T11:56:16.386 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:16.376+0000 7f895c977740 1 -- start start 2024-03-16T11:56:16.386 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:16.377+0000 7f895c977740 1 Processor -- start 2024-03-16T11:56:16.386 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:16.377+0000 7f895c977740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562a5dbacbd0 0x562a5dbb7c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:56:16.387 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:16.377+0000 7f895c977740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562a5dbad4f0 0x562a5dbb8170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:56:16.387 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:16.377+0000 7f895c977740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562a5dbbac30 0x562a5dbb86b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:56:16.387 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:16.377+0000 7f895c977740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x562a5dbbd860 con 0x562a5dbbac30 2024-03-16T11:56:16.387 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:16.377+0000 7f895c977740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x562a5dacdd70 con 0x562a5dbad4f0 2024-03-16T11:56:16.387 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:16.377+0000 7f895c977740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x562a5dbbd9e0 con 0x562a5dbacbd0 2024-03-16T11:56:16.387 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:16.377+0000 7f895c977740 10 client.0 _ll_register_callbacks cb 0x562a5d825a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T11:56:16.387 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:16.377+0000 7f8952ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562a5dbbac30 0x562a5dbb86b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:56:16.387 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:16.377+0000 7f8952ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562a5dbbac30 0x562a5dbb86b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56568/0 (socket says 172.21.15.29:56568) 2024-03-16T11:56:16.387 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:16.377+0000 7f8952ffd640 1 -- 172.21.15.29:0/3632460207 learned_addr learned my addr 172.21.15.29:0/3632460207 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:56:16.387 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:16.377+0000 7f89537fe640 1 --2- 172.21.15.29:0/3632460207 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562a5dbad4f0 0x562a5dbb8170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:56:16.387 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:16.377+0000 7f8952ffd640 1 --2- 172.21.15.29:0/3632460207 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562a5dbacbd0 0x562a5dbb7c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:56:16.388 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:16.377+0000 7f89537fe640 1 -- 172.21.15.29:0/3632460207 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562a5dbacbd0 msgr2=0x562a5dbb7c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:56:16.388 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:16.377+0000 7f89537fe640 1 --2- 172.21.15.29:0/3632460207 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562a5dbacbd0 0x562a5dbb7c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:56:16.388 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:16.377+0000 7f89537fe640 1 -- 172.21.15.29:0/3632460207 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562a5dbbac30 msgr2=0x562a5dbb86b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:56:16.388 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:16.377+0000 7f89537fe640 1 --2- 172.21.15.29:0/3632460207 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562a5dbbac30 0x562a5dbb86b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:56:16.388 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:16.377+0000 7f89537fe640 1 -- 172.21.15.29:0/3632460207 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562a5d826210 con 0x562a5dbad4f0 2024-03-16T11:56:16.388 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:16.377+0000 7f8952ffd640 1 --2- 172.21.15.29:0/3632460207 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562a5dbbac30 0x562a5dbb86b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:56:16.388 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:16.378+0000 7f89537fe640 1 --2- 172.21.15.29:0/3632460207 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562a5dbad4f0 0x562a5dbb8170 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f8944009230 tx=0x7f894402c800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2343159a8572f5a1 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:56:16.388 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:16.378+0000 7f8932ffd640 10 client.0 ms_handle_connect on v2:172.21.15.152:3300/0 2024-03-16T11:56:16.388 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:16.378+0000 7f8932ffd640 1 -- 172.21.15.29:0/3632460207 <== mon.2 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8944005d50 con 0x562a5dbad4f0 2024-03-16T11:56:16.388 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:16.378+0000 7f895c977740 1 -- 172.21.15.29:0/3632460207 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f894800fe60 con 0x562a5dbad4f0 2024-03-16T11:56:16.388 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:16.378+0000 7f8932ffd640 1 -- 172.21.15.29:0/3632460207 <== mon.2 v2:172.21.15.152:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8512+0+0 (secure 0 0 0) 0x7f894402cab0 con 0x562a5dbad4f0 2024-03-16T11:56:16.388 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:16.378+0000 7f8932ffd640 1 -- 172.21.15.29:0/3632460207 <== mon.2 v2:172.21.15.152:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f894402ccb0 con 0x562a5dbad4f0 2024-03-16T11:56:16.388 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:16.378+0000 7f895c977740 10 client.25661 fetch_fsmap learned FSMap version 24 2024-03-16T11:56:16.388 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:16.378+0000 7f895c977740 1 -- 172.21.15.29:0/3632460207 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x562a5d8f2620 con 0x562a5dbad4f0 2024-03-16T11:56:16.388 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:16.378+0000 7f8932ffd640 1 -- 172.21.15.29:0/3632460207 <== mon.2 v2:172.21.15.152:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f894800fe60 con 0x562a5dbad4f0 2024-03-16T11:56:16.389 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:16.378+0000 7f8932ffd640 1 -- 172.21.15.29:0/3632460207 <== mon.2 v2:172.21.15.152:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7f89440035f0 con 0x562a5dbad4f0 2024-03-16T11:56:16.389 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:16.378+0000 7f895c977740 10 client.25661 fetch_fsmap finished waiting for FSMap version 24 2024-03-16T11:56:16.389 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:16.378+0000 7f895c977740 10 client.25661 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T11:56:16.389 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:16.378+0000 7f895c977740 20 client.25661 populate_metadata read hostname 'smithi029' 2024-03-16T11:56:16.389 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:16.379+0000 7f895c977740 1 --2- 172.21.15.29:0/3632460207 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x562a5dbcae90 0x562a5dbcd280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:56:16.389 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:16.379+0000 7f895c977740 4 client.25661 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T11:56:16.389 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:16.379+0000 7f895c977740 1 -- 172.21.15.29:0/3632460207 --> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562a5db1ef80 con 0x562a5dbcae90 2024-03-16T11:56:16.389 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:16.379+0000 7f8952ffd640 1 --2- 172.21.15.29:0/3632460207 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x562a5dbcae90 0x562a5dbcd280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:56:16.389 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:16.379+0000 7f8952ffd640 1 --2- 172.21.15.29:0/3632460207 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x562a5dbcae90 0x562a5dbcd280 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ed5fd4442755ed47 server_cookie=84ecda86a04c5245 in_seq=0 out_seq=0 2024-03-16T11:56:16.389 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:16.379+0000 7f8932ffd640 10 client.25661 ms_handle_connect on v2:172.21.15.152:6838/1548731833 2024-03-16T11:56:16.389 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:16.381+0000 7f8932ffd640 1 -- 172.21.15.29:0/3632460207 <== mds.0 v2:172.21.15.152:6838/1548731833 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x562a5db1ef80 con 0x562a5dbcae90 2024-03-16T11:56:16.389 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:16.381+0000 7f8932ffd640 10 client.25661 handle_command_reply: tid=0 2024-03-16T11:56:16.389 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:16.381+0000 7f895c977740 1 client.25661 shutdown 2024-03-16T11:56:16.389 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:16.382+0000 7f895c977740 1 -- 172.21.15.29:0/3632460207 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562a5dbad4f0 msgr2=0x562a5dbb8170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:56:16.389 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:16.382+0000 7f895c977740 1 --2- 172.21.15.29:0/3632460207 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562a5dbad4f0 0x562a5dbb8170 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f8944009230 tx=0x7f894402c800 comp rx=0 tx=0).stop 2024-03-16T11:56:16.390 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:16.382+0000 7f895c977740 1 -- 172.21.15.29:0/3632460207 shutdown_connections 2024-03-16T11:56:16.390 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:16.382+0000 7f895c977740 1 --2- 172.21.15.29:0/3632460207 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562a5dbbac30 0x562a5dbb86b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:56:16.390 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:16.382+0000 7f895c977740 1 --2- 172.21.15.29:0/3632460207 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x562a5dbcae90 0x562a5dbcd280 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:56:16.390 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:16.382+0000 7f895c977740 1 --2- 172.21.15.29:0/3632460207 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562a5dbad4f0 0x562a5dbb8170 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:56:16.390 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:16.382+0000 7f895c977740 1 --2- 172.21.15.29:0/3632460207 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562a5dbacbd0 0x562a5dbb7c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:56:16.390 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:16.382+0000 7f8932ffd640 0 client.25661 ms_handle_reset on v2:172.21.15.152:6838/1548731833 2024-03-16T11:56:16.390 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:16.382+0000 7f895c977740 1 -- 172.21.15.29:0/3632460207 >> 172.21.15.29:0/3632460207 conn(0x562a5dba7a50 msgr2=0x562a5dba97a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:56:16.390 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:16.382+0000 7f895c977740 1 -- 172.21.15.29:0/3632460207 shutdown_connections 2024-03-16T11:56:16.390 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:16.382+0000 7f895c977740 1 -- 172.21.15.29:0/3632460207 wait complete. 2024-03-16T11:56:16.390 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:16.382+0000 7f895c977740 20 client.25661 trim_cache size 0 max 16384 2024-03-16T11:56:16.398 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:16.397+0000 7f895c977740 1 Processor -- start 2024-03-16T11:56:16.398 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:16.397+0000 7f895c977740 1 -- start start 2024-03-16T11:56:16.398 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:16.397+0000 7f895c977740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562a5dbacbd0 0x562a5dbc6640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:56:16.398 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:16.397+0000 7f895c977740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562a5dbad4f0 0x562a5dbb84d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:56:16.398 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:16.397+0000 7f895c977740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562a5dbbac30 0x562a5dbd2b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:56:16.398 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:16.397+0000 7f895c977740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f89541ada00 con 0x562a5dbad4f0 2024-03-16T11:56:16.398 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:16.397+0000 7f895c977740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f89541ab2d0 con 0x562a5dbacbd0 2024-03-16T11:56:16.399 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:16.397+0000 7f895c977740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f89541a9100 con 0x562a5dbbac30 2024-03-16T11:56:16.399 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:16.397+0000 7f89537fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562a5dbad4f0 0x562a5dbb84d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:56:16.399 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:16.397+0000 7f89537fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562a5dbad4f0 0x562a5dbb84d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56582/0 (socket says 172.21.15.29:56582) 2024-03-16T11:56:16.399 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:16.397+0000 7f89537fe640 1 -- 172.21.15.29:0/1726697389 learned_addr learned my addr 172.21.15.29:0/1726697389 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:56:16.399 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:16.397+0000 7f8952ffd640 1 --2- 172.21.15.29:0/1726697389 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562a5dbacbd0 0x562a5dbc6640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:56:16.399 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:16.397+0000 7f8952ffd640 1 --2- 172.21.15.29:0/1726697389 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562a5dbbac30 0x562a5dbd2b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:56:16.399 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:16.398+0000 7f89537fe640 1 -- 172.21.15.29:0/1726697389 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562a5dbbac30 msgr2=0x562a5dbd2b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:56:16.399 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:16.398+0000 7f89537fe640 1 --2- 172.21.15.29:0/1726697389 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562a5dbbac30 0x562a5dbd2b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:56:16.399 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:16.398+0000 7f89537fe640 1 -- 172.21.15.29:0/1726697389 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562a5dbacbd0 msgr2=0x562a5dbc6640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:56:16.399 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:16.398+0000 7f89537fe640 1 --2- 172.21.15.29:0/1726697389 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562a5dbacbd0 0x562a5dbc6640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:56:16.399 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:16.398+0000 7f89537fe640 1 -- 172.21.15.29:0/1726697389 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562a5dbad320 con 0x562a5dbad4f0 2024-03-16T11:56:16.399 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:16.398+0000 7f89537fe640 1 --2- 172.21.15.29:0/1726697389 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562a5dbad4f0 0x562a5dbb84d0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f8944003fe0 tx=0x7f894402ce50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b82cdc9ab1dddfea server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:56:16.399 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:16.398+0000 7f89397fa640 1 -- 172.21.15.29:0/1726697389 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8944005d50 con 0x562a5dbad4f0 2024-03-16T11:56:16.399 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:16.398+0000 7f89397fa640 1 -- 172.21.15.29:0/1726697389 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8944004ba0 con 0x562a5dbad4f0 2024-03-16T11:56:16.399 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:16.398+0000 7f89397fa640 1 -- 172.21.15.29:0/1726697389 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f894402c8b0 con 0x562a5dbad4f0 2024-03-16T11:56:16.400 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:16.399+0000 7f895c977740 1 -- 172.21.15.29:0/1726697389 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562a5dbad4f0 msgr2=0x562a5dbb84d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:56:16.400 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:16.399+0000 7f895c977740 1 --2- 172.21.15.29:0/1726697389 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562a5dbad4f0 0x562a5dbb84d0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f8944003fe0 tx=0x7f894402ce50 comp rx=0 tx=0).stop 2024-03-16T11:56:16.400 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:16.399+0000 7f895c977740 1 -- 172.21.15.29:0/1726697389 shutdown_connections 2024-03-16T11:56:16.400 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:16.399+0000 7f895c977740 1 --2- 172.21.15.29:0/1726697389 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562a5dbbac30 0x562a5dbd2b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:56:16.400 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:16.399+0000 7f895c977740 1 --2- 172.21.15.29:0/1726697389 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562a5dbad4f0 0x562a5dbb84d0 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:56:16.400 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:16.399+0000 7f895c977740 1 --2- 172.21.15.29:0/1726697389 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562a5dbacbd0 0x562a5dbc6640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:56:16.400 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:16.399+0000 7f895c977740 1 -- 172.21.15.29:0/1726697389 >> 172.21.15.29:0/1726697389 conn(0x562a5dba7a50 msgr2=0x562a5dba9730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:56:16.400 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:16.399+0000 7f895c977740 1 -- 172.21.15.29:0/1726697389 shutdown_connections 2024-03-16T11:56:16.400 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:16.399+0000 7f895c977740 1 -- 172.21.15.29:0/1726697389 wait complete. 2024-03-16T11:56:16.400 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:16.399+0000 7f895c977740 1 -- start start 2024-03-16T11:56:16.400 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:16.399+0000 7f895c977740 1 Processor -- start 2024-03-16T11:56:16.403 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:16.400+0000 7f895c977740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562a5dbacbd0 0x562a5dbc30d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:56:16.403 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:16.400+0000 7f895c977740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562a5dbad4f0 0x562a5dbbe440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:56:16.403 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:16.400+0000 7f895c977740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562a5dbbac30 0x562a5dbbeb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:56:16.403 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:16.400+0000 7f895c977740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f89541a9790 con 0x562a5dbad4f0 2024-03-16T11:56:16.403 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:16.400+0000 7f895c977740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f89541aac00 con 0x562a5dbacbd0 2024-03-16T11:56:16.403 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:16.400+0000 7f895c977740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f89541a9e60 con 0x562a5dbbac30 2024-03-16T11:56:16.403 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:16.400+0000 7f895c977740 10 client.0 _ll_register_callbacks cb 0x7f8944009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T11:56:16.404 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:16.400+0000 7f89537fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562a5dbad4f0 0x562a5dbbe440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:56:16.404 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:16.400+0000 7f89537fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562a5dbad4f0 0x562a5dbbe440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56586/0 (socket says 172.21.15.29:56586) 2024-03-16T11:56:16.404 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:16.400+0000 7f89537fe640 1 -- 172.21.15.29:0/4131535418 learned_addr learned my addr 172.21.15.29:0/4131535418 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:56:16.404 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:16.400+0000 7f8952ffd640 1 --2- 172.21.15.29:0/4131535418 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562a5dbacbd0 0x562a5dbc30d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:56:16.404 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:16.400+0000 7f8952ffd640 1 --2- 172.21.15.29:0/4131535418 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562a5dbbac30 0x562a5dbbeb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:56:16.404 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:16.400+0000 7f89537fe640 1 -- 172.21.15.29:0/4131535418 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562a5dbbac30 msgr2=0x562a5dbbeb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:56:16.404 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:16.400+0000 7f89537fe640 1 --2- 172.21.15.29:0/4131535418 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562a5dbbac30 0x562a5dbbeb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:56:16.404 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:16.400+0000 7f89537fe640 1 -- 172.21.15.29:0/4131535418 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562a5dbacbd0 msgr2=0x562a5dbc30d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:56:16.404 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:16.400+0000 7f89537fe640 1 --2- 172.21.15.29:0/4131535418 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562a5dbacbd0 0x562a5dbc30d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:56:16.404 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:16.400+0000 7f89537fe640 1 -- 172.21.15.29:0/4131535418 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562a5dc30170 con 0x562a5dbad4f0 2024-03-16T11:56:16.404 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:16.400+0000 7f8952ffd640 1 --2- 172.21.15.29:0/4131535418 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562a5dbacbd0 0x562a5dbc30d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:56:16.404 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:16.400+0000 7f89537fe640 1 --2- 172.21.15.29:0/4131535418 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562a5dbad4f0 0x562a5dbbe440 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f8944009230 tx=0x7f8944004dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cef96f5af8192619 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:56:16.404 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:16.400+0000 7f8932ffd640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-16T11:56:16.404 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:16.400+0000 7f8932ffd640 1 -- 172.21.15.29:0/4131535418 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f894402c8b0 con 0x562a5dbad4f0 2024-03-16T11:56:16.405 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:16.400+0000 7f8932ffd640 1 -- 172.21.15.29:0/4131535418 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8512+0+0 (secure 0 0 0) 0x7f894402ca50 con 0x562a5dbad4f0 2024-03-16T11:56:16.405 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:16.400+0000 7f895c977740 1 -- 172.21.15.29:0/4131535418 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8944004ba0 con 0x562a5dbad4f0 2024-03-16T11:56:16.405 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:16.400+0000 7f8932ffd640 1 -- 172.21.15.29:0/4131535418 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f89440046c0 con 0x562a5dbad4f0 2024-03-16T11:56:16.405 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:16.400+0000 7f895c977740 10 client.15852 fetch_fsmap learned FSMap version 24 2024-03-16T11:56:16.405 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:16.400+0000 7f895c977740 1 -- 172.21.15.29:0/4131535418 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x562a5dc31340 con 0x562a5dbad4f0 2024-03-16T11:56:16.405 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:16.401+0000 7f8932ffd640 1 -- 172.21.15.29:0/4131535418 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8944004ba0 con 0x562a5dbad4f0 2024-03-16T11:56:16.405 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:16.401+0000 7f8932ffd640 1 -- 172.21.15.29:0/4131535418 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7f8944060020 con 0x562a5dbad4f0 2024-03-16T11:56:16.405 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:16.401+0000 7f895c977740 10 client.15852 fetch_fsmap finished waiting for FSMap version 24 2024-03-16T11:56:16.405 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:16.401+0000 7f895c977740 10 client.15852 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T11:56:16.405 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:16.401+0000 7f895c977740 20 client.15852 populate_metadata read hostname 'smithi029' 2024-03-16T11:56:16.405 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:16.401+0000 7f895c977740 1 --2- 172.21.15.29:0/4131535418 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x562a5dc31990 0x562a5dc33db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:56:16.405 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:16.401+0000 7f895c977740 4 client.15852 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-16T11:56:16.405 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:16.401+0000 7f895c977740 1 -- 172.21.15.29:0/4131535418 --> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] -- command(tid 0: {"prefix": "flush journal"}) -- 0x562a5db192d0 con 0x562a5dc31990 2024-03-16T11:56:16.405 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:16.401+0000 7f8952ffd640 1 --2- 172.21.15.29:0/4131535418 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x562a5dc31990 0x562a5dc33db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:56:16.405 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:16.401+0000 7f8932ffd640 10 client.15852 ms_handle_connect on v2:172.21.15.152:6838/1548731833 2024-03-16T11:56:16.405 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:16.401+0000 7f8952ffd640 1 --2- 172.21.15.29:0/4131535418 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x562a5dc31990 0x562a5dc33db0 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=25398cc035a998fb server_cookie=ecb058fb2885ee79 in_seq=0 out_seq=0 2024-03-16T11:56:16.641 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:16.642+0000 7f8932ffd640 1 -- 172.21.15.29:0/4131535418 <== mds.0 v2:172.21.15.152:6838/1548731833 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x562a5db192d0 con 0x562a5dc31990 2024-03-16T11:56:16.642 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:16.642+0000 7f8932ffd640 10 client.15852 handle_command_reply: tid=0 2024-03-16T11:56:16.642 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:16.642+0000 7f895c977740 1 client.15852 shutdown 2024-03-16T11:56:16.643 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:16.642+0000 7f895c977740 1 -- 172.21.15.29:0/4131535418 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562a5dbad4f0 msgr2=0x562a5dbbe440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:56:16.643 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:16.642+0000 7f895c977740 1 --2- 172.21.15.29:0/4131535418 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562a5dbad4f0 0x562a5dbbe440 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f8944009230 tx=0x7f8944004dc0 comp rx=0 tx=0).stop 2024-03-16T11:56:16.643 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:16.642+0000 7f895c977740 1 -- 172.21.15.29:0/4131535418 shutdown_connections 2024-03-16T11:56:16.643 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:16.642+0000 7f895c977740 1 --2- 172.21.15.29:0/4131535418 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562a5dbbac30 0x562a5dbbeb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:56:16.643 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:16.642+0000 7f895c977740 1 --2- 172.21.15.29:0/4131535418 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562a5dbad4f0 0x562a5dbbe440 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:56:16.643 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:16.642+0000 7f895c977740 1 --2- 172.21.15.29:0/4131535418 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x562a5dc31990 0x562a5dc33db0 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:56:16.644 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:16.642+0000 7f895c977740 1 --2- 172.21.15.29:0/4131535418 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562a5dbacbd0 0x562a5dbc30d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:56:16.644 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:16.642+0000 7f8932ffd640 0 client.15852 ms_handle_reset on v2:172.21.15.152:6838/1548731833 2024-03-16T11:56:16.644 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:16.642+0000 7f895c977740 1 -- 172.21.15.29:0/4131535418 >> 172.21.15.29:0/4131535418 conn(0x562a5dba7a50 msgr2=0x562a5dba6320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:56:16.644 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:16.643+0000 7f895c977740 1 -- 172.21.15.29:0/4131535418 shutdown_connections 2024-03-16T11:56:16.644 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:16.643+0000 7f895c977740 1 -- 172.21.15.29:0/4131535418 wait complete. 2024-03-16T11:56:16.644 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:16.643+0000 7f895c977740 20 client.15852 trim_cache size 0 max 16384 2024-03-16T11:56:16.645 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:{ 2024-03-16T11:56:16.645 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "message": "", 2024-03-16T11:56:16.645 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "return_code": 0 2024-03-16T11:56:16.645 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:} 2024-03-16T11:56:16.646 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:16.643+0000 7f8959d90640 1 -- 172.21.15.29:0/1666038505 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f893c0a2a10 msgr2=0x7f893c0a4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:56:16.646 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:16.643+0000 7f8959d90640 1 --2- 172.21.15.29:0/1666038505 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f893c0a2a10 0x7f893c0a4ed0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f894c097ea0 tx=0x7f894c086040 comp rx=0 tx=0).stop 2024-03-16T11:56:16.646 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:16.643+0000 7f8959d90640 1 -- 172.21.15.29:0/1666038505 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8954107860 msgr2=0x7f895419c820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:56:16.646 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:16.643+0000 7f8959d90640 1 --2- 172.21.15.29:0/1666038505 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8954107860 0x7f895419c820 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f894c00eb10 tx=0x7f894c00c2d0 comp rx=0 tx=0).stop 2024-03-16T11:56:16.646 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:16.643+0000 7f8959d90640 1 -- 172.21.15.29:0/1666038505 shutdown_connections 2024-03-16T11:56:16.647 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:16.643+0000 7f8959d90640 1 --2- 172.21.15.29:0/1666038505 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f893c0a2a10 0x7f893c0a4ed0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:56:16.647 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:16.643+0000 7f8959d90640 1 --2- 172.21.15.29:0/1666038505 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8954107860 0x7f895419c820 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:56:16.647 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:16.643+0000 7f8959d90640 1 --2- 172.21.15.29:0/1666038505 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f895406a9a0 0x7f89541984a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:56:16.647 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:16.643+0000 7f8959d90640 1 --2- 172.21.15.29:0/1666038505 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8954069fe0 0x7f8954197f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:56:16.647 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:16.643+0000 7f8959d90640 1 -- 172.21.15.29:0/1666038505 >> 172.21.15.29:0/1666038505 conn(0x7f89540774b0 msgr2=0x7f89540fe330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:56:16.647 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:16.643+0000 7f8959d90640 1 -- 172.21.15.29:0/1666038505 shutdown_connections 2024-03-16T11:56:16.647 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:16.643+0000 7f8959d90640 1 -- 172.21.15.29:0/1666038505 wait complete. 2024-03-16T11:56:16.788 INFO:tasks.workunit.client.0.smithi029.stdout: 1 532711 11.77 MB/sec execute 121 sec latency 114.099 ms 2024-03-16T11:56:17.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:56:16 smithi029 ceph-mon[31975]: pgmap v165: 97 pgs: 97 active+clean; 55 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 3.0 MiB/s wr, 312 op/s 2024-03-16T11:56:17.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:56:16 smithi152 ceph-mon[36291]: pgmap v165: 97 pgs: 97 active+clean; 55 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 3.0 MiB/s wr, 312 op/s 2024-03-16T11:56:17.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:56:16 smithi154 ceph-mon[36320]: pgmap v165: 97 pgs: 97 active+clean; 55 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 3.0 MiB/s wr, 312 op/s 2024-03-16T11:56:17.785 INFO:tasks.workunit.client.0.smithi029.stdout: 1 533995 11.73 MB/sec execute 122 sec latency 123.071 ms 2024-03-16T11:56:18.785 INFO:tasks.workunit.client.0.smithi029.stdout: 1 536690 11.77 MB/sec execute 123 sec latency 12.724 ms 2024-03-16T11:56:19.087 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:56:18 smithi029 ceph-mon[31975]: pgmap v166: 97 pgs: 97 active+clean; 61 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 4.5 MiB/s wr, 408 op/s 2024-03-16T11:56:19.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:56:18 smithi152 ceph-mon[36291]: pgmap v166: 97 pgs: 97 active+clean; 61 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 4.5 MiB/s wr, 408 op/s 2024-03-16T11:56:19.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:56:18 smithi154 ceph-mon[36320]: pgmap v166: 97 pgs: 97 active+clean; 61 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 4.5 MiB/s wr, 408 op/s 2024-03-16T11:56:19.785 INFO:tasks.workunit.client.0.smithi029.stdout: 1 538929 11.74 MB/sec execute 124 sec latency 19.306 ms 2024-03-16T11:56:20.785 INFO:tasks.workunit.client.0.smithi029.stdout: 1 540792 11.74 MB/sec execute 125 sec latency 20.949 ms 2024-03-16T11:56:21.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:56:20 smithi029 ceph-mon[31975]: pgmap v167: 97 pgs: 97 active+clean; 61 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 995 KiB/s rd, 3.0 MiB/s wr, 193 op/s 2024-03-16T11:56:21.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:56:20 smithi152 ceph-mon[36291]: pgmap v167: 97 pgs: 97 active+clean; 61 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 995 KiB/s rd, 3.0 MiB/s wr, 193 op/s 2024-03-16T11:56:21.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:56:20 smithi154 ceph-mon[36320]: pgmap v167: 97 pgs: 97 active+clean; 61 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 995 KiB/s rd, 3.0 MiB/s wr, 193 op/s 2024-03-16T11:56:21.786 INFO:tasks.workunit.client.0.smithi029.stdout: 1 543487 11.79 MB/sec execute 126 sec latency 70.092 ms 2024-03-16T11:56:22.786 INFO:tasks.workunit.client.0.smithi029.stdout: 1 544675 11.72 MB/sec execute 127 sec latency 149.530 ms 2024-03-16T11:56:23.087 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:56:22 smithi029 ceph-mon[31975]: pgmap v168: 97 pgs: 97 active+clean; 70 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 6.0 MiB/s wr, 388 op/s 2024-03-16T11:56:23.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:56:22 smithi152 ceph-mon[36291]: pgmap v168: 97 pgs: 97 active+clean; 70 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 6.0 MiB/s wr, 388 op/s 2024-03-16T11:56:23.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:56:22 smithi154 ceph-mon[36320]: pgmap v168: 97 pgs: 97 active+clean; 70 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 6.0 MiB/s wr, 388 op/s 2024-03-16T11:56:23.786 INFO:tasks.workunit.client.0.smithi029.stdout: 1 547064 11.75 MB/sec execute 128 sec latency 19.373 ms 2024-03-16T11:56:24.786 INFO:tasks.workunit.client.0.smithi029.stdout: 1 548480 11.71 MB/sec execute 129 sec latency 44.247 ms 2024-03-16T11:56:25.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:56:24 smithi029 ceph-mon[31975]: pgmap v169: 97 pgs: 97 active+clean; 70 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 4.5 MiB/s wr, 290 op/s 2024-03-16T11:56:25.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:56:24 smithi152 ceph-mon[36291]: pgmap v169: 97 pgs: 97 active+clean; 70 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 4.5 MiB/s wr, 290 op/s 2024-03-16T11:56:25.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:56:24 smithi154 ceph-mon[36320]: pgmap v169: 97 pgs: 97 active+clean; 70 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 4.5 MiB/s wr, 290 op/s 2024-03-16T11:56:25.786 INFO:tasks.workunit.client.0.smithi029.stdout: 1 550708 11.72 MB/sec execute 130 sec latency 24.728 ms 2024-03-16T11:56:26.786 INFO:tasks.workunit.client.0.smithi029.stdout: 1 552132 11.68 MB/sec execute 131 sec latency 18.143 ms 2024-03-16T11:56:27.087 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:56:26 smithi029 ceph-mon[31975]: pgmap v170: 97 pgs: 97 active+clean; 70 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 4.5 MiB/s wr, 290 op/s 2024-03-16T11:56:27.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:56:26 smithi152 ceph-mon[36291]: pgmap v170: 97 pgs: 97 active+clean; 70 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 4.5 MiB/s wr, 290 op/s 2024-03-16T11:56:27.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:56:26 smithi154 ceph-mon[36320]: pgmap v170: 97 pgs: 97 active+clean; 70 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 4.5 MiB/s wr, 290 op/s 2024-03-16T11:56:27.786 INFO:tasks.workunit.client.0.smithi029.stdout: 1 554142 11.68 MB/sec execute 132 sec latency 135.129 ms 2024-03-16T11:56:28.786 INFO:tasks.workunit.client.0.smithi029.stdout: 1 555653 11.65 MB/sec execute 133 sec latency 22.994 ms 2024-03-16T11:56:29.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:56:28 smithi029 ceph-mon[31975]: pgmap v171: 97 pgs: 97 active+clean; 74 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 5.8 MiB/s wr, 385 op/s 2024-03-16T11:56:29.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:56:28 smithi152 ceph-mon[36291]: pgmap v171: 97 pgs: 97 active+clean; 74 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 5.8 MiB/s wr, 385 op/s 2024-03-16T11:56:29.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:56:28 smithi154 ceph-mon[36320]: pgmap v171: 97 pgs: 97 active+clean; 74 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 5.8 MiB/s wr, 385 op/s 2024-03-16T11:56:29.732 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:29.730+0000 7f3ba7e12640 1 -- 172.21.15.29:0/868493364 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3ba0101790 msgr2=0x7f3ba0101c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:56:29.733 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:29.730+0000 7f3ba7e12640 1 --2- 172.21.15.29:0/868493364 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3ba0101790 0x7f3ba0101c10 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f3b90009930 tx=0x7f3b9002bc30 comp rx=0 tx=0).stop 2024-03-16T11:56:29.733 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:29.730+0000 7f3ba7e12640 1 -- 172.21.15.29:0/868493364 shutdown_connections 2024-03-16T11:56:29.733 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:29.730+0000 7f3ba7e12640 1 --2- 172.21.15.29:0/868493364 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3ba0102150 0x7f3ba010e660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:56:29.733 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:29.730+0000 7f3ba7e12640 1 --2- 172.21.15.29:0/868493364 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3ba0101790 0x7f3ba0101c10 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:56:29.733 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:29.730+0000 7f3ba7e12640 1 --2- 172.21.15.29:0/868493364 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3ba01077a0 0x7f3ba0107b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:56:29.733 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:29.730+0000 7f3ba7e12640 1 -- 172.21.15.29:0/868493364 >> 172.21.15.29:0/868493364 conn(0x7f3ba00fd2a0 msgr2=0x7f3ba00ff6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:56:29.733 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:29.730+0000 7f3ba7e12640 1 -- 172.21.15.29:0/868493364 shutdown_connections 2024-03-16T11:56:29.733 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:29.730+0000 7f3ba7e12640 1 -- 172.21.15.29:0/868493364 wait complete. 2024-03-16T11:56:29.733 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:29.731+0000 7f3ba7e12640 1 Processor -- start 2024-03-16T11:56:29.733 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:29.731+0000 7f3ba7e12640 1 -- start start 2024-03-16T11:56:29.733 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:29.731+0000 7f3ba7e12640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3ba0101790 0x7f3ba0073ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:56:29.734 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:29.731+0000 7f3ba7e12640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3ba0102150 0x7f3ba0074020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:56:29.734 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:29.731+0000 7f3ba7e12640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3ba01077a0 0x7f3ba0074560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:56:29.734 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:29.731+0000 7f3ba7e12640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f3ba0111bb0 con 0x7f3ba0102150 2024-03-16T11:56:29.734 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:29.731+0000 7f3ba7e12640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f3ba0111a30 con 0x7f3ba0101790 2024-03-16T11:56:29.734 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:29.731+0000 7f3ba7e12640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f3ba0111d30 con 0x7f3ba01077a0 2024-03-16T11:56:29.734 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:29.731+0000 7f3ba53ad640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3ba0102150 0x7f3ba0074020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:56:29.734 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:29.731+0000 7f3ba53ad640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3ba0102150 0x7f3ba0074020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42800/0 (socket says 172.21.15.29:42800) 2024-03-16T11:56:29.734 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:29.731+0000 7f3ba53ad640 1 -- 172.21.15.29:0/619632278 learned_addr learned my addr 172.21.15.29:0/619632278 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:56:29.734 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:29.731+0000 7f3ba5bae640 1 --2- 172.21.15.29:0/619632278 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3ba0101790 0x7f3ba0073ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:56:29.734 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:29.731+0000 7f3ba63af640 1 --2- 172.21.15.29:0/619632278 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3ba01077a0 0x7f3ba0074560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:56:29.734 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:29.731+0000 7f3ba53ad640 1 -- 172.21.15.29:0/619632278 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3ba01077a0 msgr2=0x7f3ba0074560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:56:29.735 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:29.731+0000 7f3ba53ad640 1 --2- 172.21.15.29:0/619632278 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3ba01077a0 0x7f3ba0074560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:56:29.735 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:29.731+0000 7f3ba53ad640 1 -- 172.21.15.29:0/619632278 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3ba0101790 msgr2=0x7f3ba0073ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:56:29.735 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:29.731+0000 7f3ba53ad640 1 --2- 172.21.15.29:0/619632278 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3ba0101790 0x7f3ba0073ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:56:29.735 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:29.731+0000 7f3ba53ad640 1 -- 172.21.15.29:0/619632278 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ba01a64f0 con 0x7f3ba0102150 2024-03-16T11:56:29.735 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:29.732+0000 7f3ba53ad640 1 --2- 172.21.15.29:0/619632278 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3ba0102150 0x7f3ba0074020 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f3b90009230 tx=0x7f3b9002cb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:56:29.735 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:29.732+0000 7f3b96ffd640 1 -- 172.21.15.29:0/619632278 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b90041070 con 0x7f3ba0102150 2024-03-16T11:56:29.735 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:29.732+0000 7f3ba7e12640 1 -- 172.21.15.29:0/619632278 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3ba01a6780 con 0x7f3ba0102150 2024-03-16T11:56:29.735 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:29.732+0000 7f3b96ffd640 1 -- 172.21.15.29:0/619632278 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3b90004450 con 0x7f3ba0102150 2024-03-16T11:56:29.735 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:29.732+0000 7f3b96ffd640 1 -- 172.21.15.29:0/619632278 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b90003e50 con 0x7f3ba0102150 2024-03-16T11:56:29.735 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:29.732+0000 7f3ba7e12640 1 -- 172.21.15.29:0/619632278 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3ba01a6cc0 con 0x7f3ba0102150 2024-03-16T11:56:29.741 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:29.734+0000 7f3b96ffd640 1 -- 172.21.15.29:0/619632278 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7f3b90044070 con 0x7f3ba0102150 2024-03-16T11:56:29.741 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:29.735+0000 7f3baaa04740 1 Processor -- start 2024-03-16T11:56:29.742 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:29.735+0000 7f3baaa04740 1 -- start start 2024-03-16T11:56:29.742 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:29.735+0000 7f3b96ffd640 1 --2- 172.21.15.29:0/619632278 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f3b8c0a2bb0 0x7f3b8c0a5070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:56:29.742 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:29.735+0000 7f3baaa04740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5613f80cd130 0x5613f80cd510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:56:29.742 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:29.735+0000 7f3baaa04740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5613f80cda50 0x5613f80d62e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:56:29.742 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:29.735+0000 7f3ba63af640 1 --2- 172.21.15.29:0/619632278 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f3b8c0a2bb0 0x7f3b8c0a5070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:56:29.742 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:29.735+0000 7f3baaa04740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5613f80d6820 0x5613f80d8c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:56:29.742 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:29.735+0000 7f3b96ffd640 1 -- 172.21.15.29:0/619632278 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8512+0+0 (secure 0 0 0) 0x7f3b900f0650 con 0x7f3ba0102150 2024-03-16T11:56:29.742 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:29.735+0000 7f3baaa04740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x5613f8031a40 con 0x5613f80d6820 2024-03-16T11:56:29.742 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:29.735+0000 7f3baaa04740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x5613f7fda920 con 0x5613f80cda50 2024-03-16T11:56:29.742 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:29.735+0000 7f3baaa04740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5613f802dae0 con 0x5613f80cd130 2024-03-16T11:56:29.742 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:29.735+0000 7f3ba63af640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5613f80d6820 0x5613f80d8c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:56:29.742 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:29.735+0000 7f3ba63af640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5613f80d6820 0x5613f80d8c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42804/0 (socket says 172.21.15.29:42804) 2024-03-16T11:56:29.742 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:29.735+0000 7f3ba63af640 1 -- 172.21.15.29:0/1560020296 learned_addr learned my addr 172.21.15.29:0/1560020296 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:56:29.743 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:29.735+0000 7f3ba5bae640 1 --2- 172.21.15.29:0/1560020296 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5613f80cd130 0x5613f80cd510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:56:29.743 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:29.735+0000 7f3ba53ad640 1 --2- 172.21.15.29:0/1560020296 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5613f80cda50 0x5613f80d62e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:56:29.743 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:29.735+0000 7f3ba5bae640 1 -- 172.21.15.29:0/1560020296 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5613f80cda50 msgr2=0x5613f80d62e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:56:29.743 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:29.735+0000 7f3ba5bae640 1 --2- 172.21.15.29:0/1560020296 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5613f80cda50 0x5613f80d62e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:56:29.743 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:29.735+0000 7f3ba5bae640 1 -- 172.21.15.29:0/1560020296 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5613f80d6820 msgr2=0x5613f80d8c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:56:29.743 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:29.735+0000 7f3ba5bae640 1 --2- 172.21.15.29:0/1560020296 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5613f80d6820 0x5613f80d8c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:56:29.743 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:29.736+0000 7f3ba5bae640 1 -- 172.21.15.29:0/1560020296 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5613f7ff2460 con 0x5613f80cd130 2024-03-16T11:56:29.743 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:29.736+0000 7f3ba63af640 1 --2- 172.21.15.29:0/1560020296 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5613f80d6820 0x5613f80d8c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:56:29.743 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:29.736+0000 7f3ba63af640 1 --2- 172.21.15.29:0/619632278 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f3b8c0a2bb0 0x7f3b8c0a5070 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f3b98013cf0 tx=0x7f3b98009f10 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:56:29.743 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:29.736+0000 7f3ba5bae640 1 --2- 172.21.15.29:0/1560020296 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5613f80cd130 0x5613f80cd510 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f3b9c009c20 tx=0x7f3b9c011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3c5b9ca6d3a57e7c server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:56:29.743 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:29.736+0000 7f3b8bfff640 1 -- 172.21.15.29:0/1560020296 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b9c004050 con 0x5613f80cd130 2024-03-16T11:56:29.743 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:29.736+0000 7f3b8bfff640 1 -- 172.21.15.29:0/1560020296 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3b9c005500 con 0x5613f80cd130 2024-03-16T11:56:29.743 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:29.736+0000 7f3b8bfff640 1 -- 172.21.15.29:0/1560020296 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b9c004d00 con 0x5613f80cd130 2024-03-16T11:56:29.743 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:29.737+0000 7f3baaa04740 1 -- 172.21.15.29:0/1560020296 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5613f80cd130 msgr2=0x5613f80cd510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:56:29.744 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:29.737+0000 7f3baaa04740 1 --2- 172.21.15.29:0/1560020296 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5613f80cd130 0x5613f80cd510 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f3b9c009c20 tx=0x7f3b9c011810 comp rx=0 tx=0).stop 2024-03-16T11:56:29.744 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:29.737+0000 7f3baaa04740 1 -- 172.21.15.29:0/1560020296 shutdown_connections 2024-03-16T11:56:29.744 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:29.737+0000 7f3baaa04740 1 --2- 172.21.15.29:0/1560020296 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5613f80d6820 0x5613f80d8c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:56:29.744 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:29.737+0000 7f3baaa04740 1 --2- 172.21.15.29:0/1560020296 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5613f80cda50 0x5613f80d62e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:56:29.744 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:29.737+0000 7f3baaa04740 1 --2- 172.21.15.29:0/1560020296 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5613f80cd130 0x5613f80cd510 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:56:29.744 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:29.737+0000 7f3baaa04740 1 -- 172.21.15.29:0/1560020296 >> 172.21.15.29:0/1560020296 conn(0x5613f80c7a50 msgr2=0x5613f80c7e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:56:29.744 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:29.737+0000 7f3baaa04740 1 -- 172.21.15.29:0/1560020296 shutdown_connections 2024-03-16T11:56:29.744 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:29.737+0000 7f3baaa04740 1 -- 172.21.15.29:0/1560020296 wait complete. 2024-03-16T11:56:29.744 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:29.737+0000 7f3baaa04740 1 -- start start 2024-03-16T11:56:29.744 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:29.737+0000 7f3baaa04740 1 Processor -- start 2024-03-16T11:56:29.744 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:29.738+0000 7f3baaa04740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5613f80cd130 0x5613f80d37e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:56:29.744 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:29.738+0000 7f3baaa04740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5613f80cda50 0x5613f80d3d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:56:29.744 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:29.738+0000 7f3baaa04740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5613f80d6820 0x5613f80d4260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:56:29.744 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:29.738+0000 7f3baaa04740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x5613f80d9450 con 0x5613f80d6820 2024-03-16T11:56:29.744 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:29.738+0000 7f3baaa04740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x5613f7fedd70 con 0x5613f80cd130 2024-03-16T11:56:29.745 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:29.738+0000 7f3baaa04740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5613f80d95d0 con 0x5613f80cda50 2024-03-16T11:56:29.745 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:29.738+0000 7f3baaa04740 10 client.0 _ll_register_callbacks cb 0x5613f7d45a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T11:56:29.745 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:29.738+0000 7f3ba63af640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5613f80d6820 0x5613f80d4260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:56:29.745 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:29.738+0000 7f3ba5bae640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5613f80cd130 0x5613f80d37e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:56:29.745 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:29.738+0000 7f3ba53ad640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5613f80cda50 0x5613f80d3d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:56:29.745 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:29.738+0000 7f3ba5bae640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5613f80cd130 0x5613f80d37e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.29:57748/0 (socket says 172.21.15.29:57748) 2024-03-16T11:56:29.745 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:29.738+0000 7f3ba5bae640 1 -- 172.21.15.29:0/1736310950 learned_addr learned my addr 172.21.15.29:0/1736310950 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:56:29.745 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:29.738+0000 7f3ba63af640 1 -- 172.21.15.29:0/1736310950 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5613f80cda50 msgr2=0x5613f80d3d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:56:29.745 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:29.738+0000 7f3ba63af640 1 --2- 172.21.15.29:0/1736310950 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5613f80cda50 0x5613f80d3d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:56:29.745 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:29.738+0000 7f3ba63af640 1 -- 172.21.15.29:0/1736310950 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5613f80cd130 msgr2=0x5613f80d37e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:56:29.745 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:29.738+0000 7f3ba63af640 1 --2- 172.21.15.29:0/1736310950 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5613f80cd130 0x5613f80d37e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:56:29.745 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:29.738+0000 7f3ba63af640 1 -- 172.21.15.29:0/1736310950 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5613f7d46210 con 0x5613f80d6820 2024-03-16T11:56:29.745 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:29.738+0000 7f3ba5bae640 1 --2- 172.21.15.29:0/1736310950 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5613f80cd130 0x5613f80d37e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:56:29.745 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:29.738+0000 7f3ba63af640 1 --2- 172.21.15.29:0/1736310950 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5613f80d6820 0x5613f80d4260 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f3b98009270 tx=0x7f3b98004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64b4d33ba45e313e server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:56:29.746 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:29.738+0000 7f3b897fa640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-16T11:56:29.746 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:29.738+0000 7f3b897fa640 1 -- 172.21.15.29:0/1736310950 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b98028070 con 0x5613f80d6820 2024-03-16T11:56:29.746 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:29.738+0000 7f3b897fa640 1 -- 172.21.15.29:0/1736310950 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8512+0+0 (secure 0 0 0) 0x7f3b98003970 con 0x5613f80d6820 2024-03-16T11:56:29.746 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:29.738+0000 7f3baaa04740 1 -- 172.21.15.29:0/1736310950 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3b9c005500 con 0x5613f80d6820 2024-03-16T11:56:29.746 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:29.738+0000 7f3b897fa640 1 -- 172.21.15.29:0/1736310950 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f3b98033050 con 0x5613f80d6820 2024-03-16T11:56:29.746 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:29.739+0000 7f3baaa04740 10 client.15864 fetch_fsmap learned FSMap version 24 2024-03-16T11:56:29.746 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:29.739+0000 7f3baaa04740 1 -- 172.21.15.29:0/1736310950 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x5613f7e12620 con 0x5613f80d6820 2024-03-16T11:56:29.746 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:29.739+0000 7f3b897fa640 1 -- 172.21.15.29:0/1736310950 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3b9c005500 con 0x5613f80d6820 2024-03-16T11:56:29.746 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:29.739+0000 7f3b897fa640 1 -- 172.21.15.29:0/1736310950 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7f3b98020050 con 0x5613f80d6820 2024-03-16T11:56:29.746 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:29.739+0000 7f3baaa04740 10 client.15864 fetch_fsmap finished waiting for FSMap version 24 2024-03-16T11:56:29.746 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:29.739+0000 7f3baaa04740 10 client.15864 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T11:56:29.746 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:29.739+0000 7f3baaa04740 20 client.15864 populate_metadata read hostname 'smithi029' 2024-03-16T11:56:29.746 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:29.739+0000 7f3baaa04740 1 --2- 172.21.15.29:0/1736310950 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x5613f80e6960 0x5613f80e8d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:56:29.746 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:29.739+0000 7f3baaa04740 4 client.15864 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T11:56:29.747 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:29.739+0000 7f3baaa04740 1 -- 172.21.15.29:0/1736310950 --> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5613f803ef80 con 0x5613f80e6960 2024-03-16T11:56:29.747 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:29.739+0000 7f3ba5bae640 1 --2- 172.21.15.29:0/1736310950 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x5613f80e6960 0x5613f80e8d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:56:29.747 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:29.740+0000 7f3b897fa640 10 client.15864 ms_handle_connect on v2:172.21.15.152:6838/1548731833 2024-03-16T11:56:29.747 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:29.740+0000 7f3ba5bae640 1 --2- 172.21.15.29:0/1736310950 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x5613f80e6960 0x5613f80e8d70 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8420b8f5f01dac2f server_cookie=5f7b2de8a21ae8e4 in_seq=0 out_seq=0 2024-03-16T11:56:29.748 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:29.745+0000 7f3b897fa640 1 -- 172.21.15.29:0/1736310950 <== mds.0 v2:172.21.15.152:6838/1548731833 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5613f803ef80 con 0x5613f80e6960 2024-03-16T11:56:29.748 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:29.745+0000 7f3b897fa640 10 client.15864 handle_command_reply: tid=0 2024-03-16T11:56:29.748 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:29.745+0000 7f3baaa04740 1 client.15864 shutdown 2024-03-16T11:56:29.748 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:29.745+0000 7f3baaa04740 1 -- 172.21.15.29:0/1736310950 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5613f80d6820 msgr2=0x5613f80d4260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:56:29.748 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:29.745+0000 7f3baaa04740 1 --2- 172.21.15.29:0/1736310950 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5613f80d6820 0x5613f80d4260 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f3b98009270 tx=0x7f3b98004ab0 comp rx=0 tx=0).stop 2024-03-16T11:56:29.748 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:29.745+0000 7f3baaa04740 1 -- 172.21.15.29:0/1736310950 shutdown_connections 2024-03-16T11:56:29.748 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:29.745+0000 7f3baaa04740 1 --2- 172.21.15.29:0/1736310950 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5613f80d6820 0x5613f80d4260 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:56:29.748 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:29.745+0000 7f3baaa04740 1 --2- 172.21.15.29:0/1736310950 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5613f80cda50 0x5613f80d3d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:56:29.748 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:29.745+0000 7f3baaa04740 1 --2- 172.21.15.29:0/1736310950 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x5613f80e6960 0x5613f80e8d70 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:56:29.748 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:29.745+0000 7f3baaa04740 1 --2- 172.21.15.29:0/1736310950 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5613f80cd130 0x5613f80d37e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:56:29.749 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:29.745+0000 7f3b897fa640 0 client.15864 ms_handle_reset on v2:172.21.15.152:6838/1548731833 2024-03-16T11:56:29.749 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:29.745+0000 7f3baaa04740 1 -- 172.21.15.29:0/1736310950 >> 172.21.15.29:0/1736310950 conn(0x5613f80c7a50 msgr2=0x5613f80c97a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:56:29.749 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:29.745+0000 7f3baaa04740 1 -- 172.21.15.29:0/1736310950 shutdown_connections 2024-03-16T11:56:29.749 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:29.745+0000 7f3baaa04740 1 -- 172.21.15.29:0/1736310950 wait complete. 2024-03-16T11:56:29.749 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:29.745+0000 7f3baaa04740 20 client.15864 trim_cache size 0 max 16384 2024-03-16T11:56:29.765 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:29.760+0000 7f3baaa04740 1 Processor -- start 2024-03-16T11:56:29.765 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:29.760+0000 7f3baaa04740 1 -- start start 2024-03-16T11:56:29.765 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:29.760+0000 7f3baaa04740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5613f80cd130 0x5613f80d5f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:56:29.765 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:29.760+0000 7f3baaa04740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5613f80cda50 0x5613f80e21f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:56:29.765 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:29.760+0000 7f3baaa04740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5613f80d6820 0x5613f80c2a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:56:29.765 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:29.760+0000 7f3baaa04740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f3ba01b1f40 con 0x5613f80d6820 2024-03-16T11:56:29.765 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:29.760+0000 7f3baaa04740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f3ba01af810 con 0x5613f80cda50 2024-03-16T11:56:29.765 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:29.760+0000 7f3baaa04740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f3ba01ad640 con 0x5613f80cd130 2024-03-16T11:56:29.765 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:29.760+0000 7f3ba63af640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5613f80d6820 0x5613f80c2a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:56:29.765 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:29.761+0000 7f3ba63af640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5613f80d6820 0x5613f80c2a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42828/0 (socket says 172.21.15.29:42828) 2024-03-16T11:56:29.765 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:29.761+0000 7f3ba63af640 1 -- 172.21.15.29:0/3256180018 learned_addr learned my addr 172.21.15.29:0/3256180018 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:56:29.766 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:29.761+0000 7f3ba5bae640 1 --2- 172.21.15.29:0/3256180018 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5613f80cd130 0x5613f80d5f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:56:29.766 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:29.761+0000 7f3ba53ad640 1 --2- 172.21.15.29:0/3256180018 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5613f80cda50 0x5613f80e21f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:56:29.766 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:29.761+0000 7f3ba63af640 1 -- 172.21.15.29:0/3256180018 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5613f80cd130 msgr2=0x5613f80d5f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:56:29.766 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:29.761+0000 7f3ba63af640 1 --2- 172.21.15.29:0/3256180018 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5613f80cd130 0x5613f80d5f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:56:29.766 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:29.761+0000 7f3ba63af640 1 -- 172.21.15.29:0/3256180018 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5613f80cda50 msgr2=0x5613f80e21f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:56:29.766 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:29.761+0000 7f3ba63af640 1 --2- 172.21.15.29:0/3256180018 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5613f80cda50 0x5613f80e21f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:56:29.766 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:29.761+0000 7f3ba63af640 1 -- 172.21.15.29:0/3256180018 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5613f80d27b0 con 0x5613f80d6820 2024-03-16T11:56:29.766 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:29.761+0000 7f3ba53ad640 1 --2- 172.21.15.29:0/3256180018 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5613f80cda50 0x5613f80e21f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T11:56:29.766 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:29.761+0000 7f3ba63af640 1 --2- 172.21.15.29:0/3256180018 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5613f80d6820 0x5613f80c2a40 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f3b98013810 tx=0x7f3b980135a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3aa8c73f52bfd653 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:56:29.766 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:29.761+0000 7f3b8b7fe640 1 -- 172.21.15.29:0/3256180018 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b98028040 con 0x5613f80d6820 2024-03-16T11:56:29.766 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:29.761+0000 7f3b8b7fe640 1 -- 172.21.15.29:0/3256180018 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3b98003e50 con 0x5613f80d6820 2024-03-16T11:56:29.766 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:29.761+0000 7f3b8b7fe640 1 -- 172.21.15.29:0/3256180018 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b980216e0 con 0x5613f80d6820 2024-03-16T11:56:29.766 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:29.762+0000 7f3baaa04740 1 -- 172.21.15.29:0/3256180018 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5613f80d6820 msgr2=0x5613f80c2a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:56:29.767 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:29.762+0000 7f3baaa04740 1 --2- 172.21.15.29:0/3256180018 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5613f80d6820 0x5613f80c2a40 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f3b98013810 tx=0x7f3b980135a0 comp rx=0 tx=0).stop 2024-03-16T11:56:29.767 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:29.762+0000 7f3baaa04740 1 -- 172.21.15.29:0/3256180018 shutdown_connections 2024-03-16T11:56:29.767 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:29.762+0000 7f3baaa04740 1 --2- 172.21.15.29:0/3256180018 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5613f80d6820 0x5613f80c2a40 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:56:29.767 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:29.762+0000 7f3baaa04740 1 --2- 172.21.15.29:0/3256180018 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5613f80cda50 0x5613f80e21f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:56:29.767 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:29.762+0000 7f3baaa04740 1 --2- 172.21.15.29:0/3256180018 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5613f80cd130 0x5613f80d5f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:56:29.767 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:29.762+0000 7f3baaa04740 1 -- 172.21.15.29:0/3256180018 >> 172.21.15.29:0/3256180018 conn(0x5613f80c7a50 msgr2=0x5613f80d4030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:56:29.767 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:29.762+0000 7f3baaa04740 1 -- 172.21.15.29:0/3256180018 shutdown_connections 2024-03-16T11:56:29.767 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:29.762+0000 7f3baaa04740 1 -- 172.21.15.29:0/3256180018 wait complete. 2024-03-16T11:56:29.767 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:29.762+0000 7f3baaa04740 1 -- start start 2024-03-16T11:56:29.767 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:29.762+0000 7f3baaa04740 1 Processor -- start 2024-03-16T11:56:29.767 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:29.762+0000 7f3baaa04740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5613f80cd130 0x5613f80d0ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:56:29.767 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:29.762+0000 7f3baaa04740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5613f80cda50 0x5613f80d1020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:56:29.767 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:29.762+0000 7f3baaa04740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5613f80d6820 0x5613f814bcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:56:29.767 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:29.762+0000 7f3baaa04740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f3ba01adcd0 con 0x5613f80cd130 2024-03-16T11:56:29.768 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:29.762+0000 7f3baaa04740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f3ba01af140 con 0x5613f80cda50 2024-03-16T11:56:29.768 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:29.762+0000 7f3baaa04740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f3ba01ae3a0 con 0x5613f80d6820 2024-03-16T11:56:29.768 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:29.762+0000 7f3baaa04740 10 client.0 _ll_register_callbacks cb 0x7f3b98013760 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T11:56:29.768 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:29.763+0000 7f3ba5bae640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5613f80cd130 0x5613f80d0ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:56:29.768 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:29.763+0000 7f3ba5bae640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5613f80cd130 0x5613f80d0ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42830/0 (socket says 172.21.15.29:42830) 2024-03-16T11:56:29.768 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:29.763+0000 7f3ba5bae640 1 -- 172.21.15.29:0/626720048 learned_addr learned my addr 172.21.15.29:0/626720048 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:56:29.768 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:29.763+0000 7f3ba53ad640 1 --2- 172.21.15.29:0/626720048 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5613f80cda50 0x5613f80d1020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:56:29.768 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:29.763+0000 7f3ba63af640 1 --2- 172.21.15.29:0/626720048 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5613f80d6820 0x5613f814bcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:56:29.768 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:29.763+0000 7f3ba5bae640 1 -- 172.21.15.29:0/626720048 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5613f80d6820 msgr2=0x5613f814bcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:56:29.768 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:29.763+0000 7f3ba5bae640 1 --2- 172.21.15.29:0/626720048 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5613f80d6820 0x5613f814bcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:56:29.768 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:29.763+0000 7f3ba5bae640 1 -- 172.21.15.29:0/626720048 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5613f80cda50 msgr2=0x5613f80d1020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:56:29.768 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:29.763+0000 7f3ba5bae640 1 --2- 172.21.15.29:0/626720048 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5613f80cda50 0x5613f80d1020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:56:29.768 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:29.763+0000 7f3ba5bae640 1 -- 172.21.15.29:0/626720048 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5613f814c1f0 con 0x5613f80cd130 2024-03-16T11:56:29.769 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:29.763+0000 7f3ba5bae640 1 --2- 172.21.15.29:0/626720048 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5613f80cd130 0x5613f80d0ae0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f3b9c009380 tx=0x7f3b9c011d40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=961bbaf894b39ad9 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:56:29.769 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:29.763+0000 7f3b897fa640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-16T11:56:29.769 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:29.763+0000 7f3b897fa640 1 -- 172.21.15.29:0/626720048 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b9c0043e0 con 0x5613f80cd130 2024-03-16T11:56:29.769 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:29.763+0000 7f3b897fa640 1 -- 172.21.15.29:0/626720048 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8512+0+0 (secure 0 0 0) 0x7f3b9c004580 con 0x5613f80cd130 2024-03-16T11:56:29.769 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:29.763+0000 7f3baaa04740 1 -- 172.21.15.29:0/626720048 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3b98003e50 con 0x5613f80cd130 2024-03-16T11:56:29.769 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:29.763+0000 7f3b897fa640 1 -- 172.21.15.29:0/626720048 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f3b9c03c050 con 0x5613f80cd130 2024-03-16T11:56:29.769 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:29.763+0000 7f3baaa04740 10 client.15870 fetch_fsmap learned FSMap version 24 2024-03-16T11:56:29.769 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:29.763+0000 7f3baaa04740 1 -- 172.21.15.29:0/626720048 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x5613f814d4b0 con 0x5613f80cd130 2024-03-16T11:56:29.769 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:29.764+0000 7f3b897fa640 1 -- 172.21.15.29:0/626720048 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3b98003e50 con 0x5613f80cd130 2024-03-16T11:56:29.769 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:29.764+0000 7f3b897fa640 1 -- 172.21.15.29:0/626720048 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7f3b9c003aa0 con 0x5613f80cd130 2024-03-16T11:56:29.769 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:29.764+0000 7f3baaa04740 10 client.15870 fetch_fsmap finished waiting for FSMap version 24 2024-03-16T11:56:29.769 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:29.764+0000 7f3baaa04740 10 client.15870 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T11:56:29.769 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:29.764+0000 7f3baaa04740 20 client.15870 populate_metadata read hostname 'smithi029' 2024-03-16T11:56:29.769 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:29.764+0000 7f3baaa04740 1 --2- 172.21.15.29:0/626720048 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x5613f814d9d0 0x5613f814fdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:56:29.770 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:29.764+0000 7f3baaa04740 4 client.15870 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-16T11:56:29.770 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:29.764+0000 7f3baaa04740 1 -- 172.21.15.29:0/626720048 --> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5613f80392d0 con 0x5613f814d9d0 2024-03-16T11:56:29.770 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:29.764+0000 7f3ba53ad640 1 --2- 172.21.15.29:0/626720048 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x5613f814d9d0 0x5613f814fdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:56:29.770 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:29.764+0000 7f3ba53ad640 1 --2- 172.21.15.29:0/626720048 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x5613f814d9d0 0x5613f814fdc0 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=93245bc32379b1d7 server_cookie=ed6635a76d03fe91 in_seq=0 out_seq=0 2024-03-16T11:56:29.770 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:29.764+0000 7f3b897fa640 10 client.15870 ms_handle_connect on v2:172.21.15.152:6838/1548731833 2024-03-16T11:56:29.786 INFO:tasks.workunit.client.0.smithi029.stdout: 1 557845 11.66 MB/sec execute 134 sec latency 21.701 ms 2024-03-16T11:56:29.951 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:29.951+0000 7f3b897fa640 1 -- 172.21.15.29:0/626720048 <== mds.0 v2:172.21.15.152:6838/1548731833 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5613f80392d0 con 0x5613f814d9d0 2024-03-16T11:56:29.952 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:29.951+0000 7f3b897fa640 10 client.15870 handle_command_reply: tid=0 2024-03-16T11:56:29.952 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:29.951+0000 7f3baaa04740 1 client.15870 shutdown 2024-03-16T11:56:29.952 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:29.951+0000 7f3baaa04740 1 -- 172.21.15.29:0/626720048 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5613f80cd130 msgr2=0x5613f80d0ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:56:29.952 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:29.951+0000 7f3baaa04740 1 --2- 172.21.15.29:0/626720048 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5613f80cd130 0x5613f80d0ae0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f3b9c009380 tx=0x7f3b9c011d40 comp rx=0 tx=0).stop 2024-03-16T11:56:29.952 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:29.951+0000 7f3baaa04740 1 -- 172.21.15.29:0/626720048 shutdown_connections 2024-03-16T11:56:29.952 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:29.951+0000 7f3baaa04740 1 --2- 172.21.15.29:0/626720048 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5613f80d6820 0x5613f814bcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:56:29.952 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:29.951+0000 7f3baaa04740 1 --2- 172.21.15.29:0/626720048 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x5613f814d9d0 0x5613f814fdc0 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:56:29.952 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:29.951+0000 7f3baaa04740 1 --2- 172.21.15.29:0/626720048 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5613f80cda50 0x5613f80d1020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:56:29.952 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:29.951+0000 7f3baaa04740 1 --2- 172.21.15.29:0/626720048 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5613f80cd130 0x5613f80d0ae0 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:56:29.952 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:29.951+0000 7f3b897fa640 0 client.15870 ms_handle_reset on v2:172.21.15.152:6838/1548731833 2024-03-16T11:56:29.952 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:29.951+0000 7f3baaa04740 1 -- 172.21.15.29:0/626720048 >> 172.21.15.29:0/626720048 conn(0x5613f80c7a50 msgr2=0x5613f80c6320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:56:29.952 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:29.951+0000 7f3baaa04740 1 -- 172.21.15.29:0/626720048 shutdown_connections 2024-03-16T11:56:29.953 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:29.951+0000 7f3baaa04740 1 -- 172.21.15.29:0/626720048 wait complete. 2024-03-16T11:56:29.953 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:29.951+0000 7f3baaa04740 20 client.15870 trim_cache size 0 max 16384 2024-03-16T11:56:29.953 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:{ 2024-03-16T11:56:29.953 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "message": "", 2024-03-16T11:56:29.953 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "return_code": 0 2024-03-16T11:56:29.953 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:} 2024-03-16T11:56:29.954 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:29.952+0000 7f3ba7e12640 1 -- 172.21.15.29:0/619632278 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f3b8c0a2bb0 msgr2=0x7f3b8c0a5070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:56:29.954 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:29.952+0000 7f3ba7e12640 1 --2- 172.21.15.29:0/619632278 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f3b8c0a2bb0 0x7f3b8c0a5070 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f3b98013cf0 tx=0x7f3b98009f10 comp rx=0 tx=0).stop 2024-03-16T11:56:29.955 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:29.952+0000 7f3ba7e12640 1 -- 172.21.15.29:0/619632278 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3ba0102150 msgr2=0x7f3ba0074020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:56:29.955 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:29.952+0000 7f3ba7e12640 1 --2- 172.21.15.29:0/619632278 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3ba0102150 0x7f3ba0074020 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f3b90009230 tx=0x7f3b9002cb30 comp rx=0 tx=0).stop 2024-03-16T11:56:29.955 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:29.952+0000 7f3ba7e12640 1 -- 172.21.15.29:0/619632278 shutdown_connections 2024-03-16T11:56:29.955 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:29.952+0000 7f3ba7e12640 1 --2- 172.21.15.29:0/619632278 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f3b8c0a2bb0 0x7f3b8c0a5070 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:56:29.955 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:29.952+0000 7f3ba7e12640 1 --2- 172.21.15.29:0/619632278 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3ba01077a0 0x7f3ba0074560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:56:29.955 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:29.952+0000 7f3ba7e12640 1 --2- 172.21.15.29:0/619632278 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3ba0102150 0x7f3ba0074020 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:56:29.955 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:29.952+0000 7f3ba7e12640 1 --2- 172.21.15.29:0/619632278 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3ba0101790 0x7f3ba0073ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:56:29.955 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:29.952+0000 7f3ba7e12640 1 -- 172.21.15.29:0/619632278 >> 172.21.15.29:0/619632278 conn(0x7f3ba00fd2a0 msgr2=0x7f3ba00fd840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:56:29.955 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:29.952+0000 7f3ba7e12640 1 -- 172.21.15.29:0/619632278 shutdown_connections 2024-03-16T11:56:29.955 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:29.952+0000 7f3ba7e12640 1 -- 172.21.15.29:0/619632278 wait complete. 2024-03-16T11:56:30.786 INFO:tasks.workunit.client.0.smithi029.stdout: 1 559115 11.62 MB/sec execute 135 sec latency 99.506 ms 2024-03-16T11:56:31.090 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:56:30 smithi029 ceph-mon[31975]: pgmap v172: 97 pgs: 97 active+clean; 74 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 4.4 MiB/s wr, 289 op/s 2024-03-16T11:56:31.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:56:30 smithi152 ceph-mon[36291]: pgmap v172: 97 pgs: 97 active+clean; 74 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 4.4 MiB/s wr, 289 op/s 2024-03-16T11:56:31.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:56:30 smithi154 ceph-mon[36320]: pgmap v172: 97 pgs: 97 active+clean; 74 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 4.4 MiB/s wr, 289 op/s 2024-03-16T11:56:31.786 INFO:tasks.workunit.client.0.smithi029.stdout: 1 561311 11.62 MB/sec execute 136 sec latency 22.403 ms 2024-03-16T11:56:32.786 INFO:tasks.workunit.client.0.smithi029.stdout: 1 562322 11.56 MB/sec execute 137 sec latency 113.048 ms 2024-03-16T11:56:33.087 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:56:32 smithi029 ceph-mon[31975]: pgmap v173: 97 pgs: 97 active+clean; 63 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 5.9 MiB/s wr, 488 op/s 2024-03-16T11:56:33.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:56:32 smithi152 ceph-mon[36291]: pgmap v173: 97 pgs: 97 active+clean; 63 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 5.9 MiB/s wr, 488 op/s 2024-03-16T11:56:33.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:56:32 smithi154 ceph-mon[36320]: pgmap v173: 97 pgs: 97 active+clean; 63 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 5.9 MiB/s wr, 488 op/s 2024-03-16T11:56:33.786 INFO:tasks.workunit.client.0.smithi029.stdout: 1 564629 11.57 MB/sec execute 138 sec latency 22.013 ms 2024-03-16T11:56:34.786 INFO:tasks.workunit.client.0.smithi029.stdout: 1 566229 11.56 MB/sec execute 139 sec latency 18.648 ms 2024-03-16T11:56:35.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:56:34 smithi029 ceph-mon[31975]: pgmap v174: 97 pgs: 97 active+clean; 63 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 2.8 MiB/s wr, 293 op/s 2024-03-16T11:56:35.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:56:34 smithi152 ceph-mon[36291]: pgmap v174: 97 pgs: 97 active+clean; 63 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 2.8 MiB/s wr, 293 op/s 2024-03-16T11:56:35.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:56:34 smithi154 ceph-mon[36320]: pgmap v174: 97 pgs: 97 active+clean; 63 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 2.8 MiB/s wr, 293 op/s 2024-03-16T11:56:35.786 INFO:tasks.workunit.client.0.smithi029.stdout: 1 568481 11.57 MB/sec execute 140 sec latency 18.353 ms 2024-03-16T11:56:36.786 INFO:tasks.workunit.client.0.smithi029.stdout: 1 569906 11.53 MB/sec execute 141 sec latency 22.783 ms 2024-03-16T11:56:37.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:56:36 smithi029 ceph-mon[31975]: pgmap v175: 97 pgs: 97 active+clean; 63 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 2.8 MiB/s wr, 293 op/s 2024-03-16T11:56:37.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:56:36 smithi152 ceph-mon[36291]: pgmap v175: 97 pgs: 97 active+clean; 63 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 2.8 MiB/s wr, 293 op/s 2024-03-16T11:56:37.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:56:36 smithi154 ceph-mon[36320]: pgmap v175: 97 pgs: 97 active+clean; 63 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 2.8 MiB/s wr, 293 op/s 2024-03-16T11:56:37.789 INFO:tasks.workunit.client.0.smithi029.stdout: 1 572026 11.54 MB/sec execute 142 sec latency 149.767 ms 2024-03-16T11:56:38.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:56:37 smithi154 ceph-mon[36320]: pgmap v176: 97 pgs: 97 active+clean; 68 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 4.2 MiB/s wr, 381 op/s 2024-03-16T11:56:38.337 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:56:37 smithi029 ceph-mon[31975]: pgmap v176: 97 pgs: 97 active+clean; 68 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 4.2 MiB/s wr, 381 op/s 2024-03-16T11:56:38.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:56:37 smithi152 ceph-mon[36291]: pgmap v176: 97 pgs: 97 active+clean; 68 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 4.2 MiB/s wr, 381 op/s 2024-03-16T11:56:38.787 INFO:tasks.workunit.client.0.smithi029.stdout: 1 573410 11.50 MB/sec execute 143 sec latency 22.708 ms 2024-03-16T11:56:39.787 INFO:tasks.workunit.client.0.smithi029.stdout: 1 575660 11.52 MB/sec execute 144 sec latency 22.213 ms 2024-03-16T11:56:40.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:56:39 smithi154 ceph-mon[36320]: pgmap v177: 97 pgs: 97 active+clean; 68 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 2.9 MiB/s wr, 286 op/s 2024-03-16T11:56:40.337 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:56:39 smithi029 ceph-mon[31975]: pgmap v177: 97 pgs: 97 active+clean; 68 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 2.9 MiB/s wr, 286 op/s 2024-03-16T11:56:40.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:56:39 smithi152 ceph-mon[36291]: pgmap v177: 97 pgs: 97 active+clean; 68 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 2.9 MiB/s wr, 286 op/s 2024-03-16T11:56:40.787 INFO:tasks.workunit.client.0.smithi029.stdout: 1 577102 11.48 MB/sec execute 145 sec latency 19.490 ms 2024-03-16T11:56:41.787 INFO:tasks.workunit.client.0.smithi029.stdout: 1 579306 11.49 MB/sec execute 146 sec latency 23.516 ms 2024-03-16T11:56:42.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:56:41 smithi154 ceph-mon[36320]: pgmap v178: 97 pgs: 97 active+clean; 73 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 4.2 MiB/s wr, 374 op/s 2024-03-16T11:56:42.337 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:56:41 smithi029 ceph-mon[31975]: pgmap v178: 97 pgs: 97 active+clean; 73 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 4.2 MiB/s wr, 374 op/s 2024-03-16T11:56:42.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:56:41 smithi152 ceph-mon[36291]: pgmap v178: 97 pgs: 97 active+clean; 73 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 4.2 MiB/s wr, 374 op/s 2024-03-16T11:56:42.787 INFO:tasks.workunit.client.0.smithi029.stdout: 1 580669 11.45 MB/sec execute 147 sec latency 74.995 ms 2024-03-16T11:56:43.044 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:43.043+0000 7f2e0312f640 1 -- 172.21.15.29:0/2389403974 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2dfc1012f0 msgr2=0x7f2dfc10d510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:56:43.044 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:43.043+0000 7f2e0312f640 1 --2- 172.21.15.29:0/2389403974 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2dfc1012f0 0x7f2dfc10d510 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f2df4009ca0 tx=0x7f2df402c0f0 comp rx=0 tx=0).stop 2024-03-16T11:56:43.044 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:43.043+0000 7f2e0312f640 1 -- 172.21.15.29:0/2389403974 shutdown_connections 2024-03-16T11:56:43.044 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:43.043+0000 7f2e0312f640 1 --2- 172.21.15.29:0/2389403974 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2dfc1012f0 0x7f2dfc10d510 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:56:43.044 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:43.043+0000 7f2e0312f640 1 --2- 172.21.15.29:0/2389403974 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2dfc100930 0x7f2dfc100db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:56:43.044 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:43.043+0000 7f2e0312f640 1 --2- 172.21.15.29:0/2389403974 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2dfc105680 0x7f2dfc105a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:56:43.044 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:43.043+0000 7f2e0312f640 1 -- 172.21.15.29:0/2389403974 >> 172.21.15.29:0/2389403974 conn(0x7f2dfc077530 msgr2=0x7f2dfc077940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:56:43.045 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:43.043+0000 7f2e0312f640 1 -- 172.21.15.29:0/2389403974 shutdown_connections 2024-03-16T11:56:43.045 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:43.043+0000 7f2e0312f640 1 -- 172.21.15.29:0/2389403974 wait complete. 2024-03-16T11:56:43.045 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:43.043+0000 7f2e0312f640 1 Processor -- start 2024-03-16T11:56:43.045 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:43.043+0000 7f2e0312f640 1 -- start start 2024-03-16T11:56:43.045 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:43.043+0000 7f2e0312f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2dfc100930 0x7f2dfc197fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:56:43.045 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:43.043+0000 7f2e0312f640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2dfc1012f0 0x7f2dfc1984f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:56:43.045 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:43.044+0000 7f2e0312f640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2dfc105680 0x7f2dfc19c870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:56:43.045 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:43.044+0000 7f2e0312f640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f2dfc110ba0 con 0x7f2dfc100930 2024-03-16T11:56:43.045 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:43.044+0000 7f2e0312f640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f2dfc110a20 con 0x7f2dfc105680 2024-03-16T11:56:43.045 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:43.044+0000 7f2e0312f640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f2dfc110d20 con 0x7f2dfc1012f0 2024-03-16T11:56:43.045 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:43.044+0000 7f2e00ecb640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2dfc100930 0x7f2dfc197fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:56:43.045 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:43.044+0000 7f2e00ecb640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2dfc100930 0x7f2dfc197fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60120/0 (socket says 172.21.15.29:60120) 2024-03-16T11:56:43.045 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:43.044+0000 7f2e00ecb640 1 -- 172.21.15.29:0/155283634 learned_addr learned my addr 172.21.15.29:0/155283634 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:56:43.046 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:43.044+0000 7f2dfbfff640 1 --2- 172.21.15.29:0/155283634 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2dfc1012f0 0x7f2dfc1984f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:56:43.046 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:43.044+0000 7f2e016cc640 1 --2- 172.21.15.29:0/155283634 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2dfc105680 0x7f2dfc19c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:56:43.046 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:43.044+0000 7f2e00ecb640 1 -- 172.21.15.29:0/155283634 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2dfc1012f0 msgr2=0x7f2dfc1984f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:56:43.046 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:43.044+0000 7f2e00ecb640 1 --2- 172.21.15.29:0/155283634 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2dfc1012f0 0x7f2dfc1984f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:56:43.046 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:43.044+0000 7f2e00ecb640 1 -- 172.21.15.29:0/155283634 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2dfc105680 msgr2=0x7f2dfc19c870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:56:43.046 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:43.044+0000 7f2e00ecb640 1 --2- 172.21.15.29:0/155283634 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2dfc105680 0x7f2dfc19c870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:56:43.046 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:43.044+0000 7f2e00ecb640 1 -- 172.21.15.29:0/155283634 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2dfc19cfa0 con 0x7f2dfc100930 2024-03-16T11:56:43.046 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:43.044+0000 7f2dfbfff640 1 --2- 172.21.15.29:0/155283634 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2dfc1012f0 0x7f2dfc1984f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:56:43.046 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:43.044+0000 7f2e00ecb640 1 --2- 172.21.15.29:0/155283634 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2dfc100930 0x7f2dfc197fb0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f2de400e9f0 tx=0x7f2de400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:56:43.046 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:43.044+0000 7f2df9ffb640 1 -- 172.21.15.29:0/155283634 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2de400cd80 con 0x7f2dfc100930 2024-03-16T11:56:43.046 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:43.044+0000 7f2e0312f640 1 -- 172.21.15.29:0/155283634 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2dfc19d290 con 0x7f2dfc100930 2024-03-16T11:56:43.046 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:43.044+0000 7f2df9ffb640 1 -- 172.21.15.29:0/155283634 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2de4010910 con 0x7f2dfc100930 2024-03-16T11:56:43.046 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:43.045+0000 7f2df9ffb640 1 -- 172.21.15.29:0/155283634 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2de40188f0 con 0x7f2dfc100930 2024-03-16T11:56:43.046 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:43.045+0000 7f2e0312f640 1 -- 172.21.15.29:0/155283634 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2dfc1a4b20 con 0x7f2dfc100930 2024-03-16T11:56:43.053 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:43.047+0000 7f2df9ffb640 1 -- 172.21.15.29:0/155283634 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7f2de4010ac0 con 0x7f2dfc100930 2024-03-16T11:56:43.053 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:43.047+0000 7f2e05df0740 1 Processor -- start 2024-03-16T11:56:43.053 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:43.047+0000 7f2e05df0740 1 -- start start 2024-03-16T11:56:43.053 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:43.048+0000 7f2e05df0740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56021ad86130 0x56021ad86510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:56:43.053 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:43.048+0000 7f2df9ffb640 1 --2- 172.21.15.29:0/155283634 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f2de80a2a10 0x7f2de80a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:56:43.053 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:43.048+0000 7f2e05df0740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56021ad86a50 0x56021ad8f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:56:43.054 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:43.048+0000 7f2e05df0740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56021ad8f820 0x56021ad91c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:56:43.054 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:43.048+0000 7f2e05df0740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x56021aceaa40 con 0x56021ad86130 2024-03-16T11:56:43.054 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:43.048+0000 7f2e05df0740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x56021ac93920 con 0x56021ad8f820 2024-03-16T11:56:43.054 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:43.048+0000 7f2e05df0740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x56021ace6ae0 con 0x56021ad86a50 2024-03-16T11:56:43.054 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:43.048+0000 7f2e016cc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56021ad86130 0x56021ad86510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:56:43.054 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:43.048+0000 7f2e00ecb640 1 --2- 172.21.15.29:0/155283634 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f2de80a2a10 0x7f2de80a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:56:43.054 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:43.048+0000 7f2dfbfff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56021ad86a50 0x56021ad8f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:56:43.054 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:43.048+0000 7f2e016cc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56021ad86130 0x56021ad86510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60134/0 (socket says 172.21.15.29:60134) 2024-03-16T11:56:43.054 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:43.048+0000 7f2e016cc640 1 -- 172.21.15.29:0/4221282248 learned_addr learned my addr 172.21.15.29:0/4221282248 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:56:43.054 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:43.048+0000 7f2e016cc640 1 --2- 172.21.15.29:0/4221282248 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56021ad8f820 0x56021ad91c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:56:43.054 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:43.048+0000 7f2df9ffb640 1 -- 172.21.15.29:0/155283634 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8512+0+0 (secure 0 0 0) 0x7f2de40cf240 con 0x7f2dfc100930 2024-03-16T11:56:43.054 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:43.048+0000 7f2dfbfff640 1 -- 172.21.15.29:0/4221282248 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56021ad8f820 msgr2=0x56021ad91c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:56:43.054 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:43.048+0000 7f2dfbfff640 1 --2- 172.21.15.29:0/4221282248 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56021ad8f820 0x56021ad91c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:56:43.054 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:43.048+0000 7f2dfbfff640 1 -- 172.21.15.29:0/4221282248 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56021ad86130 msgr2=0x56021ad86510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:56:43.054 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:43.048+0000 7f2dfbfff640 1 --2- 172.21.15.29:0/4221282248 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56021ad86130 0x56021ad86510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:56:43.055 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:43.048+0000 7f2dfbfff640 1 -- 172.21.15.29:0/4221282248 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56021acab460 con 0x56021ad86a50 2024-03-16T11:56:43.055 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:43.048+0000 7f2e00ecb640 1 --2- 172.21.15.29:0/155283634 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f2de80a2a10 0x7f2de80a4ed0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f2de409a7e0 tx=0x7f2de4097bc0 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:56:43.055 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:43.048+0000 7f2dfbfff640 1 --2- 172.21.15.29:0/4221282248 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56021ad86a50 0x56021ad8f2e0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f2dec009fa0 tx=0x7f2dec0118d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=272d3b4ff86dc09c server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:56:43.055 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:43.048+0000 7f2df2ffd640 1 -- 172.21.15.29:0/4221282248 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2dec004030 con 0x56021ad86a50 2024-03-16T11:56:43.055 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:43.049+0000 7f2df2ffd640 1 -- 172.21.15.29:0/4221282248 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2dec005500 con 0x56021ad86a50 2024-03-16T11:56:43.055 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:43.049+0000 7f2df2ffd640 1 -- 172.21.15.29:0/4221282248 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2dec004d50 con 0x56021ad86a50 2024-03-16T11:56:43.055 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:43.049+0000 7f2e05df0740 1 -- 172.21.15.29:0/4221282248 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56021ad86a50 msgr2=0x56021ad8f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:56:43.055 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:43.049+0000 7f2e05df0740 1 --2- 172.21.15.29:0/4221282248 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56021ad86a50 0x56021ad8f2e0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f2dec009fa0 tx=0x7f2dec0118d0 comp rx=0 tx=0).stop 2024-03-16T11:56:43.055 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:43.049+0000 7f2e05df0740 1 -- 172.21.15.29:0/4221282248 shutdown_connections 2024-03-16T11:56:43.055 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:43.049+0000 7f2e05df0740 1 --2- 172.21.15.29:0/4221282248 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56021ad8f820 0x56021ad91c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:56:43.055 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:43.049+0000 7f2e05df0740 1 --2- 172.21.15.29:0/4221282248 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56021ad86a50 0x56021ad8f2e0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:56:43.055 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:43.049+0000 7f2e05df0740 1 --2- 172.21.15.29:0/4221282248 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56021ad86130 0x56021ad86510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:56:43.055 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:43.049+0000 7f2e05df0740 1 -- 172.21.15.29:0/4221282248 >> 172.21.15.29:0/4221282248 conn(0x56021ad80a50 msgr2=0x56021ad80e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:56:43.055 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:43.049+0000 7f2e05df0740 1 -- 172.21.15.29:0/4221282248 shutdown_connections 2024-03-16T11:56:43.055 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:43.049+0000 7f2e05df0740 1 -- 172.21.15.29:0/4221282248 wait complete. 2024-03-16T11:56:43.056 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:43.050+0000 7f2e05df0740 1 -- start start 2024-03-16T11:56:43.056 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:43.050+0000 7f2e05df0740 1 Processor -- start 2024-03-16T11:56:43.056 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:43.050+0000 7f2e05df0740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56021ad86130 0x56021ad8c890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:56:43.056 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:43.050+0000 7f2e05df0740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56021ad86a50 0x56021ad8cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:56:43.056 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:43.050+0000 7f2e05df0740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56021ad8f820 0x56021ad8d310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:56:43.056 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:43.050+0000 7f2e05df0740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x56021ad92450 con 0x56021ad86a50 2024-03-16T11:56:43.056 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:43.050+0000 7f2e05df0740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x56021aca6d70 con 0x56021ad8f820 2024-03-16T11:56:43.056 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:43.050+0000 7f2e05df0740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x56021ad925d0 con 0x56021ad86130 2024-03-16T11:56:43.056 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:43.050+0000 7f2e05df0740 10 client.0 _ll_register_callbacks cb 0x56021a9fea00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T11:56:43.056 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:43.050+0000 7f2dfbfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56021ad86a50 0x56021ad8cdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:56:43.056 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:43.050+0000 7f2dfbfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56021ad86a50 0x56021ad8cdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60148/0 (socket says 172.21.15.29:60148) 2024-03-16T11:56:43.056 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:43.050+0000 7f2dfbfff640 1 -- 172.21.15.29:0/2118554898 learned_addr learned my addr 172.21.15.29:0/2118554898 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:56:43.056 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:43.050+0000 7f2e016cc640 1 --2- 172.21.15.29:0/2118554898 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56021ad86130 0x56021ad8c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:56:43.056 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:43.050+0000 7f2e016cc640 1 --2- 172.21.15.29:0/2118554898 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56021ad8f820 0x56021ad8d310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:56:43.057 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:43.050+0000 7f2dfbfff640 1 -- 172.21.15.29:0/2118554898 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56021ad86130 msgr2=0x56021ad8c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:56:43.057 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:43.050+0000 7f2dfbfff640 1 --2- 172.21.15.29:0/2118554898 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56021ad86130 0x56021ad8c890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:56:43.057 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:43.050+0000 7f2dfbfff640 1 -- 172.21.15.29:0/2118554898 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56021ad8f820 msgr2=0x56021ad8d310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:56:43.057 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:43.050+0000 7f2dfbfff640 1 --2- 172.21.15.29:0/2118554898 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56021ad8f820 0x56021ad8d310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:56:43.057 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:43.050+0000 7f2dfbfff640 1 -- 172.21.15.29:0/2118554898 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56021a9ff210 con 0x56021ad86a50 2024-03-16T11:56:43.057 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:43.050+0000 7f2e016cc640 1 --2- 172.21.15.29:0/2118554898 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56021ad86130 0x56021ad8c890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:56:43.057 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:43.051+0000 7f2dfbfff640 1 --2- 172.21.15.29:0/2118554898 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56021ad86a50 0x56021ad8cdd0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f2dec00a010 tx=0x7f2dec004e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8534f4fc9ae5bec server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:56:43.057 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:43.051+0000 7f2ddbfff640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-16T11:56:43.057 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:43.051+0000 7f2ddbfff640 1 -- 172.21.15.29:0/2118554898 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2dec027070 con 0x56021ad86a50 2024-03-16T11:56:43.057 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:43.051+0000 7f2ddbfff640 1 -- 172.21.15.29:0/2118554898 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8512+0+0 (secure 0 0 0) 0x7f2dec005260 con 0x56021ad86a50 2024-03-16T11:56:43.057 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:43.051+0000 7f2e05df0740 1 -- 172.21.15.29:0/2118554898 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2dec005500 con 0x56021ad86a50 2024-03-16T11:56:43.057 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:43.051+0000 7f2ddbfff640 1 -- 172.21.15.29:0/2118554898 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f2dec033050 con 0x56021ad86a50 2024-03-16T11:56:43.057 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:43.051+0000 7f2e05df0740 10 client.15882 fetch_fsmap learned FSMap version 24 2024-03-16T11:56:43.057 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:43.051+0000 7f2e05df0740 1 -- 172.21.15.29:0/2118554898 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x56021aacb620 con 0x56021ad86a50 2024-03-16T11:56:43.058 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:43.051+0000 7f2ddbfff640 1 -- 172.21.15.29:0/2118554898 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2dec005500 con 0x56021ad86a50 2024-03-16T11:56:43.058 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:43.051+0000 7f2ddbfff640 1 -- 172.21.15.29:0/2118554898 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7f2dec022460 con 0x56021ad86a50 2024-03-16T11:56:43.058 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:43.051+0000 7f2e05df0740 10 client.15882 fetch_fsmap finished waiting for FSMap version 24 2024-03-16T11:56:43.058 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:43.051+0000 7f2e05df0740 10 client.15882 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T11:56:43.058 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:43.051+0000 7f2e05df0740 20 client.15882 populate_metadata read hostname 'smithi029' 2024-03-16T11:56:43.058 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:43.051+0000 7f2e05df0740 1 --2- 172.21.15.29:0/2118554898 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x56021ad9f9a0 0x56021ada1d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:56:43.058 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:43.051+0000 7f2e05df0740 4 client.15882 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T11:56:43.058 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:43.051+0000 7f2e05df0740 1 -- 172.21.15.29:0/2118554898 --> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56021acf7f80 con 0x56021ad9f9a0 2024-03-16T11:56:43.058 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:43.052+0000 7f2e016cc640 1 --2- 172.21.15.29:0/2118554898 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x56021ad9f9a0 0x56021ada1d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:56:43.058 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:43.052+0000 7f2e016cc640 1 --2- 172.21.15.29:0/2118554898 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x56021ad9f9a0 0x56021ada1d90 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=db2009913c2ba9d1 server_cookie=56dd6ed5ebbaeb8f in_seq=0 out_seq=0 2024-03-16T11:56:43.058 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:43.052+0000 7f2ddbfff640 10 client.15882 ms_handle_connect on v2:172.21.15.152:6838/1548731833 2024-03-16T11:56:43.058 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:43.053+0000 7f2ddbfff640 1 -- 172.21.15.29:0/2118554898 <== mds.0 v2:172.21.15.152:6838/1548731833 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x56021acf7f80 con 0x56021ad9f9a0 2024-03-16T11:56:43.058 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:43.053+0000 7f2ddbfff640 10 client.15882 handle_command_reply: tid=0 2024-03-16T11:56:43.058 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:43.053+0000 7f2e05df0740 1 client.15882 shutdown 2024-03-16T11:56:43.059 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:43.053+0000 7f2e05df0740 1 -- 172.21.15.29:0/2118554898 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56021ad86a50 msgr2=0x56021ad8cdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:56:43.059 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:43.053+0000 7f2e05df0740 1 --2- 172.21.15.29:0/2118554898 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56021ad86a50 0x56021ad8cdd0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f2dec00a010 tx=0x7f2dec004e60 comp rx=0 tx=0).stop 2024-03-16T11:56:43.059 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:43.053+0000 7f2e05df0740 1 -- 172.21.15.29:0/2118554898 shutdown_connections 2024-03-16T11:56:43.059 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:43.053+0000 7f2e05df0740 1 --2- 172.21.15.29:0/2118554898 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x56021ad9f9a0 0x56021ada1d90 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:56:43.059 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:43.053+0000 7f2e05df0740 1 --2- 172.21.15.29:0/2118554898 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56021ad8f820 0x56021ad8d310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:56:43.059 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:43.053+0000 7f2e05df0740 1 --2- 172.21.15.29:0/2118554898 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56021ad86a50 0x56021ad8cdd0 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:56:43.059 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:43.053+0000 7f2e05df0740 1 --2- 172.21.15.29:0/2118554898 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56021ad86130 0x56021ad8c890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:56:43.059 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:43.053+0000 7f2e05df0740 1 -- 172.21.15.29:0/2118554898 >> 172.21.15.29:0/2118554898 conn(0x56021ad80a50 msgr2=0x56021ad827a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:56:43.059 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:43.053+0000 7f2ddbfff640 0 client.15882 ms_handle_reset on v2:172.21.15.152:6838/1548731833 2024-03-16T11:56:43.059 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:43.053+0000 7f2e05df0740 1 -- 172.21.15.29:0/2118554898 shutdown_connections 2024-03-16T11:56:43.059 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:43.053+0000 7f2e05df0740 1 -- 172.21.15.29:0/2118554898 wait complete. 2024-03-16T11:56:43.059 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:43.054+0000 7f2e05df0740 20 client.15882 trim_cache size 0 max 16384 2024-03-16T11:56:43.069 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:43.069+0000 7f2e05df0740 1 Processor -- start 2024-03-16T11:56:43.069 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:43.069+0000 7f2e05df0740 1 -- start start 2024-03-16T11:56:43.069 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:43.069+0000 7f2e05df0740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56021ad86130 0x56021ad8efc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:56:43.070 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:43.069+0000 7f2e05df0740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56021ad86a50 0x56021ad9b1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:56:43.070 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:43.069+0000 7f2e05df0740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56021ad8f820 0x56021ada7520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:56:43.070 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:43.069+0000 7f2e05df0740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f2dfc1afb80 con 0x56021ad86a50 2024-03-16T11:56:43.070 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:43.069+0000 7f2e05df0740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f2dfc1ad450 con 0x56021ad86130 2024-03-16T11:56:43.070 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:43.069+0000 7f2e05df0740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f2dfc1ab280 con 0x56021ad8f820 2024-03-16T11:56:43.070 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:43.069+0000 7f2dfbfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56021ad86a50 0x56021ad9b1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:56:43.070 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:43.070+0000 7f2e016cc640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56021ad86130 0x56021ad8efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:56:43.070 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:43.070+0000 7f2dfbfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56021ad86a50 0x56021ad9b1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60154/0 (socket says 172.21.15.29:60154) 2024-03-16T11:56:43.070 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:43.070+0000 7f2dfbfff640 1 -- 172.21.15.29:0/3822403020 learned_addr learned my addr 172.21.15.29:0/3822403020 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:56:43.070 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:43.070+0000 7f2e016cc640 1 --2- 172.21.15.29:0/3822403020 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56021ad8f820 0x56021ada7520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:56:43.070 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:43.070+0000 7f2dfbfff640 1 -- 172.21.15.29:0/3822403020 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56021ad8f820 msgr2=0x56021ada7520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:56:43.070 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:43.070+0000 7f2dfbfff640 1 --2- 172.21.15.29:0/3822403020 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56021ad8f820 0x56021ada7520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:56:43.070 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:43.070+0000 7f2dfbfff640 1 -- 172.21.15.29:0/3822403020 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56021ad86130 msgr2=0x56021ad8efc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:56:43.071 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:43.070+0000 7f2dfbfff640 1 --2- 172.21.15.29:0/3822403020 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56021ad86130 0x56021ad8efc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:56:43.071 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:43.070+0000 7f2dfbfff640 1 -- 172.21.15.29:0/3822403020 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56021ad9b730 con 0x56021ad86a50 2024-03-16T11:56:43.071 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:43.070+0000 7f2dfbfff640 1 --2- 172.21.15.29:0/3822403020 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56021ad86a50 0x56021ad9b1f0 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f2dec004940 tx=0x7f2dec004de0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8106c2edc20cdf server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:56:43.071 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:43.070+0000 7f2df27fc640 1 -- 172.21.15.29:0/3822403020 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2dec027040 con 0x56021ad86a50 2024-03-16T11:56:43.071 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:43.070+0000 7f2df27fc640 1 -- 172.21.15.29:0/3822403020 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2dec009b50 con 0x56021ad86a50 2024-03-16T11:56:43.071 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:43.070+0000 7f2df27fc640 1 -- 172.21.15.29:0/3822403020 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2dec01d040 con 0x56021ad86a50 2024-03-16T11:56:43.074 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:43.071+0000 7f2e05df0740 1 -- 172.21.15.29:0/3822403020 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56021ad86a50 msgr2=0x56021ad9b1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:56:43.075 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:43.071+0000 7f2e05df0740 1 --2- 172.21.15.29:0/3822403020 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56021ad86a50 0x56021ad9b1f0 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f2dec004940 tx=0x7f2dec004de0 comp rx=0 tx=0).stop 2024-03-16T11:56:43.075 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:43.071+0000 7f2e05df0740 1 -- 172.21.15.29:0/3822403020 shutdown_connections 2024-03-16T11:56:43.075 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:43.071+0000 7f2e05df0740 1 --2- 172.21.15.29:0/3822403020 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56021ad8f820 0x56021ada7520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:56:43.075 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:43.071+0000 7f2e05df0740 1 --2- 172.21.15.29:0/3822403020 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56021ad86a50 0x56021ad9b1f0 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:56:43.075 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:43.071+0000 7f2e05df0740 1 --2- 172.21.15.29:0/3822403020 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56021ad86130 0x56021ad8efc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:56:43.075 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:43.071+0000 7f2e05df0740 1 -- 172.21.15.29:0/3822403020 >> 172.21.15.29:0/3822403020 conn(0x56021ad80a50 msgr2=0x56021ada19b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:56:43.075 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:43.071+0000 7f2e05df0740 1 -- 172.21.15.29:0/3822403020 shutdown_connections 2024-03-16T11:56:43.075 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:43.071+0000 7f2e05df0740 1 -- 172.21.15.29:0/3822403020 wait complete. 2024-03-16T11:56:43.075 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:43.071+0000 7f2e05df0740 1 -- start start 2024-03-16T11:56:43.075 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:43.071+0000 7f2e05df0740 1 Processor -- start 2024-03-16T11:56:43.075 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:43.072+0000 7f2e05df0740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56021ad86130 0x56021ad92a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:56:43.075 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:43.072+0000 7f2e05df0740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56021ad86a50 0x56021ad92f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:56:43.075 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:43.072+0000 7f2e05df0740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56021ad8f820 0x56021adfdbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:56:43.075 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:43.072+0000 7f2e05df0740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f2dfc1ab910 con 0x56021ad86130 2024-03-16T11:56:43.075 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:43.072+0000 7f2e05df0740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f2dfc1acd80 con 0x56021ad86a50 2024-03-16T11:56:43.076 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:43.072+0000 7f2e05df0740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f2dfc1abfe0 con 0x56021ad8f820 2024-03-16T11:56:43.076 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:43.072+0000 7f2e05df0740 10 client.0 _ll_register_callbacks cb 0x7f2df4037070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T11:56:43.076 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:43.072+0000 7f2e016cc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56021ad86130 0x56021ad92a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:56:43.076 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:43.072+0000 7f2e016cc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56021ad86130 0x56021ad92a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60164/0 (socket says 172.21.15.29:60164) 2024-03-16T11:56:43.076 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:43.072+0000 7f2e016cc640 1 -- 172.21.15.29:0/471484481 learned_addr learned my addr 172.21.15.29:0/471484481 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:56:43.076 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:43.072+0000 7f2dfbfff640 1 --2- 172.21.15.29:0/471484481 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56021ad86a50 0x56021ad92f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:56:43.076 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:43.072+0000 7f2e016cc640 1 --2- 172.21.15.29:0/471484481 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56021ad8f820 0x56021adfdbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:56:43.076 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:43.072+0000 7f2e016cc640 1 -- 172.21.15.29:0/471484481 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56021ad8f820 msgr2=0x56021adfdbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:56:43.076 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:43.072+0000 7f2e016cc640 1 --2- 172.21.15.29:0/471484481 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56021ad8f820 0x56021adfdbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:56:43.076 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:43.072+0000 7f2e016cc640 1 -- 172.21.15.29:0/471484481 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56021ad86a50 msgr2=0x56021ad92f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:56:43.076 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:43.072+0000 7f2e016cc640 1 --2- 172.21.15.29:0/471484481 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56021ad86a50 0x56021ad92f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:56:43.076 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:43.072+0000 7f2e016cc640 1 -- 172.21.15.29:0/471484481 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56021adfe230 con 0x56021ad86130 2024-03-16T11:56:43.076 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:43.072+0000 7f2e016cc640 1 --2- 172.21.15.29:0/471484481 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56021ad86130 0x56021ad92a10 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f2df4006010 tx=0x7f2df40095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6799a8399381bfa2 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:56:43.076 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:43.072+0000 7f2ddbfff640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-16T11:56:43.076 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:43.072+0000 7f2ddbfff640 1 -- 172.21.15.29:0/471484481 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2df4056070 con 0x56021ad86130 2024-03-16T11:56:43.077 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:43.072+0000 7f2ddbfff640 1 -- 172.21.15.29:0/471484481 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8512+0+0 (secure 0 0 0) 0x7f2df404fb30 con 0x56021ad86130 2024-03-16T11:56:43.077 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:43.072+0000 7f2e05df0740 1 -- 172.21.15.29:0/471484481 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2dec009b50 con 0x56021ad86130 2024-03-16T11:56:43.077 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:43.073+0000 7f2ddbfff640 1 -- 172.21.15.29:0/471484481 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f2df405f050 con 0x56021ad86130 2024-03-16T11:56:43.077 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:43.073+0000 7f2e05df0740 10 client.15888 fetch_fsmap learned FSMap version 24 2024-03-16T11:56:43.077 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:43.073+0000 7f2e05df0740 1 -- 172.21.15.29:0/471484481 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x56021ae05610 con 0x56021ad86130 2024-03-16T11:56:43.077 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:43.073+0000 7f2ddbfff640 1 -- 172.21.15.29:0/471484481 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2dec009b50 con 0x56021ad86130 2024-03-16T11:56:43.077 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:43.073+0000 7f2ddbfff640 1 -- 172.21.15.29:0/471484481 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7f2df4059020 con 0x56021ad86130 2024-03-16T11:56:43.077 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:43.073+0000 7f2e05df0740 10 client.15888 fetch_fsmap finished waiting for FSMap version 24 2024-03-16T11:56:43.077 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:43.073+0000 7f2e05df0740 10 client.15888 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T11:56:43.077 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:43.073+0000 7f2e05df0740 20 client.15888 populate_metadata read hostname 'smithi029' 2024-03-16T11:56:43.077 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:43.073+0000 7f2e05df0740 1 --2- 172.21.15.29:0/471484481 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x56021ae05d90 0x56021ae081b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:56:43.077 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:43.073+0000 7f2e05df0740 4 client.15888 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-16T11:56:43.077 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:43.073+0000 7f2e05df0740 1 -- 172.21.15.29:0/471484481 --> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] -- command(tid 0: {"prefix": "flush journal"}) -- 0x56021acf22d0 con 0x56021ae05d90 2024-03-16T11:56:43.077 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:43.073+0000 7f2dfbfff640 1 --2- 172.21.15.29:0/471484481 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x56021ae05d90 0x56021ae081b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:56:43.077 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:43.074+0000 7f2ddbfff640 10 client.15888 ms_handle_connect on v2:172.21.15.152:6838/1548731833 2024-03-16T11:56:43.077 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:43.074+0000 7f2dfbfff640 1 --2- 172.21.15.29:0/471484481 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x56021ae05d90 0x56021ae081b0 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7a19b65473bdb76a server_cookie=a636ce16974fcdb0 in_seq=0 out_seq=0 2024-03-16T11:56:43.262 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:43.262+0000 7f2ddbfff640 1 -- 172.21.15.29:0/471484481 <== mds.0 v2:172.21.15.152:6838/1548731833 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x56021acf22d0 con 0x56021ae05d90 2024-03-16T11:56:43.262 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:43.262+0000 7f2ddbfff640 10 client.15888 handle_command_reply: tid=0 2024-03-16T11:56:43.262 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:43.262+0000 7f2e05df0740 1 client.15888 shutdown 2024-03-16T11:56:43.262 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:43.262+0000 7f2e05df0740 1 -- 172.21.15.29:0/471484481 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56021ad86130 msgr2=0x56021ad92a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:56:43.262 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:43.262+0000 7f2e05df0740 1 --2- 172.21.15.29:0/471484481 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56021ad86130 0x56021ad92a10 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f2df4006010 tx=0x7f2df40095c0 comp rx=0 tx=0).stop 2024-03-16T11:56:43.263 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:43.262+0000 7f2e05df0740 1 -- 172.21.15.29:0/471484481 shutdown_connections 2024-03-16T11:56:43.263 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:43.262+0000 7f2e05df0740 1 --2- 172.21.15.29:0/471484481 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56021ad8f820 0x56021adfdbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:56:43.263 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:43.262+0000 7f2e05df0740 1 --2- 172.21.15.29:0/471484481 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x56021ae05d90 0x56021ae081b0 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:56:43.263 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:43.262+0000 7f2e05df0740 1 --2- 172.21.15.29:0/471484481 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56021ad86a50 0x56021ad92f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:56:43.263 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:43.263+0000 7f2e05df0740 1 --2- 172.21.15.29:0/471484481 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56021ad86130 0x56021ad92a10 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:56:43.263 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:43.263+0000 7f2ddbfff640 0 client.15888 ms_handle_reset on v2:172.21.15.152:6838/1548731833 2024-03-16T11:56:43.263 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:43.263+0000 7f2e05df0740 1 -- 172.21.15.29:0/471484481 >> 172.21.15.29:0/471484481 conn(0x56021ad80a50 msgr2=0x56021ad8c830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:56:43.263 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:43.263+0000 7f2e05df0740 1 -- 172.21.15.29:0/471484481 shutdown_connections 2024-03-16T11:56:43.263 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:43.263+0000 7f2e05df0740 1 -- 172.21.15.29:0/471484481 wait complete. 2024-03-16T11:56:43.263 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:43.263+0000 7f2e05df0740 20 client.15888 trim_cache size 0 max 16384 2024-03-16T11:56:43.264 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:{ 2024-03-16T11:56:43.264 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "message": "", 2024-03-16T11:56:43.264 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "return_code": 0 2024-03-16T11:56:43.264 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:} 2024-03-16T11:56:43.265 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:43.264+0000 7f2e0312f640 1 -- 172.21.15.29:0/155283634 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f2de80a2a10 msgr2=0x7f2de80a4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:56:43.265 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:43.264+0000 7f2e0312f640 1 --2- 172.21.15.29:0/155283634 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f2de80a2a10 0x7f2de80a4ed0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f2de409a7e0 tx=0x7f2de4097bc0 comp rx=0 tx=0).stop 2024-03-16T11:56:43.265 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:43.264+0000 7f2e0312f640 1 -- 172.21.15.29:0/155283634 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2dfc100930 msgr2=0x7f2dfc197fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:56:43.265 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:43.264+0000 7f2e0312f640 1 --2- 172.21.15.29:0/155283634 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2dfc100930 0x7f2dfc197fb0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f2de400e9f0 tx=0x7f2de400eeb0 comp rx=0 tx=0).stop 2024-03-16T11:56:43.265 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:43.264+0000 7f2e0312f640 1 -- 172.21.15.29:0/155283634 shutdown_connections 2024-03-16T11:56:43.265 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:43.264+0000 7f2e0312f640 1 --2- 172.21.15.29:0/155283634 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f2de80a2a10 0x7f2de80a4ed0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:56:43.265 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:43.264+0000 7f2e0312f640 1 --2- 172.21.15.29:0/155283634 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2dfc105680 0x7f2dfc19c870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:56:43.265 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:43.264+0000 7f2e0312f640 1 --2- 172.21.15.29:0/155283634 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2dfc1012f0 0x7f2dfc1984f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:56:43.265 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:43.264+0000 7f2e0312f640 1 --2- 172.21.15.29:0/155283634 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2dfc100930 0x7f2dfc197fb0 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:56:43.265 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:43.264+0000 7f2e0312f640 1 -- 172.21.15.29:0/155283634 >> 172.21.15.29:0/155283634 conn(0x7f2dfc077530 msgr2=0x7f2dfc0696f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:56:43.265 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:43.264+0000 7f2e0312f640 1 -- 172.21.15.29:0/155283634 shutdown_connections 2024-03-16T11:56:43.265 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:43.264+0000 7f2e0312f640 1 -- 172.21.15.29:0/155283634 wait complete. 2024-03-16T11:56:43.787 INFO:tasks.workunit.client.0.smithi029.stdout: 1 582757 11.46 MB/sec execute 148 sec latency 77.941 ms 2024-03-16T11:56:44.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:56:44 smithi154 ceph-mon[36320]: pgmap v179: 97 pgs: 97 active+clean; 73 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 2.8 MiB/s wr, 176 op/s 2024-03-16T11:56:44.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:56:44 smithi029 ceph-mon[31975]: pgmap v179: 97 pgs: 97 active+clean; 73 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 2.8 MiB/s wr, 176 op/s 2024-03-16T11:56:44.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:56:44 smithi152 ceph-mon[36291]: pgmap v179: 97 pgs: 97 active+clean; 73 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 2.8 MiB/s wr, 176 op/s 2024-03-16T11:56:44.787 INFO:tasks.workunit.client.0.smithi029.stdout: 1 584234 11.42 MB/sec execute 149 sec latency 40.670 ms 2024-03-16T11:56:45.787 INFO:tasks.workunit.client.0.smithi029.stdout: 1 586498 11.44 MB/sec execute 150 sec latency 16.715 ms 2024-03-16T11:56:46.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:56:46 smithi154 ceph-mon[36320]: pgmap v180: 97 pgs: 97 active+clean; 73 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 2.8 MiB/s wr, 176 op/s 2024-03-16T11:56:46.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:56:46 smithi029 ceph-mon[31975]: pgmap v180: 97 pgs: 97 active+clean; 73 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 2.8 MiB/s wr, 176 op/s 2024-03-16T11:56:46.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:56:46 smithi152 ceph-mon[36291]: pgmap v180: 97 pgs: 97 active+clean; 73 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 2.8 MiB/s wr, 176 op/s 2024-03-16T11:56:46.787 INFO:tasks.workunit.client.0.smithi029.stdout: 1 588275 11.40 MB/sec execute 151 sec latency 27.200 ms 2024-03-16T11:56:47.787 INFO:tasks.workunit.client.0.smithi029.stdout: 1 590043 11.42 MB/sec execute 152 sec latency 57.327 ms 2024-03-16T11:56:48.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:56:48 smithi154 ceph-mon[36320]: pgmap v181: 97 pgs: 97 active+clean; 66 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 4.2 MiB/s wr, 364 op/s 2024-03-16T11:56:48.337 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:56:48 smithi029 ceph-mon[31975]: pgmap v181: 97 pgs: 97 active+clean; 66 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 4.2 MiB/s wr, 364 op/s 2024-03-16T11:56:48.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:56:48 smithi152 ceph-mon[36291]: pgmap v181: 97 pgs: 97 active+clean; 66 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 4.2 MiB/s wr, 364 op/s 2024-03-16T11:56:48.787 INFO:tasks.workunit.client.0.smithi029.stdout: 1 591179 11.38 MB/sec execute 153 sec latency 23.895 ms 2024-03-16T11:56:49.787 INFO:tasks.workunit.client.0.smithi029.stdout: 1 593418 11.39 MB/sec execute 154 sec latency 18.331 ms 2024-03-16T11:56:50.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:56:50 smithi154 ceph-mon[36320]: pgmap v182: 97 pgs: 97 active+clean; 66 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.8 MiB/s wr, 277 op/s 2024-03-16T11:56:50.337 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:56:50 smithi029 ceph-mon[31975]: pgmap v182: 97 pgs: 97 active+clean; 66 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.8 MiB/s wr, 277 op/s 2024-03-16T11:56:50.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:56:50 smithi152 ceph-mon[36291]: pgmap v182: 97 pgs: 97 active+clean; 66 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.8 MiB/s wr, 277 op/s 2024-03-16T11:56:50.787 INFO:tasks.workunit.client.0.smithi029.stdout: 1 594868 11.35 MB/sec execute 155 sec latency 19.335 ms 2024-03-16T11:56:51.787 INFO:tasks.workunit.client.0.smithi029.stdout: 1 597159 11.37 MB/sec execute 156 sec latency 18.792 ms 2024-03-16T11:56:52.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:56:52 smithi154 ceph-mon[36320]: pgmap v183: 97 pgs: 97 active+clean; 72 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 4.2 MiB/s wr, 367 op/s 2024-03-16T11:56:52.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:56:52 smithi029 ceph-mon[31975]: pgmap v183: 97 pgs: 97 active+clean; 72 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 4.2 MiB/s wr, 367 op/s 2024-03-16T11:56:52.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:56:52 smithi152 ceph-mon[36291]: pgmap v183: 97 pgs: 97 active+clean; 72 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 4.2 MiB/s wr, 367 op/s 2024-03-16T11:56:52.787 INFO:tasks.workunit.client.0.smithi029.stdout: 1 598626 11.33 MB/sec execute 157 sec latency 72.550 ms 2024-03-16T11:56:53.787 INFO:tasks.workunit.client.0.smithi029.stdout: 1 600680 11.35 MB/sec execute 158 sec latency 29.361 ms 2024-03-16T11:56:54.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:56:54 smithi154 ceph-mon[36320]: pgmap v184: 97 pgs: 97 active+clean; 72 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 2.8 MiB/s wr, 279 op/s 2024-03-16T11:56:54.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:56:54 smithi029 ceph-mon[31975]: pgmap v184: 97 pgs: 97 active+clean; 72 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 2.8 MiB/s wr, 279 op/s 2024-03-16T11:56:54.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:56:54 smithi152 ceph-mon[36291]: pgmap v184: 97 pgs: 97 active+clean; 72 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 2.8 MiB/s wr, 279 op/s 2024-03-16T11:56:54.787 INFO:tasks.workunit.client.0.smithi029.stdout: 1 602346 11.31 MB/sec execute 159 sec latency 18.334 ms 2024-03-16T11:56:55.787 INFO:tasks.workunit.client.0.smithi029.stdout: 1 604357 11.32 MB/sec execute 160 sec latency 17.517 ms 2024-03-16T11:56:56.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:56:56 smithi154 ceph-mon[36320]: pgmap v185: 97 pgs: 97 active+clean; 72 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 2.8 MiB/s wr, 279 op/s 2024-03-16T11:56:56.337 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:56:56 smithi029 ceph-mon[31975]: pgmap v185: 97 pgs: 97 active+clean; 72 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 2.8 MiB/s wr, 279 op/s 2024-03-16T11:56:56.358 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:56.357+0000 7f9364868640 1 -- 172.21.15.29:0/1276618801 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9360069800 msgr2=0x7f936010e670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:56:56.359 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:56.357+0000 7f9364868640 1 --2- 172.21.15.29:0/1276618801 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9360069800 0x7f936010e670 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f9358009ca0 tx=0x7f935802c0f0 comp rx=0 tx=0).stop 2024-03-16T11:56:56.359 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:56.357+0000 7f9364868640 1 -- 172.21.15.29:0/1276618801 shutdown_connections 2024-03-16T11:56:56.359 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:56.357+0000 7f9364868640 1 --2- 172.21.15.29:0/1276618801 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9360069800 0x7f936010e670 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:56:56.359 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:56.357+0000 7f9364868640 1 --2- 172.21.15.29:0/1276618801 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9360068e60 0x7f93600692c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:56:56.359 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:56.357+0000 7f9364868640 1 --2- 172.21.15.29:0/1276618801 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f93601078d0 0x7f9360107cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:56:56.359 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:56.357+0000 7f9364868640 1 -- 172.21.15.29:0/1276618801 >> 172.21.15.29:0/1276618801 conn(0x7f9360077530 msgr2=0x7f9360077940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:56:56.359 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:56.357+0000 7f9364868640 1 -- 172.21.15.29:0/1276618801 shutdown_connections 2024-03-16T11:56:56.359 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:56.357+0000 7f9364868640 1 -- 172.21.15.29:0/1276618801 wait complete. 2024-03-16T11:56:56.359 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:56.358+0000 7f9364868640 1 Processor -- start 2024-03-16T11:56:56.359 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:56.358+0000 7f9364868640 1 -- start start 2024-03-16T11:56:56.359 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:56.358+0000 7f9364868640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9360068e60 0x7f9360100a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:56:56.359 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:56.358+0000 7f9364868640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9360069800 0x7f9360100f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:56:56.359 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:56.358+0000 7f9364868640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f93601078d0 0x7f9360101480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:56:56.359 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:56.358+0000 7f9364868640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f9360111c50 con 0x7f9360069800 2024-03-16T11:56:56.360 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:56.358+0000 7f9364868640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f9360111ad0 con 0x7f9360068e60 2024-03-16T11:56:56.360 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:56.358+0000 7f9364868640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f9360111dd0 con 0x7f93601078d0 2024-03-16T11:56:56.360 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:56.358+0000 7f9357fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9360069800 0x7f9360100f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:56:56.360 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:56.358+0000 7f935e59c640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9360068e60 0x7f9360100a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:56:56.360 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:56.358+0000 7f935ed9d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f93601078d0 0x7f9360101480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:56:56.360 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:56.358+0000 7f9357fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9360069800 0x7f9360100f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36700/0 (socket says 172.21.15.29:36700) 2024-03-16T11:56:56.360 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:56.358+0000 7f9357fff640 1 -- 172.21.15.29:0/393045429 learned_addr learned my addr 172.21.15.29:0/393045429 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:56:56.360 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:56.358+0000 7f935ed9d640 1 -- 172.21.15.29:0/393045429 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9360068e60 msgr2=0x7f9360100a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:56:56.360 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:56.358+0000 7f935ed9d640 1 --2- 172.21.15.29:0/393045429 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9360068e60 0x7f9360100a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:56:56.360 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:56.358+0000 7f935ed9d640 1 -- 172.21.15.29:0/393045429 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9360069800 msgr2=0x7f9360100f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:56:56.360 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:56.358+0000 7f935ed9d640 1 --2- 172.21.15.29:0/393045429 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9360069800 0x7f9360100f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:56:56.360 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:56.358+0000 7f935ed9d640 1 -- 172.21.15.29:0/393045429 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93601a6450 con 0x7f93601078d0 2024-03-16T11:56:56.360 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:56.358+0000 7f9357fff640 1 --2- 172.21.15.29:0/393045429 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9360069800 0x7f9360100f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:56:56.360 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:56.358+0000 7f935e59c640 1 --2- 172.21.15.29:0/393045429 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9360068e60 0x7f9360100a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:56:56.360 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:56.359+0000 7f935ed9d640 1 --2- 172.21.15.29:0/393045429 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f93601078d0 0x7f9360101480 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f935802c700 tx=0x7f9358032f20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:56:56.361 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:56.359+0000 7f93577fe640 1 -- 172.21.15.29:0/393045429 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9358041020 con 0x7f93601078d0 2024-03-16T11:56:56.361 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:56.359+0000 7f9364868640 1 -- 172.21.15.29:0/393045429 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f93601a66e0 con 0x7f93601078d0 2024-03-16T11:56:56.361 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:56.359+0000 7f93577fe640 1 -- 172.21.15.29:0/393045429 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9358034cd0 con 0x7f93601078d0 2024-03-16T11:56:56.361 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:56.359+0000 7f93577fe640 1 -- 172.21.15.29:0/393045429 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9358039040 con 0x7f93601078d0 2024-03-16T11:56:56.361 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:56.359+0000 7f9364868640 1 -- 172.21.15.29:0/393045429 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f93601a6ba0 con 0x7f93601078d0 2024-03-16T11:56:56.367 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:56.361+0000 7f936744f740 1 Processor -- start 2024-03-16T11:56:56.368 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:56.361+0000 7f93577fe640 1 -- 172.21.15.29:0/393045429 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7f9358044020 con 0x7f93601078d0 2024-03-16T11:56:56.368 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:56.361+0000 7f936744f740 1 -- start start 2024-03-16T11:56:56.368 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:56.362+0000 7f936744f740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558857acd130 0x558857acd510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:56:56.368 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:56.362+0000 7f936744f740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558857acda50 0x558857ad62e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:56:56.368 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:56.362+0000 7f936744f740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558857ad6820 0x558857ad8c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:56:56.368 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:56.362+0000 7f936744f740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x558857a31a40 con 0x558857acda50 2024-03-16T11:56:56.368 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:56.362+0000 7f936744f740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x5588579da920 con 0x558857acd130 2024-03-16T11:56:56.368 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:56.362+0000 7f936744f740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x558857a2dae0 con 0x558857ad6820 2024-03-16T11:56:56.368 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:56.362+0000 7f935e59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558857acda50 0x558857ad62e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:56:56.368 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:56.362+0000 7f935e59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558857acda50 0x558857ad62e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36714/0 (socket says 172.21.15.29:36714) 2024-03-16T11:56:56.368 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:56.362+0000 7f935e59c640 1 -- 172.21.15.29:0/1892046937 learned_addr learned my addr 172.21.15.29:0/1892046937 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:56:56.368 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:56.362+0000 7f9357fff640 1 --2- 172.21.15.29:0/1892046937 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558857acd130 0x558857acd510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:56:56.368 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:56.362+0000 7f9357fff640 1 --2- 172.21.15.29:0/1892046937 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558857ad6820 0x558857ad8c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:56:56.368 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:56.362+0000 7f93577fe640 1 --2- 172.21.15.29:0/393045429 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f934c0a2a40 0x7f934c0a4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:56:56.369 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:56.362+0000 7f935e59c640 1 -- 172.21.15.29:0/1892046937 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558857ad6820 msgr2=0x558857ad8c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:56:56.369 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:56.362+0000 7f935e59c640 1 --2- 172.21.15.29:0/1892046937 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558857ad6820 0x558857ad8c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:56:56.369 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:56.362+0000 7f935e59c640 1 -- 172.21.15.29:0/1892046937 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558857acd130 msgr2=0x558857acd510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:56:56.369 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:56.362+0000 7f935e59c640 1 --2- 172.21.15.29:0/1892046937 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558857acd130 0x558857acd510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:56:56.369 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:56.362+0000 7f935e59c640 1 -- 172.21.15.29:0/1892046937 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5588579f2460 con 0x558857acda50 2024-03-16T11:56:56.369 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:56.362+0000 7f9357fff640 1 --2- 172.21.15.29:0/1892046937 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558857acd130 0x558857acd510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T11:56:56.369 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:56.362+0000 7f935ed9d640 1 --2- 172.21.15.29:0/393045429 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f934c0a2a40 0x7f934c0a4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:56:56.369 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:56.362+0000 7f935e59c640 1 --2- 172.21.15.29:0/1892046937 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558857acda50 0x558857ad62e0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f93500114c0 tx=0x7f9350011980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6475b37b44df81d8 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:56:56.369 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:56.362+0000 7f93577fe640 1 -- 172.21.15.29:0/393045429 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8512+0+0 (secure 0 0 0) 0x7f93580f0950 con 0x7f93601078d0 2024-03-16T11:56:56.369 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:56.362+0000 7f9354ff9640 1 -- 172.21.15.29:0/1892046937 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9350004020 con 0x558857acda50 2024-03-16T11:56:56.369 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:56.363+0000 7f9354ff9640 1 -- 172.21.15.29:0/1892046937 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f93500041c0 con 0x558857acda50 2024-03-16T11:56:56.369 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:56.363+0000 7f9354ff9640 1 -- 172.21.15.29:0/1892046937 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f935000f8f0 con 0x558857acda50 2024-03-16T11:56:56.369 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:56.363+0000 7f936744f740 1 -- 172.21.15.29:0/1892046937 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558857acda50 msgr2=0x558857ad62e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:56:56.369 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:56.363+0000 7f935ed9d640 1 --2- 172.21.15.29:0/393045429 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f934c0a2a40 0x7f934c0a4f00 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f9358075630 tx=0x7f935803d000 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:56:56.370 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:56.363+0000 7f936744f740 1 --2- 172.21.15.29:0/1892046937 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558857acda50 0x558857ad62e0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f93500114c0 tx=0x7f9350011980 comp rx=0 tx=0).stop 2024-03-16T11:56:56.370 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:56.363+0000 7f936744f740 1 -- 172.21.15.29:0/1892046937 shutdown_connections 2024-03-16T11:56:56.370 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:56.363+0000 7f936744f740 1 --2- 172.21.15.29:0/1892046937 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558857ad6820 0x558857ad8c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:56:56.370 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:56.363+0000 7f936744f740 1 --2- 172.21.15.29:0/1892046937 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558857acda50 0x558857ad62e0 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:56:56.370 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:56.363+0000 7f936744f740 1 --2- 172.21.15.29:0/1892046937 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558857acd130 0x558857acd510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:56:56.370 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:56.363+0000 7f936744f740 1 -- 172.21.15.29:0/1892046937 >> 172.21.15.29:0/1892046937 conn(0x558857ac7a50 msgr2=0x558857ac7e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:56:56.370 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:56.363+0000 7f936744f740 1 -- 172.21.15.29:0/1892046937 shutdown_connections 2024-03-16T11:56:56.370 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:56.363+0000 7f936744f740 1 -- 172.21.15.29:0/1892046937 wait complete. 2024-03-16T11:56:56.370 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:56.364+0000 7f936744f740 1 -- start start 2024-03-16T11:56:56.370 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:56.364+0000 7f936744f740 1 Processor -- start 2024-03-16T11:56:56.370 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:56.364+0000 7f936744f740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558857acd130 0x558857ad3890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:56:56.370 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:56.364+0000 7f936744f740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558857acda50 0x558857ad3dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:56:56.370 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:56.364+0000 7f936744f740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558857ad6820 0x558857ad4310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:56:56.370 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:56.364+0000 7f936744f740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x558857ad9450 con 0x558857acd130 2024-03-16T11:56:56.371 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:56.364+0000 7f936744f740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x5588579edd70 con 0x558857acda50 2024-03-16T11:56:56.371 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:56.364+0000 7f936744f740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x558857ad95d0 con 0x558857ad6820 2024-03-16T11:56:56.371 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:56.364+0000 7f936744f740 10 client.0 _ll_register_callbacks cb 0x558857745a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T11:56:56.371 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:56.364+0000 7f9357fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558857acd130 0x558857ad3890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:56:56.371 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:56.364+0000 7f9357fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558857acd130 0x558857ad3890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36730/0 (socket says 172.21.15.29:36730) 2024-03-16T11:56:56.371 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:56.364+0000 7f9357fff640 1 -- 172.21.15.29:0/4093917303 learned_addr learned my addr 172.21.15.29:0/4093917303 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:56:56.371 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:56.364+0000 7f935e59c640 1 --2- 172.21.15.29:0/4093917303 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558857acda50 0x558857ad3dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:56:56.371 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:56.364+0000 7f9357fff640 1 --2- 172.21.15.29:0/4093917303 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558857ad6820 0x558857ad4310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:56:56.371 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:56.364+0000 7f9357fff640 1 -- 172.21.15.29:0/4093917303 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558857ad6820 msgr2=0x558857ad4310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:56:56.371 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:56.364+0000 7f9357fff640 1 --2- 172.21.15.29:0/4093917303 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558857ad6820 0x558857ad4310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:56:56.371 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:56.364+0000 7f9357fff640 1 -- 172.21.15.29:0/4093917303 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558857acda50 msgr2=0x558857ad3dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:56:56.371 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:56.364+0000 7f9357fff640 1 --2- 172.21.15.29:0/4093917303 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558857acda50 0x558857ad3dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:56:56.371 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:56.364+0000 7f9357fff640 1 -- 172.21.15.29:0/4093917303 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558857746210 con 0x558857acd130 2024-03-16T11:56:56.371 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:56.365+0000 7f9357fff640 1 --2- 172.21.15.29:0/4093917303 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558857acd130 0x558857ad3890 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f9348012e80 tx=0x7f9348004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=273db320ea805617 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:56:56.371 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:56.365+0000 7f9345ffb640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-16T11:56:56.372 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:56.365+0000 7f9345ffb640 1 -- 172.21.15.29:0/4093917303 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9348012110 con 0x558857acd130 2024-03-16T11:56:56.372 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:56.365+0000 7f9345ffb640 1 -- 172.21.15.29:0/4093917303 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8512+0+0 (secure 0 0 0) 0x7f9348004030 con 0x558857acd130 2024-03-16T11:56:56.372 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:56.365+0000 7f936744f740 1 -- 172.21.15.29:0/4093917303 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f93500041c0 con 0x558857acd130 2024-03-16T11:56:56.372 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:56.365+0000 7f9345ffb640 1 -- 172.21.15.29:0/4093917303 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f93480122b0 con 0x558857acd130 2024-03-16T11:56:56.372 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:56.365+0000 7f936744f740 10 client.15900 fetch_fsmap learned FSMap version 24 2024-03-16T11:56:56.372 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:56.365+0000 7f936744f740 1 -- 172.21.15.29:0/4093917303 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x558857812620 con 0x558857acd130 2024-03-16T11:56:56.372 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:56.365+0000 7f9345ffb640 1 -- 172.21.15.29:0/4093917303 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f93500041c0 con 0x558857acd130 2024-03-16T11:56:56.372 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:56.365+0000 7f9345ffb640 1 -- 172.21.15.29:0/4093917303 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7f93480035f0 con 0x558857acd130 2024-03-16T11:56:56.372 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:56.365+0000 7f936744f740 10 client.15900 fetch_fsmap finished waiting for FSMap version 24 2024-03-16T11:56:56.372 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:56.365+0000 7f936744f740 10 client.15900 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T11:56:56.372 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:56.365+0000 7f936744f740 20 client.15900 populate_metadata read hostname 'smithi029' 2024-03-16T11:56:56.372 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:56.365+0000 7f936744f740 1 --2- 172.21.15.29:0/4093917303 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x558857ae6a50 0x558857ae8e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:56:56.372 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:56.365+0000 7f936744f740 4 client.15900 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T11:56:56.372 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:56.365+0000 7f936744f740 1 -- 172.21.15.29:0/4093917303 --> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558857a3ef80 con 0x558857ae6a50 2024-03-16T11:56:56.372 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:56.366+0000 7f935e59c640 1 --2- 172.21.15.29:0/4093917303 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x558857ae6a50 0x558857ae8e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:56:56.373 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:56.366+0000 7f9345ffb640 10 client.15900 ms_handle_connect on v2:172.21.15.152:6838/1548731833 2024-03-16T11:56:56.373 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:56.366+0000 7f935e59c640 1 --2- 172.21.15.29:0/4093917303 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x558857ae6a50 0x558857ae8e40 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7bbd39859923f9e8 server_cookie=f9779fd1d3607abb in_seq=0 out_seq=0 2024-03-16T11:56:56.373 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:56.367+0000 7f9345ffb640 1 -- 172.21.15.29:0/4093917303 <== mds.0 v2:172.21.15.152:6838/1548731833 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x558857a3ef80 con 0x558857ae6a50 2024-03-16T11:56:56.373 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:56.367+0000 7f9345ffb640 10 client.15900 handle_command_reply: tid=0 2024-03-16T11:56:56.373 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:56.367+0000 7f936744f740 1 client.15900 shutdown 2024-03-16T11:56:56.373 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:56.367+0000 7f936744f740 1 -- 172.21.15.29:0/4093917303 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558857acd130 msgr2=0x558857ad3890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:56:56.373 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:56.367+0000 7f936744f740 1 --2- 172.21.15.29:0/4093917303 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558857acd130 0x558857ad3890 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f9348012e80 tx=0x7f9348004730 comp rx=0 tx=0).stop 2024-03-16T11:56:56.373 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:56.367+0000 7f936744f740 1 -- 172.21.15.29:0/4093917303 shutdown_connections 2024-03-16T11:56:56.373 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:56.367+0000 7f936744f740 1 --2- 172.21.15.29:0/4093917303 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558857ad6820 0x558857ad4310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:56:56.373 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:56.367+0000 7f936744f740 1 --2- 172.21.15.29:0/4093917303 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x558857ae6a50 0x558857ae8e40 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:56:56.373 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:56.367+0000 7f936744f740 1 --2- 172.21.15.29:0/4093917303 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558857acda50 0x558857ad3dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:56:56.373 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:56.367+0000 7f936744f740 1 --2- 172.21.15.29:0/4093917303 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558857acd130 0x558857ad3890 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:56:56.373 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:56.367+0000 7f9345ffb640 0 client.15900 ms_handle_reset on v2:172.21.15.152:6838/1548731833 2024-03-16T11:56:56.373 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:56.367+0000 7f936744f740 1 -- 172.21.15.29:0/4093917303 >> 172.21.15.29:0/4093917303 conn(0x558857ac7a50 msgr2=0x558857ac97a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:56:56.374 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:56.367+0000 7f936744f740 1 -- 172.21.15.29:0/4093917303 shutdown_connections 2024-03-16T11:56:56.374 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:56.367+0000 7f936744f740 1 -- 172.21.15.29:0/4093917303 wait complete. 2024-03-16T11:56:56.374 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:56.367+0000 7f936744f740 20 client.15900 trim_cache size 0 max 16384 2024-03-16T11:56:56.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:56:56 smithi152 ceph-mon[36291]: pgmap v185: 97 pgs: 97 active+clean; 72 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 2.8 MiB/s wr, 279 op/s 2024-03-16T11:56:56.386 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:56.382+0000 7f936744f740 1 Processor -- start 2024-03-16T11:56:56.386 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:56.382+0000 7f936744f740 1 -- start start 2024-03-16T11:56:56.386 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:56.382+0000 7f936744f740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558857acd130 0x558857ae21f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:56:56.386 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:56.382+0000 7f936744f740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558857acda50 0x558857aee700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:56:56.386 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:56.382+0000 7f936744f740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558857ad6820 0x558857ad3850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:56:56.386 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:56.382+0000 7f936744f740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f93601b1e60 con 0x558857acda50 2024-03-16T11:56:56.386 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:56.382+0000 7f936744f740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f93601af730 con 0x558857ad6820 2024-03-16T11:56:56.386 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:56.382+0000 7f936744f740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f93601ad560 con 0x558857acd130 2024-03-16T11:56:56.386 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:56.382+0000 7f935e59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558857acda50 0x558857aee700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:56:56.387 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:56.383+0000 7f935e59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558857acda50 0x558857aee700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36736/0 (socket says 172.21.15.29:36736) 2024-03-16T11:56:56.387 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:56.383+0000 7f935e59c640 1 -- 172.21.15.29:0/2394201183 learned_addr learned my addr 172.21.15.29:0/2394201183 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:56:56.387 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:56.383+0000 7f9357fff640 1 --2- 172.21.15.29:0/2394201183 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558857acd130 0x558857ae21f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:56:56.387 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:56.383+0000 7f9357fff640 1 --2- 172.21.15.29:0/2394201183 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558857ad6820 0x558857ad3850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:56:56.387 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:56.383+0000 7f935e59c640 1 -- 172.21.15.29:0/2394201183 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558857acd130 msgr2=0x558857ae21f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:56:56.387 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:56.383+0000 7f935e59c640 1 --2- 172.21.15.29:0/2394201183 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558857acd130 0x558857ae21f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:56:56.387 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:56.383+0000 7f935e59c640 1 -- 172.21.15.29:0/2394201183 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558857ad6820 msgr2=0x558857ad3850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:56:56.387 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:56.383+0000 7f935e59c640 1 --2- 172.21.15.29:0/2394201183 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558857ad6820 0x558857ad3850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:56:56.387 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:56.383+0000 7f935e59c640 1 -- 172.21.15.29:0/2394201183 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558857ac7580 con 0x558857acda50 2024-03-16T11:56:56.387 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:56.383+0000 7f935e59c640 1 --2- 172.21.15.29:0/2394201183 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558857acda50 0x558857aee700 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f9350005f10 tx=0x7f93500041e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b11e349327a52e5c server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:56:56.387 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:56.383+0000 7f9347fff640 1 -- 172.21.15.29:0/2394201183 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f935001d040 con 0x558857acda50 2024-03-16T11:56:56.387 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:56.383+0000 7f9347fff640 1 -- 172.21.15.29:0/2394201183 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9350029070 con 0x558857acda50 2024-03-16T11:56:56.387 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:56.383+0000 7f9347fff640 1 -- 172.21.15.29:0/2394201183 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93500050d0 con 0x558857acda50 2024-03-16T11:56:56.387 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:56.384+0000 7f936744f740 1 -- 172.21.15.29:0/2394201183 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558857acda50 msgr2=0x558857aee700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:56:56.387 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:56.384+0000 7f936744f740 1 --2- 172.21.15.29:0/2394201183 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558857acda50 0x558857aee700 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f9350005f10 tx=0x7f93500041e0 comp rx=0 tx=0).stop 2024-03-16T11:56:56.388 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:56.384+0000 7f936744f740 1 -- 172.21.15.29:0/2394201183 shutdown_connections 2024-03-16T11:56:56.388 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:56.384+0000 7f936744f740 1 --2- 172.21.15.29:0/2394201183 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558857ad6820 0x558857ad3850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:56:56.388 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:56.384+0000 7f936744f740 1 --2- 172.21.15.29:0/2394201183 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558857acda50 0x558857aee700 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:56:56.388 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:56.384+0000 7f936744f740 1 --2- 172.21.15.29:0/2394201183 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558857acd130 0x558857ae21f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:56:56.388 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:56.384+0000 7f936744f740 1 -- 172.21.15.29:0/2394201183 >> 172.21.15.29:0/2394201183 conn(0x558857ac7a50 msgr2=0x558857acec40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:56:56.388 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:56.384+0000 7f936744f740 1 -- 172.21.15.29:0/2394201183 shutdown_connections 2024-03-16T11:56:56.388 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:56.384+0000 7f936744f740 1 -- 172.21.15.29:0/2394201183 wait complete. 2024-03-16T11:56:56.388 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:56.384+0000 7f936744f740 1 -- start start 2024-03-16T11:56:56.388 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:56.384+0000 7f936744f740 1 Processor -- start 2024-03-16T11:56:56.388 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:56.385+0000 7f936744f740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558857acd130 0x558857adea20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:56:56.388 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:56.385+0000 7f936744f740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558857acda50 0x558857ad9de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:56:56.388 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:56.385+0000 7f936744f740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558857ad6820 0x558857ada320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:56:56.388 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:56.385+0000 7f936744f740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f93601adbf0 con 0x558857acda50 2024-03-16T11:56:56.388 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:56.385+0000 7f936744f740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f93601af060 con 0x558857ad6820 2024-03-16T11:56:56.388 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:56.385+0000 7f936744f740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f93601ae2c0 con 0x558857acd130 2024-03-16T11:56:56.389 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:56.385+0000 7f936744f740 10 client.0 _ll_register_callbacks cb 0x7f9348009ec0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T11:56:56.389 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:56.385+0000 7f935e59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558857acda50 0x558857ad9de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:56:56.389 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:56.385+0000 7f935e59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558857acda50 0x558857ad9de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36746/0 (socket says 172.21.15.29:36746) 2024-03-16T11:56:56.389 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:56.385+0000 7f935e59c640 1 -- 172.21.15.29:0/3247502013 learned_addr learned my addr 172.21.15.29:0/3247502013 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:56:56.389 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:56.385+0000 7f9357fff640 1 --2- 172.21.15.29:0/3247502013 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558857acd130 0x558857adea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:56:56.389 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:56.385+0000 7f9357fff640 1 --2- 172.21.15.29:0/3247502013 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558857ad6820 0x558857ada320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:56:56.389 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:56.385+0000 7f935e59c640 1 -- 172.21.15.29:0/3247502013 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558857acd130 msgr2=0x558857adea20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:56:56.389 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:56.385+0000 7f935e59c640 1 --2- 172.21.15.29:0/3247502013 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558857acd130 0x558857adea20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:56:56.389 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:56.385+0000 7f935e59c640 1 -- 172.21.15.29:0/3247502013 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558857ad6820 msgr2=0x558857ada320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:56:56.389 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:56.385+0000 7f935e59c640 1 --2- 172.21.15.29:0/3247502013 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558857ad6820 0x558857ada320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:56:56.389 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:56.385+0000 7f935e59c640 1 -- 172.21.15.29:0/3247502013 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558857adab20 con 0x558857acda50 2024-03-16T11:56:56.389 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:56.385+0000 7f9357fff640 1 --2- 172.21.15.29:0/3247502013 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558857acd130 0x558857adea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:56:56.389 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:56.385+0000 7f935e59c640 1 --2- 172.21.15.29:0/3247502013 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558857acda50 0x558857ad9de0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f9350014040 tx=0x7f9350005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7eb3c39c1d116af3 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:56:56.390 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:56.385+0000 7f9345ffb640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-16T11:56:56.390 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:56.385+0000 7f9345ffb640 1 -- 172.21.15.29:0/3247502013 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9350037020 con 0x558857acda50 2024-03-16T11:56:56.390 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:56.385+0000 7f9345ffb640 1 -- 172.21.15.29:0/3247502013 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8512+0+0 (secure 0 0 0) 0x7f9350009bf0 con 0x558857acda50 2024-03-16T11:56:56.390 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:56.385+0000 7f936744f740 1 -- 172.21.15.29:0/3247502013 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9350029070 con 0x558857acda50 2024-03-16T11:56:56.390 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:56.386+0000 7f9345ffb640 1 -- 172.21.15.29:0/3247502013 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f935001d070 con 0x558857acda50 2024-03-16T11:56:56.390 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:56.386+0000 7f936744f740 10 client.15906 fetch_fsmap learned FSMap version 24 2024-03-16T11:56:56.390 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:56.386+0000 7f936744f740 1 -- 172.21.15.29:0/3247502013 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x558857b4ccb0 con 0x558857acda50 2024-03-16T11:56:56.390 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:56.386+0000 7f9345ffb640 1 -- 172.21.15.29:0/3247502013 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9350029070 con 0x558857acda50 2024-03-16T11:56:56.390 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:56.386+0000 7f9345ffb640 1 -- 172.21.15.29:0/3247502013 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7f9350005430 con 0x558857acda50 2024-03-16T11:56:56.390 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:56.386+0000 7f936744f740 10 client.15906 fetch_fsmap finished waiting for FSMap version 24 2024-03-16T11:56:56.390 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:56.386+0000 7f936744f740 10 client.15906 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T11:56:56.390 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:56.386+0000 7f936744f740 20 client.15906 populate_metadata read hostname 'smithi029' 2024-03-16T11:56:56.390 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:56.386+0000 7f936744f740 1 --2- 172.21.15.29:0/3247502013 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x558857b4d230 0x558857b4f650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:56:56.390 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:56.386+0000 7f936744f740 4 client.15906 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-16T11:56:56.391 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:56.386+0000 7f936744f740 1 -- 172.21.15.29:0/3247502013 --> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] -- command(tid 0: {"prefix": "flush journal"}) -- 0x558857a392d0 con 0x558857b4d230 2024-03-16T11:56:56.391 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:56.386+0000 7f9357fff640 1 --2- 172.21.15.29:0/3247502013 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x558857b4d230 0x558857b4f650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:56:56.391 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:56.387+0000 7f9357fff640 1 --2- 172.21.15.29:0/3247502013 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x558857b4d230 0x558857b4f650 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=823780300aa83de server_cookie=e557796d8e6fdf89 in_seq=0 out_seq=0 2024-03-16T11:56:56.391 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:56.387+0000 7f9345ffb640 10 client.15906 ms_handle_connect on v2:172.21.15.152:6838/1548731833 2024-03-16T11:56:56.546 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:56.547+0000 7f9345ffb640 1 -- 172.21.15.29:0/3247502013 <== mds.0 v2:172.21.15.152:6838/1548731833 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x558857a392d0 con 0x558857b4d230 2024-03-16T11:56:56.547 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:56.547+0000 7f9345ffb640 10 client.15906 handle_command_reply: tid=0 2024-03-16T11:56:56.547 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:56.547+0000 7f936744f740 1 client.15906 shutdown 2024-03-16T11:56:56.547 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:56.547+0000 7f936744f740 1 -- 172.21.15.29:0/3247502013 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558857acda50 msgr2=0x558857ad9de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:56:56.547 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:56.547+0000 7f936744f740 1 --2- 172.21.15.29:0/3247502013 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558857acda50 0x558857ad9de0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f9350014040 tx=0x7f9350005fd0 comp rx=0 tx=0).stop 2024-03-16T11:56:56.547 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:56.547+0000 7f936744f740 1 -- 172.21.15.29:0/3247502013 shutdown_connections 2024-03-16T11:56:56.547 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:56.547+0000 7f936744f740 1 --2- 172.21.15.29:0/3247502013 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x558857b4d230 0x558857b4f650 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:56:56.547 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:56.547+0000 7f936744f740 1 --2- 172.21.15.29:0/3247502013 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558857ad6820 0x558857ada320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:56:56.547 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:56.547+0000 7f936744f740 1 --2- 172.21.15.29:0/3247502013 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558857acda50 0x558857ad9de0 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:56:56.547 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:56.547+0000 7f9345ffb640 0 client.15906 ms_handle_reset on v2:172.21.15.152:6838/1548731833 2024-03-16T11:56:56.547 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:56.547+0000 7f936744f740 1 --2- 172.21.15.29:0/3247502013 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558857acd130 0x558857adea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:56:56.547 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:56.547+0000 7f936744f740 1 -- 172.21.15.29:0/3247502013 >> 172.21.15.29:0/3247502013 conn(0x558857ac7a50 msgr2=0x558857ad3850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:56:56.548 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:56.547+0000 7f936744f740 1 -- 172.21.15.29:0/3247502013 shutdown_connections 2024-03-16T11:56:56.548 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:56.547+0000 7f936744f740 1 -- 172.21.15.29:0/3247502013 wait complete. 2024-03-16T11:56:56.548 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:56.547+0000 7f936744f740 20 client.15906 trim_cache size 0 max 16384 2024-03-16T11:56:56.548 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:{ 2024-03-16T11:56:56.548 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "message": "", 2024-03-16T11:56:56.548 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "return_code": 0 2024-03-16T11:56:56.548 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:} 2024-03-16T11:56:56.549 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:56.547+0000 7f9364868640 1 -- 172.21.15.29:0/393045429 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f934c0a2a40 msgr2=0x7f934c0a4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:56:56.549 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:56.547+0000 7f9364868640 1 --2- 172.21.15.29:0/393045429 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f934c0a2a40 0x7f934c0a4f00 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f9358075630 tx=0x7f935803d000 comp rx=0 tx=0).stop 2024-03-16T11:56:56.549 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:56.547+0000 7f9364868640 1 -- 172.21.15.29:0/393045429 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f93601078d0 msgr2=0x7f9360101480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:56:56.549 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:56.547+0000 7f9364868640 1 --2- 172.21.15.29:0/393045429 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f93601078d0 0x7f9360101480 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f935802c700 tx=0x7f9358032f20 comp rx=0 tx=0).stop 2024-03-16T11:56:56.549 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:56.548+0000 7f9364868640 1 -- 172.21.15.29:0/393045429 shutdown_connections 2024-03-16T11:56:56.549 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:56.548+0000 7f9364868640 1 --2- 172.21.15.29:0/393045429 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f934c0a2a40 0x7f934c0a4f00 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:56:56.549 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:56.548+0000 7f9364868640 1 --2- 172.21.15.29:0/393045429 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f93601078d0 0x7f9360101480 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:56:56.549 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:56.548+0000 7f9364868640 1 --2- 172.21.15.29:0/393045429 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9360069800 0x7f9360100f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:56:56.550 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:56.548+0000 7f9364868640 1 --2- 172.21.15.29:0/393045429 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9360068e60 0x7f9360100a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:56:56.550 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:56.548+0000 7f9364868640 1 -- 172.21.15.29:0/393045429 >> 172.21.15.29:0/393045429 conn(0x7f9360077530 msgr2=0x7f93600fe8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:56:56.550 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:56.548+0000 7f9364868640 1 -- 172.21.15.29:0/393045429 shutdown_connections 2024-03-16T11:56:56.550 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:56:56.548+0000 7f9364868640 1 -- 172.21.15.29:0/393045429 wait complete. 2024-03-16T11:56:56.787 INFO:tasks.workunit.client.0.smithi029.stdout: 1 605946 11.29 MB/sec execute 161 sec latency 92.330 ms 2024-03-16T11:56:57.787 INFO:tasks.workunit.client.0.smithi029.stdout: 1 607733 11.30 MB/sec execute 162 sec latency 79.949 ms 2024-03-16T11:56:58.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:56:58 smithi154 ceph-mon[36320]: pgmap v186: 97 pgs: 97 active+clean; 77 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 4.2 MiB/s wr, 367 op/s 2024-03-16T11:56:58.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:56:58 smithi029 ceph-mon[31975]: pgmap v186: 97 pgs: 97 active+clean; 77 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 4.2 MiB/s wr, 367 op/s 2024-03-16T11:56:58.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:56:58 smithi152 ceph-mon[36291]: pgmap v186: 97 pgs: 97 active+clean; 77 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 4.2 MiB/s wr, 367 op/s 2024-03-16T11:56:58.788 INFO:tasks.workunit.client.0.smithi029.stdout: 1 609170 11.27 MB/sec execute 163 sec latency 29.444 ms 2024-03-16T11:56:59.788 INFO:tasks.workunit.client.0.smithi029.stdout: 1 611357 11.28 MB/sec execute 164 sec latency 17.117 ms 2024-03-16T11:57:00.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:57:00 smithi154 ceph-mon[36320]: pgmap v187: 97 pgs: 97 active+clean; 77 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 2.8 MiB/s wr, 179 op/s 2024-03-16T11:57:00.337 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:57:00 smithi029 ceph-mon[31975]: pgmap v187: 97 pgs: 97 active+clean; 77 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 2.8 MiB/s wr, 179 op/s 2024-03-16T11:57:00.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:57:00 smithi152 ceph-mon[36291]: pgmap v187: 97 pgs: 97 active+clean; 77 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 2.8 MiB/s wr, 179 op/s 2024-03-16T11:57:00.788 INFO:tasks.workunit.client.0.smithi029.stdout: 1 613283 11.25 MB/sec execute 165 sec latency 22.315 ms 2024-03-16T11:57:01.788 INFO:tasks.workunit.client.0.smithi029.stdout: 1 614731 11.26 MB/sec execute 166 sec latency 25.440 ms 2024-03-16T11:57:02.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:57:02 smithi154 ceph-mon[36320]: pgmap v188: 97 pgs: 97 active+clean; 66 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 4.0 MiB/s wr, 377 op/s 2024-03-16T11:57:02.337 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:57:02 smithi029 ceph-mon[31975]: pgmap v188: 97 pgs: 97 active+clean; 66 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 4.0 MiB/s wr, 377 op/s 2024-03-16T11:57:02.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:57:02 smithi152 ceph-mon[36291]: pgmap v188: 97 pgs: 97 active+clean; 66 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 4.0 MiB/s wr, 377 op/s 2024-03-16T11:57:02.788 INFO:tasks.workunit.client.0.smithi029.stdout: 1 616086 11.22 MB/sec execute 167 sec latency 38.912 ms 2024-03-16T11:57:03.788 INFO:tasks.workunit.client.0.smithi029.stdout: 1 618246 11.24 MB/sec execute 168 sec latency 18.379 ms 2024-03-16T11:57:04.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:57:04 smithi154 ceph-mon[36320]: pgmap v189: 97 pgs: 97 active+clean; 66 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.6 MiB/s wr, 286 op/s 2024-03-16T11:57:04.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:57:04 smithi029 ceph-mon[31975]: pgmap v189: 97 pgs: 97 active+clean; 66 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.6 MiB/s wr, 286 op/s 2024-03-16T11:57:04.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:57:04 smithi152 ceph-mon[36291]: pgmap v189: 97 pgs: 97 active+clean; 66 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.6 MiB/s wr, 286 op/s 2024-03-16T11:57:04.788 INFO:tasks.workunit.client.0.smithi029.stdout: 1 618791 11.18 MB/sec execute 169 sec latency 94.660 ms 2024-03-16T11:57:05.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:57:05 smithi154 ceph-mon[36320]: osdmap e91: 12 total, 12 up, 12 in 2024-03-16T11:57:05.337 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:57:05 smithi029 ceph-mon[31975]: osdmap e91: 12 total, 12 up, 12 in 2024-03-16T11:57:05.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:57:05 smithi152 ceph-mon[36291]: osdmap e91: 12 total, 12 up, 12 in 2024-03-16T11:57:05.788 INFO:tasks.workunit.client.0.smithi029.stdout: 1 619651 11.14 MB/sec execute 170 sec latency 39.836 ms 2024-03-16T11:57:06.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:57:06 smithi154 ceph-mon[36320]: pgmap v191: 97 pgs: 97 active+clean; 66 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.1 MiB/s wr, 343 op/s 2024-03-16T11:57:06.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:57:06 smithi154 ceph-mon[36320]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T11:57:06.337 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:57:06 smithi029 ceph-mon[31975]: pgmap v191: 97 pgs: 97 active+clean; 66 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.1 MiB/s wr, 343 op/s 2024-03-16T11:57:06.337 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:57:06 smithi029 ceph-mon[31975]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T11:57:06.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:57:06 smithi152 ceph-mon[36291]: pgmap v191: 97 pgs: 97 active+clean; 66 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.1 MiB/s wr, 343 op/s 2024-03-16T11:57:06.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:57:06 smithi152 ceph-mon[36291]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T11:57:06.788 INFO:tasks.workunit.client.0.smithi029.stdout: 1 621839 11.15 MB/sec execute 171 sec latency 19.974 ms 2024-03-16T11:57:07.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:57:07 smithi154 ceph-mon[36320]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:57:07.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:57:07 smithi154 ceph-mon[36320]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T11:57:07.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:57:07 smithi154 ceph-mon[36320]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' 2024-03-16T11:57:07.337 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:57:07 smithi029 ceph-mon[31975]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:57:07.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:57:07 smithi029 ceph-mon[31975]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T11:57:07.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:57:07 smithi029 ceph-mon[31975]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' 2024-03-16T11:57:07.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:57:07 smithi152 ceph-mon[36291]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:57:07.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:57:07 smithi152 ceph-mon[36291]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T11:57:07.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:57:07 smithi152 ceph-mon[36291]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' 2024-03-16T11:57:07.788 INFO:tasks.workunit.client.0.smithi029.stdout: 1 623417 11.12 MB/sec execute 172 sec latency 26.487 ms 2024-03-16T11:57:08.627 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:57:08 smithi152 ceph-mon[36291]: pgmap v192: 97 pgs: 97 active+clean; 70 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.0 MiB/s wr, 345 op/s 2024-03-16T11:57:08.788 INFO:tasks.workunit.client.0.smithi029.stdout: 1 625343 11.13 MB/sec execute 173 sec latency 64.231 ms 2024-03-16T11:57:08.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:57:08 smithi154 ceph-mon[36320]: pgmap v192: 97 pgs: 97 active+clean; 70 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.0 MiB/s wr, 345 op/s 2024-03-16T11:57:08.837 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:57:08 smithi029 ceph-mon[31975]: pgmap v192: 97 pgs: 97 active+clean; 70 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.0 MiB/s wr, 345 op/s 2024-03-16T11:57:09.646 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:09.644+0000 7fd4a6e31640 1 -- 172.21.15.29:0/3832788408 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd4a0101380 msgr2=0x7fd4a010d500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:57:09.646 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:09.644+0000 7fd4a6e31640 1 --2- 172.21.15.29:0/3832788408 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd4a0101380 0x7fd4a010d500 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7fd498009ca0 tx=0x7fd49802c0f0 comp rx=0 tx=0).stop 2024-03-16T11:57:09.646 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:09.644+0000 7fd4a6e31640 1 -- 172.21.15.29:0/3832788408 shutdown_connections 2024-03-16T11:57:09.646 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:09.644+0000 7fd4a6e31640 1 --2- 172.21.15.29:0/3832788408 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd4a0101380 0x7fd4a010d500 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:57:09.646 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:09.644+0000 7fd4a6e31640 1 --2- 172.21.15.29:0/3832788408 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd4a01009c0 0x7fd4a0100e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:57:09.646 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:09.644+0000 7fd4a6e31640 1 --2- 172.21.15.29:0/3832788408 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd4a0105670 0x7fd4a0105a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:57:09.646 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:09.644+0000 7fd4a6e31640 1 -- 172.21.15.29:0/3832788408 >> 172.21.15.29:0/3832788408 conn(0x7fd4a00776c0 msgr2=0x7fd4a0077ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:57:09.646 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:09.645+0000 7fd4a6e31640 1 -- 172.21.15.29:0/3832788408 shutdown_connections 2024-03-16T11:57:09.647 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:09.645+0000 7fd4a6e31640 1 -- 172.21.15.29:0/3832788408 wait complete. 2024-03-16T11:57:09.647 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:09.645+0000 7fd4a6e31640 1 Processor -- start 2024-03-16T11:57:09.647 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:09.645+0000 7fd4a6e31640 1 -- start start 2024-03-16T11:57:09.647 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:09.645+0000 7fd4a6e31640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd4a01009c0 0x7fd4a0197fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:57:09.647 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:09.645+0000 7fd4a6e31640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd4a0101380 0x7fd4a0198500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:57:09.647 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:09.645+0000 7fd4a6e31640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd4a0105670 0x7fd4a019c830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:57:09.647 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:09.645+0000 7fd4a6e31640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fd4a0110ab0 con 0x7fd4a0101380 2024-03-16T11:57:09.647 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:09.645+0000 7fd4a6e31640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fd4a0110930 con 0x7fd4a0105670 2024-03-16T11:57:09.647 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:09.645+0000 7fd4a6e31640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fd4a0110c30 con 0x7fd4a01009c0 2024-03-16T11:57:09.647 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:09.646+0000 7fd49ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd4a0101380 0x7fd4a0198500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:57:09.647 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:09.646+0000 7fd49ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd4a0101380 0x7fd4a0198500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45916/0 (socket says 172.21.15.29:45916) 2024-03-16T11:57:09.647 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:09.646+0000 7fd4a4bcd640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd4a01009c0 0x7fd4a0197fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:57:09.647 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:09.646+0000 7fd49ffff640 1 -- 172.21.15.29:0/759680852 learned_addr learned my addr 172.21.15.29:0/759680852 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:57:09.647 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:09.646+0000 7fd4a53ce640 1 --2- 172.21.15.29:0/759680852 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd4a0105670 0x7fd4a019c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:57:09.647 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:09.646+0000 7fd49ffff640 1 -- 172.21.15.29:0/759680852 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd4a01009c0 msgr2=0x7fd4a0197fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:57:09.648 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:09.646+0000 7fd49ffff640 1 --2- 172.21.15.29:0/759680852 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd4a01009c0 0x7fd4a0197fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:57:09.648 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:09.646+0000 7fd49ffff640 1 -- 172.21.15.29:0/759680852 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd4a0105670 msgr2=0x7fd4a019c830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:57:09.648 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:09.646+0000 7fd49ffff640 1 --2- 172.21.15.29:0/759680852 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd4a0105670 0x7fd4a019c830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:57:09.648 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:09.646+0000 7fd49ffff640 1 -- 172.21.15.29:0/759680852 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd4a019cf60 con 0x7fd4a0101380 2024-03-16T11:57:09.648 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:09.646+0000 7fd49ffff640 1 --2- 172.21.15.29:0/759680852 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd4a0101380 0x7fd4a0198500 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7fd48c00e9f0 tx=0x7fd48c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:57:09.648 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:09.646+0000 7fd49dffb640 1 -- 172.21.15.29:0/759680852 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd48c00cd80 con 0x7fd4a0101380 2024-03-16T11:57:09.648 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:09.646+0000 7fd4a6e31640 1 -- 172.21.15.29:0/759680852 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd4a019d250 con 0x7fd4a0101380 2024-03-16T11:57:09.648 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:09.646+0000 7fd49dffb640 1 -- 172.21.15.29:0/759680852 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd48c010910 con 0x7fd4a0101380 2024-03-16T11:57:09.648 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:09.646+0000 7fd49dffb640 1 -- 172.21.15.29:0/759680852 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd48c0188f0 con 0x7fd4a0101380 2024-03-16T11:57:09.648 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:09.647+0000 7fd4a6e31640 1 -- 172.21.15.29:0/759680852 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd4a01a4ae0 con 0x7fd4a0101380 2024-03-16T11:57:09.654 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:09.649+0000 7fd49dffb640 1 -- 172.21.15.29:0/759680852 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7fd48c010ac0 con 0x7fd4a0101380 2024-03-16T11:57:09.654 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:09.649+0000 7fd4a9a18740 1 Processor -- start 2024-03-16T11:57:09.654 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:09.649+0000 7fd4a9a18740 1 -- start start 2024-03-16T11:57:09.654 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:09.650+0000 7fd4a9a18740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55bdfa663130 0x55bdfa663510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:57:09.654 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:09.650+0000 7fd4a9a18740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bdfa663a50 0x55bdfa66c2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:57:09.654 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:09.650+0000 7fd4a9a18740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bdfa66c820 0x55bdfa66ec10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:57:09.654 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:09.650+0000 7fd4a9a18740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55bdfa5c7a40 con 0x55bdfa66c820 2024-03-16T11:57:09.654 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:09.650+0000 7fd4a9a18740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x55bdfa570920 con 0x55bdfa663130 2024-03-16T11:57:09.654 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:09.650+0000 7fd4a9a18740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55bdfa5c3ae0 con 0x55bdfa663a50 2024-03-16T11:57:09.654 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:09.650+0000 7fd49dffb640 1 --2- 172.21.15.29:0/759680852 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7fd4900a2a10 0x7fd4900a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:57:09.654 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:09.650+0000 7fd4a53ce640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55bdfa663130 0x55bdfa663510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:57:09.655 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:09.650+0000 7fd4a4bcd640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bdfa663a50 0x55bdfa66c2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:57:09.655 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:09.650+0000 7fd49ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bdfa66c820 0x55bdfa66ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:57:09.655 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:09.650+0000 7fd4a53ce640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55bdfa663130 0x55bdfa663510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.29:44682/0 (socket says 172.21.15.29:44682) 2024-03-16T11:57:09.655 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:09.650+0000 7fd4a53ce640 1 -- 172.21.15.29:0/2041529606 learned_addr learned my addr 172.21.15.29:0/2041529606 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:57:09.655 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:09.650+0000 7fd4a4bcd640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bdfa663a50 0x55bdfa66c2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.29:60156/0 (socket says 172.21.15.29:60156) 2024-03-16T11:57:09.655 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:09.650+0000 7fd4a53ce640 1 --2- 172.21.15.29:0/759680852 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7fd4900a2a10 0x7fd4900a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:57:09.655 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:09.650+0000 7fd4a53ce640 1 -- 172.21.15.29:0/2041529606 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bdfa663a50 msgr2=0x55bdfa66c2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:57:09.655 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:09.650+0000 7fd4a53ce640 1 --2- 172.21.15.29:0/2041529606 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bdfa663a50 0x55bdfa66c2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:57:09.655 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:09.650+0000 7fd4a53ce640 1 -- 172.21.15.29:0/2041529606 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bdfa66c820 msgr2=0x55bdfa66ec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:57:09.655 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:09.650+0000 7fd4a53ce640 1 --2- 172.21.15.29:0/2041529606 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bdfa66c820 0x55bdfa66ec10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:57:09.655 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:09.650+0000 7fd49dffb640 1 -- 172.21.15.29:0/759680852 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8600+0+0 (secure 0 0 0) 0x7fd48c0cf2c0 con 0x7fd4a0101380 2024-03-16T11:57:09.655 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:09.650+0000 7fd4a53ce640 1 -- 172.21.15.29:0/2041529606 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bdfa588460 con 0x55bdfa663130 2024-03-16T11:57:09.655 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:09.650+0000 7fd4a4bcd640 1 --2- 172.21.15.29:0/2041529606 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bdfa663a50 0x55bdfa66c2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:57:09.655 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:09.650+0000 7fd4a53ce640 1 --2- 172.21.15.29:0/2041529606 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55bdfa663130 0x55bdfa663510 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fd498004af0 tx=0x7fd498032e70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b6854f6efa318b7f server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:57:09.656 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:09.651+0000 7fd4a53ce640 1 --2- 172.21.15.29:0/759680852 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7fd4900a2a10 0x7fd4900a4ed0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fd498047000 tx=0x7fd498049000 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:57:09.656 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:09.651+0000 7fd48affd640 1 -- 172.21.15.29:0/2041529606 <== mon.2 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd498039040 con 0x55bdfa663130 2024-03-16T11:57:09.656 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:09.651+0000 7fd48affd640 1 -- 172.21.15.29:0/2041529606 <== mon.2 v2:172.21.15.152:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd498003c30 con 0x55bdfa663130 2024-03-16T11:57:09.656 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:09.651+0000 7fd48affd640 1 -- 172.21.15.29:0/2041529606 <== mon.2 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd49804b660 con 0x55bdfa663130 2024-03-16T11:57:09.656 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:09.651+0000 7fd4a9a18740 1 -- 172.21.15.29:0/2041529606 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55bdfa663130 msgr2=0x55bdfa663510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:57:09.656 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:09.651+0000 7fd4a9a18740 1 --2- 172.21.15.29:0/2041529606 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55bdfa663130 0x55bdfa663510 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fd498004af0 tx=0x7fd498032e70 comp rx=0 tx=0).stop 2024-03-16T11:57:09.656 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:09.651+0000 7fd4a9a18740 1 -- 172.21.15.29:0/2041529606 shutdown_connections 2024-03-16T11:57:09.656 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:09.651+0000 7fd4a9a18740 1 --2- 172.21.15.29:0/2041529606 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bdfa66c820 0x55bdfa66ec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:57:09.656 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:09.651+0000 7fd4a9a18740 1 --2- 172.21.15.29:0/2041529606 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bdfa663a50 0x55bdfa66c2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:57:09.656 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:09.651+0000 7fd4a9a18740 1 --2- 172.21.15.29:0/2041529606 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55bdfa663130 0x55bdfa663510 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:57:09.656 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:09.651+0000 7fd4a9a18740 1 -- 172.21.15.29:0/2041529606 >> 172.21.15.29:0/2041529606 conn(0x55bdfa65da50 msgr2=0x55bdfa65de30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:57:09.656 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:09.651+0000 7fd4a9a18740 1 -- 172.21.15.29:0/2041529606 shutdown_connections 2024-03-16T11:57:09.656 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:09.651+0000 7fd4a9a18740 1 -- 172.21.15.29:0/2041529606 wait complete. 2024-03-16T11:57:09.656 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:09.652+0000 7fd4a9a18740 1 -- start start 2024-03-16T11:57:09.656 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:09.652+0000 7fd4a9a18740 1 Processor -- start 2024-03-16T11:57:09.657 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:09.652+0000 7fd4a9a18740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bdfa663130 0x55bdfa6698b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:57:09.657 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:09.652+0000 7fd4a9a18740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55bdfa663a50 0x55bdfa669df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:57:09.657 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:09.652+0000 7fd4a9a18740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bdfa66c820 0x55bdfa66a330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:57:09.657 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:09.652+0000 7fd4a9a18740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55bdfa66f390 con 0x55bdfa663130 2024-03-16T11:57:09.657 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:09.652+0000 7fd4a9a18740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x55bdfa583d70 con 0x55bdfa663a50 2024-03-16T11:57:09.657 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:09.652+0000 7fd4a9a18740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55bdfa66f510 con 0x55bdfa66c820 2024-03-16T11:57:09.657 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:09.652+0000 7fd4a9a18740 10 client.0 _ll_register_callbacks cb 0x55bdfa2dba00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T11:57:09.657 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:09.652+0000 7fd4a4bcd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bdfa663130 0x55bdfa6698b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:57:09.657 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:09.652+0000 7fd4a4bcd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bdfa663130 0x55bdfa6698b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45934/0 (socket says 172.21.15.29:45934) 2024-03-16T11:57:09.657 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:09.652+0000 7fd4a4bcd640 1 -- 172.21.15.29:0/2531233192 learned_addr learned my addr 172.21.15.29:0/2531233192 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:57:09.657 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:09.652+0000 7fd49ffff640 1 --2- 172.21.15.29:0/2531233192 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55bdfa663a50 0x55bdfa669df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:57:09.657 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:09.652+0000 7fd4a53ce640 1 --2- 172.21.15.29:0/2531233192 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bdfa66c820 0x55bdfa66a330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:57:09.657 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:09.653+0000 7fd4a4bcd640 1 -- 172.21.15.29:0/2531233192 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bdfa66c820 msgr2=0x55bdfa66a330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:57:09.657 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:09.653+0000 7fd4a4bcd640 1 --2- 172.21.15.29:0/2531233192 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bdfa66c820 0x55bdfa66a330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:57:09.657 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:09.653+0000 7fd4a4bcd640 1 -- 172.21.15.29:0/2531233192 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55bdfa663a50 msgr2=0x55bdfa669df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:57:09.658 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:09.653+0000 7fd4a4bcd640 1 --2- 172.21.15.29:0/2531233192 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55bdfa663a50 0x55bdfa669df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:57:09.658 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:09.653+0000 7fd4a4bcd640 1 -- 172.21.15.29:0/2531233192 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bdfa2dc210 con 0x55bdfa663130 2024-03-16T11:57:09.658 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:09.653+0000 7fd4a4bcd640 1 --2- 172.21.15.29:0/2531233192 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bdfa663130 0x55bdfa6698b0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7fd4940126e0 tx=0x7fd494012ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=852c0a2287ece99e server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:57:09.658 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:09.653+0000 7fd47ffff640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-16T11:57:09.658 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:09.653+0000 7fd47ffff640 1 -- 172.21.15.29:0/2531233192 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd49400f920 con 0x55bdfa663130 2024-03-16T11:57:09.658 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:09.653+0000 7fd47ffff640 1 -- 172.21.15.29:0/2531233192 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8600+0+0 (secure 0 0 0) 0x7fd49400fac0 con 0x55bdfa663130 2024-03-16T11:57:09.658 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:09.653+0000 7fd4a9a18740 1 -- 172.21.15.29:0/2531233192 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd498003c30 con 0x55bdfa663130 2024-03-16T11:57:09.658 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:09.653+0000 7fd47ffff640 1 -- 172.21.15.29:0/2531233192 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fd49400fcc0 con 0x55bdfa663130 2024-03-16T11:57:09.658 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:09.653+0000 7fd4a9a18740 10 client.15918 fetch_fsmap learned FSMap version 24 2024-03-16T11:57:09.658 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:09.653+0000 7fd4a9a18740 1 -- 172.21.15.29:0/2531233192 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55bdfa3a8620 con 0x55bdfa663130 2024-03-16T11:57:09.658 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:09.653+0000 7fd47ffff640 1 -- 172.21.15.29:0/2531233192 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd498003c30 con 0x55bdfa663130 2024-03-16T11:57:09.658 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:09.654+0000 7fd47ffff640 1 -- 172.21.15.29:0/2531233192 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7fd494010550 con 0x55bdfa663130 2024-03-16T11:57:09.658 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:09.654+0000 7fd4a9a18740 10 client.15918 fetch_fsmap finished waiting for FSMap version 24 2024-03-16T11:57:09.658 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:09.654+0000 7fd4a9a18740 10 client.15918 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T11:57:09.659 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:09.654+0000 7fd4a9a18740 20 client.15918 populate_metadata read hostname 'smithi029' 2024-03-16T11:57:09.659 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:09.654+0000 7fd4a9a18740 1 --2- 172.21.15.29:0/2531233192 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55bdfa67c9f0 0x55bdfa67ee80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:57:09.659 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:09.654+0000 7fd4a9a18740 4 client.15918 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T11:57:09.659 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:09.654+0000 7fd4a9a18740 1 -- 172.21.15.29:0/2531233192 --> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bdfa5d4f80 con 0x55bdfa67c9f0 2024-03-16T11:57:09.659 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:09.654+0000 7fd49ffff640 1 --2- 172.21.15.29:0/2531233192 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55bdfa67c9f0 0x55bdfa67ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:57:09.659 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:09.654+0000 7fd47ffff640 10 client.15918 ms_handle_connect on v2:172.21.15.152:6838/1548731833 2024-03-16T11:57:09.659 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:09.654+0000 7fd49ffff640 1 --2- 172.21.15.29:0/2531233192 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55bdfa67c9f0 0x55bdfa67ee80 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=20a173be0ce653b1 server_cookie=4b3eea51f219057c in_seq=0 out_seq=0 2024-03-16T11:57:09.660 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:09.656+0000 7fd47ffff640 1 -- 172.21.15.29:0/2531233192 <== mds.0 v2:172.21.15.152:6838/1548731833 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55bdfa5d4f80 con 0x55bdfa67c9f0 2024-03-16T11:57:09.660 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:09.656+0000 7fd47ffff640 10 client.15918 handle_command_reply: tid=0 2024-03-16T11:57:09.660 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:09.656+0000 7fd4a9a18740 1 client.15918 shutdown 2024-03-16T11:57:09.660 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:09.656+0000 7fd4a9a18740 1 -- 172.21.15.29:0/2531233192 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bdfa663130 msgr2=0x55bdfa6698b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:57:09.660 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:09.656+0000 7fd4a9a18740 1 --2- 172.21.15.29:0/2531233192 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bdfa663130 0x55bdfa6698b0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7fd4940126e0 tx=0x7fd494012ba0 comp rx=0 tx=0).stop 2024-03-16T11:57:09.660 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:09.656+0000 7fd4a9a18740 1 -- 172.21.15.29:0/2531233192 shutdown_connections 2024-03-16T11:57:09.660 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:09.656+0000 7fd4a9a18740 1 --2- 172.21.15.29:0/2531233192 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bdfa66c820 0x55bdfa66a330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:57:09.661 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:09.656+0000 7fd4a9a18740 1 --2- 172.21.15.29:0/2531233192 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55bdfa67c9f0 0x55bdfa67ee80 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:57:09.661 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:09.656+0000 7fd4a9a18740 1 --2- 172.21.15.29:0/2531233192 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55bdfa663a50 0x55bdfa669df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:57:09.661 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:09.656+0000 7fd47ffff640 0 client.15918 ms_handle_reset on v2:172.21.15.152:6838/1548731833 2024-03-16T11:57:09.661 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:09.656+0000 7fd4a9a18740 1 --2- 172.21.15.29:0/2531233192 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bdfa663130 0x55bdfa6698b0 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:57:09.661 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:09.656+0000 7fd4a9a18740 1 -- 172.21.15.29:0/2531233192 >> 172.21.15.29:0/2531233192 conn(0x55bdfa65da50 msgr2=0x55bdfa65f7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:57:09.661 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:09.656+0000 7fd4a9a18740 1 -- 172.21.15.29:0/2531233192 shutdown_connections 2024-03-16T11:57:09.661 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:09.656+0000 7fd4a9a18740 1 -- 172.21.15.29:0/2531233192 wait complete. 2024-03-16T11:57:09.661 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:09.656+0000 7fd4a9a18740 20 client.15918 trim_cache size 0 max 16384 2024-03-16T11:57:09.673 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:09.672+0000 7fd4a9a18740 1 Processor -- start 2024-03-16T11:57:09.674 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:09.672+0000 7fd4a9a18740 1 -- start start 2024-03-16T11:57:09.674 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:09.673+0000 7fd4a9a18740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bdfa663130 0x55bdfa678130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:57:09.674 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:09.673+0000 7fd4a9a18740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bdfa663a50 0x55bdfa684790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:57:09.674 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:09.673+0000 7fd4a9a18740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55bdfa66c820 0x55bdfa674750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:57:09.674 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:09.673+0000 7fd4a9a18740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fd4a01afda0 con 0x55bdfa663130 2024-03-16T11:57:09.674 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:09.673+0000 7fd4a9a18740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fd4a01ad670 con 0x55bdfa66c820 2024-03-16T11:57:09.674 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:09.673+0000 7fd4a9a18740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fd4a01ab4e0 con 0x55bdfa663a50 2024-03-16T11:57:09.674 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:09.673+0000 7fd4a4bcd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bdfa663130 0x55bdfa678130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:57:09.674 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:09.673+0000 7fd49ffff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bdfa663a50 0x55bdfa684790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:57:09.674 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:09.673+0000 7fd4a4bcd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bdfa663130 0x55bdfa678130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45944/0 (socket says 172.21.15.29:45944) 2024-03-16T11:57:09.674 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:09.673+0000 7fd4a4bcd640 1 -- 172.21.15.29:0/173327191 learned_addr learned my addr 172.21.15.29:0/173327191 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:57:09.674 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:09.673+0000 7fd4a53ce640 1 --2- 172.21.15.29:0/173327191 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55bdfa66c820 0x55bdfa674750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:57:09.675 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:09.673+0000 7fd4a4bcd640 1 -- 172.21.15.29:0/173327191 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bdfa663a50 msgr2=0x55bdfa684790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:57:09.675 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:09.673+0000 7fd4a4bcd640 1 --2- 172.21.15.29:0/173327191 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bdfa663a50 0x55bdfa684790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:57:09.675 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:09.673+0000 7fd4a4bcd640 1 -- 172.21.15.29:0/173327191 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55bdfa66c820 msgr2=0x55bdfa674750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:57:09.675 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:09.673+0000 7fd4a4bcd640 1 --2- 172.21.15.29:0/173327191 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55bdfa66c820 0x55bdfa674750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:57:09.675 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:09.673+0000 7fd4a4bcd640 1 -- 172.21.15.29:0/173327191 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bdfa678670 con 0x55bdfa663130 2024-03-16T11:57:09.675 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:09.673+0000 7fd4a4bcd640 1 --2- 172.21.15.29:0/173327191 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bdfa663130 0x55bdfa678130 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7fd4940059f0 tx=0x7fd494009640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd190d919b32c36d server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:57:09.675 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:09.673+0000 7fd48a7fc640 1 -- 172.21.15.29:0/173327191 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd49400f920 con 0x55bdfa663130 2024-03-16T11:57:09.675 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:09.673+0000 7fd48a7fc640 1 -- 172.21.15.29:0/173327191 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd49400fac0 con 0x55bdfa663130 2024-03-16T11:57:09.675 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:09.674+0000 7fd48a7fc640 1 -- 172.21.15.29:0/173327191 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd494005720 con 0x55bdfa663130 2024-03-16T11:57:09.675 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:09.674+0000 7fd4a9a18740 1 -- 172.21.15.29:0/173327191 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bdfa663130 msgr2=0x55bdfa678130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:57:09.675 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:09.674+0000 7fd4a9a18740 1 --2- 172.21.15.29:0/173327191 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bdfa663130 0x55bdfa678130 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7fd4940059f0 tx=0x7fd494009640 comp rx=0 tx=0).stop 2024-03-16T11:57:09.675 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:09.674+0000 7fd4a9a18740 1 -- 172.21.15.29:0/173327191 shutdown_connections 2024-03-16T11:57:09.675 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:09.674+0000 7fd4a9a18740 1 --2- 172.21.15.29:0/173327191 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55bdfa66c820 0x55bdfa674750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:57:09.675 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:09.674+0000 7fd4a9a18740 1 --2- 172.21.15.29:0/173327191 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bdfa663a50 0x55bdfa684790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:57:09.675 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:09.674+0000 7fd4a9a18740 1 --2- 172.21.15.29:0/173327191 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bdfa663130 0x55bdfa678130 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:57:09.676 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:09.674+0000 7fd4a9a18740 1 -- 172.21.15.29:0/173327191 >> 172.21.15.29:0/173327191 conn(0x55bdfa65da50 msgr2=0x55bdfa669870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:57:09.676 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:09.674+0000 7fd4a9a18740 1 -- 172.21.15.29:0/173327191 shutdown_connections 2024-03-16T11:57:09.676 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:09.674+0000 7fd4a9a18740 1 -- 172.21.15.29:0/173327191 wait complete. 2024-03-16T11:57:09.676 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:09.675+0000 7fd4a9a18740 1 -- start start 2024-03-16T11:57:09.676 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:09.675+0000 7fd4a9a18740 1 Processor -- start 2024-03-16T11:57:09.678 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:09.675+0000 7fd4a9a18740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55bdfa663130 0x55bdfa671450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:57:09.678 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:09.675+0000 7fd4a9a18740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bdfa663a50 0x55bdfa671990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:57:09.679 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:09.675+0000 7fd4a9a18740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bdfa66c820 0x55bdfa6e2dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:57:09.679 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:09.675+0000 7fd4a9a18740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fd4a01abb30 con 0x55bdfa66c820 2024-03-16T11:57:09.679 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:09.675+0000 7fd4a9a18740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fd4a01acfa0 con 0x55bdfa663130 2024-03-16T11:57:09.679 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:09.675+0000 7fd4a9a18740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fd4a01ac200 con 0x55bdfa663a50 2024-03-16T11:57:09.679 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:09.675+0000 7fd4a9a18740 10 client.0 _ll_register_callbacks cb 0x7fd494009910 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T11:57:09.679 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:09.675+0000 7fd4a53ce640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bdfa66c820 0x55bdfa6e2dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:57:09.679 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:09.675+0000 7fd4a53ce640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bdfa66c820 0x55bdfa6e2dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45946/0 (socket says 172.21.15.29:45946) 2024-03-16T11:57:09.679 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:09.675+0000 7fd4a53ce640 1 -- 172.21.15.29:0/3627221269 learned_addr learned my addr 172.21.15.29:0/3627221269 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:57:09.679 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:09.675+0000 7fd49ffff640 1 --2- 172.21.15.29:0/3627221269 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bdfa663a50 0x55bdfa671990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:57:09.679 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:09.675+0000 7fd4a4bcd640 1 --2- 172.21.15.29:0/3627221269 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55bdfa663130 0x55bdfa671450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:57:09.679 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:09.675+0000 7fd4a53ce640 1 -- 172.21.15.29:0/3627221269 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bdfa663a50 msgr2=0x55bdfa671990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:57:09.679 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:09.675+0000 7fd4a53ce640 1 --2- 172.21.15.29:0/3627221269 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bdfa663a50 0x55bdfa671990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:57:09.679 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:09.675+0000 7fd4a53ce640 1 -- 172.21.15.29:0/3627221269 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55bdfa663130 msgr2=0x55bdfa671450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:57:09.679 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:09.675+0000 7fd4a53ce640 1 --2- 172.21.15.29:0/3627221269 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55bdfa663130 0x55bdfa671450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:57:09.680 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:09.675+0000 7fd4a53ce640 1 -- 172.21.15.29:0/3627221269 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bdfa6e3310 con 0x55bdfa66c820 2024-03-16T11:57:09.680 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:09.676+0000 7fd4a53ce640 1 --2- 172.21.15.29:0/3627221269 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bdfa66c820 0x55bdfa6e2dd0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7fd498032530 tx=0x7fd498005ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d63162432fa7a71 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:57:09.680 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:09.676+0000 7fd47ffff640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-16T11:57:09.680 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:09.676+0000 7fd47ffff640 1 -- 172.21.15.29:0/3627221269 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd498003e40 con 0x55bdfa66c820 2024-03-16T11:57:09.680 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:09.676+0000 7fd47ffff640 1 -- 172.21.15.29:0/3627221269 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8600+0+0 (secure 0 0 0) 0x7fd49804b850 con 0x55bdfa66c820 2024-03-16T11:57:09.680 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:09.676+0000 7fd4a9a18740 1 -- 172.21.15.29:0/3627221269 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd49400fac0 con 0x55bdfa66c820 2024-03-16T11:57:09.680 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:09.676+0000 7fd47ffff640 1 -- 172.21.15.29:0/3627221269 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fd49804ba50 con 0x55bdfa66c820 2024-03-16T11:57:09.680 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:09.676+0000 7fd4a9a18740 10 client.15924 fetch_fsmap learned FSMap version 24 2024-03-16T11:57:09.680 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:09.676+0000 7fd4a9a18740 1 -- 172.21.15.29:0/3627221269 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55bdfa6e44c0 con 0x55bdfa66c820 2024-03-16T11:57:09.680 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:09.676+0000 7fd47ffff640 1 -- 172.21.15.29:0/3627221269 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd49400fac0 con 0x55bdfa66c820 2024-03-16T11:57:09.680 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:09.676+0000 7fd47ffff640 1 -- 172.21.15.29:0/3627221269 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7fd49803f070 con 0x55bdfa66c820 2024-03-16T11:57:09.680 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:09.676+0000 7fd4a9a18740 10 client.15924 fetch_fsmap finished waiting for FSMap version 24 2024-03-16T11:57:09.680 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:09.676+0000 7fd4a9a18740 10 client.15924 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T11:57:09.680 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:09.676+0000 7fd4a9a18740 20 client.15924 populate_metadata read hostname 'smithi029' 2024-03-16T11:57:09.680 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:09.676+0000 7fd4a9a18740 1 --2- 172.21.15.29:0/3627221269 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55bdfa6e4800 0x55bdfa6e6bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:57:09.681 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:09.676+0000 7fd4a9a18740 4 client.15924 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-16T11:57:09.681 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:09.676+0000 7fd4a9a18740 1 -- 172.21.15.29:0/3627221269 --> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55bdfa5cf2d0 con 0x55bdfa6e4800 2024-03-16T11:57:09.681 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:09.676+0000 7fd4a4bcd640 1 --2- 172.21.15.29:0/3627221269 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55bdfa6e4800 0x55bdfa6e6bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:57:09.681 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:09.677+0000 7fd47ffff640 10 client.15924 ms_handle_connect on v2:172.21.15.152:6838/1548731833 2024-03-16T11:57:09.681 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:09.677+0000 7fd4a4bcd640 1 --2- 172.21.15.29:0/3627221269 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55bdfa6e4800 0x55bdfa6e6bf0 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=45943d46783617d9 server_cookie=8ac3a8223b9ba20 in_seq=0 out_seq=0 2024-03-16T11:57:09.788 INFO:tasks.workunit.client.0.smithi029.stdout: 1 626744 11.10 MB/sec execute 174 sec latency 112.124 ms 2024-03-16T11:57:09.846 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:09.846+0000 7fd47ffff640 1 -- 172.21.15.29:0/3627221269 <== mds.0 v2:172.21.15.152:6838/1548731833 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55bdfa5cf2d0 con 0x55bdfa6e4800 2024-03-16T11:57:09.846 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:09.846+0000 7fd47ffff640 10 client.15924 handle_command_reply: tid=0 2024-03-16T11:57:09.846 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:09.846+0000 7fd4a9a18740 1 client.15924 shutdown 2024-03-16T11:57:09.846 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:09.846+0000 7fd4a9a18740 1 -- 172.21.15.29:0/3627221269 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bdfa66c820 msgr2=0x55bdfa6e2dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:57:09.846 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:09.846+0000 7fd4a9a18740 1 --2- 172.21.15.29:0/3627221269 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bdfa66c820 0x55bdfa6e2dd0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7fd498032530 tx=0x7fd498005ef0 comp rx=0 tx=0).stop 2024-03-16T11:57:09.846 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:09.846+0000 7fd4a9a18740 1 -- 172.21.15.29:0/3627221269 shutdown_connections 2024-03-16T11:57:09.846 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:09.846+0000 7fd4a9a18740 1 --2- 172.21.15.29:0/3627221269 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bdfa66c820 0x55bdfa6e2dd0 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:57:09.847 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:09.846+0000 7fd4a9a18740 1 --2- 172.21.15.29:0/3627221269 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bdfa663a50 0x55bdfa671990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:57:09.847 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:09.846+0000 7fd4a9a18740 1 --2- 172.21.15.29:0/3627221269 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55bdfa6e4800 0x55bdfa6e6bf0 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:57:09.847 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:09.846+0000 7fd47ffff640 0 client.15924 ms_handle_reset on v2:172.21.15.152:6838/1548731833 2024-03-16T11:57:09.847 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:09.846+0000 7fd4a9a18740 1 --2- 172.21.15.29:0/3627221269 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55bdfa663130 0x55bdfa671450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:57:09.847 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:09.846+0000 7fd4a9a18740 1 -- 172.21.15.29:0/3627221269 >> 172.21.15.29:0/3627221269 conn(0x55bdfa65da50 msgr2=0x55bdfa65f5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:57:09.847 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:09.847+0000 7fd4a9a18740 1 -- 172.21.15.29:0/3627221269 shutdown_connections 2024-03-16T11:57:09.847 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:09.847+0000 7fd4a9a18740 1 -- 172.21.15.29:0/3627221269 wait complete. 2024-03-16T11:57:09.847 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:09.847+0000 7fd4a9a18740 20 client.15924 trim_cache size 0 max 16384 2024-03-16T11:57:09.847 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:{ 2024-03-16T11:57:09.847 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "message": "", 2024-03-16T11:57:09.848 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "return_code": 0 2024-03-16T11:57:09.848 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:} 2024-03-16T11:57:09.848 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:09.848+0000 7fd4a6e31640 1 -- 172.21.15.29:0/759680852 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7fd4900a2a10 msgr2=0x7fd4900a4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:57:09.848 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:09.848+0000 7fd4a6e31640 1 --2- 172.21.15.29:0/759680852 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7fd4900a2a10 0x7fd4900a4ed0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fd498047000 tx=0x7fd498049000 comp rx=0 tx=0).stop 2024-03-16T11:57:09.849 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:09.848+0000 7fd4a6e31640 1 -- 172.21.15.29:0/759680852 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd4a0101380 msgr2=0x7fd4a0198500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:57:09.849 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:09.848+0000 7fd4a6e31640 1 --2- 172.21.15.29:0/759680852 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd4a0101380 0x7fd4a0198500 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7fd48c00e9f0 tx=0x7fd48c00eeb0 comp rx=0 tx=0).stop 2024-03-16T11:57:09.849 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:09.848+0000 7fd4a6e31640 1 -- 172.21.15.29:0/759680852 shutdown_connections 2024-03-16T11:57:09.849 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:09.848+0000 7fd4a6e31640 1 --2- 172.21.15.29:0/759680852 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7fd4900a2a10 0x7fd4900a4ed0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:57:09.849 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:09.848+0000 7fd4a6e31640 1 --2- 172.21.15.29:0/759680852 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd4a0105670 0x7fd4a019c830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:57:09.849 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:09.848+0000 7fd4a6e31640 1 --2- 172.21.15.29:0/759680852 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd4a0101380 0x7fd4a0198500 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:57:09.849 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:09.848+0000 7fd4a6e31640 1 --2- 172.21.15.29:0/759680852 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd4a01009c0 0x7fd4a0197fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:57:09.849 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:09.848+0000 7fd4a6e31640 1 -- 172.21.15.29:0/759680852 >> 172.21.15.29:0/759680852 conn(0x7fd4a00776c0 msgr2=0x7fd4a006a1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:57:09.849 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:09.848+0000 7fd4a6e31640 1 -- 172.21.15.29:0/759680852 shutdown_connections 2024-03-16T11:57:09.849 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:09.848+0000 7fd4a6e31640 1 -- 172.21.15.29:0/759680852 wait complete. 2024-03-16T11:57:10.627 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:57:10 smithi152 ceph-mon[36291]: pgmap v193: 97 pgs: 97 active+clean; 70 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.0 MiB/s wr, 345 op/s 2024-03-16T11:57:10.788 INFO:tasks.workunit.client.0.smithi029.stdout: 1 628913 11.12 MB/sec execute 175 sec latency 38.735 ms 2024-03-16T11:57:10.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:57:10 smithi154 ceph-mon[36320]: pgmap v193: 97 pgs: 97 active+clean; 70 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.0 MiB/s wr, 345 op/s 2024-03-16T11:57:10.837 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:57:10 smithi029 ceph-mon[31975]: pgmap v193: 97 pgs: 97 active+clean; 70 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.0 MiB/s wr, 345 op/s 2024-03-16T11:57:11.788 INFO:tasks.workunit.client.0.smithi029.stdout: 1 630443 11.09 MB/sec execute 176 sec latency 19.151 ms 2024-03-16T11:57:12.627 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:57:12 smithi152 ceph-mon[36291]: pgmap v194: 97 pgs: 97 active+clean; 65 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 3.2 MiB/s wr, 320 op/s 2024-03-16T11:57:12.627 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:57:12 smithi152 ceph-mon[36291]: osdmap e92: 12 total, 12 up, 12 in 2024-03-16T11:57:12.788 INFO:tasks.workunit.client.0.smithi029.stdout: 1 632629 11.10 MB/sec execute 177 sec latency 13.397 ms 2024-03-16T11:57:12.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:57:12 smithi154 ceph-mon[36320]: pgmap v194: 97 pgs: 97 active+clean; 65 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 3.2 MiB/s wr, 320 op/s 2024-03-16T11:57:12.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:57:12 smithi154 ceph-mon[36320]: osdmap e92: 12 total, 12 up, 12 in 2024-03-16T11:57:12.837 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:57:12 smithi029 ceph-mon[31975]: pgmap v194: 97 pgs: 97 active+clean; 65 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 3.2 MiB/s wr, 320 op/s 2024-03-16T11:57:12.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:57:12 smithi029 ceph-mon[31975]: osdmap e92: 12 total, 12 up, 12 in 2024-03-16T11:57:13.788 INFO:tasks.workunit.client.0.smithi029.stdout: 1 633870 11.07 MB/sec execute 178 sec latency 107.897 ms 2024-03-16T11:57:14.788 INFO:tasks.workunit.client.0.smithi029.stdout: 1 636103 11.08 MB/sec execute 179 sec latency 30.046 ms 2024-03-16T11:57:14.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:57:14 smithi154 ceph-mon[36320]: pgmap v196: 97 pgs: 97 active+clean; 65 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 3.6 MiB/s wr, 358 op/s 2024-03-16T11:57:14.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:57:14 smithi029 ceph-mon[31975]: pgmap v196: 97 pgs: 97 active+clean; 65 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 3.6 MiB/s wr, 358 op/s 2024-03-16T11:57:14.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:57:14 smithi152 ceph-mon[36291]: pgmap v196: 97 pgs: 97 active+clean; 65 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 3.6 MiB/s wr, 358 op/s 2024-03-16T11:57:15.788 INFO:tasks.workunit.client.0.smithi029.stdout: 1 638044 11.06 MB/sec execute 180 sec latency 21.531 ms 2024-03-16T11:57:16.789 INFO:tasks.workunit.client.0.smithi029.stdout: 1 639865 11.07 MB/sec execute 181 sec latency 16.186 ms 2024-03-16T11:57:16.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:57:16 smithi154 ceph-mon[36320]: pgmap v197: 97 pgs: 97 active+clean; 65 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 3.2 MiB/s wr, 320 op/s 2024-03-16T11:57:16.837 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:57:16 smithi029 ceph-mon[31975]: pgmap v197: 97 pgs: 97 active+clean; 65 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 3.2 MiB/s wr, 320 op/s 2024-03-16T11:57:16.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:57:16 smithi152 ceph-mon[36291]: pgmap v197: 97 pgs: 97 active+clean; 65 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 3.2 MiB/s wr, 320 op/s 2024-03-16T11:57:17.789 INFO:tasks.workunit.client.0.smithi029.stdout: 1 641754 11.05 MB/sec execute 182 sec latency 23.734 ms 2024-03-16T11:57:18.789 INFO:tasks.workunit.client.0.smithi029.stdout: 1 643378 11.05 MB/sec execute 183 sec latency 43.574 ms 2024-03-16T11:57:18.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:57:18 smithi154 ceph-mon[36320]: pgmap v198: 97 pgs: 97 active+clean; 71 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 3.2 MiB/s wr, 315 op/s 2024-03-16T11:57:18.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:57:18 smithi029 ceph-mon[31975]: pgmap v198: 97 pgs: 97 active+clean; 71 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 3.2 MiB/s wr, 315 op/s 2024-03-16T11:57:18.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:57:18 smithi152 ceph-mon[36291]: pgmap v198: 97 pgs: 97 active+clean; 71 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 3.2 MiB/s wr, 315 op/s 2024-03-16T11:57:19.789 INFO:tasks.workunit.client.0.smithi029.stdout: 1 645254 11.03 MB/sec execute 184 sec latency 22.822 ms 2024-03-16T11:57:20.789 INFO:tasks.workunit.client.0.smithi029.stdout: 1 647035 11.03 MB/sec execute 185 sec latency 14.113 ms 2024-03-16T11:57:20.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:57:20 smithi154 ceph-mon[36320]: pgmap v199: 97 pgs: 97 active+clean; 71 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 3.2 MiB/s wr, 315 op/s 2024-03-16T11:57:20.837 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:57:20 smithi029 ceph-mon[31975]: pgmap v199: 97 pgs: 97 active+clean; 71 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 3.2 MiB/s wr, 315 op/s 2024-03-16T11:57:20.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:57:20 smithi152 ceph-mon[36291]: pgmap v199: 97 pgs: 97 active+clean; 71 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 3.2 MiB/s wr, 315 op/s 2024-03-16T11:57:21.789 INFO:tasks.workunit.client.0.smithi029.stdout: 1 649127 11.03 MB/sec execute 186 sec latency 22.584 ms 2024-03-16T11:57:22.789 INFO:tasks.workunit.client.0.smithi029.stdout: 1 650765 11.02 MB/sec execute 187 sec latency 26.176 ms 2024-03-16T11:57:22.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:57:22 smithi154 ceph-mon[36320]: pgmap v200: 97 pgs: 97 active+clean; 78 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 3.2 MiB/s wr, 211 op/s 2024-03-16T11:57:22.837 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:57:22 smithi029 ceph-mon[31975]: pgmap v200: 97 pgs: 97 active+clean; 78 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 3.2 MiB/s wr, 211 op/s 2024-03-16T11:57:22.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:57:22 smithi152 ceph-mon[36291]: pgmap v200: 97 pgs: 97 active+clean; 78 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 3.2 MiB/s wr, 211 op/s 2024-03-16T11:57:22.943 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:22.942+0000 7f9597a68640 1 -- 172.21.15.29:0/1141137301 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9590101060 msgr2=0x7f959010d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:57:22.943 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:22.942+0000 7f9597a68640 1 --2- 172.21.15.29:0/1141137301 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9590101060 0x7f959010d570 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f9588009cd0 tx=0x7f958802c0f0 comp rx=0 tx=0).stop 2024-03-16T11:57:22.943 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:22.942+0000 7f9597a68640 1 -- 172.21.15.29:0/1141137301 shutdown_connections 2024-03-16T11:57:22.943 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:22.942+0000 7f9597a68640 1 --2- 172.21.15.29:0/1141137301 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9590101060 0x7f959010d570 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:57:22.943 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:22.942+0000 7f9597a68640 1 --2- 172.21.15.29:0/1141137301 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f95901006a0 0x7f9590100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:57:22.944 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:22.942+0000 7f9597a68640 1 --2- 172.21.15.29:0/1141137301 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f95901066b0 0x7f9590106a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:57:22.944 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:22.942+0000 7f9597a68640 1 -- 172.21.15.29:0/1141137301 >> 172.21.15.29:0/1141137301 conn(0x7f9590077800 msgr2=0x7f9590077c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:57:22.944 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:22.942+0000 7f9597a68640 1 -- 172.21.15.29:0/1141137301 shutdown_connections 2024-03-16T11:57:22.944 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:22.942+0000 7f9597a68640 1 -- 172.21.15.29:0/1141137301 wait complete. 2024-03-16T11:57:22.944 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:22.943+0000 7f9597a68640 1 Processor -- start 2024-03-16T11:57:22.944 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:22.943+0000 7f9597a68640 1 -- start start 2024-03-16T11:57:22.944 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:22.943+0000 7f9597a68640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f95901006a0 0x7f9590196f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:57:22.944 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:22.943+0000 7f9597a68640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9590101060 0x7f9590197440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:57:22.944 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:22.943+0000 7f9597a68640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f95901066b0 0x7f959019b7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:57:22.944 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:22.943+0000 7f9597a68640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f9590110ac0 con 0x7f95901006a0 2024-03-16T11:57:22.944 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:22.943+0000 7f9597a68640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f9590110940 con 0x7f95901066b0 2024-03-16T11:57:22.944 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:22.943+0000 7f9597a68640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f9590110c40 con 0x7f9590101060 2024-03-16T11:57:22.944 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:22.943+0000 7f9595804640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f95901006a0 0x7f9590196f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:57:22.944 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:22.943+0000 7f9595804640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f95901006a0 0x7f9590196f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36292/0 (socket says 172.21.15.29:36292) 2024-03-16T11:57:22.945 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:22.943+0000 7f9595804640 1 -- 172.21.15.29:0/3177556338 learned_addr learned my addr 172.21.15.29:0/3177556338 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:57:22.945 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:22.943+0000 7f9595003640 1 --2- 172.21.15.29:0/3177556338 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9590101060 0x7f9590197440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:57:22.945 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:22.943+0000 7f9596005640 1 --2- 172.21.15.29:0/3177556338 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f95901066b0 0x7f959019b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:57:22.945 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:22.943+0000 7f9595804640 1 -- 172.21.15.29:0/3177556338 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9590101060 msgr2=0x7f9590197440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:57:22.945 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:22.943+0000 7f9595804640 1 --2- 172.21.15.29:0/3177556338 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9590101060 0x7f9590197440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:57:22.945 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:22.943+0000 7f9595804640 1 -- 172.21.15.29:0/3177556338 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f95901066b0 msgr2=0x7f959019b7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:57:22.945 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:22.943+0000 7f9595804640 1 --2- 172.21.15.29:0/3177556338 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f95901066b0 0x7f959019b7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:57:22.945 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:22.943+0000 7f9595804640 1 -- 172.21.15.29:0/3177556338 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f959019bef0 con 0x7f95901006a0 2024-03-16T11:57:22.945 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:22.943+0000 7f9595804640 1 --2- 172.21.15.29:0/3177556338 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f95901006a0 0x7f9590196f00 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f958c00e9f0 tx=0x7f958c00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:57:22.945 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:22.944+0000 7f9586ffd640 1 -- 172.21.15.29:0/3177556338 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f958c00cd80 con 0x7f95901006a0 2024-03-16T11:57:22.945 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:22.944+0000 7f9597a68640 1 -- 172.21.15.29:0/3177556338 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f959019c1e0 con 0x7f95901006a0 2024-03-16T11:57:22.945 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:22.944+0000 7f9586ffd640 1 -- 172.21.15.29:0/3177556338 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f958c005500 con 0x7f95901006a0 2024-03-16T11:57:22.945 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:22.944+0000 7f9586ffd640 1 -- 172.21.15.29:0/3177556338 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f958c010690 con 0x7f95901006a0 2024-03-16T11:57:22.945 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:22.944+0000 7f9597a68640 1 -- 172.21.15.29:0/3177556338 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f95901a3a70 con 0x7f95901006a0 2024-03-16T11:57:22.952 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:22.946+0000 7f9586ffd640 1 -- 172.21.15.29:0/3177556338 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7f958c0046c0 con 0x7f95901006a0 2024-03-16T11:57:22.952 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:22.946+0000 7f959a650740 1 Processor -- start 2024-03-16T11:57:22.952 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:22.946+0000 7f959a650740 1 -- start start 2024-03-16T11:57:22.952 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:22.947+0000 7f959a650740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563319b1d130 0x563319b1d510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:57:22.952 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:22.947+0000 7f959a650740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563319b1da50 0x563319b262e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:57:22.952 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:22.947+0000 7f959a650740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563319b26820 0x563319b28c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:57:22.953 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:22.947+0000 7f9586ffd640 1 --2- 172.21.15.29:0/3177556338 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f957c0a2a10 0x7f957c0a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:57:22.953 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:22.947+0000 7f959a650740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x563319a81a40 con 0x563319b1da50 2024-03-16T11:57:22.953 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:22.947+0000 7f959a650740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x563319a2a920 con 0x563319b1d130 2024-03-16T11:57:22.953 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:22.947+0000 7f959a650740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x563319a7dae0 con 0x563319b26820 2024-03-16T11:57:22.953 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:22.947+0000 7f9595804640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563319b1da50 0x563319b262e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:57:22.953 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:22.947+0000 7f9596005640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563319b1d130 0x563319b1d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:57:22.953 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:22.947+0000 7f9596005640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563319b1d130 0x563319b1d510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.29:52536/0 (socket says 172.21.15.29:52536) 2024-03-16T11:57:22.953 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:22.947+0000 7f9596005640 1 -- 172.21.15.29:0/1720308147 learned_addr learned my addr 172.21.15.29:0/1720308147 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:57:22.953 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:22.947+0000 7f9595003640 1 --2- 172.21.15.29:0/1720308147 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563319b26820 0x563319b28c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:57:22.953 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:22.947+0000 7f9596005640 1 --2- 172.21.15.29:0/3177556338 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f957c0a2a10 0x7f957c0a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:57:22.953 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:22.947+0000 7f9586ffd640 1 -- 172.21.15.29:0/3177556338 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8556+0+0 (secure 0 0 0) 0x7f958c0cf220 con 0x7f95901006a0 2024-03-16T11:57:22.953 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:22.947+0000 7f9595804640 1 -- 172.21.15.29:0/1720308147 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563319b26820 msgr2=0x563319b28c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:57:22.953 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:22.947+0000 7f9595804640 1 --2- 172.21.15.29:0/1720308147 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563319b26820 0x563319b28c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:57:22.953 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:22.947+0000 7f9595804640 1 -- 172.21.15.29:0/1720308147 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563319b1d130 msgr2=0x563319b1d510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:57:22.954 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:22.947+0000 7f9595804640 1 --2- 172.21.15.29:0/1720308147 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563319b1d130 0x563319b1d510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:57:22.954 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:22.947+0000 7f9595804640 1 -- 172.21.15.29:0/1720308147 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563319a42460 con 0x563319b1da50 2024-03-16T11:57:22.954 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:22.947+0000 7f9596005640 1 --2- 172.21.15.29:0/1720308147 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563319b1d130 0x563319b1d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:57:22.954 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:22.947+0000 7f9595003640 1 --2- 172.21.15.29:0/1720308147 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563319b26820 0x563319b28c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:57:22.954 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:22.948+0000 7f9595804640 1 --2- 172.21.15.29:0/1720308147 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563319b1da50 0x563319b262e0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f958c0cc930 tx=0x7f958c06d530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41d48ab3ca6b3c82 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:57:22.954 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:22.948+0000 7f9596005640 1 --2- 172.21.15.29:0/3177556338 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f957c0a2a10 0x7f957c0a4ed0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f9588032530 tx=0x7f9588005c50 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:57:22.954 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:22.948+0000 7f957bfff640 1 -- 172.21.15.29:0/1720308147 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f958c09b1a0 con 0x563319b1da50 2024-03-16T11:57:22.954 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:22.948+0000 7f957bfff640 1 -- 172.21.15.29:0/1720308147 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f958c09bf40 con 0x563319b1da50 2024-03-16T11:57:22.954 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:22.948+0000 7f957bfff640 1 -- 172.21.15.29:0/1720308147 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f958c0b5ed0 con 0x563319b1da50 2024-03-16T11:57:22.954 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:22.948+0000 7f959a650740 1 -- 172.21.15.29:0/1720308147 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563319b1da50 msgr2=0x563319b262e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:57:22.954 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:22.948+0000 7f959a650740 1 --2- 172.21.15.29:0/1720308147 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563319b1da50 0x563319b262e0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f958c0cc930 tx=0x7f958c06d530 comp rx=0 tx=0).stop 2024-03-16T11:57:22.954 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:22.949+0000 7f959a650740 1 -- 172.21.15.29:0/1720308147 shutdown_connections 2024-03-16T11:57:22.954 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:22.949+0000 7f959a650740 1 --2- 172.21.15.29:0/1720308147 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563319b26820 0x563319b28c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:57:22.954 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:22.949+0000 7f959a650740 1 --2- 172.21.15.29:0/1720308147 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563319b1da50 0x563319b262e0 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:57:22.954 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:22.949+0000 7f959a650740 1 --2- 172.21.15.29:0/1720308147 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563319b1d130 0x563319b1d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:57:22.955 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:22.949+0000 7f959a650740 1 -- 172.21.15.29:0/1720308147 >> 172.21.15.29:0/1720308147 conn(0x563319b17a50 msgr2=0x563319b17e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:57:22.955 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:22.949+0000 7f959a650740 1 -- 172.21.15.29:0/1720308147 shutdown_connections 2024-03-16T11:57:22.955 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:22.949+0000 7f959a650740 1 -- 172.21.15.29:0/1720308147 wait complete. 2024-03-16T11:57:22.955 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:22.949+0000 7f959a650740 1 -- start start 2024-03-16T11:57:22.955 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:22.949+0000 7f959a650740 1 Processor -- start 2024-03-16T11:57:22.955 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:22.949+0000 7f959a650740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563319b1d130 0x563319b23850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:57:22.955 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:22.949+0000 7f959a650740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563319b1da50 0x563319b23d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:57:22.955 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:22.949+0000 7f959a650740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563319b26820 0x563319b242d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:57:22.955 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:22.949+0000 7f959a650740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x563319b293f0 con 0x563319b1d130 2024-03-16T11:57:22.955 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:22.949+0000 7f959a650740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x563319a3dd70 con 0x563319b26820 2024-03-16T11:57:22.955 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:22.949+0000 7f959a650740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x563319b29570 con 0x563319b1da50 2024-03-16T11:57:22.955 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:22.950+0000 7f959a650740 10 client.0 _ll_register_callbacks cb 0x563319795a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T11:57:22.955 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:22.950+0000 7f9595804640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563319b1d130 0x563319b23850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:57:22.955 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:22.950+0000 7f9596005640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563319b26820 0x563319b242d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:57:22.956 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:22.950+0000 7f9596005640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563319b26820 0x563319b242d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.29:52538/0 (socket says 172.21.15.29:52538) 2024-03-16T11:57:22.956 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:22.950+0000 7f9595804640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563319b1d130 0x563319b23850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36314/0 (socket says 172.21.15.29:36314) 2024-03-16T11:57:22.956 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:22.950+0000 7f9596005640 1 -- 172.21.15.29:0/808605331 learned_addr learned my addr 172.21.15.29:0/808605331 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:57:22.956 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:22.950+0000 7f9595003640 1 --2- 172.21.15.29:0/808605331 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563319b1da50 0x563319b23d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:57:22.956 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:22.950+0000 7f9596005640 1 -- 172.21.15.29:0/808605331 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563319b1da50 msgr2=0x563319b23d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:57:22.956 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:22.950+0000 7f9596005640 1 --2- 172.21.15.29:0/808605331 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563319b1da50 0x563319b23d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:57:22.956 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:22.950+0000 7f9596005640 1 -- 172.21.15.29:0/808605331 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563319b1d130 msgr2=0x563319b23850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:57:22.956 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:22.950+0000 7f9596005640 1 --2- 172.21.15.29:0/808605331 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563319b1d130 0x563319b23850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:57:22.956 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:22.950+0000 7f9596005640 1 -- 172.21.15.29:0/808605331 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563319796210 con 0x563319b26820 2024-03-16T11:57:22.956 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:22.950+0000 7f9595804640 1 --2- 172.21.15.29:0/808605331 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563319b1d130 0x563319b23850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:57:22.956 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:22.950+0000 7f9596005640 1 --2- 172.21.15.29:0/808605331 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563319b26820 0x563319b242d0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f958802c630 tx=0x7f958802c0f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a88a0511d88025a0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:57:22.956 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:22.950+0000 7f95797fa640 10 client.0 ms_handle_connect on v2:172.21.15.152:3300/0 2024-03-16T11:57:22.956 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:22.950+0000 7f95797fa640 1 -- 172.21.15.29:0/808605331 <== mon.2 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9588053060 con 0x563319b26820 2024-03-16T11:57:22.956 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:22.950+0000 7f95797fa640 1 -- 172.21.15.29:0/808605331 <== mon.2 v2:172.21.15.152:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8556+0+0 (secure 0 0 0) 0x7f9588004040 con 0x563319b26820 2024-03-16T11:57:22.957 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:22.950+0000 7f959a650740 1 -- 172.21.15.29:0/808605331 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f958c09bf40 con 0x563319b26820 2024-03-16T11:57:22.957 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:22.951+0000 7f95797fa640 1 -- 172.21.15.29:0/808605331 <== mon.2 v2:172.21.15.152:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f9588039040 con 0x563319b26820 2024-03-16T11:57:22.957 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:22.951+0000 7f959a650740 10 client.25751 fetch_fsmap learned FSMap version 24 2024-03-16T11:57:22.957 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:22.951+0000 7f959a650740 1 -- 172.21.15.29:0/808605331 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x563319862620 con 0x563319b26820 2024-03-16T11:57:22.957 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:22.951+0000 7f95797fa640 1 -- 172.21.15.29:0/808605331 <== mon.2 v2:172.21.15.152:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f958c09bf40 con 0x563319b26820 2024-03-16T11:57:22.957 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:22.951+0000 7f95797fa640 1 -- 172.21.15.29:0/808605331 <== mon.2 v2:172.21.15.152:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7f958804e740 con 0x563319b26820 2024-03-16T11:57:22.957 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:22.951+0000 7f959a650740 10 client.25751 fetch_fsmap finished waiting for FSMap version 24 2024-03-16T11:57:22.957 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:22.951+0000 7f959a650740 10 client.25751 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T11:57:22.957 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:22.951+0000 7f959a650740 20 client.25751 populate_metadata read hostname 'smithi029' 2024-03-16T11:57:22.957 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:22.951+0000 7f959a650740 1 --2- 172.21.15.29:0/808605331 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x563319b36940 0x563319b38d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:57:22.957 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:22.951+0000 7f959a650740 4 client.25751 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T11:57:22.957 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:22.951+0000 7f959a650740 1 -- 172.21.15.29:0/808605331 --> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563319a8ef80 con 0x563319b36940 2024-03-16T11:57:22.957 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:22.951+0000 7f9595804640 1 --2- 172.21.15.29:0/808605331 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x563319b36940 0x563319b38d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:57:22.957 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:22.952+0000 7f9595804640 1 --2- 172.21.15.29:0/808605331 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x563319b36940 0x563319b38d50 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c3ce1a2490f79563 server_cookie=1b24647177eb7acd in_seq=0 out_seq=0 2024-03-16T11:57:22.957 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:22.952+0000 7f95797fa640 10 client.25751 ms_handle_connect on v2:172.21.15.152:6838/1548731833 2024-03-16T11:57:22.958 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:22.953+0000 7f95797fa640 1 -- 172.21.15.29:0/808605331 <== mds.0 v2:172.21.15.152:6838/1548731833 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x563319a8ef80 con 0x563319b36940 2024-03-16T11:57:22.958 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:22.953+0000 7f95797fa640 10 client.25751 handle_command_reply: tid=0 2024-03-16T11:57:22.958 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:22.953+0000 7f959a650740 1 client.25751 shutdown 2024-03-16T11:57:22.958 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:22.953+0000 7f959a650740 1 -- 172.21.15.29:0/808605331 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563319b26820 msgr2=0x563319b242d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:57:22.958 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:22.953+0000 7f959a650740 1 --2- 172.21.15.29:0/808605331 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563319b26820 0x563319b242d0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f958802c630 tx=0x7f958802c0f0 comp rx=0 tx=0).stop 2024-03-16T11:57:22.958 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:22.953+0000 7f959a650740 1 -- 172.21.15.29:0/808605331 shutdown_connections 2024-03-16T11:57:22.958 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:22.953+0000 7f959a650740 1 --2- 172.21.15.29:0/808605331 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x563319b36940 0x563319b38d50 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:57:22.958 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:22.953+0000 7f959a650740 1 --2- 172.21.15.29:0/808605331 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563319b26820 0x563319b242d0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:57:22.958 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:22.953+0000 7f959a650740 1 --2- 172.21.15.29:0/808605331 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563319b1da50 0x563319b23d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:57:22.958 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:22.953+0000 7f959a650740 1 --2- 172.21.15.29:0/808605331 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563319b1d130 0x563319b23850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:57:22.958 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:22.953+0000 7f95797fa640 0 client.25751 ms_handle_reset on v2:172.21.15.152:6838/1548731833 2024-03-16T11:57:22.958 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:22.953+0000 7f959a650740 1 -- 172.21.15.29:0/808605331 >> 172.21.15.29:0/808605331 conn(0x563319b17a50 msgr2=0x563319b197a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:57:22.958 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:22.953+0000 7f959a650740 1 -- 172.21.15.29:0/808605331 shutdown_connections 2024-03-16T11:57:22.958 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:22.953+0000 7f959a650740 1 -- 172.21.15.29:0/808605331 wait complete. 2024-03-16T11:57:22.958 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:22.953+0000 7f959a650740 20 client.25751 trim_cache size 0 max 16384 2024-03-16T11:57:22.968 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:22.968+0000 7f959a650740 1 Processor -- start 2024-03-16T11:57:22.968 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:22.968+0000 7f959a650740 1 -- start start 2024-03-16T11:57:22.969 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:22.968+0000 7f959a650740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563319b1d130 0x563319b25f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:57:22.969 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:22.968+0000 7f959a650740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563319b1da50 0x563319b32190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:57:22.969 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:22.968+0000 7f959a650740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563319b26820 0x563319b2e9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:57:22.969 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:22.968+0000 7f959a650740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f95901aecf0 con 0x563319b1da50 2024-03-16T11:57:22.969 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:22.968+0000 7f959a650740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f95901ac5c0 con 0x563319b26820 2024-03-16T11:57:22.969 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:22.968+0000 7f959a650740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f95901aa430 con 0x563319b1d130 2024-03-16T11:57:22.969 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:22.968+0000 7f9595003640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563319b1da50 0x563319b32190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:57:22.969 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:22.968+0000 7f9595003640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563319b1da50 0x563319b32190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36316/0 (socket says 172.21.15.29:36316) 2024-03-16T11:57:22.969 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:22.968+0000 7f9595003640 1 -- 172.21.15.29:0/220811294 learned_addr learned my addr 172.21.15.29:0/220811294 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:57:22.969 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:22.968+0000 7f9595804640 1 --2- 172.21.15.29:0/220811294 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563319b1d130 0x563319b25f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:57:22.969 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:22.968+0000 7f9596005640 1 --2- 172.21.15.29:0/220811294 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563319b26820 0x563319b2e9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:57:22.969 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:22.969+0000 7f9595003640 1 -- 172.21.15.29:0/220811294 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563319b1d130 msgr2=0x563319b25f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:57:22.969 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:22.969+0000 7f9595003640 1 --2- 172.21.15.29:0/220811294 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563319b1d130 0x563319b25f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:57:22.969 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:22.969+0000 7f9595003640 1 -- 172.21.15.29:0/220811294 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563319b26820 msgr2=0x563319b2e9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:57:22.969 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:22.969+0000 7f9595003640 1 --2- 172.21.15.29:0/220811294 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563319b26820 0x563319b2e9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:57:22.970 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:22.969+0000 7f9595003640 1 -- 172.21.15.29:0/220811294 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563319b28c10 con 0x563319b1da50 2024-03-16T11:57:22.970 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:22.969+0000 7f9595003640 1 --2- 172.21.15.29:0/220811294 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563319b1da50 0x563319b32190 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f9580009de0 tx=0x7f9580005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78a728b9ea2ab7c5 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:57:22.970 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:22.969+0000 7f957b7fe640 1 -- 172.21.15.29:0/220811294 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95800120d0 con 0x563319b1da50 2024-03-16T11:57:22.970 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:22.969+0000 7f957b7fe640 1 -- 172.21.15.29:0/220811294 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9580012270 con 0x563319b1da50 2024-03-16T11:57:22.970 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:22.969+0000 7f957b7fe640 1 -- 172.21.15.29:0/220811294 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9580010430 con 0x563319b1da50 2024-03-16T11:57:22.973 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:22.970+0000 7f959a650740 1 -- 172.21.15.29:0/220811294 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563319b1da50 msgr2=0x563319b32190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:57:22.973 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:22.970+0000 7f959a650740 1 --2- 172.21.15.29:0/220811294 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563319b1da50 0x563319b32190 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f9580009de0 tx=0x7f9580005f90 comp rx=0 tx=0).stop 2024-03-16T11:57:22.974 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:22.970+0000 7f959a650740 1 -- 172.21.15.29:0/220811294 shutdown_connections 2024-03-16T11:57:22.974 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:22.970+0000 7f959a650740 1 --2- 172.21.15.29:0/220811294 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563319b26820 0x563319b2e9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:57:22.974 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:22.970+0000 7f959a650740 1 --2- 172.21.15.29:0/220811294 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563319b1da50 0x563319b32190 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:57:22.974 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:22.970+0000 7f959a650740 1 --2- 172.21.15.29:0/220811294 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563319b1d130 0x563319b25f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:57:22.974 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:22.970+0000 7f959a650740 1 -- 172.21.15.29:0/220811294 >> 172.21.15.29:0/220811294 conn(0x563319b17a50 msgr2=0x563319b12a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:57:22.974 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:22.970+0000 7f959a650740 1 -- 172.21.15.29:0/220811294 shutdown_connections 2024-03-16T11:57:22.974 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:22.970+0000 7f959a650740 1 -- 172.21.15.29:0/220811294 wait complete. 2024-03-16T11:57:22.974 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:22.970+0000 7f959a650740 1 -- start start 2024-03-16T11:57:22.974 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:22.970+0000 7f959a650740 1 Processor -- start 2024-03-16T11:57:22.974 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:22.971+0000 7f959a650740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563319b1d130 0x563319b3ded0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:57:22.974 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:22.971+0000 7f959a650740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563319b1da50 0x563319b3e410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:57:22.974 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:22.971+0000 7f959a650740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563319b26820 0x563319b3e950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:57:22.974 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:22.971+0000 7f959a650740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f95901aaa80 con 0x563319b26820 2024-03-16T11:57:22.975 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:22.971+0000 7f959a650740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f95901abef0 con 0x563319b1d130 2024-03-16T11:57:22.975 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:22.971+0000 7f959a650740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f95901ab150 con 0x563319b1da50 2024-03-16T11:57:22.975 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:22.971+0000 7f959a650740 10 client.0 _ll_register_callbacks cb 0x7f958c005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T11:57:22.975 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:22.971+0000 7f9596005640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563319b26820 0x563319b3e950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:57:22.975 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:22.971+0000 7f9595003640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563319b1da50 0x563319b3e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:57:22.975 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:22.971+0000 7f9596005640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563319b26820 0x563319b3e950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36332/0 (socket says 172.21.15.29:36332) 2024-03-16T11:57:22.975 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:22.971+0000 7f9596005640 1 -- 172.21.15.29:0/315088052 learned_addr learned my addr 172.21.15.29:0/315088052 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:57:22.975 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:22.971+0000 7f9595804640 1 --2- 172.21.15.29:0/315088052 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563319b1d130 0x563319b3ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:57:22.975 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:22.971+0000 7f9596005640 1 -- 172.21.15.29:0/315088052 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563319b1da50 msgr2=0x563319b3e410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:57:22.975 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:22.971+0000 7f9596005640 1 --2- 172.21.15.29:0/315088052 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563319b1da50 0x563319b3e410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:57:22.975 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:22.971+0000 7f9596005640 1 -- 172.21.15.29:0/315088052 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563319b1d130 msgr2=0x563319b3ded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:57:22.975 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:22.971+0000 7f9596005640 1 --2- 172.21.15.29:0/315088052 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563319b1d130 0x563319b3ded0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:57:22.975 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:22.971+0000 7f9596005640 1 -- 172.21.15.29:0/315088052 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563319b9dcb0 con 0x563319b26820 2024-03-16T11:57:22.975 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:22.971+0000 7f9596005640 1 --2- 172.21.15.29:0/315088052 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563319b26820 0x563319b3e950 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f958804ecb0 tx=0x7f958804e960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31f3b6cb26fd6e37 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:57:22.976 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:22.971+0000 7f95797fa640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-16T11:57:22.976 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:22.971+0000 7f95797fa640 1 -- 172.21.15.29:0/315088052 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9588053050 con 0x563319b26820 2024-03-16T11:57:22.976 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:22.971+0000 7f95797fa640 1 -- 172.21.15.29:0/315088052 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8556+0+0 (secure 0 0 0) 0x7f95880041e0 con 0x563319b26820 2024-03-16T11:57:22.976 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:22.971+0000 7f959a650740 1 -- 172.21.15.29:0/315088052 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9580012270 con 0x563319b26820 2024-03-16T11:57:22.976 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:22.972+0000 7f95797fa640 1 -- 172.21.15.29:0/315088052 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f9588009b20 con 0x563319b26820 2024-03-16T11:57:22.976 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:22.972+0000 7f959a650740 10 client.15942 fetch_fsmap learned FSMap version 24 2024-03-16T11:57:22.976 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:22.972+0000 7f959a650740 1 -- 172.21.15.29:0/315088052 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x563319b9ef00 con 0x563319b26820 2024-03-16T11:57:22.976 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:22.972+0000 7f95797fa640 1 -- 172.21.15.29:0/315088052 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9580012270 con 0x563319b26820 2024-03-16T11:57:22.976 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:22.972+0000 7f95797fa640 1 -- 172.21.15.29:0/315088052 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7f958805f030 con 0x563319b26820 2024-03-16T11:57:22.976 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:22.972+0000 7f959a650740 10 client.15942 fetch_fsmap finished waiting for FSMap version 24 2024-03-16T11:57:22.976 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:22.972+0000 7f959a650740 10 client.15942 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T11:57:22.976 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:22.972+0000 7f959a650740 20 client.15942 populate_metadata read hostname 'smithi029' 2024-03-16T11:57:22.976 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:22.972+0000 7f959a650740 1 --2- 172.21.15.29:0/315088052 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x563319b9f660 0x563319ba1a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:57:22.976 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:22.972+0000 7f959a650740 4 client.15942 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-16T11:57:22.976 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:22.972+0000 7f959a650740 1 -- 172.21.15.29:0/315088052 --> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] -- command(tid 0: {"prefix": "flush journal"}) -- 0x563319a892d0 con 0x563319b9f660 2024-03-16T11:57:22.976 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:22.972+0000 7f9595804640 1 --2- 172.21.15.29:0/315088052 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x563319b9f660 0x563319ba1a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:57:22.977 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:22.973+0000 7f9595804640 1 --2- 172.21.15.29:0/315088052 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x563319b9f660 0x563319ba1a80 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9e10a37d347ae875 server_cookie=d755fa9f3e1d7dfa in_seq=0 out_seq=0 2024-03-16T11:57:22.977 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:22.973+0000 7f95797fa640 10 client.15942 ms_handle_connect on v2:172.21.15.152:6838/1548731833 2024-03-16T11:57:23.184 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:23.184+0000 7f95797fa640 1 -- 172.21.15.29:0/315088052 <== mds.0 v2:172.21.15.152:6838/1548731833 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x563319a892d0 con 0x563319b9f660 2024-03-16T11:57:23.184 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:23.184+0000 7f95797fa640 10 client.15942 handle_command_reply: tid=0 2024-03-16T11:57:23.184 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:23.184+0000 7f959a650740 1 client.15942 shutdown 2024-03-16T11:57:23.184 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:23.184+0000 7f959a650740 1 -- 172.21.15.29:0/315088052 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563319b26820 msgr2=0x563319b3e950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:57:23.184 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:23.184+0000 7f959a650740 1 --2- 172.21.15.29:0/315088052 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563319b26820 0x563319b3e950 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f958804ecb0 tx=0x7f958804e960 comp rx=0 tx=0).stop 2024-03-16T11:57:23.184 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:23.184+0000 7f959a650740 1 -- 172.21.15.29:0/315088052 shutdown_connections 2024-03-16T11:57:23.184 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:23.184+0000 7f959a650740 1 --2- 172.21.15.29:0/315088052 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563319b26820 0x563319b3e950 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:57:23.184 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:23.184+0000 7f959a650740 1 --2- 172.21.15.29:0/315088052 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563319b1da50 0x563319b3e410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:57:23.184 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:23.184+0000 7f959a650740 1 --2- 172.21.15.29:0/315088052 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x563319b9f660 0x563319ba1a80 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:57:23.184 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:23.184+0000 7f95797fa640 0 client.15942 ms_handle_reset on v2:172.21.15.152:6838/1548731833 2024-03-16T11:57:23.185 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:23.184+0000 7f959a650740 1 --2- 172.21.15.29:0/315088052 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563319b1d130 0x563319b3ded0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:57:23.185 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:23.184+0000 7f959a650740 1 -- 172.21.15.29:0/315088052 >> 172.21.15.29:0/315088052 conn(0x563319b17a50 msgr2=0x563319b23810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:57:23.185 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:23.184+0000 7f959a650740 1 -- 172.21.15.29:0/315088052 shutdown_connections 2024-03-16T11:57:23.185 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:23.184+0000 7f959a650740 1 -- 172.21.15.29:0/315088052 wait complete. 2024-03-16T11:57:23.185 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:23.184+0000 7f959a650740 20 client.15942 trim_cache size 0 max 16384 2024-03-16T11:57:23.185 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:{ 2024-03-16T11:57:23.185 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "message": "", 2024-03-16T11:57:23.185 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "return_code": 0 2024-03-16T11:57:23.185 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:} 2024-03-16T11:57:23.186 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:23.185+0000 7f9597a68640 1 -- 172.21.15.29:0/3177556338 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f957c0a2a10 msgr2=0x7f957c0a4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:57:23.186 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:23.185+0000 7f9597a68640 1 --2- 172.21.15.29:0/3177556338 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f957c0a2a10 0x7f957c0a4ed0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f9588032530 tx=0x7f9588005c50 comp rx=0 tx=0).stop 2024-03-16T11:57:23.186 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:23.185+0000 7f9597a68640 1 -- 172.21.15.29:0/3177556338 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f95901006a0 msgr2=0x7f9590196f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:57:23.186 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:23.185+0000 7f9597a68640 1 --2- 172.21.15.29:0/3177556338 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f95901006a0 0x7f9590196f00 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f958c00e9f0 tx=0x7f958c00edf0 comp rx=0 tx=0).stop 2024-03-16T11:57:23.186 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:23.185+0000 7f9597a68640 1 -- 172.21.15.29:0/3177556338 shutdown_connections 2024-03-16T11:57:23.186 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:23.185+0000 7f9597a68640 1 --2- 172.21.15.29:0/3177556338 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f957c0a2a10 0x7f957c0a4ed0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:57:23.186 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:23.185+0000 7f9597a68640 1 --2- 172.21.15.29:0/3177556338 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f95901066b0 0x7f959019b7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:57:23.187 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:23.185+0000 7f9597a68640 1 --2- 172.21.15.29:0/3177556338 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9590101060 0x7f9590197440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:57:23.187 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:23.185+0000 7f9597a68640 1 --2- 172.21.15.29:0/3177556338 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f95901006a0 0x7f9590196f00 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:57:23.187 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:23.185+0000 7f9597a68640 1 -- 172.21.15.29:0/3177556338 >> 172.21.15.29:0/3177556338 conn(0x7f9590077800 msgr2=0x7f9590103270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:57:23.187 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:23.185+0000 7f9597a68640 1 -- 172.21.15.29:0/3177556338 shutdown_connections 2024-03-16T11:57:23.187 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:23.185+0000 7f9597a68640 1 -- 172.21.15.29:0/3177556338 wait complete. 2024-03-16T11:57:23.789 INFO:tasks.workunit.client.0.smithi029.stdout: 1 651982 10.99 MB/sec execute 188 sec latency 109.336 ms 2024-03-16T11:57:24.789 INFO:tasks.workunit.client.0.smithi029.stdout: 1 653895 11.00 MB/sec execute 189 sec latency 18.218 ms 2024-03-16T11:57:24.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:57:24 smithi154 ceph-mon[36320]: pgmap v201: 97 pgs: 97 active+clean; 78 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.0 MiB/s wr, 194 op/s 2024-03-16T11:57:24.837 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:57:24 smithi029 ceph-mon[31975]: pgmap v201: 97 pgs: 97 active+clean; 78 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.0 MiB/s wr, 194 op/s 2024-03-16T11:57:24.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:57:24 smithi152 ceph-mon[36291]: pgmap v201: 97 pgs: 97 active+clean; 78 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.0 MiB/s wr, 194 op/s 2024-03-16T11:57:25.789 INFO:tasks.workunit.client.0.smithi029.stdout: 1 655808 10.98 MB/sec execute 190 sec latency 19.814 ms 2024-03-16T11:57:26.789 INFO:tasks.workunit.client.0.smithi029.stdout: 1 657576 10.99 MB/sec execute 191 sec latency 16.970 ms 2024-03-16T11:57:26.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:57:26 smithi154 ceph-mon[36320]: pgmap v202: 97 pgs: 97 active+clean; 78 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 2.7 MiB/s wr, 176 op/s 2024-03-16T11:57:26.837 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:57:26 smithi029 ceph-mon[31975]: pgmap v202: 97 pgs: 97 active+clean; 78 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 2.7 MiB/s wr, 176 op/s 2024-03-16T11:57:26.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:57:26 smithi152 ceph-mon[36291]: pgmap v202: 97 pgs: 97 active+clean; 78 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 2.7 MiB/s wr, 176 op/s 2024-03-16T11:57:27.789 INFO:tasks.workunit.client.0.smithi029.stdout: 1 659536 10.97 MB/sec execute 192 sec latency 24.047 ms 2024-03-16T11:57:28.789 INFO:tasks.workunit.client.0.smithi029.stdout: 1 661211 10.97 MB/sec execute 193 sec latency 85.138 ms 2024-03-16T11:57:28.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:57:28 smithi154 ceph-mon[36320]: pgmap v203: 97 pgs: 97 active+clean; 67 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 4.0 MiB/s wr, 360 op/s 2024-03-16T11:57:28.837 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:57:28 smithi029 ceph-mon[31975]: pgmap v203: 97 pgs: 97 active+clean; 67 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 4.0 MiB/s wr, 360 op/s 2024-03-16T11:57:28.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:57:28 smithi152 ceph-mon[36291]: pgmap v203: 97 pgs: 97 active+clean; 67 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 4.0 MiB/s wr, 360 op/s 2024-03-16T11:57:29.789 INFO:tasks.workunit.client.0.smithi029.stdout: 1 663291 10.97 MB/sec execute 194 sec latency 28.954 ms 2024-03-16T11:57:30.789 INFO:tasks.workunit.client.0.smithi029.stdout: 1 664986 10.96 MB/sec execute 195 sec latency 23.717 ms 2024-03-16T11:57:30.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:57:30 smithi154 ceph-mon[36320]: pgmap v204: 97 pgs: 97 active+clean; 67 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 2.7 MiB/s wr, 274 op/s 2024-03-16T11:57:30.837 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:57:30 smithi029 ceph-mon[31975]: pgmap v204: 97 pgs: 97 active+clean; 67 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 2.7 MiB/s wr, 274 op/s 2024-03-16T11:57:30.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:57:30 smithi152 ceph-mon[36291]: pgmap v204: 97 pgs: 97 active+clean; 67 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 2.7 MiB/s wr, 274 op/s 2024-03-16T11:57:31.789 INFO:tasks.workunit.client.0.smithi029.stdout: 1 667076 10.97 MB/sec execute 196 sec latency 41.821 ms 2024-03-16T11:57:32.789 INFO:tasks.workunit.client.0.smithi029.stdout: 1 668786 10.95 MB/sec execute 197 sec latency 22.991 ms 2024-03-16T11:57:32.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:57:32 smithi154 ceph-mon[36320]: pgmap v205: 97 pgs: 97 active+clean; 73 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 4.1 MiB/s wr, 364 op/s 2024-03-16T11:57:32.837 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:57:32 smithi029 ceph-mon[31975]: pgmap v205: 97 pgs: 97 active+clean; 73 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 4.1 MiB/s wr, 364 op/s 2024-03-16T11:57:32.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:57:32 smithi152 ceph-mon[36291]: pgmap v205: 97 pgs: 97 active+clean; 73 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 4.1 MiB/s wr, 364 op/s 2024-03-16T11:57:33.789 INFO:tasks.workunit.client.0.smithi029.stdout: 1 670809 10.96 MB/sec execute 198 sec latency 49.952 ms 2024-03-16T11:57:34.789 INFO:tasks.workunit.client.0.smithi029.stdout: 1 672152 10.93 MB/sec execute 199 sec latency 20.561 ms 2024-03-16T11:57:34.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:57:34 smithi154 ceph-mon[36320]: pgmap v206: 97 pgs: 97 active+clean; 73 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 2.7 MiB/s wr, 273 op/s 2024-03-16T11:57:34.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:57:34 smithi029 ceph-mon[31975]: pgmap v206: 97 pgs: 97 active+clean; 73 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 2.7 MiB/s wr, 273 op/s 2024-03-16T11:57:34.878 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:57:34 smithi152 ceph-mon[36291]: pgmap v206: 97 pgs: 97 active+clean; 73 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 2.7 MiB/s wr, 273 op/s 2024-03-16T11:57:35.789 INFO:tasks.workunit.client.0.smithi029.stdout: 1 674575 10.95 MB/sec execute 200 sec latency 18.897 ms 2024-03-16T11:57:36.278 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:36.276+0000 7f9f763c6640 1 -- 172.21.15.29:0/2454445517 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9f701077c0 msgr2=0x7f9f70107ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:57:36.278 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:36.276+0000 7f9f763c6640 1 --2- 172.21.15.29:0/2454445517 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9f701077c0 0x7f9f70107ba0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f9f60009930 tx=0x7f9f6002bc30 comp rx=0 tx=0).stop 2024-03-16T11:57:36.278 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:36.276+0000 7f9f763c6640 1 -- 172.21.15.29:0/2454445517 shutdown_connections 2024-03-16T11:57:36.278 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:36.276+0000 7f9f763c6640 1 --2- 172.21.15.29:0/2454445517 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9f70102170 0x7f9f7010e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:57:36.278 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:36.276+0000 7f9f763c6640 1 --2- 172.21.15.29:0/2454445517 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9f701017b0 0x7f9f70101c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:57:36.279 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:36.276+0000 7f9f763c6640 1 --2- 172.21.15.29:0/2454445517 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9f701077c0 0x7f9f70107ba0 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:57:36.279 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:36.276+0000 7f9f763c6640 1 -- 172.21.15.29:0/2454445517 >> 172.21.15.29:0/2454445517 conn(0x7f9f700fd2c0 msgr2=0x7f9f700ff6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:57:36.279 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:36.276+0000 7f9f763c6640 1 -- 172.21.15.29:0/2454445517 shutdown_connections 2024-03-16T11:57:36.279 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:36.276+0000 7f9f763c6640 1 -- 172.21.15.29:0/2454445517 wait complete. 2024-03-16T11:57:36.279 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:36.276+0000 7f9f763c6640 1 Processor -- start 2024-03-16T11:57:36.279 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:36.276+0000 7f9f763c6640 1 -- start start 2024-03-16T11:57:36.279 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:36.277+0000 7f9f763c6640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9f701017b0 0x7f9f70198030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:57:36.279 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:36.277+0000 7f9f763c6640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9f70102170 0x7f9f70198570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:57:36.279 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:36.277+0000 7f9f763c6640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9f701077c0 0x7f9f7019c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:57:36.279 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:36.277+0000 7f9f763c6640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f9f70111bd0 con 0x7f9f701077c0 2024-03-16T11:57:36.279 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:36.277+0000 7f9f763c6640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f9f70111a50 con 0x7f9f701017b0 2024-03-16T11:57:36.279 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:36.277+0000 7f9f763c6640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f9f70111d50 con 0x7f9f70102170 2024-03-16T11:57:36.279 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:36.277+0000 7f9f74963640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9f701077c0 0x7f9f7019c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:57:36.279 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:36.277+0000 7f9f6ffff640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9f701017b0 0x7f9f70198030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:57:36.279 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:36.277+0000 7f9f74963640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9f701077c0 0x7f9f7019c8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41616/0 (socket says 172.21.15.29:41616) 2024-03-16T11:57:36.280 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:36.277+0000 7f9f74963640 1 -- 172.21.15.29:0/224337734 learned_addr learned my addr 172.21.15.29:0/224337734 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:57:36.280 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:36.277+0000 7f9f6f7fe640 1 --2- 172.21.15.29:0/224337734 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9f70102170 0x7f9f70198570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:57:36.280 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:36.277+0000 7f9f74963640 1 -- 172.21.15.29:0/224337734 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9f70102170 msgr2=0x7f9f70198570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:57:36.280 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:36.277+0000 7f9f74963640 1 --2- 172.21.15.29:0/224337734 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9f70102170 0x7f9f70198570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:57:36.280 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:36.277+0000 7f9f74963640 1 -- 172.21.15.29:0/224337734 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9f701017b0 msgr2=0x7f9f70198030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:57:36.280 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:36.277+0000 7f9f74963640 1 --2- 172.21.15.29:0/224337734 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9f701017b0 0x7f9f70198030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:57:36.280 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:36.277+0000 7f9f74963640 1 -- 172.21.15.29:0/224337734 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f7019d020 con 0x7f9f701077c0 2024-03-16T11:57:36.280 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:36.277+0000 7f9f74963640 1 --2- 172.21.15.29:0/224337734 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9f701077c0 0x7f9f7019c8f0 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f9f6800eb10 tx=0x7f9f6800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:57:36.280 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:36.278+0000 7f9f6d7fa640 1 -- 172.21.15.29:0/224337734 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f68004b20 con 0x7f9f701077c0 2024-03-16T11:57:36.280 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:36.278+0000 7f9f6d7fa640 1 -- 172.21.15.29:0/224337734 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9f68004cc0 con 0x7f9f701077c0 2024-03-16T11:57:36.280 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:36.278+0000 7f9f763c6640 1 -- 172.21.15.29:0/224337734 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9f7019d310 con 0x7f9f701077c0 2024-03-16T11:57:36.280 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:36.278+0000 7f9f6d7fa640 1 -- 172.21.15.29:0/224337734 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f68010760 con 0x7f9f701077c0 2024-03-16T11:57:36.280 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:36.278+0000 7f9f763c6640 1 -- 172.21.15.29:0/224337734 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9f701a4ba0 con 0x7f9f701077c0 2024-03-16T11:57:36.285 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:36.280+0000 7f9f6d7fa640 1 -- 172.21.15.29:0/224337734 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7f9f68005020 con 0x7f9f701077c0 2024-03-16T11:57:36.285 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:36.280+0000 7f9f78fb4740 1 Processor -- start 2024-03-16T11:57:36.286 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:36.280+0000 7f9f78fb4740 1 -- start start 2024-03-16T11:57:36.286 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:36.280+0000 7f9f78fb4740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ac7b306130 0x55ac7b306510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:57:36.286 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:36.280+0000 7f9f78fb4740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ac7b306a50 0x55ac7b30f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:57:36.286 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:36.280+0000 7f9f78fb4740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ac7b30f820 0x55ac7b311c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:57:36.286 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:36.280+0000 7f9f78fb4740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55ac7b26aa40 con 0x55ac7b306a50 2024-03-16T11:57:36.286 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:36.280+0000 7f9f78fb4740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x55ac7b213920 con 0x55ac7b30f820 2024-03-16T11:57:36.286 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:36.280+0000 7f9f78fb4740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55ac7b266ae0 con 0x55ac7b306130 2024-03-16T11:57:36.286 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:36.280+0000 7f9f6d7fa640 1 --2- 172.21.15.29:0/224337734 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f9f580a2a10 0x7f9f580a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:57:36.286 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:36.280+0000 7f9f6ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ac7b306a50 0x55ac7b30f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:57:36.286 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:36.280+0000 7f9f6f7fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ac7b306130 0x55ac7b306510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:57:36.286 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:36.280+0000 7f9f6ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ac7b306a50 0x55ac7b30f2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41630/0 (socket says 172.21.15.29:41630) 2024-03-16T11:57:36.286 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:36.280+0000 7f9f6ffff640 1 -- 172.21.15.29:0/380780180 learned_addr learned my addr 172.21.15.29:0/380780180 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:57:36.286 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:36.281+0000 7f9f6f7fe640 1 --2- 172.21.15.29:0/380780180 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ac7b30f820 0x55ac7b311c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:57:36.286 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:36.281+0000 7f9f74963640 1 --2- 172.21.15.29:0/224337734 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f9f580a2a10 0x7f9f580a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:57:36.287 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:36.281+0000 7f9f6d7fa640 1 -- 172.21.15.29:0/224337734 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8556+0+0 (secure 0 0 0) 0x7f9f680cf4d0 con 0x7f9f701077c0 2024-03-16T11:57:36.287 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:36.281+0000 7f9f6f7fe640 1 -- 172.21.15.29:0/380780180 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ac7b30f820 msgr2=0x55ac7b311c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:57:36.287 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:36.281+0000 7f9f6f7fe640 1 --2- 172.21.15.29:0/380780180 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ac7b30f820 0x55ac7b311c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:57:36.287 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:36.281+0000 7f9f6f7fe640 1 -- 172.21.15.29:0/380780180 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ac7b306a50 msgr2=0x55ac7b30f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:57:36.287 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:36.281+0000 7f9f6f7fe640 1 --2- 172.21.15.29:0/380780180 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ac7b306a50 0x55ac7b30f2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:57:36.287 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:36.281+0000 7f9f6f7fe640 1 -- 172.21.15.29:0/380780180 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ac7b22b460 con 0x55ac7b306130 2024-03-16T11:57:36.287 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:36.281+0000 7f9f6ffff640 1 --2- 172.21.15.29:0/380780180 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ac7b306a50 0x55ac7b30f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:57:36.287 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:36.281+0000 7f9f74963640 1 --2- 172.21.15.29:0/224337734 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f9f580a2a10 0x7f9f580a4ed0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f9f6809b810 tx=0x7f9f6800efd0 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:57:36.287 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:36.281+0000 7f9f6f7fe640 1 --2- 172.21.15.29:0/380780180 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ac7b306130 0x55ac7b306510 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f9f64009fd0 tx=0x7f9f640096c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2b53119388023ee0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:57:36.287 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:36.281+0000 7f9f567fc640 1 -- 172.21.15.29:0/380780180 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f64004050 con 0x55ac7b306130 2024-03-16T11:57:36.287 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:36.281+0000 7f9f567fc640 1 -- 172.21.15.29:0/380780180 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9f64005500 con 0x55ac7b306130 2024-03-16T11:57:36.287 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:36.281+0000 7f9f567fc640 1 -- 172.21.15.29:0/380780180 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f64004ca0 con 0x55ac7b306130 2024-03-16T11:57:36.287 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:36.282+0000 7f9f78fb4740 1 -- 172.21.15.29:0/380780180 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ac7b306130 msgr2=0x55ac7b306510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:57:36.287 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:36.282+0000 7f9f78fb4740 1 --2- 172.21.15.29:0/380780180 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ac7b306130 0x55ac7b306510 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f9f64009fd0 tx=0x7f9f640096c0 comp rx=0 tx=0).stop 2024-03-16T11:57:36.288 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:36.282+0000 7f9f78fb4740 1 -- 172.21.15.29:0/380780180 shutdown_connections 2024-03-16T11:57:36.288 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:36.282+0000 7f9f78fb4740 1 --2- 172.21.15.29:0/380780180 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ac7b30f820 0x55ac7b311c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:57:36.288 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:36.282+0000 7f9f78fb4740 1 --2- 172.21.15.29:0/380780180 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ac7b306a50 0x55ac7b30f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:57:36.288 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:36.282+0000 7f9f78fb4740 1 --2- 172.21.15.29:0/380780180 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ac7b306130 0x55ac7b306510 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:57:36.288 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:36.282+0000 7f9f78fb4740 1 -- 172.21.15.29:0/380780180 >> 172.21.15.29:0/380780180 conn(0x55ac7b300a50 msgr2=0x55ac7b300e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:57:36.288 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:36.282+0000 7f9f78fb4740 1 -- 172.21.15.29:0/380780180 shutdown_connections 2024-03-16T11:57:36.288 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:36.282+0000 7f9f78fb4740 1 -- 172.21.15.29:0/380780180 wait complete. 2024-03-16T11:57:36.288 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:36.282+0000 7f9f78fb4740 1 -- start start 2024-03-16T11:57:36.288 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:36.283+0000 7f9f78fb4740 1 Processor -- start 2024-03-16T11:57:36.288 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:36.283+0000 7f9f78fb4740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ac7b306130 0x55ac7b30c780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:57:36.288 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:36.283+0000 7f9f78fb4740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ac7b306a50 0x55ac7b30ccc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:57:36.288 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:36.283+0000 7f9f78fb4740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ac7b30f820 0x55ac7b30d200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:57:36.288 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:36.283+0000 7f9f78fb4740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55ac7b312450 con 0x55ac7b306130 2024-03-16T11:57:36.288 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:36.283+0000 7f9f78fb4740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x55ac7b226d70 con 0x55ac7b30f820 2024-03-16T11:57:36.288 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:36.283+0000 7f9f78fb4740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55ac7b3125d0 con 0x55ac7b306a50 2024-03-16T11:57:36.289 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:36.283+0000 7f9f78fb4740 10 client.0 _ll_register_callbacks cb 0x55ac7af7ea00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T11:57:36.289 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:36.283+0000 7f9f6f7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ac7b306130 0x55ac7b30c780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:57:36.289 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:36.283+0000 7f9f6f7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ac7b306130 0x55ac7b30c780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41632/0 (socket says 172.21.15.29:41632) 2024-03-16T11:57:36.289 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:36.283+0000 7f9f6f7fe640 1 -- 172.21.15.29:0/4208502703 learned_addr learned my addr 172.21.15.29:0/4208502703 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:57:36.289 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:36.283+0000 7f9f6ffff640 1 --2- 172.21.15.29:0/4208502703 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ac7b306a50 0x55ac7b30ccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:57:36.289 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:36.283+0000 7f9f6f7fe640 1 --2- 172.21.15.29:0/4208502703 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ac7b30f820 0x55ac7b30d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:57:36.289 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:36.283+0000 7f9f6f7fe640 1 -- 172.21.15.29:0/4208502703 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ac7b306a50 msgr2=0x55ac7b30ccc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:57:36.289 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:36.283+0000 7f9f6f7fe640 1 --2- 172.21.15.29:0/4208502703 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ac7b306a50 0x55ac7b30ccc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:57:36.289 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:36.283+0000 7f9f6f7fe640 1 -- 172.21.15.29:0/4208502703 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ac7b30f820 msgr2=0x55ac7b30d200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:57:36.289 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:36.283+0000 7f9f6f7fe640 1 --2- 172.21.15.29:0/4208502703 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ac7b30f820 0x55ac7b30d200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:57:36.289 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:36.283+0000 7f9f6f7fe640 1 -- 172.21.15.29:0/4208502703 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ac7af7f210 con 0x55ac7b306130 2024-03-16T11:57:36.289 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:36.283+0000 7f9f6ffff640 1 --2- 172.21.15.29:0/4208502703 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ac7b306a50 0x55ac7b30ccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:57:36.289 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:36.283+0000 7f9f6f7fe640 1 --2- 172.21.15.29:0/4208502703 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ac7b306130 0x55ac7b30c780 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f9f640058c0 tx=0x7f9f64005850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=875b7cdb7158c056 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:57:36.289 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:36.283+0000 7f9f4f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-16T11:57:36.289 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:36.284+0000 7f9f4f7fe640 1 -- 172.21.15.29:0/4208502703 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f64004d80 con 0x55ac7b306130 2024-03-16T11:57:36.290 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:36.284+0000 7f9f4f7fe640 1 -- 172.21.15.29:0/4208502703 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8556+0+0 (secure 0 0 0) 0x7f9f6400f790 con 0x55ac7b306130 2024-03-16T11:57:36.290 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:36.284+0000 7f9f78fb4740 1 -- 172.21.15.29:0/4208502703 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9f64005500 con 0x55ac7b306130 2024-03-16T11:57:36.290 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:36.284+0000 7f9f4f7fe640 1 -- 172.21.15.29:0/4208502703 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f9f6400f990 con 0x55ac7b306130 2024-03-16T11:57:36.290 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:36.284+0000 7f9f78fb4740 10 client.15954 fetch_fsmap learned FSMap version 24 2024-03-16T11:57:36.290 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:36.284+0000 7f9f78fb4740 1 -- 172.21.15.29:0/4208502703 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55ac7b04b620 con 0x55ac7b306130 2024-03-16T11:57:36.290 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:36.284+0000 7f9f4f7fe640 1 -- 172.21.15.29:0/4208502703 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9f64005500 con 0x55ac7b306130 2024-03-16T11:57:36.290 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:36.284+0000 7f9f4f7fe640 1 -- 172.21.15.29:0/4208502703 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7f9f640369b0 con 0x55ac7b306130 2024-03-16T11:57:36.290 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:36.284+0000 7f9f78fb4740 10 client.15954 fetch_fsmap finished waiting for FSMap version 24 2024-03-16T11:57:36.290 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:36.284+0000 7f9f78fb4740 10 client.15954 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T11:57:36.290 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:36.284+0000 7f9f78fb4740 20 client.15954 populate_metadata read hostname 'smithi029' 2024-03-16T11:57:36.290 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:36.284+0000 7f9f78fb4740 1 --2- 172.21.15.29:0/4208502703 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55ac7b31f7d0 0x55ac7b321bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:57:36.290 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:36.284+0000 7f9f78fb4740 4 client.15954 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T11:57:36.290 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:36.284+0000 7f9f78fb4740 1 -- 172.21.15.29:0/4208502703 --> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ac7b277f80 con 0x55ac7b31f7d0 2024-03-16T11:57:36.290 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:36.285+0000 7f9f6ffff640 1 --2- 172.21.15.29:0/4208502703 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55ac7b31f7d0 0x55ac7b321bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:57:36.290 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:36.285+0000 7f9f6ffff640 1 --2- 172.21.15.29:0/4208502703 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55ac7b31f7d0 0x55ac7b321bc0 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d5e1435b1898752e server_cookie=8a94f71dc985e0b8 in_seq=0 out_seq=0 2024-03-16T11:57:36.291 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:36.285+0000 7f9f4f7fe640 10 client.15954 ms_handle_connect on v2:172.21.15.152:6838/1548731833 2024-03-16T11:57:36.292 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:36.286+0000 7f9f4f7fe640 1 -- 172.21.15.29:0/4208502703 <== mds.0 v2:172.21.15.152:6838/1548731833 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55ac7b277f80 con 0x55ac7b31f7d0 2024-03-16T11:57:36.292 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:36.286+0000 7f9f4f7fe640 10 client.15954 handle_command_reply: tid=0 2024-03-16T11:57:36.292 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:36.286+0000 7f9f78fb4740 1 client.15954 shutdown 2024-03-16T11:57:36.292 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:36.286+0000 7f9f78fb4740 1 -- 172.21.15.29:0/4208502703 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ac7b306130 msgr2=0x55ac7b30c780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:57:36.292 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:36.286+0000 7f9f78fb4740 1 --2- 172.21.15.29:0/4208502703 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ac7b306130 0x55ac7b30c780 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f9f640058c0 tx=0x7f9f64005850 comp rx=0 tx=0).stop 2024-03-16T11:57:36.292 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:36.286+0000 7f9f78fb4740 1 -- 172.21.15.29:0/4208502703 shutdown_connections 2024-03-16T11:57:36.292 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:36.286+0000 7f9f78fb4740 1 --2- 172.21.15.29:0/4208502703 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55ac7b31f7d0 0x55ac7b321bc0 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:57:36.292 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:36.286+0000 7f9f78fb4740 1 --2- 172.21.15.29:0/4208502703 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ac7b30f820 0x55ac7b30d200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:57:36.292 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:36.286+0000 7f9f78fb4740 1 --2- 172.21.15.29:0/4208502703 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ac7b306a50 0x55ac7b30ccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:57:36.292 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:36.286+0000 7f9f4f7fe640 0 client.15954 ms_handle_reset on v2:172.21.15.152:6838/1548731833 2024-03-16T11:57:36.293 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:36.286+0000 7f9f78fb4740 1 --2- 172.21.15.29:0/4208502703 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ac7b306130 0x55ac7b30c780 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:57:36.293 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:36.286+0000 7f9f78fb4740 1 -- 172.21.15.29:0/4208502703 >> 172.21.15.29:0/4208502703 conn(0x55ac7b300a50 msgr2=0x55ac7b3027a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:57:36.293 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:36.287+0000 7f9f78fb4740 1 -- 172.21.15.29:0/4208502703 shutdown_connections 2024-03-16T11:57:36.293 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:36.287+0000 7f9f78fb4740 1 -- 172.21.15.29:0/4208502703 wait complete. 2024-03-16T11:57:36.293 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:36.287+0000 7f9f78fb4740 20 client.15954 trim_cache size 0 max 16384 2024-03-16T11:57:36.306 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:36.302+0000 7f9f78fb4740 1 Processor -- start 2024-03-16T11:57:36.306 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:36.302+0000 7f9f78fb4740 1 -- start start 2024-03-16T11:57:36.306 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:36.302+0000 7f9f78fb4740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ac7b306130 0x55ac7b30eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:57:36.306 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:36.302+0000 7f9f78fb4740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ac7b306a50 0x55ac7b31b220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:57:36.306 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:36.302+0000 7f9f78fb4740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ac7b30f820 0x55ac7b30af20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:57:36.306 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:36.302+0000 7f9f78fb4740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f9f701afe50 con 0x55ac7b30f820 2024-03-16T11:57:36.306 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:36.302+0000 7f9f78fb4740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f9f701ad720 con 0x55ac7b306a50 2024-03-16T11:57:36.306 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:36.302+0000 7f9f78fb4740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f9f701ab590 con 0x55ac7b306130 2024-03-16T11:57:36.306 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:36.302+0000 7f9f6f7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ac7b30f820 0x55ac7b30af20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:57:36.307 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:36.302+0000 7f9f6ffff640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ac7b306a50 0x55ac7b31b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:57:36.307 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:36.302+0000 7f9f6f7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ac7b30f820 0x55ac7b30af20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41640/0 (socket says 172.21.15.29:41640) 2024-03-16T11:57:36.307 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:36.302+0000 7f9f6f7fe640 1 -- 172.21.15.29:0/1748353744 learned_addr learned my addr 172.21.15.29:0/1748353744 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:57:36.307 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:36.302+0000 7f9f6f7fe640 1 --2- 172.21.15.29:0/1748353744 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ac7b306130 0x55ac7b30eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:57:36.307 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:36.302+0000 7f9f6f7fe640 1 -- 172.21.15.29:0/1748353744 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ac7b306130 msgr2=0x55ac7b30eeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:57:36.307 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:36.302+0000 7f9f6f7fe640 1 --2- 172.21.15.29:0/1748353744 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ac7b306130 0x55ac7b30eeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:57:36.307 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:36.302+0000 7f9f6f7fe640 1 -- 172.21.15.29:0/1748353744 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ac7b306a50 msgr2=0x55ac7b31b220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:57:36.307 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:36.302+0000 7f9f6f7fe640 1 --2- 172.21.15.29:0/1748353744 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ac7b306a50 0x55ac7b31b220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:57:36.307 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:36.302+0000 7f9f6f7fe640 1 -- 172.21.15.29:0/1748353744 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ac7b300880 con 0x55ac7b30f820 2024-03-16T11:57:36.307 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:36.302+0000 7f9f6f7fe640 1 --2- 172.21.15.29:0/1748353744 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ac7b30f820 0x55ac7b30af20 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f9f64006010 tx=0x7f9f640096c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e89774c0bc8b088 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:57:36.307 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:36.303+0000 7f9f55ffb640 1 -- 172.21.15.29:0/1748353744 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f6403c070 con 0x55ac7b30f820 2024-03-16T11:57:36.307 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:36.303+0000 7f9f55ffb640 1 -- 172.21.15.29:0/1748353744 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9f6400f790 con 0x55ac7b30f820 2024-03-16T11:57:36.307 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:36.303+0000 7f9f55ffb640 1 -- 172.21.15.29:0/1748353744 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f640300b0 con 0x55ac7b30f820 2024-03-16T11:57:36.307 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:36.303+0000 7f9f78fb4740 1 -- 172.21.15.29:0/1748353744 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ac7b30f820 msgr2=0x55ac7b30af20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:57:36.307 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:36.303+0000 7f9f78fb4740 1 --2- 172.21.15.29:0/1748353744 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ac7b30f820 0x55ac7b30af20 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f9f64006010 tx=0x7f9f640096c0 comp rx=0 tx=0).stop 2024-03-16T11:57:36.308 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:36.303+0000 7f9f78fb4740 1 -- 172.21.15.29:0/1748353744 shutdown_connections 2024-03-16T11:57:36.308 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:36.303+0000 7f9f78fb4740 1 --2- 172.21.15.29:0/1748353744 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ac7b30f820 0x55ac7b30af20 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:57:36.308 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:36.303+0000 7f9f78fb4740 1 --2- 172.21.15.29:0/1748353744 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ac7b306a50 0x55ac7b31b220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:57:36.308 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:36.303+0000 7f9f78fb4740 1 --2- 172.21.15.29:0/1748353744 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ac7b306130 0x55ac7b30eeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:57:36.308 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:36.303+0000 7f9f78fb4740 1 -- 172.21.15.29:0/1748353744 >> 172.21.15.29:0/1748353744 conn(0x55ac7b300a50 msgr2=0x55ac7b2ffbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:57:36.308 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:36.303+0000 7f9f78fb4740 1 -- 172.21.15.29:0/1748353744 shutdown_connections 2024-03-16T11:57:36.308 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:36.303+0000 7f9f78fb4740 1 -- 172.21.15.29:0/1748353744 wait complete. 2024-03-16T11:57:36.308 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:36.303+0000 7f9f78fb4740 1 -- start start 2024-03-16T11:57:36.308 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:36.304+0000 7f9f78fb4740 1 Processor -- start 2024-03-16T11:57:36.308 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:36.304+0000 7f9f78fb4740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ac7b306130 0x55ac7b3133c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:57:36.308 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:36.304+0000 7f9f78fb4740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ac7b306a50 0x55ac7b313900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:57:36.308 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:36.304+0000 7f9f78fb4740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ac7b30f820 0x55ac7b313e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:57:36.308 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:36.304+0000 7f9f78fb4740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f9f701abbe0 con 0x55ac7b30f820 2024-03-16T11:57:36.308 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:36.304+0000 7f9f78fb4740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f9f701ad050 con 0x55ac7b306a50 2024-03-16T11:57:36.308 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:36.304+0000 7f9f78fb4740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f9f701ac2b0 con 0x55ac7b306130 2024-03-16T11:57:36.309 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:36.304+0000 7f9f78fb4740 10 client.0 _ll_register_callbacks cb 0x7f9f680093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T11:57:36.309 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:36.304+0000 7f9f6f7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ac7b30f820 0x55ac7b313e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:57:36.309 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:36.304+0000 7f9f6f7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ac7b30f820 0x55ac7b313e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41642/0 (socket says 172.21.15.29:41642) 2024-03-16T11:57:36.309 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:36.304+0000 7f9f6f7fe640 1 -- 172.21.15.29:0/3772646423 learned_addr learned my addr 172.21.15.29:0/3772646423 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:57:36.309 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:36.304+0000 7f9f6f7fe640 1 --2- 172.21.15.29:0/3772646423 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ac7b306130 0x55ac7b3133c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:57:36.309 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:36.304+0000 7f9f6ffff640 1 --2- 172.21.15.29:0/3772646423 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ac7b306a50 0x55ac7b313900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:57:36.309 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:36.304+0000 7f9f6f7fe640 1 -- 172.21.15.29:0/3772646423 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ac7b306130 msgr2=0x55ac7b3133c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:57:36.309 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:36.304+0000 7f9f6f7fe640 1 --2- 172.21.15.29:0/3772646423 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ac7b306130 0x55ac7b3133c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:57:36.309 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:36.304+0000 7f9f6f7fe640 1 -- 172.21.15.29:0/3772646423 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ac7b306a50 msgr2=0x55ac7b313900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:57:36.309 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:36.304+0000 7f9f6f7fe640 1 --2- 172.21.15.29:0/3772646423 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ac7b306a50 0x55ac7b313900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:57:36.309 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:36.304+0000 7f9f6f7fe640 1 -- 172.21.15.29:0/3772646423 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ac7b384bd0 con 0x55ac7b30f820 2024-03-16T11:57:36.309 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:36.304+0000 7f9f6f7fe640 1 --2- 172.21.15.29:0/3772646423 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ac7b30f820 0x55ac7b313e40 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f9f64022040 tx=0x7f9f64013040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b1a7208fe0147b5 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:57:36.309 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:36.304+0000 7f9f4f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-16T11:57:36.309 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:36.305+0000 7f9f4f7fe640 1 -- 172.21.15.29:0/3772646423 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f6403c040 con 0x55ac7b30f820 2024-03-16T11:57:36.309 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:36.305+0000 7f9f4f7fe640 1 -- 172.21.15.29:0/3772646423 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8556+0+0 (secure 0 0 0) 0x7f9f64004870 con 0x55ac7b30f820 2024-03-16T11:57:36.309 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:36.305+0000 7f9f78fb4740 1 -- 172.21.15.29:0/3772646423 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9f6400f790 con 0x55ac7b30f820 2024-03-16T11:57:36.310 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:36.305+0000 7f9f4f7fe640 1 -- 172.21.15.29:0/3772646423 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f9f64004d80 con 0x55ac7b30f820 2024-03-16T11:57:36.310 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:36.305+0000 7f9f78fb4740 10 client.15960 fetch_fsmap learned FSMap version 24 2024-03-16T11:57:36.310 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:36.305+0000 7f9f78fb4740 1 -- 172.21.15.29:0/3772646423 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55ac7b385ea0 con 0x55ac7b30f820 2024-03-16T11:57:36.310 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:36.305+0000 7f9f4f7fe640 1 -- 172.21.15.29:0/3772646423 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9f6400f790 con 0x55ac7b30f820 2024-03-16T11:57:36.310 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:36.305+0000 7f9f4f7fe640 1 -- 172.21.15.29:0/3772646423 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7f9f64004040 con 0x55ac7b30f820 2024-03-16T11:57:36.310 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:36.305+0000 7f9f78fb4740 10 client.15960 fetch_fsmap finished waiting for FSMap version 24 2024-03-16T11:57:36.310 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:36.305+0000 7f9f78fb4740 10 client.15960 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T11:57:36.310 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:36.305+0000 7f9f78fb4740 20 client.15960 populate_metadata read hostname 'smithi029' 2024-03-16T11:57:36.310 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:36.305+0000 7f9f78fb4740 1 --2- 172.21.15.29:0/3772646423 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55ac7b386450 0x55ac7b388840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:57:36.310 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:36.305+0000 7f9f78fb4740 4 client.15960 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-16T11:57:36.310 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:36.305+0000 7f9f78fb4740 1 -- 172.21.15.29:0/3772646423 --> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55ac7b2722d0 con 0x55ac7b386450 2024-03-16T11:57:36.310 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:36.305+0000 7f9f6ffff640 1 --2- 172.21.15.29:0/3772646423 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55ac7b386450 0x55ac7b388840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:57:36.310 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:36.306+0000 7f9f4f7fe640 10 client.15960 ms_handle_connect on v2:172.21.15.152:6838/1548731833 2024-03-16T11:57:36.310 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:36.306+0000 7f9f6ffff640 1 --2- 172.21.15.29:0/3772646423 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55ac7b386450 0x55ac7b388840 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fa06fe74da3f3f6c server_cookie=ff9d3c67596044c0 in_seq=0 out_seq=0 2024-03-16T11:57:36.517 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:36.516+0000 7f9f4f7fe640 1 -- 172.21.15.29:0/3772646423 <== mds.0 v2:172.21.15.152:6838/1548731833 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55ac7b2722d0 con 0x55ac7b386450 2024-03-16T11:57:36.517 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:36.516+0000 7f9f4f7fe640 10 client.15960 handle_command_reply: tid=0 2024-03-16T11:57:36.517 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:36.516+0000 7f9f78fb4740 1 client.15960 shutdown 2024-03-16T11:57:36.517 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:36.516+0000 7f9f78fb4740 1 -- 172.21.15.29:0/3772646423 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ac7b30f820 msgr2=0x55ac7b313e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:57:36.518 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:36.516+0000 7f9f78fb4740 1 --2- 172.21.15.29:0/3772646423 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ac7b30f820 0x55ac7b313e40 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f9f64022040 tx=0x7f9f64013040 comp rx=0 tx=0).stop 2024-03-16T11:57:36.518 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:36.517+0000 7f9f78fb4740 1 -- 172.21.15.29:0/3772646423 shutdown_connections 2024-03-16T11:57:36.518 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:36.517+0000 7f9f78fb4740 1 --2- 172.21.15.29:0/3772646423 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ac7b30f820 0x55ac7b313e40 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:57:36.518 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:36.517+0000 7f9f78fb4740 1 --2- 172.21.15.29:0/3772646423 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55ac7b386450 0x55ac7b388840 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:57:36.518 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:36.517+0000 7f9f4f7fe640 0 client.15960 ms_handle_reset on v2:172.21.15.152:6838/1548731833 2024-03-16T11:57:36.518 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:36.517+0000 7f9f78fb4740 1 --2- 172.21.15.29:0/3772646423 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ac7b306a50 0x55ac7b313900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:57:36.518 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:36.517+0000 7f9f78fb4740 1 --2- 172.21.15.29:0/3772646423 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ac7b306130 0x55ac7b3133c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:57:36.518 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:36.517+0000 7f9f78fb4740 1 -- 172.21.15.29:0/3772646423 >> 172.21.15.29:0/3772646423 conn(0x55ac7b300a50 msgr2=0x55ac7b2ff320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:57:36.518 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:36.517+0000 7f9f78fb4740 1 -- 172.21.15.29:0/3772646423 shutdown_connections 2024-03-16T11:57:36.518 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:36.517+0000 7f9f78fb4740 1 -- 172.21.15.29:0/3772646423 wait complete. 2024-03-16T11:57:36.518 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:36.517+0000 7f9f78fb4740 20 client.15960 trim_cache size 0 max 16384 2024-03-16T11:57:36.519 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:{ 2024-03-16T11:57:36.519 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "message": "", 2024-03-16T11:57:36.519 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "return_code": 0 2024-03-16T11:57:36.519 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:} 2024-03-16T11:57:36.519 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:36.518+0000 7f9f763c6640 1 -- 172.21.15.29:0/224337734 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f9f580a2a10 msgr2=0x7f9f580a4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:57:36.520 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:36.518+0000 7f9f763c6640 1 --2- 172.21.15.29:0/224337734 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f9f580a2a10 0x7f9f580a4ed0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f9f6809b810 tx=0x7f9f6800efd0 comp rx=0 tx=0).stop 2024-03-16T11:57:36.520 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:36.518+0000 7f9f763c6640 1 -- 172.21.15.29:0/224337734 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9f701077c0 msgr2=0x7f9f7019c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:57:36.520 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:36.518+0000 7f9f763c6640 1 --2- 172.21.15.29:0/224337734 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9f701077c0 0x7f9f7019c8f0 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f9f6800eb10 tx=0x7f9f6800c2d0 comp rx=0 tx=0).stop 2024-03-16T11:57:36.520 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:36.518+0000 7f9f763c6640 1 -- 172.21.15.29:0/224337734 shutdown_connections 2024-03-16T11:57:36.520 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:36.518+0000 7f9f763c6640 1 --2- 172.21.15.29:0/224337734 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f9f580a2a10 0x7f9f580a4ed0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:57:36.520 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:36.518+0000 7f9f763c6640 1 --2- 172.21.15.29:0/224337734 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9f701077c0 0x7f9f7019c8f0 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:57:36.520 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:36.518+0000 7f9f763c6640 1 --2- 172.21.15.29:0/224337734 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9f70102170 0x7f9f70198570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:57:36.520 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:36.518+0000 7f9f763c6640 1 --2- 172.21.15.29:0/224337734 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9f701017b0 0x7f9f70198030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:57:36.520 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:36.518+0000 7f9f763c6640 1 -- 172.21.15.29:0/224337734 >> 172.21.15.29:0/224337734 conn(0x7f9f700fd2c0 msgr2=0x7f9f700fd880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:57:36.520 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:36.518+0000 7f9f763c6640 1 -- 172.21.15.29:0/224337734 shutdown_connections 2024-03-16T11:57:36.520 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:36.518+0000 7f9f763c6640 1 -- 172.21.15.29:0/224337734 wait complete. 2024-03-16T11:57:36.790 INFO:tasks.workunit.client.0.smithi029.stdout: 1 675746 10.92 MB/sec execute 201 sec latency 37.717 ms 2024-03-16T11:57:36.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:57:36 smithi154 ceph-mon[36320]: pgmap v207: 97 pgs: 97 active+clean; 73 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 2.7 MiB/s wr, 273 op/s 2024-03-16T11:57:36.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:57:36 smithi029 ceph-mon[31975]: pgmap v207: 97 pgs: 97 active+clean; 73 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 2.7 MiB/s wr, 273 op/s 2024-03-16T11:57:36.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:57:36 smithi152 ceph-mon[36291]: pgmap v207: 97 pgs: 97 active+clean; 73 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 2.7 MiB/s wr, 273 op/s 2024-03-16T11:57:37.790 INFO:tasks.workunit.client.0.smithi029.stdout: 1 677973 10.93 MB/sec execute 202 sec latency 22.629 ms 2024-03-16T11:57:38.790 INFO:tasks.workunit.client.0.smithi029.stdout: 1 679224 10.91 MB/sec execute 203 sec latency 74.284 ms 2024-03-16T11:57:38.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:57:38 smithi154 ceph-mon[36320]: pgmap v208: 97 pgs: 97 active+clean; 79 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 3.9 MiB/s wr, 358 op/s 2024-03-16T11:57:38.837 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:57:38 smithi029 ceph-mon[31975]: pgmap v208: 97 pgs: 97 active+clean; 79 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 3.9 MiB/s wr, 358 op/s 2024-03-16T11:57:38.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:57:38 smithi152 ceph-mon[36291]: pgmap v208: 97 pgs: 97 active+clean; 79 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 3.9 MiB/s wr, 358 op/s 2024-03-16T11:57:39.790 INFO:tasks.workunit.client.0.smithi029.stdout: 1 681237 10.91 MB/sec execute 204 sec latency 38.999 ms 2024-03-16T11:57:40.790 INFO:tasks.workunit.client.0.smithi029.stdout: 1 682684 10.89 MB/sec execute 205 sec latency 22.972 ms 2024-03-16T11:57:40.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:57:40 smithi154 ceph-mon[36320]: pgmap v209: 97 pgs: 97 active+clean; 79 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 2.6 MiB/s wr, 175 op/s 2024-03-16T11:57:40.837 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:57:40 smithi029 ceph-mon[31975]: pgmap v209: 97 pgs: 97 active+clean; 79 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 2.6 MiB/s wr, 175 op/s 2024-03-16T11:57:40.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:57:40 smithi152 ceph-mon[36291]: pgmap v209: 97 pgs: 97 active+clean; 79 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 2.6 MiB/s wr, 175 op/s 2024-03-16T11:57:41.790 INFO:tasks.workunit.client.0.smithi029.stdout: 1 684742 10.89 MB/sec execute 206 sec latency 21.886 ms 2024-03-16T11:57:42.790 INFO:tasks.workunit.client.0.smithi029.stdout: 1 686209 10.88 MB/sec execute 207 sec latency 22.701 ms 2024-03-16T11:57:42.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:57:42 smithi154 ceph-mon[36320]: pgmap v210: 97 pgs: 97 active+clean; 69 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 3.9 MiB/s wr, 357 op/s 2024-03-16T11:57:42.837 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:57:42 smithi029 ceph-mon[31975]: pgmap v210: 97 pgs: 97 active+clean; 69 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 3.9 MiB/s wr, 357 op/s 2024-03-16T11:57:42.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:57:42 smithi152 ceph-mon[36291]: pgmap v210: 97 pgs: 97 active+clean; 69 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 3.9 MiB/s wr, 357 op/s 2024-03-16T11:57:43.790 INFO:tasks.workunit.client.0.smithi029.stdout: 1 687995 10.86 MB/sec execute 208 sec latency 70.905 ms 2024-03-16T11:57:44.678 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:57:44 smithi152 ceph-mon[36291]: pgmap v211: 97 pgs: 97 active+clean; 69 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 2.5 MiB/s wr, 267 op/s 2024-03-16T11:57:44.790 INFO:tasks.workunit.client.0.smithi029.stdout: 1 689703 10.87 MB/sec execute 209 sec latency 19.101 ms 2024-03-16T11:57:44.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:57:44 smithi154 ceph-mon[36320]: pgmap v211: 97 pgs: 97 active+clean; 69 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 2.5 MiB/s wr, 267 op/s 2024-03-16T11:57:44.837 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:57:44 smithi029 ceph-mon[31975]: pgmap v211: 97 pgs: 97 active+clean; 69 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 2.5 MiB/s wr, 267 op/s 2024-03-16T11:57:45.790 INFO:tasks.workunit.client.0.smithi029.stdout: 1 691849 10.87 MB/sec execute 210 sec latency 25.605 ms 2024-03-16T11:57:46.790 INFO:tasks.workunit.client.0.smithi029.stdout: 1 693428 10.86 MB/sec execute 211 sec latency 14.304 ms 2024-03-16T11:57:46.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:57:46 smithi154 ceph-mon[36320]: pgmap v212: 97 pgs: 97 active+clean; 69 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 2.5 MiB/s wr, 267 op/s 2024-03-16T11:57:46.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:57:46 smithi029 ceph-mon[31975]: pgmap v212: 97 pgs: 97 active+clean; 69 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 2.5 MiB/s wr, 267 op/s 2024-03-16T11:57:46.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:57:46 smithi152 ceph-mon[36291]: pgmap v212: 97 pgs: 97 active+clean; 69 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 2.5 MiB/s wr, 267 op/s 2024-03-16T11:57:47.790 INFO:tasks.workunit.client.0.smithi029.stdout: 1 695698 10.86 MB/sec execute 212 sec latency 18.573 ms 2024-03-16T11:57:48.790 INFO:tasks.workunit.client.0.smithi029.stdout: 1 697040 10.84 MB/sec execute 213 sec latency 38.673 ms 2024-03-16T11:57:48.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:57:48 smithi154 ceph-mon[36320]: pgmap v213: 97 pgs: 97 active+clean; 74 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 3.7 MiB/s wr, 350 op/s 2024-03-16T11:57:48.837 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:57:48 smithi029 ceph-mon[31975]: pgmap v213: 97 pgs: 97 active+clean; 74 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 3.7 MiB/s wr, 350 op/s 2024-03-16T11:57:48.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:57:48 smithi152 ceph-mon[36291]: pgmap v213: 97 pgs: 97 active+clean; 74 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 3.7 MiB/s wr, 350 op/s 2024-03-16T11:57:49.613 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.611+0000 7f1520227640 1 -- 172.21.15.29:0/2144999701 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1518104350 msgr2=0x7f1518110860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:57:49.613 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.611+0000 7f1520227640 1 --2- 172.21.15.29:0/2144999701 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1518104350 0x7f1518110860 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f1510009cd0 tx=0x7f151002c0f0 comp rx=0 tx=0).stop 2024-03-16T11:57:49.613 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.611+0000 7f1520227640 1 -- 172.21.15.29:0/2144999701 shutdown_connections 2024-03-16T11:57:49.613 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.611+0000 7f1520227640 1 --2- 172.21.15.29:0/2144999701 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1518104350 0x7f1518110860 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:57:49.613 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.611+0000 7f1520227640 1 --2- 172.21.15.29:0/2144999701 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1518103990 0x7f1518103e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:57:49.613 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.611+0000 7f1520227640 1 --2- 172.21.15.29:0/2144999701 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f15181099a0 0x7f1518109d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:57:49.613 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.611+0000 7f1520227640 1 -- 172.21.15.29:0/2144999701 >> 172.21.15.29:0/2144999701 conn(0x7f15180ff460 msgr2=0x7f1518101880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:57:49.613 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.611+0000 7f1520227640 1 -- 172.21.15.29:0/2144999701 shutdown_connections 2024-03-16T11:57:49.613 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.611+0000 7f1520227640 1 -- 172.21.15.29:0/2144999701 wait complete. 2024-03-16T11:57:49.614 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.612+0000 7f1520227640 1 Processor -- start 2024-03-16T11:57:49.614 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.612+0000 7f1520227640 1 -- start start 2024-03-16T11:57:49.614 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.612+0000 7f1520227640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1518103990 0x7f151819a1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:57:49.614 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.612+0000 7f1520227640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1518104350 0x7f151819a720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:57:49.614 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.612+0000 7f1520227640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f15181099a0 0x7f151819eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:57:49.614 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.612+0000 7f1520227640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f1518113d80 con 0x7f1518104350 2024-03-16T11:57:49.614 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.612+0000 7f1520227640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f1518113c00 con 0x7f15181099a0 2024-03-16T11:57:49.614 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.612+0000 7f1520227640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f1518113f00 con 0x7f1518103990 2024-03-16T11:57:49.614 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.612+0000 7f151d7c2640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1518104350 0x7f151819a720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:57:49.614 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.612+0000 7f151dfc3640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1518103990 0x7f151819a1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:57:49.614 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.612+0000 7f151dfc3640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1518103990 0x7f151819a1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.29:36292/0 (socket says 172.21.15.29:36292) 2024-03-16T11:57:49.614 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.612+0000 7f151dfc3640 1 -- 172.21.15.29:0/3487570614 learned_addr learned my addr 172.21.15.29:0/3487570614 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:57:49.614 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.612+0000 7f151e7c4640 1 --2- 172.21.15.29:0/3487570614 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f15181099a0 0x7f151819eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:57:49.615 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.613+0000 7f151dfc3640 1 -- 172.21.15.29:0/3487570614 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f15181099a0 msgr2=0x7f151819eaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:57:49.615 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.613+0000 7f151dfc3640 1 --2- 172.21.15.29:0/3487570614 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f15181099a0 0x7f151819eaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:57:49.615 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.613+0000 7f151dfc3640 1 -- 172.21.15.29:0/3487570614 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1518104350 msgr2=0x7f151819a720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:57:49.615 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.613+0000 7f151dfc3640 1 --2- 172.21.15.29:0/3487570614 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1518104350 0x7f151819a720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:57:49.615 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.613+0000 7f151dfc3640 1 -- 172.21.15.29:0/3487570614 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f151819f1d0 con 0x7f1518103990 2024-03-16T11:57:49.615 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.613+0000 7f151dfc3640 1 --2- 172.21.15.29:0/3487570614 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1518103990 0x7f151819a1e0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f151400e9f0 tx=0x7f151400eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:57:49.615 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.613+0000 7f150effd640 1 -- 172.21.15.29:0/3487570614 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f151400cd80 con 0x7f1518103990 2024-03-16T11:57:49.615 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.613+0000 7f1520227640 1 -- 172.21.15.29:0/3487570614 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f151819f4c0 con 0x7f1518103990 2024-03-16T11:57:49.615 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.613+0000 7f150effd640 1 -- 172.21.15.29:0/3487570614 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1514010910 con 0x7f1518103990 2024-03-16T11:57:49.615 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.613+0000 7f1520227640 1 -- 172.21.15.29:0/3487570614 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f15181a6d50 con 0x7f1518103990 2024-03-16T11:57:49.615 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.613+0000 7f150effd640 1 -- 172.21.15.29:0/3487570614 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1514018910 con 0x7f1518103990 2024-03-16T11:57:49.620 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.615+0000 7f150effd640 1 -- 172.21.15.29:0/3487570614 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7f1514010ac0 con 0x7f1518103990 2024-03-16T11:57:49.621 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.616+0000 7f1522e0e740 1 Processor -- start 2024-03-16T11:57:49.621 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.616+0000 7f150effd640 1 --2- 172.21.15.29:0/3487570614 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f15040a2a10 0x7f15040a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:57:49.621 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.616+0000 7f150effd640 1 -- 172.21.15.29:0/3487570614 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8556+0+0 (secure 0 0 0) 0x7f15140cf120 con 0x7f1518103990 2024-03-16T11:57:49.621 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.616+0000 7f151e7c4640 1 --2- 172.21.15.29:0/3487570614 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f15040a2a10 0x7f15040a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:57:49.621 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.616+0000 7f1522e0e740 1 -- start start 2024-03-16T11:57:49.621 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.616+0000 7f1522e0e740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562d5f127130 0x562d5f127510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:57:49.621 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.616+0000 7f1522e0e740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562d5f127a50 0x562d5f1302e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:57:49.621 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.616+0000 7f1522e0e740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562d5f130820 0x562d5f132c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:57:49.621 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.616+0000 7f1522e0e740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x562d5f08ba40 con 0x562d5f127130 2024-03-16T11:57:49.621 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.616+0000 7f1522e0e740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x562d5f034920 con 0x562d5f127a50 2024-03-16T11:57:49.621 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.616+0000 7f1522e0e740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x562d5f087ae0 con 0x562d5f130820 2024-03-16T11:57:49.621 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.616+0000 7f151dfc3640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562d5f127130 0x562d5f127510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:57:49.621 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.616+0000 7f151dfc3640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562d5f127130 0x562d5f127510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42566/0 (socket says 172.21.15.29:42566) 2024-03-16T11:57:49.622 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.616+0000 7f151dfc3640 1 -- 172.21.15.29:0/852468853 learned_addr learned my addr 172.21.15.29:0/852468853 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:57:49.622 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.616+0000 7f151d7c2640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562d5f127a50 0x562d5f1302e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:57:49.622 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.617+0000 7f151e7c4640 1 --2- 172.21.15.29:0/3487570614 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f15040a2a10 0x7f15040a4ed0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f1510005cc0 tx=0x7f1510005c50 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:57:49.622 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.617+0000 7f151e7c4640 1 --2- 172.21.15.29:0/852468853 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562d5f130820 0x562d5f132c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:57:49.622 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.617+0000 7f151d7c2640 1 -- 172.21.15.29:0/852468853 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562d5f130820 msgr2=0x562d5f132c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:57:49.622 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.617+0000 7f151d7c2640 1 --2- 172.21.15.29:0/852468853 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562d5f130820 0x562d5f132c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:57:49.622 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.617+0000 7f151d7c2640 1 -- 172.21.15.29:0/852468853 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562d5f127130 msgr2=0x562d5f127510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:57:49.622 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.617+0000 7f151d7c2640 1 --2- 172.21.15.29:0/852468853 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562d5f127130 0x562d5f127510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:57:49.622 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.617+0000 7f151d7c2640 1 -- 172.21.15.29:0/852468853 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562d5f04c460 con 0x562d5f127a50 2024-03-16T11:57:49.622 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.617+0000 7f151dfc3640 1 --2- 172.21.15.29:0/852468853 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562d5f127130 0x562d5f127510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:57:49.622 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.617+0000 7f151d7c2640 1 --2- 172.21.15.29:0/852468853 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562d5f127a50 0x562d5f1302e0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f1508009bf0 tx=0x7f1508011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8ca2d534fa2e1c1f server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:57:49.622 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.617+0000 7f1503fff640 1 -- 172.21.15.29:0/852468853 <== mon.2 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1508004050 con 0x562d5f127a50 2024-03-16T11:57:49.622 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.617+0000 7f1503fff640 1 -- 172.21.15.29:0/852468853 <== mon.2 v2:172.21.15.152:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1508005500 con 0x562d5f127a50 2024-03-16T11:57:49.623 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.617+0000 7f1503fff640 1 -- 172.21.15.29:0/852468853 <== mon.2 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1508004e40 con 0x562d5f127a50 2024-03-16T11:57:49.623 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.618+0000 7f1522e0e740 1 -- 172.21.15.29:0/852468853 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562d5f127a50 msgr2=0x562d5f1302e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:57:49.623 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.618+0000 7f1522e0e740 1 --2- 172.21.15.29:0/852468853 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562d5f127a50 0x562d5f1302e0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f1508009bf0 tx=0x7f1508011810 comp rx=0 tx=0).stop 2024-03-16T11:57:49.623 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.618+0000 7f1522e0e740 1 -- 172.21.15.29:0/852468853 shutdown_connections 2024-03-16T11:57:49.623 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.618+0000 7f1522e0e740 1 --2- 172.21.15.29:0/852468853 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562d5f130820 0x562d5f132c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:57:49.623 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.618+0000 7f1522e0e740 1 --2- 172.21.15.29:0/852468853 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562d5f127a50 0x562d5f1302e0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:57:49.623 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.618+0000 7f1522e0e740 1 --2- 172.21.15.29:0/852468853 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562d5f127130 0x562d5f127510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:57:49.623 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.618+0000 7f1522e0e740 1 -- 172.21.15.29:0/852468853 >> 172.21.15.29:0/852468853 conn(0x562d5f121a50 msgr2=0x562d5f121e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:57:49.623 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.618+0000 7f1522e0e740 1 -- 172.21.15.29:0/852468853 shutdown_connections 2024-03-16T11:57:49.623 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.618+0000 7f1522e0e740 1 -- 172.21.15.29:0/852468853 wait complete. 2024-03-16T11:57:49.623 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.618+0000 7f1522e0e740 1 -- start start 2024-03-16T11:57:49.623 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.619+0000 7f1522e0e740 1 Processor -- start 2024-03-16T11:57:49.623 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.619+0000 7f1522e0e740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562d5f127130 0x562d5f12d7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:57:49.623 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.619+0000 7f1522e0e740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562d5f127a50 0x562d5f12dd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:57:49.623 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.619+0000 7f1522e0e740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562d5f130820 0x562d5f12e270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:57:49.624 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.619+0000 7f1522e0e740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x562d5f133450 con 0x562d5f127130 2024-03-16T11:57:49.624 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.619+0000 7f1522e0e740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x562d5f047d70 con 0x562d5f127a50 2024-03-16T11:57:49.624 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.619+0000 7f1522e0e740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x562d5f1335d0 con 0x562d5f130820 2024-03-16T11:57:49.624 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.619+0000 7f1522e0e740 10 client.0 _ll_register_callbacks cb 0x562d5ed9fa00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T11:57:49.624 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.619+0000 7f151dfc3640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562d5f127130 0x562d5f12d7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:57:49.624 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.619+0000 7f151dfc3640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562d5f127130 0x562d5f12d7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42572/0 (socket says 172.21.15.29:42572) 2024-03-16T11:57:49.624 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.619+0000 7f151dfc3640 1 -- 172.21.15.29:0/1618818085 learned_addr learned my addr 172.21.15.29:0/1618818085 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:57:49.624 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.619+0000 7f151e7c4640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562d5f130820 0x562d5f12e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:57:49.624 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.619+0000 7f151d7c2640 1 --2- 172.21.15.29:0/1618818085 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562d5f127a50 0x562d5f12dd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:57:49.624 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.619+0000 7f151dfc3640 1 -- 172.21.15.29:0/1618818085 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562d5f130820 msgr2=0x562d5f12e270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:57:49.624 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.619+0000 7f151dfc3640 1 --2- 172.21.15.29:0/1618818085 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562d5f130820 0x562d5f12e270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:57:49.624 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.619+0000 7f151dfc3640 1 -- 172.21.15.29:0/1618818085 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562d5f127a50 msgr2=0x562d5f12dd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:57:49.624 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.619+0000 7f151dfc3640 1 --2- 172.21.15.29:0/1618818085 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562d5f127a50 0x562d5f12dd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:57:49.624 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.619+0000 7f151dfc3640 1 -- 172.21.15.29:0/1618818085 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562d5eda0210 con 0x562d5f127130 2024-03-16T11:57:49.625 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.619+0000 7f151e7c4640 1 --2- 172.21.15.29:0/1618818085 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562d5f130820 0x562d5f12e270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:57:49.625 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.620+0000 7f151dfc3640 1 --2- 172.21.15.29:0/1618818085 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562d5f127130 0x562d5f12d7f0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f15140cc900 tx=0x7f151409b950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d98d15ac14e7551 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:57:49.625 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.620+0000 7f15017fa640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-16T11:57:49.625 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.620+0000 7f15017fa640 1 -- 172.21.15.29:0/1618818085 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15140b4060 con 0x562d5f127130 2024-03-16T11:57:49.625 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.620+0000 7f15017fa640 1 -- 172.21.15.29:0/1618818085 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8556+0+0 (secure 0 0 0) 0x7f15140afd40 con 0x562d5f127130 2024-03-16T11:57:49.625 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.620+0000 7f1522e0e740 1 -- 172.21.15.29:0/1618818085 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1508005500 con 0x562d5f127130 2024-03-16T11:57:49.625 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.620+0000 7f15017fa640 1 -- 172.21.15.29:0/1618818085 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f151409ac60 con 0x562d5f127130 2024-03-16T11:57:49.625 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.620+0000 7f1522e0e740 10 client.15972 fetch_fsmap learned FSMap version 24 2024-03-16T11:57:49.625 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.620+0000 7f1522e0e740 1 -- 172.21.15.29:0/1618818085 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x562d5ee6c620 con 0x562d5f127130 2024-03-16T11:57:49.625 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.620+0000 7f15017fa640 1 -- 172.21.15.29:0/1618818085 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1508005500 con 0x562d5f127130 2024-03-16T11:57:49.625 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.620+0000 7f15017fa640 1 -- 172.21.15.29:0/1618818085 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7f1514097720 con 0x562d5f127130 2024-03-16T11:57:49.625 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.620+0000 7f1522e0e740 10 client.15972 fetch_fsmap finished waiting for FSMap version 24 2024-03-16T11:57:49.625 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.620+0000 7f1522e0e740 10 client.15972 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T11:57:49.625 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.620+0000 7f1522e0e740 20 client.15972 populate_metadata read hostname 'smithi029' 2024-03-16T11:57:49.625 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.620+0000 7f1522e0e740 1 --2- 172.21.15.29:0/1618818085 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x562d5f1409e0 0x562d5f142dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:57:49.626 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.620+0000 7f1522e0e740 4 client.15972 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T11:57:49.626 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.620+0000 7f1522e0e740 1 -- 172.21.15.29:0/1618818085 --> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562d5f098f80 con 0x562d5f1409e0 2024-03-16T11:57:49.626 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.621+0000 7f151d7c2640 1 --2- 172.21.15.29:0/1618818085 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x562d5f1409e0 0x562d5f142dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:57:49.626 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.621+0000 7f15017fa640 10 client.15972 ms_handle_connect on v2:172.21.15.152:6838/1548731833 2024-03-16T11:57:49.626 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.621+0000 7f151d7c2640 1 --2- 172.21.15.29:0/1618818085 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x562d5f1409e0 0x562d5f142dd0 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d4e6d7c52b2798d1 server_cookie=c1166b16494a8ce8 in_seq=0 out_seq=0 2024-03-16T11:57:49.627 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.623+0000 7f15017fa640 1 -- 172.21.15.29:0/1618818085 <== mds.0 v2:172.21.15.152:6838/1548731833 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x562d5f098f80 con 0x562d5f1409e0 2024-03-16T11:57:49.627 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.623+0000 7f15017fa640 10 client.15972 handle_command_reply: tid=0 2024-03-16T11:57:49.627 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.623+0000 7f1522e0e740 1 client.15972 shutdown 2024-03-16T11:57:49.627 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.623+0000 7f1522e0e740 1 -- 172.21.15.29:0/1618818085 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562d5f127130 msgr2=0x562d5f12d7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:57:49.627 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.623+0000 7f1522e0e740 1 --2- 172.21.15.29:0/1618818085 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562d5f127130 0x562d5f12d7f0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f15140cc900 tx=0x7f151409b950 comp rx=0 tx=0).stop 2024-03-16T11:57:49.627 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.623+0000 7f1522e0e740 1 -- 172.21.15.29:0/1618818085 shutdown_connections 2024-03-16T11:57:49.627 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.623+0000 7f1522e0e740 1 --2- 172.21.15.29:0/1618818085 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562d5f130820 0x562d5f12e270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:57:49.627 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.623+0000 7f1522e0e740 1 --2- 172.21.15.29:0/1618818085 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x562d5f1409e0 0x562d5f142dd0 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:57:49.627 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.623+0000 7f1522e0e740 1 --2- 172.21.15.29:0/1618818085 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562d5f127a50 0x562d5f12dd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:57:49.628 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.623+0000 7f1522e0e740 1 --2- 172.21.15.29:0/1618818085 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562d5f127130 0x562d5f12d7f0 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:57:49.628 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.623+0000 7f15017fa640 0 client.15972 ms_handle_reset on v2:172.21.15.152:6838/1548731833 2024-03-16T11:57:49.628 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.623+0000 7f1522e0e740 1 -- 172.21.15.29:0/1618818085 >> 172.21.15.29:0/1618818085 conn(0x562d5f121a50 msgr2=0x562d5f1237a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:57:49.628 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.623+0000 7f1522e0e740 1 -- 172.21.15.29:0/1618818085 shutdown_connections 2024-03-16T11:57:49.628 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.623+0000 7f1522e0e740 1 -- 172.21.15.29:0/1618818085 wait complete. 2024-03-16T11:57:49.628 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.623+0000 7f1522e0e740 20 client.15972 trim_cache size 0 max 16384 2024-03-16T11:57:49.640 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.640+0000 7f1522e0e740 1 Processor -- start 2024-03-16T11:57:49.640 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.640+0000 7f1522e0e740 1 -- start start 2024-03-16T11:57:49.640 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.640+0000 7f1522e0e740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562d5f127130 0x562d5f12ff20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:57:49.640 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.640+0000 7f1522e0e740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562d5f127a50 0x562d5f11ca40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:57:49.640 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.640+0000 7f1522e0e740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562d5f130820 0x562d5f13c1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:57:49.640 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.640+0000 7f1522e0e740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f15181b1fc0 con 0x562d5f127130 2024-03-16T11:57:49.640 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.640+0000 7f1522e0e740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f15181af890 con 0x562d5f130820 2024-03-16T11:57:49.640 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.640+0000 7f1522e0e740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f15181ad700 con 0x562d5f127a50 2024-03-16T11:57:49.640 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.640+0000 7f151dfc3640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562d5f127130 0x562d5f12ff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:57:49.640 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.640+0000 7f151dfc3640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562d5f127130 0x562d5f12ff20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42578/0 (socket says 172.21.15.29:42578) 2024-03-16T11:57:49.641 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.640+0000 7f151dfc3640 1 -- 172.21.15.29:0/2915191565 learned_addr learned my addr 172.21.15.29:0/2915191565 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:57:49.641 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.640+0000 7f151e7c4640 1 --2- 172.21.15.29:0/2915191565 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562d5f130820 0x562d5f13c1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:57:49.641 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.640+0000 7f151d7c2640 1 --2- 172.21.15.29:0/2915191565 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562d5f127a50 0x562d5f11ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:57:49.641 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.640+0000 7f151dfc3640 1 -- 172.21.15.29:0/2915191565 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562d5f127a50 msgr2=0x562d5f11ca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:57:49.641 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.640+0000 7f151dfc3640 1 --2- 172.21.15.29:0/2915191565 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562d5f127a50 0x562d5f11ca40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:57:49.641 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.640+0000 7f151dfc3640 1 -- 172.21.15.29:0/2915191565 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562d5f130820 msgr2=0x562d5f13c1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:57:49.641 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.640+0000 7f151dfc3640 1 --2- 172.21.15.29:0/2915191565 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562d5f130820 0x562d5f13c1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:57:49.641 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.640+0000 7f151dfc3640 1 -- 172.21.15.29:0/2915191565 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562d5f12c7e0 con 0x562d5f127130 2024-03-16T11:57:49.641 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.640+0000 7f151e7c4640 1 --2- 172.21.15.29:0/2915191565 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562d5f130820 0x562d5f13c1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:57:49.641 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.640+0000 7f151dfc3640 1 --2- 172.21.15.29:0/2915191565 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562d5f127130 0x562d5f12ff20 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f15140cce50 tx=0x7f15140ba170 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d0465f103712043 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:57:49.641 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.641+0000 7f15037fe640 1 -- 172.21.15.29:0/2915191565 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15140a3f00 con 0x562d5f127130 2024-03-16T11:57:49.641 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.641+0000 7f15037fe640 1 -- 172.21.15.29:0/2915191565 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f151409ae00 con 0x562d5f127130 2024-03-16T11:57:49.641 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.641+0000 7f15037fe640 1 -- 172.21.15.29:0/2915191565 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1514097560 con 0x562d5f127130 2024-03-16T11:57:49.645 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.641+0000 7f1522e0e740 1 -- 172.21.15.29:0/2915191565 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562d5f127130 msgr2=0x562d5f12ff20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:57:49.645 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.641+0000 7f1522e0e740 1 --2- 172.21.15.29:0/2915191565 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562d5f127130 0x562d5f12ff20 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f15140cce50 tx=0x7f15140ba170 comp rx=0 tx=0).stop 2024-03-16T11:57:49.645 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.641+0000 7f1522e0e740 1 -- 172.21.15.29:0/2915191565 shutdown_connections 2024-03-16T11:57:49.645 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.641+0000 7f1522e0e740 1 --2- 172.21.15.29:0/2915191565 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562d5f130820 0x562d5f13c1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:57:49.645 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.641+0000 7f1522e0e740 1 --2- 172.21.15.29:0/2915191565 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562d5f127a50 0x562d5f11ca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:57:49.645 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.641+0000 7f1522e0e740 1 --2- 172.21.15.29:0/2915191565 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562d5f127130 0x562d5f12ff20 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:57:49.645 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.641+0000 7f1522e0e740 1 -- 172.21.15.29:0/2915191565 >> 172.21.15.29:0/2915191565 conn(0x562d5f121a50 msgr2=0x562d5f120bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:57:49.645 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.642+0000 7f1522e0e740 1 -- 172.21.15.29:0/2915191565 shutdown_connections 2024-03-16T11:57:49.646 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.642+0000 7f1522e0e740 1 -- 172.21.15.29:0/2915191565 wait complete. 2024-03-16T11:57:49.646 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.642+0000 7f1522e0e740 1 -- start start 2024-03-16T11:57:49.646 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.642+0000 7f1522e0e740 1 Processor -- start 2024-03-16T11:57:49.646 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.642+0000 7f1522e0e740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562d5f127130 0x562d5f12a990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:57:49.646 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.642+0000 7f1522e0e740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562d5f127a50 0x562d5f12aed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:57:49.646 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.642+0000 7f1522e0e740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562d5f130820 0x562d5f1a5d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:57:49.646 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.642+0000 7f1522e0e740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f15181add50 con 0x562d5f130820 2024-03-16T11:57:49.646 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.642+0000 7f1522e0e740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f15181af1c0 con 0x562d5f127a50 2024-03-16T11:57:49.646 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.642+0000 7f1522e0e740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f15181ae420 con 0x562d5f127130 2024-03-16T11:57:49.646 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.643+0000 7f1522e0e740 10 client.0 _ll_register_callbacks cb 0x7f151409b810 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T11:57:49.646 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.643+0000 7f151e7c4640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562d5f130820 0x562d5f1a5d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:57:49.646 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.643+0000 7f151d7c2640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562d5f127a50 0x562d5f12aed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:57:49.646 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.643+0000 7f151e7c4640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562d5f130820 0x562d5f1a5d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42588/0 (socket says 172.21.15.29:42588) 2024-03-16T11:57:49.646 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.643+0000 7f151e7c4640 1 -- 172.21.15.29:0/902822765 learned_addr learned my addr 172.21.15.29:0/902822765 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:57:49.646 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.643+0000 7f151d7c2640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562d5f127a50 0x562d5f12aed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.29:36684/0 (socket says 172.21.15.29:36684) 2024-03-16T11:57:49.647 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.643+0000 7f151dfc3640 1 --2- 172.21.15.29:0/902822765 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562d5f127130 0x562d5f12a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:57:49.647 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.643+0000 7f151e7c4640 1 -- 172.21.15.29:0/902822765 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562d5f127130 msgr2=0x562d5f12a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:57:49.647 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.643+0000 7f151e7c4640 1 --2- 172.21.15.29:0/902822765 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562d5f127130 0x562d5f12a990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:57:49.647 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.643+0000 7f151e7c4640 1 -- 172.21.15.29:0/902822765 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562d5f127a50 msgr2=0x562d5f12aed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:57:49.647 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.643+0000 7f151e7c4640 1 --2- 172.21.15.29:0/902822765 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562d5f127a50 0x562d5f12aed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:57:49.647 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.643+0000 7f151e7c4640 1 -- 172.21.15.29:0/902822765 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562d5f1a6250 con 0x562d5f130820 2024-03-16T11:57:49.647 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.643+0000 7f151d7c2640 1 --2- 172.21.15.29:0/902822765 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562d5f127a50 0x562d5f12aed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:57:49.647 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.643+0000 7f151dfc3640 1 --2- 172.21.15.29:0/902822765 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562d5f127130 0x562d5f12a990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:57:49.647 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.643+0000 7f151e7c4640 1 --2- 172.21.15.29:0/902822765 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562d5f130820 0x562d5f1a5d10 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f151002c700 tx=0x7f151002c0f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d415e93e84c7909 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:57:49.647 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.643+0000 7f15017fa640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-16T11:57:49.647 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.643+0000 7f15017fa640 1 -- 172.21.15.29:0/902822765 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1510039040 con 0x562d5f130820 2024-03-16T11:57:49.647 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.643+0000 7f15017fa640 1 -- 172.21.15.29:0/902822765 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8556+0+0 (secure 0 0 0) 0x7f15100045d0 con 0x562d5f130820 2024-03-16T11:57:49.647 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.643+0000 7f1522e0e740 1 -- 172.21.15.29:0/902822765 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f151409ae00 con 0x562d5f130820 2024-03-16T11:57:49.647 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.643+0000 7f15017fa640 1 -- 172.21.15.29:0/902822765 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f15100047d0 con 0x562d5f130820 2024-03-16T11:57:49.648 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.643+0000 7f1522e0e740 10 client.15978 fetch_fsmap learned FSMap version 24 2024-03-16T11:57:49.648 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.643+0000 7f1522e0e740 1 -- 172.21.15.29:0/902822765 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x562d5f1a7400 con 0x562d5f130820 2024-03-16T11:57:49.648 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.644+0000 7f15017fa640 1 -- 172.21.15.29:0/902822765 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f151409ae00 con 0x562d5f130820 2024-03-16T11:57:49.648 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.644+0000 7f15017fa640 1 -- 172.21.15.29:0/902822765 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7f151003f770 con 0x562d5f130820 2024-03-16T11:57:49.648 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.644+0000 7f1522e0e740 10 client.15978 fetch_fsmap finished waiting for FSMap version 24 2024-03-16T11:57:49.648 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.644+0000 7f1522e0e740 10 client.15978 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T11:57:49.648 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.644+0000 7f1522e0e740 20 client.15978 populate_metadata read hostname 'smithi029' 2024-03-16T11:57:49.648 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.644+0000 7f1522e0e740 1 --2- 172.21.15.29:0/902822765 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x562d5f1a7ba0 0x562d5f1a9fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:57:49.648 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.644+0000 7f1522e0e740 4 client.15978 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-16T11:57:49.648 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.644+0000 7f1522e0e740 1 -- 172.21.15.29:0/902822765 --> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] -- command(tid 0: {"prefix": "flush journal"}) -- 0x562d5f0932d0 con 0x562d5f1a7ba0 2024-03-16T11:57:49.648 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.644+0000 7f151dfc3640 1 --2- 172.21.15.29:0/902822765 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x562d5f1a7ba0 0x562d5f1a9fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:57:49.648 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.644+0000 7f15017fa640 10 client.15978 ms_handle_connect on v2:172.21.15.152:6838/1548731833 2024-03-16T11:57:49.648 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.644+0000 7f151dfc3640 1 --2- 172.21.15.29:0/902822765 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x562d5f1a7ba0 0x562d5f1a9fc0 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9a9a76763b474be2 server_cookie=fe29d0fb5e0af20b in_seq=0 out_seq=0 2024-03-16T11:57:49.790 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.791+0000 7f15017fa640 1 -- 172.21.15.29:0/902822765 <== mds.0 v2:172.21.15.152:6838/1548731833 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x562d5f0932d0 con 0x562d5f1a7ba0 2024-03-16T11:57:49.790 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.791+0000 7f15017fa640 10 client.15978 handle_command_reply: tid=0 2024-03-16T11:57:49.790 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.791+0000 7f1522e0e740 1 client.15978 shutdown 2024-03-16T11:57:49.791 INFO:tasks.workunit.client.0.smithi029.stdout: 1 698537 10.83 MB/sec execute 214 sec latency 64.782 ms 2024-03-16T11:57:49.792 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.791+0000 7f1522e0e740 1 -- 172.21.15.29:0/902822765 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562d5f130820 msgr2=0x562d5f1a5d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:57:49.792 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.791+0000 7f1522e0e740 1 --2- 172.21.15.29:0/902822765 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562d5f130820 0x562d5f1a5d10 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f151002c700 tx=0x7f151002c0f0 comp rx=0 tx=0).stop 2024-03-16T11:57:49.792 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.791+0000 7f1522e0e740 1 -- 172.21.15.29:0/902822765 shutdown_connections 2024-03-16T11:57:49.792 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.791+0000 7f1522e0e740 1 --2- 172.21.15.29:0/902822765 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562d5f130820 0x562d5f1a5d10 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:57:49.792 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.791+0000 7f1522e0e740 1 --2- 172.21.15.29:0/902822765 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x562d5f1a7ba0 0x562d5f1a9fc0 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:57:49.792 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.791+0000 7f1522e0e740 1 --2- 172.21.15.29:0/902822765 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562d5f127a50 0x562d5f12aed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:57:49.792 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.791+0000 7f1522e0e740 1 --2- 172.21.15.29:0/902822765 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562d5f127130 0x562d5f12a990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:57:49.792 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.791+0000 7f15017fa640 0 client.15978 ms_handle_reset on v2:172.21.15.152:6838/1548731833 2024-03-16T11:57:49.792 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.791+0000 7f1522e0e740 1 -- 172.21.15.29:0/902822765 >> 172.21.15.29:0/902822765 conn(0x562d5f121a50 msgr2=0x562d5f12d7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:57:49.792 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.791+0000 7f1522e0e740 1 -- 172.21.15.29:0/902822765 shutdown_connections 2024-03-16T11:57:49.792 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.791+0000 7f1522e0e740 1 -- 172.21.15.29:0/902822765 wait complete. 2024-03-16T11:57:49.792 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.792+0000 7f1522e0e740 20 client.15978 trim_cache size 0 max 16384 2024-03-16T11:57:49.793 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:{ 2024-03-16T11:57:49.793 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "message": "", 2024-03-16T11:57:49.793 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "return_code": 0 2024-03-16T11:57:49.793 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:} 2024-03-16T11:57:49.794 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.792+0000 7f1520227640 1 -- 172.21.15.29:0/3487570614 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f15040a2a10 msgr2=0x7f15040a4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:57:49.794 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.792+0000 7f1520227640 1 --2- 172.21.15.29:0/3487570614 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f15040a2a10 0x7f15040a4ed0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f1510005cc0 tx=0x7f1510005c50 comp rx=0 tx=0).stop 2024-03-16T11:57:49.794 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.792+0000 7f1520227640 1 -- 172.21.15.29:0/3487570614 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1518103990 msgr2=0x7f151819a1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:57:49.794 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.792+0000 7f1520227640 1 --2- 172.21.15.29:0/3487570614 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1518103990 0x7f151819a1e0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f151400e9f0 tx=0x7f151400eeb0 comp rx=0 tx=0).stop 2024-03-16T11:57:49.794 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.792+0000 7f1520227640 1 -- 172.21.15.29:0/3487570614 shutdown_connections 2024-03-16T11:57:49.794 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.792+0000 7f1520227640 1 --2- 172.21.15.29:0/3487570614 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f15040a2a10 0x7f15040a4ed0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:57:49.794 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.792+0000 7f1520227640 1 --2- 172.21.15.29:0/3487570614 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f15181099a0 0x7f151819eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:57:49.794 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.792+0000 7f1520227640 1 --2- 172.21.15.29:0/3487570614 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1518104350 0x7f151819a720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:57:49.794 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.792+0000 7f1520227640 1 --2- 172.21.15.29:0/3487570614 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1518103990 0x7f151819a1e0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:57:49.794 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.792+0000 7f1520227640 1 -- 172.21.15.29:0/3487570614 >> 172.21.15.29:0/3487570614 conn(0x7f15180ff460 msgr2=0x7f15180ff980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:57:49.794 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.792+0000 7f1520227640 1 -- 172.21.15.29:0/3487570614 shutdown_connections 2024-03-16T11:57:49.794 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:57:49.792+0000 7f1520227640 1 -- 172.21.15.29:0/3487570614 wait complete. 2024-03-16T11:57:50.791 INFO:tasks.workunit.client.0.smithi029.stdout: 1 700315 10.83 MB/sec execute 215 sec latency 16.516 ms 2024-03-16T11:57:50.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:57:50 smithi154 ceph-mon[36320]: pgmap v214: 97 pgs: 97 active+clean; 74 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 2.4 MiB/s wr, 265 op/s 2024-03-16T11:57:50.795 INFO:journalctl@ceph.mds.c.smithi154.stdout:Mar 16 11:57:50 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mds-c[66727]: ignoring --setuser ceph since I am not root 2024-03-16T11:57:50.795 INFO:journalctl@ceph.mds.c.smithi154.stdout:Mar 16 11:57:50 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mds-c[66727]: ignoring --setgroup ceph since I am not root 2024-03-16T11:57:50.795 INFO:journalctl@ceph.mds.c.smithi154.stdout:Mar 16 11:57:50 smithi154 ceph-mds[66731]: -- 172.21.15.154:0/1974893420 <== mon.1 v2:172.21.15.154:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x559676561a40 con 0x559676535000 2024-03-16T11:57:50.795 INFO:journalctl@ceph.mds.c.smithi154.stdout:Mar 16 11:57:50 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mds-c[66727]: starting mds.c at 2024-03-16T11:57:50.837 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:57:50 smithi029 ceph-mon[31975]: pgmap v214: 97 pgs: 97 active+clean; 74 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 2.4 MiB/s wr, 265 op/s 2024-03-16T11:57:50.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:57:50 smithi152 ceph-mon[36291]: pgmap v214: 97 pgs: 97 active+clean; 74 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 2.4 MiB/s wr, 265 op/s 2024-03-16T11:57:51.790 INFO:tasks.workunit.client.0.smithi029.stdout: 1 702481 10.83 MB/sec execute 216 sec latency 20.677 ms 2024-03-16T11:57:51.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:57:51 smithi154 ceph-mon[36320]: daemon mds.c restarted 2024-03-16T11:57:51.837 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:57:51 smithi029 ceph-mon[31975]: daemon mds.c restarted 2024-03-16T11:57:51.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:57:51 smithi152 ceph-mon[36291]: daemon mds.c restarted 2024-03-16T11:57:52.790 INFO:tasks.workunit.client.0.smithi029.stdout: 1 704043 10.82 MB/sec execute 217 sec latency 22.469 ms 2024-03-16T11:57:52.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:57:52 smithi154 ceph-mon[36320]: pgmap v215: 97 pgs: 97 active+clean; 64 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 3.9 MiB/s wr, 442 op/s 2024-03-16T11:57:52.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:57:52 smithi154 ceph-mon[36320]: osdmap e93: 12 total, 12 up, 12 in 2024-03-16T11:57:52.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:57:52 smithi154 ceph-mon[36320]: mds.? [v2:172.21.15.154:6836/733315344,v1:172.21.15.154:6837/733315344] up:boot 2024-03-16T11:57:52.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:57:52 smithi154 ceph-mon[36320]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 2 up:standby-replay 7 up:standby 2024-03-16T11:57:52.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:57:52 smithi154 ceph-mon[36320]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-16T11:57:52.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:57:52 smithi154 ceph-mon[36320]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-03-16T11:57:52.837 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:57:52 smithi029 ceph-mon[31975]: pgmap v215: 97 pgs: 97 active+clean; 64 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 3.9 MiB/s wr, 442 op/s 2024-03-16T11:57:52.837 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:57:52 smithi029 ceph-mon[31975]: osdmap e93: 12 total, 12 up, 12 in 2024-03-16T11:57:52.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:57:52 smithi029 ceph-mon[31975]: mds.? [v2:172.21.15.154:6836/733315344,v1:172.21.15.154:6837/733315344] up:boot 2024-03-16T11:57:52.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:57:52 smithi029 ceph-mon[31975]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 2 up:standby-replay 7 up:standby 2024-03-16T11:57:52.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:57:52 smithi029 ceph-mon[31975]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-16T11:57:52.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:57:52 smithi029 ceph-mon[31975]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-03-16T11:57:52.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:57:52 smithi152 ceph-mon[36291]: pgmap v215: 97 pgs: 97 active+clean; 64 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 3.9 MiB/s wr, 442 op/s 2024-03-16T11:57:52.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:57:52 smithi152 ceph-mon[36291]: osdmap e93: 12 total, 12 up, 12 in 2024-03-16T11:57:52.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:57:52 smithi152 ceph-mon[36291]: mds.? [v2:172.21.15.154:6836/733315344,v1:172.21.15.154:6837/733315344] up:boot 2024-03-16T11:57:52.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:57:52 smithi152 ceph-mon[36291]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 2 up:standby-replay 7 up:standby 2024-03-16T11:57:52.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:57:52 smithi152 ceph-mon[36291]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-16T11:57:52.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:57:52 smithi152 ceph-mon[36291]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-03-16T11:57:53.791 INFO:tasks.workunit.client.0.smithi029.stdout: 1 706144 10.82 MB/sec execute 218 sec latency 20.601 ms 2024-03-16T11:57:54.791 INFO:tasks.workunit.client.0.smithi029.stdout: 1 707397 10.81 MB/sec execute 219 sec latency 68.161 ms 2024-03-16T11:57:54.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:57:54 smithi154 ceph-mon[36320]: pgmap v217: 97 pgs: 97 active+clean; 64 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 3.2 MiB/s wr, 312 op/s 2024-03-16T11:57:54.837 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:57:54 smithi029 ceph-mon[31975]: pgmap v217: 97 pgs: 97 active+clean; 64 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 3.2 MiB/s wr, 312 op/s 2024-03-16T11:57:54.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:57:54 smithi152 ceph-mon[36291]: pgmap v217: 97 pgs: 97 active+clean; 64 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 3.2 MiB/s wr, 312 op/s 2024-03-16T11:57:55.791 INFO:tasks.workunit.client.0.smithi029.stdout: 1 709432 10.81 MB/sec execute 220 sec latency 25.042 ms 2024-03-16T11:57:56.791 INFO:tasks.workunit.client.0.smithi029.stdout: 1 711050 10.80 MB/sec execute 221 sec latency 25.605 ms 2024-03-16T11:57:56.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:57:56 smithi154 ceph-mon[36320]: pgmap v218: 97 pgs: 97 active+clean; 64 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 3.2 MiB/s wr, 312 op/s 2024-03-16T11:57:56.837 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:57:56 smithi029 ceph-mon[31975]: pgmap v218: 97 pgs: 97 active+clean; 64 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 3.2 MiB/s wr, 312 op/s 2024-03-16T11:57:56.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:57:56 smithi152 ceph-mon[36291]: pgmap v218: 97 pgs: 97 active+clean; 64 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 3.2 MiB/s wr, 312 op/s 2024-03-16T11:57:57.791 INFO:tasks.workunit.client.0.smithi029.stdout: 1 713397 10.80 MB/sec execute 222 sec latency 16.714 ms 2024-03-16T11:57:58.791 INFO:tasks.workunit.client.0.smithi029.stdout: 1 714985 10.79 MB/sec execute 223 sec latency 23.593 ms 2024-03-16T11:57:58.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:57:58 smithi154 ceph-mon[36320]: pgmap v219: 97 pgs: 97 active+clean; 70 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 3.2 MiB/s wr, 315 op/s 2024-03-16T11:57:58.837 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:57:58 smithi029 ceph-mon[31975]: pgmap v219: 97 pgs: 97 active+clean; 70 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 3.2 MiB/s wr, 315 op/s 2024-03-16T11:57:58.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:57:58 smithi152 ceph-mon[36291]: pgmap v219: 97 pgs: 97 active+clean; 70 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 3.2 MiB/s wr, 315 op/s 2024-03-16T11:57:59.791 INFO:tasks.workunit.client.0.smithi029.stdout: 1 716745 10.79 MB/sec execute 224 sec latency 118.507 ms 2024-03-16T11:58:00.791 INFO:tasks.workunit.client.0.smithi029.stdout: 1 718035 10.77 MB/sec execute 225 sec latency 164.657 ms 2024-03-16T11:58:00.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:58:00 smithi154 ceph-mon[36320]: pgmap v220: 97 pgs: 97 active+clean; 70 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 3.2 MiB/s wr, 315 op/s 2024-03-16T11:58:00.837 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:58:00 smithi029 ceph-mon[31975]: pgmap v220: 97 pgs: 97 active+clean; 70 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 3.2 MiB/s wr, 315 op/s 2024-03-16T11:58:00.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:58:00 smithi152 ceph-mon[36291]: pgmap v220: 97 pgs: 97 active+clean; 70 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 3.2 MiB/s wr, 315 op/s 2024-03-16T11:58:01.791 INFO:tasks.workunit.client.0.smithi029.stdout: 1 719203 10.75 MB/sec execute 226 sec latency 23.713 ms 2024-03-16T11:58:02.791 INFO:tasks.workunit.client.0.smithi029.stdout: 1 721254 10.76 MB/sec execute 227 sec latency 22.274 ms 2024-03-16T11:58:02.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:58:02 smithi154 ceph-mon[36320]: pgmap v221: 97 pgs: 97 active+clean; 75 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 3.0 MiB/s wr, 226 op/s 2024-03-16T11:58:02.837 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:58:02 smithi029 ceph-mon[31975]: pgmap v221: 97 pgs: 97 active+clean; 75 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 3.0 MiB/s wr, 226 op/s 2024-03-16T11:58:02.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:58:02 smithi152 ceph-mon[36291]: pgmap v221: 97 pgs: 97 active+clean; 75 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 3.0 MiB/s wr, 226 op/s 2024-03-16T11:58:02.880 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:02.880+0000 7f202739f640 1 -- 172.21.15.29:0/1180435125 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2020107660 msgr2=0x7f2020107a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:58:02.880 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:02.880+0000 7f202739f640 1 --2- 172.21.15.29:0/1180435125 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2020107660 0x7f2020107a40 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f2018009cd0 tx=0x7f201802c0f0 comp rx=0 tx=0).stop 2024-03-16T11:58:02.880 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:02.881+0000 7f202739f640 1 -- 172.21.15.29:0/1180435125 shutdown_connections 2024-03-16T11:58:02.880 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:02.881+0000 7f202739f640 1 --2- 172.21.15.29:0/1180435125 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2020101f80 0x7f202010e490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:58:02.880 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:02.881+0000 7f202739f640 1 --2- 172.21.15.29:0/1180435125 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f20201015c0 0x7f2020101a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:58:02.880 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:02.881+0000 7f202739f640 1 --2- 172.21.15.29:0/1180435125 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2020107660 0x7f2020107a40 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:58:02.881 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:02.881+0000 7f202739f640 1 -- 172.21.15.29:0/1180435125 >> 172.21.15.29:0/1180435125 conn(0x7f20200fd0f0 msgr2=0x7f20200ff510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:58:02.881 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:02.881+0000 7f202739f640 1 -- 172.21.15.29:0/1180435125 shutdown_connections 2024-03-16T11:58:02.881 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:02.881+0000 7f202739f640 1 -- 172.21.15.29:0/1180435125 wait complete. 2024-03-16T11:58:02.881 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:02.881+0000 7f202739f640 1 Processor -- start 2024-03-16T11:58:02.881 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:02.881+0000 7f202739f640 1 -- start start 2024-03-16T11:58:02.882 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:02.881+0000 7f202739f640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f20201015c0 0x7f2020197e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:58:02.883 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:02.881+0000 7f202739f640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2020101f80 0x7f20201983a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:58:02.883 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:02.881+0000 7f202739f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2020107660 0x7f202019c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:58:02.883 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:02.881+0000 7f202739f640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f20201119e0 con 0x7f2020107660 2024-03-16T11:58:02.883 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:02.881+0000 7f202739f640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f2020111860 con 0x7f2020101f80 2024-03-16T11:58:02.883 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:02.882+0000 7f202739f640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f2020111b60 con 0x7f20201015c0 2024-03-16T11:58:02.883 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:02.882+0000 7f2026b9e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2020107660 0x7f202019c720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:58:02.883 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:02.882+0000 7f2026b9e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2020107660 0x7f202019c720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59822/0 (socket says 172.21.15.29:59822) 2024-03-16T11:58:02.883 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:02.882+0000 7f2026b9e640 1 -- 172.21.15.29:0/1383625386 learned_addr learned my addr 172.21.15.29:0/1383625386 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:58:02.883 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:02.882+0000 7f2025b9c640 1 --2- 172.21.15.29:0/1383625386 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2020101f80 0x7f20201983a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:58:02.883 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:02.882+0000 7f202639d640 1 --2- 172.21.15.29:0/1383625386 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f20201015c0 0x7f2020197e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:58:02.883 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:02.882+0000 7f2026b9e640 1 -- 172.21.15.29:0/1383625386 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f20201015c0 msgr2=0x7f2020197e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:58:02.883 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:02.882+0000 7f2026b9e640 1 --2- 172.21.15.29:0/1383625386 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f20201015c0 0x7f2020197e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:58:02.883 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:02.882+0000 7f2026b9e640 1 -- 172.21.15.29:0/1383625386 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2020101f80 msgr2=0x7f20201983a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:58:02.883 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:02.882+0000 7f2026b9e640 1 --2- 172.21.15.29:0/1383625386 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2020101f80 0x7f20201983a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:58:02.884 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:02.882+0000 7f2026b9e640 1 -- 172.21.15.29:0/1383625386 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f202019ce50 con 0x7f2020107660 2024-03-16T11:58:02.884 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:02.882+0000 7f2026b9e640 1 --2- 172.21.15.29:0/1383625386 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2020107660 0x7f202019c720 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f201000e970 tx=0x7f201000ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:58:02.884 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:02.882+0000 7f201f7fe640 1 -- 172.21.15.29:0/1383625386 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f201000ccc0 con 0x7f2020107660 2024-03-16T11:58:02.884 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:02.882+0000 7f202739f640 1 -- 172.21.15.29:0/1383625386 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f202019d140 con 0x7f2020107660 2024-03-16T11:58:02.884 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:02.882+0000 7f201f7fe640 1 -- 172.21.15.29:0/1383625386 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f201000ce60 con 0x7f2020107660 2024-03-16T11:58:02.884 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:02.882+0000 7f201f7fe640 1 -- 172.21.15.29:0/1383625386 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2010010490 con 0x7f2020107660 2024-03-16T11:58:02.884 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:02.882+0000 7f202739f640 1 -- 172.21.15.29:0/1383625386 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f20201a49d0 con 0x7f2020107660 2024-03-16T11:58:02.886 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:02.885+0000 7f201f7fe640 1 -- 172.21.15.29:0/1383625386 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7f2010005020 con 0x7f2020107660 2024-03-16T11:58:02.886 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:02.885+0000 7f202b1ea740 1 Processor -- start 2024-03-16T11:58:02.886 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:02.886+0000 7f202b1ea740 1 -- start start 2024-03-16T11:58:02.886 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:02.886+0000 7f202b1ea740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563bcc5d72c0 0x563bcc5d76a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:58:02.886 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:02.886+0000 7f202b1ea740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563bcc5d7be0 0x563bcc5e0470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:58:02.886 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:02.886+0000 7f202b1ea740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563bcc5e09b0 0x563bcc5e2da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:58:02.887 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:02.886+0000 7f202b1ea740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x563bcc53ba40 con 0x563bcc5d72c0 2024-03-16T11:58:02.887 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:02.886+0000 7f202b1ea740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x563bcc4e4920 con 0x563bcc5e09b0 2024-03-16T11:58:02.887 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:02.886+0000 7f202b1ea740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x563bcc537ae0 con 0x563bcc5d7be0 2024-03-16T11:58:02.887 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:02.886+0000 7f201f7fe640 1 --2- 172.21.15.29:0/1383625386 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f20080a2960 0x7f20080a4e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:58:02.887 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:02.886+0000 7f202639d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563bcc5d7be0 0x563bcc5e0470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:58:02.887 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:02.886+0000 7f2025b9c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563bcc5d72c0 0x563bcc5d76a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:58:02.887 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:02.886+0000 7f202639d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563bcc5d7be0 0x563bcc5e0470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.29:51684/0 (socket says 172.21.15.29:51684) 2024-03-16T11:58:02.887 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:02.886+0000 7f202639d640 1 -- 172.21.15.29:0/187734105 learned_addr learned my addr 172.21.15.29:0/187734105 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:58:02.887 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:02.886+0000 7f2026b9e640 1 --2- 172.21.15.29:0/1383625386 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f20080a2960 0x7f20080a4e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:58:02.887 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:02.886+0000 7f2025b9c640 1 --2- 172.21.15.29:0/187734105 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563bcc5e09b0 0x563bcc5e2da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:58:02.887 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:02.886+0000 7f201f7fe640 1 -- 172.21.15.29:0/1383625386 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 8512+0+0 (secure 0 0 0) 0x7f20100cf2c0 con 0x7f2020107660 2024-03-16T11:58:02.887 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:02.886+0000 7f202639d640 1 -- 172.21.15.29:0/187734105 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563bcc5e09b0 msgr2=0x563bcc5e2da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:58:02.887 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:02.886+0000 7f202639d640 1 --2- 172.21.15.29:0/187734105 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563bcc5e09b0 0x563bcc5e2da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:58:02.887 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:02.886+0000 7f202639d640 1 -- 172.21.15.29:0/187734105 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563bcc5d72c0 msgr2=0x563bcc5d76a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:58:02.887 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:02.886+0000 7f202639d640 1 --2- 172.21.15.29:0/187734105 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563bcc5d72c0 0x563bcc5d76a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:58:02.887 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:02.886+0000 7f202639d640 1 -- 172.21.15.29:0/187734105 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563bcc4fc460 con 0x563bcc5d7be0 2024-03-16T11:58:02.888 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:02.887+0000 7f202639d640 1 --2- 172.21.15.29:0/187734105 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563bcc5d7be0 0x563bcc5e0470 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f2018009ca0 tx=0x7f2018009880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bdba94b35209a631 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:58:02.888 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:02.887+0000 7f2026b9e640 1 --2- 172.21.15.29:0/1383625386 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f20080a2960 0x7f20080a4e20 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f201009b5f0 tx=0x7f201007c040 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:58:02.888 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:02.887+0000 7f201cbfe640 1 -- 172.21.15.29:0/187734105 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2018043070 con 0x563bcc5d7be0 2024-03-16T11:58:02.888 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:02.887+0000 7f201cbfe640 1 -- 172.21.15.29:0/187734105 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f201802cca0 con 0x563bcc5d7be0 2024-03-16T11:58:02.888 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:02.887+0000 7f201cbfe640 1 -- 172.21.15.29:0/187734105 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2018039040 con 0x563bcc5d7be0 2024-03-16T11:58:02.892 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:02.888+0000 7f202b1ea740 1 -- 172.21.15.29:0/187734105 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563bcc5d7be0 msgr2=0x563bcc5e0470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:58:02.892 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:02.888+0000 7f202b1ea740 1 --2- 172.21.15.29:0/187734105 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563bcc5d7be0 0x563bcc5e0470 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f2018009ca0 tx=0x7f2018009880 comp rx=0 tx=0).stop 2024-03-16T11:58:02.892 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:02.888+0000 7f202b1ea740 1 -- 172.21.15.29:0/187734105 shutdown_connections 2024-03-16T11:58:02.892 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:02.888+0000 7f202b1ea740 1 --2- 172.21.15.29:0/187734105 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563bcc5e09b0 0x563bcc5e2da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:58:02.892 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:02.888+0000 7f202b1ea740 1 --2- 172.21.15.29:0/187734105 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563bcc5d7be0 0x563bcc5e0470 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:58:02.892 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:02.888+0000 7f202b1ea740 1 --2- 172.21.15.29:0/187734105 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563bcc5d72c0 0x563bcc5d76a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:58:02.892 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:02.888+0000 7f202b1ea740 1 -- 172.21.15.29:0/187734105 >> 172.21.15.29:0/187734105 conn(0x563bcc5d1be0 msgr2=0x563bcc5d1fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:58:02.892 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:02.888+0000 7f202b1ea740 1 -- 172.21.15.29:0/187734105 shutdown_connections 2024-03-16T11:58:02.893 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:02.888+0000 7f202b1ea740 1 -- 172.21.15.29:0/187734105 wait complete. 2024-03-16T11:58:02.893 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:02.888+0000 7f202b1ea740 1 -- start start 2024-03-16T11:58:02.893 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:02.888+0000 7f202b1ea740 1 Processor -- start 2024-03-16T11:58:02.893 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:02.888+0000 7f202b1ea740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563bcc5d72c0 0x563bcc5dd910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:58:02.893 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:02.888+0000 7f202b1ea740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563bcc5d7be0 0x563bcc5dde50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:58:02.893 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:02.888+0000 7f202b1ea740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563bcc5e09b0 0x563bcc5de390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:58:02.893 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:02.888+0000 7f202b1ea740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x563bcc5e3580 con 0x563bcc5e09b0 2024-03-16T11:58:02.893 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:02.888+0000 7f202b1ea740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x563bcc4f7d70 con 0x563bcc5d7be0 2024-03-16T11:58:02.893 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:02.888+0000 7f202b1ea740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x563bcc5e3700 con 0x563bcc5d72c0 2024-03-16T11:58:02.893 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:02.888+0000 7f202b1ea740 10 client.0 _ll_register_callbacks cb 0x563bcc24fa00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T11:58:02.893 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:02.889+0000 7f2025b9c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563bcc5d72c0 0x563bcc5dd910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:58:02.893 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:02.889+0000 7f202639d640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563bcc5d7be0 0x563bcc5dde50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:58:02.893 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:02.889+0000 7f2025b9c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563bcc5e09b0 0x563bcc5de390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:58:02.893 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:02.889+0000 7f202639d640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563bcc5d7be0 0x563bcc5dde50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.29:57062/0 (socket says 172.21.15.29:57062) 2024-03-16T11:58:02.893 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:02.889+0000 7f2025b9c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563bcc5d72c0 0x563bcc5dd910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.29:51698/0 (socket says 172.21.15.29:51698) 2024-03-16T11:58:02.894 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:02.889+0000 7f202639d640 1 -- 172.21.15.29:0/2351652007 learned_addr learned my addr 172.21.15.29:0/2351652007 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:58:02.894 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:02.889+0000 7f2025b9c640 1 -- 172.21.15.29:0/2351652007 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563bcc5d7be0 msgr2=0x563bcc5dde50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:58:02.894 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:02.889+0000 7f2025b9c640 1 --2- 172.21.15.29:0/2351652007 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563bcc5d7be0 0x563bcc5dde50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:58:02.894 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:02.889+0000 7f2025b9c640 1 -- 172.21.15.29:0/2351652007 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563bcc5e09b0 msgr2=0x563bcc5de390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:58:02.894 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:02.889+0000 7f2025b9c640 1 --2- 172.21.15.29:0/2351652007 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563bcc5e09b0 0x563bcc5de390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:58:02.894 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:02.889+0000 7f2025b9c640 1 -- 172.21.15.29:0/2351652007 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563bcc250210 con 0x563bcc5d72c0 2024-03-16T11:58:02.894 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:02.889+0000 7f2025b9c640 1 --2- 172.21.15.29:0/2351652007 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563bcc5d72c0 0x563bcc5dd910 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f2014015fd0 tx=0x7f2014005f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a21105958ecd26a server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:58:02.894 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:02.889+0000 7f2005ffb640 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-16T11:58:02.894 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:02.889+0000 7f2005ffb640 1 -- 172.21.15.29:0/2351652007 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2014004050 con 0x563bcc5d72c0 2024-03-16T11:58:02.894 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:02.889+0000 7f202b1ea740 1 -- 172.21.15.29:0/2351652007 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f201802cca0 con 0x563bcc5d72c0 2024-03-16T11:58:02.894 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:02.889+0000 7f2005ffb640 1 -- 172.21.15.29:0/2351652007 <== mon.1 v2:172.21.15.154:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8512+0+0 (secure 0 0 0) 0x7f20140041f0 con 0x563bcc5d72c0 2024-03-16T11:58:02.894 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:02.889+0000 7f2005ffb640 1 -- 172.21.15.29:0/2351652007 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f20140043f0 con 0x563bcc5d72c0 2024-03-16T11:58:02.894 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:02.889+0000 7f202b1ea740 10 client.25843 fetch_fsmap learned FSMap version 26 2024-03-16T11:58:02.894 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:02.889+0000 7f202b1ea740 1 -- 172.21.15.29:0/2351652007 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x563bcc31c620 con 0x563bcc5d72c0 2024-03-16T11:58:02.895 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:02.890+0000 7f2005ffb640 1 -- 172.21.15.29:0/2351652007 <== mon.1 v2:172.21.15.154:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f201802cca0 con 0x563bcc5d72c0 2024-03-16T11:58:02.895 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:02.890+0000 7f2005ffb640 1 -- 172.21.15.29:0/2351652007 <== mon.1 v2:172.21.15.154:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f20140035f0 con 0x563bcc5d72c0 2024-03-16T11:58:02.895 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:02.890+0000 7f202b1ea740 10 client.25843 fetch_fsmap finished waiting for FSMap version 26 2024-03-16T11:58:02.895 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:02.890+0000 7f202b1ea740 10 client.25843 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T11:58:02.895 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:02.890+0000 7f202b1ea740 20 client.25843 populate_metadata read hostname 'smithi029' 2024-03-16T11:58:02.895 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:02.890+0000 7f202b1ea740 1 --2- 172.21.15.29:0/2351652007 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x563bcc5f0a40 0x563bcc5f2e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:58:02.895 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:02.890+0000 7f202b1ea740 4 client.25843 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T11:58:02.895 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:02.890+0000 7f202b1ea740 1 -- 172.21.15.29:0/2351652007 --> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563bcc548f80 con 0x563bcc5f0a40 2024-03-16T11:58:02.895 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:02.890+0000 7f202639d640 1 --2- 172.21.15.29:0/2351652007 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x563bcc5f0a40 0x563bcc5f2e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:58:02.895 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:02.891+0000 7f202639d640 1 --2- 172.21.15.29:0/2351652007 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x563bcc5f0a40 0x563bcc5f2e30 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=54406bead1d9cff3 server_cookie=d1a1e9947357e9f2 in_seq=0 out_seq=0 2024-03-16T11:58:02.895 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:02.891+0000 7f2005ffb640 10 client.25843 ms_handle_connect on v2:172.21.15.152:6838/1548731833 2024-03-16T11:58:02.895 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:02.892+0000 7f2005ffb640 1 -- 172.21.15.29:0/2351652007 <== mds.0 v2:172.21.15.152:6838/1548731833 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x563bcc548f80 con 0x563bcc5f0a40 2024-03-16T11:58:02.895 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:02.892+0000 7f2005ffb640 10 client.25843 handle_command_reply: tid=0 2024-03-16T11:58:02.895 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:02.892+0000 7f202b1ea740 1 client.25843 shutdown 2024-03-16T11:58:02.895 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:02.892+0000 7f202b1ea740 1 -- 172.21.15.29:0/2351652007 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563bcc5d72c0 msgr2=0x563bcc5dd910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:58:02.895 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:02.892+0000 7f202b1ea740 1 --2- 172.21.15.29:0/2351652007 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563bcc5d72c0 0x563bcc5dd910 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f2014015fd0 tx=0x7f2014005f90 comp rx=0 tx=0).stop 2024-03-16T11:58:02.895 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:02.892+0000 7f202b1ea740 1 -- 172.21.15.29:0/2351652007 shutdown_connections 2024-03-16T11:58:02.896 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:02.892+0000 7f202b1ea740 1 --2- 172.21.15.29:0/2351652007 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563bcc5e09b0 0x563bcc5de390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:58:02.896 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:02.892+0000 7f202b1ea740 1 --2- 172.21.15.29:0/2351652007 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x563bcc5f0a40 0x563bcc5f2e30 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:58:02.896 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:02.892+0000 7f202b1ea740 1 --2- 172.21.15.29:0/2351652007 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563bcc5d7be0 0x563bcc5dde50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:58:02.896 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:02.892+0000 7f202b1ea740 1 --2- 172.21.15.29:0/2351652007 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563bcc5d72c0 0x563bcc5dd910 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:58:02.896 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:02.892+0000 7f2005ffb640 0 client.25843 ms_handle_reset on v2:172.21.15.152:6838/1548731833 2024-03-16T11:58:02.896 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:02.892+0000 7f202b1ea740 1 -- 172.21.15.29:0/2351652007 >> 172.21.15.29:0/2351652007 conn(0x563bcc5d1be0 msgr2=0x563bcc5d3930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:58:02.896 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:02.892+0000 7f202b1ea740 1 -- 172.21.15.29:0/2351652007 shutdown_connections 2024-03-16T11:58:02.896 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:02.892+0000 7f202b1ea740 1 -- 172.21.15.29:0/2351652007 wait complete. 2024-03-16T11:58:02.896 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:02.892+0000 7f202b1ea740 20 client.25843 trim_cache size 0 max 16384 2024-03-16T11:58:02.910 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:02.907+0000 7f202b1ea740 1 Processor -- start 2024-03-16T11:58:02.910 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:02.907+0000 7f202b1ea740 1 -- start start 2024-03-16T11:58:02.910 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:02.907+0000 7f202b1ea740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563bcc5d72c0 0x563bcc5e0040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:58:02.910 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:02.907+0000 7f202b1ea740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563bcc5d7be0 0x563bcc5ec320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:58:02.910 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:02.907+0000 7f202b1ea740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563bcc5e09b0 0x563bcc5f86f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:58:02.911 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:02.907+0000 7f202b1ea740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f20201ad9e0 con 0x563bcc5e09b0 2024-03-16T11:58:02.911 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:02.907+0000 7f202b1ea740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f20201ab2b0 con 0x563bcc5d72c0 2024-03-16T11:58:02.911 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:02.907+0000 7f202b1ea740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f20201a90e0 con 0x563bcc5d7be0 2024-03-16T11:58:02.911 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:02.907+0000 7f202639d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563bcc5d7be0 0x563bcc5ec320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:58:02.911 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:02.907+0000 7f2025b9c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563bcc5e09b0 0x563bcc5f86f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:58:02.911 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:02.907+0000 7f202639d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563bcc5d7be0 0x563bcc5ec320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.29:51712/0 (socket says 172.21.15.29:51712) 2024-03-16T11:58:02.911 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:02.907+0000 7f202639d640 1 -- 172.21.15.29:0/1946048830 learned_addr learned my addr 172.21.15.29:0/1946048830 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:58:02.911 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:02.907+0000 7f2025b9c640 1 --2- 172.21.15.29:0/1946048830 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563bcc5d72c0 0x563bcc5e0040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:58:02.911 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:02.907+0000 7f202639d640 1 -- 172.21.15.29:0/1946048830 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563bcc5d72c0 msgr2=0x563bcc5e0040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:58:02.911 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:02.907+0000 7f202639d640 1 --2- 172.21.15.29:0/1946048830 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563bcc5d72c0 0x563bcc5e0040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:58:02.911 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:02.907+0000 7f202639d640 1 -- 172.21.15.29:0/1946048830 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563bcc5e09b0 msgr2=0x563bcc5f86f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:58:02.911 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:02.907+0000 7f202639d640 1 --2- 172.21.15.29:0/1946048830 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563bcc5e09b0 0x563bcc5f86f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:58:02.911 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:02.907+0000 7f202639d640 1 -- 172.21.15.29:0/1946048830 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563bcc5f31a0 con 0x563bcc5d7be0 2024-03-16T11:58:02.911 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:02.907+0000 7f2025b9c640 1 --2- 172.21.15.29:0/1946048830 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563bcc5d72c0 0x563bcc5e0040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T11:58:02.912 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:02.908+0000 7f202639d640 1 --2- 172.21.15.29:0/1946048830 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563bcc5d7be0 0x563bcc5ec320 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f2018003a80 tx=0x7f201802c0f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b1a0615fea330924 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:58:02.912 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:02.908+0000 7f2007fff640 1 -- 172.21.15.29:0/1946048830 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f201802de60 con 0x563bcc5d7be0 2024-03-16T11:58:02.912 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:02.908+0000 7f2007fff640 1 -- 172.21.15.29:0/1946048830 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f20180523b0 con 0x563bcc5d7be0 2024-03-16T11:58:02.912 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:02.908+0000 7f2007fff640 1 -- 172.21.15.29:0/1946048830 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2018039040 con 0x563bcc5d7be0 2024-03-16T11:58:02.912 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:02.908+0000 7f202b1ea740 1 -- 172.21.15.29:0/1946048830 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563bcc5d7be0 msgr2=0x563bcc5ec320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:58:02.912 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:02.908+0000 7f202b1ea740 1 --2- 172.21.15.29:0/1946048830 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563bcc5d7be0 0x563bcc5ec320 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f2018003a80 tx=0x7f201802c0f0 comp rx=0 tx=0).stop 2024-03-16T11:58:02.912 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:02.908+0000 7f202b1ea740 1 -- 172.21.15.29:0/1946048830 shutdown_connections 2024-03-16T11:58:02.912 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:02.908+0000 7f202b1ea740 1 --2- 172.21.15.29:0/1946048830 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563bcc5e09b0 0x563bcc5f86f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:58:02.912 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:02.908+0000 7f202b1ea740 1 --2- 172.21.15.29:0/1946048830 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563bcc5d7be0 0x563bcc5ec320 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:58:02.912 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:02.908+0000 7f202b1ea740 1 --2- 172.21.15.29:0/1946048830 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563bcc5d72c0 0x563bcc5e0040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:58:02.912 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:02.908+0000 7f202b1ea740 1 -- 172.21.15.29:0/1946048830 >> 172.21.15.29:0/1946048830 conn(0x563bcc5d1be0 msgr2=0x563bcc5d8bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:58:02.912 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:02.908+0000 7f202b1ea740 1 -- 172.21.15.29:0/1946048830 shutdown_connections 2024-03-16T11:58:02.912 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:02.908+0000 7f202b1ea740 1 -- 172.21.15.29:0/1946048830 wait complete. 2024-03-16T11:58:02.912 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:02.909+0000 7f202b1ea740 1 -- start start 2024-03-16T11:58:02.912 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:02.909+0000 7f202b1ea740 1 Processor -- start 2024-03-16T11:58:02.913 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:02.909+0000 7f202b1ea740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563bcc5d72c0 0x563bcc5e8d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:58:02.913 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:02.909+0000 7f202b1ea740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563bcc5d7be0 0x563bcc5e4100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:58:02.913 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:02.909+0000 7f202b1ea740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563bcc5e09b0 0x563bcc5e47f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:58:02.913 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:02.909+0000 7f202b1ea740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f20201a9770 con 0x563bcc5d72c0 2024-03-16T11:58:02.913 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:02.909+0000 7f202b1ea740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f20201aabe0 con 0x563bcc5d7be0 2024-03-16T11:58:02.913 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:02.909+0000 7f202b1ea740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f20201a9e40 con 0x563bcc5e09b0 2024-03-16T11:58:02.913 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:02.909+0000 7f202b1ea740 10 client.0 _ll_register_callbacks cb 0x7f20140099c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T11:58:02.913 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:02.909+0000 7f2025b9c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563bcc5d72c0 0x563bcc5e8d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:58:02.913 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:02.909+0000 7f2025b9c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563bcc5d72c0 0x563bcc5e8d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59860/0 (socket says 172.21.15.29:59860) 2024-03-16T11:58:02.913 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:02.909+0000 7f2025b9c640 1 -- 172.21.15.29:0/3577247145 learned_addr learned my addr 172.21.15.29:0/3577247145 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:58:02.913 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:02.909+0000 7f202639d640 1 --2- 172.21.15.29:0/3577247145 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563bcc5d7be0 0x563bcc5e4100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:58:02.913 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:02.909+0000 7f2025b9c640 1 --2- 172.21.15.29:0/3577247145 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563bcc5e09b0 0x563bcc5e47f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:58:02.913 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:02.909+0000 7f2025b9c640 1 -- 172.21.15.29:0/3577247145 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563bcc5e09b0 msgr2=0x563bcc5e47f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:58:02.913 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:02.909+0000 7f2025b9c640 1 --2- 172.21.15.29:0/3577247145 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563bcc5e09b0 0x563bcc5e47f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:58:02.914 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:02.909+0000 7f2025b9c640 1 -- 172.21.15.29:0/3577247145 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563bcc5d7be0 msgr2=0x563bcc5e4100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:58:02.914 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:02.909+0000 7f2025b9c640 1 --2- 172.21.15.29:0/3577247145 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563bcc5d7be0 0x563bcc5e4100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:58:02.914 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:02.909+0000 7f2025b9c640 1 -- 172.21.15.29:0/3577247145 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563bcc655d20 con 0x563bcc5d72c0 2024-03-16T11:58:02.914 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:02.909+0000 7f2025b9c640 1 --2- 172.21.15.29:0/3577247145 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563bcc5d72c0 0x563bcc5e8d90 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f2014003940 tx=0x7f2014015b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80f74308cec909c9 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:58:02.914 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:02.909+0000 7f2005ffb640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-16T11:58:02.914 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:02.910+0000 7f2005ffb640 1 -- 172.21.15.29:0/3577247145 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20140154a0 con 0x563bcc5d72c0 2024-03-16T11:58:02.914 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:02.910+0000 7f2005ffb640 1 -- 172.21.15.29:0/3577247145 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8512+0+0 (secure 0 0 0) 0x7f2014004050 con 0x563bcc5d72c0 2024-03-16T11:58:02.914 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:02.910+0000 7f202b1ea740 1 -- 172.21.15.29:0/3577247145 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f20180523b0 con 0x563bcc5d72c0 2024-03-16T11:58:02.914 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:02.910+0000 7f2005ffb640 1 -- 172.21.15.29:0/3577247145 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f2014004250 con 0x563bcc5d72c0 2024-03-16T11:58:02.914 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:02.910+0000 7f202b1ea740 10 client.16002 fetch_fsmap learned FSMap version 26 2024-03-16T11:58:02.914 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:02.910+0000 7f202b1ea740 1 -- 172.21.15.29:0/3577247145 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x563bcc656ed0 con 0x563bcc5d72c0 2024-03-16T11:58:02.914 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:02.910+0000 7f2005ffb640 1 -- 172.21.15.29:0/3577247145 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f20180523b0 con 0x563bcc5d72c0 2024-03-16T11:58:02.914 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:02.910+0000 7f2005ffb640 1 -- 172.21.15.29:0/3577247145 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f201400eb50 con 0x563bcc5d72c0 2024-03-16T11:58:02.914 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:02.910+0000 7f202b1ea740 10 client.16002 fetch_fsmap finished waiting for FSMap version 26 2024-03-16T11:58:02.915 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:02.910+0000 7f202b1ea740 10 client.16002 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T11:58:02.915 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:02.910+0000 7f202b1ea740 20 client.16002 populate_metadata read hostname 'smithi029' 2024-03-16T11:58:02.915 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:02.910+0000 7f202b1ea740 1 --2- 172.21.15.29:0/3577247145 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x563bcc657590 0x563bcc6599b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:58:02.915 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:02.910+0000 7f202b1ea740 4 client.16002 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-16T11:58:02.915 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:02.910+0000 7f202b1ea740 1 -- 172.21.15.29:0/3577247145 --> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] -- command(tid 0: {"prefix": "flush journal"}) -- 0x563bcc5432d0 con 0x563bcc657590 2024-03-16T11:58:02.915 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:02.910+0000 7f202639d640 1 --2- 172.21.15.29:0/3577247145 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x563bcc657590 0x563bcc6599b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:58:02.915 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:02.911+0000 7f2005ffb640 10 client.16002 ms_handle_connect on v2:172.21.15.152:6838/1548731833 2024-03-16T11:58:02.915 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:02.911+0000 7f202639d640 1 --2- 172.21.15.29:0/3577247145 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x563bcc657590 0x563bcc6599b0 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f777d685db6e64b6 server_cookie=2652e4e17f16384b in_seq=0 out_seq=0 2024-03-16T11:58:03.175 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:03.175+0000 7f2005ffb640 1 -- 172.21.15.29:0/3577247145 <== mds.0 v2:172.21.15.152:6838/1548731833 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x563bcc5432d0 con 0x563bcc657590 2024-03-16T11:58:03.175 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:03.175+0000 7f2005ffb640 10 client.16002 handle_command_reply: tid=0 2024-03-16T11:58:03.175 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:03.175+0000 7f202b1ea740 1 client.16002 shutdown 2024-03-16T11:58:03.175 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:03.175+0000 7f202b1ea740 1 -- 172.21.15.29:0/3577247145 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563bcc5d72c0 msgr2=0x563bcc5e8d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:58:03.175 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:03.175+0000 7f202b1ea740 1 --2- 172.21.15.29:0/3577247145 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563bcc5d72c0 0x563bcc5e8d90 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f2014003940 tx=0x7f2014015b20 comp rx=0 tx=0).stop 2024-03-16T11:58:03.175 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:03.175+0000 7f202b1ea740 1 -- 172.21.15.29:0/3577247145 shutdown_connections 2024-03-16T11:58:03.175 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:03.175+0000 7f202b1ea740 1 --2- 172.21.15.29:0/3577247145 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563bcc5e09b0 0x563bcc5e47f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:58:03.176 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:03.175+0000 7f202b1ea740 1 --2- 172.21.15.29:0/3577247145 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x563bcc657590 0x563bcc6599b0 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:58:03.176 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:03.175+0000 7f202b1ea740 1 --2- 172.21.15.29:0/3577247145 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563bcc5d7be0 0x563bcc5e4100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:58:03.176 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:03.175+0000 7f2005ffb640 0 client.16002 ms_handle_reset on v2:172.21.15.152:6838/1548731833 2024-03-16T11:58:03.176 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:03.175+0000 7f202b1ea740 1 --2- 172.21.15.29:0/3577247145 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563bcc5d72c0 0x563bcc5e8d90 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:58:03.176 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:03.175+0000 7f202b1ea740 1 -- 172.21.15.29:0/3577247145 >> 172.21.15.29:0/3577247145 conn(0x563bcc5d1be0 msgr2=0x563bcc5d03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:58:03.176 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:03.175+0000 7f202b1ea740 1 -- 172.21.15.29:0/3577247145 shutdown_connections 2024-03-16T11:58:03.176 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:03.176+0000 7f202b1ea740 1 -- 172.21.15.29:0/3577247145 wait complete. 2024-03-16T11:58:03.176 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:03.176+0000 7f202b1ea740 20 client.16002 trim_cache size 0 max 16384 2024-03-16T11:58:03.176 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:{ 2024-03-16T11:58:03.176 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "message": "", 2024-03-16T11:58:03.176 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "return_code": 0 2024-03-16T11:58:03.176 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:} 2024-03-16T11:58:03.177 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:03.176+0000 7f202739f640 1 -- 172.21.15.29:0/1383625386 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f20080a2960 msgr2=0x7f20080a4e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:58:03.177 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:03.176+0000 7f202739f640 1 --2- 172.21.15.29:0/1383625386 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f20080a2960 0x7f20080a4e20 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f201009b5f0 tx=0x7f201007c040 comp rx=0 tx=0).stop 2024-03-16T11:58:03.177 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:03.176+0000 7f202739f640 1 -- 172.21.15.29:0/1383625386 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2020107660 msgr2=0x7f202019c720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:58:03.177 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:03.176+0000 7f202739f640 1 --2- 172.21.15.29:0/1383625386 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2020107660 0x7f202019c720 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f201000e970 tx=0x7f201000ed70 comp rx=0 tx=0).stop 2024-03-16T11:58:03.177 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:03.176+0000 7f202739f640 1 -- 172.21.15.29:0/1383625386 shutdown_connections 2024-03-16T11:58:03.178 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:03.176+0000 7f202739f640 1 --2- 172.21.15.29:0/1383625386 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f20080a2960 0x7f20080a4e20 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:58:03.178 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:03.176+0000 7f202739f640 1 --2- 172.21.15.29:0/1383625386 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2020107660 0x7f202019c720 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:58:03.178 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:03.176+0000 7f202739f640 1 --2- 172.21.15.29:0/1383625386 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2020101f80 0x7f20201983a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:58:03.178 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:03.176+0000 7f202739f640 1 --2- 172.21.15.29:0/1383625386 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f20201015c0 0x7f2020197e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:58:03.178 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:03.176+0000 7f202739f640 1 -- 172.21.15.29:0/1383625386 >> 172.21.15.29:0/1383625386 conn(0x7f20200fd0f0 msgr2=0x7f20200fe820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:58:03.178 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:03.176+0000 7f202739f640 1 -- 172.21.15.29:0/1383625386 shutdown_connections 2024-03-16T11:58:03.178 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:03.176+0000 7f202739f640 1 -- 172.21.15.29:0/1383625386 wait complete. 2024-03-16T11:58:03.791 INFO:tasks.workunit.client.0.smithi029.stdout: 1 722560 10.74 MB/sec execute 228 sec latency 77.218 ms 2024-03-16T11:58:04.791 INFO:tasks.workunit.client.0.smithi029.stdout: 1 724214 10.73 MB/sec execute 229 sec latency 67.492 ms 2024-03-16T11:58:04.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:58:04 smithi154 ceph-mon[36320]: pgmap v222: 97 pgs: 97 active+clean; 75 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 2.6 MiB/s wr, 196 op/s 2024-03-16T11:58:04.837 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:58:04 smithi029 ceph-mon[31975]: pgmap v222: 97 pgs: 97 active+clean; 75 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 2.6 MiB/s wr, 196 op/s 2024-03-16T11:58:04.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:58:04 smithi152 ceph-mon[36291]: pgmap v222: 97 pgs: 97 active+clean; 75 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 2.6 MiB/s wr, 196 op/s 2024-03-16T11:58:05.791 INFO:tasks.workunit.client.0.smithi029.stdout: 1 724712 10.70 MB/sec execute 230 sec latency 32.327 ms 2024-03-16T11:58:05.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:58:05 smithi154 ceph-mon[36320]: osdmap e94: 12 total, 12 up, 12 in 2024-03-16T11:58:05.837 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:58:05 smithi029 ceph-mon[31975]: osdmap e94: 12 total, 12 up, 12 in 2024-03-16T11:58:05.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:58:05 smithi152 ceph-mon[36291]: osdmap e94: 12 total, 12 up, 12 in 2024-03-16T11:58:06.791 INFO:tasks.workunit.client.0.smithi029.stdout: 1 725472 10.67 MB/sec execute 231 sec latency 39.769 ms 2024-03-16T11:58:06.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:58:06 smithi154 ceph-mon[36320]: pgmap v224: 97 pgs: 97 active+clean; 75 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 3.0 MiB/s wr, 226 op/s 2024-03-16T11:58:06.837 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:58:06 smithi029 ceph-mon[31975]: pgmap v224: 97 pgs: 97 active+clean; 75 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 3.0 MiB/s wr, 226 op/s 2024-03-16T11:58:06.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:58:06 smithi152 ceph-mon[36291]: pgmap v224: 97 pgs: 97 active+clean; 75 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 3.0 MiB/s wr, 226 op/s 2024-03-16T11:58:07.791 INFO:tasks.workunit.client.0.smithi029.stdout: 1 727571 10.67 MB/sec execute 232 sec latency 19.581 ms 2024-03-16T11:58:07.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:58:07 smithi154 ceph-mon[36320]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T11:58:07.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:58:07 smithi154 ceph-mon[36320]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:58:07.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:58:07 smithi154 ceph-mon[36320]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T11:58:07.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:58:07 smithi154 ceph-mon[36320]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' 2024-03-16T11:58:07.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:58:07 smithi029 ceph-mon[31975]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T11:58:07.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:58:07 smithi029 ceph-mon[31975]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:58:07.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:58:07 smithi029 ceph-mon[31975]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T11:58:07.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:58:07 smithi029 ceph-mon[31975]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' 2024-03-16T11:58:07.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:58:07 smithi152 ceph-mon[36291]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T11:58:07.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:58:07 smithi152 ceph-mon[36291]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:58:07.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:58:07 smithi152 ceph-mon[36291]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T11:58:07.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:58:07 smithi152 ceph-mon[36291]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' 2024-03-16T11:58:08.791 INFO:tasks.workunit.client.0.smithi029.stdout: 1 729116 10.66 MB/sec execute 233 sec latency 26.465 ms 2024-03-16T11:58:08.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:58:08 smithi154 ceph-mon[36320]: pgmap v225: 97 pgs: 36 active+clean+snaptrim_wait, 45 active+clean, 16 active+clean+snaptrim; 67 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 3.3 MiB/s wr, 348 op/s 2024-03-16T11:58:08.837 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:58:08 smithi029 ceph-mon[31975]: pgmap v225: 97 pgs: 36 active+clean+snaptrim_wait, 45 active+clean, 16 active+clean+snaptrim; 67 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 3.3 MiB/s wr, 348 op/s 2024-03-16T11:58:08.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:58:08 smithi152 ceph-mon[36291]: pgmap v225: 97 pgs: 36 active+clean+snaptrim_wait, 45 active+clean, 16 active+clean+snaptrim; 67 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 3.3 MiB/s wr, 348 op/s 2024-03-16T11:58:09.791 INFO:tasks.workunit.client.0.smithi029.stdout: 1 730956 10.66 MB/sec execute 234 sec latency 131.687 ms 2024-03-16T11:58:10.791 INFO:tasks.workunit.client.0.smithi029.stdout: 1 732621 10.65 MB/sec execute 235 sec latency 21.343 ms 2024-03-16T11:58:10.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:58:10 smithi154 ceph-mon[36320]: pgmap v226: 97 pgs: 36 active+clean+snaptrim_wait, 45 active+clean, 16 active+clean+snaptrim; 67 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 3.3 MiB/s wr, 348 op/s 2024-03-16T11:58:10.837 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:58:10 smithi029 ceph-mon[31975]: pgmap v226: 97 pgs: 36 active+clean+snaptrim_wait, 45 active+clean, 16 active+clean+snaptrim; 67 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 3.3 MiB/s wr, 348 op/s 2024-03-16T11:58:10.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:58:10 smithi152 ceph-mon[36291]: pgmap v226: 97 pgs: 36 active+clean+snaptrim_wait, 45 active+clean, 16 active+clean+snaptrim; 67 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 3.3 MiB/s wr, 348 op/s 2024-03-16T11:58:11.791 INFO:tasks.workunit.client.0.smithi029.stdout: 1 734669 10.65 MB/sec execute 236 sec latency 75.054 ms 2024-03-16T11:58:12.791 INFO:tasks.workunit.client.0.smithi029.stdout: 1 736130 10.64 MB/sec execute 237 sec latency 20.464 ms 2024-03-16T11:58:12.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:58:12 smithi154 ceph-mon[36320]: pgmap v227: 97 pgs: 97 active+clean; 74 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 3.1 MiB/s wr, 321 op/s 2024-03-16T11:58:12.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:58:12 smithi154 ceph-mon[36320]: osdmap e95: 12 total, 12 up, 12 in 2024-03-16T11:58:12.837 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:58:12 smithi029 ceph-mon[31975]: pgmap v227: 97 pgs: 97 active+clean; 74 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 3.1 MiB/s wr, 321 op/s 2024-03-16T11:58:12.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:58:12 smithi029 ceph-mon[31975]: osdmap e95: 12 total, 12 up, 12 in 2024-03-16T11:58:12.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:58:12 smithi152 ceph-mon[36291]: pgmap v227: 97 pgs: 97 active+clean; 74 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 3.1 MiB/s wr, 321 op/s 2024-03-16T11:58:12.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:58:12 smithi152 ceph-mon[36291]: osdmap e95: 12 total, 12 up, 12 in 2024-03-16T11:58:13.792 INFO:tasks.workunit.client.0.smithi029.stdout: 1 738528 10.66 MB/sec execute 238 sec latency 18.266 ms 2024-03-16T11:58:14.792 INFO:tasks.workunit.client.0.smithi029.stdout: 1 739655 10.63 MB/sec execute 239 sec latency 62.137 ms 2024-03-16T11:58:14.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:58:14 smithi154 ceph-mon[36320]: pgmap v229: 97 pgs: 97 active+clean; 74 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 3.6 MiB/s wr, 377 op/s 2024-03-16T11:58:14.837 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:58:14 smithi029 ceph-mon[31975]: pgmap v229: 97 pgs: 97 active+clean; 74 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 3.6 MiB/s wr, 377 op/s 2024-03-16T11:58:14.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:58:14 smithi152 ceph-mon[36291]: pgmap v229: 97 pgs: 97 active+clean; 74 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 3.6 MiB/s wr, 377 op/s 2024-03-16T11:58:15.792 INFO:tasks.workunit.client.0.smithi029.stdout: 1 741853 10.64 MB/sec execute 240 sec latency 19.948 ms 2024-03-16T11:58:16.269 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:16.267+0000 7efe854ff640 1 -- 172.21.15.29:0/1085043264 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efe80102170 msgr2=0x7efe8010e680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:58:16.269 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:16.267+0000 7efe854ff640 1 --2- 172.21.15.29:0/1085043264 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efe80102170 0x7efe8010e680 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7efe78009cd0 tx=0x7efe7802c0f0 comp rx=0 tx=0).stop 2024-03-16T11:58:16.269 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:16.267+0000 7efe854ff640 1 -- 172.21.15.29:0/1085043264 shutdown_connections 2024-03-16T11:58:16.269 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:16.267+0000 7efe854ff640 1 --2- 172.21.15.29:0/1085043264 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efe80102170 0x7efe8010e680 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:58:16.269 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:16.267+0000 7efe854ff640 1 --2- 172.21.15.29:0/1085043264 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efe801017b0 0x7efe80101c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:58:16.269 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:16.267+0000 7efe854ff640 1 --2- 172.21.15.29:0/1085043264 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7efe801077c0 0x7efe80107ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:58:16.269 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:16.267+0000 7efe854ff640 1 -- 172.21.15.29:0/1085043264 >> 172.21.15.29:0/1085043264 conn(0x7efe800fd2c0 msgr2=0x7efe800ff6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:58:16.269 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:16.267+0000 7efe854ff640 1 -- 172.21.15.29:0/1085043264 shutdown_connections 2024-03-16T11:58:16.269 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:16.267+0000 7efe854ff640 1 -- 172.21.15.29:0/1085043264 wait complete. 2024-03-16T11:58:16.269 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:16.268+0000 7efe854ff640 1 Processor -- start 2024-03-16T11:58:16.270 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:16.268+0000 7efe854ff640 1 -- start start 2024-03-16T11:58:16.270 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:16.268+0000 7efe854ff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efe801017b0 0x7efe80073ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:58:16.270 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:16.268+0000 7efe854ff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efe80102170 0x7efe80074020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:58:16.270 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:16.268+0000 7efe854ff640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7efe801077c0 0x7efe80074560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:58:16.270 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:16.268+0000 7efe854ff640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7efe80111bd0 con 0x7efe80102170 2024-03-16T11:58:16.270 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:16.268+0000 7efe854ff640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7efe80111a50 con 0x7efe801077c0 2024-03-16T11:58:16.270 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:16.268+0000 7efe854ff640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7efe80111d50 con 0x7efe801017b0 2024-03-16T11:58:16.270 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:16.268+0000 7efe77fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efe80102170 0x7efe80074020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:58:16.270 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:16.268+0000 7efe77fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efe80102170 0x7efe80074020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39484/0 (socket says 172.21.15.29:39484) 2024-03-16T11:58:16.270 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:16.268+0000 7efe77fff640 1 -- 172.21.15.29:0/622389303 learned_addr learned my addr 172.21.15.29:0/622389303 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:58:16.270 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:16.268+0000 7efe7effd640 1 --2- 172.21.15.29:0/622389303 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efe801017b0 0x7efe80073ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:58:16.270 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:16.268+0000 7efe7f7fe640 1 --2- 172.21.15.29:0/622389303 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7efe801077c0 0x7efe80074560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:58:16.270 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:16.269+0000 7efe77fff640 1 -- 172.21.15.29:0/622389303 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efe801017b0 msgr2=0x7efe80073ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:58:16.271 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:16.269+0000 7efe77fff640 1 --2- 172.21.15.29:0/622389303 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efe801017b0 0x7efe80073ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:58:16.271 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:16.269+0000 7efe77fff640 1 -- 172.21.15.29:0/622389303 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7efe801077c0 msgr2=0x7efe80074560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:58:16.271 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:16.269+0000 7efe77fff640 1 --2- 172.21.15.29:0/622389303 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7efe801077c0 0x7efe80074560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:58:16.271 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:16.269+0000 7efe77fff640 1 -- 172.21.15.29:0/622389303 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe801a6490 con 0x7efe80102170 2024-03-16T11:58:16.271 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:16.269+0000 7efe7effd640 1 --2- 172.21.15.29:0/622389303 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efe801017b0 0x7efe80073ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:58:16.271 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:16.269+0000 7efe77fff640 1 --2- 172.21.15.29:0/622389303 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efe80102170 0x7efe80074020 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7efe6800e9f0 tx=0x7efe6800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:58:16.271 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:16.269+0000 7efe7cff9640 1 -- 172.21.15.29:0/622389303 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe6800cd80 con 0x7efe80102170 2024-03-16T11:58:16.271 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:16.269+0000 7efe854ff640 1 -- 172.21.15.29:0/622389303 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efe801a6780 con 0x7efe80102170 2024-03-16T11:58:16.271 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:16.269+0000 7efe7cff9640 1 -- 172.21.15.29:0/622389303 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efe68005500 con 0x7efe80102170 2024-03-16T11:58:16.271 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:16.269+0000 7efe7cff9640 1 -- 172.21.15.29:0/622389303 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe68010690 con 0x7efe80102170 2024-03-16T11:58:16.271 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:16.269+0000 7efe854ff640 1 -- 172.21.15.29:0/622389303 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efe801a6c80 con 0x7efe80102170 2024-03-16T11:58:16.277 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:16.271+0000 7efe7cff9640 1 -- 172.21.15.29:0/622389303 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7efe680046c0 con 0x7efe80102170 2024-03-16T11:58:16.277 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:16.272+0000 7efe881c9740 1 Processor -- start 2024-03-16T11:58:16.277 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:16.272+0000 7efe881c9740 1 -- start start 2024-03-16T11:58:16.277 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:16.272+0000 7efe7cff9640 1 --2- 172.21.15.29:0/622389303 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7efe6c0a2a10 0x7efe6c0a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:58:16.277 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:16.272+0000 7efe7cff9640 1 -- 172.21.15.29:0/622389303 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8556+0+0 (secure 0 0 0) 0x7efe680cf2e0 con 0x7efe80102170 2024-03-16T11:58:16.277 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:16.272+0000 7efe7f7fe640 1 --2- 172.21.15.29:0/622389303 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7efe6c0a2a10 0x7efe6c0a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:58:16.277 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:16.272+0000 7efe881c9740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564a1d980130 0x564a1d980510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:58:16.277 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:16.272+0000 7efe881c9740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564a1d980a50 0x564a1d9892e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:58:16.277 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:16.272+0000 7efe881c9740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564a1d989820 0x564a1d98bc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:58:16.278 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:16.272+0000 7efe881c9740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x564a1d8e4a40 con 0x564a1d989820 2024-03-16T11:58:16.278 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:16.272+0000 7efe881c9740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x564a1d88d920 con 0x564a1d980130 2024-03-16T11:58:16.278 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:16.272+0000 7efe881c9740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x564a1d8e0ae0 con 0x564a1d980a50 2024-03-16T11:58:16.278 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:16.272+0000 7efe7effd640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564a1d980130 0x564a1d980510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:58:16.278 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:16.272+0000 7efe77fff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564a1d980a50 0x564a1d9892e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:58:16.278 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:16.272+0000 7efe7f7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564a1d989820 0x564a1d98bc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:58:16.278 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:16.272+0000 7efe77fff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564a1d980a50 0x564a1d9892e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.29:37836/0 (socket says 172.21.15.29:37836) 2024-03-16T11:58:16.278 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:16.272+0000 7efe77fff640 1 -- 172.21.15.29:0/4196046436 learned_addr learned my addr 172.21.15.29:0/4196046436 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:58:16.278 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:16.273+0000 7efe77fff640 1 -- 172.21.15.29:0/4196046436 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564a1d980130 msgr2=0x564a1d980510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:58:16.278 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:16.273+0000 7efe77fff640 1 --2- 172.21.15.29:0/4196046436 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564a1d980130 0x564a1d980510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:58:16.278 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:16.273+0000 7efe77fff640 1 -- 172.21.15.29:0/4196046436 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564a1d989820 msgr2=0x564a1d98bc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:58:16.278 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:16.273+0000 7efe77fff640 1 --2- 172.21.15.29:0/4196046436 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564a1d989820 0x564a1d98bc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:58:16.278 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:16.273+0000 7efe77fff640 1 -- 172.21.15.29:0/4196046436 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564a1d8a5460 con 0x564a1d980a50 2024-03-16T11:58:16.278 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:16.273+0000 7efe7f7fe640 1 --2- 172.21.15.29:0/622389303 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7efe6c0a2a10 0x7efe6c0a4ed0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7efe78005cc0 tx=0x7efe7802ce80 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:58:16.279 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:16.273+0000 7efe77fff640 1 --2- 172.21.15.29:0/4196046436 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564a1d980a50 0x564a1d9892e0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7efe68098290 tx=0x7efe6809b2e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e2fd90fade7fd2e1 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:58:16.279 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:16.273+0000 7efe757fa640 1 -- 172.21.15.29:0/4196046436 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe6809fa10 con 0x564a1d980a50 2024-03-16T11:58:16.279 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:16.273+0000 7efe757fa640 1 -- 172.21.15.29:0/4196046436 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efe680a4060 con 0x564a1d980a50 2024-03-16T11:58:16.279 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:16.273+0000 7efe757fa640 1 -- 172.21.15.29:0/4196046436 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe6809bb20 con 0x564a1d980a50 2024-03-16T11:58:16.279 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:16.274+0000 7efe881c9740 1 -- 172.21.15.29:0/4196046436 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564a1d980a50 msgr2=0x564a1d9892e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:58:16.279 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:16.274+0000 7efe881c9740 1 --2- 172.21.15.29:0/4196046436 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564a1d980a50 0x564a1d9892e0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7efe68098290 tx=0x7efe6809b2e0 comp rx=0 tx=0).stop 2024-03-16T11:58:16.279 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:16.274+0000 7efe881c9740 1 -- 172.21.15.29:0/4196046436 shutdown_connections 2024-03-16T11:58:16.279 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:16.274+0000 7efe881c9740 1 --2- 172.21.15.29:0/4196046436 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564a1d989820 0x564a1d98bc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:58:16.279 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:16.274+0000 7efe881c9740 1 --2- 172.21.15.29:0/4196046436 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564a1d980a50 0x564a1d9892e0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:58:16.279 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:16.274+0000 7efe881c9740 1 --2- 172.21.15.29:0/4196046436 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564a1d980130 0x564a1d980510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:58:16.279 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:16.274+0000 7efe881c9740 1 -- 172.21.15.29:0/4196046436 >> 172.21.15.29:0/4196046436 conn(0x564a1d97aa50 msgr2=0x564a1d97ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:58:16.279 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:16.274+0000 7efe881c9740 1 -- 172.21.15.29:0/4196046436 shutdown_connections 2024-03-16T11:58:16.279 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:16.274+0000 7efe881c9740 1 -- 172.21.15.29:0/4196046436 wait complete. 2024-03-16T11:58:16.279 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:16.274+0000 7efe881c9740 1 -- start start 2024-03-16T11:58:16.280 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:16.274+0000 7efe881c9740 1 Processor -- start 2024-03-16T11:58:16.280 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:16.275+0000 7efe881c9740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564a1d980130 0x564a1d986990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:58:16.280 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:16.275+0000 7efe881c9740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564a1d980a50 0x564a1d986ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:58:16.280 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:16.275+0000 7efe881c9740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564a1d989820 0x564a1d987410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:58:16.280 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:16.275+0000 7efe881c9740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x564a1d98c450 con 0x564a1d980130 2024-03-16T11:58:16.280 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:16.275+0000 7efe881c9740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x564a1d8a0d70 con 0x564a1d989820 2024-03-16T11:58:16.280 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:16.275+0000 7efe881c9740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x564a1d98c5d0 con 0x564a1d980a50 2024-03-16T11:58:16.280 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:16.275+0000 7efe881c9740 10 client.0 _ll_register_callbacks cb 0x564a1d5f8a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T11:58:16.280 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:16.275+0000 7efe7effd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564a1d980130 0x564a1d986990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:58:16.280 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:16.275+0000 7efe77fff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564a1d980a50 0x564a1d986ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:58:16.280 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:16.275+0000 7efe7effd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564a1d980130 0x564a1d986990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39502/0 (socket says 172.21.15.29:39502) 2024-03-16T11:58:16.280 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:16.275+0000 7efe7effd640 1 -- 172.21.15.29:0/1900568219 learned_addr learned my addr 172.21.15.29:0/1900568219 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:58:16.280 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:16.275+0000 7efe7f7fe640 1 --2- 172.21.15.29:0/1900568219 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564a1d989820 0x564a1d987410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:58:16.280 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:16.275+0000 7efe7effd640 1 -- 172.21.15.29:0/1900568219 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564a1d980a50 msgr2=0x564a1d986ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:58:16.281 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:16.275+0000 7efe7effd640 1 --2- 172.21.15.29:0/1900568219 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564a1d980a50 0x564a1d986ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:58:16.281 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:16.275+0000 7efe7effd640 1 -- 172.21.15.29:0/1900568219 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564a1d989820 msgr2=0x564a1d987410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:58:16.281 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:16.275+0000 7efe7effd640 1 --2- 172.21.15.29:0/1900568219 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564a1d989820 0x564a1d987410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:58:16.281 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:16.275+0000 7efe7effd640 1 -- 172.21.15.29:0/1900568219 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564a1d5f9210 con 0x564a1d980130 2024-03-16T11:58:16.281 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:16.275+0000 7efe77fff640 1 --2- 172.21.15.29:0/1900568219 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564a1d980a50 0x564a1d986ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:58:16.281 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:16.275+0000 7efe7effd640 1 --2- 172.21.15.29:0/1900568219 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564a1d980130 0x564a1d986990 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7efe70012d20 tx=0x7efe7000f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a0db9814de5a2c8 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:58:16.281 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:16.275+0000 7efe667fc640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-16T11:58:16.281 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:16.275+0000 7efe667fc640 1 -- 172.21.15.29:0/1900568219 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe700120d0 con 0x564a1d980130 2024-03-16T11:58:16.281 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:16.275+0000 7efe881c9740 1 -- 172.21.15.29:0/1900568219 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efe680a4060 con 0x564a1d980130 2024-03-16T11:58:16.281 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:16.275+0000 7efe667fc640 1 -- 172.21.15.29:0/1900568219 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8556+0+0 (secure 0 0 0) 0x7efe70010430 con 0x564a1d980130 2024-03-16T11:58:16.281 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:16.276+0000 7efe667fc640 1 -- 172.21.15.29:0/1900568219 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7efe70012270 con 0x564a1d980130 2024-03-16T11:58:16.281 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:16.276+0000 7efe881c9740 10 client.16014 fetch_fsmap learned FSMap version 26 2024-03-16T11:58:16.281 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:16.276+0000 7efe881c9740 1 -- 172.21.15.29:0/1900568219 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x564a1d6c5620 con 0x564a1d980130 2024-03-16T11:58:16.281 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:16.276+0000 7efe667fc640 1 -- 172.21.15.29:0/1900568219 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efe680a4060 con 0x564a1d980130 2024-03-16T11:58:16.281 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:16.276+0000 7efe667fc640 1 -- 172.21.15.29:0/1900568219 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7efe70024830 con 0x564a1d980130 2024-03-16T11:58:16.282 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:16.276+0000 7efe881c9740 10 client.16014 fetch_fsmap finished waiting for FSMap version 26 2024-03-16T11:58:16.282 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:16.276+0000 7efe881c9740 10 client.16014 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T11:58:16.282 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:16.276+0000 7efe881c9740 20 client.16014 populate_metadata read hostname 'smithi029' 2024-03-16T11:58:16.282 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:16.276+0000 7efe881c9740 1 --2- 172.21.15.29:0/1900568219 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x564a1d999b40 0x564a1d99bf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:58:16.282 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:16.276+0000 7efe881c9740 4 client.16014 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T11:58:16.282 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:16.276+0000 7efe881c9740 1 -- 172.21.15.29:0/1900568219 --> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564a1d8f1f80 con 0x564a1d999b40 2024-03-16T11:58:16.282 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:16.276+0000 7efe77fff640 1 --2- 172.21.15.29:0/1900568219 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x564a1d999b40 0x564a1d99bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:58:16.282 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:16.277+0000 7efe77fff640 1 --2- 172.21.15.29:0/1900568219 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x564a1d999b40 0x564a1d99bf70 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=30c766c9a33d8814 server_cookie=98eadbe7626d6fd1 in_seq=0 out_seq=0 2024-03-16T11:58:16.282 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:16.277+0000 7efe667fc640 10 client.16014 ms_handle_connect on v2:172.21.15.152:6838/1548731833 2024-03-16T11:58:16.283 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:16.278+0000 7efe667fc640 1 -- 172.21.15.29:0/1900568219 <== mds.0 v2:172.21.15.152:6838/1548731833 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x564a1d8f1f80 con 0x564a1d999b40 2024-03-16T11:58:16.283 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:16.278+0000 7efe667fc640 10 client.16014 handle_command_reply: tid=0 2024-03-16T11:58:16.284 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:16.278+0000 7efe881c9740 1 client.16014 shutdown 2024-03-16T11:58:16.284 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:16.278+0000 7efe881c9740 1 -- 172.21.15.29:0/1900568219 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564a1d980130 msgr2=0x564a1d986990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:58:16.284 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:16.278+0000 7efe881c9740 1 --2- 172.21.15.29:0/1900568219 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564a1d980130 0x564a1d986990 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7efe70012d20 tx=0x7efe7000f790 comp rx=0 tx=0).stop 2024-03-16T11:58:16.284 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:16.278+0000 7efe881c9740 1 -- 172.21.15.29:0/1900568219 shutdown_connections 2024-03-16T11:58:16.284 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:16.278+0000 7efe881c9740 1 --2- 172.21.15.29:0/1900568219 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x564a1d999b40 0x564a1d99bf70 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:58:16.284 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:16.278+0000 7efe881c9740 1 --2- 172.21.15.29:0/1900568219 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564a1d989820 0x564a1d987410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:58:16.284 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:16.278+0000 7efe881c9740 1 --2- 172.21.15.29:0/1900568219 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564a1d980a50 0x564a1d986ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:58:16.284 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:16.278+0000 7efe667fc640 0 client.16014 ms_handle_reset on v2:172.21.15.152:6838/1548731833 2024-03-16T11:58:16.284 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:16.278+0000 7efe881c9740 1 --2- 172.21.15.29:0/1900568219 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564a1d980130 0x564a1d986990 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:58:16.284 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:16.278+0000 7efe881c9740 1 -- 172.21.15.29:0/1900568219 >> 172.21.15.29:0/1900568219 conn(0x564a1d97aa50 msgr2=0x564a1d97c7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:58:16.284 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:16.279+0000 7efe881c9740 1 -- 172.21.15.29:0/1900568219 shutdown_connections 2024-03-16T11:58:16.284 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:16.279+0000 7efe881c9740 1 -- 172.21.15.29:0/1900568219 wait complete. 2024-03-16T11:58:16.284 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:16.279+0000 7efe881c9740 20 client.16014 trim_cache size 0 max 16384 2024-03-16T11:58:16.293 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:16.294+0000 7efe881c9740 1 Processor -- start 2024-03-16T11:58:16.293 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:16.294+0000 7efe881c9740 1 -- start start 2024-03-16T11:58:16.298 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:16.294+0000 7efe881c9740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564a1d980130 0x564a1d9890c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:58:16.298 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:16.294+0000 7efe881c9740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564a1d980a50 0x564a1d9951f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:58:16.298 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:16.294+0000 7efe881c9740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564a1d989820 0x564a1d97c540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:58:16.298 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:16.294+0000 7efe881c9740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7efe801b1f70 con 0x564a1d989820 2024-03-16T11:58:16.298 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:16.294+0000 7efe881c9740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7efe801af840 con 0x564a1d980a50 2024-03-16T11:58:16.298 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:16.294+0000 7efe881c9740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7efe801ad6b0 con 0x564a1d980130 2024-03-16T11:58:16.298 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:16.295+0000 7efe7f7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564a1d989820 0x564a1d97c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:58:16.298 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:16.295+0000 7efe77fff640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564a1d980a50 0x564a1d9951f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:58:16.299 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:16.295+0000 7efe7effd640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564a1d980130 0x564a1d9890c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:58:16.299 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:16.295+0000 7efe7f7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564a1d989820 0x564a1d97c540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39508/0 (socket says 172.21.15.29:39508) 2024-03-16T11:58:16.299 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:16.295+0000 7efe7f7fe640 1 -- 172.21.15.29:0/2441808395 learned_addr learned my addr 172.21.15.29:0/2441808395 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:58:16.299 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:16.295+0000 7efe7f7fe640 1 -- 172.21.15.29:0/2441808395 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564a1d980130 msgr2=0x564a1d9890c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:58:16.299 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:16.295+0000 7efe7f7fe640 1 --2- 172.21.15.29:0/2441808395 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564a1d980130 0x564a1d9890c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:58:16.299 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:16.295+0000 7efe7f7fe640 1 -- 172.21.15.29:0/2441808395 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564a1d980a50 msgr2=0x564a1d9951f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:58:16.299 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:16.295+0000 7efe7f7fe640 1 --2- 172.21.15.29:0/2441808395 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564a1d980a50 0x564a1d9951f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:58:16.299 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:16.295+0000 7efe7f7fe640 1 -- 172.21.15.29:0/2441808395 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564a1d998730 con 0x564a1d989820 2024-03-16T11:58:16.299 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:16.295+0000 7efe7f7fe640 1 --2- 172.21.15.29:0/2441808395 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564a1d989820 0x564a1d97c540 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7efe7802c700 tx=0x7efe7802e050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9204e80b382b5726 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:58:16.299 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:16.295+0000 7efe74ff9640 1 -- 172.21.15.29:0/2441808395 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe78052020 con 0x564a1d989820 2024-03-16T11:58:16.299 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:16.295+0000 7efe74ff9640 1 -- 172.21.15.29:0/2441808395 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efe78003e50 con 0x564a1d989820 2024-03-16T11:58:16.299 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:16.295+0000 7efe74ff9640 1 -- 172.21.15.29:0/2441808395 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe78039040 con 0x564a1d989820 2024-03-16T11:58:16.299 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:16.296+0000 7efe881c9740 1 -- 172.21.15.29:0/2441808395 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564a1d989820 msgr2=0x564a1d97c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:58:16.299 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:16.296+0000 7efe881c9740 1 --2- 172.21.15.29:0/2441808395 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564a1d989820 0x564a1d97c540 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7efe7802c700 tx=0x7efe7802e050 comp rx=0 tx=0).stop 2024-03-16T11:58:16.300 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:16.296+0000 7efe881c9740 1 -- 172.21.15.29:0/2441808395 shutdown_connections 2024-03-16T11:58:16.300 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:16.296+0000 7efe881c9740 1 --2- 172.21.15.29:0/2441808395 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564a1d989820 0x564a1d97c540 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:58:16.300 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:16.296+0000 7efe881c9740 1 --2- 172.21.15.29:0/2441808395 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564a1d980a50 0x564a1d9951f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:58:16.300 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:16.296+0000 7efe881c9740 1 --2- 172.21.15.29:0/2441808395 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564a1d980130 0x564a1d9890c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:58:16.300 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:16.296+0000 7efe881c9740 1 -- 172.21.15.29:0/2441808395 >> 172.21.15.29:0/2441808395 conn(0x564a1d97aa50 msgr2=0x564a1d97be30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:58:16.300 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:16.296+0000 7efe881c9740 1 -- 172.21.15.29:0/2441808395 shutdown_connections 2024-03-16T11:58:16.300 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:16.296+0000 7efe881c9740 1 -- 172.21.15.29:0/2441808395 wait complete. 2024-03-16T11:58:16.300 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:16.296+0000 7efe881c9740 1 -- start start 2024-03-16T11:58:16.300 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:16.296+0000 7efe881c9740 1 Processor -- start 2024-03-16T11:58:16.300 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:16.297+0000 7efe881c9740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564a1d980130 0x564a1d9837b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:58:16.300 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:16.297+0000 7efe881c9740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564a1d980a50 0x564a1d983cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:58:16.300 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:16.297+0000 7efe881c9740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564a1d989820 0x564a1d984230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:58:16.300 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:16.297+0000 7efe881c9740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7efe801add00 con 0x564a1d989820 2024-03-16T11:58:16.300 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:16.297+0000 7efe881c9740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7efe801af170 con 0x564a1d980130 2024-03-16T11:58:16.301 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:16.297+0000 7efe881c9740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7efe801ae3d0 con 0x564a1d980a50 2024-03-16T11:58:16.301 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:16.297+0000 7efe881c9740 10 client.0 _ll_register_callbacks cb 0x7efe70009ce0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T11:58:16.301 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:16.297+0000 7efe7f7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564a1d989820 0x564a1d984230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:58:16.301 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:16.297+0000 7efe7f7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564a1d989820 0x564a1d984230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39514/0 (socket says 172.21.15.29:39514) 2024-03-16T11:58:16.301 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:16.297+0000 7efe7f7fe640 1 -- 172.21.15.29:0/3240295202 learned_addr learned my addr 172.21.15.29:0/3240295202 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:58:16.301 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:16.297+0000 7efe7effd640 1 --2- 172.21.15.29:0/3240295202 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564a1d980130 0x564a1d9837b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:58:16.301 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:16.297+0000 7efe77fff640 1 --2- 172.21.15.29:0/3240295202 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564a1d980a50 0x564a1d983cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:58:16.301 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:16.297+0000 7efe7f7fe640 1 -- 172.21.15.29:0/3240295202 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564a1d980a50 msgr2=0x564a1d983cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:58:16.301 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:16.297+0000 7efe7f7fe640 1 --2- 172.21.15.29:0/3240295202 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564a1d980a50 0x564a1d983cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:58:16.301 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:16.297+0000 7efe7f7fe640 1 -- 172.21.15.29:0/3240295202 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564a1d980130 msgr2=0x564a1d9837b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:58:16.301 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:16.297+0000 7efe7f7fe640 1 --2- 172.21.15.29:0/3240295202 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564a1d980130 0x564a1d9837b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:58:16.301 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:16.297+0000 7efe7f7fe640 1 -- 172.21.15.29:0/3240295202 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564a1d9ff150 con 0x564a1d989820 2024-03-16T11:58:16.301 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:16.297+0000 7efe7f7fe640 1 --2- 172.21.15.29:0/3240295202 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564a1d989820 0x564a1d984230 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7efe780046b0 tx=0x7efe78009360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5c4f64554f7883b server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:58:16.301 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:16.297+0000 7efe667fc640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-16T11:58:16.302 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:16.297+0000 7efe667fc640 1 -- 172.21.15.29:0/3240295202 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe78052030 con 0x564a1d989820 2024-03-16T11:58:16.302 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:16.297+0000 7efe667fc640 1 -- 172.21.15.29:0/3240295202 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8556+0+0 (secure 0 0 0) 0x7efe78004000 con 0x564a1d989820 2024-03-16T11:58:16.302 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:16.297+0000 7efe881c9740 1 -- 172.21.15.29:0/3240295202 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efe78003e50 con 0x564a1d989820 2024-03-16T11:58:16.302 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:16.298+0000 7efe667fc640 1 -- 172.21.15.29:0/3240295202 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7efe78039040 con 0x564a1d989820 2024-03-16T11:58:16.302 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:16.298+0000 7efe881c9740 10 client.16020 fetch_fsmap learned FSMap version 26 2024-03-16T11:58:16.302 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:16.298+0000 7efe881c9740 1 -- 172.21.15.29:0/3240295202 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x564a1da00460 con 0x564a1d989820 2024-03-16T11:58:16.302 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:16.298+0000 7efe667fc640 1 -- 172.21.15.29:0/3240295202 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efe78003e50 con 0x564a1d989820 2024-03-16T11:58:16.302 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:16.298+0000 7efe667fc640 1 -- 172.21.15.29:0/3240295202 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7efe7803f040 con 0x564a1d989820 2024-03-16T11:58:16.302 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:16.298+0000 7efe881c9740 10 client.16020 fetch_fsmap finished waiting for FSMap version 26 2024-03-16T11:58:16.302 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:16.298+0000 7efe881c9740 10 client.16020 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T11:58:16.302 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:16.298+0000 7efe881c9740 20 client.16020 populate_metadata read hostname 'smithi029' 2024-03-16T11:58:16.302 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:16.298+0000 7efe881c9740 1 --2- 172.21.15.29:0/3240295202 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x564a1da00b20 0x564a1da02f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:58:16.302 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:16.298+0000 7efe881c9740 4 client.16020 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-16T11:58:16.302 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:16.298+0000 7efe881c9740 1 -- 172.21.15.29:0/3240295202 --> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] -- command(tid 0: {"prefix": "flush journal"}) -- 0x564a1d8ec2d0 con 0x564a1da00b20 2024-03-16T11:58:16.303 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:16.298+0000 7efe7effd640 1 --2- 172.21.15.29:0/3240295202 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x564a1da00b20 0x564a1da02f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:58:16.303 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:16.299+0000 7efe7effd640 1 --2- 172.21.15.29:0/3240295202 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x564a1da00b20 0x564a1da02f40 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bf586c898f1dd60 server_cookie=335ee167e6c6180e in_seq=0 out_seq=0 2024-03-16T11:58:16.303 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:16.299+0000 7efe667fc640 10 client.16020 ms_handle_connect on v2:172.21.15.152:6838/1548731833 2024-03-16T11:58:16.462 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:16.462+0000 7efe667fc640 1 -- 172.21.15.29:0/3240295202 <== mds.0 v2:172.21.15.152:6838/1548731833 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x564a1d8ec2d0 con 0x564a1da00b20 2024-03-16T11:58:16.462 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:16.462+0000 7efe667fc640 10 client.16020 handle_command_reply: tid=0 2024-03-16T11:58:16.462 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:16.462+0000 7efe881c9740 1 client.16020 shutdown 2024-03-16T11:58:16.462 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:16.462+0000 7efe881c9740 1 -- 172.21.15.29:0/3240295202 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564a1d989820 msgr2=0x564a1d984230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:58:16.462 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:16.462+0000 7efe881c9740 1 --2- 172.21.15.29:0/3240295202 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564a1d989820 0x564a1d984230 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7efe780046b0 tx=0x7efe78009360 comp rx=0 tx=0).stop 2024-03-16T11:58:16.462 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:16.462+0000 7efe881c9740 1 -- 172.21.15.29:0/3240295202 shutdown_connections 2024-03-16T11:58:16.462 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:16.462+0000 7efe881c9740 1 --2- 172.21.15.29:0/3240295202 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564a1d989820 0x564a1d984230 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:58:16.462 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:16.462+0000 7efe881c9740 1 --2- 172.21.15.29:0/3240295202 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564a1d980a50 0x564a1d983cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:58:16.462 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:16.462+0000 7efe881c9740 1 --2- 172.21.15.29:0/3240295202 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x564a1da00b20 0x564a1da02f40 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:58:16.462 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:16.462+0000 7efe881c9740 1 --2- 172.21.15.29:0/3240295202 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564a1d980130 0x564a1d9837b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:58:16.463 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:16.462+0000 7efe881c9740 1 -- 172.21.15.29:0/3240295202 >> 172.21.15.29:0/3240295202 conn(0x564a1d97aa50 msgr2=0x564a1d97c5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:58:16.463 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:16.462+0000 7efe667fc640 0 client.16020 ms_handle_reset on v2:172.21.15.152:6838/1548731833 2024-03-16T11:58:16.463 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:16.462+0000 7efe881c9740 1 -- 172.21.15.29:0/3240295202 shutdown_connections 2024-03-16T11:58:16.463 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:16.462+0000 7efe881c9740 1 -- 172.21.15.29:0/3240295202 wait complete. 2024-03-16T11:58:16.463 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:16.462+0000 7efe881c9740 20 client.16020 trim_cache size 0 max 16384 2024-03-16T11:58:16.463 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:{ 2024-03-16T11:58:16.463 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "message": "", 2024-03-16T11:58:16.463 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "return_code": 0 2024-03-16T11:58:16.463 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:} 2024-03-16T11:58:16.464 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:16.463+0000 7efe854ff640 1 -- 172.21.15.29:0/622389303 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7efe6c0a2a10 msgr2=0x7efe6c0a4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:58:16.464 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:16.463+0000 7efe854ff640 1 --2- 172.21.15.29:0/622389303 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7efe6c0a2a10 0x7efe6c0a4ed0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7efe78005cc0 tx=0x7efe7802ce80 comp rx=0 tx=0).stop 2024-03-16T11:58:16.464 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:16.463+0000 7efe854ff640 1 -- 172.21.15.29:0/622389303 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efe80102170 msgr2=0x7efe80074020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:58:16.464 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:16.463+0000 7efe854ff640 1 --2- 172.21.15.29:0/622389303 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efe80102170 0x7efe80074020 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7efe6800e9f0 tx=0x7efe6800edf0 comp rx=0 tx=0).stop 2024-03-16T11:58:16.464 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:16.463+0000 7efe854ff640 1 -- 172.21.15.29:0/622389303 shutdown_connections 2024-03-16T11:58:16.465 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:16.463+0000 7efe854ff640 1 --2- 172.21.15.29:0/622389303 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7efe6c0a2a10 0x7efe6c0a4ed0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:58:16.465 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:16.463+0000 7efe854ff640 1 --2- 172.21.15.29:0/622389303 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7efe801077c0 0x7efe80074560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:58:16.465 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:16.463+0000 7efe854ff640 1 --2- 172.21.15.29:0/622389303 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efe80102170 0x7efe80074020 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:58:16.465 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:16.463+0000 7efe854ff640 1 --2- 172.21.15.29:0/622389303 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efe801017b0 0x7efe80073ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:58:16.465 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:16.463+0000 7efe854ff640 1 -- 172.21.15.29:0/622389303 >> 172.21.15.29:0/622389303 conn(0x7efe800fd2c0 msgr2=0x7efe800fd7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:58:16.465 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:16.463+0000 7efe854ff640 1 -- 172.21.15.29:0/622389303 shutdown_connections 2024-03-16T11:58:16.465 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:16.463+0000 7efe854ff640 1 -- 172.21.15.29:0/622389303 wait complete. 2024-03-16T11:58:16.792 INFO:tasks.workunit.client.0.smithi029.stdout: 1 743002 10.61 MB/sec execute 241 sec latency 104.065 ms 2024-03-16T11:58:16.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:58:16 smithi154 ceph-mon[36320]: pgmap v230: 97 pgs: 97 active+clean; 74 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 3.1 MiB/s wr, 321 op/s 2024-03-16T11:58:16.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:58:16 smithi029 ceph-mon[31975]: pgmap v230: 97 pgs: 97 active+clean; 74 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 3.1 MiB/s wr, 321 op/s 2024-03-16T11:58:16.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:58:16 smithi152 ceph-mon[36291]: pgmap v230: 97 pgs: 97 active+clean; 74 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 3.1 MiB/s wr, 321 op/s 2024-03-16T11:58:17.792 INFO:tasks.workunit.client.0.smithi029.stdout: 1 745141 10.62 MB/sec execute 242 sec latency 18.135 ms 2024-03-16T11:58:18.792 INFO:tasks.workunit.client.0.smithi029.stdout: 1 746812 10.61 MB/sec execute 243 sec latency 21.767 ms 2024-03-16T11:58:18.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:58:18 smithi154 ceph-mon[36320]: pgmap v231: 97 pgs: 97 active+clean; 80 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 3.1 MiB/s wr, 201 op/s 2024-03-16T11:58:18.837 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:58:18 smithi029 ceph-mon[31975]: pgmap v231: 97 pgs: 97 active+clean; 80 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 3.1 MiB/s wr, 201 op/s 2024-03-16T11:58:18.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:58:18 smithi152 ceph-mon[36291]: pgmap v231: 97 pgs: 97 active+clean; 80 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 3.1 MiB/s wr, 201 op/s 2024-03-16T11:58:19.792 INFO:tasks.workunit.client.0.smithi029.stdout: 1 748844 10.61 MB/sec execute 244 sec latency 78.194 ms 2024-03-16T11:58:20.793 INFO:tasks.workunit.client.0.smithi029.stdout: 1 750403 10.60 MB/sec execute 245 sec latency 31.561 ms 2024-03-16T11:58:20.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:58:20 smithi154 ceph-mon[36320]: pgmap v232: 97 pgs: 97 active+clean; 80 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 3.1 MiB/s wr, 201 op/s 2024-03-16T11:58:20.837 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:58:20 smithi029 ceph-mon[31975]: pgmap v232: 97 pgs: 97 active+clean; 80 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 3.1 MiB/s wr, 201 op/s 2024-03-16T11:58:20.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:58:20 smithi152 ceph-mon[36291]: pgmap v232: 97 pgs: 97 active+clean; 80 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 3.1 MiB/s wr, 201 op/s 2024-03-16T11:58:21.792 INFO:tasks.workunit.client.0.smithi029.stdout: 1 752747 10.62 MB/sec execute 246 sec latency 36.387 ms 2024-03-16T11:58:22.792 INFO:tasks.workunit.client.0.smithi029.stdout: 1 754261 10.61 MB/sec execute 247 sec latency 24.135 ms 2024-03-16T11:58:22.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:58:22 smithi154 ceph-mon[36320]: pgmap v233: 97 pgs: 97 active+clean; 71 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 3.3 MiB/s wr, 320 op/s 2024-03-16T11:58:22.837 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:58:22 smithi029 ceph-mon[31975]: pgmap v233: 97 pgs: 97 active+clean; 71 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 3.3 MiB/s wr, 320 op/s 2024-03-16T11:58:22.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:58:22 smithi152 ceph-mon[36291]: pgmap v233: 97 pgs: 97 active+clean; 71 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 3.3 MiB/s wr, 320 op/s 2024-03-16T11:58:23.792 INFO:tasks.workunit.client.0.smithi029.stdout: 1 756529 10.62 MB/sec execute 248 sec latency 18.056 ms 2024-03-16T11:58:24.792 INFO:tasks.workunit.client.0.smithi029.stdout: 1 757836 10.60 MB/sec execute 249 sec latency 68.624 ms 2024-03-16T11:58:24.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:58:24 smithi154 ceph-mon[36320]: pgmap v234: 97 pgs: 97 active+clean; 71 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 3.0 MiB/s wr, 293 op/s 2024-03-16T11:58:24.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:58:24 smithi029 ceph-mon[31975]: pgmap v234: 97 pgs: 97 active+clean; 71 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 3.0 MiB/s wr, 293 op/s 2024-03-16T11:58:24.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:58:24 smithi152 ceph-mon[36291]: pgmap v234: 97 pgs: 97 active+clean; 71 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 3.0 MiB/s wr, 293 op/s 2024-03-16T11:58:25.792 INFO:tasks.workunit.client.0.smithi029.stdout: 1 760075 10.61 MB/sec execute 250 sec latency 19.001 ms 2024-03-16T11:58:26.792 INFO:tasks.workunit.client.0.smithi029.stdout: 1 761515 10.59 MB/sec execute 251 sec latency 22.579 ms 2024-03-16T11:58:26.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:58:26 smithi154 ceph-mon[36320]: pgmap v235: 97 pgs: 97 active+clean; 71 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 2.8 MiB/s wr, 266 op/s 2024-03-16T11:58:26.837 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:58:26 smithi029 ceph-mon[31975]: pgmap v235: 97 pgs: 97 active+clean; 71 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 2.8 MiB/s wr, 266 op/s 2024-03-16T11:58:26.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:58:26 smithi152 ceph-mon[36291]: pgmap v235: 97 pgs: 97 active+clean; 71 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 2.8 MiB/s wr, 266 op/s 2024-03-16T11:58:27.792 INFO:tasks.workunit.client.0.smithi029.stdout: 1 763687 10.60 MB/sec execute 252 sec latency 20.153 ms 2024-03-16T11:58:28.792 INFO:tasks.workunit.client.0.smithi029.stdout: 1 765101 10.58 MB/sec execute 253 sec latency 33.822 ms 2024-03-16T11:58:28.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:58:28 smithi154 ceph-mon[36320]: pgmap v236: 97 pgs: 97 active+clean; 78 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 4.2 MiB/s wr, 355 op/s 2024-03-16T11:58:28.837 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:58:28 smithi029 ceph-mon[31975]: pgmap v236: 97 pgs: 97 active+clean; 78 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 4.2 MiB/s wr, 355 op/s 2024-03-16T11:58:28.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:58:28 smithi152 ceph-mon[36291]: pgmap v236: 97 pgs: 97 active+clean; 78 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 4.2 MiB/s wr, 355 op/s 2024-03-16T11:58:29.555 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:29.553+0000 7fc637b9e640 1 -- 172.21.15.29:0/3108188938 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc6301006b0 msgr2=0x7fc630100b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:58:29.555 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:29.553+0000 7fc637b9e640 1 --2- 172.21.15.29:0/3108188938 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc6301006b0 0x7fc630100b30 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7fc620009930 tx=0x7fc62002bc30 comp rx=0 tx=0).stop 2024-03-16T11:58:29.556 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:29.553+0000 7fc637b9e640 1 -- 172.21.15.29:0/3108188938 shutdown_connections 2024-03-16T11:58:29.556 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:29.553+0000 7fc637b9e640 1 --2- 172.21.15.29:0/3108188938 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc630101070 0x7fc63010d580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:58:29.556 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:29.553+0000 7fc637b9e640 1 --2- 172.21.15.29:0/3108188938 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc6301006b0 0x7fc630100b30 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:58:29.556 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:29.553+0000 7fc637b9e640 1 --2- 172.21.15.29:0/3108188938 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc6301066c0 0x7fc630106aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:58:29.556 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:29.553+0000 7fc637b9e640 1 -- 172.21.15.29:0/3108188938 >> 172.21.15.29:0/3108188938 conn(0x7fc630077810 msgr2=0x7fc630077c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:58:29.556 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:29.553+0000 7fc637b9e640 1 -- 172.21.15.29:0/3108188938 shutdown_connections 2024-03-16T11:58:29.556 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:29.553+0000 7fc637b9e640 1 -- 172.21.15.29:0/3108188938 wait complete. 2024-03-16T11:58:29.556 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:29.554+0000 7fc637b9e640 1 Processor -- start 2024-03-16T11:58:29.556 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:29.554+0000 7fc637b9e640 1 -- start start 2024-03-16T11:58:29.556 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:29.554+0000 7fc637b9e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc6301006b0 0x7fc630197fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:58:29.556 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:29.554+0000 7fc637b9e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc630101070 0x7fc630198520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:58:29.556 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:29.554+0000 7fc637b9e640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc6301066c0 0x7fc63019c850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:58:29.556 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:29.554+0000 7fc637b9e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fc630110ad0 con 0x7fc6301006b0 2024-03-16T11:58:29.557 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:29.554+0000 7fc637b9e640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fc630110950 con 0x7fc6301066c0 2024-03-16T11:58:29.557 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:29.554+0000 7fc637b9e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fc630110c50 con 0x7fc630101070 2024-03-16T11:58:29.557 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:29.554+0000 7fc63593a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc6301006b0 0x7fc630197fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:58:29.557 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:29.554+0000 7fc63593a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc6301006b0 0x7fc630197fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:34994/0 (socket says 172.21.15.29:34994) 2024-03-16T11:58:29.557 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:29.554+0000 7fc63593a640 1 -- 172.21.15.29:0/1850734825 learned_addr learned my addr 172.21.15.29:0/1850734825 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:58:29.557 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:29.554+0000 7fc635139640 1 --2- 172.21.15.29:0/1850734825 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc630101070 0x7fc630198520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:58:29.557 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:29.554+0000 7fc63613b640 1 --2- 172.21.15.29:0/1850734825 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc6301066c0 0x7fc63019c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:58:29.557 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:29.554+0000 7fc63593a640 1 -- 172.21.15.29:0/1850734825 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc630101070 msgr2=0x7fc630198520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:58:29.557 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:29.554+0000 7fc63593a640 1 --2- 172.21.15.29:0/1850734825 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc630101070 0x7fc630198520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:58:29.557 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:29.554+0000 7fc63593a640 1 -- 172.21.15.29:0/1850734825 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc6301066c0 msgr2=0x7fc63019c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:58:29.557 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:29.554+0000 7fc63593a640 1 --2- 172.21.15.29:0/1850734825 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc6301066c0 0x7fc63019c850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:58:29.557 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:29.554+0000 7fc63593a640 1 -- 172.21.15.29:0/1850734825 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc63019cf80 con 0x7fc6301006b0 2024-03-16T11:58:29.557 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:29.554+0000 7fc635139640 1 --2- 172.21.15.29:0/1850734825 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc630101070 0x7fc630198520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:58:29.557 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:29.554+0000 7fc63593a640 1 --2- 172.21.15.29:0/1850734825 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc6301006b0 0x7fc630197fe0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7fc62c00e9f0 tx=0x7fc62c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:58:29.558 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:29.555+0000 7fc626ffd640 1 -- 172.21.15.29:0/1850734825 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc62c00cd80 con 0x7fc6301006b0 2024-03-16T11:58:29.558 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:29.555+0000 7fc637b9e640 1 -- 172.21.15.29:0/1850734825 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc63019d270 con 0x7fc6301006b0 2024-03-16T11:58:29.558 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:29.555+0000 7fc626ffd640 1 -- 172.21.15.29:0/1850734825 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc62c010910 con 0x7fc6301006b0 2024-03-16T11:58:29.558 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:29.555+0000 7fc626ffd640 1 -- 172.21.15.29:0/1850734825 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc62c0188f0 con 0x7fc6301006b0 2024-03-16T11:58:29.558 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:29.555+0000 7fc637b9e640 1 -- 172.21.15.29:0/1850734825 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc6301a4b00 con 0x7fc6301006b0 2024-03-16T11:58:29.558 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:29.558+0000 7fc626ffd640 1 -- 172.21.15.29:0/1850734825 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7fc62c010ac0 con 0x7fc6301006b0 2024-03-16T11:58:29.559 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:29.558+0000 7fc63a789740 1 Processor -- start 2024-03-16T11:58:29.559 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:29.559+0000 7fc63a789740 1 -- start start 2024-03-16T11:58:29.559 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:29.559+0000 7fc63a789740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55faecde8130 0x55faecde8510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:58:29.560 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:29.559+0000 7fc63a789740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55faecde8a50 0x55faecdf12e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:58:29.560 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:29.559+0000 7fc63a789740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55faecdf1820 0x55faecdf3c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:58:29.560 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:29.559+0000 7fc63a789740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55faecd4ca40 con 0x55faecdf1820 2024-03-16T11:58:29.560 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:29.559+0000 7fc63a789740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x55faeccf5920 con 0x55faecde8130 2024-03-16T11:58:29.560 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:29.559+0000 7fc63a789740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55faecd48ae0 con 0x55faecde8a50 2024-03-16T11:58:29.560 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:29.559+0000 7fc626ffd640 1 --2- 172.21.15.29:0/1850734825 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7fc61c0ab220 0x7fc61c0ad6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:58:29.560 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:29.559+0000 7fc635139640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55faecdf1820 0x55faecdf3c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:58:29.560 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:29.559+0000 7fc635139640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55faecdf1820 0x55faecdf3c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:35006/0 (socket says 172.21.15.29:35006) 2024-03-16T11:58:29.560 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:29.559+0000 7fc635139640 1 -- 172.21.15.29:0/1986318106 learned_addr learned my addr 172.21.15.29:0/1986318106 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:58:29.560 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:29.559+0000 7fc63613b640 1 --2- 172.21.15.29:0/1986318106 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55faecde8130 0x55faecde8510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:58:29.560 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:29.559+0000 7fc63613b640 1 --2- 172.21.15.29:0/1850734825 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7fc61c0ab220 0x7fc61c0ad6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:58:29.560 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:29.559+0000 7fc626ffd640 1 -- 172.21.15.29:0/1850734825 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8556+0+0 (secure 0 0 0) 0x7fc62c0cf1e0 con 0x7fc6301006b0 2024-03-16T11:58:29.560 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:29.559+0000 7fc63593a640 1 --2- 172.21.15.29:0/1986318106 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55faecde8a50 0x55faecdf12e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:58:29.561 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:29.560+0000 7fc63613b640 1 -- 172.21.15.29:0/1986318106 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55faecde8a50 msgr2=0x55faecdf12e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:58:29.561 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:29.560+0000 7fc63613b640 1 --2- 172.21.15.29:0/1986318106 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55faecde8a50 0x55faecdf12e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:58:29.561 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:29.560+0000 7fc63613b640 1 -- 172.21.15.29:0/1986318106 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55faecdf1820 msgr2=0x55faecdf3c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:58:29.561 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:29.560+0000 7fc63613b640 1 --2- 172.21.15.29:0/1986318106 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55faecdf1820 0x55faecdf3c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:58:29.561 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:29.560+0000 7fc63613b640 1 -- 172.21.15.29:0/1986318106 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55faecd0d460 con 0x55faecde8130 2024-03-16T11:58:29.561 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:29.560+0000 7fc63613b640 1 --2- 172.21.15.29:0/1850734825 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7fc61c0ab220 0x7fc61c0ad6e0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fc628005fd0 tx=0x7fc628009b80 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:58:29.563 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:29.561+0000 7fc63613b640 1 --2- 172.21.15.29:0/1986318106 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55faecde8130 0x55faecde8510 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fc628011a10 tx=0x7fc62801d000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5b3621d310f62426 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:58:29.563 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:29.562+0000 7fc61bfff640 1 -- 172.21.15.29:0/1986318106 <== mon.2 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc628023070 con 0x55faecde8130 2024-03-16T11:58:29.563 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:29.562+0000 7fc61bfff640 1 -- 172.21.15.29:0/1986318106 <== mon.2 v2:172.21.15.152:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc628003970 con 0x55faecde8130 2024-03-16T11:58:29.563 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:29.562+0000 7fc61bfff640 1 -- 172.21.15.29:0/1986318106 <== mon.2 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc628004300 con 0x55faecde8130 2024-03-16T11:58:29.563 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:29.562+0000 7fc63a789740 1 -- 172.21.15.29:0/1986318106 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55faecde8130 msgr2=0x55faecde8510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:58:29.563 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:29.562+0000 7fc63a789740 1 --2- 172.21.15.29:0/1986318106 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55faecde8130 0x55faecde8510 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fc628011a10 tx=0x7fc62801d000 comp rx=0 tx=0).stop 2024-03-16T11:58:29.563 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:29.563+0000 7fc63a789740 1 -- 172.21.15.29:0/1986318106 shutdown_connections 2024-03-16T11:58:29.563 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:29.563+0000 7fc63a789740 1 --2- 172.21.15.29:0/1986318106 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55faecdf1820 0x55faecdf3c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:58:29.563 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:29.563+0000 7fc63a789740 1 --2- 172.21.15.29:0/1986318106 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55faecde8a50 0x55faecdf12e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:58:29.564 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:29.563+0000 7fc63a789740 1 --2- 172.21.15.29:0/1986318106 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55faecde8130 0x55faecde8510 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:58:29.564 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:29.563+0000 7fc63a789740 1 -- 172.21.15.29:0/1986318106 >> 172.21.15.29:0/1986318106 conn(0x55faecde2a50 msgr2=0x55faecde2e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:58:29.564 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:29.563+0000 7fc63a789740 1 -- 172.21.15.29:0/1986318106 shutdown_connections 2024-03-16T11:58:29.564 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:29.563+0000 7fc63a789740 1 -- 172.21.15.29:0/1986318106 wait complete. 2024-03-16T11:58:29.564 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:29.563+0000 7fc63a789740 1 -- start start 2024-03-16T11:58:29.564 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:29.563+0000 7fc63a789740 1 Processor -- start 2024-03-16T11:58:29.567 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:29.564+0000 7fc63a789740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55faecde8130 0x55faecdeaa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:58:29.567 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:29.564+0000 7fc63a789740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55faecde8a50 0x55faecdeafc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:58:29.567 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:29.564+0000 7fc63a789740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55faecdf1820 0x55faecdef240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:58:29.567 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:29.564+0000 7fc63a789740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55faecdf4330 con 0x55faecde8130 2024-03-16T11:58:29.567 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:29.564+0000 7fc63a789740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x55faecd08d70 con 0x55faecde8a50 2024-03-16T11:58:29.567 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:29.564+0000 7fc63a789740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55faecdf44b0 con 0x55faecdf1820 2024-03-16T11:58:29.567 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:29.564+0000 7fc63593a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55faecde8130 0x55faecdeaa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:58:29.568 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:29.564+0000 7fc635139640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55faecde8a50 0x55faecdeafc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:58:29.568 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:29.564+0000 7fc635139640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55faecde8a50 0x55faecdeafc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.29:41296/0 (socket says 172.21.15.29:41296) 2024-03-16T11:58:29.568 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:29.564+0000 7fc635139640 1 -- 172.21.15.29:0/4149638994 learned_addr learned my addr 172.21.15.29:0/4149638994 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:58:29.568 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:29.564+0000 7fc63613b640 1 --2- 172.21.15.29:0/4149638994 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55faecdf1820 0x55faecdef240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:58:29.568 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:29.564+0000 7fc63a789740 10 client.0 _ll_register_callbacks cb 0x55faeca60a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T11:58:29.568 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:29.565+0000 7fc63593a640 1 -- 172.21.15.29:0/4149638994 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55faecdf1820 msgr2=0x55faecdef240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:58:29.568 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:29.565+0000 7fc63593a640 1 --2- 172.21.15.29:0/4149638994 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55faecdf1820 0x55faecdef240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:58:29.568 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:29.565+0000 7fc63593a640 1 -- 172.21.15.29:0/4149638994 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55faecde8a50 msgr2=0x55faecdeafc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:58:29.568 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:29.565+0000 7fc63593a640 1 --2- 172.21.15.29:0/4149638994 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55faecde8a50 0x55faecdeafc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:58:29.568 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:29.565+0000 7fc63593a640 1 -- 172.21.15.29:0/4149638994 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55faeca61210 con 0x55faecde8130 2024-03-16T11:58:29.568 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:29.565+0000 7fc63613b640 1 --2- 172.21.15.29:0/4149638994 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55faecdf1820 0x55faecdef240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:58:29.568 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:29.565+0000 7fc63593a640 1 --2- 172.21.15.29:0/4149638994 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55faecde8130 0x55faecdeaa80 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7fc62c0cc910 tx=0x7fc62c09b990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2c40b489e74b4ba server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:58:29.568 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:29.565+0000 7fc6197fa640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-16T11:58:29.568 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:29.565+0000 7fc6197fa640 1 -- 172.21.15.29:0/4149638994 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc62c0b4060 con 0x55faecde8130 2024-03-16T11:58:29.569 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:29.565+0000 7fc63a789740 1 -- 172.21.15.29:0/4149638994 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc628003970 con 0x55faecde8130 2024-03-16T11:58:29.569 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:29.565+0000 7fc6197fa640 1 -- 172.21.15.29:0/4149638994 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8556+0+0 (secure 0 0 0) 0x7fc62c0a3d30 con 0x55faecde8130 2024-03-16T11:58:29.569 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:29.565+0000 7fc6197fa640 1 -- 172.21.15.29:0/4149638994 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fc62c097040 con 0x55faecde8130 2024-03-16T11:58:29.569 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:29.565+0000 7fc63a789740 10 client.16032 fetch_fsmap learned FSMap version 26 2024-03-16T11:58:29.569 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:29.565+0000 7fc63a789740 1 -- 172.21.15.29:0/4149638994 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55faecb2d620 con 0x55faecde8130 2024-03-16T11:58:29.569 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:29.566+0000 7fc6197fa640 1 -- 172.21.15.29:0/4149638994 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc628003970 con 0x55faecde8130 2024-03-16T11:58:29.569 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:29.566+0000 7fc6197fa640 1 -- 172.21.15.29:0/4149638994 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7fc62c09ad20 con 0x55faecde8130 2024-03-16T11:58:29.569 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:29.566+0000 7fc63a789740 10 client.16032 fetch_fsmap finished waiting for FSMap version 26 2024-03-16T11:58:29.569 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:29.566+0000 7fc63a789740 10 client.16032 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T11:58:29.569 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:29.566+0000 7fc63a789740 20 client.16032 populate_metadata read hostname 'smithi029' 2024-03-16T11:58:29.569 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:29.566+0000 7fc63a789740 1 --2- 172.21.15.29:0/4149638994 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55faece019a0 0x55faece03d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:58:29.569 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:29.566+0000 7fc63a789740 4 client.16032 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T11:58:29.569 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:29.566+0000 7fc63a789740 1 -- 172.21.15.29:0/4149638994 --> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55faecd59f80 con 0x55faece019a0 2024-03-16T11:58:29.569 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:29.566+0000 7fc635139640 1 --2- 172.21.15.29:0/4149638994 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55faece019a0 0x55faece03d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:58:29.571 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:29.568+0000 7fc6197fa640 10 client.16032 ms_handle_connect on v2:172.21.15.152:6838/1548731833 2024-03-16T11:58:29.571 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:29.568+0000 7fc635139640 1 --2- 172.21.15.29:0/4149638994 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55faece019a0 0x55faece03d90 crc :-1 s=READY pgs=106 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e2b71af9bbbf334f server_cookie=b3bf7e0f72df248c in_seq=0 out_seq=0 2024-03-16T11:58:29.571 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:29.569+0000 7fc6197fa640 1 -- 172.21.15.29:0/4149638994 <== mds.0 v2:172.21.15.152:6838/1548731833 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55faecd59f80 con 0x55faece019a0 2024-03-16T11:58:29.571 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:29.569+0000 7fc6197fa640 10 client.16032 handle_command_reply: tid=0 2024-03-16T11:58:29.571 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:29.569+0000 7fc63a789740 1 client.16032 shutdown 2024-03-16T11:58:29.571 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:29.569+0000 7fc63a789740 1 -- 172.21.15.29:0/4149638994 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55faecde8130 msgr2=0x55faecdeaa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:58:29.571 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:29.569+0000 7fc63a789740 1 --2- 172.21.15.29:0/4149638994 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55faecde8130 0x55faecdeaa80 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7fc62c0cc910 tx=0x7fc62c09b990 comp rx=0 tx=0).stop 2024-03-16T11:58:29.571 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:29.569+0000 7fc63a789740 1 -- 172.21.15.29:0/4149638994 shutdown_connections 2024-03-16T11:58:29.571 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:29.569+0000 7fc63a789740 1 --2- 172.21.15.29:0/4149638994 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55faecdf1820 0x55faecdef240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:58:29.571 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:29.569+0000 7fc63a789740 1 --2- 172.21.15.29:0/4149638994 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55faece019a0 0x55faece03d90 crc :-1 s=READY pgs=106 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:58:29.572 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:29.569+0000 7fc63a789740 1 --2- 172.21.15.29:0/4149638994 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55faecde8a50 0x55faecdeafc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:58:29.572 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:29.569+0000 7fc6197fa640 0 client.16032 ms_handle_reset on v2:172.21.15.152:6838/1548731833 2024-03-16T11:58:29.572 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:29.569+0000 7fc63a789740 1 --2- 172.21.15.29:0/4149638994 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55faecde8130 0x55faecdeaa80 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:58:29.572 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:29.569+0000 7fc63a789740 1 -- 172.21.15.29:0/4149638994 >> 172.21.15.29:0/4149638994 conn(0x55faecde2a50 msgr2=0x55faecde47a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:58:29.572 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:29.569+0000 7fc63a789740 1 -- 172.21.15.29:0/4149638994 shutdown_connections 2024-03-16T11:58:29.572 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:29.569+0000 7fc63a789740 1 -- 172.21.15.29:0/4149638994 wait complete. 2024-03-16T11:58:29.572 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:29.569+0000 7fc63a789740 20 client.16032 trim_cache size 0 max 16384 2024-03-16T11:58:29.591 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:29.589+0000 7fc63a789740 1 Processor -- start 2024-03-16T11:58:29.591 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:29.589+0000 7fc63a789740 1 -- start start 2024-03-16T11:58:29.591 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:29.589+0000 7fc63a789740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55faecde8130 0x55faecddda40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:58:29.591 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:29.589+0000 7fc63a789740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55faecde8a50 0x55faecdef070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:58:29.592 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:29.589+0000 7fc63a789740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55faecdf1820 0x55faece095d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:58:29.592 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:29.589+0000 7fc63a789740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fc6301adb70 con 0x55faecde8130 2024-03-16T11:58:29.592 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:29.589+0000 7fc63a789740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fc6301ab440 con 0x55faecde8a50 2024-03-16T11:58:29.592 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:29.589+0000 7fc63a789740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fc6301a92b0 con 0x55faecdf1820 2024-03-16T11:58:29.592 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:29.589+0000 7fc63593a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55faecde8130 0x55faecddda40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:58:29.592 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:29.589+0000 7fc63593a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55faecde8130 0x55faecddda40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:35022/0 (socket says 172.21.15.29:35022) 2024-03-16T11:58:29.592 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:29.589+0000 7fc63593a640 1 -- 172.21.15.29:0/4219286536 learned_addr learned my addr 172.21.15.29:0/4219286536 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:58:29.592 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:29.589+0000 7fc635139640 1 --2- 172.21.15.29:0/4219286536 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55faecde8a50 0x55faecdef070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:58:29.592 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:29.590+0000 7fc63613b640 1 --2- 172.21.15.29:0/4219286536 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55faecdf1820 0x55faece095d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:58:29.592 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:29.590+0000 7fc63593a640 1 -- 172.21.15.29:0/4219286536 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55faecdf1820 msgr2=0x55faece095d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:58:29.592 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:29.590+0000 7fc63593a640 1 --2- 172.21.15.29:0/4219286536 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55faecdf1820 0x55faece095d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:58:29.592 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:29.590+0000 7fc63593a640 1 -- 172.21.15.29:0/4219286536 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55faecde8a50 msgr2=0x55faecdef070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:58:29.592 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:29.590+0000 7fc63593a640 1 --2- 172.21.15.29:0/4219286536 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55faecde8a50 0x55faecdef070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:58:29.592 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:29.590+0000 7fc63593a640 1 -- 172.21.15.29:0/4219286536 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55faecde9a30 con 0x55faecde8130 2024-03-16T11:58:29.593 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:29.590+0000 7fc63593a640 1 --2- 172.21.15.29:0/4219286536 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55faecde8130 0x55faecddda40 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7fc62c09b860 tx=0x7fc62c097ca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c74a9ea4cae3741 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:58:29.593 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:29.590+0000 7fc61b7fe640 1 -- 172.21.15.29:0/4219286536 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc62c0b4050 con 0x55faecde8130 2024-03-16T11:58:29.593 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:29.590+0000 7fc61b7fe640 1 -- 172.21.15.29:0/4219286536 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc62c0973e0 con 0x55faecde8130 2024-03-16T11:58:29.593 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:29.590+0000 7fc61b7fe640 1 -- 172.21.15.29:0/4219286536 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc62c09cb50 con 0x55faecde8130 2024-03-16T11:58:29.593 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:29.591+0000 7fc63a789740 1 -- 172.21.15.29:0/4219286536 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55faecde8130 msgr2=0x55faecddda40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:58:29.593 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:29.591+0000 7fc63a789740 1 --2- 172.21.15.29:0/4219286536 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55faecde8130 0x55faecddda40 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7fc62c09b860 tx=0x7fc62c097ca0 comp rx=0 tx=0).stop 2024-03-16T11:58:29.593 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:29.591+0000 7fc63a789740 1 -- 172.21.15.29:0/4219286536 shutdown_connections 2024-03-16T11:58:29.593 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:29.591+0000 7fc63a789740 1 --2- 172.21.15.29:0/4219286536 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55faecdf1820 0x55faece095d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:58:29.593 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:29.591+0000 7fc63a789740 1 --2- 172.21.15.29:0/4219286536 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55faecde8a50 0x55faecdef070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:58:29.593 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:29.591+0000 7fc63a789740 1 --2- 172.21.15.29:0/4219286536 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55faecde8130 0x55faecddda40 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:58:29.593 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:29.591+0000 7fc63a789740 1 -- 172.21.15.29:0/4219286536 >> 172.21.15.29:0/4219286536 conn(0x55faecde2a50 msgr2=0x55faecde1bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:58:29.593 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:29.591+0000 7fc63a789740 1 -- 172.21.15.29:0/4219286536 shutdown_connections 2024-03-16T11:58:29.593 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:29.591+0000 7fc63a789740 1 -- 172.21.15.29:0/4219286536 wait complete. 2024-03-16T11:58:29.593 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:29.591+0000 7fc63a789740 1 -- start start 2024-03-16T11:58:29.593 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:29.591+0000 7fc63a789740 1 Processor -- start 2024-03-16T11:58:29.596 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:29.592+0000 7fc63a789740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55faecde8130 0x55faecdf98b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:58:29.597 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:29.592+0000 7fc63a789740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55faecde8a50 0x55faecdf9df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:58:29.597 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:29.592+0000 7fc63a789740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55faecdf1820 0x55faecdfa330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:58:29.597 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:29.592+0000 7fc63a789740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fc6301a9900 con 0x55faecdf1820 2024-03-16T11:58:29.597 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:29.592+0000 7fc63a789740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fc6301aad70 con 0x55faecde8130 2024-03-16T11:58:29.597 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:29.592+0000 7fc63a789740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fc6301a9fd0 con 0x55faecde8a50 2024-03-16T11:58:29.597 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:29.592+0000 7fc63a789740 10 client.0 _ll_register_callbacks cb 0x7fc62c09b6a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T11:58:29.597 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:29.592+0000 7fc63613b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55faecdf1820 0x55faecdfa330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:58:29.597 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:29.592+0000 7fc63613b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55faecdf1820 0x55faecdfa330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:35032/0 (socket says 172.21.15.29:35032) 2024-03-16T11:58:29.597 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:29.592+0000 7fc63613b640 1 -- 172.21.15.29:0/382699643 learned_addr learned my addr 172.21.15.29:0/382699643 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:58:29.597 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:29.592+0000 7fc63593a640 1 --2- 172.21.15.29:0/382699643 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55faecde8130 0x55faecdf98b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:58:29.597 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:29.592+0000 7fc635139640 1 --2- 172.21.15.29:0/382699643 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55faecde8a50 0x55faecdf9df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:58:29.597 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:29.592+0000 7fc63613b640 1 -- 172.21.15.29:0/382699643 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55faecde8a50 msgr2=0x55faecdf9df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:58:29.597 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:29.592+0000 7fc63613b640 1 --2- 172.21.15.29:0/382699643 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55faecde8a50 0x55faecdf9df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:58:29.597 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:29.592+0000 7fc63613b640 1 -- 172.21.15.29:0/382699643 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55faecde8130 msgr2=0x55faecdf98b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:58:29.598 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:29.592+0000 7fc63613b640 1 --2- 172.21.15.29:0/382699643 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55faecde8130 0x55faecdf98b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:58:29.598 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:29.592+0000 7fc63613b640 1 -- 172.21.15.29:0/382699643 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55faecde8880 con 0x55faecdf1820 2024-03-16T11:58:29.598 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:29.592+0000 7fc63593a640 1 --2- 172.21.15.29:0/382699643 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55faecde8130 0x55faecdf98b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:58:29.598 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:29.592+0000 7fc63613b640 1 --2- 172.21.15.29:0/382699643 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55faecdf1820 0x55faecdfa330 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7fc628000f80 tx=0x7fc628003eb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3429b6dd6ca2ed55 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:58:29.598 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:29.592+0000 7fc6197fa640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-16T11:58:29.598 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:29.593+0000 7fc6197fa640 1 -- 172.21.15.29:0/382699643 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc628034020 con 0x55faecdf1820 2024-03-16T11:58:29.598 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:29.593+0000 7fc6197fa640 1 -- 172.21.15.29:0/382699643 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8556+0+0 (secure 0 0 0) 0x7fc628005420 con 0x55faecdf1820 2024-03-16T11:58:29.598 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:29.593+0000 7fc63a789740 1 -- 172.21.15.29:0/382699643 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc62c0973e0 con 0x55faecdf1820 2024-03-16T11:58:29.598 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:29.593+0000 7fc6197fa640 1 -- 172.21.15.29:0/382699643 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fc628023040 con 0x55faecdf1820 2024-03-16T11:58:29.598 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:29.593+0000 7fc63a789740 10 client.16038 fetch_fsmap learned FSMap version 26 2024-03-16T11:58:29.598 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:29.593+0000 7fc63a789740 1 -- 172.21.15.29:0/382699643 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55faece67e40 con 0x55faecdf1820 2024-03-16T11:58:29.598 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:29.593+0000 7fc6197fa640 1 -- 172.21.15.29:0/382699643 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc62c0973e0 con 0x55faecdf1820 2024-03-16T11:58:29.598 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:29.593+0000 7fc6197fa640 1 -- 172.21.15.29:0/382699643 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7fc62802f070 con 0x55faecdf1820 2024-03-16T11:58:29.598 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:29.593+0000 7fc63a789740 10 client.16038 fetch_fsmap finished waiting for FSMap version 26 2024-03-16T11:58:29.598 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:29.593+0000 7fc63a789740 10 client.16038 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T11:58:29.599 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:29.593+0000 7fc63a789740 20 client.16038 populate_metadata read hostname 'smithi029' 2024-03-16T11:58:29.599 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:29.593+0000 7fc63a789740 1 --2- 172.21.15.29:0/382699643 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55faece683a0 0x55faece6a7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:58:29.599 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:29.593+0000 7fc63a789740 4 client.16038 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-16T11:58:29.599 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:29.593+0000 7fc63a789740 1 -- 172.21.15.29:0/382699643 --> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55faecd542d0 con 0x55faece683a0 2024-03-16T11:58:29.599 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:29.593+0000 7fc63593a640 1 --2- 172.21.15.29:0/382699643 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55faece683a0 0x55faece6a7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:58:29.599 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:29.594+0000 7fc63593a640 1 --2- 172.21.15.29:0/382699643 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55faece683a0 0x55faece6a7c0 crc :-1 s=READY pgs=107 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a82b0b3d23343052 server_cookie=7dc90593b9f8220d in_seq=0 out_seq=0 2024-03-16T11:58:29.599 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:29.594+0000 7fc6197fa640 10 client.16038 ms_handle_connect on v2:172.21.15.152:6838/1548731833 2024-03-16T11:58:29.793 INFO:tasks.workunit.client.0.smithi029.stdout: 1 767071 10.59 MB/sec execute 254 sec latency 92.476 ms 2024-03-16T11:58:29.824 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:29.823+0000 7fc6197fa640 1 -- 172.21.15.29:0/382699643 <== mds.0 v2:172.21.15.152:6838/1548731833 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55faecd542d0 con 0x55faece683a0 2024-03-16T11:58:29.824 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:29.823+0000 7fc6197fa640 10 client.16038 handle_command_reply: tid=0 2024-03-16T11:58:29.824 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:29.823+0000 7fc63a789740 1 client.16038 shutdown 2024-03-16T11:58:29.824 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:29.823+0000 7fc63a789740 1 -- 172.21.15.29:0/382699643 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55faecdf1820 msgr2=0x55faecdfa330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:58:29.825 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:29.823+0000 7fc63a789740 1 --2- 172.21.15.29:0/382699643 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55faecdf1820 0x55faecdfa330 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7fc628000f80 tx=0x7fc628003eb0 comp rx=0 tx=0).stop 2024-03-16T11:58:29.825 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:29.824+0000 7fc63a789740 1 -- 172.21.15.29:0/382699643 shutdown_connections 2024-03-16T11:58:29.825 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:29.824+0000 7fc63a789740 1 --2- 172.21.15.29:0/382699643 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55faecdf1820 0x55faecdfa330 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:58:29.825 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:29.824+0000 7fc63a789740 1 --2- 172.21.15.29:0/382699643 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55faecde8a50 0x55faecdf9df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:58:29.825 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:29.824+0000 7fc63a789740 1 --2- 172.21.15.29:0/382699643 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55faece683a0 0x55faece6a7c0 crc :-1 s=READY pgs=107 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:58:29.825 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:29.824+0000 7fc63a789740 1 --2- 172.21.15.29:0/382699643 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55faecde8130 0x55faecdf98b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:58:29.825 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:29.824+0000 7fc63a789740 1 -- 172.21.15.29:0/382699643 >> 172.21.15.29:0/382699643 conn(0x55faecde2a50 msgr2=0x55faecde3e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:58:29.825 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:29.824+0000 7fc6197fa640 0 client.16038 ms_handle_reset on v2:172.21.15.152:6838/1548731833 2024-03-16T11:58:29.825 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:29.824+0000 7fc63a789740 1 -- 172.21.15.29:0/382699643 shutdown_connections 2024-03-16T11:58:29.825 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:29.824+0000 7fc63a789740 1 -- 172.21.15.29:0/382699643 wait complete. 2024-03-16T11:58:29.825 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:29.824+0000 7fc63a789740 20 client.16038 trim_cache size 0 max 16384 2024-03-16T11:58:29.826 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:{ 2024-03-16T11:58:29.826 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "message": "", 2024-03-16T11:58:29.826 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "return_code": 0 2024-03-16T11:58:29.826 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:} 2024-03-16T11:58:29.827 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:29.824+0000 7fc637b9e640 1 -- 172.21.15.29:0/1850734825 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7fc61c0ab220 msgr2=0x7fc61c0ad6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:58:29.827 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:29.824+0000 7fc637b9e640 1 --2- 172.21.15.29:0/1850734825 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7fc61c0ab220 0x7fc61c0ad6e0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fc628005fd0 tx=0x7fc628009b80 comp rx=0 tx=0).stop 2024-03-16T11:58:29.827 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:29.825+0000 7fc637b9e640 1 -- 172.21.15.29:0/1850734825 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc6301006b0 msgr2=0x7fc630197fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:58:29.827 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:29.825+0000 7fc637b9e640 1 --2- 172.21.15.29:0/1850734825 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc6301006b0 0x7fc630197fe0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7fc62c00e9f0 tx=0x7fc62c00eeb0 comp rx=0 tx=0).stop 2024-03-16T11:58:29.827 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:29.825+0000 7fc637b9e640 1 -- 172.21.15.29:0/1850734825 shutdown_connections 2024-03-16T11:58:29.827 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:29.825+0000 7fc637b9e640 1 --2- 172.21.15.29:0/1850734825 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7fc61c0ab220 0x7fc61c0ad6e0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:58:29.827 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:29.825+0000 7fc637b9e640 1 --2- 172.21.15.29:0/1850734825 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc6301066c0 0x7fc63019c850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:58:29.827 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:29.825+0000 7fc637b9e640 1 --2- 172.21.15.29:0/1850734825 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc630101070 0x7fc630198520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:58:29.827 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:29.825+0000 7fc637b9e640 1 --2- 172.21.15.29:0/1850734825 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc6301006b0 0x7fc630197fe0 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:58:29.827 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:29.825+0000 7fc637b9e640 1 -- 172.21.15.29:0/1850734825 >> 172.21.15.29:0/1850734825 conn(0x7fc630077810 msgr2=0x7fc63006a200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:58:29.827 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:29.825+0000 7fc637b9e640 1 -- 172.21.15.29:0/1850734825 shutdown_connections 2024-03-16T11:58:29.828 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:29.825+0000 7fc637b9e640 1 -- 172.21.15.29:0/1850734825 wait complete. 2024-03-16T11:58:30.792 INFO:tasks.workunit.client.0.smithi029.stdout: 1 768141 10.56 MB/sec execute 255 sec latency 117.722 ms 2024-03-16T11:58:30.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:58:30 smithi154 ceph-mon[36320]: pgmap v237: 97 pgs: 97 active+clean; 78 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 2.8 MiB/s wr, 268 op/s 2024-03-16T11:58:30.837 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:58:30 smithi029 ceph-mon[31975]: pgmap v237: 97 pgs: 97 active+clean; 78 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 2.8 MiB/s wr, 268 op/s 2024-03-16T11:58:30.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:58:30 smithi152 ceph-mon[36291]: pgmap v237: 97 pgs: 97 active+clean; 78 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 2.8 MiB/s wr, 268 op/s 2024-03-16T11:58:31.792 INFO:tasks.workunit.client.0.smithi029.stdout: 1 769448 10.54 MB/sec execute 256 sec latency 68.941 ms 2024-03-16T11:58:32.792 INFO:tasks.workunit.client.0.smithi029.stdout: 1 771119 10.55 MB/sec execute 257 sec latency 20.303 ms 2024-03-16T11:58:32.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:58:32 smithi154 ceph-mon[36320]: pgmap v238: 97 pgs: 97 active+clean; 68 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 4.1 MiB/s wr, 437 op/s 2024-03-16T11:58:32.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:58:32 smithi029 ceph-mon[31975]: pgmap v238: 97 pgs: 97 active+clean; 68 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 4.1 MiB/s wr, 437 op/s 2024-03-16T11:58:32.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:58:32 smithi152 ceph-mon[36291]: pgmap v238: 97 pgs: 97 active+clean; 68 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 4.1 MiB/s wr, 437 op/s 2024-03-16T11:58:33.793 INFO:tasks.workunit.client.0.smithi029.stdout: 1 772955 10.53 MB/sec execute 258 sec latency 29.502 ms 2024-03-16T11:58:34.792 INFO:tasks.workunit.client.0.smithi029.stdout: 1 774781 10.54 MB/sec execute 259 sec latency 21.794 ms 2024-03-16T11:58:34.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:58:34 smithi154 ceph-mon[36320]: pgmap v239: 97 pgs: 97 active+clean; 68 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 2.7 MiB/s wr, 257 op/s 2024-03-16T11:58:34.837 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:58:34 smithi029 ceph-mon[31975]: pgmap v239: 97 pgs: 97 active+clean; 68 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 2.7 MiB/s wr, 257 op/s 2024-03-16T11:58:34.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:58:34 smithi152 ceph-mon[36291]: pgmap v239: 97 pgs: 97 active+clean; 68 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 2.7 MiB/s wr, 257 op/s 2024-03-16T11:58:35.793 INFO:tasks.workunit.client.0.smithi029.stdout: 1 776503 10.53 MB/sec execute 260 sec latency 64.895 ms 2024-03-16T11:58:36.793 INFO:tasks.workunit.client.0.smithi029.stdout: 1 778314 10.53 MB/sec execute 261 sec latency 18.475 ms 2024-03-16T11:58:36.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:58:36 smithi154 ceph-mon[36320]: pgmap v240: 97 pgs: 97 active+clean; 68 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 2.7 MiB/s wr, 257 op/s 2024-03-16T11:58:36.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:58:36 smithi029 ceph-mon[31975]: pgmap v240: 97 pgs: 97 active+clean; 68 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 2.7 MiB/s wr, 257 op/s 2024-03-16T11:58:36.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:58:36 smithi152 ceph-mon[36291]: pgmap v240: 97 pgs: 97 active+clean; 68 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 2.7 MiB/s wr, 257 op/s 2024-03-16T11:58:37.793 INFO:tasks.workunit.client.0.smithi029.stdout: 1 780484 10.54 MB/sec execute 262 sec latency 20.472 ms 2024-03-16T11:58:38.793 INFO:tasks.workunit.client.0.smithi029.stdout: 1 782051 10.53 MB/sec execute 263 sec latency 27.724 ms 2024-03-16T11:58:38.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:58:38 smithi154 ceph-mon[36320]: pgmap v241: 97 pgs: 97 active+clean; 74 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 4.0 MiB/s wr, 337 op/s 2024-03-16T11:58:38.837 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:58:38 smithi029 ceph-mon[31975]: pgmap v241: 97 pgs: 97 active+clean; 74 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 4.0 MiB/s wr, 337 op/s 2024-03-16T11:58:38.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:58:38 smithi152 ceph-mon[36291]: pgmap v241: 97 pgs: 97 active+clean; 74 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 4.0 MiB/s wr, 337 op/s 2024-03-16T11:58:39.794 INFO:tasks.workunit.client.0.smithi029.stdout: 1 784012 10.53 MB/sec execute 264 sec latency 53.514 ms 2024-03-16T11:58:40.793 INFO:tasks.workunit.client.0.smithi029.stdout: 1 785301 10.52 MB/sec execute 265 sec latency 76.968 ms 2024-03-16T11:58:40.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:58:40 smithi154 ceph-mon[36320]: pgmap v242: 97 pgs: 97 active+clean; 74 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 2.6 MiB/s wr, 248 op/s 2024-03-16T11:58:40.837 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:58:40 smithi029 ceph-mon[31975]: pgmap v242: 97 pgs: 97 active+clean; 74 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 2.6 MiB/s wr, 248 op/s 2024-03-16T11:58:40.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:58:40 smithi152 ceph-mon[36291]: pgmap v242: 97 pgs: 97 active+clean; 74 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 2.6 MiB/s wr, 248 op/s 2024-03-16T11:58:41.793 INFO:tasks.workunit.client.0.smithi029.stdout: 1 787138 10.50 MB/sec execute 266 sec latency 27.318 ms 2024-03-16T11:58:42.793 INFO:tasks.workunit.client.0.smithi029.stdout: 1 788913 10.51 MB/sec execute 267 sec latency 23.358 ms 2024-03-16T11:58:42.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:58:42 smithi154 ceph-mon[36320]: pgmap v243: 97 pgs: 97 active+clean; 80 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 3.8 MiB/s wr, 329 op/s 2024-03-16T11:58:42.837 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:58:42 smithi029 ceph-mon[31975]: pgmap v243: 97 pgs: 97 active+clean; 80 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 3.8 MiB/s wr, 329 op/s 2024-03-16T11:58:42.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:58:42 smithi152 ceph-mon[36291]: pgmap v243: 97 pgs: 97 active+clean; 80 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 3.8 MiB/s wr, 329 op/s 2024-03-16T11:58:42.920 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:42.919+0000 7f03a52e8640 1 -- 172.21.15.29:0/2903660596 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f03a0102190 msgr2=0x7f03a010e6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:58:42.920 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:42.919+0000 7f03a52e8640 1 --2- 172.21.15.29:0/2903660596 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f03a0102190 0x7f03a010e6a0 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f0398009cd0 tx=0x7f039802c0f0 comp rx=0 tx=0).stop 2024-03-16T11:58:42.920 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:42.919+0000 7f03a52e8640 1 -- 172.21.15.29:0/2903660596 shutdown_connections 2024-03-16T11:58:42.920 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:42.919+0000 7f03a52e8640 1 --2- 172.21.15.29:0/2903660596 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f03a0102190 0x7f03a010e6a0 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:58:42.920 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:42.919+0000 7f03a52e8640 1 --2- 172.21.15.29:0/2903660596 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f03a01017d0 0x7f03a0101c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:58:42.920 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:42.919+0000 7f03a52e8640 1 --2- 172.21.15.29:0/2903660596 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f03a01077e0 0x7f03a0107bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:58:42.920 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:42.919+0000 7f03a52e8640 1 -- 172.21.15.29:0/2903660596 >> 172.21.15.29:0/2903660596 conn(0x7f03a00fd2a0 msgr2=0x7f03a00ff6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:58:42.921 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:42.919+0000 7f03a52e8640 1 -- 172.21.15.29:0/2903660596 shutdown_connections 2024-03-16T11:58:42.921 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:42.919+0000 7f03a52e8640 1 -- 172.21.15.29:0/2903660596 wait complete. 2024-03-16T11:58:42.921 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:42.919+0000 7f03a52e8640 1 Processor -- start 2024-03-16T11:58:42.921 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:42.919+0000 7f03a52e8640 1 -- start start 2024-03-16T11:58:42.921 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:42.920+0000 7f03a52e8640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f03a01017d0 0x7f03a0198030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:58:42.921 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:42.920+0000 7f03a52e8640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f03a0102190 0x7f03a0198570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:58:42.921 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:42.920+0000 7f03a52e8640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f03a01077e0 0x7f03a019c8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:58:42.921 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:42.920+0000 7f03a52e8640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f03a0111bf0 con 0x7f03a01017d0 2024-03-16T11:58:42.921 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:42.920+0000 7f03a52e8640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f03a0111a70 con 0x7f03a01077e0 2024-03-16T11:58:42.921 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:42.920+0000 7f03a52e8640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f03a0111d70 con 0x7f03a0102190 2024-03-16T11:58:42.921 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:42.920+0000 7f039effd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f03a01017d0 0x7f03a0198030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:58:42.922 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:42.920+0000 7f039effd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f03a01017d0 0x7f03a0198030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55792/0 (socket says 172.21.15.29:55792) 2024-03-16T11:58:42.922 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:42.920+0000 7f039effd640 1 -- 172.21.15.29:0/350428363 learned_addr learned my addr 172.21.15.29:0/350428363 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:58:42.922 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:42.920+0000 7f039e7fc640 1 --2- 172.21.15.29:0/350428363 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f03a0102190 0x7f03a0198570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:58:42.922 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:42.920+0000 7f039f7fe640 1 --2- 172.21.15.29:0/350428363 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f03a01077e0 0x7f03a019c8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:58:42.922 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:42.920+0000 7f039effd640 1 -- 172.21.15.29:0/350428363 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f03a0102190 msgr2=0x7f03a0198570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:58:42.922 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:42.920+0000 7f039effd640 1 --2- 172.21.15.29:0/350428363 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f03a0102190 0x7f03a0198570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:58:42.922 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:42.920+0000 7f039effd640 1 -- 172.21.15.29:0/350428363 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f03a01077e0 msgr2=0x7f03a019c8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:58:42.922 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:42.920+0000 7f039effd640 1 --2- 172.21.15.29:0/350428363 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f03a01077e0 0x7f03a019c8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:58:42.922 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:42.920+0000 7f039effd640 1 -- 172.21.15.29:0/350428363 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03a019cfd0 con 0x7f03a01017d0 2024-03-16T11:58:42.922 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:42.920+0000 7f039effd640 1 --2- 172.21.15.29:0/350428363 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f03a01017d0 0x7f03a0198030 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f039000e9f0 tx=0x7f039000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:58:42.922 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:42.920+0000 7f0387fff640 1 -- 172.21.15.29:0/350428363 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f039000cd80 con 0x7f03a01017d0 2024-03-16T11:58:42.922 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:42.920+0000 7f03a52e8640 1 -- 172.21.15.29:0/350428363 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f03a019d2c0 con 0x7f03a01017d0 2024-03-16T11:58:42.922 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:42.920+0000 7f0387fff640 1 -- 172.21.15.29:0/350428363 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0390010910 con 0x7f03a01017d0 2024-03-16T11:58:42.922 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:42.920+0000 7f0387fff640 1 -- 172.21.15.29:0/350428363 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0390018990 con 0x7f03a01017d0 2024-03-16T11:58:42.923 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:42.920+0000 7f03a52e8640 1 -- 172.21.15.29:0/350428363 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f03a01a4b50 con 0x7f03a01017d0 2024-03-16T11:58:42.929 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:42.922+0000 7f0387fff640 1 -- 172.21.15.29:0/350428363 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7f0390010ac0 con 0x7f03a01017d0 2024-03-16T11:58:42.929 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:42.923+0000 7f03a7fab740 1 Processor -- start 2024-03-16T11:58:42.929 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:42.923+0000 7f0387fff640 1 --2- 172.21.15.29:0/350428363 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f03880a2a10 0x7f03880a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:58:42.929 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:42.923+0000 7f03a7fab740 1 -- start start 2024-03-16T11:58:42.929 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:42.923+0000 7f039f7fe640 1 --2- 172.21.15.29:0/350428363 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f03880a2a10 0x7f03880a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:58:42.929 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:42.923+0000 7f0387fff640 1 -- 172.21.15.29:0/350428363 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8556+0+0 (secure 0 0 0) 0x7f03900cf140 con 0x7f03a01017d0 2024-03-16T11:58:42.929 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:42.923+0000 7f03a7fab740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f171a8b130 0x55f171a8b510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:58:42.929 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:42.923+0000 7f03a7fab740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f171a8ba50 0x55f171a942e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:58:42.929 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:42.923+0000 7f03a7fab740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f171a94820 0x55f171a96c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:58:42.929 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:42.923+0000 7f03a7fab740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55f1719efa40 con 0x55f171a8ba50 2024-03-16T11:58:42.930 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:42.923+0000 7f03a7fab740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x55f171998920 con 0x55f171a8b130 2024-03-16T11:58:42.930 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:42.923+0000 7f03a7fab740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55f1719ebae0 con 0x55f171a94820 2024-03-16T11:58:42.930 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:42.923+0000 7f039e7fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f171a8ba50 0x55f171a942e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:58:42.930 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:42.923+0000 7f039e7fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f171a8ba50 0x55f171a942e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55794/0 (socket says 172.21.15.29:55794) 2024-03-16T11:58:42.930 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:42.924+0000 7f039e7fc640 1 -- 172.21.15.29:0/2688777448 learned_addr learned my addr 172.21.15.29:0/2688777448 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:58:42.930 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:42.924+0000 7f039f7fe640 1 --2- 172.21.15.29:0/2688777448 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f171a94820 0x55f171a96c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:58:42.930 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:42.924+0000 7f039effd640 1 --2- 172.21.15.29:0/2688777448 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f171a8b130 0x55f171a8b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:58:42.930 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:42.924+0000 7f039e7fc640 1 -- 172.21.15.29:0/2688777448 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f171a94820 msgr2=0x55f171a96c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:58:42.930 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:42.924+0000 7f039e7fc640 1 --2- 172.21.15.29:0/2688777448 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f171a94820 0x55f171a96c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:58:42.930 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:42.924+0000 7f039e7fc640 1 -- 172.21.15.29:0/2688777448 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f171a8b130 msgr2=0x55f171a8b510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:58:42.930 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:42.924+0000 7f039e7fc640 1 --2- 172.21.15.29:0/2688777448 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f171a8b130 0x55f171a8b510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:58:42.930 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:42.924+0000 7f039e7fc640 1 -- 172.21.15.29:0/2688777448 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f1719b0460 con 0x55f171a8ba50 2024-03-16T11:58:42.930 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:42.924+0000 7f039f7fe640 1 --2- 172.21.15.29:0/350428363 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f03880a2a10 0x7f03880a4ed0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f0398005cc0 tx=0x7f039802ce60 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:58:42.930 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:42.924+0000 7f039e7fc640 1 --2- 172.21.15.29:0/2688777448 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f171a8ba50 0x55f171a942e0 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f0394009ca0 tx=0x7f0394011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2efd296f21627a95 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:58:42.931 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:42.924+0000 7f03857fa640 1 -- 172.21.15.29:0/2688777448 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f039401e070 con 0x55f171a8ba50 2024-03-16T11:58:42.931 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:42.924+0000 7f03857fa640 1 -- 172.21.15.29:0/2688777448 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0394009e50 con 0x55f171a8ba50 2024-03-16T11:58:42.931 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:42.924+0000 7f03857fa640 1 -- 172.21.15.29:0/2688777448 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0394018790 con 0x55f171a8ba50 2024-03-16T11:58:42.931 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:42.925+0000 7f03a7fab740 1 -- 172.21.15.29:0/2688777448 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f171a8ba50 msgr2=0x55f171a942e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:58:42.931 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:42.925+0000 7f03a7fab740 1 --2- 172.21.15.29:0/2688777448 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f171a8ba50 0x55f171a942e0 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f0394009ca0 tx=0x7f0394011810 comp rx=0 tx=0).stop 2024-03-16T11:58:42.931 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:42.925+0000 7f03a7fab740 1 -- 172.21.15.29:0/2688777448 shutdown_connections 2024-03-16T11:58:42.931 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:42.925+0000 7f03a7fab740 1 --2- 172.21.15.29:0/2688777448 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f171a94820 0x55f171a96c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:58:42.931 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:42.925+0000 7f03a7fab740 1 --2- 172.21.15.29:0/2688777448 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f171a8ba50 0x55f171a942e0 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:58:42.931 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:42.925+0000 7f03a7fab740 1 --2- 172.21.15.29:0/2688777448 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f171a8b130 0x55f171a8b510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:58:42.931 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:42.925+0000 7f03a7fab740 1 -- 172.21.15.29:0/2688777448 >> 172.21.15.29:0/2688777448 conn(0x55f171a85a50 msgr2=0x55f171a85e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:58:42.931 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:42.925+0000 7f03a7fab740 1 -- 172.21.15.29:0/2688777448 shutdown_connections 2024-03-16T11:58:42.931 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:42.925+0000 7f03a7fab740 1 -- 172.21.15.29:0/2688777448 wait complete. 2024-03-16T11:58:42.931 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:42.925+0000 7f03a7fab740 1 -- start start 2024-03-16T11:58:42.931 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:42.925+0000 7f03a7fab740 1 Processor -- start 2024-03-16T11:58:42.931 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:42.926+0000 7f03a7fab740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f171a8b130 0x55f171a91850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:58:42.932 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:42.926+0000 7f03a7fab740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f171a8ba50 0x55f171a91d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:58:42.932 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:42.926+0000 7f03a7fab740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f171a94820 0x55f171a922d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:58:42.932 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:42.926+0000 7f03a7fab740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55f171a97450 con 0x55f171a8b130 2024-03-16T11:58:42.932 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:42.926+0000 7f03a7fab740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x55f1719abd70 con 0x55f171a8ba50 2024-03-16T11:58:42.932 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:42.926+0000 7f03a7fab740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55f171a975d0 con 0x55f171a94820 2024-03-16T11:58:42.932 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:42.926+0000 7f03a7fab740 10 client.0 _ll_register_callbacks cb 0x55f171703a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T11:58:42.932 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:42.926+0000 7f039effd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f171a8b130 0x55f171a91850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:58:42.932 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:42.926+0000 7f039effd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f171a8b130 0x55f171a91850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55796/0 (socket says 172.21.15.29:55796) 2024-03-16T11:58:42.932 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:42.926+0000 7f039effd640 1 -- 172.21.15.29:0/4126535878 learned_addr learned my addr 172.21.15.29:0/4126535878 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:58:42.932 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:42.926+0000 7f039f7fe640 1 --2- 172.21.15.29:0/4126535878 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f171a94820 0x55f171a922d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:58:42.932 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:42.926+0000 7f039e7fc640 1 --2- 172.21.15.29:0/4126535878 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f171a8ba50 0x55f171a91d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:58:42.932 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:42.926+0000 7f039effd640 1 -- 172.21.15.29:0/4126535878 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f171a94820 msgr2=0x55f171a922d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:58:42.932 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:42.926+0000 7f039effd640 1 --2- 172.21.15.29:0/4126535878 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f171a94820 0x55f171a922d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:58:42.932 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:42.926+0000 7f039effd640 1 -- 172.21.15.29:0/4126535878 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f171a8ba50 msgr2=0x55f171a91d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:58:42.933 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:42.926+0000 7f039effd640 1 --2- 172.21.15.29:0/4126535878 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f171a8ba50 0x55f171a91d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:58:42.933 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:42.926+0000 7f039effd640 1 -- 172.21.15.29:0/4126535878 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f171704210 con 0x55f171a8b130 2024-03-16T11:58:42.933 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:42.926+0000 7f039effd640 1 --2- 172.21.15.29:0/4126535878 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f171a8b130 0x55f171a91850 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f0390097c40 tx=0x7f039009b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac09ea5acdcccb51 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:58:42.933 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:42.926+0000 7f0382ff5640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-16T11:58:42.933 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:42.926+0000 7f0382ff5640 1 -- 172.21.15.29:0/4126535878 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03900b4060 con 0x55f171a8b130 2024-03-16T11:58:42.933 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:42.926+0000 7f0382ff5640 1 -- 172.21.15.29:0/4126535878 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8556+0+0 (secure 0 0 0) 0x7f03900afb60 con 0x55f171a8b130 2024-03-16T11:58:42.933 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:42.926+0000 7f03a7fab740 1 -- 172.21.15.29:0/4126535878 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0394009e50 con 0x55f171a8b130 2024-03-16T11:58:42.933 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:42.927+0000 7f0382ff5640 1 -- 172.21.15.29:0/4126535878 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f03900a5050 con 0x55f171a8b130 2024-03-16T11:58:42.933 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:42.927+0000 7f03a7fab740 10 client.16050 fetch_fsmap learned FSMap version 26 2024-03-16T11:58:42.933 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:42.927+0000 7f03a7fab740 1 -- 172.21.15.29:0/4126535878 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55f1717d0620 con 0x55f171a8b130 2024-03-16T11:58:42.933 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:42.927+0000 7f0382ff5640 1 -- 172.21.15.29:0/4126535878 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0394009e50 con 0x55f171a8b130 2024-03-16T11:58:42.933 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:42.927+0000 7f0382ff5640 1 -- 172.21.15.29:0/4126535878 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f039009ac80 con 0x55f171a8b130 2024-03-16T11:58:42.933 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:42.927+0000 7f03a7fab740 10 client.16050 fetch_fsmap finished waiting for FSMap version 26 2024-03-16T11:58:42.933 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:42.927+0000 7f03a7fab740 10 client.16050 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T11:58:42.933 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:42.927+0000 7f03a7fab740 20 client.16050 populate_metadata read hostname 'smithi029' 2024-03-16T11:58:42.934 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:42.927+0000 7f03a7fab740 1 --2- 172.21.15.29:0/4126535878 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55f171aa4950 0x55f171aa6d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:58:42.934 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:42.927+0000 7f03a7fab740 4 client.16050 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T11:58:42.934 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:42.927+0000 7f03a7fab740 1 -- 172.21.15.29:0/4126535878 --> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f1719fcf80 con 0x55f171aa4950 2024-03-16T11:58:42.934 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:42.927+0000 7f039e7fc640 1 --2- 172.21.15.29:0/4126535878 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55f171aa4950 0x55f171aa6d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:58:42.934 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:42.928+0000 7f0382ff5640 10 client.16050 ms_handle_connect on v2:172.21.15.152:6838/1548731833 2024-03-16T11:58:42.934 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:42.928+0000 7f039e7fc640 1 --2- 172.21.15.29:0/4126535878 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55f171aa4950 0x55f171aa6d40 crc :-1 s=READY pgs=108 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8dfba4ba500b6a98 server_cookie=bcc79dffb7907ad5 in_seq=0 out_seq=0 2024-03-16T11:58:42.934 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:42.929+0000 7f0382ff5640 1 -- 172.21.15.29:0/4126535878 <== mds.0 v2:172.21.15.152:6838/1548731833 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55f1719fcf80 con 0x55f171aa4950 2024-03-16T11:58:42.934 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:42.929+0000 7f0382ff5640 10 client.16050 handle_command_reply: tid=0 2024-03-16T11:58:42.934 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:42.929+0000 7f03a7fab740 1 client.16050 shutdown 2024-03-16T11:58:42.934 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:42.929+0000 7f03a7fab740 1 -- 172.21.15.29:0/4126535878 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f171a8b130 msgr2=0x55f171a91850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:58:42.934 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:42.929+0000 7f03a7fab740 1 --2- 172.21.15.29:0/4126535878 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f171a8b130 0x55f171a91850 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f0390097c40 tx=0x7f039009b8f0 comp rx=0 tx=0).stop 2024-03-16T11:58:42.934 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:42.929+0000 7f03a7fab740 1 -- 172.21.15.29:0/4126535878 shutdown_connections 2024-03-16T11:58:42.934 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:42.929+0000 7f03a7fab740 1 --2- 172.21.15.29:0/4126535878 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f171a94820 0x55f171a922d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:58:42.934 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:42.929+0000 7f03a7fab740 1 --2- 172.21.15.29:0/4126535878 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55f171aa4950 0x55f171aa6d40 crc :-1 s=READY pgs=108 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:58:42.935 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:42.930+0000 7f03a7fab740 1 --2- 172.21.15.29:0/4126535878 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f171a8ba50 0x55f171a91d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:58:42.935 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:42.930+0000 7f03a7fab740 1 --2- 172.21.15.29:0/4126535878 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f171a8b130 0x55f171a91850 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:58:42.935 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:42.930+0000 7f0382ff5640 0 client.16050 ms_handle_reset on v2:172.21.15.152:6838/1548731833 2024-03-16T11:58:42.935 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:42.930+0000 7f03a7fab740 1 -- 172.21.15.29:0/4126535878 >> 172.21.15.29:0/4126535878 conn(0x55f171a85a50 msgr2=0x55f171a877a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:58:42.935 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:42.930+0000 7f03a7fab740 1 -- 172.21.15.29:0/4126535878 shutdown_connections 2024-03-16T11:58:42.935 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:42.930+0000 7f03a7fab740 1 -- 172.21.15.29:0/4126535878 wait complete. 2024-03-16T11:58:42.935 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:42.930+0000 7f03a7fab740 20 client.16050 trim_cache size 0 max 16384 2024-03-16T11:58:42.946 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:42.946+0000 7f03a7fab740 1 Processor -- start 2024-03-16T11:58:42.946 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:42.946+0000 7f03a7fab740 1 -- start start 2024-03-16T11:58:42.947 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:42.946+0000 7f03a7fab740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f171a8b130 0x55f171a93f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:58:42.947 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:42.946+0000 7f03a7fab740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f171a8ba50 0x55f171aa01f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:58:42.947 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:42.946+0000 7f03a7fab740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f171a94820 0x55f171aac620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:58:42.947 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:42.946+0000 7f03a7fab740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f03a01afdd0 con 0x55f171a8b130 2024-03-16T11:58:42.947 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:42.946+0000 7f03a7fab740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f03a01ad6a0 con 0x55f171a94820 2024-03-16T11:58:42.947 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:42.946+0000 7f03a7fab740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f03a01ab510 con 0x55f171a8ba50 2024-03-16T11:58:42.947 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:42.947+0000 7f039effd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f171a8b130 0x55f171a93f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:58:42.947 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:42.947+0000 7f039effd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f171a8b130 0x55f171a93f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55804/0 (socket says 172.21.15.29:55804) 2024-03-16T11:58:42.947 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:42.947+0000 7f039effd640 1 -- 172.21.15.29:0/3437502305 learned_addr learned my addr 172.21.15.29:0/3437502305 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:58:42.947 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:42.947+0000 7f039e7fc640 1 --2- 172.21.15.29:0/3437502305 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f171a8ba50 0x55f171aa01f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:58:42.947 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:42.947+0000 7f039f7fe640 1 --2- 172.21.15.29:0/3437502305 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f171a94820 0x55f171aac620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:58:42.947 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:42.947+0000 7f039effd640 1 -- 172.21.15.29:0/3437502305 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f171a8ba50 msgr2=0x55f171aa01f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:58:42.947 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:42.947+0000 7f039effd640 1 --2- 172.21.15.29:0/3437502305 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f171a8ba50 0x55f171aa01f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:58:42.947 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:42.947+0000 7f039effd640 1 -- 172.21.15.29:0/3437502305 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f171a94820 msgr2=0x55f171aac620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:58:42.947 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:42.947+0000 7f039effd640 1 --2- 172.21.15.29:0/3437502305 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f171a94820 0x55f171aac620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:58:42.948 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:42.947+0000 7f039effd640 1 -- 172.21.15.29:0/3437502305 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f171a8b880 con 0x55f171a8b130 2024-03-16T11:58:42.948 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:42.947+0000 7f039effd640 1 --2- 172.21.15.29:0/3437502305 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f171a8b130 0x55f171a93f80 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f0390099ae0 tx=0x7f03900a40a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1335f49d29c561bd server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:58:42.948 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:42.947+0000 7f0384ff9640 1 -- 172.21.15.29:0/3437502305 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03900b4040 con 0x55f171a8b130 2024-03-16T11:58:42.948 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:42.947+0000 7f0384ff9640 1 -- 172.21.15.29:0/3437502305 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f039009f8d0 con 0x55f171a8b130 2024-03-16T11:58:42.948 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:42.947+0000 7f0384ff9640 1 -- 172.21.15.29:0/3437502305 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03900afb60 con 0x55f171a8b130 2024-03-16T11:58:42.951 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:42.948+0000 7f03a7fab740 1 -- 172.21.15.29:0/3437502305 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f171a8b130 msgr2=0x55f171a93f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:58:42.952 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:42.948+0000 7f03a7fab740 1 --2- 172.21.15.29:0/3437502305 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f171a8b130 0x55f171a93f80 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f0390099ae0 tx=0x7f03900a40a0 comp rx=0 tx=0).stop 2024-03-16T11:58:42.952 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:42.948+0000 7f03a7fab740 1 -- 172.21.15.29:0/3437502305 shutdown_connections 2024-03-16T11:58:42.952 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:42.948+0000 7f03a7fab740 1 --2- 172.21.15.29:0/3437502305 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f171a94820 0x55f171aac620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:58:42.952 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:42.948+0000 7f03a7fab740 1 --2- 172.21.15.29:0/3437502305 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f171a8ba50 0x55f171aa01f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:58:42.952 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:42.948+0000 7f03a7fab740 1 --2- 172.21.15.29:0/3437502305 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f171a8b130 0x55f171a93f80 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:58:42.952 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:42.948+0000 7f03a7fab740 1 -- 172.21.15.29:0/3437502305 >> 172.21.15.29:0/3437502305 conn(0x55f171a85a50 msgr2=0x55f171a876c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:58:42.952 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:42.948+0000 7f03a7fab740 1 -- 172.21.15.29:0/3437502305 shutdown_connections 2024-03-16T11:58:42.952 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:42.948+0000 7f03a7fab740 1 -- 172.21.15.29:0/3437502305 wait complete. 2024-03-16T11:58:42.952 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:42.948+0000 7f03a7fab740 1 -- start start 2024-03-16T11:58:42.952 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:42.948+0000 7f03a7fab740 1 Processor -- start 2024-03-16T11:58:42.952 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:42.949+0000 7f03a7fab740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f171a8b130 0x55f171a98990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:58:42.952 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:42.949+0000 7f03a7fab740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f171a8ba50 0x55f171a98ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:58:42.952 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:42.949+0000 7f03a7fab740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f171a94820 0x55f171b03c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:58:42.952 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:42.949+0000 7f03a7fab740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f03a01abb60 con 0x55f171a8b130 2024-03-16T11:58:42.953 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:42.949+0000 7f03a7fab740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f03a01acfd0 con 0x55f171a8ba50 2024-03-16T11:58:42.953 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:42.949+0000 7f03a7fab740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f03a01ac230 con 0x55f171a94820 2024-03-16T11:58:42.953 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:42.949+0000 7f03a7fab740 10 client.0 _ll_register_callbacks cb 0x7f039009b690 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T11:58:42.953 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:42.949+0000 7f039effd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f171a8b130 0x55f171a98990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:58:42.953 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:42.949+0000 7f039effd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f171a8b130 0x55f171a98990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55806/0 (socket says 172.21.15.29:55806) 2024-03-16T11:58:42.953 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:42.949+0000 7f039effd640 1 -- 172.21.15.29:0/3111442701 learned_addr learned my addr 172.21.15.29:0/3111442701 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:58:42.953 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:42.949+0000 7f039f7fe640 1 --2- 172.21.15.29:0/3111442701 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f171a94820 0x55f171b03c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:58:42.953 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:42.949+0000 7f039e7fc640 1 --2- 172.21.15.29:0/3111442701 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f171a8ba50 0x55f171a98ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:58:42.953 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:42.949+0000 7f039effd640 1 -- 172.21.15.29:0/3111442701 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f171a94820 msgr2=0x55f171b03c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:58:42.953 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:42.949+0000 7f039effd640 1 --2- 172.21.15.29:0/3111442701 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f171a94820 0x55f171b03c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:58:42.953 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:42.949+0000 7f039effd640 1 -- 172.21.15.29:0/3111442701 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f171a8ba50 msgr2=0x55f171a98ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:58:42.953 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:42.949+0000 7f039effd640 1 --2- 172.21.15.29:0/3111442701 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f171a8ba50 0x55f171a98ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:58:42.953 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:42.949+0000 7f039effd640 1 -- 172.21.15.29:0/3111442701 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f171b04260 con 0x55f171a8b130 2024-03-16T11:58:42.953 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:42.949+0000 7f039f7fe640 1 --2- 172.21.15.29:0/3111442701 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f171a94820 0x55f171b03c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:58:42.953 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:42.949+0000 7f039effd640 1 --2- 172.21.15.29:0/3111442701 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f171a8b130 0x55f171a98990 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f039000e9c0 tx=0x7f039009ac80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e5b0eea7ef50222 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:58:42.954 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:42.949+0000 7f0382ff5640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-16T11:58:42.954 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:42.949+0000 7f0382ff5640 1 -- 172.21.15.29:0/3111442701 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03900b4070 con 0x55f171a8b130 2024-03-16T11:58:42.954 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:42.950+0000 7f03a7fab740 1 -- 172.21.15.29:0/3111442701 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f039009f8d0 con 0x55f171a8b130 2024-03-16T11:58:42.954 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:42.950+0000 7f0382ff5640 1 -- 172.21.15.29:0/3111442701 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8556+0+0 (secure 0 0 0) 0x7f03900afd60 con 0x55f171a8b130 2024-03-16T11:58:42.954 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:42.950+0000 7f0382ff5640 1 -- 172.21.15.29:0/3111442701 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f0390097ad0 con 0x55f171a8b130 2024-03-16T11:58:42.954 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:42.950+0000 7f03a7fab740 10 client.16056 fetch_fsmap learned FSMap version 26 2024-03-16T11:58:42.954 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:42.950+0000 7f03a7fab740 1 -- 172.21.15.29:0/3111442701 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55f171b0b5e0 con 0x55f171a8b130 2024-03-16T11:58:42.954 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:42.950+0000 7f0382ff5640 1 -- 172.21.15.29:0/3111442701 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f039009f8d0 con 0x55f171a8b130 2024-03-16T11:58:42.954 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:42.950+0000 7f0382ff5640 1 -- 172.21.15.29:0/3111442701 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f039009ccb0 con 0x55f171a8b130 2024-03-16T11:58:42.954 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:42.950+0000 7f03a7fab740 10 client.16056 fetch_fsmap finished waiting for FSMap version 26 2024-03-16T11:58:42.954 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:42.950+0000 7f03a7fab740 10 client.16056 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T11:58:42.954 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:42.950+0000 7f03a7fab740 20 client.16056 populate_metadata read hostname 'smithi029' 2024-03-16T11:58:42.954 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:42.950+0000 7f03a7fab740 1 --2- 172.21.15.29:0/3111442701 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55f171b0b950 0x55f171b0dd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:58:42.954 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:42.950+0000 7f03a7fab740 4 client.16056 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-16T11:58:42.955 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:42.950+0000 7f03a7fab740 1 -- 172.21.15.29:0/3111442701 --> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55f1719f72d0 con 0x55f171b0b950 2024-03-16T11:58:42.955 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:42.950+0000 7f039e7fc640 1 --2- 172.21.15.29:0/3111442701 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55f171b0b950 0x55f171b0dd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:58:42.955 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:42.951+0000 7f039e7fc640 1 --2- 172.21.15.29:0/3111442701 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55f171b0b950 0x55f171b0dd70 crc :-1 s=READY pgs=109 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=30e7d29043c384bc server_cookie=a6e03c32fba2381b in_seq=0 out_seq=0 2024-03-16T11:58:42.955 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:42.951+0000 7f0382ff5640 10 client.16056 ms_handle_connect on v2:172.21.15.152:6838/1548731833 2024-03-16T11:58:43.118 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:43.117+0000 7f0382ff5640 1 -- 172.21.15.29:0/3111442701 <== mds.0 v2:172.21.15.152:6838/1548731833 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55f1719f72d0 con 0x55f171b0b950 2024-03-16T11:58:43.118 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:43.117+0000 7f0382ff5640 10 client.16056 handle_command_reply: tid=0 2024-03-16T11:58:43.118 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:43.118+0000 7f03a7fab740 1 client.16056 shutdown 2024-03-16T11:58:43.118 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:43.118+0000 7f03a7fab740 1 -- 172.21.15.29:0/3111442701 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f171a8b130 msgr2=0x55f171a98990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:58:43.118 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:43.118+0000 7f03a7fab740 1 --2- 172.21.15.29:0/3111442701 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f171a8b130 0x55f171a98990 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f039000e9c0 tx=0x7f039009ac80 comp rx=0 tx=0).stop 2024-03-16T11:58:43.118 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:43.118+0000 7f03a7fab740 1 -- 172.21.15.29:0/3111442701 shutdown_connections 2024-03-16T11:58:43.118 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:43.118+0000 7f03a7fab740 1 --2- 172.21.15.29:0/3111442701 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f171a94820 0x55f171b03c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:58:43.118 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:43.118+0000 7f03a7fab740 1 --2- 172.21.15.29:0/3111442701 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55f171b0b950 0x55f171b0dd70 crc :-1 s=READY pgs=109 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:58:43.118 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:43.118+0000 7f0382ff5640 0 client.16056 ms_handle_reset on v2:172.21.15.152:6838/1548731833 2024-03-16T11:58:43.119 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:43.118+0000 7f03a7fab740 1 --2- 172.21.15.29:0/3111442701 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f171a8ba50 0x55f171a98ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:58:43.119 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:43.118+0000 7f03a7fab740 1 --2- 172.21.15.29:0/3111442701 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f171a8b130 0x55f171a98990 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:58:43.119 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:43.118+0000 7f03a7fab740 1 -- 172.21.15.29:0/3111442701 >> 172.21.15.29:0/3111442701 conn(0x55f171a85a50 msgr2=0x55f171a86dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:58:43.119 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:43.118+0000 7f03a7fab740 1 -- 172.21.15.29:0/3111442701 shutdown_connections 2024-03-16T11:58:43.119 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:43.118+0000 7f03a7fab740 1 -- 172.21.15.29:0/3111442701 wait complete. 2024-03-16T11:58:43.119 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:43.118+0000 7f03a7fab740 20 client.16056 trim_cache size 0 max 16384 2024-03-16T11:58:43.119 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:{ 2024-03-16T11:58:43.119 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "message": "", 2024-03-16T11:58:43.119 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "return_code": 0 2024-03-16T11:58:43.120 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:} 2024-03-16T11:58:43.121 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:43.119+0000 7f03a52e8640 1 -- 172.21.15.29:0/350428363 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f03880a2a10 msgr2=0x7f03880a4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:58:43.121 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:43.119+0000 7f03a52e8640 1 --2- 172.21.15.29:0/350428363 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f03880a2a10 0x7f03880a4ed0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f0398005cc0 tx=0x7f039802ce60 comp rx=0 tx=0).stop 2024-03-16T11:58:43.121 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:43.119+0000 7f03a52e8640 1 -- 172.21.15.29:0/350428363 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f03a01017d0 msgr2=0x7f03a0198030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:58:43.121 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:43.119+0000 7f03a52e8640 1 --2- 172.21.15.29:0/350428363 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f03a01017d0 0x7f03a0198030 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f039000e9f0 tx=0x7f039000eeb0 comp rx=0 tx=0).stop 2024-03-16T11:58:43.121 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:43.119+0000 7f03a52e8640 1 -- 172.21.15.29:0/350428363 shutdown_connections 2024-03-16T11:58:43.121 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:43.119+0000 7f03a52e8640 1 --2- 172.21.15.29:0/350428363 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f03880a2a10 0x7f03880a4ed0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:58:43.121 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:43.119+0000 7f03a52e8640 1 --2- 172.21.15.29:0/350428363 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f03a01077e0 0x7f03a019c8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:58:43.121 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:43.119+0000 7f03a52e8640 1 --2- 172.21.15.29:0/350428363 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f03a0102190 0x7f03a0198570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:58:43.121 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:43.120+0000 7f03a52e8640 1 --2- 172.21.15.29:0/350428363 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f03a01017d0 0x7f03a0198030 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:58:43.122 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:43.120+0000 7f03a52e8640 1 -- 172.21.15.29:0/350428363 >> 172.21.15.29:0/350428363 conn(0x7f03a00fd2a0 msgr2=0x7f03a00fd7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:58:43.122 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:43.120+0000 7f03a52e8640 1 -- 172.21.15.29:0/350428363 shutdown_connections 2024-03-16T11:58:43.122 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:43.120+0000 7f03a52e8640 1 -- 172.21.15.29:0/350428363 wait complete. 2024-03-16T11:58:43.793 INFO:tasks.workunit.client.0.smithi029.stdout: 1 790627 10.50 MB/sec execute 268 sec latency 102.135 ms 2024-03-16T11:58:44.793 INFO:tasks.workunit.client.0.smithi029.stdout: 1 792378 10.50 MB/sec execute 269 sec latency 17.023 ms 2024-03-16T11:58:44.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:58:44 smithi154 ceph-mon[36320]: pgmap v244: 97 pgs: 97 active+clean; 80 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 2.5 MiB/s wr, 160 op/s 2024-03-16T11:58:44.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:58:44 smithi029 ceph-mon[31975]: pgmap v244: 97 pgs: 97 active+clean; 80 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 2.5 MiB/s wr, 160 op/s 2024-03-16T11:58:44.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:58:44 smithi152 ceph-mon[36291]: pgmap v244: 97 pgs: 97 active+clean; 80 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 2.5 MiB/s wr, 160 op/s 2024-03-16T11:58:45.793 INFO:tasks.workunit.client.0.smithi029.stdout: 1 794021 10.49 MB/sec execute 270 sec latency 49.685 ms 2024-03-16T11:58:46.793 INFO:tasks.workunit.client.0.smithi029.stdout: 1 795918 10.50 MB/sec execute 271 sec latency 23.193 ms 2024-03-16T11:58:46.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:58:46 smithi154 ceph-mon[36320]: pgmap v245: 97 pgs: 97 active+clean; 80 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 2.5 MiB/s wr, 160 op/s 2024-03-16T11:58:46.837 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:58:46 smithi029 ceph-mon[31975]: pgmap v245: 97 pgs: 97 active+clean; 80 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 2.5 MiB/s wr, 160 op/s 2024-03-16T11:58:46.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:58:46 smithi152 ceph-mon[36291]: pgmap v245: 97 pgs: 97 active+clean; 80 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 2.5 MiB/s wr, 160 op/s 2024-03-16T11:58:47.793 INFO:tasks.workunit.client.0.smithi029.stdout: 1 797880 10.49 MB/sec execute 272 sec latency 19.081 ms 2024-03-16T11:58:48.793 INFO:tasks.workunit.client.0.smithi029.stdout: 1 799584 10.49 MB/sec execute 273 sec latency 22.862 ms 2024-03-16T11:58:48.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:58:48 smithi154 ceph-mon[36320]: pgmap v246: 97 pgs: 97 active+clean; 70 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 3.7 MiB/s wr, 331 op/s 2024-03-16T11:58:48.837 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:58:48 smithi029 ceph-mon[31975]: pgmap v246: 97 pgs: 97 active+clean; 70 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 3.7 MiB/s wr, 331 op/s 2024-03-16T11:58:48.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:58:48 smithi152 ceph-mon[36291]: pgmap v246: 97 pgs: 97 active+clean; 70 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 3.7 MiB/s wr, 331 op/s 2024-03-16T11:58:49.793 INFO:tasks.workunit.client.0.smithi029.stdout: 1 801614 10.49 MB/sec execute 274 sec latency 19.112 ms 2024-03-16T11:58:50.793 INFO:tasks.workunit.client.0.smithi029.stdout: 1 802978 10.48 MB/sec execute 275 sec latency 114.815 ms 2024-03-16T11:58:50.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:58:50 smithi154 ceph-mon[36320]: pgmap v247: 97 pgs: 97 active+clean; 70 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 2.4 MiB/s wr, 252 op/s 2024-03-16T11:58:50.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:58:50 smithi029 ceph-mon[31975]: pgmap v247: 97 pgs: 97 active+clean; 70 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 2.4 MiB/s wr, 252 op/s 2024-03-16T11:58:50.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:58:50 smithi152 ceph-mon[36291]: pgmap v247: 97 pgs: 97 active+clean; 70 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 2.4 MiB/s wr, 252 op/s 2024-03-16T11:58:51.793 INFO:tasks.workunit.client.0.smithi029.stdout: 1 804851 10.47 MB/sec execute 276 sec latency 20.488 ms 2024-03-16T11:58:52.793 INFO:tasks.workunit.client.0.smithi029.stdout: 1 806574 10.48 MB/sec execute 277 sec latency 15.558 ms 2024-03-16T11:58:52.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:58:52 smithi154 ceph-mon[36320]: pgmap v248: 97 pgs: 97 active+clean; 76 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 3.9 MiB/s wr, 340 op/s 2024-03-16T11:58:52.837 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:58:52 smithi029 ceph-mon[31975]: pgmap v248: 97 pgs: 97 active+clean; 76 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 3.9 MiB/s wr, 340 op/s 2024-03-16T11:58:52.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:58:52 smithi152 ceph-mon[36291]: pgmap v248: 97 pgs: 97 active+clean; 76 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 3.9 MiB/s wr, 340 op/s 2024-03-16T11:58:53.794 INFO:tasks.workunit.client.0.smithi029.stdout: 1 808513 10.47 MB/sec execute 278 sec latency 20.738 ms 2024-03-16T11:58:54.794 INFO:tasks.workunit.client.0.smithi029.stdout: 1 810385 10.47 MB/sec execute 279 sec latency 26.459 ms 2024-03-16T11:58:54.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:58:54 smithi154 ceph-mon[36320]: pgmap v249: 97 pgs: 97 active+clean; 76 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 2.7 MiB/s wr, 259 op/s 2024-03-16T11:58:54.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:58:54 smithi029 ceph-mon[31975]: pgmap v249: 97 pgs: 97 active+clean; 76 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 2.7 MiB/s wr, 259 op/s 2024-03-16T11:58:54.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:58:54 smithi152 ceph-mon[36291]: pgmap v249: 97 pgs: 97 active+clean; 76 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 2.7 MiB/s wr, 259 op/s 2024-03-16T11:58:55.794 INFO:tasks.workunit.client.0.smithi029.stdout: 1 812002 10.46 MB/sec execute 280 sec latency 127.737 ms 2024-03-16T11:58:56.212 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.210+0000 7fc38e959640 1 -- 172.21.15.29:0/983515886 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc388102160 msgr2=0x7fc38810e670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:58:56.212 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.210+0000 7fc38e959640 1 --2- 172.21.15.29:0/983515886 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc388102160 0x7fc38810e670 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7fc380009cd0 tx=0x7fc38002c0f0 comp rx=0 tx=0).stop 2024-03-16T11:58:56.212 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.210+0000 7fc38e959640 1 -- 172.21.15.29:0/983515886 shutdown_connections 2024-03-16T11:58:56.212 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.210+0000 7fc38e959640 1 --2- 172.21.15.29:0/983515886 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc388102160 0x7fc38810e670 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:58:56.212 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.210+0000 7fc38e959640 1 --2- 172.21.15.29:0/983515886 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc3881017a0 0x7fc388101c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:58:56.212 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.210+0000 7fc38e959640 1 --2- 172.21.15.29:0/983515886 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc3881077b0 0x7fc388107b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:58:56.212 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.210+0000 7fc38e959640 1 -- 172.21.15.29:0/983515886 >> 172.21.15.29:0/983515886 conn(0x7fc3880fd290 msgr2=0x7fc3880ff6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:58:56.212 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.210+0000 7fc38e959640 1 -- 172.21.15.29:0/983515886 shutdown_connections 2024-03-16T11:58:56.212 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.210+0000 7fc38e959640 1 -- 172.21.15.29:0/983515886 wait complete. 2024-03-16T11:58:56.212 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.211+0000 7fc38e959640 1 Processor -- start 2024-03-16T11:58:56.213 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.211+0000 7fc38e959640 1 -- start start 2024-03-16T11:58:56.213 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.211+0000 7fc38e959640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc3881017a0 0x7fc388197fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:58:56.213 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.211+0000 7fc38e959640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc388102160 0x7fc3881984e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:58:56.213 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.211+0000 7fc38e959640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc3881077b0 0x7fc38819c860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:58:56.213 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.211+0000 7fc38e959640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fc388111bc0 con 0x7fc3881077b0 2024-03-16T11:58:56.213 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.211+0000 7fc38e959640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fc388111a40 con 0x7fc388102160 2024-03-16T11:58:56.213 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.211+0000 7fc38e959640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fc388111d40 con 0x7fc3881017a0 2024-03-16T11:58:56.213 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.212+0000 7fc38cef6640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc3881077b0 0x7fc38819c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:58:56.213 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.212+0000 7fc387fff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc3881017a0 0x7fc388197fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:58:56.213 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.212+0000 7fc3877fe640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc388102160 0x7fc3881984e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:58:56.213 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.212+0000 7fc38cef6640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc3881077b0 0x7fc38819c860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60728/0 (socket says 172.21.15.29:60728) 2024-03-16T11:58:56.213 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.212+0000 7fc38cef6640 1 -- 172.21.15.29:0/2366729666 learned_addr learned my addr 172.21.15.29:0/2366729666 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:58:56.213 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.212+0000 7fc38cef6640 1 -- 172.21.15.29:0/2366729666 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc3881017a0 msgr2=0x7fc388197fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:58:56.213 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.212+0000 7fc38cef6640 1 --2- 172.21.15.29:0/2366729666 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc3881017a0 0x7fc388197fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:58:56.214 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.212+0000 7fc38cef6640 1 -- 172.21.15.29:0/2366729666 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc388102160 msgr2=0x7fc3881984e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:58:56.214 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.212+0000 7fc38cef6640 1 --2- 172.21.15.29:0/2366729666 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc388102160 0x7fc3881984e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:58:56.214 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.212+0000 7fc38cef6640 1 -- 172.21.15.29:0/2366729666 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc38819cf90 con 0x7fc3881077b0 2024-03-16T11:58:56.214 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.212+0000 7fc387fff640 1 --2- 172.21.15.29:0/2366729666 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc3881017a0 0x7fc388197fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:58:56.214 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.212+0000 7fc3877fe640 1 --2- 172.21.15.29:0/2366729666 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc388102160 0x7fc3881984e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:58:56.214 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.212+0000 7fc38cef6640 1 --2- 172.21.15.29:0/2366729666 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc3881077b0 0x7fc38819c860 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7fc3800348f0 tx=0x7fc380032c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:58:56.214 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.212+0000 7fc3857fa640 1 -- 172.21.15.29:0/2366729666 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc380041020 con 0x7fc3881077b0 2024-03-16T11:58:56.214 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.212+0000 7fc3857fa640 1 -- 172.21.15.29:0/2366729666 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc380032e80 con 0x7fc3881077b0 2024-03-16T11:58:56.214 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.212+0000 7fc38e959640 1 -- 172.21.15.29:0/2366729666 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc38819d220 con 0x7fc3881077b0 2024-03-16T11:58:56.214 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.212+0000 7fc3857fa640 1 -- 172.21.15.29:0/2366729666 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc380039040 con 0x7fc3881077b0 2024-03-16T11:58:56.214 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.212+0000 7fc38e959640 1 -- 172.21.15.29:0/2366729666 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc3881a4b10 con 0x7fc3881077b0 2024-03-16T11:58:56.220 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.215+0000 7fc3857fa640 1 -- 172.21.15.29:0/2366729666 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7fc380044020 con 0x7fc3881077b0 2024-03-16T11:58:56.220 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.215+0000 7fc391611740 1 Processor -- start 2024-03-16T11:58:56.220 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.215+0000 7fc391611740 1 -- start start 2024-03-16T11:58:56.220 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.215+0000 7fc391611740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55bdb3e25130 0x55bdb3e25510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:58:56.220 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.215+0000 7fc391611740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bdb3e25a50 0x55bdb3e2e2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:58:56.220 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.215+0000 7fc391611740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bdb3e2e820 0x55bdb3e30c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:58:56.220 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.215+0000 7fc391611740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55bdb3d89a40 con 0x55bdb3e2e820 2024-03-16T11:58:56.221 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.215+0000 7fc391611740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x55bdb3d32920 con 0x55bdb3e25130 2024-03-16T11:58:56.221 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.215+0000 7fc391611740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55bdb3d85ae0 con 0x55bdb3e25a50 2024-03-16T11:58:56.221 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.215+0000 7fc3857fa640 1 --2- 172.21.15.29:0/2366729666 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7fc3700a2a10 0x7fc3700a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:58:56.221 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.215+0000 7fc3877fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bdb3e2e820 0x55bdb3e30c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:58:56.221 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.215+0000 7fc387fff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bdb3e25a50 0x55bdb3e2e2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:58:56.221 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.215+0000 7fc38cef6640 1 --2- 172.21.15.29:0/2366729666 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7fc3700a2a10 0x7fc3700a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:58:56.221 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.215+0000 7fc387fff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bdb3e25a50 0x55bdb3e2e2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.29:39014/0 (socket says 172.21.15.29:39014) 2024-03-16T11:58:56.221 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.215+0000 7fc387fff640 1 -- 172.21.15.29:0/2551872906 learned_addr learned my addr 172.21.15.29:0/2551872906 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:58:56.221 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.215+0000 7fc3877fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bdb3e2e820 0x55bdb3e30c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60740/0 (socket says 172.21.15.29:60740) 2024-03-16T11:58:56.221 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.216+0000 7fc3877fe640 1 --2- 172.21.15.29:0/2551872906 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55bdb3e25130 0x55bdb3e25510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:58:56.221 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.216+0000 7fc3857fa640 1 -- 172.21.15.29:0/2366729666 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8556+0+0 (secure 0 0 0) 0x7fc3800f0910 con 0x7fc3881077b0 2024-03-16T11:58:56.221 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.216+0000 7fc387fff640 1 -- 172.21.15.29:0/2551872906 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55bdb3e25130 msgr2=0x55bdb3e25510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:58:56.221 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.216+0000 7fc387fff640 1 --2- 172.21.15.29:0/2551872906 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55bdb3e25130 0x55bdb3e25510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:58:56.221 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.216+0000 7fc387fff640 1 -- 172.21.15.29:0/2551872906 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bdb3e2e820 msgr2=0x55bdb3e30c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:58:56.221 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.216+0000 7fc387fff640 1 --2- 172.21.15.29:0/2551872906 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bdb3e2e820 0x55bdb3e30c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:58:56.222 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.216+0000 7fc387fff640 1 -- 172.21.15.29:0/2551872906 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bdb3d4a460 con 0x55bdb3e25a50 2024-03-16T11:58:56.222 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.216+0000 7fc3877fe640 1 --2- 172.21.15.29:0/2551872906 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55bdb3e25130 0x55bdb3e25510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T11:58:56.222 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.216+0000 7fc387fff640 1 --2- 172.21.15.29:0/2551872906 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bdb3e25a50 0x55bdb3e2e2e0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fc37c0114f0 tx=0x7fc37c0119b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5ebbc27376c8b7a8 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:58:56.222 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.216+0000 7fc38cef6640 1 --2- 172.21.15.29:0/2366729666 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7fc3700a2a10 0x7fc3700a4ed0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fc3800ee200 tx=0x7fc3800dd040 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:58:56.222 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.216+0000 7fc36e7fc640 1 -- 172.21.15.29:0/2551872906 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc37c01e070 con 0x55bdb3e25a50 2024-03-16T11:58:56.222 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.216+0000 7fc36e7fc640 1 -- 172.21.15.29:0/2551872906 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc37c00faa0 con 0x55bdb3e25a50 2024-03-16T11:58:56.222 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.216+0000 7fc36e7fc640 1 -- 172.21.15.29:0/2551872906 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc37c019b50 con 0x55bdb3e25a50 2024-03-16T11:58:56.222 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.217+0000 7fc391611740 1 -- 172.21.15.29:0/2551872906 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bdb3e25a50 msgr2=0x55bdb3e2e2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:58:56.222 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.217+0000 7fc391611740 1 --2- 172.21.15.29:0/2551872906 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bdb3e25a50 0x55bdb3e2e2e0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fc37c0114f0 tx=0x7fc37c0119b0 comp rx=0 tx=0).stop 2024-03-16T11:58:56.222 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.217+0000 7fc391611740 1 -- 172.21.15.29:0/2551872906 shutdown_connections 2024-03-16T11:58:56.222 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.217+0000 7fc391611740 1 --2- 172.21.15.29:0/2551872906 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bdb3e2e820 0x55bdb3e30c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:58:56.222 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.217+0000 7fc391611740 1 --2- 172.21.15.29:0/2551872906 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bdb3e25a50 0x55bdb3e2e2e0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:58:56.222 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.217+0000 7fc391611740 1 --2- 172.21.15.29:0/2551872906 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55bdb3e25130 0x55bdb3e25510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:58:56.223 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.217+0000 7fc391611740 1 -- 172.21.15.29:0/2551872906 >> 172.21.15.29:0/2551872906 conn(0x55bdb3e1fa50 msgr2=0x55bdb3e1fe30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:58:56.223 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.217+0000 7fc391611740 1 -- 172.21.15.29:0/2551872906 shutdown_connections 2024-03-16T11:58:56.223 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.217+0000 7fc391611740 1 -- 172.21.15.29:0/2551872906 wait complete. 2024-03-16T11:58:56.223 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.217+0000 7fc391611740 1 -- start start 2024-03-16T11:58:56.223 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.217+0000 7fc391611740 1 Processor -- start 2024-03-16T11:58:56.223 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.218+0000 7fc391611740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bdb3e25130 0x55bdb3e2b870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:58:56.223 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.218+0000 7fc391611740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bdb3e25a50 0x55bdb3e2bdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:58:56.223 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.218+0000 7fc391611740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55bdb3e2e820 0x55bdb3e2c2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:58:56.223 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.218+0000 7fc391611740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55bdb3e31450 con 0x55bdb3e25a50 2024-03-16T11:58:56.223 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.218+0000 7fc391611740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x55bdb3d45d70 con 0x55bdb3e2e820 2024-03-16T11:58:56.223 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.218+0000 7fc391611740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55bdb3e315d0 con 0x55bdb3e25130 2024-03-16T11:58:56.223 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.218+0000 7fc391611740 10 client.0 _ll_register_callbacks cb 0x55bdb3a9da00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T11:58:56.223 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.218+0000 7fc387fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bdb3e25a50 0x55bdb3e2bdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:58:56.223 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.218+0000 7fc387fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bdb3e25a50 0x55bdb3e2bdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60744/0 (socket says 172.21.15.29:60744) 2024-03-16T11:58:56.224 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.218+0000 7fc387fff640 1 -- 172.21.15.29:0/106662272 learned_addr learned my addr 172.21.15.29:0/106662272 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:58:56.224 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.218+0000 7fc3877fe640 1 --2- 172.21.15.29:0/106662272 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bdb3e25130 0x55bdb3e2b870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:58:56.224 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.218+0000 7fc3877fe640 1 --2- 172.21.15.29:0/106662272 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55bdb3e2e820 0x55bdb3e2c2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:58:56.224 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.218+0000 7fc387fff640 1 -- 172.21.15.29:0/106662272 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bdb3e25130 msgr2=0x55bdb3e2b870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:58:56.224 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.218+0000 7fc387fff640 1 --2- 172.21.15.29:0/106662272 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bdb3e25130 0x55bdb3e2b870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:58:56.224 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.218+0000 7fc387fff640 1 -- 172.21.15.29:0/106662272 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55bdb3e2e820 msgr2=0x55bdb3e2c2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:58:56.224 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.218+0000 7fc387fff640 1 --2- 172.21.15.29:0/106662272 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55bdb3e2e820 0x55bdb3e2c2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:58:56.224 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.218+0000 7fc387fff640 1 -- 172.21.15.29:0/106662272 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bdb3a9e210 con 0x55bdb3e25a50 2024-03-16T11:58:56.224 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.218+0000 7fc387fff640 1 --2- 172.21.15.29:0/106662272 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bdb3e25a50 0x55bdb3e2bdb0 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7fc37c01c040 tx=0x7fc37c005710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=848c3c0713434287 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:58:56.224 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.218+0000 7fc3677fe640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-16T11:58:56.224 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.218+0000 7fc3677fe640 1 -- 172.21.15.29:0/106662272 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc37c026020 con 0x55bdb3e25a50 2024-03-16T11:58:56.224 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.218+0000 7fc391611740 1 -- 172.21.15.29:0/106662272 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc37c00faa0 con 0x55bdb3e25a50 2024-03-16T11:58:56.224 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.218+0000 7fc3677fe640 1 -- 172.21.15.29:0/106662272 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8556+0+0 (secure 0 0 0) 0x7fc37c00f640 con 0x55bdb3e25a50 2024-03-16T11:58:56.224 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.219+0000 7fc3677fe640 1 -- 172.21.15.29:0/106662272 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fc37c01e040 con 0x55bdb3e25a50 2024-03-16T11:58:56.224 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.219+0000 7fc391611740 10 client.16068 fetch_fsmap learned FSMap version 26 2024-03-16T11:58:56.225 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.219+0000 7fc391611740 1 -- 172.21.15.29:0/106662272 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55bdb3b6a620 con 0x55bdb3e25a50 2024-03-16T11:58:56.225 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.219+0000 7fc3677fe640 1 -- 172.21.15.29:0/106662272 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc37c00faa0 con 0x55bdb3e25a50 2024-03-16T11:58:56.225 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.219+0000 7fc3677fe640 1 -- 172.21.15.29:0/106662272 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7fc37c021070 con 0x55bdb3e25a50 2024-03-16T11:58:56.225 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.219+0000 7fc391611740 10 client.16068 fetch_fsmap finished waiting for FSMap version 26 2024-03-16T11:58:56.225 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.219+0000 7fc391611740 10 client.16068 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T11:58:56.225 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.219+0000 7fc391611740 20 client.16068 populate_metadata read hostname 'smithi029' 2024-03-16T11:58:56.225 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.219+0000 7fc391611740 1 --2- 172.21.15.29:0/106662272 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55bdb3e3e950 0x55bdb3e40d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:58:56.225 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.219+0000 7fc391611740 4 client.16068 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T11:58:56.225 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.219+0000 7fc391611740 1 -- 172.21.15.29:0/106662272 --> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bdb3d96f80 con 0x55bdb3e3e950 2024-03-16T11:58:56.225 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.219+0000 7fc3877fe640 1 --2- 172.21.15.29:0/106662272 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55bdb3e3e950 0x55bdb3e40d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:58:56.225 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.220+0000 7fc3677fe640 10 client.16068 ms_handle_connect on v2:172.21.15.152:6838/1548731833 2024-03-16T11:58:56.225 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.220+0000 7fc3877fe640 1 --2- 172.21.15.29:0/106662272 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55bdb3e3e950 0x55bdb3e40d40 crc :-1 s=READY pgs=110 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=194e85c9ff296600 server_cookie=5f0a1b75a4faba4 in_seq=0 out_seq=0 2024-03-16T11:58:56.226 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.222+0000 7fc3677fe640 1 -- 172.21.15.29:0/106662272 <== mds.0 v2:172.21.15.152:6838/1548731833 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55bdb3d96f80 con 0x55bdb3e3e950 2024-03-16T11:58:56.226 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.222+0000 7fc3677fe640 10 client.16068 handle_command_reply: tid=0 2024-03-16T11:58:56.227 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.222+0000 7fc391611740 1 client.16068 shutdown 2024-03-16T11:58:56.227 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.222+0000 7fc391611740 1 -- 172.21.15.29:0/106662272 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bdb3e25a50 msgr2=0x55bdb3e2bdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:58:56.227 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.222+0000 7fc391611740 1 --2- 172.21.15.29:0/106662272 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bdb3e25a50 0x55bdb3e2bdb0 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7fc37c01c040 tx=0x7fc37c005710 comp rx=0 tx=0).stop 2024-03-16T11:58:56.227 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.222+0000 7fc391611740 1 -- 172.21.15.29:0/106662272 shutdown_connections 2024-03-16T11:58:56.227 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.222+0000 7fc391611740 1 --2- 172.21.15.29:0/106662272 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55bdb3e3e950 0x55bdb3e40d40 crc :-1 s=READY pgs=110 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:58:56.227 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.222+0000 7fc391611740 1 --2- 172.21.15.29:0/106662272 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55bdb3e2e820 0x55bdb3e2c2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:58:56.227 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.222+0000 7fc3677fe640 0 client.16068 ms_handle_reset on v2:172.21.15.152:6838/1548731833 2024-03-16T11:58:56.227 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.222+0000 7fc391611740 1 --2- 172.21.15.29:0/106662272 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bdb3e25a50 0x55bdb3e2bdb0 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:58:56.227 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.222+0000 7fc391611740 1 --2- 172.21.15.29:0/106662272 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bdb3e25130 0x55bdb3e2b870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:58:56.227 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.222+0000 7fc391611740 1 -- 172.21.15.29:0/106662272 >> 172.21.15.29:0/106662272 conn(0x55bdb3e1fa50 msgr2=0x55bdb3e217a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:58:56.227 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.222+0000 7fc391611740 1 -- 172.21.15.29:0/106662272 shutdown_connections 2024-03-16T11:58:56.227 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.222+0000 7fc391611740 1 -- 172.21.15.29:0/106662272 wait complete. 2024-03-16T11:58:56.227 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.223+0000 7fc391611740 20 client.16068 trim_cache size 0 max 16384 2024-03-16T11:58:56.241 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.237+0000 7fc391611740 1 Processor -- start 2024-03-16T11:58:56.241 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.237+0000 7fc391611740 1 -- start start 2024-03-16T11:58:56.241 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.237+0000 7fc391611740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55bdb3e25130 0x55bdb3e2dfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:58:56.241 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.237+0000 7fc391611740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bdb3e25a50 0x55bdb3e3a1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:58:56.241 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.237+0000 7fc391611740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bdb3e2e820 0x55bdb3e464f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:58:56.241 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.238+0000 7fc391611740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fc3881afe00 con 0x55bdb3e25a50 2024-03-16T11:58:56.241 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.238+0000 7fc391611740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fc3881ad6d0 con 0x55bdb3e25130 2024-03-16T11:58:56.241 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.238+0000 7fc391611740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fc3881ab540 con 0x55bdb3e2e820 2024-03-16T11:58:56.241 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.238+0000 7fc387fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bdb3e25a50 0x55bdb3e3a1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:58:56.241 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.238+0000 7fc387fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bdb3e25a50 0x55bdb3e3a1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60746/0 (socket says 172.21.15.29:60746) 2024-03-16T11:58:56.241 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.238+0000 7fc387fff640 1 -- 172.21.15.29:0/1240698122 learned_addr learned my addr 172.21.15.29:0/1240698122 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:58:56.241 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.238+0000 7fc3877fe640 1 --2- 172.21.15.29:0/1240698122 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bdb3e2e820 0x55bdb3e464f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:58:56.241 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.238+0000 7fc3877fe640 1 --2- 172.21.15.29:0/1240698122 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55bdb3e25130 0x55bdb3e2dfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:58:56.242 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.238+0000 7fc387fff640 1 -- 172.21.15.29:0/1240698122 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bdb3e2e820 msgr2=0x55bdb3e464f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:58:56.242 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.238+0000 7fc387fff640 1 --2- 172.21.15.29:0/1240698122 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bdb3e2e820 0x55bdb3e464f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:58:56.242 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.238+0000 7fc387fff640 1 -- 172.21.15.29:0/1240698122 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55bdb3e25130 msgr2=0x55bdb3e2dfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:58:56.242 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.238+0000 7fc387fff640 1 --2- 172.21.15.29:0/1240698122 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55bdb3e25130 0x55bdb3e2dfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:58:56.242 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.238+0000 7fc387fff640 1 -- 172.21.15.29:0/1240698122 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bdb3e19fc0 con 0x55bdb3e25a50 2024-03-16T11:58:56.242 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.238+0000 7fc3877fe640 1 --2- 172.21.15.29:0/1240698122 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bdb3e2e820 0x55bdb3e464f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T11:58:56.242 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.238+0000 7fc387fff640 1 --2- 172.21.15.29:0/1240698122 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bdb3e25a50 0x55bdb3e3a1f0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7fc37c02c040 tx=0x7fc37c005780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=479fd67c7ef7058e server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:58:56.242 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.238+0000 7fc36dffb640 1 -- 172.21.15.29:0/1240698122 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc37c015020 con 0x55bdb3e25a50 2024-03-16T11:58:56.242 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.238+0000 7fc36dffb640 1 -- 172.21.15.29:0/1240698122 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc37c019c00 con 0x55bdb3e25a50 2024-03-16T11:58:56.242 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.238+0000 7fc36dffb640 1 -- 172.21.15.29:0/1240698122 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc37c026070 con 0x55bdb3e25a50 2024-03-16T11:58:56.242 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.239+0000 7fc391611740 1 -- 172.21.15.29:0/1240698122 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bdb3e25a50 msgr2=0x55bdb3e3a1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:58:56.242 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.239+0000 7fc391611740 1 --2- 172.21.15.29:0/1240698122 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bdb3e25a50 0x55bdb3e3a1f0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7fc37c02c040 tx=0x7fc37c005780 comp rx=0 tx=0).stop 2024-03-16T11:58:56.242 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.239+0000 7fc391611740 1 -- 172.21.15.29:0/1240698122 shutdown_connections 2024-03-16T11:58:56.242 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.239+0000 7fc391611740 1 --2- 172.21.15.29:0/1240698122 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bdb3e2e820 0x55bdb3e464f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:58:56.243 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.239+0000 7fc391611740 1 --2- 172.21.15.29:0/1240698122 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bdb3e25a50 0x55bdb3e3a1f0 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:58:56.243 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.239+0000 7fc391611740 1 --2- 172.21.15.29:0/1240698122 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55bdb3e25130 0x55bdb3e2dfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:58:56.243 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.239+0000 7fc391611740 1 -- 172.21.15.29:0/1240698122 >> 172.21.15.29:0/1240698122 conn(0x55bdb3e1fa50 msgr2=0x55bdb3e2b830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:58:56.243 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.239+0000 7fc391611740 1 -- 172.21.15.29:0/1240698122 shutdown_connections 2024-03-16T11:58:56.243 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.239+0000 7fc391611740 1 -- 172.21.15.29:0/1240698122 wait complete. 2024-03-16T11:58:56.243 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.239+0000 7fc391611740 1 -- start start 2024-03-16T11:58:56.243 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.239+0000 7fc391611740 1 Processor -- start 2024-03-16T11:58:56.243 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.240+0000 7fc391611740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bdb3e25130 0x55bdb3e28620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:58:56.243 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.240+0000 7fc391611740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55bdb3e25a50 0x55bdb3e28b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:58:56.243 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.240+0000 7fc391611740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bdb3e2e820 0x55bdb3e290a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:58:56.243 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.240+0000 7fc391611740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fc3881abb90 con 0x55bdb3e25130 2024-03-16T11:58:56.243 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.240+0000 7fc391611740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fc3881ad000 con 0x55bdb3e25a50 2024-03-16T11:58:56.243 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.240+0000 7fc391611740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fc3881ac260 con 0x55bdb3e2e820 2024-03-16T11:58:56.243 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.240+0000 7fc391611740 10 client.0 _ll_register_callbacks cb 0x7fc37c01c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T11:58:56.243 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.240+0000 7fc3877fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bdb3e25130 0x55bdb3e28620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:58:56.244 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.240+0000 7fc3877fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bdb3e25130 0x55bdb3e28620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60748/0 (socket says 172.21.15.29:60748) 2024-03-16T11:58:56.244 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.240+0000 7fc3877fe640 1 -- 172.21.15.29:0/950381852 learned_addr learned my addr 172.21.15.29:0/950381852 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:58:56.244 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.240+0000 7fc387fff640 1 --2- 172.21.15.29:0/950381852 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55bdb3e25a50 0x55bdb3e28b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:58:56.244 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.240+0000 7fc3877fe640 1 --2- 172.21.15.29:0/950381852 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bdb3e2e820 0x55bdb3e290a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:58:56.244 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.240+0000 7fc3877fe640 1 -- 172.21.15.29:0/950381852 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bdb3e2e820 msgr2=0x55bdb3e290a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:58:56.244 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.240+0000 7fc3877fe640 1 --2- 172.21.15.29:0/950381852 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bdb3e2e820 0x55bdb3e290a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:58:56.244 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.240+0000 7fc3877fe640 1 -- 172.21.15.29:0/950381852 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55bdb3e25a50 msgr2=0x55bdb3e28b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:58:56.244 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.240+0000 7fc3877fe640 1 --2- 172.21.15.29:0/950381852 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55bdb3e25a50 0x55bdb3e28b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:58:56.244 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.240+0000 7fc3877fe640 1 -- 172.21.15.29:0/950381852 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bdb3ea3d20 con 0x55bdb3e25130 2024-03-16T11:58:56.244 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.240+0000 7fc3877fe640 1 --2- 172.21.15.29:0/950381852 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bdb3e25130 0x55bdb3e28620 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7fc378000c00 tx=0x7fc378013040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a18587c885178c54 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:58:56.244 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.240+0000 7fc3677fe640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-16T11:58:56.244 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.240+0000 7fc3677fe640 1 -- 172.21.15.29:0/950381852 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc37802c020 con 0x55bdb3e25130 2024-03-16T11:58:56.244 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.240+0000 7fc3677fe640 1 -- 172.21.15.29:0/950381852 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8556+0+0 (secure 0 0 0) 0x7fc378025690 con 0x55bdb3e25130 2024-03-16T11:58:56.244 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.240+0000 7fc391611740 1 -- 172.21.15.29:0/950381852 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc37c019c00 con 0x55bdb3e25130 2024-03-16T11:58:56.244 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.240+0000 7fc3677fe640 1 -- 172.21.15.29:0/950381852 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fc378005020 con 0x55bdb3e25130 2024-03-16T11:58:56.245 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.240+0000 7fc391611740 10 client.16074 fetch_fsmap learned FSMap version 26 2024-03-16T11:58:56.245 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.240+0000 7fc391611740 1 -- 172.21.15.29:0/950381852 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55bdb3ea5070 con 0x55bdb3e25130 2024-03-16T11:58:56.245 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.241+0000 7fc3677fe640 1 -- 172.21.15.29:0/950381852 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc37c019c00 con 0x55bdb3e25130 2024-03-16T11:58:56.245 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.241+0000 7fc3677fe640 1 -- 172.21.15.29:0/950381852 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7fc3780035f0 con 0x55bdb3e25130 2024-03-16T11:58:56.245 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.241+0000 7fc391611740 10 client.16074 fetch_fsmap finished waiting for FSMap version 26 2024-03-16T11:58:56.245 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.241+0000 7fc391611740 10 client.16074 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T11:58:56.245 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.241+0000 7fc391611740 20 client.16074 populate_metadata read hostname 'smithi029' 2024-03-16T11:58:56.245 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.241+0000 7fc391611740 1 --2- 172.21.15.29:0/950381852 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55bdb3ea5820 0x55bdb3ea7c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:58:56.245 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.241+0000 7fc391611740 4 client.16074 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-16T11:58:56.245 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.241+0000 7fc391611740 1 -- 172.21.15.29:0/950381852 --> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55bdb3d912d0 con 0x55bdb3ea5820 2024-03-16T11:58:56.245 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.241+0000 7fc387fff640 1 --2- 172.21.15.29:0/950381852 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55bdb3ea5820 0x55bdb3ea7c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:58:56.245 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.241+0000 7fc3677fe640 10 client.16074 ms_handle_connect on v2:172.21.15.152:6838/1548731833 2024-03-16T11:58:56.245 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.241+0000 7fc387fff640 1 --2- 172.21.15.29:0/950381852 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55bdb3ea5820 0x55bdb3ea7c40 crc :-1 s=READY pgs=111 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fe2d18a20ca81d55 server_cookie=2449dde0c9e4acea in_seq=0 out_seq=0 2024-03-16T11:58:56.407 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.407+0000 7fc3677fe640 1 -- 172.21.15.29:0/950381852 <== mds.0 v2:172.21.15.152:6838/1548731833 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55bdb3d912d0 con 0x55bdb3ea5820 2024-03-16T11:58:56.408 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.408+0000 7fc3677fe640 10 client.16074 handle_command_reply: tid=0 2024-03-16T11:58:56.408 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.408+0000 7fc391611740 1 client.16074 shutdown 2024-03-16T11:58:56.408 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.408+0000 7fc391611740 1 -- 172.21.15.29:0/950381852 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bdb3e25130 msgr2=0x55bdb3e28620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:58:56.408 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.408+0000 7fc391611740 1 --2- 172.21.15.29:0/950381852 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bdb3e25130 0x55bdb3e28620 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7fc378000c00 tx=0x7fc378013040 comp rx=0 tx=0).stop 2024-03-16T11:58:56.408 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.408+0000 7fc391611740 1 -- 172.21.15.29:0/950381852 shutdown_connections 2024-03-16T11:58:56.408 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.408+0000 7fc391611740 1 --2- 172.21.15.29:0/950381852 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bdb3e2e820 0x55bdb3e290a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:58:56.408 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.408+0000 7fc391611740 1 --2- 172.21.15.29:0/950381852 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55bdb3ea5820 0x55bdb3ea7c40 crc :-1 s=READY pgs=111 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:58:56.408 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.408+0000 7fc3677fe640 0 client.16074 ms_handle_reset on v2:172.21.15.152:6838/1548731833 2024-03-16T11:58:56.408 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.408+0000 7fc391611740 1 --2- 172.21.15.29:0/950381852 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55bdb3e25a50 0x55bdb3e28b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:58:56.408 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.408+0000 7fc391611740 1 --2- 172.21.15.29:0/950381852 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bdb3e25130 0x55bdb3e28620 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:58:56.408 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.408+0000 7fc391611740 1 -- 172.21.15.29:0/950381852 >> 172.21.15.29:0/950381852 conn(0x55bdb3e1fa50 msgr2=0x55bdb3e1ebb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:58:56.408 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.408+0000 7fc391611740 1 -- 172.21.15.29:0/950381852 shutdown_connections 2024-03-16T11:58:56.408 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.408+0000 7fc391611740 1 -- 172.21.15.29:0/950381852 wait complete. 2024-03-16T11:58:56.409 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.408+0000 7fc391611740 20 client.16074 trim_cache size 0 max 16384 2024-03-16T11:58:56.409 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:{ 2024-03-16T11:58:56.409 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "message": "", 2024-03-16T11:58:56.409 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "return_code": 0 2024-03-16T11:58:56.409 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:} 2024-03-16T11:58:56.410 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.409+0000 7fc38e959640 1 -- 172.21.15.29:0/2366729666 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7fc3700a2a10 msgr2=0x7fc3700a4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:58:56.411 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.409+0000 7fc38e959640 1 --2- 172.21.15.29:0/2366729666 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7fc3700a2a10 0x7fc3700a4ed0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fc3800ee200 tx=0x7fc3800dd040 comp rx=0 tx=0).stop 2024-03-16T11:58:56.411 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.409+0000 7fc38e959640 1 -- 172.21.15.29:0/2366729666 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc3881077b0 msgr2=0x7fc38819c860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:58:56.411 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.409+0000 7fc38e959640 1 --2- 172.21.15.29:0/2366729666 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc3881077b0 0x7fc38819c860 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7fc3800348f0 tx=0x7fc380032c90 comp rx=0 tx=0).stop 2024-03-16T11:58:56.411 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.409+0000 7fc38e959640 1 -- 172.21.15.29:0/2366729666 shutdown_connections 2024-03-16T11:58:56.411 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.410+0000 7fc38e959640 1 --2- 172.21.15.29:0/2366729666 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7fc3700a2a10 0x7fc3700a4ed0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:58:56.411 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.410+0000 7fc38e959640 1 --2- 172.21.15.29:0/2366729666 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc3881077b0 0x7fc38819c860 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:58:56.411 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.410+0000 7fc38e959640 1 --2- 172.21.15.29:0/2366729666 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc388102160 0x7fc3881984e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:58:56.411 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.410+0000 7fc38e959640 1 --2- 172.21.15.29:0/2366729666 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc3881017a0 0x7fc388197fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:58:56.411 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.410+0000 7fc38e959640 1 -- 172.21.15.29:0/2366729666 >> 172.21.15.29:0/2366729666 conn(0x7fc3880fd290 msgr2=0x7fc3880fd7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:58:56.411 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.410+0000 7fc38e959640 1 -- 172.21.15.29:0/2366729666 shutdown_connections 2024-03-16T11:58:56.411 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:58:56.410+0000 7fc38e959640 1 -- 172.21.15.29:0/2366729666 wait complete. 2024-03-16T11:58:56.794 INFO:tasks.workunit.client.0.smithi029.stdout: 1 813629 10.46 MB/sec execute 281 sec latency 67.092 ms 2024-03-16T11:58:56.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:58:56 smithi154 ceph-mon[36320]: pgmap v250: 97 pgs: 97 active+clean; 76 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 2.7 MiB/s wr, 259 op/s 2024-03-16T11:58:56.837 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:58:56 smithi029 ceph-mon[31975]: pgmap v250: 97 pgs: 97 active+clean; 76 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 2.7 MiB/s wr, 259 op/s 2024-03-16T11:58:56.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:58:56 smithi152 ceph-mon[36291]: pgmap v250: 97 pgs: 97 active+clean; 76 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 2.7 MiB/s wr, 259 op/s 2024-03-16T11:58:57.794 INFO:tasks.workunit.client.0.smithi029.stdout: 1 815542 10.45 MB/sec execute 282 sec latency 26.362 ms 2024-03-16T11:58:58.794 INFO:tasks.workunit.client.0.smithi029.stdout: 1 817304 10.46 MB/sec execute 283 sec latency 34.738 ms 2024-03-16T11:58:58.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:58:58 smithi154 ceph-mon[36320]: pgmap v251: 97 pgs: 97 active+clean; 82 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 4.0 MiB/s wr, 348 op/s 2024-03-16T11:58:58.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:58:58 smithi029 ceph-mon[31975]: pgmap v251: 97 pgs: 97 active+clean; 82 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 4.0 MiB/s wr, 348 op/s 2024-03-16T11:58:58.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:58:58 smithi152 ceph-mon[36291]: pgmap v251: 97 pgs: 97 active+clean; 82 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 4.0 MiB/s wr, 348 op/s 2024-03-16T11:58:59.794 INFO:tasks.workunit.client.0.smithi029.stdout: 1 819388 10.46 MB/sec execute 284 sec latency 17.717 ms 2024-03-16T11:59:00.794 INFO:tasks.workunit.client.0.smithi029.stdout: 1 820800 10.45 MB/sec execute 285 sec latency 74.604 ms 2024-03-16T11:59:00.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:59:00 smithi154 ceph-mon[36320]: pgmap v252: 97 pgs: 97 active+clean; 82 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 2.8 MiB/s wr, 177 op/s 2024-03-16T11:59:00.837 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:59:00 smithi029 ceph-mon[31975]: pgmap v252: 97 pgs: 97 active+clean; 82 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 2.8 MiB/s wr, 177 op/s 2024-03-16T11:59:00.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:59:00 smithi152 ceph-mon[36291]: pgmap v252: 97 pgs: 97 active+clean; 82 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 2.8 MiB/s wr, 177 op/s 2024-03-16T11:59:01.794 INFO:tasks.workunit.client.0.smithi029.stdout: 1 821722 10.43 MB/sec execute 286 sec latency 48.106 ms 2024-03-16T11:59:02.794 INFO:tasks.workunit.client.0.smithi029.stdout: 1 823831 10.44 MB/sec execute 287 sec latency 47.248 ms 2024-03-16T11:59:02.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:59:02 smithi154 ceph-mon[36320]: pgmap v253: 97 pgs: 97 active+clean; 73 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 4.1 MiB/s wr, 341 op/s 2024-03-16T11:59:02.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:59:02 smithi029 ceph-mon[31975]: pgmap v253: 97 pgs: 97 active+clean; 73 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 4.1 MiB/s wr, 341 op/s 2024-03-16T11:59:02.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:59:02 smithi152 ceph-mon[36291]: pgmap v253: 97 pgs: 97 active+clean; 73 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 4.1 MiB/s wr, 341 op/s 2024-03-16T11:59:03.794 INFO:tasks.workunit.client.0.smithi029.stdout: 1 825214 10.42 MB/sec execute 288 sec latency 20.634 ms 2024-03-16T11:59:04.794 INFO:tasks.workunit.client.0.smithi029.stdout: 1 827228 10.43 MB/sec execute 289 sec latency 36.002 ms 2024-03-16T11:59:04.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:59:04 smithi154 ceph-mon[36320]: pgmap v254: 97 pgs: 97 active+clean; 73 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 2.6 MiB/s wr, 253 op/s 2024-03-16T11:59:04.837 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:59:04 smithi029 ceph-mon[31975]: pgmap v254: 97 pgs: 97 active+clean; 73 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 2.6 MiB/s wr, 253 op/s 2024-03-16T11:59:04.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:59:04 smithi152 ceph-mon[36291]: pgmap v254: 97 pgs: 97 active+clean; 73 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 2.6 MiB/s wr, 253 op/s 2024-03-16T11:59:05.794 INFO:tasks.workunit.client.0.smithi029.stdout: 1 827571 10.40 MB/sec execute 290 sec latency 116.048 ms 2024-03-16T11:59:05.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:59:05 smithi154 ceph-mon[36320]: osdmap e96: 12 total, 12 up, 12 in 2024-03-16T11:59:05.837 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:59:05 smithi029 ceph-mon[31975]: osdmap e96: 12 total, 12 up, 12 in 2024-03-16T11:59:05.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:59:05 smithi152 ceph-mon[36291]: osdmap e96: 12 total, 12 up, 12 in 2024-03-16T11:59:06.794 INFO:tasks.workunit.client.0.smithi029.stdout: 1 828151 10.37 MB/sec execute 291 sec latency 45.839 ms 2024-03-16T11:59:06.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:59:06 smithi154 ceph-mon[36320]: pgmap v256: 97 pgs: 97 active+clean; 73 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 3.2 MiB/s wr, 303 op/s 2024-03-16T11:59:06.837 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:59:06 smithi029 ceph-mon[31975]: pgmap v256: 97 pgs: 97 active+clean; 73 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 3.2 MiB/s wr, 303 op/s 2024-03-16T11:59:06.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:59:06 smithi152 ceph-mon[36291]: pgmap v256: 97 pgs: 97 active+clean; 73 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 3.2 MiB/s wr, 303 op/s 2024-03-16T11:59:07.794 INFO:tasks.workunit.client.0.smithi029.stdout: 1 830106 10.37 MB/sec execute 292 sec latency 20.760 ms 2024-03-16T11:59:08.794 INFO:tasks.workunit.client.0.smithi029.stdout: 1 831556 10.36 MB/sec execute 293 sec latency 25.376 ms 2024-03-16T11:59:08.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:59:08 smithi154 ceph-mon[36320]: pgmap v257: 97 pgs: 18 active+clean+snaptrim, 38 active+clean+snaptrim_wait, 41 active+clean; 82 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.1 MiB/s wr, 315 op/s 2024-03-16T11:59:08.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:59:08 smithi154 ceph-mon[36320]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T11:59:08.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:59:08 smithi154 ceph-mon[36320]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:59:08.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:59:08 smithi154 ceph-mon[36320]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T11:59:08.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:59:08 smithi154 ceph-mon[36320]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' 2024-03-16T11:59:08.837 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:59:08 smithi029 ceph-mon[31975]: pgmap v257: 97 pgs: 18 active+clean+snaptrim, 38 active+clean+snaptrim_wait, 41 active+clean; 82 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.1 MiB/s wr, 315 op/s 2024-03-16T11:59:08.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:59:08 smithi029 ceph-mon[31975]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T11:59:08.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:59:08 smithi029 ceph-mon[31975]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:59:08.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:59:08 smithi029 ceph-mon[31975]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T11:59:08.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:59:08 smithi029 ceph-mon[31975]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' 2024-03-16T11:59:08.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:59:08 smithi152 ceph-mon[36291]: pgmap v257: 97 pgs: 18 active+clean+snaptrim, 38 active+clean+snaptrim_wait, 41 active+clean; 82 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.1 MiB/s wr, 315 op/s 2024-03-16T11:59:08.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:59:08 smithi152 ceph-mon[36291]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T11:59:08.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:59:08 smithi152 ceph-mon[36291]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T11:59:08.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:59:08 smithi152 ceph-mon[36291]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T11:59:08.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:59:08 smithi152 ceph-mon[36291]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' 2024-03-16T11:59:09.505 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.503+0000 7f929d427640 1 -- 172.21.15.29:0/2325205404 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f92981099c0 msgr2=0x7f9298109da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:59:09.505 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.503+0000 7f929d427640 1 --2- 172.21.15.29:0/2325205404 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f92981099c0 0x7f9298109da0 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f9290009930 tx=0x7f929002bc30 comp rx=0 tx=0).stop 2024-03-16T11:59:09.505 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.503+0000 7f929d427640 1 -- 172.21.15.29:0/2325205404 shutdown_connections 2024-03-16T11:59:09.505 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.503+0000 7f929d427640 1 --2- 172.21.15.29:0/2325205404 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9298104370 0x7f9298110880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:59:09.505 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.503+0000 7f929d427640 1 --2- 172.21.15.29:0/2325205404 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f92981039b0 0x7f9298103e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:59:09.505 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.503+0000 7f929d427640 1 --2- 172.21.15.29:0/2325205404 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f92981099c0 0x7f9298109da0 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:59:09.505 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.503+0000 7f929d427640 1 -- 172.21.15.29:0/2325205404 >> 172.21.15.29:0/2325205404 conn(0x7f92980ff4a0 msgr2=0x7f92981018c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:59:09.505 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.503+0000 7f929d427640 1 -- 172.21.15.29:0/2325205404 shutdown_connections 2024-03-16T11:59:09.505 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.503+0000 7f929d427640 1 -- 172.21.15.29:0/2325205404 wait complete. 2024-03-16T11:59:09.505 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.504+0000 7f929d427640 1 Processor -- start 2024-03-16T11:59:09.506 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.504+0000 7f929d427640 1 -- start start 2024-03-16T11:59:09.506 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.504+0000 7f929d427640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f92981039b0 0x7f929819a200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:59:09.506 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.504+0000 7f929d427640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9298104370 0x7f929819a740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:59:09.506 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.504+0000 7f929d427640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f92981099c0 0x7f929819eac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:59:09.506 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.504+0000 7f929d427640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f9298113da0 con 0x7f9298104370 2024-03-16T11:59:09.506 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.504+0000 7f929d427640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f9298113c20 con 0x7f92981039b0 2024-03-16T11:59:09.506 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.504+0000 7f929d427640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f9298113f20 con 0x7f92981099c0 2024-03-16T11:59:09.506 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.504+0000 7f92967fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9298104370 0x7f929819a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:59:09.506 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.505+0000 7f92977fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f92981099c0 0x7f929819eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:59:09.506 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.505+0000 7f9296ffd640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f92981039b0 0x7f929819a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:59:09.506 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.505+0000 7f92977fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f92981099c0 0x7f929819eac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.29:51434/0 (socket says 172.21.15.29:51434) 2024-03-16T11:59:09.506 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.505+0000 7f9296ffd640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f92981039b0 0x7f929819a200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.29:38128/0 (socket says 172.21.15.29:38128) 2024-03-16T11:59:09.506 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.505+0000 7f92977fe640 1 -- 172.21.15.29:0/4182958333 learned_addr learned my addr 172.21.15.29:0/4182958333 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:59:09.507 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.505+0000 7f92967fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9298104370 0x7f929819a740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:48614/0 (socket says 172.21.15.29:48614) 2024-03-16T11:59:09.507 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.505+0000 7f92977fe640 1 -- 172.21.15.29:0/4182958333 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f92981039b0 msgr2=0x7f929819a200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:59:09.507 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.505+0000 7f92977fe640 1 --2- 172.21.15.29:0/4182958333 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f92981039b0 0x7f929819a200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:59:09.507 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.505+0000 7f92977fe640 1 -- 172.21.15.29:0/4182958333 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9298104370 msgr2=0x7f929819a740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:59:09.507 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.505+0000 7f92977fe640 1 --2- 172.21.15.29:0/4182958333 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9298104370 0x7f929819a740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:59:09.507 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.505+0000 7f92977fe640 1 -- 172.21.15.29:0/4182958333 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f929819f1f0 con 0x7f92981099c0 2024-03-16T11:59:09.507 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.505+0000 7f92967fc640 1 --2- 172.21.15.29:0/4182958333 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9298104370 0x7f929819a740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:59:09.507 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.505+0000 7f92977fe640 1 --2- 172.21.15.29:0/4182958333 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f92981099c0 0x7f929819eac0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f928800eb10 tx=0x7f928800c2d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:59:09.507 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.505+0000 7f928ffff640 1 -- 172.21.15.29:0/4182958333 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9288004b20 con 0x7f92981099c0 2024-03-16T11:59:09.507 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.505+0000 7f929d427640 1 -- 172.21.15.29:0/4182958333 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f929819f4e0 con 0x7f92981099c0 2024-03-16T11:59:09.507 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.505+0000 7f928ffff640 1 -- 172.21.15.29:0/4182958333 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9288004cc0 con 0x7f92981099c0 2024-03-16T11:59:09.507 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.505+0000 7f928ffff640 1 -- 172.21.15.29:0/4182958333 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9288010760 con 0x7f92981099c0 2024-03-16T11:59:09.507 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.505+0000 7f929d427640 1 -- 172.21.15.29:0/4182958333 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f92981a6d70 con 0x7f92981099c0 2024-03-16T11:59:09.513 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.507+0000 7f928ffff640 1 -- 172.21.15.29:0/4182958333 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7f9288005020 con 0x7f92981099c0 2024-03-16T11:59:09.513 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.508+0000 7f92a0010740 1 Processor -- start 2024-03-16T11:59:09.513 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.508+0000 7f928ffff640 1 --2- 172.21.15.29:0/4182958333 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f92840a2a10 0x7f92840a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:59:09.513 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.508+0000 7f92a0010740 1 -- start start 2024-03-16T11:59:09.514 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.508+0000 7f92967fc640 1 --2- 172.21.15.29:0/4182958333 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f92840a2a10 0x7f92840a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:59:09.514 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.508+0000 7f928ffff640 1 -- 172.21.15.29:0/4182958333 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8600+0+0 (secure 0 0 0) 0x7f92880cf420 con 0x7f92981099c0 2024-03-16T11:59:09.514 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.508+0000 7f92a0010740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5602f0f29130 0x5602f0f29510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:59:09.514 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.508+0000 7f92a0010740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5602f0f29a50 0x5602f0f322e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:59:09.514 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.508+0000 7f92a0010740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5602f0f32820 0x5602f0f34c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:59:09.514 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.508+0000 7f92a0010740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x5602f0e8da40 con 0x5602f0f32820 2024-03-16T11:59:09.514 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.508+0000 7f92a0010740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x5602f0e36920 con 0x5602f0f29a50 2024-03-16T11:59:09.514 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.508+0000 7f92a0010740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5602f0e89ae0 con 0x5602f0f29130 2024-03-16T11:59:09.514 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.508+0000 7f92977fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5602f0f32820 0x5602f0f34c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:59:09.514 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.508+0000 7f9296ffd640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5602f0f29130 0x5602f0f29510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:59:09.514 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.508+0000 7f92967fc640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5602f0f29a50 0x5602f0f322e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:59:09.514 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.508+0000 7f92977fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5602f0f32820 0x5602f0f34c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:48626/0 (socket says 172.21.15.29:48626) 2024-03-16T11:59:09.515 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.508+0000 7f92977fe640 1 -- 172.21.15.29:0/3079313252 learned_addr learned my addr 172.21.15.29:0/3079313252 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:59:09.515 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.509+0000 7f92967fc640 1 --2- 172.21.15.29:0/4182958333 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f92840a2a10 0x7f92840a4ed0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f9298101a70 tx=0x7f9280005f90 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:59:09.515 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.509+0000 7f92977fe640 1 -- 172.21.15.29:0/3079313252 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5602f0f29130 msgr2=0x5602f0f29510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:59:09.515 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.509+0000 7f92977fe640 1 --2- 172.21.15.29:0/3079313252 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5602f0f29130 0x5602f0f29510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:59:09.515 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.509+0000 7f92977fe640 1 -- 172.21.15.29:0/3079313252 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5602f0f29a50 msgr2=0x5602f0f322e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:59:09.515 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.509+0000 7f92977fe640 1 --2- 172.21.15.29:0/3079313252 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5602f0f29a50 0x5602f0f322e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:59:09.515 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.509+0000 7f92977fe640 1 -- 172.21.15.29:0/3079313252 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5602f0e4e460 con 0x5602f0f32820 2024-03-16T11:59:09.515 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.509+0000 7f9296ffd640 1 --2- 172.21.15.29:0/3079313252 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5602f0f29130 0x5602f0f29510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:59:09.515 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.509+0000 7f92977fe640 1 --2- 172.21.15.29:0/3079313252 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5602f0f32820 0x5602f0f34c10 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f92880a3c40 tx=0x7f928809b510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d759c2efaeb696c3 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:59:09.515 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.509+0000 7f928d7fa640 1 -- 172.21.15.29:0/3079313252 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f928809b9d0 con 0x5602f0f32820 2024-03-16T11:59:09.515 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.509+0000 7f928d7fa640 1 -- 172.21.15.29:0/3079313252 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f928809fb20 con 0x5602f0f32820 2024-03-16T11:59:09.515 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.509+0000 7f928d7fa640 1 -- 172.21.15.29:0/3079313252 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92880a3ce0 con 0x5602f0f32820 2024-03-16T11:59:09.515 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.510+0000 7f92a0010740 1 -- 172.21.15.29:0/3079313252 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5602f0f32820 msgr2=0x5602f0f34c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:59:09.515 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.510+0000 7f92a0010740 1 --2- 172.21.15.29:0/3079313252 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5602f0f32820 0x5602f0f34c10 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f92880a3c40 tx=0x7f928809b510 comp rx=0 tx=0).stop 2024-03-16T11:59:09.516 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.510+0000 7f92a0010740 1 -- 172.21.15.29:0/3079313252 shutdown_connections 2024-03-16T11:59:09.516 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.510+0000 7f92a0010740 1 --2- 172.21.15.29:0/3079313252 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5602f0f32820 0x5602f0f34c10 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:59:09.516 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.510+0000 7f92a0010740 1 --2- 172.21.15.29:0/3079313252 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5602f0f29a50 0x5602f0f322e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:59:09.516 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.510+0000 7f92a0010740 1 --2- 172.21.15.29:0/3079313252 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5602f0f29130 0x5602f0f29510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:59:09.516 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.510+0000 7f92a0010740 1 -- 172.21.15.29:0/3079313252 >> 172.21.15.29:0/3079313252 conn(0x5602f0f23a50 msgr2=0x5602f0f23e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:59:09.516 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.510+0000 7f92a0010740 1 -- 172.21.15.29:0/3079313252 shutdown_connections 2024-03-16T11:59:09.516 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.510+0000 7f92a0010740 1 -- 172.21.15.29:0/3079313252 wait complete. 2024-03-16T11:59:09.516 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.510+0000 7f92a0010740 1 -- start start 2024-03-16T11:59:09.516 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.510+0000 7f92a0010740 1 Processor -- start 2024-03-16T11:59:09.516 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.510+0000 7f92a0010740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5602f0f29130 0x5602f0f2f770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:59:09.517 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.510+0000 7f92a0010740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5602f0f29a50 0x5602f0f2fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:59:09.517 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.511+0000 7f92a0010740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5602f0f32820 0x5602f0f301f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:59:09.517 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.511+0000 7f92a0010740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x5602f0f353f0 con 0x5602f0f32820 2024-03-16T11:59:09.517 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.511+0000 7f92a0010740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x5602f0e49d70 con 0x5602f0f29130 2024-03-16T11:59:09.517 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.511+0000 7f92a0010740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5602f0f35570 con 0x5602f0f29a50 2024-03-16T11:59:09.517 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.511+0000 7f92a0010740 10 client.0 _ll_register_callbacks cb 0x5602f0ba1a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T11:59:09.517 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.511+0000 7f92977fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5602f0f32820 0x5602f0f301f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:59:09.517 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.511+0000 7f9296ffd640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5602f0f29130 0x5602f0f2f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:59:09.517 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.511+0000 7f9296ffd640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5602f0f29130 0x5602f0f2f770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.29:38144/0 (socket says 172.21.15.29:38144) 2024-03-16T11:59:09.517 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.511+0000 7f92977fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5602f0f32820 0x5602f0f301f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:48642/0 (socket says 172.21.15.29:48642) 2024-03-16T11:59:09.517 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.511+0000 7f9296ffd640 1 -- 172.21.15.29:0/1872719497 learned_addr learned my addr 172.21.15.29:0/1872719497 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:59:09.517 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.511+0000 7f92967fc640 1 --2- 172.21.15.29:0/1872719497 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5602f0f29a50 0x5602f0f2fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:59:09.518 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.511+0000 7f9296ffd640 1 -- 172.21.15.29:0/1872719497 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5602f0f29a50 msgr2=0x5602f0f2fcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:59:09.518 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.511+0000 7f9296ffd640 1 --2- 172.21.15.29:0/1872719497 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5602f0f29a50 0x5602f0f2fcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:59:09.518 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.511+0000 7f9296ffd640 1 -- 172.21.15.29:0/1872719497 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5602f0f32820 msgr2=0x5602f0f301f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:59:09.518 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.511+0000 7f9296ffd640 1 --2- 172.21.15.29:0/1872719497 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5602f0f32820 0x5602f0f301f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:59:09.518 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.511+0000 7f9296ffd640 1 -- 172.21.15.29:0/1872719497 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5602f0ba2210 con 0x5602f0f29130 2024-03-16T11:59:09.518 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.511+0000 7f92977fe640 1 --2- 172.21.15.29:0/1872719497 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5602f0f32820 0x5602f0f301f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:59:09.518 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.511+0000 7f9296ffd640 1 --2- 172.21.15.29:0/1872719497 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5602f0f29130 0x5602f0f2f770 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f929002cfd0 tx=0x7f9290009600 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b1af0cad7019361a server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:59:09.518 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.511+0000 7f927e7fc640 10 client.0 ms_handle_connect on v2:172.21.15.152:3300/0 2024-03-16T11:59:09.518 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.511+0000 7f927e7fc640 1 -- 172.21.15.29:0/1872719497 <== mon.2 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9290004240 con 0x5602f0f29130 2024-03-16T11:59:09.518 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.511+0000 7f92a0010740 1 -- 172.21.15.29:0/1872719497 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f928809fb20 con 0x5602f0f29130 2024-03-16T11:59:09.518 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.511+0000 7f927e7fc640 1 -- 172.21.15.29:0/1872719497 <== mon.2 v2:172.21.15.152:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8600+0+0 (secure 0 0 0) 0x7f92900043e0 con 0x5602f0f29130 2024-03-16T11:59:09.519 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.512+0000 7f927e7fc640 1 -- 172.21.15.29:0/1872719497 <== mon.2 v2:172.21.15.152:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f92900046c0 con 0x5602f0f29130 2024-03-16T11:59:09.519 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.512+0000 7f92a0010740 10 client.25901 fetch_fsmap learned FSMap version 26 2024-03-16T11:59:09.519 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.512+0000 7f92a0010740 1 -- 172.21.15.29:0/1872719497 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5602f0c6e620 con 0x5602f0f29130 2024-03-16T11:59:09.519 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.512+0000 7f927e7fc640 1 -- 172.21.15.29:0/1872719497 <== mon.2 v2:172.21.15.152:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f928809fb20 con 0x5602f0f29130 2024-03-16T11:59:09.519 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.512+0000 7f927e7fc640 1 -- 172.21.15.29:0/1872719497 <== mon.2 v2:172.21.15.152:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f92900035f0 con 0x5602f0f29130 2024-03-16T11:59:09.519 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.512+0000 7f92a0010740 10 client.25901 fetch_fsmap finished waiting for FSMap version 26 2024-03-16T11:59:09.519 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.512+0000 7f92a0010740 10 client.25901 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T11:59:09.519 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.512+0000 7f92a0010740 20 client.25901 populate_metadata read hostname 'smithi029' 2024-03-16T11:59:09.519 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.512+0000 7f92a0010740 1 --2- 172.21.15.29:0/1872719497 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x5602f0f42990 0x5602f0f44d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:59:09.519 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.512+0000 7f92a0010740 4 client.25901 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T11:59:09.519 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.512+0000 7f92a0010740 1 -- 172.21.15.29:0/1872719497 --> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5602f0e9af80 con 0x5602f0f42990 2024-03-16T11:59:09.519 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.512+0000 7f92967fc640 1 --2- 172.21.15.29:0/1872719497 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x5602f0f42990 0x5602f0f44d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:59:09.519 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.513+0000 7f927e7fc640 10 client.25901 ms_handle_connect on v2:172.21.15.152:6838/1548731833 2024-03-16T11:59:09.520 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.513+0000 7f92967fc640 1 --2- 172.21.15.29:0/1872719497 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x5602f0f42990 0x5602f0f44d80 crc :-1 s=READY pgs=112 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d7109af42a1b5545 server_cookie=ec9db529194e1db2 in_seq=0 out_seq=0 2024-03-16T11:59:09.521 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.514+0000 7f927e7fc640 1 -- 172.21.15.29:0/1872719497 <== mds.0 v2:172.21.15.152:6838/1548731833 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5602f0e9af80 con 0x5602f0f42990 2024-03-16T11:59:09.522 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.514+0000 7f927e7fc640 10 client.25901 handle_command_reply: tid=0 2024-03-16T11:59:09.522 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.514+0000 7f92a0010740 1 client.25901 shutdown 2024-03-16T11:59:09.522 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.515+0000 7f92a0010740 1 -- 172.21.15.29:0/1872719497 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5602f0f29130 msgr2=0x5602f0f2f770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:59:09.522 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.515+0000 7f92a0010740 1 --2- 172.21.15.29:0/1872719497 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5602f0f29130 0x5602f0f2f770 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f929002cfd0 tx=0x7f9290009600 comp rx=0 tx=0).stop 2024-03-16T11:59:09.522 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.515+0000 7f92a0010740 1 -- 172.21.15.29:0/1872719497 shutdown_connections 2024-03-16T11:59:09.522 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.515+0000 7f92a0010740 1 --2- 172.21.15.29:0/1872719497 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5602f0f32820 0x5602f0f301f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:59:09.522 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.515+0000 7f92a0010740 1 --2- 172.21.15.29:0/1872719497 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5602f0f29a50 0x5602f0f2fcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:59:09.522 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.515+0000 7f92a0010740 1 --2- 172.21.15.29:0/1872719497 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x5602f0f42990 0x5602f0f44d80 crc :-1 s=READY pgs=112 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:59:09.522 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.515+0000 7f92a0010740 1 --2- 172.21.15.29:0/1872719497 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5602f0f29130 0x5602f0f2f770 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:59:09.522 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.515+0000 7f927e7fc640 0 client.25901 ms_handle_reset on v2:172.21.15.152:6838/1548731833 2024-03-16T11:59:09.522 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.515+0000 7f92a0010740 1 -- 172.21.15.29:0/1872719497 >> 172.21.15.29:0/1872719497 conn(0x5602f0f23a50 msgr2=0x5602f0f257a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:59:09.522 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.515+0000 7f92a0010740 1 -- 172.21.15.29:0/1872719497 shutdown_connections 2024-03-16T11:59:09.523 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.515+0000 7f92a0010740 1 -- 172.21.15.29:0/1872719497 wait complete. 2024-03-16T11:59:09.523 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.515+0000 7f92a0010740 20 client.25901 trim_cache size 0 max 16384 2024-03-16T11:59:09.528 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.530+0000 7f92a0010740 1 Processor -- start 2024-03-16T11:59:09.529 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.530+0000 7f92a0010740 1 -- start start 2024-03-16T11:59:09.533 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.530+0000 7f92a0010740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5602f0f29130 0x5602f0f31ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:59:09.533 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.530+0000 7f92a0010740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5602f0f29a50 0x5602f0f3e1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:59:09.534 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.530+0000 7f92a0010740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5602f0f32820 0x5602f0f4a650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:59:09.534 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.530+0000 7f92a0010740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f92981b2000 con 0x5602f0f29a50 2024-03-16T11:59:09.534 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.530+0000 7f92a0010740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f92981af8d0 con 0x5602f0f32820 2024-03-16T11:59:09.534 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.530+0000 7f92a0010740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f92981ad740 con 0x5602f0f29130 2024-03-16T11:59:09.534 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.530+0000 7f92967fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5602f0f29a50 0x5602f0f3e1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:59:09.534 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.530+0000 7f92967fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5602f0f29a50 0x5602f0f3e1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:48654/0 (socket says 172.21.15.29:48654) 2024-03-16T11:59:09.534 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.530+0000 7f92967fc640 1 -- 172.21.15.29:0/2911026496 learned_addr learned my addr 172.21.15.29:0/2911026496 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:59:09.534 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.530+0000 7f9296ffd640 1 --2- 172.21.15.29:0/2911026496 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5602f0f29130 0x5602f0f31ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:59:09.534 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.530+0000 7f92977fe640 1 --2- 172.21.15.29:0/2911026496 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5602f0f32820 0x5602f0f4a650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:59:09.534 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.530+0000 7f92967fc640 1 -- 172.21.15.29:0/2911026496 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5602f0f29130 msgr2=0x5602f0f31ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:59:09.534 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.530+0000 7f92967fc640 1 --2- 172.21.15.29:0/2911026496 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5602f0f29130 0x5602f0f31ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:59:09.534 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.530+0000 7f92967fc640 1 -- 172.21.15.29:0/2911026496 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5602f0f32820 msgr2=0x5602f0f4a650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:59:09.534 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.530+0000 7f92967fc640 1 --2- 172.21.15.29:0/2911026496 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5602f0f32820 0x5602f0f4a650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:59:09.534 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.530+0000 7f92967fc640 1 -- 172.21.15.29:0/2911026496 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5602f0f1dfc0 con 0x5602f0f29a50 2024-03-16T11:59:09.535 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.531+0000 7f92967fc640 1 --2- 172.21.15.29:0/2911026496 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5602f0f29a50 0x5602f0f3e1c0 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f9280013010 tx=0x7f9280019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c1c05a93701270c server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:59:09.535 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.531+0000 7f928cff9640 1 -- 172.21.15.29:0/2911026496 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9280030070 con 0x5602f0f29a50 2024-03-16T11:59:09.535 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.531+0000 7f928cff9640 1 -- 172.21.15.29:0/2911026496 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f928002a710 con 0x5602f0f29a50 2024-03-16T11:59:09.535 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.531+0000 7f928cff9640 1 -- 172.21.15.29:0/2911026496 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f928001c440 con 0x5602f0f29a50 2024-03-16T11:59:09.535 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.531+0000 7f92a0010740 1 -- 172.21.15.29:0/2911026496 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5602f0f29a50 msgr2=0x5602f0f3e1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:59:09.535 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.531+0000 7f92a0010740 1 --2- 172.21.15.29:0/2911026496 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5602f0f29a50 0x5602f0f3e1c0 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f9280013010 tx=0x7f9280019000 comp rx=0 tx=0).stop 2024-03-16T11:59:09.535 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.532+0000 7f92a0010740 1 -- 172.21.15.29:0/2911026496 shutdown_connections 2024-03-16T11:59:09.535 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.532+0000 7f92a0010740 1 --2- 172.21.15.29:0/2911026496 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5602f0f32820 0x5602f0f4a650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:59:09.535 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.532+0000 7f92a0010740 1 --2- 172.21.15.29:0/2911026496 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5602f0f29a50 0x5602f0f3e1c0 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:59:09.535 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.532+0000 7f92a0010740 1 --2- 172.21.15.29:0/2911026496 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5602f0f29130 0x5602f0f31ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:59:09.535 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.532+0000 7f92a0010740 1 -- 172.21.15.29:0/2911026496 >> 172.21.15.29:0/2911026496 conn(0x5602f0f23a50 msgr2=0x5602f0f25730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:59:09.535 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.532+0000 7f92a0010740 1 -- 172.21.15.29:0/2911026496 shutdown_connections 2024-03-16T11:59:09.535 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.532+0000 7f92a0010740 1 -- 172.21.15.29:0/2911026496 wait complete. 2024-03-16T11:59:09.535 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.532+0000 7f92a0010740 1 -- start start 2024-03-16T11:59:09.535 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.532+0000 7f92a0010740 1 Processor -- start 2024-03-16T11:59:09.536 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.532+0000 7f92a0010740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5602f0f29130 0x5602f0f2c3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:59:09.536 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.532+0000 7f92a0010740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5602f0f29a50 0x5602f0f2c900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:59:09.536 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.532+0000 7f92a0010740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5602f0f32820 0x5602f0f2ce40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:59:09.536 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.532+0000 7f92a0010740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f92981add90 con 0x5602f0f29a50 2024-03-16T11:59:09.536 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.532+0000 7f92a0010740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f92981af200 con 0x5602f0f32820 2024-03-16T11:59:09.536 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.532+0000 7f92a0010740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f92981ae460 con 0x5602f0f29130 2024-03-16T11:59:09.536 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.532+0000 7f92a0010740 10 client.0 _ll_register_callbacks cb 0x7f929002cd60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T11:59:09.536 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.532+0000 7f92967fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5602f0f29a50 0x5602f0f2c900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:59:09.536 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.532+0000 7f92967fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5602f0f29a50 0x5602f0f2c900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:48662/0 (socket says 172.21.15.29:48662) 2024-03-16T11:59:09.536 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.532+0000 7f9296ffd640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5602f0f29130 0x5602f0f2c3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:59:09.536 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.532+0000 7f92967fc640 1 -- 172.21.15.29:0/2349945769 learned_addr learned my addr 172.21.15.29:0/2349945769 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:59:09.536 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.532+0000 7f92977fe640 1 --2- 172.21.15.29:0/2349945769 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5602f0f32820 0x5602f0f2ce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:59:09.536 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.533+0000 7f92967fc640 1 -- 172.21.15.29:0/2349945769 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5602f0f29130 msgr2=0x5602f0f2c3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:59:09.536 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.533+0000 7f92967fc640 1 --2- 172.21.15.29:0/2349945769 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5602f0f29130 0x5602f0f2c3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:59:09.537 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.533+0000 7f92967fc640 1 -- 172.21.15.29:0/2349945769 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5602f0f32820 msgr2=0x5602f0f2ce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:59:09.537 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.533+0000 7f92967fc640 1 --2- 172.21.15.29:0/2349945769 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5602f0f32820 0x5602f0f2ce40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:59:09.537 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.533+0000 7f92967fc640 1 -- 172.21.15.29:0/2349945769 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5602f0fa7d30 con 0x5602f0f29a50 2024-03-16T11:59:09.537 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.533+0000 7f9296ffd640 1 --2- 172.21.15.29:0/2349945769 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5602f0f29130 0x5602f0f2c3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:59:09.537 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.533+0000 7f92967fc640 1 --2- 172.21.15.29:0/2349945769 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5602f0f29a50 0x5602f0f2c900 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f928002e040 tx=0x7f928002a950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf1d9444016f0c98 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:59:09.537 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.533+0000 7f927e7fc640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-16T11:59:09.537 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.533+0000 7f927e7fc640 1 -- 172.21.15.29:0/2349945769 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9280030070 con 0x5602f0f29a50 2024-03-16T11:59:09.537 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.533+0000 7f927e7fc640 1 -- 172.21.15.29:0/2349945769 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8600+0+0 (secure 0 0 0) 0x7f928000fce0 con 0x5602f0f29a50 2024-03-16T11:59:09.537 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.533+0000 7f92a0010740 1 -- 172.21.15.29:0/2349945769 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f928002a710 con 0x5602f0f29a50 2024-03-16T11:59:09.537 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.533+0000 7f927e7fc640 1 -- 172.21.15.29:0/2349945769 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f928000f480 con 0x5602f0f29a50 2024-03-16T11:59:09.537 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.533+0000 7f92a0010740 10 client.16092 fetch_fsmap learned FSMap version 26 2024-03-16T11:59:09.537 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.533+0000 7f92a0010740 1 -- 172.21.15.29:0/2349945769 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5602f0fa9000 con 0x5602f0f29a50 2024-03-16T11:59:09.537 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.533+0000 7f927e7fc640 1 -- 172.21.15.29:0/2349945769 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f928002a710 con 0x5602f0f29a50 2024-03-16T11:59:09.537 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.533+0000 7f927e7fc640 1 -- 172.21.15.29:0/2349945769 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f928003c730 con 0x5602f0f29a50 2024-03-16T11:59:09.537 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.533+0000 7f92a0010740 10 client.16092 fetch_fsmap finished waiting for FSMap version 26 2024-03-16T11:59:09.538 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.533+0000 7f92a0010740 10 client.16092 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T11:59:09.538 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.533+0000 7f92a0010740 20 client.16092 populate_metadata read hostname 'smithi029' 2024-03-16T11:59:09.538 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.533+0000 7f92a0010740 1 --2- 172.21.15.29:0/2349945769 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x5602f0fa9740 0x5602f0fabb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:59:09.538 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.533+0000 7f92a0010740 4 client.16092 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-16T11:59:09.538 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.533+0000 7f92a0010740 1 -- 172.21.15.29:0/2349945769 --> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5602f0e952d0 con 0x5602f0fa9740 2024-03-16T11:59:09.538 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.534+0000 7f9296ffd640 1 --2- 172.21.15.29:0/2349945769 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x5602f0fa9740 0x5602f0fabb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:59:09.538 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.534+0000 7f9296ffd640 1 --2- 172.21.15.29:0/2349945769 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x5602f0fa9740 0x5602f0fabb60 crc :-1 s=READY pgs=113 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=29c11582dfd2eeff server_cookie=2a906ef6b688e5b6 in_seq=0 out_seq=0 2024-03-16T11:59:09.538 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.534+0000 7f927e7fc640 10 client.16092 ms_handle_connect on v2:172.21.15.152:6838/1548731833 2024-03-16T11:59:09.708 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.708+0000 7f927e7fc640 1 -- 172.21.15.29:0/2349945769 <== mds.0 v2:172.21.15.152:6838/1548731833 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5602f0e952d0 con 0x5602f0fa9740 2024-03-16T11:59:09.708 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.708+0000 7f927e7fc640 10 client.16092 handle_command_reply: tid=0 2024-03-16T11:59:09.708 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.708+0000 7f92a0010740 1 client.16092 shutdown 2024-03-16T11:59:09.708 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.708+0000 7f92a0010740 1 -- 172.21.15.29:0/2349945769 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5602f0f29a50 msgr2=0x5602f0f2c900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:59:09.708 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.708+0000 7f92a0010740 1 --2- 172.21.15.29:0/2349945769 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5602f0f29a50 0x5602f0f2c900 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f928002e040 tx=0x7f928002a950 comp rx=0 tx=0).stop 2024-03-16T11:59:09.708 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.708+0000 7f92a0010740 1 -- 172.21.15.29:0/2349945769 shutdown_connections 2024-03-16T11:59:09.709 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.708+0000 7f92a0010740 1 --2- 172.21.15.29:0/2349945769 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x5602f0fa9740 0x5602f0fabb60 crc :-1 s=READY pgs=113 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:59:09.709 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.708+0000 7f92a0010740 1 --2- 172.21.15.29:0/2349945769 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5602f0f32820 0x5602f0f2ce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:59:09.709 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.708+0000 7f92a0010740 1 --2- 172.21.15.29:0/2349945769 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5602f0f29a50 0x5602f0f2c900 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:59:09.709 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.708+0000 7f92a0010740 1 --2- 172.21.15.29:0/2349945769 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5602f0f29130 0x5602f0f2c3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:59:09.709 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.708+0000 7f92a0010740 1 -- 172.21.15.29:0/2349945769 >> 172.21.15.29:0/2349945769 conn(0x5602f0f23a50 msgr2=0x5602f0f22320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:59:09.709 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.708+0000 7f927e7fc640 0 client.16092 ms_handle_reset on v2:172.21.15.152:6838/1548731833 2024-03-16T11:59:09.709 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.708+0000 7f92a0010740 1 -- 172.21.15.29:0/2349945769 shutdown_connections 2024-03-16T11:59:09.709 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.709+0000 7f92a0010740 1 -- 172.21.15.29:0/2349945769 wait complete. 2024-03-16T11:59:09.709 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.709+0000 7f92a0010740 20 client.16092 trim_cache size 0 max 16384 2024-03-16T11:59:09.709 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:{ 2024-03-16T11:59:09.709 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "message": "", 2024-03-16T11:59:09.710 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "return_code": 0 2024-03-16T11:59:09.710 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:} 2024-03-16T11:59:09.710 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.709+0000 7f929d427640 1 -- 172.21.15.29:0/4182958333 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f92840a2a10 msgr2=0x7f92840a4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:59:09.711 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.709+0000 7f929d427640 1 --2- 172.21.15.29:0/4182958333 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f92840a2a10 0x7f92840a4ed0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f9298101a70 tx=0x7f9280005f90 comp rx=0 tx=0).stop 2024-03-16T11:59:09.711 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.709+0000 7f929d427640 1 -- 172.21.15.29:0/4182958333 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f92981099c0 msgr2=0x7f929819eac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:59:09.711 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.709+0000 7f929d427640 1 --2- 172.21.15.29:0/4182958333 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f92981099c0 0x7f929819eac0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f928800eb10 tx=0x7f928800c2d0 comp rx=0 tx=0).stop 2024-03-16T11:59:09.711 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.709+0000 7f929d427640 1 -- 172.21.15.29:0/4182958333 shutdown_connections 2024-03-16T11:59:09.711 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.709+0000 7f929d427640 1 --2- 172.21.15.29:0/4182958333 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f92840a2a10 0x7f92840a4ed0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:59:09.711 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.709+0000 7f929d427640 1 --2- 172.21.15.29:0/4182958333 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f92981099c0 0x7f929819eac0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:59:09.711 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.709+0000 7f929d427640 1 --2- 172.21.15.29:0/4182958333 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9298104370 0x7f929819a740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:59:09.711 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.709+0000 7f929d427640 1 --2- 172.21.15.29:0/4182958333 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f92981039b0 0x7f929819a200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:59:09.711 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.709+0000 7f929d427640 1 -- 172.21.15.29:0/4182958333 >> 172.21.15.29:0/4182958333 conn(0x7f92980ff4a0 msgr2=0x7f92980ffb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:59:09.711 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.710+0000 7f929d427640 1 -- 172.21.15.29:0/4182958333 shutdown_connections 2024-03-16T11:59:09.711 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:09.710+0000 7f929d427640 1 -- 172.21.15.29:0/4182958333 wait complete. 2024-03-16T11:59:09.794 INFO:tasks.workunit.client.0.smithi029.stdout: 1 833213 10.35 MB/sec execute 294 sec latency 93.049 ms 2024-03-16T11:59:10.794 INFO:tasks.workunit.client.0.smithi029.stdout: 1 834879 10.35 MB/sec execute 295 sec latency 52.870 ms 2024-03-16T11:59:10.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:59:10 smithi154 ceph-mon[36320]: pgmap v258: 97 pgs: 18 active+clean+snaptrim, 38 active+clean+snaptrim_wait, 41 active+clean; 82 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.1 MiB/s wr, 315 op/s 2024-03-16T11:59:10.837 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:59:10 smithi029 ceph-mon[31975]: pgmap v258: 97 pgs: 18 active+clean+snaptrim, 38 active+clean+snaptrim_wait, 41 active+clean; 82 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.1 MiB/s wr, 315 op/s 2024-03-16T11:59:10.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:59:10 smithi152 ceph-mon[36291]: pgmap v258: 97 pgs: 18 active+clean+snaptrim, 38 active+clean+snaptrim_wait, 41 active+clean; 82 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.1 MiB/s wr, 315 op/s 2024-03-16T11:59:11.794 INFO:tasks.workunit.client.0.smithi029.stdout: 1 836693 10.34 MB/sec execute 296 sec latency 63.358 ms 2024-03-16T11:59:12.794 INFO:tasks.workunit.client.0.smithi029.stdout: 1 838352 10.35 MB/sec execute 297 sec latency 82.420 ms 2024-03-16T11:59:12.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:59:12 smithi154 ceph-mon[36320]: pgmap v259: 97 pgs: 2 active+clean+snaptrim, 4 active+clean+snaptrim_wait, 91 active+clean; 76 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 2.9 MiB/s wr, 277 op/s 2024-03-16T11:59:12.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:59:12 smithi029 ceph-mon[31975]: pgmap v259: 97 pgs: 2 active+clean+snaptrim, 4 active+clean+snaptrim_wait, 91 active+clean; 76 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 2.9 MiB/s wr, 277 op/s 2024-03-16T11:59:12.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:59:12 smithi152 ceph-mon[36291]: pgmap v259: 97 pgs: 2 active+clean+snaptrim, 4 active+clean+snaptrim_wait, 91 active+clean; 76 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 2.9 MiB/s wr, 277 op/s 2024-03-16T11:59:13.794 INFO:tasks.workunit.client.0.smithi029.stdout: 1 839687 10.33 MB/sec execute 298 sec latency 166.322 ms 2024-03-16T11:59:14.795 INFO:tasks.workunit.client.0.smithi029.stdout: 1 841777 10.34 MB/sec execute 299 sec latency 18.115 ms 2024-03-16T11:59:14.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:59:14 smithi154 ceph-mon[36320]: pgmap v260: 97 pgs: 97 active+clean; 73 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 3.1 MiB/s wr, 295 op/s 2024-03-16T11:59:14.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:59:14 smithi029 ceph-mon[31975]: pgmap v260: 97 pgs: 97 active+clean; 73 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 3.1 MiB/s wr, 295 op/s 2024-03-16T11:59:14.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:59:14 smithi152 ceph-mon[36291]: pgmap v260: 97 pgs: 97 active+clean; 73 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 3.1 MiB/s wr, 295 op/s 2024-03-16T11:59:15.794 INFO:tasks.workunit.client.0.smithi029.stdout: 1 843152 10.33 MB/sec execute 300 sec latency 40.264 ms 2024-03-16T11:59:16.794 INFO:tasks.workunit.client.0.smithi029.stdout: 1 845241 10.34 MB/sec execute 301 sec latency 115.425 ms 2024-03-16T11:59:16.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:59:16 smithi154 ceph-mon[36320]: pgmap v261: 97 pgs: 97 active+clean; 73 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.0 MiB/s wr, 282 op/s 2024-03-16T11:59:16.837 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:59:16 smithi029 ceph-mon[31975]: pgmap v261: 97 pgs: 97 active+clean; 73 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.0 MiB/s wr, 282 op/s 2024-03-16T11:59:16.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:59:16 smithi152 ceph-mon[36291]: pgmap v261: 97 pgs: 97 active+clean; 73 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.0 MiB/s wr, 282 op/s 2024-03-16T11:59:17.795 INFO:tasks.workunit.client.0.smithi029.stdout: 1 846611 10.32 MB/sec execute 302 sec latency 25.803 ms 2024-03-16T11:59:18.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:59:18 smithi029 ceph-mon[31975]: pgmap v262: 97 pgs: 97 active+clean; 77 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 3.8 MiB/s wr, 331 op/s 2024-03-16T11:59:18.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:59:18 smithi029 ceph-mon[31975]: osdmap e97: 12 total, 12 up, 12 in 2024-03-16T11:59:18.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:59:18 smithi152 ceph-mon[36291]: pgmap v262: 97 pgs: 97 active+clean; 77 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 3.8 MiB/s wr, 331 op/s 2024-03-16T11:59:18.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:59:18 smithi152 ceph-mon[36291]: osdmap e97: 12 total, 12 up, 12 in 2024-03-16T11:59:18.545 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:59:18 smithi154 ceph-mon[36320]: pgmap v262: 97 pgs: 97 active+clean; 77 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 3.8 MiB/s wr, 331 op/s 2024-03-16T11:59:18.545 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:59:18 smithi154 ceph-mon[36320]: osdmap e97: 12 total, 12 up, 12 in 2024-03-16T11:59:18.795 INFO:tasks.workunit.client.0.smithi029.stdout: 1 848811 10.33 MB/sec execute 303 sec latency 21.090 ms 2024-03-16T11:59:19.795 INFO:tasks.workunit.client.0.smithi029.stdout: 1 850258 10.32 MB/sec execute 304 sec latency 22.632 ms 2024-03-16T11:59:20.337 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:59:20 smithi029 ceph-mon[31975]: pgmap v264: 97 pgs: 97 active+clean; 77 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 3.1 MiB/s wr, 278 op/s 2024-03-16T11:59:20.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:59:20 smithi152 ceph-mon[36291]: pgmap v264: 97 pgs: 97 active+clean; 77 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 3.1 MiB/s wr, 278 op/s 2024-03-16T11:59:20.545 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:59:20 smithi154 ceph-mon[36320]: pgmap v264: 97 pgs: 97 active+clean; 77 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 3.1 MiB/s wr, 278 op/s 2024-03-16T11:59:20.795 INFO:tasks.workunit.client.0.smithi029.stdout: 1 852396 10.33 MB/sec execute 305 sec latency 19.077 ms 2024-03-16T11:59:21.795 INFO:tasks.workunit.client.0.smithi029.stdout: 1 853692 10.31 MB/sec execute 306 sec latency 84.935 ms 2024-03-16T11:59:22.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:59:22 smithi029 ceph-mon[31975]: pgmap v265: 97 pgs: 97 active+clean; 82 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 2.9 MiB/s wr, 213 op/s 2024-03-16T11:59:22.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:59:22 smithi152 ceph-mon[36291]: pgmap v265: 97 pgs: 97 active+clean; 82 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 2.9 MiB/s wr, 213 op/s 2024-03-16T11:59:22.545 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:59:22 smithi154 ceph-mon[36320]: pgmap v265: 97 pgs: 97 active+clean; 82 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 2.9 MiB/s wr, 213 op/s 2024-03-16T11:59:22.795 INFO:tasks.workunit.client.0.smithi029.stdout: 1 855857 10.32 MB/sec execute 307 sec latency 41.426 ms 2024-03-16T11:59:22.808 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.805+0000 7fef47687640 1 -- 172.21.15.29:0/2432820645 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fef4006a920 msgr2=0x7fef4010dfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:59:22.808 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.805+0000 7fef47687640 1 --2- 172.21.15.29:0/2432820645 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fef4006a920 0x7fef4010dfa0 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7fef34009900 tx=0x7fef3402bd10 comp rx=0 tx=0).stop 2024-03-16T11:59:22.808 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.806+0000 7fef47687640 1 -- 172.21.15.29:0/2432820645 shutdown_connections 2024-03-16T11:59:22.808 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.806+0000 7fef47687640 1 --2- 172.21.15.29:0/2432820645 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fef4010e4e0 0x7fef401108d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:59:22.809 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.806+0000 7fef47687640 1 --2- 172.21.15.29:0/2432820645 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fef4006a920 0x7fef4010dfa0 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:59:22.809 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.806+0000 7fef47687640 1 --2- 172.21.15.29:0/2432820645 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fef4006a000 0x7fef4006a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:59:22.809 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.806+0000 7fef47687640 1 -- 172.21.15.29:0/2432820645 >> 172.21.15.29:0/2432820645 conn(0x7fef40077540 msgr2=0x7fef40077950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:59:22.809 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.806+0000 7fef47687640 1 -- 172.21.15.29:0/2432820645 shutdown_connections 2024-03-16T11:59:22.809 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.806+0000 7fef47687640 1 -- 172.21.15.29:0/2432820645 wait complete. 2024-03-16T11:59:22.809 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.806+0000 7fef47687640 1 Processor -- start 2024-03-16T11:59:22.809 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.806+0000 7fef47687640 1 -- start start 2024-03-16T11:59:22.809 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.806+0000 7fef47687640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fef4006a000 0x7fef4019a1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:59:22.809 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.806+0000 7fef47687640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fef4006a920 0x7fef4019a6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:59:22.809 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.806+0000 7fef47687640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fef4010e4e0 0x7fef4019ea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:59:22.809 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.806+0000 7fef47687640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fef40113de0 con 0x7fef4010e4e0 2024-03-16T11:59:22.809 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.806+0000 7fef47687640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fef40113c60 con 0x7fef4006a000 2024-03-16T11:59:22.810 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.806+0000 7fef47687640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fef40113f60 con 0x7fef4006a920 2024-03-16T11:59:22.810 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.807+0000 7fef44c22640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fef4006a920 0x7fef4019a6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:59:22.810 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.807+0000 7fef45423640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fef4006a000 0x7fef4019a1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:59:22.810 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.807+0000 7fef45c24640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fef4010e4e0 0x7fef4019ea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:59:22.810 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.807+0000 7fef44c22640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fef4006a920 0x7fef4019a6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.29:46750/0 (socket says 172.21.15.29:46750) 2024-03-16T11:59:22.810 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.807+0000 7fef45423640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fef4006a000 0x7fef4019a1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.29:51932/0 (socket says 172.21.15.29:51932) 2024-03-16T11:59:22.810 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.807+0000 7fef44c22640 1 -- 172.21.15.29:0/1662843024 learned_addr learned my addr 172.21.15.29:0/1662843024 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:59:22.810 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.807+0000 7fef45c24640 1 -- 172.21.15.29:0/1662843024 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fef4006a920 msgr2=0x7fef4019a6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:59:22.810 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.807+0000 7fef45c24640 1 --2- 172.21.15.29:0/1662843024 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fef4006a920 0x7fef4019a6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:59:22.810 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.807+0000 7fef45c24640 1 -- 172.21.15.29:0/1662843024 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fef4006a000 msgr2=0x7fef4019a1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:59:22.810 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.807+0000 7fef45c24640 1 --2- 172.21.15.29:0/1662843024 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fef4006a000 0x7fef4019a1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:59:22.810 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.807+0000 7fef45c24640 1 -- 172.21.15.29:0/1662843024 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef4019f190 con 0x7fef4010e4e0 2024-03-16T11:59:22.811 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.807+0000 7fef44c22640 1 --2- 172.21.15.29:0/1662843024 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fef4006a920 0x7fef4019a6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:59:22.811 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.807+0000 7fef45c24640 1 --2- 172.21.15.29:0/1662843024 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fef4010e4e0 0x7fef4019ea60 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7fef40069b60 tx=0x7fef3000ee60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:59:22.811 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.807+0000 7fef3e7fc640 1 -- 172.21.15.29:0/1662843024 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef3000cd50 con 0x7fef4010e4e0 2024-03-16T11:59:22.811 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.807+0000 7fef3e7fc640 1 -- 172.21.15.29:0/1662843024 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fef30005020 con 0x7fef4010e4e0 2024-03-16T11:59:22.811 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.807+0000 7fef3e7fc640 1 -- 172.21.15.29:0/1662843024 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef30010690 con 0x7fef4010e4e0 2024-03-16T11:59:22.811 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.807+0000 7fef47687640 1 -- 172.21.15.29:0/1662843024 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fef4019f480 con 0x7fef4010e4e0 2024-03-16T11:59:22.811 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.807+0000 7fef47687640 1 -- 172.21.15.29:0/1662843024 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fef401a6d10 con 0x7fef4010e4e0 2024-03-16T11:59:22.818 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.809+0000 7fef3e7fc640 1 -- 172.21.15.29:0/1662843024 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7fef300051d0 con 0x7fef4010e4e0 2024-03-16T11:59:22.818 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.810+0000 7fef4a26a740 1 Processor -- start 2024-03-16T11:59:22.818 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.810+0000 7fef4a26a740 1 -- start start 2024-03-16T11:59:22.818 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.810+0000 7fef4a26a740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5598151ab130 0x5598151ab510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:59:22.818 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.810+0000 7fef4a26a740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5598151aba50 0x5598151b42e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:59:22.818 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.810+0000 7fef3e7fc640 1 --2- 172.21.15.29:0/1662843024 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7fef280a2a90 0x7fef280a4f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:59:22.818 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.810+0000 7fef4a26a740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5598151b4820 0x5598151b6c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:59:22.819 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.810+0000 7fef4a26a740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55981510fa40 con 0x5598151aba50 2024-03-16T11:59:22.819 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.810+0000 7fef4a26a740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x5598150b8920 con 0x5598151ab130 2024-03-16T11:59:22.819 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.810+0000 7fef4a26a740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55981510bae0 con 0x5598151b4820 2024-03-16T11:59:22.819 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.810+0000 7fef45423640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5598151aba50 0x5598151b42e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:59:22.819 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.810+0000 7fef44c22640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5598151ab130 0x5598151ab510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:59:22.819 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.810+0000 7fef45423640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5598151aba50 0x5598151b42e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46292/0 (socket says 172.21.15.29:46292) 2024-03-16T11:59:22.819 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.810+0000 7fef45423640 1 -- 172.21.15.29:0/3051951325 learned_addr learned my addr 172.21.15.29:0/3051951325 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:59:22.819 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.810+0000 7fef44c22640 1 --2- 172.21.15.29:0/1662843024 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7fef280a2a90 0x7fef280a4f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:59:22.819 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.810+0000 7fef45c24640 1 --2- 172.21.15.29:0/3051951325 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5598151b4820 0x5598151b6c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:59:22.819 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.810+0000 7fef3e7fc640 1 -- 172.21.15.29:0/1662843024 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 8556+0+0 (secure 0 0 0) 0x7fef300cf490 con 0x7fef4010e4e0 2024-03-16T11:59:22.819 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.811+0000 7fef45423640 1 -- 172.21.15.29:0/3051951325 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5598151b4820 msgr2=0x5598151b6c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:59:22.819 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.811+0000 7fef45423640 1 --2- 172.21.15.29:0/3051951325 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5598151b4820 0x5598151b6c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:59:22.820 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.811+0000 7fef45423640 1 -- 172.21.15.29:0/3051951325 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5598151ab130 msgr2=0x5598151ab510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:59:22.820 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.811+0000 7fef45423640 1 --2- 172.21.15.29:0/3051951325 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5598151ab130 0x5598151ab510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:59:22.820 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.811+0000 7fef45423640 1 -- 172.21.15.29:0/3051951325 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5598150d0460 con 0x5598151aba50 2024-03-16T11:59:22.820 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.811+0000 7fef45423640 1 --2- 172.21.15.29:0/3051951325 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5598151aba50 0x5598151b42e0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7fef38009ca0 tx=0x7fef38011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2866c64e452fe949 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:59:22.820 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.811+0000 7fef44c22640 1 --2- 172.21.15.29:0/1662843024 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7fef280a2a90 0x7fef280a4f50 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fef401003c0 tx=0x7fef34005bc0 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:59:22.820 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.811+0000 7fef277fe640 1 -- 172.21.15.29:0/3051951325 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef38004220 con 0x5598151aba50 2024-03-16T11:59:22.820 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.811+0000 7fef277fe640 1 -- 172.21.15.29:0/3051951325 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fef380043c0 con 0x5598151aba50 2024-03-16T11:59:22.820 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.811+0000 7fef277fe640 1 -- 172.21.15.29:0/3051951325 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef380198b0 con 0x5598151aba50 2024-03-16T11:59:22.820 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.812+0000 7fef4a26a740 1 -- 172.21.15.29:0/3051951325 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5598151aba50 msgr2=0x5598151b42e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:59:22.820 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.812+0000 7fef4a26a740 1 --2- 172.21.15.29:0/3051951325 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5598151aba50 0x5598151b42e0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7fef38009ca0 tx=0x7fef38011810 comp rx=0 tx=0).stop 2024-03-16T11:59:22.820 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.812+0000 7fef4a26a740 1 -- 172.21.15.29:0/3051951325 shutdown_connections 2024-03-16T11:59:22.820 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.812+0000 7fef4a26a740 1 --2- 172.21.15.29:0/3051951325 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5598151b4820 0x5598151b6c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:59:22.820 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.812+0000 7fef4a26a740 1 --2- 172.21.15.29:0/3051951325 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5598151aba50 0x5598151b42e0 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:59:22.821 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.812+0000 7fef4a26a740 1 --2- 172.21.15.29:0/3051951325 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5598151ab130 0x5598151ab510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:59:22.821 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.812+0000 7fef4a26a740 1 -- 172.21.15.29:0/3051951325 >> 172.21.15.29:0/3051951325 conn(0x5598151a5a50 msgr2=0x5598151a5e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:59:22.821 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.812+0000 7fef4a26a740 1 -- 172.21.15.29:0/3051951325 shutdown_connections 2024-03-16T11:59:22.821 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.812+0000 7fef4a26a740 1 -- 172.21.15.29:0/3051951325 wait complete. 2024-03-16T11:59:22.821 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.812+0000 7fef4a26a740 1 -- start start 2024-03-16T11:59:22.821 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.812+0000 7fef4a26a740 1 Processor -- start 2024-03-16T11:59:22.821 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.812+0000 7fef4a26a740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5598151ab130 0x5598151b1810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:59:22.821 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.812+0000 7fef4a26a740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5598151aba50 0x5598151b1d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:59:22.821 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.812+0000 7fef4a26a740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5598151b4820 0x5598151b2290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:59:22.821 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.812+0000 7fef4a26a740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x5598151b7450 con 0x5598151aba50 2024-03-16T11:59:22.821 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.812+0000 7fef4a26a740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x5598150cbd70 con 0x5598151b4820 2024-03-16T11:59:22.821 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.812+0000 7fef4a26a740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5598151b75d0 con 0x5598151ab130 2024-03-16T11:59:22.822 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.813+0000 7fef4a26a740 10 client.0 _ll_register_callbacks cb 0x559814e23a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T11:59:22.822 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.813+0000 7fef44c22640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5598151aba50 0x5598151b1d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:59:22.822 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.813+0000 7fef45423640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5598151ab130 0x5598151b1810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:59:22.822 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.813+0000 7fef44c22640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5598151aba50 0x5598151b1d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46298/0 (socket says 172.21.15.29:46298) 2024-03-16T11:59:22.822 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.813+0000 7fef44c22640 1 -- 172.21.15.29:0/2588450069 learned_addr learned my addr 172.21.15.29:0/2588450069 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:59:22.822 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.813+0000 7fef45423640 1 --2- 172.21.15.29:0/2588450069 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5598151ab130 0x5598151b1810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.29:46766/0 (socket says 172.21.15.29:46766) 2024-03-16T11:59:22.822 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.813+0000 7fef45c24640 1 --2- 172.21.15.29:0/2588450069 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5598151b4820 0x5598151b2290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:59:22.822 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.813+0000 7fef44c22640 1 -- 172.21.15.29:0/2588450069 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5598151ab130 msgr2=0x5598151b1810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:59:22.822 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.813+0000 7fef44c22640 1 --2- 172.21.15.29:0/2588450069 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5598151ab130 0x5598151b1810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:59:22.822 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.813+0000 7fef44c22640 1 -- 172.21.15.29:0/2588450069 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5598151b4820 msgr2=0x5598151b2290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:59:22.822 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.813+0000 7fef44c22640 1 --2- 172.21.15.29:0/2588450069 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5598151b4820 0x5598151b2290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:59:22.822 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.813+0000 7fef44c22640 1 -- 172.21.15.29:0/2588450069 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559814e24210 con 0x5598151aba50 2024-03-16T11:59:22.823 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.813+0000 7fef45423640 1 --2- 172.21.15.29:0/2588450069 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5598151ab130 0x5598151b1810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:59:22.823 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.813+0000 7fef44c22640 1 --2- 172.21.15.29:0/2588450069 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5598151aba50 0x5598151b1d50 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7fef34009430 tx=0x7fef34005b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c3961dd05abc614 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:59:22.823 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.813+0000 7fef24ff9640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-16T11:59:22.823 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.813+0000 7fef24ff9640 1 -- 172.21.15.29:0/2588450069 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef34052020 con 0x5598151aba50 2024-03-16T11:59:22.823 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.813+0000 7fef24ff9640 1 -- 172.21.15.29:0/2588450069 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8556+0+0 (secure 0 0 0) 0x7fef3403a040 con 0x5598151aba50 2024-03-16T11:59:22.823 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.813+0000 7fef4a26a740 1 -- 172.21.15.29:0/2588450069 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fef380043c0 con 0x5598151aba50 2024-03-16T11:59:22.823 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.813+0000 7fef24ff9640 1 -- 172.21.15.29:0/2588450069 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fef3405d050 con 0x5598151aba50 2024-03-16T11:59:22.823 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.813+0000 7fef4a26a740 10 client.16104 fetch_fsmap learned FSMap version 26 2024-03-16T11:59:22.823 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.814+0000 7fef4a26a740 1 -- 172.21.15.29:0/2588450069 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x559814ef0620 con 0x5598151aba50 2024-03-16T11:59:22.823 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.814+0000 7fef24ff9640 1 -- 172.21.15.29:0/2588450069 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fef380043c0 con 0x5598151aba50 2024-03-16T11:59:22.823 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.814+0000 7fef24ff9640 1 -- 172.21.15.29:0/2588450069 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7fef34003b20 con 0x5598151aba50 2024-03-16T11:59:22.823 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.814+0000 7fef4a26a740 10 client.16104 fetch_fsmap finished waiting for FSMap version 26 2024-03-16T11:59:22.824 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.814+0000 7fef4a26a740 10 client.16104 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T11:59:22.824 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.814+0000 7fef4a26a740 20 client.16104 populate_metadata read hostname 'smithi029' 2024-03-16T11:59:22.824 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.814+0000 7fef4a26a740 1 --2- 172.21.15.29:0/2588450069 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x5598151c4950 0x5598151c6de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:59:22.824 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.814+0000 7fef4a26a740 4 client.16104 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T11:59:22.824 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.814+0000 7fef4a26a740 1 -- 172.21.15.29:0/2588450069 --> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55981511cf80 con 0x5598151c4950 2024-03-16T11:59:22.824 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.814+0000 7fef45423640 1 --2- 172.21.15.29:0/2588450069 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x5598151c4950 0x5598151c6de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:59:22.824 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.814+0000 7fef24ff9640 10 client.16104 ms_handle_connect on v2:172.21.15.152:6838/1548731833 2024-03-16T11:59:22.824 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.814+0000 7fef45423640 1 --2- 172.21.15.29:0/2588450069 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x5598151c4950 0x5598151c6de0 crc :-1 s=READY pgs=114 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=27dcd8f788de0a3b server_cookie=221240fd3088e661 in_seq=0 out_seq=0 2024-03-16T11:59:22.824 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.817+0000 7fef24ff9640 1 -- 172.21.15.29:0/2588450069 <== mds.0 v2:172.21.15.152:6838/1548731833 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55981511cf80 con 0x5598151c4950 2024-03-16T11:59:22.824 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.817+0000 7fef24ff9640 10 client.16104 handle_command_reply: tid=0 2024-03-16T11:59:22.824 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.817+0000 7fef4a26a740 1 client.16104 shutdown 2024-03-16T11:59:22.824 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.817+0000 7fef4a26a740 1 -- 172.21.15.29:0/2588450069 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5598151aba50 msgr2=0x5598151b1d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:59:22.825 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.817+0000 7fef4a26a740 1 --2- 172.21.15.29:0/2588450069 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5598151aba50 0x5598151b1d50 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7fef34009430 tx=0x7fef34005b50 comp rx=0 tx=0).stop 2024-03-16T11:59:22.825 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.817+0000 7fef4a26a740 1 -- 172.21.15.29:0/2588450069 shutdown_connections 2024-03-16T11:59:22.825 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.817+0000 7fef4a26a740 1 --2- 172.21.15.29:0/2588450069 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x5598151c4950 0x5598151c6de0 crc :-1 s=READY pgs=114 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:59:22.825 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.817+0000 7fef4a26a740 1 --2- 172.21.15.29:0/2588450069 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5598151b4820 0x5598151b2290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:59:22.825 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.817+0000 7fef4a26a740 1 --2- 172.21.15.29:0/2588450069 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5598151aba50 0x5598151b1d50 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:59:22.825 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.817+0000 7fef4a26a740 1 --2- 172.21.15.29:0/2588450069 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5598151ab130 0x5598151b1810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:59:22.825 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.817+0000 7fef24ff9640 0 client.16104 ms_handle_reset on v2:172.21.15.152:6838/1548731833 2024-03-16T11:59:22.825 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.817+0000 7fef4a26a740 1 -- 172.21.15.29:0/2588450069 >> 172.21.15.29:0/2588450069 conn(0x5598151a5a50 msgr2=0x5598151a77a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:59:22.825 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.817+0000 7fef4a26a740 1 -- 172.21.15.29:0/2588450069 shutdown_connections 2024-03-16T11:59:22.825 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.817+0000 7fef4a26a740 1 -- 172.21.15.29:0/2588450069 wait complete. 2024-03-16T11:59:22.825 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.817+0000 7fef4a26a740 20 client.16104 trim_cache size 0 max 16384 2024-03-16T11:59:22.838 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.833+0000 7fef4a26a740 1 Processor -- start 2024-03-16T11:59:22.838 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.833+0000 7fef4a26a740 1 -- start start 2024-03-16T11:59:22.838 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.833+0000 7fef4a26a740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5598151ab130 0x5598151c01f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:59:22.838 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.833+0000 7fef4a26a740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5598151aba50 0x5598151cc6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:59:22.838 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.833+0000 7fef4a26a740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5598151b4820 0x5598151b17d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:59:22.838 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.833+0000 7fef4a26a740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fef401b1fb0 con 0x5598151ab130 2024-03-16T11:59:22.838 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.833+0000 7fef4a26a740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fef401af880 con 0x5598151b4820 2024-03-16T11:59:22.839 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.833+0000 7fef4a26a740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fef401ad6f0 con 0x5598151aba50 2024-03-16T11:59:22.839 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.833+0000 7fef45423640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5598151ab130 0x5598151c01f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:59:22.839 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.833+0000 7fef45423640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5598151ab130 0x5598151c01f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46310/0 (socket says 172.21.15.29:46310) 2024-03-16T11:59:22.839 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.833+0000 7fef45423640 1 -- 172.21.15.29:0/178807514 learned_addr learned my addr 172.21.15.29:0/178807514 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:59:22.839 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.833+0000 7fef44c22640 1 --2- 172.21.15.29:0/178807514 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5598151aba50 0x5598151cc6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:59:22.839 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.833+0000 7fef45c24640 1 --2- 172.21.15.29:0/178807514 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5598151b4820 0x5598151b17d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:59:22.839 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.833+0000 7fef45423640 1 -- 172.21.15.29:0/178807514 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5598151aba50 msgr2=0x5598151cc6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:59:22.839 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.833+0000 7fef45423640 1 --2- 172.21.15.29:0/178807514 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5598151aba50 0x5598151cc6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:59:22.839 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.833+0000 7fef45423640 1 -- 172.21.15.29:0/178807514 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5598151b4820 msgr2=0x5598151b17d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:59:22.839 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.833+0000 7fef45423640 1 --2- 172.21.15.29:0/178807514 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5598151b4820 0x5598151b17d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:59:22.839 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.833+0000 7fef45423640 1 -- 172.21.15.29:0/178807514 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5598151c0730 con 0x5598151ab130 2024-03-16T11:59:22.839 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.833+0000 7fef45c24640 1 --2- 172.21.15.29:0/178807514 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5598151b4820 0x5598151b17d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T11:59:22.840 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.834+0000 7fef45423640 1 --2- 172.21.15.29:0/178807514 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5598151ab130 0x5598151c01f0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7fef380099b0 tx=0x7fef38009310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18c58ac1bf1bbd24 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:59:22.840 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.834+0000 7fef26ffd640 1 -- 172.21.15.29:0/178807514 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef3802cc40 con 0x5598151ab130 2024-03-16T11:59:22.840 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.834+0000 7fef26ffd640 1 -- 172.21.15.29:0/178807514 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fef3802c7f0 con 0x5598151ab130 2024-03-16T11:59:22.840 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.834+0000 7fef26ffd640 1 -- 172.21.15.29:0/178807514 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef380055a0 con 0x5598151ab130 2024-03-16T11:59:22.840 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.834+0000 7fef4a26a740 1 -- 172.21.15.29:0/178807514 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5598151ab130 msgr2=0x5598151c01f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:59:22.840 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.834+0000 7fef4a26a740 1 --2- 172.21.15.29:0/178807514 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5598151ab130 0x5598151c01f0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7fef380099b0 tx=0x7fef38009310 comp rx=0 tx=0).stop 2024-03-16T11:59:22.840 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.834+0000 7fef4a26a740 1 -- 172.21.15.29:0/178807514 shutdown_connections 2024-03-16T11:59:22.840 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.834+0000 7fef4a26a740 1 --2- 172.21.15.29:0/178807514 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5598151b4820 0x5598151b17d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:59:22.840 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.834+0000 7fef4a26a740 1 --2- 172.21.15.29:0/178807514 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5598151aba50 0x5598151cc6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:59:22.840 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.834+0000 7fef4a26a740 1 --2- 172.21.15.29:0/178807514 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5598151ab130 0x5598151c01f0 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:59:22.840 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.834+0000 7fef4a26a740 1 -- 172.21.15.29:0/178807514 >> 172.21.15.29:0/178807514 conn(0x5598151a5a50 msgr2=0x5598151acba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:59:22.840 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.834+0000 7fef4a26a740 1 -- 172.21.15.29:0/178807514 shutdown_connections 2024-03-16T11:59:22.841 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.834+0000 7fef4a26a740 1 -- 172.21.15.29:0/178807514 wait complete. 2024-03-16T11:59:22.841 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.835+0000 7fef4a26a740 1 -- start start 2024-03-16T11:59:22.841 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.835+0000 7fef4a26a740 1 Processor -- start 2024-03-16T11:59:22.841 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.835+0000 7fef4a26a740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5598151ab130 0x5598151bcbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:59:22.841 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.835+0000 7fef4a26a740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5598151aba50 0x5598151ae1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:59:22.841 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.835+0000 7fef4a26a740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5598151b4820 0x5598151ae820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:59:22.841 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.835+0000 7fef4a26a740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fef401add40 con 0x5598151aba50 2024-03-16T11:59:22.841 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.835+0000 7fef4a26a740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fef401af1b0 con 0x5598151ab130 2024-03-16T11:59:22.841 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.835+0000 7fef4a26a740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fef401ae410 con 0x5598151b4820 2024-03-16T11:59:22.841 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.835+0000 7fef4a26a740 10 client.0 _ll_register_callbacks cb 0x7fef34009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T11:59:22.841 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.835+0000 7fef44c22640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5598151aba50 0x5598151ae1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:59:22.841 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.835+0000 7fef44c22640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5598151aba50 0x5598151ae1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46312/0 (socket says 172.21.15.29:46312) 2024-03-16T11:59:22.841 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.835+0000 7fef44c22640 1 -- 172.21.15.29:0/3063202932 learned_addr learned my addr 172.21.15.29:0/3063202932 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:59:22.842 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.835+0000 7fef45423640 1 --2- 172.21.15.29:0/3063202932 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5598151ab130 0x5598151bcbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:59:22.842 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.835+0000 7fef45c24640 1 --2- 172.21.15.29:0/3063202932 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5598151b4820 0x5598151ae820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:59:22.842 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.836+0000 7fef44c22640 1 -- 172.21.15.29:0/3063202932 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5598151b4820 msgr2=0x5598151ae820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:59:22.842 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.836+0000 7fef44c22640 1 --2- 172.21.15.29:0/3063202932 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5598151b4820 0x5598151ae820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:59:22.842 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.836+0000 7fef44c22640 1 -- 172.21.15.29:0/3063202932 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5598151ab130 msgr2=0x5598151bcbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:59:22.842 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.836+0000 7fef44c22640 1 --2- 172.21.15.29:0/3063202932 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5598151ab130 0x5598151bcbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:59:22.842 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.836+0000 7fef44c22640 1 -- 172.21.15.29:0/3063202932 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5598151aeff0 con 0x5598151aba50 2024-03-16T11:59:22.842 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.836+0000 7fef44c22640 1 --2- 172.21.15.29:0/3063202932 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5598151aba50 0x5598151ae1f0 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7fef34003e90 tx=0x7fef340038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4fb6c3ab90f6aa70 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:59:22.842 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.836+0000 7fef24ff9640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-16T11:59:22.842 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.836+0000 7fef24ff9640 1 -- 172.21.15.29:0/3063202932 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef34052040 con 0x5598151aba50 2024-03-16T11:59:22.842 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.836+0000 7fef24ff9640 1 -- 172.21.15.29:0/3063202932 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8556+0+0 (secure 0 0 0) 0x7fef34003a90 con 0x5598151aba50 2024-03-16T11:59:22.842 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.836+0000 7fef4a26a740 1 -- 172.21.15.29:0/3063202932 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fef3802c7f0 con 0x5598151aba50 2024-03-16T11:59:22.843 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.836+0000 7fef24ff9640 1 -- 172.21.15.29:0/3063202932 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fef34004410 con 0x5598151aba50 2024-03-16T11:59:22.843 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.836+0000 7fef4a26a740 10 client.16110 fetch_fsmap learned FSMap version 26 2024-03-16T11:59:22.843 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.836+0000 7fef4a26a740 1 -- 172.21.15.29:0/3063202932 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55981522ae30 con 0x5598151aba50 2024-03-16T11:59:22.843 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.836+0000 7fef24ff9640 1 -- 172.21.15.29:0/3063202932 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fef3802c7f0 con 0x5598151aba50 2024-03-16T11:59:22.843 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.836+0000 7fef24ff9640 1 -- 172.21.15.29:0/3063202932 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7fef3404d650 con 0x5598151aba50 2024-03-16T11:59:22.843 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.836+0000 7fef4a26a740 10 client.16110 fetch_fsmap finished waiting for FSMap version 26 2024-03-16T11:59:22.843 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.836+0000 7fef4a26a740 10 client.16110 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T11:59:22.843 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.836+0000 7fef4a26a740 20 client.16110 populate_metadata read hostname 'smithi029' 2024-03-16T11:59:22.843 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.836+0000 7fef4a26a740 1 --2- 172.21.15.29:0/3063202932 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55981522b570 0x55981522d990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:59:22.843 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.836+0000 7fef4a26a740 4 client.16110 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-16T11:59:22.843 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.836+0000 7fef4a26a740 1 -- 172.21.15.29:0/3063202932 --> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5598151172d0 con 0x55981522b570 2024-03-16T11:59:22.843 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.837+0000 7fef45423640 1 --2- 172.21.15.29:0/3063202932 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55981522b570 0x55981522d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:59:22.843 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.837+0000 7fef24ff9640 10 client.16110 ms_handle_connect on v2:172.21.15.152:6838/1548731833 2024-03-16T11:59:22.844 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.837+0000 7fef45423640 1 --2- 172.21.15.29:0/3063202932 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55981522b570 0x55981522d990 crc :-1 s=READY pgs=115 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=aed72b6b256d0e76 server_cookie=f474542030b0bfc8 in_seq=0 out_seq=0 2024-03-16T11:59:22.996 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.995+0000 7fef24ff9640 1 -- 172.21.15.29:0/3063202932 <== mds.0 v2:172.21.15.152:6838/1548731833 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5598151172d0 con 0x55981522b570 2024-03-16T11:59:22.997 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.995+0000 7fef24ff9640 10 client.16110 handle_command_reply: tid=0 2024-03-16T11:59:22.997 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.995+0000 7fef4a26a740 1 client.16110 shutdown 2024-03-16T11:59:22.997 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.995+0000 7fef4a26a740 1 -- 172.21.15.29:0/3063202932 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5598151aba50 msgr2=0x5598151ae1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:59:22.997 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.995+0000 7fef4a26a740 1 --2- 172.21.15.29:0/3063202932 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5598151aba50 0x5598151ae1f0 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7fef34003e90 tx=0x7fef340038a0 comp rx=0 tx=0).stop 2024-03-16T11:59:22.997 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.995+0000 7fef4a26a740 1 -- 172.21.15.29:0/3063202932 shutdown_connections 2024-03-16T11:59:22.997 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.996+0000 7fef4a26a740 1 --2- 172.21.15.29:0/3063202932 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5598151b4820 0x5598151ae820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:59:22.997 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.996+0000 7fef4a26a740 1 --2- 172.21.15.29:0/3063202932 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5598151aba50 0x5598151ae1f0 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:59:22.997 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.996+0000 7fef4a26a740 1 --2- 172.21.15.29:0/3063202932 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55981522b570 0x55981522d990 crc :-1 s=READY pgs=115 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:59:22.997 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.996+0000 7fef4a26a740 1 --2- 172.21.15.29:0/3063202932 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5598151ab130 0x5598151bcbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:59:22.997 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.996+0000 7fef4a26a740 1 -- 172.21.15.29:0/3063202932 >> 172.21.15.29:0/3063202932 conn(0x5598151a5a50 msgr2=0x5598151b2e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:59:22.997 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.996+0000 7fef24ff9640 0 client.16110 ms_handle_reset on v2:172.21.15.152:6838/1548731833 2024-03-16T11:59:22.997 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.996+0000 7fef4a26a740 1 -- 172.21.15.29:0/3063202932 shutdown_connections 2024-03-16T11:59:22.997 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.996+0000 7fef4a26a740 1 -- 172.21.15.29:0/3063202932 wait complete. 2024-03-16T11:59:22.998 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.996+0000 7fef4a26a740 20 client.16110 trim_cache size 0 max 16384 2024-03-16T11:59:22.998 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:{ 2024-03-16T11:59:22.998 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "message": "", 2024-03-16T11:59:22.998 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "return_code": 0 2024-03-16T11:59:22.998 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:} 2024-03-16T11:59:22.999 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.997+0000 7fef47687640 1 -- 172.21.15.29:0/1662843024 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7fef280a2a90 msgr2=0x7fef280a4f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:59:22.999 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.997+0000 7fef47687640 1 --2- 172.21.15.29:0/1662843024 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7fef280a2a90 0x7fef280a4f50 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fef401003c0 tx=0x7fef34005bc0 comp rx=0 tx=0).stop 2024-03-16T11:59:22.999 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.997+0000 7fef47687640 1 -- 172.21.15.29:0/1662843024 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fef4010e4e0 msgr2=0x7fef4019ea60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:59:22.999 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.997+0000 7fef47687640 1 --2- 172.21.15.29:0/1662843024 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fef4010e4e0 0x7fef4019ea60 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7fef40069b60 tx=0x7fef3000ee60 comp rx=0 tx=0).stop 2024-03-16T11:59:22.999 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.997+0000 7fef47687640 1 -- 172.21.15.29:0/1662843024 shutdown_connections 2024-03-16T11:59:22.999 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.997+0000 7fef47687640 1 --2- 172.21.15.29:0/1662843024 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7fef280a2a90 0x7fef280a4f50 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:59:22.999 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.997+0000 7fef47687640 1 --2- 172.21.15.29:0/1662843024 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fef4010e4e0 0x7fef4019ea60 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:59:22.999 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.997+0000 7fef47687640 1 --2- 172.21.15.29:0/1662843024 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fef4006a920 0x7fef4019a6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:59:22.999 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.997+0000 7fef47687640 1 --2- 172.21.15.29:0/1662843024 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fef4006a000 0x7fef4019a1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:59:23.000 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.997+0000 7fef47687640 1 -- 172.21.15.29:0/1662843024 >> 172.21.15.29:0/1662843024 conn(0x7fef40077540 msgr2=0x7fef4010f840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:59:23.000 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.997+0000 7fef47687640 1 -- 172.21.15.29:0/1662843024 shutdown_connections 2024-03-16T11:59:23.000 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:22.997+0000 7fef47687640 1 -- 172.21.15.29:0/1662843024 wait complete. 2024-03-16T11:59:23.795 INFO:tasks.workunit.client.0.smithi029.stdout: 1 856836 10.30 MB/sec execute 308 sec latency 104.913 ms 2024-03-16T11:59:24.337 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:59:24 smithi029 ceph-mon[31975]: pgmap v266: 97 pgs: 97 active+clean; 83 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 2.8 MiB/s wr, 201 op/s 2024-03-16T11:59:24.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:59:24 smithi152 ceph-mon[36291]: pgmap v266: 97 pgs: 97 active+clean; 83 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 2.8 MiB/s wr, 201 op/s 2024-03-16T11:59:24.545 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:59:24 smithi154 ceph-mon[36320]: pgmap v266: 97 pgs: 97 active+clean; 83 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 2.8 MiB/s wr, 201 op/s 2024-03-16T11:59:24.795 INFO:tasks.workunit.client.0.smithi029.stdout: 1 859206 10.31 MB/sec execute 309 sec latency 17.647 ms 2024-03-16T11:59:25.795 INFO:tasks.workunit.client.0.smithi029.stdout: 1 860627 10.30 MB/sec execute 310 sec latency 30.376 ms 2024-03-16T11:59:26.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:59:26 smithi029 ceph-mon[31975]: pgmap v267: 97 pgs: 97 active+clean; 83 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 2.8 MiB/s wr, 201 op/s 2024-03-16T11:59:26.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:59:26 smithi152 ceph-mon[36291]: pgmap v267: 97 pgs: 97 active+clean; 83 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 2.8 MiB/s wr, 201 op/s 2024-03-16T11:59:26.545 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:59:26 smithi154 ceph-mon[36320]: pgmap v267: 97 pgs: 97 active+clean; 83 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 2.8 MiB/s wr, 201 op/s 2024-03-16T11:59:26.795 INFO:tasks.workunit.client.0.smithi029.stdout: 1 862223 10.29 MB/sec execute 311 sec latency 152.198 ms 2024-03-16T11:59:27.795 INFO:tasks.workunit.client.0.smithi029.stdout: 1 863628 10.28 MB/sec execute 312 sec latency 26.280 ms 2024-03-16T11:59:28.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:59:28 smithi152 ceph-mon[36291]: pgmap v268: 97 pgs: 97 active+clean; 68 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 2.7 MiB/s wr, 281 op/s 2024-03-16T11:59:28.545 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:59:28 smithi154 ceph-mon[36320]: pgmap v268: 97 pgs: 97 active+clean; 68 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 2.7 MiB/s wr, 281 op/s 2024-03-16T11:59:28.587 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:59:28 smithi029 ceph-mon[31975]: pgmap v268: 97 pgs: 97 active+clean; 68 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 2.7 MiB/s wr, 281 op/s 2024-03-16T11:59:28.795 INFO:tasks.workunit.client.0.smithi029.stdout: 1 865676 10.29 MB/sec execute 313 sec latency 20.274 ms 2024-03-16T11:59:29.795 INFO:tasks.workunit.client.0.smithi029.stdout: 1 867082 10.28 MB/sec execute 314 sec latency 22.327 ms 2024-03-16T11:59:30.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:59:30 smithi152 ceph-mon[36291]: pgmap v269: 97 pgs: 97 active+clean; 68 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 2.2 MiB/s wr, 236 op/s 2024-03-16T11:59:30.545 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:59:30 smithi154 ceph-mon[36320]: pgmap v269: 97 pgs: 97 active+clean; 68 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 2.2 MiB/s wr, 236 op/s 2024-03-16T11:59:30.587 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:59:30 smithi029 ceph-mon[31975]: pgmap v269: 97 pgs: 97 active+clean; 68 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 2.2 MiB/s wr, 236 op/s 2024-03-16T11:59:30.795 INFO:tasks.workunit.client.0.smithi029.stdout: 1 869070 10.28 MB/sec execute 315 sec latency 19.739 ms 2024-03-16T11:59:31.795 INFO:tasks.workunit.client.0.smithi029.stdout: 1 870502 10.27 MB/sec execute 316 sec latency 52.580 ms 2024-03-16T11:59:32.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:59:32 smithi152 ceph-mon[36291]: pgmap v270: 97 pgs: 97 active+clean; 74 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 3.4 MiB/s wr, 313 op/s 2024-03-16T11:59:32.545 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:59:32 smithi154 ceph-mon[36320]: pgmap v270: 97 pgs: 97 active+clean; 74 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 3.4 MiB/s wr, 313 op/s 2024-03-16T11:59:32.588 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:59:32 smithi029 ceph-mon[31975]: pgmap v270: 97 pgs: 97 active+clean; 74 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 3.4 MiB/s wr, 313 op/s 2024-03-16T11:59:32.795 INFO:tasks.workunit.client.0.smithi029.stdout: 1 872279 10.26 MB/sec execute 317 sec latency 34.440 ms 2024-03-16T11:59:33.796 INFO:tasks.workunit.client.0.smithi029.stdout: 1 873983 10.27 MB/sec execute 318 sec latency 14.664 ms 2024-03-16T11:59:34.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:59:34 smithi152 ceph-mon[36291]: pgmap v271: 97 pgs: 97 active+clean; 74 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 2.5 MiB/s wr, 240 op/s 2024-03-16T11:59:34.545 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:59:34 smithi154 ceph-mon[36320]: pgmap v271: 97 pgs: 97 active+clean; 74 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 2.5 MiB/s wr, 240 op/s 2024-03-16T11:59:34.590 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:59:34 smithi029 ceph-mon[31975]: pgmap v271: 97 pgs: 97 active+clean; 74 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 2.5 MiB/s wr, 240 op/s 2024-03-16T11:59:34.795 INFO:tasks.workunit.client.0.smithi029.stdout: 1 875875 10.26 MB/sec execute 319 sec latency 32.283 ms 2024-03-16T11:59:35.795 INFO:tasks.workunit.client.0.smithi029.stdout: 1 877650 10.26 MB/sec execute 320 sec latency 22.022 ms 2024-03-16T11:59:36.095 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:36.093+0000 7ff95e13b640 1 -- 172.21.15.29:0/1043492454 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff958101810 msgr2=0x7ff95810d520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:59:36.095 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:36.093+0000 7ff95e13b640 1 --2- 172.21.15.29:0/1043492454 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff958101810 0x7ff95810d520 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7ff950009ca0 tx=0x7ff95002c0f0 comp rx=0 tx=0).stop 2024-03-16T11:59:36.095 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:36.093+0000 7ff95e13b640 1 -- 172.21.15.29:0/1043492454 shutdown_connections 2024-03-16T11:59:36.095 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:36.093+0000 7ff95e13b640 1 --2- 172.21.15.29:0/1043492454 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff958101810 0x7ff95810d520 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:59:36.095 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:36.093+0000 7ff95e13b640 1 --2- 172.21.15.29:0/1043492454 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff958100e50 0x7ff9581012d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:59:36.095 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:36.093+0000 7ff95e13b640 1 --2- 172.21.15.29:0/1043492454 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff958105690 0x7ff958105a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:59:36.095 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:36.093+0000 7ff95e13b640 1 -- 172.21.15.29:0/1043492454 >> 172.21.15.29:0/1043492454 conn(0x7ff958076c70 msgr2=0x7ff9580fdff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:59:36.095 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:36.093+0000 7ff95e13b640 1 -- 172.21.15.29:0/1043492454 shutdown_connections 2024-03-16T11:59:36.095 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:36.093+0000 7ff95e13b640 1 -- 172.21.15.29:0/1043492454 wait complete. 2024-03-16T11:59:36.096 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:36.094+0000 7ff95e13b640 1 Processor -- start 2024-03-16T11:59:36.096 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:36.094+0000 7ff95e13b640 1 -- start start 2024-03-16T11:59:36.096 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:36.094+0000 7ff95e13b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff958100e50 0x7ff958197ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:59:36.096 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:36.094+0000 7ff95e13b640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff958101810 0x7ff958198420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:59:36.096 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:36.094+0000 7ff95e13b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff958105690 0x7ff95819c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:59:36.096 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:36.094+0000 7ff95e13b640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7ff958110a90 con 0x7ff958100e50 2024-03-16T11:59:36.096 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:36.094+0000 7ff95e13b640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7ff958110910 con 0x7ff958101810 2024-03-16T11:59:36.096 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:36.094+0000 7ff95e13b640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7ff958110c10 con 0x7ff958105690 2024-03-16T11:59:36.096 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:36.094+0000 7ff9577fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff958100e50 0x7ff958197ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:59:36.096 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:36.094+0000 7ff9577fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff958100e50 0x7ff958197ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50154/0 (socket says 172.21.15.29:50154) 2024-03-16T11:59:36.096 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:36.094+0000 7ff9577fe640 1 -- 172.21.15.29:0/3009795496 learned_addr learned my addr 172.21.15.29:0/3009795496 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:59:36.097 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:36.094+0000 7ff956ffd640 1 --2- 172.21.15.29:0/3009795496 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff958101810 0x7ff958198420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:59:36.097 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:36.094+0000 7ff957fff640 1 --2- 172.21.15.29:0/3009795496 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff958105690 0x7ff95819c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:59:36.097 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:36.094+0000 7ff9577fe640 1 -- 172.21.15.29:0/3009795496 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff958105690 msgr2=0x7ff95819c7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:59:36.097 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:36.094+0000 7ff9577fe640 1 --2- 172.21.15.29:0/3009795496 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff958105690 0x7ff95819c7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:59:36.097 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:36.094+0000 7ff9577fe640 1 -- 172.21.15.29:0/3009795496 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff958101810 msgr2=0x7ff958198420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:59:36.097 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:36.094+0000 7ff9577fe640 1 --2- 172.21.15.29:0/3009795496 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff958101810 0x7ff958198420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:59:36.097 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:36.094+0000 7ff9577fe640 1 -- 172.21.15.29:0/3009795496 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff95819ced0 con 0x7ff958100e50 2024-03-16T11:59:36.097 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:36.094+0000 7ff957fff640 1 --2- 172.21.15.29:0/3009795496 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff958105690 0x7ff95819c7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T11:59:36.097 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:36.095+0000 7ff9577fe640 1 --2- 172.21.15.29:0/3009795496 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff958100e50 0x7ff958197ee0 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7ff94800e9f0 tx=0x7ff94800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:59:36.097 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:36.095+0000 7ff954ff9640 1 -- 172.21.15.29:0/3009795496 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff94800cd80 con 0x7ff958100e50 2024-03-16T11:59:36.097 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:36.095+0000 7ff95e13b640 1 -- 172.21.15.29:0/3009795496 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff95819d1c0 con 0x7ff958100e50 2024-03-16T11:59:36.097 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:36.095+0000 7ff954ff9640 1 -- 172.21.15.29:0/3009795496 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff948010910 con 0x7ff958100e50 2024-03-16T11:59:36.097 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:36.095+0000 7ff954ff9640 1 -- 172.21.15.29:0/3009795496 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9480188f0 con 0x7ff958100e50 2024-03-16T11:59:36.097 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:36.095+0000 7ff95e13b640 1 -- 172.21.15.29:0/3009795496 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff9581a4a50 con 0x7ff958100e50 2024-03-16T11:59:36.104 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:36.098+0000 7ff960e02740 1 Processor -- start 2024-03-16T11:59:36.104 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:36.098+0000 7ff960e02740 1 -- start start 2024-03-16T11:59:36.104 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:36.098+0000 7ff960e02740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d9f8b0c130 0x55d9f8b0c510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:59:36.104 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:36.098+0000 7ff960e02740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d9f8b0ca50 0x55d9f8b152e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:59:36.104 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:36.098+0000 7ff954ff9640 1 -- 172.21.15.29:0/3009795496 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7ff948010ac0 con 0x7ff958100e50 2024-03-16T11:59:36.104 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:36.098+0000 7ff960e02740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d9f8b15820 0x55d9f8b17c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:59:36.105 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:36.098+0000 7ff960e02740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55d9f8a70a40 con 0x55d9f8b15820 2024-03-16T11:59:36.105 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:36.098+0000 7ff960e02740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x55d9f8a19920 con 0x55d9f8b0c130 2024-03-16T11:59:36.105 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:36.098+0000 7ff960e02740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55d9f8a6cae0 con 0x55d9f8b0ca50 2024-03-16T11:59:36.105 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:36.098+0000 7ff956ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d9f8b15820 0x55d9f8b17c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:59:36.105 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:36.098+0000 7ff957fff640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d9f8b0c130 0x55d9f8b0c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:59:36.105 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:36.098+0000 7ff956ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d9f8b15820 0x55d9f8b17c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50168/0 (socket says 172.21.15.29:50168) 2024-03-16T11:59:36.105 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:36.098+0000 7ff956ffd640 1 -- 172.21.15.29:0/62883821 learned_addr learned my addr 172.21.15.29:0/62883821 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:59:36.105 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:36.098+0000 7ff9577fe640 1 --2- 172.21.15.29:0/62883821 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d9f8b0ca50 0x55d9f8b152e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:59:36.105 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:36.098+0000 7ff957fff640 1 -- 172.21.15.29:0/62883821 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d9f8b0ca50 msgr2=0x55d9f8b152e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:59:36.105 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:36.098+0000 7ff957fff640 1 --2- 172.21.15.29:0/62883821 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d9f8b0ca50 0x55d9f8b152e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:59:36.105 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:36.098+0000 7ff957fff640 1 -- 172.21.15.29:0/62883821 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d9f8b15820 msgr2=0x55d9f8b17c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:59:36.105 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:36.098+0000 7ff957fff640 1 --2- 172.21.15.29:0/62883821 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d9f8b15820 0x55d9f8b17c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:59:36.106 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:36.098+0000 7ff957fff640 1 -- 172.21.15.29:0/62883821 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d9f8a31460 con 0x55d9f8b0c130 2024-03-16T11:59:36.106 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:36.099+0000 7ff954ff9640 1 --2- 172.21.15.29:0/3009795496 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7ff9400a2a10 0x7ff9400a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:59:36.106 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:36.099+0000 7ff957fff640 1 --2- 172.21.15.29:0/62883821 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d9f8b0c130 0x55d9f8b0c510 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7ff95002c700 tx=0x7ff950005fa0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=261e8b69a62bf365 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:59:36.106 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:36.099+0000 7ff9577fe640 1 --2- 172.21.15.29:0/3009795496 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7ff9400a2a10 0x7ff9400a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:59:36.106 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:36.099+0000 7ff954ff9640 1 -- 172.21.15.29:0/3009795496 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 8556+0+0 (secure 0 0 0) 0x7ff9480cf2e0 con 0x7ff958100e50 2024-03-16T11:59:36.106 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:36.099+0000 7ff9577fe640 1 --2- 172.21.15.29:0/3009795496 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7ff9400a2a10 0x7ff9400a4ed0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7ff9480ccac0 tx=0x7ff9480d2e30 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:59:36.106 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:36.100+0000 7ff93dffb640 1 -- 172.21.15.29:0/62883821 <== mon.2 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff950044020 con 0x55d9f8b0c130 2024-03-16T11:59:36.106 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:36.100+0000 7ff93dffb640 1 -- 172.21.15.29:0/62883821 <== mon.2 v2:172.21.15.152:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff950032c90 con 0x55d9f8b0c130 2024-03-16T11:59:36.107 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:36.100+0000 7ff93dffb640 1 -- 172.21.15.29:0/62883821 <== mon.2 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff950039040 con 0x55d9f8b0c130 2024-03-16T11:59:36.107 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:36.101+0000 7ff960e02740 1 -- 172.21.15.29:0/62883821 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d9f8b0c130 msgr2=0x55d9f8b0c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:59:36.107 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:36.101+0000 7ff960e02740 1 --2- 172.21.15.29:0/62883821 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d9f8b0c130 0x55d9f8b0c510 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7ff95002c700 tx=0x7ff950005fa0 comp rx=0 tx=0).stop 2024-03-16T11:59:36.107 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:36.101+0000 7ff960e02740 1 -- 172.21.15.29:0/62883821 shutdown_connections 2024-03-16T11:59:36.107 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:36.101+0000 7ff960e02740 1 --2- 172.21.15.29:0/62883821 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d9f8b15820 0x55d9f8b17c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:59:36.107 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:36.101+0000 7ff960e02740 1 --2- 172.21.15.29:0/62883821 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d9f8b0ca50 0x55d9f8b152e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:59:36.107 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:36.101+0000 7ff960e02740 1 --2- 172.21.15.29:0/62883821 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d9f8b0c130 0x55d9f8b0c510 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:59:36.107 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:36.101+0000 7ff960e02740 1 -- 172.21.15.29:0/62883821 >> 172.21.15.29:0/62883821 conn(0x55d9f8b06a50 msgr2=0x55d9f8b06e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:59:36.107 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:36.101+0000 7ff960e02740 1 -- 172.21.15.29:0/62883821 shutdown_connections 2024-03-16T11:59:36.107 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:36.101+0000 7ff960e02740 1 -- 172.21.15.29:0/62883821 wait complete. 2024-03-16T11:59:36.107 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:36.101+0000 7ff960e02740 1 -- start start 2024-03-16T11:59:36.107 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:36.101+0000 7ff960e02740 1 Processor -- start 2024-03-16T11:59:36.107 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:36.102+0000 7ff960e02740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d9f8b0c130 0x55d9f8b12820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:59:36.107 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:36.102+0000 7ff960e02740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d9f8b0ca50 0x55d9f8b12d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:59:36.108 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:36.102+0000 7ff960e02740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d9f8b15820 0x55d9f8b132a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:59:36.108 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:36.102+0000 7ff960e02740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55d9f8b183f0 con 0x55d9f8b0ca50 2024-03-16T11:59:36.108 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:36.102+0000 7ff960e02740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x55d9f8a2cd70 con 0x55d9f8b0c130 2024-03-16T11:59:36.108 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:36.102+0000 7ff960e02740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55d9f8b18570 con 0x55d9f8b15820 2024-03-16T11:59:36.108 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:36.102+0000 7ff960e02740 10 client.0 _ll_register_callbacks cb 0x55d9f8784a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T11:59:36.108 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:36.102+0000 7ff956ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d9f8b0ca50 0x55d9f8b12d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:59:36.108 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:36.102+0000 7ff956ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d9f8b0ca50 0x55d9f8b12d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50182/0 (socket says 172.21.15.29:50182) 2024-03-16T11:59:36.108 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:36.102+0000 7ff956ffd640 1 -- 172.21.15.29:0/2302004718 learned_addr learned my addr 172.21.15.29:0/2302004718 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:59:36.108 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:36.102+0000 7ff957fff640 1 --2- 172.21.15.29:0/2302004718 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d9f8b0c130 0x55d9f8b12820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:59:36.108 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:36.102+0000 7ff957fff640 1 --2- 172.21.15.29:0/2302004718 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d9f8b15820 0x55d9f8b132a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:59:36.108 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:36.102+0000 7ff956ffd640 1 -- 172.21.15.29:0/2302004718 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d9f8b15820 msgr2=0x55d9f8b132a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:59:36.108 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:36.102+0000 7ff956ffd640 1 --2- 172.21.15.29:0/2302004718 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d9f8b15820 0x55d9f8b132a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:59:36.108 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:36.102+0000 7ff956ffd640 1 -- 172.21.15.29:0/2302004718 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d9f8b0c130 msgr2=0x55d9f8b12820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:59:36.108 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:36.102+0000 7ff956ffd640 1 --2- 172.21.15.29:0/2302004718 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d9f8b0c130 0x55d9f8b12820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:59:36.109 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:36.102+0000 7ff956ffd640 1 -- 172.21.15.29:0/2302004718 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d9f8785210 con 0x55d9f8b0ca50 2024-03-16T11:59:36.109 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:36.102+0000 7ff956ffd640 1 --2- 172.21.15.29:0/2302004718 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d9f8b0ca50 0x55d9f8b12d60 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7ff94c012700 tx=0x7ff94c012bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79872876cac85c4d server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:59:36.109 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:36.102+0000 7ff936ffd640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-16T11:59:36.109 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:36.102+0000 7ff936ffd640 1 -- 172.21.15.29:0/2302004718 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff94c00f920 con 0x55d9f8b0ca50 2024-03-16T11:59:36.109 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:36.102+0000 7ff936ffd640 1 -- 172.21.15.29:0/2302004718 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8556+0+0 (secure 0 0 0) 0x7ff94c00fac0 con 0x55d9f8b0ca50 2024-03-16T11:59:36.109 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:36.102+0000 7ff960e02740 1 -- 172.21.15.29:0/2302004718 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff950032c90 con 0x55d9f8b0ca50 2024-03-16T11:59:36.109 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:36.103+0000 7ff936ffd640 1 -- 172.21.15.29:0/2302004718 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7ff94c00fcc0 con 0x55d9f8b0ca50 2024-03-16T11:59:36.109 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:36.103+0000 7ff960e02740 10 client.16122 fetch_fsmap learned FSMap version 26 2024-03-16T11:59:36.109 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:36.103+0000 7ff960e02740 1 -- 172.21.15.29:0/2302004718 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55d9f8851620 con 0x55d9f8b0ca50 2024-03-16T11:59:36.109 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:36.103+0000 7ff936ffd640 1 -- 172.21.15.29:0/2302004718 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff950032c90 con 0x55d9f8b0ca50 2024-03-16T11:59:36.109 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:36.103+0000 7ff936ffd640 1 -- 172.21.15.29:0/2302004718 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7ff94c010430 con 0x55d9f8b0ca50 2024-03-16T11:59:36.109 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:36.103+0000 7ff960e02740 10 client.16122 fetch_fsmap finished waiting for FSMap version 26 2024-03-16T11:59:36.109 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:36.103+0000 7ff960e02740 10 client.16122 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T11:59:36.109 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:36.103+0000 7ff960e02740 20 client.16122 populate_metadata read hostname 'smithi029' 2024-03-16T11:59:36.109 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:36.103+0000 7ff960e02740 1 --2- 172.21.15.29:0/2302004718 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55d9f8b259b0 0x55d9f8b27da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:59:36.109 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:36.103+0000 7ff960e02740 4 client.16122 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T11:59:36.110 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:36.103+0000 7ff960e02740 1 -- 172.21.15.29:0/2302004718 --> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d9f8a7df80 con 0x55d9f8b259b0 2024-03-16T11:59:36.110 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:36.103+0000 7ff957fff640 1 --2- 172.21.15.29:0/2302004718 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55d9f8b259b0 0x55d9f8b27da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:59:36.110 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:36.104+0000 7ff936ffd640 10 client.16122 ms_handle_connect on v2:172.21.15.152:6838/1548731833 2024-03-16T11:59:36.110 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:36.104+0000 7ff957fff640 1 --2- 172.21.15.29:0/2302004718 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55d9f8b259b0 0x55d9f8b27da0 crc :-1 s=READY pgs=116 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f12f6f44fdaa427a server_cookie=7f04e3bb37f4913c in_seq=0 out_seq=0 2024-03-16T11:59:36.111 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:36.106+0000 7ff936ffd640 1 -- 172.21.15.29:0/2302004718 <== mds.0 v2:172.21.15.152:6838/1548731833 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55d9f8a7df80 con 0x55d9f8b259b0 2024-03-16T11:59:36.111 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:36.106+0000 7ff936ffd640 10 client.16122 handle_command_reply: tid=0 2024-03-16T11:59:36.111 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:36.106+0000 7ff960e02740 1 client.16122 shutdown 2024-03-16T11:59:36.111 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:36.106+0000 7ff960e02740 1 -- 172.21.15.29:0/2302004718 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d9f8b0ca50 msgr2=0x55d9f8b12d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:59:36.111 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:36.106+0000 7ff960e02740 1 --2- 172.21.15.29:0/2302004718 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d9f8b0ca50 0x55d9f8b12d60 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7ff94c012700 tx=0x7ff94c012bc0 comp rx=0 tx=0).stop 2024-03-16T11:59:36.111 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:36.106+0000 7ff960e02740 1 -- 172.21.15.29:0/2302004718 shutdown_connections 2024-03-16T11:59:36.111 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:36.106+0000 7ff960e02740 1 --2- 172.21.15.29:0/2302004718 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d9f8b15820 0x55d9f8b132a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:59:36.111 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:36.106+0000 7ff960e02740 1 --2- 172.21.15.29:0/2302004718 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d9f8b0ca50 0x55d9f8b12d60 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:59:36.112 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:36.106+0000 7ff960e02740 1 --2- 172.21.15.29:0/2302004718 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55d9f8b259b0 0x55d9f8b27da0 crc :-1 s=READY pgs=116 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:59:36.112 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:36.106+0000 7ff960e02740 1 --2- 172.21.15.29:0/2302004718 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d9f8b0c130 0x55d9f8b12820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:59:36.112 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:36.106+0000 7ff936ffd640 0 client.16122 ms_handle_reset on v2:172.21.15.152:6838/1548731833 2024-03-16T11:59:36.112 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:36.106+0000 7ff960e02740 1 -- 172.21.15.29:0/2302004718 >> 172.21.15.29:0/2302004718 conn(0x55d9f8b06a50 msgr2=0x55d9f8b087a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:59:36.112 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:36.106+0000 7ff960e02740 1 -- 172.21.15.29:0/2302004718 shutdown_connections 2024-03-16T11:59:36.112 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:36.106+0000 7ff960e02740 1 -- 172.21.15.29:0/2302004718 wait complete. 2024-03-16T11:59:36.112 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:36.106+0000 7ff960e02740 20 client.16122 trim_cache size 0 max 16384 2024-03-16T11:59:36.125 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:36.121+0000 7ff960e02740 1 Processor -- start 2024-03-16T11:59:36.125 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:36.121+0000 7ff960e02740 1 -- start start 2024-03-16T11:59:36.125 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:36.121+0000 7ff960e02740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d9f8b0c130 0x55d9f8b21190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:59:36.125 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:36.121+0000 7ff960e02740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d9f8b0ca50 0x55d9f8b08540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:59:36.125 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:36.121+0000 7ff960e02740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d9f8b15820 0x55d9f8b2d670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:59:36.125 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:36.121+0000 7ff960e02740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7ff9581afcb0 con 0x55d9f8b0c130 2024-03-16T11:59:36.125 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:36.121+0000 7ff960e02740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7ff9581ad580 con 0x55d9f8b15820 2024-03-16T11:59:36.125 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:36.121+0000 7ff960e02740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7ff9581ab3f0 con 0x55d9f8b0ca50 2024-03-16T11:59:36.125 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:36.121+0000 7ff957fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d9f8b0c130 0x55d9f8b21190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:59:36.125 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:36.122+0000 7ff957fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d9f8b0c130 0x55d9f8b21190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50184/0 (socket says 172.21.15.29:50184) 2024-03-16T11:59:36.126 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:36.122+0000 7ff957fff640 1 -- 172.21.15.29:0/2277196171 learned_addr learned my addr 172.21.15.29:0/2277196171 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:59:36.126 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:36.122+0000 7ff956ffd640 1 --2- 172.21.15.29:0/2277196171 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d9f8b0ca50 0x55d9f8b08540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:59:36.126 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:36.122+0000 7ff957fff640 1 --2- 172.21.15.29:0/2277196171 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d9f8b15820 0x55d9f8b2d670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:59:36.126 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:36.122+0000 7ff957fff640 1 -- 172.21.15.29:0/2277196171 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d9f8b0ca50 msgr2=0x55d9f8b08540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:59:36.126 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:36.122+0000 7ff957fff640 1 --2- 172.21.15.29:0/2277196171 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d9f8b0ca50 0x55d9f8b08540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:59:36.126 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:36.122+0000 7ff957fff640 1 -- 172.21.15.29:0/2277196171 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d9f8b15820 msgr2=0x55d9f8b2d670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:59:36.126 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:36.122+0000 7ff957fff640 1 --2- 172.21.15.29:0/2277196171 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d9f8b15820 0x55d9f8b2d670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:59:36.126 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:36.122+0000 7ff957fff640 1 -- 172.21.15.29:0/2277196171 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d9f8b24550 con 0x55d9f8b0c130 2024-03-16T11:59:36.126 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:36.122+0000 7ff957fff640 1 --2- 172.21.15.29:0/2277196171 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d9f8b0c130 0x55d9f8b21190 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7ff950000f80 tx=0x7ff950004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3059fae20564e039 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:59:36.126 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:36.122+0000 7ff93d7fa640 1 -- 172.21.15.29:0/2277196171 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff950044040 con 0x55d9f8b0c130 2024-03-16T11:59:36.126 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:36.122+0000 7ff93d7fa640 1 -- 172.21.15.29:0/2277196171 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff950032c90 con 0x55d9f8b0c130 2024-03-16T11:59:36.126 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:36.122+0000 7ff93d7fa640 1 -- 172.21.15.29:0/2277196171 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9500329c0 con 0x55d9f8b0c130 2024-03-16T11:59:36.126 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:36.123+0000 7ff960e02740 1 -- 172.21.15.29:0/2277196171 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d9f8b0c130 msgr2=0x55d9f8b21190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:59:36.126 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:36.123+0000 7ff960e02740 1 --2- 172.21.15.29:0/2277196171 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d9f8b0c130 0x55d9f8b21190 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7ff950000f80 tx=0x7ff950004a40 comp rx=0 tx=0).stop 2024-03-16T11:59:36.127 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:36.123+0000 7ff960e02740 1 -- 172.21.15.29:0/2277196171 shutdown_connections 2024-03-16T11:59:36.127 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:36.123+0000 7ff960e02740 1 --2- 172.21.15.29:0/2277196171 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d9f8b15820 0x55d9f8b2d670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:59:36.127 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:36.123+0000 7ff960e02740 1 --2- 172.21.15.29:0/2277196171 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d9f8b0ca50 0x55d9f8b08540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:59:36.127 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:36.123+0000 7ff960e02740 1 --2- 172.21.15.29:0/2277196171 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d9f8b0c130 0x55d9f8b21190 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:59:36.127 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:36.123+0000 7ff960e02740 1 -- 172.21.15.29:0/2277196171 >> 172.21.15.29:0/2277196171 conn(0x55d9f8b06a50 msgr2=0x55d9f8b07e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:59:36.127 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:36.123+0000 7ff960e02740 1 -- 172.21.15.29:0/2277196171 shutdown_connections 2024-03-16T11:59:36.127 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:36.123+0000 7ff960e02740 1 -- 172.21.15.29:0/2277196171 wait complete. 2024-03-16T11:59:36.127 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:36.123+0000 7ff960e02740 1 -- start start 2024-03-16T11:59:36.127 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:36.123+0000 7ff960e02740 1 Processor -- start 2024-03-16T11:59:36.127 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:36.123+0000 7ff960e02740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d9f8b0c130 0x55d9f8b84fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:59:36.127 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:36.124+0000 7ff960e02740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d9f8b0ca50 0x55d9f8b854f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:59:36.127 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:36.124+0000 7ff960e02740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d9f8b15820 0x55d9f8b198a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:59:36.127 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:36.124+0000 7ff960e02740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7ff9581aba40 con 0x55d9f8b0ca50 2024-03-16T11:59:36.127 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:36.124+0000 7ff960e02740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7ff9581aceb0 con 0x55d9f8b0c130 2024-03-16T11:59:36.127 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:36.124+0000 7ff960e02740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7ff9581ac110 con 0x55d9f8b15820 2024-03-16T11:59:36.128 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:36.124+0000 7ff960e02740 10 client.0 _ll_register_callbacks cb 0x7ff94c0097a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T11:59:36.128 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:36.124+0000 7ff956ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d9f8b0ca50 0x55d9f8b854f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:59:36.128 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:36.124+0000 7ff956ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d9f8b0ca50 0x55d9f8b854f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50188/0 (socket says 172.21.15.29:50188) 2024-03-16T11:59:36.128 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:36.124+0000 7ff956ffd640 1 -- 172.21.15.29:0/1956951327 learned_addr learned my addr 172.21.15.29:0/1956951327 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:59:36.128 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:36.124+0000 7ff957fff640 1 --2- 172.21.15.29:0/1956951327 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d9f8b0c130 0x55d9f8b84fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:59:36.128 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:36.124+0000 7ff957fff640 1 --2- 172.21.15.29:0/1956951327 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d9f8b15820 0x55d9f8b198a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:59:36.128 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:36.124+0000 7ff956ffd640 1 -- 172.21.15.29:0/1956951327 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d9f8b15820 msgr2=0x55d9f8b198a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:59:36.128 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:36.124+0000 7ff956ffd640 1 --2- 172.21.15.29:0/1956951327 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d9f8b15820 0x55d9f8b198a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:59:36.128 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:36.124+0000 7ff956ffd640 1 -- 172.21.15.29:0/1956951327 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d9f8b0c130 msgr2=0x55d9f8b84fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:59:36.128 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:36.124+0000 7ff956ffd640 1 --2- 172.21.15.29:0/1956951327 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d9f8b0c130 0x55d9f8b84fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:59:36.128 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:36.124+0000 7ff956ffd640 1 -- 172.21.15.29:0/1956951327 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d9f8b17c10 con 0x55d9f8b0ca50 2024-03-16T11:59:36.129 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:36.124+0000 7ff956ffd640 1 --2- 172.21.15.29:0/1956951327 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d9f8b0ca50 0x55d9f8b854f0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x55d9f8b13df0 tx=0x7ff94c009ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=edb018ed51ac8319 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:59:36.129 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:36.124+0000 7ff936ffd640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-16T11:59:36.129 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:36.124+0000 7ff936ffd640 1 -- 172.21.15.29:0/1956951327 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff94c029040 con 0x55d9f8b0ca50 2024-03-16T11:59:36.129 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:36.124+0000 7ff960e02740 1 -- 172.21.15.29:0/1956951327 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff950032c90 con 0x55d9f8b0ca50 2024-03-16T11:59:36.129 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:36.124+0000 7ff936ffd640 1 -- 172.21.15.29:0/1956951327 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8556+0+0 (secure 0 0 0) 0x7ff94c00f920 con 0x55d9f8b0ca50 2024-03-16T11:59:36.129 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:36.124+0000 7ff936ffd640 1 -- 172.21.15.29:0/1956951327 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7ff94c00fb20 con 0x55d9f8b0ca50 2024-03-16T11:59:36.129 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:36.124+0000 7ff960e02740 10 client.16128 fetch_fsmap learned FSMap version 26 2024-03-16T11:59:36.129 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:36.124+0000 7ff960e02740 1 -- 172.21.15.29:0/1956951327 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55d9f8b8c5f0 con 0x55d9f8b0ca50 2024-03-16T11:59:36.129 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:36.125+0000 7ff936ffd640 1 -- 172.21.15.29:0/1956951327 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff950032c90 con 0x55d9f8b0ca50 2024-03-16T11:59:36.129 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:36.125+0000 7ff936ffd640 1 -- 172.21.15.29:0/1956951327 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7ff94c0053d0 con 0x55d9f8b0ca50 2024-03-16T11:59:36.129 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:36.125+0000 7ff960e02740 10 client.16128 fetch_fsmap finished waiting for FSMap version 26 2024-03-16T11:59:36.129 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:36.125+0000 7ff960e02740 10 client.16128 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T11:59:36.129 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:36.125+0000 7ff960e02740 20 client.16128 populate_metadata read hostname 'smithi029' 2024-03-16T11:59:36.130 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:36.125+0000 7ff960e02740 1 --2- 172.21.15.29:0/1956951327 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55d9f8b8cda0 0x55d9f8b8f1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:59:36.130 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:36.125+0000 7ff960e02740 4 client.16128 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-16T11:59:36.130 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:36.125+0000 7ff960e02740 1 -- 172.21.15.29:0/1956951327 --> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55d9f8a782d0 con 0x55d9f8b8cda0 2024-03-16T11:59:36.130 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:36.125+0000 7ff957fff640 1 --2- 172.21.15.29:0/1956951327 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55d9f8b8cda0 0x55d9f8b8f1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:59:36.130 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:36.125+0000 7ff936ffd640 10 client.16128 ms_handle_connect on v2:172.21.15.152:6838/1548731833 2024-03-16T11:59:36.130 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:36.125+0000 7ff957fff640 1 --2- 172.21.15.29:0/1956951327 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55d9f8b8cda0 0x55d9f8b8f1c0 crc :-1 s=READY pgs=117 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f4eda79ee97c9e55 server_cookie=20f4a35fb651bd35 in_seq=0 out_seq=0 2024-03-16T11:59:36.346 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:36.346+0000 7ff936ffd640 1 -- 172.21.15.29:0/1956951327 <== mds.0 v2:172.21.15.152:6838/1548731833 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55d9f8a782d0 con 0x55d9f8b8cda0 2024-03-16T11:59:36.346 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:36.346+0000 7ff936ffd640 10 client.16128 handle_command_reply: tid=0 2024-03-16T11:59:36.346 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:36.346+0000 7ff960e02740 1 client.16128 shutdown 2024-03-16T11:59:36.346 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:36.346+0000 7ff960e02740 1 -- 172.21.15.29:0/1956951327 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d9f8b0ca50 msgr2=0x55d9f8b854f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:59:36.346 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:36.346+0000 7ff960e02740 1 --2- 172.21.15.29:0/1956951327 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d9f8b0ca50 0x55d9f8b854f0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x55d9f8b13df0 tx=0x7ff94c009ee0 comp rx=0 tx=0).stop 2024-03-16T11:59:36.346 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:36.346+0000 7ff960e02740 1 -- 172.21.15.29:0/1956951327 shutdown_connections 2024-03-16T11:59:36.346 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:36.346+0000 7ff960e02740 1 --2- 172.21.15.29:0/1956951327 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d9f8b15820 0x55d9f8b198a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:59:36.346 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:36.346+0000 7ff960e02740 1 --2- 172.21.15.29:0/1956951327 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d9f8b0ca50 0x55d9f8b854f0 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:59:36.347 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:36.346+0000 7ff960e02740 1 --2- 172.21.15.29:0/1956951327 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55d9f8b8cda0 0x55d9f8b8f1c0 crc :-1 s=READY pgs=117 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:59:36.347 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:36.346+0000 7ff960e02740 1 --2- 172.21.15.29:0/1956951327 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d9f8b0c130 0x55d9f8b84fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:59:36.347 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:36.346+0000 7ff936ffd640 0 client.16128 ms_handle_reset on v2:172.21.15.152:6838/1548731833 2024-03-16T11:59:36.347 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:36.346+0000 7ff960e02740 1 -- 172.21.15.29:0/1956951327 >> 172.21.15.29:0/1956951327 conn(0x55d9f8b06a50 msgr2=0x55d9f8b07d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:59:36.347 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:36.346+0000 7ff960e02740 1 -- 172.21.15.29:0/1956951327 shutdown_connections 2024-03-16T11:59:36.347 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:36.346+0000 7ff960e02740 1 -- 172.21.15.29:0/1956951327 wait complete. 2024-03-16T11:59:36.347 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:36.346+0000 7ff960e02740 20 client.16128 trim_cache size 0 max 16384 2024-03-16T11:59:36.347 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:{ 2024-03-16T11:59:36.347 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "message": "", 2024-03-16T11:59:36.347 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "return_code": 0 2024-03-16T11:59:36.347 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:} 2024-03-16T11:59:36.348 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:36.347+0000 7ff95e13b640 1 -- 172.21.15.29:0/3009795496 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7ff9400a2a10 msgr2=0x7ff9400a4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:59:36.348 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:36.347+0000 7ff95e13b640 1 --2- 172.21.15.29:0/3009795496 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7ff9400a2a10 0x7ff9400a4ed0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7ff9480ccac0 tx=0x7ff9480d2e30 comp rx=0 tx=0).stop 2024-03-16T11:59:36.348 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:36.347+0000 7ff95e13b640 1 -- 172.21.15.29:0/3009795496 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff958100e50 msgr2=0x7ff958197ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:59:36.348 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:36.347+0000 7ff95e13b640 1 --2- 172.21.15.29:0/3009795496 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff958100e50 0x7ff958197ee0 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7ff94800e9f0 tx=0x7ff94800eeb0 comp rx=0 tx=0).stop 2024-03-16T11:59:36.349 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:36.347+0000 7ff95e13b640 1 -- 172.21.15.29:0/3009795496 shutdown_connections 2024-03-16T11:59:36.349 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:36.347+0000 7ff95e13b640 1 --2- 172.21.15.29:0/3009795496 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7ff9400a2a10 0x7ff9400a4ed0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:59:36.349 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:36.347+0000 7ff95e13b640 1 --2- 172.21.15.29:0/3009795496 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff958105690 0x7ff95819c7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:59:36.349 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:36.347+0000 7ff95e13b640 1 --2- 172.21.15.29:0/3009795496 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff958101810 0x7ff958198420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:59:36.349 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:36.347+0000 7ff95e13b640 1 --2- 172.21.15.29:0/3009795496 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff958100e50 0x7ff958197ee0 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:59:36.349 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:36.347+0000 7ff95e13b640 1 -- 172.21.15.29:0/3009795496 >> 172.21.15.29:0/3009795496 conn(0x7ff958076c70 msgr2=0x7ff9580ffe80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:59:36.349 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:36.347+0000 7ff95e13b640 1 -- 172.21.15.29:0/3009795496 shutdown_connections 2024-03-16T11:59:36.349 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:36.347+0000 7ff95e13b640 1 -- 172.21.15.29:0/3009795496 wait complete. 2024-03-16T11:59:36.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:59:36 smithi152 ceph-mon[36291]: pgmap v272: 97 pgs: 97 active+clean; 74 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 2.4 MiB/s wr, 235 op/s 2024-03-16T11:59:36.545 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:59:36 smithi154 ceph-mon[36320]: pgmap v272: 97 pgs: 97 active+clean; 74 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 2.4 MiB/s wr, 235 op/s 2024-03-16T11:59:36.587 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:59:36 smithi029 ceph-mon[31975]: pgmap v272: 97 pgs: 97 active+clean; 74 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 2.4 MiB/s wr, 235 op/s 2024-03-16T11:59:36.795 INFO:tasks.workunit.client.0.smithi029.stdout: 1 879068 10.25 MB/sec execute 321 sec latency 208.350 ms 2024-03-16T11:59:37.795 INFO:tasks.workunit.client.0.smithi029.stdout: 1 880882 10.26 MB/sec execute 322 sec latency 30.904 ms 2024-03-16T11:59:38.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:59:38 smithi152 ceph-mon[36291]: pgmap v273: 97 pgs: 97 active+clean; 80 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 3.7 MiB/s wr, 320 op/s 2024-03-16T11:59:38.545 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:59:38 smithi154 ceph-mon[36320]: pgmap v273: 97 pgs: 97 active+clean; 80 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 3.7 MiB/s wr, 320 op/s 2024-03-16T11:59:38.587 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:59:38 smithi029 ceph-mon[31975]: pgmap v273: 97 pgs: 97 active+clean; 80 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 3.7 MiB/s wr, 320 op/s 2024-03-16T11:59:38.795 INFO:tasks.workunit.client.0.smithi029.stdout: 1 882375 10.24 MB/sec execute 323 sec latency 20.361 ms 2024-03-16T11:59:39.796 INFO:tasks.workunit.client.0.smithi029.stdout: 1 884409 10.25 MB/sec execute 324 sec latency 30.354 ms 2024-03-16T11:59:40.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:59:40 smithi152 ceph-mon[36291]: pgmap v274: 97 pgs: 97 active+clean; 80 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 2.6 MiB/s wr, 168 op/s 2024-03-16T11:59:40.545 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:59:40 smithi154 ceph-mon[36320]: pgmap v274: 97 pgs: 97 active+clean; 80 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 2.6 MiB/s wr, 168 op/s 2024-03-16T11:59:40.588 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:59:40 smithi029 ceph-mon[31975]: pgmap v274: 97 pgs: 97 active+clean; 80 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 2.6 MiB/s wr, 168 op/s 2024-03-16T11:59:40.796 INFO:tasks.workunit.client.0.smithi029.stdout: 1 885878 10.24 MB/sec execute 325 sec latency 18.033 ms 2024-03-16T11:59:41.796 INFO:tasks.workunit.client.0.smithi029.stdout: 1 887827 10.25 MB/sec execute 326 sec latency 86.933 ms 2024-03-16T11:59:42.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:59:42 smithi152 ceph-mon[36291]: pgmap v275: 97 pgs: 97 active+clean; 70 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 3.8 MiB/s wr, 318 op/s 2024-03-16T11:59:42.545 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:59:42 smithi154 ceph-mon[36320]: pgmap v275: 97 pgs: 97 active+clean; 70 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 3.8 MiB/s wr, 318 op/s 2024-03-16T11:59:42.587 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:59:42 smithi029 ceph-mon[31975]: pgmap v275: 97 pgs: 97 active+clean; 70 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 3.8 MiB/s wr, 318 op/s 2024-03-16T11:59:42.796 INFO:tasks.workunit.client.0.smithi029.stdout: 1 889204 10.23 MB/sec execute 327 sec latency 21.762 ms 2024-03-16T11:59:43.796 INFO:tasks.workunit.client.0.smithi029.stdout: 1 891364 10.24 MB/sec execute 328 sec latency 26.959 ms 2024-03-16T11:59:44.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:59:44 smithi152 ceph-mon[36291]: pgmap v276: 97 pgs: 97 active+clean; 70 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 2.6 MiB/s wr, 249 op/s 2024-03-16T11:59:44.545 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:59:44 smithi154 ceph-mon[36320]: pgmap v276: 97 pgs: 97 active+clean; 70 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 2.6 MiB/s wr, 249 op/s 2024-03-16T11:59:44.588 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:59:44 smithi029 ceph-mon[31975]: pgmap v276: 97 pgs: 97 active+clean; 70 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 2.6 MiB/s wr, 249 op/s 2024-03-16T11:59:44.796 INFO:tasks.workunit.client.0.smithi029.stdout: 1 892814 10.23 MB/sec execute 329 sec latency 19.339 ms 2024-03-16T11:59:45.796 INFO:tasks.workunit.client.0.smithi029.stdout: 1 894994 10.24 MB/sec execute 330 sec latency 17.639 ms 2024-03-16T11:59:46.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:59:46 smithi152 ceph-mon[36291]: pgmap v277: 97 pgs: 97 active+clean; 70 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 2.5 MiB/s wr, 244 op/s 2024-03-16T11:59:46.545 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:59:46 smithi154 ceph-mon[36320]: pgmap v277: 97 pgs: 97 active+clean; 70 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 2.5 MiB/s wr, 244 op/s 2024-03-16T11:59:46.588 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:59:46 smithi029 ceph-mon[31975]: pgmap v277: 97 pgs: 97 active+clean; 70 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 2.5 MiB/s wr, 244 op/s 2024-03-16T11:59:46.796 INFO:tasks.workunit.client.0.smithi029.stdout: 1 896244 10.23 MB/sec execute 331 sec latency 77.767 ms 2024-03-16T11:59:47.796 INFO:tasks.workunit.client.0.smithi029.stdout: 1 898414 10.24 MB/sec execute 332 sec latency 34.953 ms 2024-03-16T11:59:48.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:59:48 smithi152 ceph-mon[36291]: pgmap v278: 97 pgs: 97 active+clean; 76 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 3.6 MiB/s wr, 327 op/s 2024-03-16T11:59:48.545 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:59:48 smithi154 ceph-mon[36320]: pgmap v278: 97 pgs: 97 active+clean; 76 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 3.6 MiB/s wr, 327 op/s 2024-03-16T11:59:48.587 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:59:48 smithi029 ceph-mon[31975]: pgmap v278: 97 pgs: 97 active+clean; 76 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 3.6 MiB/s wr, 327 op/s 2024-03-16T11:59:48.796 INFO:tasks.workunit.client.0.smithi029.stdout: 1 899837 10.22 MB/sec execute 333 sec latency 26.923 ms 2024-03-16T11:59:49.437 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.438+0000 7f7f3e4dc640 1 -- 172.21.15.29:0/3815360847 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7f381077a0 msgr2=0x7f7f38107b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:59:49.438 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.438+0000 7f7f3e4dc640 1 --2- 172.21.15.29:0/3815360847 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7f381077a0 0x7f7f38107b80 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f7f28009930 tx=0x7f7f2802bc30 comp rx=0 tx=0).stop 2024-03-16T11:59:49.438 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.438+0000 7f7f3e4dc640 1 -- 172.21.15.29:0/3815360847 shutdown_connections 2024-03-16T11:59:49.438 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.438+0000 7f7f3e4dc640 1 --2- 172.21.15.29:0/3815360847 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7f38102150 0x7f7f3810e660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:59:49.438 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.438+0000 7f7f3e4dc640 1 --2- 172.21.15.29:0/3815360847 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7f38101790 0x7f7f38101c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:59:49.438 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.438+0000 7f7f3e4dc640 1 --2- 172.21.15.29:0/3815360847 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7f381077a0 0x7f7f38107b80 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:59:49.438 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.438+0000 7f7f3e4dc640 1 -- 172.21.15.29:0/3815360847 >> 172.21.15.29:0/3815360847 conn(0x7f7f380fd2a0 msgr2=0x7f7f380ff6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:59:49.438 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.438+0000 7f7f3e4dc640 1 -- 172.21.15.29:0/3815360847 shutdown_connections 2024-03-16T11:59:49.438 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.438+0000 7f7f3e4dc640 1 -- 172.21.15.29:0/3815360847 wait complete. 2024-03-16T11:59:49.439 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.438+0000 7f7f3e4dc640 1 Processor -- start 2024-03-16T11:59:49.439 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.438+0000 7f7f3e4dc640 1 -- start start 2024-03-16T11:59:49.446 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.439+0000 7f7f3e4dc640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7f38101790 0x7f7f38197ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:59:49.447 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.439+0000 7f7f3e4dc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7f38102150 0x7f7f38198530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:59:49.447 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.439+0000 7f7f3e4dc640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7f381077a0 0x7f7f3819c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:59:49.447 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.439+0000 7f7f3e4dc640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f7f38111b70 con 0x7f7f38102150 2024-03-16T11:59:49.447 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.439+0000 7f7f3e4dc640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f7f381119f0 con 0x7f7f381077a0 2024-03-16T11:59:49.447 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.439+0000 7f7f3e4dc640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f7f38111cf0 con 0x7f7f38101790 2024-03-16T11:59:49.447 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.439+0000 7f7f377fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7f38102150 0x7f7f38198530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:59:49.447 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.439+0000 7f7f37fff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7f38101790 0x7f7f38197ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:59:49.447 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.439+0000 7f7f377fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7f38102150 0x7f7f38198530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53908/0 (socket says 172.21.15.29:53908) 2024-03-16T11:59:49.447 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.439+0000 7f7f37fff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7f38101790 0x7f7f38197ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.29:33118/0 (socket says 172.21.15.29:33118) 2024-03-16T11:59:49.447 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.439+0000 7f7f377fe640 1 -- 172.21.15.29:0/43993373 learned_addr learned my addr 172.21.15.29:0/43993373 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:59:49.447 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.439+0000 7f7f3ca79640 1 --2- 172.21.15.29:0/43993373 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7f381077a0 0x7f7f3819c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:59:49.447 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.439+0000 7f7f377fe640 1 -- 172.21.15.29:0/43993373 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7f38101790 msgr2=0x7f7f38197ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:59:49.448 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.439+0000 7f7f377fe640 1 --2- 172.21.15.29:0/43993373 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7f38101790 0x7f7f38197ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:59:49.448 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.439+0000 7f7f377fe640 1 -- 172.21.15.29:0/43993373 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7f381077a0 msgr2=0x7f7f3819c8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:59:49.448 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.439+0000 7f7f377fe640 1 --2- 172.21.15.29:0/43993373 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7f381077a0 0x7f7f3819c8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:59:49.448 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.439+0000 7f7f377fe640 1 -- 172.21.15.29:0/43993373 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f3819cfe0 con 0x7f7f38102150 2024-03-16T11:59:49.448 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.439+0000 7f7f37fff640 1 --2- 172.21.15.29:0/43993373 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7f38101790 0x7f7f38197ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:59:49.448 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.439+0000 7f7f377fe640 1 --2- 172.21.15.29:0/43993373 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7f38102150 0x7f7f38198530 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f7f2000e9f0 tx=0x7f7f2000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:59:49.448 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.440+0000 7f7f357fa640 1 -- 172.21.15.29:0/43993373 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f2000cd80 con 0x7f7f38102150 2024-03-16T11:59:49.448 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.440+0000 7f7f3e4dc640 1 -- 172.21.15.29:0/43993373 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7f3819d2d0 con 0x7f7f38102150 2024-03-16T11:59:49.448 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.440+0000 7f7f357fa640 1 -- 172.21.15.29:0/43993373 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7f20010910 con 0x7f7f38102150 2024-03-16T11:59:49.448 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.440+0000 7f7f357fa640 1 -- 172.21.15.29:0/43993373 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f200188f0 con 0x7f7f38102150 2024-03-16T11:59:49.448 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.440+0000 7f7f3e4dc640 1 -- 172.21.15.29:0/43993373 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7f381a4b60 con 0x7f7f38102150 2024-03-16T11:59:49.448 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.442+0000 7f7f357fa640 1 -- 172.21.15.29:0/43993373 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7f7f20010ac0 con 0x7f7f38102150 2024-03-16T11:59:49.448 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.442+0000 7f7f411a5740 1 Processor -- start 2024-03-16T11:59:49.448 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.442+0000 7f7f357fa640 1 --2- 172.21.15.29:0/43993373 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f7f240a2a10 0x7f7f240a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:59:49.449 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.442+0000 7f7f411a5740 1 -- start start 2024-03-16T11:59:49.449 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.442+0000 7f7f357fa640 1 -- 172.21.15.29:0/43993373 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 8556+0+0 (secure 0 0 0) 0x7f7f200cf230 con 0x7f7f38102150 2024-03-16T11:59:49.449 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.442+0000 7f7f3ca79640 1 --2- 172.21.15.29:0/43993373 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f7f240a2a10 0x7f7f240a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:59:49.449 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.442+0000 7f7f411a5740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558affed7130 0x558affed7510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:59:49.449 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.442+0000 7f7f411a5740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558affed7a50 0x558affee02e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:59:49.449 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.442+0000 7f7f411a5740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558affee0820 0x558affee2c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:59:49.449 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.442+0000 7f7f411a5740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x558affe3ba40 con 0x558affed7a50 2024-03-16T11:59:49.449 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.442+0000 7f7f411a5740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x558affde4920 con 0x558affee0820 2024-03-16T11:59:49.449 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.442+0000 7f7f411a5740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x558affe37ae0 con 0x558affed7130 2024-03-16T11:59:49.449 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.443+0000 7f7f377fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558affed7a50 0x558affee02e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:59:49.449 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.443+0000 7f7f377fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558affed7a50 0x558affee02e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53916/0 (socket says 172.21.15.29:53916) 2024-03-16T11:59:49.449 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.443+0000 7f7f377fe640 1 -- 172.21.15.29:0/1267067651 learned_addr learned my addr 172.21.15.29:0/1267067651 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:59:49.449 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.443+0000 7f7f37fff640 1 --2- 172.21.15.29:0/1267067651 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558affed7130 0x558affed7510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:59:49.449 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.443+0000 7f7f3ca79640 1 --2- 172.21.15.29:0/1267067651 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558affee0820 0x558affee2c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:59:49.449 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.443+0000 7f7f377fe640 1 -- 172.21.15.29:0/1267067651 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558affed7130 msgr2=0x558affed7510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:59:49.450 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.443+0000 7f7f377fe640 1 --2- 172.21.15.29:0/1267067651 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558affed7130 0x558affed7510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:59:49.450 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.443+0000 7f7f377fe640 1 -- 172.21.15.29:0/1267067651 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558affee0820 msgr2=0x558affee2c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:59:49.450 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.443+0000 7f7f377fe640 1 --2- 172.21.15.29:0/1267067651 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558affee0820 0x558affee2c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:59:49.450 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.443+0000 7f7f377fe640 1 -- 172.21.15.29:0/1267067651 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558affdfc460 con 0x558affed7a50 2024-03-16T11:59:49.450 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.443+0000 7f7f37fff640 1 --2- 172.21.15.29:0/1267067651 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558affed7130 0x558affed7510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:59:49.450 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.443+0000 7f7f3ca79640 1 --2- 172.21.15.29:0/43993373 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f7f240a2a10 0x7f7f240a4ed0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f7f30005fd0 tx=0x7f7f30011b40 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:59:49.450 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.443+0000 7f7f377fe640 1 --2- 172.21.15.29:0/1267067651 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558affed7a50 0x558affee02e0 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f7f200982f0 tx=0x7f7f2009f9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f4d6eddec1e6a67 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:59:49.450 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.443+0000 7f7f2e7fc640 1 -- 172.21.15.29:0/1267067651 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f20084e60 con 0x558affed7a50 2024-03-16T11:59:49.450 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.443+0000 7f7f2e7fc640 1 -- 172.21.15.29:0/1267067651 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7f20097f90 con 0x558affed7a50 2024-03-16T11:59:49.450 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.443+0000 7f7f2e7fc640 1 -- 172.21.15.29:0/1267067651 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f2009cda0 con 0x558affed7a50 2024-03-16T11:59:49.450 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.444+0000 7f7f411a5740 1 -- 172.21.15.29:0/1267067651 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558affed7a50 msgr2=0x558affee02e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:59:49.450 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.444+0000 7f7f411a5740 1 --2- 172.21.15.29:0/1267067651 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558affed7a50 0x558affee02e0 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f7f200982f0 tx=0x7f7f2009f9c0 comp rx=0 tx=0).stop 2024-03-16T11:59:49.450 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.444+0000 7f7f411a5740 1 -- 172.21.15.29:0/1267067651 shutdown_connections 2024-03-16T11:59:49.450 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.444+0000 7f7f411a5740 1 --2- 172.21.15.29:0/1267067651 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558affee0820 0x558affee2c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:59:49.451 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.444+0000 7f7f411a5740 1 --2- 172.21.15.29:0/1267067651 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558affed7a50 0x558affee02e0 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:59:49.451 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.444+0000 7f7f411a5740 1 --2- 172.21.15.29:0/1267067651 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558affed7130 0x558affed7510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:59:49.451 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.444+0000 7f7f411a5740 1 -- 172.21.15.29:0/1267067651 >> 172.21.15.29:0/1267067651 conn(0x558affed1a50 msgr2=0x558affed1e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:59:49.451 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.444+0000 7f7f411a5740 1 -- 172.21.15.29:0/1267067651 shutdown_connections 2024-03-16T11:59:49.451 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.444+0000 7f7f411a5740 1 -- 172.21.15.29:0/1267067651 wait complete. 2024-03-16T11:59:49.451 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.444+0000 7f7f411a5740 1 -- start start 2024-03-16T11:59:49.451 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.445+0000 7f7f411a5740 1 Processor -- start 2024-03-16T11:59:49.451 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.445+0000 7f7f411a5740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558affed7130 0x558affedd8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:59:49.451 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.445+0000 7f7f411a5740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558affed7a50 0x558affedde00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:59:49.451 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.445+0000 7f7f411a5740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558affee0820 0x558affede340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:59:49.451 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.445+0000 7f7f411a5740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x558affee33f0 con 0x558affed7130 2024-03-16T11:59:49.451 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.445+0000 7f7f411a5740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x558affdf7d70 con 0x558affed7a50 2024-03-16T11:59:49.451 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.445+0000 7f7f411a5740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x558affee3570 con 0x558affee0820 2024-03-16T11:59:49.451 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.445+0000 7f7f411a5740 10 client.0 _ll_register_callbacks cb 0x558affb4fa00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T11:59:49.452 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.445+0000 7f7f37fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558affed7130 0x558affedd8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:59:49.452 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.445+0000 7f7f37fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558affed7130 0x558affedd8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53924/0 (socket says 172.21.15.29:53924) 2024-03-16T11:59:49.452 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.445+0000 7f7f37fff640 1 -- 172.21.15.29:0/2358547804 learned_addr learned my addr 172.21.15.29:0/2358547804 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:59:49.452 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.445+0000 7f7f377fe640 1 --2- 172.21.15.29:0/2358547804 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558affed7a50 0x558affedde00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:59:49.452 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.445+0000 7f7f3ca79640 1 --2- 172.21.15.29:0/2358547804 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558affee0820 0x558affede340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:59:49.452 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.445+0000 7f7f377fe640 1 -- 172.21.15.29:0/2358547804 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558affee0820 msgr2=0x558affede340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:59:49.452 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.445+0000 7f7f377fe640 1 --2- 172.21.15.29:0/2358547804 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558affee0820 0x558affede340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:59:49.452 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.445+0000 7f7f377fe640 1 -- 172.21.15.29:0/2358547804 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558affed7130 msgr2=0x558affedd8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:59:49.452 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.445+0000 7f7f377fe640 1 --2- 172.21.15.29:0/2358547804 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558affed7130 0x558affedd8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:59:49.452 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.445+0000 7f7f377fe640 1 -- 172.21.15.29:0/2358547804 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558affb50210 con 0x558affed7a50 2024-03-16T11:59:49.452 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.445+0000 7f7f37fff640 1 --2- 172.21.15.29:0/2358547804 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558affed7130 0x558affedd8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:59:49.452 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.445+0000 7f7f3ca79640 1 --2- 172.21.15.29:0/2358547804 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558affee0820 0x558affede340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T11:59:49.452 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.446+0000 7f7f377fe640 1 --2- 172.21.15.29:0/2358547804 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558affed7a50 0x558affedde00 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x558affed6be0 tx=0x7f7f2009b020 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6ffed8494859dcdf server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:59:49.452 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.446+0000 7f7f177fe640 10 client.0 ms_handle_connect on v2:172.21.15.152:3300/0 2024-03-16T11:59:49.453 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.446+0000 7f7f177fe640 1 -- 172.21.15.29:0/2358547804 <== mon.2 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f20084e60 con 0x558affed7a50 2024-03-16T11:59:49.453 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.446+0000 7f7f177fe640 1 -- 172.21.15.29:0/2358547804 <== mon.2 v2:172.21.15.152:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8556+0+0 (secure 0 0 0) 0x7f7f2009cda0 con 0x558affed7a50 2024-03-16T11:59:49.453 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.446+0000 7f7f411a5740 1 -- 172.21.15.29:0/2358547804 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7f20097f90 con 0x558affed7a50 2024-03-16T11:59:49.453 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.446+0000 7f7f177fe640 1 -- 172.21.15.29:0/2358547804 <== mon.2 v2:172.21.15.152:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f7f20010430 con 0x558affed7a50 2024-03-16T11:59:49.453 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.446+0000 7f7f411a5740 10 client.25955 fetch_fsmap learned FSMap version 26 2024-03-16T11:59:49.453 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.446+0000 7f7f411a5740 1 -- 172.21.15.29:0/2358547804 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x558affc1c620 con 0x558affed7a50 2024-03-16T11:59:49.453 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.446+0000 7f7f177fe640 1 -- 172.21.15.29:0/2358547804 <== mon.2 v2:172.21.15.152:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7f20097f90 con 0x558affed7a50 2024-03-16T11:59:49.453 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.446+0000 7f7f177fe640 1 -- 172.21.15.29:0/2358547804 <== mon.2 v2:172.21.15.152:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f7f2009a7e0 con 0x558affed7a50 2024-03-16T11:59:49.453 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.446+0000 7f7f411a5740 10 client.25955 fetch_fsmap finished waiting for FSMap version 26 2024-03-16T11:59:49.453 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.446+0000 7f7f411a5740 10 client.25955 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T11:59:49.453 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.446+0000 7f7f411a5740 20 client.25955 populate_metadata read hostname 'smithi029' 2024-03-16T11:59:49.453 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.446+0000 7f7f411a5740 1 --2- 172.21.15.29:0/2358547804 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x558affef0870 0x558affef2c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:59:49.453 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.446+0000 7f7f411a5740 4 client.25955 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T11:59:49.453 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.446+0000 7f7f411a5740 1 -- 172.21.15.29:0/2358547804 --> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558affe48f80 con 0x558affef0870 2024-03-16T11:59:49.453 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.447+0000 7f7f37fff640 1 --2- 172.21.15.29:0/2358547804 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x558affef0870 0x558affef2c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:59:49.454 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.447+0000 7f7f177fe640 10 client.25955 ms_handle_connect on v2:172.21.15.152:6838/1548731833 2024-03-16T11:59:49.454 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.447+0000 7f7f37fff640 1 --2- 172.21.15.29:0/2358547804 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x558affef0870 0x558affef2c60 crc :-1 s=READY pgs=118 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ac40b4bfe7353d61 server_cookie=600597941e303785 in_seq=0 out_seq=0 2024-03-16T11:59:49.455 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.449+0000 7f7f177fe640 1 -- 172.21.15.29:0/2358547804 <== mds.0 v2:172.21.15.152:6838/1548731833 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x558affe48f80 con 0x558affef0870 2024-03-16T11:59:49.455 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.449+0000 7f7f177fe640 10 client.25955 handle_command_reply: tid=0 2024-03-16T11:59:49.455 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.449+0000 7f7f411a5740 1 client.25955 shutdown 2024-03-16T11:59:49.455 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.449+0000 7f7f411a5740 1 -- 172.21.15.29:0/2358547804 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558affed7a50 msgr2=0x558affedde00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:59:49.455 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.449+0000 7f7f411a5740 1 --2- 172.21.15.29:0/2358547804 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558affed7a50 0x558affedde00 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x558affed6be0 tx=0x7f7f2009b020 comp rx=0 tx=0).stop 2024-03-16T11:59:49.455 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.449+0000 7f7f411a5740 1 -- 172.21.15.29:0/2358547804 shutdown_connections 2024-03-16T11:59:49.455 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.449+0000 7f7f411a5740 1 --2- 172.21.15.29:0/2358547804 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558affee0820 0x558affede340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:59:49.455 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.449+0000 7f7f411a5740 1 --2- 172.21.15.29:0/2358547804 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x558affef0870 0x558affef2c60 crc :-1 s=READY pgs=118 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:59:49.455 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.449+0000 7f7f411a5740 1 --2- 172.21.15.29:0/2358547804 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558affed7a50 0x558affedde00 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:59:49.455 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.449+0000 7f7f411a5740 1 --2- 172.21.15.29:0/2358547804 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558affed7130 0x558affedd8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:59:49.456 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.449+0000 7f7f177fe640 0 client.25955 ms_handle_reset on v2:172.21.15.152:6838/1548731833 2024-03-16T11:59:49.456 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.449+0000 7f7f411a5740 1 -- 172.21.15.29:0/2358547804 >> 172.21.15.29:0/2358547804 conn(0x558affed1a50 msgr2=0x558affed37a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:59:49.456 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.449+0000 7f7f411a5740 1 -- 172.21.15.29:0/2358547804 shutdown_connections 2024-03-16T11:59:49.456 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.449+0000 7f7f411a5740 1 -- 172.21.15.29:0/2358547804 wait complete. 2024-03-16T11:59:49.456 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.449+0000 7f7f411a5740 20 client.25955 trim_cache size 0 max 16384 2024-03-16T11:59:49.467 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.466+0000 7f7f411a5740 1 Processor -- start 2024-03-16T11:59:49.468 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.466+0000 7f7f411a5740 1 -- start start 2024-03-16T11:59:49.468 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.466+0000 7f7f411a5740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558affed7130 0x558affedfff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:59:49.468 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.466+0000 7f7f411a5740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558affed7a50 0x558affecca40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:59:49.468 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.466+0000 7f7f411a5740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558affee0820 0x558affede160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:59:49.468 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.466+0000 7f7f411a5740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f7f381afcf0 con 0x558affed7130 2024-03-16T11:59:49.468 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.466+0000 7f7f411a5740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f7f381ad5c0 con 0x558affee0820 2024-03-16T11:59:49.468 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.466+0000 7f7f411a5740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f7f381ab430 con 0x558affed7a50 2024-03-16T11:59:49.468 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.466+0000 7f7f37fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558affed7130 0x558affedfff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:59:49.468 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.466+0000 7f7f37fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558affed7130 0x558affedfff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53940/0 (socket says 172.21.15.29:53940) 2024-03-16T11:59:49.468 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.467+0000 7f7f37fff640 1 -- 172.21.15.29:0/1761314873 learned_addr learned my addr 172.21.15.29:0/1761314873 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:59:49.468 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.467+0000 7f7f3ca79640 1 --2- 172.21.15.29:0/1761314873 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558affee0820 0x558affede160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:59:49.469 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.467+0000 7f7f377fe640 1 --2- 172.21.15.29:0/1761314873 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558affed7a50 0x558affecca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:59:49.469 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.467+0000 7f7f37fff640 1 -- 172.21.15.29:0/1761314873 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558affed7a50 msgr2=0x558affecca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:59:49.469 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.467+0000 7f7f37fff640 1 --2- 172.21.15.29:0/1761314873 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558affed7a50 0x558affecca40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:59:49.469 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.467+0000 7f7f37fff640 1 -- 172.21.15.29:0/1761314873 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558affee0820 msgr2=0x558affede160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:59:49.469 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.467+0000 7f7f37fff640 1 --2- 172.21.15.29:0/1761314873 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558affee0820 0x558affede160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:59:49.469 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.467+0000 7f7f37fff640 1 -- 172.21.15.29:0/1761314873 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558affedc870 con 0x558affed7130 2024-03-16T11:59:49.469 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.467+0000 7f7f37fff640 1 --2- 172.21.15.29:0/1761314873 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558affed7130 0x558affedfff0 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f7f28000c00 tx=0x7f7f28009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aed1f11cf71449e5 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:59:49.469 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.467+0000 7f7f2dffb640 1 -- 172.21.15.29:0/1761314873 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f2804e070 con 0x558affed7130 2024-03-16T11:59:49.469 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.467+0000 7f7f2dffb640 1 -- 172.21.15.29:0/1761314873 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7f28048da0 con 0x558affed7130 2024-03-16T11:59:49.469 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.467+0000 7f7f2dffb640 1 -- 172.21.15.29:0/1761314873 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f28004f30 con 0x558affed7130 2024-03-16T11:59:49.469 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.468+0000 7f7f411a5740 1 -- 172.21.15.29:0/1761314873 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558affed7130 msgr2=0x558affedfff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:59:49.469 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.468+0000 7f7f411a5740 1 --2- 172.21.15.29:0/1761314873 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558affed7130 0x558affedfff0 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f7f28000c00 tx=0x7f7f28009260 comp rx=0 tx=0).stop 2024-03-16T11:59:49.469 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.468+0000 7f7f411a5740 1 -- 172.21.15.29:0/1761314873 shutdown_connections 2024-03-16T11:59:49.470 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.468+0000 7f7f411a5740 1 --2- 172.21.15.29:0/1761314873 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558affee0820 0x558affede160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:59:49.470 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.468+0000 7f7f411a5740 1 --2- 172.21.15.29:0/1761314873 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558affed7a50 0x558affecca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:59:49.470 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.468+0000 7f7f411a5740 1 --2- 172.21.15.29:0/1761314873 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558affed7130 0x558affedfff0 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:59:49.470 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.468+0000 7f7f411a5740 1 -- 172.21.15.29:0/1761314873 >> 172.21.15.29:0/1761314873 conn(0x558affed1a50 msgr2=0x558affed0bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:59:49.470 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.468+0000 7f7f411a5740 1 -- 172.21.15.29:0/1761314873 shutdown_connections 2024-03-16T11:59:49.470 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.468+0000 7f7f411a5740 1 -- 172.21.15.29:0/1761314873 wait complete. 2024-03-16T11:59:49.470 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.468+0000 7f7f411a5740 1 -- start start 2024-03-16T11:59:49.470 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.469+0000 7f7f411a5740 1 Processor -- start 2024-03-16T11:59:49.474 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.469+0000 7f7f411a5740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558affed7130 0x558affef3760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:59:49.474 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.469+0000 7f7f411a5740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558affed7a50 0x558affef3ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:59:49.474 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.469+0000 7f7f411a5740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558affee0820 0x558affef41e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:59:49.474 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.469+0000 7f7f411a5740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f7f381aba80 con 0x558affed7130 2024-03-16T11:59:49.474 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.469+0000 7f7f411a5740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f7f381acef0 con 0x558affed7a50 2024-03-16T11:59:49.474 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.469+0000 7f7f411a5740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f7f381ac150 con 0x558affee0820 2024-03-16T11:59:49.474 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.469+0000 7f7f411a5740 10 client.0 _ll_register_callbacks cb 0x7f7f2009f070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T11:59:49.475 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.469+0000 7f7f37fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558affed7130 0x558affef3760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:59:49.475 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.469+0000 7f7f37fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558affed7130 0x558affef3760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36780/0 (socket says 172.21.15.29:36780) 2024-03-16T11:59:49.475 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.469+0000 7f7f37fff640 1 -- 172.21.15.29:0/1132759260 learned_addr learned my addr 172.21.15.29:0/1132759260 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T11:59:49.475 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.469+0000 7f7f3ca79640 1 --2- 172.21.15.29:0/1132759260 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558affee0820 0x558affef41e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:59:49.475 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.469+0000 7f7f377fe640 1 --2- 172.21.15.29:0/1132759260 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558affed7a50 0x558affef3ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:59:49.475 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.469+0000 7f7f37fff640 1 -- 172.21.15.29:0/1132759260 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558affee0820 msgr2=0x558affef41e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:59:49.475 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.469+0000 7f7f37fff640 1 --2- 172.21.15.29:0/1132759260 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558affee0820 0x558affef41e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:59:49.475 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.469+0000 7f7f37fff640 1 -- 172.21.15.29:0/1132759260 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558affed7a50 msgr2=0x558affef3ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T11:59:49.475 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.469+0000 7f7f37fff640 1 --2- 172.21.15.29:0/1132759260 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558affed7a50 0x558affef3ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:59:49.475 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.469+0000 7f7f37fff640 1 -- 172.21.15.29:0/1132759260 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558affed7880 con 0x558affed7130 2024-03-16T11:59:49.475 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.470+0000 7f7f37fff640 1 --2- 172.21.15.29:0/1132759260 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558affed7130 0x558affef3760 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f7f28009230 tx=0x7f7f28004e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8467c5295fe54c35 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T11:59:49.475 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.470+0000 7f7f177fe640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-16T11:59:49.475 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.470+0000 7f7f177fe640 1 -- 172.21.15.29:0/1132759260 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f28059060 con 0x558affed7130 2024-03-16T11:59:49.475 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.470+0000 7f7f411a5740 1 -- 172.21.15.29:0/1132759260 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7f28048da0 con 0x558affed7130 2024-03-16T11:59:49.476 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.470+0000 7f7f177fe640 1 -- 172.21.15.29:0/1132759260 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8556+0+0 (secure 0 0 0) 0x7f7f28048480 con 0x558affed7130 2024-03-16T11:59:49.476 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.470+0000 7f7f177fe640 1 -- 172.21.15.29:0/1132759260 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f7f2804e030 con 0x558affed7130 2024-03-16T11:59:49.476 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.470+0000 7f7f411a5740 10 client.16146 fetch_fsmap learned FSMap version 26 2024-03-16T11:59:49.476 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.470+0000 7f7f411a5740 1 -- 172.21.15.29:0/1132759260 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x558afff56a90 con 0x558affed7130 2024-03-16T11:59:49.476 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.470+0000 7f7f177fe640 1 -- 172.21.15.29:0/1132759260 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7f28048da0 con 0x558affed7130 2024-03-16T11:59:49.476 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.470+0000 7f7f177fe640 1 -- 172.21.15.29:0/1132759260 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f7f2805d480 con 0x558affed7130 2024-03-16T11:59:49.476 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.470+0000 7f7f411a5740 10 client.16146 fetch_fsmap finished waiting for FSMap version 26 2024-03-16T11:59:49.476 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.470+0000 7f7f411a5740 10 client.16146 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T11:59:49.476 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.470+0000 7f7f411a5740 20 client.16146 populate_metadata read hostname 'smithi029' 2024-03-16T11:59:49.476 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.470+0000 7f7f411a5740 1 --2- 172.21.15.29:0/1132759260 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x558afff57210 0x558afff59630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T11:59:49.476 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.470+0000 7f7f411a5740 4 client.16146 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-16T11:59:49.476 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.470+0000 7f7f411a5740 1 -- 172.21.15.29:0/1132759260 --> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] -- command(tid 0: {"prefix": "flush journal"}) -- 0x558affe432d0 con 0x558afff57210 2024-03-16T11:59:49.476 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.471+0000 7f7f377fe640 1 --2- 172.21.15.29:0/1132759260 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x558afff57210 0x558afff59630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T11:59:49.476 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.471+0000 7f7f177fe640 10 client.16146 ms_handle_connect on v2:172.21.15.152:6838/1548731833 2024-03-16T11:59:49.476 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.471+0000 7f7f377fe640 1 --2- 172.21.15.29:0/1132759260 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x558afff57210 0x558afff59630 crc :-1 s=READY pgs=119 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bda0905afa4b1b01 server_cookie=7bba17e97e092d27 in_seq=0 out_seq=0 2024-03-16T11:59:49.638 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.637+0000 7f7f177fe640 1 -- 172.21.15.29:0/1132759260 <== mds.0 v2:172.21.15.152:6838/1548731833 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x558affe432d0 con 0x558afff57210 2024-03-16T11:59:49.638 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.637+0000 7f7f177fe640 10 client.16146 handle_command_reply: tid=0 2024-03-16T11:59:49.638 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.637+0000 7f7f411a5740 1 client.16146 shutdown 2024-03-16T11:59:49.638 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.638+0000 7f7f411a5740 1 -- 172.21.15.29:0/1132759260 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558affed7130 msgr2=0x558affef3760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:59:49.638 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.638+0000 7f7f411a5740 1 --2- 172.21.15.29:0/1132759260 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558affed7130 0x558affef3760 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f7f28009230 tx=0x7f7f28004e70 comp rx=0 tx=0).stop 2024-03-16T11:59:49.638 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.638+0000 7f7f411a5740 1 -- 172.21.15.29:0/1132759260 shutdown_connections 2024-03-16T11:59:49.638 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.638+0000 7f7f411a5740 1 --2- 172.21.15.29:0/1132759260 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558affee0820 0x558affef41e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:59:49.639 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.638+0000 7f7f411a5740 1 --2- 172.21.15.29:0/1132759260 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x558afff57210 0x558afff59630 crc :-1 s=READY pgs=119 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:59:49.639 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.638+0000 7f7f411a5740 1 --2- 172.21.15.29:0/1132759260 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558affed7a50 0x558affef3ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:59:49.639 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.638+0000 7f7f411a5740 1 --2- 172.21.15.29:0/1132759260 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558affed7130 0x558affef3760 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:59:49.639 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.638+0000 7f7f411a5740 1 -- 172.21.15.29:0/1132759260 >> 172.21.15.29:0/1132759260 conn(0x558affed1a50 msgr2=0x558affed2ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:59:49.639 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.638+0000 7f7f177fe640 0 client.16146 ms_handle_reset on v2:172.21.15.152:6838/1548731833 2024-03-16T11:59:49.639 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.638+0000 7f7f411a5740 1 -- 172.21.15.29:0/1132759260 shutdown_connections 2024-03-16T11:59:49.639 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.638+0000 7f7f411a5740 1 -- 172.21.15.29:0/1132759260 wait complete. 2024-03-16T11:59:49.639 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.638+0000 7f7f411a5740 20 client.16146 trim_cache size 0 max 16384 2024-03-16T11:59:49.640 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:{ 2024-03-16T11:59:49.640 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "message": "", 2024-03-16T11:59:49.640 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "return_code": 0 2024-03-16T11:59:49.640 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:} 2024-03-16T11:59:49.641 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.639+0000 7f7f3e4dc640 1 -- 172.21.15.29:0/43993373 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f7f240a2a10 msgr2=0x7f7f240a4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:59:49.641 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.639+0000 7f7f3e4dc640 1 --2- 172.21.15.29:0/43993373 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f7f240a2a10 0x7f7f240a4ed0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f7f30005fd0 tx=0x7f7f30011b40 comp rx=0 tx=0).stop 2024-03-16T11:59:49.641 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.639+0000 7f7f3e4dc640 1 -- 172.21.15.29:0/43993373 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7f38102150 msgr2=0x7f7f38198530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T11:59:49.641 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.639+0000 7f7f3e4dc640 1 --2- 172.21.15.29:0/43993373 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7f38102150 0x7f7f38198530 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f7f2000e9f0 tx=0x7f7f2000eeb0 comp rx=0 tx=0).stop 2024-03-16T11:59:49.641 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.639+0000 7f7f3e4dc640 1 -- 172.21.15.29:0/43993373 shutdown_connections 2024-03-16T11:59:49.641 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.639+0000 7f7f3e4dc640 1 --2- 172.21.15.29:0/43993373 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f7f240a2a10 0x7f7f240a4ed0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:59:49.641 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.639+0000 7f7f3e4dc640 1 --2- 172.21.15.29:0/43993373 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7f381077a0 0x7f7f3819c8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:59:49.642 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.639+0000 7f7f3e4dc640 1 --2- 172.21.15.29:0/43993373 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7f38102150 0x7f7f38198530 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:59:49.642 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.639+0000 7f7f3e4dc640 1 --2- 172.21.15.29:0/43993373 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7f38101790 0x7f7f38197ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T11:59:49.642 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.639+0000 7f7f3e4dc640 1 -- 172.21.15.29:0/43993373 >> 172.21.15.29:0/43993373 conn(0x7f7f380fd2a0 msgr2=0x7f7f380fd860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T11:59:49.642 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.639+0000 7f7f3e4dc640 1 -- 172.21.15.29:0/43993373 shutdown_connections 2024-03-16T11:59:49.642 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T11:59:49.639+0000 7f7f3e4dc640 1 -- 172.21.15.29:0/43993373 wait complete. 2024-03-16T11:59:49.796 INFO:tasks.workunit.client.0.smithi029.stdout: 1 901865 10.23 MB/sec execute 334 sec latency 112.069 ms 2024-03-16T11:59:50.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:59:50 smithi152 ceph-mon[36291]: pgmap v279: 97 pgs: 97 active+clean; 76 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 2.3 MiB/s wr, 243 op/s 2024-03-16T11:59:50.545 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:59:50 smithi154 ceph-mon[36320]: pgmap v279: 97 pgs: 97 active+clean; 76 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 2.3 MiB/s wr, 243 op/s 2024-03-16T11:59:50.587 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:59:50 smithi029 ceph-mon[31975]: pgmap v279: 97 pgs: 97 active+clean; 76 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 2.3 MiB/s wr, 243 op/s 2024-03-16T11:59:50.796 INFO:tasks.workunit.client.0.smithi029.stdout: 1 903217 10.22 MB/sec execute 335 sec latency 23.768 ms 2024-03-16T11:59:51.796 INFO:tasks.workunit.client.0.smithi029.stdout: 1 903840 10.19 MB/sec execute 336 sec latency 135.142 ms 2024-03-16T11:59:52.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:59:52 smithi152 ceph-mon[36291]: pgmap v280: 97 pgs: 97 active+clean; 66 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 3.7 MiB/s wr, 414 op/s 2024-03-16T11:59:52.545 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:59:52 smithi154 ceph-mon[36320]: pgmap v280: 97 pgs: 97 active+clean; 66 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 3.7 MiB/s wr, 414 op/s 2024-03-16T11:59:52.587 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:59:52 smithi029 ceph-mon[31975]: pgmap v280: 97 pgs: 97 active+clean; 66 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 3.7 MiB/s wr, 414 op/s 2024-03-16T11:59:52.796 INFO:tasks.workunit.client.0.smithi029.stdout: 1 905861 10.20 MB/sec execute 337 sec latency 42.302 ms 2024-03-16T11:59:53.796 INFO:tasks.workunit.client.0.smithi029.stdout: 1 907692 10.19 MB/sec execute 338 sec latency 95.588 ms 2024-03-16T11:59:54.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:59:54 smithi152 ceph-mon[36291]: pgmap v281: 97 pgs: 97 active+clean; 66 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 2.6 MiB/s wr, 264 op/s 2024-03-16T11:59:54.545 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:59:54 smithi154 ceph-mon[36320]: pgmap v281: 97 pgs: 97 active+clean; 66 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 2.6 MiB/s wr, 264 op/s 2024-03-16T11:59:54.587 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:59:54 smithi029 ceph-mon[31975]: pgmap v281: 97 pgs: 97 active+clean; 66 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 2.6 MiB/s wr, 264 op/s 2024-03-16T11:59:54.796 INFO:tasks.workunit.client.0.smithi029.stdout: 1 909517 10.20 MB/sec execute 339 sec latency 28.538 ms 2024-03-16T11:59:55.796 INFO:tasks.workunit.client.0.smithi029.stdout: 1 911598 10.20 MB/sec execute 340 sec latency 21.226 ms 2024-03-16T11:59:56.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:59:56 smithi152 ceph-mon[36291]: pgmap v282: 97 pgs: 97 active+clean; 66 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 2.6 MiB/s wr, 254 op/s 2024-03-16T11:59:56.545 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:59:56 smithi154 ceph-mon[36320]: pgmap v282: 97 pgs: 97 active+clean; 66 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 2.6 MiB/s wr, 254 op/s 2024-03-16T11:59:56.588 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:59:56 smithi029 ceph-mon[31975]: pgmap v282: 97 pgs: 97 active+clean; 66 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 2.6 MiB/s wr, 254 op/s 2024-03-16T11:59:56.796 INFO:tasks.workunit.client.0.smithi029.stdout: 1 913201 10.19 MB/sec execute 341 sec latency 37.918 ms 2024-03-16T11:59:57.796 INFO:tasks.workunit.client.0.smithi029.stdout: 1 915296 10.20 MB/sec execute 342 sec latency 70.050 ms 2024-03-16T11:59:58.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 11:59:58 smithi152 ceph-mon[36291]: pgmap v283: 97 pgs: 97 active+clean; 72 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 3.8 MiB/s wr, 335 op/s 2024-03-16T11:59:58.390 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 11:59:58 smithi029 ceph-mon[31975]: pgmap v283: 97 pgs: 97 active+clean; 72 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 3.8 MiB/s wr, 335 op/s 2024-03-16T11:59:58.545 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 11:59:58 smithi154 ceph-mon[36320]: pgmap v283: 97 pgs: 97 active+clean; 72 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 3.8 MiB/s wr, 335 op/s 2024-03-16T11:59:58.797 INFO:tasks.workunit.client.0.smithi029.stdout: 1 916688 10.19 MB/sec execute 343 sec latency 121.844 ms 2024-03-16T11:59:59.796 INFO:tasks.workunit.client.0.smithi029.stdout: 1 917965 10.23 MB/sec execute 344 sec latency 115.981 ms 2024-03-16T12:00:00.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:00:00 smithi152 ceph-mon[36291]: pgmap v284: 97 pgs: 97 active+clean; 72 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 2.6 MiB/s wr, 252 op/s 2024-03-16T12:00:00.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:00:00 smithi152 ceph-mon[36291]: overall HEALTH_OK 2024-03-16T12:00:00.545 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:00:00 smithi154 ceph-mon[36320]: pgmap v284: 97 pgs: 97 active+clean; 72 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 2.6 MiB/s wr, 252 op/s 2024-03-16T12:00:00.545 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:00:00 smithi154 ceph-mon[36320]: overall HEALTH_OK 2024-03-16T12:00:00.587 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:00:00 smithi029 ceph-mon[31975]: pgmap v284: 97 pgs: 97 active+clean; 72 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 2.6 MiB/s wr, 252 op/s 2024-03-16T12:00:00.588 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:00:00 smithi029 ceph-mon[31975]: overall HEALTH_OK 2024-03-16T12:00:00.796 INFO:tasks.workunit.client.0.smithi029.stdout: 1 920377 10.24 MB/sec execute 345 sec latency 22.196 ms 2024-03-16T12:00:01.796 INFO:tasks.workunit.client.0.smithi029.stdout: 1 921280 10.22 MB/sec execute 346 sec latency 27.172 ms 2024-03-16T12:00:02.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:00:02 smithi152 ceph-mon[36291]: pgmap v285: 97 pgs: 97 active+clean; 78 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 4.0 MiB/s wr, 334 op/s 2024-03-16T12:00:02.545 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:00:02 smithi154 ceph-mon[36320]: pgmap v285: 97 pgs: 97 active+clean; 78 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 4.0 MiB/s wr, 334 op/s 2024-03-16T12:00:02.588 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:00:02 smithi029 ceph-mon[31975]: pgmap v285: 97 pgs: 97 active+clean; 78 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 4.0 MiB/s wr, 334 op/s 2024-03-16T12:00:02.730 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:02.728+0000 7f09827d3640 1 -- 172.21.15.29:0/4293417800 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f097c1029c0 msgr2=0x7f097c102da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:00:02.730 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:02.728+0000 7f09827d3640 1 --2- 172.21.15.29:0/4293417800 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f097c1029c0 0x7f097c102da0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f096c009900 tx=0x7f096c02bc30 comp rx=0 tx=0).stop 2024-03-16T12:00:02.730 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:02.728+0000 7f09827d3640 1 -- 172.21.15.29:0/4293417800 shutdown_connections 2024-03-16T12:00:02.730 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:02.728+0000 7f09827d3640 1 --2- 172.21.15.29:0/4293417800 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f097c10be30 0x7f097c10e2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:00:02.730 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:02.728+0000 7f09827d3640 1 --2- 172.21.15.29:0/4293417800 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f097c1032e0 0x7f097c10b7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:00:02.731 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:02.728+0000 7f09827d3640 1 --2- 172.21.15.29:0/4293417800 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f097c1029c0 0x7f097c102da0 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:00:02.731 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:02.728+0000 7f09827d3640 1 -- 172.21.15.29:0/4293417800 >> 172.21.15.29:0/4293417800 conn(0x7f097c077810 msgr2=0x7f097c077c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:00:02.731 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:02.728+0000 7f09827d3640 1 -- 172.21.15.29:0/4293417800 shutdown_connections 2024-03-16T12:00:02.731 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:02.728+0000 7f09827d3640 1 -- 172.21.15.29:0/4293417800 wait complete. 2024-03-16T12:00:02.731 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:02.729+0000 7f09827d3640 1 Processor -- start 2024-03-16T12:00:02.731 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:02.729+0000 7f09827d3640 1 -- start start 2024-03-16T12:00:02.731 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:02.729+0000 7f09827d3640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f097c1029c0 0x7f097c1980b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:00:02.731 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:02.729+0000 7f09827d3640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f097c1032e0 0x7f097c1985f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:00:02.731 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:02.729+0000 7f09827d3640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f097c10be30 0x7f097c19c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:00:02.731 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:02.729+0000 7f09827d3640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f097c111d10 con 0x7f097c1029c0 2024-03-16T12:00:02.731 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:02.729+0000 7f09827d3640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f097c111b90 con 0x7f097c10be30 2024-03-16T12:00:02.731 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:02.729+0000 7f09827d3640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f097c111e90 con 0x7f097c1032e0 2024-03-16T12:00:02.731 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:02.729+0000 7f097bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f097c1029c0 0x7f097c1980b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:00:02.732 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:02.729+0000 7f097bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f097c1029c0 0x7f097c1980b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50472/0 (socket says 172.21.15.29:50472) 2024-03-16T12:00:02.732 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:02.729+0000 7f097bfff640 1 -- 172.21.15.29:0/42148377 learned_addr learned my addr 172.21.15.29:0/42148377 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T12:00:02.732 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:02.729+0000 7f0980d70640 1 --2- 172.21.15.29:0/42148377 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f097c10be30 0x7f097c19c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:00:02.732 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:02.729+0000 7f097b7fe640 1 --2- 172.21.15.29:0/42148377 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f097c1032e0 0x7f097c1985f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:00:02.732 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:02.729+0000 7f097bfff640 1 -- 172.21.15.29:0/42148377 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f097c1032e0 msgr2=0x7f097c1985f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:00:02.732 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:02.729+0000 7f097bfff640 1 --2- 172.21.15.29:0/42148377 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f097c1032e0 0x7f097c1985f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:00:02.732 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:02.729+0000 7f097bfff640 1 -- 172.21.15.29:0/42148377 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f097c10be30 msgr2=0x7f097c19c920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:00:02.732 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:02.729+0000 7f097bfff640 1 --2- 172.21.15.29:0/42148377 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f097c10be30 0x7f097c19c920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:00:02.732 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:02.729+0000 7f097bfff640 1 -- 172.21.15.29:0/42148377 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f097c19d050 con 0x7f097c1029c0 2024-03-16T12:00:02.732 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:02.730+0000 7f097bfff640 1 --2- 172.21.15.29:0/42148377 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f097c1029c0 0x7f097c1980b0 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f096c005ab0 tx=0x7f096c0045e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:00:02.732 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:02.730+0000 7f09797fa640 1 -- 172.21.15.29:0/42148377 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f096c043020 con 0x7f097c1029c0 2024-03-16T12:00:02.732 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:02.730+0000 7f09797fa640 1 -- 172.21.15.29:0/42148377 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f096c0054e0 con 0x7f097c1029c0 2024-03-16T12:00:02.732 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:02.730+0000 7f09827d3640 1 -- 172.21.15.29:0/42148377 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f097c19d2e0 con 0x7f097c1029c0 2024-03-16T12:00:02.732 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:02.730+0000 7f09797fa640 1 -- 172.21.15.29:0/42148377 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f096c003c90 con 0x7f097c1029c0 2024-03-16T12:00:02.733 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:02.730+0000 7f09827d3640 1 -- 172.21.15.29:0/42148377 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f097c1a4bd0 con 0x7f097c1029c0 2024-03-16T12:00:02.739 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:02.732+0000 7f09797fa640 1 -- 172.21.15.29:0/42148377 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7f096c046020 con 0x7f097c1029c0 2024-03-16T12:00:02.739 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:02.732+0000 7f09853b6740 1 Processor -- start 2024-03-16T12:00:02.739 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:02.732+0000 7f09853b6740 1 -- start start 2024-03-16T12:00:02.739 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:02.733+0000 7f09853b6740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561040d09130 0x561040d09510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:00:02.739 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:02.733+0000 7f09853b6740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561040d09a50 0x561040d122e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:00:02.739 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:02.733+0000 7f09853b6740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561040d12820 0x561040d14c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:00:02.739 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:02.733+0000 7f09853b6740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x561040c6da40 con 0x561040d12820 2024-03-16T12:00:02.739 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:02.733+0000 7f09853b6740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x561040c16920 con 0x561040d09a50 2024-03-16T12:00:02.739 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:02.733+0000 7f09853b6740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x561040c69ae0 con 0x561040d09130 2024-03-16T12:00:02.740 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:02.733+0000 7f09797fa640 1 --2- 172.21.15.29:0/42148377 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f09640a2a10 0x7f09640a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:00:02.740 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:02.733+0000 7f097b7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561040d12820 0x561040d14c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:00:02.740 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:02.733+0000 7f097bfff640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561040d09a50 0x561040d122e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:00:02.740 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:02.733+0000 7f0980d70640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561040d09130 0x561040d09510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:00:02.740 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:02.733+0000 7f097b7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561040d12820 0x561040d14c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50478/0 (socket says 172.21.15.29:50478) 2024-03-16T12:00:02.740 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:02.733+0000 7f097b7fe640 1 -- 172.21.15.29:0/3688070871 learned_addr learned my addr 172.21.15.29:0/3688070871 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T12:00:02.740 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:02.733+0000 7f0980d70640 1 --2- 172.21.15.29:0/42148377 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f09640a2a10 0x7f09640a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:00:02.740 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:02.733+0000 7f09797fa640 1 -- 172.21.15.29:0/42148377 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 8556+0+0 (secure 0 0 0) 0x7f096c0f0fa0 con 0x7f097c1029c0 2024-03-16T12:00:02.740 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:02.733+0000 7f097b7fe640 1 -- 172.21.15.29:0/3688070871 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561040d09130 msgr2=0x561040d09510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:00:02.740 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:02.733+0000 7f097b7fe640 1 --2- 172.21.15.29:0/3688070871 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561040d09130 0x561040d09510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:00:02.740 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:02.733+0000 7f097b7fe640 1 -- 172.21.15.29:0/3688070871 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561040d09a50 msgr2=0x561040d122e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:00:02.740 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:02.733+0000 7f097b7fe640 1 --2- 172.21.15.29:0/3688070871 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561040d09a50 0x561040d122e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:00:02.740 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:02.733+0000 7f097b7fe640 1 -- 172.21.15.29:0/3688070871 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561040c2e460 con 0x561040d12820 2024-03-16T12:00:02.740 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:02.733+0000 7f097b7fe640 1 --2- 172.21.15.29:0/3688070871 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561040d12820 0x561040d14c10 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f0970009a40 tx=0x7f0970009f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8536079e278661db server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:00:02.740 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:02.733+0000 7f0980d70640 1 --2- 172.21.15.29:0/42148377 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f09640a2a10 0x7f09640a4ed0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f097400bd50 tx=0x7f0974009650 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:00:02.741 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:02.734+0000 7f09627fc640 1 -- 172.21.15.29:0/3688070871 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0970017070 con 0x561040d12820 2024-03-16T12:00:02.741 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:02.734+0000 7f09627fc640 1 -- 172.21.15.29:0/3688070871 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0970013470 con 0x561040d12820 2024-03-16T12:00:02.741 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:02.734+0000 7f09627fc640 1 -- 172.21.15.29:0/3688070871 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0970012630 con 0x561040d12820 2024-03-16T12:00:02.741 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:02.734+0000 7f09853b6740 1 -- 172.21.15.29:0/3688070871 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561040d12820 msgr2=0x561040d14c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:00:02.741 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:02.734+0000 7f09853b6740 1 --2- 172.21.15.29:0/3688070871 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561040d12820 0x561040d14c10 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f0970009a40 tx=0x7f0970009f00 comp rx=0 tx=0).stop 2024-03-16T12:00:02.741 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:02.734+0000 7f09853b6740 1 -- 172.21.15.29:0/3688070871 shutdown_connections 2024-03-16T12:00:02.741 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:02.734+0000 7f09853b6740 1 --2- 172.21.15.29:0/3688070871 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561040d12820 0x561040d14c10 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:00:02.741 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:02.734+0000 7f09853b6740 1 --2- 172.21.15.29:0/3688070871 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561040d09a50 0x561040d122e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:00:02.741 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:02.734+0000 7f09853b6740 1 --2- 172.21.15.29:0/3688070871 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561040d09130 0x561040d09510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:00:02.741 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:02.734+0000 7f09853b6740 1 -- 172.21.15.29:0/3688070871 >> 172.21.15.29:0/3688070871 conn(0x561040d03a50 msgr2=0x561040d03e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:00:02.741 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:02.734+0000 7f09853b6740 1 -- 172.21.15.29:0/3688070871 shutdown_connections 2024-03-16T12:00:02.741 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:02.734+0000 7f09853b6740 1 -- 172.21.15.29:0/3688070871 wait complete. 2024-03-16T12:00:02.741 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:02.735+0000 7f09853b6740 1 -- start start 2024-03-16T12:00:02.741 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:02.735+0000 7f09853b6740 1 Processor -- start 2024-03-16T12:00:02.741 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:02.735+0000 7f09853b6740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561040d09130 0x561040d0f7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:00:02.742 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:02.735+0000 7f09853b6740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561040d09a50 0x561040d0fd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:00:02.742 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:02.735+0000 7f09853b6740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561040d12820 0x561040d10270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:00:02.742 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:02.735+0000 7f09853b6740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x561040d15450 con 0x561040d12820 2024-03-16T12:00:02.742 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:02.735+0000 7f09853b6740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x561040c29d70 con 0x561040d09130 2024-03-16T12:00:02.742 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:02.735+0000 7f09853b6740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x561040d155d0 con 0x561040d09a50 2024-03-16T12:00:02.742 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:02.735+0000 7f09853b6740 10 client.0 _ll_register_callbacks cb 0x561040981a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T12:00:02.742 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:02.735+0000 7f0980d70640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561040d12820 0x561040d10270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:00:02.742 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:02.735+0000 7f0980d70640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561040d12820 0x561040d10270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50492/0 (socket says 172.21.15.29:50492) 2024-03-16T12:00:02.742 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:02.735+0000 7f097bfff640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561040d09130 0x561040d0f7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:00:02.742 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:02.735+0000 7f0980d70640 1 -- 172.21.15.29:0/4284853399 learned_addr learned my addr 172.21.15.29:0/4284853399 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T12:00:02.742 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:02.735+0000 7f097b7fe640 1 --2- 172.21.15.29:0/4284853399 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561040d09a50 0x561040d0fd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:00:02.742 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:02.736+0000 7f0980d70640 1 -- 172.21.15.29:0/4284853399 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561040d09a50 msgr2=0x561040d0fd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:00:02.742 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:02.736+0000 7f0980d70640 1 --2- 172.21.15.29:0/4284853399 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561040d09a50 0x561040d0fd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:00:02.742 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:02.736+0000 7f0980d70640 1 -- 172.21.15.29:0/4284853399 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561040d09130 msgr2=0x561040d0f7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:00:02.742 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:02.736+0000 7f0980d70640 1 --2- 172.21.15.29:0/4284853399 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561040d09130 0x561040d0f7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:00:02.743 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:02.736+0000 7f0980d70640 1 -- 172.21.15.29:0/4284853399 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561040982210 con 0x561040d12820 2024-03-16T12:00:02.743 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:02.736+0000 7f0980d70640 1 --2- 172.21.15.29:0/4284853399 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561040d12820 0x561040d10270 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f0974017e50 tx=0x7f0974004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1fa573bfd00f179d server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:00:02.743 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:02.736+0000 7f095b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-16T12:00:02.743 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:02.736+0000 7f095b7fe640 1 -- 172.21.15.29:0/4284853399 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f097402b060 con 0x561040d12820 2024-03-16T12:00:02.743 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:02.736+0000 7f095b7fe640 1 -- 172.21.15.29:0/4284853399 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8556+0+0 (secure 0 0 0) 0x7f097401bd90 con 0x561040d12820 2024-03-16T12:00:02.743 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:02.736+0000 7f09853b6740 1 -- 172.21.15.29:0/4284853399 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0970013470 con 0x561040d12820 2024-03-16T12:00:02.743 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:02.736+0000 7f095b7fe640 1 -- 172.21.15.29:0/4284853399 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f0974014040 con 0x561040d12820 2024-03-16T12:00:02.743 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:02.736+0000 7f09853b6740 10 client.16158 fetch_fsmap learned FSMap version 26 2024-03-16T12:00:02.743 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:02.736+0000 7f09853b6740 1 -- 172.21.15.29:0/4284853399 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x561040a4e620 con 0x561040d12820 2024-03-16T12:00:02.743 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:02.736+0000 7f095b7fe640 1 -- 172.21.15.29:0/4284853399 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0970013470 con 0x561040d12820 2024-03-16T12:00:02.743 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:02.736+0000 7f095b7fe640 1 -- 172.21.15.29:0/4284853399 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f097402e3d0 con 0x561040d12820 2024-03-16T12:00:02.743 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:02.737+0000 7f09853b6740 10 client.16158 fetch_fsmap finished waiting for FSMap version 26 2024-03-16T12:00:02.743 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:02.737+0000 7f09853b6740 10 client.16158 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T12:00:02.743 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:02.737+0000 7f09853b6740 20 client.16158 populate_metadata read hostname 'smithi029' 2024-03-16T12:00:02.743 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:02.737+0000 7f09853b6740 1 --2- 172.21.15.29:0/4284853399 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x561040d229a0 0x561040d24db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:00:02.743 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:02.737+0000 7f09853b6740 4 client.16158 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T12:00:02.744 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:02.737+0000 7f09853b6740 1 -- 172.21.15.29:0/4284853399 --> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561040c7af80 con 0x561040d229a0 2024-03-16T12:00:02.744 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:02.737+0000 7f097bfff640 1 --2- 172.21.15.29:0/4284853399 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x561040d229a0 0x561040d24db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:00:02.744 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:02.737+0000 7f095b7fe640 10 client.16158 ms_handle_connect on v2:172.21.15.152:6838/1548731833 2024-03-16T12:00:02.744 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:02.737+0000 7f097bfff640 1 --2- 172.21.15.29:0/4284853399 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x561040d229a0 0x561040d24db0 crc :-1 s=READY pgs=120 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=75f3d931e0a31fb0 server_cookie=e001f418c07cd07e in_seq=0 out_seq=0 2024-03-16T12:00:02.744 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:02.738+0000 7f095b7fe640 1 -- 172.21.15.29:0/4284853399 <== mds.0 v2:172.21.15.152:6838/1548731833 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x561040c7af80 con 0x561040d229a0 2024-03-16T12:00:02.744 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:02.738+0000 7f095b7fe640 10 client.16158 handle_command_reply: tid=0 2024-03-16T12:00:02.744 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:02.738+0000 7f09853b6740 1 client.16158 shutdown 2024-03-16T12:00:02.744 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:02.738+0000 7f09853b6740 1 -- 172.21.15.29:0/4284853399 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561040d12820 msgr2=0x561040d10270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:00:02.744 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:02.738+0000 7f09853b6740 1 --2- 172.21.15.29:0/4284853399 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561040d12820 0x561040d10270 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f0974017e50 tx=0x7f0974004a40 comp rx=0 tx=0).stop 2024-03-16T12:00:02.744 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:02.738+0000 7f09853b6740 1 -- 172.21.15.29:0/4284853399 shutdown_connections 2024-03-16T12:00:02.744 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:02.738+0000 7f09853b6740 1 --2- 172.21.15.29:0/4284853399 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561040d12820 0x561040d10270 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:00:02.744 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:02.738+0000 7f09853b6740 1 --2- 172.21.15.29:0/4284853399 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561040d09a50 0x561040d0fd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:00:02.744 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:02.738+0000 7f09853b6740 1 --2- 172.21.15.29:0/4284853399 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x561040d229a0 0x561040d24db0 crc :-1 s=READY pgs=120 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:00:02.744 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:02.738+0000 7f09853b6740 1 --2- 172.21.15.29:0/4284853399 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561040d09130 0x561040d0f7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:00:02.744 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:02.738+0000 7f095b7fe640 0 client.16158 ms_handle_reset on v2:172.21.15.152:6838/1548731833 2024-03-16T12:00:02.745 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:02.738+0000 7f09853b6740 1 -- 172.21.15.29:0/4284853399 >> 172.21.15.29:0/4284853399 conn(0x561040d03a50 msgr2=0x561040d057a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:00:02.745 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:02.738+0000 7f09853b6740 1 -- 172.21.15.29:0/4284853399 shutdown_connections 2024-03-16T12:00:02.745 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:02.738+0000 7f09853b6740 1 -- 172.21.15.29:0/4284853399 wait complete. 2024-03-16T12:00:02.745 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:02.738+0000 7f09853b6740 20 client.16158 trim_cache size 0 max 16384 2024-03-16T12:00:02.758 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:02.753+0000 7f09853b6740 1 Processor -- start 2024-03-16T12:00:02.758 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:02.753+0000 7f09853b6740 1 -- start start 2024-03-16T12:00:02.758 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:02.753+0000 7f09853b6740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561040d09130 0x561040d11f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:00:02.758 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:02.753+0000 7f09853b6740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561040d09a50 0x561040d1e1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:00:02.758 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:02.753+0000 7f09853b6740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561040d12820 0x561040cfea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:00:02.758 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:02.753+0000 7f09853b6740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f097c1afd20 con 0x561040d09130 2024-03-16T12:00:02.758 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:02.753+0000 7f09853b6740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f097c1ad5f0 con 0x561040d09a50 2024-03-16T12:00:02.758 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:02.753+0000 7f09853b6740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f097c1ab460 con 0x561040d12820 2024-03-16T12:00:02.759 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:02.754+0000 7f097bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561040d09130 0x561040d11f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:00:02.759 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:02.754+0000 7f097bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561040d09130 0x561040d11f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50498/0 (socket says 172.21.15.29:50498) 2024-03-16T12:00:02.759 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:02.754+0000 7f097bfff640 1 -- 172.21.15.29:0/1810462500 learned_addr learned my addr 172.21.15.29:0/1810462500 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T12:00:02.759 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:02.754+0000 7f097b7fe640 1 --2- 172.21.15.29:0/1810462500 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561040d09a50 0x561040d1e1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:00:02.759 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:02.754+0000 7f0980d70640 1 --2- 172.21.15.29:0/1810462500 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561040d12820 0x561040cfea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:00:02.759 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:02.754+0000 7f097bfff640 1 -- 172.21.15.29:0/1810462500 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561040d12820 msgr2=0x561040cfea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:00:02.759 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:02.754+0000 7f097bfff640 1 --2- 172.21.15.29:0/1810462500 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561040d12820 0x561040cfea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:00:02.759 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:02.754+0000 7f097bfff640 1 -- 172.21.15.29:0/1810462500 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561040d09a50 msgr2=0x561040d1e1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:00:02.759 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:02.754+0000 7f097bfff640 1 --2- 172.21.15.29:0/1810462500 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561040d09a50 0x561040d1e1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:00:02.759 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:02.754+0000 7f097bfff640 1 -- 172.21.15.29:0/1810462500 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561040d0e7a0 con 0x561040d09130 2024-03-16T12:00:02.760 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:02.754+0000 7f097bfff640 1 --2- 172.21.15.29:0/1810462500 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561040d09130 0x561040d11f20 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f096c0bd3f0 tx=0x7f096c0904f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=199dbffda0d8d55a server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:00:02.760 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:02.754+0000 7f0961ffb640 1 -- 172.21.15.29:0/1810462500 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f096c004c80 con 0x561040d09130 2024-03-16T12:00:02.760 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:02.754+0000 7f0961ffb640 1 -- 172.21.15.29:0/1810462500 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f096c003e30 con 0x561040d09130 2024-03-16T12:00:02.760 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:02.754+0000 7f0961ffb640 1 -- 172.21.15.29:0/1810462500 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f096c0c5e90 con 0x561040d09130 2024-03-16T12:00:02.760 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:02.755+0000 7f09853b6740 1 -- 172.21.15.29:0/1810462500 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561040d09130 msgr2=0x561040d11f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:00:02.760 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:02.755+0000 7f09853b6740 1 --2- 172.21.15.29:0/1810462500 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561040d09130 0x561040d11f20 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f096c0bd3f0 tx=0x7f096c0904f0 comp rx=0 tx=0).stop 2024-03-16T12:00:02.760 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:02.755+0000 7f09853b6740 1 -- 172.21.15.29:0/1810462500 shutdown_connections 2024-03-16T12:00:02.760 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:02.755+0000 7f09853b6740 1 --2- 172.21.15.29:0/1810462500 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561040d12820 0x561040cfea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:00:02.760 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:02.755+0000 7f09853b6740 1 --2- 172.21.15.29:0/1810462500 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561040d09a50 0x561040d1e1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:00:02.760 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:02.755+0000 7f09853b6740 1 --2- 172.21.15.29:0/1810462500 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561040d09130 0x561040d11f20 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:00:02.761 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:02.755+0000 7f09853b6740 1 -- 172.21.15.29:0/1810462500 >> 172.21.15.29:0/1810462500 conn(0x561040d03a50 msgr2=0x561040d05730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:00:02.761 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:02.755+0000 7f09853b6740 1 -- 172.21.15.29:0/1810462500 shutdown_connections 2024-03-16T12:00:02.761 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:02.755+0000 7f09853b6740 1 -- 172.21.15.29:0/1810462500 wait complete. 2024-03-16T12:00:02.761 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:02.755+0000 7f09853b6740 1 -- start start 2024-03-16T12:00:02.761 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:02.755+0000 7f09853b6740 1 Processor -- start 2024-03-16T12:00:02.761 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:02.756+0000 7f09853b6740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561040d09130 0x561040d162b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:00:02.761 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:02.756+0000 7f09853b6740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561040d09a50 0x561040d167f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:00:02.761 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:02.756+0000 7f09853b6740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561040d12820 0x561040d16d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:00:02.761 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:02.756+0000 7f09853b6740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f097c1abab0 con 0x561040d09130 2024-03-16T12:00:02.762 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:02.756+0000 7f09853b6740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f097c1acf20 con 0x561040d09a50 2024-03-16T12:00:02.762 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:02.756+0000 7f09853b6740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f097c1ac180 con 0x561040d12820 2024-03-16T12:00:02.762 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:02.756+0000 7f09853b6740 10 client.0 _ll_register_callbacks cb 0x7f0974009e00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T12:00:02.762 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:02.756+0000 7f097bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561040d09130 0x561040d162b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:00:02.762 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:02.756+0000 7f097bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561040d09130 0x561040d162b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50508/0 (socket says 172.21.15.29:50508) 2024-03-16T12:00:02.762 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:02.756+0000 7f097bfff640 1 -- 172.21.15.29:0/264816306 learned_addr learned my addr 172.21.15.29:0/264816306 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T12:00:02.762 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:02.756+0000 7f0980d70640 1 --2- 172.21.15.29:0/264816306 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561040d12820 0x561040d16d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:00:02.762 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:02.756+0000 7f097b7fe640 1 --2- 172.21.15.29:0/264816306 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561040d09a50 0x561040d167f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:00:02.762 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:02.756+0000 7f097bfff640 1 -- 172.21.15.29:0/264816306 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561040d12820 msgr2=0x561040d16d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:00:02.762 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:02.756+0000 7f097bfff640 1 --2- 172.21.15.29:0/264816306 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561040d12820 0x561040d16d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:00:02.762 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:02.756+0000 7f097bfff640 1 -- 172.21.15.29:0/264816306 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561040d09a50 msgr2=0x561040d167f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:00:02.762 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:02.756+0000 7f097bfff640 1 --2- 172.21.15.29:0/264816306 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561040d09a50 0x561040d167f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:00:02.763 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:02.756+0000 7f097bfff640 1 -- 172.21.15.29:0/264816306 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561040d87ec0 con 0x561040d09130 2024-03-16T12:00:02.763 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:02.756+0000 7f097bfff640 1 --2- 172.21.15.29:0/264816306 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561040d09130 0x561040d162b0 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f096c02c4e0 tx=0x7f096c005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f44f98f9da4df301 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:00:02.763 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:02.756+0000 7f095b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-16T12:00:02.763 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:02.756+0000 7f095b7fe640 1 -- 172.21.15.29:0/264816306 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f096c08d060 con 0x561040d09130 2024-03-16T12:00:02.763 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:02.756+0000 7f095b7fe640 1 -- 172.21.15.29:0/264816306 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8556+0+0 (secure 0 0 0) 0x7f096c0b9d60 con 0x561040d09130 2024-03-16T12:00:02.763 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:02.756+0000 7f09853b6740 1 -- 172.21.15.29:0/264816306 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f096c003e30 con 0x561040d09130 2024-03-16T12:00:02.763 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:02.757+0000 7f095b7fe640 1 -- 172.21.15.29:0/264816306 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f096c004c80 con 0x561040d09130 2024-03-16T12:00:02.763 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:02.757+0000 7f09853b6740 10 client.16164 fetch_fsmap learned FSMap version 26 2024-03-16T12:00:02.763 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:02.757+0000 7f09853b6740 1 -- 172.21.15.29:0/264816306 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x561040d89190 con 0x561040d09130 2024-03-16T12:00:02.763 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:02.757+0000 7f095b7fe640 1 -- 172.21.15.29:0/264816306 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f096c082e50 con 0x561040d09130 2024-03-16T12:00:02.763 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:02.757+0000 7f095b7fe640 1 -- 172.21.15.29:0/264816306 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f096c0c5a00 con 0x561040d09130 2024-03-16T12:00:02.763 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:02.757+0000 7f09853b6740 10 client.16164 fetch_fsmap finished waiting for FSMap version 26 2024-03-16T12:00:02.763 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:02.757+0000 7f09853b6740 10 client.16164 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T12:00:02.763 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:02.757+0000 7f09853b6740 20 client.16164 populate_metadata read hostname 'smithi029' 2024-03-16T12:00:02.763 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:02.757+0000 7f09853b6740 1 --2- 172.21.15.29:0/264816306 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x561040d897a0 0x561040d8bb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:00:02.764 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:02.757+0000 7f09853b6740 4 client.16164 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-16T12:00:02.764 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:02.757+0000 7f09853b6740 1 -- 172.21.15.29:0/264816306 --> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] -- command(tid 0: {"prefix": "flush journal"}) -- 0x561040c752d0 con 0x561040d897a0 2024-03-16T12:00:02.764 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:02.757+0000 7f097b7fe640 1 --2- 172.21.15.29:0/264816306 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x561040d897a0 0x561040d8bb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:00:02.764 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:02.758+0000 7f097b7fe640 1 --2- 172.21.15.29:0/264816306 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x561040d897a0 0x561040d8bb90 crc :-1 s=READY pgs=121 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ff2589a8f64f1db4 server_cookie=9c60fdca55ea67fc in_seq=0 out_seq=0 2024-03-16T12:00:02.764 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:02.758+0000 7f095b7fe640 10 client.16164 ms_handle_connect on v2:172.21.15.152:6838/1548731833 2024-03-16T12:00:02.797 INFO:tasks.workunit.client.0.smithi029.stdout: 1 922842 10.21 MB/sec execute 347 sec latency 68.070 ms 2024-03-16T12:00:02.959 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:02.959+0000 7f095b7fe640 1 -- 172.21.15.29:0/264816306 <== mds.0 v2:172.21.15.152:6838/1548731833 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x561040c752d0 con 0x561040d897a0 2024-03-16T12:00:02.959 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:02.959+0000 7f095b7fe640 10 client.16164 handle_command_reply: tid=0 2024-03-16T12:00:02.959 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:02.959+0000 7f09853b6740 1 client.16164 shutdown 2024-03-16T12:00:02.959 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:02.959+0000 7f09853b6740 1 -- 172.21.15.29:0/264816306 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561040d09130 msgr2=0x561040d162b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:00:02.960 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:02.959+0000 7f09853b6740 1 --2- 172.21.15.29:0/264816306 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561040d09130 0x561040d162b0 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f096c02c4e0 tx=0x7f096c005fd0 comp rx=0 tx=0).stop 2024-03-16T12:00:02.960 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:02.960+0000 7f09853b6740 1 -- 172.21.15.29:0/264816306 shutdown_connections 2024-03-16T12:00:02.960 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:02.960+0000 7f09853b6740 1 --2- 172.21.15.29:0/264816306 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561040d12820 0x561040d16d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:00:02.960 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:02.960+0000 7f09853b6740 1 --2- 172.21.15.29:0/264816306 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x561040d897a0 0x561040d8bb90 crc :-1 s=READY pgs=121 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:00:02.960 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:02.960+0000 7f09853b6740 1 --2- 172.21.15.29:0/264816306 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561040d09a50 0x561040d167f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:00:02.960 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:02.960+0000 7f09853b6740 1 --2- 172.21.15.29:0/264816306 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561040d09130 0x561040d162b0 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:00:02.960 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:02.960+0000 7f09853b6740 1 -- 172.21.15.29:0/264816306 >> 172.21.15.29:0/264816306 conn(0x561040d03a50 msgr2=0x561040d02320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:00:02.960 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:02.960+0000 7f095b7fe640 0 client.16164 ms_handle_reset on v2:172.21.15.152:6838/1548731833 2024-03-16T12:00:02.960 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:02.960+0000 7f09853b6740 1 -- 172.21.15.29:0/264816306 shutdown_connections 2024-03-16T12:00:02.960 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:02.960+0000 7f09853b6740 1 -- 172.21.15.29:0/264816306 wait complete. 2024-03-16T12:00:02.960 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:02.960+0000 7f09853b6740 20 client.16164 trim_cache size 0 max 16384 2024-03-16T12:00:02.960 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:{ 2024-03-16T12:00:02.961 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "message": "", 2024-03-16T12:00:02.961 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "return_code": 0 2024-03-16T12:00:02.961 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:} 2024-03-16T12:00:02.962 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:02.961+0000 7f09827d3640 1 -- 172.21.15.29:0/42148377 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f09640a2a10 msgr2=0x7f09640a4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:00:02.962 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:02.961+0000 7f09827d3640 1 --2- 172.21.15.29:0/42148377 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f09640a2a10 0x7f09640a4ed0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f097400bd50 tx=0x7f0974009650 comp rx=0 tx=0).stop 2024-03-16T12:00:02.962 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:02.961+0000 7f09827d3640 1 -- 172.21.15.29:0/42148377 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f097c1029c0 msgr2=0x7f097c1980b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:00:02.962 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:02.961+0000 7f09827d3640 1 --2- 172.21.15.29:0/42148377 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f097c1029c0 0x7f097c1980b0 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f096c005ab0 tx=0x7f096c0045e0 comp rx=0 tx=0).stop 2024-03-16T12:00:02.962 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:02.961+0000 7f09827d3640 1 -- 172.21.15.29:0/42148377 shutdown_connections 2024-03-16T12:00:02.962 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:02.961+0000 7f09827d3640 1 --2- 172.21.15.29:0/42148377 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f09640a2a10 0x7f09640a4ed0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:00:02.962 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:02.961+0000 7f09827d3640 1 --2- 172.21.15.29:0/42148377 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f097c10be30 0x7f097c19c920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:00:02.962 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:02.961+0000 7f09827d3640 1 --2- 172.21.15.29:0/42148377 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f097c1032e0 0x7f097c1985f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:00:02.963 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:02.961+0000 7f09827d3640 1 --2- 172.21.15.29:0/42148377 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f097c1029c0 0x7f097c1980b0 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:00:02.963 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:02.961+0000 7f09827d3640 1 -- 172.21.15.29:0/42148377 >> 172.21.15.29:0/42148377 conn(0x7f097c077810 msgr2=0x7f097c0fd5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:00:02.963 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:02.961+0000 7f09827d3640 1 -- 172.21.15.29:0/42148377 shutdown_connections 2024-03-16T12:00:02.963 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:02.961+0000 7f09827d3640 1 -- 172.21.15.29:0/42148377 wait complete. 2024-03-16T12:00:03.797 INFO:tasks.workunit.client.0.smithi029.stdout: 1 924511 10.21 MB/sec execute 348 sec latency 125.048 ms 2024-03-16T12:00:04.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:00:04 smithi152 ceph-mon[36291]: pgmap v286: 97 pgs: 97 active+clean; 78 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.6 MiB/s wr, 163 op/s 2024-03-16T12:00:04.545 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:00:04 smithi154 ceph-mon[36320]: pgmap v286: 97 pgs: 97 active+clean; 78 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.6 MiB/s wr, 163 op/s 2024-03-16T12:00:04.588 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:00:04 smithi029 ceph-mon[31975]: pgmap v286: 97 pgs: 97 active+clean; 78 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.6 MiB/s wr, 163 op/s 2024-03-16T12:00:04.797 INFO:tasks.workunit.client.0.smithi029.stdout: 1 925485 10.20 MB/sec execute 349 sec latency 69.040 ms 2024-03-16T12:00:05.545 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:00:05 smithi154 ceph-mon[36320]: osdmap e98: 12 total, 12 up, 12 in 2024-03-16T12:00:05.588 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:00:05 smithi029 ceph-mon[31975]: osdmap e98: 12 total, 12 up, 12 in 2024-03-16T12:00:05.627 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:00:05 smithi152 ceph-mon[36291]: osdmap e98: 12 total, 12 up, 12 in 2024-03-16T12:00:05.797 INFO:tasks.workunit.client.0.smithi029.stdout: 1 926281 10.18 MB/sec execute 350 sec latency 58.821 ms 2024-03-16T12:00:06.545 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:00:06 smithi154 ceph-mon[36320]: pgmap v288: 97 pgs: 97 active+clean; 78 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 3.1 MiB/s wr, 196 op/s 2024-03-16T12:00:06.588 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:00:06 smithi029 ceph-mon[31975]: pgmap v288: 97 pgs: 97 active+clean; 78 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 3.1 MiB/s wr, 196 op/s 2024-03-16T12:00:06.627 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:00:06 smithi152 ceph-mon[36291]: pgmap v288: 97 pgs: 97 active+clean; 78 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 3.1 MiB/s wr, 196 op/s 2024-03-16T12:00:06.797 INFO:tasks.workunit.client.0.smithi029.stdout: 1 927722 10.18 MB/sec execute 351 sec latency 21.573 ms 2024-03-16T12:00:07.797 INFO:tasks.workunit.client.0.smithi029.stdout: 1 929149 10.17 MB/sec execute 352 sec latency 49.572 ms 2024-03-16T12:00:08.545 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:00:08 smithi154 ceph-mon[36320]: pgmap v289: 97 pgs: 16 active+clean+snaptrim, 34 active+clean+snaptrim_wait, 47 active+clean; 67 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.1 MiB/s wr, 315 op/s 2024-03-16T12:00:08.587 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:00:08 smithi029 ceph-mon[31975]: pgmap v289: 97 pgs: 16 active+clean+snaptrim, 34 active+clean+snaptrim_wait, 47 active+clean; 67 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.1 MiB/s wr, 315 op/s 2024-03-16T12:00:08.627 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:00:08 smithi152 ceph-mon[36291]: pgmap v289: 97 pgs: 16 active+clean+snaptrim, 34 active+clean+snaptrim_wait, 47 active+clean; 67 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.1 MiB/s wr, 315 op/s 2024-03-16T12:00:08.797 INFO:tasks.workunit.client.0.smithi029.stdout: 1 930829 10.17 MB/sec execute 353 sec latency 362.705 ms 2024-03-16T12:00:09.545 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:00:09 smithi154 ceph-mon[36320]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T12:00:09.588 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:00:09 smithi029 ceph-mon[31975]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T12:00:09.627 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:00:09 smithi152 ceph-mon[36291]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T12:00:09.797 INFO:tasks.workunit.client.0.smithi029.stdout: 1 930829 10.14 MB/sec execute 354 sec latency 1362.776 ms 2024-03-16T12:00:10.545 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:00:10 smithi154 ceph-mon[36320]: pgmap v290: 97 pgs: 16 active+clean+snaptrim, 34 active+clean+snaptrim_wait, 47 active+clean; 67 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.1 MiB/s wr, 315 op/s 2024-03-16T12:00:10.545 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:00:10 smithi154 ceph-mon[36320]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:00:10.545 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:00:10 smithi154 ceph-mon[36320]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T12:00:10.545 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:00:10 smithi154 ceph-mon[36320]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' 2024-03-16T12:00:10.588 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:00:10 smithi029 ceph-mon[31975]: pgmap v290: 97 pgs: 16 active+clean+snaptrim, 34 active+clean+snaptrim_wait, 47 active+clean; 67 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.1 MiB/s wr, 315 op/s 2024-03-16T12:00:10.588 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:00:10 smithi029 ceph-mon[31975]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:00:10.588 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:00:10 smithi029 ceph-mon[31975]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T12:00:10.588 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:00:10 smithi029 ceph-mon[31975]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' 2024-03-16T12:00:10.627 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:00:10 smithi152 ceph-mon[36291]: pgmap v290: 97 pgs: 16 active+clean+snaptrim, 34 active+clean+snaptrim_wait, 47 active+clean; 67 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.1 MiB/s wr, 315 op/s 2024-03-16T12:00:10.627 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:00:10 smithi152 ceph-mon[36291]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:00:10.627 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:00:10 smithi152 ceph-mon[36291]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T12:00:10.627 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:00:10 smithi152 ceph-mon[36291]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' 2024-03-16T12:00:10.797 INFO:tasks.workunit.client.0.smithi029.stdout: 1 930829 10.11 MB/sec execute 355 sec latency 2362.862 ms 2024-03-16T12:00:11.798 INFO:tasks.workunit.client.0.smithi029.stdout: 1 930829 10.09 MB/sec execute 356 sec latency 3362.900 ms 2024-03-16T12:00:12.545 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:00:12 smithi154 ceph-mon[36320]: pgmap v291: 97 pgs: 2 active+clean+snaptrim, 4 active+clean+snaptrim_wait, 91 active+clean; 69 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 870 KiB/s rd, 2.4 MiB/s wr, 275 op/s 2024-03-16T12:00:12.588 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:00:12 smithi029 ceph-mon[31975]: pgmap v291: 97 pgs: 2 active+clean+snaptrim, 4 active+clean+snaptrim_wait, 91 active+clean; 69 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 870 KiB/s rd, 2.4 MiB/s wr, 275 op/s 2024-03-16T12:00:12.627 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:00:12 smithi152 ceph-mon[36291]: pgmap v291: 97 pgs: 2 active+clean+snaptrim, 4 active+clean+snaptrim_wait, 91 active+clean; 69 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 870 KiB/s rd, 2.4 MiB/s wr, 275 op/s 2024-03-16T12:00:12.798 INFO:tasks.workunit.client.0.smithi029.stdout: 1 930829 10.06 MB/sec execute 357 sec latency 4363.027 ms 2024-03-16T12:00:13.798 INFO:tasks.workunit.client.0.smithi029.stdout: 1 930829 10.03 MB/sec execute 358 sec latency 5363.127 ms 2024-03-16T12:00:14.429 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:00:14 smithi152 ceph-mon[36291]: pgmap v292: 97 pgs: 97 active+clean; 69 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 870 KiB/s rd, 2.4 MiB/s wr, 278 op/s 2024-03-16T12:00:14.545 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:00:14 smithi154 ceph-mon[36320]: pgmap v292: 97 pgs: 97 active+clean; 69 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 870 KiB/s rd, 2.4 MiB/s wr, 278 op/s 2024-03-16T12:00:14.588 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:00:14 smithi029 ceph-mon[31975]: pgmap v292: 97 pgs: 97 active+clean; 69 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 870 KiB/s rd, 2.4 MiB/s wr, 278 op/s 2024-03-16T12:00:14.798 INFO:tasks.workunit.client.0.smithi029.stdout: 1 930829 10.00 MB/sec execute 359 sec latency 6363.248 ms 2024-03-16T12:00:15.798 INFO:tasks.workunit.client.0.smithi029.stdout: 1 930829 9.97 MB/sec execute 360 sec latency 7363.377 ms 2024-03-16T12:00:16.066 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:16.065+0000 7fbf60ff4640 1 -- 172.21.15.29:0/1587392973 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbf5c1078b0 msgr2=0x7fbf5c107c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:00:16.066 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:16.065+0000 7fbf60ff4640 1 --2- 172.21.15.29:0/1587392973 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbf5c1078b0 0x7fbf5c107c90 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7fbf4c009930 tx=0x7fbf4c02bc30 comp rx=0 tx=0).stop 2024-03-16T12:00:16.066 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:16.065+0000 7fbf60ff4640 1 -- 172.21.15.29:0/1587392973 shutdown_connections 2024-03-16T12:00:16.066 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:16.065+0000 7fbf60ff4640 1 --2- 172.21.15.29:0/1587392973 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbf5c069800 0x7fbf5c10e670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:00:16.066 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:16.065+0000 7fbf60ff4640 1 --2- 172.21.15.29:0/1587392973 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbf5c068e60 0x7fbf5c0692c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:00:16.066 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:16.065+0000 7fbf60ff4640 1 --2- 172.21.15.29:0/1587392973 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbf5c1078b0 0x7fbf5c107c90 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:00:16.067 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:16.065+0000 7fbf60ff4640 1 -- 172.21.15.29:0/1587392973 >> 172.21.15.29:0/1587392973 conn(0x7fbf5c077530 msgr2=0x7fbf5c077940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:00:16.067 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:16.065+0000 7fbf60ff4640 1 -- 172.21.15.29:0/1587392973 shutdown_connections 2024-03-16T12:00:16.067 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:16.065+0000 7fbf60ff4640 1 -- 172.21.15.29:0/1587392973 wait complete. 2024-03-16T12:00:16.067 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:16.066+0000 7fbf60ff4640 1 Processor -- start 2024-03-16T12:00:16.067 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:16.066+0000 7fbf60ff4640 1 -- start start 2024-03-16T12:00:16.067 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:16.066+0000 7fbf60ff4640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbf5c068e60 0x7fbf5c197f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:00:16.067 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:16.066+0000 7fbf60ff4640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbf5c069800 0x7fbf5c1984a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:00:16.067 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:16.066+0000 7fbf60ff4640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbf5c1078b0 0x7fbf5c19c7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:00:16.067 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:16.066+0000 7fbf60ff4640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fbf5c111ba0 con 0x7fbf5c1078b0 2024-03-16T12:00:16.067 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:16.066+0000 7fbf60ff4640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fbf5c111a20 con 0x7fbf5c069800 2024-03-16T12:00:16.067 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:16.066+0000 7fbf60ff4640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fbf5c111d20 con 0x7fbf5c068e60 2024-03-16T12:00:16.067 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:16.066+0000 7fbf5ad9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbf5c1078b0 0x7fbf5c19c7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:00:16.067 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:16.066+0000 7fbf5ad9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbf5c1078b0 0x7fbf5c19c7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55832/0 (socket says 172.21.15.29:55832) 2024-03-16T12:00:16.067 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:16.066+0000 7fbf5ad9d640 1 -- 172.21.15.29:0/3403821178 learned_addr learned my addr 172.21.15.29:0/3403821178 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T12:00:16.068 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:16.066+0000 7fbf5a59c640 1 --2- 172.21.15.29:0/3403821178 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbf5c068e60 0x7fbf5c197f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:00:16.068 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:16.066+0000 7fbf59d9b640 1 --2- 172.21.15.29:0/3403821178 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbf5c069800 0x7fbf5c1984a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:00:16.068 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:16.066+0000 7fbf5ad9d640 1 -- 172.21.15.29:0/3403821178 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbf5c068e60 msgr2=0x7fbf5c197f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:00:16.068 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:16.066+0000 7fbf5ad9d640 1 --2- 172.21.15.29:0/3403821178 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbf5c068e60 0x7fbf5c197f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:00:16.068 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:16.066+0000 7fbf5ad9d640 1 -- 172.21.15.29:0/3403821178 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbf5c069800 msgr2=0x7fbf5c1984a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:00:16.068 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:16.066+0000 7fbf5ad9d640 1 --2- 172.21.15.29:0/3403821178 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbf5c069800 0x7fbf5c1984a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:00:16.068 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:16.066+0000 7fbf5ad9d640 1 -- 172.21.15.29:0/3403821178 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf5c19cf00 con 0x7fbf5c1078b0 2024-03-16T12:00:16.068 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:16.066+0000 7fbf5a59c640 1 --2- 172.21.15.29:0/3403821178 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbf5c068e60 0x7fbf5c197f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T12:00:16.068 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:16.066+0000 7fbf59d9b640 1 --2- 172.21.15.29:0/3403821178 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbf5c069800 0x7fbf5c1984a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T12:00:16.068 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:16.067+0000 7fbf5ad9d640 1 --2- 172.21.15.29:0/3403821178 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbf5c1078b0 0x7fbf5c19c7d0 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7fbf5400ea60 tx=0x7fbf5400ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:00:16.068 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:16.067+0000 7fbf4b7fe640 1 -- 172.21.15.29:0/3403821178 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf5400ce40 con 0x7fbf5c1078b0 2024-03-16T12:00:16.068 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:16.067+0000 7fbf4b7fe640 1 -- 172.21.15.29:0/3403821178 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbf54005020 con 0x7fbf5c1078b0 2024-03-16T12:00:16.068 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:16.067+0000 7fbf60ff4640 1 -- 172.21.15.29:0/3403821178 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbf5c19d1f0 con 0x7fbf5c1078b0 2024-03-16T12:00:16.068 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:16.067+0000 7fbf4b7fe640 1 -- 172.21.15.29:0/3403821178 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf54010690 con 0x7fbf5c1078b0 2024-03-16T12:00:16.069 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:16.067+0000 7fbf60ff4640 1 -- 172.21.15.29:0/3403821178 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbf5c1a4a80 con 0x7fbf5c1078b0 2024-03-16T12:00:16.075 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:16.069+0000 7fbf4b7fe640 1 -- 172.21.15.29:0/3403821178 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7fbf54005520 con 0x7fbf5c1078b0 2024-03-16T12:00:16.075 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:16.070+0000 7fbf63be6740 1 Processor -- start 2024-03-16T12:00:16.075 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:16.070+0000 7fbf63be6740 1 -- start start 2024-03-16T12:00:16.075 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:16.070+0000 7fbf4b7fe640 1 --2- 172.21.15.29:0/3403821178 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7fbf440a2a90 0x7fbf440a4f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:00:16.075 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:16.070+0000 7fbf63be6740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bc14332130 0x55bc14332510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:00:16.075 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:16.070+0000 7fbf63be6740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55bc14332a50 0x55bc1433b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:00:16.075 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:16.070+0000 7fbf63be6740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bc1433b820 0x55bc1433dc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:00:16.076 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:16.070+0000 7fbf63be6740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55bc14296a40 con 0x55bc1433b820 2024-03-16T12:00:16.076 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:16.070+0000 7fbf63be6740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x55bc1423f920 con 0x55bc14332a50 2024-03-16T12:00:16.076 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:16.070+0000 7fbf63be6740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55bc14292ae0 con 0x55bc14332130 2024-03-16T12:00:16.076 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:16.070+0000 7fbf59d9b640 1 --2- 172.21.15.29:0/3403821178 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7fbf440a2a90 0x7fbf440a4f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:00:16.076 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:16.070+0000 7fbf4b7fe640 1 -- 172.21.15.29:0/3403821178 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 8600+0+0 (secure 0 0 0) 0x7fbf540cf490 con 0x7fbf5c1078b0 2024-03-16T12:00:16.076 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:16.070+0000 7fbf5a59c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bc14332130 0x55bc14332510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:00:16.076 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:16.070+0000 7fbf59d9b640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55bc14332a50 0x55bc1433b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:00:16.076 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:16.070+0000 7fbf5a59c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bc14332130 0x55bc14332510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.29:58812/0 (socket says 172.21.15.29:58812) 2024-03-16T12:00:16.076 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:16.070+0000 7fbf5a59c640 1 -- 172.21.15.29:0/665277546 learned_addr learned my addr 172.21.15.29:0/665277546 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T12:00:16.076 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:16.070+0000 7fbf5ad9d640 1 --2- 172.21.15.29:0/665277546 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bc1433b820 0x55bc1433dc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:00:16.076 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:16.071+0000 7fbf5a59c640 1 -- 172.21.15.29:0/665277546 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55bc14332a50 msgr2=0x55bc1433b2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:00:16.076 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:16.071+0000 7fbf5a59c640 1 --2- 172.21.15.29:0/665277546 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55bc14332a50 0x55bc1433b2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:00:16.076 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:16.071+0000 7fbf5a59c640 1 -- 172.21.15.29:0/665277546 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bc1433b820 msgr2=0x55bc1433dc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:00:16.076 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:16.071+0000 7fbf5a59c640 1 --2- 172.21.15.29:0/665277546 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bc1433b820 0x55bc1433dc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:00:16.076 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:16.071+0000 7fbf5a59c640 1 -- 172.21.15.29:0/665277546 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bc14257460 con 0x55bc14332130 2024-03-16T12:00:16.076 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:16.071+0000 7fbf59d9b640 1 --2- 172.21.15.29:0/3403821178 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7fbf440a2a90 0x7fbf440a4f50 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fbf5c077a20 tx=0x7fbf50009b10 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:00:16.077 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:16.071+0000 7fbf5a59c640 1 --2- 172.21.15.29:0/665277546 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bc14332130 0x55bc14332510 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fbf4c000c00 tx=0x7fbf4c005e50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ab848299632509ff server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:00:16.077 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:16.071+0000 7fbf48ff9640 1 -- 172.21.15.29:0/665277546 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf4c042070 con 0x55bc14332130 2024-03-16T12:00:16.077 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:16.071+0000 7fbf48ff9640 1 -- 172.21.15.29:0/665277546 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbf4c004240 con 0x55bc14332130 2024-03-16T12:00:16.077 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:16.071+0000 7fbf48ff9640 1 -- 172.21.15.29:0/665277546 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf4c0046c0 con 0x55bc14332130 2024-03-16T12:00:16.077 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:16.072+0000 7fbf63be6740 1 -- 172.21.15.29:0/665277546 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bc14332130 msgr2=0x55bc14332510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:00:16.077 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:16.072+0000 7fbf63be6740 1 --2- 172.21.15.29:0/665277546 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bc14332130 0x55bc14332510 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fbf4c000c00 tx=0x7fbf4c005e50 comp rx=0 tx=0).stop 2024-03-16T12:00:16.077 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:16.072+0000 7fbf63be6740 1 -- 172.21.15.29:0/665277546 shutdown_connections 2024-03-16T12:00:16.077 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:16.072+0000 7fbf63be6740 1 --2- 172.21.15.29:0/665277546 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bc1433b820 0x55bc1433dc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:00:16.077 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:16.072+0000 7fbf63be6740 1 --2- 172.21.15.29:0/665277546 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55bc14332a50 0x55bc1433b2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:00:16.077 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:16.072+0000 7fbf63be6740 1 --2- 172.21.15.29:0/665277546 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bc14332130 0x55bc14332510 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:00:16.077 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:16.072+0000 7fbf63be6740 1 -- 172.21.15.29:0/665277546 >> 172.21.15.29:0/665277546 conn(0x55bc1432ca50 msgr2=0x55bc1432ce30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:00:16.077 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:16.072+0000 7fbf63be6740 1 -- 172.21.15.29:0/665277546 shutdown_connections 2024-03-16T12:00:16.077 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:16.072+0000 7fbf63be6740 1 -- 172.21.15.29:0/665277546 wait complete. 2024-03-16T12:00:16.077 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:16.072+0000 7fbf63be6740 1 -- start start 2024-03-16T12:00:16.077 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:16.072+0000 7fbf63be6740 1 Processor -- start 2024-03-16T12:00:16.077 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:16.073+0000 7fbf63be6740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bc14332130 0x55bc14338840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:00:16.078 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:16.073+0000 7fbf63be6740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bc14332a50 0x55bc14338d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:00:16.078 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:16.073+0000 7fbf63be6740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55bc1433b820 0x55bc143392c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:00:16.078 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:16.073+0000 7fbf63be6740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55bc1433e3f0 con 0x55bc14332130 2024-03-16T12:00:16.078 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:16.073+0000 7fbf63be6740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x55bc14252d70 con 0x55bc1433b820 2024-03-16T12:00:16.078 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:16.073+0000 7fbf63be6740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55bc1433e570 con 0x55bc14332a50 2024-03-16T12:00:16.078 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:16.073+0000 7fbf63be6740 10 client.0 _ll_register_callbacks cb 0x55bc13faaa00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T12:00:16.078 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:16.073+0000 7fbf5a59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bc14332130 0x55bc14338840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:00:16.078 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:16.073+0000 7fbf5a59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bc14332130 0x55bc14338840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55850/0 (socket says 172.21.15.29:55850) 2024-03-16T12:00:16.078 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:16.073+0000 7fbf5a59c640 1 -- 172.21.15.29:0/4114528265 learned_addr learned my addr 172.21.15.29:0/4114528265 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T12:00:16.078 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:16.073+0000 7fbf5ad9d640 1 --2- 172.21.15.29:0/4114528265 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55bc1433b820 0x55bc143392c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:00:16.078 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:16.073+0000 7fbf59d9b640 1 --2- 172.21.15.29:0/4114528265 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bc14332a50 0x55bc14338d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:00:16.078 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:16.073+0000 7fbf5ad9d640 1 -- 172.21.15.29:0/4114528265 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bc14332a50 msgr2=0x55bc14338d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:00:16.078 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:16.073+0000 7fbf5ad9d640 1 --2- 172.21.15.29:0/4114528265 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bc14332a50 0x55bc14338d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:00:16.078 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:16.073+0000 7fbf5ad9d640 1 -- 172.21.15.29:0/4114528265 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bc14332130 msgr2=0x55bc14338840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:00:16.079 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:16.073+0000 7fbf5ad9d640 1 --2- 172.21.15.29:0/4114528265 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bc14332130 0x55bc14338840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:00:16.079 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:16.073+0000 7fbf5ad9d640 1 -- 172.21.15.29:0/4114528265 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bc13fab210 con 0x55bc1433b820 2024-03-16T12:00:16.079 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:16.073+0000 7fbf5a59c640 1 --2- 172.21.15.29:0/4114528265 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bc14332130 0x55bc14338840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:00:16.079 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:16.073+0000 7fbf59d9b640 1 --2- 172.21.15.29:0/4114528265 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bc14332a50 0x55bc14338d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:00:16.079 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:16.073+0000 7fbf5ad9d640 1 --2- 172.21.15.29:0/4114528265 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55bc1433b820 0x55bc143392c0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fbf540ccb30 tx=0x7fbf5409c1c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a311ffa1d9063da4 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:00:16.079 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:16.073+0000 7fbf41ffb640 10 client.0 ms_handle_connect on v2:172.21.15.152:3300/0 2024-03-16T12:00:16.079 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:16.073+0000 7fbf41ffb640 1 -- 172.21.15.29:0/4114528265 <== mon.2 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf540b3070 con 0x55bc1433b820 2024-03-16T12:00:16.079 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:16.073+0000 7fbf41ffb640 1 -- 172.21.15.29:0/4114528265 <== mon.2 v2:172.21.15.152:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 8600+0+0 (secure 0 0 0) 0x7fbf540980b0 con 0x55bc1433b820 2024-03-16T12:00:16.079 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:16.074+0000 7fbf63be6740 1 -- 172.21.15.29:0/4114528265 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbf4c004240 con 0x55bc1433b820 2024-03-16T12:00:16.079 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:16.074+0000 7fbf41ffb640 1 -- 172.21.15.29:0/4114528265 <== mon.2 v2:172.21.15.152:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fbf540a41e0 con 0x55bc1433b820 2024-03-16T12:00:16.079 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:16.074+0000 7fbf63be6740 10 client.25988 fetch_fsmap learned FSMap version 26 2024-03-16T12:00:16.079 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:16.074+0000 7fbf63be6740 1 -- 172.21.15.29:0/4114528265 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55bc14077620 con 0x55bc1433b820 2024-03-16T12:00:16.079 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:16.074+0000 7fbf41ffb640 1 -- 172.21.15.29:0/4114528265 <== mon.2 v2:172.21.15.152:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbf4c004240 con 0x55bc1433b820 2024-03-16T12:00:16.079 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:16.074+0000 7fbf41ffb640 1 -- 172.21.15.29:0/4114528265 <== mon.2 v2:172.21.15.152:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7fbf5409ae90 con 0x55bc1433b820 2024-03-16T12:00:16.079 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:16.074+0000 7fbf63be6740 10 client.25988 fetch_fsmap finished waiting for FSMap version 26 2024-03-16T12:00:16.079 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:16.074+0000 7fbf63be6740 10 client.25988 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T12:00:16.080 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:16.074+0000 7fbf63be6740 20 client.25988 populate_metadata read hostname 'smithi029' 2024-03-16T12:00:16.080 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:16.074+0000 7fbf63be6740 1 --2- 172.21.15.29:0/4114528265 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55bc1434b920 0x55bc1434dd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:00:16.080 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:16.074+0000 7fbf63be6740 4 client.25988 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T12:00:16.080 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:16.074+0000 7fbf63be6740 1 -- 172.21.15.29:0/4114528265 --> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bc142a3f80 con 0x55bc1434b920 2024-03-16T12:00:16.080 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:16.074+0000 7fbf5a59c640 1 --2- 172.21.15.29:0/4114528265 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55bc1434b920 0x55bc1434dd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:00:16.080 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:16.075+0000 7fbf41ffb640 10 client.25988 ms_handle_connect on v2:172.21.15.152:6838/1548731833 2024-03-16T12:00:16.080 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:16.075+0000 7fbf5a59c640 1 --2- 172.21.15.29:0/4114528265 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55bc1434b920 0x55bc1434dd70 crc :-1 s=READY pgs=122 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5ff55fe64d01d39 server_cookie=5d28afccbfba1e6a in_seq=0 out_seq=0 2024-03-16T12:00:16.080 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:16.075+0000 7fbf41ffb640 1 -- 172.21.15.29:0/4114528265 <== mds.0 v2:172.21.15.152:6838/1548731833 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55bc142a3f80 con 0x55bc1434b920 2024-03-16T12:00:16.080 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:16.075+0000 7fbf41ffb640 10 client.25988 handle_command_reply: tid=0 2024-03-16T12:00:16.080 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:16.075+0000 7fbf63be6740 1 client.25988 shutdown 2024-03-16T12:00:16.080 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:16.076+0000 7fbf63be6740 1 -- 172.21.15.29:0/4114528265 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55bc1433b820 msgr2=0x55bc143392c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:00:16.080 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:16.076+0000 7fbf63be6740 1 --2- 172.21.15.29:0/4114528265 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55bc1433b820 0x55bc143392c0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fbf540ccb30 tx=0x7fbf5409c1c0 comp rx=0 tx=0).stop 2024-03-16T12:00:16.080 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:16.076+0000 7fbf63be6740 1 -- 172.21.15.29:0/4114528265 shutdown_connections 2024-03-16T12:00:16.080 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:16.076+0000 7fbf63be6740 1 --2- 172.21.15.29:0/4114528265 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55bc1434b920 0x55bc1434dd70 crc :-1 s=READY pgs=122 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:00:16.080 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:16.076+0000 7fbf63be6740 1 --2- 172.21.15.29:0/4114528265 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55bc1433b820 0x55bc143392c0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:00:16.080 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:16.076+0000 7fbf41ffb640 0 client.25988 ms_handle_reset on v2:172.21.15.152:6838/1548731833 2024-03-16T12:00:16.081 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:16.076+0000 7fbf63be6740 1 --2- 172.21.15.29:0/4114528265 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bc14332a50 0x55bc14338d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:00:16.081 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:16.076+0000 7fbf63be6740 1 --2- 172.21.15.29:0/4114528265 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bc14332130 0x55bc14338840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:00:16.081 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:16.076+0000 7fbf63be6740 1 -- 172.21.15.29:0/4114528265 >> 172.21.15.29:0/4114528265 conn(0x55bc1432ca50 msgr2=0x55bc1432e7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:00:16.081 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:16.076+0000 7fbf63be6740 1 -- 172.21.15.29:0/4114528265 shutdown_connections 2024-03-16T12:00:16.081 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:16.076+0000 7fbf63be6740 1 -- 172.21.15.29:0/4114528265 wait complete. 2024-03-16T12:00:16.081 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:16.076+0000 7fbf63be6740 20 client.25988 trim_cache size 0 max 16384 2024-03-16T12:00:16.094 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:16.091+0000 7fbf63be6740 1 Processor -- start 2024-03-16T12:00:16.094 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:16.091+0000 7fbf63be6740 1 -- start start 2024-03-16T12:00:16.095 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:16.091+0000 7fbf63be6740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bc14332130 0x55bc1433af70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:00:16.095 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:16.091+0000 7fbf63be6740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bc14332a50 0x55bc14347190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:00:16.095 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:16.091+0000 7fbf63be6740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55bc1433b820 0x55bc14353680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:00:16.095 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:16.091+0000 7fbf63be6740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fbf5c1afc40 con 0x55bc14332130 2024-03-16T12:00:16.095 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:16.091+0000 7fbf63be6740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fbf5c1ad510 con 0x55bc1433b820 2024-03-16T12:00:16.095 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:16.091+0000 7fbf63be6740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fbf5c1ab380 con 0x55bc14332a50 2024-03-16T12:00:16.095 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:16.091+0000 7fbf5a59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bc14332130 0x55bc1433af70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:00:16.095 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:16.091+0000 7fbf5a59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bc14332130 0x55bc1433af70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55860/0 (socket says 172.21.15.29:55860) 2024-03-16T12:00:16.095 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:16.091+0000 7fbf5a59c640 1 -- 172.21.15.29:0/859066484 learned_addr learned my addr 172.21.15.29:0/859066484 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T12:00:16.095 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:16.091+0000 7fbf59d9b640 1 --2- 172.21.15.29:0/859066484 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bc14332a50 0x55bc14347190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:00:16.095 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:16.091+0000 7fbf5ad9d640 1 --2- 172.21.15.29:0/859066484 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55bc1433b820 0x55bc14353680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:00:16.095 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:16.091+0000 7fbf5a59c640 1 -- 172.21.15.29:0/859066484 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bc14332a50 msgr2=0x55bc14347190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:00:16.095 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:16.091+0000 7fbf5a59c640 1 --2- 172.21.15.29:0/859066484 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bc14332a50 0x55bc14347190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:00:16.095 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:16.091+0000 7fbf5a59c640 1 -- 172.21.15.29:0/859066484 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55bc1433b820 msgr2=0x55bc14353680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:00:16.095 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:16.091+0000 7fbf5a59c640 1 --2- 172.21.15.29:0/859066484 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55bc1433b820 0x55bc14353680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:00:16.096 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:16.091+0000 7fbf5a59c640 1 -- 172.21.15.29:0/859066484 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bc1432c880 con 0x55bc14332130 2024-03-16T12:00:16.096 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:16.091+0000 7fbf5a59c640 1 --2- 172.21.15.29:0/859066484 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bc14332130 0x55bc1433af70 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7fbf4c000c00 tx=0x7fbf4c005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9fd14d644d49579 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:00:16.096 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:16.092+0000 7fbf43fff640 1 -- 172.21.15.29:0/859066484 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf4c042040 con 0x55bc14332130 2024-03-16T12:00:16.096 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:16.092+0000 7fbf43fff640 1 -- 172.21.15.29:0/859066484 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbf4c02cd40 con 0x55bc14332130 2024-03-16T12:00:16.096 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:16.092+0000 7fbf43fff640 1 -- 172.21.15.29:0/859066484 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf4c047870 con 0x55bc14332130 2024-03-16T12:00:16.096 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:16.092+0000 7fbf63be6740 1 -- 172.21.15.29:0/859066484 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bc14332130 msgr2=0x55bc1433af70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:00:16.096 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:16.092+0000 7fbf63be6740 1 --2- 172.21.15.29:0/859066484 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bc14332130 0x55bc1433af70 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7fbf4c000c00 tx=0x7fbf4c005fd0 comp rx=0 tx=0).stop 2024-03-16T12:00:16.096 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:16.092+0000 7fbf63be6740 1 -- 172.21.15.29:0/859066484 shutdown_connections 2024-03-16T12:00:16.096 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:16.092+0000 7fbf63be6740 1 --2- 172.21.15.29:0/859066484 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55bc1433b820 0x55bc14353680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:00:16.096 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:16.092+0000 7fbf63be6740 1 --2- 172.21.15.29:0/859066484 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bc14332a50 0x55bc14347190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:00:16.096 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:16.092+0000 7fbf63be6740 1 --2- 172.21.15.29:0/859066484 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bc14332130 0x55bc1433af70 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:00:16.096 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:16.092+0000 7fbf63be6740 1 -- 172.21.15.29:0/859066484 >> 172.21.15.29:0/859066484 conn(0x55bc1432ca50 msgr2=0x55bc1432ce30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:00:16.096 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:16.092+0000 7fbf63be6740 1 -- 172.21.15.29:0/859066484 shutdown_connections 2024-03-16T12:00:16.096 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:16.092+0000 7fbf63be6740 1 -- 172.21.15.29:0/859066484 wait complete. 2024-03-16T12:00:16.096 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:16.092+0000 7fbf63be6740 1 -- start start 2024-03-16T12:00:16.097 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:16.093+0000 7fbf63be6740 1 Processor -- start 2024-03-16T12:00:16.097 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:16.093+0000 7fbf63be6740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bc14332130 0x55bc143436e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:00:16.097 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:16.093+0000 7fbf63be6740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bc14332a50 0x55bc14343c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:00:16.097 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:16.093+0000 7fbf63be6740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55bc1433b820 0x55bc1433ed70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:00:16.097 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:16.093+0000 7fbf63be6740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fbf5c1ab9d0 con 0x55bc14332a50 2024-03-16T12:00:16.097 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:16.093+0000 7fbf63be6740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fbf5c1ace40 con 0x55bc1433b820 2024-03-16T12:00:16.097 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:16.093+0000 7fbf63be6740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fbf5c1ac0a0 con 0x55bc14332130 2024-03-16T12:00:16.097 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:16.093+0000 7fbf63be6740 10 client.0 _ll_register_callbacks cb 0x7fbf5409b580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T12:00:16.097 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:16.093+0000 7fbf59d9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bc14332a50 0x55bc14343c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:00:16.097 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:16.093+0000 7fbf59d9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bc14332a50 0x55bc14343c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55866/0 (socket says 172.21.15.29:55866) 2024-03-16T12:00:16.097 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:16.093+0000 7fbf59d9b640 1 -- 172.21.15.29:0/1351468286 learned_addr learned my addr 172.21.15.29:0/1351468286 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T12:00:16.097 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:16.093+0000 7fbf5a59c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bc14332130 0x55bc143436e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:00:16.097 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:16.093+0000 7fbf5ad9d640 1 --2- 172.21.15.29:0/1351468286 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55bc1433b820 0x55bc1433ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:00:16.097 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:16.093+0000 7fbf59d9b640 1 -- 172.21.15.29:0/1351468286 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bc14332130 msgr2=0x55bc143436e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:00:16.097 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:16.093+0000 7fbf59d9b640 1 --2- 172.21.15.29:0/1351468286 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bc14332130 0x55bc143436e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:00:16.098 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:16.093+0000 7fbf59d9b640 1 -- 172.21.15.29:0/1351468286 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55bc1433b820 msgr2=0x55bc1433ed70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:00:16.098 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:16.093+0000 7fbf59d9b640 1 --2- 172.21.15.29:0/1351468286 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55bc1433b820 0x55bc1433ed70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:00:16.098 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:16.093+0000 7fbf59d9b640 1 -- 172.21.15.29:0/1351468286 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bc1433f5a0 con 0x55bc14332a50 2024-03-16T12:00:16.098 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:16.093+0000 7fbf59d9b640 1 --2- 172.21.15.29:0/1351468286 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bc14332a50 0x55bc14343c20 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x55bc14331be0 tx=0x7fbf50004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb692b9fddbd3fdc server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:00:16.098 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:16.093+0000 7fbf41ffb640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-16T12:00:16.098 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:16.094+0000 7fbf41ffb640 1 -- 172.21.15.29:0/1351468286 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf5002f060 con 0x55bc14332a50 2024-03-16T12:00:16.098 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:16.094+0000 7fbf41ffb640 1 -- 172.21.15.29:0/1351468286 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 8600+0+0 (secure 0 0 0) 0x7fbf5001c040 con 0x55bc14332a50 2024-03-16T12:00:16.098 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:16.094+0000 7fbf63be6740 1 -- 172.21.15.29:0/1351468286 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbf4c02cd40 con 0x55bc14332a50 2024-03-16T12:00:16.098 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:16.094+0000 7fbf41ffb640 1 -- 172.21.15.29:0/1351468286 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fbf50003cd0 con 0x55bc14332a50 2024-03-16T12:00:16.098 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:16.094+0000 7fbf63be6740 10 client.16182 fetch_fsmap learned FSMap version 26 2024-03-16T12:00:16.098 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:16.094+0000 7fbf63be6740 1 -- 172.21.15.29:0/1351468286 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55bc143b19c0 con 0x55bc14332a50 2024-03-16T12:00:16.098 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:16.094+0000 7fbf41ffb640 1 -- 172.21.15.29:0/1351468286 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbf4c02cd40 con 0x55bc14332a50 2024-03-16T12:00:16.098 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:16.094+0000 7fbf41ffb640 1 -- 172.21.15.29:0/1351468286 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7fbf50029070 con 0x55bc14332a50 2024-03-16T12:00:16.099 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:16.094+0000 7fbf63be6740 10 client.16182 fetch_fsmap finished waiting for FSMap version 26 2024-03-16T12:00:16.099 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:16.094+0000 7fbf63be6740 10 client.16182 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T12:00:16.099 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:16.094+0000 7fbf63be6740 20 client.16182 populate_metadata read hostname 'smithi029' 2024-03-16T12:00:16.099 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:16.094+0000 7fbf63be6740 1 --2- 172.21.15.29:0/1351468286 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55bc1434b920 0x55bc143b41e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:00:16.099 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:16.094+0000 7fbf63be6740 4 client.16182 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-16T12:00:16.099 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:16.094+0000 7fbf63be6740 1 -- 172.21.15.29:0/1351468286 --> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55bc1429e2d0 con 0x55bc1434b920 2024-03-16T12:00:16.099 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:16.094+0000 7fbf5a59c640 1 --2- 172.21.15.29:0/1351468286 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55bc1434b920 0x55bc143b41e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:00:16.099 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:16.095+0000 7fbf5a59c640 1 --2- 172.21.15.29:0/1351468286 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55bc1434b920 0x55bc143b41e0 crc :-1 s=READY pgs=123 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1d52ae05c8f3bf77 server_cookie=a0f94c459a1a4ee6 in_seq=0 out_seq=0 2024-03-16T12:00:16.099 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:16.095+0000 7fbf41ffb640 10 client.16182 ms_handle_connect on v2:172.21.15.152:6838/1548731833 2024-03-16T12:00:16.207 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:16.207+0000 7fbf41ffb640 1 -- 172.21.15.29:0/1351468286 <== mds.0 v2:172.21.15.152:6838/1548731833 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55bc1429e2d0 con 0x55bc1434b920 2024-03-16T12:00:16.207 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:16.207+0000 7fbf41ffb640 10 client.16182 handle_command_reply: tid=0 2024-03-16T12:00:16.207 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:16.207+0000 7fbf63be6740 1 client.16182 shutdown 2024-03-16T12:00:16.207 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:16.207+0000 7fbf63be6740 1 -- 172.21.15.29:0/1351468286 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bc14332a50 msgr2=0x55bc14343c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:00:16.207 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:16.207+0000 7fbf63be6740 1 --2- 172.21.15.29:0/1351468286 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bc14332a50 0x55bc14343c20 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x55bc14331be0 tx=0x7fbf50004730 comp rx=0 tx=0).stop 2024-03-16T12:00:16.207 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:16.207+0000 7fbf63be6740 1 -- 172.21.15.29:0/1351468286 shutdown_connections 2024-03-16T12:00:16.207 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:16.207+0000 7fbf63be6740 1 --2- 172.21.15.29:0/1351468286 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55bc1434b920 0x55bc143b41e0 crc :-1 s=READY pgs=123 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:00:16.207 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:16.207+0000 7fbf63be6740 1 --2- 172.21.15.29:0/1351468286 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55bc1433b820 0x55bc1433ed70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:00:16.207 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:16.207+0000 7fbf41ffb640 0 client.16182 ms_handle_reset on v2:172.21.15.152:6838/1548731833 2024-03-16T12:00:16.208 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:16.207+0000 7fbf63be6740 1 --2- 172.21.15.29:0/1351468286 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bc14332a50 0x55bc14343c20 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:00:16.208 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:16.207+0000 7fbf63be6740 1 --2- 172.21.15.29:0/1351468286 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bc14332130 0x55bc143436e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:00:16.208 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:16.207+0000 7fbf63be6740 1 -- 172.21.15.29:0/1351468286 >> 172.21.15.29:0/1351468286 conn(0x55bc1432ca50 msgr2=0x55bc14338800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:00:16.208 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:16.207+0000 7fbf63be6740 1 -- 172.21.15.29:0/1351468286 shutdown_connections 2024-03-16T12:00:16.208 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:16.207+0000 7fbf63be6740 1 -- 172.21.15.29:0/1351468286 wait complete. 2024-03-16T12:00:16.208 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:16.207+0000 7fbf63be6740 20 client.16182 trim_cache size 0 max 16384 2024-03-16T12:00:16.208 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:{ 2024-03-16T12:00:16.208 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "message": "", 2024-03-16T12:00:16.208 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "return_code": 0 2024-03-16T12:00:16.208 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:} 2024-03-16T12:00:16.209 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:16.208+0000 7fbf60ff4640 1 -- 172.21.15.29:0/3403821178 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7fbf440a2a90 msgr2=0x7fbf440a4f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:00:16.209 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:16.208+0000 7fbf60ff4640 1 --2- 172.21.15.29:0/3403821178 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7fbf440a2a90 0x7fbf440a4f50 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fbf5c077a20 tx=0x7fbf50009b10 comp rx=0 tx=0).stop 2024-03-16T12:00:16.209 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:16.208+0000 7fbf60ff4640 1 -- 172.21.15.29:0/3403821178 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbf5c1078b0 msgr2=0x7fbf5c19c7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:00:16.209 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:16.208+0000 7fbf60ff4640 1 --2- 172.21.15.29:0/3403821178 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbf5c1078b0 0x7fbf5c19c7d0 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7fbf5400ea60 tx=0x7fbf5400ef20 comp rx=0 tx=0).stop 2024-03-16T12:00:16.209 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:16.208+0000 7fbf60ff4640 1 -- 172.21.15.29:0/3403821178 shutdown_connections 2024-03-16T12:00:16.209 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:16.208+0000 7fbf60ff4640 1 --2- 172.21.15.29:0/3403821178 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7fbf440a2a90 0x7fbf440a4f50 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:00:16.209 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:16.208+0000 7fbf60ff4640 1 --2- 172.21.15.29:0/3403821178 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbf5c1078b0 0x7fbf5c19c7d0 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:00:16.210 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:16.208+0000 7fbf60ff4640 1 --2- 172.21.15.29:0/3403821178 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbf5c069800 0x7fbf5c1984a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:00:16.210 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:16.208+0000 7fbf60ff4640 1 --2- 172.21.15.29:0/3403821178 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbf5c068e60 0x7fbf5c197f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:00:16.210 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:16.208+0000 7fbf60ff4640 1 -- 172.21.15.29:0/3403821178 >> 172.21.15.29:0/3403821178 conn(0x7fbf5c077530 msgr2=0x7fbf5c0fe8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:00:16.210 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:16.208+0000 7fbf60ff4640 1 -- 172.21.15.29:0/3403821178 shutdown_connections 2024-03-16T12:00:16.210 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:16.208+0000 7fbf60ff4640 1 -- 172.21.15.29:0/3403821178 wait complete. 2024-03-16T12:00:16.545 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:00:16 smithi154 ceph-mon[36320]: pgmap v293: 97 pgs: 97 active+clean; 69 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 802 KiB/s rd, 2.2 MiB/s wr, 256 op/s 2024-03-16T12:00:16.587 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:00:16 smithi029 ceph-mon[31975]: pgmap v293: 97 pgs: 97 active+clean; 69 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 802 KiB/s rd, 2.2 MiB/s wr, 256 op/s 2024-03-16T12:00:16.627 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:00:16 smithi152 ceph-mon[36291]: pgmap v293: 97 pgs: 97 active+clean; 69 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 802 KiB/s rd, 2.2 MiB/s wr, 256 op/s 2024-03-16T12:00:16.798 INFO:tasks.workunit.client.0.smithi029.stdout: 1 931441 9.95 MB/sec execute 361 sec latency 7755.869 ms 2024-03-16T12:00:17.798 INFO:tasks.workunit.client.0.smithi029.stdout: 1 932865 9.94 MB/sec execute 362 sec latency 59.649 ms 2024-03-16T12:00:18.398 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:00:18 smithi152 ceph-mon[36291]: pgmap v294: 97 pgs: 97 active+clean; 70 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 759 KiB/s rd, 2.9 MiB/s wr, 259 op/s 2024-03-16T12:00:18.398 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:00:18 smithi152 ceph-mon[36291]: osdmap e99: 12 total, 12 up, 12 in 2024-03-16T12:00:18.545 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:00:18 smithi154 ceph-mon[36320]: pgmap v294: 97 pgs: 97 active+clean; 70 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 759 KiB/s rd, 2.9 MiB/s wr, 259 op/s 2024-03-16T12:00:18.545 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:00:18 smithi154 ceph-mon[36320]: osdmap e99: 12 total, 12 up, 12 in 2024-03-16T12:00:18.587 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:00:18 smithi029 ceph-mon[31975]: pgmap v294: 97 pgs: 97 active+clean; 70 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 759 KiB/s rd, 2.9 MiB/s wr, 259 op/s 2024-03-16T12:00:18.588 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:00:18 smithi029 ceph-mon[31975]: osdmap e99: 12 total, 12 up, 12 in 2024-03-16T12:00:18.798 INFO:tasks.workunit.client.0.smithi029.stdout: 1 934830 9.95 MB/sec execute 363 sec latency 18.425 ms 2024-03-16T12:00:19.798 INFO:tasks.workunit.client.0.smithi029.stdout: 1 936654 9.94 MB/sec execute 364 sec latency 16.591 ms 2024-03-16T12:00:20.398 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:00:20 smithi152 ceph-mon[36291]: pgmap v296: 97 pgs: 97 active+clean; 70 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 367 KiB/s rd, 2.1 MiB/s wr, 94 op/s 2024-03-16T12:00:20.545 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:00:20 smithi154 ceph-mon[36320]: pgmap v296: 97 pgs: 97 active+clean; 70 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 367 KiB/s rd, 2.1 MiB/s wr, 94 op/s 2024-03-16T12:00:20.587 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:00:20 smithi029 ceph-mon[31975]: pgmap v296: 97 pgs: 97 active+clean; 70 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 367 KiB/s rd, 2.1 MiB/s wr, 94 op/s 2024-03-16T12:00:20.798 INFO:tasks.workunit.client.0.smithi029.stdout: 1 938471 9.95 MB/sec execute 365 sec latency 16.240 ms 2024-03-16T12:00:21.798 INFO:tasks.workunit.client.0.smithi029.stdout: 1 940415 9.94 MB/sec execute 366 sec latency 22.154 ms 2024-03-16T12:00:22.396 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:00:22 smithi152 ceph-mon[36291]: pgmap v297: 97 pgs: 97 active+clean; 67 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 634 KiB/s rd, 2.3 MiB/s wr, 159 op/s 2024-03-16T12:00:22.545 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:00:22 smithi154 ceph-mon[36320]: pgmap v297: 97 pgs: 97 active+clean; 67 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 634 KiB/s rd, 2.3 MiB/s wr, 159 op/s 2024-03-16T12:00:22.587 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:00:22 smithi029 ceph-mon[31975]: pgmap v297: 97 pgs: 97 active+clean; 67 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 634 KiB/s rd, 2.3 MiB/s wr, 159 op/s 2024-03-16T12:00:22.798 INFO:tasks.workunit.client.0.smithi029.stdout: 1 942002 9.94 MB/sec execute 367 sec latency 99.116 ms 2024-03-16T12:00:23.798 INFO:tasks.workunit.client.0.smithi029.stdout: 1 943389 9.93 MB/sec execute 368 sec latency 32.484 ms 2024-03-16T12:00:24.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:00:24 smithi152 ceph-mon[36291]: pgmap v298: 97 pgs: 97 active+clean; 67 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 634 KiB/s rd, 2.4 MiB/s wr, 162 op/s 2024-03-16T12:00:24.545 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:00:24 smithi154 ceph-mon[36320]: pgmap v298: 97 pgs: 97 active+clean; 67 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 634 KiB/s rd, 2.4 MiB/s wr, 162 op/s 2024-03-16T12:00:24.588 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:00:24 smithi029 ceph-mon[31975]: pgmap v298: 97 pgs: 97 active+clean; 67 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 634 KiB/s rd, 2.4 MiB/s wr, 162 op/s 2024-03-16T12:00:24.799 INFO:tasks.workunit.client.0.smithi029.stdout: 1 945256 9.94 MB/sec execute 369 sec latency 128.203 ms 2024-03-16T12:00:25.799 INFO:tasks.workunit.client.0.smithi029.stdout: 1 946598 9.93 MB/sec execute 370 sec latency 19.359 ms 2024-03-16T12:00:26.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:00:26 smithi152 ceph-mon[36291]: pgmap v299: 97 pgs: 97 active+clean; 67 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 634 KiB/s rd, 2.4 MiB/s wr, 162 op/s 2024-03-16T12:00:26.545 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:00:26 smithi154 ceph-mon[36320]: pgmap v299: 97 pgs: 97 active+clean; 67 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 634 KiB/s rd, 2.4 MiB/s wr, 162 op/s 2024-03-16T12:00:26.587 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:00:26 smithi029 ceph-mon[31975]: pgmap v299: 97 pgs: 97 active+clean; 67 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 634 KiB/s rd, 2.4 MiB/s wr, 162 op/s 2024-03-16T12:00:26.799 INFO:tasks.workunit.client.0.smithi029.stdout: 1 948729 9.94 MB/sec execute 371 sec latency 17.576 ms 2024-03-16T12:00:27.799 INFO:tasks.workunit.client.0.smithi029.stdout: 1 949951 9.92 MB/sec execute 372 sec latency 57.021 ms 2024-03-16T12:00:28.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:00:28 smithi152 ceph-mon[36291]: pgmap v300: 97 pgs: 97 active+clean; 72 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 3.0 MiB/s wr, 216 op/s 2024-03-16T12:00:28.389 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:00:28 smithi029 ceph-mon[31975]: pgmap v300: 97 pgs: 97 active+clean; 72 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 3.0 MiB/s wr, 216 op/s 2024-03-16T12:00:28.545 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:00:28 smithi154 ceph-mon[36320]: pgmap v300: 97 pgs: 97 active+clean; 72 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 3.0 MiB/s wr, 216 op/s 2024-03-16T12:00:28.799 INFO:tasks.workunit.client.0.smithi029.stdout: 1 952154 9.93 MB/sec execute 373 sec latency 48.190 ms 2024-03-16T12:00:29.301 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:29.300+0000 7f32fffc6640 1 -- 172.21.15.29:0/149608885 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f32f8105690 msgr2=0x7f32f8105a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:00:29.301 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:29.300+0000 7f32fffc6640 1 --2- 172.21.15.29:0/149608885 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f32f8105690 0x7f32f8105a70 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f32f4009900 tx=0x7f32f402bc30 comp rx=0 tx=0).stop 2024-03-16T12:00:29.301 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:29.300+0000 7f32fffc6640 1 -- 172.21.15.29:0/149608885 shutdown_connections 2024-03-16T12:00:29.301 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:29.300+0000 7f32fffc6640 1 --2- 172.21.15.29:0/149608885 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f32f8101380 0x7f32f810d520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:00:29.301 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:29.300+0000 7f32fffc6640 1 --2- 172.21.15.29:0/149608885 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f32f81009c0 0x7f32f8100e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:00:29.301 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:29.300+0000 7f32fffc6640 1 --2- 172.21.15.29:0/149608885 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f32f8105690 0x7f32f8105a70 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:00:29.301 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:29.300+0000 7f32fffc6640 1 -- 172.21.15.29:0/149608885 >> 172.21.15.29:0/149608885 conn(0x7f32f80774c0 msgr2=0x7f32f80778d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:00:29.301 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:29.300+0000 7f32fffc6640 1 -- 172.21.15.29:0/149608885 shutdown_connections 2024-03-16T12:00:29.301 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:29.300+0000 7f32fffc6640 1 -- 172.21.15.29:0/149608885 wait complete. 2024-03-16T12:00:29.303 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:29.301+0000 7f32fffc6640 1 Processor -- start 2024-03-16T12:00:29.303 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:29.301+0000 7f32fffc6640 1 -- start start 2024-03-16T12:00:29.304 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:29.301+0000 7f32fffc6640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f32f81009c0 0x7f32f80fe380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:00:29.304 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:29.301+0000 7f32fffc6640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f32f8101380 0x7f32f80fe8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:00:29.304 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:29.301+0000 7f32fffc6640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f32f8105690 0x7f32f80fee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:00:29.304 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:29.301+0000 7f32fffc6640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f32f8110ad0 con 0x7f32f8105690 2024-03-16T12:00:29.304 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:29.301+0000 7f32fffc6640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f32f8110950 con 0x7f32f81009c0 2024-03-16T12:00:29.304 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:29.301+0000 7f32fffc6640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f32f8110c50 con 0x7f32f8101380 2024-03-16T12:00:29.304 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:29.302+0000 7f32fe563640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f32f8105690 0x7f32f80fee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:00:29.304 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:29.302+0000 7f32fe563640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f32f8105690 0x7f32f80fee00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56974/0 (socket says 172.21.15.29:56974) 2024-03-16T12:00:29.304 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:29.302+0000 7f32fe563640 1 -- 172.21.15.29:0/4018871661 learned_addr learned my addr 172.21.15.29:0/4018871661 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T12:00:29.304 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:29.302+0000 7f32fdd62640 1 --2- 172.21.15.29:0/4018871661 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f32f81009c0 0x7f32f80fe380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:00:29.304 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:29.302+0000 7f32fd561640 1 --2- 172.21.15.29:0/4018871661 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f32f8101380 0x7f32f80fe8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:00:29.304 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:29.302+0000 7f32fe563640 1 -- 172.21.15.29:0/4018871661 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f32f8101380 msgr2=0x7f32f80fe8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:00:29.304 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:29.302+0000 7f32fe563640 1 --2- 172.21.15.29:0/4018871661 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f32f8101380 0x7f32f80fe8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:00:29.304 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:29.302+0000 7f32fe563640 1 -- 172.21.15.29:0/4018871661 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f32f81009c0 msgr2=0x7f32f80fe380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:00:29.304 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:29.302+0000 7f32fe563640 1 --2- 172.21.15.29:0/4018871661 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f32f81009c0 0x7f32f80fe380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:00:29.304 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:29.302+0000 7f32fe563640 1 -- 172.21.15.29:0/4018871661 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32f81a64a0 con 0x7f32f8105690 2024-03-16T12:00:29.305 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:29.302+0000 7f32fe563640 1 --2- 172.21.15.29:0/4018871661 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f32f8105690 0x7f32f80fee00 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f32f000ea60 tx=0x7f32f000ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:00:29.305 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:29.302+0000 7f32eeffd640 1 -- 172.21.15.29:0/4018871661 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32f000ce40 con 0x7f32f8105690 2024-03-16T12:00:29.305 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:29.302+0000 7f32eeffd640 1 -- 172.21.15.29:0/4018871661 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f32f0005020 con 0x7f32f8105690 2024-03-16T12:00:29.305 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:29.302+0000 7f32fffc6640 1 -- 172.21.15.29:0/4018871661 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f32f81a6790 con 0x7f32f8105690 2024-03-16T12:00:29.305 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:29.302+0000 7f32eeffd640 1 -- 172.21.15.29:0/4018871661 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32f0010690 con 0x7f32f8105690 2024-03-16T12:00:29.305 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:29.302+0000 7f32fffc6640 1 -- 172.21.15.29:0/4018871661 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f32f81a6c90 con 0x7f32f8105690 2024-03-16T12:00:29.307 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:29.304+0000 7f32eeffd640 1 -- 172.21.15.29:0/4018871661 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7f32f0005520 con 0x7f32f8105690 2024-03-16T12:00:29.308 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:29.305+0000 7f3302bb5740 1 Processor -- start 2024-03-16T12:00:29.308 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:29.305+0000 7f3302bb5740 1 -- start start 2024-03-16T12:00:29.308 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:29.305+0000 7f32eeffd640 1 --2- 172.21.15.29:0/4018871661 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f32e40a2a10 0x7f32e40a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:00:29.308 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:29.305+0000 7f3302bb5740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d3e70ff130 0x55d3e70ff510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:00:29.308 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:29.305+0000 7f3302bb5740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d3e70ffa50 0x55d3e71082e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:00:29.308 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:29.305+0000 7f3302bb5740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d3e7108820 0x55d3e710ac10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:00:29.308 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:29.305+0000 7f3302bb5740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55d3e7063a40 con 0x55d3e70ffa50 2024-03-16T12:00:29.308 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:29.305+0000 7f3302bb5740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x55d3e700c920 con 0x55d3e7108820 2024-03-16T12:00:29.308 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:29.305+0000 7f3302bb5740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55d3e705fae0 con 0x55d3e70ff130 2024-03-16T12:00:29.308 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:29.305+0000 7f32fd561640 1 --2- 172.21.15.29:0/4018871661 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f32e40a2a10 0x7f32e40a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:00:29.308 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:29.305+0000 7f32eeffd640 1 -- 172.21.15.29:0/4018871661 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 8556+0+0 (secure 0 0 0) 0x7f32f00cf2f0 con 0x7f32f8105690 2024-03-16T12:00:29.308 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:29.305+0000 7f32fdd62640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d3e70ff130 0x55d3e70ff510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:00:29.309 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:29.306+0000 7f32fe563640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d3e7108820 0x55d3e710ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:00:29.309 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:29.306+0000 7f32fdd62640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d3e70ff130 0x55d3e70ff510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.29:33050/0 (socket says 172.21.15.29:33050) 2024-03-16T12:00:29.309 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:29.306+0000 7f32fdd62640 1 -- 172.21.15.29:0/1582891333 learned_addr learned my addr 172.21.15.29:0/1582891333 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T12:00:29.309 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:29.306+0000 7f32fd561640 1 --2- 172.21.15.29:0/1582891333 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d3e70ffa50 0x55d3e71082e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:00:29.309 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:29.306+0000 7f32fdd62640 1 -- 172.21.15.29:0/1582891333 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d3e7108820 msgr2=0x55d3e710ac10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:00:29.309 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:29.306+0000 7f32fdd62640 1 --2- 172.21.15.29:0/1582891333 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d3e7108820 0x55d3e710ac10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:00:29.309 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:29.306+0000 7f32fdd62640 1 -- 172.21.15.29:0/1582891333 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d3e70ffa50 msgr2=0x55d3e71082e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:00:29.309 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:29.306+0000 7f32fdd62640 1 --2- 172.21.15.29:0/1582891333 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d3e70ffa50 0x55d3e71082e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:00:29.309 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:29.306+0000 7f32fdd62640 1 -- 172.21.15.29:0/1582891333 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d3e7024460 con 0x55d3e70ff130 2024-03-16T12:00:29.309 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:29.306+0000 7f32fe563640 1 --2- 172.21.15.29:0/1582891333 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d3e7108820 0x55d3e710ac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:00:29.309 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:29.306+0000 7f32fd561640 1 --2- 172.21.15.29:0/4018871661 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f32e40a2a10 0x7f32e40a4ed0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f32e8009fd0 tx=0x7f32e8009b80 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:00:29.309 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:29.306+0000 7f32fdd62640 1 --2- 172.21.15.29:0/1582891333 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d3e70ff130 0x55d3e70ff510 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f32f4009900 tx=0x7f32f4005be0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3640101f76c5828d server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:00:29.309 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:29.306+0000 7f32e3fff640 1 -- 172.21.15.29:0/1582891333 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32f4042070 con 0x55d3e70ff130 2024-03-16T12:00:29.309 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:29.306+0000 7f32e3fff640 1 -- 172.21.15.29:0/1582891333 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f32f402c8b0 con 0x55d3e70ff130 2024-03-16T12:00:29.309 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:29.306+0000 7f32e3fff640 1 -- 172.21.15.29:0/1582891333 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32f4004c90 con 0x55d3e70ff130 2024-03-16T12:00:29.310 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:29.307+0000 7f3302bb5740 1 -- 172.21.15.29:0/1582891333 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d3e70ff130 msgr2=0x55d3e70ff510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:00:29.310 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:29.307+0000 7f3302bb5740 1 --2- 172.21.15.29:0/1582891333 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d3e70ff130 0x55d3e70ff510 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f32f4009900 tx=0x7f32f4005be0 comp rx=0 tx=0).stop 2024-03-16T12:00:29.310 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:29.307+0000 7f3302bb5740 1 -- 172.21.15.29:0/1582891333 shutdown_connections 2024-03-16T12:00:29.310 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:29.307+0000 7f3302bb5740 1 --2- 172.21.15.29:0/1582891333 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d3e7108820 0x55d3e710ac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:00:29.310 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:29.307+0000 7f3302bb5740 1 --2- 172.21.15.29:0/1582891333 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d3e70ffa50 0x55d3e71082e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:00:29.310 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:29.307+0000 7f3302bb5740 1 --2- 172.21.15.29:0/1582891333 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d3e70ff130 0x55d3e70ff510 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:00:29.310 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:29.307+0000 7f3302bb5740 1 -- 172.21.15.29:0/1582891333 >> 172.21.15.29:0/1582891333 conn(0x55d3e70f9a50 msgr2=0x55d3e70f9e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:00:29.310 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:29.307+0000 7f3302bb5740 1 -- 172.21.15.29:0/1582891333 shutdown_connections 2024-03-16T12:00:29.310 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:29.307+0000 7f3302bb5740 1 -- 172.21.15.29:0/1582891333 wait complete. 2024-03-16T12:00:29.310 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:29.307+0000 7f3302bb5740 1 -- start start 2024-03-16T12:00:29.310 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:29.308+0000 7f3302bb5740 1 Processor -- start 2024-03-16T12:00:29.313 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:29.308+0000 7f3302bb5740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d3e70ff130 0x55d3e71057d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:00:29.313 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:29.308+0000 7f3302bb5740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d3e70ffa50 0x55d3e7105d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:00:29.313 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:29.308+0000 7f3302bb5740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d3e7108820 0x55d3e7106250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:00:29.313 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:29.308+0000 7f3302bb5740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55d3e710b3f0 con 0x55d3e70ff130 2024-03-16T12:00:29.313 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:29.308+0000 7f3302bb5740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x55d3e701fd70 con 0x55d3e70ffa50 2024-03-16T12:00:29.313 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:29.308+0000 7f3302bb5740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55d3e710b570 con 0x55d3e7108820 2024-03-16T12:00:29.313 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:29.308+0000 7f3302bb5740 10 client.0 _ll_register_callbacks cb 0x55d3e6d77a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T12:00:29.313 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:29.308+0000 7f32fdd62640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d3e70ff130 0x55d3e71057d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:00:29.313 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:29.308+0000 7f32fdd62640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d3e70ff130 0x55d3e71057d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56990/0 (socket says 172.21.15.29:56990) 2024-03-16T12:00:29.313 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:29.308+0000 7f32fdd62640 1 -- 172.21.15.29:0/820238638 learned_addr learned my addr 172.21.15.29:0/820238638 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T12:00:29.314 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:29.308+0000 7f32fd561640 1 --2- 172.21.15.29:0/820238638 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d3e70ffa50 0x55d3e7105d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:00:29.314 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:29.308+0000 7f32fe563640 1 --2- 172.21.15.29:0/820238638 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d3e7108820 0x55d3e7106250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:00:29.314 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:29.308+0000 7f32fdd62640 1 -- 172.21.15.29:0/820238638 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d3e7108820 msgr2=0x55d3e7106250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:00:29.314 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:29.308+0000 7f32fdd62640 1 --2- 172.21.15.29:0/820238638 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d3e7108820 0x55d3e7106250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:00:29.314 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:29.308+0000 7f32fdd62640 1 -- 172.21.15.29:0/820238638 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d3e70ffa50 msgr2=0x55d3e7105d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:00:29.314 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:29.308+0000 7f32fdd62640 1 --2- 172.21.15.29:0/820238638 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d3e70ffa50 0x55d3e7105d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:00:29.314 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:29.308+0000 7f32fdd62640 1 -- 172.21.15.29:0/820238638 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d3e6d78210 con 0x55d3e70ff130 2024-03-16T12:00:29.314 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:29.309+0000 7f32fdd62640 1 --2- 172.21.15.29:0/820238638 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d3e70ff130 0x55d3e71057d0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f32f402cd10 tx=0x7f32f402cd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea6c319ea0ae35e9 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:00:29.314 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:29.309+0000 7f32e17fa640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-16T12:00:29.314 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:29.309+0000 7f32e17fa640 1 -- 172.21.15.29:0/820238638 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32f404b070 con 0x55d3e70ff130 2024-03-16T12:00:29.314 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:29.309+0000 7f32e17fa640 1 -- 172.21.15.29:0/820238638 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 8556+0+0 (secure 0 0 0) 0x7f32f4005010 con 0x55d3e70ff130 2024-03-16T12:00:29.314 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:29.309+0000 7f3302bb5740 1 -- 172.21.15.29:0/820238638 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f32f402c8b0 con 0x55d3e70ff130 2024-03-16T12:00:29.314 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:29.309+0000 7f32e17fa640 1 -- 172.21.15.29:0/820238638 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f32f4057050 con 0x55d3e70ff130 2024-03-16T12:00:29.314 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:29.309+0000 7f3302bb5740 10 client.16194 fetch_fsmap learned FSMap version 26 2024-03-16T12:00:29.314 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:29.309+0000 7f3302bb5740 1 -- 172.21.15.29:0/820238638 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55d3e6e44620 con 0x55d3e70ff130 2024-03-16T12:00:29.315 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:29.309+0000 7f32e17fa640 1 -- 172.21.15.29:0/820238638 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f32f402c8b0 con 0x55d3e70ff130 2024-03-16T12:00:29.315 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:29.309+0000 7f32e17fa640 1 -- 172.21.15.29:0/820238638 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f32f4047c60 con 0x55d3e70ff130 2024-03-16T12:00:29.315 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:29.310+0000 7f3302bb5740 10 client.16194 fetch_fsmap finished waiting for FSMap version 26 2024-03-16T12:00:29.315 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:29.310+0000 7f3302bb5740 10 client.16194 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T12:00:29.315 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:29.310+0000 7f3302bb5740 20 client.16194 populate_metadata read hostname 'smithi029' 2024-03-16T12:00:29.315 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:29.310+0000 7f3302bb5740 1 --2- 172.21.15.29:0/820238638 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55d3e7118750 0x55d3e711ab40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:00:29.315 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:29.310+0000 7f3302bb5740 4 client.16194 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T12:00:29.315 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:29.310+0000 7f3302bb5740 1 -- 172.21.15.29:0/820238638 --> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d3e7070f80 con 0x55d3e7118750 2024-03-16T12:00:29.315 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:29.310+0000 7f32fd561640 1 --2- 172.21.15.29:0/820238638 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55d3e7118750 0x55d3e711ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:00:29.315 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:29.310+0000 7f32e17fa640 10 client.16194 ms_handle_connect on v2:172.21.15.152:6838/1548731833 2024-03-16T12:00:29.315 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:29.310+0000 7f32fd561640 1 --2- 172.21.15.29:0/820238638 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55d3e7118750 0x55d3e711ab40 crc :-1 s=READY pgs=124 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=54e6c1d49fe5229c server_cookie=18ab4ff7cb7160a5 in_seq=0 out_seq=0 2024-03-16T12:00:29.316 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:29.314+0000 7f32e17fa640 1 -- 172.21.15.29:0/820238638 <== mds.0 v2:172.21.15.152:6838/1548731833 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55d3e7070f80 con 0x55d3e7118750 2024-03-16T12:00:29.316 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:29.314+0000 7f32e17fa640 10 client.16194 handle_command_reply: tid=0 2024-03-16T12:00:29.317 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:29.314+0000 7f3302bb5740 1 client.16194 shutdown 2024-03-16T12:00:29.317 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:29.314+0000 7f3302bb5740 1 -- 172.21.15.29:0/820238638 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d3e70ff130 msgr2=0x55d3e71057d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:00:29.317 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:29.314+0000 7f3302bb5740 1 --2- 172.21.15.29:0/820238638 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d3e70ff130 0x55d3e71057d0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f32f402cd10 tx=0x7f32f402cd40 comp rx=0 tx=0).stop 2024-03-16T12:00:29.317 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:29.314+0000 7f3302bb5740 1 -- 172.21.15.29:0/820238638 shutdown_connections 2024-03-16T12:00:29.317 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:29.314+0000 7f3302bb5740 1 --2- 172.21.15.29:0/820238638 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d3e7108820 0x55d3e7106250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:00:29.317 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:29.314+0000 7f3302bb5740 1 --2- 172.21.15.29:0/820238638 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55d3e7118750 0x55d3e711ab40 crc :-1 s=READY pgs=124 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:00:29.317 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:29.314+0000 7f3302bb5740 1 --2- 172.21.15.29:0/820238638 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d3e70ffa50 0x55d3e7105d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:00:29.317 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:29.314+0000 7f3302bb5740 1 --2- 172.21.15.29:0/820238638 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d3e70ff130 0x55d3e71057d0 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:00:29.317 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:29.314+0000 7f32e17fa640 0 client.16194 ms_handle_reset on v2:172.21.15.152:6838/1548731833 2024-03-16T12:00:29.317 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:29.314+0000 7f3302bb5740 1 -- 172.21.15.29:0/820238638 >> 172.21.15.29:0/820238638 conn(0x55d3e70f9a50 msgr2=0x55d3e70fb7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:00:29.317 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:29.314+0000 7f3302bb5740 1 -- 172.21.15.29:0/820238638 shutdown_connections 2024-03-16T12:00:29.317 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:29.314+0000 7f3302bb5740 1 -- 172.21.15.29:0/820238638 wait complete. 2024-03-16T12:00:29.317 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:29.314+0000 7f3302bb5740 20 client.16194 trim_cache size 0 max 16384 2024-03-16T12:00:29.332 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:29.330+0000 7f3302bb5740 1 Processor -- start 2024-03-16T12:00:29.332 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:29.331+0000 7f3302bb5740 1 -- start start 2024-03-16T12:00:29.332 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:29.331+0000 7f3302bb5740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d3e70ff130 0x55d3e7107f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:00:29.332 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:29.331+0000 7f3302bb5740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d3e70ffa50 0x55d3e7114190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:00:29.332 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:29.331+0000 7f3302bb5740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d3e7108820 0x55d3e70f4a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:00:29.332 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:29.331+0000 7f3302bb5740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f32f81b1ec0 con 0x55d3e7108820 2024-03-16T12:00:29.332 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:29.331+0000 7f3302bb5740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f32f81af790 con 0x55d3e70ff130 2024-03-16T12:00:29.332 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:29.331+0000 7f3302bb5740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f32f81ad600 con 0x55d3e70ffa50 2024-03-16T12:00:29.332 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:29.331+0000 7f32fe563640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d3e7108820 0x55d3e70f4a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:00:29.333 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:29.331+0000 7f32fe563640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d3e7108820 0x55d3e70f4a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56998/0 (socket says 172.21.15.29:56998) 2024-03-16T12:00:29.333 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:29.331+0000 7f32fe563640 1 -- 172.21.15.29:0/3837020714 learned_addr learned my addr 172.21.15.29:0/3837020714 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T12:00:29.333 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:29.331+0000 7f32fdd62640 1 --2- 172.21.15.29:0/3837020714 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d3e70ff130 0x55d3e7107f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:00:29.333 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:29.331+0000 7f32fd561640 1 --2- 172.21.15.29:0/3837020714 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d3e70ffa50 0x55d3e7114190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:00:29.333 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:29.331+0000 7f32fe563640 1 -- 172.21.15.29:0/3837020714 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d3e70ffa50 msgr2=0x55d3e7114190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:00:29.333 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:29.331+0000 7f32fe563640 1 --2- 172.21.15.29:0/3837020714 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d3e70ffa50 0x55d3e7114190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:00:29.333 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:29.331+0000 7f32fe563640 1 -- 172.21.15.29:0/3837020714 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d3e70ff130 msgr2=0x55d3e7107f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:00:29.333 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:29.331+0000 7f32fe563640 1 --2- 172.21.15.29:0/3837020714 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d3e70ff130 0x55d3e7107f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:00:29.333 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:29.331+0000 7f32fe563640 1 -- 172.21.15.29:0/3837020714 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d3e7104780 con 0x55d3e7108820 2024-03-16T12:00:29.333 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:29.331+0000 7f32fe563640 1 --2- 172.21.15.29:0/3837020714 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d3e7108820 0x55d3e70f4a40 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f32f00ccad0 tx=0x7f32f00cf280 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87803026c3b3f529 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:00:29.333 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:29.331+0000 7f32e37fe640 1 -- 172.21.15.29:0/3837020714 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32f00b4070 con 0x55d3e7108820 2024-03-16T12:00:29.333 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:29.332+0000 7f32e37fe640 1 -- 172.21.15.29:0/3837020714 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f32f00cf040 con 0x55d3e7108820 2024-03-16T12:00:29.333 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:29.332+0000 7f32e37fe640 1 -- 172.21.15.29:0/3837020714 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32f00ce030 con 0x55d3e7108820 2024-03-16T12:00:29.337 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:29.332+0000 7f3302bb5740 1 -- 172.21.15.29:0/3837020714 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d3e7108820 msgr2=0x55d3e70f4a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:00:29.337 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:29.332+0000 7f3302bb5740 1 --2- 172.21.15.29:0/3837020714 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d3e7108820 0x55d3e70f4a40 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f32f00ccad0 tx=0x7f32f00cf280 comp rx=0 tx=0).stop 2024-03-16T12:00:29.337 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:29.332+0000 7f3302bb5740 1 -- 172.21.15.29:0/3837020714 shutdown_connections 2024-03-16T12:00:29.337 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:29.332+0000 7f3302bb5740 1 --2- 172.21.15.29:0/3837020714 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d3e7108820 0x55d3e70f4a40 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:00:29.337 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:29.332+0000 7f3302bb5740 1 --2- 172.21.15.29:0/3837020714 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d3e70ffa50 0x55d3e7114190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:00:29.337 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:29.332+0000 7f3302bb5740 1 --2- 172.21.15.29:0/3837020714 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d3e70ff130 0x55d3e7107f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:00:29.337 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:29.332+0000 7f3302bb5740 1 -- 172.21.15.29:0/3837020714 >> 172.21.15.29:0/3837020714 conn(0x55d3e70f9a50 msgr2=0x55d3e70f8bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:00:29.337 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:29.332+0000 7f3302bb5740 1 -- 172.21.15.29:0/3837020714 shutdown_connections 2024-03-16T12:00:29.338 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:29.332+0000 7f3302bb5740 1 -- 172.21.15.29:0/3837020714 wait complete. 2024-03-16T12:00:29.338 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:29.333+0000 7f3302bb5740 1 -- start start 2024-03-16T12:00:29.338 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:29.333+0000 7f3302bb5740 1 Processor -- start 2024-03-16T12:00:29.338 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:29.333+0000 7f3302bb5740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d3e70ff130 0x55d3e71109b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:00:29.338 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:29.333+0000 7f3302bb5740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d3e70ffa50 0x55d3e710d790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:00:29.338 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:29.333+0000 7f3302bb5740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d3e7108820 0x55d3e710dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:00:29.338 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:29.333+0000 7f3302bb5740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f32f81adc50 con 0x55d3e70ffa50 2024-03-16T12:00:29.338 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:29.333+0000 7f3302bb5740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f32f81af0c0 con 0x55d3e70ff130 2024-03-16T12:00:29.338 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:29.333+0000 7f3302bb5740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f32f81ae320 con 0x55d3e7108820 2024-03-16T12:00:29.338 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:29.333+0000 7f3302bb5740 10 client.0 _ll_register_callbacks cb 0x7f32e80114c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T12:00:29.338 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:29.333+0000 7f32fd561640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d3e70ffa50 0x55d3e710d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:00:29.338 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:29.333+0000 7f32fd561640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d3e70ffa50 0x55d3e710d790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57010/0 (socket says 172.21.15.29:57010) 2024-03-16T12:00:29.338 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:29.333+0000 7f32fd561640 1 -- 172.21.15.29:0/3396780524 learned_addr learned my addr 172.21.15.29:0/3396780524 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T12:00:29.338 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:29.333+0000 7f32fdd62640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d3e70ff130 0x55d3e71109b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:00:29.339 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:29.333+0000 7f32fe563640 1 --2- 172.21.15.29:0/3396780524 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d3e7108820 0x55d3e710dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:00:29.339 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:29.333+0000 7f32fd561640 1 -- 172.21.15.29:0/3396780524 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d3e7108820 msgr2=0x55d3e710dcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:00:29.339 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:29.333+0000 7f32fd561640 1 --2- 172.21.15.29:0/3396780524 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d3e7108820 0x55d3e710dcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:00:29.339 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:29.333+0000 7f32fd561640 1 -- 172.21.15.29:0/3396780524 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d3e70ff130 msgr2=0x55d3e71109b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:00:29.339 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:29.333+0000 7f32fd561640 1 --2- 172.21.15.29:0/3396780524 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d3e70ff130 0x55d3e71109b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:00:29.339 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:29.333+0000 7f32fd561640 1 -- 172.21.15.29:0/3396780524 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d3e710e4d0 con 0x55d3e70ffa50 2024-03-16T12:00:29.339 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:29.333+0000 7f32fdd62640 1 --2- 172.21.15.29:0/3396780524 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d3e70ff130 0x55d3e71109b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:00:29.339 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:29.334+0000 7f32fd561640 1 --2- 172.21.15.29:0/3396780524 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d3e70ffa50 0x55d3e710d790 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f32e8009980 tx=0x7f32e801d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d42f44c385e2348 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:00:29.339 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:29.334+0000 7f32e17fa640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-16T12:00:29.339 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:29.334+0000 7f32e17fa640 1 -- 172.21.15.29:0/3396780524 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32e803a060 con 0x55d3e70ffa50 2024-03-16T12:00:29.339 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:29.334+0000 7f32e17fa640 1 -- 172.21.15.29:0/3396780524 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 8556+0+0 (secure 0 0 0) 0x7f32e801a070 con 0x55d3e70ffa50 2024-03-16T12:00:29.339 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:29.334+0000 7f3302bb5740 1 -- 172.21.15.29:0/3396780524 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f32f00cf040 con 0x55d3e70ffa50 2024-03-16T12:00:29.339 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:29.334+0000 7f32e17fa640 1 -- 172.21.15.29:0/3396780524 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f32e8011190 con 0x55d3e70ffa50 2024-03-16T12:00:29.339 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:29.334+0000 7f3302bb5740 10 client.16200 fetch_fsmap learned FSMap version 26 2024-03-16T12:00:29.339 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:29.334+0000 7f3302bb5740 1 -- 172.21.15.29:0/3396780524 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55d3e717f900 con 0x55d3e70ffa50 2024-03-16T12:00:29.339 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:29.334+0000 7f32e17fa640 1 -- 172.21.15.29:0/3396780524 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f32f00cf040 con 0x55d3e70ffa50 2024-03-16T12:00:29.339 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:29.334+0000 7f32e17fa640 1 -- 172.21.15.29:0/3396780524 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f32e80035f0 con 0x55d3e70ffa50 2024-03-16T12:00:29.340 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:29.334+0000 7f3302bb5740 10 client.16200 fetch_fsmap finished waiting for FSMap version 26 2024-03-16T12:00:29.340 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:29.334+0000 7f3302bb5740 10 client.16200 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T12:00:29.340 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:29.334+0000 7f3302bb5740 20 client.16200 populate_metadata read hostname 'smithi029' 2024-03-16T12:00:29.340 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:29.334+0000 7f3302bb5740 1 --2- 172.21.15.29:0/3396780524 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55d3e7180060 0x55d3e7182480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:00:29.340 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:29.334+0000 7f3302bb5740 4 client.16200 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-16T12:00:29.340 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:29.334+0000 7f3302bb5740 1 -- 172.21.15.29:0/3396780524 --> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55d3e706b2d0 con 0x55d3e7180060 2024-03-16T12:00:29.340 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:29.335+0000 7f32fdd62640 1 --2- 172.21.15.29:0/3396780524 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55d3e7180060 0x55d3e7182480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:00:29.340 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:29.335+0000 7f32e17fa640 10 client.16200 ms_handle_connect on v2:172.21.15.152:6838/1548731833 2024-03-16T12:00:29.340 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:29.335+0000 7f32fdd62640 1 --2- 172.21.15.29:0/3396780524 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55d3e7180060 0x55d3e7182480 crc :-1 s=READY pgs=125 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d582b3b799c410a server_cookie=f03921d42cd70bd3 in_seq=0 out_seq=0 2024-03-16T12:00:29.526 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:29.525+0000 7f32e17fa640 1 -- 172.21.15.29:0/3396780524 <== mds.0 v2:172.21.15.152:6838/1548731833 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55d3e706b2d0 con 0x55d3e7180060 2024-03-16T12:00:29.526 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:29.525+0000 7f32e17fa640 10 client.16200 handle_command_reply: tid=0 2024-03-16T12:00:29.526 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:29.525+0000 7f3302bb5740 1 client.16200 shutdown 2024-03-16T12:00:29.526 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:29.525+0000 7f3302bb5740 1 -- 172.21.15.29:0/3396780524 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d3e70ffa50 msgr2=0x55d3e710d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:00:29.526 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:29.525+0000 7f3302bb5740 1 --2- 172.21.15.29:0/3396780524 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d3e70ffa50 0x55d3e710d790 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f32e8009980 tx=0x7f32e801d000 comp rx=0 tx=0).stop 2024-03-16T12:00:29.526 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:29.526+0000 7f3302bb5740 1 -- 172.21.15.29:0/3396780524 shutdown_connections 2024-03-16T12:00:29.527 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:29.526+0000 7f3302bb5740 1 --2- 172.21.15.29:0/3396780524 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d3e7108820 0x55d3e710dcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:00:29.527 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:29.526+0000 7f3302bb5740 1 --2- 172.21.15.29:0/3396780524 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d3e70ffa50 0x55d3e710d790 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:00:29.527 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:29.526+0000 7f3302bb5740 1 --2- 172.21.15.29:0/3396780524 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55d3e7180060 0x55d3e7182480 crc :-1 s=READY pgs=125 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:00:29.527 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:29.526+0000 7f3302bb5740 1 --2- 172.21.15.29:0/3396780524 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d3e70ff130 0x55d3e71109b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:00:29.527 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:29.526+0000 7f3302bb5740 1 -- 172.21.15.29:0/3396780524 >> 172.21.15.29:0/3396780524 conn(0x55d3e70f9a50 msgr2=0x55d3e7105770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:00:29.527 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:29.526+0000 7f32e17fa640 0 client.16200 ms_handle_reset on v2:172.21.15.152:6838/1548731833 2024-03-16T12:00:29.527 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:29.526+0000 7f3302bb5740 1 -- 172.21.15.29:0/3396780524 shutdown_connections 2024-03-16T12:00:29.527 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:29.526+0000 7f3302bb5740 1 -- 172.21.15.29:0/3396780524 wait complete. 2024-03-16T12:00:29.527 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:29.526+0000 7f3302bb5740 20 client.16200 trim_cache size 0 max 16384 2024-03-16T12:00:29.527 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:{ 2024-03-16T12:00:29.527 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "message": "", 2024-03-16T12:00:29.528 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "return_code": 0 2024-03-16T12:00:29.528 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:} 2024-03-16T12:00:29.531 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:29.527+0000 7f32fffc6640 1 -- 172.21.15.29:0/4018871661 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f32e40a2a10 msgr2=0x7f32e40a4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:00:29.531 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:29.527+0000 7f32fffc6640 1 --2- 172.21.15.29:0/4018871661 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f32e40a2a10 0x7f32e40a4ed0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f32e8009fd0 tx=0x7f32e8009b80 comp rx=0 tx=0).stop 2024-03-16T12:00:29.531 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:29.527+0000 7f32fffc6640 1 -- 172.21.15.29:0/4018871661 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f32f8105690 msgr2=0x7f32f80fee00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:00:29.531 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:29.527+0000 7f32fffc6640 1 --2- 172.21.15.29:0/4018871661 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f32f8105690 0x7f32f80fee00 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f32f000ea60 tx=0x7f32f000ef20 comp rx=0 tx=0).stop 2024-03-16T12:00:29.531 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:29.527+0000 7f32fffc6640 1 -- 172.21.15.29:0/4018871661 shutdown_connections 2024-03-16T12:00:29.531 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:29.527+0000 7f32fffc6640 1 --2- 172.21.15.29:0/4018871661 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f32e40a2a10 0x7f32e40a4ed0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:00:29.531 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:29.527+0000 7f32fffc6640 1 --2- 172.21.15.29:0/4018871661 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f32f8105690 0x7f32f80fee00 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:00:29.531 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:29.527+0000 7f32fffc6640 1 --2- 172.21.15.29:0/4018871661 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f32f8101380 0x7f32f80fe8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:00:29.531 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:29.527+0000 7f32fffc6640 1 --2- 172.21.15.29:0/4018871661 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f32f81009c0 0x7f32f80fe380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:00:29.531 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:29.527+0000 7f32fffc6640 1 -- 172.21.15.29:0/4018871661 >> 172.21.15.29:0/4018871661 conn(0x7f32f80774c0 msgr2=0x7f32f806a220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:00:29.531 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:29.527+0000 7f32fffc6640 1 -- 172.21.15.29:0/4018871661 shutdown_connections 2024-03-16T12:00:29.531 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:29.527+0000 7f32fffc6640 1 -- 172.21.15.29:0/4018871661 wait complete. 2024-03-16T12:00:29.799 INFO:tasks.workunit.client.0.smithi029.stdout: 1 953175 9.91 MB/sec execute 374 sec latency 25.162 ms 2024-03-16T12:00:30.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:00:30 smithi152 ceph-mon[36291]: pgmap v301: 97 pgs: 97 active+clean; 72 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 879 KiB/s rd, 2.5 MiB/s wr, 182 op/s 2024-03-16T12:00:30.545 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:00:30 smithi154 ceph-mon[36320]: pgmap v301: 97 pgs: 97 active+clean; 72 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 879 KiB/s rd, 2.5 MiB/s wr, 182 op/s 2024-03-16T12:00:30.587 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:00:30 smithi029 ceph-mon[31975]: pgmap v301: 97 pgs: 97 active+clean; 72 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 879 KiB/s rd, 2.5 MiB/s wr, 182 op/s 2024-03-16T12:00:30.799 INFO:tasks.workunit.client.0.smithi029.stdout: 1 955489 9.92 MB/sec execute 375 sec latency 22.601 ms 2024-03-16T12:00:31.799 INFO:tasks.workunit.client.0.smithi029.stdout: 1 956898 9.91 MB/sec execute 376 sec latency 22.653 ms 2024-03-16T12:00:32.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:00:32 smithi152 ceph-mon[36291]: pgmap v302: 97 pgs: 97 active+clean; 65 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.7 MiB/s wr, 327 op/s 2024-03-16T12:00:32.545 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:00:32 smithi154 ceph-mon[36320]: pgmap v302: 97 pgs: 97 active+clean; 65 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.7 MiB/s wr, 327 op/s 2024-03-16T12:00:32.590 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:00:32 smithi029 ceph-mon[31975]: pgmap v302: 97 pgs: 97 active+clean; 65 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.7 MiB/s wr, 327 op/s 2024-03-16T12:00:32.799 INFO:tasks.workunit.client.0.smithi029.stdout: 1 958676 9.92 MB/sec execute 377 sec latency 96.884 ms 2024-03-16T12:00:33.799 INFO:tasks.workunit.client.0.smithi029.stdout: 1 959706 9.91 MB/sec execute 378 sec latency 44.413 ms 2024-03-16T12:00:34.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:00:34 smithi152 ceph-mon[36291]: pgmap v303: 97 pgs: 97 active+clean; 65 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 868 KiB/s rd, 2.7 MiB/s wr, 234 op/s 2024-03-16T12:00:34.545 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:00:34 smithi154 ceph-mon[36320]: pgmap v303: 97 pgs: 97 active+clean; 65 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 868 KiB/s rd, 2.7 MiB/s wr, 234 op/s 2024-03-16T12:00:34.588 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:00:34 smithi029 ceph-mon[31975]: pgmap v303: 97 pgs: 97 active+clean; 65 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 868 KiB/s rd, 2.7 MiB/s wr, 234 op/s 2024-03-16T12:00:34.799 INFO:tasks.workunit.client.0.smithi029.stdout: 1 961267 9.90 MB/sec execute 379 sec latency 20.192 ms 2024-03-16T12:00:35.799 INFO:tasks.workunit.client.0.smithi029.stdout: 1 962992 9.90 MB/sec execute 380 sec latency 28.543 ms 2024-03-16T12:00:36.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:00:36 smithi152 ceph-mon[36291]: pgmap v304: 97 pgs: 97 active+clean; 65 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 868 KiB/s rd, 2.6 MiB/s wr, 229 op/s 2024-03-16T12:00:36.545 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:00:36 smithi154 ceph-mon[36320]: pgmap v304: 97 pgs: 97 active+clean; 65 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 868 KiB/s rd, 2.6 MiB/s wr, 229 op/s 2024-03-16T12:00:36.588 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:00:36 smithi029 ceph-mon[31975]: pgmap v304: 97 pgs: 97 active+clean; 65 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 868 KiB/s rd, 2.6 MiB/s wr, 229 op/s 2024-03-16T12:00:36.799 INFO:tasks.workunit.client.0.smithi029.stdout: 1 964358 9.89 MB/sec execute 381 sec latency 20.600 ms 2024-03-16T12:00:37.799 INFO:tasks.workunit.client.0.smithi029.stdout: 1 966352 9.90 MB/sec execute 382 sec latency 120.866 ms 2024-03-16T12:00:38.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:00:38 smithi152 ceph-mon[36291]: pgmap v305: 97 pgs: 97 active+clean; 90 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 5.1 MiB/s wr, 303 op/s 2024-03-16T12:00:38.545 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:00:38 smithi154 ceph-mon[36320]: pgmap v305: 97 pgs: 97 active+clean; 90 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 5.1 MiB/s wr, 303 op/s 2024-03-16T12:00:38.587 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:00:38 smithi029 ceph-mon[31975]: pgmap v305: 97 pgs: 97 active+clean; 90 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 5.1 MiB/s wr, 303 op/s 2024-03-16T12:00:38.799 INFO:tasks.workunit.client.0.smithi029.stdout: 1 966966 9.88 MB/sec execute 383 sec latency 35.083 ms 2024-03-16T12:00:39.799 INFO:tasks.workunit.client.0.smithi029.stdout: 1 968763 9.87 MB/sec execute 384 sec latency 25.478 ms 2024-03-16T12:00:40.379 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:00:40 smithi152 ceph-mon[36291]: pgmap v306: 97 pgs: 97 active+clean; 90 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.7 MiB/s wr, 230 op/s 2024-03-16T12:00:40.545 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:00:40 smithi154 ceph-mon[36320]: pgmap v306: 97 pgs: 97 active+clean; 90 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.7 MiB/s wr, 230 op/s 2024-03-16T12:00:40.589 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:00:40 smithi029 ceph-mon[31975]: pgmap v306: 97 pgs: 97 active+clean; 90 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.7 MiB/s wr, 230 op/s 2024-03-16T12:00:40.799 INFO:tasks.workunit.client.0.smithi029.stdout: 1 970154 9.88 MB/sec execute 385 sec latency 21.436 ms 2024-03-16T12:00:41.799 INFO:tasks.workunit.client.0.smithi029.stdout: 1 971583 9.86 MB/sec execute 386 sec latency 19.113 ms 2024-03-16T12:00:42.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:00:42 smithi152 ceph-mon[36291]: pgmap v307: 97 pgs: 97 active+clean; 95 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 4.7 MiB/s wr, 297 op/s 2024-03-16T12:00:42.545 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:00:42 smithi154 ceph-mon[36320]: pgmap v307: 97 pgs: 97 active+clean; 95 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 4.7 MiB/s wr, 297 op/s 2024-03-16T12:00:42.587 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:00:42 smithi029 ceph-mon[31975]: pgmap v307: 97 pgs: 97 active+clean; 95 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 4.7 MiB/s wr, 297 op/s 2024-03-16T12:00:42.619 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:42.618+0000 7f88852ff640 1 -- 172.21.15.29:0/2660369686 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8880100930 msgr2=0x7f8880100d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:00:42.619 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:42.618+0000 7f88852ff640 1 --2- 172.21.15.29:0/2660369686 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8880100930 0x7f8880100d90 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f8868009900 tx=0x7f886802bc30 comp rx=0 tx=0).stop 2024-03-16T12:00:42.619 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:42.618+0000 7f88852ff640 1 -- 172.21.15.29:0/2660369686 shutdown_connections 2024-03-16T12:00:42.620 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:42.618+0000 7f88852ff640 1 --2- 172.21.15.29:0/2660369686 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f88801012d0 0x7f888010d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:00:42.620 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:42.618+0000 7f88852ff640 1 --2- 172.21.15.29:0/2660369686 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8880100930 0x7f8880100d90 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:00:42.620 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:42.618+0000 7f88852ff640 1 --2- 172.21.15.29:0/2660369686 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8880105660 0x7f8880105a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:00:42.620 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:42.618+0000 7f88852ff640 1 -- 172.21.15.29:0/2660369686 >> 172.21.15.29:0/2660369686 conn(0x7f88800774b0 msgr2=0x7f88800778c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:00:42.620 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:42.618+0000 7f88852ff640 1 -- 172.21.15.29:0/2660369686 shutdown_connections 2024-03-16T12:00:42.620 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:42.618+0000 7f88852ff640 1 -- 172.21.15.29:0/2660369686 wait complete. 2024-03-16T12:00:42.620 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:42.618+0000 7f88852ff640 1 Processor -- start 2024-03-16T12:00:42.620 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:42.618+0000 7f88852ff640 1 -- start start 2024-03-16T12:00:42.620 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:42.618+0000 7f88852ff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8880100930 0x7f88801980b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:00:42.620 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:42.618+0000 7f88852ff640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f88801012d0 0x7f88801985f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:00:42.620 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:42.618+0000 7f88852ff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8880105660 0x7f888019c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:00:42.620 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:42.619+0000 7f88852ff640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f8880111c90 con 0x7f8880100930 2024-03-16T12:00:42.620 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:42.619+0000 7f88852ff640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f8880111b10 con 0x7f88801012d0 2024-03-16T12:00:42.620 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:42.619+0000 7f88852ff640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f8880111e10 con 0x7f8880105660 2024-03-16T12:00:42.621 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:42.619+0000 7f887effd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8880100930 0x7f88801980b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:00:42.621 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:42.619+0000 7f887effd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8880100930 0x7f88801980b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33386/0 (socket says 172.21.15.29:33386) 2024-03-16T12:00:42.621 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:42.619+0000 7f887effd640 1 -- 172.21.15.29:0/3657351055 learned_addr learned my addr 172.21.15.29:0/3657351055 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T12:00:42.621 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:42.619+0000 7f88767fc640 1 --2- 172.21.15.29:0/3657351055 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f88801012d0 0x7f88801985f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:00:42.621 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:42.619+0000 7f887f7fe640 1 --2- 172.21.15.29:0/3657351055 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8880105660 0x7f888019c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:00:42.621 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:42.619+0000 7f887effd640 1 -- 172.21.15.29:0/3657351055 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8880105660 msgr2=0x7f888019c920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:00:42.621 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:42.619+0000 7f887effd640 1 --2- 172.21.15.29:0/3657351055 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8880105660 0x7f888019c920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:00:42.621 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:42.619+0000 7f887effd640 1 -- 172.21.15.29:0/3657351055 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f88801012d0 msgr2=0x7f88801985f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:00:42.621 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:42.619+0000 7f887effd640 1 --2- 172.21.15.29:0/3657351055 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f88801012d0 0x7f88801985f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:00:42.621 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:42.619+0000 7f887effd640 1 -- 172.21.15.29:0/3657351055 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f888019d050 con 0x7f8880100930 2024-03-16T12:00:42.621 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:42.619+0000 7f887effd640 1 --2- 172.21.15.29:0/3657351055 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8880100930 0x7f88801980b0 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f887000e9f0 tx=0x7f887000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:00:42.621 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:42.619+0000 7f887cff9640 1 -- 172.21.15.29:0/3657351055 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f887000cd80 con 0x7f8880100930 2024-03-16T12:00:42.621 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:42.619+0000 7f88852ff640 1 -- 172.21.15.29:0/3657351055 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f888019d340 con 0x7f8880100930 2024-03-16T12:00:42.621 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:42.619+0000 7f887cff9640 1 -- 172.21.15.29:0/3657351055 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8870010910 con 0x7f8880100930 2024-03-16T12:00:42.621 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:42.619+0000 7f887cff9640 1 -- 172.21.15.29:0/3657351055 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88700188f0 con 0x7f8880100930 2024-03-16T12:00:42.621 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:42.619+0000 7f88852ff640 1 -- 172.21.15.29:0/3657351055 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f88801a4bd0 con 0x7f8880100930 2024-03-16T12:00:42.631 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:42.621+0000 7f887cff9640 1 -- 172.21.15.29:0/3657351055 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7f8870010ac0 con 0x7f8880100930 2024-03-16T12:00:42.631 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:42.622+0000 7f8887fc7740 1 Processor -- start 2024-03-16T12:00:42.631 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:42.622+0000 7f8887fc7740 1 -- start start 2024-03-16T12:00:42.631 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:42.622+0000 7f887cff9640 1 --2- 172.21.15.29:0/3657351055 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f886c0a2a10 0x7f886c0a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:00:42.631 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:42.622+0000 7f8887fc7740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a94cd14130 0x55a94cd14510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:00:42.631 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:42.622+0000 7f8887fc7740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a94cd14a50 0x55a94cd1d2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:00:42.631 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:42.622+0000 7f8887fc7740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a94cd1d820 0x55a94cd1fc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:00:42.631 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:42.622+0000 7f8887fc7740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55a94cc78a40 con 0x55a94cd14a50 2024-03-16T12:00:42.631 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:42.622+0000 7f8887fc7740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x55a94cc21920 con 0x55a94cd14130 2024-03-16T12:00:42.631 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:42.622+0000 7f8887fc7740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55a94cc74ae0 con 0x55a94cd1d820 2024-03-16T12:00:42.631 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:42.622+0000 7f887f7fe640 1 --2- 172.21.15.29:0/3657351055 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f886c0a2a10 0x7f886c0a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:00:42.632 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:42.622+0000 7f887cff9640 1 -- 172.21.15.29:0/3657351055 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 8556+0+0 (secure 0 0 0) 0x7f88700cf0f0 con 0x7f8880100930 2024-03-16T12:00:42.632 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:42.622+0000 7f887effd640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a94cd14130 0x55a94cd14510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:00:42.632 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:42.622+0000 7f88767fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a94cd14a50 0x55a94cd1d2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:00:42.632 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:42.622+0000 7f88767fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a94cd14a50 0x55a94cd1d2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33402/0 (socket says 172.21.15.29:33402) 2024-03-16T12:00:42.632 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:42.622+0000 7f88767fc640 1 -- 172.21.15.29:0/3690192019 learned_addr learned my addr 172.21.15.29:0/3690192019 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T12:00:42.632 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:42.622+0000 7f887f7fe640 1 --2- 172.21.15.29:0/3690192019 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a94cd1d820 0x55a94cd1fc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:00:42.632 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:42.623+0000 7f887effd640 1 -- 172.21.15.29:0/3690192019 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a94cd1d820 msgr2=0x55a94cd1fc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:00:42.632 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:42.623+0000 7f887effd640 1 --2- 172.21.15.29:0/3690192019 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a94cd1d820 0x55a94cd1fc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:00:42.632 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:42.623+0000 7f887effd640 1 -- 172.21.15.29:0/3690192019 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a94cd14a50 msgr2=0x55a94cd1d2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:00:42.632 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:42.623+0000 7f887effd640 1 --2- 172.21.15.29:0/3690192019 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a94cd14a50 0x55a94cd1d2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:00:42.632 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:42.623+0000 7f887effd640 1 -- 172.21.15.29:0/3690192019 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a94cc39460 con 0x55a94cd14130 2024-03-16T12:00:42.632 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:42.623+0000 7f887f7fe640 1 --2- 172.21.15.29:0/3657351055 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f886c0a2a10 0x7f886c0a4ed0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x55a94cd0fe70 tx=0x7f8878009bf0 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:00:42.632 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:42.623+0000 7f887effd640 1 --2- 172.21.15.29:0/3690192019 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a94cd14130 0x55a94cd14510 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f88700cc8c0 tx=0x7f887009b4d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=af4f160025232a4 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:00:42.632 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:42.623+0000 7f88757fa640 1 -- 172.21.15.29:0/3690192019 <== mon.2 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f887009b880 con 0x55a94cd14130 2024-03-16T12:00:42.632 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:42.623+0000 7f88757fa640 1 -- 172.21.15.29:0/3690192019 <== mon.2 v2:172.21.15.152:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f887009f920 con 0x55a94cd14130 2024-03-16T12:00:42.632 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:42.623+0000 7f88757fa640 1 -- 172.21.15.29:0/3690192019 <== mon.2 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88700860e0 con 0x55a94cd14130 2024-03-16T12:00:42.633 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:42.624+0000 7f8887fc7740 1 -- 172.21.15.29:0/3690192019 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a94cd14130 msgr2=0x55a94cd14510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:00:42.633 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:42.624+0000 7f8887fc7740 1 --2- 172.21.15.29:0/3690192019 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a94cd14130 0x55a94cd14510 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f88700cc8c0 tx=0x7f887009b4d0 comp rx=0 tx=0).stop 2024-03-16T12:00:42.633 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:42.624+0000 7f8887fc7740 1 -- 172.21.15.29:0/3690192019 shutdown_connections 2024-03-16T12:00:42.633 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:42.624+0000 7f8887fc7740 1 --2- 172.21.15.29:0/3690192019 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a94cd1d820 0x55a94cd1fc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:00:42.633 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:42.624+0000 7f8887fc7740 1 --2- 172.21.15.29:0/3690192019 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a94cd14a50 0x55a94cd1d2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:00:42.633 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:42.624+0000 7f8887fc7740 1 --2- 172.21.15.29:0/3690192019 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a94cd14130 0x55a94cd14510 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:00:42.633 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:42.624+0000 7f8887fc7740 1 -- 172.21.15.29:0/3690192019 >> 172.21.15.29:0/3690192019 conn(0x55a94cd0ea50 msgr2=0x55a94cd0ee30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:00:42.633 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:42.624+0000 7f8887fc7740 1 -- 172.21.15.29:0/3690192019 shutdown_connections 2024-03-16T12:00:42.633 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:42.624+0000 7f8887fc7740 1 -- 172.21.15.29:0/3690192019 wait complete. 2024-03-16T12:00:42.633 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:42.624+0000 7f8887fc7740 1 -- start start 2024-03-16T12:00:42.633 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:42.624+0000 7f8887fc7740 1 Processor -- start 2024-03-16T12:00:42.633 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:42.624+0000 7f8887fc7740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a94cd14130 0x55a94cd1a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:00:42.633 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:42.625+0000 7f8887fc7740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a94cd14a50 0x55a94cd1ae30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:00:42.633 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:42.625+0000 7f8887fc7740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a94cd1d820 0x55a94cd1b370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:00:42.633 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:42.625+0000 7f8887fc7740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55a94cd20450 con 0x55a94cd1d820 2024-03-16T12:00:42.633 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:42.625+0000 7f8887fc7740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x55a94cc34d70 con 0x55a94cd14a50 2024-03-16T12:00:42.634 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:42.625+0000 7f8887fc7740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55a94cd205d0 con 0x55a94cd14130 2024-03-16T12:00:42.634 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:42.625+0000 7f8887fc7740 10 client.0 _ll_register_callbacks cb 0x55a94c98ca00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T12:00:42.634 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:42.625+0000 7f887f7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a94cd1d820 0x55a94cd1b370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:00:42.634 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:42.625+0000 7f88767fc640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a94cd14a50 0x55a94cd1ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:00:42.634 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:42.625+0000 7f887f7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a94cd1d820 0x55a94cd1b370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33414/0 (socket says 172.21.15.29:33414) 2024-03-16T12:00:42.634 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:42.625+0000 7f887effd640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a94cd14130 0x55a94cd1a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:00:42.634 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:42.625+0000 7f887f7fe640 1 -- 172.21.15.29:0/3077757684 learned_addr learned my addr 172.21.15.29:0/3077757684 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T12:00:42.634 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:42.625+0000 7f887f7fe640 1 -- 172.21.15.29:0/3077757684 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a94cd14130 msgr2=0x55a94cd1a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:00:42.634 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:42.625+0000 7f887f7fe640 1 --2- 172.21.15.29:0/3077757684 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a94cd14130 0x55a94cd1a8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:00:42.634 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:42.625+0000 7f887f7fe640 1 -- 172.21.15.29:0/3077757684 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a94cd14a50 msgr2=0x55a94cd1ae30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:00:42.634 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:42.625+0000 7f887f7fe640 1 --2- 172.21.15.29:0/3077757684 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a94cd14a50 0x55a94cd1ae30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:00:42.634 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:42.625+0000 7f887f7fe640 1 -- 172.21.15.29:0/3077757684 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a94c98d210 con 0x55a94cd1d820 2024-03-16T12:00:42.634 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:42.625+0000 7f887f7fe640 1 --2- 172.21.15.29:0/3077757684 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a94cd1d820 0x55a94cd1b370 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f8878009270 tx=0x7f8878009500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=abc401dbf39edaf9 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:00:42.634 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:42.625+0000 7f88667fc640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-16T12:00:42.635 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:42.625+0000 7f88667fc640 1 -- 172.21.15.29:0/3077757684 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8878012be0 con 0x55a94cd1d820 2024-03-16T12:00:42.635 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:42.625+0000 7f88667fc640 1 -- 172.21.15.29:0/3077757684 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 8556+0+0 (secure 0 0 0) 0x7f8878012d80 con 0x55a94cd1d820 2024-03-16T12:00:42.635 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:42.625+0000 7f8887fc7740 1 -- 172.21.15.29:0/3077757684 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f887009f920 con 0x55a94cd1d820 2024-03-16T12:00:42.635 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:42.625+0000 7f88667fc640 1 -- 172.21.15.29:0/3077757684 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f8878021440 con 0x55a94cd1d820 2024-03-16T12:00:42.635 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:42.626+0000 7f8887fc7740 10 client.16212 fetch_fsmap learned FSMap version 26 2024-03-16T12:00:42.635 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:42.626+0000 7f8887fc7740 1 -- 172.21.15.29:0/3077757684 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55a94ca59620 con 0x55a94cd1d820 2024-03-16T12:00:42.635 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:42.626+0000 7f88667fc640 1 -- 172.21.15.29:0/3077757684 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f887009f920 con 0x55a94cd1d820 2024-03-16T12:00:42.635 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:42.626+0000 7f88667fc640 1 -- 172.21.15.29:0/3077757684 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f887801f730 con 0x55a94cd1d820 2024-03-16T12:00:42.635 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:42.626+0000 7f8887fc7740 10 client.16212 fetch_fsmap finished waiting for FSMap version 26 2024-03-16T12:00:42.635 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:42.626+0000 7f8887fc7740 10 client.16212 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T12:00:42.635 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:42.626+0000 7f8887fc7740 20 client.16212 populate_metadata read hostname 'smithi029' 2024-03-16T12:00:42.635 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:42.626+0000 7f8887fc7740 1 --2- 172.21.15.29:0/3077757684 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55a94cd2da30 0x55a94cd2fe20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:00:42.635 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:42.626+0000 7f8887fc7740 4 client.16212 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T12:00:42.635 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:42.626+0000 7f8887fc7740 1 -- 172.21.15.29:0/3077757684 --> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a94cc85f80 con 0x55a94cd2da30 2024-03-16T12:00:42.635 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:42.626+0000 7f887effd640 1 --2- 172.21.15.29:0/3077757684 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55a94cd2da30 0x55a94cd2fe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:00:42.636 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:42.626+0000 7f88667fc640 10 client.16212 ms_handle_connect on v2:172.21.15.152:6838/1548731833 2024-03-16T12:00:42.636 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:42.626+0000 7f887effd640 1 --2- 172.21.15.29:0/3077757684 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55a94cd2da30 0x55a94cd2fe20 crc :-1 s=READY pgs=126 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1d91eb65b7a39d78 server_cookie=bb29e2cf092e4a02 in_seq=0 out_seq=0 2024-03-16T12:00:42.636 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:42.628+0000 7f88667fc640 1 -- 172.21.15.29:0/3077757684 <== mds.0 v2:172.21.15.152:6838/1548731833 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55a94cc85f80 con 0x55a94cd2da30 2024-03-16T12:00:42.636 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:42.628+0000 7f88667fc640 10 client.16212 handle_command_reply: tid=0 2024-03-16T12:00:42.636 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:42.628+0000 7f8887fc7740 1 client.16212 shutdown 2024-03-16T12:00:42.636 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:42.628+0000 7f8887fc7740 1 -- 172.21.15.29:0/3077757684 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a94cd1d820 msgr2=0x55a94cd1b370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:00:42.636 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:42.628+0000 7f8887fc7740 1 --2- 172.21.15.29:0/3077757684 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a94cd1d820 0x55a94cd1b370 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f8878009270 tx=0x7f8878009500 comp rx=0 tx=0).stop 2024-03-16T12:00:42.636 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:42.628+0000 7f8887fc7740 1 -- 172.21.15.29:0/3077757684 shutdown_connections 2024-03-16T12:00:42.636 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:42.628+0000 7f8887fc7740 1 --2- 172.21.15.29:0/3077757684 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a94cd1d820 0x55a94cd1b370 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:00:42.636 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:42.628+0000 7f8887fc7740 1 --2- 172.21.15.29:0/3077757684 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55a94cd2da30 0x55a94cd2fe20 crc :-1 s=READY pgs=126 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:00:42.636 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:42.628+0000 7f8887fc7740 1 --2- 172.21.15.29:0/3077757684 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a94cd14a50 0x55a94cd1ae30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:00:42.636 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:42.628+0000 7f8887fc7740 1 --2- 172.21.15.29:0/3077757684 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a94cd14130 0x55a94cd1a8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:00:42.636 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:42.628+0000 7f88667fc640 0 client.16212 ms_handle_reset on v2:172.21.15.152:6838/1548731833 2024-03-16T12:00:42.636 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:42.628+0000 7f8887fc7740 1 -- 172.21.15.29:0/3077757684 >> 172.21.15.29:0/3077757684 conn(0x55a94cd0ea50 msgr2=0x55a94cd107a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:00:42.637 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:42.628+0000 7f8887fc7740 1 -- 172.21.15.29:0/3077757684 shutdown_connections 2024-03-16T12:00:42.637 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:42.628+0000 7f8887fc7740 1 -- 172.21.15.29:0/3077757684 wait complete. 2024-03-16T12:00:42.637 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:42.628+0000 7f8887fc7740 20 client.16212 trim_cache size 0 max 16384 2024-03-16T12:00:42.648 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:42.644+0000 7f8887fc7740 1 Processor -- start 2024-03-16T12:00:42.648 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:42.644+0000 7f8887fc7740 1 -- start start 2024-03-16T12:00:42.648 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:42.644+0000 7f8887fc7740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a94cd14130 0x55a94cd291f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:00:42.649 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:42.644+0000 7f8887fc7740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a94cd14a50 0x55a94cd35710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:00:42.649 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:42.644+0000 7f8887fc7740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a94cd1d820 0x55a94cd0d320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:00:42.649 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:42.644+0000 7f8887fc7740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f88801afd90 con 0x55a94cd14130 2024-03-16T12:00:42.649 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:42.644+0000 7f8887fc7740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f88801ad660 con 0x55a94cd14a50 2024-03-16T12:00:42.649 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:42.644+0000 7f8887fc7740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f88801ab4d0 con 0x55a94cd1d820 2024-03-16T12:00:42.649 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:42.644+0000 7f887effd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a94cd14130 0x55a94cd291f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:00:42.649 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:42.644+0000 7f887effd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a94cd14130 0x55a94cd291f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33420/0 (socket says 172.21.15.29:33420) 2024-03-16T12:00:42.649 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:42.644+0000 7f887effd640 1 -- 172.21.15.29:0/264096313 learned_addr learned my addr 172.21.15.29:0/264096313 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T12:00:42.649 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:42.644+0000 7f88767fc640 1 --2- 172.21.15.29:0/264096313 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a94cd14a50 0x55a94cd35710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:00:42.649 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:42.644+0000 7f887f7fe640 1 --2- 172.21.15.29:0/264096313 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a94cd1d820 0x55a94cd0d320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:00:42.649 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:42.644+0000 7f887effd640 1 -- 172.21.15.29:0/264096313 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a94cd1d820 msgr2=0x55a94cd0d320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:00:42.649 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:42.644+0000 7f887effd640 1 --2- 172.21.15.29:0/264096313 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a94cd1d820 0x55a94cd0d320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:00:42.649 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:42.644+0000 7f887effd640 1 -- 172.21.15.29:0/264096313 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a94cd14a50 msgr2=0x55a94cd35710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:00:42.649 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:42.644+0000 7f887effd640 1 --2- 172.21.15.29:0/264096313 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a94cd14a50 0x55a94cd35710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:00:42.650 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:42.644+0000 7f887effd640 1 -- 172.21.15.29:0/264096313 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a94cd15090 con 0x55a94cd14130 2024-03-16T12:00:42.650 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:42.644+0000 7f887effd640 1 --2- 172.21.15.29:0/264096313 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a94cd14130 0x55a94cd291f0 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f887009b7c0 tx=0x7f887006fb60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd55b1ed54d56aad server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:00:42.650 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:42.645+0000 7f8874ff9640 1 -- 172.21.15.29:0/264096313 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88700b3020 con 0x55a94cd14130 2024-03-16T12:00:42.650 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:42.645+0000 7f8874ff9640 1 -- 172.21.15.29:0/264096313 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f887009b880 con 0x55a94cd14130 2024-03-16T12:00:42.650 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:42.645+0000 7f8874ff9640 1 -- 172.21.15.29:0/264096313 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88700a3ae0 con 0x55a94cd14130 2024-03-16T12:00:42.650 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:42.645+0000 7f8887fc7740 1 -- 172.21.15.29:0/264096313 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a94cd14130 msgr2=0x55a94cd291f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:00:42.650 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:42.645+0000 7f8887fc7740 1 --2- 172.21.15.29:0/264096313 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a94cd14130 0x55a94cd291f0 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f887009b7c0 tx=0x7f887006fb60 comp rx=0 tx=0).stop 2024-03-16T12:00:42.650 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:42.645+0000 7f8887fc7740 1 -- 172.21.15.29:0/264096313 shutdown_connections 2024-03-16T12:00:42.650 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:42.645+0000 7f8887fc7740 1 --2- 172.21.15.29:0/264096313 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a94cd1d820 0x55a94cd0d320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:00:42.650 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:42.645+0000 7f8887fc7740 1 --2- 172.21.15.29:0/264096313 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a94cd14a50 0x55a94cd35710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:00:42.650 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:42.645+0000 7f8887fc7740 1 --2- 172.21.15.29:0/264096313 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a94cd14130 0x55a94cd291f0 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:00:42.650 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:42.645+0000 7f8887fc7740 1 -- 172.21.15.29:0/264096313 >> 172.21.15.29:0/264096313 conn(0x55a94cd0ea50 msgr2=0x55a94cd09a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:00:42.650 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:42.645+0000 7f8887fc7740 1 -- 172.21.15.29:0/264096313 shutdown_connections 2024-03-16T12:00:42.650 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:42.645+0000 7f8887fc7740 1 -- 172.21.15.29:0/264096313 wait complete. 2024-03-16T12:00:42.651 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:42.646+0000 7f8887fc7740 1 -- start start 2024-03-16T12:00:42.651 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:42.646+0000 7f8887fc7740 1 Processor -- start 2024-03-16T12:00:42.651 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:42.646+0000 7f8887fc7740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a94cd14130 0x55a94cd17a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:00:42.651 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:42.646+0000 7f8887fc7740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a94cd14a50 0x55a94cd17fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:00:42.651 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:42.646+0000 7f8887fc7740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a94cd1d820 0x55a94cd92d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:00:42.651 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:42.646+0000 7f8887fc7740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f88801abb20 con 0x55a94cd14a50 2024-03-16T12:00:42.651 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:42.646+0000 7f8887fc7740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f88801acf90 con 0x55a94cd1d820 2024-03-16T12:00:42.651 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:42.646+0000 7f8887fc7740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f88801ac1f0 con 0x55a94cd14130 2024-03-16T12:00:42.651 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:42.646+0000 7f8887fc7740 10 client.0 _ll_register_callbacks cb 0x7f88780135a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T12:00:42.651 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:42.646+0000 7f88767fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a94cd14a50 0x55a94cd17fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:00:42.651 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:42.646+0000 7f88767fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a94cd14a50 0x55a94cd17fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33426/0 (socket says 172.21.15.29:33426) 2024-03-16T12:00:42.651 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:42.646+0000 7f88767fc640 1 -- 172.21.15.29:0/4121622460 learned_addr learned my addr 172.21.15.29:0/4121622460 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T12:00:42.651 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:42.646+0000 7f887effd640 1 --2- 172.21.15.29:0/4121622460 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a94cd14130 0x55a94cd17a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:00:42.651 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:42.646+0000 7f887f7fe640 1 --2- 172.21.15.29:0/4121622460 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a94cd1d820 0x55a94cd92d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:00:42.651 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:42.646+0000 7f88767fc640 1 -- 172.21.15.29:0/4121622460 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a94cd14130 msgr2=0x55a94cd17a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:00:42.652 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:42.646+0000 7f88767fc640 1 --2- 172.21.15.29:0/4121622460 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a94cd14130 0x55a94cd17a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:00:42.652 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:42.646+0000 7f88767fc640 1 -- 172.21.15.29:0/4121622460 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a94cd1d820 msgr2=0x55a94cd92d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:00:42.652 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:42.646+0000 7f88767fc640 1 --2- 172.21.15.29:0/4121622460 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a94cd1d820 0x55a94cd92d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:00:42.652 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:42.646+0000 7f88767fc640 1 -- 172.21.15.29:0/4121622460 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a94cd93280 con 0x55a94cd14a50 2024-03-16T12:00:42.652 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:42.646+0000 7f88767fc640 1 --2- 172.21.15.29:0/4121622460 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a94cd14a50 0x55a94cd17fa0 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f886802e010 tx=0x7f886802ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5422f1df33e0a602 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:00:42.652 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:42.646+0000 7f88667fc640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-16T12:00:42.652 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:42.647+0000 7f88667fc640 1 -- 172.21.15.29:0/4121622460 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f886804f060 con 0x55a94cd14a50 2024-03-16T12:00:42.652 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:42.647+0000 7f88667fc640 1 -- 172.21.15.29:0/4121622460 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 8556+0+0 (secure 0 0 0) 0x7f88680042f0 con 0x55a94cd14a50 2024-03-16T12:00:42.652 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:42.647+0000 7f8887fc7740 1 -- 172.21.15.29:0/4121622460 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f887009b880 con 0x55a94cd14a50 2024-03-16T12:00:42.652 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:42.647+0000 7f88667fc640 1 -- 172.21.15.29:0/4121622460 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f88680046c0 con 0x55a94cd14a50 2024-03-16T12:00:42.652 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:42.647+0000 7f8887fc7740 10 client.16218 fetch_fsmap learned FSMap version 26 2024-03-16T12:00:42.652 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:42.647+0000 7f8887fc7740 1 -- 172.21.15.29:0/4121622460 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55a94cd94430 con 0x55a94cd14a50 2024-03-16T12:00:42.652 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:42.647+0000 7f88667fc640 1 -- 172.21.15.29:0/4121622460 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f887009b880 con 0x55a94cd14a50 2024-03-16T12:00:42.652 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:42.647+0000 7f88667fc640 1 -- 172.21.15.29:0/4121622460 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f88680035f0 con 0x55a94cd14a50 2024-03-16T12:00:42.652 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:42.647+0000 7f8887fc7740 10 client.16218 fetch_fsmap finished waiting for FSMap version 26 2024-03-16T12:00:42.653 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:42.647+0000 7f8887fc7740 10 client.16218 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T12:00:42.653 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:42.647+0000 7f8887fc7740 20 client.16218 populate_metadata read hostname 'smithi029' 2024-03-16T12:00:42.653 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:42.647+0000 7f8887fc7740 1 --2- 172.21.15.29:0/4121622460 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55a94cd94bb0 0x55a94cd96fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:00:42.653 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:42.647+0000 7f8887fc7740 4 client.16218 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-16T12:00:42.653 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:42.647+0000 7f8887fc7740 1 -- 172.21.15.29:0/4121622460 --> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55a94cc802d0 con 0x55a94cd94bb0 2024-03-16T12:00:42.653 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:42.647+0000 7f887effd640 1 --2- 172.21.15.29:0/4121622460 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55a94cd94bb0 0x55a94cd96fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:00:42.653 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:42.648+0000 7f88667fc640 10 client.16218 ms_handle_connect on v2:172.21.15.152:6838/1548731833 2024-03-16T12:00:42.653 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:42.648+0000 7f887effd640 1 --2- 172.21.15.29:0/4121622460 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55a94cd94bb0 0x55a94cd96fd0 crc :-1 s=READY pgs=127 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=46a11fb35dcd11c9 server_cookie=5d280597686f8a72 in_seq=0 out_seq=0 2024-03-16T12:00:42.799 INFO:tasks.workunit.client.0.smithi029.stdout: 1 973482 9.87 MB/sec execute 387 sec latency 121.976 ms 2024-03-16T12:00:42.848 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:42.848+0000 7f88667fc640 1 -- 172.21.15.29:0/4121622460 <== mds.0 v2:172.21.15.152:6838/1548731833 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55a94cc802d0 con 0x55a94cd94bb0 2024-03-16T12:00:42.848 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:42.848+0000 7f88667fc640 10 client.16218 handle_command_reply: tid=0 2024-03-16T12:00:42.848 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:42.848+0000 7f8887fc7740 1 client.16218 shutdown 2024-03-16T12:00:42.848 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:42.848+0000 7f8887fc7740 1 -- 172.21.15.29:0/4121622460 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a94cd14a50 msgr2=0x55a94cd17fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:00:42.848 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:42.848+0000 7f8887fc7740 1 --2- 172.21.15.29:0/4121622460 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a94cd14a50 0x55a94cd17fa0 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f886802e010 tx=0x7f886802ce30 comp rx=0 tx=0).stop 2024-03-16T12:00:42.848 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:42.848+0000 7f8887fc7740 1 -- 172.21.15.29:0/4121622460 shutdown_connections 2024-03-16T12:00:42.848 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:42.848+0000 7f8887fc7740 1 --2- 172.21.15.29:0/4121622460 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55a94cd94bb0 0x55a94cd96fd0 crc :-1 s=READY pgs=127 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:00:42.848 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:42.848+0000 7f8887fc7740 1 --2- 172.21.15.29:0/4121622460 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a94cd1d820 0x55a94cd92d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:00:42.848 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:42.848+0000 7f88667fc640 0 client.16218 ms_handle_reset on v2:172.21.15.152:6838/1548731833 2024-03-16T12:00:42.849 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:42.848+0000 7f8887fc7740 1 --2- 172.21.15.29:0/4121622460 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a94cd14a50 0x55a94cd17fa0 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:00:42.849 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:42.848+0000 7f8887fc7740 1 --2- 172.21.15.29:0/4121622460 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a94cd14130 0x55a94cd17a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:00:42.849 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:42.848+0000 7f8887fc7740 1 -- 172.21.15.29:0/4121622460 >> 172.21.15.29:0/4121622460 conn(0x55a94cd0ea50 msgr2=0x55a94cd0d320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:00:42.849 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:42.848+0000 7f8887fc7740 1 -- 172.21.15.29:0/4121622460 shutdown_connections 2024-03-16T12:00:42.849 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:42.848+0000 7f8887fc7740 1 -- 172.21.15.29:0/4121622460 wait complete. 2024-03-16T12:00:42.849 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:42.848+0000 7f8887fc7740 20 client.16218 trim_cache size 0 max 16384 2024-03-16T12:00:42.849 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:{ 2024-03-16T12:00:42.849 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "message": "", 2024-03-16T12:00:42.849 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "return_code": 0 2024-03-16T12:00:42.849 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:} 2024-03-16T12:00:42.850 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:42.849+0000 7f88852ff640 1 -- 172.21.15.29:0/3657351055 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f886c0a2a10 msgr2=0x7f886c0a4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:00:42.850 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:42.849+0000 7f88852ff640 1 --2- 172.21.15.29:0/3657351055 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f886c0a2a10 0x7f886c0a4ed0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x55a94cd0fe70 tx=0x7f8878009bf0 comp rx=0 tx=0).stop 2024-03-16T12:00:42.850 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:42.849+0000 7f88852ff640 1 -- 172.21.15.29:0/3657351055 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8880100930 msgr2=0x7f88801980b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:00:42.851 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:42.849+0000 7f88852ff640 1 --2- 172.21.15.29:0/3657351055 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8880100930 0x7f88801980b0 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f887000e9f0 tx=0x7f887000eeb0 comp rx=0 tx=0).stop 2024-03-16T12:00:42.851 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:42.849+0000 7f88852ff640 1 -- 172.21.15.29:0/3657351055 shutdown_connections 2024-03-16T12:00:42.851 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:42.849+0000 7f88852ff640 1 --2- 172.21.15.29:0/3657351055 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f886c0a2a10 0x7f886c0a4ed0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:00:42.851 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:42.849+0000 7f88852ff640 1 --2- 172.21.15.29:0/3657351055 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8880105660 0x7f888019c920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:00:42.851 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:42.849+0000 7f88852ff640 1 --2- 172.21.15.29:0/3657351055 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f88801012d0 0x7f88801985f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:00:42.851 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:42.849+0000 7f88852ff640 1 --2- 172.21.15.29:0/3657351055 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8880100930 0x7f88801980b0 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:00:42.851 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:42.849+0000 7f88852ff640 1 -- 172.21.15.29:0/3657351055 >> 172.21.15.29:0/3657351055 conn(0x7f88800774b0 msgr2=0x7f88800fda50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:00:42.851 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:42.849+0000 7f88852ff640 1 -- 172.21.15.29:0/3657351055 shutdown_connections 2024-03-16T12:00:42.851 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:42.849+0000 7f88852ff640 1 -- 172.21.15.29:0/3657351055 wait complete. 2024-03-16T12:00:43.800 INFO:tasks.workunit.client.0.smithi029.stdout: 1 974392 9.85 MB/sec execute 388 sec latency 94.333 ms 2024-03-16T12:00:44.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:00:44 smithi152 ceph-mon[36291]: pgmap v308: 97 pgs: 97 active+clean; 95 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 3.6 MiB/s wr, 153 op/s 2024-03-16T12:00:44.545 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:00:44 smithi154 ceph-mon[36320]: pgmap v308: 97 pgs: 97 active+clean; 95 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 3.6 MiB/s wr, 153 op/s 2024-03-16T12:00:44.587 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:00:44 smithi029 ceph-mon[31975]: pgmap v308: 97 pgs: 97 active+clean; 95 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 3.6 MiB/s wr, 153 op/s 2024-03-16T12:00:44.800 INFO:tasks.workunit.client.0.smithi029.stdout: 1 976458 9.86 MB/sec execute 389 sec latency 31.008 ms 2024-03-16T12:00:45.800 INFO:tasks.workunit.client.0.smithi029.stdout: 1 977636 9.85 MB/sec execute 390 sec latency 33.648 ms 2024-03-16T12:00:46.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:00:46 smithi152 ceph-mon[36291]: pgmap v309: 97 pgs: 97 active+clean; 95 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 3.5 MiB/s wr, 143 op/s 2024-03-16T12:00:46.545 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:00:46 smithi154 ceph-mon[36320]: pgmap v309: 97 pgs: 97 active+clean; 95 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 3.5 MiB/s wr, 143 op/s 2024-03-16T12:00:46.587 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:00:46 smithi029 ceph-mon[31975]: pgmap v309: 97 pgs: 97 active+clean; 95 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 3.5 MiB/s wr, 143 op/s 2024-03-16T12:00:46.800 INFO:tasks.workunit.client.0.smithi029.stdout: 1 979575 9.84 MB/sec execute 391 sec latency 22.869 ms 2024-03-16T12:00:47.800 INFO:tasks.workunit.client.0.smithi029.stdout: 1 980965 9.85 MB/sec execute 392 sec latency 39.926 ms 2024-03-16T12:00:48.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:00:48 smithi152 ceph-mon[36291]: pgmap v310: 97 pgs: 97 active+clean; 86 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 4.5 MiB/s wr, 293 op/s 2024-03-16T12:00:48.545 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:00:48 smithi154 ceph-mon[36320]: pgmap v310: 97 pgs: 97 active+clean; 86 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 4.5 MiB/s wr, 293 op/s 2024-03-16T12:00:48.587 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:00:48 smithi029 ceph-mon[31975]: pgmap v310: 97 pgs: 97 active+clean; 86 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 4.5 MiB/s wr, 293 op/s 2024-03-16T12:00:48.800 INFO:tasks.workunit.client.0.smithi029.stdout: 1 982262 9.83 MB/sec execute 393 sec latency 63.790 ms 2024-03-16T12:00:49.800 INFO:tasks.workunit.client.0.smithi029.stdout: 1 984223 9.84 MB/sec execute 394 sec latency 36.873 ms 2024-03-16T12:00:50.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:00:50 smithi152 ceph-mon[36291]: pgmap v311: 97 pgs: 97 active+clean; 86 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 2.0 MiB/s wr, 219 op/s 2024-03-16T12:00:50.545 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:00:50 smithi154 ceph-mon[36320]: pgmap v311: 97 pgs: 97 active+clean; 86 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 2.0 MiB/s wr, 219 op/s 2024-03-16T12:00:50.587 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:00:50 smithi029 ceph-mon[31975]: pgmap v311: 97 pgs: 97 active+clean; 86 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 2.0 MiB/s wr, 219 op/s 2024-03-16T12:00:50.802 INFO:tasks.workunit.client.0.smithi029.stdout: 1 985561 9.83 MB/sec execute 395 sec latency 24.496 ms 2024-03-16T12:00:51.800 INFO:tasks.workunit.client.0.smithi029.stdout: 1 987651 9.84 MB/sec execute 396 sec latency 52.970 ms 2024-03-16T12:00:52.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:00:52 smithi152 ceph-mon[36291]: pgmap v312: 97 pgs: 97 active+clean; 93 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 2.7 MiB/s wr, 284 op/s 2024-03-16T12:00:52.545 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:00:52 smithi154 ceph-mon[36320]: pgmap v312: 97 pgs: 97 active+clean; 93 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 2.7 MiB/s wr, 284 op/s 2024-03-16T12:00:52.588 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:00:52 smithi029 ceph-mon[31975]: pgmap v312: 97 pgs: 97 active+clean; 93 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 2.7 MiB/s wr, 284 op/s 2024-03-16T12:00:52.800 INFO:tasks.workunit.client.0.smithi029.stdout: 1 988451 9.82 MB/sec execute 397 sec latency 34.361 ms 2024-03-16T12:00:53.800 INFO:tasks.workunit.client.0.smithi029.stdout: 1 990139 9.81 MB/sec execute 398 sec latency 116.569 ms 2024-03-16T12:00:54.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:00:54 smithi152 ceph-mon[36291]: pgmap v313: 97 pgs: 97 active+clean; 93 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 1.7 MiB/s wr, 218 op/s 2024-03-16T12:00:54.545 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:00:54 smithi154 ceph-mon[36320]: pgmap v313: 97 pgs: 97 active+clean; 93 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 1.7 MiB/s wr, 218 op/s 2024-03-16T12:00:54.587 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:00:54 smithi029 ceph-mon[31975]: pgmap v313: 97 pgs: 97 active+clean; 93 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 1.7 MiB/s wr, 218 op/s 2024-03-16T12:00:54.800 INFO:tasks.workunit.client.0.smithi029.stdout: 1 991516 9.82 MB/sec execute 399 sec latency 18.003 ms 2024-03-16T12:00:55.800 INFO:tasks.workunit.client.0.smithi029.stdout: 1 992935 9.81 MB/sec execute 400 sec latency 27.308 ms 2024-03-16T12:00:55.938 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:55.937+0000 7fec87e7e640 1 -- 172.21.15.29:0/830929809 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fec80068e60 msgr2=0x7fec800692c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:00:55.938 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:55.937+0000 7fec87e7e640 1 --2- 172.21.15.29:0/830929809 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fec80068e60 0x7fec800692c0 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7fec70009900 tx=0x7fec7002bc30 comp rx=0 tx=0).stop 2024-03-16T12:00:55.938 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:55.937+0000 7fec87e7e640 1 -- 172.21.15.29:0/830929809 shutdown_connections 2024-03-16T12:00:55.938 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:55.937+0000 7fec87e7e640 1 --2- 172.21.15.29:0/830929809 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fec80069800 0x7fec8010e670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:00:55.938 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:55.937+0000 7fec87e7e640 1 --2- 172.21.15.29:0/830929809 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fec80068e60 0x7fec800692c0 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:00:55.938 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:55.937+0000 7fec87e7e640 1 --2- 172.21.15.29:0/830929809 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fec801078d0 0x7fec80107cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:00:55.939 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:55.937+0000 7fec87e7e640 1 -- 172.21.15.29:0/830929809 >> 172.21.15.29:0/830929809 conn(0x7fec80077530 msgr2=0x7fec80077940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:00:55.939 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:55.937+0000 7fec87e7e640 1 -- 172.21.15.29:0/830929809 shutdown_connections 2024-03-16T12:00:55.939 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:55.937+0000 7fec87e7e640 1 -- 172.21.15.29:0/830929809 wait complete. 2024-03-16T12:00:55.939 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:55.938+0000 7fec87e7e640 1 Processor -- start 2024-03-16T12:00:55.939 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:55.938+0000 7fec87e7e640 1 -- start start 2024-03-16T12:00:55.939 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:55.938+0000 7fec87e7e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fec80068e60 0x7fec80198030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:00:55.939 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:55.938+0000 7fec87e7e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fec80069800 0x7fec80198570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:00:55.939 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:55.938+0000 7fec87e7e640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fec801078d0 0x7fec8019c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:00:55.939 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:55.938+0000 7fec87e7e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fec80111c50 con 0x7fec80069800 2024-03-16T12:00:55.939 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:55.938+0000 7fec87e7e640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fec80111ad0 con 0x7fec801078d0 2024-03-16T12:00:55.939 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:55.938+0000 7fec87e7e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fec80111dd0 con 0x7fec80068e60 2024-03-16T12:00:55.939 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:55.938+0000 7fec85419640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fec80069800 0x7fec80198570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:00:55.942 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:55.938+0000 7fec85419640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fec80069800 0x7fec80198570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:34042/0 (socket says 172.21.15.29:34042) 2024-03-16T12:00:55.942 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:55.938+0000 7fec85419640 1 -- 172.21.15.29:0/1014531212 learned_addr learned my addr 172.21.15.29:0/1014531212 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T12:00:55.942 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:55.938+0000 7fec8641b640 1 --2- 172.21.15.29:0/1014531212 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fec801078d0 0x7fec8019c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:00:55.942 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:55.938+0000 7fec85c1a640 1 --2- 172.21.15.29:0/1014531212 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fec80068e60 0x7fec80198030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:00:55.945 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:55.939+0000 7fec85419640 1 -- 172.21.15.29:0/1014531212 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fec80068e60 msgr2=0x7fec80198030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:00:55.945 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:55.939+0000 7fec85419640 1 --2- 172.21.15.29:0/1014531212 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fec80068e60 0x7fec80198030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:00:55.946 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:55.939+0000 7fec85419640 1 -- 172.21.15.29:0/1014531212 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fec801078d0 msgr2=0x7fec8019c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:00:55.946 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:55.939+0000 7fec85419640 1 --2- 172.21.15.29:0/1014531212 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fec801078d0 0x7fec8019c8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:00:55.946 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:55.939+0000 7fec85419640 1 -- 172.21.15.29:0/1014531212 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec8019d020 con 0x7fec80069800 2024-03-16T12:00:55.946 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:55.939+0000 7fec85419640 1 --2- 172.21.15.29:0/1014531212 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fec80069800 0x7fec80198570 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7fec70006010 tx=0x7fec7002cb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:00:55.946 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:55.939+0000 7fec76ffd640 1 -- 172.21.15.29:0/1014531212 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec70041070 con 0x7fec80069800 2024-03-16T12:00:55.946 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:55.939+0000 7fec76ffd640 1 -- 172.21.15.29:0/1014531212 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fec70004450 con 0x7fec80069800 2024-03-16T12:00:55.946 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:55.939+0000 7fec87e7e640 1 -- 172.21.15.29:0/1014531212 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fec8019d2b0 con 0x7fec80069800 2024-03-16T12:00:55.946 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:55.939+0000 7fec76ffd640 1 -- 172.21.15.29:0/1014531212 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec70003e00 con 0x7fec80069800 2024-03-16T12:00:55.946 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:55.939+0000 7fec87e7e640 1 -- 172.21.15.29:0/1014531212 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fec801a4ba0 con 0x7fec80069800 2024-03-16T12:00:55.946 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:55.941+0000 7fec76ffd640 1 -- 172.21.15.29:0/1014531212 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7fec70044070 con 0x7fec80069800 2024-03-16T12:00:55.946 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:55.941+0000 7fec8aa62740 1 Processor -- start 2024-03-16T12:00:55.946 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:55.942+0000 7fec8aa62740 1 -- start start 2024-03-16T12:00:55.946 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:55.942+0000 7fec76ffd640 1 --2- 172.21.15.29:0/1014531212 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7fec6c0a2a10 0x7fec6c0a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:00:55.947 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:55.942+0000 7fec8aa62740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56079b5b0c50 0x56079b5b1030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:00:55.947 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:55.942+0000 7fec8aa62740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56079b5b1570 0x56079b5be6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:00:55.947 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:55.942+0000 7fec8641b640 1 --2- 172.21.15.29:0/1014531212 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7fec6c0a2a10 0x7fec6c0a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:00:55.947 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:55.942+0000 7fec76ffd640 1 -- 172.21.15.29:0/1014531212 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 8556+0+0 (secure 0 0 0) 0x7fec700f07a0 con 0x7fec80069800 2024-03-16T12:00:55.947 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:55.942+0000 7fec8aa62740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56079b5bec30 0x56079b5c1020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:00:55.947 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:55.942+0000 7fec8aa62740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x56079b515a40 con 0x56079b5b1570 2024-03-16T12:00:55.947 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:55.942+0000 7fec8aa62740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x56079b4be920 con 0x56079b5b0c50 2024-03-16T12:00:55.947 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:55.942+0000 7fec8aa62740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x56079b511ae0 con 0x56079b5bec30 2024-03-16T12:00:55.947 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:55.942+0000 7fec85c1a640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56079b5b0c50 0x56079b5b1030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:00:55.947 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:55.942+0000 7fec85c1a640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56079b5b0c50 0x56079b5b1030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.29:46234/0 (socket says 172.21.15.29:46234) 2024-03-16T12:00:55.947 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:55.942+0000 7fec85c1a640 1 -- 172.21.15.29:0/2993590019 learned_addr learned my addr 172.21.15.29:0/2993590019 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T12:00:55.947 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:55.942+0000 7fec85419640 1 --2- 172.21.15.29:0/2993590019 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56079b5b1570 0x56079b5be6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:00:55.947 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:55.942+0000 7fec8641b640 1 --2- 172.21.15.29:0/2993590019 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56079b5bec30 0x56079b5c1020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:00:55.947 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:55.942+0000 7fec85c1a640 1 -- 172.21.15.29:0/2993590019 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56079b5bec30 msgr2=0x56079b5c1020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:00:55.948 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:55.942+0000 7fec85c1a640 1 --2- 172.21.15.29:0/2993590019 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56079b5bec30 0x56079b5c1020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:00:55.948 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:55.942+0000 7fec85c1a640 1 -- 172.21.15.29:0/2993590019 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56079b5b1570 msgr2=0x56079b5be6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:00:55.948 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:55.942+0000 7fec85c1a640 1 --2- 172.21.15.29:0/2993590019 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56079b5b1570 0x56079b5be6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:00:55.948 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:55.942+0000 7fec85c1a640 1 -- 172.21.15.29:0/2993590019 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56079b4d6460 con 0x56079b5b0c50 2024-03-16T12:00:55.948 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:55.942+0000 7fec8641b640 1 --2- 172.21.15.29:0/1014531212 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7fec6c0a2a10 0x7fec6c0a4ed0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fec78009fd0 tx=0x7fec78011bc0 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:00:55.948 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:55.943+0000 7fec85c1a640 1 --2- 172.21.15.29:0/2993590019 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56079b5b0c50 0x56079b5b1030 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fec7c009cf0 tx=0x7fec7c011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4440438b9d25356e server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:00:55.948 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:55.943+0000 7fec6bfff640 1 -- 172.21.15.29:0/2993590019 <== mon.2 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec7c01e070 con 0x56079b5b0c50 2024-03-16T12:00:55.948 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:55.943+0000 7fec6bfff640 1 -- 172.21.15.29:0/2993590019 <== mon.2 v2:172.21.15.152:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fec7c009de0 con 0x56079b5b0c50 2024-03-16T12:00:55.948 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:55.943+0000 7fec6bfff640 1 -- 172.21.15.29:0/2993590019 <== mon.2 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec7c0196b0 con 0x56079b5b0c50 2024-03-16T12:00:55.948 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:55.943+0000 7fec8aa62740 1 -- 172.21.15.29:0/2993590019 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56079b5b0c50 msgr2=0x56079b5b1030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:00:55.948 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:55.943+0000 7fec8aa62740 1 --2- 172.21.15.29:0/2993590019 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56079b5b0c50 0x56079b5b1030 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fec7c009cf0 tx=0x7fec7c011810 comp rx=0 tx=0).stop 2024-03-16T12:00:55.948 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:55.944+0000 7fec8aa62740 1 -- 172.21.15.29:0/2993590019 shutdown_connections 2024-03-16T12:00:55.948 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:55.944+0000 7fec8aa62740 1 --2- 172.21.15.29:0/2993590019 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56079b5bec30 0x56079b5c1020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:00:55.948 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:55.944+0000 7fec8aa62740 1 --2- 172.21.15.29:0/2993590019 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56079b5b1570 0x56079b5be6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:00:55.948 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:55.944+0000 7fec8aa62740 1 --2- 172.21.15.29:0/2993590019 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56079b5b0c50 0x56079b5b1030 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:00:55.949 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:55.944+0000 7fec8aa62740 1 -- 172.21.15.29:0/2993590019 >> 172.21.15.29:0/2993590019 conn(0x56079b5aba50 msgr2=0x56079b5abe30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:00:55.949 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:55.944+0000 7fec8aa62740 1 -- 172.21.15.29:0/2993590019 shutdown_connections 2024-03-16T12:00:55.949 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:55.944+0000 7fec8aa62740 1 -- 172.21.15.29:0/2993590019 wait complete. 2024-03-16T12:00:55.953 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:55.944+0000 7fec8aa62740 1 -- start start 2024-03-16T12:00:55.953 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:55.944+0000 7fec8aa62740 1 Processor -- start 2024-03-16T12:00:55.953 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:55.944+0000 7fec8aa62740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56079b5b0c50 0x56079b5bbc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:00:55.953 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:55.944+0000 7fec8aa62740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56079b5b1570 0x56079b5bc150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:00:55.953 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:55.944+0000 7fec8aa62740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56079b5bec30 0x56079b5bc690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:00:55.953 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:55.944+0000 7fec8aa62740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x56079b5c17f0 con 0x56079b5b1570 2024-03-16T12:00:55.953 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:55.945+0000 7fec8aa62740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x56079b4d1d70 con 0x56079b5bec30 2024-03-16T12:00:55.953 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:55.945+0000 7fec8aa62740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x56079b5c1970 con 0x56079b5b0c50 2024-03-16T12:00:55.954 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:55.945+0000 7fec8aa62740 10 client.0 _ll_register_callbacks cb 0x56079b229a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T12:00:55.954 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:55.945+0000 7fec85419640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56079b5b1570 0x56079b5bc150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:00:55.954 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:55.945+0000 7fec85c1a640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56079b5b0c50 0x56079b5bbc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:00:55.954 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:55.945+0000 7fec85419640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56079b5b1570 0x56079b5bc150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:34060/0 (socket says 172.21.15.29:34060) 2024-03-16T12:00:55.954 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:55.945+0000 7fec85419640 1 -- 172.21.15.29:0/1581934138 learned_addr learned my addr 172.21.15.29:0/1581934138 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T12:00:55.954 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:55.945+0000 7fec8641b640 1 --2- 172.21.15.29:0/1581934138 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56079b5bec30 0x56079b5bc690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:00:55.954 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:55.945+0000 7fec85419640 1 -- 172.21.15.29:0/1581934138 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56079b5b0c50 msgr2=0x56079b5bbc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:00:55.954 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:55.945+0000 7fec85419640 1 --2- 172.21.15.29:0/1581934138 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56079b5b0c50 0x56079b5bbc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:00:55.954 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:55.945+0000 7fec85419640 1 -- 172.21.15.29:0/1581934138 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56079b5bec30 msgr2=0x56079b5bc690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:00:55.954 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:55.945+0000 7fec85419640 1 --2- 172.21.15.29:0/1581934138 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56079b5bec30 0x56079b5bc690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:00:55.954 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:55.945+0000 7fec85419640 1 -- 172.21.15.29:0/1581934138 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56079b22a210 con 0x56079b5b1570 2024-03-16T12:00:55.954 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:55.945+0000 7fec85419640 1 --2- 172.21.15.29:0/1581934138 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56079b5b1570 0x56079b5bc150 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7fec700bcca0 tx=0x7fec7002cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ade1398d0390894e server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:00:55.955 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:55.945+0000 7fec697fa640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-16T12:00:55.955 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:55.945+0000 7fec697fa640 1 -- 172.21.15.29:0/1581934138 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec7004b040 con 0x56079b5b1570 2024-03-16T12:00:55.955 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:55.945+0000 7fec697fa640 1 -- 172.21.15.29:0/1581934138 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 8556+0+0 (secure 0 0 0) 0x7fec700b9610 con 0x56079b5b1570 2024-03-16T12:00:55.955 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:55.945+0000 7fec8aa62740 1 -- 172.21.15.29:0/1581934138 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fec7c009de0 con 0x56079b5b1570 2024-03-16T12:00:55.955 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:55.945+0000 7fec697fa640 1 -- 172.21.15.29:0/1581934138 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fec700c5690 con 0x56079b5b1570 2024-03-16T12:00:55.955 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:55.945+0000 7fec8aa62740 10 client.16230 fetch_fsmap learned FSMap version 26 2024-03-16T12:00:55.955 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:55.945+0000 7fec8aa62740 1 -- 172.21.15.29:0/1581934138 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x56079b2f6620 con 0x56079b5b1570 2024-03-16T12:00:55.955 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:55.946+0000 7fec697fa640 1 -- 172.21.15.29:0/1581934138 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fec7c009de0 con 0x56079b5b1570 2024-03-16T12:00:55.955 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:55.946+0000 7fec697fa640 1 -- 172.21.15.29:0/1581934138 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7fec700bc4f0 con 0x56079b5b1570 2024-03-16T12:00:55.955 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:55.946+0000 7fec8aa62740 10 client.16230 fetch_fsmap finished waiting for FSMap version 26 2024-03-16T12:00:55.955 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:55.946+0000 7fec8aa62740 10 client.16230 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T12:00:55.955 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:55.946+0000 7fec8aa62740 20 client.16230 populate_metadata read hostname 'smithi029' 2024-03-16T12:00:55.955 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:55.946+0000 7fec8aa62740 1 --2- 172.21.15.29:0/1581934138 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x56079b5ced10 0x56079b5d1130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:00:55.955 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:55.946+0000 7fec8aa62740 4 client.16230 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T12:00:55.955 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:55.946+0000 7fec8aa62740 1 -- 172.21.15.29:0/1581934138 --> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56079b522f80 con 0x56079b5ced10 2024-03-16T12:00:55.956 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:55.946+0000 7fec85c1a640 1 --2- 172.21.15.29:0/1581934138 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x56079b5ced10 0x56079b5d1130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:00:55.956 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:55.946+0000 7fec697fa640 10 client.16230 ms_handle_connect on v2:172.21.15.152:6838/1548731833 2024-03-16T12:00:55.956 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:55.946+0000 7fec85c1a640 1 --2- 172.21.15.29:0/1581934138 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x56079b5ced10 0x56079b5d1130 crc :-1 s=READY pgs=128 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=29fdfbfd1b5dfc81 server_cookie=71ce125158f2bd0a in_seq=0 out_seq=0 2024-03-16T12:00:55.956 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:55.949+0000 7fec697fa640 1 -- 172.21.15.29:0/1581934138 <== mds.0 v2:172.21.15.152:6838/1548731833 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x56079b522f80 con 0x56079b5ced10 2024-03-16T12:00:55.956 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:55.949+0000 7fec697fa640 10 client.16230 handle_command_reply: tid=0 2024-03-16T12:00:55.956 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:55.949+0000 7fec8aa62740 1 client.16230 shutdown 2024-03-16T12:00:55.956 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:55.949+0000 7fec8aa62740 1 -- 172.21.15.29:0/1581934138 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56079b5b1570 msgr2=0x56079b5bc150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:00:55.956 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:55.949+0000 7fec8aa62740 1 --2- 172.21.15.29:0/1581934138 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56079b5b1570 0x56079b5bc150 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7fec700bcca0 tx=0x7fec7002cde0 comp rx=0 tx=0).stop 2024-03-16T12:00:55.956 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:55.949+0000 7fec8aa62740 1 -- 172.21.15.29:0/1581934138 shutdown_connections 2024-03-16T12:00:55.956 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:55.949+0000 7fec8aa62740 1 --2- 172.21.15.29:0/1581934138 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x56079b5ced10 0x56079b5d1130 crc :-1 s=READY pgs=128 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:00:55.956 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:55.949+0000 7fec8aa62740 1 --2- 172.21.15.29:0/1581934138 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56079b5bec30 0x56079b5bc690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:00:55.956 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:55.949+0000 7fec697fa640 0 client.16230 ms_handle_reset on v2:172.21.15.152:6838/1548731833 2024-03-16T12:00:55.956 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:55.949+0000 7fec8aa62740 1 --2- 172.21.15.29:0/1581934138 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56079b5b1570 0x56079b5bc150 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:00:55.956 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:55.949+0000 7fec8aa62740 1 --2- 172.21.15.29:0/1581934138 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56079b5b0c50 0x56079b5bbc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:00:55.956 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:55.949+0000 7fec8aa62740 1 -- 172.21.15.29:0/1581934138 >> 172.21.15.29:0/1581934138 conn(0x56079b5aba50 msgr2=0x56079b5ad7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:00:55.957 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:55.950+0000 7fec8aa62740 1 -- 172.21.15.29:0/1581934138 shutdown_connections 2024-03-16T12:00:55.957 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:55.950+0000 7fec8aa62740 1 -- 172.21.15.29:0/1581934138 wait complete. 2024-03-16T12:00:55.957 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:55.950+0000 7fec8aa62740 20 client.16230 trim_cache size 0 max 16384 2024-03-16T12:00:55.966 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:55.965+0000 7fec8aa62740 1 Processor -- start 2024-03-16T12:00:55.966 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:55.965+0000 7fec8aa62740 1 -- start start 2024-03-16T12:00:55.966 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:55.965+0000 7fec8aa62740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56079b5b0c50 0x56079b5ca5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:00:55.967 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:55.965+0000 7fec8aa62740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56079b5b1570 0x56079b5aaa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:00:55.967 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:55.965+0000 7fec8aa62740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56079b5bec30 0x56079b5d6a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:00:55.967 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:55.965+0000 7fec8aa62740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fec801adb80 con 0x56079b5b0c50 2024-03-16T12:00:55.967 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:55.965+0000 7fec8aa62740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fec801ab450 con 0x56079b5b1570 2024-03-16T12:00:55.967 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:55.965+0000 7fec8aa62740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fec801a92c0 con 0x56079b5bec30 2024-03-16T12:00:55.967 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:55.965+0000 7fec85c1a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56079b5b0c50 0x56079b5ca5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:00:55.967 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:55.965+0000 7fec85c1a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56079b5b0c50 0x56079b5ca5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:34074/0 (socket says 172.21.15.29:34074) 2024-03-16T12:00:55.967 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:55.965+0000 7fec85c1a640 1 -- 172.21.15.29:0/551442772 learned_addr learned my addr 172.21.15.29:0/551442772 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T12:00:55.967 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:55.965+0000 7fec85419640 1 --2- 172.21.15.29:0/551442772 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56079b5b1570 0x56079b5aaa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:00:55.967 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:55.965+0000 7fec8641b640 1 --2- 172.21.15.29:0/551442772 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56079b5bec30 0x56079b5d6a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:00:55.967 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:55.965+0000 7fec85c1a640 1 -- 172.21.15.29:0/551442772 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56079b5bec30 msgr2=0x56079b5d6a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:00:55.967 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:55.965+0000 7fec85c1a640 1 --2- 172.21.15.29:0/551442772 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56079b5bec30 0x56079b5d6a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:00:55.968 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:55.965+0000 7fec85c1a640 1 -- 172.21.15.29:0/551442772 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56079b5b1570 msgr2=0x56079b5aaa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:00:55.968 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:55.965+0000 7fec85c1a640 1 --2- 172.21.15.29:0/551442772 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56079b5b1570 0x56079b5aaa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:00:55.968 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:55.965+0000 7fec85c1a640 1 -- 172.21.15.29:0/551442772 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56079b5c1020 con 0x56079b5b0c50 2024-03-16T12:00:55.968 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:55.965+0000 7fec85c1a640 1 --2- 172.21.15.29:0/551442772 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56079b5b0c50 0x56079b5ca5d0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7fec7c003ca0 tx=0x7fec7c003cd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d028f937236294e8 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:00:55.968 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:55.966+0000 7fec6b7fe640 1 -- 172.21.15.29:0/551442772 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec7c02f070 con 0x56079b5b0c50 2024-03-16T12:00:55.968 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:55.966+0000 7fec6b7fe640 1 -- 172.21.15.29:0/551442772 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fec7c009de0 con 0x56079b5b0c50 2024-03-16T12:00:55.968 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:55.966+0000 7fec6b7fe640 1 -- 172.21.15.29:0/551442772 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec7c01e040 con 0x56079b5b0c50 2024-03-16T12:00:55.968 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:55.966+0000 7fec8aa62740 1 -- 172.21.15.29:0/551442772 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56079b5b0c50 msgr2=0x56079b5ca5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:00:55.968 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:55.966+0000 7fec8aa62740 1 --2- 172.21.15.29:0/551442772 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56079b5b0c50 0x56079b5ca5d0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7fec7c003ca0 tx=0x7fec7c003cd0 comp rx=0 tx=0).stop 2024-03-16T12:00:55.968 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:55.966+0000 7fec8aa62740 1 -- 172.21.15.29:0/551442772 shutdown_connections 2024-03-16T12:00:55.969 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:55.966+0000 7fec8aa62740 1 --2- 172.21.15.29:0/551442772 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56079b5bec30 0x56079b5d6a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:00:55.969 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:55.966+0000 7fec8aa62740 1 --2- 172.21.15.29:0/551442772 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56079b5b1570 0x56079b5aaa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:00:55.969 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:55.966+0000 7fec8aa62740 1 --2- 172.21.15.29:0/551442772 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56079b5b0c50 0x56079b5ca5d0 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:00:55.969 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:55.966+0000 7fec8aa62740 1 -- 172.21.15.29:0/551442772 >> 172.21.15.29:0/551442772 conn(0x56079b5aba50 msgr2=0x56079b5aa320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:00:55.969 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:55.966+0000 7fec8aa62740 1 -- 172.21.15.29:0/551442772 shutdown_connections 2024-03-16T12:00:55.973 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:55.966+0000 7fec8aa62740 1 -- 172.21.15.29:0/551442772 wait complete. 2024-03-16T12:00:55.974 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:55.967+0000 7fec8aa62740 1 -- start start 2024-03-16T12:00:55.974 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:55.967+0000 7fec8aa62740 1 Processor -- start 2024-03-16T12:00:55.974 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:55.967+0000 7fec8aa62740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56079b5b0c50 0x56079b5c3110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:00:55.974 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:55.967+0000 7fec8aa62740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56079b5b1570 0x56079b5c3650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:00:55.974 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:55.967+0000 7fec8aa62740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56079b5bec30 0x56079b62d010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:00:55.974 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:55.967+0000 7fec8aa62740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fec801a9910 con 0x56079b5b1570 2024-03-16T12:00:55.974 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:55.967+0000 7fec8aa62740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fec801aad80 con 0x56079b5bec30 2024-03-16T12:00:55.974 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:55.967+0000 7fec8aa62740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fec801a9fe0 con 0x56079b5b0c50 2024-03-16T12:00:55.974 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:55.967+0000 7fec8aa62740 10 client.0 _ll_register_callbacks cb 0x7fec700b9310 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T12:00:55.974 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:55.967+0000 7fec85419640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56079b5b1570 0x56079b5c3650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:00:55.974 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:55.967+0000 7fec85419640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56079b5b1570 0x56079b5c3650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:34086/0 (socket says 172.21.15.29:34086) 2024-03-16T12:00:55.975 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:55.967+0000 7fec85419640 1 -- 172.21.15.29:0/3406581355 learned_addr learned my addr 172.21.15.29:0/3406581355 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T12:00:55.975 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:55.967+0000 7fec85c1a640 1 --2- 172.21.15.29:0/3406581355 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56079b5b0c50 0x56079b5c3110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:00:55.975 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:55.967+0000 7fec8641b640 1 --2- 172.21.15.29:0/3406581355 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56079b5bec30 0x56079b62d010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:00:55.975 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:55.967+0000 7fec85419640 1 -- 172.21.15.29:0/3406581355 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56079b5b0c50 msgr2=0x56079b5c3110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:00:55.975 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:55.967+0000 7fec85419640 1 --2- 172.21.15.29:0/3406581355 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56079b5b0c50 0x56079b5c3110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:00:55.975 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:55.967+0000 7fec85419640 1 -- 172.21.15.29:0/3406581355 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56079b5bec30 msgr2=0x56079b62d010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:00:55.975 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:55.967+0000 7fec85419640 1 --2- 172.21.15.29:0/3406581355 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56079b5bec30 0x56079b62d010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:00:55.975 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:55.967+0000 7fec85419640 1 -- 172.21.15.29:0/3406581355 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56079b62d810 con 0x56079b5b1570 2024-03-16T12:00:55.975 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:55.967+0000 7fec8641b640 1 --2- 172.21.15.29:0/3406581355 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56079b5bec30 0x56079b62d010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T12:00:55.975 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:55.968+0000 7fec85419640 1 --2- 172.21.15.29:0/3406581355 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56079b5b1570 0x56079b5c3650 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7fec700bbe60 tx=0x7fec7002bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1619fc93aff0fe3 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:00:55.975 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:55.968+0000 7fec697fa640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-16T12:00:55.975 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:55.968+0000 7fec697fa640 1 -- 172.21.15.29:0/3406581355 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec7004b070 con 0x56079b5b1570 2024-03-16T12:00:55.975 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:55.968+0000 7fec697fa640 1 -- 172.21.15.29:0/3406581355 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 8556+0+0 (secure 0 0 0) 0x7fec700bc3f0 con 0x56079b5b1570 2024-03-16T12:00:55.975 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:55.968+0000 7fec8aa62740 1 -- 172.21.15.29:0/3406581355 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fec7c009de0 con 0x56079b5b1570 2024-03-16T12:00:55.976 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:55.968+0000 7fec697fa640 1 -- 172.21.15.29:0/3406581355 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fec700ca050 con 0x56079b5b1570 2024-03-16T12:00:55.976 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:55.968+0000 7fec8aa62740 10 client.16236 fetch_fsmap learned FSMap version 26 2024-03-16T12:00:55.976 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:55.968+0000 7fec8aa62740 1 -- 172.21.15.29:0/3406581355 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x56079b635c60 con 0x56079b5b1570 2024-03-16T12:00:55.976 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:55.968+0000 7fec697fa640 1 -- 172.21.15.29:0/3406581355 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fec7c009de0 con 0x56079b5b1570 2024-03-16T12:00:55.976 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:55.968+0000 7fec697fa640 1 -- 172.21.15.29:0/3406581355 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7fec700a60d0 con 0x56079b5b1570 2024-03-16T12:00:55.976 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:55.968+0000 7fec8aa62740 10 client.16236 fetch_fsmap finished waiting for FSMap version 26 2024-03-16T12:00:55.976 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:55.968+0000 7fec8aa62740 10 client.16236 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T12:00:55.976 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:55.968+0000 7fec8aa62740 20 client.16236 populate_metadata read hostname 'smithi029' 2024-03-16T12:00:55.976 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:55.968+0000 7fec8aa62740 1 --2- 172.21.15.29:0/3406581355 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x56079b636280 0x56079b6386a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:00:55.976 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:55.968+0000 7fec8aa62740 4 client.16236 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-16T12:00:55.976 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:55.968+0000 7fec8aa62740 1 -- 172.21.15.29:0/3406581355 --> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] -- command(tid 0: {"prefix": "flush journal"}) -- 0x56079b51d2d0 con 0x56079b636280 2024-03-16T12:00:55.976 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:55.968+0000 7fec85c1a640 1 --2- 172.21.15.29:0/3406581355 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x56079b636280 0x56079b6386a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:00:55.976 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:55.969+0000 7fec85c1a640 1 --2- 172.21.15.29:0/3406581355 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x56079b636280 0x56079b6386a0 crc :-1 s=READY pgs=129 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=593beb1974a16400 server_cookie=2875086f1525e9e2 in_seq=0 out_seq=0 2024-03-16T12:00:55.976 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:55.969+0000 7fec697fa640 10 client.16236 ms_handle_connect on v2:172.21.15.152:6838/1548731833 2024-03-16T12:00:56.174 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:56.173+0000 7fec697fa640 1 -- 172.21.15.29:0/3406581355 <== mds.0 v2:172.21.15.152:6838/1548731833 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x56079b51d2d0 con 0x56079b636280 2024-03-16T12:00:56.174 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:56.173+0000 7fec697fa640 10 client.16236 handle_command_reply: tid=0 2024-03-16T12:00:56.174 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:56.173+0000 7fec8aa62740 1 client.16236 shutdown 2024-03-16T12:00:56.174 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:56.174+0000 7fec8aa62740 1 -- 172.21.15.29:0/3406581355 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56079b5b1570 msgr2=0x56079b5c3650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:00:56.174 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:56.174+0000 7fec8aa62740 1 --2- 172.21.15.29:0/3406581355 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56079b5b1570 0x56079b5c3650 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7fec700bbe60 tx=0x7fec7002bc30 comp rx=0 tx=0).stop 2024-03-16T12:00:56.174 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:56.174+0000 7fec8aa62740 1 -- 172.21.15.29:0/3406581355 shutdown_connections 2024-03-16T12:00:56.174 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:56.174+0000 7fec8aa62740 1 --2- 172.21.15.29:0/3406581355 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x56079b636280 0x56079b6386a0 crc :-1 s=READY pgs=129 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:00:56.175 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:56.174+0000 7fec697fa640 0 client.16236 ms_handle_reset on v2:172.21.15.152:6838/1548731833 2024-03-16T12:00:56.175 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:56.174+0000 7fec8aa62740 1 --2- 172.21.15.29:0/3406581355 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56079b5bec30 0x56079b62d010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:00:56.175 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:56.174+0000 7fec8aa62740 1 --2- 172.21.15.29:0/3406581355 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56079b5b1570 0x56079b5c3650 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:00:56.175 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:56.174+0000 7fec8aa62740 1 --2- 172.21.15.29:0/3406581355 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56079b5b0c50 0x56079b5c3110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:00:56.175 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:56.174+0000 7fec8aa62740 1 -- 172.21.15.29:0/3406581355 >> 172.21.15.29:0/3406581355 conn(0x56079b5aba50 msgr2=0x56079b5bbbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:00:56.175 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:56.174+0000 7fec8aa62740 1 -- 172.21.15.29:0/3406581355 shutdown_connections 2024-03-16T12:00:56.175 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:56.174+0000 7fec8aa62740 1 -- 172.21.15.29:0/3406581355 wait complete. 2024-03-16T12:00:56.175 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:56.174+0000 7fec8aa62740 20 client.16236 trim_cache size 0 max 16384 2024-03-16T12:00:56.175 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:{ 2024-03-16T12:00:56.175 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "message": "", 2024-03-16T12:00:56.175 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "return_code": 0 2024-03-16T12:00:56.176 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:} 2024-03-16T12:00:56.176 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:56.175+0000 7fec87e7e640 1 -- 172.21.15.29:0/1014531212 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7fec6c0a2a10 msgr2=0x7fec6c0a4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:00:56.176 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:56.175+0000 7fec87e7e640 1 --2- 172.21.15.29:0/1014531212 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7fec6c0a2a10 0x7fec6c0a4ed0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fec78009fd0 tx=0x7fec78011bc0 comp rx=0 tx=0).stop 2024-03-16T12:00:56.176 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:56.175+0000 7fec87e7e640 1 -- 172.21.15.29:0/1014531212 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fec80069800 msgr2=0x7fec80198570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:00:56.176 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:56.175+0000 7fec87e7e640 1 --2- 172.21.15.29:0/1014531212 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fec80069800 0x7fec80198570 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7fec70006010 tx=0x7fec7002cb30 comp rx=0 tx=0).stop 2024-03-16T12:00:56.177 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:56.175+0000 7fec87e7e640 1 -- 172.21.15.29:0/1014531212 shutdown_connections 2024-03-16T12:00:56.177 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:56.175+0000 7fec87e7e640 1 --2- 172.21.15.29:0/1014531212 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7fec6c0a2a10 0x7fec6c0a4ed0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:00:56.177 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:56.175+0000 7fec87e7e640 1 --2- 172.21.15.29:0/1014531212 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fec801078d0 0x7fec8019c8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:00:56.177 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:56.175+0000 7fec87e7e640 1 --2- 172.21.15.29:0/1014531212 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fec80069800 0x7fec80198570 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:00:56.177 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:56.175+0000 7fec87e7e640 1 --2- 172.21.15.29:0/1014531212 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fec80068e60 0x7fec80198030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:00:56.177 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:56.175+0000 7fec87e7e640 1 -- 172.21.15.29:0/1014531212 >> 172.21.15.29:0/1014531212 conn(0x7fec80077530 msgr2=0x7fec800fe9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:00:56.177 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:56.175+0000 7fec87e7e640 1 -- 172.21.15.29:0/1014531212 shutdown_connections 2024-03-16T12:00:56.177 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:00:56.175+0000 7fec87e7e640 1 -- 172.21.15.29:0/1014531212 wait complete. 2024-03-16T12:00:56.545 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:00:56 smithi154 ceph-mon[36320]: pgmap v314: 97 pgs: 97 active+clean; 93 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 1.7 MiB/s wr, 215 op/s 2024-03-16T12:00:56.587 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:00:56 smithi029 ceph-mon[31975]: pgmap v314: 97 pgs: 97 active+clean; 93 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 1.7 MiB/s wr, 215 op/s 2024-03-16T12:00:56.627 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:00:56 smithi152 ceph-mon[36291]: pgmap v314: 97 pgs: 97 active+clean; 93 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 1.7 MiB/s wr, 215 op/s 2024-03-16T12:00:56.800 INFO:tasks.workunit.client.0.smithi029.stdout: 1 994849 9.81 MB/sec execute 401 sec latency 114.444 ms 2024-03-16T12:00:57.800 INFO:tasks.workunit.client.0.smithi029.stdout: 1 995753 9.79 MB/sec execute 402 sec latency 23.587 ms 2024-03-16T12:00:58.390 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:00:58 smithi029 ceph-mon[31975]: pgmap v315: 97 pgs: 97 active+clean; 98 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 3.0 MiB/s wr, 291 op/s 2024-03-16T12:00:58.545 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:00:58 smithi154 ceph-mon[36320]: pgmap v315: 97 pgs: 97 active+clean; 98 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 3.0 MiB/s wr, 291 op/s 2024-03-16T12:00:58.627 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:00:58 smithi152 ceph-mon[36291]: pgmap v315: 97 pgs: 97 active+clean; 98 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 3.0 MiB/s wr, 291 op/s 2024-03-16T12:00:58.800 INFO:tasks.workunit.client.0.smithi029.stdout: 1 997551 9.79 MB/sec execute 403 sec latency 133.672 ms 2024-03-16T12:00:59.801 INFO:tasks.workunit.client.0.smithi029.stdout: 1 998688 9.79 MB/sec execute 404 sec latency 38.852 ms 2024-03-16T12:01:00.545 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:01:00 smithi154 ceph-mon[36320]: pgmap v316: 97 pgs: 97 active+clean; 98 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 2.0 MiB/s wr, 142 op/s 2024-03-16T12:01:00.587 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:01:00 smithi029 ceph-mon[31975]: pgmap v316: 97 pgs: 97 active+clean; 98 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 2.0 MiB/s wr, 142 op/s 2024-03-16T12:01:00.627 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:01:00 smithi152 ceph-mon[36291]: pgmap v316: 97 pgs: 97 active+clean; 98 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 2.0 MiB/s wr, 142 op/s 2024-03-16T12:01:00.801 INFO:tasks.workunit.client.0.smithi029.stdout: 1 999545 9.77 MB/sec execute 405 sec latency 29.555 ms 2024-03-16T12:01:01.801 INFO:tasks.workunit.client.0.smithi029.stdout: 1 1000193 9.76 MB/sec execute 406 sec latency 23.004 ms 2024-03-16T12:01:02.451 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:01:02 smithi152 ceph-mon[36291]: pgmap v317: 97 pgs: 97 active+clean; 91 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 3.1 MiB/s wr, 309 op/s 2024-03-16T12:01:02.545 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:01:02 smithi154 ceph-mon[36320]: pgmap v317: 97 pgs: 97 active+clean; 91 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 3.1 MiB/s wr, 309 op/s 2024-03-16T12:01:02.588 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:01:02 smithi029 ceph-mon[31975]: pgmap v317: 97 pgs: 97 active+clean; 91 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 3.1 MiB/s wr, 309 op/s 2024-03-16T12:01:02.801 INFO:tasks.workunit.client.0.smithi029.stdout: 1 1001984 9.76 MB/sec execute 407 sec latency 23.348 ms 2024-03-16T12:01:03.801 INFO:tasks.workunit.client.0.smithi029.stdout: 1 1002843 9.74 MB/sec execute 408 sec latency 62.079 ms 2024-03-16T12:01:04.545 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:01:04 smithi154 ceph-mon[36320]: pgmap v318: 97 pgs: 97 active+clean; 91 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 2.5 MiB/s wr, 257 op/s 2024-03-16T12:01:04.588 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:01:04 smithi029 ceph-mon[31975]: pgmap v318: 97 pgs: 97 active+clean; 91 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 2.5 MiB/s wr, 257 op/s 2024-03-16T12:01:04.627 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:01:04 smithi152 ceph-mon[36291]: pgmap v318: 97 pgs: 97 active+clean; 91 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 2.5 MiB/s wr, 257 op/s 2024-03-16T12:01:04.801 INFO:tasks.workunit.client.0.smithi029.stdout: 1 1003690 9.73 MB/sec execute 409 sec latency 83.088 ms 2024-03-16T12:01:05.545 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:01:05 smithi154 ceph-mon[36320]: osdmap e100: 12 total, 12 up, 12 in 2024-03-16T12:01:05.588 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:01:05 smithi029 ceph-mon[31975]: osdmap e100: 12 total, 12 up, 12 in 2024-03-16T12:01:05.627 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:01:05 smithi152 ceph-mon[36291]: osdmap e100: 12 total, 12 up, 12 in 2024-03-16T12:01:05.801 INFO:tasks.workunit.client.0.smithi029.stdout: 1 1004426 9.71 MB/sec execute 410 sec latency 48.242 ms 2024-03-16T12:01:06.545 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:01:06 smithi154 ceph-mon[36320]: pgmap v320: 97 pgs: 97 active+clean; 91 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 3.0 MiB/s wr, 307 op/s 2024-03-16T12:01:06.587 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:01:06 smithi029 ceph-mon[31975]: pgmap v320: 97 pgs: 97 active+clean; 91 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 3.0 MiB/s wr, 307 op/s 2024-03-16T12:01:06.627 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:01:06 smithi152 ceph-mon[36291]: pgmap v320: 97 pgs: 97 active+clean; 91 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 3.0 MiB/s wr, 307 op/s 2024-03-16T12:01:06.801 INFO:tasks.workunit.client.0.smithi029.stdout: 1 1005494 9.71 MB/sec execute 411 sec latency 39.926 ms 2024-03-16T12:01:07.801 INFO:tasks.workunit.client.0.smithi029.stdout: 1 1006540 9.70 MB/sec execute 412 sec latency 23.194 ms 2024-03-16T12:01:08.545 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:01:08 smithi154 ceph-mon[36320]: pgmap v321: 97 pgs: 15 active+clean+snaptrim, 28 active+clean+snaptrim_wait, 54 active+clean; 76 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 2.9 MiB/s wr, 289 op/s 2024-03-16T12:01:08.588 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:01:08 smithi029 ceph-mon[31975]: pgmap v321: 97 pgs: 15 active+clean+snaptrim, 28 active+clean+snaptrim_wait, 54 active+clean; 76 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 2.9 MiB/s wr, 289 op/s 2024-03-16T12:01:08.627 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:01:08 smithi152 ceph-mon[36291]: pgmap v321: 97 pgs: 15 active+clean+snaptrim, 28 active+clean+snaptrim_wait, 54 active+clean; 76 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 2.9 MiB/s wr, 289 op/s 2024-03-16T12:01:08.801 INFO:tasks.workunit.client.0.smithi029.stdout: 1 1008337 9.70 MB/sec execute 413 sec latency 101.560 ms 2024-03-16T12:01:09.266 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:09.265+0000 7f5725b22640 1 -- 172.21.15.29:0/1374678428 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f57201017b0 msgr2=0x7f5720101c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:01:09.266 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:09.265+0000 7f5725b22640 1 --2- 172.21.15.29:0/1374678428 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f57201017b0 0x7f5720101c30 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f5714009930 tx=0x7f571402bc30 comp rx=0 tx=0).stop 2024-03-16T12:01:09.266 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:09.265+0000 7f5725b22640 1 -- 172.21.15.29:0/1374678428 shutdown_connections 2024-03-16T12:01:09.266 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:09.265+0000 7f5725b22640 1 --2- 172.21.15.29:0/1374678428 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5720102170 0x7f572010e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:01:09.266 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:09.265+0000 7f5725b22640 1 --2- 172.21.15.29:0/1374678428 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f57201017b0 0x7f5720101c30 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:01:09.266 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:09.265+0000 7f5725b22640 1 --2- 172.21.15.29:0/1374678428 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f57201077c0 0x7f5720107ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:01:09.267 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:09.265+0000 7f5725b22640 1 -- 172.21.15.29:0/1374678428 >> 172.21.15.29:0/1374678428 conn(0x7f57200fd2c0 msgr2=0x7f57200ff6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:01:09.267 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:09.265+0000 7f5725b22640 1 -- 172.21.15.29:0/1374678428 shutdown_connections 2024-03-16T12:01:09.267 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:09.265+0000 7f5725b22640 1 -- 172.21.15.29:0/1374678428 wait complete. 2024-03-16T12:01:09.267 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:09.265+0000 7f5725b22640 1 Processor -- start 2024-03-16T12:01:09.267 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:09.266+0000 7f5725b22640 1 -- start start 2024-03-16T12:01:09.267 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:09.266+0000 7f5725b22640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f57201017b0 0x7f5720198010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:01:09.267 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:09.266+0000 7f5725b22640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5720102170 0x7f5720198550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:01:09.267 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:09.266+0000 7f5725b22640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f57201077c0 0x7f572019c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:01:09.267 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:09.266+0000 7f5725b22640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f5720111bd0 con 0x7f5720102170 2024-03-16T12:01:09.267 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:09.266+0000 7f5725b22640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f5720111a50 con 0x7f57201077c0 2024-03-16T12:01:09.267 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:09.266+0000 7f5725b22640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f5720111d50 con 0x7f57201017b0 2024-03-16T12:01:09.267 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:09.266+0000 7f571effd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5720102170 0x7f5720198550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:01:09.267 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:09.266+0000 7f571f7fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f57201017b0 0x7f5720198010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:01:09.268 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:09.266+0000 7f571effd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5720102170 0x7f5720198550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:40528/0 (socket says 172.21.15.29:40528) 2024-03-16T12:01:09.268 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:09.266+0000 7f571effd640 1 -- 172.21.15.29:0/2054974882 learned_addr learned my addr 172.21.15.29:0/2054974882 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T12:01:09.268 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:09.266+0000 7f571ffff640 1 --2- 172.21.15.29:0/2054974882 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f57201077c0 0x7f572019c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:01:09.268 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:09.266+0000 7f571effd640 1 -- 172.21.15.29:0/2054974882 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f57201017b0 msgr2=0x7f5720198010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:01:09.268 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:09.266+0000 7f571effd640 1 --2- 172.21.15.29:0/2054974882 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f57201017b0 0x7f5720198010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:01:09.268 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:09.266+0000 7f571effd640 1 -- 172.21.15.29:0/2054974882 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f57201077c0 msgr2=0x7f572019c8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:01:09.268 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:09.266+0000 7f571effd640 1 --2- 172.21.15.29:0/2054974882 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f57201077c0 0x7f572019c8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:01:09.268 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:09.266+0000 7f571effd640 1 -- 172.21.15.29:0/2054974882 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f572019d000 con 0x7f5720102170 2024-03-16T12:01:09.268 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:09.266+0000 7f571f7fe640 1 --2- 172.21.15.29:0/2054974882 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f57201017b0 0x7f5720198010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:01:09.268 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:09.266+0000 7f571effd640 1 --2- 172.21.15.29:0/2054974882 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5720102170 0x7f5720198550 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f5714009230 tx=0x7f571402cb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:01:09.269 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:09.266+0000 7f571cff9640 1 -- 172.21.15.29:0/2054974882 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5714041070 con 0x7f5720102170 2024-03-16T12:01:09.269 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:09.267+0000 7f571cff9640 1 -- 172.21.15.29:0/2054974882 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5714004450 con 0x7f5720102170 2024-03-16T12:01:09.269 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:09.267+0000 7f5725b22640 1 -- 172.21.15.29:0/2054974882 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f572019d290 con 0x7f5720102170 2024-03-16T12:01:09.269 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:09.267+0000 7f571cff9640 1 -- 172.21.15.29:0/2054974882 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5714003e50 con 0x7f5720102170 2024-03-16T12:01:09.269 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:09.267+0000 7f5725b22640 1 -- 172.21.15.29:0/2054974882 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f57201a4b80 con 0x7f5720102170 2024-03-16T12:01:09.277 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:09.269+0000 7f571cff9640 1 -- 172.21.15.29:0/2054974882 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7f5714044070 con 0x7f5720102170 2024-03-16T12:01:09.277 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:09.269+0000 7f57287eb740 1 Processor -- start 2024-03-16T12:01:09.277 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:09.269+0000 7f57287eb740 1 -- start start 2024-03-16T12:01:09.277 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:09.269+0000 7f57287eb740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557bd2700130 0x557bd2700510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:01:09.277 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:09.269+0000 7f57287eb740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557bd2700a50 0x557bd27092e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:01:09.277 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:09.269+0000 7f57287eb740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557bd2709820 0x557bd270bc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:01:09.277 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:09.270+0000 7f571cff9640 1 --2- 172.21.15.29:0/2054974882 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f57080a2bb0 0x7f57080a5070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:01:09.277 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:09.270+0000 7f57287eb740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x557bd2664a40 con 0x557bd2709820 2024-03-16T12:01:09.278 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:09.270+0000 7f57287eb740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x557bd260d920 con 0x557bd2700a50 2024-03-16T12:01:09.278 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:09.270+0000 7f57287eb740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x557bd2660ae0 con 0x557bd2700130 2024-03-16T12:01:09.278 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:09.270+0000 7f571effd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557bd2709820 0x557bd270bc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:01:09.278 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:09.270+0000 7f571f7fe640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557bd2700a50 0x557bd27092e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:01:09.278 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:09.270+0000 7f571ffff640 1 --2- 172.21.15.29:0/2054974882 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f57080a2bb0 0x7f57080a5070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:01:09.278 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:09.270+0000 7f571f7fe640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557bd2700a50 0x557bd27092e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.29:56396/0 (socket says 172.21.15.29:56396) 2024-03-16T12:01:09.278 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:09.270+0000 7f571f7fe640 1 -- 172.21.15.29:0/4002216285 learned_addr learned my addr 172.21.15.29:0/4002216285 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T12:01:09.278 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:09.270+0000 7f571ffff640 1 --2- 172.21.15.29:0/4002216285 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557bd2700130 0x557bd2700510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:01:09.278 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:09.270+0000 7f571cff9640 1 -- 172.21.15.29:0/2054974882 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 8600+0+0 (secure 0 0 0) 0x7f57140f06d0 con 0x7f5720102170 2024-03-16T12:01:09.278 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:09.270+0000 7f571f7fe640 1 -- 172.21.15.29:0/4002216285 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557bd2700130 msgr2=0x557bd2700510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:01:09.278 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:09.270+0000 7f571f7fe640 1 --2- 172.21.15.29:0/4002216285 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557bd2700130 0x557bd2700510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:01:09.279 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:09.270+0000 7f571f7fe640 1 -- 172.21.15.29:0/4002216285 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557bd2709820 msgr2=0x557bd270bc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:01:09.279 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:09.270+0000 7f571f7fe640 1 --2- 172.21.15.29:0/4002216285 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557bd2709820 0x557bd270bc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:01:09.279 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:09.270+0000 7f571f7fe640 1 -- 172.21.15.29:0/4002216285 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557bd2625460 con 0x557bd2700a50 2024-03-16T12:01:09.279 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:09.270+0000 7f571effd640 1 --2- 172.21.15.29:0/4002216285 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557bd2709820 0x557bd270bc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:01:09.279 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:09.270+0000 7f571ffff640 1 --2- 172.21.15.29:0/2054974882 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f57080a2bb0 0x7f57080a5070 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f5718005fd0 tx=0x7f5718009f00 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:01:09.279 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:09.270+0000 7f571f7fe640 1 --2- 172.21.15.29:0/4002216285 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557bd2700a50 0x557bd27092e0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f57100114f0 tx=0x7f57100119b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=51cc78fb47064e0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:01:09.279 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:09.270+0000 7f5705ffb640 1 -- 172.21.15.29:0/4002216285 <== mon.2 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f571001e070 con 0x557bd2700a50 2024-03-16T12:01:09.279 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:09.271+0000 7f5705ffb640 1 -- 172.21.15.29:0/4002216285 <== mon.2 v2:172.21.15.152:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f571000faa0 con 0x557bd2700a50 2024-03-16T12:01:09.279 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:09.271+0000 7f5705ffb640 1 -- 172.21.15.29:0/4002216285 <== mon.2 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5710019bb0 con 0x557bd2700a50 2024-03-16T12:01:09.279 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:09.271+0000 7f57287eb740 1 -- 172.21.15.29:0/4002216285 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557bd2700a50 msgr2=0x557bd27092e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:01:09.279 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:09.271+0000 7f57287eb740 1 --2- 172.21.15.29:0/4002216285 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557bd2700a50 0x557bd27092e0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f57100114f0 tx=0x7f57100119b0 comp rx=0 tx=0).stop 2024-03-16T12:01:09.279 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:09.271+0000 7f57287eb740 1 -- 172.21.15.29:0/4002216285 shutdown_connections 2024-03-16T12:01:09.279 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:09.271+0000 7f57287eb740 1 --2- 172.21.15.29:0/4002216285 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557bd2709820 0x557bd270bc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:01:09.279 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:09.271+0000 7f57287eb740 1 --2- 172.21.15.29:0/4002216285 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557bd2700a50 0x557bd27092e0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:01:09.280 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:09.271+0000 7f57287eb740 1 --2- 172.21.15.29:0/4002216285 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557bd2700130 0x557bd2700510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:01:09.280 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:09.271+0000 7f57287eb740 1 -- 172.21.15.29:0/4002216285 >> 172.21.15.29:0/4002216285 conn(0x557bd26faa50 msgr2=0x557bd26fae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:01:09.280 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:09.271+0000 7f57287eb740 1 -- 172.21.15.29:0/4002216285 shutdown_connections 2024-03-16T12:01:09.280 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:09.271+0000 7f57287eb740 1 -- 172.21.15.29:0/4002216285 wait complete. 2024-03-16T12:01:09.280 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:09.272+0000 7f57287eb740 1 -- start start 2024-03-16T12:01:09.280 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:09.272+0000 7f57287eb740 1 Processor -- start 2024-03-16T12:01:09.280 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:09.272+0000 7f57287eb740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557bd2700130 0x557bd27067f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:01:09.280 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:09.272+0000 7f57287eb740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557bd2700a50 0x557bd2706d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:01:09.280 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:09.272+0000 7f57287eb740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557bd2709820 0x557bd2707270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:01:09.280 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:09.272+0000 7f57287eb740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x557bd270c450 con 0x557bd2709820 2024-03-16T12:01:09.280 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:09.272+0000 7f57287eb740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x557bd2620d70 con 0x557bd2700130 2024-03-16T12:01:09.280 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:09.272+0000 7f57287eb740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x557bd270c5d0 con 0x557bd2700a50 2024-03-16T12:01:09.280 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:09.272+0000 7f57287eb740 10 client.0 _ll_register_callbacks cb 0x557bd2378a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T12:01:09.281 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:09.272+0000 7f571ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557bd2709820 0x557bd2707270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:01:09.281 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:09.272+0000 7f571f7fe640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557bd2700130 0x557bd27067f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:01:09.281 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:09.272+0000 7f571ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557bd2709820 0x557bd2707270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:40554/0 (socket says 172.21.15.29:40554) 2024-03-16T12:01:09.281 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:09.272+0000 7f571ffff640 1 -- 172.21.15.29:0/499068055 learned_addr learned my addr 172.21.15.29:0/499068055 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T12:01:09.281 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:09.272+0000 7f571effd640 1 --2- 172.21.15.29:0/499068055 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557bd2700a50 0x557bd2706d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:01:09.281 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:09.272+0000 7f571ffff640 1 -- 172.21.15.29:0/499068055 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557bd2700a50 msgr2=0x557bd2706d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:01:09.281 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:09.272+0000 7f571ffff640 1 --2- 172.21.15.29:0/499068055 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557bd2700a50 0x557bd2706d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:01:09.281 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:09.272+0000 7f571ffff640 1 -- 172.21.15.29:0/499068055 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557bd2700130 msgr2=0x557bd27067f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:01:09.281 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:09.272+0000 7f571ffff640 1 --2- 172.21.15.29:0/499068055 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557bd2700130 0x557bd27067f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:01:09.281 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:09.272+0000 7f571ffff640 1 -- 172.21.15.29:0/499068055 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557bd2379210 con 0x557bd2709820 2024-03-16T12:01:09.281 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:09.273+0000 7f571ffff640 1 --2- 172.21.15.29:0/499068055 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557bd2709820 0x557bd2707270 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f5718003b90 tx=0x7f571801d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a233d81bea953e04 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:01:09.281 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:09.273+0000 7f56feffd640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-16T12:01:09.281 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:09.273+0000 7f56feffd640 1 -- 172.21.15.29:0/499068055 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5718024070 con 0x557bd2709820 2024-03-16T12:01:09.282 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:09.273+0000 7f56feffd640 1 -- 172.21.15.29:0/499068055 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 8600+0+0 (secure 0 0 0) 0x7f571801f450 con 0x557bd2709820 2024-03-16T12:01:09.282 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:09.273+0000 7f57287eb740 1 -- 172.21.15.29:0/499068055 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f571000faa0 con 0x557bd2709820 2024-03-16T12:01:09.282 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:09.273+0000 7f56feffd640 1 -- 172.21.15.29:0/499068055 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f571801f650 con 0x557bd2709820 2024-03-16T12:01:09.282 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:09.273+0000 7f57287eb740 10 client.16248 fetch_fsmap learned FSMap version 26 2024-03-16T12:01:09.282 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:09.273+0000 7f57287eb740 1 -- 172.21.15.29:0/499068055 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x557bd2445620 con 0x557bd2709820 2024-03-16T12:01:09.282 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:09.273+0000 7f56feffd640 1 -- 172.21.15.29:0/499068055 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f571000faa0 con 0x557bd2709820 2024-03-16T12:01:09.282 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:09.273+0000 7f56feffd640 1 -- 172.21.15.29:0/499068055 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f5718028650 con 0x557bd2709820 2024-03-16T12:01:09.282 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:09.273+0000 7f57287eb740 10 client.16248 fetch_fsmap finished waiting for FSMap version 26 2024-03-16T12:01:09.282 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:09.273+0000 7f57287eb740 10 client.16248 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T12:01:09.282 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:09.273+0000 7f57287eb740 20 client.16248 populate_metadata read hostname 'smithi029' 2024-03-16T12:01:09.282 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:09.273+0000 7f57287eb740 1 --2- 172.21.15.29:0/499068055 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x557bd2719990 0x557bd271bda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:01:09.282 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:09.273+0000 7f57287eb740 4 client.16248 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T12:01:09.282 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:09.273+0000 7f57287eb740 1 -- 172.21.15.29:0/499068055 --> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557bd2671f80 con 0x557bd2719990 2024-03-16T12:01:09.283 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:09.274+0000 7f571f7fe640 1 --2- 172.21.15.29:0/499068055 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x557bd2719990 0x557bd271bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:01:09.283 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:09.274+0000 7f56feffd640 10 client.16248 ms_handle_connect on v2:172.21.15.152:6838/1548731833 2024-03-16T12:01:09.283 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:09.274+0000 7f571f7fe640 1 --2- 172.21.15.29:0/499068055 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x557bd2719990 0x557bd271bda0 crc :-1 s=READY pgs=130 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=aeeb1a385e633e72 server_cookie=d3813b3144a8da40 in_seq=0 out_seq=0 2024-03-16T12:01:09.283 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:09.275+0000 7f56feffd640 1 -- 172.21.15.29:0/499068055 <== mds.0 v2:172.21.15.152:6838/1548731833 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x557bd2671f80 con 0x557bd2719990 2024-03-16T12:01:09.283 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:09.275+0000 7f56feffd640 10 client.16248 handle_command_reply: tid=0 2024-03-16T12:01:09.283 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:09.275+0000 7f57287eb740 1 client.16248 shutdown 2024-03-16T12:01:09.283 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:09.275+0000 7f57287eb740 1 -- 172.21.15.29:0/499068055 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557bd2709820 msgr2=0x557bd2707270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:01:09.283 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:09.275+0000 7f57287eb740 1 --2- 172.21.15.29:0/499068055 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557bd2709820 0x557bd2707270 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f5718003b90 tx=0x7f571801d000 comp rx=0 tx=0).stop 2024-03-16T12:01:09.283 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:09.275+0000 7f57287eb740 1 -- 172.21.15.29:0/499068055 shutdown_connections 2024-03-16T12:01:09.283 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:09.275+0000 7f57287eb740 1 --2- 172.21.15.29:0/499068055 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557bd2709820 0x557bd2707270 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:01:09.283 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:09.275+0000 7f57287eb740 1 --2- 172.21.15.29:0/499068055 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557bd2700a50 0x557bd2706d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:01:09.283 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:09.275+0000 7f57287eb740 1 --2- 172.21.15.29:0/499068055 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x557bd2719990 0x557bd271bda0 crc :-1 s=READY pgs=130 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:01:09.283 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:09.275+0000 7f57287eb740 1 --2- 172.21.15.29:0/499068055 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557bd2700130 0x557bd27067f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:01:09.284 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:09.275+0000 7f56feffd640 0 client.16248 ms_handle_reset on v2:172.21.15.152:6838/1548731833 2024-03-16T12:01:09.284 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:09.275+0000 7f57287eb740 1 -- 172.21.15.29:0/499068055 >> 172.21.15.29:0/499068055 conn(0x557bd26faa50 msgr2=0x557bd26fc7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:01:09.284 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:09.275+0000 7f57287eb740 1 -- 172.21.15.29:0/499068055 shutdown_connections 2024-03-16T12:01:09.284 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:09.275+0000 7f57287eb740 1 -- 172.21.15.29:0/499068055 wait complete. 2024-03-16T12:01:09.284 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:09.275+0000 7f57287eb740 20 client.16248 trim_cache size 0 max 16384 2024-03-16T12:01:09.295 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:09.290+0000 7f57287eb740 1 Processor -- start 2024-03-16T12:01:09.295 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:09.290+0000 7f57287eb740 1 -- start start 2024-03-16T12:01:09.295 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:09.291+0000 7f57287eb740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557bd2700130 0x557bd2708f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:01:09.295 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:09.291+0000 7f57287eb740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557bd2700a50 0x557bd27151f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:01:09.295 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:09.291+0000 7f57287eb740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557bd2709820 0x557bd26f5a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:01:09.295 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:09.291+0000 7f57287eb740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f57201afe10 con 0x557bd2700130 2024-03-16T12:01:09.295 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:09.291+0000 7f57287eb740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f57201ad6e0 con 0x557bd2709820 2024-03-16T12:01:09.295 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:09.291+0000 7f57287eb740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f57201ab550 con 0x557bd2700a50 2024-03-16T12:01:09.295 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:09.291+0000 7f571f7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557bd2700130 0x557bd2708f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:01:09.295 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:09.291+0000 7f571f7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557bd2700130 0x557bd2708f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:40568/0 (socket says 172.21.15.29:40568) 2024-03-16T12:01:09.295 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:09.291+0000 7f571f7fe640 1 -- 172.21.15.29:0/3736445978 learned_addr learned my addr 172.21.15.29:0/3736445978 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T12:01:09.296 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:09.291+0000 7f571effd640 1 --2- 172.21.15.29:0/3736445978 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557bd2700a50 0x557bd27151f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:01:09.296 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:09.291+0000 7f571ffff640 1 --2- 172.21.15.29:0/3736445978 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557bd2709820 0x557bd26f5a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:01:09.296 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:09.291+0000 7f571f7fe640 1 -- 172.21.15.29:0/3736445978 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557bd2700a50 msgr2=0x557bd27151f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:01:09.296 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:09.291+0000 7f571f7fe640 1 --2- 172.21.15.29:0/3736445978 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557bd2700a50 0x557bd27151f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:01:09.296 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:09.291+0000 7f571f7fe640 1 -- 172.21.15.29:0/3736445978 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557bd2709820 msgr2=0x557bd26f5a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:01:09.296 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:09.291+0000 7f571f7fe640 1 --2- 172.21.15.29:0/3736445978 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557bd2709820 0x557bd26f5a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:01:09.296 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:09.291+0000 7f571f7fe640 1 -- 172.21.15.29:0/3736445978 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557bd27057a0 con 0x557bd2700130 2024-03-16T12:01:09.296 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:09.291+0000 7f571ffff640 1 --2- 172.21.15.29:0/3736445978 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557bd2709820 0x557bd26f5a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T12:01:09.296 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:09.291+0000 7f571f7fe640 1 --2- 172.21.15.29:0/3736445978 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557bd2700130 0x557bd2708f20 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f57100114f0 tx=0x7f571000ff60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ae08bad2c649310 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:01:09.296 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:09.291+0000 7f57057fa640 1 -- 172.21.15.29:0/3736445978 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f571002f070 con 0x557bd2700130 2024-03-16T12:01:09.296 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:09.291+0000 7f57057fa640 1 -- 172.21.15.29:0/3736445978 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f571000faa0 con 0x557bd2700130 2024-03-16T12:01:09.296 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:09.291+0000 7f57057fa640 1 -- 172.21.15.29:0/3736445978 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f571001e040 con 0x557bd2700130 2024-03-16T12:01:09.296 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:09.292+0000 7f57287eb740 1 -- 172.21.15.29:0/3736445978 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557bd2700130 msgr2=0x557bd2708f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:01:09.296 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:09.292+0000 7f57287eb740 1 --2- 172.21.15.29:0/3736445978 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557bd2700130 0x557bd2708f20 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f57100114f0 tx=0x7f571000ff60 comp rx=0 tx=0).stop 2024-03-16T12:01:09.297 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:09.292+0000 7f57287eb740 1 -- 172.21.15.29:0/3736445978 shutdown_connections 2024-03-16T12:01:09.297 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:09.292+0000 7f57287eb740 1 --2- 172.21.15.29:0/3736445978 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557bd2709820 0x557bd26f5a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:01:09.297 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:09.292+0000 7f57287eb740 1 --2- 172.21.15.29:0/3736445978 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557bd2700a50 0x557bd27151f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:01:09.297 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:09.292+0000 7f57287eb740 1 --2- 172.21.15.29:0/3736445978 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557bd2700130 0x557bd2708f20 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:01:09.297 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:09.292+0000 7f57287eb740 1 -- 172.21.15.29:0/3736445978 >> 172.21.15.29:0/3736445978 conn(0x557bd26faa50 msgr2=0x557bd2707040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:01:09.297 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:09.292+0000 7f57287eb740 1 -- 172.21.15.29:0/3736445978 shutdown_connections 2024-03-16T12:01:09.297 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:09.292+0000 7f57287eb740 1 -- 172.21.15.29:0/3736445978 wait complete. 2024-03-16T12:01:09.297 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:09.292+0000 7f57287eb740 1 -- start start 2024-03-16T12:01:09.297 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:09.292+0000 7f57287eb740 1 Processor -- start 2024-03-16T12:01:09.297 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:09.293+0000 7f57287eb740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557bd2700130 0x557bd270d1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:01:09.297 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:09.293+0000 7f57287eb740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557bd2700a50 0x557bd270d710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:01:09.297 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:09.293+0000 7f57287eb740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557bd2709820 0x557bd270dc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:01:09.297 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:09.293+0000 7f57287eb740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f57201abba0 con 0x557bd2700a50 2024-03-16T12:01:09.297 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:09.293+0000 7f57287eb740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f57201ad010 con 0x557bd2709820 2024-03-16T12:01:09.297 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:09.293+0000 7f57287eb740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f57201ac270 con 0x557bd2700130 2024-03-16T12:01:09.298 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:09.293+0000 7f57287eb740 10 client.0 _ll_register_callbacks cb 0x7f5718009aa0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T12:01:09.298 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:09.293+0000 7f571effd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557bd2700a50 0x557bd270d710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:01:09.298 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:09.293+0000 7f571f7fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557bd2700130 0x557bd270d1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:01:09.298 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:09.293+0000 7f571effd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557bd2700a50 0x557bd270d710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:40582/0 (socket says 172.21.15.29:40582) 2024-03-16T12:01:09.298 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:09.293+0000 7f571effd640 1 -- 172.21.15.29:0/379531055 learned_addr learned my addr 172.21.15.29:0/379531055 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T12:01:09.298 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:09.293+0000 7f571ffff640 1 --2- 172.21.15.29:0/379531055 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557bd2709820 0x557bd270dc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:01:09.298 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:09.293+0000 7f571effd640 1 -- 172.21.15.29:0/379531055 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557bd2700130 msgr2=0x557bd270d1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:01:09.298 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:09.293+0000 7f571effd640 1 --2- 172.21.15.29:0/379531055 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557bd2700130 0x557bd270d1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:01:09.298 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:09.293+0000 7f571effd640 1 -- 172.21.15.29:0/379531055 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557bd2709820 msgr2=0x557bd270dc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:01:09.298 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:09.293+0000 7f571effd640 1 --2- 172.21.15.29:0/379531055 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557bd2709820 0x557bd270dc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:01:09.298 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:09.293+0000 7f571effd640 1 -- 172.21.15.29:0/379531055 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557bd277ed90 con 0x557bd2700a50 2024-03-16T12:01:09.298 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:09.293+0000 7f571f7fe640 1 --2- 172.21.15.29:0/379531055 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557bd2700130 0x557bd270d1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:01:09.298 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:09.293+0000 7f571effd640 1 --2- 172.21.15.29:0/379531055 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557bd2700a50 0x557bd270d710 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f57140b9810 tx=0x7f5714003920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae7c4e8c8c235a48 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:01:09.299 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:09.293+0000 7f56feffd640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-16T12:01:09.299 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:09.293+0000 7f56feffd640 1 -- 172.21.15.29:0/379531055 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57140d2020 con 0x557bd2700a50 2024-03-16T12:01:09.299 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:09.294+0000 7f56feffd640 1 -- 172.21.15.29:0/379531055 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 8600+0+0 (secure 0 0 0) 0x7f57140edff0 con 0x557bd2700a50 2024-03-16T12:01:09.299 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:09.294+0000 7f57287eb740 1 -- 172.21.15.29:0/379531055 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f571000faa0 con 0x557bd2700a50 2024-03-16T12:01:09.299 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:09.294+0000 7f56feffd640 1 -- 172.21.15.29:0/379531055 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f57140da050 con 0x557bd2700a50 2024-03-16T12:01:09.299 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:09.294+0000 7f57287eb740 10 client.16254 fetch_fsmap learned FSMap version 26 2024-03-16T12:01:09.299 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:09.294+0000 7f57287eb740 1 -- 172.21.15.29:0/379531055 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x557bd27800e0 con 0x557bd2700a50 2024-03-16T12:01:09.299 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:09.294+0000 7f56feffd640 1 -- 172.21.15.29:0/379531055 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f571000faa0 con 0x557bd2700a50 2024-03-16T12:01:09.299 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:09.294+0000 7f56feffd640 1 -- 172.21.15.29:0/379531055 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f57140864f0 con 0x557bd2700a50 2024-03-16T12:01:09.299 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:09.294+0000 7f57287eb740 10 client.16254 fetch_fsmap finished waiting for FSMap version 26 2024-03-16T12:01:09.299 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:09.294+0000 7f57287eb740 10 client.16254 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T12:01:09.299 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:09.294+0000 7f57287eb740 20 client.16254 populate_metadata read hostname 'smithi029' 2024-03-16T12:01:09.299 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:09.294+0000 7f57287eb740 1 --2- 172.21.15.29:0/379531055 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x557bd2780860 0x557bd2782c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:01:09.299 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:09.294+0000 7f57287eb740 4 client.16254 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-16T12:01:09.299 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:09.294+0000 7f57287eb740 1 -- 172.21.15.29:0/379531055 --> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] -- command(tid 0: {"prefix": "flush journal"}) -- 0x557bd266c2d0 con 0x557bd2780860 2024-03-16T12:01:09.300 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:09.294+0000 7f571f7fe640 1 --2- 172.21.15.29:0/379531055 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x557bd2780860 0x557bd2782c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:01:09.300 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:09.295+0000 7f571f7fe640 1 --2- 172.21.15.29:0/379531055 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x557bd2780860 0x557bd2782c80 crc :-1 s=READY pgs=131 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a42cd245e4910e2e server_cookie=c019bbf532a2095b in_seq=0 out_seq=0 2024-03-16T12:01:09.300 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:09.295+0000 7f56feffd640 10 client.16254 ms_handle_connect on v2:172.21.15.152:6838/1548731833 2024-03-16T12:01:09.459 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:09.459+0000 7f56feffd640 1 -- 172.21.15.29:0/379531055 <== mds.0 v2:172.21.15.152:6838/1548731833 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x557bd266c2d0 con 0x557bd2780860 2024-03-16T12:01:09.459 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:09.459+0000 7f56feffd640 10 client.16254 handle_command_reply: tid=0 2024-03-16T12:01:09.459 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:09.459+0000 7f57287eb740 1 client.16254 shutdown 2024-03-16T12:01:09.460 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:09.460+0000 7f57287eb740 1 -- 172.21.15.29:0/379531055 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557bd2700a50 msgr2=0x557bd270d710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:01:09.460 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:09.460+0000 7f57287eb740 1 --2- 172.21.15.29:0/379531055 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557bd2700a50 0x557bd270d710 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f57140b9810 tx=0x7f5714003920 comp rx=0 tx=0).stop 2024-03-16T12:01:09.460 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:09.460+0000 7f57287eb740 1 -- 172.21.15.29:0/379531055 shutdown_connections 2024-03-16T12:01:09.460 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:09.460+0000 7f57287eb740 1 --2- 172.21.15.29:0/379531055 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x557bd2780860 0x557bd2782c80 crc :-1 s=READY pgs=131 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:01:09.460 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:09.460+0000 7f57287eb740 1 --2- 172.21.15.29:0/379531055 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557bd2709820 0x557bd270dc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:01:09.460 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:09.460+0000 7f57287eb740 1 --2- 172.21.15.29:0/379531055 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557bd2700a50 0x557bd270d710 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:01:09.460 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:09.460+0000 7f57287eb740 1 --2- 172.21.15.29:0/379531055 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557bd2700130 0x557bd270d1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:01:09.460 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:09.460+0000 7f56feffd640 0 client.16254 ms_handle_reset on v2:172.21.15.152:6838/1548731833 2024-03-16T12:01:09.460 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:09.460+0000 7f57287eb740 1 -- 172.21.15.29:0/379531055 >> 172.21.15.29:0/379531055 conn(0x557bd26faa50 msgr2=0x557bd26f9320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:01:09.460 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:09.460+0000 7f57287eb740 1 -- 172.21.15.29:0/379531055 shutdown_connections 2024-03-16T12:01:09.460 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:09.460+0000 7f57287eb740 1 -- 172.21.15.29:0/379531055 wait complete. 2024-03-16T12:01:09.460 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:09.460+0000 7f57287eb740 20 client.16254 trim_cache size 0 max 16384 2024-03-16T12:01:09.461 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:{ 2024-03-16T12:01:09.461 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "message": "", 2024-03-16T12:01:09.461 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "return_code": 0 2024-03-16T12:01:09.461 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:} 2024-03-16T12:01:09.462 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:09.460+0000 7f5725b22640 1 -- 172.21.15.29:0/2054974882 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f57080a2bb0 msgr2=0x7f57080a5070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:01:09.462 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:09.460+0000 7f5725b22640 1 --2- 172.21.15.29:0/2054974882 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f57080a2bb0 0x7f57080a5070 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f5718005fd0 tx=0x7f5718009f00 comp rx=0 tx=0).stop 2024-03-16T12:01:09.462 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:09.460+0000 7f5725b22640 1 -- 172.21.15.29:0/2054974882 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5720102170 msgr2=0x7f5720198550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:01:09.462 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:09.460+0000 7f5725b22640 1 --2- 172.21.15.29:0/2054974882 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5720102170 0x7f5720198550 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f5714009230 tx=0x7f571402cb30 comp rx=0 tx=0).stop 2024-03-16T12:01:09.462 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:09.460+0000 7f5725b22640 1 -- 172.21.15.29:0/2054974882 shutdown_connections 2024-03-16T12:01:09.462 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:09.460+0000 7f5725b22640 1 --2- 172.21.15.29:0/2054974882 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f57080a2bb0 0x7f57080a5070 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:01:09.462 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:09.460+0000 7f5725b22640 1 --2- 172.21.15.29:0/2054974882 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f57201077c0 0x7f572019c8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:01:09.462 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:09.460+0000 7f5725b22640 1 --2- 172.21.15.29:0/2054974882 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5720102170 0x7f5720198550 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:01:09.462 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:09.460+0000 7f5725b22640 1 --2- 172.21.15.29:0/2054974882 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f57201017b0 0x7f5720198010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:01:09.462 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:09.460+0000 7f5725b22640 1 -- 172.21.15.29:0/2054974882 >> 172.21.15.29:0/2054974882 conn(0x7f57200fd2c0 msgr2=0x7f57200fd840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:01:09.462 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:09.461+0000 7f5725b22640 1 -- 172.21.15.29:0/2054974882 shutdown_connections 2024-03-16T12:01:09.462 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:09.461+0000 7f5725b22640 1 -- 172.21.15.29:0/2054974882 wait complete. 2024-03-16T12:01:09.801 INFO:tasks.workunit.client.0.smithi029.stdout: 1 1009266 9.69 MB/sec execute 414 sec latency 102.373 ms 2024-03-16T12:01:10.045 INFO:journalctl@ceph.mds.c.smithi154.stdout:Mar 16 12:01:09 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mds-c[66727]: ignoring --setuser ceph since I am not root 2024-03-16T12:01:10.045 INFO:journalctl@ceph.mds.c.smithi154.stdout:Mar 16 12:01:09 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mds-c[66727]: ignoring --setgroup ceph since I am not root 2024-03-16T12:01:10.045 INFO:journalctl@ceph.mds.c.smithi154.stdout:Mar 16 12:01:09 smithi154 ceph-mds[66731]: -- 172.21.15.154:0/12354084 <== mon.1 v2:172.21.15.154:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x561c89fdda40 con 0x561c89fb0800 2024-03-16T12:01:10.045 INFO:journalctl@ceph.mds.c.smithi154.stdout:Mar 16 12:01:09 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mds-c[66727]: starting mds.c at 2024-03-16T12:01:10.545 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:01:10 smithi154 ceph-mon[36320]: pgmap v322: 97 pgs: 15 active+clean+snaptrim, 28 active+clean+snaptrim_wait, 54 active+clean; 76 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 2.9 MiB/s wr, 289 op/s 2024-03-16T12:01:10.545 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:01:10 smithi154 ceph-mon[36320]: daemon mds.c restarted 2024-03-16T12:01:10.545 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:01:10 smithi154 ceph-mon[36320]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T12:01:10.588 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:01:10 smithi029 ceph-mon[31975]: pgmap v322: 97 pgs: 15 active+clean+snaptrim, 28 active+clean+snaptrim_wait, 54 active+clean; 76 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 2.9 MiB/s wr, 289 op/s 2024-03-16T12:01:10.588 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:01:10 smithi029 ceph-mon[31975]: daemon mds.c restarted 2024-03-16T12:01:10.588 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:01:10 smithi029 ceph-mon[31975]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T12:01:10.627 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:01:10 smithi152 ceph-mon[36291]: pgmap v322: 97 pgs: 15 active+clean+snaptrim, 28 active+clean+snaptrim_wait, 54 active+clean; 76 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 2.9 MiB/s wr, 289 op/s 2024-03-16T12:01:10.627 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:01:10 smithi152 ceph-mon[36291]: daemon mds.c restarted 2024-03-16T12:01:10.627 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:01:10 smithi152 ceph-mon[36291]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T12:01:10.801 INFO:tasks.workunit.client.0.smithi029.stdout: 1 1010595 9.68 MB/sec execute 415 sec latency 29.016 ms 2024-03-16T12:01:11.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:01:11 smithi152 ceph-mon[36291]: osdmap e101: 12 total, 12 up, 12 in 2024-03-16T12:01:11.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:01:11 smithi152 ceph-mon[36291]: mds.? [v2:172.21.15.154:6836/2335630227,v1:172.21.15.154:6837/2335630227] up:boot 2024-03-16T12:01:11.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:01:11 smithi152 ceph-mon[36291]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 2 up:standby-replay 7 up:standby 2024-03-16T12:01:11.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:01:11 smithi152 ceph-mon[36291]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-16T12:01:11.378 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:01:11 smithi152 ceph-mon[36291]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-03-16T12:01:11.545 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:01:11 smithi154 ceph-mon[36320]: osdmap e101: 12 total, 12 up, 12 in 2024-03-16T12:01:11.545 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:01:11 smithi154 ceph-mon[36320]: mds.? [v2:172.21.15.154:6836/2335630227,v1:172.21.15.154:6837/2335630227] up:boot 2024-03-16T12:01:11.545 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:01:11 smithi154 ceph-mon[36320]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 2 up:standby-replay 7 up:standby 2024-03-16T12:01:11.545 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:01:11 smithi154 ceph-mon[36320]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-16T12:01:11.545 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:01:11 smithi154 ceph-mon[36320]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-03-16T12:01:11.588 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:01:11 smithi029 ceph-mon[31975]: osdmap e101: 12 total, 12 up, 12 in 2024-03-16T12:01:11.588 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:01:11 smithi029 ceph-mon[31975]: mds.? [v2:172.21.15.154:6836/2335630227,v1:172.21.15.154:6837/2335630227] up:boot 2024-03-16T12:01:11.588 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:01:11 smithi029 ceph-mon[31975]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 2 up:standby-replay 7 up:standby 2024-03-16T12:01:11.588 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:01:11 smithi029 ceph-mon[31975]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-16T12:01:11.588 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:01:11 smithi029 ceph-mon[31975]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-03-16T12:01:11.801 INFO:tasks.workunit.client.0.smithi029.stdout: 1 1012575 9.69 MB/sec execute 416 sec latency 21.361 ms 2024-03-16T12:01:12.801 INFO:tasks.workunit.client.0.smithi029.stdout: 1 1013171 9.67 MB/sec execute 417 sec latency 203.351 ms 2024-03-16T12:01:13.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:01:12 smithi152 ceph-mon[36291]: pgmap v324: 97 pgs: 2 active+clean+snaptrim, 4 active+clean+snaptrim_wait, 91 active+clean; 63 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.3 MiB/s wr, 265 op/s 2024-03-16T12:01:13.128 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:01:12 smithi152 ceph-mon[36291]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' 2024-03-16T12:01:13.128 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:01:12 smithi152 ceph-mon[36291]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' 2024-03-16T12:01:13.128 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:01:12 smithi152 ceph-mon[36291]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' 2024-03-16T12:01:13.128 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:01:12 smithi152 ceph-mon[36291]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' 2024-03-16T12:01:13.128 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:01:12 smithi152 ceph-mon[36291]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' 2024-03-16T12:01:13.128 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:01:12 smithi152 ceph-mon[36291]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' 2024-03-16T12:01:13.128 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:01:12 smithi152 ceph-mon[36291]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-16T12:01:13.128 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:01:12 smithi152 ceph-mon[36291]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-16T12:01:13.128 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:01:12 smithi152 ceph-mon[36291]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-16T12:01:13.128 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:01:12 smithi152 ceph-mon[36291]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-16T12:01:13.128 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:01:12 smithi152 ceph-mon[36291]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-16T12:01:13.128 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:01:12 smithi152 ceph-mon[36291]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-16T12:01:13.129 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:01:12 smithi152 ceph-mon[36291]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-16T12:01:13.129 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:01:12 smithi152 ceph-mon[36291]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-16T12:01:13.129 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:01:12 smithi152 ceph-mon[36291]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-16T12:01:13.129 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:01:12 smithi152 ceph-mon[36291]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-16T12:01:13.129 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:01:12 smithi152 ceph-mon[36291]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-16T12:01:13.129 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:01:12 smithi152 ceph-mon[36291]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-16T12:01:13.129 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:01:12 smithi152 ceph-mon[36291]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:01:13.129 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:01:12 smithi152 ceph-mon[36291]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T12:01:13.129 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:01:12 smithi152 ceph-mon[36291]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' 2024-03-16T12:01:13.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:01:12 smithi154 ceph-mon[36320]: pgmap v324: 97 pgs: 2 active+clean+snaptrim, 4 active+clean+snaptrim_wait, 91 active+clean; 63 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.3 MiB/s wr, 265 op/s 2024-03-16T12:01:13.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:01:12 smithi154 ceph-mon[36320]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' 2024-03-16T12:01:13.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:01:12 smithi154 ceph-mon[36320]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' 2024-03-16T12:01:13.296 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:01:12 smithi154 ceph-mon[36320]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' 2024-03-16T12:01:13.296 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:01:12 smithi154 ceph-mon[36320]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' 2024-03-16T12:01:13.296 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:01:12 smithi154 ceph-mon[36320]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' 2024-03-16T12:01:13.296 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:01:12 smithi154 ceph-mon[36320]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' 2024-03-16T12:01:13.296 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:01:12 smithi154 ceph-mon[36320]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-16T12:01:13.296 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:01:12 smithi154 ceph-mon[36320]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-16T12:01:13.296 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:01:12 smithi154 ceph-mon[36320]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-16T12:01:13.296 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:01:12 smithi154 ceph-mon[36320]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-16T12:01:13.296 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:01:12 smithi154 ceph-mon[36320]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-16T12:01:13.296 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:01:12 smithi154 ceph-mon[36320]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-16T12:01:13.296 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:01:12 smithi154 ceph-mon[36320]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-16T12:01:13.296 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:01:12 smithi154 ceph-mon[36320]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-16T12:01:13.296 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:01:12 smithi154 ceph-mon[36320]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-16T12:01:13.296 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:01:12 smithi154 ceph-mon[36320]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-16T12:01:13.297 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:01:12 smithi154 ceph-mon[36320]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-16T12:01:13.297 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:01:12 smithi154 ceph-mon[36320]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-16T12:01:13.297 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:01:12 smithi154 ceph-mon[36320]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:01:13.297 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:01:12 smithi154 ceph-mon[36320]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T12:01:13.297 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:01:12 smithi154 ceph-mon[36320]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' 2024-03-16T12:01:13.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:01:12 smithi029 ceph-mon[31975]: pgmap v324: 97 pgs: 2 active+clean+snaptrim, 4 active+clean+snaptrim_wait, 91 active+clean; 63 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.3 MiB/s wr, 265 op/s 2024-03-16T12:01:13.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:01:12 smithi029 ceph-mon[31975]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' 2024-03-16T12:01:13.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:01:12 smithi029 ceph-mon[31975]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' 2024-03-16T12:01:13.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:01:12 smithi029 ceph-mon[31975]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' 2024-03-16T12:01:13.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:01:12 smithi029 ceph-mon[31975]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' 2024-03-16T12:01:13.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:01:12 smithi029 ceph-mon[31975]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' 2024-03-16T12:01:13.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:01:12 smithi029 ceph-mon[31975]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' 2024-03-16T12:01:13.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:01:12 smithi029 ceph-mon[31975]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-16T12:01:13.339 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:01:12 smithi029 ceph-mon[31975]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-16T12:01:13.339 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:01:12 smithi029 ceph-mon[31975]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-16T12:01:13.339 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:01:12 smithi029 ceph-mon[31975]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-16T12:01:13.339 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:01:12 smithi029 ceph-mon[31975]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-16T12:01:13.339 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:01:12 smithi029 ceph-mon[31975]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-16T12:01:13.339 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:01:12 smithi029 ceph-mon[31975]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-16T12:01:13.339 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:01:12 smithi029 ceph-mon[31975]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-16T12:01:13.339 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:01:12 smithi029 ceph-mon[31975]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-16T12:01:13.339 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:01:12 smithi029 ceph-mon[31975]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-16T12:01:13.339 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:01:12 smithi029 ceph-mon[31975]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-16T12:01:13.339 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:01:12 smithi029 ceph-mon[31975]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-16T12:01:13.339 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:01:12 smithi029 ceph-mon[31975]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:01:13.340 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:01:12 smithi029 ceph-mon[31975]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T12:01:13.340 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:01:12 smithi029 ceph-mon[31975]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' 2024-03-16T12:01:13.801 INFO:tasks.workunit.client.0.smithi029.stdout: 1 1014945 9.66 MB/sec execute 418 sec latency 39.485 ms 2024-03-16T12:01:14.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:01:13 smithi152 ceph-mon[36291]: Adjusting osd_memory_target on smithi154 to 174.1M 2024-03-16T12:01:14.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:01:13 smithi152 ceph-mon[36291]: Unable to set osd_memory_target on smithi154 to 182636339: error parsing value: Value '182636339' is below minimum 939524096 2024-03-16T12:01:14.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:01:13 smithi152 ceph-mon[36291]: Adjusting osd_memory_target on smithi029 to 177.2M 2024-03-16T12:01:14.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:01:13 smithi152 ceph-mon[36291]: Unable to set osd_memory_target on smithi029 to 185894912: error parsing value: Value '185894912' is below minimum 939524096 2024-03-16T12:01:14.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:01:13 smithi152 ceph-mon[36291]: Adjusting osd_memory_target on smithi152 to 174.1M 2024-03-16T12:01:14.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:01:13 smithi152 ceph-mon[36291]: Unable to set osd_memory_target on smithi152 to 182635622: error parsing value: Value '182635622' is below minimum 939524096 2024-03-16T12:01:14.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:01:13 smithi154 ceph-mon[36320]: Adjusting osd_memory_target on smithi154 to 174.1M 2024-03-16T12:01:14.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:01:13 smithi154 ceph-mon[36320]: Unable to set osd_memory_target on smithi154 to 182636339: error parsing value: Value '182636339' is below minimum 939524096 2024-03-16T12:01:14.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:01:13 smithi154 ceph-mon[36320]: Adjusting osd_memory_target on smithi029 to 177.2M 2024-03-16T12:01:14.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:01:13 smithi154 ceph-mon[36320]: Unable to set osd_memory_target on smithi029 to 185894912: error parsing value: Value '185894912' is below minimum 939524096 2024-03-16T12:01:14.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:01:13 smithi154 ceph-mon[36320]: Adjusting osd_memory_target on smithi152 to 174.1M 2024-03-16T12:01:14.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:01:13 smithi154 ceph-mon[36320]: Unable to set osd_memory_target on smithi152 to 182635622: error parsing value: Value '182635622' is below minimum 939524096 2024-03-16T12:01:14.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:01:13 smithi029 ceph-mon[31975]: Adjusting osd_memory_target on smithi154 to 174.1M 2024-03-16T12:01:14.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:01:13 smithi029 ceph-mon[31975]: Unable to set osd_memory_target on smithi154 to 182636339: error parsing value: Value '182636339' is below minimum 939524096 2024-03-16T12:01:14.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:01:13 smithi029 ceph-mon[31975]: Adjusting osd_memory_target on smithi029 to 177.2M 2024-03-16T12:01:14.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:01:13 smithi029 ceph-mon[31975]: Unable to set osd_memory_target on smithi029 to 185894912: error parsing value: Value '185894912' is below minimum 939524096 2024-03-16T12:01:14.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:01:13 smithi029 ceph-mon[31975]: Adjusting osd_memory_target on smithi152 to 174.1M 2024-03-16T12:01:14.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:01:13 smithi029 ceph-mon[31975]: Unable to set osd_memory_target on smithi152 to 182635622: error parsing value: Value '182635622' is below minimum 939524096 2024-03-16T12:01:14.802 INFO:tasks.workunit.client.0.smithi029.stdout: 1 1016281 9.67 MB/sec execute 419 sec latency 37.078 ms 2024-03-16T12:01:15.127 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:01:14 smithi152 ceph-mon[36291]: pgmap v325: 97 pgs: 97 active+clean; 62 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 3.1 MiB/s wr, 250 op/s 2024-03-16T12:01:15.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:01:14 smithi154 ceph-mon[36320]: pgmap v325: 97 pgs: 97 active+clean; 62 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 3.1 MiB/s wr, 250 op/s 2024-03-16T12:01:15.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:01:14 smithi029 ceph-mon[31975]: pgmap v325: 97 pgs: 97 active+clean; 62 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 3.1 MiB/s wr, 250 op/s 2024-03-16T12:01:15.801 INFO:tasks.workunit.client.0.smithi029.stdout: 1 1017643 9.66 MB/sec execute 420 sec latency 24.036 ms 2024-03-16T12:01:16.802 INFO:tasks.workunit.client.0.smithi029.stdout: 1 1019605 9.66 MB/sec execute 421 sec latency 21.022 ms 2024-03-16T12:01:17.166 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:01:16 smithi152 ceph-mon[36291]: pgmap v326: 97 pgs: 97 active+clean; 62 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 2.7 MiB/s wr, 221 op/s 2024-03-16T12:01:17.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:01:16 smithi154 ceph-mon[36320]: pgmap v326: 97 pgs: 97 active+clean; 62 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 2.7 MiB/s wr, 221 op/s 2024-03-16T12:01:17.338 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:01:16 smithi029 ceph-mon[31975]: pgmap v326: 97 pgs: 97 active+clean; 62 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 2.7 MiB/s wr, 221 op/s 2024-03-16T12:01:17.801 INFO:tasks.workunit.client.0.smithi029.stdout: 1 1020281 9.65 MB/sec execute 422 sec latency 49.138 ms 2024-03-16T12:01:18.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:01:18 smithi152 ceph-mon[36291]: pgmap v327: 97 pgs: 97 active+clean; 67 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 2.7 MiB/s wr, 233 op/s 2024-03-16T12:01:18.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:01:18 smithi152 ceph-mon[36291]: osdmap e102: 12 total, 12 up, 12 in 2024-03-16T12:01:18.545 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:01:18 smithi154 ceph-mon[36320]: pgmap v327: 97 pgs: 97 active+clean; 67 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 2.7 MiB/s wr, 233 op/s 2024-03-16T12:01:18.545 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:01:18 smithi154 ceph-mon[36320]: osdmap e102: 12 total, 12 up, 12 in 2024-03-16T12:01:18.587 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:01:18 smithi029 ceph-mon[31975]: pgmap v327: 97 pgs: 97 active+clean; 67 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 2.7 MiB/s wr, 233 op/s 2024-03-16T12:01:18.587 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:01:18 smithi029 ceph-mon[31975]: osdmap e102: 12 total, 12 up, 12 in 2024-03-16T12:01:18.801 INFO:tasks.workunit.client.0.smithi029.stdout: 1 1021459 9.64 MB/sec execute 423 sec latency 205.792 ms 2024-03-16T12:01:19.801 INFO:tasks.workunit.client.0.smithi029.stdout: 1 1023259 9.64 MB/sec execute 424 sec latency 36.389 ms 2024-03-16T12:01:20.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:01:20 smithi152 ceph-mon[36291]: pgmap v329: 97 pgs: 97 active+clean; 67 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.1 MiB/s wr, 263 op/s 2024-03-16T12:01:20.545 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:01:20 smithi154 ceph-mon[36320]: pgmap v329: 97 pgs: 97 active+clean; 67 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.1 MiB/s wr, 263 op/s 2024-03-16T12:01:20.588 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:01:20 smithi029 ceph-mon[31975]: pgmap v329: 97 pgs: 97 active+clean; 67 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.1 MiB/s wr, 263 op/s 2024-03-16T12:01:20.802 INFO:tasks.workunit.client.0.smithi029.stdout: 1 1024573 9.63 MB/sec execute 425 sec latency 30.649 ms 2024-03-16T12:01:21.802 INFO:tasks.workunit.client.0.smithi029.stdout: 1 1026693 9.64 MB/sec execute 426 sec latency 19.186 ms 2024-03-16T12:01:22.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:01:22 smithi152 ceph-mon[36291]: pgmap v330: 97 pgs: 97 active+clean; 72 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 2.7 MiB/s wr, 176 op/s 2024-03-16T12:01:22.545 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:01:22 smithi154 ceph-mon[36320]: pgmap v330: 97 pgs: 97 active+clean; 72 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 2.7 MiB/s wr, 176 op/s 2024-03-16T12:01:22.550 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:22.550+0000 7f1e06a7e640 1 -- 172.21.15.29:0/3041275973 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1e00105680 msgr2=0x7f1e00105a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:01:22.550 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:22.550+0000 7f1e06a7e640 1 --2- 172.21.15.29:0/3041275973 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1e00105680 0x7f1e00105a60 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f1df8009900 tx=0x7f1df802bc30 comp rx=0 tx=0).stop 2024-03-16T12:01:22.551 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:22.550+0000 7f1e06a7e640 1 -- 172.21.15.29:0/3041275973 shutdown_connections 2024-03-16T12:01:22.551 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:22.550+0000 7f1e06a7e640 1 --2- 172.21.15.29:0/3041275973 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1e001012f0 0x7f1e0010d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:01:22.551 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:22.550+0000 7f1e06a7e640 1 --2- 172.21.15.29:0/3041275973 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1e00100930 0x7f1e00100db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:01:22.551 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:22.550+0000 7f1e06a7e640 1 --2- 172.21.15.29:0/3041275973 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1e00105680 0x7f1e00105a60 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:01:22.551 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:22.550+0000 7f1e06a7e640 1 -- 172.21.15.29:0/3041275973 >> 172.21.15.29:0/3041275973 conn(0x7f1e00077530 msgr2=0x7f1e00077940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:01:22.551 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:22.550+0000 7f1e06a7e640 1 -- 172.21.15.29:0/3041275973 shutdown_connections 2024-03-16T12:01:22.551 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:22.550+0000 7f1e06a7e640 1 -- 172.21.15.29:0/3041275973 wait complete. 2024-03-16T12:01:22.551 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:22.550+0000 7f1e06a7e640 1 Processor -- start 2024-03-16T12:01:22.551 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:22.551+0000 7f1e06a7e640 1 -- start start 2024-03-16T12:01:22.553 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:22.551+0000 7f1e06a7e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1e00100930 0x7f1e00198f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:01:22.553 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:22.551+0000 7f1e06a7e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1e001012f0 0x7f1e00199440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:01:22.553 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:22.551+0000 7f1e06a7e640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1e00105680 0x7f1e001a3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:01:22.553 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:22.551+0000 7f1e06a7e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f1e00110ba0 con 0x7f1e001012f0 2024-03-16T12:01:22.553 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:22.551+0000 7f1e06a7e640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f1e00110a20 con 0x7f1e00105680 2024-03-16T12:01:22.553 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:22.551+0000 7f1e06a7e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f1e00110d20 con 0x7f1e00100930 2024-03-16T12:01:22.553 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:22.551+0000 7f1e0481a640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1e00100930 0x7f1e00198f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:01:22.553 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:22.551+0000 7f1dfffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1e001012f0 0x7f1e00199440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:01:22.553 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:22.551+0000 7f1e0481a640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1e00100930 0x7f1e00198f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.29:60044/0 (socket says 172.21.15.29:60044) 2024-03-16T12:01:22.554 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:22.551+0000 7f1e0481a640 1 -- 172.21.15.29:0/4190744623 learned_addr learned my addr 172.21.15.29:0/4190744623 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T12:01:22.554 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:22.551+0000 7f1e0501b640 1 --2- 172.21.15.29:0/4190744623 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1e00105680 0x7f1e001a3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:01:22.554 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:22.551+0000 7f1dfffff640 1 -- 172.21.15.29:0/4190744623 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1e00100930 msgr2=0x7f1e00198f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:01:22.554 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:22.551+0000 7f1dfffff640 1 --2- 172.21.15.29:0/4190744623 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1e00100930 0x7f1e00198f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:01:22.554 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:22.551+0000 7f1dfffff640 1 -- 172.21.15.29:0/4190744623 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1e00105680 msgr2=0x7f1e001a3f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:01:22.554 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:22.551+0000 7f1dfffff640 1 --2- 172.21.15.29:0/4190744623 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1e00105680 0x7f1e001a3f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:01:22.554 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:22.551+0000 7f1dfffff640 1 -- 172.21.15.29:0/4190744623 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e001a44c0 con 0x7f1e001012f0 2024-03-16T12:01:22.554 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:22.551+0000 7f1dfffff640 1 --2- 172.21.15.29:0/4190744623 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1e001012f0 0x7f1e00199440 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f1df0009fd0 tx=0x7f1df000ef90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:01:22.554 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:22.552+0000 7f1dfdffb640 1 -- 172.21.15.29:0/4190744623 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1df0004b20 con 0x7f1e001012f0 2024-03-16T12:01:22.554 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:22.552+0000 7f1dfdffb640 1 -- 172.21.15.29:0/4190744623 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1df0004cc0 con 0x7f1e001012f0 2024-03-16T12:01:22.554 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:22.552+0000 7f1e06a7e640 1 -- 172.21.15.29:0/4190744623 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1e001a4690 con 0x7f1e001012f0 2024-03-16T12:01:22.554 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:22.552+0000 7f1dfdffb640 1 -- 172.21.15.29:0/4190744623 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1df00107b0 con 0x7f1e001012f0 2024-03-16T12:01:22.554 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:22.552+0000 7f1e06a7e640 1 -- 172.21.15.29:0/4190744623 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1e001a4b90 con 0x7f1e001012f0 2024-03-16T12:01:22.561 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:22.554+0000 7f1dfdffb640 1 -- 172.21.15.29:0/4190744623 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7f1df0005020 con 0x7f1e001012f0 2024-03-16T12:01:22.561 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:22.554+0000 7f1dfdffb640 1 --2- 172.21.15.29:0/4190744623 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f1dec0a2a10 0x7f1dec0a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:01:22.561 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:22.554+0000 7f1e09663740 1 Processor -- start 2024-03-16T12:01:22.561 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:22.554+0000 7f1e09663740 1 -- start start 2024-03-16T12:01:22.561 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:22.554+0000 7f1dfdffb640 1 -- 172.21.15.29:0/4190744623 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 8556+0+0 (secure 0 0 0) 0x7f1df00cf5e0 con 0x7f1e001012f0 2024-03-16T12:01:22.561 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:22.554+0000 7f1e0481a640 1 --2- 172.21.15.29:0/4190744623 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f1dec0a2a10 0x7f1dec0a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:01:22.561 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:22.555+0000 7f1e09663740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560f0c9b5130 0x560f0c9b5510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:01:22.561 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:22.555+0000 7f1e09663740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560f0c9b5a50 0x560f0c9be2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:01:22.561 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:22.555+0000 7f1e09663740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560f0c9be820 0x560f0c9c0c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:01:22.562 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:22.555+0000 7f1e09663740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x560f0c919a40 con 0x560f0c9be820 2024-03-16T12:01:22.562 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:22.555+0000 7f1e09663740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x560f0c8c2920 con 0x560f0c9b5a50 2024-03-16T12:01:22.562 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:22.555+0000 7f1e09663740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x560f0c915ae0 con 0x560f0c9b5130 2024-03-16T12:01:22.562 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:22.555+0000 7f1e0501b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560f0c9be820 0x560f0c9c0c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:01:22.562 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:22.555+0000 7f1e0501b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560f0c9be820 0x560f0c9c0c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46462/0 (socket says 172.21.15.29:46462) 2024-03-16T12:01:22.562 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:22.555+0000 7f1e0501b640 1 -- 172.21.15.29:0/4003944631 learned_addr learned my addr 172.21.15.29:0/4003944631 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T12:01:22.562 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:22.555+0000 7f1dfffff640 1 --2- 172.21.15.29:0/4003944631 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560f0c9b5a50 0x560f0c9be2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:01:22.562 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:22.555+0000 7f1e0481a640 1 --2- 172.21.15.29:0/4003944631 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560f0c9b5130 0x560f0c9b5510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:01:22.562 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:22.555+0000 7f1e0481a640 1 --2- 172.21.15.29:0/4190744623 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f1dec0a2a10 0x7f1dec0a4ed0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f1df8000c00 tx=0x7f1df80058d0 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:01:22.562 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:22.555+0000 7f1e0501b640 1 -- 172.21.15.29:0/4003944631 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560f0c9b5130 msgr2=0x560f0c9b5510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:01:22.562 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:22.555+0000 7f1e0501b640 1 --2- 172.21.15.29:0/4003944631 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560f0c9b5130 0x560f0c9b5510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:01:22.562 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:22.555+0000 7f1e0501b640 1 -- 172.21.15.29:0/4003944631 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560f0c9b5a50 msgr2=0x560f0c9be2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:01:22.562 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:22.555+0000 7f1e0501b640 1 --2- 172.21.15.29:0/4003944631 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560f0c9b5a50 0x560f0c9be2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:01:22.562 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:22.555+0000 7f1e0501b640 1 -- 172.21.15.29:0/4003944631 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560f0c8da460 con 0x560f0c9be820 2024-03-16T12:01:22.563 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:22.555+0000 7f1e0481a640 1 --2- 172.21.15.29:0/4003944631 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560f0c9b5130 0x560f0c9b5510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T12:01:22.563 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:22.555+0000 7f1dfffff640 1 --2- 172.21.15.29:0/4003944631 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560f0c9b5a50 0x560f0c9be2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:01:22.563 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:22.555+0000 7f1e0501b640 1 --2- 172.21.15.29:0/4003944631 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560f0c9be820 0x560f0c9c0c10 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f1de8009c10 tx=0x7f1de8011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6abb1e9c9ac327d server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:01:22.563 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:22.555+0000 7f1df6ffd640 1 -- 172.21.15.29:0/4003944631 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1de801e070 con 0x560f0c9be820 2024-03-16T12:01:22.563 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:22.555+0000 7f1df6ffd640 1 -- 172.21.15.29:0/4003944631 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1de8019b10 con 0x560f0c9be820 2024-03-16T12:01:22.563 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:22.555+0000 7f1df6ffd640 1 -- 172.21.15.29:0/4003944631 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1de800e970 con 0x560f0c9be820 2024-03-16T12:01:22.563 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:22.556+0000 7f1e09663740 1 -- 172.21.15.29:0/4003944631 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560f0c9be820 msgr2=0x560f0c9c0c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:01:22.563 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:22.556+0000 7f1e09663740 1 --2- 172.21.15.29:0/4003944631 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560f0c9be820 0x560f0c9c0c10 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f1de8009c10 tx=0x7f1de8011810 comp rx=0 tx=0).stop 2024-03-16T12:01:22.563 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:22.556+0000 7f1e09663740 1 -- 172.21.15.29:0/4003944631 shutdown_connections 2024-03-16T12:01:22.563 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:22.556+0000 7f1e09663740 1 --2- 172.21.15.29:0/4003944631 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560f0c9be820 0x560f0c9c0c10 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:01:22.563 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:22.556+0000 7f1e09663740 1 --2- 172.21.15.29:0/4003944631 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560f0c9b5a50 0x560f0c9be2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:01:22.563 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:22.556+0000 7f1e09663740 1 --2- 172.21.15.29:0/4003944631 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560f0c9b5130 0x560f0c9b5510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:01:22.563 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:22.556+0000 7f1e09663740 1 -- 172.21.15.29:0/4003944631 >> 172.21.15.29:0/4003944631 conn(0x560f0c9afa50 msgr2=0x560f0c9afe30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:01:22.563 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:22.556+0000 7f1e09663740 1 -- 172.21.15.29:0/4003944631 shutdown_connections 2024-03-16T12:01:22.563 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:22.556+0000 7f1e09663740 1 -- 172.21.15.29:0/4003944631 wait complete. 2024-03-16T12:01:22.564 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:22.556+0000 7f1e09663740 1 -- start start 2024-03-16T12:01:22.564 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:22.557+0000 7f1e09663740 1 Processor -- start 2024-03-16T12:01:22.564 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:22.557+0000 7f1e09663740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560f0c9b5130 0x560f0c9bb8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:01:22.564 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:22.557+0000 7f1e09663740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560f0c9b5a50 0x560f0c9bbe10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:01:22.564 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:22.557+0000 7f1e09663740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560f0c9be820 0x560f0c9bc350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:01:22.564 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:22.557+0000 7f1e09663740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x560f0c9c1450 con 0x560f0c9b5130 2024-03-16T12:01:22.564 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:22.557+0000 7f1e09663740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x560f0c8d5d70 con 0x560f0c9b5a50 2024-03-16T12:01:22.564 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:22.557+0000 7f1e09663740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x560f0c9c15d0 con 0x560f0c9be820 2024-03-16T12:01:22.564 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:22.557+0000 7f1e09663740 10 client.0 _ll_register_callbacks cb 0x560f0c62da00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T12:01:22.564 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:22.557+0000 7f1e0481a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560f0c9b5130 0x560f0c9bb8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:01:22.564 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:22.557+0000 7f1e0481a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560f0c9b5130 0x560f0c9bb8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46470/0 (socket says 172.21.15.29:46470) 2024-03-16T12:01:22.564 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:22.557+0000 7f1e0481a640 1 -- 172.21.15.29:0/3603667261 learned_addr learned my addr 172.21.15.29:0/3603667261 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T12:01:22.564 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:22.557+0000 7f1dfffff640 1 --2- 172.21.15.29:0/3603667261 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560f0c9b5a50 0x560f0c9bbe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:01:22.564 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:22.557+0000 7f1e0501b640 1 --2- 172.21.15.29:0/3603667261 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560f0c9be820 0x560f0c9bc350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:01:22.565 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:22.557+0000 7f1e0481a640 1 -- 172.21.15.29:0/3603667261 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560f0c9be820 msgr2=0x560f0c9bc350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:01:22.565 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:22.557+0000 7f1e0481a640 1 --2- 172.21.15.29:0/3603667261 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560f0c9be820 0x560f0c9bc350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:01:22.565 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:22.557+0000 7f1e0481a640 1 -- 172.21.15.29:0/3603667261 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560f0c9b5a50 msgr2=0x560f0c9bbe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:01:22.565 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:22.557+0000 7f1e0481a640 1 --2- 172.21.15.29:0/3603667261 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560f0c9b5a50 0x560f0c9bbe10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:01:22.565 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:22.557+0000 7f1e0481a640 1 -- 172.21.15.29:0/3603667261 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560f0c62e210 con 0x560f0c9b5130 2024-03-16T12:01:22.565 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:22.557+0000 7f1e0501b640 1 --2- 172.21.15.29:0/3603667261 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560f0c9be820 0x560f0c9bc350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T12:01:22.565 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:22.558+0000 7f1e0481a640 1 --2- 172.21.15.29:0/3603667261 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560f0c9b5130 0x560f0c9bb8d0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f1df8036040 tx=0x7f1df802bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=479476b2a9a8f71e server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:01:22.565 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:22.558+0000 7f1ddffff640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-16T12:01:22.565 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:22.558+0000 7f1ddffff640 1 -- 172.21.15.29:0/3603667261 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1df8004b90 con 0x560f0c9b5130 2024-03-16T12:01:22.565 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:22.558+0000 7f1ddffff640 1 -- 172.21.15.29:0/3603667261 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 8556+0+0 (secure 0 0 0) 0x7f1df8004ec0 con 0x560f0c9b5130 2024-03-16T12:01:22.565 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:22.558+0000 7f1e09663740 1 -- 172.21.15.29:0/3603667261 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1de8019b10 con 0x560f0c9b5130 2024-03-16T12:01:22.565 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:22.558+0000 7f1ddffff640 1 -- 172.21.15.29:0/3603667261 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f1df80050c0 con 0x560f0c9b5130 2024-03-16T12:01:22.565 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:22.558+0000 7f1e09663740 10 client.16272 fetch_fsmap learned FSMap version 28 2024-03-16T12:01:22.565 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:22.558+0000 7f1e09663740 1 -- 172.21.15.29:0/3603667261 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x560f0c6fa620 con 0x560f0c9b5130 2024-03-16T12:01:22.566 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:22.558+0000 7f1ddffff640 1 -- 172.21.15.29:0/3603667261 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1de8019b10 con 0x560f0c9b5130 2024-03-16T12:01:22.566 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:22.558+0000 7f1ddffff640 1 -- 172.21.15.29:0/3603667261 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 28) ==== 8338+0+0 (secure 0 0 0) 0x7f1df805c450 con 0x560f0c9b5130 2024-03-16T12:01:22.566 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:22.558+0000 7f1e09663740 10 client.16272 fetch_fsmap finished waiting for FSMap version 28 2024-03-16T12:01:22.566 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:22.558+0000 7f1e09663740 10 client.16272 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T12:01:22.566 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:22.558+0000 7f1e09663740 20 client.16272 populate_metadata read hostname 'smithi029' 2024-03-16T12:01:22.566 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:22.558+0000 7f1e09663740 1 --2- 172.21.15.29:0/3603667261 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x560f0c9ce9e0 0x560f0c9d0dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:01:22.566 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:22.558+0000 7f1e09663740 4 client.16272 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T12:01:22.566 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:22.558+0000 7f1e09663740 1 -- 172.21.15.29:0/3603667261 --> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560f0c926f80 con 0x560f0c9ce9e0 2024-03-16T12:01:22.566 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:22.559+0000 7f1dfffff640 1 --2- 172.21.15.29:0/3603667261 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x560f0c9ce9e0 0x560f0c9d0dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:01:22.566 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:22.559+0000 7f1dfffff640 1 --2- 172.21.15.29:0/3603667261 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x560f0c9ce9e0 0x560f0c9d0dd0 crc :-1 s=READY pgs=132 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3c557e03b0333a58 server_cookie=10f94418f5e6cd4a in_seq=0 out_seq=0 2024-03-16T12:01:22.566 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:22.559+0000 7f1ddffff640 10 client.16272 ms_handle_connect on v2:172.21.15.152:6838/1548731833 2024-03-16T12:01:22.566 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:22.560+0000 7f1ddffff640 1 -- 172.21.15.29:0/3603667261 <== mds.0 v2:172.21.15.152:6838/1548731833 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x560f0c926f80 con 0x560f0c9ce9e0 2024-03-16T12:01:22.566 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:22.560+0000 7f1ddffff640 10 client.16272 handle_command_reply: tid=0 2024-03-16T12:01:22.566 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:22.560+0000 7f1e09663740 1 client.16272 shutdown 2024-03-16T12:01:22.566 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:22.560+0000 7f1e09663740 1 -- 172.21.15.29:0/3603667261 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560f0c9b5130 msgr2=0x560f0c9bb8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:01:22.567 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:22.560+0000 7f1e09663740 1 --2- 172.21.15.29:0/3603667261 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560f0c9b5130 0x560f0c9bb8d0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f1df8036040 tx=0x7f1df802bc30 comp rx=0 tx=0).stop 2024-03-16T12:01:22.567 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:22.560+0000 7f1e09663740 1 -- 172.21.15.29:0/3603667261 shutdown_connections 2024-03-16T12:01:22.567 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:22.560+0000 7f1e09663740 1 --2- 172.21.15.29:0/3603667261 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560f0c9be820 0x560f0c9bc350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:01:22.567 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:22.560+0000 7f1e09663740 1 --2- 172.21.15.29:0/3603667261 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x560f0c9ce9e0 0x560f0c9d0dd0 crc :-1 s=READY pgs=132 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:01:22.567 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:22.560+0000 7f1e09663740 1 --2- 172.21.15.29:0/3603667261 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560f0c9b5a50 0x560f0c9bbe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:01:22.567 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:22.560+0000 7f1ddffff640 0 client.16272 ms_handle_reset on v2:172.21.15.152:6838/1548731833 2024-03-16T12:01:22.567 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:22.560+0000 7f1e09663740 1 --2- 172.21.15.29:0/3603667261 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560f0c9b5130 0x560f0c9bb8d0 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:01:22.567 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:22.560+0000 7f1e09663740 1 -- 172.21.15.29:0/3603667261 >> 172.21.15.29:0/3603667261 conn(0x560f0c9afa50 msgr2=0x560f0c9b17a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:01:22.567 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:22.560+0000 7f1e09663740 1 -- 172.21.15.29:0/3603667261 shutdown_connections 2024-03-16T12:01:22.567 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:22.560+0000 7f1e09663740 1 -- 172.21.15.29:0/3603667261 wait complete. 2024-03-16T12:01:22.567 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:22.561+0000 7f1e09663740 20 client.16272 trim_cache size 0 max 16384 2024-03-16T12:01:22.576 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:22.575+0000 7f1e09663740 1 Processor -- start 2024-03-16T12:01:22.576 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:22.575+0000 7f1e09663740 1 -- start start 2024-03-16T12:01:22.576 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:22.575+0000 7f1e09663740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560f0c9b5130 0x560f0c9be000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:01:22.577 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:22.575+0000 7f1e09663740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560f0c9b5a50 0x560f0c9ca1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:01:22.577 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:22.575+0000 7f1e09663740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560f0c9be820 0x560f0c9b15b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:01:22.577 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:22.575+0000 7f1e09663740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f1e001afde0 con 0x560f0c9b5130 2024-03-16T12:01:22.577 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:22.575+0000 7f1e09663740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f1e001ad6b0 con 0x560f0c9b5a50 2024-03-16T12:01:22.577 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:22.575+0000 7f1e09663740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f1e001ab520 con 0x560f0c9be820 2024-03-16T12:01:22.577 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:22.576+0000 7f1e0481a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560f0c9b5130 0x560f0c9be000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:01:22.577 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:22.576+0000 7f1e0481a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560f0c9b5130 0x560f0c9be000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46472/0 (socket says 172.21.15.29:46472) 2024-03-16T12:01:22.577 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:22.576+0000 7f1e0481a640 1 -- 172.21.15.29:0/1212859976 learned_addr learned my addr 172.21.15.29:0/1212859976 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T12:01:22.577 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:22.576+0000 7f1e0501b640 1 --2- 172.21.15.29:0/1212859976 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560f0c9be820 0x560f0c9b15b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:01:22.577 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:22.576+0000 7f1dfffff640 1 --2- 172.21.15.29:0/1212859976 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560f0c9b5a50 0x560f0c9ca1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:01:22.577 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:22.576+0000 7f1e0481a640 1 -- 172.21.15.29:0/1212859976 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560f0c9be820 msgr2=0x560f0c9b15b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:01:22.577 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:22.576+0000 7f1e0481a640 1 --2- 172.21.15.29:0/1212859976 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560f0c9be820 0x560f0c9b15b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:01:22.577 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:22.576+0000 7f1e0481a640 1 -- 172.21.15.29:0/1212859976 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560f0c9b5a50 msgr2=0x560f0c9ca1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:01:22.577 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:22.576+0000 7f1e0481a640 1 --2- 172.21.15.29:0/1212859976 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560f0c9b5a50 0x560f0c9ca1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:01:22.578 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:22.576+0000 7f1e0481a640 1 -- 172.21.15.29:0/1212859976 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560f0c9d1140 con 0x560f0c9b5130 2024-03-16T12:01:22.578 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:22.576+0000 7f1e0481a640 1 --2- 172.21.15.29:0/1212859976 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560f0c9b5130 0x560f0c9be000 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f1df8009430 tx=0x7f1df8005db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ef9c148621aabf1 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:01:22.578 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:22.576+0000 7f1df67fc640 1 -- 172.21.15.29:0/1212859976 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1df8061070 con 0x560f0c9b5130 2024-03-16T12:01:22.578 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:22.576+0000 7f1df67fc640 1 -- 172.21.15.29:0/1212859976 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1df804e070 con 0x560f0c9b5130 2024-03-16T12:01:22.578 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:22.576+0000 7f1df67fc640 1 -- 172.21.15.29:0/1212859976 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1df8004e10 con 0x560f0c9b5130 2024-03-16T12:01:22.581 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:22.577+0000 7f1e09663740 1 -- 172.21.15.29:0/1212859976 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560f0c9b5130 msgr2=0x560f0c9be000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:01:22.581 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:22.577+0000 7f1e09663740 1 --2- 172.21.15.29:0/1212859976 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560f0c9b5130 0x560f0c9be000 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f1df8009430 tx=0x7f1df8005db0 comp rx=0 tx=0).stop 2024-03-16T12:01:22.582 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:22.577+0000 7f1e09663740 1 -- 172.21.15.29:0/1212859976 shutdown_connections 2024-03-16T12:01:22.582 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:22.577+0000 7f1e09663740 1 --2- 172.21.15.29:0/1212859976 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560f0c9be820 0x560f0c9b15b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:01:22.582 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:22.577+0000 7f1e09663740 1 --2- 172.21.15.29:0/1212859976 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560f0c9b5a50 0x560f0c9ca1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:01:22.582 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:22.577+0000 7f1e09663740 1 --2- 172.21.15.29:0/1212859976 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560f0c9b5130 0x560f0c9be000 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:01:22.582 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:22.577+0000 7f1e09663740 1 -- 172.21.15.29:0/1212859976 >> 172.21.15.29:0/1212859976 conn(0x560f0c9afa50 msgr2=0x560f0c9b0ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:01:22.582 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:22.577+0000 7f1e09663740 1 -- 172.21.15.29:0/1212859976 shutdown_connections 2024-03-16T12:01:22.582 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:22.577+0000 7f1e09663740 1 -- 172.21.15.29:0/1212859976 wait complete. 2024-03-16T12:01:22.582 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:22.577+0000 7f1e09663740 1 -- start start 2024-03-16T12:01:22.582 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:22.577+0000 7f1e09663740 1 Processor -- start 2024-03-16T12:01:22.582 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:22.578+0000 7f1e09663740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560f0c9b5130 0x560f0c9b8ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:01:22.582 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:22.578+0000 7f1e09663740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560f0c9b5a50 0x560f0c9b9000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:01:22.582 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:22.578+0000 7f1e09663740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560f0c9be820 0x560f0ca33d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:01:22.582 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:22.578+0000 7f1e09663740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f1e001abb70 con 0x560f0c9be820 2024-03-16T12:01:22.583 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:22.578+0000 7f1e09663740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f1e001acfe0 con 0x560f0c9b5a50 2024-03-16T12:01:22.583 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:22.578+0000 7f1e09663740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f1e001ac240 con 0x560f0c9b5130 2024-03-16T12:01:22.583 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:22.578+0000 7f1e09663740 10 client.0 _ll_register_callbacks cb 0x7f1df8042070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T12:01:22.583 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:22.578+0000 7f1e0501b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560f0c9be820 0x560f0ca33d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:01:22.583 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:22.578+0000 7f1e0501b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560f0c9be820 0x560f0ca33d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46478/0 (socket says 172.21.15.29:46478) 2024-03-16T12:01:22.583 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:22.578+0000 7f1e0501b640 1 -- 172.21.15.29:0/3455001479 learned_addr learned my addr 172.21.15.29:0/3455001479 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T12:01:22.583 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:22.578+0000 7f1dfffff640 1 --2- 172.21.15.29:0/3455001479 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560f0c9b5a50 0x560f0c9b9000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:01:22.583 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:22.578+0000 7f1e0481a640 1 --2- 172.21.15.29:0/3455001479 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560f0c9b5130 0x560f0c9b8ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:01:22.583 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:22.578+0000 7f1e0501b640 1 -- 172.21.15.29:0/3455001479 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560f0c9b5130 msgr2=0x560f0c9b8ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:01:22.583 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:22.578+0000 7f1e0501b640 1 --2- 172.21.15.29:0/3455001479 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560f0c9b5130 0x560f0c9b8ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:01:22.583 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:22.578+0000 7f1e0501b640 1 -- 172.21.15.29:0/3455001479 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560f0c9b5a50 msgr2=0x560f0c9b9000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:01:22.583 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:22.578+0000 7f1e0501b640 1 --2- 172.21.15.29:0/3455001479 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560f0c9b5a50 0x560f0c9b9000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:01:22.583 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:22.578+0000 7f1e0501b640 1 -- 172.21.15.29:0/3455001479 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560f0ca34250 con 0x560f0c9be820 2024-03-16T12:01:22.584 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:22.579+0000 7f1e0501b640 1 --2- 172.21.15.29:0/3455001479 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560f0c9be820 0x560f0ca33d10 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f1de8011d10 tx=0x7f1de80099a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c63f8e2b54227545 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:01:22.584 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:22.579+0000 7f1ddffff640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-16T12:01:22.584 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:22.579+0000 7f1ddffff640 1 -- 172.21.15.29:0/3455001479 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1de8029070 con 0x560f0c9be820 2024-03-16T12:01:22.584 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:22.579+0000 7f1e09663740 1 -- 172.21.15.29:0/3455001479 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1df804e070 con 0x560f0c9be820 2024-03-16T12:01:22.584 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:22.579+0000 7f1ddffff640 1 -- 172.21.15.29:0/3455001479 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 8556+0+0 (secure 0 0 0) 0x7f1de8005300 con 0x560f0c9be820 2024-03-16T12:01:22.584 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:22.579+0000 7f1ddffff640 1 -- 172.21.15.29:0/3455001479 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f1de801e040 con 0x560f0c9be820 2024-03-16T12:01:22.584 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:22.579+0000 7f1e09663740 10 client.16278 fetch_fsmap learned FSMap version 28 2024-03-16T12:01:22.584 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:22.579+0000 7f1e09663740 1 -- 172.21.15.29:0/3455001479 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x560f0ca35460 con 0x560f0c9be820 2024-03-16T12:01:22.584 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:22.579+0000 7f1ddffff640 1 -- 172.21.15.29:0/3455001479 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1df804e070 con 0x560f0c9be820 2024-03-16T12:01:22.584 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:22.579+0000 7f1ddffff640 1 -- 172.21.15.29:0/3455001479 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 28) ==== 8338+0+0 (secure 0 0 0) 0x7f1de8025620 con 0x560f0c9be820 2024-03-16T12:01:22.584 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:22.579+0000 7f1e09663740 10 client.16278 fetch_fsmap finished waiting for FSMap version 28 2024-03-16T12:01:22.584 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:22.579+0000 7f1e09663740 10 client.16278 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T12:01:22.584 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:22.579+0000 7f1e09663740 20 client.16278 populate_metadata read hostname 'smithi029' 2024-03-16T12:01:22.584 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:22.579+0000 7f1e09663740 1 --2- 172.21.15.29:0/3455001479 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x560f0ca359c0 0x560f0ca37de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:01:22.584 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:22.579+0000 7f1e09663740 4 client.16278 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-16T12:01:22.585 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:22.579+0000 7f1e09663740 1 -- 172.21.15.29:0/3455001479 --> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] -- command(tid 0: {"prefix": "flush journal"}) -- 0x560f0c9212d0 con 0x560f0ca359c0 2024-03-16T12:01:22.585 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:22.579+0000 7f1e0481a640 1 --2- 172.21.15.29:0/3455001479 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x560f0ca359c0 0x560f0ca37de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:01:22.585 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:22.580+0000 7f1ddffff640 10 client.16278 ms_handle_connect on v2:172.21.15.152:6838/1548731833 2024-03-16T12:01:22.585 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:22.580+0000 7f1e0481a640 1 --2- 172.21.15.29:0/3455001479 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x560f0ca359c0 0x560f0ca37de0 crc :-1 s=READY pgs=133 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3db9beb2644739b6 server_cookie=e62f535b4159678d in_seq=0 out_seq=0 2024-03-16T12:01:22.587 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:01:22 smithi029 ceph-mon[31975]: pgmap v330: 97 pgs: 97 active+clean; 72 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 2.7 MiB/s wr, 176 op/s 2024-03-16T12:01:22.739 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:22.738+0000 7f1ddffff640 1 -- 172.21.15.29:0/3455001479 <== mds.0 v2:172.21.15.152:6838/1548731833 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x560f0c9212d0 con 0x560f0ca359c0 2024-03-16T12:01:22.739 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:22.738+0000 7f1ddffff640 10 client.16278 handle_command_reply: tid=0 2024-03-16T12:01:22.739 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:22.738+0000 7f1e09663740 1 client.16278 shutdown 2024-03-16T12:01:22.739 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:22.738+0000 7f1e09663740 1 -- 172.21.15.29:0/3455001479 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560f0c9be820 msgr2=0x560f0ca33d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:01:22.739 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:22.738+0000 7f1e09663740 1 --2- 172.21.15.29:0/3455001479 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560f0c9be820 0x560f0ca33d10 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f1de8011d10 tx=0x7f1de80099a0 comp rx=0 tx=0).stop 2024-03-16T12:01:22.739 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:22.738+0000 7f1e09663740 1 -- 172.21.15.29:0/3455001479 shutdown_connections 2024-03-16T12:01:22.740 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:22.738+0000 7f1e09663740 1 --2- 172.21.15.29:0/3455001479 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560f0c9be820 0x560f0ca33d10 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:01:22.740 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:22.738+0000 7f1e09663740 1 --2- 172.21.15.29:0/3455001479 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x560f0ca359c0 0x560f0ca37de0 crc :-1 s=READY pgs=133 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:01:22.740 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:22.738+0000 7f1e09663740 1 --2- 172.21.15.29:0/3455001479 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560f0c9b5a50 0x560f0c9b9000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:01:22.740 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:22.738+0000 7f1e09663740 1 --2- 172.21.15.29:0/3455001479 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560f0c9b5130 0x560f0c9b8ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:01:22.740 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:22.738+0000 7f1e09663740 1 -- 172.21.15.29:0/3455001479 >> 172.21.15.29:0/3455001479 conn(0x560f0c9afa50 msgr2=0x560f0c9b0ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:01:22.740 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:22.738+0000 7f1ddffff640 0 client.16278 ms_handle_reset on v2:172.21.15.152:6838/1548731833 2024-03-16T12:01:22.740 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:22.738+0000 7f1e09663740 1 -- 172.21.15.29:0/3455001479 shutdown_connections 2024-03-16T12:01:22.740 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:22.738+0000 7f1e09663740 1 -- 172.21.15.29:0/3455001479 wait complete. 2024-03-16T12:01:22.740 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:22.738+0000 7f1e09663740 20 client.16278 trim_cache size 0 max 16384 2024-03-16T12:01:22.740 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:{ 2024-03-16T12:01:22.740 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "message": "", 2024-03-16T12:01:22.741 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "return_code": 0 2024-03-16T12:01:22.741 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:} 2024-03-16T12:01:22.741 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:22.739+0000 7f1e06a7e640 1 -- 172.21.15.29:0/4190744623 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f1dec0a2a10 msgr2=0x7f1dec0a4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:01:22.741 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:22.739+0000 7f1e06a7e640 1 --2- 172.21.15.29:0/4190744623 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f1dec0a2a10 0x7f1dec0a4ed0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f1df8000c00 tx=0x7f1df80058d0 comp rx=0 tx=0).stop 2024-03-16T12:01:22.742 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:22.739+0000 7f1e06a7e640 1 -- 172.21.15.29:0/4190744623 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1e001012f0 msgr2=0x7f1e00199440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:01:22.742 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:22.739+0000 7f1e06a7e640 1 --2- 172.21.15.29:0/4190744623 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1e001012f0 0x7f1e00199440 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f1df0009fd0 tx=0x7f1df000ef90 comp rx=0 tx=0).stop 2024-03-16T12:01:22.742 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:22.739+0000 7f1e06a7e640 1 -- 172.21.15.29:0/4190744623 shutdown_connections 2024-03-16T12:01:22.742 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:22.739+0000 7f1e06a7e640 1 --2- 172.21.15.29:0/4190744623 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f1dec0a2a10 0x7f1dec0a4ed0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:01:22.742 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:22.739+0000 7f1e06a7e640 1 --2- 172.21.15.29:0/4190744623 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1e00105680 0x7f1e001a3f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:01:22.742 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:22.739+0000 7f1e06a7e640 1 --2- 172.21.15.29:0/4190744623 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1e001012f0 0x7f1e00199440 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:01:22.742 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:22.739+0000 7f1e06a7e640 1 --2- 172.21.15.29:0/4190744623 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1e00100930 0x7f1e00198f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:01:22.742 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:22.739+0000 7f1e06a7e640 1 -- 172.21.15.29:0/4190744623 >> 172.21.15.29:0/4190744623 conn(0x7f1e00077530 msgr2=0x7f1e000fe940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:01:22.742 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:22.739+0000 7f1e06a7e640 1 -- 172.21.15.29:0/4190744623 shutdown_connections 2024-03-16T12:01:22.742 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:22.739+0000 7f1e06a7e640 1 -- 172.21.15.29:0/4190744623 wait complete. 2024-03-16T12:01:22.802 INFO:tasks.workunit.client.0.smithi029.stdout: 1 1027088 9.62 MB/sec execute 427 sec latency 221.515 ms 2024-03-16T12:01:23.802 INFO:tasks.workunit.client.0.smithi029.stdout: 1 1028514 9.61 MB/sec execute 428 sec latency 48.075 ms 2024-03-16T12:01:24.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:01:24 smithi152 ceph-mon[36291]: pgmap v331: 97 pgs: 97 active+clean; 72 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 2.7 MiB/s wr, 169 op/s 2024-03-16T12:01:24.545 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:01:24 smithi154 ceph-mon[36320]: pgmap v331: 97 pgs: 97 active+clean; 72 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 2.7 MiB/s wr, 169 op/s 2024-03-16T12:01:24.588 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:01:24 smithi029 ceph-mon[31975]: pgmap v331: 97 pgs: 97 active+clean; 72 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 2.7 MiB/s wr, 169 op/s 2024-03-16T12:01:24.802 INFO:tasks.workunit.client.0.smithi029.stdout: 1 1030318 9.62 MB/sec execute 429 sec latency 110.754 ms 2024-03-16T12:01:25.802 INFO:tasks.workunit.client.0.smithi029.stdout: 1 1031343 9.60 MB/sec execute 430 sec latency 23.194 ms 2024-03-16T12:01:26.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:01:26 smithi152 ceph-mon[36291]: pgmap v332: 97 pgs: 97 active+clean; 72 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 2.7 MiB/s wr, 169 op/s 2024-03-16T12:01:26.545 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:01:26 smithi154 ceph-mon[36320]: pgmap v332: 97 pgs: 97 active+clean; 72 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 2.7 MiB/s wr, 169 op/s 2024-03-16T12:01:26.587 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:01:26 smithi029 ceph-mon[31975]: pgmap v332: 97 pgs: 97 active+clean; 72 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 2.7 MiB/s wr, 169 op/s 2024-03-16T12:01:26.802 INFO:tasks.workunit.client.0.smithi029.stdout: 1 1033499 9.61 MB/sec execute 431 sec latency 25.415 ms 2024-03-16T12:01:27.802 INFO:tasks.workunit.client.0.smithi029.stdout: 1 1034336 9.60 MB/sec execute 432 sec latency 20.633 ms 2024-03-16T12:01:28.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:01:28 smithi152 ceph-mon[36291]: pgmap v333: 97 pgs: 97 active+clean; 65 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 2.6 MiB/s wr, 247 op/s 2024-03-16T12:01:28.390 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:01:28 smithi029 ceph-mon[31975]: pgmap v333: 97 pgs: 97 active+clean; 65 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 2.6 MiB/s wr, 247 op/s 2024-03-16T12:01:28.545 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:01:28 smithi154 ceph-mon[36320]: pgmap v333: 97 pgs: 97 active+clean; 65 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 2.6 MiB/s wr, 247 op/s 2024-03-16T12:01:28.802 INFO:tasks.workunit.client.0.smithi029.stdout: 1 1035919 9.59 MB/sec execute 433 sec latency 26.228 ms 2024-03-16T12:01:29.802 INFO:tasks.workunit.client.0.smithi029.stdout: 1 1037408 9.60 MB/sec execute 434 sec latency 77.984 ms 2024-03-16T12:01:30.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:01:30 smithi152 ceph-mon[36291]: pgmap v334: 97 pgs: 97 active+clean; 65 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.2 MiB/s wr, 207 op/s 2024-03-16T12:01:30.545 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:01:30 smithi154 ceph-mon[36320]: pgmap v334: 97 pgs: 97 active+clean; 65 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.2 MiB/s wr, 207 op/s 2024-03-16T12:01:30.587 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:01:30 smithi029 ceph-mon[31975]: pgmap v334: 97 pgs: 97 active+clean; 65 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.2 MiB/s wr, 207 op/s 2024-03-16T12:01:30.802 INFO:tasks.workunit.client.0.smithi029.stdout: 1 1038447 9.58 MB/sec execute 435 sec latency 22.206 ms 2024-03-16T12:01:31.802 INFO:tasks.workunit.client.0.smithi029.stdout: 1 1040493 9.58 MB/sec execute 436 sec latency 30.002 ms 2024-03-16T12:01:32.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:01:32 smithi152 ceph-mon[36291]: pgmap v335: 97 pgs: 97 active+clean; 70 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 3.3 MiB/s wr, 274 op/s 2024-03-16T12:01:32.545 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:01:32 smithi154 ceph-mon[36320]: pgmap v335: 97 pgs: 97 active+clean; 70 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 3.3 MiB/s wr, 274 op/s 2024-03-16T12:01:32.587 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:01:32 smithi029 ceph-mon[31975]: pgmap v335: 97 pgs: 97 active+clean; 70 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 3.3 MiB/s wr, 274 op/s 2024-03-16T12:01:32.802 INFO:tasks.workunit.client.0.smithi029.stdout: 1 1041417 9.58 MB/sec execute 437 sec latency 23.455 ms 2024-03-16T12:01:33.802 INFO:tasks.workunit.client.0.smithi029.stdout: 1 1043075 9.57 MB/sec execute 438 sec latency 24.744 ms 2024-03-16T12:01:34.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:01:34 smithi152 ceph-mon[36291]: pgmap v336: 97 pgs: 97 active+clean; 70 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 2.3 MiB/s wr, 209 op/s 2024-03-16T12:01:34.545 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:01:34 smithi154 ceph-mon[36320]: pgmap v336: 97 pgs: 97 active+clean; 70 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 2.3 MiB/s wr, 209 op/s 2024-03-16T12:01:34.588 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:01:34 smithi029 ceph-mon[31975]: pgmap v336: 97 pgs: 97 active+clean; 70 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 2.3 MiB/s wr, 209 op/s 2024-03-16T12:01:34.802 INFO:tasks.workunit.client.0.smithi029.stdout: 1 1044527 9.57 MB/sec execute 439 sec latency 51.010 ms 2024-03-16T12:01:35.802 INFO:tasks.workunit.client.0.smithi029.stdout: 1 1045557 9.56 MB/sec execute 440 sec latency 27.955 ms 2024-03-16T12:01:35.831 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:35.831+0000 7f56eb910640 1 -- 172.21.15.29:0/3437189708 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f56e406a980 msgr2=0x7f56e410e620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:01:35.831 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:35.831+0000 7f56eb910640 1 --2- 172.21.15.29:0/3437189708 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f56e406a980 0x7f56e410e620 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f56dc009ca0 tx=0x7f56dc02c0f0 comp rx=0 tx=0).stop 2024-03-16T12:01:35.831 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:35.831+0000 7f56eb910640 1 -- 172.21.15.29:0/3437189708 shutdown_connections 2024-03-16T12:01:35.831 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:35.831+0000 7f56eb910640 1 --2- 172.21.15.29:0/3437189708 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f56e406a980 0x7f56e410e620 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:01:35.831 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:35.831+0000 7f56eb910640 1 --2- 172.21.15.29:0/3437189708 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f56e4069fe0 0x7f56e406a440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:01:35.831 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:35.831+0000 7f56eb910640 1 --2- 172.21.15.29:0/3437189708 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f56e41077d0 0x7f56e4107bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:01:35.831 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:35.831+0000 7f56eb910640 1 -- 172.21.15.29:0/3437189708 >> 172.21.15.29:0/3437189708 conn(0x7f56e40774b0 msgr2=0x7f56e40778c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:01:35.831 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:35.832+0000 7f56eb910640 1 -- 172.21.15.29:0/3437189708 shutdown_connections 2024-03-16T12:01:35.831 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:35.832+0000 7f56eb910640 1 -- 172.21.15.29:0/3437189708 wait complete. 2024-03-16T12:01:35.833 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:35.832+0000 7f56eb910640 1 Processor -- start 2024-03-16T12:01:35.833 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:35.832+0000 7f56eb910640 1 -- start start 2024-03-16T12:01:35.833 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:35.832+0000 7f56eb910640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f56e4069fe0 0x7f56e4197fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:01:35.833 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:35.832+0000 7f56eb910640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f56e406a980 0x7f56e41984f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:01:35.834 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:35.832+0000 7f56eb910640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f56e41077d0 0x7f56e419c870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:01:35.834 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:35.832+0000 7f56eb910640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f56e4111bb0 con 0x7f56e4069fe0 2024-03-16T12:01:35.834 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:35.832+0000 7f56eb910640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f56e4111a30 con 0x7f56e406a980 2024-03-16T12:01:35.834 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:35.832+0000 7f56eb910640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f56e4111d30 con 0x7f56e41077d0 2024-03-16T12:01:35.834 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:35.832+0000 7f56e96ac640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f56e4069fe0 0x7f56e4197fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:01:35.834 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:35.832+0000 7f56e96ac640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f56e4069fe0 0x7f56e4197fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50132/0 (socket says 172.21.15.29:50132) 2024-03-16T12:01:35.834 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:35.832+0000 7f56e96ac640 1 -- 172.21.15.29:0/1533195092 learned_addr learned my addr 172.21.15.29:0/1533195092 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T12:01:35.834 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:35.833+0000 7f56e9ead640 1 --2- 172.21.15.29:0/1533195092 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f56e41077d0 0x7f56e419c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:01:35.834 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:35.833+0000 7f56e8eab640 1 --2- 172.21.15.29:0/1533195092 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f56e406a980 0x7f56e41984f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:01:35.834 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:35.833+0000 7f56e96ac640 1 -- 172.21.15.29:0/1533195092 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f56e41077d0 msgr2=0x7f56e419c870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:01:35.834 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:35.833+0000 7f56e96ac640 1 --2- 172.21.15.29:0/1533195092 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f56e41077d0 0x7f56e419c870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:01:35.834 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:35.833+0000 7f56e96ac640 1 -- 172.21.15.29:0/1533195092 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f56e406a980 msgr2=0x7f56e41984f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:01:35.834 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:35.833+0000 7f56e96ac640 1 --2- 172.21.15.29:0/1533195092 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f56e406a980 0x7f56e41984f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:01:35.835 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:35.833+0000 7f56e96ac640 1 -- 172.21.15.29:0/1533195092 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56e419cfa0 con 0x7f56e4069fe0 2024-03-16T12:01:35.835 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:35.833+0000 7f56e96ac640 1 --2- 172.21.15.29:0/1533195092 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f56e4069fe0 0x7f56e4197fb0 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f56e000e9f0 tx=0x7f56e000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:01:35.835 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:35.833+0000 7f56da7fc640 1 -- 172.21.15.29:0/1533195092 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56e000cd80 con 0x7f56e4069fe0 2024-03-16T12:01:35.835 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:35.833+0000 7f56da7fc640 1 -- 172.21.15.29:0/1533195092 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f56e0005500 con 0x7f56e4069fe0 2024-03-16T12:01:35.835 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:35.833+0000 7f56da7fc640 1 -- 172.21.15.29:0/1533195092 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56e0010690 con 0x7f56e4069fe0 2024-03-16T12:01:35.835 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:35.833+0000 7f56eb910640 1 -- 172.21.15.29:0/1533195092 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f56e419d290 con 0x7f56e4069fe0 2024-03-16T12:01:35.835 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:35.833+0000 7f56eb910640 1 -- 172.21.15.29:0/1533195092 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f56e41a4b20 con 0x7f56e4069fe0 2024-03-16T12:01:35.841 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:35.835+0000 7f56da7fc640 1 -- 172.21.15.29:0/1533195092 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7f56e00046c0 con 0x7f56e4069fe0 2024-03-16T12:01:35.841 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:35.836+0000 7f56ee5da740 1 Processor -- start 2024-03-16T12:01:35.841 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:35.836+0000 7f56ee5da740 1 -- start start 2024-03-16T12:01:35.841 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:35.836+0000 7f56da7fc640 1 --2- 172.21.15.29:0/1533195092 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f56d00a2a10 0x7f56d00a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:01:35.841 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:35.836+0000 7f56ee5da740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e343e91130 0x55e343e91510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:01:35.841 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:35.836+0000 7f56e9ead640 1 --2- 172.21.15.29:0/1533195092 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f56d00a2a10 0x7f56d00a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:01:35.841 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:35.836+0000 7f56ee5da740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e343e91a50 0x55e343e9a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:01:35.841 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:35.837+0000 7f56ee5da740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e343e9a820 0x55e343e9cc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:01:35.841 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:35.837+0000 7f56ee5da740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55e343df5a40 con 0x55e343e91130 2024-03-16T12:01:35.841 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:35.837+0000 7f56ee5da740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x55e343d9e920 con 0x55e343e91a50 2024-03-16T12:01:35.841 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:35.837+0000 7f56ee5da740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55e343df1ae0 con 0x55e343e9a820 2024-03-16T12:01:35.841 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:35.837+0000 7f56da7fc640 1 -- 172.21.15.29:0/1533195092 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 8556+0+0 (secure 0 0 0) 0x7f56e00cf270 con 0x7f56e4069fe0 2024-03-16T12:01:35.841 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:35.837+0000 7f56e96ac640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e343e91130 0x55e343e91510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:01:35.842 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:35.837+0000 7f56e96ac640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e343e91130 0x55e343e91510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50140/0 (socket says 172.21.15.29:50140) 2024-03-16T12:01:35.842 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:35.837+0000 7f56e96ac640 1 -- 172.21.15.29:0/993519958 learned_addr learned my addr 172.21.15.29:0/993519958 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T12:01:35.842 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:35.837+0000 7f56e8eab640 1 --2- 172.21.15.29:0/993519958 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e343e91a50 0x55e343e9a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:01:35.842 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:35.837+0000 7f56e9ead640 1 --2- 172.21.15.29:0/993519958 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e343e9a820 0x55e343e9cc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:01:35.842 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:35.837+0000 7f56e96ac640 1 -- 172.21.15.29:0/993519958 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e343e9a820 msgr2=0x55e343e9cc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:01:35.842 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:35.837+0000 7f56e96ac640 1 --2- 172.21.15.29:0/993519958 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e343e9a820 0x55e343e9cc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:01:35.842 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:35.837+0000 7f56e96ac640 1 -- 172.21.15.29:0/993519958 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e343e91a50 msgr2=0x55e343e9a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:01:35.842 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:35.837+0000 7f56e96ac640 1 --2- 172.21.15.29:0/993519958 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e343e91a50 0x55e343e9a2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:01:35.842 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:35.837+0000 7f56e96ac640 1 -- 172.21.15.29:0/993519958 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e343db6460 con 0x55e343e91130 2024-03-16T12:01:35.842 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:35.837+0000 7f56e9ead640 1 --2- 172.21.15.29:0/1533195092 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f56d00a2a10 0x7f56d00a4ed0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f56dc004af0 tx=0x7f56dc032e70 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:01:35.842 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:35.837+0000 7f56e96ac640 1 --2- 172.21.15.29:0/993519958 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e343e91130 0x55e343e91510 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f56e0098220 tx=0x7f56e009b6e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd73e98494046590 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:01:35.842 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:35.837+0000 7f56cf7fe640 1 -- 172.21.15.29:0/993519958 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56e0056600 con 0x55e343e91130 2024-03-16T12:01:35.842 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:35.837+0000 7f56cf7fe640 1 -- 172.21.15.29:0/993519958 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f56e009bf50 con 0x55e343e91130 2024-03-16T12:01:35.842 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:35.837+0000 7f56cf7fe640 1 -- 172.21.15.29:0/993519958 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56e0097ec0 con 0x55e343e91130 2024-03-16T12:01:35.843 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:35.838+0000 7f56ee5da740 1 -- 172.21.15.29:0/993519958 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e343e91130 msgr2=0x55e343e91510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:01:35.843 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:35.838+0000 7f56ee5da740 1 --2- 172.21.15.29:0/993519958 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e343e91130 0x55e343e91510 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f56e0098220 tx=0x7f56e009b6e0 comp rx=0 tx=0).stop 2024-03-16T12:01:35.843 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:35.838+0000 7f56ee5da740 1 -- 172.21.15.29:0/993519958 shutdown_connections 2024-03-16T12:01:35.843 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:35.838+0000 7f56ee5da740 1 --2- 172.21.15.29:0/993519958 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e343e9a820 0x55e343e9cc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:01:35.843 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:35.838+0000 7f56ee5da740 1 --2- 172.21.15.29:0/993519958 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e343e91a50 0x55e343e9a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:01:35.843 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:35.838+0000 7f56ee5da740 1 --2- 172.21.15.29:0/993519958 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e343e91130 0x55e343e91510 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:01:35.843 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:35.838+0000 7f56ee5da740 1 -- 172.21.15.29:0/993519958 >> 172.21.15.29:0/993519958 conn(0x55e343e8ba50 msgr2=0x55e343e8be30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:01:35.843 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:35.838+0000 7f56ee5da740 1 -- 172.21.15.29:0/993519958 shutdown_connections 2024-03-16T12:01:35.843 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:35.838+0000 7f56ee5da740 1 -- 172.21.15.29:0/993519958 wait complete. 2024-03-16T12:01:35.843 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:35.839+0000 7f56ee5da740 1 -- start start 2024-03-16T12:01:35.843 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:35.839+0000 7f56ee5da740 1 Processor -- start 2024-03-16T12:01:35.843 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:35.839+0000 7f56ee5da740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e343e91130 0x55e343e97910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:01:35.843 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:35.839+0000 7f56ee5da740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e343e91a50 0x55e343e97e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:01:35.843 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:35.839+0000 7f56ee5da740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e343e9a820 0x55e343e98390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:01:35.843 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:35.839+0000 7f56ee5da740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55e343e9d3f0 con 0x55e343e91130 2024-03-16T12:01:35.844 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:35.839+0000 7f56ee5da740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x55e343db1d70 con 0x55e343e91a50 2024-03-16T12:01:35.844 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:35.839+0000 7f56ee5da740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55e343e9d570 con 0x55e343e9a820 2024-03-16T12:01:35.844 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:35.839+0000 7f56ee5da740 10 client.0 _ll_register_callbacks cb 0x55e343b09a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T12:01:35.844 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:35.839+0000 7f56e96ac640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e343e91130 0x55e343e97910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:01:35.844 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:35.839+0000 7f56e96ac640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e343e91130 0x55e343e97910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50152/0 (socket says 172.21.15.29:50152) 2024-03-16T12:01:35.844 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:35.839+0000 7f56e96ac640 1 -- 172.21.15.29:0/844629739 learned_addr learned my addr 172.21.15.29:0/844629739 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T12:01:35.844 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:35.839+0000 7f56e8eab640 1 --2- 172.21.15.29:0/844629739 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e343e91a50 0x55e343e97e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:01:35.844 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:35.839+0000 7f56e9ead640 1 --2- 172.21.15.29:0/844629739 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e343e9a820 0x55e343e98390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:01:35.844 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:35.840+0000 7f56e96ac640 1 -- 172.21.15.29:0/844629739 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e343e9a820 msgr2=0x55e343e98390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:01:35.844 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:35.840+0000 7f56e96ac640 1 --2- 172.21.15.29:0/844629739 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e343e9a820 0x55e343e98390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:01:35.844 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:35.840+0000 7f56e96ac640 1 -- 172.21.15.29:0/844629739 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e343e91a50 msgr2=0x55e343e97e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:01:35.844 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:35.840+0000 7f56e96ac640 1 --2- 172.21.15.29:0/844629739 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e343e91a50 0x55e343e97e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:01:35.844 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:35.840+0000 7f56e96ac640 1 -- 172.21.15.29:0/844629739 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e343b0a210 con 0x55e343e91130 2024-03-16T12:01:35.844 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:35.840+0000 7f56e96ac640 1 --2- 172.21.15.29:0/844629739 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e343e91130 0x55e343e97910 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f56e009b550 tx=0x7f56e00980f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e2c52ecc74054a5 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:01:35.845 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:35.840+0000 7f56ccff9640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-16T12:01:35.845 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:35.840+0000 7f56ccff9640 1 -- 172.21.15.29:0/844629739 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56e00ccdd0 con 0x55e343e91130 2024-03-16T12:01:35.845 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:35.840+0000 7f56ccff9640 1 -- 172.21.15.29:0/844629739 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 8556+0+0 (secure 0 0 0) 0x7f56e00561d0 con 0x55e343e91130 2024-03-16T12:01:35.845 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:35.840+0000 7f56ee5da740 1 -- 172.21.15.29:0/844629739 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f56e009bf50 con 0x55e343e91130 2024-03-16T12:01:35.845 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:35.840+0000 7f56ccff9640 1 -- 172.21.15.29:0/844629739 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f56e0056410 con 0x55e343e91130 2024-03-16T12:01:35.845 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:35.840+0000 7f56ee5da740 10 client.16290 fetch_fsmap learned FSMap version 28 2024-03-16T12:01:35.845 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:35.840+0000 7f56ee5da740 1 -- 172.21.15.29:0/844629739 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55e343bd6620 con 0x55e343e91130 2024-03-16T12:01:35.845 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:35.840+0000 7f56ccff9640 1 -- 172.21.15.29:0/844629739 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f56e009bf50 con 0x55e343e91130 2024-03-16T12:01:35.845 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:35.840+0000 7f56ccff9640 1 -- 172.21.15.29:0/844629739 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 28) ==== 8338+0+0 (secure 0 0 0) 0x7f56e009cb80 con 0x55e343e91130 2024-03-16T12:01:35.845 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:35.840+0000 7f56ee5da740 10 client.16290 fetch_fsmap finished waiting for FSMap version 28 2024-03-16T12:01:35.845 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:35.840+0000 7f56ee5da740 10 client.16290 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T12:01:35.845 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:35.840+0000 7f56ee5da740 20 client.16290 populate_metadata read hostname 'smithi029' 2024-03-16T12:01:35.845 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:35.841+0000 7f56ee5da740 1 --2- 172.21.15.29:0/844629739 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55e343eaa890 0x55e343eacc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:01:35.845 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:35.841+0000 7f56ee5da740 4 client.16290 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T12:01:35.845 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:35.841+0000 7f56ee5da740 1 -- 172.21.15.29:0/844629739 --> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e343e02f80 con 0x55e343eaa890 2024-03-16T12:01:35.846 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:35.841+0000 7f56e8eab640 1 --2- 172.21.15.29:0/844629739 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55e343eaa890 0x55e343eacc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:01:35.846 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:35.841+0000 7f56e8eab640 1 --2- 172.21.15.29:0/844629739 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55e343eaa890 0x55e343eacc80 crc :-1 s=READY pgs=134 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=50c315f1aca045bf server_cookie=220bcfd462c43580 in_seq=0 out_seq=0 2024-03-16T12:01:35.846 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:35.841+0000 7f56ccff9640 10 client.16290 ms_handle_connect on v2:172.21.15.152:6838/1548731833 2024-03-16T12:01:35.847 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:35.843+0000 7f56ccff9640 1 -- 172.21.15.29:0/844629739 <== mds.0 v2:172.21.15.152:6838/1548731833 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55e343e02f80 con 0x55e343eaa890 2024-03-16T12:01:35.847 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:35.843+0000 7f56ccff9640 10 client.16290 handle_command_reply: tid=0 2024-03-16T12:01:35.847 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:35.843+0000 7f56ee5da740 1 client.16290 shutdown 2024-03-16T12:01:35.847 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:35.843+0000 7f56ee5da740 1 -- 172.21.15.29:0/844629739 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e343e91130 msgr2=0x55e343e97910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:01:35.847 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:35.843+0000 7f56ee5da740 1 --2- 172.21.15.29:0/844629739 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e343e91130 0x55e343e97910 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f56e009b550 tx=0x7f56e00980f0 comp rx=0 tx=0).stop 2024-03-16T12:01:35.847 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:35.843+0000 7f56ee5da740 1 -- 172.21.15.29:0/844629739 shutdown_connections 2024-03-16T12:01:35.847 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:35.843+0000 7f56ee5da740 1 --2- 172.21.15.29:0/844629739 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e343e9a820 0x55e343e98390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:01:35.847 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:35.843+0000 7f56ee5da740 1 --2- 172.21.15.29:0/844629739 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55e343eaa890 0x55e343eacc80 crc :-1 s=READY pgs=134 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:01:35.847 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:35.843+0000 7f56ee5da740 1 --2- 172.21.15.29:0/844629739 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e343e91a50 0x55e343e97e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:01:35.847 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:35.843+0000 7f56ee5da740 1 --2- 172.21.15.29:0/844629739 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e343e91130 0x55e343e97910 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:01:35.848 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:35.843+0000 7f56ccff9640 0 client.16290 ms_handle_reset on v2:172.21.15.152:6838/1548731833 2024-03-16T12:01:35.848 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:35.843+0000 7f56ee5da740 1 -- 172.21.15.29:0/844629739 >> 172.21.15.29:0/844629739 conn(0x55e343e8ba50 msgr2=0x55e343e8d7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:01:35.848 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:35.843+0000 7f56ee5da740 1 -- 172.21.15.29:0/844629739 shutdown_connections 2024-03-16T12:01:35.848 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:35.843+0000 7f56ee5da740 1 -- 172.21.15.29:0/844629739 wait complete. 2024-03-16T12:01:35.848 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:35.844+0000 7f56ee5da740 20 client.16290 trim_cache size 0 max 16384 2024-03-16T12:01:35.863 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:35.859+0000 7f56ee5da740 1 Processor -- start 2024-03-16T12:01:35.863 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:35.859+0000 7f56ee5da740 1 -- start start 2024-03-16T12:01:35.863 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:35.859+0000 7f56ee5da740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e343e91130 0x55e343e9a040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:01:35.863 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:35.859+0000 7f56ee5da740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e343e91a50 0x55e343eafca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:01:35.864 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:35.859+0000 7f56ee5da740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e343e9a820 0x55e343e94b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:01:35.864 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:35.859+0000 7f56ee5da740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f56e41afda0 con 0x55e343e9a820 2024-03-16T12:01:35.864 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:35.859+0000 7f56ee5da740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f56e41ad670 con 0x55e343e91130 2024-03-16T12:01:35.864 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:35.859+0000 7f56ee5da740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f56e41ab4e0 con 0x55e343e91a50 2024-03-16T12:01:35.864 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:35.859+0000 7f56e9ead640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e343e9a820 0x55e343e94b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:01:35.864 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:35.859+0000 7f56e9ead640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e343e9a820 0x55e343e94b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50168/0 (socket says 172.21.15.29:50168) 2024-03-16T12:01:35.864 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:35.859+0000 7f56e9ead640 1 -- 172.21.15.29:0/1157429028 learned_addr learned my addr 172.21.15.29:0/1157429028 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T12:01:35.864 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:35.859+0000 7f56e8eab640 1 --2- 172.21.15.29:0/1157429028 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e343e91a50 0x55e343eafca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:01:35.864 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:35.859+0000 7f56e96ac640 1 --2- 172.21.15.29:0/1157429028 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e343e91130 0x55e343e9a040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:01:35.864 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:35.860+0000 7f56e9ead640 1 -- 172.21.15.29:0/1157429028 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e343e91a50 msgr2=0x55e343eafca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:01:35.864 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:35.860+0000 7f56e9ead640 1 --2- 172.21.15.29:0/1157429028 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e343e91a50 0x55e343eafca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:01:35.864 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:35.860+0000 7f56e9ead640 1 -- 172.21.15.29:0/1157429028 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e343e91130 msgr2=0x55e343e9a040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:01:35.864 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:35.860+0000 7f56e9ead640 1 --2- 172.21.15.29:0/1157429028 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e343e91130 0x55e343e9a040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:01:35.865 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:35.860+0000 7f56e9ead640 1 -- 172.21.15.29:0/1157429028 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e343e934f0 con 0x55e343e9a820 2024-03-16T12:01:35.865 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:35.860+0000 7f56e96ac640 1 --2- 172.21.15.29:0/1157429028 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e343e91130 0x55e343e9a040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T12:01:35.865 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:35.860+0000 7f56e9ead640 1 --2- 172.21.15.29:0/1157429028 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e343e9a820 0x55e343e94b90 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f56dc037040 tx=0x7f56dc045000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79c9090293baa763 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:01:35.865 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:35.860+0000 7f56ceffd640 1 -- 172.21.15.29:0/1157429028 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56dc039040 con 0x55e343e9a820 2024-03-16T12:01:35.865 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:35.860+0000 7f56ceffd640 1 -- 172.21.15.29:0/1157429028 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f56dc003d70 con 0x55e343e9a820 2024-03-16T12:01:35.865 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:35.860+0000 7f56ceffd640 1 -- 172.21.15.29:0/1157429028 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56dc04e9c0 con 0x55e343e9a820 2024-03-16T12:01:35.865 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:35.861+0000 7f56ee5da740 1 -- 172.21.15.29:0/1157429028 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e343e9a820 msgr2=0x55e343e94b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:01:35.865 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:35.861+0000 7f56ee5da740 1 --2- 172.21.15.29:0/1157429028 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e343e9a820 0x55e343e94b90 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f56dc037040 tx=0x7f56dc045000 comp rx=0 tx=0).stop 2024-03-16T12:01:35.865 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:35.861+0000 7f56ee5da740 1 -- 172.21.15.29:0/1157429028 shutdown_connections 2024-03-16T12:01:35.865 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:35.861+0000 7f56ee5da740 1 --2- 172.21.15.29:0/1157429028 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e343e9a820 0x55e343e94b90 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:01:35.865 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:35.861+0000 7f56ee5da740 1 --2- 172.21.15.29:0/1157429028 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e343e91a50 0x55e343eafca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:01:35.865 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:35.861+0000 7f56ee5da740 1 --2- 172.21.15.29:0/1157429028 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e343e91130 0x55e343e9a040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:01:35.865 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:35.861+0000 7f56ee5da740 1 -- 172.21.15.29:0/1157429028 >> 172.21.15.29:0/1157429028 conn(0x55e343e8ba50 msgr2=0x55e343ea6050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:01:35.865 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:35.861+0000 7f56ee5da740 1 -- 172.21.15.29:0/1157429028 shutdown_connections 2024-03-16T12:01:35.865 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:35.861+0000 7f56ee5da740 1 -- 172.21.15.29:0/1157429028 wait complete. 2024-03-16T12:01:35.866 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:35.861+0000 7f56ee5da740 1 -- start start 2024-03-16T12:01:35.866 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:35.861+0000 7f56ee5da740 1 Processor -- start 2024-03-16T12:01:35.866 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:35.861+0000 7f56ee5da740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e343e91130 0x55e343eaf8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:01:35.866 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:35.861+0000 7f56ee5da740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e343e91a50 0x55e343eafde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:01:35.866 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:35.861+0000 7f56ee5da740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e343e9a820 0x55e343eadd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:01:35.866 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:35.861+0000 7f56ee5da740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f56e41abb30 con 0x55e343e9a820 2024-03-16T12:01:35.866 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:35.862+0000 7f56ee5da740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f56e41acfa0 con 0x55e343e91a50 2024-03-16T12:01:35.866 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:35.862+0000 7f56ee5da740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f56e41ac200 con 0x55e343e91130 2024-03-16T12:01:35.866 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:35.862+0000 7f56ee5da740 10 client.0 _ll_register_callbacks cb 0x7f56d40097a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T12:01:35.866 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:35.862+0000 7f56e9ead640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e343e9a820 0x55e343eadd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:01:35.866 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:35.862+0000 7f56e9ead640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e343e9a820 0x55e343eadd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50170/0 (socket says 172.21.15.29:50170) 2024-03-16T12:01:35.866 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:35.862+0000 7f56e9ead640 1 -- 172.21.15.29:0/185096357 learned_addr learned my addr 172.21.15.29:0/185096357 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T12:01:35.866 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:35.862+0000 7f56e8eab640 1 --2- 172.21.15.29:0/185096357 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e343e91a50 0x55e343eafde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:01:35.866 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:35.862+0000 7f56e96ac640 1 --2- 172.21.15.29:0/185096357 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e343e91130 0x55e343eaf8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:01:35.867 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:35.862+0000 7f56e9ead640 1 -- 172.21.15.29:0/185096357 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e343e91130 msgr2=0x55e343eaf8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:01:35.867 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:35.862+0000 7f56e9ead640 1 --2- 172.21.15.29:0/185096357 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e343e91130 0x55e343eaf8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:01:35.867 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:35.862+0000 7f56e9ead640 1 -- 172.21.15.29:0/185096357 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e343e91a50 msgr2=0x55e343eafde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:01:35.867 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:35.862+0000 7f56e9ead640 1 --2- 172.21.15.29:0/185096357 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e343e91a50 0x55e343eafde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:01:35.867 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:35.862+0000 7f56e9ead640 1 -- 172.21.15.29:0/185096357 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e343eae4d0 con 0x55e343e9a820 2024-03-16T12:01:35.867 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:35.862+0000 7f56e9ead640 1 --2- 172.21.15.29:0/185096357 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e343e9a820 0x55e343eadd40 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f56dc02c630 tx=0x7f56dc005c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b385cd70dcd0fb00 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:01:35.867 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:35.862+0000 7f56ccff9640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-16T12:01:35.867 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:35.862+0000 7f56ccff9640 1 -- 172.21.15.29:0/185096357 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56dc039040 con 0x55e343e9a820 2024-03-16T12:01:35.867 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:35.862+0000 7f56ccff9640 1 -- 172.21.15.29:0/185096357 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 8556+0+0 (secure 0 0 0) 0x7f56dc04eb50 con 0x55e343e9a820 2024-03-16T12:01:35.867 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:35.862+0000 7f56ee5da740 1 -- 172.21.15.29:0/185096357 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f56dc003d70 con 0x55e343e9a820 2024-03-16T12:01:35.867 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:35.862+0000 7f56ccff9640 1 -- 172.21.15.29:0/185096357 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f56dc068050 con 0x55e343e9a820 2024-03-16T12:01:35.867 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:35.862+0000 7f56ee5da740 10 client.16296 fetch_fsmap learned FSMap version 28 2024-03-16T12:01:35.867 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:35.862+0000 7f56ee5da740 1 -- 172.21.15.29:0/185096357 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55e343f11290 con 0x55e343e9a820 2024-03-16T12:01:35.867 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:35.863+0000 7f56ccff9640 1 -- 172.21.15.29:0/185096357 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f56dc003d70 con 0x55e343e9a820 2024-03-16T12:01:35.868 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:35.863+0000 7f56ccff9640 1 -- 172.21.15.29:0/185096357 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 28) ==== 8338+0+0 (secure 0 0 0) 0x7f56dc034440 con 0x55e343e9a820 2024-03-16T12:01:35.868 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:35.863+0000 7f56ee5da740 10 client.16296 fetch_fsmap finished waiting for FSMap version 28 2024-03-16T12:01:35.868 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:35.863+0000 7f56ee5da740 10 client.16296 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T12:01:35.868 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:35.863+0000 7f56ee5da740 20 client.16296 populate_metadata read hostname 'smithi029' 2024-03-16T12:01:35.868 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:35.863+0000 7f56ee5da740 1 --2- 172.21.15.29:0/185096357 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55e343f11a40 0x55e343f13e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:01:35.868 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:35.863+0000 7f56ee5da740 4 client.16296 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-16T12:01:35.868 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:35.863+0000 7f56ee5da740 1 -- 172.21.15.29:0/185096357 --> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55e343dfd2d0 con 0x55e343f11a40 2024-03-16T12:01:35.868 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:35.863+0000 7f56e96ac640 1 --2- 172.21.15.29:0/185096357 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55e343f11a40 0x55e343f13e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:01:35.868 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:35.863+0000 7f56ccff9640 10 client.16296 ms_handle_connect on v2:172.21.15.152:6838/1548731833 2024-03-16T12:01:35.868 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:35.863+0000 7f56e96ac640 1 --2- 172.21.15.29:0/185096357 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55e343f11a40 0x55e343f13e60 crc :-1 s=READY pgs=135 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=29f4c52861a5658b server_cookie=16a2fe48f33d34f9 in_seq=0 out_seq=0 2024-03-16T12:01:36.055 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:36.055+0000 7f56ccff9640 1 -- 172.21.15.29:0/185096357 <== mds.0 v2:172.21.15.152:6838/1548731833 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55e343dfd2d0 con 0x55e343f11a40 2024-03-16T12:01:36.055 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:36.055+0000 7f56ccff9640 10 client.16296 handle_command_reply: tid=0 2024-03-16T12:01:36.055 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:36.055+0000 7f56ee5da740 1 client.16296 shutdown 2024-03-16T12:01:36.055 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:36.055+0000 7f56ee5da740 1 -- 172.21.15.29:0/185096357 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e343e9a820 msgr2=0x55e343eadd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:01:36.055 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:36.055+0000 7f56ee5da740 1 --2- 172.21.15.29:0/185096357 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e343e9a820 0x55e343eadd40 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f56dc02c630 tx=0x7f56dc005c50 comp rx=0 tx=0).stop 2024-03-16T12:01:36.055 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:36.055+0000 7f56ee5da740 1 -- 172.21.15.29:0/185096357 shutdown_connections 2024-03-16T12:01:36.055 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:36.055+0000 7f56ee5da740 1 --2- 172.21.15.29:0/185096357 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e343e9a820 0x55e343eadd40 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:01:36.055 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:36.055+0000 7f56ee5da740 1 --2- 172.21.15.29:0/185096357 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55e343f11a40 0x55e343f13e60 crc :-1 s=READY pgs=135 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:01:36.055 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:36.055+0000 7f56ee5da740 1 --2- 172.21.15.29:0/185096357 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e343e91a50 0x55e343eafde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:01:36.055 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:36.055+0000 7f56ccff9640 0 client.16296 ms_handle_reset on v2:172.21.15.152:6838/1548731833 2024-03-16T12:01:36.055 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:36.055+0000 7f56ee5da740 1 --2- 172.21.15.29:0/185096357 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e343e91130 0x55e343eaf8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:01:36.055 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:36.055+0000 7f56ee5da740 1 -- 172.21.15.29:0/185096357 >> 172.21.15.29:0/185096357 conn(0x55e343e8ba50 msgr2=0x55e343ea57c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:01:36.056 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:36.055+0000 7f56ee5da740 1 -- 172.21.15.29:0/185096357 shutdown_connections 2024-03-16T12:01:36.056 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:36.055+0000 7f56ee5da740 1 -- 172.21.15.29:0/185096357 wait complete. 2024-03-16T12:01:36.056 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:36.055+0000 7f56ee5da740 20 client.16296 trim_cache size 0 max 16384 2024-03-16T12:01:36.056 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:{ 2024-03-16T12:01:36.056 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "message": "", 2024-03-16T12:01:36.056 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "return_code": 0 2024-03-16T12:01:36.056 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:} 2024-03-16T12:01:36.057 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:36.056+0000 7f56eb910640 1 -- 172.21.15.29:0/1533195092 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f56d00a2a10 msgr2=0x7f56d00a4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:01:36.057 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:36.056+0000 7f56eb910640 1 --2- 172.21.15.29:0/1533195092 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f56d00a2a10 0x7f56d00a4ed0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f56dc004af0 tx=0x7f56dc032e70 comp rx=0 tx=0).stop 2024-03-16T12:01:36.057 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:36.056+0000 7f56eb910640 1 -- 172.21.15.29:0/1533195092 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f56e4069fe0 msgr2=0x7f56e4197fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:01:36.057 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:36.056+0000 7f56eb910640 1 --2- 172.21.15.29:0/1533195092 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f56e4069fe0 0x7f56e4197fb0 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f56e000e9f0 tx=0x7f56e000edf0 comp rx=0 tx=0).stop 2024-03-16T12:01:36.057 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:36.056+0000 7f56eb910640 1 -- 172.21.15.29:0/1533195092 shutdown_connections 2024-03-16T12:01:36.057 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:36.056+0000 7f56eb910640 1 --2- 172.21.15.29:0/1533195092 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f56d00a2a10 0x7f56d00a4ed0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:01:36.057 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:36.056+0000 7f56eb910640 1 --2- 172.21.15.29:0/1533195092 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f56e41077d0 0x7f56e419c870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:01:36.057 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:36.056+0000 7f56eb910640 1 --2- 172.21.15.29:0/1533195092 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f56e406a980 0x7f56e41984f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:01:36.057 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:36.056+0000 7f56eb910640 1 --2- 172.21.15.29:0/1533195092 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f56e4069fe0 0x7f56e4197fb0 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:01:36.057 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:36.056+0000 7f56eb910640 1 -- 172.21.15.29:0/1533195092 >> 172.21.15.29:0/1533195092 conn(0x7f56e40774b0 msgr2=0x7f56e40fe430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:01:36.058 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:36.056+0000 7f56eb910640 1 -- 172.21.15.29:0/1533195092 shutdown_connections 2024-03-16T12:01:36.058 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:36.056+0000 7f56eb910640 1 -- 172.21.15.29:0/1533195092 wait complete. 2024-03-16T12:01:36.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:01:36 smithi152 ceph-mon[36291]: pgmap v337: 97 pgs: 97 active+clean; 70 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.3 MiB/s wr, 206 op/s 2024-03-16T12:01:36.545 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:01:36 smithi154 ceph-mon[36320]: pgmap v337: 97 pgs: 97 active+clean; 70 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.3 MiB/s wr, 206 op/s 2024-03-16T12:01:36.588 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:01:36 smithi029 ceph-mon[31975]: pgmap v337: 97 pgs: 97 active+clean; 70 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.3 MiB/s wr, 206 op/s 2024-03-16T12:01:36.803 INFO:tasks.workunit.client.0.smithi029.stdout: 1 1047011 9.55 MB/sec execute 441 sec latency 123.284 ms 2024-03-16T12:01:37.803 INFO:tasks.workunit.client.0.smithi029.stdout: 1 1048216 9.55 MB/sec execute 442 sec latency 26.923 ms 2024-03-16T12:01:38.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:01:38 smithi152 ceph-mon[36291]: pgmap v338: 97 pgs: 97 active+clean; 75 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 3.2 MiB/s wr, 308 op/s 2024-03-16T12:01:38.545 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:01:38 smithi154 ceph-mon[36320]: pgmap v338: 97 pgs: 97 active+clean; 75 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 3.2 MiB/s wr, 308 op/s 2024-03-16T12:01:38.587 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:01:38 smithi029 ceph-mon[31975]: pgmap v338: 97 pgs: 97 active+clean; 75 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 3.2 MiB/s wr, 308 op/s 2024-03-16T12:01:38.803 INFO:tasks.workunit.client.0.smithi029.stdout: 1 1049613 9.54 MB/sec execute 443 sec latency 21.593 ms 2024-03-16T12:01:39.803 INFO:tasks.workunit.client.0.smithi029.stdout: 1 1051515 9.55 MB/sec execute 444 sec latency 63.392 ms 2024-03-16T12:01:40.377 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:01:40 smithi152 ceph-mon[36291]: pgmap v339: 97 pgs: 97 active+clean; 75 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.2 MiB/s wr, 172 op/s 2024-03-16T12:01:40.545 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:01:40 smithi154 ceph-mon[36320]: pgmap v339: 97 pgs: 97 active+clean; 75 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.2 MiB/s wr, 172 op/s 2024-03-16T12:01:40.587 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:01:40 smithi029 ceph-mon[31975]: pgmap v339: 97 pgs: 97 active+clean; 75 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.2 MiB/s wr, 172 op/s 2024-03-16T12:01:40.803 INFO:tasks.workunit.client.0.smithi029.stdout: 1 1052219 9.53 MB/sec execute 445 sec latency 30.564 ms 2024-03-16T12:01:41.803 INFO:tasks.workunit.client.0.smithi029.stdout: 1 1054090 9.53 MB/sec execute 446 sec latency 23.500 ms 2024-03-16T12:01:42.545 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:01:42 smithi154 ceph-mon[36320]: pgmap v340: 97 pgs: 97 active+clean; 67 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 2.8 MiB/s wr, 256 op/s 2024-03-16T12:01:42.588 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:01:42 smithi029 ceph-mon[31975]: pgmap v340: 97 pgs: 97 active+clean; 67 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 2.8 MiB/s wr, 256 op/s 2024-03-16T12:01:42.627 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:01:42 smithi152 ceph-mon[36291]: pgmap v340: 97 pgs: 97 active+clean; 67 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 2.8 MiB/s wr, 256 op/s 2024-03-16T12:01:42.803 INFO:tasks.workunit.client.0.smithi029.stdout: 1 1055399 9.53 MB/sec execute 447 sec latency 26.537 ms 2024-03-16T12:01:43.803 INFO:tasks.workunit.client.0.smithi029.stdout: 1 1056759 9.52 MB/sec execute 448 sec latency 21.601 ms 2024-03-16T12:01:44.428 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:01:44 smithi152 ceph-mon[36291]: pgmap v341: 97 pgs: 97 active+clean; 68 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.0 MiB/s wr, 204 op/s 2024-03-16T12:01:44.545 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:01:44 smithi154 ceph-mon[36320]: pgmap v341: 97 pgs: 97 active+clean; 68 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.0 MiB/s wr, 204 op/s 2024-03-16T12:01:44.587 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:01:44 smithi029 ceph-mon[31975]: pgmap v341: 97 pgs: 97 active+clean; 68 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.0 MiB/s wr, 204 op/s 2024-03-16T12:01:44.803 INFO:tasks.workunit.client.0.smithi029.stdout: 1 1058610 9.53 MB/sec execute 449 sec latency 57.979 ms 2024-03-16T12:01:45.803 INFO:tasks.workunit.client.0.smithi029.stdout: 1 1059388 9.51 MB/sec execute 450 sec latency 20.951 ms 2024-03-16T12:01:46.545 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:01:46 smithi154 ceph-mon[36320]: pgmap v342: 97 pgs: 97 active+clean; 68 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 1.9 MiB/s wr, 201 op/s 2024-03-16T12:01:46.587 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:01:46 smithi029 ceph-mon[31975]: pgmap v342: 97 pgs: 97 active+clean; 68 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 1.9 MiB/s wr, 201 op/s 2024-03-16T12:01:46.627 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:01:46 smithi152 ceph-mon[36291]: pgmap v342: 97 pgs: 97 active+clean; 68 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 1.9 MiB/s wr, 201 op/s 2024-03-16T12:01:46.803 INFO:tasks.workunit.client.0.smithi029.stdout: 1 1061497 9.52 MB/sec execute 451 sec latency 20.751 ms 2024-03-16T12:01:47.804 INFO:tasks.workunit.client.0.smithi029.stdout: 1 1062608 9.51 MB/sec execute 452 sec latency 25.139 ms 2024-03-16T12:01:48.545 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:01:48 smithi154 ceph-mon[36320]: pgmap v343: 97 pgs: 97 active+clean; 73 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.0 MiB/s wr, 271 op/s 2024-03-16T12:01:48.588 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:01:48 smithi029 ceph-mon[31975]: pgmap v343: 97 pgs: 97 active+clean; 73 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.0 MiB/s wr, 271 op/s 2024-03-16T12:01:48.628 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:01:48 smithi152 ceph-mon[36291]: pgmap v343: 97 pgs: 97 active+clean; 73 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.0 MiB/s wr, 271 op/s 2024-03-16T12:01:48.803 INFO:tasks.workunit.client.0.smithi029.stdout: 1 1064212 9.50 MB/sec execute 453 sec latency 22.666 ms 2024-03-16T12:01:49.149 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:49.147+0000 7f072ca59640 1 -- 172.21.15.29:0/35117285 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f07281006a0 msgr2=0x7f072810e670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:01:49.149 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:49.147+0000 7f072ca59640 1 --2- 172.21.15.29:0/35117285 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f07281006a0 0x7f072810e670 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f0720009cd0 tx=0x7f072002c0f0 comp rx=0 tx=0).stop 2024-03-16T12:01:49.149 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:49.148+0000 7f072ca59640 1 -- 172.21.15.29:0/35117285 shutdown_connections 2024-03-16T12:01:49.149 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:49.148+0000 7f072ca59640 1 --2- 172.21.15.29:0/35117285 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f07281006a0 0x7f072810e670 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:01:49.149 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:49.148+0000 7f072ca59640 1 --2- 172.21.15.29:0/35117285 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f07280ffce0 0x7f0728100160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:01:49.149 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:49.148+0000 7f072ca59640 1 --2- 172.21.15.29:0/35117285 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f07281067c0 0x7f0728106ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:01:49.149 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:49.148+0000 7f072ca59640 1 -- 172.21.15.29:0/35117285 >> 172.21.15.29:0/35117285 conn(0x7f0728076ca0 msgr2=0x7f07280fdff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:01:49.149 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:49.148+0000 7f072ca59640 1 -- 172.21.15.29:0/35117285 shutdown_connections 2024-03-16T12:01:49.149 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:49.148+0000 7f072ca59640 1 -- 172.21.15.29:0/35117285 wait complete. 2024-03-16T12:01:49.149 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:49.148+0000 7f072ca59640 1 Processor -- start 2024-03-16T12:01:49.149 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:49.148+0000 7f072ca59640 1 -- start start 2024-03-16T12:01:49.150 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:49.148+0000 7f072ca59640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f07280ffce0 0x7f0728197fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:01:49.150 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:49.149+0000 7f072ca59640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f07281006a0 0x7f07281984f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:01:49.150 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:49.149+0000 7f072ca59640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f07281067c0 0x7f072819c870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:01:49.150 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:49.149+0000 7f072ca59640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f0728111c90 con 0x7f07280ffce0 2024-03-16T12:01:49.150 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:49.149+0000 7f072ca59640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f0728111b10 con 0x7f07281067c0 2024-03-16T12:01:49.150 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:49.149+0000 7f072ca59640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f0728111e10 con 0x7f07281006a0 2024-03-16T12:01:49.150 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:49.149+0000 7f072659c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f07280ffce0 0x7f0728197fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:01:49.150 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:49.149+0000 7f072659c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f07280ffce0 0x7f0728197fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59882/0 (socket says 172.21.15.29:59882) 2024-03-16T12:01:49.150 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:49.149+0000 7f072659c640 1 -- 172.21.15.29:0/3300027608 learned_addr learned my addr 172.21.15.29:0/3300027608 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T12:01:49.150 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:49.149+0000 7f0726d9d640 1 --2- 172.21.15.29:0/3300027608 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f07281067c0 0x7f072819c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:01:49.150 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:49.149+0000 7f0725d9b640 1 --2- 172.21.15.29:0/3300027608 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f07281006a0 0x7f07281984f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:01:49.150 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:49.149+0000 7f072659c640 1 -- 172.21.15.29:0/3300027608 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f07281006a0 msgr2=0x7f07281984f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:01:49.150 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:49.149+0000 7f072659c640 1 --2- 172.21.15.29:0/3300027608 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f07281006a0 0x7f07281984f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:01:49.150 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:49.149+0000 7f072659c640 1 -- 172.21.15.29:0/3300027608 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f07281067c0 msgr2=0x7f072819c870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:01:49.150 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:49.149+0000 7f072659c640 1 --2- 172.21.15.29:0/3300027608 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f07281067c0 0x7f072819c870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:01:49.151 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:49.149+0000 7f072659c640 1 -- 172.21.15.29:0/3300027608 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f072819cfa0 con 0x7f07280ffce0 2024-03-16T12:01:49.151 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:49.149+0000 7f072659c640 1 --2- 172.21.15.29:0/3300027608 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f07280ffce0 0x7f0728197fb0 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f071800e9f0 tx=0x7f071800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:01:49.151 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:49.149+0000 7f07177fe640 1 -- 172.21.15.29:0/3300027608 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f071800cd80 con 0x7f07280ffce0 2024-03-16T12:01:49.151 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:49.149+0000 7f072ca59640 1 -- 172.21.15.29:0/3300027608 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f072819d290 con 0x7f07280ffce0 2024-03-16T12:01:49.151 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:49.149+0000 7f07177fe640 1 -- 172.21.15.29:0/3300027608 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0718010910 con 0x7f07280ffce0 2024-03-16T12:01:49.151 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:49.149+0000 7f07177fe640 1 -- 172.21.15.29:0/3300027608 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0718018990 con 0x7f07280ffce0 2024-03-16T12:01:49.151 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:49.149+0000 7f072ca59640 1 -- 172.21.15.29:0/3300027608 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f07281a4b20 con 0x7f07280ffce0 2024-03-16T12:01:49.156 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:49.152+0000 7f07177fe640 1 -- 172.21.15.29:0/3300027608 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7f0718010ac0 con 0x7f07280ffce0 2024-03-16T12:01:49.157 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:49.152+0000 7f072f641740 1 Processor -- start 2024-03-16T12:01:49.157 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:49.152+0000 7f072f641740 1 -- start start 2024-03-16T12:01:49.157 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:49.152+0000 7f072f641740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dcca4c2bd0 0x55dcca4c2fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:01:49.157 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:49.152+0000 7f072f641740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55dcca4c34f0 0x55dcca4d06f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:01:49.157 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:49.153+0000 7f072f641740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55dcca4d0c30 0x55dcca4d3020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:01:49.157 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:49.153+0000 7f072f641740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55dcca427a40 con 0x55dcca4c34f0 2024-03-16T12:01:49.157 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:49.153+0000 7f072f641740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x55dcca3d0920 con 0x55dcca4d0c30 2024-03-16T12:01:49.157 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:49.153+0000 7f072f641740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55dcca423ae0 con 0x55dcca4c2bd0 2024-03-16T12:01:49.157 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:49.153+0000 7f07177fe640 1 --2- 172.21.15.29:0/3300027608 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f07100a2a10 0x7f07100a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:01:49.157 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:49.153+0000 7f0726d9d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dcca4c2bd0 0x55dcca4c2fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:01:49.157 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:49.153+0000 7f072659c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55dcca4c34f0 0x55dcca4d06f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:01:49.157 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:49.153+0000 7f0726d9d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dcca4c2bd0 0x55dcca4c2fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.29:57262/0 (socket says 172.21.15.29:57262) 2024-03-16T12:01:49.157 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:49.153+0000 7f0726d9d640 1 -- 172.21.15.29:0/186101258 learned_addr learned my addr 172.21.15.29:0/186101258 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T12:01:49.157 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:49.153+0000 7f0725d9b640 1 --2- 172.21.15.29:0/186101258 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55dcca4d0c30 0x55dcca4d3020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:01:49.157 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:49.153+0000 7f0726d9d640 1 --2- 172.21.15.29:0/3300027608 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f07100a2a10 0x7f07100a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:01:49.157 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:49.153+0000 7f0726d9d640 1 -- 172.21.15.29:0/186101258 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55dcca4d0c30 msgr2=0x55dcca4d3020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:01:49.158 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:49.153+0000 7f0726d9d640 1 --2- 172.21.15.29:0/186101258 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55dcca4d0c30 0x55dcca4d3020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:01:49.158 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:49.153+0000 7f0726d9d640 1 -- 172.21.15.29:0/186101258 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55dcca4c34f0 msgr2=0x55dcca4d06f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:01:49.158 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:49.153+0000 7f0726d9d640 1 --2- 172.21.15.29:0/186101258 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55dcca4c34f0 0x55dcca4d06f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:01:49.158 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:49.153+0000 7f0726d9d640 1 -- 172.21.15.29:0/186101258 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dcca3e8460 con 0x55dcca4c2bd0 2024-03-16T12:01:49.158 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:49.153+0000 7f0725d9b640 1 --2- 172.21.15.29:0/186101258 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55dcca4d0c30 0x55dcca4d3020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:01:49.158 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:49.153+0000 7f07177fe640 1 -- 172.21.15.29:0/3300027608 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 8556+0+0 (secure 0 0 0) 0x7f07180cf120 con 0x7f07280ffce0 2024-03-16T12:01:49.158 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:49.153+0000 7f0726d9d640 1 --2- 172.21.15.29:0/186101258 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dcca4c2bd0 0x55dcca4c2fb0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f0720004af0 tx=0x7f072002cef0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b16691dcc44d040b server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:01:49.158 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:49.153+0000 7f0726d9d640 1 --2- 172.21.15.29:0/3300027608 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f07100a2a10 0x7f07100a4ed0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f0720047000 tx=0x7f0720048000 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:01:49.158 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:49.153+0000 7f0714ff9640 1 -- 172.21.15.29:0/186101258 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0720050070 con 0x55dcca4c2bd0 2024-03-16T12:01:49.158 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:49.154+0000 7f0714ff9640 1 -- 172.21.15.29:0/186101258 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0720003e50 con 0x55dcca4c2bd0 2024-03-16T12:01:49.158 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:49.154+0000 7f0714ff9640 1 -- 172.21.15.29:0/186101258 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0720039040 con 0x55dcca4c2bd0 2024-03-16T12:01:49.158 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:49.154+0000 7f072f641740 1 -- 172.21.15.29:0/186101258 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dcca4c2bd0 msgr2=0x55dcca4c2fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:01:49.158 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:49.154+0000 7f072f641740 1 --2- 172.21.15.29:0/186101258 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dcca4c2bd0 0x55dcca4c2fb0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f0720004af0 tx=0x7f072002cef0 comp rx=0 tx=0).stop 2024-03-16T12:01:49.158 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:49.154+0000 7f072f641740 1 -- 172.21.15.29:0/186101258 shutdown_connections 2024-03-16T12:01:49.158 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:49.154+0000 7f072f641740 1 --2- 172.21.15.29:0/186101258 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55dcca4d0c30 0x55dcca4d3020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:01:49.159 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:49.154+0000 7f072f641740 1 --2- 172.21.15.29:0/186101258 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55dcca4c34f0 0x55dcca4d06f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:01:49.159 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:49.154+0000 7f072f641740 1 --2- 172.21.15.29:0/186101258 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dcca4c2bd0 0x55dcca4c2fb0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:01:49.159 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:49.154+0000 7f072f641740 1 -- 172.21.15.29:0/186101258 >> 172.21.15.29:0/186101258 conn(0x55dcca4bda50 msgr2=0x55dcca4bde30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:01:49.159 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:49.154+0000 7f072f641740 1 -- 172.21.15.29:0/186101258 shutdown_connections 2024-03-16T12:01:49.159 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:49.154+0000 7f072f641740 1 -- 172.21.15.29:0/186101258 wait complete. 2024-03-16T12:01:49.159 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:49.155+0000 7f072f641740 1 -- start start 2024-03-16T12:01:49.159 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:49.155+0000 7f072f641740 1 Processor -- start 2024-03-16T12:01:49.159 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:49.155+0000 7f072f641740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55dcca4c2bd0 0x55dcca4cdca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:01:49.159 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:49.155+0000 7f072f641740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55dcca4c34f0 0x55dcca4ce1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:01:49.159 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:49.155+0000 7f072f641740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dcca4d0c30 0x55dcca4ce720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:01:49.159 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:49.155+0000 7f072f641740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55dcca4d3800 con 0x55dcca4c34f0 2024-03-16T12:01:49.159 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:49.155+0000 7f072f641740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x55dcca3e3d70 con 0x55dcca4c2bd0 2024-03-16T12:01:49.159 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:49.155+0000 7f072f641740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55dcca4d3980 con 0x55dcca4d0c30 2024-03-16T12:01:49.159 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:49.155+0000 7f072f641740 10 client.0 _ll_register_callbacks cb 0x55dcca13ba00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T12:01:49.159 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:49.155+0000 7f0725d9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55dcca4c34f0 0x55dcca4ce1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:01:49.159 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:49.155+0000 7f072659c640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55dcca4c2bd0 0x55dcca4cdca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:01:49.159 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:49.155+0000 7f0725d9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55dcca4c34f0 0x55dcca4ce1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59894/0 (socket says 172.21.15.29:59894) 2024-03-16T12:01:49.160 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:49.155+0000 7f0725d9b640 1 -- 172.21.15.29:0/3222901298 learned_addr learned my addr 172.21.15.29:0/3222901298 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T12:01:49.160 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:49.155+0000 7f0726d9d640 1 --2- 172.21.15.29:0/3222901298 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dcca4d0c30 0x55dcca4ce720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:01:49.160 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:49.155+0000 7f0725d9b640 1 -- 172.21.15.29:0/3222901298 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dcca4d0c30 msgr2=0x55dcca4ce720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:01:49.160 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:49.155+0000 7f0725d9b640 1 --2- 172.21.15.29:0/3222901298 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dcca4d0c30 0x55dcca4ce720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:01:49.160 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:49.155+0000 7f0725d9b640 1 -- 172.21.15.29:0/3222901298 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55dcca4c2bd0 msgr2=0x55dcca4cdca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:01:49.160 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:49.155+0000 7f0725d9b640 1 --2- 172.21.15.29:0/3222901298 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55dcca4c2bd0 0x55dcca4cdca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:01:49.160 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:49.155+0000 7f0725d9b640 1 -- 172.21.15.29:0/3222901298 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dcca13c210 con 0x55dcca4c34f0 2024-03-16T12:01:49.160 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:49.156+0000 7f0725d9b640 1 --2- 172.21.15.29:0/3222901298 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55dcca4c34f0 0x55dcca4ce1e0 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f071c012700 tx=0x7f071c012bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e06a21f92e3aae4 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:01:49.160 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:49.156+0000 7f070dffb640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-16T12:01:49.160 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:49.156+0000 7f070dffb640 1 -- 172.21.15.29:0/3222901298 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f071c004030 con 0x55dcca4c34f0 2024-03-16T12:01:49.160 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:49.156+0000 7f072f641740 1 -- 172.21.15.29:0/3222901298 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0720003e50 con 0x55dcca4c34f0 2024-03-16T12:01:49.160 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:49.156+0000 7f070dffb640 1 -- 172.21.15.29:0/3222901298 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 8556+0+0 (secure 0 0 0) 0x7f071c0041d0 con 0x55dcca4c34f0 2024-03-16T12:01:49.160 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:49.156+0000 7f070dffb640 1 -- 172.21.15.29:0/3222901298 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f071c0043d0 con 0x55dcca4c34f0 2024-03-16T12:01:49.160 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:49.156+0000 7f072f641740 10 client.16308 fetch_fsmap learned FSMap version 28 2024-03-16T12:01:49.160 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:49.156+0000 7f072f641740 1 -- 172.21.15.29:0/3222901298 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55dcca208620 con 0x55dcca4c34f0 2024-03-16T12:01:49.160 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:49.156+0000 7f070dffb640 1 -- 172.21.15.29:0/3222901298 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0720003e50 con 0x55dcca4c34f0 2024-03-16T12:01:49.161 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:49.156+0000 7f070dffb640 1 -- 172.21.15.29:0/3222901298 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 28) ==== 8338+0+0 (secure 0 0 0) 0x7f071c003b60 con 0x55dcca4c34f0 2024-03-16T12:01:49.161 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:49.156+0000 7f072f641740 10 client.16308 fetch_fsmap finished waiting for FSMap version 28 2024-03-16T12:01:49.161 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:49.156+0000 7f072f641740 10 client.16308 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T12:01:49.161 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:49.156+0000 7f072f641740 20 client.16308 populate_metadata read hostname 'smithi029' 2024-03-16T12:01:49.161 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:49.156+0000 7f072f641740 1 --2- 172.21.15.29:0/3222901298 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55dcca4e0ec0 0x55dcca4e32b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:01:49.161 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:49.156+0000 7f072f641740 4 client.16308 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T12:01:49.161 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:49.156+0000 7f072f641740 1 -- 172.21.15.29:0/3222901298 --> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55dcca434f80 con 0x55dcca4e0ec0 2024-03-16T12:01:49.161 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:49.157+0000 7f072659c640 1 --2- 172.21.15.29:0/3222901298 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55dcca4e0ec0 0x55dcca4e32b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:01:49.161 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:49.157+0000 7f070dffb640 10 client.16308 ms_handle_connect on v2:172.21.15.152:6838/1548731833 2024-03-16T12:01:49.161 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:49.157+0000 7f072659c640 1 --2- 172.21.15.29:0/3222901298 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55dcca4e0ec0 0x55dcca4e32b0 crc :-1 s=READY pgs=136 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a76d8ce9f4c93292 server_cookie=daa82896c164716d in_seq=0 out_seq=0 2024-03-16T12:01:49.162 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:49.159+0000 7f070dffb640 1 -- 172.21.15.29:0/3222901298 <== mds.0 v2:172.21.15.152:6838/1548731833 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55dcca434f80 con 0x55dcca4e0ec0 2024-03-16T12:01:49.162 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:49.159+0000 7f070dffb640 10 client.16308 handle_command_reply: tid=0 2024-03-16T12:01:49.162 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:49.159+0000 7f072f641740 1 client.16308 shutdown 2024-03-16T12:01:49.162 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:49.159+0000 7f072f641740 1 -- 172.21.15.29:0/3222901298 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55dcca4c34f0 msgr2=0x55dcca4ce1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:01:49.162 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:49.159+0000 7f072f641740 1 --2- 172.21.15.29:0/3222901298 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55dcca4c34f0 0x55dcca4ce1e0 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f071c012700 tx=0x7f071c012bc0 comp rx=0 tx=0).stop 2024-03-16T12:01:49.162 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:49.159+0000 7f072f641740 1 -- 172.21.15.29:0/3222901298 shutdown_connections 2024-03-16T12:01:49.162 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:49.159+0000 7f072f641740 1 --2- 172.21.15.29:0/3222901298 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dcca4d0c30 0x55dcca4ce720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:01:49.163 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:49.159+0000 7f072f641740 1 --2- 172.21.15.29:0/3222901298 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55dcca4c34f0 0x55dcca4ce1e0 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:01:49.163 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:49.159+0000 7f072f641740 1 --2- 172.21.15.29:0/3222901298 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55dcca4e0ec0 0x55dcca4e32b0 crc :-1 s=READY pgs=136 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:01:49.163 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:49.159+0000 7f072f641740 1 --2- 172.21.15.29:0/3222901298 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55dcca4c2bd0 0x55dcca4cdca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:01:49.163 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:49.159+0000 7f070dffb640 0 client.16308 ms_handle_reset on v2:172.21.15.152:6838/1548731833 2024-03-16T12:01:49.163 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:49.159+0000 7f072f641740 1 -- 172.21.15.29:0/3222901298 >> 172.21.15.29:0/3222901298 conn(0x55dcca4bda50 msgr2=0x55dcca4bf7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:01:49.163 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:49.159+0000 7f072f641740 1 -- 172.21.15.29:0/3222901298 shutdown_connections 2024-03-16T12:01:49.163 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:49.159+0000 7f072f641740 1 -- 172.21.15.29:0/3222901298 wait complete. 2024-03-16T12:01:49.163 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:49.159+0000 7f072f641740 20 client.16308 trim_cache size 0 max 16384 2024-03-16T12:01:49.177 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:49.174+0000 7f072f641740 1 Processor -- start 2024-03-16T12:01:49.177 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:49.174+0000 7f072f641740 1 -- start start 2024-03-16T12:01:49.177 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:49.174+0000 7f072f641740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55dcca4c2bd0 0x55dcca4d03d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:01:49.177 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:49.174+0000 7f072f641740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55dcca4c34f0 0x55dcca4dc5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:01:49.178 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:49.174+0000 7f072f641740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dcca4d0c30 0x55dcca4e8bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:01:49.178 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:49.174+0000 7f072f641740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f07281ada90 con 0x55dcca4c34f0 2024-03-16T12:01:49.178 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:49.174+0000 7f072f641740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f07281ab360 con 0x55dcca4c2bd0 2024-03-16T12:01:49.178 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:49.174+0000 7f072f641740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f07281a91d0 con 0x55dcca4d0c30 2024-03-16T12:01:49.178 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:49.174+0000 7f0725d9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55dcca4c34f0 0x55dcca4dc5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:01:49.178 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:49.174+0000 7f0725d9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55dcca4c34f0 0x55dcca4dc5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59902/0 (socket says 172.21.15.29:59902) 2024-03-16T12:01:49.178 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:49.174+0000 7f0725d9b640 1 -- 172.21.15.29:0/1263570025 learned_addr learned my addr 172.21.15.29:0/1263570025 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T12:01:49.178 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:49.174+0000 7f0726d9d640 1 --2- 172.21.15.29:0/1263570025 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dcca4d0c30 0x55dcca4e8bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:01:49.178 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:49.174+0000 7f072659c640 1 --2- 172.21.15.29:0/1263570025 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55dcca4c2bd0 0x55dcca4d03d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:01:49.178 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:49.175+0000 7f0725d9b640 1 -- 172.21.15.29:0/1263570025 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dcca4d0c30 msgr2=0x55dcca4e8bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:01:49.178 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:49.175+0000 7f0725d9b640 1 --2- 172.21.15.29:0/1263570025 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dcca4d0c30 0x55dcca4e8bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:01:49.178 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:49.175+0000 7f0725d9b640 1 -- 172.21.15.29:0/1263570025 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55dcca4c2bd0 msgr2=0x55dcca4d03d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:01:49.178 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:49.175+0000 7f0725d9b640 1 --2- 172.21.15.29:0/1263570025 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55dcca4c2bd0 0x55dcca4d03d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:01:49.179 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:49.175+0000 7f0725d9b640 1 -- 172.21.15.29:0/1263570025 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dcca4dfa60 con 0x55dcca4c34f0 2024-03-16T12:01:49.179 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:49.175+0000 7f0725d9b640 1 --2- 172.21.15.29:0/1263570025 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55dcca4c34f0 0x55dcca4dc5d0 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f071c010430 tx=0x7f071c0096c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2cafd09c629313e0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:01:49.179 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:49.175+0000 7f070ffff640 1 -- 172.21.15.29:0/1263570025 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f071c029070 con 0x55dcca4c34f0 2024-03-16T12:01:49.179 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:49.175+0000 7f070ffff640 1 -- 172.21.15.29:0/1263570025 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f071c004000 con 0x55dcca4c34f0 2024-03-16T12:01:49.179 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:49.175+0000 7f070ffff640 1 -- 172.21.15.29:0/1263570025 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f071c003bc0 con 0x55dcca4c34f0 2024-03-16T12:01:49.179 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:49.176+0000 7f072f641740 1 -- 172.21.15.29:0/1263570025 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55dcca4c34f0 msgr2=0x55dcca4dc5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:01:49.179 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:49.176+0000 7f072f641740 1 --2- 172.21.15.29:0/1263570025 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55dcca4c34f0 0x55dcca4dc5d0 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f071c010430 tx=0x7f071c0096c0 comp rx=0 tx=0).stop 2024-03-16T12:01:49.179 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:49.176+0000 7f072f641740 1 -- 172.21.15.29:0/1263570025 shutdown_connections 2024-03-16T12:01:49.179 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:49.176+0000 7f072f641740 1 --2- 172.21.15.29:0/1263570025 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dcca4d0c30 0x55dcca4e8bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:01:49.179 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:49.176+0000 7f072f641740 1 --2- 172.21.15.29:0/1263570025 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55dcca4c34f0 0x55dcca4dc5d0 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:01:49.179 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:49.176+0000 7f072f641740 1 --2- 172.21.15.29:0/1263570025 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55dcca4c2bd0 0x55dcca4d03d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:01:49.179 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:49.176+0000 7f072f641740 1 -- 172.21.15.29:0/1263570025 >> 172.21.15.29:0/1263570025 conn(0x55dcca4bda50 msgr2=0x55dcca4b8a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:01:49.179 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:49.176+0000 7f072f641740 1 -- 172.21.15.29:0/1263570025 shutdown_connections 2024-03-16T12:01:49.179 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:49.176+0000 7f072f641740 1 -- 172.21.15.29:0/1263570025 wait complete. 2024-03-16T12:01:49.179 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:49.176+0000 7f072f641740 1 -- start start 2024-03-16T12:01:49.180 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:49.176+0000 7f072f641740 1 Processor -- start 2024-03-16T12:01:49.180 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:49.176+0000 7f072f641740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55dcca4c2bd0 0x55dcca4d4730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:01:49.180 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:49.176+0000 7f072f641740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dcca4c34f0 0x55dcca4d4c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:01:49.180 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:49.176+0000 7f072f641740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55dcca4d0c30 0x55dcca4d51b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:01:49.180 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:49.176+0000 7f072f641740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f07281a9820 con 0x55dcca4c2bd0 2024-03-16T12:01:49.180 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:49.176+0000 7f072f641740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f07281aac90 con 0x55dcca4d0c30 2024-03-16T12:01:49.180 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:49.176+0000 7f072f641740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f07281a9ef0 con 0x55dcca4c34f0 2024-03-16T12:01:49.180 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:49.176+0000 7f072f641740 10 client.0 _ll_register_callbacks cb 0x7f071c009930 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T12:01:49.180 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:49.176+0000 7f072659c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55dcca4c2bd0 0x55dcca4d4730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:01:49.180 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:49.176+0000 7f072659c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55dcca4c2bd0 0x55dcca4d4730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59910/0 (socket says 172.21.15.29:59910) 2024-03-16T12:01:49.180 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:49.176+0000 7f072659c640 1 -- 172.21.15.29:0/2790595256 learned_addr learned my addr 172.21.15.29:0/2790595256 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T12:01:49.180 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:49.176+0000 7f0725d9b640 1 --2- 172.21.15.29:0/2790595256 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dcca4c34f0 0x55dcca4d4c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:01:49.180 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:49.176+0000 7f0726d9d640 1 --2- 172.21.15.29:0/2790595256 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55dcca4d0c30 0x55dcca4d51b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:01:49.180 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:49.177+0000 7f072659c640 1 -- 172.21.15.29:0/2790595256 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dcca4c34f0 msgr2=0x55dcca4d4c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:01:49.181 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:49.177+0000 7f072659c640 1 --2- 172.21.15.29:0/2790595256 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dcca4c34f0 0x55dcca4d4c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:01:49.181 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:49.177+0000 7f072659c640 1 -- 172.21.15.29:0/2790595256 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55dcca4d0c30 msgr2=0x55dcca4d51b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:01:49.181 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:49.177+0000 7f072659c640 1 --2- 172.21.15.29:0/2790595256 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55dcca4d0c30 0x55dcca4d51b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:01:49.181 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:49.177+0000 7f072659c640 1 -- 172.21.15.29:0/2790595256 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dcca546440 con 0x55dcca4c2bd0 2024-03-16T12:01:49.181 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:49.177+0000 7f072659c640 1 --2- 172.21.15.29:0/2790595256 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55dcca4c2bd0 0x55dcca4d4730 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f0718097d10 tx=0x7f0718086040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df40c1572bab1f49 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:01:49.181 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:49.177+0000 7f070dffb640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-16T12:01:49.181 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:49.177+0000 7f070dffb640 1 -- 172.21.15.29:0/2790595256 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07180b3070 con 0x55dcca4c2bd0 2024-03-16T12:01:49.181 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:49.177+0000 7f070dffb640 1 -- 172.21.15.29:0/2790595256 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 8556+0+0 (secure 0 0 0) 0x7f07180ccb70 con 0x55dcca4c2bd0 2024-03-16T12:01:49.181 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:49.177+0000 7f072f641740 1 -- 172.21.15.29:0/2790595256 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f071c004000 con 0x55dcca4c2bd0 2024-03-16T12:01:49.181 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:49.177+0000 7f070dffb640 1 -- 172.21.15.29:0/2790595256 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f0718097f20 con 0x55dcca4c2bd0 2024-03-16T12:01:49.181 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:49.177+0000 7f072f641740 10 client.16314 fetch_fsmap learned FSMap version 28 2024-03-16T12:01:49.181 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:49.177+0000 7f072f641740 1 -- 172.21.15.29:0/2790595256 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55dcca547790 con 0x55dcca4c2bd0 2024-03-16T12:01:49.181 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:49.177+0000 7f070dffb640 1 -- 172.21.15.29:0/2790595256 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f071c004000 con 0x55dcca4c2bd0 2024-03-16T12:01:49.181 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:49.177+0000 7f070dffb640 1 -- 172.21.15.29:0/2790595256 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 28) ==== 8338+0+0 (secure 0 0 0) 0x7f07180afce0 con 0x55dcca4c2bd0 2024-03-16T12:01:49.181 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:49.177+0000 7f072f641740 10 client.16314 fetch_fsmap finished waiting for FSMap version 28 2024-03-16T12:01:49.181 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:49.177+0000 7f072f641740 10 client.16314 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T12:01:49.182 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:49.177+0000 7f072f641740 20 client.16314 populate_metadata read hostname 'smithi029' 2024-03-16T12:01:49.182 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:49.178+0000 7f072f641740 1 --2- 172.21.15.29:0/2790595256 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55dcca547eb0 0x55dcca54a2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:01:49.182 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:49.178+0000 7f072f641740 4 client.16314 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-16T12:01:49.182 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:49.178+0000 7f072f641740 1 -- 172.21.15.29:0/2790595256 --> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55dcca42f2d0 con 0x55dcca547eb0 2024-03-16T12:01:49.182 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:49.178+0000 7f0725d9b640 1 --2- 172.21.15.29:0/2790595256 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55dcca547eb0 0x55dcca54a2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:01:49.182 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:49.178+0000 7f0725d9b640 1 --2- 172.21.15.29:0/2790595256 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55dcca547eb0 0x55dcca54a2c0 crc :-1 s=READY pgs=137 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=29001bf5e857a477 server_cookie=8f1e60e5ea6b9e in_seq=0 out_seq=0 2024-03-16T12:01:49.182 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:49.178+0000 7f070dffb640 10 client.16314 ms_handle_connect on v2:172.21.15.152:6838/1548731833 2024-03-16T12:01:49.339 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:49.339+0000 7f070dffb640 1 -- 172.21.15.29:0/2790595256 <== mds.0 v2:172.21.15.152:6838/1548731833 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55dcca42f2d0 con 0x55dcca547eb0 2024-03-16T12:01:49.339 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:49.339+0000 7f070dffb640 10 client.16314 handle_command_reply: tid=0 2024-03-16T12:01:49.339 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:49.339+0000 7f072f641740 1 client.16314 shutdown 2024-03-16T12:01:49.339 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:49.339+0000 7f072f641740 1 -- 172.21.15.29:0/2790595256 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55dcca4c2bd0 msgr2=0x55dcca4d4730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:01:49.339 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:49.339+0000 7f072f641740 1 --2- 172.21.15.29:0/2790595256 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55dcca4c2bd0 0x55dcca4d4730 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f0718097d10 tx=0x7f0718086040 comp rx=0 tx=0).stop 2024-03-16T12:01:49.339 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:49.339+0000 7f072f641740 1 -- 172.21.15.29:0/2790595256 shutdown_connections 2024-03-16T12:01:49.339 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:49.339+0000 7f072f641740 1 --2- 172.21.15.29:0/2790595256 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55dcca547eb0 0x55dcca54a2c0 crc :-1 s=READY pgs=137 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:01:49.339 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:49.339+0000 7f072f641740 1 --2- 172.21.15.29:0/2790595256 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55dcca4d0c30 0x55dcca4d51b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:01:49.339 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:49.339+0000 7f072f641740 1 --2- 172.21.15.29:0/2790595256 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dcca4c34f0 0x55dcca4d4c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:01:49.339 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:49.339+0000 7f072f641740 1 --2- 172.21.15.29:0/2790595256 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55dcca4c2bd0 0x55dcca4d4730 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:01:49.339 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:49.339+0000 7f070dffb640 0 client.16314 ms_handle_reset on v2:172.21.15.152:6838/1548731833 2024-03-16T12:01:49.339 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:49.339+0000 7f072f641740 1 -- 172.21.15.29:0/2790595256 >> 172.21.15.29:0/2790595256 conn(0x55dcca4bda50 msgr2=0x55dcca4cdc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:01:49.339 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:49.339+0000 7f072f641740 1 -- 172.21.15.29:0/2790595256 shutdown_connections 2024-03-16T12:01:49.339 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:49.339+0000 7f072f641740 1 -- 172.21.15.29:0/2790595256 wait complete. 2024-03-16T12:01:49.340 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:49.339+0000 7f072f641740 20 client.16314 trim_cache size 0 max 16384 2024-03-16T12:01:49.340 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:{ 2024-03-16T12:01:49.340 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "message": "", 2024-03-16T12:01:49.340 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "return_code": 0 2024-03-16T12:01:49.340 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:} 2024-03-16T12:01:49.341 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:49.340+0000 7f072ca59640 1 -- 172.21.15.29:0/3300027608 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f07100a2a10 msgr2=0x7f07100a4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:01:49.341 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:49.340+0000 7f072ca59640 1 --2- 172.21.15.29:0/3300027608 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f07100a2a10 0x7f07100a4ed0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f0720047000 tx=0x7f0720048000 comp rx=0 tx=0).stop 2024-03-16T12:01:49.341 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:49.340+0000 7f072ca59640 1 -- 172.21.15.29:0/3300027608 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f07280ffce0 msgr2=0x7f0728197fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:01:49.341 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:49.340+0000 7f072ca59640 1 --2- 172.21.15.29:0/3300027608 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f07280ffce0 0x7f0728197fb0 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f071800e9f0 tx=0x7f071800eeb0 comp rx=0 tx=0).stop 2024-03-16T12:01:49.341 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:49.340+0000 7f072ca59640 1 -- 172.21.15.29:0/3300027608 shutdown_connections 2024-03-16T12:01:49.341 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:49.340+0000 7f072ca59640 1 --2- 172.21.15.29:0/3300027608 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f07100a2a10 0x7f07100a4ed0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:01:49.341 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:49.340+0000 7f072ca59640 1 --2- 172.21.15.29:0/3300027608 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f07281067c0 0x7f072819c870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:01:49.341 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:49.340+0000 7f072ca59640 1 --2- 172.21.15.29:0/3300027608 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f07281006a0 0x7f07281984f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:01:49.341 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:49.340+0000 7f072ca59640 1 --2- 172.21.15.29:0/3300027608 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f07280ffce0 0x7f0728197fb0 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:01:49.341 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:49.340+0000 7f072ca59640 1 -- 172.21.15.29:0/3300027608 >> 172.21.15.29:0/3300027608 conn(0x7f0728076ca0 msgr2=0x7f0728104510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:01:49.342 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:49.340+0000 7f072ca59640 1 -- 172.21.15.29:0/3300027608 shutdown_connections 2024-03-16T12:01:49.342 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:01:49.340+0000 7f072ca59640 1 -- 172.21.15.29:0/3300027608 wait complete. 2024-03-16T12:01:49.803 INFO:tasks.workunit.client.0.smithi029.stdout: 1 1065763 9.51 MB/sec execute 454 sec latency 107.749 ms 2024-03-16T12:01:50.545 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:01:50 smithi154 ceph-mon[36320]: pgmap v344: 97 pgs: 97 active+clean; 73 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.0 MiB/s wr, 170 op/s 2024-03-16T12:01:50.588 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:01:50 smithi029 ceph-mon[31975]: pgmap v344: 97 pgs: 97 active+clean; 73 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.0 MiB/s wr, 170 op/s 2024-03-16T12:01:50.627 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:01:50 smithi152 ceph-mon[36291]: pgmap v344: 97 pgs: 97 active+clean; 73 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.0 MiB/s wr, 170 op/s 2024-03-16T12:01:50.803 INFO:tasks.workunit.client.0.smithi029.stdout: 1 1066572 9.49 MB/sec execute 455 sec latency 26.621 ms 2024-03-16T12:01:51.803 INFO:tasks.workunit.client.0.smithi029.stdout: 1 1068471 9.50 MB/sec execute 456 sec latency 25.945 ms 2024-03-16T12:01:52.546 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:01:52 smithi154 ceph-mon[36320]: pgmap v345: 97 pgs: 97 active+clean; 66 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.1 MiB/s wr, 297 op/s 2024-03-16T12:01:52.587 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:01:52 smithi029 ceph-mon[31975]: pgmap v345: 97 pgs: 97 active+clean; 66 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.1 MiB/s wr, 297 op/s 2024-03-16T12:01:52.627 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:01:52 smithi152 ceph-mon[36291]: pgmap v345: 97 pgs: 97 active+clean; 66 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.1 MiB/s wr, 297 op/s 2024-03-16T12:01:52.803 INFO:tasks.workunit.client.0.smithi029.stdout: 1 1069606 9.49 MB/sec execute 457 sec latency 22.102 ms 2024-03-16T12:01:53.803 INFO:tasks.workunit.client.0.smithi029.stdout: 1 1071049 9.48 MB/sec execute 458 sec latency 26.901 ms 2024-03-16T12:01:54.545 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:01:54 smithi154 ceph-mon[36320]: pgmap v346: 97 pgs: 97 active+clean; 66 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.4 MiB/s wr, 217 op/s 2024-03-16T12:01:54.587 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:01:54 smithi029 ceph-mon[31975]: pgmap v346: 97 pgs: 97 active+clean; 66 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.4 MiB/s wr, 217 op/s 2024-03-16T12:01:54.626 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:01:54 smithi152 ceph-mon[36291]: pgmap v346: 97 pgs: 97 active+clean; 66 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.4 MiB/s wr, 217 op/s 2024-03-16T12:01:54.803 INFO:tasks.workunit.client.0.smithi029.stdout: 1 1072869 9.49 MB/sec execute 459 sec latency 58.803 ms 2024-03-16T12:01:55.803 INFO:tasks.workunit.client.0.smithi029.stdout: 1 1073945 9.48 MB/sec execute 460 sec latency 35.687 ms 2024-03-16T12:01:56.544 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:01:56 smithi154 ceph-mon[36320]: pgmap v347: 97 pgs: 97 active+clean; 66 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.1 MiB/s wr, 201 op/s 2024-03-16T12:01:56.587 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:01:56 smithi029 ceph-mon[31975]: pgmap v347: 97 pgs: 97 active+clean; 66 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.1 MiB/s wr, 201 op/s 2024-03-16T12:01:56.626 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:01:56 smithi152 ceph-mon[36291]: pgmap v347: 97 pgs: 97 active+clean; 66 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.1 MiB/s wr, 201 op/s 2024-03-16T12:01:56.803 INFO:tasks.workunit.client.0.smithi029.stdout: 1 1076075 9.48 MB/sec execute 461 sec latency 47.456 ms 2024-03-16T12:01:57.803 INFO:tasks.workunit.client.0.smithi029.stdout: 1 1076830 9.47 MB/sec execute 462 sec latency 61.088 ms 2024-03-16T12:01:58.544 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:01:58 smithi154 ceph-mon[36320]: pgmap v348: 97 pgs: 97 active+clean; 71 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 3.1 MiB/s wr, 269 op/s 2024-03-16T12:01:58.587 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:01:58 smithi029 ceph-mon[31975]: pgmap v348: 97 pgs: 97 active+clean; 71 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 3.1 MiB/s wr, 269 op/s 2024-03-16T12:01:58.626 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:01:58 smithi152 ceph-mon[36291]: pgmap v348: 97 pgs: 97 active+clean; 71 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 3.1 MiB/s wr, 269 op/s 2024-03-16T12:01:58.803 INFO:tasks.workunit.client.0.smithi029.stdout: 1 1078674 9.47 MB/sec execute 463 sec latency 27.775 ms 2024-03-16T12:01:59.803 INFO:tasks.workunit.client.0.smithi029.stdout: 1 1080013 9.47 MB/sec execute 464 sec latency 125.705 ms 2024-03-16T12:02:00.544 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:02:00 smithi154 ceph-mon[36320]: pgmap v349: 97 pgs: 97 active+clean; 71 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.1 MiB/s wr, 200 op/s 2024-03-16T12:02:00.587 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:02:00 smithi029 ceph-mon[31975]: pgmap v349: 97 pgs: 97 active+clean; 71 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.1 MiB/s wr, 200 op/s 2024-03-16T12:02:00.626 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:02:00 smithi152 ceph-mon[36291]: pgmap v349: 97 pgs: 97 active+clean; 71 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.1 MiB/s wr, 200 op/s 2024-03-16T12:02:00.803 INFO:tasks.workunit.client.0.smithi029.stdout: 1 1080900 9.46 MB/sec execute 465 sec latency 41.156 ms 2024-03-16T12:02:01.803 INFO:tasks.workunit.client.0.smithi029.stdout: 1 1082194 9.45 MB/sec execute 466 sec latency 30.372 ms 2024-03-16T12:02:02.433 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:02.432+0000 7ff14a685640 1 -- 172.21.15.29:0/2123715244 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff1441012d0 msgr2=0x7ff14410d510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:02:02.433 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:02.432+0000 7ff14a685640 1 --2- 172.21.15.29:0/2123715244 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff1441012d0 0x7ff14410d510 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7ff13c009ca0 tx=0x7ff13c02c0f0 comp rx=0 tx=0).stop 2024-03-16T12:02:02.433 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:02.432+0000 7ff14a685640 1 -- 172.21.15.29:0/2123715244 shutdown_connections 2024-03-16T12:02:02.433 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:02.432+0000 7ff14a685640 1 --2- 172.21.15.29:0/2123715244 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff1441012d0 0x7ff14410d510 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:02:02.433 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:02.432+0000 7ff14a685640 1 --2- 172.21.15.29:0/2123715244 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff144100930 0x7ff144100d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:02:02.433 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:02.432+0000 7ff14a685640 1 --2- 172.21.15.29:0/2123715244 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff144105660 0x7ff144105a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:02:02.433 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:02.432+0000 7ff14a685640 1 -- 172.21.15.29:0/2123715244 >> 172.21.15.29:0/2123715244 conn(0x7ff1440774b0 msgr2=0x7ff1440778c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:02:02.434 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:02.432+0000 7ff14a685640 1 -- 172.21.15.29:0/2123715244 shutdown_connections 2024-03-16T12:02:02.434 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:02.432+0000 7ff14a685640 1 -- 172.21.15.29:0/2123715244 wait complete. 2024-03-16T12:02:02.434 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:02.432+0000 7ff14a685640 1 Processor -- start 2024-03-16T12:02:02.434 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:02.432+0000 7ff14a685640 1 -- start start 2024-03-16T12:02:02.434 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:02.433+0000 7ff14a685640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff144100930 0x7ff144197fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:02:02.434 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:02.433+0000 7ff14a685640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff1441012d0 0x7ff1441984e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:02:02.434 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:02.433+0000 7ff14a685640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff144105660 0x7ff14419c810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:02:02.434 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:02.433+0000 7ff14a685640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7ff144110b70 con 0x7ff1441012d0 2024-03-16T12:02:02.434 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:02.433+0000 7ff14a685640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7ff1441109f0 con 0x7ff144100930 2024-03-16T12:02:02.434 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:02.433+0000 7ff14a685640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7ff144110cf0 con 0x7ff144105660 2024-03-16T12:02:02.434 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:02.433+0000 7ff1437fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff1441012d0 0x7ff1441984e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:02:02.434 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:02.433+0000 7ff1437fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff1441012d0 0x7ff1441984e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51676/0 (socket says 172.21.15.29:51676) 2024-03-16T12:02:02.434 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:02.433+0000 7ff1437fe640 1 -- 172.21.15.29:0/857678049 learned_addr learned my addr 172.21.15.29:0/857678049 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T12:02:02.434 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:02.433+0000 7ff143fff640 1 --2- 172.21.15.29:0/857678049 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff144100930 0x7ff144197fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:02:02.434 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:02.433+0000 7ff148c22640 1 --2- 172.21.15.29:0/857678049 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff144105660 0x7ff14419c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:02:02.435 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:02.433+0000 7ff1437fe640 1 -- 172.21.15.29:0/857678049 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff144105660 msgr2=0x7ff14419c810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:02:02.435 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:02.433+0000 7ff1437fe640 1 --2- 172.21.15.29:0/857678049 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff144105660 0x7ff14419c810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:02:02.435 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:02.433+0000 7ff1437fe640 1 -- 172.21.15.29:0/857678049 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff144100930 msgr2=0x7ff144197fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:02:02.435 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:02.433+0000 7ff1437fe640 1 --2- 172.21.15.29:0/857678049 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff144100930 0x7ff144197fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:02:02.435 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:02.433+0000 7ff1437fe640 1 -- 172.21.15.29:0/857678049 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff14419cf40 con 0x7ff1441012d0 2024-03-16T12:02:02.435 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:02.433+0000 7ff143fff640 1 --2- 172.21.15.29:0/857678049 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff144100930 0x7ff144197fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:02:02.435 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:02.433+0000 7ff1437fe640 1 --2- 172.21.15.29:0/857678049 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff1441012d0 0x7ff1441984e0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7ff13800e9f0 tx=0x7ff13800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:02:02.435 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:02.433+0000 7ff1417fa640 1 -- 172.21.15.29:0/857678049 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff13800cd80 con 0x7ff1441012d0 2024-03-16T12:02:02.435 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:02.434+0000 7ff1417fa640 1 -- 172.21.15.29:0/857678049 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff138010910 con 0x7ff1441012d0 2024-03-16T12:02:02.435 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:02.434+0000 7ff14a685640 1 -- 172.21.15.29:0/857678049 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff14419d230 con 0x7ff1441012d0 2024-03-16T12:02:02.435 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:02.434+0000 7ff1417fa640 1 -- 172.21.15.29:0/857678049 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff138018990 con 0x7ff1441012d0 2024-03-16T12:02:02.435 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:02.434+0000 7ff14a685640 1 -- 172.21.15.29:0/857678049 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff1440fe220 con 0x7ff1441012d0 2024-03-16T12:02:02.441 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:02.436+0000 7ff1417fa640 1 -- 172.21.15.29:0/857678049 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7ff138010ac0 con 0x7ff1441012d0 2024-03-16T12:02:02.441 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:02.436+0000 7ff14d276740 1 Processor -- start 2024-03-16T12:02:02.441 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:02.436+0000 7ff14d276740 1 -- start start 2024-03-16T12:02:02.442 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:02.436+0000 7ff14d276740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561224fd5130 0x561224fd5510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:02:02.442 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:02.436+0000 7ff14d276740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561224fd5a50 0x561224fde2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:02:02.442 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:02.437+0000 7ff14d276740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561224fde820 0x561224fe0c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:02:02.442 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:02.437+0000 7ff14d276740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x561224f39a40 con 0x561224fd5a50 2024-03-16T12:02:02.442 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:02.437+0000 7ff14d276740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x561224ee2920 con 0x561224fd5130 2024-03-16T12:02:02.442 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:02.437+0000 7ff14d276740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x561224f35ae0 con 0x561224fde820 2024-03-16T12:02:02.442 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:02.437+0000 7ff1417fa640 1 --2- 172.21.15.29:0/857678049 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7ff12c0a2a10 0x7ff12c0a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:02:02.442 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:02.437+0000 7ff143fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561224fd5a50 0x561224fde2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:02:02.442 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:02.437+0000 7ff148c22640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561224fd5130 0x561224fd5510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:02:02.442 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:02.437+0000 7ff148c22640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561224fd5130 0x561224fd5510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.29:44102/0 (socket says 172.21.15.29:44102) 2024-03-16T12:02:02.442 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:02.437+0000 7ff143fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561224fd5a50 0x561224fde2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51684/0 (socket says 172.21.15.29:51684) 2024-03-16T12:02:02.442 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:02.437+0000 7ff148c22640 1 -- 172.21.15.29:0/3298157777 learned_addr learned my addr 172.21.15.29:0/3298157777 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T12:02:02.443 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:02.437+0000 7ff1437fe640 1 --2- 172.21.15.29:0/3298157777 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561224fde820 0x561224fe0c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:02:02.443 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:02.437+0000 7ff148c22640 1 --2- 172.21.15.29:0/857678049 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7ff12c0a2a10 0x7ff12c0a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:02:02.443 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:02.437+0000 7ff1417fa640 1 -- 172.21.15.29:0/857678049 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 8556+0+0 (secure 0 0 0) 0x7ff1380cf1e0 con 0x7ff1441012d0 2024-03-16T12:02:02.443 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:02.437+0000 7ff148c22640 1 -- 172.21.15.29:0/3298157777 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561224fde820 msgr2=0x561224fe0c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:02:02.443 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:02.437+0000 7ff148c22640 1 --2- 172.21.15.29:0/3298157777 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561224fde820 0x561224fe0c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:02:02.443 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:02.437+0000 7ff148c22640 1 -- 172.21.15.29:0/3298157777 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561224fd5a50 msgr2=0x561224fde2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:02:02.443 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:02.437+0000 7ff148c22640 1 --2- 172.21.15.29:0/3298157777 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561224fd5a50 0x561224fde2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:02:02.443 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:02.437+0000 7ff148c22640 1 -- 172.21.15.29:0/3298157777 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561224efa460 con 0x561224fd5130 2024-03-16T12:02:02.443 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:02.437+0000 7ff143fff640 1 --2- 172.21.15.29:0/3298157777 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561224fd5a50 0x561224fde2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:02:02.443 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:02.437+0000 7ff148c22640 1 --2- 172.21.15.29:0/857678049 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7ff12c0a2a10 0x7ff12c0a4ed0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7ff13c009ca0 tx=0x7ff13c046000 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:02:02.443 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:02.437+0000 7ff148c22640 1 --2- 172.21.15.29:0/3298157777 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561224fd5130 0x561224fd5510 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7ff13c037040 tx=0x7ff13c032ee0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7c0f89924fe58359 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:02:02.443 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:02.437+0000 7ff12a7fc640 1 -- 172.21.15.29:0/3298157777 <== mon.2 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff13c045040 con 0x561224fd5130 2024-03-16T12:02:02.443 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:02.438+0000 7ff12a7fc640 1 -- 172.21.15.29:0/3298157777 <== mon.2 v2:172.21.15.152:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff13c004d40 con 0x561224fd5130 2024-03-16T12:02:02.444 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:02.438+0000 7ff12a7fc640 1 -- 172.21.15.29:0/3298157777 <== mon.2 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff13c005250 con 0x561224fd5130 2024-03-16T12:02:02.444 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:02.438+0000 7ff14d276740 1 -- 172.21.15.29:0/3298157777 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561224fd5130 msgr2=0x561224fd5510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:02:02.444 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:02.438+0000 7ff14d276740 1 --2- 172.21.15.29:0/3298157777 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561224fd5130 0x561224fd5510 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7ff13c037040 tx=0x7ff13c032ee0 comp rx=0 tx=0).stop 2024-03-16T12:02:02.444 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:02.438+0000 7ff14d276740 1 -- 172.21.15.29:0/3298157777 shutdown_connections 2024-03-16T12:02:02.444 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:02.438+0000 7ff14d276740 1 --2- 172.21.15.29:0/3298157777 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561224fde820 0x561224fe0c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:02:02.444 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:02.438+0000 7ff14d276740 1 --2- 172.21.15.29:0/3298157777 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561224fd5a50 0x561224fde2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:02:02.444 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:02.438+0000 7ff14d276740 1 --2- 172.21.15.29:0/3298157777 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561224fd5130 0x561224fd5510 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:02:02.444 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:02.438+0000 7ff14d276740 1 -- 172.21.15.29:0/3298157777 >> 172.21.15.29:0/3298157777 conn(0x561224fcfa50 msgr2=0x561224fcfe30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:02:02.444 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:02.438+0000 7ff14d276740 1 -- 172.21.15.29:0/3298157777 shutdown_connections 2024-03-16T12:02:02.444 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:02.438+0000 7ff14d276740 1 -- 172.21.15.29:0/3298157777 wait complete. 2024-03-16T12:02:02.444 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:02.439+0000 7ff14d276740 1 -- start start 2024-03-16T12:02:02.444 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:02.439+0000 7ff14d276740 1 Processor -- start 2024-03-16T12:02:02.444 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:02.439+0000 7ff14d276740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561224fd5130 0x561224fdb8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:02:02.444 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:02.439+0000 7ff14d276740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561224fd5a50 0x561224fdbdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:02:02.445 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:02.439+0000 7ff14d276740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561224fde820 0x561224fdc330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:02:02.445 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:02.439+0000 7ff14d276740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x561224fe13f0 con 0x561224fd5a50 2024-03-16T12:02:02.445 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:02.439+0000 7ff14d276740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x561224ef5d70 con 0x561224fd5130 2024-03-16T12:02:02.445 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:02.439+0000 7ff14d276740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x561224fe1570 con 0x561224fde820 2024-03-16T12:02:02.445 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:02.439+0000 7ff14d276740 10 client.0 _ll_register_callbacks cb 0x561224c4da00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T12:02:02.445 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:02.439+0000 7ff1437fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561224fd5a50 0x561224fdbdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:02:02.445 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:02.439+0000 7ff1437fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561224fd5a50 0x561224fdbdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51696/0 (socket says 172.21.15.29:51696) 2024-03-16T12:02:02.445 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:02.439+0000 7ff1437fe640 1 -- 172.21.15.29:0/1016454655 learned_addr learned my addr 172.21.15.29:0/1016454655 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T12:02:02.445 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:02.439+0000 7ff143fff640 1 --2- 172.21.15.29:0/1016454655 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561224fd5130 0x561224fdb8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:02:02.445 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:02.439+0000 7ff148c22640 1 --2- 172.21.15.29:0/1016454655 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561224fde820 0x561224fdc330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:02:02.445 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:02.439+0000 7ff1437fe640 1 -- 172.21.15.29:0/1016454655 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561224fde820 msgr2=0x561224fdc330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:02:02.445 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:02.439+0000 7ff1437fe640 1 --2- 172.21.15.29:0/1016454655 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561224fde820 0x561224fdc330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:02:02.445 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:02.439+0000 7ff1437fe640 1 -- 172.21.15.29:0/1016454655 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561224fd5130 msgr2=0x561224fdb8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:02:02.445 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:02.439+0000 7ff1437fe640 1 --2- 172.21.15.29:0/1016454655 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561224fd5130 0x561224fdb8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:02:02.445 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:02.439+0000 7ff1437fe640 1 -- 172.21.15.29:0/1016454655 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561224c4e210 con 0x561224fd5a50 2024-03-16T12:02:02.446 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:02.440+0000 7ff1437fe640 1 --2- 172.21.15.29:0/1016454655 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561224fd5a50 0x561224fdbdf0 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7ff1380982a0 tx=0x7ff13806b460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c64e586b6a087ee server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:02:02.446 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:02.440+0000 7ff1237fe640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-16T12:02:02.446 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:02.440+0000 7ff1237fe640 1 -- 172.21.15.29:0/1016454655 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff13809f970 con 0x561224fd5a50 2024-03-16T12:02:02.446 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:02.440+0000 7ff1237fe640 1 -- 172.21.15.29:0/1016454655 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 8556+0+0 (secure 0 0 0) 0x7ff13809cb50 con 0x561224fd5a50 2024-03-16T12:02:02.446 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:02.440+0000 7ff14d276740 1 -- 172.21.15.29:0/1016454655 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff13c004d40 con 0x561224fd5a50 2024-03-16T12:02:02.446 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:02.440+0000 7ff1237fe640 1 -- 172.21.15.29:0/1016454655 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7ff1380a3fc0 con 0x561224fd5a50 2024-03-16T12:02:02.446 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:02.440+0000 7ff14d276740 10 client.16326 fetch_fsmap learned FSMap version 28 2024-03-16T12:02:02.446 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:02.440+0000 7ff14d276740 1 -- 172.21.15.29:0/1016454655 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x561224d1a620 con 0x561224fd5a50 2024-03-16T12:02:02.446 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:02.440+0000 7ff1237fe640 1 -- 172.21.15.29:0/1016454655 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff13c004d40 con 0x561224fd5a50 2024-03-16T12:02:02.446 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:02.440+0000 7ff1237fe640 1 -- 172.21.15.29:0/1016454655 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 28) ==== 8338+0+0 (secure 0 0 0) 0x7ff13809ad20 con 0x561224fd5a50 2024-03-16T12:02:02.446 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:02.440+0000 7ff14d276740 10 client.16326 fetch_fsmap finished waiting for FSMap version 28 2024-03-16T12:02:02.446 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:02.440+0000 7ff14d276740 10 client.16326 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T12:02:02.446 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:02.440+0000 7ff14d276740 20 client.16326 populate_metadata read hostname 'smithi029' 2024-03-16T12:02:02.447 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:02.440+0000 7ff14d276740 1 --2- 172.21.15.29:0/1016454655 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x561224feea50 0x561224ff0e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:02:02.447 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:02.440+0000 7ff14d276740 4 client.16326 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T12:02:02.447 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:02.440+0000 7ff14d276740 1 -- 172.21.15.29:0/1016454655 --> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561224f46f80 con 0x561224feea50 2024-03-16T12:02:02.447 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:02.441+0000 7ff143fff640 1 --2- 172.21.15.29:0/1016454655 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x561224feea50 0x561224ff0e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:02:02.447 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:02.441+0000 7ff143fff640 1 --2- 172.21.15.29:0/1016454655 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x561224feea50 0x561224ff0e40 crc :-1 s=READY pgs=138 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=83da93a376bdaa8e server_cookie=f97d10589d8816c7 in_seq=0 out_seq=0 2024-03-16T12:02:02.447 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:02.441+0000 7ff1237fe640 10 client.16326 ms_handle_connect on v2:172.21.15.152:6838/1548731833 2024-03-16T12:02:02.459 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:02.459+0000 7ff1237fe640 1 -- 172.21.15.29:0/1016454655 <== mds.0 v2:172.21.15.152:6838/1548731833 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x561224f46f80 con 0x561224feea50 2024-03-16T12:02:02.459 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:02.460+0000 7ff1237fe640 10 client.16326 handle_command_reply: tid=0 2024-03-16T12:02:02.459 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:02.460+0000 7ff14d276740 1 client.16326 shutdown 2024-03-16T12:02:02.459 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:02.460+0000 7ff14d276740 1 -- 172.21.15.29:0/1016454655 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561224fd5a50 msgr2=0x561224fdbdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:02:02.459 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:02.460+0000 7ff14d276740 1 --2- 172.21.15.29:0/1016454655 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561224fd5a50 0x561224fdbdf0 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7ff1380982a0 tx=0x7ff13806b460 comp rx=0 tx=0).stop 2024-03-16T12:02:02.459 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:02.460+0000 7ff14d276740 1 -- 172.21.15.29:0/1016454655 shutdown_connections 2024-03-16T12:02:02.459 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:02.460+0000 7ff14d276740 1 --2- 172.21.15.29:0/1016454655 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561224fde820 0x561224fdc330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:02:02.459 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:02.460+0000 7ff14d276740 1 --2- 172.21.15.29:0/1016454655 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561224fd5a50 0x561224fdbdf0 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:02:02.460 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:02.460+0000 7ff14d276740 1 --2- 172.21.15.29:0/1016454655 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x561224feea50 0x561224ff0e40 crc :-1 s=READY pgs=138 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:02:02.460 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:02.460+0000 7ff14d276740 1 --2- 172.21.15.29:0/1016454655 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561224fd5130 0x561224fdb8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:02:02.460 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:02.460+0000 7ff1237fe640 0 client.16326 ms_handle_reset on v2:172.21.15.152:6838/1548731833 2024-03-16T12:02:02.460 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:02.460+0000 7ff14d276740 1 -- 172.21.15.29:0/1016454655 >> 172.21.15.29:0/1016454655 conn(0x561224fcfa50 msgr2=0x561224fd17a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:02:02.460 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:02.460+0000 7ff14d276740 1 -- 172.21.15.29:0/1016454655 shutdown_connections 2024-03-16T12:02:02.460 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:02.460+0000 7ff14d276740 1 -- 172.21.15.29:0/1016454655 wait complete. 2024-03-16T12:02:02.460 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:02.460+0000 7ff14d276740 20 client.16326 trim_cache size 0 max 16384 2024-03-16T12:02:02.475 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:02.475+0000 7ff14d276740 1 Processor -- start 2024-03-16T12:02:02.475 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:02.475+0000 7ff14d276740 1 -- start start 2024-03-16T12:02:02.475 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:02.475+0000 7ff14d276740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561224fd5130 0x561224fddfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:02:02.475 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:02.475+0000 7ff14d276740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561224fd5a50 0x561224fea190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:02:02.475 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:02.475+0000 7ff14d276740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561224fde820 0x561224ff6660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:02:02.475 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:02.475+0000 7ff14d276740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7ff1441adad0 con 0x561224fde820 2024-03-16T12:02:02.475 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:02.475+0000 7ff14d276740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7ff1441ab3a0 con 0x561224fd5130 2024-03-16T12:02:02.476 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:02.475+0000 7ff14d276740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7ff1441a9230 con 0x561224fd5a50 2024-03-16T12:02:02.476 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:02.476+0000 7ff148c22640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561224fde820 0x561224ff6660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:02:02.476 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:02.476+0000 7ff1437fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561224fd5a50 0x561224fea190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:02:02.476 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:02.476+0000 7ff148c22640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561224fde820 0x561224ff6660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51702/0 (socket says 172.21.15.29:51702) 2024-03-16T12:02:02.476 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:02.476+0000 7ff143fff640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561224fd5130 0x561224fddfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:02:02.476 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:02.476+0000 7ff148c22640 1 -- 172.21.15.29:0/3385949703 learned_addr learned my addr 172.21.15.29:0/3385949703 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T12:02:02.476 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:02.476+0000 7ff148c22640 1 -- 172.21.15.29:0/3385949703 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561224fd5a50 msgr2=0x561224fea190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:02:02.476 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:02.476+0000 7ff148c22640 1 --2- 172.21.15.29:0/3385949703 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561224fd5a50 0x561224fea190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:02:02.476 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:02.476+0000 7ff148c22640 1 -- 172.21.15.29:0/3385949703 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561224fd5130 msgr2=0x561224fddfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:02:02.476 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:02.476+0000 7ff148c22640 1 --2- 172.21.15.29:0/3385949703 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561224fd5130 0x561224fddfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:02:02.476 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:02.476+0000 7ff148c22640 1 -- 172.21.15.29:0/3385949703 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561224fcb310 con 0x561224fde820 2024-03-16T12:02:02.476 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:02.476+0000 7ff148c22640 1 --2- 172.21.15.29:0/3385949703 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561224fde820 0x561224ff6660 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7ff13c009a60 tx=0x7ff13c03e040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d0880bb1e563cab server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:02:02.476 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:02.476+0000 7ff129ffb640 1 -- 172.21.15.29:0/3385949703 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff13c060070 con 0x561224fde820 2024-03-16T12:02:02.476 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:02.476+0000 7ff129ffb640 1 -- 172.21.15.29:0/3385949703 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff13c003e50 con 0x561224fde820 2024-03-16T12:02:02.476 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:02.476+0000 7ff129ffb640 1 -- 172.21.15.29:0/3385949703 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff13c03c7e0 con 0x561224fde820 2024-03-16T12:02:02.480 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:02.477+0000 7ff14d276740 1 -- 172.21.15.29:0/3385949703 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561224fde820 msgr2=0x561224ff6660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:02:02.480 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:02.477+0000 7ff14d276740 1 --2- 172.21.15.29:0/3385949703 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561224fde820 0x561224ff6660 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7ff13c009a60 tx=0x7ff13c03e040 comp rx=0 tx=0).stop 2024-03-16T12:02:02.480 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:02.477+0000 7ff14d276740 1 -- 172.21.15.29:0/3385949703 shutdown_connections 2024-03-16T12:02:02.480 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:02.477+0000 7ff14d276740 1 --2- 172.21.15.29:0/3385949703 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561224fde820 0x561224ff6660 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:02:02.480 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:02.477+0000 7ff14d276740 1 --2- 172.21.15.29:0/3385949703 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561224fd5a50 0x561224fea190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:02:02.480 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:02.477+0000 7ff14d276740 1 --2- 172.21.15.29:0/3385949703 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561224fd5130 0x561224fddfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:02:02.480 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:02.477+0000 7ff14d276740 1 -- 172.21.15.29:0/3385949703 >> 172.21.15.29:0/3385949703 conn(0x561224fcfa50 msgr2=0x561224fdb870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:02:02.480 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:02.477+0000 7ff14d276740 1 -- 172.21.15.29:0/3385949703 shutdown_connections 2024-03-16T12:02:02.480 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:02.477+0000 7ff14d276740 1 -- 172.21.15.29:0/3385949703 wait complete. 2024-03-16T12:02:02.481 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:02.477+0000 7ff14d276740 1 -- start start 2024-03-16T12:02:02.481 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:02.478+0000 7ff14d276740 1 Processor -- start 2024-03-16T12:02:02.481 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:02.478+0000 7ff14d276740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561224fd5130 0x561224fe21d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:02:02.481 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:02.478+0000 7ff14d276740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561224fd5a50 0x561224fe2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:02:02.481 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:02.478+0000 7ff14d276740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561224fde820 0x561224fe2c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:02:02.481 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:02.478+0000 7ff14d276740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7ff1441a9880 con 0x561224fd5a50 2024-03-16T12:02:02.481 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:02.478+0000 7ff14d276740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7ff1441aacd0 con 0x561224fde820 2024-03-16T12:02:02.481 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:02.478+0000 7ff14d276740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7ff1441a9f30 con 0x561224fd5130 2024-03-16T12:02:02.481 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:02.478+0000 7ff14d276740 10 client.0 _ll_register_callbacks cb 0x7ff13800cf20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T12:02:02.481 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:02.478+0000 7ff1437fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561224fd5a50 0x561224fe2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:02:02.481 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:02.478+0000 7ff1437fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561224fd5a50 0x561224fe2710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51718/0 (socket says 172.21.15.29:51718) 2024-03-16T12:02:02.481 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:02.478+0000 7ff1437fe640 1 -- 172.21.15.29:0/3780175565 learned_addr learned my addr 172.21.15.29:0/3780175565 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T12:02:02.481 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:02.478+0000 7ff148c22640 1 --2- 172.21.15.29:0/3780175565 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561224fde820 0x561224fe2c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:02:02.481 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:02.478+0000 7ff143fff640 1 --2- 172.21.15.29:0/3780175565 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561224fd5130 0x561224fe21d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:02:02.481 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:02.478+0000 7ff1437fe640 1 -- 172.21.15.29:0/3780175565 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561224fd5130 msgr2=0x561224fe21d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:02:02.481 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:02.478+0000 7ff1437fe640 1 --2- 172.21.15.29:0/3780175565 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561224fd5130 0x561224fe21d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:02:02.482 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:02.478+0000 7ff1437fe640 1 -- 172.21.15.29:0/3780175565 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561224fde820 msgr2=0x561224fe2c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:02:02.482 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:02.478+0000 7ff1437fe640 1 --2- 172.21.15.29:0/3780175565 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561224fde820 0x561224fe2c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:02:02.482 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:02.478+0000 7ff1437fe640 1 -- 172.21.15.29:0/3780175565 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561225053d90 con 0x561224fd5a50 2024-03-16T12:02:02.482 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:02.478+0000 7ff143fff640 1 --2- 172.21.15.29:0/3780175565 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561224fd5130 0x561224fe21d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:02:02.482 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:02.478+0000 7ff1437fe640 1 --2- 172.21.15.29:0/3780175565 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561224fd5a50 0x561224fe2710 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7ff13800cfd0 tx=0x7ff138018420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1e8eeb27f7e10f6 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:02:02.482 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:02.478+0000 7ff1237fe640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-16T12:02:02.482 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:02.479+0000 7ff1237fe640 1 -- 172.21.15.29:0/3780175565 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff13809f970 con 0x561224fd5a50 2024-03-16T12:02:02.482 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:02.479+0000 7ff1237fe640 1 -- 172.21.15.29:0/3780175565 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 8556+0+0 (secure 0 0 0) 0x7ff13806e1f0 con 0x561224fd5a50 2024-03-16T12:02:02.482 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:02.479+0000 7ff14d276740 1 -- 172.21.15.29:0/3780175565 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff13c003e50 con 0x561224fd5a50 2024-03-16T12:02:02.482 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:02.479+0000 7ff1237fe640 1 -- 172.21.15.29:0/3780175565 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7ff138097b70 con 0x561224fd5a50 2024-03-16T12:02:02.482 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:02.479+0000 7ff14d276740 10 client.16332 fetch_fsmap learned FSMap version 28 2024-03-16T12:02:02.482 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:02.479+0000 7ff14d276740 1 -- 172.21.15.29:0/3780175565 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5612250550a0 con 0x561224fd5a50 2024-03-16T12:02:02.482 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:02.479+0000 7ff1237fe640 1 -- 172.21.15.29:0/3780175565 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff13c003e50 con 0x561224fd5a50 2024-03-16T12:02:02.482 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:02.479+0000 7ff1237fe640 1 -- 172.21.15.29:0/3780175565 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 28) ==== 8338+0+0 (secure 0 0 0) 0x7ff13809a790 con 0x561224fd5a50 2024-03-16T12:02:02.482 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:02.479+0000 7ff14d276740 10 client.16332 fetch_fsmap finished waiting for FSMap version 28 2024-03-16T12:02:02.483 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:02.479+0000 7ff14d276740 10 client.16332 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T12:02:02.483 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:02.479+0000 7ff14d276740 20 client.16332 populate_metadata read hostname 'smithi029' 2024-03-16T12:02:02.483 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:02.479+0000 7ff14d276740 1 --2- 172.21.15.29:0/3780175565 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x5612250556c0 0x561225057ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:02:02.483 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:02.479+0000 7ff14d276740 4 client.16332 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-16T12:02:02.483 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:02.479+0000 7ff14d276740 1 -- 172.21.15.29:0/3780175565 --> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] -- command(tid 0: {"prefix": "flush journal"}) -- 0x561224f412d0 con 0x5612250556c0 2024-03-16T12:02:02.483 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:02.479+0000 7ff143fff640 1 --2- 172.21.15.29:0/3780175565 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x5612250556c0 0x561225057ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:02:02.483 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:02.480+0000 7ff1237fe640 10 client.16332 ms_handle_connect on v2:172.21.15.152:6838/1548731833 2024-03-16T12:02:02.483 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:02.480+0000 7ff143fff640 1 --2- 172.21.15.29:0/3780175565 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x5612250556c0 0x561225057ae0 crc :-1 s=READY pgs=139 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3e66aa8b92f3b3c7 server_cookie=e213ac106abfa6a8 in_seq=0 out_seq=0 2024-03-16T12:02:02.544 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:02:02 smithi154 ceph-mon[36320]: pgmap v350: 97 pgs: 97 active+clean; 76 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 3.2 MiB/s wr, 293 op/s 2024-03-16T12:02:02.589 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:02:02 smithi029 ceph-mon[31975]: pgmap v350: 97 pgs: 97 active+clean; 76 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 3.2 MiB/s wr, 293 op/s 2024-03-16T12:02:02.626 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:02:02 smithi152 ceph-mon[36291]: pgmap v350: 97 pgs: 97 active+clean; 76 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 3.2 MiB/s wr, 293 op/s 2024-03-16T12:02:02.664 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:02.665+0000 7ff1237fe640 1 -- 172.21.15.29:0/3780175565 <== mds.0 v2:172.21.15.152:6838/1548731833 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x561224f412d0 con 0x5612250556c0 2024-03-16T12:02:02.665 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:02.665+0000 7ff1237fe640 10 client.16332 handle_command_reply: tid=0 2024-03-16T12:02:02.665 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:02.665+0000 7ff14d276740 1 client.16332 shutdown 2024-03-16T12:02:02.665 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:02.665+0000 7ff14d276740 1 -- 172.21.15.29:0/3780175565 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561224fd5a50 msgr2=0x561224fe2710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:02:02.665 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:02.665+0000 7ff14d276740 1 --2- 172.21.15.29:0/3780175565 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561224fd5a50 0x561224fe2710 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7ff13800cfd0 tx=0x7ff138018420 comp rx=0 tx=0).stop 2024-03-16T12:02:02.665 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:02.666+0000 7ff14d276740 1 -- 172.21.15.29:0/3780175565 shutdown_connections 2024-03-16T12:02:02.665 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:02.666+0000 7ff14d276740 1 --2- 172.21.15.29:0/3780175565 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x5612250556c0 0x561225057ae0 crc :-1 s=READY pgs=139 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:02:02.665 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:02.666+0000 7ff1237fe640 0 client.16332 ms_handle_reset on v2:172.21.15.152:6838/1548731833 2024-03-16T12:02:02.665 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:02.666+0000 7ff14d276740 1 --2- 172.21.15.29:0/3780175565 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561224fde820 0x561224fe2c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:02:02.665 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:02.666+0000 7ff14d276740 1 --2- 172.21.15.29:0/3780175565 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561224fd5a50 0x561224fe2710 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:02:02.665 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:02.666+0000 7ff14d276740 1 --2- 172.21.15.29:0/3780175565 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561224fd5130 0x561224fe21d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:02:02.665 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:02.666+0000 7ff14d276740 1 -- 172.21.15.29:0/3780175565 >> 172.21.15.29:0/3780175565 conn(0x561224fcfa50 msgr2=0x561224fcebb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:02:02.665 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:02.666+0000 7ff14d276740 1 -- 172.21.15.29:0/3780175565 shutdown_connections 2024-03-16T12:02:02.665 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:02.666+0000 7ff14d276740 1 -- 172.21.15.29:0/3780175565 wait complete. 2024-03-16T12:02:02.665 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:02.666+0000 7ff14d276740 20 client.16332 trim_cache size 0 max 16384 2024-03-16T12:02:02.666 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:{ 2024-03-16T12:02:02.666 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "message": "", 2024-03-16T12:02:02.666 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "return_code": 0 2024-03-16T12:02:02.666 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:} 2024-03-16T12:02:02.667 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:02.667+0000 7ff14a685640 1 -- 172.21.15.29:0/857678049 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7ff12c0a2a10 msgr2=0x7ff12c0a4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:02:02.667 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:02.667+0000 7ff14a685640 1 --2- 172.21.15.29:0/857678049 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7ff12c0a2a10 0x7ff12c0a4ed0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7ff13c009ca0 tx=0x7ff13c046000 comp rx=0 tx=0).stop 2024-03-16T12:02:02.667 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:02.667+0000 7ff14a685640 1 -- 172.21.15.29:0/857678049 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff1441012d0 msgr2=0x7ff1441984e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:02:02.667 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:02.667+0000 7ff14a685640 1 --2- 172.21.15.29:0/857678049 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff1441012d0 0x7ff1441984e0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7ff13800e9f0 tx=0x7ff13800eeb0 comp rx=0 tx=0).stop 2024-03-16T12:02:02.667 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:02.667+0000 7ff14a685640 1 -- 172.21.15.29:0/857678049 shutdown_connections 2024-03-16T12:02:02.667 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:02.667+0000 7ff14a685640 1 --2- 172.21.15.29:0/857678049 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7ff12c0a2a10 0x7ff12c0a4ed0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:02:02.667 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:02.667+0000 7ff14a685640 1 --2- 172.21.15.29:0/857678049 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff144105660 0x7ff14419c810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:02:02.667 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:02.667+0000 7ff14a685640 1 --2- 172.21.15.29:0/857678049 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff1441012d0 0x7ff1441984e0 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:02:02.667 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:02.667+0000 7ff14a685640 1 --2- 172.21.15.29:0/857678049 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff144100930 0x7ff144197fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:02:02.667 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:02.667+0000 7ff14a685640 1 -- 172.21.15.29:0/857678049 >> 172.21.15.29:0/857678049 conn(0x7ff1440774b0 msgr2=0x7ff14406a120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:02:02.667 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:02.667+0000 7ff14a685640 1 -- 172.21.15.29:0/857678049 shutdown_connections 2024-03-16T12:02:02.667 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:02.667+0000 7ff14a685640 1 -- 172.21.15.29:0/857678049 wait complete. 2024-03-16T12:02:02.803 INFO:tasks.workunit.client.0.smithi029.stdout: 1 1083523 9.45 MB/sec execute 467 sec latency 120.526 ms 2024-03-16T12:02:03.803 INFO:tasks.workunit.client.0.smithi029.stdout: 1 1084805 9.44 MB/sec execute 468 sec latency 23.557 ms 2024-03-16T12:02:04.544 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:02:04 smithi154 ceph-mon[36320]: pgmap v351: 97 pgs: 97 active+clean; 76 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.2 MiB/s wr, 173 op/s 2024-03-16T12:02:04.587 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:02:04 smithi029 ceph-mon[31975]: pgmap v351: 97 pgs: 97 active+clean; 76 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.2 MiB/s wr, 173 op/s 2024-03-16T12:02:04.626 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:02:04 smithi152 ceph-mon[36291]: pgmap v351: 97 pgs: 97 active+clean; 76 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.2 MiB/s wr, 173 op/s 2024-03-16T12:02:04.803 INFO:tasks.workunit.client.0.smithi029.stdout: 1 1085681 9.42 MB/sec execute 469 sec latency 67.255 ms 2024-03-16T12:02:05.544 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:02:05 smithi154 ceph-mon[36320]: osdmap e103: 12 total, 12 up, 12 in 2024-03-16T12:02:05.587 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:02:05 smithi029 ceph-mon[31975]: osdmap e103: 12 total, 12 up, 12 in 2024-03-16T12:02:05.626 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:02:05 smithi152 ceph-mon[36291]: osdmap e103: 12 total, 12 up, 12 in 2024-03-16T12:02:05.803 INFO:tasks.workunit.client.0.smithi029.stdout: 1 1086406 9.42 MB/sec execute 470 sec latency 71.406 ms 2024-03-16T12:02:06.544 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:02:06 smithi154 ceph-mon[36320]: pgmap v353: 97 pgs: 97 active+clean; 76 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 2.6 MiB/s wr, 203 op/s 2024-03-16T12:02:06.587 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:02:06 smithi029 ceph-mon[31975]: pgmap v353: 97 pgs: 97 active+clean; 76 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 2.6 MiB/s wr, 203 op/s 2024-03-16T12:02:06.626 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:02:06 smithi152 ceph-mon[36291]: pgmap v353: 97 pgs: 97 active+clean; 76 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 2.6 MiB/s wr, 203 op/s 2024-03-16T12:02:06.803 INFO:tasks.workunit.client.0.smithi029.stdout: 1 1087102 9.41 MB/sec execute 471 sec latency 46.567 ms 2024-03-16T12:02:07.804 INFO:tasks.workunit.client.0.smithi029.stdout: 1 1088520 9.40 MB/sec execute 472 sec latency 21.912 ms 2024-03-16T12:02:08.544 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:02:08 smithi154 ceph-mon[36320]: pgmap v354: 97 pgs: 30 active+clean+snaptrim_wait, 14 active+clean+snaptrim, 53 active+clean; 64 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 2.7 MiB/s wr, 274 op/s 2024-03-16T12:02:08.587 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:02:08 smithi029 ceph-mon[31975]: pgmap v354: 97 pgs: 30 active+clean+snaptrim_wait, 14 active+clean+snaptrim, 53 active+clean; 64 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 2.7 MiB/s wr, 274 op/s 2024-03-16T12:02:08.626 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:02:08 smithi152 ceph-mon[36291]: pgmap v354: 97 pgs: 30 active+clean+snaptrim_wait, 14 active+clean+snaptrim, 53 active+clean; 64 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 2.7 MiB/s wr, 274 op/s 2024-03-16T12:02:08.803 INFO:tasks.workunit.client.0.smithi029.stdout: 1 1090591 9.41 MB/sec execute 473 sec latency 31.637 ms 2024-03-16T12:02:09.803 INFO:tasks.workunit.client.0.smithi029.stdout: 1 1091768 9.40 MB/sec execute 474 sec latency 29.242 ms 2024-03-16T12:02:10.544 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:02:10 smithi154 ceph-mon[36320]: pgmap v355: 97 pgs: 30 active+clean+snaptrim_wait, 14 active+clean+snaptrim, 53 active+clean; 64 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 2.7 MiB/s wr, 274 op/s 2024-03-16T12:02:10.587 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:02:10 smithi029 ceph-mon[31975]: pgmap v355: 97 pgs: 30 active+clean+snaptrim_wait, 14 active+clean+snaptrim, 53 active+clean; 64 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 2.7 MiB/s wr, 274 op/s 2024-03-16T12:02:10.626 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:02:10 smithi152 ceph-mon[36291]: pgmap v355: 97 pgs: 30 active+clean+snaptrim_wait, 14 active+clean+snaptrim, 53 active+clean; 64 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 2.7 MiB/s wr, 274 op/s 2024-03-16T12:02:10.804 INFO:tasks.workunit.client.0.smithi029.stdout: 1 1093522 9.40 MB/sec execute 475 sec latency 116.466 ms 2024-03-16T12:02:11.804 INFO:tasks.workunit.client.0.smithi029.stdout: 1 1094607 9.39 MB/sec execute 476 sec latency 17.832 ms 2024-03-16T12:02:12.544 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:02:12 smithi154 ceph-mon[36320]: pgmap v356: 97 pgs: 10 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 83 active+clean; 69 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.5 MiB/s wr, 236 op/s 2024-03-16T12:02:12.587 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:02:12 smithi029 ceph-mon[31975]: pgmap v356: 97 pgs: 10 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 83 active+clean; 69 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.5 MiB/s wr, 236 op/s 2024-03-16T12:02:12.626 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:02:12 smithi152 ceph-mon[36291]: pgmap v356: 97 pgs: 10 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 83 active+clean; 69 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.5 MiB/s wr, 236 op/s 2024-03-16T12:02:12.804 INFO:tasks.workunit.client.0.smithi029.stdout: 1 1096431 9.39 MB/sec execute 477 sec latency 22.662 ms 2024-03-16T12:02:13.544 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:02:13 smithi154 ceph-mon[36320]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T12:02:13.587 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:02:13 smithi029 ceph-mon[31975]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T12:02:13.626 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:02:13 smithi152 ceph-mon[36291]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T12:02:13.804 INFO:tasks.workunit.client.0.smithi029.stdout: 1 1097739 9.39 MB/sec execute 478 sec latency 91.421 ms 2024-03-16T12:02:14.794 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:02:14 smithi154 ceph-mon[36320]: pgmap v357: 97 pgs: 97 active+clean; 69 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 2.6 MiB/s wr, 236 op/s 2024-03-16T12:02:14.794 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:02:14 smithi154 ceph-mon[36320]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' 2024-03-16T12:02:14.794 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:02:14 smithi154 ceph-mon[36320]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' 2024-03-16T12:02:14.794 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:02:14 smithi154 ceph-mon[36320]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' 2024-03-16T12:02:14.794 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:02:14 smithi154 ceph-mon[36320]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' 2024-03-16T12:02:14.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:02:14 smithi154 ceph-mon[36320]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:02:14.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:02:14 smithi154 ceph-mon[36320]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T12:02:14.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:02:14 smithi154 ceph-mon[36320]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' 2024-03-16T12:02:14.803 INFO:tasks.workunit.client.0.smithi029.stdout: 1 1099025 9.39 MB/sec execute 479 sec latency 54.084 ms 2024-03-16T12:02:14.837 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:02:14 smithi029 ceph-mon[31975]: pgmap v357: 97 pgs: 97 active+clean; 69 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 2.6 MiB/s wr, 236 op/s 2024-03-16T12:02:14.837 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:02:14 smithi029 ceph-mon[31975]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' 2024-03-16T12:02:14.837 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:02:14 smithi029 ceph-mon[31975]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' 2024-03-16T12:02:14.837 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:02:14 smithi029 ceph-mon[31975]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' 2024-03-16T12:02:14.837 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:02:14 smithi029 ceph-mon[31975]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' 2024-03-16T12:02:14.837 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:02:14 smithi029 ceph-mon[31975]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:02:14.837 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:02:14 smithi029 ceph-mon[31975]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T12:02:14.837 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:02:14 smithi029 ceph-mon[31975]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' 2024-03-16T12:02:14.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:02:14 smithi152 ceph-mon[36291]: pgmap v357: 97 pgs: 97 active+clean; 69 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 2.6 MiB/s wr, 236 op/s 2024-03-16T12:02:14.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:02:14 smithi152 ceph-mon[36291]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' 2024-03-16T12:02:14.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:02:14 smithi152 ceph-mon[36291]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' 2024-03-16T12:02:14.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:02:14 smithi152 ceph-mon[36291]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' 2024-03-16T12:02:14.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:02:14 smithi152 ceph-mon[36291]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' 2024-03-16T12:02:14.877 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:02:14 smithi152 ceph-mon[36291]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:02:14.878 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:02:14 smithi152 ceph-mon[36291]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T12:02:14.878 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:02:14 smithi152 ceph-mon[36291]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' 2024-03-16T12:02:15.760 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.761+0000 7fd9b1001640 1 -- 172.21.15.29:0/2138569858 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd9ac1027c0 msgr2=0x7fd9ac102ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:02:15.760 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.761+0000 7fd9b1001640 1 --2- 172.21.15.29:0/2138569858 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd9ac1027c0 0x7fd9ac102ba0 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7fd99c009900 tx=0x7fd99c02bcd0 comp rx=0 tx=0).stop 2024-03-16T12:02:15.761 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.761+0000 7fd9b1001640 1 -- 172.21.15.29:0/2138569858 shutdown_connections 2024-03-16T12:02:15.761 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.761+0000 7fd9b1001640 1 --2- 172.21.15.29:0/2138569858 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd9ac10bbe0 0x7fd9ac10e080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:02:15.761 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.761+0000 7fd9b1001640 1 --2- 172.21.15.29:0/2138569858 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd9ac1030e0 0x7fd9ac10b5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:02:15.761 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.761+0000 7fd9b1001640 1 --2- 172.21.15.29:0/2138569858 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd9ac1027c0 0x7fd9ac102ba0 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:02:15.761 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.761+0000 7fd9b1001640 1 -- 172.21.15.29:0/2138569858 >> 172.21.15.29:0/2138569858 conn(0x7fd9ac0fc020 msgr2=0x7fd9ac0fe440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:02:15.761 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.761+0000 7fd9b1001640 1 -- 172.21.15.29:0/2138569858 shutdown_connections 2024-03-16T12:02:15.761 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.761+0000 7fd9b1001640 1 -- 172.21.15.29:0/2138569858 wait complete. 2024-03-16T12:02:15.761 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.762+0000 7fd9b1001640 1 Processor -- start 2024-03-16T12:02:15.761 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.762+0000 7fd9b1001640 1 -- start start 2024-03-16T12:02:15.763 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.762+0000 7fd9b1001640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd9ac1027c0 0x7fd9ac197f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:02:15.763 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.763+0000 7fd9b1001640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd9ac1030e0 0x7fd9ac198460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:02:15.763 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.763+0000 7fd9b1001640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd9ac10bbe0 0x7fd9ac19c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:02:15.763 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.763+0000 7fd9b1001640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fd9ac1119e0 con 0x7fd9ac1030e0 2024-03-16T12:02:15.763 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.763+0000 7fd9b1001640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fd9ac111860 con 0x7fd9ac10bbe0 2024-03-16T12:02:15.764 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.763+0000 7fd9b1001640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fd9ac111b60 con 0x7fd9ac1027c0 2024-03-16T12:02:15.764 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.763+0000 7fd9aa59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd9ac1030e0 0x7fd9ac198460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:02:15.764 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.763+0000 7fd9aa59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd9ac1030e0 0x7fd9ac198460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33900/0 (socket says 172.21.15.29:33900) 2024-03-16T12:02:15.764 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.763+0000 7fd9aa59c640 1 -- 172.21.15.29:0/695485121 learned_addr learned my addr 172.21.15.29:0/695485121 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T12:02:15.764 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.763+0000 7fd9aad9d640 1 --2- 172.21.15.29:0/695485121 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd9ac1027c0 0x7fd9ac197f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:02:15.764 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.763+0000 7fd9ab59e640 1 --2- 172.21.15.29:0/695485121 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd9ac10bbe0 0x7fd9ac19c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:02:15.764 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.763+0000 7fd9aa59c640 1 -- 172.21.15.29:0/695485121 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd9ac1027c0 msgr2=0x7fd9ac197f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:02:15.764 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.763+0000 7fd9aa59c640 1 --2- 172.21.15.29:0/695485121 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd9ac1027c0 0x7fd9ac197f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:02:15.764 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.763+0000 7fd9aa59c640 1 -- 172.21.15.29:0/695485121 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd9ac10bbe0 msgr2=0x7fd9ac19c7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:02:15.764 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.763+0000 7fd9aa59c640 1 --2- 172.21.15.29:0/695485121 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd9ac10bbe0 0x7fd9ac19c7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:02:15.764 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.763+0000 7fd9aa59c640 1 -- 172.21.15.29:0/695485121 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9ac19cf10 con 0x7fd9ac1030e0 2024-03-16T12:02:15.764 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.763+0000 7fd9aad9d640 1 --2- 172.21.15.29:0/695485121 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd9ac1027c0 0x7fd9ac197f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:02:15.764 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.763+0000 7fd9aa59c640 1 --2- 172.21.15.29:0/695485121 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd9ac1030e0 0x7fd9ac198460 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7fd9a000e9f0 tx=0x7fd9a000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:02:15.764 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.763+0000 7fd993fff640 1 -- 172.21.15.29:0/695485121 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9a000ccc0 con 0x7fd9ac1030e0 2024-03-16T12:02:15.765 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.764+0000 7fd9b1001640 1 -- 172.21.15.29:0/695485121 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd9ac19d200 con 0x7fd9ac1030e0 2024-03-16T12:02:15.765 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.764+0000 7fd993fff640 1 -- 172.21.15.29:0/695485121 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd9a000ce60 con 0x7fd9ac1030e0 2024-03-16T12:02:15.765 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.764+0000 7fd993fff640 1 -- 172.21.15.29:0/695485121 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9a0010690 con 0x7fd9ac1030e0 2024-03-16T12:02:15.765 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.764+0000 7fd9b1001640 1 -- 172.21.15.29:0/695485121 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9ac1a4a90 con 0x7fd9ac1030e0 2024-03-16T12:02:15.771 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.766+0000 7fd993fff640 1 -- 172.21.15.29:0/695485121 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7fd9a00046c0 con 0x7fd9ac1030e0 2024-03-16T12:02:15.771 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.766+0000 7fd9b3bdf740 1 Processor -- start 2024-03-16T12:02:15.771 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.766+0000 7fd9b3bdf740 1 -- start start 2024-03-16T12:02:15.771 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.766+0000 7fd993fff640 1 --2- 172.21.15.29:0/695485121 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7fd9940a2960 0x7fd9940a4e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:02:15.771 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.767+0000 7fd9b3bdf740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556c93a902c0 0x556c93a906a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:02:15.771 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.767+0000 7fd9b3bdf740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556c93a90be0 0x556c93a99470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:02:15.771 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.767+0000 7fd9b3bdf740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556c93a999b0 0x556c93a9bda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:02:15.771 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.767+0000 7fd9b3bdf740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x556c939f4a40 con 0x556c93a902c0 2024-03-16T12:02:15.771 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.767+0000 7fd9b3bdf740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x556c9399d920 con 0x556c93a90be0 2024-03-16T12:02:15.771 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.767+0000 7fd9b3bdf740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x556c939f0ae0 con 0x556c93a999b0 2024-03-16T12:02:15.772 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.767+0000 7fd9ab59e640 1 --2- 172.21.15.29:0/695485121 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7fd9940a2960 0x7fd9940a4e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:02:15.772 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.767+0000 7fd993fff640 1 -- 172.21.15.29:0/695485121 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 8600+0+0 (secure 0 0 0) 0x7fd9a00cf130 con 0x7fd9ac1030e0 2024-03-16T12:02:15.772 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.767+0000 7fd9aad9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556c93a902c0 0x556c93a906a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:02:15.772 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.767+0000 7fd9aa59c640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556c93a90be0 0x556c93a99470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:02:15.772 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.767+0000 7fd9aad9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556c93a902c0 0x556c93a906a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33910/0 (socket says 172.21.15.29:33910) 2024-03-16T12:02:15.772 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.767+0000 7fd9aa59c640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556c93a90be0 0x556c93a99470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.29:59940/0 (socket says 172.21.15.29:59940) 2024-03-16T12:02:15.772 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.767+0000 7fd9aad9d640 1 -- 172.21.15.29:0/1263527859 learned_addr learned my addr 172.21.15.29:0/1263527859 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T12:02:15.772 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.767+0000 7fd9ab59e640 1 --2- 172.21.15.29:0/1263527859 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556c93a999b0 0x556c93a9bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:02:15.772 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.767+0000 7fd9aa59c640 1 -- 172.21.15.29:0/1263527859 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556c93a999b0 msgr2=0x556c93a9bda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:02:15.772 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.767+0000 7fd9aa59c640 1 --2- 172.21.15.29:0/1263527859 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556c93a999b0 0x556c93a9bda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:02:15.772 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.767+0000 7fd9aa59c640 1 -- 172.21.15.29:0/1263527859 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556c93a902c0 msgr2=0x556c93a906a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:02:15.772 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.767+0000 7fd9aa59c640 1 --2- 172.21.15.29:0/1263527859 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556c93a902c0 0x556c93a906a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:02:15.772 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.767+0000 7fd9aa59c640 1 -- 172.21.15.29:0/1263527859 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556c939b5460 con 0x556c93a90be0 2024-03-16T12:02:15.772 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.767+0000 7fd9aad9d640 1 --2- 172.21.15.29:0/1263527859 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556c93a902c0 0x556c93a906a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:02:15.773 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.767+0000 7fd9ab59e640 1 --2- 172.21.15.29:0/695485121 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7fd9940a2960 0x7fd9940a4e20 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fd9a4009fd0 tx=0x7fd9a4009d60 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:02:15.773 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.767+0000 7fd9aa59c640 1 --2- 172.21.15.29:0/1263527859 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556c93a90be0 0x556c93a99470 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fd9a00cc8d0 tx=0x7fd9a0097480 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7d66a5e64692b101 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:02:15.773 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.768+0000 7fd9917fa640 1 -- 172.21.15.29:0/1263527859 <== mon.2 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9a001f670 con 0x556c93a90be0 2024-03-16T12:02:15.773 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.768+0000 7fd9917fa640 1 -- 172.21.15.29:0/1263527859 <== mon.2 v2:172.21.15.152:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd9a001fe70 con 0x556c93a90be0 2024-03-16T12:02:15.773 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.768+0000 7fd9917fa640 1 -- 172.21.15.29:0/1263527859 <== mon.2 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9a001f340 con 0x556c93a90be0 2024-03-16T12:02:15.773 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.768+0000 7fd9b3bdf740 1 -- 172.21.15.29:0/1263527859 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556c93a90be0 msgr2=0x556c93a99470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:02:15.773 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.768+0000 7fd9b3bdf740 1 --2- 172.21.15.29:0/1263527859 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556c93a90be0 0x556c93a99470 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fd9a00cc8d0 tx=0x7fd9a0097480 comp rx=0 tx=0).stop 2024-03-16T12:02:15.773 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.768+0000 7fd9b3bdf740 1 -- 172.21.15.29:0/1263527859 shutdown_connections 2024-03-16T12:02:15.773 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.768+0000 7fd9b3bdf740 1 --2- 172.21.15.29:0/1263527859 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556c93a999b0 0x556c93a9bda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:02:15.773 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.768+0000 7fd9b3bdf740 1 --2- 172.21.15.29:0/1263527859 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556c93a90be0 0x556c93a99470 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:02:15.773 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.768+0000 7fd9b3bdf740 1 --2- 172.21.15.29:0/1263527859 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556c93a902c0 0x556c93a906a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:02:15.773 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.769+0000 7fd9b3bdf740 1 -- 172.21.15.29:0/1263527859 >> 172.21.15.29:0/1263527859 conn(0x556c93a8abe0 msgr2=0x556c93a8afc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:02:15.773 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.769+0000 7fd9b3bdf740 1 -- 172.21.15.29:0/1263527859 shutdown_connections 2024-03-16T12:02:15.774 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.769+0000 7fd9b3bdf740 1 -- 172.21.15.29:0/1263527859 wait complete. 2024-03-16T12:02:15.774 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.769+0000 7fd9b3bdf740 1 -- start start 2024-03-16T12:02:15.774 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.769+0000 7fd9b3bdf740 1 Processor -- start 2024-03-16T12:02:15.774 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.769+0000 7fd9b3bdf740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556c93a902c0 0x556c93a96a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:02:15.774 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.769+0000 7fd9b3bdf740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556c93a90be0 0x556c93a96fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:02:15.774 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.769+0000 7fd9b3bdf740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556c93a999b0 0x556c93a97510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:02:15.774 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.769+0000 7fd9b3bdf740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x556c93a9c5e0 con 0x556c93a999b0 2024-03-16T12:02:15.774 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.769+0000 7fd9b3bdf740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x556c939b0d70 con 0x556c93a902c0 2024-03-16T12:02:15.774 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.769+0000 7fd9b3bdf740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x556c93a9c760 con 0x556c93a90be0 2024-03-16T12:02:15.774 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.769+0000 7fd9b3bdf740 10 client.0 _ll_register_callbacks cb 0x556c93708a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T12:02:15.774 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.769+0000 7fd9aa59c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556c93a90be0 0x556c93a96fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:02:15.774 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.769+0000 7fd9ab59e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556c93a999b0 0x556c93a97510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:02:15.774 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.770+0000 7fd9aa59c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556c93a90be0 0x556c93a96fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.29:36746/0 (socket says 172.21.15.29:36746) 2024-03-16T12:02:15.774 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.770+0000 7fd9aad9d640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556c93a902c0 0x556c93a96a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:02:15.775 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.770+0000 7fd9ab59e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556c93a999b0 0x556c93a97510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33918/0 (socket says 172.21.15.29:33918) 2024-03-16T12:02:15.775 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.770+0000 7fd9aa59c640 1 -- 172.21.15.29:0/2532369262 learned_addr learned my addr 172.21.15.29:0/2532369262 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T12:02:15.775 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.770+0000 7fd9ab59e640 1 -- 172.21.15.29:0/2532369262 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556c93a90be0 msgr2=0x556c93a96fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:02:15.775 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.770+0000 7fd9ab59e640 1 --2- 172.21.15.29:0/2532369262 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556c93a90be0 0x556c93a96fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:02:15.775 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.770+0000 7fd9ab59e640 1 -- 172.21.15.29:0/2532369262 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556c93a902c0 msgr2=0x556c93a96a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:02:15.775 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.770+0000 7fd9ab59e640 1 --2- 172.21.15.29:0/2532369262 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556c93a902c0 0x556c93a96a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:02:15.775 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.770+0000 7fd9ab59e640 1 -- 172.21.15.29:0/2532369262 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556c93709210 con 0x556c93a999b0 2024-03-16T12:02:15.775 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.770+0000 7fd9aa59c640 1 --2- 172.21.15.29:0/2532369262 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556c93a90be0 0x556c93a96fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:02:15.775 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.770+0000 7fd9ab59e640 1 --2- 172.21.15.29:0/2532369262 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556c93a999b0 0x556c93a97510 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x556c93a8bd70 tx=0x7fd9a4004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f2f3beecf52c42da server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:02:15.775 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.770+0000 7fd98eff5640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-16T12:02:15.775 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.770+0000 7fd98eff5640 1 -- 172.21.15.29:0/2532369262 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9a4025070 con 0x556c93a999b0 2024-03-16T12:02:15.775 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.770+0000 7fd98eff5640 1 -- 172.21.15.29:0/2532369262 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 8600+0+0 (secure 0 0 0) 0x7fd9a4004b20 con 0x556c93a999b0 2024-03-16T12:02:15.775 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.770+0000 7fd9b3bdf740 1 -- 172.21.15.29:0/2532369262 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x556c93a01f80 con 0x556c93a999b0 2024-03-16T12:02:15.775 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.770+0000 7fd98eff5640 1 -- 172.21.15.29:0/2532369262 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fd9a4004d60 con 0x556c93a999b0 2024-03-16T12:02:15.775 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.770+0000 7fd9b3bdf740 10 client.16344 fetch_fsmap learned FSMap version 28 2024-03-16T12:02:15.776 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.770+0000 7fd9b3bdf740 1 -- 172.21.15.29:0/2532369262 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x556c937d5620 con 0x556c93a999b0 2024-03-16T12:02:15.776 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.771+0000 7fd98eff5640 1 -- 172.21.15.29:0/2532369262 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x556c93a01f80 con 0x556c93a999b0 2024-03-16T12:02:15.776 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.771+0000 7fd98eff5640 1 -- 172.21.15.29:0/2532369262 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 28) ==== 8338+0+0 (secure 0 0 0) 0x7fd9a4020050 con 0x556c93a999b0 2024-03-16T12:02:15.776 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.771+0000 7fd9b3bdf740 10 client.16344 fetch_fsmap finished waiting for FSMap version 28 2024-03-16T12:02:15.776 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.771+0000 7fd9b3bdf740 10 client.16344 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T12:02:15.776 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.771+0000 7fd9b3bdf740 20 client.16344 populate_metadata read hostname 'smithi029' 2024-03-16T12:02:15.776 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.771+0000 7fd9b3bdf740 1 --2- 172.21.15.29:0/2532369262 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x556c93aa9bf0 0x556c93aabfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:02:15.776 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.771+0000 7fd9b3bdf740 4 client.16344 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T12:02:15.776 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.771+0000 7fd9b3bdf740 1 -- 172.21.15.29:0/2532369262 --> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556c939fc2d0 con 0x556c93aa9bf0 2024-03-16T12:02:15.776 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.771+0000 7fd9aad9d640 1 --2- 172.21.15.29:0/2532369262 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x556c93aa9bf0 0x556c93aabfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:02:15.776 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.771+0000 7fd98eff5640 10 client.16344 ms_handle_connect on v2:172.21.15.152:6838/1548731833 2024-03-16T12:02:15.776 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.771+0000 7fd9aad9d640 1 --2- 172.21.15.29:0/2532369262 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x556c93aa9bf0 0x556c93aabfe0 crc :-1 s=READY pgs=140 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7c38429bb25adcc2 server_cookie=587cfab2086aeed1 in_seq=0 out_seq=0 2024-03-16T12:02:15.777 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.774+0000 7fd98eff5640 1 -- 172.21.15.29:0/2532369262 <== mds.0 v2:172.21.15.152:6838/1548731833 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x556c939fc2d0 con 0x556c93aa9bf0 2024-03-16T12:02:15.777 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.774+0000 7fd98eff5640 10 client.16344 handle_command_reply: tid=0 2024-03-16T12:02:15.777 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.774+0000 7fd9b3bdf740 1 client.16344 shutdown 2024-03-16T12:02:15.778 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.774+0000 7fd9b3bdf740 1 -- 172.21.15.29:0/2532369262 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556c93a999b0 msgr2=0x556c93a97510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:02:15.778 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.774+0000 7fd9b3bdf740 1 --2- 172.21.15.29:0/2532369262 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556c93a999b0 0x556c93a97510 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x556c93a8bd70 tx=0x7fd9a4004ab0 comp rx=0 tx=0).stop 2024-03-16T12:02:15.778 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.774+0000 7fd9b3bdf740 1 -- 172.21.15.29:0/2532369262 shutdown_connections 2024-03-16T12:02:15.778 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.774+0000 7fd9b3bdf740 1 --2- 172.21.15.29:0/2532369262 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556c93a999b0 0x556c93a97510 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:02:15.778 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.774+0000 7fd9b3bdf740 1 --2- 172.21.15.29:0/2532369262 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556c93a90be0 0x556c93a96fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:02:15.778 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.774+0000 7fd9b3bdf740 1 --2- 172.21.15.29:0/2532369262 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x556c93aa9bf0 0x556c93aabfe0 crc :-1 s=READY pgs=140 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:02:15.778 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.774+0000 7fd9b3bdf740 1 --2- 172.21.15.29:0/2532369262 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556c93a902c0 0x556c93a96a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:02:15.778 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.774+0000 7fd98eff5640 0 client.16344 ms_handle_reset on v2:172.21.15.152:6838/1548731833 2024-03-16T12:02:15.778 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.774+0000 7fd9b3bdf740 1 -- 172.21.15.29:0/2532369262 >> 172.21.15.29:0/2532369262 conn(0x556c93a8abe0 msgr2=0x556c93a8c930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:02:15.778 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.774+0000 7fd9b3bdf740 1 -- 172.21.15.29:0/2532369262 shutdown_connections 2024-03-16T12:02:15.778 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.774+0000 7fd9b3bdf740 1 -- 172.21.15.29:0/2532369262 wait complete. 2024-03-16T12:02:15.778 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.775+0000 7fd9b3bdf740 20 client.16344 trim_cache size 0 max 16384 2024-03-16T12:02:15.791 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.791+0000 7fd9b3bdf740 1 Processor -- start 2024-03-16T12:02:15.791 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.792+0000 7fd9b3bdf740 1 -- start start 2024-03-16T12:02:15.791 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.792+0000 7fd9b3bdf740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556c93a902c0 0x556c93a991c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:02:15.791 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.792+0000 7fd9b3bdf740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556c93a90be0 0x556c93aa5380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:02:15.792 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.792+0000 7fd9b3bdf740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556c93a999b0 0x556c93ab18b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:02:15.792 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.792+0000 7fd9b3bdf740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fd9ac1afd20 con 0x556c93a90be0 2024-03-16T12:02:15.792 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.792+0000 7fd9b3bdf740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fd9ac1ad5f0 con 0x556c93a902c0 2024-03-16T12:02:15.792 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.792+0000 7fd9b3bdf740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fd9ac1ab460 con 0x556c93a999b0 2024-03-16T12:02:15.792 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.792+0000 7fd9ab59e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556c93a999b0 0x556c93ab18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:02:15.792 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.792+0000 7fd9aa59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556c93a90be0 0x556c93aa5380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:02:15.792 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.792+0000 7fd9ab59e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556c93a999b0 0x556c93ab18b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.29:36748/0 (socket says 172.21.15.29:36748) 2024-03-16T12:02:15.792 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.792+0000 7fd9ab59e640 1 -- 172.21.15.29:0/135387355 learned_addr learned my addr 172.21.15.29:0/135387355 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T12:02:15.792 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.792+0000 7fd9aad9d640 1 --2- 172.21.15.29:0/135387355 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556c93a902c0 0x556c93a991c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:02:15.792 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.792+0000 7fd9ab59e640 1 -- 172.21.15.29:0/135387355 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556c93a902c0 msgr2=0x556c93a991c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:02:15.792 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.792+0000 7fd9ab59e640 1 --2- 172.21.15.29:0/135387355 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556c93a902c0 0x556c93a991c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:02:15.792 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.792+0000 7fd9ab59e640 1 -- 172.21.15.29:0/135387355 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556c93a90be0 msgr2=0x556c93aa5380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:02:15.792 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.792+0000 7fd9ab59e640 1 --2- 172.21.15.29:0/135387355 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556c93a90be0 0x556c93aa5380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:02:15.792 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.792+0000 7fd9ab59e640 1 -- 172.21.15.29:0/135387355 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556c93a9c0d0 con 0x556c93a999b0 2024-03-16T12:02:15.793 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.793+0000 7fd9ab59e640 1 --2- 172.21.15.29:0/135387355 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556c93a999b0 0x556c93ab18b0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fd9a4004620 tx=0x7fd9a400fb70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=53e902e67dd826cc server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:02:15.793 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.793+0000 7fd990ff9640 1 -- 172.21.15.29:0/135387355 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9a4035070 con 0x556c93a999b0 2024-03-16T12:02:15.793 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.793+0000 7fd990ff9640 1 -- 172.21.15.29:0/135387355 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd9a40138f0 con 0x556c93a999b0 2024-03-16T12:02:15.793 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.793+0000 7fd990ff9640 1 -- 172.21.15.29:0/135387355 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9a4025070 con 0x556c93a999b0 2024-03-16T12:02:15.796 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.793+0000 7fd9b3bdf740 1 -- 172.21.15.29:0/135387355 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556c93a999b0 msgr2=0x556c93ab18b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:02:15.796 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.793+0000 7fd9b3bdf740 1 --2- 172.21.15.29:0/135387355 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556c93a999b0 0x556c93ab18b0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fd9a4004620 tx=0x7fd9a400fb70 comp rx=0 tx=0).stop 2024-03-16T12:02:15.796 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.794+0000 7fd9b3bdf740 1 -- 172.21.15.29:0/135387355 shutdown_connections 2024-03-16T12:02:15.796 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.794+0000 7fd9b3bdf740 1 --2- 172.21.15.29:0/135387355 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556c93a999b0 0x556c93ab18b0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:02:15.796 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.794+0000 7fd9b3bdf740 1 --2- 172.21.15.29:0/135387355 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556c93a90be0 0x556c93aa5380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:02:15.796 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.794+0000 7fd9b3bdf740 1 --2- 172.21.15.29:0/135387355 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556c93a902c0 0x556c93a991c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:02:15.796 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.794+0000 7fd9b3bdf740 1 -- 172.21.15.29:0/135387355 >> 172.21.15.29:0/135387355 conn(0x556c93a8abe0 msgr2=0x556c93a89c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:02:15.797 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.794+0000 7fd9b3bdf740 1 -- 172.21.15.29:0/135387355 shutdown_connections 2024-03-16T12:02:15.797 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.794+0000 7fd9b3bdf740 1 -- 172.21.15.29:0/135387355 wait complete. 2024-03-16T12:02:15.797 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.794+0000 7fd9b3bdf740 1 -- start start 2024-03-16T12:02:15.797 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.794+0000 7fd9b3bdf740 1 Processor -- start 2024-03-16T12:02:15.797 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.794+0000 7fd9b3bdf740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556c93a902c0 0x556c93a9d520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:02:15.797 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.794+0000 7fd9b3bdf740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556c93a90be0 0x556c93a9da60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:02:15.797 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.794+0000 7fd9b3bdf740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556c93a999b0 0x556c93a9dfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:02:15.797 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.794+0000 7fd9b3bdf740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fd9ac1abab0 con 0x556c93a90be0 2024-03-16T12:02:15.797 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.794+0000 7fd9b3bdf740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fd9ac1acf20 con 0x556c93a999b0 2024-03-16T12:02:15.797 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.794+0000 7fd9b3bdf740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fd9ac1ac180 con 0x556c93a902c0 2024-03-16T12:02:15.797 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.794+0000 7fd9b3bdf740 10 client.0 _ll_register_callbacks cb 0x7fd9a4013520 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T12:02:15.797 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.794+0000 7fd9aa59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556c93a90be0 0x556c93a9da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:02:15.797 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.794+0000 7fd9aa59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556c93a90be0 0x556c93a9da60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33938/0 (socket says 172.21.15.29:33938) 2024-03-16T12:02:15.797 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.794+0000 7fd9aa59c640 1 -- 172.21.15.29:0/992731040 learned_addr learned my addr 172.21.15.29:0/992731040 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T12:02:15.798 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.794+0000 7fd9aad9d640 1 --2- 172.21.15.29:0/992731040 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556c93a902c0 0x556c93a9d520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:02:15.798 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.795+0000 7fd9ab59e640 1 --2- 172.21.15.29:0/992731040 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556c93a999b0 0x556c93a9dfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:02:15.798 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.795+0000 7fd9aa59c640 1 -- 172.21.15.29:0/992731040 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556c93a902c0 msgr2=0x556c93a9d520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:02:15.798 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.795+0000 7fd9aa59c640 1 --2- 172.21.15.29:0/992731040 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556c93a902c0 0x556c93a9d520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:02:15.798 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.795+0000 7fd9aa59c640 1 -- 172.21.15.29:0/992731040 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556c93a999b0 msgr2=0x556c93a9dfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:02:15.798 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.795+0000 7fd9aa59c640 1 --2- 172.21.15.29:0/992731040 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556c93a999b0 0x556c93a9dfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:02:15.798 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.795+0000 7fd9aa59c640 1 -- 172.21.15.29:0/992731040 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556c93b0f130 con 0x556c93a90be0 2024-03-16T12:02:15.798 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.795+0000 7fd9ab59e640 1 --2- 172.21.15.29:0/992731040 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556c93a999b0 0x556c93a9dfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T12:02:15.798 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.795+0000 7fd9aa59c640 1 --2- 172.21.15.29:0/992731040 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556c93a90be0 0x556c93a9da60 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7fd9a0099a90 tx=0x7fd9a0009310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e1eda039e7b821e server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:02:15.798 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.795+0000 7fd98eff5640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-16T12:02:15.798 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.795+0000 7fd98eff5640 1 -- 172.21.15.29:0/992731040 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9a00b8070 con 0x556c93a90be0 2024-03-16T12:02:15.798 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.795+0000 7fd98eff5640 1 -- 172.21.15.29:0/992731040 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 8600+0+0 (secure 0 0 0) 0x7fd9a0097e50 con 0x556c93a90be0 2024-03-16T12:02:15.798 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.795+0000 7fd9b3bdf740 1 -- 172.21.15.29:0/992731040 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd9a40138f0 con 0x556c93a90be0 2024-03-16T12:02:15.798 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.796+0000 7fd98eff5640 1 -- 172.21.15.29:0/992731040 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fd9a00a3f80 con 0x556c93a90be0 2024-03-16T12:02:15.799 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.796+0000 7fd9b3bdf740 10 client.16350 fetch_fsmap learned FSMap version 28 2024-03-16T12:02:15.799 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.796+0000 7fd9b3bdf740 1 -- 172.21.15.29:0/992731040 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x556c93b10400 con 0x556c93a90be0 2024-03-16T12:02:15.799 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.796+0000 7fd98eff5640 1 -- 172.21.15.29:0/992731040 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd9a40138f0 con 0x556c93a90be0 2024-03-16T12:02:15.799 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.796+0000 7fd98eff5640 1 -- 172.21.15.29:0/992731040 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 28) ==== 8338+0+0 (secure 0 0 0) 0x7fd9a00afbc0 con 0x556c93a90be0 2024-03-16T12:02:15.799 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.796+0000 7fd9b3bdf740 10 client.16350 fetch_fsmap finished waiting for FSMap version 28 2024-03-16T12:02:15.799 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.796+0000 7fd9b3bdf740 10 client.16350 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T12:02:15.799 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.796+0000 7fd9b3bdf740 20 client.16350 populate_metadata read hostname 'smithi029' 2024-03-16T12:02:15.799 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.796+0000 7fd9b3bdf740 1 --2- 172.21.15.29:0/992731040 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x556c93b10740 0x556c93b12b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:02:15.799 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.796+0000 7fd9b3bdf740 4 client.16350 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-16T12:02:15.799 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.796+0000 7fd9b3bdf740 1 -- 172.21.15.29:0/992731040 --> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] -- command(tid 0: {"prefix": "flush journal"}) -- 0x556c9397a770 con 0x556c93b10740 2024-03-16T12:02:15.799 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.796+0000 7fd9aad9d640 1 --2- 172.21.15.29:0/992731040 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x556c93b10740 0x556c93b12b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:02:15.799 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.797+0000 7fd9aad9d640 1 --2- 172.21.15.29:0/992731040 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x556c93b10740 0x556c93b12b30 crc :-1 s=READY pgs=141 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a6c9d6657f10a21d server_cookie=4246bf4a5c5773f8 in_seq=0 out_seq=0 2024-03-16T12:02:15.799 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.797+0000 7fd98eff5640 10 client.16350 ms_handle_connect on v2:172.21.15.152:6838/1548731833 2024-03-16T12:02:15.803 INFO:tasks.workunit.client.0.smithi029.stdout: 1 1100941 9.39 MB/sec execute 480 sec latency 91.536 ms 2024-03-16T12:02:15.960 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.960+0000 7fd98eff5640 1 -- 172.21.15.29:0/992731040 <== mds.0 v2:172.21.15.152:6838/1548731833 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x556c9397a770 con 0x556c93b10740 2024-03-16T12:02:15.960 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.960+0000 7fd98eff5640 10 client.16350 handle_command_reply: tid=0 2024-03-16T12:02:15.960 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.960+0000 7fd9b3bdf740 1 client.16350 shutdown 2024-03-16T12:02:15.960 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.960+0000 7fd9b3bdf740 1 -- 172.21.15.29:0/992731040 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556c93a90be0 msgr2=0x556c93a9da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:02:15.960 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.960+0000 7fd9b3bdf740 1 --2- 172.21.15.29:0/992731040 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556c93a90be0 0x556c93a9da60 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7fd9a0099a90 tx=0x7fd9a0009310 comp rx=0 tx=0).stop 2024-03-16T12:02:15.960 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.960+0000 7fd9b3bdf740 1 -- 172.21.15.29:0/992731040 shutdown_connections 2024-03-16T12:02:15.961 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.960+0000 7fd9b3bdf740 1 --2- 172.21.15.29:0/992731040 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x556c93b10740 0x556c93b12b30 crc :-1 s=READY pgs=141 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:02:15.961 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.960+0000 7fd98eff5640 0 client.16350 ms_handle_reset on v2:172.21.15.152:6838/1548731833 2024-03-16T12:02:15.961 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.960+0000 7fd9b3bdf740 1 --2- 172.21.15.29:0/992731040 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556c93a999b0 0x556c93a9dfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:02:15.961 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.960+0000 7fd9b3bdf740 1 --2- 172.21.15.29:0/992731040 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556c93a90be0 0x556c93a9da60 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:02:15.961 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.960+0000 7fd9b3bdf740 1 --2- 172.21.15.29:0/992731040 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556c93a902c0 0x556c93a9d520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:02:15.961 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.960+0000 7fd9b3bdf740 1 -- 172.21.15.29:0/992731040 >> 172.21.15.29:0/992731040 conn(0x556c93a8abe0 msgr2=0x556c93a893d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:02:15.961 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.961+0000 7fd9b3bdf740 1 -- 172.21.15.29:0/992731040 shutdown_connections 2024-03-16T12:02:15.961 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.961+0000 7fd9b3bdf740 1 -- 172.21.15.29:0/992731040 wait complete. 2024-03-16T12:02:15.961 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.961+0000 7fd9b3bdf740 20 client.16350 trim_cache size 0 max 16384 2024-03-16T12:02:15.962 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:{ 2024-03-16T12:02:15.962 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "message": "", 2024-03-16T12:02:15.962 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "return_code": 0 2024-03-16T12:02:15.962 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:} 2024-03-16T12:02:15.963 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.961+0000 7fd9b1001640 1 -- 172.21.15.29:0/695485121 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7fd9940a2960 msgr2=0x7fd9940a4e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:02:15.963 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.961+0000 7fd9b1001640 1 --2- 172.21.15.29:0/695485121 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7fd9940a2960 0x7fd9940a4e20 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fd9a4009fd0 tx=0x7fd9a4009d60 comp rx=0 tx=0).stop 2024-03-16T12:02:15.963 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.961+0000 7fd9b1001640 1 -- 172.21.15.29:0/695485121 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd9ac1030e0 msgr2=0x7fd9ac198460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:02:15.963 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.961+0000 7fd9b1001640 1 --2- 172.21.15.29:0/695485121 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd9ac1030e0 0x7fd9ac198460 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7fd9a000e9f0 tx=0x7fd9a000eeb0 comp rx=0 tx=0).stop 2024-03-16T12:02:15.963 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.962+0000 7fd9b1001640 1 -- 172.21.15.29:0/695485121 shutdown_connections 2024-03-16T12:02:15.963 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.962+0000 7fd9b1001640 1 --2- 172.21.15.29:0/695485121 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7fd9940a2960 0x7fd9940a4e20 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:02:15.963 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.962+0000 7fd9b1001640 1 --2- 172.21.15.29:0/695485121 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd9ac10bbe0 0x7fd9ac19c7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:02:15.963 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.962+0000 7fd9b1001640 1 --2- 172.21.15.29:0/695485121 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd9ac1030e0 0x7fd9ac198460 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:02:15.964 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.962+0000 7fd9b1001640 1 --2- 172.21.15.29:0/695485121 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd9ac1027c0 0x7fd9ac197f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:02:15.964 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.962+0000 7fd9b1001640 1 -- 172.21.15.29:0/695485121 >> 172.21.15.29:0/695485121 conn(0x7fd9ac0fc020 msgr2=0x7fd9ac0fcd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:02:15.964 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.962+0000 7fd9b1001640 1 -- 172.21.15.29:0/695485121 shutdown_connections 2024-03-16T12:02:15.964 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:15.962+0000 7fd9b1001640 1 -- 172.21.15.29:0/695485121 wait complete. 2024-03-16T12:02:16.794 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:02:16 smithi154 ceph-mon[36320]: pgmap v358: 97 pgs: 97 active+clean; 69 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.4 MiB/s wr, 217 op/s 2024-03-16T12:02:16.803 INFO:tasks.workunit.client.0.smithi029.stdout: 1 1101615 9.38 MB/sec execute 481 sec latency 70.706 ms 2024-03-16T12:02:16.836 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:02:16 smithi029 ceph-mon[31975]: pgmap v358: 97 pgs: 97 active+clean; 69 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.4 MiB/s wr, 217 op/s 2024-03-16T12:02:16.876 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:02:16 smithi152 ceph-mon[36291]: pgmap v358: 97 pgs: 97 active+clean; 69 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.4 MiB/s wr, 217 op/s 2024-03-16T12:02:17.804 INFO:tasks.workunit.client.0.smithi029.stdout: 1 1103393 9.37 MB/sec execute 482 sec latency 21.381 ms 2024-03-16T12:02:18.376 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:02:18 smithi152 ceph-mon[36291]: pgmap v359: 97 pgs: 97 active+clean; 73 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 3.5 MiB/s wr, 331 op/s 2024-03-16T12:02:18.376 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:02:18 smithi152 ceph-mon[36291]: osdmap e104: 12 total, 12 up, 12 in 2024-03-16T12:02:18.544 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:02:18 smithi154 ceph-mon[36320]: pgmap v359: 97 pgs: 97 active+clean; 73 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 3.5 MiB/s wr, 331 op/s 2024-03-16T12:02:18.544 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:02:18 smithi154 ceph-mon[36320]: osdmap e104: 12 total, 12 up, 12 in 2024-03-16T12:02:18.587 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:02:18 smithi029 ceph-mon[31975]: pgmap v359: 97 pgs: 97 active+clean; 73 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 3.5 MiB/s wr, 331 op/s 2024-03-16T12:02:18.587 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:02:18 smithi029 ceph-mon[31975]: osdmap e104: 12 total, 12 up, 12 in 2024-03-16T12:02:18.804 INFO:tasks.workunit.client.0.smithi029.stdout: 1 1105018 9.38 MB/sec execute 483 sec latency 26.400 ms 2024-03-16T12:02:19.804 INFO:tasks.workunit.client.0.smithi029.stdout: 1 1106586 9.37 MB/sec execute 484 sec latency 26.572 ms 2024-03-16T12:02:20.376 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:02:20 smithi152 ceph-mon[36291]: pgmap v361: 97 pgs: 97 active+clean; 73 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 2.9 MiB/s wr, 244 op/s 2024-03-16T12:02:20.544 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:02:20 smithi154 ceph-mon[36320]: pgmap v361: 97 pgs: 97 active+clean; 73 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 2.9 MiB/s wr, 244 op/s 2024-03-16T12:02:20.586 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:02:20 smithi029 ceph-mon[31975]: pgmap v361: 97 pgs: 97 active+clean; 73 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 2.9 MiB/s wr, 244 op/s 2024-03-16T12:02:20.804 INFO:tasks.workunit.client.0.smithi029.stdout: 1 1108344 9.37 MB/sec execute 485 sec latency 57.567 ms 2024-03-16T12:02:21.804 INFO:tasks.workunit.client.0.smithi029.stdout: 1 1109466 9.36 MB/sec execute 486 sec latency 32.124 ms 2024-03-16T12:02:22.376 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:02:22 smithi152 ceph-mon[36291]: pgmap v362: 97 pgs: 97 active+clean; 70 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 2.8 MiB/s wr, 244 op/s 2024-03-16T12:02:22.544 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:02:22 smithi154 ceph-mon[36320]: pgmap v362: 97 pgs: 97 active+clean; 70 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 2.8 MiB/s wr, 244 op/s 2024-03-16T12:02:22.586 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:02:22 smithi029 ceph-mon[31975]: pgmap v362: 97 pgs: 97 active+clean; 70 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 2.8 MiB/s wr, 244 op/s 2024-03-16T12:02:22.804 INFO:tasks.workunit.client.0.smithi029.stdout: 1 1111764 9.38 MB/sec execute 487 sec latency 19.906 ms 2024-03-16T12:02:23.804 INFO:tasks.workunit.client.0.smithi029.stdout: 1 1112565 9.36 MB/sec execute 488 sec latency 21.972 ms 2024-03-16T12:02:24.376 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:02:24 smithi152 ceph-mon[36291]: pgmap v363: 97 pgs: 97 active+clean; 70 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 2.9 MiB/s wr, 245 op/s 2024-03-16T12:02:24.544 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:02:24 smithi154 ceph-mon[36320]: pgmap v363: 97 pgs: 97 active+clean; 70 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 2.9 MiB/s wr, 245 op/s 2024-03-16T12:02:24.586 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:02:24 smithi029 ceph-mon[31975]: pgmap v363: 97 pgs: 97 active+clean; 70 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 2.9 MiB/s wr, 245 op/s 2024-03-16T12:02:24.804 INFO:tasks.workunit.client.0.smithi029.stdout: 1 1114549 9.36 MB/sec execute 489 sec latency 19.962 ms 2024-03-16T12:02:25.804 INFO:tasks.workunit.client.0.smithi029.stdout: 1 1115696 9.36 MB/sec execute 490 sec latency 53.213 ms 2024-03-16T12:02:26.376 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:02:26 smithi152 ceph-mon[36291]: pgmap v364: 97 pgs: 97 active+clean; 70 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 2.9 MiB/s wr, 245 op/s 2024-03-16T12:02:26.543 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:02:26 smithi154 ceph-mon[36320]: pgmap v364: 97 pgs: 97 active+clean; 70 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 2.9 MiB/s wr, 245 op/s 2024-03-16T12:02:26.586 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:02:26 smithi029 ceph-mon[31975]: pgmap v364: 97 pgs: 97 active+clean; 70 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 2.9 MiB/s wr, 245 op/s 2024-03-16T12:02:26.804 INFO:tasks.workunit.client.0.smithi029.stdout: 1 1117307 9.35 MB/sec execute 491 sec latency 20.223 ms 2024-03-16T12:02:27.804 INFO:tasks.workunit.client.0.smithi029.stdout: 1 1119022 9.36 MB/sec execute 492 sec latency 23.966 ms 2024-03-16T12:02:28.376 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:02:28 smithi152 ceph-mon[36291]: pgmap v365: 97 pgs: 97 active+clean; 76 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 2.7 MiB/s wr, 175 op/s 2024-03-16T12:02:28.388 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:02:28 smithi029 ceph-mon[31975]: pgmap v365: 97 pgs: 97 active+clean; 76 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 2.7 MiB/s wr, 175 op/s 2024-03-16T12:02:28.544 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:02:28 smithi154 ceph-mon[36320]: pgmap v365: 97 pgs: 97 active+clean; 76 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 2.7 MiB/s wr, 175 op/s 2024-03-16T12:02:28.804 INFO:tasks.workunit.client.0.smithi029.stdout: 1 1120447 9.35 MB/sec execute 493 sec latency 22.529 ms 2024-03-16T12:02:29.053 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:29.052+0000 7f1319d3b640 1 -- 172.21.15.29:0/2100995619 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f13141077a0 msgr2=0x7f1314107b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:02:29.054 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:29.052+0000 7f1319d3b640 1 --2- 172.21.15.29:0/2100995619 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f13141077a0 0x7f1314107b80 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f1304009930 tx=0x7f130402bc30 comp rx=0 tx=0).stop 2024-03-16T12:02:29.054 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:29.052+0000 7f1319d3b640 1 -- 172.21.15.29:0/2100995619 shutdown_connections 2024-03-16T12:02:29.054 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:29.052+0000 7f1319d3b640 1 --2- 172.21.15.29:0/2100995619 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1314102150 0x7f131410e660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:02:29.054 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:29.052+0000 7f1319d3b640 1 --2- 172.21.15.29:0/2100995619 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1314101790 0x7f1314101c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:02:29.054 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:29.052+0000 7f1319d3b640 1 --2- 172.21.15.29:0/2100995619 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f13141077a0 0x7f1314107b80 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:02:29.054 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:29.052+0000 7f1319d3b640 1 -- 172.21.15.29:0/2100995619 >> 172.21.15.29:0/2100995619 conn(0x7f13140fd2a0 msgr2=0x7f13140ff6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:02:29.054 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:29.052+0000 7f1319d3b640 1 -- 172.21.15.29:0/2100995619 shutdown_connections 2024-03-16T12:02:29.054 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:29.052+0000 7f1319d3b640 1 -- 172.21.15.29:0/2100995619 wait complete. 2024-03-16T12:02:29.054 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:29.053+0000 7f1319d3b640 1 Processor -- start 2024-03-16T12:02:29.054 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:29.053+0000 7f1319d3b640 1 -- start start 2024-03-16T12:02:29.055 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:29.053+0000 7f1319d3b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1314101790 0x7f1314198010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:02:29.055 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:29.053+0000 7f1319d3b640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1314102150 0x7f1314198550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:02:29.055 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:29.053+0000 7f1319d3b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f13141077a0 0x7f131419c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:02:29.055 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:29.053+0000 7f1319d3b640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f1314111bb0 con 0x7f1314101790 2024-03-16T12:02:29.055 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:29.053+0000 7f1319d3b640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f1314111a30 con 0x7f1314102150 2024-03-16T12:02:29.055 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:29.053+0000 7f1319d3b640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f1314111d30 con 0x7f13141077a0 2024-03-16T12:02:29.055 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:29.053+0000 7f13137fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1314101790 0x7f1314198010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:02:29.055 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:29.053+0000 7f13137fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1314101790 0x7f1314198010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52338/0 (socket says 172.21.15.29:52338) 2024-03-16T12:02:29.055 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:29.053+0000 7f13137fe640 1 -- 172.21.15.29:0/147736254 learned_addr learned my addr 172.21.15.29:0/147736254 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T12:02:29.055 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:29.053+0000 7f1312ffd640 1 --2- 172.21.15.29:0/147736254 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1314102150 0x7f1314198550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:02:29.055 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:29.053+0000 7f1313fff640 1 --2- 172.21.15.29:0/147736254 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f13141077a0 0x7f131419c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:02:29.055 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:29.053+0000 7f13137fe640 1 -- 172.21.15.29:0/147736254 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f13141077a0 msgr2=0x7f131419c8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:02:29.055 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:29.053+0000 7f13137fe640 1 --2- 172.21.15.29:0/147736254 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f13141077a0 0x7f131419c8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:02:29.055 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:29.053+0000 7f13137fe640 1 -- 172.21.15.29:0/147736254 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1314102150 msgr2=0x7f1314198550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:02:29.055 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:29.053+0000 7f13137fe640 1 --2- 172.21.15.29:0/147736254 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1314102150 0x7f1314198550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:02:29.056 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:29.054+0000 7f13137fe640 1 -- 172.21.15.29:0/147736254 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f131419d000 con 0x7f1314101790 2024-03-16T12:02:29.056 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:29.054+0000 7f13137fe640 1 --2- 172.21.15.29:0/147736254 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1314101790 0x7f1314198010 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f1304009230 tx=0x7f130402cb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:02:29.056 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:29.054+0000 7f1310ff9640 1 -- 172.21.15.29:0/147736254 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1304041070 con 0x7f1314101790 2024-03-16T12:02:29.056 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:29.054+0000 7f1310ff9640 1 -- 172.21.15.29:0/147736254 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1304004450 con 0x7f1314101790 2024-03-16T12:02:29.056 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:29.054+0000 7f1319d3b640 1 -- 172.21.15.29:0/147736254 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f131419d290 con 0x7f1314101790 2024-03-16T12:02:29.056 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:29.054+0000 7f1310ff9640 1 -- 172.21.15.29:0/147736254 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1304003e50 con 0x7f1314101790 2024-03-16T12:02:29.056 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:29.054+0000 7f1319d3b640 1 -- 172.21.15.29:0/147736254 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f13141a4b80 con 0x7f1314101790 2024-03-16T12:02:29.063 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:29.056+0000 7f1310ff9640 1 -- 172.21.15.29:0/147736254 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7f1304044070 con 0x7f1314101790 2024-03-16T12:02:29.063 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:29.056+0000 7f131ca03740 1 Processor -- start 2024-03-16T12:02:29.063 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:29.056+0000 7f131ca03740 1 -- start start 2024-03-16T12:02:29.063 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:29.057+0000 7f131ca03740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55fcf5046130 0x55fcf5046510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:02:29.063 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:29.057+0000 7f1310ff9640 1 --2- 172.21.15.29:0/147736254 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f12fc0a2bb0 0x7f12fc0a5070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:02:29.063 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:29.057+0000 7f131ca03740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55fcf5046a50 0x55fcf504f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:02:29.063 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:29.057+0000 7f131ca03740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fcf504f820 0x55fcf5051c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:02:29.063 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:29.057+0000 7f131ca03740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55fcf4faaa40 con 0x55fcf5046130 2024-03-16T12:02:29.063 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:29.057+0000 7f131ca03740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x55fcf4f53920 con 0x55fcf5046a50 2024-03-16T12:02:29.063 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:29.057+0000 7f131ca03740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55fcf4fa6ae0 con 0x55fcf504f820 2024-03-16T12:02:29.063 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:29.057+0000 7f13137fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55fcf5046130 0x55fcf5046510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:02:29.063 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:29.057+0000 7f1313fff640 1 --2- 172.21.15.29:0/147736254 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f12fc0a2bb0 0x7f12fc0a5070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:02:29.063 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:29.057+0000 7f1312ffd640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55fcf5046a50 0x55fcf504f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:02:29.064 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:29.057+0000 7f13137fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55fcf5046130 0x55fcf5046510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52350/0 (socket says 172.21.15.29:52350) 2024-03-16T12:02:29.064 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:29.057+0000 7f13137fe640 1 -- 172.21.15.29:0/4077757268 learned_addr learned my addr 172.21.15.29:0/4077757268 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T12:02:29.064 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:29.057+0000 7f1313fff640 1 --2- 172.21.15.29:0/4077757268 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fcf504f820 0x55fcf5051c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:02:29.064 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:29.057+0000 7f1310ff9640 1 -- 172.21.15.29:0/147736254 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 8556+0+0 (secure 0 0 0) 0x7f13040f0630 con 0x7f1314101790 2024-03-16T12:02:29.064 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:29.057+0000 7f13137fe640 1 -- 172.21.15.29:0/4077757268 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fcf504f820 msgr2=0x55fcf5051c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:02:29.064 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:29.057+0000 7f13137fe640 1 --2- 172.21.15.29:0/4077757268 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fcf504f820 0x55fcf5051c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:02:29.064 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:29.057+0000 7f13137fe640 1 -- 172.21.15.29:0/4077757268 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55fcf5046a50 msgr2=0x55fcf504f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:02:29.064 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:29.057+0000 7f13137fe640 1 --2- 172.21.15.29:0/4077757268 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55fcf5046a50 0x55fcf504f2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:02:29.064 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:29.057+0000 7f13137fe640 1 -- 172.21.15.29:0/4077757268 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fcf4f6b460 con 0x55fcf5046130 2024-03-16T12:02:29.064 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:29.057+0000 7f1312ffd640 1 --2- 172.21.15.29:0/4077757268 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55fcf5046a50 0x55fcf504f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:02:29.064 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:29.057+0000 7f1313fff640 1 --2- 172.21.15.29:0/147736254 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f12fc0a2bb0 0x7f12fc0a5070 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f130c009fd0 tx=0x7f130c009de0 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:02:29.064 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:29.057+0000 7f13137fe640 1 --2- 172.21.15.29:0/4077757268 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55fcf5046130 0x55fcf5046510 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f13040c4ff0 tx=0x7f1304003920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=713e0495f847d1c3 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:02:29.064 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:29.058+0000 7f12f9ffb640 1 -- 172.21.15.29:0/4077757268 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f130404b050 con 0x55fcf5046130 2024-03-16T12:02:29.064 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:29.058+0000 7f12f9ffb640 1 -- 172.21.15.29:0/4077757268 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1304004c80 con 0x55fcf5046130 2024-03-16T12:02:29.065 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:29.058+0000 7f12f9ffb640 1 -- 172.21.15.29:0/4077757268 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13040c0ed0 con 0x55fcf5046130 2024-03-16T12:02:29.065 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:29.058+0000 7f131ca03740 1 -- 172.21.15.29:0/4077757268 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55fcf5046130 msgr2=0x55fcf5046510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:02:29.065 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:29.058+0000 7f131ca03740 1 --2- 172.21.15.29:0/4077757268 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55fcf5046130 0x55fcf5046510 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f13040c4ff0 tx=0x7f1304003920 comp rx=0 tx=0).stop 2024-03-16T12:02:29.065 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:29.058+0000 7f131ca03740 1 -- 172.21.15.29:0/4077757268 shutdown_connections 2024-03-16T12:02:29.065 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:29.058+0000 7f131ca03740 1 --2- 172.21.15.29:0/4077757268 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fcf504f820 0x55fcf5051c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:02:29.065 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:29.058+0000 7f131ca03740 1 --2- 172.21.15.29:0/4077757268 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55fcf5046a50 0x55fcf504f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:02:29.065 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:29.058+0000 7f131ca03740 1 --2- 172.21.15.29:0/4077757268 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55fcf5046130 0x55fcf5046510 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:02:29.065 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:29.058+0000 7f131ca03740 1 -- 172.21.15.29:0/4077757268 >> 172.21.15.29:0/4077757268 conn(0x55fcf5040a50 msgr2=0x55fcf5040e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:02:29.065 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:29.058+0000 7f131ca03740 1 -- 172.21.15.29:0/4077757268 shutdown_connections 2024-03-16T12:02:29.065 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:29.058+0000 7f131ca03740 1 -- 172.21.15.29:0/4077757268 wait complete. 2024-03-16T12:02:29.065 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:29.059+0000 7f131ca03740 1 -- start start 2024-03-16T12:02:29.065 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:29.059+0000 7f131ca03740 1 Processor -- start 2024-03-16T12:02:29.065 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:29.059+0000 7f131ca03740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55fcf5046130 0x55fcf504c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:02:29.065 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:29.059+0000 7f131ca03740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55fcf5046a50 0x55fcf504ce30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:02:29.066 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:29.059+0000 7f131ca03740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fcf504f820 0x55fcf504d370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:02:29.066 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:29.059+0000 7f131ca03740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55fcf4faaa40 con 0x55fcf5046130 2024-03-16T12:02:29.066 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:29.059+0000 7f131ca03740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x55fcf4f66d70 con 0x55fcf5046a50 2024-03-16T12:02:29.066 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:29.059+0000 7f131ca03740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55fcf5052210 con 0x55fcf504f820 2024-03-16T12:02:29.066 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:29.059+0000 7f131ca03740 10 client.0 _ll_register_callbacks cb 0x55fcf4cbea00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T12:02:29.066 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:29.059+0000 7f13137fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55fcf5046130 0x55fcf504c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:02:29.066 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:29.059+0000 7f13137fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55fcf5046130 0x55fcf504c8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52354/0 (socket says 172.21.15.29:52354) 2024-03-16T12:02:29.066 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:29.059+0000 7f13137fe640 1 -- 172.21.15.29:0/2262293470 learned_addr learned my addr 172.21.15.29:0/2262293470 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T12:02:29.066 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:29.059+0000 7f1312ffd640 1 --2- 172.21.15.29:0/2262293470 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55fcf5046a50 0x55fcf504ce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:02:29.066 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:29.059+0000 7f1313fff640 1 --2- 172.21.15.29:0/2262293470 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fcf504f820 0x55fcf504d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:02:29.066 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:29.060+0000 7f13137fe640 1 -- 172.21.15.29:0/2262293470 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fcf504f820 msgr2=0x55fcf504d370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:02:29.066 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:29.060+0000 7f13137fe640 1 --2- 172.21.15.29:0/2262293470 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fcf504f820 0x55fcf504d370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:02:29.067 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:29.060+0000 7f13137fe640 1 -- 172.21.15.29:0/2262293470 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55fcf5046a50 msgr2=0x55fcf504ce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:02:29.067 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:29.060+0000 7f13137fe640 1 --2- 172.21.15.29:0/2262293470 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55fcf5046a50 0x55fcf504ce30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:02:29.067 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:29.060+0000 7f13137fe640 1 -- 172.21.15.29:0/2262293470 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fcf4cbf210 con 0x55fcf5046130 2024-03-16T12:02:29.067 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:29.060+0000 7f1312ffd640 1 --2- 172.21.15.29:0/2262293470 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55fcf5046a50 0x55fcf504ce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T12:02:29.067 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:29.060+0000 7f13137fe640 1 --2- 172.21.15.29:0/2262293470 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55fcf5046130 0x55fcf504c8f0 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f13040bcb30 tx=0x7f13040bcea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=adfcb8ac1d902560 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:02:29.067 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:29.060+0000 7f12f2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-16T12:02:29.067 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:29.060+0000 7f12f2ffd640 1 -- 172.21.15.29:0/2262293470 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f130404b050 con 0x55fcf5046130 2024-03-16T12:02:29.067 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:29.060+0000 7f12f2ffd640 1 -- 172.21.15.29:0/2262293470 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 8556+0+0 (secure 0 0 0) 0x7f13040edfe0 con 0x55fcf5046130 2024-03-16T12:02:29.067 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:29.060+0000 7f131ca03740 1 -- 172.21.15.29:0/2262293470 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1304004c80 con 0x55fcf5046130 2024-03-16T12:02:29.067 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:29.060+0000 7f12f2ffd640 1 -- 172.21.15.29:0/2262293470 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f13040bc790 con 0x55fcf5046130 2024-03-16T12:02:29.067 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:29.060+0000 7f131ca03740 10 client.16362 fetch_fsmap learned FSMap version 28 2024-03-16T12:02:29.068 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:29.060+0000 7f131ca03740 1 -- 172.21.15.29:0/2262293470 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55fcf4d8b620 con 0x55fcf5046130 2024-03-16T12:02:29.068 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:29.060+0000 7f12f2ffd640 1 -- 172.21.15.29:0/2262293470 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1304004c80 con 0x55fcf5046130 2024-03-16T12:02:29.068 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:29.060+0000 7f12f2ffd640 1 -- 172.21.15.29:0/2262293470 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 28) ==== 8338+0+0 (secure 0 0 0) 0x7f13040c5090 con 0x55fcf5046130 2024-03-16T12:02:29.068 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:29.060+0000 7f131ca03740 10 client.16362 fetch_fsmap finished waiting for FSMap version 28 2024-03-16T12:02:29.068 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:29.060+0000 7f131ca03740 10 client.16362 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T12:02:29.068 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:29.060+0000 7f131ca03740 20 client.16362 populate_metadata read hostname 'smithi029' 2024-03-16T12:02:29.068 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:29.060+0000 7f131ca03740 1 --2- 172.21.15.29:0/2262293470 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55fcf505f500 0x55fcf50618f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:02:29.068 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:29.060+0000 7f131ca03740 4 client.16362 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T12:02:29.068 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:29.060+0000 7f131ca03740 1 -- 172.21.15.29:0/2262293470 --> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fcf4fb7f80 con 0x55fcf505f500 2024-03-16T12:02:29.068 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:29.061+0000 7f1312ffd640 1 --2- 172.21.15.29:0/2262293470 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55fcf505f500 0x55fcf50618f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:02:29.068 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:29.061+0000 7f1312ffd640 1 --2- 172.21.15.29:0/2262293470 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55fcf505f500 0x55fcf50618f0 crc :-1 s=READY pgs=142 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b9423cca7244c24 server_cookie=a7e76184efcbf8dd in_seq=0 out_seq=0 2024-03-16T12:02:29.068 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:29.061+0000 7f12f2ffd640 10 client.16362 ms_handle_connect on v2:172.21.15.152:6838/1548731833 2024-03-16T12:02:29.068 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:29.063+0000 7f12f2ffd640 1 -- 172.21.15.29:0/2262293470 <== mds.0 v2:172.21.15.152:6838/1548731833 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55fcf4fb7f80 con 0x55fcf505f500 2024-03-16T12:02:29.069 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:29.063+0000 7f12f2ffd640 10 client.16362 handle_command_reply: tid=0 2024-03-16T12:02:29.069 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:29.063+0000 7f131ca03740 1 client.16362 shutdown 2024-03-16T12:02:29.069 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:29.063+0000 7f131ca03740 1 -- 172.21.15.29:0/2262293470 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55fcf5046130 msgr2=0x55fcf504c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:02:29.069 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:29.063+0000 7f131ca03740 1 --2- 172.21.15.29:0/2262293470 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55fcf5046130 0x55fcf504c8f0 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f13040bcb30 tx=0x7f13040bcea0 comp rx=0 tx=0).stop 2024-03-16T12:02:29.069 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:29.063+0000 7f131ca03740 1 -- 172.21.15.29:0/2262293470 shutdown_connections 2024-03-16T12:02:29.069 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:29.063+0000 7f131ca03740 1 --2- 172.21.15.29:0/2262293470 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fcf504f820 0x55fcf504d370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:02:29.069 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:29.063+0000 7f131ca03740 1 --2- 172.21.15.29:0/2262293470 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55fcf505f500 0x55fcf50618f0 crc :-1 s=READY pgs=142 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:02:29.069 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:29.063+0000 7f131ca03740 1 --2- 172.21.15.29:0/2262293470 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55fcf5046a50 0x55fcf504ce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:02:29.069 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:29.063+0000 7f131ca03740 1 --2- 172.21.15.29:0/2262293470 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55fcf5046130 0x55fcf504c8f0 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:02:29.070 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:29.063+0000 7f12f2ffd640 0 client.16362 ms_handle_reset on v2:172.21.15.152:6838/1548731833 2024-03-16T12:02:29.070 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:29.063+0000 7f131ca03740 1 -- 172.21.15.29:0/2262293470 >> 172.21.15.29:0/2262293470 conn(0x55fcf5040a50 msgr2=0x55fcf50427a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:02:29.070 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:29.063+0000 7f131ca03740 1 -- 172.21.15.29:0/2262293470 shutdown_connections 2024-03-16T12:02:29.070 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:29.063+0000 7f131ca03740 1 -- 172.21.15.29:0/2262293470 wait complete. 2024-03-16T12:02:29.070 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:29.063+0000 7f131ca03740 20 client.16362 trim_cache size 0 max 16384 2024-03-16T12:02:29.078 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:29.078+0000 7f131ca03740 1 Processor -- start 2024-03-16T12:02:29.078 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:29.078+0000 7f131ca03740 1 -- start start 2024-03-16T12:02:29.079 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:29.078+0000 7f131ca03740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fcf5046130 0x55fcf504f020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:02:29.079 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:29.078+0000 7f131ca03740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55fcf5046a50 0x55fcf505ae30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:02:29.079 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:29.078+0000 7f131ca03740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55fcf504f820 0x55fcf503ba40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:02:29.079 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:29.078+0000 7f131ca03740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f13141afd60 con 0x55fcf5046a50 2024-03-16T12:02:29.079 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:29.078+0000 7f131ca03740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f13141ad630 con 0x55fcf504f820 2024-03-16T12:02:29.079 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:29.078+0000 7f131ca03740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f13141ab4a0 con 0x55fcf5046130 2024-03-16T12:02:29.079 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:29.079+0000 7f1312ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55fcf5046a50 0x55fcf505ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:02:29.079 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:29.079+0000 7f1312ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55fcf5046a50 0x55fcf505ae30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52358/0 (socket says 172.21.15.29:52358) 2024-03-16T12:02:29.079 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:29.079+0000 7f1312ffd640 1 -- 172.21.15.29:0/2075755237 learned_addr learned my addr 172.21.15.29:0/2075755237 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T12:02:29.079 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:29.079+0000 7f1312ffd640 1 -- 172.21.15.29:0/2075755237 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fcf5046130 msgr2=0x55fcf504f020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:02:29.079 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:29.079+0000 7f1312ffd640 1 --2- 172.21.15.29:0/2075755237 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fcf5046130 0x55fcf504f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:02:29.079 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:29.079+0000 7f1312ffd640 1 -- 172.21.15.29:0/2075755237 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55fcf504f820 msgr2=0x55fcf503ba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:02:29.080 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:29.079+0000 7f1312ffd640 1 --2- 172.21.15.29:0/2075755237 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55fcf504f820 0x55fcf503ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:02:29.080 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:29.079+0000 7f1312ffd640 1 -- 172.21.15.29:0/2075755237 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fcf504b8a0 con 0x55fcf5046a50 2024-03-16T12:02:29.080 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:29.079+0000 7f1312ffd640 1 --2- 172.21.15.29:0/2075755237 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55fcf5046a50 0x55fcf505ae30 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x55fcf5045be0 tx=0x7f1308005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de7afa5c3dad28f0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:02:29.080 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:29.079+0000 7f12f97fa640 1 -- 172.21.15.29:0/2075755237 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1308005220 con 0x55fcf5046a50 2024-03-16T12:02:29.080 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:29.079+0000 7f12f97fa640 1 -- 172.21.15.29:0/2075755237 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1308024030 con 0x55fcf5046a50 2024-03-16T12:02:29.080 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:29.079+0000 7f12f97fa640 1 -- 172.21.15.29:0/2075755237 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1308004b80 con 0x55fcf5046a50 2024-03-16T12:02:29.083 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:29.080+0000 7f131ca03740 1 -- 172.21.15.29:0/2075755237 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55fcf5046a50 msgr2=0x55fcf505ae30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:02:29.083 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:29.080+0000 7f131ca03740 1 --2- 172.21.15.29:0/2075755237 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55fcf5046a50 0x55fcf505ae30 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x55fcf5045be0 tx=0x7f1308005f90 comp rx=0 tx=0).stop 2024-03-16T12:02:29.083 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:29.080+0000 7f131ca03740 1 -- 172.21.15.29:0/2075755237 shutdown_connections 2024-03-16T12:02:29.083 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:29.080+0000 7f131ca03740 1 --2- 172.21.15.29:0/2075755237 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55fcf504f820 0x55fcf503ba40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:02:29.084 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:29.080+0000 7f131ca03740 1 --2- 172.21.15.29:0/2075755237 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55fcf5046a50 0x55fcf505ae30 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:02:29.084 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:29.080+0000 7f131ca03740 1 --2- 172.21.15.29:0/2075755237 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fcf5046130 0x55fcf504f020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:02:29.084 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:29.080+0000 7f131ca03740 1 -- 172.21.15.29:0/2075755237 >> 172.21.15.29:0/2075755237 conn(0x55fcf5040a50 msgr2=0x55fcf503fbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:02:29.084 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:29.080+0000 7f131ca03740 1 -- 172.21.15.29:0/2075755237 shutdown_connections 2024-03-16T12:02:29.084 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:29.080+0000 7f131ca03740 1 -- 172.21.15.29:0/2075755237 wait complete. 2024-03-16T12:02:29.084 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:29.080+0000 7f131ca03740 1 -- start start 2024-03-16T12:02:29.084 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:29.081+0000 7f131ca03740 1 Processor -- start 2024-03-16T12:02:29.084 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:29.081+0000 7f131ca03740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fcf5046130 0x55fcf5053640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:02:29.084 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:29.081+0000 7f131ca03740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55fcf5046a50 0x55fcf5053b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:02:29.084 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:29.081+0000 7f131ca03740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55fcf504f820 0x55fcf50be770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:02:29.084 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:29.081+0000 7f131ca03740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f13141abaf0 con 0x55fcf504f820 2024-03-16T12:02:29.084 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:29.081+0000 7f131ca03740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f13141acf60 con 0x55fcf5046a50 2024-03-16T12:02:29.084 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:29.081+0000 7f131ca03740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f13141ac1c0 con 0x55fcf5046130 2024-03-16T12:02:29.084 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:29.081+0000 7f131ca03740 10 client.0 _ll_register_callbacks cb 0x7f13080097a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T12:02:29.084 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:29.081+0000 7f1313fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55fcf504f820 0x55fcf50be770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:02:29.085 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:29.081+0000 7f1313fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55fcf504f820 0x55fcf50be770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52366/0 (socket says 172.21.15.29:52366) 2024-03-16T12:02:29.085 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:29.081+0000 7f1313fff640 1 -- 172.21.15.29:0/2304116914 learned_addr learned my addr 172.21.15.29:0/2304116914 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T12:02:29.085 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:29.081+0000 7f1312ffd640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55fcf5046a50 0x55fcf5053b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:02:29.085 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:29.081+0000 7f13137fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fcf5046130 0x55fcf5053640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:02:29.085 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:29.081+0000 7f1313fff640 1 -- 172.21.15.29:0/2304116914 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fcf5046130 msgr2=0x55fcf5053640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:02:29.085 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:29.081+0000 7f1313fff640 1 --2- 172.21.15.29:0/2304116914 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fcf5046130 0x55fcf5053640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:02:29.085 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:29.081+0000 7f1313fff640 1 -- 172.21.15.29:0/2304116914 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55fcf5046a50 msgr2=0x55fcf5053b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:02:29.085 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:29.081+0000 7f1313fff640 1 --2- 172.21.15.29:0/2304116914 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55fcf5046a50 0x55fcf5053b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:02:29.085 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:29.081+0000 7f1313fff640 1 -- 172.21.15.29:0/2304116914 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fcf50bede0 con 0x55fcf504f820 2024-03-16T12:02:29.085 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:29.082+0000 7f1313fff640 1 --2- 172.21.15.29:0/2304116914 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55fcf504f820 0x55fcf50be770 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f130c005fd0 tx=0x7f130c004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c38c3edf4ad3476 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:02:29.085 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:29.082+0000 7f12f2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-16T12:02:29.085 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:29.082+0000 7f12f2ffd640 1 -- 172.21.15.29:0/2304116914 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f130c02f060 con 0x55fcf504f820 2024-03-16T12:02:29.085 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:29.082+0000 7f12f2ffd640 1 -- 172.21.15.29:0/2304116914 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 8556+0+0 (secure 0 0 0) 0x7f130c01c040 con 0x55fcf504f820 2024-03-16T12:02:29.085 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:29.082+0000 7f131ca03740 1 -- 172.21.15.29:0/2304116914 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1308024030 con 0x55fcf504f820 2024-03-16T12:02:29.085 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:29.082+0000 7f12f2ffd640 1 -- 172.21.15.29:0/2304116914 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f130c011e10 con 0x55fcf504f820 2024-03-16T12:02:29.086 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:29.082+0000 7f131ca03740 10 client.16368 fetch_fsmap learned FSMap version 28 2024-03-16T12:02:29.086 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:29.082+0000 7f131ca03740 1 -- 172.21.15.29:0/2304116914 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55fcf50c6190 con 0x55fcf504f820 2024-03-16T12:02:29.086 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:29.082+0000 7f12f2ffd640 1 -- 172.21.15.29:0/2304116914 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1308024030 con 0x55fcf504f820 2024-03-16T12:02:29.086 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:29.082+0000 7f12f2ffd640 1 -- 172.21.15.29:0/2304116914 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 28) ==== 8338+0+0 (secure 0 0 0) 0x7f130c01f040 con 0x55fcf504f820 2024-03-16T12:02:29.086 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:29.082+0000 7f131ca03740 10 client.16368 fetch_fsmap finished waiting for FSMap version 28 2024-03-16T12:02:29.086 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:29.082+0000 7f131ca03740 10 client.16368 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T12:02:29.086 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:29.082+0000 7f131ca03740 20 client.16368 populate_metadata read hostname 'smithi029' 2024-03-16T12:02:29.086 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:29.082+0000 7f131ca03740 1 --2- 172.21.15.29:0/2304116914 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55fcf50c6910 0x55fcf50c8d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:02:29.086 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:29.082+0000 7f131ca03740 4 client.16368 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-16T12:02:29.086 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:29.082+0000 7f131ca03740 1 -- 172.21.15.29:0/2304116914 --> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55fcf4fb22d0 con 0x55fcf50c6910 2024-03-16T12:02:29.087 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:29.082+0000 7f13137fe640 1 --2- 172.21.15.29:0/2304116914 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55fcf50c6910 0x55fcf50c8d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:02:29.087 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:29.083+0000 7f13137fe640 1 --2- 172.21.15.29:0/2304116914 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55fcf50c6910 0x55fcf50c8d30 crc :-1 s=READY pgs=143 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4b463247ae0672fa server_cookie=e07731cae0f2bc6a in_seq=0 out_seq=0 2024-03-16T12:02:29.087 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:29.083+0000 7f12f2ffd640 10 client.16368 ms_handle_connect on v2:172.21.15.152:6838/1548731833 2024-03-16T12:02:29.259 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:29.259+0000 7f12f2ffd640 1 -- 172.21.15.29:0/2304116914 <== mds.0 v2:172.21.15.152:6838/1548731833 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55fcf4fb22d0 con 0x55fcf50c6910 2024-03-16T12:02:29.259 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:29.259+0000 7f12f2ffd640 10 client.16368 handle_command_reply: tid=0 2024-03-16T12:02:29.259 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:29.259+0000 7f131ca03740 1 client.16368 shutdown 2024-03-16T12:02:29.259 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:29.259+0000 7f131ca03740 1 -- 172.21.15.29:0/2304116914 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55fcf504f820 msgr2=0x55fcf50be770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:02:29.259 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:29.259+0000 7f131ca03740 1 --2- 172.21.15.29:0/2304116914 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55fcf504f820 0x55fcf50be770 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f130c005fd0 tx=0x7f130c004ab0 comp rx=0 tx=0).stop 2024-03-16T12:02:29.260 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:29.259+0000 7f131ca03740 1 -- 172.21.15.29:0/2304116914 shutdown_connections 2024-03-16T12:02:29.260 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:29.259+0000 7f131ca03740 1 --2- 172.21.15.29:0/2304116914 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55fcf504f820 0x55fcf50be770 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:02:29.260 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:29.259+0000 7f131ca03740 1 --2- 172.21.15.29:0/2304116914 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55fcf50c6910 0x55fcf50c8d30 crc :-1 s=READY pgs=143 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:02:29.260 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:29.259+0000 7f131ca03740 1 --2- 172.21.15.29:0/2304116914 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55fcf5046a50 0x55fcf5053b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:02:29.260 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:29.259+0000 7f131ca03740 1 --2- 172.21.15.29:0/2304116914 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fcf5046130 0x55fcf5053640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:02:29.260 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:29.259+0000 7f12f2ffd640 0 client.16368 ms_handle_reset on v2:172.21.15.152:6838/1548731833 2024-03-16T12:02:29.260 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:29.259+0000 7f131ca03740 1 -- 172.21.15.29:0/2304116914 >> 172.21.15.29:0/2304116914 conn(0x55fcf5040a50 msgr2=0x55fcf5041ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:02:29.260 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:29.260+0000 7f131ca03740 1 -- 172.21.15.29:0/2304116914 shutdown_connections 2024-03-16T12:02:29.260 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:29.260+0000 7f131ca03740 1 -- 172.21.15.29:0/2304116914 wait complete. 2024-03-16T12:02:29.260 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:29.260+0000 7f131ca03740 20 client.16368 trim_cache size 0 max 16384 2024-03-16T12:02:29.260 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:{ 2024-03-16T12:02:29.261 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "message": "", 2024-03-16T12:02:29.261 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "return_code": 0 2024-03-16T12:02:29.261 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:} 2024-03-16T12:02:29.262 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:29.261+0000 7f1319d3b640 1 -- 172.21.15.29:0/147736254 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f12fc0a2bb0 msgr2=0x7f12fc0a5070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:02:29.262 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:29.261+0000 7f1319d3b640 1 --2- 172.21.15.29:0/147736254 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f12fc0a2bb0 0x7f12fc0a5070 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f130c009fd0 tx=0x7f130c009de0 comp rx=0 tx=0).stop 2024-03-16T12:02:29.262 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:29.261+0000 7f1319d3b640 1 -- 172.21.15.29:0/147736254 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1314101790 msgr2=0x7f1314198010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:02:29.262 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:29.261+0000 7f1319d3b640 1 --2- 172.21.15.29:0/147736254 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1314101790 0x7f1314198010 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f1304009230 tx=0x7f130402cb30 comp rx=0 tx=0).stop 2024-03-16T12:02:29.262 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:29.261+0000 7f1319d3b640 1 -- 172.21.15.29:0/147736254 shutdown_connections 2024-03-16T12:02:29.262 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:29.261+0000 7f1319d3b640 1 --2- 172.21.15.29:0/147736254 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f12fc0a2bb0 0x7f12fc0a5070 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:02:29.262 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:29.261+0000 7f1319d3b640 1 --2- 172.21.15.29:0/147736254 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f13141077a0 0x7f131419c8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:02:29.262 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:29.261+0000 7f1319d3b640 1 --2- 172.21.15.29:0/147736254 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1314102150 0x7f1314198550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:02:29.262 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:29.261+0000 7f1319d3b640 1 --2- 172.21.15.29:0/147736254 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1314101790 0x7f1314198010 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:02:29.262 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:29.261+0000 7f1319d3b640 1 -- 172.21.15.29:0/147736254 >> 172.21.15.29:0/147736254 conn(0x7f13140fd2a0 msgr2=0x7f13140fd7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:02:29.262 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:29.261+0000 7f1319d3b640 1 -- 172.21.15.29:0/147736254 shutdown_connections 2024-03-16T12:02:29.263 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:29.261+0000 7f1319d3b640 1 -- 172.21.15.29:0/147736254 wait complete. 2024-03-16T12:02:29.804 INFO:tasks.workunit.client.0.smithi029.stdout: 1 1122417 9.36 MB/sec execute 494 sec latency 92.526 ms 2024-03-16T12:02:30.543 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:02:30 smithi154 ceph-mon[36320]: pgmap v366: 97 pgs: 97 active+clean; 76 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 2.3 MiB/s wr, 147 op/s 2024-03-16T12:02:30.586 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:02:30 smithi029 ceph-mon[31975]: pgmap v366: 97 pgs: 97 active+clean; 76 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 2.3 MiB/s wr, 147 op/s 2024-03-16T12:02:30.625 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:02:30 smithi152 ceph-mon[36291]: pgmap v366: 97 pgs: 97 active+clean; 76 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 2.3 MiB/s wr, 147 op/s 2024-03-16T12:02:30.804 INFO:tasks.workunit.client.0.smithi029.stdout: 1 1123084 9.34 MB/sec execute 495 sec latency 36.092 ms 2024-03-16T12:02:31.804 INFO:tasks.workunit.client.0.smithi029.stdout: 1 1125018 9.34 MB/sec execute 496 sec latency 19.910 ms 2024-03-16T12:02:32.543 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:02:32 smithi154 ceph-mon[36320]: pgmap v367: 97 pgs: 97 active+clean; 67 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 2.9 MiB/s wr, 250 op/s 2024-03-16T12:02:32.586 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:02:32 smithi029 ceph-mon[31975]: pgmap v367: 97 pgs: 97 active+clean; 67 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 2.9 MiB/s wr, 250 op/s 2024-03-16T12:02:32.625 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:02:32 smithi152 ceph-mon[36291]: pgmap v367: 97 pgs: 97 active+clean; 67 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 2.9 MiB/s wr, 250 op/s 2024-03-16T12:02:32.804 INFO:tasks.workunit.client.0.smithi029.stdout: 1 1126524 9.34 MB/sec execute 497 sec latency 22.275 ms 2024-03-16T12:02:33.804 INFO:tasks.workunit.client.0.smithi029.stdout: 1 1128368 9.34 MB/sec execute 498 sec latency 20.055 ms 2024-03-16T12:02:34.543 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:02:34 smithi154 ceph-mon[36320]: pgmap v368: 97 pgs: 97 active+clean; 66 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 2.3 MiB/s wr, 229 op/s 2024-03-16T12:02:34.586 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:02:34 smithi029 ceph-mon[31975]: pgmap v368: 97 pgs: 97 active+clean; 66 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 2.3 MiB/s wr, 229 op/s 2024-03-16T12:02:34.625 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:02:34 smithi152 ceph-mon[36291]: pgmap v368: 97 pgs: 97 active+clean; 66 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 2.3 MiB/s wr, 229 op/s 2024-03-16T12:02:34.804 INFO:tasks.workunit.client.0.smithi029.stdout: 1 1129727 9.34 MB/sec execute 499 sec latency 34.348 ms 2024-03-16T12:02:35.804 INFO:tasks.workunit.client.0.smithi029.stdout: 1 1130961 9.34 MB/sec execute 500 sec latency 40.405 ms 2024-03-16T12:02:36.544 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:02:36 smithi154 ceph-mon[36320]: pgmap v369: 97 pgs: 97 active+clean; 66 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.1 MiB/s wr, 221 op/s 2024-03-16T12:02:36.586 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:02:36 smithi029 ceph-mon[31975]: pgmap v369: 97 pgs: 97 active+clean; 66 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.1 MiB/s wr, 221 op/s 2024-03-16T12:02:36.625 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:02:36 smithi152 ceph-mon[36291]: pgmap v369: 97 pgs: 97 active+clean; 66 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.1 MiB/s wr, 221 op/s 2024-03-16T12:02:36.804 INFO:tasks.workunit.client.0.smithi029.stdout: 1 1133105 9.34 MB/sec execute 501 sec latency 18.398 ms 2024-03-16T12:02:37.804 INFO:tasks.workunit.client.0.smithi029.stdout: 1 1133855 9.33 MB/sec execute 502 sec latency 20.801 ms 2024-03-16T12:02:38.543 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:02:38 smithi154 ceph-mon[36320]: pgmap v370: 97 pgs: 97 active+clean; 76 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 3.1 MiB/s wr, 282 op/s 2024-03-16T12:02:38.586 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:02:38 smithi029 ceph-mon[31975]: pgmap v370: 97 pgs: 97 active+clean; 76 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 3.1 MiB/s wr, 282 op/s 2024-03-16T12:02:38.625 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:02:38 smithi152 ceph-mon[36291]: pgmap v370: 97 pgs: 97 active+clean; 76 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 3.1 MiB/s wr, 282 op/s 2024-03-16T12:02:38.804 INFO:tasks.workunit.client.0.smithi029.stdout: 1 1135879 9.33 MB/sec execute 503 sec latency 24.450 ms 2024-03-16T12:02:39.804 INFO:tasks.workunit.client.0.smithi029.stdout: 1 1137023 9.33 MB/sec execute 504 sec latency 45.863 ms 2024-03-16T12:02:40.543 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:02:40 smithi154 ceph-mon[36320]: pgmap v371: 97 pgs: 97 active+clean; 76 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 1.9 MiB/s wr, 206 op/s 2024-03-16T12:02:40.586 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:02:40 smithi029 ceph-mon[31975]: pgmap v371: 97 pgs: 97 active+clean; 76 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 1.9 MiB/s wr, 206 op/s 2024-03-16T12:02:40.625 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:02:40 smithi152 ceph-mon[36291]: pgmap v371: 97 pgs: 97 active+clean; 76 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 1.9 MiB/s wr, 206 op/s 2024-03-16T12:02:40.804 INFO:tasks.workunit.client.0.smithi029.stdout: 1 1138345 9.32 MB/sec execute 505 sec latency 82.456 ms 2024-03-16T12:02:41.804 INFO:tasks.workunit.client.0.smithi029.stdout: 1 1140277 9.33 MB/sec execute 506 sec latency 39.279 ms 2024-03-16T12:02:42.359 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:42.359+0000 7f3bc44b2640 1 -- 172.21.15.29:0/643253818 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3bbc105660 msgr2=0x7f3bbc105a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:02:42.359 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:42.359+0000 7f3bc44b2640 1 --2- 172.21.15.29:0/643253818 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3bbc105660 0x7f3bbc105a40 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f3bb8009900 tx=0x7f3bb802bc30 comp rx=0 tx=0).stop 2024-03-16T12:02:42.359 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:42.360+0000 7f3bc44b2640 1 -- 172.21.15.29:0/643253818 shutdown_connections 2024-03-16T12:02:42.359 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:42.360+0000 7f3bc44b2640 1 --2- 172.21.15.29:0/643253818 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3bbc1012d0 0x7f3bbc10d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:02:42.359 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:42.360+0000 7f3bc44b2640 1 --2- 172.21.15.29:0/643253818 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3bbc100930 0x7f3bbc100d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:02:42.359 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:42.360+0000 7f3bc44b2640 1 --2- 172.21.15.29:0/643253818 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3bbc105660 0x7f3bbc105a40 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:02:42.360 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:42.360+0000 7f3bc44b2640 1 -- 172.21.15.29:0/643253818 >> 172.21.15.29:0/643253818 conn(0x7f3bbc0774b0 msgr2=0x7f3bbc0778c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:02:42.360 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:42.360+0000 7f3bc44b2640 1 -- 172.21.15.29:0/643253818 shutdown_connections 2024-03-16T12:02:42.360 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:42.360+0000 7f3bc44b2640 1 -- 172.21.15.29:0/643253818 wait complete. 2024-03-16T12:02:42.360 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:42.360+0000 7f3bc44b2640 1 Processor -- start 2024-03-16T12:02:42.360 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:42.360+0000 7f3bc44b2640 1 -- start start 2024-03-16T12:02:42.360 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:42.360+0000 7f3bc44b2640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3bbc100930 0x7f3bbc198050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:02:42.360 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:42.360+0000 7f3bc44b2640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3bbc1012d0 0x7f3bbc198590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:02:42.360 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:42.360+0000 7f3bc44b2640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3bbc105660 0x7f3bbc19c910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:02:42.360 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:42.360+0000 7f3bc44b2640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f3bbc111c90 con 0x7f3bbc105660 2024-03-16T12:02:42.360 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:42.360+0000 7f3bc44b2640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f3bbc111b10 con 0x7f3bbc100930 2024-03-16T12:02:42.360 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:42.360+0000 7f3bc44b2640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f3bbc111e10 con 0x7f3bbc1012d0 2024-03-16T12:02:42.360 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:42.361+0000 7f3bc2a4f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3bbc105660 0x7f3bbc19c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:02:42.360 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:42.361+0000 7f3bc2a4f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3bbc105660 0x7f3bbc19c910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39160/0 (socket says 172.21.15.29:39160) 2024-03-16T12:02:42.360 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:42.361+0000 7f3bc2a4f640 1 -- 172.21.15.29:0/1818965994 learned_addr learned my addr 172.21.15.29:0/1818965994 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T12:02:42.361 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:42.361+0000 7f3bc1a4d640 1 --2- 172.21.15.29:0/1818965994 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3bbc1012d0 0x7f3bbc198590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:02:42.361 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:42.361+0000 7f3bc2a4f640 1 -- 172.21.15.29:0/1818965994 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3bbc1012d0 msgr2=0x7f3bbc198590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:02:42.361 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:42.361+0000 7f3bc2a4f640 1 --2- 172.21.15.29:0/1818965994 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3bbc1012d0 0x7f3bbc198590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:02:42.361 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:42.361+0000 7f3bc2a4f640 1 -- 172.21.15.29:0/1818965994 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3bbc100930 msgr2=0x7f3bbc198050 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-16T12:02:42.361 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:42.361+0000 7f3bc2a4f640 1 --2- 172.21.15.29:0/1818965994 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3bbc100930 0x7f3bbc198050 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:02:42.361 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:42.361+0000 7f3bc2a4f640 1 -- 172.21.15.29:0/1818965994 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3bbc19d040 con 0x7f3bbc105660 2024-03-16T12:02:42.361 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:42.361+0000 7f3bc2a4f640 1 --2- 172.21.15.29:0/1818965994 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3bbc105660 0x7f3bbc19c910 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f3bb400eb10 tx=0x7f3bb400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:02:42.361 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:42.361+0000 7f3bb37fe640 1 -- 172.21.15.29:0/1818965994 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bb4004b20 con 0x7f3bbc105660 2024-03-16T12:02:42.361 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:42.361+0000 7f3bc44b2640 1 -- 172.21.15.29:0/1818965994 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3bbc19d330 con 0x7f3bbc105660 2024-03-16T12:02:42.361 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:42.361+0000 7f3bb37fe640 1 -- 172.21.15.29:0/1818965994 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3bb4004cc0 con 0x7f3bbc105660 2024-03-16T12:02:42.361 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:42.361+0000 7f3bb37fe640 1 -- 172.21.15.29:0/1818965994 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bb4010760 con 0x7f3bbc105660 2024-03-16T12:02:42.361 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:42.361+0000 7f3bc44b2640 1 -- 172.21.15.29:0/1818965994 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3bbc1a4bc0 con 0x7f3bbc105660 2024-03-16T12:02:42.363 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:42.364+0000 7f3bb37fe640 1 -- 172.21.15.29:0/1818965994 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7f3bb4005020 con 0x7f3bbc105660 2024-03-16T12:02:42.363 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:42.364+0000 7f3bc7179740 1 Processor -- start 2024-03-16T12:02:42.364 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:42.365+0000 7f3bc7179740 1 -- start start 2024-03-16T12:02:42.364 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:42.365+0000 7f3bc7179740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f128650bd0 0x55f128650fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:02:42.364 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:42.365+0000 7f3bc7179740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f1286514f0 0x55f12865e6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:02:42.364 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:42.365+0000 7f3bc7179740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f12865ec30 0x55f128661020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:02:42.364 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:42.365+0000 7f3bc7179740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55f1285b5a40 con 0x55f128650bd0 2024-03-16T12:02:42.364 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:42.365+0000 7f3bc7179740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x55f12855e920 con 0x55f1286514f0 2024-03-16T12:02:42.364 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:42.365+0000 7f3bc7179740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55f1285b1ae0 con 0x55f12865ec30 2024-03-16T12:02:42.364 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:42.365+0000 7f3bc224e640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f1286514f0 0x55f12865e6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:02:42.364 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:42.365+0000 7f3bc1a4d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f128650bd0 0x55f128650fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:02:42.364 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:42.365+0000 7f3bc224e640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f1286514f0 0x55f12865e6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.29:54768/0 (socket says 172.21.15.29:54768) 2024-03-16T12:02:42.364 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:42.365+0000 7f3bc1a4d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f128650bd0 0x55f128650fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39170/0 (socket says 172.21.15.29:39170) 2024-03-16T12:02:42.364 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:42.365+0000 7f3bc224e640 1 -- 172.21.15.29:0/1860385469 learned_addr learned my addr 172.21.15.29:0/1860385469 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T12:02:42.364 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:42.365+0000 7f3bc1a4d640 1 --2- 172.21.15.29:0/1860385469 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f12865ec30 0x55f128661020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:02:42.364 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:42.365+0000 7f3bb37fe640 1 --2- 172.21.15.29:0/1818965994 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f3ba80a2a10 0x7f3ba80a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:02:42.365 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:42.365+0000 7f3bc224e640 1 -- 172.21.15.29:0/1860385469 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f12865ec30 msgr2=0x55f128661020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:02:42.365 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:42.365+0000 7f3bc224e640 1 --2- 172.21.15.29:0/1860385469 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f12865ec30 0x55f128661020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:02:42.365 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:42.365+0000 7f3bc224e640 1 -- 172.21.15.29:0/1860385469 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f128650bd0 msgr2=0x55f128650fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:02:42.365 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:42.365+0000 7f3bc224e640 1 --2- 172.21.15.29:0/1860385469 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f128650bd0 0x55f128650fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:02:42.365 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:42.365+0000 7f3bc2a4f640 1 --2- 172.21.15.29:0/1818965994 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f3ba80a2a10 0x7f3ba80a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:02:42.365 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:42.365+0000 7f3bc224e640 1 -- 172.21.15.29:0/1860385469 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f128576460 con 0x55f1286514f0 2024-03-16T12:02:42.365 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:42.365+0000 7f3bc1a4d640 1 --2- 172.21.15.29:0/1860385469 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f12865ec30 0x55f128661020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T12:02:42.365 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:42.365+0000 7f3bb37fe640 1 -- 172.21.15.29:0/1818965994 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 8556+0+0 (secure 0 0 0) 0x7f3bb40cf4b0 con 0x7f3bbc105660 2024-03-16T12:02:42.365 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:42.366+0000 7f3bc224e640 1 --2- 172.21.15.29:0/1860385469 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f1286514f0 0x55f12865e6f0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f3bb802c170 tx=0x7f3bb8005e80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3bd4fb0348041a2f server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:02:42.365 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:42.366+0000 7f3bc2a4f640 1 --2- 172.21.15.29:0/1818965994 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f3ba80a2a10 0x7f3ba80a4ed0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f3bb409b7f0 tx=0x7f3bb40667b0 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:02:42.365 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:42.366+0000 7f3bb0bfe640 1 -- 172.21.15.29:0/1860385469 <== mon.2 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bb802ca80 con 0x55f1286514f0 2024-03-16T12:02:42.365 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:42.366+0000 7f3bb0bfe640 1 -- 172.21.15.29:0/1860385469 <== mon.2 v2:172.21.15.152:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3bb8042070 con 0x55f1286514f0 2024-03-16T12:02:42.365 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:42.366+0000 7f3bb0bfe640 1 -- 172.21.15.29:0/1860385469 <== mon.2 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bb80046c0 con 0x55f1286514f0 2024-03-16T12:02:42.369 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:42.367+0000 7f3bc7179740 1 -- 172.21.15.29:0/1860385469 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f1286514f0 msgr2=0x55f12865e6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:02:42.369 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:42.367+0000 7f3bc7179740 1 --2- 172.21.15.29:0/1860385469 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f1286514f0 0x55f12865e6f0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f3bb802c170 tx=0x7f3bb8005e80 comp rx=0 tx=0).stop 2024-03-16T12:02:42.369 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:42.367+0000 7f3bc7179740 1 -- 172.21.15.29:0/1860385469 shutdown_connections 2024-03-16T12:02:42.369 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:42.367+0000 7f3bc7179740 1 --2- 172.21.15.29:0/1860385469 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f12865ec30 0x55f128661020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:02:42.369 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:42.367+0000 7f3bc7179740 1 --2- 172.21.15.29:0/1860385469 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f1286514f0 0x55f12865e6f0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:02:42.369 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:42.367+0000 7f3bc7179740 1 --2- 172.21.15.29:0/1860385469 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f128650bd0 0x55f128650fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:02:42.369 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:42.367+0000 7f3bc7179740 1 -- 172.21.15.29:0/1860385469 >> 172.21.15.29:0/1860385469 conn(0x55f12864ba50 msgr2=0x55f12864be30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:02:42.369 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:42.367+0000 7f3bc7179740 1 -- 172.21.15.29:0/1860385469 shutdown_connections 2024-03-16T12:02:42.370 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:42.367+0000 7f3bc7179740 1 -- 172.21.15.29:0/1860385469 wait complete. 2024-03-16T12:02:42.370 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:42.367+0000 7f3bc7179740 1 -- start start 2024-03-16T12:02:42.370 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:42.367+0000 7f3bc7179740 1 Processor -- start 2024-03-16T12:02:42.370 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:42.367+0000 7f3bc7179740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f128650bd0 0x55f12865bb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:02:42.370 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:42.367+0000 7f3bc7179740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f1286514f0 0x55f12865c090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:02:42.370 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:42.368+0000 7f3bc7179740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f12865ec30 0x55f12865c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:02:42.370 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:42.368+0000 7f3bc7179740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55f128661800 con 0x55f1286514f0 2024-03-16T12:02:42.370 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:42.368+0000 7f3bc7179740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x55f128571d70 con 0x55f12865ec30 2024-03-16T12:02:42.370 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:42.368+0000 7f3bc7179740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55f128661980 con 0x55f128650bd0 2024-03-16T12:02:42.370 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:42.368+0000 7f3bc7179740 10 client.0 _ll_register_callbacks cb 0x55f1282c9a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T12:02:42.370 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:42.368+0000 7f3bc224e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f1286514f0 0x55f12865c090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:02:42.370 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:42.368+0000 7f3bc224e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f1286514f0 0x55f12865c090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39180/0 (socket says 172.21.15.29:39180) 2024-03-16T12:02:42.370 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:42.368+0000 7f3bc224e640 1 -- 172.21.15.29:0/2956275745 learned_addr learned my addr 172.21.15.29:0/2956275745 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T12:02:42.370 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:42.368+0000 7f3bc1a4d640 1 --2- 172.21.15.29:0/2956275745 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f128650bd0 0x55f12865bb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:02:42.370 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:42.368+0000 7f3bc1a4d640 1 --2- 172.21.15.29:0/2956275745 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f12865ec30 0x55f12865c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:02:42.371 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:42.368+0000 7f3bc1a4d640 1 -- 172.21.15.29:0/2956275745 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f12865ec30 msgr2=0x55f12865c5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:02:42.371 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:42.368+0000 7f3bc1a4d640 1 --2- 172.21.15.29:0/2956275745 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f12865ec30 0x55f12865c5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:02:42.371 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:42.368+0000 7f3bc1a4d640 1 -- 172.21.15.29:0/2956275745 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f1286514f0 msgr2=0x55f12865c090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:02:42.371 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:42.368+0000 7f3bc1a4d640 1 --2- 172.21.15.29:0/2956275745 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f1286514f0 0x55f12865c090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:02:42.371 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:42.368+0000 7f3bc1a4d640 1 -- 172.21.15.29:0/2956275745 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f1282ca210 con 0x55f128650bd0 2024-03-16T12:02:42.371 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:42.368+0000 7f3bc224e640 1 --2- 172.21.15.29:0/2956275745 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f1286514f0 0x55f12865c090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:02:42.371 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:42.368+0000 7f3bc1a4d640 1 --2- 172.21.15.29:0/2956275745 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f128650bd0 0x55f12865bb50 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x55f128650ab0 tx=0x7f3bac015040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5bfb210b9882166b server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:02:42.371 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:42.368+0000 7f3ba5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-16T12:02:42.371 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:42.369+0000 7f3ba5ffb640 1 -- 172.21.15.29:0/2956275745 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bac014030 con 0x55f128650bd0 2024-03-16T12:02:42.371 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:42.369+0000 7f3bc7179740 1 -- 172.21.15.29:0/2956275745 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3bb8042070 con 0x55f128650bd0 2024-03-16T12:02:42.371 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:42.369+0000 7f3ba5ffb640 1 -- 172.21.15.29:0/2956275745 <== mon.1 v2:172.21.15.154:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 8556+0+0 (secure 0 0 0) 0x7f3bac00fe30 con 0x55f128650bd0 2024-03-16T12:02:42.371 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:42.369+0000 7f3ba5ffb640 1 -- 172.21.15.29:0/2956275745 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f3bac029050 con 0x55f128650bd0 2024-03-16T12:02:42.371 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:42.369+0000 7f3bc7179740 10 client.26221 fetch_fsmap learned FSMap version 28 2024-03-16T12:02:42.371 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:42.369+0000 7f3bc7179740 1 -- 172.21.15.29:0/2956275745 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55f128396620 con 0x55f128650bd0 2024-03-16T12:02:42.371 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:42.369+0000 7f3ba5ffb640 1 -- 172.21.15.29:0/2956275745 <== mon.1 v2:172.21.15.154:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3bb8042070 con 0x55f128650bd0 2024-03-16T12:02:42.372 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:42.369+0000 7f3ba5ffb640 1 -- 172.21.15.29:0/2956275745 <== mon.1 v2:172.21.15.154:3300/0 5 ==== fsmap(e 28) ==== 8338+0+0 (secure 0 0 0) 0x7f3bac004c50 con 0x55f128650bd0 2024-03-16T12:02:42.372 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:42.369+0000 7f3bc7179740 10 client.26221 fetch_fsmap finished waiting for FSMap version 28 2024-03-16T12:02:42.372 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:42.369+0000 7f3bc7179740 10 client.26221 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T12:02:42.372 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:42.369+0000 7f3bc7179740 20 client.26221 populate_metadata read hostname 'smithi029' 2024-03-16T12:02:42.372 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:42.369+0000 7f3bc7179740 1 --2- 172.21.15.29:0/2956275745 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55f12866ed50 0x55f128671180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:02:42.372 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:42.369+0000 7f3bc7179740 4 client.26221 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T12:02:42.372 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:42.369+0000 7f3bc7179740 1 -- 172.21.15.29:0/2956275745 --> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f1285c2f80 con 0x55f12866ed50 2024-03-16T12:02:42.372 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:42.369+0000 7f3bc224e640 1 --2- 172.21.15.29:0/2956275745 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55f12866ed50 0x55f128671180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:02:42.372 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:42.370+0000 7f3ba5ffb640 10 client.26221 ms_handle_connect on v2:172.21.15.152:6838/1548731833 2024-03-16T12:02:42.372 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:42.370+0000 7f3bc224e640 1 --2- 172.21.15.29:0/2956275745 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55f12866ed50 0x55f128671180 crc :-1 s=READY pgs=144 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=90c8df4ea458183f server_cookie=5fcc7830529cf862 in_seq=0 out_seq=0 2024-03-16T12:02:42.373 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:42.372+0000 7f3ba5ffb640 1 -- 172.21.15.29:0/2956275745 <== mds.0 v2:172.21.15.152:6838/1548731833 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55f1285c2f80 con 0x55f12866ed50 2024-03-16T12:02:42.373 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:42.372+0000 7f3ba5ffb640 10 client.26221 handle_command_reply: tid=0 2024-03-16T12:02:42.373 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:42.372+0000 7f3bc7179740 1 client.26221 shutdown 2024-03-16T12:02:42.374 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:42.372+0000 7f3bc7179740 1 -- 172.21.15.29:0/2956275745 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f128650bd0 msgr2=0x55f12865bb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:02:42.374 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:42.372+0000 7f3bc7179740 1 --2- 172.21.15.29:0/2956275745 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f128650bd0 0x55f12865bb50 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x55f128650ab0 tx=0x7f3bac015040 comp rx=0 tx=0).stop 2024-03-16T12:02:42.374 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:42.372+0000 7f3bc7179740 1 -- 172.21.15.29:0/2956275745 shutdown_connections 2024-03-16T12:02:42.374 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:42.372+0000 7f3bc7179740 1 --2- 172.21.15.29:0/2956275745 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55f12866ed50 0x55f128671180 crc :-1 s=READY pgs=144 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:02:42.374 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:42.372+0000 7f3bc7179740 1 --2- 172.21.15.29:0/2956275745 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f12865ec30 0x55f12865c5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:02:42.374 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:42.372+0000 7f3bc7179740 1 --2- 172.21.15.29:0/2956275745 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f1286514f0 0x55f12865c090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:02:42.374 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:42.372+0000 7f3bc7179740 1 --2- 172.21.15.29:0/2956275745 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f128650bd0 0x55f12865bb50 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:02:42.374 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:42.372+0000 7f3ba5ffb640 0 client.26221 ms_handle_reset on v2:172.21.15.152:6838/1548731833 2024-03-16T12:02:42.374 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:42.372+0000 7f3bc7179740 1 -- 172.21.15.29:0/2956275745 >> 172.21.15.29:0/2956275745 conn(0x55f12864ba50 msgr2=0x55f12864d7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:02:42.374 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:42.372+0000 7f3bc7179740 1 -- 172.21.15.29:0/2956275745 shutdown_connections 2024-03-16T12:02:42.374 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:42.372+0000 7f3bc7179740 1 -- 172.21.15.29:0/2956275745 wait complete. 2024-03-16T12:02:42.374 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:42.373+0000 7f3bc7179740 20 client.26221 trim_cache size 0 max 16384 2024-03-16T12:02:42.390 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:42.389+0000 7f3bc7179740 1 Processor -- start 2024-03-16T12:02:42.390 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:42.389+0000 7f3bc7179740 1 -- start start 2024-03-16T12:02:42.391 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:42.389+0000 7f3bc7179740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f128650bd0 0x55f12865e280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:02:42.391 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:42.389+0000 7f3bc7179740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f1286514f0 0x55f12866a610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:02:42.391 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:42.389+0000 7f3bc7179740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f12865ec30 0x55f12865acb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:02:42.391 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:42.389+0000 7f3bc7179740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f3bbc1adad0 con 0x55f1286514f0 2024-03-16T12:02:42.391 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:42.389+0000 7f3bc7179740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f3bbc1ab3a0 con 0x55f12865ec30 2024-03-16T12:02:42.391 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:42.389+0000 7f3bc7179740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f3bbc1a9210 con 0x55f128650bd0 2024-03-16T12:02:42.391 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:42.389+0000 7f3bc224e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f1286514f0 0x55f12866a610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:02:42.391 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:42.389+0000 7f3bc224e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f1286514f0 0x55f12866a610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39182/0 (socket says 172.21.15.29:39182) 2024-03-16T12:02:42.391 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:42.389+0000 7f3bc1a4d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f128650bd0 0x55f12865e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:02:42.391 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:42.389+0000 7f3bc224e640 1 -- 172.21.15.29:0/567912694 learned_addr learned my addr 172.21.15.29:0/567912694 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T12:02:42.391 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:42.389+0000 7f3bc1a4d640 1 --2- 172.21.15.29:0/567912694 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f12865ec30 0x55f12865acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:02:42.391 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:42.389+0000 7f3bc1a4d640 1 -- 172.21.15.29:0/567912694 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f12865ec30 msgr2=0x55f12865acb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:02:42.391 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:42.389+0000 7f3bc1a4d640 1 --2- 172.21.15.29:0/567912694 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f12865ec30 0x55f12865acb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:02:42.391 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:42.389+0000 7f3bc1a4d640 1 -- 172.21.15.29:0/567912694 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f1286514f0 msgr2=0x55f12866a610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:02:42.392 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:42.389+0000 7f3bc1a4d640 1 --2- 172.21.15.29:0/567912694 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f1286514f0 0x55f12866a610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:02:42.392 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:42.389+0000 7f3bc1a4d640 1 -- 172.21.15.29:0/567912694 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f12864b880 con 0x55f128650bd0 2024-03-16T12:02:42.392 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:42.389+0000 7f3bc224e640 1 --2- 172.21.15.29:0/567912694 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f1286514f0 0x55f12866a610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:02:42.392 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:42.389+0000 7f3bc1a4d640 1 --2- 172.21.15.29:0/567912694 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f128650bd0 0x55f12865e280 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f3bac001eb0 tx=0x7f3bac005f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a1364fc3e486cb90 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:02:42.392 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:42.390+0000 7f3ba7fff640 1 -- 172.21.15.29:0/567912694 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bac014030 con 0x55f128650bd0 2024-03-16T12:02:42.392 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:42.390+0000 7f3ba7fff640 1 -- 172.21.15.29:0/567912694 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3bac00fe30 con 0x55f128650bd0 2024-03-16T12:02:42.392 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:42.390+0000 7f3ba7fff640 1 -- 172.21.15.29:0/567912694 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bac005400 con 0x55f128650bd0 2024-03-16T12:02:42.392 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:42.390+0000 7f3bc7179740 1 -- 172.21.15.29:0/567912694 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f128650bd0 msgr2=0x55f12865e280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:02:42.392 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:42.390+0000 7f3bc7179740 1 --2- 172.21.15.29:0/567912694 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f128650bd0 0x55f12865e280 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f3bac001eb0 tx=0x7f3bac005f90 comp rx=0 tx=0).stop 2024-03-16T12:02:42.392 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:42.390+0000 7f3bc7179740 1 -- 172.21.15.29:0/567912694 shutdown_connections 2024-03-16T12:02:42.392 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:42.390+0000 7f3bc7179740 1 --2- 172.21.15.29:0/567912694 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f12865ec30 0x55f12865acb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:02:42.392 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:42.390+0000 7f3bc7179740 1 --2- 172.21.15.29:0/567912694 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f1286514f0 0x55f12866a610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:02:42.392 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:42.390+0000 7f3bc7179740 1 --2- 172.21.15.29:0/567912694 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f128650bd0 0x55f12865e280 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:02:42.392 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:42.390+0000 7f3bc7179740 1 -- 172.21.15.29:0/567912694 >> 172.21.15.29:0/567912694 conn(0x55f12864ba50 msgr2=0x55f12864cea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:02:42.392 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:42.390+0000 7f3bc7179740 1 -- 172.21.15.29:0/567912694 shutdown_connections 2024-03-16T12:02:42.393 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:42.390+0000 7f3bc7179740 1 -- 172.21.15.29:0/567912694 wait complete. 2024-03-16T12:02:42.393 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:42.391+0000 7f3bc7179740 1 -- start start 2024-03-16T12:02:42.393 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:42.391+0000 7f3bc7179740 1 Processor -- start 2024-03-16T12:02:42.393 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:42.391+0000 7f3bc7179740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f128650bd0 0x55f1286668e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:02:42.393 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:42.391+0000 7f3bc7179740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f1286514f0 0x55f128666e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:02:42.393 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:42.391+0000 7f3bc7179740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f12865ec30 0x55f128661ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:02:42.393 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:42.391+0000 7f3bc7179740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f3bbc1a9860 con 0x55f12865ec30 2024-03-16T12:02:42.393 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:42.391+0000 7f3bc7179740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f3bbc1aacd0 con 0x55f1286514f0 2024-03-16T12:02:42.393 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:42.391+0000 7f3bc7179740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f3bbc1a9f30 con 0x55f128650bd0 2024-03-16T12:02:42.393 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:42.391+0000 7f3bc7179740 10 client.0 _ll_register_callbacks cb 0x7f3bac009d50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T12:02:42.393 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:42.391+0000 7f3bc1a4d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f12865ec30 0x55f128661ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:02:42.393 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:42.391+0000 7f3bc1a4d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f128650bd0 0x55f1286668e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:02:42.393 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:42.391+0000 7f3bc1a4d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f128650bd0 0x55f1286668e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.29:51724/0 (socket says 172.21.15.29:51724) 2024-03-16T12:02:42.393 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:42.391+0000 7f3bc1a4d640 1 -- 172.21.15.29:0/2948310349 learned_addr learned my addr 172.21.15.29:0/2948310349 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T12:02:42.394 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:42.391+0000 7f3bc224e640 1 --2- 172.21.15.29:0/2948310349 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f1286514f0 0x55f128666e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:02:42.394 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:42.392+0000 7f3bc1a4d640 1 -- 172.21.15.29:0/2948310349 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f1286514f0 msgr2=0x55f128666e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:02:42.394 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:42.392+0000 7f3bc1a4d640 1 --2- 172.21.15.29:0/2948310349 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f1286514f0 0x55f128666e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:02:42.394 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:42.392+0000 7f3bc1a4d640 1 -- 172.21.15.29:0/2948310349 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f12865ec30 msgr2=0x55f128661ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:02:42.394 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:42.392+0000 7f3bc1a4d640 1 --2- 172.21.15.29:0/2948310349 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f12865ec30 0x55f128661ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:02:42.394 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:42.392+0000 7f3bc1a4d640 1 -- 172.21.15.29:0/2948310349 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f1286624e0 con 0x55f128650bd0 2024-03-16T12:02:42.394 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:42.392+0000 7f3bc1a4d640 1 --2- 172.21.15.29:0/2948310349 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f128650bd0 0x55f1286668e0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f3bac010d70 tx=0x7f3bac000bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d6bdc7ec302d3ca0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:02:42.394 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:42.392+0000 7f3ba5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-16T12:02:42.394 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:42.392+0000 7f3ba5ffb640 1 -- 172.21.15.29:0/2948310349 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bac014070 con 0x55f128650bd0 2024-03-16T12:02:42.394 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:42.392+0000 7f3ba5ffb640 1 -- 172.21.15.29:0/2948310349 <== mon.1 v2:172.21.15.154:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 8556+0+0 (secure 0 0 0) 0x7f3bac021a00 con 0x55f128650bd0 2024-03-16T12:02:42.394 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:42.392+0000 7f3bc7179740 1 -- 172.21.15.29:0/2948310349 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3bac00fe30 con 0x55f128650bd0 2024-03-16T12:02:42.394 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:42.392+0000 7f3ba5ffb640 1 -- 172.21.15.29:0/2948310349 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f3bac033040 con 0x55f128650bd0 2024-03-16T12:02:42.394 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:42.392+0000 7f3bc7179740 10 client.26227 fetch_fsmap learned FSMap version 28 2024-03-16T12:02:42.394 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:42.392+0000 7f3bc7179740 1 -- 172.21.15.29:0/2948310349 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55f1286d4b70 con 0x55f128650bd0 2024-03-16T12:02:42.395 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:42.393+0000 7f3ba5ffb640 1 -- 172.21.15.29:0/2948310349 <== mon.1 v2:172.21.15.154:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3bac00fe30 con 0x55f128650bd0 2024-03-16T12:02:42.395 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:42.393+0000 7f3ba5ffb640 1 -- 172.21.15.29:0/2948310349 <== mon.1 v2:172.21.15.154:3300/0 5 ==== fsmap(e 28) ==== 8338+0+0 (secure 0 0 0) 0x7f3bac004d30 con 0x55f128650bd0 2024-03-16T12:02:42.395 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:42.393+0000 7f3bc7179740 10 client.26227 fetch_fsmap finished waiting for FSMap version 28 2024-03-16T12:02:42.395 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:42.393+0000 7f3bc7179740 10 client.26227 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T12:02:42.395 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:42.393+0000 7f3bc7179740 20 client.26227 populate_metadata read hostname 'smithi029' 2024-03-16T12:02:42.395 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:42.393+0000 7f3bc7179740 1 --2- 172.21.15.29:0/2948310349 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55f1286d5180 0x55f1286d7570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:02:42.395 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:42.393+0000 7f3bc7179740 4 client.26227 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-16T12:02:42.395 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:42.393+0000 7f3bc7179740 1 -- 172.21.15.29:0/2948310349 --> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55f1285bd2d0 con 0x55f1286d5180 2024-03-16T12:02:42.395 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:42.393+0000 7f3bc224e640 1 --2- 172.21.15.29:0/2948310349 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55f1286d5180 0x55f1286d7570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:02:42.395 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:42.393+0000 7f3bc224e640 1 --2- 172.21.15.29:0/2948310349 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55f1286d5180 0x55f1286d7570 crc :-1 s=READY pgs=145 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=eec28948107f4ddb server_cookie=f990229e6564ef71 in_seq=0 out_seq=0 2024-03-16T12:02:42.395 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:42.393+0000 7f3ba5ffb640 10 client.26227 ms_handle_connect on v2:172.21.15.152:6838/1548731833 2024-03-16T12:02:42.543 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:02:42 smithi154 ceph-mon[36320]: pgmap v372: 97 pgs: 97 active+clean; 76 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 2.3 MiB/s wr, 241 op/s 2024-03-16T12:02:42.575 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:42.576+0000 7f3ba5ffb640 1 -- 172.21.15.29:0/2948310349 <== mds.0 v2:172.21.15.152:6838/1548731833 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55f1285bd2d0 con 0x55f1286d5180 2024-03-16T12:02:42.575 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:42.576+0000 7f3ba5ffb640 10 client.26227 handle_command_reply: tid=0 2024-03-16T12:02:42.575 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:42.576+0000 7f3bc7179740 1 client.26227 shutdown 2024-03-16T12:02:42.575 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:42.576+0000 7f3bc7179740 1 -- 172.21.15.29:0/2948310349 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f128650bd0 msgr2=0x55f1286668e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:02:42.575 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:42.576+0000 7f3bc7179740 1 --2- 172.21.15.29:0/2948310349 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f128650bd0 0x55f1286668e0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f3bac010d70 tx=0x7f3bac000bc0 comp rx=0 tx=0).stop 2024-03-16T12:02:42.575 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:42.577+0000 7f3bc7179740 1 -- 172.21.15.29:0/2948310349 shutdown_connections 2024-03-16T12:02:42.575 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:42.577+0000 7f3bc7179740 1 --2- 172.21.15.29:0/2948310349 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f12865ec30 0x55f128661ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:02:42.575 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:42.577+0000 7f3bc7179740 1 --2- 172.21.15.29:0/2948310349 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55f1286d5180 0x55f1286d7570 crc :-1 s=READY pgs=145 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:02:42.575 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:42.577+0000 7f3bc7179740 1 --2- 172.21.15.29:0/2948310349 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f1286514f0 0x55f128666e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:02:42.575 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:42.577+0000 7f3ba5ffb640 0 client.26227 ms_handle_reset on v2:172.21.15.152:6838/1548731833 2024-03-16T12:02:42.575 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:42.577+0000 7f3bc7179740 1 --2- 172.21.15.29:0/2948310349 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f128650bd0 0x55f1286668e0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:02:42.575 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:42.577+0000 7f3bc7179740 1 -- 172.21.15.29:0/2948310349 >> 172.21.15.29:0/2948310349 conn(0x55f12864ba50 msgr2=0x55f12864cea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:02:42.576 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:42.577+0000 7f3bc7179740 1 -- 172.21.15.29:0/2948310349 shutdown_connections 2024-03-16T12:02:42.576 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:42.577+0000 7f3bc7179740 1 -- 172.21.15.29:0/2948310349 wait complete. 2024-03-16T12:02:42.576 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:42.577+0000 7f3bc7179740 20 client.26227 trim_cache size 0 max 16384 2024-03-16T12:02:42.576 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:{ 2024-03-16T12:02:42.576 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "message": "", 2024-03-16T12:02:42.576 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "return_code": 0 2024-03-16T12:02:42.576 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:} 2024-03-16T12:02:42.577 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:42.578+0000 7f3bc44b2640 1 -- 172.21.15.29:0/1818965994 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f3ba80a2a10 msgr2=0x7f3ba80a4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:02:42.577 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:42.578+0000 7f3bc44b2640 1 --2- 172.21.15.29:0/1818965994 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f3ba80a2a10 0x7f3ba80a4ed0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f3bb409b7f0 tx=0x7f3bb40667b0 comp rx=0 tx=0).stop 2024-03-16T12:02:42.577 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:42.578+0000 7f3bc44b2640 1 -- 172.21.15.29:0/1818965994 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3bbc105660 msgr2=0x7f3bbc19c910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:02:42.577 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:42.578+0000 7f3bc44b2640 1 --2- 172.21.15.29:0/1818965994 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3bbc105660 0x7f3bbc19c910 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f3bb400eb10 tx=0x7f3bb400c2d0 comp rx=0 tx=0).stop 2024-03-16T12:02:42.577 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:42.578+0000 7f3bc44b2640 1 -- 172.21.15.29:0/1818965994 shutdown_connections 2024-03-16T12:02:42.577 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:42.578+0000 7f3bc44b2640 1 --2- 172.21.15.29:0/1818965994 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f3ba80a2a10 0x7f3ba80a4ed0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:02:42.577 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:42.578+0000 7f3bc44b2640 1 --2- 172.21.15.29:0/1818965994 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3bbc105660 0x7f3bbc19c910 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:02:42.577 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:42.578+0000 7f3bc44b2640 1 --2- 172.21.15.29:0/1818965994 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3bbc1012d0 0x7f3bbc198590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:02:42.578 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:42.578+0000 7f3bc44b2640 1 --2- 172.21.15.29:0/1818965994 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3bbc100930 0x7f3bbc198050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:02:42.578 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:42.578+0000 7f3bc44b2640 1 -- 172.21.15.29:0/1818965994 >> 172.21.15.29:0/1818965994 conn(0x7f3bbc0774b0 msgr2=0x7f3bbc0fe480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:02:42.578 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:42.578+0000 7f3bc44b2640 1 -- 172.21.15.29:0/1818965994 shutdown_connections 2024-03-16T12:02:42.578 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:42.578+0000 7f3bc44b2640 1 -- 172.21.15.29:0/1818965994 wait complete. 2024-03-16T12:02:42.586 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:02:42 smithi029 ceph-mon[31975]: pgmap v372: 97 pgs: 97 active+clean; 76 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 2.3 MiB/s wr, 241 op/s 2024-03-16T12:02:42.625 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:02:42 smithi152 ceph-mon[36291]: pgmap v372: 97 pgs: 97 active+clean; 76 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 2.3 MiB/s wr, 241 op/s 2024-03-16T12:02:42.804 INFO:tasks.workunit.client.0.smithi029.stdout: 1 1140946 9.31 MB/sec execute 507 sec latency 101.731 ms 2024-03-16T12:02:43.804 INFO:tasks.workunit.client.0.smithi029.stdout: 1 1142760 9.31 MB/sec execute 508 sec latency 22.495 ms 2024-03-16T12:02:44.427 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:02:44 smithi152 ceph-mon[36291]: pgmap v373: 97 pgs: 97 active+clean; 82 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 2.3 MiB/s wr, 174 op/s 2024-03-16T12:02:44.543 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:02:44 smithi154 ceph-mon[36320]: pgmap v373: 97 pgs: 97 active+clean; 82 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 2.3 MiB/s wr, 174 op/s 2024-03-16T12:02:44.586 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:02:44 smithi029 ceph-mon[31975]: pgmap v373: 97 pgs: 97 active+clean; 82 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 2.3 MiB/s wr, 174 op/s 2024-03-16T12:02:44.804 INFO:tasks.workunit.client.0.smithi029.stdout: 1 1144030 9.31 MB/sec execute 509 sec latency 93.062 ms 2024-03-16T12:02:45.804 INFO:tasks.workunit.client.0.smithi029.stdout: 1 1145353 9.30 MB/sec execute 510 sec latency 108.023 ms 2024-03-16T12:02:46.543 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:02:46 smithi154 ceph-mon[36320]: pgmap v374: 97 pgs: 97 active+clean; 82 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 2.0 MiB/s wr, 132 op/s 2024-03-16T12:02:46.586 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:02:46 smithi029 ceph-mon[31975]: pgmap v374: 97 pgs: 97 active+clean; 82 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 2.0 MiB/s wr, 132 op/s 2024-03-16T12:02:46.625 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:02:46 smithi152 ceph-mon[36291]: pgmap v374: 97 pgs: 97 active+clean; 82 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 2.0 MiB/s wr, 132 op/s 2024-03-16T12:02:46.804 INFO:tasks.workunit.client.0.smithi029.stdout: 1 1147361 9.31 MB/sec execute 511 sec latency 49.429 ms 2024-03-16T12:02:47.804 INFO:tasks.workunit.client.0.smithi029.stdout: 1 1148361 9.30 MB/sec execute 512 sec latency 22.873 ms 2024-03-16T12:02:48.543 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:02:48 smithi154 ceph-mon[36320]: pgmap v375: 97 pgs: 97 active+clean; 78 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 3.2 MiB/s wr, 274 op/s 2024-03-16T12:02:48.586 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:02:48 smithi029 ceph-mon[31975]: pgmap v375: 97 pgs: 97 active+clean; 78 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 3.2 MiB/s wr, 274 op/s 2024-03-16T12:02:48.625 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:02:48 smithi152 ceph-mon[36291]: pgmap v375: 97 pgs: 97 active+clean; 78 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 3.2 MiB/s wr, 274 op/s 2024-03-16T12:02:48.804 INFO:tasks.workunit.client.0.smithi029.stdout: 1 1150668 9.30 MB/sec execute 513 sec latency 17.676 ms 2024-03-16T12:02:49.804 INFO:tasks.workunit.client.0.smithi029.stdout: 1 1151428 9.30 MB/sec execute 514 sec latency 28.388 ms 2024-03-16T12:02:50.543 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:02:50 smithi154 ceph-mon[36320]: pgmap v376: 97 pgs: 97 active+clean; 78 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.2 MiB/s wr, 212 op/s 2024-03-16T12:02:50.586 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:02:50 smithi029 ceph-mon[31975]: pgmap v376: 97 pgs: 97 active+clean; 78 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.2 MiB/s wr, 212 op/s 2024-03-16T12:02:50.625 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:02:50 smithi152 ceph-mon[36291]: pgmap v376: 97 pgs: 97 active+clean; 78 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.2 MiB/s wr, 212 op/s 2024-03-16T12:02:50.804 INFO:tasks.workunit.client.0.smithi029.stdout: 1 1153253 9.29 MB/sec execute 515 sec latency 20.030 ms 2024-03-16T12:02:51.804 INFO:tasks.workunit.client.0.smithi029.stdout: 1 1154544 9.29 MB/sec execute 516 sec latency 65.787 ms 2024-03-16T12:02:52.543 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:02:52 smithi154 ceph-mon[36320]: pgmap v377: 97 pgs: 97 active+clean; 82 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 3.3 MiB/s wr, 282 op/s 2024-03-16T12:02:52.586 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:02:52 smithi029 ceph-mon[31975]: pgmap v377: 97 pgs: 97 active+clean; 82 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 3.3 MiB/s wr, 282 op/s 2024-03-16T12:02:52.625 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:02:52 smithi152 ceph-mon[36291]: pgmap v377: 97 pgs: 97 active+clean; 82 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 3.3 MiB/s wr, 282 op/s 2024-03-16T12:02:52.805 INFO:tasks.workunit.client.0.smithi029.stdout: 1 1155993 9.29 MB/sec execute 517 sec latency 18.759 ms 2024-03-16T12:02:53.805 INFO:tasks.workunit.client.0.smithi029.stdout: 1 1158015 9.29 MB/sec execute 518 sec latency 18.159 ms 2024-03-16T12:02:54.543 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:02:54 smithi154 ceph-mon[36320]: pgmap v378: 97 pgs: 97 active+clean; 81 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 2.9 MiB/s wr, 251 op/s 2024-03-16T12:02:54.585 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:02:54 smithi029 ceph-mon[31975]: pgmap v378: 97 pgs: 97 active+clean; 81 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 2.9 MiB/s wr, 251 op/s 2024-03-16T12:02:54.625 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:02:54 smithi152 ceph-mon[36291]: pgmap v378: 97 pgs: 97 active+clean; 81 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 2.9 MiB/s wr, 251 op/s 2024-03-16T12:02:54.805 INFO:tasks.workunit.client.0.smithi029.stdout: 1 1159113 9.28 MB/sec execute 519 sec latency 31.885 ms 2024-03-16T12:02:55.670 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:55.671+0000 7f4cc3cac640 1 -- 172.21.15.29:0/1413119758 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4cbc1012f0 msgr2=0x7f4cbc10d4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:02:55.671 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:55.671+0000 7f4cc3cac640 1 --2- 172.21.15.29:0/1413119758 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4cbc1012f0 0x7f4cbc10d4f0 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f4cac009ca0 tx=0x7f4cac02c0f0 comp rx=0 tx=0).stop 2024-03-16T12:02:55.671 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:55.671+0000 7f4cc3cac640 1 -- 172.21.15.29:0/1413119758 shutdown_connections 2024-03-16T12:02:55.671 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:55.671+0000 7f4cc3cac640 1 --2- 172.21.15.29:0/1413119758 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4cbc1012f0 0x7f4cbc10d4f0 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:02:55.671 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:55.671+0000 7f4cc3cac640 1 --2- 172.21.15.29:0/1413119758 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4cbc100930 0x7f4cbc100db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:02:55.671 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:55.671+0000 7f4cc3cac640 1 --2- 172.21.15.29:0/1413119758 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4cbc105670 0x7f4cbc105a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:02:55.671 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:55.671+0000 7f4cc3cac640 1 -- 172.21.15.29:0/1413119758 >> 172.21.15.29:0/1413119758 conn(0x7f4cbc077530 msgr2=0x7f4cbc077940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:02:55.671 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:55.671+0000 7f4cc3cac640 1 -- 172.21.15.29:0/1413119758 shutdown_connections 2024-03-16T12:02:55.671 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:55.671+0000 7f4cc3cac640 1 -- 172.21.15.29:0/1413119758 wait complete. 2024-03-16T12:02:55.671 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:55.672+0000 7f4cc3cac640 1 Processor -- start 2024-03-16T12:02:55.671 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:55.672+0000 7f4cc3cac640 1 -- start start 2024-03-16T12:02:55.671 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:55.672+0000 7f4cc3cac640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4cbc100930 0x7f4cbc197f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:02:55.671 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:55.672+0000 7f4cc3cac640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4cbc1012f0 0x7f4cbc1984b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:02:55.672 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:55.672+0000 7f4cc3cac640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4cbc105670 0x7f4cbc19c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:02:55.672 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:55.672+0000 7f4cc3cac640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f4cbc111b50 con 0x7f4cbc105670 2024-03-16T12:02:55.672 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:55.672+0000 7f4cc3cac640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f4cbc1119d0 con 0x7f4cbc1012f0 2024-03-16T12:02:55.672 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:55.672+0000 7f4cc3cac640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f4cbc111cd0 con 0x7f4cbc100930 2024-03-16T12:02:55.672 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:55.672+0000 7f4cc1a48640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4cbc100930 0x7f4cbc197f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:02:55.672 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:55.672+0000 7f4cc1247640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4cbc1012f0 0x7f4cbc1984b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:02:55.673 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:55.672+0000 7f4cc1a48640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4cbc100930 0x7f4cbc197f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.29:50130/0 (socket says 172.21.15.29:50130) 2024-03-16T12:02:55.673 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:55.672+0000 7f4cc1a48640 1 -- 172.21.15.29:0/999187369 learned_addr learned my addr 172.21.15.29:0/999187369 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T12:02:55.673 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:55.672+0000 7f4cc2249640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4cbc105670 0x7f4cbc19c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:02:55.673 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:55.673+0000 7f4cc1247640 1 -- 172.21.15.29:0/999187369 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4cbc100930 msgr2=0x7f4cbc197f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:02:55.673 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:55.673+0000 7f4cc1247640 1 --2- 172.21.15.29:0/999187369 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4cbc100930 0x7f4cbc197f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:02:55.673 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:55.673+0000 7f4cc1247640 1 -- 172.21.15.29:0/999187369 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4cbc105670 msgr2=0x7f4cbc19c7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:02:55.673 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:55.673+0000 7f4cc1247640 1 --2- 172.21.15.29:0/999187369 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4cbc105670 0x7f4cbc19c7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:02:55.673 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:55.673+0000 7f4cc1247640 1 -- 172.21.15.29:0/999187369 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4cbc19cf10 con 0x7f4cbc1012f0 2024-03-16T12:02:55.673 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:55.673+0000 7f4cc2249640 1 --2- 172.21.15.29:0/999187369 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4cbc105670 0x7f4cbc19c7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:02:55.673 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:55.673+0000 7f4cc1247640 1 --2- 172.21.15.29:0/999187369 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4cbc1012f0 0x7f4cbc1984b0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f4cb000e9f0 tx=0x7f4cb000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:02:55.673 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:55.673+0000 7f4cbaffd640 1 -- 172.21.15.29:0/999187369 <== mon.2 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cb000ccc0 con 0x7f4cbc1012f0 2024-03-16T12:02:55.673 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:55.673+0000 7f4cc3cac640 1 -- 172.21.15.29:0/999187369 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4cbc19d200 con 0x7f4cbc1012f0 2024-03-16T12:02:55.673 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:55.673+0000 7f4cbaffd640 1 -- 172.21.15.29:0/999187369 <== mon.2 v2:172.21.15.152:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4cb0010910 con 0x7f4cbc1012f0 2024-03-16T12:02:55.673 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:55.673+0000 7f4cbaffd640 1 -- 172.21.15.29:0/999187369 <== mon.2 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cb0018960 con 0x7f4cbc1012f0 2024-03-16T12:02:55.674 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:55.673+0000 7f4cc3cac640 1 -- 172.21.15.29:0/999187369 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4cbc1a4a90 con 0x7f4cbc1012f0 2024-03-16T12:02:55.679 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:55.675+0000 7f4cbaffd640 1 -- 172.21.15.29:0/999187369 <== mon.2 v2:172.21.15.152:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7f4cb0010ac0 con 0x7f4cbc1012f0 2024-03-16T12:02:55.679 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:55.676+0000 7f4cbaffd640 1 --2- 172.21.15.29:0/999187369 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f4ca40a2a10 0x7f4ca40a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:02:55.679 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:55.676+0000 7f4cbaffd640 1 -- 172.21.15.29:0/999187369 <== mon.2 v2:172.21.15.152:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 8556+0+0 (secure 0 0 0) 0x7f4cb00cf2a0 con 0x7f4cbc1012f0 2024-03-16T12:02:55.679 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:55.676+0000 7f4cc1a48640 1 --2- 172.21.15.29:0/999187369 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f4ca40a2a10 0x7f4ca40a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:02:55.680 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:55.676+0000 7f4cc6972740 1 Processor -- start 2024-03-16T12:02:55.680 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:55.676+0000 7f4cc6972740 1 -- start start 2024-03-16T12:02:55.680 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:55.676+0000 7f4cc1a48640 1 --2- 172.21.15.29:0/999187369 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f4ca40a2a10 0x7f4ca40a4ed0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f4cb4009c50 tx=0x7f4cb4005f90 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:02:55.680 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:55.677+0000 7f4cc6972740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c742464130 0x55c742464510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:02:55.680 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:55.677+0000 7f4cc6972740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c742464a50 0x55c74246d2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:02:55.680 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:55.677+0000 7f4cc6972740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c74246d820 0x55c74246fc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:02:55.680 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:55.677+0000 7f4cc6972740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55c7423c8a40 con 0x55c742464a50 2024-03-16T12:02:55.680 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:55.677+0000 7f4cc6972740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x55c742371920 con 0x55c74246d820 2024-03-16T12:02:55.680 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:55.677+0000 7f4cc6972740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55c7423c4ae0 con 0x55c742464130 2024-03-16T12:02:55.680 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:55.677+0000 7f4cc1247640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c742464a50 0x55c74246d2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:02:55.680 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:55.677+0000 7f4cc1247640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c742464a50 0x55c74246d2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59856/0 (socket says 172.21.15.29:59856) 2024-03-16T12:02:55.680 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:55.677+0000 7f4cc1247640 1 -- 172.21.15.29:0/4261299817 learned_addr learned my addr 172.21.15.29:0/4261299817 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T12:02:55.680 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:55.677+0000 7f4cc1a48640 1 --2- 172.21.15.29:0/4261299817 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c742464130 0x55c742464510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:02:55.681 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:55.677+0000 7f4cc2249640 1 --2- 172.21.15.29:0/4261299817 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c74246d820 0x55c74246fc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:02:55.681 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:55.677+0000 7f4cc1247640 1 -- 172.21.15.29:0/4261299817 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c742464130 msgr2=0x55c742464510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:02:55.681 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:55.677+0000 7f4cc1247640 1 --2- 172.21.15.29:0/4261299817 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c742464130 0x55c742464510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:02:55.681 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:55.677+0000 7f4cc1247640 1 -- 172.21.15.29:0/4261299817 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c74246d820 msgr2=0x55c74246fc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:02:55.681 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:55.677+0000 7f4cc1247640 1 --2- 172.21.15.29:0/4261299817 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c74246d820 0x55c74246fc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:02:55.681 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:55.677+0000 7f4cc1247640 1 -- 172.21.15.29:0/4261299817 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c742389460 con 0x55c742464a50 2024-03-16T12:02:55.681 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:55.677+0000 7f4cc1247640 1 --2- 172.21.15.29:0/4261299817 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c742464a50 0x55c74246d2e0 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f4cb00a3af0 tx=0x7f4cb009b9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a01436021d7d53f8 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:02:55.681 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:55.677+0000 7f4ca3fff640 1 -- 172.21.15.29:0/4261299817 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cb00ccbf0 con 0x55c742464a50 2024-03-16T12:02:55.681 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:55.678+0000 7f4ca3fff640 1 -- 172.21.15.29:0/4261299817 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4cb0097fa0 con 0x55c742464a50 2024-03-16T12:02:55.682 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:55.678+0000 7f4ca3fff640 1 -- 172.21.15.29:0/4261299817 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cb00862d0 con 0x55c742464a50 2024-03-16T12:02:55.682 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:55.678+0000 7f4cc6972740 1 -- 172.21.15.29:0/4261299817 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c742464a50 msgr2=0x55c74246d2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:02:55.682 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:55.678+0000 7f4cc6972740 1 --2- 172.21.15.29:0/4261299817 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c742464a50 0x55c74246d2e0 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f4cb00a3af0 tx=0x7f4cb009b9f0 comp rx=0 tx=0).stop 2024-03-16T12:02:55.682 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:55.678+0000 7f4cc6972740 1 -- 172.21.15.29:0/4261299817 shutdown_connections 2024-03-16T12:02:55.682 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:55.678+0000 7f4cc6972740 1 --2- 172.21.15.29:0/4261299817 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c74246d820 0x55c74246fc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:02:55.682 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:55.678+0000 7f4cc6972740 1 --2- 172.21.15.29:0/4261299817 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c742464a50 0x55c74246d2e0 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:02:55.682 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:55.678+0000 7f4cc6972740 1 --2- 172.21.15.29:0/4261299817 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c742464130 0x55c742464510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:02:55.682 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:55.678+0000 7f4cc6972740 1 -- 172.21.15.29:0/4261299817 >> 172.21.15.29:0/4261299817 conn(0x55c74245ea50 msgr2=0x55c74245ee30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:02:55.682 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:55.678+0000 7f4cc6972740 1 -- 172.21.15.29:0/4261299817 shutdown_connections 2024-03-16T12:02:55.682 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:55.678+0000 7f4cc6972740 1 -- 172.21.15.29:0/4261299817 wait complete. 2024-03-16T12:02:55.682 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:55.679+0000 7f4cc6972740 1 -- start start 2024-03-16T12:02:55.682 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:55.679+0000 7f4cc6972740 1 Processor -- start 2024-03-16T12:02:55.682 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:55.679+0000 7f4cc6972740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c742464130 0x55c74246a900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:02:55.683 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:55.679+0000 7f4cc6972740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c742464a50 0x55c74246ae40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:02:55.683 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:55.679+0000 7f4cc6972740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c74246d820 0x55c74246b380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:02:55.683 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:55.679+0000 7f4cc6972740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55c7424703f0 con 0x55c742464130 2024-03-16T12:02:55.683 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:55.679+0000 7f4cc6972740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x55c742384d70 con 0x55c74246d820 2024-03-16T12:02:55.683 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:55.679+0000 7f4cc6972740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55c742470570 con 0x55c742464a50 2024-03-16T12:02:55.683 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:55.679+0000 7f4cc6972740 10 client.0 _ll_register_callbacks cb 0x55c7420dca00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T12:02:55.683 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:55.679+0000 7f4cc1a48640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c742464130 0x55c74246a900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:02:55.683 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:55.679+0000 7f4cc1a48640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c742464130 0x55c74246a900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59868/0 (socket says 172.21.15.29:59868) 2024-03-16T12:02:55.683 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:55.679+0000 7f4cc1a48640 1 -- 172.21.15.29:0/271994411 learned_addr learned my addr 172.21.15.29:0/271994411 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T12:02:55.683 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:55.679+0000 7f4cc1247640 1 --2- 172.21.15.29:0/271994411 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c742464a50 0x55c74246ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:02:55.683 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:55.679+0000 7f4cc2249640 1 --2- 172.21.15.29:0/271994411 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c74246d820 0x55c74246b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:02:55.683 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:55.680+0000 7f4cc1a48640 1 -- 172.21.15.29:0/271994411 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c742464a50 msgr2=0x55c74246ae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:02:55.683 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:55.680+0000 7f4cc1a48640 1 --2- 172.21.15.29:0/271994411 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c742464a50 0x55c74246ae40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:02:55.684 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:55.680+0000 7f4cc1a48640 1 -- 172.21.15.29:0/271994411 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c74246d820 msgr2=0x55c74246b380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:02:55.684 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:55.680+0000 7f4cc1a48640 1 --2- 172.21.15.29:0/271994411 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c74246d820 0x55c74246b380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:02:55.684 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:55.680+0000 7f4cc1a48640 1 -- 172.21.15.29:0/271994411 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c7420dd210 con 0x55c742464130 2024-03-16T12:02:55.684 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:55.680+0000 7f4cc1a48640 1 --2- 172.21.15.29:0/271994411 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c742464130 0x55c74246a900 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f4cb401b9e0 tx=0x7f4cb4013040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec1520b50ddbd79e server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:02:55.684 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:55.680+0000 7f4ca17fa640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-16T12:02:55.684 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:55.680+0000 7f4ca17fa640 1 -- 172.21.15.29:0/271994411 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cb402c060 con 0x55c742464130 2024-03-16T12:02:55.684 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:55.680+0000 7f4cc6972740 1 -- 172.21.15.29:0/271994411 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4cb0097fa0 con 0x55c742464130 2024-03-16T12:02:55.684 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:55.680+0000 7f4ca17fa640 1 -- 172.21.15.29:0/271994411 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 8556+0+0 (secure 0 0 0) 0x7f4cb401ad20 con 0x55c742464130 2024-03-16T12:02:55.684 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:55.680+0000 7f4ca17fa640 1 -- 172.21.15.29:0/271994411 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f4cb4030430 con 0x55c742464130 2024-03-16T12:02:55.684 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:55.680+0000 7f4cc6972740 10 client.16398 fetch_fsmap learned FSMap version 28 2024-03-16T12:02:55.684 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:55.680+0000 7f4cc6972740 1 -- 172.21.15.29:0/271994411 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55c7421a9620 con 0x55c742464130 2024-03-16T12:02:55.684 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:55.681+0000 7f4ca17fa640 1 -- 172.21.15.29:0/271994411 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4cb0097fa0 con 0x55c742464130 2024-03-16T12:02:55.684 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:55.681+0000 7f4ca17fa640 1 -- 172.21.15.29:0/271994411 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 28) ==== 8338+0+0 (secure 0 0 0) 0x7f4cb4026070 con 0x55c742464130 2024-03-16T12:02:55.684 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:55.681+0000 7f4cc6972740 10 client.16398 fetch_fsmap finished waiting for FSMap version 28 2024-03-16T12:02:55.684 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:55.681+0000 7f4cc6972740 10 client.16398 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T12:02:55.685 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:55.681+0000 7f4cc6972740 20 client.16398 populate_metadata read hostname 'smithi029' 2024-03-16T12:02:55.685 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:55.681+0000 7f4cc6972740 1 --2- 172.21.15.29:0/271994411 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55c74247dab0 0x55c74247fea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:02:55.685 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:55.681+0000 7f4cc6972740 4 client.16398 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T12:02:55.685 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:55.681+0000 7f4cc6972740 1 -- 172.21.15.29:0/271994411 --> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c7423d5f80 con 0x55c74247dab0 2024-03-16T12:02:55.685 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:55.681+0000 7f4cc1247640 1 --2- 172.21.15.29:0/271994411 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55c74247dab0 0x55c74247fea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:02:55.685 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:55.682+0000 7f4ca17fa640 10 client.16398 ms_handle_connect on v2:172.21.15.152:6838/1548731833 2024-03-16T12:02:55.685 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:55.682+0000 7f4cc1247640 1 --2- 172.21.15.29:0/271994411 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55c74247dab0 0x55c74247fea0 crc :-1 s=READY pgs=146 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=14fc4f0fad6461ce server_cookie=e241c40b8e58480b in_seq=0 out_seq=0 2024-03-16T12:02:55.686 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:55.683+0000 7f4ca17fa640 1 -- 172.21.15.29:0/271994411 <== mds.0 v2:172.21.15.152:6838/1548731833 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55c7423d5f80 con 0x55c74247dab0 2024-03-16T12:02:55.687 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:55.683+0000 7f4ca17fa640 10 client.16398 handle_command_reply: tid=0 2024-03-16T12:02:55.687 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:55.683+0000 7f4cc6972740 1 client.16398 shutdown 2024-03-16T12:02:55.687 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:55.683+0000 7f4cc6972740 1 -- 172.21.15.29:0/271994411 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c742464130 msgr2=0x55c74246a900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:02:55.687 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:55.683+0000 7f4cc6972740 1 --2- 172.21.15.29:0/271994411 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c742464130 0x55c74246a900 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f4cb401b9e0 tx=0x7f4cb4013040 comp rx=0 tx=0).stop 2024-03-16T12:02:55.687 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:55.683+0000 7f4cc6972740 1 -- 172.21.15.29:0/271994411 shutdown_connections 2024-03-16T12:02:55.687 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:55.683+0000 7f4cc6972740 1 --2- 172.21.15.29:0/271994411 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55c74247dab0 0x55c74247fea0 crc :-1 s=READY pgs=146 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:02:55.687 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:55.683+0000 7f4cc6972740 1 --2- 172.21.15.29:0/271994411 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c74246d820 0x55c74246b380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:02:55.687 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:55.683+0000 7f4cc6972740 1 --2- 172.21.15.29:0/271994411 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c742464a50 0x55c74246ae40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:02:55.687 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:55.683+0000 7f4cc6972740 1 --2- 172.21.15.29:0/271994411 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c742464130 0x55c74246a900 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:02:55.687 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:55.683+0000 7f4cc6972740 1 -- 172.21.15.29:0/271994411 >> 172.21.15.29:0/271994411 conn(0x55c74245ea50 msgr2=0x55c7424607a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:02:55.687 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:55.684+0000 7f4ca17fa640 0 client.16398 ms_handle_reset on v2:172.21.15.152:6838/1548731833 2024-03-16T12:02:55.687 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:55.684+0000 7f4cc6972740 1 -- 172.21.15.29:0/271994411 shutdown_connections 2024-03-16T12:02:55.687 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:55.684+0000 7f4cc6972740 1 -- 172.21.15.29:0/271994411 wait complete. 2024-03-16T12:02:55.687 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:55.684+0000 7f4cc6972740 20 client.16398 trim_cache size 0 max 16384 2024-03-16T12:02:55.700 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:55.701+0000 7f4cc6972740 1 Processor -- start 2024-03-16T12:02:55.700 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:55.701+0000 7f4cc6972740 1 -- start start 2024-03-16T12:02:55.700 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:55.701+0000 7f4cc6972740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c742464130 0x55c742479190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:02:55.700 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:55.701+0000 7f4cc6972740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c742464a50 0x55c74246c730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:02:55.700 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:55.701+0000 7f4cc6972740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c74246d820 0x55c742485760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:02:55.700 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:55.702+0000 7f4cc6972740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f4cbc1afd30 con 0x55c74246d820 2024-03-16T12:02:55.700 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:55.702+0000 7f4cc6972740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f4cbc1ad600 con 0x55c742464130 2024-03-16T12:02:55.700 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:55.702+0000 7f4cc6972740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f4cbc1ab470 con 0x55c742464a50 2024-03-16T12:02:55.700 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:55.702+0000 7f4cc2249640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c74246d820 0x55c742485760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:02:55.700 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:55.702+0000 7f4cc2249640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c74246d820 0x55c742485760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59872/0 (socket says 172.21.15.29:59872) 2024-03-16T12:02:55.701 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:55.702+0000 7f4cc2249640 1 -- 172.21.15.29:0/2153604128 learned_addr learned my addr 172.21.15.29:0/2153604128 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T12:02:55.701 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:55.702+0000 7f4cc1247640 1 --2- 172.21.15.29:0/2153604128 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c742464a50 0x55c74246c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:02:55.701 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:55.702+0000 7f4cc1a48640 1 --2- 172.21.15.29:0/2153604128 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c742464130 0x55c742479190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:02:55.701 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:55.702+0000 7f4cc2249640 1 -- 172.21.15.29:0/2153604128 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c742464a50 msgr2=0x55c74246c730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:02:55.701 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:55.702+0000 7f4cc2249640 1 --2- 172.21.15.29:0/2153604128 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c742464a50 0x55c74246c730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:02:55.701 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:55.702+0000 7f4cc2249640 1 -- 172.21.15.29:0/2153604128 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c742464130 msgr2=0x55c742479190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:02:55.701 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:55.702+0000 7f4cc2249640 1 --2- 172.21.15.29:0/2153604128 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c742464130 0x55c742479190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:02:55.701 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:55.702+0000 7f4cc2249640 1 -- 172.21.15.29:0/2153604128 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c74245a310 con 0x55c74246d820 2024-03-16T12:02:55.701 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:55.702+0000 7f4cc1247640 1 --2- 172.21.15.29:0/2153604128 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c742464a50 0x55c74246c730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:02:55.701 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:55.702+0000 7f4cc2249640 1 --2- 172.21.15.29:0/2153604128 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c74246d820 0x55c742485760 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f4cac009ca0 tx=0x7f4cac005c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51422cc15a094e62 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:02:55.701 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:55.703+0000 7f4ca37fe640 1 -- 172.21.15.29:0/2153604128 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cac039040 con 0x55c74246d820 2024-03-16T12:02:55.701 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:55.703+0000 7f4ca37fe640 1 -- 172.21.15.29:0/2153604128 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4cac032c90 con 0x55c74246d820 2024-03-16T12:02:55.701 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:55.703+0000 7f4ca37fe640 1 -- 172.21.15.29:0/2153604128 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cac0347f0 con 0x55c74246d820 2024-03-16T12:02:55.705 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:55.703+0000 7f4cc6972740 1 -- 172.21.15.29:0/2153604128 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c74246d820 msgr2=0x55c742485760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:02:55.705 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:55.703+0000 7f4cc6972740 1 --2- 172.21.15.29:0/2153604128 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c74246d820 0x55c742485760 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f4cac009ca0 tx=0x7f4cac005c50 comp rx=0 tx=0).stop 2024-03-16T12:02:55.705 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:55.703+0000 7f4cc6972740 1 -- 172.21.15.29:0/2153604128 shutdown_connections 2024-03-16T12:02:55.705 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:55.703+0000 7f4cc6972740 1 --2- 172.21.15.29:0/2153604128 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c74246d820 0x55c742485760 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:02:55.706 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:55.703+0000 7f4cc6972740 1 --2- 172.21.15.29:0/2153604128 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c742464a50 0x55c74246c730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:02:55.706 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:55.703+0000 7f4cc6972740 1 --2- 172.21.15.29:0/2153604128 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c742464130 0x55c742479190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:02:55.706 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:55.703+0000 7f4cc6972740 1 -- 172.21.15.29:0/2153604128 >> 172.21.15.29:0/2153604128 conn(0x55c74245ea50 msgr2=0x55c74246c020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:02:55.706 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:55.703+0000 7f4cc6972740 1 -- 172.21.15.29:0/2153604128 shutdown_connections 2024-03-16T12:02:55.706 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:55.703+0000 7f4cc6972740 1 -- 172.21.15.29:0/2153604128 wait complete. 2024-03-16T12:02:55.706 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:55.704+0000 7f4cc6972740 1 -- start start 2024-03-16T12:02:55.706 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:55.704+0000 7f4cc6972740 1 Processor -- start 2024-03-16T12:02:55.706 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:55.704+0000 7f4cc6972740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c742464130 0x55c742475b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:02:55.706 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:55.704+0000 7f4cc6972740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c742464a50 0x55c742470ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:02:55.706 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:55.704+0000 7f4cc6972740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c74246d820 0x55c7424714d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:02:55.706 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:55.704+0000 7f4cc6972740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f4cbc1abac0 con 0x55c74246d820 2024-03-16T12:02:55.707 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:55.704+0000 7f4cc6972740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f4cbc1acf30 con 0x55c742464a50 2024-03-16T12:02:55.707 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:55.704+0000 7f4cc6972740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f4cbc1ac190 con 0x55c742464130 2024-03-16T12:02:55.707 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:55.704+0000 7f4cc6972740 10 client.0 _ll_register_callbacks cb 0x7f4cb4009a70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T12:02:55.707 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:55.704+0000 7f4cc2249640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c74246d820 0x55c7424714d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:02:55.707 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:55.704+0000 7f4cc2249640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c74246d820 0x55c7424714d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59878/0 (socket says 172.21.15.29:59878) 2024-03-16T12:02:55.707 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:55.704+0000 7f4cc2249640 1 -- 172.21.15.29:0/1564992291 learned_addr learned my addr 172.21.15.29:0/1564992291 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T12:02:55.707 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:55.704+0000 7f4cc1247640 1 --2- 172.21.15.29:0/1564992291 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c742464a50 0x55c742470ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:02:55.707 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:55.704+0000 7f4cc1a48640 1 --2- 172.21.15.29:0/1564992291 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c742464130 0x55c742475b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:02:55.707 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:55.705+0000 7f4cc2249640 1 -- 172.21.15.29:0/1564992291 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c742464130 msgr2=0x55c742475b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:02:55.707 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:55.705+0000 7f4cc2249640 1 --2- 172.21.15.29:0/1564992291 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c742464130 0x55c742475b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:02:55.707 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:55.705+0000 7f4cc2249640 1 -- 172.21.15.29:0/1564992291 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c742464a50 msgr2=0x55c742470ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:02:55.707 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:55.705+0000 7f4cc2249640 1 --2- 172.21.15.29:0/1564992291 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c742464a50 0x55c742470ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:02:55.707 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:55.705+0000 7f4cc2249640 1 -- 172.21.15.29:0/1564992291 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c742471d00 con 0x55c74246d820 2024-03-16T12:02:55.707 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:55.705+0000 7f4cc2249640 1 --2- 172.21.15.29:0/1564992291 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c74246d820 0x55c7424714d0 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f4cac02e010 tx=0x7f4cac005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=134c29ef8d7a032d server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:02:55.708 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:55.705+0000 7f4ca17fa640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-16T12:02:55.708 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:55.705+0000 7f4ca17fa640 1 -- 172.21.15.29:0/1564992291 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cac057060 con 0x55c74246d820 2024-03-16T12:02:55.708 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:55.705+0000 7f4ca17fa640 1 -- 172.21.15.29:0/1564992291 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 8556+0+0 (secure 0 0 0) 0x7f4cac032590 con 0x55c74246d820 2024-03-16T12:02:55.708 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:55.705+0000 7f4cc6972740 1 -- 172.21.15.29:0/1564992291 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4cac032c90 con 0x55c74246d820 2024-03-16T12:02:55.708 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:55.705+0000 7f4ca17fa640 1 -- 172.21.15.29:0/1564992291 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f4cac039050 con 0x55c74246d820 2024-03-16T12:02:55.708 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:55.705+0000 7f4cc6972740 10 client.16404 fetch_fsmap learned FSMap version 28 2024-03-16T12:02:55.708 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:55.705+0000 7f4cc6972740 1 -- 172.21.15.29:0/1564992291 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55c7424e3ea0 con 0x55c74246d820 2024-03-16T12:02:55.708 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:55.705+0000 7f4ca17fa640 1 -- 172.21.15.29:0/1564992291 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4cac032c90 con 0x55c74246d820 2024-03-16T12:02:55.708 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:55.705+0000 7f4ca17fa640 1 -- 172.21.15.29:0/1564992291 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 28) ==== 8338+0+0 (secure 0 0 0) 0x7f4cac0414d0 con 0x55c74246d820 2024-03-16T12:02:55.708 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:55.705+0000 7f4cc6972740 10 client.16404 fetch_fsmap finished waiting for FSMap version 28 2024-03-16T12:02:55.708 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:55.705+0000 7f4cc6972740 10 client.16404 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T12:02:55.708 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:55.705+0000 7f4cc6972740 20 client.16404 populate_metadata read hostname 'smithi029' 2024-03-16T12:02:55.708 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:55.705+0000 7f4cc6972740 1 --2- 172.21.15.29:0/1564992291 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55c7424e4560 0x55c7424e6980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:02:55.708 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:55.705+0000 7f4cc6972740 4 client.16404 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-16T12:02:55.708 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:55.705+0000 7f4cc6972740 1 -- 172.21.15.29:0/1564992291 --> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55c7423d02d0 con 0x55c7424e4560 2024-03-16T12:02:55.709 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:55.706+0000 7f4cc1a48640 1 --2- 172.21.15.29:0/1564992291 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55c7424e4560 0x55c7424e6980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:02:55.709 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:55.706+0000 7f4cc1a48640 1 --2- 172.21.15.29:0/1564992291 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55c7424e4560 0x55c7424e6980 crc :-1 s=READY pgs=147 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2f007952d43b4bc server_cookie=6c598955a32a2a99 in_seq=0 out_seq=0 2024-03-16T12:02:55.709 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:55.706+0000 7f4ca17fa640 10 client.16404 ms_handle_connect on v2:172.21.15.152:6838/1548731833 2024-03-16T12:02:55.805 INFO:tasks.workunit.client.0.smithi029.stdout: 1 1161136 9.29 MB/sec execute 520 sec latency 66.638 ms 2024-03-16T12:02:55.910 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:55.912+0000 7f4ca17fa640 1 -- 172.21.15.29:0/1564992291 <== mds.0 v2:172.21.15.152:6838/1548731833 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55c7423d02d0 con 0x55c7424e4560 2024-03-16T12:02:55.911 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:55.912+0000 7f4ca17fa640 10 client.16404 handle_command_reply: tid=0 2024-03-16T12:02:55.911 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:55.912+0000 7f4cc6972740 1 client.16404 shutdown 2024-03-16T12:02:55.911 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:55.913+0000 7f4cc6972740 1 -- 172.21.15.29:0/1564992291 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c74246d820 msgr2=0x55c7424714d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:02:55.911 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:55.913+0000 7f4cc6972740 1 --2- 172.21.15.29:0/1564992291 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c74246d820 0x55c7424714d0 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f4cac02e010 tx=0x7f4cac005f90 comp rx=0 tx=0).stop 2024-03-16T12:02:55.911 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:55.913+0000 7f4cc6972740 1 -- 172.21.15.29:0/1564992291 shutdown_connections 2024-03-16T12:02:55.911 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:55.913+0000 7f4cc6972740 1 --2- 172.21.15.29:0/1564992291 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c74246d820 0x55c7424714d0 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:02:55.911 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:55.913+0000 7f4cc6972740 1 --2- 172.21.15.29:0/1564992291 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55c7424e4560 0x55c7424e6980 crc :-1 s=READY pgs=147 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:02:55.911 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:55.913+0000 7f4ca17fa640 0 client.16404 ms_handle_reset on v2:172.21.15.152:6838/1548731833 2024-03-16T12:02:55.911 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:55.913+0000 7f4cc6972740 1 --2- 172.21.15.29:0/1564992291 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c742464a50 0x55c742470ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:02:55.911 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:55.913+0000 7f4cc6972740 1 --2- 172.21.15.29:0/1564992291 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c742464130 0x55c742475b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:02:55.911 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:55.913+0000 7f4cc6972740 1 -- 172.21.15.29:0/1564992291 >> 172.21.15.29:0/1564992291 conn(0x55c74245ea50 msgr2=0x55c74245dbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:02:55.911 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:55.913+0000 7f4cc6972740 1 -- 172.21.15.29:0/1564992291 shutdown_connections 2024-03-16T12:02:55.911 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:55.913+0000 7f4cc6972740 1 -- 172.21.15.29:0/1564992291 wait complete. 2024-03-16T12:02:55.911 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:55.913+0000 7f4cc6972740 20 client.16404 trim_cache size 0 max 16384 2024-03-16T12:02:55.912 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:{ 2024-03-16T12:02:55.912 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "message": "", 2024-03-16T12:02:55.912 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "return_code": 0 2024-03-16T12:02:55.912 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:} 2024-03-16T12:02:55.913 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:55.914+0000 7f4cc3cac640 1 -- 172.21.15.29:0/999187369 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f4ca40a2a10 msgr2=0x7f4ca40a4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:02:55.913 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:55.914+0000 7f4cc3cac640 1 --2- 172.21.15.29:0/999187369 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f4ca40a2a10 0x7f4ca40a4ed0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f4cb4009c50 tx=0x7f4cb4005f90 comp rx=0 tx=0).stop 2024-03-16T12:02:55.913 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:55.914+0000 7f4cc3cac640 1 -- 172.21.15.29:0/999187369 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4cbc1012f0 msgr2=0x7f4cbc1984b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:02:55.913 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:55.914+0000 7f4cc3cac640 1 --2- 172.21.15.29:0/999187369 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4cbc1012f0 0x7f4cbc1984b0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f4cb000e9f0 tx=0x7f4cb000eeb0 comp rx=0 tx=0).stop 2024-03-16T12:02:55.913 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:55.915+0000 7f4cc3cac640 1 -- 172.21.15.29:0/999187369 shutdown_connections 2024-03-16T12:02:55.914 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:55.915+0000 7f4cc3cac640 1 --2- 172.21.15.29:0/999187369 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f4ca40a2a10 0x7f4ca40a4ed0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:02:55.914 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:55.915+0000 7f4cc3cac640 1 --2- 172.21.15.29:0/999187369 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4cbc105670 0x7f4cbc19c7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:02:55.914 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:55.915+0000 7f4cc3cac640 1 --2- 172.21.15.29:0/999187369 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4cbc1012f0 0x7f4cbc1984b0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:02:55.914 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:55.915+0000 7f4cc3cac640 1 --2- 172.21.15.29:0/999187369 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4cbc100930 0x7f4cbc197f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:02:55.914 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:55.915+0000 7f4cc3cac640 1 -- 172.21.15.29:0/999187369 >> 172.21.15.29:0/999187369 conn(0x7f4cbc077530 msgr2=0x7f4cbc0fe480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:02:55.914 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:55.915+0000 7f4cc3cac640 1 -- 172.21.15.29:0/999187369 shutdown_connections 2024-03-16T12:02:55.914 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:02:55.915+0000 7f4cc3cac640 1 -- 172.21.15.29:0/999187369 wait complete. 2024-03-16T12:02:56.543 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:02:56 smithi154 ceph-mon[36320]: pgmap v379: 97 pgs: 97 active+clean; 81 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.3 MiB/s wr, 215 op/s 2024-03-16T12:02:56.586 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:02:56 smithi029 ceph-mon[31975]: pgmap v379: 97 pgs: 97 active+clean; 81 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.3 MiB/s wr, 215 op/s 2024-03-16T12:02:56.625 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:02:56 smithi152 ceph-mon[36291]: pgmap v379: 97 pgs: 97 active+clean; 81 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.3 MiB/s wr, 215 op/s 2024-03-16T12:02:56.805 INFO:tasks.workunit.client.0.smithi029.stdout: 1 1161783 9.28 MB/sec execute 521 sec latency 130.868 ms 2024-03-16T12:02:57.043 INFO:journalctl@ceph.mds.c.smithi154.stdout:Mar 16 12:02:56 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mds-c[66727]: ignoring --setuser ceph since I am not root 2024-03-16T12:02:57.043 INFO:journalctl@ceph.mds.c.smithi154.stdout:Mar 16 12:02:56 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mds-c[66727]: ignoring --setgroup ceph since I am not root 2024-03-16T12:02:57.043 INFO:journalctl@ceph.mds.c.smithi154.stdout:Mar 16 12:02:56 smithi154 ceph-mds[66731]: -- 172.21.15.154:0/3477985660 <== mon.1 v2:172.21.15.154:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x560623fffa40 con 0x560623fd2800 2024-03-16T12:02:57.043 INFO:journalctl@ceph.mds.c.smithi154.stdout:Mar 16 12:02:56 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mds-c[66727]: starting mds.c at 2024-03-16T12:02:57.543 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:02:57 smithi154 ceph-mon[36320]: daemon mds.c restarted 2024-03-16T12:02:57.543 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:02:57 smithi154 ceph-mon[36320]: osdmap e105: 12 total, 12 up, 12 in 2024-03-16T12:02:57.543 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:02:57 smithi154 ceph-mon[36320]: mds.? [v2:172.21.15.154:6836/3101252460,v1:172.21.15.154:6837/3101252460] up:boot 2024-03-16T12:02:57.543 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:02:57 smithi154 ceph-mon[36320]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-03-16T12:02:57.543 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:02:57 smithi154 ceph-mon[36320]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-16T12:02:57.585 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:02:57 smithi029 ceph-mon[31975]: daemon mds.c restarted 2024-03-16T12:02:57.585 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:02:57 smithi029 ceph-mon[31975]: osdmap e105: 12 total, 12 up, 12 in 2024-03-16T12:02:57.586 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:02:57 smithi029 ceph-mon[31975]: mds.? [v2:172.21.15.154:6836/3101252460,v1:172.21.15.154:6837/3101252460] up:boot 2024-03-16T12:02:57.586 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:02:57 smithi029 ceph-mon[31975]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-03-16T12:02:57.586 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:02:57 smithi029 ceph-mon[31975]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-16T12:02:57.625 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:02:57 smithi152 ceph-mon[36291]: daemon mds.c restarted 2024-03-16T12:02:57.625 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:02:57 smithi152 ceph-mon[36291]: osdmap e105: 12 total, 12 up, 12 in 2024-03-16T12:02:57.625 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:02:57 smithi152 ceph-mon[36291]: mds.? [v2:172.21.15.154:6836/3101252460,v1:172.21.15.154:6837/3101252460] up:boot 2024-03-16T12:02:57.625 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:02:57 smithi152 ceph-mon[36291]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-03-16T12:02:57.625 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:02:57 smithi152 ceph-mon[36291]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-16T12:02:57.805 INFO:tasks.workunit.client.0.smithi029.stdout: 1 1163142 9.27 MB/sec execute 522 sec latency 32.569 ms 2024-03-16T12:02:58.487 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:02:58 smithi029 ceph-mon[31975]: pgmap v380: 97 pgs: 97 active+clean; 69 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 3.4 MiB/s wr, 362 op/s 2024-03-16T12:02:58.542 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:02:58 smithi154 ceph-mon[36320]: pgmap v380: 97 pgs: 97 active+clean; 69 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 3.4 MiB/s wr, 362 op/s 2024-03-16T12:02:58.625 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:02:58 smithi152 ceph-mon[36291]: pgmap v380: 97 pgs: 97 active+clean; 69 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 3.4 MiB/s wr, 362 op/s 2024-03-16T12:02:58.805 INFO:tasks.workunit.client.0.smithi029.stdout: 1 1165092 9.28 MB/sec execute 523 sec latency 22.179 ms 2024-03-16T12:02:59.805 INFO:tasks.workunit.client.0.smithi029.stdout: 1 1165941 9.27 MB/sec execute 524 sec latency 18.053 ms 2024-03-16T12:03:00.542 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:03:00 smithi154 ceph-mon[36320]: pgmap v382: 97 pgs: 97 active+clean; 69 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 2.6 MiB/s wr, 264 op/s 2024-03-16T12:03:00.585 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:03:00 smithi029 ceph-mon[31975]: pgmap v382: 97 pgs: 97 active+clean; 69 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 2.6 MiB/s wr, 264 op/s 2024-03-16T12:03:00.625 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:03:00 smithi152 ceph-mon[36291]: pgmap v382: 97 pgs: 97 active+clean; 69 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 2.6 MiB/s wr, 264 op/s 2024-03-16T12:03:00.805 INFO:tasks.workunit.client.0.smithi029.stdout: 1 1167303 9.26 MB/sec execute 525 sec latency 40.815 ms 2024-03-16T12:03:01.805 INFO:tasks.workunit.client.0.smithi029.stdout: 1 1168574 9.26 MB/sec execute 526 sec latency 79.872 ms 2024-03-16T12:03:02.543 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:03:02 smithi154 ceph-mon[36320]: pgmap v383: 97 pgs: 97 active+clean; 70 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 2.1 MiB/s wr, 287 op/s 2024-03-16T12:03:02.585 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:03:02 smithi029 ceph-mon[31975]: pgmap v383: 97 pgs: 97 active+clean; 70 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 2.1 MiB/s wr, 287 op/s 2024-03-16T12:03:02.625 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:03:02 smithi152 ceph-mon[36291]: pgmap v383: 97 pgs: 97 active+clean; 70 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 2.1 MiB/s wr, 287 op/s 2024-03-16T12:03:02.805 INFO:tasks.workunit.client.0.smithi029.stdout: 1 1169380 9.25 MB/sec execute 527 sec latency 37.133 ms 2024-03-16T12:03:03.805 INFO:tasks.workunit.client.0.smithi029.stdout: 1 1171445 9.25 MB/sec execute 528 sec latency 23.382 ms 2024-03-16T12:03:04.542 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:03:04 smithi154 ceph-mon[36320]: pgmap v384: 97 pgs: 97 active+clean; 70 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 2.2 MiB/s wr, 294 op/s 2024-03-16T12:03:04.585 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:03:04 smithi029 ceph-mon[31975]: pgmap v384: 97 pgs: 97 active+clean; 70 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 2.2 MiB/s wr, 294 op/s 2024-03-16T12:03:04.625 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:03:04 smithi152 ceph-mon[36291]: pgmap v384: 97 pgs: 97 active+clean; 70 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 2.2 MiB/s wr, 294 op/s 2024-03-16T12:03:04.805 INFO:tasks.workunit.client.0.smithi029.stdout: 1 1172220 9.25 MB/sec execute 529 sec latency 82.779 ms 2024-03-16T12:03:05.543 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:03:05 smithi154 ceph-mon[36320]: osdmap e106: 12 total, 12 up, 12 in 2024-03-16T12:03:05.585 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:03:05 smithi029 ceph-mon[31975]: osdmap e106: 12 total, 12 up, 12 in 2024-03-16T12:03:05.625 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:03:05 smithi152 ceph-mon[36291]: osdmap e106: 12 total, 12 up, 12 in 2024-03-16T12:03:05.805 INFO:tasks.workunit.client.0.smithi029.stdout: 1 1172419 9.23 MB/sec execute 530 sec latency 70.768 ms 2024-03-16T12:03:06.543 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:03:06 smithi154 ceph-mon[36320]: pgmap v386: 97 pgs: 97 active+clean; 70 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.2 MiB/s wr, 147 op/s 2024-03-16T12:03:06.585 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:03:06 smithi029 ceph-mon[31975]: pgmap v386: 97 pgs: 97 active+clean; 70 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.2 MiB/s wr, 147 op/s 2024-03-16T12:03:06.625 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:03:06 smithi152 ceph-mon[36291]: pgmap v386: 97 pgs: 97 active+clean; 70 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.2 MiB/s wr, 147 op/s 2024-03-16T12:03:06.805 INFO:tasks.workunit.client.0.smithi029.stdout: 1 1172992 9.21 MB/sec execute 531 sec latency 106.501 ms 2024-03-16T12:03:07.805 INFO:tasks.workunit.client.0.smithi029.stdout: 1 1175008 9.22 MB/sec execute 532 sec latency 21.674 ms 2024-03-16T12:03:08.542 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:03:08 smithi154 ceph-mon[36320]: pgmap v387: 97 pgs: 12 active+clean+snaptrim, 22 active+clean+snaptrim_wait, 63 active+clean; 72 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 2.1 MiB/s wr, 190 op/s 2024-03-16T12:03:08.585 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:03:08 smithi029 ceph-mon[31975]: pgmap v387: 97 pgs: 12 active+clean+snaptrim, 22 active+clean+snaptrim_wait, 63 active+clean; 72 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 2.1 MiB/s wr, 190 op/s 2024-03-16T12:03:08.624 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:03:08 smithi152 ceph-mon[36291]: pgmap v387: 97 pgs: 12 active+clean+snaptrim, 22 active+clean+snaptrim_wait, 63 active+clean; 72 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 2.1 MiB/s wr, 190 op/s 2024-03-16T12:03:08.805 INFO:tasks.workunit.client.0.smithi029.stdout: 1 1176096 9.21 MB/sec execute 533 sec latency 16.969 ms 2024-03-16T12:03:09.009 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:09.010+0000 7fd840590640 1 -- 172.21.15.29:0/1461612360 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd838102150 msgr2=0x7fd83810e660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:03:09.010 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:09.010+0000 7fd840590640 1 --2- 172.21.15.29:0/1461612360 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd838102150 0x7fd83810e660 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7fd830009cd0 tx=0x7fd83002c0f0 comp rx=0 tx=0).stop 2024-03-16T12:03:09.010 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:09.010+0000 7fd840590640 1 -- 172.21.15.29:0/1461612360 shutdown_connections 2024-03-16T12:03:09.010 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:09.010+0000 7fd840590640 1 --2- 172.21.15.29:0/1461612360 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd838102150 0x7fd83810e660 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:03:09.010 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:09.010+0000 7fd840590640 1 --2- 172.21.15.29:0/1461612360 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd838101790 0x7fd838101c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:03:09.010 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:09.010+0000 7fd840590640 1 --2- 172.21.15.29:0/1461612360 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd8381077a0 0x7fd838107b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:03:09.010 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:09.010+0000 7fd840590640 1 -- 172.21.15.29:0/1461612360 >> 172.21.15.29:0/1461612360 conn(0x7fd8380fd2a0 msgr2=0x7fd8380ff6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:03:09.010 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:09.010+0000 7fd840590640 1 -- 172.21.15.29:0/1461612360 shutdown_connections 2024-03-16T12:03:09.010 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:09.010+0000 7fd840590640 1 -- 172.21.15.29:0/1461612360 wait complete. 2024-03-16T12:03:09.010 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:09.011+0000 7fd840590640 1 Processor -- start 2024-03-16T12:03:09.010 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:09.011+0000 7fd840590640 1 -- start start 2024-03-16T12:03:09.010 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:09.011+0000 7fd840590640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd838101790 0x7fd838197f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:03:09.011 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:09.011+0000 7fd840590640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd838102150 0x7fd838198490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:03:09.011 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:09.011+0000 7fd840590640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd8381077a0 0x7fd83819c810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:03:09.011 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:09.011+0000 7fd840590640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fd838111bb0 con 0x7fd838101790 2024-03-16T12:03:09.011 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:09.011+0000 7fd840590640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fd838111a30 con 0x7fd8381077a0 2024-03-16T12:03:09.011 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:09.011+0000 7fd840590640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fd838111d30 con 0x7fd838102150 2024-03-16T12:03:09.011 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:09.011+0000 7fd83e32c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd838101790 0x7fd838197f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:03:09.011 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:09.011+0000 7fd83e32c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd838101790 0x7fd838197f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:34970/0 (socket says 172.21.15.29:34970) 2024-03-16T12:03:09.011 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:09.011+0000 7fd83e32c640 1 -- 172.21.15.29:0/1863936492 learned_addr learned my addr 172.21.15.29:0/1863936492 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T12:03:09.011 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:09.011+0000 7fd83db2b640 1 --2- 172.21.15.29:0/1863936492 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd838102150 0x7fd838198490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:03:09.011 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:09.011+0000 7fd83eb2d640 1 --2- 172.21.15.29:0/1863936492 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd8381077a0 0x7fd83819c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:03:09.011 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:09.012+0000 7fd83e32c640 1 -- 172.21.15.29:0/1863936492 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd838102150 msgr2=0x7fd838198490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:03:09.011 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:09.012+0000 7fd83e32c640 1 --2- 172.21.15.29:0/1863936492 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd838102150 0x7fd838198490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:03:09.011 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:09.012+0000 7fd83e32c640 1 -- 172.21.15.29:0/1863936492 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd8381077a0 msgr2=0x7fd83819c810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:03:09.012 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:09.012+0000 7fd83e32c640 1 --2- 172.21.15.29:0/1863936492 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd8381077a0 0x7fd83819c810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:03:09.012 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:09.012+0000 7fd83e32c640 1 -- 172.21.15.29:0/1863936492 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd83819cf40 con 0x7fd838101790 2024-03-16T12:03:09.012 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:09.012+0000 7fd83e32c640 1 --2- 172.21.15.29:0/1863936492 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd838101790 0x7fd838197f50 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7fd83400e9f0 tx=0x7fd83400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:03:09.012 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:09.012+0000 7fd82f7fe640 1 -- 172.21.15.29:0/1863936492 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd83400cd80 con 0x7fd838101790 2024-03-16T12:03:09.012 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:09.012+0000 7fd82f7fe640 1 -- 172.21.15.29:0/1863936492 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd834010910 con 0x7fd838101790 2024-03-16T12:03:09.012 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:09.012+0000 7fd840590640 1 -- 172.21.15.29:0/1863936492 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd83819d230 con 0x7fd838101790 2024-03-16T12:03:09.012 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:09.012+0000 7fd82f7fe640 1 -- 172.21.15.29:0/1863936492 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd834018990 con 0x7fd838101790 2024-03-16T12:03:09.012 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:09.012+0000 7fd840590640 1 -- 172.21.15.29:0/1863936492 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd8381a4ac0 con 0x7fd838101790 2024-03-16T12:03:09.019 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:09.014+0000 7fd82f7fe640 1 -- 172.21.15.29:0/1863936492 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7fd834010ac0 con 0x7fd838101790 2024-03-16T12:03:09.019 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:09.015+0000 7fd84317e740 1 Processor -- start 2024-03-16T12:03:09.019 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:09.015+0000 7fd82f7fe640 1 --2- 172.21.15.29:0/1863936492 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7fd8240a2a10 0x7fd8240a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:03:09.019 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:09.015+0000 7fd82f7fe640 1 -- 172.21.15.29:0/1863936492 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 8600+0+0 (secure 0 0 0) 0x7fd8340cf170 con 0x7fd838101790 2024-03-16T12:03:09.019 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:09.015+0000 7fd83eb2d640 1 --2- 172.21.15.29:0/1863936492 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7fd8240a2a10 0x7fd8240a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:03:09.019 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:09.015+0000 7fd84317e740 1 -- start start 2024-03-16T12:03:09.019 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:09.015+0000 7fd84317e740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558ddef4f130 0x558ddef4f510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:03:09.020 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:09.015+0000 7fd84317e740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558ddef4fa50 0x558ddef582e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:03:09.020 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:09.015+0000 7fd84317e740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558ddef58820 0x558ddef5ac10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:03:09.020 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:09.015+0000 7fd84317e740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x558ddeeb3a40 con 0x558ddef58820 2024-03-16T12:03:09.020 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:09.015+0000 7fd84317e740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x558ddee5c920 con 0x558ddef4f130 2024-03-16T12:03:09.020 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:09.015+0000 7fd84317e740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x558ddeeafae0 con 0x558ddef4fa50 2024-03-16T12:03:09.020 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:09.015+0000 7fd83e32c640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558ddef4f130 0x558ddef4f510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:03:09.020 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:09.015+0000 7fd83e32c640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558ddef4f130 0x558ddef4f510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.29:52604/0 (socket says 172.21.15.29:52604) 2024-03-16T12:03:09.020 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:09.015+0000 7fd83e32c640 1 -- 172.21.15.29:0/2420550867 learned_addr learned my addr 172.21.15.29:0/2420550867 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T12:03:09.020 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:09.015+0000 7fd83db2b640 1 --2- 172.21.15.29:0/2420550867 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558ddef4fa50 0x558ddef582e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:03:09.020 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:09.015+0000 7fd83eb2d640 1 --2- 172.21.15.29:0/2420550867 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558ddef58820 0x558ddef5ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:03:09.020 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:09.016+0000 7fd83eb2d640 1 --2- 172.21.15.29:0/1863936492 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7fd8240a2a10 0x7fd8240a4ed0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fd830005cc0 tx=0x7fd830032340 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:03:09.020 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:09.016+0000 7fd83e32c640 1 -- 172.21.15.29:0/2420550867 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558ddef4fa50 msgr2=0x558ddef582e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:03:09.020 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:09.016+0000 7fd83e32c640 1 --2- 172.21.15.29:0/2420550867 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558ddef4fa50 0x558ddef582e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:03:09.020 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:09.016+0000 7fd83e32c640 1 -- 172.21.15.29:0/2420550867 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558ddef58820 msgr2=0x558ddef5ac10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:03:09.021 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:09.016+0000 7fd83e32c640 1 --2- 172.21.15.29:0/2420550867 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558ddef58820 0x558ddef5ac10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:03:09.021 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:09.016+0000 7fd83e32c640 1 -- 172.21.15.29:0/2420550867 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558ddee74460 con 0x558ddef4f130 2024-03-16T12:03:09.021 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:09.016+0000 7fd83e32c640 1 --2- 172.21.15.29:0/2420550867 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558ddef4f130 0x558ddef4f510 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fd834098200 tx=0x7fd83409b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=559dc09a792a1a94 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:03:09.021 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:09.016+0000 7fd82cbfe640 1 -- 172.21.15.29:0/2420550867 <== mon.2 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8340860b0 con 0x558ddef4f130 2024-03-16T12:03:09.021 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:09.016+0000 7fd82cbfe640 1 -- 172.21.15.29:0/2420550867 <== mon.2 v2:172.21.15.152:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd8340ccaf0 con 0x558ddef4f130 2024-03-16T12:03:09.021 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:09.016+0000 7fd82cbfe640 1 -- 172.21.15.29:0/2420550867 <== mon.2 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd83409ac80 con 0x558ddef4f130 2024-03-16T12:03:09.021 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:09.017+0000 7fd84317e740 1 -- 172.21.15.29:0/2420550867 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558ddef4f130 msgr2=0x558ddef4f510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:03:09.021 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:09.017+0000 7fd84317e740 1 --2- 172.21.15.29:0/2420550867 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558ddef4f130 0x558ddef4f510 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fd834098200 tx=0x7fd83409b8f0 comp rx=0 tx=0).stop 2024-03-16T12:03:09.021 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:09.017+0000 7fd84317e740 1 -- 172.21.15.29:0/2420550867 shutdown_connections 2024-03-16T12:03:09.021 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:09.017+0000 7fd84317e740 1 --2- 172.21.15.29:0/2420550867 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558ddef58820 0x558ddef5ac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:03:09.021 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:09.017+0000 7fd84317e740 1 --2- 172.21.15.29:0/2420550867 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558ddef4fa50 0x558ddef582e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:03:09.021 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:09.017+0000 7fd84317e740 1 --2- 172.21.15.29:0/2420550867 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558ddef4f130 0x558ddef4f510 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:03:09.021 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:09.017+0000 7fd84317e740 1 -- 172.21.15.29:0/2420550867 >> 172.21.15.29:0/2420550867 conn(0x558ddef49a50 msgr2=0x558ddef49e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:03:09.022 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:09.017+0000 7fd84317e740 1 -- 172.21.15.29:0/2420550867 shutdown_connections 2024-03-16T12:03:09.022 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:09.017+0000 7fd84317e740 1 -- 172.21.15.29:0/2420550867 wait complete. 2024-03-16T12:03:09.022 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:09.017+0000 7fd84317e740 1 -- start start 2024-03-16T12:03:09.022 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:09.017+0000 7fd84317e740 1 Processor -- start 2024-03-16T12:03:09.022 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:09.018+0000 7fd84317e740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558ddef4f130 0x558ddef55950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:03:09.022 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:09.018+0000 7fd84317e740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558ddef4fa50 0x558ddef55e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:03:09.022 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:09.018+0000 7fd84317e740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558ddef58820 0x558ddef563d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:03:09.022 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:09.018+0000 7fd84317e740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x558ddef5b3f0 con 0x558ddef4fa50 2024-03-16T12:03:09.022 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:09.018+0000 7fd84317e740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x558ddee6fd70 con 0x558ddef4f130 2024-03-16T12:03:09.022 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:09.018+0000 7fd84317e740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x558ddef5b570 con 0x558ddef58820 2024-03-16T12:03:09.022 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:09.018+0000 7fd84317e740 10 client.0 _ll_register_callbacks cb 0x558ddebc7a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T12:03:09.022 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:09.018+0000 7fd83db2b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558ddef4fa50 0x558ddef55e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:03:09.022 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:09.018+0000 7fd83db2b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558ddef4fa50 0x558ddef55e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:34976/0 (socket says 172.21.15.29:34976) 2024-03-16T12:03:09.022 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:09.018+0000 7fd83db2b640 1 -- 172.21.15.29:0/2561838052 learned_addr learned my addr 172.21.15.29:0/2561838052 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T12:03:09.023 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:09.018+0000 7fd83eb2d640 1 --2- 172.21.15.29:0/2561838052 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558ddef58820 0x558ddef563d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:03:09.023 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:09.018+0000 7fd83e32c640 1 --2- 172.21.15.29:0/2561838052 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558ddef4f130 0x558ddef55950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:03:09.023 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:09.018+0000 7fd83db2b640 1 -- 172.21.15.29:0/2561838052 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558ddef58820 msgr2=0x558ddef563d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:03:09.023 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:09.018+0000 7fd83db2b640 1 --2- 172.21.15.29:0/2561838052 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558ddef58820 0x558ddef563d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:03:09.023 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:09.018+0000 7fd83db2b640 1 -- 172.21.15.29:0/2561838052 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558ddef4f130 msgr2=0x558ddef55950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:03:09.023 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:09.018+0000 7fd83db2b640 1 --2- 172.21.15.29:0/2561838052 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558ddef4f130 0x558ddef55950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:03:09.023 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:09.018+0000 7fd83db2b640 1 -- 172.21.15.29:0/2561838052 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558ddebc8210 con 0x558ddef4fa50 2024-03-16T12:03:09.023 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:09.018+0000 7fd83db2b640 1 --2- 172.21.15.29:0/2561838052 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558ddef4fa50 0x558ddef55e90 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7fd828012720 tx=0x7fd828012be0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2261e5ca08e370f5 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:03:09.023 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:09.018+0000 7fd821ffb640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-16T12:03:09.023 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:09.018+0000 7fd821ffb640 1 -- 172.21.15.29:0/2561838052 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd82800f920 con 0x558ddef4fa50 2024-03-16T12:03:09.023 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:09.018+0000 7fd821ffb640 1 -- 172.21.15.29:0/2561838052 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 8600+0+0 (secure 0 0 0) 0x7fd82800fac0 con 0x558ddef4fa50 2024-03-16T12:03:09.023 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:09.018+0000 7fd84317e740 1 -- 172.21.15.29:0/2561838052 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd8340ccaf0 con 0x558ddef4fa50 2024-03-16T12:03:09.023 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:09.018+0000 7fd821ffb640 1 -- 172.21.15.29:0/2561838052 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fd82800fd00 con 0x558ddef4fa50 2024-03-16T12:03:09.023 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:09.019+0000 7fd84317e740 10 client.16419 fetch_fsmap learned FSMap version 29 2024-03-16T12:03:09.024 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:09.019+0000 7fd84317e740 1 -- 172.21.15.29:0/2561838052 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x558ddec94620 con 0x558ddef4fa50 2024-03-16T12:03:09.024 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:09.019+0000 7fd821ffb640 1 -- 172.21.15.29:0/2561838052 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd8340ccaf0 con 0x558ddef4fa50 2024-03-16T12:03:09.024 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:09.019+0000 7fd821ffb640 1 -- 172.21.15.29:0/2561838052 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 29) ==== 8338+0+0 (secure 0 0 0) 0x7fd828010550 con 0x558ddef4fa50 2024-03-16T12:03:09.024 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:09.019+0000 7fd84317e740 10 client.16419 fetch_fsmap finished waiting for FSMap version 29 2024-03-16T12:03:09.024 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:09.019+0000 7fd84317e740 10 client.16419 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T12:03:09.024 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:09.019+0000 7fd84317e740 20 client.16419 populate_metadata read hostname 'smithi029' 2024-03-16T12:03:09.024 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:09.019+0000 7fd84317e740 1 --2- 172.21.15.29:0/2561838052 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x558ddef68ab0 0x558ddef6aea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:03:09.024 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:09.019+0000 7fd84317e740 4 client.16419 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T12:03:09.024 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:09.019+0000 7fd84317e740 1 -- 172.21.15.29:0/2561838052 --> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558ddeec0f80 con 0x558ddef68ab0 2024-03-16T12:03:09.024 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:09.019+0000 7fd83e32c640 1 --2- 172.21.15.29:0/2561838052 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x558ddef68ab0 0x558ddef6aea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:03:09.024 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:09.019+0000 7fd821ffb640 10 client.16419 ms_handle_connect on v2:172.21.15.152:6838/1548731833 2024-03-16T12:03:09.024 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:09.019+0000 7fd83e32c640 1 --2- 172.21.15.29:0/2561838052 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x558ddef68ab0 0x558ddef6aea0 crc :-1 s=READY pgs=148 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=adadd71ca9568314 server_cookie=fff32ce56c484384 in_seq=0 out_seq=0 2024-03-16T12:03:09.024 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:09.022+0000 7fd821ffb640 1 -- 172.21.15.29:0/2561838052 <== mds.0 v2:172.21.15.152:6838/1548731833 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x558ddeec0f80 con 0x558ddef68ab0 2024-03-16T12:03:09.024 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:09.022+0000 7fd821ffb640 10 client.16419 handle_command_reply: tid=0 2024-03-16T12:03:09.025 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:09.022+0000 7fd84317e740 1 client.16419 shutdown 2024-03-16T12:03:09.025 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:09.022+0000 7fd84317e740 1 -- 172.21.15.29:0/2561838052 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558ddef4fa50 msgr2=0x558ddef55e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:03:09.025 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:09.022+0000 7fd84317e740 1 --2- 172.21.15.29:0/2561838052 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558ddef4fa50 0x558ddef55e90 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7fd828012720 tx=0x7fd828012be0 comp rx=0 tx=0).stop 2024-03-16T12:03:09.025 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:09.022+0000 7fd84317e740 1 -- 172.21.15.29:0/2561838052 shutdown_connections 2024-03-16T12:03:09.025 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:09.022+0000 7fd84317e740 1 --2- 172.21.15.29:0/2561838052 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558ddef58820 0x558ddef563d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:03:09.025 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:09.022+0000 7fd84317e740 1 --2- 172.21.15.29:0/2561838052 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558ddef4fa50 0x558ddef55e90 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:03:09.025 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:09.022+0000 7fd84317e740 1 --2- 172.21.15.29:0/2561838052 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x558ddef68ab0 0x558ddef6aea0 crc :-1 s=READY pgs=148 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:03:09.025 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:09.022+0000 7fd84317e740 1 --2- 172.21.15.29:0/2561838052 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558ddef4f130 0x558ddef55950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:03:09.025 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:09.022+0000 7fd821ffb640 0 client.16419 ms_handle_reset on v2:172.21.15.152:6838/1548731833 2024-03-16T12:03:09.025 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:09.022+0000 7fd84317e740 1 -- 172.21.15.29:0/2561838052 >> 172.21.15.29:0/2561838052 conn(0x558ddef49a50 msgr2=0x558ddef4b7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:03:09.025 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:09.022+0000 7fd84317e740 1 -- 172.21.15.29:0/2561838052 shutdown_connections 2024-03-16T12:03:09.025 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:09.022+0000 7fd84317e740 1 -- 172.21.15.29:0/2561838052 wait complete. 2024-03-16T12:03:09.025 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:09.022+0000 7fd84317e740 20 client.16419 trim_cache size 0 max 16384 2024-03-16T12:03:09.038 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:09.037+0000 7fd84317e740 1 Processor -- start 2024-03-16T12:03:09.038 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:09.037+0000 7fd84317e740 1 -- start start 2024-03-16T12:03:09.038 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:09.037+0000 7fd84317e740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558ddef4f130 0x558ddef58080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:03:09.039 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:09.037+0000 7fd84317e740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558ddef4fa50 0x558ddef64190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:03:09.039 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:09.037+0000 7fd84317e740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558ddef58820 0x558ddef70790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:03:09.039 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:09.037+0000 7fd84317e740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fd8381afe80 con 0x558ddef58820 2024-03-16T12:03:09.039 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:09.037+0000 7fd84317e740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fd8381ad750 con 0x558ddef4f130 2024-03-16T12:03:09.039 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:09.037+0000 7fd84317e740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fd8381ab5c0 con 0x558ddef4fa50 2024-03-16T12:03:09.039 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:09.037+0000 7fd83eb2d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558ddef58820 0x558ddef70790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:03:09.039 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:09.037+0000 7fd83db2b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558ddef4fa50 0x558ddef64190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:03:09.039 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:09.037+0000 7fd83db2b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558ddef4fa50 0x558ddef64190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.29:44570/0 (socket says 172.21.15.29:44570) 2024-03-16T12:03:09.039 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:09.037+0000 7fd83e32c640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558ddef4f130 0x558ddef58080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:03:09.039 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:09.037+0000 7fd83db2b640 1 -- 172.21.15.29:0/245847756 learned_addr learned my addr 172.21.15.29:0/245847756 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T12:03:09.039 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:09.038+0000 7fd83eb2d640 1 -- 172.21.15.29:0/245847756 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558ddef4fa50 msgr2=0x558ddef64190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:03:09.039 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:09.038+0000 7fd83eb2d640 1 --2- 172.21.15.29:0/245847756 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558ddef4fa50 0x558ddef64190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:03:09.039 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:09.038+0000 7fd83eb2d640 1 -- 172.21.15.29:0/245847756 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558ddef4f130 msgr2=0x558ddef58080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:03:09.039 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:09.038+0000 7fd83eb2d640 1 --2- 172.21.15.29:0/245847756 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558ddef4f130 0x558ddef58080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:03:09.040 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:09.038+0000 7fd83eb2d640 1 -- 172.21.15.29:0/245847756 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558ddef6b210 con 0x558ddef58820 2024-03-16T12:03:09.040 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:09.038+0000 7fd83db2b640 1 --2- 172.21.15.29:0/245847756 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558ddef4fa50 0x558ddef64190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:03:09.040 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:09.038+0000 7fd83eb2d640 1 --2- 172.21.15.29:0/245847756 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558ddef58820 0x558ddef70790 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7fd830037040 tx=0x7fd830040000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc5942bb5ef6af36 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:03:09.040 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:09.038+0000 7fd823fff640 1 -- 172.21.15.29:0/245847756 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd830050070 con 0x558ddef58820 2024-03-16T12:03:09.040 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:09.038+0000 7fd823fff640 1 -- 172.21.15.29:0/245847756 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd830047070 con 0x558ddef58820 2024-03-16T12:03:09.040 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:09.038+0000 7fd823fff640 1 -- 172.21.15.29:0/245847756 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd830039040 con 0x558ddef58820 2024-03-16T12:03:09.040 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:09.039+0000 7fd84317e740 1 -- 172.21.15.29:0/245847756 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558ddef58820 msgr2=0x558ddef70790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:03:09.040 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:09.039+0000 7fd84317e740 1 --2- 172.21.15.29:0/245847756 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558ddef58820 0x558ddef70790 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7fd830037040 tx=0x7fd830040000 comp rx=0 tx=0).stop 2024-03-16T12:03:09.040 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:09.039+0000 7fd84317e740 1 -- 172.21.15.29:0/245847756 shutdown_connections 2024-03-16T12:03:09.040 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:09.039+0000 7fd84317e740 1 --2- 172.21.15.29:0/245847756 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558ddef58820 0x558ddef70790 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:03:09.040 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:09.039+0000 7fd84317e740 1 --2- 172.21.15.29:0/245847756 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558ddef4fa50 0x558ddef64190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:03:09.040 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:09.039+0000 7fd84317e740 1 --2- 172.21.15.29:0/245847756 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558ddef4f130 0x558ddef58080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:03:09.040 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:09.039+0000 7fd84317e740 1 -- 172.21.15.29:0/245847756 >> 172.21.15.29:0/245847756 conn(0x558ddef49a50 msgr2=0x558ddef48bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:03:09.041 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:09.039+0000 7fd84317e740 1 -- 172.21.15.29:0/245847756 shutdown_connections 2024-03-16T12:03:09.041 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:09.039+0000 7fd84317e740 1 -- 172.21.15.29:0/245847756 wait complete. 2024-03-16T12:03:09.041 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:09.039+0000 7fd84317e740 1 -- start start 2024-03-16T12:03:09.041 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:09.039+0000 7fd84317e740 1 Processor -- start 2024-03-16T12:03:09.041 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:09.039+0000 7fd84317e740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558ddef4f130 0x558ddef52d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:03:09.041 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:09.039+0000 7fd84317e740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558ddef4fa50 0x558ddef53250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:03:09.041 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:09.039+0000 7fd84317e740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558ddef58820 0x558ddefcddc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:03:09.041 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:09.039+0000 7fd84317e740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fd8381abc10 con 0x558ddef4fa50 2024-03-16T12:03:09.041 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:09.039+0000 7fd84317e740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fd8381ad080 con 0x558ddef58820 2024-03-16T12:03:09.041 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:09.039+0000 7fd84317e740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fd8381ac2e0 con 0x558ddef4f130 2024-03-16T12:03:09.041 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:09.039+0000 7fd84317e740 10 client.0 _ll_register_callbacks cb 0x7fd828009930 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T12:03:09.041 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:09.039+0000 7fd83db2b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558ddef4fa50 0x558ddef53250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:03:09.041 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:09.039+0000 7fd83e32c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558ddef4f130 0x558ddef52d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:03:09.041 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:09.039+0000 7fd83db2b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558ddef4fa50 0x558ddef53250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:34996/0 (socket says 172.21.15.29:34996) 2024-03-16T12:03:09.042 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:09.039+0000 7fd83db2b640 1 -- 172.21.15.29:0/654620454 learned_addr learned my addr 172.21.15.29:0/654620454 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T12:03:09.042 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:09.039+0000 7fd83eb2d640 1 --2- 172.21.15.29:0/654620454 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558ddef58820 0x558ddefcddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:03:09.042 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:09.040+0000 7fd83db2b640 1 -- 172.21.15.29:0/654620454 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558ddef4f130 msgr2=0x558ddef52d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:03:09.042 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:09.040+0000 7fd83db2b640 1 --2- 172.21.15.29:0/654620454 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558ddef4f130 0x558ddef52d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:03:09.042 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:09.040+0000 7fd83db2b640 1 -- 172.21.15.29:0/654620454 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558ddef58820 msgr2=0x558ddefcddc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:03:09.042 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:09.040+0000 7fd83db2b640 1 --2- 172.21.15.29:0/654620454 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558ddef58820 0x558ddefcddc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:03:09.042 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:09.040+0000 7fd83db2b640 1 -- 172.21.15.29:0/654620454 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558ddefce3d0 con 0x558ddef4fa50 2024-03-16T12:03:09.042 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:09.040+0000 7fd83e32c640 1 --2- 172.21.15.29:0/654620454 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558ddef4f130 0x558ddef52d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:03:09.042 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:09.040+0000 7fd83db2b640 1 --2- 172.21.15.29:0/654620454 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558ddef4fa50 0x558ddef53250 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7fd828000c00 tx=0x7fd82800fbd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1db820c9d02b054 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:03:09.042 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:09.040+0000 7fd821ffb640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-16T12:03:09.042 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:09.040+0000 7fd821ffb640 1 -- 172.21.15.29:0/654620454 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd828028070 con 0x558ddef4fa50 2024-03-16T12:03:09.042 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:09.040+0000 7fd821ffb640 1 -- 172.21.15.29:0/654620454 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 8600+0+0 (secure 0 0 0) 0x7fd828010640 con 0x558ddef4fa50 2024-03-16T12:03:09.042 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:09.040+0000 7fd84317e740 1 -- 172.21.15.29:0/654620454 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd830047070 con 0x558ddef4fa50 2024-03-16T12:03:09.042 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:09.040+0000 7fd821ffb640 1 -- 172.21.15.29:0/654620454 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fd828031050 con 0x558ddef4fa50 2024-03-16T12:03:09.043 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:09.040+0000 7fd84317e740 10 client.16425 fetch_fsmap learned FSMap version 29 2024-03-16T12:03:09.043 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:09.040+0000 7fd84317e740 1 -- 172.21.15.29:0/654620454 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x558ddefcf6e0 con 0x558ddef4fa50 2024-03-16T12:03:09.043 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:09.040+0000 7fd821ffb640 1 -- 172.21.15.29:0/654620454 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd830047070 con 0x558ddef4fa50 2024-03-16T12:03:09.043 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:09.040+0000 7fd821ffb640 1 -- 172.21.15.29:0/654620454 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 29) ==== 8338+0+0 (secure 0 0 0) 0x7fd82802b020 con 0x558ddef4fa50 2024-03-16T12:03:09.043 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:09.040+0000 7fd84317e740 10 client.16425 fetch_fsmap finished waiting for FSMap version 29 2024-03-16T12:03:09.043 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:09.040+0000 7fd84317e740 10 client.16425 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T12:03:09.043 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:09.040+0000 7fd84317e740 20 client.16425 populate_metadata read hostname 'smithi029' 2024-03-16T12:03:09.043 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:09.041+0000 7fd84317e740 1 --2- 172.21.15.29:0/654620454 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x558ddefcfe00 0x558ddefd2220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:03:09.043 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:09.041+0000 7fd84317e740 4 client.16425 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-16T12:03:09.043 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:09.041+0000 7fd84317e740 1 -- 172.21.15.29:0/654620454 --> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] -- command(tid 0: {"prefix": "flush journal"}) -- 0x558ddeebb2d0 con 0x558ddefcfe00 2024-03-16T12:03:09.043 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:09.041+0000 7fd83e32c640 1 --2- 172.21.15.29:0/654620454 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x558ddefcfe00 0x558ddefd2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:03:09.043 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:09.041+0000 7fd83e32c640 1 --2- 172.21.15.29:0/654620454 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x558ddefcfe00 0x558ddefd2220 crc :-1 s=READY pgs=149 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=33d289b1b7529be9 server_cookie=d4b1a028e050b183 in_seq=0 out_seq=0 2024-03-16T12:03:09.043 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:09.041+0000 7fd821ffb640 10 client.16425 ms_handle_connect on v2:172.21.15.152:6838/1548731833 2024-03-16T12:03:09.260 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:09.263+0000 7fd821ffb640 1 -- 172.21.15.29:0/654620454 <== mds.0 v2:172.21.15.152:6838/1548731833 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x558ddeebb2d0 con 0x558ddefcfe00 2024-03-16T12:03:09.260 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:09.263+0000 7fd821ffb640 10 client.16425 handle_command_reply: tid=0 2024-03-16T12:03:09.261 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:09.263+0000 7fd84317e740 1 client.16425 shutdown 2024-03-16T12:03:09.261 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:09.263+0000 7fd84317e740 1 -- 172.21.15.29:0/654620454 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558ddef4fa50 msgr2=0x558ddef53250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:03:09.261 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:09.263+0000 7fd84317e740 1 --2- 172.21.15.29:0/654620454 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558ddef4fa50 0x558ddef53250 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7fd828000c00 tx=0x7fd82800fbd0 comp rx=0 tx=0).stop 2024-03-16T12:03:09.261 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:09.263+0000 7fd84317e740 1 -- 172.21.15.29:0/654620454 shutdown_connections 2024-03-16T12:03:09.261 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:09.263+0000 7fd84317e740 1 --2- 172.21.15.29:0/654620454 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x558ddefcfe00 0x558ddefd2220 crc :-1 s=READY pgs=149 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:03:09.261 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:09.263+0000 7fd821ffb640 0 client.16425 ms_handle_reset on v2:172.21.15.152:6838/1548731833 2024-03-16T12:03:09.261 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:09.263+0000 7fd84317e740 1 --2- 172.21.15.29:0/654620454 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558ddef58820 0x558ddefcddc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:03:09.261 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:09.263+0000 7fd84317e740 1 --2- 172.21.15.29:0/654620454 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558ddef4fa50 0x558ddef53250 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:03:09.261 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:09.263+0000 7fd84317e740 1 --2- 172.21.15.29:0/654620454 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558ddef4f130 0x558ddef52d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:03:09.261 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:09.263+0000 7fd84317e740 1 -- 172.21.15.29:0/654620454 >> 172.21.15.29:0/654620454 conn(0x558ddef49a50 msgr2=0x558ddef55740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:03:09.261 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:09.263+0000 7fd84317e740 1 -- 172.21.15.29:0/654620454 shutdown_connections 2024-03-16T12:03:09.261 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:09.263+0000 7fd84317e740 1 -- 172.21.15.29:0/654620454 wait complete. 2024-03-16T12:03:09.261 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:09.263+0000 7fd84317e740 20 client.16425 trim_cache size 0 max 16384 2024-03-16T12:03:09.262 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:{ 2024-03-16T12:03:09.262 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "message": "", 2024-03-16T12:03:09.262 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "return_code": 0 2024-03-16T12:03:09.262 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:} 2024-03-16T12:03:09.263 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:09.264+0000 7fd840590640 1 -- 172.21.15.29:0/1863936492 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7fd8240a2a10 msgr2=0x7fd8240a4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:03:09.263 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:09.264+0000 7fd840590640 1 --2- 172.21.15.29:0/1863936492 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7fd8240a2a10 0x7fd8240a4ed0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fd830005cc0 tx=0x7fd830032340 comp rx=0 tx=0).stop 2024-03-16T12:03:09.263 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:09.264+0000 7fd840590640 1 -- 172.21.15.29:0/1863936492 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd838101790 msgr2=0x7fd838197f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:03:09.263 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:09.264+0000 7fd840590640 1 --2- 172.21.15.29:0/1863936492 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd838101790 0x7fd838197f50 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7fd83400e9f0 tx=0x7fd83400eeb0 comp rx=0 tx=0).stop 2024-03-16T12:03:09.263 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:09.264+0000 7fd840590640 1 -- 172.21.15.29:0/1863936492 shutdown_connections 2024-03-16T12:03:09.263 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:09.264+0000 7fd840590640 1 --2- 172.21.15.29:0/1863936492 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7fd8240a2a10 0x7fd8240a4ed0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:03:09.263 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:09.264+0000 7fd840590640 1 --2- 172.21.15.29:0/1863936492 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd8381077a0 0x7fd83819c810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:03:09.263 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:09.264+0000 7fd840590640 1 --2- 172.21.15.29:0/1863936492 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd838102150 0x7fd838198490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:03:09.263 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:09.264+0000 7fd840590640 1 --2- 172.21.15.29:0/1863936492 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd838101790 0x7fd838197f50 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:03:09.263 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:09.264+0000 7fd840590640 1 -- 172.21.15.29:0/1863936492 >> 172.21.15.29:0/1863936492 conn(0x7fd8380fd2a0 msgr2=0x7fd8380fd780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:03:09.264 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:09.264+0000 7fd840590640 1 -- 172.21.15.29:0/1863936492 shutdown_connections 2024-03-16T12:03:09.264 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:09.264+0000 7fd840590640 1 -- 172.21.15.29:0/1863936492 wait complete. 2024-03-16T12:03:09.805 INFO:tasks.workunit.client.0.smithi029.stdout: 1 1177219 9.21 MB/sec execute 534 sec latency 125.529 ms 2024-03-16T12:03:10.542 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:03:10 smithi154 ceph-mon[36320]: pgmap v388: 97 pgs: 12 active+clean+snaptrim, 22 active+clean+snaptrim_wait, 63 active+clean; 72 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 2.0 MiB/s wr, 189 op/s 2024-03-16T12:03:10.585 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:03:10 smithi029 ceph-mon[31975]: pgmap v388: 97 pgs: 12 active+clean+snaptrim, 22 active+clean+snaptrim_wait, 63 active+clean; 72 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 2.0 MiB/s wr, 189 op/s 2024-03-16T12:03:10.624 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:03:10 smithi152 ceph-mon[36291]: pgmap v388: 97 pgs: 12 active+clean+snaptrim, 22 active+clean+snaptrim_wait, 63 active+clean; 72 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 2.0 MiB/s wr, 189 op/s 2024-03-16T12:03:10.805 INFO:tasks.workunit.client.0.smithi029.stdout: 1 1179272 9.21 MB/sec execute 535 sec latency 18.939 ms 2024-03-16T12:03:11.805 INFO:tasks.workunit.client.0.smithi029.stdout: 1 1179867 9.20 MB/sec execute 536 sec latency 35.412 ms 2024-03-16T12:03:12.542 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:03:12 smithi154 ceph-mon[36320]: pgmap v389: 97 pgs: 2 active+clean+snaptrim, 4 active+clean+snaptrim_wait, 91 active+clean; 66 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 2.5 MiB/s wr, 229 op/s 2024-03-16T12:03:12.585 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:03:12 smithi029 ceph-mon[31975]: pgmap v389: 97 pgs: 2 active+clean+snaptrim, 4 active+clean+snaptrim_wait, 91 active+clean; 66 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 2.5 MiB/s wr, 229 op/s 2024-03-16T12:03:12.624 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:03:12 smithi152 ceph-mon[36291]: pgmap v389: 97 pgs: 2 active+clean+snaptrim, 4 active+clean+snaptrim_wait, 91 active+clean; 66 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 2.5 MiB/s wr, 229 op/s 2024-03-16T12:03:12.805 INFO:tasks.workunit.client.0.smithi029.stdout: 1 1181776 9.20 MB/sec execute 537 sec latency 22.235 ms 2024-03-16T12:03:13.805 INFO:tasks.workunit.client.0.smithi029.stdout: 1 1183155 9.20 MB/sec execute 538 sec latency 22.957 ms 2024-03-16T12:03:14.542 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:03:14 smithi154 ceph-mon[36320]: pgmap v390: 97 pgs: 97 active+clean; 66 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 2.5 MiB/s wr, 232 op/s 2024-03-16T12:03:14.585 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:03:14 smithi029 ceph-mon[31975]: pgmap v390: 97 pgs: 97 active+clean; 66 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 2.5 MiB/s wr, 232 op/s 2024-03-16T12:03:14.624 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:03:14 smithi152 ceph-mon[36291]: pgmap v390: 97 pgs: 97 active+clean; 66 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 2.5 MiB/s wr, 232 op/s 2024-03-16T12:03:14.805 INFO:tasks.workunit.client.0.smithi029.stdout: 1 1184638 9.19 MB/sec execute 539 sec latency 29.004 ms 2024-03-16T12:03:15.542 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:03:15 smithi154 ceph-mon[36320]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T12:03:15.585 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:03:15 smithi029 ceph-mon[31975]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T12:03:15.624 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:03:15 smithi152 ceph-mon[36291]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T12:03:15.805 INFO:tasks.workunit.client.0.smithi029.stdout: 1 1186410 9.20 MB/sec execute 540 sec latency 19.202 ms 2024-03-16T12:03:16.542 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:03:16 smithi154 ceph-mon[36320]: pgmap v391: 97 pgs: 97 active+clean; 66 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 2.3 MiB/s wr, 214 op/s 2024-03-16T12:03:16.542 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:03:16 smithi154 ceph-mon[36320]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' 2024-03-16T12:03:16.542 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:03:16 smithi154 ceph-mon[36320]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' 2024-03-16T12:03:16.542 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:03:16 smithi154 ceph-mon[36320]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:03:16.543 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:03:16 smithi154 ceph-mon[36320]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T12:03:16.543 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:03:16 smithi154 ceph-mon[36320]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' 2024-03-16T12:03:16.585 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:03:16 smithi029 ceph-mon[31975]: pgmap v391: 97 pgs: 97 active+clean; 66 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 2.3 MiB/s wr, 214 op/s 2024-03-16T12:03:16.585 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:03:16 smithi029 ceph-mon[31975]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' 2024-03-16T12:03:16.585 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:03:16 smithi029 ceph-mon[31975]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' 2024-03-16T12:03:16.585 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:03:16 smithi029 ceph-mon[31975]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:03:16.585 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:03:16 smithi029 ceph-mon[31975]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T12:03:16.585 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:03:16 smithi029 ceph-mon[31975]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' 2024-03-16T12:03:16.626 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:03:16 smithi152 ceph-mon[36291]: pgmap v391: 97 pgs: 97 active+clean; 66 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 2.3 MiB/s wr, 214 op/s 2024-03-16T12:03:16.626 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:03:16 smithi152 ceph-mon[36291]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' 2024-03-16T12:03:16.626 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:03:16 smithi152 ceph-mon[36291]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' 2024-03-16T12:03:16.626 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:03:16 smithi152 ceph-mon[36291]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:03:16.626 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:03:16 smithi152 ceph-mon[36291]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T12:03:16.626 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:03:16 smithi152 ceph-mon[36291]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' 2024-03-16T12:03:16.805 INFO:tasks.workunit.client.0.smithi029.stdout: 1 1187280 9.19 MB/sec execute 541 sec latency 66.513 ms 2024-03-16T12:03:17.805 INFO:tasks.workunit.client.0.smithi029.stdout: 1 1189464 9.19 MB/sec execute 542 sec latency 19.814 ms 2024-03-16T12:03:18.374 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:03:18 smithi152 ceph-mon[36291]: pgmap v392: 97 pgs: 97 active+clean; 71 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 3.2 MiB/s wr, 265 op/s 2024-03-16T12:03:18.374 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:03:18 smithi152 ceph-mon[36291]: osdmap e107: 12 total, 12 up, 12 in 2024-03-16T12:03:18.542 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:03:18 smithi154 ceph-mon[36320]: pgmap v392: 97 pgs: 97 active+clean; 71 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 3.2 MiB/s wr, 265 op/s 2024-03-16T12:03:18.542 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:03:18 smithi154 ceph-mon[36320]: osdmap e107: 12 total, 12 up, 12 in 2024-03-16T12:03:18.585 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:03:18 smithi029 ceph-mon[31975]: pgmap v392: 97 pgs: 97 active+clean; 71 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 3.2 MiB/s wr, 265 op/s 2024-03-16T12:03:18.585 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:03:18 smithi029 ceph-mon[31975]: osdmap e107: 12 total, 12 up, 12 in 2024-03-16T12:03:18.805 INFO:tasks.workunit.client.0.smithi029.stdout: 1 1190433 9.19 MB/sec execute 543 sec latency 24.304 ms 2024-03-16T12:03:19.805 INFO:tasks.workunit.client.0.smithi029.stdout: 1 1192167 9.18 MB/sec execute 544 sec latency 99.423 ms 2024-03-16T12:03:20.374 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:03:20 smithi152 ceph-mon[36291]: pgmap v394: 97 pgs: 97 active+clean; 71 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 2.8 MiB/s wr, 247 op/s 2024-03-16T12:03:20.542 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:03:20 smithi154 ceph-mon[36320]: pgmap v394: 97 pgs: 97 active+clean; 71 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 2.8 MiB/s wr, 247 op/s 2024-03-16T12:03:20.585 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:03:20 smithi029 ceph-mon[31975]: pgmap v394: 97 pgs: 97 active+clean; 71 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 2.8 MiB/s wr, 247 op/s 2024-03-16T12:03:20.805 INFO:tasks.workunit.client.0.smithi029.stdout: 1 1193617 9.19 MB/sec execute 545 sec latency 16.417 ms 2024-03-16T12:03:21.805 INFO:tasks.workunit.client.0.smithi029.stdout: 1 1194869 9.18 MB/sec execute 546 sec latency 65.275 ms 2024-03-16T12:03:22.356 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:22.356+0000 7fd709ec6640 1 -- 172.21.15.29:0/3288088268 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd704100930 msgr2=0x7fd704100db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:03:22.356 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:22.356+0000 7fd709ec6640 1 --2- 172.21.15.29:0/3288088268 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd704100930 0x7fd704100db0 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7fd6f8009900 tx=0x7fd6f802bc30 comp rx=0 tx=0).stop 2024-03-16T12:03:22.356 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:22.356+0000 7fd709ec6640 1 -- 172.21.15.29:0/3288088268 shutdown_connections 2024-03-16T12:03:22.357 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:22.356+0000 7fd709ec6640 1 --2- 172.21.15.29:0/3288088268 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd7041012f0 0x7fd70410d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:03:22.357 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:22.356+0000 7fd709ec6640 1 --2- 172.21.15.29:0/3288088268 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd704100930 0x7fd704100db0 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:03:22.357 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:22.356+0000 7fd709ec6640 1 --2- 172.21.15.29:0/3288088268 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd704105680 0x7fd704105a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:03:22.357 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:22.356+0000 7fd709ec6640 1 -- 172.21.15.29:0/3288088268 >> 172.21.15.29:0/3288088268 conn(0x7fd704077530 msgr2=0x7fd704077940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:03:22.357 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:22.356+0000 7fd709ec6640 1 -- 172.21.15.29:0/3288088268 shutdown_connections 2024-03-16T12:03:22.357 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:22.356+0000 7fd709ec6640 1 -- 172.21.15.29:0/3288088268 wait complete. 2024-03-16T12:03:22.357 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:22.356+0000 7fd709ec6640 1 Processor -- start 2024-03-16T12:03:22.357 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:22.356+0000 7fd709ec6640 1 -- start start 2024-03-16T12:03:22.357 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:22.357+0000 7fd709ec6640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd704100930 0x7fd704102350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:03:22.357 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:22.357+0000 7fd709ec6640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd7041012f0 0x7fd704102890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:03:22.357 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:22.357+0000 7fd709ec6640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd704105680 0x7fd704102dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:03:22.357 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:22.357+0000 7fd709ec6640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fd704110ba0 con 0x7fd7041012f0 2024-03-16T12:03:22.357 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:22.357+0000 7fd709ec6640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fd704110a20 con 0x7fd704105680 2024-03-16T12:03:22.357 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:22.357+0000 7fd709ec6640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fd704110d20 con 0x7fd704100930 2024-03-16T12:03:22.358 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:22.357+0000 7fd702ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd7041012f0 0x7fd704102890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:03:22.358 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:22.357+0000 7fd702ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd7041012f0 0x7fd704102890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50700/0 (socket says 172.21.15.29:50700) 2024-03-16T12:03:22.358 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:22.357+0000 7fd702ffd640 1 -- 172.21.15.29:0/3538164856 learned_addr learned my addr 172.21.15.29:0/3538164856 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T12:03:22.358 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:22.357+0000 7fd7037fe640 1 --2- 172.21.15.29:0/3538164856 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd704100930 0x7fd704102350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:03:22.358 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:22.357+0000 7fd703fff640 1 --2- 172.21.15.29:0/3538164856 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd704105680 0x7fd704102dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:03:22.358 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:22.357+0000 7fd702ffd640 1 -- 172.21.15.29:0/3538164856 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd704100930 msgr2=0x7fd704102350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:03:22.358 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:22.357+0000 7fd702ffd640 1 --2- 172.21.15.29:0/3538164856 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd704100930 0x7fd704102350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:03:22.358 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:22.357+0000 7fd702ffd640 1 -- 172.21.15.29:0/3538164856 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd704105680 msgr2=0x7fd704102dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:03:22.358 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:22.357+0000 7fd702ffd640 1 --2- 172.21.15.29:0/3538164856 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd704105680 0x7fd704102dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:03:22.358 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:22.357+0000 7fd702ffd640 1 -- 172.21.15.29:0/3538164856 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7041a6600 con 0x7fd7041012f0 2024-03-16T12:03:22.358 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:22.357+0000 7fd7037fe640 1 --2- 172.21.15.29:0/3538164856 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd704100930 0x7fd704102350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:03:22.358 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:22.357+0000 7fd703fff640 1 --2- 172.21.15.29:0/3538164856 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd704105680 0x7fd704102dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:03:22.358 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:22.357+0000 7fd702ffd640 1 --2- 172.21.15.29:0/3538164856 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd7041012f0 0x7fd704102890 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7fd6f8005940 tx=0x7fd6f8004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:03:22.359 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:22.357+0000 7fd700ff9640 1 -- 172.21.15.29:0/3538164856 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6f8041070 con 0x7fd7041012f0 2024-03-16T12:03:22.359 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:22.357+0000 7fd709ec6640 1 -- 172.21.15.29:0/3538164856 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd7041a6890 con 0x7fd7041012f0 2024-03-16T12:03:22.359 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:22.357+0000 7fd700ff9640 1 -- 172.21.15.29:0/3538164856 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd6f80054e0 con 0x7fd7041012f0 2024-03-16T12:03:22.359 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:22.358+0000 7fd700ff9640 1 -- 172.21.15.29:0/3538164856 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6f8003d00 con 0x7fd7041012f0 2024-03-16T12:03:22.359 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:22.358+0000 7fd709ec6640 1 -- 172.21.15.29:0/3538164856 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd7041a6dd0 con 0x7fd7041012f0 2024-03-16T12:03:22.365 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:22.360+0000 7fd700ff9640 1 -- 172.21.15.29:0/3538164856 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7fd6f8044070 con 0x7fd7041012f0 2024-03-16T12:03:22.365 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:22.360+0000 7fd70cb90740 1 Processor -- start 2024-03-16T12:03:22.365 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:22.360+0000 7fd70cb90740 1 -- start start 2024-03-16T12:03:22.365 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:22.360+0000 7fd70cb90740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55968ee54130 0x55968ee54510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:03:22.365 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:22.360+0000 7fd70cb90740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55968ee54a50 0x55968ee5d2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:03:22.365 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:22.360+0000 7fd70cb90740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55968ee5d820 0x55968ee5fc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:03:22.365 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:22.360+0000 7fd70cb90740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55968edb8a40 con 0x55968ee5d820 2024-03-16T12:03:22.366 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:22.360+0000 7fd70cb90740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x55968ed61920 con 0x55968ee54a50 2024-03-16T12:03:22.366 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:22.360+0000 7fd70cb90740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55968edb4ae0 con 0x55968ee54130 2024-03-16T12:03:22.366 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:22.360+0000 7fd700ff9640 1 --2- 172.21.15.29:0/3538164856 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7fd6ec0a2a10 0x7fd6ec0a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:03:22.366 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:22.361+0000 7fd703fff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55968ee54130 0x55968ee54510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:03:22.366 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:22.361+0000 7fd702ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55968ee5d820 0x55968ee5fc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:03:22.366 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:22.361+0000 7fd703fff640 1 --2- 172.21.15.29:0/3538164856 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7fd6ec0a2a10 0x7fd6ec0a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:03:22.366 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:22.361+0000 7fd7037fe640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55968ee54a50 0x55968ee5d2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:03:22.366 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:22.361+0000 7fd703fff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55968ee54130 0x55968ee54510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.29:41628/0 (socket says 172.21.15.29:41628) 2024-03-16T12:03:22.366 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:22.361+0000 7fd7037fe640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55968ee54a50 0x55968ee5d2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.29:36610/0 (socket says 172.21.15.29:36610) 2024-03-16T12:03:22.366 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:22.361+0000 7fd703fff640 1 -- 172.21.15.29:0/941351048 learned_addr learned my addr 172.21.15.29:0/941351048 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T12:03:22.366 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:22.361+0000 7fd700ff9640 1 -- 172.21.15.29:0/3538164856 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 8556+0+0 (secure 0 0 0) 0x7fd6f80f07e0 con 0x7fd7041012f0 2024-03-16T12:03:22.366 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:22.361+0000 7fd703fff640 1 -- 172.21.15.29:0/941351048 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55968ee54a50 msgr2=0x55968ee5d2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:03:22.366 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:22.361+0000 7fd703fff640 1 --2- 172.21.15.29:0/941351048 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55968ee54a50 0x55968ee5d2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:03:22.367 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:22.361+0000 7fd703fff640 1 -- 172.21.15.29:0/941351048 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55968ee5d820 msgr2=0x55968ee5fc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:03:22.367 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:22.361+0000 7fd703fff640 1 --2- 172.21.15.29:0/941351048 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55968ee5d820 0x55968ee5fc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:03:22.367 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:22.361+0000 7fd703fff640 1 -- 172.21.15.29:0/941351048 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55968ed79460 con 0x55968ee54130 2024-03-16T12:03:22.367 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:22.361+0000 7fd703fff640 1 --2- 172.21.15.29:0/3538164856 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7fd6ec0a2a10 0x7fd6ec0a4ed0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fd6fc00fc90 tx=0x7fd6fc018000 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:03:22.367 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:22.361+0000 7fd703fff640 1 --2- 172.21.15.29:0/941351048 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55968ee54130 0x55968ee54510 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fd6fc010e90 tx=0x7fd6fc00f790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a58762a2209e1fcf server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:03:22.367 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:22.361+0000 7fd6e9ffb640 1 -- 172.21.15.29:0/941351048 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6fc019040 con 0x55968ee54130 2024-03-16T12:03:22.367 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:22.361+0000 7fd6e9ffb640 1 -- 172.21.15.29:0/941351048 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd6fc00ec50 con 0x55968ee54130 2024-03-16T12:03:22.367 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:22.361+0000 7fd6e9ffb640 1 -- 172.21.15.29:0/941351048 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6fc0253a0 con 0x55968ee54130 2024-03-16T12:03:22.367 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:22.362+0000 7fd70cb90740 1 -- 172.21.15.29:0/941351048 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55968ee54130 msgr2=0x55968ee54510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:03:22.367 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:22.362+0000 7fd70cb90740 1 --2- 172.21.15.29:0/941351048 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55968ee54130 0x55968ee54510 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fd6fc010e90 tx=0x7fd6fc00f790 comp rx=0 tx=0).stop 2024-03-16T12:03:22.367 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:22.362+0000 7fd70cb90740 1 -- 172.21.15.29:0/941351048 shutdown_connections 2024-03-16T12:03:22.367 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:22.362+0000 7fd70cb90740 1 --2- 172.21.15.29:0/941351048 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55968ee5d820 0x55968ee5fc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:03:22.367 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:22.362+0000 7fd70cb90740 1 --2- 172.21.15.29:0/941351048 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55968ee54a50 0x55968ee5d2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:03:22.367 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:22.362+0000 7fd70cb90740 1 --2- 172.21.15.29:0/941351048 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55968ee54130 0x55968ee54510 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:03:22.367 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:22.362+0000 7fd70cb90740 1 -- 172.21.15.29:0/941351048 >> 172.21.15.29:0/941351048 conn(0x55968ee4ea50 msgr2=0x55968ee4ee30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:03:22.368 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:22.362+0000 7fd70cb90740 1 -- 172.21.15.29:0/941351048 shutdown_connections 2024-03-16T12:03:22.368 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:22.362+0000 7fd70cb90740 1 -- 172.21.15.29:0/941351048 wait complete. 2024-03-16T12:03:22.368 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:22.362+0000 7fd70cb90740 1 -- start start 2024-03-16T12:03:22.368 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:22.363+0000 7fd70cb90740 1 Processor -- start 2024-03-16T12:03:22.368 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:22.363+0000 7fd70cb90740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55968ee54130 0x55968ee5a850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:03:22.368 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:22.363+0000 7fd70cb90740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55968ee54a50 0x55968ee5ad90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:03:22.368 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:22.363+0000 7fd70cb90740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55968ee5d820 0x55968ee5b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:03:22.368 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:22.363+0000 7fd70cb90740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55968ee603f0 con 0x55968ee5d820 2024-03-16T12:03:22.368 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:22.363+0000 7fd70cb90740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x55968ed74d70 con 0x55968ee54130 2024-03-16T12:03:22.368 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:22.363+0000 7fd70cb90740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55968ee60570 con 0x55968ee54a50 2024-03-16T12:03:22.368 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:22.363+0000 7fd70cb90740 10 client.0 _ll_register_callbacks cb 0x55968eacca00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T12:03:22.368 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:22.363+0000 7fd703fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55968ee5d820 0x55968ee5b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:03:22.368 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:22.363+0000 7fd702ffd640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55968ee54a50 0x55968ee5ad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:03:22.369 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:22.363+0000 7fd7037fe640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55968ee54130 0x55968ee5a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:03:22.369 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:22.363+0000 7fd702ffd640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55968ee54a50 0x55968ee5ad90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.29:41642/0 (socket says 172.21.15.29:41642) 2024-03-16T12:03:22.369 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:22.363+0000 7fd702ffd640 1 -- 172.21.15.29:0/262474677 learned_addr learned my addr 172.21.15.29:0/262474677 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T12:03:22.369 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:22.363+0000 7fd703fff640 1 -- 172.21.15.29:0/262474677 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55968ee54a50 msgr2=0x55968ee5ad90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:03:22.369 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:22.363+0000 7fd703fff640 1 --2- 172.21.15.29:0/262474677 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55968ee54a50 0x55968ee5ad90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:03:22.369 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:22.363+0000 7fd703fff640 1 -- 172.21.15.29:0/262474677 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55968ee54130 msgr2=0x55968ee5a850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:03:22.369 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:22.363+0000 7fd703fff640 1 --2- 172.21.15.29:0/262474677 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55968ee54130 0x55968ee5a850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:03:22.369 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:22.363+0000 7fd703fff640 1 -- 172.21.15.29:0/262474677 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55968eacd210 con 0x55968ee5d820 2024-03-16T12:03:22.369 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:22.364+0000 7fd703fff640 1 --2- 172.21.15.29:0/262474677 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55968ee5d820 0x55968ee5b2d0 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7fd6fc009660 tx=0x7fd6fc01ef50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7e8e882d8520f4c server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:03:22.369 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:22.364+0000 7fd6e2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-16T12:03:22.369 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:22.364+0000 7fd6e2ffd640 1 -- 172.21.15.29:0/262474677 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6fc00ee00 con 0x55968ee5d820 2024-03-16T12:03:22.369 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:22.364+0000 7fd6e2ffd640 1 -- 172.21.15.29:0/262474677 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(107..107 src has 1..107) ==== 8556+0+0 (secure 0 0 0) 0x7fd6fc02b040 con 0x55968ee5d820 2024-03-16T12:03:22.369 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:22.364+0000 7fd70cb90740 1 -- 172.21.15.29:0/262474677 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd6fc00ec50 con 0x55968ee5d820 2024-03-16T12:03:22.369 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:22.364+0000 7fd6e2ffd640 1 -- 172.21.15.29:0/262474677 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fd6fc03b050 con 0x55968ee5d820 2024-03-16T12:03:22.370 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:22.364+0000 7fd70cb90740 10 client.16437 fetch_fsmap learned FSMap version 29 2024-03-16T12:03:22.370 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:22.364+0000 7fd70cb90740 1 -- 172.21.15.29:0/262474677 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55968eb99620 con 0x55968ee5d820 2024-03-16T12:03:22.370 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:22.364+0000 7fd6e2ffd640 1 -- 172.21.15.29:0/262474677 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd6fc00ec50 con 0x55968ee5d820 2024-03-16T12:03:22.370 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:22.364+0000 7fd6e2ffd640 1 -- 172.21.15.29:0/262474677 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 29) ==== 8338+0+0 (secure 0 0 0) 0x7fd6fc01e690 con 0x55968ee5d820 2024-03-16T12:03:22.370 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:22.364+0000 7fd70cb90740 10 client.16437 fetch_fsmap finished waiting for FSMap version 29 2024-03-16T12:03:22.370 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:22.364+0000 7fd70cb90740 10 client.16437 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T12:03:22.370 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:22.364+0000 7fd70cb90740 20 client.16437 populate_metadata read hostname 'smithi029' 2024-03-16T12:03:22.370 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:22.364+0000 7fd70cb90740 1 --2- 172.21.15.29:0/262474677 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55968ee6d8e0 0x55968ee6fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:03:22.370 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:22.364+0000 7fd70cb90740 4 client.16437 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T12:03:22.370 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:22.364+0000 7fd70cb90740 1 -- 172.21.15.29:0/262474677 --> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55968edc5f80 con 0x55968ee6d8e0 2024-03-16T12:03:22.370 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:22.365+0000 7fd7037fe640 1 --2- 172.21.15.29:0/262474677 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55968ee6d8e0 0x55968ee6fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:03:22.370 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:22.365+0000 7fd7037fe640 1 --2- 172.21.15.29:0/262474677 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55968ee6d8e0 0x55968ee6fcd0 crc :-1 s=READY pgs=150 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=28d91ffb6a219e66 server_cookie=83c81c477e5315e3 in_seq=0 out_seq=0 2024-03-16T12:03:22.370 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:22.365+0000 7fd6e2ffd640 10 client.16437 ms_handle_connect on v2:172.21.15.152:6838/1548731833 2024-03-16T12:03:22.370 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:22.366+0000 7fd6e2ffd640 1 -- 172.21.15.29:0/262474677 <== mds.0 v2:172.21.15.152:6838/1548731833 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55968edc5f80 con 0x55968ee6d8e0 2024-03-16T12:03:22.371 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:22.366+0000 7fd6e2ffd640 10 client.16437 handle_command_reply: tid=0 2024-03-16T12:03:22.371 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:22.366+0000 7fd70cb90740 1 client.16437 shutdown 2024-03-16T12:03:22.371 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:22.366+0000 7fd70cb90740 1 -- 172.21.15.29:0/262474677 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55968ee5d820 msgr2=0x55968ee5b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:03:22.371 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:22.366+0000 7fd70cb90740 1 --2- 172.21.15.29:0/262474677 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55968ee5d820 0x55968ee5b2d0 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7fd6fc009660 tx=0x7fd6fc01ef50 comp rx=0 tx=0).stop 2024-03-16T12:03:22.371 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:22.366+0000 7fd70cb90740 1 -- 172.21.15.29:0/262474677 shutdown_connections 2024-03-16T12:03:22.371 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:22.366+0000 7fd70cb90740 1 --2- 172.21.15.29:0/262474677 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55968ee5d820 0x55968ee5b2d0 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:03:22.371 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:22.366+0000 7fd70cb90740 1 --2- 172.21.15.29:0/262474677 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55968ee54a50 0x55968ee5ad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:03:22.371 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:22.366+0000 7fd70cb90740 1 --2- 172.21.15.29:0/262474677 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55968ee6d8e0 0x55968ee6fcd0 crc :-1 s=READY pgs=150 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:03:22.371 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:22.366+0000 7fd70cb90740 1 --2- 172.21.15.29:0/262474677 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55968ee54130 0x55968ee5a850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:03:22.371 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:22.366+0000 7fd70cb90740 1 -- 172.21.15.29:0/262474677 >> 172.21.15.29:0/262474677 conn(0x55968ee4ea50 msgr2=0x55968ee507a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:03:22.371 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:22.366+0000 7fd6e2ffd640 0 client.16437 ms_handle_reset on v2:172.21.15.152:6838/1548731833 2024-03-16T12:03:22.371 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:22.366+0000 7fd70cb90740 1 -- 172.21.15.29:0/262474677 shutdown_connections 2024-03-16T12:03:22.371 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:22.366+0000 7fd70cb90740 1 -- 172.21.15.29:0/262474677 wait complete. 2024-03-16T12:03:22.371 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:22.366+0000 7fd70cb90740 20 client.16437 trim_cache size 0 max 16384 2024-03-16T12:03:22.374 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:03:22 smithi152 ceph-mon[36291]: pgmap v395: 97 pgs: 97 active+clean; 77 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 2.9 MiB/s wr, 183 op/s 2024-03-16T12:03:22.380 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:22.381+0000 7fd70cb90740 1 Processor -- start 2024-03-16T12:03:22.380 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:22.382+0000 7fd70cb90740 1 -- start start 2024-03-16T12:03:22.380 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:22.382+0000 7fd70cb90740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55968ee54130 0x55968ee5a130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:03:22.380 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:22.382+0000 7fd70cb90740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55968ee54a50 0x55968ee5a810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:03:22.380 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:22.382+0000 7fd70cb90740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55968ee5d820 0x55968ee69190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:03:22.380 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:22.382+0000 7fd70cb90740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fd7041b1f60 con 0x55968ee54a50 2024-03-16T12:03:22.381 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:22.382+0000 7fd70cb90740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fd7041af830 con 0x55968ee54130 2024-03-16T12:03:22.381 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:22.382+0000 7fd70cb90740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fd7041ad6a0 con 0x55968ee5d820 2024-03-16T12:03:22.381 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:22.382+0000 7fd702ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55968ee54a50 0x55968ee5a810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:03:22.381 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:22.382+0000 7fd702ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55968ee54a50 0x55968ee5a810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50730/0 (socket says 172.21.15.29:50730) 2024-03-16T12:03:22.381 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:22.382+0000 7fd702ffd640 1 -- 172.21.15.29:0/3325469583 learned_addr learned my addr 172.21.15.29:0/3325469583 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T12:03:22.381 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:22.382+0000 7fd703fff640 1 --2- 172.21.15.29:0/3325469583 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55968ee5d820 0x55968ee69190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:03:22.381 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:22.382+0000 7fd7037fe640 1 --2- 172.21.15.29:0/3325469583 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55968ee54130 0x55968ee5a130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:03:22.381 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:22.382+0000 7fd702ffd640 1 -- 172.21.15.29:0/3325469583 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55968ee5d820 msgr2=0x55968ee69190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:03:22.381 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:22.382+0000 7fd702ffd640 1 --2- 172.21.15.29:0/3325469583 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55968ee5d820 0x55968ee69190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:03:22.381 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:22.382+0000 7fd702ffd640 1 -- 172.21.15.29:0/3325469583 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55968ee54130 msgr2=0x55968ee5a130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:03:22.381 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:22.382+0000 7fd702ffd640 1 --2- 172.21.15.29:0/3325469583 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55968ee54130 0x55968ee5a130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:03:22.381 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:22.382+0000 7fd702ffd640 1 -- 172.21.15.29:0/3325469583 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55968ee54880 con 0x55968ee54a50 2024-03-16T12:03:22.381 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:22.382+0000 7fd702ffd640 1 --2- 172.21.15.29:0/3325469583 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55968ee54a50 0x55968ee5a810 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7fd6f80ee510 tx=0x7fd6f8042f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=767c0541b68c0cdc server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:03:22.381 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:22.383+0000 7fd6e97fa640 1 -- 172.21.15.29:0/3325469583 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6f804b040 con 0x55968ee54a50 2024-03-16T12:03:22.382 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:22.383+0000 7fd6e97fa640 1 -- 172.21.15.29:0/3325469583 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd6f802ccd0 con 0x55968ee54a50 2024-03-16T12:03:22.382 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:22.383+0000 7fd6e97fa640 1 -- 172.21.15.29:0/3325469583 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6f80c56d0 con 0x55968ee54a50 2024-03-16T12:03:22.385 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:22.383+0000 7fd70cb90740 1 -- 172.21.15.29:0/3325469583 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55968ee54a50 msgr2=0x55968ee5a810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:03:22.385 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:22.383+0000 7fd70cb90740 1 --2- 172.21.15.29:0/3325469583 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55968ee54a50 0x55968ee5a810 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7fd6f80ee510 tx=0x7fd6f8042f50 comp rx=0 tx=0).stop 2024-03-16T12:03:22.385 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:22.383+0000 7fd70cb90740 1 -- 172.21.15.29:0/3325469583 shutdown_connections 2024-03-16T12:03:22.385 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:22.383+0000 7fd70cb90740 1 --2- 172.21.15.29:0/3325469583 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55968ee5d820 0x55968ee69190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:03:22.385 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:22.383+0000 7fd70cb90740 1 --2- 172.21.15.29:0/3325469583 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55968ee54a50 0x55968ee5a810 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:03:22.386 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:22.383+0000 7fd70cb90740 1 --2- 172.21.15.29:0/3325469583 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55968ee54130 0x55968ee5a130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:03:22.386 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:22.383+0000 7fd70cb90740 1 -- 172.21.15.29:0/3325469583 >> 172.21.15.29:0/3325469583 conn(0x55968ee4ea50 msgr2=0x55968ee4dbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:03:22.386 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:22.383+0000 7fd70cb90740 1 -- 172.21.15.29:0/3325469583 shutdown_connections 2024-03-16T12:03:22.386 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:22.383+0000 7fd70cb90740 1 -- 172.21.15.29:0/3325469583 wait complete. 2024-03-16T12:03:22.386 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:22.384+0000 7fd70cb90740 1 -- start start 2024-03-16T12:03:22.386 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:22.384+0000 7fd70cb90740 1 Processor -- start 2024-03-16T12:03:22.386 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:22.384+0000 7fd70cb90740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55968ee54130 0x55968ee65a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:03:22.386 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:22.384+0000 7fd70cb90740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55968ee54a50 0x55968ee60e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:03:22.386 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:22.384+0000 7fd70cb90740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55968ee5d820 0x55968ee613a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:03:22.386 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:22.384+0000 7fd70cb90740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fd7041adcf0 con 0x55968ee54a50 2024-03-16T12:03:22.386 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:22.384+0000 7fd70cb90740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fd7041af160 con 0x55968ee54130 2024-03-16T12:03:22.386 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:22.384+0000 7fd70cb90740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fd7041ae3c0 con 0x55968ee5d820 2024-03-16T12:03:22.386 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:22.384+0000 7fd70cb90740 10 client.0 _ll_register_callbacks cb 0x7fd6f80b9330 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T12:03:22.386 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:22.384+0000 7fd702ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55968ee54a50 0x55968ee60e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:03:22.387 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:22.384+0000 7fd702ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55968ee54a50 0x55968ee60e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50732/0 (socket says 172.21.15.29:50732) 2024-03-16T12:03:22.387 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:22.384+0000 7fd702ffd640 1 -- 172.21.15.29:0/3056428736 learned_addr learned my addr 172.21.15.29:0/3056428736 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T12:03:22.387 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:22.384+0000 7fd7037fe640 1 --2- 172.21.15.29:0/3056428736 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55968ee54130 0x55968ee65a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:03:22.387 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:22.384+0000 7fd703fff640 1 --2- 172.21.15.29:0/3056428736 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55968ee5d820 0x55968ee613a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:03:22.387 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:22.384+0000 7fd702ffd640 1 -- 172.21.15.29:0/3056428736 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55968ee5d820 msgr2=0x55968ee613a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:03:22.387 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:22.384+0000 7fd702ffd640 1 --2- 172.21.15.29:0/3056428736 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55968ee5d820 0x55968ee613a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:03:22.387 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:22.384+0000 7fd702ffd640 1 -- 172.21.15.29:0/3056428736 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55968ee54130 msgr2=0x55968ee65a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:03:22.387 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:22.384+0000 7fd702ffd640 1 --2- 172.21.15.29:0/3056428736 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55968ee54130 0x55968ee65a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:03:22.387 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:22.384+0000 7fd702ffd640 1 -- 172.21.15.29:0/3056428736 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55968ee61bd0 con 0x55968ee54a50 2024-03-16T12:03:22.387 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:22.385+0000 7fd702ffd640 1 --2- 172.21.15.29:0/3056428736 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55968ee54a50 0x55968ee60e60 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7fd6f802e040 tx=0x7fd6f80bcc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24ecebfee9401c3d server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:03:22.387 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:22.385+0000 7fd6e2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-16T12:03:22.387 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:22.385+0000 7fd6e2ffd640 1 -- 172.21.15.29:0/3056428736 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6f80bc940 con 0x55968ee54a50 2024-03-16T12:03:22.387 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:22.385+0000 7fd70cb90740 1 -- 172.21.15.29:0/3056428736 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55968ee56450 con 0x55968ee54a50 2024-03-16T12:03:22.387 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:22.385+0000 7fd6e2ffd640 1 -- 172.21.15.29:0/3056428736 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(107..107 src has 1..107) ==== 8556+0+0 (secure 0 0 0) 0x7fd6f80823c0 con 0x55968ee54a50 2024-03-16T12:03:22.388 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:22.385+0000 7fd6e2ffd640 1 -- 172.21.15.29:0/3056428736 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fd6f8005690 con 0x55968ee54a50 2024-03-16T12:03:22.388 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:22.385+0000 7fd70cb90740 10 client.16443 fetch_fsmap learned FSMap version 29 2024-03-16T12:03:22.388 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:22.385+0000 7fd70cb90740 1 -- 172.21.15.29:0/3056428736 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55968eed38d0 con 0x55968ee54a50 2024-03-16T12:03:22.388 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:22.385+0000 7fd6e2ffd640 1 -- 172.21.15.29:0/3056428736 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55968ee56450 con 0x55968ee54a50 2024-03-16T12:03:22.388 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:22.385+0000 7fd6e2ffd640 1 -- 172.21.15.29:0/3056428736 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 29) ==== 8338+0+0 (secure 0 0 0) 0x7fd6f80c5240 con 0x55968ee54a50 2024-03-16T12:03:22.388 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:22.385+0000 7fd70cb90740 10 client.16443 fetch_fsmap finished waiting for FSMap version 29 2024-03-16T12:03:22.388 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:22.385+0000 7fd70cb90740 10 client.16443 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T12:03:22.388 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:22.385+0000 7fd70cb90740 20 client.16443 populate_metadata read hostname 'smithi029' 2024-03-16T12:03:22.388 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:22.385+0000 7fd70cb90740 1 --2- 172.21.15.29:0/3056428736 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55968eed3f70 0x55968eed6390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:03:22.388 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:22.385+0000 7fd70cb90740 4 client.16443 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-16T12:03:22.388 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:22.385+0000 7fd70cb90740 1 -- 172.21.15.29:0/3056428736 --> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55968edc02d0 con 0x55968eed3f70 2024-03-16T12:03:22.388 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:22.385+0000 7fd7037fe640 1 --2- 172.21.15.29:0/3056428736 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55968eed3f70 0x55968eed6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:03:22.388 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:22.386+0000 7fd6e2ffd640 10 client.16443 ms_handle_connect on v2:172.21.15.152:6838/1548731833 2024-03-16T12:03:22.388 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:22.386+0000 7fd7037fe640 1 --2- 172.21.15.29:0/3056428736 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55968eed3f70 0x55968eed6390 crc :-1 s=READY pgs=151 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a8a9413add50849a server_cookie=64d28ed5e3f639e0 in_seq=0 out_seq=0 2024-03-16T12:03:22.542 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:03:22 smithi154 ceph-mon[36320]: pgmap v395: 97 pgs: 97 active+clean; 77 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 2.9 MiB/s wr, 183 op/s 2024-03-16T12:03:22.559 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:22.560+0000 7fd6e2ffd640 1 -- 172.21.15.29:0/3056428736 <== mds.0 v2:172.21.15.152:6838/1548731833 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55968edc02d0 con 0x55968eed3f70 2024-03-16T12:03:22.559 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:22.560+0000 7fd6e2ffd640 10 client.16443 handle_command_reply: tid=0 2024-03-16T12:03:22.559 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:22.560+0000 7fd70cb90740 1 client.16443 shutdown 2024-03-16T12:03:22.559 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:22.561+0000 7fd70cb90740 1 -- 172.21.15.29:0/3056428736 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55968ee54a50 msgr2=0x55968ee60e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:03:22.559 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:22.561+0000 7fd70cb90740 1 --2- 172.21.15.29:0/3056428736 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55968ee54a50 0x55968ee60e60 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7fd6f802e040 tx=0x7fd6f80bcc20 comp rx=0 tx=0).stop 2024-03-16T12:03:22.559 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:22.561+0000 7fd70cb90740 1 -- 172.21.15.29:0/3056428736 shutdown_connections 2024-03-16T12:03:22.559 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:22.561+0000 7fd70cb90740 1 --2- 172.21.15.29:0/3056428736 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55968ee5d820 0x55968ee613a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:03:22.559 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:22.561+0000 7fd70cb90740 1 --2- 172.21.15.29:0/3056428736 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55968ee54a50 0x55968ee60e60 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:03:22.559 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:22.561+0000 7fd70cb90740 1 --2- 172.21.15.29:0/3056428736 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55968eed3f70 0x55968eed6390 crc :-1 s=READY pgs=151 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:03:22.559 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:22.561+0000 7fd70cb90740 1 --2- 172.21.15.29:0/3056428736 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55968ee54130 0x55968ee65a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:03:22.559 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:22.561+0000 7fd70cb90740 1 -- 172.21.15.29:0/3056428736 >> 172.21.15.29:0/3056428736 conn(0x55968ee4ea50 msgr2=0x55968ee4fd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:03:22.560 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:22.561+0000 7fd6e2ffd640 0 client.16443 ms_handle_reset on v2:172.21.15.152:6838/1548731833 2024-03-16T12:03:22.560 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:22.561+0000 7fd70cb90740 1 -- 172.21.15.29:0/3056428736 shutdown_connections 2024-03-16T12:03:22.560 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:22.561+0000 7fd70cb90740 1 -- 172.21.15.29:0/3056428736 wait complete. 2024-03-16T12:03:22.560 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:22.561+0000 7fd70cb90740 20 client.16443 trim_cache size 0 max 16384 2024-03-16T12:03:22.560 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:{ 2024-03-16T12:03:22.560 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "message": "", 2024-03-16T12:03:22.560 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "return_code": 0 2024-03-16T12:03:22.560 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:} 2024-03-16T12:03:22.561 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:22.562+0000 7fd709ec6640 1 -- 172.21.15.29:0/3538164856 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7fd6ec0a2a10 msgr2=0x7fd6ec0a4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:03:22.561 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:22.562+0000 7fd709ec6640 1 --2- 172.21.15.29:0/3538164856 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7fd6ec0a2a10 0x7fd6ec0a4ed0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fd6fc00fc90 tx=0x7fd6fc018000 comp rx=0 tx=0).stop 2024-03-16T12:03:22.561 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:22.562+0000 7fd709ec6640 1 -- 172.21.15.29:0/3538164856 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd7041012f0 msgr2=0x7fd704102890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:03:22.561 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:22.562+0000 7fd709ec6640 1 --2- 172.21.15.29:0/3538164856 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd7041012f0 0x7fd704102890 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7fd6f8005940 tx=0x7fd6f8004270 comp rx=0 tx=0).stop 2024-03-16T12:03:22.561 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:22.562+0000 7fd709ec6640 1 -- 172.21.15.29:0/3538164856 shutdown_connections 2024-03-16T12:03:22.561 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:22.562+0000 7fd709ec6640 1 --2- 172.21.15.29:0/3538164856 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7fd6ec0a2a10 0x7fd6ec0a4ed0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:03:22.561 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:22.562+0000 7fd709ec6640 1 --2- 172.21.15.29:0/3538164856 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd704105680 0x7fd704102dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:03:22.561 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:22.562+0000 7fd709ec6640 1 --2- 172.21.15.29:0/3538164856 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd7041012f0 0x7fd704102890 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:03:22.562 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:22.562+0000 7fd709ec6640 1 --2- 172.21.15.29:0/3538164856 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd704100930 0x7fd704102350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:03:22.562 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:22.562+0000 7fd709ec6640 1 -- 172.21.15.29:0/3538164856 >> 172.21.15.29:0/3538164856 conn(0x7fd704077530 msgr2=0x7fd704069200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:03:22.562 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:22.562+0000 7fd709ec6640 1 -- 172.21.15.29:0/3538164856 shutdown_connections 2024-03-16T12:03:22.562 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:22.562+0000 7fd709ec6640 1 -- 172.21.15.29:0/3538164856 wait complete. 2024-03-16T12:03:22.585 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:03:22 smithi029 ceph-mon[31975]: pgmap v395: 97 pgs: 97 active+clean; 77 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 2.9 MiB/s wr, 183 op/s 2024-03-16T12:03:22.805 INFO:tasks.workunit.client.0.smithi029.stdout: 1 1196744 9.18 MB/sec execute 547 sec latency 101.999 ms 2024-03-16T12:03:23.805 INFO:tasks.workunit.client.0.smithi029.stdout: 1 1197550 9.17 MB/sec execute 548 sec latency 17.006 ms 2024-03-16T12:03:24.542 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:03:24 smithi154 ceph-mon[36320]: pgmap v396: 97 pgs: 97 active+clean; 77 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 2.9 MiB/s wr, 173 op/s 2024-03-16T12:03:24.585 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:03:24 smithi029 ceph-mon[31975]: pgmap v396: 97 pgs: 97 active+clean; 77 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 2.9 MiB/s wr, 173 op/s 2024-03-16T12:03:24.624 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:03:24 smithi152 ceph-mon[36291]: pgmap v396: 97 pgs: 97 active+clean; 77 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 2.9 MiB/s wr, 173 op/s 2024-03-16T12:03:24.805 INFO:tasks.workunit.client.0.smithi029.stdout: 1 1199276 9.17 MB/sec execute 549 sec latency 56.912 ms 2024-03-16T12:03:25.805 INFO:tasks.workunit.client.0.smithi029.stdout: 1 1200688 9.17 MB/sec execute 550 sec latency 19.079 ms 2024-03-16T12:03:26.542 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:03:26 smithi154 ceph-mon[36320]: pgmap v397: 97 pgs: 97 active+clean; 77 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 2.9 MiB/s wr, 173 op/s 2024-03-16T12:03:26.584 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:03:26 smithi029 ceph-mon[31975]: pgmap v397: 97 pgs: 97 active+clean; 77 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 2.9 MiB/s wr, 173 op/s 2024-03-16T12:03:26.624 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:03:26 smithi152 ceph-mon[36291]: pgmap v397: 97 pgs: 97 active+clean; 77 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 2.9 MiB/s wr, 173 op/s 2024-03-16T12:03:26.805 INFO:tasks.workunit.client.0.smithi029.stdout: 1 1201874 9.17 MB/sec execute 551 sec latency 89.638 ms 2024-03-16T12:03:27.805 INFO:tasks.workunit.client.0.smithi029.stdout: 1 1203842 9.17 MB/sec execute 552 sec latency 21.278 ms 2024-03-16T12:03:28.386 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:03:28 smithi029 ceph-mon[31975]: pgmap v398: 97 pgs: 97 active+clean; 70 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 2.9 MiB/s wr, 249 op/s 2024-03-16T12:03:28.542 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:03:28 smithi154 ceph-mon[36320]: pgmap v398: 97 pgs: 97 active+clean; 70 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 2.9 MiB/s wr, 249 op/s 2024-03-16T12:03:28.624 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:03:28 smithi152 ceph-mon[36291]: pgmap v398: 97 pgs: 97 active+clean; 70 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 2.9 MiB/s wr, 249 op/s 2024-03-16T12:03:28.805 INFO:tasks.workunit.client.0.smithi029.stdout: 1 1204742 9.16 MB/sec execute 553 sec latency 24.281 ms 2024-03-16T12:03:29.805 INFO:tasks.workunit.client.0.smithi029.stdout: 1 1206511 9.16 MB/sec execute 554 sec latency 23.861 ms 2024-03-16T12:03:30.542 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:03:30 smithi154 ceph-mon[36320]: pgmap v399: 97 pgs: 97 active+clean; 70 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 2.4 MiB/s wr, 209 op/s 2024-03-16T12:03:30.584 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:03:30 smithi029 ceph-mon[31975]: pgmap v399: 97 pgs: 97 active+clean; 70 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 2.4 MiB/s wr, 209 op/s 2024-03-16T12:03:30.624 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:03:30 smithi152 ceph-mon[36291]: pgmap v399: 97 pgs: 97 active+clean; 70 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 2.4 MiB/s wr, 209 op/s 2024-03-16T12:03:30.805 INFO:tasks.workunit.client.0.smithi029.stdout: 1 1207933 9.16 MB/sec execute 555 sec latency 16.926 ms 2024-03-16T12:03:31.805 INFO:tasks.workunit.client.0.smithi029.stdout: 1 1209231 9.15 MB/sec execute 556 sec latency 55.578 ms 2024-03-16T12:03:32.542 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:03:32 smithi154 ceph-mon[36320]: pgmap v400: 97 pgs: 97 active+clean; 71 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 3.4 MiB/s wr, 273 op/s 2024-03-16T12:03:32.584 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:03:32 smithi029 ceph-mon[31975]: pgmap v400: 97 pgs: 97 active+clean; 71 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 3.4 MiB/s wr, 273 op/s 2024-03-16T12:03:32.624 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:03:32 smithi152 ceph-mon[36291]: pgmap v400: 97 pgs: 97 active+clean; 71 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 3.4 MiB/s wr, 273 op/s 2024-03-16T12:03:32.805 INFO:tasks.workunit.client.0.smithi029.stdout: 1 1211120 9.16 MB/sec execute 557 sec latency 66.201 ms 2024-03-16T12:03:33.806 INFO:tasks.workunit.client.0.smithi029.stdout: 1 1211980 9.15 MB/sec execute 558 sec latency 26.456 ms 2024-03-16T12:03:34.542 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:03:34 smithi154 ceph-mon[36320]: pgmap v401: 97 pgs: 97 active+clean; 71 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.3 MiB/s wr, 207 op/s 2024-03-16T12:03:34.584 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:03:34 smithi029 ceph-mon[31975]: pgmap v401: 97 pgs: 97 active+clean; 71 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.3 MiB/s wr, 207 op/s 2024-03-16T12:03:34.624 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:03:34 smithi152 ceph-mon[36291]: pgmap v401: 97 pgs: 97 active+clean; 71 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.3 MiB/s wr, 207 op/s 2024-03-16T12:03:34.806 INFO:tasks.workunit.client.0.smithi029.stdout: 1 1213996 9.15 MB/sec execute 559 sec latency 20.211 ms 2024-03-16T12:03:35.656 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.660+0000 7fce5771b640 1 -- 172.21.15.29:0/2229504739 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fce501006a0 msgr2=0x7fce50100b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:03:35.657 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.660+0000 7fce5771b640 1 --2- 172.21.15.29:0/2229504739 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fce501006a0 0x7fce50100b20 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7fce40009930 tx=0x7fce4002bc60 comp rx=0 tx=0).stop 2024-03-16T12:03:35.657 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.660+0000 7fce5771b640 1 -- 172.21.15.29:0/2229504739 shutdown_connections 2024-03-16T12:03:35.657 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.660+0000 7fce5771b640 1 --2- 172.21.15.29:0/2229504739 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fce50101060 0x7fce5010d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:03:35.657 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.660+0000 7fce5771b640 1 --2- 172.21.15.29:0/2229504739 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fce501006a0 0x7fce50100b20 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:03:35.657 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.660+0000 7fce5771b640 1 --2- 172.21.15.29:0/2229504739 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fce501066b0 0x7fce50106a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:03:35.657 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.660+0000 7fce5771b640 1 -- 172.21.15.29:0/2229504739 >> 172.21.15.29:0/2229504739 conn(0x7fce50077810 msgr2=0x7fce50077c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:03:35.657 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.661+0000 7fce5771b640 1 -- 172.21.15.29:0/2229504739 shutdown_connections 2024-03-16T12:03:35.657 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.661+0000 7fce5771b640 1 -- 172.21.15.29:0/2229504739 wait complete. 2024-03-16T12:03:35.659 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.661+0000 7fce5771b640 1 Processor -- start 2024-03-16T12:03:35.659 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.661+0000 7fce5771b640 1 -- start start 2024-03-16T12:03:35.659 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.662+0000 7fce5771b640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fce501006a0 0x7fce50195da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:03:35.659 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.662+0000 7fce5771b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fce50101060 0x7fce501962e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:03:35.659 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.662+0000 7fce5771b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fce501066b0 0x7fce5019a660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:03:35.659 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.662+0000 7fce5771b640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fce5010fa00 con 0x7fce501066b0 2024-03-16T12:03:35.659 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.662+0000 7fce5771b640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fce5010f880 con 0x7fce501006a0 2024-03-16T12:03:35.659 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.662+0000 7fce5771b640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fce5010fb80 con 0x7fce50101060 2024-03-16T12:03:35.659 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.662+0000 7fce55cb8640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fce501066b0 0x7fce5019a660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:03:35.659 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.662+0000 7fce54cb6640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fce50101060 0x7fce501962e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:03:35.659 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.662+0000 7fce55cb8640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fce501066b0 0x7fce5019a660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58378/0 (socket says 172.21.15.29:58378) 2024-03-16T12:03:35.659 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.662+0000 7fce55cb8640 1 -- 172.21.15.29:0/2916736363 learned_addr learned my addr 172.21.15.29:0/2916736363 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T12:03:35.659 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.662+0000 7fce54cb6640 1 -- 172.21.15.29:0/2916736363 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fce501006a0 msgr2=0x7fce50195da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:03:35.659 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.662+0000 7fce54cb6640 1 --2- 172.21.15.29:0/2916736363 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fce501006a0 0x7fce50195da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:03:35.660 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.662+0000 7fce54cb6640 1 -- 172.21.15.29:0/2916736363 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fce501066b0 msgr2=0x7fce5019a660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:03:35.660 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.662+0000 7fce54cb6640 1 --2- 172.21.15.29:0/2916736363 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fce501066b0 0x7fce5019a660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:03:35.660 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.662+0000 7fce54cb6640 1 -- 172.21.15.29:0/2916736363 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce5019ad90 con 0x7fce50101060 2024-03-16T12:03:35.660 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.662+0000 7fce55cb8640 1 --2- 172.21.15.29:0/2916736363 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fce501066b0 0x7fce5019a660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:03:35.660 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.663+0000 7fce54cb6640 1 --2- 172.21.15.29:0/2916736363 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fce50101060 0x7fce501962e0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fce4002c1a0 tx=0x7fce400045e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:03:35.661 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.663+0000 7fce467fc640 1 -- 172.21.15.29:0/2916736363 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce40041070 con 0x7fce50101060 2024-03-16T12:03:35.661 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.663+0000 7fce5771b640 1 -- 172.21.15.29:0/2916736363 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fce5019b020 con 0x7fce50101060 2024-03-16T12:03:35.661 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.663+0000 7fce467fc640 1 -- 172.21.15.29:0/2916736363 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fce4002cd40 con 0x7fce50101060 2024-03-16T12:03:35.661 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.663+0000 7fce5771b640 1 -- 172.21.15.29:0/2916736363 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fce501a2910 con 0x7fce50101060 2024-03-16T12:03:35.661 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.663+0000 7fce467fc640 1 -- 172.21.15.29:0/2916736363 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce4004a060 con 0x7fce50101060 2024-03-16T12:03:35.664 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.666+0000 7fce5a3e8740 1 Processor -- start 2024-03-16T12:03:35.665 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.666+0000 7fce5a3e8740 1 -- start start 2024-03-16T12:03:35.665 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.666+0000 7fce467fc640 1 -- 172.21.15.29:0/2916736363 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7fce40003bf0 con 0x7fce50101060 2024-03-16T12:03:35.665 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.666+0000 7fce5a3e8740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562051e91130 0x562051e91510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:03:35.665 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.666+0000 7fce5a3e8740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562051e91a50 0x562051e9a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:03:35.665 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.666+0000 7fce5a3e8740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562051e9a820 0x562051e9cc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:03:35.665 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.666+0000 7fce5a3e8740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x562051df5a40 con 0x562051e91130 2024-03-16T12:03:35.665 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.666+0000 7fce5a3e8740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x562051d9e920 con 0x562051e9a820 2024-03-16T12:03:35.665 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.666+0000 7fce5a3e8740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x562051df1ae0 con 0x562051e91a50 2024-03-16T12:03:35.665 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.666+0000 7fce55cb8640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562051e91130 0x562051e91510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:03:35.665 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.667+0000 7fce554b7640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562051e91a50 0x562051e9a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:03:35.665 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.667+0000 7fce554b7640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562051e91a50 0x562051e9a2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.29:57732/0 (socket says 172.21.15.29:57732) 2024-03-16T12:03:35.665 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.667+0000 7fce55cb8640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562051e91130 0x562051e91510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58380/0 (socket says 172.21.15.29:58380) 2024-03-16T12:03:35.665 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.667+0000 7fce554b7640 1 -- 172.21.15.29:0/3093464496 learned_addr learned my addr 172.21.15.29:0/3093464496 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T12:03:35.666 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.667+0000 7fce54cb6640 1 --2- 172.21.15.29:0/3093464496 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562051e9a820 0x562051e9cc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:03:35.666 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.667+0000 7fce554b7640 1 -- 172.21.15.29:0/3093464496 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562051e9a820 msgr2=0x562051e9cc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:03:35.666 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.667+0000 7fce554b7640 1 --2- 172.21.15.29:0/3093464496 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562051e9a820 0x562051e9cc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:03:35.666 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.667+0000 7fce554b7640 1 -- 172.21.15.29:0/3093464496 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562051e91130 msgr2=0x562051e91510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:03:35.666 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.667+0000 7fce554b7640 1 --2- 172.21.15.29:0/3093464496 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562051e91130 0x562051e91510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:03:35.666 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.667+0000 7fce554b7640 1 -- 172.21.15.29:0/3093464496 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562051db6460 con 0x562051e91a50 2024-03-16T12:03:35.666 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.667+0000 7fce467fc640 1 --2- 172.21.15.29:0/2916736363 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7fce3c0a4c20 0x7fce3c0a70e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:03:35.666 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.667+0000 7fce554b7640 1 --2- 172.21.15.29:0/3093464496 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562051e91a50 0x562051e9a2e0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fce4c009ae0 tx=0x7fce4c00e350 comp rx=0 tx=0).ready entity=mon.1 client_cookie=64f01dde0cb2ebf4 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:03:35.666 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.667+0000 7fce55cb8640 1 --2- 172.21.15.29:0/2916736363 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7fce3c0a4c20 0x7fce3c0a70e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:03:35.666 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.667+0000 7fce467fc640 1 -- 172.21.15.29:0/2916736363 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 8556+0+0 (secure 0 0 0) 0x7fce400f0ac0 con 0x7fce50101060 2024-03-16T12:03:35.666 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.667+0000 7fce3b7fe640 1 -- 172.21.15.29:0/3093464496 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce4c00ed20 con 0x562051e91a50 2024-03-16T12:03:35.666 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.667+0000 7fce3b7fe640 1 -- 172.21.15.29:0/3093464496 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fce4c0035f0 con 0x562051e91a50 2024-03-16T12:03:35.666 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.667+0000 7fce3b7fe640 1 -- 172.21.15.29:0/3093464496 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce4c010690 con 0x562051e91a50 2024-03-16T12:03:35.667 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.668+0000 7fce5a3e8740 1 -- 172.21.15.29:0/3093464496 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562051e91a50 msgr2=0x562051e9a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:03:35.667 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.668+0000 7fce55cb8640 1 --2- 172.21.15.29:0/2916736363 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7fce3c0a4c20 0x7fce3c0a70e0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fce48009270 tx=0x7fce48009680 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:03:35.667 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.668+0000 7fce5a3e8740 1 --2- 172.21.15.29:0/3093464496 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562051e91a50 0x562051e9a2e0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fce4c009ae0 tx=0x7fce4c00e350 comp rx=0 tx=0).stop 2024-03-16T12:03:35.667 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.668+0000 7fce5a3e8740 1 -- 172.21.15.29:0/3093464496 shutdown_connections 2024-03-16T12:03:35.667 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.668+0000 7fce5a3e8740 1 --2- 172.21.15.29:0/3093464496 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562051e9a820 0x562051e9cc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:03:35.667 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.668+0000 7fce5a3e8740 1 --2- 172.21.15.29:0/3093464496 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562051e91a50 0x562051e9a2e0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:03:35.667 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.668+0000 7fce5a3e8740 1 --2- 172.21.15.29:0/3093464496 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562051e91130 0x562051e91510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:03:35.667 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.668+0000 7fce5a3e8740 1 -- 172.21.15.29:0/3093464496 >> 172.21.15.29:0/3093464496 conn(0x562051e8ba50 msgr2=0x562051e8be30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:03:35.667 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.668+0000 7fce5a3e8740 1 -- 172.21.15.29:0/3093464496 shutdown_connections 2024-03-16T12:03:35.667 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.668+0000 7fce5a3e8740 1 -- 172.21.15.29:0/3093464496 wait complete. 2024-03-16T12:03:35.667 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.669+0000 7fce5a3e8740 1 -- start start 2024-03-16T12:03:35.667 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.669+0000 7fce5a3e8740 1 Processor -- start 2024-03-16T12:03:35.668 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.669+0000 7fce5a3e8740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562051e91130 0x562051e97860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:03:35.669 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.669+0000 7fce5a3e8740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562051e91a50 0x562051e97da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:03:35.669 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.669+0000 7fce5a3e8740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562051e9a820 0x562051e982e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:03:35.669 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.669+0000 7fce5a3e8740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x562051e9d450 con 0x562051e91a50 2024-03-16T12:03:35.669 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.669+0000 7fce5a3e8740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x562051db1d70 con 0x562051e9a820 2024-03-16T12:03:35.669 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.669+0000 7fce5a3e8740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x562051e9d5d0 con 0x562051e91130 2024-03-16T12:03:35.669 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.669+0000 7fce5a3e8740 10 client.0 _ll_register_callbacks cb 0x562051b09a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T12:03:35.669 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.670+0000 7fce554b7640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562051e91130 0x562051e97860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:03:35.669 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.670+0000 7fce554b7640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562051e91130 0x562051e97860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.29:57738/0 (socket says 172.21.15.29:57738) 2024-03-16T12:03:35.669 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.670+0000 7fce554b7640 1 -- 172.21.15.29:0/4004834193 learned_addr learned my addr 172.21.15.29:0/4004834193 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T12:03:35.669 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.670+0000 7fce54cb6640 1 --2- 172.21.15.29:0/4004834193 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562051e91a50 0x562051e97da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:03:35.669 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.670+0000 7fce55cb8640 1 --2- 172.21.15.29:0/4004834193 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562051e9a820 0x562051e982e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:03:35.669 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.672+0000 7fce54cb6640 1 -- 172.21.15.29:0/4004834193 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562051e91130 msgr2=0x562051e97860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:03:35.669 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.672+0000 7fce54cb6640 1 --2- 172.21.15.29:0/4004834193 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562051e91130 0x562051e97860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:03:35.669 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.672+0000 7fce54cb6640 1 -- 172.21.15.29:0/4004834193 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562051e9a820 msgr2=0x562051e982e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:03:35.670 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.672+0000 7fce54cb6640 1 --2- 172.21.15.29:0/4004834193 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562051e9a820 0x562051e982e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:03:35.670 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.672+0000 7fce54cb6640 1 -- 172.21.15.29:0/4004834193 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562051b0a210 con 0x562051e91a50 2024-03-16T12:03:35.670 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.672+0000 7fce554b7640 1 --2- 172.21.15.29:0/4004834193 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562051e91130 0x562051e97860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:03:35.670 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.672+0000 7fce55cb8640 1 --2- 172.21.15.29:0/4004834193 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562051e9a820 0x562051e982e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:03:35.670 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.673+0000 7fce54cb6640 1 --2- 172.21.15.29:0/4004834193 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562051e91a50 0x562051e97da0 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7fce400ee2d0 tx=0x7fce400b94b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47ced629996274c0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:03:35.670 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.673+0000 7fce38ff9640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-16T12:03:35.671 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.673+0000 7fce38ff9640 1 -- 172.21.15.29:0/4004834193 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce400bcb70 con 0x562051e91a50 2024-03-16T12:03:35.671 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.673+0000 7fce38ff9640 1 -- 172.21.15.29:0/4004834193 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(107..107 src has 1..107) ==== 8556+0+0 (secure 0 0 0) 0x7fce400bd470 con 0x562051e91a50 2024-03-16T12:03:35.671 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.673+0000 7fce5a3e8740 1 -- 172.21.15.29:0/4004834193 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fce4c0035f0 con 0x562051e91a50 2024-03-16T12:03:35.671 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.673+0000 7fce38ff9640 1 -- 172.21.15.29:0/4004834193 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fce400035f0 con 0x562051e91a50 2024-03-16T12:03:35.671 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.673+0000 7fce5a3e8740 10 client.16455 fetch_fsmap learned FSMap version 29 2024-03-16T12:03:35.671 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.673+0000 7fce5a3e8740 1 -- 172.21.15.29:0/4004834193 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x562051bd6620 con 0x562051e91a50 2024-03-16T12:03:35.671 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.673+0000 7fce38ff9640 1 -- 172.21.15.29:0/4004834193 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fce4c0035f0 con 0x562051e91a50 2024-03-16T12:03:35.671 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.674+0000 7fce38ff9640 1 -- 172.21.15.29:0/4004834193 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 29) ==== 8338+0+0 (secure 0 0 0) 0x7fce400c5520 con 0x562051e91a50 2024-03-16T12:03:35.671 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.674+0000 7fce5a3e8740 10 client.16455 fetch_fsmap finished waiting for FSMap version 29 2024-03-16T12:03:35.671 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.674+0000 7fce5a3e8740 10 client.16455 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T12:03:35.671 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.674+0000 7fce5a3e8740 20 client.16455 populate_metadata read hostname 'smithi029' 2024-03-16T12:03:35.672 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.674+0000 7fce5a3e8740 1 --2- 172.21.15.29:0/4004834193 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x562051eaa8c0 0x562051eacd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:03:35.672 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.674+0000 7fce5a3e8740 4 client.16455 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T12:03:35.672 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.674+0000 7fce5a3e8740 1 -- 172.21.15.29:0/4004834193 --> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562051e02f80 con 0x562051eaa8c0 2024-03-16T12:03:35.672 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.675+0000 7fce554b7640 1 --2- 172.21.15.29:0/4004834193 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x562051eaa8c0 0x562051eacd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:03:35.672 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.675+0000 7fce38ff9640 10 client.16455 ms_handle_connect on v2:172.21.15.152:6838/1548731833 2024-03-16T12:03:35.672 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.675+0000 7fce554b7640 1 --2- 172.21.15.29:0/4004834193 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x562051eaa8c0 0x562051eacd00 crc :-1 s=READY pgs=152 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3a5b1156f833474e server_cookie=570acd65b03bf15e in_seq=0 out_seq=0 2024-03-16T12:03:35.675 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.679+0000 7fce38ff9640 1 -- 172.21.15.29:0/4004834193 <== mds.0 v2:172.21.15.152:6838/1548731833 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x562051e02f80 con 0x562051eaa8c0 2024-03-16T12:03:35.675 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.679+0000 7fce38ff9640 10 client.16455 handle_command_reply: tid=0 2024-03-16T12:03:35.675 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.679+0000 7fce5a3e8740 1 client.16455 shutdown 2024-03-16T12:03:35.675 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.679+0000 7fce5a3e8740 1 -- 172.21.15.29:0/4004834193 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562051e91a50 msgr2=0x562051e97da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:03:35.676 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.679+0000 7fce5a3e8740 1 --2- 172.21.15.29:0/4004834193 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562051e91a50 0x562051e97da0 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7fce400ee2d0 tx=0x7fce400b94b0 comp rx=0 tx=0).stop 2024-03-16T12:03:35.676 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.679+0000 7fce5a3e8740 1 -- 172.21.15.29:0/4004834193 shutdown_connections 2024-03-16T12:03:35.676 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.679+0000 7fce5a3e8740 1 --2- 172.21.15.29:0/4004834193 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x562051eaa8c0 0x562051eacd00 crc :-1 s=READY pgs=152 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:03:35.676 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.679+0000 7fce5a3e8740 1 --2- 172.21.15.29:0/4004834193 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562051e9a820 0x562051e982e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:03:35.676 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.679+0000 7fce5a3e8740 1 --2- 172.21.15.29:0/4004834193 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562051e91a50 0x562051e97da0 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:03:35.676 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.679+0000 7fce38ff9640 0 client.16455 ms_handle_reset on v2:172.21.15.152:6838/1548731833 2024-03-16T12:03:35.676 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.679+0000 7fce5a3e8740 1 --2- 172.21.15.29:0/4004834193 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562051e91130 0x562051e97860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:03:35.676 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.679+0000 7fce5a3e8740 1 -- 172.21.15.29:0/4004834193 >> 172.21.15.29:0/4004834193 conn(0x562051e8ba50 msgr2=0x562051e8d7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:03:35.676 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.679+0000 7fce5a3e8740 1 -- 172.21.15.29:0/4004834193 shutdown_connections 2024-03-16T12:03:35.676 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.679+0000 7fce5a3e8740 1 -- 172.21.15.29:0/4004834193 wait complete. 2024-03-16T12:03:35.676 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.679+0000 7fce5a3e8740 20 client.16455 trim_cache size 0 max 16384 2024-03-16T12:03:35.692 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.695+0000 7fce5a3e8740 1 Processor -- start 2024-03-16T12:03:35.692 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.695+0000 7fce5a3e8740 1 -- start start 2024-03-16T12:03:35.692 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.695+0000 7fce5a3e8740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562051e91130 0x562051ea61f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:03:35.693 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.695+0000 7fce5a3e8740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562051e91a50 0x562051eb2610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:03:35.693 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.695+0000 7fce5a3e8740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562051e9a820 0x562051ea2810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:03:35.693 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.695+0000 7fce5a3e8740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fce501adb90 con 0x562051e91a50 2024-03-16T12:03:35.693 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.695+0000 7fce5a3e8740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fce501ab460 con 0x562051e9a820 2024-03-16T12:03:35.693 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.695+0000 7fce5a3e8740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fce501a92d0 con 0x562051e91130 2024-03-16T12:03:35.693 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.695+0000 7fce54cb6640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562051e91a50 0x562051eb2610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:03:35.693 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.696+0000 7fce54cb6640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562051e91a50 0x562051eb2610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58408/0 (socket says 172.21.15.29:58408) 2024-03-16T12:03:35.693 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.696+0000 7fce54cb6640 1 -- 172.21.15.29:0/2039912244 learned_addr learned my addr 172.21.15.29:0/2039912244 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T12:03:35.693 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.696+0000 7fce554b7640 1 --2- 172.21.15.29:0/2039912244 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562051e91130 0x562051ea61f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:03:35.693 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.696+0000 7fce55cb8640 1 --2- 172.21.15.29:0/2039912244 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562051e9a820 0x562051ea2810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:03:35.693 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.696+0000 7fce54cb6640 1 -- 172.21.15.29:0/2039912244 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562051e91130 msgr2=0x562051ea61f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:03:35.693 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.696+0000 7fce54cb6640 1 --2- 172.21.15.29:0/2039912244 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562051e91130 0x562051ea61f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:03:35.693 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.696+0000 7fce54cb6640 1 -- 172.21.15.29:0/2039912244 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562051e9a820 msgr2=0x562051ea2810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:03:35.693 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.696+0000 7fce54cb6640 1 --2- 172.21.15.29:0/2039912244 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562051e9a820 0x562051ea2810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:03:35.693 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.696+0000 7fce54cb6640 1 -- 172.21.15.29:0/2039912244 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562051ead380 con 0x562051e91a50 2024-03-16T12:03:35.694 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.696+0000 7fce55cb8640 1 --2- 172.21.15.29:0/2039912244 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562051e9a820 0x562051ea2810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T12:03:35.694 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.696+0000 7fce54cb6640 1 --2- 172.21.15.29:0/2039912244 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562051e91a50 0x562051eb2610 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7fce40009340 tx=0x7fce400ee300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12b67443463c7adc server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:03:35.694 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.696+0000 7fce3affd640 1 -- 172.21.15.29:0/2039912244 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce400bcb70 con 0x562051e91a50 2024-03-16T12:03:35.694 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.696+0000 7fce3affd640 1 -- 172.21.15.29:0/2039912244 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fce400bd910 con 0x562051e91a50 2024-03-16T12:03:35.694 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.696+0000 7fce3affd640 1 -- 172.21.15.29:0/2039912244 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce400c1360 con 0x562051e91a50 2024-03-16T12:03:35.698 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.697+0000 7fce5a3e8740 1 -- 172.21.15.29:0/2039912244 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562051e91a50 msgr2=0x562051eb2610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:03:35.698 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.697+0000 7fce5a3e8740 1 --2- 172.21.15.29:0/2039912244 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562051e91a50 0x562051eb2610 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7fce40009340 tx=0x7fce400ee300 comp rx=0 tx=0).stop 2024-03-16T12:03:35.698 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.697+0000 7fce5a3e8740 1 -- 172.21.15.29:0/2039912244 shutdown_connections 2024-03-16T12:03:35.698 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.697+0000 7fce5a3e8740 1 --2- 172.21.15.29:0/2039912244 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562051e9a820 0x562051ea2810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:03:35.698 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.697+0000 7fce5a3e8740 1 --2- 172.21.15.29:0/2039912244 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562051e91a50 0x562051eb2610 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:03:35.698 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.697+0000 7fce5a3e8740 1 --2- 172.21.15.29:0/2039912244 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562051e91130 0x562051ea61f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:03:35.698 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.697+0000 7fce5a3e8740 1 -- 172.21.15.29:0/2039912244 >> 172.21.15.29:0/2039912244 conn(0x562051e8ba50 msgr2=0x562051e97820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:03:35.698 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.697+0000 7fce5a3e8740 1 -- 172.21.15.29:0/2039912244 shutdown_connections 2024-03-16T12:03:35.698 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.697+0000 7fce5a3e8740 1 -- 172.21.15.29:0/2039912244 wait complete. 2024-03-16T12:03:35.698 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.698+0000 7fce5a3e8740 1 -- start start 2024-03-16T12:03:35.698 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.698+0000 7fce5a3e8740 1 Processor -- start 2024-03-16T12:03:35.698 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.698+0000 7fce5a3e8740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562051e91130 0x562051ea2420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:03:35.699 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.698+0000 7fce5a3e8740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562051e91a50 0x562051ea2960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:03:35.699 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.698+0000 7fce5a3e8740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562051e9a820 0x562051e9e8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:03:35.699 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.698+0000 7fce5a3e8740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fce501a9920 con 0x562051e91130 2024-03-16T12:03:35.699 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.698+0000 7fce5a3e8740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fce501aad90 con 0x562051e91a50 2024-03-16T12:03:35.699 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.698+0000 7fce5a3e8740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fce501a9ff0 con 0x562051e9a820 2024-03-16T12:03:35.699 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.698+0000 7fce5a3e8740 10 client.0 _ll_register_callbacks cb 0x7fce400f4140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T12:03:35.699 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.698+0000 7fce554b7640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562051e91130 0x562051ea2420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:03:35.699 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.698+0000 7fce554b7640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562051e91130 0x562051ea2420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58424/0 (socket says 172.21.15.29:58424) 2024-03-16T12:03:35.699 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.698+0000 7fce554b7640 1 -- 172.21.15.29:0/2336410641 learned_addr learned my addr 172.21.15.29:0/2336410641 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T12:03:35.699 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.698+0000 7fce54cb6640 1 --2- 172.21.15.29:0/2336410641 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562051e91a50 0x562051ea2960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:03:35.699 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.698+0000 7fce55cb8640 1 --2- 172.21.15.29:0/2336410641 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562051e9a820 0x562051e9e8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:03:35.699 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.699+0000 7fce554b7640 1 -- 172.21.15.29:0/2336410641 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562051e9a820 msgr2=0x562051e9e8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:03:35.699 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.699+0000 7fce554b7640 1 --2- 172.21.15.29:0/2336410641 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562051e9a820 0x562051e9e8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:03:35.699 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.699+0000 7fce554b7640 1 -- 172.21.15.29:0/2336410641 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562051e91a50 msgr2=0x562051ea2960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:03:35.700 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.699+0000 7fce554b7640 1 --2- 172.21.15.29:0/2336410641 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562051e91a50 0x562051ea2960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:03:35.700 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.699+0000 7fce554b7640 1 -- 172.21.15.29:0/2336410641 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562051e9f080 con 0x562051e91130 2024-03-16T12:03:35.700 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.699+0000 7fce55cb8640 1 --2- 172.21.15.29:0/2336410641 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562051e9a820 0x562051e9e8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:03:35.700 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.699+0000 7fce554b7640 1 --2- 172.21.15.29:0/2336410641 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562051e91130 0x562051ea2420 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7fce4c000c00 tx=0x7fce4c0097d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c30c01b38671006 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:03:35.700 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.699+0000 7fce38ff9640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-16T12:03:35.700 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.699+0000 7fce38ff9640 1 -- 172.21.15.29:0/2336410641 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce4c01d040 con 0x562051e91130 2024-03-16T12:03:35.700 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.699+0000 7fce38ff9640 1 -- 172.21.15.29:0/2336410641 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(107..107 src has 1..107) ==== 8556+0+0 (secure 0 0 0) 0x7fce4c0293e0 con 0x562051e91130 2024-03-16T12:03:35.700 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.699+0000 7fce5a3e8740 1 -- 172.21.15.29:0/2336410641 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fce400bd910 con 0x562051e91130 2024-03-16T12:03:35.700 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.699+0000 7fce38ff9640 1 -- 172.21.15.29:0/2336410641 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fce4c039050 con 0x562051e91130 2024-03-16T12:03:35.700 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.699+0000 7fce5a3e8740 10 client.16461 fetch_fsmap learned FSMap version 29 2024-03-16T12:03:35.700 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.699+0000 7fce5a3e8740 1 -- 172.21.15.29:0/2336410641 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x562051f116e0 con 0x562051e91130 2024-03-16T12:03:35.700 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.699+0000 7fce38ff9640 1 -- 172.21.15.29:0/2336410641 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fce400bd910 con 0x562051e91130 2024-03-16T12:03:35.700 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.699+0000 7fce38ff9640 1 -- 172.21.15.29:0/2336410641 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 29) ==== 8338+0+0 (secure 0 0 0) 0x7fce4c004810 con 0x562051e91130 2024-03-16T12:03:35.701 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.699+0000 7fce5a3e8740 10 client.16461 fetch_fsmap finished waiting for FSMap version 29 2024-03-16T12:03:35.701 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.699+0000 7fce5a3e8740 10 client.16461 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T12:03:35.701 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.699+0000 7fce5a3e8740 20 client.16461 populate_metadata read hostname 'smithi029' 2024-03-16T12:03:35.701 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.699+0000 7fce5a3e8740 1 --2- 172.21.15.29:0/2336410641 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x562051f11ad0 0x562051f13ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:03:35.701 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.699+0000 7fce5a3e8740 4 client.16461 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-16T12:03:35.701 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.699+0000 7fce5a3e8740 1 -- 172.21.15.29:0/2336410641 --> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] -- command(tid 0: {"prefix": "flush journal"}) -- 0x562051dfd2d0 con 0x562051f11ad0 2024-03-16T12:03:35.701 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.700+0000 7fce54cb6640 1 --2- 172.21.15.29:0/2336410641 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x562051f11ad0 0x562051f13ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:03:35.701 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.700+0000 7fce38ff9640 10 client.16461 ms_handle_connect on v2:172.21.15.152:6838/1548731833 2024-03-16T12:03:35.701 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.700+0000 7fce54cb6640 1 --2- 172.21.15.29:0/2336410641 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x562051f11ad0 0x562051f13ec0 crc :-1 s=READY pgs=153 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2b3da7170c53c14d server_cookie=a146fe2d729932a0 in_seq=0 out_seq=0 2024-03-16T12:03:35.806 INFO:tasks.workunit.client.0.smithi029.stdout: 1 1214957 9.15 MB/sec execute 560 sec latency 44.615 ms 2024-03-16T12:03:35.877 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.880+0000 7fce38ff9640 1 -- 172.21.15.29:0/2336410641 <== mds.0 v2:172.21.15.152:6838/1548731833 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x562051dfd2d0 con 0x562051f11ad0 2024-03-16T12:03:35.877 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.880+0000 7fce38ff9640 10 client.16461 handle_command_reply: tid=0 2024-03-16T12:03:35.877 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.880+0000 7fce5a3e8740 1 client.16461 shutdown 2024-03-16T12:03:35.877 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.880+0000 7fce5a3e8740 1 -- 172.21.15.29:0/2336410641 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562051e91130 msgr2=0x562051ea2420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:03:35.877 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.880+0000 7fce5a3e8740 1 --2- 172.21.15.29:0/2336410641 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562051e91130 0x562051ea2420 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7fce4c000c00 tx=0x7fce4c0097d0 comp rx=0 tx=0).stop 2024-03-16T12:03:35.878 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.881+0000 7fce5a3e8740 1 -- 172.21.15.29:0/2336410641 shutdown_connections 2024-03-16T12:03:35.878 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.881+0000 7fce5a3e8740 1 --2- 172.21.15.29:0/2336410641 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562051e9a820 0x562051e9e8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:03:35.878 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.881+0000 7fce5a3e8740 1 --2- 172.21.15.29:0/2336410641 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x562051f11ad0 0x562051f13ec0 crc :-1 s=READY pgs=153 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:03:35.878 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.881+0000 7fce5a3e8740 1 --2- 172.21.15.29:0/2336410641 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562051e91a50 0x562051ea2960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:03:35.878 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.881+0000 7fce5a3e8740 1 --2- 172.21.15.29:0/2336410641 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562051e91130 0x562051ea2420 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:03:35.878 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.881+0000 7fce5a3e8740 1 -- 172.21.15.29:0/2336410641 >> 172.21.15.29:0/2336410641 conn(0x562051e8ba50 msgr2=0x562051e980b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:03:35.878 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.881+0000 7fce38ff9640 0 client.16461 ms_handle_reset on v2:172.21.15.152:6838/1548731833 2024-03-16T12:03:35.878 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.881+0000 7fce5a3e8740 1 -- 172.21.15.29:0/2336410641 shutdown_connections 2024-03-16T12:03:35.878 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.881+0000 7fce5a3e8740 1 -- 172.21.15.29:0/2336410641 wait complete. 2024-03-16T12:03:35.878 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.881+0000 7fce5a3e8740 20 client.16461 trim_cache size 0 max 16384 2024-03-16T12:03:35.879 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:{ 2024-03-16T12:03:35.879 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "message": "", 2024-03-16T12:03:35.879 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "return_code": 0 2024-03-16T12:03:35.879 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:} 2024-03-16T12:03:35.879 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.881+0000 7fce5771b640 1 -- 172.21.15.29:0/2916736363 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7fce3c0a4c20 msgr2=0x7fce3c0a70e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:03:35.880 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.881+0000 7fce5771b640 1 --2- 172.21.15.29:0/2916736363 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7fce3c0a4c20 0x7fce3c0a70e0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fce48009270 tx=0x7fce48009680 comp rx=0 tx=0).stop 2024-03-16T12:03:35.880 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.881+0000 7fce5771b640 1 -- 172.21.15.29:0/2916736363 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fce50101060 msgr2=0x7fce501962e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:03:35.880 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.881+0000 7fce5771b640 1 --2- 172.21.15.29:0/2916736363 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fce50101060 0x7fce501962e0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fce4002c1a0 tx=0x7fce400045e0 comp rx=0 tx=0).stop 2024-03-16T12:03:35.880 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.881+0000 7fce5771b640 1 -- 172.21.15.29:0/2916736363 shutdown_connections 2024-03-16T12:03:35.880 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.881+0000 7fce5771b640 1 --2- 172.21.15.29:0/2916736363 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7fce3c0a4c20 0x7fce3c0a70e0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:03:35.880 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.881+0000 7fce5771b640 1 --2- 172.21.15.29:0/2916736363 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fce501066b0 0x7fce5019a660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:03:35.880 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.881+0000 7fce5771b640 1 --2- 172.21.15.29:0/2916736363 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fce50101060 0x7fce501962e0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:03:35.880 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.881+0000 7fce5771b640 1 --2- 172.21.15.29:0/2916736363 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fce501006a0 0x7fce50195da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:03:35.880 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.881+0000 7fce5771b640 1 -- 172.21.15.29:0/2916736363 >> 172.21.15.29:0/2916736363 conn(0x7fce50077810 msgr2=0x7fce50103170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:03:35.880 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.882+0000 7fce5771b640 1 -- 172.21.15.29:0/2916736363 shutdown_connections 2024-03-16T12:03:35.880 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:35.882+0000 7fce5771b640 1 -- 172.21.15.29:0/2916736363 wait complete. 2024-03-16T12:03:36.542 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:03:36 smithi154 ceph-mon[36320]: pgmap v402: 97 pgs: 97 active+clean; 71 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.2 MiB/s wr, 203 op/s 2024-03-16T12:03:36.542 INFO:journalctl@ceph.mds.c.smithi154.stdout:Mar 16 12:03:36 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mds-c[66727]: ignoring --setuser ceph since I am not root 2024-03-16T12:03:36.542 INFO:journalctl@ceph.mds.c.smithi154.stdout:Mar 16 12:03:36 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mds-c[66727]: ignoring --setgroup ceph since I am not root 2024-03-16T12:03:36.542 INFO:journalctl@ceph.mds.c.smithi154.stdout:Mar 16 12:03:36 smithi154 ceph-mds[66731]: -- 172.21.15.154:0/1678040786 <== mon.1 v2:172.21.15.154:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55897677da40 con 0x558976751000 2024-03-16T12:03:36.542 INFO:journalctl@ceph.mds.c.smithi154.stdout:Mar 16 12:03:36 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mds-c[66727]: starting mds.c at 2024-03-16T12:03:36.584 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:03:36 smithi029 ceph-mon[31975]: pgmap v402: 97 pgs: 97 active+clean; 71 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.2 MiB/s wr, 203 op/s 2024-03-16T12:03:36.624 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:03:36 smithi152 ceph-mon[36291]: pgmap v402: 97 pgs: 97 active+clean; 71 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.2 MiB/s wr, 203 op/s 2024-03-16T12:03:36.806 INFO:tasks.workunit.client.0.smithi029.stdout: 1 1216310 9.14 MB/sec execute 561 sec latency 89.269 ms 2024-03-16T12:03:37.542 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:03:37 smithi154 ceph-mon[36320]: daemon mds.c restarted 2024-03-16T12:03:37.542 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:03:37 smithi154 ceph-mon[36320]: osdmap e108: 12 total, 12 up, 12 in 2024-03-16T12:03:37.542 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:03:37 smithi154 ceph-mon[36320]: mds.? [v2:172.21.15.154:6836/1624851783,v1:172.21.15.154:6837/1624851783] up:boot 2024-03-16T12:03:37.542 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:03:37 smithi154 ceph-mon[36320]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-03-16T12:03:37.542 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:03:37 smithi154 ceph-mon[36320]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-16T12:03:37.584 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:03:37 smithi029 ceph-mon[31975]: daemon mds.c restarted 2024-03-16T12:03:37.585 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:03:37 smithi029 ceph-mon[31975]: osdmap e108: 12 total, 12 up, 12 in 2024-03-16T12:03:37.585 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:03:37 smithi029 ceph-mon[31975]: mds.? [v2:172.21.15.154:6836/1624851783,v1:172.21.15.154:6837/1624851783] up:boot 2024-03-16T12:03:37.585 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:03:37 smithi029 ceph-mon[31975]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-03-16T12:03:37.585 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:03:37 smithi029 ceph-mon[31975]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-16T12:03:37.624 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:03:37 smithi152 ceph-mon[36291]: daemon mds.c restarted 2024-03-16T12:03:37.624 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:03:37 smithi152 ceph-mon[36291]: osdmap e108: 12 total, 12 up, 12 in 2024-03-16T12:03:37.624 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:03:37 smithi152 ceph-mon[36291]: mds.? [v2:172.21.15.154:6836/1624851783,v1:172.21.15.154:6837/1624851783] up:boot 2024-03-16T12:03:37.624 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:03:37 smithi152 ceph-mon[36291]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-03-16T12:03:37.624 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:03:37 smithi152 ceph-mon[36291]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-16T12:03:37.806 INFO:tasks.workunit.client.0.smithi029.stdout: 1 1218279 9.15 MB/sec execute 562 sec latency 56.382 ms 2024-03-16T12:03:38.542 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:03:38 smithi154 ceph-mon[36320]: pgmap v403: 97 pgs: 97 active+clean; 60 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 3.2 MiB/s wr, 336 op/s 2024-03-16T12:03:38.584 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:03:38 smithi029 ceph-mon[31975]: pgmap v403: 97 pgs: 97 active+clean; 60 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 3.2 MiB/s wr, 336 op/s 2024-03-16T12:03:38.624 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:03:38 smithi152 ceph-mon[36291]: pgmap v403: 97 pgs: 97 active+clean; 60 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 3.2 MiB/s wr, 336 op/s 2024-03-16T12:03:38.806 INFO:tasks.workunit.client.0.smithi029.stdout: 1 1219147 9.13 MB/sec execute 563 sec latency 20.033 ms 2024-03-16T12:03:39.806 INFO:tasks.workunit.client.0.smithi029.stdout: 1 1221405 9.14 MB/sec execute 564 sec latency 19.191 ms 2024-03-16T12:03:40.541 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:03:40 smithi154 ceph-mon[36320]: pgmap v405: 97 pgs: 97 active+clean; 60 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 2.5 MiB/s wr, 242 op/s 2024-03-16T12:03:40.584 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:03:40 smithi029 ceph-mon[31975]: pgmap v405: 97 pgs: 97 active+clean; 60 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 2.5 MiB/s wr, 242 op/s 2024-03-16T12:03:40.624 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:03:40 smithi152 ceph-mon[36291]: pgmap v405: 97 pgs: 97 active+clean; 60 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 2.5 MiB/s wr, 242 op/s 2024-03-16T12:03:40.806 INFO:tasks.workunit.client.0.smithi029.stdout: 1 1222311 9.13 MB/sec execute 565 sec latency 22.324 ms 2024-03-16T12:03:41.806 INFO:tasks.workunit.client.0.smithi029.stdout: 1 1224143 9.13 MB/sec execute 566 sec latency 21.815 ms 2024-03-16T12:03:42.541 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:03:42 smithi154 ceph-mon[36320]: pgmap v406: 97 pgs: 97 active+clean; 62 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 2.0 MiB/s wr, 210 op/s 2024-03-16T12:03:42.584 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:03:42 smithi029 ceph-mon[31975]: pgmap v406: 97 pgs: 97 active+clean; 62 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 2.0 MiB/s wr, 210 op/s 2024-03-16T12:03:42.624 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:03:42 smithi152 ceph-mon[36291]: pgmap v406: 97 pgs: 97 active+clean; 62 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 2.0 MiB/s wr, 210 op/s 2024-03-16T12:03:42.806 INFO:tasks.workunit.client.0.smithi029.stdout: 1 1225430 9.13 MB/sec execute 567 sec latency 61.676 ms 2024-03-16T12:03:43.806 INFO:tasks.workunit.client.0.smithi029.stdout: 1 1226773 9.13 MB/sec execute 568 sec latency 20.623 ms 2024-03-16T12:03:44.426 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:03:44 smithi152 ceph-mon[36291]: pgmap v407: 97 pgs: 97 active+clean; 66 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 2.4 MiB/s wr, 242 op/s 2024-03-16T12:03:44.541 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:03:44 smithi154 ceph-mon[36320]: pgmap v407: 97 pgs: 97 active+clean; 66 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 2.4 MiB/s wr, 242 op/s 2024-03-16T12:03:44.584 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:03:44 smithi029 ceph-mon[31975]: pgmap v407: 97 pgs: 97 active+clean; 66 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 2.4 MiB/s wr, 242 op/s 2024-03-16T12:03:44.806 INFO:tasks.workunit.client.0.smithi029.stdout: 1 1228852 9.13 MB/sec execute 569 sec latency 24.314 ms 2024-03-16T12:03:45.806 INFO:tasks.workunit.client.0.smithi029.stdout: 1 1229644 9.12 MB/sec execute 570 sec latency 24.828 ms 2024-03-16T12:03:46.541 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:03:46 smithi154 ceph-mon[36320]: pgmap v408: 97 pgs: 97 active+clean; 66 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 2.4 MiB/s wr, 242 op/s 2024-03-16T12:03:46.584 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:03:46 smithi029 ceph-mon[31975]: pgmap v408: 97 pgs: 97 active+clean; 66 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 2.4 MiB/s wr, 242 op/s 2024-03-16T12:03:46.623 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:03:46 smithi152 ceph-mon[36291]: pgmap v408: 97 pgs: 97 active+clean; 66 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 2.4 MiB/s wr, 242 op/s 2024-03-16T12:03:46.807 INFO:tasks.workunit.client.0.smithi029.stdout: 1 1231742 9.12 MB/sec execute 571 sec latency 20.534 ms 2024-03-16T12:03:47.806 INFO:tasks.workunit.client.0.smithi029.stdout: 1 1232592 9.12 MB/sec execute 572 sec latency 141.181 ms 2024-03-16T12:03:48.541 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:03:48 smithi154 ceph-mon[36320]: pgmap v409: 97 pgs: 97 active+clean; 71 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 2.5 MiB/s wr, 170 op/s 2024-03-16T12:03:48.584 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:03:48 smithi029 ceph-mon[31975]: pgmap v409: 97 pgs: 97 active+clean; 71 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 2.5 MiB/s wr, 170 op/s 2024-03-16T12:03:48.624 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:03:48 smithi152 ceph-mon[36291]: pgmap v409: 97 pgs: 97 active+clean; 71 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 2.5 MiB/s wr, 170 op/s 2024-03-16T12:03:48.806 INFO:tasks.workunit.client.0.smithi029.stdout: 1 1233922 9.11 MB/sec execute 573 sec latency 29.881 ms 2024-03-16T12:03:48.965 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:48.969+0000 7f2900ec0640 1 -- 172.21.15.29:0/3795887116 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f28fc1077b0 msgr2=0x7f28fc107b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:03:48.966 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:48.969+0000 7f2900ec0640 1 --2- 172.21.15.29:0/3795887116 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f28fc1077b0 0x7f28fc107b90 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f28ec009900 tx=0x7f28ec02bc30 comp rx=0 tx=0).stop 2024-03-16T12:03:48.966 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:48.969+0000 7f2900ec0640 1 -- 172.21.15.29:0/3795887116 shutdown_connections 2024-03-16T12:03:48.966 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:48.969+0000 7f2900ec0640 1 --2- 172.21.15.29:0/3795887116 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f28fc06a920 0x7f28fc10e600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:03:48.966 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:48.969+0000 7f2900ec0640 1 --2- 172.21.15.29:0/3795887116 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f28fc069f80 0x7f28fc06a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:03:48.966 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:48.969+0000 7f2900ec0640 1 --2- 172.21.15.29:0/3795887116 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f28fc1077b0 0x7f28fc107b90 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:03:48.966 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:48.969+0000 7f2900ec0640 1 -- 172.21.15.29:0/3795887116 >> 172.21.15.29:0/3795887116 conn(0x7f28fc0776c0 msgr2=0x7f28fc077ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:03:48.966 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:48.969+0000 7f2900ec0640 1 -- 172.21.15.29:0/3795887116 shutdown_connections 2024-03-16T12:03:48.966 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:48.969+0000 7f2900ec0640 1 -- 172.21.15.29:0/3795887116 wait complete. 2024-03-16T12:03:48.968 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:48.970+0000 7f2900ec0640 1 Processor -- start 2024-03-16T12:03:48.968 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:48.970+0000 7f2900ec0640 1 -- start start 2024-03-16T12:03:48.968 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:48.971+0000 7f2900ec0640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f28fc069f80 0x7f28fc197f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:03:48.969 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:48.971+0000 7f2900ec0640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f28fc06a920 0x7f28fc1984c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:03:48.969 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:48.971+0000 7f2900ec0640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f28fc1077b0 0x7f28fc19c840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:03:48.969 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:48.971+0000 7f2900ec0640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f28fc111ba0 con 0x7f28fc1077b0 2024-03-16T12:03:48.969 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:48.971+0000 7f2900ec0640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f28fc111a20 con 0x7f28fc06a920 2024-03-16T12:03:48.969 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:48.971+0000 7f2900ec0640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f28fc111d20 con 0x7f28fc069f80 2024-03-16T12:03:48.969 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:48.971+0000 7f28fad9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f28fc1077b0 0x7f28fc19c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:03:48.969 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:48.971+0000 7f28fad9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f28fc1077b0 0x7f28fc19c840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36722/0 (socket says 172.21.15.29:36722) 2024-03-16T12:03:48.969 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:48.971+0000 7f28fa59c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f28fc069f80 0x7f28fc197f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:03:48.969 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:48.971+0000 7f28fad9d640 1 -- 172.21.15.29:0/4224262762 learned_addr learned my addr 172.21.15.29:0/4224262762 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T12:03:48.969 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:48.971+0000 7f28f1d9b640 1 --2- 172.21.15.29:0/4224262762 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f28fc06a920 0x7f28fc1984c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:03:48.969 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:48.971+0000 7f28fad9d640 1 -- 172.21.15.29:0/4224262762 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f28fc069f80 msgr2=0x7f28fc197f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:03:48.969 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:48.971+0000 7f28fad9d640 1 --2- 172.21.15.29:0/4224262762 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f28fc069f80 0x7f28fc197f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:03:48.969 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:48.971+0000 7f28fad9d640 1 -- 172.21.15.29:0/4224262762 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f28fc06a920 msgr2=0x7f28fc1984c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:03:48.969 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:48.971+0000 7f28fad9d640 1 --2- 172.21.15.29:0/4224262762 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f28fc06a920 0x7f28fc1984c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:03:48.969 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:48.971+0000 7f28fad9d640 1 -- 172.21.15.29:0/4224262762 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28fc19cf70 con 0x7f28fc1077b0 2024-03-16T12:03:48.970 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:48.971+0000 7f28fa59c640 1 --2- 172.21.15.29:0/4224262762 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f28fc069f80 0x7f28fc197f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:03:48.970 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:48.971+0000 7f28fad9d640 1 --2- 172.21.15.29:0/4224262762 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f28fc1077b0 0x7f28fc19c840 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f28f400eb10 tx=0x7f28f400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:03:48.970 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:48.971+0000 7f28f3fff640 1 -- 172.21.15.29:0/4224262762 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28f4004b20 con 0x7f28fc1077b0 2024-03-16T12:03:48.970 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:48.971+0000 7f2900ec0640 1 -- 172.21.15.29:0/4224262762 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f28fc19d260 con 0x7f28fc1077b0 2024-03-16T12:03:48.970 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:48.971+0000 7f28f3fff640 1 -- 172.21.15.29:0/4224262762 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f28f4004cc0 con 0x7f28fc1077b0 2024-03-16T12:03:48.970 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:48.971+0000 7f28f3fff640 1 -- 172.21.15.29:0/4224262762 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28f4010760 con 0x7f28fc1077b0 2024-03-16T12:03:48.970 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:48.971+0000 7f2900ec0640 1 -- 172.21.15.29:0/4224262762 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f28fc1a4af0 con 0x7f28fc1077b0 2024-03-16T12:03:48.977 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:48.973+0000 7f28f3fff640 1 -- 172.21.15.29:0/4224262762 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7f28f4005020 con 0x7f28fc1077b0 2024-03-16T12:03:48.977 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:48.974+0000 7f2903b84740 1 Processor -- start 2024-03-16T12:03:48.977 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:48.974+0000 7f2903b84740 1 -- start start 2024-03-16T12:03:48.977 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:48.974+0000 7f28f3fff640 1 --2- 172.21.15.29:0/4224262762 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f28e80a2a10 0x7f28e80a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:03:48.978 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:48.974+0000 7f2903b84740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55614faf9130 0x55614faf9510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:03:48.978 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:48.974+0000 7f2903b84740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55614faf9a50 0x55614fb022e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:03:48.978 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:48.974+0000 7f2903b84740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55614fb02820 0x55614fb04c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:03:48.978 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:48.974+0000 7f2903b84740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55614fa5da40 con 0x55614faf9130 2024-03-16T12:03:48.978 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:48.974+0000 7f2903b84740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x55614fa06920 con 0x55614fb02820 2024-03-16T12:03:48.978 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:48.974+0000 7f2903b84740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55614fa59ae0 con 0x55614faf9a50 2024-03-16T12:03:48.978 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:48.974+0000 7f28f3fff640 1 -- 172.21.15.29:0/4224262762 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 8512+0+0 (secure 0 0 0) 0x7f28f40cf4a0 con 0x7f28fc1077b0 2024-03-16T12:03:48.979 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:48.974+0000 7f28f1d9b640 1 --2- 172.21.15.29:0/4224262762 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f28e80a2a10 0x7f28e80a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:03:48.979 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:48.974+0000 7f28fa59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55614faf9130 0x55614faf9510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:03:48.979 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:48.974+0000 7f28f1d9b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55614faf9a50 0x55614fb022e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:03:48.979 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:48.974+0000 7f28fa59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55614faf9130 0x55614faf9510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36730/0 (socket says 172.21.15.29:36730) 2024-03-16T12:03:48.979 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:48.975+0000 7f28fa59c640 1 -- 172.21.15.29:0/3824181338 learned_addr learned my addr 172.21.15.29:0/3824181338 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T12:03:48.979 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:48.975+0000 7f28fad9d640 1 --2- 172.21.15.29:0/3824181338 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55614fb02820 0x55614fb04c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:03:48.979 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:48.975+0000 7f28fa59c640 1 -- 172.21.15.29:0/3824181338 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55614faf9a50 msgr2=0x55614fb022e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:03:48.979 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:48.975+0000 7f28fa59c640 1 --2- 172.21.15.29:0/3824181338 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55614faf9a50 0x55614fb022e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:03:48.979 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:48.975+0000 7f28fa59c640 1 -- 172.21.15.29:0/3824181338 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55614fb02820 msgr2=0x55614fb04c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:03:48.979 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:48.975+0000 7f28fa59c640 1 --2- 172.21.15.29:0/3824181338 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55614fb02820 0x55614fb04c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:03:48.979 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:48.975+0000 7f28fa59c640 1 -- 172.21.15.29:0/3824181338 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55614fa1e460 con 0x55614faf9130 2024-03-16T12:03:48.979 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:48.975+0000 7f28fad9d640 1 --2- 172.21.15.29:0/3824181338 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55614fb02820 0x55614fb04c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:03:48.979 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:48.975+0000 7f28f1d9b640 1 --2- 172.21.15.29:0/4224262762 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f28e80a2a10 0x7f28e80a4ed0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f28e400bfd0 tx=0x7f28e400bb40 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:03:48.979 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:48.975+0000 7f28fa59c640 1 --2- 172.21.15.29:0/3824181338 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55614faf9130 0x55614faf9510 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f28ec02c170 tx=0x7f28ec009610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7be9666ec0e4306f server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:03:48.979 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:48.975+0000 7f28f0d99640 1 -- 172.21.15.29:0/3824181338 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28ec042070 con 0x55614faf9130 2024-03-16T12:03:48.979 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:48.975+0000 7f28f0d99640 1 -- 172.21.15.29:0/3824181338 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f28ec004a70 con 0x55614faf9130 2024-03-16T12:03:48.980 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:48.975+0000 7f28f0d99640 1 -- 172.21.15.29:0/3824181338 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28ec005160 con 0x55614faf9130 2024-03-16T12:03:48.980 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:48.976+0000 7f2903b84740 1 -- 172.21.15.29:0/3824181338 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55614faf9130 msgr2=0x55614faf9510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:03:48.980 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:48.976+0000 7f2903b84740 1 --2- 172.21.15.29:0/3824181338 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55614faf9130 0x55614faf9510 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f28ec02c170 tx=0x7f28ec009610 comp rx=0 tx=0).stop 2024-03-16T12:03:48.980 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:48.976+0000 7f2903b84740 1 -- 172.21.15.29:0/3824181338 shutdown_connections 2024-03-16T12:03:48.980 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:48.976+0000 7f2903b84740 1 --2- 172.21.15.29:0/3824181338 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55614fb02820 0x55614fb04c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:03:48.980 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:48.976+0000 7f2903b84740 1 --2- 172.21.15.29:0/3824181338 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55614faf9a50 0x55614fb022e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:03:48.980 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:48.976+0000 7f2903b84740 1 --2- 172.21.15.29:0/3824181338 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55614faf9130 0x55614faf9510 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:03:48.980 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:48.976+0000 7f2903b84740 1 -- 172.21.15.29:0/3824181338 >> 172.21.15.29:0/3824181338 conn(0x55614faf3a50 msgr2=0x55614faf3e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:03:48.980 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:48.976+0000 7f2903b84740 1 -- 172.21.15.29:0/3824181338 shutdown_connections 2024-03-16T12:03:48.980 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:48.976+0000 7f2903b84740 1 -- 172.21.15.29:0/3824181338 wait complete. 2024-03-16T12:03:48.980 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:48.976+0000 7f2903b84740 1 -- start start 2024-03-16T12:03:48.980 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:48.976+0000 7f2903b84740 1 Processor -- start 2024-03-16T12:03:48.980 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:48.977+0000 7f2903b84740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55614faf9130 0x55614faff8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:03:48.980 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:48.977+0000 7f2903b84740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55614faf9a50 0x55614faffdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:03:48.980 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:48.977+0000 7f2903b84740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55614fb02820 0x55614fb00330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:03:48.980 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:48.977+0000 7f2903b84740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55614fb053f0 con 0x55614faf9a50 2024-03-16T12:03:48.981 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:48.977+0000 7f2903b84740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x55614fa19d70 con 0x55614fb02820 2024-03-16T12:03:48.981 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:48.977+0000 7f2903b84740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55614fb05570 con 0x55614faf9130 2024-03-16T12:03:48.981 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:48.977+0000 7f2903b84740 10 client.0 _ll_register_callbacks cb 0x55614f771a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T12:03:48.981 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:48.977+0000 7f28f1d9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55614faf9a50 0x55614faffdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:03:48.981 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:48.977+0000 7f28f1d9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55614faf9a50 0x55614faffdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36746/0 (socket says 172.21.15.29:36746) 2024-03-16T12:03:48.981 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:48.977+0000 7f28f1d9b640 1 -- 172.21.15.29:0/3693810868 learned_addr learned my addr 172.21.15.29:0/3693810868 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T12:03:48.981 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:48.977+0000 7f28fa59c640 1 --2- 172.21.15.29:0/3693810868 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55614faf9130 0x55614faff8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:03:48.981 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:48.977+0000 7f28fad9d640 1 --2- 172.21.15.29:0/3693810868 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55614fb02820 0x55614fb00330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:03:48.981 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:48.977+0000 7f28f1d9b640 1 -- 172.21.15.29:0/3693810868 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55614faf9130 msgr2=0x55614faff8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:03:48.981 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:48.977+0000 7f28f1d9b640 1 --2- 172.21.15.29:0/3693810868 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55614faf9130 0x55614faff8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:03:48.981 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:48.977+0000 7f28f1d9b640 1 -- 172.21.15.29:0/3693810868 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55614fb02820 msgr2=0x55614fb00330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:03:48.981 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:48.977+0000 7f28f1d9b640 1 --2- 172.21.15.29:0/3693810868 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55614fb02820 0x55614fb00330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:03:48.982 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:48.977+0000 7f28f1d9b640 1 -- 172.21.15.29:0/3693810868 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55614f772210 con 0x55614faf9a50 2024-03-16T12:03:48.982 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:48.977+0000 7f28f1d9b640 1 --2- 172.21.15.29:0/3693810868 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55614faf9a50 0x55614faffdf0 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f28e4010b70 tx=0x7f28e400b410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b486a5a115b7bb6 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:03:48.982 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:48.977+0000 7f28e1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-16T12:03:48.982 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:48.977+0000 7f28e1ffb640 1 -- 172.21.15.29:0/3693810868 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28e401ec00 con 0x55614faf9a50 2024-03-16T12:03:48.982 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:48.977+0000 7f28e1ffb640 1 -- 172.21.15.29:0/3693810868 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(108..108 src has 1..108) ==== 8512+0+0 (secure 0 0 0) 0x7f28e401eda0 con 0x55614faf9a50 2024-03-16T12:03:48.982 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:48.977+0000 7f2903b84740 1 -- 172.21.15.29:0/3693810868 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f28ec004a70 con 0x55614faf9a50 2024-03-16T12:03:48.982 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:48.978+0000 7f28e1ffb640 1 -- 172.21.15.29:0/3693810868 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f28e402a410 con 0x55614faf9a50 2024-03-16T12:03:48.982 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:48.978+0000 7f2903b84740 10 client.16479 fetch_fsmap learned FSMap version 30 2024-03-16T12:03:48.982 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:48.978+0000 7f2903b84740 1 -- 172.21.15.29:0/3693810868 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55614f83e620 con 0x55614faf9a50 2024-03-16T12:03:48.982 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:48.978+0000 7f28e1ffb640 1 -- 172.21.15.29:0/3693810868 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f28ec004a70 con 0x55614faf9a50 2024-03-16T12:03:48.982 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:48.978+0000 7f28e1ffb640 1 -- 172.21.15.29:0/3693810868 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 30) ==== 8338+0+0 (secure 0 0 0) 0x7f28e401f070 con 0x55614faf9a50 2024-03-16T12:03:48.982 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:48.978+0000 7f2903b84740 10 client.16479 fetch_fsmap finished waiting for FSMap version 30 2024-03-16T12:03:48.982 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:48.978+0000 7f2903b84740 10 client.16479 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T12:03:48.982 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:48.978+0000 7f2903b84740 20 client.16479 populate_metadata read hostname 'smithi029' 2024-03-16T12:03:48.983 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:48.978+0000 7f2903b84740 1 --2- 172.21.15.29:0/3693810868 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55614fb129a0 0x55614fb14df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:03:48.983 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:48.978+0000 7f2903b84740 4 client.16479 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T12:03:48.983 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:48.978+0000 7f2903b84740 1 -- 172.21.15.29:0/3693810868 --> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55614fa6af80 con 0x55614fb129a0 2024-03-16T12:03:48.983 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:48.978+0000 7f28fa59c640 1 --2- 172.21.15.29:0/3693810868 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55614fb129a0 0x55614fb14df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:03:48.983 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:48.979+0000 7f28fa59c640 1 --2- 172.21.15.29:0/3693810868 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55614fb129a0 0x55614fb14df0 crc :-1 s=READY pgs=154 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3b5f60cf781ceaae server_cookie=36da0f7570f25407 in_seq=0 out_seq=0 2024-03-16T12:03:48.983 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:48.979+0000 7f28e1ffb640 10 client.16479 ms_handle_connect on v2:172.21.15.152:6838/1548731833 2024-03-16T12:03:48.983 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:48.980+0000 7f28e1ffb640 1 -- 172.21.15.29:0/3693810868 <== mds.0 v2:172.21.15.152:6838/1548731833 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55614fa6af80 con 0x55614fb129a0 2024-03-16T12:03:48.983 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:48.980+0000 7f28e1ffb640 10 client.16479 handle_command_reply: tid=0 2024-03-16T12:03:48.983 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:48.980+0000 7f2903b84740 1 client.16479 shutdown 2024-03-16T12:03:48.983 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:48.980+0000 7f2903b84740 1 -- 172.21.15.29:0/3693810868 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55614faf9a50 msgr2=0x55614faffdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:03:48.983 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:48.980+0000 7f2903b84740 1 --2- 172.21.15.29:0/3693810868 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55614faf9a50 0x55614faffdf0 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f28e4010b70 tx=0x7f28e400b410 comp rx=0 tx=0).stop 2024-03-16T12:03:48.983 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:48.980+0000 7f2903b84740 1 -- 172.21.15.29:0/3693810868 shutdown_connections 2024-03-16T12:03:48.983 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:48.980+0000 7f2903b84740 1 --2- 172.21.15.29:0/3693810868 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55614fb129a0 0x55614fb14df0 crc :-1 s=READY pgs=154 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:03:48.983 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:48.980+0000 7f2903b84740 1 --2- 172.21.15.29:0/3693810868 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55614fb02820 0x55614fb00330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:03:48.983 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:48.980+0000 7f28e1ffb640 0 client.16479 ms_handle_reset on v2:172.21.15.152:6838/1548731833 2024-03-16T12:03:48.984 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:48.980+0000 7f2903b84740 1 --2- 172.21.15.29:0/3693810868 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55614faf9a50 0x55614faffdf0 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:03:48.984 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:48.980+0000 7f2903b84740 1 --2- 172.21.15.29:0/3693810868 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55614faf9130 0x55614faff8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:03:48.984 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:48.980+0000 7f2903b84740 1 -- 172.21.15.29:0/3693810868 >> 172.21.15.29:0/3693810868 conn(0x55614faf3a50 msgr2=0x55614faf57a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:03:48.984 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:48.980+0000 7f2903b84740 1 -- 172.21.15.29:0/3693810868 shutdown_connections 2024-03-16T12:03:48.984 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:48.980+0000 7f2903b84740 1 -- 172.21.15.29:0/3693810868 wait complete. 2024-03-16T12:03:48.984 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:48.980+0000 7f2903b84740 20 client.16479 trim_cache size 0 max 16384 2024-03-16T12:03:48.996 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:48.995+0000 7f2903b84740 1 Processor -- start 2024-03-16T12:03:48.996 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:48.996+0000 7f2903b84740 1 -- start start 2024-03-16T12:03:48.996 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:48.996+0000 7f2903b84740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55614faf9130 0x55614fb01fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:03:48.996 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:48.996+0000 7f2903b84740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55614faf9a50 0x55614fb0e190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:03:48.996 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:48.996+0000 7f2903b84740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55614fb02820 0x55614faf5540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:03:48.996 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:48.996+0000 7f2903b84740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f28fc1afcc0 con 0x55614fb02820 2024-03-16T12:03:48.996 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:48.996+0000 7f2903b84740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f28fc1ad590 con 0x55614faf9a50 2024-03-16T12:03:48.996 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:48.996+0000 7f2903b84740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f28fc1ab3c0 con 0x55614faf9130 2024-03-16T12:03:48.997 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:48.996+0000 7f28fad9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55614fb02820 0x55614faf5540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:03:48.997 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:48.996+0000 7f28fad9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55614fb02820 0x55614faf5540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36752/0 (socket says 172.21.15.29:36752) 2024-03-16T12:03:48.997 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:48.996+0000 7f28fad9d640 1 -- 172.21.15.29:0/3717287341 learned_addr learned my addr 172.21.15.29:0/3717287341 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T12:03:48.997 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:48.996+0000 7f28fa59c640 1 --2- 172.21.15.29:0/3717287341 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55614faf9130 0x55614fb01fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:03:48.997 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:48.996+0000 7f28f1d9b640 1 --2- 172.21.15.29:0/3717287341 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55614faf9a50 0x55614fb0e190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:03:48.997 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:48.996+0000 7f28fad9d640 1 -- 172.21.15.29:0/3717287341 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55614faf9130 msgr2=0x55614fb01fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:03:48.997 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:48.996+0000 7f28fad9d640 1 --2- 172.21.15.29:0/3717287341 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55614faf9130 0x55614fb01fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:03:48.997 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:48.996+0000 7f28fad9d640 1 -- 172.21.15.29:0/3717287341 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55614faf9a50 msgr2=0x55614fb0e190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:03:48.997 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:48.996+0000 7f28fad9d640 1 --2- 172.21.15.29:0/3717287341 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55614faf9a50 0x55614fb0e190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:03:48.997 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:48.996+0000 7f28fad9d640 1 -- 172.21.15.29:0/3717287341 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55614fae5140 con 0x55614fb02820 2024-03-16T12:03:48.997 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:48.996+0000 7f28fad9d640 1 --2- 172.21.15.29:0/3717287341 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55614fb02820 0x55614faf5540 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f28f409baa0 tx=0x7f28f400cf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9296d833907c43a server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:03:48.997 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:48.996+0000 7f28e3fff640 1 -- 172.21.15.29:0/3717287341 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28f409bb10 con 0x55614fb02820 2024-03-16T12:03:48.997 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:48.997+0000 7f28e3fff640 1 -- 172.21.15.29:0/3717287341 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f28f409fc60 con 0x55614fb02820 2024-03-16T12:03:48.997 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:48.997+0000 7f28e3fff640 1 -- 172.21.15.29:0/3717287341 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28f4005570 con 0x55614fb02820 2024-03-16T12:03:48.998 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:48.997+0000 7f2903b84740 1 -- 172.21.15.29:0/3717287341 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55614fb02820 msgr2=0x55614faf5540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:03:48.998 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:48.997+0000 7f2903b84740 1 --2- 172.21.15.29:0/3717287341 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55614fb02820 0x55614faf5540 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f28f409baa0 tx=0x7f28f400cf40 comp rx=0 tx=0).stop 2024-03-16T12:03:48.998 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:48.997+0000 7f2903b84740 1 -- 172.21.15.29:0/3717287341 shutdown_connections 2024-03-16T12:03:48.998 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:48.997+0000 7f2903b84740 1 --2- 172.21.15.29:0/3717287341 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55614fb02820 0x55614faf5540 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:03:48.998 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:48.997+0000 7f2903b84740 1 --2- 172.21.15.29:0/3717287341 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55614faf9a50 0x55614fb0e190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:03:48.998 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:48.997+0000 7f2903b84740 1 --2- 172.21.15.29:0/3717287341 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55614faf9130 0x55614fb01fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:03:48.998 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:48.997+0000 7f2903b84740 1 -- 172.21.15.29:0/3717287341 >> 172.21.15.29:0/3717287341 conn(0x55614faf3a50 msgr2=0x55614faf4e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:03:48.998 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:48.997+0000 7f2903b84740 1 -- 172.21.15.29:0/3717287341 shutdown_connections 2024-03-16T12:03:48.998 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:48.997+0000 7f2903b84740 1 -- 172.21.15.29:0/3717287341 wait complete. 2024-03-16T12:03:48.998 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:48.997+0000 7f2903b84740 1 -- start start 2024-03-16T12:03:48.998 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:48.998+0000 7f2903b84740 1 Processor -- start 2024-03-16T12:03:48.998 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:48.998+0000 7f2903b84740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55614faf9130 0x55614fb06480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:03:48.998 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:48.998+0000 7f2903b84740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55614faf9a50 0x55614fb069c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:03:48.998 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:48.998+0000 7f2903b84740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55614fb02820 0x55614fb77d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:03:48.998 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:48.998+0000 7f2903b84740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f28fc1aba50 con 0x55614faf9130 2024-03-16T12:03:48.999 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:48.998+0000 7f2903b84740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f28fc1acec0 con 0x55614faf9a50 2024-03-16T12:03:48.999 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:48.998+0000 7f2903b84740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f28fc1ac120 con 0x55614fb02820 2024-03-16T12:03:48.999 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:48.998+0000 7f2903b84740 10 client.0 _ll_register_callbacks cb 0x7f28e400bd50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T12:03:48.999 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:48.998+0000 7f28fa59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55614faf9130 0x55614fb06480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:03:48.999 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:48.998+0000 7f28fa59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55614faf9130 0x55614fb06480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36760/0 (socket says 172.21.15.29:36760) 2024-03-16T12:03:48.999 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:48.998+0000 7f28fa59c640 1 -- 172.21.15.29:0/3647685873 learned_addr learned my addr 172.21.15.29:0/3647685873 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T12:03:48.999 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:48.998+0000 7f28f1d9b640 1 --2- 172.21.15.29:0/3647685873 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55614faf9a50 0x55614fb069c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:03:48.999 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:48.998+0000 7f28fad9d640 1 --2- 172.21.15.29:0/3647685873 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55614fb02820 0x55614fb77d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:03:48.999 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:48.998+0000 7f28fa59c640 1 -- 172.21.15.29:0/3647685873 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55614fb02820 msgr2=0x55614fb77d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:03:48.999 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:48.998+0000 7f28fa59c640 1 --2- 172.21.15.29:0/3647685873 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55614fb02820 0x55614fb77d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:03:48.999 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:48.998+0000 7f28fa59c640 1 -- 172.21.15.29:0/3647685873 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55614faf9a50 msgr2=0x55614fb069c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:03:48.999 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:48.998+0000 7f28fa59c640 1 --2- 172.21.15.29:0/3647685873 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55614faf9a50 0x55614fb069c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:03:48.999 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:48.998+0000 7f28fa59c640 1 -- 172.21.15.29:0/3647685873 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55614fb78270 con 0x55614faf9130 2024-03-16T12:03:48.999 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:48.998+0000 7f28fa59c640 1 --2- 172.21.15.29:0/3647685873 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55614faf9130 0x55614fb06480 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f28ec004470 tx=0x7f28ec009680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=334d4606df2c8af2 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:03:49.000 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:48.998+0000 7f28e1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-16T12:03:49.000 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:48.998+0000 7f28e1ffb640 1 -- 172.21.15.29:0/3647685873 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28ec042040 con 0x55614faf9130 2024-03-16T12:03:49.000 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:48.998+0000 7f28e1ffb640 1 -- 172.21.15.29:0/3647685873 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(108..108 src has 1..108) ==== 8512+0+0 (secure 0 0 0) 0x7f28ec04ea30 con 0x55614faf9130 2024-03-16T12:03:49.000 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:48.998+0000 7f2903b84740 1 -- 172.21.15.29:0/3647685873 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f28f409fc60 con 0x55614faf9130 2024-03-16T12:03:49.000 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:48.999+0000 7f28e1ffb640 1 -- 172.21.15.29:0/3647685873 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f28ec051080 con 0x55614faf9130 2024-03-16T12:03:49.000 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:48.999+0000 7f2903b84740 10 client.16485 fetch_fsmap learned FSMap version 30 2024-03-16T12:03:49.000 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:48.999+0000 7f2903b84740 1 -- 172.21.15.29:0/3647685873 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55614fb79460 con 0x55614faf9130 2024-03-16T12:03:49.000 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:48.999+0000 7f28e1ffb640 1 -- 172.21.15.29:0/3647685873 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f28f409fc60 con 0x55614faf9130 2024-03-16T12:03:49.000 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:48.999+0000 7f28e1ffb640 1 -- 172.21.15.29:0/3647685873 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 30) ==== 8338+0+0 (secure 0 0 0) 0x7f28ec04e030 con 0x55614faf9130 2024-03-16T12:03:49.000 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:48.999+0000 7f2903b84740 10 client.16485 fetch_fsmap finished waiting for FSMap version 30 2024-03-16T12:03:49.000 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:48.999+0000 7f2903b84740 10 client.16485 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T12:03:49.000 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:48.999+0000 7f2903b84740 20 client.16485 populate_metadata read hostname 'smithi029' 2024-03-16T12:03:49.000 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:48.999+0000 7f2903b84740 1 --2- 172.21.15.29:0/3647685873 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55614fb797f0 0x55614fb7bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:03:49.000 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:48.999+0000 7f2903b84740 4 client.16485 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-16T12:03:49.000 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:48.999+0000 7f2903b84740 1 -- 172.21.15.29:0/3647685873 --> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55614fa652d0 con 0x55614fb797f0 2024-03-16T12:03:49.001 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:48.999+0000 7f28f1d9b640 1 --2- 172.21.15.29:0/3647685873 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55614fb797f0 0x55614fb7bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:03:49.001 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:49.000+0000 7f28e1ffb640 10 client.16485 ms_handle_connect on v2:172.21.15.152:6838/1548731833 2024-03-16T12:03:49.001 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:49.000+0000 7f28f1d9b640 1 --2- 172.21.15.29:0/3647685873 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55614fb797f0 0x55614fb7bbe0 crc :-1 s=READY pgs=155 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=43426db96f5aae10 server_cookie=e8627ad891580c96 in_seq=0 out_seq=0 2024-03-16T12:03:49.185 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:49.188+0000 7f28e1ffb640 1 -- 172.21.15.29:0/3647685873 <== mds.0 v2:172.21.15.152:6838/1548731833 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55614fa652d0 con 0x55614fb797f0 2024-03-16T12:03:49.185 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:49.188+0000 7f28e1ffb640 10 client.16485 handle_command_reply: tid=0 2024-03-16T12:03:49.185 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:49.188+0000 7f2903b84740 1 client.16485 shutdown 2024-03-16T12:03:49.185 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:49.189+0000 7f2903b84740 1 -- 172.21.15.29:0/3647685873 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55614faf9130 msgr2=0x55614fb06480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:03:49.185 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:49.189+0000 7f2903b84740 1 --2- 172.21.15.29:0/3647685873 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55614faf9130 0x55614fb06480 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f28ec004470 tx=0x7f28ec009680 comp rx=0 tx=0).stop 2024-03-16T12:03:49.185 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:49.189+0000 7f2903b84740 1 -- 172.21.15.29:0/3647685873 shutdown_connections 2024-03-16T12:03:49.185 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:49.189+0000 7f2903b84740 1 --2- 172.21.15.29:0/3647685873 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55614fb02820 0x55614fb77d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:03:49.185 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:49.189+0000 7f2903b84740 1 --2- 172.21.15.29:0/3647685873 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55614fb797f0 0x55614fb7bbe0 crc :-1 s=READY pgs=155 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:03:49.185 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:49.189+0000 7f2903b84740 1 --2- 172.21.15.29:0/3647685873 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55614faf9a50 0x55614fb069c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:03:49.185 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:49.189+0000 7f2903b84740 1 --2- 172.21.15.29:0/3647685873 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55614faf9130 0x55614fb06480 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:03:49.186 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:49.189+0000 7f28e1ffb640 0 client.16485 ms_handle_reset on v2:172.21.15.152:6838/1548731833 2024-03-16T12:03:49.186 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:49.189+0000 7f2903b84740 1 -- 172.21.15.29:0/3647685873 >> 172.21.15.29:0/3647685873 conn(0x55614faf3a50 msgr2=0x55614faf5610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:03:49.186 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:49.189+0000 7f2903b84740 1 -- 172.21.15.29:0/3647685873 shutdown_connections 2024-03-16T12:03:49.186 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:49.189+0000 7f2903b84740 1 -- 172.21.15.29:0/3647685873 wait complete. 2024-03-16T12:03:49.186 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:49.189+0000 7f2903b84740 20 client.16485 trim_cache size 0 max 16384 2024-03-16T12:03:49.186 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:{ 2024-03-16T12:03:49.186 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "message": "", 2024-03-16T12:03:49.186 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "return_code": 0 2024-03-16T12:03:49.186 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:} 2024-03-16T12:03:49.187 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:49.189+0000 7f2900ec0640 1 -- 172.21.15.29:0/4224262762 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f28e80a2a10 msgr2=0x7f28e80a4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:03:49.187 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:49.189+0000 7f2900ec0640 1 --2- 172.21.15.29:0/4224262762 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f28e80a2a10 0x7f28e80a4ed0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f28e400bfd0 tx=0x7f28e400bb40 comp rx=0 tx=0).stop 2024-03-16T12:03:49.187 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:49.189+0000 7f2900ec0640 1 -- 172.21.15.29:0/4224262762 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f28fc1077b0 msgr2=0x7f28fc19c840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:03:49.187 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:49.189+0000 7f2900ec0640 1 --2- 172.21.15.29:0/4224262762 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f28fc1077b0 0x7f28fc19c840 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f28f400eb10 tx=0x7f28f400c2d0 comp rx=0 tx=0).stop 2024-03-16T12:03:49.187 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:49.189+0000 7f2900ec0640 1 -- 172.21.15.29:0/4224262762 shutdown_connections 2024-03-16T12:03:49.187 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:49.189+0000 7f2900ec0640 1 --2- 172.21.15.29:0/4224262762 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f28e80a2a10 0x7f28e80a4ed0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:03:49.187 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:49.189+0000 7f2900ec0640 1 --2- 172.21.15.29:0/4224262762 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f28fc1077b0 0x7f28fc19c840 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:03:49.187 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:49.189+0000 7f2900ec0640 1 --2- 172.21.15.29:0/4224262762 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f28fc06a920 0x7f28fc1984c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:03:49.187 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:49.189+0000 7f2900ec0640 1 --2- 172.21.15.29:0/4224262762 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f28fc069f80 0x7f28fc197f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:03:49.188 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:49.189+0000 7f2900ec0640 1 -- 172.21.15.29:0/4224262762 >> 172.21.15.29:0/4224262762 conn(0x7f28fc0776c0 msgr2=0x7f28fc0ff450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:03:49.188 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:49.190+0000 7f2900ec0640 1 -- 172.21.15.29:0/4224262762 shutdown_connections 2024-03-16T12:03:49.188 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:03:49.190+0000 7f2900ec0640 1 -- 172.21.15.29:0/4224262762 wait complete. 2024-03-16T12:03:49.806 INFO:tasks.workunit.client.0.smithi029.stdout: 1 1235900 9.12 MB/sec execute 574 sec latency 108.100 ms 2024-03-16T12:03:50.541 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:03:50 smithi154 ceph-mon[36320]: pgmap v410: 97 pgs: 97 active+clean; 71 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 2.1 MiB/s wr, 142 op/s 2024-03-16T12:03:50.584 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:03:50 smithi029 ceph-mon[31975]: pgmap v410: 97 pgs: 97 active+clean; 71 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 2.1 MiB/s wr, 142 op/s 2024-03-16T12:03:50.623 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:03:50 smithi152 ceph-mon[36291]: pgmap v410: 97 pgs: 97 active+clean; 71 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 2.1 MiB/s wr, 142 op/s 2024-03-16T12:03:50.806 INFO:tasks.workunit.client.0.smithi029.stdout: 1 1236554 9.11 MB/sec execute 575 sec latency 18.520 ms 2024-03-16T12:03:51.806 INFO:tasks.workunit.client.0.smithi029.stdout: 1 1238402 9.10 MB/sec execute 576 sec latency 23.475 ms 2024-03-16T12:03:52.541 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:03:52 smithi154 ceph-mon[36320]: pgmap v411: 97 pgs: 97 active+clean; 64 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 2.7 MiB/s wr, 238 op/s 2024-03-16T12:03:52.584 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:03:52 smithi029 ceph-mon[31975]: pgmap v411: 97 pgs: 97 active+clean; 64 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 2.7 MiB/s wr, 238 op/s 2024-03-16T12:03:52.623 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:03:52 smithi152 ceph-mon[36291]: pgmap v411: 97 pgs: 97 active+clean; 64 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 2.7 MiB/s wr, 238 op/s 2024-03-16T12:03:52.806 INFO:tasks.workunit.client.0.smithi029.stdout: 1 1239728 9.11 MB/sec execute 577 sec latency 124.378 ms 2024-03-16T12:03:53.806 INFO:tasks.workunit.client.0.smithi029.stdout: 1 1241142 9.10 MB/sec execute 578 sec latency 24.862 ms 2024-03-16T12:03:54.541 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:03:54 smithi154 ceph-mon[36320]: pgmap v412: 97 pgs: 97 active+clean; 64 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 2.6 MiB/s wr, 238 op/s 2024-03-16T12:03:54.584 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:03:54 smithi029 ceph-mon[31975]: pgmap v412: 97 pgs: 97 active+clean; 64 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 2.6 MiB/s wr, 238 op/s 2024-03-16T12:03:54.623 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:03:54 smithi152 ceph-mon[36291]: pgmap v412: 97 pgs: 97 active+clean; 64 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 2.6 MiB/s wr, 238 op/s 2024-03-16T12:03:54.806 INFO:tasks.workunit.client.0.smithi029.stdout: 1 1243114 9.11 MB/sec execute 579 sec latency 42.255 ms 2024-03-16T12:03:55.806 INFO:tasks.workunit.client.0.smithi029.stdout: 1 1244277 9.10 MB/sec execute 580 sec latency 23.498 ms 2024-03-16T12:03:56.542 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:03:56 smithi154 ceph-mon[36320]: pgmap v413: 97 pgs: 97 active+clean; 64 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 2.2 MiB/s wr, 209 op/s 2024-03-16T12:03:56.584 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:03:56 smithi029 ceph-mon[31975]: pgmap v413: 97 pgs: 97 active+clean; 64 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 2.2 MiB/s wr, 209 op/s 2024-03-16T12:03:56.623 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:03:56 smithi152 ceph-mon[36291]: pgmap v413: 97 pgs: 97 active+clean; 64 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 2.2 MiB/s wr, 209 op/s 2024-03-16T12:03:56.807 INFO:tasks.workunit.client.0.smithi029.stdout: 1 1246525 9.11 MB/sec execute 581 sec latency 20.065 ms 2024-03-16T12:03:57.806 INFO:tasks.workunit.client.0.smithi029.stdout: 1 1247081 9.10 MB/sec execute 582 sec latency 67.080 ms 2024-03-16T12:03:58.541 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:03:58 smithi154 ceph-mon[36320]: pgmap v414: 97 pgs: 97 active+clean; 69 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 3.3 MiB/s wr, 283 op/s 2024-03-16T12:03:58.584 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:03:58 smithi029 ceph-mon[31975]: pgmap v414: 97 pgs: 97 active+clean; 69 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 3.3 MiB/s wr, 283 op/s 2024-03-16T12:03:58.623 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:03:58 smithi152 ceph-mon[36291]: pgmap v414: 97 pgs: 97 active+clean; 69 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 3.3 MiB/s wr, 283 op/s 2024-03-16T12:03:58.807 INFO:tasks.workunit.client.0.smithi029.stdout: 1 1248714 9.09 MB/sec execute 583 sec latency 26.678 ms 2024-03-16T12:03:59.807 INFO:tasks.workunit.client.0.smithi029.stdout: 1 1250250 9.10 MB/sec execute 584 sec latency 84.248 ms 2024-03-16T12:04:00.541 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:04:00 smithi154 ceph-mon[36320]: pgmap v415: 97 pgs: 97 active+clean; 69 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 2.2 MiB/s wr, 209 op/s 2024-03-16T12:04:00.584 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:04:00 smithi029 ceph-mon[31975]: pgmap v415: 97 pgs: 97 active+clean; 69 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 2.2 MiB/s wr, 209 op/s 2024-03-16T12:04:00.623 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:04:00 smithi152 ceph-mon[36291]: pgmap v415: 97 pgs: 97 active+clean; 69 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 2.2 MiB/s wr, 209 op/s 2024-03-16T12:04:00.806 INFO:tasks.workunit.client.0.smithi029.stdout: 1 1251133 9.08 MB/sec execute 585 sec latency 58.148 ms 2024-03-16T12:04:01.806 INFO:tasks.workunit.client.0.smithi029.stdout: 1 1252475 9.08 MB/sec execute 586 sec latency 33.737 ms 2024-03-16T12:04:02.278 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:02.280+0000 7fa77f252640 1 -- 172.21.15.29:0/1491516224 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa7781077c0 msgr2=0x7fa778107ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:04:02.278 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:02.280+0000 7fa77f252640 1 --2- 172.21.15.29:0/1491516224 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa7781077c0 0x7fa778107ba0 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7fa768009930 tx=0x7fa76802bc30 comp rx=0 tx=0).stop 2024-03-16T12:04:02.278 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:02.280+0000 7fa77f252640 1 -- 172.21.15.29:0/1491516224 shutdown_connections 2024-03-16T12:04:02.278 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:02.280+0000 7fa77f252640 1 --2- 172.21.15.29:0/1491516224 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa778102170 0x7fa77810e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:04:02.278 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:02.280+0000 7fa77f252640 1 --2- 172.21.15.29:0/1491516224 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa7781017b0 0x7fa778101c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:04:02.278 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:02.280+0000 7fa77f252640 1 --2- 172.21.15.29:0/1491516224 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa7781077c0 0x7fa778107ba0 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:04:02.278 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:02.280+0000 7fa77f252640 1 -- 172.21.15.29:0/1491516224 >> 172.21.15.29:0/1491516224 conn(0x7fa7780fd2c0 msgr2=0x7fa7780ff6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:04:02.278 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:02.280+0000 7fa77f252640 1 -- 172.21.15.29:0/1491516224 shutdown_connections 2024-03-16T12:04:02.278 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:02.280+0000 7fa77f252640 1 -- 172.21.15.29:0/1491516224 wait complete. 2024-03-16T12:04:02.279 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:02.280+0000 7fa77f252640 1 Processor -- start 2024-03-16T12:04:02.279 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:02.280+0000 7fa77f252640 1 -- start start 2024-03-16T12:04:02.279 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:02.281+0000 7fa77f252640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa7781017b0 0x7fa778197f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:04:02.279 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:02.281+0000 7fa77f252640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa778102170 0x7fa7781984c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:04:02.279 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:02.281+0000 7fa77f252640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa7781077c0 0x7fa77819c840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:04:02.279 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:02.281+0000 7fa77f252640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fa778111be0 con 0x7fa7781017b0 2024-03-16T12:04:02.279 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:02.281+0000 7fa77f252640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fa778111a60 con 0x7fa7781077c0 2024-03-16T12:04:02.279 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:02.281+0000 7fa77f252640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fa778111d60 con 0x7fa778102170 2024-03-16T12:04:02.279 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:02.281+0000 7fa77cfee640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa7781017b0 0x7fa778197f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:04:02.279 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:02.281+0000 7fa77cfee640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa7781017b0 0x7fa778197f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55784/0 (socket says 172.21.15.29:55784) 2024-03-16T12:04:02.279 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:02.281+0000 7fa77cfee640 1 -- 172.21.15.29:0/3185638895 learned_addr learned my addr 172.21.15.29:0/3185638895 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T12:04:02.279 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:02.281+0000 7fa77d7ef640 1 --2- 172.21.15.29:0/3185638895 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa7781077c0 0x7fa77819c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:04:02.279 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:02.281+0000 7fa777fff640 1 --2- 172.21.15.29:0/3185638895 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa778102170 0x7fa7781984c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:04:02.280 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:02.281+0000 7fa77cfee640 1 -- 172.21.15.29:0/3185638895 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa778102170 msgr2=0x7fa7781984c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:04:02.280 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:02.281+0000 7fa77cfee640 1 --2- 172.21.15.29:0/3185638895 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa778102170 0x7fa7781984c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:04:02.280 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:02.281+0000 7fa77cfee640 1 -- 172.21.15.29:0/3185638895 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa7781077c0 msgr2=0x7fa77819c840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:04:02.280 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:02.281+0000 7fa77cfee640 1 --2- 172.21.15.29:0/3185638895 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa7781077c0 0x7fa77819c840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:04:02.280 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:02.281+0000 7fa77cfee640 1 -- 172.21.15.29:0/3185638895 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa77819cf70 con 0x7fa7781017b0 2024-03-16T12:04:02.280 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:02.281+0000 7fa77cfee640 1 --2- 172.21.15.29:0/3185638895 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa7781017b0 0x7fa778197f80 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7fa768009900 tx=0x7fa768004360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:04:02.280 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:02.281+0000 7fa775ffb640 1 -- 172.21.15.29:0/3185638895 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa768043020 con 0x7fa7781017b0 2024-03-16T12:04:02.280 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:02.281+0000 7fa775ffb640 1 -- 172.21.15.29:0/3185638895 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa7680054e0 con 0x7fa7781017b0 2024-03-16T12:04:02.280 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:02.281+0000 7fa775ffb640 1 -- 172.21.15.29:0/3185638895 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa768003b80 con 0x7fa7781017b0 2024-03-16T12:04:02.280 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:02.281+0000 7fa77f252640 1 -- 172.21.15.29:0/3185638895 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa77819d200 con 0x7fa7781017b0 2024-03-16T12:04:02.280 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:02.282+0000 7fa77f252640 1 -- 172.21.15.29:0/3185638895 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa7781a4af0 con 0x7fa7781017b0 2024-03-16T12:04:02.287 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:02.284+0000 7fa775ffb640 1 -- 172.21.15.29:0/3185638895 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7fa768046020 con 0x7fa7781017b0 2024-03-16T12:04:02.287 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:02.284+0000 7fa781e3e740 1 Processor -- start 2024-03-16T12:04:02.287 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:02.284+0000 7fa781e3e740 1 -- start start 2024-03-16T12:04:02.287 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:02.284+0000 7fa781e3e740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56042a28d130 0x56042a28d510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:04:02.287 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:02.284+0000 7fa781e3e740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56042a28da50 0x56042a2962e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:04:02.287 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:02.284+0000 7fa781e3e740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56042a296820 0x56042a298c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:04:02.288 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:02.284+0000 7fa781e3e740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x56042a1f1a40 con 0x56042a296820 2024-03-16T12:04:02.288 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:02.284+0000 7fa781e3e740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x56042a19a920 con 0x56042a28da50 2024-03-16T12:04:02.288 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:02.284+0000 7fa781e3e740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x56042a1edae0 con 0x56042a28d130 2024-03-16T12:04:02.288 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:02.284+0000 7fa77cfee640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56042a28da50 0x56042a2962e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:04:02.288 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:02.284+0000 7fa777fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56042a296820 0x56042a298c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:04:02.288 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:02.284+0000 7fa777fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56042a296820 0x56042a298c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55792/0 (socket says 172.21.15.29:55792) 2024-03-16T12:04:02.288 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:02.284+0000 7fa77d7ef640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56042a28d130 0x56042a28d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:04:02.288 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:02.284+0000 7fa777fff640 1 -- 172.21.15.29:0/332890725 learned_addr learned my addr 172.21.15.29:0/332890725 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T12:04:02.288 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:02.284+0000 7fa77cfee640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56042a28da50 0x56042a2962e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.29:51086/0 (socket says 172.21.15.29:51086) 2024-03-16T12:04:02.288 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:02.284+0000 7fa775ffb640 1 --2- 172.21.15.29:0/3185638895 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7fa7600a2a10 0x7fa7600a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:04:02.288 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:02.285+0000 7fa77d7ef640 1 --2- 172.21.15.29:0/3185638895 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7fa7600a2a10 0x7fa7600a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:04:02.288 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:02.285+0000 7fa777fff640 1 -- 172.21.15.29:0/332890725 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56042a28d130 msgr2=0x56042a28d510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:04:02.289 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:02.285+0000 7fa777fff640 1 --2- 172.21.15.29:0/332890725 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56042a28d130 0x56042a28d510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:04:02.289 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:02.285+0000 7fa777fff640 1 -- 172.21.15.29:0/332890725 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56042a28da50 msgr2=0x56042a2962e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:04:02.289 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:02.285+0000 7fa777fff640 1 --2- 172.21.15.29:0/332890725 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56042a28da50 0x56042a2962e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:04:02.289 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:02.285+0000 7fa777fff640 1 -- 172.21.15.29:0/332890725 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56042a1b2460 con 0x56042a296820 2024-03-16T12:04:02.289 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:02.285+0000 7fa77cfee640 1 --2- 172.21.15.29:0/332890725 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56042a28da50 0x56042a2962e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:04:02.289 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:02.285+0000 7fa775ffb640 1 -- 172.21.15.29:0/3185638895 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 8512+0+0 (secure 0 0 0) 0x7fa7680f0e00 con 0x7fa7781017b0 2024-03-16T12:04:02.289 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:02.285+0000 7fa777fff640 1 --2- 172.21.15.29:0/332890725 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56042a296820 0x56042a298c10 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7fa76c009c90 tx=0x7fa76c011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b13a341f0381b0f1 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:04:02.289 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:02.285+0000 7fa75effd640 1 -- 172.21.15.29:0/332890725 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa76c01e070 con 0x56042a296820 2024-03-16T12:04:02.289 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:02.285+0000 7fa75effd640 1 -- 172.21.15.29:0/332890725 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa76c019b10 con 0x56042a296820 2024-03-16T12:04:02.289 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:02.285+0000 7fa75effd640 1 -- 172.21.15.29:0/332890725 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa76c00eac0 con 0x56042a296820 2024-03-16T12:04:02.289 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:02.286+0000 7fa781e3e740 1 -- 172.21.15.29:0/332890725 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56042a296820 msgr2=0x56042a298c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:04:02.289 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:02.286+0000 7fa781e3e740 1 --2- 172.21.15.29:0/332890725 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56042a296820 0x56042a298c10 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7fa76c009c90 tx=0x7fa76c011810 comp rx=0 tx=0).stop 2024-03-16T12:04:02.289 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:02.286+0000 7fa77d7ef640 1 --2- 172.21.15.29:0/3185638895 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7fa7600a2a10 0x7fa7600a4ed0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fa770007310 tx=0x7fa770004ab0 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:04:02.289 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:02.286+0000 7fa781e3e740 1 -- 172.21.15.29:0/332890725 shutdown_connections 2024-03-16T12:04:02.290 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:02.286+0000 7fa781e3e740 1 --2- 172.21.15.29:0/332890725 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56042a296820 0x56042a298c10 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:04:02.290 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:02.286+0000 7fa781e3e740 1 --2- 172.21.15.29:0/332890725 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56042a28da50 0x56042a2962e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:04:02.290 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:02.286+0000 7fa781e3e740 1 --2- 172.21.15.29:0/332890725 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56042a28d130 0x56042a28d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:04:02.290 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:02.286+0000 7fa781e3e740 1 -- 172.21.15.29:0/332890725 >> 172.21.15.29:0/332890725 conn(0x56042a287a50 msgr2=0x56042a287e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:04:02.290 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:02.286+0000 7fa781e3e740 1 -- 172.21.15.29:0/332890725 shutdown_connections 2024-03-16T12:04:02.290 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:02.286+0000 7fa781e3e740 1 -- 172.21.15.29:0/332890725 wait complete. 2024-03-16T12:04:02.290 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:02.286+0000 7fa781e3e740 1 -- start start 2024-03-16T12:04:02.290 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:02.286+0000 7fa781e3e740 1 Processor -- start 2024-03-16T12:04:02.290 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:02.287+0000 7fa781e3e740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56042a28d130 0x56042a2937c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:04:02.290 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:02.287+0000 7fa781e3e740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56042a28da50 0x56042a293d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:04:02.290 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:02.287+0000 7fa781e3e740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56042a296820 0x56042a294240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:04:02.290 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:02.287+0000 7fa781e3e740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x56042a299450 con 0x56042a28da50 2024-03-16T12:04:02.291 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:02.287+0000 7fa781e3e740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x56042a1add70 con 0x56042a28d130 2024-03-16T12:04:02.291 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:02.287+0000 7fa781e3e740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x56042a2995d0 con 0x56042a296820 2024-03-16T12:04:02.291 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:02.287+0000 7fa781e3e740 10 client.0 _ll_register_callbacks cb 0x560429f05a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T12:04:02.291 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:02.287+0000 7fa777fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56042a28da50 0x56042a293d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:04:02.291 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:02.287+0000 7fa777fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56042a28da50 0x56042a293d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55796/0 (socket says 172.21.15.29:55796) 2024-03-16T12:04:02.291 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:02.287+0000 7fa777fff640 1 -- 172.21.15.29:0/2068692437 learned_addr learned my addr 172.21.15.29:0/2068692437 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T12:04:02.291 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:02.287+0000 7fa77cfee640 1 --2- 172.21.15.29:0/2068692437 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56042a28d130 0x56042a2937c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:04:02.291 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:02.287+0000 7fa77d7ef640 1 --2- 172.21.15.29:0/2068692437 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56042a296820 0x56042a294240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:04:02.291 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:02.287+0000 7fa777fff640 1 -- 172.21.15.29:0/2068692437 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56042a296820 msgr2=0x56042a294240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:04:02.291 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:02.287+0000 7fa777fff640 1 --2- 172.21.15.29:0/2068692437 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56042a296820 0x56042a294240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:04:02.291 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:02.287+0000 7fa777fff640 1 -- 172.21.15.29:0/2068692437 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56042a28d130 msgr2=0x56042a2937c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:04:02.292 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:02.287+0000 7fa777fff640 1 --2- 172.21.15.29:0/2068692437 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56042a28d130 0x56042a2937c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:04:02.292 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:02.287+0000 7fa777fff640 1 -- 172.21.15.29:0/2068692437 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560429f06210 con 0x56042a28da50 2024-03-16T12:04:02.292 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:02.287+0000 7fa777fff640 1 --2- 172.21.15.29:0/2068692437 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56042a28da50 0x56042a293d00 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7fa76c019a30 tx=0x7fa76c005850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c13430b30e050369 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:04:02.292 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:02.287+0000 7fa757fff640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-16T12:04:02.292 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:02.287+0000 7fa757fff640 1 -- 172.21.15.29:0/2068692437 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa76c029020 con 0x56042a28da50 2024-03-16T12:04:02.292 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:02.287+0000 7fa757fff640 1 -- 172.21.15.29:0/2068692437 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(108..108 src has 1..108) ==== 8512+0+0 (secure 0 0 0) 0x7fa76c005040 con 0x56042a28da50 2024-03-16T12:04:02.292 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:02.287+0000 7fa781e3e740 1 -- 172.21.15.29:0/2068692437 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa76c019b10 con 0x56042a28da50 2024-03-16T12:04:02.292 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:02.288+0000 7fa757fff640 1 -- 172.21.15.29:0/2068692437 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fa76c034050 con 0x56042a28da50 2024-03-16T12:04:02.292 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:02.288+0000 7fa781e3e740 10 client.16497 fetch_fsmap learned FSMap version 30 2024-03-16T12:04:02.292 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:02.288+0000 7fa781e3e740 1 -- 172.21.15.29:0/2068692437 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x560429fd2620 con 0x56042a28da50 2024-03-16T12:04:02.292 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:02.288+0000 7fa757fff640 1 -- 172.21.15.29:0/2068692437 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa76c019b10 con 0x56042a28da50 2024-03-16T12:04:02.292 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:02.288+0000 7fa757fff640 1 -- 172.21.15.29:0/2068692437 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 30) ==== 8338+0+0 (secure 0 0 0) 0x7fa76c00eb90 con 0x56042a28da50 2024-03-16T12:04:02.292 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:02.288+0000 7fa781e3e740 10 client.16497 fetch_fsmap finished waiting for FSMap version 30 2024-03-16T12:04:02.292 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:02.288+0000 7fa781e3e740 10 client.16497 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T12:04:02.293 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:02.288+0000 7fa781e3e740 20 client.16497 populate_metadata read hostname 'smithi029' 2024-03-16T12:04:02.293 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:02.288+0000 7fa781e3e740 1 --2- 172.21.15.29:0/2068692437 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x56042a2a6840 0x56042a2a8c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:04:02.293 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:02.288+0000 7fa781e3e740 4 client.16497 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T12:04:02.293 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:02.288+0000 7fa781e3e740 1 -- 172.21.15.29:0/2068692437 --> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56042a1fef80 con 0x56042a2a6840 2024-03-16T12:04:02.293 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:02.288+0000 7fa77cfee640 1 --2- 172.21.15.29:0/2068692437 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x56042a2a6840 0x56042a2a8c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:04:02.293 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:02.289+0000 7fa757fff640 10 client.16497 ms_handle_connect on v2:172.21.15.152:6838/1548731833 2024-03-16T12:04:02.293 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:02.289+0000 7fa77cfee640 1 --2- 172.21.15.29:0/2068692437 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x56042a2a6840 0x56042a2a8c30 crc :-1 s=READY pgs=156 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a90df3468188207a server_cookie=2a50a0ee7da93f25 in_seq=0 out_seq=0 2024-03-16T12:04:02.293 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:02.289+0000 7fa757fff640 1 -- 172.21.15.29:0/2068692437 <== mds.0 v2:172.21.15.152:6838/1548731833 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x56042a1fef80 con 0x56042a2a6840 2024-03-16T12:04:02.293 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:02.289+0000 7fa757fff640 10 client.16497 handle_command_reply: tid=0 2024-03-16T12:04:02.293 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:02.289+0000 7fa781e3e740 1 client.16497 shutdown 2024-03-16T12:04:02.293 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:02.290+0000 7fa781e3e740 1 -- 172.21.15.29:0/2068692437 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56042a28da50 msgr2=0x56042a293d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:04:02.294 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:02.290+0000 7fa781e3e740 1 --2- 172.21.15.29:0/2068692437 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56042a28da50 0x56042a293d00 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7fa76c019a30 tx=0x7fa76c005850 comp rx=0 tx=0).stop 2024-03-16T12:04:02.294 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:02.290+0000 7fa781e3e740 1 -- 172.21.15.29:0/2068692437 shutdown_connections 2024-03-16T12:04:02.294 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:02.290+0000 7fa781e3e740 1 --2- 172.21.15.29:0/2068692437 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56042a296820 0x56042a294240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:04:02.294 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:02.290+0000 7fa781e3e740 1 --2- 172.21.15.29:0/2068692437 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56042a28da50 0x56042a293d00 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:04:02.294 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:02.290+0000 7fa781e3e740 1 --2- 172.21.15.29:0/2068692437 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x56042a2a6840 0x56042a2a8c30 crc :-1 s=READY pgs=156 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:04:02.294 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:02.290+0000 7fa757fff640 0 client.16497 ms_handle_reset on v2:172.21.15.152:6838/1548731833 2024-03-16T12:04:02.294 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:02.290+0000 7fa781e3e740 1 --2- 172.21.15.29:0/2068692437 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56042a28d130 0x56042a2937c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:04:02.294 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:02.290+0000 7fa781e3e740 1 -- 172.21.15.29:0/2068692437 >> 172.21.15.29:0/2068692437 conn(0x56042a287a50 msgr2=0x56042a2897a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:04:02.294 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:02.290+0000 7fa781e3e740 1 -- 172.21.15.29:0/2068692437 shutdown_connections 2024-03-16T12:04:02.294 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:02.290+0000 7fa781e3e740 1 -- 172.21.15.29:0/2068692437 wait complete. 2024-03-16T12:04:02.294 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:02.290+0000 7fa781e3e740 20 client.16497 trim_cache size 0 max 16384 2024-03-16T12:04:02.305 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:02.305+0000 7fa781e3e740 1 Processor -- start 2024-03-16T12:04:02.305 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:02.305+0000 7fa781e3e740 1 -- start start 2024-03-16T12:04:02.305 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:02.305+0000 7fa781e3e740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56042a28d130 0x56042a295ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:04:02.305 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:02.305+0000 7fa781e3e740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56042a28da50 0x56042a2a21f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:04:02.305 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:02.305+0000 7fa781e3e740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56042a296820 0x56042a289560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:04:02.305 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:02.305+0000 7fa781e3e740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fa7781afcb0 con 0x56042a296820 2024-03-16T12:04:02.305 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:02.305+0000 7fa781e3e740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fa7781ad580 con 0x56042a28d130 2024-03-16T12:04:02.305 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:02.305+0000 7fa781e3e740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fa7781ab3b0 con 0x56042a28da50 2024-03-16T12:04:02.305 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:02.305+0000 7fa77d7ef640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56042a296820 0x56042a289560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:04:02.305 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:02.305+0000 7fa777fff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56042a28da50 0x56042a2a21f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:04:02.305 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:02.305+0000 7fa77d7ef640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56042a296820 0x56042a289560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55802/0 (socket says 172.21.15.29:55802) 2024-03-16T12:04:02.305 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:02.305+0000 7fa77d7ef640 1 -- 172.21.15.29:0/2346289582 learned_addr learned my addr 172.21.15.29:0/2346289582 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T12:04:02.305 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:02.305+0000 7fa77cfee640 1 --2- 172.21.15.29:0/2346289582 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56042a28d130 0x56042a295ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:04:02.305 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:02.305+0000 7fa777fff640 1 -- 172.21.15.29:0/2346289582 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56042a28d130 msgr2=0x56042a295ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:04:02.306 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:02.305+0000 7fa777fff640 1 --2- 172.21.15.29:0/2346289582 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56042a28d130 0x56042a295ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:04:02.306 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:02.305+0000 7fa777fff640 1 -- 172.21.15.29:0/2346289582 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56042a296820 msgr2=0x56042a289560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:04:02.306 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:02.305+0000 7fa777fff640 1 --2- 172.21.15.29:0/2346289582 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56042a296820 0x56042a289560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:04:02.306 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:02.305+0000 7fa777fff640 1 -- 172.21.15.29:0/2346289582 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56042a287880 con 0x56042a28da50 2024-03-16T12:04:02.306 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:02.305+0000 7fa77d7ef640 1 --2- 172.21.15.29:0/2346289582 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56042a296820 0x56042a289560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:04:02.306 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:02.305+0000 7fa777fff640 1 --2- 172.21.15.29:0/2346289582 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56042a28da50 0x56042a2a21f0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fa76c00efb0 tx=0x7fa76c012f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=10e08314c3f9f609 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:04:02.306 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:02.306+0000 7fa75e7fc640 1 -- 172.21.15.29:0/2346289582 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa76c029040 con 0x56042a28da50 2024-03-16T12:04:02.306 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:02.306+0000 7fa75e7fc640 1 -- 172.21.15.29:0/2346289582 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa76c0097a0 con 0x56042a28da50 2024-03-16T12:04:02.306 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:02.306+0000 7fa75e7fc640 1 -- 172.21.15.29:0/2346289582 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa76c01e040 con 0x56042a28da50 2024-03-16T12:04:02.306 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:02.306+0000 7fa781e3e740 1 -- 172.21.15.29:0/2346289582 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56042a28da50 msgr2=0x56042a2a21f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:04:02.306 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:02.306+0000 7fa781e3e740 1 --2- 172.21.15.29:0/2346289582 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56042a28da50 0x56042a2a21f0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fa76c00efb0 tx=0x7fa76c012f90 comp rx=0 tx=0).stop 2024-03-16T12:04:02.306 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:02.306+0000 7fa781e3e740 1 -- 172.21.15.29:0/2346289582 shutdown_connections 2024-03-16T12:04:02.306 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:02.306+0000 7fa781e3e740 1 --2- 172.21.15.29:0/2346289582 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56042a296820 0x56042a289560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:04:02.306 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:02.306+0000 7fa781e3e740 1 --2- 172.21.15.29:0/2346289582 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56042a28da50 0x56042a2a21f0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:04:02.307 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:02.306+0000 7fa781e3e740 1 --2- 172.21.15.29:0/2346289582 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56042a28d130 0x56042a295ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:04:02.307 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:02.306+0000 7fa781e3e740 1 -- 172.21.15.29:0/2346289582 >> 172.21.15.29:0/2346289582 conn(0x56042a287a50 msgr2=0x56042a288e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:04:02.307 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:02.306+0000 7fa781e3e740 1 -- 172.21.15.29:0/2346289582 shutdown_connections 2024-03-16T12:04:02.307 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:02.306+0000 7fa781e3e740 1 -- 172.21.15.29:0/2346289582 wait complete. 2024-03-16T12:04:02.307 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:02.307+0000 7fa781e3e740 1 -- start start 2024-03-16T12:04:02.307 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:02.307+0000 7fa781e3e740 1 Processor -- start 2024-03-16T12:04:02.307 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:02.307+0000 7fa781e3e740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56042a28d130 0x56042a29a820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:04:02.307 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:02.307+0000 7fa781e3e740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56042a28da50 0x56042a29ad60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:04:02.307 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:02.307+0000 7fa781e3e740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56042a296820 0x56042a30b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:04:02.307 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:02.307+0000 7fa781e3e740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fa7781aba40 con 0x56042a28da50 2024-03-16T12:04:02.307 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:02.307+0000 7fa781e3e740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fa7781aceb0 con 0x56042a296820 2024-03-16T12:04:02.307 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:02.307+0000 7fa781e3e740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fa7781ac110 con 0x56042a28d130 2024-03-16T12:04:02.307 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:02.307+0000 7fa781e3e740 10 client.0 _ll_register_callbacks cb 0x7fa76c01c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T12:04:02.307 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:02.307+0000 7fa777fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56042a28da50 0x56042a29ad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:04:02.307 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:02.307+0000 7fa777fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56042a28da50 0x56042a29ad60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55818/0 (socket says 172.21.15.29:55818) 2024-03-16T12:04:02.307 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:02.307+0000 7fa777fff640 1 -- 172.21.15.29:0/2654032100 learned_addr learned my addr 172.21.15.29:0/2654032100 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T12:04:02.308 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:02.307+0000 7fa77cfee640 1 --2- 172.21.15.29:0/2654032100 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56042a28d130 0x56042a29a820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:04:02.308 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:02.307+0000 7fa77d7ef640 1 --2- 172.21.15.29:0/2654032100 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56042a296820 0x56042a30b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:04:02.308 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:02.307+0000 7fa777fff640 1 -- 172.21.15.29:0/2654032100 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56042a28d130 msgr2=0x56042a29a820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:04:02.308 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:02.307+0000 7fa777fff640 1 --2- 172.21.15.29:0/2654032100 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56042a28d130 0x56042a29a820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:04:02.308 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:02.307+0000 7fa777fff640 1 -- 172.21.15.29:0/2654032100 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56042a296820 msgr2=0x56042a30b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:04:02.308 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:02.307+0000 7fa777fff640 1 --2- 172.21.15.29:0/2654032100 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56042a296820 0x56042a30b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:04:02.308 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:02.307+0000 7fa777fff640 1 -- 172.21.15.29:0/2654032100 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56042a30bf10 con 0x56042a28da50 2024-03-16T12:04:02.308 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:02.308+0000 7fa777fff640 1 --2- 172.21.15.29:0/2654032100 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56042a28da50 0x56042a29ad60 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7fa76c004860 tx=0x7fa76c00e5a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f39bbd2f1cfa273 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:04:02.308 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:02.308+0000 7fa757fff640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-16T12:04:02.308 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:02.308+0000 7fa757fff640 1 -- 172.21.15.29:0/2654032100 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa76c03d070 con 0x56042a28da50 2024-03-16T12:04:02.308 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:02.308+0000 7fa757fff640 1 -- 172.21.15.29:0/2654032100 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(108..108 src has 1..108) ==== 8512+0+0 (secure 0 0 0) 0x7fa76c004250 con 0x56042a28da50 2024-03-16T12:04:02.308 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:02.308+0000 7fa781e3e740 1 -- 172.21.15.29:0/2654032100 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa76c0097a0 con 0x56042a28da50 2024-03-16T12:04:02.308 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:02.308+0000 7fa757fff640 1 -- 172.21.15.29:0/2654032100 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fa76c046050 con 0x56042a28da50 2024-03-16T12:04:02.308 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:02.308+0000 7fa781e3e740 10 client.16503 fetch_fsmap learned FSMap version 30 2024-03-16T12:04:02.309 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:02.308+0000 7fa781e3e740 1 -- 172.21.15.29:0/2654032100 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x56042a30d1f0 con 0x56042a28da50 2024-03-16T12:04:02.309 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:02.308+0000 7fa757fff640 1 -- 172.21.15.29:0/2654032100 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa76c0097a0 con 0x56042a28da50 2024-03-16T12:04:02.309 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:02.308+0000 7fa757fff640 1 -- 172.21.15.29:0/2654032100 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 30) ==== 8338+0+0 (secure 0 0 0) 0x7fa76c040020 con 0x56042a28da50 2024-03-16T12:04:02.309 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:02.308+0000 7fa781e3e740 10 client.16503 fetch_fsmap finished waiting for FSMap version 30 2024-03-16T12:04:02.309 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:02.308+0000 7fa781e3e740 10 client.16503 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T12:04:02.309 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:02.308+0000 7fa781e3e740 20 client.16503 populate_metadata read hostname 'smithi029' 2024-03-16T12:04:02.309 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:02.308+0000 7fa781e3e740 1 --2- 172.21.15.29:0/2654032100 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x56042a30d870 0x56042a30fc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:04:02.309 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:02.308+0000 7fa781e3e740 4 client.16503 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-16T12:04:02.309 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:02.308+0000 7fa781e3e740 1 -- 172.21.15.29:0/2654032100 --> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] -- command(tid 0: {"prefix": "flush journal"}) -- 0x56042a1f92d0 con 0x56042a30d870 2024-03-16T12:04:02.309 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:02.309+0000 7fa77cfee640 1 --2- 172.21.15.29:0/2654032100 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x56042a30d870 0x56042a30fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:04:02.309 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:02.309+0000 7fa77cfee640 1 --2- 172.21.15.29:0/2654032100 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x56042a30d870 0x56042a30fc90 crc :-1 s=READY pgs=157 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=77df685a78fc3cdd server_cookie=85a7061d1a6af005 in_seq=0 out_seq=0 2024-03-16T12:04:02.310 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:02.309+0000 7fa757fff640 10 client.16503 ms_handle_connect on v2:172.21.15.152:6838/1548731833 2024-03-16T12:04:02.541 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:04:02 smithi154 ceph-mon[36320]: pgmap v416: 97 pgs: 97 active+clean; 71 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 3.0 MiB/s wr, 277 op/s 2024-03-16T12:04:02.574 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:02.578+0000 7fa757fff640 1 -- 172.21.15.29:0/2654032100 <== mds.0 v2:172.21.15.152:6838/1548731833 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x56042a1f92d0 con 0x56042a30d870 2024-03-16T12:04:02.574 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:02.578+0000 7fa757fff640 10 client.16503 handle_command_reply: tid=0 2024-03-16T12:04:02.574 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:02.578+0000 7fa781e3e740 1 client.16503 shutdown 2024-03-16T12:04:02.574 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:02.578+0000 7fa781e3e740 1 -- 172.21.15.29:0/2654032100 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56042a28da50 msgr2=0x56042a29ad60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:04:02.574 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:02.578+0000 7fa781e3e740 1 --2- 172.21.15.29:0/2654032100 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56042a28da50 0x56042a29ad60 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7fa76c004860 tx=0x7fa76c00e5a0 comp rx=0 tx=0).stop 2024-03-16T12:04:02.574 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:02.578+0000 7fa781e3e740 1 -- 172.21.15.29:0/2654032100 shutdown_connections 2024-03-16T12:04:02.575 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:02.578+0000 7fa781e3e740 1 --2- 172.21.15.29:0/2654032100 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x56042a30d870 0x56042a30fc90 crc :-1 s=READY pgs=157 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:04:02.575 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:02.578+0000 7fa757fff640 0 client.16503 ms_handle_reset on v2:172.21.15.152:6838/1548731833 2024-03-16T12:04:02.575 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:02.578+0000 7fa781e3e740 1 --2- 172.21.15.29:0/2654032100 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56042a296820 0x56042a30b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:04:02.575 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:02.578+0000 7fa781e3e740 1 --2- 172.21.15.29:0/2654032100 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56042a28da50 0x56042a29ad60 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:04:02.575 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:02.578+0000 7fa781e3e740 1 --2- 172.21.15.29:0/2654032100 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56042a28d130 0x56042a29a820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:04:02.575 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:02.578+0000 7fa781e3e740 1 -- 172.21.15.29:0/2654032100 >> 172.21.15.29:0/2654032100 conn(0x56042a287a50 msgr2=0x56042a286bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:04:02.575 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:02.578+0000 7fa781e3e740 1 -- 172.21.15.29:0/2654032100 shutdown_connections 2024-03-16T12:04:02.575 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:02.578+0000 7fa781e3e740 1 -- 172.21.15.29:0/2654032100 wait complete. 2024-03-16T12:04:02.575 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:02.578+0000 7fa781e3e740 20 client.16503 trim_cache size 0 max 16384 2024-03-16T12:04:02.575 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:{ 2024-03-16T12:04:02.575 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "message": "", 2024-03-16T12:04:02.576 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "return_code": 0 2024-03-16T12:04:02.576 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:} 2024-03-16T12:04:02.576 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:02.579+0000 7fa77f252640 1 -- 172.21.15.29:0/3185638895 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7fa7600a2a10 msgr2=0x7fa7600a4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:04:02.576 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:02.579+0000 7fa77f252640 1 --2- 172.21.15.29:0/3185638895 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7fa7600a2a10 0x7fa7600a4ed0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fa770007310 tx=0x7fa770004ab0 comp rx=0 tx=0).stop 2024-03-16T12:04:02.576 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:02.579+0000 7fa77f252640 1 -- 172.21.15.29:0/3185638895 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa7781017b0 msgr2=0x7fa778197f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:04:02.577 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:02.579+0000 7fa77f252640 1 --2- 172.21.15.29:0/3185638895 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa7781017b0 0x7fa778197f80 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7fa768009900 tx=0x7fa768004360 comp rx=0 tx=0).stop 2024-03-16T12:04:02.577 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:02.579+0000 7fa77f252640 1 -- 172.21.15.29:0/3185638895 shutdown_connections 2024-03-16T12:04:02.577 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:02.579+0000 7fa77f252640 1 --2- 172.21.15.29:0/3185638895 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7fa7600a2a10 0x7fa7600a4ed0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:04:02.577 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:02.579+0000 7fa77f252640 1 --2- 172.21.15.29:0/3185638895 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa7781077c0 0x7fa77819c840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:04:02.577 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:02.579+0000 7fa77f252640 1 --2- 172.21.15.29:0/3185638895 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa778102170 0x7fa7781984c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:04:02.577 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:02.579+0000 7fa77f252640 1 --2- 172.21.15.29:0/3185638895 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa7781017b0 0x7fa778197f80 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:04:02.577 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:02.579+0000 7fa77f252640 1 -- 172.21.15.29:0/3185638895 >> 172.21.15.29:0/3185638895 conn(0x7fa7780fd2c0 msgr2=0x7fa7780fd7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:04:02.577 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:02.580+0000 7fa77f252640 1 -- 172.21.15.29:0/3185638895 shutdown_connections 2024-03-16T12:04:02.577 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:02.580+0000 7fa77f252640 1 -- 172.21.15.29:0/3185638895 wait complete. 2024-03-16T12:04:02.584 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:04:02 smithi029 ceph-mon[31975]: pgmap v416: 97 pgs: 97 active+clean; 71 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 3.0 MiB/s wr, 277 op/s 2024-03-16T12:04:02.623 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:04:02 smithi152 ceph-mon[36291]: pgmap v416: 97 pgs: 97 active+clean; 71 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 3.0 MiB/s wr, 277 op/s 2024-03-16T12:04:02.807 INFO:tasks.workunit.client.0.smithi029.stdout: 1 1253723 9.08 MB/sec execute 587 sec latency 173.153 ms 2024-03-16T12:04:03.807 INFO:tasks.workunit.client.0.smithi029.stdout: 1 1254472 9.07 MB/sec execute 588 sec latency 50.549 ms 2024-03-16T12:04:04.541 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:04:04 smithi154 ceph-mon[36320]: pgmap v417: 97 pgs: 97 active+clean; 74 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 2.6 MiB/s wr, 209 op/s 2024-03-16T12:04:04.584 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:04:04 smithi029 ceph-mon[31975]: pgmap v417: 97 pgs: 97 active+clean; 74 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 2.6 MiB/s wr, 209 op/s 2024-03-16T12:04:04.623 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:04:04 smithi152 ceph-mon[36291]: pgmap v417: 97 pgs: 97 active+clean; 74 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 2.6 MiB/s wr, 209 op/s 2024-03-16T12:04:04.807 INFO:tasks.workunit.client.0.smithi029.stdout: 1 1255361 9.06 MB/sec execute 589 sec latency 134.517 ms 2024-03-16T12:04:05.541 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:04:05 smithi154 ceph-mon[36320]: osdmap e109: 12 total, 12 up, 12 in 2024-03-16T12:04:05.583 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:04:05 smithi029 ceph-mon[31975]: osdmap e109: 12 total, 12 up, 12 in 2024-03-16T12:04:05.623 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:04:05 smithi152 ceph-mon[36291]: osdmap e109: 12 total, 12 up, 12 in 2024-03-16T12:04:05.807 INFO:tasks.workunit.client.0.smithi029.stdout: 1 1256012 9.05 MB/sec execute 590 sec latency 93.593 ms 2024-03-16T12:04:06.541 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:04:06 smithi154 ceph-mon[36320]: pgmap v419: 97 pgs: 97 active+clean; 74 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 2.6 MiB/s wr, 204 op/s 2024-03-16T12:04:06.584 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:04:06 smithi029 ceph-mon[31975]: pgmap v419: 97 pgs: 97 active+clean; 74 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 2.6 MiB/s wr, 204 op/s 2024-03-16T12:04:06.623 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:04:06 smithi152 ceph-mon[36291]: pgmap v419: 97 pgs: 97 active+clean; 74 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 2.6 MiB/s wr, 204 op/s 2024-03-16T12:04:06.807 INFO:tasks.workunit.client.0.smithi029.stdout: 1 1257206 9.05 MB/sec execute 591 sec latency 38.043 ms 2024-03-16T12:04:07.807 INFO:tasks.workunit.client.0.smithi029.stdout: 1 1257764 9.04 MB/sec execute 592 sec latency 75.119 ms 2024-03-16T12:04:08.541 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:04:08 smithi154 ceph-mon[36320]: pgmap v420: 97 pgs: 14 active+clean+snaptrim, 31 active+clean+snaptrim_wait, 52 active+clean; 64 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.6 MiB/s wr, 278 op/s 2024-03-16T12:04:08.583 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:04:08 smithi029 ceph-mon[31975]: pgmap v420: 97 pgs: 14 active+clean+snaptrim, 31 active+clean+snaptrim_wait, 52 active+clean; 64 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.6 MiB/s wr, 278 op/s 2024-03-16T12:04:08.623 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:04:08 smithi152 ceph-mon[36291]: pgmap v420: 97 pgs: 14 active+clean+snaptrim, 31 active+clean+snaptrim_wait, 52 active+clean; 64 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.6 MiB/s wr, 278 op/s 2024-03-16T12:04:08.807 INFO:tasks.workunit.client.0.smithi029.stdout: 1 1259347 9.03 MB/sec execute 593 sec latency 28.554 ms 2024-03-16T12:04:09.807 INFO:tasks.workunit.client.0.smithi029.stdout: 1 1260903 9.04 MB/sec execute 594 sec latency 34.528 ms 2024-03-16T12:04:10.541 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:04:10 smithi154 ceph-mon[36320]: pgmap v421: 97 pgs: 14 active+clean+snaptrim, 31 active+clean+snaptrim_wait, 52 active+clean; 64 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.6 MiB/s wr, 278 op/s 2024-03-16T12:04:10.584 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:04:10 smithi029 ceph-mon[31975]: pgmap v421: 97 pgs: 14 active+clean+snaptrim, 31 active+clean+snaptrim_wait, 52 active+clean; 64 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.6 MiB/s wr, 278 op/s 2024-03-16T12:04:10.623 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:04:10 smithi152 ceph-mon[36291]: pgmap v421: 97 pgs: 14 active+clean+snaptrim, 31 active+clean+snaptrim_wait, 52 active+clean; 64 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.6 MiB/s wr, 278 op/s 2024-03-16T12:04:10.807 INFO:tasks.workunit.client.0.smithi029.stdout: 1 1262261 9.03 MB/sec execute 595 sec latency 29.114 ms 2024-03-16T12:04:11.807 INFO:tasks.workunit.client.0.smithi029.stdout: 1 1264307 9.04 MB/sec execute 596 sec latency 18.898 ms 2024-03-16T12:04:12.541 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:04:12 smithi154 ceph-mon[36320]: pgmap v422: 97 pgs: 2 active+clean+snaptrim, 4 active+clean+snaptrim_wait, 91 active+clean; 69 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.9 MiB/s wr, 271 op/s 2024-03-16T12:04:12.584 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:04:12 smithi029 ceph-mon[31975]: pgmap v422: 97 pgs: 2 active+clean+snaptrim, 4 active+clean+snaptrim_wait, 91 active+clean; 69 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.9 MiB/s wr, 271 op/s 2024-03-16T12:04:12.623 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:04:12 smithi152 ceph-mon[36291]: pgmap v422: 97 pgs: 2 active+clean+snaptrim, 4 active+clean+snaptrim_wait, 91 active+clean; 69 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.9 MiB/s wr, 271 op/s 2024-03-16T12:04:12.807 INFO:tasks.workunit.client.0.smithi029.stdout: 1 1265020 9.03 MB/sec execute 597 sec latency 65.965 ms 2024-03-16T12:04:13.807 INFO:tasks.workunit.client.0.smithi029.stdout: 1 1266716 9.02 MB/sec execute 598 sec latency 123.345 ms 2024-03-16T12:04:14.541 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:04:14 smithi154 ceph-mon[36320]: pgmap v423: 97 pgs: 97 active+clean; 69 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.6 MiB/s wr, 242 op/s 2024-03-16T12:04:14.583 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:04:14 smithi029 ceph-mon[31975]: pgmap v423: 97 pgs: 97 active+clean; 69 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.6 MiB/s wr, 242 op/s 2024-03-16T12:04:14.623 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:04:14 smithi152 ceph-mon[36291]: pgmap v423: 97 pgs: 97 active+clean; 69 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.6 MiB/s wr, 242 op/s 2024-03-16T12:04:14.807 INFO:tasks.workunit.client.0.smithi029.stdout: 1 1267988 9.03 MB/sec execute 599 sec latency 72.915 ms 2024-03-16T12:04:15.668 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:15.671+0000 7f365e1c6640 1 -- 172.21.15.29:0/1348446006 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f36581012f0 msgr2=0x7f365810d510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:04:15.668 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:15.671+0000 7f365e1c6640 1 --2- 172.21.15.29:0/1348446006 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f36581012f0 0x7f365810d510 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f3650009ca0 tx=0x7f365002c0f0 comp rx=0 tx=0).stop 2024-03-16T12:04:15.668 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:15.671+0000 7f365e1c6640 1 -- 172.21.15.29:0/1348446006 shutdown_connections 2024-03-16T12:04:15.668 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:15.671+0000 7f365e1c6640 1 --2- 172.21.15.29:0/1348446006 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f36581012f0 0x7f365810d510 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:04:15.668 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:15.671+0000 7f365e1c6640 1 --2- 172.21.15.29:0/1348446006 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3658100930 0x7f3658100db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:04:15.669 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:15.671+0000 7f365e1c6640 1 --2- 172.21.15.29:0/1348446006 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3658105680 0x7f3658105a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:04:15.669 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:15.671+0000 7f365e1c6640 1 -- 172.21.15.29:0/1348446006 >> 172.21.15.29:0/1348446006 conn(0x7f3658077530 msgr2=0x7f3658077940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:04:15.669 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:15.671+0000 7f365e1c6640 1 -- 172.21.15.29:0/1348446006 shutdown_connections 2024-03-16T12:04:15.669 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:15.671+0000 7f365e1c6640 1 -- 172.21.15.29:0/1348446006 wait complete. 2024-03-16T12:04:15.669 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:15.672+0000 7f365e1c6640 1 Processor -- start 2024-03-16T12:04:15.669 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:15.672+0000 7f365e1c6640 1 -- start start 2024-03-16T12:04:15.669 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:15.672+0000 7f365e1c6640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3658100930 0x7f3658196f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:04:15.669 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:15.672+0000 7f365e1c6640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f36581012f0 0x7f3658197440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:04:15.669 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:15.672+0000 7f365e1c6640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3658105680 0x7f365819b7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:04:15.669 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:15.672+0000 7f365e1c6640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f3658110ba0 con 0x7f36581012f0 2024-03-16T12:04:15.669 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:15.672+0000 7f365e1c6640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f3658110a20 con 0x7f3658105680 2024-03-16T12:04:15.669 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:15.672+0000 7f365e1c6640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f3658110d20 con 0x7f3658100930 2024-03-16T12:04:15.669 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:15.672+0000 7f3656ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f36581012f0 0x7f3658197440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:04:15.670 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:15.672+0000 7f36577fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3658100930 0x7f3658196f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:04:15.670 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:15.672+0000 7f3656ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f36581012f0 0x7f3658197440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50402/0 (socket says 172.21.15.29:50402) 2024-03-16T12:04:15.670 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:15.672+0000 7f3656ffd640 1 -- 172.21.15.29:0/2823994824 learned_addr learned my addr 172.21.15.29:0/2823994824 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T12:04:15.670 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:15.672+0000 7f3657fff640 1 --2- 172.21.15.29:0/2823994824 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3658105680 0x7f365819b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:04:15.670 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:15.673+0000 7f3656ffd640 1 -- 172.21.15.29:0/2823994824 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3658100930 msgr2=0x7f3658196f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:04:15.670 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:15.673+0000 7f3656ffd640 1 --2- 172.21.15.29:0/2823994824 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3658100930 0x7f3658196f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:04:15.670 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:15.673+0000 7f3656ffd640 1 -- 172.21.15.29:0/2823994824 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3658105680 msgr2=0x7f365819b7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:04:15.670 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:15.673+0000 7f3656ffd640 1 --2- 172.21.15.29:0/2823994824 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3658105680 0x7f365819b7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:04:15.670 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:15.673+0000 7f3656ffd640 1 -- 172.21.15.29:0/2823994824 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f365819bef0 con 0x7f36581012f0 2024-03-16T12:04:15.670 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:15.673+0000 7f3656ffd640 1 --2- 172.21.15.29:0/2823994824 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f36581012f0 0x7f3658197440 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f364c00d950 tx=0x7f364c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:04:15.670 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:15.673+0000 7f3654ff9640 1 -- 172.21.15.29:0/2823994824 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f364c017030 con 0x7f36581012f0 2024-03-16T12:04:15.670 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:15.673+0000 7f3654ff9640 1 -- 172.21.15.29:0/2823994824 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f364c004020 con 0x7f36581012f0 2024-03-16T12:04:15.670 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:15.673+0000 7f365e1c6640 1 -- 172.21.15.29:0/2823994824 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f365819c1e0 con 0x7f36581012f0 2024-03-16T12:04:15.670 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:15.673+0000 7f3654ff9640 1 -- 172.21.15.29:0/2823994824 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f364c0047a0 con 0x7f36581012f0 2024-03-16T12:04:15.671 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:15.673+0000 7f365e1c6640 1 -- 172.21.15.29:0/2823994824 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f36581a3a70 con 0x7f36581012f0 2024-03-16T12:04:15.676 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:15.675+0000 7f3654ff9640 1 -- 172.21.15.29:0/2823994824 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7f364c0041d0 con 0x7f36581012f0 2024-03-16T12:04:15.676 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:15.676+0000 7f3660db1740 1 Processor -- start 2024-03-16T12:04:15.676 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:15.676+0000 7f3660db1740 1 -- start start 2024-03-16T12:04:15.676 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:15.676+0000 7f3654ff9640 1 --2- 172.21.15.29:0/2823994824 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f36400a2a10 0x7f36400a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:04:15.676 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:15.676+0000 7f3660db1740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55639e05d340 0x55639e05d720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:04:15.677 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:15.676+0000 7f3660db1740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55639e05dc60 0x55639e0664f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:04:15.677 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:15.676+0000 7f3660db1740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55639e066a30 0x55639e068e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:04:15.677 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:15.676+0000 7f3660db1740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55639dfbfa40 con 0x55639e05d340 2024-03-16T12:04:15.677 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:15.676+0000 7f3660db1740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x55639df68920 con 0x55639e05dc60 2024-03-16T12:04:15.677 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:15.676+0000 7f3660db1740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55639dfbbae0 con 0x55639e066a30 2024-03-16T12:04:15.677 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:15.676+0000 7f3654ff9640 1 -- 172.21.15.29:0/2823994824 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 8600+0+0 (secure 0 0 0) 0x7f364c0ce860 con 0x7f36581012f0 2024-03-16T12:04:15.677 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:15.676+0000 7f3657fff640 1 --2- 172.21.15.29:0/2823994824 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f36400a2a10 0x7f36400a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:04:15.677 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:15.676+0000 7f36577fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55639e05d340 0x55639e05d720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:04:15.677 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:15.676+0000 7f36577fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55639e05d340 0x55639e05d720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50406/0 (socket says 172.21.15.29:50406) 2024-03-16T12:04:15.677 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:15.676+0000 7f36577fe640 1 -- 172.21.15.29:0/2802358310 learned_addr learned my addr 172.21.15.29:0/2802358310 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T12:04:15.677 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:15.676+0000 7f3656ffd640 1 --2- 172.21.15.29:0/2802358310 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55639e05dc60 0x55639e0664f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:04:15.677 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:15.676+0000 7f3657fff640 1 --2- 172.21.15.29:0/2802358310 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55639e066a30 0x55639e068e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:04:15.677 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:15.676+0000 7f36577fe640 1 -- 172.21.15.29:0/2802358310 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55639e066a30 msgr2=0x55639e068e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:04:15.677 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:15.676+0000 7f36577fe640 1 --2- 172.21.15.29:0/2802358310 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55639e066a30 0x55639e068e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:04:15.678 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:15.676+0000 7f36577fe640 1 -- 172.21.15.29:0/2802358310 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55639e05dc60 msgr2=0x55639e0664f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:04:15.678 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:15.676+0000 7f36577fe640 1 --2- 172.21.15.29:0/2802358310 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55639e05dc60 0x55639e0664f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:04:15.678 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:15.676+0000 7f36577fe640 1 -- 172.21.15.29:0/2802358310 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55639df80460 con 0x55639e05d340 2024-03-16T12:04:15.678 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:15.676+0000 7f3656ffd640 1 --2- 172.21.15.29:0/2802358310 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55639e05dc60 0x55639e0664f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:04:15.678 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:15.677+0000 7f3657fff640 1 --2- 172.21.15.29:0/2823994824 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f36400a2a10 0x7f36400a4ed0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f3650004af0 tx=0x7f36500323c0 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:04:15.678 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:15.677+0000 7f36577fe640 1 --2- 172.21.15.29:0/2802358310 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55639e05d340 0x55639e05d720 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f3648009ca0 tx=0x7f3648011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4eda25af6c590ab6 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:04:15.678 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:15.677+0000 7f363dffb640 1 -- 172.21.15.29:0/2802358310 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3648019620 con 0x55639e05d340 2024-03-16T12:04:15.678 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:15.677+0000 7f363dffb640 1 -- 172.21.15.29:0/2802358310 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3648019ca0 con 0x55639e05d340 2024-03-16T12:04:15.678 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:15.677+0000 7f363dffb640 1 -- 172.21.15.29:0/2802358310 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3648018bf0 con 0x55639e05d340 2024-03-16T12:04:15.678 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:15.677+0000 7f3660db1740 1 -- 172.21.15.29:0/2802358310 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55639e05d340 msgr2=0x55639e05d720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:04:15.678 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:15.677+0000 7f3660db1740 1 --2- 172.21.15.29:0/2802358310 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55639e05d340 0x55639e05d720 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f3648009ca0 tx=0x7f3648011810 comp rx=0 tx=0).stop 2024-03-16T12:04:15.678 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:15.677+0000 7f3660db1740 1 -- 172.21.15.29:0/2802358310 shutdown_connections 2024-03-16T12:04:15.678 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:15.678+0000 7f3660db1740 1 --2- 172.21.15.29:0/2802358310 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55639e066a30 0x55639e068e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:04:15.678 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:15.678+0000 7f3660db1740 1 --2- 172.21.15.29:0/2802358310 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55639e05dc60 0x55639e0664f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:04:15.679 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:15.678+0000 7f3660db1740 1 --2- 172.21.15.29:0/2802358310 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55639e05d340 0x55639e05d720 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:04:15.679 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:15.678+0000 7f3660db1740 1 -- 172.21.15.29:0/2802358310 >> 172.21.15.29:0/2802358310 conn(0x55639e057c60 msgr2=0x55639e058040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:04:15.679 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:15.678+0000 7f3660db1740 1 -- 172.21.15.29:0/2802358310 shutdown_connections 2024-03-16T12:04:15.679 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:15.678+0000 7f3660db1740 1 -- 172.21.15.29:0/2802358310 wait complete. 2024-03-16T12:04:15.679 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:15.678+0000 7f3660db1740 1 -- start start 2024-03-16T12:04:15.679 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:15.678+0000 7f3660db1740 1 Processor -- start 2024-03-16T12:04:15.679 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:15.678+0000 7f3660db1740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55639e05d340 0x55639e0639c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:04:15.679 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:15.678+0000 7f3660db1740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55639e05dc60 0x55639e063f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:04:15.679 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:15.678+0000 7f3660db1740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55639e066a30 0x55639e064440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:04:15.679 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:15.678+0000 7f3660db1740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55639e069660 con 0x55639e066a30 2024-03-16T12:04:15.679 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:15.678+0000 7f3660db1740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x55639df7bd70 con 0x55639e05dc60 2024-03-16T12:04:15.679 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:15.678+0000 7f3660db1740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55639e0697e0 con 0x55639e05d340 2024-03-16T12:04:15.679 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:15.678+0000 7f3660db1740 10 client.0 _ll_register_callbacks cb 0x55639dcd3a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T12:04:15.679 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:15.679+0000 7f3657fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55639e066a30 0x55639e064440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:04:15.680 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:15.679+0000 7f36577fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55639e05d340 0x55639e0639c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:04:15.680 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:15.679+0000 7f3656ffd640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55639e05dc60 0x55639e063f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:04:15.680 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:15.679+0000 7f3657fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55639e066a30 0x55639e064440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50414/0 (socket says 172.21.15.29:50414) 2024-03-16T12:04:15.680 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:15.679+0000 7f3657fff640 1 -- 172.21.15.29:0/3578169348 learned_addr learned my addr 172.21.15.29:0/3578169348 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T12:04:15.680 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:15.679+0000 7f36577fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55639e05d340 0x55639e0639c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.29:50648/0 (socket says 172.21.15.29:50648) 2024-03-16T12:04:15.680 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:15.679+0000 7f3657fff640 1 -- 172.21.15.29:0/3578169348 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55639e05d340 msgr2=0x55639e0639c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:04:15.680 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:15.679+0000 7f3657fff640 1 --2- 172.21.15.29:0/3578169348 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55639e05d340 0x55639e0639c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:04:15.680 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:15.679+0000 7f3657fff640 1 -- 172.21.15.29:0/3578169348 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55639e05dc60 msgr2=0x55639e063f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:04:15.680 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:15.679+0000 7f3657fff640 1 --2- 172.21.15.29:0/3578169348 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55639e05dc60 0x55639e063f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:04:15.680 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:15.679+0000 7f3657fff640 1 -- 172.21.15.29:0/3578169348 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55639dcd4210 con 0x55639e066a30 2024-03-16T12:04:15.680 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:15.679+0000 7f3657fff640 1 --2- 172.21.15.29:0/3578169348 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55639e066a30 0x55639e064440 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f3650009ca0 tx=0x7f365002c0f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec287dbdfc10574c server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:04:15.680 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:15.679+0000 7f3636ffd640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-16T12:04:15.680 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:15.679+0000 7f3636ffd640 1 -- 172.21.15.29:0/3578169348 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3650039040 con 0x55639e066a30 2024-03-16T12:04:15.680 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:15.679+0000 7f3636ffd640 1 -- 172.21.15.29:0/3578169348 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(109..109 src has 1..109) ==== 8600+0+0 (secure 0 0 0) 0x7f3650003af0 con 0x55639e066a30 2024-03-16T12:04:15.681 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:15.679+0000 7f3660db1740 1 -- 172.21.15.29:0/3578169348 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3648019ca0 con 0x55639e066a30 2024-03-16T12:04:15.681 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:15.680+0000 7f3636ffd640 1 -- 172.21.15.29:0/3578169348 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f365005b050 con 0x55639e066a30 2024-03-16T12:04:15.681 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:15.680+0000 7f3660db1740 10 client.16515 fetch_fsmap learned FSMap version 30 2024-03-16T12:04:15.681 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:15.680+0000 7f3660db1740 1 -- 172.21.15.29:0/3578169348 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55639dda0620 con 0x55639e066a30 2024-03-16T12:04:15.681 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:15.680+0000 7f3636ffd640 1 -- 172.21.15.29:0/3578169348 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3648019ca0 con 0x55639e066a30 2024-03-16T12:04:15.681 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:15.680+0000 7f3636ffd640 1 -- 172.21.15.29:0/3578169348 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 30) ==== 8338+0+0 (secure 0 0 0) 0x7f365004d440 con 0x55639e066a30 2024-03-16T12:04:15.681 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:15.680+0000 7f3660db1740 10 client.16515 fetch_fsmap finished waiting for FSMap version 30 2024-03-16T12:04:15.681 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:15.680+0000 7f3660db1740 10 client.16515 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T12:04:15.681 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:15.680+0000 7f3660db1740 20 client.16515 populate_metadata read hostname 'smithi029' 2024-03-16T12:04:15.681 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:15.680+0000 7f3660db1740 1 --2- 172.21.15.29:0/3578169348 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55639e076b60 0x55639e078f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:04:15.681 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:15.680+0000 7f3660db1740 4 client.16515 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T12:04:15.681 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:15.680+0000 7f3660db1740 1 -- 172.21.15.29:0/3578169348 --> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55639dfccf80 con 0x55639e076b60 2024-03-16T12:04:15.681 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:15.680+0000 7f36577fe640 1 --2- 172.21.15.29:0/3578169348 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55639e076b60 0x55639e078f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:04:15.681 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:15.681+0000 7f36577fe640 1 --2- 172.21.15.29:0/3578169348 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55639e076b60 0x55639e078f70 crc :-1 s=READY pgs=158 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3bf38ae9f47ca66f server_cookie=b8ea9565fe452c5a in_seq=0 out_seq=0 2024-03-16T12:04:15.681 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:15.681+0000 7f3636ffd640 10 client.16515 ms_handle_connect on v2:172.21.15.152:6838/1548731833 2024-03-16T12:04:15.683 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:15.682+0000 7f3636ffd640 1 -- 172.21.15.29:0/3578169348 <== mds.0 v2:172.21.15.152:6838/1548731833 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55639dfccf80 con 0x55639e076b60 2024-03-16T12:04:15.683 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:15.682+0000 7f3636ffd640 10 client.16515 handle_command_reply: tid=0 2024-03-16T12:04:15.683 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:15.682+0000 7f3660db1740 1 client.16515 shutdown 2024-03-16T12:04:15.683 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:15.683+0000 7f3660db1740 1 -- 172.21.15.29:0/3578169348 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55639e066a30 msgr2=0x55639e064440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:04:15.683 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:15.683+0000 7f3660db1740 1 --2- 172.21.15.29:0/3578169348 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55639e066a30 0x55639e064440 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f3650009ca0 tx=0x7f365002c0f0 comp rx=0 tx=0).stop 2024-03-16T12:04:15.683 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:15.683+0000 7f3660db1740 1 -- 172.21.15.29:0/3578169348 shutdown_connections 2024-03-16T12:04:15.683 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:15.683+0000 7f3660db1740 1 --2- 172.21.15.29:0/3578169348 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55639e066a30 0x55639e064440 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:04:15.683 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:15.683+0000 7f3660db1740 1 --2- 172.21.15.29:0/3578169348 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55639e076b60 0x55639e078f70 crc :-1 s=READY pgs=158 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:04:15.683 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:15.683+0000 7f3660db1740 1 --2- 172.21.15.29:0/3578169348 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55639e05dc60 0x55639e063f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:04:15.683 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:15.683+0000 7f3660db1740 1 --2- 172.21.15.29:0/3578169348 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55639e05d340 0x55639e0639c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:04:15.683 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:15.683+0000 7f3636ffd640 0 client.16515 ms_handle_reset on v2:172.21.15.152:6838/1548731833 2024-03-16T12:04:15.683 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:15.683+0000 7f3660db1740 1 -- 172.21.15.29:0/3578169348 >> 172.21.15.29:0/3578169348 conn(0x55639e057c60 msgr2=0x55639e054bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:04:15.684 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:15.683+0000 7f3660db1740 1 -- 172.21.15.29:0/3578169348 shutdown_connections 2024-03-16T12:04:15.684 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:15.683+0000 7f3660db1740 1 -- 172.21.15.29:0/3578169348 wait complete. 2024-03-16T12:04:15.684 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:15.683+0000 7f3660db1740 20 client.16515 trim_cache size 0 max 16384 2024-03-16T12:04:15.694 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:15.698+0000 7f3660db1740 1 Processor -- start 2024-03-16T12:04:15.694 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:15.698+0000 7f3660db1740 1 -- start start 2024-03-16T12:04:15.694 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:15.698+0000 7f3660db1740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55639e05d340 0x55639e0660f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:04:15.694 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:15.698+0000 7f3660db1740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55639e05dc60 0x55639e072400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:04:15.694 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:15.698+0000 7f3660db1740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55639e066a30 0x55639e07e880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:04:15.694 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:15.698+0000 7f3660db1740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f36581adb20 con 0x55639e05d340 2024-03-16T12:04:15.694 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:15.698+0000 7f3660db1740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f36581ab3f0 con 0x55639e05dc60 2024-03-16T12:04:15.694 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:15.698+0000 7f3660db1740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f36581a9260 con 0x55639e066a30 2024-03-16T12:04:15.694 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:15.698+0000 7f36577fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55639e05d340 0x55639e0660f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:04:15.694 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:15.698+0000 7f3656ffd640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55639e05dc60 0x55639e072400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:04:15.694 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:15.698+0000 7f36577fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55639e05d340 0x55639e0660f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50420/0 (socket says 172.21.15.29:50420) 2024-03-16T12:04:15.694 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:15.698+0000 7f3657fff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55639e066a30 0x55639e07e880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:04:15.695 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:15.698+0000 7f36577fe640 1 -- 172.21.15.29:0/3714829186 learned_addr learned my addr 172.21.15.29:0/3714829186 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T12:04:15.695 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:15.698+0000 7f36577fe640 1 -- 172.21.15.29:0/3714829186 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55639e066a30 msgr2=0x55639e07e880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:04:15.695 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:15.698+0000 7f36577fe640 1 --2- 172.21.15.29:0/3714829186 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55639e066a30 0x55639e07e880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:04:15.695 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:15.698+0000 7f36577fe640 1 -- 172.21.15.29:0/3714829186 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55639e05dc60 msgr2=0x55639e072400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:04:15.695 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:15.698+0000 7f36577fe640 1 --2- 172.21.15.29:0/3714829186 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55639e05dc60 0x55639e072400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:04:15.695 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:15.698+0000 7f36577fe640 1 -- 172.21.15.29:0/3714829186 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55639e05da90 con 0x55639e05d340 2024-03-16T12:04:15.695 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:15.698+0000 7f3657fff640 1 --2- 172.21.15.29:0/3714829186 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55639e066a30 0x55639e07e880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:04:15.695 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:15.698+0000 7f3656ffd640 1 --2- 172.21.15.29:0/3714829186 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55639e05dc60 0x55639e072400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:04:15.695 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:15.698+0000 7f36577fe640 1 --2- 172.21.15.29:0/3714829186 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55639e05d340 0x55639e0660f0 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f364801b040 tx=0x7f3648009910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea6fffdd94521123 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:04:15.695 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:15.699+0000 7f363d7fa640 1 -- 172.21.15.29:0/3714829186 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3648030070 con 0x55639e05d340 2024-03-16T12:04:15.695 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:15.699+0000 7f363d7fa640 1 -- 172.21.15.29:0/3714829186 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3648005040 con 0x55639e05d340 2024-03-16T12:04:15.695 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:15.699+0000 7f363d7fa640 1 -- 172.21.15.29:0/3714829186 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3648004840 con 0x55639e05d340 2024-03-16T12:04:15.699 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:15.699+0000 7f3660db1740 1 -- 172.21.15.29:0/3714829186 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55639e05d340 msgr2=0x55639e0660f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:04:15.699 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:15.699+0000 7f3660db1740 1 --2- 172.21.15.29:0/3714829186 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55639e05d340 0x55639e0660f0 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f364801b040 tx=0x7f3648009910 comp rx=0 tx=0).stop 2024-03-16T12:04:15.699 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:15.699+0000 7f3660db1740 1 -- 172.21.15.29:0/3714829186 shutdown_connections 2024-03-16T12:04:15.699 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:15.699+0000 7f3660db1740 1 --2- 172.21.15.29:0/3714829186 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55639e066a30 0x55639e07e880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:04:15.699 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:15.699+0000 7f3660db1740 1 --2- 172.21.15.29:0/3714829186 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55639e05dc60 0x55639e072400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:04:15.699 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:15.699+0000 7f3660db1740 1 --2- 172.21.15.29:0/3714829186 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55639e05d340 0x55639e0660f0 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:04:15.699 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:15.699+0000 7f3660db1740 1 -- 172.21.15.29:0/3714829186 >> 172.21.15.29:0/3714829186 conn(0x55639e057c60 msgr2=0x55639e054bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:04:15.700 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:15.699+0000 7f3660db1740 1 -- 172.21.15.29:0/3714829186 shutdown_connections 2024-03-16T12:04:15.700 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:15.699+0000 7f3660db1740 1 -- 172.21.15.29:0/3714829186 wait complete. 2024-03-16T12:04:15.700 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:15.700+0000 7f3660db1740 1 -- start start 2024-03-16T12:04:15.700 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:15.700+0000 7f3660db1740 1 Processor -- start 2024-03-16T12:04:15.700 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:15.700+0000 7f3660db1740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55639e05d340 0x55639e06eac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:04:15.700 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:15.700+0000 7f3660db1740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55639e05dc60 0x55639e06ae90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:04:15.700 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:15.700+0000 7f3660db1740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55639e066a30 0x55639e06b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:04:15.700 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:15.700+0000 7f3660db1740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f36581a98b0 con 0x55639e05d340 2024-03-16T12:04:15.700 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:15.700+0000 7f3660db1740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f36581aad20 con 0x55639e066a30 2024-03-16T12:04:15.700 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:15.700+0000 7f3660db1740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f36581a9f80 con 0x55639e05dc60 2024-03-16T12:04:15.700 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:15.700+0000 7f3660db1740 10 client.0 _ll_register_callbacks cb 0x7f3648005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T12:04:15.700 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:15.700+0000 7f36577fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55639e05d340 0x55639e06eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:04:15.700 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:15.700+0000 7f36577fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55639e05d340 0x55639e06eac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50432/0 (socket says 172.21.15.29:50432) 2024-03-16T12:04:15.700 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:15.700+0000 7f36577fe640 1 -- 172.21.15.29:0/1498526476 learned_addr learned my addr 172.21.15.29:0/1498526476 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T12:04:15.701 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:15.700+0000 7f3657fff640 1 --2- 172.21.15.29:0/1498526476 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55639e066a30 0x55639e06b3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:04:15.701 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:15.700+0000 7f3656ffd640 1 --2- 172.21.15.29:0/1498526476 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55639e05dc60 0x55639e06ae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:04:15.701 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:15.701+0000 7f36577fe640 1 -- 172.21.15.29:0/1498526476 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55639e05dc60 msgr2=0x55639e06ae90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:04:15.701 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:15.701+0000 7f36577fe640 1 --2- 172.21.15.29:0/1498526476 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55639e05dc60 0x55639e06ae90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:04:15.701 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:15.701+0000 7f36577fe640 1 -- 172.21.15.29:0/1498526476 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55639e066a30 msgr2=0x55639e06b3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:04:15.701 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:15.701+0000 7f36577fe640 1 --2- 172.21.15.29:0/1498526476 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55639e066a30 0x55639e06b3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:04:15.701 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:15.701+0000 7f36577fe640 1 -- 172.21.15.29:0/1498526476 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55639e06b9c0 con 0x55639e05d340 2024-03-16T12:04:15.701 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:15.701+0000 7f36577fe640 1 --2- 172.21.15.29:0/1498526476 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55639e05d340 0x55639e06eac0 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f3648011850 tx=0x7f3648009640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e0e45425d9369fb server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:04:15.701 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:15.701+0000 7f3636ffd640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-16T12:04:15.701 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:15.701+0000 7f3636ffd640 1 -- 172.21.15.29:0/1498526476 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3648039060 con 0x55639e05d340 2024-03-16T12:04:15.701 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:15.701+0000 7f3636ffd640 1 -- 172.21.15.29:0/1498526476 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(109..109 src has 1..109) ==== 8600+0+0 (secure 0 0 0) 0x7f36480188a0 con 0x55639e05d340 2024-03-16T12:04:15.701 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:15.701+0000 7f3660db1740 1 -- 172.21.15.29:0/1498526476 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3648005040 con 0x55639e05d340 2024-03-16T12:04:15.701 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:15.701+0000 7f3636ffd640 1 -- 172.21.15.29:0/1498526476 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f3648030070 con 0x55639e05d340 2024-03-16T12:04:15.702 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:15.701+0000 7f3660db1740 10 client.16521 fetch_fsmap learned FSMap version 30 2024-03-16T12:04:15.702 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:15.701+0000 7f3660db1740 1 -- 172.21.15.29:0/1498526476 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55639e0ddcb0 con 0x55639e05d340 2024-03-16T12:04:15.702 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:15.701+0000 7f3636ffd640 1 -- 172.21.15.29:0/1498526476 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3648005040 con 0x55639e05d340 2024-03-16T12:04:15.702 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:15.701+0000 7f3636ffd640 1 -- 172.21.15.29:0/1498526476 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 30) ==== 8338+0+0 (secure 0 0 0) 0x7f364802b040 con 0x55639e05d340 2024-03-16T12:04:15.702 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:15.701+0000 7f3660db1740 10 client.16521 fetch_fsmap finished waiting for FSMap version 30 2024-03-16T12:04:15.702 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:15.701+0000 7f3660db1740 10 client.16521 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T12:04:15.702 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:15.701+0000 7f3660db1740 20 client.16521 populate_metadata read hostname 'smithi029' 2024-03-16T12:04:15.702 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:15.701+0000 7f3660db1740 1 --2- 172.21.15.29:0/1498526476 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55639e0de180 0x55639e0e0570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:04:15.702 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:15.701+0000 7f3660db1740 4 client.16521 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-16T12:04:15.702 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:15.701+0000 7f3660db1740 1 -- 172.21.15.29:0/1498526476 --> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55639dfc72d0 con 0x55639e0de180 2024-03-16T12:04:15.702 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:15.702+0000 7f3656ffd640 1 --2- 172.21.15.29:0/1498526476 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55639e0de180 0x55639e0e0570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:04:15.702 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:15.702+0000 7f3636ffd640 10 client.16521 ms_handle_connect on v2:172.21.15.152:6838/1548731833 2024-03-16T12:04:15.702 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:15.702+0000 7f3656ffd640 1 --2- 172.21.15.29:0/1498526476 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55639e0de180 0x55639e0e0570 crc :-1 s=READY pgs=159 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f4debafd42159317 server_cookie=b5bb27fb4d9208cd in_seq=0 out_seq=0 2024-03-16T12:04:15.807 INFO:tasks.workunit.client.0.smithi029.stdout: 1 cleanup 600 sec 2024-03-16T12:04:15.903 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:15.908+0000 7f3636ffd640 1 -- 172.21.15.29:0/1498526476 <== mds.0 v2:172.21.15.152:6838/1548731833 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55639dfc72d0 con 0x55639e0de180 2024-03-16T12:04:15.903 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:15.908+0000 7f3636ffd640 10 client.16521 handle_command_reply: tid=0 2024-03-16T12:04:15.904 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:15.908+0000 7f3660db1740 1 client.16521 shutdown 2024-03-16T12:04:15.904 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:15.908+0000 7f3660db1740 1 -- 172.21.15.29:0/1498526476 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55639e05d340 msgr2=0x55639e06eac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:04:15.904 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:15.908+0000 7f3660db1740 1 --2- 172.21.15.29:0/1498526476 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55639e05d340 0x55639e06eac0 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f3648011850 tx=0x7f3648009640 comp rx=0 tx=0).stop 2024-03-16T12:04:15.904 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:15.908+0000 7f3660db1740 1 -- 172.21.15.29:0/1498526476 shutdown_connections 2024-03-16T12:04:15.904 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:15.908+0000 7f3660db1740 1 --2- 172.21.15.29:0/1498526476 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55639e0de180 0x55639e0e0570 crc :-1 s=READY pgs=159 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:04:15.904 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:15.908+0000 7f3660db1740 1 --2- 172.21.15.29:0/1498526476 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55639e066a30 0x55639e06b3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:04:15.904 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:15.908+0000 7f3660db1740 1 --2- 172.21.15.29:0/1498526476 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55639e05dc60 0x55639e06ae90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:04:15.904 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:15.908+0000 7f3660db1740 1 --2- 172.21.15.29:0/1498526476 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55639e05d340 0x55639e06eac0 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:04:15.904 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:15.908+0000 7f3636ffd640 0 client.16521 ms_handle_reset on v2:172.21.15.152:6838/1548731833 2024-03-16T12:04:15.904 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:15.908+0000 7f3660db1740 1 -- 172.21.15.29:0/1498526476 >> 172.21.15.29:0/1498526476 conn(0x55639e057c60 msgr2=0x55639e063980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:04:15.904 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:15.908+0000 7f3660db1740 1 -- 172.21.15.29:0/1498526476 shutdown_connections 2024-03-16T12:04:15.904 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:15.908+0000 7f3660db1740 1 -- 172.21.15.29:0/1498526476 wait complete. 2024-03-16T12:04:15.904 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:15.908+0000 7f3660db1740 20 client.16521 trim_cache size 0 max 16384 2024-03-16T12:04:15.905 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:{ 2024-03-16T12:04:15.905 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "message": "", 2024-03-16T12:04:15.905 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "return_code": 0 2024-03-16T12:04:15.905 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:} 2024-03-16T12:04:15.906 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:15.909+0000 7f365e1c6640 1 -- 172.21.15.29:0/2823994824 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f36400a2a10 msgr2=0x7f36400a4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:04:15.906 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:15.909+0000 7f365e1c6640 1 --2- 172.21.15.29:0/2823994824 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f36400a2a10 0x7f36400a4ed0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f3650004af0 tx=0x7f36500323c0 comp rx=0 tx=0).stop 2024-03-16T12:04:15.906 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:15.909+0000 7f365e1c6640 1 -- 172.21.15.29:0/2823994824 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f36581012f0 msgr2=0x7f3658197440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:04:15.906 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:15.909+0000 7f365e1c6640 1 --2- 172.21.15.29:0/2823994824 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f36581012f0 0x7f3658197440 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f364c00d950 tx=0x7f364c00de10 comp rx=0 tx=0).stop 2024-03-16T12:04:15.906 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:15.909+0000 7f365e1c6640 1 -- 172.21.15.29:0/2823994824 shutdown_connections 2024-03-16T12:04:15.906 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:15.909+0000 7f365e1c6640 1 --2- 172.21.15.29:0/2823994824 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f36400a2a10 0x7f36400a4ed0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:04:15.906 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:15.909+0000 7f365e1c6640 1 --2- 172.21.15.29:0/2823994824 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3658105680 0x7f365819b7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:04:15.906 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:15.909+0000 7f365e1c6640 1 --2- 172.21.15.29:0/2823994824 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f36581012f0 0x7f3658197440 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:04:15.906 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:15.909+0000 7f365e1c6640 1 --2- 172.21.15.29:0/2823994824 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3658100930 0x7f3658196f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:04:15.906 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:15.909+0000 7f365e1c6640 1 -- 172.21.15.29:0/2823994824 >> 172.21.15.29:0/2823994824 conn(0x7f3658077530 msgr2=0x7f36580fe8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:04:15.907 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:15.909+0000 7f365e1c6640 1 -- 172.21.15.29:0/2823994824 shutdown_connections 2024-03-16T12:04:15.907 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:15.909+0000 7f365e1c6640 1 -- 172.21.15.29:0/2823994824 wait complete. 2024-03-16T12:04:16.081 INFO:tasks.workunit.client.0.smithi029.stdout: 0 cleanup 600 sec 2024-03-16T12:04:16.081 INFO:tasks.workunit.client.0.smithi029.stdout: 2024-03-16T12:04:16.081 INFO:tasks.workunit.client.0.smithi029.stdout: Operation Count AvgLat MaxLat 2024-03-16T12:04:16.081 INFO:tasks.workunit.client.0.smithi029.stdout: ---------------------------------------- 2024-03-16T12:04:16.081 INFO:tasks.workunit.client.0.smithi029.stdout: NTCreateX 172357 1.321 164.648 2024-03-16T12:04:16.081 INFO:tasks.workunit.client.0.smithi029.stdout: Close 126576 0.003 0.122 2024-03-16T12:04:16.081 INFO:tasks.workunit.client.0.smithi029.stdout: Rename 7306 5.975 134.505 2024-03-16T12:04:16.081 INFO:tasks.workunit.client.0.smithi029.stdout: Unlink 34838 1.947 84.237 2024-03-16T12:04:16.081 INFO:tasks.workunit.client.0.smithi029.stdout: Deltree 4 118.387 149.588 2024-03-16T12:04:16.082 INFO:tasks.workunit.client.0.smithi029.stdout: Mkdir 2 0.013 0.015 2024-03-16T12:04:16.082 INFO:tasks.workunit.client.0.smithi029.stdout: Qpathinfo 156219 1.224 7755.860 2024-03-16T12:04:16.082 INFO:tasks.workunit.client.0.smithi029.stdout: Qfileinfo 27365 0.002 0.089 2024-03-16T12:04:16.082 INFO:tasks.workunit.client.0.smithi029.stdout: Qfsinfo 28677 0.130 7.836 2024-03-16T12:04:16.082 INFO:tasks.workunit.client.0.smithi029.stdout: Sfileinfo 14054 0.020 4.376 2024-03-16T12:04:16.082 INFO:tasks.workunit.client.0.smithi029.stdout: Find 60418 0.149 107.888 2024-03-16T12:04:16.082 INFO:tasks.workunit.client.0.smithi029.stdout: WriteX 85886 0.060 68.063 2024-03-16T12:04:16.082 INFO:tasks.workunit.client.0.smithi029.stdout: ReadX 270372 0.019 26.283 2024-03-16T12:04:16.082 INFO:tasks.workunit.client.0.smithi029.stdout: LockX 562 1.224 9.394 2024-03-16T12:04:16.082 INFO:tasks.workunit.client.0.smithi029.stdout: UnlockX 562 1.119 45.859 2024-03-16T12:04:16.082 INFO:tasks.workunit.client.0.smithi029.stdout: Flush 12088 3.465 205.785 2024-03-16T12:04:16.082 INFO:tasks.workunit.client.0.smithi029.stdout: 2024-03-16T12:04:16.082 INFO:tasks.workunit.client.0.smithi029.stdout:Throughput 9.02733 MB/sec 1 clients 1 procs max_latency=7755.869 ms 2024-03-16T12:04:16.083 INFO:tasks.workunit.client.0.smithi029.stderr:+ dbench 10 2024-03-16T12:04:16.083 INFO:tasks.workunit.client.0.smithi029.stdout:dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 2024-03-16T12:04:16.083 INFO:tasks.workunit.client.0.smithi029.stdout: 2024-03-16T12:04:16.083 INFO:tasks.workunit.client.0.smithi029.stdout:Running for 600 seconds with load '/usr/share/dbench/client.txt' and minimum warmup 120 secs 2024-03-16T12:04:16.083 INFO:tasks.workunit.client.0.smithi029.stdout:8 of 10 processes prepared for launch 0 sec 2024-03-16T12:04:16.182 INFO:tasks.workunit.client.0.smithi029.stdout:10 of 10 processes prepared for launch 0 sec 2024-03-16T12:04:16.183 INFO:tasks.workunit.client.0.smithi029.stdout:releasing clients 2024-03-16T12:04:16.541 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:04:16 smithi154 ceph-mon[36320]: pgmap v424: 97 pgs: 97 active+clean; 69 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.4 MiB/s wr, 222 op/s 2024-03-16T12:04:16.541 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:04:16 smithi154 ceph-mon[36320]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T12:04:16.583 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:04:16 smithi029 ceph-mon[31975]: pgmap v424: 97 pgs: 97 active+clean; 69 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.4 MiB/s wr, 222 op/s 2024-03-16T12:04:16.584 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:04:16 smithi029 ceph-mon[31975]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T12:04:16.623 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:04:16 smithi152 ceph-mon[36291]: pgmap v424: 97 pgs: 97 active+clean; 69 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.4 MiB/s wr, 222 op/s 2024-03-16T12:04:16.623 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:04:16 smithi152 ceph-mon[36291]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T12:04:17.183 INFO:tasks.workunit.client.0.smithi029.stdout: 10 273 102.36 MB/sec warmup 1 sec latency 115.382 ms 2024-03-16T12:04:17.541 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:04:17 smithi154 ceph-mon[36320]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:04:17.541 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:04:17 smithi154 ceph-mon[36320]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T12:04:17.541 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:04:17 smithi154 ceph-mon[36320]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' 2024-03-16T12:04:17.541 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:04:17 smithi154 ceph-mon[36320]: osdmap e110: 12 total, 12 up, 12 in 2024-03-16T12:04:17.583 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:04:17 smithi029 ceph-mon[31975]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:04:17.584 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:04:17 smithi029 ceph-mon[31975]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T12:04:17.584 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:04:17 smithi029 ceph-mon[31975]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' 2024-03-16T12:04:17.584 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:04:17 smithi029 ceph-mon[31975]: osdmap e110: 12 total, 12 up, 12 in 2024-03-16T12:04:17.623 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:04:17 smithi152 ceph-mon[36291]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:04:17.623 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:04:17 smithi152 ceph-mon[36291]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T12:04:17.623 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:04:17 smithi152 ceph-mon[36291]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' 2024-03-16T12:04:17.623 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:04:17 smithi152 ceph-mon[36291]: osdmap e110: 12 total, 12 up, 12 in 2024-03-16T12:04:18.182 INFO:tasks.workunit.client.0.smithi029.stdout: 10 463 81.22 MB/sec warmup 2 sec latency 265.882 ms 2024-03-16T12:04:18.541 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:04:18 smithi154 ceph-mon[36320]: pgmap v425: 97 pgs: 97 active+clean; 63 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 3.3 MiB/s wr, 332 op/s 2024-03-16T12:04:18.583 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:04:18 smithi029 ceph-mon[31975]: pgmap v425: 97 pgs: 97 active+clean; 63 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 3.3 MiB/s wr, 332 op/s 2024-03-16T12:04:18.623 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:04:18 smithi152 ceph-mon[36291]: pgmap v425: 97 pgs: 97 active+clean; 63 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 3.3 MiB/s wr, 332 op/s 2024-03-16T12:04:19.182 INFO:tasks.workunit.client.0.smithi029.stdout: 10 778 68.03 MB/sec warmup 3 sec latency 322.878 ms 2024-03-16T12:04:20.183 INFO:tasks.workunit.client.0.smithi029.stdout: 10 1099 55.07 MB/sec warmup 4 sec latency 303.326 ms 2024-03-16T12:04:20.540 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:04:20 smithi154 ceph-mon[36320]: pgmap v427: 97 pgs: 97 active+clean; 63 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.7 MiB/s wr, 236 op/s 2024-03-16T12:04:20.541 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:04:20 smithi154 ceph-mon[36320]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-03-16T12:04:20.583 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:04:20 smithi029 ceph-mon[31975]: pgmap v427: 97 pgs: 97 active+clean; 63 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.7 MiB/s wr, 236 op/s 2024-03-16T12:04:20.583 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:04:20 smithi029 ceph-mon[31975]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-03-16T12:04:20.623 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:04:20 smithi152 ceph-mon[36291]: pgmap v427: 97 pgs: 97 active+clean; 63 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.7 MiB/s wr, 236 op/s 2024-03-16T12:04:20.623 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:04:20 smithi152 ceph-mon[36291]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-03-16T12:04:21.183 INFO:tasks.workunit.client.0.smithi029.stdout: 10 1550 50.07 MB/sec warmup 5 sec latency 231.907 ms 2024-03-16T12:04:22.183 INFO:tasks.workunit.client.0.smithi029.stdout: 10 2210 44.59 MB/sec warmup 6 sec latency 248.132 ms 2024-03-16T12:04:22.540 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:04:22 smithi154 ceph-mon[36320]: pgmap v428: 97 pgs: 97 active+clean; 66 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.4 MiB/s wr, 221 op/s 2024-03-16T12:04:22.583 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:04:22 smithi029 ceph-mon[31975]: pgmap v428: 97 pgs: 97 active+clean; 66 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.4 MiB/s wr, 221 op/s 2024-03-16T12:04:22.622 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:04:22 smithi152 ceph-mon[36291]: pgmap v428: 97 pgs: 97 active+clean; 66 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.4 MiB/s wr, 221 op/s 2024-03-16T12:04:23.183 INFO:tasks.workunit.client.0.smithi029.stdout: 10 2561 43.63 MB/sec warmup 7 sec latency 295.964 ms 2024-03-16T12:04:24.183 INFO:tasks.workunit.client.0.smithi029.stdout: 10 2974 41.46 MB/sec warmup 8 sec latency 223.893 ms 2024-03-16T12:04:24.540 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:04:24 smithi154 ceph-mon[36320]: pgmap v429: 97 pgs: 97 active+clean; 70 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.9 MiB/s wr, 267 op/s 2024-03-16T12:04:24.583 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:04:24 smithi029 ceph-mon[31975]: pgmap v429: 97 pgs: 97 active+clean; 70 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.9 MiB/s wr, 267 op/s 2024-03-16T12:04:24.623 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:04:24 smithi152 ceph-mon[36291]: pgmap v429: 97 pgs: 97 active+clean; 70 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.9 MiB/s wr, 267 op/s 2024-03-16T12:04:25.182 INFO:tasks.workunit.client.0.smithi029.stdout: 10 3404 41.25 MB/sec warmup 9 sec latency 62.601 ms 2024-03-16T12:04:26.183 INFO:tasks.workunit.client.0.smithi029.stdout: 10 3599 38.39 MB/sec warmup 10 sec latency 164.220 ms 2024-03-16T12:04:26.540 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:04:26 smithi154 ceph-mon[36320]: pgmap v430: 97 pgs: 97 active+clean; 70 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.9 MiB/s wr, 267 op/s 2024-03-16T12:04:26.583 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:04:26 smithi029 ceph-mon[31975]: pgmap v430: 97 pgs: 97 active+clean; 70 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.9 MiB/s wr, 267 op/s 2024-03-16T12:04:26.623 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:04:26 smithi152 ceph-mon[36291]: pgmap v430: 97 pgs: 97 active+clean; 70 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.9 MiB/s wr, 267 op/s 2024-03-16T12:04:27.183 INFO:tasks.workunit.client.0.smithi029.stdout: 10 3734 35.15 MB/sec warmup 11 sec latency 210.289 ms 2024-03-16T12:04:28.183 INFO:tasks.workunit.client.0.smithi029.stdout: 10 3867 32.24 MB/sec warmup 12 sec latency 338.398 ms 2024-03-16T12:04:28.540 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:04:28 smithi154 ceph-mon[36320]: pgmap v431: 97 pgs: 97 active+clean; 79 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 3.4 MiB/s wr, 206 op/s 2024-03-16T12:04:28.583 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:04:28 smithi029 ceph-mon[31975]: pgmap v431: 97 pgs: 97 active+clean; 79 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 3.4 MiB/s wr, 206 op/s 2024-03-16T12:04:28.622 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:04:28 smithi152 ceph-mon[36291]: pgmap v431: 97 pgs: 97 active+clean; 79 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 3.4 MiB/s wr, 206 op/s 2024-03-16T12:04:29.019 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:29.022+0000 7f4a4099d640 1 -- 172.21.15.29:0/2283965036 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4a3c073b80 msgr2=0x7f4a3c073f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:04:29.020 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:29.022+0000 7f4a4099d640 1 --2- 172.21.15.29:0/2283965036 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4a3c073b80 0x7f4a3c073f60 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f4a30009900 tx=0x7f4a3002bcd0 comp rx=0 tx=0).stop 2024-03-16T12:04:29.020 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:29.023+0000 7f4a4099d640 1 -- 172.21.15.29:0/2283965036 shutdown_connections 2024-03-16T12:04:29.020 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:29.023+0000 7f4a4099d640 1 --2- 172.21.15.29:0/2283965036 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4a3c1125d0 0x7f4a3c1149c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:04:29.020 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:29.023+0000 7f4a4099d640 1 --2- 172.21.15.29:0/2283965036 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4a3c0744a0 0x7f4a3c112090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:04:29.020 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:29.023+0000 7f4a4099d640 1 --2- 172.21.15.29:0/2283965036 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4a3c073b80 0x7f4a3c073f60 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:04:29.020 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:29.023+0000 7f4a4099d640 1 -- 172.21.15.29:0/2283965036 >> 172.21.15.29:0/2283965036 conn(0x7f4a3c0fd110 msgr2=0x7f4a3c0ff530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:04:29.020 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:29.023+0000 7f4a4099d640 1 -- 172.21.15.29:0/2283965036 shutdown_connections 2024-03-16T12:04:29.020 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:29.023+0000 7f4a4099d640 1 -- 172.21.15.29:0/2283965036 wait complete. 2024-03-16T12:04:29.020 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:29.024+0000 7f4a4099d640 1 Processor -- start 2024-03-16T12:04:29.020 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:29.024+0000 7f4a4099d640 1 -- start start 2024-03-16T12:04:29.021 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:29.024+0000 7f4a4099d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4a3c073b80 0x7f4a3c19e580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:04:29.021 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:29.024+0000 7f4a4099d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4a3c0744a0 0x7f4a3c19eac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:04:29.021 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:29.024+0000 7f4a4099d640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4a3c1125d0 0x7f4a3c1a2df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:04:29.021 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:29.024+0000 7f4a4099d640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f4a3c118020 con 0x7f4a3c0744a0 2024-03-16T12:04:29.021 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:29.024+0000 7f4a4099d640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f4a3c117ea0 con 0x7f4a3c1125d0 2024-03-16T12:04:29.022 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:29.024+0000 7f4a4099d640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f4a3c1181a0 con 0x7f4a3c073b80 2024-03-16T12:04:29.022 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:29.025+0000 7f4a3affd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4a3c0744a0 0x7f4a3c19eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:04:29.022 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:29.025+0000 7f4a3bfff640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4a3c1125d0 0x7f4a3c1a2df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:04:29.022 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:29.025+0000 7f4a3affd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4a3c0744a0 0x7f4a3c19eac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39830/0 (socket says 172.21.15.29:39830) 2024-03-16T12:04:29.022 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:29.025+0000 7f4a3affd640 1 -- 172.21.15.29:0/3430201667 learned_addr learned my addr 172.21.15.29:0/3430201667 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T12:04:29.022 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:29.025+0000 7f4a3b7fe640 1 --2- 172.21.15.29:0/3430201667 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4a3c073b80 0x7f4a3c19e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:04:29.022 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:29.025+0000 7f4a3bfff640 1 -- 172.21.15.29:0/3430201667 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4a3c073b80 msgr2=0x7f4a3c19e580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:04:29.022 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:29.025+0000 7f4a3bfff640 1 --2- 172.21.15.29:0/3430201667 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4a3c073b80 0x7f4a3c19e580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:04:29.022 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:29.025+0000 7f4a3bfff640 1 -- 172.21.15.29:0/3430201667 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4a3c0744a0 msgr2=0x7f4a3c19eac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:04:29.022 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:29.025+0000 7f4a3bfff640 1 --2- 172.21.15.29:0/3430201667 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4a3c0744a0 0x7f4a3c19eac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:04:29.022 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:29.025+0000 7f4a3bfff640 1 -- 172.21.15.29:0/3430201667 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a3c1a3520 con 0x7f4a3c1125d0 2024-03-16T12:04:29.022 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:29.025+0000 7f4a3affd640 1 --2- 172.21.15.29:0/3430201667 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4a3c0744a0 0x7f4a3c19eac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:04:29.022 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:29.026+0000 7f4a3bfff640 1 --2- 172.21.15.29:0/3430201667 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4a3c1125d0 0x7f4a3c1a2df0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f4a34005fd0 tx=0x7f4a3400ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:04:29.023 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:29.027+0000 7f4a38ff9640 1 -- 172.21.15.29:0/3430201667 <== mon.2 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a3400cdf0 con 0x7f4a3c1125d0 2024-03-16T12:04:29.023 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:29.027+0000 7f4a4099d640 1 -- 172.21.15.29:0/3430201667 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4a3c1a3810 con 0x7f4a3c1125d0 2024-03-16T12:04:29.023 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:29.027+0000 7f4a38ff9640 1 -- 172.21.15.29:0/3430201667 <== mon.2 v2:172.21.15.152:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4a34005020 con 0x7f4a3c1125d0 2024-03-16T12:04:29.023 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:29.027+0000 7f4a4099d640 1 -- 172.21.15.29:0/3430201667 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4a3c1ab0a0 con 0x7f4a3c1125d0 2024-03-16T12:04:29.023 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:29.027+0000 7f4a38ff9640 1 -- 172.21.15.29:0/3430201667 <== mon.2 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a340106d0 con 0x7f4a3c1125d0 2024-03-16T12:04:29.027 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:29.031+0000 7f4a447ec740 1 Processor -- start 2024-03-16T12:04:29.027 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:29.031+0000 7f4a447ec740 1 -- start start 2024-03-16T12:04:29.027 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:29.031+0000 7f4a447ec740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562c481da2c0 0x562c481da6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:04:29.027 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:29.031+0000 7f4a447ec740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562c481dabe0 0x562c481e3470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:04:29.027 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:29.031+0000 7f4a447ec740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562c481e39b0 0x562c481e5da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:04:29.027 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:29.031+0000 7f4a447ec740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x562c4813ea40 con 0x562c481da2c0 2024-03-16T12:04:29.028 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:29.031+0000 7f4a447ec740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x562c480e7920 con 0x562c481dabe0 2024-03-16T12:04:29.028 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:29.031+0000 7f4a447ec740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x562c4813aae0 con 0x562c481e39b0 2024-03-16T12:04:29.028 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:29.032+0000 7f4a38ff9640 1 -- 172.21.15.29:0/3430201667 <== mon.2 v2:172.21.15.152:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7f4a34004a40 con 0x7f4a3c1125d0 2024-03-16T12:04:29.028 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:29.032+0000 7f4a3affd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562c481da2c0 0x562c481da6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:04:29.028 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:29.032+0000 7f4a3affd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562c481da2c0 0x562c481da6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39832/0 (socket says 172.21.15.29:39832) 2024-03-16T12:04:29.028 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:29.032+0000 7f4a3affd640 1 -- 172.21.15.29:0/3186797534 learned_addr learned my addr 172.21.15.29:0/3186797534 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T12:04:29.029 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:29.032+0000 7f4a38ff9640 1 --2- 172.21.15.29:0/3430201667 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f4a240a2a60 0x7f4a240a4f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:04:29.029 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:29.033+0000 7f4a3affd640 1 --2- 172.21.15.29:0/3186797534 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562c481e39b0 0x562c481e5da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:04:29.029 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:29.033+0000 7f4a3bfff640 1 --2- 172.21.15.29:0/3430201667 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f4a240a2a60 0x7f4a240a4f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:04:29.029 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:29.033+0000 7f4a38ff9640 1 -- 172.21.15.29:0/3430201667 <== mon.2 v2:172.21.15.152:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 8556+0+0 (secure 0 0 0) 0x7f4a340cf3b0 con 0x7f4a3c1125d0 2024-03-16T12:04:29.029 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:29.033+0000 7f4a3affd640 1 -- 172.21.15.29:0/3186797534 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562c481e39b0 msgr2=0x562c481e5da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:04:29.029 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:29.033+0000 7f4a3affd640 1 --2- 172.21.15.29:0/3186797534 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562c481e39b0 0x562c481e5da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:04:29.030 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:29.033+0000 7f4a3affd640 1 -- 172.21.15.29:0/3186797534 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562c481dabe0 msgr2=0x562c481e3470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:04:29.030 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:29.033+0000 7f4a3affd640 1 --2- 172.21.15.29:0/3186797534 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562c481dabe0 0x562c481e3470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:04:29.030 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:29.033+0000 7f4a3affd640 1 -- 172.21.15.29:0/3186797534 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562c480ff460 con 0x562c481da2c0 2024-03-16T12:04:29.030 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:29.033+0000 7f4a3affd640 1 --2- 172.21.15.29:0/3186797534 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562c481da2c0 0x562c481da6a0 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f4a2c009810 tx=0x7f4a2c009d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d8b9975b9e38722 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:04:29.031 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:29.034+0000 7f4a21ffb640 1 -- 172.21.15.29:0/3186797534 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a2c013120 con 0x562c481da2c0 2024-03-16T12:04:29.031 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:29.034+0000 7f4a21ffb640 1 -- 172.21.15.29:0/3186797534 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4a2c00fc50 con 0x562c481da2c0 2024-03-16T12:04:29.031 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:29.035+0000 7f4a447ec740 1 -- 172.21.15.29:0/3186797534 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562c481da2c0 msgr2=0x562c481da6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:04:29.031 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:29.035+0000 7f4a447ec740 1 --2- 172.21.15.29:0/3186797534 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562c481da2c0 0x562c481da6a0 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f4a2c009810 tx=0x7f4a2c009d00 comp rx=0 tx=0).stop 2024-03-16T12:04:29.031 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:29.035+0000 7f4a3bfff640 1 --2- 172.21.15.29:0/3430201667 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f4a240a2a60 0x7f4a240a4f20 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f4a34005c40 tx=0x7f4a3400cf90 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:04:29.031 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:29.035+0000 7f4a447ec740 1 -- 172.21.15.29:0/3186797534 shutdown_connections 2024-03-16T12:04:29.031 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:29.035+0000 7f4a447ec740 1 --2- 172.21.15.29:0/3186797534 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562c481e39b0 0x562c481e5da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:04:29.031 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:29.035+0000 7f4a447ec740 1 --2- 172.21.15.29:0/3186797534 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562c481dabe0 0x562c481e3470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:04:29.031 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:29.035+0000 7f4a447ec740 1 --2- 172.21.15.29:0/3186797534 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562c481da2c0 0x562c481da6a0 secure :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f4a2c009810 tx=0x7f4a2c009d00 comp rx=0 tx=0).stop 2024-03-16T12:04:29.031 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:29.035+0000 7f4a447ec740 1 -- 172.21.15.29:0/3186797534 >> 172.21.15.29:0/3186797534 conn(0x562c481d4be0 msgr2=0x562c481d4fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:04:29.032 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:29.035+0000 7f4a447ec740 1 -- 172.21.15.29:0/3186797534 shutdown_connections 2024-03-16T12:04:29.032 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:29.035+0000 7f4a447ec740 1 -- 172.21.15.29:0/3186797534 wait complete. 2024-03-16T12:04:29.032 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:29.036+0000 7f4a447ec740 1 -- start start 2024-03-16T12:04:29.033 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:29.036+0000 7f4a447ec740 1 Processor -- start 2024-03-16T12:04:29.033 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:29.036+0000 7f4a447ec740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562c481dabe0 0x562c481e0a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:04:29.033 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:29.036+0000 7f4a447ec740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562c481e39b0 0x562c481e0f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:04:29.033 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:29.037+0000 7f4a447ec740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562c481e14c0 0x562c481efaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:04:29.033 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:29.037+0000 7f4a447ec740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x562c481e6580 con 0x562c481e39b0 2024-03-16T12:04:29.033 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:29.037+0000 7f4a447ec740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x562c480fad70 con 0x562c481dabe0 2024-03-16T12:04:29.033 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:29.037+0000 7f4a447ec740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x562c481e6700 con 0x562c481e14c0 2024-03-16T12:04:29.033 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:29.037+0000 7f4a447ec740 10 client.0 _ll_register_callbacks cb 0x562c47e52a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T12:04:29.034 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:29.037+0000 7f4a3b7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562c481e39b0 0x562c481e0f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:04:29.034 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:29.037+0000 7f4a3b7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562c481e39b0 0x562c481e0f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39834/0 (socket says 172.21.15.29:39834) 2024-03-16T12:04:29.035 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:29.037+0000 7f4a3b7fe640 1 -- 172.21.15.29:0/245516498 learned_addr learned my addr 172.21.15.29:0/245516498 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T12:04:29.035 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:29.037+0000 7f4a3b7fe640 1 -- 172.21.15.29:0/245516498 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562c481e14c0 msgr2=0x562c481efaa0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-16T12:04:29.035 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:29.037+0000 7f4a3b7fe640 1 --2- 172.21.15.29:0/245516498 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562c481e14c0 0x562c481efaa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:04:29.035 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:29.038+0000 7f4a3b7fe640 1 -- 172.21.15.29:0/245516498 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562c481dabe0 msgr2=0x562c481e0a40 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-16T12:04:29.035 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:29.038+0000 7f4a3b7fe640 1 --2- 172.21.15.29:0/245516498 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562c481dabe0 0x562c481e0a40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:04:29.035 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:29.038+0000 7f4a3b7fe640 1 -- 172.21.15.29:0/245516498 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562c47e53210 con 0x562c481e39b0 2024-03-16T12:04:29.035 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:29.038+0000 7f4a3b7fe640 1 --2- 172.21.15.29:0/245516498 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562c481e39b0 0x562c481e0f80 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f4a30009900 tx=0x7f4a300045b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54c3c142ac7c4244 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:04:29.035 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:29.038+0000 7f4a1affd640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-16T12:04:29.035 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:29.038+0000 7f4a1affd640 1 -- 172.21.15.29:0/245516498 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a30043070 con 0x562c481e39b0 2024-03-16T12:04:29.035 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:29.038+0000 7f4a447ec740 1 -- 172.21.15.29:0/245516498 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x562c4814bf80 con 0x562c481e39b0 2024-03-16T12:04:29.035 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:29.038+0000 7f4a1affd640 1 -- 172.21.15.29:0/245516498 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(110..110 src has 1..110) ==== 8556+0+0 (secure 0 0 0) 0x7f4a3002cb30 con 0x562c481e39b0 2024-03-16T12:04:29.035 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:29.039+0000 7f4a1affd640 1 -- 172.21.15.29:0/245516498 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f4a3004f050 con 0x562c481e39b0 2024-03-16T12:04:29.035 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:29.039+0000 7f4a447ec740 10 client.16533 fetch_fsmap learned FSMap version 31 2024-03-16T12:04:29.036 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:29.039+0000 7f4a447ec740 1 -- 172.21.15.29:0/245516498 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x562c47f1f620 con 0x562c481e39b0 2024-03-16T12:04:29.036 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:29.039+0000 7f4a1affd640 1 -- 172.21.15.29:0/245516498 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x562c4814bf80 con 0x562c481e39b0 2024-03-16T12:04:29.036 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:29.039+0000 7f4a1affd640 1 -- 172.21.15.29:0/245516498 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 31) ==== 8346+0+0 (secure 0 0 0) 0x7f4a300052c0 con 0x562c481e39b0 2024-03-16T12:04:29.036 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:29.040+0000 7f4a447ec740 10 client.16533 fetch_fsmap finished waiting for FSMap version 31 2024-03-16T12:04:29.036 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:29.040+0000 7f4a447ec740 10 client.16533 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T12:04:29.036 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:29.040+0000 7f4a447ec740 20 client.16533 populate_metadata read hostname 'smithi029' 2024-03-16T12:04:29.037 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:29.040+0000 7f4a447ec740 1 --2- 172.21.15.29:0/245516498 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x562c481f4740 0x562c481f6ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:04:29.037 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:29.040+0000 7f4a447ec740 4 client.16533 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T12:04:29.037 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:29.040+0000 7f4a447ec740 1 -- 172.21.15.29:0/245516498 --> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562c481462d0 con 0x562c481f4740 2024-03-16T12:04:29.037 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:29.040+0000 7f4a3affd640 1 --2- 172.21.15.29:0/245516498 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x562c481f4740 0x562c481f6ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:04:29.037 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:29.042+0000 7f4a3affd640 1 --2- 172.21.15.29:0/245516498 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x562c481f4740 0x562c481f6ba0 crc :-1 s=READY pgs=160 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f08c7f9fedef8d21 server_cookie=795b77ab62af1ecd in_seq=0 out_seq=0 2024-03-16T12:04:29.037 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:29.042+0000 7f4a1affd640 10 client.16533 ms_handle_connect on v2:172.21.15.152:6838/1548731833 2024-03-16T12:04:29.041 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:29.046+0000 7f4a1affd640 1 -- 172.21.15.29:0/245516498 <== mds.0 v2:172.21.15.152:6838/1548731833 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x562c481462d0 con 0x562c481f4740 2024-03-16T12:04:29.041 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:29.046+0000 7f4a1affd640 10 client.16533 handle_command_reply: tid=0 2024-03-16T12:04:29.041 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:29.046+0000 7f4a447ec740 1 client.16533 shutdown 2024-03-16T12:04:29.042 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:29.046+0000 7f4a447ec740 1 -- 172.21.15.29:0/245516498 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562c481e39b0 msgr2=0x562c481e0f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:04:29.043 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:29.046+0000 7f4a447ec740 1 --2- 172.21.15.29:0/245516498 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562c481e39b0 0x562c481e0f80 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f4a30009900 tx=0x7f4a300045b0 comp rx=0 tx=0).stop 2024-03-16T12:04:29.043 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:29.046+0000 7f4a447ec740 1 -- 172.21.15.29:0/245516498 shutdown_connections 2024-03-16T12:04:29.043 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:29.046+0000 7f4a447ec740 1 --2- 172.21.15.29:0/245516498 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562c481e14c0 0x562c481efaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:04:29.043 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:29.046+0000 7f4a447ec740 1 --2- 172.21.15.29:0/245516498 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562c481e39b0 0x562c481e0f80 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:04:29.043 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:29.046+0000 7f4a447ec740 1 --2- 172.21.15.29:0/245516498 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x562c481f4740 0x562c481f6ba0 crc :-1 s=READY pgs=160 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:04:29.043 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:29.046+0000 7f4a447ec740 1 --2- 172.21.15.29:0/245516498 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562c481dabe0 0x562c481e0a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:04:29.043 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:29.046+0000 7f4a447ec740 1 -- 172.21.15.29:0/245516498 >> 172.21.15.29:0/245516498 conn(0x562c481d4be0 msgr2=0x562c481d6930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:04:29.043 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:29.046+0000 7f4a1affd640 0 client.16533 ms_handle_reset on v2:172.21.15.152:6838/1548731833 2024-03-16T12:04:29.043 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:29.047+0000 7f4a447ec740 1 -- 172.21.15.29:0/245516498 shutdown_connections 2024-03-16T12:04:29.043 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:29.047+0000 7f4a447ec740 1 -- 172.21.15.29:0/245516498 wait complete. 2024-03-16T12:04:29.043 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:29.047+0000 7f4a447ec740 20 client.16533 trim_cache size 0 max 16384 2024-03-16T12:04:29.072 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:29.075+0000 7f4a447ec740 1 Processor -- start 2024-03-16T12:04:29.073 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:29.075+0000 7f4a447ec740 1 -- start start 2024-03-16T12:04:29.073 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:29.076+0000 7f4a447ec740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562c481dabe0 0x562c481fc470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:04:29.073 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:29.076+0000 7f4a447ec740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562c481e14c0 0x562c481d65f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:04:29.073 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:29.076+0000 7f4a447ec740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562c481e39b0 0x562c481d6b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:04:29.073 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:29.076+0000 7f4a447ec740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f4a280111a0 con 0x562c481dabe0 2024-03-16T12:04:29.073 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:29.076+0000 7f4a447ec740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f4a2800ea10 con 0x562c481e14c0 2024-03-16T12:04:29.073 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:29.076+0000 7f4a447ec740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f4a2800c880 con 0x562c481e39b0 2024-03-16T12:04:29.073 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:29.076+0000 7f4a3affd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562c481dabe0 0x562c481fc470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:04:29.073 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:29.076+0000 7f4a3affd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562c481dabe0 0x562c481fc470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39846/0 (socket says 172.21.15.29:39846) 2024-03-16T12:04:29.073 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:29.076+0000 7f4a3affd640 1 -- 172.21.15.29:0/3763716449 learned_addr learned my addr 172.21.15.29:0/3763716449 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T12:04:29.073 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:29.076+0000 7f4a3b7fe640 1 --2- 172.21.15.29:0/3763716449 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562c481e14c0 0x562c481d65f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:04:29.073 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:29.076+0000 7f4a3affd640 1 --2- 172.21.15.29:0/3763716449 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562c481e39b0 0x562c481d6b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:04:29.073 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:29.076+0000 7f4a3affd640 1 -- 172.21.15.29:0/3763716449 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562c481e39b0 msgr2=0x562c481d6b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:04:29.074 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:29.076+0000 7f4a3affd640 1 --2- 172.21.15.29:0/3763716449 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562c481e39b0 0x562c481d6b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:04:29.074 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:29.076+0000 7f4a3affd640 1 -- 172.21.15.29:0/3763716449 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562c481e14c0 msgr2=0x562c481d65f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:04:29.074 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:29.076+0000 7f4a3affd640 1 --2- 172.21.15.29:0/3763716449 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562c481e14c0 0x562c481d65f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:04:29.074 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:29.076+0000 7f4a3affd640 1 -- 172.21.15.29:0/3763716449 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562c481f2d10 con 0x562c481dabe0 2024-03-16T12:04:29.074 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:29.076+0000 7f4a3affd640 1 --2- 172.21.15.29:0/3763716449 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562c481dabe0 0x562c481fc470 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f4a2c01d040 tx=0x7f4a2c009530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e19dd5867be745f1 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:04:29.074 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:29.076+0000 7f4a217fa640 1 -- 172.21.15.29:0/3763716449 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a2c013120 con 0x562c481dabe0 2024-03-16T12:04:29.074 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:29.077+0000 7f4a217fa640 1 -- 172.21.15.29:0/3763716449 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4a2c005520 con 0x562c481dabe0 2024-03-16T12:04:29.074 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:29.077+0000 7f4a217fa640 1 -- 172.21.15.29:0/3763716449 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a2c010850 con 0x562c481dabe0 2024-03-16T12:04:29.078 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:29.078+0000 7f4a447ec740 1 -- 172.21.15.29:0/3763716449 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562c481dabe0 msgr2=0x562c481fc470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:04:29.078 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:29.078+0000 7f4a447ec740 1 --2- 172.21.15.29:0/3763716449 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562c481dabe0 0x562c481fc470 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f4a2c01d040 tx=0x7f4a2c009530 comp rx=0 tx=0).stop 2024-03-16T12:04:29.078 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:29.078+0000 7f4a447ec740 1 -- 172.21.15.29:0/3763716449 shutdown_connections 2024-03-16T12:04:29.078 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:29.078+0000 7f4a447ec740 1 --2- 172.21.15.29:0/3763716449 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562c481e39b0 0x562c481d6b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:04:29.078 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:29.078+0000 7f4a447ec740 1 --2- 172.21.15.29:0/3763716449 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562c481e14c0 0x562c481d65f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:04:29.078 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:29.078+0000 7f4a447ec740 1 --2- 172.21.15.29:0/3763716449 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562c481dabe0 0x562c481fc470 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:04:29.079 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:29.078+0000 7f4a447ec740 1 -- 172.21.15.29:0/3763716449 >> 172.21.15.29:0/3763716449 conn(0x562c481d4be0 msgr2=0x562c481d5ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:04:29.079 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:29.078+0000 7f4a447ec740 1 -- 172.21.15.29:0/3763716449 shutdown_connections 2024-03-16T12:04:29.079 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:29.078+0000 7f4a447ec740 1 -- 172.21.15.29:0/3763716449 wait complete. 2024-03-16T12:04:29.079 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:29.078+0000 7f4a447ec740 1 -- start start 2024-03-16T12:04:29.079 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:29.078+0000 7f4a447ec740 1 Processor -- start 2024-03-16T12:04:29.079 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:29.079+0000 7f4a447ec740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562c481dabe0 0x562c481eba00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:04:29.079 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:29.079+0000 7f4a447ec740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562c481e14c0 0x562c481dd080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:04:29.079 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:29.079+0000 7f4a447ec740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562c481e39b0 0x562c481dd5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:04:29.079 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:29.079+0000 7f4a447ec740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f4a2800ced0 con 0x562c481e39b0 2024-03-16T12:04:29.079 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:29.079+0000 7f4a447ec740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f4a2800e340 con 0x562c481dabe0 2024-03-16T12:04:29.079 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:29.079+0000 7f4a447ec740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f4a2800d5a0 con 0x562c481e14c0 2024-03-16T12:04:29.079 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:29.079+0000 7f4a447ec740 10 client.0 _ll_register_callbacks cb 0x7f4a2c005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T12:04:29.079 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:29.079+0000 7f4a3affd640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562c481dabe0 0x562c481eba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:04:29.080 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:29.079+0000 7f4a3b7fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562c481e14c0 0x562c481dd080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:04:29.080 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:29.079+0000 7f4a3b7fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562c481e14c0 0x562c481dd080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.29:44712/0 (socket says 172.21.15.29:44712) 2024-03-16T12:04:29.080 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:29.079+0000 7f4a3b7fe640 1 -- 172.21.15.29:0/1687348347 learned_addr learned my addr 172.21.15.29:0/1687348347 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T12:04:29.080 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:29.079+0000 7f4a3affd640 1 --2- 172.21.15.29:0/1687348347 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562c481e39b0 0x562c481dd5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:04:29.080 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:29.079+0000 7f4a3b7fe640 1 -- 172.21.15.29:0/1687348347 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562c481dabe0 msgr2=0x562c481eba00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:04:29.080 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:29.079+0000 7f4a3b7fe640 1 --2- 172.21.15.29:0/1687348347 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562c481dabe0 0x562c481eba00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:04:29.080 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:29.079+0000 7f4a3b7fe640 1 -- 172.21.15.29:0/1687348347 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562c481e39b0 msgr2=0x562c481dd5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:04:29.080 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:29.079+0000 7f4a3b7fe640 1 --2- 172.21.15.29:0/1687348347 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562c481e39b0 0x562c481dd5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:04:29.080 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:29.079+0000 7f4a3b7fe640 1 -- 172.21.15.29:0/1687348347 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562c481ddb00 con 0x562c481e14c0 2024-03-16T12:04:29.080 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:29.079+0000 7f4a3b7fe640 1 --2- 172.21.15.29:0/1687348347 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562c481e14c0 0x562c481dd080 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x562c481e2010 tx=0x7f4a3002cd40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9051393047c0b919 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:04:29.080 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:29.080+0000 7f4a1affd640 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-16T12:04:29.080 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:29.080+0000 7f4a1affd640 1 -- 172.21.15.29:0/1687348347 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a30058060 con 0x562c481e14c0 2024-03-16T12:04:29.080 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:29.080+0000 7f4a1affd640 1 -- 172.21.15.29:0/1687348347 <== mon.1 v2:172.21.15.154:3300/0 2 ==== osd_map(110..110 src has 1..110) ==== 8556+0+0 (secure 0 0 0) 0x7f4a30003e40 con 0x562c481e14c0 2024-03-16T12:04:29.080 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:29.080+0000 7f4a447ec740 1 -- 172.21.15.29:0/1687348347 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4a2c005520 con 0x562c481e14c0 2024-03-16T12:04:29.080 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:29.080+0000 7f4a1affd640 1 -- 172.21.15.29:0/1687348347 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f4a30043030 con 0x562c481e14c0 2024-03-16T12:04:29.081 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:29.080+0000 7f4a447ec740 10 client.26377 fetch_fsmap learned FSMap version 31 2024-03-16T12:04:29.081 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:29.080+0000 7f4a447ec740 1 -- 172.21.15.29:0/1687348347 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x562c4825a740 con 0x562c481e14c0 2024-03-16T12:04:29.081 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:29.080+0000 7f4a1affd640 1 -- 172.21.15.29:0/1687348347 <== mon.1 v2:172.21.15.154:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4a2c005520 con 0x562c481e14c0 2024-03-16T12:04:29.081 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:29.080+0000 7f4a1affd640 1 -- 172.21.15.29:0/1687348347 <== mon.1 v2:172.21.15.154:3300/0 5 ==== fsmap(e 31) ==== 8346+0+0 (secure 0 0 0) 0x7f4a30004880 con 0x562c481e14c0 2024-03-16T12:04:29.081 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:29.080+0000 7f4a447ec740 10 client.26377 fetch_fsmap finished waiting for FSMap version 31 2024-03-16T12:04:29.081 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:29.080+0000 7f4a447ec740 10 client.26377 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T12:04:29.081 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:29.080+0000 7f4a447ec740 20 client.26377 populate_metadata read hostname 'smithi029' 2024-03-16T12:04:29.081 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:29.080+0000 7f4a447ec740 1 --2- 172.21.15.29:0/1687348347 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x562c4825af50 0x562c4825d370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:04:29.081 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:29.080+0000 7f4a447ec740 4 client.26377 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-16T12:04:29.081 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:29.080+0000 7f4a447ec740 1 -- 172.21.15.29:0/1687348347 --> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] -- command(tid 0: {"prefix": "flush journal"}) -- 0x562c480c4770 con 0x562c4825af50 2024-03-16T12:04:29.081 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:29.081+0000 7f4a3affd640 1 --2- 172.21.15.29:0/1687348347 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x562c4825af50 0x562c4825d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:04:29.081 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:29.081+0000 7f4a1affd640 10 client.26377 ms_handle_connect on v2:172.21.15.152:6838/1548731833 2024-03-16T12:04:29.081 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:29.081+0000 7f4a3affd640 1 --2- 172.21.15.29:0/1687348347 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x562c4825af50 0x562c4825d370 crc :-1 s=READY pgs=161 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5baa480da21962ac server_cookie=91ab0d68a350baab in_seq=0 out_seq=0 2024-03-16T12:04:29.183 INFO:tasks.workunit.client.0.smithi029.stdout: 10 4008 29.92 MB/sec warmup 13 sec latency 476.465 ms 2024-03-16T12:04:29.756 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:29.759+0000 7f4a1affd640 1 -- 172.21.15.29:0/1687348347 <== mds.0 v2:172.21.15.152:6838/1548731833 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x562c480c4770 con 0x562c4825af50 2024-03-16T12:04:29.756 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:29.759+0000 7f4a1affd640 10 client.26377 handle_command_reply: tid=0 2024-03-16T12:04:29.756 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:29.759+0000 7f4a447ec740 1 client.26377 shutdown 2024-03-16T12:04:29.756 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:29.760+0000 7f4a447ec740 1 -- 172.21.15.29:0/1687348347 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562c481e14c0 msgr2=0x562c481dd080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:04:29.756 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:29.760+0000 7f4a447ec740 1 --2- 172.21.15.29:0/1687348347 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562c481e14c0 0x562c481dd080 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x562c481e2010 tx=0x7f4a3002cd40 comp rx=0 tx=0).stop 2024-03-16T12:04:29.756 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:29.760+0000 7f4a447ec740 1 -- 172.21.15.29:0/1687348347 shutdown_connections 2024-03-16T12:04:29.756 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:29.760+0000 7f4a447ec740 1 --2- 172.21.15.29:0/1687348347 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x562c481e39b0 0x562c481dd5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:04:29.757 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:29.760+0000 7f4a447ec740 1 --2- 172.21.15.29:0/1687348347 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562c481e14c0 0x562c481dd080 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:04:29.757 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:29.760+0000 7f4a447ec740 1 --2- 172.21.15.29:0/1687348347 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x562c4825af50 0x562c4825d370 crc :-1 s=READY pgs=161 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:04:29.757 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:29.760+0000 7f4a1affd640 0 client.26377 ms_handle_reset on v2:172.21.15.152:6838/1548731833 2024-03-16T12:04:29.757 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:29.760+0000 7f4a447ec740 1 --2- 172.21.15.29:0/1687348347 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562c481dabe0 0x562c481eba00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:04:29.757 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:29.760+0000 7f4a447ec740 1 -- 172.21.15.29:0/1687348347 >> 172.21.15.29:0/1687348347 conn(0x562c481d4be0 msgr2=0x562c481d6770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:04:29.757 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:29.760+0000 7f4a447ec740 1 -- 172.21.15.29:0/1687348347 shutdown_connections 2024-03-16T12:04:29.757 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:29.760+0000 7f4a447ec740 1 -- 172.21.15.29:0/1687348347 wait complete. 2024-03-16T12:04:29.757 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:29.760+0000 7f4a447ec740 20 client.26377 trim_cache size 0 max 16384 2024-03-16T12:04:29.757 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:{ 2024-03-16T12:04:29.757 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "message": "", 2024-03-16T12:04:29.757 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "return_code": 0 2024-03-16T12:04:29.758 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:} 2024-03-16T12:04:29.759 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:29.761+0000 7f4a4099d640 1 -- 172.21.15.29:0/3430201667 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f4a240a2a60 msgr2=0x7f4a240a4f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:04:29.759 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:29.761+0000 7f4a4099d640 1 --2- 172.21.15.29:0/3430201667 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f4a240a2a60 0x7f4a240a4f20 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f4a34005c40 tx=0x7f4a3400cf90 comp rx=0 tx=0).stop 2024-03-16T12:04:29.759 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:29.761+0000 7f4a4099d640 1 -- 172.21.15.29:0/3430201667 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4a3c1125d0 msgr2=0x7f4a3c1a2df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:04:29.759 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:29.761+0000 7f4a4099d640 1 --2- 172.21.15.29:0/3430201667 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4a3c1125d0 0x7f4a3c1a2df0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f4a34005fd0 tx=0x7f4a3400ef10 comp rx=0 tx=0).stop 2024-03-16T12:04:29.759 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:29.761+0000 7f4a4099d640 1 -- 172.21.15.29:0/3430201667 shutdown_connections 2024-03-16T12:04:29.759 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:29.761+0000 7f4a4099d640 1 --2- 172.21.15.29:0/3430201667 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f4a240a2a60 0x7f4a240a4f20 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:04:29.759 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:29.761+0000 7f4a4099d640 1 --2- 172.21.15.29:0/3430201667 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4a3c1125d0 0x7f4a3c1a2df0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:04:29.759 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:29.761+0000 7f4a4099d640 1 --2- 172.21.15.29:0/3430201667 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4a3c0744a0 0x7f4a3c19eac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:04:29.759 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:29.761+0000 7f4a4099d640 1 --2- 172.21.15.29:0/3430201667 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4a3c073b80 0x7f4a3c19e580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:04:29.759 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:29.761+0000 7f4a4099d640 1 -- 172.21.15.29:0/3430201667 >> 172.21.15.29:0/3430201667 conn(0x7f4a3c0fd110 msgr2=0x7f4a3c0fde20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:04:29.759 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:29.761+0000 7f4a4099d640 1 -- 172.21.15.29:0/3430201667 shutdown_connections 2024-03-16T12:04:29.760 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:29.761+0000 7f4a4099d640 1 -- 172.21.15.29:0/3430201667 wait complete. 2024-03-16T12:04:30.183 INFO:tasks.workunit.client.0.smithi029.stdout: 10 4170 28.08 MB/sec warmup 14 sec latency 487.664 ms 2024-03-16T12:04:30.540 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:04:30 smithi154 ceph-mon[36320]: pgmap v432: 97 pgs: 97 active+clean; 79 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 920 KiB/s rd, 2.8 MiB/s wr, 173 op/s 2024-03-16T12:04:30.583 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:04:30 smithi029 ceph-mon[31975]: pgmap v432: 97 pgs: 97 active+clean; 79 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 920 KiB/s rd, 2.8 MiB/s wr, 173 op/s 2024-03-16T12:04:30.622 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:04:30 smithi152 ceph-mon[36291]: pgmap v432: 97 pgs: 97 active+clean; 79 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 920 KiB/s rd, 2.8 MiB/s wr, 173 op/s 2024-03-16T12:04:31.183 INFO:tasks.workunit.client.0.smithi029.stdout: 10 4488 27.06 MB/sec warmup 15 sec latency 573.371 ms 2024-03-16T12:04:32.183 INFO:tasks.workunit.client.0.smithi029.stdout: 10 4992 27.20 MB/sec warmup 16 sec latency 226.552 ms 2024-03-16T12:04:32.540 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:04:32 smithi154 ceph-mon[36320]: pgmap v433: 97 pgs: 97 active+clean; 73 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 4.1 MiB/s wr, 437 op/s 2024-03-16T12:04:32.583 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:04:32 smithi029 ceph-mon[31975]: pgmap v433: 97 pgs: 97 active+clean; 73 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 4.1 MiB/s wr, 437 op/s 2024-03-16T12:04:32.622 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:04:32 smithi152 ceph-mon[36291]: pgmap v433: 97 pgs: 97 active+clean; 73 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 4.1 MiB/s wr, 437 op/s 2024-03-16T12:04:33.183 INFO:tasks.workunit.client.0.smithi029.stdout: 10 5616 26.56 MB/sec warmup 17 sec latency 216.523 ms 2024-03-16T12:04:34.183 INFO:tasks.workunit.client.0.smithi029.stdout: 10 5995 27.12 MB/sec warmup 18 sec latency 91.304 ms 2024-03-16T12:04:34.540 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:04:34 smithi154 ceph-mon[36320]: pgmap v434: 97 pgs: 97 active+clean; 69 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.4 MiB/s wr, 419 op/s 2024-03-16T12:04:34.583 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:04:34 smithi029 ceph-mon[31975]: pgmap v434: 97 pgs: 97 active+clean; 69 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.4 MiB/s wr, 419 op/s 2024-03-16T12:04:34.622 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:04:34 smithi152 ceph-mon[36291]: pgmap v434: 97 pgs: 97 active+clean; 69 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.4 MiB/s wr, 419 op/s 2024-03-16T12:04:35.183 INFO:tasks.workunit.client.0.smithi029.stdout: 10 6465 27.49 MB/sec warmup 19 sec latency 69.341 ms 2024-03-16T12:04:36.183 INFO:tasks.workunit.client.0.smithi029.stdout: 10 6845 27.90 MB/sec warmup 20 sec latency 168.981 ms 2024-03-16T12:04:36.540 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:04:36 smithi154 ceph-mon[36320]: pgmap v435: 97 pgs: 97 active+clean; 69 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 920 KiB/s rd, 2.9 MiB/s wr, 377 op/s 2024-03-16T12:04:36.583 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:04:36 smithi029 ceph-mon[31975]: pgmap v435: 97 pgs: 97 active+clean; 69 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 920 KiB/s rd, 2.9 MiB/s wr, 377 op/s 2024-03-16T12:04:36.622 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:04:36 smithi152 ceph-mon[36291]: pgmap v435: 97 pgs: 97 active+clean; 69 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 920 KiB/s rd, 2.9 MiB/s wr, 377 op/s 2024-03-16T12:04:37.183 INFO:tasks.workunit.client.0.smithi029.stdout: 10 6998 27.02 MB/sec warmup 21 sec latency 206.449 ms 2024-03-16T12:04:38.183 INFO:tasks.workunit.client.0.smithi029.stdout: 10 7142 25.96 MB/sec warmup 22 sec latency 216.640 ms 2024-03-16T12:04:38.540 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:04:38 smithi154 ceph-mon[36320]: pgmap v436: 97 pgs: 97 active+clean; 77 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 4.1 MiB/s wr, 457 op/s 2024-03-16T12:04:38.583 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:04:38 smithi029 ceph-mon[31975]: pgmap v436: 97 pgs: 97 active+clean; 77 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 4.1 MiB/s wr, 457 op/s 2024-03-16T12:04:38.622 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:04:38 smithi152 ceph-mon[36291]: pgmap v436: 97 pgs: 97 active+clean; 77 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 4.1 MiB/s wr, 457 op/s 2024-03-16T12:04:39.183 INFO:tasks.workunit.client.0.smithi029.stdout: 10 7267 24.86 MB/sec warmup 23 sec latency 325.929 ms 2024-03-16T12:04:40.183 INFO:tasks.workunit.client.0.smithi029.stdout: 10 7447 24.00 MB/sec warmup 24 sec latency 524.717 ms 2024-03-16T12:04:40.540 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:04:40 smithi154 ceph-mon[36320]: pgmap v437: 97 pgs: 97 active+clean; 77 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 817 KiB/s rd, 2.6 MiB/s wr, 377 op/s 2024-03-16T12:04:40.583 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:04:40 smithi029 ceph-mon[31975]: pgmap v437: 97 pgs: 97 active+clean; 77 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 817 KiB/s rd, 2.6 MiB/s wr, 377 op/s 2024-03-16T12:04:40.622 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:04:40 smithi152 ceph-mon[36291]: pgmap v437: 97 pgs: 97 active+clean; 77 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 817 KiB/s rd, 2.6 MiB/s wr, 377 op/s 2024-03-16T12:04:41.183 INFO:tasks.workunit.client.0.smithi029.stdout: 10 7750 23.55 MB/sec warmup 25 sec latency 66.522 ms 2024-03-16T12:04:42.183 INFO:tasks.workunit.client.0.smithi029.stdout: 10 8159 23.70 MB/sec warmup 26 sec latency 81.981 ms 2024-03-16T12:04:42.540 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:04:42 smithi154 ceph-mon[36320]: pgmap v438: 97 pgs: 97 active+clean; 79 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 3.3 MiB/s wr, 425 op/s 2024-03-16T12:04:42.583 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:04:42 smithi029 ceph-mon[31975]: pgmap v438: 97 pgs: 97 active+clean; 79 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 3.3 MiB/s wr, 425 op/s 2024-03-16T12:04:42.622 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:04:42 smithi152 ceph-mon[36291]: pgmap v438: 97 pgs: 97 active+clean; 79 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 3.3 MiB/s wr, 425 op/s 2024-03-16T12:04:42.851 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:42.856+0000 7f890bec0640 1 -- 172.21.15.29:0/3881199877 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8904069800 msgr2=0x7f890410e670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:04:42.851 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:42.856+0000 7f890bec0640 1 --2- 172.21.15.29:0/3881199877 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8904069800 0x7f890410e670 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f88fc009c10 tx=0x7f88fc02bef0 comp rx=0 tx=0).stop 2024-03-16T12:04:42.851 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:42.856+0000 7f890bec0640 1 -- 172.21.15.29:0/3881199877 shutdown_connections 2024-03-16T12:04:42.851 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:42.856+0000 7f890bec0640 1 --2- 172.21.15.29:0/3881199877 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8904069800 0x7f890410e670 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:04:42.851 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:42.856+0000 7f890bec0640 1 --2- 172.21.15.29:0/3881199877 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8904068e60 0x7f89040692c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:04:42.851 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:42.856+0000 7f890bec0640 1 --2- 172.21.15.29:0/3881199877 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f89041078b0 0x7f8904107c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:04:42.852 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:42.856+0000 7f890bec0640 1 -- 172.21.15.29:0/3881199877 >> 172.21.15.29:0/3881199877 conn(0x7f8904077530 msgr2=0x7f8904077940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:04:42.852 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:42.856+0000 7f890bec0640 1 -- 172.21.15.29:0/3881199877 shutdown_connections 2024-03-16T12:04:42.852 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:42.856+0000 7f890bec0640 1 -- 172.21.15.29:0/3881199877 wait complete. 2024-03-16T12:04:42.852 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:42.857+0000 7f890bec0640 1 Processor -- start 2024-03-16T12:04:42.854 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:42.857+0000 7f890bec0640 1 -- start start 2024-03-16T12:04:42.854 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:42.857+0000 7f890bec0640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8904068e60 0x7f8904100a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:04:42.854 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:42.857+0000 7f890bec0640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8904069800 0x7f8904100f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:04:42.854 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:42.857+0000 7f890bec0640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f89041078b0 0x7f8904101480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:04:42.854 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:42.857+0000 7f890bec0640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f8904111ba0 con 0x7f8904069800 2024-03-16T12:04:42.854 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:42.857+0000 7f890bec0640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f8904111a20 con 0x7f89041078b0 2024-03-16T12:04:42.854 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:42.857+0000 7f890bec0640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f8904111d20 con 0x7f8904068e60 2024-03-16T12:04:42.854 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:42.858+0000 7f890945b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8904069800 0x7f8904100f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:04:42.854 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:42.858+0000 7f8909c5c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8904068e60 0x7f8904100a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:04:42.854 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:42.858+0000 7f890945b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8904069800 0x7f8904100f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59948/0 (socket says 172.21.15.29:59948) 2024-03-16T12:04:42.854 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:42.858+0000 7f890945b640 1 -- 172.21.15.29:0/113167430 learned_addr learned my addr 172.21.15.29:0/113167430 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T12:04:42.855 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:42.858+0000 7f890a45d640 1 --2- 172.21.15.29:0/113167430 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f89041078b0 0x7f8904101480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:04:42.855 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:42.858+0000 7f890945b640 1 -- 172.21.15.29:0/113167430 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8904068e60 msgr2=0x7f8904100a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:04:42.855 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:42.858+0000 7f890945b640 1 --2- 172.21.15.29:0/113167430 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8904068e60 0x7f8904100a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:04:42.855 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:42.858+0000 7f890945b640 1 -- 172.21.15.29:0/113167430 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f89041078b0 msgr2=0x7f8904101480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:04:42.855 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:42.858+0000 7f890945b640 1 --2- 172.21.15.29:0/113167430 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f89041078b0 0x7f8904101480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:04:42.855 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:42.858+0000 7f890945b640 1 -- 172.21.15.29:0/113167430 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89041a64a0 con 0x7f8904069800 2024-03-16T12:04:42.855 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:42.858+0000 7f890945b640 1 --2- 172.21.15.29:0/113167430 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8904069800 0x7f8904100f40 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f88f400e9f0 tx=0x7f88f400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:04:42.855 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:42.858+0000 7f88faffd640 1 -- 172.21.15.29:0/113167430 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88f400ccc0 con 0x7f8904069800 2024-03-16T12:04:42.855 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:42.858+0000 7f890bec0640 1 -- 172.21.15.29:0/113167430 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f89041a6790 con 0x7f8904069800 2024-03-16T12:04:42.855 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:42.858+0000 7f88faffd640 1 -- 172.21.15.29:0/113167430 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f88f4010910 con 0x7f8904069800 2024-03-16T12:04:42.855 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:42.858+0000 7f88faffd640 1 -- 172.21.15.29:0/113167430 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88f40188a0 con 0x7f8904069800 2024-03-16T12:04:42.855 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:42.858+0000 7f890bec0640 1 -- 172.21.15.29:0/113167430 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f89041a6ce0 con 0x7f8904069800 2024-03-16T12:04:42.861 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:42.861+0000 7f88faffd640 1 -- 172.21.15.29:0/113167430 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7f88f4010ac0 con 0x7f8904069800 2024-03-16T12:04:42.861 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:42.861+0000 7f890eb85740 1 Processor -- start 2024-03-16T12:04:42.861 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:42.861+0000 7f890eb85740 1 -- start start 2024-03-16T12:04:42.861 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:42.861+0000 7f890eb85740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5603ed354130 0x5603ed354510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:04:42.861 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:42.861+0000 7f890eb85740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5603ed354a50 0x5603ed35d2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:04:42.861 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:42.862+0000 7f890eb85740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5603ed35d820 0x5603ed35fc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:04:42.861 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:42.862+0000 7f890eb85740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x5603ed2b8a40 con 0x5603ed35d820 2024-03-16T12:04:42.861 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:42.862+0000 7f890eb85740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x5603ed261920 con 0x5603ed354130 2024-03-16T12:04:42.861 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:42.862+0000 7f890eb85740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5603ed2b4ae0 con 0x5603ed354a50 2024-03-16T12:04:42.861 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:42.862+0000 7f88faffd640 1 --2- 172.21.15.29:0/113167430 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f88f00a2a40 0x7f88f00a4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:04:42.861 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:42.862+0000 7f890a45d640 1 --2- 172.21.15.29:0/113167430 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f88f00a2a40 0x7f88f00a4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:04:42.862 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:42.862+0000 7f890945b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5603ed35d820 0x5603ed35fc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:04:42.862 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:42.862+0000 7f8909c5c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5603ed354a50 0x5603ed35d2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:04:42.862 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:42.862+0000 7f8909c5c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5603ed354a50 0x5603ed35d2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.29:37462/0 (socket says 172.21.15.29:37462) 2024-03-16T12:04:42.862 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:42.862+0000 7f890945b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5603ed35d820 0x5603ed35fc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59958/0 (socket says 172.21.15.29:59958) 2024-03-16T12:04:42.862 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:42.862+0000 7f890a45d640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5603ed354130 0x5603ed354510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:04:42.862 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:42.862+0000 7f8909c5c640 1 -- 172.21.15.29:0/3873249760 learned_addr learned my addr 172.21.15.29:0/3873249760 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T12:04:42.862 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:42.862+0000 7f88faffd640 1 -- 172.21.15.29:0/113167430 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 8556+0+0 (secure 0 0 0) 0x7f88f40cf310 con 0x7f8904069800 2024-03-16T12:04:42.862 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:42.862+0000 7f8909c5c640 1 -- 172.21.15.29:0/3873249760 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5603ed354130 msgr2=0x5603ed354510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:04:42.862 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:42.862+0000 7f8909c5c640 1 --2- 172.21.15.29:0/3873249760 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5603ed354130 0x5603ed354510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:04:42.862 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:42.862+0000 7f8909c5c640 1 -- 172.21.15.29:0/3873249760 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5603ed35d820 msgr2=0x5603ed35fc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:04:42.862 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:42.862+0000 7f8909c5c640 1 --2- 172.21.15.29:0/3873249760 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5603ed35d820 0x5603ed35fc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:04:42.862 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:42.862+0000 7f8909c5c640 1 -- 172.21.15.29:0/3873249760 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5603ed279460 con 0x5603ed354a50 2024-03-16T12:04:42.862 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:42.862+0000 7f890945b640 1 --2- 172.21.15.29:0/3873249760 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5603ed35d820 0x5603ed35fc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:04:42.862 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:42.862+0000 7f890a45d640 1 --2- 172.21.15.29:0/113167430 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f88f00a2a40 0x7f88f00a4f00 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f88fc004af0 tx=0x7f88fc033c30 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:04:42.863 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:42.862+0000 7f8909c5c640 1 --2- 172.21.15.29:0/3873249760 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5603ed354a50 0x5603ed35d2e0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f8900009c60 tx=0x7f8900011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=930732fa2f2d0c9f server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:04:42.863 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:42.863+0000 7f88effff640 1 -- 172.21.15.29:0/3873249760 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f890001e070 con 0x5603ed354a50 2024-03-16T12:04:42.863 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:42.863+0000 7f88effff640 1 -- 172.21.15.29:0/3873249760 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8900019b10 con 0x5603ed354a50 2024-03-16T12:04:42.863 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:42.863+0000 7f88effff640 1 -- 172.21.15.29:0/3873249760 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f890000ea70 con 0x5603ed354a50 2024-03-16T12:04:42.863 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:42.863+0000 7f890eb85740 1 -- 172.21.15.29:0/3873249760 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5603ed354a50 msgr2=0x5603ed35d2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:04:42.863 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:42.863+0000 7f890eb85740 1 --2- 172.21.15.29:0/3873249760 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5603ed354a50 0x5603ed35d2e0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f8900009c60 tx=0x7f8900011810 comp rx=0 tx=0).stop 2024-03-16T12:04:42.863 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:42.863+0000 7f890eb85740 1 -- 172.21.15.29:0/3873249760 shutdown_connections 2024-03-16T12:04:42.863 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:42.863+0000 7f890eb85740 1 --2- 172.21.15.29:0/3873249760 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5603ed35d820 0x5603ed35fc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:04:42.863 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:42.863+0000 7f890eb85740 1 --2- 172.21.15.29:0/3873249760 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5603ed354a50 0x5603ed35d2e0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:04:42.863 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:42.863+0000 7f890eb85740 1 --2- 172.21.15.29:0/3873249760 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5603ed354130 0x5603ed354510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:04:42.863 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:42.863+0000 7f890eb85740 1 -- 172.21.15.29:0/3873249760 >> 172.21.15.29:0/3873249760 conn(0x5603ed34ea50 msgr2=0x5603ed34ee30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:04:42.863 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:42.863+0000 7f890eb85740 1 -- 172.21.15.29:0/3873249760 shutdown_connections 2024-03-16T12:04:42.863 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:42.863+0000 7f890eb85740 1 -- 172.21.15.29:0/3873249760 wait complete. 2024-03-16T12:04:42.863 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:42.864+0000 7f890eb85740 1 -- start start 2024-03-16T12:04:42.864 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:42.864+0000 7f890eb85740 1 Processor -- start 2024-03-16T12:04:42.864 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:42.864+0000 7f890eb85740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5603ed354130 0x5603ed35a850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:04:42.864 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:42.864+0000 7f890eb85740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5603ed354a50 0x5603ed35ad90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:04:42.864 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:42.864+0000 7f890eb85740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5603ed35d820 0x5603ed35b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:04:42.864 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:42.864+0000 7f890eb85740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x5603ed360450 con 0x5603ed354130 2024-03-16T12:04:42.864 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:42.864+0000 7f890eb85740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x5603ed274d70 con 0x5603ed354a50 2024-03-16T12:04:42.864 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:42.864+0000 7f890eb85740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5603ed3605d0 con 0x5603ed35d820 2024-03-16T12:04:42.864 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:42.864+0000 7f890eb85740 10 client.0 _ll_register_callbacks cb 0x5603ecfcca00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T12:04:42.864 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:42.864+0000 7f8909c5c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5603ed354130 0x5603ed35a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:04:42.864 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:42.864+0000 7f890945b640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5603ed354a50 0x5603ed35ad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:04:42.864 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:42.864+0000 7f890a45d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5603ed35d820 0x5603ed35b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:04:42.864 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:42.864+0000 7f8909c5c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5603ed354130 0x5603ed35a850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59972/0 (socket says 172.21.15.29:59972) 2024-03-16T12:04:42.864 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:42.864+0000 7f8909c5c640 1 -- 172.21.15.29:0/3773021327 learned_addr learned my addr 172.21.15.29:0/3773021327 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T12:04:42.864 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:42.865+0000 7f8909c5c640 1 -- 172.21.15.29:0/3773021327 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5603ed35d820 msgr2=0x5603ed35b2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:04:42.864 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:42.865+0000 7f8909c5c640 1 --2- 172.21.15.29:0/3773021327 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5603ed35d820 0x5603ed35b2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:04:42.865 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:42.865+0000 7f8909c5c640 1 -- 172.21.15.29:0/3773021327 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5603ed354a50 msgr2=0x5603ed35ad90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:04:42.865 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:42.865+0000 7f8909c5c640 1 --2- 172.21.15.29:0/3773021327 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5603ed354a50 0x5603ed35ad90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:04:42.865 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:42.865+0000 7f8909c5c640 1 -- 172.21.15.29:0/3773021327 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5603ecfcd210 con 0x5603ed354130 2024-03-16T12:04:42.865 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:42.865+0000 7f890a45d640 1 --2- 172.21.15.29:0/3773021327 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5603ed35d820 0x5603ed35b2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:04:42.865 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:42.865+0000 7f890945b640 1 --2- 172.21.15.29:0/3773021327 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5603ed354a50 0x5603ed35ad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:04:42.865 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:42.865+0000 7f8909c5c640 1 --2- 172.21.15.29:0/3773021327 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5603ed354130 0x5603ed35a850 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f8900009230 tx=0x7f8900005e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b82eda2d3736794 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:04:42.865 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:42.865+0000 7f88ed7fa640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-16T12:04:42.865 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:42.865+0000 7f88ed7fa640 1 -- 172.21.15.29:0/3773021327 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8900028020 con 0x5603ed354130 2024-03-16T12:04:42.865 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:42.865+0000 7f88ed7fa640 1 -- 172.21.15.29:0/3773021327 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(110..110 src has 1..110) ==== 8556+0+0 (secure 0 0 0) 0x7f8900005040 con 0x5603ed354130 2024-03-16T12:04:42.865 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:42.865+0000 7f890eb85740 1 -- 172.21.15.29:0/3773021327 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8900019b10 con 0x5603ed354130 2024-03-16T12:04:42.865 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:42.865+0000 7f88ed7fa640 1 -- 172.21.15.29:0/3773021327 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f8900033050 con 0x5603ed354130 2024-03-16T12:04:42.865 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:42.865+0000 7f890eb85740 10 client.16551 fetch_fsmap learned FSMap version 31 2024-03-16T12:04:42.865 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:42.865+0000 7f890eb85740 1 -- 172.21.15.29:0/3773021327 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5603ed099620 con 0x5603ed354130 2024-03-16T12:04:42.865 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:42.865+0000 7f88ed7fa640 1 -- 172.21.15.29:0/3773021327 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8900019b10 con 0x5603ed354130 2024-03-16T12:04:42.865 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:42.866+0000 7f88ed7fa640 1 -- 172.21.15.29:0/3773021327 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 31) ==== 8346+0+0 (secure 0 0 0) 0x7f8900021030 con 0x5603ed354130 2024-03-16T12:04:42.866 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:42.866+0000 7f890eb85740 10 client.16551 fetch_fsmap finished waiting for FSMap version 31 2024-03-16T12:04:42.866 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:42.866+0000 7f890eb85740 10 client.16551 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T12:04:42.866 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:42.866+0000 7f890eb85740 20 client.16551 populate_metadata read hostname 'smithi029' 2024-03-16T12:04:42.866 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:42.866+0000 7f890eb85740 1 --2- 172.21.15.29:0/3773021327 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x5603ed36d940 0x5603ed36fd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:04:42.866 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:42.866+0000 7f890eb85740 4 client.16551 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T12:04:42.866 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:42.866+0000 7f890eb85740 1 -- 172.21.15.29:0/3773021327 --> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5603ed2c5f80 con 0x5603ed36d940 2024-03-16T12:04:42.866 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:42.866+0000 7f890945b640 1 --2- 172.21.15.29:0/3773021327 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x5603ed36d940 0x5603ed36fd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:04:42.866 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:42.866+0000 7f890945b640 1 --2- 172.21.15.29:0/3773021327 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x5603ed36d940 0x5603ed36fd30 crc :-1 s=READY pgs=162 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d41b56d04d5733d4 server_cookie=ed4841d89098979c in_seq=0 out_seq=0 2024-03-16T12:04:42.866 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:42.866+0000 7f88ed7fa640 10 client.16551 ms_handle_connect on v2:172.21.15.152:6838/1548731833 2024-03-16T12:04:42.867 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:42.870+0000 7f88ed7fa640 1 -- 172.21.15.29:0/3773021327 <== mds.0 v2:172.21.15.152:6838/1548731833 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5603ed2c5f80 con 0x5603ed36d940 2024-03-16T12:04:42.867 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:42.870+0000 7f88ed7fa640 10 client.16551 handle_command_reply: tid=0 2024-03-16T12:04:42.867 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:42.870+0000 7f890eb85740 1 client.16551 shutdown 2024-03-16T12:04:42.867 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:42.870+0000 7f890eb85740 1 -- 172.21.15.29:0/3773021327 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5603ed354130 msgr2=0x5603ed35a850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:04:42.867 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:42.870+0000 7f890eb85740 1 --2- 172.21.15.29:0/3773021327 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5603ed354130 0x5603ed35a850 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f8900009230 tx=0x7f8900005e30 comp rx=0 tx=0).stop 2024-03-16T12:04:42.868 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:42.870+0000 7f890eb85740 1 -- 172.21.15.29:0/3773021327 shutdown_connections 2024-03-16T12:04:42.868 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:42.870+0000 7f890eb85740 1 --2- 172.21.15.29:0/3773021327 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5603ed35d820 0x5603ed35b2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:04:42.868 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:42.870+0000 7f890eb85740 1 --2- 172.21.15.29:0/3773021327 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x5603ed36d940 0x5603ed36fd30 crc :-1 s=READY pgs=162 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:04:42.868 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:42.870+0000 7f890eb85740 1 --2- 172.21.15.29:0/3773021327 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5603ed354a50 0x5603ed35ad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:04:42.868 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:42.870+0000 7f890eb85740 1 --2- 172.21.15.29:0/3773021327 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5603ed354130 0x5603ed35a850 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:04:42.868 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:42.870+0000 7f88ed7fa640 0 client.16551 ms_handle_reset on v2:172.21.15.152:6838/1548731833 2024-03-16T12:04:42.868 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:42.870+0000 7f890eb85740 1 -- 172.21.15.29:0/3773021327 >> 172.21.15.29:0/3773021327 conn(0x5603ed34ea50 msgr2=0x5603ed3507a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:04:42.868 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:42.871+0000 7f890eb85740 1 -- 172.21.15.29:0/3773021327 shutdown_connections 2024-03-16T12:04:42.868 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:42.871+0000 7f890eb85740 1 -- 172.21.15.29:0/3773021327 wait complete. 2024-03-16T12:04:42.868 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:42.871+0000 7f890eb85740 20 client.16551 trim_cache size 0 max 16384 2024-03-16T12:04:42.884 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:42.885+0000 7f890eb85740 1 Processor -- start 2024-03-16T12:04:42.884 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:42.885+0000 7f890eb85740 1 -- start start 2024-03-16T12:04:42.884 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:42.885+0000 7f890eb85740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5603ed354130 0x5603ed35cf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:04:42.884 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:42.885+0000 7f890eb85740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5603ed354a50 0x5603ed349a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:04:42.884 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:42.885+0000 7f890eb85740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5603ed35d820 0x5603ed3691f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:04:42.885 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:42.885+0000 7f890eb85740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f89041b1fa0 con 0x5603ed354a50 2024-03-16T12:04:42.885 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:42.885+0000 7f890eb85740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f89041af810 con 0x5603ed35d820 2024-03-16T12:04:42.885 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:42.885+0000 7f890eb85740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f89041ad680 con 0x5603ed354130 2024-03-16T12:04:42.885 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:42.886+0000 7f890945b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5603ed354a50 0x5603ed349a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:04:42.885 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:42.886+0000 7f890945b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5603ed354a50 0x5603ed349a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59976/0 (socket says 172.21.15.29:59976) 2024-03-16T12:04:42.885 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:42.886+0000 7f890945b640 1 -- 172.21.15.29:0/841221410 learned_addr learned my addr 172.21.15.29:0/841221410 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T12:04:42.885 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:42.886+0000 7f8909c5c640 1 --2- 172.21.15.29:0/841221410 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5603ed354130 0x5603ed35cf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:04:42.885 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:42.886+0000 7f890a45d640 1 --2- 172.21.15.29:0/841221410 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5603ed35d820 0x5603ed3691f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:04:42.885 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:42.886+0000 7f890945b640 1 -- 172.21.15.29:0/841221410 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5603ed354130 msgr2=0x5603ed35cf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:04:42.885 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:42.886+0000 7f890945b640 1 --2- 172.21.15.29:0/841221410 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5603ed354130 0x5603ed35cf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:04:42.885 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:42.886+0000 7f890945b640 1 -- 172.21.15.29:0/841221410 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5603ed35d820 msgr2=0x5603ed3691f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:04:42.885 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:42.886+0000 7f890945b640 1 --2- 172.21.15.29:0/841221410 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5603ed35d820 0x5603ed3691f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:04:42.885 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:42.886+0000 7f890945b640 1 -- 172.21.15.29:0/841221410 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5603ed359820 con 0x5603ed354a50 2024-03-16T12:04:42.885 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:42.886+0000 7f890a45d640 1 --2- 172.21.15.29:0/841221410 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5603ed35d820 0x5603ed3691f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T12:04:42.885 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:42.886+0000 7f890945b640 1 --2- 172.21.15.29:0/841221410 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5603ed354a50 0x5603ed349a40 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f88f4097d00 tx=0x7f88f406a700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ebffc9c81c36b85 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:04:42.886 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:42.886+0000 7f88ef7fe640 1 -- 172.21.15.29:0/841221410 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88f4026040 con 0x5603ed354a50 2024-03-16T12:04:42.886 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:42.886+0000 7f88ef7fe640 1 -- 172.21.15.29:0/841221410 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f88f40a4090 con 0x5603ed354a50 2024-03-16T12:04:42.886 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:42.886+0000 7f88ef7fe640 1 -- 172.21.15.29:0/841221410 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88f40a3e60 con 0x5603ed354a50 2024-03-16T12:04:42.886 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:42.887+0000 7f890eb85740 1 -- 172.21.15.29:0/841221410 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5603ed354a50 msgr2=0x5603ed349a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:04:42.886 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:42.887+0000 7f890eb85740 1 --2- 172.21.15.29:0/841221410 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5603ed354a50 0x5603ed349a40 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f88f4097d00 tx=0x7f88f406a700 comp rx=0 tx=0).stop 2024-03-16T12:04:42.886 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:42.887+0000 7f890eb85740 1 -- 172.21.15.29:0/841221410 shutdown_connections 2024-03-16T12:04:42.886 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:42.887+0000 7f890eb85740 1 --2- 172.21.15.29:0/841221410 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5603ed35d820 0x5603ed3691f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:04:42.886 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:42.887+0000 7f890eb85740 1 --2- 172.21.15.29:0/841221410 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5603ed354a50 0x5603ed349a40 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:04:42.886 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:42.887+0000 7f890eb85740 1 --2- 172.21.15.29:0/841221410 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5603ed354130 0x5603ed35cf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:04:42.886 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:42.887+0000 7f890eb85740 1 -- 172.21.15.29:0/841221410 >> 172.21.15.29:0/841221410 conn(0x5603ed34ea50 msgr2=0x5603ed34dbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:04:42.886 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:42.887+0000 7f890eb85740 1 -- 172.21.15.29:0/841221410 shutdown_connections 2024-03-16T12:04:42.886 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:42.887+0000 7f890eb85740 1 -- 172.21.15.29:0/841221410 wait complete. 2024-03-16T12:04:42.886 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:42.887+0000 7f890eb85740 1 -- start start 2024-03-16T12:04:42.886 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:42.887+0000 7f890eb85740 1 Processor -- start 2024-03-16T12:04:42.887 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:42.888+0000 7f890eb85740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5603ed354130 0x5603ed357520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:04:42.887 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:42.888+0000 7f890eb85740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5603ed354a50 0x5603ed357a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:04:42.887 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:42.888+0000 7f890eb85740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5603ed35d820 0x5603ed357fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:04:42.887 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:42.888+0000 7f890eb85740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f89041adcd0 con 0x5603ed354130 2024-03-16T12:04:42.887 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:42.888+0000 7f890eb85740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f89041af140 con 0x5603ed354a50 2024-03-16T12:04:42.887 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:42.888+0000 7f890eb85740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f89041ae3a0 con 0x5603ed35d820 2024-03-16T12:04:42.887 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:42.888+0000 7f890eb85740 10 client.0 _ll_register_callbacks cb 0x7f890001c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T12:04:42.887 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:42.888+0000 7f8909c5c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5603ed354130 0x5603ed357520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:04:42.887 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:42.888+0000 7f8909c5c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5603ed354130 0x5603ed357520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59984/0 (socket says 172.21.15.29:59984) 2024-03-16T12:04:42.887 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:42.888+0000 7f8909c5c640 1 -- 172.21.15.29:0/14556103 learned_addr learned my addr 172.21.15.29:0/14556103 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T12:04:42.887 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:42.888+0000 7f890945b640 1 --2- 172.21.15.29:0/14556103 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5603ed354a50 0x5603ed357a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:04:42.887 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:42.888+0000 7f890a45d640 1 --2- 172.21.15.29:0/14556103 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5603ed35d820 0x5603ed357fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:04:42.887 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:42.888+0000 7f8909c5c640 1 -- 172.21.15.29:0/14556103 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5603ed35d820 msgr2=0x5603ed357fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:04:42.887 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:42.888+0000 7f8909c5c640 1 --2- 172.21.15.29:0/14556103 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5603ed35d820 0x5603ed357fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:04:42.887 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:42.888+0000 7f8909c5c640 1 -- 172.21.15.29:0/14556103 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5603ed354a50 msgr2=0x5603ed357a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:04:42.888 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:42.888+0000 7f8909c5c640 1 --2- 172.21.15.29:0/14556103 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5603ed354a50 0x5603ed357a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:04:42.888 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:42.888+0000 7f8909c5c640 1 -- 172.21.15.29:0/14556103 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5603ed3d2cb0 con 0x5603ed354130 2024-03-16T12:04:42.888 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:42.888+0000 7f890945b640 1 --2- 172.21.15.29:0/14556103 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5603ed354a50 0x5603ed357a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T12:04:42.888 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:42.888+0000 7f8909c5c640 1 --2- 172.21.15.29:0/14556103 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5603ed354130 0x5603ed357520 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f8900009b80 tx=0x7f89000092d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53e98cc0c48acf76 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:04:42.888 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:42.888+0000 7f88ed7fa640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-16T12:04:42.888 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:42.888+0000 7f88ed7fa640 1 -- 172.21.15.29:0/14556103 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8900028040 con 0x5603ed354130 2024-03-16T12:04:42.888 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:42.888+0000 7f890eb85740 1 -- 172.21.15.29:0/14556103 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f88f40a4090 con 0x5603ed354130 2024-03-16T12:04:42.888 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:42.888+0000 7f88ed7fa640 1 -- 172.21.15.29:0/14556103 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(110..110 src has 1..110) ==== 8556+0+0 (secure 0 0 0) 0x7f8900004760 con 0x5603ed354130 2024-03-16T12:04:42.888 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:42.889+0000 7f88ed7fa640 1 -- 172.21.15.29:0/14556103 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f890001e040 con 0x5603ed354130 2024-03-16T12:04:42.888 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:42.889+0000 7f890eb85740 10 client.16557 fetch_fsmap learned FSMap version 31 2024-03-16T12:04:42.888 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:42.889+0000 7f890eb85740 1 -- 172.21.15.29:0/14556103 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5603ed3d3f40 con 0x5603ed354130 2024-03-16T12:04:42.888 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:42.889+0000 7f88ed7fa640 1 -- 172.21.15.29:0/14556103 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f88f40a4090 con 0x5603ed354130 2024-03-16T12:04:42.888 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:42.889+0000 7f88ed7fa640 1 -- 172.21.15.29:0/14556103 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 31) ==== 8346+0+0 (secure 0 0 0) 0x7f8900021050 con 0x5603ed354130 2024-03-16T12:04:42.888 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:42.889+0000 7f890eb85740 10 client.16557 fetch_fsmap finished waiting for FSMap version 31 2024-03-16T12:04:42.888 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:42.889+0000 7f890eb85740 10 client.16557 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T12:04:42.889 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:42.889+0000 7f890eb85740 20 client.16557 populate_metadata read hostname 'smithi029' 2024-03-16T12:04:42.889 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:42.889+0000 7f890eb85740 1 --2- 172.21.15.29:0/14556103 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x5603ed3d4680 0x5603ed3d6aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:04:42.889 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:42.889+0000 7f890eb85740 4 client.16557 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-16T12:04:42.889 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:42.889+0000 7f890eb85740 1 -- 172.21.15.29:0/14556103 --> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5603ed2c02d0 con 0x5603ed3d4680 2024-03-16T12:04:42.889 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:42.889+0000 7f890945b640 1 --2- 172.21.15.29:0/14556103 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x5603ed3d4680 0x5603ed3d6aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:04:42.889 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:42.890+0000 7f890945b640 1 --2- 172.21.15.29:0/14556103 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x5603ed3d4680 0x5603ed3d6aa0 crc :-1 s=READY pgs=163 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bdd6c4eb46dc3e7e server_cookie=777dac4bd98074c8 in_seq=0 out_seq=0 2024-03-16T12:04:42.889 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:42.890+0000 7f88ed7fa640 10 client.16557 ms_handle_connect on v2:172.21.15.152:6838/1548731833 2024-03-16T12:04:43.183 INFO:tasks.workunit.client.0.smithi029.stdout: 10 8610 23.11 MB/sec warmup 27 sec latency 524.303 ms 2024-03-16T12:04:43.431 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:43.436+0000 7f88ed7fa640 1 -- 172.21.15.29:0/14556103 <== mds.0 v2:172.21.15.152:6838/1548731833 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5603ed2c02d0 con 0x5603ed3d4680 2024-03-16T12:04:43.431 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:43.436+0000 7f88ed7fa640 10 client.16557 handle_command_reply: tid=0 2024-03-16T12:04:43.431 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:43.436+0000 7f890eb85740 1 client.16557 shutdown 2024-03-16T12:04:43.431 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:43.436+0000 7f890eb85740 1 -- 172.21.15.29:0/14556103 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5603ed354130 msgr2=0x5603ed357520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:04:43.431 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:43.436+0000 7f890eb85740 1 --2- 172.21.15.29:0/14556103 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5603ed354130 0x5603ed357520 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f8900009b80 tx=0x7f89000092d0 comp rx=0 tx=0).stop 2024-03-16T12:04:43.431 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:43.436+0000 7f890eb85740 1 -- 172.21.15.29:0/14556103 shutdown_connections 2024-03-16T12:04:43.431 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:43.436+0000 7f890eb85740 1 --2- 172.21.15.29:0/14556103 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5603ed35d820 0x5603ed357fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:04:43.431 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:43.436+0000 7f890eb85740 1 --2- 172.21.15.29:0/14556103 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x5603ed3d4680 0x5603ed3d6aa0 crc :-1 s=READY pgs=163 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:04:43.431 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:43.436+0000 7f890eb85740 1 --2- 172.21.15.29:0/14556103 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5603ed354a50 0x5603ed357a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:04:43.431 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:43.436+0000 7f890eb85740 1 --2- 172.21.15.29:0/14556103 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5603ed354130 0x5603ed357520 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:04:43.431 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:43.436+0000 7f88ed7fa640 0 client.16557 ms_handle_reset on v2:172.21.15.152:6838/1548731833 2024-03-16T12:04:43.431 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:43.436+0000 7f890eb85740 1 -- 172.21.15.29:0/14556103 >> 172.21.15.29:0/14556103 conn(0x5603ed34ea50 msgr2=0x5603ed34d320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:04:43.432 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:43.436+0000 7f890eb85740 1 -- 172.21.15.29:0/14556103 shutdown_connections 2024-03-16T12:04:43.432 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:43.436+0000 7f890eb85740 1 -- 172.21.15.29:0/14556103 wait complete. 2024-03-16T12:04:43.432 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:43.436+0000 7f890eb85740 20 client.16557 trim_cache size 0 max 16384 2024-03-16T12:04:43.432 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:{ 2024-03-16T12:04:43.432 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "message": "", 2024-03-16T12:04:43.432 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "return_code": 0 2024-03-16T12:04:43.432 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:} 2024-03-16T12:04:43.433 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:43.437+0000 7f890bec0640 1 -- 172.21.15.29:0/113167430 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f88f00a2a40 msgr2=0x7f88f00a4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:04:43.433 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:43.437+0000 7f890bec0640 1 --2- 172.21.15.29:0/113167430 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f88f00a2a40 0x7f88f00a4f00 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f88fc004af0 tx=0x7f88fc033c30 comp rx=0 tx=0).stop 2024-03-16T12:04:43.433 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:43.437+0000 7f890bec0640 1 -- 172.21.15.29:0/113167430 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8904069800 msgr2=0x7f8904100f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:04:43.433 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:43.437+0000 7f890bec0640 1 --2- 172.21.15.29:0/113167430 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8904069800 0x7f8904100f40 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f88f400e9f0 tx=0x7f88f400edf0 comp rx=0 tx=0).stop 2024-03-16T12:04:43.433 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:43.437+0000 7f890bec0640 1 -- 172.21.15.29:0/113167430 shutdown_connections 2024-03-16T12:04:43.433 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:43.437+0000 7f890bec0640 1 --2- 172.21.15.29:0/113167430 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f88f00a2a40 0x7f88f00a4f00 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:04:43.433 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:43.437+0000 7f890bec0640 1 --2- 172.21.15.29:0/113167430 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f89041078b0 0x7f8904101480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:04:43.433 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:43.437+0000 7f890bec0640 1 --2- 172.21.15.29:0/113167430 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8904069800 0x7f8904100f40 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:04:43.433 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:43.437+0000 7f890bec0640 1 --2- 172.21.15.29:0/113167430 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8904068e60 0x7f8904100a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:04:43.434 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:43.437+0000 7f890bec0640 1 -- 172.21.15.29:0/113167430 >> 172.21.15.29:0/113167430 conn(0x7f8904077530 msgr2=0x7f89040fe980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:04:43.434 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:43.437+0000 7f890bec0640 1 -- 172.21.15.29:0/113167430 shutdown_connections 2024-03-16T12:04:43.434 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:43.437+0000 7f890bec0640 1 -- 172.21.15.29:0/113167430 wait complete. 2024-03-16T12:04:44.183 INFO:tasks.workunit.client.0.smithi029.stdout: 10 8719 22.40 MB/sec warmup 28 sec latency 660.852 ms 2024-03-16T12:04:44.540 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:04:44 smithi154 ceph-mon[36320]: pgmap v439: 97 pgs: 97 active+clean; 83 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 840 KiB/s rd, 2.4 MiB/s wr, 190 op/s 2024-03-16T12:04:44.583 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:04:44 smithi029 ceph-mon[31975]: pgmap v439: 97 pgs: 97 active+clean; 83 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 840 KiB/s rd, 2.4 MiB/s wr, 190 op/s 2024-03-16T12:04:44.622 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:04:44 smithi152 ceph-mon[36291]: pgmap v439: 97 pgs: 97 active+clean; 83 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 840 KiB/s rd, 2.4 MiB/s wr, 190 op/s 2024-03-16T12:04:45.183 INFO:tasks.workunit.client.0.smithi029.stdout: 10 9003 22.04 MB/sec warmup 29 sec latency 267.898 ms 2024-03-16T12:04:46.183 INFO:tasks.workunit.client.0.smithi029.stdout: 10 9492 22.72 MB/sec warmup 30 sec latency 43.330 ms 2024-03-16T12:04:46.540 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:04:46 smithi154 ceph-mon[36320]: pgmap v440: 97 pgs: 97 active+clean; 83 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 835 KiB/s rd, 2.3 MiB/s wr, 157 op/s 2024-03-16T12:04:46.582 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:04:46 smithi029 ceph-mon[31975]: pgmap v440: 97 pgs: 97 active+clean; 83 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 835 KiB/s rd, 2.3 MiB/s wr, 157 op/s 2024-03-16T12:04:46.622 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:04:46 smithi152 ceph-mon[36291]: pgmap v440: 97 pgs: 97 active+clean; 83 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 835 KiB/s rd, 2.3 MiB/s wr, 157 op/s 2024-03-16T12:04:47.183 INFO:tasks.workunit.client.0.smithi029.stdout: 10 9918 23.11 MB/sec warmup 31 sec latency 271.324 ms 2024-03-16T12:04:48.183 INFO:tasks.workunit.client.0.smithi029.stdout: 10 10139 22.90 MB/sec warmup 32 sec latency 238.567 ms 2024-03-16T12:04:48.540 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:04:48 smithi154 ceph-mon[36320]: pgmap v441: 97 pgs: 97 active+clean; 248 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 18 MiB/s wr, 433 op/s 2024-03-16T12:04:48.583 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:04:48 smithi029 ceph-mon[31975]: pgmap v441: 97 pgs: 97 active+clean; 248 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 18 MiB/s wr, 433 op/s 2024-03-16T12:04:48.622 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:04:48 smithi152 ceph-mon[36291]: pgmap v441: 97 pgs: 97 active+clean; 248 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 18 MiB/s wr, 433 op/s 2024-03-16T12:04:49.183 INFO:tasks.workunit.client.0.smithi029.stdout: 10 10307 22.60 MB/sec warmup 33 sec latency 270.964 ms 2024-03-16T12:04:50.183 INFO:tasks.workunit.client.0.smithi029.stdout: 10 10457 22.05 MB/sec warmup 34 sec latency 230.411 ms 2024-03-16T12:04:50.540 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:04:50 smithi154 ceph-mon[36320]: pgmap v442: 97 pgs: 97 active+clean; 248 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1013 KiB/s rd, 17 MiB/s wr, 353 op/s 2024-03-16T12:04:50.583 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:04:50 smithi029 ceph-mon[31975]: pgmap v442: 97 pgs: 97 active+clean; 248 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1013 KiB/s rd, 17 MiB/s wr, 353 op/s 2024-03-16T12:04:50.622 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:04:50 smithi152 ceph-mon[36291]: pgmap v442: 97 pgs: 97 active+clean; 248 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1013 KiB/s rd, 17 MiB/s wr, 353 op/s 2024-03-16T12:04:51.183 INFO:tasks.workunit.client.0.smithi029.stdout: 10 10585 21.43 MB/sec warmup 35 sec latency 237.231 ms 2024-03-16T12:04:52.183 INFO:tasks.workunit.client.0.smithi029.stdout: 10 10749 20.90 MB/sec warmup 36 sec latency 376.327 ms 2024-03-16T12:04:52.540 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:04:52 smithi154 ceph-mon[36320]: pgmap v443: 97 pgs: 97 active+clean; 253 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 17 MiB/s wr, 412 op/s 2024-03-16T12:04:52.582 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:04:52 smithi029 ceph-mon[31975]: pgmap v443: 97 pgs: 97 active+clean; 253 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 17 MiB/s wr, 412 op/s 2024-03-16T12:04:52.622 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:04:52 smithi152 ceph-mon[36291]: pgmap v443: 97 pgs: 97 active+clean; 253 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 17 MiB/s wr, 412 op/s 2024-03-16T12:04:53.183 INFO:tasks.workunit.client.0.smithi029.stdout: 10 11052 20.57 MB/sec warmup 37 sec latency 229.939 ms 2024-03-16T12:04:54.183 INFO:tasks.workunit.client.0.smithi029.stdout: 10 11435 20.74 MB/sec warmup 38 sec latency 252.034 ms 2024-03-16T12:04:54.540 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:04:54 smithi154 ceph-mon[36320]: pgmap v444: 97 pgs: 97 active+clean; 253 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 17 MiB/s wr, 366 op/s 2024-03-16T12:04:54.582 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:04:54 smithi029 ceph-mon[31975]: pgmap v444: 97 pgs: 97 active+clean; 253 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 17 MiB/s wr, 366 op/s 2024-03-16T12:04:54.622 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:04:54 smithi152 ceph-mon[36291]: pgmap v444: 97 pgs: 97 active+clean; 253 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 17 MiB/s wr, 366 op/s 2024-03-16T12:04:55.183 INFO:tasks.workunit.client.0.smithi029.stdout: 10 11992 20.70 MB/sec warmup 39 sec latency 63.033 ms 2024-03-16T12:04:56.183 INFO:tasks.workunit.client.0.smithi029.stdout: 10 12496 20.90 MB/sec warmup 40 sec latency 87.156 ms 2024-03-16T12:04:56.523 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:56.528+0000 7f623575b640 1 -- 172.21.15.29:0/2733079999 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6230102150 msgr2=0x7f623010e660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:04:56.523 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:56.528+0000 7f623575b640 1 --2- 172.21.15.29:0/2733079999 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6230102150 0x7f623010e660 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f6228009cd0 tx=0x7f622802c0f0 comp rx=0 tx=0).stop 2024-03-16T12:04:56.523 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:56.528+0000 7f623575b640 1 -- 172.21.15.29:0/2733079999 shutdown_connections 2024-03-16T12:04:56.523 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:56.528+0000 7f623575b640 1 --2- 172.21.15.29:0/2733079999 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6230102150 0x7f623010e660 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:04:56.523 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:56.528+0000 7f623575b640 1 --2- 172.21.15.29:0/2733079999 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6230101790 0x7f6230101c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:04:56.523 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:56.528+0000 7f623575b640 1 --2- 172.21.15.29:0/2733079999 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f62301077a0 0x7f6230107b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:04:56.523 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:56.528+0000 7f623575b640 1 -- 172.21.15.29:0/2733079999 >> 172.21.15.29:0/2733079999 conn(0x7f62300fd2a0 msgr2=0x7f62300ff6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:04:56.523 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:56.528+0000 7f623575b640 1 -- 172.21.15.29:0/2733079999 shutdown_connections 2024-03-16T12:04:56.523 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:56.528+0000 7f623575b640 1 -- 172.21.15.29:0/2733079999 wait complete. 2024-03-16T12:04:56.523 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:56.529+0000 7f623575b640 1 Processor -- start 2024-03-16T12:04:56.523 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:56.529+0000 7f623575b640 1 -- start start 2024-03-16T12:04:56.525 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:56.529+0000 7f623575b640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6230101790 0x7f6230073ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:04:56.525 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:56.529+0000 7f623575b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6230102150 0x7f6230074020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:04:56.526 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:56.529+0000 7f623575b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f62301077a0 0x7f6230074560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:04:56.526 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:56.529+0000 7f623575b640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f6230111bb0 con 0x7f6230102150 2024-03-16T12:04:56.526 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:56.529+0000 7f623575b640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f6230111a30 con 0x7f6230101790 2024-03-16T12:04:56.526 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:56.529+0000 7f623575b640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f6230111d30 con 0x7f62301077a0 2024-03-16T12:04:56.526 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:56.529+0000 7f622e7fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6230102150 0x7f6230074020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:04:56.526 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:56.529+0000 7f622e7fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6230102150 0x7f6230074020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:48822/0 (socket says 172.21.15.29:48822) 2024-03-16T12:04:56.526 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:56.529+0000 7f622e7fc640 1 -- 172.21.15.29:0/3689345658 learned_addr learned my addr 172.21.15.29:0/3689345658 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T12:04:56.526 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:56.529+0000 7f622effd640 1 --2- 172.21.15.29:0/3689345658 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6230101790 0x7f6230073ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:04:56.526 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:56.529+0000 7f622f7fe640 1 --2- 172.21.15.29:0/3689345658 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f62301077a0 0x7f6230074560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:04:56.526 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:56.530+0000 7f622e7fc640 1 -- 172.21.15.29:0/3689345658 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f62301077a0 msgr2=0x7f6230074560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:04:56.526 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:56.530+0000 7f622e7fc640 1 --2- 172.21.15.29:0/3689345658 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f62301077a0 0x7f6230074560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:04:56.526 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:56.530+0000 7f622e7fc640 1 -- 172.21.15.29:0/3689345658 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6230101790 msgr2=0x7f6230073ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:04:56.526 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:56.530+0000 7f622e7fc640 1 --2- 172.21.15.29:0/3689345658 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6230101790 0x7f6230073ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:04:56.526 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:56.530+0000 7f622e7fc640 1 -- 172.21.15.29:0/3689345658 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62301a64b0 con 0x7f6230102150 2024-03-16T12:04:56.526 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:56.530+0000 7f622e7fc640 1 --2- 172.21.15.29:0/3689345658 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6230102150 0x7f6230074020 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f622400e9f0 tx=0x7f622400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:04:56.526 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:56.530+0000 7f6217fff640 1 -- 172.21.15.29:0/3689345658 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f622400cd80 con 0x7f6230102150 2024-03-16T12:04:56.527 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:56.530+0000 7f623575b640 1 -- 172.21.15.29:0/3689345658 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f62301a67a0 con 0x7f6230102150 2024-03-16T12:04:56.527 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:56.530+0000 7f623575b640 1 -- 172.21.15.29:0/3689345658 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f62301a6cf0 con 0x7f6230102150 2024-03-16T12:04:56.527 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:56.530+0000 7f6217fff640 1 -- 172.21.15.29:0/3689345658 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6224005500 con 0x7f6230102150 2024-03-16T12:04:56.527 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:56.530+0000 7f6217fff640 1 -- 172.21.15.29:0/3689345658 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6224010690 con 0x7f6230102150 2024-03-16T12:04:56.532 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:56.533+0000 7f6217fff640 1 -- 172.21.15.29:0/3689345658 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7f62240046c0 con 0x7f6230102150 2024-03-16T12:04:56.532 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:56.534+0000 7f6238428740 1 Processor -- start 2024-03-16T12:04:56.532 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:56.534+0000 7f6238428740 1 -- start start 2024-03-16T12:04:56.532 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:56.534+0000 7f6238428740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557610dbe130 0x557610dbe510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:04:56.532 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:56.534+0000 7f6238428740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557610dbea50 0x557610dc72e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:04:56.532 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:56.534+0000 7f6238428740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557610dc7820 0x557610dc9c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:04:56.532 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:56.534+0000 7f6217fff640 1 --2- 172.21.15.29:0/3689345658 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f62180a2a90 0x7f62180a4f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:04:56.533 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:56.534+0000 7f6238428740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x557610d22a40 con 0x557610dc7820 2024-03-16T12:04:56.533 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:56.534+0000 7f6238428740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x557610ccb920 con 0x557610dbea50 2024-03-16T12:04:56.533 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:56.534+0000 7f6238428740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x557610d1eae0 con 0x557610dbe130 2024-03-16T12:04:56.533 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:56.534+0000 7f622e7fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557610dc7820 0x557610dc9c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:04:56.533 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:56.534+0000 7f622f7fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557610dbe130 0x557610dbe510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:04:56.533 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:56.534+0000 7f622e7fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557610dc7820 0x557610dc9c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:48828/0 (socket says 172.21.15.29:48828) 2024-03-16T12:04:56.533 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:56.534+0000 7f622f7fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557610dbe130 0x557610dbe510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.29:46566/0 (socket says 172.21.15.29:46566) 2024-03-16T12:04:56.533 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:56.534+0000 7f622e7fc640 1 -- 172.21.15.29:0/191682390 learned_addr learned my addr 172.21.15.29:0/191682390 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T12:04:56.533 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:56.534+0000 7f622f7fe640 1 --2- 172.21.15.29:0/3689345658 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f62180a2a90 0x7f62180a4f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:04:56.533 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:56.534+0000 7f6217fff640 1 -- 172.21.15.29:0/3689345658 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 8556+0+0 (secure 0 0 0) 0x7f62240cf0c0 con 0x7f6230102150 2024-03-16T12:04:56.533 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:56.534+0000 7f622effd640 1 --2- 172.21.15.29:0/191682390 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557610dbea50 0x557610dc72e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:04:56.533 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:56.534+0000 7f622e7fc640 1 -- 172.21.15.29:0/191682390 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557610dbe130 msgr2=0x557610dbe510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:04:56.533 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:56.534+0000 7f622e7fc640 1 --2- 172.21.15.29:0/191682390 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557610dbe130 0x557610dbe510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:04:56.533 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:56.534+0000 7f622e7fc640 1 -- 172.21.15.29:0/191682390 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557610dbea50 msgr2=0x557610dc72e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:04:56.533 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:56.534+0000 7f622e7fc640 1 --2- 172.21.15.29:0/191682390 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557610dbea50 0x557610dc72e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:04:56.533 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:56.534+0000 7f622e7fc640 1 -- 172.21.15.29:0/191682390 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557610ce3460 con 0x557610dc7820 2024-03-16T12:04:56.534 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:56.535+0000 7f622e7fc640 1 --2- 172.21.15.29:0/191682390 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557610dc7820 0x557610dc9c10 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f6224098070 tx=0x7f622400cfd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4687d959677eef51 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:04:56.534 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:56.535+0000 7f622f7fe640 1 --2- 172.21.15.29:0/3689345658 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f62180a2a90 0x7f62180a4f50 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f6228005cc0 tx=0x7f6228005c50 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:04:56.534 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:56.535+0000 7f62157fa640 1 -- 172.21.15.29:0/191682390 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62240056b0 con 0x557610dc7820 2024-03-16T12:04:56.534 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:56.535+0000 7f62157fa640 1 -- 172.21.15.29:0/191682390 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f622409bda0 con 0x557610dc7820 2024-03-16T12:04:56.534 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:56.535+0000 7f62157fa640 1 -- 172.21.15.29:0/191682390 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62240cca10 con 0x557610dc7820 2024-03-16T12:04:56.534 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:56.535+0000 7f6238428740 1 -- 172.21.15.29:0/191682390 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557610dc7820 msgr2=0x557610dc9c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:04:56.534 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:56.535+0000 7f6238428740 1 --2- 172.21.15.29:0/191682390 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557610dc7820 0x557610dc9c10 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f6224098070 tx=0x7f622400cfd0 comp rx=0 tx=0).stop 2024-03-16T12:04:56.534 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:56.535+0000 7f6238428740 1 -- 172.21.15.29:0/191682390 shutdown_connections 2024-03-16T12:04:56.534 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:56.535+0000 7f6238428740 1 --2- 172.21.15.29:0/191682390 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557610dc7820 0x557610dc9c10 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:04:56.534 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:56.535+0000 7f6238428740 1 --2- 172.21.15.29:0/191682390 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557610dbea50 0x557610dc72e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:04:56.534 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:56.535+0000 7f6238428740 1 --2- 172.21.15.29:0/191682390 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557610dbe130 0x557610dbe510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:04:56.534 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:56.535+0000 7f6238428740 1 -- 172.21.15.29:0/191682390 >> 172.21.15.29:0/191682390 conn(0x557610db8a50 msgr2=0x557610db8e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:04:56.534 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:56.536+0000 7f6238428740 1 -- 172.21.15.29:0/191682390 shutdown_connections 2024-03-16T12:04:56.534 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:56.536+0000 7f6238428740 1 -- 172.21.15.29:0/191682390 wait complete. 2024-03-16T12:04:56.534 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:56.536+0000 7f6238428740 1 -- start start 2024-03-16T12:04:56.534 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:56.536+0000 7f6238428740 1 Processor -- start 2024-03-16T12:04:56.535 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:56.536+0000 7f6238428740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557610dbe130 0x557610dc4840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:04:56.535 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:56.536+0000 7f6238428740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557610dbea50 0x557610dc4d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:04:56.535 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:56.536+0000 7f6238428740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557610dc7820 0x557610dc52c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:04:56.535 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:56.536+0000 7f6238428740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x557610dca450 con 0x557610dbe130 2024-03-16T12:04:56.535 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:56.536+0000 7f6238428740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x557610cded70 con 0x557610dbea50 2024-03-16T12:04:56.535 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:56.536+0000 7f6238428740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x557610dca5d0 con 0x557610dc7820 2024-03-16T12:04:56.535 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:56.536+0000 7f6238428740 10 client.0 _ll_register_callbacks cb 0x557610a36a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T12:04:56.535 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:56.536+0000 7f622effd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557610dbe130 0x557610dc4840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:04:56.535 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:56.536+0000 7f622effd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557610dbe130 0x557610dc4840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:48832/0 (socket says 172.21.15.29:48832) 2024-03-16T12:04:56.535 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:56.536+0000 7f622effd640 1 -- 172.21.15.29:0/643082648 learned_addr learned my addr 172.21.15.29:0/643082648 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T12:04:56.535 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:56.536+0000 7f622f7fe640 1 --2- 172.21.15.29:0/643082648 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557610dc7820 0x557610dc52c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:04:56.535 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:56.536+0000 7f622e7fc640 1 --2- 172.21.15.29:0/643082648 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557610dbea50 0x557610dc4d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:04:56.535 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:56.537+0000 7f622effd640 1 -- 172.21.15.29:0/643082648 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557610dc7820 msgr2=0x557610dc52c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:04:56.535 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:56.537+0000 7f622effd640 1 --2- 172.21.15.29:0/643082648 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557610dc7820 0x557610dc52c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:04:56.535 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:56.537+0000 7f622effd640 1 -- 172.21.15.29:0/643082648 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557610dbea50 msgr2=0x557610dc4d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:04:56.536 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:56.537+0000 7f622effd640 1 --2- 172.21.15.29:0/643082648 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557610dbea50 0x557610dc4d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:04:56.536 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:56.537+0000 7f622effd640 1 -- 172.21.15.29:0/643082648 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557610a37210 con 0x557610dbe130 2024-03-16T12:04:56.536 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:56.537+0000 7f622effd640 1 --2- 172.21.15.29:0/643082648 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557610dbe130 0x557610dc4840 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f62200126d0 tx=0x7f6220012b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f440d5b9c81fc272 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:04:56.536 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:56.537+0000 7f6212ff5640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-16T12:04:56.536 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:56.537+0000 7f6212ff5640 1 -- 172.21.15.29:0/643082648 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f622000f9e0 con 0x557610dbe130 2024-03-16T12:04:56.536 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:56.537+0000 7f6212ff5640 1 -- 172.21.15.29:0/643082648 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(110..110 src has 1..110) ==== 8556+0+0 (secure 0 0 0) 0x7f622000fb80 con 0x557610dbe130 2024-03-16T12:04:56.536 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:56.537+0000 7f6238428740 1 -- 172.21.15.29:0/643082648 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f622409bda0 con 0x557610dbe130 2024-03-16T12:04:56.536 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:56.537+0000 7f6212ff5640 1 -- 172.21.15.29:0/643082648 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f622000fd80 con 0x557610dbe130 2024-03-16T12:04:56.536 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:56.537+0000 7f6238428740 10 client.16569 fetch_fsmap learned FSMap version 31 2024-03-16T12:04:56.536 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:56.537+0000 7f6238428740 1 -- 172.21.15.29:0/643082648 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x557610b03620 con 0x557610dbe130 2024-03-16T12:04:56.536 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:56.537+0000 7f6212ff5640 1 -- 172.21.15.29:0/643082648 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f622409bda0 con 0x557610dbe130 2024-03-16T12:04:56.536 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:56.537+0000 7f6212ff5640 1 -- 172.21.15.29:0/643082648 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 31) ==== 8346+0+0 (secure 0 0 0) 0x7f62200244b0 con 0x557610dbe130 2024-03-16T12:04:56.536 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:56.537+0000 7f6238428740 10 client.16569 fetch_fsmap finished waiting for FSMap version 31 2024-03-16T12:04:56.536 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:56.537+0000 7f6238428740 10 client.16569 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T12:04:56.536 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:56.537+0000 7f6238428740 20 client.16569 populate_metadata read hostname 'smithi029' 2024-03-16T12:04:56.536 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:56.537+0000 7f6238428740 1 --2- 172.21.15.29:0/643082648 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x557610dd7b10 0x557610dd9fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:04:56.537 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:56.537+0000 7f6238428740 4 client.16569 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T12:04:56.537 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:56.537+0000 7f6238428740 1 -- 172.21.15.29:0/643082648 --> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557610d2ff80 con 0x557610dd7b10 2024-03-16T12:04:56.537 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:56.538+0000 7f622e7fc640 1 --2- 172.21.15.29:0/643082648 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x557610dd7b10 0x557610dd9fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:04:56.537 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:56.538+0000 7f622e7fc640 1 --2- 172.21.15.29:0/643082648 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x557610dd7b10 0x557610dd9fa0 crc :-1 s=READY pgs=164 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1d28dade55cf7f61 server_cookie=58ad4a4f4f1f8b0a in_seq=0 out_seq=0 2024-03-16T12:04:56.537 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:56.538+0000 7f6212ff5640 10 client.16569 ms_handle_connect on v2:172.21.15.152:6838/1548731833 2024-03-16T12:04:56.539 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:56.545+0000 7f6212ff5640 1 -- 172.21.15.29:0/643082648 <== mds.0 v2:172.21.15.152:6838/1548731833 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x557610d2ff80 con 0x557610dd7b10 2024-03-16T12:04:56.539 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:56.545+0000 7f6212ff5640 10 client.16569 handle_command_reply: tid=0 2024-03-16T12:04:56.539 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:56.545+0000 7f6238428740 1 client.16569 shutdown 2024-03-16T12:04:56.540 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:56.545+0000 7f6238428740 1 -- 172.21.15.29:0/643082648 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557610dbe130 msgr2=0x557610dc4840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:04:56.540 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:56.545+0000 7f6238428740 1 --2- 172.21.15.29:0/643082648 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557610dbe130 0x557610dc4840 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f62200126d0 tx=0x7f6220012b90 comp rx=0 tx=0).stop 2024-03-16T12:04:56.540 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:56.545+0000 7f6238428740 1 -- 172.21.15.29:0/643082648 shutdown_connections 2024-03-16T12:04:56.540 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:56.545+0000 7f6238428740 1 --2- 172.21.15.29:0/643082648 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557610dc7820 0x557610dc52c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:04:56.540 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:56.545+0000 7f6238428740 1 --2- 172.21.15.29:0/643082648 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x557610dd7b10 0x557610dd9fa0 crc :-1 s=READY pgs=164 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:04:56.540 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:56.545+0000 7f6238428740 1 --2- 172.21.15.29:0/643082648 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557610dbea50 0x557610dc4d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:04:56.540 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:56.545+0000 7f6238428740 1 --2- 172.21.15.29:0/643082648 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557610dbe130 0x557610dc4840 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:04:56.540 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:56.545+0000 7f6212ff5640 0 client.16569 ms_handle_reset on v2:172.21.15.152:6838/1548731833 2024-03-16T12:04:56.540 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:56.545+0000 7f6238428740 1 -- 172.21.15.29:0/643082648 >> 172.21.15.29:0/643082648 conn(0x557610db8a50 msgr2=0x557610dba7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:04:56.540 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:56.545+0000 7f6238428740 1 -- 172.21.15.29:0/643082648 shutdown_connections 2024-03-16T12:04:56.540 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:56.545+0000 7f6238428740 1 -- 172.21.15.29:0/643082648 wait complete. 2024-03-16T12:04:56.540 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:56.545+0000 7f6238428740 20 client.16569 trim_cache size 0 max 16384 2024-03-16T12:04:56.540 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:04:56 smithi154 ceph-mon[36320]: pgmap v445: 97 pgs: 97 active+clean; 253 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 16 MiB/s wr, 337 op/s 2024-03-16T12:04:56.559 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:56.560+0000 7f6238428740 1 Processor -- start 2024-03-16T12:04:56.559 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:56.560+0000 7f6238428740 1 -- start start 2024-03-16T12:04:56.559 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:56.561+0000 7f6238428740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557610dbe130 0x557610dd3220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:04:56.559 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:56.561+0000 7f6238428740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557610dbea50 0x557610db7a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:04:56.559 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:56.561+0000 7f6238428740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557610dc7820 0x557610ddf8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:04:56.559 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:56.561+0000 7f6238428740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f621c0110d0 con 0x557610dbe130 2024-03-16T12:04:56.559 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:56.561+0000 7f6238428740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f621c00e940 con 0x557610dbea50 2024-03-16T12:04:56.559 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:56.561+0000 7f6238428740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f621c00c7b0 con 0x557610dc7820 2024-03-16T12:04:56.559 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:56.561+0000 7f622effd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557610dbe130 0x557610dd3220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:04:56.559 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:56.561+0000 7f622effd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557610dbe130 0x557610dd3220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:48834/0 (socket says 172.21.15.29:48834) 2024-03-16T12:04:56.560 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:56.561+0000 7f622effd640 1 -- 172.21.15.29:0/4142204969 learned_addr learned my addr 172.21.15.29:0/4142204969 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T12:04:56.560 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:56.561+0000 7f622f7fe640 1 --2- 172.21.15.29:0/4142204969 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557610dc7820 0x557610ddf8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:04:56.560 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:56.561+0000 7f622e7fc640 1 --2- 172.21.15.29:0/4142204969 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557610dbea50 0x557610db7a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:04:56.560 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:56.561+0000 7f622effd640 1 -- 172.21.15.29:0/4142204969 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557610dc7820 msgr2=0x557610ddf8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:04:56.560 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:56.561+0000 7f622effd640 1 --2- 172.21.15.29:0/4142204969 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557610dc7820 0x557610ddf8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:04:56.560 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:56.561+0000 7f622effd640 1 -- 172.21.15.29:0/4142204969 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557610dbea50 msgr2=0x557610db7a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:04:56.560 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:56.561+0000 7f622effd640 1 --2- 172.21.15.29:0/4142204969 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557610dbea50 0x557610db7a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:04:56.560 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:56.561+0000 7f622effd640 1 -- 172.21.15.29:0/4142204969 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557610dbe880 con 0x557610dbe130 2024-03-16T12:04:56.560 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:56.561+0000 7f622e7fc640 1 --2- 172.21.15.29:0/4142204969 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557610dbea50 0x557610db7a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T12:04:56.560 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:56.561+0000 7f622effd640 1 --2- 172.21.15.29:0/4142204969 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557610dbe130 0x557610dd3220 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f6220012420 tx=0x7f622000ff20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f2a453dcba6e1fe9 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:04:56.560 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:56.561+0000 7f6214ff9640 1 -- 172.21.15.29:0/4142204969 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f622000f9e0 con 0x557610dbe130 2024-03-16T12:04:56.560 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:56.561+0000 7f6214ff9640 1 -- 172.21.15.29:0/4142204969 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f622000fb80 con 0x557610dbe130 2024-03-16T12:04:56.560 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:56.561+0000 7f6214ff9640 1 -- 172.21.15.29:0/4142204969 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6220004680 con 0x557610dbe130 2024-03-16T12:04:56.560 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:56.562+0000 7f6238428740 1 -- 172.21.15.29:0/4142204969 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557610dbe130 msgr2=0x557610dd3220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:04:56.560 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:56.562+0000 7f6238428740 1 --2- 172.21.15.29:0/4142204969 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557610dbe130 0x557610dd3220 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f6220012420 tx=0x7f622000ff20 comp rx=0 tx=0).stop 2024-03-16T12:04:56.560 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:56.562+0000 7f6238428740 1 -- 172.21.15.29:0/4142204969 shutdown_connections 2024-03-16T12:04:56.561 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:56.562+0000 7f6238428740 1 --2- 172.21.15.29:0/4142204969 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557610dc7820 0x557610ddf8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:04:56.561 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:56.562+0000 7f6238428740 1 --2- 172.21.15.29:0/4142204969 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557610dbea50 0x557610db7a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:04:56.561 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:56.562+0000 7f6238428740 1 --2- 172.21.15.29:0/4142204969 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557610dbe130 0x557610dd3220 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:04:56.561 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:56.562+0000 7f6238428740 1 -- 172.21.15.29:0/4142204969 >> 172.21.15.29:0/4142204969 conn(0x557610db8a50 msgr2=0x557610db7320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:04:56.561 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:56.562+0000 7f6238428740 1 -- 172.21.15.29:0/4142204969 shutdown_connections 2024-03-16T12:04:56.561 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:56.562+0000 7f6238428740 1 -- 172.21.15.29:0/4142204969 wait complete. 2024-03-16T12:04:56.561 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:56.562+0000 7f6238428740 1 -- start start 2024-03-16T12:04:56.561 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:56.563+0000 7f6238428740 1 Processor -- start 2024-03-16T12:04:56.561 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:56.563+0000 7f6238428740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557610dbe130 0x557610dc1b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:04:56.561 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:56.563+0000 7f6238428740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557610dbea50 0x557610dc2070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:04:56.561 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:56.563+0000 7f6238428740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557610dc7820 0x557610e3cee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:04:56.561 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:56.563+0000 7f6238428740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f621c00ce00 con 0x557610dc7820 2024-03-16T12:04:56.561 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:56.563+0000 7f6238428740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f621c00e270 con 0x557610dbe130 2024-03-16T12:04:56.561 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:56.563+0000 7f6238428740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f621c00d4d0 con 0x557610dbea50 2024-03-16T12:04:56.561 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:56.563+0000 7f6238428740 10 client.0 _ll_register_callbacks cb 0x7f62200097a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T12:04:56.561 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:56.563+0000 7f622f7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557610dc7820 0x557610e3cee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:04:56.562 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:56.563+0000 7f622effd640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557610dbe130 0x557610dc1b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:04:56.562 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:56.563+0000 7f622f7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557610dc7820 0x557610e3cee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:48848/0 (socket says 172.21.15.29:48848) 2024-03-16T12:04:56.562 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:56.563+0000 7f622e7fc640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557610dbea50 0x557610dc2070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:04:56.562 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:56.563+0000 7f622f7fe640 1 -- 172.21.15.29:0/1953942482 learned_addr learned my addr 172.21.15.29:0/1953942482 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T12:04:56.562 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:56.563+0000 7f622f7fe640 1 -- 172.21.15.29:0/1953942482 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557610dbea50 msgr2=0x557610dc2070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:04:56.562 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:56.563+0000 7f622f7fe640 1 --2- 172.21.15.29:0/1953942482 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557610dbea50 0x557610dc2070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:04:56.562 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:56.563+0000 7f622f7fe640 1 -- 172.21.15.29:0/1953942482 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557610dbe130 msgr2=0x557610dc1b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:04:56.562 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:56.563+0000 7f622f7fe640 1 --2- 172.21.15.29:0/1953942482 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557610dbe130 0x557610dc1b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:04:56.562 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:56.563+0000 7f622f7fe640 1 -- 172.21.15.29:0/1953942482 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557610e3d450 con 0x557610dc7820 2024-03-16T12:04:56.562 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:56.563+0000 7f622f7fe640 1 --2- 172.21.15.29:0/1953942482 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557610dc7820 0x557610e3cee0 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f622802c700 tx=0x7f622802c0f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c47b30af5588f99d server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:04:56.562 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:56.563+0000 7f6212ff5640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-16T12:04:56.562 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:56.564+0000 7f6212ff5640 1 -- 172.21.15.29:0/1953942482 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6228039040 con 0x557610dc7820 2024-03-16T12:04:56.562 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:56.564+0000 7f6212ff5640 1 -- 172.21.15.29:0/1953942482 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(110..110 src has 1..110) ==== 8556+0+0 (secure 0 0 0) 0x7f6228004450 con 0x557610dc7820 2024-03-16T12:04:56.562 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:56.564+0000 7f6238428740 1 -- 172.21.15.29:0/1953942482 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f622000fb80 con 0x557610dc7820 2024-03-16T12:04:56.562 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:56.564+0000 7f6212ff5640 1 -- 172.21.15.29:0/1953942482 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f6228004650 con 0x557610dc7820 2024-03-16T12:04:56.562 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:56.564+0000 7f6238428740 10 client.16575 fetch_fsmap learned FSMap version 31 2024-03-16T12:04:56.563 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:56.564+0000 7f6238428740 1 -- 172.21.15.29:0/1953942482 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x557610e3e760 con 0x557610dc7820 2024-03-16T12:04:56.563 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:56.564+0000 7f6212ff5640 1 -- 172.21.15.29:0/1953942482 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f622000fb80 con 0x557610dc7820 2024-03-16T12:04:56.563 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:56.564+0000 7f6212ff5640 1 -- 172.21.15.29:0/1953942482 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 31) ==== 8346+0+0 (secure 0 0 0) 0x7f6228005500 con 0x557610dc7820 2024-03-16T12:04:56.563 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:56.564+0000 7f6238428740 10 client.16575 fetch_fsmap finished waiting for FSMap version 31 2024-03-16T12:04:56.563 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:56.564+0000 7f6238428740 10 client.16575 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T12:04:56.563 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:56.564+0000 7f6238428740 20 client.16575 populate_metadata read hostname 'smithi029' 2024-03-16T12:04:56.563 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:56.564+0000 7f6238428740 1 --2- 172.21.15.29:0/1953942482 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x557610e3efb0 0x557610e413d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:04:56.563 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:56.564+0000 7f6238428740 4 client.16575 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-16T12:04:56.563 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:56.564+0000 7f6238428740 1 -- 172.21.15.29:0/1953942482 --> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] -- command(tid 0: {"prefix": "flush journal"}) -- 0x557610d2a2d0 con 0x557610e3efb0 2024-03-16T12:04:56.563 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:56.564+0000 7f622effd640 1 --2- 172.21.15.29:0/1953942482 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x557610e3efb0 0x557610e413d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:04:56.563 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:56.565+0000 7f622effd640 1 --2- 172.21.15.29:0/1953942482 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x557610e3efb0 0x557610e413d0 crc :-1 s=READY pgs=165 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f2405ab57508e397 server_cookie=c167851fe110bc1 in_seq=0 out_seq=0 2024-03-16T12:04:56.563 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:56.565+0000 7f6212ff5640 10 client.16575 ms_handle_connect on v2:172.21.15.152:6838/1548731833 2024-03-16T12:04:56.582 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:04:56 smithi029 ceph-mon[31975]: pgmap v445: 97 pgs: 97 active+clean; 253 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 16 MiB/s wr, 337 op/s 2024-03-16T12:04:56.622 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:04:56 smithi152 ceph-mon[36291]: pgmap v445: 97 pgs: 97 active+clean; 253 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 16 MiB/s wr, 337 op/s 2024-03-16T12:04:57.122 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:57.128+0000 7f6212ff5640 1 -- 172.21.15.29:0/1953942482 <== mds.0 v2:172.21.15.152:6838/1548731833 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x557610d2a2d0 con 0x557610e3efb0 2024-03-16T12:04:57.123 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:57.128+0000 7f6212ff5640 10 client.16575 handle_command_reply: tid=0 2024-03-16T12:04:57.123 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:57.128+0000 7f6238428740 1 client.16575 shutdown 2024-03-16T12:04:57.123 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:57.128+0000 7f6238428740 1 -- 172.21.15.29:0/1953942482 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557610dc7820 msgr2=0x557610e3cee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:04:57.123 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:57.128+0000 7f6238428740 1 --2- 172.21.15.29:0/1953942482 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557610dc7820 0x557610e3cee0 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f622802c700 tx=0x7f622802c0f0 comp rx=0 tx=0).stop 2024-03-16T12:04:57.123 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:57.128+0000 7f6238428740 1 -- 172.21.15.29:0/1953942482 shutdown_connections 2024-03-16T12:04:57.123 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:57.128+0000 7f6238428740 1 --2- 172.21.15.29:0/1953942482 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557610dc7820 0x557610e3cee0 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:04:57.123 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:57.128+0000 7f6238428740 1 --2- 172.21.15.29:0/1953942482 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557610dbea50 0x557610dc2070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:04:57.123 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:57.128+0000 7f6238428740 1 --2- 172.21.15.29:0/1953942482 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x557610e3efb0 0x557610e413d0 crc :-1 s=READY pgs=165 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:04:57.123 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:57.128+0000 7f6238428740 1 --2- 172.21.15.29:0/1953942482 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557610dbe130 0x557610dc1b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:04:57.123 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:57.128+0000 7f6212ff5640 0 client.16575 ms_handle_reset on v2:172.21.15.152:6838/1548731833 2024-03-16T12:04:57.123 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:57.128+0000 7f6238428740 1 -- 172.21.15.29:0/1953942482 >> 172.21.15.29:0/1953942482 conn(0x557610db8a50 msgr2=0x557610db7320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:04:57.123 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:57.128+0000 7f6238428740 1 -- 172.21.15.29:0/1953942482 shutdown_connections 2024-03-16T12:04:57.124 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:57.128+0000 7f6238428740 1 -- 172.21.15.29:0/1953942482 wait complete. 2024-03-16T12:04:57.124 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:57.128+0000 7f6238428740 20 client.16575 trim_cache size 0 max 16384 2024-03-16T12:04:57.124 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:{ 2024-03-16T12:04:57.124 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "message": "", 2024-03-16T12:04:57.124 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "return_code": 0 2024-03-16T12:04:57.124 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:} 2024-03-16T12:04:57.125 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:57.129+0000 7f623575b640 1 -- 172.21.15.29:0/3689345658 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f62180a2a90 msgr2=0x7f62180a4f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:04:57.125 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:57.129+0000 7f623575b640 1 --2- 172.21.15.29:0/3689345658 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f62180a2a90 0x7f62180a4f50 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f6228005cc0 tx=0x7f6228005c50 comp rx=0 tx=0).stop 2024-03-16T12:04:57.125 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:57.129+0000 7f623575b640 1 -- 172.21.15.29:0/3689345658 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6230102150 msgr2=0x7f6230074020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:04:57.125 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:57.129+0000 7f623575b640 1 --2- 172.21.15.29:0/3689345658 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6230102150 0x7f6230074020 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f622400e9f0 tx=0x7f622400edf0 comp rx=0 tx=0).stop 2024-03-16T12:04:57.125 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:57.129+0000 7f623575b640 1 -- 172.21.15.29:0/3689345658 shutdown_connections 2024-03-16T12:04:57.125 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:57.129+0000 7f623575b640 1 --2- 172.21.15.29:0/3689345658 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f62180a2a90 0x7f62180a4f50 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:04:57.125 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:57.129+0000 7f623575b640 1 --2- 172.21.15.29:0/3689345658 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f62301077a0 0x7f6230074560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:04:57.125 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:57.129+0000 7f623575b640 1 --2- 172.21.15.29:0/3689345658 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6230102150 0x7f6230074020 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:04:57.125 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:57.129+0000 7f623575b640 1 --2- 172.21.15.29:0/3689345658 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6230101790 0x7f6230073ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:04:57.126 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:57.129+0000 7f623575b640 1 -- 172.21.15.29:0/3689345658 >> 172.21.15.29:0/3689345658 conn(0x7f62300fd2a0 msgr2=0x7f62300fd7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:04:57.126 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:57.129+0000 7f623575b640 1 -- 172.21.15.29:0/3689345658 shutdown_connections 2024-03-16T12:04:57.126 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:04:57.129+0000 7f623575b640 1 -- 172.21.15.29:0/3689345658 wait complete. 2024-03-16T12:04:57.183 INFO:tasks.workunit.client.0.smithi029.stdout: 10 12707 20.77 MB/sec warmup 41 sec latency 463.465 ms 2024-03-16T12:04:58.184 INFO:tasks.workunit.client.0.smithi029.stdout: 10 13152 21.11 MB/sec warmup 42 sec latency 247.643 ms 2024-03-16T12:04:58.540 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:04:58 smithi154 ceph-mon[36320]: pgmap v446: 97 pgs: 97 active+clean; 263 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 17 MiB/s wr, 425 op/s 2024-03-16T12:04:58.582 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:04:58 smithi029 ceph-mon[31975]: pgmap v446: 97 pgs: 97 active+clean; 263 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 17 MiB/s wr, 425 op/s 2024-03-16T12:04:58.622 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:04:58 smithi152 ceph-mon[36291]: pgmap v446: 97 pgs: 97 active+clean; 263 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 17 MiB/s wr, 425 op/s 2024-03-16T12:04:59.184 INFO:tasks.workunit.client.0.smithi029.stdout: 10 13471 21.28 MB/sec warmup 43 sec latency 209.303 ms 2024-03-16T12:05:00.184 INFO:tasks.workunit.client.0.smithi029.stdout: 10 13653 21.03 MB/sec warmup 44 sec latency 223.990 ms 2024-03-16T12:05:00.539 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:05:00 smithi154 ceph-mon[36320]: pgmap v447: 97 pgs: 97 active+clean; 263 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 1.6 MiB/s wr, 149 op/s 2024-03-16T12:05:00.582 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:05:00 smithi029 ceph-mon[31975]: pgmap v447: 97 pgs: 97 active+clean; 263 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 1.6 MiB/s wr, 149 op/s 2024-03-16T12:05:00.621 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:05:00 smithi152 ceph-mon[36291]: pgmap v447: 97 pgs: 97 active+clean; 263 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 1.6 MiB/s wr, 149 op/s 2024-03-16T12:05:01.184 INFO:tasks.workunit.client.0.smithi029.stdout: 10 13803 20.72 MB/sec warmup 45 sec latency 248.224 ms 2024-03-16T12:05:02.184 INFO:tasks.workunit.client.0.smithi029.stdout: 10 13930 20.33 MB/sec warmup 46 sec latency 338.671 ms 2024-03-16T12:05:02.540 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:05:02 smithi154 ceph-mon[36320]: pgmap v448: 97 pgs: 97 active+clean; 256 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 2.2 MiB/s wr, 378 op/s 2024-03-16T12:05:02.582 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:05:02 smithi029 ceph-mon[31975]: pgmap v448: 97 pgs: 97 active+clean; 256 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 2.2 MiB/s wr, 378 op/s 2024-03-16T12:05:02.622 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:05:02 smithi152 ceph-mon[36291]: pgmap v448: 97 pgs: 97 active+clean; 256 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 2.2 MiB/s wr, 378 op/s 2024-03-16T12:05:03.184 INFO:tasks.workunit.client.0.smithi029.stdout: 10 14077 19.95 MB/sec warmup 47 sec latency 189.878 ms 2024-03-16T12:05:04.184 INFO:tasks.workunit.client.0.smithi029.stdout: 10 14233 19.62 MB/sec warmup 48 sec latency 76.612 ms 2024-03-16T12:05:04.539 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:05:04 smithi154 ceph-mon[36320]: pgmap v449: 97 pgs: 97 active+clean; 256 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 1.7 MiB/s wr, 344 op/s 2024-03-16T12:05:04.582 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:05:04 smithi029 ceph-mon[31975]: pgmap v449: 97 pgs: 97 active+clean; 256 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 1.7 MiB/s wr, 344 op/s 2024-03-16T12:05:04.621 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:05:04 smithi152 ceph-mon[36291]: pgmap v449: 97 pgs: 97 active+clean; 256 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 1.7 MiB/s wr, 344 op/s 2024-03-16T12:05:05.184 INFO:tasks.workunit.client.0.smithi029.stdout: 10 14455 19.34 MB/sec warmup 49 sec latency 76.231 ms 2024-03-16T12:05:06.184 INFO:tasks.workunit.client.0.smithi029.stdout: 10 14697 19.24 MB/sec warmup 50 sec latency 236.069 ms 2024-03-16T12:05:06.539 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:05:06 smithi154 ceph-mon[36320]: pgmap v450: 97 pgs: 97 active+clean; 256 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 1.7 MiB/s wr, 342 op/s 2024-03-16T12:05:06.582 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:05:06 smithi029 ceph-mon[31975]: pgmap v450: 97 pgs: 97 active+clean; 256 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 1.7 MiB/s wr, 342 op/s 2024-03-16T12:05:06.621 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:05:06 smithi152 ceph-mon[36291]: pgmap v450: 97 pgs: 97 active+clean; 256 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 1.7 MiB/s wr, 342 op/s 2024-03-16T12:05:07.184 INFO:tasks.workunit.client.0.smithi029.stdout: 10 15190 19.31 MB/sec warmup 51 sec latency 464.231 ms 2024-03-16T12:05:08.184 INFO:tasks.workunit.client.0.smithi029.stdout: 10 15703 19.41 MB/sec warmup 52 sec latency 156.829 ms 2024-03-16T12:05:08.539 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:05:08 smithi154 ceph-mon[36320]: pgmap v451: 97 pgs: 97 active+clean; 272 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 3.3 MiB/s wr, 592 op/s 2024-03-16T12:05:08.582 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:05:08 smithi029 ceph-mon[31975]: pgmap v451: 97 pgs: 97 active+clean; 272 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 3.3 MiB/s wr, 592 op/s 2024-03-16T12:05:08.621 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:05:08 smithi152 ceph-mon[36291]: pgmap v451: 97 pgs: 97 active+clean; 272 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 3.3 MiB/s wr, 592 op/s 2024-03-16T12:05:09.184 INFO:tasks.workunit.client.0.smithi029.stdout: 10 16102 19.49 MB/sec warmup 53 sec latency 233.617 ms 2024-03-16T12:05:10.184 INFO:tasks.workunit.client.0.smithi029.stdout: 10 16514 19.76 MB/sec warmup 54 sec latency 250.055 ms 2024-03-16T12:05:10.224 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:10.228+0000 7fc0ff436640 1 -- 172.21.15.29:0/1717150409 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc0f81006a0 msgr2=0x7fc0f810e670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:05:10.224 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:10.228+0000 7fc0ff436640 1 --2- 172.21.15.29:0/1717150409 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc0f81006a0 0x7fc0f810e670 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7fc0f0009cd0 tx=0x7fc0f002c0f0 comp rx=0 tx=0).stop 2024-03-16T12:05:10.224 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:10.228+0000 7fc0ff436640 1 -- 172.21.15.29:0/1717150409 shutdown_connections 2024-03-16T12:05:10.224 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:10.228+0000 7fc0ff436640 1 --2- 172.21.15.29:0/1717150409 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc0f81006a0 0x7fc0f810e670 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:05:10.224 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:10.228+0000 7fc0ff436640 1 --2- 172.21.15.29:0/1717150409 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc0f80ffce0 0x7fc0f8100160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:05:10.224 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:10.228+0000 7fc0ff436640 1 --2- 172.21.15.29:0/1717150409 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc0f81067c0 0x7fc0f8106ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:05:10.224 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:10.228+0000 7fc0ff436640 1 -- 172.21.15.29:0/1717150409 >> 172.21.15.29:0/1717150409 conn(0x7fc0f8076c70 msgr2=0x7fc0f80fdff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:05:10.224 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:10.228+0000 7fc0ff436640 1 -- 172.21.15.29:0/1717150409 shutdown_connections 2024-03-16T12:05:10.224 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:10.228+0000 7fc0ff436640 1 -- 172.21.15.29:0/1717150409 wait complete. 2024-03-16T12:05:10.224 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:10.229+0000 7fc0ff436640 1 Processor -- start 2024-03-16T12:05:10.224 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:10.229+0000 7fc0ff436640 1 -- start start 2024-03-16T12:05:10.225 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:10.229+0000 7fc0ff436640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc0f80ffce0 0x7fc0f81980b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:05:10.225 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:10.229+0000 7fc0ff436640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc0f81006a0 0x7fc0f81985f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:05:10.225 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:10.229+0000 7fc0ff436640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc0f81067c0 0x7fc0f819c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:05:10.225 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:10.229+0000 7fc0ff436640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fc0f8111c90 con 0x7fc0f81006a0 2024-03-16T12:05:10.225 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:10.229+0000 7fc0ff436640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fc0f8111b10 con 0x7fc0f81067c0 2024-03-16T12:05:10.225 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:10.229+0000 7fc0ff436640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fc0f8111e10 con 0x7fc0f80ffce0 2024-03-16T12:05:10.225 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:10.229+0000 7fc0fc9d1640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc0f81006a0 0x7fc0f81985f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:05:10.225 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:10.229+0000 7fc0fc9d1640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc0f81006a0 0x7fc0f81985f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33004/0 (socket says 172.21.15.29:33004) 2024-03-16T12:05:10.225 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:10.229+0000 7fc0fc9d1640 1 -- 172.21.15.29:0/1012116722 learned_addr learned my addr 172.21.15.29:0/1012116722 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T12:05:10.225 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:10.229+0000 7fc0fd1d2640 1 --2- 172.21.15.29:0/1012116722 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc0f80ffce0 0x7fc0f81980b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:05:10.225 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:10.229+0000 7fc0fd9d3640 1 --2- 172.21.15.29:0/1012116722 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc0f81067c0 0x7fc0f819c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:05:10.225 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:10.229+0000 7fc0fc9d1640 1 -- 172.21.15.29:0/1012116722 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc0f80ffce0 msgr2=0x7fc0f81980b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:05:10.225 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:10.229+0000 7fc0fc9d1640 1 --2- 172.21.15.29:0/1012116722 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc0f80ffce0 0x7fc0f81980b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:05:10.226 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:10.229+0000 7fc0fc9d1640 1 -- 172.21.15.29:0/1012116722 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc0f81067c0 msgr2=0x7fc0f819c920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:05:10.226 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:10.229+0000 7fc0fc9d1640 1 --2- 172.21.15.29:0/1012116722 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc0f81067c0 0x7fc0f819c920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:05:10.226 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:10.229+0000 7fc0fc9d1640 1 -- 172.21.15.29:0/1012116722 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0f819d050 con 0x7fc0f81006a0 2024-03-16T12:05:10.226 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:10.230+0000 7fc0fc9d1640 1 --2- 172.21.15.29:0/1012116722 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc0f81006a0 0x7fc0f81985f0 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7fc0e800d950 tx=0x7fc0e800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:05:10.226 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:10.230+0000 7fc0ee7fc640 1 -- 172.21.15.29:0/1012116722 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0e8017030 con 0x7fc0f81006a0 2024-03-16T12:05:10.226 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:10.230+0000 7fc0ff436640 1 -- 172.21.15.29:0/1012116722 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc0f819d340 con 0x7fc0f81006a0 2024-03-16T12:05:10.226 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:10.230+0000 7fc0ee7fc640 1 -- 172.21.15.29:0/1012116722 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc0e8004020 con 0x7fc0f81006a0 2024-03-16T12:05:10.226 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:10.230+0000 7fc0ee7fc640 1 -- 172.21.15.29:0/1012116722 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0e80047a0 con 0x7fc0f81006a0 2024-03-16T12:05:10.226 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:10.230+0000 7fc0ff436640 1 -- 172.21.15.29:0/1012116722 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc0f81a4bd0 con 0x7fc0f81006a0 2024-03-16T12:05:10.232 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:10.232+0000 7fc0ee7fc640 1 -- 172.21.15.29:0/1012116722 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7fc0e80041d0 con 0x7fc0f81006a0 2024-03-16T12:05:10.232 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:10.233+0000 7fc102026740 1 Processor -- start 2024-03-16T12:05:10.232 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:10.233+0000 7fc102026740 1 -- start start 2024-03-16T12:05:10.232 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:10.233+0000 7fc0ee7fc640 1 --2- 172.21.15.29:0/1012116722 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7fc0e40a2a10 0x7fc0e40a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:05:10.232 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:10.233+0000 7fc102026740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557e2ab94130 0x557e2ab94510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:05:10.232 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:10.233+0000 7fc102026740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557e2ab94a50 0x557e2ab9d2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:05:10.232 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:10.233+0000 7fc0fd9d3640 1 --2- 172.21.15.29:0/1012116722 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7fc0e40a2a10 0x7fc0e40a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:05:10.232 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:10.233+0000 7fc102026740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557e2ab9d820 0x557e2ab9fc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:05:10.232 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:10.233+0000 7fc102026740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x557e2aaf8a40 con 0x557e2ab94a50 2024-03-16T12:05:10.232 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:10.233+0000 7fc102026740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x557e2aaa1920 con 0x557e2ab94130 2024-03-16T12:05:10.233 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:10.233+0000 7fc102026740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x557e2aaf4ae0 con 0x557e2ab9d820 2024-03-16T12:05:10.233 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:10.233+0000 7fc0ee7fc640 1 -- 172.21.15.29:0/1012116722 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 8556+0+0 (secure 0 0 0) 0x7fc0e80ce880 con 0x7fc0f81006a0 2024-03-16T12:05:10.233 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:10.233+0000 7fc0fc9d1640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557e2ab94a50 0x557e2ab9d2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:05:10.233 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:10.233+0000 7fc0fd1d2640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557e2ab94130 0x557e2ab94510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:05:10.233 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:10.233+0000 7fc0fc9d1640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557e2ab94a50 0x557e2ab9d2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33018/0 (socket says 172.21.15.29:33018) 2024-03-16T12:05:10.233 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:10.233+0000 7fc0fd1d2640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557e2ab94130 0x557e2ab94510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.29:36294/0 (socket says 172.21.15.29:36294) 2024-03-16T12:05:10.233 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:10.233+0000 7fc0fc9d1640 1 -- 172.21.15.29:0/1124627012 learned_addr learned my addr 172.21.15.29:0/1124627012 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T12:05:10.233 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:10.233+0000 7fc0fd9d3640 1 --2- 172.21.15.29:0/1124627012 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557e2ab9d820 0x557e2ab9fc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:05:10.233 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:10.233+0000 7fc0fc9d1640 1 -- 172.21.15.29:0/1124627012 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557e2ab9d820 msgr2=0x557e2ab9fc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:05:10.233 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:10.233+0000 7fc0fc9d1640 1 --2- 172.21.15.29:0/1124627012 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557e2ab9d820 0x557e2ab9fc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:05:10.233 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:10.233+0000 7fc0fc9d1640 1 -- 172.21.15.29:0/1124627012 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557e2ab94130 msgr2=0x557e2ab94510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:05:10.233 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:10.233+0000 7fc0fc9d1640 1 --2- 172.21.15.29:0/1124627012 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557e2ab94130 0x557e2ab94510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:05:10.234 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:10.233+0000 7fc0fc9d1640 1 -- 172.21.15.29:0/1124627012 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557e2aab9460 con 0x557e2ab94a50 2024-03-16T12:05:10.234 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:10.233+0000 7fc0fd1d2640 1 --2- 172.21.15.29:0/1124627012 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557e2ab94130 0x557e2ab94510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:05:10.234 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:10.233+0000 7fc0fd9d3640 1 --2- 172.21.15.29:0/1012116722 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7fc0e40a2a10 0x7fc0e40a4ed0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fc0f0005cc0 tx=0x7fc0f002ce60 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:05:10.234 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:10.234+0000 7fc0fc9d1640 1 --2- 172.21.15.29:0/1124627012 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557e2ab94a50 0x557e2ab9d2e0 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7fc0e80a30d0 tx=0x7fc0e809a880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a46c83cb0ee29c6 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:05:10.234 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:10.234+0000 7fc0e37fe640 1 -- 172.21.15.29:0/1124627012 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0e809efb0 con 0x557e2ab94a50 2024-03-16T12:05:10.234 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:10.234+0000 7fc0e37fe640 1 -- 172.21.15.29:0/1124627012 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc0e80a3600 con 0x557e2ab94a50 2024-03-16T12:05:10.234 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:10.234+0000 7fc0e37fe640 1 -- 172.21.15.29:0/1124627012 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0e80a3370 con 0x557e2ab94a50 2024-03-16T12:05:10.234 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:10.235+0000 7fc102026740 1 -- 172.21.15.29:0/1124627012 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557e2ab94a50 msgr2=0x557e2ab9d2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:05:10.234 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:10.235+0000 7fc102026740 1 --2- 172.21.15.29:0/1124627012 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557e2ab94a50 0x557e2ab9d2e0 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7fc0e80a30d0 tx=0x7fc0e809a880 comp rx=0 tx=0).stop 2024-03-16T12:05:10.234 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:10.235+0000 7fc102026740 1 -- 172.21.15.29:0/1124627012 shutdown_connections 2024-03-16T12:05:10.234 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:10.235+0000 7fc102026740 1 --2- 172.21.15.29:0/1124627012 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557e2ab9d820 0x557e2ab9fc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:05:10.234 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:10.235+0000 7fc102026740 1 --2- 172.21.15.29:0/1124627012 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557e2ab94a50 0x557e2ab9d2e0 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:05:10.234 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:10.235+0000 7fc102026740 1 --2- 172.21.15.29:0/1124627012 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557e2ab94130 0x557e2ab94510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:05:10.235 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:10.235+0000 7fc102026740 1 -- 172.21.15.29:0/1124627012 >> 172.21.15.29:0/1124627012 conn(0x557e2ab8ea50 msgr2=0x557e2ab8ee30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:05:10.235 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:10.235+0000 7fc102026740 1 -- 172.21.15.29:0/1124627012 shutdown_connections 2024-03-16T12:05:10.235 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:10.235+0000 7fc102026740 1 -- 172.21.15.29:0/1124627012 wait complete. 2024-03-16T12:05:10.235 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:10.235+0000 7fc102026740 1 -- start start 2024-03-16T12:05:10.235 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:10.235+0000 7fc102026740 1 Processor -- start 2024-03-16T12:05:10.235 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:10.235+0000 7fc102026740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557e2ab94130 0x557e2ab9a890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:05:10.235 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:10.236+0000 7fc102026740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557e2ab94a50 0x557e2ab9add0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:05:10.235 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:10.236+0000 7fc102026740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557e2ab9d820 0x557e2ab9b310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:05:10.235 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:10.236+0000 7fc102026740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x557e2aba4800 con 0x557e2ab94130 2024-03-16T12:05:10.235 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:10.236+0000 7fc102026740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x557e2aab4d70 con 0x557e2ab94a50 2024-03-16T12:05:10.235 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:10.236+0000 7fc102026740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x557e2aba4980 con 0x557e2ab9d820 2024-03-16T12:05:10.235 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:10.236+0000 7fc102026740 10 client.0 _ll_register_callbacks cb 0x557e2a80ca00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T12:05:10.235 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:10.236+0000 7fc0fd1d2640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557e2ab94130 0x557e2ab9a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:05:10.235 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:10.236+0000 7fc0fd1d2640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557e2ab94130 0x557e2ab9a890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33026/0 (socket says 172.21.15.29:33026) 2024-03-16T12:05:10.235 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:10.236+0000 7fc0fd1d2640 1 -- 172.21.15.29:0/1371181836 learned_addr learned my addr 172.21.15.29:0/1371181836 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T12:05:10.236 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:10.236+0000 7fc0fc9d1640 1 --2- 172.21.15.29:0/1371181836 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557e2ab94a50 0x557e2ab9add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:05:10.236 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:10.236+0000 7fc0fd9d3640 1 --2- 172.21.15.29:0/1371181836 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557e2ab9d820 0x557e2ab9b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:05:10.236 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:10.236+0000 7fc0fd1d2640 1 -- 172.21.15.29:0/1371181836 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557e2ab9d820 msgr2=0x557e2ab9b310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:05:10.236 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:10.236+0000 7fc0fd1d2640 1 --2- 172.21.15.29:0/1371181836 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557e2ab9d820 0x557e2ab9b310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:05:10.236 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:10.236+0000 7fc0fd1d2640 1 -- 172.21.15.29:0/1371181836 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557e2ab94a50 msgr2=0x557e2ab9add0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:05:10.236 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:10.236+0000 7fc0fd1d2640 1 --2- 172.21.15.29:0/1371181836 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557e2ab94a50 0x557e2ab9add0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:05:10.236 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:10.236+0000 7fc0fd1d2640 1 -- 172.21.15.29:0/1371181836 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557e2a80d210 con 0x557e2ab94130 2024-03-16T12:05:10.236 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:10.236+0000 7fc0fd1d2640 1 --2- 172.21.15.29:0/1371181836 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557e2ab94130 0x557e2ab9a890 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7fc0f4004650 tx=0x7fc0f400bec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=383419786818c348 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:05:10.236 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:10.236+0000 7fc0e0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-16T12:05:10.236 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:10.236+0000 7fc0e0ff9640 1 -- 172.21.15.29:0/1371181836 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0f4003e80 con 0x557e2ab94130 2024-03-16T12:05:10.236 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:10.236+0000 7fc0e0ff9640 1 -- 172.21.15.29:0/1371181836 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(110..110 src has 1..110) ==== 8556+0+0 (secure 0 0 0) 0x7fc0f4011440 con 0x557e2ab94130 2024-03-16T12:05:10.236 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:10.236+0000 7fc102026740 1 -- 172.21.15.29:0/1371181836 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc0e80a3600 con 0x557e2ab94130 2024-03-16T12:05:10.236 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:10.236+0000 7fc0e0ff9640 1 -- 172.21.15.29:0/1371181836 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fc0f4011640 con 0x557e2ab94130 2024-03-16T12:05:10.236 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:10.236+0000 7fc102026740 10 client.16587 fetch_fsmap learned FSMap version 31 2024-03-16T12:05:10.236 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:10.236+0000 7fc102026740 1 -- 172.21.15.29:0/1371181836 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x557e2a8d9620 con 0x557e2ab94130 2024-03-16T12:05:10.236 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:10.237+0000 7fc0e0ff9640 1 -- 172.21.15.29:0/1371181836 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc0e80a3600 con 0x557e2ab94130 2024-03-16T12:05:10.236 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:10.237+0000 7fc0e0ff9640 1 -- 172.21.15.29:0/1371181836 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 31) ==== 8346+0+0 (secure 0 0 0) 0x7fc0f40239c0 con 0x557e2ab94130 2024-03-16T12:05:10.237 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:10.237+0000 7fc102026740 10 client.16587 fetch_fsmap finished waiting for FSMap version 31 2024-03-16T12:05:10.237 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:10.237+0000 7fc102026740 10 client.16587 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T12:05:10.237 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:10.237+0000 7fc102026740 20 client.16587 populate_metadata read hostname 'smithi029' 2024-03-16T12:05:10.237 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:10.237+0000 7fc102026740 1 --2- 172.21.15.29:0/1371181836 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x557e2abb1ea0 0x557e2abb42d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:05:10.237 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:10.237+0000 7fc102026740 4 client.16587 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T12:05:10.237 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:10.237+0000 7fc102026740 1 -- 172.21.15.29:0/1371181836 --> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557e2ab05f80 con 0x557e2abb1ea0 2024-03-16T12:05:10.237 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:10.237+0000 7fc0fc9d1640 1 --2- 172.21.15.29:0/1371181836 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x557e2abb1ea0 0x557e2abb42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:05:10.237 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:10.237+0000 7fc0fc9d1640 1 --2- 172.21.15.29:0/1371181836 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x557e2abb1ea0 0x557e2abb42d0 crc :-1 s=READY pgs=166 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e58c1fe08ed4d5fd server_cookie=6ba7353c2e94ce03 in_seq=0 out_seq=0 2024-03-16T12:05:10.237 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:10.237+0000 7fc0e0ff9640 10 client.16587 ms_handle_connect on v2:172.21.15.152:6838/1548731833 2024-03-16T12:05:10.238 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:10.240+0000 7fc0e0ff9640 1 -- 172.21.15.29:0/1371181836 <== mds.0 v2:172.21.15.152:6838/1548731833 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x557e2ab05f80 con 0x557e2abb1ea0 2024-03-16T12:05:10.238 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:10.240+0000 7fc0e0ff9640 10 client.16587 handle_command_reply: tid=0 2024-03-16T12:05:10.238 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:10.240+0000 7fc102026740 1 client.16587 shutdown 2024-03-16T12:05:10.239 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:10.241+0000 7fc102026740 1 -- 172.21.15.29:0/1371181836 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557e2ab94130 msgr2=0x557e2ab9a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:05:10.239 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:10.241+0000 7fc102026740 1 --2- 172.21.15.29:0/1371181836 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557e2ab94130 0x557e2ab9a890 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7fc0f4004650 tx=0x7fc0f400bec0 comp rx=0 tx=0).stop 2024-03-16T12:05:10.239 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:10.241+0000 7fc102026740 1 -- 172.21.15.29:0/1371181836 shutdown_connections 2024-03-16T12:05:10.239 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:10.241+0000 7fc102026740 1 --2- 172.21.15.29:0/1371181836 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557e2ab9d820 0x557e2ab9b310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:05:10.239 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:10.241+0000 7fc102026740 1 --2- 172.21.15.29:0/1371181836 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x557e2abb1ea0 0x557e2abb42d0 crc :-1 s=READY pgs=166 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:05:10.239 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:10.241+0000 7fc0e0ff9640 0 client.16587 ms_handle_reset on v2:172.21.15.152:6838/1548731833 2024-03-16T12:05:10.239 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:10.241+0000 7fc102026740 1 --2- 172.21.15.29:0/1371181836 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557e2ab94a50 0x557e2ab9add0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:05:10.239 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:10.241+0000 7fc102026740 1 --2- 172.21.15.29:0/1371181836 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557e2ab94130 0x557e2ab9a890 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:05:10.239 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:10.241+0000 7fc102026740 1 -- 172.21.15.29:0/1371181836 >> 172.21.15.29:0/1371181836 conn(0x557e2ab8ea50 msgr2=0x557e2ab907a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:05:10.239 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:10.241+0000 7fc102026740 1 -- 172.21.15.29:0/1371181836 shutdown_connections 2024-03-16T12:05:10.239 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:10.241+0000 7fc102026740 1 -- 172.21.15.29:0/1371181836 wait complete. 2024-03-16T12:05:10.239 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:10.241+0000 7fc102026740 20 client.16587 trim_cache size 0 max 16384 2024-03-16T12:05:10.253 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:10.257+0000 7fc102026740 1 Processor -- start 2024-03-16T12:05:10.253 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:10.258+0000 7fc102026740 1 -- start start 2024-03-16T12:05:10.253 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:10.258+0000 7fc102026740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557e2ab94130 0x557e2abad5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:05:10.253 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:10.258+0000 7fc102026740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557e2ab94a50 0x557e2abb9be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:05:10.253 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:10.258+0000 7fc102026740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557e2ab9d820 0x557e2ab8d320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:05:10.253 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:10.258+0000 7fc102026740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fc0f81adb70 con 0x557e2ab9d820 2024-03-16T12:05:10.253 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:10.258+0000 7fc102026740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fc0f81ab3e0 con 0x557e2ab94a50 2024-03-16T12:05:10.253 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:10.258+0000 7fc102026740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fc0f81a9250 con 0x557e2ab94130 2024-03-16T12:05:10.253 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:10.258+0000 7fc0fd9d3640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557e2ab9d820 0x557e2ab8d320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:05:10.253 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:10.258+0000 7fc0fd1d2640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557e2ab94130 0x557e2abad5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:05:10.254 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:10.258+0000 7fc0fd9d3640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557e2ab9d820 0x557e2ab8d320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33038/0 (socket says 172.21.15.29:33038) 2024-03-16T12:05:10.254 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:10.258+0000 7fc0fd9d3640 1 -- 172.21.15.29:0/1480732834 learned_addr learned my addr 172.21.15.29:0/1480732834 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T12:05:10.254 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:10.258+0000 7fc0fc9d1640 1 --2- 172.21.15.29:0/1480732834 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557e2ab94a50 0x557e2abb9be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:05:10.254 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:10.258+0000 7fc0fd1d2640 1 -- 172.21.15.29:0/1480732834 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557e2ab94a50 msgr2=0x557e2abb9be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:05:10.254 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:10.258+0000 7fc0fd1d2640 1 --2- 172.21.15.29:0/1480732834 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557e2ab94a50 0x557e2abb9be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:05:10.254 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:10.258+0000 7fc0fd1d2640 1 -- 172.21.15.29:0/1480732834 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557e2ab9d820 msgr2=0x557e2ab8d320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:05:10.254 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:10.258+0000 7fc0fd1d2640 1 --2- 172.21.15.29:0/1480732834 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557e2ab9d820 0x557e2ab8d320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:05:10.254 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:10.258+0000 7fc0fd1d2640 1 -- 172.21.15.29:0/1480732834 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557e2abadae0 con 0x557e2ab94130 2024-03-16T12:05:10.254 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:10.259+0000 7fc0fd1d2640 1 --2- 172.21.15.29:0/1480732834 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557e2ab94130 0x557e2abad5a0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fc0f4023eb0 tx=0x7fc0f40115d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=758cc5fd040372f4 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:05:10.254 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:10.259+0000 7fc0e2ffd640 1 -- 172.21.15.29:0/1480732834 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0f4003f40 con 0x557e2ab94130 2024-03-16T12:05:10.254 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:10.259+0000 7fc0e2ffd640 1 -- 172.21.15.29:0/1480732834 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc0f4005510 con 0x557e2ab94130 2024-03-16T12:05:10.254 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:10.259+0000 7fc0e2ffd640 1 -- 172.21.15.29:0/1480732834 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0f40048e0 con 0x557e2ab94130 2024-03-16T12:05:10.256 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:10.260+0000 7fc102026740 1 -- 172.21.15.29:0/1480732834 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557e2ab94130 msgr2=0x557e2abad5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:05:10.256 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:10.260+0000 7fc102026740 1 --2- 172.21.15.29:0/1480732834 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557e2ab94130 0x557e2abad5a0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fc0f4023eb0 tx=0x7fc0f40115d0 comp rx=0 tx=0).stop 2024-03-16T12:05:10.256 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:10.260+0000 7fc102026740 1 -- 172.21.15.29:0/1480732834 shutdown_connections 2024-03-16T12:05:10.256 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:10.260+0000 7fc102026740 1 --2- 172.21.15.29:0/1480732834 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557e2ab9d820 0x557e2ab8d320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:05:10.257 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:10.260+0000 7fc102026740 1 --2- 172.21.15.29:0/1480732834 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557e2ab94a50 0x557e2abb9be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:05:10.257 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:10.260+0000 7fc102026740 1 --2- 172.21.15.29:0/1480732834 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557e2ab94130 0x557e2abad5a0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:05:10.257 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:10.260+0000 7fc102026740 1 -- 172.21.15.29:0/1480732834 >> 172.21.15.29:0/1480732834 conn(0x557e2ab8ea50 msgr2=0x557e2ab950e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:05:10.257 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:10.260+0000 7fc102026740 1 -- 172.21.15.29:0/1480732834 shutdown_connections 2024-03-16T12:05:10.257 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:10.260+0000 7fc102026740 1 -- 172.21.15.29:0/1480732834 wait complete. 2024-03-16T12:05:10.257 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:10.260+0000 7fc102026740 1 -- start start 2024-03-16T12:05:10.257 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:10.260+0000 7fc102026740 1 Processor -- start 2024-03-16T12:05:10.257 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:10.261+0000 7fc102026740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557e2ab94130 0x557e2aba5760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:05:10.257 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:10.261+0000 7fc102026740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557e2ab94a50 0x557e2aba5ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:05:10.257 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:10.261+0000 7fc102026740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557e2ab9d820 0x557e2aba61e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:05:10.257 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:10.261+0000 7fc102026740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fc0f81a98a0 con 0x557e2ab94a50 2024-03-16T12:05:10.257 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:10.261+0000 7fc102026740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fc0f81aad10 con 0x557e2ab9d820 2024-03-16T12:05:10.257 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:10.261+0000 7fc102026740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fc0f81a9f70 con 0x557e2ab94130 2024-03-16T12:05:10.257 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:10.261+0000 7fc0fd1d2640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557e2ab94130 0x557e2aba5760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:05:10.257 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:10.261+0000 7fc0fc9d1640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557e2ab94a50 0x557e2aba5ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:05:10.258 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:10.261+0000 7fc0fd1d2640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557e2ab94130 0x557e2aba5760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.29:56894/0 (socket says 172.21.15.29:56894) 2024-03-16T12:05:10.258 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:10.261+0000 7fc102026740 10 client.0 _ll_register_callbacks cb 0x7fc0f4004160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T12:05:10.258 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:10.261+0000 7fc0fd1d2640 1 -- 172.21.15.29:0/1257787019 learned_addr learned my addr 172.21.15.29:0/1257787019 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T12:05:10.258 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:10.261+0000 7fc0fd9d3640 1 --2- 172.21.15.29:0/1257787019 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557e2ab9d820 0x557e2aba61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:05:10.258 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:10.261+0000 7fc0fd9d3640 1 -- 172.21.15.29:0/1257787019 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557e2ab94130 msgr2=0x557e2aba5760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:05:10.258 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:10.261+0000 7fc0fd9d3640 1 --2- 172.21.15.29:0/1257787019 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557e2ab94130 0x557e2aba5760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:05:10.258 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:10.261+0000 7fc0fd9d3640 1 -- 172.21.15.29:0/1257787019 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557e2ab94a50 msgr2=0x557e2aba5ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:05:10.258 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:10.261+0000 7fc0fd9d3640 1 --2- 172.21.15.29:0/1257787019 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557e2ab94a50 0x557e2aba5ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:05:10.258 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:10.261+0000 7fc0fd9d3640 1 -- 172.21.15.29:0/1257787019 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557e2ac17460 con 0x557e2ab9d820 2024-03-16T12:05:10.258 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:10.262+0000 7fc0fd9d3640 1 --2- 172.21.15.29:0/1257787019 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557e2ab9d820 0x557e2aba61e0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fc0f0032380 tx=0x7fc0f002c0f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3f0a2c026d76c42f server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:05:10.258 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:10.262+0000 7fc0e0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.152:3300/0 2024-03-16T12:05:10.261 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:10.267+0000 7fc0e0ff9640 1 -- 172.21.15.29:0/1257787019 <== mon.2 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0f0039040 con 0x557e2ab9d820 2024-03-16T12:05:10.261 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:10.267+0000 7fc0e0ff9640 1 -- 172.21.15.29:0/1257787019 <== mon.2 v2:172.21.15.152:3300/0 2 ==== osd_map(110..110 src has 1..110) ==== 8556+0+0 (secure 0 0 0) 0x7fc0f0004040 con 0x557e2ab9d820 2024-03-16T12:05:10.261 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:10.267+0000 7fc102026740 1 -- 172.21.15.29:0/1257787019 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc0f4005510 con 0x557e2ab9d820 2024-03-16T12:05:10.261 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:10.267+0000 7fc0e0ff9640 1 -- 172.21.15.29:0/1257787019 <== mon.2 v2:172.21.15.152:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fc0f005e050 con 0x557e2ab9d820 2024-03-16T12:05:10.261 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:10.267+0000 7fc102026740 10 client.26393 fetch_fsmap learned FSMap version 31 2024-03-16T12:05:10.262 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:10.267+0000 7fc102026740 1 -- 172.21.15.29:0/1257787019 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x557e2ac18750 con 0x557e2ab9d820 2024-03-16T12:05:10.262 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:10.267+0000 7fc0e0ff9640 1 -- 172.21.15.29:0/1257787019 <== mon.2 v2:172.21.15.152:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc0f4005510 con 0x557e2ab9d820 2024-03-16T12:05:10.262 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:10.267+0000 7fc0e0ff9640 1 -- 172.21.15.29:0/1257787019 <== mon.2 v2:172.21.15.152:3300/0 5 ==== fsmap(e 31) ==== 8346+0+0 (secure 0 0 0) 0x7fc0f004fa10 con 0x557e2ab9d820 2024-03-16T12:05:10.262 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:10.267+0000 7fc102026740 10 client.26393 fetch_fsmap finished waiting for FSMap version 31 2024-03-16T12:05:10.262 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:10.267+0000 7fc102026740 10 client.26393 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T12:05:10.262 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:10.267+0000 7fc102026740 20 client.26393 populate_metadata read hostname 'smithi029' 2024-03-16T12:05:10.262 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:10.267+0000 7fc102026740 1 --2- 172.21.15.29:0/1257787019 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x557e2abb1ea0 0x557e2ac1af10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:05:10.262 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:10.267+0000 7fc102026740 4 client.26393 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-16T12:05:10.262 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:10.267+0000 7fc102026740 1 -- 172.21.15.29:0/1257787019 --> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] -- command(tid 0: {"prefix": "flush journal"}) -- 0x557e2ab002d0 con 0x557e2abb1ea0 2024-03-16T12:05:10.262 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:10.268+0000 7fc0fd1d2640 1 --2- 172.21.15.29:0/1257787019 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x557e2abb1ea0 0x557e2ac1af10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:05:10.263 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:10.268+0000 7fc0fd1d2640 1 --2- 172.21.15.29:0/1257787019 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x557e2abb1ea0 0x557e2ac1af10 crc :-1 s=READY pgs=167 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6650596db0fe9358 server_cookie=e987825058de5ccd in_seq=0 out_seq=0 2024-03-16T12:05:10.263 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:10.268+0000 7fc0e0ff9640 10 client.26393 ms_handle_connect on v2:172.21.15.152:6838/1548731833 2024-03-16T12:05:10.540 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:05:10 smithi154 ceph-mon[36320]: pgmap v452: 97 pgs: 97 active+clean; 272 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 2.3 MiB/s wr, 504 op/s 2024-03-16T12:05:10.582 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:05:10 smithi029 ceph-mon[31975]: pgmap v452: 97 pgs: 97 active+clean; 272 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 2.3 MiB/s wr, 504 op/s 2024-03-16T12:05:10.621 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:05:10 smithi152 ceph-mon[36291]: pgmap v452: 97 pgs: 97 active+clean; 272 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 2.3 MiB/s wr, 504 op/s 2024-03-16T12:05:10.768 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:10.773+0000 7fc0e0ff9640 1 -- 172.21.15.29:0/1257787019 <== mds.0 v2:172.21.15.152:6838/1548731833 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x557e2ab002d0 con 0x557e2abb1ea0 2024-03-16T12:05:10.768 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:10.773+0000 7fc0e0ff9640 10 client.26393 handle_command_reply: tid=0 2024-03-16T12:05:10.768 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:10.773+0000 7fc102026740 1 client.26393 shutdown 2024-03-16T12:05:10.768 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:10.773+0000 7fc102026740 1 -- 172.21.15.29:0/1257787019 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557e2ab9d820 msgr2=0x557e2aba61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:05:10.768 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:10.773+0000 7fc102026740 1 --2- 172.21.15.29:0/1257787019 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557e2ab9d820 0x557e2aba61e0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fc0f0032380 tx=0x7fc0f002c0f0 comp rx=0 tx=0).stop 2024-03-16T12:05:10.768 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:10.773+0000 7fc102026740 1 -- 172.21.15.29:0/1257787019 shutdown_connections 2024-03-16T12:05:10.768 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:10.773+0000 7fc102026740 1 --2- 172.21.15.29:0/1257787019 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x557e2abb1ea0 0x557e2ac1af10 crc :-1 s=READY pgs=167 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:05:10.768 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:10.773+0000 7fc102026740 1 --2- 172.21.15.29:0/1257787019 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557e2ab9d820 0x557e2aba61e0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:05:10.768 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:10.773+0000 7fc102026740 1 --2- 172.21.15.29:0/1257787019 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557e2ab94a50 0x557e2aba5ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:05:10.769 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:10.773+0000 7fc102026740 1 --2- 172.21.15.29:0/1257787019 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557e2ab94130 0x557e2aba5760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:05:10.769 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:10.773+0000 7fc0e0ff9640 0 client.26393 ms_handle_reset on v2:172.21.15.152:6838/1548731833 2024-03-16T12:05:10.769 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:10.773+0000 7fc102026740 1 -- 172.21.15.29:0/1257787019 >> 172.21.15.29:0/1257787019 conn(0x557e2ab8ea50 msgr2=0x557e2ab8d320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:05:10.769 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:10.774+0000 7fc102026740 1 -- 172.21.15.29:0/1257787019 shutdown_connections 2024-03-16T12:05:10.769 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:10.774+0000 7fc102026740 1 -- 172.21.15.29:0/1257787019 wait complete. 2024-03-16T12:05:10.769 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:10.774+0000 7fc102026740 20 client.26393 trim_cache size 0 max 16384 2024-03-16T12:05:10.769 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:{ 2024-03-16T12:05:10.769 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "message": "", 2024-03-16T12:05:10.769 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "return_code": 0 2024-03-16T12:05:10.769 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:} 2024-03-16T12:05:10.770 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:10.774+0000 7fc0ff436640 1 -- 172.21.15.29:0/1012116722 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7fc0e40a2a10 msgr2=0x7fc0e40a4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:05:10.770 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:10.774+0000 7fc0ff436640 1 --2- 172.21.15.29:0/1012116722 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7fc0e40a2a10 0x7fc0e40a4ed0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fc0f0005cc0 tx=0x7fc0f002ce60 comp rx=0 tx=0).stop 2024-03-16T12:05:10.770 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:10.774+0000 7fc0ff436640 1 -- 172.21.15.29:0/1012116722 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc0f81006a0 msgr2=0x7fc0f81985f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:05:10.771 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:10.774+0000 7fc0ff436640 1 --2- 172.21.15.29:0/1012116722 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc0f81006a0 0x7fc0f81985f0 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7fc0e800d950 tx=0x7fc0e800de10 comp rx=0 tx=0).stop 2024-03-16T12:05:10.771 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:10.774+0000 7fc0ff436640 1 -- 172.21.15.29:0/1012116722 shutdown_connections 2024-03-16T12:05:10.771 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:10.774+0000 7fc0ff436640 1 --2- 172.21.15.29:0/1012116722 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7fc0e40a2a10 0x7fc0e40a4ed0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:05:10.771 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:10.774+0000 7fc0ff436640 1 --2- 172.21.15.29:0/1012116722 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc0f81067c0 0x7fc0f819c920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:05:10.771 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:10.774+0000 7fc0ff436640 1 --2- 172.21.15.29:0/1012116722 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc0f81006a0 0x7fc0f81985f0 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:05:10.771 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:10.774+0000 7fc0ff436640 1 --2- 172.21.15.29:0/1012116722 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc0f80ffce0 0x7fc0f81980b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:05:10.771 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:10.774+0000 7fc0ff436640 1 -- 172.21.15.29:0/1012116722 >> 172.21.15.29:0/1012116722 conn(0x7fc0f8076c70 msgr2=0x7fc0f8104710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:05:10.771 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:10.774+0000 7fc0ff436640 1 -- 172.21.15.29:0/1012116722 shutdown_connections 2024-03-16T12:05:10.771 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:10.774+0000 7fc0ff436640 1 -- 172.21.15.29:0/1012116722 wait complete. 2024-03-16T12:05:11.184 INFO:tasks.workunit.client.0.smithi029.stdout: 10 16705 19.63 MB/sec warmup 55 sec latency 420.005 ms 2024-03-16T12:05:12.184 INFO:tasks.workunit.client.0.smithi029.stdout: 10 16932 19.64 MB/sec warmup 56 sec latency 293.794 ms 2024-03-16T12:05:12.539 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:05:12 smithi154 ceph-mon[36320]: pgmap v453: 97 pgs: 97 active+clean; 270 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 3.3 MiB/s wr, 736 op/s 2024-03-16T12:05:12.582 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:05:12 smithi029 ceph-mon[31975]: pgmap v453: 97 pgs: 97 active+clean; 270 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 3.3 MiB/s wr, 736 op/s 2024-03-16T12:05:12.621 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:05:12 smithi152 ceph-mon[36291]: pgmap v453: 97 pgs: 97 active+clean; 270 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 3.3 MiB/s wr, 736 op/s 2024-03-16T12:05:13.184 INFO:tasks.workunit.client.0.smithi029.stdout: 10 17108 19.43 MB/sec warmup 57 sec latency 263.279 ms 2024-03-16T12:05:14.184 INFO:tasks.workunit.client.0.smithi029.stdout: 10 17233 19.17 MB/sec warmup 58 sec latency 384.248 ms 2024-03-16T12:05:14.539 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:05:14 smithi154 ceph-mon[36320]: pgmap v454: 97 pgs: 97 active+clean; 266 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 2.7 MiB/s wr, 531 op/s 2024-03-16T12:05:14.582 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:05:14 smithi029 ceph-mon[31975]: pgmap v454: 97 pgs: 97 active+clean; 266 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 2.7 MiB/s wr, 531 op/s 2024-03-16T12:05:14.621 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:05:14 smithi152 ceph-mon[36291]: pgmap v454: 97 pgs: 97 active+clean; 266 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 2.7 MiB/s wr, 531 op/s 2024-03-16T12:05:15.184 INFO:tasks.workunit.client.0.smithi029.stdout: 10 17240 18.85 MB/sec warmup 59 sec latency 1384.337 ms 2024-03-16T12:05:16.184 INFO:tasks.workunit.client.0.smithi029.stdout: 10 17324 18.55 MB/sec warmup 60 sec latency 1929.772 ms 2024-03-16T12:05:16.539 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:05:16 smithi154 ceph-mon[36320]: pgmap v455: 97 pgs: 97 active+clean; 266 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 2.7 MiB/s wr, 505 op/s 2024-03-16T12:05:16.582 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:05:16 smithi029 ceph-mon[31975]: pgmap v455: 97 pgs: 97 active+clean; 266 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 2.7 MiB/s wr, 505 op/s 2024-03-16T12:05:16.621 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:05:16 smithi152 ceph-mon[36291]: pgmap v455: 97 pgs: 97 active+clean; 266 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 2.7 MiB/s wr, 505 op/s 2024-03-16T12:05:17.184 INFO:tasks.workunit.client.0.smithi029.stdout: 10 17473 18.32 MB/sec warmup 61 sec latency 213.585 ms 2024-03-16T12:05:17.539 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:05:17 smithi154 ceph-mon[36320]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T12:05:17.582 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:05:17 smithi029 ceph-mon[31975]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T12:05:17.621 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:05:17 smithi152 ceph-mon[36291]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T12:05:18.184 INFO:tasks.workunit.client.0.smithi029.stdout: 10 17649 18.12 MB/sec warmup 62 sec latency 87.740 ms 2024-03-16T12:05:18.539 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:05:18 smithi154 ceph-mon[36320]: pgmap v456: 97 pgs: 97 active+clean; 364 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 11 MiB/s wr, 670 op/s 2024-03-16T12:05:18.539 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:05:18 smithi154 ceph-mon[36320]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:05:18.539 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:05:18 smithi154 ceph-mon[36320]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T12:05:18.540 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:05:18 smithi154 ceph-mon[36320]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' 2024-03-16T12:05:18.582 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:05:18 smithi029 ceph-mon[31975]: pgmap v456: 97 pgs: 97 active+clean; 364 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 11 MiB/s wr, 670 op/s 2024-03-16T12:05:18.582 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:05:18 smithi029 ceph-mon[31975]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:05:18.582 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:05:18 smithi029 ceph-mon[31975]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T12:05:18.582 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:05:18 smithi029 ceph-mon[31975]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' 2024-03-16T12:05:18.621 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:05:18 smithi152 ceph-mon[36291]: pgmap v456: 97 pgs: 97 active+clean; 364 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 11 MiB/s wr, 670 op/s 2024-03-16T12:05:18.621 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:05:18 smithi152 ceph-mon[36291]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:05:18.622 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:05:18 smithi152 ceph-mon[36291]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T12:05:18.622 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:05:18 smithi152 ceph-mon[36291]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' 2024-03-16T12:05:19.184 INFO:tasks.workunit.client.0.smithi029.stdout: 10 17865 18.09 MB/sec warmup 63 sec latency 171.123 ms 2024-03-16T12:05:20.184 INFO:tasks.workunit.client.0.smithi029.stdout: 10 18091 17.92 MB/sec warmup 64 sec latency 244.568 ms 2024-03-16T12:05:20.539 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:05:20 smithi154 ceph-mon[36320]: pgmap v457: 97 pgs: 97 active+clean; 364 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 9.5 MiB/s wr, 420 op/s 2024-03-16T12:05:20.582 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:05:20 smithi029 ceph-mon[31975]: pgmap v457: 97 pgs: 97 active+clean; 364 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 9.5 MiB/s wr, 420 op/s 2024-03-16T12:05:20.621 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:05:20 smithi152 ceph-mon[36291]: pgmap v457: 97 pgs: 97 active+clean; 364 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 9.5 MiB/s wr, 420 op/s 2024-03-16T12:05:21.184 INFO:tasks.workunit.client.0.smithi029.stdout: 10 18495 17.89 MB/sec warmup 65 sec latency 513.189 ms 2024-03-16T12:05:22.184 INFO:tasks.workunit.client.0.smithi029.stdout: 10 18916 17.89 MB/sec warmup 66 sec latency 1194.243 ms 2024-03-16T12:05:22.539 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:05:22 smithi154 ceph-mon[36320]: pgmap v458: 97 pgs: 97 active+clean; 320 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 12 MiB/s wr, 632 op/s 2024-03-16T12:05:22.582 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:05:22 smithi029 ceph-mon[31975]: pgmap v458: 97 pgs: 97 active+clean; 320 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 12 MiB/s wr, 632 op/s 2024-03-16T12:05:22.621 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:05:22 smithi152 ceph-mon[36291]: pgmap v458: 97 pgs: 97 active+clean; 320 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 12 MiB/s wr, 632 op/s 2024-03-16T12:05:23.184 INFO:tasks.workunit.client.0.smithi029.stdout: 10 19410 18.05 MB/sec warmup 67 sec latency 224.803 ms 2024-03-16T12:05:23.864 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:23.869+0000 7f0f83027640 1 -- 172.21.15.29:0/1344133395 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0f7c105130 msgr2=0x7f0f7c10cd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:05:23.865 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:23.869+0000 7f0f83027640 1 --2- 172.21.15.29:0/1344133395 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0f7c105130 0x7f0f7c10cd00 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f0f74009ca0 tx=0x7f0f7402c0f0 comp rx=0 tx=0).stop 2024-03-16T12:05:23.865 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:23.869+0000 7f0f83027640 1 -- 172.21.15.29:0/1344133395 shutdown_connections 2024-03-16T12:05:23.865 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:23.869+0000 7f0f83027640 1 --2- 172.21.15.29:0/1344133395 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0f7c105130 0x7f0f7c10cd00 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:05:23.865 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:23.869+0000 7f0f83027640 1 --2- 172.21.15.29:0/1344133395 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0f7c100a10 0x7f0f7c104a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:05:23.865 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:23.869+0000 7f0f83027640 1 --2- 172.21.15.29:0/1344133395 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0f7c100060 0x7f0f7c100440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:05:23.865 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:23.869+0000 7f0f83027640 1 -- 172.21.15.29:0/1344133395 >> 172.21.15.29:0/1344133395 conn(0x7f0f7c076c70 msgr2=0x7f0f7c0fdff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:05:23.865 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:23.869+0000 7f0f83027640 1 -- 172.21.15.29:0/1344133395 shutdown_connections 2024-03-16T12:05:23.865 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:23.869+0000 7f0f83027640 1 -- 172.21.15.29:0/1344133395 wait complete. 2024-03-16T12:05:23.865 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:23.870+0000 7f0f83027640 1 Processor -- start 2024-03-16T12:05:23.865 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:23.870+0000 7f0f83027640 1 -- start start 2024-03-16T12:05:23.865 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:23.870+0000 7f0f83027640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0f7c100060 0x7f0f7c1980a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:05:23.865 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:23.870+0000 7f0f83027640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0f7c105130 0x7f0f7c1985e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:05:23.865 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:23.870+0000 7f0f83027640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0f7c19c960 0x7f0f7c19ce10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:05:23.865 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:23.870+0000 7f0f83027640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f0f7c10f960 con 0x7f0f7c100060 2024-03-16T12:05:23.865 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:23.870+0000 7f0f83027640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f0f7c10f7e0 con 0x7f0f7c19c960 2024-03-16T12:05:23.866 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:23.870+0000 7f0f83027640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f0f7c10fae0 con 0x7f0f7c105130 2024-03-16T12:05:23.866 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:23.870+0000 7f0f80dc3640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0f7c100060 0x7f0f7c1980a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:05:23.866 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:23.870+0000 7f0f80dc3640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0f7c100060 0x7f0f7c1980a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41108/0 (socket says 172.21.15.29:41108) 2024-03-16T12:05:23.866 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:23.870+0000 7f0f80dc3640 1 -- 172.21.15.29:0/957832779 learned_addr learned my addr 172.21.15.29:0/957832779 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T12:05:23.866 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:23.870+0000 7f0f7bfff640 1 --2- 172.21.15.29:0/957832779 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0f7c105130 0x7f0f7c1985e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:05:23.866 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:23.870+0000 7f0f815c4640 1 --2- 172.21.15.29:0/957832779 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0f7c19c960 0x7f0f7c19ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:05:23.866 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:23.870+0000 7f0f80dc3640 1 -- 172.21.15.29:0/957832779 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0f7c105130 msgr2=0x7f0f7c1985e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:05:23.866 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:23.870+0000 7f0f80dc3640 1 --2- 172.21.15.29:0/957832779 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0f7c105130 0x7f0f7c1985e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:05:23.866 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:23.870+0000 7f0f80dc3640 1 -- 172.21.15.29:0/957832779 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0f7c19c960 msgr2=0x7f0f7c19ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:05:23.866 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:23.870+0000 7f0f80dc3640 1 --2- 172.21.15.29:0/957832779 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0f7c19c960 0x7f0f7c19ce10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:05:23.866 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:23.870+0000 7f0f80dc3640 1 -- 172.21.15.29:0/957832779 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f7c19d450 con 0x7f0f7c100060 2024-03-16T12:05:23.866 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:23.871+0000 7f0f80dc3640 1 --2- 172.21.15.29:0/957832779 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0f7c100060 0x7f0f7c1980a0 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f0f7000ea20 tx=0x7f0f7000eee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:05:23.866 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:23.871+0000 7f0f79ffb640 1 -- 172.21.15.29:0/957832779 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f7000cd40 con 0x7f0f7c100060 2024-03-16T12:05:23.866 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:23.871+0000 7f0f83027640 1 -- 172.21.15.29:0/957832779 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0f7c19d740 con 0x7f0f7c100060 2024-03-16T12:05:23.866 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:23.871+0000 7f0f79ffb640 1 -- 172.21.15.29:0/957832779 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0f700168f0 con 0x7f0f7c100060 2024-03-16T12:05:23.867 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:23.871+0000 7f0f79ffb640 1 -- 172.21.15.29:0/957832779 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f7000b920 con 0x7f0f7c100060 2024-03-16T12:05:23.867 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:23.871+0000 7f0f83027640 1 -- 172.21.15.29:0/957832779 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0f7c1a4f30 con 0x7f0f7c100060 2024-03-16T12:05:23.871 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:23.873+0000 7f0f79ffb640 1 -- 172.21.15.29:0/957832779 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7f0f70016aa0 con 0x7f0f7c100060 2024-03-16T12:05:23.871 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:23.873+0000 7f0f85c12740 1 Processor -- start 2024-03-16T12:05:23.871 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:23.873+0000 7f0f85c12740 1 -- start start 2024-03-16T12:05:23.871 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:23.873+0000 7f0f79ffb640 1 --2- 172.21.15.29:0/957832779 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f0f6c0a2a10 0x7f0f6c0a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:05:23.871 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:23.873+0000 7f0f85c12740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cd6c2c6bd0 0x55cd6c2c6fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:05:23.872 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:23.874+0000 7f0f85c12740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55cd6c2c74f0 0x55cd6c2d46f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:05:23.872 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:23.874+0000 7f0f85c12740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cd6c2d4c30 0x55cd6c2d7020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:05:23.872 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:23.874+0000 7f0f85c12740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55cd6c22ba40 con 0x55cd6c2d4c30 2024-03-16T12:05:23.872 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:23.874+0000 7f0f85c12740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x55cd6c1d4920 con 0x55cd6c2c74f0 2024-03-16T12:05:23.872 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:23.874+0000 7f0f85c12740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55cd6c227ae0 con 0x55cd6c2c6bd0 2024-03-16T12:05:23.872 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:23.874+0000 7f0f80dc3640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cd6c2c6bd0 0x55cd6c2c6fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:05:23.872 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:23.874+0000 7f0f815c4640 1 --2- 172.21.15.29:0/957832779 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f0f6c0a2a10 0x7f0f6c0a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:05:23.872 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:23.874+0000 7f0f80dc3640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cd6c2c6bd0 0x55cd6c2c6fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.29:39828/0 (socket says 172.21.15.29:39828) 2024-03-16T12:05:23.872 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:23.874+0000 7f0f7bfff640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55cd6c2c74f0 0x55cd6c2d46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:05:23.872 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:23.874+0000 7f0f80dc3640 1 -- 172.21.15.29:0/1723211888 learned_addr learned my addr 172.21.15.29:0/1723211888 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T12:05:23.872 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:23.874+0000 7f0f815c4640 1 --2- 172.21.15.29:0/1723211888 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cd6c2d4c30 0x55cd6c2d7020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:05:23.872 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:23.874+0000 7f0f79ffb640 1 -- 172.21.15.29:0/957832779 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 8556+0+0 (secure 0 0 0) 0x7f0f700cece0 con 0x7f0f7c100060 2024-03-16T12:05:23.872 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:23.874+0000 7f0f80dc3640 1 -- 172.21.15.29:0/1723211888 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55cd6c2c74f0 msgr2=0x55cd6c2d46f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:05:23.872 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:23.874+0000 7f0f80dc3640 1 --2- 172.21.15.29:0/1723211888 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55cd6c2c74f0 0x55cd6c2d46f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:05:23.872 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:23.874+0000 7f0f80dc3640 1 -- 172.21.15.29:0/1723211888 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cd6c2d4c30 msgr2=0x55cd6c2d7020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:05:23.873 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:23.874+0000 7f0f80dc3640 1 --2- 172.21.15.29:0/1723211888 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cd6c2d4c30 0x55cd6c2d7020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:05:23.873 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:23.874+0000 7f0f80dc3640 1 -- 172.21.15.29:0/1723211888 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cd6c1ec460 con 0x55cd6c2c6bd0 2024-03-16T12:05:23.873 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:23.874+0000 7f0f815c4640 1 --2- 172.21.15.29:0/957832779 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f0f6c0a2a10 0x7f0f6c0a4ed0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f0f74004af0 tx=0x7f0f7402e040 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:05:23.873 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:23.874+0000 7f0f80dc3640 1 --2- 172.21.15.29:0/1723211888 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cd6c2c6bd0 0x55cd6c2c6fb0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f0f700cc470 tx=0x7f0f70012850 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2665fed04ae0c2cf server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:05:23.873 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:23.874+0000 7f0f66ffd640 1 -- 172.21.15.29:0/1723211888 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f700a0990 con 0x55cd6c2c6bd0 2024-03-16T12:05:23.873 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:23.875+0000 7f0f66ffd640 1 -- 172.21.15.29:0/1723211888 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0f7009b3d0 con 0x55cd6c2c6bd0 2024-03-16T12:05:23.873 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:23.875+0000 7f0f66ffd640 1 -- 172.21.15.29:0/1723211888 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f7009c450 con 0x55cd6c2c6bd0 2024-03-16T12:05:23.873 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:23.875+0000 7f0f85c12740 1 -- 172.21.15.29:0/1723211888 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cd6c2c6bd0 msgr2=0x55cd6c2c6fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:05:23.873 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:23.875+0000 7f0f85c12740 1 --2- 172.21.15.29:0/1723211888 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cd6c2c6bd0 0x55cd6c2c6fb0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f0f700cc470 tx=0x7f0f70012850 comp rx=0 tx=0).stop 2024-03-16T12:05:23.873 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:23.875+0000 7f0f85c12740 1 -- 172.21.15.29:0/1723211888 shutdown_connections 2024-03-16T12:05:23.873 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:23.875+0000 7f0f85c12740 1 --2- 172.21.15.29:0/1723211888 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cd6c2d4c30 0x55cd6c2d7020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:05:23.873 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:23.875+0000 7f0f85c12740 1 --2- 172.21.15.29:0/1723211888 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55cd6c2c74f0 0x55cd6c2d46f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:05:23.873 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:23.875+0000 7f0f85c12740 1 --2- 172.21.15.29:0/1723211888 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cd6c2c6bd0 0x55cd6c2c6fb0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:05:23.873 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:23.875+0000 7f0f85c12740 1 -- 172.21.15.29:0/1723211888 >> 172.21.15.29:0/1723211888 conn(0x55cd6c2c1a50 msgr2=0x55cd6c2c1e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:05:23.873 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:23.875+0000 7f0f85c12740 1 -- 172.21.15.29:0/1723211888 shutdown_connections 2024-03-16T12:05:23.873 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:23.875+0000 7f0f85c12740 1 -- 172.21.15.29:0/1723211888 wait complete. 2024-03-16T12:05:23.873 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:23.876+0000 7f0f85c12740 1 -- start start 2024-03-16T12:05:23.874 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:23.876+0000 7f0f85c12740 1 Processor -- start 2024-03-16T12:05:23.874 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:23.876+0000 7f0f85c12740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cd6c2c6bd0 0x55cd6c2d1bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:05:23.874 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:23.876+0000 7f0f85c12740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55cd6c2c74f0 0x55cd6c2d20f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:05:23.874 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:23.876+0000 7f0f85c12740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cd6c2d4c30 0x55cd6c2d2630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:05:23.874 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:23.876+0000 7f0f85c12740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55cd6c2d7800 con 0x55cd6c2c6bd0 2024-03-16T12:05:23.874 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:23.876+0000 7f0f85c12740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x55cd6c1e7d70 con 0x55cd6c2c74f0 2024-03-16T12:05:23.874 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:23.876+0000 7f0f85c12740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55cd6c2d7980 con 0x55cd6c2d4c30 2024-03-16T12:05:23.874 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:23.876+0000 7f0f80dc3640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cd6c2c6bd0 0x55cd6c2d1bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:05:23.874 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:23.876+0000 7f0f85c12740 10 client.0 _ll_register_callbacks cb 0x55cd6bf3fa00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T12:05:23.874 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:23.876+0000 7f0f80dc3640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cd6c2c6bd0 0x55cd6c2d1bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41116/0 (socket says 172.21.15.29:41116) 2024-03-16T12:05:23.874 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:23.876+0000 7f0f7bfff640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55cd6c2c74f0 0x55cd6c2d20f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:05:23.874 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:23.876+0000 7f0f80dc3640 1 -- 172.21.15.29:0/3383866322 learned_addr learned my addr 172.21.15.29:0/3383866322 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T12:05:23.874 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:23.876+0000 7f0f815c4640 1 --2- 172.21.15.29:0/3383866322 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cd6c2d4c30 0x55cd6c2d2630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:05:23.874 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:23.877+0000 7f0f80dc3640 1 -- 172.21.15.29:0/3383866322 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cd6c2d4c30 msgr2=0x55cd6c2d2630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:05:23.875 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:23.877+0000 7f0f80dc3640 1 --2- 172.21.15.29:0/3383866322 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cd6c2d4c30 0x55cd6c2d2630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:05:23.875 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:23.877+0000 7f0f80dc3640 1 -- 172.21.15.29:0/3383866322 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55cd6c2c74f0 msgr2=0x55cd6c2d20f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:05:23.875 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:23.877+0000 7f0f80dc3640 1 --2- 172.21.15.29:0/3383866322 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55cd6c2c74f0 0x55cd6c2d20f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:05:23.875 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:23.877+0000 7f0f80dc3640 1 -- 172.21.15.29:0/3383866322 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cd6bf40210 con 0x55cd6c2c6bd0 2024-03-16T12:05:23.875 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:23.877+0000 7f0f815c4640 1 --2- 172.21.15.29:0/3383866322 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cd6c2d4c30 0x55cd6c2d2630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:05:23.875 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:23.877+0000 7f0f7bfff640 1 --2- 172.21.15.29:0/3383866322 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55cd6c2c74f0 0x55cd6c2d20f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:05:23.875 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:23.877+0000 7f0f80dc3640 1 --2- 172.21.15.29:0/3383866322 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cd6c2c6bd0 0x55cd6c2d1bb0 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f0f700a3590 tx=0x7f0f700cc3f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37fb6b6dd6a71485 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:05:23.875 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:23.877+0000 7f0f5bfff640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-16T12:05:23.875 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:23.877+0000 7f0f5bfff640 1 -- 172.21.15.29:0/3383866322 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f700a0390 con 0x55cd6c2c6bd0 2024-03-16T12:05:23.875 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:23.877+0000 7f0f5bfff640 1 -- 172.21.15.29:0/3383866322 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(110..110 src has 1..110) ==== 8556+0+0 (secure 0 0 0) 0x7f0f700a0670 con 0x55cd6c2c6bd0 2024-03-16T12:05:23.875 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:23.877+0000 7f0f85c12740 1 -- 172.21.15.29:0/3383866322 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0f7009b3d0 con 0x55cd6c2c6bd0 2024-03-16T12:05:23.875 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:23.877+0000 7f0f5bfff640 1 -- 172.21.15.29:0/3383866322 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f0f700a0990 con 0x55cd6c2c6bd0 2024-03-16T12:05:23.875 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:23.877+0000 7f0f85c12740 10 client.16605 fetch_fsmap learned FSMap version 31 2024-03-16T12:05:23.875 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:23.877+0000 7f0f85c12740 1 -- 172.21.15.29:0/3383866322 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55cd6c00c620 con 0x55cd6c2c6bd0 2024-03-16T12:05:23.875 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:23.878+0000 7f0f5bfff640 1 -- 172.21.15.29:0/3383866322 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0f7009b3d0 con 0x55cd6c2c6bd0 2024-03-16T12:05:23.875 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:23.878+0000 7f0f5bfff640 1 -- 172.21.15.29:0/3383866322 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 31) ==== 8346+0+0 (secure 0 0 0) 0x7f0f7000b040 con 0x55cd6c2c6bd0 2024-03-16T12:05:23.875 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:23.878+0000 7f0f85c12740 10 client.16605 fetch_fsmap finished waiting for FSMap version 31 2024-03-16T12:05:23.876 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:23.878+0000 7f0f85c12740 10 client.16605 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T12:05:23.876 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:23.878+0000 7f0f85c12740 20 client.16605 populate_metadata read hostname 'smithi029' 2024-03-16T12:05:23.876 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:23.878+0000 7f0f85c12740 1 --2- 172.21.15.29:0/3383866322 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55cd6c2e4ba0 0x55cd6c2e6f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:05:23.876 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:23.878+0000 7f0f85c12740 4 client.16605 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T12:05:23.876 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:23.878+0000 7f0f85c12740 1 -- 172.21.15.29:0/3383866322 --> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cd6c238f80 con 0x55cd6c2e4ba0 2024-03-16T12:05:23.876 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:23.878+0000 7f0f7bfff640 1 --2- 172.21.15.29:0/3383866322 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55cd6c2e4ba0 0x55cd6c2e6f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:05:23.877 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:23.878+0000 7f0f5bfff640 10 client.16605 ms_handle_connect on v2:172.21.15.152:6838/1548731833 2024-03-16T12:05:23.877 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:23.878+0000 7f0f7bfff640 1 --2- 172.21.15.29:0/3383866322 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55cd6c2e4ba0 0x55cd6c2e6f90 crc :-1 s=READY pgs=168 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f956aaebec87fe1d server_cookie=99c37fd17ced9053 in_seq=0 out_seq=0 2024-03-16T12:05:23.877 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:23.880+0000 7f0f5bfff640 1 -- 172.21.15.29:0/3383866322 <== mds.0 v2:172.21.15.152:6838/1548731833 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55cd6c238f80 con 0x55cd6c2e4ba0 2024-03-16T12:05:23.877 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:23.880+0000 7f0f5bfff640 10 client.16605 handle_command_reply: tid=0 2024-03-16T12:05:23.877 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:23.880+0000 7f0f85c12740 1 client.16605 shutdown 2024-03-16T12:05:23.877 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:23.880+0000 7f0f85c12740 1 -- 172.21.15.29:0/3383866322 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cd6c2c6bd0 msgr2=0x55cd6c2d1bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:05:23.877 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:23.880+0000 7f0f85c12740 1 --2- 172.21.15.29:0/3383866322 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cd6c2c6bd0 0x55cd6c2d1bb0 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f0f700a3590 tx=0x7f0f700cc3f0 comp rx=0 tx=0).stop 2024-03-16T12:05:23.877 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:23.880+0000 7f0f85c12740 1 -- 172.21.15.29:0/3383866322 shutdown_connections 2024-03-16T12:05:23.878 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:23.880+0000 7f0f85c12740 1 --2- 172.21.15.29:0/3383866322 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cd6c2d4c30 0x55cd6c2d2630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:05:23.878 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:23.880+0000 7f0f85c12740 1 --2- 172.21.15.29:0/3383866322 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55cd6c2e4ba0 0x55cd6c2e6f90 crc :-1 s=READY pgs=168 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:05:23.878 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:23.880+0000 7f0f85c12740 1 --2- 172.21.15.29:0/3383866322 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55cd6c2c74f0 0x55cd6c2d20f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:05:23.878 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:23.880+0000 7f0f85c12740 1 --2- 172.21.15.29:0/3383866322 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cd6c2c6bd0 0x55cd6c2d1bb0 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:05:23.878 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:23.880+0000 7f0f85c12740 1 -- 172.21.15.29:0/3383866322 >> 172.21.15.29:0/3383866322 conn(0x55cd6c2c1a50 msgr2=0x55cd6c2c37a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:05:23.878 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:23.880+0000 7f0f5bfff640 0 client.16605 ms_handle_reset on v2:172.21.15.152:6838/1548731833 2024-03-16T12:05:23.878 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:23.880+0000 7f0f85c12740 1 -- 172.21.15.29:0/3383866322 shutdown_connections 2024-03-16T12:05:23.878 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:23.880+0000 7f0f85c12740 1 -- 172.21.15.29:0/3383866322 wait complete. 2024-03-16T12:05:23.878 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:23.881+0000 7f0f85c12740 20 client.16605 trim_cache size 0 max 16384 2024-03-16T12:05:23.892 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:23.897+0000 7f0f85c12740 1 Processor -- start 2024-03-16T12:05:23.892 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:23.897+0000 7f0f85c12740 1 -- start start 2024-03-16T12:05:23.892 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:23.898+0000 7f0f85c12740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cd6c2c6bd0 0x55cd6c2d42e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:05:23.892 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:23.898+0000 7f0f85c12740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55cd6c2c74f0 0x55cd6c2e05e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:05:23.892 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:23.898+0000 7f0f85c12740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cd6c2d4c30 0x55cd6c2c35b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:05:23.892 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:23.898+0000 7f0f85c12740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f0f7c1adb80 con 0x55cd6c2c6bd0 2024-03-16T12:05:23.892 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:23.898+0000 7f0f85c12740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f0f7c1ab3f0 con 0x55cd6c2c74f0 2024-03-16T12:05:23.892 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:23.898+0000 7f0f85c12740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f0f7c1a9280 con 0x55cd6c2d4c30 2024-03-16T12:05:23.892 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:23.898+0000 7f0f80dc3640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cd6c2c6bd0 0x55cd6c2d42e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:05:23.893 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:23.898+0000 7f0f80dc3640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cd6c2c6bd0 0x55cd6c2d42e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41122/0 (socket says 172.21.15.29:41122) 2024-03-16T12:05:23.893 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:23.898+0000 7f0f80dc3640 1 -- 172.21.15.29:0/452034146 learned_addr learned my addr 172.21.15.29:0/452034146 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T12:05:23.893 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:23.898+0000 7f0f7bfff640 1 --2- 172.21.15.29:0/452034146 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55cd6c2c74f0 0x55cd6c2e05e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:05:23.893 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:23.898+0000 7f0f815c4640 1 --2- 172.21.15.29:0/452034146 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cd6c2d4c30 0x55cd6c2c35b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:05:23.893 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:23.898+0000 7f0f80dc3640 1 -- 172.21.15.29:0/452034146 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cd6c2d4c30 msgr2=0x55cd6c2c35b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:05:23.893 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:23.898+0000 7f0f80dc3640 1 --2- 172.21.15.29:0/452034146 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cd6c2d4c30 0x55cd6c2c35b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:05:23.893 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:23.898+0000 7f0f80dc3640 1 -- 172.21.15.29:0/452034146 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55cd6c2c74f0 msgr2=0x55cd6c2e05e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:05:23.893 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:23.898+0000 7f0f80dc3640 1 --2- 172.21.15.29:0/452034146 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55cd6c2c74f0 0x55cd6c2e05e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:05:23.893 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:23.898+0000 7f0f80dc3640 1 -- 172.21.15.29:0/452034146 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cd6c2cca00 con 0x55cd6c2c6bd0 2024-03-16T12:05:23.893 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:23.898+0000 7f0f7bfff640 1 --2- 172.21.15.29:0/452034146 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55cd6c2c74f0 0x55cd6c2e05e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:05:23.893 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:23.898+0000 7f0f80dc3640 1 --2- 172.21.15.29:0/452034146 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cd6c2c6bd0 0x55cd6c2d42e0 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f0f700cc4c0 tx=0x7f0f70097670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ac473aea3fd0f3b server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:05:23.893 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:23.899+0000 7f0f667fc640 1 -- 172.21.15.29:0/452034146 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f7009b3d0 con 0x55cd6c2c6bd0 2024-03-16T12:05:23.893 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:23.899+0000 7f0f667fc640 1 -- 172.21.15.29:0/452034146 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0f7009f470 con 0x55cd6c2c6bd0 2024-03-16T12:05:23.893 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:23.899+0000 7f0f667fc640 1 -- 172.21.15.29:0/452034146 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f7000b040 con 0x55cd6c2c6bd0 2024-03-16T12:05:23.896 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:23.900+0000 7f0f85c12740 1 -- 172.21.15.29:0/452034146 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cd6c2c6bd0 msgr2=0x55cd6c2d42e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:05:23.896 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:23.900+0000 7f0f85c12740 1 --2- 172.21.15.29:0/452034146 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cd6c2c6bd0 0x55cd6c2d42e0 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f0f700cc4c0 tx=0x7f0f70097670 comp rx=0 tx=0).stop 2024-03-16T12:05:23.896 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:23.900+0000 7f0f85c12740 1 -- 172.21.15.29:0/452034146 shutdown_connections 2024-03-16T12:05:23.896 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:23.900+0000 7f0f85c12740 1 --2- 172.21.15.29:0/452034146 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cd6c2d4c30 0x55cd6c2c35b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:05:23.896 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:23.900+0000 7f0f85c12740 1 --2- 172.21.15.29:0/452034146 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55cd6c2c74f0 0x55cd6c2e05e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:05:23.896 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:23.900+0000 7f0f85c12740 1 --2- 172.21.15.29:0/452034146 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cd6c2c6bd0 0x55cd6c2d42e0 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:05:23.896 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:23.900+0000 7f0f85c12740 1 -- 172.21.15.29:0/452034146 >> 172.21.15.29:0/452034146 conn(0x55cd6c2c1a50 msgr2=0x55cd6c2c2ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:05:23.897 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:23.900+0000 7f0f85c12740 1 -- 172.21.15.29:0/452034146 shutdown_connections 2024-03-16T12:05:23.897 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:23.900+0000 7f0f85c12740 1 -- 172.21.15.29:0/452034146 wait complete. 2024-03-16T12:05:23.897 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:23.900+0000 7f0f85c12740 1 -- start start 2024-03-16T12:05:23.897 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:23.900+0000 7f0f85c12740 1 Processor -- start 2024-03-16T12:05:23.897 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:23.901+0000 7f0f85c12740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cd6c2c6bd0 0x55cd6c2dcc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:05:23.897 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:23.901+0000 7f0f85c12740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cd6c2c74f0 0x55cd6c2ce230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:05:23.897 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:23.901+0000 7f0f85c12740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55cd6c2d4c30 0x55cd6c2ce770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:05:23.897 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:23.901+0000 7f0f85c12740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f0f7c1a98d0 con 0x55cd6c2c6bd0 2024-03-16T12:05:23.897 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:23.901+0000 7f0f85c12740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f0f7c1aad20 con 0x55cd6c2d4c30 2024-03-16T12:05:23.897 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:23.901+0000 7f0f85c12740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f0f7c1a9f80 con 0x55cd6c2c74f0 2024-03-16T12:05:23.897 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:23.901+0000 7f0f85c12740 10 client.0 _ll_register_callbacks cb 0x7f0f70005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T12:05:23.897 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:23.901+0000 7f0f80dc3640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cd6c2c6bd0 0x55cd6c2dcc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:05:23.897 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:23.901+0000 7f0f7bfff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cd6c2c74f0 0x55cd6c2ce230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:05:23.897 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:23.901+0000 7f0f7bfff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cd6c2c74f0 0x55cd6c2ce230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.29:39854/0 (socket says 172.21.15.29:39854) 2024-03-16T12:05:23.897 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:23.901+0000 7f0f80dc3640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cd6c2c6bd0 0x55cd6c2dcc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41128/0 (socket says 172.21.15.29:41128) 2024-03-16T12:05:23.898 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:23.901+0000 7f0f815c4640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55cd6c2d4c30 0x55cd6c2ce770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:05:23.898 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:23.901+0000 7f0f7bfff640 1 -- 172.21.15.29:0/4242211076 learned_addr learned my addr 172.21.15.29:0/4242211076 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T12:05:23.898 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:23.901+0000 7f0f80dc3640 1 -- 172.21.15.29:0/4242211076 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cd6c2c74f0 msgr2=0x55cd6c2ce230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:05:23.898 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:23.901+0000 7f0f80dc3640 1 --2- 172.21.15.29:0/4242211076 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cd6c2c74f0 0x55cd6c2ce230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:05:23.898 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:23.901+0000 7f0f80dc3640 1 -- 172.21.15.29:0/4242211076 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55cd6c2d4c30 msgr2=0x55cd6c2ce770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:05:23.898 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:23.901+0000 7f0f80dc3640 1 --2- 172.21.15.29:0/4242211076 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55cd6c2d4c30 0x55cd6c2ce770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:05:23.898 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:23.901+0000 7f0f80dc3640 1 -- 172.21.15.29:0/4242211076 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cd6c2cecb0 con 0x55cd6c2c6bd0 2024-03-16T12:05:23.898 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:23.901+0000 7f0f80dc3640 1 --2- 172.21.15.29:0/4242211076 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cd6c2c6bd0 0x55cd6c2dcc10 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f0f70099680 tx=0x7f0f70016e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4880a9314bc64c21 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:05:23.898 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:23.901+0000 7f0f5bfff640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-16T12:05:23.898 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:23.902+0000 7f0f5bfff640 1 -- 172.21.15.29:0/4242211076 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f700a0390 con 0x55cd6c2c6bd0 2024-03-16T12:05:23.898 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:23.902+0000 7f0f5bfff640 1 -- 172.21.15.29:0/4242211076 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(110..110 src has 1..110) ==== 8556+0+0 (secure 0 0 0) 0x7f0f7009f140 con 0x55cd6c2c6bd0 2024-03-16T12:05:23.898 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:23.902+0000 7f0f85c12740 1 -- 172.21.15.29:0/4242211076 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0f7009f470 con 0x55cd6c2c6bd0 2024-03-16T12:05:23.898 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:23.902+0000 7f0f5bfff640 1 -- 172.21.15.29:0/4242211076 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f0f70084050 con 0x55cd6c2c6bd0 2024-03-16T12:05:23.898 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:23.902+0000 7f0f85c12740 10 client.16611 fetch_fsmap learned FSMap version 31 2024-03-16T12:05:23.898 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:23.902+0000 7f0f85c12740 1 -- 172.21.15.29:0/4242211076 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55cd6c2cecb0 con 0x55cd6c2c6bd0 2024-03-16T12:05:23.899 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:23.902+0000 7f0f5bfff640 1 -- 172.21.15.29:0/4242211076 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0f7009f470 con 0x55cd6c2c6bd0 2024-03-16T12:05:23.899 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:23.902+0000 7f0f5bfff640 1 -- 172.21.15.29:0/4242211076 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 31) ==== 8346+0+0 (secure 0 0 0) 0x7f0f7009a820 con 0x55cd6c2c6bd0 2024-03-16T12:05:23.899 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:23.902+0000 7f0f85c12740 10 client.16611 fetch_fsmap finished waiting for FSMap version 31 2024-03-16T12:05:23.899 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:23.902+0000 7f0f85c12740 10 client.16611 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T12:05:23.899 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:23.902+0000 7f0f85c12740 20 client.16611 populate_metadata read hostname 'smithi029' 2024-03-16T12:05:23.899 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:23.902+0000 7f0f85c12740 1 --2- 172.21.15.29:0/4242211076 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55cd6c34ab90 0x55cd6c34cfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:05:23.899 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:23.902+0000 7f0f85c12740 4 client.16611 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-16T12:05:23.899 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:23.902+0000 7f0f85c12740 1 -- 172.21.15.29:0/4242211076 --> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55cd6c2332d0 con 0x55cd6c34ab90 2024-03-16T12:05:23.899 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:23.902+0000 7f0f7bfff640 1 --2- 172.21.15.29:0/4242211076 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55cd6c34ab90 0x55cd6c34cfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:05:23.899 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:23.903+0000 7f0f5bfff640 10 client.16611 ms_handle_connect on v2:172.21.15.152:6838/1548731833 2024-03-16T12:05:23.899 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:23.903+0000 7f0f7bfff640 1 --2- 172.21.15.29:0/4242211076 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55cd6c34ab90 0x55cd6c34cfb0 crc :-1 s=READY pgs=169 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e45082c03231c7d4 server_cookie=260cbc90842d1d99 in_seq=0 out_seq=0 2024-03-16T12:05:24.184 INFO:tasks.workunit.client.0.smithi029.stdout: 10 19705 18.20 MB/sec warmup 68 sec latency 420.874 ms 2024-03-16T12:05:24.537 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:24.543+0000 7f0f5bfff640 1 -- 172.21.15.29:0/4242211076 <== mds.0 v2:172.21.15.152:6838/1548731833 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55cd6c2332d0 con 0x55cd6c34ab90 2024-03-16T12:05:24.538 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:24.543+0000 7f0f5bfff640 10 client.16611 handle_command_reply: tid=0 2024-03-16T12:05:24.538 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:24.543+0000 7f0f85c12740 1 client.16611 shutdown 2024-03-16T12:05:24.538 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:24.543+0000 7f0f85c12740 1 -- 172.21.15.29:0/4242211076 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cd6c2c6bd0 msgr2=0x55cd6c2dcc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:05:24.538 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:24.543+0000 7f0f85c12740 1 --2- 172.21.15.29:0/4242211076 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cd6c2c6bd0 0x55cd6c2dcc10 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f0f70099680 tx=0x7f0f70016e30 comp rx=0 tx=0).stop 2024-03-16T12:05:24.538 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:24.544+0000 7f0f85c12740 1 -- 172.21.15.29:0/4242211076 shutdown_connections 2024-03-16T12:05:24.538 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:24.544+0000 7f0f85c12740 1 --2- 172.21.15.29:0/4242211076 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55cd6c34ab90 0x55cd6c34cfb0 crc :-1 s=READY pgs=169 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:05:24.538 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:24.544+0000 7f0f85c12740 1 --2- 172.21.15.29:0/4242211076 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55cd6c2d4c30 0x55cd6c2ce770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:05:24.538 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:24.544+0000 7f0f85c12740 1 --2- 172.21.15.29:0/4242211076 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cd6c2c74f0 0x55cd6c2ce230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:05:24.538 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:24.544+0000 7f0f85c12740 1 --2- 172.21.15.29:0/4242211076 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cd6c2c6bd0 0x55cd6c2dcc10 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:05:24.538 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:24.544+0000 7f0f5bfff640 0 client.16611 ms_handle_reset on v2:172.21.15.152:6838/1548731833 2024-03-16T12:05:24.538 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:24.544+0000 7f0f85c12740 1 -- 172.21.15.29:0/4242211076 >> 172.21.15.29:0/4242211076 conn(0x55cd6c2c1a50 msgr2=0x55cd6c2c0bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:05:24.538 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:24.544+0000 7f0f85c12740 1 -- 172.21.15.29:0/4242211076 shutdown_connections 2024-03-16T12:05:24.539 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:24.544+0000 7f0f85c12740 1 -- 172.21.15.29:0/4242211076 wait complete. 2024-03-16T12:05:24.539 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:24.544+0000 7f0f85c12740 20 client.16611 trim_cache size 0 max 16384 2024-03-16T12:05:24.539 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:{ 2024-03-16T12:05:24.539 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "message": "", 2024-03-16T12:05:24.539 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "return_code": 0 2024-03-16T12:05:24.539 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:} 2024-03-16T12:05:24.540 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:24.545+0000 7f0f83027640 1 -- 172.21.15.29:0/957832779 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f0f6c0a2a10 msgr2=0x7f0f6c0a4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:05:24.540 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:24.545+0000 7f0f83027640 1 --2- 172.21.15.29:0/957832779 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f0f6c0a2a10 0x7f0f6c0a4ed0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f0f74004af0 tx=0x7f0f7402e040 comp rx=0 tx=0).stop 2024-03-16T12:05:24.540 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:24.545+0000 7f0f83027640 1 -- 172.21.15.29:0/957832779 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0f7c100060 msgr2=0x7f0f7c1980a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:05:24.540 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:24.545+0000 7f0f83027640 1 --2- 172.21.15.29:0/957832779 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0f7c100060 0x7f0f7c1980a0 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f0f7000ea20 tx=0x7f0f7000eee0 comp rx=0 tx=0).stop 2024-03-16T12:05:24.540 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:24.545+0000 7f0f83027640 1 -- 172.21.15.29:0/957832779 shutdown_connections 2024-03-16T12:05:24.540 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:24.545+0000 7f0f83027640 1 --2- 172.21.15.29:0/957832779 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f0f6c0a2a10 0x7f0f6c0a4ed0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:05:24.540 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:24.545+0000 7f0f83027640 1 --2- 172.21.15.29:0/957832779 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0f7c19c960 0x7f0f7c19ce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:05:24.540 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:24.545+0000 7f0f83027640 1 --2- 172.21.15.29:0/957832779 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0f7c105130 0x7f0f7c1985e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:05:24.540 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:24.545+0000 7f0f83027640 1 --2- 172.21.15.29:0/957832779 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0f7c100060 0x7f0f7c1980a0 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:05:24.540 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:24.545+0000 7f0f83027640 1 -- 172.21.15.29:0/957832779 >> 172.21.15.29:0/957832779 conn(0x7f0f7c076c70 msgr2=0x7f0f7c0fe590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:05:24.540 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:24.545+0000 7f0f83027640 1 -- 172.21.15.29:0/957832779 shutdown_connections 2024-03-16T12:05:24.540 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:24.545+0000 7f0f83027640 1 -- 172.21.15.29:0/957832779 wait complete. 2024-03-16T12:05:24.541 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:05:24 smithi154 ceph-mon[36320]: pgmap v459: 97 pgs: 97 active+clean; 316 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 12 MiB/s wr, 427 op/s 2024-03-16T12:05:24.582 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:05:24 smithi029 ceph-mon[31975]: pgmap v459: 97 pgs: 97 active+clean; 316 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 12 MiB/s wr, 427 op/s 2024-03-16T12:05:24.621 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:05:24 smithi152 ceph-mon[36291]: pgmap v459: 97 pgs: 97 active+clean; 316 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 12 MiB/s wr, 427 op/s 2024-03-16T12:05:25.184 INFO:tasks.workunit.client.0.smithi029.stdout: 10 19920 18.18 MB/sec warmup 69 sec latency 625.062 ms 2024-03-16T12:05:26.184 INFO:tasks.workunit.client.0.smithi029.stdout: 10 20184 18.12 MB/sec warmup 70 sec latency 336.028 ms 2024-03-16T12:05:26.539 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:05:26 smithi154 ceph-mon[36320]: pgmap v460: 97 pgs: 97 active+clean; 316 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 12 MiB/s wr, 404 op/s 2024-03-16T12:05:26.581 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:05:26 smithi029 ceph-mon[31975]: pgmap v460: 97 pgs: 97 active+clean; 316 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 12 MiB/s wr, 404 op/s 2024-03-16T12:05:26.621 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:05:26 smithi152 ceph-mon[36291]: pgmap v460: 97 pgs: 97 active+clean; 316 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 12 MiB/s wr, 404 op/s 2024-03-16T12:05:27.185 INFO:tasks.workunit.client.0.smithi029.stdout: 10 20368 18.06 MB/sec warmup 71 sec latency 246.474 ms 2024-03-16T12:05:28.184 INFO:tasks.workunit.client.0.smithi029.stdout: 10 20533 17.95 MB/sec warmup 72 sec latency 213.595 ms 2024-03-16T12:05:28.582 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:05:28 smithi029 ceph-mon[31975]: pgmap v461: 97 pgs: 97 active+clean; 306 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 MiB/s rd, 13 MiB/s wr, 695 op/s 2024-03-16T12:05:28.621 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:05:28 smithi152 ceph-mon[36291]: pgmap v461: 97 pgs: 97 active+clean; 306 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 MiB/s rd, 13 MiB/s wr, 695 op/s 2024-03-16T12:05:28.789 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:05:28 smithi154 ceph-mon[36320]: pgmap v461: 97 pgs: 97 active+clean; 306 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 MiB/s rd, 13 MiB/s wr, 695 op/s 2024-03-16T12:05:29.184 INFO:tasks.workunit.client.0.smithi029.stdout: 10 20702 17.75 MB/sec warmup 73 sec latency 123.320 ms 2024-03-16T12:05:30.184 INFO:tasks.workunit.client.0.smithi029.stdout: 10 20878 17.60 MB/sec warmup 74 sec latency 149.562 ms 2024-03-16T12:05:30.581 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:05:30 smithi029 ceph-mon[31975]: pgmap v462: 97 pgs: 97 active+clean; 306 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 4.7 MiB/s wr, 531 op/s 2024-03-16T12:05:30.621 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:05:30 smithi152 ceph-mon[36291]: pgmap v462: 97 pgs: 97 active+clean; 306 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 4.7 MiB/s wr, 531 op/s 2024-03-16T12:05:30.789 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:05:30 smithi154 ceph-mon[36320]: pgmap v462: 97 pgs: 97 active+clean; 306 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 4.7 MiB/s wr, 531 op/s 2024-03-16T12:05:31.184 INFO:tasks.workunit.client.0.smithi029.stdout: 10 21110 17.53 MB/sec warmup 75 sec latency 163.713 ms 2024-03-16T12:05:32.185 INFO:tasks.workunit.client.0.smithi029.stdout: 10 21372 17.43 MB/sec warmup 76 sec latency 235.113 ms 2024-03-16T12:05:32.581 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:05:32 smithi029 ceph-mon[31975]: pgmap v463: 97 pgs: 97 active+clean; 313 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 MiB/s rd, 5.6 MiB/s wr, 623 op/s 2024-03-16T12:05:32.621 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:05:32 smithi152 ceph-mon[36291]: pgmap v463: 97 pgs: 97 active+clean; 313 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 MiB/s rd, 5.6 MiB/s wr, 623 op/s 2024-03-16T12:05:32.789 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:05:32 smithi154 ceph-mon[36320]: pgmap v463: 97 pgs: 97 active+clean; 313 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 MiB/s rd, 5.6 MiB/s wr, 623 op/s 2024-03-16T12:05:33.185 INFO:tasks.workunit.client.0.smithi029.stdout: 10 21805 17.39 MB/sec warmup 77 sec latency 130.093 ms 2024-03-16T12:05:34.184 INFO:tasks.workunit.client.0.smithi029.stdout: 10 22210 17.49 MB/sec warmup 78 sec latency 209.118 ms 2024-03-16T12:05:34.582 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:05:34 smithi029 ceph-mon[31975]: pgmap v464: 97 pgs: 97 active+clean; 312 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 2.6 MiB/s wr, 416 op/s 2024-03-16T12:05:34.621 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:05:34 smithi152 ceph-mon[36291]: pgmap v464: 97 pgs: 97 active+clean; 312 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 2.6 MiB/s wr, 416 op/s 2024-03-16T12:05:34.789 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:05:34 smithi154 ceph-mon[36320]: pgmap v464: 97 pgs: 97 active+clean; 312 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 2.6 MiB/s wr, 416 op/s 2024-03-16T12:05:35.184 INFO:tasks.workunit.client.0.smithi029.stdout: 10 22693 17.55 MB/sec warmup 79 sec latency 1116.918 ms 2024-03-16T12:05:36.185 INFO:tasks.workunit.client.0.smithi029.stdout: 10 23021 17.66 MB/sec warmup 80 sec latency 2116.958 ms 2024-03-16T12:05:36.581 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:05:36 smithi029 ceph-mon[31975]: pgmap v465: 97 pgs: 97 active+clean; 312 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 2.0 MiB/s wr, 389 op/s 2024-03-16T12:05:36.621 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:05:36 smithi152 ceph-mon[36291]: pgmap v465: 97 pgs: 97 active+clean; 312 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 2.0 MiB/s wr, 389 op/s 2024-03-16T12:05:36.789 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:05:36 smithi154 ceph-mon[36320]: pgmap v465: 97 pgs: 97 active+clean; 312 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 2.0 MiB/s wr, 389 op/s 2024-03-16T12:05:37.185 INFO:tasks.workunit.client.0.smithi029.stdout: 10 23328 17.74 MB/sec warmup 81 sec latency 2788.637 ms 2024-03-16T12:05:37.630 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:37.636+0000 7f5b52244640 1 -- 172.21.15.29:0/2934808078 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5b4c1012f0 msgr2=0x7f5b4c10d510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:05:37.630 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:37.636+0000 7f5b52244640 1 --2- 172.21.15.29:0/2934808078 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5b4c1012f0 0x7f5b4c10d510 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f5b44009ca0 tx=0x7f5b4402c0f0 comp rx=0 tx=0).stop 2024-03-16T12:05:37.630 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:37.636+0000 7f5b52244640 1 -- 172.21.15.29:0/2934808078 shutdown_connections 2024-03-16T12:05:37.630 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:37.636+0000 7f5b52244640 1 --2- 172.21.15.29:0/2934808078 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5b4c1012f0 0x7f5b4c10d510 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:05:37.630 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:37.636+0000 7f5b52244640 1 --2- 172.21.15.29:0/2934808078 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5b4c100930 0x7f5b4c100db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:05:37.630 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:37.636+0000 7f5b52244640 1 --2- 172.21.15.29:0/2934808078 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5b4c105680 0x7f5b4c105a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:05:37.631 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:37.636+0000 7f5b52244640 1 -- 172.21.15.29:0/2934808078 >> 172.21.15.29:0/2934808078 conn(0x7f5b4c077530 msgr2=0x7f5b4c077940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:05:37.631 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:37.636+0000 7f5b52244640 1 -- 172.21.15.29:0/2934808078 shutdown_connections 2024-03-16T12:05:37.631 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:37.636+0000 7f5b52244640 1 -- 172.21.15.29:0/2934808078 wait complete. 2024-03-16T12:05:37.631 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:37.636+0000 7f5b52244640 1 Processor -- start 2024-03-16T12:05:37.633 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:37.637+0000 7f5b52244640 1 -- start start 2024-03-16T12:05:37.633 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:37.637+0000 7f5b52244640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5b4c100930 0x7f5b4c198090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:05:37.633 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:37.637+0000 7f5b52244640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5b4c1012f0 0x7f5b4c1985d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:05:37.633 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:37.637+0000 7f5b52244640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5b4c105680 0x7f5b4c19c900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:05:37.633 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:37.637+0000 7f5b52244640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f5b4c110ba0 con 0x7f5b4c100930 2024-03-16T12:05:37.633 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:37.637+0000 7f5b52244640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f5b4c110a20 con 0x7f5b4c1012f0 2024-03-16T12:05:37.633 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:37.637+0000 7f5b52244640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f5b4c110d20 con 0x7f5b4c105680 2024-03-16T12:05:37.633 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:37.637+0000 7f5b4b7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5b4c100930 0x7f5b4c198090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:05:37.633 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:37.637+0000 7f5b4b7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5b4c100930 0x7f5b4c198090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43584/0 (socket says 172.21.15.29:43584) 2024-03-16T12:05:37.633 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:37.637+0000 7f5b4b7fe640 1 -- 172.21.15.29:0/2867652856 learned_addr learned my addr 172.21.15.29:0/2867652856 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T12:05:37.633 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:37.637+0000 7f5b4bfff640 1 --2- 172.21.15.29:0/2867652856 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5b4c105680 0x7f5b4c19c900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:05:37.633 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:37.637+0000 7f5b4affd640 1 --2- 172.21.15.29:0/2867652856 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5b4c1012f0 0x7f5b4c1985d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:05:37.633 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:37.637+0000 7f5b4b7fe640 1 -- 172.21.15.29:0/2867652856 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5b4c105680 msgr2=0x7f5b4c19c900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:05:37.634 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:37.637+0000 7f5b4b7fe640 1 --2- 172.21.15.29:0/2867652856 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5b4c105680 0x7f5b4c19c900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:05:37.634 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:37.637+0000 7f5b4b7fe640 1 -- 172.21.15.29:0/2867652856 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5b4c1012f0 msgr2=0x7f5b4c1985d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:05:37.634 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:37.637+0000 7f5b4b7fe640 1 --2- 172.21.15.29:0/2867652856 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5b4c1012f0 0x7f5b4c1985d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:05:37.634 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:37.637+0000 7f5b4b7fe640 1 -- 172.21.15.29:0/2867652856 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b4c19d030 con 0x7f5b4c100930 2024-03-16T12:05:37.634 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:37.637+0000 7f5b4bfff640 1 --2- 172.21.15.29:0/2867652856 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5b4c105680 0x7f5b4c19c900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:05:37.634 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:37.637+0000 7f5b4b7fe640 1 --2- 172.21.15.29:0/2867652856 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5b4c100930 0x7f5b4c198090 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f5b3c00e9f0 tx=0x7f5b3c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:05:37.634 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:37.638+0000 7f5b48ff9640 1 -- 172.21.15.29:0/2867652856 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b3c00cd80 con 0x7f5b4c100930 2024-03-16T12:05:37.634 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:37.638+0000 7f5b48ff9640 1 -- 172.21.15.29:0/2867652856 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5b3c010910 con 0x7f5b4c100930 2024-03-16T12:05:37.634 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:37.638+0000 7f5b52244640 1 -- 172.21.15.29:0/2867652856 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5b4c19d320 con 0x7f5b4c100930 2024-03-16T12:05:37.634 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:37.638+0000 7f5b48ff9640 1 -- 172.21.15.29:0/2867652856 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b3c018990 con 0x7f5b4c100930 2024-03-16T12:05:37.634 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:37.638+0000 7f5b52244640 1 -- 172.21.15.29:0/2867652856 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5b4c1a4bb0 con 0x7f5b4c100930 2024-03-16T12:05:37.639 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:37.640+0000 7f5b48ff9640 1 -- 172.21.15.29:0/2867652856 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7f5b3c010ac0 con 0x7f5b4c100930 2024-03-16T12:05:37.639 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:37.640+0000 7f5b54e2e740 1 Processor -- start 2024-03-16T12:05:37.640 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:37.640+0000 7f5b54e2e740 1 -- start start 2024-03-16T12:05:37.640 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:37.640+0000 7f5b54e2e740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5574aab07af0 0x5574aab07ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:05:37.640 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:37.640+0000 7f5b48ff9640 1 --2- 172.21.15.29:0/2867652856 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f5b380a2a10 0x7f5b380a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:05:37.640 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:37.640+0000 7f5b54e2e740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5574aab08410 0x5574aab156f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:05:37.640 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:37.640+0000 7f5b54e2e740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5574aab15c30 0x5574aab18020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:05:37.640 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:37.640+0000 7f5b54e2e740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x5574aaa6ca40 con 0x5574aab08410 2024-03-16T12:05:37.640 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:37.640+0000 7f5b54e2e740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x5574aaa15920 con 0x5574aab15c30 2024-03-16T12:05:37.640 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:37.640+0000 7f5b54e2e740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5574aaa68ae0 con 0x5574aab07af0 2024-03-16T12:05:37.640 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:37.640+0000 7f5b4b7fe640 1 --2- 172.21.15.29:0/2867652856 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f5b380a2a10 0x7f5b380a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:05:37.640 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:37.640+0000 7f5b4affd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5574aab08410 0x5574aab156f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:05:37.640 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:37.641+0000 7f5b4bfff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5574aab07af0 0x5574aab07ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:05:37.640 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:37.641+0000 7f5b4bfff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5574aab07af0 0x5574aab07ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.29:52736/0 (socket says 172.21.15.29:52736) 2024-03-16T12:05:37.640 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:37.641+0000 7f5b4affd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5574aab08410 0x5574aab156f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43588/0 (socket says 172.21.15.29:43588) 2024-03-16T12:05:37.640 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:37.641+0000 7f5b4bfff640 1 -- 172.21.15.29:0/4141107087 learned_addr learned my addr 172.21.15.29:0/4141107087 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T12:05:37.640 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:37.641+0000 7f5b4bfff640 1 --2- 172.21.15.29:0/4141107087 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5574aab15c30 0x5574aab18020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:05:37.641 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:37.641+0000 7f5b48ff9640 1 -- 172.21.15.29:0/2867652856 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 8556+0+0 (secure 0 0 0) 0x7f5b3c0cf120 con 0x7f5b4c100930 2024-03-16T12:05:37.641 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:37.641+0000 7f5b4bfff640 1 -- 172.21.15.29:0/4141107087 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5574aab15c30 msgr2=0x5574aab18020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:05:37.641 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:37.641+0000 7f5b4bfff640 1 --2- 172.21.15.29:0/4141107087 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5574aab15c30 0x5574aab18020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:05:37.641 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:37.641+0000 7f5b4bfff640 1 -- 172.21.15.29:0/4141107087 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5574aab08410 msgr2=0x5574aab156f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:05:37.641 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:37.641+0000 7f5b4bfff640 1 --2- 172.21.15.29:0/4141107087 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5574aab08410 0x5574aab156f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:05:37.641 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:37.641+0000 7f5b4bfff640 1 -- 172.21.15.29:0/4141107087 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5574aaa2d460 con 0x5574aab07af0 2024-03-16T12:05:37.641 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:37.641+0000 7f5b4b7fe640 1 --2- 172.21.15.29:0/2867652856 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f5b380a2a10 0x7f5b380a4ed0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f5b3c0cc9a0 tx=0x7f5b3c018420 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:05:37.641 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:37.641+0000 7f5b4bfff640 1 --2- 172.21.15.29:0/4141107087 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5574aab07af0 0x5574aab07ed0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f5b44006010 tx=0x7f5b44009360 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c57c158414e41f37 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:05:37.641 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:37.641+0000 7f5b41ffb640 1 -- 172.21.15.29:0/4141107087 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b44039040 con 0x5574aab07af0 2024-03-16T12:05:37.641 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:37.641+0000 7f5b41ffb640 1 -- 172.21.15.29:0/4141107087 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5b44003970 con 0x5574aab07af0 2024-03-16T12:05:37.641 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:37.641+0000 7f5b41ffb640 1 -- 172.21.15.29:0/4141107087 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b44032e90 con 0x5574aab07af0 2024-03-16T12:05:37.641 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:37.642+0000 7f5b54e2e740 1 -- 172.21.15.29:0/4141107087 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5574aab07af0 msgr2=0x5574aab07ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:05:37.641 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:37.642+0000 7f5b54e2e740 1 --2- 172.21.15.29:0/4141107087 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5574aab07af0 0x5574aab07ed0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f5b44006010 tx=0x7f5b44009360 comp rx=0 tx=0).stop 2024-03-16T12:05:37.641 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:37.642+0000 7f5b54e2e740 1 -- 172.21.15.29:0/4141107087 shutdown_connections 2024-03-16T12:05:37.641 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:37.642+0000 7f5b54e2e740 1 --2- 172.21.15.29:0/4141107087 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5574aab15c30 0x5574aab18020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:05:37.641 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:37.642+0000 7f5b54e2e740 1 --2- 172.21.15.29:0/4141107087 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5574aab08410 0x5574aab156f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:05:37.642 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:37.642+0000 7f5b54e2e740 1 --2- 172.21.15.29:0/4141107087 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5574aab07af0 0x5574aab07ed0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:05:37.642 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:37.642+0000 7f5b54e2e740 1 -- 172.21.15.29:0/4141107087 >> 172.21.15.29:0/4141107087 conn(0x5574aab02a50 msgr2=0x5574aab02e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:05:37.642 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:37.642+0000 7f5b54e2e740 1 -- 172.21.15.29:0/4141107087 shutdown_connections 2024-03-16T12:05:37.642 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:37.642+0000 7f5b54e2e740 1 -- 172.21.15.29:0/4141107087 wait complete. 2024-03-16T12:05:37.642 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:37.642+0000 7f5b54e2e740 1 -- start start 2024-03-16T12:05:37.642 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:37.642+0000 7f5b54e2e740 1 Processor -- start 2024-03-16T12:05:37.642 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:37.643+0000 7f5b54e2e740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5574aab07af0 0x5574aab12bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:05:37.642 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:37.643+0000 7f5b54e2e740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5574aab08410 0x5574aab13130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:05:37.642 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:37.643+0000 7f5b54e2e740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5574aab15c30 0x5574aab13670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:05:37.642 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:37.643+0000 7f5b54e2e740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x5574aab187a0 con 0x5574aab08410 2024-03-16T12:05:37.642 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:37.643+0000 7f5b54e2e740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x5574aaa28d70 con 0x5574aab07af0 2024-03-16T12:05:37.642 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:37.643+0000 7f5b54e2e740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5574aab18920 con 0x5574aab15c30 2024-03-16T12:05:37.642 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:37.643+0000 7f5b54e2e740 10 client.0 _ll_register_callbacks cb 0x5574aa780a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T12:05:37.642 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:37.643+0000 7f5b4affd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5574aab08410 0x5574aab13130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:05:37.643 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:37.643+0000 7f5b4bfff640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5574aab07af0 0x5574aab12bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:05:37.643 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:37.643+0000 7f5b4affd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5574aab08410 0x5574aab13130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43600/0 (socket says 172.21.15.29:43600) 2024-03-16T12:05:37.643 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:37.643+0000 7f5b4affd640 1 -- 172.21.15.29:0/966269990 learned_addr learned my addr 172.21.15.29:0/966269990 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T12:05:37.643 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:37.643+0000 7f5b4bfff640 1 --2- 172.21.15.29:0/966269990 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5574aab15c30 0x5574aab13670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:05:37.643 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:37.643+0000 7f5b4affd640 1 -- 172.21.15.29:0/966269990 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5574aab15c30 msgr2=0x5574aab13670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:05:37.643 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:37.643+0000 7f5b4affd640 1 --2- 172.21.15.29:0/966269990 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5574aab15c30 0x5574aab13670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:05:37.643 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:37.643+0000 7f5b4affd640 1 -- 172.21.15.29:0/966269990 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5574aab07af0 msgr2=0x5574aab12bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:05:37.643 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:37.643+0000 7f5b4affd640 1 --2- 172.21.15.29:0/966269990 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5574aab07af0 0x5574aab12bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:05:37.643 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:37.643+0000 7f5b4affd640 1 -- 172.21.15.29:0/966269990 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5574aa781210 con 0x5574aab08410 2024-03-16T12:05:37.643 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:37.643+0000 7f5b4affd640 1 --2- 172.21.15.29:0/966269990 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5574aab08410 0x5574aab13130 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f5b34012770 tx=0x7f5b34012c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f830095bd2b0994f server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:05:37.643 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:37.643+0000 7f5b2affd640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-16T12:05:37.643 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:37.644+0000 7f5b2affd640 1 -- 172.21.15.29:0/966269990 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b3400f8c0 con 0x5574aab08410 2024-03-16T12:05:37.643 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:37.644+0000 7f5b54e2e740 1 -- 172.21.15.29:0/966269990 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5b44003970 con 0x5574aab08410 2024-03-16T12:05:37.643 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:37.644+0000 7f5b2affd640 1 -- 172.21.15.29:0/966269990 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(110..110 src has 1..110) ==== 8556+0+0 (secure 0 0 0) 0x7f5b3400fa60 con 0x5574aab08410 2024-03-16T12:05:37.643 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:37.644+0000 7f5b2affd640 1 -- 172.21.15.29:0/966269990 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f5b3400fc60 con 0x5574aab08410 2024-03-16T12:05:37.643 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:37.644+0000 7f5b54e2e740 10 client.16623 fetch_fsmap learned FSMap version 31 2024-03-16T12:05:37.643 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:37.644+0000 7f5b54e2e740 1 -- 172.21.15.29:0/966269990 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5574aa84d620 con 0x5574aab08410 2024-03-16T12:05:37.644 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:37.644+0000 7f5b2affd640 1 -- 172.21.15.29:0/966269990 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5b44003970 con 0x5574aab08410 2024-03-16T12:05:37.644 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:37.644+0000 7f5b2affd640 1 -- 172.21.15.29:0/966269990 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 31) ==== 8346+0+0 (secure 0 0 0) 0x7f5b34024420 con 0x5574aab08410 2024-03-16T12:05:37.644 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:37.644+0000 7f5b54e2e740 10 client.16623 fetch_fsmap finished waiting for FSMap version 31 2024-03-16T12:05:37.644 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:37.644+0000 7f5b54e2e740 10 client.16623 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T12:05:37.644 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:37.644+0000 7f5b54e2e740 20 client.16623 populate_metadata read hostname 'smithi029' 2024-03-16T12:05:37.644 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:37.644+0000 7f5b54e2e740 1 --2- 172.21.15.29:0/966269990 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x5574aab25da0 0x5574aab28260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:05:37.644 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:37.644+0000 7f5b54e2e740 4 client.16623 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T12:05:37.644 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:37.644+0000 7f5b54e2e740 1 -- 172.21.15.29:0/966269990 --> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5574aaa79f80 con 0x5574aab25da0 2024-03-16T12:05:37.644 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:37.644+0000 7f5b4bfff640 1 --2- 172.21.15.29:0/966269990 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x5574aab25da0 0x5574aab28260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:05:37.644 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:37.645+0000 7f5b2affd640 10 client.16623 ms_handle_connect on v2:172.21.15.152:6838/1548731833 2024-03-16T12:05:37.644 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:37.645+0000 7f5b4bfff640 1 --2- 172.21.15.29:0/966269990 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x5574aab25da0 0x5574aab28260 crc :-1 s=READY pgs=170 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b49f5aca4188a480 server_cookie=ff62cff5e2fed735 in_seq=0 out_seq=0 2024-03-16T12:05:37.645 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:37.646+0000 7f5b2affd640 1 -- 172.21.15.29:0/966269990 <== mds.0 v2:172.21.15.152:6838/1548731833 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5574aaa79f80 con 0x5574aab25da0 2024-03-16T12:05:37.645 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:37.646+0000 7f5b2affd640 10 client.16623 handle_command_reply: tid=0 2024-03-16T12:05:37.645 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:37.646+0000 7f5b54e2e740 1 client.16623 shutdown 2024-03-16T12:05:37.645 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:37.647+0000 7f5b54e2e740 1 -- 172.21.15.29:0/966269990 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5574aab08410 msgr2=0x5574aab13130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:05:37.645 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:37.647+0000 7f5b54e2e740 1 --2- 172.21.15.29:0/966269990 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5574aab08410 0x5574aab13130 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f5b34012770 tx=0x7f5b34012c30 comp rx=0 tx=0).stop 2024-03-16T12:05:37.646 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:37.647+0000 7f5b54e2e740 1 -- 172.21.15.29:0/966269990 shutdown_connections 2024-03-16T12:05:37.646 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:37.647+0000 7f5b54e2e740 1 --2- 172.21.15.29:0/966269990 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5574aab15c30 0x5574aab13670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:05:37.646 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:37.647+0000 7f5b54e2e740 1 --2- 172.21.15.29:0/966269990 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5574aab08410 0x5574aab13130 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:05:37.646 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:37.647+0000 7f5b54e2e740 1 --2- 172.21.15.29:0/966269990 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x5574aab25da0 0x5574aab28260 crc :-1 s=READY pgs=170 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:05:37.646 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:37.647+0000 7f5b54e2e740 1 --2- 172.21.15.29:0/966269990 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5574aab07af0 0x5574aab12bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:05:37.646 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:37.647+0000 7f5b54e2e740 1 -- 172.21.15.29:0/966269990 >> 172.21.15.29:0/966269990 conn(0x5574aab02a50 msgr2=0x5574aab047a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:05:37.646 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:37.647+0000 7f5b2affd640 0 client.16623 ms_handle_reset on v2:172.21.15.152:6838/1548731833 2024-03-16T12:05:37.646 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:37.647+0000 7f5b54e2e740 1 -- 172.21.15.29:0/966269990 shutdown_connections 2024-03-16T12:05:37.646 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:37.647+0000 7f5b54e2e740 1 -- 172.21.15.29:0/966269990 wait complete. 2024-03-16T12:05:37.646 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:37.647+0000 7f5b54e2e740 20 client.16623 trim_cache size 0 max 16384 2024-03-16T12:05:37.657 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:37.662+0000 7f5b54e2e740 1 Processor -- start 2024-03-16T12:05:37.657 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:37.662+0000 7f5b54e2e740 1 -- start start 2024-03-16T12:05:37.657 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:37.663+0000 7f5b54e2e740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5574aab07af0 0x5574aab15320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:05:37.657 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:37.663+0000 7f5b54e2e740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5574aab08410 0x5574aab21540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:05:37.658 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:37.663+0000 7f5b54e2e740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5574aab15c30 0x5574aab2db80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:05:37.658 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:37.663+0000 7f5b54e2e740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f5b4c1adba0 con 0x5574aab15c30 2024-03-16T12:05:37.658 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:37.663+0000 7f5b54e2e740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f5b4c1ab410 con 0x5574aab08410 2024-03-16T12:05:37.658 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:37.663+0000 7f5b54e2e740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f5b4c1a9280 con 0x5574aab07af0 2024-03-16T12:05:37.658 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:37.663+0000 7f5b4bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5574aab15c30 0x5574aab2db80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:05:37.658 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:37.663+0000 7f5b4bfff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5574aab07af0 0x5574aab15320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:05:37.658 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:37.663+0000 7f5b4affd640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5574aab08410 0x5574aab21540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:05:37.658 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:37.663+0000 7f5b4bfff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5574aab07af0 0x5574aab15320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.29:52752/0 (socket says 172.21.15.29:52752) 2024-03-16T12:05:37.658 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:37.663+0000 7f5b4affd640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5574aab08410 0x5574aab21540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.29:60380/0 (socket says 172.21.15.29:60380) 2024-03-16T12:05:37.658 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:37.663+0000 7f5b4bfff640 1 -- 172.21.15.29:0/2364486666 learned_addr learned my addr 172.21.15.29:0/2364486666 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T12:05:37.658 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:37.663+0000 7f5b4bfff640 1 -- 172.21.15.29:0/2364486666 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5574aab08410 msgr2=0x5574aab21540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:05:37.658 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:37.663+0000 7f5b4bfff640 1 --2- 172.21.15.29:0/2364486666 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5574aab08410 0x5574aab21540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:05:37.658 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:37.663+0000 7f5b4bfff640 1 -- 172.21.15.29:0/2364486666 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5574aab15c30 msgr2=0x5574aab2db80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:05:37.658 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:37.663+0000 7f5b4bfff640 1 --2- 172.21.15.29:0/2364486666 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5574aab15c30 0x5574aab2db80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:05:37.658 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:37.663+0000 7f5b4bfff640 1 -- 172.21.15.29:0/2364486666 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5574aab21a80 con 0x5574aab07af0 2024-03-16T12:05:37.659 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:37.663+0000 7f5b4bfff640 1 --2- 172.21.15.29:0/2364486666 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5574aab07af0 0x5574aab15320 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f5b44032bc0 tx=0x7f5b44009360 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b60c6a1179e0657c server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:05:37.659 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:37.663+0000 7f5b417fa640 1 -- 172.21.15.29:0/2364486666 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b44039040 con 0x5574aab07af0 2024-03-16T12:05:37.659 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:37.664+0000 7f5b417fa640 1 -- 172.21.15.29:0/2364486666 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5b44003970 con 0x5574aab07af0 2024-03-16T12:05:37.659 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:37.664+0000 7f5b417fa640 1 -- 172.21.15.29:0/2364486666 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b44034440 con 0x5574aab07af0 2024-03-16T12:05:37.662 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:37.664+0000 7f5b54e2e740 1 -- 172.21.15.29:0/2364486666 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5574aab07af0 msgr2=0x5574aab15320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:05:37.662 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:37.664+0000 7f5b54e2e740 1 --2- 172.21.15.29:0/2364486666 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5574aab07af0 0x5574aab15320 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f5b44032bc0 tx=0x7f5b44009360 comp rx=0 tx=0).stop 2024-03-16T12:05:37.662 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:37.664+0000 7f5b54e2e740 1 -- 172.21.15.29:0/2364486666 shutdown_connections 2024-03-16T12:05:37.662 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:37.664+0000 7f5b54e2e740 1 --2- 172.21.15.29:0/2364486666 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5574aab15c30 0x5574aab2db80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:05:37.662 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:37.664+0000 7f5b54e2e740 1 --2- 172.21.15.29:0/2364486666 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5574aab08410 0x5574aab21540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:05:37.662 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:37.664+0000 7f5b54e2e740 1 --2- 172.21.15.29:0/2364486666 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5574aab07af0 0x5574aab15320 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:05:37.662 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:37.664+0000 7f5b54e2e740 1 -- 172.21.15.29:0/2364486666 >> 172.21.15.29:0/2364486666 conn(0x5574aab02a50 msgr2=0x5574aab046c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:05:37.663 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:37.664+0000 7f5b54e2e740 1 -- 172.21.15.29:0/2364486666 shutdown_connections 2024-03-16T12:05:37.663 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:37.664+0000 7f5b54e2e740 1 -- 172.21.15.29:0/2364486666 wait complete. 2024-03-16T12:05:37.663 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:37.664+0000 7f5b54e2e740 1 -- start start 2024-03-16T12:05:37.663 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:37.665+0000 7f5b54e2e740 1 Processor -- start 2024-03-16T12:05:37.663 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:37.665+0000 7f5b54e2e740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5574aab07af0 0x5574aab1db70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:05:37.663 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:37.665+0000 7f5b54e2e740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5574aab08410 0x5574aab1ab40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:05:37.663 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:37.665+0000 7f5b54e2e740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5574aab15c30 0x5574aab1b080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:05:37.663 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:37.665+0000 7f5b54e2e740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f5b4c1a98d0 con 0x5574aab15c30 2024-03-16T12:05:37.663 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:37.665+0000 7f5b54e2e740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f5b4c1aad40 con 0x5574aab07af0 2024-03-16T12:05:37.663 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:37.665+0000 7f5b54e2e740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f5b4c1a9fa0 con 0x5574aab08410 2024-03-16T12:05:37.663 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:37.665+0000 7f5b54e2e740 10 client.0 _ll_register_callbacks cb 0x7f5b34009910 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T12:05:37.663 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:37.665+0000 7f5b4bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5574aab15c30 0x5574aab1b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:05:37.663 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:37.665+0000 7f5b4bfff640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5574aab07af0 0x5574aab1db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:05:37.663 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:37.665+0000 7f5b4affd640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5574aab08410 0x5574aab1ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:05:37.663 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:37.665+0000 7f5b4bfff640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5574aab07af0 0x5574aab1db70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.29:60392/0 (socket says 172.21.15.29:60392) 2024-03-16T12:05:37.663 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:37.665+0000 7f5b4bfff640 1 -- 172.21.15.29:0/739655127 learned_addr learned my addr 172.21.15.29:0/739655127 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T12:05:37.663 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:37.665+0000 7f5b4affd640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5574aab08410 0x5574aab1ab40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.29:52758/0 (socket says 172.21.15.29:52758) 2024-03-16T12:05:37.664 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:37.665+0000 7f5b4bfff640 1 -- 172.21.15.29:0/739655127 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5574aab08410 msgr2=0x5574aab1ab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:05:37.664 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:37.665+0000 7f5b4bfff640 1 --2- 172.21.15.29:0/739655127 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5574aab08410 0x5574aab1ab40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:05:37.664 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:37.665+0000 7f5b4bfff640 1 -- 172.21.15.29:0/739655127 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5574aab15c30 msgr2=0x5574aab1b080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:05:37.664 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:37.665+0000 7f5b4bfff640 1 --2- 172.21.15.29:0/739655127 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5574aab15c30 0x5574aab1b080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:05:37.664 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:37.665+0000 7f5b4bfff640 1 -- 172.21.15.29:0/739655127 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5574aab1e1e0 con 0x5574aab07af0 2024-03-16T12:05:37.664 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:37.665+0000 7f5b4affd640 1 --2- 172.21.15.29:0/739655127 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5574aab08410 0x5574aab1ab40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:05:37.664 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:37.666+0000 7f5b4bfff640 1 --2- 172.21.15.29:0/739655127 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5574aab07af0 0x5574aab1db70 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f5b44009ca0 tx=0x7f5b44000f40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7785a4866fd42539 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:05:37.664 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:37.666+0000 7f5b2affd640 10 client.0 ms_handle_connect on v2:172.21.15.152:3300/0 2024-03-16T12:05:37.664 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:37.666+0000 7f5b2affd640 1 -- 172.21.15.29:0/739655127 <== mon.2 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b44039070 con 0x5574aab07af0 2024-03-16T12:05:37.664 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:37.666+0000 7f5b2affd640 1 -- 172.21.15.29:0/739655127 <== mon.2 v2:172.21.15.152:3300/0 2 ==== osd_map(110..110 src has 1..110) ==== 8556+0+0 (secure 0 0 0) 0x7f5b44054740 con 0x5574aab07af0 2024-03-16T12:05:37.664 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:37.666+0000 7f5b54e2e740 1 -- 172.21.15.29:0/739655127 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5b44003970 con 0x5574aab07af0 2024-03-16T12:05:37.664 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:37.666+0000 7f5b2affd640 1 -- 172.21.15.29:0/739655127 <== mon.2 v2:172.21.15.152:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f5b44003b20 con 0x5574aab07af0 2024-03-16T12:05:37.664 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:37.666+0000 7f5b54e2e740 10 client.26429 fetch_fsmap learned FSMap version 31 2024-03-16T12:05:37.664 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:37.666+0000 7f5b54e2e740 1 -- 172.21.15.29:0/739655127 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5574aab8cc10 con 0x5574aab07af0 2024-03-16T12:05:37.664 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:37.666+0000 7f5b2affd640 1 -- 172.21.15.29:0/739655127 <== mon.2 v2:172.21.15.152:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5b44003970 con 0x5574aab07af0 2024-03-16T12:05:37.664 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:37.666+0000 7f5b2affd640 1 -- 172.21.15.29:0/739655127 <== mon.2 v2:172.21.15.152:3300/0 5 ==== fsmap(e 31) ==== 8346+0+0 (secure 0 0 0) 0x7f5b440344d0 con 0x5574aab07af0 2024-03-16T12:05:37.665 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:37.666+0000 7f5b54e2e740 10 client.26429 fetch_fsmap finished waiting for FSMap version 31 2024-03-16T12:05:37.665 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:37.666+0000 7f5b54e2e740 10 client.26429 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T12:05:37.665 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:37.667+0000 7f5b54e2e740 20 client.26429 populate_metadata read hostname 'smithi029' 2024-03-16T12:05:37.665 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:37.667+0000 7f5b54e2e740 1 --2- 172.21.15.29:0/739655127 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x5574aab8ce30 0x5574aab8f220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:05:37.665 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:37.667+0000 7f5b54e2e740 4 client.26429 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-16T12:05:37.665 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:37.667+0000 7f5b54e2e740 1 -- 172.21.15.29:0/739655127 --> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5574aaa742d0 con 0x5574aab8ce30 2024-03-16T12:05:37.665 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:37.667+0000 7f5b4affd640 1 --2- 172.21.15.29:0/739655127 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x5574aab8ce30 0x5574aab8f220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:05:37.665 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:37.667+0000 7f5b2affd640 10 client.26429 ms_handle_connect on v2:172.21.15.152:6838/1548731833 2024-03-16T12:05:37.665 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:37.667+0000 7f5b4affd640 1 --2- 172.21.15.29:0/739655127 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x5574aab8ce30 0x5574aab8f220 crc :-1 s=READY pgs=171 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b926a997cdf8990c server_cookie=1c405a4f98d169c2 in_seq=0 out_seq=0 2024-03-16T12:05:38.185 INFO:tasks.workunit.client.0.smithi029.stdout: 10 23491 17.70 MB/sec warmup 82 sec latency 879.149 ms 2024-03-16T12:05:38.243 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:38.249+0000 7f5b2affd640 1 -- 172.21.15.29:0/739655127 <== mds.0 v2:172.21.15.152:6838/1548731833 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5574aaa742d0 con 0x5574aab8ce30 2024-03-16T12:05:38.243 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:38.249+0000 7f5b2affd640 10 client.26429 handle_command_reply: tid=0 2024-03-16T12:05:38.243 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:38.249+0000 7f5b54e2e740 1 client.26429 shutdown 2024-03-16T12:05:38.244 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:38.249+0000 7f5b54e2e740 1 -- 172.21.15.29:0/739655127 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5574aab07af0 msgr2=0x5574aab1db70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:05:38.244 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:38.249+0000 7f5b54e2e740 1 --2- 172.21.15.29:0/739655127 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5574aab07af0 0x5574aab1db70 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f5b44009ca0 tx=0x7f5b44000f40 comp rx=0 tx=0).stop 2024-03-16T12:05:38.244 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:38.249+0000 7f5b54e2e740 1 -- 172.21.15.29:0/739655127 shutdown_connections 2024-03-16T12:05:38.244 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:38.249+0000 7f5b54e2e740 1 --2- 172.21.15.29:0/739655127 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5574aab15c30 0x5574aab1b080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:05:38.244 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:38.249+0000 7f5b54e2e740 1 --2- 172.21.15.29:0/739655127 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5574aab08410 0x5574aab1ab40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:05:38.244 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:38.249+0000 7f5b54e2e740 1 --2- 172.21.15.29:0/739655127 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x5574aab8ce30 0x5574aab8f220 crc :-1 s=READY pgs=171 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:05:38.244 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:38.249+0000 7f5b54e2e740 1 --2- 172.21.15.29:0/739655127 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5574aab07af0 0x5574aab1db70 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:05:38.244 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:38.249+0000 7f5b2affd640 0 client.26429 ms_handle_reset on v2:172.21.15.152:6838/1548731833 2024-03-16T12:05:38.244 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:38.249+0000 7f5b54e2e740 1 -- 172.21.15.29:0/739655127 >> 172.21.15.29:0/739655127 conn(0x5574aab02a50 msgr2=0x5574aab01320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:05:38.244 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:38.249+0000 7f5b54e2e740 1 -- 172.21.15.29:0/739655127 shutdown_connections 2024-03-16T12:05:38.244 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:38.249+0000 7f5b54e2e740 1 -- 172.21.15.29:0/739655127 wait complete. 2024-03-16T12:05:38.244 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:38.249+0000 7f5b54e2e740 20 client.26429 trim_cache size 0 max 16384 2024-03-16T12:05:38.245 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:{ 2024-03-16T12:05:38.245 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "message": "", 2024-03-16T12:05:38.245 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "return_code": 0 2024-03-16T12:05:38.245 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:} 2024-03-16T12:05:38.245 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:38.250+0000 7f5b52244640 1 -- 172.21.15.29:0/2867652856 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f5b380a2a10 msgr2=0x7f5b380a4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:05:38.246 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:38.250+0000 7f5b52244640 1 --2- 172.21.15.29:0/2867652856 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f5b380a2a10 0x7f5b380a4ed0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f5b3c0cc9a0 tx=0x7f5b3c018420 comp rx=0 tx=0).stop 2024-03-16T12:05:38.246 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:38.250+0000 7f5b52244640 1 -- 172.21.15.29:0/2867652856 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5b4c100930 msgr2=0x7f5b4c198090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:05:38.246 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:38.250+0000 7f5b52244640 1 --2- 172.21.15.29:0/2867652856 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5b4c100930 0x7f5b4c198090 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f5b3c00e9f0 tx=0x7f5b3c00eeb0 comp rx=0 tx=0).stop 2024-03-16T12:05:38.246 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:38.250+0000 7f5b52244640 1 -- 172.21.15.29:0/2867652856 shutdown_connections 2024-03-16T12:05:38.246 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:38.250+0000 7f5b52244640 1 --2- 172.21.15.29:0/2867652856 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f5b380a2a10 0x7f5b380a4ed0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:05:38.246 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:38.250+0000 7f5b52244640 1 --2- 172.21.15.29:0/2867652856 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5b4c105680 0x7f5b4c19c900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:05:38.246 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:38.250+0000 7f5b52244640 1 --2- 172.21.15.29:0/2867652856 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5b4c1012f0 0x7f5b4c1985d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:05:38.246 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:38.250+0000 7f5b52244640 1 --2- 172.21.15.29:0/2867652856 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5b4c100930 0x7f5b4c198090 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:05:38.246 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:38.250+0000 7f5b52244640 1 -- 172.21.15.29:0/2867652856 >> 172.21.15.29:0/2867652856 conn(0x7f5b4c077530 msgr2=0x7f5b4c0690e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:05:38.246 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:38.250+0000 7f5b52244640 1 -- 172.21.15.29:0/2867652856 shutdown_connections 2024-03-16T12:05:38.246 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:38.250+0000 7f5b52244640 1 -- 172.21.15.29:0/2867652856 wait complete. 2024-03-16T12:05:38.581 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:05:38 smithi029 ceph-mon[31975]: pgmap v466: 97 pgs: 97 active+clean; 321 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 MiB/s rd, 3.2 MiB/s wr, 485 op/s 2024-03-16T12:05:38.621 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:05:38 smithi152 ceph-mon[36291]: pgmap v466: 97 pgs: 97 active+clean; 321 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 MiB/s rd, 3.2 MiB/s wr, 485 op/s 2024-03-16T12:05:38.789 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:05:38 smithi154 ceph-mon[36320]: pgmap v466: 97 pgs: 97 active+clean; 321 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 MiB/s rd, 3.2 MiB/s wr, 485 op/s 2024-03-16T12:05:39.185 INFO:tasks.workunit.client.0.smithi029.stdout: 10 23658 17.53 MB/sec warmup 83 sec latency 1879.190 ms 2024-03-16T12:05:40.185 INFO:tasks.workunit.client.0.smithi029.stdout: 10 23764 17.37 MB/sec warmup 84 sec latency 2879.235 ms 2024-03-16T12:05:40.581 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:05:40 smithi029 ceph-mon[31975]: pgmap v467: 97 pgs: 97 active+clean; 321 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 2.1 MiB/s wr, 194 op/s 2024-03-16T12:05:40.621 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:05:40 smithi152 ceph-mon[36291]: pgmap v467: 97 pgs: 97 active+clean; 321 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 2.1 MiB/s wr, 194 op/s 2024-03-16T12:05:40.789 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:05:40 smithi154 ceph-mon[36320]: pgmap v467: 97 pgs: 97 active+clean; 321 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 2.1 MiB/s wr, 194 op/s 2024-03-16T12:05:41.185 INFO:tasks.workunit.client.0.smithi029.stdout: 10 23928 17.21 MB/sec warmup 85 sec latency 3879.261 ms 2024-03-16T12:05:42.185 INFO:tasks.workunit.client.0.smithi029.stdout: 10 24198 17.15 MB/sec warmup 86 sec latency 4568.709 ms 2024-03-16T12:05:42.581 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:05:42 smithi029 ceph-mon[31975]: pgmap v468: 97 pgs: 97 active+clean; 310 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 MiB/s rd, 3.8 MiB/s wr, 454 op/s 2024-03-16T12:05:42.621 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:05:42 smithi152 ceph-mon[36291]: pgmap v468: 97 pgs: 97 active+clean; 310 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 MiB/s rd, 3.8 MiB/s wr, 454 op/s 2024-03-16T12:05:42.788 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:05:42 smithi154 ceph-mon[36320]: pgmap v468: 97 pgs: 97 active+clean; 310 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 MiB/s rd, 3.8 MiB/s wr, 454 op/s 2024-03-16T12:05:43.185 INFO:tasks.workunit.client.0.smithi029.stdout: 10 24518 17.16 MB/sec warmup 87 sec latency 74.846 ms 2024-03-16T12:05:44.185 INFO:tasks.workunit.client.0.smithi029.stdout: 10 24978 17.17 MB/sec warmup 88 sec latency 241.058 ms 2024-03-16T12:05:44.582 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:05:44 smithi029 ceph-mon[31975]: pgmap v469: 97 pgs: 97 active+clean; 310 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 3.1 MiB/s wr, 397 op/s 2024-03-16T12:05:44.590 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:05:44 smithi154 ceph-mon[36320]: pgmap v469: 97 pgs: 97 active+clean; 310 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 3.1 MiB/s wr, 397 op/s 2024-03-16T12:05:44.621 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:05:44 smithi152 ceph-mon[36291]: pgmap v469: 97 pgs: 97 active+clean; 310 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 3.1 MiB/s wr, 397 op/s 2024-03-16T12:05:45.185 INFO:tasks.workunit.client.0.smithi029.stdout: 10 25305 17.16 MB/sec warmup 89 sec latency 290.571 ms 2024-03-16T12:05:46.185 INFO:tasks.workunit.client.0.smithi029.stdout: 10 25728 17.26 MB/sec warmup 90 sec latency 249.463 ms 2024-03-16T12:05:46.581 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:05:46 smithi029 ceph-mon[31975]: pgmap v470: 97 pgs: 97 active+clean; 310 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 3.1 MiB/s wr, 390 op/s 2024-03-16T12:05:46.621 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:05:46 smithi152 ceph-mon[36291]: pgmap v470: 97 pgs: 97 active+clean; 310 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 3.1 MiB/s wr, 390 op/s 2024-03-16T12:05:46.788 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:05:46 smithi154 ceph-mon[36320]: pgmap v470: 97 pgs: 97 active+clean; 310 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 3.1 MiB/s wr, 390 op/s 2024-03-16T12:05:47.185 INFO:tasks.workunit.client.0.smithi029.stdout: 10 26184 17.44 MB/sec warmup 91 sec latency 216.606 ms 2024-03-16T12:05:48.185 INFO:tasks.workunit.client.0.smithi029.stdout: 10 26461 17.45 MB/sec warmup 92 sec latency 219.846 ms 2024-03-16T12:05:48.581 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:05:48 smithi029 ceph-mon[31975]: pgmap v471: 97 pgs: 97 active+clean; 323 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 MiB/s rd, 5.4 MiB/s wr, 502 op/s 2024-03-16T12:05:48.621 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:05:48 smithi152 ceph-mon[36291]: pgmap v471: 97 pgs: 97 active+clean; 323 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 MiB/s rd, 5.4 MiB/s wr, 502 op/s 2024-03-16T12:05:48.788 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:05:48 smithi154 ceph-mon[36320]: pgmap v471: 97 pgs: 97 active+clean; 323 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 MiB/s rd, 5.4 MiB/s wr, 502 op/s 2024-03-16T12:05:49.185 INFO:tasks.workunit.client.0.smithi029.stdout: 10 26684 17.44 MB/sec warmup 93 sec latency 67.754 ms 2024-03-16T12:05:50.185 INFO:tasks.workunit.client.0.smithi029.stdout: 10 26896 17.43 MB/sec warmup 94 sec latency 92.372 ms 2024-03-16T12:05:50.581 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:05:50 smithi029 ceph-mon[31975]: pgmap v472: 97 pgs: 97 active+clean; 323 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 4.2 MiB/s wr, 405 op/s 2024-03-16T12:05:50.620 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:05:50 smithi152 ceph-mon[36291]: pgmap v472: 97 pgs: 97 active+clean; 323 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 4.2 MiB/s wr, 405 op/s 2024-03-16T12:05:50.788 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:05:50 smithi154 ceph-mon[36320]: pgmap v472: 97 pgs: 97 active+clean; 323 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 4.2 MiB/s wr, 405 op/s 2024-03-16T12:05:51.185 INFO:tasks.workunit.client.0.smithi029.stdout: 10 27033 17.27 MB/sec warmup 95 sec latency 110.809 ms 2024-03-16T12:05:51.333 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:51.338+0000 7f3b11f10640 1 -- 172.21.15.29:0/1638940687 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3b0c1029c0 msgr2=0x7f3b0c102da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:05:51.333 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:51.338+0000 7f3b11f10640 1 --2- 172.21.15.29:0/1638940687 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3b0c1029c0 0x7f3b0c102da0 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f3afc009900 tx=0x7f3afc02bc30 comp rx=0 tx=0).stop 2024-03-16T12:05:51.333 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:51.338+0000 7f3b11f10640 1 -- 172.21.15.29:0/1638940687 shutdown_connections 2024-03-16T12:05:51.334 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:51.338+0000 7f3b11f10640 1 --2- 172.21.15.29:0/1638940687 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3b0c10be30 0x7f3b0c10e2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:05:51.334 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:51.338+0000 7f3b11f10640 1 --2- 172.21.15.29:0/1638940687 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3b0c1032e0 0x7f3b0c10b7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:05:51.334 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:51.338+0000 7f3b11f10640 1 --2- 172.21.15.29:0/1638940687 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3b0c1029c0 0x7f3b0c102da0 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:05:51.334 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:51.338+0000 7f3b11f10640 1 -- 172.21.15.29:0/1638940687 >> 172.21.15.29:0/1638940687 conn(0x7f3b0c077810 msgr2=0x7f3b0c077c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:05:51.334 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:51.338+0000 7f3b11f10640 1 -- 172.21.15.29:0/1638940687 shutdown_connections 2024-03-16T12:05:51.334 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:51.338+0000 7f3b11f10640 1 -- 172.21.15.29:0/1638940687 wait complete. 2024-03-16T12:05:51.334 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:51.339+0000 7f3b11f10640 1 Processor -- start 2024-03-16T12:05:51.334 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:51.339+0000 7f3b11f10640 1 -- start start 2024-03-16T12:05:51.334 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:51.339+0000 7f3b11f10640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3b0c1029c0 0x7f3b0c198020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:05:51.334 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:51.339+0000 7f3b11f10640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3b0c1032e0 0x7f3b0c198560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:05:51.334 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:51.339+0000 7f3b11f10640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3b0c10be30 0x7f3b0c19c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:05:51.334 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:51.339+0000 7f3b11f10640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f3b0c111c00 con 0x7f3b0c10be30 2024-03-16T12:05:51.335 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:51.339+0000 7f3b11f10640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f3b0c111a80 con 0x7f3b0c1032e0 2024-03-16T12:05:51.335 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:51.339+0000 7f3b11f10640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f3b0c111d80 con 0x7f3b0c1029c0 2024-03-16T12:05:51.335 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:51.340+0000 7f3b0bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3b0c10be30 0x7f3b0c19c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:05:51.335 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:51.340+0000 7f3b0bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3b0c10be30 0x7f3b0c19c8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56474/0 (socket says 172.21.15.29:56474) 2024-03-16T12:05:51.335 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:51.340+0000 7f3b0bfff640 1 -- 172.21.15.29:0/699117973 learned_addr learned my addr 172.21.15.29:0/699117973 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T12:05:51.335 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:51.340+0000 7f3b0b7fe640 1 --2- 172.21.15.29:0/699117973 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3b0c1029c0 0x7f3b0c198020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:05:51.335 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:51.340+0000 7f3b02ffd640 1 --2- 172.21.15.29:0/699117973 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3b0c1032e0 0x7f3b0c198560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:05:51.335 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:51.340+0000 7f3b0bfff640 1 -- 172.21.15.29:0/699117973 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3b0c1029c0 msgr2=0x7f3b0c198020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:05:51.335 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:51.340+0000 7f3b0bfff640 1 --2- 172.21.15.29:0/699117973 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3b0c1029c0 0x7f3b0c198020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:05:51.335 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:51.340+0000 7f3b0bfff640 1 -- 172.21.15.29:0/699117973 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3b0c1032e0 msgr2=0x7f3b0c198560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:05:51.335 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:51.340+0000 7f3b0bfff640 1 --2- 172.21.15.29:0/699117973 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3b0c1032e0 0x7f3b0c198560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:05:51.335 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:51.340+0000 7f3b0bfff640 1 -- 172.21.15.29:0/699117973 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b0c19d010 con 0x7f3b0c10be30 2024-03-16T12:05:51.335 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:51.340+0000 7f3b0bfff640 1 --2- 172.21.15.29:0/699117973 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3b0c10be30 0x7f3b0c19c8e0 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f3b0400eb10 tx=0x7f3b0400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:05:51.335 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:51.340+0000 7f3b097fa640 1 -- 172.21.15.29:0/699117973 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b04004b20 con 0x7f3b0c10be30 2024-03-16T12:05:51.336 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:51.340+0000 7f3b11f10640 1 -- 172.21.15.29:0/699117973 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3b0c19d300 con 0x7f3b0c10be30 2024-03-16T12:05:51.336 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:51.340+0000 7f3b097fa640 1 -- 172.21.15.29:0/699117973 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3b04004cc0 con 0x7f3b0c10be30 2024-03-16T12:05:51.336 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:51.340+0000 7f3b097fa640 1 -- 172.21.15.29:0/699117973 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b04010760 con 0x7f3b0c10be30 2024-03-16T12:05:51.336 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:51.340+0000 7f3b11f10640 1 -- 172.21.15.29:0/699117973 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3b0c1a4b90 con 0x7f3b0c10be30 2024-03-16T12:05:51.341 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:51.342+0000 7f3b097fa640 1 -- 172.21.15.29:0/699117973 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7f3b04005000 con 0x7f3b0c10be30 2024-03-16T12:05:51.342 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:51.343+0000 7f3b14bdd740 1 Processor -- start 2024-03-16T12:05:51.342 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:51.343+0000 7f3b14bdd740 1 -- start start 2024-03-16T12:05:51.342 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:51.343+0000 7f3b14bdd740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55cf74662130 0x55cf74662510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:05:51.342 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:51.343+0000 7f3b14bdd740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cf74662a50 0x55cf7466b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:05:51.342 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:51.343+0000 7f3b14bdd740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cf7466b820 0x55cf7466dc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:05:51.342 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:51.343+0000 7f3b14bdd740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55cf745c6a40 con 0x55cf74662a50 2024-03-16T12:05:51.342 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:51.343+0000 7f3b14bdd740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x55cf7456f920 con 0x55cf74662130 2024-03-16T12:05:51.342 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:51.343+0000 7f3b14bdd740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55cf745c2ae0 con 0x55cf7466b820 2024-03-16T12:05:51.342 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:51.343+0000 7f3b0b7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cf74662a50 0x55cf7466b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:05:51.342 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:51.343+0000 7f3b02ffd640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55cf74662130 0x55cf74662510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:05:51.342 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:51.343+0000 7f3b02ffd640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55cf74662130 0x55cf74662510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.29:37230/0 (socket says 172.21.15.29:37230) 2024-03-16T12:05:51.342 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:51.343+0000 7f3b0b7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cf74662a50 0x55cf7466b2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56482/0 (socket says 172.21.15.29:56482) 2024-03-16T12:05:51.342 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:51.343+0000 7f3b02ffd640 1 -- 172.21.15.29:0/1290307223 learned_addr learned my addr 172.21.15.29:0/1290307223 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T12:05:51.343 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:51.343+0000 7f3b097fa640 1 --2- 172.21.15.29:0/699117973 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f3af80a2a10 0x7f3af80a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:05:51.343 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:51.343+0000 7f3b02ffd640 1 --2- 172.21.15.29:0/1290307223 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cf7466b820 0x55cf7466dc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:05:51.343 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:51.343+0000 7f3b0bfff640 1 --2- 172.21.15.29:0/699117973 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f3af80a2a10 0x7f3af80a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:05:51.343 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:51.343+0000 7f3b02ffd640 1 -- 172.21.15.29:0/1290307223 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cf7466b820 msgr2=0x55cf7466dc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:05:51.343 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:51.343+0000 7f3b02ffd640 1 --2- 172.21.15.29:0/1290307223 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cf7466b820 0x55cf7466dc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:05:51.343 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:51.343+0000 7f3b02ffd640 1 -- 172.21.15.29:0/1290307223 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cf74662a50 msgr2=0x55cf7466b2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:05:51.343 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:51.343+0000 7f3b02ffd640 1 --2- 172.21.15.29:0/1290307223 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cf74662a50 0x55cf7466b2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:05:51.343 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:51.343+0000 7f3b02ffd640 1 -- 172.21.15.29:0/1290307223 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cf74587460 con 0x55cf74662130 2024-03-16T12:05:51.343 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:51.343+0000 7f3b097fa640 1 -- 172.21.15.29:0/699117973 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 8556+0+0 (secure 0 0 0) 0x7f3b040cf420 con 0x7f3b0c10be30 2024-03-16T12:05:51.343 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:51.344+0000 7f3b02ffd640 1 --2- 172.21.15.29:0/1290307223 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55cf74662130 0x55cf74662510 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f3af4009fd0 tx=0x7f3af4005850 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6dfc0ff14706a7b9 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:05:51.343 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:51.344+0000 7f3b0bfff640 1 --2- 172.21.15.29:0/699117973 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f3af80a2a10 0x7f3af80a4ed0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f3b0409b760 tx=0x7f3b04086040 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:05:51.343 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:51.344+0000 7f3b01ffb640 1 -- 172.21.15.29:0/1290307223 <== mon.2 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3af4004050 con 0x55cf74662130 2024-03-16T12:05:51.343 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:51.344+0000 7f3b01ffb640 1 -- 172.21.15.29:0/1290307223 <== mon.2 v2:172.21.15.152:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3af4005500 con 0x55cf74662130 2024-03-16T12:05:51.343 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:51.344+0000 7f3b01ffb640 1 -- 172.21.15.29:0/1290307223 <== mon.2 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3af4004cd0 con 0x55cf74662130 2024-03-16T12:05:51.343 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:51.344+0000 7f3b14bdd740 1 -- 172.21.15.29:0/1290307223 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55cf74662130 msgr2=0x55cf74662510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:05:51.343 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:51.344+0000 7f3b14bdd740 1 --2- 172.21.15.29:0/1290307223 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55cf74662130 0x55cf74662510 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f3af4009fd0 tx=0x7f3af4005850 comp rx=0 tx=0).stop 2024-03-16T12:05:51.344 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:51.344+0000 7f3b14bdd740 1 -- 172.21.15.29:0/1290307223 shutdown_connections 2024-03-16T12:05:51.344 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:51.344+0000 7f3b14bdd740 1 --2- 172.21.15.29:0/1290307223 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cf7466b820 0x55cf7466dc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:05:51.344 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:51.344+0000 7f3b14bdd740 1 --2- 172.21.15.29:0/1290307223 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cf74662a50 0x55cf7466b2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:05:51.344 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:51.344+0000 7f3b14bdd740 1 --2- 172.21.15.29:0/1290307223 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55cf74662130 0x55cf74662510 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:05:51.344 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:51.344+0000 7f3b14bdd740 1 -- 172.21.15.29:0/1290307223 >> 172.21.15.29:0/1290307223 conn(0x55cf7465ca50 msgr2=0x55cf7465ce30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:05:51.344 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:51.344+0000 7f3b14bdd740 1 -- 172.21.15.29:0/1290307223 shutdown_connections 2024-03-16T12:05:51.344 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:51.345+0000 7f3b14bdd740 1 -- 172.21.15.29:0/1290307223 wait complete. 2024-03-16T12:05:51.344 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:51.345+0000 7f3b14bdd740 1 -- start start 2024-03-16T12:05:51.344 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:51.345+0000 7f3b14bdd740 1 Processor -- start 2024-03-16T12:05:51.344 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:51.345+0000 7f3b14bdd740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cf74662130 0x55cf74668830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:05:51.344 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:51.345+0000 7f3b14bdd740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55cf74662a50 0x55cf74668d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:05:51.344 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:51.345+0000 7f3b14bdd740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cf7466b820 0x55cf746692b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:05:51.344 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:51.345+0000 7f3b14bdd740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55cf7466e450 con 0x55cf7466b820 2024-03-16T12:05:51.344 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:51.345+0000 7f3b14bdd740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x55cf74582d70 con 0x55cf74662a50 2024-03-16T12:05:51.344 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:51.345+0000 7f3b14bdd740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55cf7466e5d0 con 0x55cf74662130 2024-03-16T12:05:51.344 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:51.345+0000 7f3b14bdd740 10 client.0 _ll_register_callbacks cb 0x55cf742daa00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T12:05:51.345 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:51.345+0000 7f3b02ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cf7466b820 0x55cf746692b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:05:51.345 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:51.345+0000 7f3b02ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cf7466b820 0x55cf746692b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56488/0 (socket says 172.21.15.29:56488) 2024-03-16T12:05:51.345 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:51.345+0000 7f3b02ffd640 1 -- 172.21.15.29:0/2085272834 learned_addr learned my addr 172.21.15.29:0/2085272834 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T12:05:51.345 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:51.345+0000 7f3b0b7fe640 1 --2- 172.21.15.29:0/2085272834 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55cf74662a50 0x55cf74668d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:05:51.345 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:51.345+0000 7f3b02ffd640 1 --2- 172.21.15.29:0/2085272834 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cf74662130 0x55cf74668830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:05:51.345 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:51.346+0000 7f3b02ffd640 1 -- 172.21.15.29:0/2085272834 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cf74662130 msgr2=0x55cf74668830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:05:51.345 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:51.346+0000 7f3b02ffd640 1 --2- 172.21.15.29:0/2085272834 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cf74662130 0x55cf74668830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:05:51.345 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:51.346+0000 7f3b02ffd640 1 -- 172.21.15.29:0/2085272834 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55cf74662a50 msgr2=0x55cf74668d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:05:51.345 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:51.346+0000 7f3b02ffd640 1 --2- 172.21.15.29:0/2085272834 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55cf74662a50 0x55cf74668d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:05:51.345 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:51.346+0000 7f3b02ffd640 1 -- 172.21.15.29:0/2085272834 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cf742db210 con 0x55cf7466b820 2024-03-16T12:05:51.345 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:51.346+0000 7f3b02ffd640 1 --2- 172.21.15.29:0/2085272834 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cf7466b820 0x55cf746692b0 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f3af4009e70 tx=0x7f3af40048b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5bbcf51d360e57ba server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:05:51.345 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:51.346+0000 7f3aeaffd640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-16T12:05:51.345 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:51.346+0000 7f3aeaffd640 1 -- 172.21.15.29:0/2085272834 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3af4004a50 con 0x55cf7466b820 2024-03-16T12:05:51.345 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:51.346+0000 7f3aeaffd640 1 -- 172.21.15.29:0/2085272834 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(110..110 src has 1..110) ==== 8556+0+0 (secure 0 0 0) 0x7f3af4004bf0 con 0x55cf7466b820 2024-03-16T12:05:51.345 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:51.346+0000 7f3b14bdd740 1 -- 172.21.15.29:0/2085272834 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3af4005500 con 0x55cf7466b820 2024-03-16T12:05:51.346 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:51.346+0000 7f3aeaffd640 1 -- 172.21.15.29:0/2085272834 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f3af4035050 con 0x55cf7466b820 2024-03-16T12:05:51.346 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:51.346+0000 7f3b14bdd740 10 client.16641 fetch_fsmap learned FSMap version 31 2024-03-16T12:05:51.346 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:51.346+0000 7f3b14bdd740 1 -- 172.21.15.29:0/2085272834 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55cf743a7620 con 0x55cf7466b820 2024-03-16T12:05:51.346 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:51.346+0000 7f3aeaffd640 1 -- 172.21.15.29:0/2085272834 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3af4005500 con 0x55cf7466b820 2024-03-16T12:05:51.346 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:51.346+0000 7f3aeaffd640 1 -- 172.21.15.29:0/2085272834 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 31) ==== 8346+0+0 (secure 0 0 0) 0x7f3af402d020 con 0x55cf7466b820 2024-03-16T12:05:51.346 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:51.347+0000 7f3b14bdd740 10 client.16641 fetch_fsmap finished waiting for FSMap version 31 2024-03-16T12:05:51.346 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:51.347+0000 7f3b14bdd740 10 client.16641 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T12:05:51.346 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:51.347+0000 7f3b14bdd740 20 client.16641 populate_metadata read hostname 'smithi029' 2024-03-16T12:05:51.346 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:51.347+0000 7f3b14bdd740 1 --2- 172.21.15.29:0/2085272834 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55cf7467b810 0x55cf7467dc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:05:51.346 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:51.347+0000 7f3b14bdd740 4 client.16641 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T12:05:51.346 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:51.347+0000 7f3b14bdd740 1 -- 172.21.15.29:0/2085272834 --> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cf745d3f80 con 0x55cf7467b810 2024-03-16T12:05:51.363 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:51.371+0000 7f3b0b7fe640 1 --2- 172.21.15.29:0/2085272834 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55cf7467b810 0x55cf7467dc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:05:51.363 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:51.371+0000 7f3b0b7fe640 1 --2- 172.21.15.29:0/2085272834 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55cf7467b810 0x55cf7467dc00 crc :-1 s=READY pgs=172 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5c4831331e0cf8a5 server_cookie=13b3117e53b71fed in_seq=0 out_seq=0 2024-03-16T12:05:51.364 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:51.371+0000 7f3aeaffd640 10 client.16641 ms_handle_connect on v2:172.21.15.152:6838/1548731833 2024-03-16T12:05:51.372 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:51.378+0000 7f3aeaffd640 1 -- 172.21.15.29:0/2085272834 <== mds.0 v2:172.21.15.152:6838/1548731833 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55cf745d3f80 con 0x55cf7467b810 2024-03-16T12:05:51.372 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:51.378+0000 7f3aeaffd640 10 client.16641 handle_command_reply: tid=0 2024-03-16T12:05:51.372 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:51.378+0000 7f3b14bdd740 1 client.16641 shutdown 2024-03-16T12:05:51.372 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:51.379+0000 7f3b14bdd740 1 -- 172.21.15.29:0/2085272834 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cf7466b820 msgr2=0x55cf746692b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:05:51.372 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:51.379+0000 7f3b14bdd740 1 --2- 172.21.15.29:0/2085272834 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cf7466b820 0x55cf746692b0 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f3af4009e70 tx=0x7f3af40048b0 comp rx=0 tx=0).stop 2024-03-16T12:05:51.372 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:51.379+0000 7f3b14bdd740 1 -- 172.21.15.29:0/2085272834 shutdown_connections 2024-03-16T12:05:51.372 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:51.379+0000 7f3b14bdd740 1 --2- 172.21.15.29:0/2085272834 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cf7466b820 0x55cf746692b0 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:05:51.372 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:51.379+0000 7f3b14bdd740 1 --2- 172.21.15.29:0/2085272834 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55cf7467b810 0x55cf7467dc00 crc :-1 s=READY pgs=172 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:05:51.372 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:51.379+0000 7f3b14bdd740 1 --2- 172.21.15.29:0/2085272834 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55cf74662a50 0x55cf74668d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:05:51.372 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:51.379+0000 7f3b14bdd740 1 --2- 172.21.15.29:0/2085272834 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cf74662130 0x55cf74668830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:05:51.372 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:51.379+0000 7f3b14bdd740 1 -- 172.21.15.29:0/2085272834 >> 172.21.15.29:0/2085272834 conn(0x55cf7465ca50 msgr2=0x55cf7465e7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:05:51.372 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:51.379+0000 7f3aeaffd640 0 client.16641 ms_handle_reset on v2:172.21.15.152:6838/1548731833 2024-03-16T12:05:51.372 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:51.379+0000 7f3b14bdd740 1 -- 172.21.15.29:0/2085272834 shutdown_connections 2024-03-16T12:05:51.373 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:51.379+0000 7f3b14bdd740 1 -- 172.21.15.29:0/2085272834 wait complete. 2024-03-16T12:05:51.373 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:51.379+0000 7f3b14bdd740 20 client.16641 trim_cache size 0 max 16384 2024-03-16T12:05:51.392 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:51.394+0000 7f3b14bdd740 1 Processor -- start 2024-03-16T12:05:51.392 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:51.394+0000 7f3b14bdd740 1 -- start start 2024-03-16T12:05:51.392 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:51.394+0000 7f3b14bdd740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cf74662130 0x55cf7466af60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:05:51.393 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:51.394+0000 7f3b14bdd740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cf74662a50 0x55cf74667990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:05:51.393 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:51.394+0000 7f3b14bdd740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55cf7466b820 0x55cf746771f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:05:51.393 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:51.394+0000 7f3b14bdd740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f3b0c1adbd0 con 0x55cf74662a50 2024-03-16T12:05:51.393 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:51.394+0000 7f3b14bdd740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f3b0c1ab440 con 0x55cf7466b820 2024-03-16T12:05:51.393 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:51.394+0000 7f3b14bdd740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f3b0c1a92b0 con 0x55cf74662130 2024-03-16T12:05:51.393 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:51.394+0000 7f3b0b7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cf74662a50 0x55cf74667990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:05:51.393 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:51.394+0000 7f3b0b7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cf74662a50 0x55cf74667990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56496/0 (socket says 172.21.15.29:56496) 2024-03-16T12:05:51.393 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:51.394+0000 7f3b0b7fe640 1 -- 172.21.15.29:0/3917485878 learned_addr learned my addr 172.21.15.29:0/3917485878 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T12:05:51.393 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:51.395+0000 7f3b02ffd640 1 --2- 172.21.15.29:0/3917485878 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cf74662130 0x55cf7466af60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:05:51.393 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:51.395+0000 7f3b02ffd640 1 --2- 172.21.15.29:0/3917485878 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55cf7466b820 0x55cf746771f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:05:51.393 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:51.395+0000 7f3b0b7fe640 1 -- 172.21.15.29:0/3917485878 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cf74662130 msgr2=0x55cf7466af60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:05:51.393 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:51.395+0000 7f3b0b7fe640 1 --2- 172.21.15.29:0/3917485878 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cf74662130 0x55cf7466af60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:05:51.393 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:51.395+0000 7f3b0b7fe640 1 -- 172.21.15.29:0/3917485878 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55cf7466b820 msgr2=0x55cf746771f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:05:51.393 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:51.395+0000 7f3b0b7fe640 1 --2- 172.21.15.29:0/3917485878 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55cf7466b820 0x55cf746771f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:05:51.394 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:51.395+0000 7f3b0b7fe640 1 -- 172.21.15.29:0/3917485878 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cf746616f0 con 0x55cf74662a50 2024-03-16T12:05:51.394 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:51.395+0000 7f3b02ffd640 1 --2- 172.21.15.29:0/3917485878 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55cf7466b820 0x55cf746771f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T12:05:51.394 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:51.395+0000 7f3b0b7fe640 1 --2- 172.21.15.29:0/3917485878 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cf74662a50 0x55cf74667990 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f3afc02c170 tx=0x7f3afc005dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27c3bcac282139ce server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:05:51.394 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:51.395+0000 7f3b017fa640 1 -- 172.21.15.29:0/3917485878 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3afc04e070 con 0x55cf74662a50 2024-03-16T12:05:51.394 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:51.395+0000 7f3b017fa640 1 -- 172.21.15.29:0/3917485878 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3afc04a6e0 con 0x55cf74662a50 2024-03-16T12:05:51.394 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:51.395+0000 7f3b017fa640 1 -- 172.21.15.29:0/3917485878 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3afc004920 con 0x55cf74662a50 2024-03-16T12:05:51.394 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:51.396+0000 7f3b14bdd740 1 -- 172.21.15.29:0/3917485878 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cf74662a50 msgr2=0x55cf74667990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:05:51.394 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:51.396+0000 7f3b14bdd740 1 --2- 172.21.15.29:0/3917485878 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cf74662a50 0x55cf74667990 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f3afc02c170 tx=0x7f3afc005dd0 comp rx=0 tx=0).stop 2024-03-16T12:05:51.394 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:51.396+0000 7f3b14bdd740 1 -- 172.21.15.29:0/3917485878 shutdown_connections 2024-03-16T12:05:51.394 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:51.396+0000 7f3b14bdd740 1 --2- 172.21.15.29:0/3917485878 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55cf7466b820 0x55cf746771f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:05:51.394 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:51.396+0000 7f3b14bdd740 1 --2- 172.21.15.29:0/3917485878 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cf74662a50 0x55cf74667990 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:05:51.394 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:51.396+0000 7f3b14bdd740 1 --2- 172.21.15.29:0/3917485878 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cf74662130 0x55cf7466af60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:05:51.394 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:51.396+0000 7f3b14bdd740 1 -- 172.21.15.29:0/3917485878 >> 172.21.15.29:0/3917485878 conn(0x55cf7465ca50 msgr2=0x55cf7465b320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:05:51.394 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:51.396+0000 7f3b14bdd740 1 -- 172.21.15.29:0/3917485878 shutdown_connections 2024-03-16T12:05:51.394 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:51.396+0000 7f3b14bdd740 1 -- 172.21.15.29:0/3917485878 wait complete. 2024-03-16T12:05:51.395 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:51.396+0000 7f3b14bdd740 1 -- start start 2024-03-16T12:05:51.395 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:51.396+0000 7f3b14bdd740 1 Processor -- start 2024-03-16T12:05:51.395 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:51.397+0000 7f3b14bdd740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cf74662130 0x55cf74673a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:05:51.395 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:51.397+0000 7f3b14bdd740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cf74662a50 0x55cf746650a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:05:51.395 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:51.397+0000 7f3b14bdd740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55cf7466b820 0x55cf746655e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:05:51.395 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:51.397+0000 7f3b14bdd740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f3b0c1a9900 con 0x55cf74662a50 2024-03-16T12:05:51.395 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:51.397+0000 7f3b14bdd740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f3b0c1aad70 con 0x55cf7466b820 2024-03-16T12:05:51.395 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:51.397+0000 7f3b14bdd740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f3b0c1a9fd0 con 0x55cf74662130 2024-03-16T12:05:51.395 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:51.397+0000 7f3b14bdd740 10 client.0 _ll_register_callbacks cb 0x7f3b040093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T12:05:51.395 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:51.397+0000 7f3b0b7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cf74662a50 0x55cf746650a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:05:51.395 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:51.397+0000 7f3b0b7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cf74662a50 0x55cf746650a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56498/0 (socket says 172.21.15.29:56498) 2024-03-16T12:05:51.395 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:51.397+0000 7f3b0b7fe640 1 -- 172.21.15.29:0/2090400878 learned_addr learned my addr 172.21.15.29:0/2090400878 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T12:05:51.396 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:51.397+0000 7f3b02ffd640 1 --2- 172.21.15.29:0/2090400878 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cf74662130 0x55cf74673a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:05:51.396 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:51.397+0000 7f3b02ffd640 1 --2- 172.21.15.29:0/2090400878 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55cf7466b820 0x55cf746655e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:05:51.396 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:51.397+0000 7f3b0b7fe640 1 -- 172.21.15.29:0/2090400878 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cf74662130 msgr2=0x55cf74673a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:05:51.396 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:51.397+0000 7f3b0b7fe640 1 --2- 172.21.15.29:0/2090400878 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cf74662130 0x55cf74673a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:05:51.396 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:51.397+0000 7f3b0b7fe640 1 -- 172.21.15.29:0/2090400878 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55cf7466b820 msgr2=0x55cf746655e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:05:51.396 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:51.397+0000 7f3b0b7fe640 1 --2- 172.21.15.29:0/2090400878 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55cf7466b820 0x55cf746655e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:05:51.396 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:51.397+0000 7f3b0b7fe640 1 -- 172.21.15.29:0/2090400878 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cf74665de0 con 0x55cf74662a50 2024-03-16T12:05:51.396 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:51.397+0000 7f3b0b7fe640 1 --2- 172.21.15.29:0/2090400878 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cf74662a50 0x55cf746650a0 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f3afc036040 tx=0x7f3afc04a890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2059ab913cb76f65 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:05:51.396 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:51.397+0000 7f3aeaffd640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-16T12:05:51.396 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:51.397+0000 7f3aeaffd640 1 -- 172.21.15.29:0/2090400878 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3afc056070 con 0x55cf74662a50 2024-03-16T12:05:51.396 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:51.397+0000 7f3aeaffd640 1 -- 172.21.15.29:0/2090400878 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(110..110 src has 1..110) ==== 8556+0+0 (secure 0 0 0) 0x7f3afc0046c0 con 0x55cf74662a50 2024-03-16T12:05:51.396 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:51.397+0000 7f3b14bdd740 1 -- 172.21.15.29:0/2090400878 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3afc04a6e0 con 0x55cf74662a50 2024-03-16T12:05:51.396 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:51.397+0000 7f3aeaffd640 1 -- 172.21.15.29:0/2090400878 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f3afc061050 con 0x55cf74662a50 2024-03-16T12:05:51.396 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:51.397+0000 7f3b14bdd740 10 client.16647 fetch_fsmap learned FSMap version 31 2024-03-16T12:05:51.396 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:51.397+0000 7f3b14bdd740 1 -- 172.21.15.29:0/2090400878 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55cf746e17f0 con 0x55cf74662a50 2024-03-16T12:05:51.396 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:51.398+0000 7f3aeaffd640 1 -- 172.21.15.29:0/2090400878 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3afc04a6e0 con 0x55cf74662a50 2024-03-16T12:05:51.397 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:51.398+0000 7f3aeaffd640 1 -- 172.21.15.29:0/2090400878 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 31) ==== 8346+0+0 (secure 0 0 0) 0x7f3afc047070 con 0x55cf74662a50 2024-03-16T12:05:51.397 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:51.398+0000 7f3b14bdd740 10 client.16647 fetch_fsmap finished waiting for FSMap version 31 2024-03-16T12:05:51.397 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:51.398+0000 7f3b14bdd740 10 client.16647 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T12:05:51.397 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:51.398+0000 7f3b14bdd740 20 client.16647 populate_metadata read hostname 'smithi029' 2024-03-16T12:05:51.397 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:51.398+0000 7f3b14bdd740 1 --2- 172.21.15.29:0/2090400878 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55cf746e1fa0 0x55cf746e43c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:05:51.397 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:51.398+0000 7f3b14bdd740 4 client.16647 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-16T12:05:51.397 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:51.398+0000 7f3b14bdd740 1 -- 172.21.15.29:0/2090400878 --> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55cf745ce2d0 con 0x55cf746e1fa0 2024-03-16T12:05:51.397 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:51.398+0000 7f3b02ffd640 1 --2- 172.21.15.29:0/2090400878 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55cf746e1fa0 0x55cf746e43c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:05:51.397 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:51.398+0000 7f3aeaffd640 10 client.16647 ms_handle_connect on v2:172.21.15.152:6838/1548731833 2024-03-16T12:05:51.397 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:51.398+0000 7f3b02ffd640 1 --2- 172.21.15.29:0/2090400878 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55cf746e1fa0 0x55cf746e43c0 crc :-1 s=READY pgs=173 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=89193e208c18ef11 server_cookie=f336ba5bcf32be35 in_seq=0 out_seq=0 2024-03-16T12:05:52.163 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:52.170+0000 7f3aeaffd640 1 -- 172.21.15.29:0/2090400878 <== mds.0 v2:172.21.15.152:6838/1548731833 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55cf745ce2d0 con 0x55cf746e1fa0 2024-03-16T12:05:52.164 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:52.170+0000 7f3aeaffd640 10 client.16647 handle_command_reply: tid=0 2024-03-16T12:05:52.164 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:52.170+0000 7f3b14bdd740 1 client.16647 shutdown 2024-03-16T12:05:52.164 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:52.170+0000 7f3b14bdd740 1 -- 172.21.15.29:0/2090400878 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cf74662a50 msgr2=0x55cf746650a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:05:52.164 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:52.170+0000 7f3b14bdd740 1 --2- 172.21.15.29:0/2090400878 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cf74662a50 0x55cf746650a0 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f3afc036040 tx=0x7f3afc04a890 comp rx=0 tx=0).stop 2024-03-16T12:05:52.164 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:52.170+0000 7f3b14bdd740 1 -- 172.21.15.29:0/2090400878 shutdown_connections 2024-03-16T12:05:52.164 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:52.170+0000 7f3b14bdd740 1 --2- 172.21.15.29:0/2090400878 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55cf746e1fa0 0x55cf746e43c0 crc :-1 s=READY pgs=173 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:05:52.164 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:52.170+0000 7f3b14bdd740 1 --2- 172.21.15.29:0/2090400878 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55cf7466b820 0x55cf746655e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:05:52.164 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:52.170+0000 7f3b14bdd740 1 --2- 172.21.15.29:0/2090400878 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cf74662a50 0x55cf746650a0 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:05:52.164 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:52.170+0000 7f3aeaffd640 0 client.16647 ms_handle_reset on v2:172.21.15.152:6838/1548731833 2024-03-16T12:05:52.164 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:52.170+0000 7f3b14bdd740 1 --2- 172.21.15.29:0/2090400878 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cf74662130 0x55cf74673a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:05:52.164 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:52.170+0000 7f3b14bdd740 1 -- 172.21.15.29:0/2090400878 >> 172.21.15.29:0/2090400878 conn(0x55cf7465ca50 msgr2=0x55cf746687f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:05:52.164 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:52.170+0000 7f3b14bdd740 1 -- 172.21.15.29:0/2090400878 shutdown_connections 2024-03-16T12:05:52.164 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:52.170+0000 7f3b14bdd740 1 -- 172.21.15.29:0/2090400878 wait complete. 2024-03-16T12:05:52.165 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:52.171+0000 7f3b14bdd740 20 client.16647 trim_cache size 0 max 16384 2024-03-16T12:05:52.165 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:{ 2024-03-16T12:05:52.165 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "message": "", 2024-03-16T12:05:52.165 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "return_code": 0 2024-03-16T12:05:52.165 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:} 2024-03-16T12:05:52.166 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:52.171+0000 7f3b11f10640 1 -- 172.21.15.29:0/699117973 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f3af80a2a10 msgr2=0x7f3af80a4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:05:52.166 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:52.171+0000 7f3b11f10640 1 --2- 172.21.15.29:0/699117973 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f3af80a2a10 0x7f3af80a4ed0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f3b0409b760 tx=0x7f3b04086040 comp rx=0 tx=0).stop 2024-03-16T12:05:52.166 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:52.171+0000 7f3b11f10640 1 -- 172.21.15.29:0/699117973 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3b0c10be30 msgr2=0x7f3b0c19c8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:05:52.166 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:52.171+0000 7f3b11f10640 1 --2- 172.21.15.29:0/699117973 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3b0c10be30 0x7f3b0c19c8e0 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f3b0400eb10 tx=0x7f3b0400c2d0 comp rx=0 tx=0).stop 2024-03-16T12:05:52.166 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:52.171+0000 7f3b11f10640 1 -- 172.21.15.29:0/699117973 shutdown_connections 2024-03-16T12:05:52.166 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:52.171+0000 7f3b11f10640 1 --2- 172.21.15.29:0/699117973 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f3af80a2a10 0x7f3af80a4ed0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:05:52.166 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:52.171+0000 7f3b11f10640 1 --2- 172.21.15.29:0/699117973 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3b0c10be30 0x7f3b0c19c8e0 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:05:52.166 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:52.171+0000 7f3b11f10640 1 --2- 172.21.15.29:0/699117973 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3b0c1032e0 0x7f3b0c198560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:05:52.166 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:52.171+0000 7f3b11f10640 1 --2- 172.21.15.29:0/699117973 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3b0c1029c0 0x7f3b0c198020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:05:52.167 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:52.171+0000 7f3b11f10640 1 -- 172.21.15.29:0/699117973 >> 172.21.15.29:0/699117973 conn(0x7f3b0c077810 msgr2=0x7f3b0c0fe820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:05:52.167 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:52.171+0000 7f3b11f10640 1 -- 172.21.15.29:0/699117973 shutdown_connections 2024-03-16T12:05:52.167 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:05:52.171+0000 7f3b11f10640 1 -- 172.21.15.29:0/699117973 wait complete. 2024-03-16T12:05:52.185 INFO:tasks.workunit.client.0.smithi029.stdout: 10 27090 17.12 MB/sec warmup 96 sec latency 536.559 ms 2024-03-16T12:05:52.581 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:05:52 smithi029 ceph-mon[31975]: pgmap v473: 97 pgs: 97 active+clean; 328 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 5.1 MiB/s wr, 475 op/s 2024-03-16T12:05:52.620 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:05:52 smithi152 ceph-mon[36291]: pgmap v473: 97 pgs: 97 active+clean; 328 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 5.1 MiB/s wr, 475 op/s 2024-03-16T12:05:52.788 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:05:52 smithi154 ceph-mon[36320]: pgmap v473: 97 pgs: 97 active+clean; 328 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 5.1 MiB/s wr, 475 op/s 2024-03-16T12:05:53.185 INFO:tasks.workunit.client.0.smithi029.stdout: 10 27291 16.98 MB/sec warmup 97 sec latency 225.635 ms 2024-03-16T12:05:54.185 INFO:tasks.workunit.client.0.smithi029.stdout: 10 27575 16.98 MB/sec warmup 98 sec latency 129.377 ms 2024-03-16T12:05:54.581 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:05:54 smithi029 ceph-mon[31975]: pgmap v474: 97 pgs: 97 active+clean; 328 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 3.3 MiB/s wr, 218 op/s 2024-03-16T12:05:54.620 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:05:54 smithi152 ceph-mon[36291]: pgmap v474: 97 pgs: 97 active+clean; 328 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 3.3 MiB/s wr, 218 op/s 2024-03-16T12:05:54.788 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:05:54 smithi154 ceph-mon[36320]: pgmap v474: 97 pgs: 97 active+clean; 328 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 3.3 MiB/s wr, 218 op/s 2024-03-16T12:05:55.185 INFO:tasks.workunit.client.0.smithi029.stdout: 10 27824 16.88 MB/sec warmup 99 sec latency 1129.411 ms 2024-03-16T12:05:56.185 INFO:tasks.workunit.client.0.smithi029.stdout: 10 28141 16.80 MB/sec warmup 100 sec latency 1659.044 ms 2024-03-16T12:05:56.581 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:05:56 smithi029 ceph-mon[31975]: pgmap v475: 97 pgs: 97 active+clean; 328 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 3.2 MiB/s wr, 184 op/s 2024-03-16T12:05:56.620 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:05:56 smithi152 ceph-mon[36291]: pgmap v475: 97 pgs: 97 active+clean; 328 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 3.2 MiB/s wr, 184 op/s 2024-03-16T12:05:56.788 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:05:56 smithi154 ceph-mon[36320]: pgmap v475: 97 pgs: 97 active+clean; 328 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 3.2 MiB/s wr, 184 op/s 2024-03-16T12:05:57.185 INFO:tasks.workunit.client.0.smithi029.stdout: 10 28460 16.80 MB/sec warmup 101 sec latency 256.530 ms 2024-03-16T12:05:58.185 INFO:tasks.workunit.client.0.smithi029.stdout: 10 28954 16.95 MB/sec warmup 102 sec latency 221.404 ms 2024-03-16T12:05:58.581 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:05:58 smithi029 ceph-mon[31975]: pgmap v476: 97 pgs: 97 active+clean; 335 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 5.9 MiB/s wr, 457 op/s 2024-03-16T12:05:58.620 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:05:58 smithi152 ceph-mon[36291]: pgmap v476: 97 pgs: 97 active+clean; 335 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 5.9 MiB/s wr, 457 op/s 2024-03-16T12:05:58.788 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:05:58 smithi154 ceph-mon[36320]: pgmap v476: 97 pgs: 97 active+clean; 335 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 5.9 MiB/s wr, 457 op/s 2024-03-16T12:05:59.185 INFO:tasks.workunit.client.0.smithi029.stdout: 10 29431 17.01 MB/sec warmup 103 sec latency 256.674 ms 2024-03-16T12:06:00.185 INFO:tasks.workunit.client.0.smithi029.stdout: 10 29705 17.05 MB/sec warmup 104 sec latency 224.134 ms 2024-03-16T12:06:00.581 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:06:00 smithi029 ceph-mon[31975]: pgmap v477: 97 pgs: 97 active+clean; 335 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 3.6 MiB/s wr, 346 op/s 2024-03-16T12:06:00.620 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:06:00 smithi152 ceph-mon[36291]: pgmap v477: 97 pgs: 97 active+clean; 335 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 3.6 MiB/s wr, 346 op/s 2024-03-16T12:06:00.788 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:06:00 smithi154 ceph-mon[36320]: pgmap v477: 97 pgs: 97 active+clean; 335 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 3.6 MiB/s wr, 346 op/s 2024-03-16T12:06:01.185 INFO:tasks.workunit.client.0.smithi029.stdout: 10 29825 16.99 MB/sec warmup 105 sec latency 492.033 ms 2024-03-16T12:06:02.185 INFO:tasks.workunit.client.0.smithi029.stdout: 10 29926 16.89 MB/sec warmup 106 sec latency 1309.920 ms 2024-03-16T12:06:02.581 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:06:02 smithi029 ceph-mon[31975]: pgmap v478: 97 pgs: 97 active+clean; 331 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 4.8 MiB/s wr, 444 op/s 2024-03-16T12:06:02.620 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:06:02 smithi152 ceph-mon[36291]: pgmap v478: 97 pgs: 97 active+clean; 331 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 4.8 MiB/s wr, 444 op/s 2024-03-16T12:06:02.788 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:06:02 smithi154 ceph-mon[36320]: pgmap v478: 97 pgs: 97 active+clean; 331 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 4.8 MiB/s wr, 444 op/s 2024-03-16T12:06:03.186 INFO:tasks.workunit.client.0.smithi029.stdout: 10 30134 16.83 MB/sec warmup 107 sec latency 200.056 ms 2024-03-16T12:06:04.186 INFO:tasks.workunit.client.0.smithi029.stdout: 10 30317 16.79 MB/sec warmup 108 sec latency 87.004 ms 2024-03-16T12:06:04.620 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:06:04 smithi152 ceph-mon[36291]: pgmap v479: 97 pgs: 97 active+clean; 333 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 4.3 MiB/s wr, 416 op/s 2024-03-16T12:06:04.788 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:06:04 smithi154 ceph-mon[36320]: pgmap v479: 97 pgs: 97 active+clean; 333 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 4.3 MiB/s wr, 416 op/s 2024-03-16T12:06:04.831 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:06:04 smithi029 ceph-mon[31975]: pgmap v479: 97 pgs: 97 active+clean; 333 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 4.3 MiB/s wr, 416 op/s 2024-03-16T12:06:05.186 INFO:tasks.workunit.client.0.smithi029.stdout: 10 30450 16.72 MB/sec warmup 109 sec latency 139.338 ms 2024-03-16T12:06:05.287 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:05.292+0000 7f6254dd4640 1 -- 172.21.15.29:0/2353122614 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6250101080 msgr2=0x7f625010d590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:06:05.288 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:05.292+0000 7f6254dd4640 1 --2- 172.21.15.29:0/2353122614 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6250101080 0x7f625010d590 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f624800bda0 tx=0x7f624802dc50 comp rx=0 tx=0).stop 2024-03-16T12:06:05.288 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:05.292+0000 7f6254dd4640 1 -- 172.21.15.29:0/2353122614 shutdown_connections 2024-03-16T12:06:05.288 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:05.292+0000 7f6254dd4640 1 --2- 172.21.15.29:0/2353122614 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6250101080 0x7f625010d590 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:06:05.288 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:05.292+0000 7f6254dd4640 1 --2- 172.21.15.29:0/2353122614 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f62501006c0 0x7f6250100b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:06:05.288 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:05.292+0000 7f6254dd4640 1 --2- 172.21.15.29:0/2353122614 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f62501066d0 0x7f6250106ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:06:05.288 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:05.292+0000 7f6254dd4640 1 -- 172.21.15.29:0/2353122614 >> 172.21.15.29:0/2353122614 conn(0x7f6250077800 msgr2=0x7f6250077c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:06:05.288 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:05.292+0000 7f6254dd4640 1 -- 172.21.15.29:0/2353122614 shutdown_connections 2024-03-16T12:06:05.288 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:05.292+0000 7f6254dd4640 1 -- 172.21.15.29:0/2353122614 wait complete. 2024-03-16T12:06:05.288 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:05.293+0000 7f6254dd4640 1 Processor -- start 2024-03-16T12:06:05.288 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:05.293+0000 7f6254dd4640 1 -- start start 2024-03-16T12:06:05.288 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:05.293+0000 7f6254dd4640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f62501006c0 0x7f625019a160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:06:05.288 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:05.293+0000 7f6254dd4640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6250101080 0x7f625019a6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:06:05.289 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:05.293+0000 7f6254dd4640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f62501066d0 0x7f625019ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:06:05.289 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:05.293+0000 7f6254dd4640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f625010f9d0 con 0x7f62501006c0 2024-03-16T12:06:05.289 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:05.293+0000 7f6254dd4640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f625010f850 con 0x7f6250101080 2024-03-16T12:06:05.289 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:05.293+0000 7f6254dd4640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f625010fb50 con 0x7f62501066d0 2024-03-16T12:06:05.289 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:05.294+0000 7f624e59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f62501006c0 0x7f625019a160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:06:05.289 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:05.294+0000 7f624e59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f62501006c0 0x7f625019a160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51426/0 (socket says 172.21.15.29:51426) 2024-03-16T12:06:05.289 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:05.294+0000 7f624e59c640 1 -- 172.21.15.29:0/1918706891 learned_addr learned my addr 172.21.15.29:0/1918706891 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T12:06:05.289 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:05.294+0000 7f624dd9b640 1 --2- 172.21.15.29:0/1918706891 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6250101080 0x7f625019a6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:06:05.289 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:05.294+0000 7f624e59c640 1 -- 172.21.15.29:0/1918706891 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f62501066d0 msgr2=0x7f625019ea20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:06:05.289 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:05.294+0000 7f624ed9d640 1 --2- 172.21.15.29:0/1918706891 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f62501066d0 0x7f625019ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:06:05.289 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:05.294+0000 7f624e59c640 1 --2- 172.21.15.29:0/1918706891 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f62501066d0 0x7f625019ea20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:06:05.289 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:05.294+0000 7f624e59c640 1 -- 172.21.15.29:0/1918706891 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6250101080 msgr2=0x7f625019a6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:06:05.289 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:05.294+0000 7f624e59c640 1 --2- 172.21.15.29:0/1918706891 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6250101080 0x7f625019a6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:06:05.289 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:05.294+0000 7f624e59c640 1 -- 172.21.15.29:0/1918706891 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f625019f150 con 0x7f62501006c0 2024-03-16T12:06:05.289 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:05.294+0000 7f624ed9d640 1 --2- 172.21.15.29:0/1918706891 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f62501066d0 0x7f625019ea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-16T12:06:05.289 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:05.294+0000 7f624e59c640 1 --2- 172.21.15.29:0/1918706891 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f62501006c0 0x7f625019a160 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f624400e9f0 tx=0x7f624400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:06:05.290 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:05.294+0000 7f623f7fe640 1 -- 172.21.15.29:0/1918706891 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f624400ccc0 con 0x7f62501006c0 2024-03-16T12:06:05.290 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:05.294+0000 7f6254dd4640 1 -- 172.21.15.29:0/1918706891 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f625019f440 con 0x7f62501006c0 2024-03-16T12:06:05.290 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:05.294+0000 7f623f7fe640 1 -- 172.21.15.29:0/1918706891 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6244005500 con 0x7f62501006c0 2024-03-16T12:06:05.290 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:05.294+0000 7f623f7fe640 1 -- 172.21.15.29:0/1918706891 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6244010690 con 0x7f62501006c0 2024-03-16T12:06:05.290 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:05.295+0000 7f6254dd4640 1 -- 172.21.15.29:0/1918706891 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f62501a6cd0 con 0x7f62501006c0 2024-03-16T12:06:05.290 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:05.298+0000 7f623f7fe640 1 -- 172.21.15.29:0/1918706891 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7f62440046c0 con 0x7f62501006c0 2024-03-16T12:06:05.291 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:05.298+0000 7f623f7fe640 1 --2- 172.21.15.29:0/1918706891 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f62380a2a90 0x7f62380a4f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:06:05.291 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:05.299+0000 7f623f7fe640 1 -- 172.21.15.29:0/1918706891 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 8600+0+0 (secure 0 0 0) 0x7f62440cf240 con 0x7f62501006c0 2024-03-16T12:06:05.291 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:05.299+0000 7f62579c1740 1 Processor -- start 2024-03-16T12:06:05.291 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:05.299+0000 7f624dd9b640 1 --2- 172.21.15.29:0/1918706891 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f62380a2a90 0x7f62380a4f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:06:05.291 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:05.299+0000 7f62579c1740 1 -- start start 2024-03-16T12:06:05.293 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:05.299+0000 7f62579c1740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55626aacc130 0x55626aacc510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:06:05.293 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:05.299+0000 7f62579c1740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55626aacca50 0x55626aaddaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:06:05.293 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:05.299+0000 7f62579c1740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55626aade030 0x55626aae0420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:06:05.293 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:05.299+0000 7f62579c1740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55626aa30a40 con 0x55626aacca50 2024-03-16T12:06:05.293 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:05.300+0000 7f62579c1740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x55626a9d9920 con 0x55626aade030 2024-03-16T12:06:05.293 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:05.300+0000 7f62579c1740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55626aa2cae0 con 0x55626aacc130 2024-03-16T12:06:05.294 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:05.300+0000 7f624e59c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55626aacc130 0x55626aacc510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:06:05.294 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:05.300+0000 7f624e59c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55626aacc130 0x55626aacc510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.29:51778/0 (socket says 172.21.15.29:51778) 2024-03-16T12:06:05.294 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:05.300+0000 7f624e59c640 1 -- 172.21.15.29:0/1623539910 learned_addr learned my addr 172.21.15.29:0/1623539910 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T12:06:05.294 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:05.300+0000 7f624dd9b640 1 --2- 172.21.15.29:0/1623539910 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55626aacca50 0x55626aaddaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:06:05.294 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:05.300+0000 7f624e59c640 1 -- 172.21.15.29:0/1623539910 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55626aade030 msgr2=0x55626aae0420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:06:05.294 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:05.300+0000 7f624e59c640 1 --2- 172.21.15.29:0/1623539910 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55626aade030 0x55626aae0420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:06:05.294 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:05.300+0000 7f624e59c640 1 -- 172.21.15.29:0/1623539910 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55626aacca50 msgr2=0x55626aaddaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:06:05.294 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:05.300+0000 7f624e59c640 1 --2- 172.21.15.29:0/1623539910 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55626aacca50 0x55626aaddaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:06:05.294 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:05.300+0000 7f624dd9b640 1 --2- 172.21.15.29:0/1918706891 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f62380a2a90 0x7f62380a4f50 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f6250077d20 tx=0x7f6240005f90 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:06:05.294 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:05.300+0000 7f624e59c640 1 -- 172.21.15.29:0/1623539910 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55626a9f1460 con 0x55626aacc130 2024-03-16T12:06:05.294 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:05.300+0000 7f624e59c640 1 --2- 172.21.15.29:0/1623539910 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55626aacc130 0x55626aacc510 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f62440a3a60 tx=0x7f624409b3f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=11fab6f77e19de23 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:06:05.294 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:05.301+0000 7f623cff9640 1 -- 172.21.15.29:0/1623539910 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f624409b7f0 con 0x55626aacc130 2024-03-16T12:06:05.294 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:05.301+0000 7f623cff9640 1 -- 172.21.15.29:0/1623539910 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f624409f940 con 0x55626aacc130 2024-03-16T12:06:05.294 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:05.301+0000 7f623cff9640 1 -- 172.21.15.29:0/1623539910 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62440afbd0 con 0x55626aacc130 2024-03-16T12:06:05.297 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:05.302+0000 7f62579c1740 1 -- 172.21.15.29:0/1623539910 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55626aacc130 msgr2=0x55626aacc510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:06:05.297 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:05.302+0000 7f62579c1740 1 --2- 172.21.15.29:0/1623539910 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55626aacc130 0x55626aacc510 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f62440a3a60 tx=0x7f624409b3f0 comp rx=0 tx=0).stop 2024-03-16T12:06:05.297 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:05.302+0000 7f62579c1740 1 -- 172.21.15.29:0/1623539910 shutdown_connections 2024-03-16T12:06:05.297 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:05.302+0000 7f62579c1740 1 --2- 172.21.15.29:0/1623539910 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55626aade030 0x55626aae0420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:06:05.298 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:05.302+0000 7f62579c1740 1 --2- 172.21.15.29:0/1623539910 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55626aacca50 0x55626aaddaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:06:05.298 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:05.302+0000 7f62579c1740 1 --2- 172.21.15.29:0/1623539910 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55626aacc130 0x55626aacc510 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:06:05.298 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:05.302+0000 7f62579c1740 1 -- 172.21.15.29:0/1623539910 >> 172.21.15.29:0/1623539910 conn(0x55626aac6a50 msgr2=0x55626aac6e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:06:05.298 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:05.302+0000 7f62579c1740 1 -- 172.21.15.29:0/1623539910 shutdown_connections 2024-03-16T12:06:05.298 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:05.302+0000 7f62579c1740 1 -- 172.21.15.29:0/1623539910 wait complete. 2024-03-16T12:06:05.298 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:05.302+0000 7f62579c1740 1 -- start start 2024-03-16T12:06:05.298 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:05.302+0000 7f62579c1740 1 Processor -- start 2024-03-16T12:06:05.298 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:05.303+0000 7f62579c1740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55626aacc130 0x55626aadb110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:06:05.298 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:05.303+0000 7f62579c1740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55626aacca50 0x55626aadb650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:06:05.298 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:05.303+0000 7f62579c1740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55626aade030 0x55626aadbb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:06:05.298 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:05.303+0000 7f62579c1740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55626aae0c00 con 0x55626aacc130 2024-03-16T12:06:05.298 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:05.303+0000 7f624e59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55626aacc130 0x55626aadb110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:06:05.298 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:05.303+0000 7f624e59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55626aacc130 0x55626aadb110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51444/0 (socket says 172.21.15.29:51444) 2024-03-16T12:06:05.298 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:05.303+0000 7f624e59c640 1 -- 172.21.15.29:0/2654272667 learned_addr learned my addr 172.21.15.29:0/2654272667 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T12:06:05.298 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:05.303+0000 7f624ed9d640 1 --2- 172.21.15.29:0/2654272667 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55626aade030 0x55626aadbb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:06:05.299 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:05.303+0000 7f62579c1740 1 -- 172.21.15.29:0/2654272667 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x55626a9ecd70 con 0x55626aacca50 2024-03-16T12:06:05.299 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:05.303+0000 7f62579c1740 1 -- 172.21.15.29:0/2654272667 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55626aae0d80 con 0x55626aade030 2024-03-16T12:06:05.299 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:05.303+0000 7f62579c1740 10 client.0 _ll_register_callbacks cb 0x55626a744a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T12:06:05.299 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:05.304+0000 7f624e59c640 1 -- 172.21.15.29:0/2654272667 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55626aade030 msgr2=0x55626aadbb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:06:05.299 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:05.304+0000 7f624e59c640 1 --2- 172.21.15.29:0/2654272667 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55626aade030 0x55626aadbb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:06:05.299 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:05.304+0000 7f624e59c640 1 -- 172.21.15.29:0/2654272667 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55626aacca50 msgr2=0x55626aadb650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:06:05.299 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:05.304+0000 7f624e59c640 1 --2- 172.21.15.29:0/2654272667 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55626aacca50 0x55626aadb650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:06:05.299 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:05.304+0000 7f624e59c640 1 -- 172.21.15.29:0/2654272667 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55626a745210 con 0x55626aacc130 2024-03-16T12:06:05.299 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:05.304+0000 7f624ed9d640 1 --2- 172.21.15.29:0/2654272667 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55626aade030 0x55626aadbb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:06:05.299 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:05.304+0000 7f624e59c640 1 --2- 172.21.15.29:0/2654272667 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55626aacc130 0x55626aadb110 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f6244097c50 tx=0x7f624409b460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=396df0405bb9325a server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:06:05.299 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:05.304+0000 7f6235ffb640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-16T12:06:05.299 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:05.304+0000 7f6235ffb640 1 -- 172.21.15.29:0/2654272667 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6244065060 con 0x55626aacc130 2024-03-16T12:06:05.299 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:05.304+0000 7f6235ffb640 1 -- 172.21.15.29:0/2654272667 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(111..111 src has 1..111) ==== 8600+0+0 (secure 0 0 0) 0x7f62440afbd0 con 0x55626aacc130 2024-03-16T12:06:05.299 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:05.304+0000 7f62579c1740 1 -- 172.21.15.29:0/2654272667 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f624409f940 con 0x55626aacc130 2024-03-16T12:06:05.299 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:05.304+0000 7f6235ffb640 1 -- 172.21.15.29:0/2654272667 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f62440ba070 con 0x55626aacc130 2024-03-16T12:06:05.299 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:05.305+0000 7f62579c1740 10 client.16659 fetch_fsmap learned FSMap version 31 2024-03-16T12:06:05.300 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:05.305+0000 7f62579c1740 1 -- 172.21.15.29:0/2654272667 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55626a811620 con 0x55626aacc130 2024-03-16T12:06:05.300 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:05.305+0000 7f6235ffb640 1 -- 172.21.15.29:0/2654272667 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f624409f940 con 0x55626aacc130 2024-03-16T12:06:05.300 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:05.305+0000 7f6235ffb640 1 -- 172.21.15.29:0/2654272667 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 31) ==== 8346+0+0 (secure 0 0 0) 0x7f62440bb040 con 0x55626aacc130 2024-03-16T12:06:05.300 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:05.305+0000 7f62579c1740 10 client.16659 fetch_fsmap finished waiting for FSMap version 31 2024-03-16T12:06:05.300 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:05.305+0000 7f62579c1740 10 client.16659 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T12:06:05.300 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:05.305+0000 7f62579c1740 20 client.16659 populate_metadata read hostname 'smithi029' 2024-03-16T12:06:05.300 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:05.305+0000 7f62579c1740 1 --2- 172.21.15.29:0/2654272667 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55626aaee160 0x55626aaf0570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:06:05.300 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:05.305+0000 7f62579c1740 4 client.16659 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T12:06:05.300 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:05.305+0000 7f62579c1740 1 -- 172.21.15.29:0/2654272667 --> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55626aa3df80 con 0x55626aaee160 2024-03-16T12:06:05.300 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:05.307+0000 7f624dd9b640 1 --2- 172.21.15.29:0/2654272667 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55626aaee160 0x55626aaf0570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:06:05.302 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:05.309+0000 7f624dd9b640 1 --2- 172.21.15.29:0/2654272667 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55626aaee160 0x55626aaf0570 crc :-1 s=READY pgs=174 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dbade21f7afced6e server_cookie=b2160b16c20ebd83 in_seq=0 out_seq=0 2024-03-16T12:06:05.302 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:05.310+0000 7f6235ffb640 10 client.16659 ms_handle_connect on v2:172.21.15.152:6838/1548731833 2024-03-16T12:06:05.307 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:05.314+0000 7f6235ffb640 1 -- 172.21.15.29:0/2654272667 <== mds.0 v2:172.21.15.152:6838/1548731833 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55626aa3df80 con 0x55626aaee160 2024-03-16T12:06:05.307 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:05.314+0000 7f6235ffb640 10 client.16659 handle_command_reply: tid=0 2024-03-16T12:06:05.307 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:05.314+0000 7f62579c1740 1 client.16659 shutdown 2024-03-16T12:06:05.307 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:05.314+0000 7f62579c1740 1 -- 172.21.15.29:0/2654272667 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55626aacc130 msgr2=0x55626aadb110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:06:05.307 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:05.314+0000 7f62579c1740 1 --2- 172.21.15.29:0/2654272667 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55626aacc130 0x55626aadb110 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f6244097c50 tx=0x7f624409b460 comp rx=0 tx=0).stop 2024-03-16T12:06:05.307 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:05.314+0000 7f62579c1740 1 -- 172.21.15.29:0/2654272667 shutdown_connections 2024-03-16T12:06:05.307 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:05.314+0000 7f62579c1740 1 --2- 172.21.15.29:0/2654272667 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55626aade030 0x55626aadbb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:06:05.307 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:05.314+0000 7f62579c1740 1 --2- 172.21.15.29:0/2654272667 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55626aaee160 0x55626aaf0570 crc :-1 s=READY pgs=174 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:06:05.307 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:05.314+0000 7f62579c1740 1 --2- 172.21.15.29:0/2654272667 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55626aacca50 0x55626aadb650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:06:05.307 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:05.314+0000 7f62579c1740 1 --2- 172.21.15.29:0/2654272667 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55626aacc130 0x55626aadb110 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:06:05.307 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:05.314+0000 7f62579c1740 1 -- 172.21.15.29:0/2654272667 >> 172.21.15.29:0/2654272667 conn(0x55626aac6a50 msgr2=0x55626aac87a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:06:05.307 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:05.314+0000 7f6235ffb640 0 client.16659 ms_handle_reset on v2:172.21.15.152:6838/1548731833 2024-03-16T12:06:05.308 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:05.315+0000 7f62579c1740 1 -- 172.21.15.29:0/2654272667 shutdown_connections 2024-03-16T12:06:05.308 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:05.315+0000 7f62579c1740 1 -- 172.21.15.29:0/2654272667 wait complete. 2024-03-16T12:06:05.308 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:05.315+0000 7f62579c1740 20 client.16659 trim_cache size 0 max 16384 2024-03-16T12:06:05.329 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:05.336+0000 7f62579c1740 1 Processor -- start 2024-03-16T12:06:05.329 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:05.336+0000 7f62579c1740 1 -- start start 2024-03-16T12:06:05.329 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:05.336+0000 7f62579c1740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55626aacc130 0x55626aadd840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:06:05.329 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:05.336+0000 7f62579c1740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55626aacca50 0x55626aae99a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:06:05.329 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:05.337+0000 7f62579c1740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55626aade030 0x55626aac85b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:06:05.329 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:05.337+0000 7f62579c1740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f62501adcb0 con 0x55626aade030 2024-03-16T12:06:05.329 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:05.337+0000 7f624e59c640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55626aacc130 0x55626aadd840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:06:05.330 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:05.337+0000 7f624e59c640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55626aacc130 0x55626aadd840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.29:39112/0 (socket says 172.21.15.29:39112) 2024-03-16T12:06:05.330 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:05.337+0000 7f624e59c640 1 -- 172.21.15.29:0/1731906244 learned_addr learned my addr 172.21.15.29:0/1731906244 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T12:06:05.330 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:05.337+0000 7f62579c1740 1 -- 172.21.15.29:0/1731906244 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f62501ab520 con 0x55626aacc130 2024-03-16T12:06:05.330 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:05.337+0000 7f62579c1740 1 -- 172.21.15.29:0/1731906244 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f62501a9390 con 0x55626aacca50 2024-03-16T12:06:05.330 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:05.337+0000 7f624ed9d640 1 --2- 172.21.15.29:0/1731906244 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55626aade030 0x55626aac85b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:06:05.331 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:05.337+0000 7f624dd9b640 1 --2- 172.21.15.29:0/1731906244 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55626aacca50 0x55626aae99a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:06:05.331 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:05.338+0000 7f624ed9d640 1 -- 172.21.15.29:0/1731906244 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55626aacca50 msgr2=0x55626aae99a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:06:05.331 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:05.338+0000 7f624ed9d640 1 --2- 172.21.15.29:0/1731906244 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55626aacca50 0x55626aae99a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:06:05.331 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:05.338+0000 7f624ed9d640 1 -- 172.21.15.29:0/1731906244 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55626aacc130 msgr2=0x55626aadd840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:06:05.331 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:05.338+0000 7f624ed9d640 1 --2- 172.21.15.29:0/1731906244 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55626aacc130 0x55626aadd840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:06:05.331 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:05.338+0000 7f624ed9d640 1 -- 172.21.15.29:0/1731906244 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55626aaf0bf0 con 0x55626aade030 2024-03-16T12:06:05.331 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:05.338+0000 7f624ed9d640 1 --2- 172.21.15.29:0/1731906244 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55626aade030 0x55626aac85b0 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x55626aac7e70 tx=0x7f624800ba50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f11581b4319a6375 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:06:05.331 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:05.338+0000 7f6237fff640 1 -- 172.21.15.29:0/1731906244 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6248044030 con 0x55626aade030 2024-03-16T12:06:05.331 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:05.338+0000 7f6237fff640 1 -- 172.21.15.29:0/1731906244 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6248003e50 con 0x55626aade030 2024-03-16T12:06:05.331 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:05.338+0000 7f6237fff640 1 -- 172.21.15.29:0/1731906244 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f624802ea00 con 0x55626aade030 2024-03-16T12:06:05.332 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:05.340+0000 7f62579c1740 1 -- 172.21.15.29:0/1731906244 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55626aade030 msgr2=0x55626aac85b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:06:05.333 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:05.340+0000 7f62579c1740 1 --2- 172.21.15.29:0/1731906244 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55626aade030 0x55626aac85b0 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x55626aac7e70 tx=0x7f624800ba50 comp rx=0 tx=0).stop 2024-03-16T12:06:05.333 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:05.340+0000 7f62579c1740 1 -- 172.21.15.29:0/1731906244 shutdown_connections 2024-03-16T12:06:05.333 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:05.340+0000 7f62579c1740 1 --2- 172.21.15.29:0/1731906244 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55626aade030 0x55626aac85b0 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:06:05.333 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:05.340+0000 7f62579c1740 1 --2- 172.21.15.29:0/1731906244 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55626aacca50 0x55626aae99a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:06:05.333 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:05.340+0000 7f62579c1740 1 --2- 172.21.15.29:0/1731906244 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55626aacc130 0x55626aadd840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:06:05.333 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:05.340+0000 7f62579c1740 1 -- 172.21.15.29:0/1731906244 >> 172.21.15.29:0/1731906244 conn(0x55626aac6a50 msgr2=0x55626aac7ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:06:05.333 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:05.340+0000 7f62579c1740 1 -- 172.21.15.29:0/1731906244 shutdown_connections 2024-03-16T12:06:05.333 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:05.340+0000 7f62579c1740 1 -- 172.21.15.29:0/1731906244 wait complete. 2024-03-16T12:06:05.336 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:05.341+0000 7f62579c1740 1 -- start start 2024-03-16T12:06:05.336 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:05.341+0000 7f62579c1740 1 Processor -- start 2024-03-16T12:06:05.336 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:05.341+0000 7f62579c1740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55626aacc130 0x55626aad7e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:06:05.336 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:05.342+0000 7f62579c1740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55626aacca50 0x55626aad8350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:06:05.336 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:05.342+0000 7f62579c1740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55626aade030 0x55626aad8890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:06:05.336 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:05.342+0000 7f62579c1740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f62501a99e0 con 0x55626aacc130 2024-03-16T12:06:05.336 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:05.342+0000 7f624e59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55626aacc130 0x55626aad7e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:06:05.337 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:05.342+0000 7f624e59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55626aacc130 0x55626aad7e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51458/0 (socket says 172.21.15.29:51458) 2024-03-16T12:06:05.337 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:05.342+0000 7f624e59c640 1 -- 172.21.15.29:0/1998738893 learned_addr learned my addr 172.21.15.29:0/1998738893 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T12:06:05.337 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:05.342+0000 7f62579c1740 1 -- 172.21.15.29:0/1998738893 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f62501aae50 con 0x55626aade030 2024-03-16T12:06:05.337 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:05.342+0000 7f62579c1740 1 -- 172.21.15.29:0/1998738893 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f62501aa0b0 con 0x55626aacca50 2024-03-16T12:06:05.337 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:05.342+0000 7f624dd9b640 1 --2- 172.21.15.29:0/1998738893 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55626aacca50 0x55626aad8350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:06:05.337 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:05.342+0000 7f624ed9d640 1 --2- 172.21.15.29:0/1998738893 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55626aade030 0x55626aad8890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:06:05.337 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:05.342+0000 7f62579c1740 10 client.0 _ll_register_callbacks cb 0x7f6240009900 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T12:06:05.337 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:05.342+0000 7f624ed9d640 1 -- 172.21.15.29:0/1998738893 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55626aacca50 msgr2=0x55626aad8350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:06:05.337 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:05.342+0000 7f624ed9d640 1 --2- 172.21.15.29:0/1998738893 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55626aacca50 0x55626aad8350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:06:05.337 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:05.342+0000 7f624ed9d640 1 -- 172.21.15.29:0/1998738893 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55626aacc130 msgr2=0x55626aad7e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:06:05.337 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:05.342+0000 7f624ed9d640 1 --2- 172.21.15.29:0/1998738893 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55626aacc130 0x55626aad7e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:06:05.337 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:05.342+0000 7f624ed9d640 1 -- 172.21.15.29:0/1998738893 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55626ab64620 con 0x55626aade030 2024-03-16T12:06:05.337 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:05.342+0000 7f624e59c640 1 --2- 172.21.15.29:0/1998738893 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55626aacc130 0x55626aad7e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-16T12:06:05.337 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:05.342+0000 7f624dd9b640 1 --2- 172.21.15.29:0/1998738893 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55626aacca50 0x55626aad8350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-16T12:06:05.337 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:05.342+0000 7f624ed9d640 1 --2- 172.21.15.29:0/1998738893 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55626aade030 0x55626aad8890 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f624800bda0 tx=0x7f6248036660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=902bd50b1ab2959b server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:06:05.338 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:05.342+0000 7f6235ffb640 10 client.0 ms_handle_connect on v2:172.21.15.152:3300/0 2024-03-16T12:06:05.338 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:05.343+0000 7f6235ffb640 1 -- 172.21.15.29:0/1998738893 <== mon.2 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f624804d030 con 0x55626aade030 2024-03-16T12:06:05.338 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:05.343+0000 7f62579c1740 1 -- 172.21.15.29:0/1998738893 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6248003e50 con 0x55626aade030 2024-03-16T12:06:05.338 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:05.343+0000 7f6235ffb640 1 -- 172.21.15.29:0/1998738893 <== mon.2 v2:172.21.15.152:3300/0 2 ==== osd_map(111..111 src has 1..111) ==== 8600+0+0 (secure 0 0 0) 0x7f6248004000 con 0x55626aade030 2024-03-16T12:06:05.338 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:05.343+0000 7f6235ffb640 1 -- 172.21.15.29:0/1998738893 <== mon.2 v2:172.21.15.152:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f6248057050 con 0x55626aade030 2024-03-16T12:06:05.338 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:05.343+0000 7f62579c1740 10 client.26459 fetch_fsmap learned FSMap version 31 2024-03-16T12:06:05.338 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:05.343+0000 7f62579c1740 1 -- 172.21.15.29:0/1998738893 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55626ab65910 con 0x55626aade030 2024-03-16T12:06:05.338 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:05.343+0000 7f6235ffb640 1 -- 172.21.15.29:0/1998738893 <== mon.2 v2:172.21.15.152:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6248003e50 con 0x55626aade030 2024-03-16T12:06:05.338 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:05.344+0000 7f6235ffb640 1 -- 172.21.15.29:0/1998738893 <== mon.2 v2:172.21.15.152:3300/0 5 ==== fsmap(e 31) ==== 8346+0+0 (secure 0 0 0) 0x7f6248040800 con 0x55626aade030 2024-03-16T12:06:05.338 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:05.344+0000 7f62579c1740 10 client.26459 fetch_fsmap finished waiting for FSMap version 31 2024-03-16T12:06:05.338 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:05.344+0000 7f62579c1740 10 client.26459 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T12:06:05.338 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:05.344+0000 7f62579c1740 20 client.26459 populate_metadata read hostname 'smithi029' 2024-03-16T12:06:05.338 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:05.344+0000 7f62579c1740 1 --2- 172.21.15.29:0/1998738893 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55626ab66050 0x55626ab68470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:06:05.338 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:05.344+0000 7f62579c1740 4 client.26459 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-16T12:06:05.338 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:05.344+0000 7f62579c1740 1 -- 172.21.15.29:0/1998738893 --> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55626aa382d0 con 0x55626ab66050 2024-03-16T12:06:05.339 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:05.345+0000 7f624e59c640 1 --2- 172.21.15.29:0/1998738893 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55626ab66050 0x55626ab68470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:06:05.344 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:05.351+0000 7f624e59c640 1 --2- 172.21.15.29:0/1998738893 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55626ab66050 0x55626ab68470 crc :-1 s=READY pgs=175 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=14979a0a07a5fbaa server_cookie=1c2dd25c97e4c471 in_seq=0 out_seq=0 2024-03-16T12:06:05.344 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:05.351+0000 7f6235ffb640 10 client.26459 ms_handle_connect on v2:172.21.15.152:6838/1548731833 2024-03-16T12:06:05.620 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:06:05 smithi152 ceph-mon[36291]: osdmap e111: 12 total, 12 up, 12 in 2024-03-16T12:06:05.788 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:06:05 smithi154 ceph-mon[36320]: osdmap e111: 12 total, 12 up, 12 in 2024-03-16T12:06:05.831 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:06:05 smithi029 ceph-mon[31975]: osdmap e111: 12 total, 12 up, 12 in 2024-03-16T12:06:06.186 INFO:tasks.workunit.client.0.smithi029.stdout: 10 30472 16.58 MB/sec warmup 110 sec latency 723.233 ms 2024-03-16T12:06:06.497 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:06.503+0000 7f6235ffb640 1 -- 172.21.15.29:0/1998738893 <== mds.0 v2:172.21.15.152:6838/1548731833 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55626aa382d0 con 0x55626ab66050 2024-03-16T12:06:06.497 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:06.503+0000 7f6235ffb640 10 client.26459 handle_command_reply: tid=0 2024-03-16T12:06:06.497 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:06.503+0000 7f62579c1740 1 client.26459 shutdown 2024-03-16T12:06:06.497 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:06.504+0000 7f62579c1740 1 -- 172.21.15.29:0/1998738893 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55626aade030 msgr2=0x55626aad8890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:06:06.497 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:06.504+0000 7f62579c1740 1 --2- 172.21.15.29:0/1998738893 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55626aade030 0x55626aad8890 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f624800bda0 tx=0x7f6248036660 comp rx=0 tx=0).stop 2024-03-16T12:06:06.497 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:06.504+0000 7f62579c1740 1 -- 172.21.15.29:0/1998738893 shutdown_connections 2024-03-16T12:06:06.497 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:06.504+0000 7f62579c1740 1 --2- 172.21.15.29:0/1998738893 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55626ab66050 0x55626ab68470 crc :-1 s=READY pgs=175 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:06:06.497 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:06.504+0000 7f62579c1740 1 --2- 172.21.15.29:0/1998738893 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55626aade030 0x55626aad8890 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:06:06.497 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:06.504+0000 7f62579c1740 1 --2- 172.21.15.29:0/1998738893 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55626aacca50 0x55626aad8350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:06:06.498 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:06.504+0000 7f62579c1740 1 --2- 172.21.15.29:0/1998738893 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55626aacc130 0x55626aad7e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:06:06.498 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:06.504+0000 7f6235ffb640 0 client.26459 ms_handle_reset on v2:172.21.15.152:6838/1548731833 2024-03-16T12:06:06.498 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:06.504+0000 7f62579c1740 1 -- 172.21.15.29:0/1998738893 >> 172.21.15.29:0/1998738893 conn(0x55626aac6a50 msgr2=0x55626aac5bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:06:06.498 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:06.504+0000 7f62579c1740 1 -- 172.21.15.29:0/1998738893 shutdown_connections 2024-03-16T12:06:06.498 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:06.504+0000 7f62579c1740 1 -- 172.21.15.29:0/1998738893 wait complete. 2024-03-16T12:06:06.498 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:06.504+0000 7f62579c1740 20 client.26459 trim_cache size 0 max 16384 2024-03-16T12:06:06.498 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:{ 2024-03-16T12:06:06.498 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "message": "", 2024-03-16T12:06:06.498 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "return_code": 0 2024-03-16T12:06:06.500 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:} 2024-03-16T12:06:06.501 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:06.504+0000 7f6254dd4640 1 -- 172.21.15.29:0/1918706891 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f62380a2a90 msgr2=0x7f62380a4f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:06:06.501 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:06.504+0000 7f6254dd4640 1 --2- 172.21.15.29:0/1918706891 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f62380a2a90 0x7f62380a4f50 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f6250077d20 tx=0x7f6240005f90 comp rx=0 tx=0).stop 2024-03-16T12:06:06.502 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:06.504+0000 7f6254dd4640 1 -- 172.21.15.29:0/1918706891 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f62501006c0 msgr2=0x7f625019a160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:06:06.502 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:06.504+0000 7f6254dd4640 1 --2- 172.21.15.29:0/1918706891 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f62501006c0 0x7f625019a160 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f624400e9f0 tx=0x7f624400eeb0 comp rx=0 tx=0).stop 2024-03-16T12:06:06.502 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:06.505+0000 7f6254dd4640 1 -- 172.21.15.29:0/1918706891 shutdown_connections 2024-03-16T12:06:06.502 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:06.505+0000 7f6254dd4640 1 --2- 172.21.15.29:0/1918706891 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f62380a2a90 0x7f62380a4f50 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:06:06.502 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:06.505+0000 7f6254dd4640 1 --2- 172.21.15.29:0/1918706891 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f62501066d0 0x7f625019ea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:06:06.502 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:06.505+0000 7f6254dd4640 1 --2- 172.21.15.29:0/1918706891 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6250101080 0x7f625019a6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:06:06.502 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:06.505+0000 7f6254dd4640 1 --2- 172.21.15.29:0/1918706891 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f62501006c0 0x7f625019a160 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:06:06.502 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:06.505+0000 7f6254dd4640 1 -- 172.21.15.29:0/1918706891 >> 172.21.15.29:0/1918706891 conn(0x7f6250077800 msgr2=0x7f62501031f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:06:06.502 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:06.505+0000 7f6254dd4640 1 -- 172.21.15.29:0/1918706891 shutdown_connections 2024-03-16T12:06:06.502 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:06.505+0000 7f6254dd4640 1 -- 172.21.15.29:0/1918706891 wait complete. 2024-03-16T12:06:06.620 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:06:06 smithi152 ceph-mon[36291]: pgmap v481: 97 pgs: 97 active+clean; 333 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 5.1 MiB/s wr, 494 op/s 2024-03-16T12:06:06.788 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:06:06 smithi154 ceph-mon[36320]: pgmap v481: 97 pgs: 97 active+clean; 333 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 5.1 MiB/s wr, 494 op/s 2024-03-16T12:06:06.831 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:06:06 smithi029 ceph-mon[31975]: pgmap v481: 97 pgs: 97 active+clean; 333 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 5.1 MiB/s wr, 494 op/s 2024-03-16T12:06:07.186 INFO:tasks.workunit.client.0.smithi029.stdout: 10 30512 16.44 MB/sec warmup 111 sec latency 1016.553 ms 2024-03-16T12:06:08.186 INFO:tasks.workunit.client.0.smithi029.stdout: 10 30627 16.34 MB/sec warmup 112 sec latency 211.088 ms 2024-03-16T12:06:08.620 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:06:08 smithi152 ceph-mon[36291]: pgmap v482: 97 pgs: 24 active+clean+snaptrim, 40 active+clean+snaptrim_wait, 33 active+clean; 340 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 4.2 MiB/s wr, 512 op/s 2024-03-16T12:06:08.788 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:06:08 smithi154 ceph-mon[36320]: pgmap v482: 97 pgs: 24 active+clean+snaptrim, 40 active+clean+snaptrim_wait, 33 active+clean; 340 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 4.2 MiB/s wr, 512 op/s 2024-03-16T12:06:08.831 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:06:08 smithi029 ceph-mon[31975]: pgmap v482: 97 pgs: 24 active+clean+snaptrim, 40 active+clean+snaptrim_wait, 33 active+clean; 340 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 4.2 MiB/s wr, 512 op/s 2024-03-16T12:06:09.186 INFO:tasks.workunit.client.0.smithi029.stdout: 10 30782 16.27 MB/sec warmup 113 sec latency 218.656 ms 2024-03-16T12:06:10.186 INFO:tasks.workunit.client.0.smithi029.stdout: 10 30925 16.16 MB/sec warmup 114 sec latency 273.426 ms 2024-03-16T12:06:10.620 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:06:10 smithi152 ceph-mon[36291]: pgmap v483: 97 pgs: 24 active+clean+snaptrim, 40 active+clean+snaptrim_wait, 33 active+clean; 340 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 4.2 MiB/s wr, 512 op/s 2024-03-16T12:06:10.788 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:06:10 smithi154 ceph-mon[36320]: pgmap v483: 97 pgs: 24 active+clean+snaptrim, 40 active+clean+snaptrim_wait, 33 active+clean; 340 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 4.2 MiB/s wr, 512 op/s 2024-03-16T12:06:10.831 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:06:10 smithi029 ceph-mon[31975]: pgmap v483: 97 pgs: 24 active+clean+snaptrim, 40 active+clean+snaptrim_wait, 33 active+clean; 340 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 4.2 MiB/s wr, 512 op/s 2024-03-16T12:06:11.186 INFO:tasks.workunit.client.0.smithi029.stdout: 10 31148 16.14 MB/sec warmup 115 sec latency 405.257 ms 2024-03-16T12:06:12.186 INFO:tasks.workunit.client.0.smithi029.stdout: 10 31506 16.05 MB/sec warmup 116 sec latency 307.154 ms 2024-03-16T12:06:12.620 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:06:12 smithi152 ceph-mon[36291]: pgmap v484: 97 pgs: 18 active+clean+snaptrim, 34 active+clean+snaptrim_wait, 45 active+clean; 334 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.1 MiB/s wr, 453 op/s 2024-03-16T12:06:12.788 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:06:12 smithi154 ceph-mon[36320]: pgmap v484: 97 pgs: 18 active+clean+snaptrim, 34 active+clean+snaptrim_wait, 45 active+clean; 334 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.1 MiB/s wr, 453 op/s 2024-03-16T12:06:12.831 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:06:12 smithi029 ceph-mon[31975]: pgmap v484: 97 pgs: 18 active+clean+snaptrim, 34 active+clean+snaptrim_wait, 45 active+clean; 334 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.1 MiB/s wr, 453 op/s 2024-03-16T12:06:13.186 INFO:tasks.workunit.client.0.smithi029.stdout: 10 31878 16.08 MB/sec warmup 117 sec latency 216.914 ms 2024-03-16T12:06:14.186 INFO:tasks.workunit.client.0.smithi029.stdout: 10 32303 16.19 MB/sec warmup 118 sec latency 289.192 ms 2024-03-16T12:06:14.620 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:06:14 smithi152 ceph-mon[36291]: pgmap v485: 97 pgs: 97 active+clean; 327 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 4.7 MiB/s wr, 631 op/s 2024-03-16T12:06:14.788 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:06:14 smithi154 ceph-mon[36320]: pgmap v485: 97 pgs: 97 active+clean; 327 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 4.7 MiB/s wr, 631 op/s 2024-03-16T12:06:14.831 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:06:14 smithi029 ceph-mon[31975]: pgmap v485: 97 pgs: 97 active+clean; 327 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 4.7 MiB/s wr, 631 op/s 2024-03-16T12:06:15.186 INFO:tasks.workunit.client.0.smithi029.stdout: 10 32754 16.27 MB/sec warmup 119 sec latency 296.410 ms 2024-03-16T12:06:16.620 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:06:16 smithi152 ceph-mon[36291]: pgmap v486: 97 pgs: 97 active+clean; 327 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 4.4 MiB/s wr, 588 op/s 2024-03-16T12:06:16.788 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:06:16 smithi154 ceph-mon[36320]: pgmap v486: 97 pgs: 97 active+clean; 327 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 4.4 MiB/s wr, 588 op/s 2024-03-16T12:06:16.830 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:06:16 smithi029 ceph-mon[31975]: pgmap v486: 97 pgs: 97 active+clean; 327 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 4.4 MiB/s wr, 588 op/s 2024-03-16T12:06:17.186 INFO:tasks.workunit.client.0.smithi029.stdout: 10 33208 19.63 MB/sec execute 1 sec latency 239.612 ms 2024-03-16T12:06:18.186 INFO:tasks.workunit.client.0.smithi029.stdout: 10 33483 17.93 MB/sec execute 2 sec latency 204.890 ms 2024-03-16T12:06:18.538 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:06:18 smithi154 ceph-mon[36320]: pgmap v487: 97 pgs: 97 active+clean; 337 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 5.8 MiB/s wr, 651 op/s 2024-03-16T12:06:18.538 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:06:18 smithi154 ceph-mon[36320]: osdmap e112: 12 total, 12 up, 12 in 2024-03-16T12:06:18.580 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:06:18 smithi029 ceph-mon[31975]: pgmap v487: 97 pgs: 97 active+clean; 337 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 5.8 MiB/s wr, 651 op/s 2024-03-16T12:06:18.581 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:06:18 smithi029 ceph-mon[31975]: osdmap e112: 12 total, 12 up, 12 in 2024-03-16T12:06:18.620 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:06:18 smithi152 ceph-mon[36291]: pgmap v487: 97 pgs: 97 active+clean; 337 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 5.8 MiB/s wr, 651 op/s 2024-03-16T12:06:18.620 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:06:18 smithi152 ceph-mon[36291]: osdmap e112: 12 total, 12 up, 12 in 2024-03-16T12:06:19.186 INFO:tasks.workunit.client.0.smithi029.stdout: 10 33636 16.06 MB/sec execute 3 sec latency 207.433 ms 2024-03-16T12:06:19.538 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:06:19 smithi154 ceph-mon[36320]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T12:06:19.538 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:06:19 smithi154 ceph-mon[36320]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:06:19.538 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:06:19 smithi154 ceph-mon[36320]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T12:06:19.538 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:06:19 smithi154 ceph-mon[36320]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' 2024-03-16T12:06:19.580 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:06:19 smithi029 ceph-mon[31975]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T12:06:19.581 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:06:19 smithi029 ceph-mon[31975]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:06:19.581 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:06:19 smithi029 ceph-mon[31975]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T12:06:19.581 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:06:19 smithi029 ceph-mon[31975]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' 2024-03-16T12:06:19.594 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:19.600+0000 7fa51efa9640 1 -- 172.21.15.29:0/458363882 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa5181078b0 msgr2=0x7fa518107c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:06:19.594 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:19.600+0000 7fa51efa9640 1 --2- 172.21.15.29:0/458363882 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa5181078b0 0x7fa518107c90 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7fa50c009930 tx=0x7fa50c02bc30 comp rx=0 tx=0).stop 2024-03-16T12:06:19.594 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:19.600+0000 7fa51efa9640 1 -- 172.21.15.29:0/458363882 shutdown_connections 2024-03-16T12:06:19.595 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:19.600+0000 7fa51efa9640 1 --2- 172.21.15.29:0/458363882 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa518069800 0x7fa51810e670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:06:19.595 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:19.600+0000 7fa51efa9640 1 --2- 172.21.15.29:0/458363882 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa518068e60 0x7fa5180692c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:06:19.595 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:19.600+0000 7fa51efa9640 1 --2- 172.21.15.29:0/458363882 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa5181078b0 0x7fa518107c90 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:06:19.595 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:19.600+0000 7fa51efa9640 1 -- 172.21.15.29:0/458363882 >> 172.21.15.29:0/458363882 conn(0x7fa518077530 msgr2=0x7fa518077940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:06:19.595 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:19.600+0000 7fa51efa9640 1 -- 172.21.15.29:0/458363882 shutdown_connections 2024-03-16T12:06:19.595 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:19.600+0000 7fa51efa9640 1 -- 172.21.15.29:0/458363882 wait complete. 2024-03-16T12:06:19.595 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:19.600+0000 7fa51efa9640 1 Processor -- start 2024-03-16T12:06:19.595 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:19.600+0000 7fa51efa9640 1 -- start start 2024-03-16T12:06:19.595 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:19.601+0000 7fa51efa9640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa518068e60 0x7fa518100a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:06:19.595 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:19.601+0000 7fa51efa9640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa518069800 0x7fa518100f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:06:19.595 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:19.601+0000 7fa51efa9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa5181078b0 0x7fa518101480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:06:19.595 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:19.601+0000 7fa51efa9640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fa518111ba0 con 0x7fa5181078b0 2024-03-16T12:06:19.596 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:19.601+0000 7fa51efa9640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fa518111a20 con 0x7fa518069800 2024-03-16T12:06:19.596 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:19.601+0000 7fa51efa9640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fa518111d20 con 0x7fa518068e60 2024-03-16T12:06:19.596 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:19.601+0000 7fa51d546640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa5181078b0 0x7fa518101480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:06:19.596 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:19.601+0000 7fa51d546640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa5181078b0 0x7fa518101480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51644/0 (socket says 172.21.15.29:51644) 2024-03-16T12:06:19.596 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:19.601+0000 7fa51d546640 1 -- 172.21.15.29:0/2031635710 learned_addr learned my addr 172.21.15.29:0/2031635710 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T12:06:19.596 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:19.601+0000 7fa517fff640 1 --2- 172.21.15.29:0/2031635710 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa518069800 0x7fa518100f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:06:19.596 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:19.601+0000 7fa51cd45640 1 --2- 172.21.15.29:0/2031635710 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa518068e60 0x7fa518100a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:06:19.596 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:19.601+0000 7fa51d546640 1 -- 172.21.15.29:0/2031635710 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa518068e60 msgr2=0x7fa518100a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:06:19.596 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:19.601+0000 7fa51d546640 1 --2- 172.21.15.29:0/2031635710 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa518068e60 0x7fa518100a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:06:19.596 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:19.601+0000 7fa51d546640 1 -- 172.21.15.29:0/2031635710 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa518069800 msgr2=0x7fa518100f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:06:19.596 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:19.601+0000 7fa51d546640 1 --2- 172.21.15.29:0/2031635710 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa518069800 0x7fa518100f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:06:19.596 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:19.601+0000 7fa51d546640 1 -- 172.21.15.29:0/2031635710 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5181a6500 con 0x7fa5181078b0 2024-03-16T12:06:19.596 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:19.601+0000 7fa51d546640 1 --2- 172.21.15.29:0/2031635710 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa5181078b0 0x7fa518101480 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7fa51000ea60 tx=0x7fa51000ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:06:19.597 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:19.601+0000 7fa515ffb640 1 -- 172.21.15.29:0/2031635710 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa51000ce40 con 0x7fa5181078b0 2024-03-16T12:06:19.597 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:19.601+0000 7fa515ffb640 1 -- 172.21.15.29:0/2031635710 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa510005020 con 0x7fa5181078b0 2024-03-16T12:06:19.597 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:19.602+0000 7fa51efa9640 1 -- 172.21.15.29:0/2031635710 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa5181a67f0 con 0x7fa5181078b0 2024-03-16T12:06:19.597 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:19.602+0000 7fa515ffb640 1 -- 172.21.15.29:0/2031635710 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa510010690 con 0x7fa5181078b0 2024-03-16T12:06:19.597 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:19.602+0000 7fa51efa9640 1 -- 172.21.15.29:0/2031635710 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa5181a6d40 con 0x7fa5181078b0 2024-03-16T12:06:19.600 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:19.604+0000 7fa515ffb640 1 -- 172.21.15.29:0/2031635710 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7fa510005520 con 0x7fa5181078b0 2024-03-16T12:06:19.600 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:19.604+0000 7fa521b8d740 1 Processor -- start 2024-03-16T12:06:19.600 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:19.604+0000 7fa515ffb640 1 --2- 172.21.15.29:0/2031635710 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7fa5080a2a10 0x7fa5080a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:06:19.600 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:19.604+0000 7fa521b8d740 1 -- start start 2024-03-16T12:06:19.600 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:19.604+0000 7fa517fff640 1 --2- 172.21.15.29:0/2031635710 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7fa5080a2a10 0x7fa5080a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:06:19.600 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:19.605+0000 7fa521b8d740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557891df0130 0x557891df0510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:06:19.600 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:19.605+0000 7fa515ffb640 1 -- 172.21.15.29:0/2031635710 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(112..112 src has 1..112) ==== 8556+0+0 (secure 0 0 0) 0x7fa5100cf280 con 0x7fa5181078b0 2024-03-16T12:06:19.600 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:19.605+0000 7fa521b8d740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557891df0a50 0x557891df92e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:06:19.600 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:19.605+0000 7fa521b8d740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557891df9820 0x557891dfbc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:06:19.601 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:19.605+0000 7fa521b8d740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x557891d54a40 con 0x557891df0130 2024-03-16T12:06:19.601 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:19.605+0000 7fa521b8d740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x557891cfd920 con 0x557891df0a50 2024-03-16T12:06:19.601 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:19.605+0000 7fa521b8d740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x557891d50ae0 con 0x557891df9820 2024-03-16T12:06:19.601 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:19.605+0000 7fa51cd45640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557891df0130 0x557891df0510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:06:19.601 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:19.605+0000 7fa517fff640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557891df0a50 0x557891df92e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:06:19.601 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:19.605+0000 7fa51cd45640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557891df0130 0x557891df0510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51660/0 (socket says 172.21.15.29:51660) 2024-03-16T12:06:19.601 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:19.605+0000 7fa51cd45640 1 -- 172.21.15.29:0/3053424736 learned_addr learned my addr 172.21.15.29:0/3053424736 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T12:06:19.601 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:19.605+0000 7fa51d546640 1 --2- 172.21.15.29:0/3053424736 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557891df9820 0x557891dfbc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:06:19.601 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:19.605+0000 7fa51cd45640 1 -- 172.21.15.29:0/3053424736 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557891df9820 msgr2=0x557891dfbc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:06:19.601 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:19.605+0000 7fa51cd45640 1 --2- 172.21.15.29:0/3053424736 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557891df9820 0x557891dfbc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:06:19.601 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:19.605+0000 7fa51cd45640 1 -- 172.21.15.29:0/3053424736 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557891df0a50 msgr2=0x557891df92e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:06:19.602 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:19.605+0000 7fa51cd45640 1 --2- 172.21.15.29:0/3053424736 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557891df0a50 0x557891df92e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:06:19.602 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:19.605+0000 7fa51cd45640 1 -- 172.21.15.29:0/3053424736 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557891d15460 con 0x557891df0130 2024-03-16T12:06:19.602 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:19.605+0000 7fa517fff640 1 --2- 172.21.15.29:0/2031635710 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7fa5080a2a10 0x7fa5080a4ed0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fa518102490 tx=0x7fa504014950 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:06:19.602 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:19.605+0000 7fa51cd45640 1 --2- 172.21.15.29:0/3053424736 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557891df0130 0x557891df0510 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7fa50c009900 tx=0x7fa50c005dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f3b0b62fde6be5e server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:06:19.602 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:19.606+0000 7fa502ffd640 1 -- 172.21.15.29:0/3053424736 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa50c042070 con 0x557891df0130 2024-03-16T12:06:19.602 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:19.606+0000 7fa502ffd640 1 -- 172.21.15.29:0/3053424736 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa50c02c6d0 con 0x557891df0130 2024-03-16T12:06:19.602 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:19.606+0000 7fa502ffd640 1 -- 172.21.15.29:0/3053424736 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa50c004c90 con 0x557891df0130 2024-03-16T12:06:19.602 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:19.607+0000 7fa521b8d740 1 -- 172.21.15.29:0/3053424736 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557891df0130 msgr2=0x557891df0510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:06:19.602 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:19.607+0000 7fa521b8d740 1 --2- 172.21.15.29:0/3053424736 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557891df0130 0x557891df0510 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7fa50c009900 tx=0x7fa50c005dd0 comp rx=0 tx=0).stop 2024-03-16T12:06:19.602 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:19.607+0000 7fa521b8d740 1 -- 172.21.15.29:0/3053424736 shutdown_connections 2024-03-16T12:06:19.602 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:19.607+0000 7fa521b8d740 1 --2- 172.21.15.29:0/3053424736 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557891df9820 0x557891dfbc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:06:19.602 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:19.607+0000 7fa521b8d740 1 --2- 172.21.15.29:0/3053424736 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557891df0a50 0x557891df92e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:06:19.603 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:19.607+0000 7fa521b8d740 1 --2- 172.21.15.29:0/3053424736 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557891df0130 0x557891df0510 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:06:19.603 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:19.607+0000 7fa521b8d740 1 -- 172.21.15.29:0/3053424736 >> 172.21.15.29:0/3053424736 conn(0x557891deaa50 msgr2=0x557891deae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:06:19.603 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:19.607+0000 7fa521b8d740 1 -- 172.21.15.29:0/3053424736 shutdown_connections 2024-03-16T12:06:19.603 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:19.607+0000 7fa521b8d740 1 -- 172.21.15.29:0/3053424736 wait complete. 2024-03-16T12:06:19.603 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:19.608+0000 7fa521b8d740 1 -- start start 2024-03-16T12:06:19.603 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:19.608+0000 7fa521b8d740 1 Processor -- start 2024-03-16T12:06:19.605 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:19.608+0000 7fa521b8d740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557891df0130 0x557891df6800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:06:19.606 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:19.608+0000 7fa521b8d740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557891df0a50 0x557891df6d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:06:19.606 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:19.608+0000 7fa521b8d740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557891df9820 0x557891df7280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:06:19.606 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:19.608+0000 7fa521b8d740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x557891dfc3f0 con 0x557891df0a50 2024-03-16T12:06:19.606 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:19.608+0000 7fa521b8d740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x557891d10d70 con 0x557891df9820 2024-03-16T12:06:19.606 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:19.608+0000 7fa521b8d740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x557891dfc570 con 0x557891df0130 2024-03-16T12:06:19.606 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:19.609+0000 7fa521b8d740 10 client.0 _ll_register_callbacks cb 0x557891a68a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T12:06:19.606 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:19.609+0000 7fa517fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557891df0a50 0x557891df6d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:06:19.606 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:19.609+0000 7fa51d546640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557891df9820 0x557891df7280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:06:19.606 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:19.609+0000 7fa51cd45640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557891df0130 0x557891df6800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:06:19.606 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:19.609+0000 7fa51cd45640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557891df0130 0x557891df6800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.29:49820/0 (socket says 172.21.15.29:49820) 2024-03-16T12:06:19.606 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:19.609+0000 7fa51cd45640 1 -- 172.21.15.29:0/4005249774 learned_addr learned my addr 172.21.15.29:0/4005249774 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T12:06:19.606 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:19.609+0000 7fa517fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557891df0a50 0x557891df6d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51670/0 (socket says 172.21.15.29:51670) 2024-03-16T12:06:19.606 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:19.609+0000 7fa51cd45640 1 -- 172.21.15.29:0/4005249774 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557891df9820 msgr2=0x557891df7280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:06:19.607 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:19.609+0000 7fa51cd45640 1 --2- 172.21.15.29:0/4005249774 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557891df9820 0x557891df7280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:06:19.607 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:19.609+0000 7fa51cd45640 1 -- 172.21.15.29:0/4005249774 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557891df0a50 msgr2=0x557891df6d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:06:19.607 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:19.609+0000 7fa51cd45640 1 --2- 172.21.15.29:0/4005249774 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557891df0a50 0x557891df6d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:06:19.607 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:19.609+0000 7fa51cd45640 1 -- 172.21.15.29:0/4005249774 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557891a69210 con 0x557891df0130 2024-03-16T12:06:19.607 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:19.609+0000 7fa51cd45640 1 --2- 172.21.15.29:0/4005249774 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557891df0130 0x557891df6800 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fa50c004950 tx=0x7fa50c02c8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a361f008d82670bc server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:06:19.607 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:19.609+0000 7fa4f7fff640 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-16T12:06:19.607 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:19.609+0000 7fa4f7fff640 1 -- 172.21.15.29:0/4005249774 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa50c04b070 con 0x557891df0130 2024-03-16T12:06:19.607 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:19.609+0000 7fa4f7fff640 1 -- 172.21.15.29:0/4005249774 <== mon.1 v2:172.21.15.154:3300/0 2 ==== osd_map(112..112 src has 1..112) ==== 8556+0+0 (secure 0 0 0) 0x7fa50c004f10 con 0x557891df0130 2024-03-16T12:06:19.607 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:19.609+0000 7fa521b8d740 1 -- 172.21.15.29:0/4005249774 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa50c02c6d0 con 0x557891df0130 2024-03-16T12:06:19.607 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:19.610+0000 7fa4f7fff640 1 -- 172.21.15.29:0/4005249774 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fa50c057050 con 0x557891df0130 2024-03-16T12:06:19.607 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:19.610+0000 7fa521b8d740 10 client.26509 fetch_fsmap learned FSMap version 31 2024-03-16T12:06:19.607 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:19.610+0000 7fa521b8d740 1 -- 172.21.15.29:0/4005249774 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x557891b35620 con 0x557891df0130 2024-03-16T12:06:19.607 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:19.610+0000 7fa4f7fff640 1 -- 172.21.15.29:0/4005249774 <== mon.1 v2:172.21.15.154:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa50c02c6d0 con 0x557891df0130 2024-03-16T12:06:19.607 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:19.610+0000 7fa4f7fff640 1 -- 172.21.15.29:0/4005249774 <== mon.1 v2:172.21.15.154:3300/0 5 ==== fsmap(e 31) ==== 8346+0+0 (secure 0 0 0) 0x7fa50c047c60 con 0x557891df0130 2024-03-16T12:06:19.608 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:19.610+0000 7fa521b8d740 10 client.26509 fetch_fsmap finished waiting for FSMap version 31 2024-03-16T12:06:19.608 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:19.610+0000 7fa521b8d740 10 client.26509 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T12:06:19.608 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:19.610+0000 7fa521b8d740 20 client.26509 populate_metadata read hostname 'smithi029' 2024-03-16T12:06:19.608 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:19.610+0000 7fa521b8d740 1 --2- 172.21.15.29:0/4005249774 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x557891e09780 0x557891e0bb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:06:19.608 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:19.610+0000 7fa521b8d740 4 client.26509 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T12:06:19.608 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:19.610+0000 7fa521b8d740 1 -- 172.21.15.29:0/4005249774 --> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557891d61f80 con 0x557891e09780 2024-03-16T12:06:19.608 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:19.611+0000 7fa517fff640 1 --2- 172.21.15.29:0/4005249774 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x557891e09780 0x557891e0bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:06:19.608 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:19.611+0000 7fa4f7fff640 10 client.26509 ms_handle_connect on v2:172.21.15.152:6838/1548731833 2024-03-16T12:06:19.608 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:19.611+0000 7fa517fff640 1 --2- 172.21.15.29:0/4005249774 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x557891e09780 0x557891e0bb70 crc :-1 s=READY pgs=176 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=785288268388d24b server_cookie=b24f7092b19445f0 in_seq=0 out_seq=0 2024-03-16T12:06:19.610 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:19.615+0000 7fa4f7fff640 1 -- 172.21.15.29:0/4005249774 <== mds.0 v2:172.21.15.152:6838/1548731833 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x557891d61f80 con 0x557891e09780 2024-03-16T12:06:19.610 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:19.616+0000 7fa4f7fff640 10 client.26509 handle_command_reply: tid=0 2024-03-16T12:06:19.610 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:19.616+0000 7fa521b8d740 1 client.26509 shutdown 2024-03-16T12:06:19.610 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:19.616+0000 7fa521b8d740 1 -- 172.21.15.29:0/4005249774 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557891df0130 msgr2=0x557891df6800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:06:19.610 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:19.616+0000 7fa521b8d740 1 --2- 172.21.15.29:0/4005249774 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557891df0130 0x557891df6800 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fa50c004950 tx=0x7fa50c02c8b0 comp rx=0 tx=0).stop 2024-03-16T12:06:19.610 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:19.616+0000 7fa521b8d740 1 -- 172.21.15.29:0/4005249774 shutdown_connections 2024-03-16T12:06:19.610 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:19.616+0000 7fa521b8d740 1 --2- 172.21.15.29:0/4005249774 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x557891e09780 0x557891e0bb70 crc :-1 s=READY pgs=176 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:06:19.610 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:19.616+0000 7fa521b8d740 1 --2- 172.21.15.29:0/4005249774 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557891df9820 0x557891df7280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:06:19.610 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:19.616+0000 7fa521b8d740 1 --2- 172.21.15.29:0/4005249774 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557891df0a50 0x557891df6d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:06:19.611 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:19.616+0000 7fa521b8d740 1 --2- 172.21.15.29:0/4005249774 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557891df0130 0x557891df6800 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:06:19.611 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:19.616+0000 7fa4f7fff640 0 client.26509 ms_handle_reset on v2:172.21.15.152:6838/1548731833 2024-03-16T12:06:19.611 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:19.616+0000 7fa521b8d740 1 -- 172.21.15.29:0/4005249774 >> 172.21.15.29:0/4005249774 conn(0x557891deaa50 msgr2=0x557891dec7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:06:19.611 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:19.616+0000 7fa521b8d740 1 -- 172.21.15.29:0/4005249774 shutdown_connections 2024-03-16T12:06:19.611 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:19.616+0000 7fa521b8d740 1 -- 172.21.15.29:0/4005249774 wait complete. 2024-03-16T12:06:19.611 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:19.616+0000 7fa521b8d740 20 client.26509 trim_cache size 0 max 16384 2024-03-16T12:06:19.620 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:06:19 smithi152 ceph-mon[36291]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T12:06:19.620 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:06:19 smithi152 ceph-mon[36291]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:06:19.620 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:06:19 smithi152 ceph-mon[36291]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T12:06:19.620 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:06:19 smithi152 ceph-mon[36291]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' 2024-03-16T12:06:19.625 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:19.632+0000 7fa521b8d740 1 Processor -- start 2024-03-16T12:06:19.625 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:19.632+0000 7fa521b8d740 1 -- start start 2024-03-16T12:06:19.625 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:19.632+0000 7fa521b8d740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557891df0130 0x557891e05190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:06:19.625 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:19.632+0000 7fa521b8d740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557891df0a50 0x557891e11260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:06:19.626 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:19.632+0000 7fa521b8d740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557891df9820 0x557891e017b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:06:19.626 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:19.632+0000 7fa521b8d740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fa5181b1f50 con 0x557891df0a50 2024-03-16T12:06:19.626 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:19.632+0000 7fa521b8d740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fa5181af7c0 con 0x557891df9820 2024-03-16T12:06:19.626 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:19.632+0000 7fa521b8d740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fa5181ad630 con 0x557891df0130 2024-03-16T12:06:19.626 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:19.632+0000 7fa517fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557891df0a50 0x557891e11260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:06:19.626 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:19.632+0000 7fa517fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557891df0a50 0x557891e11260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51678/0 (socket says 172.21.15.29:51678) 2024-03-16T12:06:19.626 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:19.632+0000 7fa517fff640 1 -- 172.21.15.29:0/1381251770 learned_addr learned my addr 172.21.15.29:0/1381251770 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T12:06:19.626 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:19.632+0000 7fa51cd45640 1 --2- 172.21.15.29:0/1381251770 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557891df0130 0x557891e05190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:06:19.626 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:19.632+0000 7fa51d546640 1 --2- 172.21.15.29:0/1381251770 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557891df9820 0x557891e017b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:06:19.626 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:19.633+0000 7fa517fff640 1 -- 172.21.15.29:0/1381251770 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557891df0130 msgr2=0x557891e05190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:06:19.626 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:19.633+0000 7fa517fff640 1 --2- 172.21.15.29:0/1381251770 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557891df0130 0x557891e05190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:06:19.626 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:19.633+0000 7fa517fff640 1 -- 172.21.15.29:0/1381251770 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557891df9820 msgr2=0x557891e017b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:06:19.626 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:19.633+0000 7fa517fff640 1 --2- 172.21.15.29:0/1381251770 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557891df9820 0x557891e017b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:06:19.627 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:19.633+0000 7fa517fff640 1 -- 172.21.15.29:0/1381251770 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557891e056d0 con 0x557891df0a50 2024-03-16T12:06:19.627 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:19.633+0000 7fa517fff640 1 --2- 172.21.15.29:0/1381251770 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557891df0a50 0x557891e11260 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7fa504005d00 tx=0x7fa50401e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4fbbc955a4504b99 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:06:19.627 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:19.633+0000 7fa5027fc640 1 -- 172.21.15.29:0/1381251770 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa504030070 con 0x557891df0a50 2024-03-16T12:06:19.627 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:19.633+0000 7fa5027fc640 1 -- 172.21.15.29:0/1381251770 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa50400bb50 con 0x557891df0a50 2024-03-16T12:06:19.627 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:19.633+0000 7fa5027fc640 1 -- 172.21.15.29:0/1381251770 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa504021cc0 con 0x557891df0a50 2024-03-16T12:06:19.630 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:19.634+0000 7fa521b8d740 1 -- 172.21.15.29:0/1381251770 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557891df0a50 msgr2=0x557891e11260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:06:19.630 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:19.634+0000 7fa521b8d740 1 --2- 172.21.15.29:0/1381251770 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557891df0a50 0x557891e11260 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7fa504005d00 tx=0x7fa50401e000 comp rx=0 tx=0).stop 2024-03-16T12:06:19.630 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:19.634+0000 7fa521b8d740 1 -- 172.21.15.29:0/1381251770 shutdown_connections 2024-03-16T12:06:19.631 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:19.634+0000 7fa521b8d740 1 --2- 172.21.15.29:0/1381251770 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557891df9820 0x557891e017b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:06:19.631 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:19.634+0000 7fa521b8d740 1 --2- 172.21.15.29:0/1381251770 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557891df0a50 0x557891e11260 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:06:19.631 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:19.634+0000 7fa521b8d740 1 --2- 172.21.15.29:0/1381251770 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557891df0130 0x557891e05190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:06:19.631 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:19.634+0000 7fa521b8d740 1 -- 172.21.15.29:0/1381251770 >> 172.21.15.29:0/1381251770 conn(0x557891deaa50 msgr2=0x557891df67c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:06:19.631 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:19.634+0000 7fa521b8d740 1 -- 172.21.15.29:0/1381251770 shutdown_connections 2024-03-16T12:06:19.631 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:19.634+0000 7fa521b8d740 1 -- 172.21.15.29:0/1381251770 wait complete. 2024-03-16T12:06:19.631 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:19.634+0000 7fa521b8d740 1 -- start start 2024-03-16T12:06:19.631 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:19.634+0000 7fa521b8d740 1 Processor -- start 2024-03-16T12:06:19.631 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:19.634+0000 7fa521b8d740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557891df0130 0x557891df3b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:06:19.631 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:19.634+0000 7fa521b8d740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557891df0a50 0x557891df40b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:06:19.631 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:19.634+0000 7fa521b8d740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557891df9820 0x557891e6e890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:06:19.631 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:19.635+0000 7fa521b8d740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fa5181adc80 con 0x557891df0130 2024-03-16T12:06:19.631 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:19.635+0000 7fa521b8d740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fa5181af0f0 con 0x557891df0a50 2024-03-16T12:06:19.631 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:19.635+0000 7fa521b8d740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fa5181ae350 con 0x557891df9820 2024-03-16T12:06:19.632 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:19.635+0000 7fa521b8d740 10 client.0 _ll_register_callbacks cb 0x7fa50400b960 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T12:06:19.632 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:19.635+0000 7fa51cd45640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557891df0130 0x557891df3b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:06:19.632 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:19.635+0000 7fa51cd45640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557891df0130 0x557891df3b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51688/0 (socket says 172.21.15.29:51688) 2024-03-16T12:06:19.632 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:19.635+0000 7fa51cd45640 1 -- 172.21.15.29:0/2463715023 learned_addr learned my addr 172.21.15.29:0/2463715023 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T12:06:19.632 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:19.635+0000 7fa517fff640 1 --2- 172.21.15.29:0/2463715023 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557891df0a50 0x557891df40b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:06:19.632 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:19.635+0000 7fa51d546640 1 --2- 172.21.15.29:0/2463715023 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557891df9820 0x557891e6e890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:06:19.632 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:19.635+0000 7fa51cd45640 1 -- 172.21.15.29:0/2463715023 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557891df9820 msgr2=0x557891e6e890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:06:19.632 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:19.635+0000 7fa51cd45640 1 --2- 172.21.15.29:0/2463715023 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557891df9820 0x557891e6e890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:06:19.632 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:19.635+0000 7fa51cd45640 1 -- 172.21.15.29:0/2463715023 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557891df0a50 msgr2=0x557891df40b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:06:19.632 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:19.635+0000 7fa51cd45640 1 --2- 172.21.15.29:0/2463715023 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557891df0a50 0x557891df40b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:06:19.632 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:19.635+0000 7fa51cd45640 1 -- 172.21.15.29:0/2463715023 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557891e6ee80 con 0x557891df0130 2024-03-16T12:06:19.632 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:19.635+0000 7fa51cd45640 1 --2- 172.21.15.29:0/2463715023 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557891df0130 0x557891df3b70 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7fa50c034490 tx=0x7fa50c000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb8dd4484aa44aa0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:06:19.632 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:19.635+0000 7fa4f7fff640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-16T12:06:19.632 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:19.635+0000 7fa4f7fff640 1 -- 172.21.15.29:0/2463715023 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa50c04b040 con 0x557891df0130 2024-03-16T12:06:19.633 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:19.635+0000 7fa4f7fff640 1 -- 172.21.15.29:0/2463715023 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(112..112 src has 1..112) ==== 8556+0+0 (secure 0 0 0) 0x7fa50c005320 con 0x557891df0130 2024-03-16T12:06:19.633 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:19.635+0000 7fa521b8d740 1 -- 172.21.15.29:0/2463715023 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa50400bb50 con 0x557891df0130 2024-03-16T12:06:19.633 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:19.635+0000 7fa4f7fff640 1 -- 172.21.15.29:0/2463715023 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fa50c042040 con 0x557891df0130 2024-03-16T12:06:19.633 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:19.635+0000 7fa521b8d740 10 client.16680 fetch_fsmap learned FSMap version 31 2024-03-16T12:06:19.633 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:19.635+0000 7fa521b8d740 1 -- 172.21.15.29:0/2463715023 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x557891e70170 con 0x557891df0130 2024-03-16T12:06:19.633 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:19.636+0000 7fa4f7fff640 1 -- 172.21.15.29:0/2463715023 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa50400bb50 con 0x557891df0130 2024-03-16T12:06:19.633 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:19.636+0000 7fa4f7fff640 1 -- 172.21.15.29:0/2463715023 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 31) ==== 8346+0+0 (secure 0 0 0) 0x7fa50c034ba0 con 0x557891df0130 2024-03-16T12:06:19.633 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:19.636+0000 7fa521b8d740 10 client.16680 fetch_fsmap finished waiting for FSMap version 31 2024-03-16T12:06:19.633 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:19.636+0000 7fa521b8d740 10 client.16680 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T12:06:19.633 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:19.636+0000 7fa521b8d740 20 client.16680 populate_metadata read hostname 'smithi029' 2024-03-16T12:06:19.633 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:19.636+0000 7fa521b8d740 1 --2- 172.21.15.29:0/2463715023 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x557891e70850 0x557891e72c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:06:19.633 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:19.636+0000 7fa521b8d740 4 client.16680 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-16T12:06:19.633 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:19.636+0000 7fa521b8d740 1 -- 172.21.15.29:0/2463715023 --> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] -- command(tid 0: {"prefix": "flush journal"}) -- 0x557891d5c2d0 con 0x557891e70850 2024-03-16T12:06:19.633 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:19.636+0000 7fa517fff640 1 --2- 172.21.15.29:0/2463715023 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x557891e70850 0x557891e72c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:06:19.634 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:19.636+0000 7fa517fff640 1 --2- 172.21.15.29:0/2463715023 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x557891e70850 0x557891e72c70 crc :-1 s=READY pgs=177 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cfac2876a07bbb43 server_cookie=89d518aae33afb5b in_seq=0 out_seq=0 2024-03-16T12:06:19.634 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:19.636+0000 7fa4f7fff640 10 client.16680 ms_handle_connect on v2:172.21.15.152:6838/1548731833 2024-03-16T12:06:20.189 INFO:tasks.workunit.client.0.smithi029.stdout: 10 33744 13.05 MB/sec execute 4 sec latency 337.564 ms 2024-03-16T12:06:20.461 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:20.468+0000 7fa4f7fff640 1 -- 172.21.15.29:0/2463715023 <== mds.0 v2:172.21.15.152:6838/1548731833 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x557891d5c2d0 con 0x557891e70850 2024-03-16T12:06:20.462 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:20.468+0000 7fa4f7fff640 10 client.16680 handle_command_reply: tid=0 2024-03-16T12:06:20.462 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:20.468+0000 7fa521b8d740 1 client.16680 shutdown 2024-03-16T12:06:20.462 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:20.468+0000 7fa521b8d740 1 -- 172.21.15.29:0/2463715023 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557891df0130 msgr2=0x557891df3b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:06:20.462 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:20.468+0000 7fa521b8d740 1 --2- 172.21.15.29:0/2463715023 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557891df0130 0x557891df3b70 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7fa50c034490 tx=0x7fa50c000bc0 comp rx=0 tx=0).stop 2024-03-16T12:06:20.462 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:20.469+0000 7fa521b8d740 1 -- 172.21.15.29:0/2463715023 shutdown_connections 2024-03-16T12:06:20.462 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:20.469+0000 7fa521b8d740 1 --2- 172.21.15.29:0/2463715023 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557891df9820 0x557891e6e890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:06:20.463 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:20.469+0000 7fa521b8d740 1 --2- 172.21.15.29:0/2463715023 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x557891e70850 0x557891e72c70 crc :-1 s=READY pgs=177 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:06:20.463 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:20.469+0000 7fa521b8d740 1 --2- 172.21.15.29:0/2463715023 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557891df0a50 0x557891df40b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:06:20.463 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:20.469+0000 7fa4f7fff640 0 client.16680 ms_handle_reset on v2:172.21.15.152:6838/1548731833 2024-03-16T12:06:20.463 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:20.469+0000 7fa521b8d740 1 --2- 172.21.15.29:0/2463715023 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557891df0130 0x557891df3b70 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:06:20.463 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:20.469+0000 7fa521b8d740 1 -- 172.21.15.29:0/2463715023 >> 172.21.15.29:0/2463715023 conn(0x557891deaa50 msgr2=0x557891df7030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:06:20.463 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:20.469+0000 7fa521b8d740 1 -- 172.21.15.29:0/2463715023 shutdown_connections 2024-03-16T12:06:20.463 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:20.469+0000 7fa521b8d740 1 -- 172.21.15.29:0/2463715023 wait complete. 2024-03-16T12:06:20.463 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:20.469+0000 7fa521b8d740 20 client.16680 trim_cache size 0 max 16384 2024-03-16T12:06:20.463 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:{ 2024-03-16T12:06:20.463 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "message": "", 2024-03-16T12:06:20.463 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "return_code": 0 2024-03-16T12:06:20.463 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:} 2024-03-16T12:06:20.464 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:20.469+0000 7fa51efa9640 1 -- 172.21.15.29:0/2031635710 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7fa5080a2a10 msgr2=0x7fa5080a4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:06:20.464 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:20.469+0000 7fa51efa9640 1 --2- 172.21.15.29:0/2031635710 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7fa5080a2a10 0x7fa5080a4ed0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fa518102490 tx=0x7fa504014950 comp rx=0 tx=0).stop 2024-03-16T12:06:20.464 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:20.469+0000 7fa51efa9640 1 -- 172.21.15.29:0/2031635710 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa5181078b0 msgr2=0x7fa518101480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:06:20.464 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:20.469+0000 7fa51efa9640 1 --2- 172.21.15.29:0/2031635710 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa5181078b0 0x7fa518101480 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7fa51000ea60 tx=0x7fa51000ef20 comp rx=0 tx=0).stop 2024-03-16T12:06:20.464 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:20.469+0000 7fa51efa9640 1 -- 172.21.15.29:0/2031635710 shutdown_connections 2024-03-16T12:06:20.465 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:20.469+0000 7fa51efa9640 1 --2- 172.21.15.29:0/2031635710 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7fa5080a2a10 0x7fa5080a4ed0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:06:20.465 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:20.469+0000 7fa51efa9640 1 --2- 172.21.15.29:0/2031635710 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa5181078b0 0x7fa518101480 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:06:20.465 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:20.469+0000 7fa51efa9640 1 --2- 172.21.15.29:0/2031635710 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa518069800 0x7fa518100f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:06:20.465 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:20.469+0000 7fa51efa9640 1 --2- 172.21.15.29:0/2031635710 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa518068e60 0x7fa518100a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:06:20.465 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:20.469+0000 7fa51efa9640 1 -- 172.21.15.29:0/2031635710 >> 172.21.15.29:0/2031635710 conn(0x7fa518077530 msgr2=0x7fa5180fe9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:06:20.465 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:20.470+0000 7fa51efa9640 1 -- 172.21.15.29:0/2031635710 shutdown_connections 2024-03-16T12:06:20.465 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:20.470+0000 7fa51efa9640 1 -- 172.21.15.29:0/2031635710 wait complete. 2024-03-16T12:06:20.538 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:06:20 smithi154 ceph-mon[36320]: pgmap v489: 97 pgs: 97 active+clean; 337 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 4.6 MiB/s wr, 435 op/s 2024-03-16T12:06:20.580 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:06:20 smithi029 ceph-mon[31975]: pgmap v489: 97 pgs: 97 active+clean; 337 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 4.6 MiB/s wr, 435 op/s 2024-03-16T12:06:20.620 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:06:20 smithi152 ceph-mon[36291]: pgmap v489: 97 pgs: 97 active+clean; 337 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 4.6 MiB/s wr, 435 op/s 2024-03-16T12:06:21.186 INFO:tasks.workunit.client.0.smithi029.stdout: 10 33843 10.91 MB/sec execute 5 sec latency 337.867 ms 2024-03-16T12:06:22.186 INFO:tasks.workunit.client.0.smithi029.stdout: 10 34038 10.19 MB/sec execute 6 sec latency 92.892 ms 2024-03-16T12:06:22.538 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:06:22 smithi154 ceph-mon[36320]: pgmap v490: 97 pgs: 97 active+clean; 344 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 5.6 MiB/s wr, 466 op/s 2024-03-16T12:06:22.580 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:06:22 smithi029 ceph-mon[31975]: pgmap v490: 97 pgs: 97 active+clean; 344 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 5.6 MiB/s wr, 466 op/s 2024-03-16T12:06:22.620 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:06:22 smithi152 ceph-mon[36291]: pgmap v490: 97 pgs: 97 active+clean; 344 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 5.6 MiB/s wr, 466 op/s 2024-03-16T12:06:23.186 INFO:tasks.workunit.client.0.smithi029.stdout: 10 34246 10.03 MB/sec execute 7 sec latency 270.420 ms 2024-03-16T12:06:24.186 INFO:tasks.workunit.client.0.smithi029.stdout: 10 34486 10.46 MB/sec execute 8 sec latency 379.804 ms 2024-03-16T12:06:24.538 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:06:24 smithi154 ceph-mon[36320]: pgmap v491: 97 pgs: 97 active+clean; 347 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 6.4 MiB/s wr, 465 op/s 2024-03-16T12:06:24.581 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:06:24 smithi029 ceph-mon[31975]: pgmap v491: 97 pgs: 97 active+clean; 347 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 6.4 MiB/s wr, 465 op/s 2024-03-16T12:06:24.620 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:06:24 smithi152 ceph-mon[36291]: pgmap v491: 97 pgs: 97 active+clean; 347 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 6.4 MiB/s wr, 465 op/s 2024-03-16T12:06:25.186 INFO:tasks.workunit.client.0.smithi029.stdout: 10 34812 9.95 MB/sec execute 9 sec latency 285.173 ms 2024-03-16T12:06:26.186 INFO:tasks.workunit.client.0.smithi029.stdout: 10 35115 10.60 MB/sec execute 10 sec latency 236.803 ms 2024-03-16T12:06:26.538 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:06:26 smithi154 ceph-mon[36320]: pgmap v492: 97 pgs: 97 active+clean; 347 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 6.4 MiB/s wr, 465 op/s 2024-03-16T12:06:26.580 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:06:26 smithi029 ceph-mon[31975]: pgmap v492: 97 pgs: 97 active+clean; 347 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 6.4 MiB/s wr, 465 op/s 2024-03-16T12:06:26.620 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:06:26 smithi152 ceph-mon[36291]: pgmap v492: 97 pgs: 97 active+clean; 347 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 6.4 MiB/s wr, 465 op/s 2024-03-16T12:06:27.186 INFO:tasks.workunit.client.0.smithi029.stdout: 10 35515 12.18 MB/sec execute 11 sec latency 272.221 ms 2024-03-16T12:06:28.186 INFO:tasks.workunit.client.0.smithi029.stdout: 10 36039 13.37 MB/sec execute 12 sec latency 217.835 ms 2024-03-16T12:06:28.538 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:06:28 smithi154 ceph-mon[36320]: pgmap v493: 97 pgs: 97 active+clean; 346 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 6.2 MiB/s wr, 468 op/s 2024-03-16T12:06:28.581 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:06:28 smithi029 ceph-mon[31975]: pgmap v493: 97 pgs: 97 active+clean; 346 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 6.2 MiB/s wr, 468 op/s 2024-03-16T12:06:28.620 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:06:28 smithi152 ceph-mon[36291]: pgmap v493: 97 pgs: 97 active+clean; 346 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 6.2 MiB/s wr, 468 op/s 2024-03-16T12:06:29.186 INFO:tasks.workunit.client.0.smithi029.stdout: 10 36378 13.84 MB/sec execute 13 sec latency 278.026 ms 2024-03-16T12:06:30.186 INFO:tasks.workunit.client.0.smithi029.stdout: 10 36682 14.50 MB/sec execute 14 sec latency 194.956 ms 2024-03-16T12:06:30.538 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:06:30 smithi154 ceph-mon[36320]: pgmap v494: 97 pgs: 97 active+clean; 346 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 5.2 MiB/s wr, 393 op/s 2024-03-16T12:06:30.580 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:06:30 smithi029 ceph-mon[31975]: pgmap v494: 97 pgs: 97 active+clean; 346 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 5.2 MiB/s wr, 393 op/s 2024-03-16T12:06:30.620 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:06:30 smithi152 ceph-mon[36291]: pgmap v494: 97 pgs: 97 active+clean; 346 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 5.2 MiB/s wr, 393 op/s 2024-03-16T12:06:31.186 INFO:tasks.workunit.client.0.smithi029.stdout: 10 36878 14.54 MB/sec execute 15 sec latency 236.193 ms 2024-03-16T12:06:32.186 INFO:tasks.workunit.client.0.smithi029.stdout: 10 37087 14.55 MB/sec execute 16 sec latency 323.100 ms 2024-03-16T12:06:32.538 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:06:32 smithi154 ceph-mon[36320]: pgmap v495: 97 pgs: 97 active+clean; 349 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 5.6 MiB/s wr, 423 op/s 2024-03-16T12:06:32.580 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:06:32 smithi029 ceph-mon[31975]: pgmap v495: 97 pgs: 97 active+clean; 349 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 5.6 MiB/s wr, 423 op/s 2024-03-16T12:06:32.620 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:06:32 smithi152 ceph-mon[36291]: pgmap v495: 97 pgs: 97 active+clean; 349 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 5.6 MiB/s wr, 423 op/s 2024-03-16T12:06:33.186 INFO:tasks.workunit.client.0.smithi029.stdout: 10 37273 14.29 MB/sec execute 17 sec latency 89.432 ms 2024-03-16T12:06:33.555 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:33.559+0000 7f319bac6640 1 -- 172.21.15.29:0/500431337 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3194068e60 msgr2=0x7f31940692c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:06:33.556 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:33.559+0000 7f319bac6640 1 --2- 172.21.15.29:0/500431337 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3194068e60 0x7f31940692c0 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f3188009900 tx=0x7f318802bc30 comp rx=0 tx=0).stop 2024-03-16T12:06:33.556 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:33.559+0000 7f319bac6640 1 -- 172.21.15.29:0/500431337 shutdown_connections 2024-03-16T12:06:33.556 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:33.559+0000 7f319bac6640 1 --2- 172.21.15.29:0/500431337 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3194069800 0x7f319410e670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:06:33.556 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:33.559+0000 7f319bac6640 1 --2- 172.21.15.29:0/500431337 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3194068e60 0x7f31940692c0 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:06:33.556 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:33.559+0000 7f319bac6640 1 --2- 172.21.15.29:0/500431337 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f31941078d0 0x7f3194107cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:06:33.556 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:33.559+0000 7f319bac6640 1 -- 172.21.15.29:0/500431337 >> 172.21.15.29:0/500431337 conn(0x7f3194077530 msgr2=0x7f3194077940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:06:33.556 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:33.559+0000 7f319bac6640 1 -- 172.21.15.29:0/500431337 shutdown_connections 2024-03-16T12:06:33.556 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:33.559+0000 7f319bac6640 1 -- 172.21.15.29:0/500431337 wait complete. 2024-03-16T12:06:33.556 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:33.560+0000 7f319bac6640 1 Processor -- start 2024-03-16T12:06:33.557 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:33.560+0000 7f319bac6640 1 -- start start 2024-03-16T12:06:33.557 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:33.560+0000 7f319bac6640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3194068e60 0x7f3194197ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:06:33.557 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:33.560+0000 7f319bac6640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3194069800 0x7f3194198530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:06:33.557 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:33.560+0000 7f319bac6640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f31941078d0 0x7f319419c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:06:33.557 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:33.560+0000 7f319bac6640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f3194111c50 con 0x7f3194068e60 2024-03-16T12:06:33.557 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:33.560+0000 7f319bac6640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f3194111ad0 con 0x7f31941078d0 2024-03-16T12:06:33.557 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:33.560+0000 7f319bac6640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f3194111dd0 con 0x7f3194069800 2024-03-16T12:06:33.557 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:33.560+0000 7f3199862640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3194068e60 0x7f3194197ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:06:33.557 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:33.560+0000 7f3199061640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3194069800 0x7f3194198530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:06:33.557 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:33.560+0000 7f3199862640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3194068e60 0x7f3194197ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42298/0 (socket says 172.21.15.29:42298) 2024-03-16T12:06:33.558 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:33.560+0000 7f3199862640 1 -- 172.21.15.29:0/207756851 learned_addr learned my addr 172.21.15.29:0/207756851 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T12:06:33.558 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:33.560+0000 7f319a063640 1 --2- 172.21.15.29:0/207756851 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f31941078d0 0x7f319419c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:06:33.558 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:33.561+0000 7f3199862640 1 -- 172.21.15.29:0/207756851 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3194069800 msgr2=0x7f3194198530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:06:33.558 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:33.561+0000 7f3199862640 1 --2- 172.21.15.29:0/207756851 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3194069800 0x7f3194198530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:06:33.558 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:33.561+0000 7f3199862640 1 -- 172.21.15.29:0/207756851 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f31941078d0 msgr2=0x7f319419c8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:06:33.558 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:33.561+0000 7f3199862640 1 --2- 172.21.15.29:0/207756851 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f31941078d0 0x7f319419c8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:06:33.558 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:33.561+0000 7f3199862640 1 -- 172.21.15.29:0/207756851 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f319419cfe0 con 0x7f3194068e60 2024-03-16T12:06:33.558 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:33.561+0000 7f3199061640 1 --2- 172.21.15.29:0/207756851 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3194069800 0x7f3194198530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:06:33.558 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:33.561+0000 7f3199862640 1 --2- 172.21.15.29:0/207756851 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3194068e60 0x7f3194197ff0 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f318c00d950 tx=0x7f318c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:06:33.558 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:33.561+0000 7f3192ffd640 1 -- 172.21.15.29:0/207756851 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f318c017030 con 0x7f3194068e60 2024-03-16T12:06:33.558 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:33.561+0000 7f3192ffd640 1 -- 172.21.15.29:0/207756851 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f318c004020 con 0x7f3194068e60 2024-03-16T12:06:33.559 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:33.561+0000 7f319bac6640 1 -- 172.21.15.29:0/207756851 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f319419d2d0 con 0x7f3194068e60 2024-03-16T12:06:33.559 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:33.561+0000 7f3192ffd640 1 -- 172.21.15.29:0/207756851 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f318c0047a0 con 0x7f3194068e60 2024-03-16T12:06:33.559 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:33.561+0000 7f319bac6640 1 -- 172.21.15.29:0/207756851 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f31941a4b60 con 0x7f3194068e60 2024-03-16T12:06:33.567 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:33.564+0000 7f3192ffd640 1 -- 172.21.15.29:0/207756851 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7f318c0041d0 con 0x7f3194068e60 2024-03-16T12:06:33.567 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:33.564+0000 7f319e78c740 1 Processor -- start 2024-03-16T12:06:33.567 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:33.564+0000 7f319e78c740 1 -- start start 2024-03-16T12:06:33.567 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:33.564+0000 7f319e78c740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5555ea6e6130 0x5555ea6e6510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:06:33.568 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:33.564+0000 7f3192ffd640 1 --2- 172.21.15.29:0/207756851 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f317c0a2a10 0x7f317c0a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:06:33.568 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:33.564+0000 7f319e78c740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5555ea6e6a50 0x5555ea6ef2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:06:33.568 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:33.564+0000 7f319e78c740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5555ea6ef820 0x5555ea6f1c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:06:33.568 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:33.564+0000 7f319e78c740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x5555ea64aa40 con 0x5555ea6ef820 2024-03-16T12:06:33.568 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:33.564+0000 7f319e78c740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x5555ea5f3920 con 0x5555ea6e6a50 2024-03-16T12:06:33.568 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:33.564+0000 7f319e78c740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5555ea646ae0 con 0x5555ea6e6130 2024-03-16T12:06:33.568 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:33.565+0000 7f3199862640 1 --2- 172.21.15.29:0/207756851 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f317c0a2a10 0x7f317c0a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:06:33.568 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:33.565+0000 7f3199061640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5555ea6e6a50 0x5555ea6ef2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:06:33.568 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:33.565+0000 7f3192ffd640 1 -- 172.21.15.29:0/207756851 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(112..112 src has 1..112) ==== 8556+0+0 (secure 0 0 0) 0x7f318c0ce820 con 0x7f3194068e60 2024-03-16T12:06:33.568 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:33.565+0000 7f3199061640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5555ea6e6a50 0x5555ea6ef2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.29:52194/0 (socket says 172.21.15.29:52194) 2024-03-16T12:06:33.568 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:33.565+0000 7f3199061640 1 -- 172.21.15.29:0/2590031637 learned_addr learned my addr 172.21.15.29:0/2590031637 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T12:06:33.569 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:33.565+0000 7f319a063640 1 --2- 172.21.15.29:0/2590031637 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5555ea6e6130 0x5555ea6e6510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:06:33.569 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:33.565+0000 7f319a063640 1 --2- 172.21.15.29:0/2590031637 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5555ea6ef820 0x5555ea6f1c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:06:33.569 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:33.565+0000 7f3199061640 1 -- 172.21.15.29:0/2590031637 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5555ea6e6130 msgr2=0x5555ea6e6510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:06:33.569 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:33.565+0000 7f3199061640 1 --2- 172.21.15.29:0/2590031637 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5555ea6e6130 0x5555ea6e6510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:06:33.569 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:33.565+0000 7f3199061640 1 -- 172.21.15.29:0/2590031637 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5555ea6ef820 msgr2=0x5555ea6f1c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:06:33.569 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:33.565+0000 7f3199061640 1 --2- 172.21.15.29:0/2590031637 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5555ea6ef820 0x5555ea6f1c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:06:33.569 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:33.565+0000 7f3199061640 1 -- 172.21.15.29:0/2590031637 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5555ea60b460 con 0x5555ea6e6a50 2024-03-16T12:06:33.569 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:33.565+0000 7f3199862640 1 --2- 172.21.15.29:0/207756851 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f317c0a2a10 0x7f317c0a4ed0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f318c09ab00 tx=0x7f318c069c10 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:06:33.569 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:33.565+0000 7f3199061640 1 --2- 172.21.15.29:0/2590031637 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5555ea6e6a50 0x5555ea6ef2e0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f318802c170 tx=0x7f3188009610 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a4d0c120772e2b25 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:06:33.569 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:33.566+0000 7f317bfff640 1 -- 172.21.15.29:0/2590031637 <== mon.2 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3188042070 con 0x5555ea6e6a50 2024-03-16T12:06:33.569 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:33.566+0000 7f317bfff640 1 -- 172.21.15.29:0/2590031637 <== mon.2 v2:172.21.15.152:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3188004c70 con 0x5555ea6e6a50 2024-03-16T12:06:33.570 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:33.566+0000 7f317bfff640 1 -- 172.21.15.29:0/2590031637 <== mon.2 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31880052a0 con 0x5555ea6e6a50 2024-03-16T12:06:33.570 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:33.566+0000 7f319e78c740 1 -- 172.21.15.29:0/2590031637 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5555ea6e6a50 msgr2=0x5555ea6ef2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:06:33.570 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:33.566+0000 7f319e78c740 1 --2- 172.21.15.29:0/2590031637 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5555ea6e6a50 0x5555ea6ef2e0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f318802c170 tx=0x7f3188009610 comp rx=0 tx=0).stop 2024-03-16T12:06:33.570 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:33.566+0000 7f319e78c740 1 -- 172.21.15.29:0/2590031637 shutdown_connections 2024-03-16T12:06:33.570 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:33.566+0000 7f319e78c740 1 --2- 172.21.15.29:0/2590031637 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5555ea6ef820 0x5555ea6f1c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:06:33.570 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:33.566+0000 7f319e78c740 1 --2- 172.21.15.29:0/2590031637 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5555ea6e6a50 0x5555ea6ef2e0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:06:33.570 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:33.566+0000 7f319e78c740 1 --2- 172.21.15.29:0/2590031637 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5555ea6e6130 0x5555ea6e6510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:06:33.570 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:33.566+0000 7f319e78c740 1 -- 172.21.15.29:0/2590031637 >> 172.21.15.29:0/2590031637 conn(0x5555ea6e0a50 msgr2=0x5555ea6e0e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:06:33.570 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:33.567+0000 7f319e78c740 1 -- 172.21.15.29:0/2590031637 shutdown_connections 2024-03-16T12:06:33.570 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:33.567+0000 7f319e78c740 1 -- 172.21.15.29:0/2590031637 wait complete. 2024-03-16T12:06:33.570 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:33.567+0000 7f319e78c740 1 -- start start 2024-03-16T12:06:33.571 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:33.567+0000 7f319e78c740 1 Processor -- start 2024-03-16T12:06:33.571 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:33.567+0000 7f319e78c740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5555ea6e6130 0x5555ea6ec850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:06:33.571 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:33.567+0000 7f319e78c740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5555ea6e6a50 0x5555ea6ecd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:06:33.571 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:33.567+0000 7f319e78c740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5555ea6ef820 0x5555ea6ed2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:06:33.571 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:33.567+0000 7f319e78c740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x5555ea6f23f0 con 0x5555ea6e6a50 2024-03-16T12:06:33.571 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:33.567+0000 7f319e78c740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x5555ea606d70 con 0x5555ea6e6130 2024-03-16T12:06:33.571 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:33.567+0000 7f319e78c740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5555ea6f2570 con 0x5555ea6ef820 2024-03-16T12:06:33.571 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:33.567+0000 7f319e78c740 10 client.0 _ll_register_callbacks cb 0x5555ea35ea00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T12:06:33.571 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:33.567+0000 7f3199061640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5555ea6e6a50 0x5555ea6ecd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:06:33.571 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:33.567+0000 7f3199061640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5555ea6e6a50 0x5555ea6ecd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42314/0 (socket says 172.21.15.29:42314) 2024-03-16T12:06:33.571 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:33.567+0000 7f3199061640 1 -- 172.21.15.29:0/2130818815 learned_addr learned my addr 172.21.15.29:0/2130818815 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T12:06:33.572 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:33.567+0000 7f319a063640 1 --2- 172.21.15.29:0/2130818815 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5555ea6ef820 0x5555ea6ed2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:06:33.572 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:33.568+0000 7f319a063640 1 --2- 172.21.15.29:0/2130818815 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5555ea6e6130 0x5555ea6ec850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:06:33.572 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:33.568+0000 7f3199061640 1 -- 172.21.15.29:0/2130818815 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5555ea6ef820 msgr2=0x5555ea6ed2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:06:33.572 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:33.568+0000 7f3199061640 1 --2- 172.21.15.29:0/2130818815 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5555ea6ef820 0x5555ea6ed2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:06:33.572 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:33.568+0000 7f3199061640 1 -- 172.21.15.29:0/2130818815 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5555ea6e6130 msgr2=0x5555ea6ec850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:06:33.572 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:33.568+0000 7f3199061640 1 --2- 172.21.15.29:0/2130818815 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5555ea6e6130 0x5555ea6ec850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:06:33.572 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:33.568+0000 7f3199061640 1 -- 172.21.15.29:0/2130818815 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5555ea35f210 con 0x5555ea6e6a50 2024-03-16T12:06:33.572 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:33.568+0000 7f3199061640 1 --2- 172.21.15.29:0/2130818815 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5555ea6e6a50 0x5555ea6ecd90 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f3188036040 tx=0x7f318802bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f081773ac57370e4 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:06:33.572 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:33.568+0000 7f31797fa640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-16T12:06:33.572 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:33.568+0000 7f31797fa640 1 -- 172.21.15.29:0/2130818815 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3188042070 con 0x5555ea6e6a50 2024-03-16T12:06:33.572 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:33.568+0000 7f31797fa640 1 -- 172.21.15.29:0/2130818815 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(112..112 src has 1..112) ==== 8556+0+0 (secure 0 0 0) 0x7f3188005230 con 0x5555ea6e6a50 2024-03-16T12:06:33.572 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:33.568+0000 7f319e78c740 1 -- 172.21.15.29:0/2130818815 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5555ea657f80 con 0x5555ea6e6a50 2024-03-16T12:06:33.573 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:33.568+0000 7f31797fa640 1 -- 172.21.15.29:0/2130818815 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f3188056050 con 0x5555ea6e6a50 2024-03-16T12:06:33.573 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:33.568+0000 7f319e78c740 10 client.16692 fetch_fsmap learned FSMap version 31 2024-03-16T12:06:33.573 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:33.568+0000 7f319e78c740 1 -- 172.21.15.29:0/2130818815 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5555ea42b620 con 0x5555ea6e6a50 2024-03-16T12:06:33.573 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:33.568+0000 7f31797fa640 1 -- 172.21.15.29:0/2130818815 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x5555ea657f80 con 0x5555ea6e6a50 2024-03-16T12:06:33.573 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:33.569+0000 7f31797fa640 1 -- 172.21.15.29:0/2130818815 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 31) ==== 8346+0+0 (secure 0 0 0) 0x7f318803e400 con 0x5555ea6e6a50 2024-03-16T12:06:33.573 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:33.569+0000 7f319e78c740 10 client.16692 fetch_fsmap finished waiting for FSMap version 31 2024-03-16T12:06:33.573 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:33.569+0000 7f319e78c740 10 client.16692 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T12:06:33.573 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:33.569+0000 7f319e78c740 20 client.16692 populate_metadata read hostname 'smithi029' 2024-03-16T12:06:33.573 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:33.569+0000 7f319e78c740 1 --2- 172.21.15.29:0/2130818815 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x5555ea6ff810 0x5555ea701c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:06:33.573 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:33.569+0000 7f319e78c740 4 client.16692 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T12:06:33.573 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:33.569+0000 7f319e78c740 1 -- 172.21.15.29:0/2130818815 --> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5555ea6522d0 con 0x5555ea6ff810 2024-03-16T12:06:33.574 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:33.569+0000 7f319a063640 1 --2- 172.21.15.29:0/2130818815 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x5555ea6ff810 0x5555ea701c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:06:33.574 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:33.569+0000 7f319a063640 1 --2- 172.21.15.29:0/2130818815 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x5555ea6ff810 0x5555ea701c00 crc :-1 s=READY pgs=178 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bd396e0695ea3250 server_cookie=bf1a4531af0367ab in_seq=0 out_seq=0 2024-03-16T12:06:33.574 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:33.569+0000 7f31797fa640 10 client.16692 ms_handle_connect on v2:172.21.15.152:6838/1548731833 2024-03-16T12:06:33.574 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:33.573+0000 7f31797fa640 1 -- 172.21.15.29:0/2130818815 <== mds.0 v2:172.21.15.152:6838/1548731833 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5555ea6522d0 con 0x5555ea6ff810 2024-03-16T12:06:33.574 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:33.573+0000 7f31797fa640 10 client.16692 handle_command_reply: tid=0 2024-03-16T12:06:33.574 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:33.573+0000 7f319e78c740 1 client.16692 shutdown 2024-03-16T12:06:33.574 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:33.573+0000 7f319e78c740 1 -- 172.21.15.29:0/2130818815 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5555ea6e6a50 msgr2=0x5555ea6ecd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:06:33.574 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:33.573+0000 7f319e78c740 1 --2- 172.21.15.29:0/2130818815 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5555ea6e6a50 0x5555ea6ecd90 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f3188036040 tx=0x7f318802bc30 comp rx=0 tx=0).stop 2024-03-16T12:06:33.574 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:33.573+0000 7f319e78c740 1 -- 172.21.15.29:0/2130818815 shutdown_connections 2024-03-16T12:06:33.574 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:33.573+0000 7f319e78c740 1 --2- 172.21.15.29:0/2130818815 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5555ea6ef820 0x5555ea6ed2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:06:33.574 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:33.573+0000 7f319e78c740 1 --2- 172.21.15.29:0/2130818815 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5555ea6e6a50 0x5555ea6ecd90 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:06:33.575 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:33.573+0000 7f319e78c740 1 --2- 172.21.15.29:0/2130818815 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x5555ea6ff810 0x5555ea701c00 crc :-1 s=READY pgs=178 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:06:33.575 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:33.573+0000 7f319e78c740 1 --2- 172.21.15.29:0/2130818815 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5555ea6e6130 0x5555ea6ec850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:06:33.575 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:33.573+0000 7f31797fa640 0 client.16692 ms_handle_reset on v2:172.21.15.152:6838/1548731833 2024-03-16T12:06:33.575 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:33.573+0000 7f319e78c740 1 -- 172.21.15.29:0/2130818815 >> 172.21.15.29:0/2130818815 conn(0x5555ea6e0a50 msgr2=0x5555ea6e27a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:06:33.575 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:33.573+0000 7f319e78c740 1 -- 172.21.15.29:0/2130818815 shutdown_connections 2024-03-16T12:06:33.575 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:33.574+0000 7f319e78c740 1 -- 172.21.15.29:0/2130818815 wait complete. 2024-03-16T12:06:33.575 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:33.574+0000 7f319e78c740 20 client.16692 trim_cache size 0 max 16384 2024-03-16T12:06:33.587 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:33.588+0000 7f319e78c740 1 Processor -- start 2024-03-16T12:06:33.587 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:33.588+0000 7f319e78c740 1 -- start start 2024-03-16T12:06:33.587 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:33.589+0000 7f319e78c740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5555ea6e6130 0x5555ea6eef80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:06:33.587 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:33.589+0000 7f319e78c740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5555ea6e6a50 0x5555ea6ecf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:06:33.587 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:33.589+0000 7f319e78c740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5555ea6ef820 0x5555ea6e9ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:06:33.587 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:33.589+0000 7f319e78c740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f31941afcf0 con 0x5555ea6ef820 2024-03-16T12:06:33.588 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:33.589+0000 7f319e78c740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f31941ad560 con 0x5555ea6e6130 2024-03-16T12:06:33.588 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:33.589+0000 7f319e78c740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f31941ab3d0 con 0x5555ea6e6a50 2024-03-16T12:06:33.588 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:33.589+0000 7f319a063640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5555ea6ef820 0x5555ea6e9ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:06:33.588 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:33.589+0000 7f319a063640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5555ea6ef820 0x5555ea6e9ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42324/0 (socket says 172.21.15.29:42324) 2024-03-16T12:06:33.588 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:33.589+0000 7f319a063640 1 -- 172.21.15.29:0/918881656 learned_addr learned my addr 172.21.15.29:0/918881656 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T12:06:33.588 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:33.589+0000 7f319a063640 1 --2- 172.21.15.29:0/918881656 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5555ea6e6130 0x5555ea6eef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:06:33.588 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:33.589+0000 7f3199061640 1 --2- 172.21.15.29:0/918881656 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5555ea6e6a50 0x5555ea6ecf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:06:33.588 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:33.589+0000 7f319a063640 1 -- 172.21.15.29:0/918881656 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5555ea6e6a50 msgr2=0x5555ea6ecf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:06:33.588 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:33.589+0000 7f319a063640 1 --2- 172.21.15.29:0/918881656 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5555ea6e6a50 0x5555ea6ecf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:06:33.588 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:33.589+0000 7f319a063640 1 -- 172.21.15.29:0/918881656 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5555ea6e6130 msgr2=0x5555ea6eef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:06:33.588 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:33.589+0000 7f319a063640 1 --2- 172.21.15.29:0/918881656 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5555ea6e6130 0x5555ea6eef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:06:33.589 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:33.589+0000 7f319a063640 1 -- 172.21.15.29:0/918881656 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5555ea6e1c10 con 0x5555ea6ef820 2024-03-16T12:06:33.589 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:33.589+0000 7f319a063640 1 --2- 172.21.15.29:0/918881656 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5555ea6ef820 0x5555ea6e9ab0 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x5555ea6e1e70 tx=0x7f3184009e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b836e8b0d8f4a050 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:06:33.589 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:33.589+0000 7f317b7fe640 1 -- 172.21.15.29:0/918881656 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3184023110 con 0x5555ea6ef820 2024-03-16T12:06:33.589 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:33.589+0000 7f317b7fe640 1 -- 172.21.15.29:0/918881656 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3184011020 con 0x5555ea6ef820 2024-03-16T12:06:33.589 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:33.589+0000 7f317b7fe640 1 -- 172.21.15.29:0/918881656 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31840056c0 con 0x5555ea6ef820 2024-03-16T12:06:33.589 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:33.590+0000 7f319e78c740 1 -- 172.21.15.29:0/918881656 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5555ea6ef820 msgr2=0x5555ea6e9ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:06:33.589 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:33.590+0000 7f319e78c740 1 --2- 172.21.15.29:0/918881656 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5555ea6ef820 0x5555ea6e9ab0 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x5555ea6e1e70 tx=0x7f3184009e60 comp rx=0 tx=0).stop 2024-03-16T12:06:33.589 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:33.590+0000 7f319e78c740 1 -- 172.21.15.29:0/918881656 shutdown_connections 2024-03-16T12:06:33.589 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:33.590+0000 7f319e78c740 1 --2- 172.21.15.29:0/918881656 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5555ea6ef820 0x5555ea6e9ab0 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:06:33.589 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:33.590+0000 7f319e78c740 1 --2- 172.21.15.29:0/918881656 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5555ea6e6a50 0x5555ea6ecf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:06:33.589 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:33.590+0000 7f319e78c740 1 --2- 172.21.15.29:0/918881656 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5555ea6e6130 0x5555ea6eef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:06:33.589 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:33.590+0000 7f319e78c740 1 -- 172.21.15.29:0/918881656 >> 172.21.15.29:0/918881656 conn(0x5555ea6e0a50 msgr2=0x5555ea6ec810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:06:33.590 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:33.590+0000 7f319e78c740 1 -- 172.21.15.29:0/918881656 shutdown_connections 2024-03-16T12:06:33.590 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:33.590+0000 7f319e78c740 1 -- 172.21.15.29:0/918881656 wait complete. 2024-03-16T12:06:33.590 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:33.590+0000 7f319e78c740 1 -- start start 2024-03-16T12:06:33.590 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:33.590+0000 7f319e78c740 1 Processor -- start 2024-03-16T12:06:33.590 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:33.591+0000 7f319e78c740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5555ea6e6130 0x5555ea702880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:06:33.590 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:33.591+0000 7f319e78c740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5555ea6e6a50 0x5555ea702dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:06:33.590 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:33.591+0000 7f319e78c740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5555ea6ef820 0x5555ea7647b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:06:33.590 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:33.591+0000 7f319e78c740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f31941aba20 con 0x5555ea6ef820 2024-03-16T12:06:33.590 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:33.591+0000 7f319e78c740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f31941ace90 con 0x5555ea6e6130 2024-03-16T12:06:33.590 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:33.591+0000 7f319e78c740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f31941ac0f0 con 0x5555ea6e6a50 2024-03-16T12:06:33.590 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:33.591+0000 7f319e78c740 10 client.0 _ll_register_callbacks cb 0x7f3184011540 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T12:06:33.590 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:33.591+0000 7f319a063640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5555ea6ef820 0x5555ea7647b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:06:33.591 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:33.591+0000 7f3199061640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5555ea6e6a50 0x5555ea702dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:06:33.591 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:33.591+0000 7f319a063640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5555ea6ef820 0x5555ea7647b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42332/0 (socket says 172.21.15.29:42332) 2024-03-16T12:06:33.591 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:33.591+0000 7f319a063640 1 -- 172.21.15.29:0/397139458 learned_addr learned my addr 172.21.15.29:0/397139458 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T12:06:33.591 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:33.591+0000 7f319a063640 1 --2- 172.21.15.29:0/397139458 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5555ea6e6130 0x5555ea702880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:06:33.591 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:33.591+0000 7f319a063640 1 -- 172.21.15.29:0/397139458 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5555ea6e6a50 msgr2=0x5555ea702dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:06:33.591 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:33.591+0000 7f319a063640 1 --2- 172.21.15.29:0/397139458 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5555ea6e6a50 0x5555ea702dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:06:33.591 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:33.591+0000 7f319a063640 1 -- 172.21.15.29:0/397139458 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5555ea6e6130 msgr2=0x5555ea702880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:06:33.591 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:33.591+0000 7f319a063640 1 --2- 172.21.15.29:0/397139458 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5555ea6e6130 0x5555ea702880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:06:33.591 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:33.591+0000 7f319a063640 1 -- 172.21.15.29:0/397139458 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5555ea703660 con 0x5555ea6ef820 2024-03-16T12:06:33.591 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:33.591+0000 7f3199061640 1 --2- 172.21.15.29:0/397139458 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5555ea6e6a50 0x5555ea702dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:06:33.591 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:33.592+0000 7f319a063640 1 --2- 172.21.15.29:0/397139458 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5555ea6ef820 0x5555ea7647b0 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f3184011d00 tx=0x7f31840052a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa0a02f4666bcaca server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:06:33.591 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:33.592+0000 7f31797fa640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-16T12:06:33.592 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:33.592+0000 7f31797fa640 1 -- 172.21.15.29:0/397139458 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31840111d0 con 0x5555ea6ef820 2024-03-16T12:06:33.592 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:33.592+0000 7f31797fa640 1 -- 172.21.15.29:0/397139458 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(112..112 src has 1..112) ==== 8556+0+0 (secure 0 0 0) 0x7f318401a040 con 0x5555ea6ef820 2024-03-16T12:06:33.592 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:33.592+0000 7f319e78c740 1 -- 172.21.15.29:0/397139458 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3184011020 con 0x5555ea6ef820 2024-03-16T12:06:33.592 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:33.592+0000 7f31797fa640 1 -- 172.21.15.29:0/397139458 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f3184023110 con 0x5555ea6ef820 2024-03-16T12:06:33.592 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:33.592+0000 7f319e78c740 10 client.16698 fetch_fsmap learned FSMap version 31 2024-03-16T12:06:33.592 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:33.592+0000 7f319e78c740 1 -- 172.21.15.29:0/397139458 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5555ea765dd0 con 0x5555ea6ef820 2024-03-16T12:06:33.592 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:33.592+0000 7f31797fa640 1 -- 172.21.15.29:0/397139458 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3184011020 con 0x5555ea6ef820 2024-03-16T12:06:33.592 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:33.592+0000 7f31797fa640 1 -- 172.21.15.29:0/397139458 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 31) ==== 8346+0+0 (secure 0 0 0) 0x7f3184003970 con 0x5555ea6ef820 2024-03-16T12:06:33.592 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:33.592+0000 7f319e78c740 10 client.16698 fetch_fsmap finished waiting for FSMap version 31 2024-03-16T12:06:33.592 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:33.592+0000 7f319e78c740 10 client.16698 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T12:06:33.592 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:33.592+0000 7f319e78c740 20 client.16698 populate_metadata read hostname 'smithi029' 2024-03-16T12:06:33.592 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:33.592+0000 7f319e78c740 1 --2- 172.21.15.29:0/397139458 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x5555ea7664d0 0x5555ea7688f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:06:33.593 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:33.592+0000 7f319e78c740 4 client.16698 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-16T12:06:33.593 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:33.592+0000 7f319e78c740 1 -- 172.21.15.29:0/397139458 --> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5555ea5d0770 con 0x5555ea7664d0 2024-03-16T12:06:33.593 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:33.592+0000 7f3199061640 1 --2- 172.21.15.29:0/397139458 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x5555ea7664d0 0x5555ea7688f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:06:33.593 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:33.594+0000 7f3199061640 1 --2- 172.21.15.29:0/397139458 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x5555ea7664d0 0x5555ea7688f0 crc :-1 s=READY pgs=179 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f76078105f56f627 server_cookie=edc4e04679b0ade8 in_seq=0 out_seq=0 2024-03-16T12:06:33.593 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:33.594+0000 7f31797fa640 10 client.16698 ms_handle_connect on v2:172.21.15.152:6838/1548731833 2024-03-16T12:06:34.186 INFO:tasks.workunit.client.0.smithi029.stdout: 10 37357 13.64 MB/sec execute 18 sec latency 457.531 ms 2024-03-16T12:06:34.261 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:34.267+0000 7f31797fa640 1 -- 172.21.15.29:0/397139458 <== mds.0 v2:172.21.15.152:6838/1548731833 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5555ea5d0770 con 0x5555ea7664d0 2024-03-16T12:06:34.261 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:34.267+0000 7f31797fa640 10 client.16698 handle_command_reply: tid=0 2024-03-16T12:06:34.261 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:34.267+0000 7f319e78c740 1 client.16698 shutdown 2024-03-16T12:06:34.261 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:34.267+0000 7f319e78c740 1 -- 172.21.15.29:0/397139458 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5555ea6ef820 msgr2=0x5555ea7647b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:06:34.261 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:34.267+0000 7f319e78c740 1 --2- 172.21.15.29:0/397139458 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5555ea6ef820 0x5555ea7647b0 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f3184011d00 tx=0x7f31840052a0 comp rx=0 tx=0).stop 2024-03-16T12:06:34.262 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:34.268+0000 7f319e78c740 1 -- 172.21.15.29:0/397139458 shutdown_connections 2024-03-16T12:06:34.262 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:34.268+0000 7f319e78c740 1 --2- 172.21.15.29:0/397139458 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5555ea6ef820 0x5555ea7647b0 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:06:34.262 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:34.268+0000 7f319e78c740 1 --2- 172.21.15.29:0/397139458 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5555ea6e6a50 0x5555ea702dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:06:34.262 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:34.268+0000 7f319e78c740 1 --2- 172.21.15.29:0/397139458 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x5555ea7664d0 0x5555ea7688f0 crc :-1 s=READY pgs=179 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:06:34.262 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:34.268+0000 7f319e78c740 1 --2- 172.21.15.29:0/397139458 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5555ea6e6130 0x5555ea702880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:06:34.262 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:34.268+0000 7f319e78c740 1 -- 172.21.15.29:0/397139458 >> 172.21.15.29:0/397139458 conn(0x5555ea6e0a50 msgr2=0x5555ea6dfbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:06:34.262 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:34.268+0000 7f31797fa640 0 client.16698 ms_handle_reset on v2:172.21.15.152:6838/1548731833 2024-03-16T12:06:34.262 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:34.268+0000 7f319e78c740 1 -- 172.21.15.29:0/397139458 shutdown_connections 2024-03-16T12:06:34.262 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:34.268+0000 7f319e78c740 1 -- 172.21.15.29:0/397139458 wait complete. 2024-03-16T12:06:34.262 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:34.268+0000 7f319e78c740 20 client.16698 trim_cache size 0 max 16384 2024-03-16T12:06:34.263 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:{ 2024-03-16T12:06:34.263 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "message": "", 2024-03-16T12:06:34.263 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "return_code": 0 2024-03-16T12:06:34.263 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:} 2024-03-16T12:06:34.264 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:34.268+0000 7f319bac6640 1 -- 172.21.15.29:0/207756851 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f317c0a2a10 msgr2=0x7f317c0a4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:06:34.264 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:34.268+0000 7f319bac6640 1 --2- 172.21.15.29:0/207756851 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f317c0a2a10 0x7f317c0a4ed0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f318c09ab00 tx=0x7f318c069c10 comp rx=0 tx=0).stop 2024-03-16T12:06:34.264 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:34.268+0000 7f319bac6640 1 -- 172.21.15.29:0/207756851 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3194068e60 msgr2=0x7f3194197ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:06:34.264 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:34.268+0000 7f319bac6640 1 --2- 172.21.15.29:0/207756851 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3194068e60 0x7f3194197ff0 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f318c00d950 tx=0x7f318c00de10 comp rx=0 tx=0).stop 2024-03-16T12:06:34.264 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:34.268+0000 7f319bac6640 1 -- 172.21.15.29:0/207756851 shutdown_connections 2024-03-16T12:06:34.264 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:34.268+0000 7f319bac6640 1 --2- 172.21.15.29:0/207756851 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f317c0a2a10 0x7f317c0a4ed0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:06:34.264 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:34.268+0000 7f319bac6640 1 --2- 172.21.15.29:0/207756851 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f31941078d0 0x7f319419c8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:06:34.265 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:34.268+0000 7f319bac6640 1 --2- 172.21.15.29:0/207756851 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3194069800 0x7f3194198530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:06:34.265 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:34.268+0000 7f319bac6640 1 --2- 172.21.15.29:0/207756851 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3194068e60 0x7f3194197ff0 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:06:34.265 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:34.268+0000 7f319bac6640 1 -- 172.21.15.29:0/207756851 >> 172.21.15.29:0/207756851 conn(0x7f3194077530 msgr2=0x7f31940fe940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:06:34.265 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:34.269+0000 7f319bac6640 1 -- 172.21.15.29:0/207756851 shutdown_connections 2024-03-16T12:06:34.265 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:34.269+0000 7f319bac6640 1 -- 172.21.15.29:0/207756851 wait complete. 2024-03-16T12:06:34.537 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:06:34 smithi154 ceph-mon[36320]: pgmap v496: 97 pgs: 97 active+clean; 354 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 5.3 MiB/s wr, 416 op/s 2024-03-16T12:06:34.580 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:06:34 smithi029 ceph-mon[31975]: pgmap v496: 97 pgs: 97 active+clean; 354 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 5.3 MiB/s wr, 416 op/s 2024-03-16T12:06:34.620 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:06:34 smithi152 ceph-mon[36291]: pgmap v496: 97 pgs: 97 active+clean; 354 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 5.3 MiB/s wr, 416 op/s 2024-03-16T12:06:35.187 INFO:tasks.workunit.client.0.smithi029.stdout: 10 37538 13.23 MB/sec execute 19 sec latency 389.742 ms 2024-03-16T12:06:36.186 INFO:tasks.workunit.client.0.smithi029.stdout: 10 37753 13.20 MB/sec execute 20 sec latency 260.284 ms 2024-03-16T12:06:36.538 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:06:36 smithi154 ceph-mon[36320]: pgmap v497: 97 pgs: 97 active+clean; 354 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 3.0 MiB/s wr, 228 op/s 2024-03-16T12:06:36.580 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:06:36 smithi029 ceph-mon[31975]: pgmap v497: 97 pgs: 97 active+clean; 354 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 3.0 MiB/s wr, 228 op/s 2024-03-16T12:06:36.620 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:06:36 smithi152 ceph-mon[36291]: pgmap v497: 97 pgs: 97 active+clean; 354 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 3.0 MiB/s wr, 228 op/s 2024-03-16T12:06:37.187 INFO:tasks.workunit.client.0.smithi029.stdout: 10 38033 12.87 MB/sec execute 21 sec latency 314.770 ms 2024-03-16T12:06:38.186 INFO:tasks.workunit.client.0.smithi029.stdout: 10 38417 12.84 MB/sec execute 22 sec latency 270.143 ms 2024-03-16T12:06:38.537 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:06:38 smithi154 ceph-mon[36320]: pgmap v498: 97 pgs: 97 active+clean; 342 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 MiB/s rd, 4.5 MiB/s wr, 495 op/s 2024-03-16T12:06:38.580 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:06:38 smithi029 ceph-mon[31975]: pgmap v498: 97 pgs: 97 active+clean; 342 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 MiB/s rd, 4.5 MiB/s wr, 495 op/s 2024-03-16T12:06:38.619 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:06:38 smithi152 ceph-mon[36291]: pgmap v498: 97 pgs: 97 active+clean; 342 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 MiB/s rd, 4.5 MiB/s wr, 495 op/s 2024-03-16T12:06:39.186 INFO:tasks.workunit.client.0.smithi029.stdout: 10 38778 13.44 MB/sec execute 23 sec latency 292.861 ms 2024-03-16T12:06:40.186 INFO:tasks.workunit.client.0.smithi029.stdout: 10 39297 14.02 MB/sec execute 24 sec latency 236.756 ms 2024-03-16T12:06:40.537 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:06:40 smithi154 ceph-mon[36320]: pgmap v499: 97 pgs: 97 active+clean; 342 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 2.7 MiB/s wr, 368 op/s 2024-03-16T12:06:40.580 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:06:40 smithi029 ceph-mon[31975]: pgmap v499: 97 pgs: 97 active+clean; 342 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 2.7 MiB/s wr, 368 op/s 2024-03-16T12:06:40.620 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:06:40 smithi152 ceph-mon[36291]: pgmap v499: 97 pgs: 97 active+clean; 342 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 2.7 MiB/s wr, 368 op/s 2024-03-16T12:06:41.186 INFO:tasks.workunit.client.0.smithi029.stdout: 10 39668 14.45 MB/sec execute 25 sec latency 221.848 ms 2024-03-16T12:06:42.186 INFO:tasks.workunit.client.0.smithi029.stdout: 10 39907 14.48 MB/sec execute 26 sec latency 111.549 ms 2024-03-16T12:06:42.537 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:06:42 smithi154 ceph-mon[36320]: pgmap v500: 97 pgs: 97 active+clean; 346 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 3.3 MiB/s wr, 402 op/s 2024-03-16T12:06:42.580 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:06:42 smithi029 ceph-mon[31975]: pgmap v500: 97 pgs: 97 active+clean; 346 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 3.3 MiB/s wr, 402 op/s 2024-03-16T12:06:42.619 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:06:42 smithi152 ceph-mon[36291]: pgmap v500: 97 pgs: 97 active+clean; 346 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 3.3 MiB/s wr, 402 op/s 2024-03-16T12:06:43.186 INFO:tasks.workunit.client.0.smithi029.stdout: 10 40173 14.72 MB/sec execute 27 sec latency 59.504 ms 2024-03-16T12:06:44.186 INFO:tasks.workunit.client.0.smithi029.stdout: 10 40349 14.49 MB/sec execute 28 sec latency 215.381 ms 2024-03-16T12:06:44.537 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:06:44 smithi154 ceph-mon[36320]: pgmap v501: 97 pgs: 97 active+clean; 350 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 MiB/s rd, 3.8 MiB/s wr, 437 op/s 2024-03-16T12:06:44.580 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:06:44 smithi029 ceph-mon[31975]: pgmap v501: 97 pgs: 97 active+clean; 350 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 MiB/s rd, 3.8 MiB/s wr, 437 op/s 2024-03-16T12:06:44.619 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:06:44 smithi152 ceph-mon[36291]: pgmap v501: 97 pgs: 97 active+clean; 350 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 MiB/s rd, 3.8 MiB/s wr, 437 op/s 2024-03-16T12:06:45.186 INFO:tasks.workunit.client.0.smithi029.stdout: 10 40530 14.38 MB/sec execute 29 sec latency 71.435 ms 2024-03-16T12:06:45.537 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:06:45 smithi154 ceph-mon[36320]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-03-16T12:06:45.580 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:06:45 smithi029 ceph-mon[31975]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-03-16T12:06:45.619 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:06:45 smithi152 ceph-mon[36291]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-03-16T12:06:46.186 INFO:tasks.workunit.client.0.smithi029.stdout: 10 40703 14.04 MB/sec execute 30 sec latency 79.370 ms 2024-03-16T12:06:46.537 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:06:46 smithi154 ceph-mon[36320]: pgmap v502: 97 pgs: 97 active+clean; 350 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 3.0 MiB/s wr, 369 op/s 2024-03-16T12:06:46.580 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:06:46 smithi029 ceph-mon[31975]: pgmap v502: 97 pgs: 97 active+clean; 350 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 3.0 MiB/s wr, 369 op/s 2024-03-16T12:06:46.619 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:06:46 smithi152 ceph-mon[36291]: pgmap v502: 97 pgs: 97 active+clean; 350 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 3.0 MiB/s wr, 369 op/s 2024-03-16T12:06:47.186 INFO:tasks.workunit.client.0.smithi029.stdout: 10 40827 13.83 MB/sec execute 31 sec latency 307.726 ms 2024-03-16T12:06:47.353 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:47.359+0000 7f0454e4b640 1 -- 172.21.15.29:0/589422778 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0450106720 msgr2=0x7f0450106b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:06:47.353 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:47.359+0000 7f0454e4b640 1 --2- 172.21.15.29:0/589422778 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0450106720 0x7f0450106b00 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f0440009900 tx=0x7f044002bc30 comp rx=0 tx=0).stop 2024-03-16T12:06:47.353 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:47.359+0000 7f0454e4b640 1 -- 172.21.15.29:0/589422778 shutdown_connections 2024-03-16T12:06:47.353 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:47.359+0000 7f0454e4b640 1 --2- 172.21.15.29:0/589422778 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0450101360 0x7f045010d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:06:47.353 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:47.359+0000 7f0454e4b640 1 --2- 172.21.15.29:0/589422778 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f04501009c0 0x7f0450100e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:06:47.353 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:47.359+0000 7f0454e4b640 1 --2- 172.21.15.29:0/589422778 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0450106720 0x7f0450106b00 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:06:47.353 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:47.359+0000 7f0454e4b640 1 -- 172.21.15.29:0/589422778 >> 172.21.15.29:0/589422778 conn(0x7f0450077400 msgr2=0x7f0450077810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:06:47.353 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:47.359+0000 7f0454e4b640 1 -- 172.21.15.29:0/589422778 shutdown_connections 2024-03-16T12:06:47.353 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:47.359+0000 7f0454e4b640 1 -- 172.21.15.29:0/589422778 wait complete. 2024-03-16T12:06:47.353 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:47.360+0000 7f0454e4b640 1 Processor -- start 2024-03-16T12:06:47.353 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:47.360+0000 7f0454e4b640 1 -- start start 2024-03-16T12:06:47.353 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:47.360+0000 7f0454e4b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f04501009c0 0x7f0450196e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:06:47.353 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:47.360+0000 7f0454e4b640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0450101360 0x7f04501973d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:06:47.354 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:47.360+0000 7f0454e4b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0450106720 0x7f045019b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:06:47.354 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:47.360+0000 7f0454e4b640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f0450110a90 con 0x7f0450106720 2024-03-16T12:06:47.354 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:47.360+0000 7f0454e4b640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f0450110910 con 0x7f0450101360 2024-03-16T12:06:47.354 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:47.360+0000 7f0454e4b640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f0450110c10 con 0x7f04501009c0 2024-03-16T12:06:47.354 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:47.360+0000 7f044ed9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0450106720 0x7f045019b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:06:47.354 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:47.360+0000 7f044e59c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f04501009c0 0x7f0450196e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:06:47.354 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:47.360+0000 7f044ed9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0450106720 0x7f045019b750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46660/0 (socket says 172.21.15.29:46660) 2024-03-16T12:06:47.354 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:47.360+0000 7f044ed9d640 1 -- 172.21.15.29:0/2699198738 learned_addr learned my addr 172.21.15.29:0/2699198738 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T12:06:47.354 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:47.360+0000 7f044dd9b640 1 --2- 172.21.15.29:0/2699198738 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0450101360 0x7f04501973d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:06:47.354 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:47.360+0000 7f044e59c640 1 -- 172.21.15.29:0/2699198738 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0450101360 msgr2=0x7f04501973d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:06:47.354 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:47.360+0000 7f044e59c640 1 --2- 172.21.15.29:0/2699198738 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0450101360 0x7f04501973d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:06:47.354 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:47.360+0000 7f044e59c640 1 -- 172.21.15.29:0/2699198738 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0450106720 msgr2=0x7f045019b750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:06:47.354 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:47.360+0000 7f044e59c640 1 --2- 172.21.15.29:0/2699198738 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0450106720 0x7f045019b750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:06:47.355 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:47.360+0000 7f044e59c640 1 -- 172.21.15.29:0/2699198738 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f045019be80 con 0x7f04501009c0 2024-03-16T12:06:47.355 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:47.360+0000 7f044ed9d640 1 --2- 172.21.15.29:0/2699198738 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0450106720 0x7f045019b750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:06:47.355 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:47.361+0000 7f044e59c640 1 --2- 172.21.15.29:0/2699198738 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f04501009c0 0x7f0450196e90 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f0440036040 tx=0x7f044002cbf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:06:47.355 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:47.361+0000 7f043f7fe640 1 -- 172.21.15.29:0/2699198738 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0440041070 con 0x7f04501009c0 2024-03-16T12:06:47.355 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:47.361+0000 7f0454e4b640 1 -- 172.21.15.29:0/2699198738 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f045019c110 con 0x7f04501009c0 2024-03-16T12:06:47.355 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:47.361+0000 7f043f7fe640 1 -- 172.21.15.29:0/2699198738 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0440004450 con 0x7f04501009c0 2024-03-16T12:06:47.355 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:47.361+0000 7f043f7fe640 1 -- 172.21.15.29:0/2699198738 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0440003e30 con 0x7f04501009c0 2024-03-16T12:06:47.355 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:47.361+0000 7f0454e4b640 1 -- 172.21.15.29:0/2699198738 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f04501a3a00 con 0x7f04501009c0 2024-03-16T12:06:47.361 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:47.363+0000 7f043f7fe640 1 -- 172.21.15.29:0/2699198738 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7f0440044070 con 0x7f04501009c0 2024-03-16T12:06:47.361 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:47.363+0000 7f0457a38740 1 Processor -- start 2024-03-16T12:06:47.361 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:47.363+0000 7f0457a38740 1 -- start start 2024-03-16T12:06:47.361 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:47.364+0000 7f0457a38740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564a2bcb93b0 0x564a2bcb9790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:06:47.361 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:47.364+0000 7f0457a38740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564a2bcb9d60 0x564a2bcb3b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:06:47.361 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:47.364+0000 7f0457a38740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564a2bcb4130 0x564a2bcb45b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:06:47.361 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:47.364+0000 7f0457a38740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x564a2bc14a40 con 0x564a2bcb93b0 2024-03-16T12:06:47.361 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:47.364+0000 7f0457a38740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x564a2bbbd920 con 0x564a2bcb4130 2024-03-16T12:06:47.361 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:47.364+0000 7f0457a38740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x564a2bc10ae0 con 0x564a2bcb9d60 2024-03-16T12:06:47.361 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:47.364+0000 7f044ed9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564a2bcb93b0 0x564a2bcb9790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:06:47.361 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:47.364+0000 7f044ed9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564a2bcb93b0 0x564a2bcb9790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46668/0 (socket says 172.21.15.29:46668) 2024-03-16T12:06:47.361 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:47.364+0000 7f044ed9d640 1 -- 172.21.15.29:0/154020779 learned_addr learned my addr 172.21.15.29:0/154020779 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T12:06:47.361 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:47.364+0000 7f043f7fe640 1 --2- 172.21.15.29:0/2699198738 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f04380a2a10 0x7f04380a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:06:47.362 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:47.364+0000 7f044e59c640 1 --2- 172.21.15.29:0/154020779 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564a2bcb9d60 0x564a2bcb3b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:06:47.362 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:47.364+0000 7f044dd9b640 1 --2- 172.21.15.29:0/154020779 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564a2bcb4130 0x564a2bcb45b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:06:47.362 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:47.364+0000 7f044ed9d640 1 --2- 172.21.15.29:0/2699198738 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f04380a2a10 0x7f04380a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:06:47.362 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:47.364+0000 7f044ed9d640 1 -- 172.21.15.29:0/154020779 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564a2bcb9d60 msgr2=0x564a2bcb3b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:06:47.362 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:47.364+0000 7f044ed9d640 1 --2- 172.21.15.29:0/154020779 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564a2bcb9d60 0x564a2bcb3b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:06:47.362 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:47.364+0000 7f044ed9d640 1 -- 172.21.15.29:0/154020779 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564a2bcb4130 msgr2=0x564a2bcb45b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:06:47.362 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:47.364+0000 7f044ed9d640 1 --2- 172.21.15.29:0/154020779 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564a2bcb4130 0x564a2bcb45b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:06:47.362 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:47.364+0000 7f044ed9d640 1 -- 172.21.15.29:0/154020779 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564a2bbd5460 con 0x564a2bcb93b0 2024-03-16T12:06:47.362 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:47.364+0000 7f044dd9b640 1 --2- 172.21.15.29:0/154020779 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564a2bcb4130 0x564a2bcb45b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:06:47.362 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:47.364+0000 7f043f7fe640 1 -- 172.21.15.29:0/2699198738 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(112..112 src has 1..112) ==== 8556+0+0 (secure 0 0 0) 0x7f04400f0b70 con 0x7f04501009c0 2024-03-16T12:06:47.362 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:47.364+0000 7f044ed9d640 1 --2- 172.21.15.29:0/154020779 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564a2bcb93b0 0x564a2bcb9790 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f044800feb0 tx=0x7f0448013830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97f052c065d2d48a server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:06:47.362 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:47.364+0000 7f044ed9d640 1 --2- 172.21.15.29:0/2699198738 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f04380a2a10 0x7f04380a4ed0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f0448013d30 tx=0x7f0448020000 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:06:47.362 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:47.364+0000 7f043cff9640 1 -- 172.21.15.29:0/154020779 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0448013130 con 0x564a2bcb93b0 2024-03-16T12:06:47.363 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:47.365+0000 7f043cff9640 1 -- 172.21.15.29:0/154020779 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0448023070 con 0x564a2bcb93b0 2024-03-16T12:06:47.363 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:47.365+0000 7f043cff9640 1 -- 172.21.15.29:0/154020779 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0448010690 con 0x564a2bcb93b0 2024-03-16T12:06:47.363 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:47.365+0000 7f0457a38740 1 -- 172.21.15.29:0/154020779 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564a2bcb93b0 msgr2=0x564a2bcb9790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:06:47.363 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:47.365+0000 7f0457a38740 1 --2- 172.21.15.29:0/154020779 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564a2bcb93b0 0x564a2bcb9790 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f044800feb0 tx=0x7f0448013830 comp rx=0 tx=0).stop 2024-03-16T12:06:47.363 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:47.365+0000 7f0457a38740 1 -- 172.21.15.29:0/154020779 shutdown_connections 2024-03-16T12:06:47.363 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:47.365+0000 7f0457a38740 1 --2- 172.21.15.29:0/154020779 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564a2bcb4130 0x564a2bcb45b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:06:47.363 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:47.365+0000 7f0457a38740 1 --2- 172.21.15.29:0/154020779 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564a2bcb9d60 0x564a2bcb3b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:06:47.363 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:47.365+0000 7f0457a38740 1 --2- 172.21.15.29:0/154020779 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564a2bcb93b0 0x564a2bcb9790 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:06:47.363 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:47.365+0000 7f0457a38740 1 -- 172.21.15.29:0/154020779 >> 172.21.15.29:0/154020779 conn(0x564a2bcaa790 msgr2=0x564a2bcaab70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:06:47.363 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:47.365+0000 7f0457a38740 1 -- 172.21.15.29:0/154020779 shutdown_connections 2024-03-16T12:06:47.363 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:47.365+0000 7f0457a38740 1 -- 172.21.15.29:0/154020779 wait complete. 2024-03-16T12:06:47.363 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:47.366+0000 7f0457a38740 1 -- start start 2024-03-16T12:06:47.363 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:47.366+0000 7f0457a38740 1 Processor -- start 2024-03-16T12:06:47.363 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:47.366+0000 7f0457a38740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564a2bcb4130 0x564a2bcbd2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:06:47.364 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:47.366+0000 7f0457a38740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564a2bcb93b0 0x564a2bcbd810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:06:47.364 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:47.366+0000 7f0457a38740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564a2bcb9d60 0x564a2bcbdd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:06:47.364 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:47.366+0000 7f0457a38740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x564a2bcc07a0 con 0x564a2bcb4130 2024-03-16T12:06:47.364 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:47.366+0000 7f0457a38740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x564a2bbd0d70 con 0x564a2bcb9d60 2024-03-16T12:06:47.364 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:47.366+0000 7f0457a38740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x564a2bcc0920 con 0x564a2bcb93b0 2024-03-16T12:06:47.364 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:47.366+0000 7f0457a38740 10 client.0 _ll_register_callbacks cb 0x564a2b928a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T12:06:47.364 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:47.366+0000 7f044e59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564a2bcb4130 0x564a2bcbd2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:06:47.364 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:47.366+0000 7f044e59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564a2bcb4130 0x564a2bcbd2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46674/0 (socket says 172.21.15.29:46674) 2024-03-16T12:06:47.364 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:47.366+0000 7f044e59c640 1 -- 172.21.15.29:0/1733863435 learned_addr learned my addr 172.21.15.29:0/1733863435 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T12:06:47.364 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:47.366+0000 7f044ed9d640 1 --2- 172.21.15.29:0/1733863435 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564a2bcb9d60 0x564a2bcbdd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:06:47.364 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:47.366+0000 7f044dd9b640 1 --2- 172.21.15.29:0/1733863435 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564a2bcb93b0 0x564a2bcbd810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:06:47.364 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:47.366+0000 7f044e59c640 1 -- 172.21.15.29:0/1733863435 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564a2bcb93b0 msgr2=0x564a2bcbd810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:06:47.364 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:47.366+0000 7f044e59c640 1 --2- 172.21.15.29:0/1733863435 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564a2bcb93b0 0x564a2bcbd810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:06:47.364 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:47.366+0000 7f044e59c640 1 -- 172.21.15.29:0/1733863435 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564a2bcb9d60 msgr2=0x564a2bcbdd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:06:47.365 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:47.366+0000 7f044e59c640 1 --2- 172.21.15.29:0/1733863435 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564a2bcb9d60 0x564a2bcbdd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:06:47.365 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:47.366+0000 7f044e59c640 1 -- 172.21.15.29:0/1733863435 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564a2b929210 con 0x564a2bcb4130 2024-03-16T12:06:47.365 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:47.366+0000 7f044dd9b640 1 --2- 172.21.15.29:0/1733863435 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564a2bcb93b0 0x564a2bcbd810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T12:06:47.365 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:47.366+0000 7f044ed9d640 1 --2- 172.21.15.29:0/1733863435 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564a2bcb9d60 0x564a2bcbdd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T12:06:47.365 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:47.367+0000 7f044e59c640 1 --2- 172.21.15.29:0/1733863435 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564a2bcb4130 0x564a2bcbd2d0 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f04400b9d20 tx=0x7f0440005b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da1ace2dc3449382 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:06:47.365 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:47.367+0000 7f0435ffb640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-16T12:06:47.365 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:47.367+0000 7f0435ffb640 1 -- 172.21.15.29:0/1733863435 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f044004b030 con 0x564a2bcb4130 2024-03-16T12:06:47.365 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:47.367+0000 7f0435ffb640 1 -- 172.21.15.29:0/1733863435 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(112..112 src has 1..112) ==== 8556+0+0 (secure 0 0 0) 0x7f04400ee500 con 0x564a2bcb4130 2024-03-16T12:06:47.365 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:47.367+0000 7f0457a38740 1 -- 172.21.15.29:0/1733863435 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0448023070 con 0x564a2bcb4130 2024-03-16T12:06:47.365 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:47.367+0000 7f0435ffb640 1 -- 172.21.15.29:0/1733863435 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f04400ee700 con 0x564a2bcb4130 2024-03-16T12:06:47.365 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:47.367+0000 7f0457a38740 10 client.16710 fetch_fsmap learned FSMap version 32 2024-03-16T12:06:47.365 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:47.367+0000 7f0457a38740 1 -- 172.21.15.29:0/1733863435 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x564a2b9f5620 con 0x564a2bcb4130 2024-03-16T12:06:47.365 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:47.367+0000 7f0435ffb640 1 -- 172.21.15.29:0/1733863435 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0448023070 con 0x564a2bcb4130 2024-03-16T12:06:47.365 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:47.367+0000 7f0435ffb640 1 -- 172.21.15.29:0/1733863435 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 32) ==== 8338+0+0 (secure 0 0 0) 0x7f04400bd5b0 con 0x564a2bcb4130 2024-03-16T12:06:47.365 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:47.367+0000 7f0457a38740 10 client.16710 fetch_fsmap finished waiting for FSMap version 32 2024-03-16T12:06:47.366 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:47.367+0000 7f0457a38740 10 client.16710 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T12:06:47.366 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:47.367+0000 7f0457a38740 20 client.16710 populate_metadata read hostname 'smithi029' 2024-03-16T12:06:47.366 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:47.367+0000 7f0457a38740 1 --2- 172.21.15.29:0/1733863435 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x564a2bccdd50 0x564a2bcd01a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:06:47.366 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:47.367+0000 7f0457a38740 4 client.16710 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T12:06:47.366 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:47.367+0000 7f0457a38740 1 -- 172.21.15.29:0/1733863435 --> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564a2bc21f80 con 0x564a2bccdd50 2024-03-16T12:06:47.366 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:47.368+0000 7f044dd9b640 1 --2- 172.21.15.29:0/1733863435 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x564a2bccdd50 0x564a2bcd01a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:06:47.366 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:47.368+0000 7f0435ffb640 10 client.16710 ms_handle_connect on v2:172.21.15.152:6838/1548731833 2024-03-16T12:06:47.366 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:47.368+0000 7f044dd9b640 1 --2- 172.21.15.29:0/1733863435 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x564a2bccdd50 0x564a2bcd01a0 crc :-1 s=READY pgs=180 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a7099c53b3c8a32a server_cookie=5c92092ffceb9d6d in_seq=0 out_seq=0 2024-03-16T12:06:47.367 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:47.372+0000 7f0435ffb640 1 -- 172.21.15.29:0/1733863435 <== mds.0 v2:172.21.15.152:6838/1548731833 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x564a2bc21f80 con 0x564a2bccdd50 2024-03-16T12:06:47.367 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:47.372+0000 7f0435ffb640 10 client.16710 handle_command_reply: tid=0 2024-03-16T12:06:47.367 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:47.372+0000 7f0457a38740 1 client.16710 shutdown 2024-03-16T12:06:47.367 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:47.372+0000 7f0457a38740 1 -- 172.21.15.29:0/1733863435 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564a2bcb4130 msgr2=0x564a2bcbd2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:06:47.367 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:47.372+0000 7f0457a38740 1 --2- 172.21.15.29:0/1733863435 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564a2bcb4130 0x564a2bcbd2d0 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f04400b9d20 tx=0x7f0440005b70 comp rx=0 tx=0).stop 2024-03-16T12:06:47.367 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:47.372+0000 7f0457a38740 1 -- 172.21.15.29:0/1733863435 shutdown_connections 2024-03-16T12:06:47.367 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:47.372+0000 7f0457a38740 1 --2- 172.21.15.29:0/1733863435 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x564a2bccdd50 0x564a2bcd01a0 crc :-1 s=READY pgs=180 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:06:47.368 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:47.372+0000 7f0457a38740 1 --2- 172.21.15.29:0/1733863435 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564a2bcb9d60 0x564a2bcbdd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:06:47.368 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:47.372+0000 7f0457a38740 1 --2- 172.21.15.29:0/1733863435 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564a2bcb93b0 0x564a2bcbd810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:06:47.368 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:47.372+0000 7f0457a38740 1 --2- 172.21.15.29:0/1733863435 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564a2bcb4130 0x564a2bcbd2d0 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:06:47.368 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:47.372+0000 7f0435ffb640 0 client.16710 ms_handle_reset on v2:172.21.15.152:6838/1548731833 2024-03-16T12:06:47.368 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:47.372+0000 7f0457a38740 1 -- 172.21.15.29:0/1733863435 >> 172.21.15.29:0/1733863435 conn(0x564a2bcaa790 msgr2=0x564a2bca9bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:06:47.368 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:47.372+0000 7f0457a38740 1 -- 172.21.15.29:0/1733863435 shutdown_connections 2024-03-16T12:06:47.368 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:47.372+0000 7f0457a38740 1 -- 172.21.15.29:0/1733863435 wait complete. 2024-03-16T12:06:47.368 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:47.373+0000 7f0457a38740 20 client.16710 trim_cache size 0 max 16384 2024-03-16T12:06:47.382 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:47.390+0000 7f0457a38740 1 Processor -- start 2024-03-16T12:06:47.382 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:47.390+0000 7f0457a38740 1 -- start start 2024-03-16T12:06:47.383 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:47.390+0000 7f0457a38740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564a2bcb4130 0x564a2bcbfa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:06:47.383 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:47.391+0000 7f0457a38740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564a2bcb93b0 0x564a2bcc9580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:06:47.383 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:47.391+0000 7f0457a38740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564a2bcb9d60 0x564a2bca9320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:06:47.383 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:47.391+0000 7f0457a38740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f04501adb20 con 0x564a2bcb4130 2024-03-16T12:06:47.383 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:47.391+0000 7f0457a38740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f04501ab3f0 con 0x564a2bcb93b0 2024-03-16T12:06:47.383 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:47.391+0000 7f0457a38740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f04501a9260 con 0x564a2bcb9d60 2024-03-16T12:06:47.383 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:47.391+0000 7f044ed9d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564a2bcb9d60 0x564a2bca9320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:06:47.383 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:47.391+0000 7f044ed9d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564a2bcb9d60 0x564a2bca9320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.29:50858/0 (socket says 172.21.15.29:50858) 2024-03-16T12:06:47.383 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:47.391+0000 7f044ed9d640 1 -- 172.21.15.29:0/2434609748 learned_addr learned my addr 172.21.15.29:0/2434609748 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T12:06:47.383 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:47.391+0000 7f044dd9b640 1 --2- 172.21.15.29:0/2434609748 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564a2bcb93b0 0x564a2bcc9580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:06:47.383 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:47.391+0000 7f044ed9d640 1 -- 172.21.15.29:0/2434609748 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564a2bcb93b0 msgr2=0x564a2bcc9580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:06:47.383 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:47.391+0000 7f044ed9d640 1 --2- 172.21.15.29:0/2434609748 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564a2bcb93b0 0x564a2bcc9580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:06:47.384 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:47.391+0000 7f044ed9d640 1 -- 172.21.15.29:0/2434609748 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564a2bcb4130 msgr2=0x564a2bcbfa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:06:47.384 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:47.391+0000 7f044ed9d640 1 --2- 172.21.15.29:0/2434609748 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564a2bcb4130 0x564a2bcbfa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:06:47.384 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:47.391+0000 7f044ed9d640 1 -- 172.21.15.29:0/2434609748 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564a2bcc9ac0 con 0x564a2bcb9d60 2024-03-16T12:06:47.387 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:47.392+0000 7f044ed9d640 1 --2- 172.21.15.29:0/2434609748 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564a2bcb9d60 0x564a2bca9320 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f0448013000 tx=0x7f0448015040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e42812e4f544615d server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:06:47.387 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:47.392+0000 7f0437fff640 1 -- 172.21.15.29:0/2434609748 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f044800f9d0 con 0x564a2bcb9d60 2024-03-16T12:06:47.387 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:47.392+0000 7f0437fff640 1 -- 172.21.15.29:0/2434609748 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0448023070 con 0x564a2bcb9d60 2024-03-16T12:06:47.387 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:47.392+0000 7f0437fff640 1 -- 172.21.15.29:0/2434609748 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0448005810 con 0x564a2bcb9d60 2024-03-16T12:06:47.387 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:47.393+0000 7f0457a38740 1 -- 172.21.15.29:0/2434609748 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564a2bcb9d60 msgr2=0x564a2bca9320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:06:47.387 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:47.393+0000 7f0457a38740 1 --2- 172.21.15.29:0/2434609748 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564a2bcb9d60 0x564a2bca9320 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f0448013000 tx=0x7f0448015040 comp rx=0 tx=0).stop 2024-03-16T12:06:47.387 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:47.393+0000 7f0457a38740 1 -- 172.21.15.29:0/2434609748 shutdown_connections 2024-03-16T12:06:47.387 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:47.393+0000 7f0457a38740 1 --2- 172.21.15.29:0/2434609748 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564a2bcb9d60 0x564a2bca9320 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:06:47.387 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:47.393+0000 7f0457a38740 1 --2- 172.21.15.29:0/2434609748 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564a2bcb93b0 0x564a2bcc9580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:06:47.387 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:47.393+0000 7f0457a38740 1 --2- 172.21.15.29:0/2434609748 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564a2bcb4130 0x564a2bcbfa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:06:47.387 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:47.393+0000 7f0457a38740 1 -- 172.21.15.29:0/2434609748 >> 172.21.15.29:0/2434609748 conn(0x564a2bcaa790 msgr2=0x564a2bcbbba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:06:47.388 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:47.393+0000 7f0457a38740 1 -- 172.21.15.29:0/2434609748 shutdown_connections 2024-03-16T12:06:47.388 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:47.393+0000 7f0457a38740 1 -- 172.21.15.29:0/2434609748 wait complete. 2024-03-16T12:06:47.388 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:47.393+0000 7f0457a38740 1 -- start start 2024-03-16T12:06:47.388 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:47.393+0000 7f0457a38740 1 Processor -- start 2024-03-16T12:06:47.388 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:47.394+0000 7f0457a38740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564a2bcb4130 0x564a2bcc5bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:06:47.388 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:47.394+0000 7f0457a38740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564a2bcb93b0 0x564a2bcaff40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:06:47.388 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:47.394+0000 7f0457a38740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564a2bcb9d60 0x564a2bcd5290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:06:47.388 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:47.394+0000 7f0457a38740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f04501a98b0 con 0x564a2bcb9d60 2024-03-16T12:06:47.388 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:47.394+0000 7f0457a38740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f04501aad20 con 0x564a2bcb4130 2024-03-16T12:06:47.388 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:47.394+0000 7f0457a38740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f04501a9f80 con 0x564a2bcb93b0 2024-03-16T12:06:47.388 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:47.394+0000 7f0457a38740 10 client.0 _ll_register_callbacks cb 0x7f0440009880 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T12:06:47.388 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:47.394+0000 7f044ed9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564a2bcb9d60 0x564a2bcd5290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:06:47.388 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:47.394+0000 7f044ed9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564a2bcb9d60 0x564a2bcd5290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46682/0 (socket says 172.21.15.29:46682) 2024-03-16T12:06:47.388 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:47.394+0000 7f044ed9d640 1 -- 172.21.15.29:0/1123298702 learned_addr learned my addr 172.21.15.29:0/1123298702 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T12:06:47.388 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:47.394+0000 7f044dd9b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564a2bcb93b0 0x564a2bcaff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:06:47.389 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:47.394+0000 7f044e59c640 1 --2- 172.21.15.29:0/1123298702 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564a2bcb4130 0x564a2bcc5bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:06:47.389 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:47.394+0000 7f044ed9d640 1 -- 172.21.15.29:0/1123298702 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564a2bcb93b0 msgr2=0x564a2bcaff40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:06:47.389 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:47.394+0000 7f044ed9d640 1 --2- 172.21.15.29:0/1123298702 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564a2bcb93b0 0x564a2bcaff40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:06:47.389 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:47.394+0000 7f044ed9d640 1 -- 172.21.15.29:0/1123298702 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564a2bcb4130 msgr2=0x564a2bcc5bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:06:47.389 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:47.394+0000 7f044ed9d640 1 --2- 172.21.15.29:0/1123298702 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564a2bcb4130 0x564a2bcc5bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:06:47.389 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:47.394+0000 7f044ed9d640 1 -- 172.21.15.29:0/1123298702 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564a2bcb0480 con 0x564a2bcb9d60 2024-03-16T12:06:47.389 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:47.394+0000 7f044ed9d640 1 --2- 172.21.15.29:0/1123298702 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564a2bcb9d60 0x564a2bcd5290 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f0448031040 tx=0x7f04480054b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f554f71b65064e3c server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:06:47.389 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:47.394+0000 7f0435ffb640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-16T12:06:47.389 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:47.395+0000 7f0435ffb640 1 -- 172.21.15.29:0/1123298702 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f044803e070 con 0x564a2bcb9d60 2024-03-16T12:06:47.389 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:47.395+0000 7f0435ffb640 1 -- 172.21.15.29:0/1123298702 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(112..112 src has 1..112) ==== 8556+0+0 (secure 0 0 0) 0x7f04480049d0 con 0x564a2bcb9d60 2024-03-16T12:06:47.389 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:47.395+0000 7f0457a38740 1 -- 172.21.15.29:0/1123298702 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0448023070 con 0x564a2bcb9d60 2024-03-16T12:06:47.389 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:47.395+0000 7f0435ffb640 1 -- 172.21.15.29:0/1123298702 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f04480059a0 con 0x564a2bcb9d60 2024-03-16T12:06:47.389 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:47.395+0000 7f0457a38740 10 client.16713 fetch_fsmap learned FSMap version 32 2024-03-16T12:06:47.389 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:47.395+0000 7f0457a38740 1 -- 172.21.15.29:0/1123298702 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x564a2bd38c80 con 0x564a2bcb9d60 2024-03-16T12:06:47.389 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:47.395+0000 7f0435ffb640 1 -- 172.21.15.29:0/1123298702 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0448023070 con 0x564a2bcb9d60 2024-03-16T12:06:47.390 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:47.395+0000 7f0435ffb640 1 -- 172.21.15.29:0/1123298702 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 32) ==== 8338+0+0 (secure 0 0 0) 0x7f044801f5a0 con 0x564a2bcb9d60 2024-03-16T12:06:47.390 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:47.395+0000 7f0457a38740 10 client.16713 fetch_fsmap finished waiting for FSMap version 32 2024-03-16T12:06:47.390 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:47.395+0000 7f0457a38740 10 client.16713 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T12:06:47.390 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:47.395+0000 7f0457a38740 20 client.16713 populate_metadata read hostname 'smithi029' 2024-03-16T12:06:47.390 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:47.395+0000 7f0457a38740 1 --2- 172.21.15.29:0/1123298702 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x564a2bd39100 0x564a2bd3b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:06:47.390 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:47.395+0000 7f0457a38740 4 client.16713 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-16T12:06:47.390 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:47.395+0000 7f0457a38740 1 -- 172.21.15.29:0/1123298702 --> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] -- command(tid 0: {"prefix": "flush journal"}) -- 0x564a2bc1c2d0 con 0x564a2bd39100 2024-03-16T12:06:47.390 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:47.395+0000 7f044e59c640 1 --2- 172.21.15.29:0/1123298702 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x564a2bd39100 0x564a2bd3b4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:06:47.390 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:47.396+0000 7f044e59c640 1 --2- 172.21.15.29:0/1123298702 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x564a2bd39100 0x564a2bd3b4f0 crc :-1 s=READY pgs=181 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=917c10fb0a8fba54 server_cookie=6ea7c00ed3c6eb16 in_seq=0 out_seq=0 2024-03-16T12:06:47.390 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:47.396+0000 7f0435ffb640 10 client.16713 ms_handle_connect on v2:172.21.15.152:6838/1548731833 2024-03-16T12:06:48.007 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:48.015+0000 7f0435ffb640 1 -- 172.21.15.29:0/1123298702 <== mds.0 v2:172.21.15.152:6838/1548731833 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x564a2bc1c2d0 con 0x564a2bd39100 2024-03-16T12:06:48.008 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:48.015+0000 7f0435ffb640 10 client.16713 handle_command_reply: tid=0 2024-03-16T12:06:48.008 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:48.015+0000 7f0457a38740 1 client.16713 shutdown 2024-03-16T12:06:48.008 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:48.015+0000 7f0457a38740 1 -- 172.21.15.29:0/1123298702 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564a2bcb9d60 msgr2=0x564a2bcd5290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:06:48.008 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:48.015+0000 7f0457a38740 1 --2- 172.21.15.29:0/1123298702 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564a2bcb9d60 0x564a2bcd5290 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f0448031040 tx=0x7f04480054b0 comp rx=0 tx=0).stop 2024-03-16T12:06:48.008 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:48.015+0000 7f0457a38740 1 -- 172.21.15.29:0/1123298702 shutdown_connections 2024-03-16T12:06:48.008 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:48.015+0000 7f0457a38740 1 --2- 172.21.15.29:0/1123298702 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564a2bcb9d60 0x564a2bcd5290 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:06:48.008 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:48.015+0000 7f0457a38740 1 --2- 172.21.15.29:0/1123298702 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564a2bcb93b0 0x564a2bcaff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:06:48.008 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:48.015+0000 7f0457a38740 1 --2- 172.21.15.29:0/1123298702 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x564a2bd39100 0x564a2bd3b4f0 crc :-1 s=READY pgs=181 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:06:48.008 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:48.015+0000 7f0457a38740 1 --2- 172.21.15.29:0/1123298702 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564a2bcb4130 0x564a2bcc5bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:06:48.008 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:48.015+0000 7f0435ffb640 0 client.16713 ms_handle_reset on v2:172.21.15.152:6838/1548731833 2024-03-16T12:06:48.008 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:48.015+0000 7f0457a38740 1 -- 172.21.15.29:0/1123298702 >> 172.21.15.29:0/1123298702 conn(0x564a2bcaa790 msgr2=0x564a2bcbdb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:06:48.008 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:48.016+0000 7f0457a38740 1 -- 172.21.15.29:0/1123298702 shutdown_connections 2024-03-16T12:06:48.008 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:48.016+0000 7f0457a38740 1 -- 172.21.15.29:0/1123298702 wait complete. 2024-03-16T12:06:48.009 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:48.016+0000 7f0457a38740 20 client.16713 trim_cache size 0 max 16384 2024-03-16T12:06:48.009 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:{ 2024-03-16T12:06:48.009 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "message": "", 2024-03-16T12:06:48.009 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "return_code": 0 2024-03-16T12:06:48.009 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:} 2024-03-16T12:06:48.010 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:48.016+0000 7f0454e4b640 1 -- 172.21.15.29:0/2699198738 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f04380a2a10 msgr2=0x7f04380a4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:06:48.010 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:48.016+0000 7f0454e4b640 1 --2- 172.21.15.29:0/2699198738 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f04380a2a10 0x7f04380a4ed0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f0448013d30 tx=0x7f0448020000 comp rx=0 tx=0).stop 2024-03-16T12:06:48.010 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:48.016+0000 7f0454e4b640 1 -- 172.21.15.29:0/2699198738 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f04501009c0 msgr2=0x7f0450196e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:06:48.010 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:48.016+0000 7f0454e4b640 1 --2- 172.21.15.29:0/2699198738 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f04501009c0 0x7f0450196e90 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f0440036040 tx=0x7f044002cbf0 comp rx=0 tx=0).stop 2024-03-16T12:06:48.010 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:48.016+0000 7f0454e4b640 1 -- 172.21.15.29:0/2699198738 shutdown_connections 2024-03-16T12:06:48.010 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:48.016+0000 7f0454e4b640 1 --2- 172.21.15.29:0/2699198738 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f04380a2a10 0x7f04380a4ed0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:06:48.010 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:48.016+0000 7f0454e4b640 1 --2- 172.21.15.29:0/2699198738 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0450106720 0x7f045019b750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:06:48.010 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:48.016+0000 7f0454e4b640 1 --2- 172.21.15.29:0/2699198738 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0450101360 0x7f04501973d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:06:48.010 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:48.016+0000 7f0454e4b640 1 --2- 172.21.15.29:0/2699198738 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f04501009c0 0x7f0450196e90 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:06:48.010 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:48.016+0000 7f0454e4b640 1 -- 172.21.15.29:0/2699198738 >> 172.21.15.29:0/2699198738 conn(0x7f0450077400 msgr2=0x7f04500ff440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:06:48.010 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:48.016+0000 7f0454e4b640 1 -- 172.21.15.29:0/2699198738 shutdown_connections 2024-03-16T12:06:48.011 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:06:48.016+0000 7f0454e4b640 1 -- 172.21.15.29:0/2699198738 wait complete. 2024-03-16T12:06:48.186 INFO:tasks.workunit.client.0.smithi029.stdout: 10 40954 13.61 MB/sec execute 32 sec latency 422.332 ms 2024-03-16T12:06:48.537 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:06:48 smithi154 ceph-mon[36320]: pgmap v503: 97 pgs: 97 active+clean; 356 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 MiB/s rd, 4.3 MiB/s wr, 455 op/s 2024-03-16T12:06:48.580 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:06:48 smithi029 ceph-mon[31975]: pgmap v503: 97 pgs: 97 active+clean; 356 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 MiB/s rd, 4.3 MiB/s wr, 455 op/s 2024-03-16T12:06:48.619 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:06:48 smithi152 ceph-mon[36291]: pgmap v503: 97 pgs: 97 active+clean; 356 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 MiB/s rd, 4.3 MiB/s wr, 455 op/s 2024-03-16T12:06:49.187 INFO:tasks.workunit.client.0.smithi029.stdout: 10 41215 13.55 MB/sec execute 33 sec latency 243.435 ms 2024-03-16T12:06:50.187 INFO:tasks.workunit.client.0.smithi029.stdout: 10 41586 13.39 MB/sec execute 34 sec latency 253.065 ms 2024-03-16T12:06:50.537 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:06:50 smithi154 ceph-mon[36320]: pgmap v504: 97 pgs: 97 active+clean; 356 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 2.8 MiB/s wr, 188 op/s 2024-03-16T12:06:50.580 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:06:50 smithi029 ceph-mon[31975]: pgmap v504: 97 pgs: 97 active+clean; 356 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 2.8 MiB/s wr, 188 op/s 2024-03-16T12:06:50.619 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:06:50 smithi152 ceph-mon[36291]: pgmap v504: 97 pgs: 97 active+clean; 356 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 2.8 MiB/s wr, 188 op/s 2024-03-16T12:06:51.186 INFO:tasks.workunit.client.0.smithi029.stdout: 10 41935 13.51 MB/sec execute 35 sec latency 240.947 ms 2024-03-16T12:06:52.187 INFO:tasks.workunit.client.0.smithi029.stdout: 10 42323 13.79 MB/sec execute 36 sec latency 258.681 ms 2024-03-16T12:06:52.537 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:06:52 smithi154 ceph-mon[36320]: pgmap v505: 97 pgs: 97 active+clean; 353 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 3.3 MiB/s wr, 261 op/s 2024-03-16T12:06:52.580 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:06:52 smithi029 ceph-mon[31975]: pgmap v505: 97 pgs: 97 active+clean; 353 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 3.3 MiB/s wr, 261 op/s 2024-03-16T12:06:52.619 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:06:52 smithi152 ceph-mon[36291]: pgmap v505: 97 pgs: 97 active+clean; 353 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 3.3 MiB/s wr, 261 op/s 2024-03-16T12:06:53.187 INFO:tasks.workunit.client.0.smithi029.stdout: 10 42765 14.07 MB/sec execute 37 sec latency 229.490 ms 2024-03-16T12:06:54.187 INFO:tasks.workunit.client.0.smithi029.stdout: 10 43140 14.36 MB/sec execute 38 sec latency 62.979 ms 2024-03-16T12:06:54.537 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:06:54 smithi154 ceph-mon[36320]: pgmap v506: 97 pgs: 97 active+clean; 343 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 4.5 MiB/s wr, 447 op/s 2024-03-16T12:06:54.580 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:06:54 smithi029 ceph-mon[31975]: pgmap v506: 97 pgs: 97 active+clean; 343 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 4.5 MiB/s wr, 447 op/s 2024-03-16T12:06:54.619 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:06:54 smithi152 ceph-mon[36291]: pgmap v506: 97 pgs: 97 active+clean; 343 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 4.5 MiB/s wr, 447 op/s 2024-03-16T12:06:55.187 INFO:tasks.workunit.client.0.smithi029.stdout: 10 43455 14.65 MB/sec execute 39 sec latency 75.715 ms 2024-03-16T12:06:56.187 INFO:tasks.workunit.client.0.smithi029.stdout: 10 43724 14.80 MB/sec execute 40 sec latency 92.447 ms 2024-03-16T12:06:56.537 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:06:56 smithi154 ceph-mon[36320]: pgmap v507: 97 pgs: 97 active+clean; 343 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 3.6 MiB/s wr, 379 op/s 2024-03-16T12:06:56.580 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:06:56 smithi029 ceph-mon[31975]: pgmap v507: 97 pgs: 97 active+clean; 343 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 3.6 MiB/s wr, 379 op/s 2024-03-16T12:06:56.619 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:06:56 smithi152 ceph-mon[36291]: pgmap v507: 97 pgs: 97 active+clean; 343 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 3.6 MiB/s wr, 379 op/s 2024-03-16T12:06:57.187 INFO:tasks.workunit.client.0.smithi029.stdout: 10 43876 14.57 MB/sec execute 41 sec latency 228.314 ms 2024-03-16T12:06:58.187 INFO:tasks.workunit.client.0.smithi029.stdout: 10 44046 14.45 MB/sec execute 42 sec latency 249.855 ms 2024-03-16T12:06:58.537 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:06:58 smithi154 ceph-mon[36320]: pgmap v508: 97 pgs: 97 active+clean; 351 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 5.4 MiB/s wr, 469 op/s 2024-03-16T12:06:58.580 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:06:58 smithi029 ceph-mon[31975]: pgmap v508: 97 pgs: 97 active+clean; 351 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 5.4 MiB/s wr, 469 op/s 2024-03-16T12:06:58.619 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:06:58 smithi152 ceph-mon[36291]: pgmap v508: 97 pgs: 97 active+clean; 351 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 5.4 MiB/s wr, 469 op/s 2024-03-16T12:06:59.187 INFO:tasks.workunit.client.0.smithi029.stdout: 10 44249 14.27 MB/sec execute 43 sec latency 342.440 ms 2024-03-16T12:07:00.187 INFO:tasks.workunit.client.0.smithi029.stdout: 10 44456 14.13 MB/sec execute 44 sec latency 245.666 ms 2024-03-16T12:07:00.537 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:07:00 smithi154 ceph-mon[36320]: pgmap v509: 97 pgs: 97 active+clean; 351 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 4.1 MiB/s wr, 383 op/s 2024-03-16T12:07:00.580 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:07:00 smithi029 ceph-mon[31975]: pgmap v509: 97 pgs: 97 active+clean; 351 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 4.1 MiB/s wr, 383 op/s 2024-03-16T12:07:00.619 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:07:00 smithi152 ceph-mon[36291]: pgmap v509: 97 pgs: 97 active+clean; 351 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 4.1 MiB/s wr, 383 op/s 2024-03-16T12:07:01.098 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:01.105+0000 7f9d0c5b2640 1 -- 172.21.15.29:0/2713826818 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9d04107850 msgr2=0x7f9d04107c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:07:01.099 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:01.105+0000 7f9d0c5b2640 1 --2- 172.21.15.29:0/2713826818 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9d04107850 0x7f9d04107c30 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f9d00009900 tx=0x7f9d0002bc30 comp rx=0 tx=0).stop 2024-03-16T12:07:01.099 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:01.105+0000 7f9d0c5b2640 1 -- 172.21.15.29:0/2713826818 shutdown_connections 2024-03-16T12:07:01.099 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:01.105+0000 7f9d0c5b2640 1 --2- 172.21.15.29:0/2713826818 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9d0406a980 0x7f9d0410e630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:07:01.099 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:01.105+0000 7f9d0c5b2640 1 --2- 172.21.15.29:0/2713826818 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9d04069fe0 0x7f9d0406a440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:07:01.099 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:01.105+0000 7f9d0c5b2640 1 --2- 172.21.15.29:0/2713826818 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9d04107850 0x7f9d04107c30 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:07:01.099 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:01.105+0000 7f9d0c5b2640 1 -- 172.21.15.29:0/2713826818 >> 172.21.15.29:0/2713826818 conn(0x7f9d040774b0 msgr2=0x7f9d040778c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:07:01.099 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:01.105+0000 7f9d0c5b2640 1 -- 172.21.15.29:0/2713826818 shutdown_connections 2024-03-16T12:07:01.099 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:01.105+0000 7f9d0c5b2640 1 -- 172.21.15.29:0/2713826818 wait complete. 2024-03-16T12:07:01.099 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:01.106+0000 7f9d0c5b2640 1 Processor -- start 2024-03-16T12:07:01.099 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:01.106+0000 7f9d0c5b2640 1 -- start start 2024-03-16T12:07:01.099 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:01.106+0000 7f9d0c5b2640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9d04069fe0 0x7f9d04197f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:07:01.099 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:01.106+0000 7f9d0c5b2640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9d0406a980 0x7f9d041984b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:07:01.099 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:01.106+0000 7f9d0c5b2640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9d04107850 0x7f9d0419c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:07:01.100 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:01.106+0000 7f9d0c5b2640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f9d04111b70 con 0x7f9d04107850 2024-03-16T12:07:01.100 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:01.106+0000 7f9d0c5b2640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f9d041119f0 con 0x7f9d04069fe0 2024-03-16T12:07:01.100 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:01.106+0000 7f9d0c5b2640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f9d04111cf0 con 0x7f9d0406a980 2024-03-16T12:07:01.100 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:01.106+0000 7f9d0ab4f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9d04107850 0x7f9d0419c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:07:01.100 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:01.106+0000 7f9d0ab4f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9d04107850 0x7f9d0419c7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57520/0 (socket says 172.21.15.29:57520) 2024-03-16T12:07:01.100 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:01.106+0000 7f9d0ab4f640 1 -- 172.21.15.29:0/2376677382 learned_addr learned my addr 172.21.15.29:0/2376677382 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T12:07:01.100 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:01.106+0000 7f9d0a34e640 1 --2- 172.21.15.29:0/2376677382 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9d04069fe0 0x7f9d04197f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:07:01.100 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:01.106+0000 7f9d09b4d640 1 --2- 172.21.15.29:0/2376677382 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9d0406a980 0x7f9d041984b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:07:01.100 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:01.106+0000 7f9d0ab4f640 1 -- 172.21.15.29:0/2376677382 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9d0406a980 msgr2=0x7f9d041984b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:07:01.100 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:01.106+0000 7f9d0ab4f640 1 --2- 172.21.15.29:0/2376677382 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9d0406a980 0x7f9d041984b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:07:01.100 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:01.106+0000 7f9d0ab4f640 1 -- 172.21.15.29:0/2376677382 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9d04069fe0 msgr2=0x7f9d04197f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:07:01.100 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:01.106+0000 7f9d0ab4f640 1 --2- 172.21.15.29:0/2376677382 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9d04069fe0 0x7f9d04197f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:07:01.100 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:01.106+0000 7f9d0ab4f640 1 -- 172.21.15.29:0/2376677382 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d0419cf10 con 0x7f9d04107850 2024-03-16T12:07:01.100 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:01.106+0000 7f9d0ab4f640 1 --2- 172.21.15.29:0/2376677382 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9d04107850 0x7f9d0419c7e0 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f9cfc00eb10 tx=0x7f9cfc00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:07:01.100 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:01.107+0000 7f9cfb7fe640 1 -- 172.21.15.29:0/2376677382 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cfc004b20 con 0x7f9d04107850 2024-03-16T12:07:01.101 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:01.107+0000 7f9d0c5b2640 1 -- 172.21.15.29:0/2376677382 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9d0419d200 con 0x7f9d04107850 2024-03-16T12:07:01.101 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:01.107+0000 7f9cfb7fe640 1 -- 172.21.15.29:0/2376677382 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9cfc004cc0 con 0x7f9d04107850 2024-03-16T12:07:01.101 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:01.107+0000 7f9cfb7fe640 1 -- 172.21.15.29:0/2376677382 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cfc010760 con 0x7f9d04107850 2024-03-16T12:07:01.101 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:01.107+0000 7f9d0c5b2640 1 -- 172.21.15.29:0/2376677382 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9d041a4a90 con 0x7f9d04107850 2024-03-16T12:07:01.107 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:01.110+0000 7f9d0f19e740 1 Processor -- start 2024-03-16T12:07:01.107 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:01.110+0000 7f9cfb7fe640 1 -- 172.21.15.29:0/2376677382 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7f9cfc005020 con 0x7f9d04107850 2024-03-16T12:07:01.107 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:01.110+0000 7f9d0f19e740 1 -- start start 2024-03-16T12:07:01.107 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:01.110+0000 7f9d0f19e740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563bb621f130 0x563bb621f510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:07:01.107 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:01.110+0000 7f9d0f19e740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563bb621fa50 0x563bb62282e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:07:01.107 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:01.110+0000 7f9d0f19e740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563bb6228820 0x563bb622ac10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:07:01.107 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:01.110+0000 7f9d0f19e740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x563bb6183a40 con 0x563bb6228820 2024-03-16T12:07:01.107 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:01.110+0000 7f9d0f19e740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x563bb612c920 con 0x563bb621fa50 2024-03-16T12:07:01.107 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:01.110+0000 7f9d0f19e740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x563bb617fae0 con 0x563bb621f130 2024-03-16T12:07:01.107 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:01.110+0000 7f9d09b4d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563bb6228820 0x563bb622ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:07:01.108 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:01.110+0000 7f9d09b4d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563bb6228820 0x563bb622ac10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57534/0 (socket says 172.21.15.29:57534) 2024-03-16T12:07:01.108 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:01.110+0000 7f9d0a34e640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563bb621fa50 0x563bb62282e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:07:01.108 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:01.110+0000 7f9d09b4d640 1 -- 172.21.15.29:0/2861167481 learned_addr learned my addr 172.21.15.29:0/2861167481 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T12:07:01.108 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:01.110+0000 7f9d09b4d640 1 --2- 172.21.15.29:0/2861167481 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563bb621f130 0x563bb621f510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:07:01.108 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:01.110+0000 7f9cfb7fe640 1 --2- 172.21.15.29:0/2376677382 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f9cf00a2a10 0x7f9cf00a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:07:01.108 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:01.111+0000 7f9d0ab4f640 1 --2- 172.21.15.29:0/2376677382 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f9cf00a2a10 0x7f9cf00a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:07:01.108 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:01.111+0000 7f9d09b4d640 1 -- 172.21.15.29:0/2861167481 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563bb621f130 msgr2=0x563bb621f510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:07:01.108 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:01.111+0000 7f9d09b4d640 1 --2- 172.21.15.29:0/2861167481 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563bb621f130 0x563bb621f510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:07:01.108 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:01.111+0000 7f9d09b4d640 1 -- 172.21.15.29:0/2861167481 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563bb621fa50 msgr2=0x563bb62282e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:07:01.108 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:01.111+0000 7f9d09b4d640 1 --2- 172.21.15.29:0/2861167481 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563bb621fa50 0x563bb62282e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:07:01.108 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:01.111+0000 7f9d09b4d640 1 -- 172.21.15.29:0/2861167481 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563bb6144460 con 0x563bb6228820 2024-03-16T12:07:01.108 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:01.111+0000 7f9d0a34e640 1 --2- 172.21.15.29:0/2861167481 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563bb621fa50 0x563bb62282e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:07:01.108 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:01.111+0000 7f9cfb7fe640 1 -- 172.21.15.29:0/2376677382 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(112..112 src has 1..112) ==== 8556+0+0 (secure 0 0 0) 0x7f9cfc0cf400 con 0x7f9d04107850 2024-03-16T12:07:01.108 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:01.111+0000 7f9d09b4d640 1 --2- 172.21.15.29:0/2861167481 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563bb6228820 0x563bb622ac10 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f9cf4011810 tx=0x7f9cf40097d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6841ea35de14e2b server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:07:01.108 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:01.111+0000 7f9cf8bfe640 1 -- 172.21.15.29:0/2861167481 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cf4004030 con 0x563bb6228820 2024-03-16T12:07:01.109 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:01.111+0000 7f9d0ab4f640 1 --2- 172.21.15.29:0/2376677382 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f9cf00a2a10 0x7f9cf00a4ed0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f9cfc09b740 tx=0x7f9cfc00efd0 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:07:01.109 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:01.111+0000 7f9cf8bfe640 1 -- 172.21.15.29:0/2861167481 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9cf4011e50 con 0x563bb6228820 2024-03-16T12:07:01.109 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:01.111+0000 7f9cf8bfe640 1 -- 172.21.15.29:0/2861167481 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cf4003850 con 0x563bb6228820 2024-03-16T12:07:01.109 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:01.112+0000 7f9d0f19e740 1 -- 172.21.15.29:0/2861167481 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563bb6228820 msgr2=0x563bb622ac10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:07:01.109 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:01.112+0000 7f9d0f19e740 1 --2- 172.21.15.29:0/2861167481 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563bb6228820 0x563bb622ac10 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f9cf4011810 tx=0x7f9cf40097d0 comp rx=0 tx=0).stop 2024-03-16T12:07:01.109 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:01.112+0000 7f9d0f19e740 1 -- 172.21.15.29:0/2861167481 shutdown_connections 2024-03-16T12:07:01.109 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:01.112+0000 7f9d0f19e740 1 --2- 172.21.15.29:0/2861167481 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563bb6228820 0x563bb622ac10 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:07:01.109 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:01.112+0000 7f9d0f19e740 1 --2- 172.21.15.29:0/2861167481 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563bb621fa50 0x563bb62282e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:07:01.109 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:01.112+0000 7f9d0f19e740 1 --2- 172.21.15.29:0/2861167481 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563bb621f130 0x563bb621f510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:07:01.109 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:01.112+0000 7f9d0f19e740 1 -- 172.21.15.29:0/2861167481 >> 172.21.15.29:0/2861167481 conn(0x563bb6219a50 msgr2=0x563bb6219e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:07:01.109 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:01.112+0000 7f9d0f19e740 1 -- 172.21.15.29:0/2861167481 shutdown_connections 2024-03-16T12:07:01.109 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:01.112+0000 7f9d0f19e740 1 -- 172.21.15.29:0/2861167481 wait complete. 2024-03-16T12:07:01.109 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:01.112+0000 7f9d0f19e740 1 -- start start 2024-03-16T12:07:01.109 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:01.112+0000 7f9d0f19e740 1 Processor -- start 2024-03-16T12:07:01.109 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:01.112+0000 7f9d0f19e740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563bb621f130 0x563bb6225840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:07:01.110 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:01.113+0000 7f9d0f19e740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563bb621fa50 0x563bb6225d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:07:01.110 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:01.113+0000 7f9d0f19e740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563bb6228820 0x563bb62262c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:07:01.110 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:01.113+0000 7f9d0f19e740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x563bb622b450 con 0x563bb621f130 2024-03-16T12:07:01.110 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:01.113+0000 7f9d0f19e740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x563bb613fd70 con 0x563bb6228820 2024-03-16T12:07:01.110 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:01.113+0000 7f9d0f19e740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x563bb622b5d0 con 0x563bb621fa50 2024-03-16T12:07:01.110 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:01.113+0000 7f9d0f19e740 10 client.0 _ll_register_callbacks cb 0x563bb5e97a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T12:07:01.110 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:01.113+0000 7f9d09b4d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563bb621f130 0x563bb6225840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:07:01.110 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:01.113+0000 7f9d09b4d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563bb621f130 0x563bb6225840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57540/0 (socket says 172.21.15.29:57540) 2024-03-16T12:07:01.110 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:01.113+0000 7f9d09b4d640 1 -- 172.21.15.29:0/2928667921 learned_addr learned my addr 172.21.15.29:0/2928667921 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T12:07:01.110 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:01.113+0000 7f9d0a34e640 1 --2- 172.21.15.29:0/2928667921 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563bb621fa50 0x563bb6225d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:07:01.110 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:01.113+0000 7f9d09b4d640 1 --2- 172.21.15.29:0/2928667921 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563bb6228820 0x563bb62262c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:07:01.110 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:01.113+0000 7f9d09b4d640 1 -- 172.21.15.29:0/2928667921 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563bb621fa50 msgr2=0x563bb6225d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:07:01.110 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:01.113+0000 7f9d09b4d640 1 --2- 172.21.15.29:0/2928667921 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563bb621fa50 0x563bb6225d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:07:01.110 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:01.113+0000 7f9d09b4d640 1 -- 172.21.15.29:0/2928667921 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563bb6228820 msgr2=0x563bb62262c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:07:01.111 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:01.113+0000 7f9d09b4d640 1 --2- 172.21.15.29:0/2928667921 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563bb6228820 0x563bb62262c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:07:01.111 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:01.113+0000 7f9d09b4d640 1 -- 172.21.15.29:0/2928667921 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563bb5e98210 con 0x563bb621f130 2024-03-16T12:07:01.111 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:01.113+0000 7f9d09b4d640 1 --2- 172.21.15.29:0/2928667921 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563bb621f130 0x563bb6225840 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f9cf40043e0 tx=0x7f9cf4004440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b3642b401d3be5f server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:07:01.111 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:01.113+0000 7f9cedffb640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-16T12:07:01.111 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:01.113+0000 7f9cedffb640 1 -- 172.21.15.29:0/2928667921 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cf402d060 con 0x563bb621f130 2024-03-16T12:07:01.111 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:01.113+0000 7f9cedffb640 1 -- 172.21.15.29:0/2928667921 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(112..112 src has 1..112) ==== 8556+0+0 (secure 0 0 0) 0x7f9cf400fd50 con 0x563bb621f130 2024-03-16T12:07:01.111 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:01.113+0000 7f9d0f19e740 1 -- 172.21.15.29:0/2928667921 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9cf4011e50 con 0x563bb621f130 2024-03-16T12:07:01.111 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:01.114+0000 7f9cedffb640 1 -- 172.21.15.29:0/2928667921 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f9cf4005020 con 0x563bb621f130 2024-03-16T12:07:01.111 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:01.114+0000 7f9d0f19e740 10 client.16725 fetch_fsmap learned FSMap version 32 2024-03-16T12:07:01.111 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:01.114+0000 7f9d0f19e740 1 -- 172.21.15.29:0/2928667921 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x563bb5f64620 con 0x563bb621f130 2024-03-16T12:07:01.111 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:01.114+0000 7f9cedffb640 1 -- 172.21.15.29:0/2928667921 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9cf4011e50 con 0x563bb621f130 2024-03-16T12:07:01.111 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:01.114+0000 7f9cedffb640 1 -- 172.21.15.29:0/2928667921 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 32) ==== 8338+0+0 (secure 0 0 0) 0x7f9cf4004be0 con 0x563bb621f130 2024-03-16T12:07:01.111 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:01.114+0000 7f9d0f19e740 10 client.16725 fetch_fsmap finished waiting for FSMap version 32 2024-03-16T12:07:01.111 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:01.114+0000 7f9d0f19e740 10 client.16725 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T12:07:01.111 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:01.114+0000 7f9d0f19e740 20 client.16725 populate_metadata read hostname 'smithi029' 2024-03-16T12:07:01.111 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:01.114+0000 7f9d0f19e740 1 --2- 172.21.15.29:0/2928667921 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x563bb62388c0 0x563bb623acb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:07:01.112 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:01.114+0000 7f9d0f19e740 4 client.16725 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T12:07:01.112 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:01.114+0000 7f9d0f19e740 1 -- 172.21.15.29:0/2928667921 --> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563bb6190f80 con 0x563bb62388c0 2024-03-16T12:07:01.112 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:01.114+0000 7f9d0a34e640 1 --2- 172.21.15.29:0/2928667921 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x563bb62388c0 0x563bb623acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:07:01.112 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:01.115+0000 7f9d0a34e640 1 --2- 172.21.15.29:0/2928667921 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x563bb62388c0 0x563bb623acb0 crc :-1 s=READY pgs=182 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=899f1185adf4b9cc server_cookie=b15caa1fafda0eb9 in_seq=0 out_seq=0 2024-03-16T12:07:01.112 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:01.115+0000 7f9cedffb640 10 client.16725 ms_handle_connect on v2:172.21.15.152:6838/1548731833 2024-03-16T12:07:01.112 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:01.116+0000 7f9cedffb640 1 -- 172.21.15.29:0/2928667921 <== mds.0 v2:172.21.15.152:6838/1548731833 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x563bb6190f80 con 0x563bb62388c0 2024-03-16T12:07:01.112 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:01.116+0000 7f9cedffb640 10 client.16725 handle_command_reply: tid=0 2024-03-16T12:07:01.112 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:01.116+0000 7f9d0f19e740 1 client.16725 shutdown 2024-03-16T12:07:01.113 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:01.116+0000 7f9d0f19e740 1 -- 172.21.15.29:0/2928667921 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563bb621f130 msgr2=0x563bb6225840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:07:01.113 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:01.116+0000 7f9d0f19e740 1 --2- 172.21.15.29:0/2928667921 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563bb621f130 0x563bb6225840 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f9cf40043e0 tx=0x7f9cf4004440 comp rx=0 tx=0).stop 2024-03-16T12:07:01.113 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:01.116+0000 7f9d0f19e740 1 -- 172.21.15.29:0/2928667921 shutdown_connections 2024-03-16T12:07:01.113 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:01.116+0000 7f9d0f19e740 1 --2- 172.21.15.29:0/2928667921 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x563bb62388c0 0x563bb623acb0 crc :-1 s=READY pgs=182 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:07:01.113 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:01.116+0000 7f9d0f19e740 1 --2- 172.21.15.29:0/2928667921 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563bb6228820 0x563bb62262c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:07:01.113 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:01.116+0000 7f9d0f19e740 1 --2- 172.21.15.29:0/2928667921 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563bb621fa50 0x563bb6225d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:07:01.113 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:01.116+0000 7f9cedffb640 0 client.16725 ms_handle_reset on v2:172.21.15.152:6838/1548731833 2024-03-16T12:07:01.113 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:01.116+0000 7f9d0f19e740 1 --2- 172.21.15.29:0/2928667921 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563bb621f130 0x563bb6225840 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:07:01.114 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:01.116+0000 7f9d0f19e740 1 -- 172.21.15.29:0/2928667921 >> 172.21.15.29:0/2928667921 conn(0x563bb6219a50 msgr2=0x563bb621b7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:07:01.114 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:01.116+0000 7f9d0f19e740 1 -- 172.21.15.29:0/2928667921 shutdown_connections 2024-03-16T12:07:01.114 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:01.116+0000 7f9d0f19e740 1 -- 172.21.15.29:0/2928667921 wait complete. 2024-03-16T12:07:01.114 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:01.116+0000 7f9d0f19e740 20 client.16725 trim_cache size 0 max 16384 2024-03-16T12:07:01.127 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:01.131+0000 7f9d0f19e740 1 Processor -- start 2024-03-16T12:07:01.127 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:01.131+0000 7f9d0f19e740 1 -- start start 2024-03-16T12:07:01.127 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:01.131+0000 7f9d0f19e740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563bb621f130 0x563bb621b900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:07:01.127 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:01.131+0000 7f9d0f19e740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563bb621fa50 0x563bb6225120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:07:01.128 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:01.131+0000 7f9d0f19e740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563bb6228820 0x563bb62341f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:07:01.128 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:01.131+0000 7f9d0f19e740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f9d041afd10 con 0x563bb621fa50 2024-03-16T12:07:01.128 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:01.131+0000 7f9d0f19e740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f9d041ad5e0 con 0x563bb621f130 2024-03-16T12:07:01.128 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:01.131+0000 7f9d0f19e740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f9d041ab450 con 0x563bb6228820 2024-03-16T12:07:01.128 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:01.131+0000 7f9d0a34e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563bb621fa50 0x563bb6225120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:07:01.128 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:01.132+0000 7f9d0a34e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563bb621fa50 0x563bb6225120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57556/0 (socket says 172.21.15.29:57556) 2024-03-16T12:07:01.128 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:01.132+0000 7f9d0a34e640 1 -- 172.21.15.29:0/1011440345 learned_addr learned my addr 172.21.15.29:0/1011440345 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T12:07:01.128 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:01.132+0000 7f9d09b4d640 1 --2- 172.21.15.29:0/1011440345 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563bb621f130 0x563bb621b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:07:01.128 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:01.132+0000 7f9d09b4d640 1 --2- 172.21.15.29:0/1011440345 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563bb6228820 0x563bb62341f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:07:01.128 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:01.132+0000 7f9d0a34e640 1 -- 172.21.15.29:0/1011440345 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563bb6228820 msgr2=0x563bb62341f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:07:01.128 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:01.132+0000 7f9d0a34e640 1 --2- 172.21.15.29:0/1011440345 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563bb6228820 0x563bb62341f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:07:01.128 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:01.132+0000 7f9d0a34e640 1 -- 172.21.15.29:0/1011440345 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563bb621f130 msgr2=0x563bb621b900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:07:01.128 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:01.132+0000 7f9d0a34e640 1 --2- 172.21.15.29:0/1011440345 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563bb621f130 0x563bb621b900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:07:01.128 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:01.132+0000 7f9d0a34e640 1 -- 172.21.15.29:0/1011440345 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563bb622ac10 con 0x563bb621fa50 2024-03-16T12:07:01.129 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:01.132+0000 7f9d09b4d640 1 --2- 172.21.15.29:0/1011440345 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563bb6228820 0x563bb62341f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T12:07:01.129 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:01.132+0000 7f9d0a34e640 1 --2- 172.21.15.29:0/1011440345 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563bb621fa50 0x563bb6225120 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f9d0002c170 tx=0x7f9d00005dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc8c878769fc5243 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:07:01.129 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:01.132+0000 7f9ceffff640 1 -- 172.21.15.29:0/1011440345 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d0004c070 con 0x563bb621fa50 2024-03-16T12:07:01.129 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:01.132+0000 7f9ceffff640 1 -- 172.21.15.29:0/1011440345 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9d00048d80 con 0x563bb621fa50 2024-03-16T12:07:01.129 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:01.132+0000 7f9ceffff640 1 -- 172.21.15.29:0/1011440345 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d000043c0 con 0x563bb621fa50 2024-03-16T12:07:01.129 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:01.133+0000 7f9d0f19e740 1 -- 172.21.15.29:0/1011440345 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563bb621fa50 msgr2=0x563bb6225120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:07:01.129 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:01.133+0000 7f9d0f19e740 1 --2- 172.21.15.29:0/1011440345 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563bb621fa50 0x563bb6225120 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f9d0002c170 tx=0x7f9d00005dd0 comp rx=0 tx=0).stop 2024-03-16T12:07:01.129 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:01.133+0000 7f9d0f19e740 1 -- 172.21.15.29:0/1011440345 shutdown_connections 2024-03-16T12:07:01.129 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:01.133+0000 7f9d0f19e740 1 --2- 172.21.15.29:0/1011440345 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563bb6228820 0x563bb62341f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:07:01.129 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:01.133+0000 7f9d0f19e740 1 --2- 172.21.15.29:0/1011440345 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563bb621fa50 0x563bb6225120 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:07:01.129 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:01.133+0000 7f9d0f19e740 1 --2- 172.21.15.29:0/1011440345 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563bb621f130 0x563bb621b900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:07:01.129 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:01.133+0000 7f9d0f19e740 1 -- 172.21.15.29:0/1011440345 >> 172.21.15.29:0/1011440345 conn(0x563bb6219a50 msgr2=0x563bb6218bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:07:01.129 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:01.133+0000 7f9d0f19e740 1 -- 172.21.15.29:0/1011440345 shutdown_connections 2024-03-16T12:07:01.129 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:01.133+0000 7f9d0f19e740 1 -- 172.21.15.29:0/1011440345 wait complete. 2024-03-16T12:07:01.129 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:01.133+0000 7f9d0f19e740 1 -- start start 2024-03-16T12:07:01.129 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:01.133+0000 7f9d0f19e740 1 Processor -- start 2024-03-16T12:07:01.130 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:01.133+0000 7f9d0f19e740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563bb621f130 0x563bb6230b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:07:01.130 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:01.133+0000 7f9d0f19e740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563bb621fa50 0x563bb622bec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:07:01.130 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:01.133+0000 7f9d0f19e740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563bb6228820 0x563bb622c440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:07:01.130 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:01.133+0000 7f9d0f19e740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f9d041abaa0 con 0x563bb6228820 2024-03-16T12:07:01.130 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:01.133+0000 7f9d0f19e740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f9d041acf10 con 0x563bb621f130 2024-03-16T12:07:01.130 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:01.133+0000 7f9d0f19e740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f9d041ac170 con 0x563bb621fa50 2024-03-16T12:07:01.130 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:01.134+0000 7f9d0f19e740 10 client.0 _ll_register_callbacks cb 0x7f9cf4005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T12:07:01.130 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:01.134+0000 7f9d09b4d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563bb6228820 0x563bb622c440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:07:01.130 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:01.134+0000 7f9d09b4d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563bb6228820 0x563bb622c440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57562/0 (socket says 172.21.15.29:57562) 2024-03-16T12:07:01.130 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:01.134+0000 7f9d0a34e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563bb621fa50 0x563bb622bec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:07:01.130 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:01.134+0000 7f9d09b4d640 1 -- 172.21.15.29:0/4170222939 learned_addr learned my addr 172.21.15.29:0/4170222939 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T12:07:01.130 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:01.134+0000 7f9d09b4d640 1 --2- 172.21.15.29:0/4170222939 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563bb621f130 0x563bb6230b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:07:01.130 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:01.134+0000 7f9d09b4d640 1 -- 172.21.15.29:0/4170222939 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563bb621fa50 msgr2=0x563bb622bec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:07:01.130 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:01.134+0000 7f9d09b4d640 1 --2- 172.21.15.29:0/4170222939 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563bb621fa50 0x563bb622bec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:07:01.131 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:01.134+0000 7f9d09b4d640 1 -- 172.21.15.29:0/4170222939 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563bb621f130 msgr2=0x563bb6230b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:07:01.131 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:01.134+0000 7f9d09b4d640 1 --2- 172.21.15.29:0/4170222939 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563bb621f130 0x563bb6230b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:07:01.131 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:01.134+0000 7f9d09b4d640 1 -- 172.21.15.29:0/4170222939 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563bb622cc70 con 0x563bb6228820 2024-03-16T12:07:01.131 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:01.134+0000 7f9d0a34e640 1 --2- 172.21.15.29:0/4170222939 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563bb621fa50 0x563bb622bec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:07:01.131 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:01.134+0000 7f9d09b4d640 1 --2- 172.21.15.29:0/4170222939 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563bb6228820 0x563bb622c440 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f9cf401f510 tx=0x7f9cf400faa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9933f77ce0ace9d3 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:07:01.131 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:01.134+0000 7f9cedffb640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-16T12:07:01.131 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:01.134+0000 7f9cedffb640 1 -- 172.21.15.29:0/4170222939 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cf402d070 con 0x563bb6228820 2024-03-16T12:07:01.131 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:01.134+0000 7f9cedffb640 1 -- 172.21.15.29:0/4170222939 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(112..112 src has 1..112) ==== 8556+0+0 (secure 0 0 0) 0x7f9cf401f8f0 con 0x563bb6228820 2024-03-16T12:07:01.131 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:01.134+0000 7f9d0f19e740 1 -- 172.21.15.29:0/4170222939 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9d00048d80 con 0x563bb6228820 2024-03-16T12:07:01.131 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:01.134+0000 7f9cedffb640 1 -- 172.21.15.29:0/4170222939 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f9cf400fe80 con 0x563bb6228820 2024-03-16T12:07:01.131 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:01.134+0000 7f9d0f19e740 10 client.16731 fetch_fsmap learned FSMap version 32 2024-03-16T12:07:01.131 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:01.134+0000 7f9d0f19e740 1 -- 172.21.15.29:0/4170222939 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x563bb629ea20 con 0x563bb6228820 2024-03-16T12:07:01.131 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:01.135+0000 7f9cedffb640 1 -- 172.21.15.29:0/4170222939 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9d00048d80 con 0x563bb6228820 2024-03-16T12:07:01.131 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:01.135+0000 7f9cedffb640 1 -- 172.21.15.29:0/4170222939 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 32) ==== 8338+0+0 (secure 0 0 0) 0x7f9cf40045d0 con 0x563bb6228820 2024-03-16T12:07:01.131 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:01.135+0000 7f9d0f19e740 10 client.16731 fetch_fsmap finished waiting for FSMap version 32 2024-03-16T12:07:01.131 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:01.135+0000 7f9d0f19e740 10 client.16731 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T12:07:01.132 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:01.135+0000 7f9d0f19e740 20 client.16731 populate_metadata read hostname 'smithi029' 2024-03-16T12:07:01.132 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:01.135+0000 7f9d0f19e740 1 --2- 172.21.15.29:0/4170222939 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x563bb629f030 0x563bb62a1450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:07:01.132 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:01.135+0000 7f9d0f19e740 4 client.16731 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-16T12:07:01.132 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:01.135+0000 7f9d0f19e740 1 -- 172.21.15.29:0/4170222939 --> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] -- command(tid 0: {"prefix": "flush journal"}) -- 0x563bb618b2d0 con 0x563bb629f030 2024-03-16T12:07:01.132 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:01.135+0000 7f9d0a34e640 1 --2- 172.21.15.29:0/4170222939 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x563bb629f030 0x563bb62a1450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:07:01.132 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:01.135+0000 7f9cedffb640 10 client.16731 ms_handle_connect on v2:172.21.15.152:6838/1548731833 2024-03-16T12:07:01.132 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:01.135+0000 7f9d0a34e640 1 --2- 172.21.15.29:0/4170222939 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x563bb629f030 0x563bb62a1450 crc :-1 s=READY pgs=183 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c67d33577179f9e2 server_cookie=6bf764a74e064443 in_seq=0 out_seq=0 2024-03-16T12:07:01.188 INFO:tasks.workunit.client.0.smithi029.stdout: 10 44634 14.00 MB/sec execute 45 sec latency 514.532 ms 2024-03-16T12:07:01.568 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:01.576+0000 7f9cedffb640 1 -- 172.21.15.29:0/4170222939 <== mds.0 v2:172.21.15.152:6838/1548731833 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x563bb618b2d0 con 0x563bb629f030 2024-03-16T12:07:01.568 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:01.576+0000 7f9cedffb640 10 client.16731 handle_command_reply: tid=0 2024-03-16T12:07:01.568 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:01.576+0000 7f9d0f19e740 1 client.16731 shutdown 2024-03-16T12:07:01.568 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:01.576+0000 7f9d0f19e740 1 -- 172.21.15.29:0/4170222939 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563bb6228820 msgr2=0x563bb622c440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:07:01.569 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:01.576+0000 7f9d0f19e740 1 --2- 172.21.15.29:0/4170222939 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563bb6228820 0x563bb622c440 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f9cf401f510 tx=0x7f9cf400faa0 comp rx=0 tx=0).stop 2024-03-16T12:07:01.569 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:01.576+0000 7f9d0f19e740 1 -- 172.21.15.29:0/4170222939 shutdown_connections 2024-03-16T12:07:01.569 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:01.576+0000 7f9d0f19e740 1 --2- 172.21.15.29:0/4170222939 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563bb6228820 0x563bb622c440 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:07:01.569 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:01.576+0000 7f9d0f19e740 1 --2- 172.21.15.29:0/4170222939 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563bb621fa50 0x563bb622bec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:07:01.569 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:01.576+0000 7f9d0f19e740 1 --2- 172.21.15.29:0/4170222939 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x563bb629f030 0x563bb62a1450 crc :-1 s=READY pgs=183 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:07:01.569 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:01.576+0000 7f9d0f19e740 1 --2- 172.21.15.29:0/4170222939 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563bb621f130 0x563bb6230b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:07:01.569 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:01.576+0000 7f9d0f19e740 1 -- 172.21.15.29:0/4170222939 >> 172.21.15.29:0/4170222939 conn(0x563bb6219a50 msgr2=0x563bb6218320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:07:01.569 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:01.576+0000 7f9cedffb640 0 client.16731 ms_handle_reset on v2:172.21.15.152:6838/1548731833 2024-03-16T12:07:01.569 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:01.576+0000 7f9d0f19e740 1 -- 172.21.15.29:0/4170222939 shutdown_connections 2024-03-16T12:07:01.569 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:01.576+0000 7f9d0f19e740 1 -- 172.21.15.29:0/4170222939 wait complete. 2024-03-16T12:07:01.569 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:01.577+0000 7f9d0f19e740 20 client.16731 trim_cache size 0 max 16384 2024-03-16T12:07:01.570 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:{ 2024-03-16T12:07:01.570 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "message": "", 2024-03-16T12:07:01.570 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "return_code": 0 2024-03-16T12:07:01.570 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:} 2024-03-16T12:07:01.571 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:01.577+0000 7f9d0c5b2640 1 -- 172.21.15.29:0/2376677382 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f9cf00a2a10 msgr2=0x7f9cf00a4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:07:01.571 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:01.577+0000 7f9d0c5b2640 1 --2- 172.21.15.29:0/2376677382 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f9cf00a2a10 0x7f9cf00a4ed0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f9cfc09b740 tx=0x7f9cfc00efd0 comp rx=0 tx=0).stop 2024-03-16T12:07:01.571 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:01.577+0000 7f9d0c5b2640 1 -- 172.21.15.29:0/2376677382 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9d04107850 msgr2=0x7f9d0419c7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:07:01.571 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:01.577+0000 7f9d0c5b2640 1 --2- 172.21.15.29:0/2376677382 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9d04107850 0x7f9d0419c7e0 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f9cfc00eb10 tx=0x7f9cfc00c2d0 comp rx=0 tx=0).stop 2024-03-16T12:07:01.571 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:01.577+0000 7f9d0c5b2640 1 -- 172.21.15.29:0/2376677382 shutdown_connections 2024-03-16T12:07:01.571 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:01.577+0000 7f9d0c5b2640 1 --2- 172.21.15.29:0/2376677382 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f9cf00a2a10 0x7f9cf00a4ed0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:07:01.571 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:01.578+0000 7f9d0c5b2640 1 --2- 172.21.15.29:0/2376677382 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9d04107850 0x7f9d0419c7e0 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:07:01.571 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:01.578+0000 7f9d0c5b2640 1 --2- 172.21.15.29:0/2376677382 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9d0406a980 0x7f9d041984b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:07:01.571 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:01.578+0000 7f9d0c5b2640 1 --2- 172.21.15.29:0/2376677382 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9d04069fe0 0x7f9d04197f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:07:01.571 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:01.578+0000 7f9d0c5b2640 1 -- 172.21.15.29:0/2376677382 >> 172.21.15.29:0/2376677382 conn(0x7f9d040774b0 msgr2=0x7f9d040ff460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:07:01.572 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:01.580+0000 7f9d0c5b2640 1 -- 172.21.15.29:0/2376677382 shutdown_connections 2024-03-16T12:07:01.572 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:01.580+0000 7f9d0c5b2640 1 -- 172.21.15.29:0/2376677382 wait complete. 2024-03-16T12:07:02.187 INFO:tasks.workunit.client.0.smithi029.stdout: 10 44634 13.70 MB/sec execute 46 sec latency 1514.636 ms 2024-03-16T12:07:02.537 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:07:02 smithi154 ceph-mon[36320]: pgmap v510: 97 pgs: 97 active+clean; 356 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 4.8 MiB/s wr, 437 op/s 2024-03-16T12:07:02.580 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:07:02 smithi029 ceph-mon[31975]: pgmap v510: 97 pgs: 97 active+clean; 356 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 4.8 MiB/s wr, 437 op/s 2024-03-16T12:07:02.619 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:07:02 smithi152 ceph-mon[36291]: pgmap v510: 97 pgs: 97 active+clean; 356 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 4.8 MiB/s wr, 437 op/s 2024-03-16T12:07:03.187 INFO:tasks.workunit.client.0.smithi029.stdout: 10 44634 13.41 MB/sec execute 47 sec latency 2514.742 ms 2024-03-16T12:07:04.187 INFO:tasks.workunit.client.0.smithi029.stdout: 10 44634 13.13 MB/sec execute 48 sec latency 3514.834 ms 2024-03-16T12:07:04.537 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:07:04 smithi154 ceph-mon[36320]: pgmap v511: 97 pgs: 97 active+clean; 355 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 5.0 MiB/s wr, 428 op/s 2024-03-16T12:07:04.580 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:07:04 smithi029 ceph-mon[31975]: pgmap v511: 97 pgs: 97 active+clean; 355 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 5.0 MiB/s wr, 428 op/s 2024-03-16T12:07:04.619 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:07:04 smithi152 ceph-mon[36291]: pgmap v511: 97 pgs: 97 active+clean; 355 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 5.0 MiB/s wr, 428 op/s 2024-03-16T12:07:05.187 INFO:tasks.workunit.client.0.smithi029.stdout: 10 44739 12.95 MB/sec execute 49 sec latency 4026.451 ms 2024-03-16T12:07:06.187 INFO:tasks.workunit.client.0.smithi029.stdout: 10 45033 12.81 MB/sec execute 50 sec latency 71.510 ms 2024-03-16T12:07:06.537 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:07:06 smithi154 ceph-mon[36320]: pgmap v512: 97 pgs: 97 active+clean; 355 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 3.2 MiB/s wr, 208 op/s 2024-03-16T12:07:06.579 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:07:06 smithi029 ceph-mon[31975]: pgmap v512: 97 pgs: 97 active+clean; 355 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 3.2 MiB/s wr, 208 op/s 2024-03-16T12:07:06.619 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:07:06 smithi152 ceph-mon[36291]: pgmap v512: 97 pgs: 97 active+clean; 355 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 3.2 MiB/s wr, 208 op/s 2024-03-16T12:07:07.187 INFO:tasks.workunit.client.0.smithi029.stdout: 10 45252 12.82 MB/sec execute 51 sec latency 203.440 ms 2024-03-16T12:07:08.187 INFO:tasks.workunit.client.0.smithi029.stdout: 10 45540 12.81 MB/sec execute 52 sec latency 167.899 ms 2024-03-16T12:07:08.537 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:07:08 smithi154 ceph-mon[36320]: pgmap v513: 97 pgs: 97 active+clean; 342 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 5.7 MiB/s wr, 493 op/s 2024-03-16T12:07:08.579 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:07:08 smithi029 ceph-mon[31975]: pgmap v513: 97 pgs: 97 active+clean; 342 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 5.7 MiB/s wr, 493 op/s 2024-03-16T12:07:08.619 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:07:08 smithi152 ceph-mon[36291]: pgmap v513: 97 pgs: 97 active+clean; 342 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 5.7 MiB/s wr, 493 op/s 2024-03-16T12:07:09.187 INFO:tasks.workunit.client.0.smithi029.stdout: 10 46022 13.19 MB/sec execute 53 sec latency 113.686 ms 2024-03-16T12:07:10.187 INFO:tasks.workunit.client.0.smithi029.stdout: 10 46209 13.07 MB/sec execute 54 sec latency 352.898 ms 2024-03-16T12:07:10.537 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:07:10 smithi154 ceph-mon[36320]: pgmap v514: 97 pgs: 97 active+clean; 342 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.0 MiB/s wr, 402 op/s 2024-03-16T12:07:10.537 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:07:10 smithi154 ceph-mon[36320]: osdmap e113: 12 total, 12 up, 12 in 2024-03-16T12:07:10.580 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:07:10 smithi029 ceph-mon[31975]: pgmap v514: 97 pgs: 97 active+clean; 342 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.0 MiB/s wr, 402 op/s 2024-03-16T12:07:10.580 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:07:10 smithi029 ceph-mon[31975]: osdmap e113: 12 total, 12 up, 12 in 2024-03-16T12:07:10.619 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:07:10 smithi152 ceph-mon[36291]: pgmap v514: 97 pgs: 97 active+clean; 342 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.0 MiB/s wr, 402 op/s 2024-03-16T12:07:10.619 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:07:10 smithi152 ceph-mon[36291]: osdmap e113: 12 total, 12 up, 12 in 2024-03-16T12:07:11.187 INFO:tasks.workunit.client.0.smithi029.stdout: 10 46397 13.07 MB/sec execute 55 sec latency 164.499 ms 2024-03-16T12:07:12.187 INFO:tasks.workunit.client.0.smithi029.stdout: 10 46519 13.06 MB/sec execute 56 sec latency 538.581 ms 2024-03-16T12:07:12.537 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:07:12 smithi154 ceph-mon[36320]: pgmap v516: 97 pgs: 4 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 89 active+clean; 345 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 4.5 MiB/s wr, 463 op/s 2024-03-16T12:07:12.579 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:07:12 smithi029 ceph-mon[31975]: pgmap v516: 97 pgs: 4 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 89 active+clean; 345 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 4.5 MiB/s wr, 463 op/s 2024-03-16T12:07:12.619 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:07:12 smithi152 ceph-mon[36291]: pgmap v516: 97 pgs: 4 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 89 active+clean; 345 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 4.5 MiB/s wr, 463 op/s 2024-03-16T12:07:13.187 INFO:tasks.workunit.client.0.smithi029.stdout: 10 46591 12.87 MB/sec execute 57 sec latency 1114.611 ms 2024-03-16T12:07:14.188 INFO:tasks.workunit.client.0.smithi029.stdout: 10 46780 12.87 MB/sec execute 58 sec latency 445.618 ms 2024-03-16T12:07:14.537 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:07:14 smithi154 ceph-mon[36320]: pgmap v517: 97 pgs: 38 active+clean+snaptrim_wait, 21 active+clean+snaptrim, 38 active+clean; 351 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 5.1 MiB/s wr, 570 op/s 2024-03-16T12:07:14.580 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:07:14 smithi029 ceph-mon[31975]: pgmap v517: 97 pgs: 38 active+clean+snaptrim_wait, 21 active+clean+snaptrim, 38 active+clean; 351 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 5.1 MiB/s wr, 570 op/s 2024-03-16T12:07:14.619 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:07:14 smithi152 ceph-mon[36291]: pgmap v517: 97 pgs: 38 active+clean+snaptrim_wait, 21 active+clean+snaptrim, 38 active+clean; 351 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 5.1 MiB/s wr, 570 op/s 2024-03-16T12:07:14.662 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:14.670+0000 7f750a663640 1 -- 172.21.15.29:0/3155209014 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7504100930 msgr2=0x7f7504100db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:07:14.662 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:14.670+0000 7f750a663640 1 --2- 172.21.15.29:0/3155209014 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7504100930 0x7f7504100db0 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f74f8009900 tx=0x7f74f802bc30 comp rx=0 tx=0).stop 2024-03-16T12:07:14.662 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:14.670+0000 7f750a663640 1 -- 172.21.15.29:0/3155209014 shutdown_connections 2024-03-16T12:07:14.662 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:14.670+0000 7f750a663640 1 --2- 172.21.15.29:0/3155209014 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f75041012f0 0x7f750410d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:07:14.662 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:14.670+0000 7f750a663640 1 --2- 172.21.15.29:0/3155209014 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7504100930 0x7f7504100db0 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:07:14.662 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:14.670+0000 7f750a663640 1 --2- 172.21.15.29:0/3155209014 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7504105680 0x7f7504105a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:07:14.662 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:14.670+0000 7f750a663640 1 -- 172.21.15.29:0/3155209014 >> 172.21.15.29:0/3155209014 conn(0x7f7504077530 msgr2=0x7f7504077940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:07:14.662 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:14.670+0000 7f750a663640 1 -- 172.21.15.29:0/3155209014 shutdown_connections 2024-03-16T12:07:14.662 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:14.670+0000 7f750a663640 1 -- 172.21.15.29:0/3155209014 wait complete. 2024-03-16T12:07:14.663 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:14.671+0000 7f750a663640 1 Processor -- start 2024-03-16T12:07:14.663 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:14.671+0000 7f750a663640 1 -- start start 2024-03-16T12:07:14.664 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:14.671+0000 7f750a663640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7504100930 0x7f7504196fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:07:14.664 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:14.671+0000 7f750a663640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f75041012f0 0x7f75041974e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:07:14.664 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:14.671+0000 7f750a663640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7504105680 0x7f750419b860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:07:14.664 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:14.671+0000 7f750a663640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f7504110ba0 con 0x7f7504100930 2024-03-16T12:07:14.664 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:14.671+0000 7f750a663640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f7504110a20 con 0x7f7504105680 2024-03-16T12:07:14.664 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:14.671+0000 7f750a663640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f7504110d20 con 0x7f75041012f0 2024-03-16T12:07:14.664 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:14.671+0000 7f7503fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7504100930 0x7f7504196fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:07:14.665 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:14.671+0000 7f7503fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7504100930 0x7f7504196fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:38768/0 (socket says 172.21.15.29:38768) 2024-03-16T12:07:14.665 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:14.671+0000 7f7503fff640 1 -- 172.21.15.29:0/177191404 learned_addr learned my addr 172.21.15.29:0/177191404 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T12:07:14.665 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:14.671+0000 7f7508c00640 1 --2- 172.21.15.29:0/177191404 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7504105680 0x7f750419b860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:07:14.665 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:14.671+0000 7f75037fe640 1 --2- 172.21.15.29:0/177191404 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f75041012f0 0x7f75041974e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:07:14.665 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:14.672+0000 7f7503fff640 1 -- 172.21.15.29:0/177191404 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f75041012f0 msgr2=0x7f75041974e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:07:14.665 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:14.672+0000 7f7503fff640 1 --2- 172.21.15.29:0/177191404 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f75041012f0 0x7f75041974e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:07:14.665 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:14.672+0000 7f7503fff640 1 -- 172.21.15.29:0/177191404 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7504105680 msgr2=0x7f750419b860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:07:14.665 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:14.672+0000 7f7503fff640 1 --2- 172.21.15.29:0/177191404 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7504105680 0x7f750419b860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:07:14.665 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:14.672+0000 7f7503fff640 1 -- 172.21.15.29:0/177191404 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f750419bf90 con 0x7f7504100930 2024-03-16T12:07:14.665 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:14.672+0000 7f7503fff640 1 --2- 172.21.15.29:0/177191404 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7504100930 0x7f7504196fa0 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f74f400e9f0 tx=0x7f74f400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:07:14.665 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:14.672+0000 7f75017fa640 1 -- 172.21.15.29:0/177191404 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74f400cd80 con 0x7f7504100930 2024-03-16T12:07:14.665 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:14.672+0000 7f75017fa640 1 -- 172.21.15.29:0/177191404 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f74f4005500 con 0x7f7504100930 2024-03-16T12:07:14.665 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:14.672+0000 7f750a663640 1 -- 172.21.15.29:0/177191404 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f750419c280 con 0x7f7504100930 2024-03-16T12:07:14.665 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:14.672+0000 7f75017fa640 1 -- 172.21.15.29:0/177191404 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74f4010690 con 0x7f7504100930 2024-03-16T12:07:14.666 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:14.672+0000 7f750a663640 1 -- 172.21.15.29:0/177191404 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f75041a3b10 con 0x7f7504100930 2024-03-16T12:07:14.668 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:14.674+0000 7f75017fa640 1 -- 172.21.15.29:0/177191404 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7f74f40046c0 con 0x7f7504100930 2024-03-16T12:07:14.668 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:14.675+0000 7f750d247740 1 Processor -- start 2024-03-16T12:07:14.668 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:14.675+0000 7f750d247740 1 -- start start 2024-03-16T12:07:14.668 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:14.675+0000 7f75017fa640 1 --2- 172.21.15.29:0/177191404 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f74ec0a2a10 0x7f74ec0a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:07:14.668 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:14.675+0000 7f750d247740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559011026c10 0x559011026ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:07:14.668 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:14.675+0000 7f750d247740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559011027530 0x5590110324f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:07:14.669 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:14.675+0000 7f750d247740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559011032a30 0x559011034e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:07:14.669 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:14.675+0000 7f750d247740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x559010f8ba40 con 0x559011027530 2024-03-16T12:07:14.669 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:14.675+0000 7f750d247740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x559010f34920 con 0x559011032a30 2024-03-16T12:07:14.669 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:14.675+0000 7f750d247740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x559010f87ae0 con 0x559011026c10 2024-03-16T12:07:14.669 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:14.675+0000 7f75037fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559011027530 0x5590110324f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:07:14.669 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:14.675+0000 7f7508c00640 1 --2- 172.21.15.29:0/177191404 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f74ec0a2a10 0x7f74ec0a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:07:14.669 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:14.675+0000 7f7503fff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559011026c10 0x559011026ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:07:14.669 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:14.675+0000 7f75017fa640 1 -- 172.21.15.29:0/177191404 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 8600+0+0 (secure 0 0 0) 0x7f74f40cf240 con 0x7f7504100930 2024-03-16T12:07:14.669 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:14.675+0000 7f75037fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559011027530 0x5590110324f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:38774/0 (socket says 172.21.15.29:38774) 2024-03-16T12:07:14.669 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:14.675+0000 7f75037fe640 1 -- 172.21.15.29:0/1793961966 learned_addr learned my addr 172.21.15.29:0/1793961966 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T12:07:14.669 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:14.675+0000 7f7508c00640 1 --2- 172.21.15.29:0/1793961966 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559011032a30 0x559011034e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:07:14.669 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:14.675+0000 7f7503fff640 1 -- 172.21.15.29:0/1793961966 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559011032a30 msgr2=0x559011034e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:07:14.669 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:14.675+0000 7f7503fff640 1 --2- 172.21.15.29:0/1793961966 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559011032a30 0x559011034e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:07:14.669 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:14.675+0000 7f7503fff640 1 -- 172.21.15.29:0/1793961966 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559011027530 msgr2=0x5590110324f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:07:14.670 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:14.675+0000 7f7503fff640 1 --2- 172.21.15.29:0/1793961966 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559011027530 0x5590110324f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:07:14.670 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:14.675+0000 7f7503fff640 1 -- 172.21.15.29:0/1793961966 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559010f4c460 con 0x559011026c10 2024-03-16T12:07:14.670 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:14.676+0000 7f75037fe640 1 --2- 172.21.15.29:0/1793961966 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559011027530 0x5590110324f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:07:14.670 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:14.676+0000 7f7508c00640 1 --2- 172.21.15.29:0/177191404 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f74ec0a2a10 0x7f74ec0a4ed0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f74fc0098e0 tx=0x7f74fc009ef0 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:07:14.670 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:14.676+0000 7f7503fff640 1 --2- 172.21.15.29:0/1793961966 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559011026c10 0x559011026ff0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f74f40cc940 tx=0x7f74f409b6e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5872cc234fa68e16 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:07:14.670 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:14.676+0000 7f74ea7fc640 1 -- 172.21.15.29:0/1793961966 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74f409b850 con 0x559011026c10 2024-03-16T12:07:14.670 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:14.676+0000 7f74ea7fc640 1 -- 172.21.15.29:0/1793961966 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f74f409f9a0 con 0x559011026c10 2024-03-16T12:07:14.670 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:14.676+0000 7f74ea7fc640 1 -- 172.21.15.29:0/1793961966 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74f4097ec0 con 0x559011026c10 2024-03-16T12:07:14.670 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:14.677+0000 7f750d247740 1 -- 172.21.15.29:0/1793961966 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559011026c10 msgr2=0x559011026ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:07:14.670 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:14.677+0000 7f750d247740 1 --2- 172.21.15.29:0/1793961966 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559011026c10 0x559011026ff0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f74f40cc940 tx=0x7f74f409b6e0 comp rx=0 tx=0).stop 2024-03-16T12:07:14.670 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:14.677+0000 7f750d247740 1 -- 172.21.15.29:0/1793961966 shutdown_connections 2024-03-16T12:07:14.670 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:14.677+0000 7f750d247740 1 --2- 172.21.15.29:0/1793961966 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559011032a30 0x559011034e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:07:14.670 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:14.677+0000 7f750d247740 1 --2- 172.21.15.29:0/1793961966 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559011027530 0x5590110324f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:07:14.670 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:14.677+0000 7f750d247740 1 --2- 172.21.15.29:0/1793961966 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559011026c10 0x559011026ff0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:07:14.670 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:14.677+0000 7f750d247740 1 -- 172.21.15.29:0/1793961966 >> 172.21.15.29:0/1793961966 conn(0x559011021a50 msgr2=0x559011021e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:07:14.671 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:14.677+0000 7f750d247740 1 -- 172.21.15.29:0/1793961966 shutdown_connections 2024-03-16T12:07:14.671 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:14.677+0000 7f750d247740 1 -- 172.21.15.29:0/1793961966 wait complete. 2024-03-16T12:07:14.671 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:14.677+0000 7f750d247740 1 -- start start 2024-03-16T12:07:14.671 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:14.677+0000 7f750d247740 1 Processor -- start 2024-03-16T12:07:14.674 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:14.678+0000 7f750d247740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559011026c10 0x55901102fa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:07:14.674 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:14.678+0000 7f750d247740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559011027530 0x55901102ff50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:07:14.674 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:14.678+0000 7f750d247740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559011032a30 0x559011030490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:07:14.674 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:14.678+0000 7f750d247740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x5590110355a0 con 0x559011026c10 2024-03-16T12:07:14.674 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:14.678+0000 7f750d247740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x559010f47d70 con 0x559011032a30 2024-03-16T12:07:14.674 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:14.678+0000 7f750d247740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x559011035720 con 0x559011027530 2024-03-16T12:07:14.674 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:14.678+0000 7f750d247740 10 client.0 _ll_register_callbacks cb 0x559010c9fa00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T12:07:14.675 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:14.678+0000 7f7503fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559011026c10 0x55901102fa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:07:14.675 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:14.678+0000 7f7508c00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559011032a30 0x559011030490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:07:14.675 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:14.678+0000 7f75037fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559011027530 0x55901102ff50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:07:14.675 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:14.678+0000 7f7508c00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559011032a30 0x559011030490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.29:44682/0 (socket says 172.21.15.29:44682) 2024-03-16T12:07:14.675 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:14.678+0000 7f7503fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559011026c10 0x55901102fa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:38786/0 (socket says 172.21.15.29:38786) 2024-03-16T12:07:14.675 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:14.678+0000 7f7508c00640 1 -- 172.21.15.29:0/1651440744 learned_addr learned my addr 172.21.15.29:0/1651440744 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T12:07:14.675 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:14.678+0000 7f7508c00640 1 -- 172.21.15.29:0/1651440744 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559011027530 msgr2=0x55901102ff50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:07:14.675 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:14.678+0000 7f7508c00640 1 --2- 172.21.15.29:0/1651440744 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559011027530 0x55901102ff50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:07:14.675 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:14.678+0000 7f7508c00640 1 -- 172.21.15.29:0/1651440744 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559011026c10 msgr2=0x55901102fa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:07:14.675 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:14.678+0000 7f7508c00640 1 --2- 172.21.15.29:0/1651440744 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559011026c10 0x55901102fa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:07:14.675 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:14.678+0000 7f7508c00640 1 -- 172.21.15.29:0/1651440744 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559010ca0210 con 0x559011032a30 2024-03-16T12:07:14.675 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:14.678+0000 7f7503fff640 1 --2- 172.21.15.29:0/1651440744 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559011026c10 0x55901102fa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:07:14.675 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:14.678+0000 7f75037fe640 1 --2- 172.21.15.29:0/1651440744 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559011027530 0x55901102ff50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:07:14.675 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:14.678+0000 7f7508c00640 1 --2- 172.21.15.29:0/1651440744 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559011032a30 0x559011030490 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f74fc012c70 tx=0x7f74fc004ab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2cca37c7ed00290e server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:07:14.676 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:14.678+0000 7f74e37fe640 10 client.0 ms_handle_connect on v2:172.21.15.152:3300/0 2024-03-16T12:07:14.676 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:14.679+0000 7f74e37fe640 1 -- 172.21.15.29:0/1651440744 <== mon.2 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74fc010d70 con 0x559011032a30 2024-03-16T12:07:14.676 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:14.679+0000 7f750d247740 1 -- 172.21.15.29:0/1651440744 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f74f409f9a0 con 0x559011032a30 2024-03-16T12:07:14.676 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:14.679+0000 7f74e37fe640 1 -- 172.21.15.29:0/1651440744 <== mon.2 v2:172.21.15.152:3300/0 2 ==== osd_map(113..113 src has 1..113) ==== 8600+0+0 (secure 0 0 0) 0x7f74fc012ca0 con 0x559011032a30 2024-03-16T12:07:14.676 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:14.679+0000 7f74e37fe640 1 -- 172.21.15.29:0/1651440744 <== mon.2 v2:172.21.15.152:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f74fc0237a0 con 0x559011032a30 2024-03-16T12:07:14.676 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:14.679+0000 7f750d247740 10 client.26543 fetch_fsmap learned FSMap version 32 2024-03-16T12:07:14.676 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:14.679+0000 7f750d247740 1 -- 172.21.15.29:0/1651440744 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x559010d6c620 con 0x559011032a30 2024-03-16T12:07:14.676 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:14.679+0000 7f74e37fe640 1 -- 172.21.15.29:0/1651440744 <== mon.2 v2:172.21.15.152:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f74f409f9a0 con 0x559011032a30 2024-03-16T12:07:14.676 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:14.679+0000 7f74e37fe640 1 -- 172.21.15.29:0/1651440744 <== mon.2 v2:172.21.15.152:3300/0 5 ==== fsmap(e 32) ==== 8338+0+0 (secure 0 0 0) 0x7f74fc01f070 con 0x559011032a30 2024-03-16T12:07:14.676 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:14.679+0000 7f750d247740 10 client.26543 fetch_fsmap finished waiting for FSMap version 32 2024-03-16T12:07:14.676 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:14.679+0000 7f750d247740 10 client.26543 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T12:07:14.676 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:14.679+0000 7f750d247740 20 client.26543 populate_metadata read hostname 'smithi029' 2024-03-16T12:07:14.676 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:14.679+0000 7f750d247740 1 --2- 172.21.15.29:0/1651440744 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x5590110295c0 0x5590110299e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:07:14.676 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:14.679+0000 7f750d247740 4 client.26543 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T12:07:14.676 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:14.679+0000 7f750d247740 1 -- 172.21.15.29:0/1651440744 --> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559010f98f80 con 0x5590110295c0 2024-03-16T12:07:14.677 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:14.679+0000 7f7503fff640 1 --2- 172.21.15.29:0/1651440744 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x5590110295c0 0x5590110299e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:07:14.677 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:14.680+0000 7f7503fff640 1 --2- 172.21.15.29:0/1651440744 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x5590110295c0 0x5590110299e0 crc :-1 s=READY pgs=184 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d1c6457d18e274cc server_cookie=708d2587c3032d02 in_seq=0 out_seq=0 2024-03-16T12:07:14.677 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:14.680+0000 7f74e37fe640 10 client.26543 ms_handle_connect on v2:172.21.15.152:6838/1548731833 2024-03-16T12:07:14.677 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:14.681+0000 7f74e37fe640 1 -- 172.21.15.29:0/1651440744 <== mds.0 v2:172.21.15.152:6838/1548731833 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x559010f98f80 con 0x5590110295c0 2024-03-16T12:07:14.677 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:14.681+0000 7f74e37fe640 10 client.26543 handle_command_reply: tid=0 2024-03-16T12:07:14.677 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:14.681+0000 7f750d247740 1 client.26543 shutdown 2024-03-16T12:07:14.677 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:14.682+0000 7f750d247740 1 -- 172.21.15.29:0/1651440744 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559011032a30 msgr2=0x559011030490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:07:14.677 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:14.682+0000 7f750d247740 1 --2- 172.21.15.29:0/1651440744 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559011032a30 0x559011030490 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f74fc012c70 tx=0x7f74fc004ab0 comp rx=0 tx=0).stop 2024-03-16T12:07:14.677 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:14.682+0000 7f750d247740 1 -- 172.21.15.29:0/1651440744 shutdown_connections 2024-03-16T12:07:14.677 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:14.682+0000 7f750d247740 1 --2- 172.21.15.29:0/1651440744 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x5590110295c0 0x5590110299e0 crc :-1 s=READY pgs=184 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:07:14.677 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:14.682+0000 7f750d247740 1 --2- 172.21.15.29:0/1651440744 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559011032a30 0x559011030490 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:07:14.677 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:14.682+0000 7f750d247740 1 --2- 172.21.15.29:0/1651440744 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559011027530 0x55901102ff50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:07:14.677 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:14.682+0000 7f74e37fe640 0 client.26543 ms_handle_reset on v2:172.21.15.152:6838/1548731833 2024-03-16T12:07:14.677 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:14.682+0000 7f750d247740 1 --2- 172.21.15.29:0/1651440744 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559011026c10 0x55901102fa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:07:14.678 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:14.682+0000 7f750d247740 1 -- 172.21.15.29:0/1651440744 >> 172.21.15.29:0/1651440744 conn(0x559011021a50 msgr2=0x5590110237a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:07:14.678 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:14.682+0000 7f750d247740 1 -- 172.21.15.29:0/1651440744 shutdown_connections 2024-03-16T12:07:14.678 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:14.682+0000 7f750d247740 1 -- 172.21.15.29:0/1651440744 wait complete. 2024-03-16T12:07:14.678 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:14.682+0000 7f750d247740 20 client.26543 trim_cache size 0 max 16384 2024-03-16T12:07:14.691 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:14.698+0000 7f750d247740 1 Processor -- start 2024-03-16T12:07:14.691 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:14.698+0000 7f750d247740 1 -- start start 2024-03-16T12:07:14.691 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:14.698+0000 7f750d247740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559011021a50 0x559011032140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:07:14.691 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:14.698+0000 7f750d247740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559011026c10 0x559011046770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:07:14.691 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:14.698+0000 7f750d247740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559011027530 0x55901102cc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:07:14.691 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:14.698+0000 7f750d247740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f75041adbd0 con 0x559011027530 2024-03-16T12:07:14.691 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:14.698+0000 7f750d247740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f75041ab4a0 con 0x559011026c10 2024-03-16T12:07:14.691 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:14.698+0000 7f750d247740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f75041a9310 con 0x559011021a50 2024-03-16T12:07:14.691 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:14.698+0000 7f7508c00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559011027530 0x55901102cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:07:14.691 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:14.698+0000 7f7508c00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559011027530 0x55901102cc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:38792/0 (socket says 172.21.15.29:38792) 2024-03-16T12:07:14.691 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:14.698+0000 7f7508c00640 1 -- 172.21.15.29:0/2554022545 learned_addr learned my addr 172.21.15.29:0/2554022545 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T12:07:14.691 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:14.698+0000 7f75037fe640 1 --2- 172.21.15.29:0/2554022545 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559011026c10 0x559011046770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:07:14.691 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:14.698+0000 7f7503fff640 1 --2- 172.21.15.29:0/2554022545 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559011021a50 0x559011032140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:07:14.691 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:14.698+0000 7f7508c00640 1 -- 172.21.15.29:0/2554022545 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559011021a50 msgr2=0x559011032140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:07:14.692 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:14.698+0000 7f7508c00640 1 --2- 172.21.15.29:0/2554022545 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559011021a50 0x559011032140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:07:14.692 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:14.698+0000 7f7508c00640 1 -- 172.21.15.29:0/2554022545 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559011026c10 msgr2=0x559011046770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:07:14.692 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:14.698+0000 7f7508c00640 1 --2- 172.21.15.29:0/2554022545 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559011026c10 0x559011046770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:07:14.692 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:14.698+0000 7f7508c00640 1 -- 172.21.15.29:0/2554022545 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559011013140 con 0x559011027530 2024-03-16T12:07:14.692 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:14.698+0000 7f7508c00640 1 --2- 172.21.15.29:0/2554022545 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559011027530 0x55901102cc90 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f74fc005d70 tx=0x7f74fc004950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17ce6d2f838b384d server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:07:14.692 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:14.699+0000 7f74e9ffb640 1 -- 172.21.15.29:0/2554022545 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74fc01f070 con 0x559011027530 2024-03-16T12:07:14.692 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:14.699+0000 7f74e9ffb640 1 -- 172.21.15.29:0/2554022545 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f74fc023e20 con 0x559011027530 2024-03-16T12:07:14.692 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:14.699+0000 7f74e9ffb640 1 -- 172.21.15.29:0/2554022545 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74fc012e60 con 0x559011027530 2024-03-16T12:07:14.692 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:14.699+0000 7f750d247740 1 -- 172.21.15.29:0/2554022545 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559011027530 msgr2=0x55901102cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:07:14.692 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:14.699+0000 7f750d247740 1 --2- 172.21.15.29:0/2554022545 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559011027530 0x55901102cc90 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f74fc005d70 tx=0x7f74fc004950 comp rx=0 tx=0).stop 2024-03-16T12:07:14.692 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:14.699+0000 7f750d247740 1 -- 172.21.15.29:0/2554022545 shutdown_connections 2024-03-16T12:07:14.692 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:14.699+0000 7f750d247740 1 --2- 172.21.15.29:0/2554022545 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559011027530 0x55901102cc90 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:07:14.692 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:14.699+0000 7f750d247740 1 --2- 172.21.15.29:0/2554022545 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559011026c10 0x559011046770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:07:14.692 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:14.699+0000 7f750d247740 1 --2- 172.21.15.29:0/2554022545 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559011021a50 0x559011032140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:07:14.692 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:14.699+0000 7f750d247740 1 -- 172.21.15.29:0/2554022545 >> 172.21.15.29:0/2554022545 conn(0x55901102a5f0 msgr2=0x55901103e240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:07:14.693 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:14.699+0000 7f750d247740 1 -- 172.21.15.29:0/2554022545 shutdown_connections 2024-03-16T12:07:14.693 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:14.699+0000 7f750d247740 1 -- 172.21.15.29:0/2554022545 wait complete. 2024-03-16T12:07:14.693 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:14.700+0000 7f750d247740 1 -- start start 2024-03-16T12:07:14.693 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:14.700+0000 7f750d247740 1 Processor -- start 2024-03-16T12:07:14.695 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:14.700+0000 7f750d247740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559011021a50 0x5590110468c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:07:14.695 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:14.700+0000 7f750d247740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559011026c10 0x559011043ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:07:14.695 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:14.700+0000 7f750d247740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559011027530 0x5590110441e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:07:14.695 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:14.700+0000 7f750d247740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f75041a9960 con 0x559011021a50 2024-03-16T12:07:14.696 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:14.700+0000 7f750d247740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f75041aadd0 con 0x559011027530 2024-03-16T12:07:14.696 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:14.700+0000 7f750d247740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f75041aa030 con 0x559011026c10 2024-03-16T12:07:14.696 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:14.700+0000 7f750d247740 10 client.0 _ll_register_callbacks cb 0x7f74fc009b10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T12:07:14.696 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:14.700+0000 7f7503fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559011021a50 0x5590110468c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:07:14.696 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:14.700+0000 7f7503fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559011021a50 0x5590110468c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:38802/0 (socket says 172.21.15.29:38802) 2024-03-16T12:07:14.696 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:14.700+0000 7f7503fff640 1 -- 172.21.15.29:0/4119772575 learned_addr learned my addr 172.21.15.29:0/4119772575 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T12:07:14.696 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:14.700+0000 7f75037fe640 1 --2- 172.21.15.29:0/4119772575 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559011026c10 0x559011043ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:07:14.696 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:14.700+0000 7f7508c00640 1 --2- 172.21.15.29:0/4119772575 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559011027530 0x5590110441e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:07:14.696 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:14.700+0000 7f7503fff640 1 -- 172.21.15.29:0/4119772575 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559011026c10 msgr2=0x559011043ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:07:14.696 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:14.700+0000 7f7503fff640 1 --2- 172.21.15.29:0/4119772575 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559011026c10 0x559011043ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:07:14.696 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:14.700+0000 7f7503fff640 1 -- 172.21.15.29:0/4119772575 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559011027530 msgr2=0x5590110441e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:07:14.696 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:14.701+0000 7f7503fff640 1 --2- 172.21.15.29:0/4119772575 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559011027530 0x5590110441e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:07:14.696 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:14.701+0000 7f7503fff640 1 -- 172.21.15.29:0/4119772575 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559011044880 con 0x559011021a50 2024-03-16T12:07:14.696 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:14.701+0000 7f7503fff640 1 --2- 172.21.15.29:0/4119772575 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559011021a50 0x5590110468c0 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f74f4097cb0 tx=0x7f74f409b2b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f29d63edaba53626 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:07:14.696 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:14.701+0000 7f74e37fe640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-16T12:07:14.697 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:14.701+0000 7f74e37fe640 1 -- 172.21.15.29:0/4119772575 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74f40a3ff0 con 0x559011021a50 2024-03-16T12:07:14.697 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:14.701+0000 7f74e37fe640 1 -- 172.21.15.29:0/4119772575 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(113..113 src has 1..113) ==== 8600+0+0 (secure 0 0 0) 0x7f74f40ccbc0 con 0x559011021a50 2024-03-16T12:07:14.697 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:14.701+0000 7f750d247740 1 -- 172.21.15.29:0/4119772575 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f74fc023e20 con 0x559011021a50 2024-03-16T12:07:14.697 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:14.701+0000 7f74e37fe640 1 -- 172.21.15.29:0/4119772575 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f74f409f9a0 con 0x559011021a50 2024-03-16T12:07:14.697 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:14.701+0000 7f750d247740 10 client.16749 fetch_fsmap learned FSMap version 32 2024-03-16T12:07:14.697 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:14.701+0000 7f750d247740 1 -- 172.21.15.29:0/4119772575 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5590110a6d50 con 0x559011021a50 2024-03-16T12:07:14.697 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:14.701+0000 7f74e37fe640 1 -- 172.21.15.29:0/4119772575 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f74fc023e20 con 0x559011021a50 2024-03-16T12:07:14.697 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:14.701+0000 7f74e37fe640 1 -- 172.21.15.29:0/4119772575 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 32) ==== 8338+0+0 (secure 0 0 0) 0x7f74f40a3b60 con 0x559011021a50 2024-03-16T12:07:14.697 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:14.701+0000 7f750d247740 10 client.16749 fetch_fsmap finished waiting for FSMap version 32 2024-03-16T12:07:14.697 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:14.701+0000 7f750d247740 10 client.16749 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T12:07:14.697 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:14.701+0000 7f750d247740 20 client.16749 populate_metadata read hostname 'smithi029' 2024-03-16T12:07:14.697 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:14.701+0000 7f750d247740 1 --2- 172.21.15.29:0/4119772575 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x559011032a30 0x5590110a9040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:07:14.697 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:14.701+0000 7f750d247740 4 client.16749 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-16T12:07:14.697 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:14.701+0000 7f750d247740 1 -- 172.21.15.29:0/4119772575 --> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] -- command(tid 0: {"prefix": "flush journal"}) -- 0x559010f932d0 con 0x559011032a30 2024-03-16T12:07:14.697 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:14.702+0000 7f75037fe640 1 --2- 172.21.15.29:0/4119772575 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x559011032a30 0x5590110a9040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:07:14.698 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:14.702+0000 7f75037fe640 1 --2- 172.21.15.29:0/4119772575 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x559011032a30 0x5590110a9040 crc :-1 s=READY pgs=185 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=13ffb0a59f4e1681 server_cookie=17eabe568941a2b0 in_seq=0 out_seq=0 2024-03-16T12:07:14.698 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:14.702+0000 7f74e37fe640 10 client.16749 ms_handle_connect on v2:172.21.15.152:6838/1548731833 2024-03-16T12:07:15.188 INFO:tasks.workunit.client.0.smithi029.stdout: 10 46954 12.89 MB/sec execute 59 sec latency 512.358 ms 2024-03-16T12:07:16.187 INFO:tasks.workunit.client.0.smithi029.stdout: 10 47173 12.90 MB/sec execute 60 sec latency 324.419 ms 2024-03-16T12:07:16.537 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:07:16 smithi154 ceph-mon[36320]: pgmap v518: 97 pgs: 38 active+clean+snaptrim_wait, 21 active+clean+snaptrim, 38 active+clean; 351 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 5.1 MiB/s wr, 570 op/s 2024-03-16T12:07:16.579 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:07:16 smithi029 ceph-mon[31975]: pgmap v518: 97 pgs: 38 active+clean+snaptrim_wait, 21 active+clean+snaptrim, 38 active+clean; 351 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 5.1 MiB/s wr, 570 op/s 2024-03-16T12:07:16.619 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:07:16 smithi152 ceph-mon[36291]: pgmap v518: 97 pgs: 38 active+clean+snaptrim_wait, 21 active+clean+snaptrim, 38 active+clean; 351 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 5.1 MiB/s wr, 570 op/s 2024-03-16T12:07:17.188 INFO:tasks.workunit.client.0.smithi029.stdout: 10 47364 12.89 MB/sec execute 61 sec latency 255.079 ms 2024-03-16T12:07:18.188 INFO:tasks.workunit.client.0.smithi029.stdout: 10 47515 12.77 MB/sec execute 62 sec latency 365.049 ms 2024-03-16T12:07:18.537 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:07:18 smithi154 ceph-mon[36320]: pgmap v519: 97 pgs: 97 active+clean; 355 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 4.5 MiB/s wr, 509 op/s 2024-03-16T12:07:18.579 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:07:18 smithi029 ceph-mon[31975]: pgmap v519: 97 pgs: 97 active+clean; 355 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 4.5 MiB/s wr, 509 op/s 2024-03-16T12:07:18.619 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:07:18 smithi152 ceph-mon[36291]: pgmap v519: 97 pgs: 97 active+clean; 355 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 4.5 MiB/s wr, 509 op/s 2024-03-16T12:07:19.188 INFO:tasks.workunit.client.0.smithi029.stdout: 10 47692 12.66 MB/sec execute 63 sec latency 385.188 ms 2024-03-16T12:07:19.359 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:19.367+0000 7f74e37fe640 1 -- 172.21.15.29:0/4119772575 <== mds.0 v2:172.21.15.152:6838/1548731833 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x559010f932d0 con 0x559011032a30 2024-03-16T12:07:19.359 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:19.367+0000 7f74e37fe640 10 client.16749 handle_command_reply: tid=0 2024-03-16T12:07:19.359 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:19.368+0000 7f750d247740 1 client.16749 shutdown 2024-03-16T12:07:19.359 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:19.368+0000 7f750d247740 1 -- 172.21.15.29:0/4119772575 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559011021a50 msgr2=0x5590110468c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:07:19.359 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:19.368+0000 7f750d247740 1 --2- 172.21.15.29:0/4119772575 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559011021a50 0x5590110468c0 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f74f4097cb0 tx=0x7f74f409b2b0 comp rx=0 tx=0).stop 2024-03-16T12:07:19.360 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:19.368+0000 7f750d247740 1 -- 172.21.15.29:0/4119772575 shutdown_connections 2024-03-16T12:07:19.360 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:19.368+0000 7f750d247740 1 --2- 172.21.15.29:0/4119772575 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x559011032a30 0x5590110a9040 crc :-1 s=READY pgs=185 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:07:19.360 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:19.368+0000 7f750d247740 1 --2- 172.21.15.29:0/4119772575 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559011027530 0x5590110441e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:07:19.360 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:19.368+0000 7f74e37fe640 0 client.16749 ms_handle_reset on v2:172.21.15.152:6838/1548731833 2024-03-16T12:07:19.360 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:19.368+0000 7f750d247740 1 --2- 172.21.15.29:0/4119772575 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559011026c10 0x559011043ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:07:19.360 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:19.368+0000 7f750d247740 1 --2- 172.21.15.29:0/4119772575 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559011021a50 0x5590110468c0 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:07:19.360 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:19.368+0000 7f750d247740 1 -- 172.21.15.29:0/4119772575 >> 172.21.15.29:0/4119772575 conn(0x55901102a5f0 msgr2=0x55901102f9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:07:19.360 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:19.368+0000 7f750d247740 1 -- 172.21.15.29:0/4119772575 shutdown_connections 2024-03-16T12:07:19.360 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:19.368+0000 7f750d247740 1 -- 172.21.15.29:0/4119772575 wait complete. 2024-03-16T12:07:19.360 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:19.368+0000 7f750d247740 20 client.16749 trim_cache size 0 max 16384 2024-03-16T12:07:19.360 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:{ 2024-03-16T12:07:19.361 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "message": "", 2024-03-16T12:07:19.361 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "return_code": 0 2024-03-16T12:07:19.361 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:} 2024-03-16T12:07:19.361 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:19.369+0000 7f750a663640 1 -- 172.21.15.29:0/177191404 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f74ec0a2a10 msgr2=0x7f74ec0a4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:07:19.362 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:19.369+0000 7f750a663640 1 --2- 172.21.15.29:0/177191404 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f74ec0a2a10 0x7f74ec0a4ed0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f74fc0098e0 tx=0x7f74fc009ef0 comp rx=0 tx=0).stop 2024-03-16T12:07:19.362 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:19.369+0000 7f750a663640 1 -- 172.21.15.29:0/177191404 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7504100930 msgr2=0x7f7504196fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:07:19.362 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:19.369+0000 7f750a663640 1 --2- 172.21.15.29:0/177191404 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7504100930 0x7f7504196fa0 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f74f400e9f0 tx=0x7f74f400edf0 comp rx=0 tx=0).stop 2024-03-16T12:07:19.362 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:19.369+0000 7f750a663640 1 -- 172.21.15.29:0/177191404 shutdown_connections 2024-03-16T12:07:19.362 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:19.369+0000 7f750a663640 1 --2- 172.21.15.29:0/177191404 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f74ec0a2a10 0x7f74ec0a4ed0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:07:19.362 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:19.369+0000 7f750a663640 1 --2- 172.21.15.29:0/177191404 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7504105680 0x7f750419b860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:07:19.362 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:19.369+0000 7f750a663640 1 --2- 172.21.15.29:0/177191404 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f75041012f0 0x7f75041974e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:07:19.362 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:19.369+0000 7f750a663640 1 --2- 172.21.15.29:0/177191404 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7504100930 0x7f7504196fa0 unknown :-1 s=CLOSED pgs=766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:07:19.362 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:19.369+0000 7f750a663640 1 -- 172.21.15.29:0/177191404 >> 172.21.15.29:0/177191404 conn(0x7f7504077530 msgr2=0x7f75040fe9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:07:19.362 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:19.369+0000 7f750a663640 1 -- 172.21.15.29:0/177191404 shutdown_connections 2024-03-16T12:07:19.362 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:19.369+0000 7f750a663640 1 -- 172.21.15.29:0/177191404 wait complete. 2024-03-16T12:07:20.188 INFO:tasks.workunit.client.0.smithi029.stdout: 10 47932 12.63 MB/sec execute 64 sec latency 352.430 ms 2024-03-16T12:07:20.537 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:07:20 smithi154 ceph-mon[36320]: pgmap v520: 97 pgs: 97 active+clean; 355 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 4.5 MiB/s wr, 509 op/s 2024-03-16T12:07:20.537 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:07:20 smithi154 ceph-mon[36320]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T12:07:20.579 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:07:20 smithi029 ceph-mon[31975]: pgmap v520: 97 pgs: 97 active+clean; 355 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 4.5 MiB/s wr, 509 op/s 2024-03-16T12:07:20.579 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:07:20 smithi029 ceph-mon[31975]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T12:07:20.619 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:07:20 smithi152 ceph-mon[36291]: pgmap v520: 97 pgs: 97 active+clean; 355 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 4.5 MiB/s wr, 509 op/s 2024-03-16T12:07:20.619 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:07:20 smithi152 ceph-mon[36291]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T12:07:21.188 INFO:tasks.workunit.client.0.smithi029.stdout: 10 48206 12.56 MB/sec execute 65 sec latency 264.237 ms 2024-03-16T12:07:21.537 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:07:21 smithi154 ceph-mon[36320]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:07:21.537 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:07:21 smithi154 ceph-mon[36320]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T12:07:21.537 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:07:21 smithi154 ceph-mon[36320]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' 2024-03-16T12:07:21.579 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:07:21 smithi029 ceph-mon[31975]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:07:21.579 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:07:21 smithi029 ceph-mon[31975]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T12:07:21.579 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:07:21 smithi029 ceph-mon[31975]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' 2024-03-16T12:07:21.619 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:07:21 smithi152 ceph-mon[36291]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:07:21.619 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:07:21 smithi152 ceph-mon[36291]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T12:07:21.619 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:07:21 smithi152 ceph-mon[36291]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' 2024-03-16T12:07:22.188 INFO:tasks.workunit.client.0.smithi029.stdout: 10 48573 12.61 MB/sec execute 66 sec latency 63.559 ms 2024-03-16T12:07:22.619 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:07:22 smithi152 ceph-mon[36291]: pgmap v521: 97 pgs: 97 active+clean; 350 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 4.2 MiB/s wr, 448 op/s 2024-03-16T12:07:22.619 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:07:22 smithi152 ceph-mon[36291]: osdmap e114: 12 total, 12 up, 12 in 2024-03-16T12:07:22.786 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:07:22 smithi154 ceph-mon[36320]: pgmap v521: 97 pgs: 97 active+clean; 350 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 4.2 MiB/s wr, 448 op/s 2024-03-16T12:07:22.787 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:07:22 smithi154 ceph-mon[36320]: osdmap e114: 12 total, 12 up, 12 in 2024-03-16T12:07:22.829 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:07:22 smithi029 ceph-mon[31975]: pgmap v521: 97 pgs: 97 active+clean; 350 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 4.2 MiB/s wr, 448 op/s 2024-03-16T12:07:22.829 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:07:22 smithi029 ceph-mon[31975]: osdmap e114: 12 total, 12 up, 12 in 2024-03-16T12:07:23.188 INFO:tasks.workunit.client.0.smithi029.stdout: 10 48849 12.62 MB/sec execute 67 sec latency 265.141 ms 2024-03-16T12:07:24.188 INFO:tasks.workunit.client.0.smithi029.stdout: 10 49247 12.80 MB/sec execute 68 sec latency 98.210 ms 2024-03-16T12:07:24.619 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:07:24 smithi152 ceph-mon[36291]: pgmap v523: 97 pgs: 97 active+clean; 361 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 5.5 MiB/s wr, 406 op/s 2024-03-16T12:07:24.787 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:07:24 smithi154 ceph-mon[36320]: pgmap v523: 97 pgs: 97 active+clean; 361 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 5.5 MiB/s wr, 406 op/s 2024-03-16T12:07:24.829 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:07:24 smithi029 ceph-mon[31975]: pgmap v523: 97 pgs: 97 active+clean; 361 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 5.5 MiB/s wr, 406 op/s 2024-03-16T12:07:25.188 INFO:tasks.workunit.client.0.smithi029.stdout: 10 49662 13.08 MB/sec execute 69 sec latency 199.426 ms 2024-03-16T12:07:26.188 INFO:tasks.workunit.client.0.smithi029.stdout: 10 50030 13.17 MB/sec execute 70 sec latency 222.699 ms 2024-03-16T12:07:26.619 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:07:26 smithi152 ceph-mon[36291]: pgmap v524: 97 pgs: 97 active+clean; 361 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 5.5 MiB/s wr, 406 op/s 2024-03-16T12:07:26.786 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:07:26 smithi154 ceph-mon[36320]: pgmap v524: 97 pgs: 97 active+clean; 361 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 5.5 MiB/s wr, 406 op/s 2024-03-16T12:07:26.829 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:07:26 smithi029 ceph-mon[31975]: pgmap v524: 97 pgs: 97 active+clean; 361 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 5.5 MiB/s wr, 406 op/s 2024-03-16T12:07:27.188 INFO:tasks.workunit.client.0.smithi029.stdout: 10 50286 13.23 MB/sec execute 71 sec latency 240.838 ms 2024-03-16T12:07:28.188 INFO:tasks.workunit.client.0.smithi029.stdout: 10 50457 13.25 MB/sec execute 72 sec latency 409.462 ms 2024-03-16T12:07:28.619 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:07:28 smithi152 ceph-mon[36291]: pgmap v525: 97 pgs: 97 active+clean; 370 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 5.2 MiB/s wr, 282 op/s 2024-03-16T12:07:28.630 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:07:28 smithi029 ceph-mon[31975]: pgmap v525: 97 pgs: 97 active+clean; 370 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 5.2 MiB/s wr, 282 op/s 2024-03-16T12:07:28.786 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:07:28 smithi154 ceph-mon[36320]: pgmap v525: 97 pgs: 97 active+clean; 370 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 5.2 MiB/s wr, 282 op/s 2024-03-16T12:07:29.188 INFO:tasks.workunit.client.0.smithi029.stdout: 10 50624 13.14 MB/sec execute 73 sec latency 276.714 ms 2024-03-16T12:07:30.188 INFO:tasks.workunit.client.0.smithi029.stdout: 10 50801 13.14 MB/sec execute 74 sec latency 214.730 ms 2024-03-16T12:07:30.619 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:07:30 smithi152 ceph-mon[36291]: pgmap v526: 97 pgs: 97 active+clean; 370 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 5.2 MiB/s wr, 282 op/s 2024-03-16T12:07:30.786 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:07:30 smithi154 ceph-mon[36320]: pgmap v526: 97 pgs: 97 active+clean; 370 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 5.2 MiB/s wr, 282 op/s 2024-03-16T12:07:30.829 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:07:30 smithi029 ceph-mon[31975]: pgmap v526: 97 pgs: 97 active+clean; 370 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 5.2 MiB/s wr, 282 op/s 2024-03-16T12:07:31.188 INFO:tasks.workunit.client.0.smithi029.stdout: 10 50985 13.06 MB/sec execute 75 sec latency 112.224 ms 2024-03-16T12:07:32.188 INFO:tasks.workunit.client.0.smithi029.stdout: 10 51216 12.96 MB/sec execute 76 sec latency 73.360 ms 2024-03-16T12:07:32.471 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:32.477+0000 7f77a0b53640 1 -- 172.21.15.29:0/3696653001 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f779c069800 msgr2=0x7f779c10e670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:07:32.471 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:32.477+0000 7f77a0b53640 1 --2- 172.21.15.29:0/3696653001 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f779c069800 0x7f779c10e670 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f7794009cd0 tx=0x7f779402c0f0 comp rx=0 tx=0).stop 2024-03-16T12:07:32.471 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:32.478+0000 7f77a0b53640 1 -- 172.21.15.29:0/3696653001 shutdown_connections 2024-03-16T12:07:32.471 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:32.478+0000 7f77a0b53640 1 --2- 172.21.15.29:0/3696653001 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f779c069800 0x7f779c10e670 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:07:32.472 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:32.478+0000 7f77a0b53640 1 --2- 172.21.15.29:0/3696653001 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f779c068e60 0x7f779c0692c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:07:32.472 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:32.478+0000 7f77a0b53640 1 --2- 172.21.15.29:0/3696653001 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f779c1078b0 0x7f779c107c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:07:32.472 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:32.478+0000 7f77a0b53640 1 -- 172.21.15.29:0/3696653001 >> 172.21.15.29:0/3696653001 conn(0x7f779c077530 msgr2=0x7f779c077940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:07:32.472 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:32.478+0000 7f77a0b53640 1 -- 172.21.15.29:0/3696653001 shutdown_connections 2024-03-16T12:07:32.472 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:32.478+0000 7f77a0b53640 1 -- 172.21.15.29:0/3696653001 wait complete. 2024-03-16T12:07:32.472 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:32.478+0000 7f77a0b53640 1 Processor -- start 2024-03-16T12:07:32.472 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:32.478+0000 7f77a0b53640 1 -- start start 2024-03-16T12:07:32.472 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:32.478+0000 7f77a0b53640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f779c068e60 0x7f779c197f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:07:32.472 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:32.478+0000 7f77a0b53640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f779c069800 0x7f779c198480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:07:32.472 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:32.478+0000 7f77a0b53640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f779c1078b0 0x7f779c19c800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:07:32.472 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:32.478+0000 7f77a0b53640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f779c111ba0 con 0x7f779c1078b0 2024-03-16T12:07:32.472 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:32.478+0000 7f77a0b53640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f779c111a20 con 0x7f779c068e60 2024-03-16T12:07:32.472 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:32.478+0000 7f77a0b53640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f779c111d20 con 0x7f779c069800 2024-03-16T12:07:32.472 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:32.479+0000 7f779ad9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f779c1078b0 0x7f779c19c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:07:32.472 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:32.479+0000 7f7799d9b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f779c069800 0x7f779c198480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:07:32.473 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:32.479+0000 7f779a59c640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f779c068e60 0x7f779c197f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:07:32.473 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:32.479+0000 7f779ad9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f779c1078b0 0x7f779c19c800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:37662/0 (socket says 172.21.15.29:37662) 2024-03-16T12:07:32.473 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:32.479+0000 7f779ad9d640 1 -- 172.21.15.29:0/3333287501 learned_addr learned my addr 172.21.15.29:0/3333287501 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T12:07:32.473 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:32.479+0000 7f7799d9b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f779c069800 0x7f779c198480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.29:55412/0 (socket says 172.21.15.29:55412) 2024-03-16T12:07:32.473 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:32.479+0000 7f779ad9d640 1 -- 172.21.15.29:0/3333287501 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f779c069800 msgr2=0x7f779c198480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:07:32.473 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:32.479+0000 7f779ad9d640 1 --2- 172.21.15.29:0/3333287501 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f779c069800 0x7f779c198480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:07:32.473 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:32.479+0000 7f779ad9d640 1 -- 172.21.15.29:0/3333287501 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f779c068e60 msgr2=0x7f779c197f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:07:32.473 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:32.479+0000 7f779ad9d640 1 --2- 172.21.15.29:0/3333287501 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f779c068e60 0x7f779c197f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:07:32.473 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:32.479+0000 7f779ad9d640 1 -- 172.21.15.29:0/3333287501 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f779c19cf30 con 0x7f779c1078b0 2024-03-16T12:07:32.473 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:32.479+0000 7f779ad9d640 1 --2- 172.21.15.29:0/3333287501 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f779c1078b0 0x7f779c19c800 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f7794004af0 tx=0x7f7794032c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:07:32.473 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:32.479+0000 7f778b7fe640 1 -- 172.21.15.29:0/3333287501 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7794042020 con 0x7f779c1078b0 2024-03-16T12:07:32.473 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:32.479+0000 7f77a0b53640 1 -- 172.21.15.29:0/3333287501 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f779c19d1c0 con 0x7f779c1078b0 2024-03-16T12:07:32.473 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:32.479+0000 7f778b7fe640 1 -- 172.21.15.29:0/3333287501 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7794032e80 con 0x7f779c1078b0 2024-03-16T12:07:32.473 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:32.479+0000 7f778b7fe640 1 -- 172.21.15.29:0/3333287501 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7794039040 con 0x7f779c1078b0 2024-03-16T12:07:32.473 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:32.479+0000 7f77a0b53640 1 -- 172.21.15.29:0/3333287501 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f779c1a4ab0 con 0x7f779c1078b0 2024-03-16T12:07:32.479 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:32.482+0000 7f778b7fe640 1 -- 172.21.15.29:0/3333287501 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7f7794045020 con 0x7f779c1078b0 2024-03-16T12:07:32.479 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:32.482+0000 7f77a381a740 1 Processor -- start 2024-03-16T12:07:32.479 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:32.482+0000 7f77a381a740 1 -- start start 2024-03-16T12:07:32.479 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:32.483+0000 7f77a381a740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559e9a0aa130 0x559e9a0aa510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:07:32.479 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:32.483+0000 7f77a381a740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559e9a0aaa50 0x559e9a0b32e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:07:32.479 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:32.483+0000 7f77a381a740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559e9a0b3820 0x559e9a0b5c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:07:32.480 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:32.483+0000 7f77a381a740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x559e9a00ea40 con 0x559e9a0aa130 2024-03-16T12:07:32.480 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:32.483+0000 7f77a381a740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x559e99fb7920 con 0x559e9a0b3820 2024-03-16T12:07:32.480 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:32.483+0000 7f77a381a740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x559e9a00aae0 con 0x559e9a0aaa50 2024-03-16T12:07:32.480 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:32.483+0000 7f7799d9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559e9a0aa130 0x559e9a0aa510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:07:32.480 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:32.483+0000 7f7799d9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559e9a0aa130 0x559e9a0aa510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:37674/0 (socket says 172.21.15.29:37674) 2024-03-16T12:07:32.480 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:32.483+0000 7f7799d9b640 1 -- 172.21.15.29:0/430351645 learned_addr learned my addr 172.21.15.29:0/430351645 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T12:07:32.480 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:32.483+0000 7f779a59c640 1 --2- 172.21.15.29:0/430351645 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559e9a0aaa50 0x559e9a0b32e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:07:32.480 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:32.483+0000 7f778b7fe640 1 --2- 172.21.15.29:0/3333287501 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f77840a2a10 0x7f77840a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:07:32.480 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:32.483+0000 7f7799d9b640 1 --2- 172.21.15.29:0/430351645 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559e9a0b3820 0x559e9a0b5c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:07:32.480 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:32.483+0000 7f779ad9d640 1 --2- 172.21.15.29:0/3333287501 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f77840a2a10 0x7f77840a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:07:32.480 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:32.483+0000 7f7799d9b640 1 -- 172.21.15.29:0/430351645 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559e9a0aaa50 msgr2=0x559e9a0b32e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:07:32.480 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:32.483+0000 7f7799d9b640 1 --2- 172.21.15.29:0/430351645 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559e9a0aaa50 0x559e9a0b32e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:07:32.480 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:32.483+0000 7f7799d9b640 1 -- 172.21.15.29:0/430351645 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559e9a0b3820 msgr2=0x559e9a0b5c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:07:32.480 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:32.483+0000 7f7799d9b640 1 --2- 172.21.15.29:0/430351645 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559e9a0b3820 0x559e9a0b5c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:07:32.480 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:32.483+0000 7f7799d9b640 1 -- 172.21.15.29:0/430351645 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559e99fcf460 con 0x559e9a0aa130 2024-03-16T12:07:32.480 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:32.483+0000 7f779a59c640 1 --2- 172.21.15.29:0/430351645 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559e9a0aaa50 0x559e9a0b32e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:07:32.481 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:32.483+0000 7f778b7fe640 1 -- 172.21.15.29:0/3333287501 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(114..114 src has 1..114) ==== 8556+0+0 (secure 0 0 0) 0x7f77940f0bf0 con 0x7f779c1078b0 2024-03-16T12:07:32.481 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:32.483+0000 7f7799d9b640 1 --2- 172.21.15.29:0/430351645 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559e9a0aa130 0x559e9a0aa510 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f7790009710 tx=0x7f7790009830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8d4631cfc1f37c0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:07:32.481 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:32.483+0000 7f7788ff9640 1 -- 172.21.15.29:0/430351645 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7790004030 con 0x559e9a0aa130 2024-03-16T12:07:32.481 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:32.483+0000 7f779ad9d640 1 --2- 172.21.15.29:0/3333287501 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f77840a2a10 0x7f77840a4ed0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f7794009430 tx=0x7f77940885a0 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:07:32.481 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:32.484+0000 7f7788ff9640 1 -- 172.21.15.29:0/430351645 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f77900041d0 con 0x559e9a0aa130 2024-03-16T12:07:32.481 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:32.484+0000 7f7788ff9640 1 -- 172.21.15.29:0/430351645 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f779000f760 con 0x559e9a0aa130 2024-03-16T12:07:32.481 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:32.484+0000 7f77a381a740 1 -- 172.21.15.29:0/430351645 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559e9a0aa130 msgr2=0x559e9a0aa510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:07:32.481 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:32.484+0000 7f77a381a740 1 --2- 172.21.15.29:0/430351645 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559e9a0aa130 0x559e9a0aa510 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f7790009710 tx=0x7f7790009830 comp rx=0 tx=0).stop 2024-03-16T12:07:32.481 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:32.484+0000 7f77a381a740 1 -- 172.21.15.29:0/430351645 shutdown_connections 2024-03-16T12:07:32.481 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:32.484+0000 7f77a381a740 1 --2- 172.21.15.29:0/430351645 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559e9a0b3820 0x559e9a0b5c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:07:32.481 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:32.484+0000 7f77a381a740 1 --2- 172.21.15.29:0/430351645 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559e9a0aaa50 0x559e9a0b32e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:07:32.481 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:32.484+0000 7f77a381a740 1 --2- 172.21.15.29:0/430351645 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559e9a0aa130 0x559e9a0aa510 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:07:32.481 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:32.484+0000 7f77a381a740 1 -- 172.21.15.29:0/430351645 >> 172.21.15.29:0/430351645 conn(0x559e9a0a4a50 msgr2=0x559e9a0a4e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:07:32.481 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:32.484+0000 7f77a381a740 1 -- 172.21.15.29:0/430351645 shutdown_connections 2024-03-16T12:07:32.482 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:32.484+0000 7f77a381a740 1 -- 172.21.15.29:0/430351645 wait complete. 2024-03-16T12:07:32.482 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:32.485+0000 7f77a381a740 1 -- start start 2024-03-16T12:07:32.482 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:32.485+0000 7f77a381a740 1 Processor -- start 2024-03-16T12:07:32.482 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:32.485+0000 7f77a381a740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559e9a0aa130 0x559e9a0b0810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:07:32.482 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:32.485+0000 7f77a381a740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559e9a0aaa50 0x559e9a0b0d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:07:32.482 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:32.485+0000 7f77a381a740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559e9a0b3820 0x559e9a0b1290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:07:32.482 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:32.485+0000 7f77a381a740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x559e9a0b6450 con 0x559e9a0aaa50 2024-03-16T12:07:32.482 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:32.485+0000 7f77a381a740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x559e99fcad70 con 0x559e9a0aa130 2024-03-16T12:07:32.482 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:32.485+0000 7f77a381a740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x559e9a0b65d0 con 0x559e9a0b3820 2024-03-16T12:07:32.482 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:32.485+0000 7f77a381a740 10 client.0 _ll_register_callbacks cb 0x559e99d22a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T12:07:32.482 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:32.485+0000 7f779a59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559e9a0aaa50 0x559e9a0b0d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:07:32.482 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:32.485+0000 7f779a59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559e9a0aaa50 0x559e9a0b0d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:37680/0 (socket says 172.21.15.29:37680) 2024-03-16T12:07:32.482 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:32.485+0000 7f7799d9b640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559e9a0aa130 0x559e9a0b0810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:07:32.482 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:32.485+0000 7f779a59c640 1 -- 172.21.15.29:0/2384934346 learned_addr learned my addr 172.21.15.29:0/2384934346 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T12:07:32.483 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:32.485+0000 7f7799d9b640 1 --2- 172.21.15.29:0/2384934346 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559e9a0b3820 0x559e9a0b1290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:07:32.483 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:32.485+0000 7f779a59c640 1 -- 172.21.15.29:0/2384934346 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559e9a0b3820 msgr2=0x559e9a0b1290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:07:32.483 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:32.485+0000 7f779a59c640 1 --2- 172.21.15.29:0/2384934346 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559e9a0b3820 0x559e9a0b1290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:07:32.483 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:32.485+0000 7f779a59c640 1 -- 172.21.15.29:0/2384934346 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559e9a0aa130 msgr2=0x559e9a0b0810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:07:32.483 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:32.485+0000 7f779a59c640 1 --2- 172.21.15.29:0/2384934346 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559e9a0aa130 0x559e9a0b0810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:07:32.483 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:32.485+0000 7f779a59c640 1 -- 172.21.15.29:0/2384934346 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559e99d23210 con 0x559e9a0aaa50 2024-03-16T12:07:32.483 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:32.486+0000 7f779a59c640 1 --2- 172.21.15.29:0/2384934346 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559e9a0aaa50 0x559e9a0b0d50 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f778c012740 tx=0x7f778c012c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca07a9c6d21b0699 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:07:32.483 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:32.486+0000 7f7781ffb640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-16T12:07:32.483 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:32.486+0000 7f7781ffb640 1 -- 172.21.15.29:0/2384934346 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f778c004030 con 0x559e9a0aaa50 2024-03-16T12:07:32.483 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:32.486+0000 7f7781ffb640 1 -- 172.21.15.29:0/2384934346 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(114..114 src has 1..114) ==== 8556+0+0 (secure 0 0 0) 0x7f778c0041d0 con 0x559e9a0aaa50 2024-03-16T12:07:32.483 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:32.486+0000 7f77a381a740 1 -- 172.21.15.29:0/2384934346 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f77900041d0 con 0x559e9a0aaa50 2024-03-16T12:07:32.483 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:32.486+0000 7f7781ffb640 1 -- 172.21.15.29:0/2384934346 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f778c0043d0 con 0x559e9a0aaa50 2024-03-16T12:07:32.483 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:32.486+0000 7f77a381a740 10 client.16761 fetch_fsmap learned FSMap version 32 2024-03-16T12:07:32.483 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:32.486+0000 7f77a381a740 1 -- 172.21.15.29:0/2384934346 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x559e99def620 con 0x559e9a0aaa50 2024-03-16T12:07:32.483 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:32.486+0000 7f7781ffb640 1 -- 172.21.15.29:0/2384934346 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f77900041d0 con 0x559e9a0aaa50 2024-03-16T12:07:32.483 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:32.486+0000 7f7781ffb640 1 -- 172.21.15.29:0/2384934346 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 32) ==== 8338+0+0 (secure 0 0 0) 0x7f778c003c30 con 0x559e9a0aaa50 2024-03-16T12:07:32.483 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:32.486+0000 7f77a381a740 10 client.16761 fetch_fsmap finished waiting for FSMap version 32 2024-03-16T12:07:32.484 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:32.486+0000 7f77a381a740 10 client.16761 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T12:07:32.484 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:32.486+0000 7f77a381a740 20 client.16761 populate_metadata read hostname 'smithi029' 2024-03-16T12:07:32.484 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:32.486+0000 7f77a381a740 1 --2- 172.21.15.29:0/2384934346 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x559e9a0c39f0 0x559e9a0c5de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:07:32.484 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:32.486+0000 7f77a381a740 4 client.16761 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T12:07:32.484 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:32.486+0000 7f77a381a740 1 -- 172.21.15.29:0/2384934346 --> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559e9a01bf80 con 0x559e9a0c39f0 2024-03-16T12:07:32.484 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:32.487+0000 7f7799d9b640 1 --2- 172.21.15.29:0/2384934346 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x559e9a0c39f0 0x559e9a0c5de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:07:32.484 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:32.487+0000 7f7781ffb640 10 client.16761 ms_handle_connect on v2:172.21.15.152:6838/1548731833 2024-03-16T12:07:32.484 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:32.487+0000 7f7799d9b640 1 --2- 172.21.15.29:0/2384934346 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x559e9a0c39f0 0x559e9a0c5de0 crc :-1 s=READY pgs=186 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cecb74c1e3926ab0 server_cookie=212c6803504fd9 in_seq=0 out_seq=0 2024-03-16T12:07:32.485 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:32.489+0000 7f7781ffb640 1 -- 172.21.15.29:0/2384934346 <== mds.0 v2:172.21.15.152:6838/1548731833 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x559e9a01bf80 con 0x559e9a0c39f0 2024-03-16T12:07:32.485 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:32.489+0000 7f7781ffb640 10 client.16761 handle_command_reply: tid=0 2024-03-16T12:07:32.485 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:32.489+0000 7f77a381a740 1 client.16761 shutdown 2024-03-16T12:07:32.485 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:32.490+0000 7f77a381a740 1 -- 172.21.15.29:0/2384934346 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559e9a0aaa50 msgr2=0x559e9a0b0d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:07:32.485 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:32.490+0000 7f77a381a740 1 --2- 172.21.15.29:0/2384934346 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559e9a0aaa50 0x559e9a0b0d50 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f778c012740 tx=0x7f778c012c00 comp rx=0 tx=0).stop 2024-03-16T12:07:32.486 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:32.490+0000 7f77a381a740 1 -- 172.21.15.29:0/2384934346 shutdown_connections 2024-03-16T12:07:32.486 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:32.490+0000 7f77a381a740 1 --2- 172.21.15.29:0/2384934346 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559e9a0b3820 0x559e9a0b1290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:07:32.486 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:32.490+0000 7f77a381a740 1 --2- 172.21.15.29:0/2384934346 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559e9a0aaa50 0x559e9a0b0d50 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:07:32.486 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:32.490+0000 7f77a381a740 1 --2- 172.21.15.29:0/2384934346 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x559e9a0c39f0 0x559e9a0c5de0 crc :-1 s=READY pgs=186 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:07:32.486 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:32.490+0000 7f77a381a740 1 --2- 172.21.15.29:0/2384934346 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559e9a0aa130 0x559e9a0b0810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:07:32.486 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:32.490+0000 7f7781ffb640 0 client.16761 ms_handle_reset on v2:172.21.15.152:6838/1548731833 2024-03-16T12:07:32.486 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:32.490+0000 7f77a381a740 1 -- 172.21.15.29:0/2384934346 >> 172.21.15.29:0/2384934346 conn(0x559e9a0a4a50 msgr2=0x559e9a0a67a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:07:32.486 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:32.490+0000 7f77a381a740 1 -- 172.21.15.29:0/2384934346 shutdown_connections 2024-03-16T12:07:32.486 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:32.490+0000 7f77a381a740 1 -- 172.21.15.29:0/2384934346 wait complete. 2024-03-16T12:07:32.486 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:32.490+0000 7f77a381a740 20 client.16761 trim_cache size 0 max 16384 2024-03-16T12:07:32.501 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:32.505+0000 7f77a381a740 1 Processor -- start 2024-03-16T12:07:32.501 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:32.505+0000 7f77a381a740 1 -- start start 2024-03-16T12:07:32.501 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:32.505+0000 7f77a381a740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559e9a0aa130 0x559e9a0b2f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:07:32.501 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:32.505+0000 7f77a381a740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559e9a0aaa50 0x559e9a09fa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:07:32.501 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:32.505+0000 7f77a381a740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559e9a0b3820 0x559e9a0bf1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:07:32.501 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:32.505+0000 7f77a381a740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f779c1afd20 con 0x559e9a0aa130 2024-03-16T12:07:32.501 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:32.505+0000 7f77a381a740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f779c1ad5f0 con 0x559e9a0b3820 2024-03-16T12:07:32.501 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:32.505+0000 7f77a381a740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f779c1ab460 con 0x559e9a0aaa50 2024-03-16T12:07:32.501 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:32.505+0000 7f7799d9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559e9a0aa130 0x559e9a0b2f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:07:32.502 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:32.505+0000 7f7799d9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559e9a0aa130 0x559e9a0b2f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:37684/0 (socket says 172.21.15.29:37684) 2024-03-16T12:07:32.502 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:32.505+0000 7f7799d9b640 1 -- 172.21.15.29:0/788722723 learned_addr learned my addr 172.21.15.29:0/788722723 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T12:07:32.502 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:32.505+0000 7f779a59c640 1 --2- 172.21.15.29:0/788722723 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559e9a0aaa50 0x559e9a09fa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:07:32.502 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:32.506+0000 7f7799d9b640 1 --2- 172.21.15.29:0/788722723 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559e9a0b3820 0x559e9a0bf1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:07:32.502 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:32.506+0000 7f7799d9b640 1 -- 172.21.15.29:0/788722723 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559e9a0aaa50 msgr2=0x559e9a09fa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:07:32.502 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:32.506+0000 7f7799d9b640 1 --2- 172.21.15.29:0/788722723 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559e9a0aaa50 0x559e9a09fa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:07:32.502 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:32.506+0000 7f7799d9b640 1 -- 172.21.15.29:0/788722723 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559e9a0b3820 msgr2=0x559e9a0bf1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:07:32.502 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:32.506+0000 7f7799d9b640 1 --2- 172.21.15.29:0/788722723 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559e9a0b3820 0x559e9a0bf1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:07:32.502 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:32.506+0000 7f7799d9b640 1 -- 172.21.15.29:0/788722723 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559e9a0af7c0 con 0x559e9a0aa130 2024-03-16T12:07:32.502 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:32.506+0000 7f7799d9b640 1 --2- 172.21.15.29:0/788722723 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559e9a0aa130 0x559e9a0b2f40 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f77900058c0 tx=0x7f7790003d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e5645ff2ac3c431 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:07:32.502 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:32.506+0000 7f7783fff640 1 -- 172.21.15.29:0/788722723 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7790029040 con 0x559e9a0aa130 2024-03-16T12:07:32.502 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:32.506+0000 7f7783fff640 1 -- 172.21.15.29:0/788722723 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7790004040 con 0x559e9a0aa130 2024-03-16T12:07:32.502 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:32.506+0000 7f7783fff640 1 -- 172.21.15.29:0/788722723 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7790033030 con 0x559e9a0aa130 2024-03-16T12:07:32.502 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:32.507+0000 7f77a381a740 1 -- 172.21.15.29:0/788722723 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559e9a0aa130 msgr2=0x559e9a0b2f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:07:32.502 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:32.507+0000 7f77a381a740 1 --2- 172.21.15.29:0/788722723 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559e9a0aa130 0x559e9a0b2f40 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f77900058c0 tx=0x7f7790003d10 comp rx=0 tx=0).stop 2024-03-16T12:07:32.503 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:32.507+0000 7f77a381a740 1 -- 172.21.15.29:0/788722723 shutdown_connections 2024-03-16T12:07:32.503 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:32.507+0000 7f77a381a740 1 --2- 172.21.15.29:0/788722723 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559e9a0b3820 0x559e9a0bf1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:07:32.503 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:32.507+0000 7f77a381a740 1 --2- 172.21.15.29:0/788722723 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559e9a0aaa50 0x559e9a09fa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:07:32.503 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:32.507+0000 7f77a381a740 1 --2- 172.21.15.29:0/788722723 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559e9a0aa130 0x559e9a0b2f40 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:07:32.503 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:32.507+0000 7f77a381a740 1 -- 172.21.15.29:0/788722723 >> 172.21.15.29:0/788722723 conn(0x559e9a0a4a50 msgr2=0x559e9a0b1060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:07:32.503 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:32.507+0000 7f77a381a740 1 -- 172.21.15.29:0/788722723 shutdown_connections 2024-03-16T12:07:32.503 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:32.507+0000 7f77a381a740 1 -- 172.21.15.29:0/788722723 wait complete. 2024-03-16T12:07:32.503 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:32.507+0000 7f77a381a740 1 -- start start 2024-03-16T12:07:32.503 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:32.507+0000 7f77a381a740 1 Processor -- start 2024-03-16T12:07:32.503 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:32.507+0000 7f77a381a740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559e9a0aa130 0x559e9a121ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:07:32.503 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:32.507+0000 7f77a381a740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559e9a0aaa50 0x559e9a122410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:07:32.503 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:32.507+0000 7f77a381a740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559e9a0b3820 0x559e9a126e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:07:32.503 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:32.507+0000 7f77a381a740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f779c1abab0 con 0x559e9a0aaa50 2024-03-16T12:07:32.503 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:32.507+0000 7f77a381a740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f779c1acf20 con 0x559e9a0aa130 2024-03-16T12:07:32.503 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:32.507+0000 7f77a381a740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f779c1ac180 con 0x559e9a0b3820 2024-03-16T12:07:32.503 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:32.507+0000 7f77a381a740 10 client.0 _ll_register_callbacks cb 0x7f778c009930 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T12:07:32.504 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:32.507+0000 7f779a59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559e9a0aaa50 0x559e9a122410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:07:32.504 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:32.507+0000 7f779a59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559e9a0aaa50 0x559e9a122410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:37688/0 (socket says 172.21.15.29:37688) 2024-03-16T12:07:32.504 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:32.507+0000 7f779a59c640 1 -- 172.21.15.29:0/1409789598 learned_addr learned my addr 172.21.15.29:0/1409789598 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T12:07:32.504 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:32.508+0000 7f7799d9b640 1 --2- 172.21.15.29:0/1409789598 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559e9a0b3820 0x559e9a126e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:07:32.504 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:32.508+0000 7f7799d9b640 1 --2- 172.21.15.29:0/1409789598 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559e9a0aa130 0x559e9a121ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:07:32.504 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:32.508+0000 7f779a59c640 1 -- 172.21.15.29:0/1409789598 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559e9a0b3820 msgr2=0x559e9a126e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:07:32.504 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:32.508+0000 7f779a59c640 1 --2- 172.21.15.29:0/1409789598 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559e9a0b3820 0x559e9a126e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:07:32.504 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:32.508+0000 7f779a59c640 1 -- 172.21.15.29:0/1409789598 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559e9a0aa130 msgr2=0x559e9a121ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:07:32.504 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:32.508+0000 7f779a59c640 1 --2- 172.21.15.29:0/1409789598 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559e9a0aa130 0x559e9a121ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:07:32.504 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:32.508+0000 7f779a59c640 1 -- 172.21.15.29:0/1409789598 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559e9a1274f0 con 0x559e9a0aaa50 2024-03-16T12:07:32.504 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:32.508+0000 7f779a59c640 1 --2- 172.21.15.29:0/1409789598 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559e9a0aaa50 0x559e9a122410 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x559e9a0b0070 tx=0x7f778c00fb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a9547fc92fd6168 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:07:32.504 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:32.508+0000 7f7781ffb640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-16T12:07:32.504 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:32.508+0000 7f7781ffb640 1 -- 172.21.15.29:0/1409789598 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f778c029040 con 0x559e9a0aaa50 2024-03-16T12:07:32.504 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:32.508+0000 7f7781ffb640 1 -- 172.21.15.29:0/1409789598 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(114..114 src has 1..114) ==== 8556+0+0 (secure 0 0 0) 0x7f778c00fca0 con 0x559e9a0aaa50 2024-03-16T12:07:32.504 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:32.508+0000 7f77a381a740 1 -- 172.21.15.29:0/1409789598 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7790004040 con 0x559e9a0aaa50 2024-03-16T12:07:32.504 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:32.508+0000 7f7781ffb640 1 -- 172.21.15.29:0/1409789598 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f778c005040 con 0x559e9a0aaa50 2024-03-16T12:07:32.504 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:32.508+0000 7f77a381a740 10 client.16767 fetch_fsmap learned FSMap version 32 2024-03-16T12:07:32.505 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:32.508+0000 7f77a381a740 1 -- 172.21.15.29:0/1409789598 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x559e9a12e890 con 0x559e9a0aaa50 2024-03-16T12:07:32.505 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:32.508+0000 7f7781ffb640 1 -- 172.21.15.29:0/1409789598 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7790004040 con 0x559e9a0aaa50 2024-03-16T12:07:32.505 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:32.509+0000 7f7781ffb640 1 -- 172.21.15.29:0/1409789598 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 32) ==== 8338+0+0 (secure 0 0 0) 0x7f778c010ab0 con 0x559e9a0aaa50 2024-03-16T12:07:32.505 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:32.509+0000 7f77a381a740 10 client.16767 fetch_fsmap finished waiting for FSMap version 32 2024-03-16T12:07:32.505 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:32.509+0000 7f77a381a740 10 client.16767 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T12:07:32.505 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:32.509+0000 7f77a381a740 20 client.16767 populate_metadata read hostname 'smithi029' 2024-03-16T12:07:32.505 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:32.509+0000 7f77a381a740 1 --2- 172.21.15.29:0/1409789598 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x559e9a12f040 0x559e9a131460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:07:32.505 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:32.509+0000 7f77a381a740 4 client.16767 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-16T12:07:32.505 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:32.509+0000 7f77a381a740 1 -- 172.21.15.29:0/1409789598 --> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] -- command(tid 0: {"prefix": "flush journal"}) -- 0x559e9a0162d0 con 0x559e9a12f040 2024-03-16T12:07:32.505 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:32.509+0000 7f7799d9b640 1 --2- 172.21.15.29:0/1409789598 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x559e9a12f040 0x559e9a131460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:07:32.505 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:32.509+0000 7f7799d9b640 1 --2- 172.21.15.29:0/1409789598 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x559e9a12f040 0x559e9a131460 crc :-1 s=READY pgs=187 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=22ba08db3c4f4739 server_cookie=14c16b7c79ea1e6e in_seq=0 out_seq=0 2024-03-16T12:07:32.505 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:32.509+0000 7f7781ffb640 10 client.16767 ms_handle_connect on v2:172.21.15.152:6838/1548731833 2024-03-16T12:07:32.619 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:07:32 smithi152 ceph-mon[36291]: pgmap v527: 97 pgs: 97 active+clean; 377 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 5.9 MiB/s wr, 316 op/s 2024-03-16T12:07:32.786 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:07:32 smithi154 ceph-mon[36320]: pgmap v527: 97 pgs: 97 active+clean; 377 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 5.9 MiB/s wr, 316 op/s 2024-03-16T12:07:32.829 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:07:32 smithi029 ceph-mon[31975]: pgmap v527: 97 pgs: 97 active+clean; 377 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 5.9 MiB/s wr, 316 op/s 2024-03-16T12:07:33.188 INFO:tasks.workunit.client.0.smithi029.stdout: 10 51343 12.89 MB/sec execute 77 sec latency 433.677 ms 2024-03-16T12:07:33.341 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:33.348+0000 7f7781ffb640 1 -- 172.21.15.29:0/1409789598 <== mds.0 v2:172.21.15.152:6838/1548731833 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x559e9a0162d0 con 0x559e9a12f040 2024-03-16T12:07:33.341 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:33.348+0000 7f7781ffb640 10 client.16767 handle_command_reply: tid=0 2024-03-16T12:07:33.341 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:33.348+0000 7f77a381a740 1 client.16767 shutdown 2024-03-16T12:07:33.341 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:33.348+0000 7f77a381a740 1 -- 172.21.15.29:0/1409789598 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559e9a0aaa50 msgr2=0x559e9a122410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:07:33.342 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:33.348+0000 7f77a381a740 1 --2- 172.21.15.29:0/1409789598 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559e9a0aaa50 0x559e9a122410 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x559e9a0b0070 tx=0x7f778c00fb30 comp rx=0 tx=0).stop 2024-03-16T12:07:33.342 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:33.349+0000 7f77a381a740 1 -- 172.21.15.29:0/1409789598 shutdown_connections 2024-03-16T12:07:33.342 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:33.349+0000 7f77a381a740 1 --2- 172.21.15.29:0/1409789598 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559e9a0b3820 0x559e9a126e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:07:33.342 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:33.349+0000 7f77a381a740 1 --2- 172.21.15.29:0/1409789598 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559e9a0aaa50 0x559e9a122410 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:07:33.342 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:33.349+0000 7f77a381a740 1 --2- 172.21.15.29:0/1409789598 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x559e9a12f040 0x559e9a131460 crc :-1 s=READY pgs=187 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:07:33.342 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:33.349+0000 7f7781ffb640 0 client.16767 ms_handle_reset on v2:172.21.15.152:6838/1548731833 2024-03-16T12:07:33.342 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:33.349+0000 7f77a381a740 1 --2- 172.21.15.29:0/1409789598 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559e9a0aa130 0x559e9a121ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:07:33.342 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:33.349+0000 7f77a381a740 1 -- 172.21.15.29:0/1409789598 >> 172.21.15.29:0/1409789598 conn(0x559e9a0a4a50 msgr2=0x559e9a0b0740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:07:33.342 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:33.349+0000 7f77a381a740 1 -- 172.21.15.29:0/1409789598 shutdown_connections 2024-03-16T12:07:33.342 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:33.349+0000 7f77a381a740 1 -- 172.21.15.29:0/1409789598 wait complete. 2024-03-16T12:07:33.342 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:33.350+0000 7f77a381a740 20 client.16767 trim_cache size 0 max 16384 2024-03-16T12:07:33.343 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:{ 2024-03-16T12:07:33.343 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "message": "", 2024-03-16T12:07:33.343 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "return_code": 0 2024-03-16T12:07:33.343 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:} 2024-03-16T12:07:33.343 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:33.350+0000 7f77a0b53640 1 -- 172.21.15.29:0/3333287501 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f77840a2a10 msgr2=0x7f77840a4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:07:33.344 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:33.350+0000 7f77a0b53640 1 --2- 172.21.15.29:0/3333287501 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f77840a2a10 0x7f77840a4ed0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f7794009430 tx=0x7f77940885a0 comp rx=0 tx=0).stop 2024-03-16T12:07:33.344 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:33.350+0000 7f77a0b53640 1 -- 172.21.15.29:0/3333287501 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f779c1078b0 msgr2=0x7f779c19c800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:07:33.344 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:33.350+0000 7f77a0b53640 1 --2- 172.21.15.29:0/3333287501 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f779c1078b0 0x7f779c19c800 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f7794004af0 tx=0x7f7794032c90 comp rx=0 tx=0).stop 2024-03-16T12:07:33.344 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:33.351+0000 7f77a0b53640 1 -- 172.21.15.29:0/3333287501 shutdown_connections 2024-03-16T12:07:33.344 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:33.351+0000 7f77a0b53640 1 --2- 172.21.15.29:0/3333287501 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f77840a2a10 0x7f77840a4ed0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:07:33.344 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:33.351+0000 7f77a0b53640 1 --2- 172.21.15.29:0/3333287501 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f779c1078b0 0x7f779c19c800 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:07:33.344 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:33.351+0000 7f77a0b53640 1 --2- 172.21.15.29:0/3333287501 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f779c069800 0x7f779c198480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:07:33.344 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:33.351+0000 7f77a0b53640 1 --2- 172.21.15.29:0/3333287501 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f779c068e60 0x7f779c197f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:07:33.344 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:33.351+0000 7f77a0b53640 1 -- 172.21.15.29:0/3333287501 >> 172.21.15.29:0/3333287501 conn(0x7f779c077530 msgr2=0x7f779c0fe940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:07:33.344 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:33.351+0000 7f77a0b53640 1 -- 172.21.15.29:0/3333287501 shutdown_connections 2024-03-16T12:07:33.344 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:33.351+0000 7f77a0b53640 1 -- 172.21.15.29:0/3333287501 wait complete. 2024-03-16T12:07:34.188 INFO:tasks.workunit.client.0.smithi029.stdout: 10 51538 12.80 MB/sec execute 78 sec latency 461.871 ms 2024-03-16T12:07:34.619 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:07:34 smithi152 ceph-mon[36291]: pgmap v528: 97 pgs: 97 active+clean; 380 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 4.1 MiB/s wr, 245 op/s 2024-03-16T12:07:34.786 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:07:34 smithi154 ceph-mon[36320]: pgmap v528: 97 pgs: 97 active+clean; 380 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 4.1 MiB/s wr, 245 op/s 2024-03-16T12:07:34.829 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:07:34 smithi029 ceph-mon[31975]: pgmap v528: 97 pgs: 97 active+clean; 380 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 4.1 MiB/s wr, 245 op/s 2024-03-16T12:07:35.188 INFO:tasks.workunit.client.0.smithi029.stdout: 10 51919 12.86 MB/sec execute 79 sec latency 210.863 ms 2024-03-16T12:07:36.188 INFO:tasks.workunit.client.0.smithi029.stdout: 10 52358 13.00 MB/sec execute 80 sec latency 198.639 ms 2024-03-16T12:07:36.619 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:07:36 smithi152 ceph-mon[36291]: pgmap v529: 97 pgs: 97 active+clean; 380 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 3.7 MiB/s wr, 223 op/s 2024-03-16T12:07:36.786 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:07:36 smithi154 ceph-mon[36320]: pgmap v529: 97 pgs: 97 active+clean; 380 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 3.7 MiB/s wr, 223 op/s 2024-03-16T12:07:36.829 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:07:36 smithi029 ceph-mon[31975]: pgmap v529: 97 pgs: 97 active+clean; 380 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 3.7 MiB/s wr, 223 op/s 2024-03-16T12:07:37.188 INFO:tasks.workunit.client.0.smithi029.stdout: 10 52756 13.04 MB/sec execute 81 sec latency 194.767 ms 2024-03-16T12:07:38.188 INFO:tasks.workunit.client.0.smithi029.stdout: 10 53044 13.18 MB/sec execute 82 sec latency 276.337 ms 2024-03-16T12:07:38.618 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:07:38 smithi152 ceph-mon[36291]: pgmap v530: 97 pgs: 97 active+clean; 364 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 MiB/s rd, 5.2 MiB/s wr, 497 op/s 2024-03-16T12:07:38.786 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:07:38 smithi154 ceph-mon[36320]: pgmap v530: 97 pgs: 97 active+clean; 364 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 MiB/s rd, 5.2 MiB/s wr, 497 op/s 2024-03-16T12:07:38.829 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:07:38 smithi029 ceph-mon[31975]: pgmap v530: 97 pgs: 97 active+clean; 364 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 MiB/s rd, 5.2 MiB/s wr, 497 op/s 2024-03-16T12:07:39.188 INFO:tasks.workunit.client.0.smithi029.stdout: 10 53408 13.27 MB/sec execute 83 sec latency 237.513 ms 2024-03-16T12:07:40.188 INFO:tasks.workunit.client.0.smithi029.stdout: 10 53678 13.37 MB/sec execute 84 sec latency 211.463 ms 2024-03-16T12:07:40.786 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:07:40 smithi154 ceph-mon[36320]: pgmap v531: 97 pgs: 97 active+clean; 364 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 3.5 MiB/s wr, 367 op/s 2024-03-16T12:07:40.829 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:07:40 smithi029 ceph-mon[31975]: pgmap v531: 97 pgs: 97 active+clean; 364 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 3.5 MiB/s wr, 367 op/s 2024-03-16T12:07:40.869 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:07:40 smithi152 ceph-mon[36291]: pgmap v531: 97 pgs: 97 active+clean; 364 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 3.5 MiB/s wr, 367 op/s 2024-03-16T12:07:41.188 INFO:tasks.workunit.client.0.smithi029.stdout: 10 53892 13.37 MB/sec execute 85 sec latency 249.293 ms 2024-03-16T12:07:42.188 INFO:tasks.workunit.client.0.smithi029.stdout: 10 54098 13.36 MB/sec execute 86 sec latency 258.136 ms 2024-03-16T12:07:42.786 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:07:42 smithi154 ceph-mon[36320]: pgmap v532: 97 pgs: 97 active+clean; 369 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 4.0 MiB/s wr, 411 op/s 2024-03-16T12:07:42.829 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:07:42 smithi029 ceph-mon[31975]: pgmap v532: 97 pgs: 97 active+clean; 369 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 4.0 MiB/s wr, 411 op/s 2024-03-16T12:07:42.868 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:07:42 smithi152 ceph-mon[36291]: pgmap v532: 97 pgs: 97 active+clean; 369 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 4.0 MiB/s wr, 411 op/s 2024-03-16T12:07:43.188 INFO:tasks.workunit.client.0.smithi029.stdout: 10 54235 13.25 MB/sec execute 87 sec latency 407.947 ms 2024-03-16T12:07:44.188 INFO:tasks.workunit.client.0.smithi029.stdout: 10 54399 13.18 MB/sec execute 88 sec latency 315.497 ms 2024-03-16T12:07:44.669 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:07:44 smithi152 ceph-mon[36291]: pgmap v533: 97 pgs: 97 active+clean; 373 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 MiB/s rd, 3.9 MiB/s wr, 414 op/s 2024-03-16T12:07:44.786 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:07:44 smithi154 ceph-mon[36320]: pgmap v533: 97 pgs: 97 active+clean; 373 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 MiB/s rd, 3.9 MiB/s wr, 414 op/s 2024-03-16T12:07:44.829 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:07:44 smithi029 ceph-mon[31975]: pgmap v533: 97 pgs: 97 active+clean; 373 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 MiB/s rd, 3.9 MiB/s wr, 414 op/s 2024-03-16T12:07:45.188 INFO:tasks.workunit.client.0.smithi029.stdout: 10 54656 13.11 MB/sec execute 89 sec latency 303.499 ms 2024-03-16T12:07:46.188 INFO:tasks.workunit.client.0.smithi029.stdout: 10 55003 13.11 MB/sec execute 90 sec latency 203.010 ms 2024-03-16T12:07:46.438 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:46.446+0000 7fde41d89640 1 -- 172.21.15.29:0/4206533963 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fde3c105680 msgr2=0x7fde3c105a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:07:46.438 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:46.446+0000 7fde41d89640 1 --2- 172.21.15.29:0/4206533963 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fde3c105680 0x7fde3c105a60 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7fde2c009900 tx=0x7fde2c02bc30 comp rx=0 tx=0).stop 2024-03-16T12:07:46.439 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:46.446+0000 7fde41d89640 1 -- 172.21.15.29:0/4206533963 shutdown_connections 2024-03-16T12:07:46.439 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:46.446+0000 7fde41d89640 1 --2- 172.21.15.29:0/4206533963 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fde3c1012f0 0x7fde3c10d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:07:46.439 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:46.446+0000 7fde41d89640 1 --2- 172.21.15.29:0/4206533963 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fde3c100930 0x7fde3c100db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:07:46.439 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:46.446+0000 7fde41d89640 1 --2- 172.21.15.29:0/4206533963 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fde3c105680 0x7fde3c105a60 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:07:46.439 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:46.446+0000 7fde41d89640 1 -- 172.21.15.29:0/4206533963 >> 172.21.15.29:0/4206533963 conn(0x7fde3c077530 msgr2=0x7fde3c077940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:07:46.439 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:46.446+0000 7fde41d89640 1 -- 172.21.15.29:0/4206533963 shutdown_connections 2024-03-16T12:07:46.439 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:46.446+0000 7fde41d89640 1 -- 172.21.15.29:0/4206533963 wait complete. 2024-03-16T12:07:46.439 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:46.446+0000 7fde41d89640 1 Processor -- start 2024-03-16T12:07:46.439 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:46.446+0000 7fde41d89640 1 -- start start 2024-03-16T12:07:46.439 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:46.447+0000 7fde41d89640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fde3c100930 0x7fde3c196d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:07:46.439 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:46.447+0000 7fde41d89640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fde3c1012f0 0x7fde3c197240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:07:46.439 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:46.447+0000 7fde41d89640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fde3c105680 0x7fde3c197780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:07:46.439 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:46.447+0000 7fde41d89640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fde3c110bf0 con 0x7fde3c100930 2024-03-16T12:07:46.439 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:46.447+0000 7fde41d89640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fde3c110a70 con 0x7fde3c105680 2024-03-16T12:07:46.440 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:46.447+0000 7fde41d89640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fde3c110d70 con 0x7fde3c1012f0 2024-03-16T12:07:46.440 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:46.447+0000 7fde3b7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fde3c100930 0x7fde3c196d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:07:46.440 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:46.447+0000 7fde3b7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fde3c100930 0x7fde3c196d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33206/0 (socket says 172.21.15.29:33206) 2024-03-16T12:07:46.440 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:46.447+0000 7fde3b7fe640 1 -- 172.21.15.29:0/2761312129 learned_addr learned my addr 172.21.15.29:0/2761312129 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T12:07:46.440 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:46.447+0000 7fde3bfff640 1 --2- 172.21.15.29:0/2761312129 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fde3c105680 0x7fde3c197780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:07:46.440 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:46.447+0000 7fde32ffd640 1 --2- 172.21.15.29:0/2761312129 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fde3c1012f0 0x7fde3c197240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:07:46.440 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:46.447+0000 7fde3b7fe640 1 -- 172.21.15.29:0/2761312129 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fde3c1012f0 msgr2=0x7fde3c197240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:07:46.440 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:46.447+0000 7fde3b7fe640 1 --2- 172.21.15.29:0/2761312129 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fde3c1012f0 0x7fde3c197240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:07:46.440 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:46.447+0000 7fde3b7fe640 1 -- 172.21.15.29:0/2761312129 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fde3c105680 msgr2=0x7fde3c197780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:07:46.440 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:46.447+0000 7fde3b7fe640 1 --2- 172.21.15.29:0/2761312129 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fde3c105680 0x7fde3c197780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:07:46.440 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:46.447+0000 7fde3b7fe640 1 -- 172.21.15.29:0/2761312129 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde3c1a6570 con 0x7fde3c100930 2024-03-16T12:07:46.440 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:46.447+0000 7fde3b7fe640 1 --2- 172.21.15.29:0/2761312129 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fde3c100930 0x7fde3c196d00 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7fde2c02efd0 tx=0x7fde2c0042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:07:46.440 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:46.447+0000 7fde397fa640 1 -- 172.21.15.29:0/2761312129 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde2c041070 con 0x7fde3c100930 2024-03-16T12:07:46.440 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:46.448+0000 7fde397fa640 1 -- 172.21.15.29:0/2761312129 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fde2c02ccc0 con 0x7fde3c100930 2024-03-16T12:07:46.440 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:46.448+0000 7fde41d89640 1 -- 172.21.15.29:0/2761312129 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fde3c1a6800 con 0x7fde3c100930 2024-03-16T12:07:46.441 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:46.448+0000 7fde397fa640 1 -- 172.21.15.29:0/2761312129 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde2c003d40 con 0x7fde3c100930 2024-03-16T12:07:46.441 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:46.448+0000 7fde41d89640 1 -- 172.21.15.29:0/2761312129 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fde3c1a6cf0 con 0x7fde3c100930 2024-03-16T12:07:46.446 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:46.450+0000 7fde397fa640 1 -- 172.21.15.29:0/2761312129 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7fde2c044070 con 0x7fde3c100930 2024-03-16T12:07:46.446 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:46.450+0000 7fde397fa640 1 --2- 172.21.15.29:0/2761312129 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7fde280a2a10 0x7fde280a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:07:46.447 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:46.450+0000 7fde4497a740 1 Processor -- start 2024-03-16T12:07:46.447 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:46.450+0000 7fde4497a740 1 -- start start 2024-03-16T12:07:46.447 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:46.450+0000 7fde32ffd640 1 --2- 172.21.15.29:0/2761312129 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7fde280a2a10 0x7fde280a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:07:46.447 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:46.450+0000 7fde397fa640 1 -- 172.21.15.29:0/2761312129 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(114..114 src has 1..114) ==== 8556+0+0 (secure 0 0 0) 0x7fde2c0f07f0 con 0x7fde3c100930 2024-03-16T12:07:46.447 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:46.450+0000 7fde4497a740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563032e72130 0x563032e72510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:07:46.447 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:46.450+0000 7fde4497a740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563032e72a50 0x563032e7b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:07:46.447 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:46.451+0000 7fde4497a740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563032e7b820 0x563032e7dc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:07:46.447 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:46.451+0000 7fde4497a740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x563032dd6a40 con 0x563032e72a50 2024-03-16T12:07:46.447 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:46.451+0000 7fde4497a740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x563032d7f920 con 0x563032e72130 2024-03-16T12:07:46.447 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:46.451+0000 7fde4497a740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x563032dd2ae0 con 0x563032e7b820 2024-03-16T12:07:46.447 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:46.451+0000 7fde32ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563032e72a50 0x563032e7b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:07:46.447 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:46.451+0000 7fde3b7fe640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563032e72130 0x563032e72510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:07:46.447 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:46.451+0000 7fde3b7fe640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563032e72130 0x563032e72510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.29:55392/0 (socket says 172.21.15.29:55392) 2024-03-16T12:07:46.447 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:46.451+0000 7fde32ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563032e72a50 0x563032e7b2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33218/0 (socket says 172.21.15.29:33218) 2024-03-16T12:07:46.448 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:46.451+0000 7fde3b7fe640 1 -- 172.21.15.29:0/1343976736 learned_addr learned my addr 172.21.15.29:0/1343976736 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T12:07:46.448 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:46.451+0000 7fde3bfff640 1 --2- 172.21.15.29:0/1343976736 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563032e7b820 0x563032e7dc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:07:46.448 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:46.451+0000 7fde3b7fe640 1 -- 172.21.15.29:0/1343976736 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563032e7b820 msgr2=0x563032e7dc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:07:46.448 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:46.451+0000 7fde3b7fe640 1 --2- 172.21.15.29:0/1343976736 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563032e7b820 0x563032e7dc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:07:46.448 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:46.451+0000 7fde3b7fe640 1 -- 172.21.15.29:0/1343976736 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563032e72a50 msgr2=0x563032e7b2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:07:46.448 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:46.451+0000 7fde3b7fe640 1 --2- 172.21.15.29:0/1343976736 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563032e72a50 0x563032e7b2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:07:46.448 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:46.451+0000 7fde3b7fe640 1 -- 172.21.15.29:0/1343976736 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563032d97460 con 0x563032e72130 2024-03-16T12:07:46.448 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:46.451+0000 7fde32ffd640 1 --2- 172.21.15.29:0/2761312129 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7fde280a2a10 0x7fde280a4ed0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fde3c198e60 tx=0x7fde24011920 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:07:46.448 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:46.451+0000 7fde3b7fe640 1 --2- 172.21.15.29:0/1343976736 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563032e72130 0x563032e72510 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fde2c0bccf0 tx=0x7fde2c0bcdc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ab2789ee3505658c server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:07:46.448 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:46.451+0000 7fde31ffb640 1 -- 172.21.15.29:0/1343976736 <== mon.2 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde2c04b040 con 0x563032e72130 2024-03-16T12:07:46.448 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:46.451+0000 7fde31ffb640 1 -- 172.21.15.29:0/1343976736 <== mon.2 v2:172.21.15.152:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fde2c0bcff0 con 0x563032e72130 2024-03-16T12:07:46.448 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:46.451+0000 7fde31ffb640 1 -- 172.21.15.29:0/1343976736 <== mon.2 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde2c0b9660 con 0x563032e72130 2024-03-16T12:07:46.448 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:46.452+0000 7fde4497a740 1 -- 172.21.15.29:0/1343976736 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563032e72130 msgr2=0x563032e72510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:07:46.448 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:46.452+0000 7fde4497a740 1 --2- 172.21.15.29:0/1343976736 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563032e72130 0x563032e72510 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fde2c0bccf0 tx=0x7fde2c0bcdc0 comp rx=0 tx=0).stop 2024-03-16T12:07:46.448 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:46.452+0000 7fde4497a740 1 -- 172.21.15.29:0/1343976736 shutdown_connections 2024-03-16T12:07:46.448 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:46.452+0000 7fde4497a740 1 --2- 172.21.15.29:0/1343976736 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563032e7b820 0x563032e7dc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:07:46.449 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:46.452+0000 7fde4497a740 1 --2- 172.21.15.29:0/1343976736 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563032e72a50 0x563032e7b2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:07:46.449 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:46.452+0000 7fde4497a740 1 --2- 172.21.15.29:0/1343976736 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563032e72130 0x563032e72510 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:07:46.449 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:46.452+0000 7fde4497a740 1 -- 172.21.15.29:0/1343976736 >> 172.21.15.29:0/1343976736 conn(0x563032e6ca50 msgr2=0x563032e6ce30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:07:46.449 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:46.452+0000 7fde4497a740 1 -- 172.21.15.29:0/1343976736 shutdown_connections 2024-03-16T12:07:46.449 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:46.452+0000 7fde4497a740 1 -- 172.21.15.29:0/1343976736 wait complete. 2024-03-16T12:07:46.449 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:46.452+0000 7fde4497a740 1 -- start start 2024-03-16T12:07:46.449 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:46.453+0000 7fde4497a740 1 Processor -- start 2024-03-16T12:07:46.449 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:46.453+0000 7fde4497a740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563032e72130 0x563032e788f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:07:46.449 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:46.453+0000 7fde4497a740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563032e72a50 0x563032e78e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:07:46.449 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:46.453+0000 7fde4497a740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563032e7b820 0x563032e79370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:07:46.449 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:46.453+0000 7fde4497a740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x563032e7e210 con 0x563032e72130 2024-03-16T12:07:46.449 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:46.453+0000 7fde4497a740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x563032d92d70 con 0x563032e7b820 2024-03-16T12:07:46.449 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:46.453+0000 7fde4497a740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x563032e7e390 con 0x563032e72a50 2024-03-16T12:07:46.449 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:46.453+0000 7fde4497a740 10 client.0 _ll_register_callbacks cb 0x563032aeaa00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T12:07:46.450 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:46.453+0000 7fde3b7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563032e72130 0x563032e788f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:07:46.450 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:46.453+0000 7fde3b7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563032e72130 0x563032e788f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33230/0 (socket says 172.21.15.29:33230) 2024-03-16T12:07:46.450 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:46.453+0000 7fde3b7fe640 1 -- 172.21.15.29:0/679142934 learned_addr learned my addr 172.21.15.29:0/679142934 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T12:07:46.450 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:46.453+0000 7fde32ffd640 1 --2- 172.21.15.29:0/679142934 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563032e72a50 0x563032e78e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:07:46.450 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:46.453+0000 7fde3bfff640 1 --2- 172.21.15.29:0/679142934 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563032e7b820 0x563032e79370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:07:46.450 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:46.453+0000 7fde3b7fe640 1 -- 172.21.15.29:0/679142934 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563032e72a50 msgr2=0x563032e78e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:07:46.450 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:46.453+0000 7fde3b7fe640 1 --2- 172.21.15.29:0/679142934 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563032e72a50 0x563032e78e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:07:46.450 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:46.453+0000 7fde3b7fe640 1 -- 172.21.15.29:0/679142934 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563032e7b820 msgr2=0x563032e79370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:07:46.450 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:46.453+0000 7fde3b7fe640 1 --2- 172.21.15.29:0/679142934 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563032e7b820 0x563032e79370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:07:46.450 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:46.453+0000 7fde3b7fe640 1 -- 172.21.15.29:0/679142934 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563032aeb210 con 0x563032e72130 2024-03-16T12:07:46.450 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:46.453+0000 7fde3b7fe640 1 --2- 172.21.15.29:0/679142934 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563032e72130 0x563032e788f0 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7fde2c0ee0e0 tx=0x7fde2c009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26940773e69b2e67 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:07:46.450 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:46.453+0000 7fde1affd640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-16T12:07:46.450 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:46.454+0000 7fde1affd640 1 -- 172.21.15.29:0/679142934 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde2c0d9020 con 0x563032e72130 2024-03-16T12:07:46.450 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:46.454+0000 7fde1affd640 1 -- 172.21.15.29:0/679142934 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(114..114 src has 1..114) ==== 8556+0+0 (secure 0 0 0) 0x7fde2c0c5460 con 0x563032e72130 2024-03-16T12:07:46.450 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:46.454+0000 7fde4497a740 1 -- 172.21.15.29:0/679142934 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fde2c0bcff0 con 0x563032e72130 2024-03-16T12:07:46.451 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:46.454+0000 7fde1affd640 1 -- 172.21.15.29:0/679142934 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fde2c04b070 con 0x563032e72130 2024-03-16T12:07:46.451 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:46.454+0000 7fde4497a740 10 client.16779 fetch_fsmap learned FSMap version 32 2024-03-16T12:07:46.451 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:46.454+0000 7fde4497a740 1 -- 172.21.15.29:0/679142934 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x563032bb7620 con 0x563032e72130 2024-03-16T12:07:46.451 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:46.454+0000 7fde1affd640 1 -- 172.21.15.29:0/679142934 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fde2c0bcff0 con 0x563032e72130 2024-03-16T12:07:46.451 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:46.454+0000 7fde1affd640 1 -- 172.21.15.29:0/679142934 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 32) ==== 8338+0+0 (secure 0 0 0) 0x7fde2c0b7b60 con 0x563032e72130 2024-03-16T12:07:46.451 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:46.454+0000 7fde4497a740 10 client.16779 fetch_fsmap finished waiting for FSMap version 32 2024-03-16T12:07:46.451 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:46.454+0000 7fde4497a740 10 client.16779 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T12:07:46.451 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:46.454+0000 7fde4497a740 20 client.16779 populate_metadata read hostname 'smithi029' 2024-03-16T12:07:46.451 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:46.454+0000 7fde4497a740 1 --2- 172.21.15.29:0/679142934 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x563032e8b5b0 0x563032e8d9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:07:46.451 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:46.454+0000 7fde4497a740 4 client.16779 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T12:07:46.451 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:46.454+0000 7fde4497a740 1 -- 172.21.15.29:0/679142934 --> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563032de3f80 con 0x563032e8b5b0 2024-03-16T12:07:46.451 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:46.454+0000 7fde32ffd640 1 --2- 172.21.15.29:0/679142934 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x563032e8b5b0 0x563032e8d9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:07:46.451 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:46.455+0000 7fde32ffd640 1 --2- 172.21.15.29:0/679142934 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x563032e8b5b0 0x563032e8d9a0 crc :-1 s=READY pgs=188 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=694bd15a3c0d4644 server_cookie=1235f220de7f34df in_seq=0 out_seq=0 2024-03-16T12:07:46.451 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:46.455+0000 7fde1affd640 10 client.16779 ms_handle_connect on v2:172.21.15.152:6838/1548731833 2024-03-16T12:07:46.453 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:46.460+0000 7fde1affd640 1 -- 172.21.15.29:0/679142934 <== mds.0 v2:172.21.15.152:6838/1548731833 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x563032de3f80 con 0x563032e8b5b0 2024-03-16T12:07:46.453 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:46.460+0000 7fde1affd640 10 client.16779 handle_command_reply: tid=0 2024-03-16T12:07:46.453 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:46.461+0000 7fde4497a740 1 client.16779 shutdown 2024-03-16T12:07:46.453 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:46.461+0000 7fde4497a740 1 -- 172.21.15.29:0/679142934 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563032e72130 msgr2=0x563032e788f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:07:46.453 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:46.461+0000 7fde4497a740 1 --2- 172.21.15.29:0/679142934 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563032e72130 0x563032e788f0 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7fde2c0ee0e0 tx=0x7fde2c009260 comp rx=0 tx=0).stop 2024-03-16T12:07:46.453 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:46.461+0000 7fde4497a740 1 -- 172.21.15.29:0/679142934 shutdown_connections 2024-03-16T12:07:46.453 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:46.461+0000 7fde4497a740 1 --2- 172.21.15.29:0/679142934 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x563032e8b5b0 0x563032e8d9a0 crc :-1 s=READY pgs=188 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:07:46.453 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:46.461+0000 7fde4497a740 1 --2- 172.21.15.29:0/679142934 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563032e7b820 0x563032e79370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:07:46.453 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:46.461+0000 7fde4497a740 1 --2- 172.21.15.29:0/679142934 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563032e72a50 0x563032e78e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:07:46.453 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:46.461+0000 7fde1affd640 0 client.16779 ms_handle_reset on v2:172.21.15.152:6838/1548731833 2024-03-16T12:07:46.453 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:46.461+0000 7fde4497a740 1 --2- 172.21.15.29:0/679142934 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563032e72130 0x563032e788f0 unknown :-1 s=CLOSED pgs=777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:07:46.453 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:46.461+0000 7fde4497a740 1 -- 172.21.15.29:0/679142934 >> 172.21.15.29:0/679142934 conn(0x563032e6ca50 msgr2=0x563032e6e7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:07:46.453 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:46.461+0000 7fde4497a740 1 -- 172.21.15.29:0/679142934 shutdown_connections 2024-03-16T12:07:46.454 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:46.461+0000 7fde4497a740 1 -- 172.21.15.29:0/679142934 wait complete. 2024-03-16T12:07:46.454 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:46.461+0000 7fde4497a740 20 client.16779 trim_cache size 0 max 16384 2024-03-16T12:07:46.472 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:46.476+0000 7fde4497a740 1 Processor -- start 2024-03-16T12:07:46.472 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:46.476+0000 7fde4497a740 1 -- start start 2024-03-16T12:07:46.472 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:46.476+0000 7fde4497a740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563032e72130 0x563032e86ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:07:46.473 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:46.476+0000 7fde4497a740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563032e72a50 0x563032e67a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:07:46.473 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:46.477+0000 7fde4497a740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563032e7b820 0x563032e93180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:07:46.473 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:46.477+0000 7fde4497a740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fde3c1b1f90 con 0x563032e72a50 2024-03-16T12:07:46.473 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:46.477+0000 7fde4497a740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fde3c1af860 con 0x563032e72130 2024-03-16T12:07:46.473 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:46.477+0000 7fde4497a740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fde3c1ad6d0 con 0x563032e7b820 2024-03-16T12:07:46.473 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:46.477+0000 7fde32ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563032e72a50 0x563032e67a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:07:46.473 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:46.477+0000 7fde32ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563032e72a50 0x563032e67a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33240/0 (socket says 172.21.15.29:33240) 2024-03-16T12:07:46.473 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:46.477+0000 7fde32ffd640 1 -- 172.21.15.29:0/2469373504 learned_addr learned my addr 172.21.15.29:0/2469373504 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T12:07:46.473 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:46.477+0000 7fde3b7fe640 1 --2- 172.21.15.29:0/2469373504 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563032e72130 0x563032e86ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:07:46.473 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:46.477+0000 7fde3bfff640 1 --2- 172.21.15.29:0/2469373504 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563032e7b820 0x563032e93180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:07:46.474 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:46.477+0000 7fde32ffd640 1 -- 172.21.15.29:0/2469373504 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563032e7b820 msgr2=0x563032e93180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:07:46.474 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:46.477+0000 7fde32ffd640 1 --2- 172.21.15.29:0/2469373504 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563032e7b820 0x563032e93180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:07:46.474 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:46.477+0000 7fde32ffd640 1 -- 172.21.15.29:0/2469373504 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563032e72130 msgr2=0x563032e86ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:07:46.474 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:46.477+0000 7fde32ffd640 1 --2- 172.21.15.29:0/2469373504 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563032e72130 0x563032e86ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:07:46.474 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:46.477+0000 7fde32ffd640 1 -- 172.21.15.29:0/2469373504 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563032e778a0 con 0x563032e72a50 2024-03-16T12:07:46.474 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:46.477+0000 7fde32ffd640 1 --2- 172.21.15.29:0/2469373504 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563032e72a50 0x563032e67a40 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7fde2401d000 tx=0x7fde24020000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30bd8e9696c2f04c server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:07:46.474 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:46.477+0000 7fde317fa640 1 -- 172.21.15.29:0/2469373504 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde2401a040 con 0x563032e72a50 2024-03-16T12:07:46.474 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:46.477+0000 7fde317fa640 1 -- 172.21.15.29:0/2469373504 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fde24005500 con 0x563032e72a50 2024-03-16T12:07:46.474 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:46.477+0000 7fde317fa640 1 -- 172.21.15.29:0/2469373504 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde24004a00 con 0x563032e72a50 2024-03-16T12:07:46.474 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:46.478+0000 7fde4497a740 1 -- 172.21.15.29:0/2469373504 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563032e72a50 msgr2=0x563032e67a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:07:46.474 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:46.478+0000 7fde4497a740 1 --2- 172.21.15.29:0/2469373504 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563032e72a50 0x563032e67a40 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7fde2401d000 tx=0x7fde24020000 comp rx=0 tx=0).stop 2024-03-16T12:07:46.474 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:46.478+0000 7fde4497a740 1 -- 172.21.15.29:0/2469373504 shutdown_connections 2024-03-16T12:07:46.474 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:46.478+0000 7fde4497a740 1 --2- 172.21.15.29:0/2469373504 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563032e7b820 0x563032e93180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:07:46.475 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:46.478+0000 7fde4497a740 1 --2- 172.21.15.29:0/2469373504 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563032e72a50 0x563032e67a40 unknown :-1 s=CLOSED pgs=778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:07:46.475 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:46.478+0000 7fde4497a740 1 --2- 172.21.15.29:0/2469373504 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563032e72130 0x563032e86ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:07:46.475 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:46.478+0000 7fde4497a740 1 -- 172.21.15.29:0/2469373504 >> 172.21.15.29:0/2469373504 conn(0x563032e6ca50 msgr2=0x563032e6e730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:07:46.475 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:46.478+0000 7fde4497a740 1 -- 172.21.15.29:0/2469373504 shutdown_connections 2024-03-16T12:07:46.475 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:46.478+0000 7fde4497a740 1 -- 172.21.15.29:0/2469373504 wait complete. 2024-03-16T12:07:46.475 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:46.478+0000 7fde4497a740 1 -- start start 2024-03-16T12:07:46.475 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:46.479+0000 7fde4497a740 1 Processor -- start 2024-03-16T12:07:46.475 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:46.479+0000 7fde4497a740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563032e72130 0x563032e83070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:07:46.475 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:46.479+0000 7fde4497a740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563032e72a50 0x563032e835b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:07:46.475 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:46.479+0000 7fde4497a740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563032e7b820 0x563032e7f510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:07:46.475 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:46.479+0000 7fde4497a740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fde3c1add20 con 0x563032e7b820 2024-03-16T12:07:46.475 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:46.479+0000 7fde4497a740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fde3c1af190 con 0x563032e72130 2024-03-16T12:07:46.475 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:46.479+0000 7fde4497a740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fde3c1ae3f0 con 0x563032e72a50 2024-03-16T12:07:46.475 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:46.479+0000 7fde4497a740 10 client.0 _ll_register_callbacks cb 0x7fde24009980 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T12:07:46.475 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:46.479+0000 7fde3bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563032e7b820 0x563032e7f510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:07:46.476 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:46.479+0000 7fde3bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563032e7b820 0x563032e7f510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33246/0 (socket says 172.21.15.29:33246) 2024-03-16T12:07:46.476 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:46.479+0000 7fde3bfff640 1 -- 172.21.15.29:0/3909624803 learned_addr learned my addr 172.21.15.29:0/3909624803 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T12:07:46.476 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:46.479+0000 7fde32ffd640 1 --2- 172.21.15.29:0/3909624803 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563032e72a50 0x563032e835b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:07:46.476 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:46.479+0000 7fde3b7fe640 1 --2- 172.21.15.29:0/3909624803 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563032e72130 0x563032e83070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:07:46.476 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:46.479+0000 7fde3bfff640 1 -- 172.21.15.29:0/3909624803 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563032e72a50 msgr2=0x563032e835b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:07:46.476 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:46.479+0000 7fde3bfff640 1 --2- 172.21.15.29:0/3909624803 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563032e72a50 0x563032e835b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:07:46.476 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:46.479+0000 7fde3bfff640 1 -- 172.21.15.29:0/3909624803 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563032e72130 msgr2=0x563032e83070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:07:46.476 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:46.479+0000 7fde3bfff640 1 --2- 172.21.15.29:0/3909624803 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563032e72130 0x563032e83070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:07:46.476 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:46.479+0000 7fde3bfff640 1 -- 172.21.15.29:0/3909624803 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563032e72880 con 0x563032e7b820 2024-03-16T12:07:46.476 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:46.479+0000 7fde3bfff640 1 --2- 172.21.15.29:0/3909624803 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563032e7b820 0x563032e7f510 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7fde34004270 tx=0x7fde34004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=883bac50ea6f99b3 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:07:46.476 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:46.479+0000 7fde1affd640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-16T12:07:46.476 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:46.480+0000 7fde1affd640 1 -- 172.21.15.29:0/3909624803 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde34022060 con 0x563032e7b820 2024-03-16T12:07:46.476 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:46.480+0000 7fde1affd640 1 -- 172.21.15.29:0/3909624803 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(114..114 src has 1..114) ==== 8556+0+0 (secure 0 0 0) 0x7fde34012bb0 con 0x563032e7b820 2024-03-16T12:07:46.476 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:46.480+0000 7fde4497a740 1 -- 172.21.15.29:0/3909624803 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fde24005500 con 0x563032e7b820 2024-03-16T12:07:46.476 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:46.480+0000 7fde1affd640 1 -- 172.21.15.29:0/3909624803 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fde34012db0 con 0x563032e7b820 2024-03-16T12:07:46.477 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:46.480+0000 7fde4497a740 10 client.16785 fetch_fsmap learned FSMap version 32 2024-03-16T12:07:46.477 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:46.480+0000 7fde4497a740 1 -- 172.21.15.29:0/3909624803 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x563032ef2050 con 0x563032e7b820 2024-03-16T12:07:46.477 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:46.480+0000 7fde1affd640 1 -- 172.21.15.29:0/3909624803 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fde24005500 con 0x563032e7b820 2024-03-16T12:07:46.477 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:46.480+0000 7fde1affd640 1 -- 172.21.15.29:0/3909624803 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 32) ==== 8338+0+0 (secure 0 0 0) 0x7fde340104c0 con 0x563032e7b820 2024-03-16T12:07:46.477 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:46.480+0000 7fde4497a740 10 client.16785 fetch_fsmap finished waiting for FSMap version 32 2024-03-16T12:07:46.477 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:46.480+0000 7fde4497a740 10 client.16785 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T12:07:46.477 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:46.480+0000 7fde4497a740 20 client.16785 populate_metadata read hostname 'smithi029' 2024-03-16T12:07:46.477 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:46.480+0000 7fde4497a740 1 --2- 172.21.15.29:0/3909624803 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x563032ef2800 0x563032ef4c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:07:46.477 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:46.480+0000 7fde4497a740 4 client.16785 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-16T12:07:46.477 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:46.480+0000 7fde4497a740 1 -- 172.21.15.29:0/3909624803 --> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] -- command(tid 0: {"prefix": "flush journal"}) -- 0x563032dde2d0 con 0x563032ef2800 2024-03-16T12:07:46.477 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:46.480+0000 7fde3b7fe640 1 --2- 172.21.15.29:0/3909624803 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x563032ef2800 0x563032ef4c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:07:46.477 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:46.481+0000 7fde3b7fe640 1 --2- 172.21.15.29:0/3909624803 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x563032ef2800 0x563032ef4c20 crc :-1 s=READY pgs=189 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8909f149335a7236 server_cookie=a21dd7b122029fa5 in_seq=0 out_seq=0 2024-03-16T12:07:46.477 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:46.481+0000 7fde1affd640 10 client.16785 ms_handle_connect on v2:172.21.15.152:6838/1548731833 2024-03-16T12:07:46.786 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:07:46 smithi154 ceph-mon[36320]: pgmap v534: 97 pgs: 97 active+clean; 373 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 3.0 MiB/s wr, 369 op/s 2024-03-16T12:07:46.829 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:07:46 smithi029 ceph-mon[31975]: pgmap v534: 97 pgs: 97 active+clean; 373 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 3.0 MiB/s wr, 369 op/s 2024-03-16T12:07:46.868 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:07:46 smithi152 ceph-mon[36291]: pgmap v534: 97 pgs: 97 active+clean; 373 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 3.0 MiB/s wr, 369 op/s 2024-03-16T12:07:47.100 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:47.109+0000 7fde1affd640 1 -- 172.21.15.29:0/3909624803 <== mds.0 v2:172.21.15.152:6838/1548731833 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x563032dde2d0 con 0x563032ef2800 2024-03-16T12:07:47.100 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:47.109+0000 7fde1affd640 10 client.16785 handle_command_reply: tid=0 2024-03-16T12:07:47.101 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:47.109+0000 7fde4497a740 1 client.16785 shutdown 2024-03-16T12:07:47.101 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:47.109+0000 7fde4497a740 1 -- 172.21.15.29:0/3909624803 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563032e7b820 msgr2=0x563032e7f510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:07:47.101 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:47.109+0000 7fde4497a740 1 --2- 172.21.15.29:0/3909624803 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563032e7b820 0x563032e7f510 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7fde34004270 tx=0x7fde34004730 comp rx=0 tx=0).stop 2024-03-16T12:07:47.101 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:47.109+0000 7fde4497a740 1 -- 172.21.15.29:0/3909624803 shutdown_connections 2024-03-16T12:07:47.101 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:47.109+0000 7fde4497a740 1 --2- 172.21.15.29:0/3909624803 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563032e7b820 0x563032e7f510 unknown :-1 s=CLOSED pgs=779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:07:47.101 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:47.109+0000 7fde4497a740 1 --2- 172.21.15.29:0/3909624803 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563032e72a50 0x563032e835b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:07:47.101 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:47.109+0000 7fde4497a740 1 --2- 172.21.15.29:0/3909624803 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x563032ef2800 0x563032ef4c20 crc :-1 s=READY pgs=189 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:07:47.101 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:47.109+0000 7fde1affd640 0 client.16785 ms_handle_reset on v2:172.21.15.152:6838/1548731833 2024-03-16T12:07:47.101 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:47.109+0000 7fde4497a740 1 --2- 172.21.15.29:0/3909624803 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563032e72130 0x563032e83070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:07:47.101 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:47.109+0000 7fde4497a740 1 -- 172.21.15.29:0/3909624803 >> 172.21.15.29:0/3909624803 conn(0x563032e6ca50 msgr2=0x563032e79fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:07:47.101 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:47.109+0000 7fde4497a740 1 -- 172.21.15.29:0/3909624803 shutdown_connections 2024-03-16T12:07:47.101 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:47.110+0000 7fde4497a740 1 -- 172.21.15.29:0/3909624803 wait complete. 2024-03-16T12:07:47.101 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:47.110+0000 7fde4497a740 20 client.16785 trim_cache size 0 max 16384 2024-03-16T12:07:47.102 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:{ 2024-03-16T12:07:47.102 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "message": "", 2024-03-16T12:07:47.102 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "return_code": 0 2024-03-16T12:07:47.102 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:} 2024-03-16T12:07:47.103 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:47.110+0000 7fde41d89640 1 -- 172.21.15.29:0/2761312129 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7fde280a2a10 msgr2=0x7fde280a4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:07:47.103 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:47.110+0000 7fde41d89640 1 --2- 172.21.15.29:0/2761312129 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7fde280a2a10 0x7fde280a4ed0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fde3c198e60 tx=0x7fde24011920 comp rx=0 tx=0).stop 2024-03-16T12:07:47.103 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:47.110+0000 7fde41d89640 1 -- 172.21.15.29:0/2761312129 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fde3c100930 msgr2=0x7fde3c196d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:07:47.103 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:47.110+0000 7fde41d89640 1 --2- 172.21.15.29:0/2761312129 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fde3c100930 0x7fde3c196d00 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7fde2c02efd0 tx=0x7fde2c0042f0 comp rx=0 tx=0).stop 2024-03-16T12:07:47.103 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:47.110+0000 7fde41d89640 1 -- 172.21.15.29:0/2761312129 shutdown_connections 2024-03-16T12:07:47.103 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:47.110+0000 7fde41d89640 1 --2- 172.21.15.29:0/2761312129 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7fde280a2a10 0x7fde280a4ed0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:07:47.103 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:47.110+0000 7fde41d89640 1 --2- 172.21.15.29:0/2761312129 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fde3c105680 0x7fde3c197780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:07:47.103 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:47.110+0000 7fde41d89640 1 --2- 172.21.15.29:0/2761312129 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fde3c1012f0 0x7fde3c197240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:07:47.103 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:47.110+0000 7fde41d89640 1 --2- 172.21.15.29:0/2761312129 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fde3c100930 0x7fde3c196d00 unknown :-1 s=CLOSED pgs=776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:07:47.103 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:47.110+0000 7fde41d89640 1 -- 172.21.15.29:0/2761312129 >> 172.21.15.29:0/2761312129 conn(0x7fde3c077530 msgr2=0x7fde3c0fe900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:07:47.103 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:47.110+0000 7fde41d89640 1 -- 172.21.15.29:0/2761312129 shutdown_connections 2024-03-16T12:07:47.103 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:07:47.110+0000 7fde41d89640 1 -- 172.21.15.29:0/2761312129 wait complete. 2024-03-16T12:07:47.188 INFO:tasks.workunit.client.0.smithi029.stdout: 10 55215 13.09 MB/sec execute 91 sec latency 565.370 ms 2024-03-16T12:07:48.188 INFO:tasks.workunit.client.0.smithi029.stdout: 10 55528 13.09 MB/sec execute 92 sec latency 189.474 ms 2024-03-16T12:07:48.786 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:07:48 smithi154 ceph-mon[36320]: pgmap v535: 97 pgs: 97 active+clean; 381 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 MiB/s rd, 5.2 MiB/s wr, 481 op/s 2024-03-16T12:07:48.829 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:07:48 smithi029 ceph-mon[31975]: pgmap v535: 97 pgs: 97 active+clean; 381 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 MiB/s rd, 5.2 MiB/s wr, 481 op/s 2024-03-16T12:07:48.868 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:07:48 smithi152 ceph-mon[36291]: pgmap v535: 97 pgs: 97 active+clean; 381 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 MiB/s rd, 5.2 MiB/s wr, 481 op/s 2024-03-16T12:07:49.188 INFO:tasks.workunit.client.0.smithi029.stdout: 10 55973 13.28 MB/sec execute 93 sec latency 95.412 ms 2024-03-16T12:07:50.188 INFO:tasks.workunit.client.0.smithi029.stdout: 10 56485 13.49 MB/sec execute 94 sec latency 210.724 ms 2024-03-16T12:07:50.786 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:07:50 smithi154 ceph-mon[36320]: pgmap v536: 97 pgs: 97 active+clean; 381 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 3.7 MiB/s wr, 207 op/s 2024-03-16T12:07:50.829 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:07:50 smithi029 ceph-mon[31975]: pgmap v536: 97 pgs: 97 active+clean; 381 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 3.7 MiB/s wr, 207 op/s 2024-03-16T12:07:50.868 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:07:50 smithi152 ceph-mon[36291]: pgmap v536: 97 pgs: 97 active+clean; 381 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 3.7 MiB/s wr, 207 op/s 2024-03-16T12:07:51.188 INFO:tasks.workunit.client.0.smithi029.stdout: 10 56814 13.57 MB/sec execute 95 sec latency 215.501 ms 2024-03-16T12:07:52.188 INFO:tasks.workunit.client.0.smithi029.stdout: 10 57072 13.62 MB/sec execute 96 sec latency 224.324 ms 2024-03-16T12:07:52.786 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:07:52 smithi154 ceph-mon[36320]: pgmap v537: 97 pgs: 97 active+clean; 372 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 MiB/s rd, 4.3 MiB/s wr, 285 op/s 2024-03-16T12:07:52.829 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:07:52 smithi029 ceph-mon[31975]: pgmap v537: 97 pgs: 97 active+clean; 372 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 MiB/s rd, 4.3 MiB/s wr, 285 op/s 2024-03-16T12:07:52.868 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:07:52 smithi152 ceph-mon[36291]: pgmap v537: 97 pgs: 97 active+clean; 372 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 MiB/s rd, 4.3 MiB/s wr, 285 op/s 2024-03-16T12:07:53.190 INFO:tasks.workunit.client.0.smithi029.stdout: 10 57274 13.62 MB/sec execute 97 sec latency 235.470 ms 2024-03-16T12:07:54.188 INFO:tasks.workunit.client.0.smithi029.stdout: 10 57454 13.59 MB/sec execute 98 sec latency 263.802 ms 2024-03-16T12:07:54.786 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:07:54 smithi154 ceph-mon[36320]: pgmap v538: 97 pgs: 97 active+clean; 371 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 MiB/s rd, 5.1 MiB/s wr, 429 op/s 2024-03-16T12:07:54.829 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:07:54 smithi029 ceph-mon[31975]: pgmap v538: 97 pgs: 97 active+clean; 371 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 MiB/s rd, 5.1 MiB/s wr, 429 op/s 2024-03-16T12:07:54.868 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:07:54 smithi152 ceph-mon[36291]: pgmap v538: 97 pgs: 97 active+clean; 371 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 MiB/s rd, 5.1 MiB/s wr, 429 op/s 2024-03-16T12:07:55.188 INFO:tasks.workunit.client.0.smithi029.stdout: 10 57596 13.49 MB/sec execute 99 sec latency 326.362 ms 2024-03-16T12:07:56.188 INFO:tasks.workunit.client.0.smithi029.stdout: 10 57786 13.45 MB/sec execute 100 sec latency 244.282 ms 2024-03-16T12:07:56.786 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:07:56 smithi154 ceph-mon[36320]: pgmap v539: 97 pgs: 97 active+clean; 371 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 4.2 MiB/s wr, 379 op/s 2024-03-16T12:07:56.829 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:07:56 smithi029 ceph-mon[31975]: pgmap v539: 97 pgs: 97 active+clean; 371 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 4.2 MiB/s wr, 379 op/s 2024-03-16T12:07:56.868 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:07:56 smithi152 ceph-mon[36291]: pgmap v539: 97 pgs: 97 active+clean; 371 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 4.2 MiB/s wr, 379 op/s 2024-03-16T12:07:57.188 INFO:tasks.workunit.client.0.smithi029.stdout: 10 58038 13.39 MB/sec execute 101 sec latency 102.775 ms 2024-03-16T12:07:58.189 INFO:tasks.workunit.client.0.smithi029.stdout: 10 58269 13.34 MB/sec execute 102 sec latency 156.086 ms 2024-03-16T12:07:58.786 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:07:58 smithi154 ceph-mon[36320]: pgmap v540: 97 pgs: 97 active+clean; 377 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 MiB/s rd, 5.5 MiB/s wr, 468 op/s 2024-03-16T12:07:58.829 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:07:58 smithi029 ceph-mon[31975]: pgmap v540: 97 pgs: 97 active+clean; 377 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 MiB/s rd, 5.5 MiB/s wr, 468 op/s 2024-03-16T12:07:58.868 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:07:58 smithi152 ceph-mon[36291]: pgmap v540: 97 pgs: 97 active+clean; 377 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 MiB/s rd, 5.5 MiB/s wr, 468 op/s 2024-03-16T12:07:59.189 INFO:tasks.workunit.client.0.smithi029.stdout: 10 58526 13.32 MB/sec execute 103 sec latency 284.459 ms 2024-03-16T12:08:00.189 INFO:tasks.workunit.client.0.smithi029.stdout: 10 58885 13.34 MB/sec execute 104 sec latency 226.844 ms 2024-03-16T12:08:00.196 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:08:00.205+0000 7f457ed37640 1 -- 172.21.15.29:0/3108706728 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f45781008a0 msgr2=0x7f4578106330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:08:00.196 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:08:00.205+0000 7f457ed37640 1 --2- 172.21.15.29:0/3108706728 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f45781008a0 0x7f4578106330 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f4564009900 tx=0x7f456402bc30 comp rx=0 tx=0).stop 2024-03-16T12:08:00.196 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:08:00.205+0000 7f457ed37640 1 -- 172.21.15.29:0/3108706728 shutdown_connections 2024-03-16T12:08:00.196 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:08:00.205+0000 7f457ed37640 1 --2- 172.21.15.29:0/3108706728 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4578106a10 0x7f457810e6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:08:00.196 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:08:00.205+0000 7f457ed37640 1 --2- 172.21.15.29:0/3108706728 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f45781008a0 0x7f4578106330 unknown :-1 s=CLOSED pgs=780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:08:00.196 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:08:00.205+0000 7f457ed37640 1 --2- 172.21.15.29:0/3108706728 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f45780ffef0 0x7f45781002d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:08:00.197 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:08:00.205+0000 7f457ed37640 1 -- 172.21.15.29:0/3108706728 >> 172.21.15.29:0/3108706728 conn(0x7f4578076c00 msgr2=0x7f45780fdf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:08:00.197 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:08:00.205+0000 7f457ed37640 1 -- 172.21.15.29:0/3108706728 shutdown_connections 2024-03-16T12:08:00.197 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:08:00.205+0000 7f457ed37640 1 -- 172.21.15.29:0/3108706728 wait complete. 2024-03-16T12:08:00.197 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:08:00.206+0000 7f457ed37640 1 Processor -- start 2024-03-16T12:08:00.197 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:08:00.206+0000 7f457ed37640 1 -- start start 2024-03-16T12:08:00.199 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:08:00.206+0000 7f457ed37640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f45780ffef0 0x7f4578198030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:08:00.199 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:08:00.206+0000 7f457ed37640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4578106a10 0x7f4578198570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:08:00.199 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:08:00.206+0000 7f457ed37640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f457819c8f0 0x7f457819cda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:08:00.199 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:08:00.206+0000 7f457ed37640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f4578111b90 con 0x7f45780ffef0 2024-03-16T12:08:00.199 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:08:00.206+0000 7f457ed37640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f4578111a10 con 0x7f457819c8f0 2024-03-16T12:08:00.200 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:08:00.206+0000 7f457ed37640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f4578111d10 con 0x7f4578106a10 2024-03-16T12:08:00.200 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:08:00.206+0000 7f457cad3640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f45780ffef0 0x7f4578198030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:08:00.200 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:08:00.206+0000 7f457cad3640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f45780ffef0 0x7f4578198030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57758/0 (socket says 172.21.15.29:57758) 2024-03-16T12:08:00.200 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:08:00.206+0000 7f457cad3640 1 -- 172.21.15.29:0/2811264143 learned_addr learned my addr 172.21.15.29:0/2811264143 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T12:08:00.200 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:08:00.206+0000 7f4577fff640 1 --2- 172.21.15.29:0/2811264143 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4578106a10 0x7f4578198570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:08:00.200 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:08:00.207+0000 7f457cad3640 1 -- 172.21.15.29:0/2811264143 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4578106a10 msgr2=0x7f4578198570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:08:00.200 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:08:00.207+0000 7f457cad3640 1 --2- 172.21.15.29:0/2811264143 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4578106a10 0x7f4578198570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:08:00.200 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:08:00.207+0000 7f457cad3640 1 -- 172.21.15.29:0/2811264143 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f457819c8f0 msgr2=0x7f457819cda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:08:00.200 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:08:00.207+0000 7f457cad3640 1 --2- 172.21.15.29:0/2811264143 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f457819c8f0 0x7f457819cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:08:00.200 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:08:00.207+0000 7f457cad3640 1 -- 172.21.15.29:0/2811264143 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f457819d3e0 con 0x7f45780ffef0 2024-03-16T12:08:00.200 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:08:00.207+0000 7f457cad3640 1 --2- 172.21.15.29:0/2811264143 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f45780ffef0 0x7f4578198030 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f456c00e9f0 tx=0x7f456c00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:08:00.201 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:08:00.207+0000 7f4575ffb640 1 -- 172.21.15.29:0/2811264143 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f456c00cd80 con 0x7f45780ffef0 2024-03-16T12:08:00.201 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:08:00.207+0000 7f4575ffb640 1 -- 172.21.15.29:0/2811264143 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f456c005500 con 0x7f45780ffef0 2024-03-16T12:08:00.201 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:08:00.208+0000 7f457ed37640 1 -- 172.21.15.29:0/2811264143 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f457819d6d0 con 0x7f45780ffef0 2024-03-16T12:08:00.201 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:08:00.208+0000 7f457ed37640 1 -- 172.21.15.29:0/2811264143 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f45781a4e40 con 0x7f45780ffef0 2024-03-16T12:08:00.201 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:08:00.208+0000 7f4575ffb640 1 -- 172.21.15.29:0/2811264143 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f456c010690 con 0x7f45780ffef0 2024-03-16T12:08:00.206 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:08:00.210+0000 7f4575ffb640 1 -- 172.21.15.29:0/2811264143 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7f456c0046c0 con 0x7f45780ffef0 2024-03-16T12:08:00.206 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:08:00.210+0000 7f45819f8740 1 Processor -- start 2024-03-16T12:08:00.207 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:08:00.210+0000 7f45819f8740 1 -- start start 2024-03-16T12:08:00.207 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:08:00.210+0000 7f45819f8740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55de4af48130 0x55de4af48510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:08:00.207 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:08:00.210+0000 7f45819f8740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55de4af48a50 0x55de4af512e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:08:00.207 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:08:00.210+0000 7f4575ffb640 1 --2- 172.21.15.29:0/2811264143 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f45680a2a10 0x7f45680a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:08:00.207 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:08:00.211+0000 7f45819f8740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55de4af51820 0x55de4af53c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:08:00.207 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:08:00.211+0000 7f45819f8740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55de4aeaca40 con 0x55de4af48a50 2024-03-16T12:08:00.207 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:08:00.211+0000 7f45819f8740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x55de4ae55920 con 0x55de4af51820 2024-03-16T12:08:00.207 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:08:00.211+0000 7f45819f8740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55de4aea8ae0 con 0x55de4af48130 2024-03-16T12:08:00.207 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:08:00.211+0000 7f4577fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55de4af48a50 0x55de4af512e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:08:00.207 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:08:00.211+0000 7f457d2d4640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55de4af48130 0x55de4af48510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:08:00.207 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:08:00.211+0000 7f4577fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55de4af48a50 0x55de4af512e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57760/0 (socket says 172.21.15.29:57760) 2024-03-16T12:08:00.207 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:08:00.211+0000 7f4577fff640 1 -- 172.21.15.29:0/491595697 learned_addr learned my addr 172.21.15.29:0/491595697 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T12:08:00.208 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:08:00.211+0000 7f457cad3640 1 --2- 172.21.15.29:0/2811264143 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f45680a2a10 0x7f45680a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:08:00.208 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:08:00.211+0000 7f4575ffb640 1 -- 172.21.15.29:0/2811264143 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(114..114 src has 1..114) ==== 8556+0+0 (secure 0 0 0) 0x7f456c0cf330 con 0x7f45780ffef0 2024-03-16T12:08:00.208 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:08:00.211+0000 7f457d2d4640 1 --2- 172.21.15.29:0/491595697 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55de4af51820 0x55de4af53c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:08:00.208 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:08:00.211+0000 7f4577fff640 1 -- 172.21.15.29:0/491595697 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55de4af48130 msgr2=0x55de4af48510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:08:00.208 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:08:00.211+0000 7f4577fff640 1 --2- 172.21.15.29:0/491595697 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55de4af48130 0x55de4af48510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:08:00.208 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:08:00.211+0000 7f4577fff640 1 -- 172.21.15.29:0/491595697 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55de4af51820 msgr2=0x55de4af53c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:08:00.208 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:08:00.211+0000 7f4577fff640 1 --2- 172.21.15.29:0/491595697 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55de4af51820 0x55de4af53c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:08:00.208 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:08:00.211+0000 7f4577fff640 1 -- 172.21.15.29:0/491595697 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55de4ae6d460 con 0x55de4af48a50 2024-03-16T12:08:00.208 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:08:00.211+0000 7f457d2d4640 1 --2- 172.21.15.29:0/491595697 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55de4af48130 0x55de4af48510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:08:00.208 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:08:00.211+0000 7f457cad3640 1 --2- 172.21.15.29:0/2811264143 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f45680a2a10 0x7f45680a4ed0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f456c0ccb10 tx=0x7f456c086040 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:08:00.208 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:08:00.211+0000 7f4577fff640 1 --2- 172.21.15.29:0/491595697 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55de4af48a50 0x55de4af512e0 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f4578073f00 tx=0x7f4564005be0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1f49ae31bf9b258 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:08:00.208 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:08:00.211+0000 7f4562ffd640 1 -- 172.21.15.29:0/491595697 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4564042070 con 0x55de4af48a50 2024-03-16T12:08:00.208 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:08:00.211+0000 7f4562ffd640 1 -- 172.21.15.29:0/491595697 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4564004030 con 0x55de4af48a50 2024-03-16T12:08:00.209 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:08:00.211+0000 7f4562ffd640 1 -- 172.21.15.29:0/491595697 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4564004f20 con 0x55de4af48a50 2024-03-16T12:08:00.209 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:08:00.212+0000 7f45819f8740 1 -- 172.21.15.29:0/491595697 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55de4af48a50 msgr2=0x55de4af512e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:08:00.209 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:08:00.212+0000 7f45819f8740 1 --2- 172.21.15.29:0/491595697 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55de4af48a50 0x55de4af512e0 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f4578073f00 tx=0x7f4564005be0 comp rx=0 tx=0).stop 2024-03-16T12:08:00.209 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:08:00.212+0000 7f45819f8740 1 -- 172.21.15.29:0/491595697 shutdown_connections 2024-03-16T12:08:00.209 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:08:00.212+0000 7f45819f8740 1 --2- 172.21.15.29:0/491595697 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55de4af51820 0x55de4af53c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:08:00.209 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:08:00.212+0000 7f45819f8740 1 --2- 172.21.15.29:0/491595697 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55de4af48a50 0x55de4af512e0 unknown :-1 s=CLOSED pgs=782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:08:00.209 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:08:00.212+0000 7f45819f8740 1 --2- 172.21.15.29:0/491595697 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55de4af48130 0x55de4af48510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:08:00.209 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:08:00.212+0000 7f45819f8740 1 -- 172.21.15.29:0/491595697 >> 172.21.15.29:0/491595697 conn(0x55de4af42a50 msgr2=0x55de4af42e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:08:00.209 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:08:00.212+0000 7f45819f8740 1 -- 172.21.15.29:0/491595697 shutdown_connections 2024-03-16T12:08:00.209 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:08:00.212+0000 7f45819f8740 1 -- 172.21.15.29:0/491595697 wait complete. 2024-03-16T12:08:00.209 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:08:00.213+0000 7f45819f8740 1 -- start start 2024-03-16T12:08:00.209 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:08:00.213+0000 7f45819f8740 1 Processor -- start 2024-03-16T12:08:00.210 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:08:00.213+0000 7f45819f8740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55de4af48130 0x55de4af4e890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:08:00.210 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:08:00.213+0000 7f45819f8740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55de4af48a50 0x55de4af4edd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:08:00.210 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:08:00.213+0000 7f45819f8740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55de4af51820 0x55de4af4f310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:08:00.210 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:08:00.213+0000 7f45819f8740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55de4af543f0 con 0x55de4af48130 2024-03-16T12:08:00.210 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:08:00.213+0000 7f45819f8740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x55de4ae68d70 con 0x55de4af48a50 2024-03-16T12:08:00.210 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:08:00.213+0000 7f45819f8740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55de4af54570 con 0x55de4af51820 2024-03-16T12:08:00.210 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:08:00.213+0000 7f45819f8740 10 client.0 _ll_register_callbacks cb 0x55de4abc0a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T12:08:00.210 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:08:00.213+0000 7f457d2d4640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55de4af48130 0x55de4af4e890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:08:00.210 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:08:00.213+0000 7f457d2d4640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55de4af48130 0x55de4af4e890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57768/0 (socket says 172.21.15.29:57768) 2024-03-16T12:08:00.210 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:08:00.213+0000 7f457d2d4640 1 -- 172.21.15.29:0/522590838 learned_addr learned my addr 172.21.15.29:0/522590838 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T12:08:00.210 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:08:00.213+0000 7f4577fff640 1 --2- 172.21.15.29:0/522590838 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55de4af48a50 0x55de4af4edd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:08:00.210 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:08:00.213+0000 7f457d2d4640 1 --2- 172.21.15.29:0/522590838 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55de4af51820 0x55de4af4f310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:08:00.211 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:08:00.213+0000 7f457d2d4640 1 -- 172.21.15.29:0/522590838 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55de4af51820 msgr2=0x55de4af4f310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:08:00.211 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:08:00.213+0000 7f457d2d4640 1 --2- 172.21.15.29:0/522590838 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55de4af51820 0x55de4af4f310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:08:00.211 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:08:00.213+0000 7f457d2d4640 1 -- 172.21.15.29:0/522590838 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55de4af48a50 msgr2=0x55de4af4edd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:08:00.211 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:08:00.213+0000 7f457d2d4640 1 --2- 172.21.15.29:0/522590838 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55de4af48a50 0x55de4af4edd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:08:00.211 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:08:00.213+0000 7f457d2d4640 1 -- 172.21.15.29:0/522590838 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55de4abc1210 con 0x55de4af48130 2024-03-16T12:08:00.211 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:08:00.214+0000 7f457d2d4640 1 --2- 172.21.15.29:0/522590838 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55de4af48130 0x55de4af4e890 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f4570014e80 tx=0x7f4570004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c05c5b7edd8783de server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:08:00.211 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:08:00.214+0000 7f4557fff640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-16T12:08:00.211 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:08:00.214+0000 7f4557fff640 1 -- 172.21.15.29:0/522590838 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4570004b20 con 0x55de4af48130 2024-03-16T12:08:00.211 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:08:00.214+0000 7f4557fff640 1 -- 172.21.15.29:0/522590838 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(114..114 src has 1..114) ==== 8556+0+0 (secure 0 0 0) 0x7f4570009040 con 0x55de4af48130 2024-03-16T12:08:00.211 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:08:00.214+0000 7f45819f8740 1 -- 172.21.15.29:0/522590838 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4564004030 con 0x55de4af48130 2024-03-16T12:08:00.211 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:08:00.214+0000 7f4557fff640 1 -- 172.21.15.29:0/522590838 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f4570004cc0 con 0x55de4af48130 2024-03-16T12:08:00.211 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:08:00.214+0000 7f45819f8740 10 client.16797 fetch_fsmap learned FSMap version 32 2024-03-16T12:08:00.211 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:08:00.214+0000 7f45819f8740 1 -- 172.21.15.29:0/522590838 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55de4ac8d620 con 0x55de4af48130 2024-03-16T12:08:00.212 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:08:00.214+0000 7f4557fff640 1 -- 172.21.15.29:0/522590838 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4564004030 con 0x55de4af48130 2024-03-16T12:08:00.212 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:08:00.214+0000 7f4557fff640 1 -- 172.21.15.29:0/522590838 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 32) ==== 8338+0+0 (secure 0 0 0) 0x7f457000f040 con 0x55de4af48130 2024-03-16T12:08:00.212 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:08:00.214+0000 7f45819f8740 10 client.16797 fetch_fsmap finished waiting for FSMap version 32 2024-03-16T12:08:00.212 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:08:00.214+0000 7f45819f8740 10 client.16797 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T12:08:00.212 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:08:00.214+0000 7f45819f8740 20 client.16797 populate_metadata read hostname 'smithi029' 2024-03-16T12:08:00.212 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:08:00.214+0000 7f45819f8740 1 --2- 172.21.15.29:0/522590838 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55de4af61a70 0x55de4af63e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:08:00.212 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:08:00.215+0000 7f45819f8740 4 client.16797 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T12:08:00.212 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:08:00.215+0000 7f45819f8740 1 -- 172.21.15.29:0/522590838 --> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55de4aeb9f80 con 0x55de4af61a70 2024-03-16T12:08:00.212 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:08:00.215+0000 7f4577fff640 1 --2- 172.21.15.29:0/522590838 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55de4af61a70 0x55de4af63e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:08:00.212 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:08:00.215+0000 7f4577fff640 1 --2- 172.21.15.29:0/522590838 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55de4af61a70 0x55de4af63e60 crc :-1 s=READY pgs=190 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4d9595c4f8e27020 server_cookie=451b77c8883d86f1 in_seq=0 out_seq=0 2024-03-16T12:08:00.212 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:08:00.215+0000 7f4557fff640 10 client.16797 ms_handle_connect on v2:172.21.15.152:6838/1548731833 2024-03-16T12:08:00.214 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:08:00.219+0000 7f4557fff640 1 -- 172.21.15.29:0/522590838 <== mds.0 v2:172.21.15.152:6838/1548731833 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55de4aeb9f80 con 0x55de4af61a70 2024-03-16T12:08:00.214 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:08:00.219+0000 7f4557fff640 10 client.16797 handle_command_reply: tid=0 2024-03-16T12:08:00.214 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:08:00.219+0000 7f45819f8740 1 client.16797 shutdown 2024-03-16T12:08:00.214 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:08:00.219+0000 7f45819f8740 1 -- 172.21.15.29:0/522590838 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55de4af48130 msgr2=0x55de4af4e890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:08:00.214 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:08:00.219+0000 7f45819f8740 1 --2- 172.21.15.29:0/522590838 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55de4af48130 0x55de4af4e890 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f4570014e80 tx=0x7f4570004a40 comp rx=0 tx=0).stop 2024-03-16T12:08:00.214 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:08:00.219+0000 7f45819f8740 1 -- 172.21.15.29:0/522590838 shutdown_connections 2024-03-16T12:08:00.214 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:08:00.219+0000 7f45819f8740 1 --2- 172.21.15.29:0/522590838 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55de4af51820 0x55de4af4f310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:08:00.214 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:08:00.219+0000 7f45819f8740 1 --2- 172.21.15.29:0/522590838 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55de4af61a70 0x55de4af63e60 crc :-1 s=READY pgs=190 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:08:00.214 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:08:00.219+0000 7f45819f8740 1 --2- 172.21.15.29:0/522590838 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55de4af48a50 0x55de4af4edd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:08:00.214 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:08:00.219+0000 7f45819f8740 1 --2- 172.21.15.29:0/522590838 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55de4af48130 0x55de4af4e890 unknown :-1 s=CLOSED pgs=783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:08:00.215 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:08:00.219+0000 7f45819f8740 1 -- 172.21.15.29:0/522590838 >> 172.21.15.29:0/522590838 conn(0x55de4af42a50 msgr2=0x55de4af447a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:08:00.215 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:08:00.219+0000 7f4557fff640 0 client.16797 ms_handle_reset on v2:172.21.15.152:6838/1548731833 2024-03-16T12:08:00.215 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:08:00.219+0000 7f45819f8740 1 -- 172.21.15.29:0/522590838 shutdown_connections 2024-03-16T12:08:00.215 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:08:00.219+0000 7f45819f8740 1 -- 172.21.15.29:0/522590838 wait complete. 2024-03-16T12:08:00.215 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:08:00.219+0000 7f45819f8740 20 client.16797 trim_cache size 0 max 16384 2024-03-16T12:08:00.231 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:08:00.236+0000 7f45819f8740 1 Processor -- start 2024-03-16T12:08:00.231 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:08:00.236+0000 7f45819f8740 1 -- start start 2024-03-16T12:08:00.232 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:08:00.236+0000 7f45819f8740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55de4af48130 0x55de4af50fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:08:00.232 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:08:00.236+0000 7f45819f8740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55de4af48a50 0x55de4af5d190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:08:00.232 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:08:00.236+0000 7f45819f8740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55de4af51820 0x55de4af69750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:08:00.232 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:08:00.236+0000 7f45819f8740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f45781afc80 con 0x55de4af51820 2024-03-16T12:08:00.232 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:08:00.236+0000 7f45819f8740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f45781ad550 con 0x55de4af48130 2024-03-16T12:08:00.232 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:08:00.236+0000 7f45819f8740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f45781ab3e0 con 0x55de4af48a50 2024-03-16T12:08:00.232 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:08:00.236+0000 7f4577fff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55de4af48a50 0x55de4af5d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:08:00.232 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:08:00.236+0000 7f457d2d4640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55de4af51820 0x55de4af69750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:08:00.233 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:08:00.236+0000 7f4577fff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55de4af48a50 0x55de4af5d190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.29:39106/0 (socket says 172.21.15.29:39106) 2024-03-16T12:08:00.233 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:08:00.236+0000 7f4577fff640 1 -- 172.21.15.29:0/1552017601 learned_addr learned my addr 172.21.15.29:0/1552017601 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T12:08:00.233 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:08:00.236+0000 7f457d2d4640 1 --2- 172.21.15.29:0/1552017601 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55de4af48130 0x55de4af50fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:08:00.233 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:08:00.237+0000 7f4577fff640 1 -- 172.21.15.29:0/1552017601 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55de4af48130 msgr2=0x55de4af50fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:08:00.233 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:08:00.237+0000 7f4577fff640 1 --2- 172.21.15.29:0/1552017601 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55de4af48130 0x55de4af50fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:08:00.233 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:08:00.237+0000 7f4577fff640 1 -- 172.21.15.29:0/1552017601 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55de4af51820 msgr2=0x55de4af69750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:08:00.233 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:08:00.237+0000 7f4577fff640 1 --2- 172.21.15.29:0/1552017601 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55de4af51820 0x55de4af69750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:08:00.233 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:08:00.237+0000 7f4577fff640 1 -- 172.21.15.29:0/1552017601 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55de4af3e310 con 0x55de4af48a50 2024-03-16T12:08:00.233 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:08:00.237+0000 7f457d2d4640 1 --2- 172.21.15.29:0/1552017601 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55de4af51820 0x55de4af69750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:08:00.233 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:08:00.237+0000 7f4577fff640 1 --2- 172.21.15.29:0/1552017601 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55de4af48a50 0x55de4af5d190 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f4578073f00 tx=0x7f45640048b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=160f59d9bb6adf4c server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:08:00.233 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:08:00.237+0000 7f45627fc640 1 -- 172.21.15.29:0/1552017601 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4564042040 con 0x55de4af48a50 2024-03-16T12:08:00.233 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:08:00.237+0000 7f45627fc640 1 -- 172.21.15.29:0/1552017601 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4564004920 con 0x55de4af48a50 2024-03-16T12:08:00.234 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:08:00.237+0000 7f45627fc640 1 -- 172.21.15.29:0/1552017601 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4564050310 con 0x55de4af48a50 2024-03-16T12:08:00.234 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:08:00.238+0000 7f45819f8740 1 -- 172.21.15.29:0/1552017601 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55de4af48a50 msgr2=0x55de4af5d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:08:00.234 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:08:00.238+0000 7f45819f8740 1 --2- 172.21.15.29:0/1552017601 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55de4af48a50 0x55de4af5d190 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f4578073f00 tx=0x7f45640048b0 comp rx=0 tx=0).stop 2024-03-16T12:08:00.234 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:08:00.238+0000 7f45819f8740 1 -- 172.21.15.29:0/1552017601 shutdown_connections 2024-03-16T12:08:00.234 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:08:00.238+0000 7f45819f8740 1 --2- 172.21.15.29:0/1552017601 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55de4af51820 0x55de4af69750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:08:00.234 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:08:00.238+0000 7f45819f8740 1 --2- 172.21.15.29:0/1552017601 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55de4af48a50 0x55de4af5d190 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:08:00.234 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:08:00.238+0000 7f45819f8740 1 --2- 172.21.15.29:0/1552017601 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55de4af48130 0x55de4af50fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:08:00.234 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:08:00.238+0000 7f45819f8740 1 -- 172.21.15.29:0/1552017601 >> 172.21.15.29:0/1552017601 conn(0x55de4af42a50 msgr2=0x55de4af41bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:08:00.234 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:08:00.238+0000 7f45819f8740 1 -- 172.21.15.29:0/1552017601 shutdown_connections 2024-03-16T12:08:00.234 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:08:00.238+0000 7f45819f8740 1 -- 172.21.15.29:0/1552017601 wait complete. 2024-03-16T12:08:00.234 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:08:00.238+0000 7f45819f8740 1 -- start start 2024-03-16T12:08:00.234 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:08:00.238+0000 7f45819f8740 1 Processor -- start 2024-03-16T12:08:00.234 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:08:00.239+0000 7f45819f8740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55de4af48130 0x55de4af59ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:08:00.235 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:08:00.239+0000 7f45819f8740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55de4af48a50 0x55de4af54ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:08:00.235 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:08:00.239+0000 7f45819f8740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55de4af51820 0x55de4af55400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:08:00.235 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:08:00.239+0000 7f45819f8740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f45781aba30 con 0x55de4af48a50 2024-03-16T12:08:00.235 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:08:00.239+0000 7f45819f8740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f45781ace80 con 0x55de4af48130 2024-03-16T12:08:00.235 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:08:00.239+0000 7f45819f8740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f45781ac0e0 con 0x55de4af51820 2024-03-16T12:08:00.235 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:08:00.239+0000 7f45819f8740 10 client.0 _ll_register_callbacks cb 0x7f4570014cd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T12:08:00.235 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:08:00.239+0000 7f4577fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55de4af48a50 0x55de4af54ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:08:00.235 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:08:00.239+0000 7f4577fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55de4af48a50 0x55de4af54ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57788/0 (socket says 172.21.15.29:57788) 2024-03-16T12:08:00.235 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:08:00.239+0000 7f4577fff640 1 -- 172.21.15.29:0/3651500032 learned_addr learned my addr 172.21.15.29:0/3651500032 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T12:08:00.235 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:08:00.239+0000 7f457d2d4640 1 --2- 172.21.15.29:0/3651500032 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55de4af51820 0x55de4af55400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:08:00.235 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:08:00.239+0000 7f457d2d4640 1 --2- 172.21.15.29:0/3651500032 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55de4af48130 0x55de4af59ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:08:00.235 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:08:00.239+0000 7f4577fff640 1 -- 172.21.15.29:0/3651500032 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55de4af51820 msgr2=0x55de4af55400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:08:00.236 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:08:00.239+0000 7f4577fff640 1 --2- 172.21.15.29:0/3651500032 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55de4af51820 0x55de4af55400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:08:00.236 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:08:00.239+0000 7f4577fff640 1 -- 172.21.15.29:0/3651500032 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55de4af48130 msgr2=0x55de4af59ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T12:08:00.236 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:08:00.239+0000 7f4577fff640 1 --2- 172.21.15.29:0/3651500032 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55de4af48130 0x55de4af59ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:08:00.236 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:08:00.239+0000 7f4577fff640 1 -- 172.21.15.29:0/3651500032 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55de4af55c30 con 0x55de4af48a50 2024-03-16T12:08:00.236 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:08:00.239+0000 7f457d2d4640 1 --2- 172.21.15.29:0/3651500032 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55de4af51820 0x55de4af55400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:08:00.236 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:08:00.239+0000 7f4577fff640 1 --2- 172.21.15.29:0/3651500032 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55de4af48a50 0x55de4af54ea0 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f456402bc70 tx=0x7f4564005850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d21ed920863b4c server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:08:00.236 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:08:00.239+0000 7f4557fff640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-03-16T12:08:00.236 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:08:00.239+0000 7f4557fff640 1 -- 172.21.15.29:0/3651500032 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f456405e060 con 0x55de4af48a50 2024-03-16T12:08:00.236 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:08:00.240+0000 7f4557fff640 1 -- 172.21.15.29:0/3651500032 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(114..114 src has 1..114) ==== 8556+0+0 (secure 0 0 0) 0x7f456402ca80 con 0x55de4af48a50 2024-03-16T12:08:00.236 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:08:00.240+0000 7f45819f8740 1 -- 172.21.15.29:0/3651500032 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4564004920 con 0x55de4af48a50 2024-03-16T12:08:00.236 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:08:00.240+0000 7f4557fff640 1 -- 172.21.15.29:0/3651500032 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f4564042040 con 0x55de4af48a50 2024-03-16T12:08:00.236 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:08:00.240+0000 7f45819f8740 10 client.16803 fetch_fsmap learned FSMap version 32 2024-03-16T12:08:00.236 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:08:00.240+0000 7f45819f8740 1 -- 172.21.15.29:0/3651500032 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55de4afc7e20 con 0x55de4af48a50 2024-03-16T12:08:00.237 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:08:00.240+0000 7f4557fff640 1 -- 172.21.15.29:0/3651500032 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4564004920 con 0x55de4af48a50 2024-03-16T12:08:00.237 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:08:00.240+0000 7f4557fff640 1 -- 172.21.15.29:0/3651500032 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 32) ==== 8338+0+0 (secure 0 0 0) 0x7f456404f480 con 0x55de4af48a50 2024-03-16T12:08:00.237 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:08:00.240+0000 7f45819f8740 10 client.16803 fetch_fsmap finished waiting for FSMap version 32 2024-03-16T12:08:00.237 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:08:00.240+0000 7f45819f8740 10 client.16803 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-16T12:08:00.237 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:08:00.240+0000 7f45819f8740 20 client.16803 populate_metadata read hostname 'smithi029' 2024-03-16T12:08:00.237 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:08:00.240+0000 7f45819f8740 1 --2- 172.21.15.29:0/3651500032 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55de4afc80f0 0x55de4afca4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:08:00.237 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:08:00.240+0000 7f45819f8740 4 client.16803 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-16T12:08:00.237 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:08:00.240+0000 7f45819f8740 1 -- 172.21.15.29:0/3651500032 --> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55de4aeb42d0 con 0x55de4afc80f0 2024-03-16T12:08:00.237 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:08:00.240+0000 7f457d2d4640 1 --2- 172.21.15.29:0/3651500032 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55de4afc80f0 0x55de4afca4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:08:00.237 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:08:00.241+0000 7f457d2d4640 1 --2- 172.21.15.29:0/3651500032 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55de4afc80f0 0x55de4afca4e0 crc :-1 s=READY pgs=191 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=37161202b79979ac server_cookie=8ebddd116b043985 in_seq=0 out_seq=0 2024-03-16T12:08:00.237 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:08:00.241+0000 7f4557fff640 10 client.16803 ms_handle_connect on v2:172.21.15.152:6838/1548731833 2024-03-16T12:08:00.769 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:08:00.778+0000 7f4557fff640 1 -- 172.21.15.29:0/3651500032 <== mds.0 v2:172.21.15.152:6838/1548731833 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55de4aeb42d0 con 0x55de4afc80f0 2024-03-16T12:08:00.769 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:08:00.778+0000 7f4557fff640 10 client.16803 handle_command_reply: tid=0 2024-03-16T12:08:00.769 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:08:00.778+0000 7f45819f8740 1 client.16803 shutdown 2024-03-16T12:08:00.769 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:08:00.778+0000 7f45819f8740 1 -- 172.21.15.29:0/3651500032 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55de4af48a50 msgr2=0x55de4af54ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:08:00.769 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:08:00.778+0000 7f45819f8740 1 --2- 172.21.15.29:0/3651500032 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55de4af48a50 0x55de4af54ea0 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f456402bc70 tx=0x7f4564005850 comp rx=0 tx=0).stop 2024-03-16T12:08:00.769 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:08:00.778+0000 7f45819f8740 1 -- 172.21.15.29:0/3651500032 shutdown_connections 2024-03-16T12:08:00.769 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:08:00.778+0000 7f45819f8740 1 --2- 172.21.15.29:0/3651500032 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55de4af51820 0x55de4af55400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:08:00.770 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:08:00.778+0000 7f45819f8740 1 --2- 172.21.15.29:0/3651500032 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55de4af48a50 0x55de4af54ea0 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:08:00.770 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:08:00.778+0000 7f45819f8740 1 --2- 172.21.15.29:0/3651500032 >> [v2:172.21.15.152:6838/1548731833,v1:172.21.15.152:6839/1548731833] conn(0x55de4afc80f0 0x55de4afca4e0 crc :-1 s=READY pgs=191 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:08:00.770 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:08:00.778+0000 7f45819f8740 1 --2- 172.21.15.29:0/3651500032 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55de4af48130 0x55de4af59ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:08:00.770 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:08:00.778+0000 7f4557fff640 0 client.16803 ms_handle_reset on v2:172.21.15.152:6838/1548731833 2024-03-16T12:08:00.770 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:08:00.778+0000 7f45819f8740 1 -- 172.21.15.29:0/3651500032 >> 172.21.15.29:0/3651500032 conn(0x55de4af42a50 msgr2=0x55de4af41320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:08:00.770 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:08:00.778+0000 7f45819f8740 1 -- 172.21.15.29:0/3651500032 shutdown_connections 2024-03-16T12:08:00.770 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:08:00.778+0000 7f45819f8740 1 -- 172.21.15.29:0/3651500032 wait complete. 2024-03-16T12:08:00.770 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:08:00.778+0000 7f45819f8740 20 client.16803 trim_cache size 0 max 16384 2024-03-16T12:08:00.770 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:{ 2024-03-16T12:08:00.770 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "message": "", 2024-03-16T12:08:00.771 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "return_code": 0 2024-03-16T12:08:00.771 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:} 2024-03-16T12:08:00.771 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:08:00.779+0000 7f457ed37640 1 -- 172.21.15.29:0/2811264143 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f45680a2a10 msgr2=0x7f45680a4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:08:00.771 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:08:00.779+0000 7f457ed37640 1 --2- 172.21.15.29:0/2811264143 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f45680a2a10 0x7f45680a4ed0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f456c0ccb10 tx=0x7f456c086040 comp rx=0 tx=0).stop 2024-03-16T12:08:00.771 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:08:00.779+0000 7f457ed37640 1 -- 172.21.15.29:0/2811264143 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f45780ffef0 msgr2=0x7f4578198030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:08:00.771 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:08:00.779+0000 7f457ed37640 1 --2- 172.21.15.29:0/2811264143 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f45780ffef0 0x7f4578198030 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f456c00e9f0 tx=0x7f456c00edf0 comp rx=0 tx=0).stop 2024-03-16T12:08:00.772 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:08:00.779+0000 7f457ed37640 1 -- 172.21.15.29:0/2811264143 shutdown_connections 2024-03-16T12:08:00.772 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:08:00.779+0000 7f457ed37640 1 --2- 172.21.15.29:0/2811264143 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f45680a2a10 0x7f45680a4ed0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:08:00.772 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:08:00.779+0000 7f457ed37640 1 --2- 172.21.15.29:0/2811264143 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f457819c8f0 0x7f457819cda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:08:00.772 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:08:00.779+0000 7f457ed37640 1 --2- 172.21.15.29:0/2811264143 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4578106a10 0x7f4578198570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:08:00.772 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:08:00.779+0000 7f457ed37640 1 --2- 172.21.15.29:0/2811264143 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f45780ffef0 0x7f4578198030 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:08:00.772 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:08:00.779+0000 7f457ed37640 1 -- 172.21.15.29:0/2811264143 >> 172.21.15.29:0/2811264143 conn(0x7f4578076c00 msgr2=0x7f45780fdf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:08:00.772 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:08:00.779+0000 7f457ed37640 1 -- 172.21.15.29:0/2811264143 shutdown_connections 2024-03-16T12:08:00.772 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-03-16T12:08:00.779+0000 7f457ed37640 1 -- 172.21.15.29:0/2811264143 wait complete. 2024-03-16T12:08:00.786 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:08:00 smithi154 ceph-mon[36320]: pgmap v541: 97 pgs: 97 active+clean; 377 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 3.2 MiB/s wr, 355 op/s 2024-03-16T12:08:00.829 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:08:00 smithi029 ceph-mon[31975]: pgmap v541: 97 pgs: 97 active+clean; 377 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 3.2 MiB/s wr, 355 op/s 2024-03-16T12:08:00.868 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:08:00 smithi152 ceph-mon[36291]: pgmap v541: 97 pgs: 97 active+clean; 377 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 3.2 MiB/s wr, 355 op/s 2024-03-16T12:08:01.189 INFO:tasks.workunit.client.0.smithi029.stdout: 10 58999 13.28 MB/sec execute 105 sec latency 629.445 ms 2024-03-16T12:08:02.189 INFO:tasks.workunit.client.0.smithi029.stdout: 10 58999 13.15 MB/sec execute 106 sec latency 1629.569 ms 2024-03-16T12:08:02.786 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:08:02 smithi154 ceph-mon[36320]: pgmap v542: 97 pgs: 97 active+clean; 373 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 3.8 MiB/s wr, 424 op/s 2024-03-16T12:08:02.829 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:08:02 smithi029 ceph-mon[31975]: pgmap v542: 97 pgs: 97 active+clean; 373 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 3.8 MiB/s wr, 424 op/s 2024-03-16T12:08:02.868 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:08:02 smithi152 ceph-mon[36291]: pgmap v542: 97 pgs: 97 active+clean; 373 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 3.8 MiB/s wr, 424 op/s 2024-03-16T12:08:03.189 INFO:tasks.workunit.client.0.smithi029.stdout: 10 58999 13.03 MB/sec execute 107 sec latency 2629.624 ms 2024-03-16T12:08:04.189 INFO:tasks.workunit.client.0.smithi029.stdout: 10 58999 12.91 MB/sec execute 108 sec latency 3629.730 ms 2024-03-16T12:08:04.786 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:08:04 smithi154 ceph-mon[36320]: pgmap v543: 97 pgs: 97 active+clean; 365 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 5.4 MiB/s wr, 545 op/s 2024-03-16T12:08:04.828 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:08:04 smithi029 ceph-mon[31975]: pgmap v543: 97 pgs: 97 active+clean; 365 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 5.4 MiB/s wr, 545 op/s 2024-03-16T12:08:04.868 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:08:04 smithi152 ceph-mon[36291]: pgmap v543: 97 pgs: 97 active+clean; 365 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 5.4 MiB/s wr, 545 op/s 2024-03-16T12:08:05.189 INFO:tasks.workunit.client.0.smithi029.stdout: 10 59200 12.89 MB/sec execute 109 sec latency 4226.020 ms 2024-03-16T12:08:05.189 INFO:tasks.workunit.client.0.smithi029.stdout:[58052] write failed on handle 13848 (Resource temporarily unavailable) 2024-03-16T12:08:05.189 INFO:tasks.workunit.client.0.smithi029.stdout:Child failed with status 1 2024-03-16T12:08:05.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-16T12:08:05.208 INFO:tasks.workunit:Stopping ['suites/dbench.sh'] on client.0... 2024-03-16T12:08:05.208 DEBUG:teuthology.orchestra.run.smithi029:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-03-16T12:08:05.488 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/git.ceph.com_ceph-c_3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/qa/tasks/workunit.py", line 295, in _spawn_on_all_clients p.spawn(_run_tests, ctx, refspec, role, [unit], env, File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test suites/dbench.sh) on smithi029 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=3d78f5ba07bf18880172c07d4d7b4ac00c9d30af TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh' 2024-03-16T12:08:05.731 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=6d3a9426499e4fb39f7dd60177d4bf93 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/git.ceph.com_ceph-c_3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/qa/tasks/workunit.py", line 295, in _spawn_on_all_clients p.spawn(_run_tests, ctx, refspec, role, [unit], env, File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test suites/dbench.sh) on smithi029 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=3d78f5ba07bf18880172c07d4d7b4ac00c9d30af TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh' 2024-03-16T12:08:05.734 DEBUG:teuthology.run_tasks:Unwinding manager background_exec 2024-03-16T12:08:05.746 INFO:teuthology.task.background_exec:Stopping background command on ubuntu@smithi029.front.sepia.ceph.com 2024-03-16T12:08:05.760 DEBUG:teuthology.run_tasks:Unwinding manager full_sequential_finally 2024-03-16T12:08:05.770 INFO:teuthology.task.full_sequential_finally:In full_sequential_finally, running task exec... 2024-03-16T12:08:05.770 INFO:teuthology.task.exec:Executing custom commands... 2024-03-16T12:08:05.771 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi029.front.sepia.ceph.com 2024-03-16T12:08:05.771 DEBUG:teuthology.orchestra.run.smithi029:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'date +%s > END_TIME' 2024-03-16T12:08:05.829 DEBUG:teuthology.orchestra.run.smithi029:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'START_TIME=$(cat START_TIME); END_TIME=$(cat END_TIME); DIFF_TIME=$((600-(END_TIME-START_TIME))); if [ "$DIFF_TIME" -gt 0 ]; then sleep "$DIFF_TIME"; fi' 2024-03-16T12:08:05.899 DEBUG:teuthology.orchestra.run.smithi029:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule status --fs=cephfs --path=/' 2024-03-16T12:08:06.047 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:06.054+0000 7f6feeb77640 1 -- 172.21.15.29:0/4169592708 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6fe8101790 msgr2=0x7f6fe8101c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:08:06.048 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:06.054+0000 7f6feeb77640 1 --2- 172.21.15.29:0/4169592708 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6fe8101790 0x7f6fe8101c10 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f6fd4009930 tx=0x7f6fd402bc30 comp rx=0 tx=0).stop 2024-03-16T12:08:06.048 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:06.054+0000 7f6feeb77640 1 -- 172.21.15.29:0/4169592708 shutdown_connections 2024-03-16T12:08:06.048 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:06.054+0000 7f6feeb77640 1 --2- 172.21.15.29:0/4169592708 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6fe8102150 0x7f6fe810e660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:08:06.048 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:06.054+0000 7f6feeb77640 1 --2- 172.21.15.29:0/4169592708 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6fe8101790 0x7f6fe8101c10 unknown :-1 s=CLOSED pgs=785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:08:06.049 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:06.054+0000 7f6feeb77640 1 --2- 172.21.15.29:0/4169592708 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6fe81077a0 0x7f6fe8107b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:08:06.049 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:06.054+0000 7f6feeb77640 1 -- 172.21.15.29:0/4169592708 >> 172.21.15.29:0/4169592708 conn(0x7f6fe80fd2a0 msgr2=0x7f6fe80ff6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:08:06.049 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:06.054+0000 7f6feeb77640 1 -- 172.21.15.29:0/4169592708 shutdown_connections 2024-03-16T12:08:06.049 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:06.054+0000 7f6feeb77640 1 -- 172.21.15.29:0/4169592708 wait complete. 2024-03-16T12:08:06.049 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:06.055+0000 7f6feeb77640 1 Processor -- start 2024-03-16T12:08:06.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:06.055+0000 7f6feeb77640 1 -- start start 2024-03-16T12:08:06.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:06.055+0000 7f6feeb77640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6fe8101790 0x7f6fe8198050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:08:06.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:06.055+0000 7f6feeb77640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6fe8102150 0x7f6fe8198590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:08:06.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:06.055+0000 7f6feeb77640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6fe81077a0 0x7f6fe819c910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:08:06.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:06.055+0000 7f6feeb77640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f6fe8111bb0 con 0x7f6fe8102150 2024-03-16T12:08:06.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:06.055+0000 7f6feeb77640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f6fe8111a30 con 0x7f6fe8101790 2024-03-16T12:08:06.051 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:06.055+0000 7f6feeb77640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f6fe8111d30 con 0x7f6fe81077a0 2024-03-16T12:08:06.051 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:06.055+0000 7f6fe7fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6fe8102150 0x7f6fe8198590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:08:06.051 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:06.055+0000 7f6fe7fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6fe8102150 0x7f6fe8198590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57806/0 (socket says 172.21.15.29:57806) 2024-03-16T12:08:06.051 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:06.055+0000 7f6fec913640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6fe8101790 0x7f6fe8198050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:08:06.051 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:06.055+0000 7f6fe7fff640 1 -- 172.21.15.29:0/2756478090 learned_addr learned my addr 172.21.15.29:0/2756478090 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T12:08:06.051 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:06.055+0000 7f6fed114640 1 --2- 172.21.15.29:0/2756478090 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6fe81077a0 0x7f6fe819c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:08:06.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:06.055+0000 7f6fe7fff640 1 -- 172.21.15.29:0/2756478090 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6fe81077a0 msgr2=0x7f6fe819c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:08:06.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:06.055+0000 7f6fe7fff640 1 --2- 172.21.15.29:0/2756478090 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6fe81077a0 0x7f6fe819c910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:08:06.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:06.055+0000 7f6fe7fff640 1 -- 172.21.15.29:0/2756478090 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6fe8101790 msgr2=0x7f6fe8198050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:08:06.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:06.055+0000 7f6fe7fff640 1 --2- 172.21.15.29:0/2756478090 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6fe8101790 0x7f6fe8198050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:08:06.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:06.055+0000 7f6fe7fff640 1 -- 172.21.15.29:0/2756478090 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6fe819d040 con 0x7f6fe8102150 2024-03-16T12:08:06.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:06.055+0000 7f6fec913640 1 --2- 172.21.15.29:0/2756478090 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6fe8101790 0x7f6fe8198050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:08:06.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:06.056+0000 7f6fe7fff640 1 --2- 172.21.15.29:0/2756478090 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6fe8102150 0x7f6fe8198590 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f6fd4009230 tx=0x7f6fd402cb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:08:06.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:06.056+0000 7f6fe5ffb640 1 -- 172.21.15.29:0/2756478090 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fd4041070 con 0x7f6fe8102150 2024-03-16T12:08:06.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:06.056+0000 7f6feeb77640 1 -- 172.21.15.29:0/2756478090 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6fe819d2d0 con 0x7f6fe8102150 2024-03-16T12:08:06.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:06.056+0000 7f6fe5ffb640 1 -- 172.21.15.29:0/2756478090 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6fd4004450 con 0x7f6fe8102150 2024-03-16T12:08:06.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:06.056+0000 7f6fe5ffb640 1 -- 172.21.15.29:0/2756478090 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fd4003e50 con 0x7f6fe8102150 2024-03-16T12:08:06.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:06.056+0000 7f6feeb77640 1 -- 172.21.15.29:0/2756478090 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6fe81a4bc0 con 0x7f6fe8102150 2024-03-16T12:08:06.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:06.057+0000 7f6feeb77640 1 -- 172.21.15.29:0/2756478090 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6fe804f540 con 0x7f6fe8102150 2024-03-16T12:08:06.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:06.059+0000 7f6fe5ffb640 1 -- 172.21.15.29:0/2756478090 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7f6fd4044070 con 0x7f6fe8102150 2024-03-16T12:08:06.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:06.059+0000 7f6fe5ffb640 1 --2- 172.21.15.29:0/2756478090 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f6fd80a2bb0 0x7f6fd80a5070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:08:06.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:06.060+0000 7f6fe5ffb640 1 -- 172.21.15.29:0/2756478090 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(114..114 src has 1..114) ==== 8556+0+0 (secure 0 0 0) 0x7f6fd40f0aa0 con 0x7f6fe8102150 2024-03-16T12:08:06.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:06.060+0000 7f6fec913640 1 --2- 172.21.15.29:0/2756478090 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f6fd80a2bb0 0x7f6fd80a5070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:08:06.055 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:06.060+0000 7f6fec913640 1 --2- 172.21.15.29:0/2756478090 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f6fd80a2bb0 0x7f6fd80a5070 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f6fdc009f40 tx=0x7f6fdc005f90 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:08:06.055 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:06.063+0000 7f6fe5ffb640 1 -- 172.21.15.29:0/2756478090 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f6fd40bd240 con 0x7f6fe8102150 2024-03-16T12:08:06.259 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:06.269+0000 7f6feeb77640 1 -- 172.21.15.29:0/2756478090 --> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}) -- 0x7f6fe8061c20 con 0x7f6fd80a2bb0 2024-03-16T12:08:06.268 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:06.278+0000 7f6fe5ffb640 1 -- 172.21.15.29:0/2756478090 <== mgr.15333 v2:172.21.15.29:6800/1188678802 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+351 (secure 0 0 0) 0x7f6fe8061c20 con 0x7f6fd80a2bb0 2024-03-16T12:08:06.269 INFO:teuthology.orchestra.run.smithi029.stdout:{"fs": "cephfs", "subvol": null, "group": null, "path": "/", "rel_path": "/", "schedule": "1m", "retention": {"m": 6, "h": 3}, "start": "2024-03-16T00:00:00", "created": "2024-03-16T11:50:55", "first": "2024-03-16T11:51:00", "last": "2024-03-16T12:08:00", "last_pruned": "2024-03-16T12:08:04", "created_count": 18, "pruned_count": 11, "active": true} 2024-03-16T12:08:06.271 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:06.280+0000 7f6feeb77640 1 -- 172.21.15.29:0/2756478090 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f6fd80a2bb0 msgr2=0x7f6fd80a5070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:08:06.271 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:06.280+0000 7f6feeb77640 1 --2- 172.21.15.29:0/2756478090 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f6fd80a2bb0 0x7f6fd80a5070 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f6fdc009f40 tx=0x7f6fdc005f90 comp rx=0 tx=0).stop 2024-03-16T12:08:06.271 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:06.280+0000 7f6feeb77640 1 -- 172.21.15.29:0/2756478090 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6fe8102150 msgr2=0x7f6fe8198590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:08:06.271 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:06.280+0000 7f6feeb77640 1 --2- 172.21.15.29:0/2756478090 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6fe8102150 0x7f6fe8198590 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f6fd4009230 tx=0x7f6fd402cb30 comp rx=0 tx=0).stop 2024-03-16T12:08:06.271 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:06.281+0000 7f6feeb77640 1 -- 172.21.15.29:0/2756478090 shutdown_connections 2024-03-16T12:08:06.272 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:06.281+0000 7f6feeb77640 1 --2- 172.21.15.29:0/2756478090 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f6fd80a2bb0 0x7f6fd80a5070 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:08:06.272 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:06.281+0000 7f6feeb77640 1 --2- 172.21.15.29:0/2756478090 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6fe81077a0 0x7f6fe819c910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:08:06.272 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:06.281+0000 7f6feeb77640 1 --2- 172.21.15.29:0/2756478090 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6fe8102150 0x7f6fe8198590 unknown :-1 s=CLOSED pgs=786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:08:06.272 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:06.281+0000 7f6feeb77640 1 --2- 172.21.15.29:0/2756478090 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6fe8101790 0x7f6fe8198050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:08:06.272 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:06.281+0000 7f6feeb77640 1 -- 172.21.15.29:0/2756478090 >> 172.21.15.29:0/2756478090 conn(0x7f6fe80fd2a0 msgr2=0x7f6fe80fd800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:08:06.272 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:06.281+0000 7f6feeb77640 1 -- 172.21.15.29:0/2756478090 shutdown_connections 2024-03-16T12:08:06.272 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:06.281+0000 7f6feeb77640 1 -- 172.21.15.29:0/2756478090 wait complete. 2024-03-16T12:08:06.282 DEBUG:teuthology.orchestra.run.smithi029:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true' 2024-03-16T12:08:06.390 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:06.397+0000 7fcff59b9640 1 -- 172.21.15.29:0/1362029284 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcff0105680 msgr2=0x7fcff0105a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:08:06.390 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:06.397+0000 7fcff59b9640 1 --2- 172.21.15.29:0/1362029284 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcff0105680 0x7fcff0105a60 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7fcfe0009900 tx=0x7fcfe002bc30 comp rx=0 tx=0).stop 2024-03-16T12:08:06.390 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:06.397+0000 7fcff59b9640 1 -- 172.21.15.29:0/1362029284 shutdown_connections 2024-03-16T12:08:06.390 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:06.397+0000 7fcff59b9640 1 --2- 172.21.15.29:0/1362029284 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcff01012f0 0x7fcff010d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:08:06.391 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:06.397+0000 7fcff59b9640 1 --2- 172.21.15.29:0/1362029284 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fcff0100930 0x7fcff0100db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:08:06.391 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:06.397+0000 7fcff59b9640 1 --2- 172.21.15.29:0/1362029284 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcff0105680 0x7fcff0105a60 unknown :-1 s=CLOSED pgs=787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:08:06.391 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:06.397+0000 7fcff59b9640 1 -- 172.21.15.29:0/1362029284 >> 172.21.15.29:0/1362029284 conn(0x7fcff0077530 msgr2=0x7fcff0077940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:08:06.391 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:06.397+0000 7fcff59b9640 1 -- 172.21.15.29:0/1362029284 shutdown_connections 2024-03-16T12:08:06.391 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:06.397+0000 7fcff59b9640 1 -- 172.21.15.29:0/1362029284 wait complete. 2024-03-16T12:08:06.391 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:06.397+0000 7fcff59b9640 1 Processor -- start 2024-03-16T12:08:06.391 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:06.398+0000 7fcff59b9640 1 -- start start 2024-03-16T12:08:06.391 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:06.398+0000 7fcff59b9640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fcff0100930 0x7fcff0196f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:08:06.391 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:06.398+0000 7fcff59b9640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcff01012f0 0x7fcff0197480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:08:06.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:06.398+0000 7fcff59b9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcff0105680 0x7fcff019b800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:08:06.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:06.398+0000 7fcff59b9640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fcff0110ba0 con 0x7fcff0105680 2024-03-16T12:08:06.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:06.398+0000 7fcff59b9640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fcff0110a20 con 0x7fcff0100930 2024-03-16T12:08:06.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:06.398+0000 7fcff59b9640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fcff0110d20 con 0x7fcff01012f0 2024-03-16T12:08:06.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:06.398+0000 7fcfef7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcff0105680 0x7fcff019b800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:08:06.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:06.398+0000 7fcfef7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcff0105680 0x7fcff019b800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57832/0 (socket says 172.21.15.29:57832) 2024-03-16T12:08:06.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:06.398+0000 7fcfef7fe640 1 -- 172.21.15.29:0/473058639 learned_addr learned my addr 172.21.15.29:0/473058639 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T12:08:06.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:06.398+0000 7fcfeeffd640 1 --2- 172.21.15.29:0/473058639 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fcff0100930 0x7fcff0196f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:08:06.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:06.398+0000 7fcfe7fff640 1 --2- 172.21.15.29:0/473058639 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcff01012f0 0x7fcff0197480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:08:06.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:06.398+0000 7fcfef7fe640 1 -- 172.21.15.29:0/473058639 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcff01012f0 msgr2=0x7fcff0197480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:08:06.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:06.398+0000 7fcfef7fe640 1 --2- 172.21.15.29:0/473058639 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcff01012f0 0x7fcff0197480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:08:06.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:06.398+0000 7fcfef7fe640 1 -- 172.21.15.29:0/473058639 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fcff0100930 msgr2=0x7fcff0196f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:08:06.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:06.398+0000 7fcfef7fe640 1 --2- 172.21.15.29:0/473058639 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fcff0100930 0x7fcff0196f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:08:06.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:06.398+0000 7fcfef7fe640 1 -- 172.21.15.29:0/473058639 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcff019bf30 con 0x7fcff0105680 2024-03-16T12:08:06.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:06.398+0000 7fcfeeffd640 1 --2- 172.21.15.29:0/473058639 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fcff0100930 0x7fcff0196f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:08:06.394 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:06.398+0000 7fcfef7fe640 1 --2- 172.21.15.29:0/473058639 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcff0105680 0x7fcff019b800 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7fcfe800ea60 tx=0x7fcfe800ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:08:06.394 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:06.398+0000 7fcff49b7640 1 -- 172.21.15.29:0/473058639 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfe800ce40 con 0x7fcff0105680 2024-03-16T12:08:06.394 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:06.398+0000 7fcff49b7640 1 -- 172.21.15.29:0/473058639 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcfe8005020 con 0x7fcff0105680 2024-03-16T12:08:06.394 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:06.398+0000 7fcff59b9640 1 -- 172.21.15.29:0/473058639 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcff019c220 con 0x7fcff0105680 2024-03-16T12:08:06.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:06.399+0000 7fcff49b7640 1 -- 172.21.15.29:0/473058639 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfe8010690 con 0x7fcff0105680 2024-03-16T12:08:06.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:06.399+0000 7fcff59b9640 1 -- 172.21.15.29:0/473058639 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcff01a3ab0 con 0x7fcff0105680 2024-03-16T12:08:06.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:06.400+0000 7fcff59b9640 1 -- 172.21.15.29:0/473058639 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcff004f540 con 0x7fcff0105680 2024-03-16T12:08:06.396 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:06.401+0000 7fcff49b7640 1 -- 172.21.15.29:0/473058639 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7fcfe8005520 con 0x7fcff0105680 2024-03-16T12:08:06.396 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:06.401+0000 7fcff49b7640 1 --2- 172.21.15.29:0/473058639 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7fcfdc0a2a10 0x7fcfdc0a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:08:06.396 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:06.401+0000 7fcff49b7640 1 -- 172.21.15.29:0/473058639 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(114..114 src has 1..114) ==== 8556+0+0 (secure 0 0 0) 0x7fcfe80cf540 con 0x7fcff0105680 2024-03-16T12:08:06.396 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:06.401+0000 7fcfeeffd640 1 --2- 172.21.15.29:0/473058639 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7fcfdc0a2a10 0x7fcfdc0a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:08:06.396 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:06.402+0000 7fcfeeffd640 1 --2- 172.21.15.29:0/473058639 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7fcfdc0a2a10 0x7fcfdc0a4ed0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fcfe0009230 tx=0x7fcfe0005fd0 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:08:06.396 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:06.405+0000 7fcff49b7640 1 -- 172.21.15.29:0/473058639 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fcfe8053740 con 0x7fcff0105680 2024-03-16T12:08:06.578 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:08:06 smithi029 ceph-mon[31975]: pgmap v544: 97 pgs: 97 active+clean; 365 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 4.0 MiB/s wr, 357 op/s 2024-03-16T12:08:06.606 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:06.616+0000 7fcff59b9640 1 -- 172.21.15.29:0/473058639 --> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}) -- 0x7fcff00fefb0 con 0x7fcfdc0a2a10 2024-03-16T12:08:06.615 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:06.625+0000 7fcff49b7640 1 -- 172.21.15.29:0/473058639 <== mgr.15333 v2:172.21.15.29:6800/1188678802 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+10 (secure 0 0 0) 0x7fcff00fefb0 con 0x7fcfdc0a2a10 2024-03-16T12:08:06.615 INFO:teuthology.orchestra.run.smithi029.stdout:/ 1m 6m3h 2024-03-16T12:08:06.618 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:06.627+0000 7fcff59b9640 1 -- 172.21.15.29:0/473058639 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7fcfdc0a2a10 msgr2=0x7fcfdc0a4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:08:06.619 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:06.627+0000 7fcff59b9640 1 --2- 172.21.15.29:0/473058639 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7fcfdc0a2a10 0x7fcfdc0a4ed0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fcfe0009230 tx=0x7fcfe0005fd0 comp rx=0 tx=0).stop 2024-03-16T12:08:06.619 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:06.627+0000 7fcff59b9640 1 -- 172.21.15.29:0/473058639 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcff0105680 msgr2=0x7fcff019b800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:08:06.619 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:06.627+0000 7fcff59b9640 1 --2- 172.21.15.29:0/473058639 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcff0105680 0x7fcff019b800 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7fcfe800ea60 tx=0x7fcfe800ef20 comp rx=0 tx=0).stop 2024-03-16T12:08:06.620 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:06.627+0000 7fcff59b9640 1 -- 172.21.15.29:0/473058639 shutdown_connections 2024-03-16T12:08:06.620 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:06.627+0000 7fcff59b9640 1 --2- 172.21.15.29:0/473058639 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7fcfdc0a2a10 0x7fcfdc0a4ed0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:08:06.620 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:06.627+0000 7fcff59b9640 1 --2- 172.21.15.29:0/473058639 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcff0105680 0x7fcff019b800 unknown :-1 s=CLOSED pgs=788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:08:06.621 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:06.627+0000 7fcff59b9640 1 --2- 172.21.15.29:0/473058639 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcff01012f0 0x7fcff0197480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:08:06.621 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:06.627+0000 7fcff59b9640 1 --2- 172.21.15.29:0/473058639 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fcff0100930 0x7fcff0196f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:08:06.621 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:06.627+0000 7fcff59b9640 1 -- 172.21.15.29:0/473058639 >> 172.21.15.29:0/473058639 conn(0x7fcff0077530 msgr2=0x7fcff00fe8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:08:06.621 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:06.627+0000 7fcff59b9640 1 -- 172.21.15.29:0/473058639 shutdown_connections 2024-03-16T12:08:06.621 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:06.627+0000 7fcff59b9640 1 -- 172.21.15.29:0/473058639 wait complete. 2024-03-16T12:08:06.629 DEBUG:teuthology.run_tasks:Unwinding manager check-counter 2024-03-16T12:08:06.639 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-16T12:08:06.745 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:06.752+0000 7febe95c6640 1 -- 172.21.15.29:0/3171337885 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7febe41017a0 msgr2=0x7febe4101c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:08:06.745 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:06.752+0000 7febe95c6640 1 --2- 172.21.15.29:0/3171337885 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7febe41017a0 0x7febe4101c20 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7febd4009930 tx=0x7febd402bc30 comp rx=0 tx=0).stop 2024-03-16T12:08:06.745 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:06.752+0000 7febe95c6640 1 -- 172.21.15.29:0/3171337885 shutdown_connections 2024-03-16T12:08:06.745 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:06.752+0000 7febe95c6640 1 --2- 172.21.15.29:0/3171337885 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7febe4102160 0x7febe410e670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:08:06.745 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:06.752+0000 7febe95c6640 1 --2- 172.21.15.29:0/3171337885 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7febe41017a0 0x7febe4101c20 unknown :-1 s=CLOSED pgs=789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:08:06.745 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:06.752+0000 7febe95c6640 1 --2- 172.21.15.29:0/3171337885 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7febe41077b0 0x7febe4107b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:08:06.746 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:06.752+0000 7febe95c6640 1 -- 172.21.15.29:0/3171337885 >> 172.21.15.29:0/3171337885 conn(0x7febe40fd250 msgr2=0x7febe40ff670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:08:06.746 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:06.752+0000 7febe95c6640 1 -- 172.21.15.29:0/3171337885 shutdown_connections 2024-03-16T12:08:06.746 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:06.752+0000 7febe95c6640 1 -- 172.21.15.29:0/3171337885 wait complete. 2024-03-16T12:08:06.746 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:06.753+0000 7febe95c6640 1 Processor -- start 2024-03-16T12:08:06.746 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:06.753+0000 7febe95c6640 1 -- start start 2024-03-16T12:08:06.746 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:06.753+0000 7febe95c6640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7febe41017a0 0x7febe4073ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:08:06.746 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:06.753+0000 7febe95c6640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7febe4102160 0x7febe4074020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:08:06.746 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:06.753+0000 7febe95c6640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7febe41077b0 0x7febe4074560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:08:06.747 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:06.753+0000 7febe95c6640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7febe4111bc0 con 0x7febe41017a0 2024-03-16T12:08:06.747 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:06.753+0000 7febe95c6640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7febe4111a40 con 0x7febe41077b0 2024-03-16T12:08:06.747 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:06.753+0000 7febe95c6640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7febe4111d40 con 0x7febe4102160 2024-03-16T12:08:06.747 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:06.753+0000 7febe2ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7febe41017a0 0x7febe4073ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:08:06.747 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:06.753+0000 7febe2ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7febe41017a0 0x7febe4073ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57848/0 (socket says 172.21.15.29:57848) 2024-03-16T12:08:06.747 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:06.753+0000 7febe2ffd640 1 -- 172.21.15.29:0/681775310 learned_addr learned my addr 172.21.15.29:0/681775310 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T12:08:06.747 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:06.753+0000 7febe37fe640 1 --2- 172.21.15.29:0/681775310 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7febe41077b0 0x7febe4074560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:08:06.748 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:06.753+0000 7febe27fc640 1 --2- 172.21.15.29:0/681775310 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7febe4102160 0x7febe4074020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:08:06.748 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:06.754+0000 7febe2ffd640 1 -- 172.21.15.29:0/681775310 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7febe4102160 msgr2=0x7febe4074020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:08:06.748 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:06.754+0000 7febe2ffd640 1 --2- 172.21.15.29:0/681775310 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7febe4102160 0x7febe4074020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:08:06.748 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:06.754+0000 7febe2ffd640 1 -- 172.21.15.29:0/681775310 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7febe41077b0 msgr2=0x7febe4074560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:08:06.748 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:06.754+0000 7febe2ffd640 1 --2- 172.21.15.29:0/681775310 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7febe41077b0 0x7febe4074560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:08:06.748 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:06.754+0000 7febe2ffd640 1 -- 172.21.15.29:0/681775310 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7febe41a6580 con 0x7febe41017a0 2024-03-16T12:08:06.748 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:06.754+0000 7febe2ffd640 1 --2- 172.21.15.29:0/681775310 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7febe41017a0 0x7febe4073ae0 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7febd800e9f0 tx=0x7febd800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:08:06.748 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:06.754+0000 7febcbfff640 1 -- 172.21.15.29:0/681775310 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febd800cd80 con 0x7febe41017a0 2024-03-16T12:08:06.748 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:06.754+0000 7febe95c6640 1 -- 172.21.15.29:0/681775310 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7febe41a6870 con 0x7febe41017a0 2024-03-16T12:08:06.749 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:06.754+0000 7febcbfff640 1 -- 172.21.15.29:0/681775310 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7febd8010910 con 0x7febe41017a0 2024-03-16T12:08:06.749 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:06.754+0000 7febcbfff640 1 -- 172.21.15.29:0/681775310 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febd8018990 con 0x7febe41017a0 2024-03-16T12:08:06.749 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:06.754+0000 7febe95c6640 1 -- 172.21.15.29:0/681775310 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7febe41a6d90 con 0x7febe41017a0 2024-03-16T12:08:06.750 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:06.755+0000 7febe95c6640 1 -- 172.21.15.29:0/681775310 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7febe4068a20 con 0x7febe41017a0 2024-03-16T12:08:06.750 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:06.756+0000 7febcbfff640 1 -- 172.21.15.29:0/681775310 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7febd8010ac0 con 0x7febe41017a0 2024-03-16T12:08:06.750 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:06.757+0000 7febcbfff640 1 --2- 172.21.15.29:0/681775310 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7febcc0a2970 0x7febcc0a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:08:06.750 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:06.757+0000 7febe27fc640 1 --2- 172.21.15.29:0/681775310 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7febcc0a2970 0x7febcc0a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:08:06.750 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:06.757+0000 7febcbfff640 1 -- 172.21.15.29:0/681775310 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(114..114 src has 1..114) ==== 8556+0+0 (secure 0 0 0) 0x7febd80cf5b0 con 0x7febe41017a0 2024-03-16T12:08:06.750 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:06.757+0000 7febe27fc640 1 --2- 172.21.15.29:0/681775310 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7febcc0a2970 0x7febcc0a4e30 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7febd4005ab0 tx=0x7febd4005fd0 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:08:06.751 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:06.760+0000 7febcbfff640 1 -- 172.21.15.29:0/681775310 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7febd809bc40 con 0x7febe41017a0 2024-03-16T12:08:06.786 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:08:06 smithi154 ceph-mon[36320]: pgmap v544: 97 pgs: 97 active+clean; 365 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 4.0 MiB/s wr, 357 op/s 2024-03-16T12:08:06.868 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:08:06 smithi152 ceph-mon[36291]: pgmap v544: 97 pgs: 97 active+clean; 365 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 4.0 MiB/s wr, 357 op/s 2024-03-16T12:08:06.937 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:06.948+0000 7febe95c6640 1 -- 172.21.15.29:0/681775310 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7febe4109d10 con 0x7febe41017a0 2024-03-16T12:08:06.939 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:06.949+0000 7febcbfff640 1 -- 172.21.15.29:0/681775310 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v114) ==== 74+0+23200 (secure 0 0 0) 0x7febd809fce0 con 0x7febe41017a0 2024-03-16T12:08:06.940 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-16T12:08:06.940 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":114,"fsid":"d1887f28-e389-11ee-95c9-87774f69a715","created":"2024-03-16T11:40:16.475063+0000","modified":"2024-03-16T12:07:22.163804+0000","last_up_change":"2024-03-16T11:47:44.349348+0000","last_in_change":"2024-03-16T11:47:23.672432+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-16T11:43:51.892652+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-16T11:48:36.709732+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-16T11:48:37.586140+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"113","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":28,"snap_epoch":113,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"60f5608f-a408-45ba-bdf6-5b53758f88fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":2748947913},{"type":"v1","addr":"172.21.15.29:6803","nonce":2748947913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":2748947913},{"type":"v1","addr":"172.21.15.29:6805","nonce":2748947913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":2748947913},{"type":"v1","addr":"172.21.15.29:6809","nonce":2748947913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":2748947913},{"type":"v1","addr":"172.21.15.29:6807","nonce":2748947913}]},"public_addr":"172.21.15.29:6803/2748947913","cluster_addr":"172.21.15.29:6805/2748947913","heartbeat_back_addr":"172.21.15.29:6809/2748947913","heartbeat_front_addr":"172.21.15.29:6807/2748947913","state":["exists","up"]},{"osd":1,"uuid":"e70b53d6-ee78-4e56-9a2c-263ae0501b43","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6800","nonce":2339408422},{"type":"v1","addr":"172.21.15.152:6801","nonce":2339408422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6802","nonce":2339408422},{"type":"v1","addr":"172.21.15.152:6803","nonce":2339408422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6806","nonce":2339408422},{"type":"v1","addr":"172.21.15.152:6807","nonce":2339408422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6804","nonce":2339408422},{"type":"v1","addr":"172.21.15.152:6805","nonce":2339408422}]},"public_addr":"172.21.15.152:6801/2339408422","cluster_addr":"172.21.15.152:6803/2339408422","heartbeat_back_addr":"172.21.15.152:6807/2339408422","heartbeat_front_addr":"172.21.15.152:6805/2339408422","state":["exists","up"]},{"osd":2,"uuid":"ff39389a-dfac-44b5-80ff-36abca316e78","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6800","nonce":4288325280},{"type":"v1","addr":"172.21.15.154:6801","nonce":4288325280}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6802","nonce":4288325280},{"type":"v1","addr":"172.21.15.154:6803","nonce":4288325280}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6806","nonce":4288325280},{"type":"v1","addr":"172.21.15.154:6807","nonce":4288325280}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":4288325280},{"type":"v1","addr":"172.21.15.154:6805","nonce":4288325280}]},"public_addr":"172.21.15.154:6801/4288325280","cluster_addr":"172.21.15.154:6803/4288325280","heartbeat_back_addr":"172.21.15.154:6807/4288325280","heartbeat_front_addr":"172.21.15.154:6805/4288325280","state":["exists","up"]},{"osd":3,"uuid":"069e1795-55bc-4324-b94c-eb24ef2334c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":1538750397},{"type":"v1","addr":"172.21.15.29:6811","nonce":1538750397}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":1538750397},{"type":"v1","addr":"172.21.15.29:6813","nonce":1538750397}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":1538750397},{"type":"v1","addr":"172.21.15.29:6817","nonce":1538750397}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":1538750397},{"type":"v1","addr":"172.21.15.29:6815","nonce":1538750397}]},"public_addr":"172.21.15.29:6811/1538750397","cluster_addr":"172.21.15.29:6813/1538750397","heartbeat_back_addr":"172.21.15.29:6817/1538750397","heartbeat_front_addr":"172.21.15.29:6815/1538750397","state":["exists","up"]},{"osd":4,"uuid":"5a9d43a1-e8f5-4ab5-a441-afbc6ed4066e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6808","nonce":2795081750},{"type":"v1","addr":"172.21.15.152:6809","nonce":2795081750}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6810","nonce":2795081750},{"type":"v1","addr":"172.21.15.152:6811","nonce":2795081750}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6814","nonce":2795081750},{"type":"v1","addr":"172.21.15.152:6815","nonce":2795081750}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6812","nonce":2795081750},{"type":"v1","addr":"172.21.15.152:6813","nonce":2795081750}]},"public_addr":"172.21.15.152:6809/2795081750","cluster_addr":"172.21.15.152:6811/2795081750","heartbeat_back_addr":"172.21.15.152:6815/2795081750","heartbeat_front_addr":"172.21.15.152:6813/2795081750","state":["exists","up"]},{"osd":5,"uuid":"234de428-4ac7-44be-9a54-7b58331fdf8c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":1472686560},{"type":"v1","addr":"172.21.15.154:6809","nonce":1472686560}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6810","nonce":1472686560},{"type":"v1","addr":"172.21.15.154:6811","nonce":1472686560}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6814","nonce":1472686560},{"type":"v1","addr":"172.21.15.154:6815","nonce":1472686560}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6812","nonce":1472686560},{"type":"v1","addr":"172.21.15.154:6813","nonce":1472686560}]},"public_addr":"172.21.15.154:6809/1472686560","cluster_addr":"172.21.15.154:6811/1472686560","heartbeat_back_addr":"172.21.15.154:6815/1472686560","heartbeat_front_addr":"172.21.15.154:6813/1472686560","state":["exists","up"]},{"osd":6,"uuid":"ae45b8d9-7883-40c9-a9cd-4c92b601d4f2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":816506244},{"type":"v1","addr":"172.21.15.29:6819","nonce":816506244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":816506244},{"type":"v1","addr":"172.21.15.29:6821","nonce":816506244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":816506244},{"type":"v1","addr":"172.21.15.29:6825","nonce":816506244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":816506244},{"type":"v1","addr":"172.21.15.29:6823","nonce":816506244}]},"public_addr":"172.21.15.29:6819/816506244","cluster_addr":"172.21.15.29:6821/816506244","heartbeat_back_addr":"172.21.15.29:6825/816506244","heartbeat_front_addr":"172.21.15.29:6823/816506244","state":["exists","up"]},{"osd":7,"uuid":"6f687b28-4fce-4d4a-8b4e-294173d9d6c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6816","nonce":2924651787},{"type":"v1","addr":"172.21.15.152:6817","nonce":2924651787}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6818","nonce":2924651787},{"type":"v1","addr":"172.21.15.152:6819","nonce":2924651787}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6822","nonce":2924651787},{"type":"v1","addr":"172.21.15.152:6823","nonce":2924651787}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6820","nonce":2924651787},{"type":"v1","addr":"172.21.15.152:6821","nonce":2924651787}]},"public_addr":"172.21.15.152:6817/2924651787","cluster_addr":"172.21.15.152:6819/2924651787","heartbeat_back_addr":"172.21.15.152:6823/2924651787","heartbeat_front_addr":"172.21.15.152:6821/2924651787","state":["exists","up"]},{"osd":8,"uuid":"a25fb741-caa7-429a-b540-e394d034b446","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6816","nonce":129974156},{"type":"v1","addr":"172.21.15.154:6817","nonce":129974156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6818","nonce":129974156},{"type":"v1","addr":"172.21.15.154:6819","nonce":129974156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6822","nonce":129974156},{"type":"v1","addr":"172.21.15.154:6823","nonce":129974156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6820","nonce":129974156},{"type":"v1","addr":"172.21.15.154:6821","nonce":129974156}]},"public_addr":"172.21.15.154:6817/129974156","cluster_addr":"172.21.15.154:6819/129974156","heartbeat_back_addr":"172.21.15.154:6823/129974156","heartbeat_front_addr":"172.21.15.154:6821/129974156","state":["exists","up"]},{"osd":9,"uuid":"1071cb0e-8c5f-4cd8-9805-0045e220d946","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6826","nonce":2951414243},{"type":"v1","addr":"172.21.15.29:6827","nonce":2951414243}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":2951414243},{"type":"v1","addr":"172.21.15.29:6829","nonce":2951414243}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":2951414243},{"type":"v1","addr":"172.21.15.29:6833","nonce":2951414243}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":2951414243},{"type":"v1","addr":"172.21.15.29:6831","nonce":2951414243}]},"public_addr":"172.21.15.29:6827/2951414243","cluster_addr":"172.21.15.29:6829/2951414243","heartbeat_back_addr":"172.21.15.29:6833/2951414243","heartbeat_front_addr":"172.21.15.29:6831/2951414243","state":["exists","up"]},{"osd":10,"uuid":"ee7c7d74-3f39-4cd4-be80-843fb1dd9ecc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6824","nonce":3412525480},{"type":"v1","addr":"172.21.15.152:6825","nonce":3412525480}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6826","nonce":3412525480},{"type":"v1","addr":"172.21.15.152:6827","nonce":3412525480}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6830","nonce":3412525480},{"type":"v1","addr":"172.21.15.152:6831","nonce":3412525480}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6828","nonce":3412525480},{"type":"v1","addr":"172.21.15.152:6829","nonce":3412525480}]},"public_addr":"172.21.15.152:6825/3412525480","cluster_addr":"172.21.15.152:6827/3412525480","heartbeat_back_addr":"172.21.15.152:6831/3412525480","heartbeat_front_addr":"172.21.15.152:6829/3412525480","state":["exists","up"]},{"osd":11,"uuid":"4598c803-984a-4de6-ac22-066502c17889","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6824","nonce":1160927966},{"type":"v1","addr":"172.21.15.154:6825","nonce":1160927966}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6826","nonce":1160927966},{"type":"v1","addr":"172.21.15.154:6827","nonce":1160927966}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6830","nonce":1160927966},{"type":"v1","addr":"172.21.15.154:6831","nonce":1160927966}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6828","nonce":1160927966},{"type":"v1","addr":"172.21.15.154:6829","nonce":1160927966}]},"public_addr":"172.21.15.154:6825/1160927966","cluster_addr":"172.21.15.154:6827/1160927966","heartbeat_back_addr":"172.21.15.154:6831/1160927966","heartbeat_front_addr":"172.21.15.154:6829/1160927966","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:43:02.343393+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:43:25.209151+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:43:48.946164+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:44:14.588424+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:44:39.700025+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:45:04.061197+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:45:31.214508+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:45:56.937296+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:46:22.768915+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:46:50.101011+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:47:16.298041+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:47:42.402095+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"2.1b","mappings":[{"from":2,"to":1}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"3.33","mappings":[{"from":5,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.29:0/1361425453":"2024-03-16T12:55:48.830413+0000","172.21.15.29:0/1116486932":"2024-03-17T11:41:27.531724+0000","172.21.15.29:0/1784249039":"2024-03-17T11:41:00.505678+0000","172.21.15.29:6800/4192064640":"2024-03-17T11:50:54.850617+0000","172.21.15.29:0/1330100009":"2024-03-17T11:41:00.505678+0000","172.21.15.29:6801/2275315654":"2024-03-17T11:41:00.505678+0000","172.21.15.29:6801/2798831128":"2024-03-17T11:41:27.531724+0000","172.21.15.29:6800/2275315654":"2024-03-17T11:41:00.505678+0000","172.21.15.29:6800/2798831128":"2024-03-17T11:41:27.531724+0000","172.21.15.29:0/1211659364":"2024-03-17T11:41:27.531724+0000","172.21.15.29:0/1281028285":"2024-03-17T11:41:00.505678+0000","172.21.15.29:0/1853086886":"2024-03-17T11:41:27.531724+0000","172.21.15.29:0/623985488":"2024-03-17T11:50:54.850617+0000","172.21.15.29:6801/4192064640":"2024-03-17T11:50:54.850617+0000","172.21.15.29:0/1114950131":"2024-03-17T11:50:54.850617+0000","172.21.15.29:0/2374713213":"2024-03-17T11:50:54.850617+0000","172.21.15.29:0/340047334":"2024-03-17T11:50:54.850617+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":3,"snaps":[{"begin":16,"length":1},{"begin":28,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-16T12:08:06.942 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:06.951+0000 7febe95c6640 1 -- 172.21.15.29:0/681775310 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7febcc0a2970 msgr2=0x7febcc0a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:08:06.942 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:06.951+0000 7febe95c6640 1 --2- 172.21.15.29:0/681775310 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7febcc0a2970 0x7febcc0a4e30 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7febd4005ab0 tx=0x7febd4005fd0 comp rx=0 tx=0).stop 2024-03-16T12:08:06.942 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:06.951+0000 7febe95c6640 1 -- 172.21.15.29:0/681775310 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7febe41017a0 msgr2=0x7febe4073ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:08:06.942 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:06.951+0000 7febe95c6640 1 --2- 172.21.15.29:0/681775310 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7febe41017a0 0x7febe4073ae0 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7febd800e9f0 tx=0x7febd800eeb0 comp rx=0 tx=0).stop 2024-03-16T12:08:06.942 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:06.951+0000 7febe95c6640 1 -- 172.21.15.29:0/681775310 shutdown_connections 2024-03-16T12:08:06.943 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:06.951+0000 7febe95c6640 1 --2- 172.21.15.29:0/681775310 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7febcc0a2970 0x7febcc0a4e30 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:08:06.943 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:06.951+0000 7febe95c6640 1 --2- 172.21.15.29:0/681775310 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7febe41077b0 0x7febe4074560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:08:06.943 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:06.951+0000 7febe95c6640 1 --2- 172.21.15.29:0/681775310 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7febe4102160 0x7febe4074020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:08:06.943 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:06.951+0000 7febe95c6640 1 --2- 172.21.15.29:0/681775310 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7febe41017a0 0x7febe4073ae0 unknown :-1 s=CLOSED pgs=790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:08:06.943 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:06.951+0000 7febe95c6640 1 -- 172.21.15.29:0/681775310 >> 172.21.15.29:0/681775310 conn(0x7febe40fd250 msgr2=0x7febe40fda30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:08:06.943 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:06.951+0000 7febe95c6640 1 -- 172.21.15.29:0/681775310 shutdown_connections 2024-03-16T12:08:06.943 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:06.951+0000 7febe95c6640 1 -- 172.21.15.29:0/681775310 wait complete. 2024-03-16T12:08:06.953 INFO:tasks.check_counter.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-16T11:43:51.892652+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-16T11:48:36.709732+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-16T11:48:37.586140+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '113', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 28, 'snap_epoch': 113, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-16T12:08:06.954 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-16T12:08:07.060 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:07.069+0000 7fe68a7c4640 1 -- 172.21.15.29:0/840360328 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe68410bdf0 msgr2=0x7fe68410e220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:08:07.060 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:07.069+0000 7fe68a7c4640 1 --2- 172.21.15.29:0/840360328 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe68410bdf0 0x7fe68410e220 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7fe67c009ca0 tx=0x7fe67c02c180 comp rx=0 tx=0).stop 2024-03-16T12:08:07.060 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:07.070+0000 7fe68a7c4640 1 -- 172.21.15.29:0/840360328 shutdown_connections 2024-03-16T12:08:07.060 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:07.070+0000 7fe68a7c4640 1 --2- 172.21.15.29:0/840360328 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe68410bdf0 0x7fe68410e220 unknown :-1 s=CLOSED pgs=791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:08:07.061 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:07.070+0000 7fe68a7c4640 1 --2- 172.21.15.29:0/840360328 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe684103340 0x7fe68410b820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:08:07.061 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:07.070+0000 7fe68a7c4640 1 --2- 172.21.15.29:0/840360328 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe684102a20 0x7fe684102e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:08:07.061 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:07.070+0000 7fe68a7c4640 1 -- 172.21.15.29:0/840360328 >> 172.21.15.29:0/840360328 conn(0x7fe684076a60 msgr2=0x7fe684076e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:08:07.061 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:07.070+0000 7fe68a7c4640 1 -- 172.21.15.29:0/840360328 shutdown_connections 2024-03-16T12:08:07.061 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:07.070+0000 7fe68a7c4640 1 -- 172.21.15.29:0/840360328 wait complete. 2024-03-16T12:08:07.061 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:07.070+0000 7fe68a7c4640 1 Processor -- start 2024-03-16T12:08:07.062 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:07.070+0000 7fe68a7c4640 1 -- start start 2024-03-16T12:08:07.065 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:07.070+0000 7fe68a7c4640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe684102a20 0x7fe684198040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:08:07.065 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:07.070+0000 7fe68a7c4640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe684103340 0x7fe684198580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:08:07.065 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:07.071+0000 7fe68a7c4640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe68410bdf0 0x7fe68419c900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:08:07.066 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:07.071+0000 7fe68a7c4640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fe684111b80 con 0x7fe684103340 2024-03-16T12:08:07.066 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:07.071+0000 7fe68a7c4640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fe684111a00 con 0x7fe68410bdf0 2024-03-16T12:08:07.066 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:07.071+0000 7fe68a7c4640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fe684111d00 con 0x7fe684102a20 2024-03-16T12:08:07.066 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:07.071+0000 7fe6837fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe684103340 0x7fe684198580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:08:07.066 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:07.071+0000 7fe6837fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe684103340 0x7fe684198580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57876/0 (socket says 172.21.15.29:57876) 2024-03-16T12:08:07.067 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:07.071+0000 7fe6837fe640 1 -- 172.21.15.29:0/4004481208 learned_addr learned my addr 172.21.15.29:0/4004481208 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T12:08:07.067 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:07.071+0000 7fe683fff640 1 --2- 172.21.15.29:0/4004481208 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe684102a20 0x7fe684198040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:08:07.067 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:07.071+0000 7fe688d61640 1 --2- 172.21.15.29:0/4004481208 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe68410bdf0 0x7fe68419c900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:08:07.067 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:07.071+0000 7fe6837fe640 1 -- 172.21.15.29:0/4004481208 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe684102a20 msgr2=0x7fe684198040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:08:07.067 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:07.071+0000 7fe6837fe640 1 --2- 172.21.15.29:0/4004481208 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe684102a20 0x7fe684198040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:08:07.067 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:07.071+0000 7fe6837fe640 1 -- 172.21.15.29:0/4004481208 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe68410bdf0 msgr2=0x7fe68419c900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:08:07.068 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:07.071+0000 7fe6837fe640 1 --2- 172.21.15.29:0/4004481208 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe68410bdf0 0x7fe68419c900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:08:07.068 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:07.071+0000 7fe6837fe640 1 -- 172.21.15.29:0/4004481208 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe68419d030 con 0x7fe684103340 2024-03-16T12:08:07.068 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:07.071+0000 7fe6837fe640 1 --2- 172.21.15.29:0/4004481208 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe684103340 0x7fe684198580 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7fe66c00e9f0 tx=0x7fe66c00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:08:07.068 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:07.071+0000 7fe6817fa640 1 -- 172.21.15.29:0/4004481208 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe66c00cd80 con 0x7fe684103340 2024-03-16T12:08:07.068 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:07.071+0000 7fe6817fa640 1 -- 172.21.15.29:0/4004481208 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe66c005500 con 0x7fe684103340 2024-03-16T12:08:07.068 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:07.071+0000 7fe68a7c4640 1 -- 172.21.15.29:0/4004481208 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe68419d320 con 0x7fe684103340 2024-03-16T12:08:07.068 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:07.071+0000 7fe6817fa640 1 -- 172.21.15.29:0/4004481208 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe66c010690 con 0x7fe684103340 2024-03-16T12:08:07.068 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:07.071+0000 7fe68a7c4640 1 -- 172.21.15.29:0/4004481208 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe6841a4bb0 con 0x7fe684103340 2024-03-16T12:08:07.069 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:07.072+0000 7fe68a7c4640 1 -- 172.21.15.29:0/4004481208 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe68406a5f0 con 0x7fe684103340 2024-03-16T12:08:07.069 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:07.073+0000 7fe6817fa640 1 -- 172.21.15.29:0/4004481208 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7fe66c0046c0 con 0x7fe684103340 2024-03-16T12:08:07.069 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:07.074+0000 7fe6817fa640 1 --2- 172.21.15.29:0/4004481208 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7fe6700a2970 0x7fe6700a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:08:07.069 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:07.074+0000 7fe683fff640 1 --2- 172.21.15.29:0/4004481208 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7fe6700a2970 0x7fe6700a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:08:07.069 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:07.074+0000 7fe6817fa640 1 -- 172.21.15.29:0/4004481208 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(114..114 src has 1..114) ==== 8556+0+0 (secure 0 0 0) 0x7fe66c0cf3e0 con 0x7fe684103340 2024-03-16T12:08:07.069 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:07.074+0000 7fe683fff640 1 --2- 172.21.15.29:0/4004481208 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7fe6700a2970 0x7fe6700a4e30 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fe674009bd0 tx=0x7fe674005f90 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:08:07.070 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:07.078+0000 7fe6817fa640 1 -- 172.21.15.29:0/4004481208 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fe66c053740 con 0x7fe684103340 2024-03-16T12:08:07.251 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:07.261+0000 7fe68a7c4640 1 -- 172.21.15.29:0/4004481208 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fe684199390 con 0x7fe684103340 2024-03-16T12:08:07.251 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:07.261+0000 7fe6817fa640 1 -- 172.21.15.29:0/4004481208 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v114) ==== 93+0+10 (secure 0 0 0) 0x7fe66c09c120 con 0x7fe684103340 2024-03-16T12:08:07.252 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2024-03-16T12:08:07.253 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:07.263+0000 7fe68a7c4640 1 -- 172.21.15.29:0/4004481208 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7fe6700a2970 msgr2=0x7fe6700a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:08:07.254 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:07.263+0000 7fe68a7c4640 1 --2- 172.21.15.29:0/4004481208 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7fe6700a2970 0x7fe6700a4e30 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fe674009bd0 tx=0x7fe674005f90 comp rx=0 tx=0).stop 2024-03-16T12:08:07.254 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:07.263+0000 7fe68a7c4640 1 -- 172.21.15.29:0/4004481208 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe684103340 msgr2=0x7fe684198580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:08:07.254 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:07.263+0000 7fe68a7c4640 1 --2- 172.21.15.29:0/4004481208 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe684103340 0x7fe684198580 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7fe66c00e9f0 tx=0x7fe66c00edf0 comp rx=0 tx=0).stop 2024-03-16T12:08:07.254 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:07.263+0000 7fe68a7c4640 1 -- 172.21.15.29:0/4004481208 shutdown_connections 2024-03-16T12:08:07.254 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:07.263+0000 7fe68a7c4640 1 --2- 172.21.15.29:0/4004481208 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7fe6700a2970 0x7fe6700a4e30 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:08:07.255 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:07.263+0000 7fe68a7c4640 1 --2- 172.21.15.29:0/4004481208 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe68410bdf0 0x7fe68419c900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:08:07.255 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:07.263+0000 7fe68a7c4640 1 --2- 172.21.15.29:0/4004481208 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe684103340 0x7fe684198580 unknown :-1 s=CLOSED pgs=792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:08:07.255 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:07.263+0000 7fe68a7c4640 1 --2- 172.21.15.29:0/4004481208 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe684102a20 0x7fe684198040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:08:07.255 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:07.263+0000 7fe68a7c4640 1 -- 172.21.15.29:0/4004481208 >> 172.21.15.29:0/4004481208 conn(0x7fe684076a60 msgr2=0x7fe6840fd5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:08:07.255 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:07.263+0000 7fe68a7c4640 1 -- 172.21.15.29:0/4004481208 shutdown_connections 2024-03-16T12:08:07.256 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:07.263+0000 7fe68a7c4640 1 -- 172.21.15.29:0/4004481208 wait complete. 2024-03-16T12:08:07.265 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-16T12:08:07.372 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:07.380+0000 7f47740b2640 1 -- 172.21.15.29:0/124576718 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f476c102130 msgr2=0x7f476c10e640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:08:07.372 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:07.380+0000 7f47740b2640 1 --2- 172.21.15.29:0/124576718 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f476c102130 0x7f476c10e640 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f4764009cd0 tx=0x7f476402c0f0 comp rx=0 tx=0).stop 2024-03-16T12:08:07.373 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:07.380+0000 7f47740b2640 1 -- 172.21.15.29:0/124576718 shutdown_connections 2024-03-16T12:08:07.373 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:07.380+0000 7f47740b2640 1 --2- 172.21.15.29:0/124576718 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f476c102130 0x7f476c10e640 unknown :-1 s=CLOSED pgs=793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:08:07.373 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:07.380+0000 7f47740b2640 1 --2- 172.21.15.29:0/124576718 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f476c101770 0x7f476c101bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:08:07.373 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:07.380+0000 7f47740b2640 1 --2- 172.21.15.29:0/124576718 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f476c107780 0x7f476c107b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:08:07.373 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:07.380+0000 7f47740b2640 1 -- 172.21.15.29:0/124576718 >> 172.21.15.29:0/124576718 conn(0x7f476c078be0 msgr2=0x7f476c078ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:08:07.373 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:07.380+0000 7f47740b2640 1 -- 172.21.15.29:0/124576718 shutdown_connections 2024-03-16T12:08:07.373 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:07.380+0000 7f47740b2640 1 -- 172.21.15.29:0/124576718 wait complete. 2024-03-16T12:08:07.373 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:07.381+0000 7f47740b2640 1 Processor -- start 2024-03-16T12:08:07.373 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:07.381+0000 7f47740b2640 1 -- start start 2024-03-16T12:08:07.374 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:07.381+0000 7f47740b2640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f476c101770 0x7f476c073ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:08:07.374 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:07.381+0000 7f47740b2640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f476c102130 0x7f476c074020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:08:07.374 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:07.381+0000 7f47740b2640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f476c107780 0x7f476c074560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:08:07.374 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:07.381+0000 7f47740b2640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f476c111b90 con 0x7f476c107780 2024-03-16T12:08:07.374 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:07.381+0000 7f47740b2640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f476c111a10 con 0x7f476c101770 2024-03-16T12:08:07.375 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:07.381+0000 7f47740b2640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f476c111d10 con 0x7f476c102130 2024-03-16T12:08:07.375 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:07.381+0000 7f4771e4e640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f476c101770 0x7f476c073ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:08:07.375 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:07.381+0000 7f477264f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f476c107780 0x7f476c074560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:08:07.375 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:07.381+0000 7f477264f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f476c107780 0x7f476c074560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57884/0 (socket says 172.21.15.29:57884) 2024-03-16T12:08:07.375 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:07.381+0000 7f4771e4e640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f476c101770 0x7f476c073ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.29:38016/0 (socket says 172.21.15.29:38016) 2024-03-16T12:08:07.375 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:07.381+0000 7f477264f640 1 -- 172.21.15.29:0/3454781204 learned_addr learned my addr 172.21.15.29:0/3454781204 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T12:08:07.375 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:07.381+0000 7f477164d640 1 --2- 172.21.15.29:0/3454781204 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f476c102130 0x7f476c074020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:08:07.375 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:07.382+0000 7f477264f640 1 -- 172.21.15.29:0/3454781204 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f476c102130 msgr2=0x7f476c074020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:08:07.375 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:07.382+0000 7f477264f640 1 --2- 172.21.15.29:0/3454781204 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f476c102130 0x7f476c074020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:08:07.376 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:07.382+0000 7f477264f640 1 -- 172.21.15.29:0/3454781204 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f476c101770 msgr2=0x7f476c073ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:08:07.376 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:07.382+0000 7f477264f640 1 --2- 172.21.15.29:0/3454781204 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f476c101770 0x7f476c073ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:08:07.376 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:07.382+0000 7f477264f640 1 -- 172.21.15.29:0/3454781204 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f476c1a6530 con 0x7f476c107780 2024-03-16T12:08:07.376 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:07.382+0000 7f4771e4e640 1 --2- 172.21.15.29:0/3454781204 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f476c101770 0x7f476c073ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:08:07.376 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:07.382+0000 7f477264f640 1 --2- 172.21.15.29:0/3454781204 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f476c107780 0x7f476c074560 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f47640348f0 tx=0x7f4764032c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:08:07.376 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:07.382+0000 7f4762ffd640 1 -- 172.21.15.29:0/3454781204 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4764041020 con 0x7f476c107780 2024-03-16T12:08:07.376 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:07.382+0000 7f4762ffd640 1 -- 172.21.15.29:0/3454781204 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4764032e80 con 0x7f476c107780 2024-03-16T12:08:07.376 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:07.382+0000 7f47740b2640 1 -- 172.21.15.29:0/3454781204 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f476c1a67c0 con 0x7f476c107780 2024-03-16T12:08:07.377 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:07.382+0000 7f4762ffd640 1 -- 172.21.15.29:0/3454781204 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4764039040 con 0x7f476c107780 2024-03-16T12:08:07.377 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:07.382+0000 7f47740b2640 1 -- 172.21.15.29:0/3454781204 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f476c1a6cb0 con 0x7f476c107780 2024-03-16T12:08:07.378 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:07.383+0000 7f47740b2640 1 -- 172.21.15.29:0/3454781204 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f476c068a20 con 0x7f476c107780 2024-03-16T12:08:07.379 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:07.384+0000 7f4762ffd640 1 -- 172.21.15.29:0/3454781204 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7f4764044020 con 0x7f476c107780 2024-03-16T12:08:07.379 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:07.384+0000 7f4762ffd640 1 --2- 172.21.15.29:0/3454781204 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f47580a2970 0x7f47580a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:08:07.379 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:07.384+0000 7f4771e4e640 1 --2- 172.21.15.29:0/3454781204 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f47580a2970 0x7f47580a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:08:07.379 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:07.384+0000 7f4762ffd640 1 -- 172.21.15.29:0/3454781204 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(114..114 src has 1..114) ==== 8556+0+0 (secure 0 0 0) 0x7f47640f0910 con 0x7f476c107780 2024-03-16T12:08:07.379 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:07.385+0000 7f4771e4e640 1 --2- 172.21.15.29:0/3454781204 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f47580a2970 0x7f47580a4e30 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f4768009f40 tx=0x7f4768005f90 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:08:07.379 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:07.389+0000 7f4762ffd640 1 -- 172.21.15.29:0/3454781204 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f476404b020 con 0x7f476c107780 2024-03-16T12:08:07.563 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:07.573+0000 7f47740b2640 1 -- 172.21.15.29:0/3454781204 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f476c02d1b0 con 0x7f476c107780 2024-03-16T12:08:07.565 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:07.573+0000 7f4762ffd640 1 -- 172.21.15.29:0/3454781204 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v114) ==== 104+0+11 (secure 0 0 0) 0x7f47640bd760 con 0x7f476c107780 2024-03-16T12:08:07.566 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 32 2024-03-16T12:08:07.566 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:07.575+0000 7f47740b2640 1 -- 172.21.15.29:0/3454781204 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f47580a2970 msgr2=0x7f47580a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:08:07.566 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:07.575+0000 7f47740b2640 1 --2- 172.21.15.29:0/3454781204 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f47580a2970 0x7f47580a4e30 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f4768009f40 tx=0x7f4768005f90 comp rx=0 tx=0).stop 2024-03-16T12:08:07.567 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:07.575+0000 7f47740b2640 1 -- 172.21.15.29:0/3454781204 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f476c107780 msgr2=0x7f476c074560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:08:07.567 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:07.575+0000 7f47740b2640 1 --2- 172.21.15.29:0/3454781204 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f476c107780 0x7f476c074560 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f47640348f0 tx=0x7f4764032c90 comp rx=0 tx=0).stop 2024-03-16T12:08:07.567 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:07.575+0000 7f47740b2640 1 -- 172.21.15.29:0/3454781204 shutdown_connections 2024-03-16T12:08:07.567 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:07.575+0000 7f47740b2640 1 --2- 172.21.15.29:0/3454781204 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f47580a2970 0x7f47580a4e30 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:08:07.567 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:07.575+0000 7f47740b2640 1 --2- 172.21.15.29:0/3454781204 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f476c107780 0x7f476c074560 unknown :-1 s=CLOSED pgs=794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:08:07.567 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:07.575+0000 7f47740b2640 1 --2- 172.21.15.29:0/3454781204 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f476c102130 0x7f476c074020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:08:07.567 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:07.575+0000 7f47740b2640 1 --2- 172.21.15.29:0/3454781204 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f476c101770 0x7f476c073ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:08:07.567 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:07.575+0000 7f47740b2640 1 -- 172.21.15.29:0/3454781204 >> 172.21.15.29:0/3454781204 conn(0x7f476c078be0 msgr2=0x7f476c104020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:08:07.567 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:07.575+0000 7f47740b2640 1 -- 172.21.15.29:0/3454781204 shutdown_connections 2024-03-16T12:08:07.568 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:07.575+0000 7f47740b2640 1 -- 172.21.15.29:0/3454781204 wait complete. 2024-03-16T12:08:07.576 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:08:07 smithi029 ceph-mon[31975]: from='client.16809 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T12:08:07.576 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:08:07 smithi029 ceph-mon[31975]: from='client.16815 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T12:08:07.576 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:08:07 smithi029 ceph-mon[31975]: from='client.? 172.21.15.29:0/681775310' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T12:08:07.576 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:08:07 smithi029 ceph-mon[31975]: from='client.? 172.21.15.29:0/4004481208' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-16T12:08:07.578 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-16T12:08:07.689 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:07.693+0000 7f981f7b3640 1 -- 172.21.15.29:0/2047401417 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f98180693a0 msgr2=0x7f981810e640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:08:07.690 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:07.693+0000 7f981f7b3640 1 --2- 172.21.15.29:0/2047401417 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f98180693a0 0x7f981810e640 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f9808009cd0 tx=0x7f980802c0f0 comp rx=0 tx=0).stop 2024-03-16T12:08:07.690 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:07.693+0000 7f981f7b3640 1 -- 172.21.15.29:0/2047401417 shutdown_connections 2024-03-16T12:08:07.690 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:07.693+0000 7f981f7b3640 1 --2- 172.21.15.29:0/2047401417 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f98180693a0 0x7f981810e640 unknown :-1 s=CLOSED pgs=795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:08:07.691 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:07.693+0000 7f981f7b3640 1 --2- 172.21.15.29:0/2047401417 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f98180689e0 0x7f9818068e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:08:07.691 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:07.693+0000 7f981f7b3640 1 --2- 172.21.15.29:0/2047401417 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9818107830 0x7f9818107c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:08:07.691 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:07.693+0000 7f981f7b3640 1 -- 172.21.15.29:0/2047401417 >> 172.21.15.29:0/2047401417 conn(0x7f98180777e0 msgr2=0x7f9818077bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:08:07.691 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:07.694+0000 7f981f7b3640 1 -- 172.21.15.29:0/2047401417 shutdown_connections 2024-03-16T12:08:07.692 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:07.694+0000 7f981f7b3640 1 -- 172.21.15.29:0/2047401417 wait complete. 2024-03-16T12:08:07.692 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:07.694+0000 7f981f7b3640 1 Processor -- start 2024-03-16T12:08:07.692 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:07.694+0000 7f981f7b3640 1 -- start start 2024-03-16T12:08:07.692 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:07.694+0000 7f981f7b3640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f98180689e0 0x7f9818198000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:08:07.693 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:07.694+0000 7f981f7b3640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f98180693a0 0x7f9818198540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:08:07.693 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:07.694+0000 7f981f7b3640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9818107830 0x7f981819c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:08:07.693 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:07.694+0000 7f981f7b3640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f9818111b80 con 0x7f9818107830 2024-03-16T12:08:07.693 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:07.694+0000 7f981f7b3640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f9818111a00 con 0x7f98180689e0 2024-03-16T12:08:07.694 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:07.694+0000 7f981f7b3640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f9818111d00 con 0x7f98180693a0 2024-03-16T12:08:07.694 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:07.694+0000 7f981dd50640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9818107830 0x7f981819c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:08:07.694 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:07.694+0000 7f981dd50640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9818107830 0x7f981819c8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57912/0 (socket says 172.21.15.29:57912) 2024-03-16T12:08:07.694 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:07.694+0000 7f981dd50640 1 -- 172.21.15.29:0/1190574999 learned_addr learned my addr 172.21.15.29:0/1190574999 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T12:08:07.695 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:07.694+0000 7f981d54f640 1 --2- 172.21.15.29:0/1190574999 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f98180689e0 0x7f9818198000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:08:07.695 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:07.695+0000 7f981cd4e640 1 --2- 172.21.15.29:0/1190574999 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f98180693a0 0x7f9818198540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:08:07.695 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:07.695+0000 7f981dd50640 1 -- 172.21.15.29:0/1190574999 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f98180693a0 msgr2=0x7f9818198540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:08:07.695 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:07.695+0000 7f981dd50640 1 --2- 172.21.15.29:0/1190574999 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f98180693a0 0x7f9818198540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:08:07.696 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:07.695+0000 7f981dd50640 1 -- 172.21.15.29:0/1190574999 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f98180689e0 msgr2=0x7f9818198000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:08:07.696 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:07.695+0000 7f981dd50640 1 --2- 172.21.15.29:0/1190574999 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f98180689e0 0x7f9818198000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:08:07.696 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:07.695+0000 7f981dd50640 1 -- 172.21.15.29:0/1190574999 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f981819cff0 con 0x7f9818107830 2024-03-16T12:08:07.696 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:07.695+0000 7f981cd4e640 1 --2- 172.21.15.29:0/1190574999 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f98180693a0 0x7f9818198540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T12:08:07.697 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:07.695+0000 7f981dd50640 1 --2- 172.21.15.29:0/1190574999 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9818107830 0x7f981819c8c0 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f9808004af0 tx=0x7f9808032c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:08:07.697 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:07.695+0000 7f98167fc640 1 -- 172.21.15.29:0/1190574999 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9808042020 con 0x7f9818107830 2024-03-16T12:08:07.697 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:07.695+0000 7f98167fc640 1 -- 172.21.15.29:0/1190574999 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9808032e80 con 0x7f9818107830 2024-03-16T12:08:07.697 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:07.695+0000 7f981f7b3640 1 -- 172.21.15.29:0/1190574999 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f981819d280 con 0x7f9818107830 2024-03-16T12:08:07.698 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:07.695+0000 7f98167fc640 1 -- 172.21.15.29:0/1190574999 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9808039040 con 0x7f9818107830 2024-03-16T12:08:07.698 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:07.695+0000 7f981f7b3640 1 -- 172.21.15.29:0/1190574999 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f98181a4b70 con 0x7f9818107830 2024-03-16T12:08:07.698 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:07.696+0000 7f981f7b3640 1 -- 172.21.15.29:0/1190574999 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9818199350 con 0x7f9818107830 2024-03-16T12:08:07.699 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:07.697+0000 7f98167fc640 1 -- 172.21.15.29:0/1190574999 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7f9808045020 con 0x7f9818107830 2024-03-16T12:08:07.699 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:07.698+0000 7f98167fc640 1 --2- 172.21.15.29:0/1190574999 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f98000a29f0 0x7f98000a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:08:07.699 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:07.698+0000 7f981d54f640 1 --2- 172.21.15.29:0/1190574999 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f98000a29f0 0x7f98000a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:08:07.699 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:07.698+0000 7f98167fc640 1 -- 172.21.15.29:0/1190574999 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(114..114 src has 1..114) ==== 8556+0+0 (secure 0 0 0) 0x7f98080f0cc0 con 0x7f9818107830 2024-03-16T12:08:07.700 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:07.698+0000 7f981d54f640 1 --2- 172.21.15.29:0/1190574999 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f98000a29f0 0x7f98000a4eb0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f9810009bd0 tx=0x7f9810005f90 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:08:07.700 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:07.702+0000 7f98167fc640 1 -- 172.21.15.29:0/1190574999 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f98080f7080 con 0x7f9818107830 2024-03-16T12:08:07.786 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:08:07 smithi154 ceph-mon[36320]: from='client.16809 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T12:08:07.786 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:08:07 smithi154 ceph-mon[36320]: from='client.16815 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T12:08:07.786 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:08:07 smithi154 ceph-mon[36320]: from='client.? 172.21.15.29:0/681775310' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T12:08:07.787 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:08:07 smithi154 ceph-mon[36320]: from='client.? 172.21.15.29:0/4004481208' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-16T12:08:07.868 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:08:07 smithi152 ceph-mon[36291]: from='client.16809 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T12:08:07.868 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:08:07 smithi152 ceph-mon[36291]: from='client.16815 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T12:08:07.868 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:08:07 smithi152 ceph-mon[36291]: from='client.? 172.21.15.29:0/681775310' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T12:08:07.868 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:08:07 smithi152 ceph-mon[36291]: from='client.? 172.21.15.29:0/4004481208' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-16T12:08:07.874 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:07.884+0000 7f981f7b3640 1 -- 172.21.15.29:0/1190574999 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f981802d1b0 con 0x7f9818107830 2024-03-16T12:08:07.875 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:07.884+0000 7f98167fc640 1 -- 172.21.15.29:0/1190574999 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v114) ==== 100+0+11 (secure 0 0 0) 0x7f980804c020 con 0x7f9818107830 2024-03-16T12:08:07.875 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 64 2024-03-16T12:08:07.878 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:07.886+0000 7f981f7b3640 1 -- 172.21.15.29:0/1190574999 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f98000a29f0 msgr2=0x7f98000a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:08:07.878 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:07.886+0000 7f981f7b3640 1 --2- 172.21.15.29:0/1190574999 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f98000a29f0 0x7f98000a4eb0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f9810009bd0 tx=0x7f9810005f90 comp rx=0 tx=0).stop 2024-03-16T12:08:07.878 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:07.886+0000 7f981f7b3640 1 -- 172.21.15.29:0/1190574999 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9818107830 msgr2=0x7f981819c8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:08:07.879 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:07.886+0000 7f981f7b3640 1 --2- 172.21.15.29:0/1190574999 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9818107830 0x7f981819c8c0 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f9808004af0 tx=0x7f9808032c90 comp rx=0 tx=0).stop 2024-03-16T12:08:07.879 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:07.887+0000 7f981f7b3640 1 -- 172.21.15.29:0/1190574999 shutdown_connections 2024-03-16T12:08:07.879 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:07.887+0000 7f981f7b3640 1 --2- 172.21.15.29:0/1190574999 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f98000a29f0 0x7f98000a4eb0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:08:07.879 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:07.887+0000 7f981f7b3640 1 --2- 172.21.15.29:0/1190574999 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9818107830 0x7f981819c8c0 unknown :-1 s=CLOSED pgs=796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:08:07.879 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:07.887+0000 7f981f7b3640 1 --2- 172.21.15.29:0/1190574999 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f98180693a0 0x7f9818198540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:08:07.879 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:07.887+0000 7f981f7b3640 1 --2- 172.21.15.29:0/1190574999 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f98180689e0 0x7f9818198000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:08:07.879 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:07.887+0000 7f981f7b3640 1 -- 172.21.15.29:0/1190574999 >> 172.21.15.29:0/1190574999 conn(0x7f98180777e0 msgr2=0x7f98180fe1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:08:07.880 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:07.887+0000 7f981f7b3640 1 -- 172.21.15.29:0/1190574999 shutdown_connections 2024-03-16T12:08:07.880 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:07.887+0000 7f981f7b3640 1 -- 172.21.15.29:0/1190574999 wait complete. 2024-03-16T12:08:07.888 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json-pretty 2024-03-16T12:08:07.992 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:08.000+0000 7fb02ef7e640 1 -- 172.21.15.29:0/706123974 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb028101060 msgr2=0x7fb02810d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:08:07.993 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:08.000+0000 7fb02ef7e640 1 --2- 172.21.15.29:0/706123974 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb028101060 0x7fb02810d570 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7fb018009930 tx=0x7fb01802bc30 comp rx=0 tx=0).stop 2024-03-16T12:08:07.993 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:08.000+0000 7fb02ef7e640 1 -- 172.21.15.29:0/706123974 shutdown_connections 2024-03-16T12:08:07.993 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:08.000+0000 7fb02ef7e640 1 --2- 172.21.15.29:0/706123974 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb028101060 0x7fb02810d570 unknown :-1 s=CLOSED pgs=797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:08:07.993 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:08.000+0000 7fb02ef7e640 1 --2- 172.21.15.29:0/706123974 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb0281006a0 0x7fb028100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:08:07.993 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:08.000+0000 7fb02ef7e640 1 --2- 172.21.15.29:0/706123974 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb0281066b0 0x7fb028106a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:08:07.993 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:08.000+0000 7fb02ef7e640 1 -- 172.21.15.29:0/706123974 >> 172.21.15.29:0/706123974 conn(0x7fb028077780 msgr2=0x7fb028077b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:08:07.993 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:08.000+0000 7fb02ef7e640 1 -- 172.21.15.29:0/706123974 shutdown_connections 2024-03-16T12:08:07.994 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:08.000+0000 7fb02ef7e640 1 -- 172.21.15.29:0/706123974 wait complete. 2024-03-16T12:08:07.994 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:08.001+0000 7fb02ef7e640 1 Processor -- start 2024-03-16T12:08:07.994 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:08.001+0000 7fb02ef7e640 1 -- start start 2024-03-16T12:08:07.994 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:08.001+0000 7fb02ef7e640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb0281006a0 0x7fb028196fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:08:07.994 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:08.001+0000 7fb02ef7e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb028101060 0x7fb0281974e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:08:07.994 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:08.001+0000 7fb02ef7e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb0281066b0 0x7fb02819b860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:08:07.994 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:08.001+0000 7fb02ef7e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fb028110ac0 con 0x7fb0281066b0 2024-03-16T12:08:07.994 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:08.001+0000 7fb02ef7e640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fb028110940 con 0x7fb0281006a0 2024-03-16T12:08:07.995 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:08.001+0000 7fb02ef7e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fb028110c40 con 0x7fb028101060 2024-03-16T12:08:07.995 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:08.001+0000 7fb02d51b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb0281066b0 0x7fb02819b860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:08:07.995 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:08.001+0000 7fb02d51b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb0281066b0 0x7fb02819b860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57926/0 (socket says 172.21.15.29:57926) 2024-03-16T12:08:07.995 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:08.001+0000 7fb02d51b640 1 -- 172.21.15.29:0/2899694199 learned_addr learned my addr 172.21.15.29:0/2899694199 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T12:08:07.995 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:08.001+0000 7fb02cd1a640 1 --2- 172.21.15.29:0/2899694199 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb0281006a0 0x7fb028196fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:08:07.995 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:08.001+0000 7fb027fff640 1 --2- 172.21.15.29:0/2899694199 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb028101060 0x7fb0281974e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:08:07.995 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:08.002+0000 7fb02d51b640 1 -- 172.21.15.29:0/2899694199 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb028101060 msgr2=0x7fb0281974e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:08:07.995 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:08.002+0000 7fb02d51b640 1 --2- 172.21.15.29:0/2899694199 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb028101060 0x7fb0281974e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:08:07.996 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:08.002+0000 7fb02d51b640 1 -- 172.21.15.29:0/2899694199 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb0281006a0 msgr2=0x7fb028196fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:08:07.996 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:08.002+0000 7fb02d51b640 1 --2- 172.21.15.29:0/2899694199 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb0281006a0 0x7fb028196fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:08:07.996 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:08.002+0000 7fb02d51b640 1 -- 172.21.15.29:0/2899694199 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb02819bf90 con 0x7fb0281066b0 2024-03-16T12:08:07.996 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:08.002+0000 7fb02d51b640 1 --2- 172.21.15.29:0/2899694199 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb0281066b0 0x7fb02819b860 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7fb018005940 tx=0x7fb01802cb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:08:07.996 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:08.002+0000 7fb025ffb640 1 -- 172.21.15.29:0/2899694199 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb018041070 con 0x7fb0281066b0 2024-03-16T12:08:07.996 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:08.002+0000 7fb025ffb640 1 -- 172.21.15.29:0/2899694199 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb018004450 con 0x7fb0281066b0 2024-03-16T12:08:07.996 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:08.002+0000 7fb02ef7e640 1 -- 172.21.15.29:0/2899694199 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb02819c220 con 0x7fb0281066b0 2024-03-16T12:08:07.996 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:08.002+0000 7fb025ffb640 1 -- 172.21.15.29:0/2899694199 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb018003d30 con 0x7fb0281066b0 2024-03-16T12:08:07.996 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:08.002+0000 7fb02ef7e640 1 -- 172.21.15.29:0/2899694199 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb028069e10 con 0x7fb0281066b0 2024-03-16T12:08:07.997 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:08.003+0000 7fb02ef7e640 1 -- 172.21.15.29:0/2899694199 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb028101de0 con 0x7fb0281066b0 2024-03-16T12:08:07.997 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:08.004+0000 7fb025ffb640 1 -- 172.21.15.29:0/2899694199 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7fb018044070 con 0x7fb0281066b0 2024-03-16T12:08:07.998 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:08.005+0000 7fb025ffb640 1 --2- 172.21.15.29:0/2899694199 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7fb0100a2970 0x7fb0100a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:08:07.998 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:08.005+0000 7fb02cd1a640 1 --2- 172.21.15.29:0/2899694199 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7fb0100a2970 0x7fb0100a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:08:07.998 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:08.005+0000 7fb025ffb640 1 -- 172.21.15.29:0/2899694199 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(114..114 src has 1..114) ==== 8556+0+0 (secure 0 0 0) 0x7fb0180f0c00 con 0x7fb0281066b0 2024-03-16T12:08:07.998 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:08.005+0000 7fb02cd1a640 1 --2- 172.21.15.29:0/2899694199 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7fb0100a2970 0x7fb0100a4e30 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fb020005fd0 tx=0x7fb020009500 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:08:07.998 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:08.009+0000 7fb025ffb640 1 -- 172.21.15.29:0/2899694199 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fb0180bd400 con 0x7fb0281066b0 2024-03-16T12:08:08.232 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:08.242+0000 7fb02ef7e640 1 -- 172.21.15.29:0/2899694199 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json-pretty"} v 0) -- 0x7fb028108700 con 0x7fb0281066b0 2024-03-16T12:08:08.252 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:08.262+0000 7fb025ffb640 1 -- 172.21.15.29:0/2899694199 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json-pretty"}]=0 v25) ==== 81+0+740362 (secure 0 0 0) 0x7fb0180c14a0 con 0x7fb0281066b0 2024-03-16T12:08:08.263 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-16T12:08:08.263 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-03-16T12:08:08.263 INFO:teuthology.orchestra.run.smithi029.stdout: "epoch": 25, 2024-03-16T12:08:08.264 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:08.264 INFO:teuthology.orchestra.run.smithi029.stdout: "active_gid": 15333, 2024-03-16T12:08:08.264 INFO:teuthology.orchestra.run.smithi029.stdout: "active_name": "x", 2024-03-16T12:08:08.264 INFO:teuthology.orchestra.run.smithi029.stdout: "active_addrs": { 2024-03-16T12:08:08.264 INFO:teuthology.orchestra.run.smithi029.stdout: "addrvec": [ 2024-03-16T12:08:08.264 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-16T12:08:08.265 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "v2", 2024-03-16T12:08:08.265 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": "172.21.15.29:6800", 2024-03-16T12:08:08.265 INFO:teuthology.orchestra.run.smithi029.stdout: "nonce": 1188678802 2024-03-16T12:08:08.265 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.265 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-16T12:08:08.265 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "v1", 2024-03-16T12:08:08.265 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": "172.21.15.29:6801", 2024-03-16T12:08:08.265 INFO:teuthology.orchestra.run.smithi029.stdout: "nonce": 1188678802 2024-03-16T12:08:08.265 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:08.266 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-16T12:08:08.266 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.266 INFO:teuthology.orchestra.run.smithi029.stdout: "active_addr": "172.21.15.29:6801/1188678802", 2024-03-16T12:08:08.266 INFO:teuthology.orchestra.run.smithi029.stdout: "active_change": "2024-03-16T11:50:54.850831+0000", 2024-03-16T12:08:08.266 INFO:teuthology.orchestra.run.smithi029.stdout: "active_mgr_features": 4540701547738038271, 2024-03-16T12:08:08.267 INFO:teuthology.orchestra.run.smithi029.stdout: "available": true, 2024-03-16T12:08:08.267 INFO:teuthology.orchestra.run.smithi029.stdout: "standbys": [ 2024-03-16T12:08:08.267 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-16T12:08:08.267 INFO:teuthology.orchestra.run.smithi029.stdout: "gid": 25139, 2024-03-16T12:08:08.267 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "y", 2024-03-16T12:08:08.267 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr_features": 4540701547738038271, 2024-03-16T12:08:08.267 INFO:teuthology.orchestra.run.smithi029.stdout: "available_modules": [ 2024-03-16T12:08:08.267 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-16T12:08:08.267 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "alerts", 2024-03-16T12:08:08.268 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-03-16T12:08:08.268 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-03-16T12:08:08.268 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-03-16T12:08:08.268 INFO:teuthology.orchestra.run.smithi029.stdout: "interval": { 2024-03-16T12:08:08.268 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "interval", 2024-03-16T12:08:08.268 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "secs", 2024-03-16T12:08:08.268 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.269 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:08.269 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "60", 2024-03-16T12:08:08.269 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.269 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.269 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.269 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "How frequently to reexamine health status", 2024-03-16T12:08:08.269 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.270 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.270 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.270 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.270 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-03-16T12:08:08.270 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-03-16T12:08:08.270 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:08.270 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.270 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:08.270 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:08.271 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.271 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.271 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-16T12:08:08.271 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-16T12:08:08.271 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-16T12:08:08.271 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-16T12:08:08.271 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-16T12:08:08.271 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-16T12:08:08.271 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-16T12:08:08.272 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:08.272 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:08.272 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.272 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.272 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.272 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.272 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-03-16T12:08:08.273 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-03-16T12:08:08.273 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:08.273 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.273 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:08.273 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:08.273 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.273 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.273 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.274 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:08.274 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.274 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.274 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.274 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.275 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-03-16T12:08:08.275 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-03-16T12:08:08.275 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:08.275 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.275 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:08.275 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-03-16T12:08:08.275 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.275 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.276 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-16T12:08:08.276 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-16T12:08:08.276 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-16T12:08:08.276 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-16T12:08:08.276 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-16T12:08:08.276 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-16T12:08:08.276 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-16T12:08:08.277 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:08.277 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:08.277 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.277 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.277 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.277 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.277 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-03-16T12:08:08.277 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-03-16T12:08:08.278 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:08.278 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.278 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:08.278 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:08.278 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.278 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.279 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.279 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:08.279 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.279 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.279 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.279 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.279 INFO:teuthology.orchestra.run.smithi029.stdout: "smtp_destination": { 2024-03-16T12:08:08.280 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "smtp_destination", 2024-03-16T12:08:08.280 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:08.280 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.280 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:08.280 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:08.280 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.281 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.281 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.281 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Email address to send alerts to", 2024-03-16T12:08:08.281 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.281 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.281 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.281 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.281 INFO:teuthology.orchestra.run.smithi029.stdout: "smtp_from_name": { 2024-03-16T12:08:08.281 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "smtp_from_name", 2024-03-16T12:08:08.282 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:08.282 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.282 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:08.282 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "Ceph", 2024-03-16T12:08:08.282 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.282 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.282 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.282 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Email From: name", 2024-03-16T12:08:08.283 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.283 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.283 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.283 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.283 INFO:teuthology.orchestra.run.smithi029.stdout: "smtp_host": { 2024-03-16T12:08:08.283 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "smtp_host", 2024-03-16T12:08:08.283 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:08.283 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.283 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:08.284 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:08.284 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.284 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.284 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.284 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "SMTP server", 2024-03-16T12:08:08.284 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.284 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.284 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.284 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.285 INFO:teuthology.orchestra.run.smithi029.stdout: "smtp_password": { 2024-03-16T12:08:08.285 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "smtp_password", 2024-03-16T12:08:08.285 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:08.285 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.285 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:08.285 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:08.285 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.285 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.285 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.286 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Password to authenticate with", 2024-03-16T12:08:08.286 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.286 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.286 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.286 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.286 INFO:teuthology.orchestra.run.smithi029.stdout: "smtp_port": { 2024-03-16T12:08:08.286 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "smtp_port", 2024-03-16T12:08:08.286 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-16T12:08:08.286 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.287 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:08.287 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "465", 2024-03-16T12:08:08.287 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.287 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.287 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.287 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "SMTP port", 2024-03-16T12:08:08.287 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.287 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.287 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.288 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.288 INFO:teuthology.orchestra.run.smithi029.stdout: "smtp_sender": { 2024-03-16T12:08:08.288 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "smtp_sender", 2024-03-16T12:08:08.288 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:08.288 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.288 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:08.288 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:08.288 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.289 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.289 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.289 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "SMTP envelope sender", 2024-03-16T12:08:08.289 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.289 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.289 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.289 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.289 INFO:teuthology.orchestra.run.smithi029.stdout: "smtp_ssl": { 2024-03-16T12:08:08.289 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "smtp_ssl", 2024-03-16T12:08:08.290 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:08.290 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.290 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:08.290 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-16T12:08:08.290 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.290 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.290 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.290 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Use SSL to connect to SMTP server", 2024-03-16T12:08:08.291 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.291 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.291 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.291 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.291 INFO:teuthology.orchestra.run.smithi029.stdout: "smtp_user": { 2024-03-16T12:08:08.291 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "smtp_user", 2024-03-16T12:08:08.291 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:08.291 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.292 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:08.292 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:08.292 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.292 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.292 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.292 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "User to authenticate as", 2024-03-16T12:08:08.292 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.292 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.292 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.293 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:08.293 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:08.293 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.293 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-16T12:08:08.293 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "balancer", 2024-03-16T12:08:08.293 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-03-16T12:08:08.293 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-03-16T12:08:08.293 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-03-16T12:08:08.294 INFO:teuthology.orchestra.run.smithi029.stdout: "active": { 2024-03-16T12:08:08.294 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "active", 2024-03-16T12:08:08.294 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:08.294 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.294 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:08.294 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-16T12:08:08.294 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.294 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.294 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.295 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "automatically balance PGs across cluster", 2024-03-16T12:08:08.295 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.295 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.295 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.295 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.295 INFO:teuthology.orchestra.run.smithi029.stdout: "begin_time": { 2024-03-16T12:08:08.295 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "begin_time", 2024-03-16T12:08:08.295 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:08.296 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.296 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:08.296 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "0000", 2024-03-16T12:08:08.296 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.296 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.296 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.296 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "beginning time of day to automatically balance", 2024-03-16T12:08:08.296 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-16T12:08:08.297 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.297 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.297 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.297 INFO:teuthology.orchestra.run.smithi029.stdout: "begin_weekday": { 2024-03-16T12:08:08.297 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "begin_weekday", 2024-03-16T12:08:08.297 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "uint", 2024-03-16T12:08:08.297 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.297 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:08.297 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "0", 2024-03-16T12:08:08.298 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "0", 2024-03-16T12:08:08.298 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "6", 2024-03-16T12:08:08.298 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.298 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-03-16T12:08:08.298 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-16T12:08:08.298 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.298 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.298 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.299 INFO:teuthology.orchestra.run.smithi029.stdout: "crush_compat_max_iterations": { 2024-03-16T12:08:08.299 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "crush_compat_max_iterations", 2024-03-16T12:08:08.299 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "uint", 2024-03-16T12:08:08.299 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.299 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:08.299 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "25", 2024-03-16T12:08:08.300 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "1", 2024-03-16T12:08:08.300 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "250", 2024-03-16T12:08:08.300 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.300 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-03-16T12:08:08.300 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.300 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.300 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.300 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.301 INFO:teuthology.orchestra.run.smithi029.stdout: "crush_compat_metrics": { 2024-03-16T12:08:08.301 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "crush_compat_metrics", 2024-03-16T12:08:08.301 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:08.301 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.301 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:08.301 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "pgs,objects,bytes", 2024-03-16T12:08:08.301 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.302 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.302 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.302 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-03-16T12:08:08.302 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-03-16T12:08:08.302 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.302 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.302 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.302 INFO:teuthology.orchestra.run.smithi029.stdout: "crush_compat_step": { 2024-03-16T12:08:08.303 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "crush_compat_step", 2024-03-16T12:08:08.303 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "float", 2024-03-16T12:08:08.303 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.303 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:08.303 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "0.5", 2024-03-16T12:08:08.303 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "0.001", 2024-03-16T12:08:08.304 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "0.999", 2024-03-16T12:08:08.304 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.304 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "aggressiveness of optimization", 2024-03-16T12:08:08.304 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-03-16T12:08:08.304 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.304 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.304 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.304 INFO:teuthology.orchestra.run.smithi029.stdout: "end_time": { 2024-03-16T12:08:08.305 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "end_time", 2024-03-16T12:08:08.305 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:08.305 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.305 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:08.305 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "2359", 2024-03-16T12:08:08.305 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.305 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.306 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.306 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "ending time of day to automatically balance", 2024-03-16T12:08:08.306 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-16T12:08:08.306 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.306 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.306 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.306 INFO:teuthology.orchestra.run.smithi029.stdout: "end_weekday": { 2024-03-16T12:08:08.306 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "end_weekday", 2024-03-16T12:08:08.307 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "uint", 2024-03-16T12:08:08.307 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.307 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:08.307 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "0", 2024-03-16T12:08:08.307 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "0", 2024-03-16T12:08:08.307 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "6", 2024-03-16T12:08:08.307 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.308 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-03-16T12:08:08.308 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-16T12:08:08.308 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.308 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.308 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.309 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-03-16T12:08:08.309 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-03-16T12:08:08.309 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:08.309 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.309 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:08.309 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:08.310 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.310 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.310 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-16T12:08:08.310 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-16T12:08:08.310 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-16T12:08:08.311 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-16T12:08:08.311 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-16T12:08:08.311 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-16T12:08:08.311 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-16T12:08:08.311 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:08.311 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:08.311 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.312 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.312 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.312 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.312 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-03-16T12:08:08.312 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-03-16T12:08:08.312 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:08.312 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.312 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:08.313 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:08.313 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.313 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.313 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.313 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:08.313 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.313 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.313 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.313 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.314 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-03-16T12:08:08.314 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-03-16T12:08:08.314 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:08.314 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.314 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:08.314 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-03-16T12:08:08.314 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.314 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.315 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-16T12:08:08.315 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-16T12:08:08.315 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-16T12:08:08.315 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-16T12:08:08.315 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-16T12:08:08.315 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-16T12:08:08.315 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-16T12:08:08.315 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:08.315 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:08.316 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.316 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.316 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.316 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.316 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-03-16T12:08:08.316 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-03-16T12:08:08.316 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:08.316 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.316 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:08.316 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:08.317 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.317 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.317 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.317 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:08.317 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.317 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.317 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.317 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.317 INFO:teuthology.orchestra.run.smithi029.stdout: "min_score": { 2024-03-16T12:08:08.317 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "min_score", 2024-03-16T12:08:08.318 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "float", 2024-03-16T12:08:08.318 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.318 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:08.318 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "0", 2024-03-16T12:08:08.318 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.318 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.318 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.318 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-03-16T12:08:08.318 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.319 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.319 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.319 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.319 INFO:teuthology.orchestra.run.smithi029.stdout: "mode": { 2024-03-16T12:08:08.319 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "mode", 2024-03-16T12:08:08.319 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:08.319 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.319 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:08.319 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "upmap", 2024-03-16T12:08:08.319 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.320 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.320 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-16T12:08:08.320 INFO:teuthology.orchestra.run.smithi029.stdout: "crush-compat", 2024-03-16T12:08:08.320 INFO:teuthology.orchestra.run.smithi029.stdout: "none", 2024-03-16T12:08:08.320 INFO:teuthology.orchestra.run.smithi029.stdout: "read", 2024-03-16T12:08:08.320 INFO:teuthology.orchestra.run.smithi029.stdout: "upmap", 2024-03-16T12:08:08.320 INFO:teuthology.orchestra.run.smithi029.stdout: "upmap-read" 2024-03-16T12:08:08.320 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:08.320 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Balancer mode", 2024-03-16T12:08:08.321 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.321 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.321 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.321 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.321 INFO:teuthology.orchestra.run.smithi029.stdout: "pool_ids": { 2024-03-16T12:08:08.321 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "pool_ids", 2024-03-16T12:08:08.321 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:08.321 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.321 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:08.322 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:08.322 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.322 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.322 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.322 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-03-16T12:08:08.322 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.322 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.323 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.323 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.323 INFO:teuthology.orchestra.run.smithi029.stdout: "sleep_interval": { 2024-03-16T12:08:08.323 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "sleep_interval", 2024-03-16T12:08:08.323 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "secs", 2024-03-16T12:08:08.323 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.323 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:08.323 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "60", 2024-03-16T12:08:08.323 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.324 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.324 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.324 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-03-16T12:08:08.324 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.324 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.324 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.324 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.324 INFO:teuthology.orchestra.run.smithi029.stdout: "upmap_max_deviation": { 2024-03-16T12:08:08.324 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "upmap_max_deviation", 2024-03-16T12:08:08.325 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-16T12:08:08.325 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.325 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:08.325 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "5", 2024-03-16T12:08:08.325 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "1", 2024-03-16T12:08:08.325 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.325 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.325 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "deviation below which no optimization is attempted", 2024-03-16T12:08:08.325 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-03-16T12:08:08.325 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.326 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.326 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.326 INFO:teuthology.orchestra.run.smithi029.stdout: "upmap_max_optimizations": { 2024-03-16T12:08:08.326 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "upmap_max_optimizations", 2024-03-16T12:08:08.326 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "uint", 2024-03-16T12:08:08.326 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.326 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:08.326 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "10", 2024-03-16T12:08:08.326 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.326 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.327 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.327 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-03-16T12:08:08.327 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.327 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.327 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.327 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:08.327 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:08.327 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.327 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-16T12:08:08.327 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "cephadm", 2024-03-16T12:08:08.328 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-03-16T12:08:08.328 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-03-16T12:08:08.328 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-03-16T12:08:08.328 INFO:teuthology.orchestra.run.smithi029.stdout: "agent_down_multiplier": { 2024-03-16T12:08:08.328 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "agent_down_multiplier", 2024-03-16T12:08:08.328 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "float", 2024-03-16T12:08:08.328 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.328 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:08.328 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "3.0", 2024-03-16T12:08:08.329 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.329 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.329 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.329 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-03-16T12:08:08.329 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.329 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.329 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.329 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.329 INFO:teuthology.orchestra.run.smithi029.stdout: "agent_refresh_rate": { 2024-03-16T12:08:08.330 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "agent_refresh_rate", 2024-03-16T12:08:08.330 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "secs", 2024-03-16T12:08:08.330 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.330 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:08.330 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "20", 2024-03-16T12:08:08.330 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.330 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.330 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.331 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-03-16T12:08:08.331 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.331 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.331 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.331 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.331 INFO:teuthology.orchestra.run.smithi029.stdout: "agent_starting_port": { 2024-03-16T12:08:08.332 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "agent_starting_port", 2024-03-16T12:08:08.332 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-16T12:08:08.332 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.332 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:08.332 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "4721", 2024-03-16T12:08:08.332 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.332 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.332 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.333 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-03-16T12:08:08.333 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.333 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.333 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.333 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.334 INFO:teuthology.orchestra.run.smithi029.stdout: "allow_ptrace": { 2024-03-16T12:08:08.334 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "allow_ptrace", 2024-03-16T12:08:08.334 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:08.334 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.334 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:08.334 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:08.334 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.334 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.334 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.335 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-03-16T12:08:08.335 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-03-16T12:08:08.335 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.335 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.335 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.335 INFO:teuthology.orchestra.run.smithi029.stdout: "autotune_interval": { 2024-03-16T12:08:08.335 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "autotune_interval", 2024-03-16T12:08:08.336 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "secs", 2024-03-16T12:08:08.336 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.336 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:08.336 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "600", 2024-03-16T12:08:08.336 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.336 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.336 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.337 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "how frequently to autotune daemon memory", 2024-03-16T12:08:08.337 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.337 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.337 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.337 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.337 INFO:teuthology.orchestra.run.smithi029.stdout: "autotune_memory_target_ratio": { 2024-03-16T12:08:08.338 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "autotune_memory_target_ratio", 2024-03-16T12:08:08.338 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "float", 2024-03-16T12:08:08.338 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.338 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:08.338 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "0.7", 2024-03-16T12:08:08.338 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.338 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.339 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.339 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-03-16T12:08:08.339 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.339 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.339 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.339 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.340 INFO:teuthology.orchestra.run.smithi029.stdout: "cephadm_log_destination": { 2024-03-16T12:08:08.340 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "cephadm_log_destination", 2024-03-16T12:08:08.340 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:08.340 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.340 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:08.340 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:08.340 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.340 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.340 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-16T12:08:08.341 INFO:teuthology.orchestra.run.smithi029.stdout: "file", 2024-03-16T12:08:08.341 INFO:teuthology.orchestra.run.smithi029.stdout: "file,syslog", 2024-03-16T12:08:08.341 INFO:teuthology.orchestra.run.smithi029.stdout: "syslog" 2024-03-16T12:08:08.341 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:08.341 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-03-16T12:08:08.341 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.341 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.341 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.342 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.342 INFO:teuthology.orchestra.run.smithi029.stdout: "cgroups_split": { 2024-03-16T12:08:08.342 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "cgroups_split", 2024-03-16T12:08:08.342 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:08.342 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.342 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:08.342 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-16T12:08:08.343 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.343 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.343 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.343 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-03-16T12:08:08.343 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.344 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.344 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.344 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.344 INFO:teuthology.orchestra.run.smithi029.stdout: "config_checks_enabled": { 2024-03-16T12:08:08.344 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "config_checks_enabled", 2024-03-16T12:08:08.344 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:08.345 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.345 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:08.345 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:08.345 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.345 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.346 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.346 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-03-16T12:08:08.346 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.346 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.346 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.346 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.346 INFO:teuthology.orchestra.run.smithi029.stdout: "config_dashboard": { 2024-03-16T12:08:08.347 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "config_dashboard", 2024-03-16T12:08:08.347 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:08.347 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.347 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:08.347 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-16T12:08:08.347 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.347 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.347 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.347 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-03-16T12:08:08.348 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.348 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.348 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.348 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.348 INFO:teuthology.orchestra.run.smithi029.stdout: "container_image_alertmanager": { 2024-03-16T12:08:08.348 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "container_image_alertmanager", 2024-03-16T12:08:08.348 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:08.348 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.348 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:08.348 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-03-16T12:08:08.349 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.349 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.349 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.349 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Prometheus container image", 2024-03-16T12:08:08.349 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.349 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.349 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.349 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.350 INFO:teuthology.orchestra.run.smithi029.stdout: "container_image_base": { 2024-03-16T12:08:08.350 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "container_image_base", 2024-03-16T12:08:08.350 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:08.350 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.350 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:08.350 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "quay.io/ceph/ceph", 2024-03-16T12:08:08.350 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.350 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.350 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.351 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Container image name, without the tag", 2024-03-16T12:08:08.351 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.351 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.351 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.351 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.351 INFO:teuthology.orchestra.run.smithi029.stdout: "container_image_elasticsearch": { 2024-03-16T12:08:08.351 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "container_image_elasticsearch", 2024-03-16T12:08:08.351 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:08.351 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.352 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:08.352 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-03-16T12:08:08.352 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.352 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.352 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.352 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "elasticsearch container image", 2024-03-16T12:08:08.352 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.352 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.352 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.353 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.353 INFO:teuthology.orchestra.run.smithi029.stdout: "container_image_grafana": { 2024-03-16T12:08:08.353 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "container_image_grafana", 2024-03-16T12:08:08.353 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:08.353 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.353 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:08.353 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "quay.io/ceph/ceph-grafana:9.4.12", 2024-03-16T12:08:08.353 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.353 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.353 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.354 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Prometheus container image", 2024-03-16T12:08:08.354 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.354 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.354 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.354 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.354 INFO:teuthology.orchestra.run.smithi029.stdout: "container_image_haproxy": { 2024-03-16T12:08:08.354 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "container_image_haproxy", 2024-03-16T12:08:08.354 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:08.354 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.355 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:08.355 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-03-16T12:08:08.355 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.355 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.355 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.355 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "HAproxy container image", 2024-03-16T12:08:08.355 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.355 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.355 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.356 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.356 INFO:teuthology.orchestra.run.smithi029.stdout: "container_image_jaeger_agent": { 2024-03-16T12:08:08.356 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "container_image_jaeger_agent", 2024-03-16T12:08:08.356 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:08.356 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.356 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:08.356 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-03-16T12:08:08.356 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.356 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.356 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.357 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Jaeger agent container image", 2024-03-16T12:08:08.357 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.357 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.357 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.357 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.357 INFO:teuthology.orchestra.run.smithi029.stdout: "container_image_jaeger_collector": { 2024-03-16T12:08:08.357 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "container_image_jaeger_collector", 2024-03-16T12:08:08.357 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:08.357 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.357 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:08.358 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-03-16T12:08:08.358 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.358 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.358 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.358 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Jaeger collector container image", 2024-03-16T12:08:08.358 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.358 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.358 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.358 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.359 INFO:teuthology.orchestra.run.smithi029.stdout: "container_image_jaeger_query": { 2024-03-16T12:08:08.359 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "container_image_jaeger_query", 2024-03-16T12:08:08.359 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:08.359 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.359 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:08.359 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-03-16T12:08:08.359 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.359 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.360 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.360 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Jaeger query container image", 2024-03-16T12:08:08.360 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.360 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.360 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.360 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.360 INFO:teuthology.orchestra.run.smithi029.stdout: "container_image_keepalived": { 2024-03-16T12:08:08.360 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "container_image_keepalived", 2024-03-16T12:08:08.360 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:08.361 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.361 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:08.361 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-03-16T12:08:08.361 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.361 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.361 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.361 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Keepalived container image", 2024-03-16T12:08:08.361 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.361 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.361 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.362 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.362 INFO:teuthology.orchestra.run.smithi029.stdout: "container_image_loki": { 2024-03-16T12:08:08.362 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "container_image_loki", 2024-03-16T12:08:08.362 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:08.362 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.362 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:08.362 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-03-16T12:08:08.362 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.362 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.362 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.363 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Loki container image", 2024-03-16T12:08:08.363 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.363 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.363 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.363 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.363 INFO:teuthology.orchestra.run.smithi029.stdout: "container_image_node_exporter": { 2024-03-16T12:08:08.363 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "container_image_node_exporter", 2024-03-16T12:08:08.363 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:08.363 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.364 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:08.364 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-03-16T12:08:08.364 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.364 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.364 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.364 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Prometheus container image", 2024-03-16T12:08:08.364 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.364 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.364 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.364 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.365 INFO:teuthology.orchestra.run.smithi029.stdout: "container_image_nvmeof": { 2024-03-16T12:08:08.365 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "container_image_nvmeof", 2024-03-16T12:08:08.365 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:08.365 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.365 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:08.365 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "quay.io/ceph/nvmeof:1.0.0", 2024-03-16T12:08:08.365 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.365 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.365 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.365 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Nvme-of container image", 2024-03-16T12:08:08.366 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.366 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.366 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.366 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.366 INFO:teuthology.orchestra.run.smithi029.stdout: "container_image_prometheus": { 2024-03-16T12:08:08.366 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "container_image_prometheus", 2024-03-16T12:08:08.367 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:08.367 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.367 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:08.367 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-03-16T12:08:08.367 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.367 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.367 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.367 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Prometheus container image", 2024-03-16T12:08:08.367 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.368 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.368 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.368 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.368 INFO:teuthology.orchestra.run.smithi029.stdout: "container_image_promtail": { 2024-03-16T12:08:08.368 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "container_image_promtail", 2024-03-16T12:08:08.368 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:08.368 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.369 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:08.369 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-03-16T12:08:08.369 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.369 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.369 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.369 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Promtail container image", 2024-03-16T12:08:08.369 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.370 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.370 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.370 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.370 INFO:teuthology.orchestra.run.smithi029.stdout: "container_image_snmp_gateway": { 2024-03-16T12:08:08.370 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "container_image_snmp_gateway", 2024-03-16T12:08:08.370 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:08.370 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.371 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:08.371 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-03-16T12:08:08.371 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.371 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.371 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.371 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "SNMP Gateway container image", 2024-03-16T12:08:08.371 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.371 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.372 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.372 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.372 INFO:teuthology.orchestra.run.smithi029.stdout: "container_init": { 2024-03-16T12:08:08.372 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "container_init", 2024-03-16T12:08:08.372 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:08.372 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.372 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:08.372 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-16T12:08:08.373 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.373 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.373 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.373 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Run podman/docker with `--init`", 2024-03-16T12:08:08.373 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.373 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.374 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.374 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.374 INFO:teuthology.orchestra.run.smithi029.stdout: "daemon_cache_timeout": { 2024-03-16T12:08:08.374 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "daemon_cache_timeout", 2024-03-16T12:08:08.374 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "secs", 2024-03-16T12:08:08.374 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.374 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:08.374 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "600", 2024-03-16T12:08:08.375 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.375 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.375 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.375 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-03-16T12:08:08.375 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.376 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.376 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.376 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.376 INFO:teuthology.orchestra.run.smithi029.stdout: "default_cephadm_command_timeout": { 2024-03-16T12:08:08.376 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "default_cephadm_command_timeout", 2024-03-16T12:08:08.376 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-16T12:08:08.376 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.377 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:08.377 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "900", 2024-03-16T12:08:08.377 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.377 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.377 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.378 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-03-16T12:08:08.378 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.378 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.378 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.379 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.379 INFO:teuthology.orchestra.run.smithi029.stdout: "default_registry": { 2024-03-16T12:08:08.379 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "default_registry", 2024-03-16T12:08:08.379 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:08.379 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.379 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:08.379 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "docker.io", 2024-03-16T12:08:08.379 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.379 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.380 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.380 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-03-16T12:08:08.380 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.380 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.380 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.380 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.380 INFO:teuthology.orchestra.run.smithi029.stdout: "device_cache_timeout": { 2024-03-16T12:08:08.380 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "device_cache_timeout", 2024-03-16T12:08:08.380 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "secs", 2024-03-16T12:08:08.380 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.381 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:08.381 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "1800", 2024-03-16T12:08:08.381 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.381 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.381 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.381 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "seconds to cache device inventory", 2024-03-16T12:08:08.381 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.381 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.381 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.381 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.382 INFO:teuthology.orchestra.run.smithi029.stdout: "device_enhanced_scan": { 2024-03-16T12:08:08.382 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "device_enhanced_scan", 2024-03-16T12:08:08.382 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:08.382 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.382 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:08.382 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:08.382 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.382 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.383 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.383 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Use libstoragemgmt during device scans", 2024-03-16T12:08:08.383 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.383 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.383 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.383 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.384 INFO:teuthology.orchestra.run.smithi029.stdout: "facts_cache_timeout": { 2024-03-16T12:08:08.384 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "facts_cache_timeout", 2024-03-16T12:08:08.385 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "secs", 2024-03-16T12:08:08.385 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.385 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:08.385 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "60", 2024-03-16T12:08:08.385 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.385 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.385 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.385 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "seconds to cache host facts data", 2024-03-16T12:08:08.385 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.386 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.386 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.386 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.386 INFO:teuthology.orchestra.run.smithi029.stdout: "host_check_interval": { 2024-03-16T12:08:08.386 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "host_check_interval", 2024-03-16T12:08:08.386 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "secs", 2024-03-16T12:08:08.386 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.386 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:08.386 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "600", 2024-03-16T12:08:08.387 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.387 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.387 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.387 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "how frequently to perform a host check", 2024-03-16T12:08:08.387 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.387 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.387 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.387 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.387 INFO:teuthology.orchestra.run.smithi029.stdout: "hw_monitoring": { 2024-03-16T12:08:08.387 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "hw_monitoring", 2024-03-16T12:08:08.388 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:08.388 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.388 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:08.388 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:08.388 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.388 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.388 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.388 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-03-16T12:08:08.388 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.389 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.389 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.389 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.389 INFO:teuthology.orchestra.run.smithi029.stdout: "inventory_list_all": { 2024-03-16T12:08:08.389 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "inventory_list_all", 2024-03-16T12:08:08.389 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:08.389 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.389 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:08.389 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:08.389 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.390 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.390 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.390 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-03-16T12:08:08.390 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.390 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.390 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.390 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.390 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-03-16T12:08:08.390 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-03-16T12:08:08.391 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:08.391 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.391 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:08.391 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:08.391 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.391 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.391 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-16T12:08:08.391 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-16T12:08:08.391 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-16T12:08:08.391 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-16T12:08:08.392 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-16T12:08:08.392 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-16T12:08:08.392 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-16T12:08:08.392 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:08.392 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:08.392 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.392 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.392 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.392 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.393 INFO:teuthology.orchestra.run.smithi029.stdout: "log_refresh_metadata": { 2024-03-16T12:08:08.393 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_refresh_metadata", 2024-03-16T12:08:08.393 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:08.393 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.393 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:08.393 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:08.393 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.393 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.393 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.394 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-03-16T12:08:08.394 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.394 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.394 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.394 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.394 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-03-16T12:08:08.394 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-03-16T12:08:08.394 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:08.394 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.395 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:08.395 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-16T12:08:08.395 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.395 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.395 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.395 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-03-16T12:08:08.395 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.395 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.395 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.396 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.396 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-03-16T12:08:08.396 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-03-16T12:08:08.396 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:08.396 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.396 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:08.396 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-03-16T12:08:08.396 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.396 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.397 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-16T12:08:08.397 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-16T12:08:08.397 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-16T12:08:08.397 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-16T12:08:08.397 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-16T12:08:08.397 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-16T12:08:08.397 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-16T12:08:08.397 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:08.397 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:08.398 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.398 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.398 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.398 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.398 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-03-16T12:08:08.398 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-03-16T12:08:08.398 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:08.398 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.398 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:08.399 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:08.399 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.399 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.399 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.399 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:08.400 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.400 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.400 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.400 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.400 INFO:teuthology.orchestra.run.smithi029.stdout: "manage_etc_ceph_ceph_conf": { 2024-03-16T12:08:08.400 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-03-16T12:08:08.400 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:08.400 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.401 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:08.401 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:08.401 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.401 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.401 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.401 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-03-16T12:08:08.402 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.402 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.402 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.402 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.402 INFO:teuthology.orchestra.run.smithi029.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-03-16T12:08:08.402 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-03-16T12:08:08.402 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:08.403 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.403 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:08.403 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "*", 2024-03-16T12:08:08.403 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.403 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.403 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.404 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-03-16T12:08:08.404 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.404 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.404 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.404 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.404 INFO:teuthology.orchestra.run.smithi029.stdout: "max_count_per_host": { 2024-03-16T12:08:08.404 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "max_count_per_host", 2024-03-16T12:08:08.404 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-16T12:08:08.405 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.405 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:08.405 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "10", 2024-03-16T12:08:08.405 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.405 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.406 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.406 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "max number of daemons per service per host", 2024-03-16T12:08:08.406 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.406 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.406 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.407 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.407 INFO:teuthology.orchestra.run.smithi029.stdout: "max_osd_draining_count": { 2024-03-16T12:08:08.407 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "max_osd_draining_count", 2024-03-16T12:08:08.407 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-16T12:08:08.408 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.408 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:08.408 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "10", 2024-03-16T12:08:08.408 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.409 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.409 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.409 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-03-16T12:08:08.409 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.409 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.409 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.409 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.409 INFO:teuthology.orchestra.run.smithi029.stdout: "migration_current": { 2024-03-16T12:08:08.410 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "migration_current", 2024-03-16T12:08:08.410 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-16T12:08:08.410 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.410 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:08.410 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:08.410 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.410 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.410 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.411 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "internal - do not modify", 2024-03-16T12:08:08.411 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.411 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.411 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.411 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.411 INFO:teuthology.orchestra.run.smithi029.stdout: "mode": { 2024-03-16T12:08:08.411 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "mode", 2024-03-16T12:08:08.411 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:08.411 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.412 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:08.412 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "root", 2024-03-16T12:08:08.412 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.412 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.413 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-16T12:08:08.413 INFO:teuthology.orchestra.run.smithi029.stdout: "cephadm-package", 2024-03-16T12:08:08.413 INFO:teuthology.orchestra.run.smithi029.stdout: "root" 2024-03-16T12:08:08.413 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:08.414 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "mode for remote execution of cephadm", 2024-03-16T12:08:08.414 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.414 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.414 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.414 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.415 INFO:teuthology.orchestra.run.smithi029.stdout: "oob_default_addr": { 2024-03-16T12:08:08.415 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "oob_default_addr", 2024-03-16T12:08:08.415 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:08.415 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.415 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:08.415 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "169.254.1.1", 2024-03-16T12:08:08.415 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.415 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.415 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.415 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Default address for RedFish API (oob management).", 2024-03-16T12:08:08.416 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.416 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.416 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.416 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.416 INFO:teuthology.orchestra.run.smithi029.stdout: "prometheus_alerts_path": { 2024-03-16T12:08:08.416 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "prometheus_alerts_path", 2024-03-16T12:08:08.416 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:08.416 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.416 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:08.416 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-03-16T12:08:08.417 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.417 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.417 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.417 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-03-16T12:08:08.417 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.417 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.417 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.417 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.417 INFO:teuthology.orchestra.run.smithi029.stdout: "registry_insecure": { 2024-03-16T12:08:08.418 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "registry_insecure", 2024-03-16T12:08:08.418 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:08.418 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.418 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:08.418 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:08.418 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.418 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.418 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.419 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-03-16T12:08:08.419 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.419 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.419 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.419 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.419 INFO:teuthology.orchestra.run.smithi029.stdout: "registry_password": { 2024-03-16T12:08:08.419 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "registry_password", 2024-03-16T12:08:08.419 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:08.419 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.420 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:08.420 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:08.420 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.420 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.420 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.420 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-03-16T12:08:08.420 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.420 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.420 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.420 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.421 INFO:teuthology.orchestra.run.smithi029.stdout: "registry_url": { 2024-03-16T12:08:08.421 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "registry_url", 2024-03-16T12:08:08.421 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:08.421 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.421 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:08.421 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:08.421 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.421 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.422 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.422 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-03-16T12:08:08.422 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.422 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.422 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.422 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.422 INFO:teuthology.orchestra.run.smithi029.stdout: "registry_username": { 2024-03-16T12:08:08.422 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "registry_username", 2024-03-16T12:08:08.422 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:08.422 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.423 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:08.423 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:08.423 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.423 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.423 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.423 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-03-16T12:08:08.423 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.423 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.423 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.423 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.424 INFO:teuthology.orchestra.run.smithi029.stdout: "secure_monitoring_stack": { 2024-03-16T12:08:08.424 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "secure_monitoring_stack", 2024-03-16T12:08:08.424 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:08.424 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.424 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:08.424 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:08.424 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.424 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.424 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.424 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-03-16T12:08:08.425 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.425 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.425 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.425 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.425 INFO:teuthology.orchestra.run.smithi029.stdout: "service_discovery_port": { 2024-03-16T12:08:08.425 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "service_discovery_port", 2024-03-16T12:08:08.425 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-16T12:08:08.425 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.425 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:08.425 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "8765", 2024-03-16T12:08:08.426 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.426 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.426 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.426 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "cephadm service discovery port", 2024-03-16T12:08:08.426 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.426 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.426 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.426 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.426 INFO:teuthology.orchestra.run.smithi029.stdout: "ssh_config_file": { 2024-03-16T12:08:08.427 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "ssh_config_file", 2024-03-16T12:08:08.427 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:08.427 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.427 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:08.427 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:08.427 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.427 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.427 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.427 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-03-16T12:08:08.427 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.428 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.428 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.428 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.428 INFO:teuthology.orchestra.run.smithi029.stdout: "use_agent": { 2024-03-16T12:08:08.428 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "use_agent", 2024-03-16T12:08:08.428 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:08.428 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.428 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:08.428 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:08.429 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.429 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.429 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.429 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-03-16T12:08:08.429 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.429 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.429 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.429 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.429 INFO:teuthology.orchestra.run.smithi029.stdout: "use_repo_digest": { 2024-03-16T12:08:08.429 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "use_repo_digest", 2024-03-16T12:08:08.430 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:08.430 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.430 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:08.430 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-16T12:08:08.430 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.430 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.430 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.430 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-03-16T12:08:08.430 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.430 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.431 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.431 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.431 INFO:teuthology.orchestra.run.smithi029.stdout: "warn_on_failed_host_check": { 2024-03-16T12:08:08.431 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "warn_on_failed_host_check", 2024-03-16T12:08:08.431 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:08.431 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.431 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:08.431 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-16T12:08:08.431 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.431 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.432 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.432 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "raise a health warning if the host check fails", 2024-03-16T12:08:08.432 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.432 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.432 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.432 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.432 INFO:teuthology.orchestra.run.smithi029.stdout: "warn_on_stray_daemons": { 2024-03-16T12:08:08.432 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "warn_on_stray_daemons", 2024-03-16T12:08:08.432 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:08.432 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.433 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:08.433 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-16T12:08:08.433 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.433 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.433 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.433 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-03-16T12:08:08.433 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.434 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.434 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.434 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.434 INFO:teuthology.orchestra.run.smithi029.stdout: "warn_on_stray_hosts": { 2024-03-16T12:08:08.435 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "warn_on_stray_hosts", 2024-03-16T12:08:08.435 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:08.435 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.435 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:08.435 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-16T12:08:08.435 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.435 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.435 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.436 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-03-16T12:08:08.436 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.436 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.436 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.436 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:08.437 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:08.437 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.437 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-16T12:08:08.437 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "crash", 2024-03-16T12:08:08.437 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-03-16T12:08:08.437 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-03-16T12:08:08.437 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-03-16T12:08:08.437 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-03-16T12:08:08.437 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-03-16T12:08:08.438 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:08.438 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.438 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:08.439 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:08.439 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.439 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.439 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-16T12:08:08.439 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-16T12:08:08.439 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-16T12:08:08.439 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-16T12:08:08.439 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-16T12:08:08.440 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-16T12:08:08.440 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-16T12:08:08.440 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:08.440 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:08.440 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.441 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.441 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.441 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.441 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-03-16T12:08:08.441 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-03-16T12:08:08.441 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:08.442 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.442 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:08.442 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:08.442 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.443 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.443 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.443 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:08.443 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.443 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.444 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.444 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.444 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-03-16T12:08:08.444 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-03-16T12:08:08.445 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:08.445 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.445 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:08.445 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-03-16T12:08:08.445 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.446 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.446 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-16T12:08:08.446 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-16T12:08:08.446 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-16T12:08:08.446 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-16T12:08:08.446 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-16T12:08:08.446 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-16T12:08:08.447 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-16T12:08:08.447 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:08.447 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:08.447 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.447 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.447 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.448 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.448 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-03-16T12:08:08.448 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-03-16T12:08:08.448 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:08.448 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.448 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:08.448 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:08.448 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.448 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.449 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.449 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:08.449 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.449 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.449 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.449 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.449 INFO:teuthology.orchestra.run.smithi029.stdout: "retain_interval": { 2024-03-16T12:08:08.449 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "retain_interval", 2024-03-16T12:08:08.450 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "secs", 2024-03-16T12:08:08.450 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.450 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:08.450 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "31536000", 2024-03-16T12:08:08.450 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.450 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.450 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.451 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "how long to retain crashes before pruning them", 2024-03-16T12:08:08.451 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.451 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.451 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.451 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.451 INFO:teuthology.orchestra.run.smithi029.stdout: "warn_recent_interval": { 2024-03-16T12:08:08.451 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "warn_recent_interval", 2024-03-16T12:08:08.451 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "secs", 2024-03-16T12:08:08.451 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.452 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:08.452 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "1209600", 2024-03-16T12:08:08.452 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.452 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.452 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.452 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "time interval in which to warn about recent crashes", 2024-03-16T12:08:08.452 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.452 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.452 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.453 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:08.453 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:08.453 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.453 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-16T12:08:08.453 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "dashboard", 2024-03-16T12:08:08.453 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-03-16T12:08:08.453 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-03-16T12:08:08.453 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-03-16T12:08:08.454 INFO:teuthology.orchestra.run.smithi029.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-03-16T12:08:08.454 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-03-16T12:08:08.454 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-16T12:08:08.454 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.454 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:08.454 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "10", 2024-03-16T12:08:08.454 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.455 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.455 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.455 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:08.455 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.455 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.455 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.455 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.455 INFO:teuthology.orchestra.run.smithi029.stdout: "ALERTMANAGER_API_HOST": { 2024-03-16T12:08:08.456 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "ALERTMANAGER_API_HOST", 2024-03-16T12:08:08.456 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:08.456 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.456 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:08.456 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:08.456 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.456 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.456 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.456 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:08.456 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.457 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.457 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.457 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.457 INFO:teuthology.orchestra.run.smithi029.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-03-16T12:08:08.457 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-03-16T12:08:08.457 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:08.457 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.457 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:08.458 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-16T12:08:08.458 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.458 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.458 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.458 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:08.458 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.458 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.458 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.459 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.459 INFO:teuthology.orchestra.run.smithi029.stdout: "AUDIT_API_ENABLED": { 2024-03-16T12:08:08.459 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "AUDIT_API_ENABLED", 2024-03-16T12:08:08.459 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:08.459 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.459 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:08.459 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:08.459 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.459 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.460 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.460 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:08.460 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.460 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.460 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.460 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.460 INFO:teuthology.orchestra.run.smithi029.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-03-16T12:08:08.460 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-03-16T12:08:08.460 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:08.461 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.461 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:08.461 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-16T12:08:08.461 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.461 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.461 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.461 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:08.461 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.461 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.462 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.462 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.462 INFO:teuthology.orchestra.run.smithi029.stdout: "ENABLE_BROWSABLE_API": { 2024-03-16T12:08:08.462 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "ENABLE_BROWSABLE_API", 2024-03-16T12:08:08.462 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:08.462 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.462 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:08.462 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-16T12:08:08.462 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.462 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.463 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.463 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:08.463 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.463 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.463 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.463 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.463 INFO:teuthology.orchestra.run.smithi029.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-03-16T12:08:08.463 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-03-16T12:08:08.464 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:08.464 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.464 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:08.464 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-16T12:08:08.464 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.464 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.464 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.464 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:08.464 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.464 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.465 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.465 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.465 INFO:teuthology.orchestra.run.smithi029.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-03-16T12:08:08.465 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-03-16T12:08:08.465 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:08.465 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.465 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:08.465 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-16T12:08:08.465 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.466 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.466 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.466 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:08.466 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.466 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.466 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.466 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.466 INFO:teuthology.orchestra.run.smithi029.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-03-16T12:08:08.467 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-03-16T12:08:08.467 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:08.467 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.467 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:08.467 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-16T12:08:08.467 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.468 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.468 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.468 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:08.468 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.468 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.468 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.468 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.468 INFO:teuthology.orchestra.run.smithi029.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-03-16T12:08:08.468 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-03-16T12:08:08.469 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:08.469 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.469 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:08.469 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-16T12:08:08.469 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.470 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.470 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.470 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:08.470 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.470 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.470 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.470 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.471 INFO:teuthology.orchestra.run.smithi029.stdout: "FEATURE_TOGGLE_NFS": { 2024-03-16T12:08:08.471 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-03-16T12:08:08.471 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:08.471 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.471 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:08.471 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-16T12:08:08.472 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.472 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.472 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.472 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:08.472 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.472 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.472 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.472 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.473 INFO:teuthology.orchestra.run.smithi029.stdout: "FEATURE_TOGGLE_RBD": { 2024-03-16T12:08:08.473 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-03-16T12:08:08.473 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:08.473 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.473 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:08.473 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-16T12:08:08.473 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.474 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.474 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.474 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:08.474 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.474 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.474 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.475 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.475 INFO:teuthology.orchestra.run.smithi029.stdout: "FEATURE_TOGGLE_RGW": { 2024-03-16T12:08:08.475 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-03-16T12:08:08.475 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:08.475 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.475 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:08.475 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-16T12:08:08.475 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.475 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.476 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.476 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:08.476 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.476 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.476 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.477 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.477 INFO:teuthology.orchestra.run.smithi029.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-03-16T12:08:08.477 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-03-16T12:08:08.477 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:08.477 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.478 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:08.478 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:08.478 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.478 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.478 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.478 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:08.478 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.479 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.479 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.479 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.479 INFO:teuthology.orchestra.run.smithi029.stdout: "GRAFANA_API_PASSWORD": { 2024-03-16T12:08:08.479 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "GRAFANA_API_PASSWORD", 2024-03-16T12:08:08.479 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:08.479 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.479 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:08.479 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "admin", 2024-03-16T12:08:08.480 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.480 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.480 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.480 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:08.481 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.481 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.481 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.481 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.481 INFO:teuthology.orchestra.run.smithi029.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-03-16T12:08:08.481 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-03-16T12:08:08.481 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:08.482 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.482 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:08.482 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-16T12:08:08.482 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.482 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.482 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.482 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:08.482 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.482 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.482 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.483 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.483 INFO:teuthology.orchestra.run.smithi029.stdout: "GRAFANA_API_URL": { 2024-03-16T12:08:08.483 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "GRAFANA_API_URL", 2024-03-16T12:08:08.483 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:08.483 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.483 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:08.483 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:08.483 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.483 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.483 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.484 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:08.484 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.484 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.484 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.484 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.484 INFO:teuthology.orchestra.run.smithi029.stdout: "GRAFANA_API_USERNAME": { 2024-03-16T12:08:08.484 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "GRAFANA_API_USERNAME", 2024-03-16T12:08:08.484 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:08.484 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.485 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:08.485 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "admin", 2024-03-16T12:08:08.485 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.485 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.485 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.485 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:08.485 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.485 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.485 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.486 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.486 INFO:teuthology.orchestra.run.smithi029.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-03-16T12:08:08.486 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-03-16T12:08:08.486 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:08.486 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.486 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:08.486 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:08.486 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.486 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.487 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.487 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:08.487 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.487 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.487 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.487 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.487 INFO:teuthology.orchestra.run.smithi029.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-03-16T12:08:08.487 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-03-16T12:08:08.487 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:08.487 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.488 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:08.488 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:08.488 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.488 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.488 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.488 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:08.488 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.488 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.489 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.489 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.489 INFO:teuthology.orchestra.run.smithi029.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-03-16T12:08:08.489 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-03-16T12:08:08.489 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:08.489 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.489 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:08.490 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-16T12:08:08.490 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.490 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.490 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.490 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:08.490 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.490 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.490 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.491 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.491 INFO:teuthology.orchestra.run.smithi029.stdout: "ISSUE_TRACKER_API_KEY": { 2024-03-16T12:08:08.491 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-03-16T12:08:08.491 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:08.491 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.491 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:08.491 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:08.491 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.491 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.492 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.492 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:08.492 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.492 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.492 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.492 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.492 INFO:teuthology.orchestra.run.smithi029.stdout: "MULTICLUSTER_CONFIG": { 2024-03-16T12:08:08.492 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "MULTICLUSTER_CONFIG", 2024-03-16T12:08:08.492 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:08.493 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.493 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:08.493 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "{}", 2024-03-16T12:08:08.493 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.493 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.493 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.493 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:08.493 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.493 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.494 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.494 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.494 INFO:teuthology.orchestra.run.smithi029.stdout: "PROMETHEUS_API_HOST": { 2024-03-16T12:08:08.494 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "PROMETHEUS_API_HOST", 2024-03-16T12:08:08.494 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:08.494 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.494 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:08.494 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:08.494 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.495 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.495 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.495 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:08.495 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.495 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.495 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.495 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.495 INFO:teuthology.orchestra.run.smithi029.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-03-16T12:08:08.495 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-03-16T12:08:08.496 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:08.496 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.496 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:08.496 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-16T12:08:08.496 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.496 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.496 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.496 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:08.496 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.497 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.497 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.497 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.497 INFO:teuthology.orchestra.run.smithi029.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-03-16T12:08:08.497 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-03-16T12:08:08.497 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:08.497 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.497 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:08.497 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:08.498 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.498 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.498 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.498 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:08.498 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.498 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.498 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.498 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.499 INFO:teuthology.orchestra.run.smithi029.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-03-16T12:08:08.499 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-03-16T12:08:08.499 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:08.499 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.499 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:08.499 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:08.499 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.499 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.499 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.500 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:08.500 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.500 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.500 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.500 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.500 INFO:teuthology.orchestra.run.smithi029.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-03-16T12:08:08.500 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-03-16T12:08:08.500 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:08.500 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.501 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:08.501 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-16T12:08:08.501 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.501 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.502 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.502 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:08.502 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.502 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.502 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.502 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.502 INFO:teuthology.orchestra.run.smithi029.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-03-16T12:08:08.502 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-03-16T12:08:08.503 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:08.503 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.503 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:08.503 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-16T12:08:08.504 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.504 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.504 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.504 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:08.504 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.504 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.504 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.504 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.505 INFO:teuthology.orchestra.run.smithi029.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-03-16T12:08:08.505 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-03-16T12:08:08.505 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:08.505 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.505 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:08.506 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:08.506 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.506 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.506 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.506 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:08.506 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.506 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.507 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.507 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.507 INFO:teuthology.orchestra.run.smithi029.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-03-16T12:08:08.507 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-03-16T12:08:08.508 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:08.508 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.508 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:08.508 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:08.508 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.508 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.509 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.509 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:08.509 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.509 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.509 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.509 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.509 INFO:teuthology.orchestra.run.smithi029.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-03-16T12:08:08.510 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-03-16T12:08:08.510 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:08.510 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.510 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:08.511 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:08.511 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.511 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.511 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.512 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:08.512 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.512 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.512 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.513 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.513 INFO:teuthology.orchestra.run.smithi029.stdout: "PWD_POLICY_ENABLED": { 2024-03-16T12:08:08.513 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "PWD_POLICY_ENABLED", 2024-03-16T12:08:08.513 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:08.513 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.513 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:08.513 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-16T12:08:08.514 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.514 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.514 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.514 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:08.514 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.514 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.514 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.514 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.514 INFO:teuthology.orchestra.run.smithi029.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-03-16T12:08:08.515 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-03-16T12:08:08.515 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:08.515 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.515 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:08.515 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-03-16T12:08:08.515 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.515 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.515 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.515 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:08.516 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.516 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.516 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.516 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.516 INFO:teuthology.orchestra.run.smithi029.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-03-16T12:08:08.516 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-03-16T12:08:08.516 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-16T12:08:08.516 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.517 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:08.517 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "10", 2024-03-16T12:08:08.517 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.517 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.517 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.517 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:08.517 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.517 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.517 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.518 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.518 INFO:teuthology.orchestra.run.smithi029.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-03-16T12:08:08.518 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-03-16T12:08:08.518 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-16T12:08:08.518 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.518 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:08.518 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "8", 2024-03-16T12:08:08.518 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.518 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.519 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.519 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:08.519 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.519 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.519 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.519 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.519 INFO:teuthology.orchestra.run.smithi029.stdout: "REST_REQUESTS_TIMEOUT": { 2024-03-16T12:08:08.519 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-03-16T12:08:08.519 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-16T12:08:08.520 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.520 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:08.520 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "45", 2024-03-16T12:08:08.520 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.520 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.520 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.520 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:08.520 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.520 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.521 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.521 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.521 INFO:teuthology.orchestra.run.smithi029.stdout: "RGW_API_ACCESS_KEY": { 2024-03-16T12:08:08.521 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "RGW_API_ACCESS_KEY", 2024-03-16T12:08:08.521 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:08.521 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.521 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:08.521 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:08.521 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.522 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.522 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.522 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:08.522 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.522 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.522 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.522 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.522 INFO:teuthology.orchestra.run.smithi029.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-03-16T12:08:08.522 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-03-16T12:08:08.523 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:08.523 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.523 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:08.523 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "admin", 2024-03-16T12:08:08.523 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.523 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.523 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.523 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:08.523 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.524 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.524 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.524 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.524 INFO:teuthology.orchestra.run.smithi029.stdout: "RGW_API_SECRET_KEY": { 2024-03-16T12:08:08.524 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "RGW_API_SECRET_KEY", 2024-03-16T12:08:08.524 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:08.524 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.524 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:08.524 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:08.525 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.525 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.525 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.525 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:08.525 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.525 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.525 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.525 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.525 INFO:teuthology.orchestra.run.smithi029.stdout: "RGW_API_SSL_VERIFY": { 2024-03-16T12:08:08.526 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "RGW_API_SSL_VERIFY", 2024-03-16T12:08:08.526 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:08.526 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.526 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:08.526 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-16T12:08:08.526 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.526 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.526 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.527 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:08.527 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.527 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.527 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.527 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.527 INFO:teuthology.orchestra.run.smithi029.stdout: "UNSAFE_TLS_v1_2": { 2024-03-16T12:08:08.527 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "UNSAFE_TLS_v1_2", 2024-03-16T12:08:08.527 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:08.527 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.528 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:08.528 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:08.528 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.528 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.528 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.528 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:08.528 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.528 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.528 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.528 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.529 INFO:teuthology.orchestra.run.smithi029.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-03-16T12:08:08.529 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-03-16T12:08:08.529 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-16T12:08:08.529 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.529 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:08.529 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "0", 2024-03-16T12:08:08.529 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.529 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.530 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.530 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:08.530 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.530 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.530 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.530 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.530 INFO:teuthology.orchestra.run.smithi029.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-03-16T12:08:08.530 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-03-16T12:08:08.530 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-16T12:08:08.531 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.531 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:08.531 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "10", 2024-03-16T12:08:08.531 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.531 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.531 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.531 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:08.531 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.531 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.532 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.532 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.532 INFO:teuthology.orchestra.run.smithi029.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-03-16T12:08:08.532 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-03-16T12:08:08.532 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-16T12:08:08.532 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.532 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:08.532 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "5", 2024-03-16T12:08:08.532 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.532 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.533 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.533 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:08.533 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.533 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.533 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.533 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.534 INFO:teuthology.orchestra.run.smithi029.stdout: "cross_origin_url": { 2024-03-16T12:08:08.534 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "cross_origin_url", 2024-03-16T12:08:08.534 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:08.534 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.534 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:08.534 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:08.534 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.534 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.535 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.535 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:08.535 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.535 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.535 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.535 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.535 INFO:teuthology.orchestra.run.smithi029.stdout: "crt_file": { 2024-03-16T12:08:08.536 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "crt_file", 2024-03-16T12:08:08.536 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:08.536 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.536 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:08.536 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:08.536 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.536 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.537 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.537 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:08.537 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.537 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.537 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.537 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.537 INFO:teuthology.orchestra.run.smithi029.stdout: "debug": { 2024-03-16T12:08:08.537 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "debug", 2024-03-16T12:08:08.538 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:08.538 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.538 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:08.538 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:08.538 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.538 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.538 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.539 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Enable/disable debug options", 2024-03-16T12:08:08.539 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.539 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.539 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.539 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.539 INFO:teuthology.orchestra.run.smithi029.stdout: "jwt_token_ttl": { 2024-03-16T12:08:08.540 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "jwt_token_ttl", 2024-03-16T12:08:08.540 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-16T12:08:08.540 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.540 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:08.540 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "28800", 2024-03-16T12:08:08.540 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.540 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.540 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.541 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:08.541 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.541 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.541 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.541 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.541 INFO:teuthology.orchestra.run.smithi029.stdout: "key_file": { 2024-03-16T12:08:08.542 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "key_file", 2024-03-16T12:08:08.542 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:08.542 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.542 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:08.542 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:08.543 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.543 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.543 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.543 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:08.543 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.543 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.544 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.544 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.544 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-03-16T12:08:08.544 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-03-16T12:08:08.544 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:08.544 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.545 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:08.545 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:08.545 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.545 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.545 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-16T12:08:08.545 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-16T12:08:08.546 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-16T12:08:08.546 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-16T12:08:08.546 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-16T12:08:08.546 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-16T12:08:08.546 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-16T12:08:08.546 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:08.546 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:08.546 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.547 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.547 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.547 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.547 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-03-16T12:08:08.547 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-03-16T12:08:08.547 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:08.547 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.547 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:08.547 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:08.548 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.548 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.548 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.548 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:08.548 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.548 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.548 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.548 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.548 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-03-16T12:08:08.549 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-03-16T12:08:08.549 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:08.549 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.549 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:08.549 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-03-16T12:08:08.549 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.549 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.549 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-16T12:08:08.549 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-16T12:08:08.550 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-16T12:08:08.550 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-16T12:08:08.550 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-16T12:08:08.550 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-16T12:08:08.550 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-16T12:08:08.550 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:08.550 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:08.550 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.550 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.551 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.551 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.551 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-03-16T12:08:08.551 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-03-16T12:08:08.551 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:08.551 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.551 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:08.551 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:08.551 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.552 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.552 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.552 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:08.552 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.552 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.552 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.552 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.552 INFO:teuthology.orchestra.run.smithi029.stdout: "motd": { 2024-03-16T12:08:08.553 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "motd", 2024-03-16T12:08:08.553 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:08.553 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.553 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:08.553 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:08.553 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.553 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.553 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.553 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "The message of the day", 2024-03-16T12:08:08.554 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.554 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.554 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.554 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.554 INFO:teuthology.orchestra.run.smithi029.stdout: "redirect_resolve_ip_addr": { 2024-03-16T12:08:08.554 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "redirect_resolve_ip_addr", 2024-03-16T12:08:08.554 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:08.554 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.555 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:08.555 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:08.555 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.555 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.555 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.555 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:08.555 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.555 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.555 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.556 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.556 INFO:teuthology.orchestra.run.smithi029.stdout: "server_addr": { 2024-03-16T12:08:08.556 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "server_addr", 2024-03-16T12:08:08.556 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:08.556 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.556 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:08.556 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "::", 2024-03-16T12:08:08.556 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.557 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.557 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.557 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:08.557 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.557 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.557 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.557 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.557 INFO:teuthology.orchestra.run.smithi029.stdout: "server_port": { 2024-03-16T12:08:08.558 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "server_port", 2024-03-16T12:08:08.558 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-16T12:08:08.558 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.558 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:08.558 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "8080", 2024-03-16T12:08:08.558 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.558 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.558 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.558 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:08.559 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.559 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.559 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.559 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.559 INFO:teuthology.orchestra.run.smithi029.stdout: "ssl": { 2024-03-16T12:08:08.559 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "ssl", 2024-03-16T12:08:08.559 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:08.559 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.559 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:08.560 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-16T12:08:08.560 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.560 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.560 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.560 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:08.560 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.560 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.560 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.561 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.561 INFO:teuthology.orchestra.run.smithi029.stdout: "ssl_server_port": { 2024-03-16T12:08:08.561 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "ssl_server_port", 2024-03-16T12:08:08.561 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-16T12:08:08.561 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.561 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:08.561 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "8443", 2024-03-16T12:08:08.561 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.561 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.562 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.562 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:08.562 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.562 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.562 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.562 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.562 INFO:teuthology.orchestra.run.smithi029.stdout: "standby_behaviour": { 2024-03-16T12:08:08.562 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "standby_behaviour", 2024-03-16T12:08:08.563 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:08.563 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.563 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:08.563 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "redirect", 2024-03-16T12:08:08.563 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.563 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.563 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-16T12:08:08.563 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-16T12:08:08.563 INFO:teuthology.orchestra.run.smithi029.stdout: "redirect" 2024-03-16T12:08:08.563 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:08.564 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:08.564 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.564 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.564 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.564 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.564 INFO:teuthology.orchestra.run.smithi029.stdout: "standby_error_status_code": { 2024-03-16T12:08:08.564 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "standby_error_status_code", 2024-03-16T12:08:08.564 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-16T12:08:08.564 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.565 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:08.565 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "500", 2024-03-16T12:08:08.565 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "400", 2024-03-16T12:08:08.565 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "599", 2024-03-16T12:08:08.565 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.565 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:08.565 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.565 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.565 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.566 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.566 INFO:teuthology.orchestra.run.smithi029.stdout: "url_prefix": { 2024-03-16T12:08:08.566 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "url_prefix", 2024-03-16T12:08:08.566 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:08.566 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.566 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:08.566 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:08.566 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.567 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.567 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.567 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:08.567 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.567 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.567 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.567 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:08.567 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:08.567 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.568 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-16T12:08:08.568 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "devicehealth", 2024-03-16T12:08:08.568 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-03-16T12:08:08.568 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-03-16T12:08:08.568 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-03-16T12:08:08.568 INFO:teuthology.orchestra.run.smithi029.stdout: "enable_monitoring": { 2024-03-16T12:08:08.568 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "enable_monitoring", 2024-03-16T12:08:08.568 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:08.568 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.569 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:08.569 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-16T12:08:08.569 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.569 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.569 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.569 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "monitor device health metrics", 2024-03-16T12:08:08.570 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.570 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.570 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.570 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.570 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-03-16T12:08:08.570 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-03-16T12:08:08.570 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:08.570 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.571 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:08.571 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:08.571 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.571 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.571 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-16T12:08:08.571 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-16T12:08:08.572 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-16T12:08:08.572 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-16T12:08:08.572 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-16T12:08:08.572 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-16T12:08:08.572 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-16T12:08:08.572 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:08.572 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:08.572 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.572 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.573 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.573 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.573 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-03-16T12:08:08.573 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-03-16T12:08:08.573 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:08.573 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.574 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:08.574 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:08.574 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.574 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.574 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.574 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:08.574 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.574 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.575 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.575 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.575 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-03-16T12:08:08.575 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-03-16T12:08:08.575 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:08.575 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.576 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:08.576 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-03-16T12:08:08.576 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.576 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.576 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-16T12:08:08.576 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-16T12:08:08.576 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-16T12:08:08.576 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-16T12:08:08.577 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-16T12:08:08.577 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-16T12:08:08.577 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-16T12:08:08.577 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:08.577 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:08.577 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.578 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.578 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.578 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.578 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-03-16T12:08:08.578 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-03-16T12:08:08.578 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:08.578 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.579 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:08.579 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:08.579 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.579 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.579 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.579 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:08.580 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.580 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.580 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.580 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.580 INFO:teuthology.orchestra.run.smithi029.stdout: "mark_out_threshold": { 2024-03-16T12:08:08.580 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "mark_out_threshold", 2024-03-16T12:08:08.581 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "secs", 2024-03-16T12:08:08.581 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.581 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:08.581 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "2419200", 2024-03-16T12:08:08.581 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.581 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.581 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.581 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-03-16T12:08:08.582 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.582 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.582 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.582 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.582 INFO:teuthology.orchestra.run.smithi029.stdout: "pool_name": { 2024-03-16T12:08:08.582 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "pool_name", 2024-03-16T12:08:08.582 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:08.582 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.583 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:08.583 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "device_health_metrics", 2024-03-16T12:08:08.583 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.583 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.583 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.583 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "name of pool in which to store device health metrics", 2024-03-16T12:08:08.583 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.583 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.583 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.583 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.584 INFO:teuthology.orchestra.run.smithi029.stdout: "retention_period": { 2024-03-16T12:08:08.584 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "retention_period", 2024-03-16T12:08:08.584 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "secs", 2024-03-16T12:08:08.584 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.584 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:08.584 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "15552000", 2024-03-16T12:08:08.584 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.584 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.584 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.585 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "how long to retain device health metrics", 2024-03-16T12:08:08.585 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.585 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.585 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.585 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.585 INFO:teuthology.orchestra.run.smithi029.stdout: "scrape_frequency": { 2024-03-16T12:08:08.585 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "scrape_frequency", 2024-03-16T12:08:08.585 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "secs", 2024-03-16T12:08:08.585 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.586 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:08.586 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "86400", 2024-03-16T12:08:08.586 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.586 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.586 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.586 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "how frequently to scrape device health metrics", 2024-03-16T12:08:08.586 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.587 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.587 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.587 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.587 INFO:teuthology.orchestra.run.smithi029.stdout: "self_heal": { 2024-03-16T12:08:08.587 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "self_heal", 2024-03-16T12:08:08.587 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:08.587 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.587 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:08.587 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-16T12:08:08.588 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.588 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.588 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.588 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-03-16T12:08:08.588 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.588 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.588 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.588 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.588 INFO:teuthology.orchestra.run.smithi029.stdout: "sleep_interval": { 2024-03-16T12:08:08.589 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "sleep_interval", 2024-03-16T12:08:08.589 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "secs", 2024-03-16T12:08:08.589 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.589 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:08.589 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "600", 2024-03-16T12:08:08.589 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.589 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.589 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.589 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "how frequently to wake up and check device health", 2024-03-16T12:08:08.590 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.590 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.590 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.590 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.590 INFO:teuthology.orchestra.run.smithi029.stdout: "warn_threshold": { 2024-03-16T12:08:08.590 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "warn_threshold", 2024-03-16T12:08:08.590 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "secs", 2024-03-16T12:08:08.590 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.590 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:08.591 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "7257600", 2024-03-16T12:08:08.591 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.591 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.591 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.591 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-03-16T12:08:08.591 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.591 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.591 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.591 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:08.592 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:08.592 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.592 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-16T12:08:08.592 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "diskprediction_local", 2024-03-16T12:08:08.592 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-03-16T12:08:08.592 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-03-16T12:08:08.592 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-03-16T12:08:08.592 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-03-16T12:08:08.593 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-03-16T12:08:08.593 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:08.593 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.593 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:08.593 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:08.593 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.593 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.593 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-16T12:08:08.593 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-16T12:08:08.594 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-16T12:08:08.594 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-16T12:08:08.594 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-16T12:08:08.594 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-16T12:08:08.594 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-16T12:08:08.594 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:08.594 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:08.594 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.594 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.595 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.595 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.595 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-03-16T12:08:08.595 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-03-16T12:08:08.595 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:08.595 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.595 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:08.595 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:08.596 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.596 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.596 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.596 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:08.596 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.596 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.596 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.596 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.596 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-03-16T12:08:08.597 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-03-16T12:08:08.597 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:08.597 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.597 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:08.597 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-03-16T12:08:08.597 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.597 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.597 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-16T12:08:08.597 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-16T12:08:08.598 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-16T12:08:08.598 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-16T12:08:08.598 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-16T12:08:08.598 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-16T12:08:08.598 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-16T12:08:08.598 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:08.598 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:08.598 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.598 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.599 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.599 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.599 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-03-16T12:08:08.599 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-03-16T12:08:08.599 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:08.599 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.599 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:08.599 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:08.599 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.600 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.600 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.600 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:08.600 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.600 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.600 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.600 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.601 INFO:teuthology.orchestra.run.smithi029.stdout: "predict_interval": { 2024-03-16T12:08:08.601 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "predict_interval", 2024-03-16T12:08:08.601 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:08.601 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.601 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:08.601 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "86400", 2024-03-16T12:08:08.601 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.602 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.602 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.602 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:08.602 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.602 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.602 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.602 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.602 INFO:teuthology.orchestra.run.smithi029.stdout: "predictor_model": { 2024-03-16T12:08:08.603 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "predictor_model", 2024-03-16T12:08:08.603 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:08.603 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.603 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:08.603 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "prophetstor", 2024-03-16T12:08:08.603 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.604 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.604 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.604 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:08.604 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.604 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.604 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.604 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.604 INFO:teuthology.orchestra.run.smithi029.stdout: "sleep_interval": { 2024-03-16T12:08:08.604 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "sleep_interval", 2024-03-16T12:08:08.605 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:08.605 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.605 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:08.605 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "600", 2024-03-16T12:08:08.605 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.605 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.606 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.606 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:08.606 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.606 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.606 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.606 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:08.606 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:08.607 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.607 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-16T12:08:08.607 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "influx", 2024-03-16T12:08:08.607 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": false, 2024-03-16T12:08:08.607 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "influxdb python module not found", 2024-03-16T12:08:08.607 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-03-16T12:08:08.608 INFO:teuthology.orchestra.run.smithi029.stdout: "batch_size": { 2024-03-16T12:08:08.608 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "batch_size", 2024-03-16T12:08:08.608 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-16T12:08:08.608 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.608 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:08.608 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "5000", 2024-03-16T12:08:08.608 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.609 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.609 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.609 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-03-16T12:08:08.609 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.609 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.609 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.609 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.610 INFO:teuthology.orchestra.run.smithi029.stdout: "database": { 2024-03-16T12:08:08.610 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "database", 2024-03-16T12:08:08.610 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:08.610 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.610 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:08.611 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "ceph", 2024-03-16T12:08:08.611 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.611 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.611 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.611 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-03-16T12:08:08.611 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.612 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.612 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.612 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.612 INFO:teuthology.orchestra.run.smithi029.stdout: "hostname": { 2024-03-16T12:08:08.612 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "hostname", 2024-03-16T12:08:08.612 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:08.613 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.613 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:08.613 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:08.613 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.613 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.613 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.613 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "InfluxDB server hostname", 2024-03-16T12:08:08.614 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.614 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.614 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.614 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.614 INFO:teuthology.orchestra.run.smithi029.stdout: "interval": { 2024-03-16T12:08:08.614 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "interval", 2024-03-16T12:08:08.614 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "secs", 2024-03-16T12:08:08.614 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.615 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:08.615 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "30", 2024-03-16T12:08:08.615 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "5", 2024-03-16T12:08:08.615 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.615 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.615 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-03-16T12:08:08.615 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.615 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.616 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.616 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.616 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-03-16T12:08:08.616 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-03-16T12:08:08.616 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:08.616 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.616 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:08.616 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:08.616 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.617 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.617 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-16T12:08:08.617 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-16T12:08:08.617 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-16T12:08:08.617 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-16T12:08:08.617 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-16T12:08:08.617 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-16T12:08:08.617 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-16T12:08:08.617 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:08.618 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:08.618 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.618 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.618 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.618 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.618 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-03-16T12:08:08.618 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-03-16T12:08:08.618 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:08.619 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.619 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:08.619 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:08.619 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.619 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.619 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.619 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:08.619 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.619 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.620 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.620 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.620 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-03-16T12:08:08.620 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-03-16T12:08:08.620 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:08.620 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.620 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:08.620 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-03-16T12:08:08.620 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.621 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.621 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-16T12:08:08.621 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-16T12:08:08.621 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-16T12:08:08.621 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-16T12:08:08.621 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-16T12:08:08.621 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-16T12:08:08.621 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-16T12:08:08.621 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:08.621 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:08.622 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.622 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.622 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.622 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.622 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-03-16T12:08:08.622 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-03-16T12:08:08.622 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:08.622 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.623 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:08.623 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:08.623 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.623 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.623 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.623 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:08.623 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.623 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.623 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.624 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.624 INFO:teuthology.orchestra.run.smithi029.stdout: "password": { 2024-03-16T12:08:08.624 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "password", 2024-03-16T12:08:08.624 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:08.624 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.624 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:08.624 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:08.624 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.624 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.625 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.625 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "password of InfluxDB server user", 2024-03-16T12:08:08.625 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.625 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.625 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.625 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.625 INFO:teuthology.orchestra.run.smithi029.stdout: "port": { 2024-03-16T12:08:08.625 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "port", 2024-03-16T12:08:08.625 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-16T12:08:08.626 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.626 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:08.626 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "8086", 2024-03-16T12:08:08.626 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.626 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.626 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.626 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "InfluxDB server port", 2024-03-16T12:08:08.626 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.627 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.627 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.627 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.627 INFO:teuthology.orchestra.run.smithi029.stdout: "ssl": { 2024-03-16T12:08:08.627 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "ssl", 2024-03-16T12:08:08.627 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:08.627 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.627 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:08.628 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "false", 2024-03-16T12:08:08.628 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.628 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.628 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.628 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-03-16T12:08:08.628 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.628 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.628 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.629 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.629 INFO:teuthology.orchestra.run.smithi029.stdout: "threads": { 2024-03-16T12:08:08.629 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "threads", 2024-03-16T12:08:08.629 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-16T12:08:08.629 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.629 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:08.629 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "5", 2024-03-16T12:08:08.629 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "1", 2024-03-16T12:08:08.629 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "32", 2024-03-16T12:08:08.630 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.630 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-03-16T12:08:08.630 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.630 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.630 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.630 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.630 INFO:teuthology.orchestra.run.smithi029.stdout: "username": { 2024-03-16T12:08:08.630 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "username", 2024-03-16T12:08:08.630 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:08.631 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.631 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:08.631 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:08.631 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.631 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.631 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.631 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "username of InfluxDB server user", 2024-03-16T12:08:08.631 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.631 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.631 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.632 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.632 INFO:teuthology.orchestra.run.smithi029.stdout: "verify_ssl": { 2024-03-16T12:08:08.632 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "verify_ssl", 2024-03-16T12:08:08.632 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:08.632 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.632 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:08.632 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "true", 2024-03-16T12:08:08.632 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.632 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.633 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.633 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-03-16T12:08:08.633 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.633 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.633 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.633 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:08.633 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:08.633 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.633 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-16T12:08:08.634 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "insights", 2024-03-16T12:08:08.634 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-03-16T12:08:08.634 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-03-16T12:08:08.634 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-03-16T12:08:08.634 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-03-16T12:08:08.635 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-03-16T12:08:08.635 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:08.635 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.635 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:08.635 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:08.635 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.635 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.635 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-16T12:08:08.635 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-16T12:08:08.636 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-16T12:08:08.636 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-16T12:08:08.636 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-16T12:08:08.636 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-16T12:08:08.636 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-16T12:08:08.636 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:08.636 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:08.637 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.637 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.637 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.637 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.637 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-03-16T12:08:08.637 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-03-16T12:08:08.637 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:08.637 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.638 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:08.638 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:08.638 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.638 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.638 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.638 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:08.638 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.638 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.638 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.639 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.639 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-03-16T12:08:08.639 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-03-16T12:08:08.639 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:08.639 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.639 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:08.639 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-03-16T12:08:08.639 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.639 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.640 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-16T12:08:08.640 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-16T12:08:08.640 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-16T12:08:08.640 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-16T12:08:08.640 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-16T12:08:08.640 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-16T12:08:08.640 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-16T12:08:08.640 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:08.640 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:08.641 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.641 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.641 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.641 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.641 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-03-16T12:08:08.641 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-03-16T12:08:08.641 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:08.641 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.641 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:08.642 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:08.642 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.642 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.642 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.642 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:08.643 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.643 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.643 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.643 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:08.643 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:08.643 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.643 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-16T12:08:08.643 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "iostat", 2024-03-16T12:08:08.644 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-03-16T12:08:08.644 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-03-16T12:08:08.644 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-03-16T12:08:08.644 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-03-16T12:08:08.644 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-03-16T12:08:08.644 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:08.644 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.645 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:08.645 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:08.645 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.645 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.645 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-16T12:08:08.645 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-16T12:08:08.645 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-16T12:08:08.646 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-16T12:08:08.646 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-16T12:08:08.646 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-16T12:08:08.646 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-16T12:08:08.646 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:08.646 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:08.647 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.647 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.647 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.647 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.647 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-03-16T12:08:08.647 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-03-16T12:08:08.648 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:08.648 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.648 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:08.648 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:08.648 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.648 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.648 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.649 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:08.649 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.649 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.649 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.649 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.649 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-03-16T12:08:08.650 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-03-16T12:08:08.650 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:08.650 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.650 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:08.650 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-03-16T12:08:08.650 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.650 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.651 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-16T12:08:08.651 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-16T12:08:08.651 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-16T12:08:08.651 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-16T12:08:08.651 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-16T12:08:08.651 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-16T12:08:08.651 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-16T12:08:08.651 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:08.651 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:08.652 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.652 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.652 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.652 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.652 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-03-16T12:08:08.652 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-03-16T12:08:08.652 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:08.652 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.652 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:08.652 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:08.653 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.653 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.653 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.653 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:08.653 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.653 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.653 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.653 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:08.653 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:08.654 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.654 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-16T12:08:08.654 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "k8sevents", 2024-03-16T12:08:08.654 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-03-16T12:08:08.654 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-03-16T12:08:08.654 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-03-16T12:08:08.654 INFO:teuthology.orchestra.run.smithi029.stdout: "ceph_event_retention_days": { 2024-03-16T12:08:08.654 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "ceph_event_retention_days", 2024-03-16T12:08:08.655 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-16T12:08:08.655 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.655 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:08.655 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "7", 2024-03-16T12:08:08.655 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.655 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.655 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.655 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Days to hold ceph event information within local cache", 2024-03-16T12:08:08.655 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.655 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.656 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.656 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.656 INFO:teuthology.orchestra.run.smithi029.stdout: "config_check_secs": { 2024-03-16T12:08:08.656 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "config_check_secs", 2024-03-16T12:08:08.656 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-16T12:08:08.656 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.656 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:08.656 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "10", 2024-03-16T12:08:08.657 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "10", 2024-03-16T12:08:08.657 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.657 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.657 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-03-16T12:08:08.657 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.657 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.657 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.657 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.658 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-03-16T12:08:08.658 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-03-16T12:08:08.658 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:08.658 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.658 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:08.658 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:08.658 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.658 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.659 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-16T12:08:08.659 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-16T12:08:08.659 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-16T12:08:08.659 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-16T12:08:08.659 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-16T12:08:08.659 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-16T12:08:08.659 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-16T12:08:08.659 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:08.659 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:08.660 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.660 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.660 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.660 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.660 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-03-16T12:08:08.660 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-03-16T12:08:08.660 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:08.660 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.660 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:08.660 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:08.661 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.661 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.661 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.661 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:08.661 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.661 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.661 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.661 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.661 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-03-16T12:08:08.661 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-03-16T12:08:08.662 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:08.662 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.662 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:08.662 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-03-16T12:08:08.662 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.662 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.662 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-16T12:08:08.662 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-16T12:08:08.663 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-16T12:08:08.663 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-16T12:08:08.663 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-16T12:08:08.663 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-16T12:08:08.663 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-16T12:08:08.663 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:08.663 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:08.663 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.663 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.663 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.664 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.664 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-03-16T12:08:08.664 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-03-16T12:08:08.664 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:08.664 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.664 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:08.664 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:08.664 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.664 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.665 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.665 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:08.665 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.665 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.665 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.665 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:08.665 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:08.665 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.666 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-16T12:08:08.666 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "localpool", 2024-03-16T12:08:08.666 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-03-16T12:08:08.666 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-03-16T12:08:08.666 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-03-16T12:08:08.666 INFO:teuthology.orchestra.run.smithi029.stdout: "failure_domain": { 2024-03-16T12:08:08.666 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "failure_domain", 2024-03-16T12:08:08.666 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:08.667 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.667 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:08.667 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "host", 2024-03-16T12:08:08.667 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.667 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.667 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.667 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "failure domain for any created local pool", 2024-03-16T12:08:08.667 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-03-16T12:08:08.667 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.668 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.668 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.668 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-03-16T12:08:08.668 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-03-16T12:08:08.668 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:08.668 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.668 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:08.668 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:08.669 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.669 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.669 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-16T12:08:08.669 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-16T12:08:08.669 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-16T12:08:08.669 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-16T12:08:08.669 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-16T12:08:08.669 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-16T12:08:08.669 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-16T12:08:08.670 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:08.670 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:08.670 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.670 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.670 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.670 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.670 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-03-16T12:08:08.671 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-03-16T12:08:08.671 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:08.671 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.671 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:08.671 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:08.671 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.671 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.672 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.672 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:08.672 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.672 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.672 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.672 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.672 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-03-16T12:08:08.672 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-03-16T12:08:08.672 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:08.673 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.673 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:08.673 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-03-16T12:08:08.673 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.673 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.673 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-16T12:08:08.674 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-16T12:08:08.674 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-16T12:08:08.674 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-16T12:08:08.674 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-16T12:08:08.674 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-16T12:08:08.674 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-16T12:08:08.674 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:08.674 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:08.675 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.675 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.675 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.675 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.675 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-03-16T12:08:08.675 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-03-16T12:08:08.675 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:08.676 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.676 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:08.676 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:08.676 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.676 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.676 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.676 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:08.676 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.677 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.677 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.677 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.677 INFO:teuthology.orchestra.run.smithi029.stdout: "min_size": { 2024-03-16T12:08:08.677 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "min_size", 2024-03-16T12:08:08.677 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-16T12:08:08.677 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.677 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:08.678 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:08.678 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.678 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.678 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.678 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "default min_size for any created local pool", 2024-03-16T12:08:08.678 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-03-16T12:08:08.679 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.679 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.679 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.679 INFO:teuthology.orchestra.run.smithi029.stdout: "num_rep": { 2024-03-16T12:08:08.679 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "num_rep", 2024-03-16T12:08:08.679 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-16T12:08:08.679 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.679 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:08.680 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "3", 2024-03-16T12:08:08.680 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.680 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.680 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.680 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "default replica count for any created local pool", 2024-03-16T12:08:08.680 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.681 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.681 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.681 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.681 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_num": { 2024-03-16T12:08:08.681 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "pg_num", 2024-03-16T12:08:08.681 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-16T12:08:08.681 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.682 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:08.682 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "128", 2024-03-16T12:08:08.682 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.682 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.682 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.682 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "default pg_num for any created local pool", 2024-03-16T12:08:08.682 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.683 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.683 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.683 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.683 INFO:teuthology.orchestra.run.smithi029.stdout: "prefix": { 2024-03-16T12:08:08.683 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "prefix", 2024-03-16T12:08:08.683 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:08.683 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.684 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:08.684 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:08.684 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.684 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.684 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.684 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "name prefix for any created local pool", 2024-03-16T12:08:08.684 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.684 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.684 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.685 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.685 INFO:teuthology.orchestra.run.smithi029.stdout: "subtree": { 2024-03-16T12:08:08.685 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "subtree", 2024-03-16T12:08:08.685 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:08.685 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.685 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:08.685 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "rack", 2024-03-16T12:08:08.685 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.685 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.686 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.686 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "CRUSH level for which to create a local pool", 2024-03-16T12:08:08.686 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-03-16T12:08:08.686 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.686 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.686 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:08.686 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:08.686 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.687 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-16T12:08:08.687 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "mds_autoscaler", 2024-03-16T12:08:08.687 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-03-16T12:08:08.687 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-03-16T12:08:08.687 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-03-16T12:08:08.687 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-03-16T12:08:08.687 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-03-16T12:08:08.687 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:08.687 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.688 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:08.688 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:08.688 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.688 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.688 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-16T12:08:08.688 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-16T12:08:08.688 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-16T12:08:08.688 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-16T12:08:08.688 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-16T12:08:08.689 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-16T12:08:08.689 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-16T12:08:08.689 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:08.689 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:08.689 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.689 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.689 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.689 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.689 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-03-16T12:08:08.689 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-03-16T12:08:08.690 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:08.690 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.690 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:08.690 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:08.690 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.690 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.690 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.690 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:08.690 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.691 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.691 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.691 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.691 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-03-16T12:08:08.691 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-03-16T12:08:08.691 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:08.691 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.691 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:08.691 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-03-16T12:08:08.692 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.692 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.692 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-16T12:08:08.692 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-16T12:08:08.692 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-16T12:08:08.692 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-16T12:08:08.692 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-16T12:08:08.692 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-16T12:08:08.692 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-16T12:08:08.693 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:08.693 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:08.693 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.693 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.693 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.693 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.693 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-03-16T12:08:08.693 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-03-16T12:08:08.693 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:08.694 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.694 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:08.694 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:08.694 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.694 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.694 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.694 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:08.694 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.694 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.695 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.695 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:08.695 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:08.695 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.695 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-16T12:08:08.695 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "mirroring", 2024-03-16T12:08:08.695 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-03-16T12:08:08.695 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-03-16T12:08:08.695 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-03-16T12:08:08.695 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-03-16T12:08:08.696 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-03-16T12:08:08.696 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:08.696 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.696 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:08.696 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:08.696 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.696 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.696 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-16T12:08:08.696 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-16T12:08:08.696 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-16T12:08:08.697 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-16T12:08:08.697 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-16T12:08:08.697 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-16T12:08:08.697 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-16T12:08:08.697 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:08.697 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:08.697 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.697 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.698 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.698 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.698 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-03-16T12:08:08.698 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-03-16T12:08:08.698 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:08.698 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.698 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:08.698 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:08.699 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.699 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.699 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.699 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:08.699 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.699 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.699 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.699 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.699 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-03-16T12:08:08.700 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-03-16T12:08:08.700 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:08.700 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.700 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:08.700 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-03-16T12:08:08.700 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.700 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.700 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-16T12:08:08.700 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-16T12:08:08.700 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-16T12:08:08.701 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-16T12:08:08.701 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-16T12:08:08.701 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-16T12:08:08.701 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-16T12:08:08.701 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:08.701 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:08.701 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.701 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.701 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.702 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.702 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-03-16T12:08:08.702 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-03-16T12:08:08.702 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:08.702 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.702 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:08.702 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:08.702 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.702 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.703 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.703 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:08.703 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.703 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.703 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.703 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:08.704 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:08.704 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.704 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-16T12:08:08.704 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "nfs", 2024-03-16T12:08:08.704 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-03-16T12:08:08.704 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-03-16T12:08:08.704 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-03-16T12:08:08.704 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-03-16T12:08:08.704 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-03-16T12:08:08.705 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:08.705 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.705 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:08.705 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:08.705 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.705 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.706 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-16T12:08:08.706 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-16T12:08:08.706 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-16T12:08:08.706 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-16T12:08:08.706 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-16T12:08:08.706 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-16T12:08:08.706 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-16T12:08:08.706 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:08.707 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:08.707 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.707 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.707 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.707 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.707 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-03-16T12:08:08.707 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-03-16T12:08:08.708 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:08.708 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.708 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:08.708 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:08.708 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.708 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.708 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.709 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:08.709 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.709 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.709 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.709 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.709 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-03-16T12:08:08.709 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-03-16T12:08:08.709 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:08.709 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.710 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:08.710 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-03-16T12:08:08.710 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.710 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.710 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-16T12:08:08.710 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-16T12:08:08.710 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-16T12:08:08.710 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-16T12:08:08.710 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-16T12:08:08.711 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-16T12:08:08.711 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-16T12:08:08.711 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:08.711 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:08.711 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.711 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.711 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.711 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.712 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-03-16T12:08:08.712 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-03-16T12:08:08.712 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:08.712 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.712 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:08.712 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:08.712 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.713 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.713 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.713 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:08.713 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.713 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.713 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.713 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:08.714 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:08.714 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.714 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-16T12:08:08.714 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "orchestrator", 2024-03-16T12:08:08.714 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-03-16T12:08:08.715 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-03-16T12:08:08.715 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-03-16T12:08:08.715 INFO:teuthology.orchestra.run.smithi029.stdout: "fail_fs": { 2024-03-16T12:08:08.715 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "fail_fs", 2024-03-16T12:08:08.715 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:08.715 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.715 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:08.715 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:08.716 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.716 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.716 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.716 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-03-16T12:08:08.716 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.717 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.717 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.717 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.717 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-03-16T12:08:08.717 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-03-16T12:08:08.717 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:08.718 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.718 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:08.718 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:08.718 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.718 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.719 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-16T12:08:08.719 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-16T12:08:08.719 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-16T12:08:08.719 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-16T12:08:08.719 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-16T12:08:08.719 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-16T12:08:08.719 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-16T12:08:08.720 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:08.720 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:08.720 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.720 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.720 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.720 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.720 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-03-16T12:08:08.720 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-03-16T12:08:08.721 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:08.721 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.721 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:08.721 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:08.721 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.721 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.721 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.721 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:08.721 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.722 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.722 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.722 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.722 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-03-16T12:08:08.722 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-03-16T12:08:08.722 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:08.722 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.722 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:08.723 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-03-16T12:08:08.723 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.723 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.723 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-16T12:08:08.723 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-16T12:08:08.723 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-16T12:08:08.723 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-16T12:08:08.723 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-16T12:08:08.723 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-16T12:08:08.723 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-16T12:08:08.724 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:08.724 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:08.724 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.724 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.724 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.724 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.724 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-03-16T12:08:08.724 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-03-16T12:08:08.724 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:08.724 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.725 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:08.725 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:08.725 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.725 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.725 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.725 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:08.725 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.725 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.725 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.726 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.726 INFO:teuthology.orchestra.run.smithi029.stdout: "orchestrator": { 2024-03-16T12:08:08.726 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "orchestrator", 2024-03-16T12:08:08.726 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:08.726 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.726 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:08.726 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:08.726 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.727 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.727 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-16T12:08:08.727 INFO:teuthology.orchestra.run.smithi029.stdout: "cephadm", 2024-03-16T12:08:08.727 INFO:teuthology.orchestra.run.smithi029.stdout: "rook", 2024-03-16T12:08:08.727 INFO:teuthology.orchestra.run.smithi029.stdout: "test_orchestrator" 2024-03-16T12:08:08.727 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:08.727 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Orchestrator backend", 2024-03-16T12:08:08.727 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.727 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.728 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.728 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:08.728 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:08.728 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.728 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-16T12:08:08.728 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "osd_perf_query", 2024-03-16T12:08:08.728 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-03-16T12:08:08.728 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-03-16T12:08:08.728 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-03-16T12:08:08.729 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-03-16T12:08:08.729 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-03-16T12:08:08.729 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:08.729 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.729 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:08.729 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:08.729 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.729 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.729 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-16T12:08:08.729 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-16T12:08:08.730 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-16T12:08:08.730 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-16T12:08:08.730 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-16T12:08:08.730 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-16T12:08:08.730 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-16T12:08:08.730 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:08.730 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:08.730 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.730 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.731 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.731 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.731 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-03-16T12:08:08.731 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-03-16T12:08:08.731 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:08.731 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.731 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:08.731 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:08.731 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.732 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.732 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.732 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:08.732 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.732 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.732 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.732 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.732 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-03-16T12:08:08.732 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-03-16T12:08:08.732 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:08.733 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.733 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:08.733 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-03-16T12:08:08.733 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.733 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.733 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-16T12:08:08.733 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-16T12:08:08.733 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-16T12:08:08.733 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-16T12:08:08.734 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-16T12:08:08.734 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-16T12:08:08.734 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-16T12:08:08.734 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:08.734 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:08.734 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.734 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.734 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.735 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.735 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-03-16T12:08:08.735 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-03-16T12:08:08.735 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:08.735 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.735 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:08.735 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:08.735 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.735 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.735 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.736 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:08.736 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.736 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.736 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.736 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:08.736 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:08.736 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.736 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-16T12:08:08.736 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "osd_support", 2024-03-16T12:08:08.736 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-03-16T12:08:08.737 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-03-16T12:08:08.737 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-03-16T12:08:08.737 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-03-16T12:08:08.737 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-03-16T12:08:08.737 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:08.737 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.737 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:08.737 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:08.737 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.737 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.738 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-16T12:08:08.738 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-16T12:08:08.738 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-16T12:08:08.738 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-16T12:08:08.738 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-16T12:08:08.738 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-16T12:08:08.738 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-16T12:08:08.738 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:08.739 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:08.739 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.739 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.739 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.739 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.739 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-03-16T12:08:08.740 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-03-16T12:08:08.740 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:08.740 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.740 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:08.740 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:08.740 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.740 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.740 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.740 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:08.741 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.741 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.741 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.741 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.741 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-03-16T12:08:08.741 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-03-16T12:08:08.742 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:08.742 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.742 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:08.742 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-03-16T12:08:08.742 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.742 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.742 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-16T12:08:08.742 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-16T12:08:08.742 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-16T12:08:08.743 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-16T12:08:08.743 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-16T12:08:08.743 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-16T12:08:08.743 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-16T12:08:08.743 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:08.743 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:08.744 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.744 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.744 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.744 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.744 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-03-16T12:08:08.744 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-03-16T12:08:08.744 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:08.744 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.745 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:08.745 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:08.745 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.745 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.745 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.745 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:08.746 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.746 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.746 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.746 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:08.746 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:08.746 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.746 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-16T12:08:08.747 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "pg_autoscaler", 2024-03-16T12:08:08.747 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-03-16T12:08:08.747 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-03-16T12:08:08.747 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-03-16T12:08:08.747 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-03-16T12:08:08.747 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-03-16T12:08:08.747 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:08.748 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.748 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:08.748 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:08.748 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.748 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.748 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-16T12:08:08.749 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-16T12:08:08.749 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-16T12:08:08.749 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-16T12:08:08.749 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-16T12:08:08.749 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-16T12:08:08.749 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-16T12:08:08.750 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:08.750 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:08.750 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.750 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.751 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.751 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.751 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-03-16T12:08:08.751 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-03-16T12:08:08.751 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:08.751 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.751 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:08.751 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:08.751 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.752 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.752 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.752 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:08.752 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.752 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.752 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.752 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.752 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-03-16T12:08:08.752 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-03-16T12:08:08.753 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:08.753 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.753 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:08.753 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-03-16T12:08:08.753 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.753 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.753 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-16T12:08:08.753 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-16T12:08:08.754 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-16T12:08:08.754 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-16T12:08:08.754 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-16T12:08:08.754 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-16T12:08:08.754 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-16T12:08:08.754 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:08.754 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:08.754 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.754 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.755 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.755 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.755 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-03-16T12:08:08.755 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-03-16T12:08:08.755 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:08.755 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.755 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:08.755 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:08.755 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.756 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.756 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.756 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:08.756 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.756 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.756 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.756 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.756 INFO:teuthology.orchestra.run.smithi029.stdout: "sleep_interval": { 2024-03-16T12:08:08.756 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "sleep_interval", 2024-03-16T12:08:08.756 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "secs", 2024-03-16T12:08:08.757 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.757 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:08.757 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "60", 2024-03-16T12:08:08.757 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.757 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.757 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.757 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:08.757 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.757 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.758 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.758 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.758 INFO:teuthology.orchestra.run.smithi029.stdout: "threshold": { 2024-03-16T12:08:08.758 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "threshold", 2024-03-16T12:08:08.758 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "float", 2024-03-16T12:08:08.758 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.758 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:08.758 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "3.0", 2024-03-16T12:08:08.759 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "1.0", 2024-03-16T12:08:08.759 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.759 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.759 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "scaling threshold", 2024-03-16T12:08:08.759 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-03-16T12:08:08.759 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.759 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.759 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:08.759 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:08.760 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.760 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-16T12:08:08.760 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "progress", 2024-03-16T12:08:08.760 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-03-16T12:08:08.760 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-03-16T12:08:08.760 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-03-16T12:08:08.760 INFO:teuthology.orchestra.run.smithi029.stdout: "allow_pg_recovery_event": { 2024-03-16T12:08:08.760 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "allow_pg_recovery_event", 2024-03-16T12:08:08.761 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:08.761 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.761 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:08.761 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:08.761 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.761 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.761 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.761 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "allow the module to show pg recovery progress", 2024-03-16T12:08:08.761 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.761 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.762 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.762 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.762 INFO:teuthology.orchestra.run.smithi029.stdout: "enabled": { 2024-03-16T12:08:08.762 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "enabled", 2024-03-16T12:08:08.762 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:08.762 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.762 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:08.762 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-16T12:08:08.763 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.763 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.763 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.763 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:08.763 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.763 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.763 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.763 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.764 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-03-16T12:08:08.764 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-03-16T12:08:08.764 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:08.764 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.764 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:08.764 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:08.764 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.764 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.764 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-16T12:08:08.765 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-16T12:08:08.765 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-16T12:08:08.765 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-16T12:08:08.765 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-16T12:08:08.765 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-16T12:08:08.765 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-16T12:08:08.765 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:08.765 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:08.765 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.766 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.766 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.766 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.766 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-03-16T12:08:08.766 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-03-16T12:08:08.766 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:08.766 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.766 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:08.767 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:08.767 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.767 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.767 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.767 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:08.767 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.767 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.767 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.768 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.768 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-03-16T12:08:08.768 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-03-16T12:08:08.768 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:08.768 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.768 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:08.768 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-03-16T12:08:08.768 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.769 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.769 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-16T12:08:08.769 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-16T12:08:08.769 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-16T12:08:08.769 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-16T12:08:08.769 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-16T12:08:08.769 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-16T12:08:08.769 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-16T12:08:08.769 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:08.770 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:08.770 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.770 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.770 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.770 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.770 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-03-16T12:08:08.771 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-03-16T12:08:08.771 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:08.771 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.771 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:08.771 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:08.771 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.771 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.772 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.772 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:08.772 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.772 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.772 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.772 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.772 INFO:teuthology.orchestra.run.smithi029.stdout: "max_completed_events": { 2024-03-16T12:08:08.772 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "max_completed_events", 2024-03-16T12:08:08.772 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-16T12:08:08.773 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.773 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:08.773 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "50", 2024-03-16T12:08:08.773 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.773 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.773 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.774 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "number of past completed events to remember", 2024-03-16T12:08:08.774 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.774 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.774 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.774 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.774 INFO:teuthology.orchestra.run.smithi029.stdout: "sleep_interval": { 2024-03-16T12:08:08.774 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "sleep_interval", 2024-03-16T12:08:08.774 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "secs", 2024-03-16T12:08:08.774 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.775 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:08.775 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "5", 2024-03-16T12:08:08.775 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.775 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.775 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.775 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "how long the module is going to sleep", 2024-03-16T12:08:08.775 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.776 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.776 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.776 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:08.776 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:08.776 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.776 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-16T12:08:08.776 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "prometheus", 2024-03-16T12:08:08.776 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-03-16T12:08:08.777 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-03-16T12:08:08.777 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-03-16T12:08:08.777 INFO:teuthology.orchestra.run.smithi029.stdout: "cache": { 2024-03-16T12:08:08.777 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "cache", 2024-03-16T12:08:08.777 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:08.777 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.778 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:08.778 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-16T12:08:08.778 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.778 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.778 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.778 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:08.778 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.779 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.779 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.779 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.779 INFO:teuthology.orchestra.run.smithi029.stdout: "exclude_perf_counters": { 2024-03-16T12:08:08.779 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "exclude_perf_counters", 2024-03-16T12:08:08.779 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:08.779 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.779 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:08.780 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-16T12:08:08.780 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.780 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.780 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.780 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-03-16T12:08:08.780 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-03-16T12:08:08.780 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.780 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.780 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.780 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-03-16T12:08:08.781 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-03-16T12:08:08.781 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:08.781 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.781 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:08.781 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:08.781 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.781 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.781 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-16T12:08:08.781 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-16T12:08:08.782 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-16T12:08:08.782 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-16T12:08:08.782 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-16T12:08:08.782 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-16T12:08:08.782 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-16T12:08:08.782 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:08.782 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:08.782 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.783 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.783 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.783 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.783 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-03-16T12:08:08.783 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-03-16T12:08:08.784 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:08.784 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.784 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:08.784 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:08.784 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.784 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.785 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.785 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:08.785 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.785 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.785 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.785 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.786 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-03-16T12:08:08.786 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-03-16T12:08:08.786 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:08.786 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.786 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:08.786 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-03-16T12:08:08.786 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.787 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.787 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-16T12:08:08.787 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-16T12:08:08.787 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-16T12:08:08.787 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-16T12:08:08.787 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-16T12:08:08.787 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-16T12:08:08.787 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-16T12:08:08.787 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:08.788 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:08.788 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.788 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.788 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.788 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.788 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-03-16T12:08:08.788 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-03-16T12:08:08.788 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:08.789 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.789 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:08.789 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:08.789 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.789 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.789 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.789 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:08.789 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.789 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.789 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.790 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.790 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_stats_pools": { 2024-03-16T12:08:08.790 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "rbd_stats_pools", 2024-03-16T12:08:08.790 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:08.790 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.790 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:08.790 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:08.791 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.791 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.791 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.791 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:08.792 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.792 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.792 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.792 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.792 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_stats_pools_refresh_interval": { 2024-03-16T12:08:08.793 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-03-16T12:08:08.793 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-16T12:08:08.793 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.793 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:08.794 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "300", 2024-03-16T12:08:08.794 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.794 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.795 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.795 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:08.795 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.795 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.795 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.796 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.796 INFO:teuthology.orchestra.run.smithi029.stdout: "scrape_interval": { 2024-03-16T12:08:08.796 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "scrape_interval", 2024-03-16T12:08:08.796 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "float", 2024-03-16T12:08:08.797 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.797 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:08.797 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "15.0", 2024-03-16T12:08:08.797 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.798 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.798 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.798 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:08.798 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.799 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.799 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.799 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.799 INFO:teuthology.orchestra.run.smithi029.stdout: "server_addr": { 2024-03-16T12:08:08.800 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "server_addr", 2024-03-16T12:08:08.800 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:08.800 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.800 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:08.800 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "::", 2024-03-16T12:08:08.801 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.801 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.801 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.801 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-03-16T12:08:08.802 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.802 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.802 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.802 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.803 INFO:teuthology.orchestra.run.smithi029.stdout: "server_port": { 2024-03-16T12:08:08.803 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "server_port", 2024-03-16T12:08:08.803 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-16T12:08:08.803 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.804 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:08.804 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "9283", 2024-03-16T12:08:08.804 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.804 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.805 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.805 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-03-16T12:08:08.805 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.805 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.805 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.806 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.806 INFO:teuthology.orchestra.run.smithi029.stdout: "stale_cache_strategy": { 2024-03-16T12:08:08.806 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "stale_cache_strategy", 2024-03-16T12:08:08.806 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:08.807 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.807 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:08.807 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "log", 2024-03-16T12:08:08.807 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.808 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.808 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.808 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:08.808 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.809 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.809 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.809 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.809 INFO:teuthology.orchestra.run.smithi029.stdout: "standby_behaviour": { 2024-03-16T12:08:08.810 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "standby_behaviour", 2024-03-16T12:08:08.810 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:08.810 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.810 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:08.811 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "default", 2024-03-16T12:08:08.811 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.811 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.811 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-16T12:08:08.812 INFO:teuthology.orchestra.run.smithi029.stdout: "default", 2024-03-16T12:08:08.812 INFO:teuthology.orchestra.run.smithi029.stdout: "error" 2024-03-16T12:08:08.812 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:08.812 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:08.813 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.813 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.813 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.813 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.813 INFO:teuthology.orchestra.run.smithi029.stdout: "standby_error_status_code": { 2024-03-16T12:08:08.814 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "standby_error_status_code", 2024-03-16T12:08:08.814 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-16T12:08:08.814 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.814 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:08.815 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "500", 2024-03-16T12:08:08.815 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "400", 2024-03-16T12:08:08.815 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "599", 2024-03-16T12:08:08.815 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.816 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:08.816 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.816 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.817 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.817 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:08.817 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:08.817 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.817 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-16T12:08:08.818 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "rbd_support", 2024-03-16T12:08:08.818 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-03-16T12:08:08.818 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-03-16T12:08:08.818 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-03-16T12:08:08.819 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-03-16T12:08:08.819 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-03-16T12:08:08.819 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:08.819 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.820 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:08.820 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:08.820 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.820 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.820 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-16T12:08:08.821 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-16T12:08:08.821 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-16T12:08:08.821 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-16T12:08:08.822 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-16T12:08:08.822 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-16T12:08:08.822 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-16T12:08:08.823 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:08.823 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:08.824 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.824 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.824 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.825 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.825 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-03-16T12:08:08.825 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-03-16T12:08:08.826 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:08.826 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.827 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:08.827 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:08.827 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.828 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.828 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.828 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:08.829 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.829 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.829 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.830 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.830 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-03-16T12:08:08.831 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-03-16T12:08:08.831 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:08.831 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.832 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:08.832 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-03-16T12:08:08.832 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.833 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.833 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-16T12:08:08.834 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-16T12:08:08.834 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-16T12:08:08.834 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-16T12:08:08.834 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-16T12:08:08.835 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-16T12:08:08.835 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-16T12:08:08.836 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:08.836 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:08.836 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.837 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.837 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.837 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.838 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-03-16T12:08:08.838 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-03-16T12:08:08.839 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:08.839 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.840 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:08.840 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:08.840 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.841 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.841 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.842 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:08.842 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.842 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.843 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.843 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.843 INFO:teuthology.orchestra.run.smithi029.stdout: "max_concurrent_snap_create": { 2024-03-16T12:08:08.843 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "max_concurrent_snap_create", 2024-03-16T12:08:08.844 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-16T12:08:08.844 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.844 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:08.844 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "10", 2024-03-16T12:08:08.845 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.845 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.845 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.845 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:08.846 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.846 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.846 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.846 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.847 INFO:teuthology.orchestra.run.smithi029.stdout: "mirror_snapshot_schedule": { 2024-03-16T12:08:08.847 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "mirror_snapshot_schedule", 2024-03-16T12:08:08.847 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:08.847 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.847 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:08.847 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:08.847 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.847 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.848 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.848 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:08.848 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.848 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.848 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.848 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.848 INFO:teuthology.orchestra.run.smithi029.stdout: "trash_purge_schedule": { 2024-03-16T12:08:08.848 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "trash_purge_schedule", 2024-03-16T12:08:08.848 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:08.849 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.849 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:08.849 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:08.849 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.849 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.849 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.849 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:08.849 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.849 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.850 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.850 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:08.850 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:08.850 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.850 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-16T12:08:08.850 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "restful", 2024-03-16T12:08:08.850 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-03-16T12:08:08.850 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-03-16T12:08:08.851 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-03-16T12:08:08.851 INFO:teuthology.orchestra.run.smithi029.stdout: "enable_auth": { 2024-03-16T12:08:08.851 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "enable_auth", 2024-03-16T12:08:08.851 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:08.851 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.851 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:08.851 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-16T12:08:08.851 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.851 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.852 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.852 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:08.852 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.852 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.852 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.852 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.852 INFO:teuthology.orchestra.run.smithi029.stdout: "key_file": { 2024-03-16T12:08:08.852 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "key_file", 2024-03-16T12:08:08.853 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:08.853 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.853 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:08.853 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:08.853 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.853 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.853 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.853 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:08.854 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.854 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.854 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.854 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.854 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-03-16T12:08:08.854 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-03-16T12:08:08.855 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:08.855 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.855 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:08.855 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:08.855 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.855 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.855 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-16T12:08:08.855 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-16T12:08:08.856 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-16T12:08:08.856 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-16T12:08:08.856 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-16T12:08:08.856 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-16T12:08:08.856 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-16T12:08:08.856 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:08.856 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:08.856 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.857 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.857 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.857 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.857 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-03-16T12:08:08.857 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-03-16T12:08:08.857 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:08.857 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.857 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:08.857 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:08.858 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.858 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.858 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.858 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:08.858 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.858 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.858 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.858 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.858 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-03-16T12:08:08.859 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-03-16T12:08:08.859 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:08.859 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.859 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:08.859 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-03-16T12:08:08.859 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.859 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.859 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-16T12:08:08.859 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-16T12:08:08.860 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-16T12:08:08.860 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-16T12:08:08.860 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-16T12:08:08.860 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-16T12:08:08.860 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-16T12:08:08.860 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:08.860 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:08.860 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.860 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.861 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.861 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.861 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-03-16T12:08:08.861 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-03-16T12:08:08.861 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:08.861 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.861 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:08.861 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:08.861 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.862 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.862 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.862 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:08.862 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.862 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.862 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.862 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.862 INFO:teuthology.orchestra.run.smithi029.stdout: "server_addr": { 2024-03-16T12:08:08.862 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "server_addr", 2024-03-16T12:08:08.863 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:08.863 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.863 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:08.863 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:08.863 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.863 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.863 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.863 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:08.863 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.864 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.864 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.864 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.864 INFO:teuthology.orchestra.run.smithi029.stdout: "server_port": { 2024-03-16T12:08:08.864 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "server_port", 2024-03-16T12:08:08.864 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:08.864 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.864 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:08.864 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:08.864 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.865 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.865 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.865 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:08.865 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.865 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.865 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.865 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:08.865 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:08.865 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.866 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-16T12:08:08.866 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "rgw", 2024-03-16T12:08:08.866 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-03-16T12:08:08.866 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-03-16T12:08:08.866 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-03-16T12:08:08.866 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-03-16T12:08:08.866 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-03-16T12:08:08.867 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:08.867 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.867 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:08.867 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:08.867 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.867 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.867 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-16T12:08:08.868 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-16T12:08:08.868 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-16T12:08:08.868 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-16T12:08:08.868 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-16T12:08:08.868 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-16T12:08:08.868 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-16T12:08:08.868 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:08.868 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:08.869 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.869 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.869 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.869 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.869 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-03-16T12:08:08.869 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-03-16T12:08:08.870 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:08.870 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.870 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:08.870 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:08.870 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.870 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.870 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.870 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:08.870 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.871 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.871 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.871 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.871 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-03-16T12:08:08.871 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-03-16T12:08:08.871 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:08.872 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.872 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:08.872 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-03-16T12:08:08.872 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.872 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.872 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-16T12:08:08.872 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-16T12:08:08.872 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-16T12:08:08.873 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-16T12:08:08.873 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-16T12:08:08.873 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-16T12:08:08.873 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-16T12:08:08.873 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:08.873 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:08.873 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.874 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.874 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.874 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.874 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-03-16T12:08:08.874 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-03-16T12:08:08.874 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:08.874 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.874 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:08.875 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:08.875 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.875 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.875 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.875 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:08.875 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.875 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.876 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.876 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:08.876 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:08.876 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.876 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-16T12:08:08.876 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "rook", 2024-03-16T12:08:08.877 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-03-16T12:08:08.877 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-03-16T12:08:08.877 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-03-16T12:08:08.877 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-03-16T12:08:08.877 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-03-16T12:08:08.877 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:08.877 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.877 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:08.877 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:08.878 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.878 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.878 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-16T12:08:08.878 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-16T12:08:08.878 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-16T12:08:08.878 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-16T12:08:08.878 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-16T12:08:08.878 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-16T12:08:08.879 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-16T12:08:08.879 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:08.879 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:08.879 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.879 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.879 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.879 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.880 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-03-16T12:08:08.880 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-03-16T12:08:08.880 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:08.880 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.880 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:08.880 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:08.880 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.880 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.881 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.881 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:08.881 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.881 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.881 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.881 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.882 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-03-16T12:08:08.882 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-03-16T12:08:08.882 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:08.882 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.882 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:08.883 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-03-16T12:08:08.883 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.883 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.883 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-16T12:08:08.883 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-16T12:08:08.883 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-16T12:08:08.883 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-16T12:08:08.883 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-16T12:08:08.884 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-16T12:08:08.884 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-16T12:08:08.884 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:08.884 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:08.884 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.884 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.884 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.884 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.884 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-03-16T12:08:08.885 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-03-16T12:08:08.885 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:08.885 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.885 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:08.885 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:08.885 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.885 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.885 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.885 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:08.886 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.886 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.886 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.886 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.886 INFO:teuthology.orchestra.run.smithi029.stdout: "storage_class": { 2024-03-16T12:08:08.886 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "storage_class", 2024-03-16T12:08:08.886 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:08.886 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.886 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:08.887 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "local", 2024-03-16T12:08:08.887 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.887 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.887 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.887 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-03-16T12:08:08.887 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.887 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.887 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.888 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:08.888 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:08.888 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.888 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-16T12:08:08.888 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "selftest", 2024-03-16T12:08:08.888 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-03-16T12:08:08.888 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-03-16T12:08:08.888 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-03-16T12:08:08.888 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-03-16T12:08:08.889 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-03-16T12:08:08.889 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:08.889 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.889 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:08.889 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:08.889 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.889 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.889 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-16T12:08:08.889 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-16T12:08:08.890 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-16T12:08:08.890 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-16T12:08:08.890 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-16T12:08:08.890 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-16T12:08:08.890 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-16T12:08:08.890 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:08.890 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:08.890 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.890 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.891 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.891 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.891 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-03-16T12:08:08.891 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-03-16T12:08:08.891 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:08.891 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.891 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:08.891 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:08.891 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.892 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.892 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.892 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:08.892 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.892 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.892 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.892 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.892 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-03-16T12:08:08.892 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-03-16T12:08:08.893 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:08.893 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.893 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:08.893 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-03-16T12:08:08.893 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.893 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.893 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-16T12:08:08.893 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-16T12:08:08.894 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-16T12:08:08.894 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-16T12:08:08.894 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-16T12:08:08.894 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-16T12:08:08.894 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-16T12:08:08.894 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:08.894 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:08.894 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.895 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.895 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.895 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.895 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-03-16T12:08:08.895 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-03-16T12:08:08.895 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:08.895 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.895 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:08.895 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:08.896 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.896 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.896 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.896 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:08.896 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.896 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.896 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.896 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.896 INFO:teuthology.orchestra.run.smithi029.stdout: "roption1": { 2024-03-16T12:08:08.897 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "roption1", 2024-03-16T12:08:08.897 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:08.897 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.897 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:08.897 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:08.897 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.897 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.897 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.897 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:08.898 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.898 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.898 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.898 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.898 INFO:teuthology.orchestra.run.smithi029.stdout: "roption2": { 2024-03-16T12:08:08.898 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "roption2", 2024-03-16T12:08:08.898 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:08.898 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.899 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:08.899 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "xyz", 2024-03-16T12:08:08.899 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.899 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.899 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.899 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:08.899 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.899 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.899 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.900 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.900 INFO:teuthology.orchestra.run.smithi029.stdout: "rwoption1": { 2024-03-16T12:08:08.900 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "rwoption1", 2024-03-16T12:08:08.900 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:08.900 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.900 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:08.900 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:08.900 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.901 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.901 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.901 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:08.901 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.901 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.901 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.901 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.901 INFO:teuthology.orchestra.run.smithi029.stdout: "rwoption2": { 2024-03-16T12:08:08.901 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "rwoption2", 2024-03-16T12:08:08.902 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-16T12:08:08.902 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.902 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:08.902 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:08.902 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.902 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.902 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.902 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:08.903 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.903 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.903 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.903 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.903 INFO:teuthology.orchestra.run.smithi029.stdout: "rwoption3": { 2024-03-16T12:08:08.903 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "rwoption3", 2024-03-16T12:08:08.903 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "float", 2024-03-16T12:08:08.903 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.903 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:08.904 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:08.904 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.904 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.904 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.904 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:08.904 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.904 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.904 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.905 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.905 INFO:teuthology.orchestra.run.smithi029.stdout: "rwoption4": { 2024-03-16T12:08:08.905 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "rwoption4", 2024-03-16T12:08:08.905 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:08.905 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.905 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:08.906 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:08.906 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.906 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.906 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.906 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:08.906 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.906 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.906 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.907 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.907 INFO:teuthology.orchestra.run.smithi029.stdout: "rwoption5": { 2024-03-16T12:08:08.907 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "rwoption5", 2024-03-16T12:08:08.907 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:08.907 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.907 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:08.908 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:08.908 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.908 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.908 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.908 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:08.908 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.908 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.908 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.909 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.909 INFO:teuthology.orchestra.run.smithi029.stdout: "rwoption6": { 2024-03-16T12:08:08.909 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "rwoption6", 2024-03-16T12:08:08.909 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:08.909 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.909 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:08.909 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-16T12:08:08.910 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.910 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.910 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.910 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:08.910 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.910 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.910 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.911 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.911 INFO:teuthology.orchestra.run.smithi029.stdout: "rwoption7": { 2024-03-16T12:08:08.911 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "rwoption7", 2024-03-16T12:08:08.911 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-16T12:08:08.911 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.911 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:08.911 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:08.912 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "1", 2024-03-16T12:08:08.912 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "42", 2024-03-16T12:08:08.912 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.912 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:08.912 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.912 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.912 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.913 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.913 INFO:teuthology.orchestra.run.smithi029.stdout: "testkey": { 2024-03-16T12:08:08.913 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "testkey", 2024-03-16T12:08:08.913 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:08.913 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.913 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:08.914 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:08.914 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.914 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.914 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.914 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:08.914 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.914 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.914 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.915 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.915 INFO:teuthology.orchestra.run.smithi029.stdout: "testlkey": { 2024-03-16T12:08:08.915 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "testlkey", 2024-03-16T12:08:08.915 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:08.915 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.915 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:08.916 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:08.916 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.916 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.916 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.916 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:08.916 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.917 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.917 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.917 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.917 INFO:teuthology.orchestra.run.smithi029.stdout: "testnewline": { 2024-03-16T12:08:08.917 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "testnewline", 2024-03-16T12:08:08.917 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:08.918 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.918 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:08.918 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:08.918 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.918 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.918 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.918 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:08.919 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.919 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.919 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.919 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:08.919 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:08.919 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.919 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-16T12:08:08.919 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "snap_schedule", 2024-03-16T12:08:08.920 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-03-16T12:08:08.920 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-03-16T12:08:08.920 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-03-16T12:08:08.920 INFO:teuthology.orchestra.run.smithi029.stdout: "allow_m_granularity": { 2024-03-16T12:08:08.920 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "allow_m_granularity", 2024-03-16T12:08:08.920 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:08.920 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.920 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:08.920 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:08.921 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.921 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.921 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.921 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "allow minute scheduled snapshots", 2024-03-16T12:08:08.921 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.921 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.921 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.921 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.922 INFO:teuthology.orchestra.run.smithi029.stdout: "dump_on_update": { 2024-03-16T12:08:08.922 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "dump_on_update", 2024-03-16T12:08:08.922 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:08.922 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.922 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:08.922 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:08.922 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.922 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.922 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.923 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "dump database to debug log on update", 2024-03-16T12:08:08.923 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.923 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.923 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.923 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.923 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-03-16T12:08:08.923 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-03-16T12:08:08.923 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:08.923 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.924 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:08.924 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:08.924 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.924 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.924 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-16T12:08:08.924 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-16T12:08:08.924 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-16T12:08:08.924 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-16T12:08:08.924 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-16T12:08:08.925 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-16T12:08:08.925 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-16T12:08:08.925 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:08.925 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:08.925 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.925 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.925 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.925 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.925 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-03-16T12:08:08.926 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-03-16T12:08:08.926 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:08.926 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.926 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:08.926 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:08.926 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.926 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.926 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.927 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:08.927 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.927 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.927 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.927 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.927 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-03-16T12:08:08.927 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-03-16T12:08:08.927 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:08.927 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.928 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:08.928 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-03-16T12:08:08.928 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.928 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.928 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-16T12:08:08.928 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-16T12:08:08.928 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-16T12:08:08.929 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-16T12:08:08.929 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-16T12:08:08.929 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-16T12:08:08.929 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-16T12:08:08.929 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:08.929 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:08.929 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.929 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.929 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.930 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.930 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-03-16T12:08:08.930 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-03-16T12:08:08.930 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:08.930 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.930 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:08.930 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:08.930 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.931 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.931 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.931 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:08.931 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.931 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.931 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.931 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:08.931 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:08.932 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.932 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-16T12:08:08.932 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "stats", 2024-03-16T12:08:08.932 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-03-16T12:08:08.932 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-03-16T12:08:08.932 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-03-16T12:08:08.932 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-03-16T12:08:08.932 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-03-16T12:08:08.932 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:08.933 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.933 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:08.933 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:08.933 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.933 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.933 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-16T12:08:08.933 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-16T12:08:08.933 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-16T12:08:08.934 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-16T12:08:08.934 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-16T12:08:08.934 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-16T12:08:08.934 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-16T12:08:08.934 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:08.934 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:08.934 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.935 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.935 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.935 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.935 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-03-16T12:08:08.935 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-03-16T12:08:08.935 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:08.935 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.935 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:08.936 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:08.936 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.936 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.936 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.936 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:08.936 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.936 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.936 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.937 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.937 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-03-16T12:08:08.937 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-03-16T12:08:08.937 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:08.937 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.937 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:08.937 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-03-16T12:08:08.937 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.937 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.938 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-16T12:08:08.938 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-16T12:08:08.938 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-16T12:08:08.938 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-16T12:08:08.938 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-16T12:08:08.938 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-16T12:08:08.938 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-16T12:08:08.938 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:08.938 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:08.939 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.939 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.939 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.939 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.939 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-03-16T12:08:08.939 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-03-16T12:08:08.939 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:08.939 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.940 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:08.940 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:08.940 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.940 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.940 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.940 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:08.941 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.941 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.941 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.941 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:08.941 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:08.941 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.941 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-16T12:08:08.941 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "status", 2024-03-16T12:08:08.941 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-03-16T12:08:08.942 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-03-16T12:08:08.942 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-03-16T12:08:08.942 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-03-16T12:08:08.942 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-03-16T12:08:08.942 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:08.942 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.943 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:08.943 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:08.943 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.943 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.943 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-16T12:08:08.943 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-16T12:08:08.943 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-16T12:08:08.943 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-16T12:08:08.944 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-16T12:08:08.944 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-16T12:08:08.944 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-16T12:08:08.944 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:08.944 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:08.944 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.945 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.945 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.945 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.945 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-03-16T12:08:08.945 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-03-16T12:08:08.945 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:08.945 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.945 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:08.946 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:08.946 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.946 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.946 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.946 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:08.946 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.947 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.947 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.947 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.947 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-03-16T12:08:08.947 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-03-16T12:08:08.947 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:08.947 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.947 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:08.948 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-03-16T12:08:08.948 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.948 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.948 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-16T12:08:08.948 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-16T12:08:08.948 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-16T12:08:08.948 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-16T12:08:08.949 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-16T12:08:08.949 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-16T12:08:08.949 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-16T12:08:08.949 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:08.949 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:08.949 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.949 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.950 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.950 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.950 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-03-16T12:08:08.950 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-03-16T12:08:08.950 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:08.951 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.951 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:08.951 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:08.951 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.951 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.951 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.951 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:08.952 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.952 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.952 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.952 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:08.952 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:08.952 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.952 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-16T12:08:08.952 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "telegraf", 2024-03-16T12:08:08.953 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-03-16T12:08:08.953 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-03-16T12:08:08.953 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-03-16T12:08:08.953 INFO:teuthology.orchestra.run.smithi029.stdout: "address": { 2024-03-16T12:08:08.953 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "address", 2024-03-16T12:08:08.953 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:08.953 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.953 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:08.954 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-03-16T12:08:08.954 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.954 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.954 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.954 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:08.954 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.954 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.954 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.955 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.955 INFO:teuthology.orchestra.run.smithi029.stdout: "interval": { 2024-03-16T12:08:08.955 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "interval", 2024-03-16T12:08:08.955 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "secs", 2024-03-16T12:08:08.955 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.955 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:08.956 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "15", 2024-03-16T12:08:08.956 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.956 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.956 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.956 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:08.956 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.956 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.957 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.957 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.957 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-03-16T12:08:08.957 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-03-16T12:08:08.957 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:08.957 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.957 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:08.957 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:08.958 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.958 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.958 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-16T12:08:08.958 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-16T12:08:08.958 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-16T12:08:08.958 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-16T12:08:08.958 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-16T12:08:08.958 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-16T12:08:08.958 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-16T12:08:08.959 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:08.959 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:08.959 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.959 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.959 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.959 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.959 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-03-16T12:08:08.959 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-03-16T12:08:08.960 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:08.960 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.960 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:08.960 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:08.960 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.960 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.960 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.960 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:08.960 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.961 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.961 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.961 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.961 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-03-16T12:08:08.961 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-03-16T12:08:08.961 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:08.961 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.961 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:08.961 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-03-16T12:08:08.962 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.962 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.962 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-16T12:08:08.962 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-16T12:08:08.962 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-16T12:08:08.962 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-16T12:08:08.962 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-16T12:08:08.962 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-16T12:08:08.962 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-16T12:08:08.963 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:08.963 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:08.963 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.963 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.963 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.963 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.963 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-03-16T12:08:08.964 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-03-16T12:08:08.964 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:08.964 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.964 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:08.964 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:08.964 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.964 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.964 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.965 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:08.965 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.965 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.965 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.965 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:08.965 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:08.965 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.965 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-16T12:08:08.965 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "telemetry", 2024-03-16T12:08:08.966 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-03-16T12:08:08.966 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-03-16T12:08:08.966 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-03-16T12:08:08.966 INFO:teuthology.orchestra.run.smithi029.stdout: "channel_basic": { 2024-03-16T12:08:08.966 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "channel_basic", 2024-03-16T12:08:08.966 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:08.966 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.966 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:08.967 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-16T12:08:08.967 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.967 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.967 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.967 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Share basic cluster information (size, version)", 2024-03-16T12:08:08.967 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.967 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.967 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.968 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.968 INFO:teuthology.orchestra.run.smithi029.stdout: "channel_crash": { 2024-03-16T12:08:08.968 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "channel_crash", 2024-03-16T12:08:08.968 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:08.968 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.968 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:08.968 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-16T12:08:08.968 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.968 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.969 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.969 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-03-16T12:08:08.969 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.969 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.969 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.969 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.969 INFO:teuthology.orchestra.run.smithi029.stdout: "channel_device": { 2024-03-16T12:08:08.969 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "channel_device", 2024-03-16T12:08:08.970 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:08.970 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.970 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:08.970 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-16T12:08:08.970 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.970 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.970 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.970 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-03-16T12:08:08.970 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.971 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.971 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.971 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.971 INFO:teuthology.orchestra.run.smithi029.stdout: "channel_ident": { 2024-03-16T12:08:08.971 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "channel_ident", 2024-03-16T12:08:08.971 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:08.971 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.971 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:08.972 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:08.972 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.972 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.972 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.972 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-03-16T12:08:08.972 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.972 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.972 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.973 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.973 INFO:teuthology.orchestra.run.smithi029.stdout: "channel_perf": { 2024-03-16T12:08:08.973 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "channel_perf", 2024-03-16T12:08:08.973 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:08.973 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.973 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:08.973 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:08.974 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.974 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.974 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.974 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Share various performance metrics of a cluster", 2024-03-16T12:08:08.974 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.974 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.974 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.975 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.975 INFO:teuthology.orchestra.run.smithi029.stdout: "contact": { 2024-03-16T12:08:08.975 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "contact", 2024-03-16T12:08:08.975 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:08.975 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.975 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:08.975 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:08.975 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.976 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.976 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.976 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:08.976 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.976 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.976 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.976 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.976 INFO:teuthology.orchestra.run.smithi029.stdout: "description": { 2024-03-16T12:08:08.976 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "description", 2024-03-16T12:08:08.977 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:08.977 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.977 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:08.977 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:08.977 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.977 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.977 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.977 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:08.978 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.978 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.978 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.978 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.978 INFO:teuthology.orchestra.run.smithi029.stdout: "device_url": { 2024-03-16T12:08:08.979 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "device_url", 2024-03-16T12:08:08.979 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:08.979 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.979 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:08.979 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-03-16T12:08:08.979 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.979 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.979 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.979 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:08.980 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.980 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.980 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.980 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.980 INFO:teuthology.orchestra.run.smithi029.stdout: "enabled": { 2024-03-16T12:08:08.980 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "enabled", 2024-03-16T12:08:08.981 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:08.981 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.981 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:08.981 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:08.981 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.981 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.981 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.981 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:08.981 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.982 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.982 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.982 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.982 INFO:teuthology.orchestra.run.smithi029.stdout: "interval": { 2024-03-16T12:08:08.982 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "interval", 2024-03-16T12:08:08.982 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-16T12:08:08.983 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.983 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:08.983 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "24", 2024-03-16T12:08:08.983 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "8", 2024-03-16T12:08:08.983 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.983 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.983 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:08.983 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.984 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.984 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.984 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.984 INFO:teuthology.orchestra.run.smithi029.stdout: "last_opt_revision": { 2024-03-16T12:08:08.984 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "last_opt_revision", 2024-03-16T12:08:08.984 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-16T12:08:08.984 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.985 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:08.985 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "1", 2024-03-16T12:08:08.985 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.985 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.985 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.985 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:08.985 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.986 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.986 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.986 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.986 INFO:teuthology.orchestra.run.smithi029.stdout: "leaderboard": { 2024-03-16T12:08:08.986 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "leaderboard", 2024-03-16T12:08:08.986 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:08.986 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.987 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:08.987 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:08.987 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.987 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.987 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.987 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:08.988 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.988 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.988 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.988 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.988 INFO:teuthology.orchestra.run.smithi029.stdout: "leaderboard_description": { 2024-03-16T12:08:08.988 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "leaderboard_description", 2024-03-16T12:08:08.988 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:08.989 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.989 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:08.989 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:08.989 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.989 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.989 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.990 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:08.990 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.990 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.990 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.990 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.990 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-03-16T12:08:08.991 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-03-16T12:08:08.991 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:08.991 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.991 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:08.991 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:08.991 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.991 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.992 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-16T12:08:08.992 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-16T12:08:08.992 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-16T12:08:08.992 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-16T12:08:08.992 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-16T12:08:08.992 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-16T12:08:08.992 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-16T12:08:08.992 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:08.992 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:08.992 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.993 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.993 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.993 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.993 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-03-16T12:08:08.993 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-03-16T12:08:08.993 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:08.993 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.994 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:08.994 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:08.994 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.994 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.994 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.994 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:08.994 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.994 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.995 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.995 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.995 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-03-16T12:08:08.995 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-03-16T12:08:08.995 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:08.995 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.995 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:08.995 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-03-16T12:08:08.996 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.996 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.996 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-16T12:08:08.996 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-16T12:08:08.996 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-16T12:08:08.996 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-16T12:08:08.996 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-16T12:08:08.996 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-16T12:08:08.996 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-16T12:08:08.997 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:08.997 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:08.997 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.997 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.997 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.997 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.997 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-03-16T12:08:08.997 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-03-16T12:08:08.997 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:08.998 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.998 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:08.998 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:08.998 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:08.998 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:08.998 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:08.998 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:08.998 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:08.999 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:08.999 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:08.999 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:08.999 INFO:teuthology.orchestra.run.smithi029.stdout: "organization": { 2024-03-16T12:08:08.999 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "organization", 2024-03-16T12:08:08.999 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:08.999 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:08.999 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.000 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:09.000 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.000 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.000 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.000 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.000 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.000 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.000 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.000 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.001 INFO:teuthology.orchestra.run.smithi029.stdout: "proxy": { 2024-03-16T12:08:09.001 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "proxy", 2024-03-16T12:08:09.001 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:09.001 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.001 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.001 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:09.001 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.001 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.002 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.002 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.002 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.002 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.002 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.002 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.002 INFO:teuthology.orchestra.run.smithi029.stdout: "url": { 2024-03-16T12:08:09.002 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "url", 2024-03-16T12:08:09.003 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:09.003 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.003 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.003 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-03-16T12:08:09.003 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.003 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.003 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.003 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.004 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.004 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.004 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.004 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:09.004 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:09.004 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.004 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-16T12:08:09.004 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "test_orchestrator", 2024-03-16T12:08:09.004 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-03-16T12:08:09.004 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-03-16T12:08:09.005 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-03-16T12:08:09.005 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-03-16T12:08:09.005 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-03-16T12:08:09.005 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:09.005 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.005 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.005 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:09.005 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.005 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.006 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-16T12:08:09.006 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-16T12:08:09.006 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-16T12:08:09.006 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-16T12:08:09.006 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-16T12:08:09.006 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-16T12:08:09.006 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-16T12:08:09.006 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:09.006 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.007 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.007 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.007 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.007 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.007 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-03-16T12:08:09.007 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-03-16T12:08:09.007 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:09.007 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.008 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.008 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:09.008 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.008 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.008 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.008 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.008 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.008 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.008 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.009 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.009 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-03-16T12:08:09.009 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-03-16T12:08:09.009 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:09.009 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.009 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.009 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-03-16T12:08:09.009 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.010 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.010 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-16T12:08:09.010 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-16T12:08:09.010 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-16T12:08:09.010 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-16T12:08:09.010 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-16T12:08:09.010 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-16T12:08:09.010 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-16T12:08:09.010 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:09.011 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.011 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.011 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.011 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.011 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.011 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-03-16T12:08:09.011 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-03-16T12:08:09.011 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:09.012 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.012 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.012 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:09.012 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.012 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.012 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.013 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.013 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.013 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.013 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.013 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:09.013 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:09.013 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.014 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-16T12:08:09.014 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "volumes", 2024-03-16T12:08:09.014 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-03-16T12:08:09.014 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-03-16T12:08:09.014 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-03-16T12:08:09.014 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-03-16T12:08:09.014 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-03-16T12:08:09.014 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:09.014 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.015 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.015 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:09.015 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.015 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.015 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-16T12:08:09.016 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-16T12:08:09.016 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-16T12:08:09.016 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-16T12:08:09.016 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-16T12:08:09.016 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-16T12:08:09.016 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-16T12:08:09.016 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:09.016 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.016 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.017 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.017 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.017 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.017 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-03-16T12:08:09.017 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-03-16T12:08:09.017 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:09.018 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.018 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.018 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:09.018 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.018 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.018 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.018 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.018 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.019 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.019 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.019 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.019 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-03-16T12:08:09.019 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-03-16T12:08:09.019 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:09.020 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.020 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.020 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-03-16T12:08:09.020 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.020 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.020 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-16T12:08:09.020 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-16T12:08:09.020 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-16T12:08:09.021 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-16T12:08:09.021 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-16T12:08:09.021 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-16T12:08:09.021 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-16T12:08:09.021 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:09.021 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.021 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.022 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.022 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.022 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.022 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-03-16T12:08:09.022 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-03-16T12:08:09.023 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:09.023 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.023 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.023 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:09.023 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.023 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.024 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.024 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.024 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.024 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.024 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.024 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.025 INFO:teuthology.orchestra.run.smithi029.stdout: "max_concurrent_clones": { 2024-03-16T12:08:09.025 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "max_concurrent_clones", 2024-03-16T12:08:09.025 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-16T12:08:09.025 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.025 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.025 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "4", 2024-03-16T12:08:09.025 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.026 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.026 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.026 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Number of asynchronous cloner threads", 2024-03-16T12:08:09.026 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.026 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.026 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.026 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.026 INFO:teuthology.orchestra.run.smithi029.stdout: "periodic_async_work": { 2024-03-16T12:08:09.027 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "periodic_async_work", 2024-03-16T12:08:09.027 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:09.027 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.027 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.027 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:09.027 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.027 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.027 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.027 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Periodically check for async work", 2024-03-16T12:08:09.028 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.028 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.028 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.028 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.028 INFO:teuthology.orchestra.run.smithi029.stdout: "snapshot_clone_delay": { 2024-03-16T12:08:09.028 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "snapshot_clone_delay", 2024-03-16T12:08:09.028 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-16T12:08:09.028 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.029 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.029 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "0", 2024-03-16T12:08:09.029 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.029 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.029 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.029 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-03-16T12:08:09.029 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.029 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.029 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.030 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.030 INFO:teuthology.orchestra.run.smithi029.stdout: "snapshot_clone_no_wait": { 2024-03-16T12:08:09.030 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "snapshot_clone_no_wait", 2024-03-16T12:08:09.030 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:09.030 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.030 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.030 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-16T12:08:09.030 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.031 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.031 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.031 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-03-16T12:08:09.031 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.031 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.031 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.031 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:09.031 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:09.031 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.031 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-16T12:08:09.032 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "zabbix", 2024-03-16T12:08:09.032 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-03-16T12:08:09.032 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-03-16T12:08:09.032 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-03-16T12:08:09.032 INFO:teuthology.orchestra.run.smithi029.stdout: "discovery_interval": { 2024-03-16T12:08:09.032 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "discovery_interval", 2024-03-16T12:08:09.032 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "uint", 2024-03-16T12:08:09.032 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.032 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.033 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "100", 2024-03-16T12:08:09.033 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.033 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.033 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.033 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.033 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.033 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.033 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.033 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.034 INFO:teuthology.orchestra.run.smithi029.stdout: "identifier": { 2024-03-16T12:08:09.034 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "identifier", 2024-03-16T12:08:09.034 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:09.034 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.034 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.034 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:09.034 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.034 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.034 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.035 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.035 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.035 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.035 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.035 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.035 INFO:teuthology.orchestra.run.smithi029.stdout: "interval": { 2024-03-16T12:08:09.035 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "interval", 2024-03-16T12:08:09.035 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "secs", 2024-03-16T12:08:09.035 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.035 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.036 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "60", 2024-03-16T12:08:09.036 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.036 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.036 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.036 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.036 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.036 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.036 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.037 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.037 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-03-16T12:08:09.037 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-03-16T12:08:09.037 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:09.037 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.037 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.037 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:09.037 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.037 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.037 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-16T12:08:09.038 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-16T12:08:09.038 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-16T12:08:09.038 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-16T12:08:09.038 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-16T12:08:09.038 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-16T12:08:09.038 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-16T12:08:09.038 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:09.038 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.039 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.039 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.039 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.039 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.039 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-03-16T12:08:09.039 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-03-16T12:08:09.039 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:09.039 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.039 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.039 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:09.040 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.040 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.040 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.040 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.040 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.040 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.040 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.040 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.040 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-03-16T12:08:09.041 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-03-16T12:08:09.041 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:09.041 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.041 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.041 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-03-16T12:08:09.041 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.041 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.041 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-16T12:08:09.041 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-16T12:08:09.042 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-16T12:08:09.042 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-16T12:08:09.042 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-16T12:08:09.042 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-16T12:08:09.042 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-16T12:08:09.042 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:09.042 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.042 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.042 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.043 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.043 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.043 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-03-16T12:08:09.043 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-03-16T12:08:09.043 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:09.043 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.043 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.043 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:09.043 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.043 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.044 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.044 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.044 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.044 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.044 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.044 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.044 INFO:teuthology.orchestra.run.smithi029.stdout: "zabbix_host": { 2024-03-16T12:08:09.044 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "zabbix_host", 2024-03-16T12:08:09.044 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:09.045 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.045 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.045 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:09.045 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.045 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.045 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.045 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.045 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.046 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.046 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.046 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.046 INFO:teuthology.orchestra.run.smithi029.stdout: "zabbix_port": { 2024-03-16T12:08:09.046 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "zabbix_port", 2024-03-16T12:08:09.046 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-16T12:08:09.046 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.046 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.046 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "10051", 2024-03-16T12:08:09.047 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.047 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.047 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.047 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.047 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.047 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.047 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.047 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.048 INFO:teuthology.orchestra.run.smithi029.stdout: "zabbix_sender": { 2024-03-16T12:08:09.048 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "zabbix_sender", 2024-03-16T12:08:09.048 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:09.048 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.048 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.048 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-03-16T12:08:09.048 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.048 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.049 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.049 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.049 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.049 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.049 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.049 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:09.049 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:09.050 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:09.050 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-16T12:08:09.050 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.050 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-16T12:08:09.050 INFO:teuthology.orchestra.run.smithi029.stdout: "gid": 25177, 2024-03-16T12:08:09.050 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "z", 2024-03-16T12:08:09.050 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr_features": 4540701547738038271, 2024-03-16T12:08:09.051 INFO:teuthology.orchestra.run.smithi029.stdout: "available_modules": [ 2024-03-16T12:08:09.051 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-16T12:08:09.051 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "alerts", 2024-03-16T12:08:09.051 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-03-16T12:08:09.051 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-03-16T12:08:09.051 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-03-16T12:08:09.051 INFO:teuthology.orchestra.run.smithi029.stdout: "interval": { 2024-03-16T12:08:09.051 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "interval", 2024-03-16T12:08:09.051 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "secs", 2024-03-16T12:08:09.052 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.052 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.052 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "60", 2024-03-16T12:08:09.052 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.052 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.052 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.053 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "How frequently to reexamine health status", 2024-03-16T12:08:09.053 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.053 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.053 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.053 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.053 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-03-16T12:08:09.053 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-03-16T12:08:09.053 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:09.053 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.054 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.054 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:09.054 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.054 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.054 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-16T12:08:09.054 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-16T12:08:09.055 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-16T12:08:09.055 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-16T12:08:09.055 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-16T12:08:09.055 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-16T12:08:09.055 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-16T12:08:09.055 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:09.055 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.055 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.056 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.056 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.056 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.056 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-03-16T12:08:09.056 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-03-16T12:08:09.056 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:09.056 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.056 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.057 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:09.057 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.057 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.057 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.057 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.057 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.057 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.058 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.058 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.058 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-03-16T12:08:09.058 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-03-16T12:08:09.058 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:09.058 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.059 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.059 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-03-16T12:08:09.059 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.059 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.059 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-16T12:08:09.059 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-16T12:08:09.060 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-16T12:08:09.060 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-16T12:08:09.060 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-16T12:08:09.060 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-16T12:08:09.060 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-16T12:08:09.060 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:09.061 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.061 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.061 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.061 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.061 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.061 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-03-16T12:08:09.062 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-03-16T12:08:09.062 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:09.062 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.062 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.062 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:09.062 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.062 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.063 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.063 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.063 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.063 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.063 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.063 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.063 INFO:teuthology.orchestra.run.smithi029.stdout: "smtp_destination": { 2024-03-16T12:08:09.063 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "smtp_destination", 2024-03-16T12:08:09.063 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:09.064 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.064 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.064 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:09.064 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.064 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.064 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.064 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Email address to send alerts to", 2024-03-16T12:08:09.064 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.064 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.065 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.065 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.065 INFO:teuthology.orchestra.run.smithi029.stdout: "smtp_from_name": { 2024-03-16T12:08:09.065 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "smtp_from_name", 2024-03-16T12:08:09.065 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:09.065 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.065 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.065 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "Ceph", 2024-03-16T12:08:09.066 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.066 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.066 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.066 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Email From: name", 2024-03-16T12:08:09.066 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.066 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.066 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.066 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.067 INFO:teuthology.orchestra.run.smithi029.stdout: "smtp_host": { 2024-03-16T12:08:09.067 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "smtp_host", 2024-03-16T12:08:09.067 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:09.067 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.067 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.067 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:09.067 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.067 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.067 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.068 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "SMTP server", 2024-03-16T12:08:09.068 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.068 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.068 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.068 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.068 INFO:teuthology.orchestra.run.smithi029.stdout: "smtp_password": { 2024-03-16T12:08:09.068 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "smtp_password", 2024-03-16T12:08:09.068 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:09.068 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.069 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.069 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:09.069 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.069 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.069 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.069 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Password to authenticate with", 2024-03-16T12:08:09.069 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.069 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.070 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.070 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.070 INFO:teuthology.orchestra.run.smithi029.stdout: "smtp_port": { 2024-03-16T12:08:09.070 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "smtp_port", 2024-03-16T12:08:09.070 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-16T12:08:09.070 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.070 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.070 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "465", 2024-03-16T12:08:09.070 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.071 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.071 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.071 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "SMTP port", 2024-03-16T12:08:09.071 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.071 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.071 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.071 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.071 INFO:teuthology.orchestra.run.smithi029.stdout: "smtp_sender": { 2024-03-16T12:08:09.072 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "smtp_sender", 2024-03-16T12:08:09.072 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:09.072 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.072 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.072 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:09.072 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.072 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.072 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.073 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "SMTP envelope sender", 2024-03-16T12:08:09.073 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.073 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.073 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.073 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.073 INFO:teuthology.orchestra.run.smithi029.stdout: "smtp_ssl": { 2024-03-16T12:08:09.073 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "smtp_ssl", 2024-03-16T12:08:09.073 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:09.073 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.074 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.074 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-16T12:08:09.074 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.074 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.074 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.074 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Use SSL to connect to SMTP server", 2024-03-16T12:08:09.074 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.074 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.074 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.075 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.075 INFO:teuthology.orchestra.run.smithi029.stdout: "smtp_user": { 2024-03-16T12:08:09.075 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "smtp_user", 2024-03-16T12:08:09.075 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:09.075 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.075 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.075 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:09.075 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.076 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.076 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.076 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "User to authenticate as", 2024-03-16T12:08:09.076 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.076 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.076 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.076 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:09.076 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:09.076 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.077 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-16T12:08:09.077 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "balancer", 2024-03-16T12:08:09.077 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-03-16T12:08:09.077 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-03-16T12:08:09.077 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-03-16T12:08:09.077 INFO:teuthology.orchestra.run.smithi029.stdout: "active": { 2024-03-16T12:08:09.077 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "active", 2024-03-16T12:08:09.077 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:09.077 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.078 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.078 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-16T12:08:09.078 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.078 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.078 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.078 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "automatically balance PGs across cluster", 2024-03-16T12:08:09.078 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.078 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.079 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.079 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.079 INFO:teuthology.orchestra.run.smithi029.stdout: "begin_time": { 2024-03-16T12:08:09.079 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "begin_time", 2024-03-16T12:08:09.079 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:09.079 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.079 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.079 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "0000", 2024-03-16T12:08:09.079 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.080 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.080 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.080 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "beginning time of day to automatically balance", 2024-03-16T12:08:09.080 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-16T12:08:09.080 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.080 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.080 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.080 INFO:teuthology.orchestra.run.smithi029.stdout: "begin_weekday": { 2024-03-16T12:08:09.080 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "begin_weekday", 2024-03-16T12:08:09.081 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "uint", 2024-03-16T12:08:09.081 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.081 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.081 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "0", 2024-03-16T12:08:09.081 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "0", 2024-03-16T12:08:09.081 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "6", 2024-03-16T12:08:09.081 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.081 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-03-16T12:08:09.081 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-16T12:08:09.082 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.082 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.082 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.082 INFO:teuthology.orchestra.run.smithi029.stdout: "crush_compat_max_iterations": { 2024-03-16T12:08:09.082 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "crush_compat_max_iterations", 2024-03-16T12:08:09.082 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "uint", 2024-03-16T12:08:09.082 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.082 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.083 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "25", 2024-03-16T12:08:09.083 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "1", 2024-03-16T12:08:09.083 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "250", 2024-03-16T12:08:09.083 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.083 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-03-16T12:08:09.083 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.083 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.083 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.083 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.084 INFO:teuthology.orchestra.run.smithi029.stdout: "crush_compat_metrics": { 2024-03-16T12:08:09.084 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "crush_compat_metrics", 2024-03-16T12:08:09.084 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:09.084 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.084 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.084 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "pgs,objects,bytes", 2024-03-16T12:08:09.085 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.085 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.085 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.085 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-03-16T12:08:09.085 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-03-16T12:08:09.085 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.085 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.085 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.085 INFO:teuthology.orchestra.run.smithi029.stdout: "crush_compat_step": { 2024-03-16T12:08:09.086 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "crush_compat_step", 2024-03-16T12:08:09.086 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "float", 2024-03-16T12:08:09.086 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.086 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.086 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "0.5", 2024-03-16T12:08:09.086 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "0.001", 2024-03-16T12:08:09.087 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "0.999", 2024-03-16T12:08:09.087 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.087 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "aggressiveness of optimization", 2024-03-16T12:08:09.087 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-03-16T12:08:09.087 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.087 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.087 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.087 INFO:teuthology.orchestra.run.smithi029.stdout: "end_time": { 2024-03-16T12:08:09.088 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "end_time", 2024-03-16T12:08:09.088 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:09.088 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.088 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.088 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "2359", 2024-03-16T12:08:09.088 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.088 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.089 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.089 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "ending time of day to automatically balance", 2024-03-16T12:08:09.089 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-16T12:08:09.089 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.089 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.089 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.089 INFO:teuthology.orchestra.run.smithi029.stdout: "end_weekday": { 2024-03-16T12:08:09.089 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "end_weekday", 2024-03-16T12:08:09.090 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "uint", 2024-03-16T12:08:09.090 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.090 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.090 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "0", 2024-03-16T12:08:09.090 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "0", 2024-03-16T12:08:09.090 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "6", 2024-03-16T12:08:09.090 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.091 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-03-16T12:08:09.091 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-16T12:08:09.091 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.091 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.091 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.091 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-03-16T12:08:09.091 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-03-16T12:08:09.091 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:09.092 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.092 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.092 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:09.092 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.092 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.092 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-16T12:08:09.093 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-16T12:08:09.093 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-16T12:08:09.093 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-16T12:08:09.093 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-16T12:08:09.093 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-16T12:08:09.093 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-16T12:08:09.093 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:09.094 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.094 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.094 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.094 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.094 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.094 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-03-16T12:08:09.094 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-03-16T12:08:09.095 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:09.095 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.095 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.095 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:09.095 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.095 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.095 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.096 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.096 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.096 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.096 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.096 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.096 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-03-16T12:08:09.097 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-03-16T12:08:09.097 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:09.097 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.097 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.097 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-03-16T12:08:09.097 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.097 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.097 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-16T12:08:09.097 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-16T12:08:09.098 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-16T12:08:09.098 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-16T12:08:09.098 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-16T12:08:09.098 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-16T12:08:09.098 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-16T12:08:09.098 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:09.098 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.098 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.099 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.099 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.099 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.099 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-03-16T12:08:09.099 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-03-16T12:08:09.099 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:09.099 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.099 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.099 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:09.100 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.100 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.100 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.100 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.100 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.100 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.100 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.100 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.100 INFO:teuthology.orchestra.run.smithi029.stdout: "min_score": { 2024-03-16T12:08:09.101 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "min_score", 2024-03-16T12:08:09.101 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "float", 2024-03-16T12:08:09.101 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.101 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.101 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "0", 2024-03-16T12:08:09.101 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.101 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.101 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.101 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-03-16T12:08:09.102 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.102 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.102 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.102 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.102 INFO:teuthology.orchestra.run.smithi029.stdout: "mode": { 2024-03-16T12:08:09.102 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "mode", 2024-03-16T12:08:09.102 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:09.102 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.102 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.103 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "upmap", 2024-03-16T12:08:09.103 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.103 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.103 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-16T12:08:09.103 INFO:teuthology.orchestra.run.smithi029.stdout: "crush-compat", 2024-03-16T12:08:09.103 INFO:teuthology.orchestra.run.smithi029.stdout: "none", 2024-03-16T12:08:09.103 INFO:teuthology.orchestra.run.smithi029.stdout: "read", 2024-03-16T12:08:09.103 INFO:teuthology.orchestra.run.smithi029.stdout: "upmap", 2024-03-16T12:08:09.103 INFO:teuthology.orchestra.run.smithi029.stdout: "upmap-read" 2024-03-16T12:08:09.104 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:09.104 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Balancer mode", 2024-03-16T12:08:09.104 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.104 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.104 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.104 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.104 INFO:teuthology.orchestra.run.smithi029.stdout: "pool_ids": { 2024-03-16T12:08:09.104 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "pool_ids", 2024-03-16T12:08:09.105 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:09.105 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.105 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.105 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:09.105 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.105 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.105 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.105 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-03-16T12:08:09.105 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.106 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.106 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.106 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.106 INFO:teuthology.orchestra.run.smithi029.stdout: "sleep_interval": { 2024-03-16T12:08:09.106 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "sleep_interval", 2024-03-16T12:08:09.106 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "secs", 2024-03-16T12:08:09.106 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.106 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.106 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "60", 2024-03-16T12:08:09.107 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.107 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.107 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.107 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-03-16T12:08:09.107 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.107 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.107 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.107 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.107 INFO:teuthology.orchestra.run.smithi029.stdout: "upmap_max_deviation": { 2024-03-16T12:08:09.108 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "upmap_max_deviation", 2024-03-16T12:08:09.108 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-16T12:08:09.108 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.108 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.108 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "5", 2024-03-16T12:08:09.108 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "1", 2024-03-16T12:08:09.108 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.108 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.108 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "deviation below which no optimization is attempted", 2024-03-16T12:08:09.109 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-03-16T12:08:09.109 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.109 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.109 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.109 INFO:teuthology.orchestra.run.smithi029.stdout: "upmap_max_optimizations": { 2024-03-16T12:08:09.109 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "upmap_max_optimizations", 2024-03-16T12:08:09.109 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "uint", 2024-03-16T12:08:09.109 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.109 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.110 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "10", 2024-03-16T12:08:09.110 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.110 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.110 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.110 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-03-16T12:08:09.110 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.110 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.110 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.111 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:09.111 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:09.111 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.111 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-16T12:08:09.111 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "cephadm", 2024-03-16T12:08:09.111 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-03-16T12:08:09.111 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-03-16T12:08:09.111 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-03-16T12:08:09.111 INFO:teuthology.orchestra.run.smithi029.stdout: "agent_down_multiplier": { 2024-03-16T12:08:09.112 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "agent_down_multiplier", 2024-03-16T12:08:09.112 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "float", 2024-03-16T12:08:09.112 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.112 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.112 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "3.0", 2024-03-16T12:08:09.112 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.112 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.112 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.112 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-03-16T12:08:09.112 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.113 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.113 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.113 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.113 INFO:teuthology.orchestra.run.smithi029.stdout: "agent_refresh_rate": { 2024-03-16T12:08:09.113 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "agent_refresh_rate", 2024-03-16T12:08:09.113 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "secs", 2024-03-16T12:08:09.113 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.113 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.113 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "20", 2024-03-16T12:08:09.114 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.114 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.114 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.114 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-03-16T12:08:09.114 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.114 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.114 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.114 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.115 INFO:teuthology.orchestra.run.smithi029.stdout: "agent_starting_port": { 2024-03-16T12:08:09.115 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "agent_starting_port", 2024-03-16T12:08:09.115 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-16T12:08:09.115 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.115 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.115 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "4721", 2024-03-16T12:08:09.115 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.115 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.115 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.116 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-03-16T12:08:09.116 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.116 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.116 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.116 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.116 INFO:teuthology.orchestra.run.smithi029.stdout: "allow_ptrace": { 2024-03-16T12:08:09.116 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "allow_ptrace", 2024-03-16T12:08:09.116 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:09.116 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.116 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.117 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:09.117 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.117 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.117 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.117 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-03-16T12:08:09.117 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-03-16T12:08:09.117 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.118 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.118 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.118 INFO:teuthology.orchestra.run.smithi029.stdout: "autotune_interval": { 2024-03-16T12:08:09.118 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "autotune_interval", 2024-03-16T12:08:09.118 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "secs", 2024-03-16T12:08:09.118 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.118 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.119 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "600", 2024-03-16T12:08:09.119 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.119 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.119 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.119 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "how frequently to autotune daemon memory", 2024-03-16T12:08:09.119 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.119 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.119 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.120 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.120 INFO:teuthology.orchestra.run.smithi029.stdout: "autotune_memory_target_ratio": { 2024-03-16T12:08:09.120 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "autotune_memory_target_ratio", 2024-03-16T12:08:09.120 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "float", 2024-03-16T12:08:09.120 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.120 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.120 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "0.7", 2024-03-16T12:08:09.121 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.121 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.121 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.121 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-03-16T12:08:09.121 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.121 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.121 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.121 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.121 INFO:teuthology.orchestra.run.smithi029.stdout: "cephadm_log_destination": { 2024-03-16T12:08:09.122 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "cephadm_log_destination", 2024-03-16T12:08:09.122 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:09.122 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.122 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.122 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:09.122 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.122 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.122 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-16T12:08:09.123 INFO:teuthology.orchestra.run.smithi029.stdout: "file", 2024-03-16T12:08:09.123 INFO:teuthology.orchestra.run.smithi029.stdout: "file,syslog", 2024-03-16T12:08:09.123 INFO:teuthology.orchestra.run.smithi029.stdout: "syslog" 2024-03-16T12:08:09.123 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:09.123 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-03-16T12:08:09.123 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.123 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.123 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.123 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.124 INFO:teuthology.orchestra.run.smithi029.stdout: "cgroups_split": { 2024-03-16T12:08:09.124 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "cgroups_split", 2024-03-16T12:08:09.124 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:09.124 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.124 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.124 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-16T12:08:09.124 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.124 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.125 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.125 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-03-16T12:08:09.125 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.125 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.125 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.125 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.125 INFO:teuthology.orchestra.run.smithi029.stdout: "config_checks_enabled": { 2024-03-16T12:08:09.126 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "config_checks_enabled", 2024-03-16T12:08:09.126 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:09.126 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.126 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.126 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:09.126 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.126 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.127 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.127 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-03-16T12:08:09.127 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.127 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.127 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.127 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.127 INFO:teuthology.orchestra.run.smithi029.stdout: "config_dashboard": { 2024-03-16T12:08:09.127 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "config_dashboard", 2024-03-16T12:08:09.128 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:09.128 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.128 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.128 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-16T12:08:09.128 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.128 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.129 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.129 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-03-16T12:08:09.129 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.129 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.129 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.129 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.129 INFO:teuthology.orchestra.run.smithi029.stdout: "container_image_alertmanager": { 2024-03-16T12:08:09.130 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "container_image_alertmanager", 2024-03-16T12:08:09.130 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:09.130 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.130 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.130 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-03-16T12:08:09.130 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.131 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.131 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.131 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Prometheus container image", 2024-03-16T12:08:09.131 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.131 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.132 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.132 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.132 INFO:teuthology.orchestra.run.smithi029.stdout: "container_image_base": { 2024-03-16T12:08:09.133 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "container_image_base", 2024-03-16T12:08:09.133 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:09.133 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.133 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.133 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "quay.io/ceph/ceph", 2024-03-16T12:08:09.133 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.134 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.134 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.134 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Container image name, without the tag", 2024-03-16T12:08:09.134 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.134 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.134 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.134 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.134 INFO:teuthology.orchestra.run.smithi029.stdout: "container_image_elasticsearch": { 2024-03-16T12:08:09.135 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "container_image_elasticsearch", 2024-03-16T12:08:09.135 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:09.135 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.135 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.135 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-03-16T12:08:09.135 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.135 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.135 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.135 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "elasticsearch container image", 2024-03-16T12:08:09.136 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.136 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.136 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.136 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.136 INFO:teuthology.orchestra.run.smithi029.stdout: "container_image_grafana": { 2024-03-16T12:08:09.136 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "container_image_grafana", 2024-03-16T12:08:09.136 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:09.136 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.136 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.137 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "quay.io/ceph/ceph-grafana:9.4.12", 2024-03-16T12:08:09.137 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.137 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.137 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.137 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Prometheus container image", 2024-03-16T12:08:09.137 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.137 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.137 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.137 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.138 INFO:teuthology.orchestra.run.smithi029.stdout: "container_image_haproxy": { 2024-03-16T12:08:09.138 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "container_image_haproxy", 2024-03-16T12:08:09.138 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:09.138 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.138 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.138 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-03-16T12:08:09.138 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.138 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.138 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.139 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "HAproxy container image", 2024-03-16T12:08:09.139 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.139 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.139 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.139 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.139 INFO:teuthology.orchestra.run.smithi029.stdout: "container_image_jaeger_agent": { 2024-03-16T12:08:09.139 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "container_image_jaeger_agent", 2024-03-16T12:08:09.139 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:09.139 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.140 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.140 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-03-16T12:08:09.140 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.140 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.140 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.140 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Jaeger agent container image", 2024-03-16T12:08:09.140 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.140 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.141 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.141 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.141 INFO:teuthology.orchestra.run.smithi029.stdout: "container_image_jaeger_collector": { 2024-03-16T12:08:09.141 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "container_image_jaeger_collector", 2024-03-16T12:08:09.141 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:09.141 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.141 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.141 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-03-16T12:08:09.142 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.142 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.142 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.142 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Jaeger collector container image", 2024-03-16T12:08:09.142 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.142 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.142 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.142 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.142 INFO:teuthology.orchestra.run.smithi029.stdout: "container_image_jaeger_query": { 2024-03-16T12:08:09.143 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "container_image_jaeger_query", 2024-03-16T12:08:09.143 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:09.143 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.143 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.143 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-03-16T12:08:09.143 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.143 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.143 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.143 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Jaeger query container image", 2024-03-16T12:08:09.144 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.144 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.144 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.144 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.144 INFO:teuthology.orchestra.run.smithi029.stdout: "container_image_keepalived": { 2024-03-16T12:08:09.144 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "container_image_keepalived", 2024-03-16T12:08:09.144 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:09.144 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.144 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.145 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-03-16T12:08:09.145 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.145 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.145 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.145 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Keepalived container image", 2024-03-16T12:08:09.145 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.145 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.145 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.145 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.146 INFO:teuthology.orchestra.run.smithi029.stdout: "container_image_loki": { 2024-03-16T12:08:09.146 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "container_image_loki", 2024-03-16T12:08:09.146 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:09.146 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.146 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.146 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-03-16T12:08:09.146 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.146 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.147 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.147 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Loki container image", 2024-03-16T12:08:09.147 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.147 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.147 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.147 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.147 INFO:teuthology.orchestra.run.smithi029.stdout: "container_image_node_exporter": { 2024-03-16T12:08:09.147 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "container_image_node_exporter", 2024-03-16T12:08:09.147 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:09.148 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.148 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.148 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-03-16T12:08:09.148 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.148 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.148 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.148 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Prometheus container image", 2024-03-16T12:08:09.148 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.148 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.149 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.149 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.149 INFO:teuthology.orchestra.run.smithi029.stdout: "container_image_nvmeof": { 2024-03-16T12:08:09.149 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "container_image_nvmeof", 2024-03-16T12:08:09.149 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:09.149 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.149 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.149 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "quay.io/ceph/nvmeof:1.0.0", 2024-03-16T12:08:09.149 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.150 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.150 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.150 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Nvme-of container image", 2024-03-16T12:08:09.150 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.150 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.150 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.150 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.150 INFO:teuthology.orchestra.run.smithi029.stdout: "container_image_prometheus": { 2024-03-16T12:08:09.150 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "container_image_prometheus", 2024-03-16T12:08:09.151 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:09.151 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.151 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.151 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-03-16T12:08:09.151 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.151 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.151 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.151 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Prometheus container image", 2024-03-16T12:08:09.151 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.152 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.152 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.152 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.152 INFO:teuthology.orchestra.run.smithi029.stdout: "container_image_promtail": { 2024-03-16T12:08:09.152 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "container_image_promtail", 2024-03-16T12:08:09.152 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:09.152 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.152 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.152 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-03-16T12:08:09.153 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.153 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.153 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.153 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Promtail container image", 2024-03-16T12:08:09.153 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.153 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.153 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.153 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.154 INFO:teuthology.orchestra.run.smithi029.stdout: "container_image_snmp_gateway": { 2024-03-16T12:08:09.154 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "container_image_snmp_gateway", 2024-03-16T12:08:09.154 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:09.154 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.154 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.154 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-03-16T12:08:09.155 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.155 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.155 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.155 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "SNMP Gateway container image", 2024-03-16T12:08:09.155 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.155 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.155 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.155 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.155 INFO:teuthology.orchestra.run.smithi029.stdout: "container_init": { 2024-03-16T12:08:09.156 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "container_init", 2024-03-16T12:08:09.156 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:09.156 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.156 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.156 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-16T12:08:09.156 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.157 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.157 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.157 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Run podman/docker with `--init`", 2024-03-16T12:08:09.157 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.157 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.157 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.157 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.157 INFO:teuthology.orchestra.run.smithi029.stdout: "daemon_cache_timeout": { 2024-03-16T12:08:09.158 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "daemon_cache_timeout", 2024-03-16T12:08:09.158 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "secs", 2024-03-16T12:08:09.158 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.158 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.158 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "600", 2024-03-16T12:08:09.158 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.159 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.159 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.159 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-03-16T12:08:09.159 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.159 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.159 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.159 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.159 INFO:teuthology.orchestra.run.smithi029.stdout: "default_cephadm_command_timeout": { 2024-03-16T12:08:09.159 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "default_cephadm_command_timeout", 2024-03-16T12:08:09.160 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-16T12:08:09.160 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.160 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.160 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "900", 2024-03-16T12:08:09.160 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.160 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.161 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.161 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-03-16T12:08:09.161 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.161 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.161 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.161 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.161 INFO:teuthology.orchestra.run.smithi029.stdout: "default_registry": { 2024-03-16T12:08:09.162 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "default_registry", 2024-03-16T12:08:09.162 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:09.162 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.162 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.162 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "docker.io", 2024-03-16T12:08:09.162 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.162 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.163 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.163 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-03-16T12:08:09.163 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.163 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.163 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.163 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.164 INFO:teuthology.orchestra.run.smithi029.stdout: "device_cache_timeout": { 2024-03-16T12:08:09.164 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "device_cache_timeout", 2024-03-16T12:08:09.164 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "secs", 2024-03-16T12:08:09.164 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.164 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.164 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "1800", 2024-03-16T12:08:09.164 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.165 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.165 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.165 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "seconds to cache device inventory", 2024-03-16T12:08:09.165 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.165 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.165 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.166 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.166 INFO:teuthology.orchestra.run.smithi029.stdout: "device_enhanced_scan": { 2024-03-16T12:08:09.166 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "device_enhanced_scan", 2024-03-16T12:08:09.166 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:09.166 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.166 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.167 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:09.167 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.167 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.167 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.167 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Use libstoragemgmt during device scans", 2024-03-16T12:08:09.167 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.167 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.167 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.167 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.168 INFO:teuthology.orchestra.run.smithi029.stdout: "facts_cache_timeout": { 2024-03-16T12:08:09.168 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "facts_cache_timeout", 2024-03-16T12:08:09.168 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "secs", 2024-03-16T12:08:09.168 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.168 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.168 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "60", 2024-03-16T12:08:09.168 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.168 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.168 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.169 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "seconds to cache host facts data", 2024-03-16T12:08:09.169 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.169 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.169 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.169 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.169 INFO:teuthology.orchestra.run.smithi029.stdout: "host_check_interval": { 2024-03-16T12:08:09.169 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "host_check_interval", 2024-03-16T12:08:09.169 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "secs", 2024-03-16T12:08:09.169 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.170 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.170 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "600", 2024-03-16T12:08:09.170 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.170 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.170 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.170 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "how frequently to perform a host check", 2024-03-16T12:08:09.170 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.170 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.170 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.171 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.171 INFO:teuthology.orchestra.run.smithi029.stdout: "hw_monitoring": { 2024-03-16T12:08:09.171 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "hw_monitoring", 2024-03-16T12:08:09.171 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:09.171 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.171 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.171 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:09.171 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.171 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.172 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.172 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-03-16T12:08:09.172 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.172 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.172 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.172 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.172 INFO:teuthology.orchestra.run.smithi029.stdout: "inventory_list_all": { 2024-03-16T12:08:09.172 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "inventory_list_all", 2024-03-16T12:08:09.172 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:09.173 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.173 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.173 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:09.173 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.173 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.173 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.173 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-03-16T12:08:09.173 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.173 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.174 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.174 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.174 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-03-16T12:08:09.174 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-03-16T12:08:09.174 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:09.174 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.174 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.174 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:09.175 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.175 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.175 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-16T12:08:09.175 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-16T12:08:09.175 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-16T12:08:09.175 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-16T12:08:09.175 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-16T12:08:09.175 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-16T12:08:09.175 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-16T12:08:09.176 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:09.176 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.176 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.176 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.176 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.176 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.176 INFO:teuthology.orchestra.run.smithi029.stdout: "log_refresh_metadata": { 2024-03-16T12:08:09.176 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_refresh_metadata", 2024-03-16T12:08:09.177 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:09.177 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.177 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.177 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:09.177 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.177 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.177 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.177 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-03-16T12:08:09.177 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.178 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.178 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.178 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.178 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-03-16T12:08:09.178 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-03-16T12:08:09.178 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:09.178 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.178 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.178 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-16T12:08:09.179 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.179 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.179 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.179 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-03-16T12:08:09.179 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.179 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.179 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.179 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.179 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-03-16T12:08:09.180 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-03-16T12:08:09.180 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:09.180 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.180 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.180 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-03-16T12:08:09.180 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.180 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.180 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-16T12:08:09.180 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-16T12:08:09.181 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-16T12:08:09.181 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-16T12:08:09.181 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-16T12:08:09.181 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-16T12:08:09.181 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-16T12:08:09.181 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:09.181 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.181 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.181 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.182 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.182 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.182 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-03-16T12:08:09.182 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-03-16T12:08:09.182 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:09.182 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.182 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.182 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:09.183 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.183 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.183 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.183 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.183 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.183 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.183 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.183 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.184 INFO:teuthology.orchestra.run.smithi029.stdout: "manage_etc_ceph_ceph_conf": { 2024-03-16T12:08:09.184 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-03-16T12:08:09.184 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:09.184 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.184 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.184 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:09.184 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.184 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.184 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.185 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-03-16T12:08:09.185 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.185 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.185 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.185 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.185 INFO:teuthology.orchestra.run.smithi029.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-03-16T12:08:09.185 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-03-16T12:08:09.185 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:09.185 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.186 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.186 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "*", 2024-03-16T12:08:09.186 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.186 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.186 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.186 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-03-16T12:08:09.186 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.186 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.187 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.187 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.187 INFO:teuthology.orchestra.run.smithi029.stdout: "max_count_per_host": { 2024-03-16T12:08:09.187 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "max_count_per_host", 2024-03-16T12:08:09.187 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-16T12:08:09.187 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.188 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.188 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "10", 2024-03-16T12:08:09.188 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.188 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.188 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.188 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "max number of daemons per service per host", 2024-03-16T12:08:09.188 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.188 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.189 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.189 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.189 INFO:teuthology.orchestra.run.smithi029.stdout: "max_osd_draining_count": { 2024-03-16T12:08:09.189 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "max_osd_draining_count", 2024-03-16T12:08:09.189 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-16T12:08:09.189 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.189 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.189 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "10", 2024-03-16T12:08:09.190 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.190 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.190 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.190 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-03-16T12:08:09.190 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.190 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.191 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.191 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.191 INFO:teuthology.orchestra.run.smithi029.stdout: "migration_current": { 2024-03-16T12:08:09.191 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "migration_current", 2024-03-16T12:08:09.191 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-16T12:08:09.191 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.191 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.191 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:09.192 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.192 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.192 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.192 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "internal - do not modify", 2024-03-16T12:08:09.192 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.192 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.192 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.193 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.193 INFO:teuthology.orchestra.run.smithi029.stdout: "mode": { 2024-03-16T12:08:09.193 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "mode", 2024-03-16T12:08:09.193 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:09.193 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.193 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.193 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "root", 2024-03-16T12:08:09.193 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.194 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.194 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-16T12:08:09.194 INFO:teuthology.orchestra.run.smithi029.stdout: "cephadm-package", 2024-03-16T12:08:09.194 INFO:teuthology.orchestra.run.smithi029.stdout: "root" 2024-03-16T12:08:09.194 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:09.194 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "mode for remote execution of cephadm", 2024-03-16T12:08:09.194 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.194 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.195 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.195 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.195 INFO:teuthology.orchestra.run.smithi029.stdout: "oob_default_addr": { 2024-03-16T12:08:09.195 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "oob_default_addr", 2024-03-16T12:08:09.195 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:09.195 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.195 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.195 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "169.254.1.1", 2024-03-16T12:08:09.195 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.196 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.196 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.196 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Default address for RedFish API (oob management).", 2024-03-16T12:08:09.196 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.196 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.196 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.196 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.197 INFO:teuthology.orchestra.run.smithi029.stdout: "prometheus_alerts_path": { 2024-03-16T12:08:09.197 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "prometheus_alerts_path", 2024-03-16T12:08:09.197 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:09.197 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.197 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.197 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-03-16T12:08:09.197 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.198 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.198 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.198 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-03-16T12:08:09.198 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.198 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.198 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.198 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.199 INFO:teuthology.orchestra.run.smithi029.stdout: "registry_insecure": { 2024-03-16T12:08:09.199 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "registry_insecure", 2024-03-16T12:08:09.199 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:09.199 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.199 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.199 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:09.200 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.200 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.200 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.200 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-03-16T12:08:09.200 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.200 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.200 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.201 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.201 INFO:teuthology.orchestra.run.smithi029.stdout: "registry_password": { 2024-03-16T12:08:09.201 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "registry_password", 2024-03-16T12:08:09.201 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:09.201 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.201 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.202 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:09.202 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.202 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.202 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.202 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-03-16T12:08:09.203 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.203 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.203 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.203 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.203 INFO:teuthology.orchestra.run.smithi029.stdout: "registry_url": { 2024-03-16T12:08:09.203 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "registry_url", 2024-03-16T12:08:09.203 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:09.203 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.204 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.204 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:09.204 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.204 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.204 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.204 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-03-16T12:08:09.204 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.204 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.204 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.205 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.205 INFO:teuthology.orchestra.run.smithi029.stdout: "registry_username": { 2024-03-16T12:08:09.205 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "registry_username", 2024-03-16T12:08:09.205 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:09.205 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.205 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.205 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:09.205 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.205 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.206 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.206 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-03-16T12:08:09.206 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.206 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.206 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.206 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.206 INFO:teuthology.orchestra.run.smithi029.stdout: "secure_monitoring_stack": { 2024-03-16T12:08:09.206 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "secure_monitoring_stack", 2024-03-16T12:08:09.207 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:09.207 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.207 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.207 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:09.207 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.207 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.207 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.207 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-03-16T12:08:09.207 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.208 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.208 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.208 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.208 INFO:teuthology.orchestra.run.smithi029.stdout: "service_discovery_port": { 2024-03-16T12:08:09.208 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "service_discovery_port", 2024-03-16T12:08:09.208 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-16T12:08:09.208 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.208 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.209 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "8765", 2024-03-16T12:08:09.209 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.209 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.209 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.209 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "cephadm service discovery port", 2024-03-16T12:08:09.209 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.209 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.209 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.209 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.209 INFO:teuthology.orchestra.run.smithi029.stdout: "ssh_config_file": { 2024-03-16T12:08:09.210 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "ssh_config_file", 2024-03-16T12:08:09.210 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:09.210 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.210 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.210 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:09.210 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.210 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.210 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.211 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-03-16T12:08:09.211 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.211 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.211 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.211 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.211 INFO:teuthology.orchestra.run.smithi029.stdout: "use_agent": { 2024-03-16T12:08:09.211 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "use_agent", 2024-03-16T12:08:09.211 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:09.211 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.212 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.212 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:09.212 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.212 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.212 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.212 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-03-16T12:08:09.212 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.212 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.212 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.212 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.213 INFO:teuthology.orchestra.run.smithi029.stdout: "use_repo_digest": { 2024-03-16T12:08:09.213 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "use_repo_digest", 2024-03-16T12:08:09.213 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:09.213 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.213 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.213 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-16T12:08:09.213 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.213 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.213 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.214 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-03-16T12:08:09.214 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.214 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.214 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.214 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.214 INFO:teuthology.orchestra.run.smithi029.stdout: "warn_on_failed_host_check": { 2024-03-16T12:08:09.214 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "warn_on_failed_host_check", 2024-03-16T12:08:09.214 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:09.214 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.215 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.215 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-16T12:08:09.215 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.215 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.215 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.215 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "raise a health warning if the host check fails", 2024-03-16T12:08:09.215 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.215 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.216 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.216 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.216 INFO:teuthology.orchestra.run.smithi029.stdout: "warn_on_stray_daemons": { 2024-03-16T12:08:09.216 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "warn_on_stray_daemons", 2024-03-16T12:08:09.216 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:09.216 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.216 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.216 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-16T12:08:09.217 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.217 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.217 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.217 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-03-16T12:08:09.217 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.217 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.217 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.217 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.217 INFO:teuthology.orchestra.run.smithi029.stdout: "warn_on_stray_hosts": { 2024-03-16T12:08:09.218 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "warn_on_stray_hosts", 2024-03-16T12:08:09.218 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:09.218 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.218 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.218 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-16T12:08:09.218 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.218 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.218 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.219 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-03-16T12:08:09.219 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.219 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.219 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.219 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:09.219 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:09.219 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.219 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-16T12:08:09.219 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "crash", 2024-03-16T12:08:09.220 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-03-16T12:08:09.220 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-03-16T12:08:09.220 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-03-16T12:08:09.220 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-03-16T12:08:09.220 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-03-16T12:08:09.220 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:09.220 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.220 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.220 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:09.220 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.220 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.221 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-16T12:08:09.221 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-16T12:08:09.221 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-16T12:08:09.221 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-16T12:08:09.221 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-16T12:08:09.221 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-16T12:08:09.221 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-16T12:08:09.221 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:09.221 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.222 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.222 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.222 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.222 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.222 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-03-16T12:08:09.222 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-03-16T12:08:09.222 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:09.222 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.222 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.223 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:09.223 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.223 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.223 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.223 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.223 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.223 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.223 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.223 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.224 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-03-16T12:08:09.224 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-03-16T12:08:09.224 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:09.224 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.224 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.224 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-03-16T12:08:09.225 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.225 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.225 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-16T12:08:09.225 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-16T12:08:09.225 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-16T12:08:09.225 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-16T12:08:09.225 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-16T12:08:09.225 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-16T12:08:09.225 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-16T12:08:09.226 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:09.226 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.226 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.226 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.226 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.226 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.227 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-03-16T12:08:09.227 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-03-16T12:08:09.227 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:09.227 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.227 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.227 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:09.227 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.227 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.227 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.228 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.228 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.228 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.228 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.228 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.228 INFO:teuthology.orchestra.run.smithi029.stdout: "retain_interval": { 2024-03-16T12:08:09.229 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "retain_interval", 2024-03-16T12:08:09.229 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "secs", 2024-03-16T12:08:09.229 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.229 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.229 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "31536000", 2024-03-16T12:08:09.229 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.229 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.229 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.230 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "how long to retain crashes before pruning them", 2024-03-16T12:08:09.230 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.230 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.230 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.230 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.230 INFO:teuthology.orchestra.run.smithi029.stdout: "warn_recent_interval": { 2024-03-16T12:08:09.231 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "warn_recent_interval", 2024-03-16T12:08:09.231 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "secs", 2024-03-16T12:08:09.231 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.231 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.231 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "1209600", 2024-03-16T12:08:09.231 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.231 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.231 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.232 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "time interval in which to warn about recent crashes", 2024-03-16T12:08:09.232 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.232 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.232 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.232 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:09.232 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:09.232 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.233 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-16T12:08:09.233 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "dashboard", 2024-03-16T12:08:09.233 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-03-16T12:08:09.233 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-03-16T12:08:09.233 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-03-16T12:08:09.233 INFO:teuthology.orchestra.run.smithi029.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-03-16T12:08:09.234 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-03-16T12:08:09.234 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-16T12:08:09.234 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.234 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.234 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "10", 2024-03-16T12:08:09.234 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.234 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.235 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.235 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.235 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.235 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.235 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.235 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.236 INFO:teuthology.orchestra.run.smithi029.stdout: "ALERTMANAGER_API_HOST": { 2024-03-16T12:08:09.236 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "ALERTMANAGER_API_HOST", 2024-03-16T12:08:09.236 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:09.236 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.236 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.236 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:09.237 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.237 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.237 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.237 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.237 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.237 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.237 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.237 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.237 INFO:teuthology.orchestra.run.smithi029.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-03-16T12:08:09.238 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-03-16T12:08:09.238 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:09.238 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.238 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.238 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-16T12:08:09.238 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.238 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.238 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.239 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.239 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.239 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.239 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.239 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.239 INFO:teuthology.orchestra.run.smithi029.stdout: "AUDIT_API_ENABLED": { 2024-03-16T12:08:09.239 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "AUDIT_API_ENABLED", 2024-03-16T12:08:09.239 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:09.239 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.240 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.240 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:09.240 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.240 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.240 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.240 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.240 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.240 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.240 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.241 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.241 INFO:teuthology.orchestra.run.smithi029.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-03-16T12:08:09.241 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-03-16T12:08:09.241 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:09.241 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.241 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.241 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-16T12:08:09.241 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.242 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.242 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.242 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.242 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.242 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.242 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.242 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.242 INFO:teuthology.orchestra.run.smithi029.stdout: "ENABLE_BROWSABLE_API": { 2024-03-16T12:08:09.242 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "ENABLE_BROWSABLE_API", 2024-03-16T12:08:09.243 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:09.243 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.243 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.243 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-16T12:08:09.243 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.243 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.243 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.243 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.243 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.244 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.244 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.244 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.244 INFO:teuthology.orchestra.run.smithi029.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-03-16T12:08:09.244 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-03-16T12:08:09.244 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:09.244 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.244 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.244 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-16T12:08:09.245 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.245 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.245 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.245 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.245 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.245 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.245 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.245 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.246 INFO:teuthology.orchestra.run.smithi029.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-03-16T12:08:09.246 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-03-16T12:08:09.246 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:09.246 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.246 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.246 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-16T12:08:09.246 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.246 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.247 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.247 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.247 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.247 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.247 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.247 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.247 INFO:teuthology.orchestra.run.smithi029.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-03-16T12:08:09.247 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-03-16T12:08:09.247 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:09.248 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.248 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.248 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-16T12:08:09.248 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.248 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.248 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.248 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.248 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.248 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.249 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.249 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.249 INFO:teuthology.orchestra.run.smithi029.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-03-16T12:08:09.249 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-03-16T12:08:09.249 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:09.249 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.249 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.249 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-16T12:08:09.250 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.250 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.250 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.250 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.250 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.250 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.250 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.250 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.250 INFO:teuthology.orchestra.run.smithi029.stdout: "FEATURE_TOGGLE_NFS": { 2024-03-16T12:08:09.251 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-03-16T12:08:09.251 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:09.251 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.251 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.251 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-16T12:08:09.251 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.251 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.251 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.251 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.252 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.252 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.252 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.252 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.252 INFO:teuthology.orchestra.run.smithi029.stdout: "FEATURE_TOGGLE_RBD": { 2024-03-16T12:08:09.252 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-03-16T12:08:09.252 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:09.252 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.252 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.253 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-16T12:08:09.253 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.253 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.253 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.253 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.253 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.253 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.253 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.253 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.254 INFO:teuthology.orchestra.run.smithi029.stdout: "FEATURE_TOGGLE_RGW": { 2024-03-16T12:08:09.254 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-03-16T12:08:09.254 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:09.254 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.254 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.254 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-16T12:08:09.254 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.254 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.255 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.255 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.255 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.255 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.255 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.255 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.255 INFO:teuthology.orchestra.run.smithi029.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-03-16T12:08:09.255 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-03-16T12:08:09.255 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:09.255 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.256 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.256 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:09.256 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.256 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.256 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.256 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.256 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.256 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.257 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.257 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.257 INFO:teuthology.orchestra.run.smithi029.stdout: "GRAFANA_API_PASSWORD": { 2024-03-16T12:08:09.257 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "GRAFANA_API_PASSWORD", 2024-03-16T12:08:09.257 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:09.257 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.257 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.257 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "admin", 2024-03-16T12:08:09.258 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.258 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.258 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.258 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.258 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.258 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.259 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.259 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.259 INFO:teuthology.orchestra.run.smithi029.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-03-16T12:08:09.259 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-03-16T12:08:09.259 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:09.259 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.259 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.259 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-16T12:08:09.259 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.260 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.260 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.260 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.260 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.260 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.260 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.261 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.261 INFO:teuthology.orchestra.run.smithi029.stdout: "GRAFANA_API_URL": { 2024-03-16T12:08:09.261 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "GRAFANA_API_URL", 2024-03-16T12:08:09.261 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:09.261 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.261 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.261 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:09.261 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.261 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.262 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.262 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.262 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.262 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.262 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.262 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.263 INFO:teuthology.orchestra.run.smithi029.stdout: "GRAFANA_API_USERNAME": { 2024-03-16T12:08:09.263 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "GRAFANA_API_USERNAME", 2024-03-16T12:08:09.263 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:09.263 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.263 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.263 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "admin", 2024-03-16T12:08:09.263 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.263 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.264 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.264 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.264 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.264 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.264 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.264 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.265 INFO:teuthology.orchestra.run.smithi029.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-03-16T12:08:09.265 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-03-16T12:08:09.265 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:09.265 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.265 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.265 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:09.265 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.265 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.266 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.266 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.266 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.266 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.266 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.266 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.266 INFO:teuthology.orchestra.run.smithi029.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-03-16T12:08:09.267 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-03-16T12:08:09.267 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:09.267 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.267 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.267 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:09.267 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.267 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.267 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.267 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.267 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.268 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.268 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.268 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.268 INFO:teuthology.orchestra.run.smithi029.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-03-16T12:08:09.268 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-03-16T12:08:09.268 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:09.268 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.268 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.269 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-16T12:08:09.269 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.269 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.269 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.269 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.269 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.269 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.269 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.269 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.270 INFO:teuthology.orchestra.run.smithi029.stdout: "ISSUE_TRACKER_API_KEY": { 2024-03-16T12:08:09.270 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-03-16T12:08:09.270 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:09.270 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.270 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.270 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:09.271 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.271 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.271 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.271 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.271 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.271 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.272 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.272 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.272 INFO:teuthology.orchestra.run.smithi029.stdout: "MULTICLUSTER_CONFIG": { 2024-03-16T12:08:09.272 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "MULTICLUSTER_CONFIG", 2024-03-16T12:08:09.272 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:09.273 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.273 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.273 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "{}", 2024-03-16T12:08:09.273 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.273 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.273 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.273 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.274 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.274 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.274 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.274 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.274 INFO:teuthology.orchestra.run.smithi029.stdout: "PROMETHEUS_API_HOST": { 2024-03-16T12:08:09.274 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "PROMETHEUS_API_HOST", 2024-03-16T12:08:09.274 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:09.274 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.274 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.275 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:09.275 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.275 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.275 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.275 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.275 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.275 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.275 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.275 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.276 INFO:teuthology.orchestra.run.smithi029.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-03-16T12:08:09.276 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-03-16T12:08:09.276 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:09.276 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.276 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.276 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-16T12:08:09.276 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.276 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.276 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.277 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.277 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.277 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.277 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.277 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.277 INFO:teuthology.orchestra.run.smithi029.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-03-16T12:08:09.277 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-03-16T12:08:09.277 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:09.277 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.278 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.278 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:09.278 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.278 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.278 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.278 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.278 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.278 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.279 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.279 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.279 INFO:teuthology.orchestra.run.smithi029.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-03-16T12:08:09.279 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-03-16T12:08:09.279 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:09.279 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.279 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.279 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:09.279 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.280 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.280 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.280 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.280 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.280 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.280 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.280 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.280 INFO:teuthology.orchestra.run.smithi029.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-03-16T12:08:09.280 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-03-16T12:08:09.281 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:09.281 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.281 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.281 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-16T12:08:09.281 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.281 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.281 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.281 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.281 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.282 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.282 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.282 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.282 INFO:teuthology.orchestra.run.smithi029.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-03-16T12:08:09.282 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-03-16T12:08:09.282 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:09.282 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.282 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.282 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-16T12:08:09.283 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.283 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.283 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.283 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.283 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.283 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.283 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.283 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.283 INFO:teuthology.orchestra.run.smithi029.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-03-16T12:08:09.284 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-03-16T12:08:09.284 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:09.284 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.284 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.284 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:09.284 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.284 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.284 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.284 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.285 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.285 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.285 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.285 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.285 INFO:teuthology.orchestra.run.smithi029.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-03-16T12:08:09.285 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-03-16T12:08:09.285 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:09.285 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.285 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.286 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:09.286 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.286 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.286 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.286 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.286 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.286 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.286 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.286 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.287 INFO:teuthology.orchestra.run.smithi029.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-03-16T12:08:09.287 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-03-16T12:08:09.287 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:09.287 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.287 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.287 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:09.287 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.287 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.287 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.288 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.288 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.288 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.288 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.288 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.288 INFO:teuthology.orchestra.run.smithi029.stdout: "PWD_POLICY_ENABLED": { 2024-03-16T12:08:09.288 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "PWD_POLICY_ENABLED", 2024-03-16T12:08:09.288 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:09.288 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.289 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.289 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-16T12:08:09.289 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.289 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.289 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.289 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.289 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.289 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.289 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.290 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.290 INFO:teuthology.orchestra.run.smithi029.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-03-16T12:08:09.290 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-03-16T12:08:09.290 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:09.290 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.290 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.290 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-03-16T12:08:09.290 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.291 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.291 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.291 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.291 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.291 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.291 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.291 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.291 INFO:teuthology.orchestra.run.smithi029.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-03-16T12:08:09.291 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-03-16T12:08:09.292 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-16T12:08:09.292 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.292 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.292 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "10", 2024-03-16T12:08:09.292 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.292 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.292 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.292 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.292 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.293 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.293 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.293 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.293 INFO:teuthology.orchestra.run.smithi029.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-03-16T12:08:09.293 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-03-16T12:08:09.293 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-16T12:08:09.293 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.293 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.293 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "8", 2024-03-16T12:08:09.294 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.294 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.294 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.294 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.294 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.294 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.295 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.295 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.295 INFO:teuthology.orchestra.run.smithi029.stdout: "REST_REQUESTS_TIMEOUT": { 2024-03-16T12:08:09.295 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-03-16T12:08:09.295 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-16T12:08:09.295 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.295 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.295 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "45", 2024-03-16T12:08:09.296 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.296 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.296 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.296 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.296 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.296 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.296 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.296 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.297 INFO:teuthology.orchestra.run.smithi029.stdout: "RGW_API_ACCESS_KEY": { 2024-03-16T12:08:09.297 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "RGW_API_ACCESS_KEY", 2024-03-16T12:08:09.297 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:09.297 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.297 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.297 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:09.297 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.298 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.298 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.298 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.298 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.298 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.298 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.298 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.298 INFO:teuthology.orchestra.run.smithi029.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-03-16T12:08:09.299 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-03-16T12:08:09.299 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:09.299 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.299 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.299 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "admin", 2024-03-16T12:08:09.299 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.299 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.300 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.300 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.300 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.300 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.300 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.300 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.300 INFO:teuthology.orchestra.run.smithi029.stdout: "RGW_API_SECRET_KEY": { 2024-03-16T12:08:09.300 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "RGW_API_SECRET_KEY", 2024-03-16T12:08:09.301 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:09.301 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.301 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.301 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:09.301 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.301 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.301 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.302 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.302 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.302 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.302 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.302 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.302 INFO:teuthology.orchestra.run.smithi029.stdout: "RGW_API_SSL_VERIFY": { 2024-03-16T12:08:09.302 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "RGW_API_SSL_VERIFY", 2024-03-16T12:08:09.303 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:09.303 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.303 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.303 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-16T12:08:09.303 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.303 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.303 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.304 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.304 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.304 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.304 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.304 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.304 INFO:teuthology.orchestra.run.smithi029.stdout: "UNSAFE_TLS_v1_2": { 2024-03-16T12:08:09.305 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "UNSAFE_TLS_v1_2", 2024-03-16T12:08:09.305 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:09.305 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.305 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.305 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:09.305 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.306 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.306 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.306 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.306 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.306 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.306 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.307 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.307 INFO:teuthology.orchestra.run.smithi029.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-03-16T12:08:09.307 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-03-16T12:08:09.307 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-16T12:08:09.307 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.307 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.307 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "0", 2024-03-16T12:08:09.307 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.308 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.308 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.308 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.308 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.308 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.308 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.308 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.308 INFO:teuthology.orchestra.run.smithi029.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-03-16T12:08:09.308 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-03-16T12:08:09.309 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-16T12:08:09.309 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.309 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.309 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "10", 2024-03-16T12:08:09.309 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.309 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.309 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.309 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.309 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.310 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.310 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.310 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.310 INFO:teuthology.orchestra.run.smithi029.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-03-16T12:08:09.310 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-03-16T12:08:09.310 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-16T12:08:09.310 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.310 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.311 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "5", 2024-03-16T12:08:09.311 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.311 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.311 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.311 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.311 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.311 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.311 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.311 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.312 INFO:teuthology.orchestra.run.smithi029.stdout: "cross_origin_url": { 2024-03-16T12:08:09.312 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "cross_origin_url", 2024-03-16T12:08:09.312 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:09.312 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.312 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.312 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:09.312 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.312 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.312 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.313 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.313 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.313 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.313 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.313 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.313 INFO:teuthology.orchestra.run.smithi029.stdout: "crt_file": { 2024-03-16T12:08:09.313 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "crt_file", 2024-03-16T12:08:09.313 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:09.314 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.314 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.314 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:09.314 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.314 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.314 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.314 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.314 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.314 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.315 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.315 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.315 INFO:teuthology.orchestra.run.smithi029.stdout: "debug": { 2024-03-16T12:08:09.315 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "debug", 2024-03-16T12:08:09.315 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:09.315 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.315 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.315 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:09.316 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.316 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.316 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.316 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Enable/disable debug options", 2024-03-16T12:08:09.316 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.316 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.316 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.316 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.316 INFO:teuthology.orchestra.run.smithi029.stdout: "jwt_token_ttl": { 2024-03-16T12:08:09.317 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "jwt_token_ttl", 2024-03-16T12:08:09.317 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-16T12:08:09.317 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.317 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.317 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "28800", 2024-03-16T12:08:09.317 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.317 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.317 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.317 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.318 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.318 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.318 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.318 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.318 INFO:teuthology.orchestra.run.smithi029.stdout: "key_file": { 2024-03-16T12:08:09.318 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "key_file", 2024-03-16T12:08:09.318 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:09.318 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.318 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.319 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:09.319 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.319 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.319 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.319 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.319 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.319 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.319 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.319 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.320 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-03-16T12:08:09.320 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-03-16T12:08:09.320 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:09.320 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.320 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.320 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:09.320 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.320 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.321 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-16T12:08:09.321 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-16T12:08:09.321 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-16T12:08:09.321 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-16T12:08:09.321 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-16T12:08:09.321 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-16T12:08:09.321 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-16T12:08:09.321 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:09.321 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.321 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.322 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.322 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.322 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.322 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-03-16T12:08:09.322 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-03-16T12:08:09.322 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:09.322 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.322 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.323 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:09.323 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.323 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.323 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.323 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.323 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.323 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.323 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.323 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.324 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-03-16T12:08:09.324 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-03-16T12:08:09.324 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:09.324 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.324 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.324 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-03-16T12:08:09.324 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.324 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.324 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-16T12:08:09.325 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-16T12:08:09.325 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-16T12:08:09.325 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-16T12:08:09.325 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-16T12:08:09.325 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-16T12:08:09.325 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-16T12:08:09.325 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:09.325 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.325 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.325 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.326 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.326 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.326 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-03-16T12:08:09.326 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-03-16T12:08:09.326 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:09.326 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.326 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.326 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:09.327 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.327 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.327 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.327 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.327 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.327 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.327 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.327 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.328 INFO:teuthology.orchestra.run.smithi029.stdout: "motd": { 2024-03-16T12:08:09.328 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "motd", 2024-03-16T12:08:09.328 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:09.328 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.328 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.328 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:09.329 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.329 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.329 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.329 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "The message of the day", 2024-03-16T12:08:09.329 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.329 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.329 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.329 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.330 INFO:teuthology.orchestra.run.smithi029.stdout: "redirect_resolve_ip_addr": { 2024-03-16T12:08:09.330 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "redirect_resolve_ip_addr", 2024-03-16T12:08:09.330 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:09.330 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.330 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.330 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:09.330 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.331 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.331 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.331 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.331 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.331 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.331 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.331 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.331 INFO:teuthology.orchestra.run.smithi029.stdout: "server_addr": { 2024-03-16T12:08:09.332 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "server_addr", 2024-03-16T12:08:09.332 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:09.332 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.332 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.332 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "::", 2024-03-16T12:08:09.332 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.332 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.332 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.333 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.333 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.333 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.333 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.333 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.333 INFO:teuthology.orchestra.run.smithi029.stdout: "server_port": { 2024-03-16T12:08:09.333 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "server_port", 2024-03-16T12:08:09.334 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-16T12:08:09.334 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.334 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.334 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "8080", 2024-03-16T12:08:09.334 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.334 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.334 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.335 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.335 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.335 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.335 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.335 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.335 INFO:teuthology.orchestra.run.smithi029.stdout: "ssl": { 2024-03-16T12:08:09.335 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "ssl", 2024-03-16T12:08:09.335 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:09.336 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.336 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.336 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-16T12:08:09.336 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.336 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.336 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.336 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.337 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.337 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.337 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.337 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.337 INFO:teuthology.orchestra.run.smithi029.stdout: "ssl_server_port": { 2024-03-16T12:08:09.337 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "ssl_server_port", 2024-03-16T12:08:09.338 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-16T12:08:09.338 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.338 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.338 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "8443", 2024-03-16T12:08:09.338 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.338 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.338 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.338 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.339 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.339 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.339 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.339 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.339 INFO:teuthology.orchestra.run.smithi029.stdout: "standby_behaviour": { 2024-03-16T12:08:09.339 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "standby_behaviour", 2024-03-16T12:08:09.339 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:09.339 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.339 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.340 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "redirect", 2024-03-16T12:08:09.340 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.340 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.340 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-16T12:08:09.340 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-16T12:08:09.340 INFO:teuthology.orchestra.run.smithi029.stdout: "redirect" 2024-03-16T12:08:09.340 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:09.340 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.340 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.341 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.341 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.341 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.341 INFO:teuthology.orchestra.run.smithi029.stdout: "standby_error_status_code": { 2024-03-16T12:08:09.341 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "standby_error_status_code", 2024-03-16T12:08:09.341 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-16T12:08:09.341 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.341 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.341 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "500", 2024-03-16T12:08:09.342 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "400", 2024-03-16T12:08:09.342 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "599", 2024-03-16T12:08:09.342 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.342 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.342 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.343 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.343 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.343 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.343 INFO:teuthology.orchestra.run.smithi029.stdout: "url_prefix": { 2024-03-16T12:08:09.343 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "url_prefix", 2024-03-16T12:08:09.343 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:09.344 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.344 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.344 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:09.344 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.344 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.344 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.344 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.345 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.345 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.345 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.345 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:09.345 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:09.345 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.345 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-16T12:08:09.345 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "devicehealth", 2024-03-16T12:08:09.345 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-03-16T12:08:09.346 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-03-16T12:08:09.346 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-03-16T12:08:09.346 INFO:teuthology.orchestra.run.smithi029.stdout: "enable_monitoring": { 2024-03-16T12:08:09.346 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "enable_monitoring", 2024-03-16T12:08:09.346 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:09.346 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.346 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.346 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-16T12:08:09.347 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.347 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.347 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.347 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "monitor device health metrics", 2024-03-16T12:08:09.347 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.347 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.347 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.347 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.347 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-03-16T12:08:09.348 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-03-16T12:08:09.348 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:09.348 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.348 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.348 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:09.348 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.348 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.348 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-16T12:08:09.348 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-16T12:08:09.349 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-16T12:08:09.349 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-16T12:08:09.349 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-16T12:08:09.349 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-16T12:08:09.349 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-16T12:08:09.349 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:09.349 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.349 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.349 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.350 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.350 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.350 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-03-16T12:08:09.350 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-03-16T12:08:09.350 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:09.350 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.350 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.350 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:09.350 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.351 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.351 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.351 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.351 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.351 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.351 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.351 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.351 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-03-16T12:08:09.351 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-03-16T12:08:09.352 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:09.352 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.352 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.352 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-03-16T12:08:09.352 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.352 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.352 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-16T12:08:09.352 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-16T12:08:09.353 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-16T12:08:09.353 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-16T12:08:09.353 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-16T12:08:09.353 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-16T12:08:09.353 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-16T12:08:09.353 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:09.353 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.353 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.353 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.354 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.354 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.354 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-03-16T12:08:09.354 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-03-16T12:08:09.354 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:09.354 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.354 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.354 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:09.355 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.355 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.355 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.355 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.355 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.355 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.355 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.355 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.355 INFO:teuthology.orchestra.run.smithi029.stdout: "mark_out_threshold": { 2024-03-16T12:08:09.355 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "mark_out_threshold", 2024-03-16T12:08:09.356 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "secs", 2024-03-16T12:08:09.356 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.356 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.356 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "2419200", 2024-03-16T12:08:09.356 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.356 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.356 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.356 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-03-16T12:08:09.356 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.357 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.357 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.357 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.357 INFO:teuthology.orchestra.run.smithi029.stdout: "pool_name": { 2024-03-16T12:08:09.357 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "pool_name", 2024-03-16T12:08:09.357 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:09.357 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.357 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.357 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "device_health_metrics", 2024-03-16T12:08:09.358 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.358 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.358 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.358 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "name of pool in which to store device health metrics", 2024-03-16T12:08:09.358 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.358 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.358 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.358 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.359 INFO:teuthology.orchestra.run.smithi029.stdout: "retention_period": { 2024-03-16T12:08:09.359 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "retention_period", 2024-03-16T12:08:09.359 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "secs", 2024-03-16T12:08:09.359 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.359 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.359 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "15552000", 2024-03-16T12:08:09.359 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.359 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.359 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.360 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "how long to retain device health metrics", 2024-03-16T12:08:09.360 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.360 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.360 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.360 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.360 INFO:teuthology.orchestra.run.smithi029.stdout: "scrape_frequency": { 2024-03-16T12:08:09.360 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "scrape_frequency", 2024-03-16T12:08:09.360 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "secs", 2024-03-16T12:08:09.360 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.361 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.361 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "86400", 2024-03-16T12:08:09.361 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.361 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.361 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.361 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "how frequently to scrape device health metrics", 2024-03-16T12:08:09.361 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.361 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.361 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.362 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.362 INFO:teuthology.orchestra.run.smithi029.stdout: "self_heal": { 2024-03-16T12:08:09.362 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "self_heal", 2024-03-16T12:08:09.362 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:09.362 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.362 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.362 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-16T12:08:09.362 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.363 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.363 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.363 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-03-16T12:08:09.363 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.363 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.363 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.363 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.363 INFO:teuthology.orchestra.run.smithi029.stdout: "sleep_interval": { 2024-03-16T12:08:09.364 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "sleep_interval", 2024-03-16T12:08:09.364 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "secs", 2024-03-16T12:08:09.364 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.364 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.364 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "600", 2024-03-16T12:08:09.364 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.364 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.364 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.364 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "how frequently to wake up and check device health", 2024-03-16T12:08:09.365 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.365 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.365 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.365 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.365 INFO:teuthology.orchestra.run.smithi029.stdout: "warn_threshold": { 2024-03-16T12:08:09.365 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "warn_threshold", 2024-03-16T12:08:09.366 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "secs", 2024-03-16T12:08:09.366 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.366 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.366 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "7257600", 2024-03-16T12:08:09.366 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.366 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.366 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.366 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-03-16T12:08:09.366 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.367 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.367 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.367 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:09.367 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:09.367 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.367 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-16T12:08:09.368 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "diskprediction_local", 2024-03-16T12:08:09.368 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-03-16T12:08:09.368 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-03-16T12:08:09.368 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-03-16T12:08:09.368 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-03-16T12:08:09.368 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-03-16T12:08:09.368 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:09.368 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.369 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.369 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:09.369 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.369 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.369 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-16T12:08:09.369 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-16T12:08:09.369 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-16T12:08:09.369 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-16T12:08:09.370 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-16T12:08:09.370 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-16T12:08:09.370 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-16T12:08:09.370 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:09.370 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.370 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.370 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.371 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.371 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.371 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-03-16T12:08:09.371 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-03-16T12:08:09.371 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:09.371 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.371 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.372 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:09.372 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.372 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.372 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.372 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.372 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.372 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.372 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.373 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.373 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-03-16T12:08:09.373 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-03-16T12:08:09.373 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:09.373 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.373 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.374 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-03-16T12:08:09.374 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.374 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.374 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-16T12:08:09.374 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-16T12:08:09.374 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-16T12:08:09.374 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-16T12:08:09.375 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-16T12:08:09.375 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-16T12:08:09.375 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-16T12:08:09.375 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:09.375 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.375 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.375 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.376 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.376 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.376 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-03-16T12:08:09.376 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-03-16T12:08:09.376 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:09.376 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.377 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.377 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:09.377 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.377 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.377 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.377 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.377 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.378 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.378 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.378 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.378 INFO:teuthology.orchestra.run.smithi029.stdout: "predict_interval": { 2024-03-16T12:08:09.378 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "predict_interval", 2024-03-16T12:08:09.378 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:09.378 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.378 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.379 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "86400", 2024-03-16T12:08:09.379 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.379 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.379 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.379 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.379 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.379 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.379 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.379 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.380 INFO:teuthology.orchestra.run.smithi029.stdout: "predictor_model": { 2024-03-16T12:08:09.380 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "predictor_model", 2024-03-16T12:08:09.380 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:09.380 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.380 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.380 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "prophetstor", 2024-03-16T12:08:09.380 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.380 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.380 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.381 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.381 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.381 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.381 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.381 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.381 INFO:teuthology.orchestra.run.smithi029.stdout: "sleep_interval": { 2024-03-16T12:08:09.381 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "sleep_interval", 2024-03-16T12:08:09.381 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:09.381 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.382 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.382 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "600", 2024-03-16T12:08:09.382 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.382 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.382 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.382 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.382 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.382 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.383 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.383 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:09.383 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:09.383 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.383 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-16T12:08:09.383 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "influx", 2024-03-16T12:08:09.383 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": false, 2024-03-16T12:08:09.383 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "influxdb python module not found", 2024-03-16T12:08:09.383 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-03-16T12:08:09.384 INFO:teuthology.orchestra.run.smithi029.stdout: "batch_size": { 2024-03-16T12:08:09.384 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "batch_size", 2024-03-16T12:08:09.384 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-16T12:08:09.384 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.384 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.384 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "5000", 2024-03-16T12:08:09.384 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.384 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.384 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.385 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-03-16T12:08:09.385 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.385 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.385 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.385 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.385 INFO:teuthology.orchestra.run.smithi029.stdout: "database": { 2024-03-16T12:08:09.385 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "database", 2024-03-16T12:08:09.385 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:09.385 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.386 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.386 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "ceph", 2024-03-16T12:08:09.386 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.386 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.386 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.386 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-03-16T12:08:09.386 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.386 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.387 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.387 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.387 INFO:teuthology.orchestra.run.smithi029.stdout: "hostname": { 2024-03-16T12:08:09.387 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "hostname", 2024-03-16T12:08:09.387 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:09.387 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.387 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.387 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:09.387 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.388 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.388 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.388 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "InfluxDB server hostname", 2024-03-16T12:08:09.388 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.388 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.388 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.388 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.388 INFO:teuthology.orchestra.run.smithi029.stdout: "interval": { 2024-03-16T12:08:09.389 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "interval", 2024-03-16T12:08:09.389 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "secs", 2024-03-16T12:08:09.389 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.389 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.389 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "30", 2024-03-16T12:08:09.389 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "5", 2024-03-16T12:08:09.389 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.389 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.389 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-03-16T12:08:09.390 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.390 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.390 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.390 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.390 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-03-16T12:08:09.390 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-03-16T12:08:09.390 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:09.390 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.390 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.391 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:09.391 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.391 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.391 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-16T12:08:09.391 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-16T12:08:09.391 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-16T12:08:09.391 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-16T12:08:09.391 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-16T12:08:09.391 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-16T12:08:09.392 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-16T12:08:09.392 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:09.392 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.392 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.392 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.392 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.392 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.392 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-03-16T12:08:09.392 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-03-16T12:08:09.393 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:09.393 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.393 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.393 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:09.393 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.393 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.393 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.393 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.393 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.394 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.394 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.394 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.394 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-03-16T12:08:09.394 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-03-16T12:08:09.394 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:09.394 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.394 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.394 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-03-16T12:08:09.395 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.395 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.395 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-16T12:08:09.395 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-16T12:08:09.395 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-16T12:08:09.395 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-16T12:08:09.395 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-16T12:08:09.395 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-16T12:08:09.395 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-16T12:08:09.396 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:09.396 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.396 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.396 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.396 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.396 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.396 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-03-16T12:08:09.396 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-03-16T12:08:09.396 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:09.396 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.397 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.397 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:09.397 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.397 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.397 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.397 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.397 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.397 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.397 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.398 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.398 INFO:teuthology.orchestra.run.smithi029.stdout: "password": { 2024-03-16T12:08:09.398 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "password", 2024-03-16T12:08:09.398 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:09.398 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.399 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.399 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:09.399 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.399 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.399 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.399 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "password of InfluxDB server user", 2024-03-16T12:08:09.399 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.399 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.399 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.400 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.400 INFO:teuthology.orchestra.run.smithi029.stdout: "port": { 2024-03-16T12:08:09.400 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "port", 2024-03-16T12:08:09.400 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-16T12:08:09.400 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.400 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.400 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "8086", 2024-03-16T12:08:09.401 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.401 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.401 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.401 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "InfluxDB server port", 2024-03-16T12:08:09.401 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.401 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.401 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.401 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.402 INFO:teuthology.orchestra.run.smithi029.stdout: "ssl": { 2024-03-16T12:08:09.402 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "ssl", 2024-03-16T12:08:09.402 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:09.402 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.402 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.402 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "false", 2024-03-16T12:08:09.402 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.402 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.403 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.403 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-03-16T12:08:09.403 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.403 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.403 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.403 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.403 INFO:teuthology.orchestra.run.smithi029.stdout: "threads": { 2024-03-16T12:08:09.404 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "threads", 2024-03-16T12:08:09.404 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-16T12:08:09.404 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.404 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.404 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "5", 2024-03-16T12:08:09.404 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "1", 2024-03-16T12:08:09.404 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "32", 2024-03-16T12:08:09.405 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.405 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-03-16T12:08:09.405 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.405 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.405 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.405 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.405 INFO:teuthology.orchestra.run.smithi029.stdout: "username": { 2024-03-16T12:08:09.405 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "username", 2024-03-16T12:08:09.406 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:09.406 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.406 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.406 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:09.406 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.406 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.407 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.407 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "username of InfluxDB server user", 2024-03-16T12:08:09.407 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.407 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.407 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.407 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.407 INFO:teuthology.orchestra.run.smithi029.stdout: "verify_ssl": { 2024-03-16T12:08:09.408 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "verify_ssl", 2024-03-16T12:08:09.408 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:09.408 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.408 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.408 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "true", 2024-03-16T12:08:09.408 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.408 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.409 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.409 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-03-16T12:08:09.409 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.409 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.409 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.409 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:09.409 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:09.410 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.410 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-16T12:08:09.410 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "insights", 2024-03-16T12:08:09.410 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-03-16T12:08:09.410 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-03-16T12:08:09.410 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-03-16T12:08:09.410 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-03-16T12:08:09.411 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-03-16T12:08:09.411 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:09.411 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.411 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.411 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:09.411 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.411 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.411 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-16T12:08:09.412 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-16T12:08:09.412 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-16T12:08:09.412 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-16T12:08:09.412 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-16T12:08:09.412 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-16T12:08:09.412 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-16T12:08:09.412 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:09.412 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.413 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.413 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.413 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.413 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.413 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-03-16T12:08:09.413 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-03-16T12:08:09.413 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:09.413 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.413 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.414 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:09.414 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.414 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.414 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.414 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.414 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.415 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.415 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.415 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.415 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-03-16T12:08:09.415 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-03-16T12:08:09.415 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:09.415 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.415 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.416 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-03-16T12:08:09.416 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.416 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.416 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-16T12:08:09.416 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-16T12:08:09.416 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-16T12:08:09.416 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-16T12:08:09.416 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-16T12:08:09.417 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-16T12:08:09.417 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-16T12:08:09.417 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:09.417 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.417 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.417 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.417 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.417 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.418 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-03-16T12:08:09.418 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-03-16T12:08:09.418 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:09.418 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.418 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.418 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:09.418 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.418 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.419 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.419 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.419 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.419 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.419 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.419 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:09.419 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:09.419 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.419 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-16T12:08:09.420 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "iostat", 2024-03-16T12:08:09.420 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-03-16T12:08:09.420 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-03-16T12:08:09.420 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-03-16T12:08:09.420 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-03-16T12:08:09.420 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-03-16T12:08:09.420 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:09.420 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.420 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.421 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:09.421 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.421 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.421 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-16T12:08:09.421 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-16T12:08:09.421 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-16T12:08:09.421 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-16T12:08:09.421 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-16T12:08:09.421 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-16T12:08:09.422 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-16T12:08:09.422 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:09.422 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.422 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.422 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.422 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.422 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.422 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-03-16T12:08:09.423 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-03-16T12:08:09.423 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:09.423 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.423 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.423 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:09.423 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.423 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.423 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.423 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.423 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.424 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.424 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.424 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.424 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-03-16T12:08:09.424 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-03-16T12:08:09.424 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:09.424 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.424 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.424 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-03-16T12:08:09.425 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.425 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.425 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-16T12:08:09.425 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-16T12:08:09.425 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-16T12:08:09.425 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-16T12:08:09.425 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-16T12:08:09.425 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-16T12:08:09.425 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-16T12:08:09.426 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:09.426 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.426 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.426 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.426 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.426 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.426 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-03-16T12:08:09.426 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-03-16T12:08:09.426 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:09.427 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.427 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.427 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:09.427 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.427 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.427 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.427 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.427 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.427 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.428 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.428 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:09.428 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:09.428 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.428 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-16T12:08:09.428 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "k8sevents", 2024-03-16T12:08:09.428 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-03-16T12:08:09.428 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-03-16T12:08:09.429 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-03-16T12:08:09.429 INFO:teuthology.orchestra.run.smithi029.stdout: "ceph_event_retention_days": { 2024-03-16T12:08:09.429 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "ceph_event_retention_days", 2024-03-16T12:08:09.429 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-16T12:08:09.429 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.429 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.429 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "7", 2024-03-16T12:08:09.429 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.429 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.430 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.430 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Days to hold ceph event information within local cache", 2024-03-16T12:08:09.430 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.430 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.430 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.430 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.430 INFO:teuthology.orchestra.run.smithi029.stdout: "config_check_secs": { 2024-03-16T12:08:09.430 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "config_check_secs", 2024-03-16T12:08:09.431 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-16T12:08:09.431 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.431 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.431 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "10", 2024-03-16T12:08:09.431 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "10", 2024-03-16T12:08:09.431 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.431 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.431 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-03-16T12:08:09.431 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.432 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.432 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.432 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.432 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-03-16T12:08:09.432 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-03-16T12:08:09.432 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:09.432 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.432 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.432 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:09.433 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.433 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.433 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-16T12:08:09.433 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-16T12:08:09.433 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-16T12:08:09.433 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-16T12:08:09.433 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-16T12:08:09.433 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-16T12:08:09.434 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-16T12:08:09.434 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:09.434 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.434 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.434 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.434 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.434 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.434 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-03-16T12:08:09.434 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-03-16T12:08:09.435 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:09.435 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.435 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.435 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:09.435 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.435 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.435 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.435 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.435 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.436 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.436 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.436 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.436 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-03-16T12:08:09.436 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-03-16T12:08:09.436 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:09.437 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.437 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.437 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-03-16T12:08:09.437 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.437 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.437 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-16T12:08:09.437 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-16T12:08:09.437 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-16T12:08:09.437 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-16T12:08:09.438 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-16T12:08:09.438 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-16T12:08:09.438 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-16T12:08:09.438 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:09.438 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.438 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.438 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.439 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.439 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.439 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-03-16T12:08:09.439 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-03-16T12:08:09.439 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:09.439 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.439 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.440 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:09.440 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.440 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.440 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.440 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.440 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.440 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.440 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.441 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:09.441 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:09.441 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.441 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-16T12:08:09.441 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "localpool", 2024-03-16T12:08:09.441 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-03-16T12:08:09.441 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-03-16T12:08:09.442 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-03-16T12:08:09.442 INFO:teuthology.orchestra.run.smithi029.stdout: "failure_domain": { 2024-03-16T12:08:09.442 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "failure_domain", 2024-03-16T12:08:09.442 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:09.442 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.442 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.442 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "host", 2024-03-16T12:08:09.443 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.443 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.443 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.443 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "failure domain for any created local pool", 2024-03-16T12:08:09.443 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-03-16T12:08:09.443 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.443 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.444 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.444 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-03-16T12:08:09.444 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-03-16T12:08:09.444 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:09.445 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.445 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.445 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:09.445 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.446 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.446 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-16T12:08:09.446 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-16T12:08:09.446 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-16T12:08:09.446 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-16T12:08:09.446 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-16T12:08:09.447 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-16T12:08:09.447 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-16T12:08:09.447 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:09.447 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.447 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.447 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.448 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.448 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.448 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-03-16T12:08:09.448 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-03-16T12:08:09.448 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:09.448 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.449 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.449 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:09.449 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.449 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.449 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.449 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.449 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.449 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.449 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.450 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.450 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-03-16T12:08:09.450 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-03-16T12:08:09.450 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:09.450 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.450 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.450 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-03-16T12:08:09.451 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.451 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.451 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-16T12:08:09.451 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-16T12:08:09.451 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-16T12:08:09.451 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-16T12:08:09.451 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-16T12:08:09.451 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-16T12:08:09.451 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-16T12:08:09.452 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:09.452 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.452 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.452 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.452 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.452 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.452 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-03-16T12:08:09.452 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-03-16T12:08:09.452 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:09.453 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.453 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.453 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:09.453 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.453 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.453 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.453 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.453 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.453 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.453 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.454 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.454 INFO:teuthology.orchestra.run.smithi029.stdout: "min_size": { 2024-03-16T12:08:09.454 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "min_size", 2024-03-16T12:08:09.454 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-16T12:08:09.454 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.454 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.454 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:09.454 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.455 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.455 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.455 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "default min_size for any created local pool", 2024-03-16T12:08:09.455 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-03-16T12:08:09.455 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.455 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.455 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.455 INFO:teuthology.orchestra.run.smithi029.stdout: "num_rep": { 2024-03-16T12:08:09.455 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "num_rep", 2024-03-16T12:08:09.456 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-16T12:08:09.456 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.456 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.456 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "3", 2024-03-16T12:08:09.456 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.456 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.456 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.456 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "default replica count for any created local pool", 2024-03-16T12:08:09.456 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.456 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.457 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.457 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.457 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_num": { 2024-03-16T12:08:09.457 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "pg_num", 2024-03-16T12:08:09.457 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-16T12:08:09.457 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.457 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.457 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "128", 2024-03-16T12:08:09.458 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.458 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.458 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.458 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "default pg_num for any created local pool", 2024-03-16T12:08:09.458 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.458 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.459 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.459 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.459 INFO:teuthology.orchestra.run.smithi029.stdout: "prefix": { 2024-03-16T12:08:09.459 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "prefix", 2024-03-16T12:08:09.459 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:09.459 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.459 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.459 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:09.459 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.460 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.460 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.460 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "name prefix for any created local pool", 2024-03-16T12:08:09.460 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.460 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.460 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.460 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.460 INFO:teuthology.orchestra.run.smithi029.stdout: "subtree": { 2024-03-16T12:08:09.461 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "subtree", 2024-03-16T12:08:09.461 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:09.461 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.461 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.461 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "rack", 2024-03-16T12:08:09.461 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.461 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.461 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.461 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "CRUSH level for which to create a local pool", 2024-03-16T12:08:09.462 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-03-16T12:08:09.462 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.462 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.462 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:09.462 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:09.462 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.462 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-16T12:08:09.462 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "mds_autoscaler", 2024-03-16T12:08:09.463 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-03-16T12:08:09.463 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-03-16T12:08:09.463 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-03-16T12:08:09.463 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-03-16T12:08:09.463 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-03-16T12:08:09.463 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:09.463 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.463 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.464 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:09.464 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.464 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.464 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-16T12:08:09.464 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-16T12:08:09.464 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-16T12:08:09.464 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-16T12:08:09.464 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-16T12:08:09.464 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-16T12:08:09.465 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-16T12:08:09.465 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:09.465 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.465 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.465 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.465 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.465 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.465 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-03-16T12:08:09.465 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-03-16T12:08:09.466 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:09.466 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.466 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.466 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:09.466 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.466 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.466 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.466 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.466 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.467 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.467 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.467 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.467 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-03-16T12:08:09.467 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-03-16T12:08:09.467 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:09.467 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.467 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.468 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-03-16T12:08:09.468 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.468 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.468 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-16T12:08:09.468 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-16T12:08:09.468 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-16T12:08:09.468 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-16T12:08:09.468 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-16T12:08:09.468 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-16T12:08:09.469 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-16T12:08:09.469 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:09.469 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.469 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.469 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.469 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.469 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.469 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-03-16T12:08:09.469 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-03-16T12:08:09.470 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:09.470 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.470 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.470 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:09.470 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.471 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.471 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.471 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.471 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.471 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.471 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.471 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:09.471 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:09.472 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.472 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-16T12:08:09.472 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "mirroring", 2024-03-16T12:08:09.472 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-03-16T12:08:09.472 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-03-16T12:08:09.472 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-03-16T12:08:09.473 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-03-16T12:08:09.473 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-03-16T12:08:09.473 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:09.473 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.473 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.473 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:09.473 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.473 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.474 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-16T12:08:09.474 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-16T12:08:09.474 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-16T12:08:09.474 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-16T12:08:09.474 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-16T12:08:09.474 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-16T12:08:09.474 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-16T12:08:09.475 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:09.475 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.475 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.475 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.475 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.475 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.475 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-03-16T12:08:09.476 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-03-16T12:08:09.476 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:09.476 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.476 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.476 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:09.476 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.476 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.477 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.477 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.477 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.477 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.477 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.477 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.477 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-03-16T12:08:09.477 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-03-16T12:08:09.478 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:09.478 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.478 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.478 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-03-16T12:08:09.478 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.478 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.479 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-16T12:08:09.479 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-16T12:08:09.479 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-16T12:08:09.479 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-16T12:08:09.479 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-16T12:08:09.479 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-16T12:08:09.479 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-16T12:08:09.480 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:09.480 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.480 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.480 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.480 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.480 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.481 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-03-16T12:08:09.481 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-03-16T12:08:09.481 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:09.481 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.481 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.481 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:09.481 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.482 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.482 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.482 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.482 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.482 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.482 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.483 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:09.483 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:09.483 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.483 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-16T12:08:09.483 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "nfs", 2024-03-16T12:08:09.483 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-03-16T12:08:09.483 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-03-16T12:08:09.483 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-03-16T12:08:09.484 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-03-16T12:08:09.484 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-03-16T12:08:09.484 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:09.484 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.484 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.484 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:09.484 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.484 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.485 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-16T12:08:09.485 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-16T12:08:09.485 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-16T12:08:09.485 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-16T12:08:09.485 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-16T12:08:09.485 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-16T12:08:09.485 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-16T12:08:09.485 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:09.485 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.486 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.486 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.486 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.486 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.486 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-03-16T12:08:09.486 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-03-16T12:08:09.486 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:09.486 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.487 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.487 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:09.487 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.487 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.487 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.487 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.487 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.487 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.488 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.488 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.488 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-03-16T12:08:09.488 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-03-16T12:08:09.488 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:09.488 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.488 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.488 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-03-16T12:08:09.488 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.489 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.489 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-16T12:08:09.489 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-16T12:08:09.489 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-16T12:08:09.489 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-16T12:08:09.489 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-16T12:08:09.489 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-16T12:08:09.489 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-16T12:08:09.489 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:09.490 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.490 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.490 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.490 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.490 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.490 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-03-16T12:08:09.490 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-03-16T12:08:09.490 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:09.491 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.491 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.491 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:09.491 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.491 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.491 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.491 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.491 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.491 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.492 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.492 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:09.492 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:09.492 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.492 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-16T12:08:09.492 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "orchestrator", 2024-03-16T12:08:09.492 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-03-16T12:08:09.492 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-03-16T12:08:09.492 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-03-16T12:08:09.493 INFO:teuthology.orchestra.run.smithi029.stdout: "fail_fs": { 2024-03-16T12:08:09.493 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "fail_fs", 2024-03-16T12:08:09.493 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:09.493 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.493 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.493 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:09.493 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.493 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.494 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.494 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-03-16T12:08:09.494 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.494 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.494 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.494 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.494 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-03-16T12:08:09.494 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-03-16T12:08:09.494 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:09.495 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.495 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.495 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:09.495 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.495 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.495 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-16T12:08:09.495 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-16T12:08:09.495 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-16T12:08:09.496 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-16T12:08:09.496 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-16T12:08:09.496 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-16T12:08:09.496 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-16T12:08:09.496 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:09.496 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.496 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.496 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.497 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.497 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.497 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-03-16T12:08:09.497 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-03-16T12:08:09.497 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:09.497 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.497 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.497 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:09.498 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.498 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.498 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.498 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.498 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.498 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.498 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.498 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.498 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-03-16T12:08:09.499 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-03-16T12:08:09.499 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:09.499 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.499 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.499 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-03-16T12:08:09.499 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.499 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.499 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-16T12:08:09.499 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-16T12:08:09.500 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-16T12:08:09.500 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-16T12:08:09.500 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-16T12:08:09.500 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-16T12:08:09.500 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-16T12:08:09.500 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:09.500 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.500 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.500 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.501 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.501 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.501 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-03-16T12:08:09.501 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-03-16T12:08:09.501 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:09.501 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.501 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.501 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:09.501 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.502 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.502 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.502 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.502 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.502 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.502 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.502 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.502 INFO:teuthology.orchestra.run.smithi029.stdout: "orchestrator": { 2024-03-16T12:08:09.503 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "orchestrator", 2024-03-16T12:08:09.503 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:09.503 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.503 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.503 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:09.503 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.503 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.503 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-16T12:08:09.503 INFO:teuthology.orchestra.run.smithi029.stdout: "cephadm", 2024-03-16T12:08:09.504 INFO:teuthology.orchestra.run.smithi029.stdout: "rook", 2024-03-16T12:08:09.504 INFO:teuthology.orchestra.run.smithi029.stdout: "test_orchestrator" 2024-03-16T12:08:09.504 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:09.504 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Orchestrator backend", 2024-03-16T12:08:09.504 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.504 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.504 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.504 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:09.505 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:09.505 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.505 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-16T12:08:09.505 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "osd_perf_query", 2024-03-16T12:08:09.505 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-03-16T12:08:09.505 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-03-16T12:08:09.505 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-03-16T12:08:09.505 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-03-16T12:08:09.505 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-03-16T12:08:09.506 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:09.506 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.506 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.506 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:09.506 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.506 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.506 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-16T12:08:09.506 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-16T12:08:09.506 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-16T12:08:09.507 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-16T12:08:09.507 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-16T12:08:09.507 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-16T12:08:09.507 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-16T12:08:09.507 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:09.507 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.507 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.507 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.508 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.508 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.508 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-03-16T12:08:09.508 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-03-16T12:08:09.508 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:09.508 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.508 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.509 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:09.509 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.509 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.509 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.509 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.509 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.509 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.510 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.510 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.510 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-03-16T12:08:09.510 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-03-16T12:08:09.510 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:09.510 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.510 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.510 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-03-16T12:08:09.510 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.511 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.511 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-16T12:08:09.511 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-16T12:08:09.511 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-16T12:08:09.511 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-16T12:08:09.511 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-16T12:08:09.511 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-16T12:08:09.512 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-16T12:08:09.512 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:09.512 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.512 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.512 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.512 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.512 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.512 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-03-16T12:08:09.513 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-03-16T12:08:09.513 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:09.513 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.513 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.513 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:09.513 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.513 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.514 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.514 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.514 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.514 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.514 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.514 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:09.514 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:09.514 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.515 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-16T12:08:09.515 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "osd_support", 2024-03-16T12:08:09.515 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-03-16T12:08:09.515 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-03-16T12:08:09.515 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-03-16T12:08:09.515 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-03-16T12:08:09.515 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-03-16T12:08:09.516 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:09.516 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.516 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.516 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:09.516 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.516 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.516 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-16T12:08:09.516 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-16T12:08:09.517 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-16T12:08:09.517 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-16T12:08:09.517 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-16T12:08:09.517 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-16T12:08:09.517 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-16T12:08:09.517 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:09.518 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.518 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.518 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.518 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.518 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.518 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-03-16T12:08:09.519 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-03-16T12:08:09.519 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:09.519 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.519 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.519 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:09.519 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.519 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.519 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.520 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.520 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.520 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.520 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.520 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.520 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-03-16T12:08:09.521 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-03-16T12:08:09.521 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:09.521 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.521 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.521 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-03-16T12:08:09.521 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.521 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.522 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-16T12:08:09.522 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-16T12:08:09.522 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-16T12:08:09.522 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-16T12:08:09.522 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-16T12:08:09.522 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-16T12:08:09.522 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-16T12:08:09.523 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:09.523 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.523 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.523 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.523 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.523 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.523 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-03-16T12:08:09.523 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-03-16T12:08:09.523 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:09.524 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.524 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.524 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:09.524 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.524 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.524 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.524 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.524 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.524 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.525 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.525 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:09.525 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:09.525 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.525 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-16T12:08:09.525 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "pg_autoscaler", 2024-03-16T12:08:09.525 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-03-16T12:08:09.525 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-03-16T12:08:09.525 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-03-16T12:08:09.526 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-03-16T12:08:09.526 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-03-16T12:08:09.526 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:09.526 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.526 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.526 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:09.526 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.526 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.526 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-16T12:08:09.527 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-16T12:08:09.527 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-16T12:08:09.527 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-16T12:08:09.527 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-16T12:08:09.527 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-16T12:08:09.527 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-16T12:08:09.527 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:09.527 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.527 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.528 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.528 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.528 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.528 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-03-16T12:08:09.528 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-03-16T12:08:09.528 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:09.528 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.528 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.528 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:09.529 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.529 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.529 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.529 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.529 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.529 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.529 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.529 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.529 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-03-16T12:08:09.530 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-03-16T12:08:09.530 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:09.530 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.530 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.530 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-03-16T12:08:09.530 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.530 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.530 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-16T12:08:09.530 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-16T12:08:09.531 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-16T12:08:09.531 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-16T12:08:09.531 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-16T12:08:09.531 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-16T12:08:09.531 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-16T12:08:09.531 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:09.531 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.531 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.531 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.532 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.532 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.532 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-03-16T12:08:09.532 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-03-16T12:08:09.532 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:09.532 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.532 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.532 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:09.533 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.533 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.533 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.533 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.533 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.533 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.533 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.533 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.533 INFO:teuthology.orchestra.run.smithi029.stdout: "sleep_interval": { 2024-03-16T12:08:09.534 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "sleep_interval", 2024-03-16T12:08:09.534 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "secs", 2024-03-16T12:08:09.534 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.534 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.534 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "60", 2024-03-16T12:08:09.534 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.534 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.534 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.534 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.535 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.535 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.535 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.535 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.535 INFO:teuthology.orchestra.run.smithi029.stdout: "threshold": { 2024-03-16T12:08:09.535 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "threshold", 2024-03-16T12:08:09.535 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "float", 2024-03-16T12:08:09.535 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.535 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.536 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "3.0", 2024-03-16T12:08:09.536 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "1.0", 2024-03-16T12:08:09.536 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.536 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.536 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "scaling threshold", 2024-03-16T12:08:09.536 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-03-16T12:08:09.536 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.536 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.536 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:09.537 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:09.537 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.537 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-16T12:08:09.537 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "progress", 2024-03-16T12:08:09.537 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-03-16T12:08:09.537 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-03-16T12:08:09.537 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-03-16T12:08:09.537 INFO:teuthology.orchestra.run.smithi029.stdout: "allow_pg_recovery_event": { 2024-03-16T12:08:09.537 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "allow_pg_recovery_event", 2024-03-16T12:08:09.538 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:09.538 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.538 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.538 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:09.538 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.538 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.538 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.538 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "allow the module to show pg recovery progress", 2024-03-16T12:08:09.538 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.539 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.539 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.539 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.539 INFO:teuthology.orchestra.run.smithi029.stdout: "enabled": { 2024-03-16T12:08:09.539 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "enabled", 2024-03-16T12:08:09.539 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:09.539 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.539 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.539 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-16T12:08:09.540 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.540 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.540 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.540 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.540 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.540 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.540 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.540 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.540 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-03-16T12:08:09.541 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-03-16T12:08:09.541 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:09.541 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.541 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.541 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:09.541 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.541 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.541 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-16T12:08:09.541 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-16T12:08:09.542 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-16T12:08:09.542 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-16T12:08:09.542 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-16T12:08:09.542 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-16T12:08:09.542 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-16T12:08:09.542 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:09.543 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.543 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.543 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.543 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.543 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.543 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-03-16T12:08:09.543 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-03-16T12:08:09.543 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:09.544 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.544 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.544 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:09.544 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.544 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.544 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.544 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.544 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.544 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.545 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.545 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.545 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-03-16T12:08:09.545 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-03-16T12:08:09.545 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:09.545 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.545 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.545 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-03-16T12:08:09.546 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.546 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.546 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-16T12:08:09.546 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-16T12:08:09.546 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-16T12:08:09.546 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-16T12:08:09.546 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-16T12:08:09.546 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-16T12:08:09.546 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-16T12:08:09.547 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:09.547 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.547 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.547 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.547 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.547 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.547 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-03-16T12:08:09.547 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-03-16T12:08:09.547 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:09.548 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.548 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.548 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:09.548 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.548 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.548 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.548 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.548 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.548 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.549 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.549 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.549 INFO:teuthology.orchestra.run.smithi029.stdout: "max_completed_events": { 2024-03-16T12:08:09.549 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "max_completed_events", 2024-03-16T12:08:09.549 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-16T12:08:09.549 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.549 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.549 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "50", 2024-03-16T12:08:09.549 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.550 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.550 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.550 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "number of past completed events to remember", 2024-03-16T12:08:09.550 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.550 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.550 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.550 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.550 INFO:teuthology.orchestra.run.smithi029.stdout: "sleep_interval": { 2024-03-16T12:08:09.550 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "sleep_interval", 2024-03-16T12:08:09.551 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "secs", 2024-03-16T12:08:09.551 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.551 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.551 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "5", 2024-03-16T12:08:09.551 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.551 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.551 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.551 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "how long the module is going to sleep", 2024-03-16T12:08:09.551 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.552 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.552 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.552 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:09.552 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:09.552 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.552 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-16T12:08:09.552 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "prometheus", 2024-03-16T12:08:09.552 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-03-16T12:08:09.553 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-03-16T12:08:09.553 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-03-16T12:08:09.553 INFO:teuthology.orchestra.run.smithi029.stdout: "cache": { 2024-03-16T12:08:09.553 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "cache", 2024-03-16T12:08:09.553 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:09.553 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.553 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.553 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-16T12:08:09.553 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.553 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.554 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.554 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.554 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.554 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.554 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.554 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.554 INFO:teuthology.orchestra.run.smithi029.stdout: "exclude_perf_counters": { 2024-03-16T12:08:09.555 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "exclude_perf_counters", 2024-03-16T12:08:09.555 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:09.555 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.555 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.555 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-16T12:08:09.555 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.555 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.555 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.555 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-03-16T12:08:09.556 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-03-16T12:08:09.556 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.556 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.556 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.556 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-03-16T12:08:09.556 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-03-16T12:08:09.556 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:09.556 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.556 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.557 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:09.557 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.557 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.557 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-16T12:08:09.557 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-16T12:08:09.557 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-16T12:08:09.557 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-16T12:08:09.557 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-16T12:08:09.557 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-16T12:08:09.558 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-16T12:08:09.558 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:09.558 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.558 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.558 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.558 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.558 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.558 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-03-16T12:08:09.559 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-03-16T12:08:09.559 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:09.559 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.559 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.559 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:09.559 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.559 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.559 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.559 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.560 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.560 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.560 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.560 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.560 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-03-16T12:08:09.560 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-03-16T12:08:09.560 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:09.560 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.560 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.561 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-03-16T12:08:09.561 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.561 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.561 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-16T12:08:09.561 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-16T12:08:09.561 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-16T12:08:09.561 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-16T12:08:09.561 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-16T12:08:09.562 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-16T12:08:09.562 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-16T12:08:09.562 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:09.562 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.562 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.562 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.562 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.562 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.562 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-03-16T12:08:09.563 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-03-16T12:08:09.563 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:09.563 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.563 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.563 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:09.563 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.563 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.563 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.563 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.563 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.564 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.564 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.564 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.564 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_stats_pools": { 2024-03-16T12:08:09.564 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "rbd_stats_pools", 2024-03-16T12:08:09.564 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:09.564 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.564 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.564 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:09.565 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.565 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.565 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.565 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.565 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.565 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.565 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.565 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.565 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_stats_pools_refresh_interval": { 2024-03-16T12:08:09.566 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-03-16T12:08:09.566 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-16T12:08:09.566 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.566 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.566 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "300", 2024-03-16T12:08:09.566 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.566 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.566 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.567 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.567 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.567 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.567 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.567 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.567 INFO:teuthology.orchestra.run.smithi029.stdout: "scrape_interval": { 2024-03-16T12:08:09.567 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "scrape_interval", 2024-03-16T12:08:09.567 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "float", 2024-03-16T12:08:09.568 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.568 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.568 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "15.0", 2024-03-16T12:08:09.568 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.568 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.568 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.568 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.568 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.568 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.569 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.569 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.569 INFO:teuthology.orchestra.run.smithi029.stdout: "server_addr": { 2024-03-16T12:08:09.569 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "server_addr", 2024-03-16T12:08:09.569 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:09.569 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.569 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.570 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "::", 2024-03-16T12:08:09.570 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.570 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.570 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.570 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-03-16T12:08:09.570 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.570 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.570 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.571 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.571 INFO:teuthology.orchestra.run.smithi029.stdout: "server_port": { 2024-03-16T12:08:09.571 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "server_port", 2024-03-16T12:08:09.571 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-16T12:08:09.571 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.571 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.571 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "9283", 2024-03-16T12:08:09.572 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.572 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.572 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.572 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-03-16T12:08:09.572 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.572 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.572 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.572 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.572 INFO:teuthology.orchestra.run.smithi029.stdout: "stale_cache_strategy": { 2024-03-16T12:08:09.573 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "stale_cache_strategy", 2024-03-16T12:08:09.573 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:09.573 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.573 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.573 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "log", 2024-03-16T12:08:09.573 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.573 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.573 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.573 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.574 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.574 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.574 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.574 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.574 INFO:teuthology.orchestra.run.smithi029.stdout: "standby_behaviour": { 2024-03-16T12:08:09.574 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "standby_behaviour", 2024-03-16T12:08:09.574 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:09.574 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.574 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.575 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "default", 2024-03-16T12:08:09.575 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.575 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.575 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-16T12:08:09.575 INFO:teuthology.orchestra.run.smithi029.stdout: "default", 2024-03-16T12:08:09.575 INFO:teuthology.orchestra.run.smithi029.stdout: "error" 2024-03-16T12:08:09.575 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:09.575 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.576 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.576 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.576 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.576 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.576 INFO:teuthology.orchestra.run.smithi029.stdout: "standby_error_status_code": { 2024-03-16T12:08:09.576 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "standby_error_status_code", 2024-03-16T12:08:09.576 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-16T12:08:09.576 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.576 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.576 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "500", 2024-03-16T12:08:09.577 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "400", 2024-03-16T12:08:09.577 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "599", 2024-03-16T12:08:09.577 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.577 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.577 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.577 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.577 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.577 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:09.577 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:09.578 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.578 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-16T12:08:09.578 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "rbd_support", 2024-03-16T12:08:09.578 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-03-16T12:08:09.578 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-03-16T12:08:09.578 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-03-16T12:08:09.578 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-03-16T12:08:09.579 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-03-16T12:08:09.579 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:09.579 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.579 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.579 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:09.579 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.580 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.580 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-16T12:08:09.580 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-16T12:08:09.580 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-16T12:08:09.580 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-16T12:08:09.580 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-16T12:08:09.581 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-16T12:08:09.581 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-16T12:08:09.581 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:09.581 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.581 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.581 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.582 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.582 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.582 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-03-16T12:08:09.582 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-03-16T12:08:09.582 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:09.583 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.583 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.583 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:09.583 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.583 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.583 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.584 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.584 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.584 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.584 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.584 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.584 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-03-16T12:08:09.585 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-03-16T12:08:09.585 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:09.585 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.585 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.585 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-03-16T12:08:09.586 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.586 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.586 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-16T12:08:09.586 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-16T12:08:09.586 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-16T12:08:09.587 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-16T12:08:09.587 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-16T12:08:09.587 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-16T12:08:09.587 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-16T12:08:09.587 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:09.588 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.588 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.588 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.588 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.588 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.588 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-03-16T12:08:09.589 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-03-16T12:08:09.589 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:09.589 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.589 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.589 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:09.589 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.589 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.589 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.589 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.590 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.590 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.590 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.590 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.590 INFO:teuthology.orchestra.run.smithi029.stdout: "max_concurrent_snap_create": { 2024-03-16T12:08:09.590 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "max_concurrent_snap_create", 2024-03-16T12:08:09.590 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-16T12:08:09.590 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.590 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.591 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "10", 2024-03-16T12:08:09.591 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.591 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.591 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.591 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.591 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.591 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.591 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.591 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.592 INFO:teuthology.orchestra.run.smithi029.stdout: "mirror_snapshot_schedule": { 2024-03-16T12:08:09.592 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "mirror_snapshot_schedule", 2024-03-16T12:08:09.592 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:09.592 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.592 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.592 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:09.592 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.592 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.593 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.593 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.593 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.593 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.593 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.593 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.593 INFO:teuthology.orchestra.run.smithi029.stdout: "trash_purge_schedule": { 2024-03-16T12:08:09.593 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "trash_purge_schedule", 2024-03-16T12:08:09.593 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:09.594 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.594 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.594 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:09.594 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.594 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.594 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.594 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.594 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.594 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.595 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.595 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:09.595 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:09.595 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.595 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-16T12:08:09.595 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "restful", 2024-03-16T12:08:09.595 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-03-16T12:08:09.595 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-03-16T12:08:09.595 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-03-16T12:08:09.596 INFO:teuthology.orchestra.run.smithi029.stdout: "enable_auth": { 2024-03-16T12:08:09.596 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "enable_auth", 2024-03-16T12:08:09.596 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:09.596 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.596 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.596 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-16T12:08:09.596 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.596 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.596 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.596 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.597 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.597 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.597 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.597 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.597 INFO:teuthology.orchestra.run.smithi029.stdout: "key_file": { 2024-03-16T12:08:09.597 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "key_file", 2024-03-16T12:08:09.597 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:09.597 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.597 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.597 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:09.598 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.598 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.598 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.598 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.598 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.598 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.598 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.598 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.599 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-03-16T12:08:09.599 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-03-16T12:08:09.599 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:09.599 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.599 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.599 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:09.599 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.599 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.599 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-16T12:08:09.600 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-16T12:08:09.600 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-16T12:08:09.600 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-16T12:08:09.600 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-16T12:08:09.600 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-16T12:08:09.600 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-16T12:08:09.600 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:09.600 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.600 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.600 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.601 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.601 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.601 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-03-16T12:08:09.601 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-03-16T12:08:09.601 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:09.601 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.601 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.601 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:09.601 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.602 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.602 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.602 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.602 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.602 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.602 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.602 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.602 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-03-16T12:08:09.602 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-03-16T12:08:09.602 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:09.603 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.603 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.603 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-03-16T12:08:09.603 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.603 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.603 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-16T12:08:09.603 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-16T12:08:09.603 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-16T12:08:09.603 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-16T12:08:09.604 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-16T12:08:09.604 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-16T12:08:09.604 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-16T12:08:09.604 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:09.604 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.604 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.604 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.604 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.604 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.604 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-03-16T12:08:09.605 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-03-16T12:08:09.605 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:09.605 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.605 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.605 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:09.605 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.605 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.605 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.605 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.605 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.606 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.606 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.606 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.606 INFO:teuthology.orchestra.run.smithi029.stdout: "server_addr": { 2024-03-16T12:08:09.606 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "server_addr", 2024-03-16T12:08:09.606 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:09.606 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.606 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.606 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:09.607 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.607 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.607 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.607 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.607 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.608 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.608 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.608 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.608 INFO:teuthology.orchestra.run.smithi029.stdout: "server_port": { 2024-03-16T12:08:09.608 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "server_port", 2024-03-16T12:08:09.608 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:09.608 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.608 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.609 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:09.609 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.609 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.609 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.609 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.609 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.609 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.610 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.610 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:09.610 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:09.610 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.610 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-16T12:08:09.610 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "rgw", 2024-03-16T12:08:09.610 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-03-16T12:08:09.611 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-03-16T12:08:09.611 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-03-16T12:08:09.611 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-03-16T12:08:09.611 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-03-16T12:08:09.611 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:09.611 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.612 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.612 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:09.612 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.612 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.612 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-16T12:08:09.612 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-16T12:08:09.612 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-16T12:08:09.613 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-16T12:08:09.613 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-16T12:08:09.613 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-16T12:08:09.613 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-16T12:08:09.613 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:09.613 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.613 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.614 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.614 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.614 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.614 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-03-16T12:08:09.614 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-03-16T12:08:09.614 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:09.615 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.615 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.615 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:09.615 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.615 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.615 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.615 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.616 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.616 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.616 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.616 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.616 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-03-16T12:08:09.616 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-03-16T12:08:09.616 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:09.617 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.617 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.617 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-03-16T12:08:09.617 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.617 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.617 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-16T12:08:09.618 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-16T12:08:09.618 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-16T12:08:09.618 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-16T12:08:09.618 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-16T12:08:09.618 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-16T12:08:09.619 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-16T12:08:09.619 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:09.619 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.619 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.619 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.620 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.620 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.620 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-03-16T12:08:09.620 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-03-16T12:08:09.620 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:09.620 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.620 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.620 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:09.621 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.621 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.621 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.621 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.621 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.621 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.621 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.621 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:09.621 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:09.622 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.622 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-16T12:08:09.622 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "rook", 2024-03-16T12:08:09.622 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-03-16T12:08:09.622 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-03-16T12:08:09.622 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-03-16T12:08:09.622 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-03-16T12:08:09.622 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-03-16T12:08:09.623 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:09.623 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.623 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.623 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:09.623 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.623 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.623 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-16T12:08:09.623 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-16T12:08:09.623 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-16T12:08:09.624 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-16T12:08:09.624 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-16T12:08:09.624 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-16T12:08:09.624 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-16T12:08:09.624 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:09.624 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.624 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.624 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.625 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.625 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.625 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-03-16T12:08:09.625 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-03-16T12:08:09.625 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:09.625 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.625 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.625 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:09.625 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.626 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.626 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.626 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.626 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.626 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.626 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.626 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.626 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-03-16T12:08:09.627 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-03-16T12:08:09.627 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:09.627 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.627 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.627 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-03-16T12:08:09.627 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.627 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.627 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-16T12:08:09.628 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-16T12:08:09.628 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-16T12:08:09.628 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-16T12:08:09.628 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-16T12:08:09.628 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-16T12:08:09.628 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-16T12:08:09.628 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:09.628 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.629 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.629 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.629 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.629 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.629 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-03-16T12:08:09.629 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-03-16T12:08:09.629 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:09.629 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.629 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.630 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:09.630 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.630 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.630 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.630 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.630 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.630 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.630 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.631 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.631 INFO:teuthology.orchestra.run.smithi029.stdout: "storage_class": { 2024-03-16T12:08:09.631 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "storage_class", 2024-03-16T12:08:09.631 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:09.631 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.631 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.631 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "local", 2024-03-16T12:08:09.631 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.631 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.632 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.632 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-03-16T12:08:09.632 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.632 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.632 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.632 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:09.632 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:09.632 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.632 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-16T12:08:09.632 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "selftest", 2024-03-16T12:08:09.633 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-03-16T12:08:09.633 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-03-16T12:08:09.633 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-03-16T12:08:09.633 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-03-16T12:08:09.633 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-03-16T12:08:09.633 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:09.633 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.633 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.633 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:09.634 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.634 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.634 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-16T12:08:09.634 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-16T12:08:09.634 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-16T12:08:09.634 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-16T12:08:09.634 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-16T12:08:09.634 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-16T12:08:09.635 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-16T12:08:09.635 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:09.635 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.635 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.635 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.635 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.635 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.635 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-03-16T12:08:09.635 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-03-16T12:08:09.635 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:09.636 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.636 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.636 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:09.636 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.636 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.636 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.636 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.636 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.636 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.637 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.637 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.637 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-03-16T12:08:09.637 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-03-16T12:08:09.637 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:09.637 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.637 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.637 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-03-16T12:08:09.637 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.638 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.638 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-16T12:08:09.638 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-16T12:08:09.638 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-16T12:08:09.638 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-16T12:08:09.638 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-16T12:08:09.638 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-16T12:08:09.638 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-16T12:08:09.639 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:09.639 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.639 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.639 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.639 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.639 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.639 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-03-16T12:08:09.639 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-03-16T12:08:09.639 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:09.639 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.640 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.640 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:09.640 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.640 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.640 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.640 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.640 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.640 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.640 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.641 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.641 INFO:teuthology.orchestra.run.smithi029.stdout: "roption1": { 2024-03-16T12:08:09.641 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "roption1", 2024-03-16T12:08:09.641 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:09.642 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.642 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.642 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:09.642 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.642 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.642 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.642 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.642 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.643 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.643 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.643 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.643 INFO:teuthology.orchestra.run.smithi029.stdout: "roption2": { 2024-03-16T12:08:09.643 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "roption2", 2024-03-16T12:08:09.644 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:09.644 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.644 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.644 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "xyz", 2024-03-16T12:08:09.644 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.644 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.644 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.644 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.645 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.645 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.645 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.645 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.645 INFO:teuthology.orchestra.run.smithi029.stdout: "rwoption1": { 2024-03-16T12:08:09.646 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "rwoption1", 2024-03-16T12:08:09.646 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:09.646 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.646 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.646 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:09.646 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.646 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.647 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.647 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.647 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.647 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.647 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.648 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.648 INFO:teuthology.orchestra.run.smithi029.stdout: "rwoption2": { 2024-03-16T12:08:09.648 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "rwoption2", 2024-03-16T12:08:09.648 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-16T12:08:09.648 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.648 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.649 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:09.649 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.649 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.649 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.649 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.650 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.650 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.650 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.650 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.650 INFO:teuthology.orchestra.run.smithi029.stdout: "rwoption3": { 2024-03-16T12:08:09.651 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "rwoption3", 2024-03-16T12:08:09.651 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "float", 2024-03-16T12:08:09.651 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.651 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.652 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:09.652 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.652 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.652 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.653 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.653 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.653 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.653 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.654 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.654 INFO:teuthology.orchestra.run.smithi029.stdout: "rwoption4": { 2024-03-16T12:08:09.654 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "rwoption4", 2024-03-16T12:08:09.654 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:09.654 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.654 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.654 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:09.655 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.655 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.655 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.655 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.655 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.655 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.655 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.655 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.656 INFO:teuthology.orchestra.run.smithi029.stdout: "rwoption5": { 2024-03-16T12:08:09.656 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "rwoption5", 2024-03-16T12:08:09.656 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:09.656 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.656 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.656 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:09.656 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.656 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.657 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.657 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.657 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.657 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.657 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.657 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.657 INFO:teuthology.orchestra.run.smithi029.stdout: "rwoption6": { 2024-03-16T12:08:09.657 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "rwoption6", 2024-03-16T12:08:09.658 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:09.658 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.658 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.658 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-16T12:08:09.658 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.658 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.658 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.658 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.658 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.659 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.659 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.659 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.659 INFO:teuthology.orchestra.run.smithi029.stdout: "rwoption7": { 2024-03-16T12:08:09.659 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "rwoption7", 2024-03-16T12:08:09.659 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-16T12:08:09.659 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.659 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.659 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:09.659 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "1", 2024-03-16T12:08:09.660 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "42", 2024-03-16T12:08:09.660 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.660 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.660 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.660 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.660 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.660 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.660 INFO:teuthology.orchestra.run.smithi029.stdout: "testkey": { 2024-03-16T12:08:09.660 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "testkey", 2024-03-16T12:08:09.661 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:09.661 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.661 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.661 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:09.661 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.661 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.661 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.661 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.661 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.662 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.662 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.662 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.662 INFO:teuthology.orchestra.run.smithi029.stdout: "testlkey": { 2024-03-16T12:08:09.662 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "testlkey", 2024-03-16T12:08:09.662 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:09.662 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.662 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.662 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:09.663 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.663 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.663 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.663 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.663 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.663 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.663 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.663 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.664 INFO:teuthology.orchestra.run.smithi029.stdout: "testnewline": { 2024-03-16T12:08:09.664 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "testnewline", 2024-03-16T12:08:09.664 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:09.664 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.664 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.664 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:09.664 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.664 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.665 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.665 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.665 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.665 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.665 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.665 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:09.665 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:09.665 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.665 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-16T12:08:09.666 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "snap_schedule", 2024-03-16T12:08:09.666 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-03-16T12:08:09.666 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-03-16T12:08:09.666 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-03-16T12:08:09.666 INFO:teuthology.orchestra.run.smithi029.stdout: "allow_m_granularity": { 2024-03-16T12:08:09.666 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "allow_m_granularity", 2024-03-16T12:08:09.666 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:09.666 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.667 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.667 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:09.667 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.667 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.667 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.667 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "allow minute scheduled snapshots", 2024-03-16T12:08:09.667 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.667 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.667 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.668 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.668 INFO:teuthology.orchestra.run.smithi029.stdout: "dump_on_update": { 2024-03-16T12:08:09.668 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "dump_on_update", 2024-03-16T12:08:09.668 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:09.668 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.668 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.668 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:09.668 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.669 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.669 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.669 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "dump database to debug log on update", 2024-03-16T12:08:09.669 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.669 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.669 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.669 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.669 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-03-16T12:08:09.670 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-03-16T12:08:09.670 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:09.670 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.670 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.670 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:09.670 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.670 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.670 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-16T12:08:09.671 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-16T12:08:09.671 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-16T12:08:09.671 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-16T12:08:09.671 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-16T12:08:09.671 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-16T12:08:09.671 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-16T12:08:09.671 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:09.671 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.671 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.672 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.672 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.672 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.672 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-03-16T12:08:09.672 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-03-16T12:08:09.672 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:09.672 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.672 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.672 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:09.673 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.673 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.673 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.673 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.673 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.673 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.673 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.673 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.673 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-03-16T12:08:09.674 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-03-16T12:08:09.674 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:09.674 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.674 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.675 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-03-16T12:08:09.675 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.675 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.675 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-16T12:08:09.675 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-16T12:08:09.675 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-16T12:08:09.675 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-16T12:08:09.675 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-16T12:08:09.676 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-16T12:08:09.676 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-16T12:08:09.676 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:09.676 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.676 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.676 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.676 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.676 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.676 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-03-16T12:08:09.677 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-03-16T12:08:09.677 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:09.677 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.677 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.677 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:09.677 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.677 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.678 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.678 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.678 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.678 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.678 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.678 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:09.678 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:09.678 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.678 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-16T12:08:09.679 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "stats", 2024-03-16T12:08:09.679 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-03-16T12:08:09.679 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-03-16T12:08:09.679 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-03-16T12:08:09.679 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-03-16T12:08:09.679 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-03-16T12:08:09.679 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:09.679 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.680 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.680 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:09.680 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.680 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.680 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-16T12:08:09.680 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-16T12:08:09.680 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-16T12:08:09.680 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-16T12:08:09.681 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-16T12:08:09.681 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-16T12:08:09.681 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-16T12:08:09.681 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:09.681 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.681 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.681 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.682 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.682 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.682 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-03-16T12:08:09.682 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-03-16T12:08:09.682 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:09.683 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.683 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.683 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:09.683 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.683 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.683 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.683 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.683 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.683 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.684 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.684 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.684 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-03-16T12:08:09.684 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-03-16T12:08:09.684 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:09.684 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.684 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.685 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-03-16T12:08:09.685 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.685 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.685 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-16T12:08:09.685 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-16T12:08:09.685 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-16T12:08:09.685 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-16T12:08:09.686 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-16T12:08:09.686 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-16T12:08:09.686 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-16T12:08:09.686 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:09.686 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.686 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.687 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.687 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.687 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.687 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-03-16T12:08:09.687 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-03-16T12:08:09.687 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:09.688 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.688 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.688 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:09.688 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.688 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.688 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.689 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.689 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.689 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.689 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.689 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:09.689 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:09.690 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.690 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-16T12:08:09.690 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "status", 2024-03-16T12:08:09.690 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-03-16T12:08:09.690 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-03-16T12:08:09.690 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-03-16T12:08:09.690 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-03-16T12:08:09.691 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-03-16T12:08:09.691 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:09.691 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.691 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.691 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:09.691 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.691 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.691 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-16T12:08:09.691 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-16T12:08:09.692 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-16T12:08:09.692 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-16T12:08:09.692 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-16T12:08:09.692 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-16T12:08:09.692 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-16T12:08:09.692 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:09.692 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.692 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.692 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.693 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.693 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.693 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-03-16T12:08:09.693 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-03-16T12:08:09.693 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:09.693 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.693 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.693 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:09.693 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.694 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.694 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.694 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.694 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.694 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.694 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.694 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.694 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-03-16T12:08:09.695 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-03-16T12:08:09.695 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:09.695 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.695 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.695 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-03-16T12:08:09.695 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.695 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.695 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-16T12:08:09.695 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-16T12:08:09.696 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-16T12:08:09.696 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-16T12:08:09.696 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-16T12:08:09.696 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-16T12:08:09.696 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-16T12:08:09.696 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:09.696 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.696 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.696 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.697 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.697 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.697 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-03-16T12:08:09.697 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-03-16T12:08:09.697 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:09.697 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.697 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.697 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:09.697 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.698 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.698 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.698 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.698 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.698 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.698 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.698 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:09.698 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:09.699 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.699 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-16T12:08:09.699 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "telegraf", 2024-03-16T12:08:09.699 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-03-16T12:08:09.699 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-03-16T12:08:09.699 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-03-16T12:08:09.699 INFO:teuthology.orchestra.run.smithi029.stdout: "address": { 2024-03-16T12:08:09.699 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "address", 2024-03-16T12:08:09.699 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:09.700 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.700 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.700 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-03-16T12:08:09.700 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.700 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.700 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.700 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.700 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.701 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.701 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.701 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.701 INFO:teuthology.orchestra.run.smithi029.stdout: "interval": { 2024-03-16T12:08:09.701 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "interval", 2024-03-16T12:08:09.701 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "secs", 2024-03-16T12:08:09.701 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.701 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.701 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "15", 2024-03-16T12:08:09.702 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.702 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.702 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.702 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.702 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.702 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.702 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.702 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.703 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-03-16T12:08:09.703 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-03-16T12:08:09.703 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:09.703 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.703 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.703 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:09.703 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.703 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.704 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-16T12:08:09.704 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-16T12:08:09.704 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-16T12:08:09.704 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-16T12:08:09.704 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-16T12:08:09.704 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-16T12:08:09.704 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-16T12:08:09.704 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:09.704 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.705 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.705 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.705 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.705 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.705 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-03-16T12:08:09.705 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-03-16T12:08:09.705 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:09.705 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.705 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.706 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:09.706 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.706 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.706 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.706 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.706 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.706 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.706 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.707 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.707 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-03-16T12:08:09.707 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-03-16T12:08:09.707 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:09.707 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.707 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.707 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-03-16T12:08:09.707 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.708 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.708 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-16T12:08:09.708 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-16T12:08:09.708 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-16T12:08:09.708 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-16T12:08:09.708 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-16T12:08:09.708 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-16T12:08:09.708 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-16T12:08:09.708 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:09.709 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.709 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.709 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.709 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.709 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.709 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-03-16T12:08:09.709 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-03-16T12:08:09.709 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:09.709 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.709 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.710 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:09.710 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.710 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.710 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.710 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.710 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.710 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.711 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.711 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:09.711 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:09.711 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.711 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-16T12:08:09.711 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "telemetry", 2024-03-16T12:08:09.712 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-03-16T12:08:09.712 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-03-16T12:08:09.712 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-03-16T12:08:09.712 INFO:teuthology.orchestra.run.smithi029.stdout: "channel_basic": { 2024-03-16T12:08:09.712 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "channel_basic", 2024-03-16T12:08:09.712 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:09.712 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.712 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.712 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-16T12:08:09.713 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.713 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.713 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.713 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Share basic cluster information (size, version)", 2024-03-16T12:08:09.713 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.713 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.714 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.714 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.714 INFO:teuthology.orchestra.run.smithi029.stdout: "channel_crash": { 2024-03-16T12:08:09.714 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "channel_crash", 2024-03-16T12:08:09.714 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:09.714 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.714 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.714 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-16T12:08:09.715 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.715 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.715 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.715 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-03-16T12:08:09.715 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.715 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.715 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.716 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.716 INFO:teuthology.orchestra.run.smithi029.stdout: "channel_device": { 2024-03-16T12:08:09.716 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "channel_device", 2024-03-16T12:08:09.716 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:09.716 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.716 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.716 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-16T12:08:09.717 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.717 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.717 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.717 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-03-16T12:08:09.717 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.717 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.718 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.718 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.718 INFO:teuthology.orchestra.run.smithi029.stdout: "channel_ident": { 2024-03-16T12:08:09.718 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "channel_ident", 2024-03-16T12:08:09.718 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:09.718 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.718 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.718 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:09.719 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.719 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.719 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.719 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-03-16T12:08:09.719 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.720 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.720 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.720 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.720 INFO:teuthology.orchestra.run.smithi029.stdout: "channel_perf": { 2024-03-16T12:08:09.720 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "channel_perf", 2024-03-16T12:08:09.720 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:09.720 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.721 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.721 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:09.721 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.721 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.721 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.721 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Share various performance metrics of a cluster", 2024-03-16T12:08:09.721 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.722 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.722 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.722 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.722 INFO:teuthology.orchestra.run.smithi029.stdout: "contact": { 2024-03-16T12:08:09.722 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "contact", 2024-03-16T12:08:09.723 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:09.723 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.723 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.723 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:09.723 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.723 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.724 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.724 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.724 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.724 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.724 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.724 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.724 INFO:teuthology.orchestra.run.smithi029.stdout: "description": { 2024-03-16T12:08:09.724 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "description", 2024-03-16T12:08:09.725 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:09.725 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.725 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.725 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:09.725 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.725 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.725 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.725 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.726 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.726 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.726 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.726 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.726 INFO:teuthology.orchestra.run.smithi029.stdout: "device_url": { 2024-03-16T12:08:09.726 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "device_url", 2024-03-16T12:08:09.726 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:09.726 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.727 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.727 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-03-16T12:08:09.727 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.727 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.727 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.727 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.727 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.727 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.728 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.728 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.728 INFO:teuthology.orchestra.run.smithi029.stdout: "enabled": { 2024-03-16T12:08:09.728 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "enabled", 2024-03-16T12:08:09.728 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:09.728 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.728 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.728 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:09.728 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.729 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.729 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.729 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.729 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.729 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.729 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.729 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.729 INFO:teuthology.orchestra.run.smithi029.stdout: "interval": { 2024-03-16T12:08:09.729 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "interval", 2024-03-16T12:08:09.730 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-16T12:08:09.730 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.730 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.730 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "24", 2024-03-16T12:08:09.730 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "8", 2024-03-16T12:08:09.730 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.730 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.730 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.731 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.731 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.731 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.731 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.731 INFO:teuthology.orchestra.run.smithi029.stdout: "last_opt_revision": { 2024-03-16T12:08:09.731 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "last_opt_revision", 2024-03-16T12:08:09.731 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-16T12:08:09.731 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.731 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.732 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "1", 2024-03-16T12:08:09.732 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.732 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.732 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.732 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.732 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.732 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.732 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.732 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.733 INFO:teuthology.orchestra.run.smithi029.stdout: "leaderboard": { 2024-03-16T12:08:09.733 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "leaderboard", 2024-03-16T12:08:09.733 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:09.733 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.733 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.733 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:09.733 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.733 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.733 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.734 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.734 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.734 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.734 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.734 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.734 INFO:teuthology.orchestra.run.smithi029.stdout: "leaderboard_description": { 2024-03-16T12:08:09.734 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "leaderboard_description", 2024-03-16T12:08:09.734 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:09.735 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.735 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.735 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:09.735 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.735 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.735 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.735 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.735 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.735 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.736 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.736 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.736 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-03-16T12:08:09.736 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-03-16T12:08:09.736 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:09.736 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.736 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.736 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:09.736 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.736 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.737 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-16T12:08:09.737 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-16T12:08:09.737 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-16T12:08:09.737 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-16T12:08:09.737 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-16T12:08:09.737 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-16T12:08:09.737 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-16T12:08:09.737 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:09.738 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.738 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.738 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.738 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.738 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.738 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-03-16T12:08:09.738 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-03-16T12:08:09.738 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:09.738 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.739 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.739 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:09.739 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.739 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.739 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.739 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.739 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.739 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.739 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.740 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.740 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-03-16T12:08:09.740 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-03-16T12:08:09.740 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:09.740 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.740 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.740 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-03-16T12:08:09.740 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.740 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.741 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-16T12:08:09.741 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-16T12:08:09.741 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-16T12:08:09.741 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-16T12:08:09.741 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-16T12:08:09.741 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-16T12:08:09.741 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-16T12:08:09.741 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:09.741 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.742 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.742 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.742 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.742 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.742 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-03-16T12:08:09.742 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-03-16T12:08:09.742 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:09.742 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.743 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.743 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:09.743 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.743 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.743 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.743 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.743 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.743 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.743 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.744 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.744 INFO:teuthology.orchestra.run.smithi029.stdout: "organization": { 2024-03-16T12:08:09.744 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "organization", 2024-03-16T12:08:09.744 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:09.744 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.744 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.745 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:09.745 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.745 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.745 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.745 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.745 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.745 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.745 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.746 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.746 INFO:teuthology.orchestra.run.smithi029.stdout: "proxy": { 2024-03-16T12:08:09.746 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "proxy", 2024-03-16T12:08:09.746 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:09.746 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.746 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.746 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:09.746 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.747 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.747 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.747 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.747 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.748 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.748 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.748 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.748 INFO:teuthology.orchestra.run.smithi029.stdout: "url": { 2024-03-16T12:08:09.748 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "url", 2024-03-16T12:08:09.748 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:09.748 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.748 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.749 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-03-16T12:08:09.749 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.749 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.749 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.749 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.750 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.750 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.750 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.750 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:09.750 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:09.750 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.750 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-16T12:08:09.750 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "test_orchestrator", 2024-03-16T12:08:09.751 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-03-16T12:08:09.751 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-03-16T12:08:09.751 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-03-16T12:08:09.751 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-03-16T12:08:09.751 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-03-16T12:08:09.751 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:09.751 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.751 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.752 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:09.752 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.752 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.752 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-16T12:08:09.752 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-16T12:08:09.752 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-16T12:08:09.752 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-16T12:08:09.752 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-16T12:08:09.752 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-16T12:08:09.753 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-16T12:08:09.753 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:09.753 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.753 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.754 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.754 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.754 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.754 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-03-16T12:08:09.754 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-03-16T12:08:09.754 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:09.754 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.754 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.755 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:09.755 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.755 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.755 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.755 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.756 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.756 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.756 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.756 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.757 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-03-16T12:08:09.757 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-03-16T12:08:09.757 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:09.757 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.757 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.758 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-03-16T12:08:09.758 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.758 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.758 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-16T12:08:09.759 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-16T12:08:09.759 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-16T12:08:09.759 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-16T12:08:09.759 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-16T12:08:09.759 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-16T12:08:09.760 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-16T12:08:09.760 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:09.760 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.760 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.760 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.760 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.760 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.760 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-03-16T12:08:09.761 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-03-16T12:08:09.761 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:09.761 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.761 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.761 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:09.761 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.761 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.761 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.761 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.762 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.762 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.762 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.762 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:09.762 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:09.762 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.762 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-16T12:08:09.762 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "volumes", 2024-03-16T12:08:09.763 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-03-16T12:08:09.763 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-03-16T12:08:09.763 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-03-16T12:08:09.763 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-03-16T12:08:09.763 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-03-16T12:08:09.763 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:09.763 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.763 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.763 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:09.764 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.764 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.764 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-16T12:08:09.764 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-16T12:08:09.764 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-16T12:08:09.764 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-16T12:08:09.764 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-16T12:08:09.764 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-16T12:08:09.764 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-16T12:08:09.765 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:09.765 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.765 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.765 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.765 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.765 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.765 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-03-16T12:08:09.765 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-03-16T12:08:09.766 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:09.766 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.766 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.766 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:09.766 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.766 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.766 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.766 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.767 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.767 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.767 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.767 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.767 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-03-16T12:08:09.767 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-03-16T12:08:09.767 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:09.767 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.767 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.768 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-03-16T12:08:09.768 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.768 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.768 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-16T12:08:09.768 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-16T12:08:09.768 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-16T12:08:09.768 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-16T12:08:09.768 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-16T12:08:09.769 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-16T12:08:09.769 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-16T12:08:09.769 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:09.769 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.769 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.769 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.769 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.769 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.769 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-03-16T12:08:09.770 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-03-16T12:08:09.770 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:09.770 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.770 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.770 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:09.770 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.770 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.770 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.771 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.771 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.771 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.771 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.771 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.771 INFO:teuthology.orchestra.run.smithi029.stdout: "max_concurrent_clones": { 2024-03-16T12:08:09.771 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "max_concurrent_clones", 2024-03-16T12:08:09.771 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-16T12:08:09.772 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.772 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.772 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "4", 2024-03-16T12:08:09.772 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.772 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.772 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.772 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Number of asynchronous cloner threads", 2024-03-16T12:08:09.772 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.772 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.772 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.773 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.773 INFO:teuthology.orchestra.run.smithi029.stdout: "periodic_async_work": { 2024-03-16T12:08:09.773 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "periodic_async_work", 2024-03-16T12:08:09.773 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:09.773 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.773 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.773 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:09.773 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.773 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.774 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.774 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Periodically check for async work", 2024-03-16T12:08:09.774 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.774 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.774 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.774 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.774 INFO:teuthology.orchestra.run.smithi029.stdout: "snapshot_clone_delay": { 2024-03-16T12:08:09.775 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "snapshot_clone_delay", 2024-03-16T12:08:09.775 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-16T12:08:09.775 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.775 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.775 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "0", 2024-03-16T12:08:09.775 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.775 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.775 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.775 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-03-16T12:08:09.776 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.776 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.776 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.776 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.776 INFO:teuthology.orchestra.run.smithi029.stdout: "snapshot_clone_no_wait": { 2024-03-16T12:08:09.776 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "snapshot_clone_no_wait", 2024-03-16T12:08:09.776 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:09.777 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.777 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.777 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-16T12:08:09.777 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.777 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.777 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.777 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-03-16T12:08:09.777 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.778 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.778 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.778 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:09.778 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:09.778 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.778 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-16T12:08:09.778 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "zabbix", 2024-03-16T12:08:09.778 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-03-16T12:08:09.778 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-03-16T12:08:09.779 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-03-16T12:08:09.779 INFO:teuthology.orchestra.run.smithi029.stdout: "discovery_interval": { 2024-03-16T12:08:09.779 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "discovery_interval", 2024-03-16T12:08:09.779 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "uint", 2024-03-16T12:08:09.779 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.779 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.779 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "100", 2024-03-16T12:08:09.779 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.779 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.780 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.780 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.780 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.780 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.780 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.781 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.781 INFO:teuthology.orchestra.run.smithi029.stdout: "identifier": { 2024-03-16T12:08:09.781 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "identifier", 2024-03-16T12:08:09.781 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:09.781 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.781 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.781 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:09.782 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.782 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.782 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.782 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.782 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.782 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.783 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.783 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.783 INFO:teuthology.orchestra.run.smithi029.stdout: "interval": { 2024-03-16T12:08:09.783 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "interval", 2024-03-16T12:08:09.783 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "secs", 2024-03-16T12:08:09.784 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.784 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.784 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "60", 2024-03-16T12:08:09.784 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.784 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.784 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.784 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.784 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.785 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.785 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.785 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.785 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-03-16T12:08:09.785 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-03-16T12:08:09.785 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:09.785 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.786 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.786 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:09.786 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.786 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.786 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-16T12:08:09.786 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-16T12:08:09.787 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-16T12:08:09.787 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-16T12:08:09.787 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-16T12:08:09.787 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-16T12:08:09.787 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-16T12:08:09.787 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:09.788 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.788 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.788 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.788 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.789 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.789 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-03-16T12:08:09.789 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-03-16T12:08:09.789 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:09.789 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.789 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.789 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:09.790 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.790 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.790 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.790 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.790 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.791 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.791 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.791 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.791 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-03-16T12:08:09.792 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-03-16T12:08:09.792 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:09.792 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.792 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.792 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-03-16T12:08:09.793 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.793 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.793 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-16T12:08:09.793 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-16T12:08:09.793 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-16T12:08:09.794 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-16T12:08:09.794 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-16T12:08:09.794 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-16T12:08:09.794 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-16T12:08:09.794 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:09.794 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.794 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.794 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.795 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.795 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.795 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-03-16T12:08:09.795 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-03-16T12:08:09.795 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:09.795 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.795 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.795 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:09.795 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.796 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.796 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.796 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.796 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.796 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.796 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.796 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.796 INFO:teuthology.orchestra.run.smithi029.stdout: "zabbix_host": { 2024-03-16T12:08:09.796 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "zabbix_host", 2024-03-16T12:08:09.797 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:09.797 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.797 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.797 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:09.797 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.797 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.797 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.797 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.797 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.798 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.798 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.798 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.798 INFO:teuthology.orchestra.run.smithi029.stdout: "zabbix_port": { 2024-03-16T12:08:09.798 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "zabbix_port", 2024-03-16T12:08:09.798 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-16T12:08:09.798 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.798 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.799 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "10051", 2024-03-16T12:08:09.799 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.799 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.799 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.799 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.799 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.799 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.800 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.800 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.800 INFO:teuthology.orchestra.run.smithi029.stdout: "zabbix_sender": { 2024-03-16T12:08:09.800 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "zabbix_sender", 2024-03-16T12:08:09.800 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:09.800 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.800 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.800 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-03-16T12:08:09.801 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.801 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.801 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.801 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.801 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.801 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.801 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.801 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:09.801 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:09.802 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:09.802 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-16T12:08:09.802 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:09.802 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:09.802 INFO:teuthology.orchestra.run.smithi029.stdout: "modules": [ 2024-03-16T12:08:09.802 INFO:teuthology.orchestra.run.smithi029.stdout: "cephadm", 2024-03-16T12:08:09.802 INFO:teuthology.orchestra.run.smithi029.stdout: "dashboard", 2024-03-16T12:08:09.802 INFO:teuthology.orchestra.run.smithi029.stdout: "iostat", 2024-03-16T12:08:09.803 INFO:teuthology.orchestra.run.smithi029.stdout: "nfs", 2024-03-16T12:08:09.803 INFO:teuthology.orchestra.run.smithi029.stdout: "restful", 2024-03-16T12:08:09.803 INFO:teuthology.orchestra.run.smithi029.stdout: "snap_schedule" 2024-03-16T12:08:09.803 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:09.803 INFO:teuthology.orchestra.run.smithi029.stdout: "available_modules": [ 2024-03-16T12:08:09.803 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-16T12:08:09.803 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "alerts", 2024-03-16T12:08:09.803 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-03-16T12:08:09.803 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-03-16T12:08:09.804 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-03-16T12:08:09.804 INFO:teuthology.orchestra.run.smithi029.stdout: "interval": { 2024-03-16T12:08:09.804 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "interval", 2024-03-16T12:08:09.804 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "secs", 2024-03-16T12:08:09.804 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.804 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.804 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "60", 2024-03-16T12:08:09.804 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.804 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.804 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.805 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "How frequently to reexamine health status", 2024-03-16T12:08:09.805 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.805 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.805 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.805 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.805 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-03-16T12:08:09.805 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-03-16T12:08:09.805 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:09.805 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.806 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.806 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:09.806 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.806 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.806 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-16T12:08:09.806 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-16T12:08:09.806 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-16T12:08:09.806 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-16T12:08:09.807 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-16T12:08:09.807 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-16T12:08:09.807 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-16T12:08:09.807 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:09.807 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.807 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.807 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.807 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.807 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.808 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-03-16T12:08:09.808 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-03-16T12:08:09.808 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:09.808 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.808 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.808 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:09.808 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.808 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.809 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.809 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.809 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.809 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.809 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.809 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.809 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-03-16T12:08:09.809 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-03-16T12:08:09.809 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:09.810 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.810 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.810 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-03-16T12:08:09.810 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.810 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.810 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-16T12:08:09.810 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-16T12:08:09.810 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-16T12:08:09.811 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-16T12:08:09.811 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-16T12:08:09.811 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-16T12:08:09.811 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-16T12:08:09.811 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:09.811 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.811 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.811 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.811 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.812 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.812 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-03-16T12:08:09.812 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-03-16T12:08:09.812 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:09.812 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.812 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.812 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:09.812 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.812 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.813 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.813 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.813 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.813 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.813 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.813 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.813 INFO:teuthology.orchestra.run.smithi029.stdout: "smtp_destination": { 2024-03-16T12:08:09.813 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "smtp_destination", 2024-03-16T12:08:09.813 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:09.814 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.814 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.814 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:09.814 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.814 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.814 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.815 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Email address to send alerts to", 2024-03-16T12:08:09.815 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.815 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.815 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.815 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.815 INFO:teuthology.orchestra.run.smithi029.stdout: "smtp_from_name": { 2024-03-16T12:08:09.816 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "smtp_from_name", 2024-03-16T12:08:09.816 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:09.816 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.816 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.816 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "Ceph", 2024-03-16T12:08:09.816 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.816 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.817 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.817 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Email From: name", 2024-03-16T12:08:09.817 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.817 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.818 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.818 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.818 INFO:teuthology.orchestra.run.smithi029.stdout: "smtp_host": { 2024-03-16T12:08:09.818 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "smtp_host", 2024-03-16T12:08:09.818 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:09.818 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.818 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.818 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:09.819 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.819 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.819 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.819 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "SMTP server", 2024-03-16T12:08:09.820 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.820 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.820 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.820 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.820 INFO:teuthology.orchestra.run.smithi029.stdout: "smtp_password": { 2024-03-16T12:08:09.820 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "smtp_password", 2024-03-16T12:08:09.820 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:09.820 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.821 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.821 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:09.821 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.821 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.821 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.822 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Password to authenticate with", 2024-03-16T12:08:09.822 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.822 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.822 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.822 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.822 INFO:teuthology.orchestra.run.smithi029.stdout: "smtp_port": { 2024-03-16T12:08:09.822 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "smtp_port", 2024-03-16T12:08:09.822 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-16T12:08:09.823 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.823 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.823 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "465", 2024-03-16T12:08:09.823 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.823 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.823 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.823 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "SMTP port", 2024-03-16T12:08:09.823 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.823 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.823 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.824 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.824 INFO:teuthology.orchestra.run.smithi029.stdout: "smtp_sender": { 2024-03-16T12:08:09.824 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "smtp_sender", 2024-03-16T12:08:09.824 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:09.824 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.824 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.824 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:09.824 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.824 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.824 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.825 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "SMTP envelope sender", 2024-03-16T12:08:09.825 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.825 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.825 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.826 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.826 INFO:teuthology.orchestra.run.smithi029.stdout: "smtp_ssl": { 2024-03-16T12:08:09.826 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "smtp_ssl", 2024-03-16T12:08:09.826 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:09.826 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.826 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.827 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-16T12:08:09.827 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.827 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.827 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.828 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Use SSL to connect to SMTP server", 2024-03-16T12:08:09.828 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.828 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.828 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.829 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.829 INFO:teuthology.orchestra.run.smithi029.stdout: "smtp_user": { 2024-03-16T12:08:09.829 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "smtp_user", 2024-03-16T12:08:09.829 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:09.830 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.830 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.830 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:09.830 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.830 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.830 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.830 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "User to authenticate as", 2024-03-16T12:08:09.831 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.831 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.831 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.831 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:09.831 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:09.831 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.831 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-16T12:08:09.831 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "balancer", 2024-03-16T12:08:09.831 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-03-16T12:08:09.832 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-03-16T12:08:09.832 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-03-16T12:08:09.832 INFO:teuthology.orchestra.run.smithi029.stdout: "active": { 2024-03-16T12:08:09.832 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "active", 2024-03-16T12:08:09.832 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:09.832 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.832 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.832 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-16T12:08:09.832 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.833 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.833 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.833 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "automatically balance PGs across cluster", 2024-03-16T12:08:09.833 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.833 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.833 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.833 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.833 INFO:teuthology.orchestra.run.smithi029.stdout: "begin_time": { 2024-03-16T12:08:09.833 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "begin_time", 2024-03-16T12:08:09.834 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:09.834 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.834 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.834 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "0000", 2024-03-16T12:08:09.834 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.834 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.834 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.834 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "beginning time of day to automatically balance", 2024-03-16T12:08:09.834 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-16T12:08:09.835 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.835 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.835 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.835 INFO:teuthology.orchestra.run.smithi029.stdout: "begin_weekday": { 2024-03-16T12:08:09.835 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "begin_weekday", 2024-03-16T12:08:09.835 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "uint", 2024-03-16T12:08:09.835 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.835 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.835 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "0", 2024-03-16T12:08:09.836 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "0", 2024-03-16T12:08:09.836 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "6", 2024-03-16T12:08:09.836 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.836 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-03-16T12:08:09.836 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-16T12:08:09.836 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.836 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.836 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.837 INFO:teuthology.orchestra.run.smithi029.stdout: "crush_compat_max_iterations": { 2024-03-16T12:08:09.837 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "crush_compat_max_iterations", 2024-03-16T12:08:09.837 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "uint", 2024-03-16T12:08:09.837 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.837 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.837 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "25", 2024-03-16T12:08:09.837 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "1", 2024-03-16T12:08:09.837 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "250", 2024-03-16T12:08:09.837 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.837 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-03-16T12:08:09.838 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.838 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.838 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.838 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.838 INFO:teuthology.orchestra.run.smithi029.stdout: "crush_compat_metrics": { 2024-03-16T12:08:09.838 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "crush_compat_metrics", 2024-03-16T12:08:09.838 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:09.838 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.839 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.839 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "pgs,objects,bytes", 2024-03-16T12:08:09.839 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.839 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.839 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.839 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-03-16T12:08:09.839 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-03-16T12:08:09.839 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.839 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.840 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.840 INFO:teuthology.orchestra.run.smithi029.stdout: "crush_compat_step": { 2024-03-16T12:08:09.840 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "crush_compat_step", 2024-03-16T12:08:09.840 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "float", 2024-03-16T12:08:09.840 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.840 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.840 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "0.5", 2024-03-16T12:08:09.840 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "0.001", 2024-03-16T12:08:09.841 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "0.999", 2024-03-16T12:08:09.841 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.841 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "aggressiveness of optimization", 2024-03-16T12:08:09.841 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-03-16T12:08:09.841 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.841 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.841 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.841 INFO:teuthology.orchestra.run.smithi029.stdout: "end_time": { 2024-03-16T12:08:09.841 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "end_time", 2024-03-16T12:08:09.842 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:09.842 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.842 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.842 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "2359", 2024-03-16T12:08:09.842 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.842 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.842 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.842 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "ending time of day to automatically balance", 2024-03-16T12:08:09.843 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-16T12:08:09.843 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.843 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.843 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.843 INFO:teuthology.orchestra.run.smithi029.stdout: "end_weekday": { 2024-03-16T12:08:09.843 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "end_weekday", 2024-03-16T12:08:09.843 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "uint", 2024-03-16T12:08:09.843 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.843 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.844 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "0", 2024-03-16T12:08:09.844 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "0", 2024-03-16T12:08:09.844 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "6", 2024-03-16T12:08:09.844 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.844 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-03-16T12:08:09.844 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-16T12:08:09.844 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.844 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.844 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.844 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-03-16T12:08:09.845 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-03-16T12:08:09.845 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:09.845 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.845 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.845 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:09.845 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.845 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.845 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-16T12:08:09.845 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-16T12:08:09.846 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-16T12:08:09.846 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-16T12:08:09.846 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-16T12:08:09.846 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-16T12:08:09.846 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-16T12:08:09.846 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:09.846 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.846 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.846 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.847 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.847 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.847 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-03-16T12:08:09.847 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-03-16T12:08:09.847 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:09.847 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.847 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.847 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:09.848 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.848 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.848 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.848 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.848 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.848 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.848 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.848 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.848 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-03-16T12:08:09.849 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-03-16T12:08:09.849 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:09.849 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.849 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.849 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-03-16T12:08:09.849 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.849 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.849 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-16T12:08:09.850 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-16T12:08:09.850 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-16T12:08:09.850 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-16T12:08:09.850 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-16T12:08:09.850 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-16T12:08:09.850 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-16T12:08:09.850 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:09.850 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.850 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.851 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.851 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.851 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.851 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-03-16T12:08:09.851 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-03-16T12:08:09.851 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:09.851 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.852 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.852 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:09.852 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.852 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.852 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.852 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.853 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.853 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.853 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.853 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.853 INFO:teuthology.orchestra.run.smithi029.stdout: "min_score": { 2024-03-16T12:08:09.853 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "min_score", 2024-03-16T12:08:09.853 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "float", 2024-03-16T12:08:09.853 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.853 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.854 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "0", 2024-03-16T12:08:09.854 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.854 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.854 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.854 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-03-16T12:08:09.855 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.855 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.855 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.855 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.855 INFO:teuthology.orchestra.run.smithi029.stdout: "mode": { 2024-03-16T12:08:09.855 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "mode", 2024-03-16T12:08:09.855 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:09.855 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.855 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.856 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "upmap", 2024-03-16T12:08:09.856 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.856 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.856 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-16T12:08:09.856 INFO:teuthology.orchestra.run.smithi029.stdout: "crush-compat", 2024-03-16T12:08:09.857 INFO:teuthology.orchestra.run.smithi029.stdout: "none", 2024-03-16T12:08:09.857 INFO:teuthology.orchestra.run.smithi029.stdout: "read", 2024-03-16T12:08:09.857 INFO:teuthology.orchestra.run.smithi029.stdout: "upmap", 2024-03-16T12:08:09.857 INFO:teuthology.orchestra.run.smithi029.stdout: "upmap-read" 2024-03-16T12:08:09.857 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:09.857 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Balancer mode", 2024-03-16T12:08:09.857 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.857 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.857 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.858 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.858 INFO:teuthology.orchestra.run.smithi029.stdout: "pool_ids": { 2024-03-16T12:08:09.858 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "pool_ids", 2024-03-16T12:08:09.858 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:09.858 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.859 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.859 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:09.859 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.859 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.859 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.859 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-03-16T12:08:09.859 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.860 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.860 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.860 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.860 INFO:teuthology.orchestra.run.smithi029.stdout: "sleep_interval": { 2024-03-16T12:08:09.860 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "sleep_interval", 2024-03-16T12:08:09.860 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "secs", 2024-03-16T12:08:09.860 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.861 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.861 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "60", 2024-03-16T12:08:09.861 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.861 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.861 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.861 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-03-16T12:08:09.862 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.862 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.862 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.862 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.862 INFO:teuthology.orchestra.run.smithi029.stdout: "upmap_max_deviation": { 2024-03-16T12:08:09.862 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "upmap_max_deviation", 2024-03-16T12:08:09.863 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-16T12:08:09.863 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.863 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.863 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "5", 2024-03-16T12:08:09.863 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "1", 2024-03-16T12:08:09.863 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.864 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.864 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "deviation below which no optimization is attempted", 2024-03-16T12:08:09.864 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-03-16T12:08:09.864 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.864 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.864 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.865 INFO:teuthology.orchestra.run.smithi029.stdout: "upmap_max_optimizations": { 2024-03-16T12:08:09.865 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "upmap_max_optimizations", 2024-03-16T12:08:09.865 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "uint", 2024-03-16T12:08:09.865 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.865 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.865 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "10", 2024-03-16T12:08:09.865 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.865 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.865 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.866 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-03-16T12:08:09.866 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.866 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.866 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.866 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:09.866 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:09.866 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.866 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-16T12:08:09.866 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "cephadm", 2024-03-16T12:08:09.867 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-03-16T12:08:09.867 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-03-16T12:08:09.867 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-03-16T12:08:09.867 INFO:teuthology.orchestra.run.smithi029.stdout: "agent_down_multiplier": { 2024-03-16T12:08:09.867 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "agent_down_multiplier", 2024-03-16T12:08:09.867 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "float", 2024-03-16T12:08:09.867 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.867 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.867 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "3.0", 2024-03-16T12:08:09.868 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.868 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.868 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.868 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-03-16T12:08:09.868 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.868 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.868 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.868 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.868 INFO:teuthology.orchestra.run.smithi029.stdout: "agent_refresh_rate": { 2024-03-16T12:08:09.869 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "agent_refresh_rate", 2024-03-16T12:08:09.869 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "secs", 2024-03-16T12:08:09.869 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.869 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.869 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "20", 2024-03-16T12:08:09.869 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.869 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.869 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.870 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-03-16T12:08:09.870 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.870 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.870 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.870 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.870 INFO:teuthology.orchestra.run.smithi029.stdout: "agent_starting_port": { 2024-03-16T12:08:09.870 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "agent_starting_port", 2024-03-16T12:08:09.870 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-16T12:08:09.871 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.871 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.871 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "4721", 2024-03-16T12:08:09.871 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.871 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.871 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.871 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-03-16T12:08:09.871 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.871 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.872 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.872 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.872 INFO:teuthology.orchestra.run.smithi029.stdout: "allow_ptrace": { 2024-03-16T12:08:09.872 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "allow_ptrace", 2024-03-16T12:08:09.872 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:09.872 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.872 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.872 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:09.873 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.873 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.873 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.873 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-03-16T12:08:09.873 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-03-16T12:08:09.873 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.873 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.873 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.873 INFO:teuthology.orchestra.run.smithi029.stdout: "autotune_interval": { 2024-03-16T12:08:09.874 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "autotune_interval", 2024-03-16T12:08:09.874 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "secs", 2024-03-16T12:08:09.874 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.874 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.874 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "600", 2024-03-16T12:08:09.874 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.874 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.874 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.874 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "how frequently to autotune daemon memory", 2024-03-16T12:08:09.875 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.875 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.875 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.875 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.875 INFO:teuthology.orchestra.run.smithi029.stdout: "autotune_memory_target_ratio": { 2024-03-16T12:08:09.875 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "autotune_memory_target_ratio", 2024-03-16T12:08:09.875 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "float", 2024-03-16T12:08:09.875 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.875 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.876 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "0.7", 2024-03-16T12:08:09.876 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.876 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.876 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.876 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-03-16T12:08:09.876 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.876 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.876 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.876 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.877 INFO:teuthology.orchestra.run.smithi029.stdout: "cephadm_log_destination": { 2024-03-16T12:08:09.877 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "cephadm_log_destination", 2024-03-16T12:08:09.877 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:09.877 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.877 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.877 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:09.877 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.877 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.877 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-16T12:08:09.878 INFO:teuthology.orchestra.run.smithi029.stdout: "file", 2024-03-16T12:08:09.878 INFO:teuthology.orchestra.run.smithi029.stdout: "file,syslog", 2024-03-16T12:08:09.878 INFO:teuthology.orchestra.run.smithi029.stdout: "syslog" 2024-03-16T12:08:09.878 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:09.878 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-03-16T12:08:09.878 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.878 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.878 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.879 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.879 INFO:teuthology.orchestra.run.smithi029.stdout: "cgroups_split": { 2024-03-16T12:08:09.879 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "cgroups_split", 2024-03-16T12:08:09.879 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:09.879 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.879 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.879 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-16T12:08:09.879 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.879 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.880 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.880 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-03-16T12:08:09.880 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.880 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.880 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.880 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.880 INFO:teuthology.orchestra.run.smithi029.stdout: "config_checks_enabled": { 2024-03-16T12:08:09.880 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "config_checks_enabled", 2024-03-16T12:08:09.881 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:09.881 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.881 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.881 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:09.881 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.881 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.881 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.881 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-03-16T12:08:09.881 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.882 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.882 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.882 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.882 INFO:teuthology.orchestra.run.smithi029.stdout: "config_dashboard": { 2024-03-16T12:08:09.882 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "config_dashboard", 2024-03-16T12:08:09.882 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:09.882 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.882 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.883 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-16T12:08:09.883 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.883 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.883 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.883 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-03-16T12:08:09.883 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.883 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.883 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.883 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.884 INFO:teuthology.orchestra.run.smithi029.stdout: "container_image_alertmanager": { 2024-03-16T12:08:09.884 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "container_image_alertmanager", 2024-03-16T12:08:09.884 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:09.884 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.884 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.884 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-03-16T12:08:09.884 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.884 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.885 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.885 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Prometheus container image", 2024-03-16T12:08:09.885 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.885 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.885 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.885 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.886 INFO:teuthology.orchestra.run.smithi029.stdout: "container_image_base": { 2024-03-16T12:08:09.886 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "container_image_base", 2024-03-16T12:08:09.886 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:09.886 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.886 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.886 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "quay.io/ceph/ceph", 2024-03-16T12:08:09.886 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.886 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.886 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.887 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Container image name, without the tag", 2024-03-16T12:08:09.887 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.887 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.887 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.887 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.888 INFO:teuthology.orchestra.run.smithi029.stdout: "container_image_elasticsearch": { 2024-03-16T12:08:09.888 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "container_image_elasticsearch", 2024-03-16T12:08:09.888 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:09.888 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.888 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.888 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-03-16T12:08:09.888 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.888 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.889 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.889 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "elasticsearch container image", 2024-03-16T12:08:09.889 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.889 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.889 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.889 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.890 INFO:teuthology.orchestra.run.smithi029.stdout: "container_image_grafana": { 2024-03-16T12:08:09.890 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "container_image_grafana", 2024-03-16T12:08:09.890 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:09.890 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.890 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.890 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "quay.io/ceph/ceph-grafana:9.4.12", 2024-03-16T12:08:09.890 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.890 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.891 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.891 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Prometheus container image", 2024-03-16T12:08:09.891 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.891 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.891 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.892 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.892 INFO:teuthology.orchestra.run.smithi029.stdout: "container_image_haproxy": { 2024-03-16T12:08:09.892 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "container_image_haproxy", 2024-03-16T12:08:09.892 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:09.892 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.892 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.892 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-03-16T12:08:09.893 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.893 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.893 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.893 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "HAproxy container image", 2024-03-16T12:08:09.893 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.893 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.894 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.894 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.894 INFO:teuthology.orchestra.run.smithi029.stdout: "container_image_jaeger_agent": { 2024-03-16T12:08:09.894 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "container_image_jaeger_agent", 2024-03-16T12:08:09.894 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:09.894 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.895 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.895 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-03-16T12:08:09.895 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.895 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.895 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.895 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Jaeger agent container image", 2024-03-16T12:08:09.895 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.895 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.895 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.896 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.896 INFO:teuthology.orchestra.run.smithi029.stdout: "container_image_jaeger_collector": { 2024-03-16T12:08:09.896 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "container_image_jaeger_collector", 2024-03-16T12:08:09.896 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:09.896 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.896 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.896 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-03-16T12:08:09.896 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.896 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.896 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.897 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Jaeger collector container image", 2024-03-16T12:08:09.897 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.897 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.897 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.897 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.897 INFO:teuthology.orchestra.run.smithi029.stdout: "container_image_jaeger_query": { 2024-03-16T12:08:09.897 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "container_image_jaeger_query", 2024-03-16T12:08:09.897 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:09.897 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.898 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.898 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-03-16T12:08:09.898 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.898 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.898 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.898 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Jaeger query container image", 2024-03-16T12:08:09.898 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.899 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.899 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.899 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.899 INFO:teuthology.orchestra.run.smithi029.stdout: "container_image_keepalived": { 2024-03-16T12:08:09.899 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "container_image_keepalived", 2024-03-16T12:08:09.900 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:09.900 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.900 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.900 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-03-16T12:08:09.900 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.900 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.901 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.901 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Keepalived container image", 2024-03-16T12:08:09.901 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.901 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.901 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.901 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.901 INFO:teuthology.orchestra.run.smithi029.stdout: "container_image_loki": { 2024-03-16T12:08:09.902 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "container_image_loki", 2024-03-16T12:08:09.902 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:09.902 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.902 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.902 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-03-16T12:08:09.902 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.902 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.902 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.902 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Loki container image", 2024-03-16T12:08:09.903 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.903 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.903 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.903 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.903 INFO:teuthology.orchestra.run.smithi029.stdout: "container_image_node_exporter": { 2024-03-16T12:08:09.903 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "container_image_node_exporter", 2024-03-16T12:08:09.903 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:09.903 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.903 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.904 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-03-16T12:08:09.904 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.904 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.904 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.904 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Prometheus container image", 2024-03-16T12:08:09.904 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.904 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.904 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.905 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.905 INFO:teuthology.orchestra.run.smithi029.stdout: "container_image_nvmeof": { 2024-03-16T12:08:09.905 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "container_image_nvmeof", 2024-03-16T12:08:09.905 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:09.905 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.905 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.905 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "quay.io/ceph/nvmeof:1.0.0", 2024-03-16T12:08:09.905 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.905 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.906 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.906 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Nvme-of container image", 2024-03-16T12:08:09.906 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.906 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.906 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.906 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.906 INFO:teuthology.orchestra.run.smithi029.stdout: "container_image_prometheus": { 2024-03-16T12:08:09.906 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "container_image_prometheus", 2024-03-16T12:08:09.906 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:09.907 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.907 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.907 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-03-16T12:08:09.907 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.907 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.907 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.907 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Prometheus container image", 2024-03-16T12:08:09.907 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.907 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.908 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.908 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.908 INFO:teuthology.orchestra.run.smithi029.stdout: "container_image_promtail": { 2024-03-16T12:08:09.908 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "container_image_promtail", 2024-03-16T12:08:09.908 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:09.908 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.908 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.908 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-03-16T12:08:09.909 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.909 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.909 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.909 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Promtail container image", 2024-03-16T12:08:09.909 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.909 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.909 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.909 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.909 INFO:teuthology.orchestra.run.smithi029.stdout: "container_image_snmp_gateway": { 2024-03-16T12:08:09.910 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "container_image_snmp_gateway", 2024-03-16T12:08:09.910 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:09.910 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.910 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.910 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-03-16T12:08:09.910 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.910 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.910 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.911 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "SNMP Gateway container image", 2024-03-16T12:08:09.911 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.911 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.911 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.911 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.911 INFO:teuthology.orchestra.run.smithi029.stdout: "container_init": { 2024-03-16T12:08:09.911 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "container_init", 2024-03-16T12:08:09.911 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:09.911 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.912 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.912 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-16T12:08:09.912 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.912 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.912 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.912 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Run podman/docker with `--init`", 2024-03-16T12:08:09.912 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.912 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.913 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.913 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.913 INFO:teuthology.orchestra.run.smithi029.stdout: "daemon_cache_timeout": { 2024-03-16T12:08:09.913 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "daemon_cache_timeout", 2024-03-16T12:08:09.913 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "secs", 2024-03-16T12:08:09.913 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.913 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.913 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "600", 2024-03-16T12:08:09.914 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.914 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.914 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.914 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-03-16T12:08:09.914 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.914 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.914 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.914 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.915 INFO:teuthology.orchestra.run.smithi029.stdout: "default_cephadm_command_timeout": { 2024-03-16T12:08:09.915 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "default_cephadm_command_timeout", 2024-03-16T12:08:09.915 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-16T12:08:09.915 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.915 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.915 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "900", 2024-03-16T12:08:09.915 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.915 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.915 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.916 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-03-16T12:08:09.916 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.916 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.916 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.916 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.916 INFO:teuthology.orchestra.run.smithi029.stdout: "default_registry": { 2024-03-16T12:08:09.916 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "default_registry", 2024-03-16T12:08:09.916 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:09.916 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.917 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.917 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "docker.io", 2024-03-16T12:08:09.917 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.917 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.917 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.917 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-03-16T12:08:09.917 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.917 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.917 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.918 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.918 INFO:teuthology.orchestra.run.smithi029.stdout: "device_cache_timeout": { 2024-03-16T12:08:09.918 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "device_cache_timeout", 2024-03-16T12:08:09.918 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "secs", 2024-03-16T12:08:09.918 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.918 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.918 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "1800", 2024-03-16T12:08:09.918 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.919 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.919 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.919 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "seconds to cache device inventory", 2024-03-16T12:08:09.919 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.919 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.919 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.919 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.919 INFO:teuthology.orchestra.run.smithi029.stdout: "device_enhanced_scan": { 2024-03-16T12:08:09.919 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "device_enhanced_scan", 2024-03-16T12:08:09.920 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:09.920 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.920 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.920 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:09.920 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.920 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.920 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.920 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Use libstoragemgmt during device scans", 2024-03-16T12:08:09.920 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.921 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.921 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.921 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.921 INFO:teuthology.orchestra.run.smithi029.stdout: "facts_cache_timeout": { 2024-03-16T12:08:09.921 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "facts_cache_timeout", 2024-03-16T12:08:09.921 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "secs", 2024-03-16T12:08:09.921 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.921 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.921 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "60", 2024-03-16T12:08:09.922 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.922 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.922 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.922 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "seconds to cache host facts data", 2024-03-16T12:08:09.922 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.922 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.923 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.923 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.923 INFO:teuthology.orchestra.run.smithi029.stdout: "host_check_interval": { 2024-03-16T12:08:09.923 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "host_check_interval", 2024-03-16T12:08:09.923 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "secs", 2024-03-16T12:08:09.923 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.923 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.924 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "600", 2024-03-16T12:08:09.924 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.924 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.924 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.924 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "how frequently to perform a host check", 2024-03-16T12:08:09.924 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.924 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.924 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.924 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.925 INFO:teuthology.orchestra.run.smithi029.stdout: "hw_monitoring": { 2024-03-16T12:08:09.925 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "hw_monitoring", 2024-03-16T12:08:09.925 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:09.925 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.925 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.925 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:09.926 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.926 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.926 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.926 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-03-16T12:08:09.926 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.926 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.926 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.926 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.927 INFO:teuthology.orchestra.run.smithi029.stdout: "inventory_list_all": { 2024-03-16T12:08:09.927 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "inventory_list_all", 2024-03-16T12:08:09.927 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:09.927 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.927 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.927 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:09.928 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.928 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.928 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.928 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-03-16T12:08:09.928 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.928 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.928 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.928 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.929 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-03-16T12:08:09.929 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-03-16T12:08:09.929 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:09.929 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.929 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.929 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:09.929 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.930 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.930 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-16T12:08:09.930 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-16T12:08:09.930 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-16T12:08:09.930 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-16T12:08:09.930 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-16T12:08:09.931 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-16T12:08:09.931 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-16T12:08:09.931 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:09.931 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.931 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.931 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.931 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.932 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.932 INFO:teuthology.orchestra.run.smithi029.stdout: "log_refresh_metadata": { 2024-03-16T12:08:09.932 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_refresh_metadata", 2024-03-16T12:08:09.932 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:09.932 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.932 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.933 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:09.933 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.933 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.933 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.933 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-03-16T12:08:09.933 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.934 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.934 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.934 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.934 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-03-16T12:08:09.934 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-03-16T12:08:09.935 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:09.935 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.935 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.935 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-16T12:08:09.935 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.935 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.935 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.936 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-03-16T12:08:09.936 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.936 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.936 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.936 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.936 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-03-16T12:08:09.936 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-03-16T12:08:09.936 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:09.936 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.937 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.937 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-03-16T12:08:09.937 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.937 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.937 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-16T12:08:09.937 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-16T12:08:09.937 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-16T12:08:09.937 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-16T12:08:09.937 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-16T12:08:09.938 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-16T12:08:09.938 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-16T12:08:09.938 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:09.938 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.938 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.938 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.938 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.938 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.938 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-03-16T12:08:09.939 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-03-16T12:08:09.939 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:09.939 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.939 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.939 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:09.939 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.939 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.939 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.939 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.940 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.940 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.940 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.940 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.940 INFO:teuthology.orchestra.run.smithi029.stdout: "manage_etc_ceph_ceph_conf": { 2024-03-16T12:08:09.940 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-03-16T12:08:09.940 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:09.940 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.940 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.941 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:09.941 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.941 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.941 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.941 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-03-16T12:08:09.941 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.941 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.941 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.942 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.942 INFO:teuthology.orchestra.run.smithi029.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-03-16T12:08:09.942 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-03-16T12:08:09.942 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:09.942 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.942 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.942 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "*", 2024-03-16T12:08:09.942 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.943 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.943 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.943 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-03-16T12:08:09.943 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.943 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.943 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.943 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.943 INFO:teuthology.orchestra.run.smithi029.stdout: "max_count_per_host": { 2024-03-16T12:08:09.944 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "max_count_per_host", 2024-03-16T12:08:09.944 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-16T12:08:09.944 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.944 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.944 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "10", 2024-03-16T12:08:09.944 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.944 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.944 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.944 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "max number of daemons per service per host", 2024-03-16T12:08:09.945 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.945 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.945 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.945 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.945 INFO:teuthology.orchestra.run.smithi029.stdout: "max_osd_draining_count": { 2024-03-16T12:08:09.945 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "max_osd_draining_count", 2024-03-16T12:08:09.945 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-16T12:08:09.945 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.946 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.946 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "10", 2024-03-16T12:08:09.946 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.946 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.946 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.946 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-03-16T12:08:09.946 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.946 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.947 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.947 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.947 INFO:teuthology.orchestra.run.smithi029.stdout: "migration_current": { 2024-03-16T12:08:09.947 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "migration_current", 2024-03-16T12:08:09.947 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-16T12:08:09.947 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.947 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.947 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:09.948 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.948 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.948 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.948 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "internal - do not modify", 2024-03-16T12:08:09.948 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.948 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.948 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.948 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.948 INFO:teuthology.orchestra.run.smithi029.stdout: "mode": { 2024-03-16T12:08:09.949 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "mode", 2024-03-16T12:08:09.949 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:09.949 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.949 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.949 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "root", 2024-03-16T12:08:09.949 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.949 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.949 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-16T12:08:09.949 INFO:teuthology.orchestra.run.smithi029.stdout: "cephadm-package", 2024-03-16T12:08:09.950 INFO:teuthology.orchestra.run.smithi029.stdout: "root" 2024-03-16T12:08:09.950 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:09.950 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "mode for remote execution of cephadm", 2024-03-16T12:08:09.950 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.950 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.950 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.950 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.950 INFO:teuthology.orchestra.run.smithi029.stdout: "oob_default_addr": { 2024-03-16T12:08:09.950 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "oob_default_addr", 2024-03-16T12:08:09.951 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:09.951 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.951 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.951 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "169.254.1.1", 2024-03-16T12:08:09.951 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.951 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.951 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.951 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Default address for RedFish API (oob management).", 2024-03-16T12:08:09.951 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.951 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.952 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.952 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.952 INFO:teuthology.orchestra.run.smithi029.stdout: "prometheus_alerts_path": { 2024-03-16T12:08:09.952 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "prometheus_alerts_path", 2024-03-16T12:08:09.952 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:09.952 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.952 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.952 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-03-16T12:08:09.952 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.953 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.953 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.953 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-03-16T12:08:09.953 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.953 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.953 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.953 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.953 INFO:teuthology.orchestra.run.smithi029.stdout: "registry_insecure": { 2024-03-16T12:08:09.954 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "registry_insecure", 2024-03-16T12:08:09.954 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:09.954 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.954 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.954 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:09.954 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.954 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.954 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.954 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-03-16T12:08:09.955 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.955 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.955 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.955 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.955 INFO:teuthology.orchestra.run.smithi029.stdout: "registry_password": { 2024-03-16T12:08:09.955 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "registry_password", 2024-03-16T12:08:09.955 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:09.955 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.955 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.956 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:09.956 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.956 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.956 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.956 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-03-16T12:08:09.957 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.957 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.957 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.957 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.957 INFO:teuthology.orchestra.run.smithi029.stdout: "registry_url": { 2024-03-16T12:08:09.957 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "registry_url", 2024-03-16T12:08:09.957 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:09.957 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.957 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.958 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:09.958 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.958 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.958 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.958 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-03-16T12:08:09.959 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.959 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.959 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.959 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.959 INFO:teuthology.orchestra.run.smithi029.stdout: "registry_username": { 2024-03-16T12:08:09.959 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "registry_username", 2024-03-16T12:08:09.959 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:09.960 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.960 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.960 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:09.960 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.960 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.960 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.961 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-03-16T12:08:09.961 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.961 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.961 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.961 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.961 INFO:teuthology.orchestra.run.smithi029.stdout: "secure_monitoring_stack": { 2024-03-16T12:08:09.962 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "secure_monitoring_stack", 2024-03-16T12:08:09.962 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:09.962 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.962 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.962 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:09.963 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.963 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.963 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.963 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-03-16T12:08:09.963 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.963 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.963 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.964 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.964 INFO:teuthology.orchestra.run.smithi029.stdout: "service_discovery_port": { 2024-03-16T12:08:09.964 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "service_discovery_port", 2024-03-16T12:08:09.964 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-16T12:08:09.964 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.965 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.965 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "8765", 2024-03-16T12:08:09.965 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.965 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.965 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.966 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "cephadm service discovery port", 2024-03-16T12:08:09.966 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.966 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.966 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.966 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.967 INFO:teuthology.orchestra.run.smithi029.stdout: "ssh_config_file": { 2024-03-16T12:08:09.967 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "ssh_config_file", 2024-03-16T12:08:09.967 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:09.967 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.967 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.967 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:09.967 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.968 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.968 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.968 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-03-16T12:08:09.968 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.968 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.968 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.968 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.968 INFO:teuthology.orchestra.run.smithi029.stdout: "use_agent": { 2024-03-16T12:08:09.969 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "use_agent", 2024-03-16T12:08:09.969 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:09.969 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.969 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.969 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:09.969 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.969 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.969 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.970 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-03-16T12:08:09.970 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.970 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.970 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.971 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.971 INFO:teuthology.orchestra.run.smithi029.stdout: "use_repo_digest": { 2024-03-16T12:08:09.971 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "use_repo_digest", 2024-03-16T12:08:09.971 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:09.971 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.972 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.972 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-16T12:08:09.972 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.972 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.972 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.972 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-03-16T12:08:09.972 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.972 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.973 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.973 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.973 INFO:teuthology.orchestra.run.smithi029.stdout: "warn_on_failed_host_check": { 2024-03-16T12:08:09.973 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "warn_on_failed_host_check", 2024-03-16T12:08:09.973 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:09.973 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.973 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.973 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-16T12:08:09.973 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.974 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.974 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.974 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "raise a health warning if the host check fails", 2024-03-16T12:08:09.974 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.974 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.974 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.974 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.974 INFO:teuthology.orchestra.run.smithi029.stdout: "warn_on_stray_daemons": { 2024-03-16T12:08:09.975 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "warn_on_stray_daemons", 2024-03-16T12:08:09.975 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:09.975 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.975 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.975 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-16T12:08:09.975 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.975 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.975 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.976 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-03-16T12:08:09.976 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.976 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.976 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.976 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.976 INFO:teuthology.orchestra.run.smithi029.stdout: "warn_on_stray_hosts": { 2024-03-16T12:08:09.976 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "warn_on_stray_hosts", 2024-03-16T12:08:09.976 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:09.976 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.977 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.977 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-16T12:08:09.977 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.977 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.977 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.977 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-03-16T12:08:09.977 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.977 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.977 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.978 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:09.978 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:09.978 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.978 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-16T12:08:09.978 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "crash", 2024-03-16T12:08:09.978 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-03-16T12:08:09.978 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-03-16T12:08:09.978 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-03-16T12:08:09.979 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-03-16T12:08:09.979 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-03-16T12:08:09.979 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:09.979 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.979 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.979 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:09.979 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.979 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.979 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-16T12:08:09.980 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-16T12:08:09.980 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-16T12:08:09.980 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-16T12:08:09.980 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-16T12:08:09.980 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-16T12:08:09.980 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-16T12:08:09.980 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:09.980 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.980 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.981 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.981 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.981 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.981 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-03-16T12:08:09.981 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-03-16T12:08:09.981 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:09.981 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.981 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.981 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:09.982 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.982 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.982 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.982 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.982 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.982 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.982 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.982 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.983 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-03-16T12:08:09.983 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-03-16T12:08:09.983 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:09.983 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.983 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.983 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-03-16T12:08:09.983 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.983 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.983 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-16T12:08:09.984 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-16T12:08:09.984 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-16T12:08:09.984 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-16T12:08:09.984 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-16T12:08:09.984 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-16T12:08:09.984 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-16T12:08:09.984 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:09.984 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.984 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.984 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.985 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.985 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.985 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-03-16T12:08:09.985 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-03-16T12:08:09.985 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:09.985 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.985 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.985 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:09.985 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.985 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.986 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.986 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.986 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.986 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.986 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.986 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.986 INFO:teuthology.orchestra.run.smithi029.stdout: "retain_interval": { 2024-03-16T12:08:09.986 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "retain_interval", 2024-03-16T12:08:09.987 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "secs", 2024-03-16T12:08:09.987 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.987 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.987 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "31536000", 2024-03-16T12:08:09.987 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.987 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.987 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.987 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "how long to retain crashes before pruning them", 2024-03-16T12:08:09.987 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.988 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.988 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.988 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.988 INFO:teuthology.orchestra.run.smithi029.stdout: "warn_recent_interval": { 2024-03-16T12:08:09.988 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "warn_recent_interval", 2024-03-16T12:08:09.988 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "secs", 2024-03-16T12:08:09.988 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.988 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:09.988 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "1209600", 2024-03-16T12:08:09.988 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.989 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.989 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.989 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "time interval in which to warn about recent crashes", 2024-03-16T12:08:09.989 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.989 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.989 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.989 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:09.989 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:09.989 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.990 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-16T12:08:09.990 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "dashboard", 2024-03-16T12:08:09.990 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-03-16T12:08:09.990 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-03-16T12:08:09.990 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-03-16T12:08:09.990 INFO:teuthology.orchestra.run.smithi029.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-03-16T12:08:09.990 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-03-16T12:08:09.990 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-16T12:08:09.991 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.991 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.991 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "10", 2024-03-16T12:08:09.991 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.991 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.991 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.991 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.991 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.991 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.992 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.992 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.992 INFO:teuthology.orchestra.run.smithi029.stdout: "ALERTMANAGER_API_HOST": { 2024-03-16T12:08:09.992 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "ALERTMANAGER_API_HOST", 2024-03-16T12:08:09.992 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:09.993 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.993 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.993 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:09.993 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.993 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.993 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.993 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.994 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.994 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.994 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.994 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.994 INFO:teuthology.orchestra.run.smithi029.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-03-16T12:08:09.994 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-03-16T12:08:09.994 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:09.995 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.995 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.995 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-16T12:08:09.995 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.995 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.995 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.996 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.996 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.996 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.996 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.996 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.996 INFO:teuthology.orchestra.run.smithi029.stdout: "AUDIT_API_ENABLED": { 2024-03-16T12:08:09.996 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "AUDIT_API_ENABLED", 2024-03-16T12:08:09.996 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:09.996 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.997 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.997 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:09.997 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.997 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.997 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:09.998 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:09.998 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:09.998 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:09.998 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:09.998 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:09.998 INFO:teuthology.orchestra.run.smithi029.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-03-16T12:08:09.998 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-03-16T12:08:09.998 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:09.999 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:09.999 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:09.999 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-16T12:08:09.999 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:09.999 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:09.999 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:10.000 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.000 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.000 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.000 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.000 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.000 INFO:teuthology.orchestra.run.smithi029.stdout: "ENABLE_BROWSABLE_API": { 2024-03-16T12:08:10.000 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "ENABLE_BROWSABLE_API", 2024-03-16T12:08:10.001 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:10.001 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.001 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:10.001 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-16T12:08:10.001 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.001 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.001 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:10.002 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.002 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.002 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.002 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.002 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.002 INFO:teuthology.orchestra.run.smithi029.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-03-16T12:08:10.003 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-03-16T12:08:10.003 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:10.003 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.003 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:10.003 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-16T12:08:10.003 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.004 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.004 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:10.004 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.004 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.004 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.004 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.005 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.005 INFO:teuthology.orchestra.run.smithi029.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-03-16T12:08:10.005 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-03-16T12:08:10.005 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:10.005 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.005 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:10.006 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-16T12:08:10.006 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.006 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.006 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:10.006 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.006 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.006 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.006 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.006 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.007 INFO:teuthology.orchestra.run.smithi029.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-03-16T12:08:10.007 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-03-16T12:08:10.007 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:10.007 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.007 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:10.007 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-16T12:08:10.007 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.007 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.008 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:10.008 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.008 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.008 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.008 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.008 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.008 INFO:teuthology.orchestra.run.smithi029.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-03-16T12:08:10.008 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-03-16T12:08:10.008 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:10.008 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.009 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:10.009 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-16T12:08:10.009 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.009 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.009 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:10.009 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.009 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.009 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.010 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.010 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.010 INFO:teuthology.orchestra.run.smithi029.stdout: "FEATURE_TOGGLE_NFS": { 2024-03-16T12:08:10.010 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-03-16T12:08:10.010 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:10.010 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.010 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:10.010 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-16T12:08:10.011 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.011 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.011 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:10.011 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.011 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.011 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.011 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.011 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.011 INFO:teuthology.orchestra.run.smithi029.stdout: "FEATURE_TOGGLE_RBD": { 2024-03-16T12:08:10.012 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-03-16T12:08:10.012 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:10.012 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.012 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:10.012 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-16T12:08:10.012 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.012 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.012 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:10.013 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.013 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.013 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.013 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.013 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.013 INFO:teuthology.orchestra.run.smithi029.stdout: "FEATURE_TOGGLE_RGW": { 2024-03-16T12:08:10.013 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-03-16T12:08:10.013 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:10.014 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.014 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:10.014 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-16T12:08:10.014 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.014 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.014 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:10.014 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.014 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.015 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.015 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.015 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.015 INFO:teuthology.orchestra.run.smithi029.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-03-16T12:08:10.015 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-03-16T12:08:10.015 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:10.015 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.015 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:10.016 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:10.016 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.016 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.016 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:10.016 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.016 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.016 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.016 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.016 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.016 INFO:teuthology.orchestra.run.smithi029.stdout: "GRAFANA_API_PASSWORD": { 2024-03-16T12:08:10.017 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "GRAFANA_API_PASSWORD", 2024-03-16T12:08:10.017 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:10.017 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.017 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:10.017 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "admin", 2024-03-16T12:08:10.017 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.017 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.017 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:10.017 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.018 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.018 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.018 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.018 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.018 INFO:teuthology.orchestra.run.smithi029.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-03-16T12:08:10.018 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-03-16T12:08:10.018 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:10.018 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.019 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:10.019 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-16T12:08:10.019 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.019 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.019 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:10.019 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.019 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.019 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.019 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.020 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.020 INFO:teuthology.orchestra.run.smithi029.stdout: "GRAFANA_API_URL": { 2024-03-16T12:08:10.020 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "GRAFANA_API_URL", 2024-03-16T12:08:10.020 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:10.020 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.020 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:10.020 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:10.020 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.021 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.021 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:10.021 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.021 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.021 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.021 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.021 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.021 INFO:teuthology.orchestra.run.smithi029.stdout: "GRAFANA_API_USERNAME": { 2024-03-16T12:08:10.021 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "GRAFANA_API_USERNAME", 2024-03-16T12:08:10.022 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:10.022 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.022 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:10.022 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "admin", 2024-03-16T12:08:10.022 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.022 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.022 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:10.022 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.023 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.023 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.023 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.023 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.023 INFO:teuthology.orchestra.run.smithi029.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-03-16T12:08:10.023 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-03-16T12:08:10.023 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:10.023 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.023 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:10.023 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:10.024 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.024 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.024 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:10.024 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.024 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.024 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.024 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.024 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.024 INFO:teuthology.orchestra.run.smithi029.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-03-16T12:08:10.025 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-03-16T12:08:10.025 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:10.025 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.025 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:10.025 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:10.025 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.025 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.025 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:10.025 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.026 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.026 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.026 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.026 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.026 INFO:teuthology.orchestra.run.smithi029.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-03-16T12:08:10.026 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-03-16T12:08:10.026 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:10.026 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.027 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:10.027 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-16T12:08:10.027 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.027 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.027 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:10.028 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.028 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.028 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.028 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.028 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.028 INFO:teuthology.orchestra.run.smithi029.stdout: "ISSUE_TRACKER_API_KEY": { 2024-03-16T12:08:10.028 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-03-16T12:08:10.028 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:10.028 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.029 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:10.029 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:10.029 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.029 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.029 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:10.029 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.030 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.030 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.030 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.030 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.030 INFO:teuthology.orchestra.run.smithi029.stdout: "MULTICLUSTER_CONFIG": { 2024-03-16T12:08:10.030 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "MULTICLUSTER_CONFIG", 2024-03-16T12:08:10.030 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:10.031 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.031 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:10.031 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "{}", 2024-03-16T12:08:10.031 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.031 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.031 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:10.031 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.032 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.032 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.032 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.032 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.032 INFO:teuthology.orchestra.run.smithi029.stdout: "PROMETHEUS_API_HOST": { 2024-03-16T12:08:10.032 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "PROMETHEUS_API_HOST", 2024-03-16T12:08:10.032 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:10.032 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.033 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:10.033 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:10.033 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.033 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.033 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:10.033 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.034 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.034 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.034 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.034 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.034 INFO:teuthology.orchestra.run.smithi029.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-03-16T12:08:10.034 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-03-16T12:08:10.034 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:10.035 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.035 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:10.035 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-16T12:08:10.035 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.035 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.035 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:10.035 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.036 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.036 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.036 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.036 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.036 INFO:teuthology.orchestra.run.smithi029.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-03-16T12:08:10.036 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-03-16T12:08:10.037 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:10.037 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.037 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:10.037 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:10.037 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.037 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.037 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:10.038 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.038 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.038 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.038 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.038 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.039 INFO:teuthology.orchestra.run.smithi029.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-03-16T12:08:10.039 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-03-16T12:08:10.039 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:10.039 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.039 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:10.039 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:10.039 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.040 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.040 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:10.040 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.040 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.040 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.040 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.040 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.040 INFO:teuthology.orchestra.run.smithi029.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-03-16T12:08:10.040 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-03-16T12:08:10.041 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:10.041 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.041 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:10.041 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-16T12:08:10.041 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.041 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.041 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:10.041 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.041 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.042 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.042 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.042 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.042 INFO:teuthology.orchestra.run.smithi029.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-03-16T12:08:10.042 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-03-16T12:08:10.042 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:10.042 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.042 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:10.043 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-16T12:08:10.043 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.043 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.043 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:10.043 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.043 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.043 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.043 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.044 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.044 INFO:teuthology.orchestra.run.smithi029.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-03-16T12:08:10.044 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-03-16T12:08:10.044 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:10.044 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.044 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:10.044 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:10.044 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.045 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.045 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:10.045 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.045 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.045 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.045 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.045 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.045 INFO:teuthology.orchestra.run.smithi029.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-03-16T12:08:10.045 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-03-16T12:08:10.046 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:10.046 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.046 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:10.046 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:10.046 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.046 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.046 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:10.046 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.047 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.047 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.047 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.047 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.047 INFO:teuthology.orchestra.run.smithi029.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-03-16T12:08:10.047 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-03-16T12:08:10.047 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:10.047 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.047 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:10.048 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:10.048 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.048 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.048 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:10.048 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.048 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.048 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.048 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.048 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.049 INFO:teuthology.orchestra.run.smithi029.stdout: "PWD_POLICY_ENABLED": { 2024-03-16T12:08:10.049 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "PWD_POLICY_ENABLED", 2024-03-16T12:08:10.049 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:10.049 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.049 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:10.049 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-16T12:08:10.049 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.049 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.049 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:10.050 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.050 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.050 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.050 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.050 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.050 INFO:teuthology.orchestra.run.smithi029.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-03-16T12:08:10.050 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-03-16T12:08:10.050 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:10.050 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.051 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:10.051 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-03-16T12:08:10.051 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.051 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.051 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:10.051 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.051 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.051 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.051 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.052 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.052 INFO:teuthology.orchestra.run.smithi029.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-03-16T12:08:10.052 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-03-16T12:08:10.052 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-16T12:08:10.052 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.052 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:10.052 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "10", 2024-03-16T12:08:10.052 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.053 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.053 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:10.053 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.053 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.053 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.053 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.053 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.053 INFO:teuthology.orchestra.run.smithi029.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-03-16T12:08:10.053 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-03-16T12:08:10.054 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-16T12:08:10.054 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.054 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:10.054 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "8", 2024-03-16T12:08:10.054 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.054 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.054 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:10.055 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.055 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.055 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.055 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.055 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.055 INFO:teuthology.orchestra.run.smithi029.stdout: "REST_REQUESTS_TIMEOUT": { 2024-03-16T12:08:10.055 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-03-16T12:08:10.055 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-16T12:08:10.055 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.056 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:10.056 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "45", 2024-03-16T12:08:10.056 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.056 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.056 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:10.056 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.056 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.056 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.056 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.057 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.057 INFO:teuthology.orchestra.run.smithi029.stdout: "RGW_API_ACCESS_KEY": { 2024-03-16T12:08:10.057 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "RGW_API_ACCESS_KEY", 2024-03-16T12:08:10.057 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:10.057 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.057 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:10.057 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:10.057 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.058 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.058 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:10.058 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.058 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.058 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.058 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.058 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.058 INFO:teuthology.orchestra.run.smithi029.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-03-16T12:08:10.058 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-03-16T12:08:10.059 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:10.059 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.059 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:10.059 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "admin", 2024-03-16T12:08:10.059 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.059 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.059 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:10.059 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.060 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.060 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.060 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.060 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.060 INFO:teuthology.orchestra.run.smithi029.stdout: "RGW_API_SECRET_KEY": { 2024-03-16T12:08:10.060 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "RGW_API_SECRET_KEY", 2024-03-16T12:08:10.060 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:10.060 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.060 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:10.061 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:10.061 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.061 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.061 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:10.061 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.061 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.061 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.061 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.061 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.062 INFO:teuthology.orchestra.run.smithi029.stdout: "RGW_API_SSL_VERIFY": { 2024-03-16T12:08:10.062 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "RGW_API_SSL_VERIFY", 2024-03-16T12:08:10.062 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:10.062 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.062 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:10.062 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-16T12:08:10.062 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.062 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.063 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:10.063 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.063 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.063 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.063 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.063 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.063 INFO:teuthology.orchestra.run.smithi029.stdout: "UNSAFE_TLS_v1_2": { 2024-03-16T12:08:10.063 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "UNSAFE_TLS_v1_2", 2024-03-16T12:08:10.063 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:10.064 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.064 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:10.064 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:10.064 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.064 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.064 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:10.064 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.065 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.065 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.065 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.065 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.065 INFO:teuthology.orchestra.run.smithi029.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-03-16T12:08:10.065 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-03-16T12:08:10.065 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-16T12:08:10.065 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.066 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:10.066 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "0", 2024-03-16T12:08:10.066 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.066 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.066 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:10.066 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.066 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.067 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.067 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.067 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.067 INFO:teuthology.orchestra.run.smithi029.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-03-16T12:08:10.067 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-03-16T12:08:10.067 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-16T12:08:10.067 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.068 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:10.068 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "10", 2024-03-16T12:08:10.068 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.068 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.068 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:10.068 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.068 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.068 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.069 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.069 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.069 INFO:teuthology.orchestra.run.smithi029.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-03-16T12:08:10.069 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-03-16T12:08:10.069 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-16T12:08:10.069 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.070 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:10.070 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "5", 2024-03-16T12:08:10.070 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.070 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.070 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:10.070 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.070 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.070 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.071 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.071 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.071 INFO:teuthology.orchestra.run.smithi029.stdout: "cross_origin_url": { 2024-03-16T12:08:10.071 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "cross_origin_url", 2024-03-16T12:08:10.071 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:10.071 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.072 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:10.072 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:10.072 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.072 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.072 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:10.072 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.072 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.073 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.073 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.073 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.073 INFO:teuthology.orchestra.run.smithi029.stdout: "crt_file": { 2024-03-16T12:08:10.073 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "crt_file", 2024-03-16T12:08:10.073 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:10.073 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.073 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:10.074 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:10.074 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.074 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.074 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:10.074 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.075 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.075 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.075 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.075 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.075 INFO:teuthology.orchestra.run.smithi029.stdout: "debug": { 2024-03-16T12:08:10.075 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "debug", 2024-03-16T12:08:10.076 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:10.076 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.076 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:10.076 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:10.076 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.076 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.076 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:10.077 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Enable/disable debug options", 2024-03-16T12:08:10.077 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.077 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.077 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.077 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.078 INFO:teuthology.orchestra.run.smithi029.stdout: "jwt_token_ttl": { 2024-03-16T12:08:10.078 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "jwt_token_ttl", 2024-03-16T12:08:10.078 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-16T12:08:10.078 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.078 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:10.078 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "28800", 2024-03-16T12:08:10.078 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.078 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.079 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:10.079 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.079 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.079 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.079 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.079 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.079 INFO:teuthology.orchestra.run.smithi029.stdout: "key_file": { 2024-03-16T12:08:10.079 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "key_file", 2024-03-16T12:08:10.079 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:10.080 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.080 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:10.080 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:10.080 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.080 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.080 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:10.080 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.080 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.081 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.081 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.081 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.081 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-03-16T12:08:10.081 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-03-16T12:08:10.081 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:10.081 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.081 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:10.081 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:10.081 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.082 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.082 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-16T12:08:10.082 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-16T12:08:10.082 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-16T12:08:10.082 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-16T12:08:10.082 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-16T12:08:10.082 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-16T12:08:10.082 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-16T12:08:10.082 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:10.083 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.083 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.083 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.083 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.083 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.083 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-03-16T12:08:10.083 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-03-16T12:08:10.083 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:10.083 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.084 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:10.084 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:10.084 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.084 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.084 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:10.084 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.084 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.084 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.084 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.085 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.085 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-03-16T12:08:10.085 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-03-16T12:08:10.085 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:10.085 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.085 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:10.085 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-03-16T12:08:10.085 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.085 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.086 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-16T12:08:10.086 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-16T12:08:10.086 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-16T12:08:10.086 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-16T12:08:10.086 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-16T12:08:10.086 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-16T12:08:10.086 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-16T12:08:10.086 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:10.086 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.087 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.087 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.087 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.087 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.087 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-03-16T12:08:10.087 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-03-16T12:08:10.087 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:10.087 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.087 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:10.087 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:10.088 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.088 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.088 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:10.088 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.088 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.088 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.088 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.088 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.088 INFO:teuthology.orchestra.run.smithi029.stdout: "motd": { 2024-03-16T12:08:10.089 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "motd", 2024-03-16T12:08:10.089 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:10.089 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.089 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:10.089 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:10.089 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.089 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.089 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:10.089 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "The message of the day", 2024-03-16T12:08:10.089 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.090 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.090 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.090 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.090 INFO:teuthology.orchestra.run.smithi029.stdout: "redirect_resolve_ip_addr": { 2024-03-16T12:08:10.090 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "redirect_resolve_ip_addr", 2024-03-16T12:08:10.090 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:10.090 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.090 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:10.091 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:10.091 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.091 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.091 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:10.091 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.091 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.091 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.091 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.091 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.091 INFO:teuthology.orchestra.run.smithi029.stdout: "server_addr": { 2024-03-16T12:08:10.092 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "server_addr", 2024-03-16T12:08:10.092 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:10.092 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.092 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:10.092 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "::", 2024-03-16T12:08:10.092 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.092 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.092 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:10.093 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.093 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.093 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.093 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.093 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.093 INFO:teuthology.orchestra.run.smithi029.stdout: "server_port": { 2024-03-16T12:08:10.093 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "server_port", 2024-03-16T12:08:10.093 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-16T12:08:10.093 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.094 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:10.094 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "8080", 2024-03-16T12:08:10.094 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.094 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.094 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:10.094 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.094 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.094 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.094 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.095 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.095 INFO:teuthology.orchestra.run.smithi029.stdout: "ssl": { 2024-03-16T12:08:10.095 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "ssl", 2024-03-16T12:08:10.095 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:10.095 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.095 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:10.095 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-16T12:08:10.095 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.095 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.095 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:10.096 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.096 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.096 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.096 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.096 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.096 INFO:teuthology.orchestra.run.smithi029.stdout: "ssl_server_port": { 2024-03-16T12:08:10.096 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "ssl_server_port", 2024-03-16T12:08:10.096 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-16T12:08:10.096 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.097 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:10.097 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "8443", 2024-03-16T12:08:10.097 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.097 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.097 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:10.097 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.097 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.097 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.097 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.097 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.098 INFO:teuthology.orchestra.run.smithi029.stdout: "standby_behaviour": { 2024-03-16T12:08:10.098 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "standby_behaviour", 2024-03-16T12:08:10.098 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:10.098 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.098 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:10.098 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "redirect", 2024-03-16T12:08:10.099 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.099 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.099 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-16T12:08:10.099 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-16T12:08:10.099 INFO:teuthology.orchestra.run.smithi029.stdout: "redirect" 2024-03-16T12:08:10.100 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:10.100 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.100 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.100 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.100 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.100 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.100 INFO:teuthology.orchestra.run.smithi029.stdout: "standby_error_status_code": { 2024-03-16T12:08:10.101 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "standby_error_status_code", 2024-03-16T12:08:10.101 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-16T12:08:10.101 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.101 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:10.102 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "500", 2024-03-16T12:08:10.102 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "400", 2024-03-16T12:08:10.102 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "599", 2024-03-16T12:08:10.102 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:10.102 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.102 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.102 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.103 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.103 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.103 INFO:teuthology.orchestra.run.smithi029.stdout: "url_prefix": { 2024-03-16T12:08:10.103 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "url_prefix", 2024-03-16T12:08:10.104 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:10.104 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.104 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:10.104 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:10.104 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.104 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.104 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:10.104 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.105 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.105 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.105 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.106 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:10.106 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:10.106 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.106 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-16T12:08:10.106 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "devicehealth", 2024-03-16T12:08:10.106 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-03-16T12:08:10.106 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-03-16T12:08:10.107 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-03-16T12:08:10.107 INFO:teuthology.orchestra.run.smithi029.stdout: "enable_monitoring": { 2024-03-16T12:08:10.107 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "enable_monitoring", 2024-03-16T12:08:10.107 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:10.108 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.108 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:10.108 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-16T12:08:10.108 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.108 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.109 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:10.109 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "monitor device health metrics", 2024-03-16T12:08:10.109 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.109 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.110 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.110 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.110 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-03-16T12:08:10.110 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-03-16T12:08:10.110 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:10.111 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.111 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:10.111 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:10.111 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.111 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.112 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-16T12:08:10.112 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-16T12:08:10.112 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-16T12:08:10.112 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-16T12:08:10.112 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-16T12:08:10.112 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-16T12:08:10.112 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-16T12:08:10.112 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:10.113 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.113 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.113 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.113 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.113 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.113 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-03-16T12:08:10.113 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-03-16T12:08:10.113 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:10.114 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.114 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:10.114 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:10.114 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.114 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.114 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:10.114 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.115 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.115 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.115 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.115 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.115 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-03-16T12:08:10.115 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-03-16T12:08:10.115 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:10.115 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.116 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:10.116 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-03-16T12:08:10.116 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.116 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.116 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-16T12:08:10.116 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-16T12:08:10.116 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-16T12:08:10.116 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-16T12:08:10.117 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-16T12:08:10.117 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-16T12:08:10.117 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-16T12:08:10.117 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:10.117 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.117 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.117 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.117 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.117 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.118 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-03-16T12:08:10.118 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-03-16T12:08:10.118 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:10.118 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.118 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:10.118 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:10.118 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.118 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.119 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:10.119 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.119 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.119 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.119 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.119 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.119 INFO:teuthology.orchestra.run.smithi029.stdout: "mark_out_threshold": { 2024-03-16T12:08:10.119 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "mark_out_threshold", 2024-03-16T12:08:10.119 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "secs", 2024-03-16T12:08:10.120 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.120 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:10.120 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "2419200", 2024-03-16T12:08:10.120 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.120 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.120 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:10.120 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-03-16T12:08:10.120 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.120 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.121 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.121 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.121 INFO:teuthology.orchestra.run.smithi029.stdout: "pool_name": { 2024-03-16T12:08:10.121 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "pool_name", 2024-03-16T12:08:10.121 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:10.121 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.121 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:10.121 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "device_health_metrics", 2024-03-16T12:08:10.121 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.122 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.122 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:10.122 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "name of pool in which to store device health metrics", 2024-03-16T12:08:10.122 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.122 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.122 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.122 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.122 INFO:teuthology.orchestra.run.smithi029.stdout: "retention_period": { 2024-03-16T12:08:10.122 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "retention_period", 2024-03-16T12:08:10.123 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "secs", 2024-03-16T12:08:10.123 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.123 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:10.123 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "15552000", 2024-03-16T12:08:10.123 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.123 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.123 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:10.123 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "how long to retain device health metrics", 2024-03-16T12:08:10.123 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.124 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.124 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.124 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.124 INFO:teuthology.orchestra.run.smithi029.stdout: "scrape_frequency": { 2024-03-16T12:08:10.124 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "scrape_frequency", 2024-03-16T12:08:10.124 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "secs", 2024-03-16T12:08:10.124 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.124 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:10.124 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "86400", 2024-03-16T12:08:10.125 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.125 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.125 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:10.125 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "how frequently to scrape device health metrics", 2024-03-16T12:08:10.125 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.125 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.125 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.125 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.126 INFO:teuthology.orchestra.run.smithi029.stdout: "self_heal": { 2024-03-16T12:08:10.126 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "self_heal", 2024-03-16T12:08:10.126 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:10.126 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.126 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:10.126 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-16T12:08:10.126 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.126 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.126 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:10.127 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-03-16T12:08:10.127 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.127 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.127 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.127 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.127 INFO:teuthology.orchestra.run.smithi029.stdout: "sleep_interval": { 2024-03-16T12:08:10.127 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "sleep_interval", 2024-03-16T12:08:10.127 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "secs", 2024-03-16T12:08:10.127 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.128 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:10.128 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "600", 2024-03-16T12:08:10.128 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.128 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.128 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:10.128 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "how frequently to wake up and check device health", 2024-03-16T12:08:10.128 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.128 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.128 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.129 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.129 INFO:teuthology.orchestra.run.smithi029.stdout: "warn_threshold": { 2024-03-16T12:08:10.129 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "warn_threshold", 2024-03-16T12:08:10.129 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "secs", 2024-03-16T12:08:10.129 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.129 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:10.129 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "7257600", 2024-03-16T12:08:10.129 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.129 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.130 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:10.130 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-03-16T12:08:10.130 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.130 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.130 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.130 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:10.130 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:10.130 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.131 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-16T12:08:10.131 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "diskprediction_local", 2024-03-16T12:08:10.131 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-03-16T12:08:10.131 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-03-16T12:08:10.131 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-03-16T12:08:10.131 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-03-16T12:08:10.131 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-03-16T12:08:10.131 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:10.131 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.132 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:10.132 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:10.132 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.132 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.132 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-16T12:08:10.132 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-16T12:08:10.132 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-16T12:08:10.133 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-16T12:08:10.133 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-16T12:08:10.133 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-16T12:08:10.133 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-16T12:08:10.133 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:10.133 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.133 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.134 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.134 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.134 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.134 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-03-16T12:08:10.134 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-03-16T12:08:10.134 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:10.134 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.134 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:10.135 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:10.135 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.135 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.135 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:10.135 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.135 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.135 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.135 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.135 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.136 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-03-16T12:08:10.136 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-03-16T12:08:10.136 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:10.136 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.136 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:10.136 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-03-16T12:08:10.136 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.136 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.136 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-16T12:08:10.137 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-16T12:08:10.137 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-16T12:08:10.137 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-16T12:08:10.137 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-16T12:08:10.137 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-16T12:08:10.137 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-16T12:08:10.137 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:10.137 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.138 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.138 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.138 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.138 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.138 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-03-16T12:08:10.138 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-03-16T12:08:10.139 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:10.139 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.139 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:10.139 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:10.139 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.139 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.139 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:10.139 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.140 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.140 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.140 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.140 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.140 INFO:teuthology.orchestra.run.smithi029.stdout: "predict_interval": { 2024-03-16T12:08:10.140 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "predict_interval", 2024-03-16T12:08:10.140 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:10.141 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.141 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:10.141 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "86400", 2024-03-16T12:08:10.141 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.141 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.141 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:10.141 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.141 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.142 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.142 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.142 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.142 INFO:teuthology.orchestra.run.smithi029.stdout: "predictor_model": { 2024-03-16T12:08:10.142 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "predictor_model", 2024-03-16T12:08:10.142 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:10.142 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.143 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:10.143 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "prophetstor", 2024-03-16T12:08:10.143 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.143 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.144 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:10.144 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.144 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.144 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.144 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.144 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.144 INFO:teuthology.orchestra.run.smithi029.stdout: "sleep_interval": { 2024-03-16T12:08:10.145 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "sleep_interval", 2024-03-16T12:08:10.145 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:10.145 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.145 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:10.145 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "600", 2024-03-16T12:08:10.145 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.146 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.146 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:10.146 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.146 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.146 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.146 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.147 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:10.147 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:10.147 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.147 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-16T12:08:10.147 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "influx", 2024-03-16T12:08:10.147 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": false, 2024-03-16T12:08:10.147 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "influxdb python module not found", 2024-03-16T12:08:10.148 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-03-16T12:08:10.148 INFO:teuthology.orchestra.run.smithi029.stdout: "batch_size": { 2024-03-16T12:08:10.148 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "batch_size", 2024-03-16T12:08:10.148 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-16T12:08:10.148 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.148 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:10.148 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "5000", 2024-03-16T12:08:10.148 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.148 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.149 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:10.149 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-03-16T12:08:10.149 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.149 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.149 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.149 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.149 INFO:teuthology.orchestra.run.smithi029.stdout: "database": { 2024-03-16T12:08:10.150 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "database", 2024-03-16T12:08:10.150 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:10.150 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.150 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:10.150 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "ceph", 2024-03-16T12:08:10.150 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.150 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.150 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:10.151 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-03-16T12:08:10.151 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.151 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.151 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.151 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.151 INFO:teuthology.orchestra.run.smithi029.stdout: "hostname": { 2024-03-16T12:08:10.151 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "hostname", 2024-03-16T12:08:10.151 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:10.152 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.152 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:10.152 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:10.152 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.152 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.152 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:10.152 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "InfluxDB server hostname", 2024-03-16T12:08:10.152 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.152 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.152 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.153 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.153 INFO:teuthology.orchestra.run.smithi029.stdout: "interval": { 2024-03-16T12:08:10.153 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "interval", 2024-03-16T12:08:10.153 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "secs", 2024-03-16T12:08:10.153 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.153 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:10.153 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "30", 2024-03-16T12:08:10.153 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "5", 2024-03-16T12:08:10.153 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.154 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:10.154 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-03-16T12:08:10.154 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.154 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.154 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.154 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.154 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-03-16T12:08:10.154 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-03-16T12:08:10.155 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:10.155 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.155 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:10.155 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:10.155 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.155 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.155 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-16T12:08:10.155 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-16T12:08:10.155 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-16T12:08:10.156 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-16T12:08:10.156 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-16T12:08:10.156 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-16T12:08:10.156 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-16T12:08:10.156 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:10.156 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.156 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.156 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.157 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.157 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.157 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-03-16T12:08:10.157 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-03-16T12:08:10.157 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:10.157 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.157 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:10.157 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:10.158 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.158 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.158 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:10.158 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.158 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.158 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.158 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.158 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.159 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-03-16T12:08:10.159 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-03-16T12:08:10.159 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:10.159 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.159 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:10.159 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-03-16T12:08:10.159 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.159 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.159 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-16T12:08:10.160 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-16T12:08:10.160 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-16T12:08:10.160 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-16T12:08:10.160 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-16T12:08:10.160 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-16T12:08:10.160 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-16T12:08:10.160 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:10.160 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.160 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.160 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.161 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.161 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.161 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-03-16T12:08:10.161 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-03-16T12:08:10.161 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:10.161 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.161 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:10.161 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:10.162 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.162 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.162 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:10.162 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.162 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.162 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.162 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.162 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.162 INFO:teuthology.orchestra.run.smithi029.stdout: "password": { 2024-03-16T12:08:10.163 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "password", 2024-03-16T12:08:10.163 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:10.163 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.163 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:10.163 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:10.163 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.163 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.163 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:10.164 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "password of InfluxDB server user", 2024-03-16T12:08:10.164 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.164 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.164 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.164 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.164 INFO:teuthology.orchestra.run.smithi029.stdout: "port": { 2024-03-16T12:08:10.164 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "port", 2024-03-16T12:08:10.164 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-16T12:08:10.164 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.164 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:10.165 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "8086", 2024-03-16T12:08:10.165 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.165 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.165 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:10.165 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "InfluxDB server port", 2024-03-16T12:08:10.165 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.165 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.165 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.165 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.166 INFO:teuthology.orchestra.run.smithi029.stdout: "ssl": { 2024-03-16T12:08:10.166 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "ssl", 2024-03-16T12:08:10.166 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:10.166 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.166 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:10.166 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "false", 2024-03-16T12:08:10.166 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.166 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.166 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:10.167 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-03-16T12:08:10.167 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.167 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.167 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.167 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.167 INFO:teuthology.orchestra.run.smithi029.stdout: "threads": { 2024-03-16T12:08:10.167 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "threads", 2024-03-16T12:08:10.167 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-16T12:08:10.168 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.168 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:10.168 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "5", 2024-03-16T12:08:10.168 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "1", 2024-03-16T12:08:10.168 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "32", 2024-03-16T12:08:10.168 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:10.168 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-03-16T12:08:10.168 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.169 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.169 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.169 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.169 INFO:teuthology.orchestra.run.smithi029.stdout: "username": { 2024-03-16T12:08:10.169 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "username", 2024-03-16T12:08:10.169 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:10.169 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.170 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:10.170 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:10.170 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.170 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.170 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:10.170 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "username of InfluxDB server user", 2024-03-16T12:08:10.171 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.171 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.171 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.171 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.171 INFO:teuthology.orchestra.run.smithi029.stdout: "verify_ssl": { 2024-03-16T12:08:10.171 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "verify_ssl", 2024-03-16T12:08:10.171 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:10.171 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.172 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:10.172 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "true", 2024-03-16T12:08:10.172 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.172 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.172 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:10.172 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-03-16T12:08:10.173 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.173 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.173 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.173 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:10.173 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:10.173 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.173 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-16T12:08:10.173 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "insights", 2024-03-16T12:08:10.173 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-03-16T12:08:10.174 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-03-16T12:08:10.174 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-03-16T12:08:10.174 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-03-16T12:08:10.174 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-03-16T12:08:10.174 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:10.174 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.175 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:10.175 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:10.175 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.175 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.175 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-16T12:08:10.175 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-16T12:08:10.175 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-16T12:08:10.176 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-16T12:08:10.176 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-16T12:08:10.176 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-16T12:08:10.176 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-16T12:08:10.176 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:10.176 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.176 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.177 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.177 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.177 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.177 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-03-16T12:08:10.177 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-03-16T12:08:10.177 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:10.177 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.178 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:10.178 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:10.178 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.178 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.178 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:10.178 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.179 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.179 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.179 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.179 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.179 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-03-16T12:08:10.179 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-03-16T12:08:10.179 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:10.180 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.180 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:10.180 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-03-16T12:08:10.180 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.180 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.180 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-16T12:08:10.181 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-16T12:08:10.181 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-16T12:08:10.181 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-16T12:08:10.181 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-16T12:08:10.181 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-16T12:08:10.181 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-16T12:08:10.182 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:10.182 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.182 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.182 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.182 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.182 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.182 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-03-16T12:08:10.183 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-03-16T12:08:10.183 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:10.183 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.183 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:10.183 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:10.183 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.183 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.183 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:10.183 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.184 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.184 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.184 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.184 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:10.184 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:10.184 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.184 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-16T12:08:10.185 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "iostat", 2024-03-16T12:08:10.185 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-03-16T12:08:10.185 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-03-16T12:08:10.185 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-03-16T12:08:10.185 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-03-16T12:08:10.185 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-03-16T12:08:10.185 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:10.185 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.186 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:10.186 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:10.186 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.186 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.186 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-16T12:08:10.186 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-16T12:08:10.186 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-16T12:08:10.186 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-16T12:08:10.187 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-16T12:08:10.187 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-16T12:08:10.187 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-16T12:08:10.187 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:10.187 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.187 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.187 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.187 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.187 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.188 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-03-16T12:08:10.188 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-03-16T12:08:10.188 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:10.188 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.188 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:10.188 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:10.188 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.188 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.189 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:10.189 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.189 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.189 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.189 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.189 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.189 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-03-16T12:08:10.189 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-03-16T12:08:10.189 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:10.190 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.190 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:10.190 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-03-16T12:08:10.190 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.190 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.190 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-16T12:08:10.190 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-16T12:08:10.190 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-16T12:08:10.191 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-16T12:08:10.191 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-16T12:08:10.191 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-16T12:08:10.191 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-16T12:08:10.191 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:10.191 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.191 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.191 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.191 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.192 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.192 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-03-16T12:08:10.192 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-03-16T12:08:10.192 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:10.192 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.192 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:10.192 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:10.192 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.192 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.193 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:10.193 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.193 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.193 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.193 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.193 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:10.193 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:10.193 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.194 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-16T12:08:10.194 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "k8sevents", 2024-03-16T12:08:10.194 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-03-16T12:08:10.194 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-03-16T12:08:10.194 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-03-16T12:08:10.194 INFO:teuthology.orchestra.run.smithi029.stdout: "ceph_event_retention_days": { 2024-03-16T12:08:10.194 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "ceph_event_retention_days", 2024-03-16T12:08:10.194 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-16T12:08:10.194 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.195 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:10.195 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "7", 2024-03-16T12:08:10.195 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.195 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.195 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:10.195 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Days to hold ceph event information within local cache", 2024-03-16T12:08:10.195 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.195 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.195 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.196 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.196 INFO:teuthology.orchestra.run.smithi029.stdout: "config_check_secs": { 2024-03-16T12:08:10.196 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "config_check_secs", 2024-03-16T12:08:10.196 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-16T12:08:10.196 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.196 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:10.196 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "10", 2024-03-16T12:08:10.196 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "10", 2024-03-16T12:08:10.196 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.197 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:10.197 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-03-16T12:08:10.197 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.197 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.197 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.197 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.197 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-03-16T12:08:10.197 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-03-16T12:08:10.198 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:10.198 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.198 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:10.198 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:10.198 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.198 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.198 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-16T12:08:10.198 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-16T12:08:10.198 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-16T12:08:10.199 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-16T12:08:10.199 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-16T12:08:10.199 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-16T12:08:10.199 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-16T12:08:10.199 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:10.199 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.199 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.199 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.199 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.200 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.200 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-03-16T12:08:10.200 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-03-16T12:08:10.200 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:10.200 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.200 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:10.200 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:10.200 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.201 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.201 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:10.201 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.201 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.201 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.201 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.201 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.201 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-03-16T12:08:10.201 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-03-16T12:08:10.202 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:10.202 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.202 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:10.202 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-03-16T12:08:10.202 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.202 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.202 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-16T12:08:10.203 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-16T12:08:10.203 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-16T12:08:10.203 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-16T12:08:10.203 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-16T12:08:10.203 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-16T12:08:10.203 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-16T12:08:10.204 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:10.204 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.204 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.204 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.204 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.204 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.204 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-03-16T12:08:10.204 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-03-16T12:08:10.204 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:10.205 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.205 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:10.205 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:10.205 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.205 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.205 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:10.206 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.206 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.206 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.206 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.206 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:10.206 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:10.206 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.206 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-16T12:08:10.207 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "localpool", 2024-03-16T12:08:10.207 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-03-16T12:08:10.207 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-03-16T12:08:10.207 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-03-16T12:08:10.207 INFO:teuthology.orchestra.run.smithi029.stdout: "failure_domain": { 2024-03-16T12:08:10.207 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "failure_domain", 2024-03-16T12:08:10.207 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:10.208 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.208 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:10.208 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "host", 2024-03-16T12:08:10.208 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.208 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.208 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:10.208 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "failure domain for any created local pool", 2024-03-16T12:08:10.208 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-03-16T12:08:10.209 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.209 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.209 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.209 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-03-16T12:08:10.209 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-03-16T12:08:10.209 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:10.209 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.209 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:10.209 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:10.210 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.210 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.210 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-16T12:08:10.210 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-16T12:08:10.210 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-16T12:08:10.210 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-16T12:08:10.210 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-16T12:08:10.210 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-16T12:08:10.211 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-16T12:08:10.211 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:10.211 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.211 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.211 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.211 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.211 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.211 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-03-16T12:08:10.212 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-03-16T12:08:10.212 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:10.212 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.212 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:10.212 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:10.213 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.213 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.213 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:10.213 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.213 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.213 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.213 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.213 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.214 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-03-16T12:08:10.214 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-03-16T12:08:10.214 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:10.214 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.214 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:10.214 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-03-16T12:08:10.215 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.215 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.215 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-16T12:08:10.215 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-16T12:08:10.215 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-16T12:08:10.215 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-16T12:08:10.215 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-16T12:08:10.216 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-16T12:08:10.216 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-16T12:08:10.216 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:10.216 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.216 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.217 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.217 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.217 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.217 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-03-16T12:08:10.217 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-03-16T12:08:10.217 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:10.218 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.218 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:10.218 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:10.218 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.218 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.218 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:10.218 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.218 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.219 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.219 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.219 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.219 INFO:teuthology.orchestra.run.smithi029.stdout: "min_size": { 2024-03-16T12:08:10.219 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "min_size", 2024-03-16T12:08:10.219 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-16T12:08:10.219 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.219 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:10.220 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:10.220 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.220 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.220 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:10.220 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "default min_size for any created local pool", 2024-03-16T12:08:10.220 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-03-16T12:08:10.220 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.220 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.220 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.221 INFO:teuthology.orchestra.run.smithi029.stdout: "num_rep": { 2024-03-16T12:08:10.221 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "num_rep", 2024-03-16T12:08:10.221 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-16T12:08:10.221 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.221 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:10.221 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "3", 2024-03-16T12:08:10.221 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.221 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.221 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:10.222 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "default replica count for any created local pool", 2024-03-16T12:08:10.222 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.222 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.222 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.222 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.222 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_num": { 2024-03-16T12:08:10.222 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "pg_num", 2024-03-16T12:08:10.222 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-16T12:08:10.222 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.223 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:10.223 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "128", 2024-03-16T12:08:10.223 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.223 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.223 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:10.223 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "default pg_num for any created local pool", 2024-03-16T12:08:10.223 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.223 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.223 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.224 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.224 INFO:teuthology.orchestra.run.smithi029.stdout: "prefix": { 2024-03-16T12:08:10.224 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "prefix", 2024-03-16T12:08:10.224 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:10.224 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.224 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:10.224 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:10.224 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.225 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.225 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:10.225 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "name prefix for any created local pool", 2024-03-16T12:08:10.225 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.225 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.225 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.225 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.225 INFO:teuthology.orchestra.run.smithi029.stdout: "subtree": { 2024-03-16T12:08:10.225 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "subtree", 2024-03-16T12:08:10.226 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:10.226 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.226 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:10.226 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "rack", 2024-03-16T12:08:10.226 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.226 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.226 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:10.226 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "CRUSH level for which to create a local pool", 2024-03-16T12:08:10.227 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-03-16T12:08:10.227 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.227 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.227 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:10.227 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:10.227 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.227 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-16T12:08:10.227 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "mds_autoscaler", 2024-03-16T12:08:10.227 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-03-16T12:08:10.228 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-03-16T12:08:10.228 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-03-16T12:08:10.228 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-03-16T12:08:10.228 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-03-16T12:08:10.228 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:10.228 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.228 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:10.228 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:10.228 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.229 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.229 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-16T12:08:10.229 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-16T12:08:10.229 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-16T12:08:10.229 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-16T12:08:10.229 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-16T12:08:10.229 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-16T12:08:10.229 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-16T12:08:10.229 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:10.229 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.230 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.230 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.230 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.230 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.230 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-03-16T12:08:10.230 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-03-16T12:08:10.230 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:10.230 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.230 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:10.231 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:10.231 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.231 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.231 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:10.231 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.231 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.231 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.231 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.232 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.232 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-03-16T12:08:10.232 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-03-16T12:08:10.232 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:10.232 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.232 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:10.232 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-03-16T12:08:10.232 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.233 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.233 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-16T12:08:10.233 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-16T12:08:10.233 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-16T12:08:10.233 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-16T12:08:10.233 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-16T12:08:10.233 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-16T12:08:10.233 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-16T12:08:10.233 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:10.233 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.234 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.234 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.234 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.234 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.234 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-03-16T12:08:10.234 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-03-16T12:08:10.234 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:10.234 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.234 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:10.235 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:10.235 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.235 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.235 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:10.235 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.235 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.235 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.235 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.235 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:10.235 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:10.236 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.236 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-16T12:08:10.236 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "mirroring", 2024-03-16T12:08:10.236 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-03-16T12:08:10.236 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-03-16T12:08:10.236 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-03-16T12:08:10.236 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-03-16T12:08:10.236 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-03-16T12:08:10.236 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:10.237 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.237 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:10.237 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:10.237 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.237 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.237 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-16T12:08:10.237 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-16T12:08:10.237 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-16T12:08:10.237 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-16T12:08:10.238 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-16T12:08:10.238 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-16T12:08:10.238 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-16T12:08:10.238 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:10.238 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.238 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.238 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.238 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.238 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.239 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-03-16T12:08:10.239 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-03-16T12:08:10.239 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:10.239 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.239 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:10.239 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:10.239 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.240 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.240 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:10.240 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.240 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.240 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.240 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.240 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.241 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-03-16T12:08:10.241 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-03-16T12:08:10.241 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:10.241 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.241 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:10.241 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-03-16T12:08:10.241 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.241 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.241 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-16T12:08:10.242 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-16T12:08:10.242 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-16T12:08:10.242 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-16T12:08:10.242 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-16T12:08:10.242 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-16T12:08:10.242 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-16T12:08:10.243 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:10.243 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.243 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.243 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.243 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.243 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.243 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-03-16T12:08:10.243 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-03-16T12:08:10.244 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:10.244 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.244 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:10.244 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:10.244 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.244 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.245 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:10.245 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.245 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.245 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.245 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.245 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:10.245 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:10.245 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.245 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-16T12:08:10.246 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "nfs", 2024-03-16T12:08:10.246 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-03-16T12:08:10.246 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-03-16T12:08:10.246 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-03-16T12:08:10.246 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-03-16T12:08:10.246 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-03-16T12:08:10.246 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:10.247 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.247 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:10.247 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:10.247 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.247 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.247 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-16T12:08:10.247 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-16T12:08:10.248 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-16T12:08:10.248 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-16T12:08:10.248 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-16T12:08:10.248 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-16T12:08:10.248 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-16T12:08:10.248 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:10.248 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.248 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.249 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.249 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.249 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.249 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-03-16T12:08:10.249 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-03-16T12:08:10.249 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:10.250 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.250 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:10.250 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:10.250 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.250 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.250 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:10.251 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.251 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.251 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.251 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.251 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.251 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-03-16T12:08:10.252 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-03-16T12:08:10.252 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:10.252 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.252 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:10.252 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-03-16T12:08:10.253 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.253 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.253 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-16T12:08:10.253 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-16T12:08:10.253 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-16T12:08:10.253 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-16T12:08:10.253 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-16T12:08:10.253 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-16T12:08:10.253 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-16T12:08:10.254 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:10.254 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.254 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.254 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.254 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.254 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.254 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-03-16T12:08:10.254 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-03-16T12:08:10.255 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:10.255 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.255 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:10.255 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:10.255 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.255 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.255 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:10.255 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.255 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.256 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.256 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.256 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:10.256 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:10.256 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.256 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-16T12:08:10.256 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "orchestrator", 2024-03-16T12:08:10.256 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-03-16T12:08:10.256 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-03-16T12:08:10.257 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-03-16T12:08:10.257 INFO:teuthology.orchestra.run.smithi029.stdout: "fail_fs": { 2024-03-16T12:08:10.257 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "fail_fs", 2024-03-16T12:08:10.257 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:10.257 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.257 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:10.257 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:10.257 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.257 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.258 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:10.258 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-03-16T12:08:10.258 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.258 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.258 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.258 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.258 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-03-16T12:08:10.258 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-03-16T12:08:10.259 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:10.259 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.259 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:10.259 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:10.259 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.259 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.259 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-16T12:08:10.259 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-16T12:08:10.259 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-16T12:08:10.260 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-16T12:08:10.260 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-16T12:08:10.260 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-16T12:08:10.260 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-16T12:08:10.260 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:10.260 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.260 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.260 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.260 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.261 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.261 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-03-16T12:08:10.261 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-03-16T12:08:10.261 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:10.261 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.261 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:10.261 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:10.261 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.262 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.262 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:10.262 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.262 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.262 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.262 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.262 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.262 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-03-16T12:08:10.263 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-03-16T12:08:10.263 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:10.263 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.263 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:10.263 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-03-16T12:08:10.263 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.263 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.263 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-16T12:08:10.263 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-16T12:08:10.264 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-16T12:08:10.264 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-16T12:08:10.264 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-16T12:08:10.264 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-16T12:08:10.264 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-16T12:08:10.264 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:10.264 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.264 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.264 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.265 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.265 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.265 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-03-16T12:08:10.265 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-03-16T12:08:10.265 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:10.265 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.265 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:10.265 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:10.265 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.266 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.266 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:10.266 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.266 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.266 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.266 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.266 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.266 INFO:teuthology.orchestra.run.smithi029.stdout: "orchestrator": { 2024-03-16T12:08:10.267 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "orchestrator", 2024-03-16T12:08:10.267 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:10.267 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.267 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:10.267 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:10.267 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.267 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.267 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-16T12:08:10.267 INFO:teuthology.orchestra.run.smithi029.stdout: "cephadm", 2024-03-16T12:08:10.268 INFO:teuthology.orchestra.run.smithi029.stdout: "rook", 2024-03-16T12:08:10.268 INFO:teuthology.orchestra.run.smithi029.stdout: "test_orchestrator" 2024-03-16T12:08:10.268 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:10.268 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Orchestrator backend", 2024-03-16T12:08:10.268 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.268 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.268 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.268 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:10.268 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:10.269 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.269 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-16T12:08:10.269 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "osd_perf_query", 2024-03-16T12:08:10.269 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-03-16T12:08:10.269 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-03-16T12:08:10.269 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-03-16T12:08:10.269 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-03-16T12:08:10.269 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-03-16T12:08:10.269 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:10.270 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.270 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:10.270 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:10.270 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.270 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.270 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-16T12:08:10.270 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-16T12:08:10.270 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-16T12:08:10.270 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-16T12:08:10.271 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-16T12:08:10.271 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-16T12:08:10.271 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-16T12:08:10.271 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:10.271 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.271 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.271 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.271 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.271 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.271 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-03-16T12:08:10.272 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-03-16T12:08:10.272 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:10.272 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.272 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:10.272 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:10.272 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.272 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.272 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:10.272 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.273 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.273 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.273 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.273 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.274 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-03-16T12:08:10.274 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-03-16T12:08:10.274 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:10.274 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.274 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:10.274 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-03-16T12:08:10.274 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.274 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.275 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-16T12:08:10.275 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-16T12:08:10.275 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-16T12:08:10.275 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-16T12:08:10.275 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-16T12:08:10.275 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-16T12:08:10.275 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-16T12:08:10.275 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:10.276 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.276 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.276 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.276 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.277 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.277 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-03-16T12:08:10.277 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-03-16T12:08:10.277 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:10.277 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.277 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:10.277 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:10.277 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.278 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.278 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:10.278 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.278 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.279 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.279 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.279 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:10.279 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:10.279 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.279 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-16T12:08:10.279 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "osd_support", 2024-03-16T12:08:10.279 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-03-16T12:08:10.280 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-03-16T12:08:10.280 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-03-16T12:08:10.280 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-03-16T12:08:10.280 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-03-16T12:08:10.281 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:10.281 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.281 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:10.281 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:10.281 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.281 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.281 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-16T12:08:10.281 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-16T12:08:10.281 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-16T12:08:10.282 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-16T12:08:10.282 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-16T12:08:10.282 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-16T12:08:10.282 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-16T12:08:10.282 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:10.282 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.282 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.282 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.283 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.283 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.283 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-03-16T12:08:10.283 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-03-16T12:08:10.283 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:10.283 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.283 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:10.283 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:10.284 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.284 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.284 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:10.284 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.284 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.285 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.285 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.285 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.285 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-03-16T12:08:10.285 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-03-16T12:08:10.286 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:10.286 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.286 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:10.286 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-03-16T12:08:10.287 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.287 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.287 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-16T12:08:10.287 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-16T12:08:10.288 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-16T12:08:10.288 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-16T12:08:10.288 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-16T12:08:10.288 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-16T12:08:10.288 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-16T12:08:10.289 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:10.289 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.289 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.289 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.289 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.289 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.289 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-03-16T12:08:10.289 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-03-16T12:08:10.289 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:10.290 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.290 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:10.290 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:10.290 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.290 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.290 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:10.290 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.290 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.291 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.291 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.291 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:10.291 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:10.291 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.291 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-16T12:08:10.291 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "pg_autoscaler", 2024-03-16T12:08:10.291 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-03-16T12:08:10.291 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-03-16T12:08:10.292 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-03-16T12:08:10.292 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-03-16T12:08:10.292 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-03-16T12:08:10.292 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:10.292 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.292 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:10.292 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:10.292 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.292 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.293 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-16T12:08:10.293 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-16T12:08:10.293 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-16T12:08:10.293 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-16T12:08:10.293 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-16T12:08:10.293 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-16T12:08:10.293 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-16T12:08:10.293 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:10.293 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.294 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.294 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.294 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.294 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.294 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-03-16T12:08:10.294 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-03-16T12:08:10.294 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:10.294 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.295 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:10.295 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:10.295 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.295 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.295 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:10.295 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.295 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.295 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.295 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.296 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.296 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-03-16T12:08:10.296 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-03-16T12:08:10.296 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:10.296 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.296 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:10.296 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-03-16T12:08:10.296 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.297 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.297 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-16T12:08:10.297 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-16T12:08:10.297 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-16T12:08:10.297 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-16T12:08:10.297 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-16T12:08:10.297 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-16T12:08:10.297 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-16T12:08:10.297 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:10.298 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.298 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.298 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.298 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.298 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.298 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-03-16T12:08:10.298 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-03-16T12:08:10.298 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:10.299 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.299 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:10.299 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:10.299 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.299 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.299 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:10.299 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.299 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.300 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.300 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.300 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.300 INFO:teuthology.orchestra.run.smithi029.stdout: "sleep_interval": { 2024-03-16T12:08:10.300 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "sleep_interval", 2024-03-16T12:08:10.300 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "secs", 2024-03-16T12:08:10.300 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.300 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:10.300 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "60", 2024-03-16T12:08:10.301 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.301 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.301 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:10.301 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.301 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.301 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.301 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.301 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.301 INFO:teuthology.orchestra.run.smithi029.stdout: "threshold": { 2024-03-16T12:08:10.301 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "threshold", 2024-03-16T12:08:10.302 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "float", 2024-03-16T12:08:10.302 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.302 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:10.302 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "3.0", 2024-03-16T12:08:10.302 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "1.0", 2024-03-16T12:08:10.302 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.302 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:10.303 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "scaling threshold", 2024-03-16T12:08:10.303 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-03-16T12:08:10.303 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.303 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.303 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:10.303 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:10.303 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.303 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-16T12:08:10.303 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "progress", 2024-03-16T12:08:10.304 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-03-16T12:08:10.304 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-03-16T12:08:10.304 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-03-16T12:08:10.304 INFO:teuthology.orchestra.run.smithi029.stdout: "allow_pg_recovery_event": { 2024-03-16T12:08:10.304 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "allow_pg_recovery_event", 2024-03-16T12:08:10.304 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:10.304 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.304 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:10.304 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:10.304 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.305 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.305 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:10.305 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "allow the module to show pg recovery progress", 2024-03-16T12:08:10.305 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.305 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.305 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.305 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.305 INFO:teuthology.orchestra.run.smithi029.stdout: "enabled": { 2024-03-16T12:08:10.305 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "enabled", 2024-03-16T12:08:10.306 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:10.306 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.306 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:10.306 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-16T12:08:10.306 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.306 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.306 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:10.306 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.306 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.307 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.307 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.307 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.307 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-03-16T12:08:10.307 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-03-16T12:08:10.307 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:10.307 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.307 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:10.308 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:10.308 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.308 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.308 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-16T12:08:10.308 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-16T12:08:10.308 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-16T12:08:10.308 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-16T12:08:10.308 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-16T12:08:10.308 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-16T12:08:10.309 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-16T12:08:10.309 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:10.309 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.309 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.309 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.309 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.309 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.310 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-03-16T12:08:10.310 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-03-16T12:08:10.310 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:10.310 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.310 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:10.311 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:10.311 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.311 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.311 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:10.311 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.311 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.311 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.311 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.312 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.312 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-03-16T12:08:10.312 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-03-16T12:08:10.312 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:10.313 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.313 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:10.313 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-03-16T12:08:10.313 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.313 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.313 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-16T12:08:10.313 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-16T12:08:10.313 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-16T12:08:10.313 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-16T12:08:10.314 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-16T12:08:10.314 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-16T12:08:10.314 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-16T12:08:10.314 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:10.315 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.315 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.315 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.315 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.315 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.315 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-03-16T12:08:10.315 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-03-16T12:08:10.316 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:10.316 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.316 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:10.316 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:10.316 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.317 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.317 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:10.317 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.317 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.317 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.317 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.317 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.318 INFO:teuthology.orchestra.run.smithi029.stdout: "max_completed_events": { 2024-03-16T12:08:10.318 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "max_completed_events", 2024-03-16T12:08:10.318 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-16T12:08:10.318 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.318 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:10.319 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "50", 2024-03-16T12:08:10.319 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.319 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.319 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:10.319 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "number of past completed events to remember", 2024-03-16T12:08:10.319 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.319 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.320 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.320 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.320 INFO:teuthology.orchestra.run.smithi029.stdout: "sleep_interval": { 2024-03-16T12:08:10.320 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "sleep_interval", 2024-03-16T12:08:10.320 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "secs", 2024-03-16T12:08:10.320 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.320 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:10.320 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "5", 2024-03-16T12:08:10.321 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.321 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.321 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:10.321 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "how long the module is going to sleep", 2024-03-16T12:08:10.321 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.321 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.322 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.322 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:10.322 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:10.322 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.322 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-16T12:08:10.323 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "prometheus", 2024-03-16T12:08:10.323 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-03-16T12:08:10.323 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-03-16T12:08:10.323 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-03-16T12:08:10.323 INFO:teuthology.orchestra.run.smithi029.stdout: "cache": { 2024-03-16T12:08:10.323 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "cache", 2024-03-16T12:08:10.324 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:10.324 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.324 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:10.324 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-16T12:08:10.324 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.324 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.325 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:10.325 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.325 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.325 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.325 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.325 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.325 INFO:teuthology.orchestra.run.smithi029.stdout: "exclude_perf_counters": { 2024-03-16T12:08:10.325 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "exclude_perf_counters", 2024-03-16T12:08:10.326 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:10.326 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.326 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:10.326 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-16T12:08:10.326 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.326 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.326 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:10.326 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-03-16T12:08:10.327 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-03-16T12:08:10.327 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.327 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.327 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.327 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-03-16T12:08:10.327 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-03-16T12:08:10.327 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:10.327 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.327 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:10.328 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:10.328 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.328 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.328 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-16T12:08:10.328 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-16T12:08:10.328 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-16T12:08:10.328 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-16T12:08:10.328 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-16T12:08:10.328 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-16T12:08:10.328 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-16T12:08:10.329 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:10.329 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.329 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.329 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.329 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.329 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.329 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-03-16T12:08:10.329 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-03-16T12:08:10.329 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:10.330 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.330 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:10.330 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:10.330 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.330 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.330 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:10.330 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.330 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.331 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.331 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.331 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.331 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-03-16T12:08:10.331 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-03-16T12:08:10.331 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:10.331 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.331 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:10.331 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-03-16T12:08:10.332 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.332 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.332 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-16T12:08:10.332 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-16T12:08:10.332 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-16T12:08:10.332 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-16T12:08:10.332 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-16T12:08:10.333 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-16T12:08:10.333 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-16T12:08:10.333 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:10.333 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.333 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.333 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.333 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.333 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.333 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-03-16T12:08:10.334 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-03-16T12:08:10.334 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:10.334 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.334 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:10.334 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:10.334 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.334 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.334 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:10.335 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.335 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.335 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.335 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.335 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.335 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_stats_pools": { 2024-03-16T12:08:10.335 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "rbd_stats_pools", 2024-03-16T12:08:10.335 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:10.335 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.336 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:10.336 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:10.336 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.336 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.336 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:10.336 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.336 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.336 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.337 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.337 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.337 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_stats_pools_refresh_interval": { 2024-03-16T12:08:10.337 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-03-16T12:08:10.337 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-16T12:08:10.337 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.337 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:10.337 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "300", 2024-03-16T12:08:10.337 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.338 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.338 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:10.338 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.338 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.338 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.338 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.338 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.338 INFO:teuthology.orchestra.run.smithi029.stdout: "scrape_interval": { 2024-03-16T12:08:10.339 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "scrape_interval", 2024-03-16T12:08:10.339 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "float", 2024-03-16T12:08:10.339 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.339 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:10.339 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "15.0", 2024-03-16T12:08:10.339 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.339 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.339 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:10.340 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.340 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.340 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.340 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.340 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.340 INFO:teuthology.orchestra.run.smithi029.stdout: "server_addr": { 2024-03-16T12:08:10.340 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "server_addr", 2024-03-16T12:08:10.340 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:10.341 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.341 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:10.341 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "::", 2024-03-16T12:08:10.341 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.341 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.341 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:10.341 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-03-16T12:08:10.341 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.341 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.341 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.342 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.342 INFO:teuthology.orchestra.run.smithi029.stdout: "server_port": { 2024-03-16T12:08:10.342 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "server_port", 2024-03-16T12:08:10.342 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-16T12:08:10.342 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.342 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:10.342 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "9283", 2024-03-16T12:08:10.343 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.343 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.343 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:10.343 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-03-16T12:08:10.343 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.343 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.343 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.343 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.343 INFO:teuthology.orchestra.run.smithi029.stdout: "stale_cache_strategy": { 2024-03-16T12:08:10.344 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "stale_cache_strategy", 2024-03-16T12:08:10.344 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:10.344 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.344 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:10.344 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "log", 2024-03-16T12:08:10.345 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.345 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.345 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:10.345 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.345 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.345 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.345 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.346 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.346 INFO:teuthology.orchestra.run.smithi029.stdout: "standby_behaviour": { 2024-03-16T12:08:10.346 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "standby_behaviour", 2024-03-16T12:08:10.346 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:10.346 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.346 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:10.346 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "default", 2024-03-16T12:08:10.347 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.347 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.347 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-16T12:08:10.347 INFO:teuthology.orchestra.run.smithi029.stdout: "default", 2024-03-16T12:08:10.347 INFO:teuthology.orchestra.run.smithi029.stdout: "error" 2024-03-16T12:08:10.347 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:10.348 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.348 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.348 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.348 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.348 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.348 INFO:teuthology.orchestra.run.smithi029.stdout: "standby_error_status_code": { 2024-03-16T12:08:10.348 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "standby_error_status_code", 2024-03-16T12:08:10.348 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-16T12:08:10.349 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.349 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:10.349 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "500", 2024-03-16T12:08:10.349 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "400", 2024-03-16T12:08:10.349 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "599", 2024-03-16T12:08:10.349 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:10.350 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.350 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.350 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.350 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.350 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:10.350 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:10.350 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.350 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-16T12:08:10.351 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "rbd_support", 2024-03-16T12:08:10.351 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-03-16T12:08:10.351 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-03-16T12:08:10.351 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-03-16T12:08:10.351 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-03-16T12:08:10.351 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-03-16T12:08:10.352 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:10.352 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.352 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:10.352 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:10.352 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.352 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.352 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-16T12:08:10.352 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-16T12:08:10.353 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-16T12:08:10.353 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-16T12:08:10.353 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-16T12:08:10.353 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-16T12:08:10.353 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-16T12:08:10.353 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:10.354 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.354 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.354 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.354 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.354 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.354 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-03-16T12:08:10.355 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-03-16T12:08:10.355 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:10.355 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.355 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:10.355 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:10.355 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.355 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.355 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:10.356 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.356 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.356 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.356 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.356 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.356 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-03-16T12:08:10.356 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-03-16T12:08:10.356 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:10.357 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.357 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:10.357 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-03-16T12:08:10.357 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.357 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.357 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-16T12:08:10.357 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-16T12:08:10.357 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-16T12:08:10.357 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-16T12:08:10.358 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-16T12:08:10.358 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-16T12:08:10.358 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-16T12:08:10.358 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:10.358 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.359 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.359 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.359 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.359 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.359 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-03-16T12:08:10.360 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-03-16T12:08:10.360 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:10.360 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.360 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:10.360 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:10.360 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.361 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.361 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:10.361 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.361 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.361 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.361 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.361 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.362 INFO:teuthology.orchestra.run.smithi029.stdout: "max_concurrent_snap_create": { 2024-03-16T12:08:10.362 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "max_concurrent_snap_create", 2024-03-16T12:08:10.362 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-16T12:08:10.362 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.362 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:10.362 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "10", 2024-03-16T12:08:10.362 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.362 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.363 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:10.363 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.363 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.363 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.363 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.363 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.363 INFO:teuthology.orchestra.run.smithi029.stdout: "mirror_snapshot_schedule": { 2024-03-16T12:08:10.363 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "mirror_snapshot_schedule", 2024-03-16T12:08:10.363 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:10.364 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.364 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:10.364 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:10.364 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.364 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.364 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:10.364 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.364 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.365 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.365 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.365 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.365 INFO:teuthology.orchestra.run.smithi029.stdout: "trash_purge_schedule": { 2024-03-16T12:08:10.365 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "trash_purge_schedule", 2024-03-16T12:08:10.365 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:10.365 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.365 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:10.365 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:10.366 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.366 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.366 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:10.366 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.366 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.366 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.366 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.366 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:10.367 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:10.367 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.367 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-16T12:08:10.367 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "restful", 2024-03-16T12:08:10.367 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-03-16T12:08:10.367 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-03-16T12:08:10.367 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-03-16T12:08:10.367 INFO:teuthology.orchestra.run.smithi029.stdout: "enable_auth": { 2024-03-16T12:08:10.368 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "enable_auth", 2024-03-16T12:08:10.368 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:10.368 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.368 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:10.368 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-16T12:08:10.368 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.368 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.368 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:10.368 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.368 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.369 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.369 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.369 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.369 INFO:teuthology.orchestra.run.smithi029.stdout: "key_file": { 2024-03-16T12:08:10.369 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "key_file", 2024-03-16T12:08:10.369 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:10.369 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.369 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:10.369 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:10.370 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.370 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.370 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:10.370 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.370 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.370 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.370 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.370 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.370 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-03-16T12:08:10.371 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-03-16T12:08:10.371 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:10.371 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.371 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:10.371 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:10.371 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.371 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.371 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-16T12:08:10.371 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-16T12:08:10.372 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-16T12:08:10.372 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-16T12:08:10.372 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-16T12:08:10.372 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-16T12:08:10.372 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-16T12:08:10.372 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:10.372 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.372 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.372 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.373 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.373 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.373 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-03-16T12:08:10.373 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-03-16T12:08:10.373 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:10.373 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.373 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:10.373 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:10.373 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.373 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.374 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:10.374 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.374 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.374 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.374 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.374 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.374 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-03-16T12:08:10.374 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-03-16T12:08:10.374 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:10.375 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.375 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:10.375 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-03-16T12:08:10.375 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.375 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.375 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-16T12:08:10.375 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-16T12:08:10.375 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-16T12:08:10.375 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-16T12:08:10.375 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-16T12:08:10.376 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-16T12:08:10.376 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-16T12:08:10.376 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:10.376 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.376 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.376 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.376 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.376 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.376 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-03-16T12:08:10.376 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-03-16T12:08:10.377 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:10.377 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.377 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:10.377 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:10.377 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.377 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.377 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:10.377 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.378 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.378 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.378 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.378 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.378 INFO:teuthology.orchestra.run.smithi029.stdout: "server_addr": { 2024-03-16T12:08:10.378 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "server_addr", 2024-03-16T12:08:10.378 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:10.378 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.378 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:10.379 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:10.379 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.379 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.379 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:10.379 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.379 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.379 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.379 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.379 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.379 INFO:teuthology.orchestra.run.smithi029.stdout: "server_port": { 2024-03-16T12:08:10.380 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "server_port", 2024-03-16T12:08:10.380 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:10.380 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.380 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:10.380 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:10.380 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.380 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.380 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:10.380 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.381 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.381 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.381 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.381 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:10.381 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:10.381 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.381 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-16T12:08:10.382 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "rgw", 2024-03-16T12:08:10.382 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-03-16T12:08:10.382 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-03-16T12:08:10.382 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-03-16T12:08:10.382 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-03-16T12:08:10.382 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-03-16T12:08:10.383 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:10.383 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.383 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:10.383 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:10.383 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.383 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.383 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-16T12:08:10.383 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-16T12:08:10.383 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-16T12:08:10.384 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-16T12:08:10.384 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-16T12:08:10.384 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-16T12:08:10.384 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-16T12:08:10.384 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:10.385 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.385 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.385 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.385 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.385 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.385 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-03-16T12:08:10.385 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-03-16T12:08:10.385 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:10.385 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.386 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:10.386 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:10.386 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.386 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.387 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:10.387 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.387 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.387 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.387 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.387 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.387 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-03-16T12:08:10.387 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-03-16T12:08:10.388 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:10.388 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.388 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:10.388 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-03-16T12:08:10.388 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.389 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.389 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-16T12:08:10.389 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-16T12:08:10.389 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-16T12:08:10.389 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-16T12:08:10.389 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-16T12:08:10.389 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-16T12:08:10.390 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-16T12:08:10.390 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:10.390 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.390 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.390 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.390 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.391 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.391 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-03-16T12:08:10.391 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-03-16T12:08:10.391 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:10.391 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.392 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:10.392 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:10.392 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.392 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.392 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:10.393 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.393 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.393 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.393 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.394 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:10.394 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:10.394 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.394 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-16T12:08:10.394 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "rook", 2024-03-16T12:08:10.395 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-03-16T12:08:10.395 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-03-16T12:08:10.395 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-03-16T12:08:10.395 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-03-16T12:08:10.395 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-03-16T12:08:10.395 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:10.395 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.395 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:10.395 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:10.396 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.396 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.396 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-16T12:08:10.396 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-16T12:08:10.396 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-16T12:08:10.396 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-16T12:08:10.396 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-16T12:08:10.396 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-16T12:08:10.396 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-16T12:08:10.396 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:10.397 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.397 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.397 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.397 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.397 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.397 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-03-16T12:08:10.397 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-03-16T12:08:10.397 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:10.397 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.398 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:10.398 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:10.398 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.398 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.398 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:10.398 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.398 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.398 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.399 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.399 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.399 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-03-16T12:08:10.399 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-03-16T12:08:10.399 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:10.399 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.399 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:10.399 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-03-16T12:08:10.399 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.400 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.400 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-16T12:08:10.400 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-16T12:08:10.400 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-16T12:08:10.400 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-16T12:08:10.400 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-16T12:08:10.400 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-16T12:08:10.400 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-16T12:08:10.400 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:10.400 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.401 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.401 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.401 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.401 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.401 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-03-16T12:08:10.401 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-03-16T12:08:10.401 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:10.401 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.401 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:10.401 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:10.402 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.402 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.402 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:10.402 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.402 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.402 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.402 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.402 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.403 INFO:teuthology.orchestra.run.smithi029.stdout: "storage_class": { 2024-03-16T12:08:10.403 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "storage_class", 2024-03-16T12:08:10.403 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:10.403 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.403 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:10.403 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "local", 2024-03-16T12:08:10.403 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.403 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.403 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:10.403 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-03-16T12:08:10.404 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.404 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.404 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.404 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:10.404 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:10.404 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.404 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-16T12:08:10.404 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "selftest", 2024-03-16T12:08:10.404 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-03-16T12:08:10.405 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-03-16T12:08:10.405 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-03-16T12:08:10.405 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-03-16T12:08:10.405 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-03-16T12:08:10.405 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:10.405 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.405 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:10.405 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:10.405 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.405 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.406 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-16T12:08:10.406 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-16T12:08:10.406 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-16T12:08:10.406 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-16T12:08:10.406 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-16T12:08:10.406 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-16T12:08:10.406 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-16T12:08:10.406 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:10.407 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.407 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.407 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.407 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.407 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.407 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-03-16T12:08:10.407 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-03-16T12:08:10.407 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:10.407 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.408 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:10.408 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:10.408 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.408 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.408 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:10.408 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.408 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.408 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.408 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.409 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.409 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-03-16T12:08:10.409 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-03-16T12:08:10.409 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:10.409 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.409 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:10.409 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-03-16T12:08:10.409 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.409 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.409 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-16T12:08:10.410 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-16T12:08:10.410 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-16T12:08:10.410 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-16T12:08:10.410 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-16T12:08:10.410 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-16T12:08:10.410 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-16T12:08:10.410 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:10.410 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.411 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.411 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.411 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.411 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.411 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-03-16T12:08:10.411 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-03-16T12:08:10.411 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:10.411 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.411 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:10.412 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:10.412 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.412 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.412 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:10.412 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.412 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.412 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.412 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.412 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.413 INFO:teuthology.orchestra.run.smithi029.stdout: "roption1": { 2024-03-16T12:08:10.413 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "roption1", 2024-03-16T12:08:10.413 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:10.413 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.413 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:10.413 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:10.413 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.413 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.413 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:10.414 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.414 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.414 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.414 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.414 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.414 INFO:teuthology.orchestra.run.smithi029.stdout: "roption2": { 2024-03-16T12:08:10.414 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "roption2", 2024-03-16T12:08:10.414 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:10.414 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.415 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:10.415 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "xyz", 2024-03-16T12:08:10.415 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.415 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.415 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:10.415 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.415 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.415 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.416 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.416 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.416 INFO:teuthology.orchestra.run.smithi029.stdout: "rwoption1": { 2024-03-16T12:08:10.416 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "rwoption1", 2024-03-16T12:08:10.416 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:10.416 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.416 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:10.417 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:10.417 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.417 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.417 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:10.417 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.417 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.417 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.417 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.418 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.418 INFO:teuthology.orchestra.run.smithi029.stdout: "rwoption2": { 2024-03-16T12:08:10.418 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "rwoption2", 2024-03-16T12:08:10.418 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-16T12:08:10.418 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.418 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:10.419 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:10.419 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.419 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.419 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:10.419 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.419 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.419 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.419 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.420 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.420 INFO:teuthology.orchestra.run.smithi029.stdout: "rwoption3": { 2024-03-16T12:08:10.420 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "rwoption3", 2024-03-16T12:08:10.420 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "float", 2024-03-16T12:08:10.420 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.420 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:10.421 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:10.421 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.421 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.421 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:10.421 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.421 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.421 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.421 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.422 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.422 INFO:teuthology.orchestra.run.smithi029.stdout: "rwoption4": { 2024-03-16T12:08:10.422 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "rwoption4", 2024-03-16T12:08:10.422 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:10.422 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.422 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:10.423 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:10.423 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.423 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.423 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:10.423 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.423 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.423 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.423 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.424 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.424 INFO:teuthology.orchestra.run.smithi029.stdout: "rwoption5": { 2024-03-16T12:08:10.424 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "rwoption5", 2024-03-16T12:08:10.424 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:10.424 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.424 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:10.425 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:10.425 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.425 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.425 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:10.425 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.425 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.425 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.426 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.426 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.426 INFO:teuthology.orchestra.run.smithi029.stdout: "rwoption6": { 2024-03-16T12:08:10.426 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "rwoption6", 2024-03-16T12:08:10.426 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:10.427 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.427 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:10.427 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-16T12:08:10.427 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.427 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.427 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:10.427 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.427 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.427 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.428 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.428 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.428 INFO:teuthology.orchestra.run.smithi029.stdout: "rwoption7": { 2024-03-16T12:08:10.428 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "rwoption7", 2024-03-16T12:08:10.428 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-16T12:08:10.428 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.428 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:10.428 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:10.428 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "1", 2024-03-16T12:08:10.429 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "42", 2024-03-16T12:08:10.429 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:10.429 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.429 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.429 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.429 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.430 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.430 INFO:teuthology.orchestra.run.smithi029.stdout: "testkey": { 2024-03-16T12:08:10.430 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "testkey", 2024-03-16T12:08:10.430 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:10.430 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.430 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:10.431 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:10.431 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.431 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.431 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:10.431 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.431 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.431 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.431 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.431 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.432 INFO:teuthology.orchestra.run.smithi029.stdout: "testlkey": { 2024-03-16T12:08:10.432 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "testlkey", 2024-03-16T12:08:10.432 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:10.432 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.432 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:10.432 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:10.432 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.432 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.432 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:10.433 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.433 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.433 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.433 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.433 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.433 INFO:teuthology.orchestra.run.smithi029.stdout: "testnewline": { 2024-03-16T12:08:10.433 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "testnewline", 2024-03-16T12:08:10.433 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:10.433 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.434 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:10.434 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:10.434 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.434 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.434 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:10.434 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.434 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.434 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.434 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.435 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:10.435 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:10.435 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.435 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-16T12:08:10.435 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "snap_schedule", 2024-03-16T12:08:10.435 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-03-16T12:08:10.435 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-03-16T12:08:10.435 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-03-16T12:08:10.436 INFO:teuthology.orchestra.run.smithi029.stdout: "allow_m_granularity": { 2024-03-16T12:08:10.436 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "allow_m_granularity", 2024-03-16T12:08:10.436 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:10.436 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.436 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:10.436 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:10.436 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.436 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.436 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:10.436 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "allow minute scheduled snapshots", 2024-03-16T12:08:10.437 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.437 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.437 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.437 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.437 INFO:teuthology.orchestra.run.smithi029.stdout: "dump_on_update": { 2024-03-16T12:08:10.437 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "dump_on_update", 2024-03-16T12:08:10.437 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:10.437 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.438 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:10.438 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:10.438 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.438 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.438 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:10.438 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "dump database to debug log on update", 2024-03-16T12:08:10.438 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.438 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.438 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.439 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.439 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-03-16T12:08:10.439 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-03-16T12:08:10.439 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:10.439 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.439 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:10.439 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:10.439 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.439 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.440 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-16T12:08:10.440 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-16T12:08:10.440 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-16T12:08:10.440 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-16T12:08:10.440 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-16T12:08:10.440 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-16T12:08:10.440 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-16T12:08:10.440 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:10.440 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.440 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.441 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.441 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.441 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.441 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-03-16T12:08:10.441 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-03-16T12:08:10.441 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:10.441 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.441 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:10.442 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:10.442 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.442 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.442 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:10.442 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.442 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.442 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.442 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.442 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.443 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-03-16T12:08:10.443 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-03-16T12:08:10.443 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:10.443 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.443 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:10.443 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-03-16T12:08:10.443 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.443 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.443 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-16T12:08:10.444 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-16T12:08:10.444 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-16T12:08:10.444 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-16T12:08:10.444 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-16T12:08:10.444 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-16T12:08:10.444 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-16T12:08:10.444 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:10.444 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.445 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.445 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.445 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.445 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.445 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-03-16T12:08:10.445 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-03-16T12:08:10.445 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:10.445 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.445 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:10.445 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:10.446 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.446 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.446 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:10.446 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.446 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.446 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.446 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.446 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:10.447 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:10.447 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.447 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-16T12:08:10.447 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "stats", 2024-03-16T12:08:10.447 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-03-16T12:08:10.447 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-03-16T12:08:10.447 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-03-16T12:08:10.447 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-03-16T12:08:10.447 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-03-16T12:08:10.448 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:10.448 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.448 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:10.448 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:10.448 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.448 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.448 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-16T12:08:10.448 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-16T12:08:10.448 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-16T12:08:10.449 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-16T12:08:10.449 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-16T12:08:10.449 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-16T12:08:10.449 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-16T12:08:10.449 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:10.449 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.449 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.449 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.450 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.450 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.450 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-03-16T12:08:10.450 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-03-16T12:08:10.450 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:10.450 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.450 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:10.450 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:10.451 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.451 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.451 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:10.451 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.451 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.451 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.451 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.451 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.452 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-03-16T12:08:10.452 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-03-16T12:08:10.452 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:10.452 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.452 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:10.452 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-03-16T12:08:10.453 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.453 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.453 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-16T12:08:10.453 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-16T12:08:10.453 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-16T12:08:10.453 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-16T12:08:10.453 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-16T12:08:10.453 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-16T12:08:10.453 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-16T12:08:10.453 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:10.454 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.454 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.454 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.454 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.454 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.455 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-03-16T12:08:10.455 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-03-16T12:08:10.455 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:10.455 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.455 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:10.455 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:10.455 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.455 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.456 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:10.456 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.456 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.456 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.456 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.456 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:10.456 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:10.457 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.457 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-16T12:08:10.457 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "status", 2024-03-16T12:08:10.457 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-03-16T12:08:10.457 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-03-16T12:08:10.457 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-03-16T12:08:10.457 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-03-16T12:08:10.457 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-03-16T12:08:10.458 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:10.458 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.458 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:10.458 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:10.458 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.458 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.459 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-16T12:08:10.459 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-16T12:08:10.459 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-16T12:08:10.459 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-16T12:08:10.459 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-16T12:08:10.459 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-16T12:08:10.459 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-16T12:08:10.459 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:10.460 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.460 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.460 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.460 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.460 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.460 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-03-16T12:08:10.461 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-03-16T12:08:10.461 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:10.461 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.461 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:10.461 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:10.461 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.462 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.462 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:10.462 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.462 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.462 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.462 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.463 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.463 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-03-16T12:08:10.463 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-03-16T12:08:10.463 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:10.463 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.464 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:10.464 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-03-16T12:08:10.464 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.464 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.465 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-16T12:08:10.465 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-16T12:08:10.465 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-16T12:08:10.465 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-16T12:08:10.465 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-16T12:08:10.465 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-16T12:08:10.465 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-16T12:08:10.465 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:10.466 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.466 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.466 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.466 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.466 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.466 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-03-16T12:08:10.466 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-03-16T12:08:10.466 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:10.466 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.467 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:10.467 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:10.467 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.467 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.467 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:10.467 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.467 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.467 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.468 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.468 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:10.468 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:10.468 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.468 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-16T12:08:10.468 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "telegraf", 2024-03-16T12:08:10.468 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-03-16T12:08:10.468 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-03-16T12:08:10.468 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-03-16T12:08:10.469 INFO:teuthology.orchestra.run.smithi029.stdout: "address": { 2024-03-16T12:08:10.469 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "address", 2024-03-16T12:08:10.469 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:10.469 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.469 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:10.469 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-03-16T12:08:10.469 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.469 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.469 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:10.469 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.470 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.470 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.470 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.470 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.470 INFO:teuthology.orchestra.run.smithi029.stdout: "interval": { 2024-03-16T12:08:10.470 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "interval", 2024-03-16T12:08:10.470 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "secs", 2024-03-16T12:08:10.470 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.470 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:10.471 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "15", 2024-03-16T12:08:10.471 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.471 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.471 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:10.471 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.471 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.471 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.471 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.471 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.471 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-03-16T12:08:10.472 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-03-16T12:08:10.472 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:10.472 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.472 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:10.472 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:10.472 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.472 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.472 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-16T12:08:10.472 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-16T12:08:10.473 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-16T12:08:10.473 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-16T12:08:10.473 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-16T12:08:10.473 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-16T12:08:10.473 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-16T12:08:10.473 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:10.473 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.473 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.473 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.473 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.474 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.474 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-03-16T12:08:10.474 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-03-16T12:08:10.474 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:10.474 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.474 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:10.474 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:10.474 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.474 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.475 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:10.475 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.475 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.475 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.475 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.475 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.475 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-03-16T12:08:10.475 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-03-16T12:08:10.475 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:10.476 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.476 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:10.476 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-03-16T12:08:10.476 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.476 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.476 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-16T12:08:10.476 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-16T12:08:10.476 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-16T12:08:10.476 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-16T12:08:10.476 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-16T12:08:10.477 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-16T12:08:10.477 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-16T12:08:10.477 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:10.477 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.477 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.477 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.477 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.477 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.477 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-03-16T12:08:10.478 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-03-16T12:08:10.478 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:10.478 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.478 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:10.478 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:10.478 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.478 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.478 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:10.478 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.479 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.479 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.479 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.479 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:10.479 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:10.479 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.479 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-16T12:08:10.479 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "telemetry", 2024-03-16T12:08:10.479 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-03-16T12:08:10.480 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-03-16T12:08:10.480 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-03-16T12:08:10.480 INFO:teuthology.orchestra.run.smithi029.stdout: "channel_basic": { 2024-03-16T12:08:10.480 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "channel_basic", 2024-03-16T12:08:10.480 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:10.480 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.480 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:10.480 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-16T12:08:10.480 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.480 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.481 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:10.481 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Share basic cluster information (size, version)", 2024-03-16T12:08:10.481 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.481 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.481 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.481 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.481 INFO:teuthology.orchestra.run.smithi029.stdout: "channel_crash": { 2024-03-16T12:08:10.481 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "channel_crash", 2024-03-16T12:08:10.481 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:10.481 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.482 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:10.482 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-16T12:08:10.482 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.482 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.482 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:10.482 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-03-16T12:08:10.482 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.482 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.482 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.483 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.483 INFO:teuthology.orchestra.run.smithi029.stdout: "channel_device": { 2024-03-16T12:08:10.483 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "channel_device", 2024-03-16T12:08:10.483 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:10.483 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.483 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:10.483 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-16T12:08:10.483 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.484 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.484 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:10.484 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-03-16T12:08:10.484 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.484 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.484 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.484 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.484 INFO:teuthology.orchestra.run.smithi029.stdout: "channel_ident": { 2024-03-16T12:08:10.484 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "channel_ident", 2024-03-16T12:08:10.485 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:10.485 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.485 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:10.485 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:10.485 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.485 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.485 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:10.485 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-03-16T12:08:10.486 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.486 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.486 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.487 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.487 INFO:teuthology.orchestra.run.smithi029.stdout: "channel_perf": { 2024-03-16T12:08:10.487 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "channel_perf", 2024-03-16T12:08:10.487 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:10.488 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.488 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:10.488 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:10.489 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.489 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.489 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:10.489 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Share various performance metrics of a cluster", 2024-03-16T12:08:10.490 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.490 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.490 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.491 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.491 INFO:teuthology.orchestra.run.smithi029.stdout: "contact": { 2024-03-16T12:08:10.491 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "contact", 2024-03-16T12:08:10.491 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:10.492 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.492 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:10.492 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:10.493 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.493 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.493 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:10.493 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.494 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.494 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.494 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.495 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.495 INFO:teuthology.orchestra.run.smithi029.stdout: "description": { 2024-03-16T12:08:10.495 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "description", 2024-03-16T12:08:10.496 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:10.496 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.496 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:10.496 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:10.497 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.497 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.497 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:10.497 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.498 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.498 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.498 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.498 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.498 INFO:teuthology.orchestra.run.smithi029.stdout: "device_url": { 2024-03-16T12:08:10.498 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "device_url", 2024-03-16T12:08:10.499 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:10.499 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.499 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:10.499 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-03-16T12:08:10.499 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.499 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.499 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:10.499 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.500 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.500 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.500 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.500 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.500 INFO:teuthology.orchestra.run.smithi029.stdout: "enabled": { 2024-03-16T12:08:10.500 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "enabled", 2024-03-16T12:08:10.500 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:10.501 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.501 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:10.501 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:10.501 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.501 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.501 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:10.501 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.501 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.501 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.502 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.502 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.502 INFO:teuthology.orchestra.run.smithi029.stdout: "interval": { 2024-03-16T12:08:10.502 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "interval", 2024-03-16T12:08:10.502 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-16T12:08:10.502 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.502 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:10.502 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "24", 2024-03-16T12:08:10.503 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "8", 2024-03-16T12:08:10.503 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.503 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:10.503 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.503 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.503 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.503 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.503 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.503 INFO:teuthology.orchestra.run.smithi029.stdout: "last_opt_revision": { 2024-03-16T12:08:10.504 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "last_opt_revision", 2024-03-16T12:08:10.504 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-16T12:08:10.504 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.504 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:10.504 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "1", 2024-03-16T12:08:10.504 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.504 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.504 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:10.505 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.505 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.505 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.505 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.505 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.505 INFO:teuthology.orchestra.run.smithi029.stdout: "leaderboard": { 2024-03-16T12:08:10.505 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "leaderboard", 2024-03-16T12:08:10.505 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:10.505 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.506 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:10.506 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:10.506 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.506 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.506 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:10.506 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.506 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.506 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.507 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.507 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.507 INFO:teuthology.orchestra.run.smithi029.stdout: "leaderboard_description": { 2024-03-16T12:08:10.507 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "leaderboard_description", 2024-03-16T12:08:10.507 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:10.507 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.507 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:10.507 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:10.507 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.507 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.508 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:10.508 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.508 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.508 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.508 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.508 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.508 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-03-16T12:08:10.508 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-03-16T12:08:10.509 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:10.509 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.509 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:10.509 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:10.509 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.509 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.509 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-16T12:08:10.509 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-16T12:08:10.510 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-16T12:08:10.510 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-16T12:08:10.510 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-16T12:08:10.510 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-16T12:08:10.510 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-16T12:08:10.510 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:10.510 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.510 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.510 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.511 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.511 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.511 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-03-16T12:08:10.511 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-03-16T12:08:10.511 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:10.511 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.511 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:10.511 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:10.511 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.512 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.512 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:10.512 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.512 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.512 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.512 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.512 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.512 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-03-16T12:08:10.512 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-03-16T12:08:10.513 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:10.513 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.513 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:10.513 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-03-16T12:08:10.513 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.513 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.513 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-16T12:08:10.513 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-16T12:08:10.513 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-16T12:08:10.513 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-16T12:08:10.514 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-16T12:08:10.514 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-16T12:08:10.514 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-16T12:08:10.514 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:10.514 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.514 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.514 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.514 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.514 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.515 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-03-16T12:08:10.515 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-03-16T12:08:10.515 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:10.515 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.515 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:10.515 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:10.515 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.515 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.515 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:10.516 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.516 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.516 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.516 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.516 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.516 INFO:teuthology.orchestra.run.smithi029.stdout: "organization": { 2024-03-16T12:08:10.516 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "organization", 2024-03-16T12:08:10.516 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:10.516 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.516 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:10.517 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:10.517 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.517 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.517 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:10.517 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.517 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.517 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.517 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.517 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.518 INFO:teuthology.orchestra.run.smithi029.stdout: "proxy": { 2024-03-16T12:08:10.518 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "proxy", 2024-03-16T12:08:10.518 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:10.518 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.518 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:10.518 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:10.518 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.518 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.519 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:10.519 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.519 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.519 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.519 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.519 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.519 INFO:teuthology.orchestra.run.smithi029.stdout: "url": { 2024-03-16T12:08:10.519 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "url", 2024-03-16T12:08:10.519 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:10.519 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.520 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:10.520 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-03-16T12:08:10.520 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.520 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.520 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:10.520 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.520 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.520 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.521 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.521 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:10.521 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:10.521 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.521 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-16T12:08:10.521 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "test_orchestrator", 2024-03-16T12:08:10.521 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-03-16T12:08:10.521 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-03-16T12:08:10.521 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-03-16T12:08:10.521 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-03-16T12:08:10.522 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-03-16T12:08:10.522 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:10.522 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.522 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:10.522 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:10.522 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.522 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.523 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-16T12:08:10.523 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-16T12:08:10.523 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-16T12:08:10.523 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-16T12:08:10.523 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-16T12:08:10.523 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-16T12:08:10.523 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-16T12:08:10.523 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:10.524 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.524 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.524 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.524 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.524 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.524 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-03-16T12:08:10.524 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-03-16T12:08:10.524 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:10.525 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.525 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:10.525 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:10.525 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.525 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.525 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:10.525 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.526 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.526 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.526 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.526 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.526 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-03-16T12:08:10.526 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-03-16T12:08:10.526 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:10.526 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.527 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:10.527 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-03-16T12:08:10.527 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.527 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.527 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-16T12:08:10.527 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-16T12:08:10.528 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-16T12:08:10.528 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-16T12:08:10.528 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-16T12:08:10.528 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-16T12:08:10.528 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-16T12:08:10.528 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:10.528 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.528 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.529 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.529 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.529 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.529 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-03-16T12:08:10.529 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-03-16T12:08:10.529 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:10.530 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.530 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:10.530 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:10.530 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.530 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.530 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:10.530 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.530 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.531 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.531 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.531 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:10.531 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:10.531 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.531 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-16T12:08:10.531 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "volumes", 2024-03-16T12:08:10.532 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-03-16T12:08:10.532 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-03-16T12:08:10.532 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-03-16T12:08:10.532 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-03-16T12:08:10.532 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-03-16T12:08:10.532 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:10.533 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.533 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:10.533 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:10.533 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.533 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.533 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-16T12:08:10.534 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-16T12:08:10.534 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-16T12:08:10.534 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-16T12:08:10.534 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-16T12:08:10.534 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-16T12:08:10.535 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-16T12:08:10.535 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:10.535 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.535 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.535 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.535 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.535 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.536 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-03-16T12:08:10.536 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-03-16T12:08:10.536 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:10.536 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.536 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:10.536 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:10.536 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.536 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.537 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:10.537 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.537 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.537 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.537 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.537 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.537 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-03-16T12:08:10.537 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-03-16T12:08:10.537 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:10.538 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.538 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:10.538 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-03-16T12:08:10.538 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.538 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.538 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-16T12:08:10.538 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-16T12:08:10.538 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-16T12:08:10.538 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-16T12:08:10.539 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-16T12:08:10.539 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-16T12:08:10.539 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-16T12:08:10.539 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:10.539 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.539 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.539 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.539 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.539 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.540 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-03-16T12:08:10.540 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-03-16T12:08:10.540 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:10.540 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.540 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:10.540 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:10.540 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.540 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.540 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:10.541 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.541 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.541 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.541 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.541 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.541 INFO:teuthology.orchestra.run.smithi029.stdout: "max_concurrent_clones": { 2024-03-16T12:08:10.541 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "max_concurrent_clones", 2024-03-16T12:08:10.541 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-16T12:08:10.541 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.542 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:10.542 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "4", 2024-03-16T12:08:10.542 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.542 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.542 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:10.542 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Number of asynchronous cloner threads", 2024-03-16T12:08:10.542 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.542 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.543 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.543 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.543 INFO:teuthology.orchestra.run.smithi029.stdout: "periodic_async_work": { 2024-03-16T12:08:10.543 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "periodic_async_work", 2024-03-16T12:08:10.543 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:10.543 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.543 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:10.543 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:10.544 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.544 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.544 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:10.544 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Periodically check for async work", 2024-03-16T12:08:10.544 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.544 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.544 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.544 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.545 INFO:teuthology.orchestra.run.smithi029.stdout: "snapshot_clone_delay": { 2024-03-16T12:08:10.545 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "snapshot_clone_delay", 2024-03-16T12:08:10.545 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-16T12:08:10.545 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.545 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:10.545 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "0", 2024-03-16T12:08:10.545 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.545 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.545 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:10.546 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-03-16T12:08:10.546 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.546 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.546 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.546 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.546 INFO:teuthology.orchestra.run.smithi029.stdout: "snapshot_clone_no_wait": { 2024-03-16T12:08:10.546 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "snapshot_clone_no_wait", 2024-03-16T12:08:10.546 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:10.547 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.547 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:10.547 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "True", 2024-03-16T12:08:10.547 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.547 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.547 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:10.547 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-03-16T12:08:10.547 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.547 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.548 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.548 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:10.548 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:10.548 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.548 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-16T12:08:10.548 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "zabbix", 2024-03-16T12:08:10.548 INFO:teuthology.orchestra.run.smithi029.stdout: "can_run": true, 2024-03-16T12:08:10.548 INFO:teuthology.orchestra.run.smithi029.stdout: "error_string": "", 2024-03-16T12:08:10.548 INFO:teuthology.orchestra.run.smithi029.stdout: "module_options": { 2024-03-16T12:08:10.549 INFO:teuthology.orchestra.run.smithi029.stdout: "discovery_interval": { 2024-03-16T12:08:10.549 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "discovery_interval", 2024-03-16T12:08:10.549 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "uint", 2024-03-16T12:08:10.549 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.549 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:10.549 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "100", 2024-03-16T12:08:10.549 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.549 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.549 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:10.550 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.550 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.550 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.550 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.550 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.550 INFO:teuthology.orchestra.run.smithi029.stdout: "identifier": { 2024-03-16T12:08:10.550 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "identifier", 2024-03-16T12:08:10.550 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:10.551 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.551 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:10.551 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:10.551 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.551 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.551 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:10.551 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.551 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.552 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.552 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.552 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.552 INFO:teuthology.orchestra.run.smithi029.stdout: "interval": { 2024-03-16T12:08:10.552 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "interval", 2024-03-16T12:08:10.552 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "secs", 2024-03-16T12:08:10.552 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.552 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:10.552 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "60", 2024-03-16T12:08:10.552 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.553 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.553 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:10.553 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.553 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.553 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.553 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.553 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.553 INFO:teuthology.orchestra.run.smithi029.stdout: "log_level": { 2024-03-16T12:08:10.553 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_level", 2024-03-16T12:08:10.554 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:10.554 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.554 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:10.554 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:10.554 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.554 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.554 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-16T12:08:10.554 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-16T12:08:10.555 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-16T12:08:10.555 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-16T12:08:10.555 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-16T12:08:10.555 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-16T12:08:10.555 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-16T12:08:10.555 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:10.555 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.555 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.556 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.556 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.556 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.556 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster": { 2024-03-16T12:08:10.556 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster", 2024-03-16T12:08:10.556 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:10.556 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.556 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:10.556 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:10.557 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.557 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.557 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:10.557 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.557 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.558 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.558 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.558 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.558 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_cluster_level": { 2024-03-16T12:08:10.558 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_cluster_level", 2024-03-16T12:08:10.558 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:10.558 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.558 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:10.558 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "info", 2024-03-16T12:08:10.559 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.559 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.559 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [ 2024-03-16T12:08:10.559 INFO:teuthology.orchestra.run.smithi029.stdout: "", 2024-03-16T12:08:10.559 INFO:teuthology.orchestra.run.smithi029.stdout: "critical", 2024-03-16T12:08:10.560 INFO:teuthology.orchestra.run.smithi029.stdout: "debug", 2024-03-16T12:08:10.560 INFO:teuthology.orchestra.run.smithi029.stdout: "error", 2024-03-16T12:08:10.560 INFO:teuthology.orchestra.run.smithi029.stdout: "info", 2024-03-16T12:08:10.560 INFO:teuthology.orchestra.run.smithi029.stdout: "warning" 2024-03-16T12:08:10.560 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:10.560 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.560 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.560 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.560 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.561 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.561 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": { 2024-03-16T12:08:10.561 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "log_to_file", 2024-03-16T12:08:10.561 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "bool", 2024-03-16T12:08:10.561 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.561 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-16T12:08:10.562 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "False", 2024-03-16T12:08:10.562 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.562 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.562 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:10.562 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.562 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.562 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.563 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.563 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.563 INFO:teuthology.orchestra.run.smithi029.stdout: "zabbix_host": { 2024-03-16T12:08:10.563 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "zabbix_host", 2024-03-16T12:08:10.563 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:10.563 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.563 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:10.564 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "", 2024-03-16T12:08:10.564 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.564 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.564 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:10.564 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.564 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.564 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.564 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.565 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.565 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:08.270+0000 7fb02ef7e640 1 -- 172.21.15.29:0/2899694199 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7fb0100a2970 msgr2=0x7fb0100a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:08:10.565 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:08.270+0000 7fb02ef7e640 1 --2- 172.21.15.29:0/2899694199 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7fb0100a2970 0x7fb0100a4e30 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fb020005fd0 tx=0x7fb020009500 comp rx=0 tx=0).stop 2024-03-16T12:08:10.566 INFO:teuthology.orchestra.run.smithi029.stdout: "zabbix_port": { 2024-03-16T12:08:10.566 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "zabbix_port", 2024-03-16T12:08:10.566 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "int", 2024-03-16T12:08:10.566 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.566 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:10.567 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "10051", 2024-03-16T12:08:10.567 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.567 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.567 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:10.567 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.567 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.567 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.568 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.568 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.568 INFO:teuthology.orchestra.run.smithi029.stdout: "zabbix_sender": { 2024-03-16T12:08:10.568 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "zabbix_sender", 2024-03-16T12:08:10.568 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "str", 2024-03-16T12:08:10.568 INFO:teuthology.orchestra.run.smithi029.stdout: "level": "advanced", 2024-03-16T12:08:10.569 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 0, 2024-03-16T12:08:10.569 INFO:teuthology.orchestra.run.smithi029.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-03-16T12:08:10.569 INFO:teuthology.orchestra.run.smithi029.stdout: "min": "", 2024-03-16T12:08:10.569 INFO:teuthology.orchestra.run.smithi029.stdout: "max": "", 2024-03-16T12:08:10.569 INFO:teuthology.orchestra.run.smithi029.stdout: "enum_allowed": [], 2024-03-16T12:08:10.570 INFO:teuthology.orchestra.run.smithi029.stdout: "desc": "", 2024-03-16T12:08:10.570 INFO:teuthology.orchestra.run.smithi029.stdout: "long_desc": "", 2024-03-16T12:08:10.570 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-03-16T12:08:10.570 INFO:teuthology.orchestra.run.smithi029.stdout: "see_also": [] 2024-03-16T12:08:10.570 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:10.570 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:10.570 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:10.570 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:10.570 INFO:teuthology.orchestra.run.smithi029.stdout: "services": { 2024-03-16T12:08:10.571 INFO:teuthology.orchestra.run.smithi029.stdout: "dashboard": "https://172.21.15.29:8443/" 2024-03-16T12:08:10.571 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.571 INFO:teuthology.orchestra.run.smithi029.stdout: "always_on_modules": { 2024-03-16T12:08:10.571 INFO:teuthology.orchestra.run.smithi029.stdout: "octopus": [ 2024-03-16T12:08:10.571 INFO:teuthology.orchestra.run.smithi029.stdout: "balancer", 2024-03-16T12:08:10.571 INFO:teuthology.orchestra.run.smithi029.stdout: "crash", 2024-03-16T12:08:10.571 INFO:teuthology.orchestra.run.smithi029.stdout: "devicehealth", 2024-03-16T12:08:10.571 INFO:teuthology.orchestra.run.smithi029.stdout: "orchestrator", 2024-03-16T12:08:10.571 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_autoscaler", 2024-03-16T12:08:10.572 INFO:teuthology.orchestra.run.smithi029.stdout: "progress", 2024-03-16T12:08:10.572 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_support", 2024-03-16T12:08:10.572 INFO:teuthology.orchestra.run.smithi029.stdout: "status", 2024-03-16T12:08:10.572 INFO:teuthology.orchestra.run.smithi029.stdout: "telemetry", 2024-03-16T12:08:10.572 INFO:teuthology.orchestra.run.smithi029.stdout: "volumes" 2024-03-16T12:08:10.572 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:10.572 INFO:teuthology.orchestra.run.smithi029.stdout: "pacific": [ 2024-03-16T12:08:10.572 INFO:teuthology.orchestra.run.smithi029.stdout: "balancer", 2024-03-16T12:08:10.572 INFO:teuthology.orchestra.run.smithi029.stdout: "crash", 2024-03-16T12:08:10.572 INFO:teuthology.orchestra.run.smithi029.stdout: "devicehealth", 2024-03-16T12:08:10.573 INFO:teuthology.orchestra.run.smithi029.stdout: "orchestrator", 2024-03-16T12:08:10.573 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_autoscaler", 2024-03-16T12:08:10.573 INFO:teuthology.orchestra.run.smithi029.stdout: "progress", 2024-03-16T12:08:10.573 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_support", 2024-03-16T12:08:10.573 INFO:teuthology.orchestra.run.smithi029.stdout: "status", 2024-03-16T12:08:10.573 INFO:teuthology.orchestra.run.smithi029.stdout: "telemetry", 2024-03-16T12:08:10.573 INFO:teuthology.orchestra.run.smithi029.stdout: "volumes" 2024-03-16T12:08:10.573 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:10.574 INFO:teuthology.orchestra.run.smithi029.stdout: "quincy": [ 2024-03-16T12:08:10.574 INFO:teuthology.orchestra.run.smithi029.stdout: "balancer", 2024-03-16T12:08:10.574 INFO:teuthology.orchestra.run.smithi029.stdout: "crash", 2024-03-16T12:08:10.574 INFO:teuthology.orchestra.run.smithi029.stdout: "devicehealth", 2024-03-16T12:08:10.574 INFO:teuthology.orchestra.run.smithi029.stdout: "orchestrator", 2024-03-16T12:08:10.574 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_autoscaler", 2024-03-16T12:08:10.574 INFO:teuthology.orchestra.run.smithi029.stdout: "progress", 2024-03-16T12:08:10.574 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_support", 2024-03-16T12:08:10.574 INFO:teuthology.orchestra.run.smithi029.stdout: "status", 2024-03-16T12:08:10.575 INFO:teuthology.orchestra.run.smithi029.stdout: "telemetry", 2024-03-16T12:08:10.575 INFO:teuthology.orchestra.run.smithi029.stdout: "volumes" 2024-03-16T12:08:10.575 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:10.575 INFO:teuthology.orchestra.run.smithi029.stdout: "reef": [ 2024-03-16T12:08:10.575 INFO:teuthology.orchestra.run.smithi029.stdout: "balancer", 2024-03-16T12:08:10.575 INFO:teuthology.orchestra.run.smithi029.stdout: "crash", 2024-03-16T12:08:10.575 INFO:teuthology.orchestra.run.smithi029.stdout: "devicehealth", 2024-03-16T12:08:10.575 INFO:teuthology.orchestra.run.smithi029.stdout: "orchestrator", 2024-03-16T12:08:10.576 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_autoscaler", 2024-03-16T12:08:10.576 INFO:teuthology.orchestra.run.smithi029.stdout: "progress", 2024-03-16T12:08:10.576 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_support", 2024-03-16T12:08:10.576 INFO:teuthology.orchestra.run.smithi029.stdout: "status", 2024-03-16T12:08:10.576 INFO:teuthology.orchestra.run.smithi029.stdout: "telemetry", 2024-03-16T12:08:10.576 INFO:teuthology.orchestra.run.smithi029.stdout: "volumes" 2024-03-16T12:08:10.576 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-16T12:08:10.576 INFO:teuthology.orchestra.run.smithi029.stdout: "squid": [ 2024-03-16T12:08:10.576 INFO:teuthology.orchestra.run.smithi029.stdout: "balancer", 2024-03-16T12:08:10.577 INFO:teuthology.orchestra.run.smithi029.stdout: "crash", 2024-03-16T12:08:10.577 INFO:teuthology.orchestra.run.smithi029.stdout: "devicehealth", 2024-03-16T12:08:10.577 INFO:teuthology.orchestra.run.smithi029.stdout: "orchestrator", 2024-03-16T12:08:10.577 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_autoscaler", 2024-03-16T12:08:10.577 INFO:teuthology.orchestra.run.smithi029.stdout: "progress", 2024-03-16T12:08:10.577 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_support", 2024-03-16T12:08:10.577 INFO:teuthology.orchestra.run.smithi029.stdout: "status", 2024-03-16T12:08:10.577 INFO:teuthology.orchestra.run.smithi029.stdout: "telemetry", 2024-03-16T12:08:10.577 INFO:teuthology.orchestra.run.smithi029.stdout: "volumes" 2024-03-16T12:08:10.578 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-16T12:08:10.578 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.578 INFO:teuthology.orchestra.run.smithi029.stdout: "last_failure_osd_epoch": 88, 2024-03-16T12:08:10.578 INFO:teuthology.orchestra.run.smithi029.stdout: "active_clients": [ 2024-03-16T12:08:10.578 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-16T12:08:10.578 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "devicehealth", 2024-03-16T12:08:10.578 INFO:teuthology.orchestra.run.smithi029.stdout: "addrvec": [ 2024-03-16T12:08:10.579 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-16T12:08:10.579 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "v2", 2024-03-16T12:08:10.579 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": "172.21.15.29:0", 2024-03-16T12:08:10.579 INFO:teuthology.orchestra.run.smithi029.stdout: "nonce": 3692015184 2024-03-16T12:08:10.579 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:10.579 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-16T12:08:10.579 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.579 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-16T12:08:10.579 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "libcephsqlite", 2024-03-16T12:08:10.580 INFO:teuthology.orchestra.run.smithi029.stdout: "addrvec": [ 2024-03-16T12:08:10.580 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-16T12:08:10.580 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "v2", 2024-03-16T12:08:10.580 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": "172.21.15.29:0", 2024-03-16T12:08:10.580 INFO:teuthology.orchestra.run.smithi029.stdout: "nonce": 3228165685 2024-03-16T12:08:10.580 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:10.580 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-16T12:08:10.580 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.580 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-16T12:08:10.581 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "rbd_support", 2024-03-16T12:08:10.581 INFO:teuthology.orchestra.run.smithi029.stdout: "addrvec": [ 2024-03-16T12:08:10.581 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-16T12:08:10.581 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "v2", 2024-03-16T12:08:10.581 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": "172.21.15.29:0", 2024-03-16T12:08:10.581 INFO:teuthology.orchestra.run.smithi029.stdout: "nonce": 3019592432 2024-03-16T12:08:10.581 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:10.581 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-16T12:08:10.581 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.582 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-16T12:08:10.582 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "snap_schedule", 2024-03-16T12:08:10.582 INFO:teuthology.orchestra.run.smithi029.stdout: "addrvec": [ 2024-03-16T12:08:10.582 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-16T12:08:10.582 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "v2", 2024-03-16T12:08:10.582 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": "172.21.15.29:0", 2024-03-16T12:08:10.582 INFO:teuthology.orchestra.run.smithi029.stdout: "nonce": 902942105 2024-03-16T12:08:10.582 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:10.582 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-16T12:08:10.583 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.583 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-16T12:08:10.583 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "snap_schedule", 2024-03-16T12:08:10.583 INFO:teuthology.orchestra.run.smithi029.stdout: "addrvec": [ 2024-03-16T12:08:10.583 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-16T12:08:10.583 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "v2", 2024-03-16T12:08:10.583 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": "172.21.15.29:0", 2024-03-16T12:08:10.583 INFO:teuthology.orchestra.run.smithi029.stdout: "nonce": 3160373209 2024-03-16T12:08:10.583 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:10.583 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-16T12:08:10.584 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:10.584 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-16T12:08:10.584 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "volumes", 2024-03-16T12:08:10.584 INFO:teuthology.orchestra.run.smithi029.stdout: "addrvec": [ 2024-03-16T12:08:10.584 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-16T12:08:10.584 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "v2", 2024-03-16T12:08:10.584 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": "172.21.15.29:0", 2024-03-16T12:08:10.584 INFO:teuthology.orchestra.run.smithi029.stdout: "nonce": 3258542700 2024-03-16T12:08:10.584 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:10.584 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-16T12:08:10.585 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:10.585 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-16T12:08:10.585 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-03-16T12:08:10.586 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:08:08 smithi152 ceph-mon[36291]: pgmap v545: 97 pgs: 97 active+clean; 365 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 4.9 MiB/s wr, 419 op/s 2024-03-16T12:08:10.586 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:08:08 smithi152 ceph-mon[36291]: from='client.? 172.21.15.29:0/3454781204' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-16T12:08:10.586 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:08:08 smithi152 ceph-mon[36291]: from='client.? 172.21.15.29:0/1190574999' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-16T12:08:10.586 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:08:08 smithi152 ceph-mon[36291]: from='client.? 172.21.15.29:0/2899694199' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-03-16T12:08:10.586 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:08:08 smithi154 ceph-mon[36320]: pgmap v545: 97 pgs: 97 active+clean; 365 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 4.9 MiB/s wr, 419 op/s 2024-03-16T12:08:10.586 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:08:08 smithi154 ceph-mon[36320]: from='client.? 172.21.15.29:0/3454781204' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-16T12:08:10.586 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:08:08 smithi154 ceph-mon[36320]: from='client.? 172.21.15.29:0/1190574999' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-16T12:08:10.587 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:08:08 smithi154 ceph-mon[36320]: from='client.? 172.21.15.29:0/2899694199' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-03-16T12:08:10.587 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:08.270+0000 7fb02ef7e640 1 -- 172.21.15.29:0/2899694199 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb0281066b0 msgr2=0x7fb02819b860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:08:10.588 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:08.270+0000 7fb02ef7e640 1 --2- 172.21.15.29:0/2899694199 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb0281066b0 0x7fb02819b860 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7fb018005940 tx=0x7fb01802cb30 comp rx=0 tx=0).stop 2024-03-16T12:08:10.588 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:08.272+0000 7fb02ef7e640 1 -- 172.21.15.29:0/2899694199 shutdown_connections 2024-03-16T12:08:10.588 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:08.272+0000 7fb02ef7e640 1 --2- 172.21.15.29:0/2899694199 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7fb0100a2970 0x7fb0100a4e30 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:08:10.588 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:08.272+0000 7fb02ef7e640 1 --2- 172.21.15.29:0/2899694199 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb0281066b0 0x7fb02819b860 unknown :-1 s=CLOSED pgs=798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:08:10.588 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:08.272+0000 7fb02ef7e640 1 --2- 172.21.15.29:0/2899694199 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb028101060 0x7fb0281974e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:08:10.588 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:08.272+0000 7fb02ef7e640 1 --2- 172.21.15.29:0/2899694199 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb0281006a0 0x7fb028196fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:08:10.588 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:08.272+0000 7fb02ef7e640 1 -- 172.21.15.29:0/2899694199 >> 172.21.15.29:0/2899694199 conn(0x7fb028077780 msgr2=0x7fb0280fe6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:08:10.588 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:08.272+0000 7fb02ef7e640 1 -- 172.21.15.29:0/2899694199 shutdown_connections 2024-03-16T12:08:10.588 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:08.272+0000 7fb02ef7e640 1 -- 172.21.15.29:0/2899694199 wait complete. 2024-03-16T12:08:10.595 DEBUG:tasks.check_counter:Getting stats from a 2024-03-16T12:08:10.595 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid d1887f28-e389-11ee-95c9-87774f69a715 -- ceph daemon mds.a perf dump 2024-03-16T12:08:10.595 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:08:08 smithi029 ceph-mon[31975]: pgmap v545: 97 pgs: 97 active+clean; 365 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 4.9 MiB/s wr, 419 op/s 2024-03-16T12:08:10.595 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:08:08 smithi029 ceph-mon[31975]: from='client.? 172.21.15.29:0/3454781204' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-16T12:08:10.596 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:08:08 smithi029 ceph-mon[31975]: from='client.? 172.21.15.29:0/1190574999' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-16T12:08:10.596 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:08:08 smithi029 ceph-mon[31975]: from='client.? 172.21.15.29:0/2899694199' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-03-16T12:08:10.786 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:08:10 smithi154 ceph-mon[36320]: pgmap v546: 97 pgs: 97 active+clean; 365 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.6 MiB/s wr, 329 op/s 2024-03-16T12:08:10.786 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:08:10 smithi154 ceph-mon[36320]: osdmap e115: 12 total, 12 up, 12 in 2024-03-16T12:08:10.829 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:08:10 smithi029 ceph-mon[31975]: pgmap v546: 97 pgs: 97 active+clean; 365 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.6 MiB/s wr, 329 op/s 2024-03-16T12:08:10.829 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:08:10 smithi029 ceph-mon[31975]: osdmap e115: 12 total, 12 up, 12 in 2024-03-16T12:08:10.868 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:08:10 smithi152 ceph-mon[36291]: pgmap v546: 97 pgs: 97 active+clean; 365 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.6 MiB/s wr, 329 op/s 2024-03-16T12:08:10.868 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:08:10 smithi152 ceph-mon[36291]: osdmap e115: 12 total, 12 up, 12 in 2024-03-16T12:08:10.984 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/d1887f28-e389-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T12:08:12.786 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:08:12 smithi154 ceph-mon[36320]: pgmap v548: 97 pgs: 6 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 87 active+clean; 364 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.9 MiB/s wr, 327 op/s 2024-03-16T12:08:12.829 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:08:12 smithi029 ceph-mon[31975]: pgmap v548: 97 pgs: 6 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 87 active+clean; 364 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.9 MiB/s wr, 327 op/s 2024-03-16T12:08:12.868 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:08:12 smithi152 ceph-mon[36291]: pgmap v548: 97 pgs: 6 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 87 active+clean; 364 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.9 MiB/s wr, 327 op/s 2024-03-16T12:08:13.673 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-03-16T12:08:13.673 INFO:teuthology.orchestra.run.smithi029.stdout: "AsyncMessenger::Worker-0": { 2024-03-16T12:08:13.674 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_recv_messages": 324, 2024-03-16T12:08:13.674 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_send_messages": 314, 2024-03-16T12:08:13.674 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_recv_bytes": 1466832, 2024-03-16T12:08:13.674 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_send_bytes": 194320, 2024-03-16T12:08:13.674 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_created_connections": 4, 2024-03-16T12:08:13.674 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_active_connections": 0, 2024-03-16T12:08:13.674 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_running_total_time": 0.049476483, 2024-03-16T12:08:13.674 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_running_send_time": 0.017001597, 2024-03-16T12:08:13.675 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_running_recv_time": 0.034087052, 2024-03-16T12:08:13.675 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_running_fast_dispatch_time": 0.005444439, 2024-03-16T12:08:13.675 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_send_messages_queue_lat": { 2024-03-16T12:08:13.675 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 313, 2024-03-16T12:08:13.675 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.011506970, 2024-03-16T12:08:13.675 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000036763 2024-03-16T12:08:13.675 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:13.675 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_handle_ack_lat": { 2024-03-16T12:08:13.675 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-03-16T12:08:13.676 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-03-16T12:08:13.676 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-03-16T12:08:13.676 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:13.676 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_recv_encrypted_bytes": 1466832, 2024-03-16T12:08:13.676 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_send_encrypted_bytes": 194320 2024-03-16T12:08:13.676 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:13.676 INFO:teuthology.orchestra.run.smithi029.stdout: "AsyncMessenger::Worker-1": { 2024-03-16T12:08:13.676 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_recv_messages": 7, 2024-03-16T12:08:13.676 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_send_messages": 244, 2024-03-16T12:08:13.677 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_recv_bytes": 3552, 2024-03-16T12:08:13.677 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_send_bytes": 276848, 2024-03-16T12:08:13.677 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_created_connections": 6, 2024-03-16T12:08:13.677 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_active_connections": 3, 2024-03-16T12:08:13.677 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_running_total_time": 0.016394152, 2024-03-16T12:08:13.677 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_running_send_time": 0.010979490, 2024-03-16T12:08:13.677 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_running_recv_time": 0.002517165, 2024-03-16T12:08:13.677 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-16T12:08:13.678 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_send_messages_queue_lat": { 2024-03-16T12:08:13.678 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 242, 2024-03-16T12:08:13.678 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.013860101, 2024-03-16T12:08:13.678 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000057273 2024-03-16T12:08:13.678 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:13.678 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_handle_ack_lat": { 2024-03-16T12:08:13.679 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-03-16T12:08:13.679 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-03-16T12:08:13.679 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-03-16T12:08:13.679 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:13.679 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_recv_encrypted_bytes": 3552, 2024-03-16T12:08:13.679 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_send_encrypted_bytes": 276848 2024-03-16T12:08:13.680 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:13.680 INFO:teuthology.orchestra.run.smithi029.stdout: "AsyncMessenger::Worker-2": { 2024-03-16T12:08:13.680 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_recv_messages": 0, 2024-03-16T12:08:13.680 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_send_messages": 2, 2024-03-16T12:08:13.680 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_recv_bytes": 0, 2024-03-16T12:08:13.680 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_send_bytes": 0, 2024-03-16T12:08:13.680 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_created_connections": 2, 2024-03-16T12:08:13.681 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_active_connections": 1, 2024-03-16T12:08:13.681 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_running_total_time": 0.000295079, 2024-03-16T12:08:13.681 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_running_send_time": 0.000000000, 2024-03-16T12:08:13.681 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_running_recv_time": 0.000217522, 2024-03-16T12:08:13.681 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-16T12:08:13.681 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_send_messages_queue_lat": { 2024-03-16T12:08:13.681 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-03-16T12:08:13.682 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-03-16T12:08:13.682 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-03-16T12:08:13.682 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:13.682 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_handle_ack_lat": { 2024-03-16T12:08:13.682 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-03-16T12:08:13.682 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-03-16T12:08:13.683 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-03-16T12:08:13.683 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:13.683 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-16T12:08:13.683 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_send_encrypted_bytes": 0 2024-03-16T12:08:13.683 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:13.683 INFO:teuthology.orchestra.run.smithi029.stdout: "cct": { 2024-03-16T12:08:13.683 INFO:teuthology.orchestra.run.smithi029.stdout: "total_workers": 0, 2024-03-16T12:08:13.684 INFO:teuthology.orchestra.run.smithi029.stdout: "unhealthy_workers": 0 2024-03-16T12:08:13.684 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:13.684 INFO:teuthology.orchestra.run.smithi029.stdout: "mempool": { 2024-03-16T12:08:13.684 INFO:teuthology.orchestra.run.smithi029.stdout: "bloom_filter_bytes": 0, 2024-03-16T12:08:13.684 INFO:teuthology.orchestra.run.smithi029.stdout: "bloom_filter_items": 0, 2024-03-16T12:08:13.684 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_alloc_bytes": 0, 2024-03-16T12:08:13.684 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_alloc_items": 0, 2024-03-16T12:08:13.684 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_cache_data_bytes": 0, 2024-03-16T12:08:13.685 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_cache_data_items": 0, 2024-03-16T12:08:13.685 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-16T12:08:13.685 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_cache_onode_items": 0, 2024-03-16T12:08:13.685 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-16T12:08:13.685 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_cache_meta_items": 0, 2024-03-16T12:08:13.685 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_cache_other_bytes": 0, 2024-03-16T12:08:13.685 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_cache_other_items": 0, 2024-03-16T12:08:13.686 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-16T12:08:13.686 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_cache_buffer_items": 0, 2024-03-16T12:08:13.686 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_extent_bytes": 0, 2024-03-16T12:08:13.686 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_extent_items": 0, 2024-03-16T12:08:13.686 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_blob_bytes": 0, 2024-03-16T12:08:13.686 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_blob_items": 0, 2024-03-16T12:08:13.687 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-16T12:08:13.687 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_shared_blob_items": 0, 2024-03-16T12:08:13.687 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-16T12:08:13.687 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_inline_bl_items": 0, 2024-03-16T12:08:13.687 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_fsck_bytes": 0, 2024-03-16T12:08:13.687 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_fsck_items": 0, 2024-03-16T12:08:13.687 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_txc_bytes": 0, 2024-03-16T12:08:13.687 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_txc_items": 0, 2024-03-16T12:08:13.688 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-16T12:08:13.688 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_writing_deferred_items": 0, 2024-03-16T12:08:13.688 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_writing_bytes": 0, 2024-03-16T12:08:13.688 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_writing_items": 0, 2024-03-16T12:08:13.688 INFO:teuthology.orchestra.run.smithi029.stdout: "bluefs_bytes": 0, 2024-03-16T12:08:13.688 INFO:teuthology.orchestra.run.smithi029.stdout: "bluefs_items": 0, 2024-03-16T12:08:13.688 INFO:teuthology.orchestra.run.smithi029.stdout: "bluefs_file_reader_bytes": 0, 2024-03-16T12:08:13.688 INFO:teuthology.orchestra.run.smithi029.stdout: "bluefs_file_reader_items": 0, 2024-03-16T12:08:13.688 INFO:teuthology.orchestra.run.smithi029.stdout: "bluefs_file_writer_bytes": 0, 2024-03-16T12:08:13.689 INFO:teuthology.orchestra.run.smithi029.stdout: "bluefs_file_writer_items": 0, 2024-03-16T12:08:13.689 INFO:teuthology.orchestra.run.smithi029.stdout: "buffer_anon_bytes": 148754, 2024-03-16T12:08:13.689 INFO:teuthology.orchestra.run.smithi029.stdout: "buffer_anon_items": 31, 2024-03-16T12:08:13.689 INFO:teuthology.orchestra.run.smithi029.stdout: "buffer_meta_bytes": 0, 2024-03-16T12:08:13.689 INFO:teuthology.orchestra.run.smithi029.stdout: "buffer_meta_items": 0, 2024-03-16T12:08:13.689 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_bytes": 0, 2024-03-16T12:08:13.689 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_items": 0, 2024-03-16T12:08:13.689 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_mapbl_bytes": 0, 2024-03-16T12:08:13.689 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_mapbl_items": 0, 2024-03-16T12:08:13.690 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_pglog_bytes": 0, 2024-03-16T12:08:13.690 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_pglog_items": 0, 2024-03-16T12:08:13.690 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_bytes": 0, 2024-03-16T12:08:13.690 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_items": 0, 2024-03-16T12:08:13.690 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_mapping_bytes": 0, 2024-03-16T12:08:13.690 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_mapping_items": 0, 2024-03-16T12:08:13.690 INFO:teuthology.orchestra.run.smithi029.stdout: "pgmap_bytes": 0, 2024-03-16T12:08:13.690 INFO:teuthology.orchestra.run.smithi029.stdout: "pgmap_items": 0, 2024-03-16T12:08:13.690 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_co_bytes": 928, 2024-03-16T12:08:13.691 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_co_items": 2, 2024-03-16T12:08:13.691 INFO:teuthology.orchestra.run.smithi029.stdout: "unittest_1_bytes": 0, 2024-03-16T12:08:13.691 INFO:teuthology.orchestra.run.smithi029.stdout: "unittest_1_items": 0, 2024-03-16T12:08:13.691 INFO:teuthology.orchestra.run.smithi029.stdout: "unittest_2_bytes": 0, 2024-03-16T12:08:13.691 INFO:teuthology.orchestra.run.smithi029.stdout: "unittest_2_items": 0 2024-03-16T12:08:13.691 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:13.691 INFO:teuthology.orchestra.run.smithi029.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-16T12:08:13.691 INFO:teuthology.orchestra.run.smithi029.stdout: "val": 0, 2024-03-16T12:08:13.691 INFO:teuthology.orchestra.run.smithi029.stdout: "max": 104857600, 2024-03-16T12:08:13.692 INFO:teuthology.orchestra.run.smithi029.stdout: "get_started": 0, 2024-03-16T12:08:13.692 INFO:teuthology.orchestra.run.smithi029.stdout: "get": 327, 2024-03-16T12:08:13.692 INFO:teuthology.orchestra.run.smithi029.stdout: "get_sum": 1421124, 2024-03-16T12:08:13.692 INFO:teuthology.orchestra.run.smithi029.stdout: "get_or_fail_fail": 0, 2024-03-16T12:08:13.692 INFO:teuthology.orchestra.run.smithi029.stdout: "get_or_fail_success": 327, 2024-03-16T12:08:13.692 INFO:teuthology.orchestra.run.smithi029.stdout: "take": 0, 2024-03-16T12:08:13.692 INFO:teuthology.orchestra.run.smithi029.stdout: "take_sum": 0, 2024-03-16T12:08:13.692 INFO:teuthology.orchestra.run.smithi029.stdout: "put": 327, 2024-03-16T12:08:13.692 INFO:teuthology.orchestra.run.smithi029.stdout: "put_sum": 1421124, 2024-03-16T12:08:13.692 INFO:teuthology.orchestra.run.smithi029.stdout: "wait": { 2024-03-16T12:08:13.693 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-03-16T12:08:13.693 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-03-16T12:08:13.693 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-03-16T12:08:13.693 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:13.693 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:13.693 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-03-16T12:08:14.701 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:08:14 smithi029 ceph-mon[31975]: pgmap v549: 97 pgs: 40 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 35 active+clean; 361 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 707 KiB/s rd, 2.5 MiB/s wr, 143 op/s 2024-03-16T12:08:14.727 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.a 2024-03-16T12:08:14.728 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.a 2024-03-16T12:08:14.728 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.a 2024-03-16T12:08:14.728 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.a 2024-03-16T12:08:14.728 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.a 2024-03-16T12:08:14.728 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.a 2024-03-16T12:08:14.728 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.a 2024-03-16T12:08:14.729 DEBUG:tasks.check_counter:Getting stats from d 2024-03-16T12:08:14.729 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid d1887f28-e389-11ee-95c9-87774f69a715 -- ceph daemon mds.d perf dump 2024-03-16T12:08:14.785 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:08:14 smithi154 ceph-mon[36320]: pgmap v549: 97 pgs: 40 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 35 active+clean; 361 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 707 KiB/s rd, 2.5 MiB/s wr, 143 op/s 2024-03-16T12:08:14.868 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:08:14 smithi152 ceph-mon[36291]: pgmap v549: 97 pgs: 40 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 35 active+clean; 361 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 707 KiB/s rd, 2.5 MiB/s wr, 143 op/s 2024-03-16T12:08:14.977 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/d1887f28-e389-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T12:08:16.786 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:08:16 smithi154 ceph-mon[36320]: pgmap v550: 97 pgs: 40 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 35 active+clean; 361 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 707 KiB/s rd, 2.5 MiB/s wr, 143 op/s 2024-03-16T12:08:16.828 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:08:16 smithi029 ceph-mon[31975]: pgmap v550: 97 pgs: 40 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 35 active+clean; 361 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 707 KiB/s rd, 2.5 MiB/s wr, 143 op/s 2024-03-16T12:08:16.868 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:08:16 smithi152 ceph-mon[36291]: pgmap v550: 97 pgs: 40 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 35 active+clean; 361 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 707 KiB/s rd, 2.5 MiB/s wr, 143 op/s 2024-03-16T12:08:18.199 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-03-16T12:08:18.200 INFO:teuthology.orchestra.run.smithi029.stdout: "AsyncMessenger::Worker-0": { 2024-03-16T12:08:18.200 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_recv_messages": 4, 2024-03-16T12:08:18.200 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_send_messages": 4, 2024-03-16T12:08:18.200 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_recv_bytes": 3072, 2024-03-16T12:08:18.200 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_send_bytes": 448, 2024-03-16T12:08:18.201 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_created_connections": 4, 2024-03-16T12:08:18.201 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_active_connections": 0, 2024-03-16T12:08:18.201 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_running_total_time": 0.001547074, 2024-03-16T12:08:18.201 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_running_send_time": 0.000039199, 2024-03-16T12:08:18.201 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_running_recv_time": 0.000675972, 2024-03-16T12:08:18.201 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-16T12:08:18.201 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_send_messages_queue_lat": { 2024-03-16T12:08:18.201 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 3, 2024-03-16T12:08:18.201 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000852094, 2024-03-16T12:08:18.201 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000284031 2024-03-16T12:08:18.202 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:18.202 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_handle_ack_lat": { 2024-03-16T12:08:18.202 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-03-16T12:08:18.202 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-03-16T12:08:18.202 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-03-16T12:08:18.202 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:18.202 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_recv_encrypted_bytes": 3072, 2024-03-16T12:08:18.203 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_send_encrypted_bytes": 448 2024-03-16T12:08:18.203 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:18.203 INFO:teuthology.orchestra.run.smithi029.stdout: "AsyncMessenger::Worker-1": { 2024-03-16T12:08:18.203 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_recv_messages": 331, 2024-03-16T12:08:18.203 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_send_messages": 321, 2024-03-16T12:08:18.203 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_recv_bytes": 1468800, 2024-03-16T12:08:18.203 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_send_bytes": 198320, 2024-03-16T12:08:18.203 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_created_connections": 2, 2024-03-16T12:08:18.204 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_active_connections": 1, 2024-03-16T12:08:18.204 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_running_total_time": 0.052863020, 2024-03-16T12:08:18.204 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_running_send_time": 0.017607198, 2024-03-16T12:08:18.204 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_running_recv_time": 0.037561887, 2024-03-16T12:08:18.204 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_running_fast_dispatch_time": 0.005477116, 2024-03-16T12:08:18.204 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_send_messages_queue_lat": { 2024-03-16T12:08:18.204 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 320, 2024-03-16T12:08:18.204 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.014698702, 2024-03-16T12:08:18.204 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000045933 2024-03-16T12:08:18.205 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:18.205 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_handle_ack_lat": { 2024-03-16T12:08:18.205 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-03-16T12:08:18.205 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-03-16T12:08:18.205 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-03-16T12:08:18.205 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:18.205 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_recv_encrypted_bytes": 1468800, 2024-03-16T12:08:18.205 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_send_encrypted_bytes": 198320 2024-03-16T12:08:18.205 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:18.205 INFO:teuthology.orchestra.run.smithi029.stdout: "AsyncMessenger::Worker-2": { 2024-03-16T12:08:18.206 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_recv_messages": 3, 2024-03-16T12:08:18.206 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_send_messages": 247, 2024-03-16T12:08:18.206 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_recv_bytes": 480, 2024-03-16T12:08:18.206 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_send_bytes": 277280, 2024-03-16T12:08:18.206 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_created_connections": 6, 2024-03-16T12:08:18.206 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_active_connections": 3, 2024-03-16T12:08:18.206 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_running_total_time": 0.017140737, 2024-03-16T12:08:18.206 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_running_send_time": 0.012106133, 2024-03-16T12:08:18.206 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_running_recv_time": 0.002045462, 2024-03-16T12:08:18.207 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-16T12:08:18.207 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_send_messages_queue_lat": { 2024-03-16T12:08:18.207 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 244, 2024-03-16T12:08:18.207 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.017624709, 2024-03-16T12:08:18.207 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000072232 2024-03-16T12:08:18.207 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:18.207 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_handle_ack_lat": { 2024-03-16T12:08:18.207 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-03-16T12:08:18.207 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-03-16T12:08:18.208 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-03-16T12:08:18.208 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:18.208 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_recv_encrypted_bytes": 480, 2024-03-16T12:08:18.208 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_send_encrypted_bytes": 277280 2024-03-16T12:08:18.208 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:18.208 INFO:teuthology.orchestra.run.smithi029.stdout: "cct": { 2024-03-16T12:08:18.208 INFO:teuthology.orchestra.run.smithi029.stdout: "total_workers": 0, 2024-03-16T12:08:18.208 INFO:teuthology.orchestra.run.smithi029.stdout: "unhealthy_workers": 0 2024-03-16T12:08:18.208 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:18.209 INFO:teuthology.orchestra.run.smithi029.stdout: "mempool": { 2024-03-16T12:08:18.209 INFO:teuthology.orchestra.run.smithi029.stdout: "bloom_filter_bytes": 0, 2024-03-16T12:08:18.209 INFO:teuthology.orchestra.run.smithi029.stdout: "bloom_filter_items": 0, 2024-03-16T12:08:18.209 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_alloc_bytes": 0, 2024-03-16T12:08:18.209 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_alloc_items": 0, 2024-03-16T12:08:18.209 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_cache_data_bytes": 0, 2024-03-16T12:08:18.209 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_cache_data_items": 0, 2024-03-16T12:08:18.209 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-16T12:08:18.209 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_cache_onode_items": 0, 2024-03-16T12:08:18.210 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-16T12:08:18.210 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_cache_meta_items": 0, 2024-03-16T12:08:18.210 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_cache_other_bytes": 0, 2024-03-16T12:08:18.210 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_cache_other_items": 0, 2024-03-16T12:08:18.210 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-16T12:08:18.210 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_cache_buffer_items": 0, 2024-03-16T12:08:18.210 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_extent_bytes": 0, 2024-03-16T12:08:18.210 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_extent_items": 0, 2024-03-16T12:08:18.210 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_blob_bytes": 0, 2024-03-16T12:08:18.211 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_blob_items": 0, 2024-03-16T12:08:18.211 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-16T12:08:18.211 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_shared_blob_items": 0, 2024-03-16T12:08:18.211 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-16T12:08:18.211 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_inline_bl_items": 0, 2024-03-16T12:08:18.211 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_fsck_bytes": 0, 2024-03-16T12:08:18.211 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_fsck_items": 0, 2024-03-16T12:08:18.213 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_txc_bytes": 0, 2024-03-16T12:08:18.213 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_txc_items": 0, 2024-03-16T12:08:18.213 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-16T12:08:18.213 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_writing_deferred_items": 0, 2024-03-16T12:08:18.213 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_writing_bytes": 0, 2024-03-16T12:08:18.213 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_writing_items": 0, 2024-03-16T12:08:18.214 INFO:teuthology.orchestra.run.smithi029.stdout: "bluefs_bytes": 0, 2024-03-16T12:08:18.214 INFO:teuthology.orchestra.run.smithi029.stdout: "bluefs_items": 0, 2024-03-16T12:08:18.214 INFO:teuthology.orchestra.run.smithi029.stdout: "bluefs_file_reader_bytes": 0, 2024-03-16T12:08:18.214 INFO:teuthology.orchestra.run.smithi029.stdout: "bluefs_file_reader_items": 0, 2024-03-16T12:08:18.214 INFO:teuthology.orchestra.run.smithi029.stdout: "bluefs_file_writer_bytes": 0, 2024-03-16T12:08:18.214 INFO:teuthology.orchestra.run.smithi029.stdout: "bluefs_file_writer_items": 0, 2024-03-16T12:08:18.214 INFO:teuthology.orchestra.run.smithi029.stdout: "buffer_anon_bytes": 148731, 2024-03-16T12:08:18.214 INFO:teuthology.orchestra.run.smithi029.stdout: "buffer_anon_items": 31, 2024-03-16T12:08:18.215 INFO:teuthology.orchestra.run.smithi029.stdout: "buffer_meta_bytes": 0, 2024-03-16T12:08:18.215 INFO:teuthology.orchestra.run.smithi029.stdout: "buffer_meta_items": 0, 2024-03-16T12:08:18.215 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_bytes": 0, 2024-03-16T12:08:18.215 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_items": 0, 2024-03-16T12:08:18.215 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_mapbl_bytes": 0, 2024-03-16T12:08:18.215 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_mapbl_items": 0, 2024-03-16T12:08:18.215 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_pglog_bytes": 0, 2024-03-16T12:08:18.215 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_pglog_items": 0, 2024-03-16T12:08:18.216 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_bytes": 0, 2024-03-16T12:08:18.216 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_items": 0, 2024-03-16T12:08:18.216 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_mapping_bytes": 0, 2024-03-16T12:08:18.216 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_mapping_items": 0, 2024-03-16T12:08:18.216 INFO:teuthology.orchestra.run.smithi029.stdout: "pgmap_bytes": 0, 2024-03-16T12:08:18.216 INFO:teuthology.orchestra.run.smithi029.stdout: "pgmap_items": 0, 2024-03-16T12:08:18.216 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_co_bytes": 928, 2024-03-16T12:08:18.216 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_co_items": 2, 2024-03-16T12:08:18.216 INFO:teuthology.orchestra.run.smithi029.stdout: "unittest_1_bytes": 0, 2024-03-16T12:08:18.217 INFO:teuthology.orchestra.run.smithi029.stdout: "unittest_1_items": 0, 2024-03-16T12:08:18.217 INFO:teuthology.orchestra.run.smithi029.stdout: "unittest_2_bytes": 0, 2024-03-16T12:08:18.217 INFO:teuthology.orchestra.run.smithi029.stdout: "unittest_2_items": 0 2024-03-16T12:08:18.217 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:18.217 INFO:teuthology.orchestra.run.smithi029.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-16T12:08:18.217 INFO:teuthology.orchestra.run.smithi029.stdout: "val": 0, 2024-03-16T12:08:18.217 INFO:teuthology.orchestra.run.smithi029.stdout: "max": 104857600, 2024-03-16T12:08:18.217 INFO:teuthology.orchestra.run.smithi029.stdout: "get_started": 0, 2024-03-16T12:08:18.217 INFO:teuthology.orchestra.run.smithi029.stdout: "get": 334, 2024-03-16T12:08:18.217 INFO:teuthology.orchestra.run.smithi029.stdout: "get_sum": 1422098, 2024-03-16T12:08:18.218 INFO:teuthology.orchestra.run.smithi029.stdout: "get_or_fail_fail": 0, 2024-03-16T12:08:18.218 INFO:teuthology.orchestra.run.smithi029.stdout: "get_or_fail_success": 334, 2024-03-16T12:08:18.218 INFO:teuthology.orchestra.run.smithi029.stdout: "take": 0, 2024-03-16T12:08:18.218 INFO:teuthology.orchestra.run.smithi029.stdout: "take_sum": 0, 2024-03-16T12:08:18.218 INFO:teuthology.orchestra.run.smithi029.stdout: "put": 334, 2024-03-16T12:08:18.218 INFO:teuthology.orchestra.run.smithi029.stdout: "put_sum": 1422098, 2024-03-16T12:08:18.218 INFO:teuthology.orchestra.run.smithi029.stdout: "wait": { 2024-03-16T12:08:18.218 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-03-16T12:08:18.218 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-03-16T12:08:18.219 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-03-16T12:08:18.219 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:18.219 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:18.219 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-03-16T12:08:18.786 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:08:18 smithi154 ceph-mon[36320]: pgmap v551: 97 pgs: 97 active+clean; 359 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 256 KiB/s rd, 3.1 MiB/s wr, 111 op/s 2024-03-16T12:08:18.829 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:08:18 smithi029 ceph-mon[31975]: pgmap v551: 97 pgs: 97 active+clean; 359 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 256 KiB/s rd, 3.1 MiB/s wr, 111 op/s 2024-03-16T12:08:18.868 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:08:18 smithi152 ceph-mon[36291]: pgmap v551: 97 pgs: 97 active+clean; 359 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 256 KiB/s rd, 3.1 MiB/s wr, 111 op/s 2024-03-16T12:08:18.931 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.d 2024-03-16T12:08:18.931 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.d 2024-03-16T12:08:18.932 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.d 2024-03-16T12:08:18.932 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.d 2024-03-16T12:08:18.932 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.d 2024-03-16T12:08:18.932 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.d 2024-03-16T12:08:18.932 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.d 2024-03-16T12:08:18.932 DEBUG:tasks.check_counter:Getting stats from g 2024-03-16T12:08:18.932 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid d1887f28-e389-11ee-95c9-87774f69a715 -- ceph daemon mds.g perf dump 2024-03-16T12:08:19.182 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/d1887f28-e389-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T12:08:20.786 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:08:20 smithi154 ceph-mon[36320]: pgmap v552: 97 pgs: 97 active+clean; 359 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 256 KiB/s rd, 3.1 MiB/s wr, 111 op/s 2024-03-16T12:08:20.828 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:08:20 smithi029 ceph-mon[31975]: pgmap v552: 97 pgs: 97 active+clean; 359 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 256 KiB/s rd, 3.1 MiB/s wr, 111 op/s 2024-03-16T12:08:20.868 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:08:20 smithi152 ceph-mon[36291]: pgmap v552: 97 pgs: 97 active+clean; 359 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 256 KiB/s rd, 3.1 MiB/s wr, 111 op/s 2024-03-16T12:08:21.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:08:21 smithi029 ceph-mon[31975]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T12:08:21.785 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:08:21 smithi154 ceph-mon[36320]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T12:08:21.821 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-03-16T12:08:21.821 INFO:teuthology.orchestra.run.smithi029.stdout: "AsyncMessenger::Worker-0": { 2024-03-16T12:08:21.821 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_recv_messages": 331, 2024-03-16T12:08:21.821 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_send_messages": 319, 2024-03-16T12:08:21.821 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_recv_bytes": 1470720, 2024-03-16T12:08:21.821 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_send_bytes": 196688, 2024-03-16T12:08:21.822 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_created_connections": 4, 2024-03-16T12:08:21.822 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_active_connections": 0, 2024-03-16T12:08:21.822 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_running_total_time": 0.050594386, 2024-03-16T12:08:21.822 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_running_send_time": 0.016821033, 2024-03-16T12:08:21.822 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_running_recv_time": 0.035188658, 2024-03-16T12:08:21.822 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_running_fast_dispatch_time": 0.005495720, 2024-03-16T12:08:21.822 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_send_messages_queue_lat": { 2024-03-16T12:08:21.823 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 319, 2024-03-16T12:08:21.823 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.011821862, 2024-03-16T12:08:21.823 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000037059 2024-03-16T12:08:21.823 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:21.823 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_handle_ack_lat": { 2024-03-16T12:08:21.823 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-03-16T12:08:21.823 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-03-16T12:08:21.823 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-03-16T12:08:21.824 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:21.824 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_recv_encrypted_bytes": 1470720, 2024-03-16T12:08:21.824 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_send_encrypted_bytes": 196688 2024-03-16T12:08:21.824 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:21.824 INFO:teuthology.orchestra.run.smithi029.stdout: "AsyncMessenger::Worker-1": { 2024-03-16T12:08:21.824 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_recv_messages": 3, 2024-03-16T12:08:21.824 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_send_messages": 244, 2024-03-16T12:08:21.824 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_recv_bytes": 480, 2024-03-16T12:08:21.824 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_send_bytes": 276752, 2024-03-16T12:08:21.825 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_created_connections": 6, 2024-03-16T12:08:21.825 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_active_connections": 3, 2024-03-16T12:08:21.825 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_running_total_time": 0.016748545, 2024-03-16T12:08:21.825 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_running_send_time": 0.011451688, 2024-03-16T12:08:21.825 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_running_recv_time": 0.002511902, 2024-03-16T12:08:21.825 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-16T12:08:21.825 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_send_messages_queue_lat": { 2024-03-16T12:08:21.825 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 241, 2024-03-16T12:08:21.826 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.016753955, 2024-03-16T12:08:21.826 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000069518 2024-03-16T12:08:21.826 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:21.826 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_handle_ack_lat": { 2024-03-16T12:08:21.826 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-03-16T12:08:21.826 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-03-16T12:08:21.826 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-03-16T12:08:21.826 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:21.826 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_recv_encrypted_bytes": 480, 2024-03-16T12:08:21.827 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_send_encrypted_bytes": 276752 2024-03-16T12:08:21.827 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:21.827 INFO:teuthology.orchestra.run.smithi029.stdout: "AsyncMessenger::Worker-2": { 2024-03-16T12:08:21.827 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_recv_messages": 0, 2024-03-16T12:08:21.827 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_send_messages": 2, 2024-03-16T12:08:21.827 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_recv_bytes": 0, 2024-03-16T12:08:21.827 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_send_bytes": 0, 2024-03-16T12:08:21.828 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_created_connections": 2, 2024-03-16T12:08:21.828 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_active_connections": 1, 2024-03-16T12:08:21.828 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_running_total_time": 0.000413498, 2024-03-16T12:08:21.828 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_running_send_time": 0.000000000, 2024-03-16T12:08:21.828 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_running_recv_time": 0.000340527, 2024-03-16T12:08:21.828 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-16T12:08:21.828 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_send_messages_queue_lat": { 2024-03-16T12:08:21.828 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-03-16T12:08:21.829 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-03-16T12:08:21.829 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-03-16T12:08:21.829 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:21.829 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_handle_ack_lat": { 2024-03-16T12:08:21.829 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-03-16T12:08:21.829 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-03-16T12:08:21.829 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-03-16T12:08:21.829 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:21.829 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-16T12:08:21.830 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_send_encrypted_bytes": 0 2024-03-16T12:08:21.830 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:21.830 INFO:teuthology.orchestra.run.smithi029.stdout: "cct": { 2024-03-16T12:08:21.830 INFO:teuthology.orchestra.run.smithi029.stdout: "total_workers": 0, 2024-03-16T12:08:21.830 INFO:teuthology.orchestra.run.smithi029.stdout: "unhealthy_workers": 0 2024-03-16T12:08:21.830 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:21.830 INFO:teuthology.orchestra.run.smithi029.stdout: "mempool": { 2024-03-16T12:08:21.830 INFO:teuthology.orchestra.run.smithi029.stdout: "bloom_filter_bytes": 0, 2024-03-16T12:08:21.831 INFO:teuthology.orchestra.run.smithi029.stdout: "bloom_filter_items": 0, 2024-03-16T12:08:21.831 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_alloc_bytes": 0, 2024-03-16T12:08:21.831 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_alloc_items": 0, 2024-03-16T12:08:21.831 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_cache_data_bytes": 0, 2024-03-16T12:08:21.831 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_cache_data_items": 0, 2024-03-16T12:08:21.831 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-16T12:08:21.831 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_cache_onode_items": 0, 2024-03-16T12:08:21.831 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-16T12:08:21.831 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_cache_meta_items": 0, 2024-03-16T12:08:21.832 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_cache_other_bytes": 0, 2024-03-16T12:08:21.832 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_cache_other_items": 0, 2024-03-16T12:08:21.832 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-16T12:08:21.832 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_cache_buffer_items": 0, 2024-03-16T12:08:21.832 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_extent_bytes": 0, 2024-03-16T12:08:21.832 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_extent_items": 0, 2024-03-16T12:08:21.832 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_blob_bytes": 0, 2024-03-16T12:08:21.832 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_blob_items": 0, 2024-03-16T12:08:21.833 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-16T12:08:21.833 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_shared_blob_items": 0, 2024-03-16T12:08:21.833 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-16T12:08:21.833 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_inline_bl_items": 0, 2024-03-16T12:08:21.833 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_fsck_bytes": 0, 2024-03-16T12:08:21.833 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_fsck_items": 0, 2024-03-16T12:08:21.833 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_txc_bytes": 0, 2024-03-16T12:08:21.833 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_txc_items": 0, 2024-03-16T12:08:21.833 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-16T12:08:21.834 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_writing_deferred_items": 0, 2024-03-16T12:08:21.834 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_writing_bytes": 0, 2024-03-16T12:08:21.834 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_writing_items": 0, 2024-03-16T12:08:21.834 INFO:teuthology.orchestra.run.smithi029.stdout: "bluefs_bytes": 0, 2024-03-16T12:08:21.834 INFO:teuthology.orchestra.run.smithi029.stdout: "bluefs_items": 0, 2024-03-16T12:08:21.834 INFO:teuthology.orchestra.run.smithi029.stdout: "bluefs_file_reader_bytes": 0, 2024-03-16T12:08:21.834 INFO:teuthology.orchestra.run.smithi029.stdout: "bluefs_file_reader_items": 0, 2024-03-16T12:08:21.834 INFO:teuthology.orchestra.run.smithi029.stdout: "bluefs_file_writer_bytes": 0, 2024-03-16T12:08:21.835 INFO:teuthology.orchestra.run.smithi029.stdout: "bluefs_file_writer_items": 0, 2024-03-16T12:08:21.835 INFO:teuthology.orchestra.run.smithi029.stdout: "buffer_anon_bytes": 148731, 2024-03-16T12:08:21.835 INFO:teuthology.orchestra.run.smithi029.stdout: "buffer_anon_items": 31, 2024-03-16T12:08:21.835 INFO:teuthology.orchestra.run.smithi029.stdout: "buffer_meta_bytes": 0, 2024-03-16T12:08:21.835 INFO:teuthology.orchestra.run.smithi029.stdout: "buffer_meta_items": 0, 2024-03-16T12:08:21.835 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_bytes": 0, 2024-03-16T12:08:21.835 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_items": 0, 2024-03-16T12:08:21.835 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_mapbl_bytes": 0, 2024-03-16T12:08:21.835 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_mapbl_items": 0, 2024-03-16T12:08:21.836 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_pglog_bytes": 0, 2024-03-16T12:08:21.836 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_pglog_items": 0, 2024-03-16T12:08:21.836 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_bytes": 0, 2024-03-16T12:08:21.836 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_items": 0, 2024-03-16T12:08:21.836 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_mapping_bytes": 0, 2024-03-16T12:08:21.836 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_mapping_items": 0, 2024-03-16T12:08:21.836 INFO:teuthology.orchestra.run.smithi029.stdout: "pgmap_bytes": 0, 2024-03-16T12:08:21.836 INFO:teuthology.orchestra.run.smithi029.stdout: "pgmap_items": 0, 2024-03-16T12:08:21.837 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_co_bytes": 928, 2024-03-16T12:08:21.837 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_co_items": 2, 2024-03-16T12:08:21.837 INFO:teuthology.orchestra.run.smithi029.stdout: "unittest_1_bytes": 0, 2024-03-16T12:08:21.837 INFO:teuthology.orchestra.run.smithi029.stdout: "unittest_1_items": 0, 2024-03-16T12:08:21.837 INFO:teuthology.orchestra.run.smithi029.stdout: "unittest_2_bytes": 0, 2024-03-16T12:08:21.837 INFO:teuthology.orchestra.run.smithi029.stdout: "unittest_2_items": 0 2024-03-16T12:08:21.837 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:21.837 INFO:teuthology.orchestra.run.smithi029.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-16T12:08:21.837 INFO:teuthology.orchestra.run.smithi029.stdout: "val": 0, 2024-03-16T12:08:21.838 INFO:teuthology.orchestra.run.smithi029.stdout: "max": 104857600, 2024-03-16T12:08:21.838 INFO:teuthology.orchestra.run.smithi029.stdout: "get_started": 0, 2024-03-16T12:08:21.838 INFO:teuthology.orchestra.run.smithi029.stdout: "get": 330, 2024-03-16T12:08:21.838 INFO:teuthology.orchestra.run.smithi029.stdout: "get_sum": 1421514, 2024-03-16T12:08:21.838 INFO:teuthology.orchestra.run.smithi029.stdout: "get_or_fail_fail": 0, 2024-03-16T12:08:21.838 INFO:teuthology.orchestra.run.smithi029.stdout: "get_or_fail_success": 330, 2024-03-16T12:08:21.838 INFO:teuthology.orchestra.run.smithi029.stdout: "take": 0, 2024-03-16T12:08:21.838 INFO:teuthology.orchestra.run.smithi029.stdout: "take_sum": 0, 2024-03-16T12:08:21.839 INFO:teuthology.orchestra.run.smithi029.stdout: "put": 330, 2024-03-16T12:08:21.839 INFO:teuthology.orchestra.run.smithi029.stdout: "put_sum": 1421514, 2024-03-16T12:08:21.839 INFO:teuthology.orchestra.run.smithi029.stdout: "wait": { 2024-03-16T12:08:21.839 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-03-16T12:08:21.839 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-03-16T12:08:21.839 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-03-16T12:08:21.839 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:21.839 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:21.839 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-03-16T12:08:21.868 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:08:21 smithi152 ceph-mon[36291]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T12:08:22.520 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.g 2024-03-16T12:08:22.521 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.g 2024-03-16T12:08:22.521 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.g 2024-03-16T12:08:22.521 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.g 2024-03-16T12:08:22.521 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.g 2024-03-16T12:08:22.521 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.g 2024-03-16T12:08:22.521 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.g 2024-03-16T12:08:22.521 DEBUG:tasks.check_counter:Getting stats from j 2024-03-16T12:08:22.522 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid d1887f28-e389-11ee-95c9-87774f69a715 -- ceph daemon mds.j perf dump 2024-03-16T12:08:22.769 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/d1887f28-e389-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T12:08:22.785 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:08:22 smithi154 ceph-mon[36320]: pgmap v553: 97 pgs: 97 active+clean; 359 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 220 KiB/s rd, 2.7 MiB/s wr, 101 op/s 2024-03-16T12:08:22.786 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:08:22 smithi154 ceph-mon[36320]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:08:22.786 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:08:22 smithi154 ceph-mon[36320]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T12:08:22.786 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:08:22 smithi154 ceph-mon[36320]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' 2024-03-16T12:08:22.786 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:08:22 smithi154 ceph-mon[36320]: osdmap e116: 12 total, 12 up, 12 in 2024-03-16T12:08:22.828 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:08:22 smithi029 ceph-mon[31975]: pgmap v553: 97 pgs: 97 active+clean; 359 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 220 KiB/s rd, 2.7 MiB/s wr, 101 op/s 2024-03-16T12:08:22.829 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:08:22 smithi029 ceph-mon[31975]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:08:22.829 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:08:22 smithi029 ceph-mon[31975]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T12:08:22.829 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:08:22 smithi029 ceph-mon[31975]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' 2024-03-16T12:08:22.829 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:08:22 smithi029 ceph-mon[31975]: osdmap e116: 12 total, 12 up, 12 in 2024-03-16T12:08:22.868 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:08:22 smithi152 ceph-mon[36291]: pgmap v553: 97 pgs: 97 active+clean; 359 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 220 KiB/s rd, 2.7 MiB/s wr, 101 op/s 2024-03-16T12:08:22.868 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:08:22 smithi152 ceph-mon[36291]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T12:08:22.868 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:08:22 smithi152 ceph-mon[36291]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T12:08:22.868 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:08:22 smithi152 ceph-mon[36291]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' 2024-03-16T12:08:22.868 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:08:22 smithi152 ceph-mon[36291]: osdmap e116: 12 total, 12 up, 12 in 2024-03-16T12:08:24.567 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:08:24 smithi029 ceph-mon[31975]: pgmap v555: 97 pgs: 97 active+clean; 357 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 228 KiB/s rd, 2.3 MiB/s wr, 92 op/s 2024-03-16T12:08:24.590 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-03-16T12:08:24.590 INFO:teuthology.orchestra.run.smithi029.stdout: "AsyncMessenger::Worker-0": { 2024-03-16T12:08:24.590 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_recv_messages": 0, 2024-03-16T12:08:24.590 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_send_messages": 2, 2024-03-16T12:08:24.591 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_recv_bytes": 0, 2024-03-16T12:08:24.591 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_send_bytes": 0, 2024-03-16T12:08:24.591 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_created_connections": 4, 2024-03-16T12:08:24.591 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_active_connections": 0, 2024-03-16T12:08:24.591 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_running_total_time": 0.001116134, 2024-03-16T12:08:24.591 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_running_send_time": 0.000000000, 2024-03-16T12:08:24.591 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_running_recv_time": 0.000121595, 2024-03-16T12:08:24.591 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-16T12:08:24.592 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_send_messages_queue_lat": { 2024-03-16T12:08:24.592 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-03-16T12:08:24.592 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-03-16T12:08:24.592 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-03-16T12:08:24.592 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:24.592 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_handle_ack_lat": { 2024-03-16T12:08:24.592 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-03-16T12:08:24.592 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-03-16T12:08:24.593 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-03-16T12:08:24.593 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:24.593 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-16T12:08:24.593 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_send_encrypted_bytes": 0 2024-03-16T12:08:24.593 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:24.593 INFO:teuthology.orchestra.run.smithi029.stdout: "AsyncMessenger::Worker-1": { 2024-03-16T12:08:24.593 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_recv_messages": 338, 2024-03-16T12:08:24.594 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_send_messages": 326, 2024-03-16T12:08:24.594 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_recv_bytes": 1472688, 2024-03-16T12:08:24.594 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_send_bytes": 200688, 2024-03-16T12:08:24.594 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_created_connections": 2, 2024-03-16T12:08:24.594 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_active_connections": 1, 2024-03-16T12:08:24.594 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_running_total_time": 0.052152354, 2024-03-16T12:08:24.594 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_running_send_time": 0.017318520, 2024-03-16T12:08:24.595 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_running_recv_time": 0.036681989, 2024-03-16T12:08:24.595 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_running_fast_dispatch_time": 0.005631613, 2024-03-16T12:08:24.595 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_send_messages_queue_lat": { 2024-03-16T12:08:24.595 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 326, 2024-03-16T12:08:24.595 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.012503006, 2024-03-16T12:08:24.595 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000038352 2024-03-16T12:08:24.595 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:24.595 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_handle_ack_lat": { 2024-03-16T12:08:24.596 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-03-16T12:08:24.596 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-03-16T12:08:24.596 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-03-16T12:08:24.596 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:24.596 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_recv_encrypted_bytes": 1472688, 2024-03-16T12:08:24.596 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_send_encrypted_bytes": 200688 2024-03-16T12:08:24.596 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:24.597 INFO:teuthology.orchestra.run.smithi029.stdout: "AsyncMessenger::Worker-2": { 2024-03-16T12:08:24.597 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_recv_messages": 3, 2024-03-16T12:08:24.597 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_send_messages": 248, 2024-03-16T12:08:24.597 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_recv_bytes": 480, 2024-03-16T12:08:24.597 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_send_bytes": 277456, 2024-03-16T12:08:24.597 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_created_connections": 6, 2024-03-16T12:08:24.597 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_active_connections": 3, 2024-03-16T12:08:24.598 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_running_total_time": 0.017184954, 2024-03-16T12:08:24.598 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_running_send_time": 0.012219351, 2024-03-16T12:08:24.598 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_running_recv_time": 0.001930701, 2024-03-16T12:08:24.598 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-16T12:08:24.598 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_send_messages_queue_lat": { 2024-03-16T12:08:24.598 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 245, 2024-03-16T12:08:24.598 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.013945617, 2024-03-16T12:08:24.599 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000056920 2024-03-16T12:08:24.599 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:24.599 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_handle_ack_lat": { 2024-03-16T12:08:24.599 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-03-16T12:08:24.599 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-03-16T12:08:24.599 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-03-16T12:08:24.599 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:24.600 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_recv_encrypted_bytes": 480, 2024-03-16T12:08:24.600 INFO:teuthology.orchestra.run.smithi029.stdout: "msgr_send_encrypted_bytes": 277456 2024-03-16T12:08:24.600 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:24.600 INFO:teuthology.orchestra.run.smithi029.stdout: "cct": { 2024-03-16T12:08:24.600 INFO:teuthology.orchestra.run.smithi029.stdout: "total_workers": 0, 2024-03-16T12:08:24.600 INFO:teuthology.orchestra.run.smithi029.stdout: "unhealthy_workers": 0 2024-03-16T12:08:24.600 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:24.601 INFO:teuthology.orchestra.run.smithi029.stdout: "mempool": { 2024-03-16T12:08:24.601 INFO:teuthology.orchestra.run.smithi029.stdout: "bloom_filter_bytes": 0, 2024-03-16T12:08:24.601 INFO:teuthology.orchestra.run.smithi029.stdout: "bloom_filter_items": 0, 2024-03-16T12:08:24.601 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_alloc_bytes": 0, 2024-03-16T12:08:24.601 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_alloc_items": 0, 2024-03-16T12:08:24.601 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_cache_data_bytes": 0, 2024-03-16T12:08:24.601 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_cache_data_items": 0, 2024-03-16T12:08:24.602 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-16T12:08:24.602 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_cache_onode_items": 0, 2024-03-16T12:08:24.602 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-16T12:08:24.602 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_cache_meta_items": 0, 2024-03-16T12:08:24.602 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_cache_other_bytes": 0, 2024-03-16T12:08:24.602 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_cache_other_items": 0, 2024-03-16T12:08:24.602 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-16T12:08:24.603 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_cache_buffer_items": 0, 2024-03-16T12:08:24.603 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_extent_bytes": 0, 2024-03-16T12:08:24.603 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_extent_items": 0, 2024-03-16T12:08:24.603 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_blob_bytes": 0, 2024-03-16T12:08:24.603 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_blob_items": 0, 2024-03-16T12:08:24.603 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-16T12:08:24.603 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_shared_blob_items": 0, 2024-03-16T12:08:24.603 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-16T12:08:24.604 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_inline_bl_items": 0, 2024-03-16T12:08:24.604 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_fsck_bytes": 0, 2024-03-16T12:08:24.604 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_fsck_items": 0, 2024-03-16T12:08:24.604 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_txc_bytes": 0, 2024-03-16T12:08:24.604 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_txc_items": 0, 2024-03-16T12:08:24.604 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-16T12:08:24.604 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_writing_deferred_items": 0, 2024-03-16T12:08:24.604 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_writing_bytes": 0, 2024-03-16T12:08:24.605 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_writing_items": 0, 2024-03-16T12:08:24.605 INFO:teuthology.orchestra.run.smithi029.stdout: "bluefs_bytes": 0, 2024-03-16T12:08:24.605 INFO:teuthology.orchestra.run.smithi029.stdout: "bluefs_items": 0, 2024-03-16T12:08:24.605 INFO:teuthology.orchestra.run.smithi029.stdout: "bluefs_file_reader_bytes": 0, 2024-03-16T12:08:24.605 INFO:teuthology.orchestra.run.smithi029.stdout: "bluefs_file_reader_items": 0, 2024-03-16T12:08:24.605 INFO:teuthology.orchestra.run.smithi029.stdout: "bluefs_file_writer_bytes": 0, 2024-03-16T12:08:24.605 INFO:teuthology.orchestra.run.smithi029.stdout: "bluefs_file_writer_items": 0, 2024-03-16T12:08:24.606 INFO:teuthology.orchestra.run.smithi029.stdout: "buffer_anon_bytes": 148732, 2024-03-16T12:08:24.606 INFO:teuthology.orchestra.run.smithi029.stdout: "buffer_anon_items": 31, 2024-03-16T12:08:24.606 INFO:teuthology.orchestra.run.smithi029.stdout: "buffer_meta_bytes": 0, 2024-03-16T12:08:24.606 INFO:teuthology.orchestra.run.smithi029.stdout: "buffer_meta_items": 0, 2024-03-16T12:08:24.606 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_bytes": 0, 2024-03-16T12:08:24.606 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_items": 0, 2024-03-16T12:08:24.606 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_mapbl_bytes": 0, 2024-03-16T12:08:24.606 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_mapbl_items": 0, 2024-03-16T12:08:24.607 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_pglog_bytes": 0, 2024-03-16T12:08:24.607 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_pglog_items": 0, 2024-03-16T12:08:24.607 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_bytes": 0, 2024-03-16T12:08:24.607 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_items": 0, 2024-03-16T12:08:24.607 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_mapping_bytes": 0, 2024-03-16T12:08:24.607 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_mapping_items": 0, 2024-03-16T12:08:24.607 INFO:teuthology.orchestra.run.smithi029.stdout: "pgmap_bytes": 0, 2024-03-16T12:08:24.607 INFO:teuthology.orchestra.run.smithi029.stdout: "pgmap_items": 0, 2024-03-16T12:08:24.608 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_co_bytes": 928, 2024-03-16T12:08:24.608 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_co_items": 2, 2024-03-16T12:08:24.608 INFO:teuthology.orchestra.run.smithi029.stdout: "unittest_1_bytes": 0, 2024-03-16T12:08:24.608 INFO:teuthology.orchestra.run.smithi029.stdout: "unittest_1_items": 0, 2024-03-16T12:08:24.608 INFO:teuthology.orchestra.run.smithi029.stdout: "unittest_2_bytes": 0, 2024-03-16T12:08:24.608 INFO:teuthology.orchestra.run.smithi029.stdout: "unittest_2_items": 0 2024-03-16T12:08:24.608 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-16T12:08:24.608 INFO:teuthology.orchestra.run.smithi029.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-16T12:08:24.609 INFO:teuthology.orchestra.run.smithi029.stdout: "val": 0, 2024-03-16T12:08:24.609 INFO:teuthology.orchestra.run.smithi029.stdout: "max": 104857600, 2024-03-16T12:08:24.609 INFO:teuthology.orchestra.run.smithi029.stdout: "get_started": 0, 2024-03-16T12:08:24.609 INFO:teuthology.orchestra.run.smithi029.stdout: "get": 337, 2024-03-16T12:08:24.609 INFO:teuthology.orchestra.run.smithi029.stdout: "get_sum": 1422488, 2024-03-16T12:08:24.609 INFO:teuthology.orchestra.run.smithi029.stdout: "get_or_fail_fail": 0, 2024-03-16T12:08:24.609 INFO:teuthology.orchestra.run.smithi029.stdout: "get_or_fail_success": 337, 2024-03-16T12:08:24.609 INFO:teuthology.orchestra.run.smithi029.stdout: "take": 0, 2024-03-16T12:08:24.610 INFO:teuthology.orchestra.run.smithi029.stdout: "take_sum": 0, 2024-03-16T12:08:24.610 INFO:teuthology.orchestra.run.smithi029.stdout: "put": 337, 2024-03-16T12:08:24.610 INFO:teuthology.orchestra.run.smithi029.stdout: "put_sum": 1422488, 2024-03-16T12:08:24.610 INFO:teuthology.orchestra.run.smithi029.stdout: "wait": { 2024-03-16T12:08:24.610 INFO:teuthology.orchestra.run.smithi029.stdout: "avgcount": 0, 2024-03-16T12:08:24.610 INFO:teuthology.orchestra.run.smithi029.stdout: "sum": 0.000000000, 2024-03-16T12:08:24.610 INFO:teuthology.orchestra.run.smithi029.stdout: "avgtime": 0.000000000 2024-03-16T12:08:24.610 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:24.611 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-16T12:08:24.611 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-03-16T12:08:24.785 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:08:24 smithi154 ceph-mon[36320]: pgmap v555: 97 pgs: 97 active+clean; 357 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 228 KiB/s rd, 2.3 MiB/s wr, 92 op/s 2024-03-16T12:08:24.868 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:08:24 smithi152 ceph-mon[36291]: pgmap v555: 97 pgs: 97 active+clean; 357 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 228 KiB/s rd, 2.3 MiB/s wr, 92 op/s 2024-03-16T12:08:25.352 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.j 2024-03-16T12:08:25.352 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.j 2024-03-16T12:08:25.352 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.j 2024-03-16T12:08:25.353 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.j 2024-03-16T12:08:25.353 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.j 2024-03-16T12:08:25.353 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.j 2024-03-16T12:08:25.353 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.j 2024-03-16T12:08:25.353 DEBUG:tasks.check_counter:Getting stats from b 2024-03-16T12:08:25.353 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid d1887f28-e389-11ee-95c9-87774f69a715 -- ceph daemon mds.b perf dump 2024-03-16T12:08:25.783 INFO:teuthology.orchestra.run.smithi152.stderr:Inferring config /var/lib/ceph/d1887f28-e389-11ee-95c9-87774f69a715/mon.b/config 2024-03-16T12:08:26.618 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:08:26 smithi152 ceph-mon[36291]: pgmap v556: 97 pgs: 97 active+clean; 357 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 228 KiB/s rd, 2.3 MiB/s wr, 92 op/s 2024-03-16T12:08:26.785 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:08:26 smithi154 ceph-mon[36320]: pgmap v556: 97 pgs: 97 active+clean; 357 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 228 KiB/s rd, 2.3 MiB/s wr, 92 op/s 2024-03-16T12:08:26.828 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:08:26 smithi029 ceph-mon[31975]: pgmap v556: 97 pgs: 97 active+clean; 357 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 228 KiB/s rd, 2.3 MiB/s wr, 92 op/s 2024-03-16T12:08:28.068 INFO:teuthology.orchestra.run.smithi152.stdout:{ 2024-03-16T12:08:28.068 INFO:teuthology.orchestra.run.smithi152.stdout: "AsyncMessenger::Worker-0": { 2024-03-16T12:08:28.068 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_recv_messages": 275703, 2024-03-16T12:08:28.068 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_send_messages": 459637, 2024-03-16T12:08:28.068 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_recv_bytes": 56007187, 2024-03-16T12:08:28.068 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_send_bytes": 717424055, 2024-03-16T12:08:28.069 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_created_connections": 11, 2024-03-16T12:08:28.069 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_active_connections": 4, 2024-03-16T12:08:28.069 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_running_total_time": 23.043930133, 2024-03-16T12:08:28.069 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_running_send_time": 10.274292798, 2024-03-16T12:08:28.069 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_running_recv_time": 9.957848636, 2024-03-16T12:08:28.069 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_running_fast_dispatch_time": 2.972244604, 2024-03-16T12:08:28.069 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_send_messages_queue_lat": { 2024-03-16T12:08:28.069 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 459628, 2024-03-16T12:08:28.069 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 54.303790710, 2024-03-16T12:08:28.070 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000118147 2024-03-16T12:08:28.070 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-16T12:08:28.070 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_handle_ack_lat": { 2024-03-16T12:08:28.070 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 572784, 2024-03-16T12:08:28.070 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.105912079, 2024-03-16T12:08:28.070 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000184 2024-03-16T12:08:28.070 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-16T12:08:28.070 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_recv_encrypted_bytes": 562560, 2024-03-16T12:08:28.070 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_send_encrypted_bytes": 181280 2024-03-16T12:08:28.071 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-16T12:08:28.071 INFO:teuthology.orchestra.run.smithi152.stdout: "AsyncMessenger::Worker-1": { 2024-03-16T12:08:28.071 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_recv_messages": 262152, 2024-03-16T12:08:28.071 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_send_messages": 448830, 2024-03-16T12:08:28.071 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_recv_bytes": 56263535, 2024-03-16T12:08:28.071 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_send_bytes": 517367143, 2024-03-16T12:08:28.071 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_created_connections": 160, 2024-03-16T12:08:28.071 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_active_connections": 6, 2024-03-16T12:08:28.071 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_running_total_time": 25.276819118, 2024-03-16T12:08:28.072 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_running_send_time": 12.073225211, 2024-03-16T12:08:28.072 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_running_recv_time": 11.209797366, 2024-03-16T12:08:28.072 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_running_fast_dispatch_time": 2.732947343, 2024-03-16T12:08:28.072 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_send_messages_queue_lat": { 2024-03-16T12:08:28.072 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 448832, 2024-03-16T12:08:28.072 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 60.565074895, 2024-03-16T12:08:28.072 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000134939 2024-03-16T12:08:28.072 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-16T12:08:28.072 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_handle_ack_lat": { 2024-03-16T12:08:28.072 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 568167, 2024-03-16T12:08:28.073 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.099493752, 2024-03-16T12:08:28.073 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000175 2024-03-16T12:08:28.073 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-16T12:08:28.073 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_recv_encrypted_bytes": 1613744, 2024-03-16T12:08:28.073 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_send_encrypted_bytes": 215056 2024-03-16T12:08:28.073 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-16T12:08:28.073 INFO:teuthology.orchestra.run.smithi152.stdout: "AsyncMessenger::Worker-2": { 2024-03-16T12:08:28.073 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_recv_messages": 670092, 2024-03-16T12:08:28.073 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_send_messages": 961843, 2024-03-16T12:08:28.074 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_recv_bytes": 192165466, 2024-03-16T12:08:28.074 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_send_bytes": 1070489038, 2024-03-16T12:08:28.074 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_created_connections": 18, 2024-03-16T12:08:28.074 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_active_connections": 8, 2024-03-16T12:08:28.074 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_running_total_time": 34.068412528, 2024-03-16T12:08:28.074 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_running_send_time": 17.263340938, 2024-03-16T12:08:28.074 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_running_recv_time": 20.396265969, 2024-03-16T12:08:28.074 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_running_fast_dispatch_time": 2.523049575, 2024-03-16T12:08:28.075 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_send_messages_queue_lat": { 2024-03-16T12:08:28.075 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 961834, 2024-03-16T12:08:28.075 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 72.488917834, 2024-03-16T12:08:28.075 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000075365 2024-03-16T12:08:28.075 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-16T12:08:28.075 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_handle_ack_lat": { 2024-03-16T12:08:28.075 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 844669, 2024-03-16T12:08:28.076 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.168656739, 2024-03-16T12:08:28.076 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000199 2024-03-16T12:08:28.076 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-16T12:08:28.076 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_recv_encrypted_bytes": 94208, 2024-03-16T12:08:28.076 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_send_encrypted_bytes": 635344 2024-03-16T12:08:28.076 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-16T12:08:28.076 INFO:teuthology.orchestra.run.smithi152.stdout: "cct": { 2024-03-16T12:08:28.076 INFO:teuthology.orchestra.run.smithi152.stdout: "total_workers": 1, 2024-03-16T12:08:28.076 INFO:teuthology.orchestra.run.smithi152.stdout: "unhealthy_workers": 0 2024-03-16T12:08:28.077 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-16T12:08:28.077 INFO:teuthology.orchestra.run.smithi152.stdout: "finisher-MDSRank": { 2024-03-16T12:08:28.077 INFO:teuthology.orchestra.run.smithi152.stdout: "queue_len": 0, 2024-03-16T12:08:28.077 INFO:teuthology.orchestra.run.smithi152.stdout: "complete_latency": { 2024-03-16T12:08:28.077 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 168266, 2024-03-16T12:08:28.077 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 281.626769484, 2024-03-16T12:08:28.077 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.001673699 2024-03-16T12:08:28.077 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-03-16T12:08:28.077 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-16T12:08:28.077 INFO:teuthology.orchestra.run.smithi152.stdout: "finisher-PurgeQueue": { 2024-03-16T12:08:28.078 INFO:teuthology.orchestra.run.smithi152.stdout: "queue_len": 0, 2024-03-16T12:08:28.078 INFO:teuthology.orchestra.run.smithi152.stdout: "complete_latency": { 2024-03-16T12:08:28.078 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 95216, 2024-03-16T12:08:28.078 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 29.167487208, 2024-03-16T12:08:28.078 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000306329 2024-03-16T12:08:28.078 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-03-16T12:08:28.078 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-16T12:08:28.078 INFO:teuthology.orchestra.run.smithi152.stdout: "mds": { 2024-03-16T12:08:28.079 INFO:teuthology.orchestra.run.smithi152.stdout: "request": 471023, 2024-03-16T12:08:28.079 INFO:teuthology.orchestra.run.smithi152.stdout: "reply": 470043, 2024-03-16T12:08:28.079 INFO:teuthology.orchestra.run.smithi152.stdout: "reply_latency": { 2024-03-16T12:08:28.079 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 470043, 2024-03-16T12:08:28.079 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 2603.986946150, 2024-03-16T12:08:28.079 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.005539890 2024-03-16T12:08:28.079 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-16T12:08:28.079 INFO:teuthology.orchestra.run.smithi152.stdout: "slow_reply": 0, 2024-03-16T12:08:28.079 INFO:teuthology.orchestra.run.smithi152.stdout: "forward": 980, 2024-03-16T12:08:28.080 INFO:teuthology.orchestra.run.smithi152.stdout: "dir_fetch_complete": 0, 2024-03-16T12:08:28.080 INFO:teuthology.orchestra.run.smithi152.stdout: "dir_fetch_keys": 0, 2024-03-16T12:08:28.080 INFO:teuthology.orchestra.run.smithi152.stdout: "dir_commit": 5998, 2024-03-16T12:08:28.080 INFO:teuthology.orchestra.run.smithi152.stdout: "dir_split": 33, 2024-03-16T12:08:28.080 INFO:teuthology.orchestra.run.smithi152.stdout: "dir_merge": 44, 2024-03-16T12:08:28.080 INFO:teuthology.orchestra.run.smithi152.stdout: "inodes": 6236, 2024-03-16T12:08:28.080 INFO:teuthology.orchestra.run.smithi152.stdout: "inodes_top": 3048, 2024-03-16T12:08:28.080 INFO:teuthology.orchestra.run.smithi152.stdout: "inodes_bottom": 3188, 2024-03-16T12:08:28.080 INFO:teuthology.orchestra.run.smithi152.stdout: "inodes_pin_tail": 0, 2024-03-16T12:08:28.081 INFO:teuthology.orchestra.run.smithi152.stdout: "inodes_pinned": 1881, 2024-03-16T12:08:28.081 INFO:teuthology.orchestra.run.smithi152.stdout: "inodes_expired": 133, 2024-03-16T12:08:28.081 INFO:teuthology.orchestra.run.smithi152.stdout: "inodes_with_caps": 834, 2024-03-16T12:08:28.081 INFO:teuthology.orchestra.run.smithi152.stdout: "caps": 834, 2024-03-16T12:08:28.081 INFO:teuthology.orchestra.run.smithi152.stdout: "subtrees": 9, 2024-03-16T12:08:28.081 INFO:teuthology.orchestra.run.smithi152.stdout: "traverse": 694765, 2024-03-16T12:08:28.081 INFO:teuthology.orchestra.run.smithi152.stdout: "traverse_hit": 422756, 2024-03-16T12:08:28.081 INFO:teuthology.orchestra.run.smithi152.stdout: "traverse_forward": 980, 2024-03-16T12:08:28.081 INFO:teuthology.orchestra.run.smithi152.stdout: "traverse_discover": 23, 2024-03-16T12:08:28.082 INFO:teuthology.orchestra.run.smithi152.stdout: "traverse_dir_fetch": 0, 2024-03-16T12:08:28.082 INFO:teuthology.orchestra.run.smithi152.stdout: "traverse_remote_ino": 0, 2024-03-16T12:08:28.082 INFO:teuthology.orchestra.run.smithi152.stdout: "traverse_lock": 4554, 2024-03-16T12:08:28.082 INFO:teuthology.orchestra.run.smithi152.stdout: "load_cent": 0, 2024-03-16T12:08:28.082 INFO:teuthology.orchestra.run.smithi152.stdout: "q": 0, 2024-03-16T12:08:28.082 INFO:teuthology.orchestra.run.smithi152.stdout: "exported": 5, 2024-03-16T12:08:28.082 INFO:teuthology.orchestra.run.smithi152.stdout: "exported_inodes": 58, 2024-03-16T12:08:28.082 INFO:teuthology.orchestra.run.smithi152.stdout: "imported": 0, 2024-03-16T12:08:28.082 INFO:teuthology.orchestra.run.smithi152.stdout: "imported_inodes": 0, 2024-03-16T12:08:28.083 INFO:teuthology.orchestra.run.smithi152.stdout: "openino_dir_fetch": 0, 2024-03-16T12:08:28.083 INFO:teuthology.orchestra.run.smithi152.stdout: "openino_backtrace_fetch": 0, 2024-03-16T12:08:28.083 INFO:teuthology.orchestra.run.smithi152.stdout: "openino_peer_discover": 0, 2024-03-16T12:08:28.083 INFO:teuthology.orchestra.run.smithi152.stdout: "root_rfiles": 669, 2024-03-16T12:08:28.083 INFO:teuthology.orchestra.run.smithi152.stdout: "root_rbytes": 211706842, 2024-03-16T12:08:28.083 INFO:teuthology.orchestra.run.smithi152.stdout: "root_rsnaps": 7, 2024-03-16T12:08:28.083 INFO:teuthology.orchestra.run.smithi152.stdout: "scrub_backtrace_fetch": 0, 2024-03-16T12:08:28.084 INFO:teuthology.orchestra.run.smithi152.stdout: "scrub_set_tag": 0, 2024-03-16T12:08:28.084 INFO:teuthology.orchestra.run.smithi152.stdout: "scrub_backtrace_repaired": 0, 2024-03-16T12:08:28.084 INFO:teuthology.orchestra.run.smithi152.stdout: "scrub_inotable_repaired": 0, 2024-03-16T12:08:28.084 INFO:teuthology.orchestra.run.smithi152.stdout: "scrub_dir_inodes": 0, 2024-03-16T12:08:28.084 INFO:teuthology.orchestra.run.smithi152.stdout: "scrub_dir_base_inodes": 0, 2024-03-16T12:08:28.084 INFO:teuthology.orchestra.run.smithi152.stdout: "scrub_dirfrag_rstats": 0, 2024-03-16T12:08:28.084 INFO:teuthology.orchestra.run.smithi152.stdout: "scrub_file_inodes": 0, 2024-03-16T12:08:28.084 INFO:teuthology.orchestra.run.smithi152.stdout: "handle_inode_file_caps": 5, 2024-03-16T12:08:28.084 INFO:teuthology.orchestra.run.smithi152.stdout: "ceph_cap_op_revoke": 22084, 2024-03-16T12:08:28.085 INFO:teuthology.orchestra.run.smithi152.stdout: "ceph_cap_op_grant": 226279, 2024-03-16T12:08:28.085 INFO:teuthology.orchestra.run.smithi152.stdout: "ceph_cap_op_trunc": 0, 2024-03-16T12:08:28.085 INFO:teuthology.orchestra.run.smithi152.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-16T12:08:28.085 INFO:teuthology.orchestra.run.smithi152.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-16T12:08:28.085 INFO:teuthology.orchestra.run.smithi152.stdout: "handle_client_caps": 116503, 2024-03-16T12:08:28.085 INFO:teuthology.orchestra.run.smithi152.stdout: "handle_client_caps_dirty": 51755, 2024-03-16T12:08:28.085 INFO:teuthology.orchestra.run.smithi152.stdout: "handle_client_cap_release": 22864, 2024-03-16T12:08:28.085 INFO:teuthology.orchestra.run.smithi152.stdout: "process_request_cap_release": 320958 2024-03-16T12:08:28.085 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-16T12:08:28.086 INFO:teuthology.orchestra.run.smithi152.stdout: "mds_cache": { 2024-03-16T12:08:28.086 INFO:teuthology.orchestra.run.smithi152.stdout: "dir_update": 202, 2024-03-16T12:08:28.086 INFO:teuthology.orchestra.run.smithi152.stdout: "dir_update_receipt": 0, 2024-03-16T12:08:28.086 INFO:teuthology.orchestra.run.smithi152.stdout: "dir_try_discover": 0, 2024-03-16T12:08:28.086 INFO:teuthology.orchestra.run.smithi152.stdout: "dir_send_discover": 23, 2024-03-16T12:08:28.086 INFO:teuthology.orchestra.run.smithi152.stdout: "dir_handle_discover": 552, 2024-03-16T12:08:28.086 INFO:teuthology.orchestra.run.smithi152.stdout: "num_strays": 169, 2024-03-16T12:08:28.086 INFO:teuthology.orchestra.run.smithi152.stdout: "num_strays_delayed": 0, 2024-03-16T12:08:28.086 INFO:teuthology.orchestra.run.smithi152.stdout: "num_strays_enqueuing": 0, 2024-03-16T12:08:28.087 INFO:teuthology.orchestra.run.smithi152.stdout: "strays_created": 63165, 2024-03-16T12:08:28.087 INFO:teuthology.orchestra.run.smithi152.stdout: "strays_enqueued": 62996, 2024-03-16T12:08:28.087 INFO:teuthology.orchestra.run.smithi152.stdout: "strays_reintegrated": 0, 2024-03-16T12:08:28.087 INFO:teuthology.orchestra.run.smithi152.stdout: "strays_migrated": 0, 2024-03-16T12:08:28.087 INFO:teuthology.orchestra.run.smithi152.stdout: "num_recovering_processing": 0, 2024-03-16T12:08:28.087 INFO:teuthology.orchestra.run.smithi152.stdout: "num_recovering_enqueued": 0, 2024-03-16T12:08:28.087 INFO:teuthology.orchestra.run.smithi152.stdout: "num_recovering_prioritized": 0, 2024-03-16T12:08:28.087 INFO:teuthology.orchestra.run.smithi152.stdout: "recovery_started": 0, 2024-03-16T12:08:28.087 INFO:teuthology.orchestra.run.smithi152.stdout: "recovery_completed": 0, 2024-03-16T12:08:28.088 INFO:teuthology.orchestra.run.smithi152.stdout: "ireq_quiesce_path": 0, 2024-03-16T12:08:28.088 INFO:teuthology.orchestra.run.smithi152.stdout: "ireq_quiesce_inode": 0, 2024-03-16T12:08:28.088 INFO:teuthology.orchestra.run.smithi152.stdout: "ireq_enqueue_scrub": 0, 2024-03-16T12:08:28.088 INFO:teuthology.orchestra.run.smithi152.stdout: "ireq_exportdir": 6, 2024-03-16T12:08:28.088 INFO:teuthology.orchestra.run.smithi152.stdout: "ireq_flush": 0, 2024-03-16T12:08:28.088 INFO:teuthology.orchestra.run.smithi152.stdout: "ireq_fragmentdir": 77, 2024-03-16T12:08:28.088 INFO:teuthology.orchestra.run.smithi152.stdout: "ireq_fragstats": 0, 2024-03-16T12:08:28.088 INFO:teuthology.orchestra.run.smithi152.stdout: "ireq_inodestats": 0 2024-03-16T12:08:28.088 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-16T12:08:28.089 INFO:teuthology.orchestra.run.smithi152.stdout: "mds_log": { 2024-03-16T12:08:28.089 INFO:teuthology.orchestra.run.smithi152.stdout: "evlrg": 0, 2024-03-16T12:08:28.089 INFO:teuthology.orchestra.run.smithi152.stdout: "evadd": 303821, 2024-03-16T12:08:28.089 INFO:teuthology.orchestra.run.smithi152.stdout: "evex": 301485, 2024-03-16T12:08:28.089 INFO:teuthology.orchestra.run.smithi152.stdout: "evtrm": 301485, 2024-03-16T12:08:28.089 INFO:teuthology.orchestra.run.smithi152.stdout: "ev": 2336, 2024-03-16T12:08:28.089 INFO:teuthology.orchestra.run.smithi152.stdout: "evexg": 0, 2024-03-16T12:08:28.089 INFO:teuthology.orchestra.run.smithi152.stdout: "evexd": 5918, 2024-03-16T12:08:28.089 INFO:teuthology.orchestra.run.smithi152.stdout: "segadd": 467, 2024-03-16T12:08:28.090 INFO:teuthology.orchestra.run.smithi152.stdout: "segex": 464, 2024-03-16T12:08:28.090 INFO:teuthology.orchestra.run.smithi152.stdout: "segtrm": 464, 2024-03-16T12:08:28.090 INFO:teuthology.orchestra.run.smithi152.stdout: "seg": 3, 2024-03-16T12:08:28.090 INFO:teuthology.orchestra.run.smithi152.stdout: "segmjr": 1, 2024-03-16T12:08:28.090 INFO:teuthology.orchestra.run.smithi152.stdout: "segexg": 0, 2024-03-16T12:08:28.090 INFO:teuthology.orchestra.run.smithi152.stdout: "segexd": 10, 2024-03-16T12:08:28.090 INFO:teuthology.orchestra.run.smithi152.stdout: "expos": 1063115311, 2024-03-16T12:08:28.090 INFO:teuthology.orchestra.run.smithi152.stdout: "wrpos": 1069398789, 2024-03-16T12:08:28.091 INFO:teuthology.orchestra.run.smithi152.stdout: "rdpos": 0, 2024-03-16T12:08:28.091 INFO:teuthology.orchestra.run.smithi152.stdout: "jlat": { 2024-03-16T12:08:28.091 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 90238, 2024-03-16T12:08:28.091 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 503.260346506, 2024-03-16T12:08:28.091 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.005577033 2024-03-16T12:08:28.091 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-16T12:08:28.091 INFO:teuthology.orchestra.run.smithi152.stdout: "replayed": 0 2024-03-16T12:08:28.092 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-16T12:08:28.092 INFO:teuthology.orchestra.run.smithi152.stdout: "mds_mem": { 2024-03-16T12:08:28.092 INFO:teuthology.orchestra.run.smithi152.stdout: "ino": 3365, 2024-03-16T12:08:28.092 INFO:teuthology.orchestra.run.smithi152.stdout: "ino+": 66461, 2024-03-16T12:08:28.092 INFO:teuthology.orchestra.run.smithi152.stdout: "ino-": 63096, 2024-03-16T12:08:28.092 INFO:teuthology.orchestra.run.smithi152.stdout: "dir": 197, 2024-03-16T12:08:28.092 INFO:teuthology.orchestra.run.smithi152.stdout: "dir+": 451, 2024-03-16T12:08:28.092 INFO:teuthology.orchestra.run.smithi152.stdout: "dir-": 254, 2024-03-16T12:08:28.093 INFO:teuthology.orchestra.run.smithi152.stdout: "dn": 6236, 2024-03-16T12:08:28.093 INFO:teuthology.orchestra.run.smithi152.stdout: "dn+": 151080, 2024-03-16T12:08:28.093 INFO:teuthology.orchestra.run.smithi152.stdout: "dn-": 144844, 2024-03-16T12:08:28.093 INFO:teuthology.orchestra.run.smithi152.stdout: "cap": 834, 2024-03-16T12:08:28.093 INFO:teuthology.orchestra.run.smithi152.stdout: "cap+": 70745, 2024-03-16T12:08:28.093 INFO:teuthology.orchestra.run.smithi152.stdout: "cap-": 69911, 2024-03-16T12:08:28.093 INFO:teuthology.orchestra.run.smithi152.stdout: "rss": 101928, 2024-03-16T12:08:28.093 INFO:teuthology.orchestra.run.smithi152.stdout: "heap": 231804 2024-03-16T12:08:28.093 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-16T12:08:28.094 INFO:teuthology.orchestra.run.smithi152.stdout: "mds_server": { 2024-03-16T12:08:28.094 INFO:teuthology.orchestra.run.smithi152.stdout: "dispatch_client_request": 672461, 2024-03-16T12:08:28.094 INFO:teuthology.orchestra.run.smithi152.stdout: "dispatch_server_request": 3379, 2024-03-16T12:08:28.094 INFO:teuthology.orchestra.run.smithi152.stdout: "handle_client_request": 471023, 2024-03-16T12:08:28.094 INFO:teuthology.orchestra.run.smithi152.stdout: "handle_client_session": 9018, 2024-03-16T12:08:28.094 INFO:teuthology.orchestra.run.smithi152.stdout: "handle_peer_request": 6689, 2024-03-16T12:08:28.094 INFO:teuthology.orchestra.run.smithi152.stdout: "req_create_latency": { 2024-03-16T12:08:28.094 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 63826, 2024-03-16T12:08:28.095 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 351.869427404, 2024-03-16T12:08:28.095 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.005512948 2024-03-16T12:08:28.095 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-16T12:08:28.095 INFO:teuthology.orchestra.run.smithi152.stdout: "req_getattr_latency": { 2024-03-16T12:08:28.095 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 2890, 2024-03-16T12:08:28.095 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 88.440656961, 2024-03-16T12:08:28.095 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.030602303 2024-03-16T12:08:28.095 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-16T12:08:28.095 INFO:teuthology.orchestra.run.smithi152.stdout: "req_getfilelock_latency": { 2024-03-16T12:08:28.096 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-03-16T12:08:28.096 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-03-16T12:08:28.096 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-03-16T12:08:28.096 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-16T12:08:28.096 INFO:teuthology.orchestra.run.smithi152.stdout: "req_link_latency": { 2024-03-16T12:08:28.096 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-03-16T12:08:28.096 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-03-16T12:08:28.096 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-03-16T12:08:28.096 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-16T12:08:28.096 INFO:teuthology.orchestra.run.smithi152.stdout: "req_lookup_latency": { 2024-03-16T12:08:28.097 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 151970, 2024-03-16T12:08:28.097 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 676.193019154, 2024-03-16T12:08:28.097 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.004449516 2024-03-16T12:08:28.097 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-16T12:08:28.097 INFO:teuthology.orchestra.run.smithi152.stdout: "req_lookuphash_latency": { 2024-03-16T12:08:28.097 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-03-16T12:08:28.097 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-03-16T12:08:28.097 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-03-16T12:08:28.097 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-16T12:08:28.097 INFO:teuthology.orchestra.run.smithi152.stdout: "req_lookupino_latency": { 2024-03-16T12:08:28.098 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 1, 2024-03-16T12:08:28.098 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000248009, 2024-03-16T12:08:28.098 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000248009 2024-03-16T12:08:28.098 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-16T12:08:28.098 INFO:teuthology.orchestra.run.smithi152.stdout: "req_lookupname_latency": { 2024-03-16T12:08:28.098 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-03-16T12:08:28.098 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-03-16T12:08:28.098 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-03-16T12:08:28.099 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-16T12:08:28.099 INFO:teuthology.orchestra.run.smithi152.stdout: "req_lookupparent_latency": { 2024-03-16T12:08:28.099 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-03-16T12:08:28.099 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-03-16T12:08:28.099 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-03-16T12:08:28.099 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-16T12:08:28.099 INFO:teuthology.orchestra.run.smithi152.stdout: "req_lookupsnap_latency": { 2024-03-16T12:08:28.100 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-03-16T12:08:28.100 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-03-16T12:08:28.100 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-03-16T12:08:28.100 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-16T12:08:28.101 INFO:teuthology.orchestra.run.smithi152.stdout: "req_lssnap_latency": { 2024-03-16T12:08:28.101 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 18, 2024-03-16T12:08:28.101 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.017013665, 2024-03-16T12:08:28.101 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000945203 2024-03-16T12:08:28.101 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-16T12:08:28.101 INFO:teuthology.orchestra.run.smithi152.stdout: "req_mkdir_latency": { 2024-03-16T12:08:28.101 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 119, 2024-03-16T12:08:28.101 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 1.279477816, 2024-03-16T12:08:28.101 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.010751914 2024-03-16T12:08:28.102 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-16T12:08:28.102 INFO:teuthology.orchestra.run.smithi152.stdout: "req_mknod_latency": { 2024-03-16T12:08:28.102 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-03-16T12:08:28.102 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-03-16T12:08:28.102 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-03-16T12:08:28.102 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-16T12:08:28.102 INFO:teuthology.orchestra.run.smithi152.stdout: "req_mksnap_latency": { 2024-03-16T12:08:28.102 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 18, 2024-03-16T12:08:28.102 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 8.055540564, 2024-03-16T12:08:28.103 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.447530031 2024-03-16T12:08:28.103 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-16T12:08:28.103 INFO:teuthology.orchestra.run.smithi152.stdout: "req_open_latency": { 2024-03-16T12:08:28.103 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 10, 2024-03-16T12:08:28.103 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.021466677, 2024-03-16T12:08:28.103 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.002146667 2024-03-16T12:08:28.103 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-16T12:08:28.103 INFO:teuthology.orchestra.run.smithi152.stdout: "req_readdir_latency": { 2024-03-16T12:08:28.103 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 172365, 2024-03-16T12:08:28.104 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 994.605364211, 2024-03-16T12:08:28.104 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.005770344 2024-03-16T12:08:28.104 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-16T12:08:28.104 INFO:teuthology.orchestra.run.smithi152.stdout: "req_rename_latency": { 2024-03-16T12:08:28.104 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 13403, 2024-03-16T12:08:28.104 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 84.765550565, 2024-03-16T12:08:28.104 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.006324371 2024-03-16T12:08:28.104 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-16T12:08:28.104 INFO:teuthology.orchestra.run.smithi152.stdout: "req_renamesnap_latency": { 2024-03-16T12:08:28.105 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-03-16T12:08:28.105 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-03-16T12:08:28.105 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-03-16T12:08:28.105 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-16T12:08:28.105 INFO:teuthology.orchestra.run.smithi152.stdout: "req_snapdiff_latency": { 2024-03-16T12:08:28.105 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-03-16T12:08:28.105 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-03-16T12:08:28.105 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-03-16T12:08:28.105 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-16T12:08:28.106 INFO:teuthology.orchestra.run.smithi152.stdout: "req_rmdir_latency": { 2024-03-16T12:08:28.106 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 1, 2024-03-16T12:08:28.106 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.002932487, 2024-03-16T12:08:28.106 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.002932487 2024-03-16T12:08:28.106 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-16T12:08:28.106 INFO:teuthology.orchestra.run.smithi152.stdout: "req_rmsnap_latency": { 2024-03-16T12:08:28.106 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 11, 2024-03-16T12:08:28.106 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 1.276832592, 2024-03-16T12:08:28.106 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.116075690 2024-03-16T12:08:28.107 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-16T12:08:28.107 INFO:teuthology.orchestra.run.smithi152.stdout: "req_rmxattr_latency": { 2024-03-16T12:08:28.107 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-03-16T12:08:28.107 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-03-16T12:08:28.107 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-03-16T12:08:28.107 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-16T12:08:28.107 INFO:teuthology.orchestra.run.smithi152.stdout: "req_setattr_latency": { 2024-03-16T12:08:28.107 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 158, 2024-03-16T12:08:28.108 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 6.207235068, 2024-03-16T12:08:28.108 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.039286297 2024-03-16T12:08:28.108 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-16T12:08:28.108 INFO:teuthology.orchestra.run.smithi152.stdout: "req_setdirlayout_latency": { 2024-03-16T12:08:28.108 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-03-16T12:08:28.108 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-03-16T12:08:28.108 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-03-16T12:08:28.109 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-16T12:08:28.109 INFO:teuthology.orchestra.run.smithi152.stdout: "req_setfilelock_latency": { 2024-03-16T12:08:28.109 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 2080, 2024-03-16T12:08:28.109 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 12.269642935, 2024-03-16T12:08:28.109 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.005898866 2024-03-16T12:08:28.109 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-16T12:08:28.109 INFO:teuthology.orchestra.run.smithi152.stdout: "req_setlayout_latency": { 2024-03-16T12:08:28.109 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-03-16T12:08:28.109 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-03-16T12:08:28.110 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-03-16T12:08:28.110 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-16T12:08:28.110 INFO:teuthology.orchestra.run.smithi152.stdout: "req_setxattr_latency": { 2024-03-16T12:08:28.110 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 8, 2024-03-16T12:08:28.110 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.020540768, 2024-03-16T12:08:28.110 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.002567596 2024-03-16T12:08:28.110 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-16T12:08:28.110 INFO:teuthology.orchestra.run.smithi152.stdout: "req_symlink_latency": { 2024-03-16T12:08:28.110 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-03-16T12:08:28.111 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-03-16T12:08:28.111 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-03-16T12:08:28.111 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-16T12:08:28.111 INFO:teuthology.orchestra.run.smithi152.stdout: "req_unlink_latency": { 2024-03-16T12:08:28.111 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 63165, 2024-03-16T12:08:28.111 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 378.961997274, 2024-03-16T12:08:28.111 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.005999556 2024-03-16T12:08:28.111 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-16T12:08:28.111 INFO:teuthology.orchestra.run.smithi152.stdout: "cap_revoke_eviction": 0, 2024-03-16T12:08:28.112 INFO:teuthology.orchestra.run.smithi152.stdout: "cap_acquisition_throttle": 0, 2024-03-16T12:08:28.112 INFO:teuthology.orchestra.run.smithi152.stdout: "req_getvxattr_latency": { 2024-03-16T12:08:28.112 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-03-16T12:08:28.112 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-03-16T12:08:28.112 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-03-16T12:08:28.112 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-03-16T12:08:28.112 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-16T12:08:28.112 INFO:teuthology.orchestra.run.smithi152.stdout: "mds_sessions": { 2024-03-16T12:08:28.112 INFO:teuthology.orchestra.run.smithi152.stdout: "session_count": 2, 2024-03-16T12:08:28.113 INFO:teuthology.orchestra.run.smithi152.stdout: "session_add": 6, 2024-03-16T12:08:28.113 INFO:teuthology.orchestra.run.smithi152.stdout: "session_remove": 4, 2024-03-16T12:08:28.113 INFO:teuthology.orchestra.run.smithi152.stdout: "sessions_open": 2, 2024-03-16T12:08:28.113 INFO:teuthology.orchestra.run.smithi152.stdout: "sessions_stale": 0, 2024-03-16T12:08:28.113 INFO:teuthology.orchestra.run.smithi152.stdout: "total_load": 43519, 2024-03-16T12:08:28.113 INFO:teuthology.orchestra.run.smithi152.stdout: "average_load": 21759, 2024-03-16T12:08:28.113 INFO:teuthology.orchestra.run.smithi152.stdout: "avg_session_uptime": 1051, 2024-03-16T12:08:28.113 INFO:teuthology.orchestra.run.smithi152.stdout: "mdthresh_evicted": 0 2024-03-16T12:08:28.113 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-16T12:08:28.114 INFO:teuthology.orchestra.run.smithi152.stdout: "mempool": { 2024-03-16T12:08:28.114 INFO:teuthology.orchestra.run.smithi152.stdout: "bloom_filter_bytes": 0, 2024-03-16T12:08:28.114 INFO:teuthology.orchestra.run.smithi152.stdout: "bloom_filter_items": 0, 2024-03-16T12:08:28.114 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_alloc_bytes": 0, 2024-03-16T12:08:28.114 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_alloc_items": 0, 2024-03-16T12:08:28.114 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_cache_data_bytes": 0, 2024-03-16T12:08:28.114 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_cache_data_items": 0, 2024-03-16T12:08:28.114 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-16T12:08:28.114 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_cache_onode_items": 0, 2024-03-16T12:08:28.115 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-16T12:08:28.115 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_cache_meta_items": 0, 2024-03-16T12:08:28.115 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_cache_other_bytes": 0, 2024-03-16T12:08:28.115 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_cache_other_items": 0, 2024-03-16T12:08:28.115 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-16T12:08:28.115 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_cache_buffer_items": 0, 2024-03-16T12:08:28.115 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_extent_bytes": 0, 2024-03-16T12:08:28.115 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_extent_items": 0, 2024-03-16T12:08:28.115 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_blob_bytes": 0, 2024-03-16T12:08:28.116 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_blob_items": 0, 2024-03-16T12:08:28.116 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-16T12:08:28.116 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_shared_blob_items": 0, 2024-03-16T12:08:28.116 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-16T12:08:28.116 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_inline_bl_items": 0, 2024-03-16T12:08:28.116 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_fsck_bytes": 0, 2024-03-16T12:08:28.116 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_fsck_items": 0, 2024-03-16T12:08:28.116 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_txc_bytes": 0, 2024-03-16T12:08:28.116 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_txc_items": 0, 2024-03-16T12:08:28.117 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-16T12:08:28.117 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_writing_deferred_items": 0, 2024-03-16T12:08:28.117 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_writing_bytes": 0, 2024-03-16T12:08:28.117 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_writing_items": 0, 2024-03-16T12:08:28.117 INFO:teuthology.orchestra.run.smithi152.stdout: "bluefs_bytes": 0, 2024-03-16T12:08:28.117 INFO:teuthology.orchestra.run.smithi152.stdout: "bluefs_items": 0, 2024-03-16T12:08:28.117 INFO:teuthology.orchestra.run.smithi152.stdout: "bluefs_file_reader_bytes": 0, 2024-03-16T12:08:28.117 INFO:teuthology.orchestra.run.smithi152.stdout: "bluefs_file_reader_items": 0, 2024-03-16T12:08:28.117 INFO:teuthology.orchestra.run.smithi152.stdout: "bluefs_file_writer_bytes": 0, 2024-03-16T12:08:28.118 INFO:teuthology.orchestra.run.smithi152.stdout: "bluefs_file_writer_items": 0, 2024-03-16T12:08:28.118 INFO:teuthology.orchestra.run.smithi152.stdout: "buffer_anon_bytes": 3058548, 2024-03-16T12:08:28.118 INFO:teuthology.orchestra.run.smithi152.stdout: "buffer_anon_items": 2786, 2024-03-16T12:08:28.118 INFO:teuthology.orchestra.run.smithi152.stdout: "buffer_meta_bytes": 0, 2024-03-16T12:08:28.118 INFO:teuthology.orchestra.run.smithi152.stdout: "buffer_meta_items": 0, 2024-03-16T12:08:28.118 INFO:teuthology.orchestra.run.smithi152.stdout: "osd_bytes": 0, 2024-03-16T12:08:28.118 INFO:teuthology.orchestra.run.smithi152.stdout: "osd_items": 0, 2024-03-16T12:08:28.118 INFO:teuthology.orchestra.run.smithi152.stdout: "osd_mapbl_bytes": 0, 2024-03-16T12:08:28.119 INFO:teuthology.orchestra.run.smithi152.stdout: "osd_mapbl_items": 0, 2024-03-16T12:08:28.119 INFO:teuthology.orchestra.run.smithi152.stdout: "osd_pglog_bytes": 0, 2024-03-16T12:08:28.119 INFO:teuthology.orchestra.run.smithi152.stdout: "osd_pglog_items": 0, 2024-03-16T12:08:28.119 INFO:teuthology.orchestra.run.smithi152.stdout: "osdmap_bytes": 8256, 2024-03-16T12:08:28.119 INFO:teuthology.orchestra.run.smithi152.stdout: "osdmap_items": 188, 2024-03-16T12:08:28.119 INFO:teuthology.orchestra.run.smithi152.stdout: "osdmap_mapping_bytes": 0, 2024-03-16T12:08:28.119 INFO:teuthology.orchestra.run.smithi152.stdout: "osdmap_mapping_items": 0, 2024-03-16T12:08:28.119 INFO:teuthology.orchestra.run.smithi152.stdout: "pgmap_bytes": 0, 2024-03-16T12:08:28.119 INFO:teuthology.orchestra.run.smithi152.stdout: "pgmap_items": 0, 2024-03-16T12:08:28.119 INFO:teuthology.orchestra.run.smithi152.stdout: "mds_co_bytes": 14131582, 2024-03-16T12:08:28.120 INFO:teuthology.orchestra.run.smithi152.stdout: "mds_co_items": 189380, 2024-03-16T12:08:28.120 INFO:teuthology.orchestra.run.smithi152.stdout: "unittest_1_bytes": 0, 2024-03-16T12:08:28.120 INFO:teuthology.orchestra.run.smithi152.stdout: "unittest_1_items": 0, 2024-03-16T12:08:28.120 INFO:teuthology.orchestra.run.smithi152.stdout: "unittest_2_bytes": 0, 2024-03-16T12:08:28.120 INFO:teuthology.orchestra.run.smithi152.stdout: "unittest_2_items": 0 2024-03-16T12:08:28.120 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-16T12:08:28.120 INFO:teuthology.orchestra.run.smithi152.stdout: "objecter": { 2024-03-16T12:08:28.120 INFO:teuthology.orchestra.run.smithi152.stdout: "op_active": 0, 2024-03-16T12:08:28.120 INFO:teuthology.orchestra.run.smithi152.stdout: "op_laggy": 0, 2024-03-16T12:08:28.121 INFO:teuthology.orchestra.run.smithi152.stdout: "op_send": 198847, 2024-03-16T12:08:28.121 INFO:teuthology.orchestra.run.smithi152.stdout: "op_send_bytes": 1124783227, 2024-03-16T12:08:28.121 INFO:teuthology.orchestra.run.smithi152.stdout: "op_resend": 35, 2024-03-16T12:08:28.121 INFO:teuthology.orchestra.run.smithi152.stdout: "op_reply": 198847, 2024-03-16T12:08:28.121 INFO:teuthology.orchestra.run.smithi152.stdout: "op_latency": { 2024-03-16T12:08:28.121 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 198847, 2024-03-16T12:08:28.121 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 4743.882758906, 2024-03-16T12:08:28.121 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.023856949 2024-03-16T12:08:28.121 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-16T12:08:28.122 INFO:teuthology.orchestra.run.smithi152.stdout: "op_inflight": 0, 2024-03-16T12:08:28.122 INFO:teuthology.orchestra.run.smithi152.stdout: "oplen_avg": { 2024-03-16T12:08:28.122 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 198847, 2024-03-16T12:08:28.122 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 280170 2024-03-16T12:08:28.122 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-16T12:08:28.122 INFO:teuthology.orchestra.run.smithi152.stdout: "op": 198847, 2024-03-16T12:08:28.122 INFO:teuthology.orchestra.run.smithi152.stdout: "op_r": 1816, 2024-03-16T12:08:28.122 INFO:teuthology.orchestra.run.smithi152.stdout: "op_w": 197031, 2024-03-16T12:08:28.122 INFO:teuthology.orchestra.run.smithi152.stdout: "op_rmw": 0, 2024-03-16T12:08:28.123 INFO:teuthology.orchestra.run.smithi152.stdout: "op_pg": 0, 2024-03-16T12:08:28.123 INFO:teuthology.orchestra.run.smithi152.stdout: "osdop_stat": 5564, 2024-03-16T12:08:28.123 INFO:teuthology.orchestra.run.smithi152.stdout: "osdop_create": 32474, 2024-03-16T12:08:28.123 INFO:teuthology.orchestra.run.smithi152.stdout: "osdop_read": 1816, 2024-03-16T12:08:28.123 INFO:teuthology.orchestra.run.smithi152.stdout: "osdop_write": 92054, 2024-03-16T12:08:28.123 INFO:teuthology.orchestra.run.smithi152.stdout: "osdop_writefull": 1115, 2024-03-16T12:08:28.123 INFO:teuthology.orchestra.run.smithi152.stdout: "osdop_writesame": 0, 2024-03-16T12:08:28.123 INFO:teuthology.orchestra.run.smithi152.stdout: "osdop_append": 0, 2024-03-16T12:08:28.123 INFO:teuthology.orchestra.run.smithi152.stdout: "osdop_zero": 0, 2024-03-16T12:08:28.124 INFO:teuthology.orchestra.run.smithi152.stdout: "osdop_truncate": 33, 2024-03-16T12:08:28.124 INFO:teuthology.orchestra.run.smithi152.stdout: "osdop_delete": 64235, 2024-03-16T12:08:28.124 INFO:teuthology.orchestra.run.smithi152.stdout: "osdop_mapext": 0, 2024-03-16T12:08:28.124 INFO:teuthology.orchestra.run.smithi152.stdout: "osdop_sparse_read": 0, 2024-03-16T12:08:28.124 INFO:teuthology.orchestra.run.smithi152.stdout: "osdop_clonerange": 0, 2024-03-16T12:08:28.124 INFO:teuthology.orchestra.run.smithi152.stdout: "osdop_getxattr": 0, 2024-03-16T12:08:28.124 INFO:teuthology.orchestra.run.smithi152.stdout: "osdop_setxattr": 64948, 2024-03-16T12:08:28.124 INFO:teuthology.orchestra.run.smithi152.stdout: "osdop_cmpxattr": 0, 2024-03-16T12:08:28.124 INFO:teuthology.orchestra.run.smithi152.stdout: "osdop_rmxattr": 0, 2024-03-16T12:08:28.125 INFO:teuthology.orchestra.run.smithi152.stdout: "osdop_resetxattrs": 0, 2024-03-16T12:08:28.125 INFO:teuthology.orchestra.run.smithi152.stdout: "osdop_call": 0, 2024-03-16T12:08:28.125 INFO:teuthology.orchestra.run.smithi152.stdout: "osdop_watch": 0, 2024-03-16T12:08:28.125 INFO:teuthology.orchestra.run.smithi152.stdout: "osdop_notify": 0, 2024-03-16T12:08:28.125 INFO:teuthology.orchestra.run.smithi152.stdout: "osdop_src_cmpxattr": 0, 2024-03-16T12:08:28.125 INFO:teuthology.orchestra.run.smithi152.stdout: "osdop_pgls": 0, 2024-03-16T12:08:28.125 INFO:teuthology.orchestra.run.smithi152.stdout: "osdop_pgls_filter": 0, 2024-03-16T12:08:28.125 INFO:teuthology.orchestra.run.smithi152.stdout: "osdop_other": 0, 2024-03-16T12:08:28.125 INFO:teuthology.orchestra.run.smithi152.stdout: "linger_active": 0, 2024-03-16T12:08:28.126 INFO:teuthology.orchestra.run.smithi152.stdout: "linger_send": 0, 2024-03-16T12:08:28.126 INFO:teuthology.orchestra.run.smithi152.stdout: "linger_resend": 0, 2024-03-16T12:08:28.126 INFO:teuthology.orchestra.run.smithi152.stdout: "linger_ping": 0, 2024-03-16T12:08:28.126 INFO:teuthology.orchestra.run.smithi152.stdout: "poolop_active": 0, 2024-03-16T12:08:28.126 INFO:teuthology.orchestra.run.smithi152.stdout: "poolop_send": 0, 2024-03-16T12:08:28.126 INFO:teuthology.orchestra.run.smithi152.stdout: "poolop_resend": 0, 2024-03-16T12:08:28.126 INFO:teuthology.orchestra.run.smithi152.stdout: "poolstat_active": 0, 2024-03-16T12:08:28.126 INFO:teuthology.orchestra.run.smithi152.stdout: "poolstat_send": 0, 2024-03-16T12:08:28.126 INFO:teuthology.orchestra.run.smithi152.stdout: "poolstat_resend": 0, 2024-03-16T12:08:28.127 INFO:teuthology.orchestra.run.smithi152.stdout: "statfs_active": 0, 2024-03-16T12:08:28.127 INFO:teuthology.orchestra.run.smithi152.stdout: "statfs_send": 0, 2024-03-16T12:08:28.127 INFO:teuthology.orchestra.run.smithi152.stdout: "statfs_resend": 0, 2024-03-16T12:08:28.127 INFO:teuthology.orchestra.run.smithi152.stdout: "command_active": 0, 2024-03-16T12:08:28.127 INFO:teuthology.orchestra.run.smithi152.stdout: "command_send": 0, 2024-03-16T12:08:28.127 INFO:teuthology.orchestra.run.smithi152.stdout: "command_resend": 0, 2024-03-16T12:08:28.127 INFO:teuthology.orchestra.run.smithi152.stdout: "map_epoch": 116, 2024-03-16T12:08:28.127 INFO:teuthology.orchestra.run.smithi152.stdout: "map_full": 0, 2024-03-16T12:08:28.127 INFO:teuthology.orchestra.run.smithi152.stdout: "map_inc": 32, 2024-03-16T12:08:28.128 INFO:teuthology.orchestra.run.smithi152.stdout: "osd_sessions": 12, 2024-03-16T12:08:28.128 INFO:teuthology.orchestra.run.smithi152.stdout: "osd_session_open": 12, 2024-03-16T12:08:28.128 INFO:teuthology.orchestra.run.smithi152.stdout: "osd_session_close": 0, 2024-03-16T12:08:28.128 INFO:teuthology.orchestra.run.smithi152.stdout: "osd_laggy": 0, 2024-03-16T12:08:28.128 INFO:teuthology.orchestra.run.smithi152.stdout: "omap_wr": 13179, 2024-03-16T12:08:28.128 INFO:teuthology.orchestra.run.smithi152.stdout: "omap_rd": 0, 2024-03-16T12:08:28.128 INFO:teuthology.orchestra.run.smithi152.stdout: "omap_del": 4752 2024-03-16T12:08:28.128 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-16T12:08:28.128 INFO:teuthology.orchestra.run.smithi152.stdout: "oft": { 2024-03-16T12:08:28.129 INFO:teuthology.orchestra.run.smithi152.stdout: "omap_total_objs": 1, 2024-03-16T12:08:28.129 INFO:teuthology.orchestra.run.smithi152.stdout: "omap_total_kv_pairs": 0, 2024-03-16T12:08:28.129 INFO:teuthology.orchestra.run.smithi152.stdout: "omap_total_updates": 0, 2024-03-16T12:08:28.129 INFO:teuthology.orchestra.run.smithi152.stdout: "omap_total_removes": 0 2024-03-16T12:08:28.129 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-16T12:08:28.129 INFO:teuthology.orchestra.run.smithi152.stdout: "osd-slow-ops": { 2024-03-16T12:08:28.129 INFO:teuthology.orchestra.run.smithi152.stdout: "slow_ops_count": 0 2024-03-16T12:08:28.130 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-16T12:08:28.130 INFO:teuthology.orchestra.run.smithi152.stdout: "purge_queue": { 2024-03-16T12:08:28.130 INFO:teuthology.orchestra.run.smithi152.stdout: "pq_executing_ops": 0, 2024-03-16T12:08:28.130 INFO:teuthology.orchestra.run.smithi152.stdout: "pq_executing_ops_high_water": 10, 2024-03-16T12:08:28.130 INFO:teuthology.orchestra.run.smithi152.stdout: "pq_executing": 0, 2024-03-16T12:08:28.130 INFO:teuthology.orchestra.run.smithi152.stdout: "pq_executing_high_water": 10, 2024-03-16T12:08:28.130 INFO:teuthology.orchestra.run.smithi152.stdout: "pq_executed_ops": 62996, 2024-03-16T12:08:28.130 INFO:teuthology.orchestra.run.smithi152.stdout: "pq_executed": 62996, 2024-03-16T12:08:28.130 INFO:teuthology.orchestra.run.smithi152.stdout: "pq_item_in_journal": 0 2024-03-16T12:08:28.131 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-16T12:08:28.131 INFO:teuthology.orchestra.run.smithi152.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-16T12:08:28.131 INFO:teuthology.orchestra.run.smithi152.stdout: "val": 0, 2024-03-16T12:08:28.131 INFO:teuthology.orchestra.run.smithi152.stdout: "max": 104857600, 2024-03-16T12:08:28.131 INFO:teuthology.orchestra.run.smithi152.stdout: "get_started": 0, 2024-03-16T12:08:28.131 INFO:teuthology.orchestra.run.smithi152.stdout: "get": 1207943, 2024-03-16T12:08:28.131 INFO:teuthology.orchestra.run.smithi152.stdout: "get_sum": 205545697, 2024-03-16T12:08:28.131 INFO:teuthology.orchestra.run.smithi152.stdout: "get_or_fail_fail": 0, 2024-03-16T12:08:28.131 INFO:teuthology.orchestra.run.smithi152.stdout: "get_or_fail_success": 1207943, 2024-03-16T12:08:28.132 INFO:teuthology.orchestra.run.smithi152.stdout: "take": 0, 2024-03-16T12:08:28.132 INFO:teuthology.orchestra.run.smithi152.stdout: "take_sum": 0, 2024-03-16T12:08:28.132 INFO:teuthology.orchestra.run.smithi152.stdout: "put": 1207943, 2024-03-16T12:08:28.132 INFO:teuthology.orchestra.run.smithi152.stdout: "put_sum": 205545697, 2024-03-16T12:08:28.132 INFO:teuthology.orchestra.run.smithi152.stdout: "wait": { 2024-03-16T12:08:28.132 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-03-16T12:08:28.132 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-03-16T12:08:28.132 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-03-16T12:08:28.132 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-03-16T12:08:28.133 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-16T12:08:28.133 INFO:teuthology.orchestra.run.smithi152.stdout: "throttle-objecter_bytes": { 2024-03-16T12:08:28.133 INFO:teuthology.orchestra.run.smithi152.stdout: "val": 0, 2024-03-16T12:08:28.133 INFO:teuthology.orchestra.run.smithi152.stdout: "max": 104857600, 2024-03-16T12:08:28.133 INFO:teuthology.orchestra.run.smithi152.stdout: "get_started": 0, 2024-03-16T12:08:28.133 INFO:teuthology.orchestra.run.smithi152.stdout: "get": 0, 2024-03-16T12:08:28.133 INFO:teuthology.orchestra.run.smithi152.stdout: "get_sum": 0, 2024-03-16T12:08:28.133 INFO:teuthology.orchestra.run.smithi152.stdout: "get_or_fail_fail": 0, 2024-03-16T12:08:28.133 INFO:teuthology.orchestra.run.smithi152.stdout: "get_or_fail_success": 0, 2024-03-16T12:08:28.134 INFO:teuthology.orchestra.run.smithi152.stdout: "take": 198847, 2024-03-16T12:08:28.134 INFO:teuthology.orchestra.run.smithi152.stdout: "take_sum": 1131156831, 2024-03-16T12:08:28.134 INFO:teuthology.orchestra.run.smithi152.stdout: "put": 198847, 2024-03-16T12:08:28.134 INFO:teuthology.orchestra.run.smithi152.stdout: "put_sum": 1131156831, 2024-03-16T12:08:28.134 INFO:teuthology.orchestra.run.smithi152.stdout: "wait": { 2024-03-16T12:08:28.134 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-03-16T12:08:28.134 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-03-16T12:08:28.134 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-03-16T12:08:28.134 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-03-16T12:08:28.135 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-16T12:08:28.135 INFO:teuthology.orchestra.run.smithi152.stdout: "throttle-objecter_ops": { 2024-03-16T12:08:28.135 INFO:teuthology.orchestra.run.smithi152.stdout: "val": 0, 2024-03-16T12:08:28.135 INFO:teuthology.orchestra.run.smithi152.stdout: "max": 1024, 2024-03-16T12:08:28.135 INFO:teuthology.orchestra.run.smithi152.stdout: "get_started": 0, 2024-03-16T12:08:28.135 INFO:teuthology.orchestra.run.smithi152.stdout: "get": 0, 2024-03-16T12:08:28.135 INFO:teuthology.orchestra.run.smithi152.stdout: "get_sum": 0, 2024-03-16T12:08:28.135 INFO:teuthology.orchestra.run.smithi152.stdout: "get_or_fail_fail": 0, 2024-03-16T12:08:28.135 INFO:teuthology.orchestra.run.smithi152.stdout: "get_or_fail_success": 0, 2024-03-16T12:08:28.136 INFO:teuthology.orchestra.run.smithi152.stdout: "take": 198847, 2024-03-16T12:08:28.136 INFO:teuthology.orchestra.run.smithi152.stdout: "take_sum": 198847, 2024-03-16T12:08:28.136 INFO:teuthology.orchestra.run.smithi152.stdout: "put": 198847, 2024-03-16T12:08:28.136 INFO:teuthology.orchestra.run.smithi152.stdout: "put_sum": 198847, 2024-03-16T12:08:28.136 INFO:teuthology.orchestra.run.smithi152.stdout: "wait": { 2024-03-16T12:08:28.136 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-03-16T12:08:28.136 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-03-16T12:08:28.136 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-03-16T12:08:28.136 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-03-16T12:08:28.137 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-16T12:08:28.137 INFO:teuthology.orchestra.run.smithi152.stdout: "throttle-write_buf_throttle": { 2024-03-16T12:08:28.137 INFO:teuthology.orchestra.run.smithi152.stdout: "val": 0, 2024-03-16T12:08:28.137 INFO:teuthology.orchestra.run.smithi152.stdout: "max": 3758096384, 2024-03-16T12:08:28.137 INFO:teuthology.orchestra.run.smithi152.stdout: "get_started": 0, 2024-03-16T12:08:28.137 INFO:teuthology.orchestra.run.smithi152.stdout: "get": 62996, 2024-03-16T12:08:28.137 INFO:teuthology.orchestra.run.smithi152.stdout: "get_sum": 6373604, 2024-03-16T12:08:28.137 INFO:teuthology.orchestra.run.smithi152.stdout: "get_or_fail_fail": 0, 2024-03-16T12:08:28.137 INFO:teuthology.orchestra.run.smithi152.stdout: "get_or_fail_success": 62996, 2024-03-16T12:08:28.138 INFO:teuthology.orchestra.run.smithi152.stdout: "take": 0, 2024-03-16T12:08:28.138 INFO:teuthology.orchestra.run.smithi152.stdout: "take_sum": 0, 2024-03-16T12:08:28.138 INFO:teuthology.orchestra.run.smithi152.stdout: "put": 1816, 2024-03-16T12:08:28.138 INFO:teuthology.orchestra.run.smithi152.stdout: "put_sum": 6373604, 2024-03-16T12:08:28.138 INFO:teuthology.orchestra.run.smithi152.stdout: "wait": { 2024-03-16T12:08:28.138 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-03-16T12:08:28.138 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-03-16T12:08:28.138 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-03-16T12:08:28.139 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-03-16T12:08:28.139 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-16T12:08:28.139 INFO:teuthology.orchestra.run.smithi152.stdout: "throttle-write_buf_throttle-0x565049588500": { 2024-03-16T12:08:28.139 INFO:teuthology.orchestra.run.smithi152.stdout: "val": 0, 2024-03-16T12:08:28.139 INFO:teuthology.orchestra.run.smithi152.stdout: "max": 3758096384, 2024-03-16T12:08:28.139 INFO:teuthology.orchestra.run.smithi152.stdout: "get_started": 0, 2024-03-16T12:08:28.139 INFO:teuthology.orchestra.run.smithi152.stdout: "get": 303821, 2024-03-16T12:08:28.139 INFO:teuthology.orchestra.run.smithi152.stdout: "get_sum": 1065204485, 2024-03-16T12:08:28.139 INFO:teuthology.orchestra.run.smithi152.stdout: "get_or_fail_fail": 0, 2024-03-16T12:08:28.140 INFO:teuthology.orchestra.run.smithi152.stdout: "get_or_fail_success": 303821, 2024-03-16T12:08:28.140 INFO:teuthology.orchestra.run.smithi152.stdout: "take": 0, 2024-03-16T12:08:28.140 INFO:teuthology.orchestra.run.smithi152.stdout: "take_sum": 0, 2024-03-16T12:08:28.140 INFO:teuthology.orchestra.run.smithi152.stdout: "put": 90238, 2024-03-16T12:08:28.140 INFO:teuthology.orchestra.run.smithi152.stdout: "put_sum": 1065204485, 2024-03-16T12:08:28.140 INFO:teuthology.orchestra.run.smithi152.stdout: "wait": { 2024-03-16T12:08:28.140 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-03-16T12:08:28.140 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-03-16T12:08:28.140 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-03-16T12:08:28.140 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-03-16T12:08:28.141 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-03-16T12:08:28.141 INFO:teuthology.orchestra.run.smithi152.stdout:} 2024-03-16T12:08:28.771 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:08:28 smithi152 ceph-mon[36291]: pgmap v557: 97 pgs: 97 active+clean; 356 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 198 KiB/s rd, 2.8 MiB/s wr, 108 op/s 2024-03-16T12:08:28.785 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:08:28 smithi154 ceph-mon[36320]: pgmap v557: 97 pgs: 97 active+clean; 356 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 198 KiB/s rd, 2.8 MiB/s wr, 108 op/s 2024-03-16T12:08:28.828 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:08:28 smithi029 ceph-mon[31975]: pgmap v557: 97 pgs: 97 active+clean; 356 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 198 KiB/s rd, 2.8 MiB/s wr, 108 op/s 2024-03-16T12:08:29.231 INFO:tasks.check_counter:Daemon mds.b mds.exported=5 2024-03-16T12:08:29.231 INFO:tasks.check_counter:Daemon mds.b mds.imported=0 2024-03-16T12:08:29.231 INFO:tasks.check_counter:Daemon mds.b mds_cache.dir_update=202 2024-03-16T12:08:29.231 INFO:tasks.check_counter:Daemon mds.b mds_cache.dir_update_receipt=0 2024-03-16T12:08:29.231 INFO:tasks.check_counter:Daemon mds.b mds.root_rsnaps=7 2024-03-16T12:08:29.231 INFO:tasks.check_counter:Daemon mds.b mds_server.req_mksnap_latency.avgcount=18 2024-03-16T12:08:29.231 INFO:tasks.check_counter:Daemon mds.b mds_server.req_rmsnap_latency.avgcount=11 2024-03-16T12:08:29.232 DEBUG:tasks.check_counter:Getting stats from e 2024-03-16T12:08:29.232 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid d1887f28-e389-11ee-95c9-87774f69a715 -- ceph daemon mds.e perf dump 2024-03-16T12:08:29.530 INFO:teuthology.orchestra.run.smithi152.stderr:Inferring config /var/lib/ceph/d1887f28-e389-11ee-95c9-87774f69a715/mon.b/config 2024-03-16T12:08:30.743 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:08:30 smithi152 ceph-mon[36291]: pgmap v558: 97 pgs: 97 active+clean; 356 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 198 KiB/s rd, 2.8 MiB/s wr, 108 op/s 2024-03-16T12:08:30.785 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:08:30 smithi154 ceph-mon[36320]: pgmap v558: 97 pgs: 97 active+clean; 356 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 198 KiB/s rd, 2.8 MiB/s wr, 108 op/s 2024-03-16T12:08:30.828 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:08:30 smithi029 ceph-mon[31975]: pgmap v558: 97 pgs: 97 active+clean; 356 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 198 KiB/s rd, 2.8 MiB/s wr, 108 op/s 2024-03-16T12:08:32.618 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:08:32 smithi152 ceph-mon[36291]: pgmap v559: 97 pgs: 97 active+clean; 356 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 198 KiB/s rd, 2.8 MiB/s wr, 107 op/s 2024-03-16T12:08:32.622 INFO:teuthology.orchestra.run.smithi152.stdout:{ 2024-03-16T12:08:32.622 INFO:teuthology.orchestra.run.smithi152.stdout: "AsyncMessenger::Worker-0": { 2024-03-16T12:08:32.622 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_recv_messages": 1291, 2024-03-16T12:08:32.622 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_send_messages": 1293, 2024-03-16T12:08:32.623 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_recv_bytes": 4262948, 2024-03-16T12:08:32.623 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_send_bytes": 410602, 2024-03-16T12:08:32.623 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_created_connections": 7, 2024-03-16T12:08:32.623 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_active_connections": 2, 2024-03-16T12:08:32.623 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_running_total_time": 0.090539429, 2024-03-16T12:08:32.623 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_running_send_time": 0.032443171, 2024-03-16T12:08:32.623 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_running_recv_time": 0.024227703, 2024-03-16T12:08:32.623 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_running_fast_dispatch_time": 0.026021342, 2024-03-16T12:08:32.623 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_send_messages_queue_lat": { 2024-03-16T12:08:32.624 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 1291, 2024-03-16T12:08:32.624 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.032673065, 2024-03-16T12:08:32.624 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000025308 2024-03-16T12:08:32.624 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-16T12:08:32.624 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_handle_ack_lat": { 2024-03-16T12:08:32.624 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-03-16T12:08:32.624 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-03-16T12:08:32.624 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-03-16T12:08:32.624 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-16T12:08:32.625 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-16T12:08:32.625 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_send_encrypted_bytes": 0 2024-03-16T12:08:32.625 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-16T12:08:32.625 INFO:teuthology.orchestra.run.smithi152.stdout: "AsyncMessenger::Worker-1": { 2024-03-16T12:08:32.625 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_recv_messages": 1116, 2024-03-16T12:08:32.625 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_send_messages": 1362, 2024-03-16T12:08:32.625 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_recv_bytes": 274386, 2024-03-16T12:08:32.625 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_send_bytes": 955153, 2024-03-16T12:08:32.625 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_created_connections": 7, 2024-03-16T12:08:32.626 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_active_connections": 5, 2024-03-16T12:08:32.626 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_running_total_time": 0.088656867, 2024-03-16T12:08:32.626 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_running_send_time": 0.027461584, 2024-03-16T12:08:32.626 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_running_recv_time": 0.019365668, 2024-03-16T12:08:32.626 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_running_fast_dispatch_time": 0.033758731, 2024-03-16T12:08:32.626 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_send_messages_queue_lat": { 2024-03-16T12:08:32.626 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 1359, 2024-03-16T12:08:32.626 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.048042557, 2024-03-16T12:08:32.626 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000035351 2024-03-16T12:08:32.627 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-16T12:08:32.627 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_handle_ack_lat": { 2024-03-16T12:08:32.627 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-03-16T12:08:32.627 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-03-16T12:08:32.627 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-03-16T12:08:32.627 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-16T12:08:32.627 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_recv_encrypted_bytes": 320, 2024-03-16T12:08:32.627 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_send_encrypted_bytes": 600896 2024-03-16T12:08:32.627 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-16T12:08:32.628 INFO:teuthology.orchestra.run.smithi152.stdout: "AsyncMessenger::Worker-2": { 2024-03-16T12:08:32.628 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_recv_messages": 1572, 2024-03-16T12:08:32.628 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_send_messages": 1549, 2024-03-16T12:08:32.628 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_recv_bytes": 1949520, 2024-03-16T12:08:32.628 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_send_bytes": 584442, 2024-03-16T12:08:32.628 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_created_connections": 3, 2024-03-16T12:08:32.628 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_active_connections": 2, 2024-03-16T12:08:32.628 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_running_total_time": 0.133687544, 2024-03-16T12:08:32.629 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_running_send_time": 0.040186915, 2024-03-16T12:08:32.629 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_running_recv_time": 0.060364499, 2024-03-16T12:08:32.629 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_running_fast_dispatch_time": 0.036294631, 2024-03-16T12:08:32.629 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_send_messages_queue_lat": { 2024-03-16T12:08:32.629 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 1549, 2024-03-16T12:08:32.629 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.047235091, 2024-03-16T12:08:32.629 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000030493 2024-03-16T12:08:32.629 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-16T12:08:32.630 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_handle_ack_lat": { 2024-03-16T12:08:32.630 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-03-16T12:08:32.630 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-03-16T12:08:32.630 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-03-16T12:08:32.630 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-16T12:08:32.630 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_recv_encrypted_bytes": 1550688, 2024-03-16T12:08:32.630 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_send_encrypted_bytes": 206976 2024-03-16T12:08:32.630 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-16T12:08:32.630 INFO:teuthology.orchestra.run.smithi152.stdout: "cct": { 2024-03-16T12:08:32.631 INFO:teuthology.orchestra.run.smithi152.stdout: "total_workers": 1, 2024-03-16T12:08:32.631 INFO:teuthology.orchestra.run.smithi152.stdout: "unhealthy_workers": 0 2024-03-16T12:08:32.631 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-16T12:08:32.631 INFO:teuthology.orchestra.run.smithi152.stdout: "finisher-MDSRank": { 2024-03-16T12:08:32.631 INFO:teuthology.orchestra.run.smithi152.stdout: "queue_len": 0, 2024-03-16T12:08:32.631 INFO:teuthology.orchestra.run.smithi152.stdout: "complete_latency": { 2024-03-16T12:08:32.631 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 5988, 2024-03-16T12:08:32.631 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.135519207, 2024-03-16T12:08:32.631 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000022631 2024-03-16T12:08:32.632 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-03-16T12:08:32.632 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-16T12:08:32.632 INFO:teuthology.orchestra.run.smithi152.stdout: "finisher-PurgeQueue": { 2024-03-16T12:08:32.632 INFO:teuthology.orchestra.run.smithi152.stdout: "queue_len": 0, 2024-03-16T12:08:32.632 INFO:teuthology.orchestra.run.smithi152.stdout: "complete_latency": { 2024-03-16T12:08:32.632 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-03-16T12:08:32.632 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-03-16T12:08:32.632 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-03-16T12:08:32.632 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-03-16T12:08:32.633 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-16T12:08:32.633 INFO:teuthology.orchestra.run.smithi152.stdout: "mds": { 2024-03-16T12:08:32.633 INFO:teuthology.orchestra.run.smithi152.stdout: "request": 0, 2024-03-16T12:08:32.633 INFO:teuthology.orchestra.run.smithi152.stdout: "reply": 0, 2024-03-16T12:08:32.633 INFO:teuthology.orchestra.run.smithi152.stdout: "reply_latency": { 2024-03-16T12:08:32.633 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-03-16T12:08:32.633 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-03-16T12:08:32.633 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-03-16T12:08:32.633 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-16T12:08:32.634 INFO:teuthology.orchestra.run.smithi152.stdout: "slow_reply": 0, 2024-03-16T12:08:32.634 INFO:teuthology.orchestra.run.smithi152.stdout: "forward": 0, 2024-03-16T12:08:32.634 INFO:teuthology.orchestra.run.smithi152.stdout: "dir_fetch_complete": 0, 2024-03-16T12:08:32.634 INFO:teuthology.orchestra.run.smithi152.stdout: "dir_fetch_keys": 0, 2024-03-16T12:08:32.634 INFO:teuthology.orchestra.run.smithi152.stdout: "dir_commit": 0, 2024-03-16T12:08:32.634 INFO:teuthology.orchestra.run.smithi152.stdout: "dir_split": 0, 2024-03-16T12:08:32.634 INFO:teuthology.orchestra.run.smithi152.stdout: "dir_merge": 0, 2024-03-16T12:08:32.634 INFO:teuthology.orchestra.run.smithi152.stdout: "inodes": 422, 2024-03-16T12:08:32.634 INFO:teuthology.orchestra.run.smithi152.stdout: "inodes_top": 1, 2024-03-16T12:08:32.635 INFO:teuthology.orchestra.run.smithi152.stdout: "inodes_bottom": 421, 2024-03-16T12:08:32.635 INFO:teuthology.orchestra.run.smithi152.stdout: "inodes_pin_tail": 0, 2024-03-16T12:08:32.635 INFO:teuthology.orchestra.run.smithi152.stdout: "inodes_pinned": 421, 2024-03-16T12:08:32.635 INFO:teuthology.orchestra.run.smithi152.stdout: "inodes_expired": 0, 2024-03-16T12:08:32.635 INFO:teuthology.orchestra.run.smithi152.stdout: "inodes_with_caps": 0, 2024-03-16T12:08:32.635 INFO:teuthology.orchestra.run.smithi152.stdout: "caps": 0, 2024-03-16T12:08:32.635 INFO:teuthology.orchestra.run.smithi152.stdout: "subtrees": 4, 2024-03-16T12:08:32.635 INFO:teuthology.orchestra.run.smithi152.stdout: "traverse": 0, 2024-03-16T12:08:32.636 INFO:teuthology.orchestra.run.smithi152.stdout: "traverse_hit": 0, 2024-03-16T12:08:32.636 INFO:teuthology.orchestra.run.smithi152.stdout: "traverse_forward": 0, 2024-03-16T12:08:32.636 INFO:teuthology.orchestra.run.smithi152.stdout: "traverse_discover": 0, 2024-03-16T12:08:32.636 INFO:teuthology.orchestra.run.smithi152.stdout: "traverse_dir_fetch": 0, 2024-03-16T12:08:32.636 INFO:teuthology.orchestra.run.smithi152.stdout: "traverse_remote_ino": 0, 2024-03-16T12:08:32.636 INFO:teuthology.orchestra.run.smithi152.stdout: "traverse_lock": 0, 2024-03-16T12:08:32.636 INFO:teuthology.orchestra.run.smithi152.stdout: "load_cent": 0, 2024-03-16T12:08:32.636 INFO:teuthology.orchestra.run.smithi152.stdout: "q": 0, 2024-03-16T12:08:32.636 INFO:teuthology.orchestra.run.smithi152.stdout: "exported": 0, 2024-03-16T12:08:32.637 INFO:teuthology.orchestra.run.smithi152.stdout: "exported_inodes": 0, 2024-03-16T12:08:32.637 INFO:teuthology.orchestra.run.smithi152.stdout: "imported": 0, 2024-03-16T12:08:32.637 INFO:teuthology.orchestra.run.smithi152.stdout: "imported_inodes": 0, 2024-03-16T12:08:32.637 INFO:teuthology.orchestra.run.smithi152.stdout: "openino_dir_fetch": 0, 2024-03-16T12:08:32.637 INFO:teuthology.orchestra.run.smithi152.stdout: "openino_backtrace_fetch": 0, 2024-03-16T12:08:32.637 INFO:teuthology.orchestra.run.smithi152.stdout: "openino_peer_discover": 0, 2024-03-16T12:08:32.637 INFO:teuthology.orchestra.run.smithi152.stdout: "root_rfiles": 0, 2024-03-16T12:08:32.637 INFO:teuthology.orchestra.run.smithi152.stdout: "root_rbytes": 0, 2024-03-16T12:08:32.638 INFO:teuthology.orchestra.run.smithi152.stdout: "root_rsnaps": 0, 2024-03-16T12:08:32.638 INFO:teuthology.orchestra.run.smithi152.stdout: "scrub_backtrace_fetch": 0, 2024-03-16T12:08:32.638 INFO:teuthology.orchestra.run.smithi152.stdout: "scrub_set_tag": 0, 2024-03-16T12:08:32.638 INFO:teuthology.orchestra.run.smithi152.stdout: "scrub_backtrace_repaired": 0, 2024-03-16T12:08:32.638 INFO:teuthology.orchestra.run.smithi152.stdout: "scrub_inotable_repaired": 0, 2024-03-16T12:08:32.638 INFO:teuthology.orchestra.run.smithi152.stdout: "scrub_dir_inodes": 0, 2024-03-16T12:08:32.638 INFO:teuthology.orchestra.run.smithi152.stdout: "scrub_dir_base_inodes": 0, 2024-03-16T12:08:32.638 INFO:teuthology.orchestra.run.smithi152.stdout: "scrub_dirfrag_rstats": 0, 2024-03-16T12:08:32.638 INFO:teuthology.orchestra.run.smithi152.stdout: "scrub_file_inodes": 0, 2024-03-16T12:08:32.639 INFO:teuthology.orchestra.run.smithi152.stdout: "handle_inode_file_caps": 0, 2024-03-16T12:08:32.639 INFO:teuthology.orchestra.run.smithi152.stdout: "ceph_cap_op_revoke": 0, 2024-03-16T12:08:32.639 INFO:teuthology.orchestra.run.smithi152.stdout: "ceph_cap_op_grant": 0, 2024-03-16T12:08:32.639 INFO:teuthology.orchestra.run.smithi152.stdout: "ceph_cap_op_trunc": 0, 2024-03-16T12:08:32.639 INFO:teuthology.orchestra.run.smithi152.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-16T12:08:32.639 INFO:teuthology.orchestra.run.smithi152.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-16T12:08:32.639 INFO:teuthology.orchestra.run.smithi152.stdout: "handle_client_caps": 0, 2024-03-16T12:08:32.639 INFO:teuthology.orchestra.run.smithi152.stdout: "handle_client_caps_dirty": 0, 2024-03-16T12:08:32.639 INFO:teuthology.orchestra.run.smithi152.stdout: "handle_client_cap_release": 0, 2024-03-16T12:08:32.640 INFO:teuthology.orchestra.run.smithi152.stdout: "process_request_cap_release": 0 2024-03-16T12:08:32.640 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-16T12:08:32.640 INFO:teuthology.orchestra.run.smithi152.stdout: "mds_cache": { 2024-03-16T12:08:32.640 INFO:teuthology.orchestra.run.smithi152.stdout: "dir_update": 0, 2024-03-16T12:08:32.640 INFO:teuthology.orchestra.run.smithi152.stdout: "dir_update_receipt": 0, 2024-03-16T12:08:32.640 INFO:teuthology.orchestra.run.smithi152.stdout: "dir_try_discover": 0, 2024-03-16T12:08:32.640 INFO:teuthology.orchestra.run.smithi152.stdout: "dir_send_discover": 0, 2024-03-16T12:08:32.640 INFO:teuthology.orchestra.run.smithi152.stdout: "dir_handle_discover": 0, 2024-03-16T12:08:32.640 INFO:teuthology.orchestra.run.smithi152.stdout: "num_strays": 0, 2024-03-16T12:08:32.641 INFO:teuthology.orchestra.run.smithi152.stdout: "num_strays_delayed": 0, 2024-03-16T12:08:32.641 INFO:teuthology.orchestra.run.smithi152.stdout: "num_strays_enqueuing": 0, 2024-03-16T12:08:32.641 INFO:teuthology.orchestra.run.smithi152.stdout: "strays_created": 0, 2024-03-16T12:08:32.641 INFO:teuthology.orchestra.run.smithi152.stdout: "strays_enqueued": 0, 2024-03-16T12:08:32.641 INFO:teuthology.orchestra.run.smithi152.stdout: "strays_reintegrated": 0, 2024-03-16T12:08:32.641 INFO:teuthology.orchestra.run.smithi152.stdout: "strays_migrated": 0, 2024-03-16T12:08:32.641 INFO:teuthology.orchestra.run.smithi152.stdout: "num_recovering_processing": 0, 2024-03-16T12:08:32.641 INFO:teuthology.orchestra.run.smithi152.stdout: "num_recovering_enqueued": 0, 2024-03-16T12:08:32.642 INFO:teuthology.orchestra.run.smithi152.stdout: "num_recovering_prioritized": 0, 2024-03-16T12:08:32.642 INFO:teuthology.orchestra.run.smithi152.stdout: "recovery_started": 0, 2024-03-16T12:08:32.642 INFO:teuthology.orchestra.run.smithi152.stdout: "recovery_completed": 0, 2024-03-16T12:08:32.642 INFO:teuthology.orchestra.run.smithi152.stdout: "ireq_quiesce_path": 0, 2024-03-16T12:08:32.642 INFO:teuthology.orchestra.run.smithi152.stdout: "ireq_quiesce_inode": 0, 2024-03-16T12:08:32.642 INFO:teuthology.orchestra.run.smithi152.stdout: "ireq_enqueue_scrub": 0, 2024-03-16T12:08:32.642 INFO:teuthology.orchestra.run.smithi152.stdout: "ireq_exportdir": 0, 2024-03-16T12:08:32.642 INFO:teuthology.orchestra.run.smithi152.stdout: "ireq_flush": 0, 2024-03-16T12:08:32.642 INFO:teuthology.orchestra.run.smithi152.stdout: "ireq_fragmentdir": 0, 2024-03-16T12:08:32.643 INFO:teuthology.orchestra.run.smithi152.stdout: "ireq_fragstats": 0, 2024-03-16T12:08:32.643 INFO:teuthology.orchestra.run.smithi152.stdout: "ireq_inodestats": 0 2024-03-16T12:08:32.643 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-16T12:08:32.643 INFO:teuthology.orchestra.run.smithi152.stdout: "mds_log": { 2024-03-16T12:08:32.643 INFO:teuthology.orchestra.run.smithi152.stdout: "evlrg": 0, 2024-03-16T12:08:32.643 INFO:teuthology.orchestra.run.smithi152.stdout: "evadd": 0, 2024-03-16T12:08:32.643 INFO:teuthology.orchestra.run.smithi152.stdout: "evex": 0, 2024-03-16T12:08:32.643 INFO:teuthology.orchestra.run.smithi152.stdout: "evtrm": 0, 2024-03-16T12:08:32.644 INFO:teuthology.orchestra.run.smithi152.stdout: "ev": 2731, 2024-03-16T12:08:32.644 INFO:teuthology.orchestra.run.smithi152.stdout: "evexg": 0, 2024-03-16T12:08:32.644 INFO:teuthology.orchestra.run.smithi152.stdout: "evexd": 0, 2024-03-16T12:08:32.644 INFO:teuthology.orchestra.run.smithi152.stdout: "segadd": 0, 2024-03-16T12:08:32.644 INFO:teuthology.orchestra.run.smithi152.stdout: "segex": 0, 2024-03-16T12:08:32.644 INFO:teuthology.orchestra.run.smithi152.stdout: "segtrm": 0, 2024-03-16T12:08:32.644 INFO:teuthology.orchestra.run.smithi152.stdout: "seg": 4, 2024-03-16T12:08:32.644 INFO:teuthology.orchestra.run.smithi152.stdout: "segmjr": 2, 2024-03-16T12:08:32.644 INFO:teuthology.orchestra.run.smithi152.stdout: "segexg": 0, 2024-03-16T12:08:32.645 INFO:teuthology.orchestra.run.smithi152.stdout: "segexd": 0, 2024-03-16T12:08:32.645 INFO:teuthology.orchestra.run.smithi152.stdout: "expos": 4194304, 2024-03-16T12:08:32.645 INFO:teuthology.orchestra.run.smithi152.stdout: "wrpos": 8119987, 2024-03-16T12:08:32.645 INFO:teuthology.orchestra.run.smithi152.stdout: "rdpos": 8119413, 2024-03-16T12:08:32.645 INFO:teuthology.orchestra.run.smithi152.stdout: "jlat": { 2024-03-16T12:08:32.645 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-03-16T12:08:32.645 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-03-16T12:08:32.645 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-03-16T12:08:32.645 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-16T12:08:32.646 INFO:teuthology.orchestra.run.smithi152.stdout: "replayed": 2731 2024-03-16T12:08:32.646 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-16T12:08:32.646 INFO:teuthology.orchestra.run.smithi152.stdout: "mds_mem": { 2024-03-16T12:08:32.646 INFO:teuthology.orchestra.run.smithi152.stdout: "ino": 67, 2024-03-16T12:08:32.646 INFO:teuthology.orchestra.run.smithi152.stdout: "ino+": 411, 2024-03-16T12:08:32.646 INFO:teuthology.orchestra.run.smithi152.stdout: "ino-": 344, 2024-03-16T12:08:32.646 INFO:teuthology.orchestra.run.smithi152.stdout: "dir": 18, 2024-03-16T12:08:32.646 INFO:teuthology.orchestra.run.smithi152.stdout: "dir+": 18, 2024-03-16T12:08:32.646 INFO:teuthology.orchestra.run.smithi152.stdout: "dir-": 0, 2024-03-16T12:08:32.647 INFO:teuthology.orchestra.run.smithi152.stdout: "dn": 422, 2024-03-16T12:08:32.647 INFO:teuthology.orchestra.run.smithi152.stdout: "dn+": 422, 2024-03-16T12:08:32.647 INFO:teuthology.orchestra.run.smithi152.stdout: "dn-": 0, 2024-03-16T12:08:32.647 INFO:teuthology.orchestra.run.smithi152.stdout: "cap": 0, 2024-03-16T12:08:32.647 INFO:teuthology.orchestra.run.smithi152.stdout: "cap+": 0, 2024-03-16T12:08:32.647 INFO:teuthology.orchestra.run.smithi152.stdout: "cap-": 0, 2024-03-16T12:08:32.647 INFO:teuthology.orchestra.run.smithi152.stdout: "rss": 50900, 2024-03-16T12:08:32.647 INFO:teuthology.orchestra.run.smithi152.stdout: "heap": 215420 2024-03-16T12:08:32.647 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-16T12:08:32.648 INFO:teuthology.orchestra.run.smithi152.stdout: "mds_server": { 2024-03-16T12:08:32.648 INFO:teuthology.orchestra.run.smithi152.stdout: "dispatch_client_request": 0, 2024-03-16T12:08:32.648 INFO:teuthology.orchestra.run.smithi152.stdout: "dispatch_server_request": 0, 2024-03-16T12:08:32.648 INFO:teuthology.orchestra.run.smithi152.stdout: "handle_client_request": 0, 2024-03-16T12:08:32.648 INFO:teuthology.orchestra.run.smithi152.stdout: "handle_client_session": 0, 2024-03-16T12:08:32.648 INFO:teuthology.orchestra.run.smithi152.stdout: "handle_peer_request": 0, 2024-03-16T12:08:32.648 INFO:teuthology.orchestra.run.smithi152.stdout: "req_create_latency": { 2024-03-16T12:08:32.648 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-03-16T12:08:32.648 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-03-16T12:08:32.649 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-03-16T12:08:32.649 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-16T12:08:32.649 INFO:teuthology.orchestra.run.smithi152.stdout: "req_getattr_latency": { 2024-03-16T12:08:32.649 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-03-16T12:08:32.649 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-03-16T12:08:32.649 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-03-16T12:08:32.649 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-16T12:08:32.649 INFO:teuthology.orchestra.run.smithi152.stdout: "req_getfilelock_latency": { 2024-03-16T12:08:32.649 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-03-16T12:08:32.649 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-03-16T12:08:32.650 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-03-16T12:08:32.650 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-16T12:08:32.650 INFO:teuthology.orchestra.run.smithi152.stdout: "req_link_latency": { 2024-03-16T12:08:32.650 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-03-16T12:08:32.650 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-03-16T12:08:32.650 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-03-16T12:08:32.650 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-16T12:08:32.650 INFO:teuthology.orchestra.run.smithi152.stdout: "req_lookup_latency": { 2024-03-16T12:08:32.650 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-03-16T12:08:32.651 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-03-16T12:08:32.651 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-03-16T12:08:32.651 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-16T12:08:32.651 INFO:teuthology.orchestra.run.smithi152.stdout: "req_lookuphash_latency": { 2024-03-16T12:08:32.651 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-03-16T12:08:32.651 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-03-16T12:08:32.651 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-03-16T12:08:32.651 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-16T12:08:32.651 INFO:teuthology.orchestra.run.smithi152.stdout: "req_lookupino_latency": { 2024-03-16T12:08:32.652 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-03-16T12:08:32.652 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-03-16T12:08:32.652 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-03-16T12:08:32.652 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-16T12:08:32.652 INFO:teuthology.orchestra.run.smithi152.stdout: "req_lookupname_latency": { 2024-03-16T12:08:32.652 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-03-16T12:08:32.652 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-03-16T12:08:32.652 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-03-16T12:08:32.652 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-16T12:08:32.653 INFO:teuthology.orchestra.run.smithi152.stdout: "req_lookupparent_latency": { 2024-03-16T12:08:32.653 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-03-16T12:08:32.653 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-03-16T12:08:32.653 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-03-16T12:08:32.653 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-16T12:08:32.653 INFO:teuthology.orchestra.run.smithi152.stdout: "req_lookupsnap_latency": { 2024-03-16T12:08:32.653 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-03-16T12:08:32.653 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-03-16T12:08:32.653 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-03-16T12:08:32.654 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-16T12:08:32.654 INFO:teuthology.orchestra.run.smithi152.stdout: "req_lssnap_latency": { 2024-03-16T12:08:32.654 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-03-16T12:08:32.654 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-03-16T12:08:32.654 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-03-16T12:08:32.654 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-16T12:08:32.654 INFO:teuthology.orchestra.run.smithi152.stdout: "req_mkdir_latency": { 2024-03-16T12:08:32.655 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-03-16T12:08:32.655 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-03-16T12:08:32.655 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-03-16T12:08:32.655 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-16T12:08:32.655 INFO:teuthology.orchestra.run.smithi152.stdout: "req_mknod_latency": { 2024-03-16T12:08:32.656 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-03-16T12:08:32.656 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-03-16T12:08:32.656 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-03-16T12:08:32.656 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-16T12:08:32.656 INFO:teuthology.orchestra.run.smithi152.stdout: "req_mksnap_latency": { 2024-03-16T12:08:32.656 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-03-16T12:08:32.656 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-03-16T12:08:32.656 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-03-16T12:08:32.656 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-16T12:08:32.656 INFO:teuthology.orchestra.run.smithi152.stdout: "req_open_latency": { 2024-03-16T12:08:32.657 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-03-16T12:08:32.657 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-03-16T12:08:32.657 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-03-16T12:08:32.657 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-16T12:08:32.657 INFO:teuthology.orchestra.run.smithi152.stdout: "req_readdir_latency": { 2024-03-16T12:08:32.657 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-03-16T12:08:32.657 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-03-16T12:08:32.657 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-03-16T12:08:32.657 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-16T12:08:32.658 INFO:teuthology.orchestra.run.smithi152.stdout: "req_rename_latency": { 2024-03-16T12:08:32.658 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-03-16T12:08:32.658 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-03-16T12:08:32.658 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-03-16T12:08:32.658 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-16T12:08:32.658 INFO:teuthology.orchestra.run.smithi152.stdout: "req_renamesnap_latency": { 2024-03-16T12:08:32.658 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-03-16T12:08:32.658 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-03-16T12:08:32.659 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-03-16T12:08:32.659 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-16T12:08:32.659 INFO:teuthology.orchestra.run.smithi152.stdout: "req_snapdiff_latency": { 2024-03-16T12:08:32.659 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-03-16T12:08:32.659 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-03-16T12:08:32.659 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-03-16T12:08:32.659 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-16T12:08:32.659 INFO:teuthology.orchestra.run.smithi152.stdout: "req_rmdir_latency": { 2024-03-16T12:08:32.659 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-03-16T12:08:32.660 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-03-16T12:08:32.660 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-03-16T12:08:32.660 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-16T12:08:32.660 INFO:teuthology.orchestra.run.smithi152.stdout: "req_rmsnap_latency": { 2024-03-16T12:08:32.660 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-03-16T12:08:32.660 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-03-16T12:08:32.660 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-03-16T12:08:32.660 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-16T12:08:32.660 INFO:teuthology.orchestra.run.smithi152.stdout: "req_rmxattr_latency": { 2024-03-16T12:08:32.661 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-03-16T12:08:32.661 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-03-16T12:08:32.661 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-03-16T12:08:32.661 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-16T12:08:32.661 INFO:teuthology.orchestra.run.smithi152.stdout: "req_setattr_latency": { 2024-03-16T12:08:32.661 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-03-16T12:08:32.661 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-03-16T12:08:32.661 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-03-16T12:08:32.661 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-16T12:08:32.662 INFO:teuthology.orchestra.run.smithi152.stdout: "req_setdirlayout_latency": { 2024-03-16T12:08:32.662 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-03-16T12:08:32.662 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-03-16T12:08:32.662 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-03-16T12:08:32.662 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-16T12:08:32.662 INFO:teuthology.orchestra.run.smithi152.stdout: "req_setfilelock_latency": { 2024-03-16T12:08:32.662 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-03-16T12:08:32.662 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-03-16T12:08:32.662 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-03-16T12:08:32.663 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-16T12:08:32.663 INFO:teuthology.orchestra.run.smithi152.stdout: "req_setlayout_latency": { 2024-03-16T12:08:32.663 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-03-16T12:08:32.663 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-03-16T12:08:32.663 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-03-16T12:08:32.663 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-16T12:08:32.663 INFO:teuthology.orchestra.run.smithi152.stdout: "req_setxattr_latency": { 2024-03-16T12:08:32.663 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-03-16T12:08:32.663 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-03-16T12:08:32.664 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-03-16T12:08:32.664 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-16T12:08:32.664 INFO:teuthology.orchestra.run.smithi152.stdout: "req_symlink_latency": { 2024-03-16T12:08:32.664 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-03-16T12:08:32.664 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-03-16T12:08:32.664 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-03-16T12:08:32.664 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-16T12:08:32.664 INFO:teuthology.orchestra.run.smithi152.stdout: "req_unlink_latency": { 2024-03-16T12:08:32.665 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-03-16T12:08:32.665 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-03-16T12:08:32.665 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-03-16T12:08:32.665 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-16T12:08:32.665 INFO:teuthology.orchestra.run.smithi152.stdout: "cap_revoke_eviction": 0, 2024-03-16T12:08:32.665 INFO:teuthology.orchestra.run.smithi152.stdout: "cap_acquisition_throttle": 0, 2024-03-16T12:08:32.665 INFO:teuthology.orchestra.run.smithi152.stdout: "req_getvxattr_latency": { 2024-03-16T12:08:32.665 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-03-16T12:08:32.665 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-03-16T12:08:32.665 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-03-16T12:08:32.666 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-03-16T12:08:32.666 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-16T12:08:32.666 INFO:teuthology.orchestra.run.smithi152.stdout: "mds_sessions": { 2024-03-16T12:08:32.666 INFO:teuthology.orchestra.run.smithi152.stdout: "session_count": 1, 2024-03-16T12:08:32.666 INFO:teuthology.orchestra.run.smithi152.stdout: "session_add": 5, 2024-03-16T12:08:32.666 INFO:teuthology.orchestra.run.smithi152.stdout: "session_remove": 4, 2024-03-16T12:08:32.666 INFO:teuthology.orchestra.run.smithi152.stdout: "sessions_open": 4, 2024-03-16T12:08:32.666 INFO:teuthology.orchestra.run.smithi152.stdout: "sessions_stale": 0, 2024-03-16T12:08:32.666 INFO:teuthology.orchestra.run.smithi152.stdout: "total_load": 0, 2024-03-16T12:08:32.667 INFO:teuthology.orchestra.run.smithi152.stdout: "average_load": 0, 2024-03-16T12:08:32.667 INFO:teuthology.orchestra.run.smithi152.stdout: "avg_session_uptime": 6891, 2024-03-16T12:08:32.667 INFO:teuthology.orchestra.run.smithi152.stdout: "mdthresh_evicted": 0 2024-03-16T12:08:32.667 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-16T12:08:32.667 INFO:teuthology.orchestra.run.smithi152.stdout: "mempool": { 2024-03-16T12:08:32.667 INFO:teuthology.orchestra.run.smithi152.stdout: "bloom_filter_bytes": 0, 2024-03-16T12:08:32.667 INFO:teuthology.orchestra.run.smithi152.stdout: "bloom_filter_items": 0, 2024-03-16T12:08:32.668 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_alloc_bytes": 0, 2024-03-16T12:08:32.668 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_alloc_items": 0, 2024-03-16T12:08:32.668 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_cache_data_bytes": 0, 2024-03-16T12:08:32.668 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_cache_data_items": 0, 2024-03-16T12:08:32.668 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-16T12:08:32.668 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_cache_onode_items": 0, 2024-03-16T12:08:32.668 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-16T12:08:32.668 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_cache_meta_items": 0, 2024-03-16T12:08:32.668 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_cache_other_bytes": 0, 2024-03-16T12:08:32.669 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_cache_other_items": 0, 2024-03-16T12:08:32.669 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-16T12:08:32.669 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_cache_buffer_items": 0, 2024-03-16T12:08:32.669 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_extent_bytes": 0, 2024-03-16T12:08:32.669 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_extent_items": 0, 2024-03-16T12:08:32.669 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_blob_bytes": 0, 2024-03-16T12:08:32.669 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_blob_items": 0, 2024-03-16T12:08:32.669 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-16T12:08:32.669 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_shared_blob_items": 0, 2024-03-16T12:08:32.670 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-16T12:08:32.670 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_inline_bl_items": 0, 2024-03-16T12:08:32.670 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_fsck_bytes": 0, 2024-03-16T12:08:32.670 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_fsck_items": 0, 2024-03-16T12:08:32.670 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_txc_bytes": 0, 2024-03-16T12:08:32.670 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_txc_items": 0, 2024-03-16T12:08:32.670 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-16T12:08:32.670 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_writing_deferred_items": 0, 2024-03-16T12:08:32.670 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_writing_bytes": 0, 2024-03-16T12:08:32.671 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_writing_items": 0, 2024-03-16T12:08:32.671 INFO:teuthology.orchestra.run.smithi152.stdout: "bluefs_bytes": 0, 2024-03-16T12:08:32.671 INFO:teuthology.orchestra.run.smithi152.stdout: "bluefs_items": 0, 2024-03-16T12:08:32.671 INFO:teuthology.orchestra.run.smithi152.stdout: "bluefs_file_reader_bytes": 0, 2024-03-16T12:08:32.671 INFO:teuthology.orchestra.run.smithi152.stdout: "bluefs_file_reader_items": 0, 2024-03-16T12:08:32.671 INFO:teuthology.orchestra.run.smithi152.stdout: "bluefs_file_writer_bytes": 0, 2024-03-16T12:08:32.671 INFO:teuthology.orchestra.run.smithi152.stdout: "bluefs_file_writer_items": 0, 2024-03-16T12:08:32.671 INFO:teuthology.orchestra.run.smithi152.stdout: "buffer_anon_bytes": 192440, 2024-03-16T12:08:32.671 INFO:teuthology.orchestra.run.smithi152.stdout: "buffer_anon_items": 114, 2024-03-16T12:08:32.672 INFO:teuthology.orchestra.run.smithi152.stdout: "buffer_meta_bytes": 0, 2024-03-16T12:08:32.672 INFO:teuthology.orchestra.run.smithi152.stdout: "buffer_meta_items": 0, 2024-03-16T12:08:32.672 INFO:teuthology.orchestra.run.smithi152.stdout: "osd_bytes": 0, 2024-03-16T12:08:32.672 INFO:teuthology.orchestra.run.smithi152.stdout: "osd_items": 0, 2024-03-16T12:08:32.672 INFO:teuthology.orchestra.run.smithi152.stdout: "osd_mapbl_bytes": 0, 2024-03-16T12:08:32.672 INFO:teuthology.orchestra.run.smithi152.stdout: "osd_mapbl_items": 0, 2024-03-16T12:08:32.672 INFO:teuthology.orchestra.run.smithi152.stdout: "osd_pglog_bytes": 0, 2024-03-16T12:08:32.672 INFO:teuthology.orchestra.run.smithi152.stdout: "osd_pglog_items": 0, 2024-03-16T12:08:32.672 INFO:teuthology.orchestra.run.smithi152.stdout: "osdmap_bytes": 8256, 2024-03-16T12:08:32.672 INFO:teuthology.orchestra.run.smithi152.stdout: "osdmap_items": 188, 2024-03-16T12:08:32.673 INFO:teuthology.orchestra.run.smithi152.stdout: "osdmap_mapping_bytes": 0, 2024-03-16T12:08:32.673 INFO:teuthology.orchestra.run.smithi152.stdout: "osdmap_mapping_items": 0, 2024-03-16T12:08:32.673 INFO:teuthology.orchestra.run.smithi152.stdout: "pgmap_bytes": 0, 2024-03-16T12:08:32.673 INFO:teuthology.orchestra.run.smithi152.stdout: "pgmap_items": 0, 2024-03-16T12:08:32.673 INFO:teuthology.orchestra.run.smithi152.stdout: "mds_co_bytes": 490139, 2024-03-16T12:08:32.673 INFO:teuthology.orchestra.run.smithi152.stdout: "mds_co_items": 4077, 2024-03-16T12:08:32.673 INFO:teuthology.orchestra.run.smithi152.stdout: "unittest_1_bytes": 0, 2024-03-16T12:08:32.673 INFO:teuthology.orchestra.run.smithi152.stdout: "unittest_1_items": 0, 2024-03-16T12:08:32.673 INFO:teuthology.orchestra.run.smithi152.stdout: "unittest_2_bytes": 0, 2024-03-16T12:08:32.674 INFO:teuthology.orchestra.run.smithi152.stdout: "unittest_2_items": 0 2024-03-16T12:08:32.674 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-16T12:08:32.674 INFO:teuthology.orchestra.run.smithi152.stdout: "objecter": { 2024-03-16T12:08:32.674 INFO:teuthology.orchestra.run.smithi152.stdout: "op_active": 0, 2024-03-16T12:08:32.674 INFO:teuthology.orchestra.run.smithi152.stdout: "op_laggy": 0, 2024-03-16T12:08:32.674 INFO:teuthology.orchestra.run.smithi152.stdout: "op_send": 3592, 2024-03-16T12:08:32.674 INFO:teuthology.orchestra.run.smithi152.stdout: "op_send_bytes": 21, 2024-03-16T12:08:32.674 INFO:teuthology.orchestra.run.smithi152.stdout: "op_resend": 3, 2024-03-16T12:08:32.675 INFO:teuthology.orchestra.run.smithi152.stdout: "op_reply": 3592, 2024-03-16T12:08:32.675 INFO:teuthology.orchestra.run.smithi152.stdout: "op_latency": { 2024-03-16T12:08:32.675 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 3592, 2024-03-16T12:08:32.675 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 3.128961382, 2024-03-16T12:08:32.675 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000871091 2024-03-16T12:08:32.675 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-16T12:08:32.675 INFO:teuthology.orchestra.run.smithi152.stdout: "op_inflight": 0, 2024-03-16T12:08:32.675 INFO:teuthology.orchestra.run.smithi152.stdout: "oplen_avg": { 2024-03-16T12:08:32.675 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 3592, 2024-03-16T12:08:32.676 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 3593 2024-03-16T12:08:32.676 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-16T12:08:32.676 INFO:teuthology.orchestra.run.smithi152.stdout: "op": 3592, 2024-03-16T12:08:32.676 INFO:teuthology.orchestra.run.smithi152.stdout: "op_r": 3592, 2024-03-16T12:08:32.676 INFO:teuthology.orchestra.run.smithi152.stdout: "op_w": 0, 2024-03-16T12:08:32.676 INFO:teuthology.orchestra.run.smithi152.stdout: "op_rmw": 0, 2024-03-16T12:08:32.676 INFO:teuthology.orchestra.run.smithi152.stdout: "op_pg": 0, 2024-03-16T12:08:32.676 INFO:teuthology.orchestra.run.smithi152.stdout: "osdop_stat": 2299, 2024-03-16T12:08:32.676 INFO:teuthology.orchestra.run.smithi152.stdout: "osdop_create": 0, 2024-03-16T12:08:32.677 INFO:teuthology.orchestra.run.smithi152.stdout: "osdop_read": 1291, 2024-03-16T12:08:32.677 INFO:teuthology.orchestra.run.smithi152.stdout: "osdop_write": 0, 2024-03-16T12:08:32.677 INFO:teuthology.orchestra.run.smithi152.stdout: "osdop_writefull": 0, 2024-03-16T12:08:32.677 INFO:teuthology.orchestra.run.smithi152.stdout: "osdop_writesame": 0, 2024-03-16T12:08:32.677 INFO:teuthology.orchestra.run.smithi152.stdout: "osdop_append": 0, 2024-03-16T12:08:32.677 INFO:teuthology.orchestra.run.smithi152.stdout: "osdop_zero": 0, 2024-03-16T12:08:32.677 INFO:teuthology.orchestra.run.smithi152.stdout: "osdop_truncate": 0, 2024-03-16T12:08:32.677 INFO:teuthology.orchestra.run.smithi152.stdout: "osdop_delete": 0, 2024-03-16T12:08:32.677 INFO:teuthology.orchestra.run.smithi152.stdout: "osdop_mapext": 0, 2024-03-16T12:08:32.678 INFO:teuthology.orchestra.run.smithi152.stdout: "osdop_sparse_read": 0, 2024-03-16T12:08:32.678 INFO:teuthology.orchestra.run.smithi152.stdout: "osdop_clonerange": 0, 2024-03-16T12:08:32.678 INFO:teuthology.orchestra.run.smithi152.stdout: "osdop_getxattr": 1, 2024-03-16T12:08:32.678 INFO:teuthology.orchestra.run.smithi152.stdout: "osdop_setxattr": 0, 2024-03-16T12:08:32.678 INFO:teuthology.orchestra.run.smithi152.stdout: "osdop_cmpxattr": 0, 2024-03-16T12:08:32.678 INFO:teuthology.orchestra.run.smithi152.stdout: "osdop_rmxattr": 0, 2024-03-16T12:08:32.678 INFO:teuthology.orchestra.run.smithi152.stdout: "osdop_resetxattrs": 0, 2024-03-16T12:08:32.678 INFO:teuthology.orchestra.run.smithi152.stdout: "osdop_call": 0, 2024-03-16T12:08:32.678 INFO:teuthology.orchestra.run.smithi152.stdout: "osdop_watch": 0, 2024-03-16T12:08:32.679 INFO:teuthology.orchestra.run.smithi152.stdout: "osdop_notify": 0, 2024-03-16T12:08:32.679 INFO:teuthology.orchestra.run.smithi152.stdout: "osdop_src_cmpxattr": 0, 2024-03-16T12:08:32.679 INFO:teuthology.orchestra.run.smithi152.stdout: "osdop_pgls": 0, 2024-03-16T12:08:32.679 INFO:teuthology.orchestra.run.smithi152.stdout: "osdop_pgls_filter": 0, 2024-03-16T12:08:32.679 INFO:teuthology.orchestra.run.smithi152.stdout: "osdop_other": 0, 2024-03-16T12:08:32.679 INFO:teuthology.orchestra.run.smithi152.stdout: "linger_active": 0, 2024-03-16T12:08:32.679 INFO:teuthology.orchestra.run.smithi152.stdout: "linger_send": 0, 2024-03-16T12:08:32.679 INFO:teuthology.orchestra.run.smithi152.stdout: "linger_resend": 0, 2024-03-16T12:08:32.679 INFO:teuthology.orchestra.run.smithi152.stdout: "linger_ping": 0, 2024-03-16T12:08:32.680 INFO:teuthology.orchestra.run.smithi152.stdout: "poolop_active": 0, 2024-03-16T12:08:32.680 INFO:teuthology.orchestra.run.smithi152.stdout: "poolop_send": 0, 2024-03-16T12:08:32.680 INFO:teuthology.orchestra.run.smithi152.stdout: "poolop_resend": 0, 2024-03-16T12:08:32.680 INFO:teuthology.orchestra.run.smithi152.stdout: "poolstat_active": 0, 2024-03-16T12:08:32.680 INFO:teuthology.orchestra.run.smithi152.stdout: "poolstat_send": 0, 2024-03-16T12:08:32.680 INFO:teuthology.orchestra.run.smithi152.stdout: "poolstat_resend": 0, 2024-03-16T12:08:32.680 INFO:teuthology.orchestra.run.smithi152.stdout: "statfs_active": 0, 2024-03-16T12:08:32.680 INFO:teuthology.orchestra.run.smithi152.stdout: "statfs_send": 0, 2024-03-16T12:08:32.680 INFO:teuthology.orchestra.run.smithi152.stdout: "statfs_resend": 0, 2024-03-16T12:08:32.680 INFO:teuthology.orchestra.run.smithi152.stdout: "command_active": 0, 2024-03-16T12:08:32.681 INFO:teuthology.orchestra.run.smithi152.stdout: "command_send": 0, 2024-03-16T12:08:32.681 INFO:teuthology.orchestra.run.smithi152.stdout: "command_resend": 0, 2024-03-16T12:08:32.681 INFO:teuthology.orchestra.run.smithi152.stdout: "map_epoch": 116, 2024-03-16T12:08:32.681 INFO:teuthology.orchestra.run.smithi152.stdout: "map_full": 0, 2024-03-16T12:08:32.681 INFO:teuthology.orchestra.run.smithi152.stdout: "map_inc": 31, 2024-03-16T12:08:32.681 INFO:teuthology.orchestra.run.smithi152.stdout: "osd_sessions": 5, 2024-03-16T12:08:32.681 INFO:teuthology.orchestra.run.smithi152.stdout: "osd_session_open": 6, 2024-03-16T12:08:32.681 INFO:teuthology.orchestra.run.smithi152.stdout: "osd_session_close": 1, 2024-03-16T12:08:32.681 INFO:teuthology.orchestra.run.smithi152.stdout: "osd_laggy": 0, 2024-03-16T12:08:32.682 INFO:teuthology.orchestra.run.smithi152.stdout: "omap_wr": 0, 2024-03-16T12:08:32.682 INFO:teuthology.orchestra.run.smithi152.stdout: "omap_rd": 2, 2024-03-16T12:08:32.682 INFO:teuthology.orchestra.run.smithi152.stdout: "omap_del": 0 2024-03-16T12:08:32.682 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-16T12:08:32.682 INFO:teuthology.orchestra.run.smithi152.stdout: "oft": { 2024-03-16T12:08:32.682 INFO:teuthology.orchestra.run.smithi152.stdout: "omap_total_objs": 0, 2024-03-16T12:08:32.682 INFO:teuthology.orchestra.run.smithi152.stdout: "omap_total_kv_pairs": 0, 2024-03-16T12:08:32.682 INFO:teuthology.orchestra.run.smithi152.stdout: "omap_total_updates": 0, 2024-03-16T12:08:32.682 INFO:teuthology.orchestra.run.smithi152.stdout: "omap_total_removes": 0 2024-03-16T12:08:32.682 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-16T12:08:32.683 INFO:teuthology.orchestra.run.smithi152.stdout: "osd-slow-ops": { 2024-03-16T12:08:32.683 INFO:teuthology.orchestra.run.smithi152.stdout: "slow_ops_count": 0 2024-03-16T12:08:32.683 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-16T12:08:32.683 INFO:teuthology.orchestra.run.smithi152.stdout: "purge_queue": { 2024-03-16T12:08:32.683 INFO:teuthology.orchestra.run.smithi152.stdout: "pq_executing_ops": 0, 2024-03-16T12:08:32.683 INFO:teuthology.orchestra.run.smithi152.stdout: "pq_executing_ops_high_water": 0, 2024-03-16T12:08:32.683 INFO:teuthology.orchestra.run.smithi152.stdout: "pq_executing": 0, 2024-03-16T12:08:32.683 INFO:teuthology.orchestra.run.smithi152.stdout: "pq_executing_high_water": 0, 2024-03-16T12:08:32.683 INFO:teuthology.orchestra.run.smithi152.stdout: "pq_executed_ops": 0, 2024-03-16T12:08:32.684 INFO:teuthology.orchestra.run.smithi152.stdout: "pq_executed": 0, 2024-03-16T12:08:32.684 INFO:teuthology.orchestra.run.smithi152.stdout: "pq_item_in_journal": 0 2024-03-16T12:08:32.684 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-16T12:08:32.684 INFO:teuthology.orchestra.run.smithi152.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-16T12:08:32.684 INFO:teuthology.orchestra.run.smithi152.stdout: "val": 0, 2024-03-16T12:08:32.684 INFO:teuthology.orchestra.run.smithi152.stdout: "max": 104857600, 2024-03-16T12:08:32.684 INFO:teuthology.orchestra.run.smithi152.stdout: "get_started": 0, 2024-03-16T12:08:32.684 INFO:teuthology.orchestra.run.smithi152.stdout: "get": 3975, 2024-03-16T12:08:32.684 INFO:teuthology.orchestra.run.smithi152.stdout: "get_sum": 6106629, 2024-03-16T12:08:32.685 INFO:teuthology.orchestra.run.smithi152.stdout: "get_or_fail_fail": 0, 2024-03-16T12:08:32.685 INFO:teuthology.orchestra.run.smithi152.stdout: "get_or_fail_success": 3975, 2024-03-16T12:08:32.685 INFO:teuthology.orchestra.run.smithi152.stdout: "take": 0, 2024-03-16T12:08:32.685 INFO:teuthology.orchestra.run.smithi152.stdout: "take_sum": 0, 2024-03-16T12:08:32.685 INFO:teuthology.orchestra.run.smithi152.stdout: "put": 3975, 2024-03-16T12:08:32.685 INFO:teuthology.orchestra.run.smithi152.stdout: "put_sum": 6106629, 2024-03-16T12:08:32.685 INFO:teuthology.orchestra.run.smithi152.stdout: "wait": { 2024-03-16T12:08:32.685 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-03-16T12:08:32.685 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-03-16T12:08:32.686 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-03-16T12:08:32.686 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-03-16T12:08:32.686 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-16T12:08:32.686 INFO:teuthology.orchestra.run.smithi152.stdout: "throttle-objecter_bytes": { 2024-03-16T12:08:32.686 INFO:teuthology.orchestra.run.smithi152.stdout: "val": 0, 2024-03-16T12:08:32.686 INFO:teuthology.orchestra.run.smithi152.stdout: "max": 104857600, 2024-03-16T12:08:32.686 INFO:teuthology.orchestra.run.smithi152.stdout: "get_started": 0, 2024-03-16T12:08:32.686 INFO:teuthology.orchestra.run.smithi152.stdout: "get": 0, 2024-03-16T12:08:32.686 INFO:teuthology.orchestra.run.smithi152.stdout: "get_sum": 0, 2024-03-16T12:08:32.687 INFO:teuthology.orchestra.run.smithi152.stdout: "get_or_fail_fail": 0, 2024-03-16T12:08:32.687 INFO:teuthology.orchestra.run.smithi152.stdout: "get_or_fail_success": 0, 2024-03-16T12:08:32.687 INFO:teuthology.orchestra.run.smithi152.stdout: "take": 3592, 2024-03-16T12:08:32.687 INFO:teuthology.orchestra.run.smithi152.stdout: "take_sum": 3925688, 2024-03-16T12:08:32.687 INFO:teuthology.orchestra.run.smithi152.stdout: "put": 3592, 2024-03-16T12:08:32.687 INFO:teuthology.orchestra.run.smithi152.stdout: "put_sum": 3925688, 2024-03-16T12:08:32.687 INFO:teuthology.orchestra.run.smithi152.stdout: "wait": { 2024-03-16T12:08:32.687 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-03-16T12:08:32.687 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-03-16T12:08:32.688 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-03-16T12:08:32.688 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-03-16T12:08:32.688 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-16T12:08:32.688 INFO:teuthology.orchestra.run.smithi152.stdout: "throttle-objecter_ops": { 2024-03-16T12:08:32.688 INFO:teuthology.orchestra.run.smithi152.stdout: "val": 0, 2024-03-16T12:08:32.688 INFO:teuthology.orchestra.run.smithi152.stdout: "max": 1024, 2024-03-16T12:08:32.688 INFO:teuthology.orchestra.run.smithi152.stdout: "get_started": 0, 2024-03-16T12:08:32.688 INFO:teuthology.orchestra.run.smithi152.stdout: "get": 0, 2024-03-16T12:08:32.688 INFO:teuthology.orchestra.run.smithi152.stdout: "get_sum": 0, 2024-03-16T12:08:32.689 INFO:teuthology.orchestra.run.smithi152.stdout: "get_or_fail_fail": 0, 2024-03-16T12:08:32.689 INFO:teuthology.orchestra.run.smithi152.stdout: "get_or_fail_success": 0, 2024-03-16T12:08:32.689 INFO:teuthology.orchestra.run.smithi152.stdout: "take": 3592, 2024-03-16T12:08:32.689 INFO:teuthology.orchestra.run.smithi152.stdout: "take_sum": 3592, 2024-03-16T12:08:32.689 INFO:teuthology.orchestra.run.smithi152.stdout: "put": 3592, 2024-03-16T12:08:32.689 INFO:teuthology.orchestra.run.smithi152.stdout: "put_sum": 3592, 2024-03-16T12:08:32.689 INFO:teuthology.orchestra.run.smithi152.stdout: "wait": { 2024-03-16T12:08:32.689 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-03-16T12:08:32.689 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-03-16T12:08:32.690 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-03-16T12:08:32.690 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-03-16T12:08:32.690 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-16T12:08:32.690 INFO:teuthology.orchestra.run.smithi152.stdout: "throttle-write_buf_throttle": { 2024-03-16T12:08:32.690 INFO:teuthology.orchestra.run.smithi152.stdout: "val": 0, 2024-03-16T12:08:32.690 INFO:teuthology.orchestra.run.smithi152.stdout: "max": 3758096384, 2024-03-16T12:08:32.690 INFO:teuthology.orchestra.run.smithi152.stdout: "get_started": 0, 2024-03-16T12:08:32.690 INFO:teuthology.orchestra.run.smithi152.stdout: "get": 0, 2024-03-16T12:08:32.690 INFO:teuthology.orchestra.run.smithi152.stdout: "get_sum": 0, 2024-03-16T12:08:32.691 INFO:teuthology.orchestra.run.smithi152.stdout: "get_or_fail_fail": 0, 2024-03-16T12:08:32.691 INFO:teuthology.orchestra.run.smithi152.stdout: "get_or_fail_success": 0, 2024-03-16T12:08:32.691 INFO:teuthology.orchestra.run.smithi152.stdout: "take": 0, 2024-03-16T12:08:32.691 INFO:teuthology.orchestra.run.smithi152.stdout: "take_sum": 0, 2024-03-16T12:08:32.691 INFO:teuthology.orchestra.run.smithi152.stdout: "put": 0, 2024-03-16T12:08:32.691 INFO:teuthology.orchestra.run.smithi152.stdout: "put_sum": 0, 2024-03-16T12:08:32.691 INFO:teuthology.orchestra.run.smithi152.stdout: "wait": { 2024-03-16T12:08:32.691 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-03-16T12:08:32.691 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-03-16T12:08:32.692 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-03-16T12:08:32.692 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-03-16T12:08:32.692 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-16T12:08:32.692 INFO:teuthology.orchestra.run.smithi152.stdout: "throttle-write_buf_throttle-0x55ff93c026e0": { 2024-03-16T12:08:32.692 INFO:teuthology.orchestra.run.smithi152.stdout: "val": 0, 2024-03-16T12:08:32.692 INFO:teuthology.orchestra.run.smithi152.stdout: "max": 3758096384, 2024-03-16T12:08:32.692 INFO:teuthology.orchestra.run.smithi152.stdout: "get_started": 0, 2024-03-16T12:08:32.692 INFO:teuthology.orchestra.run.smithi152.stdout: "get": 0, 2024-03-16T12:08:32.692 INFO:teuthology.orchestra.run.smithi152.stdout: "get_sum": 0, 2024-03-16T12:08:32.693 INFO:teuthology.orchestra.run.smithi152.stdout: "get_or_fail_fail": 0, 2024-03-16T12:08:32.693 INFO:teuthology.orchestra.run.smithi152.stdout: "get_or_fail_success": 0, 2024-03-16T12:08:32.693 INFO:teuthology.orchestra.run.smithi152.stdout: "take": 0, 2024-03-16T12:08:32.693 INFO:teuthology.orchestra.run.smithi152.stdout: "take_sum": 0, 2024-03-16T12:08:32.693 INFO:teuthology.orchestra.run.smithi152.stdout: "put": 0, 2024-03-16T12:08:32.693 INFO:teuthology.orchestra.run.smithi152.stdout: "put_sum": 0, 2024-03-16T12:08:32.693 INFO:teuthology.orchestra.run.smithi152.stdout: "wait": { 2024-03-16T12:08:32.693 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-03-16T12:08:32.693 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-03-16T12:08:32.694 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-03-16T12:08:32.694 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-03-16T12:08:32.694 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-03-16T12:08:32.694 INFO:teuthology.orchestra.run.smithi152.stdout:} 2024-03-16T12:08:32.785 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:08:32 smithi154 ceph-mon[36320]: pgmap v559: 97 pgs: 97 active+clean; 356 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 198 KiB/s rd, 2.8 MiB/s wr, 107 op/s 2024-03-16T12:08:32.829 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:08:32 smithi029 ceph-mon[31975]: pgmap v559: 97 pgs: 97 active+clean; 356 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 198 KiB/s rd, 2.8 MiB/s wr, 107 op/s 2024-03-16T12:08:33.499 INFO:tasks.check_counter:Daemon mds.e mds.exported=0 2024-03-16T12:08:33.500 INFO:tasks.check_counter:Daemon mds.e mds.imported=0 2024-03-16T12:08:33.500 INFO:tasks.check_counter:Daemon mds.e mds_cache.dir_update=0 2024-03-16T12:08:33.500 INFO:tasks.check_counter:Daemon mds.e mds_cache.dir_update_receipt=0 2024-03-16T12:08:33.500 INFO:tasks.check_counter:Daemon mds.e mds.root_rsnaps=0 2024-03-16T12:08:33.500 INFO:tasks.check_counter:Daemon mds.e mds_server.req_mksnap_latency.avgcount=0 2024-03-16T12:08:33.500 INFO:tasks.check_counter:Daemon mds.e mds_server.req_rmsnap_latency.avgcount=0 2024-03-16T12:08:33.501 DEBUG:tasks.check_counter:Getting stats from h 2024-03-16T12:08:33.501 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid d1887f28-e389-11ee-95c9-87774f69a715 -- ceph daemon mds.h perf dump 2024-03-16T12:08:33.782 INFO:teuthology.orchestra.run.smithi152.stderr:Inferring config /var/lib/ceph/d1887f28-e389-11ee-95c9-87774f69a715/mon.b/config 2024-03-16T12:08:34.618 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:08:34 smithi152 ceph-mon[36291]: pgmap v560: 97 pgs: 97 active+clean; 356 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 177 KiB/s rd, 2.3 MiB/s wr, 82 op/s 2024-03-16T12:08:34.790 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:08:34 smithi154 ceph-mon[36320]: pgmap v560: 97 pgs: 97 active+clean; 356 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 177 KiB/s rd, 2.3 MiB/s wr, 82 op/s 2024-03-16T12:08:34.828 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:08:34 smithi029 ceph-mon[31975]: pgmap v560: 97 pgs: 97 active+clean; 356 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 177 KiB/s rd, 2.3 MiB/s wr, 82 op/s 2024-03-16T12:08:35.608 INFO:teuthology.orchestra.run.smithi152.stdout:{ 2024-03-16T12:08:35.608 INFO:teuthology.orchestra.run.smithi152.stdout: "AsyncMessenger::Worker-0": { 2024-03-16T12:08:35.608 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_recv_messages": 46970, 2024-03-16T12:08:35.608 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_send_messages": 48842, 2024-03-16T12:08:35.609 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_recv_bytes": 11904175, 2024-03-16T12:08:35.609 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_send_bytes": 11626194, 2024-03-16T12:08:35.609 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_created_connections": 16, 2024-03-16T12:08:35.609 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_active_connections": 4, 2024-03-16T12:08:35.609 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_running_total_time": 2.100629102, 2024-03-16T12:08:35.609 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_running_send_time": 0.937548818, 2024-03-16T12:08:35.609 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_running_recv_time": 1.542133199, 2024-03-16T12:08:35.609 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_running_fast_dispatch_time": 0.031394886, 2024-03-16T12:08:35.609 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_send_messages_queue_lat": { 2024-03-16T12:08:35.610 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 48830, 2024-03-16T12:08:35.610 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 1.089627998, 2024-03-16T12:08:35.610 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000022314 2024-03-16T12:08:35.610 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-16T12:08:35.610 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_handle_ack_lat": { 2024-03-16T12:08:35.610 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 47358, 2024-03-16T12:08:35.610 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.018827376, 2024-03-16T12:08:35.610 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000397 2024-03-16T12:08:35.611 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-16T12:08:35.611 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_recv_encrypted_bytes": 4128, 2024-03-16T12:08:35.611 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_send_encrypted_bytes": 4736 2024-03-16T12:08:35.611 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-16T12:08:35.611 INFO:teuthology.orchestra.run.smithi152.stdout: "AsyncMessenger::Worker-1": { 2024-03-16T12:08:35.611 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_recv_messages": 993, 2024-03-16T12:08:35.611 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_send_messages": 965, 2024-03-16T12:08:35.611 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_recv_bytes": 1723726, 2024-03-16T12:08:35.611 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_send_bytes": 435916, 2024-03-16T12:08:35.612 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_created_connections": 10, 2024-03-16T12:08:35.612 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_active_connections": 7, 2024-03-16T12:08:35.612 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_running_total_time": 0.086564746, 2024-03-16T12:08:35.612 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_running_send_time": 0.027261651, 2024-03-16T12:08:35.612 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_running_recv_time": 0.059620396, 2024-03-16T12:08:35.612 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_running_fast_dispatch_time": 0.013274259, 2024-03-16T12:08:35.612 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_send_messages_queue_lat": { 2024-03-16T12:08:35.612 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 964, 2024-03-16T12:08:35.612 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.064343515, 2024-03-16T12:08:35.613 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000066746 2024-03-16T12:08:35.613 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-16T12:08:35.613 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_handle_ack_lat": { 2024-03-16T12:08:35.613 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 566, 2024-03-16T12:08:35.613 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000103707, 2024-03-16T12:08:35.613 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000183 2024-03-16T12:08:35.613 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-16T12:08:35.613 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_recv_encrypted_bytes": 1560528, 2024-03-16T12:08:35.613 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_send_encrypted_bytes": 212384 2024-03-16T12:08:35.614 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-16T12:08:35.614 INFO:teuthology.orchestra.run.smithi152.stdout: "AsyncMessenger::Worker-2": { 2024-03-16T12:08:35.614 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_recv_messages": 379140, 2024-03-16T12:08:35.614 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_send_messages": 192571, 2024-03-16T12:08:35.614 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_recv_bytes": 165010812, 2024-03-16T12:08:35.614 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_send_bytes": 32453336, 2024-03-16T12:08:35.614 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_created_connections": 13, 2024-03-16T12:08:35.614 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_active_connections": 6, 2024-03-16T12:08:35.615 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_running_total_time": 17.148067877, 2024-03-16T12:08:35.615 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_running_send_time": 7.679037730, 2024-03-16T12:08:35.615 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_running_recv_time": 13.079058734, 2024-03-16T12:08:35.615 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_running_fast_dispatch_time": 0.011004143, 2024-03-16T12:08:35.615 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_send_messages_queue_lat": { 2024-03-16T12:08:35.615 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 192569, 2024-03-16T12:08:35.615 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 2.975969908, 2024-03-16T12:08:35.615 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000015454 2024-03-16T12:08:35.615 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-16T12:08:35.616 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_handle_ack_lat": { 2024-03-16T12:08:35.616 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 569623, 2024-03-16T12:08:35.616 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.077256785, 2024-03-16T12:08:35.616 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000135 2024-03-16T12:08:35.616 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-16T12:08:35.616 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_recv_encrypted_bytes": 5232, 2024-03-16T12:08:35.616 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_send_encrypted_bytes": 606128 2024-03-16T12:08:35.616 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-16T12:08:35.616 INFO:teuthology.orchestra.run.smithi152.stdout: "cct": { 2024-03-16T12:08:35.617 INFO:teuthology.orchestra.run.smithi152.stdout: "total_workers": 1, 2024-03-16T12:08:35.617 INFO:teuthology.orchestra.run.smithi152.stdout: "unhealthy_workers": 0 2024-03-16T12:08:35.617 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-16T12:08:35.617 INFO:teuthology.orchestra.run.smithi152.stdout: "finisher-MDSRank": { 2024-03-16T12:08:35.617 INFO:teuthology.orchestra.run.smithi152.stdout: "queue_len": 0, 2024-03-16T12:08:35.617 INFO:teuthology.orchestra.run.smithi152.stdout: "complete_latency": { 2024-03-16T12:08:35.617 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 1976, 2024-03-16T12:08:35.617 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 1.278509300, 2024-03-16T12:08:35.617 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000647018 2024-03-16T12:08:35.618 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-03-16T12:08:35.618 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-16T12:08:35.618 INFO:teuthology.orchestra.run.smithi152.stdout: "finisher-PurgeQueue": { 2024-03-16T12:08:35.618 INFO:teuthology.orchestra.run.smithi152.stdout: "queue_len": 0, 2024-03-16T12:08:35.618 INFO:teuthology.orchestra.run.smithi152.stdout: "complete_latency": { 2024-03-16T12:08:35.618 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 757, 2024-03-16T12:08:35.618 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.042252722, 2024-03-16T12:08:35.618 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000055816 2024-03-16T12:08:35.619 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-03-16T12:08:35.619 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-16T12:08:35.619 INFO:teuthology.orchestra.run.smithi152.stdout: "mds": { 2024-03-16T12:08:35.619 INFO:teuthology.orchestra.run.smithi152.stdout: "request": 44446, 2024-03-16T12:08:35.619 INFO:teuthology.orchestra.run.smithi152.stdout: "reply": 2130, 2024-03-16T12:08:35.619 INFO:teuthology.orchestra.run.smithi152.stdout: "reply_latency": { 2024-03-16T12:08:35.619 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 2130, 2024-03-16T12:08:35.619 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 16.042043427, 2024-03-16T12:08:35.619 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.007531475 2024-03-16T12:08:35.619 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-16T12:08:35.620 INFO:teuthology.orchestra.run.smithi152.stdout: "slow_reply": 0, 2024-03-16T12:08:35.620 INFO:teuthology.orchestra.run.smithi152.stdout: "forward": 42316, 2024-03-16T12:08:35.620 INFO:teuthology.orchestra.run.smithi152.stdout: "dir_fetch_complete": 0, 2024-03-16T12:08:35.620 INFO:teuthology.orchestra.run.smithi152.stdout: "dir_fetch_keys": 0, 2024-03-16T12:08:35.620 INFO:teuthology.orchestra.run.smithi152.stdout: "dir_commit": 11, 2024-03-16T12:08:35.620 INFO:teuthology.orchestra.run.smithi152.stdout: "dir_split": 0, 2024-03-16T12:08:35.620 INFO:teuthology.orchestra.run.smithi152.stdout: "dir_merge": 0, 2024-03-16T12:08:35.620 INFO:teuthology.orchestra.run.smithi152.stdout: "inodes": 195, 2024-03-16T12:08:35.620 INFO:teuthology.orchestra.run.smithi152.stdout: "inodes_top": 66, 2024-03-16T12:08:35.621 INFO:teuthology.orchestra.run.smithi152.stdout: "inodes_bottom": 129, 2024-03-16T12:08:35.621 INFO:teuthology.orchestra.run.smithi152.stdout: "inodes_pin_tail": 0, 2024-03-16T12:08:35.621 INFO:teuthology.orchestra.run.smithi152.stdout: "inodes_pinned": 100, 2024-03-16T12:08:35.621 INFO:teuthology.orchestra.run.smithi152.stdout: "inodes_expired": 0, 2024-03-16T12:08:35.621 INFO:teuthology.orchestra.run.smithi152.stdout: "inodes_with_caps": 22, 2024-03-16T12:08:35.621 INFO:teuthology.orchestra.run.smithi152.stdout: "caps": 22, 2024-03-16T12:08:35.621 INFO:teuthology.orchestra.run.smithi152.stdout: "subtrees": 4, 2024-03-16T12:08:35.621 INFO:teuthology.orchestra.run.smithi152.stdout: "traverse": 48476, 2024-03-16T12:08:35.621 INFO:teuthology.orchestra.run.smithi152.stdout: "traverse_hit": 3324, 2024-03-16T12:08:35.622 INFO:teuthology.orchestra.run.smithi152.stdout: "traverse_forward": 42312, 2024-03-16T12:08:35.622 INFO:teuthology.orchestra.run.smithi152.stdout: "traverse_discover": 105, 2024-03-16T12:08:35.622 INFO:teuthology.orchestra.run.smithi152.stdout: "traverse_dir_fetch": 0, 2024-03-16T12:08:35.622 INFO:teuthology.orchestra.run.smithi152.stdout: "traverse_remote_ino": 0, 2024-03-16T12:08:35.622 INFO:teuthology.orchestra.run.smithi152.stdout: "traverse_lock": 160, 2024-03-16T12:08:35.622 INFO:teuthology.orchestra.run.smithi152.stdout: "load_cent": 0, 2024-03-16T12:08:35.622 INFO:teuthology.orchestra.run.smithi152.stdout: "q": 0, 2024-03-16T12:08:35.622 INFO:teuthology.orchestra.run.smithi152.stdout: "exported": 0, 2024-03-16T12:08:35.623 INFO:teuthology.orchestra.run.smithi152.stdout: "exported_inodes": 0, 2024-03-16T12:08:35.623 INFO:teuthology.orchestra.run.smithi152.stdout: "imported": 2, 2024-03-16T12:08:35.623 INFO:teuthology.orchestra.run.smithi152.stdout: "imported_inodes": 15, 2024-03-16T12:08:35.623 INFO:teuthology.orchestra.run.smithi152.stdout: "openino_dir_fetch": 0, 2024-03-16T12:08:35.623 INFO:teuthology.orchestra.run.smithi152.stdout: "openino_backtrace_fetch": 0, 2024-03-16T12:08:35.623 INFO:teuthology.orchestra.run.smithi152.stdout: "openino_peer_discover": 0, 2024-03-16T12:08:35.623 INFO:teuthology.orchestra.run.smithi152.stdout: "root_rfiles": 669, 2024-03-16T12:08:35.623 INFO:teuthology.orchestra.run.smithi152.stdout: "root_rbytes": 211722202, 2024-03-16T12:08:35.623 INFO:teuthology.orchestra.run.smithi152.stdout: "root_rsnaps": 7, 2024-03-16T12:08:35.624 INFO:teuthology.orchestra.run.smithi152.stdout: "scrub_backtrace_fetch": 0, 2024-03-16T12:08:35.624 INFO:teuthology.orchestra.run.smithi152.stdout: "scrub_set_tag": 0, 2024-03-16T12:08:35.624 INFO:teuthology.orchestra.run.smithi152.stdout: "scrub_backtrace_repaired": 0, 2024-03-16T12:08:35.624 INFO:teuthology.orchestra.run.smithi152.stdout: "scrub_inotable_repaired": 0, 2024-03-16T12:08:35.624 INFO:teuthology.orchestra.run.smithi152.stdout: "scrub_dir_inodes": 0, 2024-03-16T12:08:35.624 INFO:teuthology.orchestra.run.smithi152.stdout: "scrub_dir_base_inodes": 0, 2024-03-16T12:08:35.624 INFO:teuthology.orchestra.run.smithi152.stdout: "scrub_dirfrag_rstats": 0, 2024-03-16T12:08:35.624 INFO:teuthology.orchestra.run.smithi152.stdout: "scrub_file_inodes": 0, 2024-03-16T12:08:35.624 INFO:teuthology.orchestra.run.smithi152.stdout: "handle_inode_file_caps": 0, 2024-03-16T12:08:35.625 INFO:teuthology.orchestra.run.smithi152.stdout: "ceph_cap_op_revoke": 35, 2024-03-16T12:08:35.625 INFO:teuthology.orchestra.run.smithi152.stdout: "ceph_cap_op_grant": 1482, 2024-03-16T12:08:35.625 INFO:teuthology.orchestra.run.smithi152.stdout: "ceph_cap_op_trunc": 0, 2024-03-16T12:08:35.625 INFO:teuthology.orchestra.run.smithi152.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-16T12:08:35.625 INFO:teuthology.orchestra.run.smithi152.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-16T12:08:35.625 INFO:teuthology.orchestra.run.smithi152.stdout: "handle_client_caps": 975, 2024-03-16T12:08:35.625 INFO:teuthology.orchestra.run.smithi152.stdout: "handle_client_caps_dirty": 630, 2024-03-16T12:08:35.625 INFO:teuthology.orchestra.run.smithi152.stdout: "handle_client_cap_release": 161, 2024-03-16T12:08:35.625 INFO:teuthology.orchestra.run.smithi152.stdout: "process_request_cap_release": 2182 2024-03-16T12:08:35.626 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-16T12:08:35.626 INFO:teuthology.orchestra.run.smithi152.stdout: "mds_cache": { 2024-03-16T12:08:35.626 INFO:teuthology.orchestra.run.smithi152.stdout: "dir_update": 51, 2024-03-16T12:08:35.626 INFO:teuthology.orchestra.run.smithi152.stdout: "dir_update_receipt": 481, 2024-03-16T12:08:35.626 INFO:teuthology.orchestra.run.smithi152.stdout: "dir_try_discover": 291, 2024-03-16T12:08:35.626 INFO:teuthology.orchestra.run.smithi152.stdout: "dir_send_discover": 388, 2024-03-16T12:08:35.626 INFO:teuthology.orchestra.run.smithi152.stdout: "dir_handle_discover": 76, 2024-03-16T12:08:35.626 INFO:teuthology.orchestra.run.smithi152.stdout: "num_strays": 8, 2024-03-16T12:08:35.626 INFO:teuthology.orchestra.run.smithi152.stdout: "num_strays_delayed": 0, 2024-03-16T12:08:35.627 INFO:teuthology.orchestra.run.smithi152.stdout: "num_strays_enqueuing": 0, 2024-03-16T12:08:35.627 INFO:teuthology.orchestra.run.smithi152.stdout: "strays_created": 352, 2024-03-16T12:08:35.627 INFO:teuthology.orchestra.run.smithi152.stdout: "strays_enqueued": 344, 2024-03-16T12:08:35.627 INFO:teuthology.orchestra.run.smithi152.stdout: "strays_reintegrated": 0, 2024-03-16T12:08:35.627 INFO:teuthology.orchestra.run.smithi152.stdout: "strays_migrated": 0, 2024-03-16T12:08:35.627 INFO:teuthology.orchestra.run.smithi152.stdout: "num_recovering_processing": 0, 2024-03-16T12:08:35.627 INFO:teuthology.orchestra.run.smithi152.stdout: "num_recovering_enqueued": 0, 2024-03-16T12:08:35.627 INFO:teuthology.orchestra.run.smithi152.stdout: "num_recovering_prioritized": 0, 2024-03-16T12:08:35.627 INFO:teuthology.orchestra.run.smithi152.stdout: "recovery_started": 0, 2024-03-16T12:08:35.628 INFO:teuthology.orchestra.run.smithi152.stdout: "recovery_completed": 0, 2024-03-16T12:08:35.628 INFO:teuthology.orchestra.run.smithi152.stdout: "ireq_quiesce_path": 0, 2024-03-16T12:08:35.628 INFO:teuthology.orchestra.run.smithi152.stdout: "ireq_quiesce_inode": 0, 2024-03-16T12:08:35.628 INFO:teuthology.orchestra.run.smithi152.stdout: "ireq_enqueue_scrub": 0, 2024-03-16T12:08:35.628 INFO:teuthology.orchestra.run.smithi152.stdout: "ireq_exportdir": 0, 2024-03-16T12:08:35.628 INFO:teuthology.orchestra.run.smithi152.stdout: "ireq_flush": 0, 2024-03-16T12:08:35.628 INFO:teuthology.orchestra.run.smithi152.stdout: "ireq_fragmentdir": 0, 2024-03-16T12:08:35.628 INFO:teuthology.orchestra.run.smithi152.stdout: "ireq_fragstats": 0, 2024-03-16T12:08:35.629 INFO:teuthology.orchestra.run.smithi152.stdout: "ireq_inodestats": 0 2024-03-16T12:08:35.629 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-16T12:08:35.629 INFO:teuthology.orchestra.run.smithi152.stdout: "mds_log": { 2024-03-16T12:08:35.629 INFO:teuthology.orchestra.run.smithi152.stdout: "evlrg": 0, 2024-03-16T12:08:35.629 INFO:teuthology.orchestra.run.smithi152.stdout: "evadd": 2743, 2024-03-16T12:08:35.629 INFO:teuthology.orchestra.run.smithi152.stdout: "evex": 0, 2024-03-16T12:08:35.629 INFO:teuthology.orchestra.run.smithi152.stdout: "evtrm": 0, 2024-03-16T12:08:35.629 INFO:teuthology.orchestra.run.smithi152.stdout: "ev": 2743, 2024-03-16T12:08:35.629 INFO:teuthology.orchestra.run.smithi152.stdout: "evexg": 0, 2024-03-16T12:08:35.630 INFO:teuthology.orchestra.run.smithi152.stdout: "evexd": 0, 2024-03-16T12:08:35.630 INFO:teuthology.orchestra.run.smithi152.stdout: "segadd": 4, 2024-03-16T12:08:35.630 INFO:teuthology.orchestra.run.smithi152.stdout: "segex": 0, 2024-03-16T12:08:35.630 INFO:teuthology.orchestra.run.smithi152.stdout: "segtrm": 0, 2024-03-16T12:08:35.630 INFO:teuthology.orchestra.run.smithi152.stdout: "seg": 4, 2024-03-16T12:08:35.630 INFO:teuthology.orchestra.run.smithi152.stdout: "segmjr": 2, 2024-03-16T12:08:35.630 INFO:teuthology.orchestra.run.smithi152.stdout: "segexg": 0, 2024-03-16T12:08:35.630 INFO:teuthology.orchestra.run.smithi152.stdout: "segexd": 0, 2024-03-16T12:08:35.630 INFO:teuthology.orchestra.run.smithi152.stdout: "expos": 4194304, 2024-03-16T12:08:35.631 INFO:teuthology.orchestra.run.smithi152.stdout: "wrpos": 8153486, 2024-03-16T12:08:35.631 INFO:teuthology.orchestra.run.smithi152.stdout: "rdpos": 0, 2024-03-16T12:08:35.631 INFO:teuthology.orchestra.run.smithi152.stdout: "jlat": { 2024-03-16T12:08:35.631 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 933, 2024-03-16T12:08:35.631 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 3.819438990, 2024-03-16T12:08:35.631 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.004093718 2024-03-16T12:08:35.631 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-16T12:08:35.631 INFO:teuthology.orchestra.run.smithi152.stdout: "replayed": 0 2024-03-16T12:08:35.632 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-16T12:08:35.632 INFO:teuthology.orchestra.run.smithi152.stdout: "mds_mem": { 2024-03-16T12:08:35.632 INFO:teuthology.orchestra.run.smithi152.stdout: "ino": 180, 2024-03-16T12:08:35.632 INFO:teuthology.orchestra.run.smithi152.stdout: "ino+": 524, 2024-03-16T12:08:35.632 INFO:teuthology.orchestra.run.smithi152.stdout: "ino-": 344, 2024-03-16T12:08:35.632 INFO:teuthology.orchestra.run.smithi152.stdout: "dir": 146, 2024-03-16T12:08:35.632 INFO:teuthology.orchestra.run.smithi152.stdout: "dir+": 247, 2024-03-16T12:08:35.632 INFO:teuthology.orchestra.run.smithi152.stdout: "dir-": 101, 2024-03-16T12:08:35.632 INFO:teuthology.orchestra.run.smithi152.stdout: "dn": 197, 2024-03-16T12:08:35.632 INFO:teuthology.orchestra.run.smithi152.stdout: "dn+": 654, 2024-03-16T12:08:35.633 INFO:teuthology.orchestra.run.smithi152.stdout: "dn-": 457, 2024-03-16T12:08:35.633 INFO:teuthology.orchestra.run.smithi152.stdout: "cap": 14, 2024-03-16T12:08:35.633 INFO:teuthology.orchestra.run.smithi152.stdout: "cap+": 394, 2024-03-16T12:08:35.633 INFO:teuthology.orchestra.run.smithi152.stdout: "cap-": 380, 2024-03-16T12:08:35.633 INFO:teuthology.orchestra.run.smithi152.stdout: "rss": 62124, 2024-03-16T12:08:35.633 INFO:teuthology.orchestra.run.smithi152.stdout: "heap": 223612 2024-03-16T12:08:35.633 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-16T12:08:35.633 INFO:teuthology.orchestra.run.smithi152.stdout: "mds_server": { 2024-03-16T12:08:35.634 INFO:teuthology.orchestra.run.smithi152.stdout: "dispatch_client_request": 47222, 2024-03-16T12:08:35.634 INFO:teuthology.orchestra.run.smithi152.stdout: "dispatch_server_request": 0, 2024-03-16T12:08:35.634 INFO:teuthology.orchestra.run.smithi152.stdout: "handle_client_request": 44446, 2024-03-16T12:08:35.634 INFO:teuthology.orchestra.run.smithi152.stdout: "handle_client_session": 124, 2024-03-16T12:08:35.634 INFO:teuthology.orchestra.run.smithi152.stdout: "handle_peer_request": 930, 2024-03-16T12:08:35.634 INFO:teuthology.orchestra.run.smithi152.stdout: "req_create_latency": { 2024-03-16T12:08:35.634 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 354, 2024-03-16T12:08:35.634 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 7.290064431, 2024-03-16T12:08:35.634 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.020593402 2024-03-16T12:08:35.635 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-16T12:08:35.635 INFO:teuthology.orchestra.run.smithi152.stdout: "req_getattr_latency": { 2024-03-16T12:08:35.635 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 18, 2024-03-16T12:08:35.635 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.022009243, 2024-03-16T12:08:35.635 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.001222735 2024-03-16T12:08:35.635 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-16T12:08:35.635 INFO:teuthology.orchestra.run.smithi152.stdout: "req_getfilelock_latency": { 2024-03-16T12:08:35.635 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-03-16T12:08:35.635 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-03-16T12:08:35.636 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-03-16T12:08:35.636 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-16T12:08:35.636 INFO:teuthology.orchestra.run.smithi152.stdout: "req_link_latency": { 2024-03-16T12:08:35.636 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-03-16T12:08:35.636 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-03-16T12:08:35.636 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-03-16T12:08:35.636 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-16T12:08:35.636 INFO:teuthology.orchestra.run.smithi152.stdout: "req_lookup_latency": { 2024-03-16T12:08:35.636 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 627, 2024-03-16T12:08:35.637 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.971086801, 2024-03-16T12:08:35.637 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.001548782 2024-03-16T12:08:35.637 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-16T12:08:35.637 INFO:teuthology.orchestra.run.smithi152.stdout: "req_lookuphash_latency": { 2024-03-16T12:08:35.637 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-03-16T12:08:35.637 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-03-16T12:08:35.637 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-03-16T12:08:35.637 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-16T12:08:35.637 INFO:teuthology.orchestra.run.smithi152.stdout: "req_lookupino_latency": { 2024-03-16T12:08:35.638 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 1, 2024-03-16T12:08:35.638 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000094145, 2024-03-16T12:08:35.638 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000094145 2024-03-16T12:08:35.638 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-16T12:08:35.638 INFO:teuthology.orchestra.run.smithi152.stdout: "req_lookupname_latency": { 2024-03-16T12:08:35.638 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-03-16T12:08:35.638 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-03-16T12:08:35.638 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-03-16T12:08:35.638 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-16T12:08:35.639 INFO:teuthology.orchestra.run.smithi152.stdout: "req_lookupparent_latency": { 2024-03-16T12:08:35.639 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-03-16T12:08:35.639 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-03-16T12:08:35.639 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-03-16T12:08:35.639 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-16T12:08:35.639 INFO:teuthology.orchestra.run.smithi152.stdout: "req_lookupsnap_latency": { 2024-03-16T12:08:35.639 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-03-16T12:08:35.639 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-03-16T12:08:35.639 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-03-16T12:08:35.640 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-16T12:08:35.640 INFO:teuthology.orchestra.run.smithi152.stdout: "req_lssnap_latency": { 2024-03-16T12:08:35.640 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-03-16T12:08:35.641 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-03-16T12:08:35.641 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-03-16T12:08:35.641 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-16T12:08:35.641 INFO:teuthology.orchestra.run.smithi152.stdout: "req_mkdir_latency": { 2024-03-16T12:08:35.641 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-03-16T12:08:35.641 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-03-16T12:08:35.641 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-03-16T12:08:35.641 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-16T12:08:35.642 INFO:teuthology.orchestra.run.smithi152.stdout: "req_mknod_latency": { 2024-03-16T12:08:35.642 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-03-16T12:08:35.642 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-03-16T12:08:35.642 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-03-16T12:08:35.642 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-16T12:08:35.642 INFO:teuthology.orchestra.run.smithi152.stdout: "req_mksnap_latency": { 2024-03-16T12:08:35.642 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-03-16T12:08:35.642 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-03-16T12:08:35.642 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-03-16T12:08:35.643 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-16T12:08:35.643 INFO:teuthology.orchestra.run.smithi152.stdout: "req_open_latency": { 2024-03-16T12:08:35.643 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-03-16T12:08:35.643 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-03-16T12:08:35.643 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-03-16T12:08:35.643 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-16T12:08:35.643 INFO:teuthology.orchestra.run.smithi152.stdout: "req_readdir_latency": { 2024-03-16T12:08:35.643 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 602, 2024-03-16T12:08:35.643 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.435582760, 2024-03-16T12:08:35.644 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000723559 2024-03-16T12:08:35.644 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-16T12:08:35.644 INFO:teuthology.orchestra.run.smithi152.stdout: "req_rename_latency": { 2024-03-16T12:08:35.644 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 172, 2024-03-16T12:08:35.644 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 3.606783824, 2024-03-16T12:08:35.644 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.020969673 2024-03-16T12:08:35.644 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-16T12:08:35.644 INFO:teuthology.orchestra.run.smithi152.stdout: "req_renamesnap_latency": { 2024-03-16T12:08:35.645 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-03-16T12:08:35.645 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-03-16T12:08:35.645 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-03-16T12:08:35.645 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-16T12:08:35.645 INFO:teuthology.orchestra.run.smithi152.stdout: "req_snapdiff_latency": { 2024-03-16T12:08:35.645 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-03-16T12:08:35.645 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-03-16T12:08:35.645 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-03-16T12:08:35.645 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-16T12:08:35.646 INFO:teuthology.orchestra.run.smithi152.stdout: "req_rmdir_latency": { 2024-03-16T12:08:35.646 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-03-16T12:08:35.646 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-03-16T12:08:35.646 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-03-16T12:08:35.646 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-16T12:08:35.646 INFO:teuthology.orchestra.run.smithi152.stdout: "req_rmsnap_latency": { 2024-03-16T12:08:35.646 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-03-16T12:08:35.646 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-03-16T12:08:35.646 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-03-16T12:08:35.647 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-16T12:08:35.647 INFO:teuthology.orchestra.run.smithi152.stdout: "req_rmxattr_latency": { 2024-03-16T12:08:35.647 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-03-16T12:08:35.647 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-03-16T12:08:35.647 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-03-16T12:08:35.647 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-16T12:08:35.647 INFO:teuthology.orchestra.run.smithi152.stdout: "req_setattr_latency": { 2024-03-16T12:08:35.647 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 4, 2024-03-16T12:08:35.647 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.027867772, 2024-03-16T12:08:35.648 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.006966943 2024-03-16T12:08:35.648 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-16T12:08:35.648 INFO:teuthology.orchestra.run.smithi152.stdout: "req_setdirlayout_latency": { 2024-03-16T12:08:35.648 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-03-16T12:08:35.648 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-03-16T12:08:35.648 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-03-16T12:08:35.648 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-16T12:08:35.648 INFO:teuthology.orchestra.run.smithi152.stdout: "req_setfilelock_latency": { 2024-03-16T12:08:35.648 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-03-16T12:08:35.649 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-03-16T12:08:35.649 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-03-16T12:08:35.649 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-16T12:08:35.649 INFO:teuthology.orchestra.run.smithi152.stdout: "req_setlayout_latency": { 2024-03-16T12:08:35.649 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-03-16T12:08:35.649 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-03-16T12:08:35.649 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-03-16T12:08:35.649 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-16T12:08:35.649 INFO:teuthology.orchestra.run.smithi152.stdout: "req_setxattr_latency": { 2024-03-16T12:08:35.650 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-03-16T12:08:35.650 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-03-16T12:08:35.650 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-03-16T12:08:35.650 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-16T12:08:35.650 INFO:teuthology.orchestra.run.smithi152.stdout: "req_symlink_latency": { 2024-03-16T12:08:35.650 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-03-16T12:08:35.650 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-03-16T12:08:35.650 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-03-16T12:08:35.651 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-16T12:08:35.651 INFO:teuthology.orchestra.run.smithi152.stdout: "req_unlink_latency": { 2024-03-16T12:08:35.651 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 352, 2024-03-16T12:08:35.651 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 3.688554451, 2024-03-16T12:08:35.651 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.010478847 2024-03-16T12:08:35.651 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-16T12:08:35.651 INFO:teuthology.orchestra.run.smithi152.stdout: "cap_revoke_eviction": 0, 2024-03-16T12:08:35.651 INFO:teuthology.orchestra.run.smithi152.stdout: "cap_acquisition_throttle": 0, 2024-03-16T12:08:35.651 INFO:teuthology.orchestra.run.smithi152.stdout: "req_getvxattr_latency": { 2024-03-16T12:08:35.652 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-03-16T12:08:35.652 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-03-16T12:08:35.652 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-03-16T12:08:35.652 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-03-16T12:08:35.652 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-16T12:08:35.652 INFO:teuthology.orchestra.run.smithi152.stdout: "mds_sessions": { 2024-03-16T12:08:35.652 INFO:teuthology.orchestra.run.smithi152.stdout: "session_count": 1, 2024-03-16T12:08:35.652 INFO:teuthology.orchestra.run.smithi152.stdout: "session_add": 5, 2024-03-16T12:08:35.652 INFO:teuthology.orchestra.run.smithi152.stdout: "session_remove": 4, 2024-03-16T12:08:35.652 INFO:teuthology.orchestra.run.smithi152.stdout: "sessions_open": 1, 2024-03-16T12:08:35.653 INFO:teuthology.orchestra.run.smithi152.stdout: "sessions_stale": 0, 2024-03-16T12:08:35.653 INFO:teuthology.orchestra.run.smithi152.stdout: "total_load": 821, 2024-03-16T12:08:35.653 INFO:teuthology.orchestra.run.smithi152.stdout: "average_load": 821, 2024-03-16T12:08:35.653 INFO:teuthology.orchestra.run.smithi152.stdout: "avg_session_uptime": 1054, 2024-03-16T12:08:35.653 INFO:teuthology.orchestra.run.smithi152.stdout: "mdthresh_evicted": 0 2024-03-16T12:08:35.653 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-16T12:08:35.653 INFO:teuthology.orchestra.run.smithi152.stdout: "mempool": { 2024-03-16T12:08:35.653 INFO:teuthology.orchestra.run.smithi152.stdout: "bloom_filter_bytes": 0, 2024-03-16T12:08:35.654 INFO:teuthology.orchestra.run.smithi152.stdout: "bloom_filter_items": 0, 2024-03-16T12:08:35.654 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_alloc_bytes": 0, 2024-03-16T12:08:35.654 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_alloc_items": 0, 2024-03-16T12:08:35.654 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_cache_data_bytes": 0, 2024-03-16T12:08:35.654 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_cache_data_items": 0, 2024-03-16T12:08:35.654 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-16T12:08:35.654 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_cache_onode_items": 0, 2024-03-16T12:08:35.654 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-16T12:08:35.655 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_cache_meta_items": 0, 2024-03-16T12:08:35.655 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_cache_other_bytes": 0, 2024-03-16T12:08:35.655 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_cache_other_items": 0, 2024-03-16T12:08:35.655 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-16T12:08:35.655 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_cache_buffer_items": 0, 2024-03-16T12:08:35.655 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_extent_bytes": 0, 2024-03-16T12:08:35.655 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_extent_items": 0, 2024-03-16T12:08:35.655 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_blob_bytes": 0, 2024-03-16T12:08:35.655 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_blob_items": 0, 2024-03-16T12:08:35.656 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-16T12:08:35.656 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_shared_blob_items": 0, 2024-03-16T12:08:35.656 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-16T12:08:35.656 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_inline_bl_items": 0, 2024-03-16T12:08:35.656 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_fsck_bytes": 0, 2024-03-16T12:08:35.656 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_fsck_items": 0, 2024-03-16T12:08:35.656 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_txc_bytes": 0, 2024-03-16T12:08:35.656 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_txc_items": 0, 2024-03-16T12:08:35.656 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-16T12:08:35.657 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_writing_deferred_items": 0, 2024-03-16T12:08:35.657 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_writing_bytes": 0, 2024-03-16T12:08:35.657 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_writing_items": 0, 2024-03-16T12:08:35.657 INFO:teuthology.orchestra.run.smithi152.stdout: "bluefs_bytes": 0, 2024-03-16T12:08:35.657 INFO:teuthology.orchestra.run.smithi152.stdout: "bluefs_items": 0, 2024-03-16T12:08:35.657 INFO:teuthology.orchestra.run.smithi152.stdout: "bluefs_file_reader_bytes": 0, 2024-03-16T12:08:35.657 INFO:teuthology.orchestra.run.smithi152.stdout: "bluefs_file_reader_items": 0, 2024-03-16T12:08:35.657 INFO:teuthology.orchestra.run.smithi152.stdout: "bluefs_file_writer_bytes": 0, 2024-03-16T12:08:35.657 INFO:teuthology.orchestra.run.smithi152.stdout: "bluefs_file_writer_items": 0, 2024-03-16T12:08:35.658 INFO:teuthology.orchestra.run.smithi152.stdout: "buffer_anon_bytes": 318681, 2024-03-16T12:08:35.658 INFO:teuthology.orchestra.run.smithi152.stdout: "buffer_anon_items": 274, 2024-03-16T12:08:35.658 INFO:teuthology.orchestra.run.smithi152.stdout: "buffer_meta_bytes": 264, 2024-03-16T12:08:35.658 INFO:teuthology.orchestra.run.smithi152.stdout: "buffer_meta_items": 3, 2024-03-16T12:08:35.658 INFO:teuthology.orchestra.run.smithi152.stdout: "osd_bytes": 0, 2024-03-16T12:08:35.658 INFO:teuthology.orchestra.run.smithi152.stdout: "osd_items": 0, 2024-03-16T12:08:35.658 INFO:teuthology.orchestra.run.smithi152.stdout: "osd_mapbl_bytes": 0, 2024-03-16T12:08:35.658 INFO:teuthology.orchestra.run.smithi152.stdout: "osd_mapbl_items": 0, 2024-03-16T12:08:35.658 INFO:teuthology.orchestra.run.smithi152.stdout: "osd_pglog_bytes": 0, 2024-03-16T12:08:35.659 INFO:teuthology.orchestra.run.smithi152.stdout: "osd_pglog_items": 0, 2024-03-16T12:08:35.659 INFO:teuthology.orchestra.run.smithi152.stdout: "osdmap_bytes": 8256, 2024-03-16T12:08:35.659 INFO:teuthology.orchestra.run.smithi152.stdout: "osdmap_items": 188, 2024-03-16T12:08:35.659 INFO:teuthology.orchestra.run.smithi152.stdout: "osdmap_mapping_bytes": 0, 2024-03-16T12:08:35.659 INFO:teuthology.orchestra.run.smithi152.stdout: "osdmap_mapping_items": 0, 2024-03-16T12:08:35.659 INFO:teuthology.orchestra.run.smithi152.stdout: "pgmap_bytes": 0, 2024-03-16T12:08:35.659 INFO:teuthology.orchestra.run.smithi152.stdout: "pgmap_items": 0, 2024-03-16T12:08:35.659 INFO:teuthology.orchestra.run.smithi152.stdout: "mds_co_bytes": 769833, 2024-03-16T12:08:35.659 INFO:teuthology.orchestra.run.smithi152.stdout: "mds_co_items": 6316, 2024-03-16T12:08:35.660 INFO:teuthology.orchestra.run.smithi152.stdout: "unittest_1_bytes": 0, 2024-03-16T12:08:35.660 INFO:teuthology.orchestra.run.smithi152.stdout: "unittest_1_items": 0, 2024-03-16T12:08:35.660 INFO:teuthology.orchestra.run.smithi152.stdout: "unittest_2_bytes": 0, 2024-03-16T12:08:35.660 INFO:teuthology.orchestra.run.smithi152.stdout: "unittest_2_items": 0 2024-03-16T12:08:35.660 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-16T12:08:35.660 INFO:teuthology.orchestra.run.smithi152.stdout: "objecter": { 2024-03-16T12:08:35.660 INFO:teuthology.orchestra.run.smithi152.stdout: "op_active": 0, 2024-03-16T12:08:35.660 INFO:teuthology.orchestra.run.smithi152.stdout: "op_laggy": 0, 2024-03-16T12:08:35.660 INFO:teuthology.orchestra.run.smithi152.stdout: "op_send": 1665, 2024-03-16T12:08:35.661 INFO:teuthology.orchestra.run.smithi152.stdout: "op_send_bytes": 4021403, 2024-03-16T12:08:35.661 INFO:teuthology.orchestra.run.smithi152.stdout: "op_resend": 33, 2024-03-16T12:08:35.661 INFO:teuthology.orchestra.run.smithi152.stdout: "op_reply": 1665, 2024-03-16T12:08:35.661 INFO:teuthology.orchestra.run.smithi152.stdout: "op_latency": { 2024-03-16T12:08:35.661 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 1665, 2024-03-16T12:08:35.661 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 7.508907920, 2024-03-16T12:08:35.661 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.004509854 2024-03-16T12:08:35.661 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-16T12:08:35.661 INFO:teuthology.orchestra.run.smithi152.stdout: "op_inflight": 0, 2024-03-16T12:08:35.662 INFO:teuthology.orchestra.run.smithi152.stdout: "oplen_avg": { 2024-03-16T12:08:35.662 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 1665, 2024-03-16T12:08:35.662 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 1829 2024-03-16T12:08:35.662 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-16T12:08:35.662 INFO:teuthology.orchestra.run.smithi152.stdout: "op": 1665, 2024-03-16T12:08:35.662 INFO:teuthology.orchestra.run.smithi152.stdout: "op_r": 86, 2024-03-16T12:08:35.662 INFO:teuthology.orchestra.run.smithi152.stdout: "op_w": 1579, 2024-03-16T12:08:35.662 INFO:teuthology.orchestra.run.smithi152.stdout: "op_rmw": 0, 2024-03-16T12:08:35.663 INFO:teuthology.orchestra.run.smithi152.stdout: "op_pg": 0, 2024-03-16T12:08:35.663 INFO:teuthology.orchestra.run.smithi152.stdout: "osdop_stat": 0, 2024-03-16T12:08:35.663 INFO:teuthology.orchestra.run.smithi152.stdout: "osdop_create": 10, 2024-03-16T12:08:35.663 INFO:teuthology.orchestra.run.smithi152.stdout: "osdop_read": 86, 2024-03-16T12:08:35.663 INFO:teuthology.orchestra.run.smithi152.stdout: "osdop_write": 1019, 2024-03-16T12:08:35.663 INFO:teuthology.orchestra.run.smithi152.stdout: "osdop_writefull": 71, 2024-03-16T12:08:35.663 INFO:teuthology.orchestra.run.smithi152.stdout: "osdop_writesame": 0, 2024-03-16T12:08:35.663 INFO:teuthology.orchestra.run.smithi152.stdout: "osdop_append": 0, 2024-03-16T12:08:35.663 INFO:teuthology.orchestra.run.smithi152.stdout: "osdop_zero": 0, 2024-03-16T12:08:35.664 INFO:teuthology.orchestra.run.smithi152.stdout: "osdop_truncate": 0, 2024-03-16T12:08:35.664 INFO:teuthology.orchestra.run.smithi152.stdout: "osdop_delete": 356, 2024-03-16T12:08:35.664 INFO:teuthology.orchestra.run.smithi152.stdout: "osdop_mapext": 0, 2024-03-16T12:08:35.664 INFO:teuthology.orchestra.run.smithi152.stdout: "osdop_sparse_read": 0, 2024-03-16T12:08:35.664 INFO:teuthology.orchestra.run.smithi152.stdout: "osdop_clonerange": 0, 2024-03-16T12:08:35.664 INFO:teuthology.orchestra.run.smithi152.stdout: "osdop_getxattr": 0, 2024-03-16T12:08:35.664 INFO:teuthology.orchestra.run.smithi152.stdout: "osdop_setxattr": 20, 2024-03-16T12:08:35.664 INFO:teuthology.orchestra.run.smithi152.stdout: "osdop_cmpxattr": 0, 2024-03-16T12:08:35.664 INFO:teuthology.orchestra.run.smithi152.stdout: "osdop_rmxattr": 0, 2024-03-16T12:08:35.664 INFO:teuthology.orchestra.run.smithi152.stdout: "osdop_resetxattrs": 0, 2024-03-16T12:08:35.665 INFO:teuthology.orchestra.run.smithi152.stdout: "osdop_call": 0, 2024-03-16T12:08:35.665 INFO:teuthology.orchestra.run.smithi152.stdout: "osdop_watch": 0, 2024-03-16T12:08:35.665 INFO:teuthology.orchestra.run.smithi152.stdout: "osdop_notify": 0, 2024-03-16T12:08:35.665 INFO:teuthology.orchestra.run.smithi152.stdout: "osdop_src_cmpxattr": 0, 2024-03-16T12:08:35.665 INFO:teuthology.orchestra.run.smithi152.stdout: "osdop_pgls": 0, 2024-03-16T12:08:35.665 INFO:teuthology.orchestra.run.smithi152.stdout: "osdop_pgls_filter": 0, 2024-03-16T12:08:35.665 INFO:teuthology.orchestra.run.smithi152.stdout: "osdop_other": 0, 2024-03-16T12:08:35.665 INFO:teuthology.orchestra.run.smithi152.stdout: "linger_active": 0, 2024-03-16T12:08:35.666 INFO:teuthology.orchestra.run.smithi152.stdout: "linger_send": 0, 2024-03-16T12:08:35.666 INFO:teuthology.orchestra.run.smithi152.stdout: "linger_resend": 0, 2024-03-16T12:08:35.666 INFO:teuthology.orchestra.run.smithi152.stdout: "linger_ping": 0, 2024-03-16T12:08:35.666 INFO:teuthology.orchestra.run.smithi152.stdout: "poolop_active": 0, 2024-03-16T12:08:35.666 INFO:teuthology.orchestra.run.smithi152.stdout: "poolop_send": 0, 2024-03-16T12:08:35.666 INFO:teuthology.orchestra.run.smithi152.stdout: "poolop_resend": 0, 2024-03-16T12:08:35.666 INFO:teuthology.orchestra.run.smithi152.stdout: "poolstat_active": 0, 2024-03-16T12:08:35.666 INFO:teuthology.orchestra.run.smithi152.stdout: "poolstat_send": 0, 2024-03-16T12:08:35.666 INFO:teuthology.orchestra.run.smithi152.stdout: "poolstat_resend": 0, 2024-03-16T12:08:35.667 INFO:teuthology.orchestra.run.smithi152.stdout: "statfs_active": 0, 2024-03-16T12:08:35.667 INFO:teuthology.orchestra.run.smithi152.stdout: "statfs_send": 0, 2024-03-16T12:08:35.667 INFO:teuthology.orchestra.run.smithi152.stdout: "statfs_resend": 0, 2024-03-16T12:08:35.667 INFO:teuthology.orchestra.run.smithi152.stdout: "command_active": 0, 2024-03-16T12:08:35.667 INFO:teuthology.orchestra.run.smithi152.stdout: "command_send": 0, 2024-03-16T12:08:35.667 INFO:teuthology.orchestra.run.smithi152.stdout: "command_resend": 0, 2024-03-16T12:08:35.667 INFO:teuthology.orchestra.run.smithi152.stdout: "map_epoch": 116, 2024-03-16T12:08:35.667 INFO:teuthology.orchestra.run.smithi152.stdout: "map_full": 0, 2024-03-16T12:08:35.667 INFO:teuthology.orchestra.run.smithi152.stdout: "map_inc": 31, 2024-03-16T12:08:35.668 INFO:teuthology.orchestra.run.smithi152.stdout: "osd_sessions": 12, 2024-03-16T12:08:35.668 INFO:teuthology.orchestra.run.smithi152.stdout: "osd_session_open": 16, 2024-03-16T12:08:35.668 INFO:teuthology.orchestra.run.smithi152.stdout: "osd_session_close": 4, 2024-03-16T12:08:35.668 INFO:teuthology.orchestra.run.smithi152.stdout: "osd_laggy": 0, 2024-03-16T12:08:35.668 INFO:teuthology.orchestra.run.smithi152.stdout: "omap_wr": 196, 2024-03-16T12:08:35.668 INFO:teuthology.orchestra.run.smithi152.stdout: "omap_rd": 0, 2024-03-16T12:08:35.668 INFO:teuthology.orchestra.run.smithi152.stdout: "omap_del": 71 2024-03-16T12:08:35.668 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-16T12:08:35.668 INFO:teuthology.orchestra.run.smithi152.stdout: "oft": { 2024-03-16T12:08:35.669 INFO:teuthology.orchestra.run.smithi152.stdout: "omap_total_objs": 1, 2024-03-16T12:08:35.669 INFO:teuthology.orchestra.run.smithi152.stdout: "omap_total_kv_pairs": 0, 2024-03-16T12:08:35.669 INFO:teuthology.orchestra.run.smithi152.stdout: "omap_total_updates": 0, 2024-03-16T12:08:35.669 INFO:teuthology.orchestra.run.smithi152.stdout: "omap_total_removes": 0 2024-03-16T12:08:35.669 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-16T12:08:35.669 INFO:teuthology.orchestra.run.smithi152.stdout: "osd-slow-ops": { 2024-03-16T12:08:35.669 INFO:teuthology.orchestra.run.smithi152.stdout: "slow_ops_count": 0 2024-03-16T12:08:35.669 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-16T12:08:35.669 INFO:teuthology.orchestra.run.smithi152.stdout: "purge_queue": { 2024-03-16T12:08:35.670 INFO:teuthology.orchestra.run.smithi152.stdout: "pq_executing_ops": 0, 2024-03-16T12:08:35.670 INFO:teuthology.orchestra.run.smithi152.stdout: "pq_executing_ops_high_water": 10, 2024-03-16T12:08:35.670 INFO:teuthology.orchestra.run.smithi152.stdout: "pq_executing": 0, 2024-03-16T12:08:35.670 INFO:teuthology.orchestra.run.smithi152.stdout: "pq_executing_high_water": 10, 2024-03-16T12:08:35.670 INFO:teuthology.orchestra.run.smithi152.stdout: "pq_executed_ops": 344, 2024-03-16T12:08:35.670 INFO:teuthology.orchestra.run.smithi152.stdout: "pq_executed": 344, 2024-03-16T12:08:35.670 INFO:teuthology.orchestra.run.smithi152.stdout: "pq_item_in_journal": 0 2024-03-16T12:08:35.670 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-16T12:08:35.671 INFO:teuthology.orchestra.run.smithi152.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-16T12:08:35.671 INFO:teuthology.orchestra.run.smithi152.stdout: "val": 0, 2024-03-16T12:08:35.671 INFO:teuthology.orchestra.run.smithi152.stdout: "max": 104857600, 2024-03-16T12:08:35.671 INFO:teuthology.orchestra.run.smithi152.stdout: "get_started": 0, 2024-03-16T12:08:35.671 INFO:teuthology.orchestra.run.smithi152.stdout: "get": 427099, 2024-03-16T12:08:35.671 INFO:teuthology.orchestra.run.smithi152.stdout: "get_sum": 140908998, 2024-03-16T12:08:35.671 INFO:teuthology.orchestra.run.smithi152.stdout: "get_or_fail_fail": 0, 2024-03-16T12:08:35.671 INFO:teuthology.orchestra.run.smithi152.stdout: "get_or_fail_success": 427099, 2024-03-16T12:08:35.671 INFO:teuthology.orchestra.run.smithi152.stdout: "take": 0, 2024-03-16T12:08:35.672 INFO:teuthology.orchestra.run.smithi152.stdout: "take_sum": 0, 2024-03-16T12:08:35.672 INFO:teuthology.orchestra.run.smithi152.stdout: "put": 427099, 2024-03-16T12:08:35.672 INFO:teuthology.orchestra.run.smithi152.stdout: "put_sum": 140908998, 2024-03-16T12:08:35.672 INFO:teuthology.orchestra.run.smithi152.stdout: "wait": { 2024-03-16T12:08:35.672 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-03-16T12:08:35.672 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-03-16T12:08:35.672 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-03-16T12:08:35.672 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-03-16T12:08:35.672 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-16T12:08:35.673 INFO:teuthology.orchestra.run.smithi152.stdout: "throttle-objecter_bytes": { 2024-03-16T12:08:35.673 INFO:teuthology.orchestra.run.smithi152.stdout: "val": 0, 2024-03-16T12:08:35.673 INFO:teuthology.orchestra.run.smithi152.stdout: "max": 104857600, 2024-03-16T12:08:35.673 INFO:teuthology.orchestra.run.smithi152.stdout: "get_started": 0, 2024-03-16T12:08:35.673 INFO:teuthology.orchestra.run.smithi152.stdout: "get": 0, 2024-03-16T12:08:35.673 INFO:teuthology.orchestra.run.smithi152.stdout: "get_sum": 0, 2024-03-16T12:08:35.673 INFO:teuthology.orchestra.run.smithi152.stdout: "get_or_fail_fail": 0, 2024-03-16T12:08:35.673 INFO:teuthology.orchestra.run.smithi152.stdout: "get_or_fail_success": 0, 2024-03-16T12:08:35.673 INFO:teuthology.orchestra.run.smithi152.stdout: "take": 1665, 2024-03-16T12:08:35.674 INFO:teuthology.orchestra.run.smithi152.stdout: "take_sum": 4056227, 2024-03-16T12:08:35.674 INFO:teuthology.orchestra.run.smithi152.stdout: "put": 1665, 2024-03-16T12:08:35.674 INFO:teuthology.orchestra.run.smithi152.stdout: "put_sum": 4056227, 2024-03-16T12:08:35.674 INFO:teuthology.orchestra.run.smithi152.stdout: "wait": { 2024-03-16T12:08:35.674 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-03-16T12:08:35.674 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-03-16T12:08:35.674 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-03-16T12:08:35.674 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-03-16T12:08:35.675 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-16T12:08:35.675 INFO:teuthology.orchestra.run.smithi152.stdout: "throttle-objecter_ops": { 2024-03-16T12:08:35.675 INFO:teuthology.orchestra.run.smithi152.stdout: "val": 0, 2024-03-16T12:08:35.675 INFO:teuthology.orchestra.run.smithi152.stdout: "max": 1024, 2024-03-16T12:08:35.675 INFO:teuthology.orchestra.run.smithi152.stdout: "get_started": 0, 2024-03-16T12:08:35.675 INFO:teuthology.orchestra.run.smithi152.stdout: "get": 0, 2024-03-16T12:08:35.675 INFO:teuthology.orchestra.run.smithi152.stdout: "get_sum": 0, 2024-03-16T12:08:35.675 INFO:teuthology.orchestra.run.smithi152.stdout: "get_or_fail_fail": 0, 2024-03-16T12:08:35.675 INFO:teuthology.orchestra.run.smithi152.stdout: "get_or_fail_success": 0, 2024-03-16T12:08:35.676 INFO:teuthology.orchestra.run.smithi152.stdout: "take": 1665, 2024-03-16T12:08:35.676 INFO:teuthology.orchestra.run.smithi152.stdout: "take_sum": 1665, 2024-03-16T12:08:35.676 INFO:teuthology.orchestra.run.smithi152.stdout: "put": 1665, 2024-03-16T12:08:35.676 INFO:teuthology.orchestra.run.smithi152.stdout: "put_sum": 1665, 2024-03-16T12:08:35.676 INFO:teuthology.orchestra.run.smithi152.stdout: "wait": { 2024-03-16T12:08:35.676 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-03-16T12:08:35.676 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-03-16T12:08:35.676 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-03-16T12:08:35.676 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-03-16T12:08:35.677 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-16T12:08:35.677 INFO:teuthology.orchestra.run.smithi152.stdout: "throttle-write_buf_throttle": { 2024-03-16T12:08:35.677 INFO:teuthology.orchestra.run.smithi152.stdout: "val": 0, 2024-03-16T12:08:35.677 INFO:teuthology.orchestra.run.smithi152.stdout: "max": 3758096384, 2024-03-16T12:08:35.677 INFO:teuthology.orchestra.run.smithi152.stdout: "get_started": 0, 2024-03-16T12:08:35.677 INFO:teuthology.orchestra.run.smithi152.stdout: "get": 344, 2024-03-16T12:08:35.677 INFO:teuthology.orchestra.run.smithi152.stdout: "get_sum": 34824, 2024-03-16T12:08:35.677 INFO:teuthology.orchestra.run.smithi152.stdout: "get_or_fail_fail": 0, 2024-03-16T12:08:35.677 INFO:teuthology.orchestra.run.smithi152.stdout: "get_or_fail_success": 344, 2024-03-16T12:08:35.678 INFO:teuthology.orchestra.run.smithi152.stdout: "take": 0, 2024-03-16T12:08:35.678 INFO:teuthology.orchestra.run.smithi152.stdout: "take_sum": 0, 2024-03-16T12:08:35.678 INFO:teuthology.orchestra.run.smithi152.stdout: "put": 86, 2024-03-16T12:08:35.678 INFO:teuthology.orchestra.run.smithi152.stdout: "put_sum": 34824, 2024-03-16T12:08:35.678 INFO:teuthology.orchestra.run.smithi152.stdout: "wait": { 2024-03-16T12:08:35.678 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-03-16T12:08:35.678 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-03-16T12:08:35.678 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-03-16T12:08:35.678 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-03-16T12:08:35.679 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-16T12:08:35.679 INFO:teuthology.orchestra.run.smithi152.stdout: "throttle-write_buf_throttle-0x56244d31a500": { 2024-03-16T12:08:35.679 INFO:teuthology.orchestra.run.smithi152.stdout: "val": 33499, 2024-03-16T12:08:35.679 INFO:teuthology.orchestra.run.smithi152.stdout: "max": 3758096384, 2024-03-16T12:08:35.679 INFO:teuthology.orchestra.run.smithi152.stdout: "get_started": 0, 2024-03-16T12:08:35.679 INFO:teuthology.orchestra.run.smithi152.stdout: "get": 2743, 2024-03-16T12:08:35.679 INFO:teuthology.orchestra.run.smithi152.stdout: "get_sum": 3959182, 2024-03-16T12:08:35.679 INFO:teuthology.orchestra.run.smithi152.stdout: "get_or_fail_fail": 0, 2024-03-16T12:08:35.679 INFO:teuthology.orchestra.run.smithi152.stdout: "get_or_fail_success": 2743, 2024-03-16T12:08:35.680 INFO:teuthology.orchestra.run.smithi152.stdout: "take": 0, 2024-03-16T12:08:35.680 INFO:teuthology.orchestra.run.smithi152.stdout: "take_sum": 0, 2024-03-16T12:08:35.680 INFO:teuthology.orchestra.run.smithi152.stdout: "put": 933, 2024-03-16T12:08:35.680 INFO:teuthology.orchestra.run.smithi152.stdout: "put_sum": 3925683, 2024-03-16T12:08:35.680 INFO:teuthology.orchestra.run.smithi152.stdout: "wait": { 2024-03-16T12:08:35.680 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-03-16T12:08:35.680 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-03-16T12:08:35.680 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-03-16T12:08:35.680 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-03-16T12:08:35.681 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-03-16T12:08:35.681 INFO:teuthology.orchestra.run.smithi152.stdout:} 2024-03-16T12:08:36.785 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:08:36 smithi154 ceph-mon[36320]: pgmap v561: 97 pgs: 97 active+clean; 356 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 161 KiB/s rd, 2.1 MiB/s wr, 74 op/s 2024-03-16T12:08:36.816 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:08:36 smithi152 ceph-mon[36291]: pgmap v561: 97 pgs: 97 active+clean; 356 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 161 KiB/s rd, 2.1 MiB/s wr, 74 op/s 2024-03-16T12:08:36.828 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:08:36 smithi029 ceph-mon[31975]: pgmap v561: 97 pgs: 97 active+clean; 356 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 161 KiB/s rd, 2.1 MiB/s wr, 74 op/s 2024-03-16T12:08:36.849 INFO:tasks.check_counter:Daemon mds.h mds.exported=0 2024-03-16T12:08:36.849 INFO:tasks.check_counter:Daemon mds.h mds.imported=2 2024-03-16T12:08:36.849 INFO:tasks.check_counter:Daemon mds.h mds_cache.dir_update=51 2024-03-16T12:08:36.849 INFO:tasks.check_counter:Daemon mds.h mds_cache.dir_update_receipt=481 2024-03-16T12:08:36.850 INFO:tasks.check_counter:Daemon mds.h mds.root_rsnaps=7 2024-03-16T12:08:36.850 INFO:tasks.check_counter:Daemon mds.h mds_server.req_mksnap_latency.avgcount=0 2024-03-16T12:08:36.850 INFO:tasks.check_counter:Daemon mds.h mds_server.req_rmsnap_latency.avgcount=0 2024-03-16T12:08:36.850 DEBUG:tasks.check_counter:Getting stats from k 2024-03-16T12:08:36.850 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid d1887f28-e389-11ee-95c9-87774f69a715 -- ceph daemon mds.k perf dump 2024-03-16T12:08:37.131 INFO:teuthology.orchestra.run.smithi152.stderr:Inferring config /var/lib/ceph/d1887f28-e389-11ee-95c9-87774f69a715/mon.b/config 2024-03-16T12:08:38.596 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:08:38 smithi152 ceph-mon[36291]: pgmap v562: 97 pgs: 97 active+clean; 359 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 235 KiB/s rd, 2.6 MiB/s wr, 94 op/s 2024-03-16T12:08:38.785 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:08:38 smithi154 ceph-mon[36320]: pgmap v562: 97 pgs: 97 active+clean; 359 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 235 KiB/s rd, 2.6 MiB/s wr, 94 op/s 2024-03-16T12:08:38.828 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:08:38 smithi029 ceph-mon[31975]: pgmap v562: 97 pgs: 97 active+clean; 359 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 235 KiB/s rd, 2.6 MiB/s wr, 94 op/s 2024-03-16T12:08:39.061 INFO:teuthology.orchestra.run.smithi152.stdout:{ 2024-03-16T12:08:39.062 INFO:teuthology.orchestra.run.smithi152.stdout: "AsyncMessenger::Worker-0": { 2024-03-16T12:08:39.062 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_recv_messages": 4, 2024-03-16T12:08:39.062 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_send_messages": 4, 2024-03-16T12:08:39.062 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_recv_bytes": 3072, 2024-03-16T12:08:39.062 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_send_bytes": 448, 2024-03-16T12:08:39.062 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_created_connections": 4, 2024-03-16T12:08:39.062 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_active_connections": 0, 2024-03-16T12:08:39.062 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_running_total_time": 0.001374561, 2024-03-16T12:08:39.063 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_running_send_time": 0.000030180, 2024-03-16T12:08:39.063 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_running_recv_time": 0.000590669, 2024-03-16T12:08:39.063 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-16T12:08:39.063 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_send_messages_queue_lat": { 2024-03-16T12:08:39.063 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 3, 2024-03-16T12:08:39.063 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000720608, 2024-03-16T12:08:39.063 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000240202 2024-03-16T12:08:39.063 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-16T12:08:39.063 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_handle_ack_lat": { 2024-03-16T12:08:39.064 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-03-16T12:08:39.064 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-03-16T12:08:39.064 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-03-16T12:08:39.064 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-16T12:08:39.064 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_recv_encrypted_bytes": 3072, 2024-03-16T12:08:39.064 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_send_encrypted_bytes": 448 2024-03-16T12:08:39.064 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-16T12:08:39.064 INFO:teuthology.orchestra.run.smithi152.stdout: "AsyncMessenger::Worker-1": { 2024-03-16T12:08:39.064 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_recv_messages": 3, 2024-03-16T12:08:39.065 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_send_messages": 252, 2024-03-16T12:08:39.065 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_recv_bytes": 480, 2024-03-16T12:08:39.065 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_send_bytes": 278160, 2024-03-16T12:08:39.065 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_created_connections": 6, 2024-03-16T12:08:39.065 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_active_connections": 3, 2024-03-16T12:08:39.065 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_running_total_time": 0.011875286, 2024-03-16T12:08:39.065 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_running_send_time": 0.007922336, 2024-03-16T12:08:39.065 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_running_recv_time": 0.001913932, 2024-03-16T12:08:39.065 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-16T12:08:39.066 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_send_messages_queue_lat": { 2024-03-16T12:08:39.066 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 249, 2024-03-16T12:08:39.066 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.018736062, 2024-03-16T12:08:39.066 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000075245 2024-03-16T12:08:39.066 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-16T12:08:39.066 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_handle_ack_lat": { 2024-03-16T12:08:39.066 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-03-16T12:08:39.066 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-03-16T12:08:39.067 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-03-16T12:08:39.067 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-16T12:08:39.067 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_recv_encrypted_bytes": 480, 2024-03-16T12:08:39.067 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_send_encrypted_bytes": 278160 2024-03-16T12:08:39.067 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-16T12:08:39.067 INFO:teuthology.orchestra.run.smithi152.stdout: "AsyncMessenger::Worker-2": { 2024-03-16T12:08:39.067 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_recv_messages": 338, 2024-03-16T12:08:39.067 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_send_messages": 328, 2024-03-16T12:08:39.067 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_recv_bytes": 1470704, 2024-03-16T12:08:39.068 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_send_bytes": 202800, 2024-03-16T12:08:39.068 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_created_connections": 2, 2024-03-16T12:08:39.068 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_active_connections": 1, 2024-03-16T12:08:39.068 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_running_total_time": 0.048026633, 2024-03-16T12:08:39.068 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_running_send_time": 0.013657148, 2024-03-16T12:08:39.068 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_running_recv_time": 0.036987245, 2024-03-16T12:08:39.068 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_running_fast_dispatch_time": 0.006512646, 2024-03-16T12:08:39.068 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_send_messages_queue_lat": { 2024-03-16T12:08:39.068 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 327, 2024-03-16T12:08:39.069 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.010515427, 2024-03-16T12:08:39.069 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000032157 2024-03-16T12:08:39.069 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-16T12:08:39.069 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_handle_ack_lat": { 2024-03-16T12:08:39.069 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-03-16T12:08:39.069 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-03-16T12:08:39.069 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-03-16T12:08:39.069 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-16T12:08:39.069 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_recv_encrypted_bytes": 1470704, 2024-03-16T12:08:39.070 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_send_encrypted_bytes": 202800 2024-03-16T12:08:39.070 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-16T12:08:39.070 INFO:teuthology.orchestra.run.smithi152.stdout: "cct": { 2024-03-16T12:08:39.070 INFO:teuthology.orchestra.run.smithi152.stdout: "total_workers": 0, 2024-03-16T12:08:39.070 INFO:teuthology.orchestra.run.smithi152.stdout: "unhealthy_workers": 0 2024-03-16T12:08:39.070 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-16T12:08:39.070 INFO:teuthology.orchestra.run.smithi152.stdout: "mempool": { 2024-03-16T12:08:39.070 INFO:teuthology.orchestra.run.smithi152.stdout: "bloom_filter_bytes": 0, 2024-03-16T12:08:39.070 INFO:teuthology.orchestra.run.smithi152.stdout: "bloom_filter_items": 0, 2024-03-16T12:08:39.071 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_alloc_bytes": 0, 2024-03-16T12:08:39.071 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_alloc_items": 0, 2024-03-16T12:08:39.071 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_cache_data_bytes": 0, 2024-03-16T12:08:39.071 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_cache_data_items": 0, 2024-03-16T12:08:39.071 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-16T12:08:39.071 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_cache_onode_items": 0, 2024-03-16T12:08:39.071 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-16T12:08:39.071 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_cache_meta_items": 0, 2024-03-16T12:08:39.071 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_cache_other_bytes": 0, 2024-03-16T12:08:39.072 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_cache_other_items": 0, 2024-03-16T12:08:39.072 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-16T12:08:39.072 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_cache_buffer_items": 0, 2024-03-16T12:08:39.072 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_extent_bytes": 0, 2024-03-16T12:08:39.072 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_extent_items": 0, 2024-03-16T12:08:39.072 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_blob_bytes": 0, 2024-03-16T12:08:39.072 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_blob_items": 0, 2024-03-16T12:08:39.072 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-16T12:08:39.072 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_shared_blob_items": 0, 2024-03-16T12:08:39.073 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-16T12:08:39.073 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_inline_bl_items": 0, 2024-03-16T12:08:39.073 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_fsck_bytes": 0, 2024-03-16T12:08:39.073 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_fsck_items": 0, 2024-03-16T12:08:39.073 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_txc_bytes": 0, 2024-03-16T12:08:39.073 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_txc_items": 0, 2024-03-16T12:08:39.073 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-16T12:08:39.073 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_writing_deferred_items": 0, 2024-03-16T12:08:39.073 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_writing_bytes": 0, 2024-03-16T12:08:39.074 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_writing_items": 0, 2024-03-16T12:08:39.074 INFO:teuthology.orchestra.run.smithi152.stdout: "bluefs_bytes": 0, 2024-03-16T12:08:39.074 INFO:teuthology.orchestra.run.smithi152.stdout: "bluefs_items": 0, 2024-03-16T12:08:39.074 INFO:teuthology.orchestra.run.smithi152.stdout: "bluefs_file_reader_bytes": 0, 2024-03-16T12:08:39.074 INFO:teuthology.orchestra.run.smithi152.stdout: "bluefs_file_reader_items": 0, 2024-03-16T12:08:39.074 INFO:teuthology.orchestra.run.smithi152.stdout: "bluefs_file_writer_bytes": 0, 2024-03-16T12:08:39.074 INFO:teuthology.orchestra.run.smithi152.stdout: "bluefs_file_writer_items": 0, 2024-03-16T12:08:39.074 INFO:teuthology.orchestra.run.smithi152.stdout: "buffer_anon_bytes": 148794, 2024-03-16T12:08:39.074 INFO:teuthology.orchestra.run.smithi152.stdout: "buffer_anon_items": 32, 2024-03-16T12:08:39.075 INFO:teuthology.orchestra.run.smithi152.stdout: "buffer_meta_bytes": 0, 2024-03-16T12:08:39.075 INFO:teuthology.orchestra.run.smithi152.stdout: "buffer_meta_items": 0, 2024-03-16T12:08:39.075 INFO:teuthology.orchestra.run.smithi152.stdout: "osd_bytes": 0, 2024-03-16T12:08:39.075 INFO:teuthology.orchestra.run.smithi152.stdout: "osd_items": 0, 2024-03-16T12:08:39.075 INFO:teuthology.orchestra.run.smithi152.stdout: "osd_mapbl_bytes": 0, 2024-03-16T12:08:39.075 INFO:teuthology.orchestra.run.smithi152.stdout: "osd_mapbl_items": 0, 2024-03-16T12:08:39.075 INFO:teuthology.orchestra.run.smithi152.stdout: "osd_pglog_bytes": 0, 2024-03-16T12:08:39.075 INFO:teuthology.orchestra.run.smithi152.stdout: "osd_pglog_items": 0, 2024-03-16T12:08:39.075 INFO:teuthology.orchestra.run.smithi152.stdout: "osdmap_bytes": 0, 2024-03-16T12:08:39.075 INFO:teuthology.orchestra.run.smithi152.stdout: "osdmap_items": 0, 2024-03-16T12:08:39.076 INFO:teuthology.orchestra.run.smithi152.stdout: "osdmap_mapping_bytes": 0, 2024-03-16T12:08:39.076 INFO:teuthology.orchestra.run.smithi152.stdout: "osdmap_mapping_items": 0, 2024-03-16T12:08:39.076 INFO:teuthology.orchestra.run.smithi152.stdout: "pgmap_bytes": 0, 2024-03-16T12:08:39.076 INFO:teuthology.orchestra.run.smithi152.stdout: "pgmap_items": 0, 2024-03-16T12:08:39.076 INFO:teuthology.orchestra.run.smithi152.stdout: "mds_co_bytes": 928, 2024-03-16T12:08:39.076 INFO:teuthology.orchestra.run.smithi152.stdout: "mds_co_items": 2, 2024-03-16T12:08:39.076 INFO:teuthology.orchestra.run.smithi152.stdout: "unittest_1_bytes": 0, 2024-03-16T12:08:39.076 INFO:teuthology.orchestra.run.smithi152.stdout: "unittest_1_items": 0, 2024-03-16T12:08:39.076 INFO:teuthology.orchestra.run.smithi152.stdout: "unittest_2_bytes": 0, 2024-03-16T12:08:39.077 INFO:teuthology.orchestra.run.smithi152.stdout: "unittest_2_items": 0 2024-03-16T12:08:39.077 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-16T12:08:39.077 INFO:teuthology.orchestra.run.smithi152.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-16T12:08:39.077 INFO:teuthology.orchestra.run.smithi152.stdout: "val": 0, 2024-03-16T12:08:39.077 INFO:teuthology.orchestra.run.smithi152.stdout: "max": 104857600, 2024-03-16T12:08:39.077 INFO:teuthology.orchestra.run.smithi152.stdout: "get_started": 0, 2024-03-16T12:08:39.077 INFO:teuthology.orchestra.run.smithi152.stdout: "get": 341, 2024-03-16T12:08:39.077 INFO:teuthology.orchestra.run.smithi152.stdout: "get_sum": 1423008, 2024-03-16T12:08:39.077 INFO:teuthology.orchestra.run.smithi152.stdout: "get_or_fail_fail": 0, 2024-03-16T12:08:39.078 INFO:teuthology.orchestra.run.smithi152.stdout: "get_or_fail_success": 341, 2024-03-16T12:08:39.078 INFO:teuthology.orchestra.run.smithi152.stdout: "take": 0, 2024-03-16T12:08:39.078 INFO:teuthology.orchestra.run.smithi152.stdout: "take_sum": 0, 2024-03-16T12:08:39.078 INFO:teuthology.orchestra.run.smithi152.stdout: "put": 341, 2024-03-16T12:08:39.078 INFO:teuthology.orchestra.run.smithi152.stdout: "put_sum": 1423008, 2024-03-16T12:08:39.078 INFO:teuthology.orchestra.run.smithi152.stdout: "wait": { 2024-03-16T12:08:39.078 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-03-16T12:08:39.078 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-03-16T12:08:39.078 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-03-16T12:08:39.079 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-03-16T12:08:39.079 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-03-16T12:08:39.079 INFO:teuthology.orchestra.run.smithi152.stdout:} 2024-03-16T12:08:39.762 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.k 2024-03-16T12:08:39.762 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.k 2024-03-16T12:08:39.762 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.k 2024-03-16T12:08:39.762 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.k 2024-03-16T12:08:39.762 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.k 2024-03-16T12:08:39.763 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.k 2024-03-16T12:08:39.763 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.k 2024-03-16T12:08:39.763 DEBUG:tasks.check_counter:Getting stats from c 2024-03-16T12:08:39.763 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid d1887f28-e389-11ee-95c9-87774f69a715 -- ceph daemon mds.c perf dump 2024-03-16T12:08:40.009 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/d1887f28-e389-11ee-95c9-87774f69a715/mon.c/config 2024-03-16T12:08:40.785 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:08:40 smithi154 ceph-mon[36320]: pgmap v563: 97 pgs: 97 active+clean; 359 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 159 KiB/s rd, 878 KiB/s wr, 46 op/s 2024-03-16T12:08:40.804 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:08:40 smithi152 ceph-mon[36291]: pgmap v563: 97 pgs: 97 active+clean; 359 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 159 KiB/s rd, 878 KiB/s wr, 46 op/s 2024-03-16T12:08:40.828 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:08:40 smithi029 ceph-mon[31975]: pgmap v563: 97 pgs: 97 active+clean; 359 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 159 KiB/s rd, 878 KiB/s wr, 46 op/s 2024-03-16T12:08:42.042 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-03-16T12:08:42.042 INFO:teuthology.orchestra.run.smithi154.stdout: "AsyncMessenger::Worker-0": { 2024-03-16T12:08:42.042 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_recv_messages": 633, 2024-03-16T12:08:42.042 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_send_messages": 633, 2024-03-16T12:08:42.042 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_recv_bytes": 82013355, 2024-03-16T12:08:42.043 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_send_bytes": 200480, 2024-03-16T12:08:42.043 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_created_connections": 8, 2024-03-16T12:08:42.043 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_active_connections": 4, 2024-03-16T12:08:42.043 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_running_total_time": 0.080724219, 2024-03-16T12:08:42.043 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_running_send_time": 0.011669845, 2024-03-16T12:08:42.043 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_running_recv_time": 0.049973312, 2024-03-16T12:08:42.043 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_running_fast_dispatch_time": 0.014610047, 2024-03-16T12:08:42.043 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_send_messages_queue_lat": { 2024-03-16T12:08:42.044 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 632, 2024-03-16T12:08:42.044 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.019005174, 2024-03-16T12:08:42.044 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000030071 2024-03-16T12:08:42.044 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-16T12:08:42.044 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_handle_ack_lat": { 2024-03-16T12:08:42.044 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-16T12:08:42.044 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-16T12:08:42.044 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-16T12:08:42.044 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-16T12:08:42.045 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_recv_encrypted_bytes": 3072, 2024-03-16T12:08:42.045 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_send_encrypted_bytes": 448 2024-03-16T12:08:42.045 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-16T12:08:42.045 INFO:teuthology.orchestra.run.smithi154.stdout: "AsyncMessenger::Worker-1": { 2024-03-16T12:08:42.045 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_recv_messages": 508, 2024-03-16T12:08:42.045 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_send_messages": 506, 2024-03-16T12:08:42.045 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_recv_bytes": 126093514, 2024-03-16T12:08:42.045 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_send_bytes": 183275, 2024-03-16T12:08:42.046 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_created_connections": 5, 2024-03-16T12:08:42.046 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_active_connections": 4, 2024-03-16T12:08:42.046 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_running_total_time": 0.095566374, 2024-03-16T12:08:42.046 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_running_send_time": 0.010828832, 2024-03-16T12:08:42.046 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_running_recv_time": 0.081020232, 2024-03-16T12:08:42.046 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_running_fast_dispatch_time": 0.009571999, 2024-03-16T12:08:42.046 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_send_messages_queue_lat": { 2024-03-16T12:08:42.046 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 505, 2024-03-16T12:08:42.047 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.012794925, 2024-03-16T12:08:42.047 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000025336 2024-03-16T12:08:42.047 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-16T12:08:42.047 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_handle_ack_lat": { 2024-03-16T12:08:42.047 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-16T12:08:42.047 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-16T12:08:42.047 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-16T12:08:42.047 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-16T12:08:42.047 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_recv_encrypted_bytes": 190816, 2024-03-16T12:08:42.047 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_send_encrypted_bytes": 53792 2024-03-16T12:08:42.048 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-16T12:08:42.048 INFO:teuthology.orchestra.run.smithi154.stdout: "AsyncMessenger::Worker-2": { 2024-03-16T12:08:42.048 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_recv_messages": 478, 2024-03-16T12:08:42.048 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_send_messages": 541, 2024-03-16T12:08:42.048 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_recv_bytes": 125952849, 2024-03-16T12:08:42.048 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_send_bytes": 348576, 2024-03-16T12:08:42.048 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_created_connections": 6, 2024-03-16T12:08:42.048 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_active_connections": 5, 2024-03-16T12:08:42.048 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_running_total_time": 0.083414758, 2024-03-16T12:08:42.049 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_running_send_time": 0.010786968, 2024-03-16T12:08:42.049 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_running_recv_time": 0.060462509, 2024-03-16T12:08:42.049 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_running_fast_dispatch_time": 0.009544660, 2024-03-16T12:08:42.049 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_send_messages_queue_lat": { 2024-03-16T12:08:42.049 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 539, 2024-03-16T12:08:42.049 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.013585319, 2024-03-16T12:08:42.049 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000025204 2024-03-16T12:08:42.049 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-16T12:08:42.049 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_handle_ack_lat": { 2024-03-16T12:08:42.050 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-16T12:08:42.050 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-16T12:08:42.050 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-16T12:08:42.050 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-16T12:08:42.050 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_recv_encrypted_bytes": 160, 2024-03-16T12:08:42.050 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_send_encrypted_bytes": 196880 2024-03-16T12:08:42.050 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-16T12:08:42.050 INFO:teuthology.orchestra.run.smithi154.stdout: "cct": { 2024-03-16T12:08:42.050 INFO:teuthology.orchestra.run.smithi154.stdout: "total_workers": 1, 2024-03-16T12:08:42.051 INFO:teuthology.orchestra.run.smithi154.stdout: "unhealthy_workers": 0 2024-03-16T12:08:42.051 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-16T12:08:42.051 INFO:teuthology.orchestra.run.smithi154.stdout: "finisher-MDSRank": { 2024-03-16T12:08:42.051 INFO:teuthology.orchestra.run.smithi154.stdout: "queue_len": 0, 2024-03-16T12:08:42.051 INFO:teuthology.orchestra.run.smithi154.stdout: "complete_latency": { 2024-03-16T12:08:42.051 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 2352, 2024-03-16T12:08:42.051 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.286481472, 2024-03-16T12:08:42.051 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000121803 2024-03-16T12:08:42.051 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-03-16T12:08:42.052 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-16T12:08:42.052 INFO:teuthology.orchestra.run.smithi154.stdout: "finisher-PurgeQueue": { 2024-03-16T12:08:42.052 INFO:teuthology.orchestra.run.smithi154.stdout: "queue_len": 0, 2024-03-16T12:08:42.052 INFO:teuthology.orchestra.run.smithi154.stdout: "complete_latency": { 2024-03-16T12:08:42.052 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-16T12:08:42.052 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-16T12:08:42.052 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-16T12:08:42.052 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-03-16T12:08:42.052 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-16T12:08:42.052 INFO:teuthology.orchestra.run.smithi154.stdout: "mds": { 2024-03-16T12:08:42.053 INFO:teuthology.orchestra.run.smithi154.stdout: "request": 0, 2024-03-16T12:08:42.053 INFO:teuthology.orchestra.run.smithi154.stdout: "reply": 0, 2024-03-16T12:08:42.053 INFO:teuthology.orchestra.run.smithi154.stdout: "reply_latency": { 2024-03-16T12:08:42.053 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-16T12:08:42.053 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-16T12:08:42.053 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-16T12:08:42.053 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-16T12:08:42.053 INFO:teuthology.orchestra.run.smithi154.stdout: "slow_reply": 0, 2024-03-16T12:08:42.053 INFO:teuthology.orchestra.run.smithi154.stdout: "forward": 0, 2024-03-16T12:08:42.054 INFO:teuthology.orchestra.run.smithi154.stdout: "dir_fetch_complete": 0, 2024-03-16T12:08:42.054 INFO:teuthology.orchestra.run.smithi154.stdout: "dir_fetch_keys": 0, 2024-03-16T12:08:42.054 INFO:teuthology.orchestra.run.smithi154.stdout: "dir_commit": 0, 2024-03-16T12:08:42.054 INFO:teuthology.orchestra.run.smithi154.stdout: "dir_split": 0, 2024-03-16T12:08:42.054 INFO:teuthology.orchestra.run.smithi154.stdout: "dir_merge": 0, 2024-03-16T12:08:42.054 INFO:teuthology.orchestra.run.smithi154.stdout: "inodes": 24640, 2024-03-16T12:08:42.054 INFO:teuthology.orchestra.run.smithi154.stdout: "inodes_top": 16000, 2024-03-16T12:08:42.054 INFO:teuthology.orchestra.run.smithi154.stdout: "inodes_bottom": 8640, 2024-03-16T12:08:42.055 INFO:teuthology.orchestra.run.smithi154.stdout: "inodes_pin_tail": 0, 2024-03-16T12:08:42.055 INFO:teuthology.orchestra.run.smithi154.stdout: "inodes_pinned": 1783, 2024-03-16T12:08:42.055 INFO:teuthology.orchestra.run.smithi154.stdout: "inodes_expired": 0, 2024-03-16T12:08:42.055 INFO:teuthology.orchestra.run.smithi154.stdout: "inodes_with_caps": 0, 2024-03-16T12:08:42.055 INFO:teuthology.orchestra.run.smithi154.stdout: "caps": 0, 2024-03-16T12:08:42.055 INFO:teuthology.orchestra.run.smithi154.stdout: "subtrees": 14, 2024-03-16T12:08:42.055 INFO:teuthology.orchestra.run.smithi154.stdout: "traverse": 0, 2024-03-16T12:08:42.055 INFO:teuthology.orchestra.run.smithi154.stdout: "traverse_hit": 0, 2024-03-16T12:08:42.055 INFO:teuthology.orchestra.run.smithi154.stdout: "traverse_forward": 0, 2024-03-16T12:08:42.056 INFO:teuthology.orchestra.run.smithi154.stdout: "traverse_discover": 0, 2024-03-16T12:08:42.056 INFO:teuthology.orchestra.run.smithi154.stdout: "traverse_dir_fetch": 0, 2024-03-16T12:08:42.056 INFO:teuthology.orchestra.run.smithi154.stdout: "traverse_remote_ino": 0, 2024-03-16T12:08:42.056 INFO:teuthology.orchestra.run.smithi154.stdout: "traverse_lock": 0, 2024-03-16T12:08:42.056 INFO:teuthology.orchestra.run.smithi154.stdout: "load_cent": 0, 2024-03-16T12:08:42.056 INFO:teuthology.orchestra.run.smithi154.stdout: "q": 0, 2024-03-16T12:08:42.056 INFO:teuthology.orchestra.run.smithi154.stdout: "exported": 0, 2024-03-16T12:08:42.056 INFO:teuthology.orchestra.run.smithi154.stdout: "exported_inodes": 0, 2024-03-16T12:08:42.056 INFO:teuthology.orchestra.run.smithi154.stdout: "imported": 0, 2024-03-16T12:08:42.057 INFO:teuthology.orchestra.run.smithi154.stdout: "imported_inodes": 0, 2024-03-16T12:08:42.057 INFO:teuthology.orchestra.run.smithi154.stdout: "openino_dir_fetch": 0, 2024-03-16T12:08:42.057 INFO:teuthology.orchestra.run.smithi154.stdout: "openino_backtrace_fetch": 0, 2024-03-16T12:08:42.057 INFO:teuthology.orchestra.run.smithi154.stdout: "openino_peer_discover": 0, 2024-03-16T12:08:42.057 INFO:teuthology.orchestra.run.smithi154.stdout: "root_rfiles": 669, 2024-03-16T12:08:42.057 INFO:teuthology.orchestra.run.smithi154.stdout: "root_rbytes": 211775824, 2024-03-16T12:08:42.057 INFO:teuthology.orchestra.run.smithi154.stdout: "root_rsnaps": 7, 2024-03-16T12:08:42.057 INFO:teuthology.orchestra.run.smithi154.stdout: "scrub_backtrace_fetch": 0, 2024-03-16T12:08:42.057 INFO:teuthology.orchestra.run.smithi154.stdout: "scrub_set_tag": 0, 2024-03-16T12:08:42.058 INFO:teuthology.orchestra.run.smithi154.stdout: "scrub_backtrace_repaired": 0, 2024-03-16T12:08:42.058 INFO:teuthology.orchestra.run.smithi154.stdout: "scrub_inotable_repaired": 0, 2024-03-16T12:08:42.058 INFO:teuthology.orchestra.run.smithi154.stdout: "scrub_dir_inodes": 0, 2024-03-16T12:08:42.058 INFO:teuthology.orchestra.run.smithi154.stdout: "scrub_dir_base_inodes": 0, 2024-03-16T12:08:42.058 INFO:teuthology.orchestra.run.smithi154.stdout: "scrub_dirfrag_rstats": 0, 2024-03-16T12:08:42.058 INFO:teuthology.orchestra.run.smithi154.stdout: "scrub_file_inodes": 0, 2024-03-16T12:08:42.058 INFO:teuthology.orchestra.run.smithi154.stdout: "handle_inode_file_caps": 0, 2024-03-16T12:08:42.058 INFO:teuthology.orchestra.run.smithi154.stdout: "ceph_cap_op_revoke": 0, 2024-03-16T12:08:42.058 INFO:teuthology.orchestra.run.smithi154.stdout: "ceph_cap_op_grant": 0, 2024-03-16T12:08:42.059 INFO:teuthology.orchestra.run.smithi154.stdout: "ceph_cap_op_trunc": 0, 2024-03-16T12:08:42.059 INFO:teuthology.orchestra.run.smithi154.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-16T12:08:42.059 INFO:teuthology.orchestra.run.smithi154.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-16T12:08:42.059 INFO:teuthology.orchestra.run.smithi154.stdout: "handle_client_caps": 0, 2024-03-16T12:08:42.059 INFO:teuthology.orchestra.run.smithi154.stdout: "handle_client_caps_dirty": 0, 2024-03-16T12:08:42.059 INFO:teuthology.orchestra.run.smithi154.stdout: "handle_client_cap_release": 0, 2024-03-16T12:08:42.059 INFO:teuthology.orchestra.run.smithi154.stdout: "process_request_cap_release": 0 2024-03-16T12:08:42.059 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-16T12:08:42.059 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_cache": { 2024-03-16T12:08:42.060 INFO:teuthology.orchestra.run.smithi154.stdout: "dir_update": 0, 2024-03-16T12:08:42.060 INFO:teuthology.orchestra.run.smithi154.stdout: "dir_update_receipt": 0, 2024-03-16T12:08:42.060 INFO:teuthology.orchestra.run.smithi154.stdout: "dir_try_discover": 0, 2024-03-16T12:08:42.060 INFO:teuthology.orchestra.run.smithi154.stdout: "dir_send_discover": 0, 2024-03-16T12:08:42.060 INFO:teuthology.orchestra.run.smithi154.stdout: "dir_handle_discover": 0, 2024-03-16T12:08:42.060 INFO:teuthology.orchestra.run.smithi154.stdout: "num_strays": 0, 2024-03-16T12:08:42.060 INFO:teuthology.orchestra.run.smithi154.stdout: "num_strays_delayed": 0, 2024-03-16T12:08:42.060 INFO:teuthology.orchestra.run.smithi154.stdout: "num_strays_enqueuing": 0, 2024-03-16T12:08:42.060 INFO:teuthology.orchestra.run.smithi154.stdout: "strays_created": 0, 2024-03-16T12:08:42.061 INFO:teuthology.orchestra.run.smithi154.stdout: "strays_enqueued": 0, 2024-03-16T12:08:42.061 INFO:teuthology.orchestra.run.smithi154.stdout: "strays_reintegrated": 0, 2024-03-16T12:08:42.061 INFO:teuthology.orchestra.run.smithi154.stdout: "strays_migrated": 0, 2024-03-16T12:08:42.061 INFO:teuthology.orchestra.run.smithi154.stdout: "num_recovering_processing": 0, 2024-03-16T12:08:42.061 INFO:teuthology.orchestra.run.smithi154.stdout: "num_recovering_enqueued": 0, 2024-03-16T12:08:42.061 INFO:teuthology.orchestra.run.smithi154.stdout: "num_recovering_prioritized": 0, 2024-03-16T12:08:42.061 INFO:teuthology.orchestra.run.smithi154.stdout: "recovery_started": 0, 2024-03-16T12:08:42.062 INFO:teuthology.orchestra.run.smithi154.stdout: "recovery_completed": 0, 2024-03-16T12:08:42.062 INFO:teuthology.orchestra.run.smithi154.stdout: "ireq_quiesce_path": 0, 2024-03-16T12:08:42.062 INFO:teuthology.orchestra.run.smithi154.stdout: "ireq_quiesce_inode": 0, 2024-03-16T12:08:42.062 INFO:teuthology.orchestra.run.smithi154.stdout: "ireq_enqueue_scrub": 0, 2024-03-16T12:08:42.062 INFO:teuthology.orchestra.run.smithi154.stdout: "ireq_exportdir": 0, 2024-03-16T12:08:42.062 INFO:teuthology.orchestra.run.smithi154.stdout: "ireq_flush": 0, 2024-03-16T12:08:42.062 INFO:teuthology.orchestra.run.smithi154.stdout: "ireq_fragmentdir": 0, 2024-03-16T12:08:42.062 INFO:teuthology.orchestra.run.smithi154.stdout: "ireq_fragstats": 0, 2024-03-16T12:08:42.063 INFO:teuthology.orchestra.run.smithi154.stdout: "ireq_inodestats": 0 2024-03-16T12:08:42.063 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-16T12:08:42.063 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_log": { 2024-03-16T12:08:42.063 INFO:teuthology.orchestra.run.smithi154.stdout: "evlrg": 0, 2024-03-16T12:08:42.063 INFO:teuthology.orchestra.run.smithi154.stdout: "evadd": 0, 2024-03-16T12:08:42.063 INFO:teuthology.orchestra.run.smithi154.stdout: "evex": 0, 2024-03-16T12:08:42.063 INFO:teuthology.orchestra.run.smithi154.stdout: "evtrm": 0, 2024-03-16T12:08:42.063 INFO:teuthology.orchestra.run.smithi154.stdout: "ev": 105967, 2024-03-16T12:08:42.064 INFO:teuthology.orchestra.run.smithi154.stdout: "evexg": 0, 2024-03-16T12:08:42.064 INFO:teuthology.orchestra.run.smithi154.stdout: "evexd": 0, 2024-03-16T12:08:42.064 INFO:teuthology.orchestra.run.smithi154.stdout: "segadd": 0, 2024-03-16T12:08:42.064 INFO:teuthology.orchestra.run.smithi154.stdout: "segex": 0, 2024-03-16T12:08:42.064 INFO:teuthology.orchestra.run.smithi154.stdout: "segtrm": 0, 2024-03-16T12:08:42.064 INFO:teuthology.orchestra.run.smithi154.stdout: "seg": 4, 2024-03-16T12:08:42.064 INFO:teuthology.orchestra.run.smithi154.stdout: "segmjr": 20, 2024-03-16T12:08:42.064 INFO:teuthology.orchestra.run.smithi154.stdout: "segexg": 0, 2024-03-16T12:08:42.064 INFO:teuthology.orchestra.run.smithi154.stdout: "segexd": 0, 2024-03-16T12:08:42.065 INFO:teuthology.orchestra.run.smithi154.stdout: "expos": 1063115311, 2024-03-16T12:08:42.065 INFO:teuthology.orchestra.run.smithi154.stdout: "wrpos": 1071653073, 2024-03-16T12:08:42.065 INFO:teuthology.orchestra.run.smithi154.stdout: "rdpos": 1071642628, 2024-03-16T12:08:42.065 INFO:teuthology.orchestra.run.smithi154.stdout: "jlat": { 2024-03-16T12:08:42.065 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-16T12:08:42.065 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-16T12:08:42.065 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-16T12:08:42.065 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-16T12:08:42.066 INFO:teuthology.orchestra.run.smithi154.stdout: "replayed": 105967 2024-03-16T12:08:42.066 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-16T12:08:42.066 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_mem": { 2024-03-16T12:08:42.066 INFO:teuthology.orchestra.run.smithi154.stdout: "ino": 2671, 2024-03-16T12:08:42.066 INFO:teuthology.orchestra.run.smithi154.stdout: "ino+": 24031, 2024-03-16T12:08:42.066 INFO:teuthology.orchestra.run.smithi154.stdout: "ino-": 21360, 2024-03-16T12:08:42.066 INFO:teuthology.orchestra.run.smithi154.stdout: "dir": 191, 2024-03-16T12:08:42.066 INFO:teuthology.orchestra.run.smithi154.stdout: "dir+": 267, 2024-03-16T12:08:42.067 INFO:teuthology.orchestra.run.smithi154.stdout: "dir-": 76, 2024-03-16T12:08:42.067 INFO:teuthology.orchestra.run.smithi154.stdout: "dn": 24640, 2024-03-16T12:08:42.067 INFO:teuthology.orchestra.run.smithi154.stdout: "dn+": 24697, 2024-03-16T12:08:42.067 INFO:teuthology.orchestra.run.smithi154.stdout: "dn-": 57, 2024-03-16T12:08:42.067 INFO:teuthology.orchestra.run.smithi154.stdout: "cap": 0, 2024-03-16T12:08:42.067 INFO:teuthology.orchestra.run.smithi154.stdout: "cap+": 0, 2024-03-16T12:08:42.067 INFO:teuthology.orchestra.run.smithi154.stdout: "cap-": 0, 2024-03-16T12:08:42.067 INFO:teuthology.orchestra.run.smithi154.stdout: "rss": 82764, 2024-03-16T12:08:42.067 INFO:teuthology.orchestra.run.smithi154.stdout: "heap": 215396 2024-03-16T12:08:42.068 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-16T12:08:42.068 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_server": { 2024-03-16T12:08:42.068 INFO:teuthology.orchestra.run.smithi154.stdout: "dispatch_client_request": 0, 2024-03-16T12:08:42.068 INFO:teuthology.orchestra.run.smithi154.stdout: "dispatch_server_request": 0, 2024-03-16T12:08:42.068 INFO:teuthology.orchestra.run.smithi154.stdout: "handle_client_request": 0, 2024-03-16T12:08:42.068 INFO:teuthology.orchestra.run.smithi154.stdout: "handle_client_session": 0, 2024-03-16T12:08:42.068 INFO:teuthology.orchestra.run.smithi154.stdout: "handle_peer_request": 0, 2024-03-16T12:08:42.068 INFO:teuthology.orchestra.run.smithi154.stdout: "req_create_latency": { 2024-03-16T12:08:42.068 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-16T12:08:42.069 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-16T12:08:42.069 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-16T12:08:42.069 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-16T12:08:42.069 INFO:teuthology.orchestra.run.smithi154.stdout: "req_getattr_latency": { 2024-03-16T12:08:42.069 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-16T12:08:42.069 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-16T12:08:42.069 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-16T12:08:42.069 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-16T12:08:42.069 INFO:teuthology.orchestra.run.smithi154.stdout: "req_getfilelock_latency": { 2024-03-16T12:08:42.070 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-16T12:08:42.070 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-16T12:08:42.070 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-16T12:08:42.070 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-16T12:08:42.070 INFO:teuthology.orchestra.run.smithi154.stdout: "req_link_latency": { 2024-03-16T12:08:42.070 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-16T12:08:42.070 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-16T12:08:42.071 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-16T12:08:42.071 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-16T12:08:42.071 INFO:teuthology.orchestra.run.smithi154.stdout: "req_lookup_latency": { 2024-03-16T12:08:42.071 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-16T12:08:42.071 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-16T12:08:42.071 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-16T12:08:42.071 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-16T12:08:42.071 INFO:teuthology.orchestra.run.smithi154.stdout: "req_lookuphash_latency": { 2024-03-16T12:08:42.072 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-16T12:08:42.072 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-16T12:08:42.072 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-16T12:08:42.072 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-16T12:08:42.072 INFO:teuthology.orchestra.run.smithi154.stdout: "req_lookupino_latency": { 2024-03-16T12:08:42.072 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-16T12:08:42.072 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-16T12:08:42.072 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-16T12:08:42.072 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-16T12:08:42.073 INFO:teuthology.orchestra.run.smithi154.stdout: "req_lookupname_latency": { 2024-03-16T12:08:42.073 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-16T12:08:42.073 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-16T12:08:42.073 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-16T12:08:42.073 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-16T12:08:42.073 INFO:teuthology.orchestra.run.smithi154.stdout: "req_lookupparent_latency": { 2024-03-16T12:08:42.073 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-16T12:08:42.073 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-16T12:08:42.073 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-16T12:08:42.074 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-16T12:08:42.074 INFO:teuthology.orchestra.run.smithi154.stdout: "req_lookupsnap_latency": { 2024-03-16T12:08:42.074 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-16T12:08:42.074 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-16T12:08:42.074 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-16T12:08:42.074 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-16T12:08:42.074 INFO:teuthology.orchestra.run.smithi154.stdout: "req_lssnap_latency": { 2024-03-16T12:08:42.074 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-16T12:08:42.075 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-16T12:08:42.075 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-16T12:08:42.075 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-16T12:08:42.075 INFO:teuthology.orchestra.run.smithi154.stdout: "req_mkdir_latency": { 2024-03-16T12:08:42.076 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-16T12:08:42.076 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-16T12:08:42.076 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-16T12:08:42.076 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-16T12:08:42.076 INFO:teuthology.orchestra.run.smithi154.stdout: "req_mknod_latency": { 2024-03-16T12:08:42.076 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-16T12:08:42.076 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-16T12:08:42.076 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-16T12:08:42.077 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-16T12:08:42.077 INFO:teuthology.orchestra.run.smithi154.stdout: "req_mksnap_latency": { 2024-03-16T12:08:42.077 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-16T12:08:42.077 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-16T12:08:42.077 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-16T12:08:42.077 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-16T12:08:42.077 INFO:teuthology.orchestra.run.smithi154.stdout: "req_open_latency": { 2024-03-16T12:08:42.077 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-16T12:08:42.077 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-16T12:08:42.078 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-16T12:08:42.078 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-16T12:08:42.078 INFO:teuthology.orchestra.run.smithi154.stdout: "req_readdir_latency": { 2024-03-16T12:08:42.078 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-16T12:08:42.078 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-16T12:08:42.078 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-16T12:08:42.078 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-16T12:08:42.078 INFO:teuthology.orchestra.run.smithi154.stdout: "req_rename_latency": { 2024-03-16T12:08:42.079 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-16T12:08:42.079 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-16T12:08:42.079 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-16T12:08:42.079 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-16T12:08:42.079 INFO:teuthology.orchestra.run.smithi154.stdout: "req_renamesnap_latency": { 2024-03-16T12:08:42.079 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-16T12:08:42.079 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-16T12:08:42.079 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-16T12:08:42.080 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-16T12:08:42.080 INFO:teuthology.orchestra.run.smithi154.stdout: "req_snapdiff_latency": { 2024-03-16T12:08:42.080 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-16T12:08:42.080 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-16T12:08:42.080 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-16T12:08:42.080 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-16T12:08:42.080 INFO:teuthology.orchestra.run.smithi154.stdout: "req_rmdir_latency": { 2024-03-16T12:08:42.080 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-16T12:08:42.080 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-16T12:08:42.081 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-16T12:08:42.081 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-16T12:08:42.081 INFO:teuthology.orchestra.run.smithi154.stdout: "req_rmsnap_latency": { 2024-03-16T12:08:42.081 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-16T12:08:42.081 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-16T12:08:42.081 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-16T12:08:42.081 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-16T12:08:42.081 INFO:teuthology.orchestra.run.smithi154.stdout: "req_rmxattr_latency": { 2024-03-16T12:08:42.081 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-16T12:08:42.082 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-16T12:08:42.082 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-16T12:08:42.082 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-16T12:08:42.082 INFO:teuthology.orchestra.run.smithi154.stdout: "req_setattr_latency": { 2024-03-16T12:08:42.082 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-16T12:08:42.082 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-16T12:08:42.082 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-16T12:08:42.083 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-16T12:08:42.083 INFO:teuthology.orchestra.run.smithi154.stdout: "req_setdirlayout_latency": { 2024-03-16T12:08:42.083 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-16T12:08:42.083 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-16T12:08:42.084 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-16T12:08:42.084 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-16T12:08:42.084 INFO:teuthology.orchestra.run.smithi154.stdout: "req_setfilelock_latency": { 2024-03-16T12:08:42.084 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-16T12:08:42.084 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-16T12:08:42.085 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-16T12:08:42.085 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-16T12:08:42.085 INFO:teuthology.orchestra.run.smithi154.stdout: "req_setlayout_latency": { 2024-03-16T12:08:42.085 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-16T12:08:42.086 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-16T12:08:42.086 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-16T12:08:42.086 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-16T12:08:42.086 INFO:teuthology.orchestra.run.smithi154.stdout: "req_setxattr_latency": { 2024-03-16T12:08:42.087 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-16T12:08:42.087 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-16T12:08:42.087 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-16T12:08:42.087 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-16T12:08:42.088 INFO:teuthology.orchestra.run.smithi154.stdout: "req_symlink_latency": { 2024-03-16T12:08:42.088 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-16T12:08:42.088 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-16T12:08:42.088 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-16T12:08:42.089 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-16T12:08:42.089 INFO:teuthology.orchestra.run.smithi154.stdout: "req_unlink_latency": { 2024-03-16T12:08:42.089 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-16T12:08:42.089 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-16T12:08:42.090 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-16T12:08:42.090 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-16T12:08:42.090 INFO:teuthology.orchestra.run.smithi154.stdout: "cap_revoke_eviction": 0, 2024-03-16T12:08:42.090 INFO:teuthology.orchestra.run.smithi154.stdout: "cap_acquisition_throttle": 0, 2024-03-16T12:08:42.090 INFO:teuthology.orchestra.run.smithi154.stdout: "req_getvxattr_latency": { 2024-03-16T12:08:42.091 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-16T12:08:42.091 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-16T12:08:42.091 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-16T12:08:42.091 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-03-16T12:08:42.092 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-16T12:08:42.092 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_sessions": { 2024-03-16T12:08:42.092 INFO:teuthology.orchestra.run.smithi154.stdout: "session_count": 2, 2024-03-16T12:08:42.092 INFO:teuthology.orchestra.run.smithi154.stdout: "session_add": 2, 2024-03-16T12:08:42.093 INFO:teuthology.orchestra.run.smithi154.stdout: "session_remove": 0, 2024-03-16T12:08:42.093 INFO:teuthology.orchestra.run.smithi154.stdout: "sessions_open": 0, 2024-03-16T12:08:42.093 INFO:teuthology.orchestra.run.smithi154.stdout: "sessions_stale": 0, 2024-03-16T12:08:42.093 INFO:teuthology.orchestra.run.smithi154.stdout: "total_load": 0, 2024-03-16T12:08:42.093 INFO:teuthology.orchestra.run.smithi154.stdout: "average_load": 0, 2024-03-16T12:08:42.094 INFO:teuthology.orchestra.run.smithi154.stdout: "avg_session_uptime": 2535, 2024-03-16T12:08:42.094 INFO:teuthology.orchestra.run.smithi154.stdout: "mdthresh_evicted": 0 2024-03-16T12:08:42.094 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-16T12:08:42.094 INFO:teuthology.orchestra.run.smithi154.stdout: "mempool": { 2024-03-16T12:08:42.095 INFO:teuthology.orchestra.run.smithi154.stdout: "bloom_filter_bytes": 0, 2024-03-16T12:08:42.095 INFO:teuthology.orchestra.run.smithi154.stdout: "bloom_filter_items": 0, 2024-03-16T12:08:42.095 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_alloc_bytes": 0, 2024-03-16T12:08:42.096 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_alloc_items": 0, 2024-03-16T12:08:42.096 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_cache_data_bytes": 0, 2024-03-16T12:08:42.096 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_cache_data_items": 0, 2024-03-16T12:08:42.096 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-16T12:08:42.097 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_cache_onode_items": 0, 2024-03-16T12:08:42.097 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-16T12:08:42.097 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_cache_meta_items": 0, 2024-03-16T12:08:42.097 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_cache_other_bytes": 0, 2024-03-16T12:08:42.097 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_cache_other_items": 0, 2024-03-16T12:08:42.098 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-16T12:08:42.098 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_cache_buffer_items": 0, 2024-03-16T12:08:42.098 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_extent_bytes": 0, 2024-03-16T12:08:42.099 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_extent_items": 0, 2024-03-16T12:08:42.099 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_blob_bytes": 0, 2024-03-16T12:08:42.099 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_blob_items": 0, 2024-03-16T12:08:42.099 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-16T12:08:42.100 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_shared_blob_items": 0, 2024-03-16T12:08:42.100 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-16T12:08:42.100 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_inline_bl_items": 0, 2024-03-16T12:08:42.100 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_fsck_bytes": 0, 2024-03-16T12:08:42.101 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_fsck_items": 0, 2024-03-16T12:08:42.101 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_txc_bytes": 0, 2024-03-16T12:08:42.101 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_txc_items": 0, 2024-03-16T12:08:42.101 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-16T12:08:42.102 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_writing_deferred_items": 0, 2024-03-16T12:08:42.102 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_writing_bytes": 0, 2024-03-16T12:08:42.102 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_writing_items": 0, 2024-03-16T12:08:42.102 INFO:teuthology.orchestra.run.smithi154.stdout: "bluefs_bytes": 0, 2024-03-16T12:08:42.103 INFO:teuthology.orchestra.run.smithi154.stdout: "bluefs_items": 0, 2024-03-16T12:08:42.103 INFO:teuthology.orchestra.run.smithi154.stdout: "bluefs_file_reader_bytes": 0, 2024-03-16T12:08:42.103 INFO:teuthology.orchestra.run.smithi154.stdout: "bluefs_file_reader_items": 0, 2024-03-16T12:08:42.103 INFO:teuthology.orchestra.run.smithi154.stdout: "bluefs_file_writer_bytes": 0, 2024-03-16T12:08:42.104 INFO:teuthology.orchestra.run.smithi154.stdout: "bluefs_file_writer_items": 0, 2024-03-16T12:08:42.104 INFO:teuthology.orchestra.run.smithi154.stdout: "buffer_anon_bytes": 347516, 2024-03-16T12:08:42.104 INFO:teuthology.orchestra.run.smithi154.stdout: "buffer_anon_items": 3227, 2024-03-16T12:08:42.104 INFO:teuthology.orchestra.run.smithi154.stdout: "buffer_meta_bytes": 0, 2024-03-16T12:08:42.105 INFO:teuthology.orchestra.run.smithi154.stdout: "buffer_meta_items": 0, 2024-03-16T12:08:42.105 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_bytes": 0, 2024-03-16T12:08:42.105 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_items": 0, 2024-03-16T12:08:42.105 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_mapbl_bytes": 0, 2024-03-16T12:08:42.106 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_mapbl_items": 0, 2024-03-16T12:08:42.106 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_pglog_bytes": 0, 2024-03-16T12:08:42.106 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_pglog_items": 0, 2024-03-16T12:08:42.106 INFO:teuthology.orchestra.run.smithi154.stdout: "osdmap_bytes": 8256, 2024-03-16T12:08:42.106 INFO:teuthology.orchestra.run.smithi154.stdout: "osdmap_items": 188, 2024-03-16T12:08:42.107 INFO:teuthology.orchestra.run.smithi154.stdout: "osdmap_mapping_bytes": 0, 2024-03-16T12:08:42.107 INFO:teuthology.orchestra.run.smithi154.stdout: "osdmap_mapping_items": 0, 2024-03-16T12:08:42.107 INFO:teuthology.orchestra.run.smithi154.stdout: "pgmap_bytes": 0, 2024-03-16T12:08:42.107 INFO:teuthology.orchestra.run.smithi154.stdout: "pgmap_items": 0, 2024-03-16T12:08:42.107 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_co_bytes": 24231531, 2024-03-16T12:08:42.107 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_co_items": 171645, 2024-03-16T12:08:42.108 INFO:teuthology.orchestra.run.smithi154.stdout: "unittest_1_bytes": 0, 2024-03-16T12:08:42.108 INFO:teuthology.orchestra.run.smithi154.stdout: "unittest_1_items": 0, 2024-03-16T12:08:42.108 INFO:teuthology.orchestra.run.smithi154.stdout: "unittest_2_bytes": 0, 2024-03-16T12:08:42.108 INFO:teuthology.orchestra.run.smithi154.stdout: "unittest_2_items": 0 2024-03-16T12:08:42.108 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-16T12:08:42.109 INFO:teuthology.orchestra.run.smithi154.stdout: "objecter": { 2024-03-16T12:08:42.109 INFO:teuthology.orchestra.run.smithi154.stdout: "op_active": 0, 2024-03-16T12:08:42.109 INFO:teuthology.orchestra.run.smithi154.stdout: "op_laggy": 0, 2024-03-16T12:08:42.109 INFO:teuthology.orchestra.run.smithi154.stdout: "op_send": 1513, 2024-03-16T12:08:42.110 INFO:teuthology.orchestra.run.smithi154.stdout: "op_send_bytes": 26, 2024-03-16T12:08:42.110 INFO:teuthology.orchestra.run.smithi154.stdout: "op_resend": 4, 2024-03-16T12:08:42.110 INFO:teuthology.orchestra.run.smithi154.stdout: "op_reply": 1513, 2024-03-16T12:08:42.110 INFO:teuthology.orchestra.run.smithi154.stdout: "op_latency": { 2024-03-16T12:08:42.110 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 1513, 2024-03-16T12:08:42.110 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 2.990926912, 2024-03-16T12:08:42.111 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.001976818 2024-03-16T12:08:42.111 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-16T12:08:42.111 INFO:teuthology.orchestra.run.smithi154.stdout: "op_inflight": 0, 2024-03-16T12:08:42.111 INFO:teuthology.orchestra.run.smithi154.stdout: "oplen_avg": { 2024-03-16T12:08:42.111 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 1513, 2024-03-16T12:08:42.111 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 1514 2024-03-16T12:08:42.111 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-16T12:08:42.111 INFO:teuthology.orchestra.run.smithi154.stdout: "op": 1513, 2024-03-16T12:08:42.111 INFO:teuthology.orchestra.run.smithi154.stdout: "op_r": 1513, 2024-03-16T12:08:42.112 INFO:teuthology.orchestra.run.smithi154.stdout: "op_w": 0, 2024-03-16T12:08:42.112 INFO:teuthology.orchestra.run.smithi154.stdout: "op_rmw": 0, 2024-03-16T12:08:42.112 INFO:teuthology.orchestra.run.smithi154.stdout: "op_pg": 0, 2024-03-16T12:08:42.112 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_stat": 869, 2024-03-16T12:08:42.112 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_create": 0, 2024-03-16T12:08:42.112 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_read": 641, 2024-03-16T12:08:42.112 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_write": 0, 2024-03-16T12:08:42.112 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_writefull": 0, 2024-03-16T12:08:42.113 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_writesame": 0, 2024-03-16T12:08:42.113 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_append": 0, 2024-03-16T12:08:42.113 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_zero": 0, 2024-03-16T12:08:42.113 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_truncate": 0, 2024-03-16T12:08:42.113 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_delete": 0, 2024-03-16T12:08:42.113 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_mapext": 0, 2024-03-16T12:08:42.113 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_sparse_read": 0, 2024-03-16T12:08:42.113 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_clonerange": 0, 2024-03-16T12:08:42.113 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_getxattr": 2, 2024-03-16T12:08:42.114 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_setxattr": 0, 2024-03-16T12:08:42.114 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_cmpxattr": 0, 2024-03-16T12:08:42.114 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_rmxattr": 0, 2024-03-16T12:08:42.114 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_resetxattrs": 0, 2024-03-16T12:08:42.114 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_call": 0, 2024-03-16T12:08:42.114 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_watch": 0, 2024-03-16T12:08:42.114 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_notify": 0, 2024-03-16T12:08:42.114 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_src_cmpxattr": 0, 2024-03-16T12:08:42.115 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_pgls": 0, 2024-03-16T12:08:42.115 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_pgls_filter": 0, 2024-03-16T12:08:42.115 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_other": 0, 2024-03-16T12:08:42.115 INFO:teuthology.orchestra.run.smithi154.stdout: "linger_active": 0, 2024-03-16T12:08:42.115 INFO:teuthology.orchestra.run.smithi154.stdout: "linger_send": 0, 2024-03-16T12:08:42.115 INFO:teuthology.orchestra.run.smithi154.stdout: "linger_resend": 0, 2024-03-16T12:08:42.115 INFO:teuthology.orchestra.run.smithi154.stdout: "linger_ping": 0, 2024-03-16T12:08:42.115 INFO:teuthology.orchestra.run.smithi154.stdout: "poolop_active": 0, 2024-03-16T12:08:42.115 INFO:teuthology.orchestra.run.smithi154.stdout: "poolop_send": 0, 2024-03-16T12:08:42.116 INFO:teuthology.orchestra.run.smithi154.stdout: "poolop_resend": 0, 2024-03-16T12:08:42.116 INFO:teuthology.orchestra.run.smithi154.stdout: "poolstat_active": 0, 2024-03-16T12:08:42.116 INFO:teuthology.orchestra.run.smithi154.stdout: "poolstat_send": 0, 2024-03-16T12:08:42.116 INFO:teuthology.orchestra.run.smithi154.stdout: "poolstat_resend": 0, 2024-03-16T12:08:42.116 INFO:teuthology.orchestra.run.smithi154.stdout: "statfs_active": 0, 2024-03-16T12:08:42.116 INFO:teuthology.orchestra.run.smithi154.stdout: "statfs_send": 0, 2024-03-16T12:08:42.116 INFO:teuthology.orchestra.run.smithi154.stdout: "statfs_resend": 0, 2024-03-16T12:08:42.116 INFO:teuthology.orchestra.run.smithi154.stdout: "command_active": 0, 2024-03-16T12:08:42.116 INFO:teuthology.orchestra.run.smithi154.stdout: "command_send": 0, 2024-03-16T12:08:42.117 INFO:teuthology.orchestra.run.smithi154.stdout: "command_resend": 0, 2024-03-16T12:08:42.117 INFO:teuthology.orchestra.run.smithi154.stdout: "map_epoch": 116, 2024-03-16T12:08:42.117 INFO:teuthology.orchestra.run.smithi154.stdout: "map_full": 0, 2024-03-16T12:08:42.117 INFO:teuthology.orchestra.run.smithi154.stdout: "map_inc": 8, 2024-03-16T12:08:42.117 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_sessions": 10, 2024-03-16T12:08:42.117 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_session_open": 10, 2024-03-16T12:08:42.117 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_session_close": 0, 2024-03-16T12:08:42.117 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_laggy": 0, 2024-03-16T12:08:42.118 INFO:teuthology.orchestra.run.smithi154.stdout: "omap_wr": 0, 2024-03-16T12:08:42.118 INFO:teuthology.orchestra.run.smithi154.stdout: "omap_rd": 2, 2024-03-16T12:08:42.118 INFO:teuthology.orchestra.run.smithi154.stdout: "omap_del": 0 2024-03-16T12:08:42.118 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-16T12:08:42.118 INFO:teuthology.orchestra.run.smithi154.stdout: "oft": { 2024-03-16T12:08:42.118 INFO:teuthology.orchestra.run.smithi154.stdout: "omap_total_objs": 0, 2024-03-16T12:08:42.118 INFO:teuthology.orchestra.run.smithi154.stdout: "omap_total_kv_pairs": 0, 2024-03-16T12:08:42.118 INFO:teuthology.orchestra.run.smithi154.stdout: "omap_total_updates": 0, 2024-03-16T12:08:42.119 INFO:teuthology.orchestra.run.smithi154.stdout: "omap_total_removes": 0 2024-03-16T12:08:42.119 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-16T12:08:42.119 INFO:teuthology.orchestra.run.smithi154.stdout: "osd-slow-ops": { 2024-03-16T12:08:42.119 INFO:teuthology.orchestra.run.smithi154.stdout: "slow_ops_count": 0 2024-03-16T12:08:42.119 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-16T12:08:42.119 INFO:teuthology.orchestra.run.smithi154.stdout: "purge_queue": { 2024-03-16T12:08:42.119 INFO:teuthology.orchestra.run.smithi154.stdout: "pq_executing_ops": 0, 2024-03-16T12:08:42.119 INFO:teuthology.orchestra.run.smithi154.stdout: "pq_executing_ops_high_water": 0, 2024-03-16T12:08:42.119 INFO:teuthology.orchestra.run.smithi154.stdout: "pq_executing": 0, 2024-03-16T12:08:42.120 INFO:teuthology.orchestra.run.smithi154.stdout: "pq_executing_high_water": 0, 2024-03-16T12:08:42.120 INFO:teuthology.orchestra.run.smithi154.stdout: "pq_executed_ops": 0, 2024-03-16T12:08:42.120 INFO:teuthology.orchestra.run.smithi154.stdout: "pq_executed": 0, 2024-03-16T12:08:42.120 INFO:teuthology.orchestra.run.smithi154.stdout: "pq_item_in_journal": 0 2024-03-16T12:08:42.120 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-16T12:08:42.120 INFO:teuthology.orchestra.run.smithi154.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-16T12:08:42.120 INFO:teuthology.orchestra.run.smithi154.stdout: "val": 0, 2024-03-16T12:08:42.120 INFO:teuthology.orchestra.run.smithi154.stdout: "max": 104857600, 2024-03-16T12:08:42.120 INFO:teuthology.orchestra.run.smithi154.stdout: "get_started": 0, 2024-03-16T12:08:42.120 INFO:teuthology.orchestra.run.smithi154.stdout: "get": 1615, 2024-03-16T12:08:42.121 INFO:teuthology.orchestra.run.smithi154.stdout: "get_sum": 333906158, 2024-03-16T12:08:42.121 INFO:teuthology.orchestra.run.smithi154.stdout: "get_or_fail_fail": 0, 2024-03-16T12:08:42.121 INFO:teuthology.orchestra.run.smithi154.stdout: "get_or_fail_success": 1615, 2024-03-16T12:08:42.121 INFO:teuthology.orchestra.run.smithi154.stdout: "take": 0, 2024-03-16T12:08:42.121 INFO:teuthology.orchestra.run.smithi154.stdout: "take_sum": 0, 2024-03-16T12:08:42.121 INFO:teuthology.orchestra.run.smithi154.stdout: "put": 1615, 2024-03-16T12:08:42.121 INFO:teuthology.orchestra.run.smithi154.stdout: "put_sum": 333906158, 2024-03-16T12:08:42.121 INFO:teuthology.orchestra.run.smithi154.stdout: "wait": { 2024-03-16T12:08:42.121 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-16T12:08:42.121 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-16T12:08:42.122 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-16T12:08:42.122 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-03-16T12:08:42.122 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-16T12:08:42.122 INFO:teuthology.orchestra.run.smithi154.stdout: "throttle-objecter_bytes": { 2024-03-16T12:08:42.122 INFO:teuthology.orchestra.run.smithi154.stdout: "val": 0, 2024-03-16T12:08:42.122 INFO:teuthology.orchestra.run.smithi154.stdout: "max": 104857600, 2024-03-16T12:08:42.122 INFO:teuthology.orchestra.run.smithi154.stdout: "get_started": 0, 2024-03-16T12:08:42.122 INFO:teuthology.orchestra.run.smithi154.stdout: "get": 0, 2024-03-16T12:08:42.123 INFO:teuthology.orchestra.run.smithi154.stdout: "get_sum": 0, 2024-03-16T12:08:42.123 INFO:teuthology.orchestra.run.smithi154.stdout: "get_or_fail_fail": 0, 2024-03-16T12:08:42.123 INFO:teuthology.orchestra.run.smithi154.stdout: "get_or_fail_success": 0, 2024-03-16T12:08:42.123 INFO:teuthology.orchestra.run.smithi154.stdout: "take": 1513, 2024-03-16T12:08:42.123 INFO:teuthology.orchestra.run.smithi154.stdout: "take_sum": 333448423, 2024-03-16T12:08:42.123 INFO:teuthology.orchestra.run.smithi154.stdout: "put": 1513, 2024-03-16T12:08:42.123 INFO:teuthology.orchestra.run.smithi154.stdout: "put_sum": 333448423, 2024-03-16T12:08:42.123 INFO:teuthology.orchestra.run.smithi154.stdout: "wait": { 2024-03-16T12:08:42.123 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-16T12:08:42.124 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-16T12:08:42.124 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-16T12:08:42.124 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-03-16T12:08:42.124 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-16T12:08:42.124 INFO:teuthology.orchestra.run.smithi154.stdout: "throttle-objecter_ops": { 2024-03-16T12:08:42.124 INFO:teuthology.orchestra.run.smithi154.stdout: "val": 0, 2024-03-16T12:08:42.124 INFO:teuthology.orchestra.run.smithi154.stdout: "max": 1024, 2024-03-16T12:08:42.124 INFO:teuthology.orchestra.run.smithi154.stdout: "get_started": 0, 2024-03-16T12:08:42.124 INFO:teuthology.orchestra.run.smithi154.stdout: "get": 0, 2024-03-16T12:08:42.125 INFO:teuthology.orchestra.run.smithi154.stdout: "get_sum": 0, 2024-03-16T12:08:42.125 INFO:teuthology.orchestra.run.smithi154.stdout: "get_or_fail_fail": 0, 2024-03-16T12:08:42.125 INFO:teuthology.orchestra.run.smithi154.stdout: "get_or_fail_success": 0, 2024-03-16T12:08:42.125 INFO:teuthology.orchestra.run.smithi154.stdout: "take": 1513, 2024-03-16T12:08:42.125 INFO:teuthology.orchestra.run.smithi154.stdout: "take_sum": 1513, 2024-03-16T12:08:42.125 INFO:teuthology.orchestra.run.smithi154.stdout: "put": 1513, 2024-03-16T12:08:42.125 INFO:teuthology.orchestra.run.smithi154.stdout: "put_sum": 1513, 2024-03-16T12:08:42.125 INFO:teuthology.orchestra.run.smithi154.stdout: "wait": { 2024-03-16T12:08:42.125 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-16T12:08:42.126 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-16T12:08:42.126 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-16T12:08:42.126 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-03-16T12:08:42.126 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-16T12:08:42.126 INFO:teuthology.orchestra.run.smithi154.stdout: "throttle-write_buf_throttle": { 2024-03-16T12:08:42.126 INFO:teuthology.orchestra.run.smithi154.stdout: "val": 0, 2024-03-16T12:08:42.126 INFO:teuthology.orchestra.run.smithi154.stdout: "max": 3758096384, 2024-03-16T12:08:42.126 INFO:teuthology.orchestra.run.smithi154.stdout: "get_started": 0, 2024-03-16T12:08:42.126 INFO:teuthology.orchestra.run.smithi154.stdout: "get": 0, 2024-03-16T12:08:42.127 INFO:teuthology.orchestra.run.smithi154.stdout: "get_sum": 0, 2024-03-16T12:08:42.127 INFO:teuthology.orchestra.run.smithi154.stdout: "get_or_fail_fail": 0, 2024-03-16T12:08:42.127 INFO:teuthology.orchestra.run.smithi154.stdout: "get_or_fail_success": 0, 2024-03-16T12:08:42.127 INFO:teuthology.orchestra.run.smithi154.stdout: "take": 0, 2024-03-16T12:08:42.127 INFO:teuthology.orchestra.run.smithi154.stdout: "take_sum": 0, 2024-03-16T12:08:42.127 INFO:teuthology.orchestra.run.smithi154.stdout: "put": 0, 2024-03-16T12:08:42.127 INFO:teuthology.orchestra.run.smithi154.stdout: "put_sum": 0, 2024-03-16T12:08:42.127 INFO:teuthology.orchestra.run.smithi154.stdout: "wait": { 2024-03-16T12:08:42.127 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-16T12:08:42.128 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-16T12:08:42.128 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-16T12:08:42.128 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-03-16T12:08:42.128 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-16T12:08:42.128 INFO:teuthology.orchestra.run.smithi154.stdout: "throttle-write_buf_throttle-0x5589768a8640": { 2024-03-16T12:08:42.128 INFO:teuthology.orchestra.run.smithi154.stdout: "val": 0, 2024-03-16T12:08:42.128 INFO:teuthology.orchestra.run.smithi154.stdout: "max": 3758096384, 2024-03-16T12:08:42.128 INFO:teuthology.orchestra.run.smithi154.stdout: "get_started": 0, 2024-03-16T12:08:42.129 INFO:teuthology.orchestra.run.smithi154.stdout: "get": 0, 2024-03-16T12:08:42.129 INFO:teuthology.orchestra.run.smithi154.stdout: "get_sum": 0, 2024-03-16T12:08:42.129 INFO:teuthology.orchestra.run.smithi154.stdout: "get_or_fail_fail": 0, 2024-03-16T12:08:42.129 INFO:teuthology.orchestra.run.smithi154.stdout: "get_or_fail_success": 0, 2024-03-16T12:08:42.129 INFO:teuthology.orchestra.run.smithi154.stdout: "take": 0, 2024-03-16T12:08:42.129 INFO:teuthology.orchestra.run.smithi154.stdout: "take_sum": 0, 2024-03-16T12:08:42.129 INFO:teuthology.orchestra.run.smithi154.stdout: "put": 0, 2024-03-16T12:08:42.129 INFO:teuthology.orchestra.run.smithi154.stdout: "put_sum": 0, 2024-03-16T12:08:42.129 INFO:teuthology.orchestra.run.smithi154.stdout: "wait": { 2024-03-16T12:08:42.130 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-16T12:08:42.130 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-16T12:08:42.130 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-16T12:08:42.130 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-03-16T12:08:42.130 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-03-16T12:08:42.130 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-03-16T12:08:42.785 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:08:42 smithi154 ceph-mon[36320]: pgmap v564: 97 pgs: 97 active+clean; 358 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 160 KiB/s rd, 883 KiB/s wr, 47 op/s 2024-03-16T12:08:42.828 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:08:42 smithi029 ceph-mon[31975]: pgmap v564: 97 pgs: 97 active+clean; 358 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 160 KiB/s rd, 883 KiB/s wr, 47 op/s 2024-03-16T12:08:42.867 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:08:42 smithi152 ceph-mon[36291]: pgmap v564: 97 pgs: 97 active+clean; 358 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 160 KiB/s rd, 883 KiB/s wr, 47 op/s 2024-03-16T12:08:42.880 INFO:tasks.check_counter:Daemon mds.c mds.exported=0 2024-03-16T12:08:42.880 INFO:tasks.check_counter:Daemon mds.c mds.imported=0 2024-03-16T12:08:42.880 INFO:tasks.check_counter:Daemon mds.c mds_cache.dir_update=0 2024-03-16T12:08:42.880 INFO:tasks.check_counter:Daemon mds.c mds_cache.dir_update_receipt=0 2024-03-16T12:08:42.880 INFO:tasks.check_counter:Daemon mds.c mds.root_rsnaps=7 2024-03-16T12:08:42.880 INFO:tasks.check_counter:Daemon mds.c mds_server.req_mksnap_latency.avgcount=0 2024-03-16T12:08:42.880 INFO:tasks.check_counter:Daemon mds.c mds_server.req_rmsnap_latency.avgcount=0 2024-03-16T12:08:42.881 DEBUG:tasks.check_counter:Getting stats from f 2024-03-16T12:08:42.881 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid d1887f28-e389-11ee-95c9-87774f69a715 -- ceph daemon mds.f perf dump 2024-03-16T12:08:43.130 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/d1887f28-e389-11ee-95c9-87774f69a715/mon.c/config 2024-03-16T12:08:44.612 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:08:44 smithi154 ceph-mon[36320]: pgmap v565: 97 pgs: 97 active+clean; 362 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 213 KiB/s rd, 1.1 MiB/s wr, 52 op/s 2024-03-16T12:08:44.828 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:08:44 smithi029 ceph-mon[31975]: pgmap v565: 97 pgs: 97 active+clean; 362 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 213 KiB/s rd, 1.1 MiB/s wr, 52 op/s 2024-03-16T12:08:44.867 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:08:44 smithi152 ceph-mon[36291]: pgmap v565: 97 pgs: 97 active+clean; 362 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 213 KiB/s rd, 1.1 MiB/s wr, 52 op/s 2024-03-16T12:08:44.918 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-03-16T12:08:44.919 INFO:teuthology.orchestra.run.smithi154.stdout: "AsyncMessenger::Worker-0": { 2024-03-16T12:08:44.919 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_recv_messages": 3599, 2024-03-16T12:08:44.919 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_send_messages": 3600, 2024-03-16T12:08:44.919 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_recv_bytes": 12810058, 2024-03-16T12:08:44.919 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_send_bytes": 1144171, 2024-03-16T12:08:44.919 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_created_connections": 6, 2024-03-16T12:08:44.920 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_active_connections": 2, 2024-03-16T12:08:44.920 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_running_total_time": 0.212940943, 2024-03-16T12:08:44.920 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_running_send_time": 0.061109871, 2024-03-16T12:08:44.920 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_running_recv_time": 0.060291871, 2024-03-16T12:08:44.920 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_running_fast_dispatch_time": 0.077217635, 2024-03-16T12:08:44.920 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_send_messages_queue_lat": { 2024-03-16T12:08:44.920 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 3598, 2024-03-16T12:08:44.921 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.073742509, 2024-03-16T12:08:44.921 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000020495 2024-03-16T12:08:44.921 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-16T12:08:44.921 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_handle_ack_lat": { 2024-03-16T12:08:44.921 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-16T12:08:44.921 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-16T12:08:44.922 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-16T12:08:44.922 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-16T12:08:44.922 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-16T12:08:44.922 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_send_encrypted_bytes": 0 2024-03-16T12:08:44.922 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-16T12:08:44.922 INFO:teuthology.orchestra.run.smithi154.stdout: "AsyncMessenger::Worker-1": { 2024-03-16T12:08:44.922 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_recv_messages": 104, 2024-03-16T12:08:44.922 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_send_messages": 350, 2024-03-16T12:08:44.923 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_recv_bytes": 27478, 2024-03-16T12:08:44.923 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_send_bytes": 723404, 2024-03-16T12:08:44.923 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_created_connections": 8, 2024-03-16T12:08:44.923 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_active_connections": 5, 2024-03-16T12:08:44.923 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_running_total_time": 0.018784545, 2024-03-16T12:08:44.923 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_running_send_time": 0.008856017, 2024-03-16T12:08:44.923 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_running_recv_time": 0.004700727, 2024-03-16T12:08:44.924 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_running_fast_dispatch_time": 0.002751169, 2024-03-16T12:08:44.924 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_send_messages_queue_lat": { 2024-03-16T12:08:44.924 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 348, 2024-03-16T12:08:44.924 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.022543930, 2024-03-16T12:08:44.924 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000064781 2024-03-16T12:08:44.924 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-16T12:08:44.925 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_handle_ack_lat": { 2024-03-16T12:08:44.925 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-16T12:08:44.925 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-16T12:08:44.925 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-16T12:08:44.925 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-16T12:08:44.925 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_recv_encrypted_bytes": 3552, 2024-03-16T12:08:44.925 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_send_encrypted_bytes": 692496 2024-03-16T12:08:44.925 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-16T12:08:44.926 INFO:teuthology.orchestra.run.smithi154.stdout: "AsyncMessenger::Worker-2": { 2024-03-16T12:08:44.926 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_recv_messages": 388, 2024-03-16T12:08:44.926 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_send_messages": 364, 2024-03-16T12:08:44.926 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_recv_bytes": 1559372, 2024-03-16T12:08:44.926 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_send_bytes": 208766, 2024-03-16T12:08:44.926 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_created_connections": 4, 2024-03-16T12:08:44.927 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_active_connections": 2, 2024-03-16T12:08:44.927 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_running_total_time": 0.050478076, 2024-03-16T12:08:44.927 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_running_send_time": 0.013987858, 2024-03-16T12:08:44.927 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_running_recv_time": 0.041206626, 2024-03-16T12:08:44.927 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_running_fast_dispatch_time": 0.005695967, 2024-03-16T12:08:44.927 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_send_messages_queue_lat": { 2024-03-16T12:08:44.927 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 363, 2024-03-16T12:08:44.927 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.011234847, 2024-03-16T12:08:44.928 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000030949 2024-03-16T12:08:44.928 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-16T12:08:44.928 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_handle_ack_lat": { 2024-03-16T12:08:44.928 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-16T12:08:44.928 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-16T12:08:44.928 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-16T12:08:44.928 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-16T12:08:44.929 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_recv_encrypted_bytes": 1559104, 2024-03-16T12:08:44.929 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_send_encrypted_bytes": 208448 2024-03-16T12:08:44.929 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-16T12:08:44.929 INFO:teuthology.orchestra.run.smithi154.stdout: "cct": { 2024-03-16T12:08:44.929 INFO:teuthology.orchestra.run.smithi154.stdout: "total_workers": 1, 2024-03-16T12:08:44.929 INFO:teuthology.orchestra.run.smithi154.stdout: "unhealthy_workers": 0 2024-03-16T12:08:44.929 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-16T12:08:44.929 INFO:teuthology.orchestra.run.smithi154.stdout: "finisher-MDSRank": { 2024-03-16T12:08:44.929 INFO:teuthology.orchestra.run.smithi154.stdout: "queue_len": 0, 2024-03-16T12:08:44.930 INFO:teuthology.orchestra.run.smithi154.stdout: "complete_latency": { 2024-03-16T12:08:44.930 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 6233, 2024-03-16T12:08:44.930 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.121987573, 2024-03-16T12:08:44.930 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000019571 2024-03-16T12:08:44.930 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-03-16T12:08:44.930 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-16T12:08:44.930 INFO:teuthology.orchestra.run.smithi154.stdout: "finisher-PurgeQueue": { 2024-03-16T12:08:44.930 INFO:teuthology.orchestra.run.smithi154.stdout: "queue_len": 0, 2024-03-16T12:08:44.930 INFO:teuthology.orchestra.run.smithi154.stdout: "complete_latency": { 2024-03-16T12:08:44.931 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-16T12:08:44.931 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-16T12:08:44.931 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-16T12:08:44.931 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-03-16T12:08:44.931 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-16T12:08:44.931 INFO:teuthology.orchestra.run.smithi154.stdout: "mds": { 2024-03-16T12:08:44.931 INFO:teuthology.orchestra.run.smithi154.stdout: "request": 0, 2024-03-16T12:08:44.931 INFO:teuthology.orchestra.run.smithi154.stdout: "reply": 0, 2024-03-16T12:08:44.932 INFO:teuthology.orchestra.run.smithi154.stdout: "reply_latency": { 2024-03-16T12:08:44.932 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-16T12:08:44.932 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-16T12:08:44.932 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-16T12:08:44.932 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-16T12:08:44.932 INFO:teuthology.orchestra.run.smithi154.stdout: "slow_reply": 0, 2024-03-16T12:08:44.932 INFO:teuthology.orchestra.run.smithi154.stdout: "forward": 0, 2024-03-16T12:08:44.932 INFO:teuthology.orchestra.run.smithi154.stdout: "dir_fetch_complete": 0, 2024-03-16T12:08:44.932 INFO:teuthology.orchestra.run.smithi154.stdout: "dir_fetch_keys": 0, 2024-03-16T12:08:44.933 INFO:teuthology.orchestra.run.smithi154.stdout: "dir_commit": 0, 2024-03-16T12:08:44.933 INFO:teuthology.orchestra.run.smithi154.stdout: "dir_split": 0, 2024-03-16T12:08:44.933 INFO:teuthology.orchestra.run.smithi154.stdout: "dir_merge": 0, 2024-03-16T12:08:44.933 INFO:teuthology.orchestra.run.smithi154.stdout: "inodes": 1491, 2024-03-16T12:08:44.933 INFO:teuthology.orchestra.run.smithi154.stdout: "inodes_top": 0, 2024-03-16T12:08:44.933 INFO:teuthology.orchestra.run.smithi154.stdout: "inodes_bottom": 1491, 2024-03-16T12:08:44.933 INFO:teuthology.orchestra.run.smithi154.stdout: "inodes_pin_tail": 0, 2024-03-16T12:08:44.933 INFO:teuthology.orchestra.run.smithi154.stdout: "inodes_pinned": 1491, 2024-03-16T12:08:44.933 INFO:teuthology.orchestra.run.smithi154.stdout: "inodes_expired": 0, 2024-03-16T12:08:44.934 INFO:teuthology.orchestra.run.smithi154.stdout: "inodes_with_caps": 0, 2024-03-16T12:08:44.934 INFO:teuthology.orchestra.run.smithi154.stdout: "caps": 0, 2024-03-16T12:08:44.934 INFO:teuthology.orchestra.run.smithi154.stdout: "subtrees": 12, 2024-03-16T12:08:44.934 INFO:teuthology.orchestra.run.smithi154.stdout: "traverse": 0, 2024-03-16T12:08:44.934 INFO:teuthology.orchestra.run.smithi154.stdout: "traverse_hit": 0, 2024-03-16T12:08:44.934 INFO:teuthology.orchestra.run.smithi154.stdout: "traverse_forward": 0, 2024-03-16T12:08:44.934 INFO:teuthology.orchestra.run.smithi154.stdout: "traverse_discover": 0, 2024-03-16T12:08:44.934 INFO:teuthology.orchestra.run.smithi154.stdout: "traverse_dir_fetch": 0, 2024-03-16T12:08:44.934 INFO:teuthology.orchestra.run.smithi154.stdout: "traverse_remote_ino": 0, 2024-03-16T12:08:44.935 INFO:teuthology.orchestra.run.smithi154.stdout: "traverse_lock": 0, 2024-03-16T12:08:44.935 INFO:teuthology.orchestra.run.smithi154.stdout: "load_cent": 0, 2024-03-16T12:08:44.935 INFO:teuthology.orchestra.run.smithi154.stdout: "q": 0, 2024-03-16T12:08:44.935 INFO:teuthology.orchestra.run.smithi154.stdout: "exported": 0, 2024-03-16T12:08:44.935 INFO:teuthology.orchestra.run.smithi154.stdout: "exported_inodes": 0, 2024-03-16T12:08:44.935 INFO:teuthology.orchestra.run.smithi154.stdout: "imported": 0, 2024-03-16T12:08:44.935 INFO:teuthology.orchestra.run.smithi154.stdout: "imported_inodes": 0, 2024-03-16T12:08:44.935 INFO:teuthology.orchestra.run.smithi154.stdout: "openino_dir_fetch": 0, 2024-03-16T12:08:44.935 INFO:teuthology.orchestra.run.smithi154.stdout: "openino_backtrace_fetch": 0, 2024-03-16T12:08:44.936 INFO:teuthology.orchestra.run.smithi154.stdout: "openino_peer_discover": 0, 2024-03-16T12:08:44.936 INFO:teuthology.orchestra.run.smithi154.stdout: "root_rfiles": 0, 2024-03-16T12:08:44.936 INFO:teuthology.orchestra.run.smithi154.stdout: "root_rbytes": 0, 2024-03-16T12:08:44.936 INFO:teuthology.orchestra.run.smithi154.stdout: "root_rsnaps": 0, 2024-03-16T12:08:44.936 INFO:teuthology.orchestra.run.smithi154.stdout: "scrub_backtrace_fetch": 0, 2024-03-16T12:08:44.936 INFO:teuthology.orchestra.run.smithi154.stdout: "scrub_set_tag": 0, 2024-03-16T12:08:44.936 INFO:teuthology.orchestra.run.smithi154.stdout: "scrub_backtrace_repaired": 0, 2024-03-16T12:08:44.936 INFO:teuthology.orchestra.run.smithi154.stdout: "scrub_inotable_repaired": 0, 2024-03-16T12:08:44.936 INFO:teuthology.orchestra.run.smithi154.stdout: "scrub_dir_inodes": 0, 2024-03-16T12:08:44.937 INFO:teuthology.orchestra.run.smithi154.stdout: "scrub_dir_base_inodes": 0, 2024-03-16T12:08:44.937 INFO:teuthology.orchestra.run.smithi154.stdout: "scrub_dirfrag_rstats": 0, 2024-03-16T12:08:44.937 INFO:teuthology.orchestra.run.smithi154.stdout: "scrub_file_inodes": 0, 2024-03-16T12:08:44.937 INFO:teuthology.orchestra.run.smithi154.stdout: "handle_inode_file_caps": 0, 2024-03-16T12:08:44.937 INFO:teuthology.orchestra.run.smithi154.stdout: "ceph_cap_op_revoke": 0, 2024-03-16T12:08:44.937 INFO:teuthology.orchestra.run.smithi154.stdout: "ceph_cap_op_grant": 0, 2024-03-16T12:08:44.937 INFO:teuthology.orchestra.run.smithi154.stdout: "ceph_cap_op_trunc": 0, 2024-03-16T12:08:44.937 INFO:teuthology.orchestra.run.smithi154.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-16T12:08:44.937 INFO:teuthology.orchestra.run.smithi154.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-16T12:08:44.938 INFO:teuthology.orchestra.run.smithi154.stdout: "handle_client_caps": 0, 2024-03-16T12:08:44.938 INFO:teuthology.orchestra.run.smithi154.stdout: "handle_client_caps_dirty": 0, 2024-03-16T12:08:44.938 INFO:teuthology.orchestra.run.smithi154.stdout: "handle_client_cap_release": 0, 2024-03-16T12:08:44.938 INFO:teuthology.orchestra.run.smithi154.stdout: "process_request_cap_release": 0 2024-03-16T12:08:44.938 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-16T12:08:44.938 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_cache": { 2024-03-16T12:08:44.938 INFO:teuthology.orchestra.run.smithi154.stdout: "dir_update": 0, 2024-03-16T12:08:44.938 INFO:teuthology.orchestra.run.smithi154.stdout: "dir_update_receipt": 0, 2024-03-16T12:08:44.939 INFO:teuthology.orchestra.run.smithi154.stdout: "dir_try_discover": 0, 2024-03-16T12:08:44.939 INFO:teuthology.orchestra.run.smithi154.stdout: "dir_send_discover": 0, 2024-03-16T12:08:44.939 INFO:teuthology.orchestra.run.smithi154.stdout: "dir_handle_discover": 0, 2024-03-16T12:08:44.939 INFO:teuthology.orchestra.run.smithi154.stdout: "num_strays": 0, 2024-03-16T12:08:44.939 INFO:teuthology.orchestra.run.smithi154.stdout: "num_strays_delayed": 0, 2024-03-16T12:08:44.939 INFO:teuthology.orchestra.run.smithi154.stdout: "num_strays_enqueuing": 0, 2024-03-16T12:08:44.939 INFO:teuthology.orchestra.run.smithi154.stdout: "strays_created": 0, 2024-03-16T12:08:44.939 INFO:teuthology.orchestra.run.smithi154.stdout: "strays_enqueued": 0, 2024-03-16T12:08:44.940 INFO:teuthology.orchestra.run.smithi154.stdout: "strays_reintegrated": 0, 2024-03-16T12:08:44.940 INFO:teuthology.orchestra.run.smithi154.stdout: "strays_migrated": 0, 2024-03-16T12:08:44.940 INFO:teuthology.orchestra.run.smithi154.stdout: "num_recovering_processing": 0, 2024-03-16T12:08:44.940 INFO:teuthology.orchestra.run.smithi154.stdout: "num_recovering_enqueued": 0, 2024-03-16T12:08:44.940 INFO:teuthology.orchestra.run.smithi154.stdout: "num_recovering_prioritized": 0, 2024-03-16T12:08:44.940 INFO:teuthology.orchestra.run.smithi154.stdout: "recovery_started": 0, 2024-03-16T12:08:44.940 INFO:teuthology.orchestra.run.smithi154.stdout: "recovery_completed": 0, 2024-03-16T12:08:44.940 INFO:teuthology.orchestra.run.smithi154.stdout: "ireq_quiesce_path": 0, 2024-03-16T12:08:44.940 INFO:teuthology.orchestra.run.smithi154.stdout: "ireq_quiesce_inode": 0, 2024-03-16T12:08:44.941 INFO:teuthology.orchestra.run.smithi154.stdout: "ireq_enqueue_scrub": 0, 2024-03-16T12:08:44.941 INFO:teuthology.orchestra.run.smithi154.stdout: "ireq_exportdir": 0, 2024-03-16T12:08:44.941 INFO:teuthology.orchestra.run.smithi154.stdout: "ireq_flush": 0, 2024-03-16T12:08:44.941 INFO:teuthology.orchestra.run.smithi154.stdout: "ireq_fragmentdir": 0, 2024-03-16T12:08:44.941 INFO:teuthology.orchestra.run.smithi154.stdout: "ireq_fragstats": 0, 2024-03-16T12:08:44.941 INFO:teuthology.orchestra.run.smithi154.stdout: "ireq_inodestats": 0 2024-03-16T12:08:44.941 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-16T12:08:44.941 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_log": { 2024-03-16T12:08:44.941 INFO:teuthology.orchestra.run.smithi154.stdout: "evlrg": 0, 2024-03-16T12:08:44.942 INFO:teuthology.orchestra.run.smithi154.stdout: "evadd": 0, 2024-03-16T12:08:44.942 INFO:teuthology.orchestra.run.smithi154.stdout: "evex": 0, 2024-03-16T12:08:44.942 INFO:teuthology.orchestra.run.smithi154.stdout: "evtrm": 0, 2024-03-16T12:08:44.942 INFO:teuthology.orchestra.run.smithi154.stdout: "ev": 7856, 2024-03-16T12:08:44.942 INFO:teuthology.orchestra.run.smithi154.stdout: "evexg": 0, 2024-03-16T12:08:44.942 INFO:teuthology.orchestra.run.smithi154.stdout: "evexd": 0, 2024-03-16T12:08:44.942 INFO:teuthology.orchestra.run.smithi154.stdout: "segadd": 0, 2024-03-16T12:08:44.942 INFO:teuthology.orchestra.run.smithi154.stdout: "segex": 0, 2024-03-16T12:08:44.942 INFO:teuthology.orchestra.run.smithi154.stdout: "segtrm": 0, 2024-03-16T12:08:44.943 INFO:teuthology.orchestra.run.smithi154.stdout: "seg": 10, 2024-03-16T12:08:44.943 INFO:teuthology.orchestra.run.smithi154.stdout: "segmjr": 2, 2024-03-16T12:08:44.943 INFO:teuthology.orchestra.run.smithi154.stdout: "segexg": 0, 2024-03-16T12:08:44.943 INFO:teuthology.orchestra.run.smithi154.stdout: "segexd": 0, 2024-03-16T12:08:44.943 INFO:teuthology.orchestra.run.smithi154.stdout: "expos": 4194304, 2024-03-16T12:08:44.943 INFO:teuthology.orchestra.run.smithi154.stdout: "wrpos": 15989806, 2024-03-16T12:08:44.943 INFO:teuthology.orchestra.run.smithi154.stdout: "rdpos": 15989207, 2024-03-16T12:08:44.943 INFO:teuthology.orchestra.run.smithi154.stdout: "jlat": { 2024-03-16T12:08:44.944 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-16T12:08:44.944 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-16T12:08:44.944 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-16T12:08:44.944 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-16T12:08:44.944 INFO:teuthology.orchestra.run.smithi154.stdout: "replayed": 7856 2024-03-16T12:08:44.944 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-16T12:08:44.944 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_mem": { 2024-03-16T12:08:44.944 INFO:teuthology.orchestra.run.smithi154.stdout: "ino": 253, 2024-03-16T12:08:44.945 INFO:teuthology.orchestra.run.smithi154.stdout: "ino+": 1447, 2024-03-16T12:08:44.945 INFO:teuthology.orchestra.run.smithi154.stdout: "ino-": 1194, 2024-03-16T12:08:44.945 INFO:teuthology.orchestra.run.smithi154.stdout: "dir": 33, 2024-03-16T12:08:44.945 INFO:teuthology.orchestra.run.smithi154.stdout: "dir+": 34, 2024-03-16T12:08:44.945 INFO:teuthology.orchestra.run.smithi154.stdout: "dir-": 1, 2024-03-16T12:08:44.945 INFO:teuthology.orchestra.run.smithi154.stdout: "dn": 1491, 2024-03-16T12:08:44.945 INFO:teuthology.orchestra.run.smithi154.stdout: "dn+": 1491, 2024-03-16T12:08:44.946 INFO:teuthology.orchestra.run.smithi154.stdout: "dn-": 0, 2024-03-16T12:08:44.946 INFO:teuthology.orchestra.run.smithi154.stdout: "cap": 0, 2024-03-16T12:08:44.946 INFO:teuthology.orchestra.run.smithi154.stdout: "cap+": 0, 2024-03-16T12:08:44.946 INFO:teuthology.orchestra.run.smithi154.stdout: "cap-": 0, 2024-03-16T12:08:44.946 INFO:teuthology.orchestra.run.smithi154.stdout: "rss": 53836, 2024-03-16T12:08:44.946 INFO:teuthology.orchestra.run.smithi154.stdout: "heap": 215420 2024-03-16T12:08:44.947 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-16T12:08:44.947 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_server": { 2024-03-16T12:08:44.947 INFO:teuthology.orchestra.run.smithi154.stdout: "dispatch_client_request": 0, 2024-03-16T12:08:44.947 INFO:teuthology.orchestra.run.smithi154.stdout: "dispatch_server_request": 0, 2024-03-16T12:08:44.947 INFO:teuthology.orchestra.run.smithi154.stdout: "handle_client_request": 0, 2024-03-16T12:08:44.947 INFO:teuthology.orchestra.run.smithi154.stdout: "handle_client_session": 0, 2024-03-16T12:08:44.947 INFO:teuthology.orchestra.run.smithi154.stdout: "handle_peer_request": 0, 2024-03-16T12:08:44.948 INFO:teuthology.orchestra.run.smithi154.stdout: "req_create_latency": { 2024-03-16T12:08:44.948 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-16T12:08:44.948 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-16T12:08:44.948 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-16T12:08:44.948 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-16T12:08:44.948 INFO:teuthology.orchestra.run.smithi154.stdout: "req_getattr_latency": { 2024-03-16T12:08:44.948 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-16T12:08:44.949 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-16T12:08:44.949 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-16T12:08:44.949 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-16T12:08:44.949 INFO:teuthology.orchestra.run.smithi154.stdout: "req_getfilelock_latency": { 2024-03-16T12:08:44.949 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-16T12:08:44.949 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-16T12:08:44.950 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-16T12:08:44.950 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-16T12:08:44.950 INFO:teuthology.orchestra.run.smithi154.stdout: "req_link_latency": { 2024-03-16T12:08:44.950 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-16T12:08:44.950 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-16T12:08:44.950 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-16T12:08:44.950 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-16T12:08:44.951 INFO:teuthology.orchestra.run.smithi154.stdout: "req_lookup_latency": { 2024-03-16T12:08:44.951 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-16T12:08:44.951 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-16T12:08:44.951 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-16T12:08:44.951 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-16T12:08:44.951 INFO:teuthology.orchestra.run.smithi154.stdout: "req_lookuphash_latency": { 2024-03-16T12:08:44.951 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-16T12:08:44.952 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-16T12:08:44.952 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-16T12:08:44.952 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-16T12:08:44.952 INFO:teuthology.orchestra.run.smithi154.stdout: "req_lookupino_latency": { 2024-03-16T12:08:44.952 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-16T12:08:44.952 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-16T12:08:44.952 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-16T12:08:44.953 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-16T12:08:44.953 INFO:teuthology.orchestra.run.smithi154.stdout: "req_lookupname_latency": { 2024-03-16T12:08:44.953 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-16T12:08:44.953 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-16T12:08:44.953 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-16T12:08:44.953 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-16T12:08:44.953 INFO:teuthology.orchestra.run.smithi154.stdout: "req_lookupparent_latency": { 2024-03-16T12:08:44.954 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-16T12:08:44.954 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-16T12:08:44.954 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-16T12:08:44.954 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-16T12:08:44.954 INFO:teuthology.orchestra.run.smithi154.stdout: "req_lookupsnap_latency": { 2024-03-16T12:08:44.954 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-16T12:08:44.955 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-16T12:08:44.955 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-16T12:08:44.955 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-16T12:08:44.955 INFO:teuthology.orchestra.run.smithi154.stdout: "req_lssnap_latency": { 2024-03-16T12:08:44.955 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-16T12:08:44.955 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-16T12:08:44.955 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-16T12:08:44.956 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-16T12:08:44.956 INFO:teuthology.orchestra.run.smithi154.stdout: "req_mkdir_latency": { 2024-03-16T12:08:44.957 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-16T12:08:44.957 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-16T12:08:44.957 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-16T12:08:44.957 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-16T12:08:44.957 INFO:teuthology.orchestra.run.smithi154.stdout: "req_mknod_latency": { 2024-03-16T12:08:44.957 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-16T12:08:44.958 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-16T12:08:44.958 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-16T12:08:44.958 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-16T12:08:44.958 INFO:teuthology.orchestra.run.smithi154.stdout: "req_mksnap_latency": { 2024-03-16T12:08:44.958 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-16T12:08:44.958 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-16T12:08:44.959 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-16T12:08:44.959 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-16T12:08:44.959 INFO:teuthology.orchestra.run.smithi154.stdout: "req_open_latency": { 2024-03-16T12:08:44.959 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-16T12:08:44.959 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-16T12:08:44.959 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-16T12:08:44.959 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-16T12:08:44.960 INFO:teuthology.orchestra.run.smithi154.stdout: "req_readdir_latency": { 2024-03-16T12:08:44.960 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-16T12:08:44.960 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-16T12:08:44.960 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-16T12:08:44.960 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-16T12:08:44.960 INFO:teuthology.orchestra.run.smithi154.stdout: "req_rename_latency": { 2024-03-16T12:08:44.961 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-16T12:08:44.961 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-16T12:08:44.961 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-16T12:08:44.961 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-16T12:08:44.961 INFO:teuthology.orchestra.run.smithi154.stdout: "req_renamesnap_latency": { 2024-03-16T12:08:44.961 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-16T12:08:44.962 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-16T12:08:44.962 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-16T12:08:44.962 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-16T12:08:44.962 INFO:teuthology.orchestra.run.smithi154.stdout: "req_snapdiff_latency": { 2024-03-16T12:08:44.962 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-16T12:08:44.962 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-16T12:08:44.962 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-16T12:08:44.963 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-16T12:08:44.963 INFO:teuthology.orchestra.run.smithi154.stdout: "req_rmdir_latency": { 2024-03-16T12:08:44.963 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-16T12:08:44.963 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-16T12:08:44.963 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-16T12:08:44.963 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-16T12:08:44.964 INFO:teuthology.orchestra.run.smithi154.stdout: "req_rmsnap_latency": { 2024-03-16T12:08:44.964 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-16T12:08:44.964 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-16T12:08:44.964 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-16T12:08:44.964 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-16T12:08:44.964 INFO:teuthology.orchestra.run.smithi154.stdout: "req_rmxattr_latency": { 2024-03-16T12:08:44.964 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-16T12:08:44.965 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-16T12:08:44.965 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-16T12:08:44.965 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-16T12:08:44.965 INFO:teuthology.orchestra.run.smithi154.stdout: "req_setattr_latency": { 2024-03-16T12:08:44.965 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-16T12:08:44.965 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-16T12:08:44.965 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-16T12:08:44.965 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-16T12:08:44.966 INFO:teuthology.orchestra.run.smithi154.stdout: "req_setdirlayout_latency": { 2024-03-16T12:08:44.966 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-16T12:08:44.966 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-16T12:08:44.966 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-16T12:08:44.966 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-16T12:08:44.966 INFO:teuthology.orchestra.run.smithi154.stdout: "req_setfilelock_latency": { 2024-03-16T12:08:44.966 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-16T12:08:44.967 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-16T12:08:44.967 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-16T12:08:44.967 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-16T12:08:44.967 INFO:teuthology.orchestra.run.smithi154.stdout: "req_setlayout_latency": { 2024-03-16T12:08:44.967 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-16T12:08:44.967 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-16T12:08:44.967 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-16T12:08:44.968 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-16T12:08:44.968 INFO:teuthology.orchestra.run.smithi154.stdout: "req_setxattr_latency": { 2024-03-16T12:08:44.968 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-16T12:08:44.968 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-16T12:08:44.968 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-16T12:08:44.968 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-16T12:08:44.968 INFO:teuthology.orchestra.run.smithi154.stdout: "req_symlink_latency": { 2024-03-16T12:08:44.968 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-16T12:08:44.968 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-16T12:08:44.969 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-16T12:08:44.969 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-16T12:08:44.969 INFO:teuthology.orchestra.run.smithi154.stdout: "req_unlink_latency": { 2024-03-16T12:08:44.969 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-16T12:08:44.969 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-16T12:08:44.969 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-16T12:08:44.969 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-16T12:08:44.969 INFO:teuthology.orchestra.run.smithi154.stdout: "cap_revoke_eviction": 0, 2024-03-16T12:08:44.969 INFO:teuthology.orchestra.run.smithi154.stdout: "cap_acquisition_throttle": 0, 2024-03-16T12:08:44.970 INFO:teuthology.orchestra.run.smithi154.stdout: "req_getvxattr_latency": { 2024-03-16T12:08:44.970 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-16T12:08:44.970 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-16T12:08:44.970 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-16T12:08:44.970 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-03-16T12:08:44.970 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-16T12:08:44.970 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_sessions": { 2024-03-16T12:08:44.970 INFO:teuthology.orchestra.run.smithi154.stdout: "session_count": 2, 2024-03-16T12:08:44.971 INFO:teuthology.orchestra.run.smithi154.stdout: "session_add": 3, 2024-03-16T12:08:44.971 INFO:teuthology.orchestra.run.smithi154.stdout: "session_remove": 1, 2024-03-16T12:08:44.971 INFO:teuthology.orchestra.run.smithi154.stdout: "sessions_open": 2, 2024-03-16T12:08:44.971 INFO:teuthology.orchestra.run.smithi154.stdout: "sessions_stale": 0, 2024-03-16T12:08:44.971 INFO:teuthology.orchestra.run.smithi154.stdout: "total_load": 0, 2024-03-16T12:08:44.971 INFO:teuthology.orchestra.run.smithi154.stdout: "average_load": 0, 2024-03-16T12:08:44.971 INFO:teuthology.orchestra.run.smithi154.stdout: "avg_session_uptime": 2535, 2024-03-16T12:08:44.971 INFO:teuthology.orchestra.run.smithi154.stdout: "mdthresh_evicted": 0 2024-03-16T12:08:44.972 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-16T12:08:44.972 INFO:teuthology.orchestra.run.smithi154.stdout: "mempool": { 2024-03-16T12:08:44.972 INFO:teuthology.orchestra.run.smithi154.stdout: "bloom_filter_bytes": 0, 2024-03-16T12:08:44.972 INFO:teuthology.orchestra.run.smithi154.stdout: "bloom_filter_items": 0, 2024-03-16T12:08:44.972 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_alloc_bytes": 0, 2024-03-16T12:08:44.972 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_alloc_items": 0, 2024-03-16T12:08:44.972 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_cache_data_bytes": 0, 2024-03-16T12:08:44.972 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_cache_data_items": 0, 2024-03-16T12:08:44.973 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-16T12:08:44.973 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_cache_onode_items": 0, 2024-03-16T12:08:44.973 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-16T12:08:44.973 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_cache_meta_items": 0, 2024-03-16T12:08:44.973 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_cache_other_bytes": 0, 2024-03-16T12:08:44.973 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_cache_other_items": 0, 2024-03-16T12:08:44.973 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-16T12:08:44.973 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_cache_buffer_items": 0, 2024-03-16T12:08:44.973 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_extent_bytes": 0, 2024-03-16T12:08:44.974 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_extent_items": 0, 2024-03-16T12:08:44.974 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_blob_bytes": 0, 2024-03-16T12:08:44.974 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_blob_items": 0, 2024-03-16T12:08:44.974 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-16T12:08:44.974 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_shared_blob_items": 0, 2024-03-16T12:08:44.974 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-16T12:08:44.974 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_inline_bl_items": 0, 2024-03-16T12:08:44.974 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_fsck_bytes": 0, 2024-03-16T12:08:44.975 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_fsck_items": 0, 2024-03-16T12:08:44.975 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_txc_bytes": 0, 2024-03-16T12:08:44.975 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_txc_items": 0, 2024-03-16T12:08:44.975 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-16T12:08:44.975 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_writing_deferred_items": 0, 2024-03-16T12:08:44.975 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_writing_bytes": 0, 2024-03-16T12:08:44.975 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_writing_items": 0, 2024-03-16T12:08:44.975 INFO:teuthology.orchestra.run.smithi154.stdout: "bluefs_bytes": 0, 2024-03-16T12:08:44.976 INFO:teuthology.orchestra.run.smithi154.stdout: "bluefs_items": 0, 2024-03-16T12:08:44.976 INFO:teuthology.orchestra.run.smithi154.stdout: "bluefs_file_reader_bytes": 0, 2024-03-16T12:08:44.976 INFO:teuthology.orchestra.run.smithi154.stdout: "bluefs_file_reader_items": 0, 2024-03-16T12:08:44.976 INFO:teuthology.orchestra.run.smithi154.stdout: "bluefs_file_writer_bytes": 0, 2024-03-16T12:08:44.976 INFO:teuthology.orchestra.run.smithi154.stdout: "bluefs_file_writer_items": 0, 2024-03-16T12:08:44.976 INFO:teuthology.orchestra.run.smithi154.stdout: "buffer_anon_bytes": 199057, 2024-03-16T12:08:44.976 INFO:teuthology.orchestra.run.smithi154.stdout: "buffer_anon_items": 293, 2024-03-16T12:08:44.976 INFO:teuthology.orchestra.run.smithi154.stdout: "buffer_meta_bytes": 0, 2024-03-16T12:08:44.976 INFO:teuthology.orchestra.run.smithi154.stdout: "buffer_meta_items": 0, 2024-03-16T12:08:44.977 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_bytes": 0, 2024-03-16T12:08:44.977 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_items": 0, 2024-03-16T12:08:44.977 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_mapbl_bytes": 0, 2024-03-16T12:08:44.977 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_mapbl_items": 0, 2024-03-16T12:08:44.977 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_pglog_bytes": 0, 2024-03-16T12:08:44.977 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_pglog_items": 0, 2024-03-16T12:08:44.977 INFO:teuthology.orchestra.run.smithi154.stdout: "osdmap_bytes": 8256, 2024-03-16T12:08:44.977 INFO:teuthology.orchestra.run.smithi154.stdout: "osdmap_items": 188, 2024-03-16T12:08:44.977 INFO:teuthology.orchestra.run.smithi154.stdout: "osdmap_mapping_bytes": 0, 2024-03-16T12:08:44.977 INFO:teuthology.orchestra.run.smithi154.stdout: "osdmap_mapping_items": 0, 2024-03-16T12:08:44.978 INFO:teuthology.orchestra.run.smithi154.stdout: "pgmap_bytes": 0, 2024-03-16T12:08:44.978 INFO:teuthology.orchestra.run.smithi154.stdout: "pgmap_items": 0, 2024-03-16T12:08:44.978 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_co_bytes": 1693949, 2024-03-16T12:08:44.978 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_co_items": 11689, 2024-03-16T12:08:44.978 INFO:teuthology.orchestra.run.smithi154.stdout: "unittest_1_bytes": 0, 2024-03-16T12:08:44.978 INFO:teuthology.orchestra.run.smithi154.stdout: "unittest_1_items": 0, 2024-03-16T12:08:44.978 INFO:teuthology.orchestra.run.smithi154.stdout: "unittest_2_bytes": 0, 2024-03-16T12:08:44.978 INFO:teuthology.orchestra.run.smithi154.stdout: "unittest_2_items": 0 2024-03-16T12:08:44.979 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-16T12:08:44.979 INFO:teuthology.orchestra.run.smithi154.stdout: "objecter": { 2024-03-16T12:08:44.979 INFO:teuthology.orchestra.run.smithi154.stdout: "op_active": 0, 2024-03-16T12:08:44.979 INFO:teuthology.orchestra.run.smithi154.stdout: "op_laggy": 0, 2024-03-16T12:08:44.979 INFO:teuthology.orchestra.run.smithi154.stdout: "op_send": 3696, 2024-03-16T12:08:44.979 INFO:teuthology.orchestra.run.smithi154.stdout: "op_send_bytes": 21, 2024-03-16T12:08:44.979 INFO:teuthology.orchestra.run.smithi154.stdout: "op_resend": 3, 2024-03-16T12:08:44.979 INFO:teuthology.orchestra.run.smithi154.stdout: "op_reply": 3696, 2024-03-16T12:08:44.980 INFO:teuthology.orchestra.run.smithi154.stdout: "op_latency": { 2024-03-16T12:08:44.980 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 3696, 2024-03-16T12:08:44.980 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 2.914926858, 2024-03-16T12:08:44.980 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000788670 2024-03-16T12:08:44.980 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-16T12:08:44.980 INFO:teuthology.orchestra.run.smithi154.stdout: "op_inflight": 0, 2024-03-16T12:08:44.980 INFO:teuthology.orchestra.run.smithi154.stdout: "oplen_avg": { 2024-03-16T12:08:44.981 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 3696, 2024-03-16T12:08:44.981 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 3697 2024-03-16T12:08:44.981 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-16T12:08:44.981 INFO:teuthology.orchestra.run.smithi154.stdout: "op": 3696, 2024-03-16T12:08:44.981 INFO:teuthology.orchestra.run.smithi154.stdout: "op_r": 3696, 2024-03-16T12:08:44.981 INFO:teuthology.orchestra.run.smithi154.stdout: "op_w": 0, 2024-03-16T12:08:44.981 INFO:teuthology.orchestra.run.smithi154.stdout: "op_rmw": 0, 2024-03-16T12:08:44.982 INFO:teuthology.orchestra.run.smithi154.stdout: "op_pg": 0, 2024-03-16T12:08:44.982 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_stat": 2325, 2024-03-16T12:08:44.982 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_create": 0, 2024-03-16T12:08:44.982 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_read": 1369, 2024-03-16T12:08:44.982 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_write": 0, 2024-03-16T12:08:44.982 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_writefull": 0, 2024-03-16T12:08:44.983 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_writesame": 0, 2024-03-16T12:08:44.983 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_append": 0, 2024-03-16T12:08:44.983 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_zero": 0, 2024-03-16T12:08:44.983 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_truncate": 0, 2024-03-16T12:08:44.983 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_delete": 0, 2024-03-16T12:08:44.983 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_mapext": 0, 2024-03-16T12:08:44.983 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_sparse_read": 0, 2024-03-16T12:08:44.984 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_clonerange": 0, 2024-03-16T12:08:44.984 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_getxattr": 1, 2024-03-16T12:08:44.984 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_setxattr": 0, 2024-03-16T12:08:44.984 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_cmpxattr": 0, 2024-03-16T12:08:44.984 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_rmxattr": 0, 2024-03-16T12:08:44.984 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_resetxattrs": 0, 2024-03-16T12:08:44.984 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_call": 0, 2024-03-16T12:08:44.985 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_watch": 0, 2024-03-16T12:08:44.985 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_notify": 0, 2024-03-16T12:08:44.985 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_src_cmpxattr": 0, 2024-03-16T12:08:44.985 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_pgls": 0, 2024-03-16T12:08:44.985 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_pgls_filter": 0, 2024-03-16T12:08:44.985 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_other": 0, 2024-03-16T12:08:44.985 INFO:teuthology.orchestra.run.smithi154.stdout: "linger_active": 0, 2024-03-16T12:08:44.986 INFO:teuthology.orchestra.run.smithi154.stdout: "linger_send": 0, 2024-03-16T12:08:44.986 INFO:teuthology.orchestra.run.smithi154.stdout: "linger_resend": 0, 2024-03-16T12:08:44.986 INFO:teuthology.orchestra.run.smithi154.stdout: "linger_ping": 0, 2024-03-16T12:08:44.986 INFO:teuthology.orchestra.run.smithi154.stdout: "poolop_active": 0, 2024-03-16T12:08:44.986 INFO:teuthology.orchestra.run.smithi154.stdout: "poolop_send": 0, 2024-03-16T12:08:44.986 INFO:teuthology.orchestra.run.smithi154.stdout: "poolop_resend": 0, 2024-03-16T12:08:44.986 INFO:teuthology.orchestra.run.smithi154.stdout: "poolstat_active": 0, 2024-03-16T12:08:44.987 INFO:teuthology.orchestra.run.smithi154.stdout: "poolstat_send": 0, 2024-03-16T12:08:44.987 INFO:teuthology.orchestra.run.smithi154.stdout: "poolstat_resend": 0, 2024-03-16T12:08:44.987 INFO:teuthology.orchestra.run.smithi154.stdout: "statfs_active": 0, 2024-03-16T12:08:44.987 INFO:teuthology.orchestra.run.smithi154.stdout: "statfs_send": 0, 2024-03-16T12:08:44.987 INFO:teuthology.orchestra.run.smithi154.stdout: "statfs_resend": 0, 2024-03-16T12:08:44.987 INFO:teuthology.orchestra.run.smithi154.stdout: "command_active": 0, 2024-03-16T12:08:44.987 INFO:teuthology.orchestra.run.smithi154.stdout: "command_send": 0, 2024-03-16T12:08:44.988 INFO:teuthology.orchestra.run.smithi154.stdout: "command_resend": 0, 2024-03-16T12:08:44.988 INFO:teuthology.orchestra.run.smithi154.stdout: "map_epoch": 116, 2024-03-16T12:08:44.988 INFO:teuthology.orchestra.run.smithi154.stdout: "map_full": 0, 2024-03-16T12:08:44.988 INFO:teuthology.orchestra.run.smithi154.stdout: "map_inc": 31, 2024-03-16T12:08:44.988 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_sessions": 5, 2024-03-16T12:08:44.988 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_session_open": 6, 2024-03-16T12:08:44.988 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_session_close": 1, 2024-03-16T12:08:44.989 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_laggy": 0, 2024-03-16T12:08:44.989 INFO:teuthology.orchestra.run.smithi154.stdout: "omap_wr": 0, 2024-03-16T12:08:44.989 INFO:teuthology.orchestra.run.smithi154.stdout: "omap_rd": 2, 2024-03-16T12:08:44.989 INFO:teuthology.orchestra.run.smithi154.stdout: "omap_del": 0 2024-03-16T12:08:44.989 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-16T12:08:44.989 INFO:teuthology.orchestra.run.smithi154.stdout: "oft": { 2024-03-16T12:08:44.989 INFO:teuthology.orchestra.run.smithi154.stdout: "omap_total_objs": 0, 2024-03-16T12:08:44.990 INFO:teuthology.orchestra.run.smithi154.stdout: "omap_total_kv_pairs": 0, 2024-03-16T12:08:44.990 INFO:teuthology.orchestra.run.smithi154.stdout: "omap_total_updates": 0, 2024-03-16T12:08:44.990 INFO:teuthology.orchestra.run.smithi154.stdout: "omap_total_removes": 0 2024-03-16T12:08:44.990 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-16T12:08:44.990 INFO:teuthology.orchestra.run.smithi154.stdout: "osd-slow-ops": { 2024-03-16T12:08:44.990 INFO:teuthology.orchestra.run.smithi154.stdout: "slow_ops_count": 0 2024-03-16T12:08:44.990 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-16T12:08:44.991 INFO:teuthology.orchestra.run.smithi154.stdout: "purge_queue": { 2024-03-16T12:08:44.991 INFO:teuthology.orchestra.run.smithi154.stdout: "pq_executing_ops": 0, 2024-03-16T12:08:44.991 INFO:teuthology.orchestra.run.smithi154.stdout: "pq_executing_ops_high_water": 0, 2024-03-16T12:08:44.991 INFO:teuthology.orchestra.run.smithi154.stdout: "pq_executing": 0, 2024-03-16T12:08:44.991 INFO:teuthology.orchestra.run.smithi154.stdout: "pq_executing_high_water": 0, 2024-03-16T12:08:44.991 INFO:teuthology.orchestra.run.smithi154.stdout: "pq_executed_ops": 0, 2024-03-16T12:08:44.991 INFO:teuthology.orchestra.run.smithi154.stdout: "pq_executed": 0, 2024-03-16T12:08:44.992 INFO:teuthology.orchestra.run.smithi154.stdout: "pq_item_in_journal": 0 2024-03-16T12:08:44.992 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-16T12:08:44.992 INFO:teuthology.orchestra.run.smithi154.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-16T12:08:44.992 INFO:teuthology.orchestra.run.smithi154.stdout: "val": 0, 2024-03-16T12:08:44.992 INFO:teuthology.orchestra.run.smithi154.stdout: "max": 104857600, 2024-03-16T12:08:44.992 INFO:teuthology.orchestra.run.smithi154.stdout: "get_started": 0, 2024-03-16T12:08:44.992 INFO:teuthology.orchestra.run.smithi154.stdout: "get": 4087, 2024-03-16T12:08:44.992 INFO:teuthology.orchestra.run.smithi154.stdout: "get_sum": 14006265, 2024-03-16T12:08:44.993 INFO:teuthology.orchestra.run.smithi154.stdout: "get_or_fail_fail": 0, 2024-03-16T12:08:44.993 INFO:teuthology.orchestra.run.smithi154.stdout: "get_or_fail_success": 4087, 2024-03-16T12:08:44.993 INFO:teuthology.orchestra.run.smithi154.stdout: "take": 0, 2024-03-16T12:08:44.993 INFO:teuthology.orchestra.run.smithi154.stdout: "take_sum": 0, 2024-03-16T12:08:44.993 INFO:teuthology.orchestra.run.smithi154.stdout: "put": 4087, 2024-03-16T12:08:44.993 INFO:teuthology.orchestra.run.smithi154.stdout: "put_sum": 14006265, 2024-03-16T12:08:44.993 INFO:teuthology.orchestra.run.smithi154.stdout: "wait": { 2024-03-16T12:08:44.994 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-16T12:08:44.994 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-16T12:08:44.994 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-16T12:08:44.994 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-03-16T12:08:44.994 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-16T12:08:44.994 INFO:teuthology.orchestra.run.smithi154.stdout: "throttle-objecter_bytes": { 2024-03-16T12:08:44.994 INFO:teuthology.orchestra.run.smithi154.stdout: "val": 0, 2024-03-16T12:08:44.995 INFO:teuthology.orchestra.run.smithi154.stdout: "max": 104857600, 2024-03-16T12:08:44.995 INFO:teuthology.orchestra.run.smithi154.stdout: "get_started": 0, 2024-03-16T12:08:44.995 INFO:teuthology.orchestra.run.smithi154.stdout: "get": 0, 2024-03-16T12:08:44.995 INFO:teuthology.orchestra.run.smithi154.stdout: "get_sum": 0, 2024-03-16T12:08:44.995 INFO:teuthology.orchestra.run.smithi154.stdout: "get_or_fail_fail": 0, 2024-03-16T12:08:44.995 INFO:teuthology.orchestra.run.smithi154.stdout: "get_or_fail_success": 0, 2024-03-16T12:08:44.995 INFO:teuthology.orchestra.run.smithi154.stdout: "take": 3696, 2024-03-16T12:08:44.996 INFO:teuthology.orchestra.run.smithi154.stdout: "take_sum": 11796482, 2024-03-16T12:08:44.996 INFO:teuthology.orchestra.run.smithi154.stdout: "put": 3696, 2024-03-16T12:08:44.996 INFO:teuthology.orchestra.run.smithi154.stdout: "put_sum": 11796482, 2024-03-16T12:08:44.996 INFO:teuthology.orchestra.run.smithi154.stdout: "wait": { 2024-03-16T12:08:44.996 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-16T12:08:44.996 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-16T12:08:44.996 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-16T12:08:44.997 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-03-16T12:08:44.997 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-16T12:08:44.997 INFO:teuthology.orchestra.run.smithi154.stdout: "throttle-objecter_ops": { 2024-03-16T12:08:44.997 INFO:teuthology.orchestra.run.smithi154.stdout: "val": 0, 2024-03-16T12:08:44.997 INFO:teuthology.orchestra.run.smithi154.stdout: "max": 1024, 2024-03-16T12:08:44.997 INFO:teuthology.orchestra.run.smithi154.stdout: "get_started": 0, 2024-03-16T12:08:44.997 INFO:teuthology.orchestra.run.smithi154.stdout: "get": 0, 2024-03-16T12:08:44.998 INFO:teuthology.orchestra.run.smithi154.stdout: "get_sum": 0, 2024-03-16T12:08:44.998 INFO:teuthology.orchestra.run.smithi154.stdout: "get_or_fail_fail": 0, 2024-03-16T12:08:44.998 INFO:teuthology.orchestra.run.smithi154.stdout: "get_or_fail_success": 0, 2024-03-16T12:08:44.998 INFO:teuthology.orchestra.run.smithi154.stdout: "take": 3696, 2024-03-16T12:08:44.998 INFO:teuthology.orchestra.run.smithi154.stdout: "take_sum": 3696, 2024-03-16T12:08:44.998 INFO:teuthology.orchestra.run.smithi154.stdout: "put": 3696, 2024-03-16T12:08:44.998 INFO:teuthology.orchestra.run.smithi154.stdout: "put_sum": 3696, 2024-03-16T12:08:44.998 INFO:teuthology.orchestra.run.smithi154.stdout: "wait": { 2024-03-16T12:08:44.999 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-16T12:08:44.999 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-16T12:08:44.999 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-16T12:08:44.999 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-03-16T12:08:44.999 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-16T12:08:44.999 INFO:teuthology.orchestra.run.smithi154.stdout: "throttle-write_buf_throttle": { 2024-03-16T12:08:44.999 INFO:teuthology.orchestra.run.smithi154.stdout: "val": 0, 2024-03-16T12:08:45.000 INFO:teuthology.orchestra.run.smithi154.stdout: "max": 3758096384, 2024-03-16T12:08:45.000 INFO:teuthology.orchestra.run.smithi154.stdout: "get_started": 0, 2024-03-16T12:08:45.000 INFO:teuthology.orchestra.run.smithi154.stdout: "get": 0, 2024-03-16T12:08:45.000 INFO:teuthology.orchestra.run.smithi154.stdout: "get_sum": 0, 2024-03-16T12:08:45.000 INFO:teuthology.orchestra.run.smithi154.stdout: "get_or_fail_fail": 0, 2024-03-16T12:08:45.000 INFO:teuthology.orchestra.run.smithi154.stdout: "get_or_fail_success": 0, 2024-03-16T12:08:45.000 INFO:teuthology.orchestra.run.smithi154.stdout: "take": 0, 2024-03-16T12:08:45.001 INFO:teuthology.orchestra.run.smithi154.stdout: "take_sum": 0, 2024-03-16T12:08:45.001 INFO:teuthology.orchestra.run.smithi154.stdout: "put": 0, 2024-03-16T12:08:45.001 INFO:teuthology.orchestra.run.smithi154.stdout: "put_sum": 0, 2024-03-16T12:08:45.001 INFO:teuthology.orchestra.run.smithi154.stdout: "wait": { 2024-03-16T12:08:45.001 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-16T12:08:45.001 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-16T12:08:45.001 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-16T12:08:45.001 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-03-16T12:08:45.002 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-16T12:08:45.002 INFO:teuthology.orchestra.run.smithi154.stdout: "throttle-write_buf_throttle-0x55f3337b46e0": { 2024-03-16T12:08:45.002 INFO:teuthology.orchestra.run.smithi154.stdout: "val": 0, 2024-03-16T12:08:45.002 INFO:teuthology.orchestra.run.smithi154.stdout: "max": 3758096384, 2024-03-16T12:08:45.002 INFO:teuthology.orchestra.run.smithi154.stdout: "get_started": 0, 2024-03-16T12:08:45.002 INFO:teuthology.orchestra.run.smithi154.stdout: "get": 0, 2024-03-16T12:08:45.002 INFO:teuthology.orchestra.run.smithi154.stdout: "get_sum": 0, 2024-03-16T12:08:45.002 INFO:teuthology.orchestra.run.smithi154.stdout: "get_or_fail_fail": 0, 2024-03-16T12:08:45.002 INFO:teuthology.orchestra.run.smithi154.stdout: "get_or_fail_success": 0, 2024-03-16T12:08:45.003 INFO:teuthology.orchestra.run.smithi154.stdout: "take": 0, 2024-03-16T12:08:45.003 INFO:teuthology.orchestra.run.smithi154.stdout: "take_sum": 0, 2024-03-16T12:08:45.003 INFO:teuthology.orchestra.run.smithi154.stdout: "put": 0, 2024-03-16T12:08:45.003 INFO:teuthology.orchestra.run.smithi154.stdout: "put_sum": 0, 2024-03-16T12:08:45.003 INFO:teuthology.orchestra.run.smithi154.stdout: "wait": { 2024-03-16T12:08:45.003 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-16T12:08:45.003 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-16T12:08:45.003 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-16T12:08:45.003 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-03-16T12:08:45.004 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-03-16T12:08:45.004 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-03-16T12:08:45.757 INFO:tasks.check_counter:Daemon mds.f mds.exported=0 2024-03-16T12:08:45.757 INFO:tasks.check_counter:Daemon mds.f mds.imported=0 2024-03-16T12:08:45.757 INFO:tasks.check_counter:Daemon mds.f mds_cache.dir_update=0 2024-03-16T12:08:45.757 INFO:tasks.check_counter:Daemon mds.f mds_cache.dir_update_receipt=0 2024-03-16T12:08:45.757 INFO:tasks.check_counter:Daemon mds.f mds.root_rsnaps=0 2024-03-16T12:08:45.758 INFO:tasks.check_counter:Daemon mds.f mds_server.req_mksnap_latency.avgcount=0 2024-03-16T12:08:45.758 INFO:tasks.check_counter:Daemon mds.f mds_server.req_rmsnap_latency.avgcount=0 2024-03-16T12:08:45.758 DEBUG:tasks.check_counter:Getting stats from i 2024-03-16T12:08:45.758 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid d1887f28-e389-11ee-95c9-87774f69a715 -- ceph daemon mds.i perf dump 2024-03-16T12:08:46.011 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/d1887f28-e389-11ee-95c9-87774f69a715/mon.c/config 2024-03-16T12:08:46.617 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:08:46 smithi152 ceph-mon[36291]: pgmap v566: 97 pgs: 97 active+clean; 362 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s rd, 789 KiB/s wr, 31 op/s 2024-03-16T12:08:46.746 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:08:46 smithi154 ceph-mon[36320]: pgmap v566: 97 pgs: 97 active+clean; 362 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s rd, 789 KiB/s wr, 31 op/s 2024-03-16T12:08:46.830 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:08:46 smithi029 ceph-mon[31975]: pgmap v566: 97 pgs: 97 active+clean; 362 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s rd, 789 KiB/s wr, 31 op/s 2024-03-16T12:08:48.235 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-03-16T12:08:48.236 INFO:teuthology.orchestra.run.smithi154.stdout: "AsyncMessenger::Worker-0": { 2024-03-16T12:08:48.236 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_recv_messages": 55972, 2024-03-16T12:08:48.236 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_send_messages": 62277, 2024-03-16T12:08:48.236 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_recv_bytes": 14431123, 2024-03-16T12:08:48.236 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_send_bytes": 33351319, 2024-03-16T12:08:48.236 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_created_connections": 17, 2024-03-16T12:08:48.236 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_active_connections": 4, 2024-03-16T12:08:48.237 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_running_total_time": 2.601044687, 2024-03-16T12:08:48.237 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_running_send_time": 1.251360926, 2024-03-16T12:08:48.237 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_running_recv_time": 1.752526898, 2024-03-16T12:08:48.237 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_running_fast_dispatch_time": 0.064528071, 2024-03-16T12:08:48.237 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_send_messages_queue_lat": { 2024-03-16T12:08:48.237 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 62275, 2024-03-16T12:08:48.237 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 1.124040110, 2024-03-16T12:08:48.238 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000018049 2024-03-16T12:08:48.238 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-16T12:08:48.238 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_handle_ack_lat": { 2024-03-16T12:08:48.238 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 58896, 2024-03-16T12:08:48.238 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.018372669, 2024-03-16T12:08:48.238 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000311 2024-03-16T12:08:48.239 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-16T12:08:48.239 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_recv_encrypted_bytes": 928, 2024-03-16T12:08:48.239 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_send_encrypted_bytes": 736 2024-03-16T12:08:48.239 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-16T12:08:48.239 INFO:teuthology.orchestra.run.smithi154.stdout: "AsyncMessenger::Worker-1": { 2024-03-16T12:08:48.239 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_recv_messages": 380804, 2024-03-16T12:08:48.239 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_send_messages": 195801, 2024-03-16T12:08:48.240 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_recv_bytes": 166967379, 2024-03-16T12:08:48.240 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_send_bytes": 32982231, 2024-03-16T12:08:48.240 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_created_connections": 10, 2024-03-16T12:08:48.240 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_active_connections": 6, 2024-03-16T12:08:48.240 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_running_total_time": 13.904160440, 2024-03-16T12:08:48.240 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_running_send_time": 5.889092776, 2024-03-16T12:08:48.241 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_running_recv_time": 11.207788754, 2024-03-16T12:08:48.241 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_running_fast_dispatch_time": 0.025439996, 2024-03-16T12:08:48.241 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_send_messages_queue_lat": { 2024-03-16T12:08:48.241 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 195800, 2024-03-16T12:08:48.241 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 2.624121469, 2024-03-16T12:08:48.241 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000013402 2024-03-16T12:08:48.242 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-16T12:08:48.242 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_handle_ack_lat": { 2024-03-16T12:08:48.242 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 573333, 2024-03-16T12:08:48.242 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.058176948, 2024-03-16T12:08:48.242 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000101 2024-03-16T12:08:48.242 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-16T12:08:48.243 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_recv_encrypted_bytes": 1564816, 2024-03-16T12:08:48.243 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_send_encrypted_bytes": 211808 2024-03-16T12:08:48.243 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-16T12:08:48.243 INFO:teuthology.orchestra.run.smithi154.stdout: "AsyncMessenger::Worker-2": { 2024-03-16T12:08:48.243 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_recv_messages": 725, 2024-03-16T12:08:48.243 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_send_messages": 1028, 2024-03-16T12:08:48.243 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_recv_bytes": 304763, 2024-03-16T12:08:48.243 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_send_bytes": 1015096, 2024-03-16T12:08:48.244 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_created_connections": 12, 2024-03-16T12:08:48.244 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_active_connections": 6, 2024-03-16T12:08:48.244 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_running_total_time": 0.057381096, 2024-03-16T12:08:48.244 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_running_send_time": 0.019769377, 2024-03-16T12:08:48.244 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_running_recv_time": 0.015430909, 2024-03-16T12:08:48.244 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_running_fast_dispatch_time": 0.017770514, 2024-03-16T12:08:48.244 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_send_messages_queue_lat": { 2024-03-16T12:08:48.244 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 1026, 2024-03-16T12:08:48.245 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.047062887, 2024-03-16T12:08:48.245 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000045870 2024-03-16T12:08:48.245 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-16T12:08:48.245 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_handle_ack_lat": { 2024-03-16T12:08:48.245 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 90, 2024-03-16T12:08:48.245 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000017660, 2024-03-16T12:08:48.245 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000196 2024-03-16T12:08:48.245 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-16T12:08:48.245 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_recv_encrypted_bytes": 7328, 2024-03-16T12:08:48.246 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_send_encrypted_bytes": 626592 2024-03-16T12:08:48.246 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-16T12:08:48.246 INFO:teuthology.orchestra.run.smithi154.stdout: "cct": { 2024-03-16T12:08:48.246 INFO:teuthology.orchestra.run.smithi154.stdout: "total_workers": 1, 2024-03-16T12:08:48.246 INFO:teuthology.orchestra.run.smithi154.stdout: "unhealthy_workers": 0 2024-03-16T12:08:48.246 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-16T12:08:48.246 INFO:teuthology.orchestra.run.smithi154.stdout: "finisher-MDSRank": { 2024-03-16T12:08:48.246 INFO:teuthology.orchestra.run.smithi154.stdout: "queue_len": 0, 2024-03-16T12:08:48.247 INFO:teuthology.orchestra.run.smithi154.stdout: "complete_latency": { 2024-03-16T12:08:48.247 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 4122, 2024-03-16T12:08:48.247 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 2.426220714, 2024-03-16T12:08:48.247 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000588602 2024-03-16T12:08:48.247 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-03-16T12:08:48.247 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-16T12:08:48.247 INFO:teuthology.orchestra.run.smithi154.stdout: "finisher-PurgeQueue": { 2024-03-16T12:08:48.247 INFO:teuthology.orchestra.run.smithi154.stdout: "queue_len": 0, 2024-03-16T12:08:48.247 INFO:teuthology.orchestra.run.smithi154.stdout: "complete_latency": { 2024-03-16T12:08:48.248 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 2250, 2024-03-16T12:08:48.248 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.109736420, 2024-03-16T12:08:48.248 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000048771 2024-03-16T12:08:48.248 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-03-16T12:08:48.248 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-16T12:08:48.248 INFO:teuthology.orchestra.run.smithi154.stdout: "mds": { 2024-03-16T12:08:48.248 INFO:teuthology.orchestra.run.smithi154.stdout: "request": 48814, 2024-03-16T12:08:48.248 INFO:teuthology.orchestra.run.smithi154.stdout: "reply": 6533, 2024-03-16T12:08:48.249 INFO:teuthology.orchestra.run.smithi154.stdout: "reply_latency": { 2024-03-16T12:08:48.249 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 6533, 2024-03-16T12:08:48.249 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 41.305564130, 2024-03-16T12:08:48.249 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.006322602 2024-03-16T12:08:48.249 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-16T12:08:48.249 INFO:teuthology.orchestra.run.smithi154.stdout: "slow_reply": 0, 2024-03-16T12:08:48.249 INFO:teuthology.orchestra.run.smithi154.stdout: "forward": 42281, 2024-03-16T12:08:48.249 INFO:teuthology.orchestra.run.smithi154.stdout: "dir_fetch_complete": 0, 2024-03-16T12:08:48.249 INFO:teuthology.orchestra.run.smithi154.stdout: "dir_fetch_keys": 0, 2024-03-16T12:08:48.250 INFO:teuthology.orchestra.run.smithi154.stdout: "dir_commit": 20, 2024-03-16T12:08:48.250 INFO:teuthology.orchestra.run.smithi154.stdout: "dir_split": 1, 2024-03-16T12:08:48.250 INFO:teuthology.orchestra.run.smithi154.stdout: "dir_merge": 0, 2024-03-16T12:08:48.250 INFO:teuthology.orchestra.run.smithi154.stdout: "inodes": 439, 2024-03-16T12:08:48.250 INFO:teuthology.orchestra.run.smithi154.stdout: "inodes_top": 155, 2024-03-16T12:08:48.250 INFO:teuthology.orchestra.run.smithi154.stdout: "inodes_bottom": 284, 2024-03-16T12:08:48.250 INFO:teuthology.orchestra.run.smithi154.stdout: "inodes_pin_tail": 0, 2024-03-16T12:08:48.250 INFO:teuthology.orchestra.run.smithi154.stdout: "inodes_pinned": 217, 2024-03-16T12:08:48.251 INFO:teuthology.orchestra.run.smithi154.stdout: "inodes_expired": 0, 2024-03-16T12:08:48.251 INFO:teuthology.orchestra.run.smithi154.stdout: "inodes_with_caps": 21, 2024-03-16T12:08:48.251 INFO:teuthology.orchestra.run.smithi154.stdout: "caps": 21, 2024-03-16T12:08:48.251 INFO:teuthology.orchestra.run.smithi154.stdout: "subtrees": 12, 2024-03-16T12:08:48.251 INFO:teuthology.orchestra.run.smithi154.stdout: "traverse": 57219, 2024-03-16T12:08:48.251 INFO:teuthology.orchestra.run.smithi154.stdout: "traverse_hit": 5301, 2024-03-16T12:08:48.251 INFO:teuthology.orchestra.run.smithi154.stdout: "traverse_forward": 42279, 2024-03-16T12:08:48.251 INFO:teuthology.orchestra.run.smithi154.stdout: "traverse_discover": 103, 2024-03-16T12:08:48.251 INFO:teuthology.orchestra.run.smithi154.stdout: "traverse_dir_fetch": 0, 2024-03-16T12:08:48.252 INFO:teuthology.orchestra.run.smithi154.stdout: "traverse_remote_ino": 0, 2024-03-16T12:08:48.252 INFO:teuthology.orchestra.run.smithi154.stdout: "traverse_lock": 33, 2024-03-16T12:08:48.252 INFO:teuthology.orchestra.run.smithi154.stdout: "load_cent": 0, 2024-03-16T12:08:48.252 INFO:teuthology.orchestra.run.smithi154.stdout: "q": 0, 2024-03-16T12:08:48.252 INFO:teuthology.orchestra.run.smithi154.stdout: "exported": 0, 2024-03-16T12:08:48.252 INFO:teuthology.orchestra.run.smithi154.stdout: "exported_inodes": 0, 2024-03-16T12:08:48.252 INFO:teuthology.orchestra.run.smithi154.stdout: "imported": 3, 2024-03-16T12:08:48.252 INFO:teuthology.orchestra.run.smithi154.stdout: "imported_inodes": 43, 2024-03-16T12:08:48.253 INFO:teuthology.orchestra.run.smithi154.stdout: "openino_dir_fetch": 0, 2024-03-16T12:08:48.253 INFO:teuthology.orchestra.run.smithi154.stdout: "openino_backtrace_fetch": 0, 2024-03-16T12:08:48.253 INFO:teuthology.orchestra.run.smithi154.stdout: "openino_peer_discover": 0, 2024-03-16T12:08:48.253 INFO:teuthology.orchestra.run.smithi154.stdout: "root_rfiles": 669, 2024-03-16T12:08:48.253 INFO:teuthology.orchestra.run.smithi154.stdout: "root_rbytes": 211775824, 2024-03-16T12:08:48.253 INFO:teuthology.orchestra.run.smithi154.stdout: "root_rsnaps": 7, 2024-03-16T12:08:48.253 INFO:teuthology.orchestra.run.smithi154.stdout: "scrub_backtrace_fetch": 0, 2024-03-16T12:08:48.253 INFO:teuthology.orchestra.run.smithi154.stdout: "scrub_set_tag": 0, 2024-03-16T12:08:48.253 INFO:teuthology.orchestra.run.smithi154.stdout: "scrub_backtrace_repaired": 0, 2024-03-16T12:08:48.254 INFO:teuthology.orchestra.run.smithi154.stdout: "scrub_inotable_repaired": 0, 2024-03-16T12:08:48.254 INFO:teuthology.orchestra.run.smithi154.stdout: "scrub_dir_inodes": 0, 2024-03-16T12:08:48.254 INFO:teuthology.orchestra.run.smithi154.stdout: "scrub_dir_base_inodes": 0, 2024-03-16T12:08:48.254 INFO:teuthology.orchestra.run.smithi154.stdout: "scrub_dirfrag_rstats": 0, 2024-03-16T12:08:48.254 INFO:teuthology.orchestra.run.smithi154.stdout: "scrub_file_inodes": 0, 2024-03-16T12:08:48.254 INFO:teuthology.orchestra.run.smithi154.stdout: "handle_inode_file_caps": 0, 2024-03-16T12:08:48.254 INFO:teuthology.orchestra.run.smithi154.stdout: "ceph_cap_op_revoke": 429, 2024-03-16T12:08:48.254 INFO:teuthology.orchestra.run.smithi154.stdout: "ceph_cap_op_grant": 5638, 2024-03-16T12:08:48.254 INFO:teuthology.orchestra.run.smithi154.stdout: "ceph_cap_op_trunc": 0, 2024-03-16T12:08:48.255 INFO:teuthology.orchestra.run.smithi154.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-16T12:08:48.255 INFO:teuthology.orchestra.run.smithi154.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-16T12:08:48.255 INFO:teuthology.orchestra.run.smithi154.stdout: "handle_client_caps": 2680, 2024-03-16T12:08:48.255 INFO:teuthology.orchestra.run.smithi154.stdout: "handle_client_caps_dirty": 965, 2024-03-16T12:08:48.255 INFO:teuthology.orchestra.run.smithi154.stdout: "handle_client_cap_release": 495, 2024-03-16T12:08:48.255 INFO:teuthology.orchestra.run.smithi154.stdout: "process_request_cap_release": 6064 2024-03-16T12:08:48.255 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-16T12:08:48.255 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_cache": { 2024-03-16T12:08:48.256 INFO:teuthology.orchestra.run.smithi154.stdout: "dir_update": 89, 2024-03-16T12:08:48.256 INFO:teuthology.orchestra.run.smithi154.stdout: "dir_update_receipt": 405, 2024-03-16T12:08:48.256 INFO:teuthology.orchestra.run.smithi154.stdout: "dir_try_discover": 253, 2024-03-16T12:08:48.256 INFO:teuthology.orchestra.run.smithi154.stdout: "dir_send_discover": 309, 2024-03-16T12:08:48.256 INFO:teuthology.orchestra.run.smithi154.stdout: "dir_handle_discover": 92, 2024-03-16T12:08:48.256 INFO:teuthology.orchestra.run.smithi154.stdout: "num_strays": 0, 2024-03-16T12:08:48.256 INFO:teuthology.orchestra.run.smithi154.stdout: "num_strays_delayed": 0, 2024-03-16T12:08:48.256 INFO:teuthology.orchestra.run.smithi154.stdout: "num_strays_enqueuing": 0, 2024-03-16T12:08:48.256 INFO:teuthology.orchestra.run.smithi154.stdout: "strays_created": 1194, 2024-03-16T12:08:48.257 INFO:teuthology.orchestra.run.smithi154.stdout: "strays_enqueued": 1194, 2024-03-16T12:08:48.257 INFO:teuthology.orchestra.run.smithi154.stdout: "strays_reintegrated": 0, 2024-03-16T12:08:48.257 INFO:teuthology.orchestra.run.smithi154.stdout: "strays_migrated": 0, 2024-03-16T12:08:48.257 INFO:teuthology.orchestra.run.smithi154.stdout: "num_recovering_processing": 0, 2024-03-16T12:08:48.257 INFO:teuthology.orchestra.run.smithi154.stdout: "num_recovering_enqueued": 0, 2024-03-16T12:08:48.257 INFO:teuthology.orchestra.run.smithi154.stdout: "num_recovering_prioritized": 0, 2024-03-16T12:08:48.257 INFO:teuthology.orchestra.run.smithi154.stdout: "recovery_started": 0, 2024-03-16T12:08:48.257 INFO:teuthology.orchestra.run.smithi154.stdout: "recovery_completed": 0, 2024-03-16T12:08:48.257 INFO:teuthology.orchestra.run.smithi154.stdout: "ireq_quiesce_path": 0, 2024-03-16T12:08:48.258 INFO:teuthology.orchestra.run.smithi154.stdout: "ireq_quiesce_inode": 0, 2024-03-16T12:08:48.258 INFO:teuthology.orchestra.run.smithi154.stdout: "ireq_enqueue_scrub": 0, 2024-03-16T12:08:48.258 INFO:teuthology.orchestra.run.smithi154.stdout: "ireq_exportdir": 0, 2024-03-16T12:08:48.258 INFO:teuthology.orchestra.run.smithi154.stdout: "ireq_flush": 0, 2024-03-16T12:08:48.258 INFO:teuthology.orchestra.run.smithi154.stdout: "ireq_fragmentdir": 1, 2024-03-16T12:08:48.258 INFO:teuthology.orchestra.run.smithi154.stdout: "ireq_fragstats": 0, 2024-03-16T12:08:48.258 INFO:teuthology.orchestra.run.smithi154.stdout: "ireq_inodestats": 0 2024-03-16T12:08:48.258 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-16T12:08:48.259 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_log": { 2024-03-16T12:08:48.259 INFO:teuthology.orchestra.run.smithi154.stdout: "evlrg": 0, 2024-03-16T12:08:48.259 INFO:teuthology.orchestra.run.smithi154.stdout: "evadd": 7868, 2024-03-16T12:08:48.259 INFO:teuthology.orchestra.run.smithi154.stdout: "evex": 0, 2024-03-16T12:08:48.259 INFO:teuthology.orchestra.run.smithi154.stdout: "evtrm": 0, 2024-03-16T12:08:48.259 INFO:teuthology.orchestra.run.smithi154.stdout: "ev": 7868, 2024-03-16T12:08:48.259 INFO:teuthology.orchestra.run.smithi154.stdout: "evexg": 0, 2024-03-16T12:08:48.259 INFO:teuthology.orchestra.run.smithi154.stdout: "evexd": 0, 2024-03-16T12:08:48.259 INFO:teuthology.orchestra.run.smithi154.stdout: "segadd": 10, 2024-03-16T12:08:48.260 INFO:teuthology.orchestra.run.smithi154.stdout: "segex": 0, 2024-03-16T12:08:48.260 INFO:teuthology.orchestra.run.smithi154.stdout: "segtrm": 0, 2024-03-16T12:08:48.260 INFO:teuthology.orchestra.run.smithi154.stdout: "seg": 10, 2024-03-16T12:08:48.260 INFO:teuthology.orchestra.run.smithi154.stdout: "segmjr": 2, 2024-03-16T12:08:48.260 INFO:teuthology.orchestra.run.smithi154.stdout: "segexg": 0, 2024-03-16T12:08:48.260 INFO:teuthology.orchestra.run.smithi154.stdout: "segexd": 0, 2024-03-16T12:08:48.260 INFO:teuthology.orchestra.run.smithi154.stdout: "expos": 4194304, 2024-03-16T12:08:48.260 INFO:teuthology.orchestra.run.smithi154.stdout: "wrpos": 16002313, 2024-03-16T12:08:48.260 INFO:teuthology.orchestra.run.smithi154.stdout: "rdpos": 0, 2024-03-16T12:08:48.261 INFO:teuthology.orchestra.run.smithi154.stdout: "jlat": { 2024-03-16T12:08:48.261 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 1996, 2024-03-16T12:08:48.261 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 9.900004062, 2024-03-16T12:08:48.261 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.004959921 2024-03-16T12:08:48.261 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-16T12:08:48.261 INFO:teuthology.orchestra.run.smithi154.stdout: "replayed": 0 2024-03-16T12:08:48.261 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-16T12:08:48.261 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_mem": { 2024-03-16T12:08:48.261 INFO:teuthology.orchestra.run.smithi154.stdout: "ino": 357, 2024-03-16T12:08:48.262 INFO:teuthology.orchestra.run.smithi154.stdout: "ino+": 1551, 2024-03-16T12:08:48.262 INFO:teuthology.orchestra.run.smithi154.stdout: "ino-": 1194, 2024-03-16T12:08:48.262 INFO:teuthology.orchestra.run.smithi154.stdout: "dir": 154, 2024-03-16T12:08:48.262 INFO:teuthology.orchestra.run.smithi154.stdout: "dir+": 216, 2024-03-16T12:08:48.262 INFO:teuthology.orchestra.run.smithi154.stdout: "dir-": 62, 2024-03-16T12:08:48.262 INFO:teuthology.orchestra.run.smithi154.stdout: "dn": 440, 2024-03-16T12:08:48.262 INFO:teuthology.orchestra.run.smithi154.stdout: "dn+": 2669, 2024-03-16T12:08:48.262 INFO:teuthology.orchestra.run.smithi154.stdout: "dn-": 2229, 2024-03-16T12:08:48.263 INFO:teuthology.orchestra.run.smithi154.stdout: "cap": 20, 2024-03-16T12:08:48.263 INFO:teuthology.orchestra.run.smithi154.stdout: "cap+": 1645, 2024-03-16T12:08:48.263 INFO:teuthology.orchestra.run.smithi154.stdout: "cap-": 1625, 2024-03-16T12:08:48.263 INFO:teuthology.orchestra.run.smithi154.stdout: "rss": 61520, 2024-03-16T12:08:48.263 INFO:teuthology.orchestra.run.smithi154.stdout: "heap": 223612 2024-03-16T12:08:48.263 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-16T12:08:48.263 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_server": { 2024-03-16T12:08:48.263 INFO:teuthology.orchestra.run.smithi154.stdout: "dispatch_client_request": 55128, 2024-03-16T12:08:48.263 INFO:teuthology.orchestra.run.smithi154.stdout: "dispatch_server_request": 0, 2024-03-16T12:08:48.264 INFO:teuthology.orchestra.run.smithi154.stdout: "handle_client_request": 48814, 2024-03-16T12:08:48.264 INFO:teuthology.orchestra.run.smithi154.stdout: "handle_client_session": 98, 2024-03-16T12:08:48.264 INFO:teuthology.orchestra.run.smithi154.stdout: "handle_peer_request": 2449, 2024-03-16T12:08:48.264 INFO:teuthology.orchestra.run.smithi154.stdout: "req_create_latency": { 2024-03-16T12:08:48.264 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 1200, 2024-03-16T12:08:48.264 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 24.276791877, 2024-03-16T12:08:48.264 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.020230659 2024-03-16T12:08:48.264 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-16T12:08:48.264 INFO:teuthology.orchestra.run.smithi154.stdout: "req_getattr_latency": { 2024-03-16T12:08:48.265 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 21, 2024-03-16T12:08:48.265 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.015515992, 2024-03-16T12:08:48.265 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000738856 2024-03-16T12:08:48.265 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-16T12:08:48.265 INFO:teuthology.orchestra.run.smithi154.stdout: "req_getfilelock_latency": { 2024-03-16T12:08:48.265 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-16T12:08:48.265 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-16T12:08:48.265 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-16T12:08:48.266 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-16T12:08:48.266 INFO:teuthology.orchestra.run.smithi154.stdout: "req_link_latency": { 2024-03-16T12:08:48.266 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-16T12:08:48.266 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-16T12:08:48.266 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-16T12:08:48.266 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-16T12:08:48.270 INFO:teuthology.orchestra.run.smithi154.stdout: "req_lookup_latency": { 2024-03-16T12:08:48.270 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 1708, 2024-03-16T12:08:48.270 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.654428302, 2024-03-16T12:08:48.271 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000383154 2024-03-16T12:08:48.271 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-16T12:08:48.271 INFO:teuthology.orchestra.run.smithi154.stdout: "req_lookuphash_latency": { 2024-03-16T12:08:48.271 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-16T12:08:48.271 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-16T12:08:48.271 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-16T12:08:48.271 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-16T12:08:48.272 INFO:teuthology.orchestra.run.smithi154.stdout: "req_lookupino_latency": { 2024-03-16T12:08:48.272 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-16T12:08:48.272 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-16T12:08:48.272 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-16T12:08:48.272 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-16T12:08:48.272 INFO:teuthology.orchestra.run.smithi154.stdout: "req_lookupname_latency": { 2024-03-16T12:08:48.272 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-16T12:08:48.273 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-16T12:08:48.273 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-16T12:08:48.273 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-16T12:08:48.273 INFO:teuthology.orchestra.run.smithi154.stdout: "req_lookupparent_latency": { 2024-03-16T12:08:48.273 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-16T12:08:48.273 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-16T12:08:48.274 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-16T12:08:48.274 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-16T12:08:48.274 INFO:teuthology.orchestra.run.smithi154.stdout: "req_lookupsnap_latency": { 2024-03-16T12:08:48.274 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-16T12:08:48.274 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-16T12:08:48.274 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-16T12:08:48.274 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-16T12:08:48.274 INFO:teuthology.orchestra.run.smithi154.stdout: "req_lssnap_latency": { 2024-03-16T12:08:48.275 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-16T12:08:48.276 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-16T12:08:48.276 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-16T12:08:48.276 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-16T12:08:48.276 INFO:teuthology.orchestra.run.smithi154.stdout: "req_mkdir_latency": { 2024-03-16T12:08:48.276 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-16T12:08:48.276 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-16T12:08:48.277 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-16T12:08:48.277 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-16T12:08:48.277 INFO:teuthology.orchestra.run.smithi154.stdout: "req_mknod_latency": { 2024-03-16T12:08:48.277 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-16T12:08:48.277 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-16T12:08:48.277 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-16T12:08:48.277 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-16T12:08:48.278 INFO:teuthology.orchestra.run.smithi154.stdout: "req_mksnap_latency": { 2024-03-16T12:08:48.278 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-16T12:08:48.278 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-16T12:08:48.278 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-16T12:08:48.278 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-16T12:08:48.278 INFO:teuthology.orchestra.run.smithi154.stdout: "req_open_latency": { 2024-03-16T12:08:48.278 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-16T12:08:48.279 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-16T12:08:48.279 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-16T12:08:48.279 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-16T12:08:48.279 INFO:teuthology.orchestra.run.smithi154.stdout: "req_readdir_latency": { 2024-03-16T12:08:48.279 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 2377, 2024-03-16T12:08:48.279 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 1.689921650, 2024-03-16T12:08:48.280 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000710947 2024-03-16T12:08:48.280 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-16T12:08:48.280 INFO:teuthology.orchestra.run.smithi154.stdout: "req_rename_latency": { 2024-03-16T12:08:48.280 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 33, 2024-03-16T12:08:48.280 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.678082844, 2024-03-16T12:08:48.280 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.020547964 2024-03-16T12:08:48.281 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-16T12:08:48.281 INFO:teuthology.orchestra.run.smithi154.stdout: "req_renamesnap_latency": { 2024-03-16T12:08:48.281 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-16T12:08:48.281 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-16T12:08:48.281 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-16T12:08:48.281 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-16T12:08:48.281 INFO:teuthology.orchestra.run.smithi154.stdout: "req_snapdiff_latency": { 2024-03-16T12:08:48.281 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-16T12:08:48.282 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-16T12:08:48.282 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-16T12:08:48.282 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-16T12:08:48.282 INFO:teuthology.orchestra.run.smithi154.stdout: "req_rmdir_latency": { 2024-03-16T12:08:48.282 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-16T12:08:48.282 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-16T12:08:48.282 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-16T12:08:48.282 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-16T12:08:48.283 INFO:teuthology.orchestra.run.smithi154.stdout: "req_rmsnap_latency": { 2024-03-16T12:08:48.283 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-16T12:08:48.283 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-16T12:08:48.283 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-16T12:08:48.283 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-16T12:08:48.283 INFO:teuthology.orchestra.run.smithi154.stdout: "req_rmxattr_latency": { 2024-03-16T12:08:48.283 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-16T12:08:48.283 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-16T12:08:48.283 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-16T12:08:48.284 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-16T12:08:48.284 INFO:teuthology.orchestra.run.smithi154.stdout: "req_setattr_latency": { 2024-03-16T12:08:48.284 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-16T12:08:48.284 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-16T12:08:48.284 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-16T12:08:48.284 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-16T12:08:48.284 INFO:teuthology.orchestra.run.smithi154.stdout: "req_setdirlayout_latency": { 2024-03-16T12:08:48.284 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-16T12:08:48.284 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-16T12:08:48.285 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-16T12:08:48.285 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-16T12:08:48.285 INFO:teuthology.orchestra.run.smithi154.stdout: "req_setfilelock_latency": { 2024-03-16T12:08:48.285 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-16T12:08:48.285 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-16T12:08:48.285 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-16T12:08:48.285 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-16T12:08:48.285 INFO:teuthology.orchestra.run.smithi154.stdout: "req_setlayout_latency": { 2024-03-16T12:08:48.285 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-16T12:08:48.286 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-16T12:08:48.286 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-16T12:08:48.286 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-16T12:08:48.286 INFO:teuthology.orchestra.run.smithi154.stdout: "req_setxattr_latency": { 2024-03-16T12:08:48.286 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-16T12:08:48.286 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-16T12:08:48.286 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-16T12:08:48.286 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-16T12:08:48.287 INFO:teuthology.orchestra.run.smithi154.stdout: "req_symlink_latency": { 2024-03-16T12:08:48.287 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-16T12:08:48.287 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-16T12:08:48.287 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-16T12:08:48.287 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-16T12:08:48.287 INFO:teuthology.orchestra.run.smithi154.stdout: "req_unlink_latency": { 2024-03-16T12:08:48.287 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 1194, 2024-03-16T12:08:48.287 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 13.990823465, 2024-03-16T12:08:48.287 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.011717607 2024-03-16T12:08:48.288 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-16T12:08:48.288 INFO:teuthology.orchestra.run.smithi154.stdout: "cap_revoke_eviction": 0, 2024-03-16T12:08:48.288 INFO:teuthology.orchestra.run.smithi154.stdout: "cap_acquisition_throttle": 0, 2024-03-16T12:08:48.288 INFO:teuthology.orchestra.run.smithi154.stdout: "req_getvxattr_latency": { 2024-03-16T12:08:48.288 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-16T12:08:48.288 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-16T12:08:48.288 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-16T12:08:48.288 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-03-16T12:08:48.289 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-16T12:08:48.289 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_sessions": { 2024-03-16T12:08:48.289 INFO:teuthology.orchestra.run.smithi154.stdout: "session_count": 2, 2024-03-16T12:08:48.289 INFO:teuthology.orchestra.run.smithi154.stdout: "session_add": 3, 2024-03-16T12:08:48.289 INFO:teuthology.orchestra.run.smithi154.stdout: "session_remove": 1, 2024-03-16T12:08:48.289 INFO:teuthology.orchestra.run.smithi154.stdout: "sessions_open": 2, 2024-03-16T12:08:48.289 INFO:teuthology.orchestra.run.smithi154.stdout: "sessions_stale": 0, 2024-03-16T12:08:48.289 INFO:teuthology.orchestra.run.smithi154.stdout: "total_load": 2274, 2024-03-16T12:08:48.289 INFO:teuthology.orchestra.run.smithi154.stdout: "average_load": 1137, 2024-03-16T12:08:48.290 INFO:teuthology.orchestra.run.smithi154.stdout: "avg_session_uptime": 1071, 2024-03-16T12:08:48.290 INFO:teuthology.orchestra.run.smithi154.stdout: "mdthresh_evicted": 0 2024-03-16T12:08:48.290 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-16T12:08:48.290 INFO:teuthology.orchestra.run.smithi154.stdout: "mempool": { 2024-03-16T12:08:48.290 INFO:teuthology.orchestra.run.smithi154.stdout: "bloom_filter_bytes": 0, 2024-03-16T12:08:48.290 INFO:teuthology.orchestra.run.smithi154.stdout: "bloom_filter_items": 0, 2024-03-16T12:08:48.290 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_alloc_bytes": 0, 2024-03-16T12:08:48.290 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_alloc_items": 0, 2024-03-16T12:08:48.290 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_cache_data_bytes": 0, 2024-03-16T12:08:48.291 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_cache_data_items": 0, 2024-03-16T12:08:48.291 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-16T12:08:48.291 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_cache_onode_items": 0, 2024-03-16T12:08:48.291 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-16T12:08:48.291 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_cache_meta_items": 0, 2024-03-16T12:08:48.291 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_cache_other_bytes": 0, 2024-03-16T12:08:48.291 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_cache_other_items": 0, 2024-03-16T12:08:48.291 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-16T12:08:48.291 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_cache_buffer_items": 0, 2024-03-16T12:08:48.292 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_extent_bytes": 0, 2024-03-16T12:08:48.292 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_extent_items": 0, 2024-03-16T12:08:48.292 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_blob_bytes": 0, 2024-03-16T12:08:48.292 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_blob_items": 0, 2024-03-16T12:08:48.292 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-16T12:08:48.292 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_shared_blob_items": 0, 2024-03-16T12:08:48.292 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-16T12:08:48.292 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_inline_bl_items": 0, 2024-03-16T12:08:48.293 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_fsck_bytes": 0, 2024-03-16T12:08:48.293 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_fsck_items": 0, 2024-03-16T12:08:48.293 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_txc_bytes": 0, 2024-03-16T12:08:48.293 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_txc_items": 0, 2024-03-16T12:08:48.293 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-16T12:08:48.293 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_writing_deferred_items": 0, 2024-03-16T12:08:48.293 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_writing_bytes": 0, 2024-03-16T12:08:48.293 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_writing_items": 0, 2024-03-16T12:08:48.293 INFO:teuthology.orchestra.run.smithi154.stdout: "bluefs_bytes": 0, 2024-03-16T12:08:48.294 INFO:teuthology.orchestra.run.smithi154.stdout: "bluefs_items": 0, 2024-03-16T12:08:48.294 INFO:teuthology.orchestra.run.smithi154.stdout: "bluefs_file_reader_bytes": 0, 2024-03-16T12:08:48.294 INFO:teuthology.orchestra.run.smithi154.stdout: "bluefs_file_reader_items": 0, 2024-03-16T12:08:48.294 INFO:teuthology.orchestra.run.smithi154.stdout: "bluefs_file_writer_bytes": 0, 2024-03-16T12:08:48.294 INFO:teuthology.orchestra.run.smithi154.stdout: "bluefs_file_writer_items": 0, 2024-03-16T12:08:48.294 INFO:teuthology.orchestra.run.smithi154.stdout: "buffer_anon_bytes": 392126, 2024-03-16T12:08:48.294 INFO:teuthology.orchestra.run.smithi154.stdout: "buffer_anon_items": 337, 2024-03-16T12:08:48.294 INFO:teuthology.orchestra.run.smithi154.stdout: "buffer_meta_bytes": 704, 2024-03-16T12:08:48.294 INFO:teuthology.orchestra.run.smithi154.stdout: "buffer_meta_items": 8, 2024-03-16T12:08:48.295 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_bytes": 0, 2024-03-16T12:08:48.295 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_items": 0, 2024-03-16T12:08:48.295 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_mapbl_bytes": 0, 2024-03-16T12:08:48.295 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_mapbl_items": 0, 2024-03-16T12:08:48.295 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_pglog_bytes": 0, 2024-03-16T12:08:48.295 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_pglog_items": 0, 2024-03-16T12:08:48.295 INFO:teuthology.orchestra.run.smithi154.stdout: "osdmap_bytes": 8256, 2024-03-16T12:08:48.295 INFO:teuthology.orchestra.run.smithi154.stdout: "osdmap_items": 188, 2024-03-16T12:08:48.295 INFO:teuthology.orchestra.run.smithi154.stdout: "osdmap_mapping_bytes": 0, 2024-03-16T12:08:48.296 INFO:teuthology.orchestra.run.smithi154.stdout: "osdmap_mapping_items": 0, 2024-03-16T12:08:48.296 INFO:teuthology.orchestra.run.smithi154.stdout: "pgmap_bytes": 0, 2024-03-16T12:08:48.296 INFO:teuthology.orchestra.run.smithi154.stdout: "pgmap_items": 0, 2024-03-16T12:08:48.296 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_co_bytes": 1351921, 2024-03-16T12:08:48.296 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_co_items": 9408, 2024-03-16T12:08:48.296 INFO:teuthology.orchestra.run.smithi154.stdout: "unittest_1_bytes": 0, 2024-03-16T12:08:48.296 INFO:teuthology.orchestra.run.smithi154.stdout: "unittest_1_items": 0, 2024-03-16T12:08:48.296 INFO:teuthology.orchestra.run.smithi154.stdout: "unittest_2_bytes": 0, 2024-03-16T12:08:48.297 INFO:teuthology.orchestra.run.smithi154.stdout: "unittest_2_items": 0 2024-03-16T12:08:48.297 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-16T12:08:48.297 INFO:teuthology.orchestra.run.smithi154.stdout: "objecter": { 2024-03-16T12:08:48.297 INFO:teuthology.orchestra.run.smithi154.stdout: "op_active": 0, 2024-03-16T12:08:48.297 INFO:teuthology.orchestra.run.smithi154.stdout: "op_laggy": 0, 2024-03-16T12:08:48.297 INFO:teuthology.orchestra.run.smithi154.stdout: "op_send": 3724, 2024-03-16T12:08:48.297 INFO:teuthology.orchestra.run.smithi154.stdout: "op_send_bytes": 12238550, 2024-03-16T12:08:48.297 INFO:teuthology.orchestra.run.smithi154.stdout: "op_resend": 28, 2024-03-16T12:08:48.297 INFO:teuthology.orchestra.run.smithi154.stdout: "op_reply": 3724, 2024-03-16T12:08:48.298 INFO:teuthology.orchestra.run.smithi154.stdout: "op_latency": { 2024-03-16T12:08:48.298 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 3724, 2024-03-16T12:08:48.298 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 21.554473075, 2024-03-16T12:08:48.298 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.005787989 2024-03-16T12:08:48.298 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-16T12:08:48.298 INFO:teuthology.orchestra.run.smithi154.stdout: "op_inflight": 0, 2024-03-16T12:08:48.298 INFO:teuthology.orchestra.run.smithi154.stdout: "oplen_avg": { 2024-03-16T12:08:48.298 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 3724, 2024-03-16T12:08:48.298 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 3988 2024-03-16T12:08:48.299 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-16T12:08:48.299 INFO:teuthology.orchestra.run.smithi154.stdout: "op": 3724, 2024-03-16T12:08:48.299 INFO:teuthology.orchestra.run.smithi154.stdout: "op_r": 109, 2024-03-16T12:08:48.299 INFO:teuthology.orchestra.run.smithi154.stdout: "op_w": 3615, 2024-03-16T12:08:48.299 INFO:teuthology.orchestra.run.smithi154.stdout: "op_rmw": 0, 2024-03-16T12:08:48.299 INFO:teuthology.orchestra.run.smithi154.stdout: "op_pg": 0, 2024-03-16T12:08:48.299 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_stat": 0, 2024-03-16T12:08:48.299 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_create": 10, 2024-03-16T12:08:48.299 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_read": 109, 2024-03-16T12:08:48.300 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_write": 2105, 2024-03-16T12:08:48.300 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_writefull": 85, 2024-03-16T12:08:48.300 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_writesame": 0, 2024-03-16T12:08:48.300 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_append": 0, 2024-03-16T12:08:48.300 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_zero": 0, 2024-03-16T12:08:48.300 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_truncate": 1, 2024-03-16T12:08:48.300 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_delete": 1208, 2024-03-16T12:08:48.300 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_mapext": 0, 2024-03-16T12:08:48.300 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_sparse_read": 0, 2024-03-16T12:08:48.301 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_clonerange": 0, 2024-03-16T12:08:48.301 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_getxattr": 0, 2024-03-16T12:08:48.301 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_setxattr": 20, 2024-03-16T12:08:48.301 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_cmpxattr": 0, 2024-03-16T12:08:48.301 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_rmxattr": 0, 2024-03-16T12:08:48.302 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_resetxattrs": 0, 2024-03-16T12:08:48.302 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_call": 0, 2024-03-16T12:08:48.302 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_watch": 0, 2024-03-16T12:08:48.302 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_notify": 0, 2024-03-16T12:08:48.302 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_src_cmpxattr": 0, 2024-03-16T12:08:48.302 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_pgls": 0, 2024-03-16T12:08:48.302 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_pgls_filter": 0, 2024-03-16T12:08:48.302 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_other": 0, 2024-03-16T12:08:48.303 INFO:teuthology.orchestra.run.smithi154.stdout: "linger_active": 0, 2024-03-16T12:08:48.303 INFO:teuthology.orchestra.run.smithi154.stdout: "linger_send": 0, 2024-03-16T12:08:48.303 INFO:teuthology.orchestra.run.smithi154.stdout: "linger_resend": 0, 2024-03-16T12:08:48.303 INFO:teuthology.orchestra.run.smithi154.stdout: "linger_ping": 0, 2024-03-16T12:08:48.303 INFO:teuthology.orchestra.run.smithi154.stdout: "poolop_active": 0, 2024-03-16T12:08:48.303 INFO:teuthology.orchestra.run.smithi154.stdout: "poolop_send": 0, 2024-03-16T12:08:48.304 INFO:teuthology.orchestra.run.smithi154.stdout: "poolop_resend": 0, 2024-03-16T12:08:48.304 INFO:teuthology.orchestra.run.smithi154.stdout: "poolstat_active": 0, 2024-03-16T12:08:48.304 INFO:teuthology.orchestra.run.smithi154.stdout: "poolstat_send": 0, 2024-03-16T12:08:48.304 INFO:teuthology.orchestra.run.smithi154.stdout: "poolstat_resend": 0, 2024-03-16T12:08:48.304 INFO:teuthology.orchestra.run.smithi154.stdout: "statfs_active": 0, 2024-03-16T12:08:48.304 INFO:teuthology.orchestra.run.smithi154.stdout: "statfs_send": 0, 2024-03-16T12:08:48.304 INFO:teuthology.orchestra.run.smithi154.stdout: "statfs_resend": 0, 2024-03-16T12:08:48.304 INFO:teuthology.orchestra.run.smithi154.stdout: "command_active": 0, 2024-03-16T12:08:48.304 INFO:teuthology.orchestra.run.smithi154.stdout: "command_send": 0, 2024-03-16T12:08:48.305 INFO:teuthology.orchestra.run.smithi154.stdout: "command_resend": 0, 2024-03-16T12:08:48.305 INFO:teuthology.orchestra.run.smithi154.stdout: "map_epoch": 116, 2024-03-16T12:08:48.305 INFO:teuthology.orchestra.run.smithi154.stdout: "map_full": 0, 2024-03-16T12:08:48.305 INFO:teuthology.orchestra.run.smithi154.stdout: "map_inc": 31, 2024-03-16T12:08:48.305 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_sessions": 12, 2024-03-16T12:08:48.305 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_session_open": 18, 2024-03-16T12:08:48.306 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_session_close": 6, 2024-03-16T12:08:48.306 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_laggy": 0, 2024-03-16T12:08:48.306 INFO:teuthology.orchestra.run.smithi154.stdout: "omap_wr": 367, 2024-03-16T12:08:48.306 INFO:teuthology.orchestra.run.smithi154.stdout: "omap_rd": 0, 2024-03-16T12:08:48.306 INFO:teuthology.orchestra.run.smithi154.stdout: "omap_del": 83 2024-03-16T12:08:48.306 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-16T12:08:48.306 INFO:teuthology.orchestra.run.smithi154.stdout: "oft": { 2024-03-16T12:08:48.306 INFO:teuthology.orchestra.run.smithi154.stdout: "omap_total_objs": 1, 2024-03-16T12:08:48.307 INFO:teuthology.orchestra.run.smithi154.stdout: "omap_total_kv_pairs": 0, 2024-03-16T12:08:48.307 INFO:teuthology.orchestra.run.smithi154.stdout: "omap_total_updates": 0, 2024-03-16T12:08:48.307 INFO:teuthology.orchestra.run.smithi154.stdout: "omap_total_removes": 0 2024-03-16T12:08:48.307 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-16T12:08:48.307 INFO:teuthology.orchestra.run.smithi154.stdout: "osd-slow-ops": { 2024-03-16T12:08:48.307 INFO:teuthology.orchestra.run.smithi154.stdout: "slow_ops_count": 1 2024-03-16T12:08:48.308 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-16T12:08:48.308 INFO:teuthology.orchestra.run.smithi154.stdout: "purge_queue": { 2024-03-16T12:08:48.308 INFO:teuthology.orchestra.run.smithi154.stdout: "pq_executing_ops": 0, 2024-03-16T12:08:48.308 INFO:teuthology.orchestra.run.smithi154.stdout: "pq_executing_ops_high_water": 10, 2024-03-16T12:08:48.308 INFO:teuthology.orchestra.run.smithi154.stdout: "pq_executing": 0, 2024-03-16T12:08:48.308 INFO:teuthology.orchestra.run.smithi154.stdout: "pq_executing_high_water": 10, 2024-03-16T12:08:48.308 INFO:teuthology.orchestra.run.smithi154.stdout: "pq_executed_ops": 1194, 2024-03-16T12:08:48.308 INFO:teuthology.orchestra.run.smithi154.stdout: "pq_executed": 1194, 2024-03-16T12:08:48.309 INFO:teuthology.orchestra.run.smithi154.stdout: "pq_item_in_journal": 0 2024-03-16T12:08:48.309 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-16T12:08:48.309 INFO:teuthology.orchestra.run.smithi154.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-16T12:08:48.309 INFO:teuthology.orchestra.run.smithi154.stdout: "val": 0, 2024-03-16T12:08:48.309 INFO:teuthology.orchestra.run.smithi154.stdout: "max": 104857600, 2024-03-16T12:08:48.309 INFO:teuthology.orchestra.run.smithi154.stdout: "get_started": 0, 2024-03-16T12:08:48.310 INFO:teuthology.orchestra.run.smithi154.stdout: "get": 437497, 2024-03-16T12:08:48.310 INFO:teuthology.orchestra.run.smithi154.stdout: "get_sum": 143155278, 2024-03-16T12:08:48.310 INFO:teuthology.orchestra.run.smithi154.stdout: "get_or_fail_fail": 0, 2024-03-16T12:08:48.310 INFO:teuthology.orchestra.run.smithi154.stdout: "get_or_fail_success": 437497, 2024-03-16T12:08:48.310 INFO:teuthology.orchestra.run.smithi154.stdout: "take": 0, 2024-03-16T12:08:48.310 INFO:teuthology.orchestra.run.smithi154.stdout: "take_sum": 0, 2024-03-16T12:08:48.311 INFO:teuthology.orchestra.run.smithi154.stdout: "put": 437497, 2024-03-16T12:08:48.311 INFO:teuthology.orchestra.run.smithi154.stdout: "put_sum": 143155278, 2024-03-16T12:08:48.311 INFO:teuthology.orchestra.run.smithi154.stdout: "wait": { 2024-03-16T12:08:48.311 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-16T12:08:48.311 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-16T12:08:48.311 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-16T12:08:48.311 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-03-16T12:08:48.312 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-16T12:08:48.312 INFO:teuthology.orchestra.run.smithi154.stdout: "throttle-objecter_bytes": { 2024-03-16T12:08:48.312 INFO:teuthology.orchestra.run.smithi154.stdout: "val": 0, 2024-03-16T12:08:48.312 INFO:teuthology.orchestra.run.smithi154.stdout: "max": 104857600, 2024-03-16T12:08:48.312 INFO:teuthology.orchestra.run.smithi154.stdout: "get_started": 0, 2024-03-16T12:08:48.312 INFO:teuthology.orchestra.run.smithi154.stdout: "get": 0, 2024-03-16T12:08:48.313 INFO:teuthology.orchestra.run.smithi154.stdout: "get_sum": 0, 2024-03-16T12:08:48.313 INFO:teuthology.orchestra.run.smithi154.stdout: "get_or_fail_fail": 0, 2024-03-16T12:08:48.313 INFO:teuthology.orchestra.run.smithi154.stdout: "get_or_fail_success": 0, 2024-03-16T12:08:48.313 INFO:teuthology.orchestra.run.smithi154.stdout: "take": 3724, 2024-03-16T12:08:48.313 INFO:teuthology.orchestra.run.smithi154.stdout: "take_sum": 12359336, 2024-03-16T12:08:48.313 INFO:teuthology.orchestra.run.smithi154.stdout: "put": 3724, 2024-03-16T12:08:48.314 INFO:teuthology.orchestra.run.smithi154.stdout: "put_sum": 12359336, 2024-03-16T12:08:48.314 INFO:teuthology.orchestra.run.smithi154.stdout: "wait": { 2024-03-16T12:08:48.314 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-16T12:08:48.314 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-16T12:08:48.314 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-16T12:08:48.314 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-03-16T12:08:48.314 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-16T12:08:48.314 INFO:teuthology.orchestra.run.smithi154.stdout: "throttle-objecter_ops": { 2024-03-16T12:08:48.315 INFO:teuthology.orchestra.run.smithi154.stdout: "val": 0, 2024-03-16T12:08:48.315 INFO:teuthology.orchestra.run.smithi154.stdout: "max": 1024, 2024-03-16T12:08:48.315 INFO:teuthology.orchestra.run.smithi154.stdout: "get_started": 0, 2024-03-16T12:08:48.315 INFO:teuthology.orchestra.run.smithi154.stdout: "get": 0, 2024-03-16T12:08:48.315 INFO:teuthology.orchestra.run.smithi154.stdout: "get_sum": 0, 2024-03-16T12:08:48.315 INFO:teuthology.orchestra.run.smithi154.stdout: "get_or_fail_fail": 0, 2024-03-16T12:08:48.315 INFO:teuthology.orchestra.run.smithi154.stdout: "get_or_fail_success": 0, 2024-03-16T12:08:48.315 INFO:teuthology.orchestra.run.smithi154.stdout: "take": 3724, 2024-03-16T12:08:48.315 INFO:teuthology.orchestra.run.smithi154.stdout: "take_sum": 3724, 2024-03-16T12:08:48.316 INFO:teuthology.orchestra.run.smithi154.stdout: "put": 3724, 2024-03-16T12:08:48.316 INFO:teuthology.orchestra.run.smithi154.stdout: "put_sum": 3724, 2024-03-16T12:08:48.316 INFO:teuthology.orchestra.run.smithi154.stdout: "wait": { 2024-03-16T12:08:48.316 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-16T12:08:48.316 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-16T12:08:48.316 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-16T12:08:48.316 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-03-16T12:08:48.316 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-16T12:08:48.316 INFO:teuthology.orchestra.run.smithi154.stdout: "throttle-write_buf_throttle": { 2024-03-16T12:08:48.317 INFO:teuthology.orchestra.run.smithi154.stdout: "val": 0, 2024-03-16T12:08:48.317 INFO:teuthology.orchestra.run.smithi154.stdout: "max": 3758096384, 2024-03-16T12:08:48.317 INFO:teuthology.orchestra.run.smithi154.stdout: "get_started": 0, 2024-03-16T12:08:48.317 INFO:teuthology.orchestra.run.smithi154.stdout: "get": 1194, 2024-03-16T12:08:48.317 INFO:teuthology.orchestra.run.smithi154.stdout: "get_sum": 120786, 2024-03-16T12:08:48.317 INFO:teuthology.orchestra.run.smithi154.stdout: "get_or_fail_fail": 0, 2024-03-16T12:08:48.317 INFO:teuthology.orchestra.run.smithi154.stdout: "get_or_fail_success": 1194, 2024-03-16T12:08:48.317 INFO:teuthology.orchestra.run.smithi154.stdout: "take": 0, 2024-03-16T12:08:48.317 INFO:teuthology.orchestra.run.smithi154.stdout: "take_sum": 0, 2024-03-16T12:08:48.318 INFO:teuthology.orchestra.run.smithi154.stdout: "put": 109, 2024-03-16T12:08:48.318 INFO:teuthology.orchestra.run.smithi154.stdout: "put_sum": 120786, 2024-03-16T12:08:48.318 INFO:teuthology.orchestra.run.smithi154.stdout: "wait": { 2024-03-16T12:08:48.318 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-16T12:08:48.318 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-16T12:08:48.318 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-16T12:08:48.318 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-03-16T12:08:48.318 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-16T12:08:48.319 INFO:teuthology.orchestra.run.smithi154.stdout: "throttle-write_buf_throttle-0x55c63aed4500": { 2024-03-16T12:08:48.319 INFO:teuthology.orchestra.run.smithi154.stdout: "val": 3462, 2024-03-16T12:08:48.319 INFO:teuthology.orchestra.run.smithi154.stdout: "max": 3758096384, 2024-03-16T12:08:48.319 INFO:teuthology.orchestra.run.smithi154.stdout: "get_started": 0, 2024-03-16T12:08:48.319 INFO:teuthology.orchestra.run.smithi154.stdout: "get": 7868, 2024-03-16T12:08:48.319 INFO:teuthology.orchestra.run.smithi154.stdout: "get_sum": 11808009, 2024-03-16T12:08:48.319 INFO:teuthology.orchestra.run.smithi154.stdout: "get_or_fail_fail": 0, 2024-03-16T12:08:48.319 INFO:teuthology.orchestra.run.smithi154.stdout: "get_or_fail_success": 7868, 2024-03-16T12:08:48.319 INFO:teuthology.orchestra.run.smithi154.stdout: "take": 0, 2024-03-16T12:08:48.320 INFO:teuthology.orchestra.run.smithi154.stdout: "take_sum": 0, 2024-03-16T12:08:48.320 INFO:teuthology.orchestra.run.smithi154.stdout: "put": 1996, 2024-03-16T12:08:48.320 INFO:teuthology.orchestra.run.smithi154.stdout: "put_sum": 11804547, 2024-03-16T12:08:48.320 INFO:teuthology.orchestra.run.smithi154.stdout: "wait": { 2024-03-16T12:08:48.320 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-16T12:08:48.320 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-16T12:08:48.320 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-16T12:08:48.320 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-03-16T12:08:48.321 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-03-16T12:08:48.321 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-03-16T12:08:48.785 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:08:48 smithi154 ceph-mon[36320]: pgmap v567: 97 pgs: 97 active+clean; 362 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 144 KiB/s rd, 804 KiB/s wr, 35 op/s 2024-03-16T12:08:48.828 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:08:48 smithi029 ceph-mon[31975]: pgmap v567: 97 pgs: 97 active+clean; 362 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 144 KiB/s rd, 804 KiB/s wr, 35 op/s 2024-03-16T12:08:48.868 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:08:48 smithi152 ceph-mon[36291]: pgmap v567: 97 pgs: 97 active+clean; 362 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 144 KiB/s rd, 804 KiB/s wr, 35 op/s 2024-03-16T12:08:49.029 INFO:tasks.check_counter:Daemon mds.i mds.exported=0 2024-03-16T12:08:49.030 INFO:tasks.check_counter:Daemon mds.i mds.imported=3 2024-03-16T12:08:49.030 INFO:tasks.check_counter:Daemon mds.i mds_cache.dir_update=89 2024-03-16T12:08:49.030 INFO:tasks.check_counter:Daemon mds.i mds_cache.dir_update_receipt=405 2024-03-16T12:08:49.030 INFO:tasks.check_counter:Daemon mds.i mds.root_rsnaps=7 2024-03-16T12:08:49.030 INFO:tasks.check_counter:Daemon mds.i mds_server.req_mksnap_latency.avgcount=0 2024-03-16T12:08:49.030 INFO:tasks.check_counter:Daemon mds.i mds_server.req_rmsnap_latency.avgcount=0 2024-03-16T12:08:49.030 DEBUG:tasks.check_counter:Getting stats from l 2024-03-16T12:08:49.031 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid d1887f28-e389-11ee-95c9-87774f69a715 -- ceph daemon mds.l perf dump 2024-03-16T12:08:49.273 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/d1887f28-e389-11ee-95c9-87774f69a715/mon.c/config 2024-03-16T12:08:50.692 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:08:50 smithi154 ceph-mon[36320]: pgmap v568: 97 pgs: 97 active+clean; 362 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 318 KiB/s wr, 15 op/s 2024-03-16T12:08:50.828 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:08:50 smithi029 ceph-mon[31975]: pgmap v568: 97 pgs: 97 active+clean; 362 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 318 KiB/s wr, 15 op/s 2024-03-16T12:08:50.867 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:08:50 smithi152 ceph-mon[36291]: pgmap v568: 97 pgs: 97 active+clean; 362 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 318 KiB/s wr, 15 op/s 2024-03-16T12:08:50.969 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-03-16T12:08:50.969 INFO:teuthology.orchestra.run.smithi154.stdout: "AsyncMessenger::Worker-0": { 2024-03-16T12:08:50.969 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_recv_messages": 4, 2024-03-16T12:08:50.970 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_send_messages": 4, 2024-03-16T12:08:50.970 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_recv_bytes": 3072, 2024-03-16T12:08:50.970 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_send_bytes": 448, 2024-03-16T12:08:50.970 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_created_connections": 4, 2024-03-16T12:08:50.970 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_active_connections": 0, 2024-03-16T12:08:50.970 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_running_total_time": 0.001479779, 2024-03-16T12:08:50.970 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_running_send_time": 0.000038621, 2024-03-16T12:08:50.970 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_running_recv_time": 0.000625688, 2024-03-16T12:08:50.970 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-16T12:08:50.971 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_send_messages_queue_lat": { 2024-03-16T12:08:50.971 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 3, 2024-03-16T12:08:50.971 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000661330, 2024-03-16T12:08:50.971 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000220443 2024-03-16T12:08:50.971 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-16T12:08:50.971 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_handle_ack_lat": { 2024-03-16T12:08:50.971 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-16T12:08:50.971 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-16T12:08:50.972 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-16T12:08:50.972 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-16T12:08:50.972 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_recv_encrypted_bytes": 3072, 2024-03-16T12:08:50.972 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_send_encrypted_bytes": 448 2024-03-16T12:08:50.972 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-16T12:08:50.972 INFO:teuthology.orchestra.run.smithi154.stdout: "AsyncMessenger::Worker-1": { 2024-03-16T12:08:50.972 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_recv_messages": 343, 2024-03-16T12:08:50.972 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_send_messages": 333, 2024-03-16T12:08:50.972 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_recv_bytes": 1472128, 2024-03-16T12:08:50.973 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_send_bytes": 205520, 2024-03-16T12:08:50.973 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_created_connections": 2, 2024-03-16T12:08:50.973 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_active_connections": 1, 2024-03-16T12:08:50.973 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_running_total_time": 0.045822247, 2024-03-16T12:08:50.973 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_running_send_time": 0.012708119, 2024-03-16T12:08:50.973 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_running_recv_time": 0.035934282, 2024-03-16T12:08:50.973 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_running_fast_dispatch_time": 0.006179663, 2024-03-16T12:08:50.973 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_send_messages_queue_lat": { 2024-03-16T12:08:50.973 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 332, 2024-03-16T12:08:50.974 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.011821431, 2024-03-16T12:08:50.974 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000035606 2024-03-16T12:08:50.974 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-16T12:08:50.974 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_handle_ack_lat": { 2024-03-16T12:08:50.974 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-16T12:08:50.974 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-16T12:08:50.974 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-16T12:08:50.974 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-16T12:08:50.974 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_recv_encrypted_bytes": 1472128, 2024-03-16T12:08:50.975 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_send_encrypted_bytes": 205520 2024-03-16T12:08:50.975 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-16T12:08:50.975 INFO:teuthology.orchestra.run.smithi154.stdout: "AsyncMessenger::Worker-2": { 2024-03-16T12:08:50.975 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_recv_messages": 3, 2024-03-16T12:08:50.975 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_send_messages": 256, 2024-03-16T12:08:50.975 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_recv_bytes": 480, 2024-03-16T12:08:50.975 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_send_bytes": 278864, 2024-03-16T12:08:50.975 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_created_connections": 6, 2024-03-16T12:08:50.975 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_active_connections": 3, 2024-03-16T12:08:50.976 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_running_total_time": 0.011320812, 2024-03-16T12:08:50.976 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_running_send_time": 0.007585247, 2024-03-16T12:08:50.976 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_running_recv_time": 0.001683918, 2024-03-16T12:08:50.976 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-16T12:08:50.976 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_send_messages_queue_lat": { 2024-03-16T12:08:50.976 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 253, 2024-03-16T12:08:50.976 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.019853868, 2024-03-16T12:08:50.976 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000078473 2024-03-16T12:08:50.976 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-16T12:08:50.977 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_handle_ack_lat": { 2024-03-16T12:08:50.977 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-16T12:08:50.977 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-16T12:08:50.977 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-16T12:08:50.977 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-16T12:08:50.977 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_recv_encrypted_bytes": 480, 2024-03-16T12:08:50.977 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_send_encrypted_bytes": 278864 2024-03-16T12:08:50.977 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-16T12:08:50.977 INFO:teuthology.orchestra.run.smithi154.stdout: "cct": { 2024-03-16T12:08:50.978 INFO:teuthology.orchestra.run.smithi154.stdout: "total_workers": 0, 2024-03-16T12:08:50.978 INFO:teuthology.orchestra.run.smithi154.stdout: "unhealthy_workers": 0 2024-03-16T12:08:50.978 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-16T12:08:50.978 INFO:teuthology.orchestra.run.smithi154.stdout: "mempool": { 2024-03-16T12:08:50.978 INFO:teuthology.orchestra.run.smithi154.stdout: "bloom_filter_bytes": 0, 2024-03-16T12:08:50.978 INFO:teuthology.orchestra.run.smithi154.stdout: "bloom_filter_items": 0, 2024-03-16T12:08:50.978 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_alloc_bytes": 0, 2024-03-16T12:08:50.978 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_alloc_items": 0, 2024-03-16T12:08:50.979 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_cache_data_bytes": 0, 2024-03-16T12:08:50.979 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_cache_data_items": 0, 2024-03-16T12:08:50.979 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-16T12:08:50.979 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_cache_onode_items": 0, 2024-03-16T12:08:50.979 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-16T12:08:50.979 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_cache_meta_items": 0, 2024-03-16T12:08:50.979 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_cache_other_bytes": 0, 2024-03-16T12:08:50.979 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_cache_other_items": 0, 2024-03-16T12:08:50.979 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-16T12:08:50.980 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_cache_buffer_items": 0, 2024-03-16T12:08:50.980 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_extent_bytes": 0, 2024-03-16T12:08:50.980 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_extent_items": 0, 2024-03-16T12:08:50.980 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_blob_bytes": 0, 2024-03-16T12:08:50.980 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_blob_items": 0, 2024-03-16T12:08:50.980 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-16T12:08:50.980 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_shared_blob_items": 0, 2024-03-16T12:08:50.980 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-16T12:08:50.980 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_inline_bl_items": 0, 2024-03-16T12:08:50.981 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_fsck_bytes": 0, 2024-03-16T12:08:50.981 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_fsck_items": 0, 2024-03-16T12:08:50.981 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_txc_bytes": 0, 2024-03-16T12:08:50.981 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_txc_items": 0, 2024-03-16T12:08:50.981 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-16T12:08:50.981 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_writing_deferred_items": 0, 2024-03-16T12:08:50.981 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_writing_bytes": 0, 2024-03-16T12:08:50.981 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_writing_items": 0, 2024-03-16T12:08:50.981 INFO:teuthology.orchestra.run.smithi154.stdout: "bluefs_bytes": 0, 2024-03-16T12:08:50.982 INFO:teuthology.orchestra.run.smithi154.stdout: "bluefs_items": 0, 2024-03-16T12:08:50.982 INFO:teuthology.orchestra.run.smithi154.stdout: "bluefs_file_reader_bytes": 0, 2024-03-16T12:08:50.982 INFO:teuthology.orchestra.run.smithi154.stdout: "bluefs_file_reader_items": 0, 2024-03-16T12:08:50.982 INFO:teuthology.orchestra.run.smithi154.stdout: "bluefs_file_writer_bytes": 0, 2024-03-16T12:08:50.982 INFO:teuthology.orchestra.run.smithi154.stdout: "bluefs_file_writer_items": 0, 2024-03-16T12:08:50.982 INFO:teuthology.orchestra.run.smithi154.stdout: "buffer_anon_bytes": 148732, 2024-03-16T12:08:50.982 INFO:teuthology.orchestra.run.smithi154.stdout: "buffer_anon_items": 31, 2024-03-16T12:08:50.982 INFO:teuthology.orchestra.run.smithi154.stdout: "buffer_meta_bytes": 0, 2024-03-16T12:08:50.983 INFO:teuthology.orchestra.run.smithi154.stdout: "buffer_meta_items": 0, 2024-03-16T12:08:50.983 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_bytes": 0, 2024-03-16T12:08:50.983 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_items": 0, 2024-03-16T12:08:50.983 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_mapbl_bytes": 0, 2024-03-16T12:08:50.983 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_mapbl_items": 0, 2024-03-16T12:08:50.983 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_pglog_bytes": 0, 2024-03-16T12:08:50.983 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_pglog_items": 0, 2024-03-16T12:08:50.983 INFO:teuthology.orchestra.run.smithi154.stdout: "osdmap_bytes": 0, 2024-03-16T12:08:50.983 INFO:teuthology.orchestra.run.smithi154.stdout: "osdmap_items": 0, 2024-03-16T12:08:50.984 INFO:teuthology.orchestra.run.smithi154.stdout: "osdmap_mapping_bytes": 0, 2024-03-16T12:08:50.984 INFO:teuthology.orchestra.run.smithi154.stdout: "osdmap_mapping_items": 0, 2024-03-16T12:08:50.984 INFO:teuthology.orchestra.run.smithi154.stdout: "pgmap_bytes": 0, 2024-03-16T12:08:50.984 INFO:teuthology.orchestra.run.smithi154.stdout: "pgmap_items": 0, 2024-03-16T12:08:50.984 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_co_bytes": 928, 2024-03-16T12:08:50.984 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_co_items": 2, 2024-03-16T12:08:50.984 INFO:teuthology.orchestra.run.smithi154.stdout: "unittest_1_bytes": 0, 2024-03-16T12:08:50.984 INFO:teuthology.orchestra.run.smithi154.stdout: "unittest_1_items": 0, 2024-03-16T12:08:50.984 INFO:teuthology.orchestra.run.smithi154.stdout: "unittest_2_bytes": 0, 2024-03-16T12:08:50.985 INFO:teuthology.orchestra.run.smithi154.stdout: "unittest_2_items": 0 2024-03-16T12:08:50.985 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-16T12:08:50.985 INFO:teuthology.orchestra.run.smithi154.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-16T12:08:50.985 INFO:teuthology.orchestra.run.smithi154.stdout: "val": 0, 2024-03-16T12:08:50.985 INFO:teuthology.orchestra.run.smithi154.stdout: "max": 104857600, 2024-03-16T12:08:50.985 INFO:teuthology.orchestra.run.smithi154.stdout: "get_started": 0, 2024-03-16T12:08:50.985 INFO:teuthology.orchestra.run.smithi154.stdout: "get": 346, 2024-03-16T12:08:50.985 INFO:teuthology.orchestra.run.smithi154.stdout: "get_sum": 1423722, 2024-03-16T12:08:50.985 INFO:teuthology.orchestra.run.smithi154.stdout: "get_or_fail_fail": 0, 2024-03-16T12:08:50.986 INFO:teuthology.orchestra.run.smithi154.stdout: "get_or_fail_success": 346, 2024-03-16T12:08:50.986 INFO:teuthology.orchestra.run.smithi154.stdout: "take": 0, 2024-03-16T12:08:50.986 INFO:teuthology.orchestra.run.smithi154.stdout: "take_sum": 0, 2024-03-16T12:08:50.986 INFO:teuthology.orchestra.run.smithi154.stdout: "put": 346, 2024-03-16T12:08:50.986 INFO:teuthology.orchestra.run.smithi154.stdout: "put_sum": 1423722, 2024-03-16T12:08:50.986 INFO:teuthology.orchestra.run.smithi154.stdout: "wait": { 2024-03-16T12:08:50.986 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-16T12:08:50.986 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-16T12:08:50.986 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-16T12:08:50.987 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-03-16T12:08:50.987 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-03-16T12:08:50.987 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-03-16T12:08:51.757 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.l 2024-03-16T12:08:51.757 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.l 2024-03-16T12:08:51.757 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.l 2024-03-16T12:08:51.757 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.l 2024-03-16T12:08:51.758 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.l 2024-03-16T12:08:51.758 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.l 2024-03-16T12:08:51.758 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.l 2024-03-16T12:08:51.758 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2024-03-16T12:08:51.769 INFO:tasks.kclient:Unmounting kernel clients... 2024-03-16T12:08:51.769 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-03-16T12:08:51.769 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-16T12:08:51.791 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-03-16T12:08:51.791 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-16T12:08:51.850 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-16T12:08:52.008 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:52.017+0000 7fca3f1a9640 1 -- 172.21.15.29:0/3227959840 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fca381077e0 msgr2=0x7fca38107bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:08:52.008 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:52.017+0000 7fca3f1a9640 1 --2- 172.21.15.29:0/3227959840 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fca381077e0 0x7fca38107bc0 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7fca34009900 tx=0x7fca3402bcd0 comp rx=0 tx=0).stop 2024-03-16T12:08:52.008 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:52.017+0000 7fca3f1a9640 1 -- 172.21.15.29:0/3227959840 shutdown_connections 2024-03-16T12:08:52.009 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:52.017+0000 7fca3f1a9640 1 --2- 172.21.15.29:0/3227959840 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fca3806a830 0x7fca3810e630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:08:52.009 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:52.017+0000 7fca3f1a9640 1 --2- 172.21.15.29:0/3227959840 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fca38069e90 0x7fca3806a2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:08:52.009 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:52.017+0000 7fca3f1a9640 1 --2- 172.21.15.29:0/3227959840 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fca381077e0 0x7fca38107bc0 unknown :-1 s=CLOSED pgs=799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:08:52.009 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:52.017+0000 7fca3f1a9640 1 -- 172.21.15.29:0/3227959840 >> 172.21.15.29:0/3227959840 conn(0x7fca38077780 msgr2=0x7fca38077b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:08:52.009 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:52.017+0000 7fca3f1a9640 1 -- 172.21.15.29:0/3227959840 shutdown_connections 2024-03-16T12:08:52.009 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:52.017+0000 7fca3f1a9640 1 -- 172.21.15.29:0/3227959840 wait complete. 2024-03-16T12:08:52.009 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:52.017+0000 7fca3f1a9640 1 Processor -- start 2024-03-16T12:08:52.009 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:52.017+0000 7fca3f1a9640 1 -- start start 2024-03-16T12:08:52.009 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:52.018+0000 7fca3f1a9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fca38069e90 0x7fca381980c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:08:52.010 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:52.018+0000 7fca3f1a9640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fca3806a830 0x7fca38198600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:08:52.010 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:52.018+0000 7fca3f1a9640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fca381077e0 0x7fca3819c930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:08:52.010 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:52.018+0000 7fca3f1a9640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fca38111ba0 con 0x7fca38069e90 2024-03-16T12:08:52.010 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:52.018+0000 7fca3f1a9640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fca38111a20 con 0x7fca3806a830 2024-03-16T12:08:52.010 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:52.018+0000 7fca3f1a9640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fca38111d20 con 0x7fca381077e0 2024-03-16T12:08:52.010 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:52.018+0000 7fca3cf45640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fca38069e90 0x7fca381980c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:08:52.010 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:52.018+0000 7fca3cf45640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fca38069e90 0x7fca381980c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60296/0 (socket says 172.21.15.29:60296) 2024-03-16T12:08:52.010 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:52.018+0000 7fca3cf45640 1 -- 172.21.15.29:0/1232200700 learned_addr learned my addr 172.21.15.29:0/1232200700 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T12:08:52.010 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:52.018+0000 7fca2ffff640 1 --2- 172.21.15.29:0/1232200700 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fca3806a830 0x7fca38198600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:08:52.011 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:52.018+0000 7fca3d746640 1 --2- 172.21.15.29:0/1232200700 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fca381077e0 0x7fca3819c930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:08:52.011 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:52.018+0000 7fca3cf45640 1 -- 172.21.15.29:0/1232200700 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fca381077e0 msgr2=0x7fca3819c930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:08:52.011 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:52.018+0000 7fca3cf45640 1 --2- 172.21.15.29:0/1232200700 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fca381077e0 0x7fca3819c930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:08:52.011 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:52.018+0000 7fca3cf45640 1 -- 172.21.15.29:0/1232200700 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fca3806a830 msgr2=0x7fca38198600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:08:52.011 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:52.018+0000 7fca3cf45640 1 --2- 172.21.15.29:0/1232200700 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fca3806a830 0x7fca38198600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:08:52.011 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:52.018+0000 7fca3cf45640 1 -- 172.21.15.29:0/1232200700 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca3819d060 con 0x7fca38069e90 2024-03-16T12:08:52.011 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:52.018+0000 7fca3cf45640 1 --2- 172.21.15.29:0/1232200700 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fca38069e90 0x7fca381980c0 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7fca34005940 tx=0x7fca3402cbd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:08:52.011 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:52.018+0000 7fca2dffb640 1 -- 172.21.15.29:0/1232200700 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca34041070 con 0x7fca38069e90 2024-03-16T12:08:52.011 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:52.018+0000 7fca3f1a9640 1 -- 172.21.15.29:0/1232200700 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fca3819d2f0 con 0x7fca38069e90 2024-03-16T12:08:52.012 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:52.018+0000 7fca2dffb640 1 -- 172.21.15.29:0/1232200700 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fca34004450 con 0x7fca38069e90 2024-03-16T12:08:52.012 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:52.018+0000 7fca2dffb640 1 -- 172.21.15.29:0/1232200700 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca34003d80 con 0x7fca38069e90 2024-03-16T12:08:52.012 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:52.019+0000 7fca3f1a9640 1 -- 172.21.15.29:0/1232200700 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fca381a4be0 con 0x7fca38069e90 2024-03-16T12:08:52.013 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:52.020+0000 7fca3f1a9640 1 -- 172.21.15.29:0/1232200700 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fca38102f00 con 0x7fca38069e90 2024-03-16T12:08:52.013 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:52.021+0000 7fca2dffb640 1 -- 172.21.15.29:0/1232200700 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7fca34044070 con 0x7fca38069e90 2024-03-16T12:08:52.013 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:52.021+0000 7fca2dffb640 1 --2- 172.21.15.29:0/1232200700 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7fca200a2970 0x7fca200a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:08:52.013 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:52.022+0000 7fca2dffb640 1 -- 172.21.15.29:0/1232200700 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 8556+0+0 (secure 0 0 0) 0x7fca340f0a70 con 0x7fca38069e90 2024-03-16T12:08:52.013 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:52.022+0000 7fca2ffff640 1 --2- 172.21.15.29:0/1232200700 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7fca200a2970 0x7fca200a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:08:52.013 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:52.022+0000 7fca2ffff640 1 --2- 172.21.15.29:0/1232200700 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7fca200a2970 0x7fca200a4e30 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fca38077d60 tx=0x7fca28005f90 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:08:52.014 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:52.025+0000 7fca2dffb640 1 -- 172.21.15.29:0/1232200700 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fca340bd210 con 0x7fca38069e90 2024-03-16T12:08:52.200 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:52.211+0000 7fca3f1a9640 1 -- 172.21.15.29:0/1232200700 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fca38109c80 con 0x7fca38069e90 2024-03-16T12:08:52.202 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:52.213+0000 7fca2dffb640 1 -- 172.21.15.29:0/1232200700 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v116) ==== 74+0+23200 (secure 0 0 0) 0x7fca340c12b0 con 0x7fca38069e90 2024-03-16T12:08:52.202 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-16T12:08:52.203 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":116,"fsid":"d1887f28-e389-11ee-95c9-87774f69a715","created":"2024-03-16T11:40:16.475063+0000","modified":"2024-03-16T12:08:22.177679+0000","last_up_change":"2024-03-16T11:47:44.349348+0000","last_in_change":"2024-03-16T11:47:23.672432+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-16T11:43:51.892652+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-16T11:48:36.709732+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-16T11:48:37.586140+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"115","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":30,"snap_epoch":115,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"60f5608f-a408-45ba-bdf6-5b53758f88fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":2748947913},{"type":"v1","addr":"172.21.15.29:6803","nonce":2748947913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":2748947913},{"type":"v1","addr":"172.21.15.29:6805","nonce":2748947913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":2748947913},{"type":"v1","addr":"172.21.15.29:6809","nonce":2748947913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":2748947913},{"type":"v1","addr":"172.21.15.29:6807","nonce":2748947913}]},"public_addr":"172.21.15.29:6803/2748947913","cluster_addr":"172.21.15.29:6805/2748947913","heartbeat_back_addr":"172.21.15.29:6809/2748947913","heartbeat_front_addr":"172.21.15.29:6807/2748947913","state":["exists","up"]},{"osd":1,"uuid":"e70b53d6-ee78-4e56-9a2c-263ae0501b43","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6800","nonce":2339408422},{"type":"v1","addr":"172.21.15.152:6801","nonce":2339408422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6802","nonce":2339408422},{"type":"v1","addr":"172.21.15.152:6803","nonce":2339408422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6806","nonce":2339408422},{"type":"v1","addr":"172.21.15.152:6807","nonce":2339408422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6804","nonce":2339408422},{"type":"v1","addr":"172.21.15.152:6805","nonce":2339408422}]},"public_addr":"172.21.15.152:6801/2339408422","cluster_addr":"172.21.15.152:6803/2339408422","heartbeat_back_addr":"172.21.15.152:6807/2339408422","heartbeat_front_addr":"172.21.15.152:6805/2339408422","state":["exists","up"]},{"osd":2,"uuid":"ff39389a-dfac-44b5-80ff-36abca316e78","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6800","nonce":4288325280},{"type":"v1","addr":"172.21.15.154:6801","nonce":4288325280}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6802","nonce":4288325280},{"type":"v1","addr":"172.21.15.154:6803","nonce":4288325280}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6806","nonce":4288325280},{"type":"v1","addr":"172.21.15.154:6807","nonce":4288325280}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":4288325280},{"type":"v1","addr":"172.21.15.154:6805","nonce":4288325280}]},"public_addr":"172.21.15.154:6801/4288325280","cluster_addr":"172.21.15.154:6803/4288325280","heartbeat_back_addr":"172.21.15.154:6807/4288325280","heartbeat_front_addr":"172.21.15.154:6805/4288325280","state":["exists","up"]},{"osd":3,"uuid":"069e1795-55bc-4324-b94c-eb24ef2334c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":1538750397},{"type":"v1","addr":"172.21.15.29:6811","nonce":1538750397}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":1538750397},{"type":"v1","addr":"172.21.15.29:6813","nonce":1538750397}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":1538750397},{"type":"v1","addr":"172.21.15.29:6817","nonce":1538750397}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":1538750397},{"type":"v1","addr":"172.21.15.29:6815","nonce":1538750397}]},"public_addr":"172.21.15.29:6811/1538750397","cluster_addr":"172.21.15.29:6813/1538750397","heartbeat_back_addr":"172.21.15.29:6817/1538750397","heartbeat_front_addr":"172.21.15.29:6815/1538750397","state":["exists","up"]},{"osd":4,"uuid":"5a9d43a1-e8f5-4ab5-a441-afbc6ed4066e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6808","nonce":2795081750},{"type":"v1","addr":"172.21.15.152:6809","nonce":2795081750}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6810","nonce":2795081750},{"type":"v1","addr":"172.21.15.152:6811","nonce":2795081750}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6814","nonce":2795081750},{"type":"v1","addr":"172.21.15.152:6815","nonce":2795081750}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6812","nonce":2795081750},{"type":"v1","addr":"172.21.15.152:6813","nonce":2795081750}]},"public_addr":"172.21.15.152:6809/2795081750","cluster_addr":"172.21.15.152:6811/2795081750","heartbeat_back_addr":"172.21.15.152:6815/2795081750","heartbeat_front_addr":"172.21.15.152:6813/2795081750","state":["exists","up"]},{"osd":5,"uuid":"234de428-4ac7-44be-9a54-7b58331fdf8c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":1472686560},{"type":"v1","addr":"172.21.15.154:6809","nonce":1472686560}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6810","nonce":1472686560},{"type":"v1","addr":"172.21.15.154:6811","nonce":1472686560}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6814","nonce":1472686560},{"type":"v1","addr":"172.21.15.154:6815","nonce":1472686560}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6812","nonce":1472686560},{"type":"v1","addr":"172.21.15.154:6813","nonce":1472686560}]},"public_addr":"172.21.15.154:6809/1472686560","cluster_addr":"172.21.15.154:6811/1472686560","heartbeat_back_addr":"172.21.15.154:6815/1472686560","heartbeat_front_addr":"172.21.15.154:6813/1472686560","state":["exists","up"]},{"osd":6,"uuid":"ae45b8d9-7883-40c9-a9cd-4c92b601d4f2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":816506244},{"type":"v1","addr":"172.21.15.29:6819","nonce":816506244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":816506244},{"type":"v1","addr":"172.21.15.29:6821","nonce":816506244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":816506244},{"type":"v1","addr":"172.21.15.29:6825","nonce":816506244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":816506244},{"type":"v1","addr":"172.21.15.29:6823","nonce":816506244}]},"public_addr":"172.21.15.29:6819/816506244","cluster_addr":"172.21.15.29:6821/816506244","heartbeat_back_addr":"172.21.15.29:6825/816506244","heartbeat_front_addr":"172.21.15.29:6823/816506244","state":["exists","up"]},{"osd":7,"uuid":"6f687b28-4fce-4d4a-8b4e-294173d9d6c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6816","nonce":2924651787},{"type":"v1","addr":"172.21.15.152:6817","nonce":2924651787}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6818","nonce":2924651787},{"type":"v1","addr":"172.21.15.152:6819","nonce":2924651787}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6822","nonce":2924651787},{"type":"v1","addr":"172.21.15.152:6823","nonce":2924651787}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6820","nonce":2924651787},{"type":"v1","addr":"172.21.15.152:6821","nonce":2924651787}]},"public_addr":"172.21.15.152:6817/2924651787","cluster_addr":"172.21.15.152:6819/2924651787","heartbeat_back_addr":"172.21.15.152:6823/2924651787","heartbeat_front_addr":"172.21.15.152:6821/2924651787","state":["exists","up"]},{"osd":8,"uuid":"a25fb741-caa7-429a-b540-e394d034b446","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6816","nonce":129974156},{"type":"v1","addr":"172.21.15.154:6817","nonce":129974156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6818","nonce":129974156},{"type":"v1","addr":"172.21.15.154:6819","nonce":129974156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6822","nonce":129974156},{"type":"v1","addr":"172.21.15.154:6823","nonce":129974156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6820","nonce":129974156},{"type":"v1","addr":"172.21.15.154:6821","nonce":129974156}]},"public_addr":"172.21.15.154:6817/129974156","cluster_addr":"172.21.15.154:6819/129974156","heartbeat_back_addr":"172.21.15.154:6823/129974156","heartbeat_front_addr":"172.21.15.154:6821/129974156","state":["exists","up"]},{"osd":9,"uuid":"1071cb0e-8c5f-4cd8-9805-0045e220d946","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6826","nonce":2951414243},{"type":"v1","addr":"172.21.15.29:6827","nonce":2951414243}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":2951414243},{"type":"v1","addr":"172.21.15.29:6829","nonce":2951414243}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":2951414243},{"type":"v1","addr":"172.21.15.29:6833","nonce":2951414243}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":2951414243},{"type":"v1","addr":"172.21.15.29:6831","nonce":2951414243}]},"public_addr":"172.21.15.29:6827/2951414243","cluster_addr":"172.21.15.29:6829/2951414243","heartbeat_back_addr":"172.21.15.29:6833/2951414243","heartbeat_front_addr":"172.21.15.29:6831/2951414243","state":["exists","up"]},{"osd":10,"uuid":"ee7c7d74-3f39-4cd4-be80-843fb1dd9ecc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6824","nonce":3412525480},{"type":"v1","addr":"172.21.15.152:6825","nonce":3412525480}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6826","nonce":3412525480},{"type":"v1","addr":"172.21.15.152:6827","nonce":3412525480}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6830","nonce":3412525480},{"type":"v1","addr":"172.21.15.152:6831","nonce":3412525480}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6828","nonce":3412525480},{"type":"v1","addr":"172.21.15.152:6829","nonce":3412525480}]},"public_addr":"172.21.15.152:6825/3412525480","cluster_addr":"172.21.15.152:6827/3412525480","heartbeat_back_addr":"172.21.15.152:6831/3412525480","heartbeat_front_addr":"172.21.15.152:6829/3412525480","state":["exists","up"]},{"osd":11,"uuid":"4598c803-984a-4de6-ac22-066502c17889","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6824","nonce":1160927966},{"type":"v1","addr":"172.21.15.154:6825","nonce":1160927966}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6826","nonce":1160927966},{"type":"v1","addr":"172.21.15.154:6827","nonce":1160927966}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6830","nonce":1160927966},{"type":"v1","addr":"172.21.15.154:6831","nonce":1160927966}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6828","nonce":1160927966},{"type":"v1","addr":"172.21.15.154:6829","nonce":1160927966}]},"public_addr":"172.21.15.154:6825/1160927966","cluster_addr":"172.21.15.154:6827/1160927966","heartbeat_back_addr":"172.21.15.154:6831/1160927966","heartbeat_front_addr":"172.21.15.154:6829/1160927966","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:43:02.343393+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:43:25.209151+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:43:48.946164+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:44:14.588424+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:44:39.700025+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:45:04.061197+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:45:31.214508+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:45:56.937296+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:46:22.768915+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:46:50.101011+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:47:16.298041+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:47:42.402095+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"2.1b","mappings":[{"from":2,"to":1}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"3.33","mappings":[{"from":5,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.29:0/1361425453":"2024-03-16T12:55:48.830413+0000","172.21.15.29:0/1116486932":"2024-03-17T11:41:27.531724+0000","172.21.15.29:0/1784249039":"2024-03-17T11:41:00.505678+0000","172.21.15.29:6800/4192064640":"2024-03-17T11:50:54.850617+0000","172.21.15.29:0/1330100009":"2024-03-17T11:41:00.505678+0000","172.21.15.29:6801/2275315654":"2024-03-17T11:41:00.505678+0000","172.21.15.29:6801/2798831128":"2024-03-17T11:41:27.531724+0000","172.21.15.29:6800/2275315654":"2024-03-17T11:41:00.505678+0000","172.21.15.29:6800/2798831128":"2024-03-17T11:41:27.531724+0000","172.21.15.29:0/1211659364":"2024-03-17T11:41:27.531724+0000","172.21.15.29:0/1281028285":"2024-03-17T11:41:00.505678+0000","172.21.15.29:0/1853086886":"2024-03-17T11:41:27.531724+0000","172.21.15.29:0/623985488":"2024-03-17T11:50:54.850617+0000","172.21.15.29:6801/4192064640":"2024-03-17T11:50:54.850617+0000","172.21.15.29:0/1114950131":"2024-03-17T11:50:54.850617+0000","172.21.15.29:0/2374713213":"2024-03-17T11:50:54.850617+0000","172.21.15.29:0/340047334":"2024-03-17T11:50:54.850617+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":3,"snaps":[{"begin":18,"length":1},{"begin":30,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-16T12:08:52.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:52.215+0000 7fca3f1a9640 1 -- 172.21.15.29:0/1232200700 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7fca200a2970 msgr2=0x7fca200a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:08:52.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:52.215+0000 7fca3f1a9640 1 --2- 172.21.15.29:0/1232200700 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7fca200a2970 0x7fca200a4e30 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fca38077d60 tx=0x7fca28005f90 comp rx=0 tx=0).stop 2024-03-16T12:08:52.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:52.215+0000 7fca3f1a9640 1 -- 172.21.15.29:0/1232200700 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fca38069e90 msgr2=0x7fca381980c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:08:52.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:52.215+0000 7fca3f1a9640 1 --2- 172.21.15.29:0/1232200700 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fca38069e90 0x7fca381980c0 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7fca34005940 tx=0x7fca3402cbd0 comp rx=0 tx=0).stop 2024-03-16T12:08:52.205 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:52.215+0000 7fca3f1a9640 1 -- 172.21.15.29:0/1232200700 shutdown_connections 2024-03-16T12:08:52.205 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:52.215+0000 7fca3f1a9640 1 --2- 172.21.15.29:0/1232200700 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7fca200a2970 0x7fca200a4e30 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:08:52.205 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:52.215+0000 7fca3f1a9640 1 --2- 172.21.15.29:0/1232200700 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fca381077e0 0x7fca3819c930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:08:52.205 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:52.215+0000 7fca3f1a9640 1 --2- 172.21.15.29:0/1232200700 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fca3806a830 0x7fca38198600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:08:52.205 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:52.215+0000 7fca3f1a9640 1 --2- 172.21.15.29:0/1232200700 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fca38069e90 0x7fca381980c0 unknown :-1 s=CLOSED pgs=800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:08:52.205 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:52.215+0000 7fca3f1a9640 1 -- 172.21.15.29:0/1232200700 >> 172.21.15.29:0/1232200700 conn(0x7fca38077780 msgr2=0x7fca380ff300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:08:52.205 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:52.215+0000 7fca3f1a9640 1 -- 172.21.15.29:0/1232200700 shutdown_connections 2024-03-16T12:08:52.205 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:52.215+0000 7fca3f1a9640 1 -- 172.21.15.29:0/1232200700 wait complete. 2024-03-16T12:08:52.215 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-16T11:43:51.892652+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-16T11:48:36.709732+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-16T11:48:37.586140+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '115', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 30, 'snap_epoch': 115, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-16T12:08:52.216 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-16T12:08:52.324 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:52.333+0000 7f7a32626640 1 -- 172.21.15.29:0/152125049 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7a2c106660 msgr2=0x7f7a2c106a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:08:52.324 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:52.333+0000 7f7a32626640 1 --2- 172.21.15.29:0/152125049 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7a2c106660 0x7f7a2c106a40 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7f7a1c009930 tx=0x7f7a1c02bc30 comp rx=0 tx=0).stop 2024-03-16T12:08:52.324 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:52.333+0000 7f7a32626640 1 -- 172.21.15.29:0/152125049 shutdown_connections 2024-03-16T12:08:52.324 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:52.333+0000 7f7a32626640 1 --2- 172.21.15.29:0/152125049 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7a2c101010 0x7f7a2c10d520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:08:52.325 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:52.333+0000 7f7a32626640 1 --2- 172.21.15.29:0/152125049 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7a2c100650 0x7f7a2c100ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:08:52.325 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:52.333+0000 7f7a32626640 1 --2- 172.21.15.29:0/152125049 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7a2c106660 0x7f7a2c106a40 unknown :-1 s=CLOSED pgs=801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:08:52.325 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:52.333+0000 7f7a32626640 1 -- 172.21.15.29:0/152125049 >> 172.21.15.29:0/152125049 conn(0x7f7a2c0777e0 msgr2=0x7f7a2c077bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:08:52.325 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:52.333+0000 7f7a32626640 1 -- 172.21.15.29:0/152125049 shutdown_connections 2024-03-16T12:08:52.325 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:52.333+0000 7f7a32626640 1 -- 172.21.15.29:0/152125049 wait complete. 2024-03-16T12:08:52.325 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:52.333+0000 7f7a32626640 1 Processor -- start 2024-03-16T12:08:52.325 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:52.333+0000 7f7a32626640 1 -- start start 2024-03-16T12:08:52.326 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:52.334+0000 7f7a32626640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7a2c100650 0x7f7a2c198060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:08:52.326 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:52.334+0000 7f7a32626640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7a2c101010 0x7f7a2c1985a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:08:52.326 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:52.334+0000 7f7a32626640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7a2c106660 0x7f7a2c19c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:08:52.326 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:52.334+0000 7f7a32626640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f7a2c110a30 con 0x7f7a2c100650 2024-03-16T12:08:52.326 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:52.334+0000 7f7a32626640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f7a2c1108b0 con 0x7f7a2c101010 2024-03-16T12:08:52.326 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:52.334+0000 7f7a32626640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f7a2c110bb0 con 0x7f7a2c106660 2024-03-16T12:08:52.327 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:52.334+0000 7f7a2bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7a2c100650 0x7f7a2c198060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:08:52.327 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:52.334+0000 7f7a2bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7a2c100650 0x7f7a2c198060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60322/0 (socket says 172.21.15.29:60322) 2024-03-16T12:08:52.327 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:52.334+0000 7f7a2bfff640 1 -- 172.21.15.29:0/2474694823 learned_addr learned my addr 172.21.15.29:0/2474694823 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T12:08:52.327 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:52.334+0000 7f7a2b7fe640 1 --2- 172.21.15.29:0/2474694823 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7a2c101010 0x7f7a2c1985a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:08:52.327 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:52.334+0000 7f7a30bc3640 1 --2- 172.21.15.29:0/2474694823 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7a2c106660 0x7f7a2c19c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:08:52.327 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:52.334+0000 7f7a2bfff640 1 -- 172.21.15.29:0/2474694823 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7a2c106660 msgr2=0x7f7a2c19c920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:08:52.327 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:52.334+0000 7f7a2bfff640 1 --2- 172.21.15.29:0/2474694823 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7a2c106660 0x7f7a2c19c920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:08:52.327 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:52.334+0000 7f7a2bfff640 1 -- 172.21.15.29:0/2474694823 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7a2c101010 msgr2=0x7f7a2c1985a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:08:52.328 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:52.334+0000 7f7a2bfff640 1 --2- 172.21.15.29:0/2474694823 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7a2c101010 0x7f7a2c1985a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:08:52.328 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:52.334+0000 7f7a2bfff640 1 -- 172.21.15.29:0/2474694823 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a2c19d050 con 0x7f7a2c100650 2024-03-16T12:08:52.328 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:52.334+0000 7f7a2bfff640 1 --2- 172.21.15.29:0/2474694823 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7a2c100650 0x7f7a2c198060 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f7a1c009230 tx=0x7f7a1c02cb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:08:52.328 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:52.334+0000 7f7a297fa640 1 -- 172.21.15.29:0/2474694823 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a1c041070 con 0x7f7a2c100650 2024-03-16T12:08:52.328 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:52.334+0000 7f7a297fa640 1 -- 172.21.15.29:0/2474694823 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7a1c004450 con 0x7f7a2c100650 2024-03-16T12:08:52.328 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:52.334+0000 7f7a32626640 1 -- 172.21.15.29:0/2474694823 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7a2c19d2e0 con 0x7f7a2c100650 2024-03-16T12:08:52.328 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:52.334+0000 7f7a297fa640 1 -- 172.21.15.29:0/2474694823 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a1c003e50 con 0x7f7a2c100650 2024-03-16T12:08:52.329 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:52.334+0000 7f7a32626640 1 -- 172.21.15.29:0/2474694823 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7a2c1a4bd0 con 0x7f7a2c100650 2024-03-16T12:08:52.329 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:52.335+0000 7f7a32626640 1 -- 172.21.15.29:0/2474694823 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7a2c0685d0 con 0x7f7a2c100650 2024-03-16T12:08:52.330 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:52.337+0000 7f7a297fa640 1 -- 172.21.15.29:0/2474694823 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7f7a1c044070 con 0x7f7a2c100650 2024-03-16T12:08:52.330 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:52.337+0000 7f7a297fa640 1 --2- 172.21.15.29:0/2474694823 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f7a140a2b10 0x7f7a140a4fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:08:52.330 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:52.337+0000 7f7a297fa640 1 -- 172.21.15.29:0/2474694823 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 8556+0+0 (secure 0 0 0) 0x7f7a1c0f0a40 con 0x7f7a2c100650 2024-03-16T12:08:52.330 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:52.337+0000 7f7a2b7fe640 1 --2- 172.21.15.29:0/2474694823 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f7a140a2b10 0x7f7a140a4fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:08:52.330 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:52.338+0000 7f7a2b7fe640 1 --2- 172.21.15.29:0/2474694823 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f7a140a2b10 0x7f7a140a4fd0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f7a2c1027f0 tx=0x7f7a20005f90 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:08:52.331 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:52.341+0000 7f7a297fa640 1 -- 172.21.15.29:0/2474694823 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f7a1c0bd240 con 0x7f7a2c100650 2024-03-16T12:08:52.514 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:52.525+0000 7f7a32626640 1 -- 172.21.15.29:0/2474694823 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f7a2c109bf0 con 0x7f7a2c100650 2024-03-16T12:08:52.514 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:52.525+0000 7f7a297fa640 1 -- 172.21.15.29:0/2474694823 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v116) ==== 93+0+10 (secure 0 0 0) 0x7f7a1c0c12e0 con 0x7f7a2c100650 2024-03-16T12:08:52.515 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2024-03-16T12:08:52.517 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:52.527+0000 7f7a32626640 1 -- 172.21.15.29:0/2474694823 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f7a140a2b10 msgr2=0x7f7a140a4fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:08:52.517 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:52.527+0000 7f7a32626640 1 --2- 172.21.15.29:0/2474694823 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f7a140a2b10 0x7f7a140a4fd0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f7a2c1027f0 tx=0x7f7a20005f90 comp rx=0 tx=0).stop 2024-03-16T12:08:52.517 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:52.527+0000 7f7a32626640 1 -- 172.21.15.29:0/2474694823 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7a2c100650 msgr2=0x7f7a2c198060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:08:52.517 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:52.527+0000 7f7a32626640 1 --2- 172.21.15.29:0/2474694823 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7a2c100650 0x7f7a2c198060 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f7a1c009230 tx=0x7f7a1c02cb30 comp rx=0 tx=0).stop 2024-03-16T12:08:52.517 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:52.527+0000 7f7a32626640 1 -- 172.21.15.29:0/2474694823 shutdown_connections 2024-03-16T12:08:52.517 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:52.527+0000 7f7a32626640 1 --2- 172.21.15.29:0/2474694823 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f7a140a2b10 0x7f7a140a4fd0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:08:52.517 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:52.527+0000 7f7a32626640 1 --2- 172.21.15.29:0/2474694823 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7a2c106660 0x7f7a2c19c920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:08:52.517 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:52.527+0000 7f7a32626640 1 --2- 172.21.15.29:0/2474694823 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7a2c101010 0x7f7a2c1985a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:08:52.518 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:52.527+0000 7f7a32626640 1 --2- 172.21.15.29:0/2474694823 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7a2c100650 0x7f7a2c198060 unknown :-1 s=CLOSED pgs=802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:08:52.518 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:52.527+0000 7f7a32626640 1 -- 172.21.15.29:0/2474694823 >> 172.21.15.29:0/2474694823 conn(0x7f7a2c0777e0 msgr2=0x7f7a2c068c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:08:52.518 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:52.527+0000 7f7a32626640 1 -- 172.21.15.29:0/2474694823 shutdown_connections 2024-03-16T12:08:52.518 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:52.527+0000 7f7a32626640 1 -- 172.21.15.29:0/2474694823 wait complete. 2024-03-16T12:08:52.527 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-16T12:08:52.634 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:52.644+0000 7fdfa3b90640 1 -- 172.21.15.29:0/3998830238 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdf9c103370 msgr2=0x7fdf9c10b850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:08:52.634 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:52.644+0000 7fdfa3b90640 1 --2- 172.21.15.29:0/3998830238 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdf9c103370 0x7fdf9c10b850 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7fdf8c009900 tx=0x7fdf8c02bcd0 comp rx=0 tx=0).stop 2024-03-16T12:08:52.634 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:52.644+0000 7fdfa3b90640 1 -- 172.21.15.29:0/3998830238 shutdown_connections 2024-03-16T12:08:52.634 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:52.644+0000 7fdfa3b90640 1 --2- 172.21.15.29:0/3998830238 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdf9c10be20 0x7fdf9c10e290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:08:52.634 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:52.644+0000 7fdfa3b90640 1 --2- 172.21.15.29:0/3998830238 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdf9c103370 0x7fdf9c10b850 unknown :-1 s=CLOSED pgs=803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:08:52.634 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:52.644+0000 7fdfa3b90640 1 --2- 172.21.15.29:0/3998830238 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fdf9c102a50 0x7fdf9c102e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:08:52.635 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:52.644+0000 7fdfa3b90640 1 -- 172.21.15.29:0/3998830238 >> 172.21.15.29:0/3998830238 conn(0x7fdf9c077840 msgr2=0x7fdf9c077c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:08:52.635 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:52.644+0000 7fdfa3b90640 1 -- 172.21.15.29:0/3998830238 shutdown_connections 2024-03-16T12:08:52.635 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:52.644+0000 7fdfa3b90640 1 -- 172.21.15.29:0/3998830238 wait complete. 2024-03-16T12:08:52.637 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:52.645+0000 7fdfa3b90640 1 Processor -- start 2024-03-16T12:08:52.637 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:52.645+0000 7fdfa3b90640 1 -- start start 2024-03-16T12:08:52.637 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:52.645+0000 7fdfa3b90640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fdf9c102a50 0x7fdf9c1980f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:08:52.637 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:52.645+0000 7fdfa3b90640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdf9c103370 0x7fdf9c198630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:08:52.637 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:52.645+0000 7fdfa3b90640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdf9c10be20 0x7fdf9c19c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:08:52.638 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:52.645+0000 7fdfa3b90640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fdf9c111bf0 con 0x7fdf9c10be20 2024-03-16T12:08:52.638 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:52.645+0000 7fdfa3b90640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fdf9c111a70 con 0x7fdf9c102a50 2024-03-16T12:08:52.638 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:52.645+0000 7fdfa3b90640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fdf9c111d70 con 0x7fdf9c103370 2024-03-16T12:08:52.638 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:52.645+0000 7fdfa212d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdf9c10be20 0x7fdf9c19c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:08:52.638 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:52.645+0000 7fdfa212d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdf9c10be20 0x7fdf9c19c9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60348/0 (socket says 172.21.15.29:60348) 2024-03-16T12:08:52.638 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:52.645+0000 7fdfa212d640 1 -- 172.21.15.29:0/783423762 learned_addr learned my addr 172.21.15.29:0/783423762 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T12:08:52.638 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:52.645+0000 7fdfa192c640 1 --2- 172.21.15.29:0/783423762 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fdf9c102a50 0x7fdf9c1980f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:08:52.638 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:52.645+0000 7fdfa112b640 1 --2- 172.21.15.29:0/783423762 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdf9c103370 0x7fdf9c198630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:08:52.639 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:52.645+0000 7fdfa212d640 1 -- 172.21.15.29:0/783423762 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdf9c103370 msgr2=0x7fdf9c198630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:08:52.639 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:52.645+0000 7fdfa212d640 1 --2- 172.21.15.29:0/783423762 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdf9c103370 0x7fdf9c198630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:08:52.639 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:52.645+0000 7fdfa212d640 1 -- 172.21.15.29:0/783423762 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fdf9c102a50 msgr2=0x7fdf9c1980f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:08:52.639 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:52.645+0000 7fdfa212d640 1 --2- 172.21.15.29:0/783423762 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fdf9c102a50 0x7fdf9c1980f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:08:52.639 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:52.645+0000 7fdfa212d640 1 -- 172.21.15.29:0/783423762 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf9c19d0e0 con 0x7fdf9c10be20 2024-03-16T12:08:52.639 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:52.646+0000 7fdfa112b640 1 --2- 172.21.15.29:0/783423762 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdf9c103370 0x7fdf9c198630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:08:52.639 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:52.646+0000 7fdfa212d640 1 --2- 172.21.15.29:0/783423762 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdf9c10be20 0x7fdf9c19c9b0 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7fdf9400efd0 tx=0x7fdf9400c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:08:52.639 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:52.646+0000 7fdf92ffd640 1 -- 172.21.15.29:0/783423762 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf94003b70 con 0x7fdf9c10be20 2024-03-16T12:08:52.640 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:52.646+0000 7fdf92ffd640 1 -- 172.21.15.29:0/783423762 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdf94003d10 con 0x7fdf9c10be20 2024-03-16T12:08:52.640 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:52.646+0000 7fdfa3b90640 1 -- 172.21.15.29:0/783423762 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdf9c19d3d0 con 0x7fdf9c10be20 2024-03-16T12:08:52.640 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:52.646+0000 7fdf92ffd640 1 -- 172.21.15.29:0/783423762 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf940109f0 con 0x7fdf9c10be20 2024-03-16T12:08:52.640 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:52.646+0000 7fdfa3b90640 1 -- 172.21.15.29:0/783423762 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdf9c1a4c60 con 0x7fdf9c10be20 2024-03-16T12:08:52.640 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:52.647+0000 7fdfa3b90640 1 -- 172.21.15.29:0/783423762 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdf9c068a20 con 0x7fdf9c10be20 2024-03-16T12:08:52.640 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:52.648+0000 7fdf92ffd640 1 -- 172.21.15.29:0/783423762 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7fdf94010b90 con 0x7fdf9c10be20 2024-03-16T12:08:52.641 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:52.649+0000 7fdf92ffd640 1 --2- 172.21.15.29:0/783423762 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7fdf880a2970 0x7fdf880a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:08:52.641 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:52.649+0000 7fdfa192c640 1 --2- 172.21.15.29:0/783423762 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7fdf880a2970 0x7fdf880a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:08:52.641 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:52.649+0000 7fdf92ffd640 1 -- 172.21.15.29:0/783423762 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 8556+0+0 (secure 0 0 0) 0x7fdf940cf610 con 0x7fdf9c10be20 2024-03-16T12:08:52.641 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:52.649+0000 7fdfa192c640 1 --2- 172.21.15.29:0/783423762 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7fdf880a2970 0x7fdf880a4e30 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fdf98009bd0 tx=0x7fdf98005f90 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:08:52.642 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:52.653+0000 7fdf92ffd640 1 -- 172.21.15.29:0/783423762 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fdf9409c2f0 con 0x7fdf9c10be20 2024-03-16T12:08:52.785 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:08:52 smithi154 ceph-mon[36320]: pgmap v569: 97 pgs: 97 active+clean; 362 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 318 KiB/s wr, 16 op/s 2024-03-16T12:08:52.785 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:08:52 smithi154 ceph-mon[36320]: from='client.? 172.21.15.29:0/1232200700' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T12:08:52.824 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:52.835+0000 7fdfa3b90640 1 -- 172.21.15.29:0/783423762 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fdf9c106980 con 0x7fdf9c10be20 2024-03-16T12:08:52.825 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:52.835+0000 7fdf92ffd640 1 -- 172.21.15.29:0/783423762 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v116) ==== 104+0+11 (secure 0 0 0) 0x7fdf9409bbf0 con 0x7fdf9c10be20 2024-03-16T12:08:52.825 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 32 2024-03-16T12:08:52.827 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:52.837+0000 7fdfa3b90640 1 -- 172.21.15.29:0/783423762 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7fdf880a2970 msgr2=0x7fdf880a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:08:52.827 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:52.837+0000 7fdfa3b90640 1 --2- 172.21.15.29:0/783423762 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7fdf880a2970 0x7fdf880a4e30 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fdf98009bd0 tx=0x7fdf98005f90 comp rx=0 tx=0).stop 2024-03-16T12:08:52.827 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:52.837+0000 7fdfa3b90640 1 -- 172.21.15.29:0/783423762 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdf9c10be20 msgr2=0x7fdf9c19c9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:08:52.827 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:52.837+0000 7fdfa3b90640 1 --2- 172.21.15.29:0/783423762 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdf9c10be20 0x7fdf9c19c9b0 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7fdf9400efd0 tx=0x7fdf9400c6a0 comp rx=0 tx=0).stop 2024-03-16T12:08:52.827 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:52.837+0000 7fdfa3b90640 1 -- 172.21.15.29:0/783423762 shutdown_connections 2024-03-16T12:08:52.827 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:52.837+0000 7fdfa3b90640 1 --2- 172.21.15.29:0/783423762 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7fdf880a2970 0x7fdf880a4e30 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:08:52.827 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:52.837+0000 7fdfa3b90640 1 --2- 172.21.15.29:0/783423762 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdf9c10be20 0x7fdf9c19c9b0 unknown :-1 s=CLOSED pgs=804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:08:52.827 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:52.837+0000 7fdfa3b90640 1 --2- 172.21.15.29:0/783423762 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdf9c103370 0x7fdf9c198630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:08:52.828 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:52.837+0000 7fdfa3b90640 1 --2- 172.21.15.29:0/783423762 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fdf9c102a50 0x7fdf9c1980f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:08:52.828 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:52.837+0000 7fdfa3b90640 1 -- 172.21.15.29:0/783423762 >> 172.21.15.29:0/783423762 conn(0x7fdf9c077840 msgr2=0x7fdf9c0fd7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:08:52.828 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:52.837+0000 7fdfa3b90640 1 -- 172.21.15.29:0/783423762 shutdown_connections 2024-03-16T12:08:52.828 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:52.837+0000 7fdfa3b90640 1 -- 172.21.15.29:0/783423762 wait complete. 2024-03-16T12:08:52.828 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:08:52 smithi029 ceph-mon[31975]: pgmap v569: 97 pgs: 97 active+clean; 362 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 318 KiB/s wr, 16 op/s 2024-03-16T12:08:52.829 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:08:52 smithi029 ceph-mon[31975]: from='client.? 172.21.15.29:0/1232200700' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T12:08:52.838 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-16T12:08:52.867 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:08:52 smithi152 ceph-mon[36291]: pgmap v569: 97 pgs: 97 active+clean; 362 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 318 KiB/s wr, 16 op/s 2024-03-16T12:08:52.867 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:08:52 smithi152 ceph-mon[36291]: from='client.? 172.21.15.29:0/1232200700' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T12:08:52.944 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:52.954+0000 7f6485553640 1 -- 172.21.15.29:0/1568723967 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6480107830 msgr2=0x7f6480107c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:08:52.944 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:52.954+0000 7f6485553640 1 --2- 172.21.15.29:0/1568723967 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6480107830 0x7f6480107c10 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f64780098c0 tx=0x7f647802bb50 comp rx=0 tx=0).stop 2024-03-16T12:08:52.944 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:52.954+0000 7f6485553640 1 -- 172.21.15.29:0/1568723967 shutdown_connections 2024-03-16T12:08:52.945 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:52.954+0000 7f6485553640 1 --2- 172.21.15.29:0/1568723967 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f64800693a0 0x7f648010e640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:08:52.945 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:52.954+0000 7f6485553640 1 --2- 172.21.15.29:0/1568723967 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f64800689e0 0x7f6480068e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:08:52.945 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:52.954+0000 7f6485553640 1 --2- 172.21.15.29:0/1568723967 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6480107830 0x7f6480107c10 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:08:52.945 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:52.954+0000 7f6485553640 1 -- 172.21.15.29:0/1568723967 >> 172.21.15.29:0/1568723967 conn(0x7f64800777e0 msgr2=0x7f6480077bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:08:52.945 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:52.954+0000 7f6485553640 1 -- 172.21.15.29:0/1568723967 shutdown_connections 2024-03-16T12:08:52.945 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:52.954+0000 7f6485553640 1 -- 172.21.15.29:0/1568723967 wait complete. 2024-03-16T12:08:52.945 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:52.955+0000 7f6485553640 1 Processor -- start 2024-03-16T12:08:52.945 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:52.955+0000 7f6485553640 1 -- start start 2024-03-16T12:08:52.947 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:52.955+0000 7f6485553640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f64800689e0 0x7f6480198080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:08:52.947 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:52.955+0000 7f6485553640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f64800693a0 0x7f64801985c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:08:52.947 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:52.955+0000 7f6485553640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6480107830 0x7f648019c940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:08:52.947 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:52.955+0000 7f6485553640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f6480111b80 con 0x7f64800689e0 2024-03-16T12:08:52.948 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:52.955+0000 7f6485553640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f6480111a00 con 0x7f64800693a0 2024-03-16T12:08:52.948 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:52.955+0000 7f6485553640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f6480111d00 con 0x7f6480107830 2024-03-16T12:08:52.948 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:52.955+0000 7f647effd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f64800689e0 0x7f6480198080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:08:52.948 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:52.956+0000 7f647effd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f64800689e0 0x7f6480198080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60360/0 (socket says 172.21.15.29:60360) 2024-03-16T12:08:52.948 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:52.956+0000 7f647effd640 1 -- 172.21.15.29:0/543935505 learned_addr learned my addr 172.21.15.29:0/543935505 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T12:08:52.948 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:52.956+0000 7f647e7fc640 1 --2- 172.21.15.29:0/543935505 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f64800693a0 0x7f64801985c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:08:52.948 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:52.956+0000 7f647f7fe640 1 --2- 172.21.15.29:0/543935505 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6480107830 0x7f648019c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:08:52.948 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:52.956+0000 7f647effd640 1 -- 172.21.15.29:0/543935505 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6480107830 msgr2=0x7f648019c940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:08:52.948 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:52.956+0000 7f647effd640 1 --2- 172.21.15.29:0/543935505 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6480107830 0x7f648019c940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:08:52.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:52.956+0000 7f647effd640 1 -- 172.21.15.29:0/543935505 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f64800693a0 msgr2=0x7f64801985c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:08:52.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:52.956+0000 7f647effd640 1 --2- 172.21.15.29:0/543935505 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f64800693a0 0x7f64801985c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:08:52.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:52.956+0000 7f647effd640 1 -- 172.21.15.29:0/543935505 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f648019d070 con 0x7f64800689e0 2024-03-16T12:08:52.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:52.956+0000 7f647effd640 1 --2- 172.21.15.29:0/543935505 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f64800689e0 0x7f6480198080 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f647802ce90 tx=0x7f647802cec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:08:52.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:52.956+0000 7f6477fff640 1 -- 172.21.15.29:0/543935505 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6478043070 con 0x7f64800689e0 2024-03-16T12:08:52.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:52.956+0000 7f6485553640 1 -- 172.21.15.29:0/543935505 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f648019d300 con 0x7f64800689e0 2024-03-16T12:08:52.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:52.956+0000 7f6485553640 1 -- 172.21.15.29:0/543935505 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f64801a4bf0 con 0x7f64800689e0 2024-03-16T12:08:52.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:52.956+0000 7f6477fff640 1 -- 172.21.15.29:0/543935505 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6478005530 con 0x7f64800689e0 2024-03-16T12:08:52.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:52.956+0000 7f6477fff640 1 -- 172.21.15.29:0/543935505 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64780046c0 con 0x7f64800689e0 2024-03-16T12:08:52.950 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:52.957+0000 7f6485553640 1 -- 172.21.15.29:0/543935505 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f64800685d0 con 0x7f64800689e0 2024-03-16T12:08:52.950 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:52.959+0000 7f6477fff640 1 -- 172.21.15.29:0/543935505 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7f647802dd10 con 0x7f64800689e0 2024-03-16T12:08:52.950 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:52.959+0000 7f6477fff640 1 --2- 172.21.15.29:0/543935505 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f646c0a2970 0x7f646c0a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:08:52.951 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:52.960+0000 7f6477fff640 1 -- 172.21.15.29:0/543935505 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 8556+0+0 (secure 0 0 0) 0x7f64780f1000 con 0x7f64800689e0 2024-03-16T12:08:52.951 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:52.960+0000 7f647e7fc640 1 --2- 172.21.15.29:0/543935505 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f646c0a2970 0x7f646c0a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:08:52.951 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:52.960+0000 7f647e7fc640 1 --2- 172.21.15.29:0/543935505 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f646c0a2970 0x7f646c0a4e30 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f64800fd980 tx=0x7f6468009260 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:08:52.952 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:52.963+0000 7f6477fff640 1 -- 172.21.15.29:0/543935505 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f64780bdd40 con 0x7f64800689e0 2024-03-16T12:08:53.134 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:53.145+0000 7f6485553640 1 -- 172.21.15.29:0/543935505 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f6480109ce0 con 0x7f64800689e0 2024-03-16T12:08:53.135 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:53.146+0000 7f6477fff640 1 -- 172.21.15.29:0/543935505 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v116) ==== 100+0+11 (secure 0 0 0) 0x7f64780b9020 con 0x7f64800689e0 2024-03-16T12:08:53.135 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 64 2024-03-16T12:08:53.137 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:53.148+0000 7f6485553640 1 -- 172.21.15.29:0/543935505 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f646c0a2970 msgr2=0x7f646c0a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:08:53.138 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:53.148+0000 7f6485553640 1 --2- 172.21.15.29:0/543935505 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f646c0a2970 0x7f646c0a4e30 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f64800fd980 tx=0x7f6468009260 comp rx=0 tx=0).stop 2024-03-16T12:08:53.138 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:53.148+0000 7f6485553640 1 -- 172.21.15.29:0/543935505 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f64800689e0 msgr2=0x7f6480198080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:08:53.138 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:53.148+0000 7f6485553640 1 --2- 172.21.15.29:0/543935505 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f64800689e0 0x7f6480198080 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f647802ce90 tx=0x7f647802cec0 comp rx=0 tx=0).stop 2024-03-16T12:08:53.138 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:53.148+0000 7f6485553640 1 -- 172.21.15.29:0/543935505 shutdown_connections 2024-03-16T12:08:53.138 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:53.148+0000 7f6485553640 1 --2- 172.21.15.29:0/543935505 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f646c0a2970 0x7f646c0a4e30 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:08:53.138 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:53.148+0000 7f6485553640 1 --2- 172.21.15.29:0/543935505 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6480107830 0x7f648019c940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:08:53.138 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:53.148+0000 7f6485553640 1 --2- 172.21.15.29:0/543935505 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f64800693a0 0x7f64801985c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:08:53.139 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:53.148+0000 7f6485553640 1 --2- 172.21.15.29:0/543935505 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f64800689e0 0x7f6480198080 unknown :-1 s=CLOSED pgs=805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:08:53.139 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:53.148+0000 7f6485553640 1 -- 172.21.15.29:0/543935505 >> 172.21.15.29:0/543935505 conn(0x7f64800777e0 msgr2=0x7f64800fe2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:08:53.139 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:53.148+0000 7f6485553640 1 -- 172.21.15.29:0/543935505 shutdown_connections 2024-03-16T12:08:53.139 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:53.148+0000 7f6485553640 1 -- 172.21.15.29:0/543935505 wait complete. 2024-03-16T12:08:53.148 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-16T12:08:53.253 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:53.262+0000 7f957bf80640 1 -- 172.21.15.29:0/1510082961 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9574107780 msgr2=0x7f9574107b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:08:53.254 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:53.262+0000 7f957bf80640 1 --2- 172.21.15.29:0/1510082961 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9574107780 0x7f9574107b60 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7f9570009930 tx=0x7f957002bc30 comp rx=0 tx=0).stop 2024-03-16T12:08:53.254 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:53.262+0000 7f957bf80640 1 -- 172.21.15.29:0/1510082961 shutdown_connections 2024-03-16T12:08:53.254 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:53.262+0000 7f957bf80640 1 --2- 172.21.15.29:0/1510082961 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9574102130 0x7f957410e640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:08:53.254 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:53.262+0000 7f957bf80640 1 --2- 172.21.15.29:0/1510082961 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9574101770 0x7f9574101bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:08:53.254 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:53.262+0000 7f957bf80640 1 --2- 172.21.15.29:0/1510082961 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9574107780 0x7f9574107b60 unknown :-1 s=CLOSED pgs=806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:08:53.255 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:53.262+0000 7f957bf80640 1 -- 172.21.15.29:0/1510082961 >> 172.21.15.29:0/1510082961 conn(0x7f9574078be0 msgr2=0x7f9574078ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:08:53.255 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:53.262+0000 7f957bf80640 1 -- 172.21.15.29:0/1510082961 shutdown_connections 2024-03-16T12:08:53.255 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:53.262+0000 7f957bf80640 1 -- 172.21.15.29:0/1510082961 wait complete. 2024-03-16T12:08:53.255 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:53.263+0000 7f957bf80640 1 Processor -- start 2024-03-16T12:08:53.255 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:53.263+0000 7f957bf80640 1 -- start start 2024-03-16T12:08:53.255 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:53.263+0000 7f957bf80640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9574101770 0x7f9574073ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:08:53.255 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:53.263+0000 7f957bf80640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9574102130 0x7f9574074020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:08:53.255 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:53.263+0000 7f957bf80640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9574107780 0x7f9574074560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:08:53.255 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:53.263+0000 7f957bf80640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f9574111b50 con 0x7f9574102130 2024-03-16T12:08:53.256 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:53.263+0000 7f957bf80640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f95741119d0 con 0x7f9574107780 2024-03-16T12:08:53.256 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:53.263+0000 7f957bf80640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f9574111cd0 con 0x7f9574101770 2024-03-16T12:08:53.256 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:53.263+0000 7f957951b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9574102130 0x7f9574074020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:08:53.256 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:53.263+0000 7f957951b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9574102130 0x7f9574074020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60378/0 (socket says 172.21.15.29:60378) 2024-03-16T12:08:53.256 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:53.263+0000 7f957951b640 1 -- 172.21.15.29:0/2047743699 learned_addr learned my addr 172.21.15.29:0/2047743699 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T12:08:53.256 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:53.263+0000 7f9579d1c640 1 --2- 172.21.15.29:0/2047743699 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9574101770 0x7f9574073ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:08:53.256 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:53.263+0000 7f957a51d640 1 --2- 172.21.15.29:0/2047743699 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9574107780 0x7f9574074560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:08:53.257 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:53.263+0000 7f957951b640 1 -- 172.21.15.29:0/2047743699 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9574101770 msgr2=0x7f9574073ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:08:53.257 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:53.263+0000 7f957951b640 1 --2- 172.21.15.29:0/2047743699 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9574101770 0x7f9574073ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:08:53.257 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:53.263+0000 7f957951b640 1 -- 172.21.15.29:0/2047743699 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9574107780 msgr2=0x7f9574074560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:08:53.257 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:53.263+0000 7f957951b640 1 --2- 172.21.15.29:0/2047743699 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9574107780 0x7f9574074560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:08:53.257 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:53.263+0000 7f957951b640 1 -- 172.21.15.29:0/2047743699 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95741a6590 con 0x7f9574102130 2024-03-16T12:08:53.257 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:53.264+0000 7f957951b640 1 --2- 172.21.15.29:0/2047743699 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9574102130 0x7f9574074020 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f956400e9f0 tx=0x7f956400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:08:53.257 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:53.264+0000 7f956affd640 1 -- 172.21.15.29:0/2047743699 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f956400cd80 con 0x7f9574102130 2024-03-16T12:08:53.257 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:53.264+0000 7f957bf80640 1 -- 172.21.15.29:0/2047743699 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f95741a6880 con 0x7f9574102130 2024-03-16T12:08:53.258 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:53.264+0000 7f956affd640 1 -- 172.21.15.29:0/2047743699 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9564010910 con 0x7f9574102130 2024-03-16T12:08:53.258 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:53.264+0000 7f956affd640 1 -- 172.21.15.29:0/2047743699 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95640188f0 con 0x7f9574102130 2024-03-16T12:08:53.258 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:53.264+0000 7f957bf80640 1 -- 172.21.15.29:0/2047743699 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f95741a6e20 con 0x7f9574102130 2024-03-16T12:08:53.259 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:53.265+0000 7f957bf80640 1 -- 172.21.15.29:0/2047743699 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9574068a20 con 0x7f9574102130 2024-03-16T12:08:53.259 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:53.266+0000 7f956affd640 1 -- 172.21.15.29:0/2047743699 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7f9564010ac0 con 0x7f9574102130 2024-03-16T12:08:53.259 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:53.266+0000 7f956affd640 1 --2- 172.21.15.29:0/2047743699 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f95600a2970 0x7f95600a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:08:53.259 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:53.267+0000 7f9579d1c640 1 --2- 172.21.15.29:0/2047743699 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f95600a2970 0x7f95600a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:08:53.259 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:53.267+0000 7f956affd640 1 -- 172.21.15.29:0/2047743699 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 8556+0+0 (secure 0 0 0) 0x7f95640cf670 con 0x7f9574102130 2024-03-16T12:08:53.259 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:53.267+0000 7f9579d1c640 1 --2- 172.21.15.29:0/2047743699 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f95600a2970 0x7f95600a4e30 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f9570005ab0 tx=0x7f9570005fd0 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:08:53.260 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:53.270+0000 7f956affd640 1 -- 172.21.15.29:0/2047743699 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f956409bd00 con 0x7f9574102130 2024-03-16T12:08:53.456 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:53.466+0000 7f957bf80640 1 -- 172.21.15.29:0/2047743699 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f9574075010 con 0x7f9574102130 2024-03-16T12:08:53.456 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:53.467+0000 7f956affd640 1 -- 172.21.15.29:0/2047743699 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 17 entries v116) ==== 81+0+1002 (secure 0 0 0) 0x7f9574075010 con 0x7f9574102130 2024-03-16T12:08:53.456 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29:0/1361425453 2024-03-16T12:55:48.830413+0000 2024-03-16T12:08:53.457 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29:0/1116486932 2024-03-17T11:41:27.531724+0000 2024-03-16T12:08:53.457 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29:0/1784249039 2024-03-17T11:41:00.505678+0000 2024-03-16T12:08:53.457 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29:6800/4192064640 2024-03-17T11:50:54.850617+0000 2024-03-16T12:08:53.457 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29:0/1330100009 2024-03-17T11:41:00.505678+0000 2024-03-16T12:08:53.457 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29:6801/2275315654 2024-03-17T11:41:00.505678+0000 2024-03-16T12:08:53.457 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29:6801/2798831128 2024-03-17T11:41:27.531724+0000 2024-03-16T12:08:53.457 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29:6800/2275315654 2024-03-17T11:41:00.505678+0000 2024-03-16T12:08:53.457 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29:6800/2798831128 2024-03-17T11:41:27.531724+0000 2024-03-16T12:08:53.458 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29:0/1211659364 2024-03-17T11:41:27.531724+0000 2024-03-16T12:08:53.458 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29:0/1281028285 2024-03-17T11:41:00.505678+0000 2024-03-16T12:08:53.458 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29:0/1853086886 2024-03-17T11:41:27.531724+0000 2024-03-16T12:08:53.458 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29:0/623985488 2024-03-17T11:50:54.850617+0000 2024-03-16T12:08:53.458 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29:6801/4192064640 2024-03-17T11:50:54.850617+0000 2024-03-16T12:08:53.458 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29:0/1114950131 2024-03-17T11:50:54.850617+0000 2024-03-16T12:08:53.458 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29:0/2374713213 2024-03-17T11:50:54.850617+0000 2024-03-16T12:08:53.458 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29:0/340047334 2024-03-17T11:50:54.850617+0000 2024-03-16T12:08:53.459 INFO:teuthology.orchestra.run.smithi029.stderr:listed 17 entries 2024-03-16T12:08:53.459 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:53.468+0000 7f957bf80640 1 -- 172.21.15.29:0/2047743699 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f95600a2970 msgr2=0x7f95600a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:08:53.460 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:53.468+0000 7f957bf80640 1 --2- 172.21.15.29:0/2047743699 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f95600a2970 0x7f95600a4e30 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f9570005ab0 tx=0x7f9570005fd0 comp rx=0 tx=0).stop 2024-03-16T12:08:53.460 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:53.469+0000 7f957bf80640 1 -- 172.21.15.29:0/2047743699 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9574102130 msgr2=0x7f9574074020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:08:53.460 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:53.469+0000 7f957bf80640 1 --2- 172.21.15.29:0/2047743699 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9574102130 0x7f9574074020 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f956400e9f0 tx=0x7f956400eeb0 comp rx=0 tx=0).stop 2024-03-16T12:08:53.460 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:53.469+0000 7f957bf80640 1 -- 172.21.15.29:0/2047743699 shutdown_connections 2024-03-16T12:08:53.460 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:53.469+0000 7f957bf80640 1 --2- 172.21.15.29:0/2047743699 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f95600a2970 0x7f95600a4e30 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:08:53.460 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:53.469+0000 7f957bf80640 1 --2- 172.21.15.29:0/2047743699 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9574107780 0x7f9574074560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:08:53.460 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:53.469+0000 7f957bf80640 1 --2- 172.21.15.29:0/2047743699 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9574102130 0x7f9574074020 unknown :-1 s=CLOSED pgs=807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:08:53.460 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:53.469+0000 7f957bf80640 1 --2- 172.21.15.29:0/2047743699 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9574101770 0x7f9574073ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:08:53.460 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:53.469+0000 7f957bf80640 1 -- 172.21.15.29:0/2047743699 >> 172.21.15.29:0/2047743699 conn(0x7f9574078be0 msgr2=0x7f95741042c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:08:53.461 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:53.469+0000 7f957bf80640 1 -- 172.21.15.29:0/2047743699 shutdown_connections 2024-03-16T12:08:53.461 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:53.469+0000 7f957bf80640 1 -- 172.21.15.29:0/2047743699 wait complete. 2024-03-16T12:08:53.469 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-03-16T12:08:53.470 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-16T12:08:53.470 DEBUG:teuthology.orchestra.run.smithi029:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-03-16T12:08:53.786 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:08:53 smithi154 ceph-mon[36320]: from='client.? 172.21.15.29:0/2474694823' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-16T12:08:53.786 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:08:53 smithi154 ceph-mon[36320]: from='client.? 172.21.15.29:0/783423762' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-16T12:08:53.787 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:08:53 smithi154 ceph-mon[36320]: from='client.? 172.21.15.29:0/543935505' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-16T12:08:53.787 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:08:53 smithi154 ceph-mon[36320]: from='client.? 172.21.15.29:0/2047743699' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-16T12:08:53.828 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:08:53 smithi029 ceph-mon[31975]: from='client.? 172.21.15.29:0/2474694823' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-16T12:08:53.828 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:08:53 smithi029 ceph-mon[31975]: from='client.? 172.21.15.29:0/783423762' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-16T12:08:53.828 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:08:53 smithi029 ceph-mon[31975]: from='client.? 172.21.15.29:0/543935505' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-16T12:08:53.828 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:08:53 smithi029 ceph-mon[31975]: from='client.? 172.21.15.29:0/2047743699' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-16T12:08:53.867 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:08:53 smithi152 ceph-mon[36291]: from='client.? 172.21.15.29:0/2474694823' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-16T12:08:53.868 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:08:53 smithi152 ceph-mon[36291]: from='client.? 172.21.15.29:0/783423762' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-16T12:08:53.868 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:08:53 smithi152 ceph-mon[36291]: from='client.? 172.21.15.29:0/543935505' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-16T12:08:53.868 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:08:53 smithi152 ceph-mon[36291]: from='client.? 172.21.15.29:0/2047743699' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-16T12:08:53.996 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi029.front.sepia.ceph.com 2024-03-16T12:08:53.997 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-16T12:08:53.997 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-16T12:08:54.030 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-16T12:08:54.030 DEBUG:teuthology.orchestra.run.smithi029:> ip netns list 2024-03-16T12:08:54.087 INFO:teuthology.orchestra.run.smithi029.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-16T12:08:54.088 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-16T12:08:54.088 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-16T12:08:54.157 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-16T12:08:54.157 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip link delete ceph-brx 2024-03-16T12:08:54.246 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-03-16T12:08:54.256 INFO:tasks.cephadm:Teardown begin 2024-03-16T12:08:54.257 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-16T12:08:54.257 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T12:08:54.371 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:54.378+0000 7f2d0cb10640 1 -- 172.21.15.29:0/2612999081 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2d081010a0 msgr2=0x7f2d0810d5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:08:54.371 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:54.378+0000 7f2d0cb10640 1 --2- 172.21.15.29:0/2612999081 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2d081010a0 0x7f2d0810d5b0 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7f2d00009cd0 tx=0x7f2d0002c0f0 comp rx=0 tx=0).stop 2024-03-16T12:08:54.371 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:54.378+0000 7f2d0cb10640 1 -- 172.21.15.29:0/2612999081 shutdown_connections 2024-03-16T12:08:54.371 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:54.378+0000 7f2d0cb10640 1 --2- 172.21.15.29:0/2612999081 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2d081010a0 0x7f2d0810d5b0 unknown :-1 s=CLOSED pgs=808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:08:54.372 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:54.378+0000 7f2d0cb10640 1 --2- 172.21.15.29:0/2612999081 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2d081006e0 0x7f2d08100b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:08:54.372 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:54.378+0000 7f2d0cb10640 1 --2- 172.21.15.29:0/2612999081 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2d081066f0 0x7f2d08106ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:08:54.372 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:54.378+0000 7f2d0cb10640 1 -- 172.21.15.29:0/2612999081 >> 172.21.15.29:0/2612999081 conn(0x7f2d080777e0 msgr2=0x7f2d08077bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:08:54.372 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:54.378+0000 7f2d0cb10640 1 -- 172.21.15.29:0/2612999081 shutdown_connections 2024-03-16T12:08:54.372 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:54.378+0000 7f2d0cb10640 1 -- 172.21.15.29:0/2612999081 wait complete. 2024-03-16T12:08:54.373 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:54.379+0000 7f2d0cb10640 1 Processor -- start 2024-03-16T12:08:54.373 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:54.379+0000 7f2d0cb10640 1 -- start start 2024-03-16T12:08:54.373 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:54.379+0000 7f2d0cb10640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2d081006e0 0x7f2d08198060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:08:54.373 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:54.379+0000 7f2d0cb10640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2d081010a0 0x7f2d081985a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:08:54.373 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:54.379+0000 7f2d0cb10640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2d081066f0 0x7f2d0819c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:08:54.373 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:54.379+0000 7f2d0cb10640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f2d08110b00 con 0x7f2d081010a0 2024-03-16T12:08:54.374 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:54.379+0000 7f2d0cb10640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f2d08110980 con 0x7f2d081066f0 2024-03-16T12:08:54.374 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:54.379+0000 7f2d0cb10640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f2d08110c80 con 0x7f2d081006e0 2024-03-16T12:08:54.374 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:54.379+0000 7f2cfdd9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2d081010a0 0x7f2d081985a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:08:54.374 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:54.379+0000 7f2d0659c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2d081006e0 0x7f2d08198060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:08:54.374 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:54.379+0000 7f2cfdd9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2d081010a0 0x7f2d081985a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60390/0 (socket says 172.21.15.29:60390) 2024-03-16T12:08:54.375 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:54.379+0000 7f2cfdd9b640 1 -- 172.21.15.29:0/2636067881 learned_addr learned my addr 172.21.15.29:0/2636067881 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T12:08:54.375 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:54.379+0000 7f2d06d9d640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2d081066f0 0x7f2d0819c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:08:54.375 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:54.380+0000 7f2cfdd9b640 1 -- 172.21.15.29:0/2636067881 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2d081006e0 msgr2=0x7f2d08198060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:08:54.375 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:54.380+0000 7f2cfdd9b640 1 --2- 172.21.15.29:0/2636067881 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2d081006e0 0x7f2d08198060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:08:54.375 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:54.380+0000 7f2cfdd9b640 1 -- 172.21.15.29:0/2636067881 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2d081066f0 msgr2=0x7f2d0819c920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:08:54.375 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:54.380+0000 7f2cfdd9b640 1 --2- 172.21.15.29:0/2636067881 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2d081066f0 0x7f2d0819c920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:08:54.376 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:54.380+0000 7f2cfdd9b640 1 -- 172.21.15.29:0/2636067881 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d0819d050 con 0x7f2d081010a0 2024-03-16T12:08:54.376 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:54.380+0000 7f2d0659c640 1 --2- 172.21.15.29:0/2636067881 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2d081006e0 0x7f2d08198060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:08:54.376 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:54.380+0000 7f2cfdd9b640 1 --2- 172.21.15.29:0/2636067881 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2d081010a0 0x7f2d081985a0 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f2cf000e9f0 tx=0x7f2cf000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:08:54.376 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:54.380+0000 7f2cfffff640 1 -- 172.21.15.29:0/2636067881 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cf000cd80 con 0x7f2d081010a0 2024-03-16T12:08:54.376 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:54.380+0000 7f2d0cb10640 1 -- 172.21.15.29:0/2636067881 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2d0819d340 con 0x7f2d081010a0 2024-03-16T12:08:54.376 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:54.380+0000 7f2cfffff640 1 -- 172.21.15.29:0/2636067881 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2cf0005500 con 0x7f2d081010a0 2024-03-16T12:08:54.377 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:54.380+0000 7f2cfffff640 1 -- 172.21.15.29:0/2636067881 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cf0010690 con 0x7f2d081010a0 2024-03-16T12:08:54.377 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:54.380+0000 7f2d0cb10640 1 -- 172.21.15.29:0/2636067881 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2d081a4bd0 con 0x7f2d081010a0 2024-03-16T12:08:54.377 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:54.381+0000 7f2d0cb10640 1 -- 172.21.15.29:0/2636067881 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2d080685d0 con 0x7f2d081010a0 2024-03-16T12:08:54.378 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:54.383+0000 7f2cfffff640 1 -- 172.21.15.29:0/2636067881 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7f2cf00046c0 con 0x7f2d081010a0 2024-03-16T12:08:54.379 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:54.384+0000 7f2cfffff640 1 --2- 172.21.15.29:0/2636067881 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f2cf40a2970 0x7f2cf40a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:08:54.379 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:54.384+0000 7f2cfffff640 1 -- 172.21.15.29:0/2636067881 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 8556+0+0 (secure 0 0 0) 0x7f2cf00cf460 con 0x7f2d081010a0 2024-03-16T12:08:54.379 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:54.384+0000 7f2d0659c640 1 --2- 172.21.15.29:0/2636067881 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f2cf40a2970 0x7f2cf40a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:08:54.379 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:54.384+0000 7f2d0659c640 1 --2- 172.21.15.29:0/2636067881 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f2cf40a2970 0x7f2cf40a4e30 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f2cf8009f40 tx=0x7f2cf8005f90 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:08:54.379 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:54.387+0000 7f2cfffff640 1 -- 172.21.15.29:0/2636067881 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f2cf0053740 con 0x7f2d081010a0 2024-03-16T12:08:54.578 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:08:54 smithi029 ceph-mon[31975]: pgmap v570: 97 pgs: 97 active+clean; 362 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 357 KiB/s wr, 18 op/s 2024-03-16T12:08:54.603 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:54.614+0000 7f2d0cb10640 1 -- 172.21.15.29:0/2636067881 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2d08109c90 con 0x7f2d081010a0 2024-03-16T12:08:54.604 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:54.615+0000 7f2cfffff640 1 -- 172.21.15.29:0/2636067881 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) ==== 94+0+11875 (secure 0 0 0) 0x7f2cf009c1a0 con 0x7f2d081010a0 2024-03-16T12:08:54.605 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-16T12:08:54.605 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":32,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14574,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/3505811955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":3505811955},{"type":"v1","addr":"172.21.15.29:6835","nonce":3505811955}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14586,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/1901337868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":1901337868},{"type":"v1","addr":"172.21.15.29:6837","nonce":1901337868}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14652,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6839/544966534","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6838","nonce":544966534},{"type":"v1","addr":"172.21.15.29:6839","nonce":544966534}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14664,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6841/540705290","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6840","nonce":540705290},{"type":"v1","addr":"172.21.15.29:6841","nonce":540705290}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24407,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6833/3136024628","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6832","nonce":3136024628},{"type":"v1","addr":"172.21.15.152:6833","nonce":3136024628}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24421,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6833/2677426282","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":2677426282},{"type":"v1","addr":"172.21.15.154:6833","nonce":2677426282}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":32,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-16T11:48:38.431509+0000","modified":"2024-03-16T12:06:43.946185+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24503,"mds_1":24499,"mds_2":24464},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24434":{"gid":24434,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.152:6835/2919113178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6834","nonce":2919113178},{"type":"v1","addr":"172.21.15.152:6835","nonce":2919113178}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24464":{"gid":24464,"name":"h","rank":2,"incarnation":19,"state":"up:active","state_seq":7,"addr":"172.21.15.152:6837/1323866722","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6836","nonce":1323866722},{"type":"v1","addr":"172.21.15.152:6837","nonce":1323866722}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.154:6835/4165274681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":4165274681},{"type":"v1","addr":"172.21.15.154:6835","nonce":4165274681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":1,"incarnation":17,"state":"up:active","state_seq":6,"addr":"172.21.15.154:6839/685736862","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":685736862},{"type":"v1","addr":"172.21.15.154:6839","nonce":685736862}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.152:6839/1548731833","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6838","nonce":1548731833},{"type":"v1","addr":"172.21.15.152:6839","nonce":1548731833}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26311":{"gid":26311,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.154:6837/1624851783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":1624851783},{"type":"v1","addr":"172.21.15.154:6837","nonce":1624851783}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24503,"qdb_cluster":[24503,24499,24464]},"id":1}]} 2024-03-16T12:08:54.605 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 32 2024-03-16T12:08:54.607 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:54.617+0000 7f2d0cb10640 1 -- 172.21.15.29:0/2636067881 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f2cf40a2970 msgr2=0x7f2cf40a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:08:54.607 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:54.617+0000 7f2d0cb10640 1 --2- 172.21.15.29:0/2636067881 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f2cf40a2970 0x7f2cf40a4e30 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f2cf8009f40 tx=0x7f2cf8005f90 comp rx=0 tx=0).stop 2024-03-16T12:08:54.607 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:54.617+0000 7f2d0cb10640 1 -- 172.21.15.29:0/2636067881 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2d081010a0 msgr2=0x7f2d081985a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:08:54.607 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:54.617+0000 7f2d0cb10640 1 --2- 172.21.15.29:0/2636067881 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2d081010a0 0x7f2d081985a0 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f2cf000e9f0 tx=0x7f2cf000edf0 comp rx=0 tx=0).stop 2024-03-16T12:08:54.607 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:54.617+0000 7f2d0cb10640 1 -- 172.21.15.29:0/2636067881 shutdown_connections 2024-03-16T12:08:54.607 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:54.617+0000 7f2d0cb10640 1 --2- 172.21.15.29:0/2636067881 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f2cf40a2970 0x7f2cf40a4e30 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:08:54.607 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:54.617+0000 7f2d0cb10640 1 --2- 172.21.15.29:0/2636067881 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2d081066f0 0x7f2d0819c920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:08:54.608 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:54.617+0000 7f2d0cb10640 1 --2- 172.21.15.29:0/2636067881 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2d081010a0 0x7f2d081985a0 unknown :-1 s=CLOSED pgs=809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:08:54.608 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:54.617+0000 7f2d0cb10640 1 --2- 172.21.15.29:0/2636067881 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2d081006e0 0x7f2d08198060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:08:54.608 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:54.617+0000 7f2d0cb10640 1 -- 172.21.15.29:0/2636067881 >> 172.21.15.29:0/2636067881 conn(0x7f2d080777e0 msgr2=0x7f2d08068be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:08:54.608 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:54.617+0000 7f2d0cb10640 1 -- 172.21.15.29:0/2636067881 shutdown_connections 2024-03-16T12:08:54.608 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:54.618+0000 7f2d0cb10640 1 -- 172.21.15.29:0/2636067881 wait complete. 2024-03-16T12:08:54.618 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T12:08:54.723 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:54.731+0000 7ffbf98c0640 1 -- 172.21.15.29:0/2216219501 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffbf4101060 msgr2=0x7ffbf410d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:08:54.723 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:54.731+0000 7ffbf98c0640 1 --2- 172.21.15.29:0/2216219501 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffbf4101060 0x7ffbf410d570 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7ffbec009cd0 tx=0x7ffbec02c0f0 comp rx=0 tx=0).stop 2024-03-16T12:08:54.723 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:54.731+0000 7ffbf98c0640 1 -- 172.21.15.29:0/2216219501 shutdown_connections 2024-03-16T12:08:54.723 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:54.731+0000 7ffbf98c0640 1 --2- 172.21.15.29:0/2216219501 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffbf4101060 0x7ffbf410d570 unknown :-1 s=CLOSED pgs=810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:08:54.723 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:54.731+0000 7ffbf98c0640 1 --2- 172.21.15.29:0/2216219501 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ffbf41006a0 0x7ffbf4100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:08:54.724 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:54.731+0000 7ffbf98c0640 1 --2- 172.21.15.29:0/2216219501 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffbf41066b0 0x7ffbf4106a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:08:54.724 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:54.731+0000 7ffbf98c0640 1 -- 172.21.15.29:0/2216219501 >> 172.21.15.29:0/2216219501 conn(0x7ffbf4077780 msgr2=0x7ffbf4077b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:08:54.724 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:54.731+0000 7ffbf98c0640 1 -- 172.21.15.29:0/2216219501 shutdown_connections 2024-03-16T12:08:54.724 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:54.731+0000 7ffbf98c0640 1 -- 172.21.15.29:0/2216219501 wait complete. 2024-03-16T12:08:54.724 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:54.732+0000 7ffbf98c0640 1 Processor -- start 2024-03-16T12:08:54.724 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:54.732+0000 7ffbf98c0640 1 -- start start 2024-03-16T12:08:54.724 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:54.732+0000 7ffbf98c0640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ffbf41006a0 0x7ffbf4198020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:08:54.724 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:54.732+0000 7ffbf98c0640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffbf4101060 0x7ffbf4198560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:08:54.725 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:54.732+0000 7ffbf98c0640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffbf41066b0 0x7ffbf419c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:08:54.725 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:54.732+0000 7ffbf98c0640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7ffbf4110ac0 con 0x7ffbf41066b0 2024-03-16T12:08:54.725 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:54.732+0000 7ffbf98c0640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7ffbf4110940 con 0x7ffbf41006a0 2024-03-16T12:08:54.725 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:54.732+0000 7ffbf98c0640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7ffbf4110c40 con 0x7ffbf4101060 2024-03-16T12:08:54.725 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:54.732+0000 7ffbf37fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffbf41066b0 0x7ffbf419c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:08:54.725 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:54.732+0000 7ffbf37fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffbf41066b0 0x7ffbf419c8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60416/0 (socket says 172.21.15.29:60416) 2024-03-16T12:08:54.725 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:54.732+0000 7ffbf37fe640 1 -- 172.21.15.29:0/1281639984 learned_addr learned my addr 172.21.15.29:0/1281639984 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T12:08:54.725 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:54.732+0000 7ffbf2ffd640 1 --2- 172.21.15.29:0/1281639984 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ffbf41006a0 0x7ffbf4198020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:08:54.725 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:54.732+0000 7ffbebfff640 1 --2- 172.21.15.29:0/1281639984 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffbf4101060 0x7ffbf4198560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:08:54.726 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:54.733+0000 7ffbf37fe640 1 -- 172.21.15.29:0/1281639984 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffbf4101060 msgr2=0x7ffbf4198560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:08:54.726 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:54.733+0000 7ffbf37fe640 1 --2- 172.21.15.29:0/1281639984 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffbf4101060 0x7ffbf4198560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:08:54.726 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:54.733+0000 7ffbf37fe640 1 -- 172.21.15.29:0/1281639984 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ffbf41006a0 msgr2=0x7ffbf4198020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:08:54.726 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:54.733+0000 7ffbf37fe640 1 --2- 172.21.15.29:0/1281639984 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ffbf41006a0 0x7ffbf4198020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:08:54.726 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:54.733+0000 7ffbf37fe640 1 -- 172.21.15.29:0/1281639984 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffbf419d010 con 0x7ffbf41066b0 2024-03-16T12:08:54.726 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:54.733+0000 7ffbf37fe640 1 --2- 172.21.15.29:0/1281639984 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffbf41066b0 0x7ffbf419c8e0 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7ffbec0348f0 tx=0x7ffbec032c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:08:54.726 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:54.733+0000 7ffbf88be640 1 -- 172.21.15.29:0/1281639984 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbec041020 con 0x7ffbf41066b0 2024-03-16T12:08:54.726 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:54.733+0000 7ffbf98c0640 1 -- 172.21.15.29:0/1281639984 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffbf419d2a0 con 0x7ffbf41066b0 2024-03-16T12:08:54.727 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:54.733+0000 7ffbf88be640 1 -- 172.21.15.29:0/1281639984 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffbec032e80 con 0x7ffbf41066b0 2024-03-16T12:08:54.727 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:54.733+0000 7ffbf88be640 1 -- 172.21.15.29:0/1281639984 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbec039040 con 0x7ffbf41066b0 2024-03-16T12:08:54.727 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:54.733+0000 7ffbf98c0640 1 -- 172.21.15.29:0/1281639984 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffbf41a4b90 con 0x7ffbf41066b0 2024-03-16T12:08:54.727 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:54.734+0000 7ffbf98c0640 1 -- 172.21.15.29:0/1281639984 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffbf4068a20 con 0x7ffbf41066b0 2024-03-16T12:08:54.728 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:54.735+0000 7ffbf88be640 1 -- 172.21.15.29:0/1281639984 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7ffbec044020 con 0x7ffbf41066b0 2024-03-16T12:08:54.728 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:54.735+0000 7ffbf88be640 1 --2- 172.21.15.29:0/1281639984 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7ffbe00a2970 0x7ffbe00a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:08:54.728 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:54.736+0000 7ffbf2ffd640 1 --2- 172.21.15.29:0/1281639984 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7ffbe00a2970 0x7ffbe00a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:08:54.728 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:54.736+0000 7ffbf88be640 1 -- 172.21.15.29:0/1281639984 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 8556+0+0 (secure 0 0 0) 0x7ffbec0f0980 con 0x7ffbf41066b0 2024-03-16T12:08:54.728 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:54.736+0000 7ffbf2ffd640 1 --2- 172.21.15.29:0/1281639984 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7ffbe00a2970 0x7ffbe00a4e30 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7ffbe4009b50 tx=0x7ffbe4005f90 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:08:54.729 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:54.740+0000 7ffbf88be640 1 -- 172.21.15.29:0/1281639984 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7ffbec04b020 con 0x7ffbf41066b0 2024-03-16T12:08:54.785 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:08:54 smithi154 ceph-mon[36320]: pgmap v570: 97 pgs: 97 active+clean; 362 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 357 KiB/s wr, 18 op/s 2024-03-16T12:08:54.867 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:08:54 smithi152 ceph-mon[36291]: pgmap v570: 97 pgs: 97 active+clean; 362 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 357 KiB/s wr, 18 op/s 2024-03-16T12:08:54.951 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:54.962+0000 7ffbf98c0640 1 -- 172.21.15.29:0/1281639984 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ffbf4109c20 con 0x7ffbf41066b0 2024-03-16T12:08:54.952 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:54.963+0000 7ffbf88be640 1 -- 172.21.15.29:0/1281639984 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) ==== 94+0+11875 (secure 0 0 0) 0x7ffbec0bd770 con 0x7ffbf41066b0 2024-03-16T12:08:54.952 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-16T12:08:54.953 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":32,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14574,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/3505811955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":3505811955},{"type":"v1","addr":"172.21.15.29:6835","nonce":3505811955}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14586,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/1901337868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":1901337868},{"type":"v1","addr":"172.21.15.29:6837","nonce":1901337868}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14652,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6839/544966534","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6838","nonce":544966534},{"type":"v1","addr":"172.21.15.29:6839","nonce":544966534}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14664,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6841/540705290","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6840","nonce":540705290},{"type":"v1","addr":"172.21.15.29:6841","nonce":540705290}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24407,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6833/3136024628","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6832","nonce":3136024628},{"type":"v1","addr":"172.21.15.152:6833","nonce":3136024628}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24421,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6833/2677426282","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":2677426282},{"type":"v1","addr":"172.21.15.154:6833","nonce":2677426282}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":32,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-16T11:48:38.431509+0000","modified":"2024-03-16T12:06:43.946185+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24503,"mds_1":24499,"mds_2":24464},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24434":{"gid":24434,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.152:6835/2919113178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6834","nonce":2919113178},{"type":"v1","addr":"172.21.15.152:6835","nonce":2919113178}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24464":{"gid":24464,"name":"h","rank":2,"incarnation":19,"state":"up:active","state_seq":7,"addr":"172.21.15.152:6837/1323866722","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6836","nonce":1323866722},{"type":"v1","addr":"172.21.15.152:6837","nonce":1323866722}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.154:6835/4165274681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":4165274681},{"type":"v1","addr":"172.21.15.154:6835","nonce":4165274681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":1,"incarnation":17,"state":"up:active","state_seq":6,"addr":"172.21.15.154:6839/685736862","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":685736862},{"type":"v1","addr":"172.21.15.154:6839","nonce":685736862}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.152:6839/1548731833","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6838","nonce":1548731833},{"type":"v1","addr":"172.21.15.152:6839","nonce":1548731833}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26311":{"gid":26311,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.154:6837/1624851783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":1624851783},{"type":"v1","addr":"172.21.15.154:6837","nonce":1624851783}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24503,"qdb_cluster":[24503,24499,24464]},"id":1}]} 2024-03-16T12:08:54.953 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 32 2024-03-16T12:08:54.954 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:54.965+0000 7ffbf98c0640 1 -- 172.21.15.29:0/1281639984 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7ffbe00a2970 msgr2=0x7ffbe00a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:08:54.954 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:54.965+0000 7ffbf98c0640 1 --2- 172.21.15.29:0/1281639984 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7ffbe00a2970 0x7ffbe00a4e30 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7ffbe4009b50 tx=0x7ffbe4005f90 comp rx=0 tx=0).stop 2024-03-16T12:08:54.954 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:54.965+0000 7ffbf98c0640 1 -- 172.21.15.29:0/1281639984 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffbf41066b0 msgr2=0x7ffbf419c8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:08:54.955 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:54.965+0000 7ffbf98c0640 1 --2- 172.21.15.29:0/1281639984 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffbf41066b0 0x7ffbf419c8e0 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7ffbec0348f0 tx=0x7ffbec032c90 comp rx=0 tx=0).stop 2024-03-16T12:08:54.955 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:54.965+0000 7ffbf98c0640 1 -- 172.21.15.29:0/1281639984 shutdown_connections 2024-03-16T12:08:54.955 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:54.965+0000 7ffbf98c0640 1 --2- 172.21.15.29:0/1281639984 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7ffbe00a2970 0x7ffbe00a4e30 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:08:54.955 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:54.965+0000 7ffbf98c0640 1 --2- 172.21.15.29:0/1281639984 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffbf41066b0 0x7ffbf419c8e0 unknown :-1 s=CLOSED pgs=811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:08:54.955 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:54.965+0000 7ffbf98c0640 1 --2- 172.21.15.29:0/1281639984 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffbf4101060 0x7ffbf4198560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:08:54.955 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:54.965+0000 7ffbf98c0640 1 --2- 172.21.15.29:0/1281639984 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ffbf41006a0 0x7ffbf4198020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:08:54.955 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:54.965+0000 7ffbf98c0640 1 -- 172.21.15.29:0/1281639984 >> 172.21.15.29:0/1281639984 conn(0x7ffbf4077780 msgr2=0x7ffbf406a0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:08:54.955 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:54.965+0000 7ffbf98c0640 1 -- 172.21.15.29:0/1281639984 shutdown_connections 2024-03-16T12:08:54.956 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:54.965+0000 7ffbf98c0640 1 -- 172.21.15.29:0/1281639984 wait complete. 2024-03-16T12:08:54.965 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-16T12:08:55.070 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:55.078+0000 7f97bfe6a640 1 -- 172.21.15.29:0/2092280104 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f97b8106ba0 msgr2=0x7f97b810e7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:08:55.070 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:55.078+0000 7f97bfe6a640 1 --2- 172.21.15.29:0/2092280104 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f97b8106ba0 0x7f97b810e7b0 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7f97b0009ca0 tx=0x7f97b002c0f0 comp rx=0 tx=0).stop 2024-03-16T12:08:55.071 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:55.078+0000 7f97bfe6a640 1 -- 172.21.15.29:0/2092280104 shutdown_connections 2024-03-16T12:08:55.071 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:55.078+0000 7f97bfe6a640 1 --2- 172.21.15.29:0/2092280104 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f97b8106ba0 0x7f97b810e7b0 unknown :-1 s=CLOSED pgs=812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:08:55.071 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:55.078+0000 7f97bfe6a640 1 --2- 172.21.15.29:0/2092280104 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f97b8100d90 0x7f97b81064c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:08:55.071 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:55.078+0000 7f97bfe6a640 1 --2- 172.21.15.29:0/2092280104 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f97b81003e0 0x7f97b81007c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:08:55.071 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:55.078+0000 7f97bfe6a640 1 -- 172.21.15.29:0/2092280104 >> 172.21.15.29:0/2092280104 conn(0x7f97b80759f0 msgr2=0x7f97b8075e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:08:55.071 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:55.078+0000 7f97bfe6a640 1 -- 172.21.15.29:0/2092280104 shutdown_connections 2024-03-16T12:08:55.071 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:55.078+0000 7f97bfe6a640 1 -- 172.21.15.29:0/2092280104 wait complete. 2024-03-16T12:08:55.072 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:55.079+0000 7f97bfe6a640 1 Processor -- start 2024-03-16T12:08:55.072 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:55.079+0000 7f97bfe6a640 1 -- start start 2024-03-16T12:08:55.072 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:55.079+0000 7f97bfe6a640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f97b81003e0 0x7f97b8101350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:08:55.072 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:55.079+0000 7f97bfe6a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f97b8106ba0 0x7f97b8101890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:08:55.072 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:55.079+0000 7f97bfe6a640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f97b8101dd0 0x7f97b81a6340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:08:55.072 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:55.079+0000 7f97bfe6a640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f97b8111c90 con 0x7f97b8106ba0 2024-03-16T12:08:55.073 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:55.079+0000 7f97bfe6a640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f97b8111b10 con 0x7f97b81003e0 2024-03-16T12:08:55.073 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:55.079+0000 7f97bfe6a640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f97b8111e10 con 0x7f97b8101dd0 2024-03-16T12:08:55.073 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:55.079+0000 7f97bd405640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f97b8106ba0 0x7f97b8101890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:08:55.073 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:55.079+0000 7f97bd405640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f97b8106ba0 0x7f97b8101890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60424/0 (socket says 172.21.15.29:60424) 2024-03-16T12:08:55.073 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:55.079+0000 7f97bd405640 1 -- 172.21.15.29:0/3374868693 learned_addr learned my addr 172.21.15.29:0/3374868693 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T12:08:55.073 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:55.079+0000 7f97bdc06640 1 --2- 172.21.15.29:0/3374868693 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f97b81003e0 0x7f97b8101350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:08:55.073 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:55.079+0000 7f97be407640 1 --2- 172.21.15.29:0/3374868693 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f97b8101dd0 0x7f97b81a6340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:08:55.074 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:55.079+0000 7f97bd405640 1 -- 172.21.15.29:0/3374868693 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f97b8101dd0 msgr2=0x7f97b81a6340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:08:55.074 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:55.079+0000 7f97bd405640 1 --2- 172.21.15.29:0/3374868693 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f97b8101dd0 0x7f97b81a6340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:08:55.074 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:55.079+0000 7f97bd405640 1 -- 172.21.15.29:0/3374868693 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f97b81003e0 msgr2=0x7f97b8101350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:08:55.074 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:55.079+0000 7f97bd405640 1 --2- 172.21.15.29:0/3374868693 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f97b81003e0 0x7f97b8101350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:08:55.074 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:55.079+0000 7f97bd405640 1 -- 172.21.15.29:0/3374868693 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97b81a69e0 con 0x7f97b8106ba0 2024-03-16T12:08:55.074 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:55.079+0000 7f97bdc06640 1 --2- 172.21.15.29:0/3374868693 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f97b81003e0 0x7f97b8101350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T12:08:55.074 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:55.080+0000 7f97bd405640 1 --2- 172.21.15.29:0/3374868693 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f97b8106ba0 0x7f97b8101890 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f97a800e9c0 tx=0x7f97a800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:08:55.075 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:55.080+0000 7f97aeffd640 1 -- 172.21.15.29:0/3374868693 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97a800cd80 con 0x7f97b8106ba0 2024-03-16T12:08:55.075 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:55.080+0000 7f97aeffd640 1 -- 172.21.15.29:0/3374868693 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f97a8005500 con 0x7f97b8106ba0 2024-03-16T12:08:55.075 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:55.080+0000 7f97aeffd640 1 -- 172.21.15.29:0/3374868693 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97a8010690 con 0x7f97b8106ba0 2024-03-16T12:08:55.075 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:55.080+0000 7f97bfe6a640 1 -- 172.21.15.29:0/3374868693 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97b81a6cd0 con 0x7f97b8106ba0 2024-03-16T12:08:55.075 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:55.080+0000 7f97bfe6a640 1 -- 172.21.15.29:0/3374868693 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97b81a71d0 con 0x7f97b8106ba0 2024-03-16T12:08:55.075 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:55.081+0000 7f97bfe6a640 1 -- 172.21.15.29:0/3374868693 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f97b8069910 con 0x7f97b8106ba0 2024-03-16T12:08:55.076 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:55.082+0000 7f97aeffd640 1 -- 172.21.15.29:0/3374868693 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7f97a80046c0 con 0x7f97b8106ba0 2024-03-16T12:08:55.076 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:55.083+0000 7f97aeffd640 1 --2- 172.21.15.29:0/3374868693 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f97a40a2970 0x7f97a40a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:08:55.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:55.083+0000 7f97bdc06640 1 --2- 172.21.15.29:0/3374868693 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f97a40a2970 0x7f97a40a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:08:55.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:55.083+0000 7f97aeffd640 1 -- 172.21.15.29:0/3374868693 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 8556+0+0 (secure 0 0 0) 0x7f97a80cf170 con 0x7f97b8106ba0 2024-03-16T12:08:55.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:55.083+0000 7f97bdc06640 1 --2- 172.21.15.29:0/3374868693 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f97a40a2970 0x7f97a40a4e30 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f97b4009fc0 tx=0x7f97b4005f90 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:08:55.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:55.087+0000 7f97aeffd640 1 -- 172.21.15.29:0/3374868693 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f97a809beb0 con 0x7f97b8106ba0 2024-03-16T12:08:55.262 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:55.273+0000 7f97bfe6a640 1 -- 172.21.15.29:0/3374868693 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f97b8109e70 con 0x7f97b8106ba0 2024-03-16T12:08:55.264 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:55.275+0000 7f97aeffd640 1 -- 172.21.15.29:0/3374868693 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v116) ==== 74+0+23200 (secure 0 0 0) 0x7f97a809b7b0 con 0x7f97b8106ba0 2024-03-16T12:08:55.265 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-16T12:08:55.265 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":116,"fsid":"d1887f28-e389-11ee-95c9-87774f69a715","created":"2024-03-16T11:40:16.475063+0000","modified":"2024-03-16T12:08:22.177679+0000","last_up_change":"2024-03-16T11:47:44.349348+0000","last_in_change":"2024-03-16T11:47:23.672432+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-16T11:43:51.892652+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-16T11:48:36.709732+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-16T11:48:37.586140+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"115","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":30,"snap_epoch":115,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"60f5608f-a408-45ba-bdf6-5b53758f88fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":2748947913},{"type":"v1","addr":"172.21.15.29:6803","nonce":2748947913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":2748947913},{"type":"v1","addr":"172.21.15.29:6805","nonce":2748947913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":2748947913},{"type":"v1","addr":"172.21.15.29:6809","nonce":2748947913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":2748947913},{"type":"v1","addr":"172.21.15.29:6807","nonce":2748947913}]},"public_addr":"172.21.15.29:6803/2748947913","cluster_addr":"172.21.15.29:6805/2748947913","heartbeat_back_addr":"172.21.15.29:6809/2748947913","heartbeat_front_addr":"172.21.15.29:6807/2748947913","state":["exists","up"]},{"osd":1,"uuid":"e70b53d6-ee78-4e56-9a2c-263ae0501b43","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6800","nonce":2339408422},{"type":"v1","addr":"172.21.15.152:6801","nonce":2339408422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6802","nonce":2339408422},{"type":"v1","addr":"172.21.15.152:6803","nonce":2339408422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6806","nonce":2339408422},{"type":"v1","addr":"172.21.15.152:6807","nonce":2339408422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6804","nonce":2339408422},{"type":"v1","addr":"172.21.15.152:6805","nonce":2339408422}]},"public_addr":"172.21.15.152:6801/2339408422","cluster_addr":"172.21.15.152:6803/2339408422","heartbeat_back_addr":"172.21.15.152:6807/2339408422","heartbeat_front_addr":"172.21.15.152:6805/2339408422","state":["exists","up"]},{"osd":2,"uuid":"ff39389a-dfac-44b5-80ff-36abca316e78","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6800","nonce":4288325280},{"type":"v1","addr":"172.21.15.154:6801","nonce":4288325280}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6802","nonce":4288325280},{"type":"v1","addr":"172.21.15.154:6803","nonce":4288325280}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6806","nonce":4288325280},{"type":"v1","addr":"172.21.15.154:6807","nonce":4288325280}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":4288325280},{"type":"v1","addr":"172.21.15.154:6805","nonce":4288325280}]},"public_addr":"172.21.15.154:6801/4288325280","cluster_addr":"172.21.15.154:6803/4288325280","heartbeat_back_addr":"172.21.15.154:6807/4288325280","heartbeat_front_addr":"172.21.15.154:6805/4288325280","state":["exists","up"]},{"osd":3,"uuid":"069e1795-55bc-4324-b94c-eb24ef2334c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":1538750397},{"type":"v1","addr":"172.21.15.29:6811","nonce":1538750397}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":1538750397},{"type":"v1","addr":"172.21.15.29:6813","nonce":1538750397}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":1538750397},{"type":"v1","addr":"172.21.15.29:6817","nonce":1538750397}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":1538750397},{"type":"v1","addr":"172.21.15.29:6815","nonce":1538750397}]},"public_addr":"172.21.15.29:6811/1538750397","cluster_addr":"172.21.15.29:6813/1538750397","heartbeat_back_addr":"172.21.15.29:6817/1538750397","heartbeat_front_addr":"172.21.15.29:6815/1538750397","state":["exists","up"]},{"osd":4,"uuid":"5a9d43a1-e8f5-4ab5-a441-afbc6ed4066e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6808","nonce":2795081750},{"type":"v1","addr":"172.21.15.152:6809","nonce":2795081750}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6810","nonce":2795081750},{"type":"v1","addr":"172.21.15.152:6811","nonce":2795081750}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6814","nonce":2795081750},{"type":"v1","addr":"172.21.15.152:6815","nonce":2795081750}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6812","nonce":2795081750},{"type":"v1","addr":"172.21.15.152:6813","nonce":2795081750}]},"public_addr":"172.21.15.152:6809/2795081750","cluster_addr":"172.21.15.152:6811/2795081750","heartbeat_back_addr":"172.21.15.152:6815/2795081750","heartbeat_front_addr":"172.21.15.152:6813/2795081750","state":["exists","up"]},{"osd":5,"uuid":"234de428-4ac7-44be-9a54-7b58331fdf8c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":1472686560},{"type":"v1","addr":"172.21.15.154:6809","nonce":1472686560}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6810","nonce":1472686560},{"type":"v1","addr":"172.21.15.154:6811","nonce":1472686560}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6814","nonce":1472686560},{"type":"v1","addr":"172.21.15.154:6815","nonce":1472686560}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6812","nonce":1472686560},{"type":"v1","addr":"172.21.15.154:6813","nonce":1472686560}]},"public_addr":"172.21.15.154:6809/1472686560","cluster_addr":"172.21.15.154:6811/1472686560","heartbeat_back_addr":"172.21.15.154:6815/1472686560","heartbeat_front_addr":"172.21.15.154:6813/1472686560","state":["exists","up"]},{"osd":6,"uuid":"ae45b8d9-7883-40c9-a9cd-4c92b601d4f2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":816506244},{"type":"v1","addr":"172.21.15.29:6819","nonce":816506244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":816506244},{"type":"v1","addr":"172.21.15.29:6821","nonce":816506244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":816506244},{"type":"v1","addr":"172.21.15.29:6825","nonce":816506244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":816506244},{"type":"v1","addr":"172.21.15.29:6823","nonce":816506244}]},"public_addr":"172.21.15.29:6819/816506244","cluster_addr":"172.21.15.29:6821/816506244","heartbeat_back_addr":"172.21.15.29:6825/816506244","heartbeat_front_addr":"172.21.15.29:6823/816506244","state":["exists","up"]},{"osd":7,"uuid":"6f687b28-4fce-4d4a-8b4e-294173d9d6c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6816","nonce":2924651787},{"type":"v1","addr":"172.21.15.152:6817","nonce":2924651787}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6818","nonce":2924651787},{"type":"v1","addr":"172.21.15.152:6819","nonce":2924651787}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6822","nonce":2924651787},{"type":"v1","addr":"172.21.15.152:6823","nonce":2924651787}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6820","nonce":2924651787},{"type":"v1","addr":"172.21.15.152:6821","nonce":2924651787}]},"public_addr":"172.21.15.152:6817/2924651787","cluster_addr":"172.21.15.152:6819/2924651787","heartbeat_back_addr":"172.21.15.152:6823/2924651787","heartbeat_front_addr":"172.21.15.152:6821/2924651787","state":["exists","up"]},{"osd":8,"uuid":"a25fb741-caa7-429a-b540-e394d034b446","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6816","nonce":129974156},{"type":"v1","addr":"172.21.15.154:6817","nonce":129974156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6818","nonce":129974156},{"type":"v1","addr":"172.21.15.154:6819","nonce":129974156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6822","nonce":129974156},{"type":"v1","addr":"172.21.15.154:6823","nonce":129974156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6820","nonce":129974156},{"type":"v1","addr":"172.21.15.154:6821","nonce":129974156}]},"public_addr":"172.21.15.154:6817/129974156","cluster_addr":"172.21.15.154:6819/129974156","heartbeat_back_addr":"172.21.15.154:6823/129974156","heartbeat_front_addr":"172.21.15.154:6821/129974156","state":["exists","up"]},{"osd":9,"uuid":"1071cb0e-8c5f-4cd8-9805-0045e220d946","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6826","nonce":2951414243},{"type":"v1","addr":"172.21.15.29:6827","nonce":2951414243}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":2951414243},{"type":"v1","addr":"172.21.15.29:6829","nonce":2951414243}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":2951414243},{"type":"v1","addr":"172.21.15.29:6833","nonce":2951414243}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":2951414243},{"type":"v1","addr":"172.21.15.29:6831","nonce":2951414243}]},"public_addr":"172.21.15.29:6827/2951414243","cluster_addr":"172.21.15.29:6829/2951414243","heartbeat_back_addr":"172.21.15.29:6833/2951414243","heartbeat_front_addr":"172.21.15.29:6831/2951414243","state":["exists","up"]},{"osd":10,"uuid":"ee7c7d74-3f39-4cd4-be80-843fb1dd9ecc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6824","nonce":3412525480},{"type":"v1","addr":"172.21.15.152:6825","nonce":3412525480}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6826","nonce":3412525480},{"type":"v1","addr":"172.21.15.152:6827","nonce":3412525480}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6830","nonce":3412525480},{"type":"v1","addr":"172.21.15.152:6831","nonce":3412525480}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6828","nonce":3412525480},{"type":"v1","addr":"172.21.15.152:6829","nonce":3412525480}]},"public_addr":"172.21.15.152:6825/3412525480","cluster_addr":"172.21.15.152:6827/3412525480","heartbeat_back_addr":"172.21.15.152:6831/3412525480","heartbeat_front_addr":"172.21.15.152:6829/3412525480","state":["exists","up"]},{"osd":11,"uuid":"4598c803-984a-4de6-ac22-066502c17889","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6824","nonce":1160927966},{"type":"v1","addr":"172.21.15.154:6825","nonce":1160927966}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6826","nonce":1160927966},{"type":"v1","addr":"172.21.15.154:6827","nonce":1160927966}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6830","nonce":1160927966},{"type":"v1","addr":"172.21.15.154:6831","nonce":1160927966}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6828","nonce":1160927966},{"type":"v1","addr":"172.21.15.154:6829","nonce":1160927966}]},"public_addr":"172.21.15.154:6825/1160927966","cluster_addr":"172.21.15.154:6827/1160927966","heartbeat_back_addr":"172.21.15.154:6831/1160927966","heartbeat_front_addr":"172.21.15.154:6829/1160927966","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:43:02.343393+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:43:25.209151+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:43:48.946164+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:44:14.588424+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:44:39.700025+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:45:04.061197+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:45:31.214508+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:45:56.937296+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:46:22.768915+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:46:50.101011+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:47:16.298041+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T11:47:42.402095+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"2.1b","mappings":[{"from":2,"to":1}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"3.33","mappings":[{"from":5,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.29:0/1361425453":"2024-03-16T12:55:48.830413+0000","172.21.15.29:0/1116486932":"2024-03-17T11:41:27.531724+0000","172.21.15.29:0/1784249039":"2024-03-17T11:41:00.505678+0000","172.21.15.29:6800/4192064640":"2024-03-17T11:50:54.850617+0000","172.21.15.29:0/1330100009":"2024-03-17T11:41:00.505678+0000","172.21.15.29:6801/2275315654":"2024-03-17T11:41:00.505678+0000","172.21.15.29:6801/2798831128":"2024-03-17T11:41:27.531724+0000","172.21.15.29:6800/2275315654":"2024-03-17T11:41:00.505678+0000","172.21.15.29:6800/2798831128":"2024-03-17T11:41:27.531724+0000","172.21.15.29:0/1211659364":"2024-03-17T11:41:27.531724+0000","172.21.15.29:0/1281028285":"2024-03-17T11:41:00.505678+0000","172.21.15.29:0/1853086886":"2024-03-17T11:41:27.531724+0000","172.21.15.29:0/623985488":"2024-03-17T11:50:54.850617+0000","172.21.15.29:6801/4192064640":"2024-03-17T11:50:54.850617+0000","172.21.15.29:0/1114950131":"2024-03-17T11:50:54.850617+0000","172.21.15.29:0/2374713213":"2024-03-17T11:50:54.850617+0000","172.21.15.29:0/340047334":"2024-03-17T11:50:54.850617+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":3,"snaps":[{"begin":18,"length":1},{"begin":30,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-16T12:08:55.266 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:55.277+0000 7f97bfe6a640 1 -- 172.21.15.29:0/3374868693 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f97a40a2970 msgr2=0x7f97a40a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:08:55.266 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:55.277+0000 7f97bfe6a640 1 --2- 172.21.15.29:0/3374868693 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f97a40a2970 0x7f97a40a4e30 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f97b4009fc0 tx=0x7f97b4005f90 comp rx=0 tx=0).stop 2024-03-16T12:08:55.267 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:55.277+0000 7f97bfe6a640 1 -- 172.21.15.29:0/3374868693 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f97b8106ba0 msgr2=0x7f97b8101890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:08:55.267 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:55.277+0000 7f97bfe6a640 1 --2- 172.21.15.29:0/3374868693 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f97b8106ba0 0x7f97b8101890 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f97a800e9c0 tx=0x7f97a800ee80 comp rx=0 tx=0).stop 2024-03-16T12:08:55.267 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:55.277+0000 7f97bfe6a640 1 -- 172.21.15.29:0/3374868693 shutdown_connections 2024-03-16T12:08:55.267 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:55.277+0000 7f97bfe6a640 1 --2- 172.21.15.29:0/3374868693 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f97a40a2970 0x7f97a40a4e30 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:08:55.267 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:55.277+0000 7f97bfe6a640 1 --2- 172.21.15.29:0/3374868693 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f97b8101dd0 0x7f97b81a6340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:08:55.267 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:55.277+0000 7f97bfe6a640 1 --2- 172.21.15.29:0/3374868693 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f97b8106ba0 0x7f97b8101890 unknown :-1 s=CLOSED pgs=813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:08:55.267 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:55.277+0000 7f97bfe6a640 1 --2- 172.21.15.29:0/3374868693 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f97b81003e0 0x7f97b8101350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:08:55.267 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:55.277+0000 7f97bfe6a640 1 -- 172.21.15.29:0/3374868693 >> 172.21.15.29:0/3374868693 conn(0x7f97b80759f0 msgr2=0x7f97b80fe1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:08:55.268 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:55.277+0000 7f97bfe6a640 1 -- 172.21.15.29:0/3374868693 shutdown_connections 2024-03-16T12:08:55.268 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:55.277+0000 7f97bfe6a640 1 -- 172.21.15.29:0/3374868693 wait complete. 2024-03-16T12:08:55.278 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-16T12:08:55.384 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:55.392+0000 7fec2a2e8640 1 -- 172.21.15.29:0/749127683 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fec24103340 msgr2=0x7fec2410b820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:08:55.385 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:55.392+0000 7fec2a2e8640 1 --2- 172.21.15.29:0/749127683 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fec24103340 0x7fec2410b820 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7fec18009900 tx=0x7fec1802bcd0 comp rx=0 tx=0).stop 2024-03-16T12:08:55.385 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:55.392+0000 7fec2a2e8640 1 -- 172.21.15.29:0/749127683 shutdown_connections 2024-03-16T12:08:55.385 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:55.392+0000 7fec2a2e8640 1 --2- 172.21.15.29:0/749127683 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fec2410bdf0 0x7fec2410e220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:08:55.385 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:55.392+0000 7fec2a2e8640 1 --2- 172.21.15.29:0/749127683 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fec24103340 0x7fec2410b820 unknown :-1 s=CLOSED pgs=814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:08:55.385 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:55.392+0000 7fec2a2e8640 1 --2- 172.21.15.29:0/749127683 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fec24102a20 0x7fec24102e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:08:55.385 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:55.392+0000 7fec2a2e8640 1 -- 172.21.15.29:0/749127683 >> 172.21.15.29:0/749127683 conn(0x7fec24076a60 msgr2=0x7fec24076e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:08:55.386 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:55.392+0000 7fec2a2e8640 1 -- 172.21.15.29:0/749127683 shutdown_connections 2024-03-16T12:08:55.386 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:55.392+0000 7fec2a2e8640 1 -- 172.21.15.29:0/749127683 wait complete. 2024-03-16T12:08:55.386 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:55.393+0000 7fec2a2e8640 1 Processor -- start 2024-03-16T12:08:55.386 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:55.393+0000 7fec2a2e8640 1 -- start start 2024-03-16T12:08:55.386 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:55.393+0000 7fec2a2e8640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fec24102a20 0x7fec24198080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:08:55.386 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:55.393+0000 7fec2a2e8640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fec24103340 0x7fec241985c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:08:55.386 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:55.393+0000 7fec2a2e8640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fec2410bdf0 0x7fec2419c940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:08:55.386 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:55.393+0000 7fec2a2e8640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fec24111b80 con 0x7fec24102a20 2024-03-16T12:08:55.387 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:55.393+0000 7fec2a2e8640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fec24111a00 con 0x7fec24103340 2024-03-16T12:08:55.387 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:55.393+0000 7fec2a2e8640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fec24111d00 con 0x7fec2410bdf0 2024-03-16T12:08:55.387 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:55.393+0000 7fec23fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fec24102a20 0x7fec24198080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:08:55.387 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:55.393+0000 7fec23fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fec24102a20 0x7fec24198080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60444/0 (socket says 172.21.15.29:60444) 2024-03-16T12:08:55.387 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:55.393+0000 7fec23fff640 1 -- 172.21.15.29:0/3612385166 learned_addr learned my addr 172.21.15.29:0/3612385166 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T12:08:55.387 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:55.394+0000 7fec28885640 1 --2- 172.21.15.29:0/3612385166 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fec2410bdf0 0x7fec2419c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:08:55.387 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:55.394+0000 7fec237fe640 1 --2- 172.21.15.29:0/3612385166 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fec24103340 0x7fec241985c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:08:55.388 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:55.394+0000 7fec23fff640 1 -- 172.21.15.29:0/3612385166 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fec2410bdf0 msgr2=0x7fec2419c940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:08:55.388 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:55.394+0000 7fec23fff640 1 --2- 172.21.15.29:0/3612385166 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fec2410bdf0 0x7fec2419c940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:08:55.388 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:55.394+0000 7fec23fff640 1 -- 172.21.15.29:0/3612385166 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fec24103340 msgr2=0x7fec241985c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:08:55.388 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:55.394+0000 7fec23fff640 1 --2- 172.21.15.29:0/3612385166 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fec24103340 0x7fec241985c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:08:55.388 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:55.394+0000 7fec23fff640 1 -- 172.21.15.29:0/3612385166 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec2419d070 con 0x7fec24102a20 2024-03-16T12:08:55.388 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:55.394+0000 7fec23fff640 1 --2- 172.21.15.29:0/3612385166 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fec24102a20 0x7fec24198080 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7fec1400e9f0 tx=0x7fec1400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:08:55.388 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:55.394+0000 7fec217fa640 1 -- 172.21.15.29:0/3612385166 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec1400cd80 con 0x7fec24102a20 2024-03-16T12:08:55.388 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:55.394+0000 7fec2a2e8640 1 -- 172.21.15.29:0/3612385166 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fec2419d360 con 0x7fec24102a20 2024-03-16T12:08:55.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:55.394+0000 7fec217fa640 1 -- 172.21.15.29:0/3612385166 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fec14010910 con 0x7fec24102a20 2024-03-16T12:08:55.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:55.394+0000 7fec217fa640 1 -- 172.21.15.29:0/3612385166 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec140188f0 con 0x7fec24102a20 2024-03-16T12:08:55.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:55.394+0000 7fec2a2e8640 1 -- 172.21.15.29:0/3612385166 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fec241a4bf0 con 0x7fec24102a20 2024-03-16T12:08:55.390 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:55.395+0000 7fec2a2e8640 1 -- 172.21.15.29:0/3612385166 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fec2406a5f0 con 0x7fec24102a20 2024-03-16T12:08:55.390 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:55.397+0000 7fec217fa640 1 -- 172.21.15.29:0/3612385166 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7fec14010ac0 con 0x7fec24102a20 2024-03-16T12:08:55.390 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:55.397+0000 7fec217fa640 1 --2- 172.21.15.29:0/3612385166 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7fec0c0a2970 0x7fec0c0a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:08:55.390 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:55.398+0000 7fec237fe640 1 --2- 172.21.15.29:0/3612385166 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7fec0c0a2970 0x7fec0c0a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:08:55.390 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:55.398+0000 7fec217fa640 1 -- 172.21.15.29:0/3612385166 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 8556+0+0 (secure 0 0 0) 0x7fec140cf600 con 0x7fec24102a20 2024-03-16T12:08:55.391 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:55.398+0000 7fec237fe640 1 --2- 172.21.15.29:0/3612385166 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7fec0c0a2970 0x7fec0c0a4e30 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fec1802efd0 tx=0x7fec1802cf90 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:08:55.391 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:55.401+0000 7fec217fa640 1 -- 172.21.15.29:0/3612385166 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fec1409bcf0 con 0x7fec24102a20 2024-03-16T12:08:55.578 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:08:55 smithi029 ceph-mon[31975]: from='client.? 172.21.15.29:0/2636067881' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T12:08:55.578 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:08:55 smithi029 ceph-mon[31975]: from='client.? 172.21.15.29:0/1281639984' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T12:08:55.578 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:08:55 smithi029 ceph-mon[31975]: from='client.? 172.21.15.29:0/3374868693' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T12:08:55.613 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:55.624+0000 7fec2a2e8640 1 -- 172.21.15.29:0/3612385166 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7fec24068a20 con 0x7fec24102a20 2024-03-16T12:08:55.615 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:55.626+0000 7fec217fa640 1 -- 172.21.15.29:0/3612385166 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v33) ==== 161+0+0 (secure 0 0 0) 0x7fec1409fd90 con 0x7fec24102a20 2024-03-16T12:08:55.616 INFO:teuthology.orchestra.run.smithi029.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-16T12:08:55.618 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:55.629+0000 7fec2a2e8640 1 -- 172.21.15.29:0/3612385166 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7fec0c0a2970 msgr2=0x7fec0c0a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:08:55.618 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:55.629+0000 7fec2a2e8640 1 --2- 172.21.15.29:0/3612385166 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7fec0c0a2970 0x7fec0c0a4e30 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fec1802efd0 tx=0x7fec1802cf90 comp rx=0 tx=0).stop 2024-03-16T12:08:55.618 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:55.629+0000 7fec2a2e8640 1 -- 172.21.15.29:0/3612385166 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fec24102a20 msgr2=0x7fec24198080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:08:55.618 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:55.629+0000 7fec2a2e8640 1 --2- 172.21.15.29:0/3612385166 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fec24102a20 0x7fec24198080 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7fec1400e9f0 tx=0x7fec1400eeb0 comp rx=0 tx=0).stop 2024-03-16T12:08:55.619 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:55.629+0000 7fec2a2e8640 1 -- 172.21.15.29:0/3612385166 shutdown_connections 2024-03-16T12:08:55.619 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:55.629+0000 7fec2a2e8640 1 --2- 172.21.15.29:0/3612385166 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7fec0c0a2970 0x7fec0c0a4e30 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:08:55.619 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:55.629+0000 7fec2a2e8640 1 --2- 172.21.15.29:0/3612385166 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fec2410bdf0 0x7fec2419c940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:08:55.619 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:55.629+0000 7fec2a2e8640 1 --2- 172.21.15.29:0/3612385166 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fec24103340 0x7fec241985c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:08:55.620 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:55.629+0000 7fec2a2e8640 1 --2- 172.21.15.29:0/3612385166 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fec24102a20 0x7fec24198080 unknown :-1 s=CLOSED pgs=815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:08:55.620 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:55.629+0000 7fec2a2e8640 1 -- 172.21.15.29:0/3612385166 >> 172.21.15.29:0/3612385166 conn(0x7fec24076a60 msgr2=0x7fec240fd580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:08:55.620 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:55.630+0000 7fec2a2e8640 1 -- 172.21.15.29:0/3612385166 shutdown_connections 2024-03-16T12:08:55.620 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:55.630+0000 7fec2a2e8640 1 -- 172.21.15.29:0/3612385166 wait complete. 2024-03-16T12:08:55.630 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-16T12:08:55.738 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:55.746+0000 7f152b189640 1 -- 172.21.15.29:0/3440983218 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1524106660 msgr2=0x7f1524106a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:08:55.738 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:55.746+0000 7f152b189640 1 --2- 172.21.15.29:0/3440983218 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1524106660 0x7f1524106a40 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7f151c009cd0 tx=0x7f151c02c0f0 comp rx=0 tx=0).stop 2024-03-16T12:08:55.738 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:55.746+0000 7f152b189640 1 -- 172.21.15.29:0/3440983218 shutdown_connections 2024-03-16T12:08:55.738 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:55.746+0000 7f152b189640 1 --2- 172.21.15.29:0/3440983218 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1524101010 0x7f152410d520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:08:55.739 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:55.746+0000 7f152b189640 1 --2- 172.21.15.29:0/3440983218 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1524100650 0x7f1524100ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:08:55.739 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:55.746+0000 7f152b189640 1 --2- 172.21.15.29:0/3440983218 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1524106660 0x7f1524106a40 unknown :-1 s=CLOSED pgs=817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:08:55.739 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:55.747+0000 7f152b189640 1 -- 172.21.15.29:0/3440983218 >> 172.21.15.29:0/3440983218 conn(0x7f15240777e0 msgr2=0x7f1524077bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:08:55.739 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:55.747+0000 7f152b189640 1 -- 172.21.15.29:0/3440983218 shutdown_connections 2024-03-16T12:08:55.739 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:55.747+0000 7f152b189640 1 -- 172.21.15.29:0/3440983218 wait complete. 2024-03-16T12:08:55.739 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:55.747+0000 7f152b189640 1 Processor -- start 2024-03-16T12:08:55.739 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:55.747+0000 7f152b189640 1 -- start start 2024-03-16T12:08:55.739 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:55.747+0000 7f152b189640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1524100650 0x7f1524198080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:08:55.739 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:55.747+0000 7f152b189640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1524101010 0x7f15241985c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:08:55.740 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:55.747+0000 7f152b189640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1524106660 0x7f152419c940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:08:55.740 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:55.747+0000 7f152b189640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f1524110a30 con 0x7f1524106660 2024-03-16T12:08:55.740 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:55.747+0000 7f152b189640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f15241108b0 con 0x7f1524101010 2024-03-16T12:08:55.740 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:55.747+0000 7f152b189640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f1524110bb0 con 0x7f1524100650 2024-03-16T12:08:55.740 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:55.747+0000 7f1529726640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1524106660 0x7f152419c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:08:55.740 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:55.747+0000 7f1529726640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1524106660 0x7f152419c940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60470/0 (socket says 172.21.15.29:60470) 2024-03-16T12:08:55.740 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:55.747+0000 7f1529726640 1 -- 172.21.15.29:0/1177978614 learned_addr learned my addr 172.21.15.29:0/1177978614 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T12:08:55.740 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:55.748+0000 7f1523fff640 1 --2- 172.21.15.29:0/1177978614 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1524101010 0x7f15241985c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:08:55.741 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:55.748+0000 7f1528f25640 1 --2- 172.21.15.29:0/1177978614 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1524100650 0x7f1524198080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:08:55.741 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:55.748+0000 7f1529726640 1 -- 172.21.15.29:0/1177978614 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1524100650 msgr2=0x7f1524198080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:08:55.741 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:55.748+0000 7f1529726640 1 --2- 172.21.15.29:0/1177978614 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1524100650 0x7f1524198080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:08:55.741 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:55.748+0000 7f1529726640 1 -- 172.21.15.29:0/1177978614 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1524101010 msgr2=0x7f15241985c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:08:55.741 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:55.748+0000 7f1529726640 1 --2- 172.21.15.29:0/1177978614 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1524101010 0x7f15241985c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:08:55.741 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:55.748+0000 7f1529726640 1 -- 172.21.15.29:0/1177978614 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f152419d070 con 0x7f1524106660 2024-03-16T12:08:55.741 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:55.748+0000 7f1523fff640 1 --2- 172.21.15.29:0/1177978614 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1524101010 0x7f15241985c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T12:08:55.741 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:55.748+0000 7f1529726640 1 --2- 172.21.15.29:0/1177978614 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1524106660 0x7f152419c940 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7f151400d750 tx=0x7f151400dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:08:55.741 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:55.748+0000 7f1521ffb640 1 -- 172.21.15.29:0/1177978614 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1514017070 con 0x7f1524106660 2024-03-16T12:08:55.742 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:55.748+0000 7f152b189640 1 -- 172.21.15.29:0/1177978614 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f152419d360 con 0x7f1524106660 2024-03-16T12:08:55.742 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:55.748+0000 7f1521ffb640 1 -- 172.21.15.29:0/1177978614 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1514005500 con 0x7f1524106660 2024-03-16T12:08:55.742 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:55.748+0000 7f1521ffb640 1 -- 172.21.15.29:0/1177978614 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1514004d90 con 0x7f1524106660 2024-03-16T12:08:55.742 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:55.748+0000 7f152b189640 1 -- 172.21.15.29:0/1177978614 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f15241a4bf0 con 0x7f1524106660 2024-03-16T12:08:55.743 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:55.749+0000 7f152b189640 1 -- 172.21.15.29:0/1177978614 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f15240685d0 con 0x7f1524106660 2024-03-16T12:08:55.743 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:55.750+0000 7f1521ffb640 1 -- 172.21.15.29:0/1177978614 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7f1514004f30 con 0x7f1524106660 2024-03-16T12:08:55.743 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:55.751+0000 7f1521ffb640 1 --2- 172.21.15.29:0/1177978614 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f150c0a2970 0x7f150c0a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:08:55.743 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:55.751+0000 7f1528f25640 1 --2- 172.21.15.29:0/1177978614 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f150c0a2970 0x7f150c0a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:08:55.743 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:55.751+0000 7f1521ffb640 1 -- 172.21.15.29:0/1177978614 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(117..117 src has 1..117) ==== 8770+0+0 (secure 0 0 0) 0x7f15140cea00 con 0x7f1524106660 2024-03-16T12:08:55.743 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:55.751+0000 7f1528f25640 1 --2- 172.21.15.29:0/1177978614 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f150c0a2970 0x7f150c0a4e30 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f151c006010 tx=0x7f151c005c30 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:08:55.744 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:55.755+0000 7f1521ffb640 1 -- 172.21.15.29:0/1177978614 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f151409b600 con 0x7f1524106660 2024-03-16T12:08:55.785 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:08:55 smithi154 ceph-mon[36320]: from='client.? 172.21.15.29:0/2636067881' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T12:08:55.785 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:08:55 smithi154 ceph-mon[36320]: from='client.? 172.21.15.29:0/1281639984' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T12:08:55.786 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:08:55 smithi154 ceph-mon[36320]: from='client.? 172.21.15.29:0/3374868693' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T12:08:55.786 INFO:journalctl@ceph.mds.c.smithi154.stdout:Mar 16 12:08:55 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mds-c[66727]: ignoring --setuser ceph since I am not root 2024-03-16T12:08:55.786 INFO:journalctl@ceph.mds.c.smithi154.stdout:Mar 16 12:08:55 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mds-c[66727]: ignoring --setgroup ceph since I am not root 2024-03-16T12:08:55.786 INFO:journalctl@ceph.mds.c.smithi154.stdout:Mar 16 12:08:55 smithi154 ceph-mds[66731]: -- 172.21.15.154:0/3875595408 <== mon.1 v2:172.21.15.154:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55d51c887a40 con 0x55d51c85b000 2024-03-16T12:08:55.786 INFO:journalctl@ceph.mds.c.smithi154.stdout:Mar 16 12:08:55 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mds-c[66727]: starting mds.c at 2024-03-16T12:08:55.787 INFO:journalctl@ceph.mds.f.smithi154.stdout:Mar 16 12:08:55 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mds-f[66421]: ignoring --setuser ceph since I am not root 2024-03-16T12:08:55.787 INFO:journalctl@ceph.mds.f.smithi154.stdout:Mar 16 12:08:55 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mds-f[66421]: ignoring --setgroup ceph since I am not root 2024-03-16T12:08:55.787 INFO:journalctl@ceph.mds.f.smithi154.stdout:Mar 16 12:08:55 smithi154 ceph-mds[66425]: -- 172.21.15.154:0/2744850516 <== mon.1 v2:172.21.15.154:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5558c29e5a40 con 0x5558c29b8800 2024-03-16T12:08:55.787 INFO:journalctl@ceph.mds.f.smithi154.stdout:Mar 16 12:08:55 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mds-f[66421]: starting mds.f at 2024-03-16T12:08:55.788 INFO:journalctl@ceph.mds.i.smithi154.stdout:Mar 16 12:08:55 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mds-i[67031]: ignoring --setuser ceph since I am not root 2024-03-16T12:08:55.788 INFO:journalctl@ceph.mds.i.smithi154.stdout:Mar 16 12:08:55 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mds-i[67031]: ignoring --setgroup ceph since I am not root 2024-03-16T12:08:55.788 INFO:journalctl@ceph.mds.i.smithi154.stdout:Mar 16 12:08:55 smithi154 ceph-mds[67035]: -- 172.21.15.154:0/680476999 <== mon.1 v2:172.21.15.154:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x555d8fa4fa40 con 0x555d8fa23000 2024-03-16T12:08:55.788 INFO:journalctl@ceph.mds.i.smithi154.stdout:Mar 16 12:08:55 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mds-i[67031]: starting mds.i at 2024-03-16T12:08:55.868 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:08:55 smithi152 ceph-mon[36291]: from='client.? 172.21.15.29:0/2636067881' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T12:08:55.868 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:08:55 smithi152 ceph-mon[36291]: from='client.? 172.21.15.29:0/1281639984' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T12:08:55.868 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:08:55 smithi152 ceph-mon[36291]: from='client.? 172.21.15.29:0/3374868693' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T12:08:55.868 INFO:journalctl@ceph.mds.e.smithi152.stdout:Mar 16 12:08:55 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mds-e[66619]: ignoring --setuser ceph since I am not root 2024-03-16T12:08:55.869 INFO:journalctl@ceph.mds.e.smithi152.stdout:Mar 16 12:08:55 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mds-e[66619]: ignoring --setgroup ceph since I am not root 2024-03-16T12:08:55.869 INFO:journalctl@ceph.mds.e.smithi152.stdout:Mar 16 12:08:55 smithi152 ceph-mds[66623]: -- 172.21.15.152:0/3643727012 <== mon.2 v2:172.21.15.152:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x561a7d5e9a40 con 0x561a7d5bc800 2024-03-16T12:08:55.869 INFO:journalctl@ceph.mds.e.smithi152.stdout:Mar 16 12:08:55 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mds-e[66619]: starting mds.e at 2024-03-16T12:08:55.869 INFO:journalctl@ceph.mds.h.smithi152.stdout:Mar 16 12:08:55 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mds-h[66932]: ignoring --setuser ceph since I am not root 2024-03-16T12:08:55.869 INFO:journalctl@ceph.mds.h.smithi152.stdout:Mar 16 12:08:55 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mds-h[66932]: ignoring --setgroup ceph since I am not root 2024-03-16T12:08:55.869 INFO:journalctl@ceph.mds.h.smithi152.stdout:Mar 16 12:08:55 smithi152 ceph-mds[66936]: -- 172.21.15.152:0/968068866 <== mon.2 v2:172.21.15.152:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55689a0e9a40 con 0x55689a0bcc00 2024-03-16T12:08:55.870 INFO:journalctl@ceph.mds.h.smithi152.stdout:Mar 16 12:08:55 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mds-h[66932]: starting mds.h at 2024-03-16T12:08:55.870 INFO:journalctl@ceph.mds.b.smithi152.stdout:Mar 16 12:08:55 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mds-b[67242]: ignoring --setuser ceph since I am not root 2024-03-16T12:08:55.870 INFO:journalctl@ceph.mds.b.smithi152.stdout:Mar 16 12:08:55 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mds-b[67242]: ignoring --setgroup ceph since I am not root 2024-03-16T12:08:55.870 INFO:journalctl@ceph.mds.b.smithi152.stdout:Mar 16 12:08:55 smithi152 ceph-mds[67246]: -- 172.21.15.152:0/2050663330 <== mon.2 v2:172.21.15.152:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x562016fdfa40 con 0x562016fb2c00 2024-03-16T12:08:55.870 INFO:journalctl@ceph.mds.b.smithi152.stdout:Mar 16 12:08:55 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mds-b[67242]: starting mds.b at 2024-03-16T12:08:55.968 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:55.979+0000 7f152b189640 1 -- 172.21.15.29:0/1177978614 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f1524109c50 con 0x7f1524106660 2024-03-16T12:08:56.078 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:08:55 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mon-a[31971]: 2024-03-16T12:08:55.625+0000 7f693ba19700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-16T12:08:56.617 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:08:56 smithi152 ceph-mon[36291]: pgmap v571: 97 pgs: 97 active+clean; 362 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 60 KiB/s wr, 9 op/s 2024-03-16T12:08:56.618 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:08:56 smithi152 ceph-mon[36291]: from='client.? 172.21.15.29:0/3612385166' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-03-16T12:08:56.618 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:08:56 smithi152 ceph-mon[36291]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-16T12:08:56.618 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:08:56 smithi152 ceph-mon[36291]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-16T12:08:56.618 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:08:56 smithi152 ceph-mon[36291]: from='client.? 172.21.15.29:0/3612385166' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-03-16T12:08:56.618 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:08:56 smithi152 ceph-mon[36291]: osdmap e117: 12 total, 12 up, 12 in 2024-03-16T12:08:56.618 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:08:56 smithi152 ceph-mon[36291]: fsmap cephfs:0/3 6 up:standby, 3 failed 2024-03-16T12:08:56.619 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:08:56 smithi152 ceph-mon[36291]: from='client.? 172.21.15.29:0/1177978614' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-03-16T12:08:56.619 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:56.628+0000 7f1521ffb640 1 -- 172.21.15.29:0/1177978614 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v34) ==== 104+0+0 (secure 0 0 0) 0x7f151409af00 con 0x7f1524106660 2024-03-16T12:08:56.620 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:56.631+0000 7f152b189640 1 -- 172.21.15.29:0/1177978614 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f150c0a2970 msgr2=0x7f150c0a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:08:56.621 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:56.631+0000 7f152b189640 1 --2- 172.21.15.29:0/1177978614 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f150c0a2970 0x7f150c0a4e30 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f151c006010 tx=0x7f151c005c30 comp rx=0 tx=0).stop 2024-03-16T12:08:56.621 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:56.631+0000 7f152b189640 1 -- 172.21.15.29:0/1177978614 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1524106660 msgr2=0x7f152419c940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:08:56.621 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:56.631+0000 7f152b189640 1 --2- 172.21.15.29:0/1177978614 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1524106660 0x7f152419c940 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7f151400d750 tx=0x7f151400dc10 comp rx=0 tx=0).stop 2024-03-16T12:08:56.621 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:56.631+0000 7f152b189640 1 -- 172.21.15.29:0/1177978614 shutdown_connections 2024-03-16T12:08:56.621 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:56.631+0000 7f152b189640 1 --2- 172.21.15.29:0/1177978614 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7f150c0a2970 0x7f150c0a4e30 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:08:56.621 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:56.631+0000 7f152b189640 1 --2- 172.21.15.29:0/1177978614 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1524106660 0x7f152419c940 unknown :-1 s=CLOSED pgs=818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:08:56.622 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:56.631+0000 7f152b189640 1 --2- 172.21.15.29:0/1177978614 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1524101010 0x7f15241985c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:08:56.622 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:56.631+0000 7f152b189640 1 --2- 172.21.15.29:0/1177978614 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1524100650 0x7f1524198080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:08:56.622 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:56.631+0000 7f152b189640 1 -- 172.21.15.29:0/1177978614 >> 172.21.15.29:0/1177978614 conn(0x7f15240777e0 msgr2=0x7f1524068c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:08:56.622 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:56.631+0000 7f152b189640 1 -- 172.21.15.29:0/1177978614 shutdown_connections 2024-03-16T12:08:56.622 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:56.631+0000 7f152b189640 1 -- 172.21.15.29:0/1177978614 wait complete. 2024-03-16T12:08:56.636 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-16T12:08:56.751 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:56.759+0000 7fa86d752640 1 -- 172.21.15.29:0/3433733248 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa868102110 msgr2=0x7fa86810e620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:08:56.751 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:56.759+0000 7fa86d752640 1 --2- 172.21.15.29:0/3433733248 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa868102110 0x7fa86810e620 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7fa860009cd0 tx=0x7fa86002c0f0 comp rx=0 tx=0).stop 2024-03-16T12:08:56.751 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:56.759+0000 7fa86d752640 1 -- 172.21.15.29:0/3433733248 shutdown_connections 2024-03-16T12:08:56.751 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:56.759+0000 7fa86d752640 1 --2- 172.21.15.29:0/3433733248 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa868102110 0x7fa86810e620 unknown :-1 s=CLOSED pgs=819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:08:56.752 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:56.759+0000 7fa86d752640 1 --2- 172.21.15.29:0/3433733248 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa868101750 0x7fa868101bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:08:56.752 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:56.759+0000 7fa86d752640 1 --2- 172.21.15.29:0/3433733248 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa868107760 0x7fa868107b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:08:56.752 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:56.759+0000 7fa86d752640 1 -- 172.21.15.29:0/3433733248 >> 172.21.15.29:0/3433733248 conn(0x7fa8680fd280 msgr2=0x7fa8680ff6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:08:56.752 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:56.759+0000 7fa86d752640 1 -- 172.21.15.29:0/3433733248 shutdown_connections 2024-03-16T12:08:56.752 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:56.759+0000 7fa86d752640 1 -- 172.21.15.29:0/3433733248 wait complete. 2024-03-16T12:08:56.752 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:56.760+0000 7fa86d752640 1 Processor -- start 2024-03-16T12:08:56.752 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:56.760+0000 7fa86d752640 1 -- start start 2024-03-16T12:08:56.752 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:56.760+0000 7fa86d752640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa868101750 0x7fa868198090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:08:56.753 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:56.760+0000 7fa86d752640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa868102110 0x7fa8681985d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:08:56.753 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:56.760+0000 7fa86d752640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa868107760 0x7fa86819c900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:08:56.753 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:56.760+0000 7fa86d752640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fa868111b30 con 0x7fa868102110 2024-03-16T12:08:56.753 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:56.760+0000 7fa86d752640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fa8681119b0 con 0x7fa868107760 2024-03-16T12:08:56.753 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:56.760+0000 7fa86d752640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fa868111cb0 con 0x7fa868101750 2024-03-16T12:08:56.753 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:56.760+0000 7fa8667fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa868102110 0x7fa8681985d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:08:56.753 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:56.760+0000 7fa8667fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa868102110 0x7fa8681985d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60496/0 (socket says 172.21.15.29:60496) 2024-03-16T12:08:56.753 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:56.760+0000 7fa8667fc640 1 -- 172.21.15.29:0/3978767736 learned_addr learned my addr 172.21.15.29:0/3978767736 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T12:08:56.754 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:56.761+0000 7fa866ffd640 1 --2- 172.21.15.29:0/3978767736 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa868101750 0x7fa868198090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:08:56.754 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:56.761+0000 7fa8677fe640 1 --2- 172.21.15.29:0/3978767736 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa868107760 0x7fa86819c900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:08:56.754 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:56.761+0000 7fa8667fc640 1 -- 172.21.15.29:0/3978767736 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa868101750 msgr2=0x7fa868198090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:08:56.754 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:56.761+0000 7fa8667fc640 1 --2- 172.21.15.29:0/3978767736 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa868101750 0x7fa868198090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:08:56.754 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:56.761+0000 7fa8667fc640 1 -- 172.21.15.29:0/3978767736 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa868107760 msgr2=0x7fa86819c900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:08:56.754 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:56.761+0000 7fa8667fc640 1 --2- 172.21.15.29:0/3978767736 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa868107760 0x7fa86819c900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:08:56.754 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:56.761+0000 7fa8667fc640 1 -- 172.21.15.29:0/3978767736 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa86819d030 con 0x7fa868102110 2024-03-16T12:08:56.754 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:56.761+0000 7fa8677fe640 1 --2- 172.21.15.29:0/3978767736 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa868107760 0x7fa86819c900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-16T12:08:56.754 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:56.761+0000 7fa8667fc640 1 --2- 172.21.15.29:0/3978767736 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa868102110 0x7fa8681985d0 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7fa85c00c890 tx=0x7fa85c00cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:08:56.755 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:56.761+0000 7fa84ffff640 1 -- 172.21.15.29:0/3978767736 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa85c004220 con 0x7fa868102110 2024-03-16T12:08:56.755 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:56.761+0000 7fa86d752640 1 -- 172.21.15.29:0/3978767736 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa86819d320 con 0x7fa868102110 2024-03-16T12:08:56.755 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:56.761+0000 7fa84ffff640 1 -- 172.21.15.29:0/3978767736 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa85c0043c0 con 0x7fa868102110 2024-03-16T12:08:56.755 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:56.761+0000 7fa84ffff640 1 -- 172.21.15.29:0/3978767736 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa85c004a00 con 0x7fa868102110 2024-03-16T12:08:56.755 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:56.761+0000 7fa86d752640 1 -- 172.21.15.29:0/3978767736 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa8681a4bb0 con 0x7fa868102110 2024-03-16T12:08:56.756 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:56.762+0000 7fa86d752640 1 -- 172.21.15.29:0/3978767736 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa868068a20 con 0x7fa868102110 2024-03-16T12:08:56.756 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:56.763+0000 7fa84ffff640 1 -- 172.21.15.29:0/3978767736 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7fa85c0079b0 con 0x7fa868102110 2024-03-16T12:08:56.756 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:56.764+0000 7fa84ffff640 1 --2- 172.21.15.29:0/3978767736 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7fa8500a2970 0x7fa8500a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:08:56.756 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:56.764+0000 7fa866ffd640 1 --2- 172.21.15.29:0/3978767736 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7fa8500a2970 0x7fa8500a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:08:56.756 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:56.764+0000 7fa84ffff640 1 -- 172.21.15.29:0/3978767736 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(117..117 src has 1..117) ==== 8770+0+0 (secure 0 0 0) 0x7fa85c0cece0 con 0x7fa868102110 2024-03-16T12:08:56.757 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:56.764+0000 7fa866ffd640 1 --2- 172.21.15.29:0/3978767736 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7fa8500a2970 0x7fa8500a4e30 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fa858005d90 tx=0x7fa858005d20 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:08:56.757 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:56.768+0000 7fa84ffff640 1 -- 172.21.15.29:0/3978767736 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fa85c014030 con 0x7fa868102110 2024-03-16T12:08:56.785 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:08:56 smithi154 ceph-mon[36320]: pgmap v571: 97 pgs: 97 active+clean; 362 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 60 KiB/s wr, 9 op/s 2024-03-16T12:08:56.785 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:08:56 smithi154 ceph-mon[36320]: from='client.? 172.21.15.29:0/3612385166' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-03-16T12:08:56.785 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:08:56 smithi154 ceph-mon[36320]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-16T12:08:56.785 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:08:56 smithi154 ceph-mon[36320]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-16T12:08:56.786 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:08:56 smithi154 ceph-mon[36320]: from='client.? 172.21.15.29:0/3612385166' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-03-16T12:08:56.786 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:08:56 smithi154 ceph-mon[36320]: osdmap e117: 12 total, 12 up, 12 in 2024-03-16T12:08:56.786 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:08:56 smithi154 ceph-mon[36320]: fsmap cephfs:0/3 6 up:standby, 3 failed 2024-03-16T12:08:56.786 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:08:56 smithi154 ceph-mon[36320]: from='client.? 172.21.15.29:0/1177978614' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-03-16T12:08:56.828 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:08:56 smithi029 ceph-mon[31975]: pgmap v571: 97 pgs: 97 active+clean; 362 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 60 KiB/s wr, 9 op/s 2024-03-16T12:08:56.828 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:08:56 smithi029 ceph-mon[31975]: from='client.? 172.21.15.29:0/3612385166' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-03-16T12:08:56.828 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:08:56 smithi029 ceph-mon[31975]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-16T12:08:56.828 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:08:56 smithi029 ceph-mon[31975]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-16T12:08:56.828 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:08:56 smithi029 ceph-mon[31975]: from='client.? 172.21.15.29:0/3612385166' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-03-16T12:08:56.828 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:08:56 smithi029 ceph-mon[31975]: osdmap e117: 12 total, 12 up, 12 in 2024-03-16T12:08:56.828 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:08:56 smithi029 ceph-mon[31975]: fsmap cephfs:0/3 6 up:standby, 3 failed 2024-03-16T12:08:56.828 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:08:56 smithi029 ceph-mon[31975]: from='client.? 172.21.15.29:0/1177978614' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-03-16T12:08:56.940 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:56.951+0000 7fa86d752640 1 -- 172.21.15.29:0/3978767736 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7fa86802d1b0 con 0x7fa868102110 2024-03-16T12:08:57.622 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:57.633+0000 7fa84ffff640 1 -- 172.21.15.29:0/3978767736 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v118) ==== 181+0+0 (secure 0 0 0) 0x7fa85c09b940 con 0x7fa868102110 2024-03-16T12:08:57.623 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'cephfs_metadata' removed 2024-03-16T12:08:57.627 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:57.637+0000 7fa86d752640 1 -- 172.21.15.29:0/3978767736 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7fa8500a2970 msgr2=0x7fa8500a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:08:57.627 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:57.637+0000 7fa86d752640 1 --2- 172.21.15.29:0/3978767736 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7fa8500a2970 0x7fa8500a4e30 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fa858005d90 tx=0x7fa858005d20 comp rx=0 tx=0).stop 2024-03-16T12:08:57.627 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:57.637+0000 7fa86d752640 1 -- 172.21.15.29:0/3978767736 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa868102110 msgr2=0x7fa8681985d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:08:57.627 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:57.637+0000 7fa86d752640 1 --2- 172.21.15.29:0/3978767736 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa868102110 0x7fa8681985d0 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7fa85c00c890 tx=0x7fa85c00cd50 comp rx=0 tx=0).stop 2024-03-16T12:08:57.629 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:57.640+0000 7fa86d752640 1 -- 172.21.15.29:0/3978767736 shutdown_connections 2024-03-16T12:08:57.629 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:57.640+0000 7fa86d752640 1 --2- 172.21.15.29:0/3978767736 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7fa8500a2970 0x7fa8500a4e30 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:08:57.629 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:57.640+0000 7fa86d752640 1 --2- 172.21.15.29:0/3978767736 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa868107760 0x7fa86819c900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:08:57.629 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:57.640+0000 7fa86d752640 1 --2- 172.21.15.29:0/3978767736 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa868102110 0x7fa8681985d0 unknown :-1 s=CLOSED pgs=820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:08:57.629 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:57.640+0000 7fa86d752640 1 --2- 172.21.15.29:0/3978767736 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa868101750 0x7fa868198090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:08:57.630 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:57.640+0000 7fa86d752640 1 -- 172.21.15.29:0/3978767736 >> 172.21.15.29:0/3978767736 conn(0x7fa8680fd280 msgr2=0x7fa8680fdaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:08:57.630 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:57.640+0000 7fa86d752640 1 -- 172.21.15.29:0/3978767736 shutdown_connections 2024-03-16T12:08:57.630 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:57.640+0000 7fa86d752640 1 -- 172.21.15.29:0/3978767736 wait complete. 2024-03-16T12:08:57.647 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-16T12:08:57.764 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:57.774+0000 7fd7ca044640 1 -- 172.21.15.29:0/1323007141 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd7c4103370 msgr2=0x7fd7c410b850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:08:57.764 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:57.774+0000 7fd7ca044640 1 --2- 172.21.15.29:0/1323007141 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd7c4103370 0x7fd7c410b850 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7fd7b8009900 tx=0x7fd7b802bcd0 comp rx=0 tx=0).stop 2024-03-16T12:08:57.764 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:57.774+0000 7fd7ca044640 1 -- 172.21.15.29:0/1323007141 shutdown_connections 2024-03-16T12:08:57.764 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:57.774+0000 7fd7ca044640 1 --2- 172.21.15.29:0/1323007141 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd7c410be20 0x7fd7c410e290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:08:57.764 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:57.774+0000 7fd7ca044640 1 --2- 172.21.15.29:0/1323007141 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd7c4103370 0x7fd7c410b850 unknown :-1 s=CLOSED pgs=821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:08:57.764 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:57.774+0000 7fd7ca044640 1 --2- 172.21.15.29:0/1323007141 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd7c4102a50 0x7fd7c4102e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:08:57.765 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:57.774+0000 7fd7ca044640 1 -- 172.21.15.29:0/1323007141 >> 172.21.15.29:0/1323007141 conn(0x7fd7c4077840 msgr2=0x7fd7c4077c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:08:57.765 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:57.774+0000 7fd7ca044640 1 -- 172.21.15.29:0/1323007141 shutdown_connections 2024-03-16T12:08:57.765 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:57.774+0000 7fd7ca044640 1 -- 172.21.15.29:0/1323007141 wait complete. 2024-03-16T12:08:57.765 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:57.774+0000 7fd7ca044640 1 Processor -- start 2024-03-16T12:08:57.765 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:57.775+0000 7fd7ca044640 1 -- start start 2024-03-16T12:08:57.768 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:57.775+0000 7fd7ca044640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd7c4102a50 0x7fd7c4198150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:08:57.768 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:57.775+0000 7fd7ca044640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd7c4103370 0x7fd7c4198690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:08:57.768 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:57.775+0000 7fd7ca044640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd7c410be20 0x7fd7c419ca10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:08:57.768 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:57.775+0000 7fd7ca044640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fd7c4111bf0 con 0x7fd7c4102a50 2024-03-16T12:08:57.768 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:57.775+0000 7fd7ca044640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fd7c4111a70 con 0x7fd7c410be20 2024-03-16T12:08:57.768 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:57.775+0000 7fd7ca044640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fd7c4111d70 con 0x7fd7c4103370 2024-03-16T12:08:57.768 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:57.775+0000 7fd7c37fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd7c4102a50 0x7fd7c4198150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:08:57.769 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:57.775+0000 7fd7c37fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd7c4102a50 0x7fd7c4198150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60516/0 (socket says 172.21.15.29:60516) 2024-03-16T12:08:57.769 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:57.775+0000 7fd7c37fe640 1 -- 172.21.15.29:0/1042778405 learned_addr learned my addr 172.21.15.29:0/1042778405 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-03-16T12:08:57.769 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:57.775+0000 7fd7c2ffd640 1 --2- 172.21.15.29:0/1042778405 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd7c4103370 0x7fd7c4198690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:08:57.769 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:57.775+0000 7fd7c3fff640 1 --2- 172.21.15.29:0/1042778405 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd7c410be20 0x7fd7c419ca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:08:57.769 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:57.775+0000 7fd7c37fe640 1 -- 172.21.15.29:0/1042778405 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd7c4103370 msgr2=0x7fd7c4198690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:08:57.769 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:57.775+0000 7fd7c37fe640 1 --2- 172.21.15.29:0/1042778405 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd7c4103370 0x7fd7c4198690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:08:57.769 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:57.775+0000 7fd7c37fe640 1 -- 172.21.15.29:0/1042778405 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd7c410be20 msgr2=0x7fd7c419ca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:08:57.769 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:57.775+0000 7fd7c37fe640 1 --2- 172.21.15.29:0/1042778405 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd7c410be20 0x7fd7c419ca10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:08:57.770 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:57.775+0000 7fd7c37fe640 1 -- 172.21.15.29:0/1042778405 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7c419d140 con 0x7fd7c4102a50 2024-03-16T12:08:57.770 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:57.775+0000 7fd7c3fff640 1 --2- 172.21.15.29:0/1042778405 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd7c410be20 0x7fd7c419ca10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T12:08:57.770 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:57.776+0000 7fd7c37fe640 1 --2- 172.21.15.29:0/1042778405 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd7c4102a50 0x7fd7c4198150 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7fd7b400c970 tx=0x7fd7b400ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:08:57.770 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:57.776+0000 7fd7c0ff9640 1 -- 172.21.15.29:0/1042778405 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7b4007bb0 con 0x7fd7c4102a50 2024-03-16T12:08:57.770 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:57.776+0000 7fd7ca044640 1 -- 172.21.15.29:0/1042778405 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd7c419d430 con 0x7fd7c4102a50 2024-03-16T12:08:57.770 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:57.776+0000 7fd7c0ff9640 1 -- 172.21.15.29:0/1042778405 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd7b4007d50 con 0x7fd7c4102a50 2024-03-16T12:08:57.770 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:57.776+0000 7fd7c0ff9640 1 -- 172.21.15.29:0/1042778405 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7b400f450 con 0x7fd7c4102a50 2024-03-16T12:08:57.770 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:57.776+0000 7fd7ca044640 1 -- 172.21.15.29:0/1042778405 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd7c41a4cc0 con 0x7fd7c4102a50 2024-03-16T12:08:57.770 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:57.777+0000 7fd7ca044640 1 -- 172.21.15.29:0/1042778405 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd7c4068a20 con 0x7fd7c4102a50 2024-03-16T12:08:57.771 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:57.778+0000 7fd7c0ff9640 1 -- 172.21.15.29:0/1042778405 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7fd7b4004fe0 con 0x7fd7c4102a50 2024-03-16T12:08:57.771 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:57.778+0000 7fd7c0ff9640 1 --2- 172.21.15.29:0/1042778405 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7fd7ac0a2970 0x7fd7ac0a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T12:08:57.771 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:57.779+0000 7fd7c2ffd640 1 --2- 172.21.15.29:0/1042778405 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7fd7ac0a2970 0x7fd7ac0a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T12:08:57.771 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:57.779+0000 7fd7c0ff9640 1 -- 172.21.15.29:0/1042778405 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(118..118 src has 1..118) ==== 8216+0+0 (secure 0 0 0) 0x7fd7b40cef60 con 0x7fd7c4102a50 2024-03-16T12:08:57.771 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:57.779+0000 7fd7c2ffd640 1 --2- 172.21.15.29:0/1042778405 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7fd7ac0a2970 0x7fd7ac0a4e30 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fd7b8005ab0 tx=0x7fd7b802cf90 comp rx=0 tx=0).ready entity=mgr.15333 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T12:08:57.772 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:57.782+0000 7fd7c0ff9640 1 -- 172.21.15.29:0/1042778405 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fd7b4014030 con 0x7fd7c4102a50 2024-03-16T12:08:57.785 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:08:57 smithi154 ceph-mon[36320]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-16T12:08:57.786 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:08:57 smithi154 ceph-mon[36320]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-03-16T12:08:57.786 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:08:57 smithi154 ceph-mon[36320]: Cluster is now healthy 2024-03-16T12:08:57.786 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:08:57 smithi154 ceph-mon[36320]: from='client.? 172.21.15.29:0/1177978614' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-03-16T12:08:57.786 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:08:57 smithi154 ceph-mon[36320]: mds.? [v2:172.21.15.154:6836/1067792863,v1:172.21.15.154:6838/1067792863] up:boot 2024-03-16T12:08:57.786 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:08:57 smithi154 ceph-mon[36320]: mds.? [v2:172.21.15.154:6834/2116884753,v1:172.21.15.154:6835/2116884753] up:boot 2024-03-16T12:08:57.786 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:08:57 smithi154 ceph-mon[36320]: mds.? [v2:172.21.15.154:6837/1342931955,v1:172.21.15.154:6839/1342931955] up:boot 2024-03-16T12:08:57.786 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:08:57 smithi154 ceph-mon[36320]: mds.? [v2:172.21.15.152:6835/1745332292,v1:172.21.15.152:6837/1745332292] up:boot 2024-03-16T12:08:57.787 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:08:57 smithi154 ceph-mon[36320]: mds.? [v2:172.21.15.152:6836/1869565801,v1:172.21.15.152:6838/1869565801] up:boot 2024-03-16T12:08:57.787 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:08:57 smithi154 ceph-mon[36320]: mds.? [v2:172.21.15.152:6834/2125871571,v1:172.21.15.152:6839/2125871571] up:boot 2024-03-16T12:08:57.787 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:08:57 smithi154 ceph-mon[36320]: fsmap 12 up:standby 2024-03-16T12:08:57.787 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:08:57 smithi154 ceph-mon[36320]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-16T12:08:57.787 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:08:57 smithi154 ceph-mon[36320]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-16T12:08:57.787 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:08:57 smithi154 ceph-mon[36320]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-16T12:08:57.787 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:08:57 smithi154 ceph-mon[36320]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-16T12:08:57.787 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:08:57 smithi154 ceph-mon[36320]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-16T12:08:57.788 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:08:57 smithi154 ceph-mon[36320]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-16T12:08:57.788 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:08:57 smithi154 ceph-mon[36320]: from='client.? 172.21.15.29:0/3978767736' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-16T12:08:57.828 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:08:57 smithi029 ceph-mon[31975]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-16T12:08:57.828 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:08:57 smithi029 ceph-mon[31975]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-03-16T12:08:57.828 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:08:57 smithi029 ceph-mon[31975]: Cluster is now healthy 2024-03-16T12:08:57.828 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:08:57 smithi029 ceph-mon[31975]: from='client.? 172.21.15.29:0/1177978614' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-03-16T12:08:57.829 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:08:57 smithi029 ceph-mon[31975]: mds.? [v2:172.21.15.154:6836/1067792863,v1:172.21.15.154:6838/1067792863] up:boot 2024-03-16T12:08:57.829 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:08:57 smithi029 ceph-mon[31975]: mds.? [v2:172.21.15.154:6834/2116884753,v1:172.21.15.154:6835/2116884753] up:boot 2024-03-16T12:08:57.829 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:08:57 smithi029 ceph-mon[31975]: mds.? [v2:172.21.15.154:6837/1342931955,v1:172.21.15.154:6839/1342931955] up:boot 2024-03-16T12:08:57.829 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:08:57 smithi029 ceph-mon[31975]: mds.? [v2:172.21.15.152:6835/1745332292,v1:172.21.15.152:6837/1745332292] up:boot 2024-03-16T12:08:57.829 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:08:57 smithi029 ceph-mon[31975]: mds.? [v2:172.21.15.152:6836/1869565801,v1:172.21.15.152:6838/1869565801] up:boot 2024-03-16T12:08:57.829 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:08:57 smithi029 ceph-mon[31975]: mds.? [v2:172.21.15.152:6834/2125871571,v1:172.21.15.152:6839/2125871571] up:boot 2024-03-16T12:08:57.829 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:08:57 smithi029 ceph-mon[31975]: fsmap 12 up:standby 2024-03-16T12:08:57.829 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:08:57 smithi029 ceph-mon[31975]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-16T12:08:57.830 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:08:57 smithi029 ceph-mon[31975]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-16T12:08:57.830 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:08:57 smithi029 ceph-mon[31975]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-16T12:08:57.830 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:08:57 smithi029 ceph-mon[31975]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-16T12:08:57.830 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:08:57 smithi029 ceph-mon[31975]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-16T12:08:57.830 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:08:57 smithi029 ceph-mon[31975]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-16T12:08:57.830 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:08:57 smithi029 ceph-mon[31975]: from='client.? 172.21.15.29:0/3978767736' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-16T12:08:57.868 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:08:57 smithi152 ceph-mon[36291]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-16T12:08:57.868 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:08:57 smithi152 ceph-mon[36291]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-03-16T12:08:57.868 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:08:57 smithi152 ceph-mon[36291]: Cluster is now healthy 2024-03-16T12:08:57.868 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:08:57 smithi152 ceph-mon[36291]: from='client.? 172.21.15.29:0/1177978614' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-03-16T12:08:57.868 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:08:57 smithi152 ceph-mon[36291]: mds.? [v2:172.21.15.154:6836/1067792863,v1:172.21.15.154:6838/1067792863] up:boot 2024-03-16T12:08:57.868 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:08:57 smithi152 ceph-mon[36291]: mds.? [v2:172.21.15.154:6834/2116884753,v1:172.21.15.154:6835/2116884753] up:boot 2024-03-16T12:08:57.868 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:08:57 smithi152 ceph-mon[36291]: mds.? [v2:172.21.15.154:6837/1342931955,v1:172.21.15.154:6839/1342931955] up:boot 2024-03-16T12:08:57.868 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:08:57 smithi152 ceph-mon[36291]: mds.? [v2:172.21.15.152:6835/1745332292,v1:172.21.15.152:6837/1745332292] up:boot 2024-03-16T12:08:57.869 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:08:57 smithi152 ceph-mon[36291]: mds.? [v2:172.21.15.152:6836/1869565801,v1:172.21.15.152:6838/1869565801] up:boot 2024-03-16T12:08:57.869 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:08:57 smithi152 ceph-mon[36291]: mds.? [v2:172.21.15.152:6834/2125871571,v1:172.21.15.152:6839/2125871571] up:boot 2024-03-16T12:08:57.869 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:08:57 smithi152 ceph-mon[36291]: fsmap 12 up:standby 2024-03-16T12:08:57.869 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:08:57 smithi152 ceph-mon[36291]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-16T12:08:57.869 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:08:57 smithi152 ceph-mon[36291]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-16T12:08:57.869 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:08:57 smithi152 ceph-mon[36291]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-16T12:08:57.869 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:08:57 smithi152 ceph-mon[36291]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-16T12:08:57.869 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:08:57 smithi152 ceph-mon[36291]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-16T12:08:57.869 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:08:57 smithi152 ceph-mon[36291]: from='mgr.15333 172.21.15.29:0/702283980' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-16T12:08:57.870 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:08:57 smithi152 ceph-mon[36291]: from='client.? 172.21.15.29:0/3978767736' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-16T12:08:57.957 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:57.968+0000 7fd7ca044640 1 -- 172.21.15.29:0/1042778405 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7fd7c4106f10 con 0x7fd7c4102a50 2024-03-16T12:08:58.628 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:58.639+0000 7fd7c0ff9640 1 -- 172.21.15.29:0/1042778405 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v119) ==== 169+0+0 (secure 0 0 0) 0x7fd7b409bd90 con 0x7fd7c4102a50 2024-03-16T12:08:58.629 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'cephfs_data' removed 2024-03-16T12:08:58.632 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:58.642+0000 7fd7ca044640 1 -- 172.21.15.29:0/1042778405 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7fd7ac0a2970 msgr2=0x7fd7ac0a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:08:58.632 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:58.642+0000 7fd7ca044640 1 --2- 172.21.15.29:0/1042778405 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7fd7ac0a2970 0x7fd7ac0a4e30 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fd7b8005ab0 tx=0x7fd7b802cf90 comp rx=0 tx=0).stop 2024-03-16T12:08:58.632 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:58.642+0000 7fd7ca044640 1 -- 172.21.15.29:0/1042778405 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd7c4102a50 msgr2=0x7fd7c4198150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T12:08:58.632 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:58.642+0000 7fd7ca044640 1 --2- 172.21.15.29:0/1042778405 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd7c4102a50 0x7fd7c4198150 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7fd7b400c970 tx=0x7fd7b400ce30 comp rx=0 tx=0).stop 2024-03-16T12:08:58.633 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:58.642+0000 7fd7ca044640 1 -- 172.21.15.29:0/1042778405 shutdown_connections 2024-03-16T12:08:58.633 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:58.642+0000 7fd7ca044640 1 --2- 172.21.15.29:0/1042778405 >> [v2:172.21.15.29:6800/1188678802,v1:172.21.15.29:6801/1188678802] conn(0x7fd7ac0a2970 0x7fd7ac0a4e30 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:08:58.633 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:58.642+0000 7fd7ca044640 1 --2- 172.21.15.29:0/1042778405 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd7c410be20 0x7fd7c419ca10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:08:58.633 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:58.642+0000 7fd7ca044640 1 --2- 172.21.15.29:0/1042778405 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd7c4103370 0x7fd7c4198690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:08:58.633 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:58.642+0000 7fd7ca044640 1 --2- 172.21.15.29:0/1042778405 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd7c4102a50 0x7fd7c4198150 unknown :-1 s=CLOSED pgs=822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T12:08:58.633 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:58.642+0000 7fd7ca044640 1 -- 172.21.15.29:0/1042778405 >> 172.21.15.29:0/1042778405 conn(0x7fd7c4077840 msgr2=0x7fd7c40fd850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T12:08:58.633 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:58.643+0000 7fd7ca044640 1 -- 172.21.15.29:0/1042778405 shutdown_connections 2024-03-16T12:08:58.633 INFO:teuthology.orchestra.run.smithi029.stderr:2024-03-16T12:08:58.643+0000 7fd7ca044640 1 -- 172.21.15.29:0/1042778405 wait complete. 2024-03-16T12:08:58.651 DEBUG:teuthology.orchestra.run.smithi029:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-16T12:08:58.686 DEBUG:teuthology.orchestra.run.smithi152:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-16T12:08:58.717 DEBUG:teuthology.orchestra.run.smithi154:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-16T12:08:58.754 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-03-16T12:08:58.755 DEBUG:teuthology.orchestra.run.smithi029:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-16T12:08:58.775 DEBUG:teuthology.orchestra.run.smithi152:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-16T12:08:58.790 DEBUG:teuthology.orchestra.run.smithi154:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-16T12:08:58.812 INFO:tasks.cephadm:Stopping all daemons... 2024-03-16T12:08:58.812 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-03-16T12:08:58.812 DEBUG:teuthology.orchestra.run.smithi029:> sudo systemctl stop ceph-d1887f28-e389-11ee-95c9-87774f69a715@mon.a 2024-03-16T12:08:59.035 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:08:58 smithi154 ceph-mon[36320]: pgmap v573: 97 pgs: 97 active+clean; 363 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 172 KiB/s rd, 178 KiB/s wr, 12 op/s 2024-03-16T12:08:59.035 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:08:58 smithi154 ceph-mon[36320]: from='client.? 172.21.15.29:0/3978767736' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-03-16T12:08:59.035 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:08:58 smithi154 ceph-mon[36320]: osdmap e118: 12 total, 12 up, 12 in 2024-03-16T12:08:59.036 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:08:58 smithi154 ceph-mon[36320]: from='client.? 172.21.15.29:0/1042778405' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-16T12:08:59.073 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:08:58 smithi029 ceph-mon[31975]: pgmap v573: 97 pgs: 97 active+clean; 363 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 172 KiB/s rd, 178 KiB/s wr, 12 op/s 2024-03-16T12:08:59.073 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:08:58 smithi029 ceph-mon[31975]: from='client.? 172.21.15.29:0/3978767736' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-03-16T12:08:59.073 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:08:58 smithi029 ceph-mon[31975]: osdmap e118: 12 total, 12 up, 12 in 2024-03-16T12:08:59.073 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:08:58 smithi029 ceph-mon[31975]: from='client.? 172.21.15.29:0/1042778405' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-16T12:08:59.073 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:08:58 smithi029 systemd[1]: Stopping Ceph mon.a for d1887f28-e389-11ee-95c9-87774f69a715... 2024-03-16T12:08:59.117 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:08:58 smithi152 ceph-mon[36291]: pgmap v573: 97 pgs: 97 active+clean; 363 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 172 KiB/s rd, 178 KiB/s wr, 12 op/s 2024-03-16T12:08:59.118 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:08:58 smithi152 ceph-mon[36291]: from='client.? 172.21.15.29:0/3978767736' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-03-16T12:08:59.118 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:08:58 smithi152 ceph-mon[36291]: osdmap e118: 12 total, 12 up, 12 in 2024-03-16T12:08:59.118 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:08:58 smithi152 ceph-mon[36291]: from='client.? 172.21.15.29:0/1042778405' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-16T12:08:59.328 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:08:59 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mon-a[31971]: 2024-03-16T12:08:59.148+0000 7f6941224700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-16T12:08:59.328 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:08:59 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mon-a[31971]: 2024-03-16T12:08:59.148+0000 7f6941224700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-03-16T12:08:59.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:08:59 smithi029 podman[94799]: 2024-03-16 12:08:59.38449696 +0000 UTC m=+0.351128186 container died 4277a6990573405f3d85e1862504a887d6b73d20f334d2d4896eac22a55e389d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af, name=ceph-d1887f28-e389-11ee-95c9-87774f69a715-mon-a, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-16T12:09:00.035 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:08:59 smithi029 podman[94799]: 2024-03-16 12:08:59.777669361 +0000 UTC m=+0.744300592 container cleanup 4277a6990573405f3d85e1862504a887d6b73d20f334d2d4896eac22a55e389d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af, name=ceph-d1887f28-e389-11ee-95c9-87774f69a715-mon-a, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-03-16T12:09:00.036 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 16 12:08:59 smithi029 bash[94799]: ceph-d1887f28-e389-11ee-95c9-87774f69a715-mon-a 2024-03-16T12:09:00.195 DEBUG:teuthology.orchestra.run.smithi029:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d1887f28-e389-11ee-95c9-87774f69a715@mon.a.service' 2024-03-16T12:09:00.246 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-16T12:09:00.247 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-03-16T12:09:00.247 INFO:tasks.cephadm.mon.c:Stopping mon.b... 2024-03-16T12:09:00.247 DEBUG:teuthology.orchestra.run.smithi152:> sudo systemctl stop ceph-d1887f28-e389-11ee-95c9-87774f69a715@mon.b 2024-03-16T12:09:00.601 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:09:00 smithi152 systemd[1]: Stopping Ceph mon.b for d1887f28-e389-11ee-95c9-87774f69a715... 2024-03-16T12:09:00.867 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:09:00 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mon-b[36287]: 2024-03-16T12:09:00.611+0000 7fc7f6560700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-16T12:09:00.868 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:09:00 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mon-b[36287]: 2024-03-16T12:09:00.611+0000 7fc7f6560700 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2024-03-16T12:09:00.868 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:09:00 smithi152 podman[73662]: 2024-03-16 12:09:00.805906284 +0000 UTC m=+0.310128686 container died 315b459181464679894a446a6cdc2dc8623c966f3b2fb42bda423483e205f7bd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d1887f28-e389-11ee-95c9-87774f69a715-mon-b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b) 2024-03-16T12:09:01.078 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 16 12:09:00 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-x[32210]: 2024-03-16T12:09:00.783+0000 7fa14a351700 -1 client.15357: SimpleRADOSStriper: lock: snap_db_v0.db: lock failed: (2) No such file or directory 2024-03-16T12:09:01.078 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 16 12:09:00 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-x[32210]: 2024-03-16T12:09:00.785+0000 7fa14a351700 -1 client.15357: SimpleRADOSStriper: lock: snap_db_v0.db: lock failed: (2) No such file or directory 2024-03-16T12:09:01.367 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:09:01 smithi152 podman[73662]: 2024-03-16 12:09:01.212874647 +0000 UTC m=+0.717097056 container cleanup 315b459181464679894a446a6cdc2dc8623c966f3b2fb42bda423483e205f7bd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d1887f28-e389-11ee-95c9-87774f69a715-mon-b, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-03-16T12:09:01.368 INFO:journalctl@ceph.mon.b.smithi152.stdout:Mar 16 12:09:01 smithi152 bash[73662]: ceph-d1887f28-e389-11ee-95c9-87774f69a715-mon-b 2024-03-16T12:09:02.038 DEBUG:teuthology.orchestra.run.smithi152:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d1887f28-e389-11ee-95c9-87774f69a715@mon.b.service' 2024-03-16T12:09:02.093 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-16T12:09:02.094 INFO:tasks.cephadm.mon.c:Stopped mon.b 2024-03-16T12:09:02.094 INFO:tasks.cephadm.mon.c:Stopping mon.c... 2024-03-16T12:09:02.094 DEBUG:teuthology.orchestra.run.smithi154:> sudo systemctl stop ceph-d1887f28-e389-11ee-95c9-87774f69a715@mon.c 2024-03-16T12:09:02.411 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:09:02 smithi154 systemd[1]: Stopping Ceph mon.c for d1887f28-e389-11ee-95c9-87774f69a715... 2024-03-16T12:09:02.785 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:09:02 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mon-c[36305]: 2024-03-16T12:09:02.421+0000 7f223dcf5700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-16T12:09:02.785 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:09:02 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mon-c[36305]: 2024-03-16T12:09:02.421+0000 7f223dcf5700 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2024-03-16T12:09:02.785 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:09:02 smithi154 podman[74631]: 2024-03-16 12:09:02.574849777 +0000 UTC m=+0.276678134 container died 79732bdfb11e0070e64dcea232db0dced05450c207f15255f8c8ea2eb58f0fb1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d1887f28-e389-11ee-95c9-87774f69a715-mon-c, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True) 2024-03-16T12:09:03.285 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:09:02 smithi154 podman[74631]: 2024-03-16 12:09:02.981174114 +0000 UTC m=+0.683002450 container cleanup 79732bdfb11e0070e64dcea232db0dced05450c207f15255f8c8ea2eb58f0fb1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d1887f28-e389-11ee-95c9-87774f69a715-mon-c, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True) 2024-03-16T12:09:03.285 INFO:journalctl@ceph.mon.c.smithi154.stdout:Mar 16 12:09:02 smithi154 bash[74631]: ceph-d1887f28-e389-11ee-95c9-87774f69a715-mon-c 2024-03-16T12:09:03.533 DEBUG:teuthology.orchestra.run.smithi154:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d1887f28-e389-11ee-95c9-87774f69a715@mon.c.service' 2024-03-16T12:09:03.578 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-16T12:09:03.579 INFO:tasks.cephadm.mon.c:Stopped mon.c 2024-03-16T12:09:03.579 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-03-16T12:09:03.579 DEBUG:teuthology.orchestra.run.smithi029:> sudo systemctl stop ceph-d1887f28-e389-11ee-95c9-87774f69a715@mgr.x 2024-03-16T12:09:03.908 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 16 12:09:03 smithi029 systemd[1]: Stopping Ceph mgr.x for d1887f28-e389-11ee-95c9-87774f69a715... 2024-03-16T12:09:04.281 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 16 12:09:03 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-x[32210]: 2024-03-16T12:09:03.919+0000 7fa1fe789700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.x -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-16T12:09:04.902 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 16 12:09:03 smithi029 podman[94931]: 2024-03-16 12:09:03.949960172 +0000 UTC m=+0.152542598 container died 4ae154094330b5ca37cf71565bb7076bf30261ef14b9197acf83d63f1bf85c9f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af, name=ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-x, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-16T12:09:05.022 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 16 12:09:04 smithi029 podman[94931]: 2024-03-16 12:09:04.706325216 +0000 UTC m=+0.908907636 container cleanup 4ae154094330b5ca37cf71565bb7076bf30261ef14b9197acf83d63f1bf85c9f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af, name=ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-x, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-16T12:09:05.022 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Mar 16 12:09:04 smithi029 bash[94931]: ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-x 2024-03-16T12:09:05.177 DEBUG:teuthology.orchestra.run.smithi029:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d1887f28-e389-11ee-95c9-87774f69a715@mgr.x.service' 2024-03-16T12:09:05.215 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-16T12:09:05.215 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-03-16T12:09:05.215 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-03-16T12:09:05.216 DEBUG:teuthology.orchestra.run.smithi152:> sudo systemctl stop ceph-d1887f28-e389-11ee-95c9-87774f69a715@mgr.y 2024-03-16T12:09:05.580 INFO:journalctl@ceph.mgr.y.smithi152.stdout:Mar 16 12:09:05 smithi152 systemd[1]: Stopping Ceph mgr.y for d1887f28-e389-11ee-95c9-87774f69a715... 2024-03-16T12:09:05.860 INFO:journalctl@ceph.mgr.y.smithi152.stdout:Mar 16 12:09:05 smithi152 podman[73785]: 2024-03-16 12:09:05.590771686 +0000 UTC m=+0.163849201 container died eec9dd63927ae55b173ae886cdfe60f2852fe2cef0c8382946c155134dd686ec (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-y, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.5, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-03-16T12:09:06.117 INFO:journalctl@ceph.mgr.y.smithi152.stdout:Mar 16 12:09:05 smithi152 podman[73785]: 2024-03-16 12:09:05.979408213 +0000 UTC m=+0.552485722 container cleanup eec9dd63927ae55b173ae886cdfe60f2852fe2cef0c8382946c155134dd686ec (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-y, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-16T12:09:06.117 INFO:journalctl@ceph.mgr.y.smithi152.stdout:Mar 16 12:09:05 smithi152 bash[73785]: ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-y 2024-03-16T12:09:06.534 DEBUG:teuthology.orchestra.run.smithi152:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d1887f28-e389-11ee-95c9-87774f69a715@mgr.y.service' 2024-03-16T12:09:06.598 INFO:journalctl@ceph.mgr.y.smithi152.stdout:Mar 16 12:09:06 smithi152 podman[73800]: 2024-03-16 12:09:06.356919695 +0000 UTC m=+0.761306600 container remove eec9dd63927ae55b173ae886cdfe60f2852fe2cef0c8382946c155134dd686ec (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-y, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-03-16T12:09:06.598 INFO:journalctl@ceph.mgr.y.smithi152.stdout:Mar 16 12:09:06 smithi152 systemd[1]: ceph-d1887f28-e389-11ee-95c9-87774f69a715@mgr.y.service: Main process exited, code=exited, status=143/n/a 2024-03-16T12:09:06.599 INFO:journalctl@ceph.mgr.y.smithi152.stdout:Mar 16 12:09:06 smithi152 systemd[1]: ceph-d1887f28-e389-11ee-95c9-87774f69a715@mgr.y.service: Failed with result 'exit-code'. 2024-03-16T12:09:06.599 INFO:journalctl@ceph.mgr.y.smithi152.stdout:Mar 16 12:09:06 smithi152 systemd[1]: Stopped Ceph mgr.y for d1887f28-e389-11ee-95c9-87774f69a715. 2024-03-16T12:09:06.599 INFO:journalctl@ceph.mgr.y.smithi152.stdout:Mar 16 12:09:06 smithi152 systemd[1]: ceph-d1887f28-e389-11ee-95c9-87774f69a715@mgr.y.service: Consumed 21.541s CPU time. 2024-03-16T12:09:06.611 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-16T12:09:06.612 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-03-16T12:09:06.612 INFO:tasks.cephadm.mgr.z:Stopping mgr.z... 2024-03-16T12:09:06.612 DEBUG:teuthology.orchestra.run.smithi154:> sudo systemctl stop ceph-d1887f28-e389-11ee-95c9-87774f69a715@mgr.z 2024-03-16T12:09:06.953 INFO:journalctl@ceph.mgr.z.smithi154.stdout:Mar 16 12:09:06 smithi154 systemd[1]: Stopping Ceph mgr.z for d1887f28-e389-11ee-95c9-87774f69a715... 2024-03-16T12:09:07.285 INFO:journalctl@ceph.mgr.z.smithi154.stdout:Mar 16 12:09:06 smithi154 podman[74753]: 2024-03-16 12:09:06.965125709 +0000 UTC m=+0.148330378 container died d145605c62c862ce9ca2f08b5e8acb8ddeb48d489ef448d22e9da39d63f5455a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-z, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-03-16T12:09:08.035 INFO:journalctl@ceph.mgr.z.smithi154.stdout:Mar 16 12:09:07 smithi154 podman[74753]: 2024-03-16 12:09:07.653516875 +0000 UTC m=+0.836721534 container cleanup d145605c62c862ce9ca2f08b5e8acb8ddeb48d489ef448d22e9da39d63f5455a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-z, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.schema-version=1.0) 2024-03-16T12:09:08.035 INFO:journalctl@ceph.mgr.z.smithi154.stdout:Mar 16 12:09:07 smithi154 bash[74753]: ceph-d1887f28-e389-11ee-95c9-87774f69a715-mgr-z 2024-03-16T12:09:08.334 DEBUG:teuthology.orchestra.run.smithi154:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d1887f28-e389-11ee-95c9-87774f69a715@mgr.z.service' 2024-03-16T12:09:08.371 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-16T12:09:08.371 INFO:tasks.cephadm.mgr.z:Stopped mgr.z 2024-03-16T12:09:08.372 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-03-16T12:09:08.372 DEBUG:teuthology.orchestra.run.smithi029:> sudo systemctl stop ceph-d1887f28-e389-11ee-95c9-87774f69a715@osd.0 2024-03-16T12:09:08.804 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 16 12:09:08 smithi029 systemd[1]: Stopping Ceph osd.0 for d1887f28-e389-11ee-95c9-87774f69a715... 2024-03-16T12:09:09.078 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 16 12:09:08 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-0[46776]: 2024-03-16T12:09:08.814+0000 7f49bf696700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-16T12:09:09.078 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 16 12:09:08 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-0[46776]: 2024-03-16T12:09:08.814+0000 7f49bf696700 -1 osd.0 119 *** Got signal Terminated *** 2024-03-16T12:09:09.078 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 16 12:09:08 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-0[46776]: 2024-03-16T12:09:08.814+0000 7f49bf696700 -1 osd.0 119 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-16T12:09:14.700 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 16 12:09:14 smithi029 podman[95050]: 2024-03-16 12:09:14.420280738 +0000 UTC m=+5.842211831 container died e866dfb0c19b11e872c21347d5a5e1503c8ada92ff31e2cd630a4b0ace06f0d3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-0, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-16T12:09:15.077 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 16 12:09:14 smithi029 podman[95050]: 2024-03-16 12:09:14.893918536 +0000 UTC m=+6.315849630 container cleanup e866dfb0c19b11e872c21347d5a5e1503c8ada92ff31e2cd630a4b0ace06f0d3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0) 2024-03-16T12:09:15.078 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 16 12:09:14 smithi029 bash[95050]: ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-0 2024-03-16T12:09:15.578 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 16 12:09:15 smithi029 podman[95070]: 2024-03-16 12:09:15.289999816 +0000 UTC m=+0.864289399 container remove e866dfb0c19b11e872c21347d5a5e1503c8ada92ff31e2cd630a4b0ace06f0d3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, io.buildah.version=1.33.5, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307) 2024-03-16T12:09:16.012 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 16 12:09:15 smithi029 podman[95162]: 2024-03-16 12:09:15.573222729 +0000 UTC m=+0.018037656 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-16T12:09:16.012 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 16 12:09:15 smithi029 podman[95162]: 2024-03-16 12:09:15.700322136 +0000 UTC m=+0.145137102 container create 3d072173ed3e7471f5dcdf46d6fcff429a6b9242003b101c2faabf8582385bf9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-0-deactivate, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-16T12:09:16.012 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 16 12:09:15 smithi029 podman[95162]: 2024-03-16 12:09:15.914376411 +0000 UTC m=+0.359191350 container init 3d072173ed3e7471f5dcdf46d6fcff429a6b9242003b101c2faabf8582385bf9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-0-deactivate, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-16T12:09:16.012 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 16 12:09:15 smithi029 podman[95162]: 2024-03-16 12:09:15.917423953 +0000 UTC m=+0.362238878 container start 3d072173ed3e7471f5dcdf46d6fcff429a6b9242003b101c2faabf8582385bf9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-0-deactivate, GIT_CLEAN=True, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-03-16T12:09:16.328 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 16 12:09:16 smithi029 podman[95162]: 2024-03-16 12:09:16.023518478 +0000 UTC m=+0.468333426 container attach 3d072173ed3e7471f5dcdf46d6fcff429a6b9242003b101c2faabf8582385bf9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-0-deactivate, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, ceph=True, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-03-16T12:09:16.328 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 16 12:09:16 smithi029 podman[95162]: 2024-03-16 12:09:16.208992312 +0000 UTC m=+0.653807245 container died 3d072173ed3e7471f5dcdf46d6fcff429a6b9242003b101c2faabf8582385bf9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-0-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-16T12:09:16.869 DEBUG:teuthology.orchestra.run.smithi029:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d1887f28-e389-11ee-95c9-87774f69a715@osd.0.service' 2024-03-16T12:09:16.917 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-16T12:09:16.917 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-03-16T12:09:16.918 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-03-16T12:09:16.918 DEBUG:teuthology.orchestra.run.smithi152:> sudo systemctl stop ceph-d1887f28-e389-11ee-95c9-87774f69a715@osd.1 2024-03-16T12:09:17.250 INFO:journalctl@ceph.osd.1.smithi152.stdout:Mar 16 12:09:16 smithi152 systemd[1]: Stopping Ceph osd.1 for d1887f28-e389-11ee-95c9-87774f69a715... 2024-03-16T12:09:17.617 INFO:journalctl@ceph.osd.1.smithi152.stdout:Mar 16 12:09:17 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-1[40893]: 2024-03-16T12:09:17.260+0000 7f4feac3a700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-16T12:09:17.617 INFO:journalctl@ceph.osd.1.smithi152.stdout:Mar 16 12:09:17 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-1[40893]: 2024-03-16T12:09:17.260+0000 7f4feac3a700 -1 osd.1 119 *** Got signal Terminated *** 2024-03-16T12:09:17.617 INFO:journalctl@ceph.osd.1.smithi152.stdout:Mar 16 12:09:17 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-1[40893]: 2024-03-16T12:09:17.260+0000 7f4feac3a700 -1 osd.1 119 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-16T12:09:22.909 INFO:journalctl@ceph.osd.1.smithi152.stdout:Mar 16 12:09:22 smithi152 podman[73921]: 2024-03-16 12:09:22.631720029 +0000 UTC m=+5.497926970 container died e2be6b67d15a0fd42b1ede684a6e583f97ff103c64381489861af9d232ec299e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-1, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.build-date=20240307) 2024-03-16T12:09:23.218 INFO:journalctl@ceph.osd.1.smithi152.stdout:Mar 16 12:09:23 smithi152 podman[73921]: 2024-03-16 12:09:23.011799842 +0000 UTC m=+5.878006783 container cleanup e2be6b67d15a0fd42b1ede684a6e583f97ff103c64381489861af9d232ec299e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-1, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-03-16T12:09:23.218 INFO:journalctl@ceph.osd.1.smithi152.stdout:Mar 16 12:09:23 smithi152 bash[73921]: ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-1 2024-03-16T12:09:23.609 INFO:journalctl@ceph.osd.1.smithi152.stdout:Mar 16 12:09:23 smithi152 podman[73937]: 2024-03-16 12:09:23.229834419 +0000 UTC m=+0.597504982 container remove e2be6b67d15a0fd42b1ede684a6e583f97ff103c64381489861af9d232ec299e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-1, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-03-16T12:09:23.865 INFO:journalctl@ceph.osd.1.smithi152.stdout:Mar 16 12:09:23 smithi152 podman[74027]: 2024-03-16 12:09:23.523809119 +0000 UTC m=+0.017644454 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-16T12:09:23.865 INFO:journalctl@ceph.osd.1.smithi152.stdout:Mar 16 12:09:23 smithi152 podman[74027]: 2024-03-16 12:09:23.631097764 +0000 UTC m=+0.124933091 container create ed4b36f73e54d44a34fb071b95502e257812f58e4f410aa08eb5abd72eff7fe1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-1-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-03-16T12:09:24.117 INFO:journalctl@ceph.osd.1.smithi152.stdout:Mar 16 12:09:23 smithi152 podman[74027]: 2024-03-16 12:09:23.876304075 +0000 UTC m=+0.370139433 container init ed4b36f73e54d44a34fb071b95502e257812f58e4f410aa08eb5abd72eff7fe1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-1-deactivate, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-16T12:09:24.117 INFO:journalctl@ceph.osd.1.smithi152.stdout:Mar 16 12:09:23 smithi152 podman[74027]: 2024-03-16 12:09:23.879955933 +0000 UTC m=+0.373791263 container start ed4b36f73e54d44a34fb071b95502e257812f58e4f410aa08eb5abd72eff7fe1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-1-deactivate, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-03-16T12:09:24.117 INFO:journalctl@ceph.osd.1.smithi152.stdout:Mar 16 12:09:23 smithi152 podman[74027]: 2024-03-16 12:09:23.975838964 +0000 UTC m=+0.469674343 container attach ed4b36f73e54d44a34fb071b95502e257812f58e4f410aa08eb5abd72eff7fe1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-1-deactivate, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-03-16T12:09:24.793 INFO:journalctl@ceph.osd.1.smithi152.stdout:Mar 16 12:09:24 smithi152 podman[74027]: 2024-03-16 12:09:24.45709303 +0000 UTC m=+0.950928377 container died ed4b36f73e54d44a34fb071b95502e257812f58e4f410aa08eb5abd72eff7fe1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-1-deactivate, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-03-16T12:09:25.151 DEBUG:teuthology.orchestra.run.smithi152:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d1887f28-e389-11ee-95c9-87774f69a715@osd.1.service' 2024-03-16T12:09:25.196 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-16T12:09:25.196 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-03-16T12:09:25.196 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-03-16T12:09:25.196 DEBUG:teuthology.orchestra.run.smithi154:> sudo systemctl stop ceph-d1887f28-e389-11ee-95c9-87774f69a715@osd.2 2024-03-16T12:09:25.513 INFO:journalctl@ceph.osd.2.smithi154.stdout:Mar 16 12:09:25 smithi154 systemd[1]: Stopping Ceph osd.2 for d1887f28-e389-11ee-95c9-87774f69a715... 2024-03-16T12:09:25.785 INFO:journalctl@ceph.osd.2.smithi154.stdout:Mar 16 12:09:25 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-2[41481]: 2024-03-16T12:09:25.524+0000 7fee69251700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-16T12:09:25.785 INFO:journalctl@ceph.osd.2.smithi154.stdout:Mar 16 12:09:25 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-2[41481]: 2024-03-16T12:09:25.524+0000 7fee69251700 -1 osd.2 119 *** Got signal Terminated *** 2024-03-16T12:09:25.785 INFO:journalctl@ceph.osd.2.smithi154.stdout:Mar 16 12:09:25 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-2[41481]: 2024-03-16T12:09:25.524+0000 7fee69251700 -1 osd.2 119 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-16T12:09:31.355 INFO:journalctl@ceph.osd.2.smithi154.stdout:Mar 16 12:09:31 smithi154 podman[74891]: 2024-03-16 12:09:31.066390958 +0000 UTC m=+5.661405961 container died cb30524b674583f74863743ab0b6cb8bc75cd9c220388b70fb698b5cc9776c64 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-03-16T12:09:31.706 INFO:journalctl@ceph.osd.2.smithi154.stdout:Mar 16 12:09:31 smithi154 podman[74891]: 2024-03-16 12:09:31.474824055 +0000 UTC m=+6.069839069 container cleanup cb30524b674583f74863743ab0b6cb8bc75cd9c220388b70fb698b5cc9776c64 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-2, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-16T12:09:31.706 INFO:journalctl@ceph.osd.2.smithi154.stdout:Mar 16 12:09:31 smithi154 bash[74891]: ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-2 2024-03-16T12:09:32.022 INFO:journalctl@ceph.osd.2.smithi154.stdout:Mar 16 12:09:31 smithi154 podman[74911]: 2024-03-16 12:09:31.717858141 +0000 UTC m=+0.649014726 container remove cb30524b674583f74863743ab0b6cb8bc75cd9c220388b70fb698b5cc9776c64 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-2, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-03-16T12:09:32.285 INFO:journalctl@ceph.osd.2.smithi154.stdout:Mar 16 12:09:32 smithi154 podman[75006]: 2024-03-16 12:09:31.93710773 +0000 UTC m=+0.017844196 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-16T12:09:32.285 INFO:journalctl@ceph.osd.2.smithi154.stdout:Mar 16 12:09:32 smithi154 podman[75006]: 2024-03-16 12:09:32.060738171 +0000 UTC m=+0.141474619 container create bd7369e21a9ee9677ef7cc7e67128b6bee334f63702bf3f0d5c90b8707fe7570 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-2-deactivate, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-03-16T12:09:32.624 INFO:journalctl@ceph.osd.2.smithi154.stdout:Mar 16 12:09:32 smithi154 podman[75006]: 2024-03-16 12:09:32.339683189 +0000 UTC m=+0.420419646 container init bd7369e21a9ee9677ef7cc7e67128b6bee334f63702bf3f0d5c90b8707fe7570 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-2-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True) 2024-03-16T12:09:32.624 INFO:journalctl@ceph.osd.2.smithi154.stdout:Mar 16 12:09:32 smithi154 podman[75006]: 2024-03-16 12:09:32.342827595 +0000 UTC m=+0.423564042 container start bd7369e21a9ee9677ef7cc7e67128b6bee334f63702bf3f0d5c90b8707fe7570 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-2-deactivate, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-03-16T12:09:32.624 INFO:journalctl@ceph.osd.2.smithi154.stdout:Mar 16 12:09:32 smithi154 podman[75006]: 2024-03-16 12:09:32.447318838 +0000 UTC m=+0.528055300 container attach bd7369e21a9ee9677ef7cc7e67128b6bee334f63702bf3f0d5c90b8707fe7570 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-2-deactivate, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_CLEAN=True) 2024-03-16T12:09:32.989 INFO:journalctl@ceph.osd.2.smithi154.stdout:Mar 16 12:09:32 smithi154 podman[75006]: 2024-03-16 12:09:32.635846339 +0000 UTC m=+0.716582794 container died bd7369e21a9ee9677ef7cc7e67128b6bee334f63702bf3f0d5c90b8707fe7570 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-2-deactivate, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS) 2024-03-16T12:09:33.314 DEBUG:teuthology.orchestra.run.smithi154:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d1887f28-e389-11ee-95c9-87774f69a715@osd.2.service' 2024-03-16T12:09:33.357 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-16T12:09:33.357 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-03-16T12:09:33.357 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-03-16T12:09:33.358 DEBUG:teuthology.orchestra.run.smithi029:> sudo systemctl stop ceph-d1887f28-e389-11ee-95c9-87774f69a715@osd.3 2024-03-16T12:09:33.675 INFO:journalctl@ceph.osd.3.smithi029.stdout:Mar 16 12:09:33 smithi029 systemd[1]: Stopping Ceph osd.3 for d1887f28-e389-11ee-95c9-87774f69a715... 2024-03-16T12:09:33.983 INFO:journalctl@ceph.osd.3.smithi029.stdout:Mar 16 12:09:33 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-3[53281]: 2024-03-16T12:09:33.685+0000 7ffa11a51700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-16T12:09:33.983 INFO:journalctl@ceph.osd.3.smithi029.stdout:Mar 16 12:09:33 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-3[53281]: 2024-03-16T12:09:33.685+0000 7ffa11a51700 -1 osd.3 119 *** Got signal Terminated *** 2024-03-16T12:09:33.984 INFO:journalctl@ceph.osd.3.smithi029.stdout:Mar 16 12:09:33 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-3[53281]: 2024-03-16T12:09:33.685+0000 7ffa11a51700 -1 osd.3 119 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-16T12:09:35.867 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:09:35 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:09:35.457+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:10.713221+0000 front 2024-03-16T12:09:10.713087+0000 (oldest deadline 2024-03-16T12:09:35.413059+0000) 2024-03-16T12:09:36.077 INFO:journalctl@ceph.osd.3.smithi029.stdout:Mar 16 12:09:35 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-3[53281]: 2024-03-16T12:09:35.740+0000 7ffa0a03d700 -1 osd.3 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:13.493330+0000 front 2024-03-16T12:09:13.493359+0000 (oldest deadline 2024-03-16T12:09:35.193441+0000) 2024-03-16T12:09:36.285 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:09:35 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:09:35.800+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:13.025675+0000 front 2024-03-16T12:09:13.025847+0000 (oldest deadline 2024-03-16T12:09:35.325827+0000) 2024-03-16T12:09:36.867 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:09:36 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:09:36.429+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:10.713221+0000 front 2024-03-16T12:09:10.713087+0000 (oldest deadline 2024-03-16T12:09:35.413059+0000) 2024-03-16T12:09:37.077 INFO:journalctl@ceph.osd.3.smithi029.stdout:Mar 16 12:09:36 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-3[53281]: 2024-03-16T12:09:36.791+0000 7ffa0a03d700 -1 osd.3 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:13.493330+0000 front 2024-03-16T12:09:13.493359+0000 (oldest deadline 2024-03-16T12:09:35.193441+0000) 2024-03-16T12:09:37.077 INFO:journalctl@ceph.osd.6.smithi029.stdout:Mar 16 12:09:36 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-6[60123]: 2024-03-16T12:09:36.928+0000 7fb55a3ef700 -1 osd.6 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:11.958752+0000 front 2024-03-16T12:09:11.958869+0000 (oldest deadline 2024-03-16T12:09:36.658956+0000) 2024-03-16T12:09:37.285 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:09:36 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:09:36.841+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:13.025675+0000 front 2024-03-16T12:09:13.025847+0000 (oldest deadline 2024-03-16T12:09:35.325827+0000) 2024-03-16T12:09:37.867 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:09:37 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:09:37.460+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:10.713221+0000 front 2024-03-16T12:09:10.713087+0000 (oldest deadline 2024-03-16T12:09:35.413059+0000) 2024-03-16T12:09:38.035 INFO:journalctl@ceph.osd.5.smithi154.stdout:Mar 16 12:09:37 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-5[48458]: 2024-03-16T12:09:37.759+0000 7fc656102700 -1 osd.5 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:10.999320+0000 front 2024-03-16T12:09:10.999175+0000 (oldest deadline 2024-03-16T12:09:36.899159+0000) 2024-03-16T12:09:38.035 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:09:37 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:09:37.852+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:13.025675+0000 front 2024-03-16T12:09:13.025847+0000 (oldest deadline 2024-03-16T12:09:35.325827+0000) 2024-03-16T12:09:38.077 INFO:journalctl@ceph.osd.3.smithi029.stdout:Mar 16 12:09:37 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-3[53281]: 2024-03-16T12:09:37.824+0000 7ffa0a03d700 -1 osd.3 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:13.493330+0000 front 2024-03-16T12:09:13.493359+0000 (oldest deadline 2024-03-16T12:09:35.193441+0000) 2024-03-16T12:09:38.078 INFO:journalctl@ceph.osd.6.smithi029.stdout:Mar 16 12:09:37 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-6[60123]: 2024-03-16T12:09:37.941+0000 7fb55a3ef700 -1 osd.6 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:11.958752+0000 front 2024-03-16T12:09:11.958869+0000 (oldest deadline 2024-03-16T12:09:36.658956+0000) 2024-03-16T12:09:38.577 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 16 12:09:38 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-9[67182]: 2024-03-16T12:09:38.174+0000 7fe1cee8b700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:12.704136+0000 front 2024-03-16T12:09:12.703904+0000 (oldest deadline 2024-03-16T12:09:37.404049+0000) 2024-03-16T12:09:38.867 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:09:38 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:09:38.464+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:10.713221+0000 front 2024-03-16T12:09:10.713087+0000 (oldest deadline 2024-03-16T12:09:35.413059+0000) 2024-03-16T12:09:39.035 INFO:journalctl@ceph.osd.5.smithi154.stdout:Mar 16 12:09:38 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-5[48458]: 2024-03-16T12:09:38.728+0000 7fc656102700 -1 osd.5 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:10.999320+0000 front 2024-03-16T12:09:10.999175+0000 (oldest deadline 2024-03-16T12:09:36.899159+0000) 2024-03-16T12:09:39.035 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:09:38 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:09:38.806+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:13.025675+0000 front 2024-03-16T12:09:13.025847+0000 (oldest deadline 2024-03-16T12:09:35.325827+0000) 2024-03-16T12:09:39.328 INFO:journalctl@ceph.osd.3.smithi029.stdout:Mar 16 12:09:39 smithi029 podman[95269]: 2024-03-16 12:09:39.087634084 +0000 UTC m=+5.525741222 container died ae3a6b97b39c3ab807682d86e8a8521e700562829725212d12046108ef2d9a91 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-3, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True) 2024-03-16T12:09:39.329 INFO:journalctl@ceph.osd.6.smithi029.stdout:Mar 16 12:09:38 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-6[60123]: 2024-03-16T12:09:38.985+0000 7fb55a3ef700 -1 osd.6 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:11.958752+0000 front 2024-03-16T12:09:11.958869+0000 (oldest deadline 2024-03-16T12:09:36.658956+0000) 2024-03-16T12:09:39.329 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 16 12:09:39 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-9[67182]: 2024-03-16T12:09:39.147+0000 7fe1cee8b700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:12.704136+0000 front 2024-03-16T12:09:12.703904+0000 (oldest deadline 2024-03-16T12:09:37.404049+0000) 2024-03-16T12:09:39.707 INFO:journalctl@ceph.osd.3.smithi029.stdout:Mar 16 12:09:39 smithi029 podman[95269]: 2024-03-16 12:09:39.499744999 +0000 UTC m=+5.937852140 container cleanup ae3a6b97b39c3ab807682d86e8a8521e700562829725212d12046108ef2d9a91 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-3, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-03-16T12:09:39.707 INFO:journalctl@ceph.osd.3.smithi029.stdout:Mar 16 12:09:39 smithi029 bash[95269]: ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-3 2024-03-16T12:09:39.867 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:09:39 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:09:39.499+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:10.713221+0000 front 2024-03-16T12:09:10.713087+0000 (oldest deadline 2024-03-16T12:09:35.413059+0000) 2024-03-16T12:09:40.014 INFO:journalctl@ceph.osd.3.smithi029.stdout:Mar 16 12:09:39 smithi029 podman[95298]: 2024-03-16 12:09:39.719008186 +0000 UTC m=+0.635079322 container remove ae3a6b97b39c3ab807682d86e8a8521e700562829725212d12046108ef2d9a91 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-3, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-03-16T12:09:40.014 INFO:journalctl@ceph.osd.6.smithi029.stdout:Mar 16 12:09:39 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-6[60123]: 2024-03-16T12:09:39.947+0000 7fb55a3ef700 -1 osd.6 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:11.958752+0000 front 2024-03-16T12:09:11.958869+0000 (oldest deadline 2024-03-16T12:09:36.658956+0000) 2024-03-16T12:09:40.036 INFO:journalctl@ceph.osd.5.smithi154.stdout:Mar 16 12:09:39 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-5[48458]: 2024-03-16T12:09:39.711+0000 7fc656102700 -1 osd.5 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:10.999320+0000 front 2024-03-16T12:09:10.999175+0000 (oldest deadline 2024-03-16T12:09:36.899159+0000) 2024-03-16T12:09:40.037 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:09:39 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:09:39.784+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:13.025675+0000 front 2024-03-16T12:09:13.025847+0000 (oldest deadline 2024-03-16T12:09:35.325827+0000) 2024-03-16T12:09:40.298 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 16 12:09:40 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-9[67182]: 2024-03-16T12:09:40.129+0000 7fe1cee8b700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:12.704136+0000 front 2024-03-16T12:09:12.703904+0000 (oldest deadline 2024-03-16T12:09:37.404049+0000) 2024-03-16T12:09:40.298 INFO:journalctl@ceph.osd.3.smithi029.stdout:Mar 16 12:09:40 smithi029 podman[95386]: 2024-03-16 12:09:39.929964482 +0000 UTC m=+0.018659678 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-16T12:09:40.299 INFO:journalctl@ceph.osd.3.smithi029.stdout:Mar 16 12:09:40 smithi029 podman[95386]: 2024-03-16 12:09:40.055771659 +0000 UTC m=+0.144466854 container create 692a139780dae36452cfccc07f3dc60515d95f5ca208cdaaf61f1cf389645f1d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-3-deactivate, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-16T12:09:40.577 INFO:journalctl@ceph.osd.3.smithi029.stdout:Mar 16 12:09:40 smithi029 podman[95386]: 2024-03-16 12:09:40.310096492 +0000 UTC m=+0.398791689 container init 692a139780dae36452cfccc07f3dc60515d95f5ca208cdaaf61f1cf389645f1d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-3-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-03-16T12:09:40.578 INFO:journalctl@ceph.osd.3.smithi029.stdout:Mar 16 12:09:40 smithi029 podman[95386]: 2024-03-16 12:09:40.313523637 +0000 UTC m=+0.402218836 container start 692a139780dae36452cfccc07f3dc60515d95f5ca208cdaaf61f1cf389645f1d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-3-deactivate, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-16T12:09:40.578 INFO:journalctl@ceph.osd.3.smithi029.stdout:Mar 16 12:09:40 smithi029 podman[95386]: 2024-03-16 12:09:40.427471085 +0000 UTC m=+0.516166296 container attach 692a139780dae36452cfccc07f3dc60515d95f5ca208cdaaf61f1cf389645f1d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-3-deactivate, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307) 2024-03-16T12:09:40.867 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:09:40 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:09:40.490+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:10.713221+0000 front 2024-03-16T12:09:10.713087+0000 (oldest deadline 2024-03-16T12:09:35.413059+0000) 2024-03-16T12:09:40.942 INFO:journalctl@ceph.osd.3.smithi029.stdout:Mar 16 12:09:40 smithi029 podman[95386]: 2024-03-16 12:09:40.6080105 +0000 UTC m=+0.696705698 container died 692a139780dae36452cfccc07f3dc60515d95f5ca208cdaaf61f1cf389645f1d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-3-deactivate, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-03-16T12:09:41.035 INFO:journalctl@ceph.osd.5.smithi154.stdout:Mar 16 12:09:40 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-5[48458]: 2024-03-16T12:09:40.692+0000 7fc656102700 -1 osd.5 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:10.999320+0000 front 2024-03-16T12:09:10.999175+0000 (oldest deadline 2024-03-16T12:09:36.899159+0000) 2024-03-16T12:09:41.035 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:09:40 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:09:40.825+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:13.025675+0000 front 2024-03-16T12:09:13.025847+0000 (oldest deadline 2024-03-16T12:09:35.325827+0000) 2024-03-16T12:09:41.258 INFO:journalctl@ceph.osd.6.smithi029.stdout:Mar 16 12:09:40 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-6[60123]: 2024-03-16T12:09:40.953+0000 7fb55a3ef700 -1 osd.6 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:11.958752+0000 front 2024-03-16T12:09:11.958869+0000 (oldest deadline 2024-03-16T12:09:36.658956+0000) 2024-03-16T12:09:41.259 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 16 12:09:41 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-9[67182]: 2024-03-16T12:09:41.141+0000 7fe1cee8b700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:12.704136+0000 front 2024-03-16T12:09:12.703904+0000 (oldest deadline 2024-03-16T12:09:37.404049+0000) 2024-03-16T12:09:41.278 DEBUG:teuthology.orchestra.run.smithi029:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d1887f28-e389-11ee-95c9-87774f69a715@osd.3.service' 2024-03-16T12:09:41.324 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-16T12:09:41.325 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-03-16T12:09:41.325 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-03-16T12:09:41.325 DEBUG:teuthology.orchestra.run.smithi152:> sudo systemctl stop ceph-d1887f28-e389-11ee-95c9-87774f69a715@osd.4 2024-03-16T12:09:41.617 INFO:journalctl@ceph.osd.4.smithi152.stdout:Mar 16 12:09:41 smithi152 systemd[1]: Stopping Ceph osd.4 for d1887f28-e389-11ee-95c9-87774f69a715... 2024-03-16T12:09:41.618 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:09:41 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:09:41.462+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:10.713221+0000 front 2024-03-16T12:09:10.713087+0000 (oldest deadline 2024-03-16T12:09:35.413059+0000) 2024-03-16T12:09:42.034 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:09:41 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:09:41.819+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:13.025675+0000 front 2024-03-16T12:09:13.025847+0000 (oldest deadline 2024-03-16T12:09:35.325827+0000) 2024-03-16T12:09:42.035 INFO:journalctl@ceph.osd.5.smithi154.stdout:Mar 16 12:09:41 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-5[48458]: 2024-03-16T12:09:41.702+0000 7fc656102700 -1 osd.5 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:10.999320+0000 front 2024-03-16T12:09:10.999175+0000 (oldest deadline 2024-03-16T12:09:36.899159+0000) 2024-03-16T12:09:42.117 INFO:journalctl@ceph.osd.4.smithi152.stdout:Mar 16 12:09:41 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-4[47757]: 2024-03-16T12:09:41.648+0000 7fd439bb6700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-16T12:09:42.117 INFO:journalctl@ceph.osd.4.smithi152.stdout:Mar 16 12:09:41 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-4[47757]: 2024-03-16T12:09:41.648+0000 7fd439bb6700 -1 osd.4 119 *** Got signal Terminated *** 2024-03-16T12:09:42.117 INFO:journalctl@ceph.osd.4.smithi152.stdout:Mar 16 12:09:41 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-4[47757]: 2024-03-16T12:09:41.648+0000 7fd439bb6700 -1 osd.4 119 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-16T12:09:42.327 INFO:journalctl@ceph.osd.6.smithi029.stdout:Mar 16 12:09:41 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-6[60123]: 2024-03-16T12:09:41.947+0000 7fb55a3ef700 -1 osd.6 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:11.958752+0000 front 2024-03-16T12:09:11.958869+0000 (oldest deadline 2024-03-16T12:09:36.658956+0000) 2024-03-16T12:09:42.327 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 16 12:09:42 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-9[67182]: 2024-03-16T12:09:42.127+0000 7fe1cee8b700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:12.704136+0000 front 2024-03-16T12:09:12.703904+0000 (oldest deadline 2024-03-16T12:09:37.404049+0000) 2024-03-16T12:09:42.867 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:09:42 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:09:42.509+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:10.713221+0000 front 2024-03-16T12:09:10.713087+0000 (oldest deadline 2024-03-16T12:09:35.413059+0000) 2024-03-16T12:09:43.034 INFO:journalctl@ceph.osd.5.smithi154.stdout:Mar 16 12:09:42 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-5[48458]: 2024-03-16T12:09:42.690+0000 7fc656102700 -1 osd.5 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:10.999320+0000 front 2024-03-16T12:09:10.999175+0000 (oldest deadline 2024-03-16T12:09:36.899159+0000) 2024-03-16T12:09:43.035 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:09:42 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:09:42.831+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:13.025675+0000 front 2024-03-16T12:09:13.025847+0000 (oldest deadline 2024-03-16T12:09:35.325827+0000) 2024-03-16T12:09:43.327 INFO:journalctl@ceph.osd.6.smithi029.stdout:Mar 16 12:09:42 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-6[60123]: 2024-03-16T12:09:42.931+0000 7fb55a3ef700 -1 osd.6 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:11.958752+0000 front 2024-03-16T12:09:11.958869+0000 (oldest deadline 2024-03-16T12:09:36.658956+0000) 2024-03-16T12:09:43.328 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 16 12:09:43 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-9[67182]: 2024-03-16T12:09:43.112+0000 7fe1cee8b700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:12.704136+0000 front 2024-03-16T12:09:12.703904+0000 (oldest deadline 2024-03-16T12:09:37.404049+0000) 2024-03-16T12:09:43.328 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 16 12:09:43 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-9[67182]: 2024-03-16T12:09:43.112+0000 7fe1cee8b700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:17.404712+0000 front 2024-03-16T12:09:17.404800+0000 (oldest deadline 2024-03-16T12:09:42.704627+0000) 2024-03-16T12:09:43.617 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:09:43 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:09:43.475+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:10.713221+0000 front 2024-03-16T12:09:10.713087+0000 (oldest deadline 2024-03-16T12:09:35.413059+0000) 2024-03-16T12:09:43.617 INFO:journalctl@ceph.osd.7.smithi152.stdout:Mar 16 12:09:43 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-7[54789]: 2024-03-16T12:09:43.259+0000 7f6c60c47700 -1 osd.7 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:17.304563+0000 front 2024-03-16T12:09:17.304652+0000 (oldest deadline 2024-03-16T12:09:43.204754+0000) 2024-03-16T12:09:43.995 INFO:journalctl@ceph.osd.5.smithi154.stdout:Mar 16 12:09:43 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-5[48458]: 2024-03-16T12:09:43.738+0000 7fc656102700 -1 osd.5 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:10.999320+0000 front 2024-03-16T12:09:10.999175+0000 (oldest deadline 2024-03-16T12:09:36.899159+0000) 2024-03-16T12:09:43.996 INFO:journalctl@ceph.osd.5.smithi154.stdout:Mar 16 12:09:43 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-5[48458]: 2024-03-16T12:09:43.738+0000 7fc656102700 -1 osd.5 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:16.899742+0000 front 2024-03-16T12:09:16.899720+0000 (oldest deadline 2024-03-16T12:09:42.799723+0000) 2024-03-16T12:09:43.996 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:09:43 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:09:43.869+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:13.025675+0000 front 2024-03-16T12:09:13.025847+0000 (oldest deadline 2024-03-16T12:09:35.325827+0000) 2024-03-16T12:09:44.327 INFO:journalctl@ceph.osd.6.smithi029.stdout:Mar 16 12:09:43 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-6[60123]: 2024-03-16T12:09:43.952+0000 7fb55a3ef700 -1 osd.6 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:11.958752+0000 front 2024-03-16T12:09:11.958869+0000 (oldest deadline 2024-03-16T12:09:36.658956+0000) 2024-03-16T12:09:44.327 INFO:journalctl@ceph.osd.6.smithi029.stdout:Mar 16 12:09:43 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-6[60123]: 2024-03-16T12:09:43.952+0000 7fb55a3ef700 -1 osd.6 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:17.160443+0000 front 2024-03-16T12:09:17.160625+0000 (oldest deadline 2024-03-16T12:09:43.060171+0000) 2024-03-16T12:09:44.328 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 16 12:09:44 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-9[67182]: 2024-03-16T12:09:44.140+0000 7fe1cee8b700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:12.704136+0000 front 2024-03-16T12:09:12.703904+0000 (oldest deadline 2024-03-16T12:09:37.404049+0000) 2024-03-16T12:09:44.328 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 16 12:09:44 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-9[67182]: 2024-03-16T12:09:44.140+0000 7fe1cee8b700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:17.404712+0000 front 2024-03-16T12:09:17.404800+0000 (oldest deadline 2024-03-16T12:09:42.704627+0000) 2024-03-16T12:09:44.493 INFO:journalctl@ceph.osd.4.smithi152.stdout:Mar 16 12:09:44 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-4[47757]: 2024-03-16T12:09:44.366+0000 7fd4361aa700 -1 osd.4 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:20.606100+0000 front 2024-03-16T12:09:20.606262+0000 (oldest deadline 2024-03-16T12:09:43.506225+0000) 2024-03-16T12:09:44.493 INFO:journalctl@ceph.osd.7.smithi152.stdout:Mar 16 12:09:44 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-7[54789]: 2024-03-16T12:09:44.219+0000 7f6c60c47700 -1 osd.7 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:17.304563+0000 front 2024-03-16T12:09:17.304652+0000 (oldest deadline 2024-03-16T12:09:43.204754+0000) 2024-03-16T12:09:44.866 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:09:44 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:09:44.504+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:10.713221+0000 front 2024-03-16T12:09:10.713087+0000 (oldest deadline 2024-03-16T12:09:35.413059+0000) 2024-03-16T12:09:44.888 INFO:journalctl@ceph.osd.5.smithi154.stdout:Mar 16 12:09:44 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-5[48458]: 2024-03-16T12:09:44.774+0000 7fc656102700 -1 osd.5 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:10.999320+0000 front 2024-03-16T12:09:10.999175+0000 (oldest deadline 2024-03-16T12:09:36.899159+0000) 2024-03-16T12:09:44.889 INFO:journalctl@ceph.osd.5.smithi154.stdout:Mar 16 12:09:44 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-5[48458]: 2024-03-16T12:09:44.774+0000 7fc656102700 -1 osd.5 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:16.899742+0000 front 2024-03-16T12:09:16.899720+0000 (oldest deadline 2024-03-16T12:09:42.799723+0000) 2024-03-16T12:09:45.232 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:09:44 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:09:44.900+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:13.025675+0000 front 2024-03-16T12:09:13.025847+0000 (oldest deadline 2024-03-16T12:09:35.325827+0000) 2024-03-16T12:09:45.327 INFO:journalctl@ceph.osd.6.smithi029.stdout:Mar 16 12:09:44 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-6[60123]: 2024-03-16T12:09:44.954+0000 7fb55a3ef700 -1 osd.6 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:11.958752+0000 front 2024-03-16T12:09:11.958869+0000 (oldest deadline 2024-03-16T12:09:36.658956+0000) 2024-03-16T12:09:45.327 INFO:journalctl@ceph.osd.6.smithi029.stdout:Mar 16 12:09:44 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-6[60123]: 2024-03-16T12:09:44.954+0000 7fb55a3ef700 -1 osd.6 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:17.160443+0000 front 2024-03-16T12:09:17.160625+0000 (oldest deadline 2024-03-16T12:09:43.060171+0000) 2024-03-16T12:09:45.328 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 16 12:09:45 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-9[67182]: 2024-03-16T12:09:45.117+0000 7fe1cee8b700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:12.704136+0000 front 2024-03-16T12:09:12.703904+0000 (oldest deadline 2024-03-16T12:09:37.404049+0000) 2024-03-16T12:09:45.328 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 16 12:09:45 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-9[67182]: 2024-03-16T12:09:45.117+0000 7fe1cee8b700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:17.404712+0000 front 2024-03-16T12:09:17.404800+0000 (oldest deadline 2024-03-16T12:09:42.704627+0000) 2024-03-16T12:09:45.506 INFO:journalctl@ceph.osd.7.smithi152.stdout:Mar 16 12:09:45 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-7[54789]: 2024-03-16T12:09:45.240+0000 7f6c60c47700 -1 osd.7 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:17.304563+0000 front 2024-03-16T12:09:17.304652+0000 (oldest deadline 2024-03-16T12:09:43.204754+0000) 2024-03-16T12:09:45.507 INFO:journalctl@ceph.osd.4.smithi152.stdout:Mar 16 12:09:45 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-4[47757]: 2024-03-16T12:09:45.414+0000 7fd4361aa700 -1 osd.4 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:20.606100+0000 front 2024-03-16T12:09:20.606262+0000 (oldest deadline 2024-03-16T12:09:43.506225+0000) 2024-03-16T12:09:45.534 INFO:journalctl@ceph.osd.8.smithi154.stdout:Mar 16 12:09:45 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-8[55054]: 2024-03-16T12:09:45.244+0000 7f3f065c0700 -1 osd.8 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:19.263068+0000 front 2024-03-16T12:09:19.263120+0000 (oldest deadline 2024-03-16T12:09:44.563033+0000) 2024-03-16T12:09:45.767 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:09:45 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:09:45.538+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:10.713221+0000 front 2024-03-16T12:09:10.713087+0000 (oldest deadline 2024-03-16T12:09:35.413059+0000) 2024-03-16T12:09:45.767 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:09:45 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:09:45.538+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:20.614694+0000 front 2024-03-16T12:09:20.614671+0000 (oldest deadline 2024-03-16T12:09:45.215535+0000) 2024-03-16T12:09:46.034 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:09:45 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:09:45.900+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:13.025675+0000 front 2024-03-16T12:09:13.025847+0000 (oldest deadline 2024-03-16T12:09:35.325827+0000) 2024-03-16T12:09:46.035 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:09:45 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:09:45.900+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:19.927745+0000 front 2024-03-16T12:09:19.927761+0000 (oldest deadline 2024-03-16T12:09:45.827828+0000) 2024-03-16T12:09:46.035 INFO:journalctl@ceph.osd.5.smithi154.stdout:Mar 16 12:09:45 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-5[48458]: 2024-03-16T12:09:45.750+0000 7fc656102700 -1 osd.5 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:10.999320+0000 front 2024-03-16T12:09:10.999175+0000 (oldest deadline 2024-03-16T12:09:36.899159+0000) 2024-03-16T12:09:46.035 INFO:journalctl@ceph.osd.5.smithi154.stdout:Mar 16 12:09:45 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-5[48458]: 2024-03-16T12:09:45.750+0000 7fc656102700 -1 osd.5 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:16.899742+0000 front 2024-03-16T12:09:16.899720+0000 (oldest deadline 2024-03-16T12:09:42.799723+0000) 2024-03-16T12:09:46.327 INFO:journalctl@ceph.osd.6.smithi029.stdout:Mar 16 12:09:45 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-6[60123]: 2024-03-16T12:09:45.993+0000 7fb55a3ef700 -1 osd.6 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:11.958752+0000 front 2024-03-16T12:09:11.958869+0000 (oldest deadline 2024-03-16T12:09:36.658956+0000) 2024-03-16T12:09:46.327 INFO:journalctl@ceph.osd.6.smithi029.stdout:Mar 16 12:09:45 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-6[60123]: 2024-03-16T12:09:45.993+0000 7fb55a3ef700 -1 osd.6 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:17.160443+0000 front 2024-03-16T12:09:17.160625+0000 (oldest deadline 2024-03-16T12:09:43.060171+0000) 2024-03-16T12:09:46.328 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 16 12:09:46 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-9[67182]: 2024-03-16T12:09:46.142+0000 7fe1cee8b700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:12.704136+0000 front 2024-03-16T12:09:12.703904+0000 (oldest deadline 2024-03-16T12:09:37.404049+0000) 2024-03-16T12:09:46.328 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 16 12:09:46 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-9[67182]: 2024-03-16T12:09:46.142+0000 7fe1cee8b700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:17.404712+0000 front 2024-03-16T12:09:17.404800+0000 (oldest deadline 2024-03-16T12:09:42.704627+0000) 2024-03-16T12:09:46.443 INFO:journalctl@ceph.osd.7.smithi152.stdout:Mar 16 12:09:46 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-7[54789]: 2024-03-16T12:09:46.194+0000 7f6c60c47700 -1 osd.7 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:17.304563+0000 front 2024-03-16T12:09:17.304652+0000 (oldest deadline 2024-03-16T12:09:43.204754+0000) 2024-03-16T12:09:46.534 INFO:journalctl@ceph.osd.8.smithi154.stdout:Mar 16 12:09:46 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-8[55054]: 2024-03-16T12:09:46.277+0000 7f3f065c0700 -1 osd.8 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:19.263068+0000 front 2024-03-16T12:09:19.263120+0000 (oldest deadline 2024-03-16T12:09:44.563033+0000) 2024-03-16T12:09:46.701 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:09:46 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:09:46.578+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:10.713221+0000 front 2024-03-16T12:09:10.713087+0000 (oldest deadline 2024-03-16T12:09:35.413059+0000) 2024-03-16T12:09:46.701 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:09:46 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:09:46.578+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:20.614694+0000 front 2024-03-16T12:09:20.614671+0000 (oldest deadline 2024-03-16T12:09:45.215535+0000) 2024-03-16T12:09:46.701 INFO:journalctl@ceph.osd.4.smithi152.stdout:Mar 16 12:09:46 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-4[47757]: 2024-03-16T12:09:46.455+0000 7fd4361aa700 -1 osd.4 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:20.606100+0000 front 2024-03-16T12:09:20.606262+0000 (oldest deadline 2024-03-16T12:09:43.506225+0000) 2024-03-16T12:09:47.034 INFO:journalctl@ceph.osd.5.smithi154.stdout:Mar 16 12:09:46 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-5[48458]: 2024-03-16T12:09:46.795+0000 7fc656102700 -1 osd.5 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:10.999320+0000 front 2024-03-16T12:09:10.999175+0000 (oldest deadline 2024-03-16T12:09:36.899159+0000) 2024-03-16T12:09:47.035 INFO:journalctl@ceph.osd.5.smithi154.stdout:Mar 16 12:09:46 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-5[48458]: 2024-03-16T12:09:46.795+0000 7fc656102700 -1 osd.5 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:16.899742+0000 front 2024-03-16T12:09:16.899720+0000 (oldest deadline 2024-03-16T12:09:42.799723+0000) 2024-03-16T12:09:47.035 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:09:46 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:09:46.875+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:13.025675+0000 front 2024-03-16T12:09:13.025847+0000 (oldest deadline 2024-03-16T12:09:35.325827+0000) 2024-03-16T12:09:47.035 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:09:46 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:09:46.875+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:19.927745+0000 front 2024-03-16T12:09:19.927761+0000 (oldest deadline 2024-03-16T12:09:45.827828+0000) 2024-03-16T12:09:47.327 INFO:journalctl@ceph.osd.6.smithi029.stdout:Mar 16 12:09:47 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-6[60123]: 2024-03-16T12:09:47.021+0000 7fb55a3ef700 -1 osd.6 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:11.958752+0000 front 2024-03-16T12:09:11.958869+0000 (oldest deadline 2024-03-16T12:09:36.658956+0000) 2024-03-16T12:09:47.327 INFO:journalctl@ceph.osd.6.smithi029.stdout:Mar 16 12:09:47 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-6[60123]: 2024-03-16T12:09:47.021+0000 7fb55a3ef700 -1 osd.6 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:17.160443+0000 front 2024-03-16T12:09:17.160625+0000 (oldest deadline 2024-03-16T12:09:43.060171+0000) 2024-03-16T12:09:47.328 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 16 12:09:47 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-9[67182]: 2024-03-16T12:09:47.166+0000 7fe1cee8b700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:12.704136+0000 front 2024-03-16T12:09:12.703904+0000 (oldest deadline 2024-03-16T12:09:37.404049+0000) 2024-03-16T12:09:47.328 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 16 12:09:47 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-9[67182]: 2024-03-16T12:09:47.166+0000 7fe1cee8b700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:17.404712+0000 front 2024-03-16T12:09:17.404800+0000 (oldest deadline 2024-03-16T12:09:42.704627+0000) 2024-03-16T12:09:47.510 INFO:journalctl@ceph.osd.7.smithi152.stdout:Mar 16 12:09:47 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-7[54789]: 2024-03-16T12:09:47.195+0000 7f6c60c47700 -1 osd.7 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:17.304563+0000 front 2024-03-16T12:09:17.304652+0000 (oldest deadline 2024-03-16T12:09:43.204754+0000) 2024-03-16T12:09:47.511 INFO:journalctl@ceph.osd.4.smithi152.stdout:Mar 16 12:09:47 smithi152 podman[74131]: 2024-03-16 12:09:47.219645121 +0000 UTC m=+5.690851816 container died 78822bb59b2b1f1d6642bccb23bcc6c3f523752204959dbb7e40ba2077313d7c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-4, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, ceph=True, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-16T12:09:47.534 INFO:journalctl@ceph.osd.8.smithi154.stdout:Mar 16 12:09:47 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-8[55054]: 2024-03-16T12:09:47.281+0000 7f3f065c0700 -1 osd.8 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:19.263068+0000 front 2024-03-16T12:09:19.263120+0000 (oldest deadline 2024-03-16T12:09:44.563033+0000) 2024-03-16T12:09:47.866 INFO:journalctl@ceph.osd.4.smithi152.stdout:Mar 16 12:09:47 smithi152 podman[74131]: 2024-03-16 12:09:47.616561255 +0000 UTC m=+6.087767931 container cleanup 78822bb59b2b1f1d6642bccb23bcc6c3f523752204959dbb7e40ba2077313d7c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-4, org.label-schema.build-date=20240307, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, ceph=True, org.label-schema.license=GPLv2) 2024-03-16T12:09:47.867 INFO:journalctl@ceph.osd.4.smithi152.stdout:Mar 16 12:09:47 smithi152 bash[74131]: ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-4 2024-03-16T12:09:47.867 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:09:47 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:09:47.628+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:10.713221+0000 front 2024-03-16T12:09:10.713087+0000 (oldest deadline 2024-03-16T12:09:35.413059+0000) 2024-03-16T12:09:47.867 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:09:47 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:09:47.628+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:20.614694+0000 front 2024-03-16T12:09:20.614671+0000 (oldest deadline 2024-03-16T12:09:45.215535+0000) 2024-03-16T12:09:48.035 INFO:journalctl@ceph.osd.5.smithi154.stdout:Mar 16 12:09:47 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-5[48458]: 2024-03-16T12:09:47.770+0000 7fc656102700 -1 osd.5 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:10.999320+0000 front 2024-03-16T12:09:10.999175+0000 (oldest deadline 2024-03-16T12:09:36.899159+0000) 2024-03-16T12:09:48.035 INFO:journalctl@ceph.osd.5.smithi154.stdout:Mar 16 12:09:47 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-5[48458]: 2024-03-16T12:09:47.770+0000 7fc656102700 -1 osd.5 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:16.899742+0000 front 2024-03-16T12:09:16.899720+0000 (oldest deadline 2024-03-16T12:09:42.799723+0000) 2024-03-16T12:09:48.035 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:09:47 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:09:47.885+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:13.025675+0000 front 2024-03-16T12:09:13.025847+0000 (oldest deadline 2024-03-16T12:09:35.325827+0000) 2024-03-16T12:09:48.035 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:09:47 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:09:47.885+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:19.927745+0000 front 2024-03-16T12:09:19.927761+0000 (oldest deadline 2024-03-16T12:09:45.827828+0000) 2024-03-16T12:09:48.203 INFO:journalctl@ceph.osd.4.smithi152.stdout:Mar 16 12:09:47 smithi152 podman[74161]: 2024-03-16 12:09:47.884813045 +0000 UTC m=+0.666952589 container remove 78822bb59b2b1f1d6642bccb23bcc6c3f523752204959dbb7e40ba2077313d7c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-4, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=) 2024-03-16T12:09:48.327 INFO:journalctl@ceph.osd.6.smithi029.stdout:Mar 16 12:09:48 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-6[60123]: 2024-03-16T12:09:48.015+0000 7fb55a3ef700 -1 osd.6 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:11.958752+0000 front 2024-03-16T12:09:11.958869+0000 (oldest deadline 2024-03-16T12:09:36.658956+0000) 2024-03-16T12:09:48.327 INFO:journalctl@ceph.osd.6.smithi029.stdout:Mar 16 12:09:48 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-6[60123]: 2024-03-16T12:09:48.015+0000 7fb55a3ef700 -1 osd.6 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:17.160443+0000 front 2024-03-16T12:09:17.160625+0000 (oldest deadline 2024-03-16T12:09:43.060171+0000) 2024-03-16T12:09:48.328 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 16 12:09:48 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-9[67182]: 2024-03-16T12:09:48.170+0000 7fe1cee8b700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:12.704136+0000 front 2024-03-16T12:09:12.703904+0000 (oldest deadline 2024-03-16T12:09:37.404049+0000) 2024-03-16T12:09:48.328 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 16 12:09:48 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-9[67182]: 2024-03-16T12:09:48.170+0000 7fe1cee8b700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:17.404712+0000 front 2024-03-16T12:09:17.404800+0000 (oldest deadline 2024-03-16T12:09:42.704627+0000) 2024-03-16T12:09:48.493 INFO:journalctl@ceph.osd.7.smithi152.stdout:Mar 16 12:09:48 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-7[54789]: 2024-03-16T12:09:48.238+0000 7f6c60c47700 -1 osd.7 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:17.304563+0000 front 2024-03-16T12:09:17.304652+0000 (oldest deadline 2024-03-16T12:09:43.204754+0000) 2024-03-16T12:09:48.494 INFO:journalctl@ceph.osd.4.smithi152.stdout:Mar 16 12:09:48 smithi152 podman[74255]: 2024-03-16 12:09:48.119067085 +0000 UTC m=+0.017810389 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-16T12:09:48.494 INFO:journalctl@ceph.osd.4.smithi152.stdout:Mar 16 12:09:48 smithi152 podman[74255]: 2024-03-16 12:09:48.248825884 +0000 UTC m=+0.147569167 container create ca9fc0f6baa2a63830394a1fd2899fdd88bf12f2d7d28d3e889542d9569041b9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-4-deactivate, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, CEPH_POINT_RELEASE=) 2024-03-16T12:09:48.494 INFO:journalctl@ceph.osd.4.smithi152.stdout:Mar 16 12:09:48 smithi152 podman[74255]: 2024-03-16 12:09:48.505603583 +0000 UTC m=+0.404346875 container init ca9fc0f6baa2a63830394a1fd2899fdd88bf12f2d7d28d3e889542d9569041b9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-4-deactivate, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-03-16T12:09:48.784 INFO:journalctl@ceph.osd.8.smithi154.stdout:Mar 16 12:09:48 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-8[55054]: 2024-03-16T12:09:48.325+0000 7f3f065c0700 -1 osd.8 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:19.263068+0000 front 2024-03-16T12:09:19.263120+0000 (oldest deadline 2024-03-16T12:09:44.563033+0000) 2024-03-16T12:09:48.786 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:09:48 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:09:48.595+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:10.713221+0000 front 2024-03-16T12:09:10.713087+0000 (oldest deadline 2024-03-16T12:09:35.413059+0000) 2024-03-16T12:09:48.786 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:09:48 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:09:48.595+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:20.614694+0000 front 2024-03-16T12:09:20.614671+0000 (oldest deadline 2024-03-16T12:09:45.215535+0000) 2024-03-16T12:09:48.787 INFO:journalctl@ceph.osd.4.smithi152.stdout:Mar 16 12:09:48 smithi152 podman[74255]: 2024-03-16 12:09:48.509115413 +0000 UTC m=+0.407858695 container start ca9fc0f6baa2a63830394a1fd2899fdd88bf12f2d7d28d3e889542d9569041b9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-4-deactivate, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-03-16T12:09:48.787 INFO:journalctl@ceph.osd.4.smithi152.stdout:Mar 16 12:09:48 smithi152 podman[74255]: 2024-03-16 12:09:48.597098327 +0000 UTC m=+0.495841625 container attach ca9fc0f6baa2a63830394a1fd2899fdd88bf12f2d7d28d3e889542d9569041b9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-4-deactivate, org.label-schema.build-date=20240307, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.license=GPLv2) 2024-03-16T12:09:49.116 INFO:journalctl@ceph.osd.4.smithi152.stdout:Mar 16 12:09:48 smithi152 podman[74255]: 2024-03-16 12:09:48.799080561 +0000 UTC m=+0.697823841 container died ca9fc0f6baa2a63830394a1fd2899fdd88bf12f2d7d28d3e889542d9569041b9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-4-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240307, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b) 2024-03-16T12:09:49.274 INFO:journalctl@ceph.osd.5.smithi154.stdout:Mar 16 12:09:48 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-5[48458]: 2024-03-16T12:09:48.812+0000 7fc656102700 -1 osd.5 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:10.999320+0000 front 2024-03-16T12:09:10.999175+0000 (oldest deadline 2024-03-16T12:09:36.899159+0000) 2024-03-16T12:09:49.274 INFO:journalctl@ceph.osd.5.smithi154.stdout:Mar 16 12:09:48 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-5[48458]: 2024-03-16T12:09:48.812+0000 7fc656102700 -1 osd.5 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:16.899742+0000 front 2024-03-16T12:09:16.899720+0000 (oldest deadline 2024-03-16T12:09:42.799723+0000) 2024-03-16T12:09:49.275 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:09:48 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:09:48.889+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:13.025675+0000 front 2024-03-16T12:09:13.025847+0000 (oldest deadline 2024-03-16T12:09:35.325827+0000) 2024-03-16T12:09:49.275 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:09:48 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:09:48.889+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:19.927745+0000 front 2024-03-16T12:09:19.927761+0000 (oldest deadline 2024-03-16T12:09:45.827828+0000) 2024-03-16T12:09:49.327 INFO:journalctl@ceph.osd.6.smithi029.stdout:Mar 16 12:09:48 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-6[60123]: 2024-03-16T12:09:48.979+0000 7fb55a3ef700 -1 osd.6 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:11.958752+0000 front 2024-03-16T12:09:11.958869+0000 (oldest deadline 2024-03-16T12:09:36.658956+0000) 2024-03-16T12:09:49.327 INFO:journalctl@ceph.osd.6.smithi029.stdout:Mar 16 12:09:48 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-6[60123]: 2024-03-16T12:09:48.979+0000 7fb55a3ef700 -1 osd.6 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:17.160443+0000 front 2024-03-16T12:09:17.160625+0000 (oldest deadline 2024-03-16T12:09:43.060171+0000) 2024-03-16T12:09:49.327 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 16 12:09:49 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-9[67182]: 2024-03-16T12:09:49.153+0000 7fe1cee8b700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:12.704136+0000 front 2024-03-16T12:09:12.703904+0000 (oldest deadline 2024-03-16T12:09:37.404049+0000) 2024-03-16T12:09:49.327 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 16 12:09:49 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-9[67182]: 2024-03-16T12:09:49.153+0000 7fe1cee8b700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:17.404712+0000 front 2024-03-16T12:09:17.404800+0000 (oldest deadline 2024-03-16T12:09:42.704627+0000) 2024-03-16T12:09:49.415 INFO:journalctl@ceph.osd.7.smithi152.stdout:Mar 16 12:09:49 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-7[54789]: 2024-03-16T12:09:49.191+0000 7f6c60c47700 -1 osd.7 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:17.304563+0000 front 2024-03-16T12:09:17.304652+0000 (oldest deadline 2024-03-16T12:09:43.204754+0000) 2024-03-16T12:09:49.442 DEBUG:teuthology.orchestra.run.smithi152:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d1887f28-e389-11ee-95c9-87774f69a715@osd.4.service' 2024-03-16T12:09:49.483 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-16T12:09:49.483 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-03-16T12:09:49.483 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-03-16T12:09:49.483 DEBUG:teuthology.orchestra.run.smithi154:> sudo systemctl stop ceph-d1887f28-e389-11ee-95c9-87774f69a715@osd.5 2024-03-16T12:09:49.531 INFO:journalctl@ceph.osd.8.smithi154.stdout:Mar 16 12:09:49 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-8[55054]: 2024-03-16T12:09:49.285+0000 7f3f065c0700 -1 osd.8 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:19.263068+0000 front 2024-03-16T12:09:19.263120+0000 (oldest deadline 2024-03-16T12:09:44.563033+0000) 2024-03-16T12:09:49.784 INFO:journalctl@ceph.osd.5.smithi154.stdout:Mar 16 12:09:49 smithi154 systemd[1]: Stopping Ceph osd.5 for d1887f28-e389-11ee-95c9-87774f69a715... 2024-03-16T12:09:49.867 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:09:49 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:09:49.603+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:10.713221+0000 front 2024-03-16T12:09:10.713087+0000 (oldest deadline 2024-03-16T12:09:35.413059+0000) 2024-03-16T12:09:49.867 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:09:49 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:09:49.603+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:20.614694+0000 front 2024-03-16T12:09:20.614671+0000 (oldest deadline 2024-03-16T12:09:45.215535+0000) 2024-03-16T12:09:50.284 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:09:49 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:09:49.882+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:13.025675+0000 front 2024-03-16T12:09:13.025847+0000 (oldest deadline 2024-03-16T12:09:35.325827+0000) 2024-03-16T12:09:50.285 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:09:49 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:09:49.882+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:19.927745+0000 front 2024-03-16T12:09:19.927761+0000 (oldest deadline 2024-03-16T12:09:45.827828+0000) 2024-03-16T12:09:50.285 INFO:journalctl@ceph.osd.5.smithi154.stdout:Mar 16 12:09:49 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-5[48458]: 2024-03-16T12:09:49.816+0000 7fc656102700 -1 osd.5 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:10.999320+0000 front 2024-03-16T12:09:10.999175+0000 (oldest deadline 2024-03-16T12:09:36.899159+0000) 2024-03-16T12:09:50.285 INFO:journalctl@ceph.osd.5.smithi154.stdout:Mar 16 12:09:49 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-5[48458]: 2024-03-16T12:09:49.816+0000 7fc656102700 -1 osd.5 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:16.899742+0000 front 2024-03-16T12:09:16.899720+0000 (oldest deadline 2024-03-16T12:09:42.799723+0000) 2024-03-16T12:09:50.285 INFO:journalctl@ceph.osd.5.smithi154.stdout:Mar 16 12:09:49 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-5[48458]: 2024-03-16T12:09:49.839+0000 7fc65db16700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-16T12:09:50.285 INFO:journalctl@ceph.osd.5.smithi154.stdout:Mar 16 12:09:49 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-5[48458]: 2024-03-16T12:09:49.839+0000 7fc65db16700 -1 osd.5 119 *** Got signal Terminated *** 2024-03-16T12:09:50.285 INFO:journalctl@ceph.osd.5.smithi154.stdout:Mar 16 12:09:49 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-5[48458]: 2024-03-16T12:09:49.839+0000 7fc65db16700 -1 osd.5 119 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-16T12:09:50.327 INFO:journalctl@ceph.osd.6.smithi029.stdout:Mar 16 12:09:50 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-6[60123]: 2024-03-16T12:09:50.000+0000 7fb55a3ef700 -1 osd.6 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:11.958752+0000 front 2024-03-16T12:09:11.958869+0000 (oldest deadline 2024-03-16T12:09:36.658956+0000) 2024-03-16T12:09:50.327 INFO:journalctl@ceph.osd.6.smithi029.stdout:Mar 16 12:09:50 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-6[60123]: 2024-03-16T12:09:50.000+0000 7fb55a3ef700 -1 osd.6 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:17.160443+0000 front 2024-03-16T12:09:17.160625+0000 (oldest deadline 2024-03-16T12:09:43.060171+0000) 2024-03-16T12:09:50.327 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 16 12:09:50 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-9[67182]: 2024-03-16T12:09:50.117+0000 7fe1cee8b700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:12.704136+0000 front 2024-03-16T12:09:12.703904+0000 (oldest deadline 2024-03-16T12:09:37.404049+0000) 2024-03-16T12:09:50.328 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 16 12:09:50 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-9[67182]: 2024-03-16T12:09:50.117+0000 7fe1cee8b700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:17.404712+0000 front 2024-03-16T12:09:17.404800+0000 (oldest deadline 2024-03-16T12:09:42.704627+0000) 2024-03-16T12:09:50.616 INFO:journalctl@ceph.osd.7.smithi152.stdout:Mar 16 12:09:50 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-7[54789]: 2024-03-16T12:09:50.148+0000 7f6c60c47700 -1 osd.7 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:17.304563+0000 front 2024-03-16T12:09:17.304652+0000 (oldest deadline 2024-03-16T12:09:43.204754+0000) 2024-03-16T12:09:50.784 INFO:journalctl@ceph.osd.8.smithi154.stdout:Mar 16 12:09:50 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-8[55054]: 2024-03-16T12:09:50.305+0000 7f3f065c0700 -1 osd.8 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:19.263068+0000 front 2024-03-16T12:09:19.263120+0000 (oldest deadline 2024-03-16T12:09:44.563033+0000) 2024-03-16T12:09:50.866 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:09:50 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:09:50.628+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:10.713221+0000 front 2024-03-16T12:09:10.713087+0000 (oldest deadline 2024-03-16T12:09:35.413059+0000) 2024-03-16T12:09:50.867 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:09:50 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:09:50.628+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:20.614694+0000 front 2024-03-16T12:09:20.614671+0000 (oldest deadline 2024-03-16T12:09:45.215535+0000) 2024-03-16T12:09:51.286 INFO:journalctl@ceph.osd.5.smithi154.stdout:Mar 16 12:09:50 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-5[48458]: 2024-03-16T12:09:50.863+0000 7fc656102700 -1 osd.5 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:10.999320+0000 front 2024-03-16T12:09:10.999175+0000 (oldest deadline 2024-03-16T12:09:36.899159+0000) 2024-03-16T12:09:51.286 INFO:journalctl@ceph.osd.5.smithi154.stdout:Mar 16 12:09:50 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-5[48458]: 2024-03-16T12:09:50.863+0000 7fc656102700 -1 osd.5 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:16.899742+0000 front 2024-03-16T12:09:16.899720+0000 (oldest deadline 2024-03-16T12:09:42.799723+0000) 2024-03-16T12:09:51.286 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:09:50 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:09:50.879+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:13.025675+0000 front 2024-03-16T12:09:13.025847+0000 (oldest deadline 2024-03-16T12:09:35.325827+0000) 2024-03-16T12:09:51.286 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:09:50 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:09:50.879+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:19.927745+0000 front 2024-03-16T12:09:19.927761+0000 (oldest deadline 2024-03-16T12:09:45.827828+0000) 2024-03-16T12:09:51.327 INFO:journalctl@ceph.osd.6.smithi029.stdout:Mar 16 12:09:50 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-6[60123]: 2024-03-16T12:09:50.957+0000 7fb55a3ef700 -1 osd.6 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:11.958752+0000 front 2024-03-16T12:09:11.958869+0000 (oldest deadline 2024-03-16T12:09:36.658956+0000) 2024-03-16T12:09:51.327 INFO:journalctl@ceph.osd.6.smithi029.stdout:Mar 16 12:09:50 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-6[60123]: 2024-03-16T12:09:50.957+0000 7fb55a3ef700 -1 osd.6 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:17.160443+0000 front 2024-03-16T12:09:17.160625+0000 (oldest deadline 2024-03-16T12:09:43.060171+0000) 2024-03-16T12:09:51.327 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 16 12:09:51 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-9[67182]: 2024-03-16T12:09:51.076+0000 7fe1cee8b700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:12.704136+0000 front 2024-03-16T12:09:12.703904+0000 (oldest deadline 2024-03-16T12:09:37.404049+0000) 2024-03-16T12:09:51.328 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 16 12:09:51 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-9[67182]: 2024-03-16T12:09:51.076+0000 7fe1cee8b700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:17.404712+0000 front 2024-03-16T12:09:17.404800+0000 (oldest deadline 2024-03-16T12:09:42.704627+0000) 2024-03-16T12:09:51.607 INFO:journalctl@ceph.osd.7.smithi152.stdout:Mar 16 12:09:51 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-7[54789]: 2024-03-16T12:09:51.168+0000 7f6c60c47700 -1 osd.7 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:17.304563+0000 front 2024-03-16T12:09:17.304652+0000 (oldest deadline 2024-03-16T12:09:43.204754+0000) 2024-03-16T12:09:51.784 INFO:journalctl@ceph.osd.8.smithi154.stdout:Mar 16 12:09:51 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-8[55054]: 2024-03-16T12:09:51.323+0000 7f3f065c0700 -1 osd.8 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:19.263068+0000 front 2024-03-16T12:09:19.263120+0000 (oldest deadline 2024-03-16T12:09:44.563033+0000) 2024-03-16T12:09:51.866 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:09:51 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:09:51.619+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:10.713221+0000 front 2024-03-16T12:09:10.713087+0000 (oldest deadline 2024-03-16T12:09:35.413059+0000) 2024-03-16T12:09:51.867 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:09:51 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:09:51.619+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:20.614694+0000 front 2024-03-16T12:09:20.614671+0000 (oldest deadline 2024-03-16T12:09:45.215535+0000) 2024-03-16T12:09:52.284 INFO:journalctl@ceph.osd.5.smithi154.stdout:Mar 16 12:09:51 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-5[48458]: 2024-03-16T12:09:51.901+0000 7fc656102700 -1 osd.5 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:10.999320+0000 front 2024-03-16T12:09:10.999175+0000 (oldest deadline 2024-03-16T12:09:36.899159+0000) 2024-03-16T12:09:52.285 INFO:journalctl@ceph.osd.5.smithi154.stdout:Mar 16 12:09:51 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-5[48458]: 2024-03-16T12:09:51.901+0000 7fc656102700 -1 osd.5 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:16.899742+0000 front 2024-03-16T12:09:16.899720+0000 (oldest deadline 2024-03-16T12:09:42.799723+0000) 2024-03-16T12:09:52.285 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:09:51 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:09:51.917+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:13.025675+0000 front 2024-03-16T12:09:13.025847+0000 (oldest deadline 2024-03-16T12:09:35.325827+0000) 2024-03-16T12:09:52.285 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:09:51 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:09:51.917+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:19.927745+0000 front 2024-03-16T12:09:19.927761+0000 (oldest deadline 2024-03-16T12:09:45.827828+0000) 2024-03-16T12:09:52.327 INFO:journalctl@ceph.osd.6.smithi029.stdout:Mar 16 12:09:51 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-6[60123]: 2024-03-16T12:09:51.968+0000 7fb55a3ef700 -1 osd.6 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:11.958752+0000 front 2024-03-16T12:09:11.958869+0000 (oldest deadline 2024-03-16T12:09:36.658956+0000) 2024-03-16T12:09:52.327 INFO:journalctl@ceph.osd.6.smithi029.stdout:Mar 16 12:09:51 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-6[60123]: 2024-03-16T12:09:51.968+0000 7fb55a3ef700 -1 osd.6 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:17.160443+0000 front 2024-03-16T12:09:17.160625+0000 (oldest deadline 2024-03-16T12:09:43.060171+0000) 2024-03-16T12:09:52.327 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 16 12:09:52 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-9[67182]: 2024-03-16T12:09:52.046+0000 7fe1cee8b700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:12.704136+0000 front 2024-03-16T12:09:12.703904+0000 (oldest deadline 2024-03-16T12:09:37.404049+0000) 2024-03-16T12:09:52.328 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 16 12:09:52 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-9[67182]: 2024-03-16T12:09:52.046+0000 7fe1cee8b700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:17.404712+0000 front 2024-03-16T12:09:17.404800+0000 (oldest deadline 2024-03-16T12:09:42.704627+0000) 2024-03-16T12:09:52.571 INFO:journalctl@ceph.osd.7.smithi152.stdout:Mar 16 12:09:52 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-7[54789]: 2024-03-16T12:09:52.206+0000 7f6c60c47700 -1 osd.7 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:17.304563+0000 front 2024-03-16T12:09:17.304652+0000 (oldest deadline 2024-03-16T12:09:43.204754+0000) 2024-03-16T12:09:52.784 INFO:journalctl@ceph.osd.8.smithi154.stdout:Mar 16 12:09:52 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-8[55054]: 2024-03-16T12:09:52.298+0000 7f3f065c0700 -1 osd.8 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:19.263068+0000 front 2024-03-16T12:09:19.263120+0000 (oldest deadline 2024-03-16T12:09:44.563033+0000) 2024-03-16T12:09:52.866 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:09:52 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:09:52.583+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:10.713221+0000 front 2024-03-16T12:09:10.713087+0000 (oldest deadline 2024-03-16T12:09:35.413059+0000) 2024-03-16T12:09:52.867 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:09:52 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:09:52.583+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:20.614694+0000 front 2024-03-16T12:09:20.614671+0000 (oldest deadline 2024-03-16T12:09:45.215535+0000) 2024-03-16T12:09:53.261 INFO:journalctl@ceph.osd.5.smithi154.stdout:Mar 16 12:09:52 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-5[48458]: 2024-03-16T12:09:52.886+0000 7fc656102700 -1 osd.5 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:10.999320+0000 front 2024-03-16T12:09:10.999175+0000 (oldest deadline 2024-03-16T12:09:36.899159+0000) 2024-03-16T12:09:53.261 INFO:journalctl@ceph.osd.5.smithi154.stdout:Mar 16 12:09:52 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-5[48458]: 2024-03-16T12:09:52.886+0000 7fc656102700 -1 osd.5 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:16.899742+0000 front 2024-03-16T12:09:16.899720+0000 (oldest deadline 2024-03-16T12:09:42.799723+0000) 2024-03-16T12:09:53.261 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:09:52 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:09:52.928+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:13.025675+0000 front 2024-03-16T12:09:13.025847+0000 (oldest deadline 2024-03-16T12:09:35.325827+0000) 2024-03-16T12:09:53.261 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:09:52 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:09:52.928+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:19.927745+0000 front 2024-03-16T12:09:19.927761+0000 (oldest deadline 2024-03-16T12:09:45.827828+0000) 2024-03-16T12:09:53.262 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:09:52 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:09:52.928+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.154:6804 osd.2 since back 2024-03-16T12:09:28.629320+0000 front 2024-03-16T12:09:28.629357+0000 (oldest deadline 2024-03-16T12:09:52.129472+0000) 2024-03-16T12:09:53.330 INFO:journalctl@ceph.osd.6.smithi029.stdout:Mar 16 12:09:52 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-6[60123]: 2024-03-16T12:09:52.929+0000 7fb55a3ef700 -1 osd.6 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:11.958752+0000 front 2024-03-16T12:09:11.958869+0000 (oldest deadline 2024-03-16T12:09:36.658956+0000) 2024-03-16T12:09:53.330 INFO:journalctl@ceph.osd.6.smithi029.stdout:Mar 16 12:09:52 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-6[60123]: 2024-03-16T12:09:52.929+0000 7fb55a3ef700 -1 osd.6 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:17.160443+0000 front 2024-03-16T12:09:17.160625+0000 (oldest deadline 2024-03-16T12:09:43.060171+0000) 2024-03-16T12:09:53.330 INFO:journalctl@ceph.osd.6.smithi029.stdout:Mar 16 12:09:52 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-6[60123]: 2024-03-16T12:09:52.929+0000 7fb55a3ef700 -1 osd.6 119 heartbeat_check: no reply from 172.21.15.154:6804 osd.2 since back 2024-03-16T12:09:29.962595+0000 front 2024-03-16T12:09:29.962754+0000 (oldest deadline 2024-03-16T12:09:52.862635+0000) 2024-03-16T12:09:53.331 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 16 12:09:53 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-9[67182]: 2024-03-16T12:09:53.059+0000 7fe1cee8b700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:12.704136+0000 front 2024-03-16T12:09:12.703904+0000 (oldest deadline 2024-03-16T12:09:37.404049+0000) 2024-03-16T12:09:53.331 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 16 12:09:53 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-9[67182]: 2024-03-16T12:09:53.059+0000 7fe1cee8b700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:17.404712+0000 front 2024-03-16T12:09:17.404800+0000 (oldest deadline 2024-03-16T12:09:42.704627+0000) 2024-03-16T12:09:53.331 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 16 12:09:53 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-9[67182]: 2024-03-16T12:09:53.059+0000 7fe1cee8b700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.154:6804 osd.2 since back 2024-03-16T12:09:27.405764+0000 front 2024-03-16T12:09:27.405893+0000 (oldest deadline 2024-03-16T12:09:52.105536+0000) 2024-03-16T12:09:53.534 INFO:journalctl@ceph.osd.8.smithi154.stdout:Mar 16 12:09:53 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-8[55054]: 2024-03-16T12:09:53.273+0000 7f3f065c0700 -1 osd.8 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:19.263068+0000 front 2024-03-16T12:09:19.263120+0000 (oldest deadline 2024-03-16T12:09:44.563033+0000) 2024-03-16T12:09:53.562 INFO:journalctl@ceph.osd.7.smithi152.stdout:Mar 16 12:09:53 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-7[54789]: 2024-03-16T12:09:53.157+0000 7f6c60c47700 -1 osd.7 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:17.304563+0000 front 2024-03-16T12:09:17.304652+0000 (oldest deadline 2024-03-16T12:09:43.204754+0000) 2024-03-16T12:09:53.563 INFO:journalctl@ceph.osd.7.smithi152.stdout:Mar 16 12:09:53 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-7[54789]: 2024-03-16T12:09:53.157+0000 7f6c60c47700 -1 osd.7 119 heartbeat_check: no reply from 172.21.15.154:6804 osd.2 since back 2024-03-16T12:09:27.607556+0000 front 2024-03-16T12:09:27.607478+0000 (oldest deadline 2024-03-16T12:09:52.307502+0000) 2024-03-16T12:09:53.866 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:09:53 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:09:53.574+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:10.713221+0000 front 2024-03-16T12:09:10.713087+0000 (oldest deadline 2024-03-16T12:09:35.413059+0000) 2024-03-16T12:09:53.867 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:09:53 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:09:53.574+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:20.614694+0000 front 2024-03-16T12:09:20.614671+0000 (oldest deadline 2024-03-16T12:09:45.215535+0000) 2024-03-16T12:09:53.867 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:09:53 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:09:53.574+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.154:6804 osd.2 since back 2024-03-16T12:09:29.916590+0000 front 2024-03-16T12:09:29.916612+0000 (oldest deadline 2024-03-16T12:09:52.816525+0000) 2024-03-16T12:09:54.256 INFO:journalctl@ceph.osd.5.smithi154.stdout:Mar 16 12:09:53 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-5[48458]: 2024-03-16T12:09:53.931+0000 7fc656102700 -1 osd.5 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:10.999320+0000 front 2024-03-16T12:09:10.999175+0000 (oldest deadline 2024-03-16T12:09:36.899159+0000) 2024-03-16T12:09:54.256 INFO:journalctl@ceph.osd.5.smithi154.stdout:Mar 16 12:09:53 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-5[48458]: 2024-03-16T12:09:53.931+0000 7fc656102700 -1 osd.5 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:16.899742+0000 front 2024-03-16T12:09:16.899720+0000 (oldest deadline 2024-03-16T12:09:42.799723+0000) 2024-03-16T12:09:54.257 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:09:53 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:09:53.929+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:13.025675+0000 front 2024-03-16T12:09:13.025847+0000 (oldest deadline 2024-03-16T12:09:35.325827+0000) 2024-03-16T12:09:54.257 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:09:53 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:09:53.929+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:19.927745+0000 front 2024-03-16T12:09:19.927761+0000 (oldest deadline 2024-03-16T12:09:45.827828+0000) 2024-03-16T12:09:54.257 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:09:53 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:09:53.929+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.154:6804 osd.2 since back 2024-03-16T12:09:28.629320+0000 front 2024-03-16T12:09:28.629357+0000 (oldest deadline 2024-03-16T12:09:52.129472+0000) 2024-03-16T12:09:54.327 INFO:journalctl@ceph.osd.6.smithi029.stdout:Mar 16 12:09:53 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-6[60123]: 2024-03-16T12:09:53.953+0000 7fb55a3ef700 -1 osd.6 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:11.958752+0000 front 2024-03-16T12:09:11.958869+0000 (oldest deadline 2024-03-16T12:09:36.658956+0000) 2024-03-16T12:09:54.327 INFO:journalctl@ceph.osd.6.smithi029.stdout:Mar 16 12:09:53 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-6[60123]: 2024-03-16T12:09:53.953+0000 7fb55a3ef700 -1 osd.6 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:17.160443+0000 front 2024-03-16T12:09:17.160625+0000 (oldest deadline 2024-03-16T12:09:43.060171+0000) 2024-03-16T12:09:54.327 INFO:journalctl@ceph.osd.6.smithi029.stdout:Mar 16 12:09:53 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-6[60123]: 2024-03-16T12:09:53.953+0000 7fb55a3ef700 -1 osd.6 119 heartbeat_check: no reply from 172.21.15.154:6804 osd.2 since back 2024-03-16T12:09:29.962595+0000 front 2024-03-16T12:09:29.962754+0000 (oldest deadline 2024-03-16T12:09:52.862635+0000) 2024-03-16T12:09:54.328 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 16 12:09:54 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-9[67182]: 2024-03-16T12:09:54.104+0000 7fe1cee8b700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:12.704136+0000 front 2024-03-16T12:09:12.703904+0000 (oldest deadline 2024-03-16T12:09:37.404049+0000) 2024-03-16T12:09:54.328 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 16 12:09:54 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-9[67182]: 2024-03-16T12:09:54.104+0000 7fe1cee8b700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:17.404712+0000 front 2024-03-16T12:09:17.404800+0000 (oldest deadline 2024-03-16T12:09:42.704627+0000) 2024-03-16T12:09:54.328 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 16 12:09:54 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-9[67182]: 2024-03-16T12:09:54.104+0000 7fe1cee8b700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.154:6804 osd.2 since back 2024-03-16T12:09:27.405764+0000 front 2024-03-16T12:09:27.405893+0000 (oldest deadline 2024-03-16T12:09:52.105536+0000) 2024-03-16T12:09:54.367 INFO:journalctl@ceph.osd.7.smithi152.stdout:Mar 16 12:09:54 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-7[54789]: 2024-03-16T12:09:54.123+0000 7f6c60c47700 -1 osd.7 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:17.304563+0000 front 2024-03-16T12:09:17.304652+0000 (oldest deadline 2024-03-16T12:09:43.204754+0000) 2024-03-16T12:09:54.367 INFO:journalctl@ceph.osd.7.smithi152.stdout:Mar 16 12:09:54 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-7[54789]: 2024-03-16T12:09:54.123+0000 7f6c60c47700 -1 osd.7 119 heartbeat_check: no reply from 172.21.15.154:6804 osd.2 since back 2024-03-16T12:09:27.607556+0000 front 2024-03-16T12:09:27.607478+0000 (oldest deadline 2024-03-16T12:09:52.307502+0000) 2024-03-16T12:09:54.534 INFO:journalctl@ceph.osd.8.smithi154.stdout:Mar 16 12:09:54 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-8[55054]: 2024-03-16T12:09:54.266+0000 7f3f065c0700 -1 osd.8 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:19.263068+0000 front 2024-03-16T12:09:19.263120+0000 (oldest deadline 2024-03-16T12:09:44.563033+0000) 2024-03-16T12:09:54.534 INFO:journalctl@ceph.osd.8.smithi154.stdout:Mar 16 12:09:54 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-8[55054]: 2024-03-16T12:09:54.266+0000 7f3f065c0700 -1 osd.8 119 heartbeat_check: no reply from 172.21.15.154:6804 osd.2 since back 2024-03-16T12:09:29.566064+0000 front 2024-03-16T12:09:29.566042+0000 (oldest deadline 2024-03-16T12:09:54.266168+0000) 2024-03-16T12:09:54.866 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:09:54 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:09:54.612+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:10.713221+0000 front 2024-03-16T12:09:10.713087+0000 (oldest deadline 2024-03-16T12:09:35.413059+0000) 2024-03-16T12:09:54.867 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:09:54 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:09:54.612+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:20.614694+0000 front 2024-03-16T12:09:20.614671+0000 (oldest deadline 2024-03-16T12:09:45.215535+0000) 2024-03-16T12:09:54.867 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:09:54 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:09:54.612+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.154:6804 osd.2 since back 2024-03-16T12:09:29.916590+0000 front 2024-03-16T12:09:29.916612+0000 (oldest deadline 2024-03-16T12:09:52.816525+0000) 2024-03-16T12:09:55.207 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:09:54 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:09:54.919+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:13.025675+0000 front 2024-03-16T12:09:13.025847+0000 (oldest deadline 2024-03-16T12:09:35.325827+0000) 2024-03-16T12:09:55.207 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:09:54 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:09:54.919+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:19.927745+0000 front 2024-03-16T12:09:19.927761+0000 (oldest deadline 2024-03-16T12:09:45.827828+0000) 2024-03-16T12:09:55.208 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:09:54 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:09:54.919+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.154:6804 osd.2 since back 2024-03-16T12:09:28.629320+0000 front 2024-03-16T12:09:28.629357+0000 (oldest deadline 2024-03-16T12:09:52.129472+0000) 2024-03-16T12:09:55.327 INFO:journalctl@ceph.osd.6.smithi029.stdout:Mar 16 12:09:54 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-6[60123]: 2024-03-16T12:09:54.983+0000 7fb55a3ef700 -1 osd.6 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:11.958752+0000 front 2024-03-16T12:09:11.958869+0000 (oldest deadline 2024-03-16T12:09:36.658956+0000) 2024-03-16T12:09:55.327 INFO:journalctl@ceph.osd.6.smithi029.stdout:Mar 16 12:09:54 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-6[60123]: 2024-03-16T12:09:54.983+0000 7fb55a3ef700 -1 osd.6 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:17.160443+0000 front 2024-03-16T12:09:17.160625+0000 (oldest deadline 2024-03-16T12:09:43.060171+0000) 2024-03-16T12:09:55.327 INFO:journalctl@ceph.osd.6.smithi029.stdout:Mar 16 12:09:54 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-6[60123]: 2024-03-16T12:09:54.983+0000 7fb55a3ef700 -1 osd.6 119 heartbeat_check: no reply from 172.21.15.154:6804 osd.2 since back 2024-03-16T12:09:29.962595+0000 front 2024-03-16T12:09:29.962754+0000 (oldest deadline 2024-03-16T12:09:52.862635+0000) 2024-03-16T12:09:55.328 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 16 12:09:55 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-9[67182]: 2024-03-16T12:09:55.063+0000 7fe1cee8b700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:12.704136+0000 front 2024-03-16T12:09:12.703904+0000 (oldest deadline 2024-03-16T12:09:37.404049+0000) 2024-03-16T12:09:55.328 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 16 12:09:55 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-9[67182]: 2024-03-16T12:09:55.063+0000 7fe1cee8b700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:17.404712+0000 front 2024-03-16T12:09:17.404800+0000 (oldest deadline 2024-03-16T12:09:42.704627+0000) 2024-03-16T12:09:55.328 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 16 12:09:55 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-9[67182]: 2024-03-16T12:09:55.063+0000 7fe1cee8b700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.154:6804 osd.2 since back 2024-03-16T12:09:27.405764+0000 front 2024-03-16T12:09:27.405893+0000 (oldest deadline 2024-03-16T12:09:52.105536+0000) 2024-03-16T12:09:55.366 INFO:journalctl@ceph.osd.7.smithi152.stdout:Mar 16 12:09:55 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-7[54789]: 2024-03-16T12:09:55.077+0000 7f6c60c47700 -1 osd.7 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:17.304563+0000 front 2024-03-16T12:09:17.304652+0000 (oldest deadline 2024-03-16T12:09:43.204754+0000) 2024-03-16T12:09:55.366 INFO:journalctl@ceph.osd.7.smithi152.stdout:Mar 16 12:09:55 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-7[54789]: 2024-03-16T12:09:55.077+0000 7f6c60c47700 -1 osd.7 119 heartbeat_check: no reply from 172.21.15.154:6804 osd.2 since back 2024-03-16T12:09:27.607556+0000 front 2024-03-16T12:09:27.607478+0000 (oldest deadline 2024-03-16T12:09:52.307502+0000) 2024-03-16T12:09:55.534 INFO:journalctl@ceph.osd.5.smithi154.stdout:Mar 16 12:09:55 smithi154 podman[75123]: 2024-03-16 12:09:55.436882384 +0000 UTC m=+5.750624829 container died 87e3c930ba34a00ee9a22742560b65e95fb1681f513efdffd14a98430f60b746 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-5, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-03-16T12:09:55.535 INFO:journalctl@ceph.osd.8.smithi154.stdout:Mar 16 12:09:55 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-8[55054]: 2024-03-16T12:09:55.219+0000 7f3f065c0700 -1 osd.8 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:19.263068+0000 front 2024-03-16T12:09:19.263120+0000 (oldest deadline 2024-03-16T12:09:44.563033+0000) 2024-03-16T12:09:55.535 INFO:journalctl@ceph.osd.8.smithi154.stdout:Mar 16 12:09:55 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-8[55054]: 2024-03-16T12:09:55.219+0000 7f3f065c0700 -1 osd.8 119 heartbeat_check: no reply from 172.21.15.154:6804 osd.2 since back 2024-03-16T12:09:29.566064+0000 front 2024-03-16T12:09:29.566042+0000 (oldest deadline 2024-03-16T12:09:54.266168+0000) 2024-03-16T12:09:56.034 INFO:journalctl@ceph.osd.5.smithi154.stdout:Mar 16 12:09:55 smithi154 podman[75123]: 2024-03-16 12:09:55.838195923 +0000 UTC m=+6.151938368 container cleanup 87e3c930ba34a00ee9a22742560b65e95fb1681f513efdffd14a98430f60b746 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_BRANCH=HEAD) 2024-03-16T12:09:56.034 INFO:journalctl@ceph.osd.5.smithi154.stdout:Mar 16 12:09:55 smithi154 bash[75123]: ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-5 2024-03-16T12:09:56.035 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:09:55 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:09:55.942+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:13.025675+0000 front 2024-03-16T12:09:13.025847+0000 (oldest deadline 2024-03-16T12:09:35.325827+0000) 2024-03-16T12:09:56.035 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:09:55 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:09:55.942+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:19.927745+0000 front 2024-03-16T12:09:19.927761+0000 (oldest deadline 2024-03-16T12:09:45.827828+0000) 2024-03-16T12:09:56.035 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:09:55 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:09:55.942+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.154:6804 osd.2 since back 2024-03-16T12:09:28.629320+0000 front 2024-03-16T12:09:28.629357+0000 (oldest deadline 2024-03-16T12:09:52.129472+0000) 2024-03-16T12:09:56.060 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:09:55 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:09:55.660+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:10.713221+0000 front 2024-03-16T12:09:10.713087+0000 (oldest deadline 2024-03-16T12:09:35.413059+0000) 2024-03-16T12:09:56.060 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:09:55 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:09:55.660+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:20.614694+0000 front 2024-03-16T12:09:20.614671+0000 (oldest deadline 2024-03-16T12:09:45.215535+0000) 2024-03-16T12:09:56.061 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:09:55 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:09:55.660+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.154:6804 osd.2 since back 2024-03-16T12:09:29.916590+0000 front 2024-03-16T12:09:29.916612+0000 (oldest deadline 2024-03-16T12:09:52.816525+0000) 2024-03-16T12:09:56.327 INFO:journalctl@ceph.osd.6.smithi029.stdout:Mar 16 12:09:55 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-6[60123]: 2024-03-16T12:09:55.952+0000 7fb55a3ef700 -1 osd.6 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:11.958752+0000 front 2024-03-16T12:09:11.958869+0000 (oldest deadline 2024-03-16T12:09:36.658956+0000) 2024-03-16T12:09:56.327 INFO:journalctl@ceph.osd.6.smithi029.stdout:Mar 16 12:09:55 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-6[60123]: 2024-03-16T12:09:55.952+0000 7fb55a3ef700 -1 osd.6 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:17.160443+0000 front 2024-03-16T12:09:17.160625+0000 (oldest deadline 2024-03-16T12:09:43.060171+0000) 2024-03-16T12:09:56.327 INFO:journalctl@ceph.osd.6.smithi029.stdout:Mar 16 12:09:55 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-6[60123]: 2024-03-16T12:09:55.952+0000 7fb55a3ef700 -1 osd.6 119 heartbeat_check: no reply from 172.21.15.154:6804 osd.2 since back 2024-03-16T12:09:29.962595+0000 front 2024-03-16T12:09:29.962754+0000 (oldest deadline 2024-03-16T12:09:52.862635+0000) 2024-03-16T12:09:56.327 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 16 12:09:56 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-9[67182]: 2024-03-16T12:09:56.041+0000 7fe1cee8b700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:12.704136+0000 front 2024-03-16T12:09:12.703904+0000 (oldest deadline 2024-03-16T12:09:37.404049+0000) 2024-03-16T12:09:56.328 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 16 12:09:56 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-9[67182]: 2024-03-16T12:09:56.041+0000 7fe1cee8b700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:17.404712+0000 front 2024-03-16T12:09:17.404800+0000 (oldest deadline 2024-03-16T12:09:42.704627+0000) 2024-03-16T12:09:56.328 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 16 12:09:56 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-9[67182]: 2024-03-16T12:09:56.041+0000 7fe1cee8b700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.154:6804 osd.2 since back 2024-03-16T12:09:27.405764+0000 front 2024-03-16T12:09:27.405893+0000 (oldest deadline 2024-03-16T12:09:52.105536+0000) 2024-03-16T12:09:56.366 INFO:journalctl@ceph.osd.7.smithi152.stdout:Mar 16 12:09:56 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-7[54789]: 2024-03-16T12:09:56.072+0000 7f6c60c47700 -1 osd.7 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:17.304563+0000 front 2024-03-16T12:09:17.304652+0000 (oldest deadline 2024-03-16T12:09:43.204754+0000) 2024-03-16T12:09:56.367 INFO:journalctl@ceph.osd.7.smithi152.stdout:Mar 16 12:09:56 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-7[54789]: 2024-03-16T12:09:56.072+0000 7f6c60c47700 -1 osd.7 119 heartbeat_check: no reply from 172.21.15.154:6804 osd.2 since back 2024-03-16T12:09:27.607556+0000 front 2024-03-16T12:09:27.607478+0000 (oldest deadline 2024-03-16T12:09:52.307502+0000) 2024-03-16T12:09:56.385 INFO:journalctl@ceph.osd.5.smithi154.stdout:Mar 16 12:09:56 smithi154 podman[75140]: 2024-03-16 12:09:56.090257816 +0000 UTC m=+0.643375010 container remove 87e3c930ba34a00ee9a22742560b65e95fb1681f513efdffd14a98430f60b746 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-5, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-16T12:09:56.385 INFO:journalctl@ceph.osd.8.smithi154.stdout:Mar 16 12:09:56 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-8[55054]: 2024-03-16T12:09:56.169+0000 7f3f065c0700 -1 osd.8 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:19.263068+0000 front 2024-03-16T12:09:19.263120+0000 (oldest deadline 2024-03-16T12:09:44.563033+0000) 2024-03-16T12:09:56.386 INFO:journalctl@ceph.osd.8.smithi154.stdout:Mar 16 12:09:56 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-8[55054]: 2024-03-16T12:09:56.169+0000 7f3f065c0700 -1 osd.8 119 heartbeat_check: no reply from 172.21.15.154:6804 osd.2 since back 2024-03-16T12:09:29.566064+0000 front 2024-03-16T12:09:29.566042+0000 (oldest deadline 2024-03-16T12:09:54.266168+0000) 2024-03-16T12:09:56.707 INFO:journalctl@ceph.osd.5.smithi154.stdout:Mar 16 12:09:56 smithi154 podman[75231]: 2024-03-16 12:09:56.300302114 +0000 UTC m=+0.017608654 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-16T12:09:56.707 INFO:journalctl@ceph.osd.5.smithi154.stdout:Mar 16 12:09:56 smithi154 podman[75231]: 2024-03-16 12:09:56.441441581 +0000 UTC m=+0.158748100 container create cc582bd58c9d78c9a7f1d88229711068e48270c73a1a770014362198d4a4be3c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-5-deactivate, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-16T12:09:57.015 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:09:56 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:09:56.934+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:13.025675+0000 front 2024-03-16T12:09:13.025847+0000 (oldest deadline 2024-03-16T12:09:35.325827+0000) 2024-03-16T12:09:57.015 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:09:56 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:09:56.934+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:19.927745+0000 front 2024-03-16T12:09:19.927761+0000 (oldest deadline 2024-03-16T12:09:45.827828+0000) 2024-03-16T12:09:57.015 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:09:56 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:09:56.934+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.154:6804 osd.2 since back 2024-03-16T12:09:28.629320+0000 front 2024-03-16T12:09:28.629357+0000 (oldest deadline 2024-03-16T12:09:52.129472+0000) 2024-03-16T12:09:57.015 INFO:journalctl@ceph.osd.5.smithi154.stdout:Mar 16 12:09:56 smithi154 podman[75231]: 2024-03-16 12:09:56.719265958 +0000 UTC m=+0.436572497 container init cc582bd58c9d78c9a7f1d88229711068e48270c73a1a770014362198d4a4be3c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-5-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True) 2024-03-16T12:09:57.015 INFO:journalctl@ceph.osd.5.smithi154.stdout:Mar 16 12:09:56 smithi154 podman[75231]: 2024-03-16 12:09:56.723215635 +0000 UTC m=+0.440522163 container start cc582bd58c9d78c9a7f1d88229711068e48270c73a1a770014362198d4a4be3c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-5-deactivate, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b) 2024-03-16T12:09:57.016 INFO:journalctl@ceph.osd.5.smithi154.stdout:Mar 16 12:09:56 smithi154 podman[75231]: 2024-03-16 12:09:56.827461183 +0000 UTC m=+0.544767722 container attach cc582bd58c9d78c9a7f1d88229711068e48270c73a1a770014362198d4a4be3c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-5-deactivate, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-03-16T12:09:57.046 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:09:56 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:09:56.705+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:10.713221+0000 front 2024-03-16T12:09:10.713087+0000 (oldest deadline 2024-03-16T12:09:35.413059+0000) 2024-03-16T12:09:57.046 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:09:56 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:09:56.705+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:20.614694+0000 front 2024-03-16T12:09:20.614671+0000 (oldest deadline 2024-03-16T12:09:45.215535+0000) 2024-03-16T12:09:57.047 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:09:56 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:09:56.705+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.154:6804 osd.2 since back 2024-03-16T12:09:29.916590+0000 front 2024-03-16T12:09:29.916612+0000 (oldest deadline 2024-03-16T12:09:52.816525+0000) 2024-03-16T12:09:57.284 INFO:journalctl@ceph.osd.5.smithi154.stdout:Mar 16 12:09:57 smithi154 podman[75231]: 2024-03-16 12:09:57.027199814 +0000 UTC m=+0.744506338 container died cc582bd58c9d78c9a7f1d88229711068e48270c73a1a770014362198d4a4be3c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-5-deactivate, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, ceph=True) 2024-03-16T12:09:57.285 INFO:journalctl@ceph.osd.8.smithi154.stdout:Mar 16 12:09:57 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-8[55054]: 2024-03-16T12:09:57.157+0000 7f3f065c0700 -1 osd.8 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:19.263068+0000 front 2024-03-16T12:09:19.263120+0000 (oldest deadline 2024-03-16T12:09:44.563033+0000) 2024-03-16T12:09:57.285 INFO:journalctl@ceph.osd.8.smithi154.stdout:Mar 16 12:09:57 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-8[55054]: 2024-03-16T12:09:57.157+0000 7f3f065c0700 -1 osd.8 119 heartbeat_check: no reply from 172.21.15.154:6804 osd.2 since back 2024-03-16T12:09:29.566064+0000 front 2024-03-16T12:09:29.566042+0000 (oldest deadline 2024-03-16T12:09:54.266168+0000) 2024-03-16T12:09:57.327 INFO:journalctl@ceph.osd.6.smithi029.stdout:Mar 16 12:09:56 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-6[60123]: 2024-03-16T12:09:56.963+0000 7fb55a3ef700 -1 osd.6 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:11.958752+0000 front 2024-03-16T12:09:11.958869+0000 (oldest deadline 2024-03-16T12:09:36.658956+0000) 2024-03-16T12:09:57.327 INFO:journalctl@ceph.osd.6.smithi029.stdout:Mar 16 12:09:56 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-6[60123]: 2024-03-16T12:09:56.963+0000 7fb55a3ef700 -1 osd.6 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:17.160443+0000 front 2024-03-16T12:09:17.160625+0000 (oldest deadline 2024-03-16T12:09:43.060171+0000) 2024-03-16T12:09:57.327 INFO:journalctl@ceph.osd.6.smithi029.stdout:Mar 16 12:09:56 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-6[60123]: 2024-03-16T12:09:56.963+0000 7fb55a3ef700 -1 osd.6 119 heartbeat_check: no reply from 172.21.15.154:6804 osd.2 since back 2024-03-16T12:09:29.962595+0000 front 2024-03-16T12:09:29.962754+0000 (oldest deadline 2024-03-16T12:09:52.862635+0000) 2024-03-16T12:09:57.327 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 16 12:09:57 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-9[67182]: 2024-03-16T12:09:57.082+0000 7fe1cee8b700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:12.704136+0000 front 2024-03-16T12:09:12.703904+0000 (oldest deadline 2024-03-16T12:09:37.404049+0000) 2024-03-16T12:09:57.327 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 16 12:09:57 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-9[67182]: 2024-03-16T12:09:57.082+0000 7fe1cee8b700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:17.404712+0000 front 2024-03-16T12:09:17.404800+0000 (oldest deadline 2024-03-16T12:09:42.704627+0000) 2024-03-16T12:09:57.328 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 16 12:09:57 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-9[67182]: 2024-03-16T12:09:57.082+0000 7fe1cee8b700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.154:6804 osd.2 since back 2024-03-16T12:09:27.405764+0000 front 2024-03-16T12:09:27.405893+0000 (oldest deadline 2024-03-16T12:09:52.105536+0000) 2024-03-16T12:09:57.366 INFO:journalctl@ceph.osd.7.smithi152.stdout:Mar 16 12:09:57 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-7[54789]: 2024-03-16T12:09:57.058+0000 7f6c60c47700 -1 osd.7 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:17.304563+0000 front 2024-03-16T12:09:17.304652+0000 (oldest deadline 2024-03-16T12:09:43.204754+0000) 2024-03-16T12:09:57.366 INFO:journalctl@ceph.osd.7.smithi152.stdout:Mar 16 12:09:57 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-7[54789]: 2024-03-16T12:09:57.058+0000 7f6c60c47700 -1 osd.7 119 heartbeat_check: no reply from 172.21.15.154:6804 osd.2 since back 2024-03-16T12:09:27.607556+0000 front 2024-03-16T12:09:27.607478+0000 (oldest deadline 2024-03-16T12:09:52.307502+0000) 2024-03-16T12:09:57.736 DEBUG:teuthology.orchestra.run.smithi154:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d1887f28-e389-11ee-95c9-87774f69a715@osd.5.service' 2024-03-16T12:09:57.776 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-16T12:09:57.777 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-03-16T12:09:57.777 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-03-16T12:09:57.777 DEBUG:teuthology.orchestra.run.smithi029:> sudo systemctl stop ceph-d1887f28-e389-11ee-95c9-87774f69a715@osd.6 2024-03-16T12:09:58.034 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:09:57 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:09:57.898+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:13.025675+0000 front 2024-03-16T12:09:13.025847+0000 (oldest deadline 2024-03-16T12:09:35.325827+0000) 2024-03-16T12:09:58.035 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:09:57 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:09:57.898+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:19.927745+0000 front 2024-03-16T12:09:19.927761+0000 (oldest deadline 2024-03-16T12:09:45.827828+0000) 2024-03-16T12:09:58.035 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:09:57 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:09:57.898+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.154:6804 osd.2 since back 2024-03-16T12:09:28.629320+0000 front 2024-03-16T12:09:28.629357+0000 (oldest deadline 2024-03-16T12:09:52.129472+0000) 2024-03-16T12:09:58.077 INFO:journalctl@ceph.osd.6.smithi029.stdout:Mar 16 12:09:57 smithi029 systemd[1]: Stopping Ceph osd.6 for d1887f28-e389-11ee-95c9-87774f69a715... 2024-03-16T12:09:58.077 INFO:journalctl@ceph.osd.6.smithi029.stdout:Mar 16 12:09:57 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-6[60123]: 2024-03-16T12:09:57.924+0000 7fb55a3ef700 -1 osd.6 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:11.958752+0000 front 2024-03-16T12:09:11.958869+0000 (oldest deadline 2024-03-16T12:09:36.658956+0000) 2024-03-16T12:09:58.077 INFO:journalctl@ceph.osd.6.smithi029.stdout:Mar 16 12:09:57 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-6[60123]: 2024-03-16T12:09:57.925+0000 7fb55a3ef700 -1 osd.6 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:17.160443+0000 front 2024-03-16T12:09:17.160625+0000 (oldest deadline 2024-03-16T12:09:43.060171+0000) 2024-03-16T12:09:58.077 INFO:journalctl@ceph.osd.6.smithi029.stdout:Mar 16 12:09:57 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-6[60123]: 2024-03-16T12:09:57.925+0000 7fb55a3ef700 -1 osd.6 119 heartbeat_check: no reply from 172.21.15.154:6804 osd.2 since back 2024-03-16T12:09:29.962595+0000 front 2024-03-16T12:09:29.962754+0000 (oldest deadline 2024-03-16T12:09:52.862635+0000) 2024-03-16T12:09:58.084 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:09:57 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:09:57.673+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:10.713221+0000 front 2024-03-16T12:09:10.713087+0000 (oldest deadline 2024-03-16T12:09:35.413059+0000) 2024-03-16T12:09:58.084 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:09:57 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:09:57.673+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:20.614694+0000 front 2024-03-16T12:09:20.614671+0000 (oldest deadline 2024-03-16T12:09:45.215535+0000) 2024-03-16T12:09:58.084 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:09:57 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:09:57.673+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.154:6804 osd.2 since back 2024-03-16T12:09:29.916590+0000 front 2024-03-16T12:09:29.916612+0000 (oldest deadline 2024-03-16T12:09:52.816525+0000) 2024-03-16T12:09:58.366 INFO:journalctl@ceph.osd.7.smithi152.stdout:Mar 16 12:09:58 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-7[54789]: 2024-03-16T12:09:58.095+0000 7f6c60c47700 -1 osd.7 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:17.304563+0000 front 2024-03-16T12:09:17.304652+0000 (oldest deadline 2024-03-16T12:09:43.204754+0000) 2024-03-16T12:09:58.366 INFO:journalctl@ceph.osd.7.smithi152.stdout:Mar 16 12:09:58 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-7[54789]: 2024-03-16T12:09:58.095+0000 7f6c60c47700 -1 osd.7 119 heartbeat_check: no reply from 172.21.15.154:6804 osd.2 since back 2024-03-16T12:09:27.607556+0000 front 2024-03-16T12:09:27.607478+0000 (oldest deadline 2024-03-16T12:09:52.307502+0000) 2024-03-16T12:09:58.534 INFO:journalctl@ceph.osd.8.smithi154.stdout:Mar 16 12:09:58 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-8[55054]: 2024-03-16T12:09:58.111+0000 7f3f065c0700 -1 osd.8 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:19.263068+0000 front 2024-03-16T12:09:19.263120+0000 (oldest deadline 2024-03-16T12:09:44.563033+0000) 2024-03-16T12:09:58.534 INFO:journalctl@ceph.osd.8.smithi154.stdout:Mar 16 12:09:58 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-8[55054]: 2024-03-16T12:09:58.111+0000 7f3f065c0700 -1 osd.8 119 heartbeat_check: no reply from 172.21.15.154:6804 osd.2 since back 2024-03-16T12:09:29.566064+0000 front 2024-03-16T12:09:29.566042+0000 (oldest deadline 2024-03-16T12:09:54.266168+0000) 2024-03-16T12:09:58.577 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 16 12:09:58 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-9[67182]: 2024-03-16T12:09:58.119+0000 7fe1cee8b700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:12.704136+0000 front 2024-03-16T12:09:12.703904+0000 (oldest deadline 2024-03-16T12:09:37.404049+0000) 2024-03-16T12:09:58.577 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 16 12:09:58 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-9[67182]: 2024-03-16T12:09:58.119+0000 7fe1cee8b700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:17.404712+0000 front 2024-03-16T12:09:17.404800+0000 (oldest deadline 2024-03-16T12:09:42.704627+0000) 2024-03-16T12:09:58.577 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 16 12:09:58 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-9[67182]: 2024-03-16T12:09:58.119+0000 7fe1cee8b700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.154:6804 osd.2 since back 2024-03-16T12:09:27.405764+0000 front 2024-03-16T12:09:27.405893+0000 (oldest deadline 2024-03-16T12:09:52.105536+0000) 2024-03-16T12:09:58.577 INFO:journalctl@ceph.osd.6.smithi029.stdout:Mar 16 12:09:58 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-6[60123]: 2024-03-16T12:09:58.108+0000 7fb561e03700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-16T12:09:58.577 INFO:journalctl@ceph.osd.6.smithi029.stdout:Mar 16 12:09:58 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-6[60123]: 2024-03-16T12:09:58.108+0000 7fb561e03700 -1 osd.6 119 *** Got signal Terminated *** 2024-03-16T12:09:58.578 INFO:journalctl@ceph.osd.6.smithi029.stdout:Mar 16 12:09:58 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-6[60123]: 2024-03-16T12:09:58.108+0000 7fb561e03700 -1 osd.6 119 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-16T12:09:59.116 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:09:58 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:09:58.666+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:10.713221+0000 front 2024-03-16T12:09:10.713087+0000 (oldest deadline 2024-03-16T12:09:35.413059+0000) 2024-03-16T12:09:59.117 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:09:58 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:09:58.666+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:20.614694+0000 front 2024-03-16T12:09:20.614671+0000 (oldest deadline 2024-03-16T12:09:45.215535+0000) 2024-03-16T12:09:59.117 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:09:58 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:09:58.666+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.154:6804 osd.2 since back 2024-03-16T12:09:29.916590+0000 front 2024-03-16T12:09:29.916612+0000 (oldest deadline 2024-03-16T12:09:52.816525+0000) 2024-03-16T12:09:59.284 INFO:journalctl@ceph.osd.8.smithi154.stdout:Mar 16 12:09:59 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-8[55054]: 2024-03-16T12:09:59.069+0000 7f3f065c0700 -1 osd.8 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:19.263068+0000 front 2024-03-16T12:09:19.263120+0000 (oldest deadline 2024-03-16T12:09:44.563033+0000) 2024-03-16T12:09:59.284 INFO:journalctl@ceph.osd.8.smithi154.stdout:Mar 16 12:09:59 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-8[55054]: 2024-03-16T12:09:59.069+0000 7f3f065c0700 -1 osd.8 119 heartbeat_check: no reply from 172.21.15.154:6804 osd.2 since back 2024-03-16T12:09:29.566064+0000 front 2024-03-16T12:09:29.566042+0000 (oldest deadline 2024-03-16T12:09:54.266168+0000) 2024-03-16T12:09:59.285 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:09:58 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:09:58.880+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:13.025675+0000 front 2024-03-16T12:09:13.025847+0000 (oldest deadline 2024-03-16T12:09:35.325827+0000) 2024-03-16T12:09:59.285 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:09:58 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:09:58.880+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:19.927745+0000 front 2024-03-16T12:09:19.927761+0000 (oldest deadline 2024-03-16T12:09:45.827828+0000) 2024-03-16T12:09:59.285 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:09:58 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:09:58.880+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.154:6804 osd.2 since back 2024-03-16T12:09:28.629320+0000 front 2024-03-16T12:09:28.629357+0000 (oldest deadline 2024-03-16T12:09:52.129472+0000) 2024-03-16T12:09:59.327 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 16 12:09:59 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-9[67182]: 2024-03-16T12:09:59.080+0000 7fe1cee8b700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:12.704136+0000 front 2024-03-16T12:09:12.703904+0000 (oldest deadline 2024-03-16T12:09:37.404049+0000) 2024-03-16T12:09:59.327 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 16 12:09:59 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-9[67182]: 2024-03-16T12:09:59.080+0000 7fe1cee8b700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:17.404712+0000 front 2024-03-16T12:09:17.404800+0000 (oldest deadline 2024-03-16T12:09:42.704627+0000) 2024-03-16T12:09:59.327 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 16 12:09:59 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-9[67182]: 2024-03-16T12:09:59.080+0000 7fe1cee8b700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.154:6804 osd.2 since back 2024-03-16T12:09:27.405764+0000 front 2024-03-16T12:09:27.405893+0000 (oldest deadline 2024-03-16T12:09:52.105536+0000) 2024-03-16T12:09:59.327 INFO:journalctl@ceph.osd.6.smithi029.stdout:Mar 16 12:09:58 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-6[60123]: 2024-03-16T12:09:58.880+0000 7fb55a3ef700 -1 osd.6 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:11.958752+0000 front 2024-03-16T12:09:11.958869+0000 (oldest deadline 2024-03-16T12:09:36.658956+0000) 2024-03-16T12:09:59.328 INFO:journalctl@ceph.osd.6.smithi029.stdout:Mar 16 12:09:58 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-6[60123]: 2024-03-16T12:09:58.880+0000 7fb55a3ef700 -1 osd.6 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:17.160443+0000 front 2024-03-16T12:09:17.160625+0000 (oldest deadline 2024-03-16T12:09:43.060171+0000) 2024-03-16T12:09:59.328 INFO:journalctl@ceph.osd.6.smithi029.stdout:Mar 16 12:09:58 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-6[60123]: 2024-03-16T12:09:58.880+0000 7fb55a3ef700 -1 osd.6 119 heartbeat_check: no reply from 172.21.15.154:6804 osd.2 since back 2024-03-16T12:09:29.962595+0000 front 2024-03-16T12:09:29.962754+0000 (oldest deadline 2024-03-16T12:09:52.862635+0000) 2024-03-16T12:09:59.616 INFO:journalctl@ceph.osd.7.smithi152.stdout:Mar 16 12:09:59 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-7[54789]: 2024-03-16T12:09:59.131+0000 7f6c60c47700 -1 osd.7 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:17.304563+0000 front 2024-03-16T12:09:17.304652+0000 (oldest deadline 2024-03-16T12:09:43.204754+0000) 2024-03-16T12:09:59.617 INFO:journalctl@ceph.osd.7.smithi152.stdout:Mar 16 12:09:59 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-7[54789]: 2024-03-16T12:09:59.131+0000 7f6c60c47700 -1 osd.7 119 heartbeat_check: no reply from 172.21.15.154:6804 osd.2 since back 2024-03-16T12:09:27.607556+0000 front 2024-03-16T12:09:27.607478+0000 (oldest deadline 2024-03-16T12:09:52.307502+0000) 2024-03-16T12:10:00.116 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:09:59 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:09:59.711+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:10.713221+0000 front 2024-03-16T12:09:10.713087+0000 (oldest deadline 2024-03-16T12:09:35.413059+0000) 2024-03-16T12:10:00.117 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:09:59 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:09:59.711+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:20.614694+0000 front 2024-03-16T12:09:20.614671+0000 (oldest deadline 2024-03-16T12:09:45.215535+0000) 2024-03-16T12:10:00.117 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:09:59 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:09:59.711+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.154:6804 osd.2 since back 2024-03-16T12:09:29.916590+0000 front 2024-03-16T12:09:29.916612+0000 (oldest deadline 2024-03-16T12:09:52.816525+0000) 2024-03-16T12:10:00.284 INFO:journalctl@ceph.osd.8.smithi154.stdout:Mar 16 12:10:00 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-8[55054]: 2024-03-16T12:10:00.033+0000 7f3f065c0700 -1 osd.8 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:19.263068+0000 front 2024-03-16T12:09:19.263120+0000 (oldest deadline 2024-03-16T12:09:44.563033+0000) 2024-03-16T12:10:00.285 INFO:journalctl@ceph.osd.8.smithi154.stdout:Mar 16 12:10:00 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-8[55054]: 2024-03-16T12:10:00.033+0000 7f3f065c0700 -1 osd.8 119 heartbeat_check: no reply from 172.21.15.154:6804 osd.2 since back 2024-03-16T12:09:29.566064+0000 front 2024-03-16T12:09:29.566042+0000 (oldest deadline 2024-03-16T12:09:54.266168+0000) 2024-03-16T12:10:00.285 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:09:59 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:09:59.911+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:13.025675+0000 front 2024-03-16T12:09:13.025847+0000 (oldest deadline 2024-03-16T12:09:35.325827+0000) 2024-03-16T12:10:00.285 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:09:59 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:09:59.911+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:19.927745+0000 front 2024-03-16T12:09:19.927761+0000 (oldest deadline 2024-03-16T12:09:45.827828+0000) 2024-03-16T12:10:00.285 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:09:59 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:09:59.911+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.154:6804 osd.2 since back 2024-03-16T12:09:28.629320+0000 front 2024-03-16T12:09:28.629357+0000 (oldest deadline 2024-03-16T12:09:52.129472+0000) 2024-03-16T12:10:00.327 INFO:journalctl@ceph.osd.6.smithi029.stdout:Mar 16 12:09:59 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-6[60123]: 2024-03-16T12:09:59.844+0000 7fb55a3ef700 -1 osd.6 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:11.958752+0000 front 2024-03-16T12:09:11.958869+0000 (oldest deadline 2024-03-16T12:09:36.658956+0000) 2024-03-16T12:10:00.327 INFO:journalctl@ceph.osd.6.smithi029.stdout:Mar 16 12:09:59 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-6[60123]: 2024-03-16T12:09:59.844+0000 7fb55a3ef700 -1 osd.6 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:17.160443+0000 front 2024-03-16T12:09:17.160625+0000 (oldest deadline 2024-03-16T12:09:43.060171+0000) 2024-03-16T12:10:00.327 INFO:journalctl@ceph.osd.6.smithi029.stdout:Mar 16 12:09:59 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-6[60123]: 2024-03-16T12:09:59.844+0000 7fb55a3ef700 -1 osd.6 119 heartbeat_check: no reply from 172.21.15.154:6804 osd.2 since back 2024-03-16T12:09:29.962595+0000 front 2024-03-16T12:09:29.962754+0000 (oldest deadline 2024-03-16T12:09:52.862635+0000) 2024-03-16T12:10:00.327 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 16 12:10:00 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-9[67182]: 2024-03-16T12:10:00.049+0000 7fe1cee8b700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:12.704136+0000 front 2024-03-16T12:09:12.703904+0000 (oldest deadline 2024-03-16T12:09:37.404049+0000) 2024-03-16T12:10:00.327 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 16 12:10:00 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-9[67182]: 2024-03-16T12:10:00.049+0000 7fe1cee8b700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:17.404712+0000 front 2024-03-16T12:09:17.404800+0000 (oldest deadline 2024-03-16T12:09:42.704627+0000) 2024-03-16T12:10:00.328 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 16 12:10:00 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-9[67182]: 2024-03-16T12:10:00.049+0000 7fe1cee8b700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.154:6804 osd.2 since back 2024-03-16T12:09:27.405764+0000 front 2024-03-16T12:09:27.405893+0000 (oldest deadline 2024-03-16T12:09:52.105536+0000) 2024-03-16T12:10:00.616 INFO:journalctl@ceph.osd.7.smithi152.stdout:Mar 16 12:10:00 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-7[54789]: 2024-03-16T12:10:00.153+0000 7f6c60c47700 -1 osd.7 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:17.304563+0000 front 2024-03-16T12:09:17.304652+0000 (oldest deadline 2024-03-16T12:09:43.204754+0000) 2024-03-16T12:10:00.617 INFO:journalctl@ceph.osd.7.smithi152.stdout:Mar 16 12:10:00 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-7[54789]: 2024-03-16T12:10:00.153+0000 7f6c60c47700 -1 osd.7 119 heartbeat_check: no reply from 172.21.15.154:6804 osd.2 since back 2024-03-16T12:09:27.607556+0000 front 2024-03-16T12:09:27.607478+0000 (oldest deadline 2024-03-16T12:09:52.307502+0000) 2024-03-16T12:10:01.116 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:10:00 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:10:00.707+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:10.713221+0000 front 2024-03-16T12:09:10.713087+0000 (oldest deadline 2024-03-16T12:09:35.413059+0000) 2024-03-16T12:10:01.117 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:10:00 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:10:00.707+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:20.614694+0000 front 2024-03-16T12:09:20.614671+0000 (oldest deadline 2024-03-16T12:09:45.215535+0000) 2024-03-16T12:10:01.117 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:10:00 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:10:00.707+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.154:6804 osd.2 since back 2024-03-16T12:09:29.916590+0000 front 2024-03-16T12:09:29.916612+0000 (oldest deadline 2024-03-16T12:09:52.816525+0000) 2024-03-16T12:10:01.284 INFO:journalctl@ceph.osd.8.smithi154.stdout:Mar 16 12:10:01 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-8[55054]: 2024-03-16T12:10:01.063+0000 7f3f065c0700 -1 osd.8 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:19.263068+0000 front 2024-03-16T12:09:19.263120+0000 (oldest deadline 2024-03-16T12:09:44.563033+0000) 2024-03-16T12:10:01.284 INFO:journalctl@ceph.osd.8.smithi154.stdout:Mar 16 12:10:01 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-8[55054]: 2024-03-16T12:10:01.063+0000 7f3f065c0700 -1 osd.8 119 heartbeat_check: no reply from 172.21.15.154:6804 osd.2 since back 2024-03-16T12:09:29.566064+0000 front 2024-03-16T12:09:29.566042+0000 (oldest deadline 2024-03-16T12:09:54.266168+0000) 2024-03-16T12:10:01.285 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:00 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:00.884+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:13.025675+0000 front 2024-03-16T12:09:13.025847+0000 (oldest deadline 2024-03-16T12:09:35.325827+0000) 2024-03-16T12:10:01.285 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:00 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:00.884+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:19.927745+0000 front 2024-03-16T12:09:19.927761+0000 (oldest deadline 2024-03-16T12:09:45.827828+0000) 2024-03-16T12:10:01.285 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:00 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:00.884+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.154:6804 osd.2 since back 2024-03-16T12:09:28.629320+0000 front 2024-03-16T12:09:28.629357+0000 (oldest deadline 2024-03-16T12:09:52.129472+0000) 2024-03-16T12:10:01.327 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 16 12:10:01 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-9[67182]: 2024-03-16T12:10:01.053+0000 7fe1cee8b700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:12.704136+0000 front 2024-03-16T12:09:12.703904+0000 (oldest deadline 2024-03-16T12:09:37.404049+0000) 2024-03-16T12:10:01.327 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 16 12:10:01 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-9[67182]: 2024-03-16T12:10:01.053+0000 7fe1cee8b700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:17.404712+0000 front 2024-03-16T12:09:17.404800+0000 (oldest deadline 2024-03-16T12:09:42.704627+0000) 2024-03-16T12:10:01.327 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 16 12:10:01 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-9[67182]: 2024-03-16T12:10:01.053+0000 7fe1cee8b700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.154:6804 osd.2 since back 2024-03-16T12:09:27.405764+0000 front 2024-03-16T12:09:27.405893+0000 (oldest deadline 2024-03-16T12:09:52.105536+0000) 2024-03-16T12:10:01.327 INFO:journalctl@ceph.osd.6.smithi029.stdout:Mar 16 12:10:00 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-6[60123]: 2024-03-16T12:10:00.845+0000 7fb55a3ef700 -1 osd.6 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:11.958752+0000 front 2024-03-16T12:09:11.958869+0000 (oldest deadline 2024-03-16T12:09:36.658956+0000) 2024-03-16T12:10:01.327 INFO:journalctl@ceph.osd.6.smithi029.stdout:Mar 16 12:10:00 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-6[60123]: 2024-03-16T12:10:00.845+0000 7fb55a3ef700 -1 osd.6 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:17.160443+0000 front 2024-03-16T12:09:17.160625+0000 (oldest deadline 2024-03-16T12:09:43.060171+0000) 2024-03-16T12:10:01.327 INFO:journalctl@ceph.osd.6.smithi029.stdout:Mar 16 12:10:00 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-6[60123]: 2024-03-16T12:10:00.846+0000 7fb55a3ef700 -1 osd.6 119 heartbeat_check: no reply from 172.21.15.154:6804 osd.2 since back 2024-03-16T12:09:29.962595+0000 front 2024-03-16T12:09:29.962754+0000 (oldest deadline 2024-03-16T12:09:52.862635+0000) 2024-03-16T12:10:01.327 INFO:journalctl@ceph.osd.6.smithi029.stdout:Mar 16 12:10:00 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-6[60123]: 2024-03-16T12:10:00.846+0000 7fb55a3ef700 -1 osd.6 119 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-16T12:09:35.163635+0000 front 2024-03-16T12:09:35.163660+0000 (oldest deadline 2024-03-16T12:09:59.863756+0000) 2024-03-16T12:10:01.616 INFO:journalctl@ceph.osd.7.smithi152.stdout:Mar 16 12:10:01 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-7[54789]: 2024-03-16T12:10:01.189+0000 7f6c60c47700 -1 osd.7 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:17.304563+0000 front 2024-03-16T12:09:17.304652+0000 (oldest deadline 2024-03-16T12:09:43.204754+0000) 2024-03-16T12:10:01.616 INFO:journalctl@ceph.osd.7.smithi152.stdout:Mar 16 12:10:01 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-7[54789]: 2024-03-16T12:10:01.189+0000 7f6c60c47700 -1 osd.7 119 heartbeat_check: no reply from 172.21.15.154:6804 osd.2 since back 2024-03-16T12:09:27.607556+0000 front 2024-03-16T12:09:27.607478+0000 (oldest deadline 2024-03-16T12:09:52.307502+0000) 2024-03-16T12:10:01.617 INFO:journalctl@ceph.osd.7.smithi152.stdout:Mar 16 12:10:01 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-7[54789]: 2024-03-16T12:10:01.189+0000 7f6c60c47700 -1 osd.7 119 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-16T12:09:35.808740+0000 front 2024-03-16T12:09:35.808611+0000 (oldest deadline 2024-03-16T12:10:00.508533+0000) 2024-03-16T12:10:02.077 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 16 12:10:02 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-9[67182]: 2024-03-16T12:10:02.014+0000 7fe1cee8b700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:12.704136+0000 front 2024-03-16T12:09:12.703904+0000 (oldest deadline 2024-03-16T12:09:37.404049+0000) 2024-03-16T12:10:02.077 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 16 12:10:02 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-9[67182]: 2024-03-16T12:10:02.014+0000 7fe1cee8b700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:17.404712+0000 front 2024-03-16T12:09:17.404800+0000 (oldest deadline 2024-03-16T12:09:42.704627+0000) 2024-03-16T12:10:02.077 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 16 12:10:02 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-9[67182]: 2024-03-16T12:10:02.014+0000 7fe1cee8b700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.154:6804 osd.2 since back 2024-03-16T12:09:27.405764+0000 front 2024-03-16T12:09:27.405893+0000 (oldest deadline 2024-03-16T12:09:52.105536+0000) 2024-03-16T12:10:02.078 INFO:journalctl@ceph.osd.6.smithi029.stdout:Mar 16 12:10:01 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-6[60123]: 2024-03-16T12:10:01.813+0000 7fb55a3ef700 -1 osd.6 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:11.958752+0000 front 2024-03-16T12:09:11.958869+0000 (oldest deadline 2024-03-16T12:09:36.658956+0000) 2024-03-16T12:10:02.078 INFO:journalctl@ceph.osd.6.smithi029.stdout:Mar 16 12:10:01 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-6[60123]: 2024-03-16T12:10:01.813+0000 7fb55a3ef700 -1 osd.6 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:17.160443+0000 front 2024-03-16T12:09:17.160625+0000 (oldest deadline 2024-03-16T12:09:43.060171+0000) 2024-03-16T12:10:02.078 INFO:journalctl@ceph.osd.6.smithi029.stdout:Mar 16 12:10:01 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-6[60123]: 2024-03-16T12:10:01.813+0000 7fb55a3ef700 -1 osd.6 119 heartbeat_check: no reply from 172.21.15.154:6804 osd.2 since back 2024-03-16T12:09:29.962595+0000 front 2024-03-16T12:09:29.962754+0000 (oldest deadline 2024-03-16T12:09:52.862635+0000) 2024-03-16T12:10:02.078 INFO:journalctl@ceph.osd.6.smithi029.stdout:Mar 16 12:10:01 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-6[60123]: 2024-03-16T12:10:01.813+0000 7fb55a3ef700 -1 osd.6 119 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-16T12:09:35.163635+0000 front 2024-03-16T12:09:35.163660+0000 (oldest deadline 2024-03-16T12:09:59.863756+0000) 2024-03-16T12:10:02.116 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:10:01 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:10:01.702+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:10.713221+0000 front 2024-03-16T12:09:10.713087+0000 (oldest deadline 2024-03-16T12:09:35.413059+0000) 2024-03-16T12:10:02.116 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:10:01 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:10:01.702+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:20.614694+0000 front 2024-03-16T12:09:20.614671+0000 (oldest deadline 2024-03-16T12:09:45.215535+0000) 2024-03-16T12:10:02.117 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:10:01 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:10:01.702+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.154:6804 osd.2 since back 2024-03-16T12:09:29.916590+0000 front 2024-03-16T12:09:29.916612+0000 (oldest deadline 2024-03-16T12:09:52.816525+0000) 2024-03-16T12:10:02.284 INFO:journalctl@ceph.osd.8.smithi154.stdout:Mar 16 12:10:02 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-8[55054]: 2024-03-16T12:10:02.081+0000 7f3f065c0700 -1 osd.8 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:19.263068+0000 front 2024-03-16T12:09:19.263120+0000 (oldest deadline 2024-03-16T12:09:44.563033+0000) 2024-03-16T12:10:02.284 INFO:journalctl@ceph.osd.8.smithi154.stdout:Mar 16 12:10:02 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-8[55054]: 2024-03-16T12:10:02.081+0000 7f3f065c0700 -1 osd.8 119 heartbeat_check: no reply from 172.21.15.154:6804 osd.2 since back 2024-03-16T12:09:29.566064+0000 front 2024-03-16T12:09:29.566042+0000 (oldest deadline 2024-03-16T12:09:54.266168+0000) 2024-03-16T12:10:02.284 INFO:journalctl@ceph.osd.8.smithi154.stdout:Mar 16 12:10:02 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-8[55054]: 2024-03-16T12:10:02.081+0000 7f3f065c0700 -1 osd.8 119 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-16T12:09:38.367565+0000 front 2024-03-16T12:09:38.367474+0000 (oldest deadline 2024-03-16T12:10:01.867407+0000) 2024-03-16T12:10:02.285 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:01 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:01.895+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:13.025675+0000 front 2024-03-16T12:09:13.025847+0000 (oldest deadline 2024-03-16T12:09:35.325827+0000) 2024-03-16T12:10:02.285 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:01 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:01.895+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:19.927745+0000 front 2024-03-16T12:09:19.927761+0000 (oldest deadline 2024-03-16T12:09:45.827828+0000) 2024-03-16T12:10:02.285 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:01 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:01.895+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.154:6804 osd.2 since back 2024-03-16T12:09:28.629320+0000 front 2024-03-16T12:09:28.629357+0000 (oldest deadline 2024-03-16T12:09:52.129472+0000) 2024-03-16T12:10:02.285 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:01 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:01.895+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-16T12:09:38.432099+0000 front 2024-03-16T12:09:38.432072+0000 (oldest deadline 2024-03-16T12:10:01.332006+0000) 2024-03-16T12:10:02.616 INFO:journalctl@ceph.osd.7.smithi152.stdout:Mar 16 12:10:02 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-7[54789]: 2024-03-16T12:10:02.230+0000 7f6c60c47700 -1 osd.7 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:17.304563+0000 front 2024-03-16T12:09:17.304652+0000 (oldest deadline 2024-03-16T12:09:43.204754+0000) 2024-03-16T12:10:02.617 INFO:journalctl@ceph.osd.7.smithi152.stdout:Mar 16 12:10:02 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-7[54789]: 2024-03-16T12:10:02.230+0000 7f6c60c47700 -1 osd.7 119 heartbeat_check: no reply from 172.21.15.154:6804 osd.2 since back 2024-03-16T12:09:27.607556+0000 front 2024-03-16T12:09:27.607478+0000 (oldest deadline 2024-03-16T12:09:52.307502+0000) 2024-03-16T12:10:02.617 INFO:journalctl@ceph.osd.7.smithi152.stdout:Mar 16 12:10:02 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-7[54789]: 2024-03-16T12:10:02.230+0000 7f6c60c47700 -1 osd.7 119 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-16T12:09:35.808740+0000 front 2024-03-16T12:09:35.808611+0000 (oldest deadline 2024-03-16T12:10:00.508533+0000) 2024-03-16T12:10:02.969 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:10:02 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:10:02.731+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:10.713221+0000 front 2024-03-16T12:09:10.713087+0000 (oldest deadline 2024-03-16T12:09:35.413059+0000) 2024-03-16T12:10:02.969 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:10:02 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:10:02.731+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:20.614694+0000 front 2024-03-16T12:09:20.614671+0000 (oldest deadline 2024-03-16T12:09:45.215535+0000) 2024-03-16T12:10:02.970 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:10:02 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:10:02.731+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.154:6804 osd.2 since back 2024-03-16T12:09:29.916590+0000 front 2024-03-16T12:09:29.916612+0000 (oldest deadline 2024-03-16T12:09:52.816525+0000) 2024-03-16T12:10:03.284 INFO:journalctl@ceph.osd.8.smithi154.stdout:Mar 16 12:10:03 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-8[55054]: 2024-03-16T12:10:03.106+0000 7f3f065c0700 -1 osd.8 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:19.263068+0000 front 2024-03-16T12:09:19.263120+0000 (oldest deadline 2024-03-16T12:09:44.563033+0000) 2024-03-16T12:10:03.284 INFO:journalctl@ceph.osd.8.smithi154.stdout:Mar 16 12:10:03 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-8[55054]: 2024-03-16T12:10:03.106+0000 7f3f065c0700 -1 osd.8 119 heartbeat_check: no reply from 172.21.15.154:6804 osd.2 since back 2024-03-16T12:09:29.566064+0000 front 2024-03-16T12:09:29.566042+0000 (oldest deadline 2024-03-16T12:09:54.266168+0000) 2024-03-16T12:10:03.284 INFO:journalctl@ceph.osd.8.smithi154.stdout:Mar 16 12:10:03 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-8[55054]: 2024-03-16T12:10:03.106+0000 7f3f065c0700 -1 osd.8 119 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-16T12:09:38.367565+0000 front 2024-03-16T12:09:38.367474+0000 (oldest deadline 2024-03-16T12:10:01.867407+0000) 2024-03-16T12:10:03.285 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:02 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:02.924+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:13.025675+0000 front 2024-03-16T12:09:13.025847+0000 (oldest deadline 2024-03-16T12:09:35.325827+0000) 2024-03-16T12:10:03.285 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:02 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:02.925+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:19.927745+0000 front 2024-03-16T12:09:19.927761+0000 (oldest deadline 2024-03-16T12:09:45.827828+0000) 2024-03-16T12:10:03.285 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:02 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:02.925+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.154:6804 osd.2 since back 2024-03-16T12:09:28.629320+0000 front 2024-03-16T12:09:28.629357+0000 (oldest deadline 2024-03-16T12:09:52.129472+0000) 2024-03-16T12:10:03.285 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:02 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:02.925+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-16T12:09:38.432099+0000 front 2024-03-16T12:09:38.432072+0000 (oldest deadline 2024-03-16T12:10:01.332006+0000) 2024-03-16T12:10:03.327 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 16 12:10:03 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-9[67182]: 2024-03-16T12:10:03.041+0000 7fe1cee8b700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:12.704136+0000 front 2024-03-16T12:09:12.703904+0000 (oldest deadline 2024-03-16T12:09:37.404049+0000) 2024-03-16T12:10:03.327 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 16 12:10:03 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-9[67182]: 2024-03-16T12:10:03.041+0000 7fe1cee8b700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:17.404712+0000 front 2024-03-16T12:09:17.404800+0000 (oldest deadline 2024-03-16T12:09:42.704627+0000) 2024-03-16T12:10:03.327 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 16 12:10:03 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-9[67182]: 2024-03-16T12:10:03.041+0000 7fe1cee8b700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.154:6804 osd.2 since back 2024-03-16T12:09:27.405764+0000 front 2024-03-16T12:09:27.405893+0000 (oldest deadline 2024-03-16T12:09:52.105536+0000) 2024-03-16T12:10:03.328 INFO:journalctl@ceph.osd.6.smithi029.stdout:Mar 16 12:10:02 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-6[60123]: 2024-03-16T12:10:02.853+0000 7fb55a3ef700 -1 osd.6 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:11.958752+0000 front 2024-03-16T12:09:11.958869+0000 (oldest deadline 2024-03-16T12:09:36.658956+0000) 2024-03-16T12:10:03.328 INFO:journalctl@ceph.osd.6.smithi029.stdout:Mar 16 12:10:02 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-6[60123]: 2024-03-16T12:10:02.853+0000 7fb55a3ef700 -1 osd.6 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:17.160443+0000 front 2024-03-16T12:09:17.160625+0000 (oldest deadline 2024-03-16T12:09:43.060171+0000) 2024-03-16T12:10:03.328 INFO:journalctl@ceph.osd.6.smithi029.stdout:Mar 16 12:10:02 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-6[60123]: 2024-03-16T12:10:02.853+0000 7fb55a3ef700 -1 osd.6 119 heartbeat_check: no reply from 172.21.15.154:6804 osd.2 since back 2024-03-16T12:09:29.962595+0000 front 2024-03-16T12:09:29.962754+0000 (oldest deadline 2024-03-16T12:09:52.862635+0000) 2024-03-16T12:10:03.328 INFO:journalctl@ceph.osd.6.smithi029.stdout:Mar 16 12:10:02 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-6[60123]: 2024-03-16T12:10:02.853+0000 7fb55a3ef700 -1 osd.6 119 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-16T12:09:35.163635+0000 front 2024-03-16T12:09:35.163660+0000 (oldest deadline 2024-03-16T12:09:59.863756+0000) 2024-03-16T12:10:03.366 INFO:journalctl@ceph.osd.7.smithi152.stdout:Mar 16 12:10:03 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-7[54789]: 2024-03-16T12:10:03.227+0000 7f6c60c47700 -1 osd.7 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:17.304563+0000 front 2024-03-16T12:09:17.304652+0000 (oldest deadline 2024-03-16T12:09:43.204754+0000) 2024-03-16T12:10:03.367 INFO:journalctl@ceph.osd.7.smithi152.stdout:Mar 16 12:10:03 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-7[54789]: 2024-03-16T12:10:03.227+0000 7f6c60c47700 -1 osd.7 119 heartbeat_check: no reply from 172.21.15.154:6804 osd.2 since back 2024-03-16T12:09:27.607556+0000 front 2024-03-16T12:09:27.607478+0000 (oldest deadline 2024-03-16T12:09:52.307502+0000) 2024-03-16T12:10:03.367 INFO:journalctl@ceph.osd.7.smithi152.stdout:Mar 16 12:10:03 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-7[54789]: 2024-03-16T12:10:03.227+0000 7f6c60c47700 -1 osd.7 119 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-16T12:09:35.808740+0000 front 2024-03-16T12:09:35.808611+0000 (oldest deadline 2024-03-16T12:10:00.508533+0000) 2024-03-16T12:10:03.921 INFO:journalctl@ceph.osd.6.smithi029.stdout:Mar 16 12:10:03 smithi029 podman[95491]: 2024-03-16 12:10:03.66294271 +0000 UTC m=+5.680232533 container died 3a976a17c65d7ac0693a2a81d80fa54f0e2a2bc87e7105e979ff3e42f91af08c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-6, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-16T12:10:04.117 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:10:03 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:10:03.714+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:10.713221+0000 front 2024-03-16T12:09:10.713087+0000 (oldest deadline 2024-03-16T12:09:35.413059+0000) 2024-03-16T12:10:04.117 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:10:03 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:10:03.714+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:20.614694+0000 front 2024-03-16T12:09:20.614671+0000 (oldest deadline 2024-03-16T12:09:45.215535+0000) 2024-03-16T12:10:04.117 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:10:03 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:10:03.714+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.154:6804 osd.2 since back 2024-03-16T12:09:29.916590+0000 front 2024-03-16T12:09:29.916612+0000 (oldest deadline 2024-03-16T12:09:52.816525+0000) 2024-03-16T12:10:04.117 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:10:03 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:10:03.714+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-16T12:09:37.517617+0000 front 2024-03-16T12:09:37.517557+0000 (oldest deadline 2024-03-16T12:10:02.817580+0000) 2024-03-16T12:10:04.284 INFO:journalctl@ceph.osd.8.smithi154.stdout:Mar 16 12:10:04 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-8[55054]: 2024-03-16T12:10:04.147+0000 7f3f065c0700 -1 osd.8 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:19.263068+0000 front 2024-03-16T12:09:19.263120+0000 (oldest deadline 2024-03-16T12:09:44.563033+0000) 2024-03-16T12:10:04.284 INFO:journalctl@ceph.osd.8.smithi154.stdout:Mar 16 12:10:04 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-8[55054]: 2024-03-16T12:10:04.147+0000 7f3f065c0700 -1 osd.8 119 heartbeat_check: no reply from 172.21.15.154:6804 osd.2 since back 2024-03-16T12:09:29.566064+0000 front 2024-03-16T12:09:29.566042+0000 (oldest deadline 2024-03-16T12:09:54.266168+0000) 2024-03-16T12:10:04.285 INFO:journalctl@ceph.osd.8.smithi154.stdout:Mar 16 12:10:04 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-8[55054]: 2024-03-16T12:10:04.147+0000 7f3f065c0700 -1 osd.8 119 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-16T12:09:38.367565+0000 front 2024-03-16T12:09:38.367474+0000 (oldest deadline 2024-03-16T12:10:01.867407+0000) 2024-03-16T12:10:04.285 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:03 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:03.957+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:13.025675+0000 front 2024-03-16T12:09:13.025847+0000 (oldest deadline 2024-03-16T12:09:35.325827+0000) 2024-03-16T12:10:04.285 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:03 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:03.957+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:19.927745+0000 front 2024-03-16T12:09:19.927761+0000 (oldest deadline 2024-03-16T12:09:45.827828+0000) 2024-03-16T12:10:04.285 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:03 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:03.957+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.154:6804 osd.2 since back 2024-03-16T12:09:28.629320+0000 front 2024-03-16T12:09:28.629357+0000 (oldest deadline 2024-03-16T12:09:52.129472+0000) 2024-03-16T12:10:04.285 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:03 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:03.957+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-16T12:09:38.432099+0000 front 2024-03-16T12:09:38.432072+0000 (oldest deadline 2024-03-16T12:10:01.332006+0000) 2024-03-16T12:10:04.299 INFO:journalctl@ceph.osd.6.smithi029.stdout:Mar 16 12:10:04 smithi029 podman[95491]: 2024-03-16 12:10:04.034105503 +0000 UTC m=+6.051395323 container cleanup 3a976a17c65d7ac0693a2a81d80fa54f0e2a2bc87e7105e979ff3e42f91af08c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-6, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-16T12:10:04.300 INFO:journalctl@ceph.osd.6.smithi029.stdout:Mar 16 12:10:04 smithi029 bash[95491]: ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-6 2024-03-16T12:10:04.300 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 16 12:10:04 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-9[67182]: 2024-03-16T12:10:04.032+0000 7fe1cee8b700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:12.704136+0000 front 2024-03-16T12:09:12.703904+0000 (oldest deadline 2024-03-16T12:09:37.404049+0000) 2024-03-16T12:10:04.300 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 16 12:10:04 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-9[67182]: 2024-03-16T12:10:04.032+0000 7fe1cee8b700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:17.404712+0000 front 2024-03-16T12:09:17.404800+0000 (oldest deadline 2024-03-16T12:09:42.704627+0000) 2024-03-16T12:10:04.300 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 16 12:10:04 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-9[67182]: 2024-03-16T12:10:04.032+0000 7fe1cee8b700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.154:6804 osd.2 since back 2024-03-16T12:09:27.405764+0000 front 2024-03-16T12:09:27.405893+0000 (oldest deadline 2024-03-16T12:09:52.105536+0000) 2024-03-16T12:10:04.300 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 16 12:10:04 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-9[67182]: 2024-03-16T12:10:04.032+0000 7fe1cee8b700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-16T12:09:37.907196+0000 front 2024-03-16T12:09:37.907286+0000 (oldest deadline 2024-03-16T12:10:03.207291+0000) 2024-03-16T12:10:04.577 INFO:journalctl@ceph.osd.6.smithi029.stdout:Mar 16 12:10:04 smithi029 podman[95510]: 2024-03-16 12:10:04.311703954 +0000 UTC m=+0.652379165 container remove 3a976a17c65d7ac0693a2a81d80fa54f0e2a2bc87e7105e979ff3e42f91af08c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-6, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, io.buildah.version=1.33.5) 2024-03-16T12:10:04.616 INFO:journalctl@ceph.osd.7.smithi152.stdout:Mar 16 12:10:04 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-7[54789]: 2024-03-16T12:10:04.226+0000 7f6c60c47700 -1 osd.7 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:17.304563+0000 front 2024-03-16T12:09:17.304652+0000 (oldest deadline 2024-03-16T12:09:43.204754+0000) 2024-03-16T12:10:04.616 INFO:journalctl@ceph.osd.7.smithi152.stdout:Mar 16 12:10:04 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-7[54789]: 2024-03-16T12:10:04.226+0000 7f6c60c47700 -1 osd.7 119 heartbeat_check: no reply from 172.21.15.154:6804 osd.2 since back 2024-03-16T12:09:27.607556+0000 front 2024-03-16T12:09:27.607478+0000 (oldest deadline 2024-03-16T12:09:52.307502+0000) 2024-03-16T12:10:04.617 INFO:journalctl@ceph.osd.7.smithi152.stdout:Mar 16 12:10:04 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-7[54789]: 2024-03-16T12:10:04.226+0000 7f6c60c47700 -1 osd.7 119 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-16T12:09:35.808740+0000 front 2024-03-16T12:09:35.808611+0000 (oldest deadline 2024-03-16T12:10:00.508533+0000) 2024-03-16T12:10:04.923 INFO:journalctl@ceph.osd.6.smithi029.stdout:Mar 16 12:10:04 smithi029 podman[95614]: 2024-03-16 12:10:04.58394741 +0000 UTC m=+0.022918498 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-16T12:10:04.923 INFO:journalctl@ceph.osd.6.smithi029.stdout:Mar 16 12:10:04 smithi029 podman[95614]: 2024-03-16 12:10:04.71523383 +0000 UTC m=+0.154204910 container create be4db6dbe981c1cce90ea97ba9fb31324196d4ab7a14a33bd341cd2854b86f9d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-6-deactivate, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-16T12:10:04.923 INFO:journalctl@ceph.osd.6.smithi029.stdout:Mar 16 12:10:04 smithi029 podman[95614]: 2024-03-16 12:10:04.935788528 +0000 UTC m=+0.374759608 container init be4db6dbe981c1cce90ea97ba9fb31324196d4ab7a14a33bd341cd2854b86f9d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-6-deactivate, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-03-16T12:10:05.116 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:10:04 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:10:04.707+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:10.713221+0000 front 2024-03-16T12:09:10.713087+0000 (oldest deadline 2024-03-16T12:09:35.413059+0000) 2024-03-16T12:10:05.117 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:10:04 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:10:04.707+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:20.614694+0000 front 2024-03-16T12:09:20.614671+0000 (oldest deadline 2024-03-16T12:09:45.215535+0000) 2024-03-16T12:10:05.117 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:10:04 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:10:04.707+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.154:6804 osd.2 since back 2024-03-16T12:09:29.916590+0000 front 2024-03-16T12:09:29.916612+0000 (oldest deadline 2024-03-16T12:09:52.816525+0000) 2024-03-16T12:10:05.117 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:10:04 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:10:04.707+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-16T12:09:37.517617+0000 front 2024-03-16T12:09:37.517557+0000 (oldest deadline 2024-03-16T12:10:02.817580+0000) 2024-03-16T12:10:05.158 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:04 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:04.917+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:13.025675+0000 front 2024-03-16T12:09:13.025847+0000 (oldest deadline 2024-03-16T12:09:35.325827+0000) 2024-03-16T12:10:05.159 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:04 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:04.917+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:19.927745+0000 front 2024-03-16T12:09:19.927761+0000 (oldest deadline 2024-03-16T12:09:45.827828+0000) 2024-03-16T12:10:05.159 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:04 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:04.917+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.154:6804 osd.2 since back 2024-03-16T12:09:28.629320+0000 front 2024-03-16T12:09:28.629357+0000 (oldest deadline 2024-03-16T12:09:52.129472+0000) 2024-03-16T12:10:05.159 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:04 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:04.917+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-16T12:09:38.432099+0000 front 2024-03-16T12:09:38.432072+0000 (oldest deadline 2024-03-16T12:10:01.332006+0000) 2024-03-16T12:10:05.233 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 16 12:10:05 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-9[67182]: 2024-03-16T12:10:05.063+0000 7fe1cee8b700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:12.704136+0000 front 2024-03-16T12:09:12.703904+0000 (oldest deadline 2024-03-16T12:09:37.404049+0000) 2024-03-16T12:10:05.233 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 16 12:10:05 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-9[67182]: 2024-03-16T12:10:05.063+0000 7fe1cee8b700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:17.404712+0000 front 2024-03-16T12:09:17.404800+0000 (oldest deadline 2024-03-16T12:09:42.704627+0000) 2024-03-16T12:10:05.233 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 16 12:10:05 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-9[67182]: 2024-03-16T12:10:05.063+0000 7fe1cee8b700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.154:6804 osd.2 since back 2024-03-16T12:09:27.405764+0000 front 2024-03-16T12:09:27.405893+0000 (oldest deadline 2024-03-16T12:09:52.105536+0000) 2024-03-16T12:10:05.233 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 16 12:10:05 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-9[67182]: 2024-03-16T12:10:05.063+0000 7fe1cee8b700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-16T12:09:37.907196+0000 front 2024-03-16T12:09:37.907286+0000 (oldest deadline 2024-03-16T12:10:03.207291+0000) 2024-03-16T12:10:05.233 INFO:journalctl@ceph.osd.6.smithi029.stdout:Mar 16 12:10:04 smithi029 podman[95614]: 2024-03-16 12:10:04.93889304 +0000 UTC m=+0.377864118 container start be4db6dbe981c1cce90ea97ba9fb31324196d4ab7a14a33bd341cd2854b86f9d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-6-deactivate, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307) 2024-03-16T12:10:05.234 INFO:journalctl@ceph.osd.6.smithi029.stdout:Mar 16 12:10:05 smithi029 podman[95614]: 2024-03-16 12:10:05.053165397 +0000 UTC m=+0.492136507 container attach be4db6dbe981c1cce90ea97ba9fb31324196d4ab7a14a33bd341cd2854b86f9d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-6-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-03-16T12:10:05.528 INFO:journalctl@ceph.osd.6.smithi029.stdout:Mar 16 12:10:05 smithi029 podman[95614]: 2024-03-16 12:10:05.245538116 +0000 UTC m=+0.684509205 container died be4db6dbe981c1cce90ea97ba9fb31324196d4ab7a14a33bd341cd2854b86f9d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-6-deactivate, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-03-16T12:10:05.534 INFO:journalctl@ceph.osd.8.smithi154.stdout:Mar 16 12:10:05 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-8[55054]: 2024-03-16T12:10:05.169+0000 7f3f065c0700 -1 osd.8 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:19.263068+0000 front 2024-03-16T12:09:19.263120+0000 (oldest deadline 2024-03-16T12:09:44.563033+0000) 2024-03-16T12:10:05.534 INFO:journalctl@ceph.osd.8.smithi154.stdout:Mar 16 12:10:05 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-8[55054]: 2024-03-16T12:10:05.169+0000 7f3f065c0700 -1 osd.8 119 heartbeat_check: no reply from 172.21.15.154:6804 osd.2 since back 2024-03-16T12:09:29.566064+0000 front 2024-03-16T12:09:29.566042+0000 (oldest deadline 2024-03-16T12:09:54.266168+0000) 2024-03-16T12:10:05.534 INFO:journalctl@ceph.osd.8.smithi154.stdout:Mar 16 12:10:05 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-8[55054]: 2024-03-16T12:10:05.169+0000 7f3f065c0700 -1 osd.8 119 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-16T12:09:38.367565+0000 front 2024-03-16T12:09:38.367474+0000 (oldest deadline 2024-03-16T12:10:01.867407+0000) 2024-03-16T12:10:05.616 INFO:journalctl@ceph.osd.7.smithi152.stdout:Mar 16 12:10:05 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-7[54789]: 2024-03-16T12:10:05.239+0000 7f6c60c47700 -1 osd.7 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:17.304563+0000 front 2024-03-16T12:09:17.304652+0000 (oldest deadline 2024-03-16T12:09:43.204754+0000) 2024-03-16T12:10:05.616 INFO:journalctl@ceph.osd.7.smithi152.stdout:Mar 16 12:10:05 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-7[54789]: 2024-03-16T12:10:05.239+0000 7f6c60c47700 -1 osd.7 119 heartbeat_check: no reply from 172.21.15.154:6804 osd.2 since back 2024-03-16T12:09:27.607556+0000 front 2024-03-16T12:09:27.607478+0000 (oldest deadline 2024-03-16T12:09:52.307502+0000) 2024-03-16T12:10:05.617 INFO:journalctl@ceph.osd.7.smithi152.stdout:Mar 16 12:10:05 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-7[54789]: 2024-03-16T12:10:05.239+0000 7f6c60c47700 -1 osd.7 119 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-16T12:09:35.808740+0000 front 2024-03-16T12:09:35.808611+0000 (oldest deadline 2024-03-16T12:10:00.508533+0000) 2024-03-16T12:10:05.880 DEBUG:teuthology.orchestra.run.smithi029:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d1887f28-e389-11ee-95c9-87774f69a715@osd.6.service' 2024-03-16T12:10:05.924 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-16T12:10:05.924 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-03-16T12:10:05.924 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-03-16T12:10:05.924 DEBUG:teuthology.orchestra.run.smithi152:> sudo systemctl stop ceph-d1887f28-e389-11ee-95c9-87774f69a715@osd.7 2024-03-16T12:10:05.966 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:10:05 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:10:05.726+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:10.713221+0000 front 2024-03-16T12:09:10.713087+0000 (oldest deadline 2024-03-16T12:09:35.413059+0000) 2024-03-16T12:10:05.966 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:10:05 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:10:05.726+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:20.614694+0000 front 2024-03-16T12:09:20.614671+0000 (oldest deadline 2024-03-16T12:09:45.215535+0000) 2024-03-16T12:10:05.966 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:10:05 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:10:05.726+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.154:6804 osd.2 since back 2024-03-16T12:09:29.916590+0000 front 2024-03-16T12:09:29.916612+0000 (oldest deadline 2024-03-16T12:09:52.816525+0000) 2024-03-16T12:10:05.967 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:10:05 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:10:05.726+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-16T12:09:37.517617+0000 front 2024-03-16T12:09:37.517557+0000 (oldest deadline 2024-03-16T12:10:02.817580+0000) 2024-03-16T12:10:06.200 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:05 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:05.895+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:13.025675+0000 front 2024-03-16T12:09:13.025847+0000 (oldest deadline 2024-03-16T12:09:35.325827+0000) 2024-03-16T12:10:06.200 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:05 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:05.895+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:19.927745+0000 front 2024-03-16T12:09:19.927761+0000 (oldest deadline 2024-03-16T12:09:45.827828+0000) 2024-03-16T12:10:06.200 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:05 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:05.895+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.154:6804 osd.2 since back 2024-03-16T12:09:28.629320+0000 front 2024-03-16T12:09:28.629357+0000 (oldest deadline 2024-03-16T12:09:52.129472+0000) 2024-03-16T12:10:06.200 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:05 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:05.895+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-16T12:09:38.432099+0000 front 2024-03-16T12:09:38.432072+0000 (oldest deadline 2024-03-16T12:10:01.332006+0000) 2024-03-16T12:10:06.225 INFO:journalctl@ceph.osd.7.smithi152.stdout:Mar 16 12:10:05 smithi152 systemd[1]: Stopping Ceph osd.7 for d1887f28-e389-11ee-95c9-87774f69a715... 2024-03-16T12:10:06.327 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 16 12:10:06 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-9[67182]: 2024-03-16T12:10:06.054+0000 7fe1cee8b700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:12.704136+0000 front 2024-03-16T12:09:12.703904+0000 (oldest deadline 2024-03-16T12:09:37.404049+0000) 2024-03-16T12:10:06.327 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 16 12:10:06 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-9[67182]: 2024-03-16T12:10:06.054+0000 7fe1cee8b700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:17.404712+0000 front 2024-03-16T12:09:17.404800+0000 (oldest deadline 2024-03-16T12:09:42.704627+0000) 2024-03-16T12:10:06.327 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 16 12:10:06 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-9[67182]: 2024-03-16T12:10:06.054+0000 7fe1cee8b700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.154:6804 osd.2 since back 2024-03-16T12:09:27.405764+0000 front 2024-03-16T12:09:27.405893+0000 (oldest deadline 2024-03-16T12:09:52.105536+0000) 2024-03-16T12:10:06.327 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 16 12:10:06 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-9[67182]: 2024-03-16T12:10:06.054+0000 7fe1cee8b700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-16T12:09:37.907196+0000 front 2024-03-16T12:09:37.907286+0000 (oldest deadline 2024-03-16T12:10:03.207291+0000) 2024-03-16T12:10:06.534 INFO:journalctl@ceph.osd.8.smithi154.stdout:Mar 16 12:10:06 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-8[55054]: 2024-03-16T12:10:06.211+0000 7f3f065c0700 -1 osd.8 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:19.263068+0000 front 2024-03-16T12:09:19.263120+0000 (oldest deadline 2024-03-16T12:09:44.563033+0000) 2024-03-16T12:10:06.534 INFO:journalctl@ceph.osd.8.smithi154.stdout:Mar 16 12:10:06 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-8[55054]: 2024-03-16T12:10:06.211+0000 7f3f065c0700 -1 osd.8 119 heartbeat_check: no reply from 172.21.15.154:6804 osd.2 since back 2024-03-16T12:09:29.566064+0000 front 2024-03-16T12:09:29.566042+0000 (oldest deadline 2024-03-16T12:09:54.266168+0000) 2024-03-16T12:10:06.535 INFO:journalctl@ceph.osd.8.smithi154.stdout:Mar 16 12:10:06 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-8[55054]: 2024-03-16T12:10:06.211+0000 7f3f065c0700 -1 osd.8 119 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-16T12:09:38.367565+0000 front 2024-03-16T12:09:38.367474+0000 (oldest deadline 2024-03-16T12:10:01.867407+0000) 2024-03-16T12:10:06.617 INFO:journalctl@ceph.osd.7.smithi152.stdout:Mar 16 12:10:06 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-7[54789]: 2024-03-16T12:10:06.236+0000 7f6c60c47700 -1 osd.7 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:17.304563+0000 front 2024-03-16T12:09:17.304652+0000 (oldest deadline 2024-03-16T12:09:43.204754+0000) 2024-03-16T12:10:06.617 INFO:journalctl@ceph.osd.7.smithi152.stdout:Mar 16 12:10:06 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-7[54789]: 2024-03-16T12:10:06.236+0000 7f6c60c47700 -1 osd.7 119 heartbeat_check: no reply from 172.21.15.154:6804 osd.2 since back 2024-03-16T12:09:27.607556+0000 front 2024-03-16T12:09:27.607478+0000 (oldest deadline 2024-03-16T12:09:52.307502+0000) 2024-03-16T12:10:06.617 INFO:journalctl@ceph.osd.7.smithi152.stdout:Mar 16 12:10:06 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-7[54789]: 2024-03-16T12:10:06.236+0000 7f6c60c47700 -1 osd.7 119 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-16T12:09:35.808740+0000 front 2024-03-16T12:09:35.808611+0000 (oldest deadline 2024-03-16T12:10:00.508533+0000) 2024-03-16T12:10:06.618 INFO:journalctl@ceph.osd.7.smithi152.stdout:Mar 16 12:10:06 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-7[54789]: 2024-03-16T12:10:06.429+0000 7f6c64653700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-16T12:10:06.618 INFO:journalctl@ceph.osd.7.smithi152.stdout:Mar 16 12:10:06 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-7[54789]: 2024-03-16T12:10:06.429+0000 7f6c64653700 -1 osd.7 119 *** Got signal Terminated *** 2024-03-16T12:10:06.618 INFO:journalctl@ceph.osd.7.smithi152.stdout:Mar 16 12:10:06 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-7[54789]: 2024-03-16T12:10:06.429+0000 7f6c64653700 -1 osd.7 119 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-16T12:10:07.116 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:10:06 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:10:06.761+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:10.713221+0000 front 2024-03-16T12:09:10.713087+0000 (oldest deadline 2024-03-16T12:09:35.413059+0000) 2024-03-16T12:10:07.117 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:10:06 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:10:06.761+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:20.614694+0000 front 2024-03-16T12:09:20.614671+0000 (oldest deadline 2024-03-16T12:09:45.215535+0000) 2024-03-16T12:10:07.117 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:10:06 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:10:06.761+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.154:6804 osd.2 since back 2024-03-16T12:09:29.916590+0000 front 2024-03-16T12:09:29.916612+0000 (oldest deadline 2024-03-16T12:09:52.816525+0000) 2024-03-16T12:10:07.117 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:10:06 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:10:06.761+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-16T12:09:37.517617+0000 front 2024-03-16T12:09:37.517557+0000 (oldest deadline 2024-03-16T12:10:02.817580+0000) 2024-03-16T12:10:07.223 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:06 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:06.902+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:13.025675+0000 front 2024-03-16T12:09:13.025847+0000 (oldest deadline 2024-03-16T12:09:35.325827+0000) 2024-03-16T12:10:07.223 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:06 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:06.902+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:19.927745+0000 front 2024-03-16T12:09:19.927761+0000 (oldest deadline 2024-03-16T12:09:45.827828+0000) 2024-03-16T12:10:07.224 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:06 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:06.902+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.154:6804 osd.2 since back 2024-03-16T12:09:28.629320+0000 front 2024-03-16T12:09:28.629357+0000 (oldest deadline 2024-03-16T12:09:52.129472+0000) 2024-03-16T12:10:07.224 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:06 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:06.902+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-16T12:09:38.432099+0000 front 2024-03-16T12:09:38.432072+0000 (oldest deadline 2024-03-16T12:10:01.332006+0000) 2024-03-16T12:10:07.327 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 16 12:10:07 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-9[67182]: 2024-03-16T12:10:07.036+0000 7fe1cee8b700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:12.704136+0000 front 2024-03-16T12:09:12.703904+0000 (oldest deadline 2024-03-16T12:09:37.404049+0000) 2024-03-16T12:10:07.327 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 16 12:10:07 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-9[67182]: 2024-03-16T12:10:07.036+0000 7fe1cee8b700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:17.404712+0000 front 2024-03-16T12:09:17.404800+0000 (oldest deadline 2024-03-16T12:09:42.704627+0000) 2024-03-16T12:10:07.327 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 16 12:10:07 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-9[67182]: 2024-03-16T12:10:07.036+0000 7fe1cee8b700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.154:6804 osd.2 since back 2024-03-16T12:09:27.405764+0000 front 2024-03-16T12:09:27.405893+0000 (oldest deadline 2024-03-16T12:09:52.105536+0000) 2024-03-16T12:10:07.327 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 16 12:10:07 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-9[67182]: 2024-03-16T12:10:07.036+0000 7fe1cee8b700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-16T12:09:37.907196+0000 front 2024-03-16T12:09:37.907286+0000 (oldest deadline 2024-03-16T12:10:03.207291+0000) 2024-03-16T12:10:07.534 INFO:journalctl@ceph.osd.8.smithi154.stdout:Mar 16 12:10:07 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-8[55054]: 2024-03-16T12:10:07.235+0000 7f3f065c0700 -1 osd.8 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:19.263068+0000 front 2024-03-16T12:09:19.263120+0000 (oldest deadline 2024-03-16T12:09:44.563033+0000) 2024-03-16T12:10:07.534 INFO:journalctl@ceph.osd.8.smithi154.stdout:Mar 16 12:10:07 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-8[55054]: 2024-03-16T12:10:07.235+0000 7f3f065c0700 -1 osd.8 119 heartbeat_check: no reply from 172.21.15.154:6804 osd.2 since back 2024-03-16T12:09:29.566064+0000 front 2024-03-16T12:09:29.566042+0000 (oldest deadline 2024-03-16T12:09:54.266168+0000) 2024-03-16T12:10:07.534 INFO:journalctl@ceph.osd.8.smithi154.stdout:Mar 16 12:10:07 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-8[55054]: 2024-03-16T12:10:07.235+0000 7f3f065c0700 -1 osd.8 119 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-16T12:09:38.367565+0000 front 2024-03-16T12:09:38.367474+0000 (oldest deadline 2024-03-16T12:10:01.867407+0000) 2024-03-16T12:10:07.616 INFO:journalctl@ceph.osd.7.smithi152.stdout:Mar 16 12:10:07 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-7[54789]: 2024-03-16T12:10:07.223+0000 7f6c60c47700 -1 osd.7 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:17.304563+0000 front 2024-03-16T12:09:17.304652+0000 (oldest deadline 2024-03-16T12:09:43.204754+0000) 2024-03-16T12:10:07.617 INFO:journalctl@ceph.osd.7.smithi152.stdout:Mar 16 12:10:07 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-7[54789]: 2024-03-16T12:10:07.223+0000 7f6c60c47700 -1 osd.7 119 heartbeat_check: no reply from 172.21.15.154:6804 osd.2 since back 2024-03-16T12:09:27.607556+0000 front 2024-03-16T12:09:27.607478+0000 (oldest deadline 2024-03-16T12:09:52.307502+0000) 2024-03-16T12:10:07.617 INFO:journalctl@ceph.osd.7.smithi152.stdout:Mar 16 12:10:07 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-7[54789]: 2024-03-16T12:10:07.223+0000 7f6c60c47700 -1 osd.7 119 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-16T12:09:35.808740+0000 front 2024-03-16T12:09:35.808611+0000 (oldest deadline 2024-03-16T12:10:00.508533+0000) 2024-03-16T12:10:08.116 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:10:07 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:10:07.715+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:10.713221+0000 front 2024-03-16T12:09:10.713087+0000 (oldest deadline 2024-03-16T12:09:35.413059+0000) 2024-03-16T12:10:08.117 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:10:07 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:10:07.715+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:20.614694+0000 front 2024-03-16T12:09:20.614671+0000 (oldest deadline 2024-03-16T12:09:45.215535+0000) 2024-03-16T12:10:08.117 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:10:07 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:10:07.715+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.154:6804 osd.2 since back 2024-03-16T12:09:29.916590+0000 front 2024-03-16T12:09:29.916612+0000 (oldest deadline 2024-03-16T12:09:52.816525+0000) 2024-03-16T12:10:08.117 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:10:07 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:10:07.715+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-16T12:09:37.517617+0000 front 2024-03-16T12:09:37.517557+0000 (oldest deadline 2024-03-16T12:10:02.817580+0000) 2024-03-16T12:10:08.117 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:10:07 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:10:07.715+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.152:6812 osd.4 since back 2024-03-16T12:09:42.818002+0000 front 2024-03-16T12:09:42.818046+0000 (oldest deadline 2024-03-16T12:10:07.518142+0000) 2024-03-16T12:10:08.195 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:07 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:07.862+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:13.025675+0000 front 2024-03-16T12:09:13.025847+0000 (oldest deadline 2024-03-16T12:09:35.325827+0000) 2024-03-16T12:10:08.196 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:07 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:07.862+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:19.927745+0000 front 2024-03-16T12:09:19.927761+0000 (oldest deadline 2024-03-16T12:09:45.827828+0000) 2024-03-16T12:10:08.196 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:07 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:07.862+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.154:6804 osd.2 since back 2024-03-16T12:09:28.629320+0000 front 2024-03-16T12:09:28.629357+0000 (oldest deadline 2024-03-16T12:09:52.129472+0000) 2024-03-16T12:10:08.196 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:07 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:07.862+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-16T12:09:38.432099+0000 front 2024-03-16T12:09:38.432072+0000 (oldest deadline 2024-03-16T12:10:01.332006+0000) 2024-03-16T12:10:08.196 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:07 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:07.862+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.152:6812 osd.4 since back 2024-03-16T12:09:43.633241+0000 front 2024-03-16T12:09:43.633117+0000 (oldest deadline 2024-03-16T12:10:07.733019+0000) 2024-03-16T12:10:08.327 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 16 12:10:08 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-9[67182]: 2024-03-16T12:10:08.000+0000 7fe1cee8b700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:12.704136+0000 front 2024-03-16T12:09:12.703904+0000 (oldest deadline 2024-03-16T12:09:37.404049+0000) 2024-03-16T12:10:08.327 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 16 12:10:08 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-9[67182]: 2024-03-16T12:10:08.000+0000 7fe1cee8b700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:17.404712+0000 front 2024-03-16T12:09:17.404800+0000 (oldest deadline 2024-03-16T12:09:42.704627+0000) 2024-03-16T12:10:08.327 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 16 12:10:08 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-9[67182]: 2024-03-16T12:10:08.000+0000 7fe1cee8b700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.154:6804 osd.2 since back 2024-03-16T12:09:27.405764+0000 front 2024-03-16T12:09:27.405893+0000 (oldest deadline 2024-03-16T12:09:52.105536+0000) 2024-03-16T12:10:08.327 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 16 12:10:08 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-9[67182]: 2024-03-16T12:10:08.000+0000 7fe1cee8b700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-16T12:09:37.907196+0000 front 2024-03-16T12:09:37.907286+0000 (oldest deadline 2024-03-16T12:10:03.207291+0000) 2024-03-16T12:10:08.327 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 16 12:10:08 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-9[67182]: 2024-03-16T12:10:08.000+0000 7fe1cee8b700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.152:6812 osd.4 since back 2024-03-16T12:09:43.207829+0000 front 2024-03-16T12:09:43.207999+0000 (oldest deadline 2024-03-16T12:10:07.907827+0000) 2024-03-16T12:10:08.534 INFO:journalctl@ceph.osd.8.smithi154.stdout:Mar 16 12:10:08 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-8[55054]: 2024-03-16T12:10:08.207+0000 7f3f065c0700 -1 osd.8 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:19.263068+0000 front 2024-03-16T12:09:19.263120+0000 (oldest deadline 2024-03-16T12:09:44.563033+0000) 2024-03-16T12:10:08.534 INFO:journalctl@ceph.osd.8.smithi154.stdout:Mar 16 12:10:08 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-8[55054]: 2024-03-16T12:10:08.207+0000 7f3f065c0700 -1 osd.8 119 heartbeat_check: no reply from 172.21.15.154:6804 osd.2 since back 2024-03-16T12:09:29.566064+0000 front 2024-03-16T12:09:29.566042+0000 (oldest deadline 2024-03-16T12:09:54.266168+0000) 2024-03-16T12:10:08.535 INFO:journalctl@ceph.osd.8.smithi154.stdout:Mar 16 12:10:08 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-8[55054]: 2024-03-16T12:10:08.207+0000 7f3f065c0700 -1 osd.8 119 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-16T12:09:38.367565+0000 front 2024-03-16T12:09:38.367474+0000 (oldest deadline 2024-03-16T12:10:01.867407+0000) 2024-03-16T12:10:08.535 INFO:journalctl@ceph.osd.8.smithi154.stdout:Mar 16 12:10:08 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-8[55054]: 2024-03-16T12:10:08.207+0000 7f3f065c0700 -1 osd.8 119 heartbeat_check: no reply from 172.21.15.152:6812 osd.4 since back 2024-03-16T12:09:45.269460+0000 front 2024-03-16T12:09:45.269372+0000 (oldest deadline 2024-03-16T12:10:08.169293+0000) 2024-03-16T12:10:08.616 INFO:journalctl@ceph.osd.7.smithi152.stdout:Mar 16 12:10:08 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-7[54789]: 2024-03-16T12:10:08.253+0000 7f6c60c47700 -1 osd.7 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:17.304563+0000 front 2024-03-16T12:09:17.304652+0000 (oldest deadline 2024-03-16T12:09:43.204754+0000) 2024-03-16T12:10:08.616 INFO:journalctl@ceph.osd.7.smithi152.stdout:Mar 16 12:10:08 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-7[54789]: 2024-03-16T12:10:08.253+0000 7f6c60c47700 -1 osd.7 119 heartbeat_check: no reply from 172.21.15.154:6804 osd.2 since back 2024-03-16T12:09:27.607556+0000 front 2024-03-16T12:09:27.607478+0000 (oldest deadline 2024-03-16T12:09:52.307502+0000) 2024-03-16T12:10:08.616 INFO:journalctl@ceph.osd.7.smithi152.stdout:Mar 16 12:10:08 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-7[54789]: 2024-03-16T12:10:08.253+0000 7f6c60c47700 -1 osd.7 119 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-16T12:09:35.808740+0000 front 2024-03-16T12:09:35.808611+0000 (oldest deadline 2024-03-16T12:10:00.508533+0000) 2024-03-16T12:10:09.116 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:10:08 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:10:08.695+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:10.713221+0000 front 2024-03-16T12:09:10.713087+0000 (oldest deadline 2024-03-16T12:09:35.413059+0000) 2024-03-16T12:10:09.117 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:10:08 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:10:08.695+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:20.614694+0000 front 2024-03-16T12:09:20.614671+0000 (oldest deadline 2024-03-16T12:09:45.215535+0000) 2024-03-16T12:10:09.117 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:10:08 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:10:08.695+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.154:6804 osd.2 since back 2024-03-16T12:09:29.916590+0000 front 2024-03-16T12:09:29.916612+0000 (oldest deadline 2024-03-16T12:09:52.816525+0000) 2024-03-16T12:10:09.117 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:10:08 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:10:08.695+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-16T12:09:37.517617+0000 front 2024-03-16T12:09:37.517557+0000 (oldest deadline 2024-03-16T12:10:02.817580+0000) 2024-03-16T12:10:09.117 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:10:08 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:10:08.695+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.152:6812 osd.4 since back 2024-03-16T12:09:42.818002+0000 front 2024-03-16T12:09:42.818046+0000 (oldest deadline 2024-03-16T12:10:07.518142+0000) 2024-03-16T12:10:09.192 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:08 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:08.875+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:13.025675+0000 front 2024-03-16T12:09:13.025847+0000 (oldest deadline 2024-03-16T12:09:35.325827+0000) 2024-03-16T12:10:09.192 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:08 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:08.875+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:19.927745+0000 front 2024-03-16T12:09:19.927761+0000 (oldest deadline 2024-03-16T12:09:45.827828+0000) 2024-03-16T12:10:09.192 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:08 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:08.875+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.154:6804 osd.2 since back 2024-03-16T12:09:28.629320+0000 front 2024-03-16T12:09:28.629357+0000 (oldest deadline 2024-03-16T12:09:52.129472+0000) 2024-03-16T12:10:09.192 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:08 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:08.875+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-16T12:09:38.432099+0000 front 2024-03-16T12:09:38.432072+0000 (oldest deadline 2024-03-16T12:10:01.332006+0000) 2024-03-16T12:10:09.192 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:08 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:08.875+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.152:6812 osd.4 since back 2024-03-16T12:09:43.633241+0000 front 2024-03-16T12:09:43.633117+0000 (oldest deadline 2024-03-16T12:10:07.733019+0000) 2024-03-16T12:10:09.327 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 16 12:10:08 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-9[67182]: 2024-03-16T12:10:08.987+0000 7fe1cee8b700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:12.704136+0000 front 2024-03-16T12:09:12.703904+0000 (oldest deadline 2024-03-16T12:09:37.404049+0000) 2024-03-16T12:10:09.327 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 16 12:10:08 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-9[67182]: 2024-03-16T12:10:08.987+0000 7fe1cee8b700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:17.404712+0000 front 2024-03-16T12:09:17.404800+0000 (oldest deadline 2024-03-16T12:09:42.704627+0000) 2024-03-16T12:10:09.327 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 16 12:10:08 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-9[67182]: 2024-03-16T12:10:08.987+0000 7fe1cee8b700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.154:6804 osd.2 since back 2024-03-16T12:09:27.405764+0000 front 2024-03-16T12:09:27.405893+0000 (oldest deadline 2024-03-16T12:09:52.105536+0000) 2024-03-16T12:10:09.327 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 16 12:10:08 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-9[67182]: 2024-03-16T12:10:08.987+0000 7fe1cee8b700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-16T12:09:37.907196+0000 front 2024-03-16T12:09:37.907286+0000 (oldest deadline 2024-03-16T12:10:03.207291+0000) 2024-03-16T12:10:09.327 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 16 12:10:08 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-9[67182]: 2024-03-16T12:10:08.987+0000 7fe1cee8b700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.152:6812 osd.4 since back 2024-03-16T12:09:43.207829+0000 front 2024-03-16T12:09:43.207999+0000 (oldest deadline 2024-03-16T12:10:07.907827+0000) 2024-03-16T12:10:09.534 INFO:journalctl@ceph.osd.8.smithi154.stdout:Mar 16 12:10:09 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-8[55054]: 2024-03-16T12:10:09.203+0000 7f3f065c0700 -1 osd.8 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:19.263068+0000 front 2024-03-16T12:09:19.263120+0000 (oldest deadline 2024-03-16T12:09:44.563033+0000) 2024-03-16T12:10:09.534 INFO:journalctl@ceph.osd.8.smithi154.stdout:Mar 16 12:10:09 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-8[55054]: 2024-03-16T12:10:09.203+0000 7f3f065c0700 -1 osd.8 119 heartbeat_check: no reply from 172.21.15.154:6804 osd.2 since back 2024-03-16T12:09:29.566064+0000 front 2024-03-16T12:09:29.566042+0000 (oldest deadline 2024-03-16T12:09:54.266168+0000) 2024-03-16T12:10:09.534 INFO:journalctl@ceph.osd.8.smithi154.stdout:Mar 16 12:10:09 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-8[55054]: 2024-03-16T12:10:09.203+0000 7f3f065c0700 -1 osd.8 119 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-16T12:09:38.367565+0000 front 2024-03-16T12:09:38.367474+0000 (oldest deadline 2024-03-16T12:10:01.867407+0000) 2024-03-16T12:10:09.534 INFO:journalctl@ceph.osd.8.smithi154.stdout:Mar 16 12:10:09 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-8[55054]: 2024-03-16T12:10:09.203+0000 7f3f065c0700 -1 osd.8 119 heartbeat_check: no reply from 172.21.15.152:6812 osd.4 since back 2024-03-16T12:09:45.269460+0000 front 2024-03-16T12:09:45.269372+0000 (oldest deadline 2024-03-16T12:10:08.169293+0000) 2024-03-16T12:10:09.616 INFO:journalctl@ceph.osd.7.smithi152.stdout:Mar 16 12:10:09 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-7[54789]: 2024-03-16T12:10:09.228+0000 7f6c60c47700 -1 osd.7 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:17.304563+0000 front 2024-03-16T12:09:17.304652+0000 (oldest deadline 2024-03-16T12:09:43.204754+0000) 2024-03-16T12:10:09.616 INFO:journalctl@ceph.osd.7.smithi152.stdout:Mar 16 12:10:09 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-7[54789]: 2024-03-16T12:10:09.228+0000 7f6c60c47700 -1 osd.7 119 heartbeat_check: no reply from 172.21.15.154:6804 osd.2 since back 2024-03-16T12:09:27.607556+0000 front 2024-03-16T12:09:27.607478+0000 (oldest deadline 2024-03-16T12:09:52.307502+0000) 2024-03-16T12:10:09.616 INFO:journalctl@ceph.osd.7.smithi152.stdout:Mar 16 12:10:09 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-7[54789]: 2024-03-16T12:10:09.228+0000 7f6c60c47700 -1 osd.7 119 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-16T12:09:35.808740+0000 front 2024-03-16T12:09:35.808611+0000 (oldest deadline 2024-03-16T12:10:00.508533+0000) 2024-03-16T12:10:10.117 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:10:09 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:10:09.739+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:10.713221+0000 front 2024-03-16T12:09:10.713087+0000 (oldest deadline 2024-03-16T12:09:35.413059+0000) 2024-03-16T12:10:10.117 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:10:09 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:10:09.739+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:20.614694+0000 front 2024-03-16T12:09:20.614671+0000 (oldest deadline 2024-03-16T12:09:45.215535+0000) 2024-03-16T12:10:10.117 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:10:09 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:10:09.739+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.154:6804 osd.2 since back 2024-03-16T12:09:29.916590+0000 front 2024-03-16T12:09:29.916612+0000 (oldest deadline 2024-03-16T12:09:52.816525+0000) 2024-03-16T12:10:10.117 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:10:09 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:10:09.739+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-16T12:09:37.517617+0000 front 2024-03-16T12:09:37.517557+0000 (oldest deadline 2024-03-16T12:10:02.817580+0000) 2024-03-16T12:10:10.118 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:10:09 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:10:09.739+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.152:6812 osd.4 since back 2024-03-16T12:09:42.818002+0000 front 2024-03-16T12:09:42.818046+0000 (oldest deadline 2024-03-16T12:10:07.518142+0000) 2024-03-16T12:10:10.210 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:09 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:09.877+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:13.025675+0000 front 2024-03-16T12:09:13.025847+0000 (oldest deadline 2024-03-16T12:09:35.325827+0000) 2024-03-16T12:10:10.210 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:09 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:09.877+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:19.927745+0000 front 2024-03-16T12:09:19.927761+0000 (oldest deadline 2024-03-16T12:09:45.827828+0000) 2024-03-16T12:10:10.210 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:09 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:09.877+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.154:6804 osd.2 since back 2024-03-16T12:09:28.629320+0000 front 2024-03-16T12:09:28.629357+0000 (oldest deadline 2024-03-16T12:09:52.129472+0000) 2024-03-16T12:10:10.211 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:09 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:09.878+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-16T12:09:38.432099+0000 front 2024-03-16T12:09:38.432072+0000 (oldest deadline 2024-03-16T12:10:01.332006+0000) 2024-03-16T12:10:10.211 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:09 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:09.878+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.152:6812 osd.4 since back 2024-03-16T12:09:43.633241+0000 front 2024-03-16T12:09:43.633117+0000 (oldest deadline 2024-03-16T12:10:07.733019+0000) 2024-03-16T12:10:10.327 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 16 12:10:09 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-9[67182]: 2024-03-16T12:10:09.988+0000 7fe1cee8b700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:12.704136+0000 front 2024-03-16T12:09:12.703904+0000 (oldest deadline 2024-03-16T12:09:37.404049+0000) 2024-03-16T12:10:10.327 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 16 12:10:09 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-9[67182]: 2024-03-16T12:10:09.988+0000 7fe1cee8b700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:17.404712+0000 front 2024-03-16T12:09:17.404800+0000 (oldest deadline 2024-03-16T12:09:42.704627+0000) 2024-03-16T12:10:10.327 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 16 12:10:09 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-9[67182]: 2024-03-16T12:10:09.988+0000 7fe1cee8b700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.154:6804 osd.2 since back 2024-03-16T12:09:27.405764+0000 front 2024-03-16T12:09:27.405893+0000 (oldest deadline 2024-03-16T12:09:52.105536+0000) 2024-03-16T12:10:10.327 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 16 12:10:09 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-9[67182]: 2024-03-16T12:10:09.988+0000 7fe1cee8b700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-16T12:09:37.907196+0000 front 2024-03-16T12:09:37.907286+0000 (oldest deadline 2024-03-16T12:10:03.207291+0000) 2024-03-16T12:10:10.327 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 16 12:10:09 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-9[67182]: 2024-03-16T12:10:09.988+0000 7fe1cee8b700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.152:6812 osd.4 since back 2024-03-16T12:09:43.207829+0000 front 2024-03-16T12:09:43.207999+0000 (oldest deadline 2024-03-16T12:10:07.907827+0000) 2024-03-16T12:10:10.534 INFO:journalctl@ceph.osd.8.smithi154.stdout:Mar 16 12:10:10 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-8[55054]: 2024-03-16T12:10:10.221+0000 7f3f065c0700 -1 osd.8 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:19.263068+0000 front 2024-03-16T12:09:19.263120+0000 (oldest deadline 2024-03-16T12:09:44.563033+0000) 2024-03-16T12:10:10.534 INFO:journalctl@ceph.osd.8.smithi154.stdout:Mar 16 12:10:10 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-8[55054]: 2024-03-16T12:10:10.221+0000 7f3f065c0700 -1 osd.8 119 heartbeat_check: no reply from 172.21.15.154:6804 osd.2 since back 2024-03-16T12:09:29.566064+0000 front 2024-03-16T12:09:29.566042+0000 (oldest deadline 2024-03-16T12:09:54.266168+0000) 2024-03-16T12:10:10.534 INFO:journalctl@ceph.osd.8.smithi154.stdout:Mar 16 12:10:10 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-8[55054]: 2024-03-16T12:10:10.221+0000 7f3f065c0700 -1 osd.8 119 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-16T12:09:38.367565+0000 front 2024-03-16T12:09:38.367474+0000 (oldest deadline 2024-03-16T12:10:01.867407+0000) 2024-03-16T12:10:10.535 INFO:journalctl@ceph.osd.8.smithi154.stdout:Mar 16 12:10:10 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-8[55054]: 2024-03-16T12:10:10.221+0000 7f3f065c0700 -1 osd.8 119 heartbeat_check: no reply from 172.21.15.152:6812 osd.4 since back 2024-03-16T12:09:45.269460+0000 front 2024-03-16T12:09:45.269372+0000 (oldest deadline 2024-03-16T12:10:08.169293+0000) 2024-03-16T12:10:10.616 INFO:journalctl@ceph.osd.7.smithi152.stdout:Mar 16 12:10:10 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-7[54789]: 2024-03-16T12:10:10.224+0000 7f6c60c47700 -1 osd.7 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:17.304563+0000 front 2024-03-16T12:09:17.304652+0000 (oldest deadline 2024-03-16T12:09:43.204754+0000) 2024-03-16T12:10:10.616 INFO:journalctl@ceph.osd.7.smithi152.stdout:Mar 16 12:10:10 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-7[54789]: 2024-03-16T12:10:10.224+0000 7f6c60c47700 -1 osd.7 119 heartbeat_check: no reply from 172.21.15.154:6804 osd.2 since back 2024-03-16T12:09:27.607556+0000 front 2024-03-16T12:09:27.607478+0000 (oldest deadline 2024-03-16T12:09:52.307502+0000) 2024-03-16T12:10:10.616 INFO:journalctl@ceph.osd.7.smithi152.stdout:Mar 16 12:10:10 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-7[54789]: 2024-03-16T12:10:10.224+0000 7f6c60c47700 -1 osd.7 119 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-16T12:09:35.808740+0000 front 2024-03-16T12:09:35.808611+0000 (oldest deadline 2024-03-16T12:10:00.508533+0000) 2024-03-16T12:10:11.116 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:10:10 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:10:10.749+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:10.713221+0000 front 2024-03-16T12:09:10.713087+0000 (oldest deadline 2024-03-16T12:09:35.413059+0000) 2024-03-16T12:10:11.117 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:10:10 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:10:10.749+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:20.614694+0000 front 2024-03-16T12:09:20.614671+0000 (oldest deadline 2024-03-16T12:09:45.215535+0000) 2024-03-16T12:10:11.117 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:10:10 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:10:10.749+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.154:6804 osd.2 since back 2024-03-16T12:09:29.916590+0000 front 2024-03-16T12:09:29.916612+0000 (oldest deadline 2024-03-16T12:09:52.816525+0000) 2024-03-16T12:10:11.117 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:10:10 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:10:10.749+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-16T12:09:37.517617+0000 front 2024-03-16T12:09:37.517557+0000 (oldest deadline 2024-03-16T12:10:02.817580+0000) 2024-03-16T12:10:11.117 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:10:10 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:10:10.749+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.152:6812 osd.4 since back 2024-03-16T12:09:42.818002+0000 front 2024-03-16T12:09:42.818046+0000 (oldest deadline 2024-03-16T12:10:07.518142+0000) 2024-03-16T12:10:11.167 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:10 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:10.915+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:13.025675+0000 front 2024-03-16T12:09:13.025847+0000 (oldest deadline 2024-03-16T12:09:35.325827+0000) 2024-03-16T12:10:11.167 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:10 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:10.915+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:19.927745+0000 front 2024-03-16T12:09:19.927761+0000 (oldest deadline 2024-03-16T12:09:45.827828+0000) 2024-03-16T12:10:11.167 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:10 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:10.915+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.154:6804 osd.2 since back 2024-03-16T12:09:28.629320+0000 front 2024-03-16T12:09:28.629357+0000 (oldest deadline 2024-03-16T12:09:52.129472+0000) 2024-03-16T12:10:11.167 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:10 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:10.915+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-16T12:09:38.432099+0000 front 2024-03-16T12:09:38.432072+0000 (oldest deadline 2024-03-16T12:10:01.332006+0000) 2024-03-16T12:10:11.167 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:10 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:10.915+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.152:6812 osd.4 since back 2024-03-16T12:09:43.633241+0000 front 2024-03-16T12:09:43.633117+0000 (oldest deadline 2024-03-16T12:10:07.733019+0000) 2024-03-16T12:10:11.327 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 16 12:10:10 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-9[67182]: 2024-03-16T12:10:10.973+0000 7fe1cee8b700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:12.704136+0000 front 2024-03-16T12:09:12.703904+0000 (oldest deadline 2024-03-16T12:09:37.404049+0000) 2024-03-16T12:10:11.327 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 16 12:10:10 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-9[67182]: 2024-03-16T12:10:10.973+0000 7fe1cee8b700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:17.404712+0000 front 2024-03-16T12:09:17.404800+0000 (oldest deadline 2024-03-16T12:09:42.704627+0000) 2024-03-16T12:10:11.327 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 16 12:10:10 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-9[67182]: 2024-03-16T12:10:10.973+0000 7fe1cee8b700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.154:6804 osd.2 since back 2024-03-16T12:09:27.405764+0000 front 2024-03-16T12:09:27.405893+0000 (oldest deadline 2024-03-16T12:09:52.105536+0000) 2024-03-16T12:10:11.327 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 16 12:10:10 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-9[67182]: 2024-03-16T12:10:10.973+0000 7fe1cee8b700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-16T12:09:37.907196+0000 front 2024-03-16T12:09:37.907286+0000 (oldest deadline 2024-03-16T12:10:03.207291+0000) 2024-03-16T12:10:11.327 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 16 12:10:10 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-9[67182]: 2024-03-16T12:10:10.973+0000 7fe1cee8b700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.152:6812 osd.4 since back 2024-03-16T12:09:43.207829+0000 front 2024-03-16T12:09:43.207999+0000 (oldest deadline 2024-03-16T12:10:07.907827+0000) 2024-03-16T12:10:11.534 INFO:journalctl@ceph.osd.8.smithi154.stdout:Mar 16 12:10:11 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-8[55054]: 2024-03-16T12:10:11.178+0000 7f3f065c0700 -1 osd.8 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:19.263068+0000 front 2024-03-16T12:09:19.263120+0000 (oldest deadline 2024-03-16T12:09:44.563033+0000) 2024-03-16T12:10:11.534 INFO:journalctl@ceph.osd.8.smithi154.stdout:Mar 16 12:10:11 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-8[55054]: 2024-03-16T12:10:11.178+0000 7f3f065c0700 -1 osd.8 119 heartbeat_check: no reply from 172.21.15.154:6804 osd.2 since back 2024-03-16T12:09:29.566064+0000 front 2024-03-16T12:09:29.566042+0000 (oldest deadline 2024-03-16T12:09:54.266168+0000) 2024-03-16T12:10:11.534 INFO:journalctl@ceph.osd.8.smithi154.stdout:Mar 16 12:10:11 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-8[55054]: 2024-03-16T12:10:11.178+0000 7f3f065c0700 -1 osd.8 119 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-16T12:09:38.367565+0000 front 2024-03-16T12:09:38.367474+0000 (oldest deadline 2024-03-16T12:10:01.867407+0000) 2024-03-16T12:10:11.534 INFO:journalctl@ceph.osd.8.smithi154.stdout:Mar 16 12:10:11 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-8[55054]: 2024-03-16T12:10:11.178+0000 7f3f065c0700 -1 osd.8 119 heartbeat_check: no reply from 172.21.15.152:6812 osd.4 since back 2024-03-16T12:09:45.269460+0000 front 2024-03-16T12:09:45.269372+0000 (oldest deadline 2024-03-16T12:10:08.169293+0000) 2024-03-16T12:10:11.616 INFO:journalctl@ceph.osd.7.smithi152.stdout:Mar 16 12:10:11 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-7[54789]: 2024-03-16T12:10:11.217+0000 7f6c60c47700 -1 osd.7 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:17.304563+0000 front 2024-03-16T12:09:17.304652+0000 (oldest deadline 2024-03-16T12:09:43.204754+0000) 2024-03-16T12:10:11.616 INFO:journalctl@ceph.osd.7.smithi152.stdout:Mar 16 12:10:11 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-7[54789]: 2024-03-16T12:10:11.217+0000 7f6c60c47700 -1 osd.7 119 heartbeat_check: no reply from 172.21.15.154:6804 osd.2 since back 2024-03-16T12:09:27.607556+0000 front 2024-03-16T12:09:27.607478+0000 (oldest deadline 2024-03-16T12:09:52.307502+0000) 2024-03-16T12:10:11.617 INFO:journalctl@ceph.osd.7.smithi152.stdout:Mar 16 12:10:11 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-7[54789]: 2024-03-16T12:10:11.217+0000 7f6c60c47700 -1 osd.7 119 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-16T12:09:35.808740+0000 front 2024-03-16T12:09:35.808611+0000 (oldest deadline 2024-03-16T12:10:00.508533+0000) 2024-03-16T12:10:11.617 INFO:journalctl@ceph.osd.7.smithi152.stdout:Mar 16 12:10:11 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-7[54789]: 2024-03-16T12:10:11.217+0000 7f6c60c47700 -1 osd.7 119 heartbeat_check: no reply from 172.21.15.152:6812 osd.4 since back 2024-03-16T12:09:45.511321+0000 front 2024-03-16T12:09:45.511299+0000 (oldest deadline 2024-03-16T12:10:10.811530+0000) 2024-03-16T12:10:12.116 INFO:journalctl@ceph.osd.7.smithi152.stdout:Mar 16 12:10:11 smithi152 podman[74363]: 2024-03-16 12:10:11.939426498 +0000 UTC m=+5.808851369 container died 492ffb94294e4e753865e21143f55a0ad66bf2cc28b0c3d7960d5cee3cea92b8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-7, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-03-16T12:10:12.117 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:10:11 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:10:11.758+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:10.713221+0000 front 2024-03-16T12:09:10.713087+0000 (oldest deadline 2024-03-16T12:09:35.413059+0000) 2024-03-16T12:10:12.117 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:10:11 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:10:11.758+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:20.614694+0000 front 2024-03-16T12:09:20.614671+0000 (oldest deadline 2024-03-16T12:09:45.215535+0000) 2024-03-16T12:10:12.117 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:10:11 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:10:11.758+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.154:6804 osd.2 since back 2024-03-16T12:09:29.916590+0000 front 2024-03-16T12:09:29.916612+0000 (oldest deadline 2024-03-16T12:09:52.816525+0000) 2024-03-16T12:10:12.117 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:10:11 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:10:11.758+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-16T12:09:37.517617+0000 front 2024-03-16T12:09:37.517557+0000 (oldest deadline 2024-03-16T12:10:02.817580+0000) 2024-03-16T12:10:12.117 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:10:11 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:10:11.758+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.152:6812 osd.4 since back 2024-03-16T12:09:42.818002+0000 front 2024-03-16T12:09:42.818046+0000 (oldest deadline 2024-03-16T12:10:07.518142+0000) 2024-03-16T12:10:12.213 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:11 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:11.915+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:13.025675+0000 front 2024-03-16T12:09:13.025847+0000 (oldest deadline 2024-03-16T12:09:35.325827+0000) 2024-03-16T12:10:12.213 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:11 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:11.915+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:19.927745+0000 front 2024-03-16T12:09:19.927761+0000 (oldest deadline 2024-03-16T12:09:45.827828+0000) 2024-03-16T12:10:12.213 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:11 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:11.915+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.154:6804 osd.2 since back 2024-03-16T12:09:28.629320+0000 front 2024-03-16T12:09:28.629357+0000 (oldest deadline 2024-03-16T12:09:52.129472+0000) 2024-03-16T12:10:12.213 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:11 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:11.915+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-16T12:09:38.432099+0000 front 2024-03-16T12:09:38.432072+0000 (oldest deadline 2024-03-16T12:10:01.332006+0000) 2024-03-16T12:10:12.214 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:11 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:11.915+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.152:6812 osd.4 since back 2024-03-16T12:09:43.633241+0000 front 2024-03-16T12:09:43.633117+0000 (oldest deadline 2024-03-16T12:10:07.733019+0000) 2024-03-16T12:10:12.326 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 16 12:10:12 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-9[67182]: 2024-03-16T12:10:12.017+0000 7fe1cee8b700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:12.704136+0000 front 2024-03-16T12:09:12.703904+0000 (oldest deadline 2024-03-16T12:09:37.404049+0000) 2024-03-16T12:10:12.327 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 16 12:10:12 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-9[67182]: 2024-03-16T12:10:12.017+0000 7fe1cee8b700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:17.404712+0000 front 2024-03-16T12:09:17.404800+0000 (oldest deadline 2024-03-16T12:09:42.704627+0000) 2024-03-16T12:10:12.327 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 16 12:10:12 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-9[67182]: 2024-03-16T12:10:12.017+0000 7fe1cee8b700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.154:6804 osd.2 since back 2024-03-16T12:09:27.405764+0000 front 2024-03-16T12:09:27.405893+0000 (oldest deadline 2024-03-16T12:09:52.105536+0000) 2024-03-16T12:10:12.327 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 16 12:10:12 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-9[67182]: 2024-03-16T12:10:12.017+0000 7fe1cee8b700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-16T12:09:37.907196+0000 front 2024-03-16T12:09:37.907286+0000 (oldest deadline 2024-03-16T12:10:03.207291+0000) 2024-03-16T12:10:12.327 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 16 12:10:12 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-9[67182]: 2024-03-16T12:10:12.017+0000 7fe1cee8b700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.152:6812 osd.4 since back 2024-03-16T12:09:43.207829+0000 front 2024-03-16T12:09:43.207999+0000 (oldest deadline 2024-03-16T12:10:07.907827+0000) 2024-03-16T12:10:12.534 INFO:journalctl@ceph.osd.8.smithi154.stdout:Mar 16 12:10:12 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-8[55054]: 2024-03-16T12:10:12.224+0000 7f3f065c0700 -1 osd.8 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:19.263068+0000 front 2024-03-16T12:09:19.263120+0000 (oldest deadline 2024-03-16T12:09:44.563033+0000) 2024-03-16T12:10:12.534 INFO:journalctl@ceph.osd.8.smithi154.stdout:Mar 16 12:10:12 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-8[55054]: 2024-03-16T12:10:12.224+0000 7f3f065c0700 -1 osd.8 119 heartbeat_check: no reply from 172.21.15.154:6804 osd.2 since back 2024-03-16T12:09:29.566064+0000 front 2024-03-16T12:09:29.566042+0000 (oldest deadline 2024-03-16T12:09:54.266168+0000) 2024-03-16T12:10:12.534 INFO:journalctl@ceph.osd.8.smithi154.stdout:Mar 16 12:10:12 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-8[55054]: 2024-03-16T12:10:12.224+0000 7f3f065c0700 -1 osd.8 119 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-16T12:09:38.367565+0000 front 2024-03-16T12:09:38.367474+0000 (oldest deadline 2024-03-16T12:10:01.867407+0000) 2024-03-16T12:10:12.534 INFO:journalctl@ceph.osd.8.smithi154.stdout:Mar 16 12:10:12 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-8[55054]: 2024-03-16T12:10:12.224+0000 7f3f065c0700 -1 osd.8 119 heartbeat_check: no reply from 172.21.15.152:6812 osd.4 since back 2024-03-16T12:09:45.269460+0000 front 2024-03-16T12:09:45.269372+0000 (oldest deadline 2024-03-16T12:10:08.169293+0000) 2024-03-16T12:10:12.593 INFO:journalctl@ceph.osd.7.smithi152.stdout:Mar 16 12:10:12 smithi152 podman[74363]: 2024-03-16 12:10:12.346138294 +0000 UTC m=+6.215563164 container cleanup 492ffb94294e4e753865e21143f55a0ad66bf2cc28b0c3d7960d5cee3cea92b8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-7, ceph=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-16T12:10:12.593 INFO:journalctl@ceph.osd.7.smithi152.stdout:Mar 16 12:10:12 smithi152 bash[74363]: ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-7 2024-03-16T12:10:12.866 INFO:journalctl@ceph.osd.7.smithi152.stdout:Mar 16 12:10:12 smithi152 podman[74380]: 2024-03-16 12:10:12.605887925 +0000 UTC m=+0.664157583 container remove 492ffb94294e4e753865e21143f55a0ad66bf2cc28b0c3d7960d5cee3cea92b8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-7, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-16T12:10:12.867 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:10:12 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:10:12.756+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:10.713221+0000 front 2024-03-16T12:09:10.713087+0000 (oldest deadline 2024-03-16T12:09:35.413059+0000) 2024-03-16T12:10:12.867 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:10:12 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:10:12.756+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:20.614694+0000 front 2024-03-16T12:09:20.614671+0000 (oldest deadline 2024-03-16T12:09:45.215535+0000) 2024-03-16T12:10:12.867 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:10:12 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:10:12.756+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.154:6804 osd.2 since back 2024-03-16T12:09:29.916590+0000 front 2024-03-16T12:09:29.916612+0000 (oldest deadline 2024-03-16T12:09:52.816525+0000) 2024-03-16T12:10:12.867 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:10:12 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:10:12.756+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-16T12:09:37.517617+0000 front 2024-03-16T12:09:37.517557+0000 (oldest deadline 2024-03-16T12:10:02.817580+0000) 2024-03-16T12:10:12.867 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:10:12 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:10:12.756+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.152:6812 osd.4 since back 2024-03-16T12:09:42.818002+0000 front 2024-03-16T12:09:42.818046+0000 (oldest deadline 2024-03-16T12:10:07.518142+0000) 2024-03-16T12:10:13.155 INFO:journalctl@ceph.osd.7.smithi152.stdout:Mar 16 12:10:12 smithi152 podman[74469]: 2024-03-16 12:10:12.807556125 +0000 UTC m=+0.017534963 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-16T12:10:13.155 INFO:journalctl@ceph.osd.7.smithi152.stdout:Mar 16 12:10:12 smithi152 podman[74469]: 2024-03-16 12:10:12.923333914 +0000 UTC m=+0.133312789 container create 1ff9fb89cdf408358493c2a1d6a6665e8d9adb98583d2477000880e1e811ebed (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-7-deactivate, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5) 2024-03-16T12:10:13.155 INFO:journalctl@ceph.osd.7.smithi152.stdout:Mar 16 12:10:13 smithi152 podman[74469]: 2024-03-16 12:10:13.167450419 +0000 UTC m=+0.377429271 container init 1ff9fb89cdf408358493c2a1d6a6665e8d9adb98583d2477000880e1e811ebed (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-7-deactivate, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240314.183243-3d78f5b) 2024-03-16T12:10:13.228 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:12 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:12.927+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:13.025675+0000 front 2024-03-16T12:09:13.025847+0000 (oldest deadline 2024-03-16T12:09:35.325827+0000) 2024-03-16T12:10:13.228 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:12 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:12.927+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:19.927745+0000 front 2024-03-16T12:09:19.927761+0000 (oldest deadline 2024-03-16T12:09:45.827828+0000) 2024-03-16T12:10:13.228 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:12 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:12.927+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.154:6804 osd.2 since back 2024-03-16T12:09:28.629320+0000 front 2024-03-16T12:09:28.629357+0000 (oldest deadline 2024-03-16T12:09:52.129472+0000) 2024-03-16T12:10:13.228 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:12 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:12.927+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-16T12:09:38.432099+0000 front 2024-03-16T12:09:38.432072+0000 (oldest deadline 2024-03-16T12:10:01.332006+0000) 2024-03-16T12:10:13.228 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:12 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:12.927+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.152:6812 osd.4 since back 2024-03-16T12:09:43.633241+0000 front 2024-03-16T12:09:43.633117+0000 (oldest deadline 2024-03-16T12:10:07.733019+0000) 2024-03-16T12:10:13.327 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 16 12:10:13 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-9[67182]: 2024-03-16T12:10:13.001+0000 7fe1cee8b700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:12.704136+0000 front 2024-03-16T12:09:12.703904+0000 (oldest deadline 2024-03-16T12:09:37.404049+0000) 2024-03-16T12:10:13.327 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 16 12:10:13 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-9[67182]: 2024-03-16T12:10:13.001+0000 7fe1cee8b700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:17.404712+0000 front 2024-03-16T12:09:17.404800+0000 (oldest deadline 2024-03-16T12:09:42.704627+0000) 2024-03-16T12:10:13.327 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 16 12:10:13 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-9[67182]: 2024-03-16T12:10:13.001+0000 7fe1cee8b700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.154:6804 osd.2 since back 2024-03-16T12:09:27.405764+0000 front 2024-03-16T12:09:27.405893+0000 (oldest deadline 2024-03-16T12:09:52.105536+0000) 2024-03-16T12:10:13.327 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 16 12:10:13 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-9[67182]: 2024-03-16T12:10:13.001+0000 7fe1cee8b700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-16T12:09:37.907196+0000 front 2024-03-16T12:09:37.907286+0000 (oldest deadline 2024-03-16T12:10:03.207291+0000) 2024-03-16T12:10:13.327 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 16 12:10:13 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-9[67182]: 2024-03-16T12:10:13.001+0000 7fe1cee8b700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.152:6812 osd.4 since back 2024-03-16T12:09:43.207829+0000 front 2024-03-16T12:09:43.207999+0000 (oldest deadline 2024-03-16T12:10:07.907827+0000) 2024-03-16T12:10:13.447 INFO:journalctl@ceph.osd.7.smithi152.stdout:Mar 16 12:10:13 smithi152 podman[74469]: 2024-03-16 12:10:13.17077536 +0000 UTC m=+0.380754185 container start 1ff9fb89cdf408358493c2a1d6a6665e8d9adb98583d2477000880e1e811ebed (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-7-deactivate, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-16T12:10:13.447 INFO:journalctl@ceph.osd.7.smithi152.stdout:Mar 16 12:10:13 smithi152 podman[74469]: 2024-03-16 12:10:13.292709111 +0000 UTC m=+0.502687964 container attach 1ff9fb89cdf408358493c2a1d6a6665e8d9adb98583d2477000880e1e811ebed (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-7-deactivate, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240307, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-03-16T12:10:13.534 INFO:journalctl@ceph.osd.8.smithi154.stdout:Mar 16 12:10:13 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-8[55054]: 2024-03-16T12:10:13.239+0000 7f3f065c0700 -1 osd.8 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:19.263068+0000 front 2024-03-16T12:09:19.263120+0000 (oldest deadline 2024-03-16T12:09:44.563033+0000) 2024-03-16T12:10:13.534 INFO:journalctl@ceph.osd.8.smithi154.stdout:Mar 16 12:10:13 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-8[55054]: 2024-03-16T12:10:13.239+0000 7f3f065c0700 -1 osd.8 119 heartbeat_check: no reply from 172.21.15.154:6804 osd.2 since back 2024-03-16T12:09:29.566064+0000 front 2024-03-16T12:09:29.566042+0000 (oldest deadline 2024-03-16T12:09:54.266168+0000) 2024-03-16T12:10:13.534 INFO:journalctl@ceph.osd.8.smithi154.stdout:Mar 16 12:10:13 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-8[55054]: 2024-03-16T12:10:13.239+0000 7f3f065c0700 -1 osd.8 119 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-16T12:09:38.367565+0000 front 2024-03-16T12:09:38.367474+0000 (oldest deadline 2024-03-16T12:10:01.867407+0000) 2024-03-16T12:10:13.534 INFO:journalctl@ceph.osd.8.smithi154.stdout:Mar 16 12:10:13 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-8[55054]: 2024-03-16T12:10:13.239+0000 7f3f065c0700 -1 osd.8 119 heartbeat_check: no reply from 172.21.15.152:6812 osd.4 since back 2024-03-16T12:09:45.269460+0000 front 2024-03-16T12:09:45.269372+0000 (oldest deadline 2024-03-16T12:10:08.169293+0000) 2024-03-16T12:10:13.733 INFO:journalctl@ceph.osd.7.smithi152.stdout:Mar 16 12:10:13 smithi152 podman[74469]: 2024-03-16 12:10:13.459114874 +0000 UTC m=+0.669093707 container died 1ff9fb89cdf408358493c2a1d6a6665e8d9adb98583d2477000880e1e811ebed (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-7-deactivate, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-03-16T12:10:14.017 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:10:13 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:10:13.792+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:10.713221+0000 front 2024-03-16T12:09:10.713087+0000 (oldest deadline 2024-03-16T12:09:35.413059+0000) 2024-03-16T12:10:14.017 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:10:13 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:10:13.792+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:20.614694+0000 front 2024-03-16T12:09:20.614671+0000 (oldest deadline 2024-03-16T12:09:45.215535+0000) 2024-03-16T12:10:14.017 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:10:13 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:10:13.792+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.154:6804 osd.2 since back 2024-03-16T12:09:29.916590+0000 front 2024-03-16T12:09:29.916612+0000 (oldest deadline 2024-03-16T12:09:52.816525+0000) 2024-03-16T12:10:14.017 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:10:13 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:10:13.792+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-16T12:09:37.517617+0000 front 2024-03-16T12:09:37.517557+0000 (oldest deadline 2024-03-16T12:10:02.817580+0000) 2024-03-16T12:10:14.017 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:10:13 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:10:13.792+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.152:6812 osd.4 since back 2024-03-16T12:09:42.818002+0000 front 2024-03-16T12:09:42.818046+0000 (oldest deadline 2024-03-16T12:10:07.518142+0000) 2024-03-16T12:10:14.037 DEBUG:teuthology.orchestra.run.smithi152:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d1887f28-e389-11ee-95c9-87774f69a715@osd.7.service' 2024-03-16T12:10:14.120 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-16T12:10:14.120 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-03-16T12:10:14.121 INFO:tasks.cephadm.osd.8:Stopping osd.8... 2024-03-16T12:10:14.121 DEBUG:teuthology.orchestra.run.smithi154:> sudo systemctl stop ceph-d1887f28-e389-11ee-95c9-87774f69a715@osd.8 2024-03-16T12:10:14.255 INFO:journalctl@ceph.osd.8.smithi154.stdout:Mar 16 12:10:14 smithi154 systemd[1]: Stopping Ceph osd.8 for d1887f28-e389-11ee-95c9-87774f69a715... 2024-03-16T12:10:14.256 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:13 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:13.945+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:13.025675+0000 front 2024-03-16T12:09:13.025847+0000 (oldest deadline 2024-03-16T12:09:35.325827+0000) 2024-03-16T12:10:14.256 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:13 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:13.945+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:19.927745+0000 front 2024-03-16T12:09:19.927761+0000 (oldest deadline 2024-03-16T12:09:45.827828+0000) 2024-03-16T12:10:14.256 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:13 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:13.945+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.154:6804 osd.2 since back 2024-03-16T12:09:28.629320+0000 front 2024-03-16T12:09:28.629357+0000 (oldest deadline 2024-03-16T12:09:52.129472+0000) 2024-03-16T12:10:14.256 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:13 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:13.945+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-16T12:09:38.432099+0000 front 2024-03-16T12:09:38.432072+0000 (oldest deadline 2024-03-16T12:10:01.332006+0000) 2024-03-16T12:10:14.256 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:13 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:13.945+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.152:6812 osd.4 since back 2024-03-16T12:09:43.633241+0000 front 2024-03-16T12:09:43.633117+0000 (oldest deadline 2024-03-16T12:10:07.733019+0000) 2024-03-16T12:10:14.327 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 16 12:10:14 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-9[67182]: 2024-03-16T12:10:14.048+0000 7fe1cee8b700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:12.704136+0000 front 2024-03-16T12:09:12.703904+0000 (oldest deadline 2024-03-16T12:09:37.404049+0000) 2024-03-16T12:10:14.327 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 16 12:10:14 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-9[67182]: 2024-03-16T12:10:14.048+0000 7fe1cee8b700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:17.404712+0000 front 2024-03-16T12:09:17.404800+0000 (oldest deadline 2024-03-16T12:09:42.704627+0000) 2024-03-16T12:10:14.327 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 16 12:10:14 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-9[67182]: 2024-03-16T12:10:14.048+0000 7fe1cee8b700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.154:6804 osd.2 since back 2024-03-16T12:09:27.405764+0000 front 2024-03-16T12:09:27.405893+0000 (oldest deadline 2024-03-16T12:09:52.105536+0000) 2024-03-16T12:10:14.327 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 16 12:10:14 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-9[67182]: 2024-03-16T12:10:14.048+0000 7fe1cee8b700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-16T12:09:37.907196+0000 front 2024-03-16T12:09:37.907286+0000 (oldest deadline 2024-03-16T12:10:03.207291+0000) 2024-03-16T12:10:14.327 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 16 12:10:14 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-9[67182]: 2024-03-16T12:10:14.048+0000 7fe1cee8b700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.152:6812 osd.4 since back 2024-03-16T12:09:43.207829+0000 front 2024-03-16T12:09:43.207999+0000 (oldest deadline 2024-03-16T12:10:07.907827+0000) 2024-03-16T12:10:14.534 INFO:journalctl@ceph.osd.8.smithi154.stdout:Mar 16 12:10:14 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-8[55054]: 2024-03-16T12:10:14.267+0000 7f3f065c0700 -1 osd.8 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:19.263068+0000 front 2024-03-16T12:09:19.263120+0000 (oldest deadline 2024-03-16T12:09:44.563033+0000) 2024-03-16T12:10:14.534 INFO:journalctl@ceph.osd.8.smithi154.stdout:Mar 16 12:10:14 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-8[55054]: 2024-03-16T12:10:14.267+0000 7f3f065c0700 -1 osd.8 119 heartbeat_check: no reply from 172.21.15.154:6804 osd.2 since back 2024-03-16T12:09:29.566064+0000 front 2024-03-16T12:09:29.566042+0000 (oldest deadline 2024-03-16T12:09:54.266168+0000) 2024-03-16T12:10:14.535 INFO:journalctl@ceph.osd.8.smithi154.stdout:Mar 16 12:10:14 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-8[55054]: 2024-03-16T12:10:14.267+0000 7f3f065c0700 -1 osd.8 119 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-16T12:09:38.367565+0000 front 2024-03-16T12:09:38.367474+0000 (oldest deadline 2024-03-16T12:10:01.867407+0000) 2024-03-16T12:10:14.535 INFO:journalctl@ceph.osd.8.smithi154.stdout:Mar 16 12:10:14 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-8[55054]: 2024-03-16T12:10:14.267+0000 7f3f065c0700 -1 osd.8 119 heartbeat_check: no reply from 172.21.15.152:6812 osd.4 since back 2024-03-16T12:09:45.269460+0000 front 2024-03-16T12:09:45.269372+0000 (oldest deadline 2024-03-16T12:10:08.169293+0000) 2024-03-16T12:10:14.535 INFO:journalctl@ceph.osd.8.smithi154.stdout:Mar 16 12:10:14 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-8[55054]: 2024-03-16T12:10:14.483+0000 7f3f0dfd4700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.8 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-16T12:10:14.535 INFO:journalctl@ceph.osd.8.smithi154.stdout:Mar 16 12:10:14 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-8[55054]: 2024-03-16T12:10:14.483+0000 7f3f0dfd4700 -1 osd.8 119 *** Got signal Terminated *** 2024-03-16T12:10:14.535 INFO:journalctl@ceph.osd.8.smithi154.stdout:Mar 16 12:10:14 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-8[55054]: 2024-03-16T12:10:14.483+0000 7f3f0dfd4700 -1 osd.8 119 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-16T12:10:15.116 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:10:14 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:10:14.834+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:10.713221+0000 front 2024-03-16T12:09:10.713087+0000 (oldest deadline 2024-03-16T12:09:35.413059+0000) 2024-03-16T12:10:15.117 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:10:14 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:10:14.834+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:20.614694+0000 front 2024-03-16T12:09:20.614671+0000 (oldest deadline 2024-03-16T12:09:45.215535+0000) 2024-03-16T12:10:15.117 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:10:14 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:10:14.834+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.154:6804 osd.2 since back 2024-03-16T12:09:29.916590+0000 front 2024-03-16T12:09:29.916612+0000 (oldest deadline 2024-03-16T12:09:52.816525+0000) 2024-03-16T12:10:15.117 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:10:14 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:10:14.834+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-16T12:09:37.517617+0000 front 2024-03-16T12:09:37.517557+0000 (oldest deadline 2024-03-16T12:10:02.817580+0000) 2024-03-16T12:10:15.117 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:10:14 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:10:14.834+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.152:6812 osd.4 since back 2024-03-16T12:09:42.818002+0000 front 2024-03-16T12:09:42.818046+0000 (oldest deadline 2024-03-16T12:10:07.518142+0000) 2024-03-16T12:10:15.275 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:14 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:14.953+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:13.025675+0000 front 2024-03-16T12:09:13.025847+0000 (oldest deadline 2024-03-16T12:09:35.325827+0000) 2024-03-16T12:10:15.275 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:14 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:14.953+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:19.927745+0000 front 2024-03-16T12:09:19.927761+0000 (oldest deadline 2024-03-16T12:09:45.827828+0000) 2024-03-16T12:10:15.275 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:14 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:14.953+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.154:6804 osd.2 since back 2024-03-16T12:09:28.629320+0000 front 2024-03-16T12:09:28.629357+0000 (oldest deadline 2024-03-16T12:09:52.129472+0000) 2024-03-16T12:10:15.276 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:14 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:14.953+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-16T12:09:38.432099+0000 front 2024-03-16T12:09:38.432072+0000 (oldest deadline 2024-03-16T12:10:01.332006+0000) 2024-03-16T12:10:15.276 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:14 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:14.953+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.152:6812 osd.4 since back 2024-03-16T12:09:43.633241+0000 front 2024-03-16T12:09:43.633117+0000 (oldest deadline 2024-03-16T12:10:07.733019+0000) 2024-03-16T12:10:15.534 INFO:journalctl@ceph.osd.8.smithi154.stdout:Mar 16 12:10:15 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-8[55054]: 2024-03-16T12:10:15.286+0000 7f3f065c0700 -1 osd.8 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:19.263068+0000 front 2024-03-16T12:09:19.263120+0000 (oldest deadline 2024-03-16T12:09:44.563033+0000) 2024-03-16T12:10:15.535 INFO:journalctl@ceph.osd.8.smithi154.stdout:Mar 16 12:10:15 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-8[55054]: 2024-03-16T12:10:15.286+0000 7f3f065c0700 -1 osd.8 119 heartbeat_check: no reply from 172.21.15.154:6804 osd.2 since back 2024-03-16T12:09:29.566064+0000 front 2024-03-16T12:09:29.566042+0000 (oldest deadline 2024-03-16T12:09:54.266168+0000) 2024-03-16T12:10:15.535 INFO:journalctl@ceph.osd.8.smithi154.stdout:Mar 16 12:10:15 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-8[55054]: 2024-03-16T12:10:15.286+0000 7f3f065c0700 -1 osd.8 119 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-16T12:09:38.367565+0000 front 2024-03-16T12:09:38.367474+0000 (oldest deadline 2024-03-16T12:10:01.867407+0000) 2024-03-16T12:10:15.535 INFO:journalctl@ceph.osd.8.smithi154.stdout:Mar 16 12:10:15 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-8[55054]: 2024-03-16T12:10:15.286+0000 7f3f065c0700 -1 osd.8 119 heartbeat_check: no reply from 172.21.15.152:6812 osd.4 since back 2024-03-16T12:09:45.269460+0000 front 2024-03-16T12:09:45.269372+0000 (oldest deadline 2024-03-16T12:10:08.169293+0000) 2024-03-16T12:10:15.577 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 16 12:10:15 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-9[67182]: 2024-03-16T12:10:15.089+0000 7fe1cee8b700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:12.704136+0000 front 2024-03-16T12:09:12.703904+0000 (oldest deadline 2024-03-16T12:09:37.404049+0000) 2024-03-16T12:10:15.577 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 16 12:10:15 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-9[67182]: 2024-03-16T12:10:15.089+0000 7fe1cee8b700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:17.404712+0000 front 2024-03-16T12:09:17.404800+0000 (oldest deadline 2024-03-16T12:09:42.704627+0000) 2024-03-16T12:10:15.577 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 16 12:10:15 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-9[67182]: 2024-03-16T12:10:15.089+0000 7fe1cee8b700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.154:6804 osd.2 since back 2024-03-16T12:09:27.405764+0000 front 2024-03-16T12:09:27.405893+0000 (oldest deadline 2024-03-16T12:09:52.105536+0000) 2024-03-16T12:10:15.577 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 16 12:10:15 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-9[67182]: 2024-03-16T12:10:15.089+0000 7fe1cee8b700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-16T12:09:37.907196+0000 front 2024-03-16T12:09:37.907286+0000 (oldest deadline 2024-03-16T12:10:03.207291+0000) 2024-03-16T12:10:15.578 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 16 12:10:15 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-9[67182]: 2024-03-16T12:10:15.089+0000 7fe1cee8b700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.152:6812 osd.4 since back 2024-03-16T12:09:43.207829+0000 front 2024-03-16T12:09:43.207999+0000 (oldest deadline 2024-03-16T12:10:07.907827+0000) 2024-03-16T12:10:16.116 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:10:15 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:10:15.788+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:10.713221+0000 front 2024-03-16T12:09:10.713087+0000 (oldest deadline 2024-03-16T12:09:35.413059+0000) 2024-03-16T12:10:16.117 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:10:15 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:10:15.788+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:20.614694+0000 front 2024-03-16T12:09:20.614671+0000 (oldest deadline 2024-03-16T12:09:45.215535+0000) 2024-03-16T12:10:16.117 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:10:15 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:10:15.788+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.154:6804 osd.2 since back 2024-03-16T12:09:29.916590+0000 front 2024-03-16T12:09:29.916612+0000 (oldest deadline 2024-03-16T12:09:52.816525+0000) 2024-03-16T12:10:16.117 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:10:15 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:10:15.788+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-16T12:09:37.517617+0000 front 2024-03-16T12:09:37.517557+0000 (oldest deadline 2024-03-16T12:10:02.817580+0000) 2024-03-16T12:10:16.117 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:10:15 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:10:15.788+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.152:6812 osd.4 since back 2024-03-16T12:09:42.818002+0000 front 2024-03-16T12:09:42.818046+0000 (oldest deadline 2024-03-16T12:10:07.518142+0000) 2024-03-16T12:10:16.284 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:15 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:15.959+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:13.025675+0000 front 2024-03-16T12:09:13.025847+0000 (oldest deadline 2024-03-16T12:09:35.325827+0000) 2024-03-16T12:10:16.285 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:15 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:15.959+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:19.927745+0000 front 2024-03-16T12:09:19.927761+0000 (oldest deadline 2024-03-16T12:09:45.827828+0000) 2024-03-16T12:10:16.285 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:15 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:15.959+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.154:6804 osd.2 since back 2024-03-16T12:09:28.629320+0000 front 2024-03-16T12:09:28.629357+0000 (oldest deadline 2024-03-16T12:09:52.129472+0000) 2024-03-16T12:10:16.285 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:15 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:15.959+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-16T12:09:38.432099+0000 front 2024-03-16T12:09:38.432072+0000 (oldest deadline 2024-03-16T12:10:01.332006+0000) 2024-03-16T12:10:16.285 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:15 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:15.959+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.152:6812 osd.4 since back 2024-03-16T12:09:43.633241+0000 front 2024-03-16T12:09:43.633117+0000 (oldest deadline 2024-03-16T12:10:07.733019+0000) 2024-03-16T12:10:16.285 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:15 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:15.959+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.154:6812 osd.5 since back 2024-03-16T12:09:53.534564+0000 front 2024-03-16T12:09:53.534519+0000 (oldest deadline 2024-03-16T12:10:15.834516+0000) 2024-03-16T12:10:16.577 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 16 12:10:16 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-9[67182]: 2024-03-16T12:10:16.104+0000 7fe1cee8b700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:12.704136+0000 front 2024-03-16T12:09:12.703904+0000 (oldest deadline 2024-03-16T12:09:37.404049+0000) 2024-03-16T12:10:16.577 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 16 12:10:16 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-9[67182]: 2024-03-16T12:10:16.104+0000 7fe1cee8b700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:17.404712+0000 front 2024-03-16T12:09:17.404800+0000 (oldest deadline 2024-03-16T12:09:42.704627+0000) 2024-03-16T12:10:16.577 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 16 12:10:16 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-9[67182]: 2024-03-16T12:10:16.104+0000 7fe1cee8b700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.154:6804 osd.2 since back 2024-03-16T12:09:27.405764+0000 front 2024-03-16T12:09:27.405893+0000 (oldest deadline 2024-03-16T12:09:52.105536+0000) 2024-03-16T12:10:16.577 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 16 12:10:16 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-9[67182]: 2024-03-16T12:10:16.104+0000 7fe1cee8b700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-16T12:09:37.907196+0000 front 2024-03-16T12:09:37.907286+0000 (oldest deadline 2024-03-16T12:10:03.207291+0000) 2024-03-16T12:10:16.578 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 16 12:10:16 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-9[67182]: 2024-03-16T12:10:16.104+0000 7fe1cee8b700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.152:6812 osd.4 since back 2024-03-16T12:09:43.207829+0000 front 2024-03-16T12:09:43.207999+0000 (oldest deadline 2024-03-16T12:10:07.907827+0000) 2024-03-16T12:10:16.784 INFO:journalctl@ceph.osd.8.smithi154.stdout:Mar 16 12:10:16 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-8[55054]: 2024-03-16T12:10:16.330+0000 7f3f065c0700 -1 osd.8 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:19.263068+0000 front 2024-03-16T12:09:19.263120+0000 (oldest deadline 2024-03-16T12:09:44.563033+0000) 2024-03-16T12:10:16.784 INFO:journalctl@ceph.osd.8.smithi154.stdout:Mar 16 12:10:16 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-8[55054]: 2024-03-16T12:10:16.330+0000 7f3f065c0700 -1 osd.8 119 heartbeat_check: no reply from 172.21.15.154:6804 osd.2 since back 2024-03-16T12:09:29.566064+0000 front 2024-03-16T12:09:29.566042+0000 (oldest deadline 2024-03-16T12:09:54.266168+0000) 2024-03-16T12:10:16.784 INFO:journalctl@ceph.osd.8.smithi154.stdout:Mar 16 12:10:16 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-8[55054]: 2024-03-16T12:10:16.330+0000 7f3f065c0700 -1 osd.8 119 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-16T12:09:38.367565+0000 front 2024-03-16T12:09:38.367474+0000 (oldest deadline 2024-03-16T12:10:01.867407+0000) 2024-03-16T12:10:16.785 INFO:journalctl@ceph.osd.8.smithi154.stdout:Mar 16 12:10:16 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-8[55054]: 2024-03-16T12:10:16.330+0000 7f3f065c0700 -1 osd.8 119 heartbeat_check: no reply from 172.21.15.152:6812 osd.4 since back 2024-03-16T12:09:45.269460+0000 front 2024-03-16T12:09:45.269372+0000 (oldest deadline 2024-03-16T12:10:08.169293+0000) 2024-03-16T12:10:16.785 INFO:journalctl@ceph.osd.8.smithi154.stdout:Mar 16 12:10:16 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-8[55054]: 2024-03-16T12:10:16.330+0000 7f3f065c0700 -1 osd.8 119 heartbeat_check: no reply from 172.21.15.154:6812 osd.5 since back 2024-03-16T12:09:51.670194+0000 front 2024-03-16T12:09:51.670244+0000 (oldest deadline 2024-03-16T12:10:15.770351+0000) 2024-03-16T12:10:17.116 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:10:16 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:10:16.775+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:10.713221+0000 front 2024-03-16T12:09:10.713087+0000 (oldest deadline 2024-03-16T12:09:35.413059+0000) 2024-03-16T12:10:17.116 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:10:16 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:10:16.775+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:20.614694+0000 front 2024-03-16T12:09:20.614671+0000 (oldest deadline 2024-03-16T12:09:45.215535+0000) 2024-03-16T12:10:17.117 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:10:16 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:10:16.775+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.154:6804 osd.2 since back 2024-03-16T12:09:29.916590+0000 front 2024-03-16T12:09:29.916612+0000 (oldest deadline 2024-03-16T12:09:52.816525+0000) 2024-03-16T12:10:17.117 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:10:16 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:10:16.775+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-16T12:09:37.517617+0000 front 2024-03-16T12:09:37.517557+0000 (oldest deadline 2024-03-16T12:10:02.817580+0000) 2024-03-16T12:10:17.117 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:10:16 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:10:16.775+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.152:6812 osd.4 since back 2024-03-16T12:09:42.818002+0000 front 2024-03-16T12:09:42.818046+0000 (oldest deadline 2024-03-16T12:10:07.518142+0000) 2024-03-16T12:10:17.284 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:16 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:16.984+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:13.025675+0000 front 2024-03-16T12:09:13.025847+0000 (oldest deadline 2024-03-16T12:09:35.325827+0000) 2024-03-16T12:10:17.284 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:16 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:16.984+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:19.927745+0000 front 2024-03-16T12:09:19.927761+0000 (oldest deadline 2024-03-16T12:09:45.827828+0000) 2024-03-16T12:10:17.284 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:16 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:16.984+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.154:6804 osd.2 since back 2024-03-16T12:09:28.629320+0000 front 2024-03-16T12:09:28.629357+0000 (oldest deadline 2024-03-16T12:09:52.129472+0000) 2024-03-16T12:10:17.284 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:16 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:16.984+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-16T12:09:38.432099+0000 front 2024-03-16T12:09:38.432072+0000 (oldest deadline 2024-03-16T12:10:01.332006+0000) 2024-03-16T12:10:17.284 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:16 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:16.984+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.152:6812 osd.4 since back 2024-03-16T12:09:43.633241+0000 front 2024-03-16T12:09:43.633117+0000 (oldest deadline 2024-03-16T12:10:07.733019+0000) 2024-03-16T12:10:17.285 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:16 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:16.984+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.154:6812 osd.5 since back 2024-03-16T12:09:53.534564+0000 front 2024-03-16T12:09:53.534519+0000 (oldest deadline 2024-03-16T12:10:15.834516+0000) 2024-03-16T12:10:17.577 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 16 12:10:17 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-9[67182]: 2024-03-16T12:10:17.152+0000 7fe1cee8b700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:12.704136+0000 front 2024-03-16T12:09:12.703904+0000 (oldest deadline 2024-03-16T12:09:37.404049+0000) 2024-03-16T12:10:17.577 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 16 12:10:17 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-9[67182]: 2024-03-16T12:10:17.152+0000 7fe1cee8b700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:17.404712+0000 front 2024-03-16T12:09:17.404800+0000 (oldest deadline 2024-03-16T12:09:42.704627+0000) 2024-03-16T12:10:17.577 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 16 12:10:17 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-9[67182]: 2024-03-16T12:10:17.152+0000 7fe1cee8b700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.154:6804 osd.2 since back 2024-03-16T12:09:27.405764+0000 front 2024-03-16T12:09:27.405893+0000 (oldest deadline 2024-03-16T12:09:52.105536+0000) 2024-03-16T12:10:17.577 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 16 12:10:17 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-9[67182]: 2024-03-16T12:10:17.152+0000 7fe1cee8b700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-16T12:09:37.907196+0000 front 2024-03-16T12:09:37.907286+0000 (oldest deadline 2024-03-16T12:10:03.207291+0000) 2024-03-16T12:10:17.577 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 16 12:10:17 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-9[67182]: 2024-03-16T12:10:17.152+0000 7fe1cee8b700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.152:6812 osd.4 since back 2024-03-16T12:09:43.207829+0000 front 2024-03-16T12:09:43.207999+0000 (oldest deadline 2024-03-16T12:10:07.907827+0000) 2024-03-16T12:10:17.784 INFO:journalctl@ceph.osd.8.smithi154.stdout:Mar 16 12:10:17 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-8[55054]: 2024-03-16T12:10:17.338+0000 7f3f065c0700 -1 osd.8 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:19.263068+0000 front 2024-03-16T12:09:19.263120+0000 (oldest deadline 2024-03-16T12:09:44.563033+0000) 2024-03-16T12:10:17.784 INFO:journalctl@ceph.osd.8.smithi154.stdout:Mar 16 12:10:17 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-8[55054]: 2024-03-16T12:10:17.338+0000 7f3f065c0700 -1 osd.8 119 heartbeat_check: no reply from 172.21.15.154:6804 osd.2 since back 2024-03-16T12:09:29.566064+0000 front 2024-03-16T12:09:29.566042+0000 (oldest deadline 2024-03-16T12:09:54.266168+0000) 2024-03-16T12:10:17.784 INFO:journalctl@ceph.osd.8.smithi154.stdout:Mar 16 12:10:17 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-8[55054]: 2024-03-16T12:10:17.338+0000 7f3f065c0700 -1 osd.8 119 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-16T12:09:38.367565+0000 front 2024-03-16T12:09:38.367474+0000 (oldest deadline 2024-03-16T12:10:01.867407+0000) 2024-03-16T12:10:17.785 INFO:journalctl@ceph.osd.8.smithi154.stdout:Mar 16 12:10:17 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-8[55054]: 2024-03-16T12:10:17.338+0000 7f3f065c0700 -1 osd.8 119 heartbeat_check: no reply from 172.21.15.152:6812 osd.4 since back 2024-03-16T12:09:45.269460+0000 front 2024-03-16T12:09:45.269372+0000 (oldest deadline 2024-03-16T12:10:08.169293+0000) 2024-03-16T12:10:17.785 INFO:journalctl@ceph.osd.8.smithi154.stdout:Mar 16 12:10:17 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-8[55054]: 2024-03-16T12:10:17.338+0000 7f3f065c0700 -1 osd.8 119 heartbeat_check: no reply from 172.21.15.154:6812 osd.5 since back 2024-03-16T12:09:51.670194+0000 front 2024-03-16T12:09:51.670244+0000 (oldest deadline 2024-03-16T12:10:15.770351+0000) 2024-03-16T12:10:18.116 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:10:17 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:10:17.751+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:10.713221+0000 front 2024-03-16T12:09:10.713087+0000 (oldest deadline 2024-03-16T12:09:35.413059+0000) 2024-03-16T12:10:18.116 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:10:17 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:10:17.751+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:20.614694+0000 front 2024-03-16T12:09:20.614671+0000 (oldest deadline 2024-03-16T12:09:45.215535+0000) 2024-03-16T12:10:18.116 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:10:17 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:10:17.751+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.154:6804 osd.2 since back 2024-03-16T12:09:29.916590+0000 front 2024-03-16T12:09:29.916612+0000 (oldest deadline 2024-03-16T12:09:52.816525+0000) 2024-03-16T12:10:18.117 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:10:17 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:10:17.752+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-16T12:09:37.517617+0000 front 2024-03-16T12:09:37.517557+0000 (oldest deadline 2024-03-16T12:10:02.817580+0000) 2024-03-16T12:10:18.117 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:10:17 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:10:17.752+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.152:6812 osd.4 since back 2024-03-16T12:09:42.818002+0000 front 2024-03-16T12:09:42.818046+0000 (oldest deadline 2024-03-16T12:10:07.518142+0000) 2024-03-16T12:10:18.284 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:17 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:17.982+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:13.025675+0000 front 2024-03-16T12:09:13.025847+0000 (oldest deadline 2024-03-16T12:09:35.325827+0000) 2024-03-16T12:10:18.284 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:17 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:17.982+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:19.927745+0000 front 2024-03-16T12:09:19.927761+0000 (oldest deadline 2024-03-16T12:09:45.827828+0000) 2024-03-16T12:10:18.284 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:17 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:17.982+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.154:6804 osd.2 since back 2024-03-16T12:09:28.629320+0000 front 2024-03-16T12:09:28.629357+0000 (oldest deadline 2024-03-16T12:09:52.129472+0000) 2024-03-16T12:10:18.284 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:17 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:17.982+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-16T12:09:38.432099+0000 front 2024-03-16T12:09:38.432072+0000 (oldest deadline 2024-03-16T12:10:01.332006+0000) 2024-03-16T12:10:18.284 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:17 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:17.982+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.152:6812 osd.4 since back 2024-03-16T12:09:43.633241+0000 front 2024-03-16T12:09:43.633117+0000 (oldest deadline 2024-03-16T12:10:07.733019+0000) 2024-03-16T12:10:18.285 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:17 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:17.982+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.154:6812 osd.5 since back 2024-03-16T12:09:53.534564+0000 front 2024-03-16T12:09:53.534519+0000 (oldest deadline 2024-03-16T12:10:15.834516+0000) 2024-03-16T12:10:18.577 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 16 12:10:18 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-9[67182]: 2024-03-16T12:10:18.202+0000 7fe1cee8b700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:12.704136+0000 front 2024-03-16T12:09:12.703904+0000 (oldest deadline 2024-03-16T12:09:37.404049+0000) 2024-03-16T12:10:18.577 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 16 12:10:18 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-9[67182]: 2024-03-16T12:10:18.202+0000 7fe1cee8b700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:17.404712+0000 front 2024-03-16T12:09:17.404800+0000 (oldest deadline 2024-03-16T12:09:42.704627+0000) 2024-03-16T12:10:18.577 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 16 12:10:18 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-9[67182]: 2024-03-16T12:10:18.202+0000 7fe1cee8b700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.154:6804 osd.2 since back 2024-03-16T12:09:27.405764+0000 front 2024-03-16T12:09:27.405893+0000 (oldest deadline 2024-03-16T12:09:52.105536+0000) 2024-03-16T12:10:18.577 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 16 12:10:18 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-9[67182]: 2024-03-16T12:10:18.202+0000 7fe1cee8b700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-16T12:09:37.907196+0000 front 2024-03-16T12:09:37.907286+0000 (oldest deadline 2024-03-16T12:10:03.207291+0000) 2024-03-16T12:10:18.577 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 16 12:10:18 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-9[67182]: 2024-03-16T12:10:18.202+0000 7fe1cee8b700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.152:6812 osd.4 since back 2024-03-16T12:09:43.207829+0000 front 2024-03-16T12:09:43.207999+0000 (oldest deadline 2024-03-16T12:10:07.907827+0000) 2024-03-16T12:10:18.578 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 16 12:10:18 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-9[67182]: 2024-03-16T12:10:18.202+0000 7fe1cee8b700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.154:6812 osd.5 since back 2024-03-16T12:09:52.310701+0000 front 2024-03-16T12:09:52.310491+0000 (oldest deadline 2024-03-16T12:10:17.610385+0000) 2024-03-16T12:10:18.784 INFO:journalctl@ceph.osd.8.smithi154.stdout:Mar 16 12:10:18 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-8[55054]: 2024-03-16T12:10:18.355+0000 7f3f065c0700 -1 osd.8 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:19.263068+0000 front 2024-03-16T12:09:19.263120+0000 (oldest deadline 2024-03-16T12:09:44.563033+0000) 2024-03-16T12:10:18.784 INFO:journalctl@ceph.osd.8.smithi154.stdout:Mar 16 12:10:18 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-8[55054]: 2024-03-16T12:10:18.355+0000 7f3f065c0700 -1 osd.8 119 heartbeat_check: no reply from 172.21.15.154:6804 osd.2 since back 2024-03-16T12:09:29.566064+0000 front 2024-03-16T12:09:29.566042+0000 (oldest deadline 2024-03-16T12:09:54.266168+0000) 2024-03-16T12:10:18.784 INFO:journalctl@ceph.osd.8.smithi154.stdout:Mar 16 12:10:18 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-8[55054]: 2024-03-16T12:10:18.355+0000 7f3f065c0700 -1 osd.8 119 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-16T12:09:38.367565+0000 front 2024-03-16T12:09:38.367474+0000 (oldest deadline 2024-03-16T12:10:01.867407+0000) 2024-03-16T12:10:18.784 INFO:journalctl@ceph.osd.8.smithi154.stdout:Mar 16 12:10:18 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-8[55054]: 2024-03-16T12:10:18.355+0000 7f3f065c0700 -1 osd.8 119 heartbeat_check: no reply from 172.21.15.152:6812 osd.4 since back 2024-03-16T12:09:45.269460+0000 front 2024-03-16T12:09:45.269372+0000 (oldest deadline 2024-03-16T12:10:08.169293+0000) 2024-03-16T12:10:18.784 INFO:journalctl@ceph.osd.8.smithi154.stdout:Mar 16 12:10:18 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-8[55054]: 2024-03-16T12:10:18.355+0000 7f3f065c0700 -1 osd.8 119 heartbeat_check: no reply from 172.21.15.154:6812 osd.5 since back 2024-03-16T12:09:51.670194+0000 front 2024-03-16T12:09:51.670244+0000 (oldest deadline 2024-03-16T12:10:15.770351+0000) 2024-03-16T12:10:19.116 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:10:18 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:10:18.770+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:10.713221+0000 front 2024-03-16T12:09:10.713087+0000 (oldest deadline 2024-03-16T12:09:35.413059+0000) 2024-03-16T12:10:19.116 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:10:18 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:10:18.770+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:20.614694+0000 front 2024-03-16T12:09:20.614671+0000 (oldest deadline 2024-03-16T12:09:45.215535+0000) 2024-03-16T12:10:19.117 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:10:18 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:10:18.770+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.154:6804 osd.2 since back 2024-03-16T12:09:29.916590+0000 front 2024-03-16T12:09:29.916612+0000 (oldest deadline 2024-03-16T12:09:52.816525+0000) 2024-03-16T12:10:19.117 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:10:18 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:10:18.770+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-16T12:09:37.517617+0000 front 2024-03-16T12:09:37.517557+0000 (oldest deadline 2024-03-16T12:10:02.817580+0000) 2024-03-16T12:10:19.117 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:10:18 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:10:18.770+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.152:6812 osd.4 since back 2024-03-16T12:09:42.818002+0000 front 2024-03-16T12:09:42.818046+0000 (oldest deadline 2024-03-16T12:10:07.518142+0000) 2024-03-16T12:10:19.284 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:19 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:19.020+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:13.025675+0000 front 2024-03-16T12:09:13.025847+0000 (oldest deadline 2024-03-16T12:09:35.325827+0000) 2024-03-16T12:10:19.284 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:19 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:19.020+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:19.927745+0000 front 2024-03-16T12:09:19.927761+0000 (oldest deadline 2024-03-16T12:09:45.827828+0000) 2024-03-16T12:10:19.284 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:19 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:19.020+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.154:6804 osd.2 since back 2024-03-16T12:09:28.629320+0000 front 2024-03-16T12:09:28.629357+0000 (oldest deadline 2024-03-16T12:09:52.129472+0000) 2024-03-16T12:10:19.284 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:19 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:19.020+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-16T12:09:38.432099+0000 front 2024-03-16T12:09:38.432072+0000 (oldest deadline 2024-03-16T12:10:01.332006+0000) 2024-03-16T12:10:19.284 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:19 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:19.020+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.152:6812 osd.4 since back 2024-03-16T12:09:43.633241+0000 front 2024-03-16T12:09:43.633117+0000 (oldest deadline 2024-03-16T12:10:07.733019+0000) 2024-03-16T12:10:19.285 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:19 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:19.020+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.154:6812 osd.5 since back 2024-03-16T12:09:53.534564+0000 front 2024-03-16T12:09:53.534519+0000 (oldest deadline 2024-03-16T12:10:15.834516+0000) 2024-03-16T12:10:19.577 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 16 12:10:19 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-9[67182]: 2024-03-16T12:10:19.162+0000 7fe1cee8b700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:12.704136+0000 front 2024-03-16T12:09:12.703904+0000 (oldest deadline 2024-03-16T12:09:37.404049+0000) 2024-03-16T12:10:19.577 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 16 12:10:19 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-9[67182]: 2024-03-16T12:10:19.162+0000 7fe1cee8b700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:17.404712+0000 front 2024-03-16T12:09:17.404800+0000 (oldest deadline 2024-03-16T12:09:42.704627+0000) 2024-03-16T12:10:19.577 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 16 12:10:19 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-9[67182]: 2024-03-16T12:10:19.162+0000 7fe1cee8b700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.154:6804 osd.2 since back 2024-03-16T12:09:27.405764+0000 front 2024-03-16T12:09:27.405893+0000 (oldest deadline 2024-03-16T12:09:52.105536+0000) 2024-03-16T12:10:19.577 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 16 12:10:19 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-9[67182]: 2024-03-16T12:10:19.162+0000 7fe1cee8b700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-16T12:09:37.907196+0000 front 2024-03-16T12:09:37.907286+0000 (oldest deadline 2024-03-16T12:10:03.207291+0000) 2024-03-16T12:10:19.577 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 16 12:10:19 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-9[67182]: 2024-03-16T12:10:19.162+0000 7fe1cee8b700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.152:6812 osd.4 since back 2024-03-16T12:09:43.207829+0000 front 2024-03-16T12:09:43.207999+0000 (oldest deadline 2024-03-16T12:10:07.907827+0000) 2024-03-16T12:10:19.577 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 16 12:10:19 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-9[67182]: 2024-03-16T12:10:19.162+0000 7fe1cee8b700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.154:6812 osd.5 since back 2024-03-16T12:09:52.310701+0000 front 2024-03-16T12:09:52.310491+0000 (oldest deadline 2024-03-16T12:10:17.610385+0000) 2024-03-16T12:10:19.784 INFO:journalctl@ceph.osd.8.smithi154.stdout:Mar 16 12:10:19 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-8[55054]: 2024-03-16T12:10:19.393+0000 7f3f065c0700 -1 osd.8 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:19.263068+0000 front 2024-03-16T12:09:19.263120+0000 (oldest deadline 2024-03-16T12:09:44.563033+0000) 2024-03-16T12:10:19.784 INFO:journalctl@ceph.osd.8.smithi154.stdout:Mar 16 12:10:19 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-8[55054]: 2024-03-16T12:10:19.393+0000 7f3f065c0700 -1 osd.8 119 heartbeat_check: no reply from 172.21.15.154:6804 osd.2 since back 2024-03-16T12:09:29.566064+0000 front 2024-03-16T12:09:29.566042+0000 (oldest deadline 2024-03-16T12:09:54.266168+0000) 2024-03-16T12:10:19.785 INFO:journalctl@ceph.osd.8.smithi154.stdout:Mar 16 12:10:19 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-8[55054]: 2024-03-16T12:10:19.393+0000 7f3f065c0700 -1 osd.8 119 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-16T12:09:38.367565+0000 front 2024-03-16T12:09:38.367474+0000 (oldest deadline 2024-03-16T12:10:01.867407+0000) 2024-03-16T12:10:19.785 INFO:journalctl@ceph.osd.8.smithi154.stdout:Mar 16 12:10:19 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-8[55054]: 2024-03-16T12:10:19.393+0000 7f3f065c0700 -1 osd.8 119 heartbeat_check: no reply from 172.21.15.152:6812 osd.4 since back 2024-03-16T12:09:45.269460+0000 front 2024-03-16T12:09:45.269372+0000 (oldest deadline 2024-03-16T12:10:08.169293+0000) 2024-03-16T12:10:19.785 INFO:journalctl@ceph.osd.8.smithi154.stdout:Mar 16 12:10:19 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-8[55054]: 2024-03-16T12:10:19.393+0000 7f3f065c0700 -1 osd.8 119 heartbeat_check: no reply from 172.21.15.154:6812 osd.5 since back 2024-03-16T12:09:51.670194+0000 front 2024-03-16T12:09:51.670244+0000 (oldest deadline 2024-03-16T12:10:15.770351+0000) 2024-03-16T12:10:20.116 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:10:19 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:10:19.771+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:10.713221+0000 front 2024-03-16T12:09:10.713087+0000 (oldest deadline 2024-03-16T12:09:35.413059+0000) 2024-03-16T12:10:20.116 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:10:19 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:10:19.771+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:20.614694+0000 front 2024-03-16T12:09:20.614671+0000 (oldest deadline 2024-03-16T12:09:45.215535+0000) 2024-03-16T12:10:20.116 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:10:19 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:10:19.771+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.154:6804 osd.2 since back 2024-03-16T12:09:29.916590+0000 front 2024-03-16T12:09:29.916612+0000 (oldest deadline 2024-03-16T12:09:52.816525+0000) 2024-03-16T12:10:20.117 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:10:19 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:10:19.771+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-16T12:09:37.517617+0000 front 2024-03-16T12:09:37.517557+0000 (oldest deadline 2024-03-16T12:10:02.817580+0000) 2024-03-16T12:10:20.117 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:10:19 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:10:19.771+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.152:6812 osd.4 since back 2024-03-16T12:09:42.818002+0000 front 2024-03-16T12:09:42.818046+0000 (oldest deadline 2024-03-16T12:10:07.518142+0000) 2024-03-16T12:10:20.248 INFO:journalctl@ceph.osd.8.smithi154.stdout:Mar 16 12:10:19 smithi154 podman[75338]: 2024-03-16 12:10:19.953325389 +0000 UTC m=+5.596192937 container died 0b9a3c71e8c84019cc508406775d653795d0cefdda33f63d15543cc6244f99af (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-8, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-16T12:10:20.248 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:20 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:20.064+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:13.025675+0000 front 2024-03-16T12:09:13.025847+0000 (oldest deadline 2024-03-16T12:09:35.325827+0000) 2024-03-16T12:10:20.248 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:20 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:20.064+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:19.927745+0000 front 2024-03-16T12:09:19.927761+0000 (oldest deadline 2024-03-16T12:09:45.827828+0000) 2024-03-16T12:10:20.248 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:20 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:20.064+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.154:6804 osd.2 since back 2024-03-16T12:09:28.629320+0000 front 2024-03-16T12:09:28.629357+0000 (oldest deadline 2024-03-16T12:09:52.129472+0000) 2024-03-16T12:10:20.249 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:20 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:20.064+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-16T12:09:38.432099+0000 front 2024-03-16T12:09:38.432072+0000 (oldest deadline 2024-03-16T12:10:01.332006+0000) 2024-03-16T12:10:20.249 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:20 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:20.064+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.152:6812 osd.4 since back 2024-03-16T12:09:43.633241+0000 front 2024-03-16T12:09:43.633117+0000 (oldest deadline 2024-03-16T12:10:07.733019+0000) 2024-03-16T12:10:20.249 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:20 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:20.064+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.154:6812 osd.5 since back 2024-03-16T12:09:53.534564+0000 front 2024-03-16T12:09:53.534519+0000 (oldest deadline 2024-03-16T12:10:15.834516+0000) 2024-03-16T12:10:20.534 INFO:journalctl@ceph.osd.8.smithi154.stdout:Mar 16 12:10:20 smithi154 podman[75338]: 2024-03-16 12:10:20.359320382 +0000 UTC m=+6.002187995 container cleanup 0b9a3c71e8c84019cc508406775d653795d0cefdda33f63d15543cc6244f99af (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-8, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240307) 2024-03-16T12:10:20.534 INFO:journalctl@ceph.osd.8.smithi154.stdout:Mar 16 12:10:20 smithi154 bash[75338]: ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-8 2024-03-16T12:10:20.576 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 16 12:10:20 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-9[67182]: 2024-03-16T12:10:20.140+0000 7fe1cee8b700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:12.704136+0000 front 2024-03-16T12:09:12.703904+0000 (oldest deadline 2024-03-16T12:09:37.404049+0000) 2024-03-16T12:10:20.577 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 16 12:10:20 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-9[67182]: 2024-03-16T12:10:20.140+0000 7fe1cee8b700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:17.404712+0000 front 2024-03-16T12:09:17.404800+0000 (oldest deadline 2024-03-16T12:09:42.704627+0000) 2024-03-16T12:10:20.577 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 16 12:10:20 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-9[67182]: 2024-03-16T12:10:20.140+0000 7fe1cee8b700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.154:6804 osd.2 since back 2024-03-16T12:09:27.405764+0000 front 2024-03-16T12:09:27.405893+0000 (oldest deadline 2024-03-16T12:09:52.105536+0000) 2024-03-16T12:10:20.577 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 16 12:10:20 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-9[67182]: 2024-03-16T12:10:20.140+0000 7fe1cee8b700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-16T12:09:37.907196+0000 front 2024-03-16T12:09:37.907286+0000 (oldest deadline 2024-03-16T12:10:03.207291+0000) 2024-03-16T12:10:20.577 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 16 12:10:20 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-9[67182]: 2024-03-16T12:10:20.140+0000 7fe1cee8b700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.152:6812 osd.4 since back 2024-03-16T12:09:43.207829+0000 front 2024-03-16T12:09:43.207999+0000 (oldest deadline 2024-03-16T12:10:07.907827+0000) 2024-03-16T12:10:20.577 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 16 12:10:20 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-9[67182]: 2024-03-16T12:10:20.140+0000 7fe1cee8b700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.154:6812 osd.5 since back 2024-03-16T12:09:52.310701+0000 front 2024-03-16T12:09:52.310491+0000 (oldest deadline 2024-03-16T12:10:17.610385+0000) 2024-03-16T12:10:20.950 INFO:journalctl@ceph.osd.8.smithi154.stdout:Mar 16 12:10:20 smithi154 podman[75366]: 2024-03-16 12:10:20.6191442 +0000 UTC m=+0.666968810 container remove 0b9a3c71e8c84019cc508406775d653795d0cefdda33f63d15543cc6244f99af (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-8, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-03-16T12:10:21.116 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:10:20 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:10:20.766+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:10.713221+0000 front 2024-03-16T12:09:10.713087+0000 (oldest deadline 2024-03-16T12:09:35.413059+0000) 2024-03-16T12:10:21.116 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:10:20 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:10:20.766+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:20.614694+0000 front 2024-03-16T12:09:20.614671+0000 (oldest deadline 2024-03-16T12:09:45.215535+0000) 2024-03-16T12:10:21.117 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:10:20 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:10:20.766+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.154:6804 osd.2 since back 2024-03-16T12:09:29.916590+0000 front 2024-03-16T12:09:29.916612+0000 (oldest deadline 2024-03-16T12:09:52.816525+0000) 2024-03-16T12:10:21.117 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:10:20 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:10:20.766+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-16T12:09:37.517617+0000 front 2024-03-16T12:09:37.517557+0000 (oldest deadline 2024-03-16T12:10:02.817580+0000) 2024-03-16T12:10:21.117 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:10:20 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:10:20.766+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.152:6812 osd.4 since back 2024-03-16T12:09:42.818002+0000 front 2024-03-16T12:09:42.818046+0000 (oldest deadline 2024-03-16T12:10:07.518142+0000) 2024-03-16T12:10:21.117 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:10:20 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:10:20.766+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.154:6812 osd.5 since back 2024-03-16T12:09:53.919738+0000 front 2024-03-16T12:09:53.919614+0000 (oldest deadline 2024-03-16T12:10:19.819647+0000) 2024-03-16T12:10:21.254 INFO:journalctl@ceph.osd.8.smithi154.stdout:Mar 16 12:10:20 smithi154 podman[75457]: 2024-03-16 12:10:20.865948712 +0000 UTC m=+0.017757363 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-16T12:10:21.254 INFO:journalctl@ceph.osd.8.smithi154.stdout:Mar 16 12:10:21 smithi154 podman[75457]: 2024-03-16 12:10:21.003149819 +0000 UTC m=+0.154958459 container create e2ca37cccbf7f031f818a42bc7ca33a3773c7eaf5773ad9bc709b9befe27e647 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-8-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-03-16T12:10:21.254 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:21 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:21.104+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:13.025675+0000 front 2024-03-16T12:09:13.025847+0000 (oldest deadline 2024-03-16T12:09:35.325827+0000) 2024-03-16T12:10:21.254 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:21 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:21.104+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:19.927745+0000 front 2024-03-16T12:09:19.927761+0000 (oldest deadline 2024-03-16T12:09:45.827828+0000) 2024-03-16T12:10:21.254 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:21 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:21.104+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.154:6804 osd.2 since back 2024-03-16T12:09:28.629320+0000 front 2024-03-16T12:09:28.629357+0000 (oldest deadline 2024-03-16T12:09:52.129472+0000) 2024-03-16T12:10:21.255 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:21 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:21.104+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-16T12:09:38.432099+0000 front 2024-03-16T12:09:38.432072+0000 (oldest deadline 2024-03-16T12:10:01.332006+0000) 2024-03-16T12:10:21.255 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:21 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:21.104+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.152:6812 osd.4 since back 2024-03-16T12:09:43.633241+0000 front 2024-03-16T12:09:43.633117+0000 (oldest deadline 2024-03-16T12:10:07.733019+0000) 2024-03-16T12:10:21.255 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:21 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:21.104+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.154:6812 osd.5 since back 2024-03-16T12:09:53.534564+0000 front 2024-03-16T12:09:53.534519+0000 (oldest deadline 2024-03-16T12:10:15.834516+0000) 2024-03-16T12:10:21.534 INFO:journalctl@ceph.osd.8.smithi154.stdout:Mar 16 12:10:21 smithi154 podman[75457]: 2024-03-16 12:10:21.266221321 +0000 UTC m=+0.418029977 container init e2ca37cccbf7f031f818a42bc7ca33a3773c7eaf5773ad9bc709b9befe27e647 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-8-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-03-16T12:10:21.534 INFO:journalctl@ceph.osd.8.smithi154.stdout:Mar 16 12:10:21 smithi154 podman[75457]: 2024-03-16 12:10:21.270091835 +0000 UTC m=+0.421900476 container start e2ca37cccbf7f031f818a42bc7ca33a3773c7eaf5773ad9bc709b9befe27e647 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-8-deactivate, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240307, GIT_CLEAN=True) 2024-03-16T12:10:21.534 INFO:journalctl@ceph.osd.8.smithi154.stdout:Mar 16 12:10:21 smithi154 podman[75457]: 2024-03-16 12:10:21.374582455 +0000 UTC m=+0.526391116 container attach e2ca37cccbf7f031f818a42bc7ca33a3773c7eaf5773ad9bc709b9befe27e647 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-8-deactivate, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.license=GPLv2) 2024-03-16T12:10:21.577 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 16 12:10:21 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-9[67182]: 2024-03-16T12:10:21.116+0000 7fe1cee8b700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:12.704136+0000 front 2024-03-16T12:09:12.703904+0000 (oldest deadline 2024-03-16T12:09:37.404049+0000) 2024-03-16T12:10:21.577 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 16 12:10:21 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-9[67182]: 2024-03-16T12:10:21.116+0000 7fe1cee8b700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:17.404712+0000 front 2024-03-16T12:09:17.404800+0000 (oldest deadline 2024-03-16T12:09:42.704627+0000) 2024-03-16T12:10:21.577 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 16 12:10:21 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-9[67182]: 2024-03-16T12:10:21.116+0000 7fe1cee8b700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.154:6804 osd.2 since back 2024-03-16T12:09:27.405764+0000 front 2024-03-16T12:09:27.405893+0000 (oldest deadline 2024-03-16T12:09:52.105536+0000) 2024-03-16T12:10:21.577 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 16 12:10:21 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-9[67182]: 2024-03-16T12:10:21.116+0000 7fe1cee8b700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-16T12:09:37.907196+0000 front 2024-03-16T12:09:37.907286+0000 (oldest deadline 2024-03-16T12:10:03.207291+0000) 2024-03-16T12:10:21.577 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 16 12:10:21 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-9[67182]: 2024-03-16T12:10:21.116+0000 7fe1cee8b700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.152:6812 osd.4 since back 2024-03-16T12:09:43.207829+0000 front 2024-03-16T12:09:43.207999+0000 (oldest deadline 2024-03-16T12:10:07.907827+0000) 2024-03-16T12:10:21.577 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 16 12:10:21 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-9[67182]: 2024-03-16T12:10:21.116+0000 7fe1cee8b700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.154:6812 osd.5 since back 2024-03-16T12:09:52.310701+0000 front 2024-03-16T12:09:52.310491+0000 (oldest deadline 2024-03-16T12:10:17.610385+0000) 2024-03-16T12:10:21.856 INFO:journalctl@ceph.osd.8.smithi154.stdout:Mar 16 12:10:21 smithi154 podman[75457]: 2024-03-16 12:10:21.560387532 +0000 UTC m=+0.712196177 container died e2ca37cccbf7f031f818a42bc7ca33a3773c7eaf5773ad9bc709b9befe27e647 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-8-deactivate, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux ) 2024-03-16T12:10:22.116 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:10:21 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:10:21.806+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:10.713221+0000 front 2024-03-16T12:09:10.713087+0000 (oldest deadline 2024-03-16T12:09:35.413059+0000) 2024-03-16T12:10:22.117 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:10:21 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:10:21.806+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:20.614694+0000 front 2024-03-16T12:09:20.614671+0000 (oldest deadline 2024-03-16T12:09:45.215535+0000) 2024-03-16T12:10:22.117 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:10:21 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:10:21.806+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.154:6804 osd.2 since back 2024-03-16T12:09:29.916590+0000 front 2024-03-16T12:09:29.916612+0000 (oldest deadline 2024-03-16T12:09:52.816525+0000) 2024-03-16T12:10:22.117 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:10:21 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:10:21.806+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-16T12:09:37.517617+0000 front 2024-03-16T12:09:37.517557+0000 (oldest deadline 2024-03-16T12:10:02.817580+0000) 2024-03-16T12:10:22.117 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:10:21 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:10:21.806+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.152:6812 osd.4 since back 2024-03-16T12:09:42.818002+0000 front 2024-03-16T12:09:42.818046+0000 (oldest deadline 2024-03-16T12:10:07.518142+0000) 2024-03-16T12:10:22.117 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:10:21 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:10:21.806+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.154:6812 osd.5 since back 2024-03-16T12:09:53.919738+0000 front 2024-03-16T12:09:53.919614+0000 (oldest deadline 2024-03-16T12:10:19.819647+0000) 2024-03-16T12:10:22.165 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:22 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:22.066+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:13.025675+0000 front 2024-03-16T12:09:13.025847+0000 (oldest deadline 2024-03-16T12:09:35.325827+0000) 2024-03-16T12:10:22.165 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:22 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:22.066+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:19.927745+0000 front 2024-03-16T12:09:19.927761+0000 (oldest deadline 2024-03-16T12:09:45.827828+0000) 2024-03-16T12:10:22.166 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:22 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:22.066+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.154:6804 osd.2 since back 2024-03-16T12:09:28.629320+0000 front 2024-03-16T12:09:28.629357+0000 (oldest deadline 2024-03-16T12:09:52.129472+0000) 2024-03-16T12:10:22.166 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:22 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:22.066+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-16T12:09:38.432099+0000 front 2024-03-16T12:09:38.432072+0000 (oldest deadline 2024-03-16T12:10:01.332006+0000) 2024-03-16T12:10:22.166 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:22 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:22.066+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.152:6812 osd.4 since back 2024-03-16T12:09:43.633241+0000 front 2024-03-16T12:09:43.633117+0000 (oldest deadline 2024-03-16T12:10:07.733019+0000) 2024-03-16T12:10:22.166 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:22 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:22.066+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.154:6812 osd.5 since back 2024-03-16T12:09:53.534564+0000 front 2024-03-16T12:09:53.534519+0000 (oldest deadline 2024-03-16T12:10:15.834516+0000) 2024-03-16T12:10:22.191 DEBUG:teuthology.orchestra.run.smithi154:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d1887f28-e389-11ee-95c9-87774f69a715@osd.8.service' 2024-03-16T12:10:22.232 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-16T12:10:22.233 INFO:tasks.cephadm.osd.8:Stopped osd.8 2024-03-16T12:10:22.233 INFO:tasks.cephadm.osd.9:Stopping osd.9... 2024-03-16T12:10:22.233 DEBUG:teuthology.orchestra.run.smithi029:> sudo systemctl stop ceph-d1887f28-e389-11ee-95c9-87774f69a715@osd.9 2024-03-16T12:10:22.538 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 16 12:10:22 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-9[67182]: 2024-03-16T12:10:22.139+0000 7fe1cee8b700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:12.704136+0000 front 2024-03-16T12:09:12.703904+0000 (oldest deadline 2024-03-16T12:09:37.404049+0000) 2024-03-16T12:10:22.538 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 16 12:10:22 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-9[67182]: 2024-03-16T12:10:22.139+0000 7fe1cee8b700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:17.404712+0000 front 2024-03-16T12:09:17.404800+0000 (oldest deadline 2024-03-16T12:09:42.704627+0000) 2024-03-16T12:10:22.538 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 16 12:10:22 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-9[67182]: 2024-03-16T12:10:22.139+0000 7fe1cee8b700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.154:6804 osd.2 since back 2024-03-16T12:09:27.405764+0000 front 2024-03-16T12:09:27.405893+0000 (oldest deadline 2024-03-16T12:09:52.105536+0000) 2024-03-16T12:10:22.538 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 16 12:10:22 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-9[67182]: 2024-03-16T12:10:22.139+0000 7fe1cee8b700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-16T12:09:37.907196+0000 front 2024-03-16T12:09:37.907286+0000 (oldest deadline 2024-03-16T12:10:03.207291+0000) 2024-03-16T12:10:22.538 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 16 12:10:22 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-9[67182]: 2024-03-16T12:10:22.139+0000 7fe1cee8b700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.152:6812 osd.4 since back 2024-03-16T12:09:43.207829+0000 front 2024-03-16T12:09:43.207999+0000 (oldest deadline 2024-03-16T12:10:07.907827+0000) 2024-03-16T12:10:22.538 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 16 12:10:22 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-9[67182]: 2024-03-16T12:10:22.139+0000 7fe1cee8b700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.154:6812 osd.5 since back 2024-03-16T12:09:52.310701+0000 front 2024-03-16T12:09:52.310491+0000 (oldest deadline 2024-03-16T12:10:17.610385+0000) 2024-03-16T12:10:22.539 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 16 12:10:22 smithi029 systemd[1]: Stopping Ceph osd.9 for d1887f28-e389-11ee-95c9-87774f69a715... 2024-03-16T12:10:22.539 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 16 12:10:22 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-9[67182]: 2024-03-16T12:10:22.549+0000 7fe1d2897700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.9 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-16T12:10:22.539 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 16 12:10:22 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-9[67182]: 2024-03-16T12:10:22.549+0000 7fe1d2897700 -1 osd.9 119 *** Got signal Terminated *** 2024-03-16T12:10:22.539 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 16 12:10:22 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-9[67182]: 2024-03-16T12:10:22.549+0000 7fe1d2897700 -1 osd.9 119 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-16T12:10:23.116 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:10:22 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:10:22.835+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:10.713221+0000 front 2024-03-16T12:09:10.713087+0000 (oldest deadline 2024-03-16T12:09:35.413059+0000) 2024-03-16T12:10:23.117 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:10:22 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:10:22.835+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:20.614694+0000 front 2024-03-16T12:09:20.614671+0000 (oldest deadline 2024-03-16T12:09:45.215535+0000) 2024-03-16T12:10:23.117 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:10:22 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:10:22.835+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.154:6804 osd.2 since back 2024-03-16T12:09:29.916590+0000 front 2024-03-16T12:09:29.916612+0000 (oldest deadline 2024-03-16T12:09:52.816525+0000) 2024-03-16T12:10:23.117 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:10:22 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:10:22.835+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-16T12:09:37.517617+0000 front 2024-03-16T12:09:37.517557+0000 (oldest deadline 2024-03-16T12:10:02.817580+0000) 2024-03-16T12:10:23.117 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:10:22 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:10:22.835+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.152:6812 osd.4 since back 2024-03-16T12:09:42.818002+0000 front 2024-03-16T12:09:42.818046+0000 (oldest deadline 2024-03-16T12:10:07.518142+0000) 2024-03-16T12:10:23.117 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:10:22 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:10:22.835+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.154:6812 osd.5 since back 2024-03-16T12:09:53.919738+0000 front 2024-03-16T12:09:53.919614+0000 (oldest deadline 2024-03-16T12:10:19.819647+0000) 2024-03-16T12:10:23.534 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:23 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:23.074+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:13.025675+0000 front 2024-03-16T12:09:13.025847+0000 (oldest deadline 2024-03-16T12:09:35.325827+0000) 2024-03-16T12:10:23.534 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:23 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:23.074+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:19.927745+0000 front 2024-03-16T12:09:19.927761+0000 (oldest deadline 2024-03-16T12:09:45.827828+0000) 2024-03-16T12:10:23.534 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:23 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:23.074+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.154:6804 osd.2 since back 2024-03-16T12:09:28.629320+0000 front 2024-03-16T12:09:28.629357+0000 (oldest deadline 2024-03-16T12:09:52.129472+0000) 2024-03-16T12:10:23.534 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:23 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:23.074+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-16T12:09:38.432099+0000 front 2024-03-16T12:09:38.432072+0000 (oldest deadline 2024-03-16T12:10:01.332006+0000) 2024-03-16T12:10:23.534 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:23 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:23.074+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.152:6812 osd.4 since back 2024-03-16T12:09:43.633241+0000 front 2024-03-16T12:09:43.633117+0000 (oldest deadline 2024-03-16T12:10:07.733019+0000) 2024-03-16T12:10:23.535 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:23 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:23.074+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.154:6812 osd.5 since back 2024-03-16T12:09:53.534564+0000 front 2024-03-16T12:09:53.534519+0000 (oldest deadline 2024-03-16T12:10:15.834516+0000) 2024-03-16T12:10:23.577 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 16 12:10:23 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-9[67182]: 2024-03-16T12:10:23.186+0000 7fe1cee8b700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:12.704136+0000 front 2024-03-16T12:09:12.703904+0000 (oldest deadline 2024-03-16T12:09:37.404049+0000) 2024-03-16T12:10:23.577 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 16 12:10:23 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-9[67182]: 2024-03-16T12:10:23.186+0000 7fe1cee8b700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:17.404712+0000 front 2024-03-16T12:09:17.404800+0000 (oldest deadline 2024-03-16T12:09:42.704627+0000) 2024-03-16T12:10:23.577 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 16 12:10:23 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-9[67182]: 2024-03-16T12:10:23.186+0000 7fe1cee8b700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.154:6804 osd.2 since back 2024-03-16T12:09:27.405764+0000 front 2024-03-16T12:09:27.405893+0000 (oldest deadline 2024-03-16T12:09:52.105536+0000) 2024-03-16T12:10:23.577 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 16 12:10:23 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-9[67182]: 2024-03-16T12:10:23.186+0000 7fe1cee8b700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-16T12:09:37.907196+0000 front 2024-03-16T12:09:37.907286+0000 (oldest deadline 2024-03-16T12:10:03.207291+0000) 2024-03-16T12:10:23.577 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 16 12:10:23 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-9[67182]: 2024-03-16T12:10:23.186+0000 7fe1cee8b700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.152:6812 osd.4 since back 2024-03-16T12:09:43.207829+0000 front 2024-03-16T12:09:43.207999+0000 (oldest deadline 2024-03-16T12:10:07.907827+0000) 2024-03-16T12:10:23.577 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 16 12:10:23 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-9[67182]: 2024-03-16T12:10:23.186+0000 7fe1cee8b700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.154:6812 osd.5 since back 2024-03-16T12:09:52.310701+0000 front 2024-03-16T12:09:52.310491+0000 (oldest deadline 2024-03-16T12:10:17.610385+0000) 2024-03-16T12:10:24.116 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:10:23 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:10:23.811+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:10.713221+0000 front 2024-03-16T12:09:10.713087+0000 (oldest deadline 2024-03-16T12:09:35.413059+0000) 2024-03-16T12:10:24.117 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:10:23 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:10:23.811+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:20.614694+0000 front 2024-03-16T12:09:20.614671+0000 (oldest deadline 2024-03-16T12:09:45.215535+0000) 2024-03-16T12:10:24.117 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:10:23 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:10:23.811+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.154:6804 osd.2 since back 2024-03-16T12:09:29.916590+0000 front 2024-03-16T12:09:29.916612+0000 (oldest deadline 2024-03-16T12:09:52.816525+0000) 2024-03-16T12:10:24.117 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:10:23 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:10:23.811+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-16T12:09:37.517617+0000 front 2024-03-16T12:09:37.517557+0000 (oldest deadline 2024-03-16T12:10:02.817580+0000) 2024-03-16T12:10:24.117 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:10:23 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:10:23.811+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.152:6812 osd.4 since back 2024-03-16T12:09:42.818002+0000 front 2024-03-16T12:09:42.818046+0000 (oldest deadline 2024-03-16T12:10:07.518142+0000) 2024-03-16T12:10:24.117 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:10:23 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:10:23.811+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.154:6812 osd.5 since back 2024-03-16T12:09:53.919738+0000 front 2024-03-16T12:09:53.919614+0000 (oldest deadline 2024-03-16T12:10:19.819647+0000) 2024-03-16T12:10:24.534 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:24 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:24.061+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:13.025675+0000 front 2024-03-16T12:09:13.025847+0000 (oldest deadline 2024-03-16T12:09:35.325827+0000) 2024-03-16T12:10:24.534 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:24 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:24.061+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:19.927745+0000 front 2024-03-16T12:09:19.927761+0000 (oldest deadline 2024-03-16T12:09:45.827828+0000) 2024-03-16T12:10:24.534 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:24 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:24.061+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.154:6804 osd.2 since back 2024-03-16T12:09:28.629320+0000 front 2024-03-16T12:09:28.629357+0000 (oldest deadline 2024-03-16T12:09:52.129472+0000) 2024-03-16T12:10:24.534 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:24 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:24.061+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-16T12:09:38.432099+0000 front 2024-03-16T12:09:38.432072+0000 (oldest deadline 2024-03-16T12:10:01.332006+0000) 2024-03-16T12:10:24.534 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:24 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:24.061+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.152:6812 osd.4 since back 2024-03-16T12:09:43.633241+0000 front 2024-03-16T12:09:43.633117+0000 (oldest deadline 2024-03-16T12:10:07.733019+0000) 2024-03-16T12:10:24.534 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:24 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:24.061+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.154:6812 osd.5 since back 2024-03-16T12:09:53.534564+0000 front 2024-03-16T12:09:53.534519+0000 (oldest deadline 2024-03-16T12:10:15.834516+0000) 2024-03-16T12:10:24.576 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 16 12:10:24 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-9[67182]: 2024-03-16T12:10:24.229+0000 7fe1cee8b700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:12.704136+0000 front 2024-03-16T12:09:12.703904+0000 (oldest deadline 2024-03-16T12:09:37.404049+0000) 2024-03-16T12:10:24.577 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 16 12:10:24 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-9[67182]: 2024-03-16T12:10:24.229+0000 7fe1cee8b700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:17.404712+0000 front 2024-03-16T12:09:17.404800+0000 (oldest deadline 2024-03-16T12:09:42.704627+0000) 2024-03-16T12:10:24.577 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 16 12:10:24 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-9[67182]: 2024-03-16T12:10:24.229+0000 7fe1cee8b700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.154:6804 osd.2 since back 2024-03-16T12:09:27.405764+0000 front 2024-03-16T12:09:27.405893+0000 (oldest deadline 2024-03-16T12:09:52.105536+0000) 2024-03-16T12:10:24.577 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 16 12:10:24 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-9[67182]: 2024-03-16T12:10:24.229+0000 7fe1cee8b700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-16T12:09:37.907196+0000 front 2024-03-16T12:09:37.907286+0000 (oldest deadline 2024-03-16T12:10:03.207291+0000) 2024-03-16T12:10:24.577 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 16 12:10:24 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-9[67182]: 2024-03-16T12:10:24.229+0000 7fe1cee8b700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.152:6812 osd.4 since back 2024-03-16T12:09:43.207829+0000 front 2024-03-16T12:09:43.207999+0000 (oldest deadline 2024-03-16T12:10:07.907827+0000) 2024-03-16T12:10:24.577 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 16 12:10:24 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-9[67182]: 2024-03-16T12:10:24.229+0000 7fe1cee8b700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.154:6812 osd.5 since back 2024-03-16T12:09:52.310701+0000 front 2024-03-16T12:09:52.310491+0000 (oldest deadline 2024-03-16T12:10:17.610385+0000) 2024-03-16T12:10:25.116 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:10:24 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:10:24.763+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:10.713221+0000 front 2024-03-16T12:09:10.713087+0000 (oldest deadline 2024-03-16T12:09:35.413059+0000) 2024-03-16T12:10:25.116 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:10:24 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:10:24.763+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:20.614694+0000 front 2024-03-16T12:09:20.614671+0000 (oldest deadline 2024-03-16T12:09:45.215535+0000) 2024-03-16T12:10:25.117 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:10:24 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:10:24.763+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.154:6804 osd.2 since back 2024-03-16T12:09:29.916590+0000 front 2024-03-16T12:09:29.916612+0000 (oldest deadline 2024-03-16T12:09:52.816525+0000) 2024-03-16T12:10:25.117 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:10:24 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:10:24.763+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-16T12:09:37.517617+0000 front 2024-03-16T12:09:37.517557+0000 (oldest deadline 2024-03-16T12:10:02.817580+0000) 2024-03-16T12:10:25.117 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:10:24 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:10:24.763+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.152:6812 osd.4 since back 2024-03-16T12:09:42.818002+0000 front 2024-03-16T12:09:42.818046+0000 (oldest deadline 2024-03-16T12:10:07.518142+0000) 2024-03-16T12:10:25.117 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:10:24 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:10:24.763+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.154:6812 osd.5 since back 2024-03-16T12:09:53.919738+0000 front 2024-03-16T12:09:53.919614+0000 (oldest deadline 2024-03-16T12:10:19.819647+0000) 2024-03-16T12:10:25.117 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:10:24 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:10:24.763+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.29:6822 osd.6 since back 2024-03-16T12:09:59.820203+0000 front 2024-03-16T12:09:59.820268+0000 (oldest deadline 2024-03-16T12:10:24.520087+0000) 2024-03-16T12:10:25.284 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:25 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:25.083+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:13.025675+0000 front 2024-03-16T12:09:13.025847+0000 (oldest deadline 2024-03-16T12:09:35.325827+0000) 2024-03-16T12:10:25.284 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:25 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:25.083+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:19.927745+0000 front 2024-03-16T12:09:19.927761+0000 (oldest deadline 2024-03-16T12:09:45.827828+0000) 2024-03-16T12:10:25.284 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:25 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:25.083+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.154:6804 osd.2 since back 2024-03-16T12:09:28.629320+0000 front 2024-03-16T12:09:28.629357+0000 (oldest deadline 2024-03-16T12:09:52.129472+0000) 2024-03-16T12:10:25.284 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:25 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:25.083+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-16T12:09:38.432099+0000 front 2024-03-16T12:09:38.432072+0000 (oldest deadline 2024-03-16T12:10:01.332006+0000) 2024-03-16T12:10:25.285 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:25 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:25.083+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.152:6812 osd.4 since back 2024-03-16T12:09:43.633241+0000 front 2024-03-16T12:09:43.633117+0000 (oldest deadline 2024-03-16T12:10:07.733019+0000) 2024-03-16T12:10:25.285 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:25 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:25.083+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.154:6812 osd.5 since back 2024-03-16T12:09:53.534564+0000 front 2024-03-16T12:09:53.534519+0000 (oldest deadline 2024-03-16T12:10:15.834516+0000) 2024-03-16T12:10:25.577 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 16 12:10:25 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-9[67182]: 2024-03-16T12:10:25.221+0000 7fe1cee8b700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:12.704136+0000 front 2024-03-16T12:09:12.703904+0000 (oldest deadline 2024-03-16T12:09:37.404049+0000) 2024-03-16T12:10:25.577 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 16 12:10:25 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-9[67182]: 2024-03-16T12:10:25.221+0000 7fe1cee8b700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:17.404712+0000 front 2024-03-16T12:09:17.404800+0000 (oldest deadline 2024-03-16T12:09:42.704627+0000) 2024-03-16T12:10:25.577 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 16 12:10:25 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-9[67182]: 2024-03-16T12:10:25.221+0000 7fe1cee8b700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.154:6804 osd.2 since back 2024-03-16T12:09:27.405764+0000 front 2024-03-16T12:09:27.405893+0000 (oldest deadline 2024-03-16T12:09:52.105536+0000) 2024-03-16T12:10:25.577 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 16 12:10:25 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-9[67182]: 2024-03-16T12:10:25.221+0000 7fe1cee8b700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-16T12:09:37.907196+0000 front 2024-03-16T12:09:37.907286+0000 (oldest deadline 2024-03-16T12:10:03.207291+0000) 2024-03-16T12:10:25.577 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 16 12:10:25 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-9[67182]: 2024-03-16T12:10:25.221+0000 7fe1cee8b700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.152:6812 osd.4 since back 2024-03-16T12:09:43.207829+0000 front 2024-03-16T12:09:43.207999+0000 (oldest deadline 2024-03-16T12:10:07.907827+0000) 2024-03-16T12:10:25.577 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 16 12:10:25 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-9[67182]: 2024-03-16T12:10:25.221+0000 7fe1cee8b700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.154:6812 osd.5 since back 2024-03-16T12:09:52.310701+0000 front 2024-03-16T12:09:52.310491+0000 (oldest deadline 2024-03-16T12:10:17.610385+0000) 2024-03-16T12:10:25.578 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 16 12:10:25 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-9[67182]: 2024-03-16T12:10:25.221+0000 7fe1cee8b700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.29:6822 osd.6 since back 2024-03-16T12:09:59.311350+0000 front 2024-03-16T12:09:59.311422+0000 (oldest deadline 2024-03-16T12:10:24.611375+0000) 2024-03-16T12:10:26.116 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:10:25 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:10:25.741+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:10.713221+0000 front 2024-03-16T12:09:10.713087+0000 (oldest deadline 2024-03-16T12:09:35.413059+0000) 2024-03-16T12:10:26.116 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:10:25 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:10:25.741+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:20.614694+0000 front 2024-03-16T12:09:20.614671+0000 (oldest deadline 2024-03-16T12:09:45.215535+0000) 2024-03-16T12:10:26.117 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:10:25 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:10:25.741+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.154:6804 osd.2 since back 2024-03-16T12:09:29.916590+0000 front 2024-03-16T12:09:29.916612+0000 (oldest deadline 2024-03-16T12:09:52.816525+0000) 2024-03-16T12:10:26.117 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:10:25 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:10:25.741+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-16T12:09:37.517617+0000 front 2024-03-16T12:09:37.517557+0000 (oldest deadline 2024-03-16T12:10:02.817580+0000) 2024-03-16T12:10:26.117 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:10:25 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:10:25.741+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.152:6812 osd.4 since back 2024-03-16T12:09:42.818002+0000 front 2024-03-16T12:09:42.818046+0000 (oldest deadline 2024-03-16T12:10:07.518142+0000) 2024-03-16T12:10:26.117 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:10:25 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:10:25.741+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.154:6812 osd.5 since back 2024-03-16T12:09:53.919738+0000 front 2024-03-16T12:09:53.919614+0000 (oldest deadline 2024-03-16T12:10:19.819647+0000) 2024-03-16T12:10:26.117 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:10:25 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:10:25.741+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.29:6822 osd.6 since back 2024-03-16T12:09:59.820203+0000 front 2024-03-16T12:09:59.820268+0000 (oldest deadline 2024-03-16T12:10:24.520087+0000) 2024-03-16T12:10:26.534 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:26 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:26.113+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:13.025675+0000 front 2024-03-16T12:09:13.025847+0000 (oldest deadline 2024-03-16T12:09:35.325827+0000) 2024-03-16T12:10:26.534 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:26 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:26.113+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:19.927745+0000 front 2024-03-16T12:09:19.927761+0000 (oldest deadline 2024-03-16T12:09:45.827828+0000) 2024-03-16T12:10:26.534 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:26 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:26.113+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.154:6804 osd.2 since back 2024-03-16T12:09:28.629320+0000 front 2024-03-16T12:09:28.629357+0000 (oldest deadline 2024-03-16T12:09:52.129472+0000) 2024-03-16T12:10:26.534 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:26 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:26.113+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-16T12:09:38.432099+0000 front 2024-03-16T12:09:38.432072+0000 (oldest deadline 2024-03-16T12:10:01.332006+0000) 2024-03-16T12:10:26.535 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:26 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:26.113+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.152:6812 osd.4 since back 2024-03-16T12:09:43.633241+0000 front 2024-03-16T12:09:43.633117+0000 (oldest deadline 2024-03-16T12:10:07.733019+0000) 2024-03-16T12:10:26.535 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:26 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:26.113+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.154:6812 osd.5 since back 2024-03-16T12:09:53.534564+0000 front 2024-03-16T12:09:53.534519+0000 (oldest deadline 2024-03-16T12:10:15.834516+0000) 2024-03-16T12:10:26.576 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 16 12:10:26 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-9[67182]: 2024-03-16T12:10:26.256+0000 7fe1cee8b700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:12.704136+0000 front 2024-03-16T12:09:12.703904+0000 (oldest deadline 2024-03-16T12:09:37.404049+0000) 2024-03-16T12:10:26.577 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 16 12:10:26 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-9[67182]: 2024-03-16T12:10:26.256+0000 7fe1cee8b700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:17.404712+0000 front 2024-03-16T12:09:17.404800+0000 (oldest deadline 2024-03-16T12:09:42.704627+0000) 2024-03-16T12:10:26.577 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 16 12:10:26 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-9[67182]: 2024-03-16T12:10:26.256+0000 7fe1cee8b700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.154:6804 osd.2 since back 2024-03-16T12:09:27.405764+0000 front 2024-03-16T12:09:27.405893+0000 (oldest deadline 2024-03-16T12:09:52.105536+0000) 2024-03-16T12:10:26.577 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 16 12:10:26 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-9[67182]: 2024-03-16T12:10:26.256+0000 7fe1cee8b700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-16T12:09:37.907196+0000 front 2024-03-16T12:09:37.907286+0000 (oldest deadline 2024-03-16T12:10:03.207291+0000) 2024-03-16T12:10:26.577 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 16 12:10:26 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-9[67182]: 2024-03-16T12:10:26.256+0000 7fe1cee8b700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.152:6812 osd.4 since back 2024-03-16T12:09:43.207829+0000 front 2024-03-16T12:09:43.207999+0000 (oldest deadline 2024-03-16T12:10:07.907827+0000) 2024-03-16T12:10:26.577 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 16 12:10:26 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-9[67182]: 2024-03-16T12:10:26.256+0000 7fe1cee8b700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.154:6812 osd.5 since back 2024-03-16T12:09:52.310701+0000 front 2024-03-16T12:09:52.310491+0000 (oldest deadline 2024-03-16T12:10:17.610385+0000) 2024-03-16T12:10:26.577 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 16 12:10:26 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-9[67182]: 2024-03-16T12:10:26.256+0000 7fe1cee8b700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.29:6822 osd.6 since back 2024-03-16T12:09:59.311350+0000 front 2024-03-16T12:09:59.311422+0000 (oldest deadline 2024-03-16T12:10:24.611375+0000) 2024-03-16T12:10:27.116 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:10:26 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:10:26.730+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:10.713221+0000 front 2024-03-16T12:09:10.713087+0000 (oldest deadline 2024-03-16T12:09:35.413059+0000) 2024-03-16T12:10:27.116 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:10:26 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:10:26.730+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:20.614694+0000 front 2024-03-16T12:09:20.614671+0000 (oldest deadline 2024-03-16T12:09:45.215535+0000) 2024-03-16T12:10:27.116 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:10:26 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:10:26.730+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.154:6804 osd.2 since back 2024-03-16T12:09:29.916590+0000 front 2024-03-16T12:09:29.916612+0000 (oldest deadline 2024-03-16T12:09:52.816525+0000) 2024-03-16T12:10:27.117 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:10:26 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:10:26.730+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-16T12:09:37.517617+0000 front 2024-03-16T12:09:37.517557+0000 (oldest deadline 2024-03-16T12:10:02.817580+0000) 2024-03-16T12:10:27.117 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:10:26 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:10:26.730+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.152:6812 osd.4 since back 2024-03-16T12:09:42.818002+0000 front 2024-03-16T12:09:42.818046+0000 (oldest deadline 2024-03-16T12:10:07.518142+0000) 2024-03-16T12:10:27.117 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:10:26 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:10:26.730+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.154:6812 osd.5 since back 2024-03-16T12:09:53.919738+0000 front 2024-03-16T12:09:53.919614+0000 (oldest deadline 2024-03-16T12:10:19.819647+0000) 2024-03-16T12:10:27.117 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:10:26 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:10:26.730+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.29:6822 osd.6 since back 2024-03-16T12:09:59.820203+0000 front 2024-03-16T12:09:59.820268+0000 (oldest deadline 2024-03-16T12:10:24.520087+0000) 2024-03-16T12:10:27.534 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:27 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:27.092+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:13.025675+0000 front 2024-03-16T12:09:13.025847+0000 (oldest deadline 2024-03-16T12:09:35.325827+0000) 2024-03-16T12:10:27.534 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:27 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:27.092+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:19.927745+0000 front 2024-03-16T12:09:19.927761+0000 (oldest deadline 2024-03-16T12:09:45.827828+0000) 2024-03-16T12:10:27.534 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:27 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:27.092+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.154:6804 osd.2 since back 2024-03-16T12:09:28.629320+0000 front 2024-03-16T12:09:28.629357+0000 (oldest deadline 2024-03-16T12:09:52.129472+0000) 2024-03-16T12:10:27.534 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:27 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:27.092+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-16T12:09:38.432099+0000 front 2024-03-16T12:09:38.432072+0000 (oldest deadline 2024-03-16T12:10:01.332006+0000) 2024-03-16T12:10:27.534 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:27 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:27.092+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.152:6812 osd.4 since back 2024-03-16T12:09:43.633241+0000 front 2024-03-16T12:09:43.633117+0000 (oldest deadline 2024-03-16T12:10:07.733019+0000) 2024-03-16T12:10:27.535 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:27 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:27.092+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.154:6812 osd.5 since back 2024-03-16T12:09:53.534564+0000 front 2024-03-16T12:09:53.534519+0000 (oldest deadline 2024-03-16T12:10:15.834516+0000) 2024-03-16T12:10:27.577 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 16 12:10:27 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-9[67182]: 2024-03-16T12:10:27.280+0000 7fe1cee8b700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:12.704136+0000 front 2024-03-16T12:09:12.703904+0000 (oldest deadline 2024-03-16T12:09:37.404049+0000) 2024-03-16T12:10:27.577 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 16 12:10:27 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-9[67182]: 2024-03-16T12:10:27.280+0000 7fe1cee8b700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:17.404712+0000 front 2024-03-16T12:09:17.404800+0000 (oldest deadline 2024-03-16T12:09:42.704627+0000) 2024-03-16T12:10:27.577 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 16 12:10:27 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-9[67182]: 2024-03-16T12:10:27.280+0000 7fe1cee8b700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.154:6804 osd.2 since back 2024-03-16T12:09:27.405764+0000 front 2024-03-16T12:09:27.405893+0000 (oldest deadline 2024-03-16T12:09:52.105536+0000) 2024-03-16T12:10:27.577 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 16 12:10:27 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-9[67182]: 2024-03-16T12:10:27.280+0000 7fe1cee8b700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-16T12:09:37.907196+0000 front 2024-03-16T12:09:37.907286+0000 (oldest deadline 2024-03-16T12:10:03.207291+0000) 2024-03-16T12:10:27.577 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 16 12:10:27 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-9[67182]: 2024-03-16T12:10:27.280+0000 7fe1cee8b700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.152:6812 osd.4 since back 2024-03-16T12:09:43.207829+0000 front 2024-03-16T12:09:43.207999+0000 (oldest deadline 2024-03-16T12:10:07.907827+0000) 2024-03-16T12:10:27.577 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 16 12:10:27 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-9[67182]: 2024-03-16T12:10:27.280+0000 7fe1cee8b700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.154:6812 osd.5 since back 2024-03-16T12:09:52.310701+0000 front 2024-03-16T12:09:52.310491+0000 (oldest deadline 2024-03-16T12:10:17.610385+0000) 2024-03-16T12:10:27.577 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 16 12:10:27 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-9[67182]: 2024-03-16T12:10:27.280+0000 7fe1cee8b700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.29:6822 osd.6 since back 2024-03-16T12:09:59.311350+0000 front 2024-03-16T12:09:59.311422+0000 (oldest deadline 2024-03-16T12:10:24.611375+0000) 2024-03-16T12:10:28.117 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:10:27 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:10:27.754+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:10.713221+0000 front 2024-03-16T12:09:10.713087+0000 (oldest deadline 2024-03-16T12:09:35.413059+0000) 2024-03-16T12:10:28.117 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:10:27 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:10:27.754+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:20.614694+0000 front 2024-03-16T12:09:20.614671+0000 (oldest deadline 2024-03-16T12:09:45.215535+0000) 2024-03-16T12:10:28.118 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:10:27 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:10:27.754+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.154:6804 osd.2 since back 2024-03-16T12:09:29.916590+0000 front 2024-03-16T12:09:29.916612+0000 (oldest deadline 2024-03-16T12:09:52.816525+0000) 2024-03-16T12:10:28.118 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:10:27 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:10:27.754+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-16T12:09:37.517617+0000 front 2024-03-16T12:09:37.517557+0000 (oldest deadline 2024-03-16T12:10:02.817580+0000) 2024-03-16T12:10:28.118 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:10:27 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:10:27.754+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.152:6812 osd.4 since back 2024-03-16T12:09:42.818002+0000 front 2024-03-16T12:09:42.818046+0000 (oldest deadline 2024-03-16T12:10:07.518142+0000) 2024-03-16T12:10:28.118 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:10:27 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:10:27.754+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.154:6812 osd.5 since back 2024-03-16T12:09:53.919738+0000 front 2024-03-16T12:09:53.919614+0000 (oldest deadline 2024-03-16T12:10:19.819647+0000) 2024-03-16T12:10:28.118 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:10:27 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:10:27.754+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.29:6822 osd.6 since back 2024-03-16T12:09:59.820203+0000 front 2024-03-16T12:09:59.820268+0000 (oldest deadline 2024-03-16T12:10:24.520087+0000) 2024-03-16T12:10:28.174 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 16 12:10:27 smithi029 podman[95718]: 2024-03-16 12:10:27.919340075 +0000 UTC m=+5.473141750 container died 2f4710eef03c26e8f7b0277320a34aac534de60cf6c60f5c65e591702ed11032 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-9, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, ceph=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-03-16T12:10:28.534 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:28 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:28.103+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:13.025675+0000 front 2024-03-16T12:09:13.025847+0000 (oldest deadline 2024-03-16T12:09:35.325827+0000) 2024-03-16T12:10:28.534 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:28 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:28.103+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:19.927745+0000 front 2024-03-16T12:09:19.927761+0000 (oldest deadline 2024-03-16T12:09:45.827828+0000) 2024-03-16T12:10:28.534 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:28 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:28.103+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.154:6804 osd.2 since back 2024-03-16T12:09:28.629320+0000 front 2024-03-16T12:09:28.629357+0000 (oldest deadline 2024-03-16T12:09:52.129472+0000) 2024-03-16T12:10:28.534 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:28 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:28.103+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-16T12:09:38.432099+0000 front 2024-03-16T12:09:38.432072+0000 (oldest deadline 2024-03-16T12:10:01.332006+0000) 2024-03-16T12:10:28.534 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:28 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:28.103+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.152:6812 osd.4 since back 2024-03-16T12:09:43.633241+0000 front 2024-03-16T12:09:43.633117+0000 (oldest deadline 2024-03-16T12:10:07.733019+0000) 2024-03-16T12:10:28.534 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:28 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:28.103+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.154:6812 osd.5 since back 2024-03-16T12:09:53.534564+0000 front 2024-03-16T12:09:53.534519+0000 (oldest deadline 2024-03-16T12:10:15.834516+0000) 2024-03-16T12:10:28.551 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 16 12:10:28 smithi029 podman[95718]: 2024-03-16 12:10:28.278047032 +0000 UTC m=+5.831848707 container cleanup 2f4710eef03c26e8f7b0277320a34aac534de60cf6c60f5c65e591702ed11032 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-9, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-16T12:10:28.551 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 16 12:10:28 smithi029 bash[95718]: ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-9 2024-03-16T12:10:28.826 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 16 12:10:28 smithi029 podman[95735]: 2024-03-16 12:10:28.564106502 +0000 UTC m=+0.639123336 container remove 2f4710eef03c26e8f7b0277320a34aac534de60cf6c60f5c65e591702ed11032 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-9, org.label-schema.build-date=20240307, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD) 2024-03-16T12:10:29.116 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:10:28 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:10:28.784+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:10.713221+0000 front 2024-03-16T12:09:10.713087+0000 (oldest deadline 2024-03-16T12:09:35.413059+0000) 2024-03-16T12:10:29.116 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:10:28 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:10:28.784+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:20.614694+0000 front 2024-03-16T12:09:20.614671+0000 (oldest deadline 2024-03-16T12:09:45.215535+0000) 2024-03-16T12:10:29.116 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:10:28 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:10:28.784+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.154:6804 osd.2 since back 2024-03-16T12:09:29.916590+0000 front 2024-03-16T12:09:29.916612+0000 (oldest deadline 2024-03-16T12:09:52.816525+0000) 2024-03-16T12:10:29.116 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:10:28 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:10:28.784+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-16T12:09:37.517617+0000 front 2024-03-16T12:09:37.517557+0000 (oldest deadline 2024-03-16T12:10:02.817580+0000) 2024-03-16T12:10:29.117 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:10:28 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:10:28.784+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.152:6812 osd.4 since back 2024-03-16T12:09:42.818002+0000 front 2024-03-16T12:09:42.818046+0000 (oldest deadline 2024-03-16T12:10:07.518142+0000) 2024-03-16T12:10:29.117 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:10:28 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:10:28.784+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.154:6812 osd.5 since back 2024-03-16T12:09:53.919738+0000 front 2024-03-16T12:09:53.919614+0000 (oldest deadline 2024-03-16T12:10:19.819647+0000) 2024-03-16T12:10:29.117 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:10:28 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:10:28.784+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.29:6822 osd.6 since back 2024-03-16T12:09:59.820203+0000 front 2024-03-16T12:09:59.820268+0000 (oldest deadline 2024-03-16T12:10:24.520087+0000) 2024-03-16T12:10:29.208 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 16 12:10:28 smithi029 podman[95822]: 2024-03-16 12:10:28.796390035 +0000 UTC m=+0.018503993 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-16T12:10:29.209 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 16 12:10:28 smithi029 podman[95822]: 2024-03-16 12:10:28.900722495 +0000 UTC m=+0.122836495 container create ccda10209887fe2f82e324e16c3f4ee67c0da1db1d1b40669757c018a8afb0d1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-9-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-03-16T12:10:29.209 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 16 12:10:29 smithi029 podman[95822]: 2024-03-16 12:10:29.130306412 +0000 UTC m=+0.352420379 container init ccda10209887fe2f82e324e16c3f4ee67c0da1db1d1b40669757c018a8afb0d1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-9-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307) 2024-03-16T12:10:29.209 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 16 12:10:29 smithi029 podman[95822]: 2024-03-16 12:10:29.13344858 +0000 UTC m=+0.355562538 container start ccda10209887fe2f82e324e16c3f4ee67c0da1db1d1b40669757c018a8afb0d1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-9-deactivate, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-16T12:10:29.209 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 16 12:10:29 smithi029 podman[95822]: 2024-03-16 12:10:29.221438014 +0000 UTC m=+0.443551989 container attach ccda10209887fe2f82e324e16c3f4ee67c0da1db1d1b40669757c018a8afb0d1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-9-deactivate, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-03-16T12:10:29.534 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:29 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:29.139+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:13.025675+0000 front 2024-03-16T12:09:13.025847+0000 (oldest deadline 2024-03-16T12:09:35.325827+0000) 2024-03-16T12:10:29.534 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:29 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:29.139+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:19.927745+0000 front 2024-03-16T12:09:19.927761+0000 (oldest deadline 2024-03-16T12:09:45.827828+0000) 2024-03-16T12:10:29.534 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:29 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:29.139+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.154:6804 osd.2 since back 2024-03-16T12:09:28.629320+0000 front 2024-03-16T12:09:28.629357+0000 (oldest deadline 2024-03-16T12:09:52.129472+0000) 2024-03-16T12:10:29.534 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:29 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:29.139+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-16T12:09:38.432099+0000 front 2024-03-16T12:09:38.432072+0000 (oldest deadline 2024-03-16T12:10:01.332006+0000) 2024-03-16T12:10:29.534 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:29 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:29.139+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.152:6812 osd.4 since back 2024-03-16T12:09:43.633241+0000 front 2024-03-16T12:09:43.633117+0000 (oldest deadline 2024-03-16T12:10:07.733019+0000) 2024-03-16T12:10:29.534 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:29 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:29.139+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.154:6812 osd.5 since back 2024-03-16T12:09:53.534564+0000 front 2024-03-16T12:09:53.534519+0000 (oldest deadline 2024-03-16T12:10:15.834516+0000) 2024-03-16T12:10:29.576 INFO:journalctl@ceph.osd.9.smithi029.stdout:Mar 16 12:10:29 smithi029 podman[95822]: 2024-03-16 12:10:29.433105785 +0000 UTC m=+0.655219742 container died ccda10209887fe2f82e324e16c3f4ee67c0da1db1d1b40669757c018a8afb0d1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-9-deactivate, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5) 2024-03-16T12:10:30.011 DEBUG:teuthology.orchestra.run.smithi029:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d1887f28-e389-11ee-95c9-87774f69a715@osd.9.service' 2024-03-16T12:10:30.053 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-16T12:10:30.053 INFO:tasks.cephadm.osd.9:Stopped osd.9 2024-03-16T12:10:30.053 INFO:tasks.cephadm.osd.10:Stopping osd.10... 2024-03-16T12:10:30.054 DEBUG:teuthology.orchestra.run.smithi152:> sudo systemctl stop ceph-d1887f28-e389-11ee-95c9-87774f69a715@osd.10 2024-03-16T12:10:30.084 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:10:29 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:10:29.745+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:10.713221+0000 front 2024-03-16T12:09:10.713087+0000 (oldest deadline 2024-03-16T12:09:35.413059+0000) 2024-03-16T12:10:30.084 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:10:29 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:10:29.745+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:20.614694+0000 front 2024-03-16T12:09:20.614671+0000 (oldest deadline 2024-03-16T12:09:45.215535+0000) 2024-03-16T12:10:30.084 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:10:29 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:10:29.745+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.154:6804 osd.2 since back 2024-03-16T12:09:29.916590+0000 front 2024-03-16T12:09:29.916612+0000 (oldest deadline 2024-03-16T12:09:52.816525+0000) 2024-03-16T12:10:30.084 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:10:29 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:10:29.745+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-16T12:09:37.517617+0000 front 2024-03-16T12:09:37.517557+0000 (oldest deadline 2024-03-16T12:10:02.817580+0000) 2024-03-16T12:10:30.084 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:10:29 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:10:29.745+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.152:6812 osd.4 since back 2024-03-16T12:09:42.818002+0000 front 2024-03-16T12:09:42.818046+0000 (oldest deadline 2024-03-16T12:10:07.518142+0000) 2024-03-16T12:10:30.084 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:10:29 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:10:29.745+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.154:6812 osd.5 since back 2024-03-16T12:09:53.919738+0000 front 2024-03-16T12:09:53.919614+0000 (oldest deadline 2024-03-16T12:10:19.819647+0000) 2024-03-16T12:10:30.084 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:10:29 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:10:29.745+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.29:6822 osd.6 since back 2024-03-16T12:09:59.820203+0000 front 2024-03-16T12:09:59.820268+0000 (oldest deadline 2024-03-16T12:10:24.520087+0000) 2024-03-16T12:10:30.366 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:10:30 smithi152 systemd[1]: Stopping Ceph osd.10 for d1887f28-e389-11ee-95c9-87774f69a715... 2024-03-16T12:10:30.534 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:30 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:30.169+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:13.025675+0000 front 2024-03-16T12:09:13.025847+0000 (oldest deadline 2024-03-16T12:09:35.325827+0000) 2024-03-16T12:10:30.534 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:30 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:30.169+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:19.927745+0000 front 2024-03-16T12:09:19.927761+0000 (oldest deadline 2024-03-16T12:09:45.827828+0000) 2024-03-16T12:10:30.534 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:30 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:30.169+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.154:6804 osd.2 since back 2024-03-16T12:09:28.629320+0000 front 2024-03-16T12:09:28.629357+0000 (oldest deadline 2024-03-16T12:09:52.129472+0000) 2024-03-16T12:10:30.534 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:30 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:30.169+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-16T12:09:38.432099+0000 front 2024-03-16T12:09:38.432072+0000 (oldest deadline 2024-03-16T12:10:01.332006+0000) 2024-03-16T12:10:30.534 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:30 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:30.169+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.152:6812 osd.4 since back 2024-03-16T12:09:43.633241+0000 front 2024-03-16T12:09:43.633117+0000 (oldest deadline 2024-03-16T12:10:07.733019+0000) 2024-03-16T12:10:30.535 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:30 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:30.169+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.154:6812 osd.5 since back 2024-03-16T12:09:53.534564+0000 front 2024-03-16T12:09:53.534519+0000 (oldest deadline 2024-03-16T12:10:15.834516+0000) 2024-03-16T12:10:30.739 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:10:30 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:10:30.397+0000 7fe128f59700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.10 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-16T12:10:30.740 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:10:30 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:10:30.397+0000 7fe128f59700 -1 osd.10 119 *** Got signal Terminated *** 2024-03-16T12:10:30.740 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:10:30 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:10:30.397+0000 7fe128f59700 -1 osd.10 119 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-16T12:10:30.740 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:10:30 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:10:30.751+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:10.713221+0000 front 2024-03-16T12:09:10.713087+0000 (oldest deadline 2024-03-16T12:09:35.413059+0000) 2024-03-16T12:10:30.740 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:10:30 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:10:30.751+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:20.614694+0000 front 2024-03-16T12:09:20.614671+0000 (oldest deadline 2024-03-16T12:09:45.215535+0000) 2024-03-16T12:10:30.740 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:10:30 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:10:30.751+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.154:6804 osd.2 since back 2024-03-16T12:09:29.916590+0000 front 2024-03-16T12:09:29.916612+0000 (oldest deadline 2024-03-16T12:09:52.816525+0000) 2024-03-16T12:10:30.740 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:10:30 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:10:30.751+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-16T12:09:37.517617+0000 front 2024-03-16T12:09:37.517557+0000 (oldest deadline 2024-03-16T12:10:02.817580+0000) 2024-03-16T12:10:30.740 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:10:30 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:10:30.751+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.152:6812 osd.4 since back 2024-03-16T12:09:42.818002+0000 front 2024-03-16T12:09:42.818046+0000 (oldest deadline 2024-03-16T12:10:07.518142+0000) 2024-03-16T12:10:30.740 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:10:30 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:10:30.751+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.154:6812 osd.5 since back 2024-03-16T12:09:53.919738+0000 front 2024-03-16T12:09:53.919614+0000 (oldest deadline 2024-03-16T12:10:19.819647+0000) 2024-03-16T12:10:30.740 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:10:30 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:10:30.751+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.29:6822 osd.6 since back 2024-03-16T12:09:59.820203+0000 front 2024-03-16T12:09:59.820268+0000 (oldest deadline 2024-03-16T12:10:24.520087+0000) 2024-03-16T12:10:31.534 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:31 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:31.142+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:13.025675+0000 front 2024-03-16T12:09:13.025847+0000 (oldest deadline 2024-03-16T12:09:35.325827+0000) 2024-03-16T12:10:31.534 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:31 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:31.142+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:19.927745+0000 front 2024-03-16T12:09:19.927761+0000 (oldest deadline 2024-03-16T12:09:45.827828+0000) 2024-03-16T12:10:31.534 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:31 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:31.142+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.154:6804 osd.2 since back 2024-03-16T12:09:28.629320+0000 front 2024-03-16T12:09:28.629357+0000 (oldest deadline 2024-03-16T12:09:52.129472+0000) 2024-03-16T12:10:31.534 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:31 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:31.142+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-16T12:09:38.432099+0000 front 2024-03-16T12:09:38.432072+0000 (oldest deadline 2024-03-16T12:10:01.332006+0000) 2024-03-16T12:10:31.534 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:31 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:31.142+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.152:6812 osd.4 since back 2024-03-16T12:09:43.633241+0000 front 2024-03-16T12:09:43.633117+0000 (oldest deadline 2024-03-16T12:10:07.733019+0000) 2024-03-16T12:10:31.535 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:31 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:31.142+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.154:6812 osd.5 since back 2024-03-16T12:09:53.534564+0000 front 2024-03-16T12:09:53.534519+0000 (oldest deadline 2024-03-16T12:10:15.834516+0000) 2024-03-16T12:10:32.116 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:10:31 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:10:31.770+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:10.713221+0000 front 2024-03-16T12:09:10.713087+0000 (oldest deadline 2024-03-16T12:09:35.413059+0000) 2024-03-16T12:10:32.116 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:10:31 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:10:31.770+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:20.614694+0000 front 2024-03-16T12:09:20.614671+0000 (oldest deadline 2024-03-16T12:09:45.215535+0000) 2024-03-16T12:10:32.116 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:10:31 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:10:31.770+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.154:6804 osd.2 since back 2024-03-16T12:09:29.916590+0000 front 2024-03-16T12:09:29.916612+0000 (oldest deadline 2024-03-16T12:09:52.816525+0000) 2024-03-16T12:10:32.117 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:10:31 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:10:31.770+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-16T12:09:37.517617+0000 front 2024-03-16T12:09:37.517557+0000 (oldest deadline 2024-03-16T12:10:02.817580+0000) 2024-03-16T12:10:32.117 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:10:31 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:10:31.770+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.152:6812 osd.4 since back 2024-03-16T12:09:42.818002+0000 front 2024-03-16T12:09:42.818046+0000 (oldest deadline 2024-03-16T12:10:07.518142+0000) 2024-03-16T12:10:32.117 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:10:31 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:10:31.770+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.154:6812 osd.5 since back 2024-03-16T12:09:53.919738+0000 front 2024-03-16T12:09:53.919614+0000 (oldest deadline 2024-03-16T12:10:19.819647+0000) 2024-03-16T12:10:32.117 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:10:31 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:10:31.770+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.29:6822 osd.6 since back 2024-03-16T12:09:59.820203+0000 front 2024-03-16T12:09:59.820268+0000 (oldest deadline 2024-03-16T12:10:24.520087+0000) 2024-03-16T12:10:32.533 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:32 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:32.172+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:13.025675+0000 front 2024-03-16T12:09:13.025847+0000 (oldest deadline 2024-03-16T12:09:35.325827+0000) 2024-03-16T12:10:32.534 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:32 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:32.172+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:19.927745+0000 front 2024-03-16T12:09:19.927761+0000 (oldest deadline 2024-03-16T12:09:45.827828+0000) 2024-03-16T12:10:32.534 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:32 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:32.172+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.154:6804 osd.2 since back 2024-03-16T12:09:28.629320+0000 front 2024-03-16T12:09:28.629357+0000 (oldest deadline 2024-03-16T12:09:52.129472+0000) 2024-03-16T12:10:32.534 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:32 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:32.172+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-16T12:09:38.432099+0000 front 2024-03-16T12:09:38.432072+0000 (oldest deadline 2024-03-16T12:10:01.332006+0000) 2024-03-16T12:10:32.534 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:32 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:32.172+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.152:6812 osd.4 since back 2024-03-16T12:09:43.633241+0000 front 2024-03-16T12:09:43.633117+0000 (oldest deadline 2024-03-16T12:10:07.733019+0000) 2024-03-16T12:10:32.534 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:32 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:32.172+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.154:6812 osd.5 since back 2024-03-16T12:09:53.534564+0000 front 2024-03-16T12:09:53.534519+0000 (oldest deadline 2024-03-16T12:10:15.834516+0000) 2024-03-16T12:10:33.116 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:10:32 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:10:32.724+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:10.713221+0000 front 2024-03-16T12:09:10.713087+0000 (oldest deadline 2024-03-16T12:09:35.413059+0000) 2024-03-16T12:10:33.116 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:10:32 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:10:32.724+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:20.614694+0000 front 2024-03-16T12:09:20.614671+0000 (oldest deadline 2024-03-16T12:09:45.215535+0000) 2024-03-16T12:10:33.116 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:10:32 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:10:32.724+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.154:6804 osd.2 since back 2024-03-16T12:09:29.916590+0000 front 2024-03-16T12:09:29.916612+0000 (oldest deadline 2024-03-16T12:09:52.816525+0000) 2024-03-16T12:10:33.116 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:10:32 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:10:32.724+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-16T12:09:37.517617+0000 front 2024-03-16T12:09:37.517557+0000 (oldest deadline 2024-03-16T12:10:02.817580+0000) 2024-03-16T12:10:33.117 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:10:32 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:10:32.724+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.152:6812 osd.4 since back 2024-03-16T12:09:42.818002+0000 front 2024-03-16T12:09:42.818046+0000 (oldest deadline 2024-03-16T12:10:07.518142+0000) 2024-03-16T12:10:33.117 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:10:32 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:10:32.724+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.154:6812 osd.5 since back 2024-03-16T12:09:53.919738+0000 front 2024-03-16T12:09:53.919614+0000 (oldest deadline 2024-03-16T12:10:19.819647+0000) 2024-03-16T12:10:33.117 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:10:32 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:10:32.724+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.29:6822 osd.6 since back 2024-03-16T12:09:59.820203+0000 front 2024-03-16T12:09:59.820268+0000 (oldest deadline 2024-03-16T12:10:24.520087+0000) 2024-03-16T12:10:33.534 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:33 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:33.129+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:13.025675+0000 front 2024-03-16T12:09:13.025847+0000 (oldest deadline 2024-03-16T12:09:35.325827+0000) 2024-03-16T12:10:33.534 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:33 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:33.129+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:19.927745+0000 front 2024-03-16T12:09:19.927761+0000 (oldest deadline 2024-03-16T12:09:45.827828+0000) 2024-03-16T12:10:33.534 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:33 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:33.129+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.154:6804 osd.2 since back 2024-03-16T12:09:28.629320+0000 front 2024-03-16T12:09:28.629357+0000 (oldest deadline 2024-03-16T12:09:52.129472+0000) 2024-03-16T12:10:33.534 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:33 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:33.129+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-16T12:09:38.432099+0000 front 2024-03-16T12:09:38.432072+0000 (oldest deadline 2024-03-16T12:10:01.332006+0000) 2024-03-16T12:10:33.534 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:33 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:33.129+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.152:6812 osd.4 since back 2024-03-16T12:09:43.633241+0000 front 2024-03-16T12:09:43.633117+0000 (oldest deadline 2024-03-16T12:10:07.733019+0000) 2024-03-16T12:10:33.534 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:33 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:33.129+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.154:6812 osd.5 since back 2024-03-16T12:09:53.534564+0000 front 2024-03-16T12:09:53.534519+0000 (oldest deadline 2024-03-16T12:10:15.834516+0000) 2024-03-16T12:10:34.116 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:10:33 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:10:33.703+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:10.713221+0000 front 2024-03-16T12:09:10.713087+0000 (oldest deadline 2024-03-16T12:09:35.413059+0000) 2024-03-16T12:10:34.116 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:10:33 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:10:33.703+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:20.614694+0000 front 2024-03-16T12:09:20.614671+0000 (oldest deadline 2024-03-16T12:09:45.215535+0000) 2024-03-16T12:10:34.116 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:10:33 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:10:33.703+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.154:6804 osd.2 since back 2024-03-16T12:09:29.916590+0000 front 2024-03-16T12:09:29.916612+0000 (oldest deadline 2024-03-16T12:09:52.816525+0000) 2024-03-16T12:10:34.116 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:10:33 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:10:33.703+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-16T12:09:37.517617+0000 front 2024-03-16T12:09:37.517557+0000 (oldest deadline 2024-03-16T12:10:02.817580+0000) 2024-03-16T12:10:34.117 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:10:33 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:10:33.703+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.152:6812 osd.4 since back 2024-03-16T12:09:42.818002+0000 front 2024-03-16T12:09:42.818046+0000 (oldest deadline 2024-03-16T12:10:07.518142+0000) 2024-03-16T12:10:34.117 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:10:33 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:10:33.703+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.154:6812 osd.5 since back 2024-03-16T12:09:53.919738+0000 front 2024-03-16T12:09:53.919614+0000 (oldest deadline 2024-03-16T12:10:19.819647+0000) 2024-03-16T12:10:34.117 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:10:33 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:10:33.703+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.29:6822 osd.6 since back 2024-03-16T12:09:59.820203+0000 front 2024-03-16T12:09:59.820268+0000 (oldest deadline 2024-03-16T12:10:24.520087+0000) 2024-03-16T12:10:34.534 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:34 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:34.162+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:13.025675+0000 front 2024-03-16T12:09:13.025847+0000 (oldest deadline 2024-03-16T12:09:35.325827+0000) 2024-03-16T12:10:34.534 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:34 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:34.162+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:19.927745+0000 front 2024-03-16T12:09:19.927761+0000 (oldest deadline 2024-03-16T12:09:45.827828+0000) 2024-03-16T12:10:34.534 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:34 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:34.162+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.154:6804 osd.2 since back 2024-03-16T12:09:28.629320+0000 front 2024-03-16T12:09:28.629357+0000 (oldest deadline 2024-03-16T12:09:52.129472+0000) 2024-03-16T12:10:34.534 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:34 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:34.162+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-16T12:09:38.432099+0000 front 2024-03-16T12:09:38.432072+0000 (oldest deadline 2024-03-16T12:10:01.332006+0000) 2024-03-16T12:10:34.534 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:34 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:34.162+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.152:6812 osd.4 since back 2024-03-16T12:09:43.633241+0000 front 2024-03-16T12:09:43.633117+0000 (oldest deadline 2024-03-16T12:10:07.733019+0000) 2024-03-16T12:10:34.534 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:34 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:34.162+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.154:6812 osd.5 since back 2024-03-16T12:09:53.534564+0000 front 2024-03-16T12:09:53.534519+0000 (oldest deadline 2024-03-16T12:10:15.834516+0000) 2024-03-16T12:10:35.116 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:10:34 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:10:34.671+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:10.713221+0000 front 2024-03-16T12:09:10.713087+0000 (oldest deadline 2024-03-16T12:09:35.413059+0000) 2024-03-16T12:10:35.116 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:10:34 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:10:34.671+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:20.614694+0000 front 2024-03-16T12:09:20.614671+0000 (oldest deadline 2024-03-16T12:09:45.215535+0000) 2024-03-16T12:10:35.116 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:10:34 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:10:34.671+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.154:6804 osd.2 since back 2024-03-16T12:09:29.916590+0000 front 2024-03-16T12:09:29.916612+0000 (oldest deadline 2024-03-16T12:09:52.816525+0000) 2024-03-16T12:10:35.116 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:10:34 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:10:34.671+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-16T12:09:37.517617+0000 front 2024-03-16T12:09:37.517557+0000 (oldest deadline 2024-03-16T12:10:02.817580+0000) 2024-03-16T12:10:35.117 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:10:34 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:10:34.671+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.152:6812 osd.4 since back 2024-03-16T12:09:42.818002+0000 front 2024-03-16T12:09:42.818046+0000 (oldest deadline 2024-03-16T12:10:07.518142+0000) 2024-03-16T12:10:35.117 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:10:34 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:10:34.671+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.154:6812 osd.5 since back 2024-03-16T12:09:53.919738+0000 front 2024-03-16T12:09:53.919614+0000 (oldest deadline 2024-03-16T12:10:19.819647+0000) 2024-03-16T12:10:35.117 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:10:34 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10[61821]: 2024-03-16T12:10:34.671+0000 7fe12554d700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.29:6822 osd.6 since back 2024-03-16T12:09:59.820203+0000 front 2024-03-16T12:09:59.820268+0000 (oldest deadline 2024-03-16T12:10:24.520087+0000) 2024-03-16T12:10:35.534 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:35 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:35.201+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:13.025675+0000 front 2024-03-16T12:09:13.025847+0000 (oldest deadline 2024-03-16T12:09:35.325827+0000) 2024-03-16T12:10:35.534 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:35 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:35.201+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:19.927745+0000 front 2024-03-16T12:09:19.927761+0000 (oldest deadline 2024-03-16T12:09:45.827828+0000) 2024-03-16T12:10:35.534 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:35 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:35.201+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.154:6804 osd.2 since back 2024-03-16T12:09:28.629320+0000 front 2024-03-16T12:09:28.629357+0000 (oldest deadline 2024-03-16T12:09:52.129472+0000) 2024-03-16T12:10:35.534 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:35 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:35.201+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-16T12:09:38.432099+0000 front 2024-03-16T12:09:38.432072+0000 (oldest deadline 2024-03-16T12:10:01.332006+0000) 2024-03-16T12:10:35.534 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:35 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:35.201+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.152:6812 osd.4 since back 2024-03-16T12:09:43.633241+0000 front 2024-03-16T12:09:43.633117+0000 (oldest deadline 2024-03-16T12:10:07.733019+0000) 2024-03-16T12:10:35.534 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:35 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:35.201+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.154:6812 osd.5 since back 2024-03-16T12:09:53.534564+0000 front 2024-03-16T12:09:53.534519+0000 (oldest deadline 2024-03-16T12:10:15.834516+0000) 2024-03-16T12:10:36.045 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:10:35 smithi152 podman[74586]: 2024-03-16 12:10:35.774359455 +0000 UTC m=+5.500609889 container died 826ecf36251994139c6487aad9b1616b87b91373047d13f3a2078836653c0b6a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5) 2024-03-16T12:10:36.311 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:10:36 smithi152 podman[74586]: 2024-03-16 12:10:36.131053944 +0000 UTC m=+5.857304367 container cleanup 826ecf36251994139c6487aad9b1616b87b91373047d13f3a2078836653c0b6a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10, ceph=True, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-03-16T12:10:36.311 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:10:36 smithi152 bash[74586]: ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10 2024-03-16T12:10:36.534 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:36 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:36.191+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:13.025675+0000 front 2024-03-16T12:09:13.025847+0000 (oldest deadline 2024-03-16T12:09:35.325827+0000) 2024-03-16T12:10:36.534 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:36 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:36.191+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:19.927745+0000 front 2024-03-16T12:09:19.927761+0000 (oldest deadline 2024-03-16T12:09:45.827828+0000) 2024-03-16T12:10:36.534 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:36 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:36.191+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.154:6804 osd.2 since back 2024-03-16T12:09:28.629320+0000 front 2024-03-16T12:09:28.629357+0000 (oldest deadline 2024-03-16T12:09:52.129472+0000) 2024-03-16T12:10:36.534 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:36 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:36.191+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-16T12:09:38.432099+0000 front 2024-03-16T12:09:38.432072+0000 (oldest deadline 2024-03-16T12:10:01.332006+0000) 2024-03-16T12:10:36.534 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:36 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:36.191+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.152:6812 osd.4 since back 2024-03-16T12:09:43.633241+0000 front 2024-03-16T12:09:43.633117+0000 (oldest deadline 2024-03-16T12:10:07.733019+0000) 2024-03-16T12:10:36.535 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:36 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:36.191+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.154:6812 osd.5 since back 2024-03-16T12:09:53.534564+0000 front 2024-03-16T12:09:53.534519+0000 (oldest deadline 2024-03-16T12:10:15.834516+0000) 2024-03-16T12:10:36.616 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:10:36 smithi152 podman[74604]: 2024-03-16 12:10:36.323714814 +0000 UTC m=+0.523311343 container remove 826ecf36251994139c6487aad9b1616b87b91373047d13f3a2078836653c0b6a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b) 2024-03-16T12:10:36.997 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:10:36 smithi152 podman[74696]: 2024-03-16 12:10:36.732669287 +0000 UTC m=+0.112449264 container create b98d549455fdd607baee708885d6c66f2fbf3e83836b570b1616327687332c4d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10-deactivate, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-03-16T12:10:36.997 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:10:36 smithi152 podman[74696]: 2024-03-16 12:10:36.63812766 +0000 UTC m=+0.017907619 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-16T12:10:36.997 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:10:36 smithi152 podman[74696]: 2024-03-16 12:10:36.926625537 +0000 UTC m=+0.306405509 container init b98d549455fdd607baee708885d6c66f2fbf3e83836b570b1616327687332c4d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307) 2024-03-16T12:10:36.997 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:10:36 smithi152 podman[74696]: 2024-03-16 12:10:36.929879749 +0000 UTC m=+0.309659698 container start b98d549455fdd607baee708885d6c66f2fbf3e83836b570b1616327687332c4d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10-deactivate, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-16T12:10:36.997 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:10:37 smithi152 podman[74696]: 2024-03-16 12:10:37.009682203 +0000 UTC m=+0.389462162 container attach b98d549455fdd607baee708885d6c66f2fbf3e83836b570b1616327687332c4d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10-deactivate, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, ceph=True) 2024-03-16T12:10:37.366 INFO:journalctl@ceph.osd.10.smithi152.stdout:Mar 16 12:10:37 smithi152 podman[74696]: 2024-03-16 12:10:37.216962892 +0000 UTC m=+0.596742841 container died b98d549455fdd607baee708885d6c66f2fbf3e83836b570b1616327687332c4d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-10-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-03-16T12:10:37.534 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:37 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:37.241+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:13.025675+0000 front 2024-03-16T12:09:13.025847+0000 (oldest deadline 2024-03-16T12:09:35.325827+0000) 2024-03-16T12:10:37.534 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:37 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:37.241+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:19.927745+0000 front 2024-03-16T12:09:19.927761+0000 (oldest deadline 2024-03-16T12:09:45.827828+0000) 2024-03-16T12:10:37.534 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:37 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:37.241+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.154:6804 osd.2 since back 2024-03-16T12:09:28.629320+0000 front 2024-03-16T12:09:28.629357+0000 (oldest deadline 2024-03-16T12:09:52.129472+0000) 2024-03-16T12:10:37.534 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:37 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:37.241+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-16T12:09:38.432099+0000 front 2024-03-16T12:09:38.432072+0000 (oldest deadline 2024-03-16T12:10:01.332006+0000) 2024-03-16T12:10:37.535 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:37 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:37.241+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.152:6812 osd.4 since back 2024-03-16T12:09:43.633241+0000 front 2024-03-16T12:09:43.633117+0000 (oldest deadline 2024-03-16T12:10:07.733019+0000) 2024-03-16T12:10:37.535 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:37 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:37.241+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.154:6812 osd.5 since back 2024-03-16T12:09:53.534564+0000 front 2024-03-16T12:09:53.534519+0000 (oldest deadline 2024-03-16T12:10:15.834516+0000) 2024-03-16T12:10:37.758 DEBUG:teuthology.orchestra.run.smithi152:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d1887f28-e389-11ee-95c9-87774f69a715@osd.10.service' 2024-03-16T12:10:37.839 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-16T12:10:37.839 INFO:tasks.cephadm.osd.10:Stopped osd.10 2024-03-16T12:10:37.840 INFO:tasks.cephadm.osd.11:Stopping osd.11... 2024-03-16T12:10:37.840 DEBUG:teuthology.orchestra.run.smithi154:> sudo systemctl stop ceph-d1887f28-e389-11ee-95c9-87774f69a715@osd.11 2024-03-16T12:10:38.150 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:37 smithi154 systemd[1]: Stopping Ceph osd.11 for d1887f28-e389-11ee-95c9-87774f69a715... 2024-03-16T12:10:38.534 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:38 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:38.162+0000 7f8cfc055700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.11 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-16T12:10:38.534 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:38 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:38.162+0000 7f8cfc055700 -1 osd.11 119 *** Got signal Terminated *** 2024-03-16T12:10:38.534 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:38 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:38.162+0000 7f8cfc055700 -1 osd.11 119 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-16T12:10:38.534 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:38 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:38.232+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:13.025675+0000 front 2024-03-16T12:09:13.025847+0000 (oldest deadline 2024-03-16T12:09:35.325827+0000) 2024-03-16T12:10:38.534 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:38 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:38.232+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:19.927745+0000 front 2024-03-16T12:09:19.927761+0000 (oldest deadline 2024-03-16T12:09:45.827828+0000) 2024-03-16T12:10:38.535 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:38 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:38.232+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.154:6804 osd.2 since back 2024-03-16T12:09:28.629320+0000 front 2024-03-16T12:09:28.629357+0000 (oldest deadline 2024-03-16T12:09:52.129472+0000) 2024-03-16T12:10:38.535 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:38 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:38.232+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-16T12:09:38.432099+0000 front 2024-03-16T12:09:38.432072+0000 (oldest deadline 2024-03-16T12:10:01.332006+0000) 2024-03-16T12:10:38.535 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:38 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:38.232+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.152:6812 osd.4 since back 2024-03-16T12:09:43.633241+0000 front 2024-03-16T12:09:43.633117+0000 (oldest deadline 2024-03-16T12:10:07.733019+0000) 2024-03-16T12:10:38.535 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:38 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:38.232+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.154:6812 osd.5 since back 2024-03-16T12:09:53.534564+0000 front 2024-03-16T12:09:53.534519+0000 (oldest deadline 2024-03-16T12:10:15.834516+0000) 2024-03-16T12:10:38.535 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:38 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:38.232+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.152:6820 osd.7 since back 2024-03-16T12:10:09.737757+0000 front 2024-03-16T12:10:09.737695+0000 (oldest deadline 2024-03-16T12:10:37.338097+0000) 2024-03-16T12:10:39.534 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:39 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:39.204+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:13.025675+0000 front 2024-03-16T12:09:13.025847+0000 (oldest deadline 2024-03-16T12:09:35.325827+0000) 2024-03-16T12:10:39.534 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:39 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:39.204+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:19.927745+0000 front 2024-03-16T12:09:19.927761+0000 (oldest deadline 2024-03-16T12:09:45.827828+0000) 2024-03-16T12:10:39.534 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:39 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:39.204+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.154:6804 osd.2 since back 2024-03-16T12:09:28.629320+0000 front 2024-03-16T12:09:28.629357+0000 (oldest deadline 2024-03-16T12:09:52.129472+0000) 2024-03-16T12:10:39.534 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:39 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:39.204+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-16T12:09:38.432099+0000 front 2024-03-16T12:09:38.432072+0000 (oldest deadline 2024-03-16T12:10:01.332006+0000) 2024-03-16T12:10:39.534 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:39 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:39.204+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.152:6812 osd.4 since back 2024-03-16T12:09:43.633241+0000 front 2024-03-16T12:09:43.633117+0000 (oldest deadline 2024-03-16T12:10:07.733019+0000) 2024-03-16T12:10:39.534 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:39 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:39.204+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.154:6812 osd.5 since back 2024-03-16T12:09:53.534564+0000 front 2024-03-16T12:09:53.534519+0000 (oldest deadline 2024-03-16T12:10:15.834516+0000) 2024-03-16T12:10:39.535 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:39 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:39.204+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.152:6820 osd.7 since back 2024-03-16T12:10:09.737757+0000 front 2024-03-16T12:10:09.737695+0000 (oldest deadline 2024-03-16T12:10:37.338097+0000) 2024-03-16T12:10:40.534 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:40 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:40.155+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:13.025675+0000 front 2024-03-16T12:09:13.025847+0000 (oldest deadline 2024-03-16T12:09:35.325827+0000) 2024-03-16T12:10:40.534 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:40 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:40.155+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:19.927745+0000 front 2024-03-16T12:09:19.927761+0000 (oldest deadline 2024-03-16T12:09:45.827828+0000) 2024-03-16T12:10:40.534 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:40 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:40.155+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.154:6804 osd.2 since back 2024-03-16T12:09:28.629320+0000 front 2024-03-16T12:09:28.629357+0000 (oldest deadline 2024-03-16T12:09:52.129472+0000) 2024-03-16T12:10:40.534 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:40 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:40.155+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-16T12:09:38.432099+0000 front 2024-03-16T12:09:38.432072+0000 (oldest deadline 2024-03-16T12:10:01.332006+0000) 2024-03-16T12:10:40.535 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:40 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:40.155+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.152:6812 osd.4 since back 2024-03-16T12:09:43.633241+0000 front 2024-03-16T12:09:43.633117+0000 (oldest deadline 2024-03-16T12:10:07.733019+0000) 2024-03-16T12:10:40.535 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:40 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:40.155+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.154:6812 osd.5 since back 2024-03-16T12:09:53.534564+0000 front 2024-03-16T12:09:53.534519+0000 (oldest deadline 2024-03-16T12:10:15.834516+0000) 2024-03-16T12:10:40.535 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:40 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:40.155+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.152:6820 osd.7 since back 2024-03-16T12:10:09.737757+0000 front 2024-03-16T12:10:09.737695+0000 (oldest deadline 2024-03-16T12:10:37.338097+0000) 2024-03-16T12:10:41.534 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:41 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:41.179+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:13.025675+0000 front 2024-03-16T12:09:13.025847+0000 (oldest deadline 2024-03-16T12:09:35.325827+0000) 2024-03-16T12:10:41.534 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:41 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:41.179+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:19.927745+0000 front 2024-03-16T12:09:19.927761+0000 (oldest deadline 2024-03-16T12:09:45.827828+0000) 2024-03-16T12:10:41.534 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:41 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:41.179+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.154:6804 osd.2 since back 2024-03-16T12:09:28.629320+0000 front 2024-03-16T12:09:28.629357+0000 (oldest deadline 2024-03-16T12:09:52.129472+0000) 2024-03-16T12:10:41.534 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:41 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:41.179+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-16T12:09:38.432099+0000 front 2024-03-16T12:09:38.432072+0000 (oldest deadline 2024-03-16T12:10:01.332006+0000) 2024-03-16T12:10:41.534 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:41 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:41.179+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.152:6812 osd.4 since back 2024-03-16T12:09:43.633241+0000 front 2024-03-16T12:09:43.633117+0000 (oldest deadline 2024-03-16T12:10:07.733019+0000) 2024-03-16T12:10:41.534 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:41 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:41.179+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.154:6812 osd.5 since back 2024-03-16T12:09:53.534564+0000 front 2024-03-16T12:09:53.534519+0000 (oldest deadline 2024-03-16T12:10:15.834516+0000) 2024-03-16T12:10:41.535 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:41 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:41.179+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.152:6820 osd.7 since back 2024-03-16T12:10:09.737757+0000 front 2024-03-16T12:10:09.737695+0000 (oldest deadline 2024-03-16T12:10:37.338097+0000) 2024-03-16T12:10:42.533 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:42 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:42.195+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-03-16T12:09:13.025675+0000 front 2024-03-16T12:09:13.025847+0000 (oldest deadline 2024-03-16T12:09:35.325827+0000) 2024-03-16T12:10:42.534 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:42 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:42.195+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-03-16T12:09:19.927745+0000 front 2024-03-16T12:09:19.927761+0000 (oldest deadline 2024-03-16T12:09:45.827828+0000) 2024-03-16T12:10:42.534 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:42 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:42.195+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.154:6804 osd.2 since back 2024-03-16T12:09:28.629320+0000 front 2024-03-16T12:09:28.629357+0000 (oldest deadline 2024-03-16T12:09:52.129472+0000) 2024-03-16T12:10:42.534 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:42 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:42.195+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.29:6814 osd.3 since back 2024-03-16T12:09:38.432099+0000 front 2024-03-16T12:09:38.432072+0000 (oldest deadline 2024-03-16T12:10:01.332006+0000) 2024-03-16T12:10:42.534 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:42 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:42.195+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.152:6812 osd.4 since back 2024-03-16T12:09:43.633241+0000 front 2024-03-16T12:09:43.633117+0000 (oldest deadline 2024-03-16T12:10:07.733019+0000) 2024-03-16T12:10:42.534 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:42 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:42.195+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.154:6812 osd.5 since back 2024-03-16T12:09:53.534564+0000 front 2024-03-16T12:09:53.534519+0000 (oldest deadline 2024-03-16T12:10:15.834516+0000) 2024-03-16T12:10:42.534 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:42 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11[61700]: 2024-03-16T12:10:42.195+0000 7f8cf4641700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.152:6820 osd.7 since back 2024-03-16T12:10:09.737757+0000 front 2024-03-16T12:10:09.737695+0000 (oldest deadline 2024-03-16T12:10:37.338097+0000) 2024-03-16T12:10:43.809 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:43 smithi154 podman[75562]: 2024-03-16 12:10:43.560495928 +0000 UTC m=+5.519381594 container died 50699c793913ff0c4d1abfc5a553e267c0987c44a8ec41866355528f9ef3a662 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, maintainer=Guillaume Abrioux ) 2024-03-16T12:10:44.150 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:43 smithi154 podman[75562]: 2024-03-16 12:10:43.920792926 +0000 UTC m=+5.879678586 container cleanup 50699c793913ff0c4d1abfc5a553e267c0987c44a8ec41866355528f9ef3a662 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-03-16T12:10:44.150 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:43 smithi154 bash[75562]: ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11 2024-03-16T12:10:44.465 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:44 smithi154 podman[75581]: 2024-03-16 12:10:44.163430526 +0000 UTC m=+0.603253740 container remove 50699c793913ff0c4d1abfc5a553e267c0987c44a8ec41866355528f9ef3a662 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-03-16T12:10:44.720 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:44 smithi154 podman[75674]: 2024-03-16 12:10:44.382089596 +0000 UTC m=+0.017269186 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-16T12:10:44.720 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:44 smithi154 podman[75674]: 2024-03-16 12:10:44.514349883 +0000 UTC m=+0.149529452 container create dd8579fea55c9ab59b34210f26861e84f36d1b130bb031f41ee5bc57b54c2c1d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11-deactivate, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.build-date=20240307, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, ceph=True) 2024-03-16T12:10:44.720 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:44 smithi154 podman[75674]: 2024-03-16 12:10:44.733373732 +0000 UTC m=+0.368553321 container init dd8579fea55c9ab59b34210f26861e84f36d1b130bb031f41ee5bc57b54c2c1d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11-deactivate, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-16T12:10:45.013 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:44 smithi154 podman[75674]: 2024-03-16 12:10:44.736410148 +0000 UTC m=+0.371589730 container start dd8579fea55c9ab59b34210f26861e84f36d1b130bb031f41ee5bc57b54c2c1d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11-deactivate, ceph=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5) 2024-03-16T12:10:45.013 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:44 smithi154 podman[75674]: 2024-03-16 12:10:44.824949993 +0000 UTC m=+0.460129592 container attach dd8579fea55c9ab59b34210f26861e84f36d1b130bb031f41ee5bc57b54c2c1d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-03-16T12:10:45.281 INFO:journalctl@ceph.osd.11.smithi154.stdout:Mar 16 12:10:45 smithi154 podman[75674]: 2024-03-16 12:10:45.026231412 +0000 UTC m=+0.661410991 container died dd8579fea55c9ab59b34210f26861e84f36d1b130bb031f41ee5bc57b54c2c1d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d1887f28-e389-11ee-95c9-87774f69a715-osd-11-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-03-16T12:10:45.617 DEBUG:teuthology.orchestra.run.smithi154:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d1887f28-e389-11ee-95c9-87774f69a715@osd.11.service' 2024-03-16T12:10:45.658 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-16T12:10:45.658 INFO:tasks.cephadm.osd.11:Stopped osd.11 2024-03-16T12:10:45.659 INFO:tasks.cephadm.mds.a:Stopping mds.a... 2024-03-16T12:10:45.659 DEBUG:teuthology.orchestra.run.smithi029:> sudo systemctl stop ceph-d1887f28-e389-11ee-95c9-87774f69a715@mds.a 2024-03-16T12:10:45.969 INFO:journalctl@ceph.mds.a.smithi029.stdout:Mar 16 12:10:45 smithi029 systemd[1]: Stopping Ceph mds.a for d1887f28-e389-11ee-95c9-87774f69a715... 2024-03-16T12:10:46.326 INFO:journalctl@ceph.mds.a.smithi029.stdout:Mar 16 12:10:45 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mds-a[74212]: 2024-03-16T12:10:45.981+0000 7f1fb1a91700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-16T12:10:46.326 INFO:journalctl@ceph.mds.a.smithi029.stdout:Mar 16 12:10:45 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mds-a[74212]: 2024-03-16T12:10:45.981+0000 7f1fb1a91700 -1 mds.a *** got signal Terminated *** 2024-03-16T12:10:50.255 INFO:journalctl@ceph.mds.a.smithi029.stdout:Mar 16 12:10:49 smithi029 podman[95927]: 2024-03-16 12:10:49.991944628 +0000 UTC m=+4.123247841 container died e452e9b5df2287e9087cf48fe7d7bda47c831086bf4c87e0d8a4dd1ca16e441e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d1887f28-e389-11ee-95c9-87774f69a715-mds-a, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux ) 2024-03-16T12:10:50.576 INFO:journalctl@ceph.mds.a.smithi029.stdout:Mar 16 12:10:50 smithi029 podman[95927]: 2024-03-16 12:10:50.368399764 +0000 UTC m=+4.499702978 container cleanup e452e9b5df2287e9087cf48fe7d7bda47c831086bf4c87e0d8a4dd1ca16e441e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d1887f28-e389-11ee-95c9-87774f69a715-mds-a, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-03-16T12:10:50.576 INFO:journalctl@ceph.mds.a.smithi029.stdout:Mar 16 12:10:50 smithi029 bash[95927]: ceph-d1887f28-e389-11ee-95c9-87774f69a715-mds-a 2024-03-16T12:10:50.768 DEBUG:teuthology.orchestra.run.smithi029:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d1887f28-e389-11ee-95c9-87774f69a715@mds.a.service' 2024-03-16T12:10:50.804 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-16T12:10:50.804 INFO:tasks.cephadm.mds.a:Stopped mds.a 2024-03-16T12:10:50.804 INFO:tasks.cephadm.mds.d:Stopping mds.d... 2024-03-16T12:10:50.804 DEBUG:teuthology.orchestra.run.smithi029:> sudo systemctl stop ceph-d1887f28-e389-11ee-95c9-87774f69a715@mds.d 2024-03-16T12:10:51.212 INFO:journalctl@ceph.mds.d.smithi029.stdout:Mar 16 12:10:50 smithi029 systemd[1]: Stopping Ceph mds.d for d1887f28-e389-11ee-95c9-87774f69a715... 2024-03-16T12:10:51.576 INFO:journalctl@ceph.mds.d.smithi029.stdout:Mar 16 12:10:51 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mds-d[73221]: 2024-03-16T12:10:51.224+0000 7f5bf3efe700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.d -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-16T12:10:51.576 INFO:journalctl@ceph.mds.d.smithi029.stdout:Mar 16 12:10:51 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mds-d[73221]: 2024-03-16T12:10:51.224+0000 7f5bf3efe700 -1 mds.d *** got signal Terminated *** 2024-03-16T12:10:54.707 INFO:journalctl@ceph.mds.d.smithi029.stdout:Mar 16 12:10:54 smithi029 podman[96046]: 2024-03-16 12:10:54.455909832 +0000 UTC m=+3.418419636 container died 49650e1408b88f1fd89fd899d4f7ebab3603c9d10466d9f86c075fa5a348975b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d1887f28-e389-11ee-95c9-87774f69a715-mds-d, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-03-16T12:10:55.075 INFO:journalctl@ceph.mds.d.smithi029.stdout:Mar 16 12:10:54 smithi029 podman[96046]: 2024-03-16 12:10:54.811692582 +0000 UTC m=+3.774202425 container cleanup 49650e1408b88f1fd89fd899d4f7ebab3603c9d10466d9f86c075fa5a348975b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d1887f28-e389-11ee-95c9-87774f69a715-mds-d, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-16T12:10:55.076 INFO:journalctl@ceph.mds.d.smithi029.stdout:Mar 16 12:10:54 smithi029 bash[96046]: ceph-d1887f28-e389-11ee-95c9-87774f69a715-mds-d 2024-03-16T12:10:55.226 DEBUG:teuthology.orchestra.run.smithi029:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d1887f28-e389-11ee-95c9-87774f69a715@mds.d.service' 2024-03-16T12:10:55.261 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-16T12:10:55.261 INFO:tasks.cephadm.mds.d:Stopped mds.d 2024-03-16T12:10:55.261 INFO:tasks.cephadm.mds.g:Stopping mds.g... 2024-03-16T12:10:55.262 DEBUG:teuthology.orchestra.run.smithi029:> sudo systemctl stop ceph-d1887f28-e389-11ee-95c9-87774f69a715@mds.g 2024-03-16T12:10:55.576 INFO:journalctl@ceph.mds.g.smithi029.stdout:Mar 16 12:10:55 smithi029 systemd[1]: Stopping Ceph mds.g for d1887f28-e389-11ee-95c9-87774f69a715... 2024-03-16T12:10:56.076 INFO:journalctl@ceph.mds.g.smithi029.stdout:Mar 16 12:10:55 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mds-g[73836]: 2024-03-16T12:10:55.624+0000 7f93b57fb700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.g -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-16T12:10:56.076 INFO:journalctl@ceph.mds.g.smithi029.stdout:Mar 16 12:10:55 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mds-g[73836]: 2024-03-16T12:10:55.624+0000 7f93b57fb700 -1 mds.g *** got signal Terminated *** 2024-03-16T12:10:59.905 INFO:journalctl@ceph.mds.g.smithi029.stdout:Mar 16 12:10:59 smithi029 podman[96165]: 2024-03-16 12:10:59.633512108 +0000 UTC m=+4.142916984 container died 37fc43af5b3fae320d6d8fa8c1726415d83f134d94b06a6987d793a746a684d2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d1887f28-e389-11ee-95c9-87774f69a715-mds-g, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2) 2024-03-16T12:11:00.251 INFO:journalctl@ceph.mds.g.smithi029.stdout:Mar 16 12:10:59 smithi029 podman[96165]: 2024-03-16 12:10:59.995250523 +0000 UTC m=+4.504655396 container cleanup 37fc43af5b3fae320d6d8fa8c1726415d83f134d94b06a6987d793a746a684d2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d1887f28-e389-11ee-95c9-87774f69a715-mds-g, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5) 2024-03-16T12:11:00.251 INFO:journalctl@ceph.mds.g.smithi029.stdout:Mar 16 12:10:59 smithi029 bash[96165]: ceph-d1887f28-e389-11ee-95c9-87774f69a715-mds-g 2024-03-16T12:11:00.411 DEBUG:teuthology.orchestra.run.smithi029:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d1887f28-e389-11ee-95c9-87774f69a715@mds.g.service' 2024-03-16T12:11:00.444 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-16T12:11:00.445 INFO:tasks.cephadm.mds.g:Stopped mds.g 2024-03-16T12:11:00.445 INFO:tasks.cephadm.mds.j:Stopping mds.j... 2024-03-16T12:11:00.445 DEBUG:teuthology.orchestra.run.smithi029:> sudo systemctl stop ceph-d1887f28-e389-11ee-95c9-87774f69a715@mds.j 2024-03-16T12:11:00.826 INFO:journalctl@ceph.mds.j.smithi029.stdout:Mar 16 12:11:00 smithi029 systemd[1]: Stopping Ceph mds.j for d1887f28-e389-11ee-95c9-87774f69a715... 2024-03-16T12:11:00.826 INFO:journalctl@ceph.mds.j.smithi029.stdout:Mar 16 12:11:00 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mds-j[72870]: 2024-03-16T12:11:00.767+0000 7f717c00f700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.j -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-16T12:11:00.826 INFO:journalctl@ceph.mds.j.smithi029.stdout:Mar 16 12:11:00 smithi029 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mds-j[72870]: 2024-03-16T12:11:00.767+0000 7f717c00f700 -1 mds.j *** got signal Terminated *** 2024-03-16T12:11:03.765 INFO:journalctl@ceph.mds.j.smithi029.stdout:Mar 16 12:11:03 smithi029 podman[96287]: 2024-03-16 12:11:03.517775358 +0000 UTC m=+2.844402263 container died edc59c83e563e2e5f4d7002bd75413a1c46a25d1c3acec0ff93ed46c1504bce1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d1887f28-e389-11ee-95c9-87774f69a715-mds-j, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-03-16T12:11:04.026 INFO:journalctl@ceph.mds.j.smithi029.stdout:Mar 16 12:11:03 smithi029 podman[96287]: 2024-03-16 12:11:03.854423915 +0000 UTC m=+3.181050816 container cleanup edc59c83e563e2e5f4d7002bd75413a1c46a25d1c3acec0ff93ed46c1504bce1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d1887f28-e389-11ee-95c9-87774f69a715-mds-j, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5) 2024-03-16T12:11:04.027 INFO:journalctl@ceph.mds.j.smithi029.stdout:Mar 16 12:11:03 smithi029 bash[96287]: ceph-d1887f28-e389-11ee-95c9-87774f69a715-mds-j 2024-03-16T12:11:04.271 DEBUG:teuthology.orchestra.run.smithi029:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d1887f28-e389-11ee-95c9-87774f69a715@mds.j.service' 2024-03-16T12:11:04.294 INFO:journalctl@ceph.mds.j.smithi029.stdout:Mar 16 12:11:04 smithi029 podman[96301]: 2024-03-16 12:11:04.039861726 +0000 UTC m=+0.513163605 container remove edc59c83e563e2e5f4d7002bd75413a1c46a25d1c3acec0ff93ed46c1504bce1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d1887f28-e389-11ee-95c9-87774f69a715-mds-j, GIT_CLEAN=True, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, io.buildah.version=1.33.5) 2024-03-16T12:11:04.294 INFO:journalctl@ceph.mds.j.smithi029.stdout:Mar 16 12:11:04 smithi029 systemd[1]: ceph-d1887f28-e389-11ee-95c9-87774f69a715@mds.j.service: Deactivated successfully. 2024-03-16T12:11:04.295 INFO:journalctl@ceph.mds.j.smithi029.stdout:Mar 16 12:11:04 smithi029 systemd[1]: Stopped Ceph mds.j for d1887f28-e389-11ee-95c9-87774f69a715. 2024-03-16T12:11:04.295 INFO:journalctl@ceph.mds.j.smithi029.stdout:Mar 16 12:11:04 smithi029 systemd[1]: ceph-d1887f28-e389-11ee-95c9-87774f69a715@mds.j.service: Consumed 1.102s CPU time. 2024-03-16T12:11:04.305 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-16T12:11:04.305 INFO:tasks.cephadm.mds.j:Stopped mds.j 2024-03-16T12:11:04.306 INFO:tasks.cephadm.mds.b:Stopping mds.b... 2024-03-16T12:11:04.306 DEBUG:teuthology.orchestra.run.smithi152:> sudo systemctl stop ceph-d1887f28-e389-11ee-95c9-87774f69a715@mds.b 2024-03-16T12:11:04.600 INFO:journalctl@ceph.mds.b.smithi152.stdout:Mar 16 12:11:04 smithi152 systemd[1]: Stopping Ceph mds.b for d1887f28-e389-11ee-95c9-87774f69a715... 2024-03-16T12:11:04.863 INFO:journalctl@ceph.mds.b.smithi152.stdout:Mar 16 12:11:04 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mds-b[67242]: 2024-03-16T12:11:04.612+0000 7ff504060700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-16T12:11:04.863 INFO:journalctl@ceph.mds.b.smithi152.stdout:Mar 16 12:11:04 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mds-b[67242]: 2024-03-16T12:11:04.612+0000 7ff504060700 -1 mds.b *** got signal Terminated *** 2024-03-16T12:11:07.992 INFO:journalctl@ceph.mds.b.smithi152.stdout:Mar 16 12:11:07 smithi152 podman[74803]: 2024-03-16 12:11:07.738892156 +0000 UTC m=+3.228390840 container died a2a2b2c4d0e786b7df5056eadc30a9f614b2ed272f78b64fa3863b7e3514d204 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d1887f28-e389-11ee-95c9-87774f69a715-mds-b, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-03-16T12:11:08.266 INFO:journalctl@ceph.mds.b.smithi152.stdout:Mar 16 12:11:08 smithi152 podman[74803]: 2024-03-16 12:11:08.078633179 +0000 UTC m=+3.568131845 container cleanup a2a2b2c4d0e786b7df5056eadc30a9f614b2ed272f78b64fa3863b7e3514d204 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d1887f28-e389-11ee-95c9-87774f69a715-mds-b, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-03-16T12:11:08.267 INFO:journalctl@ceph.mds.b.smithi152.stdout:Mar 16 12:11:08 smithi152 bash[74803]: ceph-d1887f28-e389-11ee-95c9-87774f69a715-mds-b 2024-03-16T12:11:08.478 DEBUG:teuthology.orchestra.run.smithi152:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d1887f28-e389-11ee-95c9-87774f69a715@mds.b.service' 2024-03-16T12:11:08.516 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-16T12:11:08.516 INFO:tasks.cephadm.mds.b:Stopped mds.b 2024-03-16T12:11:08.516 INFO:tasks.cephadm.mds.e:Stopping mds.e... 2024-03-16T12:11:08.516 DEBUG:teuthology.orchestra.run.smithi152:> sudo systemctl stop ceph-d1887f28-e389-11ee-95c9-87774f69a715@mds.e 2024-03-16T12:11:08.866 INFO:journalctl@ceph.mds.e.smithi152.stdout:Mar 16 12:11:08 smithi152 systemd[1]: Stopping Ceph mds.e for d1887f28-e389-11ee-95c9-87774f69a715... 2024-03-16T12:11:08.866 INFO:journalctl@ceph.mds.e.smithi152.stdout:Mar 16 12:11:08 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mds-e[66619]: 2024-03-16T12:11:08.831+0000 7f886f230700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.e -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-16T12:11:08.866 INFO:journalctl@ceph.mds.e.smithi152.stdout:Mar 16 12:11:08 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mds-e[66619]: 2024-03-16T12:11:08.831+0000 7f886f230700 -1 mds.e *** got signal Terminated *** 2024-03-16T12:11:11.982 INFO:journalctl@ceph.mds.e.smithi152.stdout:Mar 16 12:11:11 smithi152 podman[74935]: 2024-03-16 12:11:11.732516041 +0000 UTC m=+2.985781839 container died 822bc521a8b8be03443426a1111409f3034b08f2308590e2bde6d3b2c1606c40 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d1887f28-e389-11ee-95c9-87774f69a715-mds-e, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True) 2024-03-16T12:11:11.982 INFO:journalctl@ceph.mds.e.smithi152.stdout:Mar 16 12:11:11 smithi152 podman[74935]: 2024-03-16 12:11:11.973807738 +0000 UTC m=+3.227073554 container cleanup 822bc521a8b8be03443426a1111409f3034b08f2308590e2bde6d3b2c1606c40 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d1887f28-e389-11ee-95c9-87774f69a715-mds-e, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0) 2024-03-16T12:11:11.982 INFO:journalctl@ceph.mds.e.smithi152.stdout:Mar 16 12:11:11 smithi152 bash[74935]: ceph-d1887f28-e389-11ee-95c9-87774f69a715-mds-e 2024-03-16T12:11:12.374 DEBUG:teuthology.orchestra.run.smithi152:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d1887f28-e389-11ee-95c9-87774f69a715@mds.e.service' 2024-03-16T12:11:12.410 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-16T12:11:12.410 INFO:tasks.cephadm.mds.e:Stopped mds.e 2024-03-16T12:11:12.411 INFO:tasks.cephadm.mds.h:Stopping mds.h... 2024-03-16T12:11:12.411 DEBUG:teuthology.orchestra.run.smithi152:> sudo systemctl stop ceph-d1887f28-e389-11ee-95c9-87774f69a715@mds.h 2024-03-16T12:11:12.731 INFO:journalctl@ceph.mds.h.smithi152.stdout:Mar 16 12:11:12 smithi152 systemd[1]: Stopping Ceph mds.h for d1887f28-e389-11ee-95c9-87774f69a715... 2024-03-16T12:11:13.115 INFO:journalctl@ceph.mds.h.smithi152.stdout:Mar 16 12:11:12 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mds-h[66932]: 2024-03-16T12:11:12.743+0000 7f85fffb7700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.h -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-16T12:11:13.115 INFO:journalctl@ceph.mds.h.smithi152.stdout:Mar 16 12:11:12 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mds-h[66932]: 2024-03-16T12:11:12.743+0000 7f85fffb7700 -1 mds.h *** got signal Terminated *** 2024-03-16T12:11:15.982 INFO:journalctl@ceph.mds.h.smithi152.stdout:Mar 16 12:11:15 smithi152 podman[75059]: 2024-03-16 12:11:15.733542874 +0000 UTC m=+3.095386344 container died 873b7794f4d43f8f59db7a9c4cad30c1cb499bf11c72872ce483c5fb1538315a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d1887f28-e389-11ee-95c9-87774f69a715-mds-h, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-03-16T12:11:16.339 INFO:journalctl@ceph.mds.h.smithi152.stdout:Mar 16 12:11:16 smithi152 podman[75059]: 2024-03-16 12:11:16.093762711 +0000 UTC m=+3.455606180 container cleanup 873b7794f4d43f8f59db7a9c4cad30c1cb499bf11c72872ce483c5fb1538315a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d1887f28-e389-11ee-95c9-87774f69a715-mds-h, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307) 2024-03-16T12:11:16.339 INFO:journalctl@ceph.mds.h.smithi152.stdout:Mar 16 12:11:16 smithi152 bash[75059]: ceph-d1887f28-e389-11ee-95c9-87774f69a715-mds-h 2024-03-16T12:11:16.490 DEBUG:teuthology.orchestra.run.smithi152:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d1887f28-e389-11ee-95c9-87774f69a715@mds.h.service' 2024-03-16T12:11:16.524 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-16T12:11:16.525 INFO:tasks.cephadm.mds.h:Stopped mds.h 2024-03-16T12:11:16.525 INFO:tasks.cephadm.mds.k:Stopping mds.k... 2024-03-16T12:11:16.525 DEBUG:teuthology.orchestra.run.smithi152:> sudo systemctl stop ceph-d1887f28-e389-11ee-95c9-87774f69a715@mds.k 2024-03-16T12:11:16.865 INFO:journalctl@ceph.mds.k.smithi152.stdout:Mar 16 12:11:16 smithi152 systemd[1]: Stopping Ceph mds.k for d1887f28-e389-11ee-95c9-87774f69a715... 2024-03-16T12:11:16.866 INFO:journalctl@ceph.mds.k.smithi152.stdout:Mar 16 12:11:16 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mds-k[66305]: 2024-03-16T12:11:16.855+0000 7f4ba3546700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.k -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-16T12:11:16.866 INFO:journalctl@ceph.mds.k.smithi152.stdout:Mar 16 12:11:16 smithi152 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mds-k[66305]: 2024-03-16T12:11:16.855+0000 7f4ba3546700 -1 mds.k *** got signal Terminated *** 2024-03-16T12:11:20.701 INFO:journalctl@ceph.mds.k.smithi152.stdout:Mar 16 12:11:20 smithi152 podman[75182]: 2024-03-16 12:11:20.461298816 +0000 UTC m=+3.712428599 container died a35e36fe4008d594250a7847588998bc2d7b8f212639a86a45552c604d974a38 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d1887f28-e389-11ee-95c9-87774f69a715-mds-k, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307) 2024-03-16T12:11:21.115 INFO:journalctl@ceph.mds.k.smithi152.stdout:Mar 16 12:11:20 smithi152 podman[75182]: 2024-03-16 12:11:20.788571087 +0000 UTC m=+4.039700868 container cleanup a35e36fe4008d594250a7847588998bc2d7b8f212639a86a45552c604d974a38 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d1887f28-e389-11ee-95c9-87774f69a715-mds-k, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0) 2024-03-16T12:11:21.115 INFO:journalctl@ceph.mds.k.smithi152.stdout:Mar 16 12:11:20 smithi152 bash[75182]: ceph-d1887f28-e389-11ee-95c9-87774f69a715-mds-k 2024-03-16T12:11:21.116 INFO:journalctl@ceph.mds.k.smithi152.stdout:Mar 16 12:11:20 smithi152 podman[75196]: 2024-03-16 12:11:20.956091783 +0000 UTC m=+0.497570604 container remove a35e36fe4008d594250a7847588998bc2d7b8f212639a86a45552c604d974a38 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d1887f28-e389-11ee-95c9-87774f69a715-mds-k, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-16T12:11:21.137 DEBUG:teuthology.orchestra.run.smithi152:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d1887f28-e389-11ee-95c9-87774f69a715@mds.k.service' 2024-03-16T12:11:21.171 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-16T12:11:21.171 INFO:tasks.cephadm.mds.k:Stopped mds.k 2024-03-16T12:11:21.172 INFO:tasks.cephadm.mds.c:Stopping mds.c... 2024-03-16T12:11:21.172 DEBUG:teuthology.orchestra.run.smithi154:> sudo systemctl stop ceph-d1887f28-e389-11ee-95c9-87774f69a715@mds.c 2024-03-16T12:11:21.456 INFO:journalctl@ceph.mds.c.smithi154.stdout:Mar 16 12:11:21 smithi154 systemd[1]: Stopping Ceph mds.c for d1887f28-e389-11ee-95c9-87774f69a715... 2024-03-16T12:11:21.718 INFO:journalctl@ceph.mds.c.smithi154.stdout:Mar 16 12:11:21 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mds-c[66727]: 2024-03-16T12:11:21.469+0000 7fc770496700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-16T12:11:21.718 INFO:journalctl@ceph.mds.c.smithi154.stdout:Mar 16 12:11:21 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mds-c[66727]: 2024-03-16T12:11:21.469+0000 7fc770496700 -1 mds.c *** got signal Terminated *** 2024-03-16T12:11:24.033 INFO:journalctl@ceph.mds.c.smithi154.stdout:Mar 16 12:11:23 smithi154 podman[75792]: 2024-03-16 12:11:23.699586678 +0000 UTC m=+2.327099287 container died 36cab96ff53329afe85b3a03b8812e86d3f5c326e2244a77e7c4b5f47f74ee43 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d1887f28-e389-11ee-95c9-87774f69a715-mds-c, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-03-16T12:11:24.414 INFO:journalctl@ceph.mds.c.smithi154.stdout:Mar 16 12:11:24 smithi154 podman[75792]: 2024-03-16 12:11:24.218216913 +0000 UTC m=+2.845729524 container cleanup 36cab96ff53329afe85b3a03b8812e86d3f5c326e2244a77e7c4b5f47f74ee43 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d1887f28-e389-11ee-95c9-87774f69a715-mds-c, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-03-16T12:11:24.414 INFO:journalctl@ceph.mds.c.smithi154.stdout:Mar 16 12:11:24 smithi154 bash[75792]: ceph-d1887f28-e389-11ee-95c9-87774f69a715-mds-c 2024-03-16T12:11:24.635 DEBUG:teuthology.orchestra.run.smithi154:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d1887f28-e389-11ee-95c9-87774f69a715@mds.c.service' 2024-03-16T12:11:24.673 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-16T12:11:24.673 INFO:tasks.cephadm.mds.c:Stopped mds.c 2024-03-16T12:11:24.673 INFO:tasks.cephadm.mds.f:Stopping mds.f... 2024-03-16T12:11:24.674 DEBUG:teuthology.orchestra.run.smithi154:> sudo systemctl stop ceph-d1887f28-e389-11ee-95c9-87774f69a715@mds.f 2024-03-16T12:11:25.009 INFO:journalctl@ceph.mds.f.smithi154.stdout:Mar 16 12:11:24 smithi154 systemd[1]: Stopping Ceph mds.f for d1887f28-e389-11ee-95c9-87774f69a715... 2024-03-16T12:11:25.283 INFO:journalctl@ceph.mds.f.smithi154.stdout:Mar 16 12:11:25 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mds-f[66421]: 2024-03-16T12:11:25.021+0000 7f07f5331700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.f -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-16T12:11:25.283 INFO:journalctl@ceph.mds.f.smithi154.stdout:Mar 16 12:11:25 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mds-f[66421]: 2024-03-16T12:11:25.021+0000 7f07f5331700 -1 mds.f *** got signal Terminated *** 2024-03-16T12:11:28.033 INFO:journalctl@ceph.mds.f.smithi154.stdout:Mar 16 12:11:27 smithi154 podman[75926]: 2024-03-16 12:11:27.695436698 +0000 UTC m=+2.787876718 container died 3e849fc2b38e390e5e52a846bcefee2dfab94f38b0d914e6b2d19b0b1761baf7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d1887f28-e389-11ee-95c9-87774f69a715-mds-f, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b) 2024-03-16T12:11:28.418 INFO:journalctl@ceph.mds.f.smithi154.stdout:Mar 16 12:11:28 smithi154 podman[75926]: 2024-03-16 12:11:28.213486858 +0000 UTC m=+3.305926885 container cleanup 3e849fc2b38e390e5e52a846bcefee2dfab94f38b0d914e6b2d19b0b1761baf7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d1887f28-e389-11ee-95c9-87774f69a715-mds-f, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True) 2024-03-16T12:11:28.418 INFO:journalctl@ceph.mds.f.smithi154.stdout:Mar 16 12:11:28 smithi154 bash[75926]: ceph-d1887f28-e389-11ee-95c9-87774f69a715-mds-f 2024-03-16T12:11:28.668 INFO:journalctl@ceph.mds.f.smithi154.stdout:Mar 16 12:11:28 smithi154 podman[75941]: 2024-03-16 12:11:28.431325151 +0000 UTC m=+0.733560718 container remove 3e849fc2b38e390e5e52a846bcefee2dfab94f38b0d914e6b2d19b0b1761baf7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d1887f28-e389-11ee-95c9-87774f69a715-mds-f, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-16T12:11:28.669 INFO:journalctl@ceph.mds.f.smithi154.stdout:Mar 16 12:11:28 smithi154 systemd[1]: ceph-d1887f28-e389-11ee-95c9-87774f69a715@mds.f.service: Deactivated successfully. 2024-03-16T12:11:28.669 INFO:journalctl@ceph.mds.f.smithi154.stdout:Mar 16 12:11:28 smithi154 systemd[1]: Stopped Ceph mds.f for d1887f28-e389-11ee-95c9-87774f69a715. 2024-03-16T12:11:28.669 INFO:journalctl@ceph.mds.f.smithi154.stdout:Mar 16 12:11:28 smithi154 systemd[1]: ceph-d1887f28-e389-11ee-95c9-87774f69a715@mds.f.service: Consumed 3.362s CPU time. 2024-03-16T12:11:28.670 DEBUG:teuthology.orchestra.run.smithi154:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d1887f28-e389-11ee-95c9-87774f69a715@mds.f.service' 2024-03-16T12:11:28.706 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-16T12:11:28.706 INFO:tasks.cephadm.mds.f:Stopped mds.f 2024-03-16T12:11:28.706 INFO:tasks.cephadm.mds.i:Stopping mds.i... 2024-03-16T12:11:28.706 DEBUG:teuthology.orchestra.run.smithi154:> sudo systemctl stop ceph-d1887f28-e389-11ee-95c9-87774f69a715@mds.i 2024-03-16T12:11:29.021 INFO:journalctl@ceph.mds.i.smithi154.stdout:Mar 16 12:11:28 smithi154 systemd[1]: Stopping Ceph mds.i for d1887f28-e389-11ee-95c9-87774f69a715... 2024-03-16T12:11:29.283 INFO:journalctl@ceph.mds.i.smithi154.stdout:Mar 16 12:11:29 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mds-i[67031]: 2024-03-16T12:11:29.033+0000 7fa415885700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.i -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-16T12:11:29.283 INFO:journalctl@ceph.mds.i.smithi154.stdout:Mar 16 12:11:29 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mds-i[67031]: 2024-03-16T12:11:29.033+0000 7fa415885700 -1 mds.i *** got signal Terminated *** 2024-03-16T12:11:32.033 INFO:journalctl@ceph.mds.i.smithi154.stdout:Mar 16 12:11:31 smithi154 podman[76048]: 2024-03-16 12:11:31.697152152 +0000 UTC m=+2.767353742 container died 31362b9e990cfc058e042686dd711c139329910e217a0f0b652db955ba522b8e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d1887f28-e389-11ee-95c9-87774f69a715-mds-i, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-03-16T12:11:32.421 INFO:journalctl@ceph.mds.i.smithi154.stdout:Mar 16 12:11:32 smithi154 podman[76048]: 2024-03-16 12:11:32.166913929 +0000 UTC m=+3.237115585 container cleanup 31362b9e990cfc058e042686dd711c139329910e217a0f0b652db955ba522b8e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d1887f28-e389-11ee-95c9-87774f69a715-mds-i, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307) 2024-03-16T12:11:32.421 INFO:journalctl@ceph.mds.i.smithi154.stdout:Mar 16 12:11:32 smithi154 bash[76048]: ceph-d1887f28-e389-11ee-95c9-87774f69a715-mds-i 2024-03-16T12:11:32.564 DEBUG:teuthology.orchestra.run.smithi154:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d1887f28-e389-11ee-95c9-87774f69a715@mds.i.service' 2024-03-16T12:11:32.599 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-16T12:11:32.599 INFO:tasks.cephadm.mds.i:Stopped mds.i 2024-03-16T12:11:32.599 INFO:tasks.cephadm.mds.l:Stopping mds.l... 2024-03-16T12:11:32.599 DEBUG:teuthology.orchestra.run.smithi154:> sudo systemctl stop ceph-d1887f28-e389-11ee-95c9-87774f69a715@mds.l 2024-03-16T12:11:33.033 INFO:journalctl@ceph.mds.l.smithi154.stdout:Mar 16 12:11:32 smithi154 systemd[1]: Stopping Ceph mds.l for d1887f28-e389-11ee-95c9-87774f69a715... 2024-03-16T12:11:33.033 INFO:journalctl@ceph.mds.l.smithi154.stdout:Mar 16 12:11:32 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mds-l[66109]: 2024-03-16T12:11:32.919+0000 7fdffef25700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.l -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-16T12:11:33.033 INFO:journalctl@ceph.mds.l.smithi154.stdout:Mar 16 12:11:32 smithi154 ceph-d1887f28-e389-11ee-95c9-87774f69a715-mds-l[66109]: 2024-03-16T12:11:32.919+0000 7fdffef25700 -1 mds.l *** got signal Terminated *** 2024-03-16T12:11:37.283 INFO:journalctl@ceph.mds.l.smithi154.stdout:Mar 16 12:11:36 smithi154 podman[76169]: 2024-03-16 12:11:36.926156808 +0000 UTC m=+4.091269124 container died 5a6f0df597bee85538643390a4f3f7a6ce3e33e3a2112a5754295e1032b0d42d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d1887f28-e389-11ee-95c9-87774f69a715-mds-l, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-03-16T12:11:37.632 INFO:journalctl@ceph.mds.l.smithi154.stdout:Mar 16 12:11:37 smithi154 podman[76169]: 2024-03-16 12:11:37.470101263 +0000 UTC m=+4.635213548 container cleanup 5a6f0df597bee85538643390a4f3f7a6ce3e33e3a2112a5754295e1032b0d42d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d1887f28-e389-11ee-95c9-87774f69a715-mds-l, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-16T12:11:37.632 INFO:journalctl@ceph.mds.l.smithi154.stdout:Mar 16 12:11:37 smithi154 bash[76169]: ceph-d1887f28-e389-11ee-95c9-87774f69a715-mds-l 2024-03-16T12:11:37.868 DEBUG:teuthology.orchestra.run.smithi154:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d1887f28-e389-11ee-95c9-87774f69a715@mds.l.service' 2024-03-16T12:11:37.891 INFO:journalctl@ceph.mds.l.smithi154.stdout:Mar 16 12:11:37 smithi154 podman[76184]: 2024-03-16 12:11:37.645919128 +0000 UTC m=+0.713642289 container remove 5a6f0df597bee85538643390a4f3f7a6ce3e33e3a2112a5754295e1032b0d42d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d1887f28-e389-11ee-95c9-87774f69a715-mds-l, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-16T12:11:37.891 INFO:journalctl@ceph.mds.l.smithi154.stdout:Mar 16 12:11:37 smithi154 systemd[1]: ceph-d1887f28-e389-11ee-95c9-87774f69a715@mds.l.service: Deactivated successfully. 2024-03-16T12:11:37.891 INFO:journalctl@ceph.mds.l.smithi154.stdout:Mar 16 12:11:37 smithi154 systemd[1]: Stopped Ceph mds.l for d1887f28-e389-11ee-95c9-87774f69a715. 2024-03-16T12:11:37.892 INFO:journalctl@ceph.mds.l.smithi154.stdout:Mar 16 12:11:37 smithi154 systemd[1]: ceph-d1887f28-e389-11ee-95c9-87774f69a715@mds.l.service: Consumed 1.073s CPU time. 2024-03-16T12:11:37.902 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-16T12:11:37.902 INFO:tasks.cephadm.mds.l:Stopped mds.l 2024-03-16T12:11:37.902 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid d1887f28-e389-11ee-95c9-87774f69a715 --force --keep-logs 2024-03-16T12:11:38.099 INFO:teuthology.orchestra.run.smithi029.stdout:Deleting cluster with fsid: d1887f28-e389-11ee-95c9-87774f69a715 2024-03-16T12:11:40.397 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid d1887f28-e389-11ee-95c9-87774f69a715 --force --keep-logs 2024-03-16T12:11:40.589 INFO:teuthology.orchestra.run.smithi152.stdout:Deleting cluster with fsid: d1887f28-e389-11ee-95c9-87774f69a715 2024-03-16T12:11:43.207 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid d1887f28-e389-11ee-95c9-87774f69a715 --force --keep-logs 2024-03-16T12:11:43.400 INFO:teuthology.orchestra.run.smithi154.stdout:Deleting cluster with fsid: d1887f28-e389-11ee-95c9-87774f69a715 2024-03-16T12:11:45.803 DEBUG:teuthology.orchestra.run.smithi029:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-16T12:11:45.837 DEBUG:teuthology.orchestra.run.smithi152:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-16T12:11:45.872 DEBUG:teuthology.orchestra.run.smithi154:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-16T12:11:45.900 INFO:tasks.cephadm:Archiving crash dumps... 2024-03-16T12:11:45.903 DEBUG:teuthology.misc:Transferring archived files from smithi029:/var/lib/ceph/d1887f28-e389-11ee-95c9-87774f69a715/crash to /home/teuthworker/archive/pdonnell-2024-03-15_13:00:41-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7602417/remote/smithi029/crash 2024-03-16T12:11:45.903 DEBUG:teuthology.orchestra.run.smithi029:> sudo tar c -f - -C /var/lib/ceph/d1887f28-e389-11ee-95c9-87774f69a715/crash -- . 2024-03-16T12:11:45.950 INFO:teuthology.orchestra.run.smithi029.stderr:tar: /var/lib/ceph/d1887f28-e389-11ee-95c9-87774f69a715/crash: Cannot open: No such file or directory 2024-03-16T12:11:45.951 INFO:teuthology.orchestra.run.smithi029.stderr:tar: Error is not recoverable: exiting now 2024-03-16T12:11:45.952 DEBUG:teuthology.misc:Transferring archived files from smithi152:/var/lib/ceph/d1887f28-e389-11ee-95c9-87774f69a715/crash to /home/teuthworker/archive/pdonnell-2024-03-15_13:00:41-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7602417/remote/smithi152/crash 2024-03-16T12:11:45.953 DEBUG:teuthology.orchestra.run.smithi152:> sudo tar c -f - -C /var/lib/ceph/d1887f28-e389-11ee-95c9-87774f69a715/crash -- . 2024-03-16T12:11:46.001 INFO:teuthology.orchestra.run.smithi152.stderr:tar: /var/lib/ceph/d1887f28-e389-11ee-95c9-87774f69a715/crash: Cannot open: No such file or directory 2024-03-16T12:11:46.001 INFO:teuthology.orchestra.run.smithi152.stderr:tar: Error is not recoverable: exiting now 2024-03-16T12:11:46.002 DEBUG:teuthology.misc:Transferring archived files from smithi154:/var/lib/ceph/d1887f28-e389-11ee-95c9-87774f69a715/crash to /home/teuthworker/archive/pdonnell-2024-03-15_13:00:41-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7602417/remote/smithi154/crash 2024-03-16T12:11:46.003 DEBUG:teuthology.orchestra.run.smithi154:> sudo tar c -f - -C /var/lib/ceph/d1887f28-e389-11ee-95c9-87774f69a715/crash -- . 2024-03-16T12:11:46.045 INFO:teuthology.orchestra.run.smithi154.stderr:tar: /var/lib/ceph/d1887f28-e389-11ee-95c9-87774f69a715/crash: Cannot open: No such file or directory 2024-03-16T12:11:46.045 INFO:teuthology.orchestra.run.smithi154.stderr:tar: Error is not recoverable: exiting now 2024-03-16T12:11:46.046 INFO:tasks.cephadm:Checking cluster log for badness... 2024-03-16T12:11:46.046 DEBUG:teuthology.orchestra.run.smithi029:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/d1887f28-e389-11ee-95c9-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v MON_DOWN | head -n 1 2024-03-16T12:11:46.082 INFO:teuthology.orchestra.run.smithi029.stdout:2024-03-16T11:42:27.096681+0000 mon.a (mon.0) 209 : cluster [WRN] Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-03-16T12:11:46.082 WARNING:tasks.cephadm:Found errors (ERR|WRN|SEC) in cluster log 2024-03-16T12:11:46.082 INFO:tasks.cephadm:Compressing logs... 2024-03-16T12:11:46.083 DEBUG:teuthology.orchestra.run.smithi029:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-16T12:11:46.125 DEBUG:teuthology.orchestra.run.smithi152:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-16T12:11:46.127 DEBUG:teuthology.orchestra.run.smithi154:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-16T12:11:46.152 INFO:teuthology.orchestra.run.smithi029.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-03-16T12:11:46.152 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-16T12:11:46.152 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/d1887f28-e389-11ee-95c9-87774f69a715/ceph-osd.6.log 2024-03-16T12:11:46.153 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/d1887f28-e389-11ee-95c9-87774f69a715/ceph-mon.a.log 2024-03-16T12:11:46.153 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/d1887f28-e389-11ee-95c9-87774f69a715/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/d1887f28-e389-11ee-95c9-87774f69a715/ceph.audit.log 2024-03-16T12:11:46.153 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/d1887f28-e389-11ee-95c9-87774f69a715/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/d1887f28-e389-11ee-95c9-87774f69a715/ceph-mds.g.log 2024-03-16T12:11:46.156 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/d1887f28-e389-11ee-95c9-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/d1887f28-e389-11ee-95c9-87774f69a715/ceph-mds.j.log 2024-03-16T12:11:46.156 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/d1887f28-e389-11ee-95c9-87774f69a715/ceph-mds.g.log: gzip -5 --verbose -- /var/log/ceph/d1887f28-e389-11ee-95c9-87774f69a715/ceph-osd.3.log 2024-03-16T12:11:46.156 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/d1887f28-e389-11ee-95c9-87774f69a715/ceph-mds.j.log: gzip -5 --verbose -- /var/log/ceph/d1887f28-e389-11ee-95c9-87774f69a715/ceph-mgr.x.log 2024-03-16T12:11:46.156 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/d1887f28-e389-11ee-95c9-87774f69a715/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/d1887f28-e389-11ee-95c9-87774f69a715/ceph-osd.0.log 2024-03-16T12:11:46.165 INFO:teuthology.orchestra.run.smithi154.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-16T12:11:46.165 INFO:teuthology.orchestra.run.smithi154.stderr:: No such file or directory 2024-03-16T12:11:46.165 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/d1887f28-e389-11ee-95c9-87774f69a715/ceph-mds.l.log 2024-03-16T12:11:46.165 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/d1887f28-e389-11ee-95c9-87774f69a715/ceph.audit.log 2024-03-16T12:11:46.165 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/cephadm.log: /var/log/ceph/d1887f28-e389-11ee-95c9-87774f69a715/ceph-mds.l.log: gzip -5 --verbose -- /var/log/ceph/d1887f28-e389-11ee-95c9-87774f69a715/ceph-mds.i.log 2024-03-16T12:11:46.165 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/d1887f28-e389-11ee-95c9-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/d1887f28-e389-11ee-95c9-87774f69a715/ceph-osd.11.log 2024-03-16T12:11:46.166 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/d1887f28-e389-11ee-95c9-87774f69a715/ceph-mds.i.log: gzip -5 --verbose -- /var/log/ceph/d1887f28-e389-11ee-95c9-87774f69a715/ceph-mgr.z.log 2024-03-16T12:11:46.166 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/d1887f28-e389-11ee-95c9-87774f69a715/ceph-osd.11.log: gzip -5 --verbose -- /var/log/ceph/d1887f28-e389-11ee-95c9-87774f69a715/ceph-osd.8.log 2024-03-16T12:11:46.166 INFO:teuthology.orchestra.run.smithi154.stderr: 92.4% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-16T12:11:46.166 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/d1887f28-e389-11ee-95c9-87774f69a715/ceph-osd.5.log 2024-03-16T12:11:46.166 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/d1887f28-e389-11ee-95c9-87774f69a715/ceph-mgr.z.log: /var/log/ceph/d1887f28-e389-11ee-95c9-87774f69a715/ceph-osd.8.log: gzip -5 --verbose -- /var/log/ceph/d1887f28-e389-11ee-95c9-87774f69a715/ceph-mds.f.log 2024-03-16T12:11:46.166 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/d1887f28-e389-11ee-95c9-87774f69a715/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/d1887f28-e389-11ee-95c9-87774f69a715/ceph.cephadm.log 2024-03-16T12:11:46.166 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/d1887f28-e389-11ee-95c9-87774f69a715/ceph-mds.f.log: 90.5% -- replaced with /var/log/ceph/d1887f28-e389-11ee-95c9-87774f69a715/ceph.audit.log.gz 2024-03-16T12:11:46.166 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/d1887f28-e389-11ee-95c9-87774f69a715/ceph.log 2024-03-16T12:11:46.166 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/d1887f28-e389-11ee-95c9-87774f69a715/ceph.cephadm.log: 83.8% -- replaced with /var/log/ceph/d1887f28-e389-11ee-95c9-87774f69a715/ceph.cephadm.log.gz 2024-03-16T12:11:46.167 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/d1887f28-e389-11ee-95c9-87774f69a715/ceph-osd.2.log 2024-03-16T12:11:46.167 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/d1887f28-e389-11ee-95c9-87774f69a715/ceph.log: 94.7% -- replaced with /var/log/ceph/d1887f28-e389-11ee-95c9-87774f69a715/ceph-mds.l.log.gz 2024-03-16T12:11:46.167 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/d1887f28-e389-11ee-95c9-87774f69a715/ceph-mds.c.log 2024-03-16T12:11:46.167 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/d1887f28-e389-11ee-95c9-87774f69a715/ceph-osd.2.log: 87.0% -- replaced with /var/log/ceph/d1887f28-e389-11ee-95c9-87774f69a715/ceph.log.gz 2024-03-16T12:11:46.167 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/d1887f28-e389-11ee-95c9-87774f69a715/ceph-volume.log 2024-03-16T12:11:46.169 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/d1887f28-e389-11ee-95c9-87774f69a715/ceph-mgr.x.log: 90.3% -- replaced with /var/log/ceph/d1887f28-e389-11ee-95c9-87774f69a715/ceph.audit.log.gz 2024-03-16T12:11:46.169 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/d1887f28-e389-11ee-95c9-87774f69a715/ceph.cephadm.log 2024-03-16T12:11:46.170 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/d1887f28-e389-11ee-95c9-87774f69a715/ceph-osd.0.log: 94.6% -- replaced with /var/log/ceph/d1887f28-e389-11ee-95c9-87774f69a715/ceph-mds.g.log.gz 2024-03-16T12:11:46.170 INFO:teuthology.orchestra.run.smithi029.stderr: 93.3% -- replaced with /var/log/ceph/cephadm.log.gzgzip 2024-03-16T12:11:46.170 INFO:teuthology.orchestra.run.smithi029.stderr: -5 --verbose -- /var/log/ceph/d1887f28-e389-11ee-95c9-87774f69a715/ceph.log 2024-03-16T12:11:46.170 INFO:teuthology.orchestra.run.smithi029.stderr: 94.6%/var/log/ceph/d1887f28-e389-11ee-95c9-87774f69a715/ceph.cephadm.log: -- replaced with /var/log/ceph/d1887f28-e389-11ee-95c9-87774f69a715/ceph-mds.j.log.gz 2024-03-16T12:11:46.170 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/d1887f28-e389-11ee-95c9-87774f69a715/ceph-volume.log 2024-03-16T12:11:46.170 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/d1887f28-e389-11ee-95c9-87774f69a715/ceph.log: 84.2% -- replaced with /var/log/ceph/d1887f28-e389-11ee-95c9-87774f69a715/ceph.cephadm.log.gz 2024-03-16T12:11:46.170 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/d1887f28-e389-11ee-95c9-87774f69a715/ceph-osd.9.log 2024-03-16T12:11:46.170 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/d1887f28-e389-11ee-95c9-87774f69a715/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/d1887f28-e389-11ee-95c9-87774f69a715/ceph-mds.d.log 2024-03-16T12:11:46.171 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/d1887f28-e389-11ee-95c9-87774f69a715/ceph-osd.9.log: 87.0% -- replaced with /var/log/ceph/d1887f28-e389-11ee-95c9-87774f69a715/ceph.log.gz 2024-03-16T12:11:46.171 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/d1887f28-e389-11ee-95c9-87774f69a715/ceph-mds.a.log 2024-03-16T12:11:46.171 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/d1887f28-e389-11ee-95c9-87774f69a715/ceph-mds.c.log: 92.2% -- replaced with /var/log/ceph/d1887f28-e389-11ee-95c9-87774f69a715/ceph-mgr.z.log.gz 2024-03-16T12:11:46.172 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/d1887f28-e389-11ee-95c9-87774f69a715/ceph-mon.c.log 2024-03-16T12:11:46.172 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/d1887f28-e389-11ee-95c9-87774f69a715/ceph-mds.d.log: 94.5% -- replaced with /var/log/ceph/d1887f28-e389-11ee-95c9-87774f69a715/ceph-mds.d.log.gz 2024-03-16T12:11:46.175 INFO:teuthology.orchestra.run.smithi152.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-03-16T12:11:46.175 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-16T12:11:46.175 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/d1887f28-e389-11ee-95c9-87774f69a715/ceph-osd.10.log 2024-03-16T12:11:46.175 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/d1887f28-e389-11ee-95c9-87774f69a715/ceph.audit.log 2024-03-16T12:11:46.175 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/d1887f28-e389-11ee-95c9-87774f69a715/ceph-osd.10.log: gzip -5 --verbose -- /var/log/ceph/d1887f28-e389-11ee-95c9-87774f69a715/ceph-mds.b.log 2024-03-16T12:11:46.175 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/d1887f28-e389-11ee-95c9-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/d1887f28-e389-11ee-95c9-87774f69a715/ceph-osd.4.log 2024-03-16T12:11:46.175 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/d1887f28-e389-11ee-95c9-87774f69a715/ceph-mds.b.log: gzip -5 --verbose -- /var/log/ceph/d1887f28-e389-11ee-95c9-87774f69a715/ceph-osd.1.log 2024-03-16T12:11:46.175 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/d1887f28-e389-11ee-95c9-87774f69a715/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/d1887f28-e389-11ee-95c9-87774f69a715/ceph-mds.k.log 2024-03-16T12:11:46.175 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/d1887f28-e389-11ee-95c9-87774f69a715/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/d1887f28-e389-11ee-95c9-87774f69a715/ceph-mds.h.log 2024-03-16T12:11:46.176 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/d1887f28-e389-11ee-95c9-87774f69a715/ceph-mds.k.log: gzip -5 --verbose -- /var/log/ceph/d1887f28-e389-11ee-95c9-87774f69a715/ceph-osd.7.log 2024-03-16T12:11:46.176 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/d1887f28-e389-11ee-95c9-87774f69a715/ceph-mds.e.log 2024-03-16T12:11:46.177 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/d1887f28-e389-11ee-95c9-87774f69a715/ceph-mds.h.log: /var/log/ceph/d1887f28-e389-11ee-95c9-87774f69a715/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/d1887f28-e389-11ee-95c9-87774f69a715/ceph-mgr.y.log 2024-03-16T12:11:46.177 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/d1887f28-e389-11ee-95c9-87774f69a715/ceph-mds.e.log: gzip -5 --verbose -- /var/log/ceph/d1887f28-e389-11ee-95c9-87774f69a715/ceph.cephadm.log 2024-03-16T12:11:46.177 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/d1887f28-e389-11ee-95c9-87774f69a715/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/d1887f28-e389-11ee-95c9-87774f69a715/ceph.log 2024-03-16T12:11:46.178 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/d1887f28-e389-11ee-95c9-87774f69a715/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/d1887f28-e389-11ee-95c9-87774f69a715/ceph-volume.log 2024-03-16T12:11:46.178 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/d1887f28-e389-11ee-95c9-87774f69a715/ceph.log: gzip -5 --verbose -- /var/log/ceph/d1887f28-e389-11ee-95c9-87774f69a715/ceph-mon.b.log 2024-03-16T12:11:46.178 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/d1887f28-e389-11ee-95c9-87774f69a715/ceph-mds.a.log: 91.0% -- replaced with /var/log/ceph/d1887f28-e389-11ee-95c9-87774f69a715/ceph-volume.log.gz 2024-03-16T12:11:46.179 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/d1887f28-e389-11ee-95c9-87774f69a715/ceph-volume.log: /var/log/ceph/d1887f28-e389-11ee-95c9-87774f69a715/ceph-mon.b.log: 94.6% -- replaced with /var/log/ceph/d1887f28-e389-11ee-95c9-87774f69a715/ceph-mds.k.log.gz 2024-03-16T12:11:46.179 INFO:teuthology.orchestra.run.smithi029.stderr: 94.5% -- replaced with /var/log/ceph/d1887f28-e389-11ee-95c9-87774f69a715/ceph-mds.a.log.gz 2024-03-16T12:11:46.180 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/d1887f28-e389-11ee-95c9-87774f69a715/ceph-volume.log: /var/log/ceph/d1887f28-e389-11ee-95c9-87774f69a715/ceph-mon.c.log: 91.0% -- replaced with /var/log/ceph/d1887f28-e389-11ee-95c9-87774f69a715/ceph-volume.log.gz 2024-03-16T12:11:46.183 INFO:teuthology.orchestra.run.smithi152.stderr: 92.2% -- replaced with /var/log/ceph/d1887f28-e389-11ee-95c9-87774f69a715/ceph-mgr.y.log.gz 2024-03-16T12:11:46.237 INFO:teuthology.orchestra.run.smithi152.stderr: 92.1% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-16T12:11:46.298 INFO:teuthology.orchestra.run.smithi152.stderr: 83.9% -- replaced with /var/log/ceph/d1887f28-e389-11ee-95c9-87774f69a715/ceph.cephadm.log.gz 2024-03-16T12:11:46.356 INFO:teuthology.orchestra.run.smithi152.stderr: 87.0% -- replaced with /var/log/ceph/d1887f28-e389-11ee-95c9-87774f69a715/ceph.log.gz 2024-03-16T12:11:46.430 INFO:teuthology.orchestra.run.smithi152.stderr: 90.4% -- replaced with /var/log/ceph/d1887f28-e389-11ee-95c9-87774f69a715/ceph.audit.log.gz 2024-03-16T12:11:46.446 INFO:teuthology.orchestra.run.smithi029.stderr: 89.9% -- replaced with /var/log/ceph/d1887f28-e389-11ee-95c9-87774f69a715/ceph-mgr.x.log.gz 2024-03-16T12:11:46.665 INFO:teuthology.orchestra.run.smithi152.stderr: 90.9% -- replaced with /var/log/ceph/d1887f28-e389-11ee-95c9-87774f69a715/ceph-volume.log.gz 2024-03-16T12:11:46.680 INFO:teuthology.orchestra.run.smithi154.stderr: 95.1% -- replaced with /var/log/ceph/d1887f28-e389-11ee-95c9-87774f69a715/ceph-mds.f.log.gz 2024-03-16T12:11:46.760 INFO:teuthology.orchestra.run.smithi154.stderr: 91.9% -- replaced with /var/log/ceph/d1887f28-e389-11ee-95c9-87774f69a715/ceph-mon.c.log.gz 2024-03-16T12:11:47.157 INFO:teuthology.orchestra.run.smithi152.stderr: 94.4% -- replaced with /var/log/ceph/d1887f28-e389-11ee-95c9-87774f69a715/ceph-mds.e.log.gz 2024-03-16T12:11:48.787 INFO:teuthology.orchestra.run.smithi029.stderr: 91.5% -- replaced with /var/log/ceph/d1887f28-e389-11ee-95c9-87774f69a715/ceph-mon.a.log.gz 2024-03-16T12:11:54.642 INFO:teuthology.orchestra.run.smithi152.stderr: 92.0% -- replaced with /var/log/ceph/d1887f28-e389-11ee-95c9-87774f69a715/ceph-mon.b.log.gz 2024-03-16T12:12:01.681 INFO:teuthology.orchestra.run.smithi154.stderr: 94.9% -- replaced with /var/log/ceph/d1887f28-e389-11ee-95c9-87774f69a715/ceph-mds.c.log.gz 2024-03-16T12:12:04.934 INFO:teuthology.orchestra.run.smithi029.stderr: 91.6% -- replaced with /var/log/ceph/d1887f28-e389-11ee-95c9-87774f69a715/ceph-osd.9.log.gz 2024-03-16T12:12:05.796 INFO:teuthology.orchestra.run.smithi154.stderr: 95.2% -- replaced with /var/log/ceph/d1887f28-e389-11ee-95c9-87774f69a715/ceph-mds.i.log.gz 2024-03-16T12:12:09.302 INFO:teuthology.orchestra.run.smithi154.stderr: 91.3% -- replaced with /var/log/ceph/d1887f28-e389-11ee-95c9-87774f69a715/ceph-osd.11.log.gz 2024-03-16T12:12:10.960 INFO:teuthology.orchestra.run.smithi029.stderr: 91.7% -- replaced with /var/log/ceph/d1887f28-e389-11ee-95c9-87774f69a715/ceph-osd.0.log.gz 2024-03-16T12:12:11.025 INFO:teuthology.orchestra.run.smithi029.stderr: 91.4% -- replaced with /var/log/ceph/d1887f28-e389-11ee-95c9-87774f69a715/ceph-osd.6.log.gz 2024-03-16T12:12:14.938 INFO:teuthology.orchestra.run.smithi154.stderr: 91.6% -- replaced with /var/log/ceph/d1887f28-e389-11ee-95c9-87774f69a715/ceph-osd.2.log.gz 2024-03-16T12:12:16.798 INFO:teuthology.orchestra.run.smithi029.stderr: 91.5% -- replaced with /var/log/ceph/d1887f28-e389-11ee-95c9-87774f69a715/ceph-osd.3.log.gz 2024-03-16T12:12:16.799 INFO:teuthology.orchestra.run.smithi029.stderr: 2024-03-16T12:12:16.800 INFO:teuthology.orchestra.run.smithi029.stderr:real 0m30.662s 2024-03-16T12:12:16.800 INFO:teuthology.orchestra.run.smithi029.stderr:user 1m38.612s 2024-03-16T12:12:16.800 INFO:teuthology.orchestra.run.smithi029.stderr:sys 0m2.897s 2024-03-16T12:12:18.321 INFO:teuthology.orchestra.run.smithi154.stderr: 91.3% -- replaced with /var/log/ceph/d1887f28-e389-11ee-95c9-87774f69a715/ceph-osd.8.log.gz 2024-03-16T12:12:24.537 INFO:teuthology.orchestra.run.smithi154.stderr: 91.2% -- replaced with /var/log/ceph/d1887f28-e389-11ee-95c9-87774f69a715/ceph-osd.5.log.gz 2024-03-16T12:12:24.538 INFO:teuthology.orchestra.run.smithi154.stderr: 2024-03-16T12:12:24.538 INFO:teuthology.orchestra.run.smithi154.stderr:real 0m38.398s 2024-03-16T12:12:24.539 INFO:teuthology.orchestra.run.smithi154.stderr:user 2m33.960s 2024-03-16T12:12:24.539 INFO:teuthology.orchestra.run.smithi154.stderr:sys 0m4.517s 2024-03-16T12:14:30.971 INFO:teuthology.orchestra.run.smithi152.stderr: 95.7% -- replaced with /var/log/ceph/d1887f28-e389-11ee-95c9-87774f69a715/ceph-mds.h.log.gz 2024-03-16T12:14:45.115 INFO:teuthology.orchestra.run.smithi152.stderr: 91.4% -- replaced with /var/log/ceph/d1887f28-e389-11ee-95c9-87774f69a715/ceph-osd.4.log.gz 2024-03-16T12:15:01.738 INFO:teuthology.orchestra.run.smithi152.stderr: 91.6% -- replaced with /var/log/ceph/d1887f28-e389-11ee-95c9-87774f69a715/ceph-osd.1.log.gz 2024-03-16T12:15:15.497 INFO:teuthology.orchestra.run.smithi152.stderr: 91.3% -- replaced with /var/log/ceph/d1887f28-e389-11ee-95c9-87774f69a715/ceph-osd.10.log.gz 2024-03-16T12:15:53.706 INFO:teuthology.orchestra.run.smithi152.stderr: 91.3% -- replaced with /var/log/ceph/d1887f28-e389-11ee-95c9-87774f69a715/ceph-osd.7.log.gz 2024-03-16T12:21:44.855 INFO:teuthology.orchestra.run.smithi152.stderr: 93.4% -- replaced with /var/log/ceph/d1887f28-e389-11ee-95c9-87774f69a715/ceph-mds.b.log.gz 2024-03-16T12:21:44.930 INFO:teuthology.orchestra.run.smithi152.stderr: 2024-03-16T12:21:44.931 INFO:teuthology.orchestra.run.smithi152.stderr:real 9m58.790s 2024-03-16T12:21:44.931 INFO:teuthology.orchestra.run.smithi152.stderr:user 7m29.823s 2024-03-16T12:21:44.931 INFO:teuthology.orchestra.run.smithi152.stderr:sys 0m18.356s 2024-03-16T12:21:44.931 INFO:tasks.cephadm:Archiving logs... 2024-03-16T12:21:44.932 DEBUG:teuthology.misc:Transferring archived files from smithi029:/var/log/ceph to /home/teuthworker/archive/pdonnell-2024-03-15_13:00:41-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7602417/remote/smithi029/log 2024-03-16T12:21:44.933 DEBUG:teuthology.orchestra.run.smithi029:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-16T12:22:01.493 DEBUG:teuthology.misc:Transferring archived files from smithi152:/var/log/ceph to /home/teuthworker/archive/pdonnell-2024-03-15_13:00:41-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7602417/remote/smithi152/log 2024-03-16T12:22:01.494 DEBUG:teuthology.orchestra.run.smithi152:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-16T12:23:15.207 DEBUG:teuthology.misc:Transferring archived files from smithi154:/var/log/ceph to /home/teuthworker/archive/pdonnell-2024-03-15_13:00:41-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7602417/remote/smithi154/log 2024-03-16T12:23:15.208 DEBUG:teuthology.orchestra.run.smithi154:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-16T12:23:37.212 INFO:tasks.cephadm:Removing cluster... 2024-03-16T12:23:37.213 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid d1887f28-e389-11ee-95c9-87774f69a715 --force 2024-03-16T12:23:37.410 INFO:teuthology.orchestra.run.smithi029.stdout:Deleting cluster with fsid: d1887f28-e389-11ee-95c9-87774f69a715 2024-03-16T12:23:37.818 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid d1887f28-e389-11ee-95c9-87774f69a715 --force 2024-03-16T12:23:38.246 INFO:teuthology.orchestra.run.smithi152.stdout:Deleting cluster with fsid: d1887f28-e389-11ee-95c9-87774f69a715 2024-03-16T12:23:38.912 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid d1887f28-e389-11ee-95c9-87774f69a715 --force 2024-03-16T12:23:39.103 INFO:teuthology.orchestra.run.smithi154.stdout:Deleting cluster with fsid: d1887f28-e389-11ee-95c9-87774f69a715 2024-03-16T12:23:39.552 INFO:tasks.cephadm:Removing cephadm ... 2024-03-16T12:23:39.553 DEBUG:teuthology.orchestra.run.smithi029:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-16T12:23:39.571 DEBUG:teuthology.orchestra.run.smithi152:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-16T12:23:39.588 DEBUG:teuthology.orchestra.run.smithi154:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-16T12:23:39.603 INFO:tasks.cephadm:Teardown complete 2024-03-16T12:23:39.603 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-16T12:23:39.614 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-16T12:23:39.614 DEBUG:teuthology.orchestra.run.smithi029:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-16T12:23:39.617 DEBUG:teuthology.orchestra.run.smithi152:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-16T12:23:39.630 DEBUG:teuthology.orchestra.run.smithi154:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-16T12:23:39.764 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-16T12:23:39.775 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-16T12:23:39.775 DEBUG:teuthology.orchestra.run.smithi029:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-16T12:23:39.778 DEBUG:teuthology.orchestra.run.smithi152:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-16T12:23:39.780 DEBUG:teuthology.orchestra.run.smithi154:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-16T12:23:39.792 INFO:teuthology.orchestra.run.smithi029.stderr:bash: line 1: ntpq: command not found 2024-03-16T12:23:39.793 INFO:teuthology.orchestra.run.smithi152.stderr:bash: line 1: ntpq: command not found 2024-03-16T12:23:39.794 INFO:teuthology.orchestra.run.smithi154.stderr:bash: line 1: ntpq: command not found 2024-03-16T12:23:39.796 INFO:teuthology.orchestra.run.smithi029.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-16T12:23:39.797 INFO:teuthology.orchestra.run.smithi029.stdout:=============================================================================== 2024-03-16T12:23:39.797 INFO:teuthology.orchestra.run.smithi029.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 22 +982us[ +983us] +/- 125ms 2024-03-16T12:23:39.797 INFO:teuthology.orchestra.run.smithi029.stdout:^* hv02.front.sepia.ceph.com 3 6 377 20 -182us[ -182us] +/- 77ms 2024-03-16T12:23:39.797 INFO:teuthology.orchestra.run.smithi029.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 16 -446us[ -446us] +/- 88ms 2024-03-16T12:23:39.797 INFO:teuthology.orchestra.run.smithi029.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-16T12:23:39.799 INFO:teuthology.orchestra.run.smithi154.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-16T12:23:39.799 INFO:teuthology.orchestra.run.smithi154.stdout:=============================================================================== 2024-03-16T12:23:39.799 INFO:teuthology.orchestra.run.smithi154.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 13 +988us[ +988us] +/- 125ms 2024-03-16T12:23:39.799 INFO:teuthology.orchestra.run.smithi154.stdout:^* hv02.front.sepia.ceph.com 3 6 377 13 -181us[ -167us] +/- 77ms 2024-03-16T12:23:39.799 INFO:teuthology.orchestra.run.smithi154.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 16 -455us[ -442us] +/- 88ms 2024-03-16T12:23:39.799 INFO:teuthology.orchestra.run.smithi154.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-16T12:23:39.910 INFO:teuthology.orchestra.run.smithi152.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-16T12:23:39.911 INFO:teuthology.orchestra.run.smithi152.stdout:=============================================================================== 2024-03-16T12:23:39.911 INFO:teuthology.orchestra.run.smithi152.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 17 +1005us[+1005us] +/- 125ms 2024-03-16T12:23:39.911 INFO:teuthology.orchestra.run.smithi152.stdout:^* hv02.front.sepia.ceph.com 3 6 377 15 -184us[ -185us] +/- 77ms 2024-03-16T12:23:39.911 INFO:teuthology.orchestra.run.smithi152.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 19 -429us[ -429us] +/- 88ms 2024-03-16T12:23:39.911 INFO:teuthology.orchestra.run.smithi152.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-16T12:23:39.913 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-16T12:23:39.923 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-16T12:23:39.924 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-16T12:23:39.933 DEBUG:teuthology.orchestra.run.smithi029:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-16T12:23:40.021 DEBUG:teuthology.orchestra.run.smithi152:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-16T12:23:40.154 DEBUG:teuthology.orchestra.run.smithi154:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-16T12:23:40.226 DEBUG:teuthology.orchestra.run.smithi029:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-16T12:23:40.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-16T12:23:40.259 DEBUG:teuthology.orchestra.run.smithi152:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-16T12:23:40.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-16T12:23:40.294 DEBUG:teuthology.orchestra.run.smithi154:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-16T12:23:40.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-16T12:23:40.324 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-16T12:23:40.338 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-16T12:23:40.349 INFO:teuthology.task.internal:Duration was 3137.683532 seconds 2024-03-16T12:23:40.349 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-16T12:23:40.362 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-16T12:23:40.362 DEBUG:teuthology.orchestra.run.smithi029:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-16T12:23:40.365 DEBUG:teuthology.orchestra.run.smithi152:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-16T12:23:40.367 DEBUG:teuthology.orchestra.run.smithi154:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-16T12:23:40.409 INFO:teuthology.orchestra.run.smithi029.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-16T12:23:40.410 INFO:teuthology.orchestra.run.smithi154.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-16T12:23:40.426 INFO:teuthology.orchestra.run.smithi152.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-16T12:23:40.938 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-16T12:23:40.938 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi029.front.sepia.ceph.com 2024-03-16T12:23:40.938 DEBUG:teuthology.orchestra.run.smithi029:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-16T12:23:40.962 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi152.front.sepia.ceph.com 2024-03-16T12:23:40.963 DEBUG:teuthology.orchestra.run.smithi152:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-16T12:23:40.992 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi154.front.sepia.ceph.com 2024-03-16T12:23:40.992 DEBUG:teuthology.orchestra.run.smithi154:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-16T12:23:41.015 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-16T12:23:41.015 DEBUG:teuthology.orchestra.run.smithi029:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-16T12:23:41.018 DEBUG:teuthology.orchestra.run.smithi152:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-16T12:23:41.036 DEBUG:teuthology.orchestra.run.smithi154:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-16T12:23:41.084 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-16T12:23:41.085 DEBUG:teuthology.orchestra.run.smithi029:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-16T12:23:41.092 DEBUG:teuthology.orchestra.run.smithi152:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-16T12:23:41.116 DEBUG:teuthology.orchestra.run.smithi154:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-16T12:23:41.482 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-16T12:23:41.497 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-16T12:23:41.497 DEBUG:teuthology.orchestra.run.smithi029:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-16T12:23:41.525 DEBUG:teuthology.orchestra.run.smithi152:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-16T12:23:41.566 DEBUG:teuthology.orchestra.run.smithi154:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-16T12:23:41.595 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-16T12:23:41.607 DEBUG:teuthology.orchestra.run.smithi029:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-16T12:23:41.609 DEBUG:teuthology.orchestra.run.smithi152:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-16T12:23:41.611 DEBUG:teuthology.orchestra.run.smithi154:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-16T12:23:41.632 INFO:teuthology.orchestra.run.smithi029.stdout:kernel.core_pattern = core 2024-03-16T12:23:41.646 INFO:teuthology.orchestra.run.smithi152.stdout:kernel.core_pattern = core 2024-03-16T12:23:41.662 INFO:teuthology.orchestra.run.smithi154.stdout:kernel.core_pattern = core 2024-03-16T12:23:41.705 DEBUG:teuthology.orchestra.run.smithi029:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-16T12:23:41.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-16T12:23:41.720 DEBUG:teuthology.orchestra.run.smithi152:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-16T12:23:41.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-16T12:23:41.734 DEBUG:teuthology.orchestra.run.smithi154:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-16T12:23:41.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-16T12:23:41.760 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-16T12:23:41.772 INFO:teuthology.task.internal:Transferring archived files... 2024-03-16T12:23:41.773 DEBUG:teuthology.misc:Transferring archived files from smithi029:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2024-03-15_13:00:41-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7602417/remote/smithi029 2024-03-16T12:23:41.773 DEBUG:teuthology.orchestra.run.smithi029:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-16T12:23:41.818 DEBUG:teuthology.misc:Transferring archived files from smithi152:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2024-03-15_13:00:41-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7602417/remote/smithi152 2024-03-16T12:23:41.818 DEBUG:teuthology.orchestra.run.smithi152:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-16T12:23:41.857 DEBUG:teuthology.misc:Transferring archived files from smithi154:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2024-03-15_13:00:41-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7602417/remote/smithi154 2024-03-16T12:23:41.858 DEBUG:teuthology.orchestra.run.smithi154:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-16T12:23:41.896 INFO:teuthology.task.internal:Removing archive directory... 2024-03-16T12:23:41.897 DEBUG:teuthology.orchestra.run.smithi029:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-16T12:23:41.899 DEBUG:teuthology.orchestra.run.smithi152:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-16T12:23:41.901 DEBUG:teuthology.orchestra.run.smithi154:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-16T12:23:41.946 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-16T12:23:41.960 INFO:teuthology.task.internal:Not uploading archives. 2024-03-16T12:23:41.960 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-16T12:23:41.971 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-16T12:23:41.972 DEBUG:teuthology.orchestra.run.smithi029:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-16T12:23:41.974 DEBUG:teuthology.orchestra.run.smithi152:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-16T12:23:41.976 DEBUG:teuthology.orchestra.run.smithi154:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-16T12:23:41.986 INFO:teuthology.orchestra.run.smithi029.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 16 12:23 /home/ubuntu/cephtest 2024-03-16T12:23:41.988 INFO:teuthology.orchestra.run.smithi152.stdout: 262162 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 16 12:23 /home/ubuntu/cephtest 2024-03-16T12:23:42.000 INFO:teuthology.orchestra.run.smithi154.stdout: 262162 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 16 12:23 /home/ubuntu/cephtest 2024-03-16T12:23:42.001 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-16T12:23:42.067 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-16T12:23:42.091 INFO:teuthology.nuke:Checking targets against current locks 2024-03-16T12:23:42.118 DEBUG:teuthology.nuke:shortname: smithi029 2024-03-16T12:23:42.119 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-16T12:23:42.146 DEBUG:teuthology.nuke:shortname: smithi152 2024-03-16T12:23:42.147 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-16T12:23:42.153 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi029.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-03-15_13:00:41-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7602417', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-16 11:22:07.987657', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDgj7nLjz/eATmFxlBE8SkOj64J+ZHgb7KgpfapFpAiVDePemMgh+wEjzllGImxykgPeb9oHGgbFSgKSubGOT4I='} 2024-03-16T12:23:42.172 DEBUG:teuthology.nuke:shortname: smithi154 2024-03-16T12:23:42.172 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-16T12:23:42.176 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi152.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-03-15_13:00:41-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7602417', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-16 11:22:07.990467', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBODmpUUIT8s4S2XV01SxUGYFTYBPzWyZZP6Gtz6xLgKr1GoKR44RiyUzDqs9CeqaDwCNBMB8+wW3db/uSS77mp4='} 2024-03-16T12:23:42.206 INFO:teuthology.orchestra.console.smithi029:Power off 2024-03-16T12:23:42.206 DEBUG:teuthology.orchestra.console.smithi029:pexpect command: ipmitool -H smithi029.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-16T12:23:42.219 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi154.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-03-15_13:00:41-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7602417', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-16 11:22:07.989189', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKELybuyIauRgYEU3ET5nVWAUfkL8SW3exbDQM8gO4XU8gg28OxV04H1Uz80XrozUoFzeXwIb3YOjjp4czJiImQ='} 2024-03-16T12:23:42.231 DEBUG:teuthology.orchestra.console.smithi029:power off output: Chassis Power Control: Down/Off 2024-03-16T12:23:42.231 DEBUG:teuthology.orchestra.console.smithi029:pexpect command: ipmitool -H smithi029.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-16T12:23:42.251 INFO:teuthology.orchestra.console.smithi152:Power off 2024-03-16T12:23:42.252 DEBUG:teuthology.orchestra.console.smithi152:pexpect command: ipmitool -H smithi152.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-16T12:23:42.267 DEBUG:teuthology.orchestra.console.smithi029:check power output: Chassis Power is on 2024-03-16T12:23:42.278 DEBUG:teuthology.orchestra.console.smithi152:power off output: Chassis Power Control: Down/Off 2024-03-16T12:23:42.278 DEBUG:teuthology.orchestra.console.smithi152:pexpect command: ipmitool -H smithi152.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-16T12:23:42.296 INFO:teuthology.orchestra.console.smithi154:Power off 2024-03-16T12:23:42.297 DEBUG:teuthology.orchestra.console.smithi154:pexpect command: ipmitool -H smithi154.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-16T12:23:42.306 DEBUG:teuthology.orchestra.console.smithi152:check power output: Chassis Power is on 2024-03-16T12:23:42.323 DEBUG:teuthology.orchestra.console.smithi154:power off output: Chassis Power Control: Down/Off 2024-03-16T12:23:42.323 DEBUG:teuthology.orchestra.console.smithi154:pexpect command: ipmitool -H smithi154.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-16T12:23:42.351 DEBUG:teuthology.orchestra.console.smithi154:check power output: Chassis Power is on 2024-03-16T12:23:46.268 DEBUG:teuthology.orchestra.console.smithi029:pexpect command: ipmitool -H smithi029.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-16T12:23:46.307 DEBUG:teuthology.orchestra.console.smithi152:pexpect command: ipmitool -H smithi152.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-16T12:23:46.353 DEBUG:teuthology.orchestra.console.smithi154:pexpect command: ipmitool -H smithi154.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-16T12:23:46.379 DEBUG:teuthology.orchestra.console.smithi029:check power output: Chassis Power is on 2024-03-16T12:23:46.418 DEBUG:teuthology.orchestra.console.smithi152:check power output: Chassis Power is on 2024-03-16T12:23:46.465 DEBUG:teuthology.orchestra.console.smithi154:check power output: Chassis Power is on 2024-03-16T12:23:50.382 DEBUG:teuthology.orchestra.console.smithi029:pexpect command: ipmitool -H smithi029.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-16T12:23:50.419 DEBUG:teuthology.orchestra.console.smithi152:pexpect command: ipmitool -H smithi152.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-16T12:23:50.466 DEBUG:teuthology.orchestra.console.smithi154:pexpect command: ipmitool -H smithi154.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-16T12:23:50.497 DEBUG:teuthology.orchestra.console.smithi029:check power output: Chassis Power is off 2024-03-16T12:23:50.530 DEBUG:teuthology.orchestra.console.smithi152:check power output: Chassis Power is on 2024-03-16T12:23:50.577 DEBUG:teuthology.orchestra.console.smithi154:check power output: Chassis Power is off 2024-03-16T12:23:50.598 INFO:teuthology.orchestra.console.smithi029:Power off completed 2024-03-16T12:23:50.678 INFO:teuthology.orchestra.console.smithi154:Power off completed 2024-03-16T12:23:54.534 DEBUG:teuthology.orchestra.console.smithi152:pexpect command: ipmitool -H smithi152.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-16T12:23:54.650 DEBUG:teuthology.orchestra.console.smithi152:check power output: Chassis Power is off 2024-03-16T12:23:54.751 INFO:teuthology.orchestra.console.smithi152:Power off completed 2024-03-16T12:23:54.858 INFO:teuthology.run:Summary data: description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/suites/dbench}} duration: 3137.683532476425 failure_reason: 'Command failed (workunit test suites/dbench.sh) on smithi029 with status 1: ''mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=3d78f5ba07bf18880172c07d4d7b4ac00c9d30af TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh''' flavor: default owner: scheduled_pdonnell@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=6d3a9426499e4fb39f7dd60177d4bf93 status: fail success: false 2024-03-16T12:23:54.858 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-16T12:23:54.957 INFO:teuthology.run:FAIL